{"action":"opened","number":1001,"pull_request":{"url":"https://api.github.com/repos/gephi/gephi/pulls/1001","id":27865999,"html_url":"https://github.com/gephi/gephi/pull/1001","diff_url":"https://github.com/gephi/gephi/pull/1001.diff","patch_url":"https://github.com/gephi/gephi/pull/1001.patch","issue_url":"https://api.github.com/repos/gephi/gephi/issues/1001","number":1001,"state":"open","locked":false,"title":"Speed up CSV import by 90%","user":{"login":"smowton","id":283483,"avatar_url":"https://avatars.githubusercontent.com/u/283483?v=3","gravatar_id":"","url":"https://api.github.com/users/smowton","html_url":"https://github.com/smowton","followers_url":"https://api.github.com/users/smowton/followers","following_url":"https://api.github.com/users/smowton/following{/other_user}","gists_url":"https://api.github.com/users/smowton/gists{/gist_id}","starred_url":"https://api.github.com/users/smowton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smowton/subscriptions","organizations_url":"https://api.github.com/users/smowton/orgs","repos_url":"https://api.github.com/users/smowton/repos","events_url":"https://api.github.com/users/smowton/events{/privacy}","received_events_url":"https://api.github.com/users/smowton/received_events","type":"User","site_admin":false},"body":"Hi Gephi devs,\r\n\r\nI'd like to present a patchset that drastically reduces the time taken to import CSV data into Gephi for potential inclusion. The motivation: some of my users like to work with large (10M+ records) data sets, but found that importing through the Data Laboratory Nodes table CSV import tool took an hour or more, consisting of 10 minutes reading the data and handling attribute events and the rest of the hour waiting for the GUI thread to clear its backlog of work. This patchset reduces that time by around 90% to around 3 or 4 minutes on my machine, and reduces CPU usage from ~300% to ~120%.\r\n\r\nIt only covers our particular use case, and there are certainly others (for example, it does not treat the edge table or other methods of data import, such as from a database); however, I have aimed to write principled code that can easily be generalised and extended as required to accelerate other cases. I have written against 0.8.2 since the Data Lab component is currently disabled in head, but the same patches will hopefully be useful there. I have also tried to write such that individual commits can be cherry-picked if you're not happy with the branch as a whole.\r\n\r\nPer-patch summary of changes:\r\n\r\n3cc05a3: Log uncaught exceptions in the CSV import thread (and other threads).\r\n\r\n7dc2d26 and aae4e30: Disable the posting AWT refresh events during import. The receiving thread discarded them anyhow, and it cost a core running at 100% to handle the constant stream of events.\r\n\r\n0ab5549: IndexedAttributeModel maintained maps of weak references to *other* weak references to the same object, for no obvious reason. They were time-consuming to create due to synchronisation per op and made GC very slow (I think it doesn't handle weakrefs very well). This is simply stubbed out.\r\n\r\nb4a1ee4 and 2ae2987: Every node creation was making several GraphController::getModel calls (which involve synchronization) and GraphModel::getGraph calls (which allocate a wrapper *and* a weak reference). Switch to making these calls once per import, rather than once per row.\r\n\r\nf34abfa, a0edc0f and a28f309: Introduce batching in communication between the import thread and the AttributeEventManager. These two threads were wasting a *lot* of time constantly fighting for the event LinkedBlockingQueue and the EventManager's mutex. Reduce the synchronisation frequency by introducing event batching: a MassAttributeUpdate object is created and passed to functions that generate attribute events (of course setValue, but also constructors and initialisation functions). The event queue is modified to carry arrays-of-events rather than events themselves. This may render the event compression mechanism redundant, but I have left it in place for now to simplify the changeset.\r\n\r\n72c3ed8: Dramatically reduce the time spent drawing the data table by introducing a hacked variant of Netbeans' Outline table, which spends 10s of minutes calculating a column's preferred width when the table is large. This is a bit of a grim solution, as the relevant function cannot be customised easily; a cleaner solution requires a small patch to netbeans, for which I have filed a bug here: https://netbeans.org/bugzilla/show_bug.cgi?id=249864\r\n\r\nLet me know what you think! Always happy to help adapt the changes and get them into a mergeable state.","created_at":"2015-01-22T18:35:03Z","updated_at":"2015-01-22T18:35:04Z","closed_at":null,"merged_at":null,"merge_commit_sha":null,"assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/gephi/gephi/pulls/1001/commits","review_comments_url":"https://api.github.com/repos/gephi/gephi/pulls/1001/comments","review_comment_url":"https://api.github.com/repos/gephi/gephi/pulls/comments/{number}","comments_url":"https://api.github.com/repos/gephi/gephi/issues/1001/comments","statuses_url":"https://api.github.com/repos/gephi/gephi/statuses/a0edc0f2993d58c76cb35a9e00a70f6cc2902e78","head":{"label":"smowton:csv-import-performance","ref":"csv-import-performance","sha":"a0edc0f2993d58c76cb35a9e00a70f6cc2902e78","user":{"login":"smowton","id":283483,"avatar_url":"https://avatars.githubusercontent.com/u/283483?v=3","gravatar_id":"","url":"https://api.github.com/users/smowton","html_url":"https://github.com/smowton","followers_url":"https://api.github.com/users/smowton/followers","following_url":"https://api.github.com/users/smowton/following{/other_user}","gists_url":"https://api.github.com/users/smowton/gists{/gist_id}","starred_url":"https://api.github.com/users/smowton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smowton/subscriptions","organizations_url":"https://api.github.com/users/smowton/orgs","repos_url":"https://api.github.com/users/smowton/repos","events_url":"https://api.github.com/users/smowton/events{/privacy}","received_events_url":"https://api.github.com/users/smowton/received_events","type":"User","site_admin":false},"repo":{"id":29692062,"name":"gephi","full_name":"smowton/gephi","owner":{"login":"smowton","id":283483,"avatar_url":"https://avatars.githubusercontent.com/u/283483?v=3","gravatar_id":"","url":"https://api.github.com/users/smowton","html_url":"https://github.com/smowton","followers_url":"https://api.github.com/users/smowton/followers","following_url":"https://api.github.com/users/smowton/following{/other_user}","gists_url":"https://api.github.com/users/smowton/gists{/gist_id}","starred_url":"https://api.github.com/users/smowton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smowton/subscriptions","organizations_url":"https://api.github.com/users/smowton/orgs","repos_url":"https://api.github.com/users/smowton/repos","events_url":"https://api.github.com/users/smowton/events{/privacy}","received_events_url":"https://api.github.com/users/smowton/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/smowton/gephi","description":"trunk","fork":true,"url":"https://api.github.com/repos/smowton/gephi","forks_url":"https://api.github.com/repos/smowton/gephi/forks","keys_url":"https://api.github.com/repos/smowton/gephi/keys{/key_id}","collaborators_url":"https://api.github.com/repos/smowton/gephi/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/smowton/gephi/teams","hooks_url":"https://api.github.com/repos/smowton/gephi/hooks","issue_events_url":"https://api.github.com/repos/smowton/gephi/issues/events{/number}","events_url":"https://api.github.com/repos/smowton/gephi/events","assignees_url":"https://api.github.com/repos/smowton/gephi/assignees{/user}","branches_url":"https://api.github.com/repos/smowton/gephi/branches{/branch}","tags_url":"https://api.github.com/repos/smowton/gephi/tags","blobs_url":"https://api.github.com/repos/smowton/gephi/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/smowton/gephi/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/smowton/gephi/git/refs{/sha}","trees_url":"https://api.github.com/repos/smowton/gephi/git/trees{/sha}","statuses_url":"https://api.github.com/repos/smowton/gephi/statuses/{sha}","languages_url":"https://api.github.com/repos/smowton/gephi/languages","stargazers_url":"https://api.github.com/repos/smowton/gephi/stargazers","contributors_url":"https://api.github.com/repos/smowton/gephi/contributors","subscribers_url":"https://api.github.com/repos/smowton/gephi/subscribers","subscription_url":"https://api.github.com/repos/smowton/gephi/subscription","commits_url":"https://api.github.com/repos/smowton/gephi/commits{/sha}","git_commits_url":"https://api.github.com/repos/smowton/gephi/git/commits{/sha}","comments_url":"https://api.github.com/repos/smowton/gephi/comments{/number}","issue_comment_url":"https://api.github.com/repos/smowton/gephi/issues/comments/{number}","contents_url":"https://api.github.com/repos/smowton/gephi/contents/{+path}","compare_url":"https://api.github.com/repos/smowton/gephi/compare/{base}...{head}","merges_url":"https://api.github.com/repos/smowton/gephi/merges","archive_url":"https://api.github.com/repos/smowton/gephi/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/smowton/gephi/downloads","issues_url":"https://api.github.com/repos/smowton/gephi/issues{/number}","pulls_url":"https://api.github.com/repos/smowton/gephi/pulls{/number}","milestones_url":"https://api.github.com/repos/smowton/gephi/milestones{/number}","notifications_url":"https://api.github.com/repos/smowton/gephi/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/smowton/gephi/labels{/name}","releases_url":"https://api.github.com/repos/smowton/gephi/releases{/id}","created_at":"2015-01-22T17:59:00Z","updated_at":"2015-01-22T17:59:11Z","pushed_at":"2015-01-22T18:00:36Z","git_url":"git://github.com/smowton/gephi.git","ssh_url":"git@github.com:smowton/gephi.git","clone_url":"https://github.com/smowton/gephi.git","svn_url":"https://github.com/smowton/gephi","homepage":"","size":103049,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"gephi:0.8.2","ref":"0.8.2","sha":"54025c0cfbcc65d0c7d3293f2c981e5adae7107e","user":{"login":"gephi","id":1144411,"avatar_url":"https://avatars.githubusercontent.com/u/1144411?v=3","gravatar_id":"","url":"https://api.github.com/users/gephi","html_url":"https://github.com/gephi","followers_url":"https://api.github.com/users/gephi/followers","following_url":"https://api.github.com/users/gephi/following{/other_user}","gists_url":"https://api.github.com/users/gephi/gists{/gist_id}","starred_url":"https://api.github.com/users/gephi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gephi/subscriptions","organizations_url":"https://api.github.com/users/gephi/orgs","repos_url":"https://api.github.com/users/gephi/repos","events_url":"https://api.github.com/users/gephi/events{/privacy}","received_events_url":"https://api.github.com/users/gephi/received_events","type":"Organization","site_admin":false},"repo":{"id":2624521,"name":"gephi","full_name":"gephi/gephi","owner":{"login":"gephi","id":1144411,"avatar_url":"https://avatars.githubusercontent.com/u/1144411?v=3","gravatar_id":"","url":"https://api.github.com/users/gephi","html_url":"https://github.com/gephi","followers_url":"https://api.github.com/users/gephi/followers","following_url":"https://api.github.com/users/gephi/following{/other_user}","gists_url":"https://api.github.com/users/gephi/gists{/gist_id}","starred_url":"https://api.github.com/users/gephi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gephi/subscriptions","organizations_url":"https://api.github.com/users/gephi/orgs","repos_url":"https://api.github.com/users/gephi/repos","events_url":"https://api.github.com/users/gephi/events{/privacy}","received_events_url":"https://api.github.com/users/gephi/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/gephi/gephi","description":"trunk","fork":false,"url":"https://api.github.com/repos/gephi/gephi","forks_url":"https://api.github.com/repos/gephi/gephi/forks","keys_url":"https://api.github.com/repos/gephi/gephi/keys{/key_id}","collaborators_url":"https://api.github.com/repos/gephi/gephi/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/gephi/gephi/teams","hooks_url":"https://api.github.com/repos/gephi/gephi/hooks","issue_events_url":"https://api.github.com/repos/gephi/gephi/issues/events{/number}","events_url":"https://api.github.com/repos/gephi/gephi/events","assignees_url":"https://api.github.com/repos/gephi/gephi/assignees{/user}","branches_url":"https://api.github.com/repos/gephi/gephi/branches{/branch}","tags_url":"https://api.github.com/repos/gephi/gephi/tags","blobs_url":"https://api.github.com/repos/gephi/gephi/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/gephi/gephi/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/gephi/gephi/git/refs{/sha}","trees_url":"https://api.github.com/repos/gephi/gephi/git/trees{/sha}","statuses_url":"https://api.github.com/repos/gephi/gephi/statuses/{sha}","languages_url":"https://api.github.com/repos/gephi/gephi/languages","stargazers_url":"https://api.github.com/repos/gephi/gephi/stargazers","contributors_url":"https://api.github.com/repos/gephi/gephi/contributors","subscribers_url":"https://api.github.com/repos/gephi/gephi/subscribers","subscription_url":"https://api.github.com/repos/gephi/gephi/subscription","commits_url":"https://api.github.com/repos/gephi/gephi/commits{/sha}","git_commits_url":"https://api.github.com/repos/gephi/gephi/git/commits{/sha}","comments_url":"https://api.github.com/repos/gephi/gephi/comments{/number}","issue_comment_url":"https://api.github.com/repos/gephi/gephi/issues/comments/{number}","contents_url":"https://api.github.com/repos/gephi/gephi/contents/{+path}","compare_url":"https://api.github.com/repos/gephi/gephi/compare/{base}...{head}","merges_url":"https://api.github.com/repos/gephi/gephi/merges","archive_url":"https://api.github.com/repos/gephi/gephi/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/gephi/gephi/downloads","issues_url":"https://api.github.com/repos/gephi/gephi/issues{/number}","pulls_url":"https://api.github.com/repos/gephi/gephi/pulls{/number}","milestones_url":"https://api.github.com/repos/gephi/gephi/milestones{/number}","notifications_url":"https://api.github.com/repos/gephi/gephi/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/gephi/gephi/labels{/name}","releases_url":"https://api.github.com/repos/gephi/gephi/releases{/id}","created_at":"2011-10-22T03:14:46Z","updated_at":"2015-01-18T15:23:11Z","pushed_at":"2014-08-19T23:57:32Z","git_url":"git://github.com/gephi/gephi.git","ssh_url":"git@github.com:gephi/gephi.git","clone_url":"https://github.com/gephi/gephi.git","svn_url":"https://github.com/gephi/gephi","homepage":"","size":109569,"stargazers_count":826,"watchers_count":826,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":726,"mirror_url":null,"open_issues_count":371,"forks":726,"open_issues":371,"watchers":826,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/gephi/gephi/pulls/1001"},"html":{"href":"https://github.com/gephi/gephi/pull/1001"},"issue":{"href":"https://api.github.com/repos/gephi/gephi/issues/1001"},"comments":{"href":"https://api.github.com/repos/gephi/gephi/issues/1001/comments"},"review_comments":{"href":"https://api.github.com/repos/gephi/gephi/pulls/1001/comments"},"review_comment":{"href":"https://api.github.com/repos/gephi/gephi/pulls/comments/{number}"},"commits":{"href":"https://api.github.com/repos/gephi/gephi/pulls/1001/commits"},"statuses":{"href":"https://api.github.com/repos/gephi/gephi/statuses/a0edc0f2993d58c76cb35a9e00a70f6cc2902e78"}},"merged":false,"mergeable":null,"mergeable_state":"unknown","merged_by":null,"comments":0,"review_comments":0,"commits":10,"additions":484,"deletions":99,"changed_files":25}}