{"action":"created","issue":{"url":"https://api.github.com/repos/jexp/batch-import/issues/113","labels_url":"https://api.github.com/repos/jexp/batch-import/issues/113/labels{/name}","comments_url":"https://api.github.com/repos/jexp/batch-import/issues/113/comments","events_url":"https://api.github.com/repos/jexp/batch-import/issues/113/events","html_url":"https://github.com/jexp/batch-import/issues/113","id":67380013,"number":113,"title":"Slow Import / 2G nodes file","user":{"login":"dav009","id":1659415,"avatar_url":"https://avatars.githubusercontent.com/u/1659415?v=3","gravatar_id":"","url":"https://api.github.com/users/dav009","html_url":"https://github.com/dav009","followers_url":"https://api.github.com/users/dav009/followers","following_url":"https://api.github.com/users/dav009/following{/other_user}","gists_url":"https://api.github.com/users/dav009/gists{/gist_id}","starred_url":"https://api.github.com/users/dav009/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dav009/subscriptions","organizations_url":"https://api.github.com/users/dav009/orgs","repos_url":"https://api.github.com/users/dav009/repos","events_url":"https://api.github.com/users/dav009/events{/privacy}","received_events_url":"https://api.github.com/users/dav009/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-04-09T14:57:09Z","updated_at":"2015-04-09T17:27:55Z","closed_at":null,"body":"So I'm trying to import what doesn't seem to be a big file but the tool is getting stucked in the first step printing \".....\" for hours.\r\n\r\nnodes.csv is  2.8G\r\nres.csv is  405M\r\n\r\nAnd `batch.properties` is as follows:\r\n\r\n```\r\nbatch_array_separator =\\\\|\r\ndump_configuration=false\r\ncache_type=none\r\nuse_memory_mapped_buffers=true\r\nneostore.propertystore.db.index.keys.mapped_memory=5M\r\nneostore.propertystore.db.index.mapped_memory=5M\r\nneostore.nodestore.db.mapped_memory=5G\r\nneostore.relationshipstore.db.mapped_memory=5G\r\nneostore.propertystore.db.mapped_memory=7G\r\nneostore.propertystore.db.strings.mapped_memory=7G\r\n```\r\n\r\nI have used both neo4j 1.9 and 2.2, I am running this on a machine with 30G, I also left it running for 3/4 hours.\r\n\r\nAfter stopping it, I realized that the neo4j database was empty and only properties has been inserted. So I assume it is taking way too long on inserting properties ? \r\n"},"comment":{"url":"https://api.github.com/repos/jexp/batch-import/issues/comments/91301289","html_url":"https://github.com/jexp/batch-import/issues/113#issuecomment-91301289","issue_url":"https://api.github.com/repos/jexp/batch-import/issues/113","id":91301289,"user":{"login":"jexp","id":67427,"avatar_url":"https://avatars.githubusercontent.com/u/67427?v=3","gravatar_id":"","url":"https://api.github.com/users/jexp","html_url":"https://github.com/jexp","followers_url":"https://api.github.com/users/jexp/followers","following_url":"https://api.github.com/users/jexp/following{/other_user}","gists_url":"https://api.github.com/users/jexp/gists{/gist_id}","starred_url":"https://api.github.com/users/jexp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jexp/subscriptions","organizations_url":"https://api.github.com/users/jexp/orgs","repos_url":"https://api.github.com/users/jexp/repos","events_url":"https://api.github.com/users/jexp/events{/privacy}","received_events_url":"https://api.github.com/users/jexp/received_events","type":"User","site_admin":false},"created_at":"2015-04-09T17:27:55Z","updated_at":"2015-04-09T17:27:55Z","body":"I don't know if this works: `batch_array_separator =\\\\|`\r\n\r\nIf you have 30G in total it probably died doing GC:\r\n\r\nConfigure your batch-importer to use 4-8G Heap (in the shell-script)\r\n\r\nif you have 30G in total that leaves 16G for the memory mapping:\r\n\r\nSo: 2G for nodes, 8G for rels and 3G for props and 3G for strings\r\n\r\nNot sure if you really have arrays in your data.\r\n\r\nI recommend that you look into the import tool that comes out of the box with Neo4j 2.2\r\nsee:\r\nhttp://neo4j.com/docs/stable/import-tool.html"}}