{"action":"created","issue":{"url":"https://api.github.com/repos/dimitri/pgloader/issues/251","labels_url":"https://api.github.com/repos/dimitri/pgloader/issues/251/labels{/name}","comments_url":"https://api.github.com/repos/dimitri/pgloader/issues/251/comments","events_url":"https://api.github.com/repos/dimitri/pgloader/issues/251/events","html_url":"https://github.com/dimitri/pgloader/issues/251","id":91322584,"number":251,"title":"Performance seems way below expectations","user":{"login":"Themanwithoutaplan","id":1080058,"avatar_url":"https://avatars.githubusercontent.com/u/1080058?v=3","gravatar_id":"","url":"https://api.github.com/users/Themanwithoutaplan","html_url":"https://github.com/Themanwithoutaplan","followers_url":"https://api.github.com/users/Themanwithoutaplan/followers","following_url":"https://api.github.com/users/Themanwithoutaplan/following{/other_user}","gists_url":"https://api.github.com/users/Themanwithoutaplan/gists{/gist_id}","starred_url":"https://api.github.com/users/Themanwithoutaplan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Themanwithoutaplan/subscriptions","organizations_url":"https://api.github.com/users/Themanwithoutaplan/orgs","repos_url":"https://api.github.com/users/Themanwithoutaplan/repos","events_url":"https://api.github.com/users/Themanwithoutaplan/events{/privacy}","received_events_url":"https://api.github.com/users/Themanwithoutaplan/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/dimitri/pgloader/labels/WishList","name":"WishList","color":"fef2c0"}],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":24,"created_at":"2015-06-26T18:01:01Z","updated_at":"2015-07-25T12:33:57Z","closed_at":"2015-07-17T15:07:17Z","body":"I have a relatively simple import from CSV to Postgres that seems to be running quite slow. The import is of about 500000 rows with around 70 columns, mainly integers and four indices. The import currently takes around 5 hours. My hard disk will happily do over 10 MB/s. Running pgbench, at which I'm not an expert, suggests TPS between 70 and 200. Even with the lower number I'd expect the important to take about half-an-hour.\r\n\r\nWhen running the import it does, indeed seem as a lot of time is spent in pgloader rather than in Postgres.\r\n\r\nWhat should I be looking at to improve performance?"},"comment":{"url":"https://api.github.com/repos/dimitri/pgloader/issues/comments/124843178","html_url":"https://github.com/dimitri/pgloader/issues/251#issuecomment-124843178","issue_url":"https://api.github.com/repos/dimitri/pgloader/issues/251","id":124843178,"user":{"login":"dimitri","id":102900,"avatar_url":"https://avatars.githubusercontent.com/u/102900?v=3","gravatar_id":"","url":"https://api.github.com/users/dimitri","html_url":"https://github.com/dimitri","followers_url":"https://api.github.com/users/dimitri/followers","following_url":"https://api.github.com/users/dimitri/following{/other_user}","gists_url":"https://api.github.com/users/dimitri/gists{/gist_id}","starred_url":"https://api.github.com/users/dimitri/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dimitri/subscriptions","organizations_url":"https://api.github.com/users/dimitri/orgs","repos_url":"https://api.github.com/users/dimitri/repos","events_url":"https://api.github.com/users/dimitri/events{/privacy}","received_events_url":"https://api.github.com/users/dimitri/received_events","type":"User","site_admin":false},"created_at":"2015-07-25T12:33:57Z","updated_at":"2015-07-25T12:33:57Z","body":"Can you please run the following query, it might be that some indexes have been only partly deleted and that we should then not worry about them here...\r\n\r\n```sql\r\nselect indrelid::regclass, indisvalid, indcheckxmin, indisready, indislive,\r\n      pg_get_indexdef(indexrelid)\r\n   from pg_index\r\n where indrelid = 'pages'::regclass;\r\n```\r\n\r\nThe other situation where I would expect your error messages is a concurrency issue where two pgloader are working in parallel against the table, thus one of those just deleted the indexes and constraint in the time between the other process having listed the indexes and wanting to delete them...\r\n\r\nIs a concurrency issue possible in your use case?"}}