{"action":"created","issue":{"url":"https://api.github.com/repos/etianen/django-watson/issues/85","labels_url":"https://api.github.com/repos/etianen/django-watson/issues/85/labels{/name}","comments_url":"https://api.github.com/repos/etianen/django-watson/issues/85/comments","events_url":"https://api.github.com/repos/etianen/django-watson/issues/85/events","html_url":"https://github.com/etianen/django-watson/issues/85","id":53103958,"number":85,"title":"Building the index from scratch takes a while","user":{"login":"thedrow","id":48936,"avatar_url":"https://avatars.githubusercontent.com/u/48936?v=3","gravatar_id":"","url":"https://api.github.com/users/thedrow","html_url":"https://github.com/thedrow","followers_url":"https://api.github.com/users/thedrow/followers","following_url":"https://api.github.com/users/thedrow/following{/other_user}","gists_url":"https://api.github.com/users/thedrow/gists{/gist_id}","starred_url":"https://api.github.com/users/thedrow/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thedrow/subscriptions","organizations_url":"https://api.github.com/users/thedrow/orgs","repos_url":"https://api.github.com/users/thedrow/repos","events_url":"https://api.github.com/users/thedrow/events{/privacy}","received_events_url":"https://api.github.com/users/thedrow/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":9,"created_at":"2014-12-30T09:41:11Z","updated_at":"2015-01-02T10:53:34Z","closed_at":null,"body":"170000 records took about an hour to build.\r\nCan we parallelize some of the work so that for example each model will use a different process to build it's index?\r\nWe could use the ProcessPoolExecutor if it's available.\r\nWhat do you think?"},"comment":{"url":"https://api.github.com/repos/etianen/django-watson/issues/comments/68519267","html_url":"https://github.com/etianen/django-watson/issues/85#issuecomment-68519267","issue_url":"https://api.github.com/repos/etianen/django-watson/issues/85","id":68519267,"user":{"login":"etianen","id":148744,"avatar_url":"https://avatars.githubusercontent.com/u/148744?v=3","gravatar_id":"","url":"https://api.github.com/users/etianen","html_url":"https://github.com/etianen","followers_url":"https://api.github.com/users/etianen/followers","following_url":"https://api.github.com/users/etianen/following{/other_user}","gists_url":"https://api.github.com/users/etianen/gists{/gist_id}","starred_url":"https://api.github.com/users/etianen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/etianen/subscriptions","organizations_url":"https://api.github.com/users/etianen/orgs","repos_url":"https://api.github.com/users/etianen/repos","events_url":"https://api.github.com/users/etianen/events{/privacy}","received_events_url":"https://api.github.com/users/etianen/received_events","type":"User","site_admin":false},"created_at":"2015-01-02T10:53:34Z","updated_at":"2015-01-02T10:53:34Z","body":"This feels very much like one of those tasks that can only be performed with any real efficiency on the DB server. Unfortunately, the configuration for django-watson is all Python-side, in the form of SearchAdapter classes. This means that all the data needs to be streamed through Python.\r\n\r\nThere is some indication that the postgres machine doesn’t handle the bulk inserts in a performant manner, and could benefit from multiple simulataneous connections. Discussion here:\r\n\r\nhttps://github.com/etianen/django-watson/issues/51\r\n\r\nThe worker pool approach described in the last comment hasn’t been attempted. If you could find a way to leverage it in a patched buildwatson command that noticeably improves performance, I’d love to see the result.\r\n\r\n> On 31 Dec 2014, at 13:26, Omer Katz <notifications@github.com> wrote:\r\n> \r\n> I'm running on RDS and I have both. I run the indexing command from EC2 so the two servers are on the same VPC.\r\n> \r\n> —\r\n> Reply to this email directly or view it on GitHub.\r\n> "}}