{"action":"closed","issue":{"url":"https://api.github.com/repos/graue/potsfyi/issues/31","labels_url":"https://api.github.com/repos/graue/potsfyi/issues/31/labels{/name}","comments_url":"https://api.github.com/repos/graue/potsfyi/issues/31/comments","events_url":"https://api.github.com/repos/graue/potsfyi/issues/31/events","html_url":"https://github.com/graue/potsfyi/issues/31","id":16721799,"number":31,"title":"Make database updates blazing fast","user":{"login":"graue","id":1730853,"avatar_url":"https://avatars.githubusercontent.com/u/1730853?v=3","gravatar_id":"","url":"https://api.github.com/users/graue","html_url":"https://github.com/graue","followers_url":"https://api.github.com/users/graue/followers","following_url":"https://api.github.com/users/graue/following{/other_user}","gists_url":"https://api.github.com/users/graue/gists{/gist_id}","starred_url":"https://api.github.com/users/graue/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/graue/subscriptions","organizations_url":"https://api.github.com/users/graue/orgs","repos_url":"https://api.github.com/users/graue/repos","events_url":"https://api.github.com/users/graue/events{/privacy}","received_events_url":"https://api.github.com/users/graue/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/graue/potsfyi/labels/Updates","name":"Updates","color":"02d7e1"}],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2013-07-14T00:14:53Z","updated_at":"2015-07-04T00:08:37Z","closed_at":"2015-07-04T00:08:37Z","body":"tl;dr: they're too slow, make 'em fast. (Original rant follows:)\r\n\r\nI just added a commit which prints time taken for create/update using this method:\r\n\r\n    start_time = datetime.today()  # at top of create/update function\r\n    # ... function body (including final db commit at the end) ...\r\n    end_time = datetime.today()    # at bottom of create/update function\r\n    print (end_time - start_time).total_seconds()\r\n\r\nI tested the time to process 664 tracks on the local SSD on a late-2012 MacBook Air 13\". I compared the post-DB overhaul commit 30dff9d, \"Print time taken to process a DB update\", run with `DEBUG=True ./manage.py update`, and the pre-DB overhaul commit e5dad7a \"Fix broken DB_URI option\", run with `DEBUG=True ./manage.py createdb`. (To the old commit, I added a timer that works the same as above.) All figures are the average of 5 trials.\r\n\r\n* 5.14 sec (avg) to create DB first time, post-overhaul, no existing DB\r\n* 0.62 sec (avg) to update existing DB, post-overhaul, no files changed\r\n* 0.85 sec (avg) to create DB, pre-overhaul\r\n\r\nSo modest speedup (27%) in the case where nothing changed, but 500% worse for initial create! This is IMHO obviously not a good tradeoff. When nothing's changed we should get a much better speedup than 27%.\r\n\r\nI'd like updates with no changes to be at least 50% faster than pre-overhaul creates, and initial creates to be at most 25% slower. Future changes should be guided by speed testing and profiling."}}