{"action":"opened","issue":{"url":"https://api.github.com/repos/OpenConceptLab/oclapi/issues/155","labels_url":"https://api.github.com/repos/OpenConceptLab/oclapi/issues/155/labels{/name}","comments_url":"https://api.github.com/repos/OpenConceptLab/oclapi/issues/155/comments","events_url":"https://api.github.com/repos/OpenConceptLab/oclapi/issues/155/events","html_url":"https://github.com/OpenConceptLab/oclapi/issues/155","id":109211120,"number":155,"title":"Reduce concept import time to 5 seconds or less each","user":{"login":"paynejd","id":1189689,"avatar_url":"https://avatars.githubusercontent.com/u/1189689?v=3","gravatar_id":"","url":"https://api.github.com/users/paynejd","html_url":"https://github.com/paynejd","followers_url":"https://api.github.com/users/paynejd/followers","following_url":"https://api.github.com/users/paynejd/following{/other_user}","gists_url":"https://api.github.com/users/paynejd/gists{/gist_id}","starred_url":"https://api.github.com/users/paynejd/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/paynejd/subscriptions","organizations_url":"https://api.github.com/users/paynejd/orgs","repos_url":"https://api.github.com/users/paynejd/repos","events_url":"https://api.github.com/users/paynejd/events{/privacy}","received_events_url":"https://api.github.com/users/paynejd/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-10-01T01:19:16Z","updated_at":"2015-10-01T01:19:16Z","closed_at":null,"body":"Adding/updating concepts is slow. It took between 30-60seconds per concept before I started optimizing. I switched to a dedicated server and made a number of code and index improvements which got the time down to ~11 seconds per concept. This is a huge improvement, however, the imports are increasingly slow as the size of the database increases, so it is critical that this time is decreased much further. The proposed **sequential source version model** should result in huge improvements, but that won't be done anytime soon. Below are the lower hanging fruits-- a few key bottlenecks that could be addressed prior to the full re-architecting.\r\n\r\nConcept lookups are indexed and are very fast. Concept version lookups could be fast, but the current setup makes them slow if the lookup includes a check on its membership in a particular source version. Updating the array of concept versions linked to a source version and updating a specific concept version are also slow and could be sped up a lot.\r\n\r\n1. The `id__in=self.source_version.concepts` expression in `handle_concept` causes a very inefficient and long-running query for every single concept that exists, whether nodiff or an update. Code here: https://github.com/OpenConceptLab/oclapi/blob/master/django-nonrel/ocl/oclapi/management/commands/import_concepts_to_source.py#L155-L156\r\n\r\nThis issue will be resolved if the **sequential source version model** is implemented. Until then, it may be possible to change the query to compare the conceptversion ObjectID directly to the sources_sourceversion.concepts array, which runs in ~20ms instead of 200-400ms. Here's an example:\r\n\r\n```\r\n# This finds source versions of a specific source that contain the specified concept version\r\ndb.sources_sourceversion.find( { versioned_object_id: '55804807525c311e7eb3858f', concepts: '560169b6525c3109d57b1e0f' }, { concepts: { $slice: 1 }, mappings: 0, extras: 0 } )\r\n```\r\n\r\n2. The next (super) long-running query occurs when the new conceptversion ID is added to the sourceversion concepts array. It is possible to use the $pull operator to remove the old conceptversion ID and the $push operator to add the new conceptversion ID. This needs to be tested, but it should be significantly faster than updating the entire array in code and then saving the update to mongodb afterwards.\r\n\r\n3. The third long-running query, although this is not nearly as long as the 2 above, is the concept version update itself. It may be possible to speed this up a lot with some index work.\r\n\r\n4. Generally, additional index work in MongoDB could probably yield some significant improvements.\r\n\r\n5. It appears that Solr is updating records in real-time, at least on production. This is not the intended behavior, but I can't seem to figure out how to turn realtime signal processing off in Haystack, despite having hacked around with settings.py and the importer's __init__.py module."}}