{"action":"created","issue":{"url":"https://api.github.com/repos/galaxyproject/galaxy/issues/1179","labels_url":"https://api.github.com/repos/galaxyproject/galaxy/issues/1179/labels{/name}","comments_url":"https://api.github.com/repos/galaxyproject/galaxy/issues/1179/comments","events_url":"https://api.github.com/repos/galaxyproject/galaxy/issues/1179/events","html_url":"https://github.com/galaxyproject/galaxy/pull/1179","id":119018777,"number":1179,"title":"Lazy load HDA/LDDA metadata to speed up history loading","user":{"login":"abretaud","id":238755,"avatar_url":"https://avatars.githubusercontent.com/u/238755?v=3","gravatar_id":"","url":"https://api.github.com/users/abretaud","html_url":"https://github.com/abretaud","followers_url":"https://api.github.com/users/abretaud/followers","following_url":"https://api.github.com/users/abretaud/following{/other_user}","gists_url":"https://api.github.com/users/abretaud/gists{/gist_id}","starred_url":"https://api.github.com/users/abretaud/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/abretaud/subscriptions","organizations_url":"https://api.github.com/users/abretaud/orgs","repos_url":"https://api.github.com/users/abretaud/repos","events_url":"https://api.github.com/users/abretaud/events{/privacy}","received_events_url":"https://api.github.com/users/abretaud/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/galaxyproject/galaxy/labels/area/framework","name":"area/framework","color":"006b75"},{"url":"https://api.github.com/repos/galaxyproject/galaxy/labels/status/review","name":"status/review","color":"C9A203"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-11-26T10:49:56Z","updated_at":"2015-12-01T16:30:52Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/galaxyproject/galaxy/pulls/1179","html_url":"https://github.com/galaxyproject/galaxy/pull/1179","diff_url":"https://github.com/galaxyproject/galaxy/pull/1179.diff","patch_url":"https://github.com/galaxyproject/galaxy/pull/1179.patch"},"body":"Hi,\r\nI have some performance problems to view a history containing ~900 datasets (not created by me :smile: ) : loading the history takes several seconds, and sometimes fails.\r\nI did some hacking, and I found out that:\r\n-it is taking a long time to load because it tries to convert from json to MutationDict huge amounts of json stored in the metadata column of HDA or LDDA in the postgres db\r\n-this huge amount of json is generated when the dataset is a bam file created by some tool. In our case we analyse genomes with >20,000 scaffolds, and they are all listed with their length in this metadata.\r\n\r\nAs I think most of the time this metadata is not read by the code listing an history content, I have written this little patch which defers the loading of this json until it is really needed (if it is).\r\nIn my test config, the history loads in <100ms instead of >2,5s before the patch.\r\n\r\nI have some profiling information if you're interested\r\n\r\nI hope there is no side effect I haven't seen. Tell me if it needs some improvements!\r\nThanks\r\nAnthony"},"comment":{"url":"https://api.github.com/repos/galaxyproject/galaxy/issues/comments/161022816","html_url":"https://github.com/galaxyproject/galaxy/pull/1179#issuecomment-161022816","issue_url":"https://api.github.com/repos/galaxyproject/galaxy/issues/1179","id":161022816,"user":{"login":"martenson","id":1814954,"avatar_url":"https://avatars.githubusercontent.com/u/1814954?v=3","gravatar_id":"","url":"https://api.github.com/users/martenson","html_url":"https://github.com/martenson","followers_url":"https://api.github.com/users/martenson/followers","following_url":"https://api.github.com/users/martenson/following{/other_user}","gists_url":"https://api.github.com/users/martenson/gists{/gist_id}","starred_url":"https://api.github.com/users/martenson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/martenson/subscriptions","organizations_url":"https://api.github.com/users/martenson/orgs","repos_url":"https://api.github.com/users/martenson/repos","events_url":"https://api.github.com/users/martenson/events{/privacy}","received_events_url":"https://api.github.com/users/martenson/received_events","type":"User","site_admin":false},"created_at":"2015-12-01T16:30:52Z","updated_at":"2015-12-01T16:30:52Z","body":"ping @carlfeberhard @jmchilton"}}