{"action":"created","issue":{"url":"https://api.github.com/repos/lsst/sims_catUtils/issues/16","labels_url":"https://api.github.com/repos/lsst/sims_catUtils/issues/16/labels{/name}","comments_url":"https://api.github.com/repos/lsst/sims_catUtils/issues/16/comments","events_url":"https://api.github.com/repos/lsst/sims_catUtils/issues/16/events","html_url":"https://github.com/lsst/sims_catUtils/issues/16","id":95723898,"number":16,"title":"Regarding Catalog Optimal Output File Format","user":{"login":"JPGlaser","id":11276213,"avatar_url":"https://avatars.githubusercontent.com/u/11276213?v=3","gravatar_id":"","url":"https://api.github.com/users/JPGlaser","html_url":"https://github.com/JPGlaser","followers_url":"https://api.github.com/users/JPGlaser/followers","following_url":"https://api.github.com/users/JPGlaser/following{/other_user}","gists_url":"https://api.github.com/users/JPGlaser/gists{/gist_id}","starred_url":"https://api.github.com/users/JPGlaser/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JPGlaser/subscriptions","organizations_url":"https://api.github.com/users/JPGlaser/orgs","repos_url":"https://api.github.com/users/JPGlaser/repos","events_url":"https://api.github.com/users/JPGlaser/events{/privacy}","received_events_url":"https://api.github.com/users/JPGlaser/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2015-07-17T19:14:54Z","updated_at":"2015-07-20T20:30:48Z","closed_at":null,"body":"Hey Guys,\r\n\r\nSo I am noticing that as I generate larger datasets from larger FoVs, CatSim's write_catalog command seems to be running rather slow over large sets of observations. I have begun creating a catalog writer that stores each observation's instance catalog into a table in a SQLite3 database. However, the thought occurred to me that I should probably ask to see if such a command already existed within the InstanceCatalog class. Is this the case?\r\n\r\n~ Joe"},"comment":{"url":"https://api.github.com/repos/lsst/sims_catUtils/issues/comments/123024933","html_url":"https://github.com/lsst/sims_catUtils/issues/16#issuecomment-123024933","issue_url":"https://api.github.com/repos/lsst/sims_catUtils/issues/16","id":123024933,"user":{"login":"JPGlaser","id":11276213,"avatar_url":"https://avatars.githubusercontent.com/u/11276213?v=3","gravatar_id":"","url":"https://api.github.com/users/JPGlaser","html_url":"https://github.com/JPGlaser","followers_url":"https://api.github.com/users/JPGlaser/followers","following_url":"https://api.github.com/users/JPGlaser/following{/other_user}","gists_url":"https://api.github.com/users/JPGlaser/gists{/gist_id}","starred_url":"https://api.github.com/users/JPGlaser/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JPGlaser/subscriptions","organizations_url":"https://api.github.com/users/JPGlaser/orgs","repos_url":"https://api.github.com/users/JPGlaser/repos","events_url":"https://api.github.com/users/JPGlaser/events{/privacy}","received_events_url":"https://api.github.com/users/JPGlaser/received_events","type":"User","site_admin":false},"created_at":"2015-07-20T20:30:48Z","updated_at":"2015-07-20T20:30:48Z","body":"Thanks @rbiswas4, I will look into your ipynb for a possible way to store the set of instance catalogs in a database for long-term storage.\r\n\r\nThanks to you as well @danielsf. I will try that out with the next run later today to see if it does in fact speed stuff up. Also, is there a way run a loop with write_catalog inside it in parallel? Might be a good thing for us to look into to speed up writing large groups of instance catalogs for independent observations.\r\n\r\n~ Joe"}}