{"action":"closed","issue":{"url":"https://api.github.com/repos/NAVADMC/ADSM/issues/369","labels_url":"https://api.github.com/repos/NAVADMC/ADSM/issues/369/labels{/name}","comments_url":"https://api.github.com/repos/NAVADMC/ADSM/issues/369/comments","events_url":"https://api.github.com/repos/NAVADMC/ADSM/issues/369/events","html_url":"https://github.com/NAVADMC/ADSM/issues/369","id":52305145,"number":369,"title":"Output Parser performance issues","user":{"login":"josiahseaman","id":2256148,"avatar_url":"https://avatars.githubusercontent.com/u/2256148?v=3","gravatar_id":"","url":"https://api.github.com/users/josiahseaman","html_url":"https://github.com/josiahseaman","followers_url":"https://api.github.com/users/josiahseaman/followers","following_url":"https://api.github.com/users/josiahseaman/following{/other_user}","gists_url":"https://api.github.com/users/josiahseaman/gists{/gist_id}","starred_url":"https://api.github.com/users/josiahseaman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/josiahseaman/subscriptions","organizations_url":"https://api.github.com/users/josiahseaman/orgs","repos_url":"https://api.github.com/users/josiahseaman/repos","events_url":"https://api.github.com/users/josiahseaman/events{/privacy}","received_events_url":"https://api.github.com/users/josiahseaman/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/NAVADMC/ADSM/labels/Technical+Debt","name":"Technical Debt","color":"207de5"}],"state":"closed","locked":false,"assignee":{"login":"josiahseaman","id":2256148,"avatar_url":"https://avatars.githubusercontent.com/u/2256148?v=3","gravatar_id":"","url":"https://api.github.com/users/josiahseaman","html_url":"https://github.com/josiahseaman","followers_url":"https://api.github.com/users/josiahseaman/followers","following_url":"https://api.github.com/users/josiahseaman/following{/other_user}","gists_url":"https://api.github.com/users/josiahseaman/gists{/gist_id}","starred_url":"https://api.github.com/users/josiahseaman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/josiahseaman/subscriptions","organizations_url":"https://api.github.com/users/josiahseaman/orgs","repos_url":"https://api.github.com/users/josiahseaman/repos","events_url":"https://api.github.com/users/josiahseaman/events{/privacy}","received_events_url":"https://api.github.com/users/josiahseaman/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/NAVADMC/ADSM/milestones/27","labels_url":"https://api.github.com/repos/NAVADMC/ADSM/milestones/27/labels","id":890678,"number":27,"title":"V3.2.22 Release Review","description":"","creator":{"login":"josiahseaman","id":2256148,"avatar_url":"https://avatars.githubusercontent.com/u/2256148?v=3","gravatar_id":"","url":"https://api.github.com/users/josiahseaman","html_url":"https://github.com/josiahseaman","followers_url":"https://api.github.com/users/josiahseaman/followers","following_url":"https://api.github.com/users/josiahseaman/following{/other_user}","gists_url":"https://api.github.com/users/josiahseaman/gists{/gist_id}","starred_url":"https://api.github.com/users/josiahseaman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/josiahseaman/subscriptions","organizations_url":"https://api.github.com/users/josiahseaman/orgs","repos_url":"https://api.github.com/users/josiahseaman/repos","events_url":"https://api.github.com/users/josiahseaman/events{/privacy}","received_events_url":"https://api.github.com/users/josiahseaman/received_events","type":"User","site_admin":false},"open_issues":0,"closed_issues":26,"state":"open","created_at":"2014-12-03T23:04:14Z","updated_at":"2015-01-15T22:42:27Z","due_on":"2014-12-19T07:00:00Z","closed_at":null},"comments":8,"created_at":"2014-12-17T23:12:41Z","updated_at":"2015-01-15T22:42:27Z","closed_at":"2015-01-15T22:42:27Z","body":"Currently the output parser takes 2x to 3x longer to completely load the database than the simulation that is actually generating the results.  This needs to be sped up so that we can actually benefit from C Engine optimizations.\r\n\r\nHere's the current state of the `simulation_process` output parser profile:\r\n\r\n```\r\n         354840 function calls (354828 primitive calls) in 3.102 seconds\r\n\r\n   Ordered by: internal time\r\n   List reduced from 277 to 20 due to restriction <20>\r\n\r\n   ncalls  tottime  percall  cumtime  percall filename:lineno(function)\r\n        1    0.803    0.803    0.803    0.803 {method 'readline' of '_io.BufferedReader' objects}\r\n     1001    0.726    0.001    1.326    0.001 C:\\Users\\Josiah\\VirtualPy341\\lib\\site-packages\\django\\db\\models\\base.py:362(__init__)\r\n    50604    0.503    0.000    0.525    0.000 C:\\Users\\Josiah\\VirtualPy341\\lib\\site-packages\\django\\db\\models\\fields\\__init__.py:713(get_default)\r\n        1    0.351    0.351    0.351    0.351 {method 'readlines' of '_io._IOBase' objects}\r\n       50    0.173    0.003    0.199    0.004 C:/Users/Josiah/Documents/SpreadModel\\Results\\output_parser.py:169(<dictcomp>)\r\n      200    0.137    0.001    0.145    0.001 C:/Users/Josiah/Documents/SpreadModel\\Results\\output_parser.py:20(build_composite_field_map)\r\n       56    0.093    0.002    0.093    0.002 {built-in method WaitForMultipleObjects}\r\n      200    0.046    0.000    0.552    0.003 C:/Users/Josiah/Documents/SpreadModel\\Results\\output_parser.py:39(populate_tables_with_matching_fields)\r\n    50605    0.028    0.000    0.028    0.000 {method 'get' of 'mappingproxy' objects}\r\n    17100    0.026    0.000    0.026    0.000 C:/Users/Josiah/Documents/SpreadModel\\Results\\output_parser.py:10(number)\r\n       50    0.025    0.000    0.153    0.003 C:/Users/Josiah/Documents/SpreadModel\\Results\\output_parser.py:89(create_DailyByZoneAndProductionType)\r\n    71054    0.022    0.000    0.022    0.000 {built-in method setattr}\r\n    55691    0.022    0.000    0.022    0.000 {built-in method isinstance}\r\n    50604    0.019    0.000    0.019    0.000 C:\\Users\\Josiah\\VirtualPy341\\lib\\site-packages\\django\\db\\models\\fields\\__init__.py:707(has_default)\r\n       52    0.018    0.000    0.019    0.000 {method 'dump' of '_pickle.Pickler' objects}\r\n      902    0.009    0.000    0.010    0.000 C:\\Users\\Josiah\\VirtualPy341\\lib\\site-packages\\django\\db\\models\\query.py:125(__iter__)\r\n      750    0.008    0.000    0.018    0.000 C:\\Users\\Josiah\\Py341\\lib\\re.py:168(sub)\r\n      750    0.008    0.000    0.026    0.000 C:/Users/Josiah/Documents/SpreadModel\\Results\\output_parser.py:6(camel_case_spaces)\r\n       50    0.008    0.000    0.815    0.016 C:/Users/Josiah/Documents/SpreadModel\\Results\\output_parser.py:69(create_DailyByProductionType)\r\n    13950    0.007    0.000    0.007    0.000 C:/Users/Josiah/Documents/SpreadModel\\Results\\models.py:32(__iter__)\r\n\r\n```"}}