{"action":"created","issue":{"url":"https://api.github.com/repos/OpenSourcePolicyCenter/dynamic/issues/107","labels_url":"https://api.github.com/repos/OpenSourcePolicyCenter/dynamic/issues/107/labels{/name}","comments_url":"https://api.github.com/repos/OpenSourcePolicyCenter/dynamic/issues/107/comments","events_url":"https://api.github.com/repos/OpenSourcePolicyCenter/dynamic/issues/107/events","html_url":"https://github.com/OpenSourcePolicyCenter/dynamic/issues/107","id":100215483,"number":107,"title":"Running `run_simulation.py` significantly changes checked in OUTPUT values","user":{"login":"talumbau","id":1891418,"avatar_url":"https://avatars.githubusercontent.com/u/1891418?v=3","gravatar_id":"","url":"https://api.github.com/users/talumbau","html_url":"https://github.com/talumbau","followers_url":"https://api.github.com/users/talumbau/followers","following_url":"https://api.github.com/users/talumbau/following{/other_user}","gists_url":"https://api.github.com/users/talumbau/gists{/gist_id}","starred_url":"https://api.github.com/users/talumbau/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/talumbau/subscriptions","organizations_url":"https://api.github.com/users/talumbau/orgs","repos_url":"https://api.github.com/users/talumbau/repos","events_url":"https://api.github.com/users/talumbau/events{/privacy}","received_events_url":"https://api.github.com/users/talumbau/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":22,"created_at":"2015-08-11T03:04:59Z","updated_at":"2015-08-19T18:54:35Z","closed_at":null,"body":"I'm interested in making significant structural changes to the `dynamic` package (soon to be renamed `ogusa`?) but I need to know that my changes aren't affecting the answer. So, I checked out a clean version of the repo, and then ran `run_simulation.py`, which was my refactored version of the original script that ran everything. Running this script changes a lot of files currently checked in to the repo. That list of files is as follows:\r\n\r\n```\r\n\tmodified:   Python/OUTPUT/SSinit/ss_init_tpi_vars.pkl\r\n\tmodified:   Python/OUTPUT/SSinit/ss_init_vars.pkl\r\n\tmodified:   Python/OUTPUT/Saved_moments/SS_init_solutions.pkl\r\n\tmodified:   Python/OUTPUT/Saved_moments/labor_data_moments.pkl\r\n\tmodified:   Python/OUTPUT/Saved_moments/wealth_data_moments.pkl\r\n\tmodified:   Python/OUTPUT/TPI_K.png\r\n\tmodified:   Python/OUTPUT/TPIinit/TPIinit_vars.pkl\r\n```\r\nFor each of the pickle files, I ran a comparison check using the utility function `pickle_file_compare` in my [recent PR](https://github.com/OpenSourcePolicyCenter/dynamic/pull/102). I checked out another clean version of the repo in a different directory (since the code overwrites what is currently in master). For each pickle file (the clean one in master and the results of running `run_simulation.py`), I captured the output of the `pickle_file_compare`. The output of this utility function is kind of rough, but I cleaned up and condensed some of it. The gist is here:\r\n\r\nhttps://gist.github.com/talumbau/7fef7b31c88a20b9fce1\r\n\r\nThere are two overall results:\r\n\r\n- Many of the values that are different are different by numerical 'noise'. It significantly higher than something like round off error, but I could set either an absolute or relative tolerance for those values and it would be fine. Some examples from the gist:\r\n\r\nFor the file Python/OUTPUT/SSinit/ss_init_tpi_vars.pkl:\r\n```\r\n'nssmat_init: \r\n[[ 0.32211472  0.37224071  0.3898083   0.39204884  0.39629503  0.38399747\\n   0.28474527]]\r\n\r\n[[ 0.32211454  0.37224101  0.38980968  0.39205109  0.39629851  0.38400308\\n   0.28475163]]',\r\n```\r\nThe first array is one element of `nssmat_init` from running `run_simulation.py`, the second is from master. Not exactly the same, but I could set a tolerance and pass a comparison.\r\n\r\n- The values in TPIinit_vars.pkl are wildly different. For example:\r\n\r\n```\r\n'rinit: [-0.03865203] [ 0.07479938]\r\n'Yinit: [ 0.45646764] [ 0.53933587]'\r\n```\r\n\r\nI think I'm probably just doing something wrong/incomplete, because @evan-magnusson merged https://github.com/OpenSourcePolicyCenter/dynamic/pull/94 which seemed to have a very thorough treatment of this issue (making sure the new Python code got the same answers as the old). However, I'm not sure how to proceed until I figure out how to write a test to make sure I get the same answer as well.\r\n\r\ncc @rickecon @evan-magnusson @MattHJensen "},"comment":{"url":"https://api.github.com/repos/OpenSourcePolicyCenter/dynamic/issues/comments/132741242","html_url":"https://github.com/OpenSourcePolicyCenter/dynamic/issues/107#issuecomment-132741242","issue_url":"https://api.github.com/repos/OpenSourcePolicyCenter/dynamic/issues/107","id":132741242,"user":{"login":"evan-magnusson","id":10715898,"avatar_url":"https://avatars.githubusercontent.com/u/10715898?v=3","gravatar_id":"","url":"https://api.github.com/users/evan-magnusson","html_url":"https://github.com/evan-magnusson","followers_url":"https://api.github.com/users/evan-magnusson/followers","following_url":"https://api.github.com/users/evan-magnusson/following{/other_user}","gists_url":"https://api.github.com/users/evan-magnusson/gists{/gist_id}","starred_url":"https://api.github.com/users/evan-magnusson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/evan-magnusson/subscriptions","organizations_url":"https://api.github.com/users/evan-magnusson/orgs","repos_url":"https://api.github.com/users/evan-magnusson/repos","events_url":"https://api.github.com/users/evan-magnusson/events{/privacy}","received_events_url":"https://api.github.com/users/evan-magnusson/received_events","type":"User","site_admin":false},"created_at":"2015-08-19T18:54:35Z","updated_at":"2015-08-19T18:54:35Z","body":"Yep, I got the correct array.  It started at .0747995"}}