{"action":"created","issue":{"url":"https://api.github.com/repos/california-civic-data-coalition/django-calaccess-raw-data/issues/257","labels_url":"https://api.github.com/repos/california-civic-data-coalition/django-calaccess-raw-data/issues/257/labels{/name}","comments_url":"https://api.github.com/repos/california-civic-data-coalition/django-calaccess-raw-data/issues/257/comments","events_url":"https://api.github.com/repos/california-civic-data-coalition/django-calaccess-raw-data/issues/257/events","html_url":"https://github.com/california-civic-data-coalition/django-calaccess-raw-data/issues/257","id":64813898,"number":257,"title":"incremental updates from downloaded data, is this possible?","user":{"login":"rkiddy","id":691368,"avatar_url":"https://avatars.githubusercontent.com/u/691368?v=3","gravatar_id":"","url":"https://api.github.com/users/rkiddy","html_url":"https://github.com/rkiddy","followers_url":"https://api.github.com/users/rkiddy/followers","following_url":"https://api.github.com/users/rkiddy/following{/other_user}","gists_url":"https://api.github.com/users/rkiddy/gists{/gist_id}","starred_url":"https://api.github.com/users/rkiddy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rkiddy/subscriptions","organizations_url":"https://api.github.com/users/rkiddy/orgs","repos_url":"https://api.github.com/users/rkiddy/repos","events_url":"https://api.github.com/users/rkiddy/events{/privacy}","received_events_url":"https://api.github.com/users/rkiddy/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/california-civic-data-coalition/django-calaccess-raw-data/labels/backend","name":"backend","color":"fbca04"},{"url":"https://api.github.com/repos/california-civic-data-coalition/django-calaccess-raw-data/labels/enhancement","name":"enhancement","color":"84b6eb"},{"url":"https://api.github.com/repos/california-civic-data-coalition/django-calaccess-raw-data/labels/question","name":"question","color":"cc317c"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-03-27T17:36:07Z","updated_at":"2015-04-08T22:26:45Z","closed_at":null,"body":"My downloader/chopper-upper does this the same as yours. It downloads the data and re-parses and re-loads everything in the entire database every time.\r\n\r\nHave you all scoped out how to do incremental loads? In other words, if I downloaded and set up the data yesterday and I fetch again today, it would be nice if it could just add the several hundred new rows instead of starting from scratch and processing millions of rows.\r\n\r\nIt really seems that this should be possible. After all, the SoS does not correct any data. So, once something is added, it should not be going away. It would be good to be able to take advantage of this and make daily update quicker.\r\n"},"comment":{"url":"https://api.github.com/repos/california-civic-data-coalition/django-calaccess-raw-data/issues/comments/91054218","html_url":"https://github.com/california-civic-data-coalition/django-calaccess-raw-data/issues/257#issuecomment-91054218","issue_url":"https://api.github.com/repos/california-civic-data-coalition/django-calaccess-raw-data/issues/257","id":91054218,"user":{"login":"rkiddy","id":691368,"avatar_url":"https://avatars.githubusercontent.com/u/691368?v=3","gravatar_id":"","url":"https://api.github.com/users/rkiddy","html_url":"https://github.com/rkiddy","followers_url":"https://api.github.com/users/rkiddy/followers","following_url":"https://api.github.com/users/rkiddy/following{/other_user}","gists_url":"https://api.github.com/users/rkiddy/gists{/gist_id}","starred_url":"https://api.github.com/users/rkiddy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rkiddy/subscriptions","organizations_url":"https://api.github.com/users/rkiddy/orgs","repos_url":"https://api.github.com/users/rkiddy/repos","events_url":"https://api.github.com/users/rkiddy/events{/privacy}","received_events_url":"https://api.github.com/users/rkiddy/received_events","type":"User","site_admin":false},"created_at":"2015-04-08T22:26:45Z","updated_at":"2015-04-08T22:26:45Z","body":"I am looking to add code which adds a id number to the import file. The model can be changed so that the primary keys of the tables are not auto-generated. In my scripts, I was taking each TSV file on it and running this:\r\n```\r\nawk '{if (NR=0) print \"id\\t\"$0; else print (NR-1)\"\\t\"$0}'\r\n```\r\nand replacing the TSV file with the output from this. I am seeing several ways to do this in the python code, but none as simple as this command.\r\n\r\nAs I was doing this, I checked the conversion of the TSV files to the csv files. Is this right? This is with the released django-calaccess-raw-data (ie doing a \"pip install\") and not with my copy of the code. I am surprised to see that so many lines of data are disappearing. Have I done something wrong here? Do you all see this as well? Why would 2/3 of the expn file, for example, be disappearing?\r\n```\r\n$ find data/tsv data/csv -type f -ls | \\\r\n    awk '{print $NF, $7}' | \\\r\n    tr '[A-Z]' '[a-z]' | \\\r\n    awk 'BEGIN{FS=\"/\"}{print $NF}' | \\\r\n    sort | \\\r\n    awk 'BEGIN{FS=\".\";last=\"\"}{if ($1 != last) print \"\"; print $0; last=$1}'\r\n\r\nacronyms_cd.csv 43105\r\nacronyms_cd.tsv 36031\r\n\r\naddress_cd.csv 19882087\r\naddress_cd.tsv 15461937\r\n\r\nballot_measures_cd.csv 18356\r\nballot_measures_cd.tsv 17028\r\n\r\ncvr2_campaign_disclosure_cd.csv 19244800\r\ncvr2_campaign_disclosure_cd.tsv 12164158\r\n\r\ncvr2_lobby_disclosure_cd.csv 8069948\r\ncvr2_lobby_disclosure_cd.tsv 5798666\r\n\r\ncvr2_registration_cd.csv 7227935\r\ncvr2_registration_cd.tsv 5176021\r\n\r\ncvr2_so_cd.csv 6448372\r\ncvr2_so_cd.tsv 4182856\r\n\r\ncvr3_verification_info_cd.csv 21227758\r\ncvr3_verification_info_cd.tsv 15724702\r\n\r\ncvr_campaign_disclosure_cd.csv 161512513\r\ncvr_campaign_disclosure_cd.tsv 104653574\r\n\r\ncvr_e530_cd.csv 54398\r\ncvr_e530_cd.tsv 42174\r\n\r\ncvr_f470_cd.csv 0\r\ncvr_f470_cd.tsv 0\r\n\r\ncvr_lobby_disclosure_cd.csv 126789418\r\ncvr_lobby_disclosure_cd.tsv 99732892\r\n\r\ncvr_registration_cd.csv 21054006\r\ncvr_registration_cd.tsv 15172265\r\n\r\ncvr_so_cd.csv 6511823\r\ncvr_so_cd.tsv 5019263\r\n\r\ndebt_cd.csv 74290444\r\ndebt_cd.tsv 48294476\r\n\r\nefs_filing_log_cd.csv 10519894\r\nefs_filing_log_cd.tsv 8758126\r\n\r\nexpn_cd.csv 1266640882\r\nexpn_cd.tsv 781034344\r\n\r\nf495p2_cd.csv 73652\r\nf495p2_cd.tsv 59348\r\n\r\nf501_502_cd.csv 5012499\r\nf501_502_cd.tsv 3392096\r\n\r\nf690p2_cd.csv 3493865\r\nf690p2_cd.tsv 3122285\r\n\r\nfiler_acronyms_cd.csv 32194\r\nfiler_acronyms_cd.tsv 25772\r\n\r\nfiler_address_cd.csv 23081169\r\nfiler_address_cd.tsv 19154659\r\n\r\nfiler_ethics_class_cd.csv 600026\r\nfiler_ethics_class_cd.tsv 514436\r\n\r\nfiler_filings_cd.csv 281257943\r\nfiler_filings_cd.tsv 223974551\r\n\r\nfiler_interests_cd.csv 1744409\r\nfiler_interests_cd.tsv 1460665\r\n\r\nfiler_links_cd.csv 26526393\r\nfiler_links_cd.tsv 21905721\r\n\r\nfilername_cd.csv 125223831\r\nfilername_cd.tsv 102389549\r\n\r\nfilers_cd.csv 3733974\r\nfilers_cd.tsv 3055062\r\n\r\nfiler_status_types_cd.csv 155\r\nfiler_status_types_cd.tsv 119\r\n\r\nfiler_to_filer_type_cd.csv 47498546\r\nfiler_to_filer_type_cd.tsv 32473906\r\n\r\nfiler_type_periods_cd.csv 0\r\nfiler_type_periods_cd.tsv 0\r\n\r\nfiler_types_cd.csv 721\r\nfiler_types_cd.tsv 531\r\n\r\nfiler_xref_cd.csv 19006052\r\nfiler_xref_cd.tsv 16290660\r\n\r\nfiling_period_cd.csv 152223\r\nfiling_period_cd.tsv 131937\r\n\r\nfilings_cd.csv 30712525\r\nfilings_cd.tsv 24047745\r\n\r\ngroup_types_cd.csv 418\r\ngroup_types_cd.tsv 292\r\n\r\nhdr_cd.csv 41885133\r\nhdr_cd.tsv 30394095\r\n\r\nheader_cd.csv 12383\r\nheader_cd.tsv 10247\r\n\r\nimage_links_cd.csv 787521\r\nimage_links_cd.tsv 645481\r\n\r\nlatt_cd.csv 10766866\r\nlatt_cd.tsv 8097026\r\n\r\nlccm_cd.csv 13810262\r\nlccm_cd.tsv 9750319\r\n\r\nlegislative_sessions_cd.csv 1058\r\nlegislative_sessions_cd.tsv 872\r\n\r\nlemp_cd.csv 76567339\r\nlemp_cd.tsv 64707902\r\n\r\nlexp_cd.csv 36398903\r\nlexp_cd.tsv 28090980\r\n\r\nloan_cd.csv 18908659\r\nloan_cd.tsv 12429061\r\n\r\nlobby_amendments_cd.csv 2655914\r\nlobby_amendments_cd.tsv 1825406\r\n\r\nlobbying_chg_log_cd.csv 11262119\r\nlobbying_chg_log_cd.tsv 8636795\r\n\r\nlobbyist_contributions1_cd.csv 694624\r\nlobbyist_contributions1_cd.tsv 603540\r\n\r\nlobbyist_contributions2_cd.csv 694624\r\nlobbyist_contributions2_cd.tsv 603540\r\n\r\nlobbyist_contributions3_cd.csv 694624\r\nlobbyist_contributions3_cd.tsv 603540\r\n\r\nlobbyist_emp_lobbyist1_cd.csv 130881\r\nlobbyist_emp_lobbyist1_cd.tsv 112851\r\n\r\nlobbyist_emp_lobbyist2_cd.csv 130881\r\nlobbyist_emp_lobbyist2_cd.tsv 112851\r\n\r\nlobbyist_employer1_cd.csv 288400\r\nlobbyist_employer1_cd.tsv 219160\r\n\r\nlobbyist_employer2_cd.csv 288417\r\nlobbyist_employer2_cd.tsv 219177\r\n\r\nlobbyist_employer3_cd.csv 288417\r\nlobbyist_employer3_cd.tsv 219177\r\n\r\nlobbyist_employer_firms1_cd.csv 32277\r\nlobbyist_employer_firms1_cd.tsv 27021\r\n\r\nlobbyist_employer_firms2_cd.csv 32277\r\nlobbyist_employer_firms2_cd.tsv 27021\r\n\r\nlobbyist_employer_history_cd.csv 0\r\nlobbyist_employer_history_cd.tsv 0\r\n\r\nlobbyist_firm1_cd.csv 36792\r\nlobbyist_firm1_cd.tsv 27538\r\n\r\nlobbyist_firm2_cd.csv 36792\r\nlobbyist_firm2_cd.tsv 27538\r\n\r\nlobbyist_firm3_cd.csv 36792\r\nlobbyist_firm3_cd.tsv 27538\r\n\r\nlobbyist_firm_employer1_cd.csv 36322\r\nlobbyist_firm_employer1_cd.tsv 32560\r\n\r\nlobbyist_firm_employer2_cd.csv 36322\r\nlobbyist_firm_employer2_cd.tsv 32560\r\n\r\nlobbyist_firm_history_cd.csv 0\r\nlobbyist_firm_history_cd.tsv 0\r\n\r\nlobbyist_firm_lobbyist1_cd.csv 45030\r\nlobbyist_firm_lobbyist1_cd.tsv 38086\r\n\r\nlobbyist_firm_lobbyist2_cd.csv 45030\r\nlobbyist_firm_lobbyist2_cd.tsv 38086\r\n\r\nlookup_codes_cd.csv 39626\r\nlookup_codes_cd.tsv 32840\r\n\r\nloth_cd.csv 2344626\r\nloth_cd.tsv 1840906\r\n\r\nlpay_cd.csv 87144015\r\nlpay_cd.tsv 65791563\r\n\r\nnames_cd.csv 36611584\r\nnames_cd.tsv 28084227\r\n\r\nrcpt_cd.csv 2716141579\r\nrcpt_cd.tsv 1610992531\r\n\r\nreceived_filings_cd.csv 44966644\r\nreceived_filings_cd.tsv 39711660\r\n\r\nreports_cd.csv 3591\r\nreports_cd.tsv 2967\r\n\r\ns401_cd.csv 7854277\r\ns401_cd.tsv 4840768\r\n\r\ns496_cd.csv 3749247\r\ns496_cd.tsv 2931806\r\n\r\ns497_cd.csv 139912920\r\ns497_cd.tsv 93681713\r\n\r\ns498_cd.csv 2079151\r\ns498_cd.tsv 1400445\r\n\r\nsmry_cd.csv 365300812\r\nsmry_cd.tsv 229793572\r\n\r\nsplt_cd.csv 207217987\r\nsplt_cd.tsv 158555699\r\n\r\ntext_memo_cd.csv 221059593\r\ntext_memo_cd.tsv 200972246\r\n```"}}