{"action":"created","comment":{"url":"https://api.github.com/repos/open-data/ckanext-canada/pulls/comments/27239366","id":27239366,"diff_hunk":"@@ -0,0 +1,59 @@\n+#!/bin/bash\n+\n+function usage() {\n+    echo \"Usage: ${0} CKAN-INI-FILE [VIRTUAL-ENV-HOME]\"\n+    echo \"The script:\"\n+    echo \"- activates virtual environment if specified\"\n+    echo \"- pushes datasets from .csv files to the solr core\"\n+    exit -1\n+}\n+\n+#\n+#\n+# THIS SCRIPT AS IT IS WRITTEN IS HOPELESS: NEEDS TO USE CKANAPI FILE STORAGE\n+#\n+#\n+\n+if [ \"$#\" -eq 0 ]\n+then\n+    usage\n+elif [ \"$#\" -gt 1 ]\n+then\n+   VENV_HOME=$(echo ${2} | sed -e 's./$..')\n+   source ${VENV_HOME}/bin/activate\n+fi\n+\n+cd \"$(dirname ${0})\"\n+BIN_HOME=$(pwd)\n+\n+# Set ini file\n+cd \"$(dirname ${1})\"\n+INI_FILE=$(basename ${1})\n+INI_PATH=\"$(pwd)/${INI_FILE}\"\n+\n+# Record ckanext-canada home\n+cd \"${BIN_HOME}/..\"\n+CXC_HOME=$(pwd)\n+\n+# Go to ckanext-recombinant\n+cd \"${BIN_HOME}/../../ckanext-recombinant\"\n+CXR_HOME=$(pwd)\n+\n+# Get target dataset names\n+TARGET_DATASETS=$(paster recombinant target-datasets -c \"${INI_PATH}\")\n+for TARG_DS in $(echo ${TARGET_DATASETS} | tr ' ' '\\n')\n+do\n+    # Identify .csv files to push to solr\n+    CSV_FILES=\"\"\n+    cd \"${CXR_HOME}\"\n+    DATASET_TYPES=$(echo $(paster recombinant dataset-types ${TARG_DS} -c \"${INI_PATH}\") | sed -e 's/.*: //')\n+    for F in ${DATASET_TYPES}\n+    do\n+      CSV_FILES=\"${CSV_FILES} ${TARG_DS}.${F}.csv\"\n+    done\n+    CSV_FILES=${CSV_FILES/ /}\n+\n+    # rebuild solr core from identified .csv files\n+    cd \"${CXC_HOME}\"\n+    paster \"${TARG_DS}\" rebuild -f ${CSV_FILES} -c \"${INI_PATH}\" ${TARG_DS}.${DS_TYPE}.csv\n+done","path":"bin/csv2solr.sh","position":59,"original_position":59,"commit_id":"a963965b2a70452dac1bfadd5b36e416fbfad2b4","original_commit_id":"a963965b2a70452dac1bfadd5b36e416fbfad2b4","user":{"login":"wardi","id":153258,"avatar_url":"https://avatars.githubusercontent.com/u/153258?v=3","gravatar_id":"","url":"https://api.github.com/users/wardi","html_url":"https://github.com/wardi","followers_url":"https://api.github.com/users/wardi/followers","following_url":"https://api.github.com/users/wardi/following{/other_user}","gists_url":"https://api.github.com/users/wardi/gists{/gist_id}","starred_url":"https://api.github.com/users/wardi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wardi/subscriptions","organizations_url":"https://api.github.com/users/wardi/orgs","repos_url":"https://api.github.com/users/wardi/repos","events_url":"https://api.github.com/users/wardi/events{/privacy}","received_events_url":"https://api.github.com/users/wardi/received_events","type":"User","site_admin":false},"body":"I think this script is complicated enough to warrant rewriting it as a `paster canada` subcommand instead. That way it's not tied to a particular directory layout and can just import and use the necessary recombinant functions.","created_at":"2015-03-26T18:04:44Z","updated_at":"2015-03-26T18:04:44Z","html_url":"https://github.com/open-data/ckanext-canada/pull/441#discussion_r27239366","pull_request_url":"https://api.github.com/repos/open-data/ckanext-canada/pulls/441","_links":{"self":{"href":"https://api.github.com/repos/open-data/ckanext-canada/pulls/comments/27239366"},"html":{"href":"https://github.com/open-data/ckanext-canada/pull/441#discussion_r27239366"},"pull_request":{"href":"https://api.github.com/repos/open-data/ckanext-canada/pulls/441"}}},"pull_request":{"url":"https://api.github.com/repos/open-data/ckanext-canada/pulls/441","id":32038871,"html_url":"https://github.com/open-data/ckanext-canada/pull/441","diff_url":"https://github.com/open-data/ckanext-canada/pull/441.diff","patch_url":"https://github.com/open-data/ckanext-canada/pull/441.patch","issue_url":"https://api.github.com/repos/open-data/ckanext-canada/issues/441","number":441,"state":"open","locked":false,"title":"Support auto-post of combined datasets from registry to portal","user":{"login":"sklump","id":9336556,"avatar_url":"https://avatars.githubusercontent.com/u/9336556?v=3","gravatar_id":"","url":"https://api.github.com/users/sklump","html_url":"https://github.com/sklump","followers_url":"https://api.github.com/users/sklump/followers","following_url":"https://api.github.com/users/sklump/following{/other_user}","gists_url":"https://api.github.com/users/sklump/gists{/gist_id}","starred_url":"https://api.github.com/users/sklump/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sklump/subscriptions","organizations_url":"https://api.github.com/users/sklump/orgs","repos_url":"https://api.github.com/users/sklump/repos","events_url":"https://api.github.com/users/sklump/events{/privacy}","received_events_url":"https://api.github.com/users/sklump/received_events","type":"User","site_admin":false},"body":"ckanext-recombinant:\r\n+ recombinant_tables.json supports \"target_dataset\" per dataset_type (e.g., \"ati\" for \"ati-summaries\" and \"ati-none\")\r\n+ paster recombinant command target-datasets command to return all target datasets defined\r\n+ paster recombinant command dataset-types lists all dataset types per input target\r\n\r\nckanext-canada:\r\n+ ckanext/canada/common code between ati.py, pd.py refactored and generalized in dataset.py\r\n+ ckanext/canada/ati.py, pd.py know their respective target datasets and pass them to dataset.py, which gets corresponding dataset types on the fly\r\n+ bin/reg2portal.sh and reg2portal.py combine dataset resources and upload to corresponding portal datasets\r\n+ bin/csv2solr.sh on portal will update solr core from uploaded csv files, but is still a work in progress","created_at":"2015-03-26T17:07:50Z","updated_at":"2015-03-26T18:04:44Z","closed_at":null,"merged_at":null,"merge_commit_sha":"a1382f8980d16b5d7d40028a1429d32465d03dd3","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/open-data/ckanext-canada/pulls/441/commits","review_comments_url":"https://api.github.com/repos/open-data/ckanext-canada/pulls/441/comments","review_comment_url":"https://api.github.com/repos/open-data/ckanext-canada/pulls/comments{/number}","comments_url":"https://api.github.com/repos/open-data/ckanext-canada/issues/441/comments","statuses_url":"https://api.github.com/repos/open-data/ckanext-canada/statuses/a963965b2a70452dac1bfadd5b36e416fbfad2b4","head":{"label":"sklump:autopost","ref":"autopost","sha":"a963965b2a70452dac1bfadd5b36e416fbfad2b4","user":{"login":"sklump","id":9336556,"avatar_url":"https://avatars.githubusercontent.com/u/9336556?v=3","gravatar_id":"","url":"https://api.github.com/users/sklump","html_url":"https://github.com/sklump","followers_url":"https://api.github.com/users/sklump/followers","following_url":"https://api.github.com/users/sklump/following{/other_user}","gists_url":"https://api.github.com/users/sklump/gists{/gist_id}","starred_url":"https://api.github.com/users/sklump/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sklump/subscriptions","organizations_url":"https://api.github.com/users/sklump/orgs","repos_url":"https://api.github.com/users/sklump/repos","events_url":"https://api.github.com/users/sklump/events{/privacy}","received_events_url":"https://api.github.com/users/sklump/received_events","type":"User","site_admin":false},"repo":{"id":30976091,"name":"ckanext-canada","full_name":"sklump/ckanext-canada","owner":{"login":"sklump","id":9336556,"avatar_url":"https://avatars.githubusercontent.com/u/9336556?v=3","gravatar_id":"","url":"https://api.github.com/users/sklump","html_url":"https://github.com/sklump","followers_url":"https://api.github.com/users/sklump/followers","following_url":"https://api.github.com/users/sklump/following{/other_user}","gists_url":"https://api.github.com/users/sklump/gists{/gist_id}","starred_url":"https://api.github.com/users/sklump/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sklump/subscriptions","organizations_url":"https://api.github.com/users/sklump/orgs","repos_url":"https://api.github.com/users/sklump/repos","events_url":"https://api.github.com/users/sklump/events{/privacy}","received_events_url":"https://api.github.com/users/sklump/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/sklump/ckanext-canada","description":"Government of Canada CKAN Extension - Extension à CKAN du Gouvernement du Canada","fork":true,"url":"https://api.github.com/repos/sklump/ckanext-canada","forks_url":"https://api.github.com/repos/sklump/ckanext-canada/forks","keys_url":"https://api.github.com/repos/sklump/ckanext-canada/keys{/key_id}","collaborators_url":"https://api.github.com/repos/sklump/ckanext-canada/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/sklump/ckanext-canada/teams","hooks_url":"https://api.github.com/repos/sklump/ckanext-canada/hooks","issue_events_url":"https://api.github.com/repos/sklump/ckanext-canada/issues/events{/number}","events_url":"https://api.github.com/repos/sklump/ckanext-canada/events","assignees_url":"https://api.github.com/repos/sklump/ckanext-canada/assignees{/user}","branches_url":"https://api.github.com/repos/sklump/ckanext-canada/branches{/branch}","tags_url":"https://api.github.com/repos/sklump/ckanext-canada/tags","blobs_url":"https://api.github.com/repos/sklump/ckanext-canada/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/sklump/ckanext-canada/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/sklump/ckanext-canada/git/refs{/sha}","trees_url":"https://api.github.com/repos/sklump/ckanext-canada/git/trees{/sha}","statuses_url":"https://api.github.com/repos/sklump/ckanext-canada/statuses/{sha}","languages_url":"https://api.github.com/repos/sklump/ckanext-canada/languages","stargazers_url":"https://api.github.com/repos/sklump/ckanext-canada/stargazers","contributors_url":"https://api.github.com/repos/sklump/ckanext-canada/contributors","subscribers_url":"https://api.github.com/repos/sklump/ckanext-canada/subscribers","subscription_url":"https://api.github.com/repos/sklump/ckanext-canada/subscription","commits_url":"https://api.github.com/repos/sklump/ckanext-canada/commits{/sha}","git_commits_url":"https://api.github.com/repos/sklump/ckanext-canada/git/commits{/sha}","comments_url":"https://api.github.com/repos/sklump/ckanext-canada/comments{/number}","issue_comment_url":"https://api.github.com/repos/sklump/ckanext-canada/issues/comments{/number}","contents_url":"https://api.github.com/repos/sklump/ckanext-canada/contents/{+path}","compare_url":"https://api.github.com/repos/sklump/ckanext-canada/compare/{base}...{head}","merges_url":"https://api.github.com/repos/sklump/ckanext-canada/merges","archive_url":"https://api.github.com/repos/sklump/ckanext-canada/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/sklump/ckanext-canada/downloads","issues_url":"https://api.github.com/repos/sklump/ckanext-canada/issues{/number}","pulls_url":"https://api.github.com/repos/sklump/ckanext-canada/pulls{/number}","milestones_url":"https://api.github.com/repos/sklump/ckanext-canada/milestones{/number}","notifications_url":"https://api.github.com/repos/sklump/ckanext-canada/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/sklump/ckanext-canada/labels{/name}","releases_url":"https://api.github.com/repos/sklump/ckanext-canada/releases{/id}","created_at":"2015-02-18T17:11:05Z","updated_at":"2015-03-20T16:19:28Z","pushed_at":"2015-03-26T16:57:21Z","git_url":"git://github.com/sklump/ckanext-canada.git","ssh_url":"git@github.com:sklump/ckanext-canada.git","clone_url":"https://github.com/sklump/ckanext-canada.git","svn_url":"https://github.com/sklump/ckanext-canada","homepage":null,"size":14109,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"open-data:master","ref":"master","sha":"82aa378b30eb363723afbd7cec78ad851e0a722d","user":{"login":"open-data","id":2651841,"avatar_url":"https://avatars.githubusercontent.com/u/2651841?v=3","gravatar_id":"","url":"https://api.github.com/users/open-data","html_url":"https://github.com/open-data","followers_url":"https://api.github.com/users/open-data/followers","following_url":"https://api.github.com/users/open-data/following{/other_user}","gists_url":"https://api.github.com/users/open-data/gists{/gist_id}","starred_url":"https://api.github.com/users/open-data/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/open-data/subscriptions","organizations_url":"https://api.github.com/users/open-data/orgs","repos_url":"https://api.github.com/users/open-data/repos","events_url":"https://api.github.com/users/open-data/events{/privacy}","received_events_url":"https://api.github.com/users/open-data/received_events","type":"Organization","site_admin":false},"repo":{"id":7136430,"name":"ckanext-canada","full_name":"open-data/ckanext-canada","owner":{"login":"open-data","id":2651841,"avatar_url":"https://avatars.githubusercontent.com/u/2651841?v=3","gravatar_id":"","url":"https://api.github.com/users/open-data","html_url":"https://github.com/open-data","followers_url":"https://api.github.com/users/open-data/followers","following_url":"https://api.github.com/users/open-data/following{/other_user}","gists_url":"https://api.github.com/users/open-data/gists{/gist_id}","starred_url":"https://api.github.com/users/open-data/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/open-data/subscriptions","organizations_url":"https://api.github.com/users/open-data/orgs","repos_url":"https://api.github.com/users/open-data/repos","events_url":"https://api.github.com/users/open-data/events{/privacy}","received_events_url":"https://api.github.com/users/open-data/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/open-data/ckanext-canada","description":"Government of Canada CKAN Extension - Extension à CKAN du Gouvernement du Canada","fork":false,"url":"https://api.github.com/repos/open-data/ckanext-canada","forks_url":"https://api.github.com/repos/open-data/ckanext-canada/forks","keys_url":"https://api.github.com/repos/open-data/ckanext-canada/keys{/key_id}","collaborators_url":"https://api.github.com/repos/open-data/ckanext-canada/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/open-data/ckanext-canada/teams","hooks_url":"https://api.github.com/repos/open-data/ckanext-canada/hooks","issue_events_url":"https://api.github.com/repos/open-data/ckanext-canada/issues/events{/number}","events_url":"https://api.github.com/repos/open-data/ckanext-canada/events","assignees_url":"https://api.github.com/repos/open-data/ckanext-canada/assignees{/user}","branches_url":"https://api.github.com/repos/open-data/ckanext-canada/branches{/branch}","tags_url":"https://api.github.com/repos/open-data/ckanext-canada/tags","blobs_url":"https://api.github.com/repos/open-data/ckanext-canada/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/open-data/ckanext-canada/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/open-data/ckanext-canada/git/refs{/sha}","trees_url":"https://api.github.com/repos/open-data/ckanext-canada/git/trees{/sha}","statuses_url":"https://api.github.com/repos/open-data/ckanext-canada/statuses/{sha}","languages_url":"https://api.github.com/repos/open-data/ckanext-canada/languages","stargazers_url":"https://api.github.com/repos/open-data/ckanext-canada/stargazers","contributors_url":"https://api.github.com/repos/open-data/ckanext-canada/contributors","subscribers_url":"https://api.github.com/repos/open-data/ckanext-canada/subscribers","subscription_url":"https://api.github.com/repos/open-data/ckanext-canada/subscription","commits_url":"https://api.github.com/repos/open-data/ckanext-canada/commits{/sha}","git_commits_url":"https://api.github.com/repos/open-data/ckanext-canada/git/commits{/sha}","comments_url":"https://api.github.com/repos/open-data/ckanext-canada/comments{/number}","issue_comment_url":"https://api.github.com/repos/open-data/ckanext-canada/issues/comments{/number}","contents_url":"https://api.github.com/repos/open-data/ckanext-canada/contents/{+path}","compare_url":"https://api.github.com/repos/open-data/ckanext-canada/compare/{base}...{head}","merges_url":"https://api.github.com/repos/open-data/ckanext-canada/merges","archive_url":"https://api.github.com/repos/open-data/ckanext-canada/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/open-data/ckanext-canada/downloads","issues_url":"https://api.github.com/repos/open-data/ckanext-canada/issues{/number}","pulls_url":"https://api.github.com/repos/open-data/ckanext-canada/pulls{/number}","milestones_url":"https://api.github.com/repos/open-data/ckanext-canada/milestones{/number}","notifications_url":"https://api.github.com/repos/open-data/ckanext-canada/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/open-data/ckanext-canada/labels{/name}","releases_url":"https://api.github.com/repos/open-data/ckanext-canada/releases{/id}","created_at":"2012-12-12T19:34:43Z","updated_at":"2015-03-16T21:49:47Z","pushed_at":"2015-03-26T17:07:50Z","git_url":"git://github.com/open-data/ckanext-canada.git","ssh_url":"git@github.com:open-data/ckanext-canada.git","clone_url":"https://github.com/open-data/ckanext-canada.git","svn_url":"https://github.com/open-data/ckanext-canada","homepage":null,"size":15655,"stargazers_count":10,"watchers_count":10,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":8,"mirror_url":null,"open_issues_count":25,"forks":8,"open_issues":25,"watchers":10,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/open-data/ckanext-canada/pulls/441"},"html":{"href":"https://github.com/open-data/ckanext-canada/pull/441"},"issue":{"href":"https://api.github.com/repos/open-data/ckanext-canada/issues/441"},"comments":{"href":"https://api.github.com/repos/open-data/ckanext-canada/issues/441/comments"},"review_comments":{"href":"https://api.github.com/repos/open-data/ckanext-canada/pulls/441/comments"},"review_comment":{"href":"https://api.github.com/repos/open-data/ckanext-canada/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/open-data/ckanext-canada/pulls/441/commits"},"statuses":{"href":"https://api.github.com/repos/open-data/ckanext-canada/statuses/a963965b2a70452dac1bfadd5b36e416fbfad2b4"}}}}