{"action":"created","comment":{"url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/pulls/comments/45854177","id":45854177,"diff_hunk":"@@ -113,28 +113,33 @@\n   (let [{:keys [project-id embed-url username password] :as opts}\n         (:options (parse-opts args cli-options))\n         {:keys [mapping-file processed-file]} config-info\n-        s3-files (list-files-in-bucket)]\n-    (log/info \"S3 files:\" s3-files)\n-    (doseq [f s3-files]\n-      (println \"> FILE \" f)\n-      (let [file-info (s3/get-object cred bucket f)\n-            input-data (with-open [r (->> (s3/get-object cred bucket f)\n-                                          :object-content\n-                                          io/reader)]\n-                         (pro/file->seq-of-maps r))\n-            processed-content (with-open [in-file (io/reader processed-file)]\n-                                (vec (csv/read-csv in-file)))\n-            data-to-save (conj (vec processed-content)\n-                               [(f/unparse fmt (t/now)) (:key file-info) (:bucket-name file-info)\n-                                (:object-metadata file-info) (:object-content file-info)])]\n-        (println \">> Pre-processing... \" f)\n-        (pro/write-to-file processed-file data-to-save)\n-        (pre-processing input-data\n-                        mapping-file project-id\n-                        embed-url username password)\n-        (println \">> Processing... \" f)\n-        (upload-measurement-data input-data mapping-file\n-                                 embed-url username password)))))\n+        processed-files (map :file_name (pro/file->seq-of-maps processed-file))\n+        s3-files (list-files-in-bucket)","path":"src/kixi/hecuba/minorca.clj","position":27,"original_position":27,"commit_id":"d3566990869c74ea1162c93d93f8f8b3db2aa04a","original_commit_id":"d3566990869c74ea1162c93d93f8f8b3db2aa04a","user":{"login":"otfrom","id":64839,"avatar_url":"https://avatars.githubusercontent.com/u/64839?v=3","gravatar_id":"","url":"https://api.github.com/users/otfrom","html_url":"https://github.com/otfrom","followers_url":"https://api.github.com/users/otfrom/followers","following_url":"https://api.github.com/users/otfrom/following{/other_user}","gists_url":"https://api.github.com/users/otfrom/gists{/gist_id}","starred_url":"https://api.github.com/users/otfrom/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/otfrom/subscriptions","organizations_url":"https://api.github.com/users/otfrom/orgs","repos_url":"https://api.github.com/users/otfrom/repos","events_url":"https://api.github.com/users/otfrom/events{/privacy}","received_events_url":"https://api.github.com/users/otfrom/received_events","type":"User","site_admin":false},"body":"I'd rather we had cred and config loaded by functions in this let rather than as top level defs. Then pass in bucket and cred rather than referring to them globally. It would be different if they were constants but they aren't as they are read from a config file that might not be there.","created_at":"2015-11-25T11:04:00Z","updated_at":"2015-11-25T11:04:00Z","html_url":"https://github.com/MastodonC/kixi.hecuba.minorca/pull/10#discussion_r45854177","pull_request_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/pulls/10","_links":{"self":{"href":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/pulls/comments/45854177"},"html":{"href":"https://github.com/MastodonC/kixi.hecuba.minorca/pull/10#discussion_r45854177"},"pull_request":{"href":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/pulls/10"}}},"pull_request":{"url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/pulls/10","id":51552694,"html_url":"https://github.com/MastodonC/kixi.hecuba.minorca/pull/10","diff_url":"https://github.com/MastodonC/kixi.hecuba.minorca/pull/10.diff","patch_url":"https://github.com/MastodonC/kixi.hecuba.minorca/pull/10.patch","issue_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/issues/10","number":10,"state":"open","locked":false,"title":"[Ready]Add tests + use non processed files","user":{"login":"Eleonore9","id":2502761,"avatar_url":"https://avatars.githubusercontent.com/u/2502761?v=3","gravatar_id":"","url":"https://api.github.com/users/Eleonore9","html_url":"https://github.com/Eleonore9","followers_url":"https://api.github.com/users/Eleonore9/followers","following_url":"https://api.github.com/users/Eleonore9/following{/other_user}","gists_url":"https://api.github.com/users/Eleonore9/gists{/gist_id}","starred_url":"https://api.github.com/users/Eleonore9/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Eleonore9/subscriptions","organizations_url":"https://api.github.com/users/Eleonore9/orgs","repos_url":"https://api.github.com/users/Eleonore9/repos","events_url":"https://api.github.com/users/Eleonore9/events{/privacy}","received_events_url":"https://api.github.com/users/Eleonore9/received_events","type":"User","site_admin":false},"body":"Check for file names in the \"files processed\" csv -> remove them from the list of files to be processed\r\n\r\nNote: \r\nFor that I had to improve the way I get info from csv. When the keys were strings with a space the space paused a problem. Now a key like \"Column One\" becomes \"column_one\". See new tests.","created_at":"2015-11-23T17:19:43Z","updated_at":"2015-11-25T11:04:00Z","closed_at":null,"merged_at":null,"merge_commit_sha":"6538aaabd5905e295f64cb1d2c23b0f6222ef654","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/pulls/10/commits","review_comments_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/pulls/10/comments","review_comment_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/pulls/comments{/number}","comments_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/issues/10/comments","statuses_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/statuses/d3566990869c74ea1162c93d93f8f8b3db2aa04a","head":{"label":"MastodonC:feature/process-only-new-files","ref":"feature/process-only-new-files","sha":"d3566990869c74ea1162c93d93f8f8b3db2aa04a","user":{"login":"MastodonC","id":2911834,"avatar_url":"https://avatars.githubusercontent.com/u/2911834?v=3","gravatar_id":"","url":"https://api.github.com/users/MastodonC","html_url":"https://github.com/MastodonC","followers_url":"https://api.github.com/users/MastodonC/followers","following_url":"https://api.github.com/users/MastodonC/following{/other_user}","gists_url":"https://api.github.com/users/MastodonC/gists{/gist_id}","starred_url":"https://api.github.com/users/MastodonC/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MastodonC/subscriptions","organizations_url":"https://api.github.com/users/MastodonC/orgs","repos_url":"https://api.github.com/users/MastodonC/repos","events_url":"https://api.github.com/users/MastodonC/events{/privacy}","received_events_url":"https://api.github.com/users/MastodonC/received_events","type":"Organization","site_admin":false},"repo":{"id":45925844,"name":"kixi.hecuba.minorca","full_name":"MastodonC/kixi.hecuba.minorca","owner":{"login":"MastodonC","id":2911834,"avatar_url":"https://avatars.githubusercontent.com/u/2911834?v=3","gravatar_id":"","url":"https://api.github.com/users/MastodonC","html_url":"https://github.com/MastodonC","followers_url":"https://api.github.com/users/MastodonC/followers","following_url":"https://api.github.com/users/MastodonC/following{/other_user}","gists_url":"https://api.github.com/users/MastodonC/gists{/gist_id}","starred_url":"https://api.github.com/users/MastodonC/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MastodonC/subscriptions","organizations_url":"https://api.github.com/users/MastodonC/orgs","repos_url":"https://api.github.com/users/MastodonC/repos","events_url":"https://api.github.com/users/MastodonC/events{/privacy}","received_events_url":"https://api.github.com/users/MastodonC/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/MastodonC/kixi.hecuba.minorca","description":"Create a new properties and devices, load measurements from csv on S3 into Hecuba (getembed.com) via the API","fork":false,"url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca","forks_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/forks","keys_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/keys{/key_id}","collaborators_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/teams","hooks_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/hooks","issue_events_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/issues/events{/number}","events_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/events","assignees_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/assignees{/user}","branches_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/branches{/branch}","tags_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/tags","blobs_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/git/refs{/sha}","trees_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/git/trees{/sha}","statuses_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/statuses/{sha}","languages_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/languages","stargazers_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/stargazers","contributors_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/contributors","subscribers_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/subscribers","subscription_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/subscription","commits_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/commits{/sha}","git_commits_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/git/commits{/sha}","comments_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/comments{/number}","issue_comment_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/issues/comments{/number}","contents_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/contents/{+path}","compare_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/compare/{base}...{head}","merges_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/merges","archive_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/downloads","issues_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/issues{/number}","pulls_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/pulls{/number}","milestones_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/milestones{/number}","notifications_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/labels{/name}","releases_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/releases{/id}","created_at":"2015-11-10T16:46:17Z","updated_at":"2015-11-23T14:44:39Z","pushed_at":"2015-11-23T19:01:46Z","git_url":"git://github.com/MastodonC/kixi.hecuba.minorca.git","ssh_url":"git@github.com:MastodonC/kixi.hecuba.minorca.git","clone_url":"https://github.com/MastodonC/kixi.hecuba.minorca.git","svn_url":"https://github.com/MastodonC/kixi.hecuba.minorca","homepage":"","size":35,"stargazers_count":0,"watchers_count":0,"language":"Clojure","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":1,"forks":0,"open_issues":1,"watchers":0,"default_branch":"master"}},"base":{"label":"MastodonC:master","ref":"master","sha":"25a76b976ca96a13c6339740f5cac48880068a5d","user":{"login":"MastodonC","id":2911834,"avatar_url":"https://avatars.githubusercontent.com/u/2911834?v=3","gravatar_id":"","url":"https://api.github.com/users/MastodonC","html_url":"https://github.com/MastodonC","followers_url":"https://api.github.com/users/MastodonC/followers","following_url":"https://api.github.com/users/MastodonC/following{/other_user}","gists_url":"https://api.github.com/users/MastodonC/gists{/gist_id}","starred_url":"https://api.github.com/users/MastodonC/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MastodonC/subscriptions","organizations_url":"https://api.github.com/users/MastodonC/orgs","repos_url":"https://api.github.com/users/MastodonC/repos","events_url":"https://api.github.com/users/MastodonC/events{/privacy}","received_events_url":"https://api.github.com/users/MastodonC/received_events","type":"Organization","site_admin":false},"repo":{"id":45925844,"name":"kixi.hecuba.minorca","full_name":"MastodonC/kixi.hecuba.minorca","owner":{"login":"MastodonC","id":2911834,"avatar_url":"https://avatars.githubusercontent.com/u/2911834?v=3","gravatar_id":"","url":"https://api.github.com/users/MastodonC","html_url":"https://github.com/MastodonC","followers_url":"https://api.github.com/users/MastodonC/followers","following_url":"https://api.github.com/users/MastodonC/following{/other_user}","gists_url":"https://api.github.com/users/MastodonC/gists{/gist_id}","starred_url":"https://api.github.com/users/MastodonC/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MastodonC/subscriptions","organizations_url":"https://api.github.com/users/MastodonC/orgs","repos_url":"https://api.github.com/users/MastodonC/repos","events_url":"https://api.github.com/users/MastodonC/events{/privacy}","received_events_url":"https://api.github.com/users/MastodonC/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/MastodonC/kixi.hecuba.minorca","description":"Create a new properties and devices, load measurements from csv on S3 into Hecuba (getembed.com) via the API","fork":false,"url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca","forks_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/forks","keys_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/keys{/key_id}","collaborators_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/teams","hooks_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/hooks","issue_events_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/issues/events{/number}","events_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/events","assignees_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/assignees{/user}","branches_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/branches{/branch}","tags_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/tags","blobs_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/git/refs{/sha}","trees_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/git/trees{/sha}","statuses_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/statuses/{sha}","languages_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/languages","stargazers_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/stargazers","contributors_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/contributors","subscribers_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/subscribers","subscription_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/subscription","commits_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/commits{/sha}","git_commits_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/git/commits{/sha}","comments_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/comments{/number}","issue_comment_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/issues/comments{/number}","contents_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/contents/{+path}","compare_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/compare/{base}...{head}","merges_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/merges","archive_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/downloads","issues_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/issues{/number}","pulls_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/pulls{/number}","milestones_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/milestones{/number}","notifications_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/labels{/name}","releases_url":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/releases{/id}","created_at":"2015-11-10T16:46:17Z","updated_at":"2015-11-23T14:44:39Z","pushed_at":"2015-11-23T19:01:46Z","git_url":"git://github.com/MastodonC/kixi.hecuba.minorca.git","ssh_url":"git@github.com:MastodonC/kixi.hecuba.minorca.git","clone_url":"https://github.com/MastodonC/kixi.hecuba.minorca.git","svn_url":"https://github.com/MastodonC/kixi.hecuba.minorca","homepage":"","size":35,"stargazers_count":0,"watchers_count":0,"language":"Clojure","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":1,"forks":0,"open_issues":1,"watchers":0,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/pulls/10"},"html":{"href":"https://github.com/MastodonC/kixi.hecuba.minorca/pull/10"},"issue":{"href":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/issues/10"},"comments":{"href":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/issues/10/comments"},"review_comments":{"href":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/pulls/10/comments"},"review_comment":{"href":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/pulls/10/commits"},"statuses":{"href":"https://api.github.com/repos/MastodonC/kixi.hecuba.minorca/statuses/d3566990869c74ea1162c93d93f8f8b3db2aa04a"}}}}