{"action":"created","comment":{"url":"https://api.github.com/repos/USGS-R/geoknife/pulls/comments/35533234","id":35533234,"diff_hunk":"@@ -65,21 +77,25 @@ parseTimeseries <- function(file, delim, with.units = FALSE){\n   return(dataOut)\n }\n \n-parseConfig = function(file, delim){\n-  featureLine = 2\n-  skipHead = 1\n-  varMarker = '# '\n-  c <- file(file,\"r\") \n-  fileLines <- readLines(c)\n-  close(c)\n-  nRead <- length(fileLines)\n-  blockStart <- grep(varMarker, fileLines)\n-  skips = blockStart+skipHead\n-  blockEnd = c(blockStart[-1] - 1, nRead)\n-  nrows = blockEnd - skips - 1\n-  features = unique(strsplit(fileLines[featureLine], split = delim)[[1]][-1])\n-  vars = sub(varMarker,\"\",fileLines[blockStart])\n-  config = list(vars = vars, features = features, skip = skips, nrows = nrows)\n+parseConfig = function(file, delim, largeFile=FALSE){\n+  if(largeFile==FALSE){\n+    featureLine = 2 # Line containing unique IDs of features (stencil) that were processed\n+    skipHead = 1 # Number of lines to skip past the variable marker header?\n+    varMarker = '# ' # Symbol that denotes a variable identifier and a new block of output.\n+    c <- file(file,\"r\") \n+    fileLines <- readLines(c) # This concerns me!!!\n+    close(c)\n+    nRead <- length(fileLines)\n+    blockStart <- grep(varMarker, fileLines) # Lines containing variable IDs.\n+    skips = blockStart+skipHead \n+    blockEnd = c(blockStart[-1] - 1, nRead) # End of blocks\n+    nrows = blockEnd - skips - 1 # Number of ros per block.\n+    features = unique(strsplit(fileLines[featureLine], split = delim)[[1]][-1]) # Parsing out feature identifiers\n+    vars = sub(varMarker,\"\",fileLines[blockStart]) # Getting the variable names from the block starts.\n+    config = list(vars = vars, features = features, skip = skips, nrows = nrows) # Return all the good stuff!\n+  } else {\n+    stop('Large File Support Not Implemented')","path":"R/parseTimeseries.R","position":71,"original_position":71,"commit_id":"4855bd96d96a7f023474fb7f094167e854122366","original_commit_id":"4855bd96d96a7f023474fb7f094167e854122366","user":{"login":"dblodgett-usgs","id":1492803,"avatar_url":"https://avatars.githubusercontent.com/u/1492803?v=3","gravatar_id":"","url":"https://api.github.com/users/dblodgett-usgs","html_url":"https://github.com/dblodgett-usgs","followers_url":"https://api.github.com/users/dblodgett-usgs/followers","following_url":"https://api.github.com/users/dblodgett-usgs/following{/other_user}","gists_url":"https://api.github.com/users/dblodgett-usgs/gists{/gist_id}","starred_url":"https://api.github.com/users/dblodgett-usgs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dblodgett-usgs/subscriptions","organizations_url":"https://api.github.com/users/dblodgett-usgs/orgs","repos_url":"https://api.github.com/users/dblodgett-usgs/repos","events_url":"https://api.github.com/users/dblodgett-usgs/events{/privacy}","received_events_url":"https://api.github.com/users/dblodgett-usgs/received_events","type":"User","site_admin":false},"body":"ASCII encoded data is way more voluminous than the binary encoded numbers that go into a data frame. I could also imagine a scenario where a person would want to iterate over the variables in an output file getting one data frame at a time. ","created_at":"2015-07-27T13:12:07Z","updated_at":"2015-07-27T13:12:07Z","html_url":"https://github.com/USGS-R/geoknife/pull/184#discussion_r35533234","pull_request_url":"https://api.github.com/repos/USGS-R/geoknife/pulls/184","_links":{"self":{"href":"https://api.github.com/repos/USGS-R/geoknife/pulls/comments/35533234"},"html":{"href":"https://github.com/USGS-R/geoknife/pull/184#discussion_r35533234"},"pull_request":{"href":"https://api.github.com/repos/USGS-R/geoknife/pulls/184"}}},"pull_request":{"url":"https://api.github.com/repos/USGS-R/geoknife/pulls/184","id":40866958,"html_url":"https://github.com/USGS-R/geoknife/pull/184","diff_url":"https://github.com/USGS-R/geoknife/pull/184.diff","patch_url":"https://github.com/USGS-R/geoknife/pull/184.patch","issue_url":"https://api.github.com/repos/USGS-R/geoknife/issues/184","number":184,"state":"open","locked":false,"title":"Multi-threshold parsing.","user":{"login":"dblodgett-usgs","id":1492803,"avatar_url":"https://avatars.githubusercontent.com/u/1492803?v=3","gravatar_id":"","url":"https://api.github.com/users/dblodgett-usgs","html_url":"https://github.com/dblodgett-usgs","followers_url":"https://api.github.com/users/dblodgett-usgs/followers","following_url":"https://api.github.com/users/dblodgett-usgs/following{/other_user}","gists_url":"https://api.github.com/users/dblodgett-usgs/gists{/gist_id}","starred_url":"https://api.github.com/users/dblodgett-usgs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dblodgett-usgs/subscriptions","organizations_url":"https://api.github.com/users/dblodgett-usgs/orgs","repos_url":"https://api.github.com/users/dblodgett-usgs/repos","events_url":"https://api.github.com/users/dblodgett-usgs/events{/privacy}","received_events_url":"https://api.github.com/users/dblodgett-usgs/received_events","type":"User","site_admin":false},"body":"Got an initial pass at multi-threshold parsing done. Did it totally test driven and made sure everything is passing after I finished. \r\n\r\nI commented up the parser config function for my benefit in the moment, happy to throw out those comments. \r\n\r\nNote that I added a placeholder for some more 'streaming' parser logic as a 'largeFile' option that could be invoked for mongo files. \r\n\r\nWe should discuss what to actually do with the threshold column content. The result here is just another collumn that will need to be dealt with after the fact, I'm thinking it might be good to extract the thresholds into seperate variables or something... ","created_at":"2015-07-27T03:02:25Z","updated_at":"2015-07-27T13:12:07Z","closed_at":null,"merged_at":null,"merge_commit_sha":"a1d10bd6cf486ae7f383d54469c6229001035388","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/USGS-R/geoknife/pulls/184/commits","review_comments_url":"https://api.github.com/repos/USGS-R/geoknife/pulls/184/comments","review_comment_url":"https://api.github.com/repos/USGS-R/geoknife/pulls/comments{/number}","comments_url":"https://api.github.com/repos/USGS-R/geoknife/issues/184/comments","statuses_url":"https://api.github.com/repos/USGS-R/geoknife/statuses/4855bd96d96a7f023474fb7f094167e854122366","head":{"label":"dblodgett-usgs:master","ref":"master","sha":"4855bd96d96a7f023474fb7f094167e854122366","user":{"login":"dblodgett-usgs","id":1492803,"avatar_url":"https://avatars.githubusercontent.com/u/1492803?v=3","gravatar_id":"","url":"https://api.github.com/users/dblodgett-usgs","html_url":"https://github.com/dblodgett-usgs","followers_url":"https://api.github.com/users/dblodgett-usgs/followers","following_url":"https://api.github.com/users/dblodgett-usgs/following{/other_user}","gists_url":"https://api.github.com/users/dblodgett-usgs/gists{/gist_id}","starred_url":"https://api.github.com/users/dblodgett-usgs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dblodgett-usgs/subscriptions","organizations_url":"https://api.github.com/users/dblodgett-usgs/orgs","repos_url":"https://api.github.com/users/dblodgett-usgs/repos","events_url":"https://api.github.com/users/dblodgett-usgs/events{/privacy}","received_events_url":"https://api.github.com/users/dblodgett-usgs/received_events","type":"User","site_admin":false},"repo":{"id":35959476,"name":"geoknife","full_name":"dblodgett-usgs/geoknife","owner":{"login":"dblodgett-usgs","id":1492803,"avatar_url":"https://avatars.githubusercontent.com/u/1492803?v=3","gravatar_id":"","url":"https://api.github.com/users/dblodgett-usgs","html_url":"https://github.com/dblodgett-usgs","followers_url":"https://api.github.com/users/dblodgett-usgs/followers","following_url":"https://api.github.com/users/dblodgett-usgs/following{/other_user}","gists_url":"https://api.github.com/users/dblodgett-usgs/gists{/gist_id}","starred_url":"https://api.github.com/users/dblodgett-usgs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dblodgett-usgs/subscriptions","organizations_url":"https://api.github.com/users/dblodgett-usgs/orgs","repos_url":"https://api.github.com/users/dblodgett-usgs/repos","events_url":"https://api.github.com/users/dblodgett-usgs/events{/privacy}","received_events_url":"https://api.github.com/users/dblodgett-usgs/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/dblodgett-usgs/geoknife","description":"R tools for geo-web processing of gridded data via the Geo Data Portal. geoknife slices up gridded data according to overlap with irregular features, such as watersheds, lakes, points, etc.","fork":true,"url":"https://api.github.com/repos/dblodgett-usgs/geoknife","forks_url":"https://api.github.com/repos/dblodgett-usgs/geoknife/forks","keys_url":"https://api.github.com/repos/dblodgett-usgs/geoknife/keys{/key_id}","collaborators_url":"https://api.github.com/repos/dblodgett-usgs/geoknife/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/dblodgett-usgs/geoknife/teams","hooks_url":"https://api.github.com/repos/dblodgett-usgs/geoknife/hooks","issue_events_url":"https://api.github.com/repos/dblodgett-usgs/geoknife/issues/events{/number}","events_url":"https://api.github.com/repos/dblodgett-usgs/geoknife/events","assignees_url":"https://api.github.com/repos/dblodgett-usgs/geoknife/assignees{/user}","branches_url":"https://api.github.com/repos/dblodgett-usgs/geoknife/branches{/branch}","tags_url":"https://api.github.com/repos/dblodgett-usgs/geoknife/tags","blobs_url":"https://api.github.com/repos/dblodgett-usgs/geoknife/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/dblodgett-usgs/geoknife/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/dblodgett-usgs/geoknife/git/refs{/sha}","trees_url":"https://api.github.com/repos/dblodgett-usgs/geoknife/git/trees{/sha}","statuses_url":"https://api.github.com/repos/dblodgett-usgs/geoknife/statuses/{sha}","languages_url":"https://api.github.com/repos/dblodgett-usgs/geoknife/languages","stargazers_url":"https://api.github.com/repos/dblodgett-usgs/geoknife/stargazers","contributors_url":"https://api.github.com/repos/dblodgett-usgs/geoknife/contributors","subscribers_url":"https://api.github.com/repos/dblodgett-usgs/geoknife/subscribers","subscription_url":"https://api.github.com/repos/dblodgett-usgs/geoknife/subscription","commits_url":"https://api.github.com/repos/dblodgett-usgs/geoknife/commits{/sha}","git_commits_url":"https://api.github.com/repos/dblodgett-usgs/geoknife/git/commits{/sha}","comments_url":"https://api.github.com/repos/dblodgett-usgs/geoknife/comments{/number}","issue_comment_url":"https://api.github.com/repos/dblodgett-usgs/geoknife/issues/comments{/number}","contents_url":"https://api.github.com/repos/dblodgett-usgs/geoknife/contents/{+path}","compare_url":"https://api.github.com/repos/dblodgett-usgs/geoknife/compare/{base}...{head}","merges_url":"https://api.github.com/repos/dblodgett-usgs/geoknife/merges","archive_url":"https://api.github.com/repos/dblodgett-usgs/geoknife/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/dblodgett-usgs/geoknife/downloads","issues_url":"https://api.github.com/repos/dblodgett-usgs/geoknife/issues{/number}","pulls_url":"https://api.github.com/repos/dblodgett-usgs/geoknife/pulls{/number}","milestones_url":"https://api.github.com/repos/dblodgett-usgs/geoknife/milestones{/number}","notifications_url":"https://api.github.com/repos/dblodgett-usgs/geoknife/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/dblodgett-usgs/geoknife/labels{/name}","releases_url":"https://api.github.com/repos/dblodgett-usgs/geoknife/releases{/id}","created_at":"2015-05-20T16:16:52Z","updated_at":"2015-05-20T16:16:52Z","pushed_at":"2015-07-27T02:59:05Z","git_url":"git://github.com/dblodgett-usgs/geoknife.git","ssh_url":"git@github.com:dblodgett-usgs/geoknife.git","clone_url":"https://github.com/dblodgett-usgs/geoknife.git","svn_url":"https://github.com/dblodgett-usgs/geoknife","homepage":"usgs-r.github.io/geoknife","size":1304,"stargazers_count":0,"watchers_count":0,"language":"R","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"USGS-R:master","ref":"master","sha":"992d52265b28e84df035550af1d209a5146d24c8","user":{"login":"USGS-R","id":3188813,"avatar_url":"https://avatars.githubusercontent.com/u/3188813?v=3","gravatar_id":"","url":"https://api.github.com/users/USGS-R","html_url":"https://github.com/USGS-R","followers_url":"https://api.github.com/users/USGS-R/followers","following_url":"https://api.github.com/users/USGS-R/following{/other_user}","gists_url":"https://api.github.com/users/USGS-R/gists{/gist_id}","starred_url":"https://api.github.com/users/USGS-R/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/USGS-R/subscriptions","organizations_url":"https://api.github.com/users/USGS-R/orgs","repos_url":"https://api.github.com/users/USGS-R/repos","events_url":"https://api.github.com/users/USGS-R/events{/privacy}","received_events_url":"https://api.github.com/users/USGS-R/received_events","type":"Organization","site_admin":false},"repo":{"id":10252218,"name":"geoknife","full_name":"USGS-R/geoknife","owner":{"login":"USGS-R","id":3188813,"avatar_url":"https://avatars.githubusercontent.com/u/3188813?v=3","gravatar_id":"","url":"https://api.github.com/users/USGS-R","html_url":"https://github.com/USGS-R","followers_url":"https://api.github.com/users/USGS-R/followers","following_url":"https://api.github.com/users/USGS-R/following{/other_user}","gists_url":"https://api.github.com/users/USGS-R/gists{/gist_id}","starred_url":"https://api.github.com/users/USGS-R/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/USGS-R/subscriptions","organizations_url":"https://api.github.com/users/USGS-R/orgs","repos_url":"https://api.github.com/users/USGS-R/repos","events_url":"https://api.github.com/users/USGS-R/events{/privacy}","received_events_url":"https://api.github.com/users/USGS-R/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/USGS-R/geoknife","description":"R tools for geo-web processing of gridded data via the Geo Data Portal. geoknife slices up gridded data according to overlap with irregular features, such as watersheds, lakes, points, etc.","fork":false,"url":"https://api.github.com/repos/USGS-R/geoknife","forks_url":"https://api.github.com/repos/USGS-R/geoknife/forks","keys_url":"https://api.github.com/repos/USGS-R/geoknife/keys{/key_id}","collaborators_url":"https://api.github.com/repos/USGS-R/geoknife/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/USGS-R/geoknife/teams","hooks_url":"https://api.github.com/repos/USGS-R/geoknife/hooks","issue_events_url":"https://api.github.com/repos/USGS-R/geoknife/issues/events{/number}","events_url":"https://api.github.com/repos/USGS-R/geoknife/events","assignees_url":"https://api.github.com/repos/USGS-R/geoknife/assignees{/user}","branches_url":"https://api.github.com/repos/USGS-R/geoknife/branches{/branch}","tags_url":"https://api.github.com/repos/USGS-R/geoknife/tags","blobs_url":"https://api.github.com/repos/USGS-R/geoknife/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/USGS-R/geoknife/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/USGS-R/geoknife/git/refs{/sha}","trees_url":"https://api.github.com/repos/USGS-R/geoknife/git/trees{/sha}","statuses_url":"https://api.github.com/repos/USGS-R/geoknife/statuses/{sha}","languages_url":"https://api.github.com/repos/USGS-R/geoknife/languages","stargazers_url":"https://api.github.com/repos/USGS-R/geoknife/stargazers","contributors_url":"https://api.github.com/repos/USGS-R/geoknife/contributors","subscribers_url":"https://api.github.com/repos/USGS-R/geoknife/subscribers","subscription_url":"https://api.github.com/repos/USGS-R/geoknife/subscription","commits_url":"https://api.github.com/repos/USGS-R/geoknife/commits{/sha}","git_commits_url":"https://api.github.com/repos/USGS-R/geoknife/git/commits{/sha}","comments_url":"https://api.github.com/repos/USGS-R/geoknife/comments{/number}","issue_comment_url":"https://api.github.com/repos/USGS-R/geoknife/issues/comments{/number}","contents_url":"https://api.github.com/repos/USGS-R/geoknife/contents/{+path}","compare_url":"https://api.github.com/repos/USGS-R/geoknife/compare/{base}...{head}","merges_url":"https://api.github.com/repos/USGS-R/geoknife/merges","archive_url":"https://api.github.com/repos/USGS-R/geoknife/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/USGS-R/geoknife/downloads","issues_url":"https://api.github.com/repos/USGS-R/geoknife/issues{/number}","pulls_url":"https://api.github.com/repos/USGS-R/geoknife/pulls{/number}","milestones_url":"https://api.github.com/repos/USGS-R/geoknife/milestones{/number}","notifications_url":"https://api.github.com/repos/USGS-R/geoknife/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/USGS-R/geoknife/labels{/name}","releases_url":"https://api.github.com/repos/USGS-R/geoknife/releases{/id}","created_at":"2013-05-23T20:00:51Z","updated_at":"2015-07-14T19:26:43Z","pushed_at":"2015-07-27T03:02:26Z","git_url":"git://github.com/USGS-R/geoknife.git","ssh_url":"git@github.com:USGS-R/geoknife.git","clone_url":"https://github.com/USGS-R/geoknife.git","svn_url":"https://github.com/USGS-R/geoknife","homepage":"usgs-r.github.io/geoknife","size":1846,"stargazers_count":6,"watchers_count":6,"language":"R","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":8,"mirror_url":null,"open_issues_count":23,"forks":8,"open_issues":23,"watchers":6,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/USGS-R/geoknife/pulls/184"},"html":{"href":"https://github.com/USGS-R/geoknife/pull/184"},"issue":{"href":"https://api.github.com/repos/USGS-R/geoknife/issues/184"},"comments":{"href":"https://api.github.com/repos/USGS-R/geoknife/issues/184/comments"},"review_comments":{"href":"https://api.github.com/repos/USGS-R/geoknife/pulls/184/comments"},"review_comment":{"href":"https://api.github.com/repos/USGS-R/geoknife/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/USGS-R/geoknife/pulls/184/commits"},"statuses":{"href":"https://api.github.com/repos/USGS-R/geoknife/statuses/4855bd96d96a7f023474fb7f094167e854122366"}}}}