{"action":"created","issue":{"url":"https://api.github.com/repos/ngds/ckanext-ngds/issues/610","labels_url":"https://api.github.com/repos/ngds/ckanext-ngds/issues/610/labels{/name}","comments_url":"https://api.github.com/repos/ngds/ckanext-ngds/issues/610/comments","events_url":"https://api.github.com/repos/ngds/ckanext-ngds/issues/610/events","html_url":"https://github.com/ngds/ckanext-ngds/issues/610","id":59822046,"number":610,"title":" GDR-NGDS integration","user":{"login":"ykhadilkar-rei","id":3237208,"avatar_url":"https://avatars.githubusercontent.com/u/3237208?v=3","gravatar_id":"","url":"https://api.github.com/users/ykhadilkar-rei","html_url":"https://github.com/ykhadilkar-rei","followers_url":"https://api.github.com/users/ykhadilkar-rei/followers","following_url":"https://api.github.com/users/ykhadilkar-rei/following{/other_user}","gists_url":"https://api.github.com/users/ykhadilkar-rei/gists{/gist_id}","starred_url":"https://api.github.com/users/ykhadilkar-rei/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ykhadilkar-rei/subscriptions","organizations_url":"https://api.github.com/users/ykhadilkar-rei/orgs","repos_url":"https://api.github.com/users/ykhadilkar-rei/repos","events_url":"https://api.github.com/users/ykhadilkar-rei/events{/privacy}","received_events_url":"https://api.github.com/users/ykhadilkar-rei/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/ngds/ckanext-ngds/labels/in%20progress","name":"in progress","color":"ededed"},{"url":"https://api.github.com/repos/ngds/ckanext-ngds/labels/NGDS-v2.0","name":"NGDS-v2.0","color":"207de5"},{"url":"https://api.github.com/repos/ngds/ckanext-ngds/labels/priority.Major","name":"priority.Major","color":"e11d21"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/ngds/ckanext-ngds/milestones/20","html_url":"https://github.com/ngds/ckanext-ngds/milestones/REI","labels_url":"https://api.github.com/repos/ngds/ckanext-ngds/milestones/20/labels","id":865478,"number":20,"title":"REI","description":"","creator":{"login":"ccaudill","id":3188489,"avatar_url":"https://avatars.githubusercontent.com/u/3188489?v=3","gravatar_id":"","url":"https://api.github.com/users/ccaudill","html_url":"https://github.com/ccaudill","followers_url":"https://api.github.com/users/ccaudill/followers","following_url":"https://api.github.com/users/ccaudill/following{/other_user}","gists_url":"https://api.github.com/users/ccaudill/gists{/gist_id}","starred_url":"https://api.github.com/users/ccaudill/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ccaudill/subscriptions","organizations_url":"https://api.github.com/users/ccaudill/orgs","repos_url":"https://api.github.com/users/ccaudill/repos","events_url":"https://api.github.com/users/ccaudill/events{/privacy}","received_events_url":"https://api.github.com/users/ccaudill/received_events","type":"User","site_admin":false},"open_issues":10,"closed_issues":13,"state":"open","created_at":"2014-11-13T18:56:33Z","updated_at":"2015-04-30T19:46:17Z","due_on":null,"closed_at":null},"comments":21,"created_at":"2015-03-04T16:26:27Z","updated_at":"2015-05-13T17:37:40Z","closed_at":null,"body":"Some background \r\n1.  Here's the part of the USGIN harvester which extends the vanilla CKAN CSW harvester: https://github.com/ngds/ckanext-metadata/blob/master/ckanext/harvest/usgin.py#L96.  This function has a look through each individual harvested metadata record and pulls out the parts of interest.  Off the top of my head, here is where I'd throw a hook in to pull out those CSV file URLs and start working with them.\r\n \r\n2.  Here's your validation service, the venerable USGIN Models API: https://github.com/usgin/usginmodels.  This is what you'd use for validation.\r\n \r\n3.  Now there's two ways you can deploy the CSV file through Geoserver.  The way that makes most sense would be to use CKAN's DataPusher software.  NGDS CKAN should already deploy with it.  You'd probably need to hook directly into this function: https://github.com/ckan/datapusher/blob/115d3fe33c1a9041e17a964a3eda9afe72f82437/datapusher/jobs.py#L250 and it would take care of everything for you.  If you don't care about storing the contents of the CSV in the database, then use GDAL/OGR to make a VRT of the CSV and feed it to Geoserver instead of using the DataPusher.\r\n \r\n4.  Then you'd use https://github.com/ngds/ckanext-geoserver to publish the OGC services you want.\r\nWith all that laid out, this task looks like it's more about chaining together a bunch of existing processes than writing new software.  There's probably multiple ways to do it, but here's two ways I can think of off the top of my head:\r\nFor both scenarios, you're probably going to want to use the DataPusher function mentioned in item #3 above.  You're going to want to alter it so that it returns a success/failure value because your going to have to implement it with or as a dependency injection.  To implement it as a dependency injection, it'll be passed as a parameter into another function that handles storing the CSV in the database and publishing to Geoserver.  Alternatively, to implement it with a dependency injection you'd pass the publish to Geoserver function in as the dependency injection to inverse the flow.  The reason inversion of control is necessary here is because the program needs to listen for the DataPusher process to finish.  Normally, it operates asynchronously but it's going to need to run as a blocking process.  However you want to implement it, I'll refer to this process as \"ComplicatedProcess\".\r\n \r\nThe easy way:\r\nExtend https://github.com/ngds/ckanext-metadata to support the new functionality.  Specifically, add a bunch more private functions to this object: https://github.com/ngds/ckanext-metadata/blob/master/ckanext/harvest/usgin.py#L11.  Ultra-pseudo-code:\r\n-- If string scraped from metadata is CSV, download CSV.\r\n-- If CSV downloads, the entire harvest process gets put on hold and we start this new process.  Else log the error and continue the harvest as usual.\r\n-- Feed CSV to USGIN Models API for validation\r\n-- If validation passes, pass the CSV to \"ComplicatedProcess\" and wait for return.  If success, enter new metadata entry.  Else, pass and return back to regular harvest process.\r\nThis would be the easiest way to implement the logic, but you'd be taking a tremendous performance hit.  Everything would run in a blocking event loop and if a harvest contains a lot of CSVs to be harvested you could expect the whole harvest to take at least twice as long, probably longer.\r\nThe sexy, elegant way:\r\nYou'd do everything as mentioned in the easy way except you'd use a framework like twisted (https://twistedmatrix.com/trac/) or tornado (http://www.tornadoweb.org/en/stable/) to create an asynchronous, non-blocking pool of processes to minimize the performance hit.\r\n"},"comment":{"url":"https://api.github.com/repos/ngds/ckanext-ngds/issues/comments/101754956","html_url":"https://github.com/ngds/ckanext-ngds/issues/610#issuecomment-101754956","issue_url":"https://api.github.com/repos/ngds/ckanext-ngds/issues/610","id":101754956,"user":{"login":"ccaudill","id":3188489,"avatar_url":"https://avatars.githubusercontent.com/u/3188489?v=3","gravatar_id":"","url":"https://api.github.com/users/ccaudill","html_url":"https://github.com/ccaudill","followers_url":"https://api.github.com/users/ccaudill/followers","following_url":"https://api.github.com/users/ccaudill/following{/other_user}","gists_url":"https://api.github.com/users/ccaudill/gists{/gist_id}","starred_url":"https://api.github.com/users/ccaudill/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ccaudill/subscriptions","organizations_url":"https://api.github.com/users/ccaudill/orgs","repos_url":"https://api.github.com/users/ccaudill/repos","events_url":"https://api.github.com/users/ccaudill/events{/privacy}","received_events_url":"https://api.github.com/users/ccaudill/received_events","type":"User","site_admin":false},"created_at":"2015-05-13T17:37:40Z","updated_at":"2015-05-13T17:37:40Z","body":"This should be completed and already in the new rpm. We need to do a test-case on our end, then this issue can be closed.\r\n\r\nA separate issue will be made for the automatic Excel - CSV conversion of the dataset (upon harvest and before publishing).\r\n"}}