{"action":"created","issue":{"url":"https://api.github.com/repos/GeoscienceAustralia/agdc/issues/69","labels_url":"https://api.github.com/repos/GeoscienceAustralia/agdc/issues/69/labels{/name}","comments_url":"https://api.github.com/repos/GeoscienceAustralia/agdc/issues/69/comments","events_url":"https://api.github.com/repos/GeoscienceAustralia/agdc/issues/69/events","html_url":"https://github.com/GeoscienceAustralia/agdc/issues/69","id":93933719,"number":69,"title":"Durable/invariant acquisition identifier","user":{"login":"smr547","id":4327895,"avatar_url":"https://avatars.githubusercontent.com/u/4327895?v=3","gravatar_id":"","url":"https://api.github.com/users/smr547","html_url":"https://github.com/smr547","followers_url":"https://api.github.com/users/smr547/followers","following_url":"https://api.github.com/users/smr547/following{/other_user}","gists_url":"https://api.github.com/users/smr547/gists{/gist_id}","starred_url":"https://api.github.com/users/smr547/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smr547/subscriptions","organizations_url":"https://api.github.com/users/smr547/orgs","repos_url":"https://api.github.com/users/smr547/repos","events_url":"https://api.github.com/users/smr547/events{/privacy}","received_events_url":"https://api.github.com/users/smr547/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-07-09T02:13:37Z","updated_at":"2015-07-14T03:30:00Z","closed_at":null,"body":"Since its inception, WOfS has encountered problems with the lack of a stable identifier for \"tiles/acquisitions\".  WOfS currently uses the AGDC convention of \"start timestamp\" which is available via the API and also encoded into the dataset filename.\r\n\r\nOne problem has been that there is fuzziness associated with this value. In the past, should a tile/acquisition be re-ingested the \"start timestamp\" is not guaranteed to retain the same value.  Any application running incremental updates may incorrectly interpret slightly changed start times as new tile.\r\n\r\nThe micorsecond resolution of these timestamps also causes problems with string representation of the timestamp changing depending on the value of (or absence of) the microsecond component. A general solution to this problem requires some REGEX gymnastics (e.g. https://github.com/smr547/ga-neo-nfrip/blob/luigi_api_refactor/wofs/timeparser.py)\r\n\r\nAnyway, I'd like to suggest that we consider using a combination of satellite_id,  \"orbit number\" and AGCD \"cell_ID\" as a durable acquisition identifier within any particular dataset type. \r\n\r\nThis ID has the highly desirable property of not changing between ingestions. The orbit number is readily available from TLEs and could be computed during pre-ingestion processing (if it is not already). \r\n\r\nThe identifier's value can also be easily predetermined. i.e. We can reliably state what acquisition IDs will be generated during the next (or any) orbit of LS8. This has significant benefits when developing QA procedures (i.e. identifiying gaps in our data).\r\n\r\nObviously it also eliminates the possibility of data duplication.\r\n\r\nAs an example\r\n\r\nLS5_TM_NBAR_150_-034_1991-01-09T23-03-58.241019.tif\r\n\r\nmight become\r\n\r\nLS5_NBAR_150_-034_23456.tif"},"comment":{"url":"https://api.github.com/repos/GeoscienceAustralia/agdc/issues/comments/121123804","html_url":"https://github.com/GeoscienceAustralia/agdc/issues/69#issuecomment-121123804","issue_url":"https://api.github.com/repos/GeoscienceAustralia/agdc/issues/69","id":121123804,"user":{"login":"smr547","id":4327895,"avatar_url":"https://avatars.githubusercontent.com/u/4327895?v=3","gravatar_id":"","url":"https://api.github.com/users/smr547","html_url":"https://github.com/smr547","followers_url":"https://api.github.com/users/smr547/followers","following_url":"https://api.github.com/users/smr547/following{/other_user}","gists_url":"https://api.github.com/users/smr547/gists{/gist_id}","starred_url":"https://api.github.com/users/smr547/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smr547/subscriptions","organizations_url":"https://api.github.com/users/smr547/orgs","repos_url":"https://api.github.com/users/smr547/repos","events_url":"https://api.github.com/users/smr547/events{/privacy}","received_events_url":"https://api.github.com/users/smr547/received_events","type":"User","site_admin":false},"created_at":"2015-07-14T03:30:00Z","updated_at":"2015-07-14T03:30:00Z","body":"Thanks Wenjun. Orbit number looks like a slippery eel.  Thanks for\nenlightening me.\n\nI'm happy for this suggestion/issue to be closed :D\n\nOn 14 July 2015 at 12:52, Wenjun Wu <notifications@github.com> wrote:\n\n> For example, LS5 had been parked once. When it came back to service, the\n> orbit number had been reset. Some satellites have different starting orbit\n> number in TLE v.s. ancillary provided by satellite owner (e.g. Radarsat-1).\n> Furthermore, TLE is a predict orbit not actual. The orbit number can be off\n> by one when it closes to equator for an ascending pass.\n>\n> From: Steven Ring [mailto:notifications@github.com]\n> Sent: Tuesday, 14 July 2015 11:24 AM\n> To: GeoscienceAustralia/agdc\n> Cc: Wu Wenjun\n> Subject: Re: [agdc] Durable/invariant acquisition identifier (#69)\n>\n> I'm surprised orbit number cannot be precisely determined. Does that mean\n> that the 'revolution number' in the published TLEs is unreliable?\n> On Jul 14, 2015 10:09 AM, \"Wenjun Wu\" <notifications@github.com<mailto:\n> notifications@github.com>> wrote:\n>\n> > Be aware that “orbit number” is not a reliable value. Depending on how\n> to\n> > define the origin, we have countered +/-2 issue for a same pass. We have\n> > deliberately avoided to use “orbit number” as an identifier. Location,\n> time\n> > and sensor information are the correct combination to form a unique\n> > identifier.\n> >\n> > Regarding the issue caused by microsecond resolution, I would say this\n> is\n> > an implantation error. When we dealing scientific computing, we should\n> > never treat a fractional value like an integer. For example when we\n> compare\n> > two float value f1 and f2, we never check if (f1==f2) but check if\n> > abs(f1-f2) less than an acceptable range.\n> >\n> > From: Steven Ring [mailto:notifications@github.com]\n> > Sent: Thursday, 9 July 2015 12:14 PM\n> > To: GeoscienceAustralia/agdc\n> > Subject: [agdc] Durable/invariant acquisition identifier (#69)\n> >\n> >\n> > Since its inception, WOfS has encountered problems with the lack of a\n> > stable identifier for \"tiles/acquisitions\". WOfS currently uses the AGDC\n> > convention of \"start timestamp\" which is available via the API and also\n> > encoded into the dataset filename.\n> >\n> > One problem has been that there is fuzziness associated with this value.\n> > In the past, should a tile/acquisition be re-ingested the \"start\n> timestamp\"\n> > is not guaranteed to retain the same value. Any application running\n> > incremental updates may incorrectly interpret slightly changed start\n> times\n> > as new tile.\n> >\n> > The micorsecond resolution of these timestamps also causes problems with\n> > string representation of the timestamp changing depending on the value\n> of\n> > (or absence of) the microsecond component. A general solution to this\n> > problem requires some REGEX gymnastics (e.g.\n> >\n> https://github.com/smr547/ga-neo-nfrip/blob/luigi_api_refactor/wofs/timeparser.py)\n>\n> >\n> >\n> > Anyway, I'd like to suggest that we consider using a combination of\n> > satellite_id, \"orbit number\" and AGCD \"cell_ID\" as a durable acquisition\n> > identifier within any particular dataset type.\n> >\n> > This ID has the highly desirable property of not changing between\n> > ingestions. The orbit number is readily available from TLEs and could be\n> > computed during pre-ingestion processing (if it is not already).\n> >\n> > The identifier's value can also be easily predetermined. i.e. We can\n> > reliably state what acquisition IDs will be generated during the next\n> (or\n> > any) orbit of LS8. This has significant benefits when developing QA\n> > procedures (i.e. identifiying gaps in our data).\n> >\n> > Obviously it also eliminates the possibility of data duplication.\n> >\n> > As an example\n> >\n> > LS5_TM_NBAR_150_-034_1991-01-09T23-03-58.241019.tif\n> >\n> > might become\n> >\n> > LS5_NBAR_150_-034_23456.tif\n> >\n> > —\n> > Reply to this email directly or view it on GitHub<\n> > https://github.com/GeoscienceAustralia/agdc/issues/69>.\n> >\n> > Geoscience Australia Disclaimer: This e-mail (and files transmitted with\n> > it) is intended only for the person or entity to which it is addressed.\n> If\n> > you are not the intended recipient, then you have received this e-mail\n> by\n> > mistake and any use, dissemination, forwarding, printing or copying of\n> this\n> > e-mail and its file attachments is prohibited. The security of emails\n> > transmitted cannot be guaranteed; by forwarding or replying to this\n> email,\n> > you acknowledge and accept these risks.\n> >\n> >\n> -------------------------------------------------------------------------------------------------------------------------\n>\n> >\n> > —\n> > Reply to this email directly or view it on GitHub\n> > <\n> https://github.com/GeoscienceAustralia/agdc/issues/69#issuecomment-121096061>\n>\n> > .\n> >\n>\n>\n> —\n> Reply to this email directly or view it on GitHub<\n> https://github.com/GeoscienceAustralia/agdc/issues/69#issuecomment-121105733>.\n>\n>\n> Geoscience Australia Disclaimer: This e-mail (and files transmitted with\n> it) is intended only for the person or entity to which it is addressed. If\n> you are not the intended recipient, then you have received this e-mail by\n> mistake and any use, dissemination, forwarding, printing or copying of this\n> e-mail and its file attachments is prohibited. The security of emails\n> transmitted cannot be guaranteed; by forwarding or replying to this email,\n> you acknowledge and accept these risks.\n>\n> -------------------------------------------------------------------------------------------------------------------------\n>\n> —\n> Reply to this email directly or view it on GitHub\n> <https://github.com/GeoscienceAustralia/agdc/issues/69#issuecomment-121116569>\n> .\n>\n\n\n\n-- \n*Steven Ring*\nSoftware Engineer\nBSc MIT (ANU)\nm: 0417 495 268\nf: +61 2 6100 9273\ns: stevering\n"}}