{"action":"created","issue":{"url":"https://api.github.com/repos/mementoweb/py-memento-client/issues/2","labels_url":"https://api.github.com/repos/mementoweb/py-memento-client/issues/2/labels{/name}","comments_url":"https://api.github.com/repos/mementoweb/py-memento-client/issues/2/comments","events_url":"https://api.github.com/repos/mementoweb/py-memento-client/issues/2/events","html_url":"https://github.com/mementoweb/py-memento-client/issues/2","id":107370191,"number":2,"title":"More efficient timegate query option","user":{"login":"ikreymer","id":1015759,"avatar_url":"https://avatars.githubusercontent.com/u/1015759?v=3","gravatar_id":"","url":"https://api.github.com/users/ikreymer","html_url":"https://github.com/ikreymer","followers_url":"https://api.github.com/users/ikreymer/followers","following_url":"https://api.github.com/users/ikreymer/following{/other_user}","gists_url":"https://api.github.com/users/ikreymer/gists{/gist_id}","starred_url":"https://api.github.com/users/ikreymer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ikreymer/subscriptions","organizations_url":"https://api.github.com/users/ikreymer/orgs","repos_url":"https://api.github.com/users/ikreymer/repos","events_url":"https://api.github.com/users/ikreymer/events{/privacy}","received_events_url":"https://api.github.com/users/ikreymer/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":16,"created_at":"2015-09-20T03:25:53Z","updated_at":"2015-09-28T23:29:20Z","closed_at":null,"body":"Great to see this library, I was just exploring using the `get_memento_info` for Netcapsule or Reconstruct and ran into the following issue.\r\n\r\nWhile the `get_memento_info` is a great general purpose function, I think there needs to be a more optimized option that performs just the basic \"Get memento from TimeGate at specified datetime\" and nothing else. Ideally, this means having a single HTTP request to the TimeGate to get the desired info.\r\n\r\nI propose adding an extra param, `include_uri_checks`, which will default to True, but when set to False, will disable any of the following additional checks.\r\n\r\nCurrently, the `get_memento_info` also includes the following:\r\n\r\n* A `get_original_uri` queries the original url to determine if its a URL-R or URl-M. This is unnecessary if the user knows that a URL-R is being passed in (and potentially slow). With `include_uri_checks` disabled, this test will be skipped.\r\n\r\n* Redirects are enabled on the TimeGate `HEAD` request, initiating potentially many redirects. This can be disabled to ensure just one request to the TimeGate. With `include_uri_checks` disabled,\r\nthe value of `Location` or `Content-Location` header will be used instead.\r\n\r\n* Redirects are followed with a head request to the URL-M to get the `Memento-Datetime` header. This also brings additional overhead. Instead, the datetime is usually already available Link header response from the TimeGate, getting the `datetime` from the `rel=memento` entry. (As a side note, I was surprised to find that this is not required, though luckily present in most implementations -- seems like the timegate should always return the datetime in the link header, rather than force user to make another request to the URL-M)\r\n\r\n* With this option, automatic redirecting, eg. `http://lanl.gov` -> `http://www.lanl.gov/worldview/` is disabled, as the users browser will follow the 302 explicitly and it will be reflected to the user.\r\n\r\nFor now, I've called this property `include_uri_checks` but can be changed to something else. If better to make a separate function, that can work as well.\r\n\r\nWith this option disabled, Reconstruct and Netcapsule can use start using this API, rather than relying on the existing Memento JSON api that's only on the aggregator.\r\n\r\nI've also included a simple test file with demonstrates these changes.\r\n\r\nLet me know if there are any questions/thoughts.\r\n"},"comment":{"url":"https://api.github.com/repos/mementoweb/py-memento-client/issues/comments/143901197","html_url":"https://github.com/mementoweb/py-memento-client/issues/2#issuecomment-143901197","issue_url":"https://api.github.com/repos/mementoweb/py-memento-client/issues/2","id":143901197,"user":{"login":"ikreymer","id":1015759,"avatar_url":"https://avatars.githubusercontent.com/u/1015759?v=3","gravatar_id":"","url":"https://api.github.com/users/ikreymer","html_url":"https://github.com/ikreymer","followers_url":"https://api.github.com/users/ikreymer/followers","following_url":"https://api.github.com/users/ikreymer/following{/other_user}","gists_url":"https://api.github.com/users/ikreymer/gists{/gist_id}","starred_url":"https://api.github.com/users/ikreymer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ikreymer/subscriptions","organizations_url":"https://api.github.com/users/ikreymer/orgs","repos_url":"https://api.github.com/users/ikreymer/repos","events_url":"https://api.github.com/users/ikreymer/events{/privacy}","received_events_url":"https://api.github.com/users/ikreymer/received_events","type":"User","site_admin":false},"created_at":"2015-09-28T23:29:20Z","updated_at":"2015-09-28T23:29:20Z","body":"> Several tests with http://www.cnn.com on Memento-compliant archives actually do return a memento relation, indicating that it is available for acquiring the Memento-Datetime. So, in those case, the Memento-Datetime is accessible from the TimeGate response. If the Accept-Datetime is not submitted as a header, then no memento relation is present, but there is last memento which can be used.\r\n\r\nAh right, in this special case last memento == closest, so it should be used instead.\r\n\r\n> Obviously, TimeGates for noncompliant archives, such as http://webcitation.org, will not have this relation, so there is no way to get this information.\r\n\r\nBy non-compliant I assume you mean those that don't include the optional relations but otherwise follow the protocol? ... if they don't comply with the protocol, then all bets are off, right? ;)\r\n\r\n> I looked at RFC-7089 and it does not require a memento (or last memento) relation in the Link header for Pattern 1.1 or 2.1, so sites may not consistently list this relation, requiring one to make a request of the URI-M for the Memento-Datetime. The only required relation is original, and it explicitly states that The response MUST NOT contain a \"Memento-Datetime\" header. The method get_memento_info holds close to this spec.\r\n\r\nYep. I alluded to this as well. I think this is an unfortunate oversight.\r\nI would say this is even backwards, seems that the `memento` or `closest memento` relation should be the required one, rather than the `original`. A user likely knows the original one as they're querying the TimeGate with it, and the whole point of talking to the TimeGate to get the URL-M and the its datetime.\r\nLuckily for now, most active memento deployments (those based on OpenWayback or pywb) *do* include the memento or last memento relation.\r\n\r\nYep, I think the below steps look good, just added extra conditions for Pattern 2.2, 2.3, etc...,\r\nwhich could have a Content-Location and a Memento-Datetime headers.\r\n\r\n\r\n1. perform the request on the URI-G with the Accept-Datetime\r\n2. get the URI-M from the Location header *or Content-Location header* in the response\r\n3. parse the link header using `parse_link_header` to get the Memento-Datetime from the memento relation or the memento last relation, depending on whether Accept-Datetime was submitted *or use actual Memento-Datetime header, if present*\r\n4. use `__prepare_memento_response`, passing it the URI-M, the Memento-Datetime, and the Link header to produce the JSON response\r\n\r\nDo we know of archives that support memento protocol but do not include any relations?\r\n\r\n\r\n"}}