{"push_id":753153184,"size":26,"distinct_size":0,"ref":"refs/heads/master","head":"bbced7ba0641316259f345e80fe93a9797926749","before":"9f5674f0a5c6bc0ba09e838ad18c08e726c63463","commits":[{"sha":"ba17d98ce9a2470a67227b19585b813fe7591da9","author":{"email":"75ef9faee755c70589550b513ad881e5a603182c@whiteink.com","name":"Nick Stenning"},"message":"Add URI normalisation behind a feature flag\n\nThis is the first step in a major cleanup of how we internally normalise\nURIs referenced by annotations. The URIs in question divide into two\ngroups:\n\n1. URIs stored on annotations which refer to the annotated resource.\n\n2. URIs discovered in the annotated resource or elsewhere, which can be\n   used as hints to tell us which other URIs refer to the same\n   underlying document (e.g. to alternate formats of the annotated\n   resource).\n\nWe use both sets of these URIs when a user visits a page and activates\nHypothesis:\n\n- the client sends a search request, which includes the URI of the page\n  on which it was activated.\n- the server must find all annotations made on that page, but *also* all\n  annotations made on alternate versions of the page, and return these\n  to the client.\n\nGetting this right is both important and hard.\n\nIt's important because poor implementation will result in a poor\nexperience of using Hypothesis. Users don't (usually) want to see\nannotations from documents other than the one they're viewing, and users\n(usually) don't want any of the annotations on the page they're viewing\nto be missing.\n\nIt's hard because in simple terms, a user expects that when they visit\nthe \"same page,\" they will see the same annotations. But \"same page\" is\nnot a well-defined concept on the web.\n\nIndeed, one of the so-called \"axioms of web architecture\" is the\n[opacity of URIs](http://www.w3.org/DesignIssues/Axioms.html#opaque),\nwhich means that a priori, there are only a limited number of\nnormalisations of a URI which are guaranteed to preserve reference.\n\nAt the time of writing, the standard which defines permitted\nnormalisations is RFC 3987[1], AKA the \"Internationalized Resource\nIdentifiers (IRIs)\" RFC.\n\n[1]: https://tools.ietf.org/html/rfc3987#section-5\n\nDealing correctly with internationalised identifiers is something we\nwill probably need to do in due course. For now, dealing correctly with\nURIs will be a substantial improvement on the status quo. IRIs won't be\nreferred to again. The relevant standard which defines permitted\nnormalisations of URIs is RFC3986[2].\n\n[2]: https://tools.ietf.org/html/rfc3986#section-6\n\nIt's worth reading the above if you're trying to understand the starting\npoint for this code, but it's also worth understanding that in reality,\nwe have to be a lot more \"heuristic\" than these documents allow if we\nwant to ensure a good user experience. The \"opacity of URIs\", sadly,\nmust go out the window (for URLs, at least) if we want to provide a good\nexperience with Hypothesis on the real web.\n\nFor example, in practice, it may be the case that lexically sorting\nURL query parameters is almost always a permissible (and thus\nexperience-improving) normalisation. But such an operation is clearly\nnot permitted by the URI normalisation specifications.\n\nSo, overall, here's the approach taken by this commit:\n\n- We introduce the concept of a URI normalisation, which maps an input\n  URI idempotently to a normalised version of that URI.\n\n- When storing new annotations, we store normalised versions of the\n  'source' property of targets (that identify the annotated document).\n\n- When searching via the API:\n\n  + the value associated with the \"uri\" keyword, if passed, is used to\n    look up alternate URIs.[^1]\n\n  + Together, the query URI and alternate URIs returned from the\n    previous step form a set for searching.\n\n  + Each of the URIs in the search set are normalised.\n\n  + The annotation database is queried to find those annotations where\n    the normalised target source field is an exact match for any of the\n    resulting set of normalised URIs.\n\n- We continue to store, unmodified, the URIs as passed by the client, so\n  that we can, if necessary, alter the URI normalisation operation in\n  the future.\n\nA later commit will introduce a mechanism to regenerate the normalised\nfields in the search index to facilitate both release of this feature in\na deployment with legacy annotations lacking such fields, but also the\nability to later update the URI normalisation procedure.\n\nThe addition of normalised fields to annotations is performed\nunconditionally, but the alternate search procedure is guarded by a\nfeature flag.\n\n[^1]: Ideally we would look up such \"URI equivalences\" using a\nnormalised URI, but this is complicated by the implementation of\n`annotator-store` at the moment, so we don't yet do so.","distinct":false,"url":"https://api.github.com/repos/hypothesis/h/commits/ba17d98ce9a2470a67227b19585b813fe7591da9"},{"sha":"018179d0563749b431da8efceef85eede6a70b75","author":{"email":"75ef9faee755c70589550b513ad881e5a603182c@whiteink.com","name":"Nick Stenning"},"message":"Add normalised URIs on insert/update\n\nIn order to search against the \"source_normalised\" field, we need to add\nthis field when we index the annotation. This commit adds support for\ncomputing \"source_normalised\" fields (for targets) when annotations are\ninserted into the ElasticSearch index.","distinct":false,"url":"https://api.github.com/repos/hypothesis/h/commits/018179d0563749b431da8efceef85eede6a70b75"},{"sha":"429ef7790bcecf02db77a4e4116fc4c073bcec26","author":{"email":"46f1a0bd5592a2f9244ca321b129902a06b53e03@seanh.cc","name":"Sean Hammond"},"message":"Implement \"Share a group\" functionality\n\nGroup pages show a \"Login to join group\" link (opens in a new tab) if\nyou're not logged in.\nIf you're logged in they show a \"Click to join group\" button, posts to\nnew join() callable that adds the user to the group.\nIf you're already a member they show a \"Share this link to invite people\" link.\n\nThe groups dropdown list has new share icon links to the group's page,\nfor each group.","distinct":false,"url":"https://api.github.com/repos/hypothesis/h/commits/429ef7790bcecf02db77a4e4116fc4c073bcec26"},{"sha":"5d2d8142cc864288c3d93a274700cd674f16ec46","author":{"email":"75ef9faee755c70589550b513ad881e5a603182c@whiteink.com","name":"Nick Stenning"},"message":"Add `hypothesis annotool` for modifying the annotation database\n\nThis commit adds a command to the `hypothesis` CLI, which enables\nrunning of functions which modify annotations, so that we can run data\nmigrations against the annotation database.\n\nThe only supported function at the moment is 'normalise-uris', which\nensures that all annotations in the database have normalised copies of\ntheir \"target.source\" attributes.","distinct":false,"url":"https://api.github.com/repos/hypothesis/h/commits/5d2d8142cc864288c3d93a274700cd674f16ec46"},{"sha":"5261ed43e17f544ef9a365701fe9f1801e21e649","author":{"email":"75ef9faee755c70589550b513ad881e5a603182c@whiteink.com","name":"Nick Stenning"},"message":"Filter 'source_normalised' keys from annotations before display","distinct":false,"url":"https://api.github.com/repos/hypothesis/h/commits/5261ed43e17f544ef9a365701fe9f1801e21e649"},{"sha":"172fe7383214d4d801c25829b29e72eb610d7290","author":{"email":"75ef9faee755c70589550b513ad881e5a603182c@whiteink.com","name":"Nick Stenning"},"message":"Move normalisation/cleanup into `h.api.search.transform`\n\nThis moves all responsibility for adding and removing the\n\"source_normalised\" fields from annotation data into a \"transform\"\nmodule in `h.api.search`.\n\nThis keeps `h.api.uri` simple.","distinct":false,"url":"https://api.github.com/repos/hypothesis/h/commits/172fe7383214d4d801c25829b29e72eb610d7290"},{"sha":"38bc599488b9100adc752fa952ead73df21bfca0","author":{"email":"75ef9faee755c70589550b513ad881e5a603182c@whiteink.com","name":"Nick Stenning"},"message":"URI normalisation: strip trailing slashes\n\nIt would be unusual for the slashless version of a URL and the slashed\nversion, e.g.\n\n    http://example.com/foo/\n    http://example.com/foo\n\nto refer to different documents. Thus, when normalising URLs, we remove\nany trailing slashes from the pathinfo.","distinct":false,"url":"https://api.github.com/repos/hypothesis/h/commits/38bc599488b9100adc752fa952ead73df21bfca0"},{"sha":"934af1ac8ea6e4f1adff876d517616a7e0bcbfb4","author":{"email":"75ef9faee755c70589550b513ad881e5a603182c@whiteink.com","name":"Nick Stenning"},"message":"Refactor `uri.normalise` for clarity\n\nIt's slightly easier to read if we separate the normalisation functions\nout into one per URL component, rather than asking each piece to do its\nown URI parsing/splitting/recombining.\n\nEach \"_normalise_*\" function gets passed the whole URI object, because\nthe correct normalised form of each component may depend on other parts\nof the URL. For example, it's only correct to remove default port\ndeclarations (such as \":80\" or \":443\") if the corresponding scheme is\nthe right one (i.e. \"http\", \"https\").","distinct":false,"url":"https://api.github.com/repos/hypothesis/h/commits/934af1ac8ea6e4f1adff876d517616a7e0bcbfb4"},{"sha":"a1189bfe20ec52c81e509aea52a97eba74366053","author":{"email":"75ef9faee755c70589550b513ad881e5a603182c@whiteink.com","name":"Nick Stenning"},"message":"URI normalisation: add query string normalisation\n\nDo some relatively uncontroversial normalisation of query strings. We\nsort the query string by key, leaving multiple query params with the\nsame key in the same relative ordering.","distinct":false,"url":"https://api.github.com/repos/hypothesis/h/commits/a1189bfe20ec52c81e509aea52a97eba74366053"},{"sha":"ec0d9832392db3c1872fa4ffdd7ea48dba9da0a6","author":{"email":"75ef9faee755c70589550b513ad881e5a603182c@whiteink.com","name":"Nick Stenning"},"message":"URI normalisation: remove Google Analytics query params\n\nGoogle Analytics allows users to provide query parameters which feed\nadditional data into the JavaScript you run on your page. These params\ncan be reasonably assumed to be irrelevant for determining document\nidentity, so we can strip them from the normalised URL.","distinct":false,"url":"https://api.github.com/repos/hypothesis/h/commits/ec0d9832392db3c1872fa4ffdd7ea48dba9da0a6"},{"sha":"b49e1aa63ed72eb9edbd8f4a8321641811e98b84","author":{"email":"75ef9faee755c70589550b513ad881e5a603182c@whiteink.com","name":"Nick Stenning"},"message":"Input URIs can be unicode, outputs are strings\n\nAnd we expect unicode in query params to be normalised to urlencoded\nform.","distinct":false,"url":"https://api.github.com/repos/hypothesis/h/commits/b49e1aa63ed72eb9edbd8f4a8321641811e98b84"},{"sha":"57feff98a8609954a80fed9ac540865af32f30e7","author":{"email":"75ef9faee755c70589550b513ad881e5a603182c@whiteink.com","name":"Nick Stenning"},"message":"URI normalisation: normalise percent-encoding\n\nNormalise percent-encoding of paths and query strings as dictated by\nRFC3986.\n\nPaths and query strings can both contain [percent-encoded][1] character\ndata. Percent-encoding is a way to include  in URLs (and URIs more\ngenerally) certain characters which may have semantic meaning in a URL\n(such as \"/\" in a URL path), or which may simply not be permitted (such\nas literal UTF-8 characters).\n\nSometimes these percent-encodings must be preserved -- percent-encoded\nUTF-8 unicode, for example. But sometimes URLs that differ only in\npercent-encoding can (according to RFC3986) be considered the same. To\npick one example, \"-\" (hyphen) is an \"unreserved character\", so:\n\n    http://example.com/foo-bar, AND\n    http://example.com/foo%2Dbar\n\ncan be considered the same. In addition, case in percent encodings does\nnot carry information, so\n\n    http://example.com/foo%2dbar, AND\n    http://example.com/foo%2Dbar\n\nare also the same URL.\n\nThis commit adds support for normalising percent-encoding of URL paths\nand query strings.\n\n[1]: https://en.wikipedia.org/wiki/Percent-encoding","distinct":false,"url":"https://api.github.com/repos/hypothesis/h/commits/57feff98a8609954a80fed9ac540865af32f30e7"},{"sha":"5cc0e962793b4b47b0970c067d19c157b9d6fb8d","author":{"email":"75ef9faee755c70589550b513ad881e5a603182c@whiteink.com","name":"Nick Stenning"},"message":"URI normalisation: strip multiple trailing slashes\n\nuri.normalise should be idempotent.","distinct":false,"url":"https://api.github.com/repos/hypothesis/h/commits/5cc0e962793b4b47b0970c067d19c157b9d6fb8d"},{"sha":"6a2f8036288dfddcd053c7073db7c44adabd8378","author":{"email":"75ef9faee755c70589550b513ad881e5a603182c@whiteink.com","name":"Nick Stenning"},"message":"Refactor query.build for clarity\n\nRather than creating half a query body and mutating it, we create the\ncomponents of it one at a time and assemble it all at the end. This\nreduces the amount of\n\n    query[\"query\"][\"filtered\"][\"query\"][\"query\"][\"term\"]...\n\ncraziness in the code.","distinct":false,"url":"https://api.github.com/repos/hypothesis/h/commits/6a2f8036288dfddcd053c7073db7c44adabd8378"},{"sha":"acab2e1d0f4dcf361ca8c31640544e232d905fff","author":{"email":"75ef9faee755c70589550b513ad881e5a603182c@whiteink.com","name":"Nick Stenning"},"message":"Merge pull request #2412 from hypothesis/share-a-group\n\nShare a group","distinct":false,"url":"https://api.github.com/repos/hypothesis/h/commits/acab2e1d0f4dcf361ca8c31640544e232d905fff"},{"sha":"0ee4749f8b72c029da42dfb7a82ed9e65c1bbf62","author":{"email":"28c4723fc4f18c9264197ab693cc29746fd1c3a5@hypothes.is","name":"Randall Leeds"},"message":"Americanise spellings :)","distinct":false,"url":"https://api.github.com/repos/hypothesis/h/commits/0ee4749f8b72c029da42dfb7a82ed9e65c1bbf62"},{"sha":"cc4ca35648dd57393e276e61685204d67ac0381d","author":{"email":"b303809df130880a9c942350c555edddd5bdabb3@bleeds.info","name":"Randall Leeds"},"message":"Merge pull request #2413 from hypothesis/uri-normalisation\n\nURI normalisation","distinct":false,"url":"https://api.github.com/repos/hypothesis/h/commits/cc4ca35648dd57393e276e61685204d67ac0381d"},{"sha":"a436310c8692b02423904a45c33b37c347ecd735","author":{"email":"28c4723fc4f18c9264197ab693cc29746fd1c3a5@hypothes.is","name":"Randall Leeds"},"message":"Americanise some spellings","distinct":false,"url":"https://api.github.com/repos/hypothesis/h/commits/a436310c8692b02423904a45c33b37c347ecd735"},{"sha":"3b5733d6ee8aa1232db97b85cb632c1c898cd76a","author":{"email":"28c4723fc4f18c9264197ab693cc29746fd1c3a5@hypothes.is","name":"Randall Leeds"},"message":"Whitespace","distinct":false,"url":"https://api.github.com/repos/hypothesis/h/commits/3b5733d6ee8aa1232db97b85cb632c1c898cd76a"},{"sha":"0629cd886f03111ee2d6f281195011e8d136d13e","author":{"email":"28c4723fc4f18c9264197ab693cc29746fd1c3a5@hypothes.is","name":"Randall Leeds"},"message":"Remove requestAnimationFrame polyfill\n\nWe work on IE10+ but not IE9. Other major browsers all support it\nfor a while.","distinct":false,"url":"https://api.github.com/repos/hypothesis/h/commits/0629cd886f03111ee2d6f281195011e8d136d13e"}]}