{"action":"closed","issue":{"url":"https://api.github.com/repos/deconst/deconst-docs/issues/187","labels_url":"https://api.github.com/repos/deconst/deconst-docs/issues/187/labels{/name}","comments_url":"https://api.github.com/repos/deconst/deconst-docs/issues/187/comments","events_url":"https://api.github.com/repos/deconst/deconst-docs/issues/187/events","html_url":"https://github.com/deconst/deconst-docs/issues/187","id":114887163,"number":187,"title":"Full-text search","user":{"login":"smashwilson","id":17565,"avatar_url":"https://avatars.githubusercontent.com/u/17565?v=3","gravatar_id":"","url":"https://api.github.com/users/smashwilson","html_url":"https://github.com/smashwilson","followers_url":"https://api.github.com/users/smashwilson/followers","following_url":"https://api.github.com/users/smashwilson/following{/other_user}","gists_url":"https://api.github.com/users/smashwilson/gists{/gist_id}","starred_url":"https://api.github.com/users/smashwilson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smashwilson/subscriptions","organizations_url":"https://api.github.com/users/smashwilson/orgs","repos_url":"https://api.github.com/users/smashwilson/repos","events_url":"https://api.github.com/users/smashwilson/events{/privacy}","received_events_url":"https://api.github.com/users/smashwilson/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/deconst/deconst-docs/labels/discussion","name":"discussion","color":"009800"},{"url":"https://api.github.com/repos/deconst/deconst-docs/labels/infrastructure","name":"infrastructure","color":"eb6420"}],"state":"closed","locked":false,"assignee":{"login":"smashwilson","id":17565,"avatar_url":"https://avatars.githubusercontent.com/u/17565?v=3","gravatar_id":"","url":"https://api.github.com/users/smashwilson","html_url":"https://github.com/smashwilson","followers_url":"https://api.github.com/users/smashwilson/followers","following_url":"https://api.github.com/users/smashwilson/following{/other_user}","gists_url":"https://api.github.com/users/smashwilson/gists{/gist_id}","starred_url":"https://api.github.com/users/smashwilson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smashwilson/subscriptions","organizations_url":"https://api.github.com/users/smashwilson/orgs","repos_url":"https://api.github.com/users/smashwilson/repos","events_url":"https://api.github.com/users/smashwilson/events{/privacy}","received_events_url":"https://api.github.com/users/smashwilson/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/deconst/deconst-docs/milestones/5","html_url":"https://github.com/deconst/deconst-docs/milestones/[0.4]%20Search","labels_url":"https://api.github.com/repos/deconst/deconst-docs/milestones/5/labels","id":1392050,"number":5,"title":"[0.4] Search","description":"This milestone has three objectives:\r\n\r\n 1. Support full-text search.\r\n 2. Improve the content authoring and site maintainance experience.\r\n 3. Refactoring and stability.","creator":{"login":"smashwilson","id":17565,"avatar_url":"https://avatars.githubusercontent.com/u/17565?v=3","gravatar_id":"","url":"https://api.github.com/users/smashwilson","html_url":"https://github.com/smashwilson","followers_url":"https://api.github.com/users/smashwilson/followers","following_url":"https://api.github.com/users/smashwilson/following{/other_user}","gists_url":"https://api.github.com/users/smashwilson/gists{/gist_id}","starred_url":"https://api.github.com/users/smashwilson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smashwilson/subscriptions","organizations_url":"https://api.github.com/users/smashwilson/orgs","repos_url":"https://api.github.com/users/smashwilson/repos","events_url":"https://api.github.com/users/smashwilson/events{/privacy}","received_events_url":"https://api.github.com/users/smashwilson/received_events","type":"User","site_admin":false},"open_issues":16,"closed_issues":4,"state":"open","created_at":"2015-11-03T19:29:08Z","updated_at":"2015-12-04T19:51:53Z","due_on":null,"closed_at":null},"comments":21,"created_at":"2015-11-03T19:08:10Z","updated_at":"2015-12-04T19:51:53Z","closed_at":"2015-12-04T19:51:53Z","body":"Enabling search across all Deconst-indexed content has been a major goal of this project since we started it back in March, but for some reason we never made a proper issue for it. We already have an Elasticsearch instance in place within our architecture for log aggregation, so it's a question of correctly storing content as it arrives, extending the content service API with endpoints to support search, then consuming them within the presenter to provide a place for sites to hook in search queries and results.\r\n\r\n**I. First, we'll need to index incoming metadata envelopes in Elasticsearch.** This could be as simple as jamming them as-is into Elasticsearch, but I'm guessing that we'd get better results if we mutate them somewhat for optimal indexing. I predict many trips to [the Elasticsearch docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html).\r\n\r\n * Do we need to scrub HTML markup from the body before indexing?\r\n * We should support some kind of well-defined \"meta\" tags for explicit keyword indexing. We could use these for SEO, then, too.\r\n * We'll need to be certain that we delete stale information from the search index when new content arrives with the same ID.\r\n\r\n**II. Then we need to design a content service API to submit searches and retrieve results.** We _could_ accept Lucene queries and pass them directly to Elasticsearch, but once again, I suspect that there's a better way.\r\n\r\n * Search results will be returned in terms of content IDs, because the content service has no knowledge of the control repository.\r\n * We'll likely want to include additional information about each match, like preview snippets with offset data to highlight matching text.\r\n * Ordering of search results is important. I'm guessing that Elasticsearch will give us the knobs to tune this to get it to the way we want, but we'll need to know what the way we want is, first.\r\n * We may want the ability to optionally constrain results to results from a specific domain.\r\n\r\n**III. Finally, we need to figure out how to render search results within a page.** This is a little more involved than it seems because the content service talks entirely in terms of content IDs, while to be useful we need search results as presented URLs. I can see two possible directions:\r\n\r\n 1. Flag certain pages as \"search result pages\" somehow (in the control repo, possibly?). Nab standard query flags from URLs on those pages (`?q=term&page=3&perPage=10`) and submit a query to the content service. Translate result objects from content ID to presented URL based on the current content mapping. Then, either: _(a)_ inject the result objects into the envelope's body HTML with a content filter, or _(b)_ make the result array available to the nunjucks templates as a variable.\r\n 2. Expose a _presenter_ API endpoint that fires queries to the content service, maps the content IDs, and returns results as a JSON object. Consume it entirely from client-side Javascript. This would make it somewhat more difficult to incorporate within each domain, but it would let us do things like implement a loading spinner while the search is being done.\r\n\r\nI don't really have any experience designing search systems, so any input from those who do is most welcome :bow:\r\n\r\n@steveortiz, @j12y: Please weigh in if the KC has any additional requirements for this.\r\n\r\n/cc @kenperkins @jnoller"}}