{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/11740","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/11740/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/11740/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/11740/events","html_url":"https://github.com/elastic/elasticsearch/pull/11740","id":89190686,"number":11740,"title":"Completion Suggester V2","user":{"login":"areek","id":753679,"avatar_url":"https://avatars.githubusercontent.com/u/753679?v=3","gravatar_id":"","url":"https://api.github.com/users/areek","html_url":"https://github.com/areek","followers_url":"https://api.github.com/users/areek/followers","following_url":"https://api.github.com/users/areek/following{/other_user}","gists_url":"https://api.github.com/users/areek/gists{/gist_id}","starred_url":"https://api.github.com/users/areek/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/areek/subscriptions","organizations_url":"https://api.github.com/users/areek/orgs","repos_url":"https://api.github.com/users/areek/repos","events_url":"https://api.github.com/users/areek/events{/privacy}","received_events_url":"https://api.github.com/users/areek/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/:Suggesters","name":":Suggesters","color":"009800"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/v2.0.0","name":"v2.0.0","color":"dddddd"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":13,"created_at":"2015-06-18T03:52:00Z","updated_at":"2015-07-02T16:37:03Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/elastic/elasticsearch/pulls/11740","html_url":"https://github.com/elastic/elasticsearch/pull/11740","diff_url":"https://github.com/elastic/elasticsearch/pull/11740.diff","patch_url":"https://github.com/elastic/elasticsearch/pull/11740.patch"},"body":"# Overview\r\nThe motivation behind the new completion field is to support auto-complete/search-as-you-type functionality. This is a navigational feature to guide users to relevant results as they are typing, improving search precision. Ideally, auto-complete functionality should be as fast as a user types to provide instant feedback relevant to what a user has already typed in.\r\n\r\nThis PR introduces a new completion field (https://github.com/elastic/elasticsearch/issues/10746) using Lucene’s new suggest API. The new field has a superset of functionalities provided by the existing completion field (CompletionSuggester and ContextSuggester), namely near-real time search, document retrieval, support for multiple contexts and flexible scoring through index and query-time boosts.\r\n\r\n# Use Case\r\n\r\nGuide users to relevant documents by suggesting song titles as they type. In this example, the suggestions to serve the user will be indexed as a completion field named `title_suggest`.\r\n\r\nThe mapping snippet below adds a completion field named `title_suggest`\r\n```json\r\n...\r\n\"properties\": {\r\n \"title_suggest\": {\r\n  \"type\": \"completion\"\r\n }\r\n}\r\n```\r\nTo add title suggestions, index documents with `title_suggest` field.\r\n```json\r\n{\r\n \"title_suggest\" : \"Californication\",\r\n}\r\n```\r\nYou can also specify an array or an object to specify multiple values and/or configure index-time weights for completion values. **Index-time Weights** (maybe call this index-time boost?) are for sorting suggestions that match a prefix. For example, is \"Californication\" or \"Can't Stop\" more relevant for a prefix query of \"ca\"?\r\n\r\n### Querying\r\nTo query against a CompletionField, use `_suggest` API (see https://github.com/elastic/elasticsearch/issues/10746 for details). \r\n\r\nApart from simple **Prefix Query**, you can use **Fuzzy Prefix Query** to get typo-tolerant suggestions. This queries a set of similar prefixes (based on edit distance) and scores the suggestions based on how similar they are to what the user has typed in. For example, a prefix query of \"cale\" would suggest \"Californication\" and \"Can't Stop\" in order. \r\n\r\nThere is support for expressing a query prefix as a **Regular Expression**. For example, `ca[l-n]` will suggest titles starting with \"cal\", \"cam\" or \"can\".\r\n\r\n### Scoring and Filtering\r\nIt is often desirable to serve suggestions filtered and/or boosted by some criteria. For example, you want to suggest song titles filtered by certain artists or you want to boost song titles based on their genre. \r\n\r\nTo achieve suggestion filtering and/or boosting, you can add **contexts** while configuring a completion field. You can define multiple contexts for a completion field (NOTE: adding contexts increases the index size). Every context has a unique name and a type. Currently, there are two types `category` and `geo`.\r\n\r\nTo use genre as a context to song suggestion, a completion field is configured with a 'genre_context' context which indexes values from field 'genre' as shown below\r\n```json\r\n...\r\n\"properties\": {\r\n \"title_suggest\": {\r\n  \"type\": \"completion\",\r\n  \"contexts\" : [\r\n   { \"name\": \"genre_context\", \"type\" : \"category\", \"path\": \"genre\" }\r\n  ]\r\n }\r\n}\r\n```\r\nSuggestions with contexts can be added by indexing a document as follows:\r\n```json\r\n{\r\n \"title_suggest\" : \"Californication\",\r\n \"genre\" : [\"funk rock\", \"alterative_rock\", \"funk metal\"],\r\n}\r\n```\r\nThe default behaviour for context-enabled fields is to suggest from *all* contexts. You can filter on defined context(s) at query time by adding query contexts. For example, to restrict song suggestions to only 'funk rock' and 'indie' genres.\r\n```json\r\n\"suggest-namespace\" : {\r\n ...\r\n \"completion\" : {\r\n  \"field\" : \"title_suggest\",\r\n  \"contexts\": {           \r\n   \"genre_context\": [\"funk rock\", \"indie\"]\r\n  }\r\n }\r\n}\r\n```\r\nYou can boost suggestions for some genres more than others. For example, restrict song suggestions to all genres that start with \"funk\" but boost song suggestions with \"funk rock\" genre.\r\n```json\r\n...\r\n\"genre_context\": [ \r\n { \"value\": \"funk\", \"exact\": false }, \r\n { \"value\": \"funk rock\", \"boost\": 4 } \r\n]\r\n```\r\nConfiguring a completion field with multiple contexts, allows filtering/boosting suggestions by multiple criteria. For example, adding artist and genre contexts as follows:\r\n```json\r\n...\r\n\"title_suggest\": {\r\n \"type\": \"completion\",\r\n \"contexts\" : [\r\n  { \"name\": \"genre_context\", \"type\" : \"category\", \"path\": \"genre\" },\r\n  { \"name\": \"artist_context\", \"type\" : \"category\", \"path\": \"artist\" }\r\n ]\r\n}\r\n```\r\nYou can restrict song suggestions to \"funk rock\" and \"alternative rock\" genre and boost songs by \"rhcp\" by adding the following query contexts.\r\n```json\r\n\"contexts\": {           \r\n \"genre_context\": [ \"funk rock\", \"alternative rock\" ],\r\n \"artist_context\" : { \"value\" : \"rhcp\", \"boost\": 4 }\r\n}\r\n```\r\nThe query response for any context-enabled field, will include all the associated context values for that entry, along with the suggestion and weight.\r\n\r\nNOTE: Adding more contexts to a completion field will result in search performance degradation (when using match all context) and increase the index size.\r\n\r\n### PR status\r\n - The PR maps the old completion field to `completion_old` type, this is just a hack to make the tests happy. **Feedback needed**: This breaks backward compatibility for the old completion field, thoughts around what to do with the old completion field is much appreciated\r\n - This PR excludes the Lucene related changes ([LUCENE-6459](https://issues.apache.org/jira/browse/LUCENE-6459) and smaller changes) to keep the PR es specific.\r\n -  The geo context mapping is still work-in-progress, now it is possible to support geo filters and queries on geo-contexts\r\n - Retrieving documents with suggestions are not implemented in this PR, and will be done in a subsequent PR\r\n - Todo: API documentation "},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/comments/118088298","html_url":"https://github.com/elastic/elasticsearch/pull/11740#issuecomment-118088298","issue_url":"https://api.github.com/repos/elastic/elasticsearch/issues/11740","id":118088298,"user":{"login":"areek","id":753679,"avatar_url":"https://avatars.githubusercontent.com/u/753679?v=3","gravatar_id":"","url":"https://api.github.com/users/areek","html_url":"https://github.com/areek","followers_url":"https://api.github.com/users/areek/followers","following_url":"https://api.github.com/users/areek/following{/other_user}","gists_url":"https://api.github.com/users/areek/gists{/gist_id}","starred_url":"https://api.github.com/users/areek/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/areek/subscriptions","organizations_url":"https://api.github.com/users/areek/orgs","repos_url":"https://api.github.com/users/areek/repos","events_url":"https://api.github.com/users/areek/events{/privacy}","received_events_url":"https://api.github.com/users/areek/received_events","type":"User","site_admin":false},"created_at":"2015-07-02T16:37:03Z","updated_at":"2015-07-02T16:37:03Z","body":"> Is it true that the new suggest API returns the complete document?\r\n\r\n@lenniboy Yes, so instead of storing a value as a payload, you can store it as a field in the document. You will get back the document with the suggestions. "}}