{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/10823","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/10823/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/10823/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/10823/events","html_url":"https://github.com/elastic/elasticsearch/issues/10823","id":71244185,"number":10823,"title":"Vectorize API","user":{"login":"alexksikes","id":43475,"avatar_url":"https://avatars.githubusercontent.com/u/43475?v=3","gravatar_id":"","url":"https://api.github.com/users/alexksikes","html_url":"https://github.com/alexksikes","followers_url":"https://api.github.com/users/alexksikes/followers","following_url":"https://api.github.com/users/alexksikes/following{/other_user}","gists_url":"https://api.github.com/users/alexksikes/gists{/gist_id}","starred_url":"https://api.github.com/users/alexksikes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alexksikes/subscriptions","organizations_url":"https://api.github.com/users/alexksikes/orgs","repos_url":"https://api.github.com/users/alexksikes/repos","events_url":"https://api.github.com/users/alexksikes/events{/privacy}","received_events_url":"https://api.github.com/users/alexksikes/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/:Term%20Vectors","name":":Term Vectors","color":"009800"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/feature","name":"feature","color":"328a58"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/v2.0.0","name":"v2.0.0","color":"dddddd"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-04-27T10:26:37Z","updated_at":"2015-05-04T15:57:41Z","closed_at":null,"body":"The vectorize API operates on top of the Term Vectors API and on `_source` in order to return a document-term matrix according to some user given specification. In a document-term (or term-document matrix), the rows correspond to documents in an index and the columns correspond to terms, or more precisely to some numerical value associated with each term such as tf or tf-idf.\r\n\r\n## Defining the Doc-Term Matrix\r\n\r\nFirst a vectorizer has to be registered. A vectorizer defines the shape and entries of the desired outputted matrix.\r\n\r\n```\r\nPUT /index/.vectorizer/my_vectorizer\r\n{\r\n    \"vectorizer\": [\r\n        {\r\n            \"field\": \"text\",\r\n            \"span\": [ ... list of terms ...],\r\n            \"value\": \"term_freq\"\r\n        },\r\n        {\r\n            \"field\": \"field_numeric_1\",\r\n            \"span\": 1\r\n        },\r\n        {\r\n            \"field\" : \"field_numeric_2\",\r\n            \"span\": 1,\r\n            \"script\": \"if _value > 0.5 then 1 else 0\"\r\n        },\r\n        {\r\n            \"field\": \"field_numeric_3\",\r\n            \"span\": 5\r\n        },\r\n        {\r\n            \"field\": \"label\",\r\n            \"span\": 1,\r\n            \"script\": \"if _value == \"yes\" then 1 else 0\"\r\n        }\r\n    ]\r\n}\r\n```\r\n\r\nIn this example, the first entry defines a first list of columns whose values are the term frequencies of the terms given in `span` in this order. The next column is simply defined as the value of the numerical field named `field_numeric_1`. For the next column, we binarize the numerical value at the field using a script. Next we assign 5 columns for the first 5 values in the multi-valued numerical field `field_numeric_3`. Last but not least, the last column is binarized as well and could be used as the target label for learning.\r\n\r\nMore precisely:\r\n\r\n* `field`: Name of the field in the index to which a mapping from field values to real values should be applied.\r\n\r\n* `span`: For numerical fields, specifies a number of columns to be assigned in the resulting vector. For string fields, specifies a list of terms, each term occupying a column in the vector if present in the document.\r\n\r\n* `value`: For numerical fields, defaults to the value at the field. For string fields, defaults to term frequency. For the later case other values are possible such as document frequency, or payload.\r\n\r\n* `script`: Allows for any field value transformation such as thresholding, categorizing, etc ...\r\n\r\n## Retrieving Vectors\r\n\r\nWe can now use the vectorize API:\r\n\r\n```\r\nGET /index/type/id/_vectorize&vectorizer=my_vectorizer\r\n\r\nResponse:\r\n\r\n{\r\n    \"shape\": [1, 9],\r\n    \"vector\": [\r\n        {\"3\": 5, \"5\": 2, \"6\": 0.55, 7\": 1, \"8\": 0}\r\n    ]\r\n}\r\n```\r\n\r\nThe matrix returned is in a sparse format, together with its dimension. This vector can then be loaded in memory in your favorite statistical environments, or fed to a machine learning package. The parameters are exactly the same as the term vectors API with the additional `vectorizer` parameter. A vectorizer could also be provided inline, meaning as part of the request.\r\n\r\n```\r\nGET /index/type/id/_vectorize\r\n{\r\n    \"vectorizer\": [\r\n        {\r\n            \"field\": \"text\",\r\n            \"span\": [ ... list of terms ...],\r\n            \"value\": \"term_freq\"\r\n        },\r\n        {\r\n            \"field\": \"label\",\r\n            \"span\": 1,\r\n            \"script\": \"if _value == \"yes\" then 1 else 0\"\r\n        }\r\n    ]\r\n}\r\n```\r\n\r\nIn order to retrieve more than one vector, use the `_mvectorize` API:\r\n\r\n```\r\nGET /index/type/_mvectorize\r\n{\r\n    \"docs\": [\r\n    {\r\n        \"_index\": \"index\",\r\n        \"_type\": \"type\",\r\n        \"_id\": \"id\",\r\n        \"vectorizer\": my_vectorizer\r\n    },\r\n    {\r\n        \"_index\": \"index\",\r\n        \"_type\": \"type\",\r\n        \"_id\": \"id2\",\r\n        \"vectorizer\": my_vectorizer\r\n    }\r\n   ]\r\n}\r\n\r\nResponse:\r\n\r\n{\r\n    \"shape\": [2, 9],\r\n    \"vector\": [\r\n        {\"3\": 5, \"5\": 2, \"6\": 0.55, 7\": 1, \"8\": 0},\r\n        {\"0\": 2, \"3\": 1, \"6\": 0.21, 7\": 1, \"8\": 1}\r\n    ]\r\n}\r\n```\r\n\r\nHere different vectorizers could have been chosen. In this case, the vectors are stacked up, columns over columns, with the largest vector defining the dimension of the matrix.\r\n\r\n## Generating Datasets\r\n\r\nFinally, we can obtain a dataset by using a new search type called `vectorize` together with a scroll:\r\n\r\n```\r\nGET /index/type/_search?search_type=vectorize&scroll=1m\r\n{\r\n    \"query\": { \"match_all\": {}},\r\n    \"size\":  1000,\r\n\r\n    \"from\": 0,\r\n    \"step\": 1,\r\n    \"sample\": \"10%\",\r\n\r\n    \"vectorizer\": \"my_vectorizer\",\r\n    \"slice\": start:stop:step\r\n}\r\n\r\nAfter submitting the scroll request, we get the response:\r\n\r\n{\r\n    \"shape\": [1000, 9],\r\n    \"vector: [\r\n        {\"3\": 5, \"5\": 2, \"6\": 0.55, 7\": 1, \"8\": 0},\r\n        {\"0\": 2, \"3\": 1, \"6\": 0.21, 7\": 1, \"8\": 1},\r\n        {\"3\": 5, \"3\": 5, \"6\": 0.45, 7\": 0, \"8\": 0},\r\n        {\"3\": 5, \"5\": 3, \"6\": 0.56, 7\": 0, \"8\": 1},\r\n        ...\r\n    ]\r\n}\r\n```\r\n\r\nHere `slice` allows us to only select some columns in this matrix. The `from`, `step` or `sample` retain documents starting after having processed a given number of documents and at every chosen step, or with x% chances. These options are useful in order to generate datasets such as a training and test set.\r\n"},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/comments/98765043","html_url":"https://github.com/elastic/elasticsearch/issues/10823#issuecomment-98765043","issue_url":"https://api.github.com/repos/elastic/elasticsearch/issues/10823","id":98765043,"user":{"login":"polyfractal","id":1224228,"avatar_url":"https://avatars.githubusercontent.com/u/1224228?v=3","gravatar_id":"","url":"https://api.github.com/users/polyfractal","html_url":"https://github.com/polyfractal","followers_url":"https://api.github.com/users/polyfractal/followers","following_url":"https://api.github.com/users/polyfractal/following{/other_user}","gists_url":"https://api.github.com/users/polyfractal/gists{/gist_id}","starred_url":"https://api.github.com/users/polyfractal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/polyfractal/subscriptions","organizations_url":"https://api.github.com/users/polyfractal/orgs","repos_url":"https://api.github.com/users/polyfractal/repos","events_url":"https://api.github.com/users/polyfractal/events{/privacy}","received_events_url":"https://api.github.com/users/polyfractal/received_events","type":"User","site_admin":false},"created_at":"2015-05-04T15:57:41Z","updated_at":"2015-05-04T15:57:41Z","body":"I like it!  I can definitely see this being used with plug-n-play tools like WEKA or scikit-learn.  Some questions / thoughts:\r\n\r\n- Does everything just throw an exception if term vectors are not enabled?  Is this a hard requirement or just a way to improve speed?\r\n\r\n- Why is `_source` required?\r\n\r\n- How are missing values handled?  Is `null` or `NaN` encoded in the matrix if the document is missing a particular field?  I know some ML packages are particular about how they handle missing fields, perhaps there could be an optional `missing_value` parameter that inserts a default if the field is missing?  Same for spans + numerics.  E.g. if you request `span: 5`, but a field only has 3 values ... the extra two will likely need to be filled with `NaN` or `null` or something?\r\n\r\n- With `_mvectorize`, is there an option to \"unstack\" the matrices?  E.g. you may just want to request multiple matrices in one request, but not combine them.\r\n\r\n- The slice syntax seems clunky, and only sorta flexible.  E.g what if you want uneven combinations such as `0-5, 7, 9, 10-12` ?  Perhaps some kind of array syntax:  `[0..5, 7, 9, 10..12]` ?\r\n\r\n- Perhaps show the resulting \"shape\" after `slice`ing a matrix to make it clear it is different from the underlying vectorizer?\r\n\r\n- Would it be useful to also add the vectorizer name to the output?  Not sure if that is something people would want.\r\n\r\n- Not necessarily a problem, just something to think about, but `step` and `sample` could get strange if a routing scheme is used.  E.g. `sample: 10%` may exhaust some shards faster than others, because certain shards are \"hot\" due to routing.  This could unbalance the sampling and skew it towards the particular routed values.  Might just be a documentation issue so that users are aware sampling doesn't necessarily mean truly random sample depending on circumstances.\r\n\r\n- I wonder if there should be a \"csv\" or \"tab\" format option, so you can get a flat matrix rather than JSON?  Might make it easier to interface with other packages?  Not sure...it's very un-ES to emit non-JSON :)\r\n"}}