{"action":"created","issue":{"url":"https://api.github.com/repos/basho/yokozuna/issues/403","labels_url":"https://api.github.com/repos/basho/yokozuna/issues/403/labels{/name}","comments_url":"https://api.github.com/repos/basho/yokozuna/issues/403/comments","events_url":"https://api.github.com/repos/basho/yokozuna/issues/403/events","html_url":"https://github.com/basho/yokozuna/issues/403","id":36089503,"number":403,"title":"Better Index/Schema Administration [JIRA: RIAK-1688]","user":{"login":"rzezeski","id":208153,"avatar_url":"https://avatars.githubusercontent.com/u/208153?v=3","gravatar_id":"","url":"https://api.github.com/users/rzezeski","html_url":"https://github.com/rzezeski","followers_url":"https://api.github.com/users/rzezeski/followers","following_url":"https://api.github.com/users/rzezeski/following{/other_user}","gists_url":"https://api.github.com/users/rzezeski/gists{/gist_id}","starred_url":"https://api.github.com/users/rzezeski/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rzezeski/subscriptions","organizations_url":"https://api.github.com/users/rzezeski/orgs","repos_url":"https://api.github.com/users/rzezeski/repos","events_url":"https://api.github.com/users/rzezeski/events{/privacy}","received_events_url":"https://api.github.com/users/rzezeski/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/basho/yokozuna/labels/JIRA:%20To%20Do","name":"JIRA: To Do","color":"ededed"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/basho/yokozuna/milestones/21","html_url":"https://github.com/basho/yokozuna/milestones/2.1","labels_url":"https://api.github.com/repos/basho/yokozuna/milestones/21/labels","id":605261,"number":21,"title":"2.1","description":"Riak 2.1","creator":{"login":"borshop","id":6749250,"avatar_url":"https://avatars.githubusercontent.com/u/6749250?v=3","gravatar_id":"","url":"https://api.github.com/users/borshop","html_url":"https://github.com/borshop","followers_url":"https://api.github.com/users/borshop/followers","following_url":"https://api.github.com/users/borshop/following{/other_user}","gists_url":"https://api.github.com/users/borshop/gists{/gist_id}","starred_url":"https://api.github.com/users/borshop/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/borshop/subscriptions","organizations_url":"https://api.github.com/users/borshop/orgs","repos_url":"https://api.github.com/users/borshop/repos","events_url":"https://api.github.com/users/borshop/events{/privacy}","received_events_url":"https://api.github.com/users/borshop/received_events","type":"User","site_admin":false},"open_issues":9,"closed_issues":4,"state":"open","created_at":"2014-03-21T17:42:14Z","updated_at":"2015-04-15T22:24:20Z","due_on":"2014-09-26T07:00:00Z","closed_at":null},"comments":6,"created_at":"2014-06-19T15:32:58Z","updated_at":"2015-07-31T17:06:03Z","closed_at":null,"body":"This is a dump of the ideas I have for better index/schema\r\nadministration in Yokozuna.\r\n\r\nCurrent State of Affairs\r\n------------------------\r\n\r\nThe current schema and index administration APIs mostly work but allow\r\ntoo many ways for users to shoot themselves in the foot.  Many users\r\nhave already found the API complex and awkward.\r\n\r\nBoth HTTP and PB allow the storing and modification of schemas.  A\r\nschema is a name along with a blob of XML.  This XML will be handed\r\noff to Solr verbatim.\r\n\r\nA user may also create and delete indexes over HTTP or PB.  When\r\ncreating an index an N value and schema name may be supplied.  If no\r\nschema is specified the built in default is used named `_yz_default`.\r\n\r\nSteps During Index Creation\r\n---------------------------\r\n\r\nThis is an overview of the steps taken by Yokozuna during index\r\ncreation.  Index creation is async and proceeds in two stages.  The\r\nfirst is the user request and the second is the creation of the Solr\r\nCores across the cluster (in Solr, indexes are called Cores).\r\n\r\n### user request ###\r\n\r\n1. If the index already exists then just return success, otherwise\r\n   proceed.\r\n\r\n2. If the schema exists then proceed, otherwise return an error saying\r\n   the schema doesn't exist.\r\n\r\n3. Write an entry to cluster-meta for the index.\r\n\r\n4. Return success.\r\n\r\nSee: `yz_wm_index:maybe_create_index`, `yz_pb_admin:maybe_create_index`, `yz_index:create`\r\n\r\n### core creation ###\r\n\r\nEvery node has an event server with a periodic tick upon which it\r\nperforms various tasks including checking the cluster-meta for new\r\nindex entries.  This is the `yz_events` server, the name comes from\r\nthe days when Yokozuna used ring events for notification which it no\r\nlonger does.\r\n\r\nRemember, these steps are occurring on _each_ node, even the one that\r\nserviced the user request.\r\n\r\n1. Tick fires.\r\n\r\n2. If Solr is up and either a) the cluster-meta hash for yz indexes\r\n   changed or b) this tick represents a \"full check\" then continue.\r\n\r\n3. Grab the list of cores from the local Solr node.\r\n\r\n4. Grab the list of indexes from the cluster-meta.\r\n\r\n5. Calculate a delta of removed, added and those that stayed the same.\r\n\r\n6. Remove indexes that were removed.\r\n\r\n7. Combine the added and same, for any that don't exist try to create\r\n   them locally and continue to next step.\r\n\r\n8. Create the directory structure needed under data dir and copy\r\n   various files, including the schema file and `solrconfig.xml`.\r\n\r\n9. Send a request to Solr to create the core.  Log return from Solr,\r\n   whether success or failure.\r\n\r\nSince this second stage is based on a tick per node that means that a)\r\nthe actual local creation of the core may occur at different times\r\nacross and cluster, b) core creation is async from the user request\r\nand c) if a core creation fails it will retry to infinity until the\r\nfailure is fixed (normally this comes in the form of a bad schema).\r\n\r\nSee: `yz_events:handle_info(tick, S)`, `yz_events:sync_indexes`, `yz_index:local_create`\r\n\r\nIssues\r\n------\r\n\r\nHere are some issues I see with the schema/index administration as it\r\nstands now.\r\n\r\n* User must understand Solr schema syntax completely.\r\n* User must remember to add special Yokozuna fields (`_yz_*`).\r\n* User cannot easily reload a schema across the cluster.\r\n* Index creation is async, there is no simple way for a user to\r\n  determine if index creation was successful across the entire\r\n  cluster.\r\n* Indefinite retry of index creation when schema is bad.  (No, you\r\n  cannot fully verify the schema because the spec is too loose and\r\n  their is no Solr API to verify.)\r\n* A different version of the schema may be used across cores if the\r\n  user modifies the schema of an existing index and then adds a new\r\n  node without first reloading the index.\r\n* No way to verify that a schema will work without creating and then\r\n  deleting the index.\r\n* No \"schemaless\" mode which users love so much even though it is a\r\n  gimmick.\r\n\r\nProposal\r\n--------\r\n\r\nHere are some of my ideas for a better administration interface.\r\n\r\n### rip out HTTP/PB admin ###\r\n\r\nRemove all HTTP/PB administration APIs except maybe for the ability to\r\nretrieve information.  But definitely no modification allowed.  Trying\r\nto make a robust administration over these protocols is annoying and\r\nerror prone.\r\n\r\n### add 'riak-admin search index' command ###\r\n\r\nThere is already a `riak-admin search` command.  Add an `index`\r\nsubcommand for administering indexes along with their schemas.  That\r\nis, there will be no separate administration for schemas.  The two\r\nwill be one.  This removes all the complexity around deleting schemas\r\nin the current system.  If each index has its own internal schema then\r\nthe problem of deleting/modifying a schema simply doesn't exist.  It\r\nalso means one less API and semantic to define.\r\n\r\nAll administration of indexes must go through the command line client.\r\nThere will be no PB/HTTP/etc channels.  Index creation should almost\r\nnever be scripted in production anyways.  For scripting in test\r\nscenarios (e.g. Riak Test) the internal Erlang APIs may be used.\r\n\r\n### synchronous + idempotent creation ###\r\n\r\nIndex creation should be synchronous to the user.  No more two stage\r\nnonsense where the user has to play games querying all the nodes to\r\ndetermine if an index was created successfully.  Creation should also\r\nbe idempotent.  If the creation fails on some nodes then the user can\r\nretry without fear of damaging the existing cores.\r\n\r\n### two-phase creation ###\r\n\r\nIndex creation should be like node join/remove.  It should be split\r\ninto the plan/test phase and a commit phase.\r\n\r\n1. Create an index with the minimal base schema.\r\n2. Add some fields.\r\n3. Write some test documents and perform queries.\r\n4. If happy with the results then 'commit' the index which will\r\n   actually create it across the cluster.\r\n\r\nWhen a user initially creates an index Yokozuna will create a\r\ntemporary index on the local node only.  The user can then index Riak\r\nobjects in this index directly without associating a bucket and run\r\nqueries to verify things are working.  When they are satisfied the\r\nindex is committed which will delete the temporary index and create\r\nthe final one across the entire cluster.  This allows users to tune\r\nthe index inside of a sandbox before creating it across the entire\r\ncluster.\r\n\r\n### \"schemaless\" mode ###\r\n\r\nFirst off, \"schemaless\" mode is a gimmick (both in Solr and\r\nElasticSearch).  You essentially start with a blank schema and as data\r\nis written fields are automatically defined on that schema.  Users\r\nlike it because they don't have to think.  The problem is that the\r\ncomputer thinks for them and unfortunately it picks the lowest common\r\ndenominator when it comes to field types.  Efficient search is all\r\nabout picking the appropriate types and analyzers.  Users should never\r\nuse \"schemaless\" mode in production.\r\n\r\nThat said, it can be useful in development while trying to figure out\r\nan initial schema.  I think it would be nice to have the ability to\r\nput an index in \"schemaless\" mode in the planning stage so that it can\r\naid the user in creating a schema.  I envision the steps as so:\r\n\r\n1. Create an index with the minimal base schema and pass the\r\n   `--schemaless-mode` option.\r\n2. Insert some data.\r\n3. Ask the index for it's current schema.\r\n4. Abort that index.\r\n5. Tweak the schema in step 3 with help from someone that understands\r\n   Solr/Lucene.\r\n6. Create an index passing in the schema from step 5.\r\n\r\n### all schema interaction via cmd line ###\r\n\r\nUsers will no longer deal directly with Solr schemas; no more hand\r\nediting of XML (except for escape hatch for \"experts\").  Creating a\r\nschema can be a daunting task for those not familiar with Solr.  It's\r\nalso very easy to mess up because of the lack of ability to verify a\r\nschema before creating the index.  A simple typo can cause issues that\r\nonly show up after creating the index.\r\n\r\nProvide an interface under the `index` subcommand for adding fields to\r\nthe schema of an index.  Not only does this avoid learning the XML but\r\nit prevents basic mistakes from being made like adding the same field\r\ntwice (which will cause a runtime error) or spelling a Java class name\r\nincorrectly.  Frankly, it gives us, the Riak devs, more control over\r\nhow users build their schemas.\r\n\r\n### provide base schemas ###\r\n\r\nRather than force users to always add all fields provide some base\r\nschemas to use.\r\n\r\n* full: The same as today's default schema.\r\n* suffix: Remove a lot of the language analysis stuff but keep the\r\n  dynamic fields.\r\n* minimal: Only the fields required by Yokozuna.  All other base\r\n  schemas must be supersets of this one.\r\n\r\nThese are just ideas for base schemas.  Another idea is base schemas\r\nper spoken language supported by Solr.  For example, Russian,\r\nJapanese, German, etc.\r\n\r\nCommand Synopsis\r\n----------------\r\n\r\n    riak-admin search index <op> [args]\r\n\r\nCreate an index:\r\n\r\n    riak-admin search index create <name> [-n <n_val>] [-b <base_schema_name>] [-s|--schemaless] ...\r\n\r\nAdd a field:\r\n\r\n    riak-admin search index add-field <name> <type> [-s|--stored] [-r|--required] [-i|-indexed] [-m|--multi-valued] ...\r\n\r\nAdd a field type:\r\n\r\n    riak-admin search index add-field-type <name> <class> [-sm|--sort-missing-last] [-ps|--precision-step <int>] [-pig|--position-increment-gap <int>] [-i|--indexed] ...\r\n\r\nCommit an index\r\n\r\n    riak-admin search index commit <name>\r\n\r\nAbort an index\r\n\r\n    riak-admin search index abort <name>\r\n\r\nDelete an index (which has been committed, maybe just use delete for abort as well?)\r\n\r\n    riak-admin search index delete <name>\r\n\r\nKeep in mind this is a straw man synopsis.  The point is that\r\nindex/schema creation would become a more controlled act versus the\r\nfree-for-all that is is now.\r\n\r\nMutating Schemas\r\n----------------\r\n\r\nMutating schemas is a very tricky thing.  Once an index has data in it\r\nyou are very limited in the schema mutations you can make.  Basically,\r\nthe only safe thing is to add fields or field types.  You cannot do\r\nthings like change the type of a field or the class name of a type.\r\nThat will cause queries and other processes to fail.  Even if you only\r\nadd fields that does not mean that all previous indexed objects will\r\nbe queryable under that field.  Action will have to be taken to\r\nreindex those objects to pick up that new field.\r\n\r\nSince Solr has APIs for swapping cores atomically I think some work\r\ncould be done on the Yokozuna side to make mutations possible (but\r\nprobably expensive).  You could extend the index subcommand to allow\r\nmutations.  It would once again be a multi-phase, plan/commit/abort,\r\nthing.  If the user commits the changes then Yokozuna will stand up a\r\nhidden core across the cluster and start reindexing all the objects in\r\nthe background.  Once that background job completes the user will be\r\nable to choose to swap the new for the old.  This allows incoming\r\nqueries to be serviced by the old index while the new one is being\r\nbuilt.  Incoming writes will have to go to both the old and new index\r\nso there will be some write amplification during this time along with\r\nthe background process.\r\n"},"comment":{"url":"https://api.github.com/repos/basho/yokozuna/issues/comments/126753353","html_url":"https://github.com/basho/yokozuna/issues/403#issuecomment-126753353","issue_url":"https://api.github.com/repos/basho/yokozuna/issues/403","id":126753353,"user":{"login":"mogadanez","id":790307,"avatar_url":"https://avatars.githubusercontent.com/u/790307?v=3","gravatar_id":"","url":"https://api.github.com/users/mogadanez","html_url":"https://github.com/mogadanez","followers_url":"https://api.github.com/users/mogadanez/followers","following_url":"https://api.github.com/users/mogadanez/following{/other_user}","gists_url":"https://api.github.com/users/mogadanez/gists{/gist_id}","starred_url":"https://api.github.com/users/mogadanez/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mogadanez/subscriptions","organizations_url":"https://api.github.com/users/mogadanez/orgs","repos_url":"https://api.github.com/users/mogadanez/repos","events_url":"https://api.github.com/users/mogadanez/events{/privacy}","received_events_url":"https://api.github.com/users/mogadanez/received_events","type":"User","site_admin":false},"created_at":"2015-07-31T17:06:03Z","updated_at":"2015-07-31T17:06:03Z","body":"does it any technical stops to have several search indices on a bucket? it can simplify migration between schemas."}}