{"action":"created","comment":{"url":"https://api.github.com/repos/cockroachdb/cockroach/pulls/comments/42001646","id":42001646,"diff_hunk":"@@ -0,0 +1,249 @@\n+- Feature Name: table_descriptor_lease\n+- Status: draft\n+- Start Date: 2015-10-09\n+- RFC PR: [#2810](https://github.com/cockroachdb/cockroach/pull/2036)\n+- Cockroach Issue: [#2036](https://github.com/cockroachdb/cockroach/issues/2036)\n+\n+# Summary\n+\n+Implement a table descriptor lease mechanism to allow safe usage of\n+cached table descriptors.\n+\n+# Motivation\n+\n+Table descriptors contain the schema for a single table and are\n+utilized by nearly every SQL operation. Fast access to the table\n+descriptor is critical for good performance. Reading the table\n+descriptor from the KV store on every operation adds significant\n+latency.\n+\n+Table descriptors are currently distributed to every node in the\n+cluster via gossipping of the system config (see\n+[schema_distribution](https://github.com/cockroachdb/cockroach/blob/master/docs/RFCS/schema_gossip.md)). Unfortunately,\n+it is not safe to use these gossipped table descriptors in almost any\n+circumstance. Consider the statements:\n+\n+```sql\n+CREATE TABLE test (key INT PRIMARY KEY, value INT);\n+CREATE INDEX foo ON test (value);\n+INSERT INTO test VALUES (1, 2);\n+```\n+\n+Depending on when the gossip of the schema change is received, the\n+`INSERT` statement may either see no cached descriptor for the `test`\n+table (which is safe because it will fall back to reading from the KV\n+store), the descriptor as it looks after the `CREATE TABLE` statement\n+or the descriptor as it looks after the `CREATE INDEX` statement. If\n+the `INSERT` statement does not see the `CREATE INDEX` statement then\n+it will merrily insert the values into `test` without creating the\n+entry for the `foo` index, thus leaving the table data in an\n+inconsistent state.\n+\n+It is easy to show similar problematic sequences of operations for\n+`DELETE`, `UPDATE` and `SELECT`. Fortunately, a solution exists in the\n+literature: break up the schema change into discrete steps for which\n+any two consecutive versions of the schema are safe for concurrent\n+use. For example, to add an index to a table we would perform the\n+following steps:\n+\n+* Add the index to the table descriptor but mark it as delete-only:\n+  update and delete operations which would delete entries for the\n+  index do so, but insert and update operations which would add new\n+  elements and read operations do not use the index.\n+* Wait for the table descriptor change to propagate to all nodes in\n+  the cluster and all uses of the previous version to finish.\n+* Mark the index as write-only: insert, update and delete operations\n+  would add entries for the index, but read operations would not use\n+  the index.\n+* Wait for the table descriptor change to propagate to all nodes in\n+  the cluster and all uses of the previous version to finish.\n+* Backfill the index.\n+* Mark the index in the table descriptor as read-write.\n+* Wait for the table descriptor change to propagate to all nodes in\n+  the cluster and all uses of the previous version to finish.\n+\n+This RFC is focused on how to wait for the table descriptor change\n+propagate to all nodes in the cluster. More accurately, it is focused\n+on how to determine when the previous version of the descriptor is no\n+longer in use. Additionally, we need mechanisms to ensure that\n+multiple updates to a table descriptor are not performed concurrently\n+and that write operations never commit once a table descriptor is too\n+old (i.e. not one of the two most recent versions).\n+\n+At a high-level, we want to provide read and write leases for table\n+descriptors. When a schema change is to be performed, the operation\n+first needs to grab a write lease for the descriptor. The write lease\n+ensures that no concurrent schema change will be performed and that\n+there is only one active version of the descriptor in the cluster.\n+Note that these are essentially the same constraint: there are only\n+two active versions of a descriptor during a schema change.\n+\n+Before using a table descriptor for a DML operation (i.e. `SELECT`,\n+`INSERT`, `UPDATE`, `DELETE`, etc), the operation needs to obtain a\n+read lease for the descriptor. The lease will be guaranteed valid for\n+a significant period of time (on the order of minutes) and the lease\n+period can be extended as long as a scheme change is not\n+performed. When the operation completes it will release the read\n+lease.\n+\n+# Detailed design\n+\n+Table descriptors will be extended with a version number that is\n+incremented on every change to the descriptor:\n+\n+```proto\n+message TableDescriptor {\n+  ...\n+  optional uint32 id;\n+  optional uint32 version;\n+  ...\n+}\n+```\n+\n+Leases will be tied to a specific version of the descriptor. Lease\n+state will be stored in a new `system.lease` table:\n+\n+```sql\n+CREATE TABLE system.lease (\n+  DescID     INT,\n+  Version    INT,\n+  NodeID     INT,\n+  Expiration TIMESTAMP,\n+  PRIMARY KEY (DescID, Version, NodeID, Expiration)\n+)\n+```\n+\n+Entries in the lease table will be added and removed as leases are\n+acquired and released. A background goroutine will periodically delete\n+expired leases. \n+\n+Leases will be granted for a duration measured in minutes (we'll\n+assume 5m for the rest of this doc, though experimentation may tune\n+this number). A node will acquire a lease before using it in an\n+operation and may release the lease when the last local operation\n+completes that was using the lease.\n+\n+The leader of the range containing a table descriptor will gossip the\n+most recent version of that table descriptor using the gossip key\n+`\"table-<descID>\"` and the value will be the version number. The\n+gossiping of the most recent table versions allows nodes to\n+asynchronously discover when a new table version is written. But note\n+that it is not necessary for correctness as the protocol for acquiring\n+a lease ensures that only the most recent version of a descriptor can\n+have a new lease granted on it.\n+\n+Lease acquisition will perform the following steps in a transaction:\n+\n+* `SELECT descriptor FROM system.descriptor WHERE id = <descID>`\n+* `INSERT INTO system.lease VALUES (<desc.ID>, <desc.version>, <nodeID>, <expiration>)`\n+\n+Nodes will maintain a map from `<descID, version>` to\n+`<TableDescriptor, expiration, localRefCount>`. The local reference\n+count will be incremented when a transaction first uses a table and\n+decremented when the transaction commits/aborts. When the node\n+discovers a new version of the table, either via gossip or by\n+acquiring a lease and discovering the version has changed it can\n+release the lease on the old version when there are no more local\n+references:\n+\n+* `DELETE FROM system.lease WHERE (DescID, Version, NodeID, Expiration) = (<descID>, <version>, <nodeID>, <expiration>)`\n+\n+A schema change operation needs to ensure that there is only one\n+version of a descriptor in use in the cluster before a mutation of the\n+descriptor. The operation will perform the following steps\n+transactionally:\n+\n+* `SELECT descriptor FROM system.descriptor WHERE id = <descID>`\n+* `SELECT DISTINCT version FROM system.lease WHERE descID = <descID> AND expiration > current_time()`\n+* Ensure that there is only one active version which is equal to the current version.\n+* Perform the edits to the descriptor and bump the descriptor version.\n+* `UPDATE system.descriptor WHERE id = <descID> SET descriptor = <desc>`\n+\n+Note that the updating of the table descriptor will cause the table\n+version to be gossipped alerting nodes to the new version and causing\n+them to release leases on the old version. The expectation is that\n+nodes will fairly quickly transition to using the new version and\n+release all leases to the old version allowing another step in the\n+schema change operation to take place. The schema change operation\n+must poll the `system.lease` table waiting for the number of active","path":"docs/RFCS/table_descriptor_lease.md","position":177,"original_position":168,"commit_id":"24c2d7b8fde507878700d9b9f1501033117b8a88","original_commit_id":"b9353ba3871b26b9deca82319b95613df4d299f1","user":{"login":"tschottdorf","id":5076964,"avatar_url":"https://avatars.githubusercontent.com/u/5076964?v=3","gravatar_id":"","url":"https://api.github.com/users/tschottdorf","html_url":"https://github.com/tschottdorf","followers_url":"https://api.github.com/users/tschottdorf/followers","following_url":"https://api.github.com/users/tschottdorf/following{/other_user}","gists_url":"https://api.github.com/users/tschottdorf/gists{/gist_id}","starred_url":"https://api.github.com/users/tschottdorf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tschottdorf/subscriptions","organizations_url":"https://api.github.com/users/tschottdorf/orgs","repos_url":"https://api.github.com/users/tschottdorf/repos","events_url":"https://api.github.com/users/tschottdorf/events{/privacy}","received_events_url":"https://api.github.com/users/tschottdorf/received_events","type":"User","site_admin":false},"body":"there's no \"outside of a (kv) transaction\", at least not currently, in SQL. Or did we change that? I don't think we need to discuss this further now, it's only a minor performance concern that could kick in on large large large clusters.","created_at":"2015-10-14T14:38:54Z","updated_at":"2015-10-14T14:38:54Z","html_url":"https://github.com/cockroachdb/cockroach/pull/2810#discussion_r42001646","pull_request_url":"https://api.github.com/repos/cockroachdb/cockroach/pulls/2810","_links":{"self":{"href":"https://api.github.com/repos/cockroachdb/cockroach/pulls/comments/42001646"},"html":{"href":"https://github.com/cockroachdb/cockroach/pull/2810#discussion_r42001646"},"pull_request":{"href":"https://api.github.com/repos/cockroachdb/cockroach/pulls/2810"}}},"pull_request":{"url":"https://api.github.com/repos/cockroachdb/cockroach/pulls/2810","id":47285317,"html_url":"https://github.com/cockroachdb/cockroach/pull/2810","diff_url":"https://github.com/cockroachdb/cockroach/pull/2810.diff","patch_url":"https://github.com/cockroachdb/cockroach/pull/2810.patch","issue_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/2810","number":2810,"state":"open","locked":false,"title":"RFC: table descriptor leases.","user":{"login":"petermattis","id":6201335,"avatar_url":"https://avatars.githubusercontent.com/u/6201335?v=3","gravatar_id":"","url":"https://api.github.com/users/petermattis","html_url":"https://github.com/petermattis","followers_url":"https://api.github.com/users/petermattis/followers","following_url":"https://api.github.com/users/petermattis/following{/other_user}","gists_url":"https://api.github.com/users/petermattis/gists{/gist_id}","starred_url":"https://api.github.com/users/petermattis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/petermattis/subscriptions","organizations_url":"https://api.github.com/users/petermattis/orgs","repos_url":"https://api.github.com/users/petermattis/repos","events_url":"https://api.github.com/users/petermattis/events{/privacy}","received_events_url":"https://api.github.com/users/petermattis/received_events","type":"User","site_admin":false},"body":"@tschottdorf I feel this is both somewhat complex, but also a manageable bit of complexity. Definitely poke holes or sketch out your thoughts as alternative if they differed significantly.","created_at":"2015-10-09T15:26:48Z","updated_at":"2015-10-14T14:38:54Z","closed_at":null,"merged_at":null,"merge_commit_sha":"fbb9fd83f0a331a4528398e2351e0b20881f5989","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/cockroachdb/cockroach/pulls/2810/commits","review_comments_url":"https://api.github.com/repos/cockroachdb/cockroach/pulls/2810/comments","review_comment_url":"https://api.github.com/repos/cockroachdb/cockroach/pulls/comments{/number}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/2810/comments","statuses_url":"https://api.github.com/repos/cockroachdb/cockroach/statuses/24c2d7b8fde507878700d9b9f1501033117b8a88","head":{"label":"cockroachdb:pmattis/table-descriptor-lease","ref":"pmattis/table-descriptor-lease","sha":"24c2d7b8fde507878700d9b9f1501033117b8a88","user":{"login":"cockroachdb","id":6748139,"avatar_url":"https://avatars.githubusercontent.com/u/6748139?v=3","gravatar_id":"","url":"https://api.github.com/users/cockroachdb","html_url":"https://github.com/cockroachdb","followers_url":"https://api.github.com/users/cockroachdb/followers","following_url":"https://api.github.com/users/cockroachdb/following{/other_user}","gists_url":"https://api.github.com/users/cockroachdb/gists{/gist_id}","starred_url":"https://api.github.com/users/cockroachdb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cockroachdb/subscriptions","organizations_url":"https://api.github.com/users/cockroachdb/orgs","repos_url":"https://api.github.com/users/cockroachdb/repos","events_url":"https://api.github.com/users/cockroachdb/events{/privacy}","received_events_url":"https://api.github.com/users/cockroachdb/received_events","type":"Organization","site_admin":false},"repo":{"id":16563587,"name":"cockroach","full_name":"cockroachdb/cockroach","owner":{"login":"cockroachdb","id":6748139,"avatar_url":"https://avatars.githubusercontent.com/u/6748139?v=3","gravatar_id":"","url":"https://api.github.com/users/cockroachdb","html_url":"https://github.com/cockroachdb","followers_url":"https://api.github.com/users/cockroachdb/followers","following_url":"https://api.github.com/users/cockroachdb/following{/other_user}","gists_url":"https://api.github.com/users/cockroachdb/gists{/gist_id}","starred_url":"https://api.github.com/users/cockroachdb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cockroachdb/subscriptions","organizations_url":"https://api.github.com/users/cockroachdb/orgs","repos_url":"https://api.github.com/users/cockroachdb/repos","events_url":"https://api.github.com/users/cockroachdb/events{/privacy}","received_events_url":"https://api.github.com/users/cockroachdb/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/cockroachdb/cockroach","description":"A Scalable, Geo-Replicated, Transactional Datastore","fork":false,"url":"https://api.github.com/repos/cockroachdb/cockroach","forks_url":"https://api.github.com/repos/cockroachdb/cockroach/forks","keys_url":"https://api.github.com/repos/cockroachdb/cockroach/keys{/key_id}","collaborators_url":"https://api.github.com/repos/cockroachdb/cockroach/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/cockroachdb/cockroach/teams","hooks_url":"https://api.github.com/repos/cockroachdb/cockroach/hooks","issue_events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/events{/number}","events_url":"https://api.github.com/repos/cockroachdb/cockroach/events","assignees_url":"https://api.github.com/repos/cockroachdb/cockroach/assignees{/user}","branches_url":"https://api.github.com/repos/cockroachdb/cockroach/branches{/branch}","tags_url":"https://api.github.com/repos/cockroachdb/cockroach/tags","blobs_url":"https://api.github.com/repos/cockroachdb/cockroach/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/cockroachdb/cockroach/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/cockroachdb/cockroach/git/refs{/sha}","trees_url":"https://api.github.com/repos/cockroachdb/cockroach/git/trees{/sha}","statuses_url":"https://api.github.com/repos/cockroachdb/cockroach/statuses/{sha}","languages_url":"https://api.github.com/repos/cockroachdb/cockroach/languages","stargazers_url":"https://api.github.com/repos/cockroachdb/cockroach/stargazers","contributors_url":"https://api.github.com/repos/cockroachdb/cockroach/contributors","subscribers_url":"https://api.github.com/repos/cockroachdb/cockroach/subscribers","subscription_url":"https://api.github.com/repos/cockroachdb/cockroach/subscription","commits_url":"https://api.github.com/repos/cockroachdb/cockroach/commits{/sha}","git_commits_url":"https://api.github.com/repos/cockroachdb/cockroach/git/commits{/sha}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/comments{/number}","issue_comment_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/comments{/number}","contents_url":"https://api.github.com/repos/cockroachdb/cockroach/contents/{+path}","compare_url":"https://api.github.com/repos/cockroachdb/cockroach/compare/{base}...{head}","merges_url":"https://api.github.com/repos/cockroachdb/cockroach/merges","archive_url":"https://api.github.com/repos/cockroachdb/cockroach/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/cockroachdb/cockroach/downloads","issues_url":"https://api.github.com/repos/cockroachdb/cockroach/issues{/number}","pulls_url":"https://api.github.com/repos/cockroachdb/cockroach/pulls{/number}","milestones_url":"https://api.github.com/repos/cockroachdb/cockroach/milestones{/number}","notifications_url":"https://api.github.com/repos/cockroachdb/cockroach/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/labels{/name}","releases_url":"https://api.github.com/repos/cockroachdb/cockroach/releases{/id}","created_at":"2014-02-06T00:18:47Z","updated_at":"2015-10-14T07:33:53Z","pushed_at":"2015-10-14T14:38:39Z","git_url":"git://github.com/cockroachdb/cockroach.git","ssh_url":"git@github.com:cockroachdb/cockroach.git","clone_url":"https://github.com/cockroachdb/cockroach.git","svn_url":"https://github.com/cockroachdb/cockroach","homepage":null,"size":72363,"stargazers_count":5435,"watchers_count":5435,"language":"Go","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":455,"mirror_url":null,"open_issues_count":192,"forks":455,"open_issues":192,"watchers":5435,"default_branch":"master"}},"base":{"label":"cockroachdb:master","ref":"master","sha":"d55f3a26bce8c557ff1ba361bc8737e2e4ab1510","user":{"login":"cockroachdb","id":6748139,"avatar_url":"https://avatars.githubusercontent.com/u/6748139?v=3","gravatar_id":"","url":"https://api.github.com/users/cockroachdb","html_url":"https://github.com/cockroachdb","followers_url":"https://api.github.com/users/cockroachdb/followers","following_url":"https://api.github.com/users/cockroachdb/following{/other_user}","gists_url":"https://api.github.com/users/cockroachdb/gists{/gist_id}","starred_url":"https://api.github.com/users/cockroachdb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cockroachdb/subscriptions","organizations_url":"https://api.github.com/users/cockroachdb/orgs","repos_url":"https://api.github.com/users/cockroachdb/repos","events_url":"https://api.github.com/users/cockroachdb/events{/privacy}","received_events_url":"https://api.github.com/users/cockroachdb/received_events","type":"Organization","site_admin":false},"repo":{"id":16563587,"name":"cockroach","full_name":"cockroachdb/cockroach","owner":{"login":"cockroachdb","id":6748139,"avatar_url":"https://avatars.githubusercontent.com/u/6748139?v=3","gravatar_id":"","url":"https://api.github.com/users/cockroachdb","html_url":"https://github.com/cockroachdb","followers_url":"https://api.github.com/users/cockroachdb/followers","following_url":"https://api.github.com/users/cockroachdb/following{/other_user}","gists_url":"https://api.github.com/users/cockroachdb/gists{/gist_id}","starred_url":"https://api.github.com/users/cockroachdb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cockroachdb/subscriptions","organizations_url":"https://api.github.com/users/cockroachdb/orgs","repos_url":"https://api.github.com/users/cockroachdb/repos","events_url":"https://api.github.com/users/cockroachdb/events{/privacy}","received_events_url":"https://api.github.com/users/cockroachdb/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/cockroachdb/cockroach","description":"A Scalable, Geo-Replicated, Transactional Datastore","fork":false,"url":"https://api.github.com/repos/cockroachdb/cockroach","forks_url":"https://api.github.com/repos/cockroachdb/cockroach/forks","keys_url":"https://api.github.com/repos/cockroachdb/cockroach/keys{/key_id}","collaborators_url":"https://api.github.com/repos/cockroachdb/cockroach/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/cockroachdb/cockroach/teams","hooks_url":"https://api.github.com/repos/cockroachdb/cockroach/hooks","issue_events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/events{/number}","events_url":"https://api.github.com/repos/cockroachdb/cockroach/events","assignees_url":"https://api.github.com/repos/cockroachdb/cockroach/assignees{/user}","branches_url":"https://api.github.com/repos/cockroachdb/cockroach/branches{/branch}","tags_url":"https://api.github.com/repos/cockroachdb/cockroach/tags","blobs_url":"https://api.github.com/repos/cockroachdb/cockroach/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/cockroachdb/cockroach/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/cockroachdb/cockroach/git/refs{/sha}","trees_url":"https://api.github.com/repos/cockroachdb/cockroach/git/trees{/sha}","statuses_url":"https://api.github.com/repos/cockroachdb/cockroach/statuses/{sha}","languages_url":"https://api.github.com/repos/cockroachdb/cockroach/languages","stargazers_url":"https://api.github.com/repos/cockroachdb/cockroach/stargazers","contributors_url":"https://api.github.com/repos/cockroachdb/cockroach/contributors","subscribers_url":"https://api.github.com/repos/cockroachdb/cockroach/subscribers","subscription_url":"https://api.github.com/repos/cockroachdb/cockroach/subscription","commits_url":"https://api.github.com/repos/cockroachdb/cockroach/commits{/sha}","git_commits_url":"https://api.github.com/repos/cockroachdb/cockroach/git/commits{/sha}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/comments{/number}","issue_comment_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/comments{/number}","contents_url":"https://api.github.com/repos/cockroachdb/cockroach/contents/{+path}","compare_url":"https://api.github.com/repos/cockroachdb/cockroach/compare/{base}...{head}","merges_url":"https://api.github.com/repos/cockroachdb/cockroach/merges","archive_url":"https://api.github.com/repos/cockroachdb/cockroach/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/cockroachdb/cockroach/downloads","issues_url":"https://api.github.com/repos/cockroachdb/cockroach/issues{/number}","pulls_url":"https://api.github.com/repos/cockroachdb/cockroach/pulls{/number}","milestones_url":"https://api.github.com/repos/cockroachdb/cockroach/milestones{/number}","notifications_url":"https://api.github.com/repos/cockroachdb/cockroach/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/labels{/name}","releases_url":"https://api.github.com/repos/cockroachdb/cockroach/releases{/id}","created_at":"2014-02-06T00:18:47Z","updated_at":"2015-10-14T07:33:53Z","pushed_at":"2015-10-14T14:38:39Z","git_url":"git://github.com/cockroachdb/cockroach.git","ssh_url":"git@github.com:cockroachdb/cockroach.git","clone_url":"https://github.com/cockroachdb/cockroach.git","svn_url":"https://github.com/cockroachdb/cockroach","homepage":null,"size":72363,"stargazers_count":5435,"watchers_count":5435,"language":"Go","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":455,"mirror_url":null,"open_issues_count":192,"forks":455,"open_issues":192,"watchers":5435,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/cockroachdb/cockroach/pulls/2810"},"html":{"href":"https://github.com/cockroachdb/cockroach/pull/2810"},"issue":{"href":"https://api.github.com/repos/cockroachdb/cockroach/issues/2810"},"comments":{"href":"https://api.github.com/repos/cockroachdb/cockroach/issues/2810/comments"},"review_comments":{"href":"https://api.github.com/repos/cockroachdb/cockroach/pulls/2810/comments"},"review_comment":{"href":"https://api.github.com/repos/cockroachdb/cockroach/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/cockroachdb/cockroach/pulls/2810/commits"},"statuses":{"href":"https://api.github.com/repos/cockroachdb/cockroach/statuses/24c2d7b8fde507878700d9b9f1501033117b8a88"}}}}