{"action":"created","issue":{"url":"https://api.github.com/repos/scylladb/scylla/issues/548","labels_url":"https://api.github.com/repos/scylladb/scylla/issues/548/labels{/name}","comments_url":"https://api.github.com/repos/scylladb/scylla/issues/548/comments","events_url":"https://api.github.com/repos/scylladb/scylla/issues/548/events","html_url":"https://github.com/scylladb/scylla/issues/548","id":116212735,"number":548,"title":"Column family has wrong ID","user":{"login":"glommer","id":331197,"avatar_url":"https://avatars.githubusercontent.com/u/331197?v=3","gravatar_id":"","url":"https://api.github.com/users/glommer","html_url":"https://github.com/glommer","followers_url":"https://api.github.com/users/glommer/followers","following_url":"https://api.github.com/users/glommer/following{/other_user}","gists_url":"https://api.github.com/users/glommer/gists{/gist_id}","starred_url":"https://api.github.com/users/glommer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/glommer/subscriptions","organizations_url":"https://api.github.com/users/glommer/orgs","repos_url":"https://api.github.com/users/glommer/repos","events_url":"https://api.github.com/users/glommer/events{/privacy}","received_events_url":"https://api.github.com/users/glommer/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/scylladb/scylla/labels/bug","name":"bug","color":"eb6420"},{"url":"https://api.github.com/repos/scylladb/scylla/labels/high","name":"high","color":"d4c5f9"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/scylladb/scylla/milestones/1","html_url":"https://github.com/scylladb/scylla/milestones/GA","labels_url":"https://api.github.com/repos/scylladb/scylla/milestones/1/labels","id":1164307,"number":1,"title":"GA","description":null,"creator":{"login":"dorlaor","id":1735237,"avatar_url":"https://avatars.githubusercontent.com/u/1735237?v=3","gravatar_id":"","url":"https://api.github.com/users/dorlaor","html_url":"https://github.com/dorlaor","followers_url":"https://api.github.com/users/dorlaor/followers","following_url":"https://api.github.com/users/dorlaor/following{/other_user}","gists_url":"https://api.github.com/users/dorlaor/gists{/gist_id}","starred_url":"https://api.github.com/users/dorlaor/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dorlaor/subscriptions","organizations_url":"https://api.github.com/users/dorlaor/orgs","repos_url":"https://api.github.com/users/dorlaor/repos","events_url":"https://api.github.com/users/dorlaor/events{/privacy}","received_events_url":"https://api.github.com/users/dorlaor/received_events","type":"User","site_admin":false},"open_issues":169,"closed_issues":40,"state":"open","created_at":"2015-06-15T06:37:52Z","updated_at":"2015-12-23T12:53:25Z","due_on":null,"closed_at":null},"comments":1,"created_at":"2015-11-10T22:04:22Z","updated_at":"2015-12-23T13:04:05Z","closed_at":null,"body":"In a cluster with 3 nodes, loaded with standard cassandra-stress write test, I am seeing this in one of the nodes:\r\n```\r\nDEBUG [shard 25] sstable - Writing TOC file\r\n/data/data/keyspace1/standard1-9005484087af11e593f7000000000001/keyspace1-standard1-ka-56-TOC.txt.tmp\r\nDEBUG [shard 25] database - Flushing done\r\nERROR [shard 25] database - failed to write sstable: std::system_error\r\n(error system:2, No such file or directory)\r\n```\r\n\r\nThe reason we fail this write, is that this column family does not exist:\r\n\r\n```\r\n[fedora@host ~]$ sudo ls /data/data/keyspace1/\r\nstandard1-9001c5d087af11e5b67a000000000000\r\n```\r\nSo the CF is there, but it has a different UUID attached to it. That\r\ndirectory is empty,  and the commitlog directory is 9G big in size.\r\n\r\nI don't know how to reproduce but something triggers the problem. I will try more tomorrow, but we should be inspecting code paths that may lead to this\r\n\r\n@penberg says on the matter: \r\n\r\n>One potential explanation I can think of is that CF was created on\r\n>different nodes simultaneously and our schema merging validation logic\r\n>in database::update_column_family() failed to kick in."},"comment":{"url":"https://api.github.com/repos/scylladb/scylla/issues/comments/166890176","html_url":"https://github.com/scylladb/scylla/issues/548#issuecomment-166890176","issue_url":"https://api.github.com/repos/scylladb/scylla/issues/548","id":166890176,"user":{"login":"slivne","id":3465480,"avatar_url":"https://avatars.githubusercontent.com/u/3465480?v=3","gravatar_id":"","url":"https://api.github.com/users/slivne","html_url":"https://github.com/slivne","followers_url":"https://api.github.com/users/slivne/followers","following_url":"https://api.github.com/users/slivne/following{/other_user}","gists_url":"https://api.github.com/users/slivne/gists{/gist_id}","starred_url":"https://api.github.com/users/slivne/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/slivne/subscriptions","organizations_url":"https://api.github.com/users/slivne/orgs","repos_url":"https://api.github.com/users/slivne/repos","events_url":"https://api.github.com/users/slivne/events{/privacy}","received_events_url":"https://api.github.com/users/slivne/received_events","type":"User","site_admin":false},"created_at":"2015-12-23T13:04:05Z","updated_at":"2015-12-23T13:04:05Z","body":"@glommer just to verify:\r\n\r\nthe scenario is the following:\r\n\r\n1. a cluster f 3 nodes\r\n2.  run multiple cassandra-stress toward the nodes"}}