{"action":"created","issue":{"url":"https://api.github.com/repos/scylladb/scylla/issues/333","labels_url":"https://api.github.com/repos/scylladb/scylla/issues/333/labels{/name}","comments_url":"https://api.github.com/repos/scylladb/scylla/issues/333/comments","events_url":"https://api.github.com/repos/scylladb/scylla/issues/333/events","html_url":"https://github.com/scylladb/scylla/issues/333","id":105769450,"number":333,"title":"cassandra-stress: fails with RF = 2 and CL = TWO","user":{"login":"asias","id":320076,"avatar_url":"https://avatars.githubusercontent.com/u/320076?v=3","gravatar_id":"","url":"https://api.github.com/users/asias","html_url":"https://github.com/asias","followers_url":"https://api.github.com/users/asias/followers","following_url":"https://api.github.com/users/asias/following{/other_user}","gists_url":"https://api.github.com/users/asias/gists{/gist_id}","starred_url":"https://api.github.com/users/asias/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/asias/subscriptions","organizations_url":"https://api.github.com/users/asias/orgs","repos_url":"https://api.github.com/users/asias/repos","events_url":"https://api.github.com/users/asias/events{/privacy}","received_events_url":"https://api.github.com/users/asias/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/scylladb/scylla/labels/bug","name":"bug","color":"eb6420"},{"url":"https://api.github.com/repos/scylladb/scylla/labels/showstopper","name":"showstopper","color":"e11d21"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/scylladb/scylla/milestones/2","html_url":"https://github.com/scylladb/scylla/milestones/anglerfish","labels_url":"https://api.github.com/repos/scylladb/scylla/milestones/2/labels","id":1225814,"number":2,"title":"anglerfish","description":"0.9.0 beta","creator":{"login":"dorlaor","id":1735237,"avatar_url":"https://avatars.githubusercontent.com/u/1735237?v=3","gravatar_id":"","url":"https://api.github.com/users/dorlaor","html_url":"https://github.com/dorlaor","followers_url":"https://api.github.com/users/dorlaor/followers","following_url":"https://api.github.com/users/dorlaor/following{/other_user}","gists_url":"https://api.github.com/users/dorlaor/gists{/gist_id}","starred_url":"https://api.github.com/users/dorlaor/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dorlaor/subscriptions","organizations_url":"https://api.github.com/users/dorlaor/orgs","repos_url":"https://api.github.com/users/dorlaor/repos","events_url":"https://api.github.com/users/dorlaor/events{/privacy}","received_events_url":"https://api.github.com/users/dorlaor/received_events","type":"User","site_admin":false},"open_issues":31,"closed_issues":148,"state":"open","created_at":"2015-07-25T20:45:27Z","updated_at":"2015-09-23T19:38:17Z","due_on":null,"closed_at":null},"comments":25,"created_at":"2015-09-10T09:31:52Z","updated_at":"2015-09-23T19:40:32Z","closed_at":null,"body":"On aws, there are 2 nodes in the cluster.\r\n\r\n```\r\ncassandra-stress  mixed ratio\\(write=95,read=5\\) n=10000 cl=TWO  -rate threads=8  -mode cql3 native -schema replication\\(strategy=org.apache.cassandra.locator.SimpleStrategy,factor=2\\) -node 172.31.14.220\r\n\\Sleeping 2s...\r\nRunning [READ, WRITE] with 8 threads for 10000 iteration\r\nINFO  09:22:21 Using data-center name 'datacenter1' for DCAwareRoundRobinPolicy (if this is incorrect, please provide the correct datacenter name with DCAwareRoundRobinPolicy constructor)\r\nINFO  09:22:21 New Cassandra host /172.31.14.220:9042 added\r\nINFO  09:22:21 New Cassandra host /172.31.15.223:9042 added\r\nConnected to cluster: Test Cluster\r\nDatatacenter: datacenter1; Host: /172.31.14.220; Rack: rack1\r\nDatatacenter: datacenter1; Host: /172.31.15.223; Rack: rack1\r\nFailed to connect over JMX; not collecting these stats\r\ntype,      total ops,    op/s,    pk/s,   row/s,    mean,     med,     .95,     .99,    .999,     max,   time,   stderr, errors,  gc: #,  max ms,  sum ms,  sdv ms,      mb\r\ncom.datastax.driver.core.exceptions.UnavailableException: Not enough replica available for query at consistency TWO (2 required but only 1 alive)\r\ncom.datastax.driver.core.exceptions.UnavailableException: Not enough replica available for query at consistency TWO (2 required but only 1 alive)\r\ncom.datastax.driver.core.exceptions.UnavailableException: Not enough replica available for query at consistency TWO (2 required but only 1 alive)\r\ncom.datastax.driver.core.exceptions.UnavailableException: Not enough replica available for query at consistency TWO (2 required but only 1 alive)\r\ncom.datastax.driver.core.exceptions.UnavailableException: Not enough replica available for query at consistency TWO (2 required but only 1 alive)\r\ncom.datastax.driver.core.exceptions.UnavailableException: Not enough replica available for query at consistency TWO (2 required but only 1 alive)\r\ncom.datastax.driver.core.exceptions.UnavailableException: Not enough replica available for query at consistency TWO (2 required but only 1 alive)\r\njava.io.IOException: Operation x10 on key(s) [4f34383236504e333330]: Error executing: (UnavailableException): Not enough replica available for query at consistency TWO (2 required but only 1 alive)\r\n\r\n        at org.apache.cassandra.stress.Operation.error(Operation.java:216)\r\n        at org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:188)\r\n        at org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:99)\r\n        at org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:107)\r\n        at org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:259)\r\n        at org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:309)\r\ncom.datastax.driver.core.exceptions.UnavailableException: Not enough replica available for query at consistency TWO (2 required but only 1 alive)java.io.IOException: Operation x10 on key(s) [334c3635384b32303930]: Error execu\r\nting: (UnavailableException): Not enough replica available for query at consistency TWO (2 required but only 1 alive)\r\n\r\n\r\n        at org.apache.cassandra.stress.Operation.error(Operation.java:216)\r\n        at org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:188)\r\n        at org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:99)\r\ncom.datastax.driver.core.exceptions.UnavailableException: Not enough replica available for query at consistency TWO (2 required but only 1 alive)\r\ncom.datastax.driver.core.exceptions.UnavailableException: Not enough replica available for query at consistency TWO (2 required but only 1 alive)\r\ncom.datastax.driver.core.exceptions.UnavailableException: Not enough replica available for query at consistency TWO (2 required but only 1 alive)\r\ncom.datastax.driver.core.exceptions.UnavailableException: Not enough replica available for query at consistency TWO (2 required but only 1 alive)\r\ncom.datastax.driver.core.exceptions.UnavailableException: Not enough replica available for query at consistency TWO (2 required but only 1 alive)\r\n        at org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:107)\r\n        at org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:259)\r\n        at org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:309)\r\njava.io.IOException: Operation x10 on key(s) [305036304b4c4f343330]: Error executing: (UnavailableException): Not enough replica available for query at consistency TWO (2 required but only 1 alive)\r\n\r\n        at org.apache.cassandra.stress.Operation.error(Operation.java:216)\r\ncom.datastax.driver.core.exceptions.UnavailableException: Not enough replica available for query at consistency TWO (2 required but only 1 alive)\r\n        at org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:188)\r\n        at org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:99)\r\ncom.datastax.driver.core.exceptions.UnavailableException: Not enough replica available for query at consistency TWO (2 required but only 1 alive)\r\n        at org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:107)\r\n        at org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:259)\r\n        at org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:309)\r\ncom.datastax.driver.core.exceptions.UnavailableException: Not enough replica available for query at consistency TWO (2 required but only 1 alive)\r\njava.io.IOException: Operation x10 on key(s) [4e5031384d4c38313530]: Error executing: (UnavailableException): Not enough replica available for query at consistency TWO (2 required but only 1 alive)\r\n\r\n        at org.apache.cassandra.stress.Operation.error(Operation.java:216)\r\n        at org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:188)\r\n        at org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:99)\r\n        at org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:107)\r\n        at org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:259)\r\n        at org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:309)\r\ncom.datastax.driver.core.exceptions.UnavailableException: Not enough replica available for query at consistency TWO (2 required but only 1 alive)\r\ncom.datastax.driver.core.exceptions.UnavailableException: Not enough replica available for query at consistency TWO (2 required but only 1 alive)\r\ncom.datastax.driver.core.exceptions.UnavailableException: Not enough replica available for query at consistency TWO (2 required but only 1 alive)\r\ncom.datastax.driver.core.exceptions.UnavailableException: Not enough replica available for query at consistency TWO (2 required but only 1 alive)\r\ncom.datastax.driver.core.exceptions.UnavailableException: Not enough replica available for query at consistency TWO (2 required but only 1 alive)\r\ncom.datastax.driver.core.exceptions.UnavailableException: Not enough replica available for query at consistency TWO (2 required but only 1 alive)\r\njava.io.IOException: Operation x10 on key(s) [4f303435383438313830]: Error executing: (UnavailableException): Not enough replica available for query at consistency TWO (2 required but only 1 alive)\r\n```"},"comment":{"url":"https://api.github.com/repos/scylladb/scylla/issues/comments/142707310","html_url":"https://github.com/scylladb/scylla/issues/333#issuecomment-142707310","issue_url":"https://api.github.com/repos/scylladb/scylla/issues/333","id":142707310,"user":{"login":"slivne","id":3465480,"avatar_url":"https://avatars.githubusercontent.com/u/3465480?v=3","gravatar_id":"","url":"https://api.github.com/users/slivne","html_url":"https://github.com/slivne","followers_url":"https://api.github.com/users/slivne/followers","following_url":"https://api.github.com/users/slivne/following{/other_user}","gists_url":"https://api.github.com/users/slivne/gists{/gist_id}","starred_url":"https://api.github.com/users/slivne/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/slivne/subscriptions","organizations_url":"https://api.github.com/users/slivne/orgs","repos_url":"https://api.github.com/users/slivne/repos","events_url":"https://api.github.com/users/slivne/events{/privacy}","received_events_url":"https://api.github.com/users/slivne/received_events","type":"User","site_admin":false},"created_at":"2015-09-23T19:40:32Z","updated_at":"2015-09-23T19:40:32Z","body":"for anglerfish we have a solution for large servers on aws in which cpu0 handles all networking interrupts and the rest of the cpus run scylla \r\n\r\nfor next milestone we may (probably) have a better solution"}}