{"action":"created","issue":{"url":"https://api.github.com/repos/codership/mysql-wsrep/issues/115","labels_url":"https://api.github.com/repos/codership/mysql-wsrep/issues/115/labels{/name}","comments_url":"https://api.github.com/repos/codership/mysql-wsrep/issues/115/comments","events_url":"https://api.github.com/repos/codership/mysql-wsrep/issues/115/events","html_url":"https://github.com/codership/mysql-wsrep/issues/115","id":71514697,"number":115,"title":"Percona: \"Query execution was interrupted\" on slave thread","user":{"login":"philip-galera","id":8738032,"avatar_url":"https://avatars.githubusercontent.com/u/8738032?v=3","gravatar_id":"","url":"https://api.github.com/users/philip-galera","html_url":"https://github.com/philip-galera","followers_url":"https://api.github.com/users/philip-galera/followers","following_url":"https://api.github.com/users/philip-galera/following{/other_user}","gists_url":"https://api.github.com/users/philip-galera/gists{/gist_id}","starred_url":"https://api.github.com/users/philip-galera/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/philip-galera/subscriptions","organizations_url":"https://api.github.com/users/philip-galera/orgs","repos_url":"https://api.github.com/users/philip-galera/repos","events_url":"https://api.github.com/users/philip-galera/events{/privacy}","received_events_url":"https://api.github.com/users/philip-galera/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/codership/mysql-wsrep/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/codership/mysql-wsrep/labels/Percona","name":"Percona","color":"fbca04"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-04-28T07:50:32Z","updated_at":"2015-04-28T08:11:26Z","closed_at":null,"body":"Percona have reported the following situation. Running a 2-minute ALTER  causes the following situation on the slave:\r\n\r\n```\r\n2015-04-28 07:05:23 2396 [ERROR] Slave SQL: Error 'Query execution was interrupted' on query. Default database: 'mobilegambling'. Query: 'alter table PlayerAc\r\ncounts add VIPStatus TINYINT(1) not null default 0', Error_code: 1317\r\n[10:16:11] Karthik Appigatla: 2015-04-28 07:05:23 2396 [Warning] WSREP: RBR event 1 Query apply warning: 1, 2146895225\r\n2015-04-28 07:05:23 2396 [Note] WSREP: applier aborted: 2146895225\r\n2015-04-28 07:05:23 2396 [Warning] WSREP: Ignoring error for TO isolated action: source: 6e53bc11-b807-11e4-b741-66e6a99cdd5e version: 3 local: 0 state: APPLY\r\nING flags: 65 conn_id: 46010735 trx_id: -1 seqnos (l: 1009433869, g: 2146895225, s: 2146895223, d: 2146895224, ts: 14509073659622545)\r\n2015-04-28 07:05:47 2396 [Note] WSREP: applier has been aborted, skipping apply_rbr: 2146895241\r\n2015-04-28 07:05:47 2396 [Warning] WSREP: Failed to apply app buffer: seqno: 2146895241, status: 1\r\n         at galera/src/trx_handle.cpp:apply():340\r\nRetrying 2th time\r\n[10:16:28] Karthik Appigatla: …\r\n[10:16:30] Karthik Appigatla: ...\r\n[10:16:31] Karthik Appigatla: Retrying 4th time\r\n2015-04-28 07:05:47 2396 [Note] WSREP: applier has been aborted, skipping apply_rbr: 2146895242\r\n2015-04-28 07:05:47 2396 [ERROR] WSREP: Failed to apply trx: source: 23ba72c9-b808-11e4-9ce7-6e457ca0b86e version: 3 local: 0 state: APPLYING flags: 1 conn_id: 21052413 trx_id: 23224468617 seqnos (l: 1009433886, g: 2146895242, s: 2146895241, d: 2146895171, ts: 17269133574201398)\r\n2015-04-28 07:05:47 2396 [ERROR] WSREP: Failed to apply trx 2146895242 4 times\r\n2015-04-28 07:05:47 2396 [ERROR] WSREP: Node consistency compromized, aborting...\r\n2015-04-28 07:05:47 2396 [ERROR] WSREP: Failed to apply trx: source: 23ba72c9-b808-11e4-9ce7-6e457ca0b86e version: 3 local: 0 state: APPLYING flags: 1 conn_id: 21052413 trx_id: 23224468616 seqnos (l: 1009433885, g: 2146895241, s: 2146895240, d: 2146895171, ts: 17269133572312340)\r\n2015-04-28 07:05:47 2396 [ERROR] WSREP: Failed to apply trx 2146895241 4 times\r\n2015-04-28 07:05:47 2396 [ERROR] WSREP: Node consistency compromized, aborting...\r\n```\r\n\r\nIt seems to me that the sequence of events is as follows:\r\n\r\n1. ALTER errors out before completing on the slave, error is ignored\r\n2. Future replication events use the column that was supposed to be added and fail.\r\n\r\nParallel replication in 16 threads is in effect.\r\n\r\nI grepped the Percona Cluster code a bit and it seems that there are various ways to get ER_QUERY_INTERRUPTED:\r\n\r\n1. If thd->killed is set, and it is set in numerous places in the code, including when threads are created and terminated\r\n2.  If InnoDB internally issues the DB_INTERRUPTED error, which is then converted to ER_QUERY_INTERRUPTED and passed on. DB_INTERRUPTED also seems to happen only if thd->killed is set.\r\n\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/codership/mysql-wsrep/issues/comments/96968365","html_url":"https://github.com/codership/mysql-wsrep/issues/115#issuecomment-96968365","issue_url":"https://api.github.com/repos/codership/mysql-wsrep/issues/115","id":96968365,"user":{"login":"philip-galera","id":8738032,"avatar_url":"https://avatars.githubusercontent.com/u/8738032?v=3","gravatar_id":"","url":"https://api.github.com/users/philip-galera","html_url":"https://github.com/philip-galera","followers_url":"https://api.github.com/users/philip-galera/followers","following_url":"https://api.github.com/users/philip-galera/following{/other_user}","gists_url":"https://api.github.com/users/philip-galera/gists{/gist_id}","starred_url":"https://api.github.com/users/philip-galera/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/philip-galera/subscriptions","organizations_url":"https://api.github.com/users/philip-galera/orgs","repos_url":"https://api.github.com/users/philip-galera/repos","events_url":"https://api.github.com/users/philip-galera/events{/privacy}","received_events_url":"https://api.github.com/users/philip-galera/received_events","type":"User","site_admin":false},"created_at":"2015-04-28T08:11:26Z","updated_at":"2015-04-28T08:11:26Z","body":"One ticket with the same error message, but not on a slave thread and in different circumstances\r\n\r\nhttps://bugs.launchpad.net/codership-mysql/+bug/1375669"}}