{"action":"opened","issue":{"url":"https://api.github.com/repos/2ndQuadrant/repmgr/issues/103","labels_url":"https://api.github.com/repos/2ndQuadrant/repmgr/issues/103/labels{/name}","comments_url":"https://api.github.com/repos/2ndQuadrant/repmgr/issues/103/comments","events_url":"https://api.github.com/repos/2ndQuadrant/repmgr/issues/103/events","html_url":"https://github.com/2ndQuadrant/repmgr/issues/103","id":105312448,"number":103,"title":"Race condition with repmgrd after converting failed master to standby","user":{"login":"soxwellfb","id":12963638,"avatar_url":"https://avatars.githubusercontent.com/u/12963638?v=3","gravatar_id":"","url":"https://api.github.com/users/soxwellfb","html_url":"https://github.com/soxwellfb","followers_url":"https://api.github.com/users/soxwellfb/followers","following_url":"https://api.github.com/users/soxwellfb/following{/other_user}","gists_url":"https://api.github.com/users/soxwellfb/gists{/gist_id}","starred_url":"https://api.github.com/users/soxwellfb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/soxwellfb/subscriptions","organizations_url":"https://api.github.com/users/soxwellfb/orgs","repos_url":"https://api.github.com/users/soxwellfb/repos","events_url":"https://api.github.com/users/soxwellfb/events{/privacy}","received_events_url":"https://api.github.com/users/soxwellfb/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-09-08T06:47:55Z","updated_at":"2015-09-08T06:47:55Z","closed_at":null,"body":"Hi,\r\n\r\nI've got repmgr working with a master, standby and witness server, and when the master fails, I have automatic failover with repmgrd.\r\n\r\nThe issue I'm seeing comes when converting the old master database (db01.server) into a standby to reestablish the cluster.\r\n\r\nI'm running the following in a script on db01.server, where db02.server is the new master:\r\n\r\n```\r\nrepmgr --config-file /etc/repmgr/repmgr.conf --force --rsync-only \\\r\n      --host db02.server --dbname rempmgr  \\\r\n      --username repmgr --verbose standby clone\r\n\r\nservice postgresql-9.4 start\r\n\r\nrepmgr --config-file /etc/repmgr/repmgr.conf --force register standby\r\n\r\nrepmgrd -f /etc/repmgr/repmgr.conf -d\r\n```\r\nBut I'm finding in the repmgrd logs it's thinking it's a master:\r\n\r\n```\r\n[2015-09-08 15:21:03] [INFO] connected to database, checking its state\r\n[2015-09-08 15:21:03] [DEBUG] get_node_info(): SELECT id, upstream_node_id, conninfo, type, slot_name, active   FROM \"repmgr_saas_cluster\".repl_nodes  WHERE cluster = 'saas_cluster'    AND id = 1\r\n[2015-09-08 15:21:03] [DEBUG] node id is 1, upstream is 0\r\n[2015-09-08 15:21:03] [INFO] checking cluster configuration with schema 'repmgr_saas_cluster'\r\n[2015-09-08 15:21:03] [INFO] checking node 1 in cluster 'saas_cluster'\r\n[2015-09-08 15:21:03] [INFO] reloading configuration file and updating repmgr tables\r\n[2015-09-08 15:21:03] [DEBUG] reload_config(): configuration has not changed\r\n[2015-09-08 15:21:03] [WARNING] Unable to create event record: ERROR:  cannot execute INSERT in a read-only transaction\r\n\r\n[2015-09-08 15:21:03] [INFO] starting continuous master connection check\r\n```\r\nIt's even trying to insert an event record into the standby database. If I restart repmgrd, I get the correct startup:\r\n\r\n```\r\n[2015-09-08 15:27:44] [INFO] connected to database, checking its state\r\n[2015-09-08 15:27:44] [DEBUG] get_node_info(): SELECT id, upstream_node_id, conninfo, type, slot_name, active   FROM \"repmgr_saas_cluster\".repl_nodes  WHERE cluster = 'saas_cluster'    AND id = 1\r\n[2015-09-08 15:27:44] [DEBUG] node id is 1, upstream is 2\r\n[2015-09-08 15:27:44] [INFO] connecting to master node 'saas_cluster'\r\n[2015-09-08 15:27:44] [INFO] finding node list for cluster 'saas_cluster'\r\n[2015-09-08 15:27:44] [INFO] checking role of cluster node '2'\r\n[2015-09-08 15:27:44] [DEBUG] connecting to: 'host=db02.server port=5432 dbname=repmgr user=repmgr fallback_application_name='repmgr''\r\n[2015-09-08 15:27:44] [DEBUG] get_master_connection(): current master node is 2\r\n[2015-09-08 15:27:44] [INFO] checking cluster configuration with schema 'repmgr_saas_cluster'\r\n[2015-09-08 15:27:44] [INFO] checking node 1 in cluster 'saas_cluster'\r\n[2015-09-08 15:27:44] [INFO] reloading configuration file and updating repmgr tables\r\n[2015-09-08 15:27:44] [DEBUG] reload_config(): configuration has not changed\r\n[2015-09-08 15:27:44] [DEBUG] Event timestamp is: 2015-09-08 15:27:44.504539+10\r\n[2015-09-08 15:27:44] [INFO] starting continuous standby node monitoring\r\n```\r\n\r\nI inserted a few `psql -c \"SELECT id, upstream_node_id, conninfo, type, slot_name, active   FROM \\\"repmgr_saas_cluster\\\".repl_nodes  WHERE cluster = 'saas_cluster'\"` commands into the script:\r\n* after starting postgres\r\n* after registering as a standby\r\n* after starting repmgrd\r\n\r\nAnd got the following:\r\nAfter postgresql start:\r\n```\r\n id | upstream_node_id |                                 conninfo                                 |  type   |   slot_name   | active\r\n----+------------------+--------------------------------------------------------------------------+---------+---------------+--------\r\n  3 |                  | host=db03.server port=5432 dbname=repmgr user=repmgr | witness |               | t\r\n  1 |                  | host=db01.server port=5432 dbname=repmgr user=repmgr | master  | repmgr_slot_1 | f\r\n  2 |                  | host=db02.server port=5432 dbname=repmgr user=repmgr | master  | repmgr_slot_2 | t\r\n(3 rows)\r\n```\r\n\r\nAfter registering standby server:\r\n```\r\n id | upstream_node_id |                                 conninfo                                 |  type   |   slot_name   | active\r\n----+------------------+--------------------------------------------------------------------------+---------+---------------+--------\r\n  3 |                  | host=db03.serverport=5432 dbname=repmgr user=repmgr | witness |               | t\r\n  1 |                  | host=db01.server port=5432 dbname=repmgr user=repmgr | master  | repmgr_slot_1 | f\r\n  2 |                  | host=db02.server port=5432 dbname=repmgr user=repmgr | master  | repmgr_slot_2 | t\r\n(3 rows)\r\n```\r\n\r\nAfter starting repmgrd:\r\n```\r\n id | upstream_node_id |                                 conninfo                                 |  type   |   slot_name   | active\r\n----+------------------+--------------------------------------------------------------------------+---------+---------------+--------\r\n  3 |                  | host=db03.server port=5432 dbname=repmgr user=repmgr | witness |               | t\r\n  2 |                  | host=db02.server port=5432 dbname=repmgr user=repmgr | master  | repmgr_slot_2 | t\r\n  1 |                2 | host=db01.server port=5432 dbname=repmgr user=repmgr | standby | repmgr_slot_1 | t\r\n(3 rows)\r\n```\r\n\r\nWhat I think I'm seeing is a race condition caused by a delay in replicating the `repl_nodes` table from the master to the standby.\r\n\r\nI see one glaring problem is that repmgrd is quite happy to think that a node that not marked as active is a master db, but I think the best solution (assuming it is replication delay) would be for `repmgr standby register` to block until the correct status is reflected in the local copy of `repl_nodes`.\r\n\r\nOtherwise, apart from a strategic `sleep 5` in my shell script, can I workaround be suggested?\r\n\r\nThanks."}}