How to migrate database on multi-node production server?
<p>I have a multi-node production server (Tomcat 8.x on CentOS 7.x; each node is a separate CentOS instance), that uses a single MySQL database server (MySQL 5.7.x). Each node of the server will be updated manually: system administrator stops each node and deploys a new version of the application (<code>.war</code> file). It means that the service won't have downtime, because at every moment there is at least one working node.</p>

<p>Database migrations are implemented using Liquibase changesets, which are placed in the <code>.war</code> file. So each node validates and updates (if requires) the database schema. Actually, only the first node executes changesets and other nodes just validate it.</p>

<p>The problem is that there is a time gap between updates of each node: when the first node is already updated with the new application version, the last node still works on the previous application version (that might use old columns in database for example). It might lead to inconsistency of the database.</p>

<p><strong>Example</strong></p>

<p>Let's say the server has 3 nodes. At this moment they work on an application of version N. 
Next releases need to change a database schema: rename a column <code>title</code> to <code>title_new</code>. </p>

<p>To make it possible to update database schema without downtime, we need to use "two-steps change": </p>

<ul>
<li>version N+1:

<ul>
<li>adds a new column <code>title_new</code>, </li>
<li>doesn't use a column <code>title</code> anymore (it's marked as deprecated);</li>
<li>copy all data from the column <code>title</code> to <code>title_new</code>;</li>
<li>uses a column <code>title_new</code>;</li>
</ul></li>
<li>version N+2 drops a column <code>title</code>.</li>
</ul>

<p>Now administrator is going to deploy version N+1. He stops the first node for update, but the other two nodes are still working on the version N. While the first node is updating, some users might change their data using node 2 or 3 (there is a load balancer, that routes requests to different nodes). So we need a way to forbid users to make any changes via nodes 2 and 3, while they are not updated with a version N+1.</p>

<p>I see two different ways to solve this problem:</p>

<ol>
<li>Use some <code>read_only</code> mode on the application level - then the application logic forbids users to make any changes. But then we need to implement some ways to enable this mode at any time using a console or admin panel (administrator must be allowed to enable this mode).</li>
<li>Use <code>read_only</code> mode on database level. But I couldn't find any ready-for-use methods for MySQL to do this.</li>
</ol>

<p><strong>The question:</strong> what's the best way to solve the described issue?</p>

<p>P.S. Application is based on the Spring 4.x framework + Hibernate 4.x.</p>