MemSQL database recovery
<p>We have single node MemSQL community edition running in production though MemSQL single node is not recommended for production use , we started as POC and deployed the same to Prod </p>

<p>Today we ran into the below issue  ,</p>

<p>(stderr) ERROR 1720 (HY000) at line 3: Leaf Error (10.0.0.28:3307): Memory usage by MemSQL for tables (26255 MB) has reached the value of 'maximum_table_memory' global variable (26064 MB). This query cannot be executed.</p>

<p>And we tried to delete some data and we were not able to do so because of the below exception 
Error Code: 1712. Leaf Error (10.0.0.28:3307): Not enough memory available to complete the current request. The request was not processed.    46.836 sec</p>

<p>Then we doubled the RAM on the machine and restarted memsql , but leaf didn't picked up the additional memory , then I have changed the memory setting at memsql.cnf ( /var/lib/memsql/leaf-3307/ ) and restarted the leaf node and then the leaf picked up the additional memory </p>

<p>Ideally memory shouldn't be hard coded , instead it should be percentage of memory available over the machine and after restart it should pick the additional memory </p>

<p>When leaf picked up the additional memory , we started encountering different set of issues </p>

<p>We were continuously getting the below 2 exceptions and as and when we are able to connect and restart the application (ETL) , we were again running into the same problem , we tried 10 times and we have no clue what's happening , we tried restarting memsql , we tried rebalancing paritions (we know it doesn't work but still) , we tried repairing database but the DB is in online mode , when the leaf picked up additional memory after bumping the machine config (AWS) , we purged (deleted) some data from one of the table , so we thought this might be creating problem and recreated the table but still no luck </p>

<p>ERROR 1731 (HY000): The database 'reports_and_summary' will be available to query in 35 seconds after recovery from disk is finished. Run SHOW DATABASES EXTENDED and visit <a href="http://docs.memsql.com/5.5/concepts/database#states" rel="nofollow noreferrer">http://docs.memsql.com/5.5/concepts/database#states</a> for more information</p>

<p>"Primary key recovery of database memsql"</p>

<p>Finally the only thing which worked is , we upgraded the memsql version and I think it did a clean install again and it started working , but what if no new version is available for upgrade :) </p>

<p>Did any one faced similar issue and what can be the root cause ? </p>