While we take every precaution to avoid crashes, regrettably they still happen. In the worst case scenario, they cause progress to be lost on accounts.
Yesterday at around 11pm GMT, one of these crashes occurred, causing up to 4 hours worth of data loss. This is pretty much the worst case scenario, so I apologize profusely for this, and am attempting to manually restore data where possible. If you've been effected by this - which is likely if you were playing between 6pm and 10pm server time on the 12th August, please contact me on Discord or by emailing [email protected]
While the server normally crashes due to a bug in the code (an "unhandled exception"), it can also crash when the virtual machine the server runs on exhausts its resources (usually RAM) which causes the program to shut down in an uncontrolled manner.
This can cause data loss as the server saves data to the database in a "non-blocking" process - this essentially means the server puts data it wants to save in a queue for the database to handle when it has time. The server doesn't wait for this queue to be processed, which means if the server crashes when this queue still has data to be saved in it, that data is lost.
The database is optimized to ensure this queue never gets too big, but if there is some external stress on the server that causes the database to slow down (previously we have had this caused by cyber attacks on the database, but it can be caused by issues with the data center where the server is located or during intensely busy server times) this queue can get quite large and if there is a crash (sometimes caused by the queue itself becoming so large it consumes all the available RAM) then data loss can be quite damaging.
While having your hard work undone is understandably devastating and alienating to players, I have assigned everyone extra construction, research and troop speedups, resource packages and mayor XP / buff items to give you the best chance of repairing the damage caused. I have also upgraded buildings in satellites that had automation set up to the maximum level. If you would like anything in particular restored, please do not hesitate to contact me.
While avoiding crashes completely isn't possible, reducing the likelihood of data loss is, and I apologize for not acting on that before, since this isn't the first time this has happened.
The server now routinely checks the database queue, and will enter a controlled shutdown if it detects it becoming too large, where it will process every item in the queue to ensure no data is lost, before restarting.
If something like this happens again, the database may start to repeatedly have outages due to this mechanism, so an underlying cause needs to be found and mitigated against, but it should limit progress loss - which is the worst case scenario.