As of Friday 6th of August at around 6PM (EST+10) we suffered a catastrophic fatal hardware failure in our bare metal dedicated machine, in which the entire machine was rendered inoperable, and unfortunately the hard drive(s) that we had used for both the CDN and Database, had also completely died with no chance for data recovery.
The backups that were ran and had completed were producing similarly malformed/corrupted backups, due to the I/O errors from the hard-drive degradation, and the only backups that are not completely affected are the ones just before the complete degradation.
As someone in IT field, hindsight is 20/20, and I noticed all the signs of hardware failure as the descent was over roughly a year(the frequent disconnects, aberrant behavior such as server lag during backups and during random times, events failing to start and why the gigabit connection was not indeed gigabit despite being reported as such by all metrics, but was just ignorant of these signs, and for that I am vehemently sorry.
Moving forward, I will actually inspect the viability of backup instead of just checking that they completed.
This is a very heavy blow to both us as a community and us as team.
We said we won't wipe, and we're committed to that promise even though we're at a junction where that might be the case for some of you.
As our entire machine was nonfunctioning, we now have acquired another server machine, which is massively upgraded compared to the previous machine, we are no longer using hybrid drives, but actual solid state drives, so that's a massive improvement on it's own as this is a new machine, we have a new network address, which means we have to change quite a lot on our end, which will require a complete redownload of both the launcher and client, and all our old clients, will no longer function.
What was lost?
Database from 30/11/2020 to 06/08/2021
Our PATCH CDN Downloads (Allowing you to download patches)
Our Network infrastructure (So you will be required to redownload)
What are the plans going forward?
Any and all URP purchases between the window indicated above will be redistributed, we have all transactions catalogued.
As this occurred during our Goodbye 2020 promotion, everyone will be receiving that reward, the upper limit of one per account still in effect as it is deposited to the accounts cash shop inventory.
All accounts effected by this will be able to post in the Disaster Recovery thread found at this link;
Not everything can be returned unfortunately, and what can be returned has an upper limit. which will be explained and how we would you like you submit what you have lost in the thread.
I plan to bring just the account server and character server online, for people to create their accounts and characters if they have been affected by the data loss, in which you can then properly send the recovery forms, why I'm doing is will be explained below.
I'm still working on my patch, which was originally going to be just small group of bugfixes, but it will be a full content and gameplay update patch, which would have also required a redownload, this has regrettably been also delayed by this unfortunate setback, no data for the patch has been lost as we keep that elsewhere, just the recovery efforts are also being led by me, as well as the patch is being also led by me, as such we won't be bringing the server back online until the patch is ready, so you don't have to redownload multiple times, and or waste any in game time.
TL;DR(Too Long Didn't Read):
Our host's dedicated machine completely died last Friday.
Lost all of this years data, due to corrupt backups.
Will be offering account/character recovery.
URP will be redistributed to those affected.
A complete redownload will be required.
Server won't be brought back online until the patch is done.