|
A long day
Feeling good yesterday as I was figuring out more about Groovy, the scripting language used by zOMG. I woke up and was ready to see people tackling the Gnome General in Barton Town, but then all the servers had the "[no event]" flag on...
Then came the bad news.
Our logging servers were severely lagged. Main problem being the replication log server who had trouble backing up data due to an excessive amount of database writes. The problem caused errors all over the sites, as most features ended up writing logs in one way or another. Many features, including the Easter event, was turned off due to the error.
"What's going on???"
Upon closer inspection, we found that there were 5 times the activity on this one database for handling transactions. We were getting a TON of gold grants for this one application labeled... 737, which is Mini Games!
We initially thought that there were some errors in the coding, so we kept testing and debugging, but nothing suspicious was found. Then we started parsing the log to further investigate...
80% of all gold grants were generated by the same group of users.
"Yup, they're bots."
"This is why we can't have nice things" - James
As the minigame was brand new, it didn't went through a full security audit and was prone to botting. What followed was a lot of repair work, mainly because a lagging server works even slower than normal, and we need to minimize the amount of work the systems are placing on it.
We ended up doing a big search on ALL the logging functions to find what are things that are unnecessary, and discovering functions that were written years ago but we've never used since then. I guess if we had to do house cleaning, this is just as good a reason as any to get things done.
The backup server was 11 hours behind schedule when we started. Hopefully it'll catch up by later this morning and everything can be turned back on.
|
