Monthly Archives: January 2016

Let’s talk about downtime

The last few days

After moving everything to a new server, it was discovered that there was corrupted value in a collection whenever Mongo tried to scan it which caused mongod to crash. This was found when backups of the database were being ran. Since the corrupted value wasn’t touched from normal usage (opening graphs on the website), it’s likely that the corrupted value is some piece of graph data that’s over a month old. There’s over a year’s worth of graph data stored in Mongo currently.

Continue reading