1 billion documents

The graph database for MCStats (using MongoDB) now contains over 1 billion documents. The 1 billionth document was inserted on January 29, 2016!

I’ve been working on getting the data into Google BigQuery so that I can play around with the data a lot more than I can with MongoDB and after a day of massaging the data I was able to load it successfully!

BigQuery Table

Loading data is a very much manual process. The data covers 2014-11-13 to now. November, 2014 is when I started keeping _all_ data stored again after moving away from Digital Ocean to elsewhere where it is more feasible to keep all data instead of purging.

Leave a Reply

Your email address will not be published. Required fields are marked *