MCStats has used PHP to generate graphs since it was created. The first “backend” (the endpoint servers send data to) was also in PHP, but it was soon written in Java once a need for caching in-process was desperately needed.
For the past few months graph generation has been taking longer and longer. It got to the point of taking ~15 minutes to generate all of the graphs for every plugin. For the graphs to even generate the backend needs to keep MySQL up to date. Keeping it very up to date requires some 2,000+ queries/sec constantly. In a single machine and keeping MySQL small this destroys MySQL and response times were through the roof as the queries done to aggregate data for the graphs themselves were expensive and took even longer with the massive amount of queries the backend was doing.
So what what was the main step towards fixing this?