As a software developer I know the impact poorly written queries can have on the scalability of a project. However, I do know a lot of them just don’t get the big picture.

My company also provides outsourcing to its clients. We have one particular piece of software which runs dramatically slow on our servers. The creators ofcourse point their fingers to us, because ‘it runs fine over there’. ‘Over there’ serves about 2% of the concurrent users ‘over here’ serves… Even though we have proof that their app is poorly written, we can’t convince them to correct the situation.

So tempting to reply “Nothing is wrong with MySQL, go learn to write and design databases and queries fit for purpose.”

As a department we don’t do coding, but still like any sysadmin task it’s necessary for figuring out what is going on. In one particular customers case their database was using up almost always 100% CPU time even when load was light, let alone when load was heavy. Logging “Queries without indexes” was shocking, but worse still the most common query involved some very memory intensive SORT, JOIN & GROUP BY processes. Even with just one or two other queries running it was taking 5 seconds to process. It really didn’t help that the bulk of their data was in a single table, 1.5m rows x 140 columns, rather than partitioned logically.

Just by breaking the query down into three or four separate ones I came up with a much, much faster way to achieve the same result in <0.5 seconds. I carefully presented the results to the customer, explained why MySQL was running slow, explained how they could structure their queries and data better.

The response? “It works fast on this other site, why is it slow here?” We’ll just ignore how the other site was smaller, completely different data set (each row was so much smaller)

Even after careful explanation, instead of altering their code or database design they decided to fork out a few hundred dollars extra a month to get a dedicated high powered database server. The code still is ugly, but hey at least the db server has the horsepower to handle it.