Bad Web Developers Don’t Scale
September 11, 2009 – 4:30 pmOver the last 5 years, I’ve read so many articles about how X doesn’t scale. PHP, MySQL, SQL Server, Apache, you name it – everything gets a bad rap. Everyone has a different idea of scale and size, and sadly most people think their site with 1 million page views a month can’t handle the load because whatever technology they chose to use supposedly doesn’t scale.
Guess what folks – the problem probably isn’t the language you chose, or the database you’re using. Unless you’re actually big (many millions of pageviews per day), you can usually run just fine with a straight up LAMP stack on a few servers. The real issue is that your dev team has no idea how to write software or use the tools they have available.
At this point, I wonder – what is everyone’s expectation? At what point does someone say “oh, X scales great! One box is handling my web server, data storage, video conversion, at 20 million page views a day, all running on 1 nice little application!” Folks – pick the right tool for the job, and put someone qualified in charge of it. Don’t use MySQL if you really need in memory graph tree traversal (friends list).
Oh, and don’t even bother explaining to me the definition of scale. This isn’t an opportunity for you to point out what it really means – I get that you should be able to add hardware and get a proportional amount out extra output. Writing software that scales to a massive level is hard. That’s why there aren’t many companies that do it well.
/rant
4 Responses to “Bad Web Developers Don’t Scale”
Amen.
By Jeff Johnson on Sep 11, 2009
One of the services my department offers our dedicated hosting customers is effectively ’sysadmin for hire’.
A fair number of the requests revolve around “Why isn’t MySQL performing well. What’s wrong with MySQL.”
So tempting to reply “Nothing is wrong with MySQL, go learn to write and design databases and queries fit for purpose.”
As a department we don’t do coding, but still like any sysadmin task it’s necessary for figuring out what is going on. In one particular customers case their database was using up almost always 100% CPU time even when load was light, let alone when load was heavy. Logging “Queries without indexes” was shocking, but worse still the most common query involved some very memory intensive SORT, JOIN & GROUP BY processes. Even with just one or two other queries running it was taking 5 seconds to process. It really didn’t help that the bulk of their data was in a single table, 1.5m rows x 140 columns, rather than partitioned logically.
Just by breaking the query down into three or four separate ones I came up with a much, much faster way to achieve the same result in <0.5 seconds. I carefully presented the results to the customer, explained why MySQL was running slow, explained how they could structure their queries and data better.
The response? “It works fast on this other site, why is it slow here?” We’ll just ignore how the other site was smaller, completely different data set (each row was so much smaller)
Even after careful explanation, instead of altering their code or database design they decided to fork out a few hundred dollars extra a month to get a dedicated high powered database server. The code still is ugly, but hey at least the db server has the horsepower to handle it.
By Twirrim on Sep 11, 2009
True. Some people have no idea how to get their application scale well. The server, technology and the customer are potential blame targets for these folks.
By Sudheer on Sep 11, 2009
Ah. The good old ‘my code is impeccable’ argument.
As a software developer I know the impact poorly written queries can have on the scalability of a project. However, I do know a lot of them just don’t get the big picture.
My company also provides outsourcing to its clients. We have one particular piece of software which runs dramatically slow on our servers. The creators ofcourse point their fingers to us, because ‘it runs fine over there’. ‘Over there’ serves about 2% of the concurrent users ‘over here’ serves… Even though we have proof that their app is poorly written, we can’t convince them to correct the situation.
By uniekje on Sep 14, 2009