Comments on: What caused that load spike? http://www.rustyrazorblade.com/2007/02/what-caused-that-load-spike/ Tech Thoughts, Mostly on LAMP - by Jon Haddad Fri, 03 Sep 2010 03:50:22 -0700 http://wordpress.org/?v=2.8.4 hourly 1 By: jon http://www.rustyrazorblade.com/2007/02/what-caused-that-load-spike/comment-page-1/#comment-15369 jon Sun, 24 Feb 2008 05:23:28 +0000 http://www.rustyrazorblade.com/index.php/2007/02/13/what-caused-that-load-spike/#comment-15369 Hey Erik, We ended up finding a few queries that were causing several simultaneous full table scans. Other queries that would normally have been very quick (1 row, based on primary key) would backlog. Due to the number of mysql connections (I think it was around 700) it was difficult to pin down the query in question, but eventually used the mysqldumpslow tool on the slow query log to find the cause. Hey Erik,
We ended up finding a few queries that were causing several simultaneous full table scans. Other queries that would normally have been very quick (1 row, based on primary key) would backlog. Due to the number of mysql connections (I think it was around 700) it was difficult to pin down the query in question, but eventually used the mysqldumpslow tool on the slow query log to find the cause.

]]> By: Erik http://www.rustyrazorblade.com/2007/02/what-caused-that-load-spike/comment-page-1/#comment-15361 Erik Sat, 23 Feb 2008 23:36:33 +0000 http://www.rustyrazorblade.com/index.php/2007/02/13/what-caused-that-load-spike/#comment-15361 May be a bit late, and I'm guessing you've sorted this by now. But hey, someone might stumble upon it and find it useful. mod_log_forensic is very useful for tracking down web requests taking their sweet time. If they end up crashing, mod_whatkilledus might be of help as well. As for the database server, you could run `iostat` to see if the disks are working overly much or `vmstat` to see if you're swapping a lot (horrendous for a DB). Most likely the apache threads are a knock-on effect of the DB being locked up. Could also have been corrupt tables or indices. But most likely it's a locking issue, difficult to say without the queries or knowing what storage engines etc are involved. If you ever figured out what it was, I'd be interested in hearing what it was! Good blog btw! Erik May be a bit late, and I’m guessing you’ve sorted this by now. But hey, someone might stumble upon it and find it useful. mod_log_forensic is very useful for tracking down web requests taking their sweet time. If they end up crashing, mod_whatkilledus might be of help as well.
As for the database server, you could run `iostat` to see if the disks are working overly much or `vmstat` to see if you’re swapping a lot (horrendous for a DB). Most likely the apache threads are a knock-on effect of the DB being locked up. Could also have been corrupt tables or indices. But most likely it’s a locking issue, difficult to say without the queries or knowing what storage engines etc are involved.

If you ever figured out what it was, I’d be interested in hearing what it was!
Good blog btw!

Erik

]]>