I was just working today on a report consisting of a series of queries, taking about a minute to generate. Some of the data would be created in a temporary table and queried against multiple times for performance reasons, and ultimately spit out into a CSV file for someone to examine later. I also would like to be able to return the result set, and perform queries on it, which is much faster than querying a view.

Fortunately, MySQL’s awesome SELECT … INTO OUTFILE can easily write CSV files to disk, so I’m covered there.

So when I started working on this, I was thinking this would be a great opportunity to demonstrate the flexibility and usefulness of stored procedures – after all, we will likely be needing this report again. However, there’s an issue. You can’t return a result set from a stored proc that you can actually do anything with. Everything gets spit directly back to the client. So, what are the alternatives?

Perhaps generate the data into a new table and swap with the old via a RENAME TABLE (if it exists)? Maybe not a bad solution. It works (see example 1 below), but now we’ve coupled the dataset to a global copy of the table, and any references we have to it are now hard coded, rather than passing back a workable dataset we can do anything we want with. Furthermore, if we want to make any changes to the dataset (UPDATE/DELETE) we are unable to – we must make a copy of the table and work from that, unless we don’t mind messing with someone work.

Temporary tables created in a stored procedure for the rest of the connection, so they can’t be used either (example 2), which is somewhat contrary to the manual.

Temporary tables are available past the execution of a stored procedure, contrary to what I wrote earlier. (example 2)

You can use the TEMPORARY keyword when creating a table. A TEMPORARY table is visible only to the current connection, and is dropped automatically when the connection is closed. This means that two different connections can use the same temporary table name without conflicting with each other or with an existing non-TEMPORARY table of the same name. (The existing table is hidden until the temporary table is dropped.) To create temporary tables, you must have the CREATE TEMPORARY TABLES privilege.

Bummer. I’m assuming the stored procedure runs on its’ own isolated connection, which does make sense from a garbage collection point of view.

What about returning the data through the useful OUT parameter? Nope. Since I can’t store more than 1 row in a variable, that’s not happening either (example 3)

You also can’t generate the data in a new table and pass back the table name in an OUT variable. (example 4)

In conclusion: of the above methods, it seems the most likely route I’ll follow in the future will be to store the results in some permanent table, and run the proc off hours to regenerate the table, hoping that no one’s using it. It would need to be read only – and working with it would require creating your own copy of the table. It has the advantage of being easily understood – you can quickly DESCRIBE the columns and know what you’re getting back – so I suppose it’s going to have to be good enough. But what I really wanted was

mysql> call some_proc() into outfile "/tmp/myfile.txt";

For more information on stored procedures, check out the Stored procedure syntax.

Example 1: Swapping tables within a stored procedure

drop procedure test_proc;
delimiter //

CREATE procedure test_proc ()
begin
create table jon_new LIKE jon;
rename table jon to jon_old, jon_new to jon;
drop table jon_old;
end;
//
delimiter ;

mysql> show tables;
+------------------+
| Tables_in_test   |
+------------------+
| jon              |
+------------------+
4 rows in set (0.00 sec)

mysql> insert into jon values (1);
Query OK, 1 row affected (0.04 sec)

mysql> select * from jon;
+----+
| id |
+----+
|  1 |
+----+
1 row in set (0.00 sec)

mysql> call test_proc();
Query OK, 0 rows affected (1.53 sec)

mysql> select * from jon;
Empty set (0.00 sec)

Example 2: Trying to create a temporary table in a stored procedure (updated)

mysql> delimiter //
mysql> create procedure test_proc3()
    -> begin
    -> CREATE TEMPORARY TABLE my_table (id serial);
    -> end
    -> //
Query OK, 0 rows affected (0.00 sec)

mysql> delimiter ;
mysql>
mysql> call test_proc3();
Query OK, 0 rows affected (0.08 sec)

mysql>
mysql> select * from my_table;
Empty set (0.00 sec)
t

Example 3: Trying to store multiple rows in a variable

mysql> create table my_table (id int primary key);
Query OK, 0 rows affected (1.46 sec)

mysql> set @v = (select * from my_table);
Query OK, 0 rows affected (0.01 sec)

mysql> insert into my_table values (1),(2),(3);
Query OK, 3 rows affected (0.03 sec)
Records: 3  Duplicates: 0  Warnings: 0

mysql> set @v = (select * from my_table);
ERROR 1242 (21000): Subquery returns more than 1 row

Example 4: Trying to select from a table where the name was defined in a variable

mysql> set @tab = "my_table";
Query OK, 0 rows affected (0.00 sec)

mysql> select @tab;
+----------+
| @tab     |
+----------+
| my_table |
+----------+
1 row in set (0.00 sec)

mysql> select * from @tab;
ERROR 1064 (42000): You have an error in your SQL syntax;
check the manual that corresponds to your
MySQL server version for the right syntax to use near '@tab' at line 1

Looking for instances particular column in a large schema can be a pain. Fortunately the information schema makes this pretty easy, if your columns have a consistent naming convention.

SELECT table_schema, table_name, column_name
FROM information_schema.columns
WHERE column_name LIKE '%some_name%';

Now, if we want to wrap this up into an easy to use stored procedure, we can do something like this:

drop procedure find_column;
delimiter //
CREATE PROCEDURE find_column(c varchar(255))
begin
	SET @a = CONCAT("%", c, "%");
	SELECT table_schema, table_name, column_name, column_type
		FROM information_schema.columns
		WHERE column_name LIKE @a;
end
//
delimiter ;

We need to use the concat statement in order to properly get the quotes in there without using the literal string “c” in the LIKE statement.

You can do a search as follows:

CALL find_column("some_column");

Learn more on the MySQL Stored Procedures section of mysql.com

Edit: added column type on suggested by Roland Bouman.

Before:

mysql> show engines;
+————+———+———————————————————–+————–+——+————+
| Engine | Support | Comment | Transactions | XA | Savepoints |
+————+———+———————————————————–+————–+——+————+
| CSV | YES | CSV storage engine | NO | NO | NO |
| MRG_MYISAM | YES | Collection of identical MyISAM tables | NO | NO | NO |
| MEMORY | YES | Hash based, stored in memory, useful for temporary tables | NO | NO | NO |
| MyISAM | DEFAULT | Default engine as of MySQL 3.23 with great performance | NO | NO | NO |
+————+———+———————————————————–+————–+——+————+
4 rows in set (0.00 sec)

After:

mysql> show engines;
+————+———+————————————————————+————–+——+————+
| Engine | Support | Comment | Transactions | XA | Savepoints |
+————+———+————————————————————+————–+——+————+
| CSV | YES | CSV storage engine | NO | NO | NO |
| MRG_MYISAM | YES | Collection of identical MyISAM tables | NO | NO | NO |
| MEMORY | YES | Hash based, stored in memory, useful for temporary tables | NO | NO | NO |
| InnoDB | YES | Supports transactions, row-level locking, and foreign keys | YES | YES | YES |
| MyISAM | DEFAULT | Default engine as of MySQL 3.23 with great performance | NO | NO | NO |
+————+———+————————————————————+————–+——+————+
5 rows in set (0.00 sec)

I had to recompile MySQL with the following:

./configure –prefix=/usr/local/mysql –with-plugins=innobase

Really interesting read about how to examine what’s stored in memcached.

Peep uses ptrace to freeze a running memcached server, dump the internal key metadata, and return the server to a running state. If you have a good host ejection mechanism in your client, such as in the Twitter libmemcached builds, you won’t even have to change the production server pool. The instance is not restarted, and no data is lost.

Highlights include scalability improvements, subquery optimizations and join improvements, improved stored procedure management, out parameters in prepared statements, and new information schema additions.

EasyVMX lets you create empty VMWare images. Very useful when coupled with the VMWare player.

I used Jira about 3 years ago at Infosearch Media, which I don’t think exists any more. Anyway, it trounced bugzilla then and still does.

I like this post, it makes a few really cool points. It explains how to get a list of all tables and their engine and how to get a list of foreign keys in your schema. Very cool.

Q4M looks like a pretty cool storage engine that aims to address twitter-like features. I’ll be trying this out for myself in the near future.

Q4M (Queue for MySQL) is a message queue licensed under GPL that works as a pluggable storage engine of MySQL 5.1, designed to be robust, fast, flexible. It is already in production quality, and is used by several web services (see Users of Q4M).

This will give you a rough idea of your innodb memory usage. I know it’s in a hundred spots, but i hate looking for it when i double check things.

innodb_buffer_pool_size
+ key_buffer_size
+ max_connections*(sort_buffer_size+read_buffer_size+binlog_cache_size +2MB)

I had to convert the second table to match the character set. This probably wouldn’t have been a problem if I wasn’t joining on a character field.

alter table exclusion CONVERT TO CHARACTER SET utf8;

First time I’ve ever seen that one.

map <C-d> :call SwitchDB()<CR>
:function SwitchDB()
: let g:current_db = input("Database > ")
:endfunction

map <C-m> :call Doquery()<CR>
:function Doquery()
: if !exists("g:current_db")
: call SwitchDB()
: endif
: let query_string = input(g:current_db . " > " )
: if query_string != ""
: exe "!mysql " . g:current_db . " -e \"" . escape(query_string, '"') . "\""
: endif
:endfunction


Control-m to execute a query. Control-d to switch databases. It’ll prompt you the first time.

What is a materialized view?

A materialized view is a database object that contains the results of a query. The FROM clause of the query can name tables, views, and other materialized views. (from Oracle).

Essentially a materialized view lets you tell the database to periodically refresh a table with the results of a query. You may join, group, and perform calculations. The goal is to increase query performance in a read-heavy environment.

Additionally, at least in the other DBs listed, you can put indexes on the fields within the view.

Disadvantages:

• There can be issues with altering underlying tables, just as if you were to remove a column that a standard view references.
• If you’re inserting and updating into the base tables frequently, you will either see a performance hit or have to deal with stale data.
• Since the data is actually stored as a table on disk, it can take up considerable space

According to the high level architecture

Support creation of materialized views, with only the bare minimum — no automatic refresh, no query rewrite.

Which would be a shame, because it seems that adding in automatic refresh would be a pretty small part of a very complex feature. I haven’t dealt with any of the MySQL codebase so this is just my speculation.

Of course, when looking at any feature, it’s important to determine how useful it actually would be to implement. What is the target audience, and are they the existing customers? What’s the goal of adding this feature?

For ther reading, Database journal has a very good overview of indexed views in SQL Server.

Kristian Koehntopp gives a good example:

Kristian Koehntopp writes:

TDS: Time delayed SQL_THREAD (Have a replication slave that is always
lagging 30 minutes behind).

Currently, replication is a rolling recovery: To set up replication you
restore from a full dump with a binlog position. You then continously
download binlog and roll forward. In case of a master crash a slave is a
readily recovered instance (as opposed to a backup, which still has to be
restored).

This protects against crashes, but not against oopses.

A time delayed slave (TDS) is a nice protection against oopses.

Sugar on top addendum: With the binlog being a table, any table type,
it will be very easy to delete an oopsing statement out of the
unapplied binlog queue. Currently, a simple time delayed slave will
protect you against oopses, but it will be very hard to extract the
relevant binlog portion out of the TDS replication log skipping the
oopsing statement.

With a MyISAM binlog table, it is just a matter of DELETE FROM
REPLICATION.BINLOG WHERE STATEMENT_ID = 1717;

Definately a cool idea. Instead of having to restore from a backup, you just take down your master, pull out the busted query, let the server catch up, and bring the slave up as the new master. Doing a restore from our current database backups takes about 3-4 hours, and we’re not even huge. It seems like your downtime here would be limited to however long it takes your slave to catch up. Additionally, once you take down your master and remove your bogus query (truncate table perhaps?) you can allow the slave to replicate everything and catch up (no longer limited to 30 min behind, ideally). Depending on your traffic, this could limit your downtime in the case of a catastrophic loss to 20 minutes or so instead of hours, and your data loss would be far less.