Pages

Monday, April 22, 2013

Copying the last rows from a large remote table into the local database

I recently ran into a problem where I needed to copy a subset of the rows in a huge remote table to the local database.  This is the kind of thing best suited for mysqldump paired with mysql, but there's no parameter that lets you export only the last n rows of the table.  There is however a WHERE parameter (--where, -w) and apparently mysqldump does not protect against SQL injection.  So the solution looks like this:

Warning!  This will drop and re-create your local table, so if you have data in it you need to keep then you need to look into the flags to skip the drop and create statements.

mysqldump -u REMOTE_USER -pREMOTE_PW -h REMOTE_HOST --compress --where "true ORDER BY id DESC LIMIT 0, N" REMOTE_DB REMOTE_TABLE | mysql -u LOCAL_USER -pLOCAL_PW -h LOCAL_HOST LOCAL_DB
Make sure you replace the values in yellow with those appropriate for your situation.
Notes:

  • N should be the numeric number of rows you want to copy (ex. 100000)
  • There is no space between the -p and the password, but there is a space between most other options and their values
  • If your password contains special characters, make sure to escape them
  • If your local host is actually the same machine you are running the command on, you can skip -h LOCAL_HOST.  In my case it wasn't because my host operates separate MySQL and web servers (I was SSHed into the web server)
  • You can skip the ORDER BY id DESC (or replace the sorting order with whatever fits you) as needed
Why does this work?  We're basically doing SQL injection by causing mysqldump to create select statements like:
SELECT ...
FROM ...
WHERE true
ORDER BY id DESC
LIMIT 0, 100000
The whole value of the yellow text is coming from our --where value.