Migrating Sidekiq Redis Data to AWS ElastiCache

Originally written over two afternoons, August 20–21, 2024.

I recently worked on migrating our main application to Amazon ECS. One of the tasks was moving data used by Sidekiq, a background job processing system that can handle many jobs concurrently in the same process, from a Redis instance on Aptible to a new Redis instance in Amazon ElastiCache.

Sidekiq uses several queues and keys to store job data. The key we care about here is schedule, because it contains many jobs that will run in the future and later send messages to our users. We don't want to lose those jobs, so we need to copy the key to the new Redis instance. Once you know what to do, the process is relatively simple:

Dump the Sidekiq schedule key from Redis on Aptible.
Restore that dump into Redis on Amazon ElastiCache.

There is one complication: our ElastiCache instance only accepts connections from within AWS. We will create an SSH tunnel through an EC2 instance so we can reach ElastiCache from the computer performing the migration.

Dump the old Redis key

We will use redis-cli. On Debian GNU/Linux, it is available in the redis-tools package:

apt-get install redis-tools

Set environment variables for both Redis hosts. Keep real credentials out of shell history and source control:

OLD_REDIS_HOST=redis-aptible.aptible.com
OLD_REDIS_PORT=6379
NEW_REDIS_HOST=redis-elasticache.amazon.com

Dump the namespaced Sidekiq schedule key to a local file:

redis-cli -h "$OLD_REDIS_HOST" \
  -p "$OLD_REDIS_PORT" \
  --pass "$OLD_REDIS_PASSWORD" \
  DUMP 'my_namespace:schedule' > redis-dump.rdb

This creates redis-dump.rdb. RDB is one of the formats Redis uses for persistence. You can read more in the Redis persistence documentation.

Create the SSH tunnel

ssh -i ~/.ssh/my-key.pem -f -N \
  -L 6380:"$NEW_REDIS_HOST":6379 \
  ec2-user@"$EC2_INSTANCE_IP"

The tunnel exposes ElastiCache locally on port 6380. Now try to restore the dump:

head -c-1 redis-dump.rdb | \
  redis-cli -h localhost -p 6380 -x \
  RESTORE 'cool_namespace:schedule' 0

(error) BUSYKEY Target key name already exists.

The key already contains data, so Redis refuses to replace it. We can inspect the existing values with ZRANGE. First, confirm that the key is a sorted set:

redis-cli -h localhost -p 6380 TYPE 'cool_namespace:schedule'

# zset

redis-cli -h localhost -p 6380 ZRANGE 'cool_namespace:schedule' 0 -1

# "{\"class\":\"RefreshTokenWorker\",\"args\":[1053],...}"

In our case, this is a new server and the existing values are only test data, so we don't need to keep them.

Warning: the next command is destructive. Verify the host, port, and key carefully before running it.

redis-cli -h localhost -p 6380 DEL 'cool_namespace:schedule'

# (integer) 1

Confirm that the key is now empty:

redis-cli -h localhost -p 6380 ZRANGE 'cool_namespace:schedule' 0 -1

# (empty array)

Restore and verify the dump

We can now restore the dump created at the beginning:

head -c-1 redis-dump.rdb | \
  redis-cli -h localhost -p 6380 -x \
  RESTORE 'cool_namespace:schedule' 0

OK

Finally, verify the restored schedule:

redis-cli -h localhost -p 6380 ZRANGE 'cool_namespace:schedule' 0 -1

# 1) "{\"class\":\"RefreshTokenWorker\",\"args\":[1053],...}"

You should see the complete contents of the sorted set. One important detail from the Redis documentation is that RESTORE checks the RDB version and checksum and returns an error if they don't match. The source and destination Redis versions therefore need compatible RDB formats.

Redis commands used

TYPE: returns the data type stored at a key.
ZRANGE: returns a range of members from a sorted set.
DEL: deletes a key.
DUMP: serializes the value stored at a key.
RESTORE: creates a key from a serialized value.

That's all, friends. Thanks for reading :)
Gramos.

 ::: If you'd like to comment, email me: ramos.gaston AT gmail.com :::

Gastón Ramos

Personal blog