Mark Atwood (fallenpegasus) wrote,
Mark Atwood

An idea for Amazon AWS S3 and MySQL cluster replication distribution

I've described this idea to a few people, but I figured I would post it here.

I've had an idea for using Amazon AWS S3 to distribute MySQL cluster replication data.

The existing architecture for MySQL clustering is as follows:
  1. The master has N slaves
  2. The master copies each binlog replication to each slave. If there are 7 slaves, then the master has to push the same data out it's network pipe 7 times
  3. The slave has a hot TCP connection to the master. The master has 7 hot TCP connections, one for each salve.
  4. The slave takes each replication chunk and applies it.

Here is my idea.
  • For each replication chunk, the master creates a handle name for it, and also the handle name for the next chunk.
  • The server copies each chuck into an S3 item, once. The item's name is it's handle, and it has a piece of S3 metadata that is the handle of the next chunk.
  • Each client tails the bucket's item list, and grabs each chunk in turn. After it's applied that chunk, it writes a short item back to the bucket, stating that it's applied the chunk.
  • A low priority reaper watches the bucket, and when every registered slave marks a given chunk as applied, the reaper deletes the chunk.

The advantages are
  • The master only has to write the chunk out to the network once. There is no increased load when the number of slaves is increased.
  • The slaves can be very geographically dispersed without additional pain.
  • The master and the slave don't need hot TCP connections, VPN connections, or firewall configurations.
  • If the network partitions for a while, the slave falls behind, but will resync without pain. Also, a network partition doesnt crash the master when it's binlog space is exhausted.
Tags: geek, mysql, s3

  • Cute. Petite. Blonde. French. Nerd. Grrl.

    My best story about sex I didn't have: I was in Lille to give the final day keynote at fOSSa, and I had not yet written my talk. Each day, i had…

  • Razors

    I'm getting ads for I think five different "all metal" "get the best shave of your life" "throw away the plastic" razor startups. They all seem to be…

  • Doing what needs to be done

    On May 1st, one of my co-residents found one of the feral rabbits that live in the area cuddled up against a corner of the house. It was seriously…

  • Post a new comment


    Comments allowed for friends only

    Anonymous comments are disabled in this journal

    default userpic

    Your reply will be screened

    Your IP address will be recorded