?

Log in

No account? Create an account
entries friends calendar profile My Website Previous Previous Next Next
"... is finished not when there is nothing left to add, but when there is nothing left to take away. - Mark Atwood
fallenpegasus
fallenpegasus
"... is finished not when there is nothing left to add, but when there is nothing left to take away.
Going in to this most recent gig, it was thought that it was going to be a heavy user of MySQL Proxy, which thus would involve a fair amount of custom Lua programming.

As the client and I talked and whiteboarded out the design, it evolved into being very complex. It was going to be a challenge, and had a lot of moving parts.

As big and expensive as it was becoming, it still was much simpler and cheaper than what *r*cl* had quoted, they wanted a mid seven digits, PER YEAR, in license fees, for their proposal.

Anyway, as this thing grew on the board, something started itching in my head, and I sat back, looked at it again, and said "we're recreating the binlogs and the replication threads".

The client was very familiar with replication in Oracle and in MS SQL, but not at all so for MySQL, especially it's robustness in the face of lag and link failure. E.G, just let them lag, and let the disconnected one catch up again out of the binlog when it can reconnect.

With that, 90% of the past day and a half's design work got erased, and the whole thing collapsed into a completely bog standard replication setup, that any trained MySQL DBA can instantly recognize, understand, and maintain. One master, a handful of slaves, using InnoDB, replicating over geographic distances, with a needed average replication link bandwidth of less then 10 kilobits/sec. It doesn't even need hot slave promotion, since the system is still live and working, just not updating, if the master fails.

They can go live in *days*, not months.


The most important skill to have regarding any advanced technique, is being able to see when you don't have to use it.

Tags:

6 comments or Leave a comment
Comments
mauser From: mauser Date: October 30th, 2008 10:25 pm (UTC) (Link)
Definitely an engineer. A sales and Marketing droid would be trying to figure out how to reach that seven digit figure....

You're in trouble now!
fallenpegasus From: fallenpegasus Date: October 31st, 2008 04:53 pm (UTC) (Link)
My boss approves. It's one of the nice things about this job, there is no pressure to upsell.
From: ag73 Date: October 31st, 2008 04:13 pm (UTC) (Link)

What about lost transactions?

Isn't your client concerned about transactions that have not yet made it to the slaves when the master fails? Depending on the nature of the master failure, those transactions might never get recovered.
fallenpegasus From: fallenpegasus Date: October 31st, 2008 04:55 pm (UTC) (Link)

Re: What about lost transactions?

A lost transaction isnt a critical problem.

If they get little bit out of sync, its pretty easy to discover and fix.
From: ag73 Date: October 31st, 2008 05:10 pm (UTC) (Link)

Re: What about lost transactions?

That's an interesting response. Do you think if they had a choice (assuming no cost or low cost) they would prefer a solution that guaranteed that transactions had at least made it to the slave replication log if the cost were a slight (5-10%) reduction in performance?

The reason I ask is that I'm currently deploying such a solution for some beta customers and I'm interested in feedback from people like you who know this space. Here's a one pager that describes our approach:

http://www.codefutures.com/dbshards-replicate/

I'd be interested to hear your thoughts (either privately or through your blog). My email address is my first name dot last name at codefutures dot com.

Andy Grove.
fallenpegasus From: fallenpegasus Date: October 31st, 2008 09:27 pm (UTC) (Link)

Re: What about lost transactions?

Neat.

I can't recommend it to clients until PS has played with it, but it looks like a good idea.
6 comments or Leave a comment