June 28th, 2007


Ideas for a MySQL queuing storage engine

krow just posted about the the difficulties of implementing a Queue Engine for MySQL.

I don't think it's really that impossible. Yes, there are some kind of SELECTs that either don't make sense, or are almost certainly not what the user wants, or are impossible to do.

But trying to use a very specialized engine for general purpose queries is not really something to worry about.

I would take the same approach that I do with my S3 storage engine, e.g., implement whatever makes sense, and then for the cases were it doesn't make sense, tell the user "So, don't do that, then!".

For a queue engine, I would separate the operations of "get next item" and "delete"

SELECT id,message FROM queue LIMIT 1;

DELETE FROM queue WHERE id="foo";

But then, I've been thinking about how to wrap the Amazon AWS SQS service up as a storage engine, and this approach fits with SQS reasonably well.

Distributed business organization

A few months ago, I had a meeting with a small local startup. Their tech and their pitch is pretty neat. They had an angel kick them a megabuck of seed to get started. It's the classic geek startup: two main guys, one wearing the CEO/sales/biz/money hat, and the other wearing the CTO/it/tech/arch/geek hat. And they've hired a couple of coders.

But they've also rented some office space: two cubes, a meeting room, a front desk, and a lockable office for the locking file cabinet. Now, it was cheap office space, but still, why?

It's a waste of the angel's money, is increasing their burn. For what it's costing them, they could hire another coder. Dev speed is their current bottleneck, and going from 2 to 3 causes only minimal invocation of Brook's Law, especially if it's early. If they start to grow, they'll have to hire #3 and #4 soon enough anyway, at which point they've outgrown their space, and now also have the cost, pain, and distraction during their growth phase of having to move.

For the locking file cabinet, keep that in the CEO's house. The geeks and the CTO can do gang programming at a house, or in a cafe. Meetings can be held in a cafe.

In fact, when I had my meeting with them, I discovered that while I had driven from CapHill to Bellevue to meet in their offices, they (the CEO & CTO) had also driven from somewhere else in Seattle back over to Eastside to meet me there. We could have just met in a cafe here on the Hill.

If you start out with your company being distributed, then it will be easy to distribute it later. It makes finding hires easier, it makes figuring out how to handle employment alternatives easier (halftime, homework, contractors, outsourcing, specialists, independent designers, etc).

There are advantages to physical proximity, I will admit. The `net doesn't yet have the bandwidth to really do hallway design or pair programming. But don't demand physical proximity and offices just because it's traditional, or it makes management oversight "easier".

Something I would like to see MySQL staff and management do more off on their blogs and at the various technology and open source conferences is to make posts and presentations about "Running an international distributed company. How MySQL AB makes it work."

This musing came to the forefront because I have a local friend who's being interviewed for a neat position with a large well known open source project foundation. However, despite the fact that it's a distributed development environment, he would still have to move to the Housing Hell that is the Bay Area, because the foundation wants all their important people to work in the same office together.

Recently, Marc Andreessen in his blog had a series of posts on how VCs work and how to get one to fund your idea. One of the things he let slip are that two of the imporant factors in getting VC money is that VCs prefer to work thru face to face referrals, and (not admitted to) the VC doesn't want to have to drive too far from his house or office to the offices of the startup he's funding.

Which pretty much guarantees that if you want to take Silicon Valley VC money, you have to deal with the housing and governmental hell that is Silicon Valley. Yet another reason not to take VC money if you can at all avoid it.