Mark Atwood

I just pushed my first public release of my MySQL storage engine for AWS S3.

You can find it at http://fallenpegasus.com/code/mysql-awss3

Lightbulb!

The only thing that makes the S3 REST interface "special" is it's proprietary but documented auth protocol.

LibCURL knows a bunch of different auth protocols.

If the S3 auth protocol is added to to libCURL, then any existing libCURL based app can immediately read/write to S3 without further change.

The folks at Amazon AWS ought to do this. Maybe they will pay me to do it...

Attach to a table by saying

CREATE TABLE atst (s3id VARCHAR(255) NOT NULL PRIMARY KEY, s3val BLOB)) ENGINE='AWSS3' connection='awss3 bucketname aws_id aws_secret'

If the bucket's name is not ASCII, you have to convert it to UTF-8 and then percent-escape it yourself.

In theory, I could push out a release now, but the field selection code mades me want to put a paper bag over my head. So not just yet...

It now does DELETE FROM table WHERE s3id='something'.

• Features
• can now do inserts
• could do single row reads a few days ago
• Bugs
• need to URL encode the bucket name and the item name (D'oh!)
• phear that S3 doesn't get non-ASCII keys right anyway
• how do I get the HTTP return code?
• can't tell if an upload went ok or not
• can't tell the difference between a successful GET and "not found"
• To Do
• need to remove the hardcoded AWS credentials
• clean up Field object selection
• push a release to hg
• Next Features
• implement rnd* and position methods
• will involve parsing the bucket contents XML
• On The Horizon
• enforce schema constraints
• access to the Content-Type, Last-Modified-Date, metadata, and ACL
• libCURL connection sharing/reuse
• HTTPS
• cerificate auth
• gz transfer encoding
• gz stored encoding
• use CREATE SERVER instead of connection string
• dont get every item in the bucket on a scan
• client-side caching
• relaxed schema (multiple data columns)
• dont get every column if not needed (HTTP ranged GET)
• secondary keys
• multiple columns other than just UNIREG

As I continue to work on my AWS S3 MySQL storage engine, I decided that I needed some S3 buckets that actually contained "stuff", to test and develop it against. So I grabbed The Devil's Dictionary off of Project Gutenburg, wrote a little Perl script to read and parse it, and created an S3 bucket that contains a item for each definition. I also grabbed 63 scans of pieces by William Bouguereau out of ArtRenewal, and created a bucket to hold them.

Unfortunately for me, ArtRenewal doesn't lend itself to being easily scraped, or I would have scriped and then walked away from the process of transferring all 226 pieces into the bucket. Also, something (I suspect the Perl S3 library), doesn't play well with non-ASCII characters in item names, all the pieces with names with accented characters didn't upload. This is annoying, since item names are supposed to handle UTF-8.

But this is enough anyway to serve as a good working test base.

Oh, and thanks many to Amazon, for supporting this work by giving me a comp'ed AWS S3 account.

After staying up until 4:30am in the morning yesterday, and then coming right back to it after waking up, and with a lot of help from krow, my AWS S3 storage engine is doing index_reads. That means that, in SQL, you can say select * from foo_tbl where s3id='objectname'; and get back the contents of the object named "bar" from the S3 bucket that corresponds to "foo_tbl".

Woo!

Next up, inserts!

The folks at Amazon AWS S3 have invited me over tomorrow afternoon to talk to them about my MySQL AWS S3 storage engine. I plan/hope to have indexed read working and some slides done by then.

My project for the MySQL Expo, A Storage Engine for Amazon S3, just got linked to by Pete Freitag.

My project for the MySQL Expo, A Storage Engine for Amazon S3, just got linked to by Stephen O'Grady.

I'm sitting here at SJC, waiting for my flight (LUV 2334) to take me back to SEA. MySQL Camp worked very well for me. I learned a great deal, and made some very good contacts for my S3 storage engine. I've gotten very excited about getting it working.

I like the "unconference" format. It's similar to the things I like about BiCamp and about good small con, in that there is no divide between "presenters" and "attendees". If you want to present something, just edit the wiki and add yourelf to the schedule. If you have a question, ask. If you want to talk to someone, stop them in the hallway and ask them.

My only expenses were airfare (and I already had some Southwest credit to burn up), and evening dining. And I ate in the evening just to fit in, the food at Google was plentiful.