Log in

No account? Create an account
entries friends calendar profile My Website Previous Previous Next Next
Mark Atwood
Writing another storage engine, "Pig Engine"
I've started hacking together another storage engine. It's one I've wanted to write since before I wrote my AWSS3 engine.

I'm calling it "pig", because it's going to eat inodes like a pig eating corn.

Each table, row, and element is a filesystem directory. Each element directory contains a file that has the current value, and also a set of version files of the value, thus the table will contain it's entire history. The table directory contains a copy of the .FRM, to enable discovery. Storage engine API events write entries into a logging directory.

As it is, it's not very "useful", in that it's not fast, and it beats the filesystem half to death.

But, on the other hand, it's going to be useful to me for in three ways:

It's a good exercise in writing a more fully featured SE than my AWSS3 SE.

I think it will be a better "example" engine than the stock example engine, in that it will actually store data in an understandable form, and will be logging SE API calls.

And I'm going to modify it to store to HTTP, HTTP/DAV, and to AWS S3/SDS.

Tags: ,
Current Location: Remedy Tea, Capitol Hill, Seattle WA
Current Mood: geeky

2 comments or Leave a comment
loganb From: loganb Date: December 23rd, 2007 07:16 pm (UTC) (Link)
What is your algorithm for locating the newest (and thus valid) row value? Is it smarter than doing a directory listing? Also, have you thought about doing one file/row rather than per value and just version entire rows? That will tradeoff more space usage for less inode usage and perhaps make it slightly more "usable."
fallenpegasus From: fallenpegasus Date: December 24th, 2007 07:27 pm (UTC) (Link)
My plan is to have the newest item be a specific filename in the item directory, and then older items get moved to a history subdirectory at that point, with a rename.

I could do one file per row as well, but part of this is an exercise in assembling and disassembling the mysql internal row structure. The one row per stored item is the model used by the memcached engine, the http engine, and the next ver of the S3 engine.
2 comments or Leave a comment