?

Log in

No account? Create an account
entries friends calendar profile My Website Previous Previous Next Next
Do banks really use strict old-skool transactional RDBMSs? - Mark Atwood
fallenpegasus
fallenpegasus
Do banks really use strict old-skool transactional RDBMSs?
There was recently an article in Infoworld, Slacker databases break all the old rules, that gave a rundown of the new non-transactional non-RDBMS database systems that are coming on the scene.

It contains a number of remarks about bank applications and their need for strict transactionally consistant RDBMS, like so:

The old guard who start reaching for their heart medication at the news of these new databases are usually bank programmers who want to make sure that the accounts balance at the end of the day.

None of them is right for everyone, and all of them are completely wrong for the bankers out there.

Last year, when I was doing the MySQL Professional Services gigs, I actually had a client who was a gigantic financial services company. I'm not going to name them here, and you've probably never heard of them, but the odds are pretty good that they manage some portion of your money, via two levels of outsourcing indirection. They manage quite literally hundreds of millions of accounts, with an combined value in the high 12 digits. Every account gets at least one change a month, often more, and it all has to balance to the penny. Every account they've ever had, and it's full transaction history back to the beginning of time, must be kept and maintained. And there is very very strict legal and accounting oversight, imposed by internal auditors, external auditors, their direct clients, and a whole alphabet soup of regulatory agencies.

And they do not use a transactional RDBMS. For many reasons:


  • They've been around so long that the term "legacy" has real meaning.
  • They have a LOT of specialized reporting applications, many of which have been signed off on and locked down by their auditors and regulators. It would cost effectively an infinite amount of money to port them over to using SQL.
  • Their dataset is so large, that it would take literally months to just copy it, let alone import it into a new RDBMs.
  • Part of the reason they are so successful in their field, is that their margins and costs are so low, that they beat all their competition purely on price, in a very price sensitive market.
  • Buying enough IBM DB2 or Oracle RAC systems would cost them more in license fees than they currently charge their clients.
  • Estimations of the scalable performance of DB2 or RAC indicate that it would take more than a day, with all the necessary interlocking transactional updates, to apply just one day's worth of updates.


But, in their entire history, they've never lost a penny, or an account. All without a traditional transactional RDBMS.

Instead they have a huge array of VMS ISAM datastores.

How do they do it? They run "transactions" intelligently, in their applications, piece-wise, and carefully.

Remember, keeping transactions in the DBMS is not a law handed down from On High. It's an assumption hack because it was assumed that programmers could not be trusted to do it in their applications, and could not be trusted to understand how to implement good data structures or good enough access patterns.

Often it's a useful assumption, just like assuming that programmers could not be trusted to handle instruction sets, direct jmps, or memory allocation, thus the rise of compilers, while loops, and garbage collection.

But sometimes you have to step back, change your assumptions, and add support for average programmers in some other way. Or make sure you get better than average programmers, and give them brain support some other way.

Tags:

8 comments or Leave a comment
Comments
blarglefiend From: blarglefiend Date: April 15th, 2009 04:59 am (UTC) (Link)
A lot of it is familiarity. A transactional RDBMS provides some of the tools to support "average programmers", and also for "better than average programmers" having a bad day. It's not that you can't do this stuff without one, just that that's the support tool a lot of people in this space are used to.

I work for a reasonably large financial services firm. Some of our programmers are really good, and some are... less so. We've got guys who "need" a PHP wrapper to send email (running on a UNIX host, the rest of the app is not written in PHP), we've got guys who can't see that running a big query where all fields referenced aren't indexed is going to be slow, and so on.

If you start out with an RDBMS you might as well stick with one. If you started long before those were around, well, of course migrating to one is probably going to cost more than it's worth.

Porting our code from Sybase to Oracle would be painful but not impossible. Porting to something radically different would be a much bigger job, and probably not worth the cost.
mauser From: mauser Date: April 15th, 2009 08:16 am (UTC) (Link)
"Often it's a useful assumption, just like assuming that programmers could not be trusted to handle instruction sets, direct jmps, or memory allocation, thus the rise of compilers, while loops, and garbage collection."

... and Java.
From: (Anonymous) Date: April 15th, 2009 01:55 pm (UTC) (Link)

Not a convinving argument

Holding out a bank that is using an app written aeons ago in pre-RDBMS days is hardly a good argument for the ability to not use them. While you *can* trust your app to do the right thing, it's far from a best practice. The company in that example just did not have a choice at the time.

"It's an assumption hack because it was assumed that programmers could not be trusted to do it in their applications"

No, it's a rational and sound decision to put important business logic as close to the data as possible, and to make it nigh impossible to do bad things, whether on purpose or accidentally. I wouldn't consider for a moment NOT using a transactional database for my data, no matter how "better than average" my programmers are. The history of computing is littered with the plaintive cries of "We really did think we could do it all in the app..."

One more nitpick:
"But, in their entire history, they've never lost a penny, or an account"
And they would admit it if they did? :)
fallenpegasus From: fallenpegasus Date: April 15th, 2009 10:20 pm (UTC) (Link)

Re: Not a convinving argument

"rational and sound decision to put important business logic as close to the data as possible"

Until you try to make it scale.

The CAP Theorem doesn't go away just because because one wishes it so. At scale, ACID is dead. Know it, accept it, live it, love it.

ACID never really existed at all, in the first place.
From: pauldf Date: April 16th, 2009 04:21 am (UTC) (Link)

Re: Not a convinving argument

Until you try to make it scale.

Agreed; that's a key point.

RDBMS life is good when you can fit the database you need on a single commodity server. Beyond that, things get interesting.
fallenpegasus From: fallenpegasus Date: April 15th, 2009 10:22 pm (UTC) (Link)

Re: Not a convinving argument

"And they would admit it if they did?"

See my comment about auditors. They are subject to constant and overlapping auditors. If one of them is misses something, another one will catch it.
From: ext_181589 Date: April 16th, 2009 03:52 pm (UTC) (Link)

last paragraph says it all

I used to have a professor in college that said the real point of an article was always found in the last sentence or paragraph.

"But sometimes you have to step back, change your assumptions, and add support for average programmers in some other way. Or make sure you get better than average programmers, and give them brain support some other way."

Harder than average problems require better than average programmers.
From: hodgesrm Date: April 17th, 2009 10:54 pm (UTC) (Link)

Uncompelling argument

Your argument conveniently ignores all the banking applications that *do* use traditional RDBMS applications on Tandem, Oracle, and DB2.

Also, I think you overlook the importance of query optimization as a driver for RDBMS usage. If you are going to argue that RDBMS does not scale, I think you need to define exactly what sort of scaling you are talking about. As much as people tend to diss SQL, RDBMS engines perform exceedingly well for a very large set of queries, especially on systems where data characteristics (e.g., cardinality) and the underlying data model change over time. That includes RDBMS implementations with all the well-known overhead of transactions.
8 comments or Leave a comment