• No technology category name is ever perfect.
• It’s particularly hard to describe NoSQL (Not Only SQL) accurately, given the basic confusion as to what NoSQL is all about.
• That said, it seems pretty clear that NoSQL is about making big websites (and perhaps other cloud-like installations) run and scale.
• Dwight Merriman (founder/CEO of MongoDB vendor 10gen) is heading in the right direction when he says that the unifying ideas of NoSQL are that you do away with transactions and joins. But if he’s ever said something like “NoSQL is Foo without joins and transactions,” I don’t know what Foo is.
• Actually, I do know what Foo is – Foo is what happens when lots of people want to get small amounts each of information in or out of a database at the same time. I just don’t know what Foo is called.
• Obviously, Foo is a lot like OLTP (OnLine Transaction Processing). However, it would be pretty silly for Foo to actually be OLTP, given that one of the core points of NoSQL is that you don’t have transactions.
• It not just the “T” part of OLTP that’s fried. Calling something “OnLine” only makes sense as long as offline is an option, and offline transaction processing has been obsolete for a very long time.*

*Sure, if you strain you can talk yourself into exceptions. But the point stands.

So we need a name for Foo, where Foo is what happens when lots of people want to get small amounts each of information in or out of a database at the same time. Thus, three major subcategories of more-or-less disk-based Foo are:

• No-compromises ACID-compliant relational OLTP
• Sharded MySQL
• NoSQL

There may be some more purely memory-centric versions too, but let’s put those aside for the moment.

Absent a better idea, I can squeeze Foo into yet another four-letter acronym:

HVSP (High-Volume Simple Processing)

That’s as imperfect as any other category name, and an awkward mouthful to boot. So I’d love to hear a better one; if you have such, please share it! In the mean time, I think “HVSP” has merit because:

• The “Processing” part should be noncontroversial.
• “High-Volume” is inherent to the challenge. If RDBMS scale well enough for your use case, using something less powerful is probably silly.* Similarly, while Oracle shines at high-volume OLTP workloads, there are many cheaper DBMS that do a fine job of OLTP at lower volumes.
• “Simple” is the core principle of NoSQL systems, which drop joins and transactions as being too much foofarah. That only makes sense at all under the assumption that you have bone-simple queries and updates, so that programming around the lack of joins and transactions isn’t all that much of a burden.
• Something similar is true of sharded MySQL.
• Less obviously, “simple” is a core principle of relational OLTP as well. The point of the relational model is to cap the complexity of data operations, or more precisely to hide that complexity from programmers.
• And overloading the word “simple” a bit, it’s fair to say that if you’re reading or writing one record at a time, you’re doing something relatively simple, at least as opposed to what you do in analytic processing. The OLTP vs. OLAP distinction is preserved in this name change.
• The whole thing matches my definition above, namely “what happens when lots of people want to get small amounts each of information in or out of a database at the same time.”

*Assuming, of course, that rows-and-tables are a good metaphor for your data structure in the first place.

Systems I’m leaving out of the HVSP and hence also NoSQL categories include:

• Hadoop and other batch-oriented MapReduce. Hadoop isn’t part of NoSQL. I’m pretty sure that Cloudera CEO Mike Olson agrees with me.
• More generally, non-SQL data stores that don’t meet the HVSP criteria. Dave Kellogg stretches things when he claims that MarkLogic is a NoSQL system. (But then, that was in a post where he seemingly praised a train wreck of an article.)

But hey – what good is a categorization if it doesn’t leave some things out?

• Facebook invented and is adopting Cassandra.
• Twitter is adopting Cassandra.
• Digg is adopting Cassandra.
• LinkedIn invented and is adopting Voldemort.
• Gee, it seems as if the super-scalable website biz has moved beyond MySQL/Memcached.

But in addition, he provides a lot of useful links, which DBMS-oriented folks such as myself might have previously overlooked. Following those trails gets one to, among other things:

• A September, 2009 post outlining Digg’s reasons for moving to Cassandra. The core idea is that joining two tables is expensive; it’s cheaper to store the results prejoined on disk. Details are provided.
• A February, 2010 post outlining Twitter’s reasons for moving to Cassandra. They boil down to “sufficiently scalable, sufficiently simple, sufficiently robust, robustly open source.”
• A Flickr slide presentation saying “normalization is for wimps”. They seemed to be staying with MySQL, but lusting after XPath.
• A nice Cassandra technical overview by Evan Weaver of Twitter.

I also recall seeing something that said “We have 13X as many queries as updates, so of course we should optimize for reads,” but I can’t find that now. The classical OLTP answer to that would probably be “Yeah, but by the time you’re two-phase-committing and integrity-checking all the part of that update, it turns out updates are still what you should optimize for.” Well, what if the update is so simple that that’s no longer a valid argument?

There certainly seem to be some non-obvious technical choices being made here, with options being conflated that perhaps shouldn’t be. In particular, I wonder whether things are being written to cheap disk in a really fast way when it might be better to keep them in more expensive RAM or, perhaps better yet, solid-state memory. Perhaps then the functionality/performance tradeoff wouldn’t be so painful.

On the other hand, the designers of the world’s most scalable websites — e-commerce sites perhaps excepted — seem pretty unanimous in thinking it’s best to bake some database/integrity management into the applications, rather than offload it all to an RDBMS. Why? Because the transactions are so simple that hand-coding all that isn’t prohibitive. And of course because of their extreme performance and scalability needs.

I’m not sure on what basis one could argue that they’re wrong.

• Oracle thinks flash memory is the most important hardware technology of the decade, one that could lead to Oracle being “bumped off” if they don’t get it right.
• Juan believes the “bulk” of Oracle’s business will move over to Exadata-like technology over the next 5-10 years. Numbers-wise, this seems to be based more on Exadata being a platform for consolidating an enterprise’s many Oracle databases than it is on Exadata running a few Especially Big Honking Database management tasks.

And by the way, Oracle doesn’t make its storage-tier software available to run on anything than Oracle-designed boxes.  At the moment, that means Exadata Versions 1 and 2. Since Exadata is by far Oracle’s best DBMS offering (at least in theory), that means Oracle’s best database offering only runs on specific Oracle-sold hardware platforms.

*E.g., I was sitting upstairs in my parents’ apartment in Columbus, OH having the call while their doctor, who I’ve never met, was visiting downstairs. He offered to make a special trip back Saturday afternoon because he missed me Wednesday, but he’s notorious for not coming when he says he will. Update: He didn’t come Saturday. On Saturday he said he’d come Sunday. He didn’t do that either.

Other high- and lowlights of our conversation included:

• Flash is the main new hardware element in Exadata Version 2. Otherwise, Exadata 2 is just an annual refresh of Exadata Version 1 to include updated components (Nehalem chips, bigger disk drives, etc.)
• Juan thinks it’s suboptimal to use flash memory through the bottleneck of disk controllers, favoring PCIe cards instead. (I emphatically agree.)
• Juan resolutely ducked questions about actual Exadata production deployment. Literally the only fact he shared in that regard is that there are at least 2 Exadata production systems running that each have 2 or more racks cabled together.
• Juan stressed that Exadata runs apps written over Oracle DBMS unchanged.
• When making mixed-workload claims for Exadata 2, Juan stressed consolidation of multiple databases, some OLTP and some analytic. He didn’t really argue with my skepticism about integrating OLTP and analytics in the same database, with one exception:
• Juan pointed out that in major OLTP apps such as ERP systems, there often is actually more processing going on in reporting and other batch stuff than there is in true OLTP.
• Exadata 2’s flash memory is designed as a disk cache, smarter than LRU (Least Recently Used). The two examples Juan gave of “smarter than LRU” are that backups and table scans don’t flush the cache.
• I forget whether this is new in Exadata 2 (I think it is), but anyhow – Exadata has a “Storage Index” that’s a lot like a Netezza zone map. I.e., for each megabyte or so of data it stores the min and max value of every column; if a query predicate rules out those ranges, that megabyte is never retrieved.
• Oracle has long offered what sounds like flexible workload management capability, and this has now been extended to specifically include I/O resources on the storage tier.
• This isn’t Exadata-specific, but Oracle has built a file system on top of its DBMS, optimized for speed, which helps with, e.g., ELT (Extract/Load/Transform). Evidently, it’s not at all the same thing as Mark Benioff’s 1990s Microsoft-annoying IFS (Internet File System) project, which seems to have morphed into a content management SDK.

Highlights specifically in the area of parallelization included:

• Juan stressed that all databases consolidated onto an Exadata machine are/should be striped across all storage units.
• On the other hand, Juan said that different databases should be confined to specific cores or CPUs on the database tier.
• But on the third hand, Juan also stressed – in what could be called a “private cloud” pitch – that there’s great elasticity as to which databases are matched to which server CPUs.
• Contrary to what I thought he and/or his colleagues told me a year ago, Juan said RAC (Real Application Clusters) is a big part of Oracle’s data warehouse processing.
• However, Juan says that what I regard(ed) as a major objection to Oracle’s database-tier parallelization — the need to manually specify “degrees of parallelism” — has now been obviated by automation. Juan thinks that few data warehouse DBAs will now need to manually tune parallelism, with minor exceptions. One exception he cites is that if a nightly report really is non-urgent, it can just be forced to run on a single core with no chance to grab more resources. (However, Juan thinks manual tuning of parallelism will continue to play a greater role in OLTP.)

OK. That’s all I can get done tonight (see above re: inconvenience of timing). Follow-on subjects I’d like to and indeed plan to post about include:

• What Juan said about hybrid columnar compression
• Oracle’s delightfully non-confidential slide deck, and a few comments about same

• MUMPS, the original Intersystems technology, was focused on health care.
• The reasons Intersystems went object-oriented have a lot to do with the structure of health-care records.
• Intersystems’ biggest and most visible ISVs are in the health-care area.
• Intersystems is actually beginning to sell an electronic health records system called TrakCare around the world (but not in the US, where it has lots of large competitive VARs).

Note: Intersystems Cache’ is sold mainly through VARs (Value-Added Resellers), aka ISVs/OEMs. I.e., it’s sold by people who write applications on top of it.

So far as I understand – and this is still pretty vague and apt to be partially erroneous – the Intersystems Cache’ technical story goes something like this:

• Intersystems Cache’ is an object-oriented DBMS.
• The preferred language for talking to Intersystems Cache’ is Java.
• Intersystems claims Cache’ has good SQL performance, for most kinds of use-case.
• Intersystems Cache’ stores data in a kind of sparse hierarchy. It uses a lot of “common character count” compression, which sounds a lot to me like Patricia tries.
• Intersystems has recently bundled some BI/reporting tools into the Cache’ stack. Surely not coincidentally, Intersystems once told me that some of its ISVs paid more to Crystal Reports than to Intersystems.
• Intersystems Cache’ has had Sybase emulation for several years, and just added Informix emulation. Most but not all stored procedures from those other DBMS run against Cache’ as well.
• Intersystems Cache’ recently added a bunch of manageability, security, etc. features, the details of which generally inspired “Oh, you didn’t have that earlier?” reactions in me.
• Intersystems’ just did a revamp of the Cache’ object model to make it more Smalltalk-like, in which messages are set to parent rather than child classes when appropriate. Thus, when you recompile a class, you don’t also have to recompile all its children, and incremental recompilation is now near-instantaneous. (Put that one in the “Oh, you didn’t have that earlier?” category too.) Versioning will be better as well.
• In the latest release, Cache’ has added what Intersystems calls “Java Event Processing.” This doesn’t sound like CEP (Complex Event Processing), and I forgot to ask whether it was memory-centric at all. Anyhow, the idea is to bang objects into the database really quickly, having them be immediately available for SQL query. “Really quickly” means >10,000 objects/core/second, with one test at the European Space Agency getting up to 85,000. By way of contrast, Intersystems asserts (based on bake-offs) that RDBMS competitors have to insert into BLOBs to get competitive performance, with associated loss of queryability.

Finally, a few financial highlights:

• Intersystems did a little over $1/4 billion in revenue in 2009.
• 85% of that was Cache’.
• Revenue growth was slightly positive in 2009, and 15% in 2008.
• Headcount growth was 25% in 2009 and is planned to be big again in 2010, after being modest in prior years.

]]> http://www.dbms2.com/2010/01/15/intersystems-cache-highlights/feed/ 2 http://www.dbms2.com/2009/11/23/boston-big-data-summit-keynote-outline/ http://www.dbms2.com/2009/11/23/boston-big-data-summit-keynote-outline/#comments Mon, 23 Nov 2009 06:25:50 +0000 Curt Monash http://www.dbms2.com/?p=1227 Last month, Bob Zurek asked me to give a talk on “Big Data”, where “big” is anything from a few terabytes on up, then moderate a panel on cloud computing. We agreed that I could talk just from notes, without slides. So, since I have them typed up, I’m posting them below.

The top two points from Q&A probably were:

• Big Data and the cloud actually have relatively little to do with each other, a few exceptions notwithstanding, especially if the data is in a shared-nothing DBMS (as opposed to, say, a MapReduce-oriented file cluster). Two principal reasons are:
  • Redistributing data from node to node is a little slow, undermining some of the elasticity benefits of the cloud.
  • Getting data into the cloud in the first place is a lot slow.
• The NoSQL movement is a lot like the Ron Paul campaign — it consists of people who are dissatisfied with the status quo, whose dissatisfaction has a lot to do with insufficient liberty and/or excessive expenditure, and who otherwise don’t have a whole lot in common with each other.

Anyhow, here are my notes for the talk, edited in just a couple of places for readability or linkage.

Quick introduction

• Big Data vs. cloud
• How big is Big Data?
• At the low end of that range, there’s little you can’t do with conventional technology if you have:
  • An unlimited budget for hardware
  • An unlimited budget for software
  • An unlimited budget for people, especially Oracle DBAs

Big Data in OLTP

• Hard-core OLTP
  • Focus of DBMS technology for a long-time
  • Big budgets because each transaction has significant value
  • Tough to get users to change technologies
• Lighter-weight OLTP
  • Classic example = web companies
    • Big ones — retail-oriented ones (eBay, Amazon) partially excepted — rolled their own technology stacks
    • Reluctant to give money to anybody
      • Open source, etc.
  • Difficulty finding market
    • Product vs. feature
      • Clustering/HA/DR/whatever
      • Ditto cloud enablement
    • True products haven’t found much traction yet

Analytic Big Data use cases

• Kinds of data for analytics
  • More of same != big
  • More detail and/or new kinds
    • Complete data sets
    • Transactions
    • Call details
    • Tick/trade history
    • Web clickstreams
    • Network event logs
    • Other machine-generated data
    • CAM bottom line
      • Anything human-generated should and will be retained in its entirety
      • Quantities of machine-generated data retained should and will grow roughly in line w/ computing cost reductions (Moore’s Law, etc.)
• Analytic uses of Big Data
  • Analytics is mainly about three things
    • Problem detection
    • Customer relationship improvement
      • (Those overlap when the customer relationship is bad)
    • Financial statements on steroids
  • Main kinds of analytics
    • What BI vendors traditionally sell
      • General reporting and dashboards
      • Ad-hoc query (now driven from those reports and dashboards)
      • Planning (allegedly integrated with BI)
    • Research
      • Ad hoc relational query (worth mentioning twice because it drives so much of the market)
      • Data mining
      • Most web search and web mining
    • Operational/near-real-time
    • Archiving/compliance
  • What gets Big?
    • Mainly research and archiving
    • But when reporting or operational get Big, you have really interesting computing problems

Technology issues and trends

• Moore’s Law
  • CPUs — All about cores, hence parallelism is key
  • RAM
  • SSDs – hence replace disks
  • Sensors – hence generate lots more data
• Kryder’s Law
  • But rotational speeds up only 12.5X since Eisenhower Administration
  • Hence solid-state memory (or RAM) will soon take over
• In the mean time, I/O bottlenecks have had to be beaten
  • Hence sequential scans
  • Hence index-light architectures
  • Hence columnar
• DBMS “overhead”
  • Raw license and maintenance fees – software increasing fraction of total
  • OLTP vestiges – locking and all that
  • DBAs
    • People costs = huge fraction of total
    • Index-lightness addresses
    • So does appliance
  • Many people don’t really know how to write SQL
• Configuration
  • Appliance/tightly-balanced
    • Netezza
    • Teradata earlier
    • Greenplum/Sun
    • Oracle
    • IBM
    • Microsoft/Madison
  • Commodity/do what you want
    • Vertica
    • Greenplum now
    • Infobright, Aster and others
    • MapReduce-oriented file systems
  • Extreme rigidity is silly
    • Teradata, Oracle have both signaled moving to more modularity
    • Big driver of that = heterogeneous storage
      • Cheap disk
      • Expensive disk
      • Solid-state
      • RAM
    • CPU/storage ratio is even more of a driver

Theoretically defensible ways to segment the market

• Latency requirements
  • High availability and low latency go together
• Query types
  • Simultaneous users for same
• Database size
• Budget

Actual segments right now

• Utter ADW/EDW
• Data mart
  • Size
  • Naturally columnar vs. naturally row-based
• Operational/frontline
• Less dramatic/smaller EDW

• Exadata is great for data warehousing. At this time, that’s a claim much better supported by marketing and theory than by practice.
• Exadata 2 is a suitable annual improvement over last year’s Exadata 1. That’s quite plausible.
• Oracle is outstanding for OLTP. That’s borne out by vast amounts of experience, especially if by “outstanding” you mean “Gets the job done really, really well at a very high cost in terms of both licenses and labor.”
• The Flash memory in Exadata 2 makes Oracle even better for OLTP.* That’s plausible too. Worst-case is probably that Flash support doesn’t really work well in this release, but will be cleaned up soon.**
• OLTP and data warehousing uses for Exadata don’t interfere with each other. That one bears some discussion.

*Oracle has repeatedly emphasized that the Flash memory in Exadata 2 is meant to speed up OLTP. By way of contrast, I’ve only noticed one vague claim that Flash memory helps data warehousing – a reference to a doubling in “user scan rates”, which perhaps was a slip of the marketing pen.

**Oracle probably has been working on Flash memory support for a long time. But it’s likely that Oracle didn’t have a strategic commitment to Sun’s specific technology until April of this year. After all, back in March it looked as if IBM would wind up owning Sun.

The integration-versus-separation argument for OLTP and analytic databases is an old one. In the early 1980s, IBM pushed both the “Information Center” (precursor to the data warehouse) and relational DBMS (portrayed as good for query and maybe for OLTP as well). In the early 1990s, Ted Codd opined that relational DBMS were good for OLTP but not analytics, instead favoring “OLAP” systems like Arbor Software’s Essbase (which, ironically, is now owned by Oracle). As the 1990s progressed, a consensus emerged that most large* enterprises should have at least one relational data warehouse separate from the core OLTP DBMS, a view that has persisted to this day. Until the announcement of Exadata 2, Oracle hadn’t seriously disputed this consensus, although it of course it always has wanted its DBMS software to run your OLTP and analytic databases alike.

*At a sufficiently small enterprise, one DBMS suffices. If a single commodity server has enough power to do all your processing, without even requiring you to have the expertise to tune very seriously, that’s probably the right way to go.

Assuming one DBMS has plenty of functionality for OLTP and analytics alike – as Oracle certainly does – the main arguments for separating OLTP and data warehousing revolve around performance. Reasons to split out a separate analytic database include:

• You might just want to run a separate brand of DBMS for your OLTP and data warehousing. Oracle thinks this is a terrible idea. (I disagree, as do a whole lot of analytic DBMS vendors – Teradata, Netezza, Greenplum, Sybase, Vertica, Aster Data, Infobright, Kognitio, et al. — and their customers.)
• You may want to lay out or index your tables differently for OLTP and data warehousing. Materialized view capabilities as flexible as Oracle’s should let you do that in a single database.
• You may want to lay out your files differently for OLTP and data warehousing (e.g., in terms of block sizes). Oracle might claim that ASM (Automatic Storage Management) and, in particular, the “Stripe and Mirror Everything” option obviate that point. I’m far from convinced.
• OLTP and analytic workloads step on each other’s toes, Part 1. For example, analytic queries that call for table scans often don’t mix well with OLTP operations that call for random reads and (especially) writes.* In principle, Flash memory could greatly reduce the problem, if the OLTP workload talks mainly to Flash, while Flash talks to disk mainly via microbatches. But I’ll be quite surprised if Oracle has aced that challenge on the first try. More likely, a longish stretch of Bottleneck Whack-A-Mole lies ahead.
• OLTP and analytic workloads step on each other’s toes, Part 2. Even more fundamentally: If you don’t have sufficiently good workload management tools, combining OLTP and analytic workloads is a ghastly performance idea, with OLTP slowing to a crawl while analytic queries rumble to completion. However, I’d think Oracle is in pretty good shape in that area.

*If this weren’t a terribly difficult problem, Oracle, IBM, and/or Teradata – all of which can do a reasonably decent job of mixing long and short queries in the same workload — would probably have solved it years ago.

Bottom line: Some day, Oracle Exadata may be a great system for integrated OLTP and data warehousing – but probably not in the current release.

The Oracle Database 11g Release 2 white paper I cited a couple of weeks ago has evidently been edited, given that a phrase I quoted last month is no longer to be found. Anyhow, here are some quotes from and comments on what evidently is the latest version.

The In-Memory Database Cache (IMDB Cache) option of Oracle Database 11g Release 2, allows data to be cached and processed in the memory of the applications themselves, off-loading the data processing to middle tier resources. Any network latency between the middle tier and the back-end database is removed from the transaction path, with the result that individual transactions can often be executed up to 10 times faster. This is particularly useful where very high rates of transaction processing is required, such as those found under market trading systems, Telco switching systems, and Real Time manufacturing environments. All data in the middle tier is fully protected through local recovery, and asynchronous posting to the back end Oracle Database. With Oracle Database 11g Release 2, the ability to transparently deploy IMDB Cache with existing Oracle applications becomes much easier – with common data types, SQL and PL/SQL support, and native support for the Oracle Call Interface (OCI).

At a guess, this sounds like it’s based on Oracle’s TimesTen acquisition.

Oracle Database 11g Release 2 adds further optimizations, including capabilities to automatically determine the most optimal degree of parallelization for a query, based on available resources. With this comes automated parallel statement queuing, where the database determines that, based on current resource availability, it is more effective to queue a query for later execution once required resources have freed up.

Sounds like a kind of automatic workload management — i.e., the kind of optimization vendors of mature products get around to putting into their systems. It does not sound like query pipelining, however.

Oracle Database 11g Release 2 will automatically distribute a large compressed table (or a smaller non-compressed table), into the available memory across all the servers in the Grid, and will then localize parallel query processing to the data in memory on the individual nodes. This dramatically improves query performance, and is especially useful where large tables can be entirely compressed into the available memory using compression capabilities.

So Oracle caches compressed data. Not stated is which compression techniques are covered.

Each Exadata Storage Server stores up to 7 Terabyte [sic] of uncompressed user data, and also comes enabled with 384 GB of solid-state Flash cache. This Flash Cache automatically caches active data of the magnetic disks in the Oracle Exadata Storage Server, delivering a 10x performance gain for read and write operations under OLTP applications.

Sounds like the Flash memory is positioned for OLTP use.

In the past, Database Administrators and System Administrators have spent a great deal of time determining to how best place data across these disk arrays, to get maximum performance and availability. The best procedure for data placement is to simply Stripe And Mirror Everything; stripe data blocks equally across all disks in an array, and then mirror the blocks on at least two disks. This approach provides the perfect balance between performance, disk utilization, and ease of use.

This is a big part of what could be called the “Administering Oracle doesn’t suck nearly as badly as it used to” pitch. (Mitchell Kertzman, who was Sybase CEO after the mid-1990s meltdown, told me his motto was “We suck less every day.” But I digress …)

Automatic Storage Management (ASM), a feature of Oracle Database 11g automates the striping and mirroring of database without the need to purchase third party volume management software. As data volumes increase, additional disks can be added, and ASM will automatically restripe and rebalance the data across available disks to ensure optimal performance. Similarly, disks that report errors can be removed from the disk array, and ASM will re-adjust accordingly.

I.e., you can add nodes without taking the system down. That’s becoming a pretty standard feature for serious parallel DBMS.

Oracle Database 11g Release 2 improves ASM in significant areas. New intelligent data placement capabilities store infrequently accessed data on the inner rings of the physical disks, while frequently accessed data is placed on the outer rings, offering better performance optimization.

Also pretty standard.

Oracle has been enhancing partitioning capabilities for over ten years. Oracle Partitioning, an option of Oracle Database 11g Release 2, allows very large tables (and their associated indexes) to be partitioned into smaller, more manageable units, providing a “divide and conquer” approach to very large database management. Partitioning also improves performance, as the optimizer will prune queries to only use the relevant partitions of a table or index in a lookup. Oracle Database 11g Release 2 provides multiple methods for partitioning data, and also allows different levels of partitioning on the same table, so that a single partitioning strategy can be used to improve both performance and manageability.

Even better might be a system that doesn’t lean heavily on complex partitioning to achieve good performance.

Oracle Partitioning can also manage the lifecycle of information. Typically, all databases have active data – the information being processed this month or quarter, and historical data that is primarily read-only. Organizations can take advantage of the inherent lifecycle of data to implement a multi-tiered storage solution and lower their overall storage costs. For example, a large table within an order-entry system could contain all the orders processed in the last 7 years. Oracle Partitioning can be used to set up monthly partitions, with the current last four months of order data partitioned onto a high-end storage array, with all the other partitions placed on a lower-cost storage solution, often 2-3 times less cost than the high end storage environment.

This is becoming a standard feature for any parallel DBMS that can support multiple kinds of storage in one system.

Oracle Database 11g also provides advanced compression techniques to further reduce storage requirements. Using Oracle Advanced Compression, an option to Oracle Database 11g, all data in a table can be compressed using a continuous table compression capability that achieves a 2-4 times compression ratio with little performance impact on OLTP or Data Warehousing workloads. This compression technology replaces duplicate values in a table with a single value, and continuously adapts to data changes over time, so compression ratios are always maintained.

Sounds like dictionary/token compression.

With Oracle Database 11g Release 2, the Exadata Storage Servers in the Sun Oracle Database Machine also enable new hybrid columnar compression technology that provides up to a 10 times compression ratio, with corresponding improvements in query performance. And, for pure historical data, a new archival level of hybrid columnar compression can be used that provides up to 50 times compression ratios.

I thought they said 40X before. But even if my memory isn’t playing tricks regarding that, single-point compression ratio estimates are always very approximate.

Any hardware component in an Oracle Grid can be dynamically added or removed as required. Disks can be added or removed online with ASM, with the data automatically rebalanced across the new disk infrastructure. Additional servers can also be easily added or removed to a Real Application Cluster with users connected to these nodes rebalanced across the infrastructure. This ability to migrate users from one server to another in a RAC cluster also enables rolling patching of the database software. If a patch needs to be applied, then a server can be removed from the cluster, patched, and then put back into the cluster. The same operation can be repeated for the next server in the cluster, and so on.

Nice. And the paper goes on in that vein for quite a while.

Oracle Total Recall, an option to Oracle Database 11g, provides a solution for the retention of historical information. With Oracle Total Recall, all changes made to data are kept to provide a complete change history of information. This means that auditors can not only see who did what when, but they can also see what the actual information was at the time – something that previously has only be [sic] available by building into the application, or by expensive backup retention policies.

Timestamping/time-travel/whatever is increasingly becoming a standard feature as well, especially given the number of PostgreSQL-based DBMS on the market.

New internal control requirements found in regulations can be difficult and expensive to implement in an environment with multiple applications. Oracle Database Vault, an option to Oracle Database 11g, allows access controls to be transparently applied underneath existing applications. Users can be prevented from accessing specific application data, or from accessing the database outside of normal hours; separation-of-duty requirements can be enforced for different Database Administrators without a costly least privilege exercise. And Oracle Advanced Security, an option to Oracle Database 11g, can be used to transparently encrypt data at all levels – data in transit on the network; data at rest on physical storage and in backups. Similarly, the Data Masking pack can be used to obfuscate data as it moves from production to development, reducing the potential violation of privacy regulations or risking sensitive data leaks.

Oracle is the gold standard in database security.

Oracle’s self-management approach takes two tacks. Firstly, wherever possible, repeatable, labor intensive and error prone tasks that can be fully automated in the database have been. For example, Storage Management, Memory Management, Statistics collection, Backup and Recovery, and SQL Tuning have all been automated. Secondly, where operations cannot be fully automated, intelligent advisors are built into the database to mentor Database Administrators on how to get the best out of their systems. Advisors are provided for Configuration Management, Patching, Indexing, Partitioning, Performance Diagnostics, Data Recovery, and, new in Oracle Database 11g Release 2, Compression and Maximum Availability.

And boy are they needed.

Recent studies performed by an independent research company shows that Database Administrators can expect to spend 26% less time managing their 11g environments over their 10g environments, and as much as 50% when compared to older Oracle9i deployments.

50% of way too much is still way too much.

*While a few use cases are exceptions, being “better than MySQL” for a DBMS is basically like being “better than Pabst Blue Ribbon” for a beer. Unless price is your top consideration, why are you even making the comparison?

Even worse, the press release, from its subhead and very first sentence, emphasizes the claim “the Groovy SQL Switch’s ability to significantly outperform relational databases.” As CEO Joe Ward quickly agreed by email, that’s not accurate.  As you would expect from the “SQL” in its name, the Groovy SQL Switch is just as relational as the products it’s being contrasted to.  Unfortunately for Joe, who I gather aspires to edit it to say something more sensible, the press release is out already in multiple places.

More favorably, Renee Blodgett has a short, laudatory post about Groovy, with some kind of embedded video.

My client actually signed a deal for EnterpriseDB’s Postgres Plus Advanced Server and GridSQL, but unwound the transaction quickly. (They say EnterpriseDB was very gracious about the reversal.) There seem to have been two main reasons for the flip-flop.  First, it seems that EnterpriseDB’s version of Postgres isn’t up to PostgreSQL’s 8.4 feature set yet, although EnterpriseDB’s timetable for catching up might have tolerable. But GridSQL apparently is further behind yet, with no timetable for up-to-date PostgreSQL compatibility.  That was the dealbreaker.

The current base-case plan is to use generic open source PostgreSQL, with scale-out achieved via hand sharding, Hibernate, or … ??? Experience and thoughts along those lines would be much appreciated.

Another option for OLTP performance and scale-out is of course memory-centric options such as VoltDB or the Groovy SQL Switch.  But this client’s database is terabyte-scale, so hardware costs could be an issue, as of course could be product maturity.

By the way, a large fraction of these updates will be actual changes, as opposed to new records, in case that matters.  I expect that the schema being updated will be very simple — i.e., clearly simpler than in a classic order entry scenario.