• More-of-the-same in line with MarkLogic’s core positioning.
• A new bi-directional Hadoop connector.
• A free MarkLogic Express edition, limited in license terms more than in actual features, as per Slide 27 of the deck MarkLogic graciously supplied for me to post.

Also, MarkLogic is early with a feature that most serious DBMS vendors will soon have – support for tiered storage, with writes going first to solid-state storage, then being flushed to disk via a caching-style algorithm.* And as befits a sometime search-engine-substitute, MarkLogic has finally licensed a large set of document filters, from an Australian company called Isys. Apparently, the special virtue of the Isys filters is that they’re good at extracting not only text, but metadata as well.

*If there’s a caching algorithm that doesn’t contain a major element of LRU (Least Recently Used), I don’t recall ever hearing about it.

MarkLogic seems to have settled on a positioning that, although distressingly buzzword-heavy, is at least partly based upon reality. The real part includes:

• MarkLogic is a serious, enterprise-class DBMS (see for example Slide 12 of the MarkLogic deck) …
• … which has been optimized from the getgo for poly-structured data.
• MarkLogic can and does scale out to handle large amounts of data.
• MarkLogic is a general-purpose DBMS, suitable for both short-request and analytic tasks.
• MarkLogic is particularly well suited for analyses with long chains of “progressive enhancement” (MarkLogic’s favorite term when talking about derived data).
• MarkLogic often plays the role of a content assembler and/or search engine, and the people who use MarkLogic in those ways are commonly doing things that can be described as research and analysis.

Based on that reality, MarkLogic talks a lot about Volume, Velocity, Variety, Big Data, unstructured data, semi-structured data, and big data analytics.

My November, 2010 overview of MarkLogic technology remains pretty relevant. One correction, however: Node heterogeneity configurations, in which “data” and “evaluation” nodes reside on separate servers, are the exception rather than the rule.

Like Vertica, MarkLogic has laudably said that true academic researchers can get MarkLogic for free without the severe license restrictions. Free MarkLogic should be of particular interest to researchers who:

• Are studying natural networks or graphs, such as social networks or biological pathways. (This might be a fit in the social or biological sciences.)
• Are managing metadata for, say, a variety of disparate kinds of experimental files. (This might be a fit anywhere in the natural sciences.)
• Are managing actual documents, images, videos, etc., or data about such things. (This might be a fit in the humanities or social sciences.)

MarkLogic provided some disclosable financial substance by email, which I shall quote verbatim:

• MarkLogic has 45% revenue growth and 55-60% license growth year over year.
• We expect to finish this year with over $85 million in revenue, up from $55 million last year.

Arithmetical purists might note that 85/55 is more than 145%, but I’m just going to settle for the information I got and move on.

Edit: I posted separately about the MarkLogic Hadoop connector. As for that Hadoop connector – stay tuned for a short follow-up post, as writing about it now would not be convenient. (My backup discipline isn’t what it should be, and the only copy of my notes about that product is on a heavy tower computer in a house that doesn’t have working power.)

• HP needs to make outstanding enterprise systems again.
• They fell away from that target under Mark Hurd, but they surely can hit it again, based on the remnants of DEC (Digital Equipment Corporation), Tandem, the higher-end part of Compaq, and of course the original HP systems group.
• In particular:
  • Rumors say that Oracle Exadata 1 boxes, made by HP, were much lower quality than Exadata 2 boxes made by Sun.
  • HP Neoview was a waste of good engineering talent.
  • I’d like to see a few excellent Vertica appliances.
  • I hope the SAP HANA appliances go well, whenever HANA finally becomes a serious product.
  • The general move from disk to solid-state memory should offer some opportunities.

• 100s of gigabytes of data at first, growing to >1 terabyte over time.
• High peak loads.
• Public cloud portability (but they have private data centers they can use today).
• Simple database design — not a lot of tables, not a lot of columns, not a lot of joins, and everything can be distributed on the same customer_ID key.
• Stream the data to a data warehouse, that will grow to a few terabytes. (Keeping only one year of OLTP data online actually makes sense in this application, but of course everything should go into the DW.)

So I’m leaning to saying:  

• They should go with a scalable, MySQL-based solution.
  • Lots of third-party software works with MySQL, in case that’s helpful.
  • Yes, any one vendor is small and not yet firmly established, but there are numerous vendors around with interesting MySQL scaling stories.
  • In a vendor emergency, just going with Oracle’s MySQL stuff would probably work …
  • … especially because there are these lovely things in the world called solid-state drives.
  • There’s also good escapability if one wants to move away from MySQL, because everybody knows how to handle MySQL data.
• The first product to look at is dbShards, because it meets all the topology needs:
  • Local scale-out (transparent sharding).
  • Local high availability.
  • Remote disaster recovery (details of that are underway).
• The first analytic DBMS to look at is Infobright.
  • Yes, I know Infobright is focused more on machine-generated data these days, but this client’s analytic needs are so straightforward Infobright should pass with flying colors.
  • The MySQL-to-MySQL aspect should make ETL dead simple.
  • Again, there’s escapability.

Mainly, this is all fine. But I’m getting pushback on the solid-state aspect, for fear that it will compromise public cloud portability.

Am I missing something here? As far as I’m concerned, if you’re planning an OLTP system with a many-year lifespan today, of course you should assume solid-state storage. Maybe you scale out just as far as you would with disk, striping indexes or entire databases across the RAM of multiple servers. It that case, having solid-state backing reduces the risk of bottlenecks. Maybe you don’t scale out as far as you would with disk. In that case, solid-state backing saves you money.

As for public-cloud support for solid-state storage, that’s coming fast, right? (Actually, I have data points in support of that theory, but they’re a bit tenuous.) A large fraction of web businesses with private data centers seem to be using solid-state storage — from Facebook on down — or so the NoSQL/NewSQL/short-request DBMS guys tell me. Surely a number of public cloud vendors are close behind.

• A Kaminario K2 DRAM-only appliance costs $100K.
• A Kaminario K2 flash-only appliance costs $30K (but nobody buys that configuration).
• A typical Kaminario K2 hybrid DRAM/flash appliance might cost $35K (which tells us that there’s a lot more flash than DRAM).

Kaminario positions DRAM as where you focus your most write-intensive/ bottlenecking loads, such as logging or temp space, with the primary benefit being performance and a secondary benefit being slowing the wear on your flash.

If you want even your mirrors to be on flash — which Kaminario says greatly reduces the temporary performance hit in case of a failure — there will be an additional charge. Perhaps Kaminario will dig up a price number and post it in the comment thread.

The flash comes in via Fusion-io cards. Kaminario stresses that it sells a SAN (Storage Area Network) kind of offering, as opposed to the shared-nothing way one might otherwise use Fusion-io cards in servers’ PCIe slots. Kamanario further asserts its built-in high availability is both smoother and less costly than Texas Memory Systems or Violin Memory alternatives; Kaminario is generally proud of its high availability features, down to redundant uninterruptible power supplies. Apparently the sweet spot of Kaminario’s market is single-chassis 5-6 TB systems, but Kaminario asserts seamless elasticity even if you grow into a second chassis.

Price resistance seems to have gotten strongly in the way of Kaminario’s growth, although the company was evasive about customer counts and the like. But it does now have 60+ employees and an aggressive hiring plan, vs. <50 when I wrote about Kaminario a year ago. I do believe that many enterprises would benefit from throwing solid-state storage at certain performance problems, at least as a band-aid, while they contemplate software changes.* But evidently Kaminario has had difficulties — especially at the DRAM-only price point — getting customers to agree, or at least to agree that Kaminario K2 was a sufficiently cost-effective way to address the issue.

*If you like, you can regard this as deferring repayment of your technical debt.

Kaminario’s comments about how its technology is or will be applied are all over the place (again, I think part of this is due to having a small number of customers overall, and wanting to conceal how small that number is). But in general Kaminario has seen more OLTP (OnLine Transaction Processing) than analytic uptake, which contributes to them thinking that low latency is a bigger deal than raw IOPS (Input/Output Per Second). Certainly Kaminario is focused on database applications of some kind or other, generally running on big-name DBMS such as Oracle or Microsoft SQL Server

memcached is a way to cache data in RAM across a cluster of servers and have it all look logically like a single memory pool, extremely popular among large internet companies. The Membase product — which is what Couchbase has been selling this year — adds persistence to memcached, an obvious improvement on requiring application developers to write both to memcached and to non-transparently-sharded MySQL. The main technical points in adding persistence seem to have been:

• A persistent backing store (duh), namely SQLite.
• A change to the hashing algorithm, to avoid losing data when the cluster configuration is changed.

Couchbase is essentially Membase improved by integrating CouchDB into it, with the main changes being:

• Changing the backing store to CouchDB (duh). This will be in the first Couchbase release.
• Adding cross data center replication on CouchDB’s consistency model. This will not, I believe, be in the first Couchbase release.
• Offering CouchDB’s programming and query interfaces as an option. So far as I can tell, this will be implemented straightforwardly in the first Couchbase release, with elegance planned for later down the road.

Let’s drill down a bit into Membase/Couchbase clustering and consistency.

• When data is written to RAM in memcached, it immediately gets copied to another server. The same is of course true in Membase/Couchbase. The terminology on all this is confusing, but I think:
  • The portion of data that is stored as a primary copy on any given server is called a “shard”.
  • That would seem to make sense, as that data could correspond to what goes — non-transparently — into an instance of MySQL in a classical memcached/MySQL set-up.
• Updates are of course also banged to disk ASAP — but at times of heavy load, that can take a while. A few seconds to a couple of minutes is normal operation; if it takes an hour, you really should buy more hardware. (Or solid-state storage.)
• Similarly, the replication of data to a second machine’s RAM may not happen at times of heavy load — and that’s another sign you don’t have enough machines.
• Each Membase/Couchbase “shard” has lots of logical sub-shards.* (1024 for now, at least as default, although Dustin finds that number excessive and is looking to lower it.)  So if you add a node, some of the sub-shards get sent over to the new node. Unlike the case for straight memcached, no data is lost from cache (and of course not also from the persistent store). Blocking of operations from such a move only happens in narrow time windows, and then only in edge cases.

*Edit: They’re called vbuckets.

So if we consider Membase technology alone, Couchbase is CA in the CAP Theorem.  CouchDB, however, is gloriously AP in the CAP Theorem, in that it was written to assume an occasionally connected topology.* Based on that, Couchbase will allow AP operation between data centers (i.e. “stay synchronized if you can, to within the limitations of physics and so on, but don’t beat yourself up on the rare occasions that you can’t.”) I don’t know that that capability will quite be in the first release of Couchbase, but it’s coming soon.

*CouchDB also has other features friendly to occasionally-connected use cases, such as a lot of flexibility as to which parts of the database are or aren’t synced when you do reconnect. These are at the heart of the Couchbase Mobile offering.

memcached and Membase have a very simple key-value interface. CouchDB adds secondary indexes and so on. I think in the first release of Couchbase this is pretty much like having two different APIs for the same product; more elegant integration is planned down the road, and more language support as well.

The highest-performing way to use Couchbase will probably always be to just pretend it is Membase, which is to say memcached+. Dustin told me of Membase users who demanded 10-40 millisecond response times, and that not even for single queries but rather for sequences of several queries in succession. He further told me of customers asking for 1-200 microsecond response, and insisting on no worse than 1 millisecond. Frankly, the first requirement could be met by lots of technologies I can think of, at least if  you don’t rely on disk; the second is thoroughly impossible if you rely on disk, and pretty demanding no matter what kind of hardware and storage you have.

Couchbase performance against disk is a work in progress. CouchDB started out 8X slower than SQLite as a backing store, apples to apples, but Couchbase is fixing that before they roll the product out. (After all, they wouldn’t want to slow the product down in the course of an upgrade.) Beyond that, when you do exploit the indexing capability of CouchDB, performance of course slows down. Work is underway to lower the performance hit; I imagine much improvement can indeed be made, given how few resources CouchDB has been able to devote to date to Bottleneck Whack-A-Mole.

10gen recommends solid-state storage in many cases. In some cases solid-state lets you get away with fewer total nodes. 10gen also likes Flashcache (Facebook-developed technology to put a flash cache in front of hard disks). But the 100-node example mentioned above uses spinning disk.

Use cases 10gen is proud of include:

• Lots of user profile maintenance, including at online ad companies. This includes full user ad impression data. (I’ve argued for a while that user profile information belongs in something like a NoSQL database.)
• A big-name web company that wants to inspect every packet that enters their network, and replaced Splunk with MongoDB for performance reasons.
• A big-name photo/video site whose metadata is all in MongoDB. (That’s the kind of thing that often makes for good MarkLogic use cases.)

But actually, the reason we had the call was to review cases where MongoDB’s schemaless nature was significant. Examples of those included:

• A couple of top examples were of the kind “A bunch of apps, similar but not the same.” For MTV, it’s a single content management system for a bunch of websites. For Disney Playdom, it’s different schemas for every game.
• For a wireless telco, the issue was a product catalog in which devices and service plans called for very different schemas, and which the telco felt had thus become unmanageable in Oracle.
• For Craigslist, the issue wasn’t programming so much as performance — ALTER TABLE operations took months in MySQL, and that’s not a typo, although I’ll confess to not understanding why this was the case.

The 10gen guys went on to claim that schemalessness is helpful for incremental development in general, the point being that you don’t have a database-modification step. To some extent, changes can even be rolled back more easily than if you actually changed your schemas.

First the compression part. Eric thinks 6-10X compression is common for “curated” Hadoop data — i.e., the data that actually gets used a lot. Brian used an overall figure of 6-8X, and told of a specific customer who had 6X or a little more. By way of comparison, it sounds as if the kinds of data involved are like what Vertica claimed 10-60X compression for almost three years ago.

Eric also made an excellent point about low-value machine-generated data. I was suggesting that as Moore’s Law made sensor networks ever more affordable: 

• There would be lots more data thrown off.
• A lot of it would be repetitive “I’m fine; nothing to report” kinds of events.
• It would be a good idea to filter this low-value information out rather than permanently storing it.

Eric retorted that such data compresses extremely well. He was, of course, correct. If you have a long sequence or other large amount of identical data, and the right compression algorithms* — yeah, that compresses really well.

*Think run-length encoding (RLE), delta, or tokenization with variable-length tokens.

While I was at it, I asked Eric what might be typical for Hadoop temp/working space. He said at Yahoo it was getting down to 1/4 of the disk, from a previous range of 1/3.

Anyhow, Yahoo’s most recent standard Hadoop nodes feature:

• 8-12 cores
• 48 gigabytes of RAM
• 12 disks of 2 or 3 TB each

If you divide 12 by 3 for standard Hadoop redundancy, and take off 1/4, then you have 6-9 TB/node. Multiple that by a compression factor of 6-10X, at least for the “curated data,” and you get to 36-90 TB of user data per node.

As an alternative, suppose we take a point figure from Cloudera’s ranges of 16 TB of spinning disk per node (8 spindles, 2 TB/disk). Go with the 6X compression figure. Lop off 1/3 for temp space. That more conservative calculation leaves us a bit over 20 TB/node, which is probably a more typical figure among today’s Hadoop users.

The present invention relates to data storage, and more particularly, to methods and systems for a high throughput storage device.

A form of on-line transaction processing (OLTP) applications requiring a high number of data block reads or writes are called H-OLTP applications. A large server or mainframe or several servers typically host an H-OLTP application. Typically, these applications involve the use of a real time operating system, a relational database, optical fiber based networking, distributed communications facilities to a user community, and the application itself. Storage solutions for these applications use a combination of mechanical disk drives and cached memory under stored program control. The techniques for the storage management of H-OLTP applications can use redundant file storage algorithms on multiple disk drives, memory cache replications, data coherency algorithms, and/or load balancing.

and ends

It would be desirable for large capacity storage to provide sufficient throughput for high-volume, real-time applications, especially, for example in emerging applications in financial, defense, research, customer management, and homeland security areas.

The independent claims are:

1. A storage system comprising: one or more memory sections, including one or more memory devices including storage locations that store data, and a memory section controller that provides addresses to the memory devices, the addresses identifying storage locations for a memory device, wherein the memory devices use the provided addresses to perform a function selected from the set of reading out and writing data to/from the memory devices; and one or more switches, comprising a configurable switch fabric, that receive a data request including a data block identifier and switch the data request to one or more of the memory sections determined by applying the data block identifier to an algorithm that selectively configures operation of the switch fabric, the data block identifier identifying a set of storage locations; wherein the memory sections to which the data request was switched forward the received data block identifier to its memory section controller which maps the data block identifier to a set of addresses for the storage locations identified by the data block identifier, and provides the set of addresses to one or more of the memory section’s memory devices.
…
16. A method for use in a storage system, comprising: storing data in storage locations in a memory device; receiving by a switch comprising a configurable switch fabric, a data request including a data block identifier; the switch switching the data request to a memory section including the memory device determined by applying the data block identifier to an algorithm that selectively configures operation of the switch, the data block identifier identifying a set of storage locations in the memory device; forwarding the received data block identifier to a memory section controller; the memory section controller mapping the data block identifier to a set of addresses for the storage locations identified by the data block identifier; and the memory section controller providing the set of addresses to the memory device; and the memory device using the provided addresses to perform a function selected from the set of reading and writing data to/from the memory device.
…
26. A storage system, comprising: means for storing, including: means for storing data in storage locations, the means for storing data in storage locations including means for reading data stored in the storage locations using an address; means for controlling the means for storing, the means for controlling including: means for mapping a data block identifier to a set of addresses, means for providing the addresses to the means for storing data in storage locations, the addresses identifying storage locations; means for switching, including means for receiving a data request including a data block identifier; means for switching the data request based on the data block identifier to a means for storing determined by applying the data block identifier to an algorithm that selectively configures operation of the means for switching, the data block identifier identifying a set of storage locations in the means for storing data in storage locations; and means for forwarding the received data block identifier to the means for storing.

27. A storage hub comprising a memory section, including a memory device including storage locations that store data, and a memory section controller that provides an address to the memory device, the address identifying a storage location, wherein the memory device uses the provided address to write data into the memory device; and a switch, comprising a configurable switch fabric, that receives a data request including a data block identifier and transmits the data request to the memory section determined by applying the data block identifier to an algorithm that selectively configures operation of the switch fabric, and that receives write data associated with the data request and transmits the write data to the determined memory section; wherein the memory section forwards the received data block identifier to the memory section controller, which determines from the data block identifier the address of a storage location and provides the address to the memory device, and the memory device stores the write data at the address.

My one thought that could have led to the patent making sense was that maybe the term “configurable switch fabric” was defined in some particularly limited way. But noooo. Indeed, the term is not defined in the patent’s body at all; rather, the patent says (somewhat ungrammatically):

The switches 22 may be any type of switch using any type of switch fabric, such as, for example, a time division multiplexed fabric or a space division multiplexed fabric. As used herein, the term “switch fabric” the physical interconnection architecture that directs data from an incoming interface to an outgoing interface. For example, the switches 22 may be a Fibre Channel switch, an ATM switch, a switched fast Ethernet switch, a switched FDDI switch, or any other type of switch. The switches 22 may also include a controller (not shown) for controlling the switch.

I would be shocked if this patent held up upon reexamination. (If it did, EMC would pretty much be out of business, or at least vulnerable to a considerable cashectomy.) This is a particularly strong example of my belief that performance-enhancement software patents are always bogus. What’s more, it seems strange to worry about this patent’s effect on HDFS in any case, because if you’re that much of a patent wimp, you probably don’t want to run afoul of Google’s (also bogus) MapReduce patent in the first place.

On the whole, I’m somewhat more sympathetic to the idea of replacing HDFS underneath Hadoop than my clients at Cloudera or IBM would wish me to be. But the Parallel Iron patent is not a serious reason in support of such a change.

• Hadoop nodes today tend to run on fairly standard boxes.
• Hadoop nodes in the past have tended to run on boxes that were light with respect to RAM.
• The number of spindles per core on Hadoop node boxes is going up even as disks get bigger.

A key input comes from Cloudera, who to my joy delegated the questions to Omer Trajman, who wrote:

Most Hadoop deployments today use systems with dual socket and quad or hex cores (8 or 12 cores total, 16 or 24 hyper-threaded). Storage has increased as well with 6-8 spindles being common and some deployments going to 12 spindles. These are SATA disks with between 1TB and 2TB capacity. The amount of RAM varies depending on the application. 24GB is common as is 36GB – all ECC RAM. HBase clusters may have more RAM so they can cache more data. Some customers put Hadoop on their “standard box” which may not be perfectly balanced (e.g. more RAM, less disk) and needs to be altered slightly to meet the above specs. The new Dell C2100 series and the HP SL170 series are both popular server lines for Hadoop.

For a year ago perspective, see this post: http://www.cloudera.com/blog/2010/03/clouderas-support-team-shares-some-basic-hardware-recommendations/

Bullet points from that year-ago link include:

4 1TB hard disks in a JBOD (Just a Bunch Of Disks) configuration

2 quad core CPUs, running at least 2-2.5GHz

16-24GBs of RAM (24-32GBs if you’re considering HBase)

Gigabit Ethernet

So basically we’re talking in the range of 2-3 GB of RAM per core — and 1 spindle per core, up from perhaps half a spindle per core a year ago.

Meanwhile, a 2009 Yahoo slide deck refers to “500 nodes, 4000 cores, 3TB RAM, 1.5PB disk”; that divides out to 8 cores, 6 GB of RAM, and 3 TB of disk per node, all on “commodity hardware.” By 2010 Yahoo was evidently up to 2 GB of RAM per core.

There are lots of data points on the Apache Hadoop wiki, but many seem a few years old, and I don’t immediately see how to time-stamp them. Overall, they seem consistent with the trends I noted at the top of the post.

One thing I haven’t done is attempted to price any of these systems.

Contributions in the comment thread would be warmly appreciated.

Fusion-io is growing very, very fast, doubling or better in revenue every 6 months.

Fusion-io’s marketing message revolves around “data centralization”. Fusion-io is competing against storage-area networks and storage arrays.

Fusion-io’s list of application types includes

… systems dedicated to decision support, high performance financial analysis, web search, content delivery and enterprise resource planning.

Fusion-io says it has shipped over 20 petabytes of storage.

Fusion-io has a shifting array of big customers, including OEMs: 

Historically, large purchases by a relatively limited number of customers have accounted for a substantial majority of our revenue, and the composition of the group of our largest customers changes from period to period. Many of our customers make concentrated purchases to complete or upgrade specific large-scale data storage installations. These concentrated purchases are short-term in nature and are typically made on a purchase order basis rather than pursuant to long-term contracts. During fiscal 2010 and the six months ended December 31, 2010, sales to the 10 largest customers in each period, including the applicable OEMs, accounted for approximately 75% and 92% of revenue, respectively. Facebook, Inc. is currently our largest customer and accounted for a substantial portion of revenue during the six months ended December 31, 2010. We expect revenue from sales to Facebook and one other end-user to account for a substantial portion of revenue for the three months ending March 31, 2011, but that revenue from sales to Facebook and the other end-user will decline significantly for the three months ending June 30, 2011 as they complete their planned deployments.

But Fusion-io invests enough in sales and marketing, including direct sales, that I’m guessing they’re out there persuading end-users to ask for product from Dell, HP, and IBM.

Fusion-io’s inventory growth of $23.3 million for the second half of 2010 is close to revenue of $26.0 million. Accounts receivable is a much smaller figure. I’m not sure what all that signifies, but I do find it ironic that Fusion-io’s marketing statements draw an analogy to “just-in-time” manufacturing.

As for what I think about Fusion-io, it starts:

• Fusion-io’s ideas are smart.
• My skepticism about specialized storage hardware for database applications applies in part but not in whole to Fusion-io.
• Right now, Fusion-io has won the market. Even if you don’t need Fusion-io hardware to optimize your use of solid-state memory, you’re apt to go with/partner with Fusion-io anyway.

I don’t have strong opinions as to how long the last point will remain true.