My quick thoughts start:

• Mainly, this isn’t for production usage. But exceptions might arise when:
  • An application, from creation to abandonment, is only expected to have a short lifespan, in support of a specific project.
  • There is an extreme internal-politics bias to operating versus capital expenses, or something like that, forcing a user department to cloud production deployment even when it doesn’t make much rational sense.
  • An application is small enough, or the situation is sufficiently desperate, that any inefficiencies are outweighed by convenience.
• There is non-production appeal. In particular:
  • Spinning up a quick cloud instance can make a lot of sense for a developer.
  • The same goes if you want to sell an Oracle-based application and need to offer demo/test capabilities.
  • The same might go for off-site replication/disaster recovery.

Of course, those are all standard observations every time something that’s basically on-premises software is offered in the cloud. They’re only reinforced by the fact that the only Oracle software Amazon can actually license you is a particularly low-end edition.

And Oracle is indeed on-premises software. In particular, Oracle is hard enough to manage when it’s on your premises, with a known hardware configuration; who would want to try to manage a production instance of Oracle in the cloud?

Jonathan’s core claims for Cassandra include:

• Cassandra is shared-nothing.
• Cassandra has good approaches to replication and partitioning, right out of the box.
• In particular, Cassandra is good for use cases that distribute a database around the world and want to access it at “local” latencies. (Indeed, Jonathan asserts that non-local replication is a significant non-big-data Cassandra use case.)
• Cassandra’s scale-out is application-transparent, unlike sharded MySQL’s.
• Cassandra is fast at both appends and range queries, which would be hard to accomplish in a pure key-value store.

In general, Jonathan positions Cassandra as being best-suited to handle a small number of operations at high volume, throughput, and speed. The rest of what you do, as far as he’s concerned, may well belong in a more traditional SQL DBMS. 

Further highlights of our talks included, as best I understood them:

• Cassandra is based in parts both on Google’s BigTable paper of 2006 and Amazon’s Dynamo paper of 2007.
  • The core of what Cassandra takes from BigTable is based on log-structured merge trees, which actually entered the computer science literature in 1996.
  • Cassandra’s approach to horizontal scaling, replication, failover, etc. seems to be based Dynamo.
• There seems to be a logical concept of “row” in Cassandra, or it’s at least meaningful to use the SQL/relational concept of a “row” when talking about Cassandra data. However, Cassandra is closer to being a column-based data store than a row-based one. (Not the same thing, but closer.)
• Even so, it only takes a single seek to return a whole Cassandra “row”.
• Cassandra writes data quite differently from the way a classical OLTP DBMS would.
  • Cassandra writes just the data elements – i.e., fields – that are actually being inserted or changed, not whole rows.
  • One benefit is that Cassandra data is very sparse. NULLs aren’t stored in any way, and hence in particular take up no space.
  • Another benefit – and one of the core concepts of Cassandra – is that you can implicitly assume different schemas for different rows of the same “table.” In particular, you can add data for columns that you didn’t envision when you first started storing “rows” of the same “table.”
  • Writes are collected into sorted “memtables,” which from time to time are sent to disk. Once data gets to disk, it’s immutable, except for occasional merge/reorganization/garbage collection.
    • Jonathan claims, plausibly, that this makes write throughput very fast (because the I/O is fundamentally sequential in nature.)
    • The default as to how long data typically stays in memory before it gets persisted to disk is “whichever comes first of {64 MB written, 300k updates, 1 hour}”.
  • Cassandra has durability – guaranteed non-loss of data – assuming fsync is turned on. fsync seems to create a 15% or so overhead.
  • However, Cassandra has no concept of a “transaction.”
  • As one would expect, data can be read even before it has been persisted to disk.
• According to Jonathan, Cassandra can do about 14,000 writes or 7,000 reads per second, on a quad-core server.
  • Those figures scale pretty linearly with the number of servers. (There’s some overhead for network latency.)
  • Those figures assume a five-column row.
  • Cassandra’s write-performance figures are only “mildly sensitive” to the width of the row. E.g., doubling row width only gives a 15-20% throughput hit, due to some fixed per-row overhead. That said, I imagine going 100X in row width would create a major slowdown, although perhaps while measuring width more in bytes than in column count.
  • Cassandra’s performance has been growing nicely in each point release. Jonathan thinks this general trend will continue.
• Jonathan thinks Cassandra is pretty good at keeping your data safe.
  • Each node has a commit log.
  • When a node goes down, its writes are buffered until it comes back up.
• You can run Hadoop MapReduce straight against Cassandra files.
• A Cassandra node might hold anything from 10s of gigabytes to multiple terabytes of data. You might want to go with the low end if you want to have lots of cache hits.
• Solid-state storage would speed up Cassandra reads, not writes, and is not widely used with Cassandra yet.
• Jonathan says Cassandra is really good at handling time series data, by which I suspect he means log files. Cloudkick is a user of this capability.

I certainly didn’t grasp everything about Cassandra replication and partitioning strategies. That wasn’t the focus of our talks, and anyway I got the impression they are so flexible that there’s little that can firmly be said about them. But I did get the impressions:

• You set your consistency rules in the Cassandra API, not on a per-table basis. (I think this means that a lack of administrative tools is supposedly a feature, not a drawback.)
• As a practical matter, Cassandra users commonly take one of two approaches to consistency:
  • RYW consistency, most commonly with N = 3 and R = W = 2.
  • Geographically dispersed eventual consistency.
• Cassandra data is most commonly distributed via consistent hashing, but other options are “pluggable.”
• If you add a node, the busiest note automagically decides to ship some data over, reducing its load. Of course, this only works if you get the new node on before the old node is so maxed out it doesn’t have time to do the shipping.

When we talked in March, the next release of Cassandra was going to be 0.7. Cassandra 0.7 was going to be a performance/scalability release, for example fixing the flaw that garbage collection read rows into memory one at a time. After that, Cassandra 0.8 was to be a feature release, with one planned feature being more automatic index management and/or materialized-view-like capability, so as to reduce the burden on Cassandra developers of schema management.

Related links

• My March NoSQL links post included the Google and Amazon papers
• The March 2, 2010 Cloudkick post also linked above goes into a lot of detail, including what they think is great about Cassandra and what they think is still missing

To me, CAP should really be PACELC — if there is a partition (P) how does the system tradeoff between availability and consistency (A and C); else (E) when the system is running as normal in the absence of partitions, how does the system tradeoff between latency (L) and consistency (C)?

and goes on to say

For example, Amazon’s Dynamo (and related systems like Cassandra and SimpleDB) are PA/EL in PACELC — upon a partition, they give up consistency for availability; and under normal operation they give up consistency for lower latency. Giving up C in both parts of PACELC makes the design simpler — once the application is configured to be able to handle inconsistencies, it makes sense to give up consistency for both availability and lower latency.

However, I think Daniel’s improved formulation is still misleading, in at least two ways:

• Daniel implicitly assumes any given NoSQL system makes a fixed set of tradeoffs, when actually — as he in fact notes in his post –  some of them offer tradeoffs that are quite tunable.
• I think Daniel is at best oversimplifying when he appears to assert that best-case network latency is an important design criterion for all that many NoSQL systems. Naively, anything that acknowledges reads or writes requires two hops. Two-phase commit (2PC) requires three hops. 33% latency reductions are not the kinds of goals that drive dramatic DBMS redesigns, even though tenths of seconds — i.e. 100s of milliseconds — matter in the kinds of environments where NoSQL is sprouting up.

Discussions of NoSQL design philosophies tend to quickly focus in on the matter of consistency. “Consistency”, however, turns out to be a rather overloaded concept, and confusion often ensues.

In this post I plan to address one essential subject, while ducking various related ones as hard as I can. It’s what Werner Vogel of Amazon called read-your-writes consistency (a term to which I was actually introduced by Justin Sheehy of Basho). It’s either identical or very similar to what is sometimes called immediate consistency, and presumably also to what Amazon has recently called the “read my last write” capability of SimpleDB.

This is something every database-savvy person should know about, but most so far still don’t. I didn’t myself until a few weeks ago.

Considering the many different kinds of consistency outlined in the Werner Vogel link above or in the Wikipedia consistency models article — whose names may not always be used in, er, a wholly consistent manner — I don’t think there’s much benefit to renaming read-your-writes consistency yet again. Rather, let’s just call it RYW consistency, come up with a way to pronounce “RYW”, and have done with it. (I suggest “ree-ooh”, which evokes two syllables from the original phrase. Thoughts?)

Definition: RYW (Read-Your-Writes) consistency is achieved when the system guarantees that, once a record has been updated, any attempt to read the record will return the updated value.

Here a “record” can be a row, a key-value pair, or any similar unit of data. An “update” can be whichever of insert/append or true change the system supports.

A conventional relational DBMS will almost always feature RYW consistency. Some NoSQL systems feature tunable consistency, in which — depending on your settings — RYW consistency may or many not be assured.

The core ideas of RYW consistency, as implemented in various NoSQL systems, are:

• Let N = the number of copies of each record distributed across nodes of a parallel system.
• Let W = the number of nodes that must successfully acknowledge a write  for it to be successfully committed. By definition, W <= N.
• Let R = the number of nodes that must send back the same value of a unit of data for it to be accepted as read by the system. By definition, R <= N.
• The greater N-R and N-W are, the more node or network failures you can typically tolerate without blocking work.
• As long as R + W > N, you are assured of RYW consistency.

That bolded part is the key point, and I suggest that you stop and convince yourself of it before reading further.

Example: Let N = 3, W = 2, and R = 2. Suppose you write a record successfully to at least two nodes out of three. Further suppose that you then poll all three of the nodes. Then the only way you can get two values that agree with each other is if at least one of them — and hence both — return the value that was correctly and successfully written to at least two nodes in the first place.

In a conventional parallel DBMS, N = R = W, which is to say N-R = N-W = 0. Thus, a single hardware failure causes data operations to fail too. For some applications — e.g., highly parallel OLTP web apps — that kind of fragility is deemed unacceptable.

On the other hand, if W< N, it is possible to construct edge cases in which two or more consecutive failures cause incorrect data values to actually be returned. So you want to clean up any discrepancies quickly and bring the system back to a consistent state. That is where the idea of eventual consistency comes in, although you definitely can — and in some famous NoSQL implementations actually do — have eventual consistency in a system that is not RYW consistent.

Much technology goes into eventual consistency, as well as into the data distribution and polling in the first place. And in tunable systems, the choices of N, R, and W — perhaps on a “table” by “table” basis — can get pretty interesting. I’m ducking all those subjects for now, however, not least because of how much I still have to learn about them.

One point I will note, however, is this — RYW consistency and table joins make for awkward companions. If you want to join two tables, each of them distributed across some kind of parallel cluster, there are only two possibilities:

• In most cases, the data you need to join is co-located on the same nodes.
• You’re going to have an awful lot of network traffic.

In an R = W = N scenario, co-location may be realistic. But when R < N and W < N, a join can return incorrect results even when both of the tables being joined would have been read correctly.

In our example above, we had N = 3 and R = W = 2. Single-table RYW consistency was ensured. But suppose you join two records, each of which had been written correctly to 2 out of 3 nodes — but with only 1 node being correct about both records. Then only that 1 node out of 3 will return a correct value for the join, and badness will ensue.

Any architecture I can think of to circumvent that problem results in — you guessed it — an awful lot of network traffic.

And that, folks, is a big part of why the NoSQL folks are so negative about joins.

Related link

• Query fault-tolerance
• Huan Liu’s skepticism as to whether RYW consistency causes a significant performance hit
• Daniel Abadi’s views on NoSQL design tradeoffs

• A little over a year ago, Julian Browne put up a great post on Eric Brewer’s CAP conjecture/theorem, which provides much of the impetus to relax the traditional requirement for atomicity/consistency.
• Even more directly inspirational to NoSQL technology development were two seminal papers: Google’s on BigTable and Amazon’s on Dynamo. (That said, I’m having trouble getting myself to actually read them from start to finish, especially since they’ve been superseded by subsequent technology development.)
• 10gen (the MongoDB guys) hosted a NoSQL conference yesterday. Much blogging has ensued. The best post I’ve seen so far was by Adam Marcus. I find the graph database notes near the bottom particularly interesting.
• Mark Callaghan hit back against the NoSQL movement hype, and in particular against the MySQL/memcached is passe‘ meme. On the other hand, he also bemoaned many failings of MySQL. On the third hand, he praised or at least expressed hope for a variety of MySQL-related technologies, including Tokutek’s TokuDB and Continuent’s Tungsten.
• In connection with that debate, Mark Rendle offered a funny rant, mainly pro-NoSQL, in the style of a Socratic dialogue.
• John Quinn of Digg recently described Digg’s move from MySQL to Cassandra, and outlined a lot of features Digg was adding to Cassandra, all of which it is open-sourcing.
• The NoSQL guys maintain their own long list of NoSQL-related links.

Roman’s latest venture is business intelligence firm Gooddata. Gooddata was founded in 2007 and has been soliciting and getting attention for a while, so I was surprised to learn that Gooddata officially launched just a few weeks ago. Anyhow, some less technical highlights of the Gooddata story include:

• Gooddata believes it makes BI easy to adopt, unlike every other BI vendor on the planet — not excluding the many other BI vendors who say the same thing about themselves.
• Gooddata is entirely cloud-based, specifically in the Amazon cloud. I.e., Gooddata is selling SaaS-based BI.
• Gooddata wants to sell to enterprises that are large enough to have more than a couple of BI users, and small enough not to be well served by the BI market leaders.
  • In revenue terms, this is the ever-popular $100 million – $1 billion market.
  • Specifically, Gooddata believes that those enterprises may have decent “back office” BI, but don’t have much in the front office. Gooddata wants to provide them with front office BI, which seems to basically mean CRM analytics. Gooddata sees this as a market in which QlikTech is the major player. Generally, Gooddata wants to emulate and go after QlikTech.
  • Even more specifically, Gooddata wants to sell to Salesforce.com customers, who it believes are not well-served by what passes for built-in analytics at Salesforce. Partnering with NetSuite didn’t work as well, since NetSuite’s customers turn out to be smaller firms than are in Gooddata’s target market.
• Something I heard from both Jaspersoft and Gooddata is that there’s a hot market in providing cloud-based BI to online gaming companies. I gather these are mainly games running on mass communication platforms such as Facebook or the iPhone. Surely not coincidentally, it seems likely that:
  • These are small companies whose success – and hence data intake – can suddenly explode.
  • The data originates in cyberspace, with no particular need ever to come to the game companies’ own premises.
• Gooddata has 50 production customers.
• Gooddata had 2500 “projects” at the end of beta in June, and is adding 100 more per month. (Those numbers look weird together.) A “project” is a lot like a database, with associated reports, security privileges, etc.
• Gooddata has close to 40 people, mainly in development.
• I didn’t detect much of a sales strategy, nor much of a marketing strategy beyond the impressive early buzz generation. Perhaps that’s a partial explanation as to why the rate of Gooddata adoption fell even before the company officially launched.
• I forgot to ask what those 50 customers were actually paying, but considering Gooddata’s price list, it appears a typical price range for Gooddata’s stuff would be $500-$2,000/month.

Gooddata technical highlights include:

• Gooddata is building an entire BI stack – reporting, dashboards, ETL, in-memory database management, everything. I doubt Gooddata would claim that the pieces are best-of-breed in many ways other than BI ease of adoption and use.
• So far I’ve seen three Gooddata ease-of-use features or feature groups that strike me as differentiated – reusability (of metrics and/or reports), collaboration, and tag clouds. More on those below. Gooddata is also building toward an agility pitch, but those features aren’t all baked yet.
• Gooddata is MySQL-based today, but plans to move to a memory-centric compressed column store in 2010. Roman doesn’t reject analogies to SAP’s BI/BW/whatever Accelerator. Yes, folks – Gooddata is yet another BI vendor doing some form of memory-centric OLAP. That’s a big trend.
• I’m guessing that a big reason Gooddata is reinventing so many technical wheels is to ensure that the Gooddata stack is seamlessly multi-tenant from top to bottom. (Hasso Plattner of SAP’s comments on a similar idea suggest a similar emphasis.)
• Gooddata has its own multidimensional query language called MAQL (the A doesn’t seem to stand for anything). Today MAQL generates SQL for MySQL. The future columnar memory-centric data store will — I think – understand MAQL natively.

Now we get to the good stuff. When I wrote about reinventing business intelligence back in May, I focused on some interesting developments I see as actually underway — at least on an experimental basis and/or from small vendors – namely:

• Text-search interfaces. Well, while I didn’t see true text search in the Gooddata demo, I did see tag clouds, which have some of the same benefits.
• Collaboration tools. Well, Gooddata has a nice-looking approach to BI collaboration, heavily reflected in its UI metaphors. (That said, I haven’t really compared Gooddata to Microsoft SharePoint or SAP’s Portal/Rooms/whatever.)
• Memory-centric analytics (for speed of exploration). As noted above, Gooddata has that coming soon.
• Data exploration that tries to ignore fixed relational schemas, ala Attivio or Splunk. Roman says Gooddata is interested in or working on that, but offers no timetable.

Meanwhile, something I’ve been seeking for years, but haven’t seen much progress on since enhancement stopped on Cognos Metrics Manager, is more user-friendly metrics management. Well, it doesn’t have a lot of bells and whistles, but at least Gooddata has the basics – a list of already-defined metrics, and a reasonable way of compounding them into other metrics. I think that kind of thing will be a major BI feature going forward, to the point that a few years from now we’ll be worrying about how to port them from one BI vendor’s tool from another.

Bottom line: If you’re interested in BI, you should look at a Gooddata demo.

]]> http://www.dbms2.com/2009/12/27/introduction-to-gooddata/feed/ 12 http://www.dbms2.com/2009/05/29/sneakernet-to-the-cloud/ http://www.dbms2.com/2009/05/29/sneakernet-to-the-cloud/#comments Sat, 30 May 2009 03:06:04 +0000 Curt Monash http://www.dbms2.com/?p=793 Recently, Amazon CTO Werner Vogels put up a blog post which suggested that, now and in the future, the best way to get large databases into the cloud is via sneakernet.  In some circumstances, he is surely right. Possible implications include:

• When sending data to the cloud, you probably want to compress it to the max before sending. Clearpace’s new RainStor structured-data archiving service emphasizes that idea. RainStor marketing says cloud, cloud, cloud — but Clearpace thinks you really should have a bit of its software onsite too, to compress the data before sending it across the wire.
• Getting data from one cloud to another cloud could be problematic. I’m fond of saying that weblog data naturally lives in the cloud at your hosting company’s location, so you should analyze it there too. But this makes the most sense if you analyze it or at least filter/reduce it in place.  (That said, the really, really big web companies have lots of different data centers, and presumably do move huge amounts of log data from place to place.)

But for one-time moves of data sets — sure, sneaker net/snail mail should work just fine.

Amazon managers found that an employee who happened to work in France had filled out a field incorrectly and more than 50,000 items got flipped over to be flagged as “adult,” the source said. (Technically, the flag for adult content was flipped from ‘false’ to ‘true.’)

“It’s no big policy change, just some field that’s been around forever filled out incorrectly,” the source said.

Amazon employees worked on the problem well past midnight, and then handed it over to an international team, he said.

This was the best practice for reversing an error — how? Is SimpleDB somehow implicated? If this story is remotely true, and if there’s a sensible database architecture, I can’t imagine why there wouldn’t be a faster fix.

This seems like a great way to experiment with MapReduce and see if you like it. But for serious use, I don’t know why you wouldn’t prefer MapReduce more closely integrated into a DBMS.

Derivative funds and large government-subsidized entities will be especially interested in FACE’s transmodal operation. They can allocate a dedicated FACE, load it up with data, and then send it out to sea to perform advanced processing in safety. The government will have absolutely no chance of acting against them, because they will be too busy trying to decide which Federal Air Regulation (FAR) was violated, not to mention scheduling news conferences.

First excellent April Fool’s joke I saw this year was from The Guardian.  The best so far is from Expedia.  Others are linked in my Twitter feed.  And personally, I’m encouraging the concept of April No-Fooling Day.