July 1, 2009

NoSQL?

Eric Lai emailed today to ask what I thought about the NoSQL folks, and especially whether I thought their ideas were useful for enterprises in general, as opposed to just Web 2.0 companies. That was the first I heard of NoSQL, which seems to be a community discussing SQL alternatives popular among the cloud/big-web-company set, such as BigTable, Hadoop, Cassandra and so on. My short answers are:

As for the longer form, let me start by noting that there are two main kinds of reason for not liking SQL. First, you might be fine with the idea of a (somewhat) nonprocedural, schema-aware DML/DDL (Data Manipulation/Description Language), but just think another kind is better, or more suited to your use case. If your reason is like that, you might favor alternatives such as:

The second class of reason for avoiding SQL is because you don’t like the idea of a separate schema-aware DML at all. Possible reasons for this orientation include:

That last point is not a joke. One of the weirder database architectures I know of is the one underlying Guild Wars. Its developer — a brilliantly impressive guy — told me flat-out that he learned in college how to build a DBMS, but he didn’t learn how to develop for a conventional one. This was instrumental in his decision to build an unconventional data management architecture that uses SQL Server as little more than a smart file manager.

The questions of SQL performance and — often-unspecified — “overhead” are interesting to view through the lens of the H-Store/VoltDB project. Mike Stonebraker et al.:

Mike himself, of course, has been all over the spectrum on SQL-like languages. First he favored QUEL vigorously over SQL for mainstream relational DBMS. Then he led the charge to extend SQL in PostgreSQL, Illustra, et al. Then he actually staked out a contrarian position in the area of complex event/stream processing by favoring a SQL-like language in an area where other alternatives were better established — but that was at what turned into StreamBase, which now emphasizes visual programming over any kind of coding language.

I need to write much more about schema flexibility, but tonight — which will be my third straight of <<8 hours sleep — is not the time for that.

Comments

30 Responses to “NoSQL?”

  1. Guy Bayes on July 1st, 2009 1:02 pm

    I think some of your points are valid, but others are a little straw-man for my taste. I think the main logic pitfall is framing that “alternative” means ” total replacement for”. As usually is true, both extreme viewpoints are fallacies, nothing is a panacea and the truth lies somewhere in the middle.

    For example Map/Reduce is not a “total replacement for” SQL and RDBMS however it is a viable alternative for some problems currently being solved in relatively inefficient means by SQL and it’s various bastard children (PL/SQL, COBOL< flavors of ETL and the rest).

    This line was especially silly.

    “*You just like to program, and want to manipulate stored data the same way you do anything else. Thus, you are bothered by an “impedance mismatch” between SQL and your favorite programming languages. This is real. It also has been overcome by many, many enterprises around the world.”

    Has it really now? Been completely overcome in a completely efficient way? Then why do we still have more lines of Cobol in the world then all other languages combined?

    SQL is a very useful language. It is a powerful language. It is very well suited for many tasks. It is completely unsuited for many tasks. Some of the tasks it is unsuited for are data related and analysis related. The industry has tried to solve these issues in a number of less then totally successful ways mosltly through a stable of pretty popular data analyis and processing tools that are not SQL based.

    The new technologies that are coming into play do nothing more then add to that stable. They are doubtlessly being oversold as SQL replacements. However, the certainly are useful and will doubtlessly perform some data and analysis jobs better then SQL. Enterprises will find benefits in migrating some things.

  2. Justin Swanhart on July 1st, 2009 5:21 pm

    MapReduce is all about throwing massive amounts of resources at a problem in order to get it done in a short amount of time.

    MapReduce is not “green” and it represents the opposite direction (IMHO) from where the industry needs to move.

    If you want to be green, you have to do more with less. If you want to attract existing users, then you need to solve existing problems, which are mostly with SQL. I really don’t think most shops are going to gain ground by abandoning the query language spoken by just about every database developer and integration product in the world.

  3. Jerome Pineau on July 1st, 2009 5:52 pm

    I see MR more as an ELT tool really — not a replacement strategy for SQL. Maybe I’m missing something.

  4. Jerome Pineau on July 1st, 2009 5:54 pm

    Regarding “schema flexibility” which I typically call being “schema agnostic”, I describe how the XSPRADA product exhibits that trait in my blog here http://jeromepineau.blogspot.com/2009/06/can-rover-roll-over-too.html

    None of which is incompatible with SQL in anyway.

  5. Justin Swanhart on July 1st, 2009 8:03 pm

    Also, any generic RDBMS is fairly elastic as schemas go. Isn’t this generally how data warehouses bloat over time? Data is repeatedly denormalized, materialized, copied, shredded, spindled, mutilated and then copied again.

    XSPRADA may be an exception there, given the unique way it uses algebraic relationships to manage extended sets. I internally try visualize what they do using “bag algebra” but I don’t fully understand it. I’ve played a bit with it, and it is definitely an interested technology to keep an eye on.

  6. Curt Monash on July 1st, 2009 9:47 pm

    Guy,

    What part of the “No” in “NoSQL” didn’t you understand? :D

    Seriously, Mike Stonebraker delivered a blistering critique of SQL at SIGMOD today, for reasons not unlike your comments. Unfortunately, I didn’t ask him to expand on it afterwards …

  7. Bo Reid on July 1st, 2009 10:37 pm

    I’ve actually used a commercial Non-SQL, Object Oriented analytical database that uses a LISP variant to access data. Although the learning curve was steeper, especially for those without any OO experience, it was vastly faster to program and at least 10 – 100 times faster in running time series related analyses.

    The database name is Vision and it was acquired by Factset Data Systems, and was marketed as “FAST”. It was in use at some of the the top 50 asset management systems in the world.

    Every time I look at PL/SQL or Transact-SQL of code that is longer about 200 lines, I start to sigh and start whining to the nearest person about how great and easy things were with this non-SQL system.

    I read that Pick programmers feel the same way.
    I’d be curious to see anyone believes that a non-SQL oriented, real 4GL analytical language will ever be adopted outside of narrow industry niches and replace SQL for transactional and analytical purposes. I don’t see this happening. I think it is about as likely as English drivers voting to drive on the right side of the road.

  8. Jonathan Ellis on July 2nd, 2009 11:47 am

    NoSQL is about scaling, particularly scaling writes, not about antipathy towards an ancient query language. You can’t have both just-add-machines-and-push-a-button scaling of writes and traditional ACID SQL support; Google’s GQL on App Engine perhaps comes closest but even that is only a superficial resemblance.

    So I’d have to disagree with the original post and say that these solutions — some more than others — are useful for anyone with massive write volumes.

    (I work for Rackspace Hosting on the Cassandra database.)

  9. Jerome Pineau on July 2nd, 2009 1:13 pm

    “XSPRADA may be an exception there” – I think so, because another thing we dont do is “load” data to begin with. We only ingest it when queried and only then do we do data conversions as needed. So we dont convert data types from the get go. This is why it doesnt matter if you store everything as text in a CSV file (notwithstanding the obvious size penalties involved). At the end of the day we only convert JIT mode.

  10. Guy Bayes on July 2nd, 2009 2:42 pm

    Kurt, we agree, poor choice of names on their part :-)

    http://nosql.eventbrite.com/

    Seems obvious that they are interested in exploring alternatives to SQL based systems for particular problems and work loads, not advocating “lets kill sql entirely from the planet for all of time”.

    I’m sure there is much over-hype.

  11. Casper Bang on July 2nd, 2009 2:54 pm

    I agree with Guy Bayes, the impedance mismatch is very much there and while there are workaround and tools for binding these layers, these all come with a price to pay. I rather like how Ted Neward refers to this issue, namely the Vietnam of Computer Science.

  12. Cubegeek on July 2nd, 2009 8:37 pm

    Part of the difficulty with the NoSQL crowd, I think, is that they mischaracterize the very nature of enterprise computing which is very structural and procedurually oriented. Scaling writes doesn’t help manage workflow or handle complex business rules and those are the kinds of issues that harangue folks in enterprise space.

    It’s true that relational database schemas are relatively inflexible and transaction processes get locked in because of this, but that problem has been largely solved by column based databases. It’s ironic that Stonebreaker is referenced here because he’s involved in Vertica, a column based database built for the cloud that uses an SQL dialect and Map Reduce.

    From my perspective, someone who has worked extensively in data warehousing and analytics with the best of relational and multidimensional technologies, there is much to admire and fear in the scalability leapfrogging that the webguys have accomplished. But I think very few of them have dealt with workflows any more complex than the path of an ecommerce shopping cart. The skills to replace relational technologies requires much more of the mindset of game programmers than of web programmers.

    I say this also because I too see the impedence mismatch as the key problem, considering all the new jive in the middle tier over the past decade. A real “4GL” integrated programming environment is what some of these NoSQL guys need to live in building something like a medical patient records application. I think that will throw a bit of water on their fire.

  13. Log Buffer #152: a Carnival of the Vanities for DBAs | Pythian Group Blog on July 3rd, 2009 12:48 pm

    [...] SQL?!? Edwards, you’re mad!) Well, it’s not me. Here’s Curt Monash on NoSQL: “Eric Lai emailed today to ask what I thought about the NoSQL folks, and especially whether [...]

  14. Curt Monash on July 3rd, 2009 3:26 pm

    Jonathan,

    What do you guys use Cassandra for? Generic web and network event logs? Other stuff?

    And what kind of write volumes would be a good threshold for, at places like yours, saying “SQL isn’t worth its baggage?”

  15. Curt Monash on July 3rd, 2009 3:30 pm

    Cubegeek,

    What were you thinking of from the game programming side? I just went to the Cornell guys’ SIGMOD tutorial on database-like technology in game design, and it was pretty interesting stuff.

    http://www.cs.cornell.edu/bigreddata/games/

  16. Cubegeek on July 3rd, 2009 8:21 pm

    I think game programmers have, at a high level of abstraction, an implicit understanding of the roles in which players are involved and how they can mature in the context of and expansive and dynamic environment. Whereas corporate IT and the enterprise software developers who build products for them are very narrowly focused and have no contextual idea of how people work in multiple environments.

    Consequently, the user experience of integrating data from one area of the business is tedious and gets tiresome very quickly, whereas one can play videogames in completely alien environments for hundreds of hours and enjoy doing so.

    Take a well understood business concept like supply chain or inventory management. For something as simple as visualization, have you ever seen anything that resembles creativity used to present the current inventory levels in a company’s warehouse? I haven’t. It has been a bar chart for 20 years.

    I have yet to see, in a Fortune 500 company, any BI that functions in realtime with as much creativity and immersion as Maxis did with A-Train in 1992. If I presented to a corporate financial planning and analysis group some UI with an interface as complex as.. oh say the main screen of an XBOX 360 I’d either be hailed as a genius or shot on sight.

    The status quo for organizing the finances of a million businesses is a warren of Excel spreadsheets in a complex nest of shared file servers. Same as it was in 1990. You can’t even visually view the links between the spreadsheets. Much less build abstract ways of dealing with the workflow.

    The next generation of mouse, thumb and touch literate users are not going to stand for it. The only question is when is the revolution going to begin.

  17. Data, Death, and Reality Distortion Fields on July 4th, 2009 11:41 am

    [...] NoSQL? (dbms2.com) [...]

  18. Jerome Pineau on July 4th, 2009 12:44 pm

    @Cubegeek: “It’s true that relational database schemas are relatively inflexible and transaction processes get locked in because of this, but that problem has been largely solved by column based databases.”

    I dont see how columnar engines have addressed this problem — they remain relational databases. The fact that they manage storage in columnar fashion doesnt mean they’re either non-relational or more flexible with schemas. I dont think anyone is this columnar space is trying to replace relational.

    “…he’s involved in Vertica, a column based database built for the cloud that uses an SQL dialect and Map Reduce.”

    I dont believe Vertica was built for the cloud at all. Initially there was no talk of any SaaS or on-demand offering. They jumped on the bandwagon conveniently when possible. Furthermore, what does it mean to be “built” for the cloud when essentially at issue there is delivery mechanism more so than the actual backend engine IMHO.

    Good discussion on here!

  19. Cubegeek on July 4th, 2009 7:15 pm

    I’m taking people at their word that Vertica is as flexible as Oracle (Hyperion) Essbase when they draw equivalences. Essbase is very flexible when it comes to adding ‘columns’ on demand. Microsoft Analysis Services is as well to a lesser degree.

    I think hub & spoke methodologies will be adapted to the new super scalable storage technologies. I’m trying to think of ways to build systems using optimized combinations of databases – but I’m a database guy so that figures.

    When I think of ‘built for the cloud’ I’m thinking about using smart partitioning logic that scales across racks of relatively small servers and some parallelization. Devil’s in the details. But I cannot get over the prejudice, and please tell me if I’m wrong, that NoSQL is adapted by people who think first of MySQL as representative of relational tech – ie guys who build websites.

  20. Curt Monash on July 5th, 2009 4:11 pm

    Vertica was designed from the ground up to be MPP. They’re in the “Whatever else was missing in the Release 1, we got MPP right the first time” camp, along with Aster, rather than the “MPP is hard but we got it right after a couple of tries, and by the way we’ve been doing this longer than the other guys” camp, where Greenplum sits.

    Among the new vendors, the former group tend to have more compelling automatic-redistribution-among-nodes stories than the latter group, but it’s not clear the edge is a big one.

    To that, that’s about as cloud-oriented as DBMS designs seem to get.

  21. Avi Rappoport, SearchTools on July 8th, 2009 9:03 pm

    I do a lot with full-text search and that is a case where it just doesn’t match the architecture of SQL-oriented DBMSs. Most of the content is unstructured and almost all that is tagged or fielded is text rather than numeric. Inverted indexes are hugely more efficient for fast lookup than anything else, and the transaction code in DBMS is enough of an overhead to kill performance. Sure, there have been SQL-like query languages, but for the raw work of keyword search, SQL just doesn’t work. (And if you want to get me really ranting, let me tell you about the so-called MySQL full-text search, it strinks.)

  22. Curt Monash on July 8th, 2009 9:26 pm

    Avi,

    I’ve long been with you on that one. :)

    http://www.dbms2.com/2005/12/09/relational-dbms-versus-text-data/

    That said, I once tried to build a start-up around Oracle’s text search, especially from 7.3.2 to 8.0.4 Whoops …

  23. cruppstahl’s blog » NoSQL Databases gaining momentum on July 17th, 2009 8:13 am

    [...] And from the other side – Curt Monash on NoSQL (BI point of view): http://www.dbms2.com/2009/07/01/nosql-sql-alternative/ [...]

  24. Dinar on July 25th, 2009 6:22 am

    Guy Bayes:”Kurt, we agree, poor choice of names on their part :)
    http://nosql.eventbrite.com/
    yes, this is called NoSQL, but it is like about non-relational databases, not about not using SQL: http://nosql.eventbrite.com/ :’This meetup is about “open source, distributed, non relational databases”.
    Have you run into limitations with traditional relational databases?’, http://blog.oskarsson.nu/2009/06/nosql-debrief.html :’The idea was to give attendees a solid introduction to how distributed, non relational databases work as well as an overview of the various projects out there.’.
    What about whether SQL language is really needed to use with relational databases?
    And i thought this way:
    Curt Monash: “You just like to program, and want to manipulate stored data the same way you do anything else. …” and “You lack familiarity with SQL.”
    i have asked about that in #php irc channel and have published the talk: http://qdb.wp.kukmara.ru/2009/07/15/is-sql-language-really-needed-to-c-like-language-to-work-with-db/ – and there are some c-like syntax instead of SQL.

  25. NoSQL Movement is Gaining Traction | Software News Daily on September 16th, 2009 1:56 pm

    [...] work with relational databases and they have no reason to change that at the moment. An article on Monash from yesterday lays out both sides of this argument and makes some good [...]

  26. Damián Digital » Archivo » NoSQL: el movimiento en contra de las bases de datos on November 2nd, 2009 4:09 pm

    [...] pesar de todo el potencial, la mayoría de las organizaciones todavía no necesitan preocuparse por lo que se pierden, dice [...]

  27. NoSQL Q and A | DBMS2 -- DataBase Management System Services on December 12th, 2009 12:17 am

    [...] Avoiding joins is a big deal because a lot of programmers didn’t learn SQL in school. Also, joins can be computationally expensive. I wrote about some of the problems with fixed schemas here and, specifically in connection with NoSQL, here. [...]

  28. Nati Shalom on December 12th, 2009 6:51 pm

    There are few other points that may explain the rational behind the NOSQL movement.

    * Actual disk failure/year is 3% (vs. estimates of 0.5 – 0.9%) – this is a 600% difference on reported vs. actual disk failure.
    * There is NO correlation between failure rate and disk type – whether it is SCSI, SATA, or fiber channel.
    * There is NO correlation between high disk temperature and failure rates

    Those analysis shows that the approach of relying on a shared storage for reliability as with most RAC clusters is broken. Instead NOSQL approach assumes that failure are inevitable and where designed to deal with those failure under extreme scenarios.

    I summarized that topic on one of my recent post: Why Existing Databases (RAC) are So Breakable!:
    http://natishalom.typepad.com/nati_shaloms_blog/2009/11/why-existing-databases-rac-are-so-breakable.html

  29. Relevance in the datacenter « JZ Talk Blogger on January 21st, 2010 3:16 am

    [...] in the datacenter Do you know SQL or do you NoSQL? MySQL has been very popular for internet-scale deployments. But times have changed and there are [...]

  30. El movimiento en contra de las BBDD : : DESARROLLOAGIL on March 1st, 2010 3:13 am

    [...] pesar de todo el potencial, la mayoría de las organizaciones todavía no necesitan preocuparse por lo que se pierden, dice [...]

Leave a Reply




Feed: DBMS (database management system), DW (data warehousing), BI (business intelligence), and analytics technology Subscribe to the Monash Research feed via RSS or email:

Login

Search our blogs and white papers

Monash Research blogs

User consulting

Building a short list? Refining your strategic plan? We can help.

Vendor advisory

We tell vendors what's happening -- and, more important, what they should do about it.

Monash Research highlights

Learn about white papers, webcasts, and blog highlights, by RSS or email.