• Both Northscale and Gear6 offered distributions of memcached.
• Both Northscale and Gear6 also wanted to sell persistent versions of memcached — in essence, simple DBMS with the memcached API in place of a substantial DML (Data Manipulation Language).

Differences included:

• Gear6 hit the market earlier.
• Gear6 forked away from open source memcached.
• Gear6 sold its memcached distribution, while Northscale’s is free. (But you’re encouraged to pay for support.)
• Gear6 is also in the caching appliance business.
• Northscale and Gear6 had different approaches to beefing up the memcached API (especially for their respective persistent memcached stores).

However, both GigaOm and eWeek are reporting that Gear6 is headed for liquidation. Even though the source of the story seems to be Gear6’s caching appliance rival Schooner, the rumor sounds legit, not least due to Gear6’s conspicuous lack of denial. Of course, we can’t rule out some kind of last-minute transaction that either keeps Gear6 going in some capacity, or else gets its technology into some other vendor’s hands.

Metrics on Gear6 as of my briefing a few weeks ago included:

• A dozen or so announced (software?) customers
• “Dozens” of paying customers in the Amazon cloud
• “Tons” of free customers in the Amazon cloud
• 25-30 employees
• Low-end pricing of $3500/year/server or, on Amazon, $0.80/hour
• Biggest known installation = about 20 servers clustered together

I of course still plan to write about Northscale and memcached, but first I wanted to get Gear6’s apparent death throes out of the way in a separate post.

Incidentally, memcached is by no means Gear6’s original business. Gear6 was founded in 2002 as Engineered Intelligence Corporation to do something in the area of general high-performance clustering. In 2006 Gear6 said it was “solely focused” on storage, specifically the “server/storage gap”. By 2008 Gear6’s business was high-performance caching appliances, not necessarily memcached-specific. memcached, in software, appliance, and cloud-based formats, came more recently.

… or something like that. I don’t intend to write seriously about NorthScale until I have a better idea of what MemBase is.

In the mean time,

• VentureBeat put up a solid post on NorthScale’s company history and so on
• Om Malik bought into the NorthScale memcached pitch
• TechCrunch has a low-quality post about NorthScale (although it wasn’t as error-riddled as the same author’s post about nStein, which Seth Grimes properly blasted)

The top two points from Q&A probably were:

• Big Data and the cloud actually have relatively little to do with each other, a few exceptions notwithstanding, especially if the data is in a shared-nothing DBMS (as opposed to, say, a MapReduce-oriented file cluster). Two principal reasons are:
  • Redistributing data from node to node is a little slow, undermining some of the elasticity benefits of the cloud.
  • Getting data into the cloud in the first place is a lot slow.
• The NoSQL movement is a lot like the Ron Paul campaign — it consists of people who are dissatisfied with the status quo, whose dissatisfaction has a lot to do with insufficient liberty and/or excessive expenditure, and who otherwise don’t have a whole lot in common with each other.

Anyhow, here are my notes for the talk, edited in just a couple of places for readability or linkage.

Quick introduction

• Big Data vs. cloud
• How big is Big Data?
• At the low end of that range, there’s little you can’t do with conventional technology if you have:
  • An unlimited budget for hardware
  • An unlimited budget for software
  • An unlimited budget for people, especially Oracle DBAs

Big Data in OLTP

• Hard-core OLTP
  • Focus of DBMS technology for a long-time
  • Big budgets because each transaction has significant value
  • Tough to get users to change technologies
• Lighter-weight OLTP
  • Classic example = web companies
    • Big ones — retail-oriented ones (eBay, Amazon) partially excepted — rolled their own technology stacks
    • Reluctant to give money to anybody
      • Open source, etc.
  • Difficulty finding market
    • Product vs. feature
      • Clustering/HA/DR/whatever
      • Ditto cloud enablement
    • True products haven’t found much traction yet

Analytic Big Data use cases

• Kinds of data for analytics
  • More of same != big
  • More detail and/or new kinds
    • Complete data sets
    • Transactions
    • Call details
    • Tick/trade history
    • Web clickstreams
    • Network event logs
    • Other machine-generated data
    • CAM bottom line
      • Anything human-generated should and will be retained in its entirety
      • Quantities of machine-generated data retained should and will grow roughly in line w/ computing cost reductions (Moore’s Law, etc.)
• Analytic uses of Big Data
  • Analytics is mainly about three things
    • Problem detection
    • Customer relationship improvement
      • (Those overlap when the customer relationship is bad)
    • Financial statements on steroids
  • Main kinds of analytics
    • What BI vendors traditionally sell
      • General reporting and dashboards
      • Ad-hoc query (now driven from those reports and dashboards)
      • Planning (allegedly integrated with BI)
    • Research
      • Ad hoc relational query (worth mentioning twice because it drives so much of the market)
      • Data mining
      • Most web search and web mining
    • Operational/near-real-time
    • Archiving/compliance
  • What gets Big?
    • Mainly research and archiving
    • But when reporting or operational get Big, you have really interesting computing problems

Technology issues and trends

• Moore’s Law
  • CPUs — All about cores, hence parallelism is key
  • RAM
  • SSDs – hence replace disks
  • Sensors – hence generate lots more data
• Kryder’s Law
  • But rotational speeds up only 12.5X since Eisenhower Administration
  • Hence solid-state memory (or RAM) will soon take over
• In the mean time, I/O bottlenecks have had to be beaten
  • Hence sequential scans
  • Hence index-light architectures
  • Hence columnar
• DBMS “overhead”
  • Raw license and maintenance fees – software increasing fraction of total
  • OLTP vestiges – locking and all that
  • DBAs
    • People costs = huge fraction of total
    • Index-lightness addresses
    • So does appliance
  • Many people don’t really know how to write SQL
• Configuration
  • Appliance/tightly-balanced
    • Netezza
    • Teradata earlier
    • Greenplum/Sun
    • Oracle
    • IBM
    • Microsoft/Madison
  • Commodity/do what you want
    • Vertica
    • Greenplum now
    • Infobright, Aster and others
    • MapReduce-oriented file systems
  • Extreme rigidity is silly
    • Teradata, Oracle have both signaled moving to more modularity
    • Big driver of that = heterogeneous storage
      • Cheap disk
      • Expensive disk
      • Solid-state
      • RAM
    • CPU/storage ratio is even more of a driver

Theoretically defensible ways to segment the market

• Latency requirements
  • High availability and low latency go together
• Query types
  • Simultaneous users for same
• Database size
• Budget

Actual segments right now

• Utter ADW/EDW
• Data mart
  • Size
  • Naturally columnar vs. naturally row-based
• Operational/frontline
• Less dramatic/smaller EDW

Dear Curt,

Thanks for the very nice story and perception on the XLDB meeting. It is a balanced view.

More philosophically I would add a few points:

1) A data management system architecture is a large collection of compromises amongst a number of competing parameters.

data management (hardware, data structure, algorithms, optimizers, languages) –> value-for-application

Given the cost to develop/maintain a dbms, we see only a few parameter constellations in the current product offerings. And the scientists have a hard time to explore the uncharted land, because of effort required and uncertain benefits. (The same holds for researchers in R&D labs of vendors.)

2) The research community needs a focus to move ahead. The array-dbms is such a focus, because it identifies an omission in the type structure being managed at all levels of a system. Articulation of this in the community will help to steer effort.

3) The recent ‘hype’ for going to a HadoopDB like approach should be positioned carefully. It is so far a single point experiment for a limited query domain space, carefully carved out to avoid all the issues that plague a distributed dbms. Within this space the techniques come from a different operating system functionality. [Not sure what he means by this.] It does not change the DBMS itself and as such it is a repetition of middleware solutions to handle a cluster of independent MySQL instances.

This paper might be worth having a look at http://ic2.epfl.ch/labos/publications/freenix2004.pdf

To generalize it to a complete solution e.g. calls for massive replication, to avoid that you have to ship data around during query execution. This is paid for with more expensive updates. Feasible in certain domains. [I'd frame that point just as saying Hadoop-based solutions are unlikely to do as well at reducing data shipping as the better MPP DBMS.]

]]> http://www.dbms2.com/2009/10/03/martin-kersten-on-issues-in-scientific-data-management/feed/ 3 http://www.dbms2.com/2009/09/11/xkoto-gridscale-highlights/ http://www.dbms2.com/2009/09/11/xkoto-gridscale-highlights/#comments Fri, 11 Sep 2009 18:36:03 +0000 Curt Monash http://www.dbms2.com/?p=881 I talked yesterday with cofounders Albert Lee and Ariff Kassam of Xkoto. Highlights included:

• Xkoto sells Gridscale, a clustering server for DB2 and, more recently, MS SQL Server.
• Xkoto Gridscale runs on a separate box, between the application and the database servers. This box is typically smaller and cheaper than the database server boxes.
• Xkoto most typically sells Gridscale into environments where there already are three database servers — one to do work, one for hot standby, and one for remote disaster recovery.
• In such environments, Gridscale’s big benefit is that you can distribute the query workload among all three servers. Xkoto believes this big performance increase is the reason customers don’t get much past 3 database servers under Xkoto (they didn’t seem quite sure as to whether the all-time record was 4 or 5). Note that even if a remote server is a little too far away for OLTP query response, it can work fine for reporting.
• Of course, if you don’t already have high/”continuous” availability and/or disaster recovery, then Xkoto would say those are core benefits of Gridscale as well.
• Gridscale sends transactions (or just SQL statements?) to all servers in the cluster. Once any of them responds affirmatively, that update is reflected in queries. Gridscale maintains a small query log to make sure it gets the other database copies in sync. It also tries to make sure that queries always go to the most current copy of the database. (I didn’t ask what happens if Server A executes Transaction T but not U, while Server B executes Transaction U and not T — but that does seem like something of an edge case.).
• Xkoto spun out of Halcyon Monitoring in 2006, starting with DB2 support. Microsoft SQL Server support was introduced in 2008.
• Xkoto likes its partnerships with IBM and Microsoft. For example, IBM provides Level 1 and 2 support for Gridscale itself. Due in large part to this partnership strategy, Xkoto says it has no plans to support DBMS beyond DB2 and SQL Server.
• Instead, Xkoto is pursuing partnerships with large application vendors and so on. (The figure “about 10″ was mentioned.) I gather the idea is to make sure that neither the application support folks nor the app itself freak out from the fact that the app isn’t exactly talking to the DBMS any more.
• Xkoto has done lab tests suggesting Gridscale offers near-linear scalability (in terms of SQL Server database throughput) on a query-only workload up to 10 servers.
• I gather that Xkoto and IBM have demos suggesting it’s a fine idea to have your disaster recovery server be in the Amazon cloud, but they haven’t yet made any sales based on that — er, based on that premise.
• Gridscale pricing is measured in the same metrics as DB2 or SQL Server pricing, and in each case is around 1/3 what database pricing would be on the same box (I’m guessing that’s for enterprise additions without add-ons, but I didn’t probe). Specifically, Gridscale charges $12K per 100 PVUs for the DB2 edition, and $12K per socket for running with Microsoft SQL Server.
• Gridscale typically runs on smaller boxes than the databases it talks to.
• Xkoto has about 35 revenue-recognized customers. Most are on DB2, the first environment Gridscale supported.
• Average Gridscale selling prices are $180K on DB2, and $40-50K in the early going for SQL Server.
• Xkoto has about 40 full-time employees, with engineering in Toronto and business operations in Waltham.

The downside to Continuent Tungsten ’s stripped down architecture is that it doesn’t solve the most extreme performance scale-out problems. Instead, Continuent focuses on the other big benefits of keeping your data in more than one place, namely high availability and data loss prevention (i.e., backup).

Continuent has been around for a number of years, starting out in Finland but now being based in Silicon Valley. For most purposes, however, it’s reasonable to think of Continuent and Tungsten as start-up efforts.

As you might guess from the references to Finland and MySQL, Continuent’s products are open source, or at least have open source versions. I’m still a little fuzzy as to which features are open sourced and which are not. For that matter, I’m still unclear as to Tungsten’s feature list overall …

My client actually signed a deal for EnterpriseDB’s Postgres Plus Advanced Server and GridSQL, but unwound the transaction quickly. (They say EnterpriseDB was very gracious about the reversal.) There seem to have been two main reasons for the flip-flop.  First, it seems that EnterpriseDB’s version of Postgres isn’t up to PostgreSQL’s 8.4 feature set yet, although EnterpriseDB’s timetable for catching up might have tolerable. But GridSQL apparently is further behind yet, with no timetable for up-to-date PostgreSQL compatibility.  That was the dealbreaker.

The current base-case plan is to use generic open source PostgreSQL, with scale-out achieved via hand sharding, Hibernate, or … ??? Experience and thoughts along those lines would be much appreciated.

Another option for OLTP performance and scale-out is of course memory-centric options such as VoltDB or the Groovy SQL Switch.  But this client’s database is terabyte-scale, so hardware costs could be an issue, as of course could be product maturity.

By the way, a large fraction of these updates will be actual changes, as opposed to new records, in case that matters.  I expect that the schema being updated will be very simple — i.e., clearly simpler than in a classic order entry scenario.

Let’s start with the part everybody pretty much knows already:

• There are two parts to a parallelization story — how you get data off of disk, and what you do with it once you have it.
• To a first approximation, the best way to get a lot of data off of disk is in parallel, specifically with different CPUs talking to different disk drives. Until last week’s announcement of Exadata, Oracle was the most prominent holdout against this view. (That dubious honor now goes to Sybase.)
• If processing units are working in parallel to get data off disk, it is optimal to, on the same node that receives the data:
  • Do your projections. I.e., whittle down the data to just the columns you need. (That’s if you have a row-based system, as Oracle does; projections are moot in column-based systems.)
  • Do your selections. I.e., execute whichever WHERE clauses can be handled before the joins start.
  • Do any joins you can. For example, you may be joining two tables that have been partitioned on the same hash key, or you may be joining a large fact table to a small dimension table, where the latter has been replicated across every node.
• Exadata is the first mainstream Oracle product that follows this optimal storage-facing parallelization strategy. Actually, opinions differ as to whether rigid coupling of processors to specific disks is actually necessary. But after supporting one extreme (the disk part of shared-everything), Oracle with Exadata has gone to the other extreme (the disk part of shared-nothing). Other vendors taking this approach include Teradata and Netezza. Greenplum and Vertica are less extreme.
• After accessing data in parallel and filtering it to the extent possible on the nodes that retrieved it, Oracle then ships the data to a conventional Oracle database server. That server does any further query processing, along with any other analytics.

All that has been pretty clear from the getgo. Less obvious has been: How does the Oracle database server process the data it receives from the Exadata component? In particular, how parallel is the Oracle database server’s processing?

It turns out that the answer has little to do with Oracle Real Application Clusters (RAC). Indeed, it has so little to do with RAC that I’m wondering what RAC does to justify its >10% share of overall Oracle Database Machine pricing. In particular, different CPUs generally do not share RAM or cache when doing what Oracle refers to as DSS (Decision Support System — an old term) work. Thus — while I’m still not clear on all the specifics or exceptions! — it is generally fair to say that Oracle’s architecture on the database server side is akin to shared-nothing.

Please note that we’re talking here about two different pools of CPUs — the ones built into the Exadata part of the system, in charge of talking to their own private disk drives, and the ones in the RAC cluster, which do non-basic query execution, along with the rest of the analytics. Indeed, those two pools of CPUs could be of completely different brands and configurations, although at the moment they are similarly-named HP servers using identical Intel chips. I.e., Oracle has moved into the node heterogeneity camp. By way of contrast, the usual-suspect MPP vendors — Teradata, Netezza, Greenplum, Vertica, Aster Data, Paraccel, Exasol, DATAllegro — do most or all of their subsequent processing on the same nodes that retrieve data. Thus, Oracle is the first major vendor for whom it is important to remember that different parts of a query plan get parallelized across completely distinct sets of processors and processing units.*

*Yes, I know that each Netezza SPU (Snippet Processing Unit) couples a PowerPC chip and an FPGA (Field-Programmable Gate Array). But that’s a very different thing from having your data access occur on 14 servers and having the initial results sent to a different set of 8 servers.

So with all that background, I’m finally ready to lay out what I’ve gleaned about Oracle query and analytic parallelization, whether from public materials or private discussions.

• Commericial database parallelization started in the mid-1990s. Indeed, I was writing about it back in 1994, and it was part of the story in my Sybase Sell recommendation that year. Ken Jacobs et al. explained to me at the time that in the choice between “static” and “dynamic” partitioning and parallelization, Oracle had opted for the “dynamic.” I.e., all CPUs could look at all data, but there would be a form of parallel processing anyway.
• Oracle introduced a feature called Parallel Query long ago. Then, with the release of 10g, Oracle said in effect “Now we’ve removed the prior limitations and gotten Parallel Query right!”
• Oracle tables or queries have to be explicitly enabled for parallelization. Degrees of parallelization (minimum and maximum processors devoted to a task) are also explicitly declared. Naturally, there are defaults and administrative tools that make that all pretty automatic or else easy. The default default, as it were, is for most or all available processors to be used in parallel.
• Oracle also says it has parallelized a broad range of analytic functionality — data mining, OLAP, and so on, including generic UDFs (User-Defined Functions).

So how good is all this parallel technology? On the one hand, we know Oracle has been shipping it for a long time, and has it widely deployed. On the other, we also know that Oracle performance has been very problematic for large parallel queries. Surely most of those problems were due to the shared-disk bottleneck. But were they all (or so close to all as not to matter)? I don’t yet know.

If a shared disk architecture is not scalable, then how is it that Oracle consistently has more customers in the Winter Group top ten data warehouses than any other vendor?

Unfortunately, the Winter Corporation list is a joke, which may be why it hasn’t been updated since 2005. (I mean it — Dick Winter seems like too good a guy to keep publishing something so misleading indefinitely.) It counts not just user data, but indices, aggregates, and everything else. Based on that, I’d guess the largest Oracle site listed there to be at 10-20 terabytes of user data, and all the others to be at 5 TB or less. Even assuming 3-5X database growth since the list was compiled, that puts Oracle behind Teradata, DATAllegro, Netezza, Dataupia, and probably SAS — just counting ones I can think of quickly — none of whom are actually represented on the list. Teradata in particular blows Oracle away on warehouse size.

And by the way — the largest Oracle warehouse by far on that list is at Yahoo. But Oracle isn’t Yahoo’s major data warehouse software provider.

If a shared disk architecture is not scalable, then how is it that Oracle is the leader in Data Warehouse Performance. It is the TPC-H leader in the 300GB, 1TB, 3TB, and 30TB categories.

TPCs are a joke too. Oracle’s third-longest-serving exec (or maybe second-longest — I always forget whether he or Ken Jacobs has been there longer) e-mailed me a few years ago, asking for my help in making them go away. Be that as it may:

• Oracle probably won the 30 TB TPC-H because it’s the only vendor to submit a result.
• Oracle is the “leader” on the 10 TB TPC-H by 10% in price-performance, using a system that hasn’t shipped yet, over a system that has already shipped. 5 months is worth more than 10% in this day and age. Anyhow, the other contenders are Microsoft and IBM, which may be why Oracle finds it reasonably easy to keep up.
• Oracle trails Exasol by a factor of 27 — that is not a typo — on the 3 TB TPC-H.
• Oracle trails Exasol by a factor of 20 — also not a typo — on the 1 TB TPC-H.
• Oracle trails Kickfire by a factor of 14 — also not a typo — on the 300 GB TPC-H

A shared disk architecture (Oracle) is more flexible. Since all processing units can see all data the system can at runtime decide what the degree of parallelism should be. In addition, some queries may be more efficiently run in serial (simple index lookups) in which case parallelism isn’t even used.

Data warehouse appliances (at least the row-based ones) excel at fast table scans.

Also, if individual servers in a cluster contain many CPUs (or cores) the parallelism can be co-located on the node. Hence, statements may run in parallel but do not require the interconnect to ship data.

Appliance makers use multicore systems too. Everybody does, these days.

Oracle’s ‘Shared Everything’ architecture provides the ability to dynamically optimize each query requirement. The current workload is examined and the degree of parallelism is adjusted rather than blindly starting with the same degree of parallelism every time. Therefore, the degree of parallelism is optimized for every query and there is no requirement for a minimal degree of parallelism across all nodes. Operations can run in parallel using one, some or all nodes of a Real Application cluster depending on the current workload, the characteristics and importance of the query.

That’s all irrelevant to the chief benefit of parallelism. Parallelism isn’t about optimizing the use of CPUs. Parallelism is about optimizing the system where it’s actually bottlenecked, which is getting data off of disk.

Parallelism is not related to the partitioning strategy of the data as in a shared-nothing environment.

Parallelism isn’t particularly related to partitioning strategy in a shared-nothing environment either. Kognitio offers a competitive shared-nothing system with no partitioning whatsoever. And many queries on most vendors’ systems relate to partitioning only in that the data is distributed so that approximately equal amounts of data may be found on each node.

True, since that’s done by hash partitioning, you try to pick hash keys so that you get lucky as often as possible, and benefit from the hash key when doing a hash join. And further partitioning can be added as an optimization. But that’s hardly a disadvantage for shared-nothing systems vs. Oracle.

With Oracle’s shared disk approach, fail over is built-in and the configuration remains balanced.

If your disks are failing often enough for that to be more than a tiny benefit, you might want to consider changing your storage supplier.

Oracle does not require re-organization of data. Oracle’s hash partitioning is also automatic and does not require re-distribution of data. The Oracle Optimizer automatically tunes queries. In addition the Oracle Database 10g ADDM, (Automatic Database Diagnostic Monitor) runs automatically to make performance recommendations. Index management is very simple in Oracle. The ADDM tool recommends indexes, generates script to create indexes and will run them with the DBA’s approval. Oracle also supports all types of data including stars, normalized and de-normalized data. Oracle supports Join Indexes and Aggregate Join Indexes.

Somebody please remind me to start an international Scrabulous tournament for Oracle DBAs, since they have nothing else to occupy their time.

In addition, Oracle supports superior concurrency and parallelism, Oracle can execute several queries at the same time (in parallel) without performance degradation. With Oracle’s model, there are several checkout counters that customers can use, which parallelizes the process and provides a higher throughput. Even if a customer at one counter takes a long time to checkout, other customers are not affected. Once all checkout counters are full, Oracle queues the remaining customers (queries) until the next checkout counter opens up and sends the next customer in line to the open counter. If this starts occurring consistently, and the processing of customers (queries) slows down, Oracle allows for more checkout counters to be added dynamically using RAC or by simply adding more CPUs to an environment.

That one’s probably real.

Related links:

• Oracle’s Optimized Warehouse Initiative
• 4 reasons to reduce your Oracle usage