• Aster Data Analytic Foundation, a set of analytic packages prebuilt in Aster’s SQL-MapReduce
• Aster Data Developer Express, an Eclipse-based IDE (Integrated Development Environment) for developing and testing applications built on Aster nCluster, Aster SQL-MapReduce, and Aster Data Analytic Foundation

And in other Aster news:

• Along with the development GUI in Aster nCluster 4.5, there is also a new administrative GUI.
• Aster has certified that nCluster works with Fusion I/O boards, because at least one retail industry prospect cares. However, that in no way means that arm’s-length Fusion I/O certification is Aster’s ultimate solid-state memory strategy.
• I had the wrong impression about how far Aster/SAS integration has gotten. So far, it’s just at the connector level.

Aster Data Developer Express evidently does some cool stuff, like providing some sort of parallelism testing right on your desktop. It also generates lots of stub code, saving humans from the tedium of doing that. Useful, obviously.

But mainly, I want to write about the analytic packages. I’m not convinced that they’re a big deal in themselves yet, or that a whole lot of person-months have gone into their combined development. Still, I think they provide a great indication of one direction in which analytic functionality is going. And by the way, Aster promises to release a lot more of that kind of thing over the next 12 months.

Aster’s flagship analytic package is nPath, which is like a regular expression matcher, but for (time) series of data rather than for character strings. The main use for nPath is in pulling specific kinds of event sequences out of web or network event logs. However, one could imagine uses in other sectors that focus on temporal or sequential data (e.g., trading, intelligence, other sensor analysis), should existing SQL- and/or CEP-based technologies not prove sufficiently flexible. Aster 4.5 adds some new aggregation capabilities around nPath.

Other not-wholly-new packages in the Aster Data Analytic Foundation announcement are for sessionization (of clickstream data and the like) and tokenization (of text/character string data). While sessionization can be done in SQL, Aster thinks its MapReduce-based version is faster, since it doesn’t require self-joins. Makes sense. Aster’s tokenization sounds lame, however – text analytics in MapReduce tends to reinvent simplistic wheels for no clear reason, and Aster doesn’t seem to be an exception. (Aster would argue, however, that anything it does in SQL-MapReduce is more flexible than pure SQL or pure MapReduce alternatives.)

Another example of better-living-without-self-joins is Aster’s new market basket package. This lets you look at a set of point-of-sale data, pick a small integer N, and pull out all the sets of N things that were bought by the same person at the same time. I haven’t probed the claim in detail, but Aster implies there’s less combinatorial explosion in its approach than it is in the self-join alternative.

Note: Gartner highlighted self joins as a performance challenge in its recent Data Warehouse Magic Quadrant.

Aster is also releasing a few statistical and general analytic functions — specifically (and I quote a slide):

• exponential moving average
• weighted moving average
• simple moving average
• volume-weighted average price
• correlation
• linear regression
• logistic regression
• approximate_percentile
• approximate_count_distinct

The point of the last two items on the list is that if you set a non-zero tolerance for error, you can you can count things or order them into bins very efficiently – especially in terms of RAM — while being guaranteed not to exceed your error tolerance.

Note: One obvious inference from this list — which Aster gladly confirms — is that Aster has high hopes of selling to the financial services industry.

Finally, Aster is releasing its first pure graph-analytic function, for finding the shortest path between a given pair of nodes.

While I had the Aster folks on the phone anyway, I also took the opportunity to ask about the Aster nCluster 4.0 capability to create fairly persistent non-relational in-memory data structures. Specifically, I asked whether different users could access the same in-memory structure, and was told that this is a little klugey but not too horrendous. That suggests Aster’s capability may be a strict superset of UDF-based (User-Defined Function) approaches to meeting the same need, at least from a functionality standpoint. However, ease of creating those in-memory structures may still be better in the more SQL/UDF-centric approach favored by Teradata.

• This is a good idea, and many software vendors should do it if they aren’t already.
• It’s no big deal.

For reasons why, let me quote an email I just sent to an inquiring reporter:

• StreamBase sells mainly to the financial services and intelligence community markets. Neither group will share much in the way of core algorithms.
• But both groups are pretty interested in open source software even so. (I think for both the price and customizability benefits.)
• Open source software commonly gets community contributions for connectors, adapters, and (national) language translations.
• But useful contributions in other areas are much rarer.
• Linden Labs is one of StreamBase’s few significant customers outside its two core markets.
• All of the above are consistent with the press release (which quotes only one StreamBase customer — guess who?).

To understand Sybase’s plans for Aleri and CEP, it helps to understand Sybase’s current CEP-oriented offering, Sybase RAP. So far as I can tell, Sybase RAP has to date only been sold in the form of Sybase RAP: The Trading Edition. In that guise, Sybase RAP has been sold to >40 outfits since its May, 2008 launch, mainly big names in the investment banking and stock exchange sectors. If I understood correctly, the next target market for Sybase RAP is telcos, for real-time network tuning and management.

In addition to any domain-specific applications, Sybase RAP has three layers:

• CEP (Complex Event Processing). Sybase RAP CEP is based on a version of the Coral8 engine Sybase licensed and has been subsequently developing.
• In-memory DBMS. Sybase’s IMDB is part of (but I guess separable from) and has the same API as Sybase’s OLTP DBMS Adaptive Server Enterprise (ASE, aka Sybase Classic).
• Sybase IQ. Actually, Sybase used the phrase “based on Sybase IQ,” but I’m guessing it’s just Sybase IQ.

In theory, there could be a DBMS other than Sybase IQ, such as Sybase ASE or even Oracle, because Sybase IMDB can talk to a variety of DBMS. I didn’t get the impression, however, that in practice there were any Sybase RAP installations whose persistent DBMS was anything other than Sybase IQ.

Aleri had all along had something called Project Ohio, to merge Coral8 with Aleri Classic. Now Sybase’s own CEP engineering team is being added to the mix, schedules are being reconsidered and haven’t been disclosed yet. (If one woman can produce one baby in nine months, how long does it take nine women to produce a baby?) Apparently Sybase has a dozen programmers in the CEP area, plus ~20 more on Sybase RAP, not counting QA, documentation, etc.; that represents a significant bump to the overall Aleri development team.

Sybase doesn’t seem to have decided what to do yet with the various business intelligence/real-time OLAP engine products and technologies it is inheriting from Aleri.

And finally, some metrics:

• The Sybase/Aleri guys estimate that 1/3 of of Aleri’s customers and even less of its revenue came from outside the financial services sector. They did say the non-financial-services business was “starting to pick up,” but not very convincingly.
• Sybase IQ is now up to >1800 customers, with >200 new ones in 2009.
• Sybase IQ indeed has users taking in market feeds up to 3 terabytes a day, so it probably matches Vertica in having at least several-hundred-terabyte databases in the financial sector.

• The folks at Sybase still haven’t figured out when to prebrief me. (Edit: I’ve been briefed subsequently.)
• Sybase/Aleri is a potentially powerful combination, if they can effectively address the point I just made about integrating disparate latencies. That said, I’m not expecting a lot, because the CEP industry always disappoints me.
• Microsoft, IBM, and (somewhat less clearly) Oracle are all trying to do CEP inhouse. Sybase is making a good choice in having serious CEP inhouse itself
• Surely the main focus and financial justification for the Sybase/Aleri acquisition is the financial services market.
• Specifically, I expect the focus of technical integration between Aleri and Sybase’s DBMS products to start with Sybase IQ.
• Coral8 had some interesting ideas about how to integrate CEP with OLTP/operational BI, but I’m not aware that they got much traction.
• I bet there are use cases where Sybase tries and fails to sell Adaptive Server SQL Anywhere that CEP would be a better technical fit, but I don’t immediately see much practical business significance to that observation.
• While this deal could easily strengthen the Vertica/StreamBase partnership, I don’t see any reason why it would lead those two companies to actually merge.

Related link

• Thinking about analytic latency

• Traditional data of the sort that’s stored in relational databases, aka “structured.”
• Everything else, aka “unstructured” or “semi-structured” or “complex.”

There are plenty of problems with these formulations, not the least of which is that the supposedly “unstructured” data is the kind that actually tends to have interesting internal structures. But of the many reasons why these distinctions don’t tend to work very well, I think the most important one is that:

Databases shouldn’t be divided into just two categories. Even as a rough-cut approximation, they should be divided into three, namely:

• Human/Tabular data –i.e., human-generated data that fits well into relational tables or arrays
• Human/Nontabular data — i.e., all other data generated by humans
• Machine-Generated data

Even that trichotomy is grossly oversimplified, for reasons such as:

• These categories overlap.
• There are kinds of data that get into fuzzy border zones.
• Not all data in each category has all the same properties.

But at least as a starting point, I think this basic categorization has some value.

By human-generated data that fits well into relational tables or arrays, what I really mean is: the input from most conventional kinds transactions – purchase/sale, inventory/manufacturing, employment status change, etc. This is the core data managed by OLTP relational DBMS everywhere. It is also the core data in analytic relational or MOLAP databases. The vast majority of what we think or know about “database management” applies primarily to data of this kind, in large part because of two fundamental properties of this information:

• It is meaningful to contemplate this data as being 100% accurate and complete (even if that goal is difficult to achieve in the real world).
• This data is precise – i.e., one can check predicates against it and (give or take regrettable data imperfections) get inarguable yes/no answers.

For most enterprises, this is the most important data they have. It was created as a result of expensive business activities. It deals directly with money, employees, physical goods, and the rest of the things that make an enterprise go. It can be fruitfully analyzed in ever more ways, which is why it should never be thrown out or even entirely relegated to tape, now that data warehouse software, hardware, and storage has become so cheap. (“Disk is the new tape.”) And because of the importance of both preserving and accessing it, it should often be stored in multiple copies – OLTP, data warehouse, data mart, in-memory analytics, near-line quasi-archive, MOLAP cubes (if you must) and so on, plus of course replicas for high throughput and availability.

But humans generate many other kinds of data as well, especially in a form directly suitable for communication – text (in many formats), documents (text or otherwise), pictures, videos, etc. Traditional relational databases are a poor home for this kind of data because:

• This data often deals with opinions or aesthetic judgments – there is little concept of perfect accuracy.
• Similarly, there is little concept of perfect completeness.
• There’s also little concept of perfectly, unarguably accurate query results – different people will have different opinions as to what comprises good results for a search.
• Queries don’t lend themselves to binary answers; rather, documents can have differing degrees of relevancy.

Systems for managing this kind of data are much less advanced than relational database managers. Nobody knows how to get all the information out of a text document, or query all of it if they could, and the story is even worse for non-text examples. The systems that give the best query results aren’t necessarily the same ones that have the best database administration features. Basically, this area is still a mess, and it’s a mess that consumes a huge fraction of all the data storage products sold today.

But give or take questions of storage efficiency and deduplication, if humans created that kind of data, they put a lot of effort into it, so it’s worth keeping. Besides, compliance regulations commonly mandate that we do so – except, perhaps, when they mandate that we throw it away.

Machine-generated data is a whole other can of worms. Paradigmatic examples of what I mean by “machine-generated data” include:

• Computer, network, and other equipment logs
• Satellite and similar telemetry (whether for espionage or science)
• Location data such as RFID chip readings, GPS system output, etc.
• Temperature and other environmental sensor readings
• Sensor readings from factories, pipelines, etc.
• Output from many kinds of medical device, in hospitals and (increasingly) homes alike

Unlike human-generated data, whose growth is constrained by macro factors such as population and total level of economic activity, machine-generated data will continue to grow as fast as Moore’s Law lets it. That fact has two profound consequences:

• It is unrealistic to hope ever to keep most or all machine-generated data, whereas I think that’s exactly what should and will happen with human-generated data
• Before long, most data (by volume) will be machine-generated

And so it is not really an exaggeration to say that machine-generated data is the future of data management.

I’d like to close this long post by immediately pointing out some of the flaws in this simple trichotomy. One obvious gray area lies in hybrid human/machine-generated data, three big examples of which are:

• Web clickstreams
• Call detail records (CDR)
• Stock trades

In all three cases, we are quickly getting to the point where this data is preserved in its entirety (even if the network event data associated with the web logs is reduced before storage). And in each case it fits pretty well into RDBMS, although Hadoop has a role to play as well. So pretending it’s purely human-generated probably isn’t all that misleading.

Another gray area lies in text that gets linguistically processed – i.e. via text-mining tools – with the output placed into a relational database. I don’t immediately see a workaround for that flaw in my labeling scheme.  So let’s just say no taxonomy is perfect.*

*Come to think of it, that’s one of the problems holding back text-mining technology.

And of course some of the NoSQL folks would note that I was oversimplifying when I tied my first category specifically to relational DBMS. So would the folks at Intersystems.

But the biggest oversimplification stems from this:

As Mike Stonebraker* and I argued a couple of years ago, I really think that database management technologies should be divided into 10+ categories.

*Note: The links to Stonebraker’s own posts will be broken until Vertica’s webmaster gets his/her act together. But you can find them under other URLs via web search.)

]]> http://www.dbms2.com/2010/01/17/three-broad-categories-of-data/feed/ 9 http://www.dbms2.com/2010/01/15/there-sure-seem-to-be-a-lot-of-inaccuracies-on-paraccels-website/ http://www.dbms2.com/2010/01/15/there-sure-seem-to-be-a-lot-of-inaccuracies-on-paraccels-website/#comments Fri, 15 Jan 2010 04:47:00 +0000 Curt Monash http://www.dbms2.com/?p=1393 In what is actually an interesting post on database compression, ParAccel CTO Barry Zane threw in

Anyone who has met with us knows ParAccel shies away from hype.

But like many things ParAccel says, that is not true.

The latest whoppers came in the form of several customers ParAccel listed on its website who hadn’t actually bought ParAccel’s DBMS, nor even decided to do so. It is fairly common to to claim a customer win, then retract the claim due to lack of permission to disclose. But that’s not what happened in these cases. Based on emails helpfully shared by a ParAccel competitor competing in some of those accounts, it seems clear that ParAccel actually posted fabricated claims of customer wins.

Another thing that was both technically and substantively false was ParAccel’s claim to be CERTIFIED price-performance leader. Obviously, this was meant to give the impression that ParAccel had been “certified” as the leader in price/performance, when the closest thing to that that was remotely true was that ParAccel had a leading position in the category of “price/performance measurements that happen to have a certification process.” At least, that was true for a short time; then ParAccel’s certification was found to have been erroneous, and got revoked, which did not however inspire ParAccel to immediately take the claim off the front page of its website.

ParAccel’s website also reflects a lot of praise from flagship customer LatiNode. What it perhaps understandably neglects to mention is that LatiNode is in a dormant state, placed there by acquirer Elandia due to LatiNode’s criminally corrupt customer acquisition practices.

I also don’t believe ParAccel’s endlessly-repeated claim that is has never lost a benchmark on performance. However, I must in fairness note that while I’ve been given names of customers who are supposed counterexamples to this claim by somebody I trust, I’ve never been able to actually verify those supposed ParAccel losses.

• RainStor expects to finish the year with > 50 users (overwhelmingly via partners)
• A big market for RainStor (at least in terms of signed partnerships and large deal activity) is retention of telecom records, for compliance purposes, typically for a 1-3 year period. This includes:
  • CDRs (Call Detail Records)
  • Mobile phone records including CDRs and missed calls
  • SMS (Short Message Service), including the complete text of same
• RainStor thinks a number of larger telcos have the need to store a billion records per day each. (I’m not sure how many subscribers such a telco would have to have).
• John further thinks that, for the same query performance, RainStor can handle such a database on 4 blades. More precisely, he says that’s what happened at a test conducted by a major technology firm. In the same test case, SenSage required 40 blades, and Oracle required 80 or more cores on a pair of big SMP machines. John further says that the Oracle solution required a new table and new tablespace every day, while RainStor’s took 3 days for initial installation and required no DBA afterwards. However, I’m in no position to verify this report independently.
• In a different kind of proof point, so extreme it gives even the RainStor folks pause, a user has retired 300 different applications and put their databases onto a single 2-core box. (Presumably, this is via RainStor’s OEM relationship with Informatica.)
• Coming Very Soon are some services tying RainStor’s DBMS to obvious-suspect SaaS offerings. The core positioning is “SaaS data escrow”.i.e., RainStor will help you ensure that, in a worst-case scenario, there’s a nice safe copy of your data you can get at. RainStor also encourages you to do basic reporting and BI against the RainStor copy of the data, if you choose.
• The idea I’ve been pushing lately of taking a heterogeneous replication offering like Continuent’s and having it feed an archiving store like RainStor’s has hit a rather basic snag. RainStor doesn’t actually consume change data capture kinds of information directly, at least as of yet, because of difficulties fitting such a stream into its guaranteed-data-immutability model.

*I coined that category description for John in the tea room of the Park Lane Hotel. He’s subsequently embraced it enthusiastically, and I kind of like it myself.

Related links

• RainStor’s approach to compression, as described by me and by RainStor itself

• The same kinds of data are stored as before, with more being added over time.
• The same kinds of data are stored as before, but in more detail.
• New kinds of data are stored.

The first of those three ways doesn’t lead to dramatic growth. If a data warehouse goes up from 5 years of data to 6, then its overall size will grow a little over 20%. (How little depends on what the underlying business growth is – i.e., on how many more business events you have next year than you had 3 years ago.) That’s almost certainly going to be well-handled, by whatever technology manages your data warehouse today, given that:

• Chips are still subject to something resembling Moore’s Law.
• Disk capacity is still subject to Kryder’s Law, which is like Moore’s Law but with yet faster growth rates.
• DBMS software gets more performant over time.

So the cost of managing your same-as-before data will go down every year, even as the volume of that data grows.

True, disk rotation speeds have only increased 12.5 times since the Eisenhower Administration. But solid-state drives (SSDs) are getting practical for data warehousing fast, so even that bottleneck eventually will get swept away. And since what we’re discussing is, basically, the first and hence presumably highest-value data to be warehoused, it’s apt to wind up on SSDs before some other kinds of data warrant that treatment. So it’s the two other factors that drive the greatest data warehouse growth.

As costs go down, the wisdom of keeping detailed data goes up. I’d go so far as to say that every piece of data generated by a human being should be preserved and kept online, legal and privacy considerations permitting.* Most forms of capital-, labor-, and/or location-based competitive advantage being commoditized and/or globalized away. But information remains a unique corporate asset. Don’t discard it lightly.

*Unless there’s an explicit law mandating data destruction, legal considerations should permit. The idea “Let’s destroy something of irreplaceable value today, against the possibility we might be brought to judgment tomorrow” is both morally and pragmatically weird. Privacy, however, may be a different matter.

What that means in practice is that “disk is the new tape.” No-apologies performance can be had on data warehouse systems for $20,000/terabyte or less – perhaps even a lot less. Tolerable performance may cost 3-4X less than that. I think a lot of the growth in data warehouse volumes is of exactly this kind.

Ultimately, however, the greatest growth in data warehouse volumes will come from new kinds of data, especially data that is partly or wholly machine-generated. Moore’s Law applied to sensor chips tells us that data creation will grow just as fast as the data storage capacity. And thus we will be throwing away most machine-generated data forever. But what we keep will grow – well, it probably will grow at Moore’s/Kryder’s Law speeds.

That’s not to say new kinds of data are all high-volume/machine-generated. Back in 2005, I wrote two pieces for Computerworld advocating aggressive pursuit of new data sources, and the examples I mentioned were:

• Loyalty cards, especially in gaming
• Location-based analytics
• Extra customer feedback (e.g., opinion surveys)
• Price/offer testing
• Text mining in general
• Medical records

Today I’d add (among others):

• RFID
• The raw output from medical test devices
• Sensors up and down the energy supply chain

But some of those older, low-data-volume ideas still head my list of low-hanging analytic fruit.

One more complication – these buckets I’m outlining are less than precise. For example:

• Telecom CDRs (Call Detail Records) are machine-generated from a seed of human activity. They have long been stored, but now are being kept in much more detail. This is why telecommunications is one of the top markets for data warehouse technology.
• Stock trade data used to be based on human decisions. Now most of it is just machines buying and selling from each other. Either way, increasingly many investment institutions want to keep 100-terabyte-scale databases of complete historical trade detail. And that is why financial services is another huge market for data warehouse technology.
• Not long ago, web and network event logs. didn’t even exist, or were tiny where they did. Now they fill the largest known commercial databases, at firms such as Yahoo, eBay, and Facebook. Even so, more is thrown away than kept, especially on the network event side, which is a multiple of the size of the pure clickstream data.
• We don’t know exactly what all data intelligence agencies collect from telemetry, from monitoring commercial telecommunication traffic, and so on. But they’re surely throwing the vast majority away, even as the small part they keep is petabyte-scale.

But none of that interferes with my main points, which are:

• Databases will continue to grow very quickly.
• One big driver is the increasing detail in which data is kept online.
• An even bigger driver will be the unending ability of machines to generate ever greater streams of at-least-somewhat interesting data.

• Registration for tomorrow’s webinars
• Replay of the first webinar
• My slides from the first webinar

The main subjects of the webinar will be:

• Some review of material from the first webinar (all three presenters)
• Discussion of how MapReduce can help with three kinds of analytics:
  • Pattern matching (Jonathan will give detail)
  • Number-crunching (I’ll cover that, and it will be short)
  • Graph analytics (I haven’t written the slides yet, but my starting point will be some of the relationship analytics ideas we discussed in August)

Arguably, aspects of data transformation fit into each of those three categories, which may help explain why data transformation has been so prominent among the early applications of MapReduce.

As you can see from Aster’s title for the webinar (which they picked while I was on vacation), at least their portion will be focused on customer analytics, e.g. web analytics.

The top two points from Q&A probably were:

• Big Data and the cloud actually have relatively little to do with each other, a few exceptions notwithstanding, especially if the data is in a shared-nothing DBMS (as opposed to, say, a MapReduce-oriented file cluster). Two principal reasons are:
  • Redistributing data from node to node is a little slow, undermining some of the elasticity benefits of the cloud.
  • Getting data into the cloud in the first place is a lot slow.
• The NoSQL movement is a lot like the Ron Paul campaign — it consists of people who are dissatisfied with the status quo, whose dissatisfaction has a lot to do with insufficient liberty and/or excessive expenditure, and who otherwise don’t have a whole lot in common with each other.

Anyhow, here are my notes for the talk, edited in just a couple of places for readability or linkage.

Quick introduction

• Big Data vs. cloud
• How big is Big Data?
• At the low end of that range, there’s little you can’t do with conventional technology if you have:
  • An unlimited budget for hardware
  • An unlimited budget for software
  • An unlimited budget for people, especially Oracle DBAs

Big Data in OLTP

• Hard-core OLTP
  • Focus of DBMS technology for a long-time
  • Big budgets because each transaction has significant value
  • Tough to get users to change technologies
• Lighter-weight OLTP
  • Classic example = web companies
    • Big ones — retail-oriented ones (eBay, Amazon) partially excepted — rolled their own technology stacks
    • Reluctant to give money to anybody
      • Open source, etc.
  • Difficulty finding market
    • Product vs. feature
      • Clustering/HA/DR/whatever
      • Ditto cloud enablement
    • True products haven’t found much traction yet

Analytic Big Data use cases

• Kinds of data for analytics
  • More of same != big
  • More detail and/or new kinds
    • Complete data sets
    • Transactions
    • Call details
    • Tick/trade history
    • Web clickstreams
    • Network event logs
    • Other machine-generated data
    • CAM bottom line
      • Anything human-generated should and will be retained in its entirety
      • Quantities of machine-generated data retained should and will grow roughly in line w/ computing cost reductions (Moore’s Law, etc.)
• Analytic uses of Big Data
  • Analytics is mainly about three things
    • Problem detection
    • Customer relationship improvement
      • (Those overlap when the customer relationship is bad)
    • Financial statements on steroids
  • Main kinds of analytics
    • What BI vendors traditionally sell
      • General reporting and dashboards
      • Ad-hoc query (now driven from those reports and dashboards)
      • Planning (allegedly integrated with BI)
    • Research
      • Ad hoc relational query (worth mentioning twice because it drives so much of the market)
      • Data mining
      • Most web search and web mining
    • Operational/near-real-time
    • Archiving/compliance
  • What gets Big?
    • Mainly research and archiving
    • But when reporting or operational get Big, you have really interesting computing problems

Technology issues and trends

• Moore’s Law
  • CPUs — All about cores, hence parallelism is key
  • RAM
  • SSDs – hence replace disks
  • Sensors – hence generate lots more data
• Kryder’s Law
  • But rotational speeds up only 12.5X since Eisenhower Administration
  • Hence solid-state memory (or RAM) will soon take over
• In the mean time, I/O bottlenecks have had to be beaten
  • Hence sequential scans
  • Hence index-light architectures
  • Hence columnar
• DBMS “overhead”
  • Raw license and maintenance fees – software increasing fraction of total
  • OLTP vestiges – locking and all that
  • DBAs
    • People costs = huge fraction of total
    • Index-lightness addresses
    • So does appliance
  • Many people don’t really know how to write SQL
• Configuration
  • Appliance/tightly-balanced
    • Netezza
    • Teradata earlier
    • Greenplum/Sun
    • Oracle
    • IBM
    • Microsoft/Madison
  • Commodity/do what you want
    • Vertica
    • Greenplum now
    • Infobright, Aster and others
    • MapReduce-oriented file systems
  • Extreme rigidity is silly
    • Teradata, Oracle have both signaled moving to more modularity
    • Big driver of that = heterogeneous storage
      • Cheap disk
      • Expensive disk
      • Solid-state
      • RAM
    • CPU/storage ratio is even more of a driver

Theoretically defensible ways to segment the market

• Latency requirements
  • High availability and low latency go together
• Query types
  • Simultaneous users for same
• Database size
• Budget

Actual segments right now

• Utter ADW/EDW
• Data mart
  • Size
  • Naturally columnar vs. naturally row-based
• Operational/frontline
• Less dramatic/smaller EDW