• If you are or want to be a serious MapReduce user – and you’re past the “play around over the weekend” stage — you probably should have either:
  • A serious non-DBMS MapReduce distribution.
  • MapReduce integrated into your analytic DBMS.
  • Both.
• The obvious choice for non-DBMS MapReduce is Hadoop.
• The obvious choice for a Hadoop distribution is Cloudera Enterprise.
• Cloudera Enterprise has three main aspects, in an inseparable bundle:
  • Distributions for a double-digit number of open source projects. It’s nice having all that in one package – unless, of course, you like playing with Tinkertoys.
  • Proprietary Cloudera code.
  • Cloudera support.
• Cloudera says its proprietary code is and in the future is planned to be concentrated – at least in large part — on integrating open source technology with closed source products. This has the virtue of being targeted directly at that segment of the market which has proven it’s actually willing to pay money for software.
• Cloudera Enterprise areas of focus, now and in the presumed future, include:
  • Core Hadoop engine, which Cloudera says is quite predictably and appropriately evolving more slowly than the tools around it.
  • Development, management and administrative tools, including:
    • Pig and Hive. Cloudera says >70% of Facebook Hadoop jobs are initiated through Hive, and the same is true of Yahoo and Pig.
    • Connectivity to commercial tools.
    • The product formerly known as “Cloudera Desktop.”
  • Workflow, which in this context refers to letting you create a Hadoop application as a sequence of small steps, rather than forcing you to kluge it into being one unwieldy thing. At the moment, this is much less widely adopted than Pig and Hive, but Cloudera has high hopes for it, because of its obvious benefits in modularity and manageability.
  • Quasi-DBMS technology. Besides Hive and Pig, this includes HBase. Cloudera says there has been considerable demand for HBase, and it is pleased that project is now mature enough to ship. Cloudera stresses that it intends HBase not for OLTP, but as an adjunct to analytic processing. E.g., Cloudera suggests HBase would be a fine vehicle for replicating dimension tables across each node of a cluster.
  • Data connectivity, e.g. to MySQL or to sensor log files.
• Cloudera Enterprise pricing is well below DBMS prices – not by a full order of magnitude, if I’m right about everybody’s quantity discount policies, but even so by a lot. Details are NDA.

Cloudera sometimes sends confusing signals about its beliefs and strategies. For example, one can get different stories depending on whether one talks to:

• Somebody at Cloudera who comes primarily from the user and open source communities.
• Somebody at Cloudera who has actually worked at a software company before.

But I predict that Cloudera will now stick for a while with more or less the strategy outlined above.

Naturally, we also talked about Hadoop adoption. Highlights of that part – no doubt somewhat biased towards Cloudera’s own customer base — included:

• Notwithstanding eBay’s prior skepticism about MapReduce, it is quoted saying nice things in a Cloudera press release, and has apparently become quite a large Hadoop user, starting out with a search-quality use case.
• Typical Hadoop deployment sizes are 10 nodes or so when experimenting, 80-500+ in production.
• 10 terabytes/node – I’m pretty sure Cloudera meant of user data — is not inconceivable, so a cost-conscious 500-node user could have 5 petabytes of data managed by Hadoop.
• Cloudera has half a dozen customers at the 75+ node production level.
• Web and financial services are the two vertical markets moving most aggressively into Hadoop production. The government is also in significant Hadoop production, but the details of that are classified.
• Web uses for Hadoop include:
  • Clickstream – sessionization, etc. – that’s a super-mainstream use.
  • Search – analyzing search attempts in conjunction with structured data.
  • Machine learning (for ad serving, etc.).
• Financial services uses for Hadoop include:
  • Internal trading rule enforcement/fraud detection.
  • Complex ETL.
  • Portfolio risk assessment (typically overnight).

None of this is inconsistent with previous surveys of Hadoop use cases.

Various users talked at the Hadoop Summit this week. I wasn’t there, and won’t write about their stories for now. That said, Twitter’s slide deck from same has some interesting stuff, including:

• 7 TB/day ETLed from MySQL.
• Petabytes-being-stored accordingly coming soon.
• Open sourcing their ETL tool Crane.
• 3-4X LZO compression at little CPU cost.
• HBase is a more usable for them than HDFS, which isn’t mutable enough.
• Pig = 5% of code and coding effort vs. vanilla Hadoop at 30% or less performance hit.

The main new activity I know of has been in the open source SciDB project.  

• A company called Zetics has been started to commercialize SciDB. As of now, the entire staff seems to be CEO Marilyn Matz, techie Paul Brown, and part of Mike Stonebraker. Marilyn says Zetics has some venture capital, but even under NDA didn’t tell me who it was from. Zetics does not have its own web site.
• Marilyn tells me there are 20-25 contributors to SciDB, led by Paul Brown and Mike Stonebraker. Brown is full-time. Persistent Systems has been donating the efforts of a few of its employees. Some LSST folks have been doing SciDB work backed by grant money. Most or all of the rest seem to be purer volunteers. Some Russians have been particularly active.
• Release 0.5 of SciDB is expected in June. Release 1.0 is expected in September. This is a rewrite; prior demo code has been scrapped. Perhaps not coincidentally, it’s also a small slip from prior project plans.
• The array data model is an example of what’s being implemented first. (Duh — you can’t have a DBMS without a data model.) Support for uncertainty is an example of what’s been deferred until later.
• As has been clear since XLDB3 last August, one major target market for SciDB is genomic research.
• It’s obvious that the oil and gas industry, with all its geospatial data, should be interested in SciDB. But there’s not much activity in that regard; outreach is evidently needed. If you can think of somebody in that sector (or anywhere else) who should be alerted to SciDB, please ping them.
• Interest from web analytics users in SciDB seems to have receded a bit from the days when eBay almost funded the project.

In other scientific data management news,

• Microsoft put out a book called The Fourth Paradigm on scientific database management. The whole thing can be downloaded, very officially, as a giant PDF. I think it’s worth skimming. I don’t think it’s worth actually reading. (I did read it.)
• XLDB4 will be at Stanford October 5-7. Unlike prior XLDBs, it will have an open (i.e., no invitation required) part.

Finally, you surely are aware of the whole “Climategate” mess, in which major climate researchers’ email was hacked and many unkind conclusions were drawn. Well, one of the most technical parts of the disclosure was in a long series of Read Me files, in which an unfortunate programmer lamented about the difficulty of reconstructing published results from files at hand. These turned out to illustrate a classic problem that SciDB or alternatives are meant to solve:

• Raw data was impossible to use without various adjustments to regularize it (the word “regridding” comes up a lot, for example). Massaging was needed before analytics could be done on it.
• The raw data was thrown out or lost, and could not be reconstructed (why they couldn’t have asked the suppliers of the data to give it to them again was unclear in this case, since it wasn’t original experimental data).
• It was thus impossible to massage the data in any new or improved way.

• What Teradata calls the Teradata Agile Analytics Cloud, which is a combination of previously existing technology plus one new portlet called the Teradata Elastic Mart(s) Builder. (Teradata’s Elastic Mart(s) Builder Viewpoint portlet is available for download from Teradata’s Developer Exchange.)
• Teradata Data Mover 2.0, coming “Soon”, which will ease copying (ETL without any significant “T”) from one Teradata system to another.
• Teradata Express DBMS crippleware (1 terabyte only, no production use), now available on Amazon EC2 and VMware. (I don’t see where this has much connection to the rest of Teradata’s cloud strategy, except insofar as it serves to fill out a slide.)
• Unannounced (and so far as I can tell largely undesigned) future products.

Teradata openly admits that its direction is heavily influenced by Oliver Ratzesberger at eBay. Like Teradata, Oliver and eBay favor virtual data marts over physical ones. That is, Oliver and eBay believe that the ideal scenario is that every piece of data is only stored once, in an integrated Teradata warehouse. But eBay believes and Teradata increasingly agrees that users need a great deal of control over their use of this data, including the ability to import additional data into private sandboxes, and join it to the warehouse data already there.

The Teradata Elastic Mart(s) Builder Viewpoint portlet automates the inclusion of outside data. If you’re already an authorized Teradata data warehouse user, you can fill in a very short form (three or so fields) and add authorization to import outside data, e.g. from a .CSV file. No fuss, little bother. Trivial as that sounds, when you combine it with Teradata’s pre-existing robust workload management tools, it creates a pretty good virtual data mart story.

Spinning out and maintaining consistency with physical data marts is a different matter. Teradata doesn’t seem too sure it believes in those. And while Teradata is obviously planning to increase its capability in that regard anyway, I didn’t get a lot of detail beyond the reference to Data Mover 2.0.

Related links

• My Greenplum-inspired post on the future of data marts, outlining issues in “private cloud” data warehousing.
• eBay’s “Analytics as a Service” pitch (about 1 ½ years old)
• A post by Teradata’s Dan Graham explaining the Teradata Agile Analytics Cloud and Elastic Mart(s) Builder Viewpoint portlet
• Home page and complete screen shot for the Teradata Elastic Mart(s) Builder Viewpoint portlet

XLDB more or less comprises:

• A series of what have now been three workshops: XLDB1 in 2007, XLDB2 in 2008, and XLDB3 in 2009 (the closest thing to a master link is probably the XLDB3 site’s related link page). Participants have included, among others:
  • A lot of big-name database-oriented computer science researchers — Mike Stonebraker, Dave DeWitt, Martin Kersten, and numerous others
  • Academics responsible for scientific database management, especially but not only in the astronomy area
  • Some vendors (although vendor participation was cut back after XLDB1) — at XLDB3, which is the one I went to, the three vendor folks who actually talked were Stephen Brobst of Teradata, Luke Lonergan of Greenplum (who worked in scientific high performance computing earlier in his career), and Jeff Hammerbacher of Cloudera.
  • eBay and to some extent other large web companies
  • A European Union funding bureaucrat
  • Me
• An attempt to kick start a broader movement, perhaps comprising (it’s not totally clear yet):
  • Computer science researchers interested in database issues
  • Database technology vendors
  • Scientific researchers (academic) who have very large or otherwise difficult database management problems
  • Scientific researchers (commercial) who have very large or otherwise difficult database management problems
  • Other commercial users who have very large database management problems

The first result or spin-out from the XLDB effort seems to have been the SciDB project. This is an effort to build an open source DBMS called SciDB that will address some of the needs the XLDB effort is uncovering. (More on that in other posts.) Somewhat confusingly, all the use cases the XLDB group is collecting are currently being posted on SciDB’s website, apparently because it’s glitzier and healthier than, say, the excessively sparse XLDB wiki. Some SciDB development has happened, but no large sugar daddy has yet been found. (It’s a fairly open secret that eBay looked seriously and favorably at funding SciDB before the economic downturn.) hit.

Numerous big-name computer scientists are associated with SciDB, indeed more closely (it would seem) than with XLDB. That said, I’m guessing Dave DeWitt’s involvement in the open-source SciDB isn’t what it would be if he hadn’t gone to Microsoft. DeWitt actually skipped XLDB3, although he was in town for VLDB. (XLDB3 was back-to-back with VLDB 2009 in Lyon, France in late August.) Stonebraker just didn’t make the flight for either conference, due to the double-knee “upgrade” he had back in March.

There’s a lot more to be said about the cross-discipline or science-specific requirements that researchers place on data management, but I’ll leave that for later and just get this posted as a start — assuming, of course, that blog outages permit.

Related links

• Paper laying out the SciDB project
• One version of a SciDB overview page, with links to academic papers

• Data marts aren’t just for performance (or price/performance). They also exist to give individual analysts or small teams control of their analytic destiny.
• Thus, it would be really cool if business users could have their own analytic “sandboxes” — virtual or physical analytic databases that they can manipulate without breaking anything else.
• In any case, business users want to analyze data when they want to analyze it. It is often unwise to ask business users to postpone analysis until after an enterprise data model can be extended to fully incorporate the new data they want to look at.
• Whether or not you agree with that, it’s an empirical fact that enterprises have many legacy data marts (or even, especially due to M&A, multiple legacy data warehouses). Similarly, it’s an empirical fact that many business users have the clout to order up new data marts as well.
• Consolidating data marts onto one common technological platform has important benefits.

In essence, Greenplum is pitching the story:

• Thesis: Enterprise Data Warehouses (EDWs)
• Antithesis: Data Warehouse Appliances
• Synthesis: Greenplum’s Enterprise Data Cloud vision

When put that starkly, it’s overstated, not least because

Specialized Analytic DBMS != Data Warehouse Appliance

But basically it makes sense, for two main reasons:

• Analysis is performed on all sorts of novel data, from sources far beyond an enterprise’s core transactions. This data neither has to fit nor particularly benefits from being tightly fitted into the core enterprise data model. Requiring it to do so is just an unnecessary and painful bureaucratic delay.
• On the other hand, consolidation can be a good idea even when systems don’t particularly interoperate. Data marts, which commonly do in part interoperate with central data stores, have all the more reason to be consolidated onto a central technology platform/stack.

Of course, the EDC vision isn’t quite as new or differentiated as Greenplum ideally would wish one to believe.

• To a first approximation, EDC sounds a lot like what eBay has already built on Teradata equipment.
• Greenplum’s EDC vision also sounds a lot like what Stuart Frost was talking about at DATAllegro, what Dell was planning to build on DATAllegro equipment, and what Stuart continues to talk about now that he’s been acquired into Microsoft.
• Something like EDC can also be presumed to be implicit in the strategies of the other one-size-fits-all vendors — i.e., Oracle and IBM.
• Greenplum has only implemented a little more of the EDC vision so far than have other firms, unless you give it credit for being cheap/fast/MPP/running on commodity hardware, but deny that credit to Teradata (specialized hardware, and not cheap in its most popular configurations), Oracle (ditto for Exadata), IBM (also not cheap), or Microsoft/DATAllegro (not released yet).
• Specifically: In Greenplum Release 3.3, which is being announced today, Greenplum is introducing the (enhanced?) ability for data marts to be spun out as a background operation, while the database otherwise remains functional. As of 3.3, spinning out a data mart is a command-line operation. But in Release 3.4, Greenplum plans to offer a web-based interface for same, at which point the “self-service data mart creation” discussion will become operative. Otherwise, EDC is a roadmap/vision/statement-of-direction much more than it is a fully-baked technical project.

One particular source of potential confusion is Greenplum’s emphasis on the buzzphrase self-service (data mart). This seems to be a conflation of two related concepts:

• End users should be able to create new data marts themselves. Strictly speaking, I view this ability as useless at most enterprises, and important at very few, because of logistical issues. (Who gives the permissions? Who decides which hardware is used?) That said, useless “end user” tools often wind up being important productivity aids for IT professionals, and this kind of “self-service” would surely be another example.  Edit: Hmm. Doug Henschen inspired me to think that over again, and I’m beginning to soften. Suppose users could order up the data mart they want, perhaps test it at a very low processing priority (if they choose), and then send the completed request to IT for approval and provisioning. That would have some value.
• End users should be able to manage data marts themselves, once created. That’s a great idea, full of agility and don’t-make-IT-a-roadblock goodness. Data miners and similar analytic professionals commonly have the technical ability to manage a simple database, and should be allowed to do so if it’s ensured that they don’t break anything for anybody else.

One thing that’s needed for this technology to come to full fruition is sophisticated data movement and synchronization. Ideally, some tables in a data mart could be virtual — views against a central database. But others would be physically recopied from the center, with all the ETL/ELT/ETLT/replication issues that entails. Meanwhile, it’s not obvious that the ideal architecture is a simpleminded hub-spoke — perhaps one should be able to spin data marts out of other marts, perhaps at least somewhat reducing the proliferation of tables and the recopying of data. And it should be easy for administrators to change deployment strategies, e.g. by starting a table out as a view and changing over to making it a physical copy as usage profiles change.

Oliver Ratzesberger of eBay also argues that workload management — not a current Greenplum strength — can be crucial. For example, if the CEO wants the CFO to get her an answer TODAY, the fastest approach may be to create an entirely virtual data mart, with very favorable SLAs (Service Level Agreements). More generally, if you’re setting up dozens of marts that contain views of the central database, sophisticated SLA management can be essential. There’s a big virtualization opportunity here — but virtualization requires a lot of system management infrastructure.

Related links

• My recent post on reinventing business intelligence
• Greenplum adviser Joe Hellerstein’s pitch for agile data warehousing
• Charlie Bachman’s “private database” idea, which never went anywhere (pp. 138-139)
• Greenplum’s EDC and Release 3.3 press releases
• An interview with some of Greenplum co-founder Scott Yara’s own words

Metrics on eBay’s main Teradata data warehouse include:

• >2 petabytes of user data
• 10s of 1000s of users
• Millions of queries per day
• 72 nodes
• >140 GB/sec of I/O, or 2 GB/node/sec, or maybe that’s a peak when the workload is scan-heavy
• 100s of production databases being fed in

Metrics on eBay’s Greenplum data warehouse (or, if you like, data mart) include:

• 6 1/2 petabytes of user data
• 17 trillion records
• 150 billion new records/day, which seems to suggest an ingest rate well over 50 terabytes/day
• 96 nodes
• 200 MB/node/sec of I/O (that’s the order of magnitude difference that triggered my post on disk drives)
• 4.5 petabytes of storage
• 70% compression
• A small number of concurrent users

eBay’s Teradata installation is a full enterprise data warehouse. Besides size and scope, it is most notable for its implementation of Oliver’s misleadingly named analytics-as-a-service vision. In essence, eBay spins out dozens of virtual data marts, which:

• Combine views and aggregations on the central data warehouse with (optionally) additional “private” data the data mart user loads in.
• Are usually <5 terabytes in size, and indeed often <500 gigabytes.
• Can be created “instantaneously” by setting permissions, resource quotas, and the like.

The whole scheme relies heavily on Teradata’s workload management software to deliver with assurance on many SLAs (Service-Level Agreements) at once. Resource partitions are a key concept in all this.

So far as I can tell, eBay uses Greenplum to manage one kind of data — web and network event logs. These seem to be managed primarily at two levels of detail — Oliver said that the 17 trillion event detail records reduce to 1 trillion real event records. When I asked where the 17:1 ratio comes from, Oliver explained that a single web page click — which is what is memorialized in an event record — resulted in 50-150 details. That leaves a missing factor of 3-8X, but perhaps other less complex kinds of events are also mixed in.

The Greenplum metrics I quoted above represent over 100 days of data. Ultimately, eBay expects to keep 90-180 days of ultimate detail, and >1 years of event data. The 6 1/2 petabyte figure comes from dividing 2 terabytes of compressed data by (100%-70%). Since that all fits on a 4 1/2 petabyte system, I presume there’s only one level of mirroring (duh), not much temp space, and even less in the way of indexes.

Two uses of eBay’s Greenplum database are disclosed — whittling down from detailed to click-level event data, and sessionization. The latter seems to be done in batch runs and take 30 minutes per day. A couple of other uses are undisclosed. I assume eBay is doing something that requires UDFs (User-Defined Functions), because Oliver remarked that he likes the language choices offered by Greenplum’s Postgres-based UDF capability. But basically eBay’s Greenplum database is used for and evidently does very nicely at:

• Data ingest — it’s the first place log data goes
• Feeding the Teradata database
• A small number of big queries

eBay’s Teradata database handles the rest.

Related links:

• Wal-Mart, Bank of America, another financial services company, and Dell also have very large Teradata databases.
• Yahoo’s web/network events database, running on proprietary software, sounded about 1/6th the size of eBay’s Greenplum system when it was described about a year ago.
• Facebook has 2 1/2 terabytes managed by Hadoop — without a DBMS!
  
• Fox Interactive Media/MySpace has multi-hundred terabyte databases running on each of Greenplum and Aster Data nCluster.
• TEOCO has 100s of terabytes running on DATAllegro.
• To a probably lesser extent, the same is now also true of Dell.
• Vertica has a couple of unnamed customers with databases in the 200 terabyte range.
• In response to this post, Greenplum CTO Luke Lonergan quickly blogged about the eBay project. Other related posts on the same blog may follow.

• There’s currently a huge — one order of magnitude — performance difference between cheap and expensive disks for data warehousing workloads.
• New disk generations coming soon will have best-of-both-worlds aspects, combining high-end performance with lower-end cost and power consumption.
• Solid-state drives will likely add one or two orders of magnitude to performance a few years down the road. Echoing the most famous logjam in VC history — namely the 60+ hard disk companies that got venture funding in the 1980s — 20+ companies are vying to cash in.

In other news, Carson likes 10 Gigabit Ethernet, dislikes Infiniband, and is “ecstatic” about Intel’s Nehalem, which will be the basis for Teradata’s next generation of servers.

Here’s the longer version.

Oliver Ratzesberger of eBay made the interesting comment to me that 15K RPM disk drives could have 10X or more the performance of 7200 RPM ones, a difference that clearly is not explained just by rotational speed. He said this was due to the large number of retries required by the cheaper drives, which eBay had tested as being in the 5-8X range on its particular equipment, for an overall 10X+ difference in effective scan rates. When I continued to probe, Oliver suggested that the guy I really should talk with is Carson Schmidt of Teradata, advice I took eagerly based on past experience.

Yesterday, Carson — who was unsurprised at Oliver’s figures* — patiently explained to me his views of the current differences between cheap and expensive disk drives. (Carson uses the terms “near-line” and “enterprise-class”.) Besides price, cheap drives optimize for power consumption, while expensive drives optimize for performance and reliability. Currently, for Teradata, “cheap” equates to SATA, “expensive” equates to Fibre Channel, and SAS 1.0 isn’t used. But SAS 2.0, coming soon, will supersede both of those interfaces, as discussed below.

*Carson did note that the performance differential varied significantly by the kind of workload. The more mixed and oriented to random reads the workload is, the bigger the difference. If you’re just doing sequential scans, it’s smaller. Oliver’s order-of-magnitude figure seemed to be based on scan-heavy tasks.

As I understand Carson’s view, mechanical features sported only by expensive drives include:

• Smaller media, more platters, and more disk heads
• Faster rotational speeds
• Enclosures that do a better job of damping vibration from disk rotation or fans.

Electronic features of expensive storage includes:

• More CPU (at least 2X)
• More RAM (also at least 2X), which is useful for caching.
• Dual ports for networking. Teradata doesn’t just use dual storage ports for reliability; it load balances across them and sometimes gets significantly enhanced performance.

Finally, there is firmware, in which expensive disk drives seem to have two major kinds of advantages:

• Command scheduling/queuing, which Carson believes provides a benefit at least comparable to the 2X derived from different rotational speeds.
• Better data integrity checking, in line with the T10 DIF standard. Not only does this seem to give much higher reliability, but it can be done closer to the platter, yielding a performance advantage.

Apparently, this isn’t even possible for SATA and SAS 1.0 disk drives, but is common for drives that use the Fibre Channel interface, and will also be possible in the forthcoming SAS 2.0 standard. (As you may have guessed, I’m a little fuzzy on the details of this firmware stuff.)

In Carson’s view, the disk drive industry has consolidated to the point that there are two credible vendors of expensive/enterprise-class disk drives: Seagate and Hitachi. What Teradata actually uses in its own systems right now is:

• In Teradata’s high-end 5550 line – Seagate Fiber Channel 3-1/2″ drives
• In Teradata’s mid-range 2550 line – SAS drives from Seagate and perhaps also Hitachi. I get the impression these have some of the electromechanical features of expensive drives, but not the firmware.
• In Teradata’s low-end 1550 line — Hitachi 1-TB cheap drives.

All this is of course subject to change. In the short term that mainly means the possible use of alternate suppliers. As the Teradata product line is repeatedly refreshed, however, greater changes will occur. Some of the biggest are:

• A new SAS 2.0 standard will allow enterprise-class firmware for cheaper disks.
• The form factor for high-end disk drives will shrink from 3 1/2″ drives to 2 1/2″ drives of 1/2 the volume.
• The rotation speed sweet spot may actually decrease, to 10K RPM, with offsetting improvements to seek and latency so as not to cut performance. Power consumption benefits will ensue.
• There probably will be multi-TB SAS drives — “fat SAS.” SATA may be enhanced to compete with those. And by the way, SAS and SATA are electrically compatible, and hence could be combined in the same system.

I got the impression that at least the first three of these developments are expected soon, perhaps within a year.

And in a few years all of this will be pretty moot, because solid-state drives (SSDs) will be taking over. Carson thinks SSDs will have a 100X performance benefit versus disk drives, a figure that took me aback. However, he’s not yet sure about how fast SSDs will mature. Also complicating things is a possible transition some years down the road from SLC (Single-Level Cell) to MLC (Multi-Level Cell) SSDs. MLC SSDs which store multiple bits of information at once, are surely denser than SLC SSDs. I don’t know whether they’re more power efficient as well.

The main weirdnesses Carson sees in SSDs are those I’ve highlighted in the following quote from Wikipedia:

One limitation of flash memory is that although it can be read or programmed a byte or a word at a time in a random access fashion, it must be erased a “block” at a time. …

Another limitation is that flash memory has a finite number of erase-write cycles. … This effect is partially offset in some chip firmware or file system drivers by counting the writes and dynamically remapping blocks in order to spread write operations between sectors; this technique is called wear leveling. Another approach is to perform write verification and remapping to spare sectors in case of write failure, a technique called bad block management (BBM).

And finally, I unearthed a couple of non-storage tidbits, since I was talking with Carson anyway:

• Carson has become a 10 GigE “bigot”, and Teradata will soon certify 10 Gigabit Ethernet cards for connectivity to external systems. Carson’s interest in Infiniband, never high, went entirely away after Cisco decommitted to it. Obviously, this stands in contrast to the endorsements of Infiniband for data warehousing by Oracle and Microsoft.
• Intel’s Nehalem will be the basis for Teradata’s next server product. Carson is “ecstatic” with Intel at the moment, which is different from his stance at other times.

Oliver believes that, on the whole, MapReduce is 6-8X slower than native functionality in an MPP DBMS, and hence should only be used sporadically. This view is based on part on simulations eBay ran of the Terasort benchmark. On 72 Teradata nodes or 96 lower-powered nodes running another (currently unnamed, as per yet another of my PR fire drills) MPP DBMS, a simulation of Terasort executed in 78 and 120 secs respectively, which is very comparable to the times Google and Yahoo got on 1000 nodes or more.

And by the way, if you use many fewer nodes, you also consume much less floor space or electric power.

The point of such distinctions is sometimes lost on me.

In related news, of Greenplum’s two customers who back in August were supposedly heading into production soon with petabyte-plus databases, one hasn’t yet made it to that size. (”As we speak” turned out to be a longer conversation than I might have anticipated ….) The other (of course unnamed) customer has, Greenplum assures me, made it that high.  But upon checking with that (unnamed, in case I forgot to mention the point) customer, I don’t detect a whole lot of enthusiasm about Greenplum.

• Large enterprise data warehouse projects are often being deferred or cut back. (My sense is that a little of this happened in 2008, but more is happening with new budget cycles in 2009.)
• Smaller projects with credible, quick ROIs are doing fine. In some cases, these are the scaled-back or nose-under-the-tent parts of bigger enterprise-wide efforts.
• Perhaps not coincidentally, the technical trend of having a variety of data marts inside an EDW – sometimes called “sandboxes” — is going strong. I’ve commented on that before in connection with Microsoft.  It’s also at the heart of eBay’s Teradata-based “analytics-as-a-service” strategy.
• Uses of data warehousing for security or compliance still seem strong. I guess “Keep us safe (and out of jail)” is still a strong motivator for buying.
• Oracle’s is the #1 installed base in which smaller vendors go fishing. Teradata’s is probably #2. Microsoft SQL Server and MySQL are also suffering data warehouse competitive replacements.
• I haven’t heard many more examples of enterprises doing more analysis because the bad economy has invalidated their prior models and assumptions. More’s the pity.
• Projects to provide data to one’s customers are going gangbusters. There are many flavors of this, from pure third-party analytics vendors to (for example) credit card companies who sell transactional data back to their merchants to governments that become more “transparent” by exposing information to their citizens.

Obviously, if the economy is bad enough, everything will be hurt. E.g., a lot of those data sellers have something to do with advertising, and the underlying business sector is in the tank. Ditto consumer or mass B-to-B marketing as well. But for now, the worst hits are being suffered by projects with large price tags, long lead times, and unclear benefits.