1. Need to be parallelized
2. Should run on SMP rather than shared-nothing hardware

and Pervasive hasn’t done a great job of explaining where #2 applies.

That said, there’s at least one use case for which DataRush should clearly be considered today.  Suppose you have a messy ETL/data transformation task that requires custom code.  Then I see three main choices:

• Write the code within the confines of an off-the-shelf ETL tool.
• Write the code to run on an analytic DBMS platform, ideally an MPP/shared-nothing one.
• Use something like DataRush (and I’m not familiar with any good alternatives to DataRush).

In some cases, DataRush may be best possibility.

• DataRush grew out of Pervasive Software’s ETL business, as the underpinnings for a new data transformation tool they were building.
• DataRush is a Java framework for doing parallel programming automagically.
• Unlike most modern parallelization technologies, DataRush is focused on single SMP (Symmetric MultiProcessing) boxes rather than loosely-coupled grids.
• DataRush is based on dataflow programming.
• Pervasive says that DataRush is really fast.

More details may be found at the rather rich Pervasive DataRush website, or in the following excerpt from an email by Pervasive’s Steve Hochschild:

… we do believe that we were among the first in the commercial tools market to understand that a new approach was required if our customers were going to be successful in moving to multicore.  We did recognize that dataflow was a well proven architecture, having implemented it in our own products, and we made the decision to offer up that infrastructure so that our thousands of existing VAR, OEM, and ISV customers could do the same, move their applications to multicore platforms.

The specific insight we claim is not only that we recognized that dataflow is The Right Way To Go, but that we painstakingly engineered it’s implementation in Java.  Our developers have implemented deadlock detection, threading, and resource management within our library such that any commercial Java developer can become parallel-productive far more quickly than with any other approach.  In particular, we know it is easier and faster to code more quickly and more successfully using DataRush than using the concurrency primitives in Java, than learning a new language, or than moving to an alternative architecture such as a data warehouse.

So the important success factors we believe we have are:

• we started before anyone else, (other than PeakStream, R.I.P.),
• we have experts in concurrency who have implemented parallel functionality in Java
• we created it as a Java library that works as a Java developer expects with regard to their IDE, JVM, etc., and
• we have decades of experience in data management tools targeted to developers.

We further are committed to riding the commodity price declines of SMB servers and storage systems, as we are totally certain that SMB prices will fall further and more quickly than blades and clusters.

I get the impression Informatica is still Salesforce-only, e.g. from this price list.

Pervasive DataCloud is currently vendor-specific too. In Pervasive’s case, the fixed point is QuickBooks Online. DataCloud, the pure cloud offering. is newish, with an undisclosed hosting partner. The most common integration is with, you guessed it, Salesforce.com, but Microsoft’s CRM is in the mix as well. Pricing is $1-2K/year.

The most comprehensive integration-as-a-service story I’ve heard may be the one Cast Iron Systems is rolling out. Cast Iron is hosting with OpSource any integration you can get in the Cast Iron appliance. To emphasize this, pricing is identical to that of the rental option for the appliance ($1K/month in the simplest two-endpoint cases), and customers are encouraged to switch between appliance and cloud usage as they see fit. (That said, I think the whole thing is way too new for such a switch ever to have happened yet; the official rollout is scheduled for October 20.) Cast Iron supports a fairly broad range of applications, SaaS and on-premise alike. (Cast Iron is particularly proud of what sounds like a beyond-Barney hug from Oracle’s CRM On Demand business.) Cast Iron claims less than a handful of direct sales of this new cloud offering. However, Cast Iron also claims 23 partners, combined from among several areas:

• SaaS vendor OEMing for ongoing data integration in the usual way
• SaaS vendor OEMing for one-time data migration
• Implementation VAR using the service

You may have noticed that everything I’ve cited above is for operational apps being connected with each other, almost always including CRM. What I haven’t heard is integration vendors getting much involved with analytics-in-the-cloud offerings, whether from data mart outsourcers or vendors with cloud DBMS offerings. Not coincidentally, I don’t think many offerings in either category have large customer counts. (Also — Kognitio, which along with Vertica is one of the two data warehouse DBMS vendors most emphasizing cloud offerings, happens to have a data migration subsidiary of its own.)

*Or it’s some of each, at which point it’s called ETLT — I bet you can work out what that stands for.

But depending on your needs, at least two other approaches to data transformation parallelization could also be considered. Pervasive Software, which has a big data integration software business of its own, built a new ETL tool. The foundation was a middle-tier multi-core-friendly Java dataflow engine, which has been now split out as Pervasive Datarush. The product is in the early stages of being released, which may be a good excuse for the website confusingly suggesting both of:

• You can have Datarush for free.
• If Datarush doesn’t produce a 30X speedup for you, you can get your money back.

The third approach is my Subject Of The Week: MapReduce. When I posted a list of canonical MapReduce applications, my friends at Aster Data offered one pushback — I left out the area of data transformation. As CEO Mayank Bawa puts it:

Large-scale transformations can be parameterized as SQL/MR functions for data cleansing and standardization, unleashing the true potential for Extract-Load-Transform pipelines and making large-scale data model normalization feasible. Push down also enables rapid discovery and data pre-processing to create analytical data sets used for advanced analytics such as SAS and SPSS.

Some of our recent links about MapReduce

• The integration of MapReduce with SQL data warehousing
• Three major applications of MapReduce
• Sound bites about MapReduce
• Other links about MapReduce

• I was just briefed by Vertica, and got the impression that data mart outsourcers may be Vertica’s #3 vertical market, after financial services and telecom. Certainly it seems like they are Vertica’s #3 market if you bundle together data mart outsourcers and more conventional OEMs.
• When Netezza started out, a bunch of its early customers were credit data-based analytics outsourcers like Acxiom.
• After nagging DATAllegro for a production reference, I finally got a good one — TEOCO. TEOCO specializes in figuring out whether inter-carrier telcom bills are correct. While there’s certainly a transactional invoice-processing aspect to this, the business seems to hinge mainly around doing calculations to figure out correct charges.
• I was talking with Pervasive about Pervasive Datarush, a beta product that lets you do super-fast analytics on data even if you never load it into a DBMS in the first place. I challenged them for use cases. One user turns out to be an insurance claims rule-checking outsourcer.
• One of Infobright’s references is a French CRM analytics outsourcer, 1024 Degres.
• 1010data has built up a client base of 50-60, including a number of financial and retail blue-chippers, with a soup-to-nuts BI/analysis/columnar database stack.
• I haven’t heard much about Verix in a while, but their niche was combining internal sales figures with external point-of-sale/prescription data to assess retail (especially pharma) microtrends.

To a first approximation, here’s what I think is going on.

Privacy laws force some outsourcing. It’s often OK to use credit data to decide what you’ll market at whom, even when it’s not OK to actually see the credit data itself. What’s more, in some cases data can’t leave a country, so if you don’t have critical business mass in that particular country, it’s natural to use an outsourcer who does.

Privacy even aside, owners of proprietary data are natural analytics outsourcers. Either you ship your data to your customers to do with as they please — and impose on them the expense of managing it — or you manage it for them.

Analytic “secret sauce” software providers also are natural outsourcers. Most proprietary analytic rules are pretty simple-minded. Outsourcing preserves mystique and pricing power.

The usual benefits of SaaS apply. Fast set-up, no fixed costs, etc. are all goodness, just as they are in the transactional world.

With that as background, the big change in the analytics outsourcing market is the same as the one sweeping the rest of the analytics world — interactive access to detail data is finally becoming affordable. If you just run weekly or monthly reports, and there may be no reason to distinguish between analytic and transactional processing. But if you want to allow ad-hoc query, unlimited drilldown, or live dashboards, then you’re talking a serious data mart technology stack.

And I do mean “data mart”. Outsourcing an enterprise data warehouse, with all of your proprietary transactional data, doesn’t make much sense unless you’re a complete SaaS shop already outsourcing that data in the first place.

Via Data Integrator, Pervasive is a leader in the low-cost integration market, with revenue split about 50/25/25 between direct sales, ISVs, and SaaS. Pervasive fondly believes that its products cost half as much as Cast Iron’s, and wind up taking no more installation effort when you factor in Pervasive’s broader capabilities in areas such as workflow. However, there’s some doubt as to whether this is apples-to-apples. Cast Iron does include hardware, after all, and as Pervasive itself points out, Cast Iron will bundle some professional services into a sale if you ask nicely.

Two things are new. First, Pervasive just introduced Pervasive DataCloud, an offering in the recently-buzzworded IaaS (Integration As A Service) market. This is a delivery option for all of Pervasive’s runtime integration, hosted by them in the cloud (I forget to ask which hosting companies they use). That’s now a third deployment option, along with on-customer-premises and OEMed-by-a-SaaS-vendor. However, when connecting on-premises software to DataCloud, it’s likely that you’ll need a subset of Data Integrator to serve as a gateway, unless your apps already have the necessary APIs to connect out.

Second, Pervasive has an ongoing effort to “package” integrations. Its most organized such offering is a Salesforce/QuickBooks integration for under $1000/organization/year, called Pervasive DataSynch. DataSynch has actually been out for 9 months or so in an on-premises version, and now is available on DataCloud as well. Also, while it’s not (yet) as neatly packaged, Pervasive happily bids fixed-price contracts to, for example, connect SAP to Salesforce.com. Presumably, much more packaging of these kinds is coming soon.

Please subscribe to our feed!

But for most applications at any enterprise – and for all applications at most enterprises – super high-end DBMS aren’t required. There are relatively few applications that wouldn’t run perfectly well on PostgreSQL or EnterpriseDB today. Ingres and Progress OpenEdge aren’t far behind (they’re a little lacking in datatype support). Ditto Intersystems Cache’, although the nonrelational architecture will be off-putting to many. And to varying degrees, you can also do fine with MySQL, Pervasive PSQL, MaxDB, or a variety of other products – or for that matter with the cheap or free crippled versions of Oracle, SQL Server, DB2, and Informix.

What’s more, these mid-range database management systems can have significant advantages over their high-end brethren. The biggest is often price, for licenses and maintenance alike. Beyond that, they can be much easier to administer then their more complex counterparts. For example, Progress OpenEdge and Informix SE have long been reseller favorites, in large part because they can be installed at small businesses and locations that lack technical staff, and rarely if ever require DBA attention. Programming and hardware costs can sometimes be lower as well.

And what these mid-range DBMS don’t do today, they likely will do soon. In the 1990s, Microsoft SQL Server was the mid-range entry threatening to disrupt the market. But it’s grown up quite nicely. EnterpriseDB is equal or superior in every way I can think of to Oracle7, a few security certifications perhaps excepted. (They’d probably argue the release number in that claim should be 1 or 2 higher, but I’d have to compare their multimedia support to what I recall of Oracle 8.1.5 before I agreed.)

Will these mid-range database management systems truly “disrupt” the DBMS market, as many open source advocates hope? Or will they be largely co-opted into the oligopoly, as Microsoft SQL Server was? That’s a discussion for another time. For now, please just keep your mind open to DBMS alternatives – the high-end approach is not always the best.

EDIT: For a contrary view, please see my follow-up post making the opposite case.

Please subscribe to our feed!

Pervasive Summit PSQL v10 has just been rolled out, and I wrote a nice little white paper to commemorate the event, describing some of the main advances over v9, primarily for the benefit of current Pervasive PSQL developers. In one major advance, Pervasive made the SQL functionality much stronger. In particular, you now can have a regular SQL data dictionary, so that the database can be used for other purposes – BI, additional apps, whatever. Apparently, that wasn’t possible before, although it had been possible in yet earlier releases. Pervasive also added view-based security permissions, which is obviously a Very Good Thing.

There also are some big performance boosts. Most of these are via a new module (which Pervasive calls a “driver”) called “Xtreme I/O.” This comes from a company called QuickShift – previously known for Microsoft SQL*Server acceleration — which sold Pervasive the source code and, I presume around the same time, ceased operations.

Xtreme I/O’s biggest virtue is – there’s our theme-of-the-month again – compression. In the case of an OLTP DBMS like Pervasive PSQL, compression helps in two big ways. First, it reduces I/O. Second, it allows more data to be kept in cache, and thus accessed at RAM speed.

PSQL v10 also has two other boosts to in-memory processing. First, it allows 64-bit addressability, and hence access to more RAM. Second, it fixes some prior-version weirdness, and allows a higher fraction of overall RAM to be devoted to cache. Between those enhancements and the compression, the net effect is that you can now put a number of gigabytes of user data into RAM.

So should I categorize Pervasive PSQL as a memory-centric system? Well, if you’re looking at data access methods for memory-centric OLTP processing, linked-list isn’t all that bad. Even so, it would be an exaggeration to say that PSQL is truly optimized for in-memory processing. So no, PSQL is not memory-centric. But even so, it runs at RAM speed for significantly large databases than it did in v9.

One area of advance in PSQL v10 that I resisted covering in the white paper was Microsoft compatibility, be that Vista support, .Net support, or a closer resemblance to SQL*Server functionality. While those are very important, they also are quite boring. I might well change my mind if the SQL*Server resemblance became so close one could swap SQL*Server out for PSQL and have things basically keep running. But I don’t think PSQL is quite at that point yet.

Oh yes — if you do choose to look at Pervasive’s website or other marketing materials, please be aware of one weird choice they made in marketing jargon. They equate “transactional” to linked-list, and hence draw a distinction between “transactional” and “relational.” I’m pretty sure this is just a marketing artifact, and doesn’t actually indicate any severe deficiencies in their SQL transactional capabilities, unless perhaps you’re looking for some kind of distributed two-phase commit capability.

Simon claimed that for similar mappings via Pervasive you’d actually have to write code. I wonder if anybody from Pervasive would care to comment on that?

My other interesting takeaway was pricing — most of their customers rent rather than buying. Many sign long-term commitments to get discounts and still choose to rent. The main reason seems to be that’s where they have the flexibility to make room in their budget. An important secondary reason seems to be that many of Cast Iron’s customers are using Cast Iron primarily to connect to SaaS, so they’re oriented to periodic payments anyway.

• I recently talked with Pervasive Software about their data integration line. A large part of Pervasive’s new business is Salesforce.com integration, including at some big-name software vendors as customer/partner switch-hitters.
• I just rechecked my notes from my January talk with Cast Iron Systems. A large part of Cast Iron’s new business is also integration with Salesforce.com, Netsuite, and other SaaS vendors.
• Informatica keeps putting out press releases about Salesforce.com integration, most recently by offering replication in SaaS form itself.

But of course this makes sense. Without good data integration, SaaS applications would be pretty useless, at least at large and medium-sized enterprises.