• Aster Data Analytic Foundation, a set of analytic packages prebuilt in Aster’s SQL-MapReduce
• Aster Data Developer Express, an Eclipse-based IDE (Integrated Development Environment) for developing and testing applications built on Aster nCluster, Aster SQL-MapReduce, and Aster Data Analytic Foundation

And in other Aster news:

• Along with the development GUI in Aster nCluster 4.5, there is also a new administrative GUI.
• Aster has certified that nCluster works with Fusion I/O boards, because at least one retail industry prospect cares. However, that in no way means that arm’s-length Fusion I/O certification is Aster’s ultimate solid-state memory strategy.
• I had the wrong impression about how far Aster/SAS integration has gotten. So far, it’s just at the connector level.

Aster Data Developer Express evidently does some cool stuff, like providing some sort of parallelism testing right on your desktop. It also generates lots of stub code, saving humans from the tedium of doing that. Useful, obviously.

But mainly, I want to write about the analytic packages. I’m not convinced that they’re a big deal in themselves yet, or that a whole lot of person-months have gone into their combined development. Still, I think they provide a great indication of one direction in which analytic functionality is going. And by the way, Aster promises to release a lot more of that kind of thing over the next 12 months.

Aster’s flagship analytic package is nPath, which is like a regular expression matcher, but for (time) series of data rather than for character strings. The main use for nPath is in pulling specific kinds of event sequences out of web or network event logs. However, one could imagine uses in other sectors that focus on temporal or sequential data (e.g., trading, intelligence, other sensor analysis), should existing SQL- and/or CEP-based technologies not prove sufficiently flexible. Aster 4.5 adds some new aggregation capabilities around nPath.

Other not-wholly-new packages in the Aster Data Analytic Foundation announcement are for sessionization (of clickstream data and the like) and tokenization (of text/character string data). While sessionization can be done in SQL, Aster thinks its MapReduce-based version is faster, since it doesn’t require self-joins. Makes sense. Aster’s tokenization sounds lame, however – text analytics in MapReduce tends to reinvent simplistic wheels for no clear reason, and Aster doesn’t seem to be an exception. (Aster would argue, however, that anything it does in SQL-MapReduce is more flexible than pure SQL or pure MapReduce alternatives.)

Another example of better-living-without-self-joins is Aster’s new market basket package. This lets you look at a set of point-of-sale data, pick a small integer N, and pull out all the sets of N things that were bought by the same person at the same time. I haven’t probed the claim in detail, but Aster implies there’s less combinatorial explosion in its approach than it is in the self-join alternative.

Note: Gartner highlighted self joins as a performance challenge in its recent Data Warehouse Magic Quadrant.

Aster is also releasing a few statistical and general analytic functions — specifically (and I quote a slide):

• exponential moving average
• weighted moving average
• simple moving average
• volume-weighted average price
• correlation
• linear regression
• logistic regression
• approximate_percentile
• approximate_count_distinct

The point of the last two items on the list is that if you set a non-zero tolerance for error, you can you can count things or order them into bins very efficiently – especially in terms of RAM — while being guaranteed not to exceed your error tolerance.

Note: One obvious inference from this list — which Aster gladly confirms — is that Aster has high hopes of selling to the financial services industry.

Finally, Aster is releasing its first pure graph-analytic function, for finding the shortest path between a given pair of nodes.

While I had the Aster folks on the phone anyway, I also took the opportunity to ask about the Aster nCluster 4.0 capability to create fairly persistent non-relational in-memory data structures. Specifically, I asked whether different users could access the same in-memory structure, and was told that this is a little klugey but not too horrendous. That suggests Aster’s capability may be a strict superset of UDF-based (User-Defined Function) approaches to meeting the same need, at least from a functionality standpoint. However, ease of creating those in-memory structures may still be better in the more SQL/UDF-centric approach favored by Teradata.

• The most important issue in database and analytic technology, in my opinion, isn’t technological at all – rather, it’s the legal and political steps needed to preserve liberty in the face of advancing, intrusive technology.
• Another important issue for society – and this one does involve a lot of technology – is scientific number crunching. In particular, database technology for scientific computing needs to be developed much further. I’ll have more to say on all this soon.
• More generally, technology needs to keep advancing for parallel analytics. Fortunately, it is. Watch this space over the next few weeks.
• Oracle has said, in effect, that its most important technological challenge of the decade is getting solid-state memory right. I agree.
• Data volumes will keep going up, up, up. Technology needs to keep evolving accordingly. Much of what I write is on that subject.
• Data needs to be processed and analyzed at very different latencies. And there’s much further to go in integrating disparate latencies.
• Analytic database management in the cloud hasn’t been solved yet, especially for Big Data. Among the reasons are the difficulty of moving data into the cloud (unless it originated there), the slowness of moving it from node to node in shared-nothing architectures (which reduces the elasticity benefit), and above all the long and unpredictable latencies of interprocessor communication while queries are running (a key subject of discussion at the Boston Big Data Summit).
• Better business intelligence user interfaces are increasingly available. I’m thinking particularly of approaches with buzzwords like visualization/interactive exploration or faceted. But they aren’t well-integrated into the overall analytic stack, as big BI vendors are trailing the smaller ones in this regards. (Part of the problem relates to my previous point.)
• Application development over text search isn’t in the same league as application development over relational DBMS. The choices are mainly XML (e.g., MarkLogic), SQL for text integrated into RDBMS (limited by the weakness of those integrations), and something like Attivio’s Java SDK. There’s a major conceptual barrier in building those apps, namely the unpredictability of query results. Still, it should be possible to do better.
• Similarly, text analytics and conventional analytics exist well side by side. They can even be in the same database and/or dashboard, although in practice that is limited by the strong SaaS focus of text mining vendors and users. But analytic integration of them is really hard. Linguistic imprecision is, in my opinion, only the #2 reason for this difficulty. The #1 reason is that trends detected by text analytics are much less precise than trends on tabular data – e.g., a 50% increase in a certain kind of complaint may be no more significant than a 5% change in a revenue variable.
• I’m increasingly persuaded that graph analytics can be handled without a graph-centric data model. But right now, it isn’t being handled well at all. Lots more needs to be done – although when it is, it will just exacerbate the privacy/liberty dangers that so concern me.

Other posts based on my January, 2010 New England Database Summit keynote address

• Data-based snooping — a huge threat to liberty that we’re all helping make worse
• Flash, other solid-state memory, and disk
• Interesting trends in database and analytic technology

• Registration for tomorrow’s webinars
• Replay of the first webinar
• My slides from the first webinar

The main subjects of the webinar will be:

• Some review of material from the first webinar (all three presenters)
• Discussion of how MapReduce can help with three kinds of analytics:
  • Pattern matching (Jonathan will give detail)
  • Number-crunching (I’ll cover that, and it will be short)
  • Graph analytics (I haven’t written the slides yet, but my starting point will be some of the relationship analytics ideas we discussed in August)

Arguably, aspects of data transformation fit into each of those three categories, which may help explain why data transformation has been so prominent among the early applications of MapReduce.

As you can see from Aster’s title for the webinar (which they picked while I was on vacation), at least their portion will be focused on customer analytics, e.g. web analytics.

• Finding bad guys (national intelligence)
• Finding bad guys (anti-fraud)
• Data mining the social graph (e.g., for advertising optimization on social networks, or to identify influencers)

There are plenty more graph-oriented applications, of course, such as the identification of biochemical pathways. But I want to focus for now on ones like those on my list. My key points are:

• There are Big Data problems that lend themselves to graphical data models.
• So far as I can tell, the database management community isn’t doing enough to address them. (If I’m wrong about that, please tell me. I plan to arrive in Lyon for VLDB/XLDB Wednesday of next week, and of course I can always be reached by email.)

Here’s what I mean.

Applications that analyze relationship graphs are commonly grouped under the name social network analysis. As I frequently point out, category names and definitions tend to be imperfect, and that one is no exception. In particular — and the Wikipedia article on social networks and social network analysis is an excellent example of this – the term tends to be construed to cover the linkages between people or organizations, but not between, say, physical addresses, email addresses, and all the other stuff those intelligence applications actually track. I tried to introduce the term relationship analytics a while back, but it unfortunately didn’t stick.

I only ever got familiar with one company that tried to do a true graph-oriented database management system, suitable for social network analysis/relationship analytics. It was called Cogito, and had some interesting ideas about graphical data structures. Unfortunately, Cogito didn’t stick either.

As per the “Metrics” section of the Wikipedia article linked above, there are a number of well-established metrics about the relationships of pairs or groups of node to each other. The usual way to calculate these metrics is to load the graph into memory and get to work. (Indeed, such uses seem to be driving a lot of the national intelligence adoption of Hadoop.) And while I’m perfectly willing to believe that relational database management systems can do a fine job of managing generic RDF, it’s less obvious that they’re well-suited to support standard graph-analysis metric computations.

The reason, in a nutshell, is that the relational approaches usually boil down to maintaining a table with a row for every node-edge-node triple, and then doing a lot of fast self-joins to identify paths. That can work if connectivity is low and paths are sparse. But for higher degrees of connectivity, such strategies can lead – BOOM! — to serious combinatorial explosion. And that’s not good, because a lot of this analysis focuses on finding exactly the parts of the graph where the connections run thickest.

Cogito’s idea was to say “What if, for every node, you could retrieve in only a few blocks all the paths leading from it, at least up to pathlength N?” Unfortunately, Cogito’s approach to creating this effect had too little to do with optimizer development or selectively redundant data storage, and too much to do with wishful thinking; not coincidentally, Cogito is no longer around. (I haven’t kept in touch with Cogito’s successor 7 Degrees, and the reason hasn’t been lack of effort or interest on my part.)

But suppose the idea had worked. Then – unlike today – it might be realistic to do on-the-fly analytics on Very Large Graphs, just as we do operational business intelligence of a more relational or MOLAP nature. That would be cool.

How cool would it be? Well, that’s a bit hard to say. Look again at the list of applications I put up top. Those are NOT ones people generally talk a lot about. Spooks and fraud-fighters are two very secretive kinds of folks. And, for a variety of reasons, the owners of the largest websites also are reluctant to publicize details of how they do or don’t profile individual users in vivid detail. And then there’s also the question of whether we even want to help improve technology whose main use is to improve the precision with which computers track individuals – but I don’t think that’s the front on which the privacy wars are best fought.

But if I were a computer science researcher right now, graph databases – optimized to support graph-analytic metrics — are one of the areas I’d look at to see if I could make an impact.

]]> http://www.dbms2.com/2009/08/21/social-network-analysis-aka-relationship-analytics/feed/ 15 http://www.dbms2.com/2009/04/15/cloudera-presents-the-mapreduce-bull-case/ http://www.dbms2.com/2009/04/15/cloudera-presents-the-mapreduce-bull-case/#comments Wed, 15 Apr 2009 06:22:13 +0000 Curt Monash http://www.dbms2.com/?p=751 Monday was fire-drill day regarding MapReduce vs. MPP relational DBMS. The upshot was that I was quoted in Computerworld and paraphrased in GigaOm as being a little more negative on MapReduce than I really am, in line with my comment

Frankly, my views on MapReduce are more balanced than [my] weary negativity would seem to imply.

Tuesday afternoon the dial turned a couple notches more positive yet, when I talked with Michael Olson and Jeff Hammerbacher of Cloudera. Cloudera is a new company, built around the open source MapReduce implementation Hadoop. So far Cloudera gives away its Hadoop distribution, without charging for any sort of maintenance or subscription, and just gets revenue from professional services. Presumably, Cloudera plans for this business model to change down the road.

Much of our discussion revolved around Facebook, where Jeff directed a huge and diverse Hadoop effort. Apparently, Hadoop played much of the role of an enterprise data warehouse at Facebook — at least for clickstream/network data — including:

• 2 1/2 petabytes of data managed via Hadoop
• 10 terabytes/day of data ingested via Hadoop (Edit: Some of these metrics have been updated in a subsequent post about Facebook.)
  
• Ad targeting queries run every 15 minutes in Hadoop
• Dashboard roll-up queries run every hour in Hadoop
• Ad-hoc research/analytic Hadoop queries run whenever
• Anti-fraud analysis done in Hadoop
• Text mining (e.g., of things written on people’s “walls”) done in Hadoop
• 100s or 1000s of simultaneous Hadoop queries
• JSON-based social network analysis in Hadoop

Some Facebook data, however, was put into an Oracle RAC cluster for business intelligence. And Jeff does concede that query execution is slower in Hadoop than in a relational DBMS. Hadoop was also used to build the index for Facebook’s custom text search engine.

Jeff’s reasons for liking Hadoop over relational DBMS at Facebook included:

• Price. Hadoop is free. MPP relational DBMS generally aren’t.
• Re-purposed data transformation logic. Facebook has lots of code sitting around in, e.g., Python to massage various specific kinds of data on its site. This code is re-used in Hadoop for ETL/ELT/ELTL/whatever.
• Resource management. Amazingly, Jeff found it easier to build a custom Hadoop resource manager to deal with the 100s or 1000s of concurrent queries than to rely on the native capabilities of a DBMS.
• Schema flexibility. This is a subject I’ve been preaching about for years. When people interact with web sites, the best schema to store data from their interactions changes just as quickly as the nature of their possible interactions does. Of course, when you add new features to a website, you can capture anything you like on a glorified entity-attribute-value basis. (Actually, I guess it would be more like EventDescriptor-SessionIdentifierClue-Timestamp.) But evolving a relational schema rapidly enough to keep up is hard. Facebook found it easier to evolve its Hadoop-based data massagers instead. (I’ve usually suggested running with XML or an XML-like approach, but notwithstanding the case of Marklogic/OpenConnect , that’s not usually the way network analytics implementers choose to go.)

More generally, Jeff argues there are tasks better programmed in Hadoop than SQL. He generally leans that way when data is complex, or when the programmers are high-performance computing types who aren’t experienced DBMS users anyway. One specific example is graph construction and traversal; there seems to be considerable adoption of Hadoop for graph analysis in the national intelligence sector.

• Two Oracle Spatial
• One “semantic,” which in Oracle lingo seems to mean — you guessed it — RDF
• One DICOM, which seems to be a medical imaging datatype.

• Text tokenization, indexing, and search
• Creation of other kinds of data structures (e.g., graphs)
• Data mining and machine learning

That covers all MapReduce apps I recall hearing about via commercial companies and users, and also includes most of what’s in the two big sources I found online. To wit:

1. In a slide presentation, Google offers the following applications of MapReduce:

• distributed grep
• distributed sort
• web link-graph reversal
• term-vector per host
• web access log stats
• inverted index construction
• document clustering
• machine learning
• statistical machine translation

2. The Hadoop applications page offers a rich trove of applications. Excerpts include:

• Aggregate, store, and analyze data related to in-stream viewing behavior of Internet video audiences.
• Analytics
• Analyze and index textual information
• Analyzing similarities of user’s behavior.
• Build scalable machine learning algorithms like canopy clustering, k-means and many more to come (naive bayes classifiers, others)
• Charts calculation and web log analysis
• Crawl Blog posts and later process them.
• Crawling, processing, serving and log analysis
• Data mining and blog crawling
• Facial similarity and recognition across large datasets.
• Filter and index our listings, removing exact duplicates and grouping similar ones.
• Filtering and indexing listing, processing log analysis, and for recommendation data.
• Flexible web search engine software
• Gathering world wide DNS data in order to discover content distribution networks and configuration issues
• Generating web graphs
• Image based video copyright protection.
• Image content based advertising and auto-tagging for social media.
• Image processing environment for image-based product recommendation system
• Image retrieval engine
• Large scale image conversions
• Latent Semantic Analysis, Collaborative Filtering
• Log analysis, data mining and machine learning
• Natural Language Search
• Open source social search tools.
• Parses and indexes mail logs for search
• Plot the entire internet
• Process apache log, analyzing user’s action and click flow and the links click with any specified page in site and more.
• Process clickstream and demographic data in order to create web analytic reports.
• Process data relating to people on the web
• Process documents from a continuous web crawl and distributed training of support vector machines
• Process whole price data user input with map/reduce.
• Produce statistics.
• Product search indices
• Recommender system for behavioral targeting, plus other clickstream analytics
• Reduce usage data for internal metrics, for search indexing and for recommendation data.
• Research for Ad Systems and Web Search
• Retrieving and Analyzing Biomedical Knowledge
• Run Naive Bayes classifiers in parallel over crawl data to discover event information
• Search engine for chiropractic information, local chiropractors, products and schools
• Serve large Lucene indexes
• Session analysis and report generation
• Source code search engine
• Statistical analysis and modeling at scale.
• Storage, log analysis, and pattern discovery/analysis.
• Store copies of internal log and dimension data sources and use it as a source for reporting/analytics and machine learning.
• Teaching and general research activities on natural language processing and machine learning.
• Vertical search engine for trustworthy wine information

There also were some research apps and some general processing speed-up apps I found harder to excerpt.

Some of our recent links about MapReduce

• The integration of MapReduce with SQL data warehousing
• Three major applications of MapReduce
• Another application of MapReduce
• Sound bites about MapReduce
• Other links about MapReduce

]]> http://www.dbms2.com/2008/08/26/known-applications-of-mapreduce/feed/ 13 http://www.dbms2.com/2008/02/16/stonebraker-database-taxonomy/ http://www.dbms2.com/2008/02/16/stonebraker-database-taxonomy/#comments Sat, 16 Feb 2008 23:21:26 +0000 Curt Monash http://www.dbms2.com/2008/02/16/stonebraker-database-taxonomy/ In a response to my recent five-part series on DBMS diversity, Mike Stonebraker has proposed his own taxonomy of data management technologies over on Vertica’s Database Column blog.

1. OLTP DBMSs focused on fast, reliable transaction processing
2. Analytic/Data Warehouse DBMSs focused on efficient load and ad-hoc query performance
3. Science DBMSs — after all MatLab does not scale to disk-sized arrays
4. RDF stores focused on efficiently storing semi-structured data in this format
5. XML stores focused on semi-structured data in this format
6. Search engines — the big players all use proprietary engines in this area
7. Stream Processing Engines focused on real-time StreamSQL
8. “Lean and Mean,” less-than-a-database engines focused on doing a small number of things very well (embedded databases are probably in this category)
9. MapReduce and Hadoop — after all Google has enough “throw weight” to define a category

He goes on to say that each will be architected differently, except that — as he already convinced me back in July — RDF will be well-managed by specialty data warehouse DBMS.

I must confess that I didn’t explicitly mention array-based data stores, whether scientific ones, nor the remaining native MOLAP (Multi-Dimensional OnLine Analytic Processing) engines, nor the sui generis SAS Intelligence Storage relational data warehouse product. So great catch there. On the not-so-great side, I think Mike’s definitions of categories #8 and #9 are a bit fuzzy (embedded DBMS tend to be full DBMS, but MapReduce is less than a DBMS). And of course any finite list like his will make over-general assumptions (e.g., it’s not obvious the StreamSQL-based CEP vendors will blow away rule-oriented Apama) and omit edge cases.

But there’s really only one point on which we have meaningful disagreement — Mike dumps all OLTP and general-purpose relational DBMS into a single bucket. Considering that such products currently represent a large majority of the multi-billion dollar DBMS market, I think some finer distinctions are in order. At a minimum, let’s break them into two categories — high-end vs. mid-range. High-end systems have maximum robustness, whether because there’s a real application need or because it just makes their owners feel good. Mid-range systems do everything high-end systems did in the 1990s, and are a cheaper/better alternative for ever more database management tasks.

The series on database diversity (more links at the bottom of Part 1):

• Part 1: Database management system choices – overview
• Part 2: Database management system choices – 4 categories of relational
• Part 3: Database management system choices – relational data warehouse
• Part 4: Database management system choices – mid-range-relational
• Part 5: Database management system choices – beyond relational

• Vertica’s claim to have “50” customers includes a bunch of unpaid licenses, many of them in academia.
• Vertica has about 15 paying customers.
• Based on conversations with mutual prospects, Vertica believes that’s more customers than DATAllegro has. (Of course, each DATAllegro sale is bigger than one of Vertica’s. Even so, I hope Vertica is wrong in its estimate, since DATAllegro told me its customer count was “double digit” quite a while ago.)
• Most Vertica customers manage over 1 terabyte of user data. A couple have bought licenses showing they intend to manage 20 terabytes or so.
• Vertica’s biggest customer/application category – existing customers and sales pipelines alike – is call detail records for telecommunications companies. (Other data warehouse specialists also have activity in the CDR area.). Major applications are billing assurance (getting the inter-carrier charges right) and marketing analysis. Call center uses are still in the future.
• Vertica’s other big market to date is investment research/tick history. Surely not coincidentally, this is a big area of focus for Mike Stonebraker, evidently at both companies for which he’s CTO. (The other, of course, is StreamBase.)
• Runners-up in market activity are clickstream analysis and general consumer analytics. These seem to be present in Vertica’s pipeline more than in the actual customer base.

• Fraud detection comes up as a specific application in multiple customer segments.
• RDF isn’t a big deal for Vertica yet. However, Vertica does have some RDF pilot projects in the biological research area.
• A lot of Vertica customers use Business Objects and/or Informatica. And as part of QA, Vertica’s product is tested against other major business intelligence tools as well.

As for the HP/Vertica appliance deal:

• Here’s the link to Vertica’s database appliance product page. Note that it mentions 10 terabytes of user data as a representative case.
• Vertica reports that a significant minority of its customers/prospects wanted an appliance alternative.
• HP now has what it surely perceives as a high-end/low-end pair of offerings – Neoview and Vertica. Similarly, Sun has what it perceives as a similar pair – Greenplum and ParAccel. Of course, neither Vertica nor ParAccel would wholly endorse that “low-end” positioning, but they’re glad to have the big-company partnerships even so.

Edit: For more on the data warehouse appliance market overall, please see this December, 2007 post on data warehouse appliance fact and fiction.

• When new kinds of data are managed, new kinds of data management are used. More precisely, the old ways are tried first — but once they fail new technologies are tried out.
• Up through the “Bowling Alley,” markets for nonstandard data management technology commonly follow the classic Geoffrey Moore pattern. However, they rarely experience a “Tornado” or mass adoption.
• I think this is apt to change. My three strongest candidates are native XML, RDF, and memory-centric event/stream processing used for data reduction (as opposed to sub-millisecond latency, which I do think will continue to be a niche requirement).