• Sumo Logic does SaaS (Software as a Service) log management.
• Sumo Logic is text indexing/Lucene-based. Thus, it is reasonable to think of Sumo Logic as “Splunk-like”. (However, Sumo Logic seems to have a stricter security/trouble-shooting orientation than Splunk, which is trying to branch out.)
• Sumo Logic has hacked Lucene for faster indexing, and says 10-30 second latencies are typical.
• Sumo Logic’s main differentiation is automated classification of events.
• There’s some kind of streaming engine in the mix, to update counters and drive alerts.
• Sumo Logic has around 30 “customers,” free (mainly) or paying (around 5) as the case may be.
• A truly typical Sumo Logic customer has single to low double digits of gigabytes of log data per day. However, Sumo Logic seems highly confident in its ability to handle a terabyte per customer per day, give or take a factor of 2.
• When I asked about the implications of shipping that much data to a remote data center, Sumo Logic observed that log data compresses really well.
• Sumo Logic recently raised a bunch of venture capital.
• Sumo Logic’s founders are out of ArcSight, a log management company HP paid a bunch of money for.
• Sumo Logic coined a marketing term “LogReduce”, but it has nothing to do with “MapReduce”. Sumo Logic seems to find this amusing.

What interests me about Sumo Logic is that automated classification story. I thought I heard Sumo Logic say:

• It’s largely unsupervised machine learning.
• It’s specific to a particular user/data set.
• It can be up and running and classifying things effectively almost instantly (i.e., on seconds’ or minutes’ worth of data).
• It’s informed by what different users tag as false positives. (Or maybe that is planned for future versions.)

I have a little trouble seeing how all those points fit exactly together, so perhaps I got some details wrong.

The payoff is that machine learning directly informs the Sumo Logic user interface. In particular, large numbers of events are bundled into a small number of categories, hopefully making it much easier for network operations types to scan the UI and pick out what’s important.

In general, the idea of machine-learning informing analytic UIs via some sort of classification is common in text-oriented technologies, notably in:

• Good ol’ text search.
• Text mining vendors’ approaches to clustering hits on words or phrases that say substantially the same thing.

But otherwise it seems kind of rare, if we stipulate that ad-serving/general internet personalization isn’t really an analytic UI — but I’d love to hear of any interesting examples I’ve overlooked.

T-Mobile’s instructions tell you that you need to know the factory defaults for network name and password. That makes sense. They don’t also tell you that you need to know your SIM card number (included), IMEI number (included), or authorization number (not included).

That’s right — you need a number that T-Mobile doesn’t tell you you need. But the story gets a lot worse from there, because it’s almost impossible to get the number from them. I eventually talked with approximately 8 T-Mobile call center associates over the course of the evening before getting successfully connected.

One of the few redeeming features in this story is that T-Mobile call center folks pick up the phone quickly. One of the many non-redeeming ones is that they efficiently give you inaccurate information after they do. The one who finally got me the right answer was a young woman who put me on hold to call an internal resource approximately four times before finally handling the situation correctly.

At one point I found somewhat helpful information by searching on T-Mobile’s website for mobile hotspot activation. However, the same information did not surface on my earlier searches on strings like activate mobile hotspot. Stemming has been a basic feature of search engines since the mid 1990s, but evidently T-Mobile’s technological choices aren’t as current as that. Other inexcusable T-Mobile mistakes include:

• Not providing information about the need for an “activation number” in the product’s paper documentation.
• Taking the buyer to a sign-on screen that doesn’t lead to the call center reps responsible for the product being signed-on for.
• Not providing call center operators with the tools they need to get callers to the right place.

Perhaps Elbonian* contractors were involved.

Elbonia is a fictitious country of outsourcers in Scott Adams’ Dilbert comic strip. Elbonian work is not noted for its high quality.

In case you haven’t guessed yet, the missing T-Mobile  “activation number” was tantamount to a telephone number, complete with area code. T-Mobile was insisting on assigning a telephone number for a service that had nothing to do with making or receiving telephone calls. While I can believe there was some legitimate database/application design reason for having such inflexibility under the covers, it’s hard to see why T-Mobile didn’t get a composite application tool and hack a front-end that automatically generates the number without call-center intervention.

I did eventually get connected, and in my limited experience with T-Mobile’s Prepaid/Mobile Hotspot 4G “Broadband” offering, I get the impression it has good speed but conventionally flaky WiFi reliability. It may well be a T-Mobile service that is worthy of great success. Edit: That’s not true.* But it’s not going to experience such success as long as T-Mobile idiotically infuriates its users at the relationship start.

*Edit: It turns out that the T-Mobile Mobile Hotspot device has terrible range. We can use it to get online in my office, Linda’s office, or the living room/dining room area, but no 2 of the 3 at once.

My takeaways from this story include:

• Use competent documentation writers.
• Run usability testing on your entire customer-experience processes.
• Test your site search engine for usefulness.

Beyond that, there’s not a single part of this story for which there isn’t a straightforward fix, most of them alluded to above.

If you see anything of your organization in this story, it’s probably time to raise your standards. This is obviously a different kind of failure as, say, the one last year at Chase. Even so, “as awful as T-Mobile” would be a sad state to endure.

• More-of-the-same in line with MarkLogic’s core positioning.
• A new bi-directional Hadoop connector.
• A free MarkLogic Express edition, limited in license terms more than in actual features, as per Slide 27 of the deck MarkLogic graciously supplied for me to post.

Also, MarkLogic is early with a feature that most serious DBMS vendors will soon have – support for tiered storage, with writes going first to solid-state storage, then being flushed to disk via a caching-style algorithm.* And as befits a sometime search-engine-substitute, MarkLogic has finally licensed a large set of document filters, from an Australian company called Isys. Apparently, the special virtue of the Isys filters is that they’re good at extracting not only text, but metadata as well.

*If there’s a caching algorithm that doesn’t contain a major element of LRU (Least Recently Used), I don’t recall ever hearing about it.

MarkLogic seems to have settled on a positioning that, although distressingly buzzword-heavy, is at least partly based upon reality. The real part includes:

• MarkLogic is a serious, enterprise-class DBMS (see for example Slide 12 of the MarkLogic deck) …
• … which has been optimized from the getgo for poly-structured data.
• MarkLogic can and does scale out to handle large amounts of data.
• MarkLogic is a general-purpose DBMS, suitable for both short-request and analytic tasks.
• MarkLogic is particularly well suited for analyses with long chains of “progressive enhancement” (MarkLogic’s favorite term when talking about derived data).
• MarkLogic often plays the role of a content assembler and/or search engine, and the people who use MarkLogic in those ways are commonly doing things that can be described as research and analysis.

Based on that reality, MarkLogic talks a lot about Volume, Velocity, Variety, Big Data, unstructured data, semi-structured data, and big data analytics.

My November, 2010 overview of MarkLogic technology remains pretty relevant. One correction, however: Node heterogeneity configurations, in which “data” and “evaluation” nodes reside on separate servers, are the exception rather than the rule.

Like Vertica, MarkLogic has laudably said that true academic researchers can get MarkLogic for free without the severe license restrictions. Free MarkLogic should be of particular interest to researchers who:

• Are studying natural networks or graphs, such as social networks or biological pathways. (This might be a fit in the social or biological sciences.)
• Are managing metadata for, say, a variety of disparate kinds of experimental files. (This might be a fit anywhere in the natural sciences.)
• Are managing actual documents, images, videos, etc., or data about such things. (This might be a fit in the humanities or social sciences.)

MarkLogic provided some disclosable financial substance by email, which I shall quote verbatim:

• MarkLogic has 45% revenue growth and 55-60% license growth year over year.
• We expect to finish this year with over $85 million in revenue, up from $55 million last year.

Arithmetical purists might note that 85/55 is more than 145%, but I’m just going to settle for the information I got and move on.

Edit: I posted separately about the MarkLogic Hadoop connector. As for that Hadoop connector – stay tuned for a short follow-up post, as writing about it now would not be convenient. (My backup discipline isn’t what it should be, and the only copy of my notes about that product is on a heavy tower computer in a house that doesn’t have working power.)

I’ve gone on for two long posts about text data management already, but even so I’ve glossed over a major point:

Using text data commonly involves a long series of data enhancement steps.

Even before you do what we’d normally think of as “analysis”, text markup can include steps such as:

• Figure out where the words break.
• Figure out where the clauses and sentences break.
• Figure out where the paragraphs, sections, and chapters break.
• (Where necessary) map the words to similar ones — spelling correction, stemming, etc.
• Figure out which words are grammatically which parts of speech.
• Figure out which pronouns and so on refer to which other words. (Technical term: Anaphora resolution.)
• Figure out what was being said, one clause at a time.
• Figure out the emotion — or “sentiment” — associated with it.

Those processes can add up to dozens of steps. And maybe, six months down the road, you’ll think of more steps yet.

So when you manage text, it is convenient to assume dynamic schemas. That would be an argument for using MarkLogic, NoSQL document stores, and/or Hadoop, rather than strictly relational systems.

That said, text analytics can be done perfectly well in relational databases. Again, I point you to the example of Attensity, which will extract for you a large fraction of the information that can be gotten out of the text, put it into a convenient relational schema, and let you get to work. Once the principal extraction has been done, there’s no reason why your derived data issues need be any more complex than others you deal with relationally, especially on the analytic side of the house.

But what if you want to do your own text enhancement, rather than using a third party tool? The first thing to ask yourself is — why? With all due respect to the 10-20 internet-centric companies that are having fun reinventing large portions of the data processing wheel — if you’re not at one of those companies, you should probably be trying to use as much third-party software as you possibly can.

I can think of a couple of cases where rolling your own technology make sense, namely:

• The hard part of what you’re doing is extracting snippets of text from some data format proprietary to you.
• You’re trying to do very simple things across a variety of languages much broader than the 10-20 that the text analytics vendors currently do a halfway decent job of handling.

I can’t think of many others.

One thing I’d definitely be wary of is using Hadoop as a big bit bucket for individual documents in a variety of formats. I don’t know what you’d do with them once they’re there. Yes, Google invented MapReduce in part to do things like document indexing — but you’d probably prefer not to reinvent the Google stack. That’s quite apart from questions as to whether your document count exceeds Hadoop’s comfortable file-count limit. Solr is a different matter; but while Solr and Hadoop are both open source projects that can be traced back to Doug Cutting, otherwise they’re rather different things.

A useful way of looking at your choices may be to ask:

After text has run through the main pipeline of manipulation and information extraction:

• What will the output look like?
• Where do I want that output to end up?

If the output has to be something that fits into a structured/relational analytic system, then it should probably go into a relational DBMS. If you’re going to do social network analysis of the sort you’d ideally like to do in a graph database — well, unless you’re an intelligence agency with blank-check resources, you’ll probably still end up opting for a relational DBMS. If the output consists of simple, homogeneous text files, plus a few fields of metadata, and you’re not going to do much analysis of it, it can pretty much go anywhere; either SQL or NoSQL might suit your purposes. If you want maximum power and flexibility, MarkLogic may be the ideal destination.

From there, the next question is:

• What pipeline should the text run through to get to its final destination?

Often, as I’ve argued, the right answer is a third-party text analytic system. Those can generally consume text in almost any kind of file format. Other times — less often than you may think — it’s Hadoop. OK, then pass it through Hadoop. Other possibilities could come up as well (text search engines aren’t really as usually as I may have seemed to be suggesting).

Anyhow, when you’ve established where text starts out (that’s usually a given), what it passes through (please see above), and where its best parts need to end up (ditto), you’ve done the hardest parts. Figuring out the rest of your text management architecture should be relatively easy by comparison.

1. Confusion about text data management.
2. Choices for text data management (general and short-request).
3. Choices for text data management (analytic).

I’ve recently given widely varied advice about managing text (and similar files — images and so on), ranging from

Sure, just keep going with your old strategy of keeping .PDFs in the file system and pointing to them from the relational database. That’s an easy performance optimization vs. having the RDBMS manage them as BLOBs.

to

I suspect MongoDB isn’t heavyweight enough for your document management needs, let alone just dumping everything into Hadoop. Why don’t you take a look at MarkLogic?

Here are some reasons why.

There are three basic kinds of text management use case:

• Text as payload.
• Text as search parameter.
• Text as analytic input.

The simplest way to manage text electronically is to:

• Store it as whole documents — scanned images, .PDFs, word processing files, whatever.
• Find it only via fielded metadata, perhaps manually created — title, author, date, and so on.

For example, an application for college admission is accompanied by recommendation letters, transcripts, and so on; those are then moved around as dumb payloads, until such time as an admissions officer reads them. Most relational database management systems can manage BLOBs (Binary Large OBjects), but performance may be better if you leave the big objects outside the relational system. For text-as-payload, that way of managing documents can often suffice.

In other cases, the text may be so short that it naturally fits into a character field in a relational database. This is particularly likely when the text is typed in at the time of record creation, e.g. by a call center operator, a doctor, or a customer entering a support ticket. In such cases, leaving it under the management of an RDBMS makes perfect sense.

In many situations you actually want to search based on the context of the text. Unless you’re doing simple search on short text snippets in relational character fields, that generally calls for some kind of text index. Text indexes are generally found in text search engines. That said, however:

• Anything that can be done in a standalone text search engine can in principle also be integrated into relational DBMS and other data stores. (How well that works in practice is another matter.)
• Text search engines commonly index data in situ that they don’t actually manage.

In theory, then:

• You can store your text in your chosen DBMS or outside it, as pleases you.
• Your chosen DBMS can have a text search capability.
• This text search capability can be integrated with the query method used to get at the rest of the data managed by that DBMS.

That theory is commonly reflected in actual products, such as Oracle and DB2.

As so often, then, the choice of how to manage text comes down to issues such as performance, programming ease, and other components of total cost of ownership (or of some other general “goodness” metric, such as time-to-value). As a general rule, it seems:

• Text indexing inside relational DBMS has poorer performance than in, say, text search engines, often drastically so.
• BLOB management inside relational databases has poorer performance than leaving the files outside the DBMS’ purview.
• Relational DBMS do just fine at managing text strings up to, say, 2048 characters long.
• Tight integration between  text search and SQL is valuable in a few applications, but irrelevant to many others.

And so, to a first approximation:

• If you just have short text snippets, it can make sense to leave them in your relational database.
• If performance is not an issue, you can just leave your BLOBs in your relational database too.
• If performance is an issue, you probably want to have your larger text files outside your RDBMS’ control.

However, that phrasing assumes the default option is a relational DBMS, which may not be the case at all. Other choices include:

• Standalone text search engines. If you want the best available text search, get a search engine. But attempts to start with a search engine and wind up with an application platform have generally run into difficulty.
• Document-oriented (or other) NoSQL systems. The story here is surprisingly like that for relational DBMS. I’ve previously noted that document-oriented NoSQL systems manage objects, not “documents” in the ordinary sense of the word. Even so, conceptually they’re no less suited for management of true documents than relational DBMS are. I’d guess that the correlation between use cases involving true documents and use cases where document-oriented NoSQL is suitable is positive, but not very strong.
• MarkLogic. From one standpoint, MarkLogic is just a heavier-weight version of document-oriented NoSQL. But MarkLogic’s XML (and XQuery) orientation, tuned-for-years indexing, and built-in search engine put it on a different level for document management than the upstarts have reached.

I’ll cover the Hadoop option in the next, more analytically-focused post.

I hope I’ve demonstrated that there are appropriate use cases for each of:

• Letting documents be managed by the file system (and pointing to them from your preferred DBMS).
• Sticking documents straight into your preferred DBMS (SQL or non-SQL as the case may be).
• Using a specialty system such as MarkLogic (or of course, in some cases, an enterprise search engine).

And that’s even before we move on to analytically-oriented text data management.

1. Confusion about text data management.
2. Choices for text data management (general and short-request).
3. Choices for text data management (analytic).

There’s much confusion about the management of text data, among technology users, vendors, and investors alike. Reasons seems to include:

• The terminology around text data is inaccurate.
• Data volume estimates for text are misleading.
• Multiple different technologies are in the mix, including:
  • Enterprise text search.
  • Text analytics — text mining, sentiment analysis, etc.
  • Document stores — e.g. document-oriented NoSQL, or MarkLogic.
  • Log management and parsing — e.g. Splunk.
  • Text archiving — e.g., various specialty email archiving products I couldn’t even name.
  • Public web search — Google et al.
• Text search vendors have disappointed, especially technically.
• Text analytics vendors have disappointed, especially financially.
• Other analytic technology vendors ignore what the text analytic vendors actually have accomplished, and reinvent inferior wheels rather than OEM the state of the art.

Above all: The use cases for text data vary greatly, just as the use cases for simply-structured databases do.

There are probably fewer people now than there were six years ago who need to be told that text and relational database management are very different things. Other misconceptions, however, appear to be on the rise. Specific points that are commonly overlooked include:

• The terms “unstructured” or “semi-structured” data are inherently misleading. That’s why I favor “multi-structured” or “poly-structured” instead. (“Multi-structured” seems to be winning; e.g., it’s been adopted by Teradata and Teradata/Aster.)
• The “social media” text data any one enterprise brings in house isn’t all that much. For example, Attensity serves many different enterprises’ social media needs from a single 20-terabyte data store, and reports that no single enterprise has required as much as 1 terabyte of text yet. Text data may consume a lot of storage on spinning disks somewhere, but it’s not that big a factor in future DBMS industry growth. (That 20 terabyte figure does seem low.)
• Structured databases are typically worth a lot more per bit than other kinds. The most valuable electronic data, per-bit, is probably records of significant economic transactions — purchases, sales, money transfers, etc. The least valuable may be sensor log files, whose contents consist mainly of “Nothing going on here; ping you again in a minute.” Email logs, web interaction data and many other kinds fall somewhere in between. Highly valuable documents — such as signed contracts — generally persist in paper as well as electronic forms. Investors commonly overlook this point.
• The enterprise text search industry is screwed up.
  • FAST was a goofy company before it was acquired for far too much money by Microsoft.
  • Autonomy was a goofy company before it was acquired for far too much money by HP.
  • Google’s enterprise efforts are quiet.
  • The integration of text search and relational DBMS — e.g. at Oracle — has languished, with poor performance and evident lack of management attention.
  • Smaller text search vendors don’t seem to be getting a lot of traction — e.g., Coveo has a decent reputation, but when’s the last time you heard much about them? What has Attivio actually accomplished?
• Text analytics is a small business. Add up the revenue for Attensity, Clarabridge, Lexalytics, Temis, and all the others, and you might poke above $100 million, especially now that Attensity had a three-way merger. Then again, you might not.
• Even so, the text analytics vendors have developed sophisticated technology. In particular, you can use it to get a pretty good idea as to what people are writing about you, individually or as groups.

• Derived data.
• Many-step processes to produce derived data.
• Schema evolution.
• Temporary data constructs.

So let’s dive in. 

When I started my discussion of derived data, I focused on five kinds:

Aggregates, when they are maintained, generally for reasons of performance or response time.

Calculated scores, commonly based on data mining/predictive analytics.

Text analytics.

The kinds of ETL (Extract/Transform/Load) Hadoop and other forms of MapReduce are commonly used for.

Adjusted data, especially in scientific contexts.

Later I added a sixth kind:

• Derived metadata, commonly for polystructured data sets (logs, text, images, video, whatever).

More kinds may yet follow.

In all cases, I was (and am) talking about data that is actually persisted into the database. Temporary tables — for example the kind frequently created by Microstrategy — are also important in data processing, as is temp space managed solely for the convenience of the DBMS. But neither are what I mean when I talk about “derived data.”

As I noted back in June, derived data naturally leads to schema evolution. You load data into an analytic database. You do some analysis and get some interesting results — interesting enough for you to want to keep them persistently. So you extend the schema to include them. You do more research; you discover something else interesting; you extend the schema again. Repeat as needed.

However, in no way is derived data the only source of analytic schema evolution. Duh. Sometimes you just have new kinds of information coming in. Of course, once it’s there, you may want to derive something from it. In marketing contexts, both parts of that might be true in spades.

When I mentioned all this to my clients at MarkLogic — which was my inspiration for the polystructured/metadata example — they perked up and said “Oh! Progressive enhancement.” Indeed, it’s long been the case that a simple text processing pipeline could have >15 steps of extraction; indeed, I learned about the “tokenization chain” in 1997. If all the “progression” in  the data enhancement occurs in a single processing run, that wouldn’t necessarily spawn much schema evolution. On the other hand, if you think of additional steps to add every now and then — in that case your schema might indeed evolve over time.

Somewhat similarly, Hadoop can be used to run “aggregation pipelines” of many 10s of steps. The output of the whole thing might be a relatively small number of fields. But again, if the number or nature of the fields changes over time, schemas will need to evolve accordingly.

So to sum up:

• Derived data — of multiple kinds — is very important.
• If you want to increase the value you get from derived data, you might need to evolve your schema accordingly.
• Data derivation happens to sometimes have long processing pipelines; those pipelines might happen to offer clues as how to do yet better at data derivation in the future; those improvements might happen to lead to schema evolution over time.

“Just the raw facts” analytic databases are, for the most part, obsolete.

• HP is buying Autonomy.
• HP is pulling back from WebOS.
• HP may spin off its PC business altogether.

On a high level, this means:

• HP is doubling down on enterprise IT.
• HP is taking a more software-centric approach to the enterprise IT business.
• HP is backing away from the consumer electronics business.
• HP in particular is backing away from the generic desktop/laptop PC business, which may with only moderate exaggeration be regarded as:
  • The intersection of the enterprise IT and consumer electronics businesses.
  • The least attractive sector of each.

My coverage of Autonomy isn’t exactly current, but I don’t know of anything that contradicts long-time competitor* Dave Kellogg’s skeptical view of Autonomy. Autonomy is a collection of businesses involved in the management, search, and retrieval of poly-structured data, in some cases with strong market share, but even so not necessarily with the strongest of reputations for technology or technology momentum. Autonomy started from a text search engine and a Bayesian search algorithm on top of that, which did a decent job for many customers. But if there’s been much in the way of impressive enhancement over the past 8-10 years, I’ve missed the news.

*Dave, of course, was CEO of MarkLogic.

Questions obviously arise about how the Autonomy acquisition relates to other HP businesses. My early thoughts include: 

• HP has clearly signaled that it intends to pursue and focus on the data management business. Thus, we can anticipate marketing messages spanning Autonomy and Vertica. It may be helpful to recall that Vertica plays nicely with both Hadoop and Attensity.
• The first two natural tuck-in acquisitions I can think to add are Attensity and MarkLogic.
• One place I’d look for synergy is with HP’s system management software business. HP has previously acquired its way into a strong position there. If you add in knowledge of how many kinds of data are used, you have a chance to set yourself apart in the system management area.
• I had enough trouble advising Vertica about how to explain what they do in terms that HP’s hardware sales force can comfortably embrace. I think I did OK with that. But Autonomy? Youch. On the other hand, …
• … HP is run by guys from SAP (Leo Apotheker) and Oracle (Ray Lane), both of whom have dealt with similarly tough sales challenges before. But even at best, HP’s sales force organization, commission structure, and training is going to consume a lot of attention at the very highest levels of HP.
• Autonomy manages documents electronically. HP prints them. The markets where that seems synergistic, however, are fairly specialized or small. (E.g., equipment for printing on demand.) Perhaps there’s some grand joint venture possibility with Xerox here, antitrust permitting.

• I’m posting draft slides.
• I’m encouraging comment (especially in the short time window before I have to actually give the talk).
• I’m offering links below to more detail on various subjects covered in the talk.

The talk concept started out as “advanced analytics” (as opposed to fast query, a subject amply covered in the rest of any Netezza event), as a lunch break in what is otherwise a detailed “best practices” session. So I suggested we constrain the subject by focusing on a specific application area — customer acquisition and retention, something of importance to almost any enterprise, and which exploits most areas of analytic technology. Then I actually prepared the slides — and guess what? The mix of subjects will be skewed somewhat more toward generalities than I first intended, specifically in the areas of investigative analytics and derived data. And, as always when I speak, I’ll try to raise consciousness about the issues of liberty and privacy, our options as a society for addressing them, and the crucial role we play as an industry in helping policymakers deal with these technologically-intense subjects.

Slide 3 refers back to a post I made last December, saying there are six useful things you can do with analytic technology:

• Operational BI/Analytically-infused operational apps: You can make an immediate decision.
• Planning and budgeting: You can plan in support of future decisions.
• Investigative analytics (multiple disciplines): You can research, investigate, and analyze in support of future decisions.
• Business intelligence: You can monitor what’s going on, to see when it necessary to decide, plan, or investigate.
• More BI: You can communicate, to help other people and organizations do these same things.
• DBMS, ETL, and other “platform” technologies: You can provide support, in technology or data gathering, for one of the other functions.

Slide 4 observes that investigative analytics:

• Is the most rapidly advancing of the six areas …
• … because it most directly exploits performance & scalability.

Slide 5 gives my simplest overview of investigative analytics technology to date: 

• Fast query
  • Persistent storage (any data volume)
  • RAM (10s -100s of gigabytes, or more)
• Fast analytics
  • Predictive modeling
  • Transformation/tagging
  • Graph

Slide 6 points out that this is all supported by cheap data creation and acquisition, specifically in the area of machine-generated data, which gets the full benefit of Moore’s Law.

Slides 7-13 point out how the example problem domain involves lots of analytic tasks performed on lots of kinds of data. Specific examples cited include text analytics and graph/relationship analytics.

Slide 14 contains the punch line, so I’ll quote it in full:

Derived data

• You can’t keep re-analyzing all that in raw form …
• … so don’t.

If you have one takeaway from this session, let it be the utter importance of derived data.

Slide 16 lists kinds of derived data that are important in the single application of reducing telco churn:

• Normalized data
  • Parsed/sessionized logs
  • Text/sentiment highlights
  • Social network graph(s)
  • Web de-anonymization
  • Household matching
• Scores and buckets
  • Demographic
  • Psychographic
  • Offer hot buttons
  • (Dis)satisfaction
  • Credit/fraud risk
  • Lifetime customer value
  • Influence on others!

And finally, Slide 17 is my first pass at best practices for dealing with derived data:

• Evolving data warehouse schema
• Data marts
  • Physical or virtual
  • Inputs/outputs to “EDW”
• “Data science”
  • Research != production
• Multiple processing pipelines
  • Log parsing
  • Text
  • Predictive analytics
  • Generic ETL
  • Streaming “ETL”

That last list looks like a starting point for a whole set of interesting future posts.

*Everybody seems to like “poly-structured” better when it has a hyphen in it — including me.

The big difference between the two is that “multi-” just means there are multiple structures, while “poly-” further means that the structures are subject to change. Upon reflection, I think the “subject to change” part is essential, so poly-structured it is.

The definitions I’m proposing are:

• A database is poly-structured to the extent that its structure is apt to be changed in the ordinary course of query, update, or programming.
• Data is poly-structured to the extent that it is best represented in a poly-structured database.
• A DBMS is poly-structured to the extent that it is oriented to managing poly-structured databases.

Please note:

• There are many different degrees of being poly-structured; that’s why I used the phrase “to the extent that”, instead of a simple “if”.
• And as always, no technology categorization is ever precise.

Examples of poly-structure include:

• XML or JSON documents/objects describe themselves. Add a new one to a database with a different structure than the others and — presto! — you have changed the overall structure. Thus:
  • XML and JSON data is apt to be poly-structured.
  • XML and JSON databases are apt to be poly-structured.
  • MarkLogic, MongoDB, et al. are poly-structured DBMS.
• A text document is inherently poly-structured. Some queries might look at it as a bag of words; others might group the words via stemming and synonyms; others might actually exploit the document’s grammatical structure. Text search engines are poly-structured because they support all those kinds of queries.
• A single log file can be somewhat poly-structured, in that different views of it might extract different kinds of name-value pair, or different temporal relationships.
• A database that seamlessly includes a variety of log files, each with its own structure(s), is quite poly-structured.
• A classic relational database is not very poly-structured, because DDL (Data Description Language) isn’t really in “the ordinary course” of programming or update.
• However, views add a bit of poly-structure to relational databases that is not present in, say, IMS databases.
• An object-oriented DBMS is highly poly-structured, as is Workday’s internal data store.

So what do you think? Do these definitions work?