• The same kinds of data are stored as before, with more being added over time.
• The same kinds of data are stored as before, but in more detail.
• New kinds of data are stored.

The first of those three ways doesn’t lead to dramatic growth. If a data warehouse goes up from 5 years of data to 6, then its overall size will grow a little over 20%. (How little depends on what the underlying business growth is – i.e., on how many more business events you have next year than you had 3 years ago.) That’s almost certainly going to be well-handled, by whatever technology manages your data warehouse today, given that:

• Chips are still subject to something resembling Moore’s Law.
• Disk capacity is still subject to Kryder’s Law, which is like Moore’s Law but with yet faster growth rates.
• DBMS software gets more performant over time.

So the cost of managing your same-as-before data will go down every year, even as the volume of that data grows.

True, disk rotation speeds have only increased 12.5 times since the Eisenhower Administration. But solid-state drives (SSDs) are getting practical for data warehousing fast, so even that bottleneck eventually will get swept away. And since what we’re discussing is, basically, the first and hence presumably highest-value data to be warehoused, it’s apt to wind up on SSDs before some other kinds of data warrant that treatment. So it’s the two other factors that drive the greatest data warehouse growth.

As costs go down, the wisdom of keeping detailed data goes up. I’d go so far as to say that every piece of data generated by a human being should be preserved and kept online, legal and privacy considerations permitting.* Most forms of capital-, labor-, and/or location-based competitive advantage being commoditized and/or globalized away. But information remains a unique corporate asset. Don’t discard it lightly.

*Unless there’s an explicit law mandating data destruction, legal considerations should permit. The idea “Let’s destroy something of irreplaceable value today, against the possibility we might be brought to judgment tomorrow” is both morally and pragmatically weird. Privacy, however, may be a different matter.

What that means in practice is that “disk is the new tape.” No-apologies performance can be had on data warehouse systems for $20,000/terabyte or less – perhaps even a lot less. Tolerable performance may cost 3-4X less than that. I think a lot of the growth in data warehouse volumes is of exactly this kind.

Ultimately, however, the greatest growth in data warehouse volumes will come from new kinds of data, especially data that is partly or wholly machine-generated. Moore’s Law applied to sensor chips tells us that data creation will grow just as fast as the data storage capacity. And thus we will be throwing away most machine-generated data forever. But what we keep will grow – well, it probably will grow at Moore’s/Kryder’s Law speeds.

That’s not to say new kinds of data are all high-volume/machine-generated. Back in 2005, I wrote two pieces for Computerworld advocating aggressive pursuit of new data sources, and the examples I mentioned were:

• Loyalty cards, especially in gaming
• Location-based analytics
• Extra customer feedback (e.g., opinion surveys)
• Price/offer testing
• Text mining in general
• Medical records

Today I’d add (among others):

• RFID
• The raw output from medical test devices
• Sensors up and down the energy supply chain

But some of those older, low-data-volume ideas still head my list of low-hanging analytic fruit.

One more complication – these buckets I’m outlining are less than precise. For example:

• Telecom CDRs (Call Detail Records) are machine-generated from a seed of human activity. They have long been stored, but now are being kept in much more detail. This is why telecommunications is one of the top markets for data warehouse technology.
• Stock trade data used to be based on human decisions. Now most of it is just machines buying and selling from each other. Either way, increasingly many investment institutions want to keep 100-terabyte-scale databases of complete historical trade detail. And that is why financial services is another huge market for data warehouse technology.
• Not long ago, web and network event logs. didn’t even exist, or were tiny where they did. Now they fill the largest known commercial databases, at firms such as Yahoo, eBay, and Facebook. Even so, more is thrown away than kept, especially on the network event side, which is a multiple of the size of the pure clickstream data.
• We don’t know exactly what all data intelligence agencies collect from telemetry, from monitoring commercial telecommunication traffic, and so on. But they’re surely throwing the vast majority away, even as the small part they keep is petabyte-scale.

But none of that interferes with my main points, which are:

• Databases will continue to grow very quickly.
• One big driver is the increasing detail in which data is kept online.
• An even bigger driver will be the unending ability of machines to generate ever greater streams of at-least-somewhat interesting data.

Splunk’s core technology indexes text and XML files or streams, especially log files. Technical highlights of that part include:

• Splunk software both reads logs and indexes them. The same code runs both on the nodes that do the indexing and on machines that simply emit logs. However, in the latter case indexing is turned off. Thus, Splunk does not portray its software as “agentless.” However, it asserts that its agent-like software runs without “material” overhead.
• The fundamental thing that Splunk looks at is an increment to a log – i.e., whatever has been added to the log since Splunk last looked at it.
• Splunk tries to figure out what the individual entries are in a section of log it looks at. In particular:
  • Time stamps are a big clue in this “inferencing” process, but they are not the be-all and end-all.
  • Nor are line boundaries, if logs are naturally broken up into lines. (Splunk threw that latter comment in as a shot at SenSage.)
• I get the impression that most Splunk entity extraction is done at search time, not at indexing time. Splunk says that, if a <name, value> pair is clearly marked, its software does a good job of recognizing same. Beyond that, fields seem to be specified by users when they define searches.
• Splunk has a simple ILM (Information Lifecycle management) story based on time. I didn’t probe for details.

Given its text search engine, Splunk does – well, it does text searches. And it stores searches, so they can be used for alerting or reporting. Indeed, Splunk persists and presumably updates results to stored searches, in a rough analog to materialized views.

Apparently, Splunk’s indexing is typically done via MapReduce jobs. I don’t know whether any actual Splunk searches are also done via MapReduce; surely they aren’t all, given the discussion of a near-real-time alerting engine and so on. Splunk fondly believes its MapReduce is an order of magnitude faster than SQL (I didn’t ask which SQL engines Splunk has in mind when they say this), and 5-10X faster than Hadoop. One efficiency trick is to look ahead and do Reduces in place where possible. This seems to be done automatically in the execution plan, ala Aster’s SQL-MapReduce, rather than having to be hand-coded. Splunk says its software can “easily” index 1-200 gigabytes of data per day on a commodity 8-core server, while maintaining an active search load, and 3-400 gigabytes are doable.

Splunk’s capabilities right now in tabular-style analytics seem to be limited to a command-line report builder, plus a GUI wizard that generates the command line. A few users have asked for support of third-party business intelligence tools, but Splunk hasn’t provided that yet. Nor can I find much evidence of ODBC/JDBC drivers for Splunk. But then, I have trouble understanding how Splunk could provide flexible and robust reporting unless it tokenized and indexed specific fields more aggressively than I think it now does.

]]> http://www.dbms2.com/2009/10/18/technical-introduction-to-splunk/feed/ 9 http://www.dbms2.com/2009/10/18/general-introduction-to-splunk/ http://www.dbms2.com/2009/10/18/general-introduction-to-splunk/#comments Sun, 18 Oct 2009 15:59:56 +0000 Curt Monash http://www.dbms2.com/?p=1119 I dropped by log analysis software vendor Splunk a few weeks ago for a chat with Marketing VP Steve Sommer (who some you may know from Cognos and/or Informix), Product Management VP Christina Noren, and above all co-founder/CTO Erik Swan. Splunk turns out to be a pretty interesting company, from both business and technical standpoints. For one thing, Splunk seems highly regarded by most people I mention it to.

Splunk’s technical stories include:

• Text search over log files.
• Business intelligence over text search. (That part sounds a lot like Attivio.)
• MapReduce with schema flexibility and smart multi-stage execution plans. (That part sounds a lot like Aster Data.)

More on those in a separate post.

Less technical Splunk highlights include:

• Splunk has ~1200 paying customers, and is adding a couple hundred more per quarter.
• Splunk has ~160 people.
• ~80% of Splunk sales are in North America.
• Typical Splunk sales prices are in the $10-50K range, with an average around $25K, or maybe that average is a bit over $30K. Some Splunk deals are six- or even seven-figure.
• Splunk is “quite profitable.”
• Splunk’s eponymous product is priced according to how much data is indexed per day. If you index half a gigabyte of logs per day or less, Splunk is completely free. So, while Splunk is closed-source, there’s something of an open-source-like Splunk adoption model.
• Splunk has been selling product for a couple of years. I gather Splunk 4 was recently released.
• Splunk’s biggest industry segments are, not too surprisingly,
  • Telco
  • Financial services
  • Government
  • “Online”
• Splunk’s paying customers seem to use it mainly for:
  • Web logs and associated network event logs (this seems to be the biggest area)
  • Security and perhaps other general IT log analysis
  • Physical security logs (mainly in the government)
  • Anti-fraud (I’m not sure how that works)
• One would think Splunk would be used to manage a lot of intelligence telemetry, but that wasn’t particularly hinted at.
• In general, the core problem Splunk is used for is log analysis for trouble-shooting purposes.
• Splunk’s nonpaying users are more diverse; examples mentioned included windmill operations and protein research.
• Splunk’s customers include Aster Data flagship accounts MySpace and LinkedIn. I bet many other top web companies are Splunk customers as well.

• Some are in heavy production with Hadoop, and closely engaged with Cloudera. Others are active Hadoop users but are very secretive. Yet others signed up for initial Hadoop training last week.
• Some have Hadoop clusters in the thousands of nodes. Many have Hadoop clusters in the 50-100 node range. Others are just prototyping Hadoop use. And one seems to be “OEMing” a small Hadoop cluster in each piece of equipment sold.
• Many export data from Hadoop to a relational DBMS; many others just leave it in HDFS (Hadoop Distributed File System), e.g. with Hive as the query language, or in exactly one case Jaql.
• Some are household names, in web businesses or otherwise. Others seem to be pretty obscure.
• Industries include financial services, telecom (Asia only, and quite new), bioinformatics (and other research), intelligence, and lots of web and/or advertising/media.
• Application areas mentioned — and these overlap in some cases — include:
  • Log and/or clickstream analysis of various kinds
  • Marketing analytics
  • Machine learning and/or sophisticated data mining
  • Image processing
  • Processing of XML messages
  • Web crawling and/or text processing
  • General archiving, including of relational/tabular data, e.g. for compliance

We went over this list so quickly that we didn’t go into much detail on any one user. But one example that stood out was of an ad serving firm that had an “aggregation pipeline” consisting of 70-80 MapReduce jobs.

I also talked yesterday again w/ Omer Trajman of Vertica, who surprised me by indicating a high single-digit number of Vertica’s customers were in production with Hadoop — i.e., over 10% of Vertica’s production customers. (Vertica recently made its 100th sale, and of course not all those buyers are in production yet.) Vertica/Hadoop usage seems to have started in Vertica’s financial services stronghold — specifically in financial trading — with web analytics and the like coming on afterwards. Based on current prototyping efforts, Omer expects bioinformatics to be the third production market for Vertica/Hadoop, with telecommunications coming in fourth.

Unsurprisingly, the general Vertica/Hadoop usage model seems to be:

• Do something to the data in Hadoop
• Dump it into Vertica to be queried

What I did find surprising is that the data often isn’t reduced by this analysis, but rather exploded in size. E.g., a complete store of mortgage trading data might be a few terabytes in size, but Hadoop-based post processing can increase that by 1 or 2 orders of magnitude. (Analogies to the importance and magnitude of “cooked” data in scientific data processing come to mind.)

And finally, I talked to Aster a few days ago about the usage of its nCluster/Hadoop connector. Aster characterized Aster/Hadoop users’ Hadoop usage as being of the batch/ETL variety, which is the classic use case one concedes to Hadoop even if one believes that MapReduce should commonly be done right in the DBMS.

Related link

• An August, 2008 round-up of MapReduce applications.

Key points include:

• This really is more like Oracle emulation than it is transparency, a term I carelessly used before.
• IBM’s Oracle emulation effort is focused on two technological goals:
  • Making it easy for an Oracle application to be ported to DB2.
  • Making it easy for an Oracle developer to develop for DB2.
• The initial target market for DB2’s Oracle emulation is ISVs (Independent Software Vendors) much more than it is enterprises. IBM suggested there were a couple hundred early adopters, and those are primarily in the ISV area.

Because of Oracle’s market share, many ISVs focus on Oracle as the underlying database management system for their applications, whether or not they actually resell it along with their own software. IBM proposed three reasons why such ISVs might want to support DB2:

• Oracle is expensive. In particular, IBM suggested it is more flexible on licensing terms for resale than Oracle is. I find that easy to believe.
• Hey, there’s a DB2 market or installed base out there of some size — why not address it?
• Acquisition-fueled expansion in applications makes Oracle a much bigger competitor to many ISVs (all around the world) than it used to be before. That one makes all kinds of sense.

And by the way — if I wanted an Oracle-emulating DBMS, I’d feel a lot happier about doing business with IBM than I would with EnterpriseDB.

IBM feels that DB2’s Oracle compatibility is a strict superset of EnterpriseDB’s, which it presumably has carried over more or less in its entirety. I didn’t press too hard for examples of what Oracle emulation DB2 offers and EnterpriseDB doesn’t, but IBM did say something about support for more programming languages. IBM was clear on one broad area where DB2 does not offer Oracle emulation, which is the specifics of various kinds of datatype support or other specialized data access methods. For example, IBM has its own syntax for querying text, geospatial, or XML data, and has not added support for Oracle’s alternative approaches.

Ron described MarkLogic Server as a DBMS for trees. That is, MarkLogic is designed for an XML data model that’s all about nodes and relationships, but not necessarily for XML data per se. The fundamental paradigm is thus searches for nodes and/or trees, for example:

• Find all the trees or nodes anywhere in the system that have certain text in them.
• Find all the nodes anywhere in the system that have certain text in their children or immediate descendants.
• Find all trees that have a pair of nodes, of a certain kind, in a certain relationship to each other.

Also important are aggregates and perhaps also — although Mark Logic rarely mentions them unless prompted — range queries.

So let’s start with some basics about Mark Logic Server’s indexing and storage strategy:

• As in a conventional text indexing system, most MarkLogic indexes are just term lists. A term is a token, which usually is just a word.
• MarkLogic can actually have two term lists for each token – one for the XML tags (i.e., attribute values), and one for the plain text content. (Most tokens, of course, never actually occur in a tag.)
• MarkLogic also indexes node names – i.e., attribute name, or type of node – and node relationships.
• “Document” has its usual meaning in MarkLogic Server. (This is worth saying because XQuery books sometimes assume that everything in an XML database is concatenated into a single cosmic document.)
• Documents can be broken up into fragments, with metadata as to how they fit together. The smallest unit MarkLogic retrieves is such a fragment (or of course an entire document).
• Largely because of how tag information is tokenized and stored, MarkLogic doesn’t store documents as serialized XML. Mark Logic says this eliminates a great deal of XML’s traditional bloat.
• If I understood correctly, Mark Logic said that a single document – perhaps one that’s 100K long as serialized XML – could be referenced by 1000s or 10s of 1000s of term lists. Frankly, that looks a little high to me, so it’s possible Mark Logic was talking about using that number of term lists for a medium- or large-sized collection. (Mark Logic says that MarkLogic has been used for databases up to the 100s of terabytes, and for civilian databases in the 10s of terabytes range.)
• MarkLogic’s term list indexes, I’m pretty sure, store not only where a term occurs but also what position in a document (section) it appears in.
• MarkLogic indexes also record the position of each node in a tree.
• Term lists are all ordered “the same way.” Therefore, Mark Logic claims, search time scales linearly with the number of documents.

In addition to those indexes, which comprise what Mark Logic calls the “Universal Index,” there are scalar indexes. These are columnar, where a column can cover either a single element name (i.e., attribute name) or a set of (presumably related) element names. Two copies of each column are kept, one sorted by TreeID and one by value. Mark Logic believes these suffice to give good performance on aggregations and range lookups.

With all those columns and term lists, the question naturally arises: What about the MarkLogic update strategy? Highlights include:

• MarkLogic uses MVCC (MultiVersion Concurrency Control). Updates are never done in place; rather, a new version of a record is appended.
• Documents are divided into subsets, each with its own set of indexes. Each subset that is being actively updated is small. This helps with update performance. It’s actually more important for the scalar columns than for the term lists. You update a term list by adding values to the end, so length hardly matters. But in a column index you insert new values into the middle, so length is indeed relevant.
• When document subsets get to be a certain size, they’re merged into yet larger subsets, which no longer have new documents added to them. This is when MVCC cruft is cleaned up as well. However, you can specify a minimum time that old versions are guaranteed to survive.
• This architecture gives an obvious form of partitioning, letting you parallelize in an obvious way. This partitioning is random by default, although there’s an API that lets you programmatically make it nonrandom.

Why not some kind of intelligent horizontal partitioning, whether range-based or otherwise? Well, we’ve finally gotten to a MarkLogic weakness. Join performance is not a MarkLogic long suit or Mark Logic priority. Indeed, Mark Logic insists that XML data is inherently denormalized, with joins (complex or otherwise) therefore rarely arising.

For many of today’s use cases, that’s probably true. For example, when I heard this I quickly started thinking “What if a book publisher changes name? That information is in a lot of individual book records.” But the fact is, people want author/publisher information that’s accurate as of the time of release, not updated for subsequent publishing company mergers and the like.

For other use cases, however, joins may be more important. For example:

• Derivatives contracts need to survive changes in brokerage firm corporate control. (IBM reports that derivatives are an important XML use case today.)
• Medical records need to be connected with a patient’s most current contact information. (IBM reports that XML is making inroads into health care applications.)
• Consumer profiling done right would benefit greatly from XML’s schema flexibility, but would also require significant use of joins.

]]> http://www.dbms2.com/2008/10/05/marklogic-architecture-deep-dive/feed/ 3 http://www.dbms2.com/2008/08/26/known-applications-of-mapreduce/ http://www.dbms2.com/2008/08/26/known-applications-of-mapreduce/#comments Tue, 26 Aug 2008 04:54:17 +0000 Curt Monash http://www.dbms2.com/?p=500 Most of the actual MapReduce applications I’ve heard of fall into a few areas:

• Text tokenization, indexing, and search
• Creation of other kinds of data structures (e.g., graphs)
• Data mining and machine learning

That covers all MapReduce apps I recall hearing about via commercial companies and users, and also includes most of what’s in the two big sources I found online. To wit:

1. In a slide presentation, Google offers the following applications of MapReduce:

• distributed grep
• distributed sort
• web link-graph reversal
• term-vector per host
• web access log stats
• inverted index construction
• document clustering
• machine learning
• statistical machine translation

2. The Hadoop applications page offers a rich trove of applications. Excerpts include:

• Aggregate, store, and analyze data related to in-stream viewing behavior of Internet video audiences.
• Analytics
• Analyze and index textual information
• Analyzing similarities of user’s behavior.
• Build scalable machine learning algorithms like canopy clustering, k-means and many more to come (naive bayes classifiers, others)
• Charts calculation and web log analysis
• Crawl Blog posts and later process them.
• Crawling, processing, serving and log analysis
• Data mining and blog crawling
• Facial similarity and recognition across large datasets.
• Filter and index our listings, removing exact duplicates and grouping similar ones.
• Filtering and indexing listing, processing log analysis, and for recommendation data.
• Flexible web search engine software
• Gathering world wide DNS data in order to discover content distribution networks and configuration issues
• Generating web graphs
• Image based video copyright protection.
• Image content based advertising and auto-tagging for social media.
• Image processing environment for image-based product recommendation system
• Image retrieval engine
• Large scale image conversions
• Latent Semantic Analysis, Collaborative Filtering
• Log analysis, data mining and machine learning
• Natural Language Search
• Open source social search tools.
• Parses and indexes mail logs for search
• Plot the entire internet
• Process apache log, analyzing user’s action and click flow and the links click with any specified page in site and more.
• Process clickstream and demographic data in order to create web analytic reports.
• Process data relating to people on the web
• Process documents from a continuous web crawl and distributed training of support vector machines
• Process whole price data user input with map/reduce.
• Produce statistics.
• Product search indices
• Recommender system for behavioral targeting, plus other clickstream analytics
• Reduce usage data for internal metrics, for search indexing and for recommendation data.
• Research for Ad Systems and Web Search
• Retrieving and Analyzing Biomedical Knowledge
• Run Naive Bayes classifiers in parallel over crawl data to discover event information
• Search engine for chiropractic information, local chiropractors, products and schools
• Serve large Lucene indexes
• Session analysis and report generation
• Source code search engine
• Statistical analysis and modeling at scale.
• Storage, log analysis, and pattern discovery/analysis.
• Store copies of internal log and dimension data sources and use it as a source for reporting/analytics and machine learning.
• Teaching and general research activities on natural language processing and machine learning.
• Vertical search engine for trustworthy wine information

There also were some research apps and some general processing speed-up apps I found harder to excerpt.

Some of our recent links about MapReduce

• The integration of MapReduce with SQL data warehousing
• Three major applications of MapReduce
• Another application of MapReduce
• Sound bites about MapReduce
• Other links about MapReduce

]]> http://www.dbms2.com/2008/08/26/known-applications-of-mapreduce/feed/ 13 http://www.dbms2.com/2008/01/27/the-4-main-approaches-to-datatype-extensibility/ http://www.dbms2.com/2008/01/27/the-4-main-approaches-to-datatype-extensibility/#comments Sun, 27 Jan 2008 05:26:46 +0000 Curt Monash http://www.dbms2.com/2008/01/27/the-4-main-approaches-to-datatype-extensibility/ Based on a variety of conversations – including some of the flames about my recent confession that mid-range DBMS aren’t suitable for everything — it seems as if a quick primer may be in order on the subject of datatype support. So here goes.

“Database management” usually deals with numeric or alphabetical data – i.e., the kind of stuff that goes nicely into tables. It commonly has a natural one-dimensional sort order, which is very useful for sort/merge joins, b-tree indexes, and the like. This kind of tabular data is what relational database management systems were invented for.

But ever more, there are important datatypes beyond character strings, numbers and dates. Leaving out generic BLOBs and CLOBs (Binary/Character Large OBjects), the big four surely are:

• Text. Text search is a huge business on the web, and a separate big business in enterprises. And text doesn’t fit well into the relational paradigm at all.
• Geospatial. Information about locations on the earth’s surface is essentially two-dimensional. Some geospatial apps use three dimensions.
• Object. There are two main reasons for using object datatypes. First, the data can have complex internal structures. Second, it can comprise a variety of simpler types. Object structures are well-suited for engineering and medical applications.
• XML. A great deal of XML is, at its heart, either relational/tabular data or text documents. Still, there are a variety of applications for which the most natural datatype truly is XML.

Numerous other datatypes are important as well, with the top runners-up probably being images, sound, video, time series (even though they’re numeric, they benefit from special handling).

Four major ways have evolved to manage data of non-tabular datatype, either on their own or within an essentially relational data management environment.

• Utterly standalone servers. There are lots of search engines, geospatial engines, object-oriented database management systems, and so on. Some may have ODBC/JDBC SQL interfaces, to handle metadata (which is commonly tabular in nature) if nothing else. But even so, there’s little relational about them.
• True RDBMS extensibility. In the 1990s, awkwardly named object-relational database management systems were introduced, boasting the awkwardly named feature abstract datatypes. Oracle, DB2, Informix, and PostgreSQL are now of this kind. They let one write data access methods for data that’s right in the basic relational table structure, and get at it through extensions to SQL.
• Tightly coupled servers. A close relative of RDBMS extension via new access methods is to create new servers for new datatypes, well-integrated with your RDBMS. Your parser and optimizer are in charge of federating them; the user just writes extended-SQL statements.
• User-defined functions. User-defined functions are like datatype extensions, but vastly easier to write, in that they don’t have any special access methods. When their performance is good enough, UDFs are often the best way to handle extended-datatype needs.

So how does this all play out in real-world examples? It’s all over the place.

• Enterprise text search is divided among three modes – integrated into the RDBMS (Oracle and IBM), tightly-couple server (Microsoft, pre-FAST acquisition), and standalone (Autonomy, FAST pre-acquisition, Google, and most other vendors).
• Geospatial datatypes are embedded into extensible DBMS – generally via technology from ESRI – for OLTP uses. But for data warehousing, where you don’t need pinpoint record retrieval, UDFs are generally believed to suffice. (E.g. Teradata, Netezza.)
• Intersystems seems to stand alone in getting nontrivial revenue from the standalone OODBMS market.
• The XML situation is really confused: Oracle has been late getting its native XML strategy together; the tightly-coupled DB2 Viper engine has been a performance disappointment; Microsoft’s integrated native XML isn’t heard from much either; and text/XML integrated engine Marklogic is getting some non-text business almost by default. In addition, every serious relational vendor has a capability to “shred” XML into relational tables, and can of course also just bulk-handle XML via BLOBs/CLOBs.

1. Relational database theorists like to talk about the “meaning” or “semantics” of data as being in the database (specifically its metadata, and more specifically its constraints). This is at best a very limited use of the words “meaning” or “semantics,” and has little to do with understanding the meaning of plain English (or other language) phrases, sentences, paragraphs, etc. that may be stored in the database. Hugh Darwen is right and his fellow relational theorists are confused.

2. The standard way to manage text is via a full-text index, designed like this: For hundreds of thousands of words, the index maintains a list of which documents the word appears in, and at what positions in the document it appears. This is a columnar, memory-centric approach, that doesn’t work well with the architecture of mainstream relational products. Oracle pulled off a decent single-server integration nonetheless, although performance concerns linger to this day. Others, like Sybase, which attempted a Verity integration, couldn’t make it work reasonably at all. Microsoft, which started from the Sybase architecture, didn’t even try, or if they tried it wasn’t for long; Microsoft’s text search strategy has been multi-server more or less from the getgo.

3. Notwithstanding point #2, Oracle, IBM, Microsoft, and others have SQL DBMS extended to handle text via the SQL3 (or SQL/MM ) standard. (Truth be told, I get the names and sequencing of the SQL standard versions mixed up.) From this standpoint, the full text of a document is in a single column, and one can write WHERE clauses on that column using a rich set of text search operators.

But while such SQL statements formally fit into the relational predicate logic model, the fit is pretty awkward. Text search functions aren’t two-valued binary yes/no types of things; rather, they give scores, e.g. with 101 possible values (the integers from 0 – 100). Compounding them into a two-valued function typically throws away information, especially since that compounding isn’t well understood (which is why it’s so hard to usefully federate text searches across different corpuses).

4. Something even trickier is going on. Text search can be carried out against many different kinds of things. One increasingly useful target is the tables of a relational database. Where a standard SQL query might have trouble finding all the references in a whole database to a particular customer organization or product line or whatever, a text search can do a better job. This kind of use is becoming increasingly frequent. And while it works OK against relational products, it doesn’t fit into the formal relational model at all (at least not without a tremendous amount of contortion).

5. Relational DBMS typically manage the data they index. Text search systems often don’t. But that difference is almost a small one compared with some of the others mentioned above, especially since it’s a checkmark item for leading RDBMS to have some sort of formal federation capability.