• Finding bad guys (national intelligence)
• Finding bad guys (anti-fraud)
• Data mining the social graph (e.g., for advertising optimization on social networks, or to identify influencers)

There are plenty more graph-oriented applications, of course, such as the identification of biochemical pathways. But I want to focus for now on ones like those on my list. My key points are:

• There are Big Data problems that lend themselves to graphical data models.
• So far as I can tell, the database management community isn’t doing enough to address them. (If I’m wrong about that, please tell me. I plan to arrive in Lyon for VLDB/XLDB Wednesday of next week, and of course I can always be reached by email.)

Here’s what I mean.

Applications that analyze relationship graphs are commonly grouped under the name social network analysis. As I frequently point out, category names and definitions tend to be imperfect, and that one is no exception. In particular — and the Wikipedia article on social networks and social network analysis is an excellent example of this – the term tends to be construed to cover the linkages between people or organizations, but not between, say, physical addresses, email addresses, and all the other stuff those intelligence applications actually track. I tried to introduce the term relationship analytics a while back, but it unfortunately didn’t stick.

I only ever got familiar with one company that tried to do a true graph-oriented database management system, suitable for social network analysis/relationship analytics. It was called Cogito, and had some interesting ideas about graphical data structures. Unfortunately, Cogito didn’t stick either.

As per the “Metrics” section of the Wikipedia article linked above, there are a number of well-established metrics about the relationships of pairs or groups of node to each other. The usual way to calculate these metrics is to load the graph into memory and get to work. (Indeed, such uses seem to be driving a lot of the national intelligence adoption of Hadoop.) And while I’m perfectly willing to believe that relational database management systems can do a fine job of managing generic RDF, it’s less obvious that they’re well-suited to support standard graph-analysis metric computations.

The reason, in a nutshell, is that the relational approaches usually boil down to maintaining a table with a row for every node-edge-node triple, and then doing a lot of fast self-joins to identify paths. That can work if connectivity is low and paths are sparse. But for higher degrees of connectivity, such strategies can lead – BOOM! — to serious combinatorial explosion. And that’s not good, because a lot of this analysis focuses on finding exactly the parts of the graph where the connections run thickest.

Cogito’s idea was to say “What if, for every node, you could retrieve in only a few blocks all the paths leading from it, at least up to pathlength N?” Unfortunately, Cogito’s approach to creating this effect had too little to do with optimizer development or selectively redundant data storage, and too much to do with wishful thinking; not coincidentally, Cogito is no longer around. (I haven’t kept in touch with Cogito’s successor 7 Degrees, and the reason hasn’t been lack of effort or interest on my part.)

But suppose the idea had worked. Then – unlike today – it might be realistic to do on-the-fly analytics on Very Large Graphs, just as we do operational business intelligence of a more relational or MOLAP nature. That would be cool.

How cool would it be? Well, that’s a bit hard to say. Look again at the list of applications I put up top. Those are NOT ones people generally talk a lot about. Spooks and fraud-fighters are two very secretive kinds of folks. And, for a variety of reasons, the owners of the largest websites also are reluctant to publicize details of how they do or don’t profile individual users in vivid detail. And then there’s also the question of whether we even want to help improve technology whose main use is to improve the precision with which computers track individuals – but I don’t think that’s the front on which the privacy wars are best fought.

But if I were a computer science researcher right now, graph databases – optimized to support graph-analytic metrics — are one of the areas I’d look at to see if I could make an impact.

]]> http://www.dbms2.com/2009/08/21/social-network-analysis-aka-relationship-analytics/feed/ 21 http://www.dbms2.com/2006/12/27/bulletin-on-cogito/ http://www.dbms2.com/2006/12/27/bulletin-on-cogito/#comments Wed, 27 Dec 2006 21:47:12 +0000 Curt Monash http://www.dbms2.com/2006/12/27/bulletin-on-cogito/ My Bulletin on Cogito — i.e., a short-short white paper — is now available for download. Thankfully, it turned out to be pretty consistent with what I previously wrote on the company and its technology. The conclusion to the paper bears quoting here:

In deciding between conventional DBMS and specialty graph-oriented tools such as Cogito’s, there’s one key criterion: Path length. If path lengths are short and predictable, there’s a good chance that relational DBMS and their forthcoming extensions can do the job. In complex graphs with longer paths, however, relational approaches may not scale well. In such cases, specialty technologies warrant serious consideration.

Disclosure: I have a business relationship with Cogito. Specifically, I will write a sponsored white paper about their technology. I also am informally (i.e., for no incremental pay at this time) advising them on their strategy. Indeed, the term “relationship analytics” is something I coined in my first briefing with them, and they immediately adopted it as their tagline. That briefing evidently had a lot to do with why they decided to pay my rates for a business relationship.

Confession: I haven’t yet finished the research for the aforementioned white paper, so I’m going to tapdance around some details in this post.

The basic idea behind Cogito’s storage mechanism and data architecture is that everything is an arc or node of a graph, or an attribute of same. Thus, there’s always a simple relational/tabular model that’s logically equivalent to a Cogito model, with one table for each type of node or arc.

But suppose your application involves tracking graph paths of nontrivial or indeterminate length. Returning a result via SQL (or some other tabular/relational query language) would require an ugly exponential explosion in the amount of work. Renowned SQL expert Celko, who’s literally written the book(s) on tree management in SQL, has documented this point on behalf of the company. (I invite Cogito — hi, WD! — to post a comment to this blog, with the relevant links of their choice.)

Cogito’s storage, however, is optimized very differently from a tabular system’s. For any given node, it stores ALL the arcs leading from it clustered together, even if those are of different types and would wind up in different SQL tables. The term I’ve coined for this is “starburst”. What’s more — for reasons I haven’t yet fully understood — it turns out that they can get a high hit rate for the following desirable outcome:

For a node referenced in a starburst, there’s a high likelihood that the node’s own starburst will be stored in the same memory block, or else in a contiguous one.

Thus, not only are all the arcs leading from a node clustered together, but most or all of the short-length graph paths are clustered together as well.

So what is this stuff good for? First of all, there are some apps that hard to describe beyond saying they retrieve and present information in the form of a relationship graph (e.g., a geneology website). Beyond that, Cogito has gotten good traction in law enforcement. Here the idea is that you’re looking for the needles of a few true relationships in an enormous haystack of apparent six-degrees-of-separation-style connections. The same “find the bad guy” kind of applications exist in principle in antifraud, epidemiology, and other bioinformatics areas, but I’m not aware of Cogito getting a lot of traction yet in those markets.

As for apps beyond “show the graph” and “find the bad guy” — well, that’s an area of research for me. Stay tuned.