Also, Aster no longer focuses much on the general internet industry where it got its earliest sales, its continued success at LinkedIn and a recent win at an (NDA) fairly-big-name internet new account Razorfish notwithstanding. That said, the first target market Aster did share with me was “digital marketing optimization,” which includes “marketing optimization” (duh), search engine optimization (SEO), clickstream analysis, and the like. Also, Aster is going after “data scientists” in general, and that’s a term I’m still seeing used most frequently in the internet area.

I’m seeing ever more granularity as companies break down internet-related market segments. DataStax showed me a chart last week of 15 different market segments it had sold into, and at least 14 were in some way internet-related.

Rather, if Aster is to name three industries in which it has pleasingly strong sales traction, it would say manufacturing (which in Teradata lingo includes resource extraction), financial services (including insurance), and retail. A cynic might note that that breakdown, like many similar ones, adds up to fairly large swaths of the economy and the computer market, but never mind that part. (Other firms might have thrown in telecommunications and health care as well, to get even more coverage.

Two of Aster’s other favorite application areas are social network analysis/influencer identification and — which is analytically very similar — fraud detection/prevention. Taken together, that’s a whole lot of graph analysis. And I note with interest that the influencer identification stuff does NOT seem to be concentrated in telecom, which is the traditional sector one would imagine it being used in; all those call records are a lovely source of graph edges. Rather, the influencers seem to be identified from sources such as social media and credit card data .

Once again, this kind of thing gives me privacy jitters.

The match between Aster’s favorite industries and application areas is pretty much as you might expect — fraud in financial services, influencer analysis in retailing (and probably consumer financial services too), and digital marketing in both. As for manufacturing, the opportunities there seem to be focused on machine-generated data. That would be at least in high-tech manufacturing (I bet especially in flow-oriented stuff such as semiconductor fab) and oil/gas. Smart grid opportunities don’t seem to have arisen yet for Aster the way they have for a couple other vendors.

As for general Aster business trends, I think they’re good, while Aster would perhaps want to portray them as very good. Aster named a couple of impressive joint Teradata/Aster wins under NDA, but only a couple. Ramping up sales headcount is proving challenging, and some sales leadership turnover probably hasn’t helped. I do believe Aster’s spin that this is a matter of somebody being promoted quickly to a bigger job, and am optimistic about the current team — still, such moves tend to have at least short-term cost.

• In this release, Vertica UDTFs can only be written in C++. Other UDTF languages are promised.
• Otherwise, Vertica UDTFs sound pretty flexible; in particular:
  • They can ingest and emit any number of rows.
  • Their assumed schemas can be defined programmatically (both input and output).
• Vertica UDTFs go in the SELECT clause, not the FROM clause. I must confess to not grasping Vertica’s argument as to why this provides great and important flexibility.
• UDTF syntax mirrors SQL 99 Analytics pretty closely.

*It looks like the “F” is in the official name, but will often be dropped colloquially.

Other Vertica analytic platform highlights include:

• Proper integrated UDT workload management is promised, and there’s a little bit of UDT workload management already.*
• Vertica is delivering some prebuilt functions for aggregation, statistics, etc.
• Vertica has cool temporal and time series features.
• Vertica’s geospatial support seems pretty basic (circles and rectangles).
• Vertica’s NDA plans moving forward are pretty much as one would hope.

*Vertica’s UDT workload management is RAM-only, and “honor system” — i.e., it assumes that the UDTFs declare their resource usage correctly, which Vertica says is the right way to handle in-process C++ routines.

Vertica also argues that fast-performing SQL in and of itself can amount to analytic functionality. For example, Vertica has tried to ensure that it offers great performance in the kinds of self-joins that are used in graph analysis. Since Vertica has plenty of customers among the kinds of Web and telco companies that use graph analysis today, I’m inclined to grant some benefit of the doubt here. That said, Vertica thinks 3 hops is plenty for most kinds of graph analysis people want to do, and I can think of applications (e.g. anti-terrorism) where that’s surely not the case.

• I’m posting draft slides.
• I’m encouraging comment (especially in the short time window before I have to actually give the talk).
• I’m offering links below to more detail on various subjects covered in the talk.

The talk concept started out as “advanced analytics” (as opposed to fast query, a subject amply covered in the rest of any Netezza event), as a lunch break in what is otherwise a detailed “best practices” session. So I suggested we constrain the subject by focusing on a specific application area — customer acquisition and retention, something of importance to almost any enterprise, and which exploits most areas of analytic technology. Then I actually prepared the slides — and guess what? The mix of subjects will be skewed somewhat more toward generalities than I first intended, specifically in the areas of investigative analytics and derived data. And, as always when I speak, I’ll try to raise consciousness about the issues of liberty and privacy, our options as a society for addressing them, and the crucial role we play as an industry in helping policymakers deal with these technologically-intense subjects.

Slide 3 refers back to a post I made last December, saying there are six useful things you can do with analytic technology:

• Operational BI/Analytically-infused operational apps: You can make an immediate decision.
• Planning and budgeting: You can plan in support of future decisions.
• Investigative analytics (multiple disciplines): You can research, investigate, and analyze in support of future decisions.
• Business intelligence: You can monitor what’s going on, to see when it necessary to decide, plan, or investigate.
• More BI: You can communicate, to help other people and organizations do these same things.
• DBMS, ETL, and other “platform” technologies: You can provide support, in technology or data gathering, for one of the other functions.

Slide 4 observes that investigative analytics:

• Is the most rapidly advancing of the six areas …
• … because it most directly exploits performance & scalability.

Slide 5 gives my simplest overview of investigative analytics technology to date: 

• Fast query
  • Persistent storage (any data volume)
  • RAM (10s -100s of gigabytes, or more)
• Fast analytics
  • Predictive modeling
  • Transformation/tagging
  • Graph

Slide 6 points out that this is all supported by cheap data creation and acquisition, specifically in the area of machine-generated data, which gets the full benefit of Moore’s Law.

Slides 7-13 point out how the example problem domain involves lots of analytic tasks performed on lots of kinds of data. Specific examples cited include text analytics and graph/relationship analytics.

Slide 14 contains the punch line, so I’ll quote it in full:

Derived data

• You can’t keep re-analyzing all that in raw form …
• … so don’t.

If you have one takeaway from this session, let it be the utter importance of derived data.

Slide 16 lists kinds of derived data that are important in the single application of reducing telco churn:

• Normalized data
  • Parsed/sessionized logs
  • Text/sentiment highlights
  • Social network graph(s)
  • Web de-anonymization
  • Household matching
• Scores and buckets
  • Demographic
  • Psychographic
  • Offer hot buttons
  • (Dis)satisfaction
  • Credit/fraud risk
  • Lifetime customer value
  • Influence on others!

And finally, Slide 17 is my first pass at best practices for dealing with derived data:

• Evolving data warehouse schema
• Data marts
  • Physical or virtual
  • Inputs/outputs to “EDW”
• “Data science”
  • Research != production
• Multiple processing pipelines
  • Log parsing
  • Text
  • Predictive analytics
  • Generic ETL
  • Streaming “ETL”

That last list looks like a starting point for a whole set of interesting future posts.

So what other markets could MarkLogic pursue? Before Ken even started work, I sent over some thoughts. They included (but were not limited to): 

• Everybody now knows that not all problems require a relational DBMS. The NoSQL movement has seen to that.
• Not everybody agrees that “heavyweight” enterprise DBMS are needed for everything. The NoSQL movement has seen to that too.
• The “document”/”object” DBMS distinction has long been blurry. XML is full of documents, but they’re really objects. The same goes for the JSON/quasi-JSON objects of CouchDB/Couchbase and MongoDB. Object-oriented DBMS vendors have dabbled in XML on and off over the years because of technical similarity. Etc.
• MarkLogic has always focused on markets where the database truly was about documents in the conventional sense — especially long text documents — aka “content”. I always thought that focus was over-narrow.
• There are various cases where law, regulation, compliance etc. mandate the production of long text documents. I’m not sure MarkLogic has penetrated those as well as it could have.
• Graph DBMS technology is going nowhere fast, largely because nobody has solved the data distribution problem in cases big enough to need scale-out, and the technology isn’t obviously needed in single-server cases. (But see my post on Objectivity’s Infinite Graph.) Even so, graph-oriented apps are exploding, and MarkLogic should think about playing in the graph area, even if by acquisition.
• I think what I described in http://www.dbms2.com/2010/06/08/profile-of-revealed-preferences/ is non-relational and a very big market. Playing there with a “heavyweight” DBMS is of course a challenge.
• Coming over from Autodesk, Ken Bado hopefully knows more about the product data management business than I do.

It will be interesting to see what MarkLogic actually does.

I have two lines of argument supporting this view. One is application-oriented. Machine-generated data will keep growing rapidly. So using that data requires ever more processing resources as well. Analytic growth, rah-rah-rah; company valuation, sis-boom-bah. Application areas include but are not at all limited to marketing, law enforcement, investing, logistics, resource extraction, health care, and science.

The other approach is to point out some computational areas where vastly more analytic processing resources could be used than are available today. Consider, if you will, statistical modeling, graph analytics, optimization, and stochastic planning. 

Statistical practice, in many cases, still goes something like this:

• A data set has, for example, a thousand columns.
• Statisticians carefully choose a few dozen columns to model on.
• They then also decide how to modify data in some of the columns for better modeling (binning, filling in nulls, whatever).
• A linear regression ensues.

That all makes sense. Sometimes using fewer variables gives better results than using more of them (because of over-fitting), and you have to pick: You can’t realistically try all 2^1000-1 variable combinations; if you allowed quadratic terms too, you’d approach 2^500,000 combinations; and the possibilities expand from there. But suppose you actually did have unlimited computational resources? Then, if nothing else, you could do a whole lot of regressions, followed by some kind of meta-analysis on the results. That’s so beyond the realm of computational reality I doubt the mathematics of same has even been carefully worked out — which is exactly my point.

Graph analytics, to a first approximation, takes order(N*(E^H)) steps, where N is the number of nodes, H is the number of hops you want to go out, and E is the average number of edges per node. And that’s only for the simple stuff, which might produce inputs into further analytic steps. Such numbers get forbiddingly big, really fast.

Last year I talked with an LTL (Less than TruckLoad) shipping company. They had to decide which freight to put into which trucks, and then where to send the trucks. I thought for a moment, and said “In other words, the traveling salesman has to decide how to pack his knapsack?”* Optimization is computationally hard.

*Wikipedia has a wonderful list of NP-complete problems.

I was a stock analyst at the dawn of electronic spreadsheets. I actually did my first training spreadsheet using a calculator and green paper, and had one older colleague who still used a slide rule. The wonderful thing about making projections via electronic spreadsheets was that you could vary assumptions, then have the conclusions automatically recalculated. Bliss! (At least when compared to the alternatives.)

After a couple of months on the job, I circulated a memo about how one SHOULD do projections. I was told they almost fired me. And they had a point, because the computational power needed was ridiculous. The first part of the idea was to pull in every variable that seemed to make sense, and postulate (or test if possible) relationships among them. The second was to look at the outcomes under many different values of the independent variables.

In other words, I was advocating Monte Carlo stochastic planning. Well, guess what! Monte Carlo analysis is getting more widely productized, due to its usefulness in Basel 3 risk analysis. Traditional business planning still stinks. The time either has come or else is coming soon when traditional business planning should invoke Monte Carlo methods.

Bottom line: The analytic need for speed will remain with us through the foreseeable future — and I didn’t even need to do a probabilistic analysis to figure that out.

• The Teradata/Aster combination genuinely is complementary.
• Teradata focuses on and excels at multi-thousand-user, often very operational data warehousing use cases.*
  • Teradata’s products also do other analytic DBMS tasks, but in most cases aren’t particularly differentiated at them.
  • The use cases Teradata excels at probably can support higher pricing than generic fast-query use cases anyway.
• Aster is focused on analytic platform use cases.
  • Aster indeed introduced the first/best vision for analytic platforms.
  • Aster Data nCluster isn’t particularly differentiated in, say, general fast-query use cases.
  • Aster can command higher prices in analytic platform use cases than generic fast-query ones anyway.
  • Recently, Aster has been particularly focused on mapping influence graphs, and not just in the telecom industry where the idea of graph-based churn analysis started.**
  • Aster’s old frontline — aka operational — positioning seem like a distant memory.
• Teradata’s and Aster’s core code lines would be very hard to integrate, and hence will not be integrated for the foreseeable future. However:
  • Teradata and the Aster Data nCluster DBMS can and should each be enhanced to run exactly the same SQL.
  • Aster Data nCluster runs on generic commodity hardware, and that’s great. But the temptation to have Teradata’s hardware engineers design Aster-optimized hardware will eventually become irresistible.
  • The Xkoto-based replication Teradata is working on should be extended to Aster Data nCluster.
  • There surely will be knowledge transfer between the Teradata and Aster development teams in various areas. (Workload management? Security? Compression, as they both improve in that area?)

*At one point I noticed Teradata reposition the classical EDW as the place for “trusted” analytic data. That’s actually a pretty good way of putting it, sweetening (and watering down) the stifling-EDW-bureaucracy lemons, so as to make them into an appealing lemonade.

**At least one analyst firm has gotten all excited about that, and is positioning Aster pretty much as a social media play. One might call that a case of not seeing the Forrest for the trees — but from a graph-theoretic standpoint, that would be wrong …

Let me say some more about ensuring that the Teradata and Aster product lines run the same SQL. It would probably be a Small Matter of Programming to give Aster nCluster the Teradata temporal SQL extensions; to make Teradata run SQL/MapReduce, for some subset of the programming languages that Aster nCluster supports; and generally to make it so that most new things you’d want to develop on one platform would also run on the other. And since Teradata and Aster seem to be SAS’s two favorite DBMS partners — Teradata for market share and Aster for technology — SAS-support compatibility seems like a reasonable goal as well.

Benefits of such compatibility would include:

• Users could start developing on one platform, then acquire the other one only after they’d dipped their toes in the water.
• In particular, a small-but-noisy group of analysts could be given a virtual slice of a Teradata system to play on until their Aster cluster made it through the budget approval cycle.
• Disaster recovery, archiving, and so on could straddle product lines.

Performance might be very different between the two product lines, of course. And certain programming techniques might not carry over easily, such as user-defined functions (UDF), stored procedures, or some non-SQL analytic processes. Still, all focused and differentiated product positioning notwithstanding, it would be a Very Good Idea to converge the Teradata and Aster platform capabilities faster than is normal in similar merger situations.

Finally, perhaps I should also say more about Teradata’s and Aster’s relative weakness in generic fast-analytic-query use cases.

• There are many use cases for which columnar storage, columnar compression, or both give winning performance. Teradata has a very limited form of columnar compression, and no column storage. Aster has a fairly basic form of column storage, and no columnar compression.
• Teradata is architected more for random than sequential I/O. In row-based systems, sequential I/O is often better for “big” queries.
• While Teradata isn’t as expensive as it used to be — if you’re just running some analytic queries, it’s also not particularly cheap.
• Aster nCluster is just a younger product than many of its competitors. Perhaps there are some use cases for which Bottleneck Whack-A-Mole hasn’t yet gone far enough.

And yes, the rumors about Aster’s customer comScore not getting into production seem to be true, even though comScore talked at the same Aster-sponsored event I did in May, 2010.

Ultimately, there are six useful things you can do with analytic technology:

• You can make an immediate decision.
• You can plan in support of future decisions.
• You can research, investigate, and analyze in support of future decisions.
• You can monitor what’s going on, to see when it necessary to decide, plan, or investigate.
• You can communicate, to help other people and organizations do these same things.
• You can provide support, in technology or data gathering, for one of the other functions.

Technology vendors often cite similar taxonomies, claiming to have all the categories (as they conceive them) nicely represented, in slickly integrated fashion. They exaggerate. Most of these categories are in rapid flux, and the rest should be. Analytic technology still has a long way to go.

In more detail: 

• You can make an immediate decision.
  • The decision can be made by:
    • A machine, and then also executed by a machine, for example in:
      • Web site personalization.
      • Algorithmic trading.
      • Network security and/or load balancing.
    • A machine, and then executed by a human, for example in a call center.
    • A human looking at a machine.
      • This case can be a lot slower than the others.
      • Analytic-operational app integration can be a special case of this, but progress is slow — today’s reality resembles what I proposed in a 2004 white paper.
  • Technologies supporting immediate decision making include:
    • SQL, most commonly.
    • Extensions that are getting added into SQL DBMS, such as in-database model scoring.
    • Other search and query languages.
    • Complex event processing (CEP), whether rules-based or SQL-like.
    • Other rules engines, rarely.
• You can plan in support of future decisions.
  • Technologies that support planning include:
    • Microsoft Excel, first and foremost.
    • Budget-centric tools.
    • Forecasting.
  • Planning hasn’t advanced as well as one would have hoped.
    • The integration of planning and business intelligence has been uninspiring, a couple years of aggressive marketing last decade — especially by Cognos — notwithstanding.
    • Specialty planning languages always seem to disappoint. River Logic is a small vendor with a great idea — which it hasn’t advanced rapidly since the 1990s. This is sadly typical.
  • One bright spot, however, has been demand forecasting.
• You can research, investigate, and analyze in support of future decisions.
  • I’ve just started calling this area investigative analytics.
  • In doing so, I am conflating several disciplines:
    • Statistics, data mining, machine learning, and/or predictive analytics. (Note: I can’t get excited about the distinctions between those closely overlapping technology categories — apologies to Sam Madden and others who do seem to care.)
    • The more research-oriented aspects of business intelligence tools:
      • Ad-hoc query.
      • Drilldown.
      • Most things done by BI-using “business analysts.”
      • Most things within BI called “data exploration.”
    • Analogous technologies as applied to non-tabular data types such as text or graph.
  • There’s a lot further to go.
    • It’s still very early days for in-database analytic technology.
    • Any two of statistics, business intelligence, and text analytics could be much better integrated with each other than they are.
• You can monitor what’s going on, to see when it necessary to decide, plan, or investigate.
  • The guts of business intelligence — reports and dashboards — are really monitoring tools.
  • Monitoring is the jumping-off point for a lot of decision making, planning, and investigation. First you notice the anomaly or need, then you set out to do something about it.
  • I think this technology could use a lot of improvement.
• You can communicate, to help other people and organizations do these same things.
  • Since the dawn of reporting, reports have used as much to communicate among colleagues as they have to truly support personal decision-making.
  • BI vendors have done decent jobs in recent years of advancing the communication aspects of BI, in two respects:
    • General share-ability of reports and the like.
    • Stakeholder-facing BI.
  • But more profound BI-centric collaboration is advancing too slowly.
• You can provide support, in technology or data gathering, for one of the other functions.
  • Well, duh. That’s most of what I write about in this blog, especially in the areas of DBMS and ETL/ELT/ETLT.

• People
• Financial trades
• Electronic networks
• Everything else

The most varied, interesting, and valuable of those four categories is the first one.

That may change some day, with the growing importance of machine-generated data, and of big-data science in particular. But I think it’s a fair assessment at the present, and for at least the next few years.

Some of the most interesting use cases are concentrated in the areas of identifying individuals, groups of people, or behaviors of (groups of) people. For example:

• comScore works hard to identify individual web surfers – i.e. to deanonymize them — even though they may have given incomplete or false personal information.
• Other companies at least try to figure out which information in a user’s profile is unreliable, so as to classify them better. (Yes, there are 62-year-old video-game-obsessed Lady Gaga fans, but that’s generally not the way to bet.)
• Multiple telecom vendors try to identify who their most influential customers are (to a first approximation, they’re the ones most often called by the most people, but it surely gets more sophisticated than that). This information is then used to reduce churn, either by working hard to retain those users, or – if they do churn – to move very fast to retain the business from their friends.
• Other kinds of companies do similar kinds of analysis, to the extent that they have enough of a social graph to do so. (This application is a case where the term “social graph” is not a misnomer.)
• Turing detectives (I just coined that phrase) try to determine whether users are humans or bots.
• Central to detecting insurance fraud is identifying suspiciously close connections between claimants, service providers, and so on.
• Identifying groups of people is also important in flagging insider trading. Even more important are other kinds of analysis, along the lines of “is this normal innocent trading behavior?”
• Intelligence agencies try to detect networks of terrorists and their sympathizers. They further try to identify unusual patterns of communication or meetings along those networks that might indicate terrorist acts are being planned. (Civilian law enforcement agencies can use similar techniques.)

In most cases, the analysis and/or run-time execution of the relevant models is done with the help of analytic DBMS. Other technologies that come into play include non-DBMS MapReduce (Hadoop), graph engines, and CEP (Complex Event Processing). The vendor most heavily represented on that list is probably Aster Data, because:

• Aster Data is focused on hard-core analytics.
• I talk a lot with Aster Data, and in particular had a long, detailed use-cases discussion with them last week.
• The comScore example happens to come from a speaker at an Aster event I also participated in.

And by the way, all this only scratches the surface of what will be possible down the road. It’s based mainly on where you live, what you purchase, how you behave on websites, and who you communicate with. Other kinds of data, which could be used to be yet more intrusive, generally aren’t involved.

I actually have two points in drawing up this list. One is golly-gee-whiz about how a lot of analytically sophisticated applications are actually getting into production. The other is to highlight the privacy and liberty threats If This Goes On Unchecked (which is why I didn’t include some other less-people-focused examples). There’s also a related danger that, to the extent we don’t get some smart regulations to keep us safe(r), we’ll get a bunch of stupid regulations instead.

The Analytic Era has only just begun.

• Objectivity is a profitable, decades-old object-oriented DBMS vendor with about 50 employees.
• Like some other object-oriented DBMS of its generation, Objectivity is as much a toolkit for building DBMS as it is a real finished DBMS product. Objectivity sales are typically for custom deals, where Objectivity helps with the programming.
• The way Objectivity works is basically:
  • You manage objects in memory, in the format of your choice.
  • Objectivity bangs them to disk, across a network.
  • Objectivity manages the (distributed) pointers to the objects.
  • You can, if you choose, hard code exactly which objects are banged to which node.
  • Objectivity’s DML for reading data is very different from Objectivity’s DML for writing data. (I think the latter is more like the program code itself, while the former is more like regular DML.)
  • The point of Objectivity is not so much to have fast I/O. Rather, it is to minimize the CPU cost of getting the data that comes across the wire into useful form.
• Darren got the idea of putting a generic graph DBMS front-end on Objectivity while doing a relationship analytics project for an Australian intelligence agency.
• Darren redoubled his efforts to sell the project internally at Objectivity after reading what I wrote about relationship analytics back in 2006 or so.
• There is now a 5 or so person team developing Infinite Graph.
• Infinite Graph is just now going out to beta test.

Infinite Graph is an API or language binding on top of Objectivity that:

• Hides a lot of Objectivity’s complexity.
• Is suitable for graph/relationship analytics.

The main point of the Infinite Graph beta test is to see whether Objectivity got the API right. By way of contrast, Objectivity is still just researching the DBMS optimization side of things. According to Darren, what makes that so hard is that if you partition the graph in some smart way, probably through some kind of costly algorithm to determine “least connectedness,” a bit more additional data can thoroughly invalidate your results. Thus, Darren is focused more on ensuring that performance is good even if data is distributed around the network in annoying ways.

One performance win that Infinite Graph seems to get (almost?) for free from being built on top of Objectivity is lots of prefetching. Specifically, graph nodes and their edges are stored together, just like objects and their pointers are in traditional Objectivity — and if a node is retrieved, the nodes it’s connected to might also get retrieved as a background operation, before they’re even needed. More generally, Objectivity has always tried to be fast about traversing pointers, and that is a whole lot like traversing graph edges.

As a future, Infinite Graph is looking at ideas from Google’s Pregel. As Darren characterizes it, in Pregel you wrap up information about a graph node and ship it off to another computing node if the next graph node you need is over there. Darren suspects that the extreme form of this strategy would not be ideal. (I gather from Darren that Google has realized the same thing from the getgo.) Instead, he’s pinning his hopes more on smarts about when to do that (costly) shipping, and when to just fetch the information back to the compute node currently being used.

The most interesting part of our discussion, in my opinion, was about applications and application functionality. In a nutshell, Darren seems to think that it’s all about the edges, rather than the nodes themselves. (My words, not his.) In particular:

• Edges are first-class citizens in Infinite Graph, just as nodes are.
• Graphs typically are polluted with lots of insignificant edges. Examples include:
  • If you’re tracking people’s telephone traffic, lots of folks call the local pizza parlor. Indeed, it’s common to look for “star” nodes like that that have very high connectivity, and excise from the graph to reduce noise.
  • Many measures of relationship include minor relationships. Facebook friends? LinkedIn connections? Occasional phone calls? Next door neighbors? All of those can indicate very minor relationships.
• Therefore, in Infinite Graph, edges (can) have weights. Darren says this is a widely-used capability in graph applications. The core reason is to let you distinguish between significant and insignificant edges. Note that these weights can be calculated based on the raw data and stored back into the database.
• In Infinite Graph, edges can also have effectiveness date intervals. E.g., if you live at an address for a certain period, that’s when the edge connecting you to it is valid.
• In general in Infinite Graph, edges can carry arbitrary or at least flexible “qualifier”/attribute information.
• For many applications, the number of possible nodes is fundamentally limited. There are only so many people in the world, so many street addresses, so many telephone numbers, and so on. (There was a time this wasn’t believed to be the case, because timestamping was done at the node rather than edge level. But I find persuasive Darren’s argument that it works better on edges.) Edit: Even so, DARPA is thinking in the billions-of-nodes range.
  
• Darren is in general agreement with my observation that the “social graph” shouldn’t primarily be regarded as a graph.
• Yes, the paradigmatic examples of intelligence agency graph analytics are telephone or even IP traffic analysis. Nodes can wind up with lots of edges connecting them. Full analysis of the graphs exceeds even the computing capacity available to governments.
• On a happy civil liberties note, Darren observed that Australian intelligence has a lot of red tape restricting them from getting this kind information. Basically, they can only get chunks of information “on demand”. An awkward side effect of this is that when they do get it, it could be in any number of formats.

]]> http://www.dbms2.com/2010/06/19/objectivity-infinite-graph/feed/ 7 http://www.dbms2.com/2010/06/08/profile-of-revealed-preferences/ http://www.dbms2.com/2010/06/08/profile-of-revealed-preferences/#comments Tue, 08 Jun 2010 05:18:36 +0000 Curt Monash http://www.dbms2.com/?p=2235 “Social graph” is a highly misleading term, and so is “social network analysis.” By this I mean:

There’s something akin to “social graphs” and “social network analysis” that is more or less worthy of all the current hype – but graphs and network analysis are only a minor part of the whole story.

In particular, the most important parts of the Facebook “social graph” are neither social nor a graph. Rather, what’s really important is an aggregate Profile of Revealed Preferences, of which person-to-person connections or other things best modeled by a graph play only a small part.

Let me hasten to note that – even when viewed narrowly — the ideas of “social graph”and “social network analysis” do have significance. Nontrivial use cases to date for big data social network analysis include:

• Intelligence agencies identify and analyze terrorist networks. Corporations and civilian law enforcement do the same for fraud networks.
• Telephone companies use calling data to figure out which of their customers are most likely to influence which other customers in the decision to keep or change service providers. (Frankly, I find that rather creepy.)
• Social networks figure out which other members you’re likely to know, and encourage you to connect with them.

Epidemiologists aspire to add to that list, based on their success to date using much more micro forms of social network analysis. But after that, I’m running out of examples. Sure, graph analytics is good for a bunch of other things (e.g., biology at the genetic or molecular level), but those have little or nothing to do with “social graphs” or social network analysis as they are commonly understood.

Note: Of course, it is also the case that everything can be modeled by entity-attribute-value triples, and those can always be modeled by graphs. But so what?

Let’s consider what, in a marketer’s ideal world, would go into your Profile of Revealed Preferences. Raw data might include:

• Personally identifyING information. Duh. This is what makes everything else possible.
• Purchase transaction data. Lots of it. Like, everything on your credit card statements.
• Demographic and lifestyle information. Address, date of birth, educational history, race, household composition, and so on.
• Affiliations. Politics, religion, group membership of any kind. (OK, that’s partly social.)
• Explicitly stated opinions, preferences and desires, including:
  • Any surveys you have filled out.
  • Any recommendations you have made (e.g., through the Facebook Like feature).
  • The text of anything you’ve written and posted – and, very ideally, of your private emails as well.
  • Any wish lists you’ve filled in.
• Attention information. What you clicked on, what you looked at, and all that stuff website owners track.
• Your movements, to the extent they are tracked. (E.g., via Foursquare and the like.)
• Your gaming activities and the like. (This is social mainly to the extent it overlaps with other categories I’ve already mentioned.)
• Your medical information.
• Who you communicate with, and what you communicate with them about. (Hey! There’s something else social!)
• Similar information about the people you communicate with.

My core privacy thoughts about that data include:

• Individuals deserve the right to control all that information. At a minimum, they deserve total control over how the data (raw or derived) is passed from the service – e.g., website – where it naturally resides (e.g., where it is originated) to any other place.
• Given a chance, individuals would make fine-grained choices about what parts of their Profile of Revealed Preferences are available to which organizations. Reasons include:
  • Individuals have rather complex trust relationships with different kinds of merchants and marketers.
  • Consumers get different benefits from sharing information with different kinds of merchants and marketers. (Sometimes personalization is a large benefit. Sometimes it’s just creepy. And some companies actively bribe you to give them information they can use to sell to you.)

When one frame things this way, two rather difficult technological questions naturally arise.

1. Suppose, implausibly, that a single entity were allowed to control and use (for marketing) all of your Profile of Revealed Preferences information. How would they store and analyze it?
2. How does the answer to #1 change because control over the information will, in fact, be fragmented?

It’s tough enough to answer these questions for data about one person. Trying to include all but the simplest information about other people is and will for years remain quite infeasible. So, for the most part, this is not “social” information.

It’s also not naturally a “graph.” Similarly, it is not a good candidate for network analysis. To see why, let me outline why I used the name “Profile of Revealed Preferences”:

• The reason marketers want this data is, mainly, because they want to know what appeals to you, and how strongly you feel about it.
• The analytic process often entails taking explicit choices you have made, and inferring other preferences from them.
• The output of the analytic process is often one or more “scores” that then get plugged into various selection algorithms to determine what you should be shown or offered. At least implicitly, these algorithms are predicting what you will or won’t respond well to.

Not much graph-like there.

This post has gotten pretty long, so I’ll stop here without spelling anything else out. But questions I still hope to address down the road include:

• How should Profile of Revealed Preferences data be stored?
• Suppose we want to pass around derived results and not the raw data. How could we ever get to standards that would make such interchange realistic?
• If we only have raw data to pass around, what are the implications for privacy, liberty, and the structure of the online industries?