Mike Stonebraker on “real column stores”
Mike Stonebraker has a post up on Vertica’s blog trying to differentiate “real” from “pretend” column stores. (Edit: That post seems to have come back down, but as of 1/19 it can be found in Google Cache.) In essence, Mike argues that the One Right Way to design a column store is Vertica’s, a position that Daniel Abadi used to share but since has retreated from.
There are some good things about that post, and some not-so-good. The worst paragraph is probably
Several row-store vendors (including Oracle, Greenplum and Aster Data) now claim to be selling a column store. Obviously, this would require a complete rewrite of a DBMS to move from Figure 1 to Figure 2. Hence, none of the “pretenders” have actually done this. Instead all have implemented some aspects of column stores, and then claim to be the real thing. This blog defines what the “real enchilada” looks like, and how to tell it from the pretenders.
which I question on two levels. Read more
| Categories: Aster Data, Columnar database management, Database compression, Michael Stonebraker, Sybase, Theory and architecture, Vertica Systems | 24 Comments |
The technology of privacy threats
This post is the second of a series. The first one was an overview of privacy dangers, replete with specific examples of kinds of data that are stored for good reasons, but can also be repurposed for more questionable uses. More on this subject may be found in my August, 2010 post Big Data is Watching You!
There are two technology trends driving electronic privacy threats. Taken together, these trends raise scenarios such as the following:
- Your web surfing behavior indicates you’re a sports car buff, and you further like to look at pictures of scantily-clad young women. A number of your Facebook friends are single women. As a result, you’re deemed a risk to have a mid-life crisis and divorce your wife, thus increasing the interest rate you have to pay when refinancing your house.
- Your cell phone GPS indicates that you drive everywhere, instead of walking. There is no evidence of you pursuing fitness activities, but forum posting activity suggests you’re highly interested in several TV series. Your credit card bills show that your taste in restaurant food tends to the fatty. Your online photos make you look fairly obese, and a couple have ashtrays in them. As a result, you’re judged a high risk of heart attack, and your medical insurance rates are jacked up accordingly.
- You did actually have that mid-life crisis and get divorced. At the child-custody hearing, your ex-spouse’s lawyer quotes a study showing that football-loving upper income Republicans are 27% more likely to beat their children than yoga-class-attending moderate Democrats, and the probability goes up another 8% if they ever bought a jersey featuring a defensive lineman. What’s more, several of the more influential people in your network of friends also fit angry-male patterns, taking the probability of abuse up another 13%. Because of the sound statistics behind such analyses, the judge listens.
Not all these stories are quite possible today, but they aren’t far off either.
| Categories: Facebook, Predictive modeling and advanced analytics, Surveillance and privacy, Telecommunications, Web analytics | 4 Comments |
Privacy dangers — an overview
This post is the first of a series. The second one delves into the technology behind the most serious electronic privacy threats.
The privacy discussion has gotten more active, and more complicated as well. A year ago, I still struggled to get people to pay attention to privacy concerns at all, at least in the United States, with my first public breakthrough coming at the end of January. But much has changed since then.
On the commercial side, Facebook modified its privacy policies, garnering great press attention and an intense user backlash, leading to a quick partial retreat. The Wall Street Journal then launched a long series of articles — 13 so far — recounting multiple kinds of privacy threats. Other media joined in, from Forbes to CNet. Various forms of US government rule-making to inhibit advertising-related tracking have been proposed as an apparent result.
In the US, the government had a lively year as well. The Transportation Security Administration (TSA) rolled out what have been dubbed “porn scanners,” and backed them up with “enhanced patdowns.” For somebody who is, for example, female, young, a sex abuse survivor, and/or a follower of certain religions, those can be highly unpleasant, if not traumatic. Meanwhile, the Wikileaks/Cablegate events have spawned a government reaction whose scope is only beginning to be seen. A couple of “highlights” so far are some very nasty laptop seizures, and the recent demand for information on over 600,000 Twitter accounts. (Christopher Soghoian provided a detailed, nuanced legal analysis of same.)
At this point, it’s fair to say there are at least six different kinds of legitimate privacy fear. Read more
| Categories: Analytic technologies, Facebook, GIS and geospatial, Health care, Surveillance and privacy, Telecommunications, Web analytics | 6 Comments |
The six useful things you can do with analytic technology
I seem to be in the mode of sharing some of my frameworks for thinking about analytic technology. Here’s another one.
Ultimately, there are six useful things you can do with analytic technology:
- You can make an immediate decision.
- You can plan in support of future decisions.
- You can research, investigate, and analyze in support of future decisions.
- You can monitor what’s going on, to see when it necessary to decide, plan, or investigate.
- You can communicate, to help other people and organizations do these same things.
- You can provide support, in technology or data gathering, for one of the other functions.
Technology vendors often cite similar taxonomies, claiming to have all the categories (as they conceive them) nicely represented, in slickly integrated fashion. They exaggerate. Most of these categories are in rapid flux, and the rest should be. Analytic technology still has a long way to go.
In more detail: Read more
| Categories: Analytic technologies, Business intelligence, Cognos, Data warehousing, RDF and graphs, Text | 13 Comments |
Examples and definition of machine-generated data
In posts made last December, January, and April, I argued:
- Much of the growth in analytic data volumes will come in the form of machine-generated data.
- Unlike human-generated data, machine-generated data will grow at Moore’s Law kinds of speeds.
- Thus, unlike human-generated data, which I advocate keeping pretty much in all its detail, machine-generated data will continue to be in large part thrown away.
Recently and somewhat belatedly, I added a somewhat obvious point — if we don’t keep all or even most of our machine-generated data, then what we keep is likely to be in some way massaged, extracted, or derived. The purpose of this post is to address a second oversight — giving a hopefully clear definition of what I actually mean by “machine-generated data.” Read more
| Categories: Data warehousing | 28 Comments |
Evolving definitions and technology categories for 2011
It seems my prediction of a limited blogging schedule in December came emphatically true. I shall re-start with a collection of quick thoughts, clearing the decks for more detailed posts to follow. Read more
| Categories: Analytic technologies, Data types, Data warehousing, DBMS product categories, MOLAP, Theory and architecture | 6 Comments |
I’m partway back
As previously noted, I cut back temporarily on blogging (and taking briefings) a couple of months ago as my parents got sicker, then suspended work altogether a month ago when they died. I am immensely grateful to be in a line of work where choices like that are possible. Once again, I thank you all for your tolerance and kindness.
Last Monday night, Linda and I returned from Columbus, leaving behind an apartment that was hardly packed up at all. We have to go back the week of 12/6; then I’m going to see clients in California the week of 12/13, as I do about once per quarter; then of course come the holidays; there also is estate-related stuff to take care of even while we’re here; and by the way, year-end is when over half of all Monash Advantage members renew. So I surely will be on a limited blogging schedule for most of December as well.
I did, however, get a few posts done this weekend, finishing up one on MarkLogic that had been in the hopper for a while, and adding two rather substantive spin-off posts from that one as well. After the New Year, I would hope to be back up to full speed.
| Categories: About this blog | 1 Comment |
Data that is derived, augmented, enhanced, adjusted, or cooked
On this food-oriented weekend, I could easily go on long metaphorical flights about the distinction between “raw” and “cooked” data. I’ll spare you that part — reluctantly, given my fondness for fresh fruit, sushi, and steak tartare — but there’s no escaping the importance of derived/augmented/enhanced/cooked/adjusted data for analytic data processing. The five areas I have in mind are, loosely speaking:
- Aggregates, when they are maintained, generally for reasons of performance or response time.
- Calculated scores, commonly based on data mining/predictive analytics.
- Text analytics.
- The kinds of ETL (Extract/Transform/Load) Hadoop and other forms of MapReduce are commonly used for.
- Adjusted data, especially in scientific contexts.
| Categories: Analytic technologies, Data warehousing, Derived data | 12 Comments |
Document-oriented DBMS without joins
When I talked with MarkLogic’s Ken Chestnut about MarkLogic 4.2, I was surprised to learn that MarkLogic really, truly doesn’t do anything like a join. Unlike some other non-SQL DBMS, MarkLogic has no SQL interface, no ODBC or JDBC. Nothing, nada. (MarkLogic has a Java interface for Xquery, but not for anything like SQL.)
| Categories: CouchDB, MarkLogic, NoSQL, Structured documents, Text, Theory and architecture | 8 Comments |
MarkLogic and its document DBMS
This post has been long in the writing for several reasons, the biggest being that I stopped working for almost a month due to family issues. Please forgive its particularly choppy writing style; having waited this long already, I now lack the patience to further clean it up.
MarkLogic:
- Is an ACID-compliant, document-oriented, non-SQL, XML-based scale-out DBMS vendor of non-trivial size and momentum.
- Still has the same technical approach I previously described.
- Recently posted an internally-written white paper with a lot of technical detail.
- Recently had a point release — MarkLogic 4.2 — a lot of which seems to be “Oh, you didn’t have that before?” kinds of stuff.
- Has given me permission to post most of the slides from same, the first few of which give a nice overview of the MarkLogic story.
- Claims 200+ each of customers and employees (that’s from a slide MarkLogic did ask me to remove from the deck).
- Is a client again.
- Not coincidentally, is interested in branching out past the vertical markets of media and government/intelligence, in particular to the financial services market.
- Has finally rationalized its company and product names so that both are now “MarkLogic.” 🙂
- Has finally grasped that if it is proud of its ACID-compliance it probably shouldn’t be trying to market itself as “NoSQL”. 🙂