• Objectivity is a profitable, decades-old object-oriented DBMS vendor with about 50 employees.
• Like some other object-oriented DBMS of its generation, Objectivity is as much a toolkit for building DBMS as it is a real finished DBMS product. Objectivity sales are typically for custom deals, where Objectivity helps with the programming.
• The way Objectivity works is basically:
  • You manage objects in memory, in the format of your choice.
  • Objectivity bangs them to disk, across a network.
  • Objectivity manages the (distributed) pointers to the objects.
  • You can, if you choose, hard code exactly which objects are banged to which node.
  • Objectivity’s DML for reading data is very different from Objectivity’s DML for writing data. (I think the latter is more like the program code itself, while the former is more like regular DML.)
  • The point of Objectivity is not so much to have fast I/O. Rather, it is to minimize the CPU cost of getting the data that comes across the wire into useful form.
• Darren got the idea of putting a generic graph DBMS front-end on Objectivity while doing a relationship analytics project for an Australian intelligence agency.
• Darren redoubled his efforts to sell the project internally at Objectivity after reading what I wrote about relationship analytics back in 2006 or so.
• There is now a 5 or so person team developing Infinite Graph.
• Infinite Graph is just now going out to beta test.

Infinite Graph is an API or language binding on top of Objectivity that:

• Hides a lot of Objectivity’s complexity.
• Is suitable for graph/relationship analytics.

The main point of the Infinite Graph beta test is to see whether Objectivity got the API right. By way of contrast, Objectivity is still just researching the DBMS optimization side of things. According to Darren, what makes that so hard is that if you partition the graph in some smart way, probably through some kind of costly algorithm to determine “least connectedness,” a bit more additional data can thoroughly invalidate your results. Thus, Darren is focused more on ensuring that performance is good even if data is distributed around the network in annoying ways.

One performance win that Infinite Graph seems to get (almost?) for free from being built on top of Objectivity is lots of prefetching. Specifically, graph nodes and their edges are stored together, just like objects and their pointers are in traditional Objectivity — and if a node is retrieved, the nodes it’s connected to might also get retrieved as a background operation, before they’re even needed. More generally, Objectivity has always tried to be fast about traversing pointers, and that is a whole lot like traversing graph edges.

As a future, Infinite Graph is looking at ideas from Google’s Pregel. As Darren characterizes it, in Pregel you wrap up information about a graph node and ship it off to another computing node if the next graph node you need is over there. Darren suspects that the extreme form of this strategy would not be ideal. (I gather from Darren that Google has realized the same thing from the getgo.) Instead, he’s pinning his hopes more on smarts about when to do that (costly) shipping, and when to just fetch the information back to the compute node currently being used.

The most interesting part of our discussion, in my opinion, was about applications and application functionality. In a nutshell, Darren seems to think that it’s all about the edges, rather than the nodes themselves. (My words, not his.) In particular:

• Edges are first-class citizens in Infinite Graph, just as nodes are.
• Graphs typically are polluted with lots of insignificant edges. Examples include:
  • If you’re tracking people’s telephone traffic, lots of folks call the local pizza parlor. Indeed, it’s common to look for “star” nodes like that that have very high connectivity, and excise from the graph to reduce noise.
  • Many measures of relationship include minor relationships. Facebook friends? LinkedIn connections? Occasional phone calls? Next door neighbors? All of those can indicate very minor relationships.
• Therefore, in Infinite Graph, edges (can) have weights. Darren says this is a widely-used capability in graph applications. The core reason is to let you distinguish between significant and insignificant edges. Note that these weights can be calculated based on the raw data and stored back into the database.
• In Infinite Graph, edges can also have effectiveness date intervals. E.g., if you live at an address for a certain period, that’s when the edge connecting you to it is valid.
• In general in Infinite Graph, edges can carry arbitrary or at least flexible “qualifier”/attribute information.
• For many applications, the number of possible nodes is fundamentally limited. There are only so many people in the world, so many street addresses, so many telephone numbers, and so on. (There was a time this wasn’t believed to be the case, because timestamping was done at the node rather than edge level. But I find persuasive Darren’s argument that it works better on edges.) Edit: Even so, DARPA is thinking in the billions-of-nodes range.
  
• Darren is in general agreement with my observation that the “social graph” shouldn’t primarily be regarded as a graph.
• Yes, the paradigmatic examples of intelligence agency graph analytics are telephone or even IP traffic analysis. Nodes can wind up with lots of edges connecting them. Full analysis of the graphs exceeds even the computing capacity available to governments.
• On a happy civil liberties note, Darren observed that Australian intelligence has a lot of red tape restricting them from getting this kind information. Basically, they can only get chunks of information “on demand”. An awkward side effect of this is that when they do get it, it could be in any number of formats.

]]> http://www.dbms2.com/2010/06/19/objectivity-infinite-graph/feed/ 5 http://www.dbms2.com/2010/04/03/akiban-highlights/ http://www.dbms2.com/2010/04/03/akiban-highlights/#comments Sat, 03 Apr 2010 05:36:42 +0000 Curt Monash http://www.dbms2.com/?p=1809 Akiban responded quickly to my complaints about its communication style, and I chatted for a couple of hours with senior Akiban techies Ori Herrnstadt, Peter Beaman and Jack Orenstein. It’s still early days for Akiban product development, so some details haven’t been determined yet, and others I just haven’t yet pinned down. Still, I know a lot more than I did a day ago. Highlights of my talk with Akiban included:

• Akiban is basically in the business of making OLTP (OnLine Transaction Processing) DBMS.
• That said, Akiban does not necessarily aspire to offer the DBMS that has the best update efficiency or throughput. In particular, the Akiban DBMS stores every datum twice, even before replication (and indexes) are taken into account.
• Akiban wants to store everything as third normal form relational databases. I didn’t ask whether 3NF is a hard requirement, a Really Good Idea if you want Akiban to run fast (that’s the one I’d guess), or merely a general design assumption.
• Akiban characterizes its core differentiators/value proposition as being:
  • Scale-out
  • No need to pay the traditional cost of joins
• Thus, Akiban is telling something like a NoSQL story.
• However, Akiban offers SQL.
• Specifically, Akiban offers SQL through a MySQL front end. However, the choice of front-end could change (Drizzle?), and non-relational front-ends (object?)* could eventually also be offered.
• Akiban’s first target market is SaaS providers, specifically ones that have true multitenancy issues. More generally, Akiban is pursuing cloud/private cloud applications with lots of tables. (Ori talks of a few thousand tables as being a small number.) At least at first, Akiban is conceding the market for huge-volume, scale-out, no-expensive-join web databases to the NoSQL contenders.
• Akiban has been in prototyping/development of some kind for several years. However, Akiban got its first angel funding early last year and its first venture funding late in 2009, so development only ramped up recently.
• Ori tells a version of the rather common “Everything I need to know in life I learned in the Israeli Army, and now I’m commercializing it” story. However, I didn’t get the sense that Akiban is necessarily a direct extension of a specific Israeli military project.

* A lot of Boston-area DBMS developers have significant non-relational experience. E.g., Jack Orenstein was an Object Design founder, and Peter Beaman used to work for Intersystems, both object-oriented DBMS vendors.

Akiban technical highlights include:

• Somewhat confusingly, Akiban databases are divided into “groups” of tables. The point of Akiban groups is:
  • Many-to-many relationships exist only within Akiban groups, not among tables in different groups.
  • Tables within Akiban groups are kind of pre-joined; more precisely, data is organized physically in a way that anticipates joins.
  • Thus, most Akiban joins can be executed without the cost of traditional join algorithms.
• One copy of the data Akiban stores is, in effect, clustered by object. E.g., a customer and her orders are stored together, or a patient and the records of her doctor visits. That’s how Akiban anticipates most joins.
• The other copy of Akiban data is stored in columns (I’m not sure if this part is strictly columnar or more hybrid row/column), which are ordered consistently. In particular, they’re in an order dictated by the organization of the other copy of the data, whatever that means. Akiban’s goal is for this copy of the data to support reporting, operational BI, etc.
• Akiban relies heavily on its optimizer to determine data layout, probably more than conventional DBMS do.
• In essence, Akiban has a MySQL front-end and a storage engine back end, each running on its own hardware cluster, with each node of one cluster talking to each node of the other.
• I gather that Akiban distributes data among nodes clustered according to, in effect, object identifier. Presumably, inter-node joins are rare. But we didn’t discuss distribution, replication, or other scale-out issues in any detail. Indeed, I gathered that significant parts of all that weren’t built yet, and perhaps even not yet architected.
  

• MUMPS, the original Intersystems technology, was focused on health care.
• The reasons Intersystems went object-oriented have a lot to do with the structure of health-care records.
• Intersystems’ biggest and most visible ISVs are in the health-care area.
• Intersystems is actually beginning to sell an electronic health records system called TrakCare around the world (but not in the US, where it has lots of large competitive VARs).

Note: Intersystems Cache’ is sold mainly through VARs (Value-Added Resellers), aka ISVs/OEMs. I.e., it’s sold by people who write applications on top of it.

So far as I understand – and this is still pretty vague and apt to be partially erroneous – the Intersystems Cache’ technical story goes something like this:

• Intersystems Cache’ is an object-oriented DBMS.
• The preferred language for talking to Intersystems Cache’ is Java.
• Intersystems claims Cache’ has good SQL performance, for most kinds of use-case.
• Intersystems Cache’ stores data in a kind of sparse hierarchy. It uses a lot of “common character count” compression, which sounds a lot to me like Patricia tries.
• Intersystems has recently bundled some BI/reporting tools into the Cache’ stack. Surely not coincidentally, Intersystems once told me that some of its ISVs paid more to Crystal Reports than to Intersystems.
• Intersystems Cache’ has had Sybase emulation for several years, and just added Informix emulation. Most but not all stored procedures from those other DBMS run against Cache’ as well.
• Intersystems Cache’ recently added a bunch of manageability, security, etc. features, the details of which generally inspired “Oh, you didn’t have that earlier?” reactions in me.
• Intersystems’ just did a revamp of the Cache’ object model to make it more Smalltalk-like, in which messages are set to parent rather than child classes when appropriate. Thus, when you recompile a class, you don’t also have to recompile all its children, and incremental recompilation is now near-instantaneous. (Put that one in the “Oh, you didn’t have that earlier?” category too.) Versioning will be better as well.
• In the latest release, Cache’ has added what Intersystems calls “Java Event Processing.” This doesn’t sound like CEP (Complex Event Processing), and I forgot to ask whether it was memory-centric at all. Anyhow, the idea is to bang objects into the database really quickly, having them be immediately available for SQL query. “Really quickly” means >10,000 objects/core/second, with one test at the European Space Agency getting up to 85,000. By way of contrast, Intersystems asserts (based on bake-offs) that RDBMS competitors have to insert into BLOBs to get competitive performance, with associated loss of queryability.

Finally, a few financial highlights:

• Intersystems did a little over $1/4 billion in revenue in 2009.
• 85% of that was Cache’.
• Revenue growth was slightly positive in 2009, and 15% in 2008.
• Headcount growth was 25% in 2009 and is planned to be big again in 2010, after being modest in prior years.

]]> http://www.dbms2.com/2010/01/15/intersystems-cache-highlights/feed/ 3 http://www.dbms2.com/2008/06/08/perst/ http://www.dbms2.com/2008/06/08/perst/#comments Sun, 08 Jun 2008 21:07:37 +0000 Curt Monash http://www.dbms2.com/?p=438 Dan Weinreb — inspired by but not linking to my recent short post on McObject’s object-oriented in-memory DBMS Perst — has posted a detailed discussion of Perst on his own blog. For context, he compares it briefly to analogous products, most especially Progress’s — which used to be ObjectStore, of which Dan was the chief architect.

This was based on documentation and general sleuthing (Dan figured out who McObject got Perst from), rather than hands-on experience, so performance figures and the like aren’t validated. Still, if you’re interested in such technology, it’s a fascinating post.

First, the McObject guys — who also sell a relational in-memory product — have an object-oriented, apparently Java-centric product called Perst. They’ve sent over various press releases about same, the details of which didn’t make much of an impression on me. (Upon review, I see that one of the main improvements they cite in Perst 3.0 is that they added 38 pages of documentation.)

Second, I just got email about something called CSQL Cache. You can read more about CSQL Cache here, if you’re willing to navigate some fractured English. CSQL’s SourceForge page is here. My impression is that CSQL Cache is an in-memory DBMS focused on, you guessed it, caching. It definitely seems to talk SQL, but possibly its native data model is of some other kind (there are references both to “file-based” and “network”.)

That’s not a ridiculous interpretation, as he explains, but it’s kind of a glass-half-full one. By Dan’s own analysis, ObjectStore did OK but not great in the CAD/CAM/CAE and CASE markets for a while, but eventually fell back due to:

1. Lack of critical mass.
2. Moore’s Law making performance advantages less important.
3. Fewer performance benefits in Java than in C++.

A later attempt to front-end RDBMS by object-caching also fell short, perhaps due to competition from object-relational mapping vendors such as Tangosol (now part of Oracle).

So far as I can tell, the other OODBMS pioneers — Versant, Servio, et al. — did even less well than ObjectStore. Versant did sell a few OLTP accounts as sort of an early Intersystems Cache’, but that’s ancient history.

I recommend the articles highly. Dan was one of the most articulate techies I knew way back in Symbolics’ heyday, as well as being a very straight shooter. He’s still going strong.

Please subscribe to our feed!

Technorati Tags: ObjectStore, Progress Software, object-oriented database, OODBMS

“Database management” usually deals with numeric or alphabetical data – i.e., the kind of stuff that goes nicely into tables. It commonly has a natural one-dimensional sort order, which is very useful for sort/merge joins, b-tree indexes, and the like. This kind of tabular data is what relational database management systems were invented for.

But ever more, there are important datatypes beyond character strings, numbers and dates. Leaving out generic BLOBs and CLOBs (Binary/Character Large OBjects), the big four surely are:

• Text. Text search is a huge business on the web, and a separate big business in enterprises. And text doesn’t fit well into the relational paradigm at all.
• Geospatial. Information about locations on the earth’s surface is essentially two-dimensional. Some geospatial apps use three dimensions.
• Object. There are two main reasons for using object datatypes. First, the data can have complex internal structures. Second, it can comprise a variety of simpler types. Object structures are well-suited for engineering and medical applications.
• XML. A great deal of XML is, at its heart, either relational/tabular data or text documents. Still, there are a variety of applications for which the most natural datatype truly is XML.

Numerous other datatypes are important as well, with the top runners-up probably being images, sound, video, time series (even though they’re numeric, they benefit from special handling).

Four major ways have evolved to manage data of non-tabular datatype, either on their own or within an essentially relational data management environment.

• Utterly standalone servers. There are lots of search engines, geospatial engines, object-oriented database management systems, and so on. Some may have ODBC/JDBC SQL interfaces, to handle metadata (which is commonly tabular in nature) if nothing else. But even so, there’s little relational about them.
• True RDBMS extensibility. In the 1990s, awkwardly named object-relational database management systems were introduced, boasting the awkwardly named feature abstract datatypes. Oracle, DB2, Informix, and PostgreSQL are now of this kind. They let one write data access methods for data that’s right in the basic relational table structure, and get at it through extensions to SQL.
• Tightly coupled servers. A close relative of RDBMS extension via new access methods is to create new servers for new datatypes, well-integrated with your RDBMS. Your parser and optimizer are in charge of federating them; the user just writes extended-SQL statements.
• User-defined functions. User-defined functions are like datatype extensions, but vastly easier to write, in that they don’t have any special access methods. When their performance is good enough, UDFs are often the best way to handle extended-datatype needs.

So how does this all play out in real-world examples? It’s all over the place.

• Enterprise text search is divided among three modes – integrated into the RDBMS (Oracle and IBM), tightly-couple server (Microsoft, pre-FAST acquisition), and standalone (Autonomy, FAST pre-acquisition, Google, and most other vendors).
• Geospatial datatypes are embedded into extensible DBMS – generally via technology from ESRI – for OLTP uses. But for data warehousing, where you don’t need pinpoint record retrieval, UDFs are generally believed to suffice. (E.g. Teradata, Netezza.)
• Intersystems seems to stand alone in getting nontrivial revenue from the standalone OODBMS market.
• The XML situation is really confused: Oracle has been late getting its native XML strategy together; the tightly-coupled DB2 Viper engine has been a performance disappointment; Microsoft’s integrated native XML isn’t heard from much either; and text/XML integrated engine Marklogic is getting some non-text business almost by default. In addition, every serious relational vendor has a capability to “shred” XML into relational tables, and can of course also just bulk-handle XML via BLOBs/CLOBs.

And it’s not just me. Two personal data points about Intersystems’ self-defeating lack of visibility include:

• A nontrivial ISV is evaluating OODBMS for a nontrivial OLTP app. Cache’ — the obvious #1 choice — isn’t even being considered, because they know so little about the product. I suggested Cache’, but they shot back a qualifying question I couldn’t answer, because I’m so out of touch with the company and product myself.
• A client of mine that’s a medical-app ISV was indeed evaluating Cache’, as they should be — but was hugely impressed that I’d heard of such an obscure product.

I probably should stop talking about the company. While their story sounds very good in parts, I don’t have enough confidence in what I’ve heard to feel comfortable proposing them anywhere outside their flagship medical vertical market.

After a reorganization at the beginning of this (November) fiscal year, the vast majority of Progress’ products fall into one of five buckets, which I shall glibly refer to in decreasing order of size as “Progress Classic,” “SOA,” “drivers,” “memory-centric,” and “EasyAsk.” Here’s a quick overview of each.

Progress Classic is focused on the old DBMS/4GL combo, with the principal brand name being “OpenEdge.” This business is 2/3 VARs, 1/3 enterprises, typically enterprises that were first introduced to the technology via VARs. The 4GL is just historically a great product, if a little outmoded in one or the other way.

The DBMS is noteworthy in that it’s extremely easy to administer. In addition, Jeff claims that even for OLTP there’s commonly a 7:1 ratio in database size vs. Oracle for the same data, due to, for example, variable record lengths. Such factors are not unheard of in data warehousing, of course; but for OLTP his comment was quite the jaw-dropper. On the other hand, OLTP databases aren’t all that big (Progress’s biggest is 2 terabytes), so it’s not clear how great the practical significance of such statistics is.

Incidentally, the ability to scale to that size is one recent enhancement. Another is Java-friendliness; there’s now a JVM, and Progress supports Java stored procedures. Come to think of it, the latter is also mainly a performance/scalability feature in practice, although it might also provide comfort to those users not wanting to be locked into the Progress OpenEdge engine.

Overall growth in this division right now is sloooow. Big VARs continue to see some growth, but smaller ones dropping off aren’t necessarily being replaced.

SOA was put together from a lot of pieces – the inhouse development of Sonic’s stuff, the Actional acquisition, and a bunch of others. I’ll reserve comment until I know more, except to point out that Progress doesn’t seem to have yet connected the dots between its history of application development excellence and the need for really great and simple composite app development tools. Growth here seems solid.

Drivers = DataDirect, which has previously been an independent company, a part of the frequently renamed Sage/Intersolv/Merant, and before that originally an independent company named Pioneer and know for its Q+E product. Basically, DataDirect pioneered ODBC in collaboration with Microsoft, and is still the default supplier of ODBC/JDBC drivers. (Already in the 1990s these were just as performant as native database access, or so close as to make no difference, but to this day many people don’t realize that.) Now the product set has XML mixed in, via Stylus Studio, which is one of the few parts of Progress’s product line that actually seems to be vigorously marketed. Overall, the division seems to be growing nicely, something else that’s a relatively well-kept secret. But then, I’m a charter member of the club that believes DataDirect will maintain its market dominance for as long as John Goodson runs development there.

Memory-centric = what used to be known as Progress Real-Time Division, more or less. Only part of that is doing well. The object-caching and object DBMS products are being deemphasized and are showing small, declining revenue. This is despite some notable product success in the past. Brand names and predecessor companies covered by that include ObjectStore, Object Design, eXcelon (those are all the same thing, basically), and Persistence.

The Apama products, however, seem to be doing nicely in their specialty markets – financial trading, intelligence, telecom. A number of ultra-low-latency technologies seem to be doing well in those areas, as the need-for-speed seems not to be abating.

Note to self: See how Streambase is doing these days.

And then there is EasyAsk. Way back in 1983, I helped Artificial Intelligence Corporation raise scads of money on the strength of its natural language query product Intellect. For a variety of reasons, that never amounted to much, although it was the base for what became a public company called Trinzic. Anyhow, founder Larry Harris tried again, and the result product eventually became EasyAsk and was acquired by Progress. The whole thing is quite incestuous – Larry was on the Progress board, Jeff was an original investor in AI Corp. as well as being in effect a technical cofounder of Progress (despite moving on to other things), etc.

Be that as it may, Progress has some interesting thoughts about the possibilities for EasyAsk. I plan to write about them over on Text Technologies.

Want to continue getting great research about DBMS, analytics, data integration, and other technologies related to data management? Then get a free subscription to our feed, by RSS/Atom or e-mail! We recommend taking the integrated feed for all our blogs, but blog-specific ones are also easily available.

Intersystems also has a hot new Cache’-based integration product, Ensemble. They attempted to brief me on it (somewhat belatedly, truth be told) last Wednesday. Through no fault of the product, however, the briefing didn’t go so well. I still look forward to learning more about Ensemble.