I’ve been talking some with the Neo Technology/Neo4j guys, including Emil Eifrem (CEO/cofounder), Johan Svensson (CTO/cofounder), and Philip Rathle (Senior Director of Products). Basics include:
- Neo Technology came up with Neo4j, open sourced it, and is building a company around the open source core product in the usual way.
- Neo4j is a graph DBMS.
- Neo4j is unlike some other graph DBMS in that:
- Neo4j is designed for OLTP (OnLine Transaction Processing), or at least as a general-purpose DBMS, rather than being focused on investigative analytics.
- To every node or edge managed by Neo4j you can associate an arbitrary collection of (name,value) pairs — i.e., what might be called a document.
Numbers and historical facts include:
- > 50 paying Neo4j customers.
- Estimated 1000s of production Neo4j users of open source version.*
- Estimated 1/3 of paying customers and free users using Neo4j as a “system of record”.
- >30,000 downloads/month, in some sense of “download”.
- 35 people in 6 countries, vs. 25 last December.
- $13 million in VC, most of it last October.
- Started in 2000 as the underpinnings for a content management system.
- A version of the technology in production in 2003.
- Neo4j first open-sourced in 2007.
- Big-name customers including Cisco, Adobe, and Deutsche Telekom.
- Pricing of either $6,000 or $24,000 per JVM per year for two different commercial versions.
*I forgot to ask why the paying/production ratio was so low, but my guesses start with:
- Relatively high price.
- No true scale-out.
I also forgot to ask if there were any OEM users involved.
If we look at my basic list of graph data model application areas, Neo4j seems to be involved in most of what you would think. Exceptions include:
- There’s some anti-fraud and anti-terrorism, but it’s not as top-of-mind as for other graph database efforts.
- Taxonomies don’t seem to be in the picture at all, probably because people who want to model those would be more likely to turn to an RDF store.
- Hardcore influencer-finding analytics aren’t mentioned much, perhaps because Neo4j isn’t tuned for big batch analytic efforts.
- There are a lot of applications focusing on people’s roles, permissions, business hierarchies, and so on.
- Some of these are clearly in the area of access control.
- Others Neo debatably characterizes as MDM (Master Data Management), in that they serve as some kind of single-source-of-truth.
To scope what kind of databases Neo4j can or can’t handle, it may be helpful to note that:
- The scale-out version of Neo4j is some number of quarters away. At this time, Neo4j doesn’t scale out, except insofar as …
- … Neo4j has read slaves, which can be used for performance, high availability, and/or disaster recovery. (There’s some level of “OK, I’m the master now” HA functionality built in.)
- Neo4j has hard limits of 32 billion nodes, 32 billion edges, and 64 billion property (name,value) pairs.
- You are advised but not required to have everything except the property (name,value) pairs in memory.
- If that winds up leaving most of the overall data on persistent storage, you are advised but not required to use solid-state rather than spinning disk.
Neo4j is built around pointers and linked lists. The record for an edge, aka relationship, consists of:
- ID for the starting node. (Neo4j graphs are directed.)
- ID for the ending node.
- ID for the relationship type.
- Pointer to the first (name,value) pair in the property list.
- Pointer to the next edge leading from the start node.
- Pointer to the previous edge leading from the start node.
- Pointer to the next edge leading to the end node.
- Pointer to the previous edge leading to the end node.
(As you might imagine, any of those pointers could conceivably be null.) The property list is, you guessed it, another double linked list, in this case of (name,value) pairs. Similarly, I believe that a node record contains a node ID, a pointer to one edge leading from the node, a pointer to one edge leading to the node, and a pointer to a property list.
The physical data retrieval story in Neo4j starts:
- It’s mainly a big pointer chase.
- The pointers are meant to (almost) always be in memory.
- The cost of the pointer chase, to a first approximation, is a constant x E^(L-1), where
- E is the typical number of edges leading from each node
- L is the length of a path being examined
- The constant is sub-microsecond.
- In particular, the cost does not directly depend on the size of the graph, which leads Neo to somewhat confusingly say that overall query cost is “constant”
An exception to the “pointers are supposed to be in memory” rule occurs when property lists are kept on disk — but when you fetch the first property in the chain, all the rest are retrieved too, placing their pointers in memory from that time on.
Neo4j records are (almost) always fixed length, so they can be found just from offset calculations. Notes on that include:
- I get the impression that record lengths are changed from time to time, as Neo tweaks its optimization.
- Those records are short; the greatest length mentioned was 120 bytes, and figures came up as low as 20.
- If a record is constrained to be shorter than you need it to be, it turns into a pointer to some other location. I didn’t get all the details of that part.
Indexes do play a limited role, in determining at which node(s) to start the pointer chase. Notes on Neo4j indexing include:
- It seems to be Lucene-based.
- Both B-trees and text search were mentioned.
- To use an index, you need to explicitly refer to it in a query. Neo Technology is working on removing that requirement.
- Overall, indexes don’t seem to have been a big area of focus for Neo. Their heart is in pointers.
You can get at Neo4j via either its “Cypher” declarative query language or an older Java API.
Finally, there’s Neo4j’s durability story — it’s just full durability/ACID, with no tunable durability or anything like that. Neo4j doesn’t seem to feel any great rush about sending a write all the way to the database in persistent storage; but there’s also an update log, and the write doesn’t get acknowledged until that log has been flushed to disk.