I visited my then-clients at Endeca in January. We focused on underpinnings (and strategic counsel) more than on coolness in what the product actually does. But going over my notes I think there’s enough to write up now.
Before saying much else about Endeca, there’s one confusion to dispose of: What’s the relationship between Endeca’s efforts in e-commerce (helping shoppers navigate websites) and business intelligence (helping people navigate their own data)? As Endeca tells it:
- Endeca’s e-commerce and business intelligence efforts are reflections of the same technical approach. Indeed, I’m pretty sure Endeca’s product lines still share much/most of the same technology.
- Endeca went after e-commerce first because that’s where the provable ROI was. As I pointed out a couple of times in 2007, Endeca became a market leader in that area.
- Endeca increased its BI efforts later.
- Circa 2009-10, Endeca differentiated its e-commerce and BI product lines from each other.
- An e-commerce line extension called Page Builder is what really got Endeca through the recent recession.
- The BI product line Latitude was launched in the fall of 2010.
Endeca’s positioning in the business intelligence market boils down to “investigative analytics for people who aren’t hardcore analysts.” Endeca’s technological support for that stresses:
- Faceted search and navigation …
- … against diverse sources of data.
Here “diverse sources of data” can mean two things:
- Tabular data with all sorts of schemas.
- Text and so on.
That said, the Endeca paradigm is really to help you make your way through a structured database, where different portions of the database have different structures. Thus, at various points in your journey, it automagically provides you a list of choices as to where you could go next.
Underneath Endeca’s visible products is an engine called MDEX, about which Endeca says:
Inside the MDEX Engine there is no overarching schema; each data record carries its own metadata. This enables the rapid combination of a wide range of structured and unstructured content into Latitude’s unified data model. Once inside, the MDEX Engine derives common dimensions and metrics from the available metadata, instantly exposing each for high-performance refinement and analysis in the Discovery Framework. Have a new data source? Simply add it and the MDEX Engine will create new relationships where possible. Changes in source data schema? No problem, adjustments on the fly are easy.
While that is rather QlikView-like in its goals, the details are different. Most notably, Endeca MDEX features a disk-based columnar DBMS, whose highlights include:
- Endeca MDEX stores each column in two sort orders — by value and by a universal record ID. As I noted previously, this is a nice approximation to column-store idealism.
- Endeca MDEX has a range of columnar compression options — dictionary/token encoding (I’m pretty sure), run-length encoding, prefix compression, etc.
- Every Endeca MDEX column has a small tree-structured index cached in RAM.
On the business intelligence market penetration side, Endeca talked mainly about:
- Other manufactured equipment
- Public sector/intelligence and law enforcement
- Consumer & packaged goods
- Financial service
Specific applications mentioned included:
- Part-finding (I think by engineers)
- Sales/promotion analysis (down on the dealer level)
- Demand planning variance analysis
- Direct spend analysis
- Consultant staffing