September 7, 2014

An idealized log management and analysis system — from whom?

I’ve talked with many companies recently that believe they are:

At best, I think such competitive claims are overwrought. Still, it’s a genuinely important subject and opportunity, so let’s consider what a great log management and analysis system might look like.

Much of this discussion could apply to machine-generated data in general. But right now I think more players are doing product management with an explicit conception either of log management or event-series analytics, so for this post I’ll share that focus too.

A short answer might be “Splunk, but with more analytic functionality and more scalable performance, at lower cost, plus numerous coupons for free pizza.” A more constructive and bottoms-up approach might start with: 

Further, there would be numerous styles of business intelligence interface, at least including:

And there would be good support for quick-turnaround, easily-operationalized predictive analytics, of the sort that’s fairly central to the visions for Kiji and Spark.

The data management part of that is particularly hard, in that:

My thoughts on strengths and weaknesses of some obvious log data management contenders start:

In the interest of length, I’ll omit discussion of smaller vendors, except to say that Platfora’s integrated-stack event series analytics story deserves attention, and I’m disappointed that I never hear about Sumo Logic. And I don’t know a lot about companies positioned as SIEM (Security Information and Event Management), especially now that SenSage has left the scene.

Comments

10 Responses to “An idealized log management and analysis system — from whom?”

  1. Örjan on September 7th, 2014 9:03 am

    Also Tibco loglogic http://www.tibco.com/products/event-processing/loglogic-for-machine-data

    I think it is based on splunk though… (havent looked at it)

  2. Rajesh Nair on September 7th, 2014 9:39 am

    What do you think about the ELK stack?
    http://www.elasticsearch.org/overview/

    -Raj

  3. Joy-Paul Tharakan on September 15th, 2014 12:54 pm

    You may like to consider Nexthink from the list of smaller vendors. http://www.nexthink.com/

  4. The WibiWeekly: How To Save Petabytes in Hadoop, Why Customer Service is Dominating Retail & More | Data Wins on September 23rd, 2014 7:57 pm

    […] Industry analyst Curt Monash's overview and evaluation of log management & analysis […]

  5. The WibiWeekly: How To Save Petabytes in Hadoop, Why Customer Service is Dominating Retail & More | WibiData on September 24th, 2014 1:38 pm

    […] Industry analyst Curt Monash's overview and evaluation of log management & analysis […]

  6. Some stuff on my mind, September 28, 2014 | DBMS 2 : DataBase Management System Services on September 28th, 2014 8:21 pm

    […] The ability to mix traditional tabular data, JSON, and log data. […]

  7. Streaming for Hadoop | DBMS 2 : DataBase Management System Services on October 5th, 2014 4:57 am

    […] This also all fits with the importance I place on log analysis. […]

  8. Simone on October 7th, 2014 9:13 am

    SenSage is now HawkEye AP owned by Hexis Cyber Solutions, a KEYW company.

    HawkEye AP is perfectly positioned to lead the up and coming Security Analytics market. HawkEye AP continues to extend its core Log Management capability, further extending its lead as the world’s most efficient way to collect, store, and analyze mass quantities of Event Data.

    HawkEye AP is designed as a complete solution for security analytics with a large scale data warehouse, collection routines to bring in everything from your IT infrastructure, and a built-in reporting module. No single construct means HawkEye AP has virtually unlimited scalability.

  9. Luca Candela on May 19th, 2015 12:33 pm

    Your description describes very faithfully Treasure Data except for a couple bullet points that are not well developed yet.

  10. Rob Burton on October 30th, 2017 6:50 am

    There is one open source centralized log management software out there which provides scalable performance, it’s called NXLog: https://nxlog.co/products/nxlog-community-edition – it scales well event to thousands or ten thousands of servers while still providing high-performance. And it is a multi platform tool, so it can collect logs from Windwos, Linux, Android, etc. It definitely should be added to the list above.

Leave a Reply




Feed: DBMS (database management system), DW (data warehousing), BI (business intelligence), and analytics technology Subscribe to the Monash Research feed via RSS or email:

Login

Search our blogs and white papers

Monash Research blogs

User consulting

Building a short list? Refining your strategic plan? We can help.

Vendor advisory

We tell vendors what's happening -- and, more important, what they should do about it.

Monash Research highlights

Learn about white papers, webcasts, and blog highlights, by RSS or email.