July 26, 2011

Remote machine-generated data

I refer often to machine-generated data, which is commonly generated inexpensively and in log-like formats, and is often best aggregated in a big bit bucket before you try to do much analysis on it. The term has caught on, to the point that perhaps it’s time to distinguish more carefully among different kinds of machine-generated data. In particular, I think it may be useful to distinguish between:

Here’s what I’m thinking of for the second category. I rather frequently hear of cases in which data is generated by large numbers of remote machines, which occasionally send messages home. For example: 

Many technologies can be used to collect and manage remote machine-generated data, but a few common points are worth nothing.


2 Responses to “Remote machine-generated data”

  1. David Aldridge on July 26th, 2011 4:28 am

    “# If a device takes the trouble to send a message across a wide-area network, that message may be somewhat more valuable than the average piece of log-vomit. Perhaps such information doesn’t need to be stored in the cheapest possible way.

    “# Similarly, a message that is sent occasionally over time, or upon a specified event, may be more structured than a random log entry. Perhaps such data is suitable for sending straight to a relational database.

    WAN diagnostics being a good example, in which routers and other devices either proactively send fault detection messages (“memory buffer at 90%”) or where failure is detected by a management server.

    These are structured messages that can also require a lot of additional information to be added to them (eg. from an inventory system to match circuits to customers) in near real time in order to meet customer notification SLA’s. Fairly low volume, unless someone puts a digger shovel through a fibre optic trunk.

  2. Some trends that will continue in 2013 | DBMS 2 : DataBase Management System Services on December 12th, 2012 8:35 pm

    […] rate of Moore’s Law, also plus 0-25%, which is a much higher total. In particular, the use of remote machine-generated data is becoming increasingly […]

Leave a Reply

Feed: DBMS (database management system), DW (data warehousing), BI (business intelligence), and analytics technology Subscribe to the Monash Research feed via RSS or email:


Search our blogs and white papers

Monash Research blogs

User consulting

Building a short list? Refining your strategic plan? We can help.

Vendor advisory

We tell vendors what's happening -- and, more important, what they should do about it.

Monash Research highlights

Learn about white papers, webcasts, and blog highlights, by RSS or email.