Short-request and analytic processing
A few years ago, I suggested that database workloads could be divided into two kinds — transactional and analytic. The advent of non-transactional NoSQL has suggested that we need a replacement term for “transactional” or “OLTP”, but finding one has been a bit difficult. Numerous tries, including high-volume simple processing, online request processing, internet request processing, network request processing, short request processing, and rapid request processing have turned out to be imperfect, as per discussion at each of those links. But then, no category name is ever perfect anyway. I’ve finally settled on short request processing, largely because I think it does a good job of preserving the analytic-vs-bang-bang-not-analytic workload distinction.
The easy part of the distinction goes roughly like this:
- Anything transactional or “OLTP” is short-request.
- Anything “OLAP” is analytic.
- Updates of small amounts of data are probably short-request, be they transactional or not.
- Retrievals of one or a few records in the ordinary course of update-intensive processing are probably short-request.
- Queries that return or aggregate large amounts of data — even in intermediate result sets — are probably analytic.
- Queries that would take a long time to run on badly-chosen or -configured DBMS are probably analytic (even if they run nice and fast on your actual system).
- Analytic processes that go beyond querying or simple arithmetic are — you guessed it! — analytic.
- Anything expressed in MDX is probably analytic.
- Driving a dashboard is usually analytic.
Where the terminology gets more difficult is in a few areas of what one might call real-time or near-real-time analytics. My first takes are:
- If you’re updating a counter 1000s of times per second, that’s short-request, even if the counter feeds something that looks like a dashboard or report.
- Serving web pages, tracking clicks, and so on is generally short-request, except that …
- … personalizing web pages can in some cases reasonably be viewed as analytic, and hence right at the boundary between or intersection of analytic and short-request processing. The same goes for the rest of what Teradata calls active data warehousing or Aster Data used to call “frontline”.
- If complex event/stream processing feeds a dashboard for humans to look at, it’s just fast analytics. If it automatically drives transactions, then it’s also at the boundary between or intersection of analytic and short-request processing.
Indeed, one of my top trends to watch these days is the integration of short request and analytic processing. Several different approaches come to mind.
You can do everything in a single instance of a general-purpose DBMS such as Oracle, DB2, Microsoft SQL Server, Sybase ASE, or MySQL. For sufficiently small enterprises with sufficiently undemanding workloads, that’s the best approach. Tokutek evidently aspires to be an improved version of the same thing.
You can do it in an analytic DBMS that is sufficiently strong in user concurrency, update speed, and so on. This is the sweet spot of Teradata’s market. It’s also where SAP HANA is alleged to be going.
You can tie together DBMS optimized for short-request and analytic processing (or use something like Hadoop for the analytics, whether or not it should be considered as a DBMS). E.g., Membase (now Couchbase) has integration stories with Hadoop and Vertica, at a couple of clients each. I think this is a major untapped opportunity in the MySQL world, and have been raising that point with various companies for some time.
You can graft short request processing onto analytic system. That’s the point of HBase.
You can superpose analytics on a short request processing system. That’s the point of DataStax Brisk.
Sybase RAP, depending on how it is configured, can fit several of these models. The same could be said of Oracle (especially Exadata) or DB2.
Fun times.
Comments
33 Responses to “Short-request and analytic processing”
Leave a Reply
[…] seems to be offering an in-memory object-based/object-oriented/whatever short-request DBMS that also talks SQL. I haven’t been briefed at this point, and hence don’t have detail […]
[…] that sounds as if he was asking about scale-out relational DBMS in general, MySQL or otherwise, short-request or analytic, it turned out that he was asking just about short-request scale-out MySQL. My thoughts and […]
[…] never quite worked out that way. For most of the history of tabular DBMS, the best-performing short-request and analytic DBMS have been designed quite differently from each other.* Non-relational systems — from […]
[…] across-the-board competitive with analytic relational DBMS. (And the same goes for HBase vs. short-request alternatives.) But the real question is whether its features are good enough for a variety of […]
[…] you do short-request and MapReduce processing against the same HBase database, the MapReduce part is usually still done […]
[…] makes a short-request processing […]
[…] that are designed strictly for analytic uses. Examples include Sybase IQ, Vertica, Greenplum, Aster, Infobright, SAND, ParAccel, Exasol, […]
[…] batch processing truly massive structured and semi-structured data sets. Hadoop’s weakness is short-request processing but tools such as Storm and HBase are starting to address […]
[…] challenges, with considerable success. But the success is usually one silo at a time — a short-request application here, an analytic database there. When it comes to integration, unsolved problems […]
[…] (OnLine Transaction Processing)/short-request-oriented SQL DBMS that are newer than […]
[…] analytics at short-request speeds is an obvious data-management-related challenge, and not yet comprehensively […]
[…] Numerous vendors are blending SQL and JSON management in their short-request DBMS. It will take some more work for me to have a strong opinion about the merits/demerits of various […]
Thanks for the information. I have been working with the analysis and databases. Apply both OLAP and analysis software easy project.
[…] “real-time analytics”. I expected the short-request/analytic distinction to blur, but even so I’m astonished by the number of NoSQL and NewSQL vendors who’ve […]
and I on the contrary, it is not convenient to use OLAP
[…] availability and security.* This is more crucial for short-request applications than analytic ones, but even your analytic systems shouldn’t leak data or […]
[…] Systems is out. “Operational” seems to be Gartner’s term for what I call short-request, in each case the point being that OLTP (OnLine Transaction Processing) is dubious term when […]
[…] sounds like an alternate query execution capability for short-request queries, the big point of which is that it saves them from being broadcast across the whole cluster, hence […]
[…] it is a new kid — but the general class of questions keeps coming. And that’s just for short-request use cases; similar questions for analytic systems arise even more […]
[…] on the performance and/or allowed functionality of joins in scale-out short-request RDBMS, and the relative lack of complaints about […]
[…] Short-request vs. analytic. […]
[…] addressed will include analytic RDBMS (including data warehouse appliances), NoSQL/non-SQL short-request DBMS, MySQL, PostgreSQL, NewSQL and […]
[…] cases run the gamut from short-request to highly analytic; no graph DBMS is well-suited for all graph use […]
[…] There are numerous short-request RDBMS indexing strategies, with various advantages and drawbacks. But better indexing, as a general […]
[…] is where short-request kinds of data stores — NoSQL or otherwise — are often stressed, especially in the […]
[…] This all fits with my view that the Current Hot Subject is human real-time data freshness — for analytics, of course, since we’ve always had low latencies in short-request processing. […]
[…] “adding analytics” to something previously focused on short-request processing, it is common to underestimate the variety of things users will soon want to do. (One common reason […]
Great article and analytic process is essential for dbms.
Great article, really work out for me. I have referred this article to my friend also.
Database is essential for modern web development and it promotes the processing requests easily
SQL and other DBMS has significant influence in modern web applications and other web platforms
As always, I appreciate your time and thought.
Nice Article…Very interesting to read this article. I have learned some new information. Thanks for sharing. Visit Now – custom software application development