Memory-centric data management whitepaper
I have finally finished and uploaded the long-awaited white paper on memory-centric data management.
This is the project for which I origially coined the term “memory-centric data management,” after realizing that the prevalent “in-memory DBMS” creates all sorts of confusion about how and whether data persists on disk. The white paper clarifies and updates points I have been making about memory-centric data management since last summer. Sponsors included:
- Applix, vendors of in-memory/memory-centric MOLAP tool TM1
- Progress Software, vendors of ObjectStore, an OODBMS that has more impressive references in-memory or otherwise memory-centric than it does in classical disk-based configurations, and also of the Apama stream processing products
- SAP, vendors of the BI Accelerator functionality of SAP NetWeaver, or whatever tortured name they want to give it this month — basically, that’s a very cool in-memory columnar data mart technology
- Solid Information Technology, vendor of hybrid in-memory/disk-based OLTP RDBMS. Historically focused on the embedded systems market, especially telecom and networking, they’ve recently been in the news because of a deal with MySQL that is designed to extend their reach.
- Intel, makers of the processors used to run a lot of the other sponsors’ products (including all BI Accelerator installations to date).
If there’s one area in my research I’m not 100% satisfied with, it may be the question of where the true hardware bottlenecks to memory-centric data management lie (it’s obvious that the bottleneck to disk-centric data management is random disk access). Is it processor interconnect (around 1 GB/sec)? Is it processor-to-cache connections (around 5 GB/sec)? My prior pronouncements, the main body of the white paper, and the Intel Q&A appendix to the white paper may actually have slightly different spins on these points.
And by the way — the current hard limit on RAM/board isn’t 2^64 bytes, but a “mere” 2^40. But don’t worry; it will be up to 2^48 long before anybody actually puts 256 gigabytes under the control of a single processor.
| Categories: Cognos, Companies and products, In-memory DBMS, Intel, Memory-centric data management, MOLAP, Open source, Progress, Apama, and DataDirect, SAP AG, solidDB | 2 Comments |
DBMS2 at IBM
I had a chat a couple of weeks ago with Bob Picciano, who runs servers (i.e., DBMS) for IBM. I came away feeling that, while they don’t use that name, they’re well down the DBMS2 path. By no means is this SAP’s level of commitment; after all, they have to cater to traditional technology strategies as well. But they definitely seem to be getting there.
Why do I say that? Well, in no particular order:
- They have a huge commitment to a data integration business, with an increasing XML focus.
- Their favorite buzzword these days is “information-intensive,” which seems to amount to semi-composite apps that may talk in part to unstructured/semi-structured data.
- They’re serious about their XML data server.
- Unprompted – well, OK, he’s clearly read my stuff, but other than that it was unprompted – Bob referred to one of the key benefits (real and perceived) of XML storage as being “schema flexibility.”
- By accident or design, IBM has a multi-server, horses-for-courses DBMS strategy: DB2 in two important flavors, XML server, Multivalue/Pick (that’s growing, by the way), and so on.
The big piece of a DBMS2 strategy that IBM seems to be lacking is a data-oriented services repository. IBM has had disasters in the past with over-grand repository plans, so they’re treading cautiously this time around. There also might be an organizational issue; DBMS and integration technology sit in separate divisions, and I doubt it’s yet appreciated throughout IBM how central data is to an SOA strategy.
But that not-so-minor detail aside, IBM definitely seems to be developing a DBMS2-like technology vision.
| Categories: EAI, EII, ETL, ELT, ETLT, IBM and DB2, OLTP, Structured documents | Leave a Comment |
Solid/MySQL fit and positioning
I felt like writing a lot about the great potential fit between MySQL and Solid over the weekend, but Solid didn’t want me to do so. Now, however, I’m not in the mood, so I’ll just say that in OLTP, Solid’s technology is strong where MySQL’s is weak, and vice-versa. E.g., Solid is so proud of its zero-administration capabilities that, without MySQL, it doesn’t have much in the way of admin tools at all. Conversely, I think that many of those websites that crash all the time with MySQL errors would crash less with the Solid engine underneath. (Solid happens to be proud of its BLOB-handling capability, efficiency-wise.)
Neither outfit is good in data warehousing, or in text search, image search, etc. (Solid slings big files around, but it doesn’t peer closely inside them). But for OLTP of tabular or dumb media data, this looks like a great fit.
Whether anybody will care, however, is a different matter.
Lisa Vaas of eWeek offers a survey of the many MySQL engine options.
EDIT: Another Lisa Vaas article makes it clear that MySQL is planning to compete in data warehousing/OLAP as well.
| Categories: Memory-centric data management, Mid-range, MySQL, OLTP, Open source, solidDB | 4 Comments |
More on Solid and MySQL?
In a stunningly self-defeating move, my friends at Solid have decided that anything about their already-leaked possible cooperation with MySQL is embargoed.
Indeed, they’ve emphasized to me multiple times that they do not wish me to write about it.
I shall honor their wishes. I hope they are pleased with the sophistication and insight of the coverage they receive from other sources.
| Categories: Memory-centric data management, Mid-range, MySQL, OLTP, Open source, solidDB | 3 Comments |
MySQL gets the Solid engine
Solid and MySQL have struck a deal (and for some odd reason I had to find out about it from Slashdot and then here rather than from one one the companies). Apparently Solid will open source a version of its storage engine, to be used with the MySQL front-end.
Solid’s core technology is a lightweight, zero-administration OLTP RDBMS. And they really mean “zero-administration,” because as they like to point out, a typical deployment is embedded in a piece of telecom equipment that doesn’t even have a keyboard. Now, that doesn’t really mean the Solid engine would still be zero-administration in other applications, but sure aren’t talking about something as prickly as, say, Oracle.
That said, Solid’s technology has its limitations. It isn’t historically designed for the query load (volume or mix) of, say, an SAP installation. It certainly doesn’t have much in the way of data warehousing functionality. And it doesn’t have much in the way of administration tools itself (although presumably MySQL will fill that gap).
One very important aspect of the Solid technology is its hybrid memory-centric design. Much more on that soon. My white paper on memory-centric data management is finally close to publication, with Solid as a co-sponsor. At some point I’ll even do a webinar for them associated with the paper.
I don’t know whether that’s part of the MySQL relationship — it would be very cool if it were.
| Categories: Memory-centric data management, Mid-range, MySQL, OLTP, Open source, solidDB | 2 Comments |
Marklogic’s experiences — from the warhorse’s mouth!
Another subject I meant to blog about is what all I’ve learned from Mark Logic about customer uses for XML.
Well, I have a great workaround for that one. Mark Logic CEO Dave Kellogg has revved up what I think is the most interesting vendor-exec blog I’ve seen. So if you’re interested in search/publishing-style uses for native XML, I strongly encourage you to go browse his blog. (And he writes about a lot of other interesting stuff as well.)
| Categories: MarkLogic, Structured documents | 1 Comment |
IBM’s definition of native XML
IBM’s recent press release on Viper says:
Viper is expected to be the only database product able to seamlessly manage both conventional relational data and pure XML data without requiring the XML data to be reformatted or placed into a large object within the database.
That, so far as I know, is true, at least among major products.
I’m willing to apply the “native” label to Microsoft’s implementation anyway, because conceptually there’s little or no necessary performance difference between their approach and IBM’s. (Dang. I thought I posted more details on that months ago. I need to remedy the lack soon.)
As for Oracle — well, right now Oracle has a bit of a competitive problem …
| Categories: IBM and DB2, Microsoft and SQL*Server, Oracle, Structured documents | 1 Comment |
SAP on SAP
Dan Farber’s blog from SAP’s developer conference isn’t, frankly, his best piece of work, since the quotes are sometimes so garbled as to be a bit unreadable. Still, it helps flesh out what we already knew about SAP’s strategy.
Basically, they claim to be reengineering their whole product line for the new services-based architcture I keep writing about. And they insist this truly is a new platform architecture. In that regard, I buy into and agree with their pitch.
They further insist that the mid-market will be a big part of their business going forward, but SaasS will not. I don’t buy into that as fully.
I’ll spell out why in another post, but not until Monday at the earliest. Watch the comments section on this one for trackbacks.
| Categories: SAP AG, Theory and architecture | Leave a Comment |
Oracle is getting touchy about XML
From Barbara Darrow’s “Unblog”:
“How we store XML on the database is, excuse me, none of your business. The point is you can write an app using XML standards,” said Mark Drake, manager of product management for XML technology for the Redwood Shores, Calif. vendor.
“Whether we shred it, parse it, it doesn’t matter. There is no such thing as a native XML storage model, there is no W3c standard or 11th stone tablet, telling us how,” he noted.
So implementation doesn’t matter? I.e., performance doesn’t matter?
That’s not generally Oracle’s viewpoint in areas where it has a performance or implementation advantage, or even parity …
| Categories: Oracle, Structured documents | 4 Comments |
DB2 Express-C — what IBM said
The following was received from IBM a few days before the DB2 Express Edition announcement. Due to an email glitch, it took a month for me to get permission to post it. Well, here it is, finally. The emphasis is all mine..
… IBM will announce DB2 Express-C, a free version of DB2 Express tailored for developer and partner communities. With the same core data server as DB2 Express Edition, DB2 Express-C will be free for download, development use, deployment, and redistribution. Support for DB2 Express-C will be available via a free online forum hosted and monitored by a community of DB2 experts. For traditional 24×7 IBM support, customers may acquire the appropriate license for DB2 Express Edition.
The following partners recognize the value DB2 Express-C adds to the solutions they provide their clients: ActiveGrid, AMD, Business Objects, Fourth Millenium Technologies, Intel, Mandriva, Mikropis, Nitix, Novell, Quest, Red Hat Linux, Retalon, Ubuntu Linux and Zend. Watch for news from many of these companies about distribution of DB2 Express-C with their offerings — for example Zend’s upcoming update to Zend Core for IBM.
DB2 Express-C Key Messages
Production-ready — DB2 Express-C provides development flexibility through support for a wide variety of software development environments and tools. DB2 Express-C also provides deployment and support flexibility by removing software license charges and offering free community support. In addition, DB2 Express-C places no database size limitations on developers.
DB2 Express-C also benefits from IBM investment in DB2 autonomics (self-management) technologies, performance optimization and resiliency. For example, its silent embedded installation and automatic object maintenance make DB2 Express-C well-suited for seamless integration into partner applications.
Developer community — Developers in a wide variety of development environments can draw on resources like developerWorks and alphaWorks to gain access to IBM support and emerging technologies from IBM research and development laboratories. Skills and applications developed with DB2 Express-C are directly applicable to all editions of DB2. At IBM, a DB2 Express-C community team has been formed to nurture DB2 community development and work with a variety of developer, ISV and open source community organizations.
Innovative technology — DB2 Express-C will be refreshed with the forthcoming “Viper” release of DB2 currently in beta test. DB2 “Viper” is the industry’s first hybrid data server -– deliveriing superior performance managing both XML and relational data structures. Early adopters have cited cost savings achieved through reduced development time and improved performance of applications using XML and relational data. These values provide critical advantages for the growing number of solutions implementing a service oriented architecture (SOA) and working with XML-based vertical industry standards.
DB2 Express-C Product Specifics
DB2 Express-C supports the Windows and Linux operating systems on various 32-bit and 64-bit processor architectures. Current Linux distributions supported include Novell Open Enterprise Server 9, Red Hat Enterprise Linux 3 and 4, SUSE Linux Enterprise Server 8 and 9, Asianux 1.0, Mandriva Corporate Server 3.0, Nitix 4.2.2a, Red Flag Advanced Server 4.1, and Ubuntu 5.04. Several Linux distributors will include DB2 Express-C in their Linux distributions. Details: http://ibm.com/db2/linux/validate
As with DB2 Express Edition, DB2 Express-C may be deployed on all systems with up to 2 processor cores, and on AMD or Intel x86 systems with up to 2 dual-core chips. 4 GB of memory is the maximum supported. There is no limit to database size however. Other editions of DB2 exist to support larger servers or clusters of servers with a seamless upgrade from DB2 Express-C.
No-charge community support for DB2 Express-C is provided via a public Web forum. For-fee support is available through purchase of a license for DB2 Express Edition, Workgroup Edition or Enterprise Server Edition.
Software development communities, environments and languages supported include PHP, Python, Perl, Rational Web Developer, .NET with Microsoft Visual Studio, Java with Eclipse, Quest Toad for DB2, Zend Studio and Zend Core for IBM.
Additional Information
For two examples of transaction processing performance with DB2 Express, visit:
On Linux: http://www.tpc.org/tpcc/results/tpcc_result_detail.asp?id=104071601
On Windows: http://www.tpc.org/tpcc/results/tpcc_result_detail.asp?id=104041401
NOTE: These benchmark results were published prior to the announcement of DB2 Express-C and were run using DB2 Express Edition. Price/performance metrics do not reflect the absence of support charges for DB2 Express-C.
For more on DB2 Express-C, including a download link, please visit http://ibm.com/db2/express . (This link will be activated at the time of our announcement.) More information on IBM Information Management Software can be found at http://ibm.com/software/data , and more information on DB2 can be found at http://ibm.com/db2 . DB2 Magazine is available at http://www.db2mag.com .
| Categories: IBM and DB2 | Leave a Comment |