To the consternation of its then-CEO, I wrote very little about my then-client Continuent. However, when I knew Schooner’s recent announcement was coming, I reached out to other MySQL scale-out vendors too. I’ve already posted accordingly about CodeFutures (the dbShards guys) and ScaleBase. Now it’s late-responding Continuent’s turn.
Actually, what I’m mainly going to do is quote a very long email that Continuent’s current CEO/former CTO Robert Hodges sent me, and which I lightly edited.
As you know we use replication to build clusters of database replicas for open source DBMS, primarily MySQL. We have two products:
Tungsten Replicator – Tungsten Replicator is the basic data movement part of Tungsten. We are going to release it as a separate product this year and push most features into open source to increase usage.
Tungsten Enterprise – Cluster management and connectivity based on master/slave replicas. Tungsten Enterprise is commercially licensed on a subscription basis.
I guess that means Continuent’s essentially failed legacy multi-master replication product has — unsurprisingly — fallen by the wayside.
Here are the major use cases we solve.
Note that operational ease is much higher on the list than performance. Note also that Robert naturally compares Continuent Tungsten to his worst competitor, rather than any of the other interesting start-ups’ products.
1. Better MySQL Replication. Tungsten Replicator significantly extends the built-in MySQL replication to handle the following sub-use cases related to master/slave replication. Tungsten is basically a configurable message-processing engine that does database replication.
1.1 Failover with multiple slaves. Tungsten has global transaction IDs, so if you have a failure you can promote the most advanced slave from a pool of slaves to master and point all others to it. MySQL replication by contrast tracks location using pointers into the master file system, so when the master disappears the slaves lose track of their position. This use case comes up both locally as well in building DR sites.
1.2 Replication between MySQL versions. Tungsten has minimal dependencies on MySQL versions and can replicate from versions 5.0/5.1/5.5 to versions 4.1/5.0/5.1/5.5. MySQL replication by contrast at best replicates up to the next version and either cannot replicate down or has limitations on replicating down version. This comes up commonly in migrations, for example going from MySQL 4.1 to 5.1.
1.3 Heterogeneous replication. Similarly, Tungsten can replicate out of MySQL into other databases such as PostgreSQL (we are currently implementing this for a large SaaS) as well as Oracle and drizzle (currently very basic and not regularly tested but available for polishing when engagements show up). This comes up commonly when users need to move reporting data out of MySQL into something more civilized for analytics. Greenplum is popular, for example; we are working on that now.
1.4 Multi-master replication. Tungsten can consolidate from multiple masters into a single slave. We also support topologies involving bi-directional replication with slaves on each master, including ability to promote slaves to take over from its master. MySQL replication cannot do this. These topologies are required for reporting as well as systems that have configuration or process merchant data across sites.
1.5 Parallel replication. Tungsten has shard-based parallel replication from the MySQL log, which is essential to get around slave lag problems. In addition, we can spread replication processing load flexibly across CPUs and even different hosts. MySQL replication is single-threaded with a hard-coded processing path and cannot spread work across CPUs, processes, or hosts to optimize resource usage or deal with delays due to i/o.
And more operational stuff.
2. Zero-downtime upgrade. Tungsten Enterprise enables applications to implement “daylight maintenance” by taking slave databases transparently offline, upgrading/maintaining them, and bringing them back online again. This is critical for 24×7 systems that cannot afford to go down for a day in order to upgrade from MySQL 5.1 to 5.5.
Tungsten provides the management commands to make this possible, for example a simple “switch” command that flips a master/slave pair without stopping applications. Tungsten also provides the necessary connectivity, namely transparent, high-performance proxies that hide the location of master and slave databases. Finally, Tungsten Replicator provides underlying flush transactions that allow us to confirm that all user transactions have safely reached slaves before switching the master.
Now we get to performance, but read-only, and even then only for limited use cases.
3. Performance scaling. Tungsten Enterprise can load balance reads across slaves which unloads the master and utilizes otherwise idle slaves. The performance increases can be quite impressive–for reads that are highly I/O intensive you get linear scaling as you add slaves. It is less beneficial for simple key lookups, especially when databases are fully memory resident.
We have a couple of ways to do this but they basically use transparent proxies or a JDBC library to split reads and writes automatically by examining incoming transactions. For this to work most effectively applications have to follow certain conventions for their transactions. However, they are very simple and many applications do them already. For example, under our “SmartScale” load balancing only auto-commit reads are eligible for automatic load balancing. We don’t try to load-balance reads within transactions as this would lead to errors for many applications. Our performance scaling capabilities will grow better as our parallel replication capabilities improve–a big problem with MySQL replication is slaves tend to lag the masters in busy systems, hence cannot be used as easily for reads that require current information.
Yet more operations.
4. Automated administration and failure recovery. Tungsten Enterprise allows users to do complex operations like switch master/slave or take database nodes on and offline. Also many operations like failover or recognizing new slave nodes occur automatically. This reduces labor enormously both on the part of administrators as well as the effort to implement such operations in the first place.
Tungsten enables this use case in two ways. It has built-in procedures for management tasks (switch, backup, restore, online, offline, etc.) Tungsten also has built-in monitoring that feeds a rules engine containing rules to handle master failure, slave failure, new slave appearing, etc. These execute automatically, though users can turn off features like automatic failover at their discretion.
5. Disaster Recovery Sites. Tungsten Enterprise allows users to set up and and manage DR sites containing up-to-date database copies. The key features that enable this are automatic VIP management (we use a virtual IP to allow the DR site to continue receiving updates even as the master moves on the main site) as well as global transaction IDs in the replicator, which ensure local and remote slaves do not become confused about their position regardless of how failure occurs. I have a full write-up of the architecture in the following article:
Finally it’s time to raise questions about Continuent competitors.
I hope this answers your questions and is not too late. I read with interest your post on DbShards. One interesting problem I have never quite figured out with their architecture is how to ensure requests are serialized. They intercept requests at the library level and apply them both locally as well as in a log that can be applied to other databases. The big problem with this approach is transaction ordering. You either have to accept extremely slow performance (e.g., by single-threading all updates) or applications have to generate order independent updates, which means a significant rewrite to follow these conventions. Otherwise you end up with slaves that look different from the master. Either way it greatly limits the market.
Schooner has always impressed me as being the most interesting play of those you are covering. It’s interesting they bailed out of hardware. Appliances for MySQL strike me as a non-starter. SSD integration has been very rapidly explored and commoditized by members of the MySQL community–people like Mark Callaghan and Peter Zaitsev are more hardware driven than any database people I have ever worked with. The fact that the code that accesses storage devices is completely open helps drive this kind of behavior. Schooner still is going to have a problem that MySQL users are very resistant to proprietary code, at least the big ones like Facebook, Yahoo, Rackspace, and Google.
A bit about future plans:
Meanwhile I expect we will take a page from Schooner and start patching the MySQL server over time to make our software run more effectively and provide better solutions to customers. Large web properties in particular are very comfortable with this approach.
And finally some mercifully short rah-rah, still on the theme “We only acknowledge one competitor and we can clobber them.” Actually, I moved this from further up, where it seemed a bit jarring.
I anticipate that over the course of 2011 Tungsten will be recognized as the best replication engine for MySQL. This is a big statement considering the entrenched position of MySQL built-in replication but our feature set is very broad and we will also have most of it available in open source by mid-year, which means that a lot more people will start using it. The list of stable improvements over MySQL is so long I cannot fit them on a single slide any more.