Comments on: Fixing Twitter in three letters: CEP

By: Twitter turmoil: when does it end? | Enterprise Alley | ZDNet.com

Twitter turmoil: when does it end? | Enterprise Alley | ZDNet.com — Thu, 24 Apr 2008 15:15:32 +0000

[...] than what Twitter needs today. (Coral8 can do the same things.) That’s a lot of headroom. http://www.dbms2.com/2008/01/16/twitter-could-e…; has some [...]

By: Text Technologies»Blog Archive » The comprehensive guide to upgrading – or replacing – Twitter

Sun, 10 Feb 2008 01:56:26 +0000

[...] and distributing messages in real time. As I’ve already pointed out, this should be done via complex event/stream processing (CEP), not by writing everything first to a database. The need for much more complex filters just makes [...]

By: Text Technologies»Blog Archive » Sturgeon’s Law, and the future technology of social technology

Tue, 05 Feb 2008 10:04:18 +0000

[...] Filtering technology for Twitter (CEP would do the job) [...]

By: Curt Monash

Curt Monash — Mon, 21 Jan 2008 21:06:44 +0000

Jim,

Thanks. I was going to post that myself. There’s a comment in the original Dave Winer scripting.com comment thread (liked above) from a Twitter guy spelling out the point.

Best,

CAM

By: Jim Deville

Jim Deville — Mon, 21 Jan 2008 17:13:56 +0000

Mike, Curt:
Yes Twitter is running on Rails, however, if you go back to the original scaling kerfufle, you’ll see that Rails wasn’t the scaling issue. DB access was. Rails can scale just fine. It just didn’t have the ability to properly scale across multiple databases at that time.

I don’t know if the keynote was a database thing or not, but I wanted to point out that Rails isn’t necessarily the problem.

By: Don Park

Don Park — Sat, 19 Jan 2008 22:58:57 +0000

As I see it, the main problem is that they don’t see the problem. Hopefully, improvements made by Joi’s Twitter Japan team will trickle up to Twitter US.

By: Curt Monash

Curt Monash — Sat, 19 Jan 2008 19:52:55 +0000

Mike,

That may be. But if it were to scale up significantly, database access could quickly become the bottleneck. Also, I think it’s likely that for usability we’ll need more filters or channels, and that compounds the data access issues.

CAM

By: Mike

Mike — Sat, 19 Jan 2008 17:54:09 +0000

AFAIK, Twitter’s performance problems are mainly caused by the fact that it’s implemented in Rails; I don’t believe it is a data management/data access issue at all. But I could be wrong.

By: Curt Monash

Curt Monash — Sat, 19 Jan 2008 04:11:17 +0000

Henri,

RAM always matters.

But you’re right that solid-state memory has a role to play here. If I’m right with my ballpark figure of 20 megabytes/minute as an initial design goal (and that only for bursts), then a few hundred gigabytes of solid-state memory would be a huge help.

Eventually Twitter will surely want to institute more searchability of message archives. That’s when solid-state memory will really shine.

CAM

By: Scripting News for 1/18/2008 « Scripting News Annex

Scripting News for 1/18/2008 « Scripting News Annex — Sat, 19 Jan 2008 03:54:25 +0000

[...] read this story on DBMS2, as part of the initial discussion, that explained there is commercial-grade software used [...]