An Open Letter to the Twitter Guys

To: Evan Williams & Biz Stone
RE: Fix it before we nix it

I am a huge fan. You are two good guys. Twitter is the most addictive thing I've experienced since I quit smoking in 1987, and it's so much better for my health. I have built a multinational circle of friends because of Twitter. We care about each other even if many times we have never met.

I could extol your virtues at length. In fact, I often have. As a writer, I love the Twitter story.

Late one night, while an exhausted, enthusiastic team slaves away, one absent founder uses the SMS hack to send a photo link of himself gloating over a glass of wine. That photo spawns an epiphany. The hack is more valuable than the company. They give investors their money back and start over again. Twitter is born. The team shares it with a few friends who share it...you've heard this part before... .

We're up to the part where Twitter goes viral, where growth accelerates, where most users have never met you and many have never even heard of you. We'e at the part where servers are straining and those who do know and like and respect and appreciate what you have done are getting frustrated with crashes and glitches and lost tweets and unexplained weirdities.

You two are on a fast track, moving at a breath taking pace, but you've just come around a sharp turn and discovered you've hit a tough patch of mud. It can slow you down, cause you to slip and fall on your butts or even get you stuck. The patch is more dangerous than you think and it requires your immediate attention.

Social media people are a fickle lot. We are more loyal t each other than we are to the sites we visit. If some of us leave and find somewhere else to hang out there can be unexpected stampedes of friends. The good news, is that will ease the obvious strain on your current system. The bad news is you can very rapidly become yesterday's news

Ev/Biz, people are talking. This last "scheduled maintenance schedule for prime time i your prime market, got moved and when you finally did it, the end result was that things were more of a mess than they were before. Tweeters are talking mostly about Twitter and they are not singing praise. Some folks say you should be bought, bring in adult supervisors to manage things or that maybe a circle of friends should go give Pownce a try. I do know of such a case, but I'll wager that somewhere this week a couple of young entrepreneurs started working on a business plan to replace you.

I hope not. I'm still hopeful and joyful about Twitter. I like it there. I have a wish list a mile long for Twitter, but before I start gushing about that stuff, I need to feel confident that Twitter can walk solidly for a while without limping or stumbling.

Here are just a few ideas, you might want to mull over:

• Fix what's broken. OK, Twitter is a hack built on thin Rails. It was neat for a few dozen people, but you have gone viral and can be potentially huge very quickly. You need to rebuild the whole thing. My technical friends say you should use PHP, not as a patch in a few places, but you need to build the foundation so that it is solid and scalable and you need to do it fast because you have fallen behind. How do you do this in transition? You talk with your userbase.  You keep them informed. You kill yourselves to do what you say you will do.

 

• Reveal monetization plan. If you have it, share it. We users need to understand how it will impact us. Biz, I loved the time we shared in Spain, but it disturbed me when you smiled and said, "there's so many things that we can do." Silicon Valley graveyards are filled with startup corpses, whose founders uttered the same words. My 25 years of consulting companies taught me the #1 cause of death is not lack of financing. Its lack of focus. I have heard so many gushes of what a startup COULD do. The trick is to figure out what you SHOULD do.

• Share your data. Like Facebook, Twitter seems to be a bit squirrelly with its numbers. A little more transparency might help you. It is useful to a great many people to know how many users you have, how often the average Tweeter posts, what the average number of followers a Tweeter has, how many messages a day, geographic distribution, etc. I simply do not understand why you are not willing to share such data. How can sharing it hurt you? Sharing it will allow the crowd to share wisdom with you on what it means.

• Don't sell while broken. You'll suffer the same loss of return that you woud selling a Mercedes with a bashed in fender.

• Start a Twitter blog. You both have decent blogs but you need one that is focused on talking to Twitterville. Hell Evan, you of all people do not need advice on the value of blogging. But there is a need for conversations deeper than 140 characters allows. Share a few aspirations and fears. We are a passionate and knowledgeable community. But neither of you appear to me to be particularly interested in having an ongoing dialogue with customers and that is a bit disappointing from my perspective.

I believe that the two of you have an opportunity of a lifetime and right now you are at a crossroads, one direction puts you on an interactive superhighway and the other will give you an ending like Thelma & Louise.

Sincerely,

ShelIsrael

Comments

[yndygo]

Nice to see that Biz made it over here and is actually addressing your points Shel!

Also nice to see where the issues are obviously actually being considered seriously and with intent to fix.

I'm certainly lost as to what some of the commenters are saying - but I agree utterly with Susan Reynolds that I would miss our Twitter Village were it to become a ghost town.

I think I'd add one more thing tho - while functionality is critical, there's also a little room for improvement in strict site design. A link to the blog on the main Twitter page would've made it easier to find, for instance.

Let's hope that while they work on the engineering aspect, they also remember that UI is a huge part of attracting and keeping new users.

[mousewords]

I'm glad to read the comments and post together! Now I have confidence that Twitter's going to be just fine. The community I've discovered here is truly a wonderful one, and I look forward to spending time in it and watching it grow.

[Des Walsh]

Yes. No Twitter blog = "trust us, we know what's best for you and anyway even if we don't it's your tough luck".

[Linda Sherman]

I have only been on Twitter since Christmas but I have found it to be of exceptional personal value. I feel privileged to be part of this community.
Aside from some genuine friendships that I have had the opportunity to begin forming....Where else can you happen to be awake at 3AM in LA, get a heads up from new Twitterbud @mkrigsman about an important blog from @shelisrael and - bam! - there you are. Life in real time. Thanks for bringing me along for the ride.
@lindasherman

[Allen Stern]

When I wrote about Twitter's lack of a business model and how it will hurt later on, Twittervestory Fred Wilson said it didn't matter... I agree with you that they should reveal the plan now so we know whether to go all in or not.

http://www.centernetworks.com/twitter-business-model

[Nahum Gershon]

Great post, Shel. It expresses our sentiments & the hurdles we've all experienced. I hope Ev/Biz take your advice to heart and do something. Otherwise, it would take time to rebuilt this wonderful community.

Ev/Biz, we greatly appreciate what you've done for our hearts & minds and are looking forward to living a long life in Twitterville.

@nahumg
www.SocioSphere.com

[Nahum Gershon]

Sorry for the mistake- my site is www.SocioSphere.net

[cvander]

That's the kind of pressure we need to stand for. Twitter did it right on the beginning and I hope it's not a lucky hit that will sink to their current challenges.

[Michael Galpin]

The criticisms of Ruby on Rails is justified. Rails has a basic scalability problem: it cannot talk to multiple databases at the same time.

So let's say you get so many users and tweets that putting them in the same database causes any queries to be ... slow to say the least. So you put user 1 and his tweets in database 1, and user 2 and her tweets in database 2. But what happens when user 3 follows both user 1 and user 2? You need to be able to get user 1's tweets from database 1 and user 2's tweets from database 2. This is hard to do in Rails.

When Alex posted issues with this to the Rails community (and encountered unfair criticism for doing so) there were some plugins to Rails created. I don't think any of those really did what Twitter needed and Twitter is still using a single database. This was the information I got from Alex and other engineers at Twitter last fall.

It's easy to be a backseat driver and say they need to ditch Rails. Truth is that scalability issues like this or not easy regardless of technology stack. They are just more well known in things outside of Rails.

[bob]

Please correct the subheader at the top of the page.

"Social Media & it's impact on business & culture."

should be

"Social Media & its impact on business & culture."

[Mark Evans]

Shel,

Great post - with all the right questions asked. Twitter really finds itself at a huge fork in the road strategically. If it can come up with a business model, it could be huge as long as it follows some of the steps you offer.

Mark

[Todd Sieling]

> No Twitter blog = "trust us, we know what's best for you and anyway even if we don't it's your tough luck".

That's an incredibly unfair statement and can't go unchallenged. First of all, Twitter is itself a microblogging platform of sorts, and you could look at twitter.com/twitter. Or you could go here: http://blog.twitter.com/ and find -oh, it's a blog. A blog mentioned in their tweets. And linked at the bottom of their pages.

But moreover, while a blog is largely expected from web companies, it's not requisite and there are alternative ways to communicate effectively. To equate the absence of a blog with the dismissive attitude described above is absurd in Twitter's case and pretty much any other. Biz even came to this blog and commented, which takes that much more effort to reach out. But don't let that stop the ad hominem; where would the fun in posting comments be, right?

[John]

Quoting Michael
"Rails has a basic scalability problem: it cannot talk to multiple databases at the same time."

If this is Rails only roadblock to scalability, it can be easily solved by providing a plugin (I think there is at one if not more now) for it. Come on people, I am sure other scripting languages like PHP, Perl, Python went through the same set of issues, except without the notoriety.

[Ray Grieselhuber]

Fantastic post, Shel.

I have to say, as a user, and as someone who is building a service that makes heavy use of Twitter, I see real value in simplicity of Twitter as a utility. More importantly, the calibre of people who, with great generosity, share their inights, thoughts, and feelings on a daily basis makes it, IMHO, the best casual social network out there.

Regarding monetization - I'm a little unsure why this matters so much with regards to whether or not the community should cast their lot with Twitter. Everybody needs to eat, and at some point, Ev and Biz will want to monetize the platform (maybe not in 08).

If Twitter was some fly-by-night operation run by people we didn't know, I could see being concerned about what will happen as a result of monetization. These are well-known, high-integrity individuals who obviously care about building something of value, in the community sense. I have no problem committing to such a high quality community and platform, because I trust that the guys behind this app will do the right thing, by themselves and by the rest of us.

[Ontario Emperor]

I think that Biz's delaying of the monetization model until later is unwise, since the monetization model could affect the growth model.

As for what the monetization model could be, I'm not sure (although LifeLock Twitter Incorporated resonates with me)...

[shel israel]

To Bob who kindly asked me not to comment on Rails because I am not a developer, I've been holding my breath for 24 hours, but I'm disturbed by your comment. Kindly do not tell me what I can or cannot talk about. If I won't take that from my government, why on Earth would I accept that from you. I may not be technical, but I do hang out with a few geeks who know a bit about the subject. If I write something in error, I encourage you to point it out. Other than that, I'll talk about whatever I damned well please.

[Ethan Jewett]

Shel, I don't think there's any problem with you or I speculating about the root cause of Twitter's scalability problems. Even if the result is that I accuse you on Twitter of trolling the Rails community :-)

The only problem with speculation is that we might be (probably are?) wrong. I think in this case you and your technical contacts are probably on the wrong track. This conversation has already occurred once, with the base accusation that Rails can't talk to multiple databases. Once aired, the problem was solved relatively quickly and easily via a code contribution from the community: http://drnicwilliams.com/2007/04/12/magic-multi-connections-a-facility-in-rails-to-talk-to-more-than-one-database-at-a-time/

So even at the beginning, the problem was not the Rails application server architecture, but an issue of database contention. David HH's response to the criticism may have seemed a bit defensive, but at root he was correct that the best way to engage the community is to air your issues in community forums rather than try to work through your problems silently and then accuse the product of a shortcoming out of frustration during an interview. It certainly seems that the whole thing could have been handled better.

DHH also seems to be correct that at the application server level, Rails scales easily (though expensively) by simply throwing hardware at the problem. However, Twitter wasn't dealing with an application server problem and therefore wasn't dealing with a specifically "Rails" problem.

Moving on to the current discussion, it seems the root issue is still database contention, possibly in conjunction with a client polling architecture that may be reaching the limit of scale. See http://www.highscalability.com/scaling-twitter-making-twitter-10000-percent-faster and http://www.readwriteweb.com/archives/xmpp_web.php for decent overviews.

Having used Rails, Ruby and PHP (though not having scaled anything), I can tell you that there is nothing inherent in either that solves or significantly exacerbates either of these issues. On the database side, applications built using either language/framework will generate SQL statements that query the database. In pure PHP, you usually write the SQL yourself. Rails does a lot of the SQL work for you, but it is certainly possible to override that assist and do it yourself if you've developed a situation where further optimization is required.

In short, all factors seem to point to inherent architectural issues that Twitter is struggling with. There are always ways to approach these issues, but it's far more complex than switching from one language or framework to another.

[Quackking]

This thread airs lots of frustration at the inability of Twitter's backend to handle the flood of new users and new uses, but the unstated assumption of all the whining (I don't mean to diss the whiners, but the word seems appropriate -- after all, Twitter is *free*, dudes, and so are you, to write and launch a competitor..) is what I focused on in a piece I wrote in April for Broadcasting & Cable Magazine -- the 'namespace' piece. Twitter users are so heavily invested in the reputation and brand that their names represent inside the Twitter namespace that they are more willing to suffer and *beg* Twitter to fix things than to move.

FYI, you can read the B&C version here - and there's a link to the slightly different Mediazulu.com think piece here.

[Michael Galpin]

Quoting John: "If this is Rails only roadblock to scalability, it can be easily solved by providing a plugin"

Quoting Ethan: "This conversation has already occurred once, with the base accusation that Rails can't talk to multiple databases. Once aired, the problem was solved relatively quickly and easily via a code contribution from the community:"

Wouldn't it be great if things were this cut and dry? But go down to the "Scaling Twitter" link also provided by Ethan. All of Twitter is still running on a single 8-core server. What I had heard from the Twitter developers was that none of the various pluging proposes/created to solve the multi-database issue were able to work for their situation.

Could this be an issue with Twitter's architecture? Maybe. But take a look at that "magic multi-connection" plugin. It does not tackle the kind of federated database architecture you need to scale both horizontally and vertically.

Twitter's recent architectural evolution has pushed the load away from the database by using messaging and caching. They had to homegrown on the messaging. This is what all big web companies have to do. And it was they will have to do again if they want to scale their data access with Rails.

[Bob]

Shel,

I didn't mean to offend your sensibility by implying that you shouldn't have your first amendment right to freedom of speech. What frustrates me is the perpetuation of the "Rails can't scale" myth (unjustified IMO) taken out of context taking on a life of its own.

As an example, it has long been rumored and speculated by the tech industry that Microsoft VBSCript / ASP / SQL Server stack doesn't scale and that everyone should use UNIX based system / architecture. Well, MySpace has obviously proven that rumor wrong because it is built, first with Cold Fusion, and later on with ASP and ASP.NET. There is a great article on baselinemag.com that chronicled the scalability growth and challenges faced by MySpace
http://www.baselinemag.com/c/a/Projects-Networks-and-Storage/Inside-MySpacecom/

Unfortunately the article contents doesn't appear to be available anymore but it described in good details how MySpace handled (or rather struggled) with changing their architecture repeatedly at each stage of their growth spurts. It is very clear that a lot of the scaling issues centered about connectivity to databases, adding SAN, caching, etc. The scripting language (VBScript) is such a tiny part of it that I don't think it was even mentioned in the article.

Anyway, my point is that scalability is a very complex area that only a handful of people in the world can claim intimate knowledge with it. It is however very convenient for a 20-some year old programmer (with no background in designing scalable sites) to point the finger at Rails for their scalability problem. By the way, scaling up is absolutely the wrong way to go because it will tap out at some point in time. Scaling out is the only way to go.

[shel israel]

Bob,
Thanks for respecting my 1 Amendment Rights. I guess I would have more respect for you input, if you would identify yourself so that we would know where you are coming from. Your email translates into "Anonymous Troll" which doesn't give you a lot of street creds in this particular neighborhood.

[Bob]

Me: 20+ years software professional, many years of hands-on development and senior management experience, delivered commercial products, websites, etc, on Windows and UNIX/ Linux platforms. In fact, for my current startup, I did all the development work with Ruby on Rails, so I know a thing or two about this subject. I also have a database background. Even then, I don't claim to be one of those handful of experts that have worked on scalable sites. However, I have managed enough developers, written enough code, and worked on enough varied system architectures to know a BS excuse when I see one, ie, the blame laid on Rails by Twitter. As for my anonymous troll handle, it's a private inside joke and will remain private like my full identity. Thanks for letting me air my thoughts.

[shel israel]

So you say, but as far as I and my readers can see, you are nothing but an anonymous troll. On this site we take down anonymous commenters. It's called the Living Room Policy. I don't let strangers hang out in either place. If you choose to remain anonymous in future Comments, I will take you down and will not let you post here again.

[Maureen Francis]

This post applies to other social networks I belong to. I should share it with their founders.

[Krista Neher]

Wow. Great post and Great Advice. I love the points on transparency and earning the trust of your users.

I especially want to echo the "Share your Data" point. There are a number of other sites taking advantage of the deficiencies of data on Twitter, which is really a lost opportunity to capture that traffic and engagement.

I still love Twitter but HOPE that it improves :)