To: Evan Williams & Biz Stone
RE: Fix it before we nix it
I am a huge fan. You are two good guys. Twitter is the most addictive thing I've experienced since I quit smoking in 1987, and it's so much better for my health. I have built a multinational circle of friends because of Twitter. We care about each other even if many times we have never met.
I could extol your virtues at length. In fact, I often have. As a writer, I love the Twitter story.
Late one night, while an exhausted, enthusiastic team slaves away, one absent founder uses the SMS hack to send a photo link of himself gloating over a glass of wine. That photo spawns an epiphany. The hack is more valuable than the company. They give investors their money back and start over again. Twitter is born. The team shares it with a few friends who share it...you've heard this part before... .
We're up to the part where Twitter goes viral, where growth accelerates, where most users have never met you and many have never even heard of you. We'e at the part where servers are straining and those who do know and like and respect and appreciate what you have done are getting frustrated with crashes and glitches and lost tweets and unexplained weirdities.
You two are on a fast track, moving at a breath taking pace, but you've just come around a sharp turn and discovered you've hit a tough patch of mud. It can slow you down, cause you to slip and fall on your butts or even get you stuck. The patch is more dangerous than you think and it requires your immediate attention.
Social media people are a fickle lot. We are more loyal t each other than we are to the sites we visit. If some of us leave and find somewhere else to hang out there can be unexpected stampedes of friends. The good news, is that will ease the obvious strain on your current system. The bad news is you can very rapidly become yesterday's news
Ev/Biz, people are talking. This last "scheduled maintenance schedule for prime time i your prime market, got moved and when you finally did it, the end result was that things were more of a mess than they were before. Tweeters are talking mostly about Twitter and they are not singing praise. Some folks say you should be bought, bring in adult supervisors to manage things or that maybe a circle of friends should go give Pownce a try. I do know of such a case, but I'll wager that somewhere this week a couple of young entrepreneurs started working on a business plan to replace you.
I hope not. I'm still hopeful and joyful about Twitter. I like it there. I have a wish list a mile long for Twitter, but before I start gushing about that stuff, I need to feel confident that Twitter can walk solidly for a while without limping or stumbling.
Here are just a few ideas, you might want to mull over:
- Fix what's broken. OK, Twitter is a hack built on thin Rails. It was neat for a few dozen people, but you have gone viral and can be potentially huge very quickly. You need to rebuild the whole thing. My technical friends say you should use PHP, not as a patch in a few places, but you need to build the foundation so that it is solid and scalable and you need to do it fast because you have fallen behind. How do you do this in transition? You talk with your userbase. You keep them informed. You kill yourselves to do what you say you will do.
- Reveal monetization plan. If you have it, share it. We users need to understand how it will impact us. Biz, I loved the time we shared in Spain, but it disturbed me when you smiled and said, "there's so many things that we can do." Silicon Valley graveyards are filled with startup corpses, whose founders uttered the same words. My 25 years of consulting companies taught me the #1 cause of death is not lack of financing. Its lack of focus. I have heard so many gushes of what a startup COULD do. The trick is to figure out what you SHOULD do.
- Share your data. Like Facebook, Twitter seems to be a bit squirrelly with its numbers. A little more transparency might help you. It is useful to a great many people to know how many users you have, how often the average Tweeter posts, what the average number of followers a Tweeter has, how many messages a day, geographic distribution, etc. I simply do not understand why you are not willing to share such data. How can sharing it hurt you? Sharing it will allow the crowd to share wisdom with you on what it means.
- Don't sell while broken. You'll suffer the same loss of return that you woud selling a Mercedes with a bashed in fender.
- Start a Twitter blog. You both have decent blogs but you need one that is focused on talking to Twitterville. Hell Evan, you of all people do not need advice on the value of blogging. But there is a need for conversations deeper than 140 characters allows. Share a few aspirations and fears. We are a passionate and knowledgeable community. But neither of you appear to me to be particularly interested in having an ongoing dialogue with customers and that is a bit disappointing from my perspective.
I believe that the two of you have an opportunity of a lifetime and right now you are at a crossroads, one direction puts you on an interactive superhighway and the other will give you an ending like Thelma & Louise.
Sincerely,
ShelIsrael



Shel, great post. I would like to add "faster International support" though Joi just sign deal with Twitter to expand Japan market.
Posted by: isaac | January 25, 2008 at 04:40 PM
Shel, I can only say that I hope Ev & Biz listen to your wise counsel. My neighborhood in Twitterville is populated with some amazing people -- and I would never have met them any other way. So we are very, very loyal to Twitter. We tried Jaiku. We tried Pownce. And still go there from time when Twitter is down. But we always come back. Twitter is our home because it's where our "tweeps" (Twitter peeps) are.
But our home is dilapidated and falling down around us. We don't want to flee to the suburbs; we want to stay in Twitterville. Unlike Shel, I don't know what it will take to put the house back in shape; I just know that we cannot keep growing a vibrant community in the online equivalent of a fixer-upper.
Posted by: Connie Reece | January 25, 2008 at 04:46 PM
Wow.
That's 100% dead on in my book.
I really hope Ev & Biz not only read it, but take it to heart.
I loathe the prospect of having to 'recreate' my Twitter socnet elsewhere - but there comes a point where that's less of a hassle than putting up with the issues if I remain.
They need to address it before that pinnacle point is reached by too many folks.
Great post Shel. Seconding everything you said.
Posted by: yndygo | January 25, 2008 at 04:52 PM
Shel, you totally burst my bubble. You mean just as I'm starting to have fun with my 150 or so Twitter friends, the tech Gods might take it away from me? Say it isn't so.
Posted by: Kara Soluri | January 25, 2008 at 05:07 PM
Shel - Great post. These days, I'm evangelizing social media/transparency services to healthcare provider clients - who are surprisingly receptive - only to have them call me and say "It's broke" when it comes to important components like Twitter. Commercial success for Twitter lies beyond the Silicon Valley bubble where bugs, crashes and outages are breakfast conversation starters at Bucks. Keep those open letters coming.
Posted by: Tom | January 25, 2008 at 06:02 PM
http://blog.twitter.com/ ?
Posted by: 123 | January 25, 2008 at 06:03 PM
Twitter Blog... check!
http://blog.twitter.com/
I agree with your other points.
Posted by: Joe Lazarus | January 25, 2008 at 06:14 PM
Very heart felt post. I have come to enjoy twitter very much. I hope they focus and find a way to keep it going. If they get the support of twitter users I'm sure they can do almost anything and several of us would be willing to help them figure it out.
Posted by: Aruni Gunasegaram | January 25, 2008 at 06:20 PM
Maybe we should talk to them about XMPP. http://tinyurl.com/ystmq9
Posted by: Sam Lawrence | January 25, 2008 at 06:48 PM
Spot on, Shel.
Posted by: Warner Crocker | January 25, 2008 at 07:22 PM
Twitter is a bit like a rundown tenant, where we all love the homies and try to overlook the rats. It is a powerful community, but right now I wouldn't build an important communication network on its back. It has great promise as a crisis communication platform, but it is hard to rely on something in a crisis that is in its own crisis.
Posted by: Kami Huyse | January 25, 2008 at 07:48 PM
Great post. As usual: Right on target and right on time. I have several things I would like to do with Twitter, but every time one of the extended bouncing ball sessions comes along, I wonder if it will support the load. Twitter could easily be a great help to those who do user support. But if we can't depend on it to deliver the message, it won't work.
Just my two cents!
Posted by: Kathy Jacobs | January 25, 2008 at 07:49 PM
I would love to go to my blog and talk about this right now, but you nailed it right on the head, I couldn't say it any better.
Posted by: Shawn Kirsch | January 25, 2008 at 08:04 PM
Dead nuts on! I hope they read your post and act. Thanks writing what you did and doing it so well.
Posted by: Keith Shepard | January 25, 2008 at 08:24 PM
Indulge me as I try to respond in the spirit. ["You" and "y'all" refers to ... heh, the twitmeisters. *snort*]
Earlier this evening I tweeted (c/w "Trick question" warning), "Is software in any sense fractal?"
I was hitting a nail right on the head as hard as I could.
["Social object" bla-yada-blah. *Gack Ptui! And you know where you can shove your cigar!*]
Any fool can pick up a diamond. (Woa up, kidz; this aulde dawg doesn't use sarcasm, and when I feel a need to gore your ox your ox will know it indubitably ... the sophistic can veil the existential, but only the existential leaves blood dripping to the floor.)
Point is: most would not. /Can/ VS /Does/.
"Is" VS "ought". Bodhicitta is foundational, but it's only foundational. "Karma" is about entailments ... it's a noun/verb construct that pertainst to /action/.
(BTW: if someone can get me a Dominican Republic 1st level domain I gotta dandy name for you. W/respect to ma.tt this one has been in the works for a while.)
Lateral: benzene ring ... a principled practitioner pondered the tracings of rain on the window. Eureka? Yaa, shurr. *shrug* I worked 28 years to get my insight. "Eureka" captures it, but it hardly it.
Since I brought him up: kudos to Matt for WP.
And kudos to you for grabbing hold of the chain where it was closest to you.
Me? *shrug* I can always plead age ... or ill-health ... or drunkeness *Alas, unlike the beats, I've the addictive personality but no appetite for addiction!* From time to time I direct this venerable old Toughbook towards a promising vein and do a bit of plumbing. *shrug* What I know is so long as /y'all/ are happy with "80% sufficient 80% of the time for 80% of the time" I'm shit outta luck.
Who knows, maybe if you guyz break through the "sushi for lunch in the ca'" paradigm google has set ... we might keep SpaceShip Earth from augering in. (I've got audio of Columbia's last seconds. Also [lost] audio from Apollo 13. You can't beat principled practioners. Can't. Cannot. Can notRPTnot. All depends where you place mind, is whott.)
~~~ ~~ @ Shel
--bentrem
Posted by: Ben Tremblay | January 25, 2008 at 08:36 PM
First off, I would ask kindly that non-developers refrain from hating on Rails. Per Alex Payne, Rails has become such a convenient scapegoat for Twitter miserable uptime. A scalable website goes far beyond a language and framework. Alex doesn't have the background or the skills to build a scalable site, so he blames it on Rails. But seriously, I am sick of hearing non-techie talk about Rails failing to scale when they can't tell me the difference between Ruby and Rails.
Posted by: Bob | January 25, 2008 at 09:00 PM
Alex Payne is freaking clueless about scalability, he has all the experience of a new college graduate who wouldn’t know scalability if it bit him in the ass. Saying that all the scalability problem lies with Ruby on Rails is like a blind man holding onto an elephant’s tail and saying that an elephant is a thin wiry spineless animal
Posted by: John | January 25, 2008 at 09:06 PM
Interesting post. We'll have to see if it does anything. I love Twitter,.. and I know it can be better.
Posted by: Duane Brown | January 25, 2008 at 09:32 PM
How many startups have to experience crib death before people realize that Rails is a trap - easy to get something up and running but a disaster for scalable deployment.
Remember how iLike scrambled around borrowing all those servers when they went onto Facebook? They have quietly moved out of the Rails ghetto. Time for Twitter to do the same...
Posted by: Jasper | January 25, 2008 at 09:34 PM
Without twitter I'd be in contact with about .001% of the amazing people I really do feel that I "know" in a community that is - to me - irreplaceable. That said, my twitter circle includes people like you, Shel's, and being of a certain age and given a certain life perspective I couldn't think of anyone I'd rather have advice from.
I hope Ev and Biz follow through on even some of your wise counsel. Things just wouldn't be the same without the connectivity we've come to look forward to each day, and the friends and opportunities that the twitter birdie dropped into my life at a time when I needed them most.
Posted by: Susan Reynolds | January 25, 2008 at 09:36 PM
Shell,thank you for these words. Your concerns reflect Twitter's stated mission: to build a global messaging utility which can be depended on 99.99% of the time.
It's important that you realize the efforts we are making toward this goal are not a quick fix. Twitter made a big move forward when we hired Lee Mighdoll to lead our engineering and operations team. Lee has been working tirelessly to accomplish the simultaneous tasks of bolstering our current infrastructure while seeking genius engineers to build up our tiny team (currently: 4).
We hope that you can have enough patience to see us through the efforts we are currently engaged in to bring solid stability to Twitter. Again, it won't be something that happens with one down-time or one miracle weekend. It will be a sustained effort.
With regard to revealing our monetization plans I can tell you honestly that we are far more focused on growth and reliability in 2008. Our own hunches as well as advice from some very knowledgeable advisors have us adding great ideas to a list of revenue solutions which we will visit in earnest when we are ready.
Sharing data is something we're happy to do. Until now, nobody has asked us "how often the average Tweeter posts" or "what the average number of followers a Tweeter has." Geographic distribution is easy enough to share as well. I'll see if I can dig that stuff up for you next week.
Reporters, analysts, and advertising agencies are very interested in the total number of Twitter users but we have decided not to share that particular statistic. We're happy sharing other data like API traffic or average number of updates but total users is ultimately not an indication of success--especially when you are talking about something as fragile and significant as community.
Finally, you requested that we start a Twitter blog. We blog more at the Twitter blog than our own blogs! If you don't already, I hope you subscribe to the Twitter Blog because I pretty much publish everything I can think of there.
Posted by: Biz Stone | January 25, 2008 at 09:46 PM
Ahem~
I know the people who loaded the computers onto The Shuttle Columbia-and what happened was much more...than…I’m going to discuss when talking about the direction a board would take a specific advanced new media outlet in a specified regional geography-where talent and need are at high standard. (Lockheed Martin, Edwards AFB, Dresden, all within the scope of my passion…trust me.)
shel is a perfect combo of wit and wisdom in that he is simply taking a serious and personal tone to inspire those his remarks are directed to, to engage those involved to heed the actions he has declared are necessary in a capacity to serve-to wit his afformentionable stake and declaration warrant perfectional situation, and thus, so forth and so on…
So there you sit, right where I lost you at, with them fancy kinda words we (albeit in my case somewhat displaced) Midwesterner’s come up with…
Keep seeking what you want to know…c’mon…read more…
when speaking a vocal directive, with honest professionalism, sure, why not. After all, it says so, when it says… “consultant.”
I’d believe it, too. - and I can’t believe any less would be expected in this arena of an area…even though I no longer live on my favorite coasts…did I ever tell you the time I swam at Jones beach one day and at the Rincon the next….ok I won’t tell the rest of it either…I’m just saying…when we spend enough time ‘round these parts to get good at what we’re good at, we ought be listened to when we discuss something, and, if we put forward directives and give actual ways to achieve what we want to see done, which is the step up from simply whining, a la mr. trembley, then we really begin to see something done. Or, in sooo much simpler terms, “just do it.”
Oh wait, “ what did you want done again?”
Yeah…see…and it reminds me of politics-a term I almost know is nearly synonymous with ethical dilemma, greed and despair…say what you’re gonna wanna see and also say how you’re gonna wanna see what you’re gonna wanna see gotten to, get it?
Tudiousity…is his, though. I think I’ve said enough…
~!jesatiu
Posted by: jesseloop | January 25, 2008 at 10:00 PM
It's great to see the conversation here, and I hope that some concrete actions back up the strong words.
Posted by: rslux | January 25, 2008 at 10:02 PM
Looking forward to the statistics that Biz promised us next week.
Posted by: Baher | January 25, 2008 at 10:15 PM
My have two special thanks. First to Susan Reynolds. I was worried about what you would say here, because what happened around your misfortune showed this community at its very best. I continue to be moved by what has happened.
My second thanks for the prompt, sincere and informative response by Twitter co-founder Biz Stone. Biz, I really did not know there was a Twitter Blog. I think we should both be a bit embarrassed by that fact. I have now subscribed and I look forward to hearing from you next week.
Posted by: shel israel | January 25, 2008 at 11:02 PM
Nice to see that Biz made it over here and is actually addressing your points Shel!
Also nice to see where the issues are obviously actually being considered seriously and with intent to fix.
I'm certainly lost as to what some of the commenters are saying - but I agree utterly with Susan Reynolds that I would miss our Twitter Village were it to become a ghost town.
I think I'd add one more thing tho - while functionality is critical, there's also a little room for improvement in strict site design. A link to the blog on the main Twitter page would've made it easier to find, for instance.
Let's hope that while they work on the engineering aspect, they also remember that UI is a huge part of attracting and keeping new users.
Posted by: yndygo | January 25, 2008 at 11:13 PM
I'm glad to read the comments and post together! Now I have confidence that Twitter's going to be just fine. The community I've discovered here is truly a wonderful one, and I look forward to spending time in it and watching it grow.
Posted by: mousewords | January 25, 2008 at 11:44 PM
Yes. No Twitter blog = "trust us, we know what's best for you and anyway even if we don't it's your tough luck".
Posted by: Des Walsh | January 26, 2008 at 01:08 AM
I have only been on Twitter since Christmas but I have found it to be of exceptional personal value. I feel privileged to be part of this community.
Aside from some genuine friendships that I have had the opportunity to begin forming....Where else can you happen to be awake at 3AM in LA, get a heads up from new Twitterbud @mkrigsman about an important blog from @shelisrael and - bam! - there you are. Life in real time. Thanks for bringing me along for the ride.
@lindasherman
Posted by: Linda Sherman | January 26, 2008 at 03:21 AM
When I wrote about Twitter's lack of a business model and how it will hurt later on, Twittervestory Fred Wilson said it didn't matter... I agree with you that they should reveal the plan now so we know whether to go all in or not.
http://www.centernetworks.com/twitter-business-model
Posted by: Allen Stern | January 26, 2008 at 05:25 AM
Great post, Shel. It expresses our sentiments & the hurdles we've all experienced. I hope Ev/Biz take your advice to heart and do something. Otherwise, it would take time to rebuilt this wonderful community.
Ev/Biz, we greatly appreciate what you've done for our hearts & minds and are looking forward to living a long life in Twitterville.
@nahumg
www.SocioSphere.com
Posted by: Nahum Gershon | January 26, 2008 at 06:03 AM
Sorry for the mistake- my site is www.SocioSphere.net
Posted by: Nahum Gershon | January 26, 2008 at 06:33 AM
That's the kind of pressure we need to stand for. Twitter did it right on the beginning and I hope it's not a lucky hit that will sink to their current challenges.
Posted by: cvander | January 26, 2008 at 06:44 AM
The criticisms of Ruby on Rails is justified. Rails has a basic scalability problem: it cannot talk to multiple databases at the same time.
So let's say you get so many users and tweets that putting them in the same database causes any queries to be ... slow to say the least. So you put user 1 and his tweets in database 1, and user 2 and her tweets in database 2. But what happens when user 3 follows both user 1 and user 2? You need to be able to get user 1's tweets from database 1 and user 2's tweets from database 2. This is hard to do in Rails.
When Alex posted issues with this to the Rails community (and encountered unfair criticism for doing so) there were some plugins to Rails created. I don't think any of those really did what Twitter needed and Twitter is still using a single database. This was the information I got from Alex and other engineers at Twitter last fall.
It's easy to be a backseat driver and say they need to ditch Rails. Truth is that scalability issues like this or not easy regardless of technology stack. They are just more well known in things outside of Rails.
Posted by: Michael Galpin | January 26, 2008 at 08:12 AM
Please correct the subheader at the top of the page.
"Social Media & it's impact on business & culture."
should be
"Social Media & its impact on business & culture."
Posted by: bob | January 26, 2008 at 09:37 AM
Shel,
Great post - with all the right questions asked. Twitter really finds itself at a huge fork in the road strategically. If it can come up with a business model, it could be huge as long as it follows some of the steps you offer.
Mark
Posted by: Mark Evans | January 26, 2008 at 10:28 AM
> No Twitter blog = "trust us, we know what's best for you and anyway even if we don't it's your tough luck".
That's an incredibly unfair statement and can't go unchallenged. First of all, Twitter is itself a microblogging platform of sorts, and you could look at twitter.com/twitter. Or you could go here: http://blog.twitter.com/ and find -oh, it's a blog. A blog mentioned in their tweets. And linked at the bottom of their pages.
But moreover, while a blog is largely expected from web companies, it's not requisite and there are alternative ways to communicate effectively. To equate the absence of a blog with the dismissive attitude described above is absurd in Twitter's case and pretty much any other. Biz even came to this blog and commented, which takes that much more effort to reach out. But don't let that stop the ad hominem; where would the fun in posting comments be, right?
Posted by: Todd Sieling | January 26, 2008 at 11:06 AM
Quoting Michael
"Rails has a basic scalability problem: it cannot talk to multiple databases at the same time."
If this is Rails only roadblock to scalability, it can be easily solved by providing a plugin (I think there is at one if not more now) for it. Come on people, I am sure other scripting languages like PHP, Perl, Python went through the same set of issues, except without the notoriety.
Posted by: John | January 26, 2008 at 11:08 AM
Fantastic post, Shel.
I have to say, as a user, and as someone who is building a service that makes heavy use of Twitter, I see real value in simplicity of Twitter as a utility. More importantly, the calibre of people who, with great generosity, share their inights, thoughts, and feelings on a daily basis makes it, IMHO, the best casual social network out there.
Regarding monetization - I'm a little unsure why this matters so much with regards to whether or not the community should cast their lot with Twitter. Everybody needs to eat, and at some point, Ev and Biz will want to monetize the platform (maybe not in 08).
If Twitter was some fly-by-night operation run by people we didn't know, I could see being concerned about what will happen as a result of monetization. These are well-known, high-integrity individuals who obviously care about building something of value, in the community sense. I have no problem committing to such a high quality community and platform, because I trust that the guys behind this app will do the right thing, by themselves and by the rest of us.
Posted by: Ray Grieselhuber | January 26, 2008 at 11:44 AM
I think that Biz's delaying of the monetization model until later is unwise, since the monetization model could affect the growth model.
As for what the monetization model could be, I'm not sure (although LifeLock Twitter Incorporated resonates with me)...
Posted by: Ontario Emperor | January 26, 2008 at 06:23 PM
To Bob who kindly asked me not to comment on Rails because I am not a developer, I've been holding my breath for 24 hours, but I'm disturbed by your comment. Kindly do not tell me what I can or cannot talk about. If I won't take that from my government, why on Earth would I accept that from you. I may not be technical, but I do hang out with a few geeks who know a bit about the subject. If I write something in error, I encourage you to point it out. Other than that, I'll talk about whatever I damned well please.
Posted by: shel israel | January 26, 2008 at 06:44 PM
Shel, I don't think there's any problem with you or I speculating about the root cause of Twitter's scalability problems. Even if the result is that I accuse you on Twitter of trolling the Rails community :-)
The only problem with speculation is that we might be (probably are?) wrong. I think in this case you and your technical contacts are probably on the wrong track. This conversation has already occurred once, with the base accusation that Rails can't talk to multiple databases. Once aired, the problem was solved relatively quickly and easily via a code contribution from the community: http://drnicwilliams.com/2007/04/12/magic-multi-connections-a-facility-in-rails-to-talk-to-more-than-one-database-at-a-time/
So even at the beginning, the problem was not the Rails application server architecture, but an issue of database contention. David HH's response to the criticism may have seemed a bit defensive, but at root he was correct that the best way to engage the community is to air your issues in community forums rather than try to work through your problems silently and then accuse the product of a shortcoming out of frustration during an interview. It certainly seems that the whole thing could have been handled better.
DHH also seems to be correct that at the application server level, Rails scales easily (though expensively) by simply throwing hardware at the problem. However, Twitter wasn't dealing with an application server problem and therefore wasn't dealing with a specifically "Rails" problem.
Moving on to the current discussion, it seems the root issue is still database contention, possibly in conjunction with a client polling architecture that may be reaching the limit of scale. See http://www.highscalability.com/scaling-twitter-making-twitter-10000-percent-faster and http://www.readwriteweb.com/archives/xmpp_web.php for decent overviews.
Having used Rails, Ruby and PHP (though not having scaled anything), I can tell you that there is nothing inherent in either that solves or significantly exacerbates either of these issues. On the database side, applications built using either language/framework will generate SQL statements that query the database. In pure PHP, you usually write the SQL yourself. Rails does a lot of the SQL work for you, but it is certainly possible to override that assist and do it yourself if you've developed a situation where further optimization is required.
In short, all factors seem to point to inherent architectural issues that Twitter is struggling with. There are always ways to approach these issues, but it's far more complex than switching from one language or framework to another.
Posted by: Ethan Jewett | January 27, 2008 at 06:59 AM
This thread airs lots of frustration at the inability of Twitter's backend to handle the flood of new users and new uses, but the unstated assumption of all the whining (I don't mean to diss the whiners, but the word seems appropriate -- after all, Twitter is *free*, dudes, and so are you, to write and launch a competitor..) is what I focused on in a piece I wrote in April for Broadcasting & Cable Magazine -- the 'namespace' piece. Twitter users are so heavily invested in the reputation and brand that their names represent inside the Twitter namespace that they are more willing to suffer and *beg* Twitter to fix things than to move.
FYI, you can read the B&C version here - and there's a link to the slightly different Mediazulu.com think piece here.
Posted by: Quackking | January 27, 2008 at 08:53 AM
Quoting John: "If this is Rails only roadblock to scalability, it can be easily solved by providing a plugin"
Quoting Ethan: "This conversation has already occurred once, with the base accusation that Rails can't talk to multiple databases. Once aired, the problem was solved relatively quickly and easily via a code contribution from the community:"
Wouldn't it be great if things were this cut and dry? But go down to the "Scaling Twitter" link also provided by Ethan. All of Twitter is still running on a single 8-core server. What I had heard from the Twitter developers was that none of the various pluging proposes/created to solve the multi-database issue were able to work for their situation.
Could this be an issue with Twitter's architecture? Maybe. But take a look at that "magic multi-connection" plugin. It does not tackle the kind of federated database architecture you need to scale both horizontally and vertically.
Twitter's recent architectural evolution has pushed the load away from the database by using messaging and caching. They had to homegrown on the messaging. This is what all big web companies have to do. And it was they will have to do again if they want to scale their data access with Rails.
Posted by: Michael Galpin | January 27, 2008 at 09:17 AM
Shel,
I didn't mean to offend your sensibility by implying that you shouldn't have your first amendment right to freedom of speech. What frustrates me is the perpetuation of the "Rails can't scale" myth (unjustified IMO) taken out of context taking on a life of its own.
As an example, it has long been rumored and speculated by the tech industry that Microsoft VBSCript / ASP / SQL Server stack doesn't scale and that everyone should use UNIX based system / architecture. Well, MySpace has obviously proven that rumor wrong because it is built, first with Cold Fusion, and later on with ASP and ASP.NET. There is a great article on baselinemag.com that chronicled the scalability growth and challenges faced by MySpace
http://www.baselinemag.com/c/a/Projects-Networks-and-Storage/Inside-MySpacecom/
Unfortunately the article contents doesn't appear to be available anymore but it described in good details how MySpace handled (or rather struggled) with changing their architecture repeatedly at each stage of their growth spurts. It is very clear that a lot of the scaling issues centered about connectivity to databases, adding SAN, caching, etc. The scripting language (VBScript) is such a tiny part of it that I don't think it was even mentioned in the article.
Anyway, my point is that scalability is a very complex area that only a handful of people in the world can claim intimate knowledge with it. It is however very convenient for a 20-some year old programmer (with no background in designing scalable sites) to point the finger at Rails for their scalability problem. By the way, scaling up is absolutely the wrong way to go because it will tap out at some point in time. Scaling out is the only way to go.
Posted by: Bob | January 27, 2008 at 12:41 PM
Bob,
Thanks for respecting my 1 Amendment Rights. I guess I would have more respect for you input, if you would identify yourself so that we would know where you are coming from. Your email translates into "Anonymous Troll" which doesn't give you a lot of street creds in this particular neighborhood.
Posted by: shel israel | January 27, 2008 at 01:13 PM
Me: 20+ years software professional, many years of hands-on development and senior management experience, delivered commercial products, websites, etc, on Windows and UNIX/ Linux platforms. In fact, for my current startup, I did all the development work with Ruby on Rails, so I know a thing or two about this subject. I also have a database background. Even then, I don't claim to be one of those handful of experts that have worked on scalable sites. However, I have managed enough developers, written enough code, and worked on enough varied system architectures to know a BS excuse when I see one, ie, the blame laid on Rails by Twitter. As for my anonymous troll handle, it's a private inside joke and will remain private like my full identity. Thanks for letting me air my thoughts.
Posted by: Bob | January 27, 2008 at 03:16 PM
So you say, but as far as I and my readers can see, you are nothing but an anonymous troll. On this site we take down anonymous commenters. It's called the Living Room Policy. I don't let strangers hang out in either place. If you choose to remain anonymous in future Comments, I will take you down and will not let you post here again.
Posted by: shel israel | January 27, 2008 at 03:30 PM
This post applies to other social networks I belong to. I should share it with their founders.
Posted by: Maureen Francis | January 27, 2008 at 07:37 PM
Wow. Great post and Great Advice. I love the points on transparency and earning the trust of your users.
I especially want to echo the "Share your Data" point. There are a number of other sites taking advantage of the deficiencies of data on Twitter, which is really a lost opportunity to capture that traffic and engagement.
I still love Twitter but HOPE that it improves :)
Posted by: Krista Neher | February 01, 2008 at 04:49 PM
Thanks! I hadn't heard the story and always find it facscinating how things end of viral.
From my limited experience, your advice is right on...
Thanks for story.
Posted by: Linda | June 17, 2008 at 01:33 PM
I read this very well and find good knowledge. I want to give you thanks because it is great comment.
________________________________
william rozar
Wide Circles
Posted by: william rozar | July 27, 2008 at 12:45 AM