Monday, March 3, 2008


One of the few social networking sites that I still use is LinkedIn. This decision basically came down to one simple concept... if I needed to network with people who could help me, what site would I use? Needless to say, I'm still not sure if it would help me, but it is still fun to follow along with how they change their system over time. So, today I logged in and found a whole new interface and a new feature called "Status". This feature is pretty pointless, you can update your homepage with a short description of what you are doing right now. Can you say "I wanna be twitter?" Heading over to their engineering blog, there is a posting which talks briefly about the implementation of this grand new feature. Let me quote:
We've converted our back end to use this new service, and late last year we used the new Network Updates Service to create a time-based feed of the events in your network. We are currently processing over 40 million updates posted daily, which will continue to increase as the network grows. For someone like myself who has recently graduated from university, this is a daunting number, and yet, it's why I continue to enjoy working at LinkedIn.

To handle this capacity, we use a cluster of ActiveMQ machines for distributed JMS processing, EHCache for fast in-process caching, and an efficient update procedure for when new updates are persisted. We've designed the system to be scalable right from the start, horizontally partitioning across multiple databases as growth demands.

We will continue to provide new updates into your feed, and for those of you who do not visit the site regularly, you will be receiving these updates in a bi-weekly email so you can be kept up to date with the latest activity in your network. I would like to thank the team that worked on the Status feature, particularly Scott Schlegel (Web Development), James Lau (Engineering), and Qian Su (Engineering).

I'm simply astounded. It took 3 people, who knows how long, to implement one of the simplest possible features ever. 40 million updates a day? That is only 463 updates a second! A 2 cpu server could handle that without even breaking a sweat. I'm glad that you all are thinking about scalability, but these days it isn't rocket science and getting so excited over the ability to post a sticky isn't all that great.


Seanpile said...

Hi Jon,

I wrote that post; I didn't intend for it to come across as too "self-congratulatory", which I assume is what got your feathers ruffled. The blog is there to create some transparency into our engineering and regardless of how trivial you think this feature may be, I'm sure there are others who are interested.


Jon Scott Stevens said...

Hey Sean,

I do really appreciate more transparency. Keep that coming! However, if you could please talk about more interesting features, that would be a lot more informative. I also question the fact that it took so many of you to add such a simple feature. Maybe if you discussed things in more detail, that would make your postings more enjoyable as well.