About Squidoo
Compete recently reported that Squidoo was the 14th fastest growing website during 2007 in the entire world. According to Alexa, Squidoo has ranked in the top 1,000 most visited sites on the web for at least the past year. Quantcast currently ranks us in the mid 300s.
As of January 2008 Squidoo has over 150,000 users authoring almost 400,000 pages. We currently see over 7 million monthly visits, and at least twice as many pageviews. Factoring in Ajax, RSS, and API calls, our servers currently process about 45 application requests per second.
With just a tiny full-time team of four five, not to mention all our volunteers and Citizens, we think it's pretty incredible to keep such a high traffic site online while also managing to release new features and bug fixes on a regular basis. And so I thought it'd be interesting to share a little background on our engineering efforts , the technical challenges we've faced, and how we got to where we are today. Enjoy!
What's Here
- The Starting Point
- Choosing a Platform
- Techies Take Note
- Beginning Development
- Servers, servers, servers!
- Launching Our Beta
- A full page spread in the local paper!
- Making the Transition
- Server Gremlins
- Like a Slug
- Squidoo accepts an award at SXSW 2007
- Spam: The Internet's Worst Nightmare
- How We Handle Customer Service
- Breaking It Down
- Where We're At Today
- Batch Processing
- Now Accepting Your Feedback
The Starting Point
Summer, 2005
From the beginning we knew the traditional approach wouldn't work. We had a tight schedule, yet we had no VC money or plans for hiring a massive staff. How would we accomplish such lofty goals with such a tiny budget?
Where It All Started
Our hip, modern office is located just a few feet away from the mighty Hudson River, 10 short miles from New York City in a 100% authentic town called Irvington, NY.
It's named after Washington Irving, although I try not to stick around late enough at night to find out if there really is a headless horseman.
IMG_0785.JPG
IMG_0784.JPG
IMG_0783.JPG
IMG_0782.JPG
IMG_0781.JPG
IMG_0780.JPG
IMG_0787.JPG
IMG_0788.JPG
IMG_0779.JPG
IMG_0786.JPG
Choosing a Platform
Squidoo's first stage of development began in the fall of 2005. At the time, Ruby on Rails was starting to emerge as a dominant force in what was only recently being dubbed Web 2.0. Although we considered using Rails, it was still a relatively immature and unproven technology platform (boy, a lot has changed since then), and we chose PHP in its place. Since PHP goes hand in hand with Linux, Apache, and MySQL, adopting the entire LAMP platform was a no-brainer.Choosing PHP—with its extensive documentation, ubiquitous server support, abundance of third party libraries and API wrappers, and optimizable speed—was a reasonable decision. PHP has its detractors, but with the right architecture and proper discipline in place, it has no shortage of potential for driving high-powered web applications.
Speaking of architecture, ours might appear very familiar to a Rails developer—featuring a clear distinction between actions, models, helpers, and so on. We even have our own dispatcher, which came in handy when one of our main requirements was designing a human-friendly URL structure.
Techies Take Note
This lens is just a story. If you're looking for a more detailed account of the technical stuff, check out my Scaling a Web App lens. Sorry, it's not quite done yet, but I promise to finish it soon.
Beginning Development
Setting up a team
To keep our staff as lean as possible, we considered several options, ultimately leading to a parnership with Viget Labs, who helped us build version 1.0 of Squidoo.Hiring in-house employees vs an outside consulting firm can be a difficult choice for any startup. On one hand, hiring employees can be a great (and sometimes inexpensive) way to ensure dedication to your project, but the security of fixed cost development and no long-term employee commitments proved to be the right choice for us.
Working closely with Viget's designers and developers, we created and managed a constantly evolving wiki that served as our team's development guidelines. The wiki technology actually turned out to be a great asset during development, allowing our team to iterate through new design ideas without waisting valuable time reconstructing a more formal specifications document. The same wiki system is still in use today, and remains a hub for our project planning and software specs.
We quickly established a habit of 2 week development iterations. On every third week our aim was to lock down a feature list for the next iteration as quickly as possible, reserving the latter part of the week for deploying updates, testing code, and fixing bugs. This approach allowed us to prototype new features and release them into the wild with lightning speed.
Servers, servers, servers!
While I had maintained my own servers for years, I had relatively little experience with high traffic, high availability server clusters. To be on the safe side, we figured it wise to find a partner who could help fill in the gaps when it came to tough technical issues. That partner became Rackspace, a managed hosting provider well-respected for their focus on "fanatical support". Although their services come at a significant premium, Rackspace has since helped us navigate effortlessly through a number of technical hurdles.
With several high profile launch announcements, we expected traffic to grow quickly. To make sure we were properly prepared, our initial setup included a four server configuration: 1 staging server, 2 web/app servers, and 1 database server. Although a software-based load balancer was considerably cheaper, we chose a Cisco hardware appliance to split the load between our web servers—wary of the downtime required during the inevitable upgrade from a software to hardware-based solution once traffic demanded it.
In retrospect, configuring two web servers may have been borderline premature optimization. Although our usage quickly grew to full capacity on both servers, the effort involved in synchronizing the two systems was tedious and error-prone.
With two web servers, we began running into all sorts of filesystem-related issues. We installed Unison (a two-way rsync) to make sure the avatars our forum users uploaded were available on both servers, and we invested in Zend Platform for multi-server persistent sessions, code acceleration, and aggregated error logs. Unfortunately, in a totally dynamic, user-centric application, the content caching feature of this commercial product isn't quite as useful.
Launching Our Beta
December 2005
I screwed up. After our first few encounters with the blogosphere, we managed to compile a list of enough interested participants to begin a limited beta. Sure, our product was still a little buggy, and definitely needed some improvement, but it was finally ready to be tested by real, live humans. The mailing list was compiled, the email copy was written, and the beta invitations were sent....three times!It was as public a statement as we could make that what we were offering was truly a beta product. And although we quickly learned from the mistake and added "take more care" as our mantra for potentially dangerous tasks, to this day I still haven't recovered from the anxiety of emailing large groups of people. If you've ever wished you could take back an email you wrote, you know what I mean. Daily Candy and Photojojo, I don't know how you do it.
A full page spread in the local paper!
Making the Transition
Our contract with Viget had just ended, and it was a difficult time for us. Suddenly it was just down to one designer (Corey) and developer (myself). Our platform wasn't anywhere near complete, and we had ginormous goals to extend it with a plethora of new features.
Although I worked closely with Viget while developing our beta, there was still a significant learning curve in getting up to speed with the system (at that point it was approaching nearly 50,000 lines of code). Luckily, their team was very supportive during this process.
Hat tip to Ben for putting up with my hourly IM conversations, which usually went something like "So I'm building an Ajax call for saving tags from the profile page. Which controller do I put that in again?" Frustrating for him, but it didn't take me long to get the hang of things.
The larger issue was how to balance new development with resolving bug reports, responding to our community, recruiting freelance developers (for smaller projects), and, last but not least, marketing!
We struggled initially, but over time we were able to find a development process that worked for us, and things were just peachy...for a while.
Server Gremlins
One day, out of the blue, all of our servers just crashed. I couldn't login from the command prompt, and we called Rackspace to manually reboot them. Once rebooted, the site ran normally, and I could find no trace of what happened.Everything was fine for a few days, but then it happened again. And again. And again. It seemed we could only keep our servers online for a few hours at a time without requiring a hard reboot. At this point, it's safe to say I was seriously considering the benefits and lack of responsibility that come along with delivering pizzas for a living.
We soldiered on, and with a few days of effort and the help of a guru from Rackspace, we were able to trace the problem. A few new Squidooers had unwittingly added RSS modules which linked to the RSS feeds of their own lenses. When a surfer visited one of these corrupt lenses, the RSS module would update by visiting the lens, which would update the RSS module, which would visit the lens, which would update the RSS module...ok I'm going to stop now, because all this recursion is making me dizzy.
Once we found the culprit it was an easy fix. The RSS module now carefully screens for self-referencing feeds, preventing the same disaster and keeping the servers online so Squidooers can do their thing.
Like a Slug
Our luck was turning, and the Squidoo platform was picking up steam. We'd managed to develop quite an extensive list of features (considering the size of our team), and traffic was growing exponentially. But with more features and more users came more congestion. Our database, which was growing at about a gigabyte every week, soon started to lag.Although we had already spent some time optimizing the database layer (by being picky about the data we selected, creating indexes, etc), the database just couldn't keep up. Squidoo was crawling. We weren't database experts, but once again Rackspace came to the rescue. It turns out that even though we had a dedicated MySQL server, only small portions of its available resources were being used.
With a few quick tweaks (to the key buffer, innodb buffer, table cache, and a few others), Squidoo was flying again. Even better, our database server was a good sport and didn't even complain about the extra load. Thanks, db1!
Squidoo accepts an award at SXSW 2007
Spam: The Internet's Worst Nightmare
Things were going so well, in fact, that spammers started to notice. Within the course of a few short weeks, an enterprising spammer began massively polluting the blogosphere with links pointing to their Squidoo lenses. The spammer exploited the fact that we allowed <iframe> tags for more flexibility. In this case it proved too flexible, and the effect was that clicking on a link from a blog would bring you to a Squidoo lens, but immediately redirect you to a site that hosted—you guessed it—pornography.
The method used by the spammer made it appear that Squidoo was somehow affiliated, and shockwaves rippled throughout the blogosphere. We got a fair amount of heat, but did our best to remedy the situation as quickly as possible. Soon after, I posted followup notes explaining our position on spam and the new steps we were taking to keep Squidoo spam free. But the damage was done.
Lesson learned. Spam is a major problem on the internet, and in an age where reputation is everything, organizations can't afford not to have an aggressive anti-spam policy.
How We Handle Customer Service
On the technical end, more users means more opportunities for running into obscure bugs, and when major ones hit it means responding can take a long time.
Our "release early and often" philosophy means that we get plenty of feedback from users on new features (sometimes within minutes of deployment).
In addition, keeping spam at bay is a major concern. Although we have a number of automated tools designed to block spam before it starts, we also rely on the community to report suspect activity.
All this leads to lots and lots of feedback. To save our sanity, all feedback is filtered through FogBugz, where our wonderfully dedicated customer service supervisor helps with common customer service issues, filtering and prioritizing any remaining bug reports, which then get sent to our development team. We also have a dedicated anti-spam editor who reviews spam reports (and other suspicious activity) and takes action as necessary to keep Squidoo a safe, well-lighted community.
Breaking It Down
When our application servers started working overtime, we offloaded images and other static content to a lightweight web server (lighttpd) and optimized it for caching. No longer having to serve up a dozen or more static files with each request, our app servers gained a little breathing room.
When our Javascript codebase kept growing, we minified it into one tiny, compressed file. Now there are fewer web requests and less code to download.
When our database started slowing down, we installed memcached and scrutinized our model classes to cache everything possible. The result was a faster Squidoo, and a happier db1!
Where We're At Today
Our sister site, SquidU, features a lively discussion forum where lensmasters share tips, get help with common issues, and keep up to date with the newest features. The forum has over 70,000 posts and grows by a few hundred posts each day.
We have a number of exciting new projects just ahead, and we're currently looking for a new developer to help us make things happen.
Now Accepting Your Feedback
Did I miss something? If you have any questions or comments please feel free to comment below.
Ramkitten wrote...
Really interesting read. I'm glad you stopped by a lens of mine and commented, which led me here (well, to your list of lenses, but I came to this one first). I've been so busy creating lenses, I've never really looked into Squidoo as a company. Better a bit late than never! Now on to your other lenses.....
Posted April 07, 2009
GrowWear wrote...
Wonderful lens! Y'all look so cool on that couch. Love Seth's socks. :) Loved seeing your pix. Mostly I am amazed and awed by the tremendous job you guys do! Very proud to be a Squidooer!
Posted November 01, 2008
businessonapage wrote...
Once at Uni I made a 10 minute speech about a medical paper that explained how doctors and nurses treated a victim of a knife attack.
The majority of the paper explained how they had to calm the person down and then how they did the technical thing to clean, repair and bandage the wound.
Your lens covering the technical (servers, databases etc...) and non-technical (anti-spamming, elevator speech etc...) issues that arose in this project reminded me of the paper and provides a wonderful insight into what you guys have created.
Thank You for the history lesson and providing this platform for the Squidoo community. I'm glad to hear that there is more to come.
PS It's great to see the photos. Was the kitchen intentionally meant to look bigger than the equipment room?
Posted October 28, 2008
DrRichard wrote...
As a php and mysql geek myself this is really impressive. I've only built small applications so far and the challenges of designing and implementing for a high-traffic site are very interesting. Great work!!
Posted October 16, 2008
OhMe wrote...
I just love the fact that I can sit back and enjoy making my little lenses without having to understand all of this. Thank you for making it happen.
Posted October 03, 2008
ZBT wrote...
Being not even a month old here, your lens "rocked it" for me! Thank you for the insight to how it all began.
Posted October 02, 2008
Spook wrote...
I take my hat off to you and all your hard work and effort. I hope you believe it was all worth it.
Posted October 01, 2008
Lisa-Marie-Mary wrote...
I just love that you've laid out 'The Whole Squidstory' for us here! That is very cool! Thanks for sharing it! :)
Posted September 30, 2008
hesika wrote...
Thank you for giving us a glance about the history of development of Squidoo. It is very interesting and there is a lot of things to learn. For me, for example, the FogBugz could be the software, I could need in the next time. Thank you so much. Keep on your excellent work.
Posted September 12, 2008
wstrauss73 wrote...
Gil, thanks for posting this lens.. with all the info you provided it makes people think how much we take for granted when it comes to using this site. Without YOU and the rest of the Squidoo Team this wouldn't even be here. Thanks for all the time, hard work and brilliant concepts. Definitely 5-Stars :)
Bill S.
Posted September 06, 2008
credit-card wrote...
I understand - it's just a quick overview of the history and you, probably, had to deal with so many more things (besides the ones that were mentioned) for all this to happen and be the way it is. This lens makes me and, I bet, everyone who read it appreciate your work and the site so much more. Thank you, guys!
Posted August 09, 2008
samej wrote...
Interesting read.
Will be a bit more understanding now when waiting for a response to a feedback email, etc...
Posted July 29, 2008
The_Homeopath wrote...
This is so cool! Thank you for all of your work to make it easier for schmucks like me to play online!
Posted July 22, 2008
Gimmesome wrote...
Gil,
We usually don't hear much about the behind-the-scenes story except when there's a (thankfully rare) problem. It's nice to get this background about the company and the great folks who keep everything running. Thanks to you and everyone else at HQ!
Posted July 15, 2008
ebay-grandma wrote...
What a journey - all so we can have fun as lensmasters - well and make a little money along the way for charity and for us. Thanks
Posted July 14, 2008
RedSportNiac wrote...
Nice stories. All the hardship and perseverance squidoo staff went thru definitely would inspire lensmater to produce quality lens and make squidoo the best internet community years after years. Great job.
Posted July 07, 2008
LindaJM wrote...
Thanks for the details. I read the whole thing, mesmerized. I appreciate knowing how this got started.
Posted June 18, 2008
MyKidsInheritance wrote...
Awesome job, Gil. Thanks for "inviting us all in" on how it all began. LOVE Squidoo.
Melissa
Posted June 10, 2008
RyanRE wrote...
You guys did a great job with Squidoo. PHP is a challenging, or at least it was for me. This was a cool lens. I like to see that backend side of things, no pun intended of course :)
Posted May 23, 2008
happy-jack wrote...
Thanks this makes me very
happy and the photos save
me a long air trip to visit you people. The very best to all of yoy
Posted May 09, 2008
BFuniv.com wrote...
Like a new theater production. Nothing seems to work right, but somehow everyone enjoys it. It is nice to know the route to the pleasures we now enjoy. Keep up the great work.
Posted May 08, 2008
DMedley wrote...
Thanks for some great insight into the workings of Squidoo. I have been building squidoo lenses for a year and this information brings a lot into perspective for me.
Posted May 06, 2008
jacquelinestone wrote...
Gil, thanks for never giving up. You guys have created a wonderful community here. I am thrilled to be a part of it. :)
Posted May 01, 2008
Karendelac wrote...
An Easy 5 stars !
Gil, this was really a good read....you made it hard to put down, I have been on this site for 45 minutes even looking up technical terms.
Very Well Done for persistence and long hours and overcoming all barriers and problems.
Best wishes,
Karen
Posted April 25, 2008
ronpass wrote...
Gil, it's great to see some of the history of Squidoo. You continue to innovate and provide a superb platform to promote affiliate products and not-for-profits. I appreciate the opportunity to donate my Squidoo earnings to worthwhile charities.
Posted April 05, 2008
JaguarJulie wrote...
Say did I already tell you what a great lens this is? You know I'm blonde, so sometimes I need a little help ya know! AB xoxoxo
Posted April 02, 2008
RickBasset wrote...
Squidoo is amazing! The work you and the others have put in to make this happen is incredible. I am an official Squidoo junkie.
Gotta go...need a fix!
Peace!
Posted March 28, 2008
Marelisa wrote...
Hi Gil: I just launched my first lens on Monday of this week and I canīt tell you how much fun I'm having with squidoo. I can't stop logging on to ad dnew modules or tweak the ones I've already added. You guys are doing a fantastic job!
Posted March 22, 2008
Stephene wrote...
thanks for useful information.. i hv joined your fan club.. (^o^)
Posted March 16, 2008
driewe wrote...
This is fascinating. My word press blogs are getting jealous because I am spending more and more time on Squidoo :-)
Posted March 05, 2008
secondchancegifts wrote...
WOW, very interesting story and thanks for sharing it with us.
Posted February 18, 2008
moviestarstyle wrote...
great story about a great site! xoxo, moviestarstyle
Posted February 17, 2008
snaz wrote...
Thanks for the peek under the hood Gil, it's an informative & entertaining read. (LOL at the moments of pizza delivery contemplation) Congratulations to you and the others at squidoo-HQ. Also a HUGE Thank You for this wonderful community we all have come to know & love :)
Posted February 17, 2008
BobTheTeacher wrote...
Great to have such an excellent story told in such a fantastic manner. Very effective use of different modules to tell the story, too.
Bob Jenkins
- the Jerky lens guy and SquidooSecrets.com founder
Posted February 15, 2008
neverburnoutrx wrote...
just genuinely appreciate the openness and transparency -- great job.
Posted February 11, 2008
stormyweather wrote...
Really heartened to see you chose PHP. There is so much written about why PHP is not good enough for serious developments and it clearly is. What an endorsement!
Posted February 09, 2008
levikujala wrote...
Truly a great story. I didn't even care that most of your references didn't make sense to me:) It's no wonder the community vibe is so strong here.
Posted February 08, 2008
Plato wrote...
Very interesting beginnings. I too suffer from large group email anxiety due to a mishap I won't soon forget. "take more care" good mantra. I hope that loser spammer got the boot straight away.
Onward & Upward
Posted February 06, 2008
mikehz wrote...
Gil, this is remarkable, I've wondered often how you keep this large site going with so little staff! Keep up the good work
Posted February 04, 2008
SingingLessons wrote...
Wow, great to hear that such a success story had its bumps in the road. Makes it seem more personal, like a real community should be.
Posted February 02, 2008
XanTastic wrote...
Squidoo Rocks...thanks for sharing the story as we've shared it from our end too!
Posted February 01, 2008
jackclee wrote...
Congratulations to the small team that created Squidoo. It is a brilliant concept and very impressive.
5 stars...
Posted February 01, 2008
bucks123 wrote...
For someone that is usually not short of words, the only thing I can think of is I am thoroughly impressed with Squidoo and all the staff as we'll as all the members it has attracted. I'm probably your newest member (joined a few hours ago) and still stuck on this site.
The more I read, the more I want to keep reading and the more I want to contribute to a community that REALLY does have a clear cut "long term" objective on where it's headed.
I Am Impressed!
Posted January 31, 2008
The_Bard wrote...
This is a real "eye-opener". You guys have my utmost respect. I'm certainly proud to be a part of Squidoo in my own small way. You have my full support. Keep up the great work. - Paul.
Posted January 31, 2008
by giltotherescue
Related Topics
giltotherescue Recommends...
Favorited By
- jacquelinestone
- ChapelHillFiddler
- allsewnupfabrics
- CreativeLifeInProgress
- triathlontraining
- hesika
- MrLewisSmile
- clouda9
- OrganicGiftsByDiana
- emarketingwizard
- digitalscrap
- DrRichard
- Happiegrrrl
- keonij
- Sippriano
- MortenBlaabjerg
- Spook
- ZBT
- FoxMusic
- CharlesCotter
- garethjax
- jbcat47
- Romaxx
- Janusz
- BizGuides
- happy-jack
- Barkely
- LakeMom
- MyKidsInheritance
- OhMe
- See all 56 people
