Skip to navigation | Skip to content

Share your knowledge. Make a difference.

How We Engineered Squidoo

1 - I can do better 2 - Jury's out 3 - Pretty darn good 4 - Splendiferous 5 - Awesometastic (by 69 people)   Your rating: 1 - I can do better 2 - Jury's out 3 - Pretty darn good 4 - Splendiferous 5 - Awesometastic

Ranked #564 in How-To, #5540 overall

Donates to Squidoo Charity Fund

Rated G. (Control what you see)

About Squidoo

 

Founded in mid 2005, Squidoo launched in early 2006 and quickly gained traction as a platform for sharing ideas online. On Squidoo, everyday experts (yes, that means you) can handcraft individual web pages (just to be difficult, we call them lenses) about hobbies and interests and stuff for sale and ideas worth spreading.

Compete recently reported that Squidoo was the 14th fastest growing website during 2007 in the entire world. According to Alexa, Squidoo has ranked in the top 1,000 most visited sites on the web for at least the past year. Quantcast currently ranks us in the mid 300s.

As of January 2008 Squidoo has over 150,000 users authoring almost 400,000 pages. We currently see over 7 million monthly visits, and at least twice as many pageviews. Factoring in Ajax, RSS, and API calls, our servers currently process about 45 application requests per second.

With just a tiny full-time team of four five, not to mention all our volunteers and Citizens, we think it's pretty incredible to keep such a high traffic site online while also managing to release new features and bug fixes on a regular basis. And so I thought it'd be interesting to share a little background on our engineering efforts , the technical challenges we've faced, and how we got to where we are today. Enjoy!

The Starting Point 

Summer, 2005

Here we were—sitting in a small, sparsely furnished room, discussing how we were going to build a new platform (and ultimately, an ecosystem, a community) which we dreamed would change the way people find things online.

From the beginning we knew the traditional approach wouldn't work. We had a tight schedule, yet we had no VC money or plans for hiring a massive staff. How would we accomplish such lofty goals with such a tiny budget?

Where It All Started 

Our hip, modern office is located just a few feet away from the mighty Hudson River, 10 short miles from New York City in a 100% authentic town called Irvington, NY.

It's named after Washington Irving, although I try not to stick around late enough at night to find out if there really is a headless horseman.

IMG_0785.JPG by squid11

IMG_0785.JPG

IMG_0784.JPG by squid11

IMG_0784.JPG

IMG_0783.JPG by squid11

IMG_0783.JPG

IMG_0782.JPG by squid11

IMG_0782.JPG

IMG_0781.JPG by squid11

IMG_0781.JPG

IMG_0780.JPG by squid11

IMG_0780.JPG

IMG_0787.JPG by squid11

IMG_0787.JPG

IMG_0788.JPG by squid11

IMG_0788.JPG

IMG_0779.JPG by squid11

IMG_0779.JPG

IMG_0786.JPG by squid11

IMG_0786.JPG

Choosing a Platform 

Squidoo's first stage of development began in the fall of 2005. At the time, Ruby on Rails was starting to emerge as a dominant force in what was only recently being dubbed Web 2.0. Although we considered using Rails, it was still a relatively immature and unproven technology platform (boy, a lot has changed since then), and we chose PHP in its place. Since PHP goes hand in hand with Linux, Apache, and MySQL, adopting the entire LAMP platform was a no-brainer.

Choosing PHP—with its extensive documentation, ubiquitous server support, abundance of third party libraries and API wrappers, and optimizable speed—was a reasonable decision. PHP has its detractors, but with the right architecture and proper discipline in place, it has no shortage of potential for driving high-powered web applications.

Speaking of architecture, ours might appear very familiar to a Rails developer—featuring a clear distinction between actions, models, helpers, and so on. We even have our own dispatcher, which came in handy when one of our main requirements was designing a human-friendly URL structure.

Techies Take Note

This lens is just a story. If you're looking for a more detailed account of the technical stuff, check out my Scaling a Web App lens. Sorry, it's not quite done yet, but I promise to finish it soon.

Beginning Development 

Setting up a team

To keep our staff as lean as possible, we considered several options, ultimately leading to a parnership with Viget Labs, who helped us build version 1.0 of Squidoo.

Hiring in-house employees vs an outside consulting firm can be a difficult choice for any startup. On one hand, hiring employees can be a great (and sometimes inexpensive) way to ensure dedication to your project, but the security of fixed cost development and no long-term employee commitments proved to be the right choice for us.

Working closely with Viget's designers and developers, we created and managed a constantly evolving wiki that served as our team's development guidelines. The wiki technology actually turned out to be a great asset during development, allowing our team to iterate through new design ideas without waisting valuable time reconstructing a more formal specifications document. The same wiki system is still in use today, and remains a hub for our project planning and software specs.

We quickly established a habit of 2 week development iterations. On every third week our aim was to lock down a feature list for the next iteration as quickly as possible, reserving the latter part of the week for deploying updates, testing code, and fixing bugs. This approach allowed us to prototype new features and release them into the wild with lightning speed.

Servers, servers, servers! 

We couldn't afford to hire a dedicated server administrator (or two). In startup mode most employees wear multiple hats, and my role included not only development, but designing and configuring our server installation as well.

While I had maintained my own servers for years, I had relatively little experience with high traffic, high availability server clusters. To be on the safe side, we figured it wise to find a partner who could help fill in the gaps when it came to tough technical issues. That partner became Rackspace, a managed hosting provider well-respected for their focus on "fanatical support". Although their services come at a significant premium, Rackspace has since helped us navigate effortlessly through a number of technical hurdles.

With several high profile launch announcements, we expected traffic to grow quickly. To make sure we were properly prepared, our initial setup included a four server configuration: 1 staging server, 2 web/app servers, and 1 database server. Although a software-based load balancer was considerably cheaper, we chose a Cisco hardware appliance to split the load between our web servers—wary of the downtime required during the inevitable upgrade from a software to hardware-based solution once traffic demanded it.

In retrospect, configuring two web servers may have been borderline premature optimization. Although our usage quickly grew to full capacity on both servers, the effort involved in synchronizing the two systems was tedious and error-prone.

With two web servers, we began running into all sorts of filesystem-related issues. We installed Unison (a two-way rsync) to make sure the avatars our forum users uploaded were available on both servers, and we invested in Zend Platform for multi-server persistent sessions, code acceleration, and aggregated error logs. Unfortunately, in a totally dynamic, user-centric application, the content caching feature of this commercial product isn't quite as useful.

Launching Our Beta 

December 2005

I screwed up. After our first few encounters with the blogosphere, we managed to compile a list of enough interested participants to begin a limited beta. Sure, our product was still a little buggy, and definitely needed some improvement, but it was finally ready to be tested by real, live humans. The mailing list was compiled, the email copy was written, and the beta invitations were sent....three times!

It was as public a statement as we could make that what we were offering was truly a beta product. And although we quickly learned from the mistake and added "take more care" as our mantra for potentially dangerous tasks, to this day I still haven't recovered from the anxiety of emailing large groups of people. If you've ever wished you could take back an email you wrote, you know what I mean. Daily Candy and Photojojo, I don't know how you do it.

A full page spread in the local paper!

Making the Transition 

Finally, in March 2006, our beta was complete. The world was ours—to infinity and beyond we marched!

Our contract with Viget had just ended, and it was a difficult time for us. Suddenly it was just down to one designer (Corey) and developer (myself). Our platform wasn't anywhere near complete, and we had ginormous goals to extend it with a plethora of new features.

Although I worked closely with Viget while developing our beta, there was still a significant learning curve in getting up to speed with the system (at that point it was approaching nearly 50,000 lines of code). Luckily, their team was very supportive during this process.

Hat tip to Ben for putting up with my hourly IM conversations, which usually went something like "So I'm building an Ajax call for saving tags from the profile page. Which controller do I put that in again?" Frustrating for him, but it didn't take me long to get the hang of things.

The larger issue was how to balance new development with resolving bug reports, responding to our community, recruiting freelance developers (for smaller projects), and, last but not least, marketing!

We struggled initially, but over time we were able to find a development process that worked for us, and things were just peachy...for a while.

Server Gremlins 

One day, out of the blue, all of our servers just crashed. I couldn't login from the command prompt, and we called Rackspace to manually reboot them. Once rebooted, the site ran normally, and I could find no trace of what happened.

Everything was fine for a few days, but then it happened again. And again. And again. It seemed we could only keep our servers online for a few hours at a time without requiring a hard reboot. At this point, it's safe to say I was seriously considering the benefits and lack of responsibility that come along with delivering pizzas for a living.

We soldiered on, and with a few days of effort and the help of a guru from Rackspace, we were able to trace the problem. A few new Squidooers had unwittingly added RSS modules which linked to the RSS feeds of their own lenses. When a surfer visited one of these corrupt lenses, the RSS module would update by visiting the lens, which would update the RSS module, which would visit the lens, which would update the RSS module...ok I'm going to stop now, because all this recursion is making me dizzy.

Once we found the culprit it was an easy fix. The RSS module now carefully screens for self-referencing feeds, preventing the same disaster and keeping the servers online so Squidooers can do their thing.

Like a Slug 

Our luck was turning, and the Squidoo platform was picking up steam. We'd managed to develop quite an extensive list of features (considering the size of our team), and traffic was growing exponentially. But with more features and more users came more congestion. Our database, which was growing at about a gigabyte every week, soon started to lag.

Although we had already spent some time optimizing the database layer (by being picky about the data we selected, creating indexes, etc), the database just couldn't keep up. Squidoo was crawling. We weren't database experts, but once again Rackspace came to the rescue. It turns out that even though we had a dedicated MySQL server, only small portions of its available resources were being used.

With a few quick tweaks (to the key buffer, innodb buffer, table cache, and a few others), Squidoo was flying again. Even better, our database server was a good sport and didn't even complain about the extra load. Thanks, db1!

Squidoo accepts an award at SXSW 2007

Spam: The Internet's Worst Nightmare 

By June 2007, fresh from winning an award for "best community site" at SXSW, Squidoo was in great shape. Our traffic was continuing to grow, and, if I got lucky, at least one person at every party I went to had at least heard of us (note to new startup founders: perfect your elevator pitch, because you're going to be using it a lot).

Things were going so well, in fact, that spammers started to notice. Within the course of a few short weeks, an enterprising spammer began massively polluting the blogosphere with links pointing to their Squidoo lenses. The spammer exploited the fact that we allowed <iframe> tags for more flexibility. In this case it proved too flexible, and the effect was that clicking on a link from a blog would bring you to a Squidoo lens, but immediately redirect you to a site that hosted—you guessed it—pornography.

The method used by the spammer made it appear that Squidoo was somehow affiliated, and shockwaves rippled throughout the blogosphere. We got a fair amount of heat, but did our best to remedy the situation as quickly as possible. Soon after, I posted followup notes explaining our position on spam and the new steps we were taking to keep Squidoo spam free. But the damage was done.

Lesson learned. Spam is a major problem on the internet, and in an age where reputation is everything, organizations can't afford not to have an aggressive anti-spam policy.

How We Handle Customer Service 

With so many users, it's only natural that customer service is going to be a big issue.

On the technical end, more users means more opportunities for running into obscure bugs, and when major ones hit it means responding can take a long time.

Our "release early and often" philosophy means that we get plenty of feedback from users on new features (sometimes within minutes of deployment).

In addition, keeping spam at bay is a major concern. Although we have a number of automated tools designed to block spam before it starts, we also rely on the community to report suspect activity.

All this leads to lots and lots of feedback. To save our sanity, all feedback is filtered through FogBugz, where our wonderfully dedicated customer service supervisor helps with common customer service issues, filtering and prioritizing any remaining bug reports, which then get sent to our development team. We also have a dedicated anti-spam editor who reviews spam reports (and other suspicious activity) and takes action as necessary to keep Squidoo a safe, well-lighted community.

Breaking It Down 

The key to creating a scalable application is designing an architecture that can be broken down into bite-sized chunks.

When our application servers started working overtime, we offloaded images and other static content to a lightweight web server (lighttpd) and optimized it for caching. No longer having to serve up a dozen or more static files with each request, our app servers gained a little breathing room.

When our Javascript codebase kept growing, we minified it into one tiny, compressed file. Now there are fewer web requests and less code to download.

When our database started slowing down, we installed memcached and scrutinized our model classes to cache everything possible. The result was a faster Squidoo, and a happier db1!

Where We're At Today 

Although some of the challenges chronicled above seem quite scary, we've learned a lot along the way and have had virtually no downtime in the past 6 months. We continue to release new features on a regular basis, and do our best to extinguish bugs as soon as we hear about them.

Our sister site, SquidU, features a lively discussion forum where lensmasters share tips, get help with common issues, and keep up to date with the newest features. The forum has over 70,000 posts and grows by a few hundred posts each day.

We have a number of exciting new projects just ahead, and we're currently looking for a new developer to help us make things happen.

Now Accepting Your Feedback 

Did I miss something? If you have any questions or comments please feel free to comment below.

ebay-grandma

What a journey - all so we can have fun as lensmasters - well and make a little money along the way for charity and for us. Thanks

Posted July 14, 2008

RedSportNiac

Nice stories. All the hardship and perseverance squidoo staff went thru definitely would inspire lensmater to produce quality lens and make squidoo the best internet community years after years. Great job.

Posted July 07, 2008

LindaJM

Thanks for the details. I read the whole thing, mesmerized. I appreciate knowing how this got started.

Posted June 18, 2008

MyKidsInheritance

Awesome job, Gil. Thanks for "inviting us all in" on how it all began. LOVE Squidoo.
Melissa

Posted June 10, 2008

RyanRE

You guys did a great job with Squidoo. PHP is a challenging, or at least it was for me. This was a cool lens. I like to see that backend side of things, no pun intended of course :)

Posted May 23, 2008

happy-jack

Thanks this makes me very
happy and the photos save
me a long air trip to visit you people. The very best to all of yoy

Posted May 09, 2008

BFuniv.com

Like a new theater production. Nothing seems to work right, but somehow everyone enjoys it. It is nice to know the route to the pleasures we now enjoy. Keep up the great work.

Posted May 08, 2008

DMedley

Thanks for some great insight into the workings of Squidoo. I have been building squidoo lenses for a year and this information brings a lot into perspective for me.

Posted May 06, 2008

jacquelinestone

Gil, thanks for never giving up. You guys have created a wonderful community here. I am thrilled to be a part of it. :)

Posted May 01, 2008

FoxMusic

Thanks for the insight into Squidoo

Posted April 27, 2008

Karendelac

An Easy 5 stars !
Gil, this was really a good read....you made it hard to put down, I have been on this site for 45 minutes even looking up technical terms.
Very Well Done for persistence and long hours and overcoming all barriers and problems.
Best wishes,
Karen

Posted April 25, 2008

natdg

Good to know about Squidoo and how it started. :)

Posted April 09, 2008

ronpass

Gil, it's great to see some of the history of Squidoo. You continue to innovate and provide a superb platform to promote affiliate products and not-for-profits. I appreciate the opportunity to donate my Squidoo earnings to worthwhile charities.

Posted April 05, 2008

JaguarJulie

Say did I already tell you what a great lens this is? You know I'm blonde, so sometimes I need a little help ya know! AB xoxoxo

Posted April 02, 2008

RickBasset

Squidoo is amazing! The work you and the others have put in to make this happen is incredible. I am an official Squidoo junkie.
Gotta go...need a fix!

Peace!

Posted March 28, 2008

BusyQueen

Thanks for the amazing story on Squidoo

Posted March 25, 2008

Marelisa

Hi Gil: I just launched my first lens on Monday of this week and I canīt tell you how much fun I'm having with squidoo. I can't stop logging on to ad dnew modules or tweak the ones I've already added. You guys are doing a fantastic job!

Posted March 22, 2008

Stephene

thanks for useful information.. i hv joined your fan club.. (^o^)

Posted March 16, 2008

driewe

This is fascinating. My word press blogs are getting jealous because I am spending more and more time on Squidoo :-)

Posted March 05, 2008

secondchancegifts

WOW, very interesting story and thanks for sharing it with us.

Posted February 18, 2008

moviestarstyle

great story about a great site! xoxo, moviestarstyle

Posted February 17, 2008

snaz

Thanks for the peek under the hood Gil, it's an informative & entertaining read. (LOL at the moments of pizza delivery contemplation) Congratulations to you and the others at squidoo-HQ. Also a HUGE Thank You for this wonderful community we all have come to know & love :)

Posted February 17, 2008

BobTheTeacher

Great to have such an excellent story told in such a fantastic manner. Very effective use of different modules to tell the story, too.

Bob Jenkins
- the Jerky lens guy and SquidooSecrets.com founder

Posted February 15, 2008

thomasz

I like this lens. 5 stars to you.

Posted February 13, 2008

neverburnoutrx

just genuinely appreciate the openness and transparency -- great job.

Posted February 11, 2008

700islands

Really heartened to see you chose PHP. There is so much written about why PHP is not good enough for serious developments and it clearly is. What an endorsement!

Posted February 09, 2008

levikujala

Truly a great story. I didn't even care that most of your references didn't make sense to me:) It's no wonder the community vibe is so strong here.

Posted February 08, 2008

Plato

Very interesting beginnings. I too suffer from large group email anxiety due to a mishap I won't soon forget. "take more care" good mantra. I hope that loser spammer got the boot straight away.

Onward & Upward

Posted February 06, 2008

mikehz

Gil, this is remarkable, I've wondered often how you keep this large site going with so little staff! Keep up the good work

Posted February 04, 2008

SingingLessons

Wow, great to hear that such a success story had its bumps in the road. Makes it seem more personal, like a real community should be.

Posted February 02, 2008

XanTastic

Squidoo Rocks...thanks for sharing the story as we've shared it from our end too!

Posted February 01, 2008

jackclee

Congratulations to the small team that created Squidoo. It is a brilliant concept and very impressive.
5 stars...

Posted February 01, 2008

bucks123

For someone that is usually not short of words, the only thing I can think of is I am thoroughly impressed with Squidoo and all the staff as we'll as all the members it has attracted. I'm probably your newest member (joined a few hours ago) and still stuck on this site.
The more I read, the more I want to keep reading and the more I want to contribute to a community that REALLY does have a clear cut "long term" objective on where it's headed.
I Am Impressed!

Posted January 31, 2008

PABro

This is a real "eye-opener". You guys have my utmost respect. I'm certainly proud to be a part of Squidoo in my own small way. You have my full support. Keep up the great work. - Paul.

Posted January 31, 2008

surianee

whoa... glad to be part of squidoo :D

Posted January 31, 2008

sonia_simone

This is deeply cool, I love having this "squidoo baby book." Thanks Gil. You really do rock, and the squids love you.

Posted January 28, 2008

coolandy

Great story. Its truly inspirational. I'd be glad if u tell me how u managed so much popularity. And got into alexa's top 800. I think squidoo's now posing a threat to ezinearticles at google.

Posted January 27, 2008

CarolynLBoyd

What I love most about Squidoo is that you have harnessed the power of people's passionate interests. Even when I'm not interested in a particular subject, reading what someone who is passionate about it has to say is so often fascinating. What a wonderful tool for gathering and sharing knowledge!

Posted January 26, 2008

Susan52

I won't pretend to understand every detail, but I get the gist of it. That is quite a story and, having been a programmer and analyst way back before the term "IT" was invented, as well as married to the head of computer operations, I particularly appreciate the parts about tackling the crashes and problems that came up along the way. Thanks for sharing, Gil. I'm impressed and very proud to be a part of Squidoo.

Posted January 26, 2008

badmsm

Wow, and I thought I was impressed before...we are unworthy!!! Still, very glad I found Squidoo, it feels like home (only more aquatic).

Posted January 25, 2008

flicker

Thanks, gil, for telling us the history of Squidoo. Very interesting.

Posted January 25, 2008

PhotoDub

Great story! Like Diane, I'd love to join your team but can't tell my SNOBOL from my COBOL...

Nice view of the Hudson you have there!

Mike

Posted January 25, 2008

Chadrew

Nice story. It's pretty amazing that you are able to run such a huge site with only 5 people.

Posted January 25, 2008

garethjax

that's a splendid story, one that every startup should read and spread! I've favourite the lens, if you update please squidcast :D

Posted January 25, 2008

charlino

Reading this brought back start-up memories of long ago. It is great to see this lens here. Well done.

Posted January 25, 2008

giltotherescue

To answer LMP's questions:

*You said Squidoo didn't use VC financing (smart move) but you didnt say how it was financed?

Gil: We're self funded. No plans for an IPO, probably ever.

* I imagine that you lived very frugally for the first year, but hopefully things are a bit earsier now. I admire the fact that you guys have not dramatically expanded with growth. Many small companies make that mistake.

Gil: Every decision we make is based around that fact--it's the only way we've survived.

* From your photos it looks like the office is on the ground floor...is that wise? Seems like it would be more secure to be 1 or 2 floors up. Or is security not a major issue as squidoo is hosted elsewhere?

Gil: Our servers are hosted in a specially designed datacenter at Rackspace.

* Finally Gil, from your photos you look about 21-22 MAX. This must be wrong or else you started with Squidoo just out of puberty.

Yep, I'm pretty young. More here: http://www.squidoo.com/gil#module1708207

Posted January 25, 2008

DianeFusco

Reading this fab story makes me want to join your team -- but unfortunately, I'm too tech-challenged!

Congrats!

Posted January 25, 2008

LMP

Great Post!
I learned alot but have a few questions:
*You said Squidoo didn't use VC financing (smart move) but you didnt say how it was financed? Is it personally bankrolled by Seth? Were there outside investors? Is there a plan for a IPO anytime soon?

* I imagine that you lived very frugally for the first year, but hopefully things are a bit earsier now. I admire the fact that you guys have not dramatically expanded with growth. Many small companies make that mistake.

* From your photos it looks like the office is on the ground floor...is that wise? Seems like it would be more secure to be 1 or 2 floors up. Or is security not a major issue as squidoo is hosted elsewhere?

* Finally Gil, from your photos you look about 21-22 MAX. This must be wrong or else you started with Squidoo just out of puberty. Are they old pics?

Posted January 25, 2008

Christopher_Scott

Great to learn more about Squidoo. Thanks Gil.

Posted January 24, 2008

jstonedotorg

Good story. What are the plans for the future?

Posted January 24, 2008

 
1 of 2 pages
X
giltotherescue

About giltotherescue

Gil Hildebrand, Jr. is an experienced software developer based in New York City. He is currently running things as the Chief Engineer of Squidoo, and has been plugging away as a LAMP developer since 1999.

giltotherescue's Pages

See all of giltotherescue's pages