Ajax Crawling

Pulling the content in data-only form (json, xml) and using JavaScript to build the HTML representation brings many advantages:

• the amount of information transferred through the wire is minimal,


• new content can be added to the existing one without the need to reload the page,


• different representational forms can be rendered from the same data without touching the server over and over and


• user interaction with the page can be refined far beyond what can be done in plain or modestly JavaScript assisted HTML

to mention just a few.

There are downsides too. Besides some technical challenges there was an obstacle of epic proportions: JavaScript generated content was invisible for the search engines condemning that shiny new website to the island of isolation and oblivion.

The problem lies in the way JavaScript rich site fakes URL history and in-site links. The only way to prevent browser reloading the whole page (until HTML 5) was to add content behind "#" which was intended for bookmarking the page. This region of url is called "the fragment" and is never sent to the server.

http://si.draagle.com/#!browse/group/root/

and

http://si.draagle.com/#!source/est/drug=esu&fact=est_cyy

are pointing to the same page from servers (and crawler's) perspective.

The obvious solution would be to build a different tree just for search engines and deliver them some stripped down version of the content. Unfortunately this tactic is commonly used by spammers and is brutally penalised by most search engines. Don't do it at home!

Google proposed a different deal. Site owners should modify their ajax link to include #! instead of just #. Whenever googlebot sees a #! in the URL, it considers it an ajax crawlable link and converts #! temporally to ?_escaped_fragment=, thus:

(1) http://si.draagle.com/#!drug/kxi/?sub=10

becomes

(2) http://si.draagle.com/?_escaped_fragment=drug%2fkxi%2f%3fsub%3d10

Note, how the fragment part got URL escaped.

Your part of a contract is to generate exactly the same content when goolgebot ask for (2) as the browser would generate for (1).

For almost two years the only big search engine providing ability to search ajax content was Google's and considering it's market dominance it wasn't such a big deal. Still, we must admit, Bing is a great search tool in many ways comparable to the big G and it offers sort of second opinion. It's market share has grown noticeably in last year and the new paradigms such as smart phones invasion and social searching might shuffle market shares of the search engines even more.

Few days ago, I noticed an important change in Bing's Webmaster tools, namely the Configure your site to have bingbot crawl escaped fragmented URLs containing #! check box.

The dilemma having your site in most of search engine's index vs providing superior user experience by applying the heavy artillery of JavaScript is diminishing fast. And this is good news equally for users and web builders.