« Keywords In URLs - Not Just For Search Engines | Main | Google Keyword Association »
Del.icio.us Leaves a Bad Taste
August 30, 2006
If this has been covered already, let me know. If so, I'll graciously provide attribution.
"Social tagging," the process of users sharing bookmarks and feedback about specific sites and pages, is near the top of the list of cornfields on which SEOs are trying to erect slick new subdivisions.
As social media sites have gained popularity, many SEOs have lamented the fact that Del.icio.us uses the robots meta tags nofollow, noindex, and noarchive as a way to avoid spam. If links don't pass popularity, then they won't be abused, so the theory goes. (Don't confuse this nofollow with link attribute nofollow.)
This has left many people wondering why a query for [site:del.icio.us] shows about a million and a half pages indexed, and why the site ranks for queries like [seo] and [popular]. Some people believe it's due to incoming linkage and Google's tendency to show URLs in results even though Google has been told not to index them.
For better or worse, the truth is much simpler. Google was never told to not index Del.icio.us pages. YOU were told that GOOGLE was told not to index pages. But Google? They never got the message, because Del.icio.us has been using user-agent delivery (yes, cloaking) to tell you one thing, and engines another.
Following is the famous meta tag from the Del.icio.us "SEO" tag page - the meta tag that makes everyone think the page won't be crawled:
But if you set your user-agent to Googlebot, here's what you see:
Where did those highlights go? Only her hairdresser knows for sure.
The robots.txt file for the site is no different. Here's the file for standard user-agents:
I left some extra whitespace in the screen shot to show that nothing follows the code lines.
User-agents Googlebot and Slurp each get additional lines in their versions of robots.txt. Following is what Google sees:
What annoys me about this process is not that Del.icio.us is trying to put one over on Google or Yahoo. (The latter would be especially odd, given that Yahoo owns Del.icio.us), but that Del.icio.us is trying to put one over on YOU. Certainly Google and Yahoo know what's going on. Millions of pages don't magically appear when valid noindex tags are in place. Del.icio.us wants to be a popular destination, wants its search engine rankings, but it doesn't want all the riff-raff that popularity brings. Old-school cloaking that a 10-year-old could detect isn't a way to achieve that.
All posts by Erik Dafforn
posted by Erik Dafforn at August 30, 2006 04:52 PM
Intrapromote: [ Case studies | SEO services | Bios ]
Trackback Pings
To TrackBack this entry, use the following URL:
http://seoblog.intrapromote.com/mt-tb.cgi/277
Comments
"(Don't confuse this nofollow with link attribute nofollow.)"
I know one is global and one is used for individual links. But end result, what is the difference? Both tell robots not to follow the links.
The reason I bring this up, is because del.icio.us still has the "nofollow" link attribute on the individual links, regardless if your user agent is "normal" or Googlebot or whatever. So the link juice still doesn't pass, regardless of the meta tag.
I know that isn't what you are getting at with the post. Del.icio.us IS still using cloaking to (attempt to) rank in search engines, BUT I think your post kind of skips over the fact that link juice is still NOT passed even though Google doesn't see the NOINDEX, NOFOLLOW meta tag. Just wanted to clear that up, in case anyone else was confused for a second, like myself!
Anyway, great catch on the cloaking. Very interesting read.
Posted by: Jason at August 31, 2006 09:49 AM
Jason,
Excellent point about Delicious NOT cloaking the nofollow link attribute - that certainly changes the thrust of the issue, because link pop is not passed. Google's cache of Delicious pages confirms that the nofollow link attribute does remain intact.
As for the nofollow meta tag and the nofollow link attribute having the same effect, that's not my understanding. As I understand it, the nofollow meta tag tells the bot to literally not crawl the target page, while the nofollow link attribute does NOT instruct the bot to avoid crawling the link, but instead, tells it to merely not pass link popularity (or PR, or however you want to think about it).
So having a nofollow meta tag on a page does (or should) put a stop to indexing pages linked from that page, while having the nofollow link attribute enables indexing of links on the page but does not allow them to pass popularity.
So what does this leave us with? People can still take popular (i.e., well ranking) Delicious pages, tag their own content so that it appears on the page, and enjoy some trickle-down traffic from that. Sites can also use Delicious as a secondary method of getting their own deep content indexed, because bots WILL crawl the links on the Delicious page, because they don't see the nofollow meta tag like everyone thinks they do.
Both of these seem like fairly high effort-to-results ratio activities.
All of which still makes me wonder (more), what is the purpose of the cloak? What does Delicious gain from it, other than some sort of ill-gotten reputation for cleanliness?
Posted by: Erik Dafforn at August 31, 2006 11:41 AM
Erik, check Matt Cutts on the nofollows (at least with respect to Google) -- it's a very valuable post, IMHO.
As far as purpose goes, it has to be for ranking (and an attempt to build traffic) like you said... I can't think of anything else that makes sense, given who owns del.icio.us.
Still discussion-worthy, regardless!
Posted by: Jason at August 31, 2006 12:09 PM
Good points. I'm not sure why I'd split up the respective duties of the two nofollow versions. I think I must have read something early on that I took for gospel - but shouldn't have.
Posted by: Erik Dafforn at August 31, 2006 12:23 PM
Yes more and more social bookmarking sites are using no-follows and also its starting to happen in article directories now also. I think it is a shame as many uses this for help not only getting indexed but also building links. There will always be tools to take advantage of this but i think the genuine users of the social bookmarking sites will lose out. As far as your cloaking info on how they are doing it thank you.
Posted by: Michael Francis at September 9, 2007 08:50 AM
The official claim is that links with the rel=nofollow attribute do not influence the search engine rankings of the target page. In addition to Google, Yahoo and MSN also support the rel=nofollow attribute.
i think it helps indexing
Posted by: John Illnes at April 10, 2008 04:56 AM