« Wikipedia Guest Stars in Office Episode | Main | Sitemaps Protocol News / SEO Humor »
Del.icio.us Cloaking Update, More on Google Link Data
April 10, 2007
Last August, I wrote about how Del.icio.us was cloaking its robots.txt file, showing engines one version (which gave them full access) and showing users another (which appeared to restrict crawling and indexing). In addition, it was showing a set of robots meta tags to users, but not showing them to regular users.
Here's an example of what Del.icio.us was doing back then, at the page meta tag level:
Following is the famous meta tag from the Del.icio.us "SEO" tag page - the meta tag that makes everyone think the page won't be crawled:
But if you set your user-agent to Googlebot, here's what you see:
Since then, Del.icio.us has stopped one of these two techniques. The site still cloaks at the page level -- showing the robots meta tags above to users, but not to engines. But the robots.txt issue (discussed in the first paragraph above) has been changed. Now everyone sees the same version, with all major engines given these crawling parameters:
Allow: /
Disallow: /inbox
Disallow: /subscriptions
Disallow: /network
Disallow: /search
Disallow: /post
Disallow: /login
Disallow: /rss
The /subscriptions and /rss lines above, for those keeping score, are new since August.
Also note that Del.icio.us has used the "nofollow" link attribute for quite a while -- possibly since its inception. As a result, the cloaking matter is moot to many people, because to them, who cares if a page is crawled or indexed if the OBL aren't given any weight anyway?
The other reason I'm writing about Del.icio.us today is due to a comment on a recent post about Google Webmaster linking data. Offhandedly, I mentioned to "remember that Google reports nofollowed links" in its reports of incoming links to specific URLs, and I'm not sure a lot of people realize this.
(Important: Now, the "nofollow" I'm talking about is the link attribute, not the robots meta tag.)
So let me rephrase:
Just because Google sees and reports a link coming into your site does not mean that link does you any good.
As an example, I've looked through many Google link reports and gone to the specific page linking in to our site or our clients' sites. Links such as the following will show up in Google link reports, but according to everything Google has said over the past two years, the links aren't helping you:
- Del.icio.us
- Stumbleupon
- Links from comments and signatures from any blog/forum site that utilizes "nofollow"
- etc.
So again, don't take those linking reports at face value, at least to the point of making an assumption that all links are beneficial, even when the site they come from is highly respected and authoritative. Certainly, they're important for the potential traffic, but not for building your site's link popularity.
All posts by Erik Dafforn
posted by Erik Dafforn at April 10, 2007 10:56 AM
Intrapromote: [ Case studies | SEO services | Bios ]
Trackback Pings
To TrackBack this entry, use the following URL:
http://seoblog.intrapromote.com/mt-tb.cgi/387
Comments
I understand links in to a site are not always beneficial, but I was wondering whether they could be detrimental. I would love your opinion on this.
Recently I came across a site that had published a portion of a story of mine, and included a link back to my blog without my consent.
Upon further inspection of the site I realized their content is (I'm assuming) delivered without much human input (the site is a mess of font sizes, colors, advertisements, and I suspect other yuck things :).
Is this typical of the Web now, and, if it is, what can writers do to minimize the effects (if any) of this type of duplicate content? Also, how are search engines looking at this?
Thanks!
Posted by: Lidija Davis at April 26, 2007 08:31 AM