SEO Speedwagon

You might have noticed that Google rolled out sitelinks for a new batch of sites a couple weeks ago. This blog was included in that batch, as you can see if you do a query for [seo speedwagon].

The goal here isn't to beat up on Google, but I think it's significant enough that site owners should be aware of it. In a couple cases, the sitelinks that Google shows (or showed) for our site have been links specifically excluded from robots, either via robots.txt or by the "noindex" attribute in the robots Meta tag. Following is a screen shot of the [seo speedwagon] query taken on February 26, which is roughly when the new batch of sites started noticing their sitelinks:

Note the two red-outlined links. The one in the left column, ip login, is our staff login page. It's been excluded by our robots.txt file for almost three years. Coincidentally, Google couldn't index that page if it wanted to, as it's password-protected. I know that robots.txt exclusion isn't a totally reliable way to keep a URL from showing up in SERPs, as it often causes what's known as a "partially-indexed" URL (example). But come on -- a Sitelink?

The outlined link in the right column (November 2007) is a typical (if capriciously chosen) monthly archive page -- exactly the kind you see in the third column of this blog. They're ugly, more or less useless (both for SEO and for people), and I'll probably eventually do away with them, but for now, there they are. But the important thing here is that I added the robots "noindex" tag to them well over a year ago.

Just this week, Google changed the format slightly. Here's a current shot:

The November 2007 link (excluded via Meta tag) is now off the list (automatically -- I didn't do it), but the ip login link remains.

Yes, I know I could block specific sitelinks from within Webmaster Tools. And I might, but I wanted to show it to you first.

It seems like excluding specific URLs via robots.txt or via the robots meta tag should be a sufficient method of opting URLs out of sitelinks.

This topic is especially timely as Matt Cutts just recently asked users how they'd prefer that a meta-tag-excluded URL appear -- if at all -- in the Google index. As of this writing, 83% say "Don't show a link at all." I don't want to speak for his readership, let alone all site owners, but I can confidently predict that most people don't want a robots-excluded URL (regardless of whether the exclusion mechanism was robots.txt or a robots "noindex" Meta tag) showing up in a Sitelink.