« The Hunt for the Perfect SEO: Testing Knowledge vs. Testing Patience | Main | A Universal Truth from Yahoo Site Explorer »
Diagnosing Crawling and Indexing Issues with Yahoo Site Explorer
October 12, 2005
Yahoo recently released the Yahoo Site Explorer, a very helpful way to diagnose issues that may be hampering your performance in Yahoo natural search.
Enter a URL in the Search box, and Yahoo returns two values: First, the number of indexed pages for a given site, and as a second option, the number of incoming links pointing at that site. The following image shows the location of the key data points.
Drilling down, you can select specific URLs from the results page, and "explore" those pages in depth - finding, for example, the number of pages from a specific section of your site that have been indexed, or the incoming links pointing to a specific page of your site. To find an index count for a specific site section, enter or click a URL such as http://www.site.com/press/. This returns indexing and linking results for this specific URL, as well as any pages in the /press/ directory.
Unlike Google, which purposely returns only a percentage of a site's incoming links, Yahoo Site Explorer claims to show all incoming links that it knows about. This can come in very handy when performing a competitive link analysis for sites in your industry.
One of Site Explorer's largest drawbacks is the ability to download only the first 50 results into TSV format, for import into programs like Excel. It would be wonderful to have an entire site's worth of data to sort and play with in a spreadsheet program, but it's unlikely that Yahoo is too eager to spend processing time creating TSV files with tens of thousands of rows. A resourceful programmer named John Mueller has used the Yahoo Search API to create a custom version of Yahoo Site Explorer that overcomes some of these common obstacles; it's worth a look.
If you find large blocks of URLs from your site that Yahoo has not indexed, it offers a submission system similar to that of Google Sitemaps. You can simply fill a text file (such as pages.txt) with the URLs you want Yahoo to index, separated by a hard return. Upload the text file to your web server, then submit the entire URL of the text file (such as http://www.site.com/pages.txt) to the Yahoo Free Submit page.
We don't yet have significant data on the crawl rate for URLs submitted via this method, but we'll be sure to publish any information we collect.
All posts by Erik Dafforn
posted by Erik Dafforn at October 12, 2005 06:39 PM
Intrapromote: [ Case studies | SEO services | Bios ]
Trackback Pings
To TrackBack this entry, use the following URL:
http://seoblog.intrapromote.com/mt-tb.cgi/80