magnifying glass

Image via Wikipedia

I got an e-mail recently from someone very concerned that his product catalog does not have links to every page on his site from his sitemap, although he has a complete link structure emanating from his home page. Should he beat his programmer about the face and body until he creates a proper sitemap?

Seems like we've been getting a lot of questions on sitemaps recently, and I'll try to cover different ground than Diane Aull covered recently.

Diane talked about whether it's a problem if you're missing a sitemap (and the bigger problem in getting scammed to create one), but my reader had a different question. He has an old sitemap out there that contains 40 or so links, but his catalog now has 60,000 pages. Will this screw everything up?

First off, a sitemap is not required for search engines to crawl your site. If your pages are designed properly, Google and friends will happily add your pages to their indexes.

But what about a sitemap that's plain wrong, like this one? What happens then?

There's an easy way to figure out if you've got a problem. Check to see if your pages are indexed. If those catalog pages are already being indexed, then you have nothing to be concerned about, so just delete the sitemap from you server and don't give it another thought.

But do you know how to tell if your pages are indexed? You need to check each search engine separately, which you can do with the site: operator. Search for "" and you'll see all the pages indexed. If you have 60,000 pages, like my buddy, the best you can do is spot-check with this method by using the site: operator naming specific pages in different parts of your site.

For Yahoo!, there's a better method, Yahoo! Site Explorer. Like the site: operator, you can see how many pages are indexed, but you can also export up to 1000 pages to an external file for more analysis. By working your way through the search engines' results lists, you can see which pages are indexed and which are not.

So, rather than becoming fixated with sitemaps, keep your eye on the ball. Test to see if your pages are indexed in each of the search engines you are targeting. If you have pages missing, there are many things you can do to fix the problem--that's when you start considering a sitemap.

Reblog this post [with Zemanta]

December 21, 2008

Mike is an expert in search marketing, search technology, social media, publishing, text analytics, and web metrics, who regularly makes speaking appearances.

Mike's previous appearances include Text Analytics World, Rutgers Business School, SEMRush webinar, ClickZ Live.

Mike also founded and writes for Biznology, is the co-author of Outside-In Marketing (with James Mathewson) and the best-selling Search Engine Marketing, Inc. (now in its 3rd edition, and sole author of Do It Wrong Quickly, named by the Miami Herald as one of the 11 best business books of 2007.


i guess if the site's well structured and probably e.g site wide links helps, then a sitemap is not essential. still, it's one of those things that GWMT seems to advocate to further ensure that pages are indexed. Still, it's effectiveness likely pales in comparison to getting quality backlinks.

Site maps are essential as an optimization technique and should always be linked to from your index page or home page. It allows you to help the search engines to clearly understand what your web site is about and presents some unique opportunities in optimization. So make sure that no broken links for your sitemap.

Comments closed after 30 days to combat spam.

Search Engine Guide > Mike Moran > Can a Broken Sitemap Hurt You?