Image via Wikipedia

I got an e-mail recently from someone very concerned that his product catalog does not have links to every page on his site from his sitemap, although he has a complete link structure emanating from his home page. Should he beat his programmer about the face and body until he creates a proper sitemap?

Seems like we've been getting a lot of questions on sitemaps recently, and I'll try to cover different ground than Diane Aull covered recently.

Diane talked about whether it's a problem if you're missing a sitemap (and the bigger problem in getting scammed to create one), but my reader had a different question. He has an old sitemap out there that contains 40 or so links, but his catalog now has 60,000 pages. Will this screw everything up?

First off, a sitemap is not required for search engines to crawl your site. If your pages are designed properly, Google and friends will happily add your pages to their indexes.

But what about a sitemap that's plain wrong, like this one? What happens then?

There's an easy way to figure out if you've got a problem. Check to see if your pages are indexed. If those catalog pages are already being indexed, then you have nothing to be concerned about, so just delete the sitemap from you server and don't give it another thought.

But do you know how to tell if your pages are indexed? You need to check each search engine separately, which you can do with the site: operator. Search for "site:yourdomain.com" and you'll see all the pages indexed. If you have 60,000 pages, like my buddy, the best you can do is spot-check with this method by using the site: operator naming specific pages in different parts of your site.

For Yahoo!, there's a better method, Yahoo! Site Explorer. Like the site: operator, you can see how many pages are indexed, but you can also export up to 1000 pages to an external file for more analysis. By working your way through the search engines' results lists, you can see which pages are indexed and which are not.

So, rather than becoming fixated with sitemaps, keep your eye on the ball. Test to see if your pages are indexed in each of the search engines you are targeting. If you have pages missing, there are many things you can do to fix the problem--that's when you start considering a sitemap.