More pages in the search index means more chances to be found. So maybe you never really have enough pages--since every extra page is a lottery ticket in the search sweepstakes. You've gotta be in it to win it. But this does not really offer us any answers. There are obviously some amount of pages that seem OK and other amounts that are probably Bad, like, zero would be bad. How do you determine how many pages you have in the search index, and if your number is enough?
First off, you need to understand that there is no single search index-each search engine has its own search index. Google has its own, Bing has its own, and so do many other search engines. So, you need to know which search engines are worth worrying about-in the U.S., it's Google and Bing.
So how do you find out how many pages are in Google's index and how many are in Bing's?
Both Google and Bing have a tool called the "site:" command. You can just enter into each one the word "site:" along with your domain name (Such as "site:biznology.com"). For some sites, this handy command works just fine and you can see how many pages are stored in each index. If your results look right, great. But sometimes the results just look nuts. For example, "site:ibm.com" yields 2.8 million pages on Bing but a crazy 12.2 million pages on Google.
To avoid such inaccuracies, use each search engine's Webmaster Tools sites. Both Google and Bing will tell your Webmaster exactly how many pages are in the index and will even let you know which pages they are having trouble grabbing. It's possible that the IBM Webmaster is aware that there actually is a big discrepancy between Google and Bing, which might be just fine or might be something they are working on.
I've spoken to a few experts and they have varying theories. One told me that Bing stops crawling when more than 1% of the pages get errors-the Bing Webmaster site will clue you in on this. Another speculated that Bing is only returning counts of pages that get search visits, not every page in their index. No one I spoke with knew for sure why this is happening, but it shows you the importance of checking your numbers.
Likewise, big swings in indexed pages (1,000 pages indexed in Google today vs. 5,000 yesterday) mean that you should look into it. And, in general, an inclusion ratio (pages indexed divided by actual pages) below 70% is something that should give you pause, although with these Bing errors who knows what a good inclusion ration is for Bing right now.
Regardless. knowing how many pages are indexed is the first step to seeing if you have a problem.
Originally posted on Biznology
Mike is an expert in search marketing, search technology, social media, publishing, text analytics, and web metrics, who regularly makes speaking appearances.
Mike's previous appearances include Text Analytics World, Rutgers Business School, SEMRush webinar, ClickZ Live.
Mike also founded and writes for Biznology, is the co-author of Outside-In Marketing (with James Mathewson) and the best-selling Search Engine Marketing, Inc. (now in its 3rd edition, and sole author of Do It Wrong Quickly, named by the Miami Herald as one of the 11 best business books of 2007.
Copyright © 1998 - 2017 K. Clough, Inc. All Rights Reserved. Privacy