At this point in the game, we are all well aware that obtaining good Google rankings early in your search engine optimization campaign is a highly important variable to launching your campaign. With regards to non-monetary search engine submissions, Google is one of the few left, but also one of the fastest for inclusion times. As with most things, with this come risks and potential problems that webmasters and SEOs need to bear in mind while in production and pre-launch phases of the campaign. While we do know that Google spiders before and after certain phases, we are not certain at what point in the month they will do their spidering and update their database. Below we will discuss what the Google cycle entails, when and how to catch Googlebot at the right time, and what this means for your search engine optimization campaign.

The Google Cycle

While the Google update cycle is well documented, over the past year this cycle has become less of a pattern and more of a shot in the dark for those who await the update. The cycle begins with Google doing a major crawl (crawl 1), or spidering, which is when it sends Googlebot out to spider the current sites within its database (DB) as well as to find new websites that have been put online. Once Google has completed this crawl, grabbing all of the webpages for its next update, there will be an update shortly (roughly two weeks). Google will then update its database, showing the new results on www2.google.com and www3.google.com. During the update, the results are often shuffled between the primary database and the second and third database, and since Google uses over 10,000 servers, people in all areas of the world are usually seeing different results until the dust completely settles from the update. The update will continue for a few days, but usually no longer than a week in duration.

During and directly following the database update, Google will again begin heavy crawling (crawl 2), or spidering, of the existing websites in its database and new websites that have been launched. After this crawl by Googlebot, the cycle returns to the beginning and starts all over again for the next month.

Catching Google and Googlebot at the Right Time

To have your website included in the Google database, or have your site's updates reflected in the database, as soon as possible, you need to do some planning and preparing so that you can catch Googlebot at the correct position in the cycle. We know that there is the initial Googlebot crawl in the beginning of the month, as well as a crawl during and directly after the update.

So, if you wish to have a new website included in the DB, will either of these crawls insure your inclusion into the database? It has been seen that this is not always the case. Mainly, if your website is crawled in the beginning of the month, the chances are that your website will not be included in that months update. If your website is crawled during the second crawl of the month, which is directly following the update, it is possible that your website will be revisited in the next crawl and then included in the next update. Other times Google will visit a new site and grab only the robots.txt and the homepage. This is a good indication that Googlebot will be back during the next major crawl and your website will be included in the update following that second crawl. So, looking back, it seems that for your new site to be included in the Google database it will take two visits from Googlebot. This is true for most cases.

To get the fastest inclusion time possible, there are a few things that you can do. If your website is crawled for the first time by Googlebot during or directly after the update, then you are in good shape as it is more than likely your website will be included in the next month's update. If your website is not crawled at this point, but during the next crawl, you will have to wait even longer for your website to be included in Google's database. So, how do you get Googlebot to crawl your website during that specific time period? You can either hope that it will happen this way (not very reliable, is it?), or you can do your homework and plan it out. If you have other websites that are in the Google database, you can watch the crawling and update dates and then plan your launch accordingly. If you don't have any websites in the Google database that you can monitor, you can either watch Google.com for the updates, or you can read about crawling and update times at WebMasterWorld.com's Google forum.

While there is almost no way to be 100% sure that your website is going to be crawled, there are certain things that you can do to get Googlebot's attention and attract the robot to your site. The first thing to do is obtain links to your site from other websites with a high PageRank. The higher the PageRank of a website, the more that website will be crawled and refreshed by Google, which means your link will be picked up more quickly. Secondly, you can submit your website to Google through their Add URL page. This is not a definite way into the database, and is not 100% reliable. The third thing that you can do is install the Google Toolbar, and then visit your own website through the toolbar. There have been numerous reports of a direct correlation between a website's inclusion into the database and a visit through the Google Toolbar. While it will cost you $299 annually, a Yahoo! directory listing is also a help in getting into Google's database, and Yahoo! offers quick inclusion times (7 days) into their directory. An Open Directory Project listing is also a very good way to have your website included in the database, although this can sometimes take long periods of time, and is not 100% reliable.

How This Affects Your SEO Campaign

The information that is available to SEO professionals and webmasters regarding Google's crawling and update patterns can unquestionably help in planning and executing our search engine optimization campaigns. Not only can it help drastically with the above, it can also help in our schedules as updates and new developments need to be launched online by a certain date and time to be included in the database. As Google commands a high percentage of search engine referrals and traffic, having a rough idea of when this will kick in can he a large help.