Another Google Blackballing?

Hi Jill,

Is it common for Google to index a site very quickly (almost 2 days) after being submitted and then kick it out of the index a little bit later?

I have one site that I submitted and after being indexed almost immediately it was kicked out a week later. I waited for 6 months and then moved the site to a dedicated server. I re-submitted it and it was indexed again quickly but a week later it was kicked out again. Has this ever happened to you?

I know it has inbound links and the pages are optimized without spam. I can't figure out if this has to do with the program I use to submit, WebPosition Gold, or if something else causes this. I use that program to submit all (maybe 40 so far) of my sites and this has never happened to any of them. I am afraid to submit it again because I don't want Google to think I'm spamming them, but I have to do something to get this site indexed and keep it in there. If you have any advice I would love to hear it.

Thanks,
Anthony


Jill's Reply

When I first read Anthony's email, I didn't read it carefully and sent him my somewhat standard reply as follows:

"Yes, that's exactly how Google works. In fact, submitting to Google does absolutely nothing. Google finds pages to add to its database by crawling the Web through links. As long as there are links to your pages, Google will find them and add them. If there are no links and you submit, your pages won't stay in the database long. Even if there are links to your site, the pages may still disappear and reappear for 4 - 6 weeks. After that (assuming you have high-quality links pointing to your site) it should stabilize.

"If you're submitting through an automated program such as WebPosition Gold, you should immediately cease this. It won't help you in the least, and since it's against their terms of service, it can definitely hinder your listings."

However, this morning while looking for something to write about in today's newsletter, I reread Anthony's question and realized he said that this had been happening to him for 6 months! That is definitely not normal, considering he claimed to have links to his site.

So, being curious (and wanting to further procrastinate doing my writing!) I started to do some digging. He had given me the URL to his site, and the first thing I did was take a look at it. It was for a major hotel in Hilton Head, and the site was your typical hotel site. It looked nice and didn't appear to have anything spammy on it.

A good percentage of the time, when someone tells me that their URL isn't indexed, it actually is indexed, so I was almost certain that this one would be too. First I right-clicked my mouse on the home page and tried to view Google's cache of it. (You have to have the Google toolbar installed to do this.) Hmm...nope, Google said it didn't have it. So I went to Google and typed in the domain name. Nope, still nothing. Then I tried an allinurl:domain-name.com search in Google (substituting the actual domain name for "domain-name" of course!). This one did bring up a bunch of pages within the site, but still not the home page.

Typically when a page's URL doesn't show up at all in Google's database, it's because there's some sort of mirror site or other duplicate content that is showing up instead. So that's the next thing I always check in cases like this. To do this, you simply copy and paste a random group of words (or a full sentence) from the page in question into Google's search box surrounded by quotation marks. For this page, I put in the following phrase from the site: "draped oak tree, the deep rough or the rim of a".

Bingo! (You can try it yourself to see what came up.) What showed up was some really weird, long URL that was definitely not the same as Anthony's hotel domain, yet it had the same Title and snippet from the page. When I viewed Google's cache of the URL, I saw that it was exactly the same site (minus the graphics which weren't showing up). When I clicked on the actual URL, I was immediately redirected to Anthony's actual site.

So now the mystery was partly solved. Most likely, since the weird URL was already indexed, the real page appeared to simply be a duplicate, and that's why it was not getting indexed.

I couldn't tell exactly what was happening on the long-URL page because the redirect was happening so quickly (sometimes I can catch it by hitting my "esc" key really fast, but it wasn't working this time). I was still curious, so I checked the long URL in an "Lynx" browser. This is an old trick to get a rough idea of what a search engine sees. (You can view URLs this way here: http://www.delorie.com/web/lynxview.html.)

Aha! All it showed me was the following:

REFRESH(0 sec): http://www.the-actual-domain-name.com

A server header check (which can also be done at that delorie.com site) showed a normal 200 OK response instead of a 301 permanent redirect response, which I believe was falsely telling Google that this was the right URL for that page.

I was pretty sure that the existence of that weird URL page was the reason why the real URL could not seem to stay in Google's index. So I wrote Anthony back with my findings, and he immediately discovered who owned the URL and contacted them about it. They told him about a page that their hotel clients have which use those long URLs with redirects on them for tracking purposes here: http://www.wliinc3.com/cgi/foxweb.dll/cwlink/direcatn?catid=99&client=HHICC

Just as I thought this mystery was solved (I was already writing it up here), I started wondering about the other links on that page above. What about the other redirected URLs on that page? Were they all having the same problem? Strangely enough, they weren't. In fact, they were doing different things. For some of the links, Google was showing the right page indexed and not the weird URL, but for others, it had them both indexed. And for others still, Google realized that the redirected URL was the very same page as the resulting real URL. I could tell this because when I clicked to view the cache of the long URL, Google said, "This is Google's cache of http://www.the-real-domain.com and not "This is Google's cache of http:///www.weird-long-url.com."

I haven't figured out why Anthony's real URL is not indexed and these other ones are. It's possible that it all comes back to the fact that he was submitting it automatically via WPG (although I doubt it). That along with the redirect, which may have looked spammy, possibly could have gotten it flagged. Or it could simply be another case of Google getting it wrong. They mix up redirected URLs all the time.

It's really hard to say for sure. Hopefully, Anthony can get rid of that redirected link altogether and eventually get the actual URL indexed! At any rate, it gave me something to do this morning, and some interesting fodder for the newsletter. I always love trying to solve these kinds of mysteries! And if worse comes to worst, Anthony can email Google with this article, and maybe they'll straighten things out by hand.
February 26, 2004