If you use Google Analytics to monitor your site, Google provides a number of reports that can be used to find issues related to Google's crawling of your site, such as Google Index Status and Crawl Errors.
For instance, I often use Google to find articles on my own site when
I encounter a problem that I resolved previously while working on someone's
computer when I can't remember exactly what I did previously or the cause
of the same issue when I previously encountered it. I go to google.com and
spcify some keywords I know would be on the page or blog posting I created
and follow the keywords with site:support.moonpoint.com so
Google returns only results from my site. But I've recently found that
Google is reporting it has found no results for my site when I know it should
be finding them since I know the Googlebot has crawled the site subsequent
to my posting the entries on the site, some of which I posted quite some
time ago, and since it has shown me results that indicate it has
indexed material from the site only the day before. Checking the
Index Status report for my site, I found that the number
of pages indexed has declined considerably since this time
last year, whereas I haven't deleted pages and have added over
one hundred pages/blog entries since that time. Checking the Crawl Errors
report, I see quite a few "Not found" entries I need to fix,
but they don't explain the decline in the number of indexed entries,
so I know I need to do further investigation.
