When I viewed a page on the site this morning, the page took a long time to load in my browser. I checked the Apache access log to see what it was showing in regards to site activity, since I had noticed several web crawlers, aka web spiders, accessing the site yesterday. Checking the site's Apache access log for today, I saw that it was being crawled by four web spiders simultaneously: baiduspider, bingbot, MegaIndex.ru, and the linkdexbot. The first, baiduspider, is associated with the Chinese search engine company, Baidu, and the second, bingbot, is Microsoft's web crawler, which is used by its Bing search engine. The latter two, MegaIndex.ru and linkdexbot are associated with search engine optimization (SEO) companies.
[ More Info ]
