MoonPoint Support Logo

 


Shop Amazon Warehouse Deals - Deep Discounts on Open-box and Used ProductsAmazon Warehouse Deals



Advanced Search
April
Sun Mon Tue Wed Thu Fri Sat
         
23
2016
Months
Apr


Sat, Apr 23, 2016 10:53 pm

Web Spiders Crawling Site on 2016-04-23

When I viewed a page on the site this morning, the page took a long time to load in my browser. I checked the Apache access log to see what it was showing in regards to site activity, since I had noticed several web crawlers, aka web spiders, accessing the site yesterday. Checking the site's Apache access log for today, I saw that it was being crawled by four web spiders simultaneously: baiduspider, bingbot, MegaIndex.ru, and the linkdexbot. The first, baiduspider, is associated with the Chinese search engine company, Baidu, and the second, bingbot, is Microsoft's web crawler, which is used by its Bing search engine. The latter two, MegaIndex.ru and linkdexbot are associated with search engine optimization (SEO) companies.

[ More Info ]

[/network/web/crawlers] permanent link

Once You Know, You Newegg AliExpress by Alibaba.com

Shop Amazon Local - Subscribe to Deals in Your Neighborhood

Valid HTML 4.01 Transitional

Privacy Policy   Contact

Blosxom logo