MoonPoint Support Logo


Shop Amazon Warehouse Deals - Deep Discounts on Open-box and Used ProductsAmazon Warehouse Deals

Advanced Search
Sun Mon Tue Wed Thu Fri Sat

Sat, Apr 23, 2016 10:53 pm

Web Spiders Crawling Site on 2016-04-23

When I viewed a page on the site this morning, the page took a long time to load in my browser. I checked the Apache access log to see what it was showing in regards to site activity, since I had noticed several web crawlers, aka web spiders, accessing the site yesterday. Checking the site's Apache access log for today, I saw that it was being crawled by four web spiders simultaneously: baiduspider, bingbot,, and the linkdexbot. The first, baiduspider, is associated with the Chinese search engine company, Baidu, and the second, bingbot, is Microsoft's web crawler, which is used by its Bing search engine. The latter two, and linkdexbot are associated with search engine optimization (SEO) companies.

[ More Info ]

[/network/web/crawlers] permanent link

Once You Know, You Newegg AliExpress by

Shop Amazon Local - Subscribe to Deals in Your Neighborhood

Valid HTML 4.01 Transitional

Privacy Policy   Contact

Blosxom logo