 |
http://www.jafsoft.com/searchengines/webbots.html
John A. Fotheringham presents data in tabular form on the robots sent by search engines and other sites to read and index Web pages: their origins, names and IP addresses.
Keywords:
msproxy,
architectspider,
go!zilla,
linkalarm, cnet_snoop, linkwalker, msiecrawler, alexa, diibot, download wonder, ecatch, getright, getsmart, googlebot, gulliver, ia_archiver, linkbot, linkchecker, linklint, linkscan, linkverify, lycos_spider, netmind, nttdirectory_robot, search engine robots, slurp, netzip download, scooter, ultraseek, xenu`s link sleuth
|
 |
http://www.searchtools.com/robots/
Search Tools Consulting explains how the search engine programs called "robots" or "spiders" work, and reviews related sites.
Keywords:
|
 |
http://www.iplists.com/
Lists IP addresses of search engine spiders. Can be searched by IP address. Also links to resources on spiders.
Keywords:
|
 |
http://www.psychedelix.com/agents.html
Andreas Staeding`s large list of search engine spiders, similar Web robots, and Web browsers: their web-log identification and links to their originators.
Keywords: agent, bookmark, bot, bots, browser, check, checking, client, collector, crawler, database, download, e-mail, engine, favourites, fetch, grab, http, link, list, mail, manager, meta, referer, refferer, robots, search, search-engine, searchengine, spider, spiders, user, user-agent, useragent, validation, validator, robot
|
 |
http://www.sxw.org.uk/computing/robots/check.html
Online tool for syntax verification to robots.txt files, provided by Simon Wilkinson.
Keywords:
|
|
|
