123people|123people|
200please|200PleaseBot|
80legs|www.80legs.com/webcrawler|
@mail.ru|Mail.ru|
A6 Corporation|A6-Indexer|
Abonti|Abonti|
AboutUs|AboutUsBot|
AboutUs|www.aboutusbot.org|
AhrefsBot|ahrefs.com/robot/|
AideRSS|aiderss.com|
aiHit|aiHitBot|
Affectv|AffectvRobot|
AgHaven|aghaven|
Ahrefs|AhrefsBot|
Alexa|ia_archiver|
AlexaBoostup|AlexaBoostup|
AltaVista|Scooter-|
Alyze.info|alyze.info|
Ant.com|AntBot|
Apercite|apercite.fr/robot|
Applebot|www.apple.com/go/applebot|
Apple PubSub|Apple-PubSub|
Apple Syndication|AppleSyndication|
Applebot|compatible;Fetcher/0.1|
Applebot|Applebot/0.1|
Applebot|Applebot/0.3|
AraTurka|Bot.AraTurka.com|
Archive.org|archive.org_bot|
Ask.com/Teoma|Jeeves/Teoma|
Avira|SafeSearchmicrodatacrawler|
BackType|kame|
BackType|support@backtype.com|
Baiduspider|Baiduspider|
BecomeBot|BecomeBot|
Best Buy Ware Bay|WBSearchBot|
Bibliotheque Nationale de France|bnf.fr_bot|
Bing|www.bing.com|
Bing|bingbot|
Bing|MSMOBOT|
Bing|msnbot|
Bit.ly|bitlybot|
Biz360|Biz360|
Blekko|Blekkobot|
Blogged|blogged_crawl|
Bloggsi|Bloggsi|
Bloggsi|http://bloggsi.com/|
Bloglines|Bloglines|
Bloglovin|Bloglovin/1.0|
Bloglovin|bloglovin/metadataparser|
BlogPulse|BlogPulse|
BlogScope|BlogScope|
BlogScope|www.blogscope.net|
BlogShares.com|Zipping Along V|
BlogsNow|BlogsNowBot|
Bluecoat proxy|Bluecoat|
Bookmark Sync2it|sync2it.com|
BotProoXi|BotProoXi|
Brandwatch Magpie-Crawler|brandwatch.net|
Browsershots|Browsershots|
Browsershots|Browsershots.org|
BUbiNG|BUbiNG|
BuzzLogic|BuzzLogic|
BuzzLogic|www.buzzlogic.com|
Camdiscover|Camcrawler|
Camdiscover|www.camdiscover.com|
CatchBot|CatchBot|
CazoodleBot|cazoodle|
Cazoodle|Cazoodle|
CCBot|CCBot|
CCBot|commoncrawl.org/bot.html|
ChekParams|DCPbot|
Chlooe|bot-pge.chlooe.com|
Cityreview|cityreview|
Cliqz|CliqzBot|
Cloud Server Market|CloudServerMarketSpider|
CMS Crawler|www.cmscrawler.com|
Coc Coc|coccoc|
CoinCorner|CoinCornerBot|
Commentluv|Commentluv|
Commentluv|http://www.commentluv.com|
Comodo Web Inspector|Comodo-Webinspector-Crawler|
Coriolis|www.coriolis.ch|
Corpora from the web|berlin-fu-cow|
Covario|Covario-IDS|
crawler4j|crawler4j|
CRAZY Web Crawler|CRAZYWEBCRAWLER|
crowsnest|crowsnest|
Cuil|Twiceler|
cURL|curl/|
datagnion|datagnionbot|
DataSift|TweetmemeBot/4.0|
Dazoo|Dazoobot|
DictaNova|DictaCrawler|
discobot|discobot|
Discovery Engine|discoverybot|
Discovery Engine|disco/Nutch|
Discovery Engine|discobot|
DoCoMo|DoCoMo|
DomainCrawler|DomainSigmaCrawler|
DomainOptima|OptimizationCrawler|
DomainSigma|DomainSigma|
DotBot|DotBot|
DuckDuckGo|DuckDuckGo-Favicons-Bot|
eCairn|eCairn-Grabber|
EasyBib|EasyBibAutoCite|
Echo|JS-Kit|
emefge|emefgebot|
Embedly|Embedly|
EnaBall|EnaBot|
Entireweb|entireweb.com|
embed.ly|Embedly/0.2|
EveryoneSocialBot|EveryoneSocialBot|
Exabot|Exabot|
Exalead|NG/2.0|
Ezooms|ezooms.bot@gmail.com|
Everyfeed-spider|everyfeed)|
EveryoneSocialBot|EveryoneSocialBot|
Facebook|Facebot|
Facebook|facebookexternalhit/|
Fastsearch|FASTMetaWebCrawler|
Fastsearch|fastsearch|
FeedBot|FeedBot|
Feedspot|feedspot.palkeo.com|
FeedBurner|FeedBurner|
Feed::Find|Feed::Find|
Feedly|Feedly/1.0|
Feedly|FeedlyApp/1.0|
Flipboard RSS Crawler|FlipboardRSS|
FlipBoard|flipboard.com/crawler|
Feedster|Feedster|
Feedster|Feedster Crawler|
Feedzirra|feedzirra|
Findxbot|Findxbot|
flipboard|http://flipboard.com/browserproxy
FriendFeed|FriendFeedBot|
Gaisbot|Gaisbot|
Geliyoo|GeliyooBot|
Galbot|www.galbot.com|
GG Network|GGPeekBot|
GIDNetwork|GIDBot|
Generic libwww-perl|libwww|
Gigablast|Gigabot|
Gluten Free Crawler|GlutenFreeCrawler|
Gigabot|Gigabot|
Google AdSense|Mediapartners-Google|
Google Reader|Feedfetcher-Google|
Google Images|Googlebot-Image|
Google Mobile|Googlebot-Mobile|
Google Desktop|GoogleDesktop|
Google Feedfetcher|Feedfetcher-Google|
Google|googlebot/|
Google|Googlebot/|
Google Images|Googlebot-Image|
Google Labs|Python-urllib|
Google Adsbot|AdsBot-Google|
Google Sitemaps|Google-Sitemaps/|
Google Page Speed|GooglePageSpeed|
google Newsstand Producer|GoogleProducer|
Google|Googlebot|
GrapeShot|GrapeshotCrawler|
Grub crawler|Grub.org crawler|
Grepler|Greplerbot|
GTmetrix|GTmetrix|
Guruji|GurujiBot|
Harry|Harry|
Hatena|Hatena|
HatenaScreenshot|HatenaScreenshot|
Heritrix|heritrix|
Hive Spy|HiveSpy|
HMSE_Robot|HMSE_Robot|
Horizon|Horizon|
Hourlypress|hourlypress|
ICC-Crawler|ICC-Crawler|
Icerocket|BlogSearch|
ichiro|ichiro|
IDGCrawler|IDGCrawler|
IDMARCH|idmarchAutomatic|
iltrovatore.it|IlTrovatore-Setaccio|
InAGist|InAGist|
Ilial/Nutch|ilial/Nutch|
Indian Language Web|IIITBOT|
Indian Language Web|webkhoj.iiit.net|
Indy Library|IndyLibrary|
InfoHelfer|Infohelfer|
IntegromeDB|integromedb.org/Crawler|
ismyblogworking.com|ismyblogworking.com|
ismyblogworking.com|ismyblogbroken.com|
IRLbot|IRLbot|
ISC Systems iRc Search|ISCSystems|
Jakarta Commons|JakartaCommons|
Java|Java/|
James BOT|James BOT|
Jyxobot|jyxo.cz|
KaloogaBot|KaloogaBot|
KiNSHOo|KiNShooboT|
KiwiStatus|KiwiStatus|
Komodia|KomodiaBot|
LarBin|larbin_|
LargeSmall|LargeSmallCrawler|
LinkedIn|www.linkedin.com|
Linkdex|linkdex.com|
LinkPad|LinkpadBot|
Radarly|CMRadar|
LinkWalker|LinkWalker|
LiteFinder|LiteFinder|
Livedoor|livedoorScreenShot|
Livlap|LivelapBot/0.2|
llssbot|llssbot|
Livelap|LivelapBot|
LoadTime|load-time.com|
LOGICIEL|LOGICIEL|
LookSmart|grub-client|
ltx71|ltx71|
Magpie-crawler|brandwatch.net|
MagpieRSS|MagpieRSS|
Majestic-12|MJ12bot|
Map of the Internet|MOTIbot|
meanpath|meanpathbot|
mediarithmics|mcsbot|
McBot|McBot|
Megite|Megite2.0|
metager2|metager2|
METASpider|METASpider|
MetaURI|MetaURIAPI/2.0|
Micosi JP|micosi.jp|
Micro$oft Bot |MSRBOT|
Microsoft URL Control|MicrosoftURLControl|
Microsoft Office |Microsoft Office Protocol Discovery|
MLBot|metadatalabs.com|
Mojeek|MojeekBot|
Moreover|Moreoverbot|
MOZ|rogerbot|
MP3 Bot|mp3realm.org|
MSN Mobile|MSMOBOT|
MSN|msnbot|
My Nutch Spider|My Nutch Spider|
Naver|Cowbot|
Naver|naver.com|
Naver|NaverBot|
Neofonie|neofoniesearch:robot|
Naver|naver.com|
Netcraft|Netcraft|
NetSeer Teemer|Teemer|
NetSeer|NetSeer|
NetSeer|ilial/Nutch|
NetNewsWire|NetNewsWire|
Netvibes|Netvibes|
Netseer|www.netseer.com|
Neugierig|Neugierig|
Nexen|www.nexen.net|
Nigma.ru|Nigma.ru|
Odnoklassnik|OdklBot|
Online Broken Link Checker|BrokenLinkCheck.com|
OpenHose|OpenHoseBot|
OpenLinkProfiler|spbot|
NG 2.0|NG/2.0|
OrangeBot|OrangeBot/2.0|
OrangeSpider|OrangeSpider|
Outbrain|SphereScout|
PagePeeker|pagepeeker|
PaloAlto|paloaltonetworks.com|
Paper.li|PaperLiBot|
Parsijoo|parsijoo-crawler|
Peepow|Peepowbot|
Peerindex|peerindex|
PHP version tracker Nexen.net|www.nexen.net|
Picsearch|www.picsearch.com|
Ping.Blo.gs|ping.blo.gs|
Pingdom|Pingdom.com_bot|
Pinterest|Pinterest|
Pi-Monster|Pi-Monster;https://pricepi.com/|
PlantyNet|PlantyNet|
PostRank|PostRank|
PostRank|postrank.com|
ProCog|ProCogBot|
Prlog|Prlog|
Proximic|www.proximic|
Python Bot|Python-urllib/2.6|
Python Bot|python-requests/2.5.1|
QuerySeeker|QuerySeekerSpider|
Qwant|qwantify|
Qwant|Qwantify/|
Radian6|R6_CommentReader|
Radian6|R6_FeedFetcher|
RadianRss|radiarss|
Radian6|www.radian6.com|
Readability|Readability|
RedBot|redbot.org|
Riddler|Riddler|
Rivva|Rivva|
relevantNOISE|www.relevantnoise.com|
Rome Client|RomeClient|
Runnk|runnk|
SearchmetricsBot|www.searchmetrics.com/en/searchmetrics-bot/|
ScoutJet|www.scoutjet.com|
ScreenerBot|ScreenerBotCrawler|
Screaming Frog SEO|ScreamingFrogSEOSpider|
ScSpider|ScSpider|
Searchme.com|searchme.com|
Searchmetrics|SearchmetricsBot|
Search Web Engine|swebot-crawler.net|
Seexie|seexie.com|
Semiocast|semiocast.com|
Seo Hunter|4SeoHuntBot|
SEOkicks.de|SEOkicks-Robot|
SeoMoz|rogerbot-crawler|
SEOprofiler|www.seoprofiler.com|
Seznam|SeznamBot|
Seznam|Seznamscreenshot-generator|
SEMRush|SemrushBot|
shelob|shelob|
ShowyouBot|ShowyouBot|
SimplePie|SimplePie/|
Sindup|Sindup|
Sistrix|crawler.sistrix|
Sistrix|007ac9Crawler|
Site Bot|sitebot|
Site-Perf.com|Site-Perf.comperformancetestingbot|
SiteIntel|SiteIntel.net|
Skimlinks|skimlinks|
Slack|Slackbot-LinkExpanding|
Snapbot|Snap|
SocialBookmarking|socialbm_bot|
SociaLayer|socialayer.com|
Sogou|Sogou|
Sogou|sogou.com|
Sosospider|soso.com|
Sphere|SphereScout|
Spinn3r|Spinn3r|
Spinn3r|Tailrank|
Spinn3r|www.tailrank.com|
SpokeSpider|SpokeSpider|
SputnikBot|SputnikBot|
Steeler|Steeler|
Summify|Summify|
Superfeedr|http://superfeedr.com|
Synoo|Synoobot|
Spinn3r|Tailrank|
Statsbot|Statsbot|
Symfony|SymfonySpider|
Technorati|Technoratibot|
TestBot|TestBot|
The Old Reader|theoldreader|
Thumbshots|thumbshots|
TinEye|TinEye|
Tiscali|IstellaBot|
Topsy|topsy.com|
Toshiba|TosCrawler|uMBot-LN|
Trendiction|trendictionbot|
Turnitin|TurnitinBot|
Twiceler|Twiceler|
Twitter|Twitterbot|
Twitturls|twitturls|
Twitturly|Twitturly|
Twingly|TwinglyRecon|
UMBC Memeta|UMBC-Memeta|
UniversalFeedParser|UniversalFeedParser/|
Unknown Spam bot|python-requests/2.1.0|
Unknown Spam bot|python-requests/0.12.1|
Unknown Spam bot|python-requests/0.13.2|
Unknown Spam bot|Google-HTTP-Java-Client/1.17.0-rc|
Unknown Spam Bot|Mozilla/4.0(compatible;)|
Unknown Spam Bot|Mozilla/4.0(compatible;ICS)|
Unknown Spam Bot|Mozilla/4.0 (compatible; ICS 1.2.105)|
Unknown Spam Bot|Mozilla/4.0(compatible;Synapse)|
Unknown Spam Bot|Mozilla/4.0(compatible;MSIE6.0;WindowsNT5.1;SV1;.NETCLR2.0.50727)|
Unknown Spam Bot|Mozilla/4.0(compatible;MSIE6.0;WindowsNT5.0;)|
Unknown Spam Bot|Mozilla/5.001|
Unknown Spam Bot|Mozilla/0.6Beta(Windows)|
Unknown Spam Bot|Mozilla/0.91Beta(Windows)|
Unknown Spam Bot|Java/1.4.1_04|
Unknown Spam Bot|Java/1.7.0_05|
Unknown Spam Bot|Java/1.7.0_10|
Unknown Spam Bot|Java/1.7.0_09|
Unknown Spam Bot|Java/1.7.0|
Unknown Spam Bot|Java/1.6.0_04|
Unknown Spam Bot|Java/1.6.0_07|
Unknown Spam Bot|Java/1.6.0_20|
Unknown Spam Bot|Java/1.6.0_27|
Unknown Spam Bot|Java/1.6.0_26|
Unknown Spam Bot|Java/1.6.0_33|
Unknown Crawler|Python-urllib|
Unknown Crawler|larbin_2.6.3|
Unknown Spam Bot|DLE_Spider.exe|
Unknown Spam bot|Zend_Http_Client|
urlfan|urlfan-bot|
urlfan|www.urlfan.com|
URI::Fetch|URI::Fetch|
VadixBot|VadixBot|
Ventro|VENTRO|
VerticalSearchWorks|ConveraCrawler|
WiseGuys|wise-guys|
vBSEO|vBSEO|
Vik.com|vikspider|
Visvo|Visvo|
Vk|vkShare|
Voila|VoilaBot|
WebForge|WebForgeBlinkBot|
Website-Datenbank.de|netEstateNECrawler|
W3C Feed|FeedValidator|
W3C Nu|Validator.nu|
W3C Validator|W3C_Validator|
W3C Unicorn|W3C_Unicorn|
W3C CSS validation|W3C_CSS_Validator_JFouffa|
W3C Link Checker|W3C-checklink|
W3C Internationalization Checker|W3C_I18n-Checker|
WASALive|blog.wasalive.com|
WASALive|wasalive-bots|
WebAlta|WebAltaCrawler/|
Web Cookies Scanner|WebCookies|
webmeup|BLEXbot|
WebNL|WebNL|
WeSEE|WeSEE|
Weive|weive|
Weive|weivel|
Wget|WgetWiseGuys|wise-guys|/1.|
Whitevector|Whitevector|
Whitevector|whitevector|
Wikio|Wikio|
Wikio|wikio|
Wikiwix|wikiwix-bot|
WoCoDi|WocodiWebCrawler|
WordPress|WordPress|
Worio|woriobot|
Wotbox|www.wotbox.com/bot|
WPScan|WPScan|
WSCheck|wscheck.com|
WSowner|bot.wsowner|
Xenu Link Sleuth|XenuLink|
xMind|xMind|
XOVI|XoviBot|
x100bot|x100bot|
YaCy|yacybot|
Yahoo FeedSeeker|YahooFeedSeeker|
Yahoo|Yahoo-Yahoo-YSM|
Yahoo Japan|Y!J-|
Yahoo|MMCrawler|
Yahoo|slurp@inktomi|
Yahoo|Yahoo-Test|
Yahoo|Yahoo!Slurp|
Yandex|yandex|
Yandex|Yandex|
Yanga|Yanga|
Yanga WorldSearch Bot|Yanga|
YodaoBot|YodaoBot|
ZemlyaCrawl|ZemlyaCrawl|
Zerbynet|zerbybot|
Zite|zitebot|
Zooka|Zookabot|
Zum|ZumBot|
ZyBorg|ZyBorg|