robots.txt
101h4x.com
2360Spider
3404checker
4404enemy
580legs
6ADmantX
7AIBOT
8ALittle Client
9ASPSeek
10Abonti
11Aboundex
12Aboundexbot
13Acunetix
14AdsTxtCrawlerTP
15AfD-Verbotsverfahren
16AhrefsBot
17Ai2Bot
18AiHitBot
19Aipbot
20Alexibot
21Aliyun
22AliyunSecBot
23AllSubmitter
24Alligator
25AlphaBot
26Anarchie
27Anarchy
28Anarchy99
29Ankit
30Anthill
31Apexoo
32Aspiegel
33Asterias
34Atomseobot
35Attach
36AwarioBot
37AwarioRssBot
38AwarioSmartBot
39BBBike
40BDCbot
41BDFetch
42BLEXBot
43BackDoorBot
44BackStreet
45BackWeb
46Backlink-Ceck
47BacklinkCrawler
48BacklinksExtendedBot
49Badass
50Bandit
51Barkrowler
52BatchFTP
53Battleztar Bazinga
54BetaBot
55Bigfoot
56Bitacle
57BlackWidow
58Black Hole
59Blackboard
60Blow
61BlowFish
62Boardreader
63Bolt
64BotALot
65Brandprotect
66Brandwatch
67Buck
68Buddy
69BuiltBotTough
70BuiltWith
71Bullseye
72BunnySlippers
73BuzzSumo
74Bytespider
75CATExplorador
76CCBot
77CODE87
78CSHttp
79Calculon
80CazoodleBot
81Cegbfeieh
82CensysInspect
83ChatGPT-User
84CheTeam
85CheeseBot
86CherryPicker
87ChinaClaw
88Chlooe
89Citoid
90Claritybot
91ClaudeBot
92Cliqzbot
93Cloud mapping
94Cocolyzebot
95Cogentbot
96Collector
97Copier
98CopyRightCheck
99Copyscape
100Cosmos
101Craftbot
102Crawling at Home Project
103CrazyWebCrawler
104Crescent
105CrunchBot
106Curious
107Custo
108CyotekWebCopy
109DBLBot
110DIIbot
111DSearch
112DTS Agent
113DataCha0s
114DatabaseDriverMysqli
115Demon
116Deusu
117Devil
118Digincore
119DigitalPebble
120Dirbuster
121Disco
122Discobot
123Discoverybot
124Dispatch
125DittoSpyder
126DnBCrawler-Analytics
127DnyzBot
128DomCopBot
129DomainAppender
130DomainCrawler
131DomainSigmaCrawler
132DomainStatsBot
133Domains Project
134Dotbot
135Download Wonder
136Dragonfly
137Drip
138ECCP/1.0
139EMail Siphon
140EMail Wolf
141EasyDL
142Ebingbong
143Ecxi
144EirGrabber
145EroCrawler
146Evil
147Exabot
148Express WebPictures
149ExtLinksBot
150Extractor
151ExtractorPro
152Extreme Picture Finder
153EyeNetIE
154Ezooms
155FDM
156FHscan
157FacebookBot
158FemtosearchBot
159Fimap
160Firefox/7.0
161FlashGet
162Flunky
163Foobot
164Freeuploader
165FrontPage
166Fuzz
167FyberSpider
168Fyrebot
169G-i-g-a-b-o-t
170GPTBot
171GT::WWW
172GalaxyBot
173GeedoProductSearch
174Genieo
175GermCrawler
176GetRight
177GetWeb
178Getintent
179Gigabot
180Go!Zilla
181Go-Ahead-Got-It
182GoZilla
183Gotit
184GrabNet
185Grabber
186Grafula
187GrapeFX
188GrapeshotCrawler
189GridBot
190HEADMasterSEO
191HMView
192HTMLparser
193HTTP::Lite
194HTTrack
195Haansoft
196HaosouSpider
197Harvest
198Havij
199Heritrix
200Hloader
201HonoluluBot
202Humanlinks
203HybridBot
204IDBTE4M
205IDBot
206IRLbot
207Iblog
208Id-search
209IlseBot
210Image Fetch
211Image Sucker
212ImagesiftBot
213IndeedBot
214Indy Library
215InfoNaviRobot
216InfoTekies
217Information Security Team InfraSec Scanner
218InfraSec Scanner
219Intelliseek
220InterGET
221InternetMeasurement
222InternetSeer
223Internet Ninja
224Iria
225Iskanie
226IstellaBot
227JOC Web Spider
228JamesBOT
229Jbrofuzz
230JennyBot
231JetCar
232Jetty
233JikeSpider
234Joomla
235Jorgee
236JustView
237Jyxobot
238Kenjin Spider
239Keybot Translation-Search-Machine
240Keyword Density
241Kinza
242Kozmosbot
243LNSpiderguy
244LWP::Simple
245Lanshanbot
246Larbin
247Leap
248LeechFTP
249LeechGet
250LexiBot
251Lftp
252LibWeb
253Libwhisker
254LieBaoFast
255Lightspeedsystems
256Likse
257LinkScan
258LinkWalker
259Linkbot
260LinkextractorPro
261LinkpadBot
262LinksManager
263LinqiaMetadataDownloaderBot
264LinqiaRSSBot
265LinqiaScrapeBot
266Lipperhey
267Lipperhey Spider
268Litemage_walker
269Lmspider
270Ltx71
271MFC_Tear_Sample
272MIDown tool
273MIIxpc
274MJ12bot
275MQQBrowser
276MSFrontPage
277MSIECrawler
278MTRobot
279Mag-Net
280Magnet
281Mail.RU_Bot
282Majestic-SEO
283Majestic12
284Majestic SEO
285MarkMonitor
286MarkWatch
287Mass Downloader
288Masscan
289Mata Hari
290MauiBot
291Mb2345Browser
292MeanPath Bot
293Meanpathbot
294Mediatoolkitbot
295MegaIndex.ru
296Metauri
297MicroMessenger
298Microsoft Data Access
299Microsoft URL Control
300Minefield
301Mister PiX
302Moblie Safari
303Mojeek
304Mojolicious
305MolokaiBot
306Morfeus Fucking Scanner
307Mozlila
308Mr.4x3
309Msrabot
310Musobot
311NICErsPRO
312NPbot
313Name Intelligence
314Nameprotect
315Navroad
316NearSite
317Needle
318Nessus
319NetAnts
320NetLyzer
321NetMechanic
322NetSpider
323NetZIP
324Net Vampire
325Netcraft
326Nettrack
327Netvibes
328NextGenSearchBot
329Nibbler
330Niki-bot
331Nikto
332NimbleCrawler
333Nimbostratus
334Ninja
335Nmap
336Nuclei
337Nutch
338Octopus
339Offline Explorer
340Offline Navigator
341OnCrawl
342OpenLinkProfiler
343OpenVAS
344Openfind
345Openvas
346OrangeBot
347OrangeSpider
348OutclicksBot
349OutfoxBot
350PECL::HTTP
351PHPCrawl
352POE-Component-Client-HTTP
353PageAnalyzer
354PageGrabber
355PageScorer
356PageThing.com
357Page Analyzer
358Pandalytics
359Panscient
360Papa Foto
361Pavuk
362PeoplePal
363Petalbot
364Pi-Monster
365Picscout
366Picsearch
367PictureFinder
368Piepmatz
369Pimonster
370Pixray
371PleaseCrawl
372Pockey
373ProPowerBot
374ProWebWalker
375Probethenet
376Proximic
377Psbot
378Pu_iN
379Pump
380PxBroker
381PyCurl
382QueryN Metasearch
383Quick-Crawler
384RSSingBot
385Rainbot
386RankActive
387RankActiveLinkBot
388RankFlex
389RankingBot
390RankingBot2
391Rankivabot
392RankurBot
393Re-re
394ReGet
395RealDownload
396Reaper
397RebelMouse
398Recorder
399RedesScrapy
400RepoMonkey
401Ripper
402RocketCrawler
403Rogerbot
404SBIder
405SEOkicks
406SEOkicks-Robot
407SEOlyt
408SEOlyticsCrawler
409SEOprofiler
410SEOstats
411SISTRIX
412SMARTSEO-checker
413SMTBot
414SalesIntelligent
415ScanAlert
416Scanbot
417ScoutJet
418Scrapy
419Screaming
420ScreenerBot
421ScrepyBot
422Searchestate
423SearchmetricsBot
424Seekport
425SeekportBot
426SemanticJuice
427Semrush
428SemrushBot
429SemrushBot-BA
430SemrushBot-FT
431SemrushBot-OCOB
432SemrushBot-SI
433SemrushBot-SWA
434SentiBot
435SenutoBot
436SeoCherryBot
437SeoSiteCheckup
438SeobilityBot
439Seomoz
440Shodan
441Siphon
442SiteAuditBot
443SiteCheckerBotCrawler
444SiteExplorer
445SiteLockSpider
446SiteSnagger
447SiteSucker
448Site Sucker
449Sitebeam
450Siteimprove
451Sitevigil
452SlySearch
453SmartDownload
454Snake
455Snapbot
456Snoopy
457SocialRankIOBot
458Sociscraper
459Sogou web spider
460Sosospider
461Sottopop
462SpaceBison
463Spammen
464SpankBot
465Spanner
466Spbot
467Spider_Bot
468Spider_Bot/3.0
469Spinn3r
470SplitSignalBot
471SputnikBot
472Sqlmap
473Sqlworm
474Sqworm
475Steeler
476Stripper
477Sucker
478Sucuri
479SuperBot
480SuperHTTP
481Surfbot
482SurveyBot
483Suzuran
484Swiftbot
485Szukacz
486T0PHackTeam
487T8Abot
488Teleport
489TeleportPro
490Telesoft
491Telesphoreo
492Telesphorep
493TheNomad
494The Intraformant
495Thumbor
496TightTwatBot
497TinyTestBot
498Titan
499Toata
500Toweyabot
501Tracemyfile
502Trendiction
503Trendictionbot
504True_Robot
505Turingos
506Turnitin
507TurnitinBot
508TwengaBot
509Twice
510Typhoeus
511URLy.Warning
512URLy Warning
513UnisterBot
514Upflow
515V-BOT
516VB Project
517VCI
518Vacuum
519Vagabondo
520VelenPublicWebCrawler
521VeriCiteCrawler
522VidibleScraper
523Virusdie
524VoidEYE
525Voil
526Voltron
527WASALive-Bot
528WBSearchBot
529WEBDAV
530WISENutbot
531WPScan
532WWW-Collector-E
533WWW-Mechanize
534WWW::Mechanize
535WWWOFFLE
536Wallpapers
537Wallpapers/3.0
538WallpapersHD
539WeSEE
540WebAuto
541WebBandit
542WebCollage
543WebCopier
544WebEnhancer
545WebFetch
546WebFuck
547WebGo IS
548WebImageCollector
549WebLeacher
550WebPix
551WebReaper
552WebSauger
553WebStripper
554WebSucker
555WebWhacker
556WebZIP
557Web Auto
558Web Collage
559Web Enhancer
560Web Fetch
561Web Fuck
562Web Pix
563Web Sauger
564Web Sucker
565Webalta
566WebmasterWorldForumBot
567Webshag
568WebsiteExtractor
569WebsiteQuester
570Website Quester
571Webster
572Whack
573Whacker
574Whatweb
575Who.is Bot
576Widow
577WinHTTrack
578WiseGuys Robot
579Wonderbot
580Woobot
581Wotbox
582Wprecon
583Xaldon WebSpider
584Xaldon_WebSpider
585Xenu
586YaK
587YoudaoBot
588Zade
589Zauba
590Zermelo
591Zeus
592Zitebot
593ZmEu
594ZoomBot
595ZoominfoBot
596ZumBot
597ZyBorg
598adscanner
599allenai.org
600anthropic-ai
601archive.org_bot
602arquivo-web-crawler
603arquivo.pt
604autoemailspider
605awario.com
606backlink-check
607cah.io.community
608check1.exe
609clark-crawler
610coccocbot
611cognitiveseo
612cohere-ai
613com.plumanalytics
614crawl.sogou.com
615crawler.feedback
616crawler4j
617dataforseo.com
618dataforseobot
619dataprovider
620demandbase-bot
621domainsproject.org
622eCatch
623evc-batch
624everyfeed-spider
625facebookscraper
626gopher
627heritrix
628imagesift.com
629instabid
630internetVista monitor
631ips-agent
632isitwp.com
633iubenda-radar
634l9scan
635leakix
636linkdexbot
637linkfluence
638lwp-request
639lwp-trivial
640magpie-crawler
641meanpathbot
642mediawords
643muhstik-scan
644netEstate NE Crawler
645oBot
646omgili
647openai
648openai.com
649page scorer
650pcBrowser
651plumanalytics
652polaris version
653probe-image-size
654ripz
655s1z.ru
656satoristudio.net
657scalaj-http
658scan.lol
659seobility
660seocompany.store
661seoscanners
662seostar
663serpstatbot
664sexsearcher
665sitechecker.pro
666siteripz
667smart-seo-tools.sbs
668sogouspider
669sp_auditbot
670spyfu
671sysscan
672tAkeOut
673trendiction.com
674trendiction.de
675ubermetrics-technologies.com
676voyagerx.com
677webgains-bot
678webmeup-crawler
679webpros.com
680webprosbot
681x09Mozilla
682x22Mozilla
683xpymep1.exe
684zauba.io
685zgrab
