Comment bloquer les mauvais robots sur votre site avec Fail2ban

Si vous avez un serveur pour héberger vos sites internet et que vous consultez de temps en temps vos fichiers logs vous avez certainement constater que vous avez un nombre inconsidérable de visites de robots. Ces derniers ne sont pas tous mauvais et permettent notamment aux moteurs de recherche de comprendre et découvrir votre contenu pour pouvoir l'indexer, mais certains ne vous seront d'aucune utilité pour le développement de votre site et ils vont même consommer de la ressource (CPU, mémoire) inutilement.

Voici ce que j'applique actuellement et franchement c'est très très efficace puisque la majorité des "bad bots" sont maintenant bloqués et que j'ai donc pu récupérer de la disponibilité en processeur et en RAM ce qui est appréciable finalement!

La logique pour bloquer un "bad bots" est de lui indiquer que vous ne souhaitez pas que votre contenu soit consulté mais malheureusement les bad bots n'ont pas pour habitude de respecter votre fameux fichier "robots.txt" il faut donc une méthode plus expéditive que l'on peut mettre en place avec un superbe outil comme Fail2ban si vous serveur fonctionne avec apache!

Pour installer Fail2ban je vous recommande de consulter les tutoriels sur le sujet il y en a plein et même en français, cet article va donc simplement vous expliquer comment écrire la règle pour lutter contre les bad bots!

Premièrement, il faut créer le filtre que l'on va appeler simplement "badbots.conf" qui se situe dans le répertoire "filter.d" de fail2ban et ajouter la règle "badbots" dans votre fichier de règle (defaults-debian.conf normalement si vous êtes sous debian) comme ceci :

[badbots]
# Fail2Ban configuration file
#
# Regexp to catch known spambots and software alike. Please verify
# that it is your intent to block IPs which were driven by
# above mentioned bots.


[Definition]

badbotscustom = \bQwantify\b|\bCFNetwork\b
badbots = \b173\.212soso\b|\b192\.comagent\b|\b1noonbot\b|\b1on1searchbot\b|\b360Spider\b|\b3de_search2\b|\b3d_search\b|\b3g bot\b|\b3gse\b|\b404checker\b|\b404enemy\b|\b50\.nu\b|\b80legs\b|\bA1 sitemap generator\b|\bA1 website download\b|\bA6-indexer\b|\bAasp\b|\bAbachobot\b|\bAbonti\b|\bAbotemailsearch\b|\bAboundex\b|\bAboutusbot\b|\bAccmonitor compliance server\b|\bAccoon\b|\bAchulkov\.net page walker\b|\bAcme\.spider\b|\bAcoonbot\b|\bAcquia-crawler\b|\bActivetouristbot\b|\bAcunetix\b|\bAdamm bot\b|\bAdbeat_bot\b|\bADmantX\b|\bAdminshop\.com\b|\bAd muncher\b|\bAdvanced email extractor\b|\bAesop_com_spiderman\b|\bAespider\b|\bAfD-Verbotsverfahren\b|\bAf knowledge now verity spider\b|\bAggregator\:vocus\b|\bAh-ha\.com crawler\b|\bAhrefs\b|\bAhrefsBot\b|\bAIBOT\b|\bAidu\b|\bAiHitBot\b|\bAipbot\b|\bAisiid\b|\bAitcsrobot/1\.1\b|\bAjsitemap\b|\bAkamai-sitesnapshot\b|\bAlexawebsearchplatform\b|\bAlexfdownload\b|\bAlexibot\b|\bAlkalinebot\b|\bAll acronyms bot\b|\bAlligator\b|\bAllSubmitter\b|\bAlphaBot\b|\bAlpha search agent\b|\bAmerla search bot\b|\bAmfibibot\b|\bAmpmppc\.com\b|\bAmznkassocbot\b|\bAnarchie\b|\bAnemone\b|\bAnonymous\b|\bAnotherbot\b|\bAnswerbot\b|\bAnswerbus\b|\bAnswerchase prove\b|\bAntbot\b|\bAntibot\b|\bAntisantyworm\b|\bAntro\.net\b|\bAonde-spider\b|\bApexoo\b|\bAport\b|\bAppengine-google\b|\bAppid\: s~stremor-crawler-\b|\bAqua_products\b|\bArabot\b|\bArachmo\b|\bArachnophilia\b|\bAria equalizer\b|\bArianna\.libero\.it\b|\bArikus_spider\b|\bArtavisbot\b|\bArtera\b|\bArt-online\.com\b|\bAsaha search engine turkey\b|\bAsk\b|\bAspider\b|\bASPSeek\b|\bAsterias\b|\bAstrofind\b|\bAthenusbot\b|\bAtlocalbot\b|\bAtomic_email_hunter\b|\bAttach\b|\bAttrakt\b|\bAttributor\b|\bAugurfind\b|\bAuresys\b|\bAutobaron crawler\b|\bAutoemailspider\b|\bAutowebdir\b|\bAvsearch-\b|\bAxfeedsbot\b|\bAxonize-bot\b|\bAyna\b|\bB2w\b|\bBackDoorBot\b|\bBacklink-Ceck\b|\bBacklink-check\b|\bBacklinkCrawler\b|\bBackrub\b|\bBackStreet\b|\bBackstreet browser\b|\bBackWeb\b|\bBadass\b|\bBandit\b|\bBarkrowler\b|\bBatchFTP\b|\bBattleztar Bazinga\b|\bBaypup\b|\bBBBike\b|\bBDCbot\b|\bBDFetch\b|\bBecomebot\b|\bBecomejpbot\b|\bBeetlebot\b|\bBender\b|\bBesserscheitern-crawl\b|\bBetaBot\b|\bBigado\.com\b|\bBig brother\b|\bBigcliquebot\b|\bBig data\b|\bBigfoot\b|\bBiglotron\b|\bBilbo\b|\bBilgibetabot\b|\bBilgibot\b|\bBintellibot\b|\bBitacle\b|\bBitlybot\b|\bBitvouseragent\b|\bBizbot\b|\bBizworks retriever\b|\bBlackbird\b|\bBlackboard\b|\bBlack\.hole\b|\bBlack Hole\b|\bBlackmask\.net search engine\b|\bBlackWidow\b|\bBladder fusion\b|\bBlaiz-bee\b|\bBlexbot\b|\bBlinkx\b|\bBlitzbot\b|\bBlog conversation project\b|\bBlogmyway\b|\bBlogpulselive\b|\bBlogrefsbot\b|\bBlogscope\b|\bBlogslive\b|\bBloobybot\b|\bBlow\b|\bBlowFish\b|\bBlt\b|\bBnf\.fr_bot\b|\bBoaconstrictor\b|\bBoardreader\b|\bBoia\.org\b|\bBoia-scan-agent\b|\bBoi_crawl_00\b|\bBoitho\b|\bBolt\b|\bBookmark buddy bookmark checker\b|\bBookmark search tool\b|\bBosug\b|\bBotALot\b|\bBot apoena\b|\bBotrighthere\b|\bBotswana\b|\bBottybot\b|\bBpbot\b|\bBraintime_search\b|\bBrandprotect\b|\bBrandwatch\b|\bBrokenlinkcheck\.com\b|\bBrowseremulator\b|\bBrowsermob\b|\bBruinbot\b|\bBsearchr&d\b|\bBspider\b|\bBtbot\b|\bBtsearch\b|\bBubing\b|\bBuddy\b|\bBuibui\b|\bBuildcms crawler\b|\bBuiltBotTough\b|\bBuiltWith\b|\bBullseye\b|\bBumblebee\b|\bBunnySlippers\b|\bBuscadorclarin\b|\bBuscaplus robi\b|\bButterfly\b|\bBuyhawaiibot\b|\bBuzzbot\b|\bBuzzSumo\b|\bByindia\b|\bByspider\b|\bByteserver\b|\bBzbot\b|\bCacheblaster\b|\bCaddbot\b|\bCafi\b|\bCalculon\b|\bCamcrawler\b|\bCamelstampede\b|\bCanon-webrecord\b|\bCareerbot\b|\bCataguru\b|\bCatchbot\b|\bCATExplorador\b|\bCazoodle\b|\bCCBot\b|\bCcgcrawl\b|\bCcubee\b|\bCd-preload\b|\bCegbfeieh\b|\bCe-preload\b|\bCerberian drtrs\b|\bCert figleafbot\b|\bCfetch\b|\bCfnetwork\b|\bChameleon\b|\bCharlotte\b|\bCheckbot\b|\bCheck&get\b|\bChecklinks\b|\bCheeseBot\b|\bChemiede-nodebot\b|\bCherryPicker\b|\bChilkat\b|\bChinaClaw\b|\bChlooe\b|\bChromeframe\b|\bCipinetbot\b|\bCis455crawler\b|\bCiteseerxbot\b|\bCizilla\b|\bClariabot\b|\bClaritybot\b|\bClimate ark\b|\bClimateark spider\b|\bCliqzbot\b|\bCloud mapping\b|\bClshttp\b|\bClushbot\b|\bCoast scan engine\b|\bCoast webmaster pro\b|\bCoccoc\b|\bCoccocbot-web\b|\bCogentbot\b|\bCognitiveseo\b|\bCollapsarweb\b|\bCollector\b|\bColocrossing\b|\bCombine\b|\bCom\.plumanalytics\b|\bConnectsearch\b|\bConpilot\b|\bContentsmartz\b|\bContextad bot\b|\bContype\b|\bCookienet\b|\bCoolbot\b|\bCoolcheck\b|\bCopernic\b|\bCopier\b|\bCopyRightCheck\b|\bCopyscape\b|\bCore-project\b|\bCosmos\b|\bCovario-ids\b|\bCowbot-\b|\bCowdog bot\b|\bCrabbybot\b|\bCraftbot\b|\bC r a w l 3 r\b|\bCrawl_application\b|\bCrawler@\b|\bCrawler43\.ejupiter\.com\b|\bCrawler4j\b|\bCrawler\.feedback\b|\bCrawler_for_infomine\b|\bCrawler\.kpricorn\.org\b|\bcrawl\.sogou\.com\b|\bCrawly\b|\bCrazyWebCrawler\b|\bCreativecommons\b|\bCrescent\b|\bCs-crawler\b|\bCse html validator\b|\bCSHttp\b|\bCshttpclient\b|\bCuasarbot\b|\bCulsearch\b|\bCurious\b|\bCusto\b|\bCvaulev\b|\bCyberdog\b|\bCybernavi_webget\b|\bCyberpatrol sitecat webbot\b|\bCyberspyder\b|\bCydralspider\b|\bD1garabicengine\b|\bDatabaseDriverMysqli\b|\bDataCha0s\b|\bDatafountains\b|\bDataparksearch\b|\bDataprovider\.com\b|\bDatascape robot\b|\bDataspearspiderbot\b|\bDataspider\b|\bDattatec\.com\b|\bDaumoa\b|\bDBLBot\b|\bDcpbot\b|\bDeclumbot\b|\bDeepindex\b|\bDeepnet crawler\b|\bDeeptrawl\b|\bDejan\b|\bDel\.icio\.us-thumbnails\b|\bDeltascan\b|\bDelvubot\b|\bDemandbase-bot\b|\bDemon\b|\bDer gro§e bildersauger\b|\bDer große bildersauger\b|\bDeusu\b|\bDevil\b|\bDfs-fetch\b|\bDiagem\b|\bDiamond\b|\bDibot\b|\bDidaxusbot\b|\bDigext\b|\bDigger\b|\bDigincore\b|\bDigi-rssbot\b|\bDigitalarchivesbot\b|\bDigitalPebble\b|\bDigout4u\b|\bDIIbot\b|\bDillo\b|\bDirbuster\b|\bDir_snatch\.exe\b|\bDisco\b|\bDiscobot\b|\bDiscoverybot\b|\bDistilled-reputation-monitor\b|\bDittoSpyder\b|\bDjangotraineebot\b|\bDkimrepbot\b|\bDmoz downloader\b|\bDnyzBot\b|\bDocomo\b|\bDof-verify\b|\bDomainAppender\b|\bDomainCrawler\b|\bDomainscan\b|\bDomainSigmaCrawler\b|\bDomainStatsBot\b|\bDomainwatcher bot\b|\bDotbot\b|\bDotspotsbot\b|\bDow jones searchbot\b|\bDownload\b|\bDownload Wonder\b|\bDoy\b|\bDragonfly\b|\bDrip\b|\bDrone\b|\bDtaagent\b|\bDTS Agent\b|\bDtsearchspider\b|\bDumbot\b|\bDwaar\b|\bDxseeker\b|\bEah\b|\bEarth platform indexer\b|\bEarth science educator  robot\b|\bEasyDL\b|\bEbingbong\b|\bEc2linkfinder\b|\bEcairn-grabber\b|\bECatch\b|\bECCP/1\.0\b|\bEchoosebot\b|\bEcxi\b|\bEdisterbot\b|\bEdugovsearch\b|\bEgothor\b|\bEidetica\.com\b|\bEirGrabber\b|\bElblindo the blind bot\b|\bElisabot\b|\bEllerdalebot\b|\bEmailcollector\b|\bEmail extractor\b|\bEmailleach\b|\bEmailsiphon\b|\bEMail Siphon\b|\bEmailwolf\b|\bEMail Wolf\b|\bEmeraldshield\b|\bEmpas_robot\b|\bEnabot\b|\bEndeca\b|\bEnigmabot\b|\bEnswer neuro bot\b|\bEnter user-agent\b|\bEntitycubebot\b|\bEroCrawler\b|\bE-societyrobot\b|\bEstylesearch\b|\bEsyndicat bot\b|\bEurosoft-bot\b|\bEvaal\b|\bEvc-batch\b|\bEventware\b|\bEverest-vulcan inc\.\b|\bEvil\b|\bExabot\b|\bExactsearch\b|\bExactseek\b|\bExooba\b|\bExploder\b|\bExpress WebPictures\b|\bExtLinksBot\b|\bExtractor\b|\bExtractorPro\b|\bExtreme Picture Finder\b|\bEyenetie\b|\bEyeNetIE\b|\bEzooms\b|\bEz-robot\b|\bFactbot\b|\bFairad client\b|\bFalcon\b|\bFastbot crawler\b|\bFastbot\.de crawler\b|\bFast data search document retriever\b|\bFast esp\b|\bFast-search-engine\b|\bFatbot\b|\bFavcollector\b|\bFaviconizer\b|\bFavorites sweeper\b|\bF-bot test pilot\b|\bFDM\b|\bFdse robot\b|\bFedcontractorbot\b|\bFembot\b|\bFemtosearchBot\b|\bFetch api request\b|\bFetch_ici\b|\bFgcrawler\b|\bFHscan\b|\bFilangy\b|\bFilehound\b|\bFimap\b|\bFindanisp\.com_isp_finder\b|\bFindlinks\b|\bFindweb\b|\bFirebat\b|\bFirefox/7\.0\b|\bFirstgov\.gov search\b|\bFlaming attackbot\b|\bFlamingo_searchengine\b|\bFlashcapture\b|\bFlashGet\b|\bFlickysearchbot\b|\bFluffy the spider\b|\bFlunky\b|\bFocused_crawler\b|\bFollowsite\b|\bFoobot\b|\bFooooo_web_video_crawl\b|\bFopper\b|\bFormulafinderbot\b|\bForschungsportal\b|\bFrancis\b|\bFr_crawler\b|\bFreeuploader\b|\bFreewebmonitoring sitechecker\b|\bFreshcrawler\b|\bFreshdownload\b|\bFreshlinks\.exe\b|\bFriendfeedbot\b|\bFrodo\.at\b|\bFroggle\b|\bFrontPage\b|\bFroola bot\b|\bFull_breadth_crawler\b|\bFu-nbi\b|\bFunnelback\b|\bFurlbot\b|\bFyberSpider\b|\bFyrebot\b|\bG10-bot\b|\bGaisbot\b|\bGalaxyBot\b|\bGazz\b|\bGbplugin\b|\bGenerate_infomine_category_classifiers\b|\bGenevabot\b|\bGeniebot\b|\bGenieo\b|\bGeomaxenginebot\b|\bGeometabot\b|\bGeonabot\b|\bGeovisu\b|\bGermCrawler\b|\bGethtmlcontents\b|\bGetintent\b|\bGetleft\b|\bGetRight\b|\bGetsmart\b|\bGeturl\.rexx\b|\bGetWeb\b|\bGiant\b|\bGigablast\b|\bGigablastopensource\b|\bGigabot\b|\bG-i-g-a-b-o-t\b|\bGirafabot\b|\bGleamebot\b|\bGnome-vfs\b|\bGo-Ahead-Got-It\b|\bGoforitbot\b|\bGoforit\.com\b|\bGo-http-client\b|\bGold crawler\b|\bGoldfire server\b|\bGolem\b|\bGoodjelly\b|\bGordon-college-google-mini\b|\bGoroam\b|\bGoseebot\b|\bGotit\b|\bGovbot\b|\bGoZilla\b|\bGo\!Zilla\b|\bGpu p2p crawler\b|\bGrabber\b|\bGrabNet\b|\bGrafula\b|\bGrapeFX\b|\bGrapeshot\b|\bGrbot\b|\bGreenyogi\b|\bGridBot\b|\bGromit\b|\bGrub\b|\bGslfbot\b|\bGT\:\:WWW\b|\bGulliver\b|\bGulperbot\b|\bGurujibot\b|\bGvcbot\.com\b|\bGvc business crawler\b|\bGvc crawler\b|\bGvc search bot\b|\bGvc web crawler\b|\bGvc weblink crawler\b|\bGvc world links\b|\bHaansoft\b|\bHaosouSpider\b|\bHappyfunbot\b|\bHarvest\b|\bHatena antenna\b|\bHavij\b|\bHawler\b|\bHcat\b|\bHclsreport-crawler\b|\bHd nutch agent\b|\bHeader_test_client\b|\bHEADMasterSEO\b|\bHealia\b|\bHelix\b|\bHere will be link to crawler site\b|\bHeritrix\b|\bHiscan\b|\bHisoftware accmonitor server\b|\bHisoftware accverify\b|\bHitcrawler\b|\bHivabot\b|\bHloader\b|\bHmsebot\b|\bHMView\b|\bHoge\b|\bHolmes\b|\bHomepagesearch\b|\bHooblybot-image\b|\bHoowwwer\b|\bHostcrawler\b|\bHsft - link scanner\b|\bHsft - lvu scanner\b|\bHslide\b|\bHt\://check\b|\bHtdig\b|\bHtml link validator\b|\bHTMLparser\b|\bHttplib\b|\bHTTP\:\:Lite\b|\bHTTrack\b|\bHuaweisymantecspider\b|\bHul-wax\b|\bHumanlinks\b|\bHybridBot\b|\bHyperestraier\b|\bHyperix\b|\bIblog\b|\bIbuena\b|\bIcab\b|\bIcds-ingestion\b|\bIchiro\b|\bIcopyright conductor\b|\bIDBot\b|\bIDG/UK\b|\bId-search\b|\bIeautodiscovery\b|\bIecheck\b|\bIhwebchecker\b|\bIiitbot\b|\bIim_405\b|\bIlseBot\b|\bIltrovatore\b|\bImagebot\b|\bImage Fetch\b|\bImage-fetcher\b|\bImagefortress\b|\bImageshereimagesthereimageseverywhere\b|\bImage stripper\b|\bImage Sucker\b|\bImagevisu\b|\bImds_monitor\b|\bImo-google-robot-intelink\b|\bInagist\.com url crawler\b|\bIndeedBot\b|\bIndexer\b|\bIndustry cortex webcrawler\b|\bIndylabs_marius\b|\bIndy Library\b|\bInelabot\b|\bInet32 ctrl\b|\bInetbot\b|\bInfolink\b|\bInfomine\b|\bInfoNaviRobot\b|\bInformant\b|\bInfo seeker\b|\bInfoseek sidewinder\b|\bInfoTekies\b|\bInfousabot\b|\bIngrid\b|\bInktomi\b|\bInsightscollector\b|\bInsightsworksbot\b|\bInspirebot\b|\bInstabid\b|\bInsumascout\b|\bIntelix\b|\bIntelliseek\b|\bInterGET\b|\bInternetlinkagent\b|\bInternet Ninja\b|\bInternet radio crawler\b|\bInternetSeer\b|\bInternetVista monitor\b|\bInterseek\b|\bIoi\b|\bIpadd bot\b|\bIps-agent\b|\bIpselonbot\b|\bIp-web-crawler\.com\b|\bIria\b|\bIRLbot\b|\bIron33\b|\bIsara\b|\bIsearch\b|\bIsilox\b|\bIskanie\b|\bIstellaBot\b|\bIts-learning crawler\b|\bIu_csci_b659_class_crawler\b|\bIvia\b|\bJadynave\b|\bJamesBOT\b|\bJbot\b|\bJbrofuzz\b|\bJemmathetourist\b|\bJennyBot\b|\bJetbot\b|\bJetbrains omea pro\b|\bJetCar\b|\bJikeSpider\b|\bJim\b|\bJobo\b|\bJobspider_ba\b|\bJoc\b|\bJOC Web Spider\b|\bJoedog\b|\bJoomla\b|\bJorgee\b|\bJoyscapebot\b|\bJspyda\b|\bJunut bot\b|\bJustView\b|\bJyxobot\b|\bKakclebot\b|\bKalooga\b|\bKatatudo-spider\b|\bKbeta1\b|\bKeepni web site monitor\b|\bKenjin\.spider\b|\bKenjin Spider\b|\bKeybot translation-search-machine\b|\bKeywenbot\b|\bKeyword\.density\b|\bKeyword Density\b|\bKinjabot\b|\bKitenga-crawler-bot\b|\bKiwistatus\b|\bKmbot-\b|\bKmccrew bot search\b|\bKnight\b|\bKnowitall\b|\bKnowledge\.com\b|\bKnowledge engine\b|\bKoepabot\b|\bKoninklijke\b|\bKorniki\b|\bKozmosbot\b|\bKrowler\b|\bKsbot\b|\bK\.s\.bot\b|\bKuloko-bot\b|\bKulturarw3\b|\bKummhttp\b|\bKurzor\b|\bKyluka crawl\b|\bLabhoo\b|\bLabourunions411\b|\bLachesis\b|\bLament\b|\bLamerexterminator\b|\bLanshanbot\b|\bLapozzbot\b|\bLarbin\b|\bLbot\b|\bLeaptag\b|\bLeechFTP\b|\bLeechGet\b|\bLetscrawl\.com\b|\bLexiBot\b|\bLexxebot\b|\bLftp\b|\bLibcrawl\b|\bLibiviacore\b|\bLibw\b|\bLibWeb\b|\bLibwhisker\b|\bLightspeedsystems\b|\bLikse\b|\bLinguee bot\b|\bLinkalarm\b|\bLinkbot\b|\bLinkcheck by siteimprove\.com\b|\bLinkchecker\b|\bLink_checker\b|\bLink checker\b|\bLinkcheck scanner\b|\bLinkdexbot\b|\bLinkdex\.com\b|\bLinkextractorPro\b|\bLinklint\b|\bLinklooker\b|\bLinkman\b|\bLinkpadBot\b|\bLinkScan\b|\bLinksManager\b|\bLinksmanager\.com_bot\b|\bLinks sql\b|\bLinksweeper\b|\bLink validator\b|\bLinkWalker\b|\bLinqiaMetadataDownloaderBot\b|\bLinqiaRSSBot\b|\bLinqiaScrapeBot\b|\bLipperhey\b|\bLitefinder\b|\bLitemage_walker\b|\bLitlrbot\b|\bLittle grabber at skanktale\.com\b|\bLivelapbot\b|\bLm harvester\b|\bLmqueuebot\b|\bLmspider\b|\bLNSpiderguy\b|\bLoadtimebot\b|\bLocalcombot\b|\bLocust\b|\bLolongbot\b|\bLookbot\b|\bLsearch\b|\bLssbot\b|\bLt scotland checklink\b|\bLtx71\b|\bL\.webis\b|\bLWP\b|\bLWP\:\:Simple\b|\bLwp-trivial\b|\bLycos_spider\b|\bLydia entity spider\b|\bLynnbot\b|\bLytranslate\b|\bMagnet\b|\bMag-Net\b|\bMagpie-crawler\b|\bMagus bot\b|\bMail\.ru\b|\bMail\.RU_Bot\b|\bMainseek_bot\b|\bMajestic12\b|\bMammoth\b|\bMap robot\b|\bMarkMonitor\b|\bMarkWatch\b|\bMasagool\b|\bMasidani_bot_\b|\bMasscan\b|\bMass Downloader\b|\bMata\.hari\b|\bMata Hari\b|\bMatentzn at cs dot man dot ac dot uk\b|\bMauiBot\b|\bMaxamine\.com\b|\bMaxomobot\b|\bMcbot\b|\bMeanpathbot\b|\bMediawords\b|\bMedrabbit\b|\bMegaIndex\.ru\b|\bMegite\b|\bMemacbot\b|\bMemo\b|\bMendeleybot\b|\bMercator-\b|\bMercuryboard_user_agent_sql_injection\.nasl\b|\bMetacarta\b|\bMetaeuro web search\b|\bMetager2\b|\bMetagloss\b|\bMetal crawler\b|\bMetaquerier\b|\bMetaspider\b|\bMetaspinner\b|\bMetauri\b|\bMfcrawler\b|\bMFC_Tear_Sample\b|\bMfhttpscan\b|\bMicrosoft Data Access\b|\bMicrosoft URL Control\b|\bMIDown tool\b|\bMIIxpc\b|\bMinibot\b|\bMinirank\b|\bMini-robot\b|\bMirror\b|\bMissigua locator\b|\bMister\.pix\b|\bMister PiX\b|\bMiva\b|\bMJ12bot\b|\bMnogosearch\b|\bMod_accessibility\b|\bModuna\.com\b|\bMoget\b|\bMojeek\b|\bMonkeycrawl\b|\bMorfeus Fucking Scanner\b|\bMoses\b|\bMowserbot\b|\bMqbot\b|\bMr\.4x3\b|\bMse360\b|\bMSFrontPage\b|\bMSIECrawler\b|\bMsindianwebcrawl\b|\bMsmobot\b|\bMsnptc\b|\bMsrabot\b|\bMsrbot\b|\bMS Web Services Client Protocol\b|\bMt-soft\b|\bMuhstik-scan\b|\bMultitext\b|\bMusobot\b|\bMyapp\b|\bMycompanybot\b|\bMycrawler\b|\bMyengines-us-bot\b|\bMyfamilybot\b|\bMy-heritrix-crawler\b|\bMy_little_searchengine_project\b|\bMyra\b|\bNabot\b|\bNajdi\.si\b|\bNambu\b|\bName Intelligence\b|\bNameprotect\b|\bNasa search\b|\bNatchcvs\b|\bNatweb-bad-link-mailer\b|\bNaver\b|\bNavroad\b|\bNearSite\b|\bNec-meshexplorer\b|\bNeedle\b|\bNeosciocrawler\b|\bNerdbynature\.bot\b|\bNerdybot\b|\bNerima-crawl-\b|\bNessus\b|\bNestreader\b|\bNetAnts\b|\bNetcarta cyberpilot pro\b|\bNetcraft\b|\bNetEstate NE Crawler\b|\bNetexperts\b|\bNetid\.com bot\b|\bNetLyzer\b|\bNetMechanic\b|\bNetprospector\b|\bNetresearchserver\b|\bNetseer\b|\bNetshift\b|\bNetsongbot\b|\bNetsparker\b|\bNetSpider\b|\bNetsrcherp\b|\bNettrack\b|\bNet\:\:trackback\b|\bNet Vampire\b|\bNetvibes\b|\bNetZIP\b|\bNewmedhunt\b|\bNews bot\b|\bNewsgatherer\b|\bNewsgroupreporter\b|\bNews_search_app\b|\bNewstrovebot\b|\bNextGenSearchBot\b|\bNextthing\.org\b|\bNibbler\b|\bNicebot\b|\bNICErsPRO\b|\bNiki-bot\b|\bNikto\b|\bNimbleCrawler\b|\bNimbus-1\b|\bNinetowns\b|\bNinja\b|\bNjuicebot\b|\bNlese\b|\bNmap\b|\bNogate\b|\bNorbert the spider\b|\bNoteworthybot\b|\bNPbot\b|\bNrcan intranet crawler\b|\bNsdl_search_bot\b|\bNuggetize\.com bot\b|\bNusearch spider\b|\bNutch\b|\bNu_tch\b|\bNwspider\b|\bNymesis\b|\bNys-crawler\b|\bObjectssearch\b|\bOBot\b|\bObvius external linkcheck\b|\bOcelli\b|\bOctopus\b|\bOdp entries t_st\b|\bOegp\b|\bOffline\.explorer\b|\bOffline Explorer\b|\bOffline Navigator\b|\bOgspider\b|\bOmiexplorer_bot\b|\bOmniexplorer\b|\bOmnifind\b|\bOmniweb\b|\bOnetszukaj\b|\bOnline link validator\b|\bOozbot\b|\bOpenbot\b|\bOpenfind\b|\bOpenintelligencedata\b|\bOpenisearch\b|\bOpenLinkProfiler\b|\bOpenlink virtuoso rdf crawler\b|\bOpensearchserver_bot\b|\bOpenVAS\b|\bOpidig\b|\bOptidiscover\b|\bOracle secure enterprise search\b|\bOracle ultra search\b|\bOrangeBot\b|\bOrangeSpider\b|\bOrisbot\b|\bOrnl_crawler\b|\bOrnl_mercury\b|\bOsis-project\.jp\b|\bOso\b|\bOutclicksBot\b|\bOutfoxBot\b|\bOutfoxmelonbot\b|\bOwler-bot\b|\bOwsbot\b|\bOzelot\b|\bP3p client\b|\bPageAnalyzer\b|\bPage Analyzer\b|\bPagebiteshyperbot\b|\bPagebull\b|\bPagedown\b|\bPagefetcher\b|\bPageGrabber\b|\bPagepeeker\b|\bPagerank monitor\b|\bPage scorer\b|\bPageScorer\b|\bPage_verifier\b|\bPamsnbot\.htm\b|\bPanopy bot\b|\bPanscient\b|\bPansophica\b|\bPapa Foto\b|\bPaperlibot\b|\bParasite\b|\bParsijoo\b|\bPathtraq\b|\bPattern\b|\bPatwebbot\b|\bPavuk\b|\bPaxleframework\b|\bPbbot\b|\bPcBrowser\b|\bPcore-http\b|\bPd-crawler\b|\bPECL\:\:HTTP\b|\bPenthesila\b|\bPeoplePal\b|\bPerform_crawl\b|\bPerman\b|\bPersonal ultimate crawler\b|\bPHPCrawl\b|\bPhpdig\b|\bPhp version tracker\b|\bPicosearch\b|\bPicscout\b|\bPicsearch\b|\bPictureFinder\b|\bPieno robot\b|\bPimonster\b|\bPi-Monster\b|\bPipbot\b|\bPipeliner\b|\bPita\b|\bPixfinder\b|\bPixray\b|\bPiyushbot\b|\bPlanetwork bot search\b|\bPleaseCrawl\b|\bPlucker\b|\bPlukkie\b|\bPlumanalytics\b|\bPlumtree\b|\bPockey\b|\bPocohttp\b|\bPOE-Component-Client-HTTP\b|\bPogodak\.ba\b|\bPogodak\.co\.yu\b|\bPoirot\b|\bPolybot\b|\bPompos\b|\bPoodle predictor\b|\bPopscreenbot\b|\bPostpost\b|\bPrivacyfinder\b|\bProbethenet\b|\bProjectwf-java-test-crawler\b|\bProPowerBot\b|\bProWebWalker\b|\bProxem websearch\b|\bProximic\b|\bProxy crawler\b|\bPsbot\b|\bPss-bot\b|\bPsycheclone\b|\bPub-crawler\b|\bPucl\b|\bPulsebot\b|\bPump\b|\bPwebot\b|\bPxBroker\b|\bPyCurl\b|\bQeavis agent\b|\bQfkbot\b|\bQualidade\b|\bQualidator\.com bot\b|\bQuepasacreep\b|\bQueryn\.metasearch\b|\bQueryn metasearch\b|\bQueryN Metasearch\b|\bQuest\.durato\b|\bQuick-Crawler\b|\bQuintura-crw\b|\bQunarbot\b|\bQwantify\b|\bQweerybot\b|\bQweery_robot\.txt_checkbot\b|\bR2ibot\b|\bR6_commentreader\b|\bR6_feedfetcher\b|\bR6_votereader\b|\bRabot\b|\bRadian6\b|\bRadiation retriever\b|\bRampybot\b|\bRankActive\b|\bRankFlex\b|\bRankingBot\b|\bRankingBot2\b|\bRankivabot\b|\bRankur\b|\bRankurBot\b|\bRational sitecheck\b|\bRcstartbot\b|\bRealDownload\b|\bReaper\b|\bRebelMouse\b|\bRebi-shoveler\b|\bRecorder\b|\bRedbot\b|\bRedcarpet\b|\bRedesScrapy\b|\bReGet\b|\bRepoMonkey\b|\bResearch robot\b|\bRiddler\b|\bRiight\b|\bRipper\b|\bRisenetbot\b|\bRiverglassscanner\b|\bRobopal\b|\bRobosourcer\b|\bRobotek\b|\bRobozilla\b|\bRocketCrawler\b|\bRoger\b|\bRogerbot\b|\bRome client\b|\bRondello\b|\bRotondo\b|\bRoverbot\b|\bRpt-httpclient\b|\bRtgibot\b|\bRufusbot\b|\bRunnk online rss reader\b|\bRunnk rss aggregator\b|\bS2bot\b|\bSafaribookmarkchecker\b|\bSafednsbot\b|\bSafetynet robot\b|\bSaladspoon\b|\bSalesIntelligent\b|\bSapienti\b|\bSapphireweb\b|\bSBIder\b|\bSbl-bot\b|\bScanAlert\b|\bScanbot\b|\bScan\.lol\b|\bScfcrawler\b|\bScich\b|\bScientificcommons\.org\b|\bScollspider\b|\bScooperbot\b|\bScooter\b|\bScoutjet\b|\bScrapebox\b|\bScrapy\b|\bScrawltest\b|\bScreaming\b|\bScreenerBot\b|\bScrubby\b|\bScspider\b|\bScumbot\b|\bSearcharoo\.net\b|\bSearchblox\b|\bSearchbot\b|\bSearch-channel\b|\bSearchengine\b|\bSearch-engine-studio\b|\bSearchestate\b|\bSearchhippo\.com\b|\bSearchit-bot\b|\bSearch\.kumkie\.com\b|\bSearchmarking\b|\bSearchmarks\b|\bSearchmee\b|\bSearchmetricsBot\b|\bSearchmining\b|\bSearchnowbot\b|\bSearchpreview\b|\bSearch publisher\b|\bSearchspider\.com\b|\bSearch\.updated\.com\b|\bSearch\.usgs\.gov\b|\bSearch x-bot\b|\bSearqubot\b|\bSeb spider\b|\bSeekbot\b|\bSeeker\.lookseek\.com\b|\bSeeqbot\b|\bSeeqpod-vertical-crawler\b|\bSelflinkchecker\b|\bSemager\b|\bSemanticdiscovery\b|\bSemantifire\b|\bSemisearch\b|\bSemrush\b|\bSemrushBot\b|\bSeoengworldbot\b|\bSEOkicks\b|\bSEOlyticsCrawler\b|\bSeomoz\b|\bSEOprofiler\b|\bSeoscanners\b|\bSEOstats\b|\bSexsearcher\b|\bSeznam\b|\bSeznamBot\b|\bShablastbot\b|\bShadowwebanalyzer\b|\bShareaza\b|\bShelob\b|\bSherlock\b|\bShim-crawler\b|\bShodan\b|\bShopsalad\b|\bShopwiki\b|\bShowlinks\b|\bShowyoubot\b|\bSiclab\b|\bSilk\b|\bSimplepie\b|\bSiphon\b|\bSISTRIX\b|\bSitebeam\b|\bSitebot\b|\bSitecheck\b|\bSiteExplorer\b|\bSitefinder\b|\bSiteguardbot\b|\bSiteimprove\b|\bSiteLockSpider\b|\bSiteorbiter\b|\bSiteSnagger\b|\bSiteSucker\b|\bSite Sucker\b|\bSitesweeper\b|\bSitevigil\b|\bSitexpert\b|\bSkimbot\b|\bSkimwordsbot\b|\bSkreemrbot\b|\bSkywalker\b|\bSlackbot-LinkExpanding\b|\bSleipnir\b|\bSlow-crawler\b|\bSlySearch\b|\bSmart-crawler\b|\bSmartDownload\b|\bSmarte bot\b|\bSmartwit\.com\b|\bSMTBot\b|\bSnake\b|\bSnapbot\b|\bSnap\.com beta crawler\b|\bSnappreviewbot\b|\bSnappy\b|\bSnookit\b|\bSnooper\b|\bSnoopy\b|\bSocialRankIOBot\b|\bSocietyrobot\b|\bSocscibot\b|\bSoft411 directory\b|\bSogou\b|\bsogouspider\b|\bSogou web spider\b|\bSohu agent\b|\bSohu-search\b|\bSokitomi crawl\b|\bSolbot\b|\bSondeur\b|\bSootle\b|\bSosospider\b|\bSottopop\b|\bSpacebison\b|\bSpace bison\b|\bSpaceBison\b|\bSpace fung\b|\bSpammen\b|\bSpankBot\b|\bSpanner\b|\bSpatineo monitor controller\b|\bSpatineo serval controller\b|\bSpatineo serval getmapbot\b|\bSpbot\b|\bSpecial_archiver\b|\bSpeedy\b|\bSphere scout\b|\bSphider\b|\bSpiderengine\b|\bSpiderku\b|\bSpiderman\b|\bSpider\.terranautic\.net\b|\bSpinn3r\b|\bSpinne\b|\bSportcrew-bot\b|\bSproose\b|\bSputnikBot\b|\bSpyder3\.microsys\.com\b|\bSqlmap\b|\bSqlworm\b|\bSquidclamav_redirector\b|\bSquid-prefetch\b|\bSq webscanner\b|\bSqworm\b|\bSrevbot\b|\bSslbot\b|\bSsm agent\b|\bS~stremor-crawler\b|\bStackrambler\b|\bStardownloader\b|\bStatbot\b|\bStatcrawler\b|\bStatedept-crawler\b|\bSteeler\b|\bStegmann-bot\b|\bStero\b|\bStripper\b|\bStumbler\b|\bSuchclip\b|\bSucker\b|\bSucuri\b|\bSumeetbot\b|\bSumitbot\b|\bSummizebot\b|\bSummizefeedreader\b|\bSunrise xp\b|\bSuperBot\b|\bSuperHTTP\b|\bSuperlumin downloader\b|\bSuperpagesbot\b|\bSupremesearch\.net\b|\bSupybot\b|\bSurdotlybot\b|\bSurf\b|\bSurfbot\b|\bSurveyBot\b|\bSuzuran\b|\bSwebot\b|\bSwiftbot\b|\bSwish-e\b|\bSygolbot\b|\bSynapticwalker\b|\bSyntryx ant scout chassis pheromone\b|\bSysscan\b|\bSystemsearch-robot\b|\bSzukacz\b|\bT0PHackTeam\b|\bT8Abot\b|\bTailrank\b|\bTAkeOut\b|\bTalkro web-shot\b|\bTamu_crawler\b|\bTapuzbot\b|\bTarantula\b|\bTargetblaster\.com\b|\bTargetyournews\.com bot\b|\bTausdatabot\b|\bTaxinomiabot\b|\bTeamsoft wininet component\b|\bTecomi bot\b|\bTeezirbot\b|\bTeleport\b|\bTeleportPro\b|\bTelesoft\b|\bTelesphoreo\b|\bTelesphorep\b|\bTeradex mapper\b|\bTeragram_crawler\b|\bTerrawizbot\b|\bTestbot\b|\bTesting of bot\b|\bTextbot\b|\bThatrobotsite\.com\b|\bThe dyslexalizer\b|\bThe\.intraformant\b|\bThe Intraformant\b|\bTheNomad\b|\bTheophrastus\b|\bTheusefulbot\b|\bThumbbot\b|\bThumbnail\.cz robot\b|\bThumbshots-de-bot\b|\bT-h-u-n-d-e-r-s-t-o-n-e\b|\bTigerbot\b|\bTighttwatbot\b|\bTightTwatBot\b|\bTineye\b|\bTitan\b|\bToata\b|\bTocrawl\b|\bTo-dress_ru_bot_\b|\bTo-night-bot\b|\bTopicalizer\b|\bTopicblogs\b|\bToplistbot\b|\bTopserver php\b|\bTopyx-crawler\b|\bTouche\b|\bTourlentascanner\b|\bToweyabot\b|\bTpsystem\b|\bTraazi\b|\bTracemyfile\b|\bTransgenikbot\b|\bTravelbot\b|\bTravellazerbot\b|\bTravel-search\b|\bTreezy\b|\bTrendiction\b|\bTrendictionbot\b|\bTrendiction\.com\b|\bTrendiction\.de\b|\bTrex\b|\bTridentspider\b|\bTrovator\b|\bTrue_Robot\b|\bTscholarsbot\b|\bTsm translation-search-machine\b|\bTswebbot\b|\bTulipchain\b|\bTuringos\b|\bTurnitin\b|\bTurnitinBot\b|\bTutorgigbot\b|\bTweetedtimes bot\b|\bTweetmemebot\b|\bTwengaBot\b|\bTwice\b|\bTwikle\b|\bTwinuffbot\b|\bTwisted pagegetter\b|\bTwitturls\b|\bTwitturly\b|\bTygobot\b|\bTygoprowler\b|\bTyphoeus\b|\bUberbot\b|\bUcb-nutch\b|\bUdmsearch\b|\bUfam-crawler-\b|\bUltraseek\b|\bUnchaos\b|\bUnidentified\b|\bUnisterBot\b|\bUnitek uniengine\b|\bUniversalsearch\b|\bUnwindfetchor\b|\bUoftdb_experiment\b|\bUpdated\b|\bUrlappendbot\b|\bUrlblaze\b|\bUrlchecker\b|\bUrl-checker\b|\bUrlck\b|\bUrl control\b|\bUrldispatcher\b|\bUrl_gather\b|\bUrlspiderpro\b|\bURLy\.Warning\b|\bURLy Warning\b|\bUsaf afkn k2spider\b|\bUsasearch\b|\bU\.s\. government printing office\b|\bUss-cosmix\b|\bUsyd-nlp-spider\b|\bVacobot\b|\bVacuum\b|\bVadixbot\b|\bVagabondo\b|\bValidator\b|\bValkyrie\b|\bVB Project\b|\bVbseo\b|\bVCI\b|\bVci webviewer vci webviewer win32\b|\bVerbstarbot\b|\bVeriCiteCrawler\b|\bVerifactrola\b|\bVerity-url-gateway\b|\bVermut\b|\bVersus crawler\b|\bVersus\.integis\.ch\b|\bViasarchivinginformation\.html\b|\bVidibleScraper\b|\bVipr\b|\bVirus_detector\b|\bVirus-detector\b|\bVirusdie\b|\bVisbot\b|\bVishal for clia\b|\bVisweb\b|\bVital search'n urchin\b|\bVlad\b|\bVlsearch\b|\bVmbot\b|\bVocusbot\b|\bVoidEYE\b|\bVoil\b|\bVoilabot\b|\bVoltron\b|\bVortex\b|\bVoyager\b|\bVspider\b|\bW3c_unicorn\b|\bW3c-webcon\b|\bW3search\b|\bWacbot\b|\bWallpapers/3\.0\b|\bWallpapersHD\b|\bWanadoo\b|\bWASALive-Bot\b|\bWastrix\b|\bWater conserve portal\b|\bWater conserve spider\b|\bWatzbot\b|\bWauuu\b|\bWavefire\b|\bWaypath\b|\bWazzup\b|\bWbdbot\b|\bWBSearchBot\b|\bWeb-agent\b|\bWebalta\b|\bWebaltbot\b|\bWebAuto\b|\bWeb Auto\b|\bWebBandit\b|\bWebbot\b|\bWebbul-bot\b|\bWebcapture\b|\bWeb ceo online robot\b|\bWebcheck\b|\bWebclipping\.com\b|\bWebCollage\b|\bWeb Collage\b|\bWebCopier\b|\bWebcopy\b|\bWebcorp\b|\bWebcrawler\b|\bWeb crawler\b|\bWebcrawl\.net\b|\bWebdatacentrebot\b|\bWEBDAV\b|\bWeb downloader\b|\bWebdownloader for x\b|\bWebdup\b|\bWebemailextrac\b|\bWebEnhancer\b|\bWeb Enhancer\b|\bWebFetch\b|\bWeb Fetch\b|\bWebFuck\b|\bWeb Fuck\b|\bWebgather\b|\bWebgobbler\b|\bWebGo IS\b|\bWeb\.image\.collector\b|\bWeb image collector\b|\bWebImageCollector\b|\bWebimages\b|\bWebinator-search2\b|\bWebinator-wbi\b|\bWebindex\b|\bWeblayers\b|\bWebLeacher\b|\bWeblexbot\b|\bWeblinker\b|\bWeb link validator\b|\bWeblyzard\b|\bWeb magnet\b|\bWebmastercoffee\b|\bWebmasterworld extractor\b|\bWebmasterWorldForumBot\b|\bWebmeup-crawler\b|\bWebminer\b|\bWebmoose\b|\bWebot\b|\bWebPix\b|\bWeb Pix\b|\bWebReaper\b|\bWebripper\b|\bWebSauger\b|\bWeb Sauger\b|\bWebscan\b|\bWebsearchbench\b|\bWebshag\b|\bWebsite\b|\bWeb site downloader\b|\bWebsiteExtractor\b|\bWebsiteQuester\b|\bWebsite Quester\b|\bWeb-sniffer\b|\bWebspear\b|\bWebsphinx\b|\bWebspider\b|\bWebster\b|\bWebStripper\b|\bWebSucker\b|\bWeb Sucker\b|\bWebtrafficexpress\b|\bWebtrends link analyzer\b|\bWebvac\b|\bWebwalk\b|\bWebwasher\b|\bWebwatch\b|\bWebWhacker\b|\bWebxm\b|\bWebZIP\b|\bWeddings\.info\b|\bWenbin\b|\bWepa\b|\bWep search\b|\bWerelatebot\b|\bWeSEE\b|\bWhack\b|\bWhacker\b|\bWhatweb\b|\bWhirlpool web engine\b|\bWho\.is Bot\b|\bWhowhere robot\b|\bWidow\b|\bWikiabot\b|\bWikio\b|\bWikiwix-bot-\b|\bWinhttp\b|\bWinHTTrack\b|\bWire\b|\bWisebot\b|\bWiseGuys Robot\b|\bWISENutbot\b|\bWish-la\b|\bWish-project\b|\bWisponbot\b|\bWmcai-robot\b|\bWminer\b|\bWmsbot\b|\bWonderbot\b|\bWoobot\b|\bWoriobot\b|\bWorldshop\b|\bWorqmada\b|\bWotbox\b|\bWprecon\b|\bWPScan\b|\bWume_crawler\b|\bWww collector\b|\bWww-collector-e\b|\bWWW-Collector-E\b|\bWww-mechanize\b|\bWWW-Mechanize\b|\bWWW\:\:Mechanize\b|\bWWWOFFLE\b|\bWwwrobot\b|\bWwwster\b|\bWwwwanderer\b|\bWwwxref\b|\bWysigot\b|\bX09Mozilla\b|\bX22Mozilla\b|\bXaldon\b|\bXaldon_WebSpider\b|\bXaldon WebSpider\b|\bX-clawler\b|\bX-crawler\b|\bXenu\b|\bXerka metabot\b|\bXerka webbot\b|\bXget\b|\bXirq\b|\bXmarksfetch\b|\bXpymep1\.exe\b|\bXqrobot\b|\bYacybot\b|\bYacy\.net\b|\bYanga worldsearch bot\b|\bYarienavoir\.net\b|\bYasaklibot\b|\bYats crawler\b|\bYbot\b|\bYebolbot\b|\bYellowjacket\b|\bYeti\b|\bY\!j\b|\bYolinkbot\b|\bYooglifetchagent\b|\bYoono\b|\bYottacars_bot\b|\bYoudaoBot\b|\bYourls\b|\bZ-add link checker\b|\bZade\b|\bZagrebin\b|\bZao\b|\bZauba\b|\bZauba\.io\b|\bZedzo\.validate\b|\bZermelo\b|\bZeus\b|\bZgrab\b|\bZibber-v\b|\bZimeno\b|\bZing-bottabo\b|\bZipppbot\b|\bZitebot\b|\bZmEu\b|\bZongbot\b|\bZoomspider\b|\bZotag search\b|\bZsebot\b|\bZuibot\b|\bZumBot\b|\bZyBorg\b|\bZyte\b|\bapplebot\b|\barchive\.org_bot\b|\bBaidu\b|\bBaiduspider\b|\bCFNetwork\b|\bFirefox/21\.0\b|\bia_archiver\b|\bMozilla/4\.76\b|\bMSIE 5\.\b|\bMSIE 6\.0\b|\bQwantify\b|\bSafeDNSBot\b|\bUptimebot\b|\bYahoo\! Slurp\b|\bSeekport\b

failregex = (?i)<HOST> -.*"(GET|POST|HEAD|OPTIONS).*HTTP.*(?:%(badbotscustom)s|%(badbots)s).*"$

ignoreregex = 

[badbots]
enabled = true
port    = http,https
filter  = badbots
logpath = /var/log/virtualmin/*_access_log
maxretry = 1
findtime = 3600
bantime = 300

Une fois fait il vous suffit de recharger fail2ban avec cette commande :

service fail2ban reload

Et voilà vous êtes protégé contre ces saloperies de robots!