User-agent: BecomeBot Crawl-Delay: 3333333333333333333333333333333333333333333333333333333333333333333333330 Disallow: / User-agent: TurnitinBot Disallow: / User-agent: discobot Disallow: / User-agent: Theophrastus Disallow: / User-agent: Spock Crawler Disallow: * User-agent: e-SocietyRobot Disallow: / User-agent: heritrix Disallow: / User-agent: noxtrumbot/1.0 Disallow: / User-agent: noxtrumbot/1.0 Disallow: * User-agent: dotbot Disallow: / User-agent: Mag-Net Disallow: / User-agent: Wget Disallow: / User-agent: MSIECrawler Disallow: / User-agent: SlySearch Disallow: / User-agent: asterias Disallow: / User-agent: Mercator-2.0 Disallow: / User-agent: SSM Agent 1.0 Disallow: / User-agent: Microsoft URL Control Disallow: / User-agent: Mercator Disallow: / User-agent: TurnitinBot Disallow: / User-agent: NPBot Disallow: / User-Agent: OmniExplorer_Bot/1.09 Crawl-delay: 5 User-agent: http://www.almaden.ibm.com/cs/crawler Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler Disallow: * User-agent: Yandex Disallow: / User-agent: YandexSomething Disallow: / User-agent: Yandex Disallow: * User-agent: YandexSomething Disallow: * User-Agent: sitecheck.internetseer.com Disallow: / User-Agent: QPCreep Test Rig Disallow: / User-Agent: QPCreep Disallow: / User-Agent: Spinn3r Disallow: / User-agent: * Disallow: /contacts.html Disallow: /logs Disallow: /stats Disallow: /cgi-bin Disallow: /flash Disallow: /test Disallow: /staff