[Midnightbsd-cvs] www [546] trunk/robots.txt: add crawl delay for BLEXBot

[laffer1 at midnightbsd.org]

Revision: 546
          http://svnweb.midnightbsd.org/www/?rev=546
Author:   laffer1
Date:     2014-06-13 07:59:32 -0400 (Fri, 13 Jun 2014)
Log Message:
-----------
add crawl delay for BLEXBot

Modified Paths:
--------------
    trunk/robots.txt

Modified: trunk/robots.txt
===================================================================
--- trunk/robots.txt	2014-04-11 23:59:55 UTC (rev 545)
+++ trunk/robots.txt	2014-06-13 11:59:32 UTC (rev 546)
@@ -4,3 +4,6 @@
 Disallow: /css/
 Disallow: /js/
 Disallow: /assets/
+
+User-agent: BLEXBot
+Crawl-delay: 10