[Midnightbsd-cvs] www [547] trunk/robots.txt: crawl delay slowdown
laffer1 at midnightbsd.org
laffer1 at midnightbsd.org
Fri Jun 13 08:03:39 EDT 2014
Revision: 547
http://svnweb.midnightbsd.org/www/?rev=547
Author: laffer1
Date: 2014-06-13 08:03:39 -0400 (Fri, 13 Jun 2014)
Log Message:
-----------
crawl delay slowdown
Modified Paths:
--------------
trunk/robots.txt
Modified: trunk/robots.txt
===================================================================
--- trunk/robots.txt 2014-06-13 11:59:32 UTC (rev 546)
+++ trunk/robots.txt 2014-06-13 12:03:39 UTC (rev 547)
@@ -4,6 +4,7 @@
Disallow: /css/
Disallow: /js/
Disallow: /assets/
+Crawl-delay: 5
User-agent: BLEXBot
Crawl-delay: 10
More information about the Midnightbsd-cvs
mailing list