[Midnightbsd-cvs] www [549] trunk/robots.txt: just do a standard * include now
laffer1 at midnightbsd.org
laffer1 at midnightbsd.org
Fri Jun 13 08:06:42 EDT 2014
Revision: 549
http://svnweb.midnightbsd.org/www/?rev=549
Author: laffer1
Date: 2014-06-13 08:06:41 -0400 (Fri, 13 Jun 2014)
Log Message:
-----------
just do a standard * include now
Modified Paths:
--------------
trunk/robots.txt
Modified: trunk/robots.txt
===================================================================
--- trunk/robots.txt 2014-06-13 12:05:36 UTC (rev 548)
+++ trunk/robots.txt 2014-06-13 12:06:41 UTC (rev 549)
@@ -7,6 +7,3 @@
Crawl-delay: 5
Sitemap: http://www.midnightbsd.org/sitemap.xml.gz
Host: www.midnightbsd.org
-
-User-agent: BLEXBot
-Crawl-delay: 10
More information about the Midnightbsd-cvs
mailing list