[Midnightbsd-cvs] www [548] trunk/robots.txt: add host and sitemap directives

[laffer1 at midnightbsd.org]

Revision: 548
          http://svnweb.midnightbsd.org/www/?rev=548
Author:   laffer1
Date:     2014-06-13 08:05:36 -0400 (Fri, 13 Jun 2014)
Log Message:
-----------
add host and sitemap directives

Modified Paths:
--------------
    trunk/robots.txt

Modified: trunk/robots.txt
===================================================================
--- trunk/robots.txt	2014-06-13 12:03:39 UTC (rev 547)
+++ trunk/robots.txt	2014-06-13 12:05:36 UTC (rev 548)
@@ -5,6 +5,8 @@
 Disallow: /js/
 Disallow: /assets/
 Crawl-delay: 5
+Sitemap: http://www.midnightbsd.org/sitemap.xml.gz
+Host: www.midnightbsd.org
 
 User-agent: BLEXBot
 Crawl-delay: 10