www/trunk/robots.txt

User-agent: *
Disallow: /cgi-bin/
Disallow: /shadowbox/
Disallow: /css/
Disallow: /js/
Disallow: /assets/
Crawl-delay: 5
Sitemap: http://www.midnightbsd.org/sitemap.xml.gz
Host: www.midnightbsd.org
Revision:	549
Committed:	Fri Jun 13 12:06:41 2014 UTC (9 years, 10 months ago) by laffer1
Content type:	text/plain
File size:	198 byte(s)
Log Message:	just do a standard * include now
#	Content
1	User-agent: *
2	Disallow: /cgi-bin/
3	Disallow: /shadowbox/
4	Disallow: /css/
5	Disallow: /js/
6	Disallow: /assets/
7	Crawl-delay: 5
8	Sitemap: http://www.midnightbsd.org/sitemap.xml.gz
9	Host: www.midnightbsd.org