www/trunk/robots.txt

User-agent: *
Disallow: /cgi-bin/
Disallow: /shadowbox/
Disallow: /assets/
Disallow: /magus/auth/
Crawl-delay: 5
Sitemap: http://www.midnightbsd.org/sitemap.xml.gz
Host: www.midnightbsd.org
Revision:	593
Committed:	Fri Apr 17 12:06:52 2015 UTC (9 years ago) by laffer1
Content type:	text/plain
File size:	190 byte(s)
Log Message:	dont hide css or js because googlebot makes wrong assumptions about mobile for their new filtering.
#	Content
1	User-agent: *
2	Disallow: /cgi-bin/
3	Disallow: /shadowbox/
4	Disallow: /assets/
5	Disallow: /magus/auth/
6	Crawl-delay: 5
7	Sitemap: http://www.midnightbsd.org/sitemap.xml.gz
8	Host: www.midnightbsd.org