www/trunk/robots.txt

User-agent: *
Disallow: /bin/
Disallow: /cgi-bin/
Disallow: /shadowbox/
Disallow: /assets/
Disallow: /magus/auth/
Crawl-delay: 5
Sitemap: http://www.midnightbsd.org/sitemap.xml.gz
Host: www.midnightbsd.org
Revision:	605
Committed:	Thu Jun 25 23:16:38 2015 UTC (8 years, 10 months ago) by laffer1
Content type:	text/plain
File size:	206 byte(s)
Log Message:	add another directory
#	Content
1	User-agent: *
2	Disallow: /bin/
3	Disallow: /cgi-bin/
4	Disallow: /shadowbox/
5	Disallow: /assets/
6	Disallow: /magus/auth/
7	Crawl-delay: 5
8	Sitemap: http://www.midnightbsd.org/sitemap.xml.gz
9	Host: www.midnightbsd.org