[Midnightbsd-cvs] www [667] trunk/robots.txt: don't index opengrok

[laffer1 at midnightbsd.org]

Revision: 667
          http://svnweb.midnightbsd.org/www/?rev=667
Author:   laffer1
Date:     2015-10-15 22:32:32 -0400 (Thu, 15 Oct 2015)
Log Message:
-----------
don't index opengrok

Modified Paths:
--------------
    trunk/robots.txt

Modified: trunk/robots.txt
===================================================================
--- trunk/robots.txt	2015-10-11 15:48:32 UTC (rev 666)
+++ trunk/robots.txt	2015-10-16 02:32:32 UTC (rev 667)
@@ -2,10 +2,12 @@
 Disallow: /bin/
 Disallow: /cgi-bin/
 Disallow: /html5lightbox/index.html
+Disallow: /html5lightbox/skins/index.html
 Disallow: /assets/
 Disallow: /magus/auth/
 Disallow: /ftp/MidnightBSD/distfiles/*.asc
 Disallow: /ftp/MidnightBSD/distfiles/*.sig
+Disallow: /opengrok/
 Crawl-delay: 5
 Sitemap: http://www.midnightbsd.org/sitemap.xml
 Sitemap: http://www.midnightbsd.org/sitemap1.xml