Revision 667 - (view) (download) (annotate) - [select for diffs]
Modified Fri Oct 16 02:32:32 2015 UTC (8 years, 6 months ago) by laffer1
File length: 807 byte(s)
Diff to previous 666

don't index opengrok

add sitemaps

Revision 665 - (view) (download) (annotate) - [select for diffs]
Modified Sun Oct 11 15:36:16 2015 UTC (8 years, 6 months ago) by laffer1
File length: 306 byte(s)
Diff to previous 605

add more exclusions nad drop a bunch of old files

add another directory

Revision 593 - (view) (download) (annotate) - [select for diffs]
Modified Fri Apr 17 12:06:52 2015 UTC (9 years ago) by laffer1
File length: 190 byte(s)
Diff to previous 591

dont hide css or js because googlebot makes wrong assumptions about mobile for their new filtering.

block auth

Revision 549 - (view) (download) (annotate) - [select for diffs]
Modified Fri Jun 13 12:06:41 2014 UTC (9 years, 10 months ago) by laffer1
File length: 198 byte(s)
Diff to previous 548

just do a standard * include now

add host and sitemap directives

Revision 547 - (view) (download) (annotate) - [select for diffs]
Modified Fri Jun 13 12:03:39 2014 UTC (9 years, 10 months ago) by laffer1
File length: 158 byte(s)
Diff to previous 546

crawl delay slowdown

add crawl delay for BLEXBot

Revision 365 - (view) (download) (annotate) - [select for diffs]
Modified Mon May 25 21:28:43 2009 UTC (14 years, 10 months ago) by laffer1
File length: 106 byte(s)
Diff to previous 272

Add cgi-bin directory stuff so that we don't get hammered by bots

add a robots.txt file.  We'll deal with this later.  I'm tired of the 404s