SE bots seem to ignore robots.txt exclusion of newreply.php

711 · #1 04-06-2008, 01:36 AM

I have installed the Track Guest Visits modification, which shows guest and SE bot activity on a forum.

While browsing the spider activity page, I noticed lots of request to newreply.php, even though I have that page disallowed in my robots.txt.

Here is a screenshot of what I mean:

Notice all the:

Called with DO = 'newreply'

Here is my current robots.txt content:

HTML Code:

User-agent: *
Disallow: /forums/admincp/
Disallow: /forums/attachment.php
Disallow: /forums/avatar.php
Disallow: /forums/calendar.php
Disallow: /forums/cgi-bin/
Disallow: /forums/clientscript/
Disallow: /forums/cron.php
Disallow: /forums/editpost.php
Disallow: /forums/faq.php
Disallow: /forums/image.php
Disallow: /forums/images/
Disallow: /forums/includes/
Disallow: /forums/install/
Disallow: /forums/ispy.php
Disallow: /forums/joinrequests.php
Disallow: /forums/login.php
Disallow: /forums/member.php
Disallow: /forums/member2.php
Disallow: /forums/misc.php
Disallow: /forums/modcp/
Disallow: /forums/moderator.php
Disallow: /forums/newreply.php
Disallow: /forums/newthread.php
Disallow: /forums/online.php
Disallow: /forums/payments.php
Disallow: /forums/poll.php
Disallow: /forums/postings.php
Disallow: /forums/printthread.php
Disallow: /forums/private.php
Disallow: /forums/private2.php
Disallow: /forums/profile.php
Disallow: /forums/register.php
Disallow: /forums/reputation.php
Disallow: /forums/search.php
Disallow: /forums/sendmessage.php
Disallow: /forums/sendmessage.php?do=
Disallow: /forums/sendtofriend.php
Disallow: /forums/showgroups.php
Disallow: /forums/showpost.php
Disallow: /forums/sitemap/
Disallow: /forums/spy.php
Disallow: /forums/subscription.php
Disallow: /forums/tags/
Disallow: /forums/threadrate.php
Disallow: /forums/upload.php
Disallow: /forums/usercp.php
Disallow: /forums/weeklystats.php
Disallow: /forums/statistics.php
Disallow: /forums/stats.php
Disallow: /forums/infraction.php
Disallow: /forums/ajax.php
Disallow: /forums/arcade.php
 
User-agent: Slurp
Disallow: /gallery/
Crawl-delay: 90
Sitemap: http://www.entropiaforum.com/forums/sitemap_index.xml.gz

Any ideas or suggestions on how to prevent the spiders from visiting newreply.php (or any other unwanted pages) would be appreciated, thank!

briansol · #2 04-06-2008, 05:47 PM

the robots file is merely a suggestion to the SE. It won't 100% stop anything.

711 · #3 04-06-2008, 06:30 PM

Quote:

Originally Posted by briansol

the robots file is merely a suggestion to the SE. It won't 100% stop anything.

True, though I thought Googlebot tended to be more "well-behaved", and usually respected the robots.txt suggestions?

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Blocking bots in robots.txt - how do they see URLs?	Dave Hybrid	General Discussion	0	06-20-2007 06:53 PM
Just a test, please ignore	Lian	Off-Topic & Chit Chat	1	01-16-2007 02:00 AM

SEO

SE bots seem to ignore robots.txt exclusion of newreply.php

SE bots seem to ignore robots.txt exclusion of newreply.php

Similar Threads