SEO
vBulletin Search Engine Optimization
Blocking bots in robots.txt - how do they see URLs?
This is a discussion on Blocking bots in robots.txt - how do they see URLs? within the General Discussion forums, part of the vBSEO SEO Plugin category; I have my archive/ set to rewrite to sitemap/ in vBSEO CP . I have noticed spiders still showing the ...
| |||||||
Enhancing 80 million pages. | Register | FAQ | Members List | Calendar | Search | Today's Posts | Mark Forums Read |
|
#1
06-20-2007, 06:53 PM
| ||||
| ||||
| Blocking bots in robots.txt - how do they see URLs?
I have my archive/ set to rewrite to sitemap/ in vBSEO CP. I have noticed spiders still showing the url they are following is archive/xxxx in whos online. I also see them seeing threads as showthread.php as the URL they are following. If i have archive/ blocked in my robots.txt am i effectivly blocking them using my sitemap or would sitemap/ have to be in the robots.txt to do that. Basically does blocking an old dynamic url effectivly block the rewritten ones also? Like if i blocked showthread.php would it stop my rewritten threads being indexed? Just trying to understand bots better. Thanks.       |
 |
| Thread Tools | |
| |
Similar Threads | ||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| Google Sitemap-Listing of old url's option-from 1.7 | T2DMan | Member Articles | 0 | 04-02-2006 08:46 AM |
All times are GMT -4. The time now is 03:08 AM.
