Briansol's Ultimate Guide to vBSEO

Originally Posted by briansol

because you just told hackers where they are.

No,, I did not, I understand just what your saying now.

So if the robots.txt file can be read by anyone

How do you stop them from being crawled?

your private areas should be protected by htpasswd directives.
htpasswd - Manage user files for basic authentication - Apache HTTP Server
http://www.htaccesstools.com/htpasswd-generator/

Does Technorati spider follow these robots.txt rules? Isn't it external.php disallowed to these kind of spiders when there is a "Disallow: *.php" rule in robots file?

robots files are merely 'suggestions'. good bots will obey, but not all of them do.

Yes I know

I'm getting " The URL you provided is invalid. Please check the URL and try again" message when I'm trying to claim my forum on Technorati and I wonder if the "Disallow: *.php"could be the the reason.

Thanks

technorati goes to the homepage, so make sure the code is on the homepage. you could make a special rule to allow techorati through (i do'nt know the bot name off hand)

Originally Posted by briansol

you could make a special rule to allow techorati through (i do'nt know the bot name off hand)

Yes I didn't think of that. I'm gonna give it a try...

Thanks

With the new 2.5 sitemap settings allowing a priority range, what range(s) do you recommend for those using your settings on 2.2 and now upgrading to 2.5?

Hi guys, i know i still need to update this. Real life events have kept me away from the forums recently and I haven't had time to get this updated to 3.8/3.3 yet.

I know you're busy, but an update to this would be fantastic

So should I Disable the archive?

thanks

yes

Ok, so I have followed your tutorial and done what you suggested. I added the robots.txt file and where do I import the vbseo_all.xml file?

I am running vbseo 3.3.0 and VB 3.8.3

Thanks

You can import vbseo_all.xml file from vBSEO Control Panel.

So, I don't think this is exactly what you use anymore by the looks of hondaswap's links and my sites.