robots.txt managed by vBSEO

It would be great to have the robots.txt file managed by vbseo. Like having a section in vbseocp dedicated to it; where we can select which URLs to allow, what to deny, the crawl delay, bots to which rules should apply, etc. In addition we can have a textarea for custom robot entries.

Some of the settings in the CP could be like :

- Disallow showpost.php : Yes / No
- Disallow /archive/ : Yes / No
- Disallow member lists : Yes / No
- Crawl Delay : 10

And of course an option to disable this feature, so users who want to manage it on their own can do it like they are doing it now, via the filesystem (to avoid opening the CP everytime they want to make a change).


Inspired by : Understanding: robots.txt

Hello Shadab ,

That may not work in many cases as robots.txt suppose to be uploaded to root folder and most users are using a folder for their forum installations on their site.

Ah... Yes. Didn't think of it. It wouldn't work in that case by default.
But maybe a .htaccess rule in root can take care of it ?

Like RewriteRule robots.txt /folder/vbseo.php?vbseourl=robots.txt [L]

Nice suggestion.

That may not work in many cases as robots.txt suppose to be uploaded to root folder and most users are using a folder for their forum installations on their site.

Can't it be done via some htaccess rewrite maybe, just like sitemap generator creates sitemap.gz file in an inside data folder but we are able to access it from root.

Originally Posted by webwizzy

Nice suggestion.

Thank you

Originally Posted by webwizzy

just like sitemap generator creates sitemap.gz file in an inside data folder but we are able to access it from root.

Not always. The sitemap is served from the forum root. (which may or may not coincide with the site root).
If the forum is in /forums/, the sitemap will be at /forums/sitemap_index.xml.gz

We could definitely find a technical solution for this, and I do believe the htaccess method used for the sitemap would be applicable.

Robots.txt file should be readable on root folder. If it is a redirect , it cannot be read correctly. Thats why i donot think a .htaccess rule will solve the problem. I thought about adding a robotx.txt file to the docs folder so user can upload that but even that will not work because the robots.txt file should be edited by the user itself if they have their forum under a folder. Maybe it is best to create a robots.txt generator on vBSEO.com

Originally Posted by Mert Gökçeimam

If it is a redirect , it cannot be read correctly.

No, not the redirect. I'm talking about rewriting it.

Just like vBSEO serves pseudo-html pages via server-side rewriting for normal forum pages, it can also easily serve a text file (ie, robots.txt) pre-processed by a php script.