Robots.txt

Hello-

I've been told that is not necessary to have such file in my directory as long as I have the .htaccess file in there. Is this correct:

Here is how my .ht access looks like:
Rewrite Rule ^/index.HTML /forums/index.Php [L,R=301]

and here is how my existing robot.txt looks like:
User-agent: *
Disallow:



Any input is greatly appreciated.

Thanks

Effectively, your robots.txt file is blank, so it will do nothing.

Robots.txt ONLY works in the root of your site. There is no such thing as site.com/forums/robots.txt (at least as far as bots are concerned)


All that said, I don't really understand what your question is...

My question is:
Do I need to have a robots.txt or htaccess is enough and will work just ast well the robots.txt. Also do you have any recommendations on what should I have in my robots.txt? Like a sample template?

Sorry for so many questions. I'm new at this.

Thanks

robots and htaccess are 2 different things

the robots.txt file is a file where you can block certain pages/folders/scripts from search engine crawling.

htaccess is used for many things, such as re-writing, secure directories, or loading modules or apache settings on the fly, etc etc.


what YOU need in your robots file will vary, as it's what's set up on your site.

The basic jist of the robots file is to exclude all non-important stuff and old scripts from getting indexed.

Is there a wild card command that I can use to block certain files from been scanned by robots? For example what if I want to block all PHP files at once? Those are really not necessary correct?


Thanks

Originally Posted by hdeassis

Is there a wild card command that I can use to block certain files from been scanned by robots? For example what if I want to block all PHP files at once?

Code:

User-agent: *
Disallow: *.php

See Post #4 in thread : Robots.txt, by briansol.

Thanks everyoone for their input.