Results 1 to 4 of 4

Robots.txt file changes

This is a discussion on Robots.txt file changes within the General Discussion forums, part of the vBSEO Google/Yahoo Sitemap category; I converted from phpbb3 and had phpbbseo. With that modification I had to add a few things to robots.txt. I ...

  1. #1
    Member Array
    Real Name
    Justin
    Join Date
    Aug 2009
    Posts
    32
    Liked
    0 times

    Robots.txt file changes

    I converted from phpbb3 and had phpbbseo. With that modification I had to add a few things to robots.txt. I am wondering if I need to remove any of the following from that file:

    Disallow: /posting.php
    Disallow: /search.php?
    Disallow: /ucp.php
    Disallow: /mcp.php
    Disallow: /post
    Disallow: /faq.php
    Disallow: /viewtopic.php
    Disallow: /viewforum.php

  2. #2
    Senior Member Array
    Real Name
    Marco Mamdouh
    Join Date
    May 2008
    Location
    Egypt
    Posts
    2,627
    Liked
    3 times
    Blog Entries
    1
    If you didn't have any additional .php file and need it to index you can safety use this robots.txt
    Code:
    User-agent: *
    
    Disallow: *.php
    
    Sitemap: http://domain . com/sitemap_index.xml.gz
    But if you have .php file and need to index it in search engines you can use this file (Recommended to use)
    Code:
    User-agent: *
    
    Disallow: /calendar.php
    Disallow: /login.php
    Disallow: /member.php
    Disallow: /memberlist.php
    Disallow: /newthread.php
    Disallow: /newreply.php
    Disallow: /printthread.php
    Disallow: /private.php
    Disallow: /profile.php
    Disallow: /register.php
    Disallow: /search.php
    Disallow: /sendmessage.php
    Disallow: /member.php
    Disallow: /archive/
    Disallow: /misc.php
    
    Sitemap: http://domain . com/sitemap_index.xml.gz

  3. #3
    Member Array
    Real Name
    Justin
    Join Date
    Aug 2009
    Posts
    32
    Liked
    0 times
    Thanks that is what I wanted

  4. #4
    Senior Member Array
    Real Name
    Marco Mamdouh
    Join Date
    May 2008
    Location
    Egypt
    Posts
    2,627
    Liked
    3 times
    Blog Entries
    1
    You are welcome

Similar Threads

  1. Robots.txt now support the sitemap file
    By Mohamed Mostafa in forum General Discussion
    Replies: 35
    Last Post: 08-24-2007, 09:33 AM
  2. Please review my robots.txt file...
    By Sonnie in forum General Discussion
    Replies: 11
    Last Post: 09-08-2006, 03:30 PM
  3. Question about location of robots.txt file
    By Mike in forum General Discussion
    Replies: 2
    Last Post: 05-22-2006, 01:05 PM
  4. Robots file and the rewritten urls....
    By BamaStangGuy in forum General Discussion
    Replies: 9
    Last Post: 02-28-2006, 11:10 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •