Results 1 to 2 of 2

About robots.txt

This is a discussion on About robots.txt within the General Discussion forums, part of the vBulletin SEO Discussion category; Hello everyone, I want to block certain threads manually via robots.txt and I have a question doing so: eg. if ...

  1. #1
    Member
    Join Date
    Jan 2006
    Posts
    31
    Liked
    0 times

    About robots.txt

    Hello everyone,
    I want to block certain threads manually via robots.txt and I have a question doing so:
    eg. if I want to include a thread with several pages into robots.txt like this:
    www.mysite.com/forums/1234-testthread.html
    www.mysite.com/forums/1234-testthread-2.html
    www.mysite.com/forums/1234-testthread-3.html

    is it possible to have a syntax like this to include all pages of the thread without typing everyone?
    www.mysite.com/forums/1234-*

    I tried this one but without success.
    Can someone help me out?

    Thank you so much in advance
    Diver

  2. #2
    Senior Member briansol's Avatar
    Real Name
    Brian
    Join Date
    Apr 2006
    Location
    Central CT, USA
    Posts
    6,981
    Liked
    8 times
    the * should work. but you shouldn't include your .com. just relative path from root.

    not sure if you can end in a *, but the * is valid.

    try
    Disallow: /forums/1234-testthread*.html


    -=-------------

    also, keep in mind that it may take a month before this gets acknowledged. it's certainly not an instant process.

Similar Threads

  1. robots.txt
    By Zenith in forum General Discussion
    Replies: 64
    Last Post: 12-01-2010, 07:52 PM
  2. robots.txt
    By wolf32 in forum General Discussion
    Replies: 0
    Last Post: 06-30-2007, 09:29 PM
  3. Robots.txt help
    By Dave Hybrid in forum General Discussion
    Replies: 2
    Last Post: 05-29-2007, 10:16 AM
  4. No Robots.txt
    By friendly in forum Troubleshooting
    Replies: 1
    Last Post: 10-26-2006, 10:30 AM
  5. robots.txt
    By croportal in forum General Discussion
    Replies: 1
    Last Post: 02-05-2006, 12:31 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •