Results 1 to 9 of 9

Robot.txt

This is a discussion on Robot.txt within the Pre-Sales Questions forums, part of the Announcements & Pre-Sales category; hi when i add my site map in Google webmaster i receive an error : Code: Network unreachable: robots.txt unreachable ...

  1. #1
    Member
    Real Name
    susan
    Join Date
    Sep 2008
    Location
    IRAN
    Posts
    62
    Liked
    0 times

    Question Robot.txt

    hi

    when i add my site map in Google webmaster i receive an error :


    Code:
    Network unreachable: robots.txt unreachable
    We were unable to crawl your Sitemap because we found a robots.txt file at the root of your site but were unable to download it. Please ensure that it is accessible or remove it completely.









    i delete this file but Google say again to me u have this File !!

    please help me

  2. #2
    Member
    Real Name
    susan
    Join Date
    Sep 2008
    Location
    IRAN
    Posts
    62
    Liked
    0 times
    Hello !!

  3. #3
    Junior Member
    Real Name
    Krish
    Join Date
    Sep 2008
    Posts
    9
    Liked
    0 times
    Whats the permission of robots.txt file and whats the content do you have in it?

  4. #4
    Member
    Real Name
    susan
    Join Date
    Sep 2008
    Location
    IRAN
    Posts
    62
    Liked
    0 times
    Code:
    # Allow Archiver
    User-agent: ia_archiver
    Allow: /
    
    
    User-agent: Slurp
    Crawl-delay: 60
    
    
    User-agent: *
    
    Disallow: *.php
    Disallow: *.js
    Disallow: *.jsp
    Disallow: *.cfm
    Disallow: *.asp
    Disallow: *.html
    Disallow: *.htm
    Disallow: *.aspx
    Disallow: *.cgi
    
    Disallow: /includes/
    Disallow: /install/
    Disallow: /customavatars/
    Disallow: /archive/
    Disallow: /sitemap/
    
    Sitemap: http://forums.parsforums.net/sitemap_index.xml.gz

  5. #5
    Member
    Real Name
    susan
    Join Date
    Sep 2008
    Location
    IRAN
    Posts
    62
    Liked
    0 times
    is this ok ? or have a problem ?

  6. #6
    Senior Member
    Real Name
    Alex
    Join Date
    Mar 2007
    Location
    Italy
    Posts
    508
    Liked
    0 times
    you need to wait maybe few hours or a day for the next attempt to crawl your site by google

  7. #7
    Junior Member
    Real Name
    Krish
    Join Date
    Sep 2008
    Posts
    9
    Liked
    0 times
    Where have you placed robots.txt?
    check it using New Robots.txt Syntax Checker: a validator for robots.txt files

  8. #8
    Member
    Real Name
    susan
    Join Date
    Sep 2008
    Location
    IRAN
    Posts
    62
    Liked
    0 times
    thanks , its about 777 and 666 !!

  9. #9
    Senior Member briansol's Avatar
    Real Name
    Brian
    Join Date
    Apr 2006
    Location
    Central CT, USA
    Posts
    6,981
    Liked
    8 times
    644 is generally the setting you want for regular files.

Similar Threads

  1. robot.txt dosyasında ne yazmalı
    By bendensin890 in forum Türkçe
    Replies: 22
    Last Post: 02-03-2009, 01:47 PM
  2. Robot Text Do I need it?
    By photochimper in forum General Discussion
    Replies: 2
    Last Post: 05-21-2008, 12:31 AM
  3. vbseo and robot.txt?
    By Julio in forum General Discussion
    Replies: 4
    Last Post: 12-31-2007, 06:25 PM
  4. Robot.txt file
    By BamaStangGuy in forum General Discussion
    Replies: 3
    Last Post: 06-17-2006, 01:09 PM
  5. I need some help - concerned about robot activity
    By Mike in forum Off-Topic & Chit Chat
    Replies: 10
    Last Post: 01-17-2006, 10:31 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •