View RSS Feed

Recent Blogs Posts

  1. robots.txt

    by , 07-25-2010 at 05:32 PM
    Quote Originally Posted by KURTZ View Post
    thanks Brian just tested this:

    Code:
    # Allow Archive.org to save snapshots of everything
    User-agent: ia_archiver
    Allow: /
    
    # Tame yahoo... it tends to eat a ton of resources without a delay
    User-agent: Slurp
    Crawl-delay: 60
    
    
    #list individual pages and files here that all bots should ignore, as well as group extentions.
    #If you re-write everything to .html, you can disallow *.php, but note that if you don't have a CRR
    ...
    Categories
    SEO Tips