• Rank Fishkin outlines mistakes with Robots.txt file that hurt SEO
  • Provides two key take aways (click link).
  • 3 Ways to Block a Page from the search engines: Robots.txt, Meta NoIndex, Nofollowing Links
    • Robots.txt stops them from crawling it again, but does not stop it from being in the index.
    • Meta Noindex lets them get the page, but not display it in the SERPs
    • Nofollowing can save link juice (but pages still end up in search index sometimes)
  • A blocked page cannot share link value (i.e. juice) even if it has some to give.