Best way to Robots.txt out a temporary web page (CMS Article)

I have been doing some reading and saw that its a good idea whenever you have a page that you know will be temporary to robots.txt that page out so that google wont index it for better SEO...

So if this is in fact true then what is the best way to robots.txt out a CMS article for example?

Would you do something like this and keep the /content/ in there as well?

User-agent: *
Disallow: /content/temporary-example-page.html

Yes you can use it like that also you can give exact url. But google will not index as long as it's not in sitemaps or linked from your website where spiders visits. If it's background then no need for robots.txt. Once it's spidered then you can use 301 to original page.

Robots.txt is not a guarantee. For example yahoo slurp bots may still deny robots.txt. At least it takes time for them to follow robots.txt rules. Really no need to remove any cms articles. The more content is the better. But you can remove low quality pages like member profiles and keep them away spidered. I use this robots.txt for my forum. You can check. http://www.webmastersitesi.com/robots.txt

Robots runs in root only, so you need to address the path as if site.com was in front of it.

Originally Posted by webmastersitesi

Yes you can use it like that also you can give exact url. But google will not index as long as it's not in sitemaps or linked from your website where spiders visits. If it's background then no need for robots.txt. Once it's spidered then you can use 301 to original page.

Robots.txt is not a guarantee. For example yahoo slurp bots may still deny robots.txt. At least it takes time for them to follow robots.txt rules. Really no need to remove any cms articles. The more content is the better. But you can remove low quality pages like member profiles and keep them away spidered. I use this robots.txt for my forum. You can check. http://www.webmastersitesi.com/robots.txt

Thanks for your response. So what is the best way to make it so it's not included in the sitemaps? I might have missed it but is there a disallow section somewhere in the vbseo control panel which tells what you don't want submitted in the sitemap? Or is it done through webmaster tools somewhere?

I don't really understand what you mean about being in background?

Thanks for sharing your robots.txt. I only have a couple items on mine right now but I'm slowly but surely understanding which ones to include for best SEO on my site thanks to this website and everyone here.

Originally Posted by Brian Cummiskey

Robots runs in root only, so you need to address the path as if site.com was in front of it.

Ok so what you're saying is I need to use the complete URL and make it look like this instead right:

User-agent: *
Disallow: www.site.com/content/temporary-example-page.html

There is no way to exclude a section/page individually in the sitemap tool. Blocking it via robots is fine. You will probably get a flag in webmaster tools under the 'in sitemaps' tab, but it is expected.

Disallow: /forum/page/

is the correct syntax if you are installed in /forum


Since you're in root,

Disallow: /content/section/122-virtual-video-listings.html

is an example url

Originally Posted by Brian Cummiskey

There is no way to exclude a section/page individually in the sitemap tool. Blocking it via robots is fine. You will probably get a flag in webmaster tools under the 'in sitemaps' tab, but it is expected.

Disallow: /forum/page/

is the correct syntax if you are installed in /forum


Since you're in root,

Disallow: /content/section/122-virtual-video-listings.html

is an example url

Ok I understand now, since the root is in a way the domain itself all you need is every page that follows that root. Cool thanks Brian.