problem with crawling member profile page

Hi all

I'm using vbseo with vbseo sitemap generator and have a problem with indexing member profile. Althought i added nofollow to all link to member profile, it's still inform me in SE bot activity that bots crawled a lot of member profile page - as i show in attachment file. in fact when i use site:mydomain.com there is a lot of member profile page are showed at the top ( my site is : http:forum.tech24.vn ). I'm afraid of if SE bots crawl al lot of member page than they will not have enough time to crawl other more important things in website ( showthread, homepage, forumdisplay ).

Can any body help me on this ? Thanks for every suggestion

You need to make sure you are not including member profiles to your sitemap files and you are excluding member profiles getting indexed

You can add a line
disallow: profilepath
in robots.txt and then you can use google's "url removal tool" in webmaster tools for early indexed pages in google.

Mert Gökçeimam :
1. Member profile are not included in sitemap.xml
2. Do you mean that i have to include /member.php in robots.txt ? I also thinked about this early but have a question will it block all profile member to be indexed in search result ?

Hope to hear from you soon ! thanks for your answer

2) yes. if you re-write, you should have

Disallow: /forum/member/*

or your actual path to the member script.

Yes, this will block bots from viewing the pages.


In general, just leave the member pages out of the sitemap, but leave them open (do not block with robots.txt).