robots.txt

This is the contents of my current robots.txt file, but I feel that it's not what it should be. I have noticed that vBSEO hasn't been boosting my website as much as I thought it would. I read all these articles on how satisfied people were and how people saying their traffic trippled in months, but so far I can just see that it's a load of bull. Either that or I'm going wrong somewhere.

But I was just wondering if someone could take a look at my robots.txt file and just let me know what I need to add or remove in order to get the maximum potential out of it. Here are the contents of my robots.txt file:

Code:

User-agent: *
Disallow: /archive/
Disallow: /lo-fi/
Disallow: /general-announcements/
Disallow: /admincp/
Disallow: /cgi-bin/
Disallow: /clientscript/
Disallow: /includes/
Disallow: /install/
Disallow: /modcp/
Disallow: /sitemap/
Disallow: /tags/
Disallow: /printthread.php
Disallow: /subscription.php
Disallow: /payments.php
Disallow: /profile.php
Disallow: /faq.php
Disallow: /calendar.php
Disallow: /private.php
Disallow: /sendmessage.php
Disallow: /sendmessage.php?do=
Disallow: /showgroups.php
Disallow: /reputation.php
Disallow: /report.php
Disallow: /threadrate.php
Disallow: /postings.php
Disallow: /newthread.php
Disallow: /search.php
Disallow: /newreply.php
Disallow: /register.php
Disallow: /login.php
Disallow: /faq.php
Disallow: /image.php
Disallow: /cron.php
Disallow: /joinrequests.php
Disallow: /usercp.php
Disallow: /vbseocp.php

rule #1 of robots.txt, NEVER include private/hidden directories such as modcp or admincp. This is a big security issue, as you basically just told would be hackers where your admin panel is..... and sometimes, that's half the battle. Remove those two entries. I would also suggest renaming the admincp folder and setting that up in the vb options (and also adjusthing the htaccesss file, replacing admincp with xxxxxxx whatever you change the folder name to).

Otherwise, your robots file looks fine.

I see you have lo-fi in there. did you convert from ipb recently?
Your google results look good. all results seem to be re-written and vb urls.

The main thing to consider is that you are in a VERY competitve niche. Some of the biggest forums out there are coding/web related (digital point, sitepoint, dream in code, experts exchange, etc etc) Further, you segmented yourself with your TLD choice. Not many non-UK folks tend to visit .co.uk sites, and non-UK google searchers are likely to not get a .co.uk result page unless its a VERY specific search term that only you have ranking for in the top 3 results

You also have a critical markup error.
[Invalid] Markup Validation of http://www.codershangout.co.uk/ - W3C Markup Validator

You're missing the table row/cell start. Thus, this renders your H1 basically useless, as html is a top-down approach as read by bots.

I would also remove your sitename from titles for better long tail. see: [How To] Remove Forum Title from Thread Pages for Long Tail Optimization / Increased Thread Title Relevancy

I would also remove the top5 stats for guests (you can wrap it in a simple template conditional, <if condition="$show['member']"> code </if> ). You can also consider using the new VBSEO virtual HTML feature. Lots of links and bloat in there that don't help the spidering down of your important content.

The other thing to consider, and probably the most important thing I've noticed on your site, is that your thread titles (by no fault of your own) are mostly generic.

"php help"
"mysql error"

next to no one is going to search for that.

I would take a proactive apprach to re-naming people's bad thread titles to what it actually is.
ie,
PHP Mysql error - Web Design Forums
I would rename
mysql_num_rows() gives invalid username/password combination
Or something to that effect. THAT is what people are going to search for, and THAT is how you are going to get good google/se traffic.... other people who are having the same problem.

rule #1 of robots.txt, NEVER include private/hidden directories such as modcp or admincp. This is a big security issue, as you basically just told would be hackers where your admin panel is..... and sometimes, that's half the battle. Remove those two entries. I would also suggest renaming the admincp folder and setting that up in the vb options (and also adjusthing the htaccesss file, replacing admincp with xxxxxxx whatever you change the folder name to).

I have actually renamed the admincp / modcp directories but forgot to take them out of the robots file, sorry, my mistake there.

You're missing the table row/cell start. Thus, this renders your H1 basically useless, as html is a top-down approach as read by bots.

I'm a bit confused on what I need to do to fix this error :-/ Slightly confused.

I see you have lo-fi in there. did you convert from ipb recently?
Your google results look good. all results seem to be re-written and vb urls.

No I didn't convert from ipb, why? :-/ vBulletin is the only forum software that has been installed on my website ever.

The main thing to consider is that you are in a VERY competitve niche. Some of the biggest forums out there are coding/web related (digital point, sitepoint, dream in code, experts exchange, etc etc) Further, you segmented yourself with your TLD choice. Not many non-UK folks tend to visit .co.uk sites, and non-UK google searchers are likely to not get a .co.uk result page unless its a VERY specific search term that only you have ranking for in the top 3 results

I know, unfortunetly the .com wasn't availible, but I do intend to back-order it as it expires in a couple of months.

Even though I have got this code on my site:

Code:

<h1 id="logotext">Web Design Forums</h1></span></a>
<h2 id="logotext1">Advice forums for all of your web design, graphics and coding needs.</h2>

and this is the css

Code:

h1#logotext {text-indent:-9999999px;margin:0;padding:0;}
h2#logotext1 {text-indent:-9999999px;margin:0;padding:0;}

I know I have to fix the error but will I get a black mark from Google for indenting and hiding this from view, which is technically cheating and I don't really want to challenge Google as I will come off worst lol.

Thankyou for your help mate, much appreciated

HTML Code:

<h1 id="logotext">Web Design Forums</h1></span></a>

Where are the opening 'span' and 'a' tags ?

Also you can't have H1 inside any of them; as H1 is a block level element.
This is better:

HTML Code:

<h1 id="logotext"><a href="your url">Web Design Forums</a></h1>

Thankyou, I have set it to this as I don't want it as a link:

Code:

<h1 id="logotext">Web Design Forums</h1>

I won't get a black mark by Google for hiding text behind an image will I? I just don't want challenge Google lol.

Sorry to re-open this thread, but Briansol, you mentioned this:

I would also remove the top5 stats for guests (you can wrap it in a simple template conditional, <if condition="$show['member']"> code </if> ). You can also consider using the new VBSEO virtual HTML feature. Lots of links and bloat in there that don't help the spidering down of your important content.

Why do I need to wrap if statements around it? Wouldn't disabling it for guests in the options for that product be sufficient?

Kind Regards
Stormraven.

Bump

Hello ,

SE's suggest 100~ links on a page. Adding top5 stats will increase active link numbers and will not have positive effect.

Thankyou mert, but which is the best way to hide it? The plugin options have an option to disable it for the usergroup guests or do I need to wrap <if> tags around it? Which is the best?

Kind Regards
Stormraven.

If the plugin has an usergroup option , then you should use that.

Thankyou very much Mert, I have rated your support 5 out of 5 thankyou mate.

Originally Posted by Stormraven

Thankyou very much Mert, I have rated your support 5 out of 5 thankyou mate.

You're welcome