This site has been up a few months, but still fails to show up on Google. Could some kind soul point out my failings! I have styled the text - but not an h1 tag. Should all text be in p, h1, h2 tags and not added ones with names like ‘main heading’ etc? Is this an issue? Many thanks.
Actually on closer inspection it looks like this shouldn’t block the crawling of the site but I’d be inclined to get rid of the file anyway just to be sure.
Regards,
Tim.
Hi Noel,
It looks like it make have been added by default by the host. If you’ve an FTP app then FTP into the server space and simply deleter the robots.txt file that should be at the root level of the site.
Regards,
Tim.
On 13 May 2014, at 14:12, Noel Sergeant wrote:
Thanks for info, Tim. But I’ve never heard of this. How have I applied it and how do I remove it?
The robots exclusion file tells search engines (and other bots) what they should have access to. If you don’t want a certain page showing up on Google then you can add it to the file and all good search engines will simply ignore it. Likewise it is worth blocking access to things like log-in pages for a content management systems to make it harder for people to simply search for a site to attack.
Regards,
Tim.
Last one from me (for the moment). The site does use webyeb cms. Does this mean that if you you CMS you should have the robot with the downside that Google will not search for your site? I presume, because the file is at root level, you cannot just apply it to a single page, only the whole site.
You can move the file into a deeper folder. What is the actual URL (well, don’t tell us all here, but think about this question) of the login page? If that’s inside a folder somewhere, then put the robots.txt file inside that folder, so you don’t end up with example.com/admin/login.php links in your google results.
Walter
On May 13, 2014, at 10:26 AM, Noel Sergeant wrote:
Last one from me (for the moment). The site does use webyeb cms. Does this mean that if you you CMS you should have the robot with the downside that Google will not search for your site? I presume, because the file is at root level, you cannot just apply it to a single page, only the whole site.
Sorry, back again. The webyeb .php pages appear in the root folder along with the other html pages. In that folder there is a webyeb folder which contains data and program folders. Not sure what folder you would put the robot file into.
Well, then my suggestion won’t help. Try asking Max – he has a lot of experience with this system, and may have some specific light he can shed on this problem.
Walter
On May 13, 2014, at 10:54 AM, Noel Sergeant wrote:
Sorry, back again. The webyeb .php pages appear in the root folder along with the other html pages. In that folder there is a webyeb folder which contains data and program folders. Not sure what folder you would put the robot file into.