I have some webpages that we give links to in marketing emails. The links go to webpages that cannot be accessed if you go to our site home page, but you can get to our home page from those pages.
What I want to know is:
1/ how do I stop search engines from indexing these pages
2/ If there is not link back to them from our home page, I’m I right in thinking they cannot be ripped down by programs like Blue Crab?
On 16 Jan 2011, at 10:08, “Kevin Cheesman” email@hidden wrote:
I have some webpages that we give links to in marketing emails. The links go to webpages that cannot be accessed if you go to our site home page, but you can get to our home page from those pages.
What I want to know is:
1/ how do I stop search engines from indexing these pages
2/ If there is not link back to them from our home page, I’m I right in thinking they cannot be ripped down by programs like Blue Crab?
There is another chance for these to be indexed, and that’s called
referer leakage. If you have any links on these “private” pages that
go to other sites, your page will be captured in the “referer” (yes, I
know that’s not how the word is usually spelled) header, which is a
hidden part of any Web request.
Google (and other sites that follow their algorithmic lead) will look
back at the page that sent the link. Since it’s not in their index,
then they may add it. IF they follow the robots.txt or robots meta
tags for instructions, then all is well, there won’t be any leakage.
But not all robots are benign. And this data will hang around in the
servers that you linked to until the logs rotate.
This, by the way, forms the bedrock of how Google works – by seeing
who links to your site, they can assign the page rank based on the
popularity of each referer.
Walter
On Jan 16, 2011, at 5:08 AM, Kevin Cheesman wrote:
2/ If there is not link back to them from our home page, I’m I right
in thinking they cannot be ripped down by programs like Blue Crab?
