How do I disallow all of the websites on my development server if the only way you access said sites is with a subdomain redirect that bypasses the root?
At present I have a robots.txt file in the root folder with the following code:
User-agent: *
Disallow: /
offtopic mailing list
email@hidden
Update your subscriptions at:
http://freewaytalk.net/person/options
Are you looking to avoid people being able to browse to www.example.com/subdomain and force them to go to subdomain.example.com? If so, you want an htaccess redirect, not a robots.txt. The robots file is purely for well-behaved spiders, and does nothing for normal browsers or spiders that are not well-behaved.
Google: mod_rewrite redirect to subdomain
Walter
On Feb 12, 2013, at 10:59 AM, RavenManiac wrote:
How do I disallow all of the websites on my development server if the only way you access said sites is with a subdomain redirect that bypasses the root?
At present I have a robots.txt file in the root folder with the following code:
User-agent: *
Disallow: /
offtopic mailing list
email@hidden
Update your subscriptions at:
http://freewaytalk.net/person/options
offtopic mailing list
email@hidden
Update your subscriptions at:
http://freewaytalk.net/person/options
No, I just don’t want search engines to index my internal server.
offtopic mailing list
email@hidden
Update your subscriptions at:
http://freewaytalk.net/person/options
Then what I would recommend is that you use your cPanel (or local equivalent) to set a simple password on that folder. Tell all your clients what it is. That will stop all crawlers cold, good and evil. Even better is to have a separate password for each client’s folder. It doesn’t have to be a serious password (it will be sent in plain text anyway) but just the fact that it is there will keep the honest folk honest, and will keep the bots out entirely.
Walter
On Feb 12, 2013, at 11:50 AM, RavenManiac wrote:
No, I just don’t want search engines to index my internal server.
offtopic mailing list
email@hidden
Update your subscriptions at:
http://freewaytalk.net/person/options
offtopic mailing list
email@hidden
Update your subscriptions at:
http://freewaytalk.net/person/options