This is a topic I have circled around a few times and have tried different approaches. When developing a new site for our companies we typically create a folder dedicated to the development of the site while the site that is currently live lives in the root directory. Have used the robots.txt no follow approach, as well as the same approach but in the the head of the page.
I would think that there is a more straight forward way to include a .htaccess file in the root of the development folder nested in the main directory. Since we are using the same domain and page names are also the same, just a level or two down from the root, I want to be sure that the development content is not being indexed and showing up in search engines. I would value your input.
Was hoping to see how others might be doing this via .htaccess as our VPS host does not use cPanel.
Most likely we will end up moving all our company sites to Amazon for greater future expansion. From my preliminary research they have their own way of doing things.
Thanks guys… Dave for now I may just go with the Password Protect .htaccess method, fairly simple. Also thanks for the link. I have used Andreas’s resource before, very helpful.
Using this method I don’t need to change my robots.txt or meta info on the pages in the protected folder, correct?
Also is this a separate .htaccess file in the sub directory?
Here is a link to a great article by Marvin Russell. He does a nice job of unpacking this topic in a simple to understand way. Hope this is helpful to someone else as well.