While I am developing a site, I do not want to have the half finished pages indexed by search engines etc. As I understand it, noindex is a more thorough method than robots.txt. How do I implement noindex on an XWay site.
In Xway’s Page inspector, find and expand the Meta Tags section. In there, click the + and enter “robots” for the name, “none” for the content. Alternatively you might want only “noindex” rather than none (noindex, nofollow), Google’s website has more information about possible values here: Robots Meta Tags Specifications | Google Search Central | Documentation | Google for Developers
You can apply this to your master page(s) if you want every page to be omitted until you’re finished making the site or if you want to allow indexing gradually as you finish pages, you can apply it to the individual pages and later tick Use Master Meta Tags to restore the connection to master page meta tags (removing the noindex entry in the process).
My understanding is that robots.txt and the meta tag serve slightly different but often overlapping or sometimes conflicting purposes, robots.txt is for informing search engines how they should (or should not) crawl your website during indexing, whereas the robots meta tag on a page informs the search engine how it should handle that page after it has decided to index it.
A key difference there is that if a search engine has decided to index your page as a result of following a link on a different website, it will read the meta tag while indexing but most will not read robots.txt at that stage.
My knowledge of it is limited though so I don’t fully understand the implications of using one, the other or both.
Thank you, Simon. That was easy.
Does password protecting a site stop the robots indexing?
Is it possible to password a site from Xway?
Password protecting would not necessarily stop robots indexing a URL but would prevent them from indexing the content. E.g. if the robot follows a link from your own or another website and encounters password protection, it may index that a password protected page exists there.
Xway does not directly provide password protection but integrating software to handle presenting content in different ways, e.g. by hiding it behind authentication, is possible with a combination of markup on your page(s) and markup boxes.
It sounds like this would be overkill if it’s only meant as a temporary measure while developing the site but this would usually be a piece of CMS software uploaded to your server, where markup (provided by the CMS) added to your page would initialise it and markup boxes would be used on your page where you want its content to appear.
The other way of getting password protection would be using Basic Authentication. This is part of HTTP and is provided by web server software, it’s rarely used on websites meant for general consumption but is a simple setup for something meant to be private for an individual or to a limited number of people. If you’ve seen your browser (as opposed to something on the website) asking for a username and password when trying to open a URL, this is Basic Authentication.
How exactly you use this will depend on how your hosting provider has configured things so you would need to check their website or contact them about setting it up.
As far as I know there is no further customisation which can be applied to this for search engines, they will not know what or even if there is something behind the URL, they will just be told their access is unauthorised.
Again, very clear and useful. Thank you. I will investigate how to set passwords on my VPS.