can someone explain how to write a simple effective robot txt file
dynamo mailing list
email@hidden
Update your subscriptions at:
http://freewaytalk.net/person/options
Google is your friend
I found this:
http://www.robotstxt.org/orig.html
and
http://www.thesitewizard.com/archive/robotstxt.shtml
The first one seems fairly clear.
HTH
On Nov 7, 2009, at 7:45 PM, marathon wrote:
can someone explain how to write a simple effective robot txt file
dynamo mailing list
email@hidden
Update your subscriptions at:
http://freewaytalk.net/person/options
dynamo mailing list
email@hidden
Update your subscriptions at:
http://freewaytalk.net/person/options
Thanks for the info, so I see where to place the name in meta tags–
but what is the best way and how to write the correct format is my
question. Can anyone copy and paste how this should look, thanks!
On Nov 7, 2009, at 2:29 PM, Mike B wrote:
Google is your friend
I found this:
The Web Robots Pagesand
http://www.thesitewizard.com/archive/robotstxt.shtml
The first one seems fairly clear.
HTH
On Nov 7, 2009, at 7:45 PM, marathon wrote:
can someone explain how to write a simple effective robot txt file
dynamo mailing list
email@hidden
Update your subscriptions at:
http://freewaytalk.net/person/options
dynamo mailing list
email@hidden
Update your subscriptions at:
http://freewaytalk.net/person/options
dynamo mailing list
email@hidden
Update your subscriptions at:
http://freewaytalk.net/person/options
There are some nice examples at the bottom of this page-
http://www.robotstxt.org/orig.html#examples
If that answers your question
Cheers!
|
On Nov 8, 2009, at 12:51 PM, marathon wrote:
Thanks for the info, so I see where to place the name in meta tags–
but what is the best way and how to write the correct format is my
question. Can anyone copy and paste how this should look, thanks!
On Nov 7, 2009, at 2:29 PM, Mike B wrote:
Google is your friend
I found this:
The Web Robots Pagesand
http://www.thesitewizard.com/archive/robotstxt.shtml
The first one seems fairly clear.
HTH
On Nov 7, 2009, at 7:45 PM, marathon wrote:
can someone explain how to write a simple effective robot txt file
dynamo mailing list
email@hidden
Update your subscriptions at:
http://freewaytalk.net/person/options
dynamo mailing list
email@hidden
Update your subscriptions at:
http://freewaytalk.net/person/options
dynamo mailing list
email@hidden
Update your subscriptions at:
http://freewaytalk.net/person/options
dynamo mailing list
email@hidden
Update your subscriptions at:
http://freewaytalk.net/person/options