Are you familiar with the Robot Exclusion Standard? This is a way
whereby you can inform search engines and other automated spiders
(programs which scan your web sites for information) what pages
and directories are not to be examined. The standard is precisely
defined and a little obscure, and while not overly complicated it
can be difficult to understand.
There is help available, believe it or not. That obscure file
called "robots.txt" no longer needs to be hand-edited. No longer
do you need to type in server path names, remember robot and
spider names and enter strange symbols which don't seem to make
any sense.
Okay, here's a quick review. The "robots.txt" file simply lists
the spiders that you want to prevent from listing all or part of
your site. Along with each spider is the list of directories and
files that it is not allowed to examine. Wildcards are allowed.
It's not very difficult but it is time consuming if you have lots
of robots and files to specify. In addition, it's easy to misspell
something and fali to achieve your goal.
Now, someone has created a nice little piece of shareware called
RoboGen, and it's not very expensive considering the benefits that
you will receive.
I stumbled across this entirely by accident, since it never
occurred to me that someone would write a program such as this. I
downloaded the 30 day demonstration and quickly discovered how
wonderful it is. Robogen has already saved me hours and hours of
work.
Okay, here's how it works. You enter your FTP login information
and log into your site. You then select robots by name in the left
pane, and indicate which directories and/or files are to be
excluded by the robot in the right pane.
You can create a new file or edit an existing one. Even better,
Robogen will work directly off the "robots.txt" file on your web
site. It will log in, read the existing file, allow you to edit
to your hearts content, then update the file directly.
I like little specialized tool such as this which truly make a
webmasters life easy. The product is inexpensive (there is a
limited free version as well), well documented and complete.
Highly recommended.
About the Author
Richard Lowe Jr. is the webmaster of Internet Tips And Secrets
at http://www.internet-tips.net - Visit our website any time to
read over 1,000 complete FREE articles about how to improve your
internet profits, enjoyment and knowledge.