perl-www-robotrules: perl-www-robotrules (DB of robots.txt-derived perms)
perl-www-robotrules:
perl-www-robotrules: This perl module parses /robots.txt files as specified
perl-www-robotrules: in "A Standard for Robot Exclusion". Webmasters can use
perl-www-robotrules: the /robots.txt file to forbid conforming robots from
perl-www-robotrules: accessing parts of their web site
perl-www-robotrules:
perl-www-robotrules: http://search.cpan.org/dist/WWW-RobotRules/ 
perl-www-robotrules:
perl-www-robotrules:
perl-www-robotrules: