robots_txt_processor library

Classes

RobotsTxt
RobotsTxt object class
RobotsTxtGroup
RobotsTxt group object class
RobotsTxtProcessor
A robots.txt processor