features/web_scraping/robots_txt_handler library

Classes

RobotsTxtHandler
A class for handling robots.txt files
RobotsTxtRules
A class for representing robots.txt rules