Lightweight, sturdy and fully-documented parser of the `robots.txt` file.
More...
robots.txt
Rule
example.dart
vxern.dev
Repository (GitHub)View/report issues
unknown (license)
sprint, web_scraper
Packages that depend on robots_txt
Back