robots_txt library

Lightweight, fully documented robots.txt file parser.

Classes

Robots

Allows for parsing of a host's robots.txt to get information about which of its resources may or may not be accessed, as well as which of its pages cannot be traversed.

Rule

A single rule (either Allow or Disallow) inside the robots.txt file.

Extensions

RulingOnPath on List<Rule>

Extends List<Rule> with a method for getting the Rule with the highest Rule.priority.

Typedefs

Element = Map<String, Map<String, dynamic>>

Abstracts away the rather convoluted declaration for an element with two fields; 'title' and 'attributes'. 'attributes' is a map containing the attributes of the element.