RobotsTxtHandler class
A class for handling robots.txt files
Constructors
- RobotsTxtHandler.new({Client? httpClient, ProxyManager? proxyManager, ScrapingLogger? logger, String? defaultUserAgent, bool respectRobotsTxt = true, int cacheExpirationMs = 3600000})
- Creates a new RobotsTxtHandler with the given parameters
Properties
- hashCode → int
-
The hash code for this object.
no setterinherited
- runtimeType → Type
-
A representation of the runtime type of the object.
no setterinherited
Methods
-
clearCache(
) → void - Clears the cached rules
-
close(
) → void - Closes the HTTP client
-
getRules(
String domain) → Future< RobotsTxtRules?> - Gets the robots.txt rules for a domain
-
isAllowed(
String url, [String? userAgent]) → Future< bool> - Checks if a URL is allowed to be crawled
-
noSuchMethod(
Invocation invocation) → dynamic -
Invoked when a nonexistent method or property is accessed.
inherited
-
toString(
) → String -
A string representation of this object.
inherited
Operators
-
operator ==(
Object other) → bool -
The equality operator.
inherited