AdvancedWebScraper class
An advanced web scraper with proxy rotation, rate limiting, and more
Constructors
- AdvancedWebScraper({required ProxyManager proxyManager, ProxyHttpClient? httpClient, RateLimiter? rateLimiter, UserAgentRotator? userAgentRotator, CookieManager? cookieManager, int defaultTimeout = 30000, int maxRetries = 3, bool handleCookies = true, bool followRedirects = true})
- Creates a new AdvancedWebScraper with the given parameters
Properties
- hashCode → int
-
The hash code for this object.
no setterinherited
- runtimeType → Type
-
A representation of the runtime type of the object.
no setterinherited
Methods
-
close(
) → void - Closes the HTTP client
-
extractData(
{required String html, required String selector, String? attribute, bool asText = true}) → List< String> - Parses HTML content and extracts data using CSS selectors
-
extractStructuredData(
{required String html, required Map< String, String> selectors, Map<String, String?> ? attributes}) → List<Map< String, String> > - Parses HTML content and extracts structured data using CSS selectors
-
fetchHtml(
{required String url, Map< String, String> ? headers, int? timeout, int? retries}) → Future<String> - Fetches HTML content from the given URL
-
fetchJson(
{required String url, Map< String, String> ? headers, int? timeout, int? retries}) → Future<Map< String, dynamic> > - Fetches JSON content from the given URL
-
noSuchMethod(
Invocation invocation) → dynamic -
Invoked when a nonexistent method or property is accessed.
inherited
-
submitForm(
{required String url, String method = 'POST', required Map< String, String> formData, Map<String, String> ? headers, int? timeout, int? retries}) → Future<String> - Submits a form with the given data
-
toString(
) → String -
A string representation of this object.
inherited
Operators
-
operator ==(
Object other) → bool -
The equality operator.
inherited
Static Methods
-
create(
{required ProxyManager proxyManager, int defaultTimeout = 30000, int maxRetries = 3, bool handleCookies = true, bool followRedirects = true}) → Future< AdvancedWebScraper> - Factory constructor to create an AdvancedWebScraper with default components