WebScraperIntelligentScrapingExtension extension
Extension methods for WebScraper to add intelligent scraping capabilities
- on
Methods
-
detectMainContent(
String html) → ContentDetectionResult -
Available on WebScraper, provided by the WebScraperIntelligentScrapingExtension extension
Detects the main content area of a webpage -
extractContentWithPagination(
{required String url, required PaginationConfig paginationConfig, LazyLoadConfig? lazyLoadConfig, TextExtractionOptions textExtractionOptions = const TextExtractionOptions(), Map< String, String> ? headers, int? timeout, int? retries}) → Future<List< TextExtractionResult> > -
Available on WebScraper, provided by the WebScraperIntelligentScrapingExtension extension
Extracts the main content from multiple pages with pagination -
extractText(
String html, {TextExtractionOptions options = const TextExtractionOptions()}) → TextExtractionResult -
Available on WebScraper, provided by the WebScraperIntelligentScrapingExtension extension
Extracts clean, readable text from HTML -
fetchHtmlWithLazyLoading(
{required String url, LazyLoadConfig config = const LazyLoadConfig(), Map< String, String> ? headers}) → Future<LazyLoadResult> -
Available on WebScraper, provided by the WebScraperIntelligentScrapingExtension extension
Fetches HTML content with lazy loading support -
scrapeWithLazyLoadingAndPagination<
T> ({required String url, required PaginationConfig paginationConfig, required LazyLoadConfig lazyLoadConfig, required Future< T> extractor(String html, String pageUrl), Map<String, String> ? headers, int? timeout, int? retries}) → Future<PaginationResult< T> > -
Available on WebScraper, provided by the WebScraperIntelligentScrapingExtension extension
Fetches HTML content with both lazy loading and pagination support -
scrapeWithPagination<
T> ({required String url, required PaginationConfig config, required Future< T> extractor(String html, String pageUrl), Map<String, String> ? headers, int? timeout, int? retries}) → Future<PaginationResult< T> > -
Available on WebScraper, provided by the WebScraperIntelligentScrapingExtension extension
Scrapes multiple pages with pagination