WebScraperPerformanceExtension extension
Extension methods for WebScraper to add performance optimization capabilities
- on
Methods
-
createCacheManager(
{String namespace = 'web_scraper', Logger? logger}) → DataCacheManager -
Available on WebScraper, provided by the WebScraperPerformanceExtension extension
Creates a data cache manager for caching scraping results -
createDataChunker(
{int chunkSize = DataChunker.defaultChunkSize, Logger? logger}) → DataChunker -
Available on WebScraper, provided by the WebScraperPerformanceExtension extension
Creates a data chunker for handling large datasets -
createTaskScheduler(
{TaskSchedulerConfig? config, ResourceMonitor? resourceMonitor, Logger? logger}) → TaskScheduler -
Available on WebScraper, provided by the WebScraperPerformanceExtension extension
Creates a task scheduler for parallel scraping -
fetchHtmlWithCache(
{required String url, required DataCacheManager cacheManager, Map< String, String> ? headers, int? timeout, int? retries, DataCacheOptions cacheOptions = const DataCacheOptions()}) → Future<String> -
Available on WebScraper, provided by the WebScraperPerformanceExtension extension
Fetches HTML with caching -
scrapeInParallel<
T> ({required List< String> urls, required Future<T> extractor(String html, String url), required TaskScheduler scheduler, Map<String, String> ? headers, int? timeout, int? retries, TaskPriority priority = TaskPriority.normal, int maxRetries = 3}) → Future<List< T> > -
Available on WebScraper, provided by the WebScraperPerformanceExtension extension
Scrapes multiple URLs in parallel -
scrapeWithChunking<
T> ({required String url, required DataChunker dataChunker, required FutureOr< T> processor(String chunk, T? previousResult), Map<String, String> ? headers, int? timeout, int? retries, T? initialResult}) → Future<T> -
Available on WebScraper, provided by the WebScraperPerformanceExtension extension
Scrapes a URL with chunked processing for large HTML documents