pivox library
Classes
- AdaptiveScrapingStrategy
- Generates adaptive scraping strategies based on site reputation and error patterns
- AdvancedProxyManager
- Advanced proxy manager with additional features
- AdvancedProxyManagerAdapter
- Adapter for using AdvancedProxyManager as a ProxyManager
- AdvancedRotationStrategy
- Advanced proxy rotation strategy that uses multiple factors to select the next proxy
- AdvancedWebScraper
- An advanced web scraper with proxy rotation, rate limiting, and more
- AppConstants
- Constants used throughout the application
- ConcurrentWebScraper
- A web scraper with concurrency control
- CookieManager
- A manager for cookies to maintain sessions across requests
- DataCacheManager
- A manager for storing scraped data using data_cache_x
- DynamicUserAgentManager
- A manager for dynamically fetching and using user agents
- EnhancedRateLimiter
- An enhanced rate limiter for web scraping with advanced features
- GeoRotationStrategy
- Geo-based proxy rotation strategy that rotates through proxies from different countries
- GetProxies
- Use case for getting proxies
- GetValidatedProxies
- Use case for getting validated proxies
- HeadlessBrowser
- A headless browser implementation for web scraping
- HeadlessBrowserConfig
- Configuration options for the headless browser
- HeadlessBrowserFactory
- Factory for creating headless browser instances
- HeadlessBrowserResult
- Result of a headless browser operation
- HeadlessBrowserService
- Service for headless browser operations
- LeastRecentlyUsedStrategy
- Least recently used proxy rotation strategy
- MemoryEfficientParser
- A memory-efficient HTML parser for large documents
- OnlineKhabarHandler
- A specialized handler for onlinekhabar.com
-
ParallelProcessor<
T, R> - A utility class for processing tasks in parallel using Isolates
- Pivox
- Factory methods for quick initialization
- PivoxBuilder
- A builder class for simplifying Pivox initialization
- PivoxFactory
- Factory class for creating Pivox components
- PivoxScraper
- A comprehensive web scraping solution built on Pivox
- Proxy
- Represents a proxy server
- ProxyAnalyticsService
- Service for tracking proxy usage analytics
- ProxyAnalyticsServiceImpl
- Implementation of ProxyAnalyticsService that uses SharedPreferences for storage
- ProxyAuth
- Authentication credentials for a proxy
- ProxyCacheManager
- Manager for caching proxies in a tiered system
- ProxyFilterOptions
- Options for filtering proxies
- ProxyHttpClient
- HTTP client with proxy support
- ProxyInterceptor
- Dio interceptor for proxy support
- ProxyLocalDataSource
- Interface for local data source to cache proxies
- ProxyLocalDataSourceImpl
- Implementation of ProxyLocalDataSource using SharedPreferences
- ProxyManager
- Manager for proxy operations
- ProxyModel
- Data model for a proxy server
- ProxyPreloaderService
- Service for preloading and background validation of proxies
- ProxyRemoteDataSource
- Interface for remote data source to fetch proxies
- ProxyRemoteDataSourceImpl
- Implementation of ProxyRemoteDataSource
- ProxyRepository
- Repository interface for proxy management
- ProxyRepositoryImpl
- Implementation of ProxyRepository
- ProxyRotationStrategy
- Abstract class for proxy rotation strategies
- ProxyRotationStrategyFactory
- Factory for creating proxy rotation strategies
- ProxyScore
- Represents a score for a proxy server
- ProxyValidationOptions
- Options for proxy validation
- RandomStrategy
- Random proxy rotation strategy
- RateLimiter
- A rate limiter for web scraping to avoid overwhelming target websites
- RobotsTxtHandler
- A class for handling robots.txt files
- RobotsTxtRules
- A class for representing robots.txt rules
- RotationStrategyFactory
- Factory for creating proxy rotation strategies
- RoundRobinStrategy
- Round-robin proxy rotation strategy
- ScrapingJob
- A scraping job
- ScrapingJobScheduler
- A scheduler for recurring scraping jobs
- ScrapingLogEntry
- Represents a log entry in the scraping process
- ScrapingLogger
- A logger for scraping operations
- ScrapingStrategy
- Defines the strategy for scraping a website
- ScrapingTaskQueue
- A task queue for web scraping with concurrency control
- SiteReputation
- Represents the reputation of a website for scraping
- SiteReputationTracker
- Tracks the reputation and success rate of websites for scraping
- SocksProxyValidator
- Validator for SOCKS proxies
- SpecializedHeadlessHandlers
- Specialized handlers for problematic sites using headless browser
- SpecializedSiteHandler
- A specialized handler for problematic websites
- SpecializedSiteHandlerRegistry
- A registry of specialized site handlers
- StreamingHtmlParser
- A streaming HTML parser that can process HTML incrementally
- UserAgentRotator
- A rotator for user agents to avoid detection
- ValidateProxy
- Use case for validating a proxy
- VegaMoviesHandler
- A specialized handler for vegamovies.td
- WebScraper
- A web scraper that uses proxies to avoid detection and blocking
- WeightedRotationStrategy
- Weighted proxy rotation strategy that selects proxies based on their performance scores
- WeightedStrategy
- Weighted random proxy rotation strategy based on proxy scores
Enums
- BrowserType
- Browser types for user agent selection
- CacheTier
- Cache tier levels for proxies
- LogType
- The type of log entry
- ProxyAuthMethod
- Authentication method for proxies
- ProxyProtocol
- Protocol used by a proxy
- RotationStrategyType
- Type of proxy rotation strategy
- ScrapingExceptionType
- Exception types for web scraping
Extensions
- ProxyProtocolExtension on ProxyProtocol
- Extension methods for ProxyProtocol
- WebScraperExtension on WebScraper
- Extension methods for WebScraper to enhance error handling and retry logic
Exceptions / Errors
- NoValidProxiesException
- Exception thrown when no valid proxies are available
- PivoxException
- Base exception class for Pivox
- ProxyFetchException
- Exception thrown when proxy fetching fails
- ProxyRotationException
- Exception thrown when proxy rotation fails
- ProxyValidationException
- Exception thrown when proxy validation fails
- ScrapingException
- Exception thrown when a scraping operation fails