spidr library

Classes

AiExtractor: The core interface for AI-powered scraper extensions.
BasicProxyPool: Base proxy pool implementation.
BasicSchemaConfig: A basic implementation of SchemaConfig for key-value dictionary mappings.
ConcurrentStreamTransformer<S, T>: A custom StreamTransformer that executes conversion futures concurrently up to a maximum concurrency limit.
CrawlerScheduler: Manages crawling request frontiers, filtering duplicate visited URLs, and enforcing boundaries.
DefaultCrawlerScheduler: Default implementation of CrawlerScheduler supporting FIFO (BFS) and LIFO (DFS).
DioSpidrClient: Concrete implementation of SpidrClient using the robust cross-platform Dio package.
ElementFingerprint: Represents a persistent visual, behavioral, and structural fingerprint of a DOM element.
FileStorageAdapter
FingerprintStore: Abstract persistence contract for storing and retrieving element fingerprints.
GeminiLlmBackend: Concrete LLM backend targeting the Google Gemini API.
HtmlParseOptions: Configuration options for HTML parsing optimizations.
HtmlSpidrElement: Concrete implementation of SpidrElement backed by package:html DOM Element.
HtmlSpidrPage: Concrete implementation of SpidrPage executing CSS and XPath parsing.
IndexedDbStorageAdapter
IsarStorageAdapter
LlmAiExtractor: LLM-powered implementation of AiExtractor parsing dynamic document structures.
LlmBackend: Base abstract interface for large language model backends.
MemoryFingerprintStore: Volatile in-memory implementation of FingerprintStore.
MemorySessionStore: Volatile in-memory implementation of SpidrSessionStore.
MemoryStorageAdapter: Fallback key-value storage implementation executing entirely in volatile heap memory.
MockAiExtractor: Mock AI implementation for basic testing.
MockLlmBackend: Lightweight mock LLM backend returning predefined static mock payloads.
OpenAiLlmBackend: Concrete LLM backend targeting the OpenAI chat completions API.
ProxyInfo: Represents connection attributes of a proxy server.
ProxyMiddleware: Request middleware to automatically select and inject proxy details on requests.
ProxyPool: Manages collections of proxies and selection strategies.
RetryConfig: Retry policy definitions for handling network failures or transient error responses.
RobotsDirective: Represents a single robots.txt directive rule.
RobotsManager: Helper that downloads, parses, and caches robots.txt configuration dynamically.
RobotsTxt: Parser for robots.txt rules.
SchemaConfig: Base schema configuration representing target attributes to extract.
SimpleSpidrElement: Basic implementation of SpidrElement used for stubs and fallback parsing.
SimpleSpidrPage: Basic implementation of SpidrPage used for stubs and fallback document wrappers.
Spider: Subclass this to define spider-specific starting conditions and response parsing workflows.
Spidr: The root orchestration facade for the SPIDR web scraping framework.
SpidrBrowser: Entry interface for orchestrating Chrome DevTools Protocol (CDP) automated browsers.
SpidrBrowserPage: Abstract wrapper exposing automation page capabilities.
SpidrCapabilities: Exposes the platform capability profile of the current running host.
SpidrClient: The core interface for components executing network requests.
SpidrCookie: Represents a parsed HTTP Cookie structure inside SPIDR.
SpidrCookieJar: A platform-agnostic container managing, matching, and storing Cookies.
SpidrCrawler: Orchestrates the crawling and execution loop.
SpidrDecoderRegistry: A central registry to map and lookup model decoder functions at runtime.
SpidrElement: Represents a strongly-typed, abstract wrapper around a parsed DOM element.
SpidrFingerprintRegistry: Global registry holding the active FingerprintStore instance.
SpidrLoggingMiddleware: A middleware that logs HTTP requests and responses to the console or developer log.
SpidrMiddleware: Middleware hook interface for intercepting requests and responses in SPIDR.
SpidrPage: Represents a parsed page document, either from a raw HTTP response or a browser window.
SpidrPlugin: Represents a modular extension that attaches functionality to the SPIDR ecosystem.
SpidrPluginRegistry: Manages registration and lookups of SPIDR plugins at runtime.
SpidrRateLimiter: Throttling helper enforcing requests-per-second and concurrency limits in SPIDR.
SpidrRendererRegistry: A global registry that decoupled browser packages can use to register their page rendering engines.
SpidrRequest: Represents an HTTP or Browser navigation request in SPIDR.
SpidrResponse: Represents the completed response retrieved from a SPIDR scrape.
SpidrSession: Represents a complete snapshot of network/browser states, including cookies, headers, local storage, and IndexedDB data.
SpidrSessionRegistry: Global registry for configuring the active session storage manager.
SpidrSessionStore: Abstract contract for managing session storage environments.
SpidrStorage: Factory utility to resolve the best available storage engine.
SpidrStreamExtractor: Helper that feeds a stream of requests into a concurrent executing pipeline.
SpidrWorkerPool: A persistent pool of Dart Isolates used to execute scraping network requests in parallel.
SpidrXpath: A lightweight, custom XPath parser and selector evaluator walking standard HTML DOM trees.
SqliteStorageAdapter
StealthConfig: Configuration profiles for the evasion and anti-fingerprinting plugin.
StealthMiddleware: Middleware hook to apply User-Agent rotation and header sequence randomizations.
StealthPlugin: Core interface defining anti-fingerprinting and browser masking configurations.
StorageAdapter: Database adapter interface for storing crawl checkpoint frontiers, dynamic cookies, and fingerprints.
StorageFingerprintStore: Persistent implementation of FingerprintStore backed by a StorageAdapter.
StorageSessionStore: Persistent implementation of SpidrSessionStore backed by a StorageAdapter.
UserAgentGenerator: Generates organic User-Agent profiles for requests and browser engines.
ValidatingProxyPool: Validation proxy pool extension supporting validation timeouts and status checks.
WorkerResult: Represents the execution results returned from a worker isolate.
XpathStep: Represents a parsed step configuration within an XPath expression path query.

Enums

BackoffStrategy: Strategies to compute delays between retry attempts.
CrawlStrategy: Supported crawler search strategies.
ProxyProtocol: Supported networking proxy protocols.
ProxyStrategy: Proxy rotation strategies.

Functions

calculateSimilarity(ElementFingerprint a, ElementFingerprint b) → double: Calculates a similarity score between two fingerprints, returning a value between 0.0 and 1.0.
findBestMatch(SpidrElement rootElement, ElementFingerprint target) → SpidrElement?: Traverses all elements inside rootElement and returns the best matching candidate for target. Returns null if no candidate meets the threshold (0.6).

Typedefs

PageRenderer = Future<SpidrPage> Function(SpidrPage page, {List<String> scriptTriggers, Duration? timeout, String? waitSelector})

Exceptions / Errors

SpidrBrowserException: Thrown when a browser automation action fails.
SpidrException: Base class for all exceptions thrown by the SPIDR framework.
SpidrNetworkException: Thrown when request execution fails (e.g., timeout, network connection, proxy issues).
SpidrParseException: Thrown when parsing HTML/XML DOM structures fails.
UnsupportedCapabilityException: Thrown when a requested framework capability is not supported on the current platform.

Classes

Enums

Functions

Typedefs

Exceptions / Errors

spidr package

spidr library