crawlberg library

Classes

ActionResult

Result from a single page action execution.

ArticleMetadata

Article metadata extracted from article:* Open Graph tags.

AuthConfig

BatchCrawlResult

Result from a single URL in a batch crawl operation.

BatchCrawlResults

Aggregate result of a batch crawl, exposing per-URL results plus precomputed counts.

BatchCrawlStreamRequest

Request to begin a multi-URL streaming crawl.

BatchScrapeResult

Result from a single URL in a batch scrape operation.

BatchScrapeResults

Aggregate result of a batch scrape, exposing per-URL results plus precomputed counts.

BrowserConfig

Browser fallback configuration.

BrowserExtras

Browser-specific extras populated when the native browser backend was used.

CitationReference

A single numbered reference in a citation list — produced by the citation extractor when content uses inline [N]-style markers.

CitationResult

Result of citation conversion.

ContentConfig

Content extraction and conversion configuration.

CookieInfo

Information about an HTTP cookie received from a response.

CrawlbergBridge

CrawlConfig

Configuration for crawl, scrape, and map operations.

CrawlEngineHandle

CrawlError

CrawlEvent

CrawlPageResult

The result of crawling a single page during a crawl operation.

CrawlResult

The result of a multi-page crawl operation.

CrawlStreamRequest

Request to begin a single-URL streaming crawl.

DownloadedAsset

A downloaded asset from a page.

DownloadedDocument

A downloaded non-HTML document (PDF, DOCX, image, code file, etc.).

ExtractionMeta

Metadata about an LLM extraction pass.

FaviconInfo

Information about a favicon or icon link.

FeedInfo

Information about a feed link found on a page.

HeadingInfo

A heading element extracted from the page.

HostMatcher

HreflangEntry

An hreflang alternate link entry.

ImageInfo

Information about an image found on a page.

InteractionResult

Result of executing a sequence of page interaction actions.

JsonLdEntry

A JSON-LD structured data entry found on a page.

LinkInfo

Information about a link found on a page.

MapResult

The result of a map operation, containing discovered URLs.

MarkdownResult

Rich markdown conversion result from HTML processing.

PageAction

PageMetadata

Metadata extracted from an HTML page's <meta> tags and <title> element.

ProxyConfig

Proxy configuration for HTTP requests.

ResponseMeta

Response metadata extracted from HTTP headers.

ScrapeResult

The result of a single-page scrape operation.

SitemapUrl

A URL entry from a sitemap.

SsrfError

SsrfPolicy

SSRF policy configuration.

Enums

AssetCategory

The category of a downloaded asset.

BrowserBackend

Browser backend used for JavaScript rendering.

BrowserMode

When to use the headless browser fallback.

BrowserWait

Wait strategy for browser page rendering.

ContentFilterKind

Content filter applied to each crawled page before it reaches the result.

CrawlStrategyKind

Traversal order for a crawl.

DocumentContentEncoding

Opt-in encoding applied to a downloaded document's bytes for callers who need the content available in a serializable field rather than reading it from disk.

FeedType

The type of a feed (RSS, Atom, or JSON Feed).

ImageSource

The source of an image reference.

LinkType

The classification of a link.

ScrollDirection

Direction for a scroll action.

Extensions

AssetCategoryWireValue on AssetCategory

AuthConfigPatterns on AuthConfig

Adds pattern-matching-related methods to AuthConfig.

BrowserBackendWireValue on BrowserBackend

BrowserModeWireValue on BrowserMode

BrowserWaitWireValue on BrowserWait

ContentFilterKindWireValue on ContentFilterKind

CrawlErrorPatterns on CrawlError

Adds pattern-matching-related methods to CrawlError.

CrawlEventPatterns on CrawlEvent

Adds pattern-matching-related methods to CrawlEvent.

CrawlStrategyKindWireValue on CrawlStrategyKind

DocumentContentEncodingWireValue on DocumentContentEncoding

FeedTypeWireValue on FeedType

HostMatcherPatterns on HostMatcher

Adds pattern-matching-related methods to HostMatcher.

ImageSourceWireValue on ImageSource

LinkTypeWireValue on LinkType

PageActionPatterns on PageAction

Adds pattern-matching-related methods to PageAction.

ScrollDirectionWireValue on ScrollDirection

SsrfErrorPatterns on SsrfError

Adds pattern-matching-related methods to SsrfError.

Functions

batchCrawl({required CrawlEngineHandle engine, required List<String> urls}) → Future<BatchCrawlResults>

Crawl multiple seed URLs concurrently, each following links to configured depth.

batchScrape({required CrawlEngineHandle engine, required List<String> urls}) → Future<BatchScrapeResults>

Scrape multiple URLs concurrently.

crawl({required CrawlEngineHandle engine, required String url}) → Future<CrawlResult>

Crawl a website starting from url, following links up to the configured depth.

createActionResultFromJson({required String json}) → Future<ActionResult>

createArticleMetadataFromJson({required String json}) → Future<ArticleMetadata>

createAssetCategoryFromJson({required String json}) → Future<AssetCategory>

createAuthConfigFromJson({required String json}) → Future<AuthConfig>

createBatchCrawlResultFromJson({required String json}) → Future<BatchCrawlResult>

createBatchCrawlResultsFromJson({required String json}) → Future<BatchCrawlResults>

createBatchCrawlStreamRequestFromJson({required String json}) → Future<BatchCrawlStreamRequest>

createBatchScrapeResultFromJson({required String json}) → Future<BatchScrapeResult>

createBatchScrapeResultsFromJson({required String json}) → Future<BatchScrapeResults>

createBrowserBackendFromJson({required String json}) → Future<BrowserBackend>

createBrowserConfigFromJson({required String json}) → Future<BrowserConfig>

createBrowserExtrasFromJson({required String json}) → Future<BrowserExtras>

createBrowserModeFromJson({required String json}) → Future<BrowserMode>

createBrowserWaitFromJson({required String json}) → Future<BrowserWait>

createCitationReferenceFromJson({required String json}) → Future<CitationReference>

createCitationResultFromJson({required String json}) → Future<CitationResult>

createContentConfigFromJson({required String json}) → Future<ContentConfig>

createContentFilterKindFromJson({required String json}) → Future<ContentFilterKind>

createCookieInfoFromJson({required String json}) → Future<CookieInfo>

createCrawlConfigFromJson({required String json}) → Future<CrawlConfig>

createCrawlEventFromJson({required String json}) → Future<CrawlEvent>

createCrawlPageResultFromJson({required String json}) → Future<CrawlPageResult>

createCrawlResultFromJson({required String json}) → Future<CrawlResult>

createCrawlStrategyKindFromJson({required String json}) → Future<CrawlStrategyKind>

createCrawlStreamRequestFromJson({required String json}) → Future<CrawlStreamRequest>

createDocumentContentEncodingFromJson({required String json}) → Future<DocumentContentEncoding>

createDownloadedAssetFromJson({required String json}) → Future<DownloadedAsset>

createDownloadedDocumentFromJson({required String json}) → Future<DownloadedDocument>

createEngine({CrawlConfig? config}) → Future<CrawlEngineHandle>

Create a new crawl engine with the given configuration.

createExtractionMetaFromJson({required String json}) → Future<ExtractionMeta>

createFaviconInfoFromJson({required String json}) → Future<FaviconInfo>

createFeedInfoFromJson({required String json}) → Future<FeedInfo>

createFeedTypeFromJson({required String json}) → Future<FeedType>

createHeadingInfoFromJson({required String json}) → Future<HeadingInfo>

createHostMatcherFromJson({required String json}) → Future<HostMatcher>

createHreflangEntryFromJson({required String json}) → Future<HreflangEntry>

createImageInfoFromJson({required String json}) → Future<ImageInfo>

createImageSourceFromJson({required String json}) → Future<ImageSource>

createInteractionResultFromJson({required String json}) → Future<InteractionResult>

createJsonLdEntryFromJson({required String json}) → Future<JsonLdEntry>

createLinkInfoFromJson({required String json}) → Future<LinkInfo>

createLinkTypeFromJson({required String json}) → Future<LinkType>

createMapResultFromJson({required String json}) → Future<MapResult>

createMarkdownResultFromJson({required String json}) → Future<MarkdownResult>

createPageActionFromJson({required String json}) → Future<PageAction>

createPageMetadataFromJson({required String json}) → Future<PageMetadata>

createProxyConfigFromJson({required String json}) → Future<ProxyConfig>

createResponseMetaFromJson({required String json}) → Future<ResponseMeta>

createScrapeResultFromJson({required String json}) → Future<ScrapeResult>

createScrollDirectionFromJson({required String json}) → Future<ScrollDirection>

createSitemapUrlFromJson({required String json}) → Future<SitemapUrl>

createSsrfPolicyFromJson({required String json}) → Future<SsrfPolicy>

generateCitations({required String markdown}) → Future<CitationResult>

Convert markdown links to numbered citations.

interact({required CrawlEngineHandle engine, required String url, required List<PageAction> actions}) → Future<InteractionResult>

Execute browser actions on a single page.

mapUrls({required CrawlEngineHandle engine, required String url}) → Future<MapResult>

Discover all pages on a website by following links and sitemaps.

scrape({required CrawlEngineHandle engine, required String url}) → Future<ScrapeResult>

Scrape a single URL, returning extracted page data.