ai_webscraper library
A basic AI-powered web scraper for Dart
This library provides a simple yet powerful way to scrape web content using AI providers like OpenAI GPT and Google Gemini for structured data extraction.
Example usage:
final scraper = AIWebScraper(
aiModel: AIModel.gpt4o,
apiKey: 'your-openai-api-key',
);
final result = await scraper.extractFromUrl(
url: 'https://example.com',
schema: {
'title': 'string',
'description': 'string',
'price': 'number',
},
);
if (result.success) {
print('Extracted data: ${result.data}');
} else {
print('Error: ${result.error}');
}
Classes
- AIWebScraper
- Main AI-powered web scraper class.
- BatchProcessor
- Utility class for processing items in batches with concurrency control.
- Logger
- Simple logging utility for the AI WebScraper package.
- SchemaValidator
- Utility class for validating schema definitions.
- ScopedLogger
- A scoped logger that adds a prefix to all log messages.
- ScrapingResult
- Represents the result of a web scraping operation.
- Semaphore
- A semaphore implementation for controlling concurrency.
- URLValidator
- Utility class for validating URLs.
Enums
- AIModel
- Enumeration of all available AI models from supported providers
- AIProvider
- Enumeration of supported AI providers for web scraping.
- LogLevel
- Log levels for controlling logging output.
- SchemaType
- Enumeration of supported schema types for data extraction.
Extensions
- AIProviderExtension on AIProvider
- Extension methods for AIProvider enum.
- SchemaTypeExtension on SchemaType
- Extension methods for SchemaType enum.
Exceptions / Errors
- AIClientException
- Exception thrown when AI client operations fail.
- AIProviderException
- Exception thrown when AI provider operations fail.
- AIWebScraperException
- Base exception class for all AI WebScraper related exceptions.
- BatchProcessingException
- Exception thrown when batch processing fails.
- InvalidApiKeyException
- Exception thrown when API key validation fails.
- JavaScriptScrapingException
- Exception thrown when JavaScript scraping fails.
- ParsingException
- Exception thrown when data parsing fails.
- SchemaValidationException
- Exception thrown when schema validation fails.
- ScrapingException
- Exception thrown when web scraping fails.
- TimeoutException
- Exception thrown when timeout occurs.
- UnsupportedProviderException
- Exception thrown when an unsupported AI provider is requested.
- URLValidationException
- Exception thrown when URL validation fails.