scrapeFromUrl static method

Future scrapeFromUrl(Map<String, Config> masterConfig, String? url, { String? html, Extract extract = Extract.product, Map<String, Config> priceConfig = const {}, })

Entry point for scraper. Requires, Map of Config and a url. You can pass pre-fetched html if it's available, so scraper can skip fetching HTML and start parsing directly. extract and priceConfig was meant to be used for e-commerce price fetching when only price was needed. You can ignore it.

Implementation

static Future scrapeFromUrl(
  Map<String, Config> masterConfig,
  String? url, {
  String? html,
  Extract extract = Extract.product,
  Map<String, Config> priceConfig = const {},
}) async {
  if (url != null) {
    //Find configuration file for URL and Extraction type!
    Config? config = getConfig(
      masterConfig,
      url,
      extract,
      priceConfig: priceConfig,
    );
    if (config == null) {
      return SpiderError(
        500,
        'This URL is not supported, Please try again!',
      );
    }

    //Pass config, url and html (optional) to scraping handler
    return await scrapingHandler(url, config, html: html);
  } else {
    return SpiderError(
      500,
      'URL should not be empty, Please try again.',
    );
  }
}