Pub

A Simple Web Scraper for Dart & Flutter

A very basic web scraper implementation to scrap html elements from a web page.

Pull requests are most welcome.

Getting Started

In your pubspec.yaml root add:

dependencies:
  web_scraper:

then,

import 'package:web_scraper/web_scraper.dart';

Note that as of version 0.0.6, the project supports not only Flutter projects, but also Dart projects.

Implementation

    final webScraper = WebScraper('https://webscraper.io');
    if (await webScraper.loadWebPage('/test-sites/e-commerce/allinone')) {
        List<Map<String, dynamic>> elements = webScraper.getElement('h3.title > a.caption', ['href']);
        print(elements);
    }

Checkout web_scraper_test.dart file to have closer look on all functionalities.

Methods

MethodDescriptionArgumentsReturn Type
loadWebPageLoads the webpage into response object and then parse it into the document objectString routeFuture <bool>
loadFromURLLoads the webpage from provided URL into response object and then parse it into the document objectString pageFuture <bool>
loadFromStringLoads the webpage from a String (usually stored by the getPageContent method) into response object and then parse it into the document object. This operation is completely synchronous and exists as a helper method to perform compute() flutter operations and avoid jankString responseBodyAsStringFuture <bool>
getPageContentReturns webpage's html in string formatVoidString body
getElementReturns List of elements found at specified addressString address, List <String> attributesList <Map<String, dynamic>>
getElementTitleReturns List of element titles found at specified addressString addressList <String>
getElementAttributeReturns List of elements single attribute found at specified address (if you wish to get multiple attributes at once, please use getElement instead)String address, List <String> attributesList <String>
getAllScriptsReturns the list of all data enclosed in script tags of the documentVoidList <String>
getScriptVariablesReturns Map between given variable names and list of their occurence in the script tagsList <String> variableNamesMap <String, dynamic>

Contributing

  • Please branch from develop to implement bug fix/new feature.
  • Ensure that code is formatted according to base dart rules & using the latest stable version of dart.
  • Open a PR with develop as the PR target with a clear description.

Libraries

web_scraper
Web Scraper Library [...]