web_scraper 0.1.4 copy "web_scraper: ^0.1.4" to clipboard
web_scraper: ^0.1.4 copied to clipboard

A simple web scraper to scrape HTML tags and their attributes to cast them into Lists and Maps for dart and flutter.

Pub

A Simple Web Scraper for Dart & Flutter #

A very basic web scraper implementation to scrap html elements from a web page.

Pull requests are most welcome.

Getting Started #

In your pubspec.yaml root add:

dependencies:
  web_scraper:

then,

import 'package:web_scraper/web_scraper.dart';

Note that as of version 0.0.6, the project supports not only Flutter projects, but also Dart projects.

Implementation #

    final webScraper = WebScraper('https://webscraper.io');
    if (await webScraper.loadWebPage('/test-sites/e-commerce/allinone')) {
        List<Map<String, dynamic>> elements = webScraper.getElement('h3.title > a.caption', ['href']);
        print(elements);
    }

Checkout web_scraper_test.dart file to have closer look on all functionalities.

Methods #

Method Description Arguments Return Type
loadWebPage Loads the webpage into response object and then parse it into the document object String route Future <bool>
loadFromURL Loads the webpage from provided URL into response object and then parse it into the document object String page Future <bool>
loadFromString Loads the webpage from a String (usually stored by the getPageContent method) into response object and then parse it into the document object. This operation is completely synchronous and exists as a helper method to perform compute() flutter operations and avoid jank String responseBodyAsString Future <bool>
getPageContent Returns webpage's html in string format Void String body
getElement Returns List of elements found at specified address String address, List <String> attributes List <Map<String, dynamic>>
getElementTitle Returns List of element titles found at specified address String address List <String>
getElementAttribute Returns List of elements single attribute found at specified address (if you wish to get multiple attributes at once, please use getElement instead) String address, List <String> attributes List <String>
getAllScripts Returns the list of all data enclosed in script tags of the document Void List <String>
getScriptVariables Returns Map between given variable names and list of their occurence in the script tags List <String> variableNames Map <String, dynamic>

Contributing #

  • Please branch from develop to implement bug fix/new feature.
  • Ensure that code is formatted according to base dart rules & using the latest stable version of dart.
  • Open a PR with develop as the PR target with a clear description.
288
likes
140
pub points
89%
popularity

Publisher

verified publishertusharojha.com

A simple web scraper to scrape HTML tags and their attributes to cast them into Lists and Maps for dart and flutter.

Repository (GitHub)
View/report issues

Documentation

API reference

License

Apache-2.0 (license)

Dependencies

html, http

More

Packages that depend on web_scraper