web_scraper 0.0.5

  • Readme
  • Changelog
  • Example
  • Installing
  • new79

Pub

A Simple Web Scraper for Dart & Flutter #

A very basic web scraper implementation to scrap html elements from a web page.

Pull requests certainly welcome.

Installation #

In your pubspec.yaml root add:

dependencies:
  web_scraper: LATEST_VERSION_NUMBER

then,

import 'package:web_scraper/web_scraper.dart';

Implementation #

    final webScraper = WebScraper('https://webscraper.io');
    if(await webScraper.loadWebPage('/test-sites/e-commerce/allinone')){
        List<Map<String, dynamic>> elements = webScraper.getElement('h3.title > a.caption', ['href']);
        print(elements);
    }

Methods #

MethodDescriptionArgumentsReturn Type
loadWebPageLoads the webpage into response objectString routeFuture <bool>
getPageContentReturns webpage's html in string formatVoidString body
getElementReturns List of elements found at specified addressString address, List <String> attributesList <Map<String, dynamic>>
getAllScriptsReturns the list of all data enclosed in script tags of the documentVoidList <String>
getScriptVariablesReturns Map between given variable names and list of their occurence in the script tagsList <String> variableNamesMap <String, dynamic>

Contribute to the package at GitHub. #

  • File bugs, features, etc.
  • Fix bugs and send pull requests
  • Review pull requests

[0.0.5] - 03/04/2020 #

  • getAllScripts added
  • getScriptVariables implemented
  • Improved error reporting

[0.0.3] - 22/03/2020 #

  • Converted List of Maps of attributes to a single Map in repsonse
  • Web_Scraper Tests added
  • Added more description

[0.0.1] - 20/03/2020 #

  • Initial Web Scraper Created.

example/lib/main.dart

import 'package:flutter/material.dart';
import 'package:url_launcher/url_launcher.dart';
import 'package:web_scraper/web_scraper.dart';

void main() => runApp(WebScraperApp());

class WebScraperApp extends StatefulWidget {
  @override
  _WebScraperAppState createState() => _WebScraperAppState();
}

class _WebScraperAppState extends State<WebScraperApp> {

  // initialize WebScraper by passing base url of website
  final webScraper = WebScraper('https://webscraper.io');

  // Response of getElement is always List<Map<String, dynamic>>
  List<Map<String, dynamic>> productNames;

  void fetchProducts() async {
    // Loads web page and downloads into local state of library
    if (await webScraper
        .loadWebPage('/test-sites/e-commerce/allinone/computers/laptops')) {
      setState(() {
        // getElement takes the address of html tag/element and attributes you want to scrap from website
        // it will return the attributes in the same order passed
        productNames = webScraper.getElement(
            'div.thumbnail > div.caption > h4 > a.title', ['href', 'title']);
      });
    }
  }

  @override
  void initState() {
    super.initState();
    // Requesting to fetch before UI drawing starts
    fetchProducts();
  }

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Fetch Data Example',
      theme: ThemeData(
        primarySwatch: Colors.blue,
      ),
      home: Scaffold(
          appBar: AppBar(
            title: Text("Product Catalog"),
          ),
          body: SafeArea(
              child: productNames == null
                  ? Center(
                      child: CircularProgressIndicator(), // Loads Circular Loading Animation
                    )
                  : ListView.builder(
                      itemCount: productNames.length,
                      itemBuilder: (BuildContext context, int index) {
                        // Attributes are in the form of List<Map<String, dynamic>>.
                        Map<String, dynamic> attributes =
                            productNames[index]['attributes'];
                        return ExpansionTile(
                          title: Text(attributes['title']),
                          children: <Widget>[
                            Padding(
                              padding: const EdgeInsets.all(10.0),
                              child: InkWell(
                                onTap: () {
                                  // uses UI Launcher to launch in web browser & minor tweaks to generate url
                                  launch(webScraper.baseUrl +
                                      attributes['href']);
                                },
                                child: Text(
                                  "View Product",
                                  style: TextStyle(color: Colors.blue),
                                ),
                              ),
                            )
                          ],
                        );
                      }))),
    );
  }
}

Use this package as a library

1. Depend on it

Add this to your package's pubspec.yaml file:


dependencies:
  web_scraper: ^0.0.5

2. Install it

You can install packages from the command line:

with Flutter:


$ flutter pub get

Alternatively, your editor might support flutter pub get. Check the docs for your editor to learn more.

3. Import it

Now in your Dart code, you can use:


import 'package:web_scraper/web_scraper.dart';
  
Popularity:
Describes how popular the package is relative to other packages. [more]
61
Health:
Code health derived from static analysis. [more]
100
Maintenance:
Reflects how tidy and up-to-date the package is. [more]
90
Overall:
Weighted score of the above. [more]
79
Learn more about scoring.

We analyzed this package on Apr 7, 2020, and provided a score, details, and suggestions below. Analysis was completed with status completed using:

  • Dart: 2.7.1
  • pana: 0.13.6
  • Flutter: 1.12.13+hotfix.8

Health suggestions

Format lib/web_scraper.dart.

Run flutter format to format lib/web_scraper.dart.

Maintenance suggestions

Package is pre-v0.1 release. (-10 points)

While nothing is inherently wrong with versions of 0.0.*, it might mean that the author is still experimenting with the general direction of the API.

Dependencies

Package Constraint Resolved Available
Direct dependencies
Dart SDK >=2.1.0 <3.0.0
flutter 0.0.0
html ^0.14.0+3 0.14.0+3
http ^0.12.0+4 0.12.0+4
Transitive dependencies
async 2.4.1
charcode 1.1.3
collection 1.14.11 1.14.12
csslib 0.16.1
http_parser 3.1.4
meta 1.1.8
path 1.6.4
pedantic 1.9.0
sky_engine 0.0.99
source_span 1.7.0
string_scanner 1.0.5
term_glyph 1.1.0
typed_data 1.1.6
vector_math 2.0.8
Dev dependencies
flutter_test