puppeteer_page_walker 0.1.0+1

  • Readme
  • Changelog
  • Example
  • Installing
  • 66

Pub

puppeteer-page-walker #

A wrapper library of puppeteer for humane scraping :) Let's write the scraping scenario separately for each browsing URL.

Install #

Just add dependency into pubspec.yaml,

dependencies:
  puppeteer_page_walker: ^0.1.0

or specify the github url for using the latest functions.

dependencies:
  puppeteer_page_walker:
    git:
      url: git://github.com/YusukeIwaki/puppeteer-dart-page-walker.git

Enjoy! #

import 'dart:io';

import 'package:puppeteer/puppeteer.dart';
import 'package:puppeteer_page_walker/puppeteer_page_walker.dart';

main(List<String> args) async {
  final browser = await puppeteer.launch();

  await PageWalker(browser).initWith((page) async {
    // browse github.com.
    await page.setViewport(DeviceViewport(width: 1200, height: 480));
    await page.goto("https://github.com/");
  }).forEachPage((page) async {
    // debug print page url for each access.
    print("[${DateTime.now()}] ${page.url}");
  }).andIfUrlIs("https://github.com/", (page) async {
    // search "puppeteer" in github.com.
    final form = await page.$("form.js-site-search-form");
    final searchInput = await form.$("input.header-search-input");
    await searchInput.type("puppeteer");
    await searchInput.press(Key.enter);
  }).andIf((url) => url.startsWith("https://github.com/search"), (page) async {
    // extract repo title from search results.
    final repoList = await page.$("ul.repo-list");
    final repoItems = await repoList.$$("h3");
    await Future.forEach(repoItems, (item) async {
      final String title = await item.$eval("a", "a => a.innerText");
      print("==> $title");
    });

    // goodbye!
    await browser.close();
  }).startWalking();
}

0.1.0 #

0.1.0+1 #

  • Just modifying README, CHANGELOG. No functional updates.

example/puppeteer_page_walker.dart

import 'dart:io';

import 'package:puppeteer/puppeteer.dart';
import 'package:puppeteer_page_walker/puppeteer_page_walker.dart';

main(List<String> args) async {
  final browser = await puppeteer.launch(
    executablePath: Platform.environment['PUPPETEER_EXECUTABLE_PATH'],
    headless: false,
  );

  await PageWalker(browser).initWith((page) async {
    // browse github.com.
    await page.setViewport(DeviceViewport(width: 1200, height: 480));
    await page.goto("https://github.com/");
  }).forEachPage((page) async {
    // debug print page url for each access.
    print("[${DateTime.now()}] ${page.url}");
  }).andIfUrlIs("https://github.com/", (page) async {
    // search "puppeteer" in github.com.
    final form = await page.$("form.js-site-search-form");
    final searchInput = await form.$("input.header-search-input");
    await searchInput.type("puppeteer");
    await searchInput.press(Key.enter);
  }).andIf((url) => url.startsWith("https://github.com/search"), (page) async {
    // extract repo title from search results.
    final repoList = await page.$("ul.repo-list");
    final repoItems = await repoList.$$("h3");
    await Future.forEach(repoItems, (item) async {
      final String title = await item.$eval("a", "a => a.innerText");
      print("==> $title");
    });

    // goodbye!
    await browser.close();
  }).startWalking();
}

Use this package as a library

1. Depend on it

Add this to your package's pubspec.yaml file:


dependencies:
  puppeteer_page_walker: ^0.1.0+1

2. Install it

You can install packages from the command line:

with pub:


$ pub get

with Flutter:


$ flutter pub get

Alternatively, your editor might support pub get or flutter pub get. Check the docs for your editor to learn more.

3. Import it

Now in your Dart code, you can use:


import 'package:puppeteer_page_walker/puppeteer_page_walker.dart';
  
Popularity:
Describes how popular the package is relative to other packages. [more]
32
Health:
Code health derived from static analysis. [more]
99
Maintenance:
Reflects how tidy and up-to-date the package is. [more]
100
Overall:
Weighted score of the above. [more]
66
Learn more about scoring.

We analyzed this package on Mar 30, 2020, and provided a score, details, and suggestions below. Analysis was completed with status completed using:

  • Dart: 2.7.1
  • pana: 0.13.6

Health issues and suggestions

Document public APIs. (-1 points)

9 out of 9 API elements have no dartdoc comment.Providing good documentation for libraries, classes, functions, and other API elements improves code readability and helps developers find and use your API.

Dependencies

Package Constraint Resolved Available
Direct dependencies
Dart SDK >=2.6.0-dev.6.0 <3.0.0
puppeteer ^1.14.1 1.16.1
Transitive dependencies
archive 2.0.13
args 1.6.0
async 2.4.1
charcode 1.1.3
collection 1.14.12
convert 2.1.1
crypto 2.1.4
http 0.12.0+4
http_parser 3.1.4
logging 0.11.4
meta 1.1.8
mime 0.9.6+3
path 1.6.4
petitparser 3.0.2
pool 1.4.0
source_span 1.7.0
stack_trace 1.9.3
string_scanner 1.0.5
term_glyph 1.1.0
typed_data 1.1.6
Dev dependencies
pedantic ^1.8.0 1.9.0