pub_mirror 1.0.5

Pub Mirror #

Pub Package Docker Image

A multi-thread tool to download content from Pub. Then the result can be served by a http server and taken as the backend of pub, flutter or even Pub Mirror itself.

With this tool, you can build a local mirror of pub with no need of unstable reverse proxy or dynamic server like pub_server.

Installation #

It can be installed from pub by:

$ pub global activate pub_mirror

or installed as a docker image by:

$ docker pull huiyiqun/pub_mirror

or installed from source directly by:

$ cd path/to/pub-mirror
$ pub get
# then it can be run by following command
$ dart bin/dart_mirror.dart

Using the tool #

$ pub_mirror --help
Usage:
pub_mirror [options] <dest-path> <serving-url>

Example: pub_mirror /tmp/pub/ file:///tmp/pub/

Options:
-h, --help              print usage and exit
-v, --verbose           more verbose output
-o, --[no-]overwrite    overwrite existing meta files
-u, --upstream          the upstream to mirror from
                        (defaults to "https://pub.dartlang.org/api")

-c, --connections       max number of connections
                        (defaults to "10")

-p, --concurrency       max number of packages to download in parallel
                        (defaults to "1")

The dest-path is where you would like to save the packages and serving-url is the base url that you would like to serving on.

file:///tmp/pub/ is just used as an example, the file scheme is not supported by pub.

The packages are downloaded incrementally, which means:

  1. If the process is interrupted and resumed, packages and versions has been downloaded will be skipped.
  2. If the process is completed and restarted again, only new packages and new releases will be downloaded.

Service #

In order to make the mirror accessible by the client, we must serve it with a web server.

An example for nginx is:

http {
  server {
    listen 80;  # identical to the scheme port in the serving-url
    listen ssl 443;  # if the scheme in the serving-url is https
    server_name example.com;  # identical to the hostname in the serving-url

    location /pub/ {  # identical to the path in the serving-url
      root path/to/path;  # identical to the dest-path
      location /pub/api/ {
        try_files $uri $uri/meta.json =404;  # information of packages and versions are saved in meta.json
      }
    }
  }
}

Start the nginx and enjoy the speed!

Test #

To test the configuration of the web server, You can visit following URLs:

  1. http://example.com/pub/api/packages # a large json file
  2. http://example.com/pub/api/packages/pub_mirror # a json file
  3. http://example.com/pub/api/packages/pub_mirror/versions/0.1.0 # a json file
  4. http://example.com/pub/packages/pub_mirror/versions/0.1.0.tar.gz # an archive file

Using the mirror #

Setting the environment PUB_HOSTED_URL to the serving-url, then both pub and flutter will download packages from your mirror.

$ export PUB_HOSTED_URL="http://example.com/pub/"
$ pub get  # downloaded from http://example.com/pub/
$ flutter packages get  # downloaded from http://example.com/pub/

Use case #

Dart repo of TUNA mirrors is served by this tool.

0.1.0 #

  • initial version with the basic functions

0.1.1 #

  • saving meta.json to temp file and move it to the destination (not working)
  • fix some bugs

0.1.2 #

  • add dependency on http
  • fix some bugs

0.1.3 #

  • make the pub_mirror available as an executable
  • mark unwanted Futures

0.1.5 #

  • make the comments dartdoc-friendly

0.1.6 #

  • downgrade pedantic to 1.4.0

0.1.7 #

  • published as a docker image
  • fix the bug that the meta.json of the package is incorrect when some versions of the package is skipped
  • new options to overwrite the meta.json even if it has existed.

0.1.8 #

  • save meta.json for available packages

0.1.9 #

  • add documentations
  • add examples
  • upstream become an optional argument of PubMirrorTool

1.0.0 #

  • install all the dependencies from pub

1.0.1 #

  • improve the code

1.0.2 #

  • print the status of the executor during downloading

1.0.3 #

  • display the progress of downloading

example/pub_mirror.dart

import 'package:pub_mirror/pub_mirror.dart' as pub_mirror;

main() async {
  final tool = pub_mirror.PubMirrorTool('/tmp/pub/', 'http://example.com/pub/',
      upstream: 'https://pub.dartlang.org/api',
      verbose: true,
      maxConnections: 10);

  // iterate over all the packages
  await for (var package in tool.listAllPackages()) {
    print('-> ${package.name}');
  }

  // download the single package
  await tool.downloadPackage('pub_mirror');

  // download all packages with 100 threads
  await tool.download(100, overwrite: false);
}

Use this package as an executable

1. Install it

You can install the package from the command line:


$ pub global activate pub_mirror

2. Use it

The package has the following executables:


$ pub_mirror

Use this package as a library

1. Depend on it

Add this to your package's pubspec.yaml file:


dependencies:
  pub_mirror: ^1.0.5

2. Install it

You can install packages from the command line:

with pub:


$ pub get

Alternatively, your editor might support pub get. Check the docs for your editor to learn more.

3. Import it

Now in your Dart code, you can use:


import 'package:pub_mirror/pub_mirror.dart';
  
Popularity:
Describes how popular the package is relative to other packages. [more]
0
Health:
Code health derived from static analysis. [more]
100
Maintenance:
Reflects how tidy and up-to-date the package is. [more]
90
Overall:
Weighted score of the above. [more]
48
Learn more about scoring.

We analyzed this package on Jul 20, 2019, and provided a score, details, and suggestions below. Analysis was completed with status completed using:

  • Dart: 2.4.0
  • pana: 0.12.19

Platforms

Detected platforms: other

Primary library: package:pub_mirror/pub_mirror.dart with components: io, mirrors.

Health suggestions

Format lib/http.dart.

Run dartfmt to format lib/http.dart.

Maintenance issues and suggestions

Support latest dependencies. (-10 points)

The version constraint in pubspec.yaml does not support the latest published versions for 1 dependency (http).

Dependencies

Package Constraint Resolved Available
Direct dependencies
Dart SDK >=2.1.0 <3.0.0
args ^1.5.1 1.5.2
executor ^2.1.2 2.1.2
http ^0.11.3+17 0.11.3+17 0.12.0+2
logging ^0.11.3+2 0.11.3+2
path ^1.6.2 1.6.2
pedantic ^1.4.0 1.8.0+1
pub_client ^3.0.3 3.0.3
quiver_log ^1.0.6 1.0.6
Transitive dependencies
async 2.3.0
charcode 1.1.2
collection 1.14.11
http_parser 3.1.3
intl 0.15.8
json_annotation 2.4.0
meta 1.1.7
source_span 1.5.5
stack_trace 1.9.3
string_scanner 1.0.4
term_glyph 1.1.0
typed_data 1.1.6

Admin