jsonstreamreader 1.1.9

  • Readme
  • Changelog
  • Example
  • Installing
  • 63

Json Stream Reader #

A Flutter Json Stream Reader

This project processes a local json file by splitting it into smaller chunks, with automatic garbage collecting.

Classes #

Streamer #

This class defines the size of each chunk,optionally values include

  1. chunksize

    default is 100

  2. start

    default is 0 and defines where Reader should start parsing file

  3. end

    default is the end of the file and defines where Reader should stop Reading file

Readers #

All readers have optional values which include

  1. int delay

how long the reader should wait before reading next chunk in microseconds

  1. int combine

    If combine is 1, array only values will be combined ie. [1,2,3] will result in {"0":1,"1":2,"2":3}, object values in an array are not affected ie. [{},{}] which mean [1,2,{"third":3}] will result in {"0":1,"1":2,"2":3, {"third":3} }

    If combine is 2, array values will be collasped into an object closest to the root eg. {'first':{'second':{ "third":3 } } } will be collaped to {'first':{'/second/third/':3 } }

  2. Reader

  3. StreamReader Provides access to json data by using StreamController

  4. StreamItem

  5. ReaderState

  6. StreamReaderState

  7. GlobalReaderManager It is used to pause or resume all or one Reader or StreamReader.

  8. FileState It is used to determine when a file is no longer in use.

Using JsonPath "Hack" in filter #

ExpressionPath
$.^\/$
[0:9][0>= <=9]\/$
[:9]+[0>= <=9]+\/$
..\/.*\/.*\/$
...\/.*\/.*\/.*\/$
$.*/.* & /
.*.*
[*].*
{name}any value with key "name"

Operators #

SymbolMeaning
&AND
|OR
!NOT

Operations can be used with path or jsonpath combinations. example:

  1. "{name} & /items/"

    all direct descendants of items with at least one key name

  2. "({name} | !/) & $.items.*"

    all objects with at least one key name or not a direct descendant of root and that is a descendant of items

The order does not matter(Operations was improved using https://github.com/riichard/boolean-parser-js)

Future #

After doing some benchmarking I found out these function are very expensive, so I added futures to the mix, now these futures are optional and must not be null.

What order are futures executed? #

  1. trailing
  2. filter

In most cases not all parts of an object is present in the first value instance, in this case a transformer will be handy for StreamReader while other methods can be used for Reader.

Benchmarks

  1. 02.json takes 2 milliseconds to execute
  2. citylots.json takes 8 minutes to execute this is mostly because of how complex the dataset is.
  3. www.carqueryapi.com takes about 2.3 milliseconds, please note that the file had to be modified I remove ?( from the beginning and ); at the end.

Execution time depends exclusively on the complexity of the file and the number of items to be processed. ie. a file 3 times larger than citylots.json can take approximately five minutes to execute.Using combine:2 can speed up processing considerably.

This project was tested using

  1. https://github.com/zemirco/sf-city-lots-json/blob/master/citylots.json
  2. https://github.com/thaiwsa/aws-speed/blob/master/JsonProcess/jsondata/02.json
  3. http://www.carqueryapi.com/api/0.3/?callback=?&cmd=getMakes&year=1970&sold_in_us=1&utm_medium=referral&utm_campaign=ZEEF&utm_source=https%3A%2F%2Fjson-datasets.zeef.com%2Fjdorfman
  4. https://catalogue.data.gov.bc.ca/dataset/children-and-family-development-cases-in-care-demographics

[1.1.1] - TODO: July 24.

  • Bug Fixes
  • Limited pause added to RemoteSource
  • pause and resume added to Reader and StreamReader

[1.1.6] - TODO: July 4.

  • Optimization
  • Better Api References

[1.1.7] - TODO: July 8.

  • Optimization
  • Chunksize bug fix
  • Added unit to Streamer

[1.1.8] - TODO: July 8.

  • Optimization
  • findLength property and setLength method added to Streamer 8 With combine 1, citylots now takes on average 2mins 20seconds

[1.1.9] - TODO: July 8.

  • Filter Bug Fix

[1.2.0] - TODO: July 14.

  • Optimization for RemoteSource

example/main.dart

import 'package:jsonstreamreader/jsonstreamreader.dart';
import 'dart:async';

void main() {
  RemoteSource f =
      new RemoteSource('https://jsonplaceholder.typicode.com/posts');
  f.setBody({'userId': '1'});
  Streamer s = new Streamer(f, chunksize: 100);
  //Not Ready to start reading so I use hault
  Reader r = new Reader(s, combine: 1)..hault();
  //StreamReader r = new StreamReader(s, combine: 1)..hault();
  r.trailing((dynamic value, String key) {
    print("key is ${key} and value is ${value}");
  });

  ///r.trailing.listen((StreamItem item) {
  ///  print("key is ${item.key} and value is ${item.value}");
  ///});
  r.progress((p) {
    print(p);
  }).done(() {
    print('Completed');
  }).fail((error) {
    print(error);
  });

  ///r.progress.listen((double p) {
  ///  print("completed ${p}%");
  ///});
  ///r.done.listen((_) {
  ///  print("completed");
  ///});
  ///r.error.listen((error) {
  ///  print(error);
  ///});
  new Timer(new Duration(seconds: 1), () {
    //start reading
    r.begin();
  });
  FileState.notinuse.listen((d) {
    print('this file is free to delete');
    print(d);
  });
  r.onpause((ReaderState state) {
    print(state.toString());
  });

  ///Pause the reader
  new Timer(new Duration(seconds: 3), () {
    r.pause().then((ReaderState state) {
      //here you code store state.toString() in a file or just resume
      new Timer(new Duration(seconds: 3), () {
        r.resume();
      });
    });
  });
  f.downloaded.listen((onData) {
    //File will always be download before parsing is done
    //but parsing is started as soon as data is available,i.e while the file is downloading
    print('File has been downloaded');
  });
  f.download();
}

Use this package as a library

1. Depend on it

Add this to your package's pubspec.yaml file:


dependencies:
  jsonstreamreader: ^1.1.9

2. Install it

You can install packages from the command line:

with Flutter:


$ flutter pub get

Alternatively, your editor might support flutter pub get. Check the docs for your editor to learn more.

3. Import it

Now in your Dart code, you can use:


import 'package:jsonstreamreader/jsonstreamreader.dart';
  
Popularity:
Describes how popular the package is relative to other packages. [more]
25
Health:
Code health derived from static analysis. [more]
100
Maintenance:
Reflects how tidy and up-to-date the package is. [more]
100
Overall:
Weighted score of the above. [more]
63
Learn more about scoring.

We analyzed this package on Oct 18, 2019, and provided a score, details, and suggestions below. Analysis was completed with status completed using:

  • Dart: 2.5.1
  • pana: 0.12.21
  • Flutter: 1.9.1+hotfix.4

Platforms

Detected platforms: Flutter

References Flutter, and has no conflicting libraries.

Health suggestions

Format lib/source/remote/futureread.dart.

Run flutter format to format lib/source/remote/futureread.dart.

Format lib/streamerunit.dart.

Run flutter format to format lib/streamerunit.dart.

Dependencies

Package Constraint Resolved Available
Direct dependencies
Dart SDK >=2.1.0 <3.0.0
flutter 0.0.0
heavylist ^0.1.0 0.1.1
path ^1.6.2 1.6.4
path_provider ^1.1.0 1.3.1
Transitive dependencies
collection 1.14.11 1.14.12
meta 1.1.7
platform 2.2.1
sky_engine 0.0.99
typed_data 1.1.6
vector_math 2.0.8
Dev dependencies
flutter_test