Json Stream Reader

A Flutter Json Stream Reader

This project processes a local json file by splitting it into smaller chunks, with automatic garbage collecting.

Classes

Streamer

This class defines the size of each chunk,optionally values include

  1. chunksize

    default is 100

  2. start

    default is 0 and defines where Reader should start parsing file

  3. end

    default is the end of the file and defines where Reader should stop Reading file

Readers

All readers have optional values which include

  1. int delay

how long the reader should wait before reading next chunk in microseconds

  1. int combine

    If combine is 1, array only values will be combined ie. 1,2,3 will result in {"0":1,"1":2,"2":3}, object values in an array are not affected ie. {},{} which mean 1,2,{"third":3} will result in {"0":1,"1":2,"2":3, {"third":3} }

    If combine is 2, array values will be collasped into an object closest to the root eg. {'first':{'second':{ "third":3 } } } will be collaped to {'first':{'/second/third/':3 } }

  2. Reader

  3. StreamReader Provides access to json data by using StreamController

  4. StreamItem

  5. ReaderState

  6. StreamReaderState

  7. GlobalReaderManager It is used to pause or resume all or one Reader or StreamReader.

  8. FileState It is used to determine when a file is no longer in use.

Using JsonPath "Hack" in filter

ExpressionPath
$.^\/$
0:90>= <=9\/$
:9+0>= <=9+\/$
..\/.*\/.*\/$
...\/.*\/.*\/.*\/$
$.*/.* & /
.*.*
*.*
{name}any value with key "name"

Operators

SymbolMeaning
&AND
|OR
!NOT

Operations can be used with path or jsonpath combinations. example:

  1. "{name} & /items/"

    all direct descendants of items with at least one key name

  2. "({name} | !/) & $.items.*"

    all objects with at least one key name or not a direct descendant of root and that is a descendant of items

The order does not matter(Operations was improved using https://github.com/riichard/boolean-parser-js)

Future

After doing some benchmarking I found out these function are very expensive, so I added futures to the mix, now these futures are optional and must not be null.

What order are futures executed?

  1. trailing
  2. filter

In most cases not all parts of an object is present in the first value instance, in this case a transformer will be handy for StreamReader while other methods can be used for Reader.

Benchmarks

  1. 02.json takes 2 milliseconds to execute
  2. citylots.json takes 8 minutes to execute this is mostly because of how complex the dataset is.
  3. www.carqueryapi.com takes about 2.3 milliseconds, please note that the file had to be modified I remove ?( from the beginning and ); at the end.

Execution time depends exclusively on the complexity of the file and the number of items to be processed. ie. a file 3 times larger than citylots.json can take approximately five minutes to execute.Using combine:2 can speed up processing considerably.

This project was tested using

  1. https://github.com/zemirco/sf-city-lots-json/blob/master/citylots.json
  2. https://github.com/thaiwsa/aws-speed/blob/master/JsonProcess/jsondata/02.json
  3. http://www.carqueryapi.com/api/0.3/?callback=?&cmd=getMakes&year=1970&sold_in_us=1&utm_medium=referral&utm_campaign=ZEEF&utm_source=https%3A%2F%2Fjson-datasets.zeef.com%2Fjdorfman
  4. https://catalogue.data.gov.bc.ca/dataset/children-and-family-development-cases-in-care-demographics

Libraries

abstractreader
abstractstatereader
booleanparser
complexexpression
filestate
filestateprovider
futureread
globalreadermanager
globalreaderstatemanager
jsonexpression
jsonstreamreader
jsontoken
jsontokenizer
jsontokenstate
normalized
reader
readerstate
remotemethod
remotescheme
remotesource
simpletoken
streamer
streamerunit
streamitem
streamreader
streamreaderstate