jsonstreamreader 1.1.9

This project processes local and remote json files/sources by splitting it into smaller chunks, with automatic garbage collecting.

Json Stream Reader

A Flutter Json Stream Reader

This project processes a local json file by splitting it into smaller chunks, with automatic garbage collecting.

Classes

Streamer

This class defines the size of each chunk,optionally values include

1. chunksize

   default is 100

2. start

   default is 0 and defines where Reader should start parsing file

3. end

   default is the end of the file and defines where Reader should stop Reading file

Readers

All readers have optional values which include

1. int delay

how long the reader should wait before reading next chunk in microseconds

2. int combine

   If combine is 1, array only values will be combined ie. [1,2,3] will result in {"0":1,"1":2,"2":3}, object values in an array are not affected ie. [{},{}] which mean [1,2,{"third":3}] will result in {"0":1,"1":2,"2":3, {"third":3} }

   If combine is 2, array values will be collasped into an object closest to the root eg. {'first':{'second':{ "third":3 } } } will be collaped to {'first':{'/second/third/':3 } }

3. Reader

4. StreamReader Provides access to json data by using StreamController

5. StreamItem

6. ReaderState

7. StreamReaderState

8. GlobalReaderManager It is used to pause or resume all or one Reader or StreamReader.

9. FileState It is used to determine when a file is no longer in use.

Using JsonPath "Hack" in filter

Expression	Path
$.	^\/$
[0:9]	[0>= <=9]\/$
[:9]+	[0>= <=9]+\/$
..	\/.*\/.*\/$
...	\/.*\/.*\/.*\/$
$.*	/.* & /
.*	.*
[*]	.*
{name}	any value with key "name"

Operators

Symbol	Meaning
&	AND
|	OR
!	NOT

Operations can be used with path or jsonpath combinations. example:

1. "{name} & /items/"

   all direct descendants of items with at least one key name

2. "({name} | !/) & $.items.*"

   all objects with at least one key name or not a direct descendant of root and that is a descendant of items

The order does not matter(Operations was improved using https://github.com/riichard/boolean-parser-js)

Future

After doing some benchmarking I found out these function are very expensive, so I added futures to the mix, now these futures are optional and must not be null.

What order are futures executed?

1. trailing
2. filter

In most cases not all parts of an object is present in the first value instance, in this case a transformer will be handy for StreamReader while other methods can be used for Reader.

Benchmarks

1. 02.json takes 2 milliseconds to execute
2. citylots.json takes 8 minutes to execute this is mostly because of how complex the dataset is.
3. www.carqueryapi.com takes about 2.3 milliseconds, please note that the file had to be modified I remove ?( from the beginning and ); at the end.

Execution time depends exclusively on the complexity of the file and the number of items to be processed. ie. a file 3 times larger than citylots.json can take approximately five minutes to execute.Using combine:2 can speed up processing considerably.

This project was tested using

1. https://github.com/zemirco/sf-city-lots-json/blob/master/citylots.json
2. https://github.com/thaiwsa/aws-speed/blob/master/JsonProcess/jsondata/02.json
3. http://www.carqueryapi.com/api/0.3/?callback=?&cmd=getMakes&year=1970&sold_in_us=1&utm_medium=referral&utm_campaign=ZEEF&utm_source=https%3A%2F%2Fjson-datasets.zeef.com%2Fjdorfman
4. https://catalogue.data.gov.bc.ca/dataset/children-and-family-development-cases-in-care-demographics