jsonstreamreader 0.0.6

This project is processes a local json file by splitting it into smaller chunks, with automatic garbage collecting.Please use latest version.

Json Stream Reader Beta

A Flutter Json Stream Reader

Getting Started

This project is processes a local json file by splitting it into smaller chunks, with automatic garbage collecting.Please use latest version.

1. Create file and streamer instance

File file = new File(path.path + '/citylots.json');

Streamer s = new Streamer(file);

/* or Streamer streamer = new Streamer(file,chunksize=200); */

default chunksize is 100(100kb).

2. Create Reader instance

Reader reader = new Reader(streamer);

//or Reader r = new Reader(s, delay: 100); delay is in microseconds

3. Register a function to recieve data, your options are
   • filter(String expression,void (dynamic value, String key)) - value is either an JsonObject or a String
   • trailing(void (dynamic value, String key))
   • pipe(void (dynamic value, String key)) - With pipe only array values(example in [3] array value is 3) or completed - objects(example {"3":4,"5":6} will be processed but {"3":4, will never be, in which case you should use trailing.
4. Example

getApplicationDocumentsDirectory().then((Directory path) {

File file = new File(path.path + '/citylots.json');

Streamer streamer = new Streamer(file);

Reader r = new Reader(streamer);

r.filter("\$.*", (dynamic value, String key) {

//items are received here

print(value);

})

.done(() {//called when all items have been processed

print("Completed");

}).fail((err) {//called when and if processing fails

print(err); });

}).catchError((error) {

print(error);

});

How are keys Represented?

Root is represented as /

1. Example 1

{
    "glossary": {
        "title": "example glossary",
		"GlossDiv": {
            "title": "S",
			"GlossList": {
                "GlossEntry": {
                    "ID": "SGML",
					"SortAs": "SGML",
					"GlossTerm": "Standard Generalized Markup Language",
					"Acronym": "SGML",
					"Abbrev": "ISO 8879:1986",
					"GlossDef": {
                        "para": "A meta-markup language, used to create markup languages such as DocBook.",
						"GlossSeeAlso": ["GML", "XML"]
                    },
					"GlossSee": "markup"
                }
            }
        }
    }
}

• object['glossary'] has a key of /
• object['glossary']['title'] has a key of /glossary/
• object['glossary']['GlossDiv'] has a key of /glossary/
• object['glossary']['GlossDiv']['title'] has a key of /glossary/title/
• object['glossary']['GlossDiv']['GlossList']['GlossEntry'] has a key of /glossary/GlossDiv/GlossList

So from Root(/) key is Root + key/

2. Example 2

[ {"age":12}, 2, 3, 4 ]

• object[0]['age'] has a key of /0/age/
• object[1] has a key of /1/

Using Expression in filter

reader.filter(Expression, (dynamic value, String key) {

//items are received here

print(value);

})

1. If you start your expression with "/" normal regular expressions rules apply so for object[0]['age'] I can use "\/0\/age\/". You have to escape "/".

   reader.filter("\/0\/age\/", (dynamic value, String key) {

   //items are received here

   print(value);

   })

2. For all children of object['glossary']['GlossDiv'] I can use "\/glossary\/GlossDiv\/.*".

   reader.filter("\/glossary\/GlossDiv\/.*", (dynamic value, String key) {

   //items are received here

   print(value);

   })

Using JsonPath "Hack" in filter

Expression	Path
$.	^\/$
[0:9]	[0-9]\/$
[0:9]+	[0-9]+\/$
..	\/.*\/.*\/$
...	\/.*\/.*\/.*\/$
.*	.*
[*]	.*
{name}	any object with key "name"

Operators

Symbol	Meaning
&	AND

Operations can be used with path and jsonpath combinations example

1. "{name} & /items/"
2. "{name} & $.items"

The order does not matter

1. so for object[0]['age'] I can use "$.0.age" which is equivalent to "^\/0\/age\/$".

   reader.filter("$.0.age", (dynamic value, String key) {

   //items are received here print(value);

   })

2. For all children of object['glossary']['GlossDiv'] I can use $.glossary.GlossDiv.* which is equivalent ot "\/glossary\/GlossDiv\/.*".

   reader.filter("$.glossary.GlossDiv.*", (dynamic value, String key) {

   //items are received here

   print(value);

   })

Future

After doing some benchmarking I found out these function are very expensive, so I added futures to the mix, now these futures are optional and must not be null.

How To use Futures?

reader.filter("$.glossary.GlossDiv.*", (dynamic value, String key) {

//items are received here

Completer<String> future = new Completer<String>();

return future.future;

})

Futures are always processed before the next function is called which makes for some very interesting. How so well I can do this.

1. Say I have a function that returns a future after adding a list of items to a database, I would only add every ten items.

addToDatabase(List<Map<String,String>> items){

//implement here

//dont forget to empty list

items.clear();

}

With that out of the way now I will need to generate the list, I will maintain a list of items.

List<Map<String,String>> items = List<Map<String,String>>();

reader.filter("$.glossary.GlossDiv.*", (dynamic value, String key) { if(value.contains('lastkey')){

if(items.keys.last.constains('lastkey')){

  //new item
  
  items.add(value);

}

else{

  //new item
  
  items.last.addAll(value);

}

}

else{

if(items.keys.last.constains('lastkey')){

  //new item
  
  items.add(value);

}

else{

  //new item
  
  items.last.addAll(value);

}

} /// now we add the items to database if there are 100 or more of them

if(items.length>99){

//add to database return future so that this is processed before we starting adding

//more items

return addToDatabase(items);

}

})

.done((){

if(items.length>0){

addToDatabase(items);

}

});

The reader will handle the rest. Please note version 1 will have alot of breaking features

1. I will be removing pipe
2. Trailing will be modified

Benchmarks

1. 02.json takes 2 milliseconds to execute
2. citylots.json takes 8 minutes to execute this is mostly because of how complex the dataset is.
3. www.carqueryapi.com takes about 2.3 milliseconds, please not that the file had to be modified I remove ?( from the beginning and ); at the end.

This project was tested using

1. https://github.com/zemirco/sf-city-lots-json/blob/master/citylots.json
2. https://github.com/thaiwsa/aws-speed/blob/master/JsonProcess/jsondata/02.json
3. http://www.carqueryapi.com/api/0.3/?callback=?&cmd=getMakes&year=1970&sold_in_us=1&utm_medium=referral&utm_campaign=ZEEF&utm_source=https%3A%2F%2Fjson-datasets.zeef.com%2Fjdorfman
4. https://catalogue.data.gov.bc.ca/dataset/children-and-family-development-cases-in-care-demographics