delta_trace_db 0.0.7

A secure, in-memory database built on Dart with class-based functionality and detailed operation history tracking.

delta_trace_db

(en)Japanese ver is here.
(ja) この解説の日本語版はここ にあります。

Overview

"DeltaTraceDB" is an in-memory database (NoSQL) that stores data in units of lists of classes. This database allows you to register class structures as they are in the database,
and allows full-text searches of the elements of the registered classes.
In addition, queries are also classes, and can have DB operation information consisting of
who, when, what, why, and from.
If serialized and saved, it provides a very rich source of information for security audits and usage analysis.
This is particularly useful in projects with various constraints, such as medical use.
In addition, for when, the TemporalTrace class has a complete tracing function for
the communication path and each arrival time.
I think this would be useful, for example, in space-scale communication networks and relay servers, where non-negligible delays occur even at the speed of light.

DB structure

The structure of this DB is as follows.
In other words, each collection corresponds to a list of each class.
For this reason, the user is barely aware of the difference between the front end and the back end,
and can concentrate on the operation of "obtaining the required class object."

📦 Database (DeltaTraceDB)
├── 🗂️ CollectionA (key: "collection_a")
│   ├── 📄 Item (ClassA)
│   │   ├── id: int
│   │   ├── name: String
│   │   └── timestamp: String
│   └── ...
├── 🗂️ CollectionB (key: "collection_b")
│   ├── 📄 Item (ClassB)
│   │   ├── uid: String
│   │   └── data: Map<String, dynamic>
└── ...

Basic DB operations

All operations in this DB are class-based.
There is no need to learn a new query language.

📦 1. Define the model class

First, prepare the model class.
It is convenient to have the model class inherit ClonableFile from the file_state_manager package.
If you do not want to inherit ClonableFile, you can also handle Map<String,dynamic> directly with RawQueryBuilder.
Here is an example of how to inherit ClonableFile.

class User extends CloneableFile {
  final String id;
  final String name;
  final int age;
  final DateTime createdAt;
  final DateTime updatedAt;
  final Map<String, dynamic> nestedObj;

  User({
    required this.id,
    required this.name,
    required this.age,
    required this.createdAt,
    required this.updatedAt,
    required this.nestedObj,
  });

  static User fromDict(Map<String, dynamic> src) =>
      User(
        id: src['id'],
        name: src['name'],
        age: src['age'],
        createdAt: DateTime.parse(src['createdAt']),
        updatedAt: DateTime.parse(src['updatedAt']),
        nestedObj: src['nestedObj'],
      );

  @override
  Map<String, dynamic> toDict() =>
      {
        'id': id,
        'name': name,
        'age': age,
        'createdAt': createdAt.toIso8601String(),
        'updatedAt': DateTime.now().toIso8601String(),
        'nestedObj': {...nestedObj},
      };

  @override
  User clone() => User.fromDict(toDict());
}

🏗️ 2. Initialize the DB and add data

final db = DeltaTraceDatabase();
final now = DateTime.now();
final users = [
  User(id: '1',
      name: 'Taro',
      age: 25,
      createdAt: now,
      updatedAt: now,
      nestedObj: {"a": "a"}),
  User(id: '2',
      name: 'Jiro',
      age: 30,
      createdAt: now,
      updatedAt: now,
      nestedObj: {"a": "b"}),
];

final query = QueryBuilder.add(target: 'users', addData: users).build();
// <User> here is not needed on the server.
// This is a type specification for conversion processing when retrieving data.
final result = db.executeQuery<User>(query);

💡 Key Points

• You can add, update, and delete data by simply passing the query generated by QueryBuilder to executeQuery.
• This query can be serialized as a Map<String, dynamic> using the toDict method, so it can be sent to the server as is and reflected in a remote database if necessary.
• If you prepare a process equivalent to executeQuery on the server side, you can process it with the same query structure.
• Most of what is required on the server side is user permission confirmation and logging, and logging is also very easy since you can just save the query as is.
• To return results from the server to the frontend, simply toDict the QueryResult of executeQuery and return it.

🔍 3. Data search (filter + sort + paging)

final searchQuery = QueryBuilder.search(
  target: 'users',
  queryNode: FieldContains("name", "ro"), // Name contains "ro" (so Taro and Jiro are the targets)
  sortObj: SingleSort(field: 'age', reversed: true), // Descending order by age
  limit: 1, // Only get one result
).build();
final searchResult = db.executeQuery<User>(searchQuery);
// Jiro is obtained.
final matchedUsers = searchResult.convert(User.fromDict);
// Query to paging to the next page.
final pagingQuery = QueryBuilder.search(
    target: 'users',
    queryNode: FieldContains("name", "ro"),
    // When paging, the parameters must be the same as before.
    sortObj: SingleSort(field: 'age', reversed: true),
    limit: 1,
    // Get only one result
    startAfter: searchResult.result
        .last // Specify paging. Note that you can also specify an offset instead of an object.
).build();
final nextPageSearchResult = db.executeQuery<User>(pagingQuery);
// Get Taro.
final nextPageUsers = nextPageSearchResult.convert(User.fromDict);

✏️ 4. Data update (multiple conditions/partial update)

final updateQuery = QueryBuilder.update(
  target: 'users',
  queryNode: OrNode([
    FieldEquals('name', 'Taro'),
    FieldEquals('name', 'Jiro'),
  ]),
  overrideData: {'age': 99},
  returnData: true,
).build();

final updateResult = db.executeQuery<User>(updateQuery);
final updated = updateResult.convert(User.fromDict);

❌ 5. Data Deletion

final deleteQuery = QueryBuilder.delete(
  target: 'users',
  queryNode: FieldEquals("name", "Jiro"),
).build();

final deleteResult = db.executeQuery<User>(deleteQuery);

💾 6. Saving and restoring (serialization supported)

// Save (obtained in Map format). After this, you can encrypt it as you like or save it using your favorite package.
final saved = db.toDict();
// Restore
final loaded = DeltaTraceDatabase.fromDict(saved);

🕰️ Full restoration (Query log) is also possible using change logs.
In delta_trace_db, all change operations (add, update, delete, etc.) are expressed in the Query class.
So, by saving this Query as a log in chronological order, you can completely reproduce the DB state at any point in time.

💡 Why can it be restored?

• All data operations are recorded by Query.
• The same state can be obtained by re-executing the saved query logs in order on an empty DB in the initial state.

I recommend a configuration where normal saving is done as a date and time snapshot, and the query itself is logged.
This way, you can restore to just before the problem occurred just by applying the query log from the snapshot saving point onwards,
and since all other operations are also left in the log, it becomes easy to rebuild the DB.
The query can have DB operation information consisting of "who, when, what, why, from" as a Cause class,
so if you set this appropriately, it will be even easier to identify the problem.

🧠 7. Searching in nested fields and searching using other nodes

For nested fields, you can specify the key separated by a ".", such as "nestedObj.a".
The nodes that can be used in searches are "LogicalNode (And and Or)" and "ComparisonNode (Equals, etc.)". Please check the following for available types.
logicalNode
comparisonNode

🧬 8. Type conversion/template conversion (conformToTemplate)

The conformToTemplate function is useful when you want to convert saved data into a new structure.
This function can complement and convert saved data according to a template even if the data is missing.

// Original class
class ClassA extends ClonableFile {
  String id;
  String name;

  ClassB(this.id, this.name)
// Omitted
}

final db = DeltaTraceDatabase();
final users = [
  ClassA(id: 'u003', name: 'Hanako')
];
final query = QueryBuilder.add(target: 'users', addData: users).build();
final result = db.executeQuery<User>(query);
// New class to be changed from ClassA
class ClassB extends ClonableFile {
  String id;
  String name;
  int age;

  ClassB(this.id, this.name, this.age)
// Omitted
}

final Query conformQuery = QueryBuilder.conformToTemplate(
  target: 'users',
  template: ClassB(
      id: '', name: '', age: -1), // Undefined age is filled with the initial value of -1
).build();
final QueryResult<ClassB> _ = db.executeQuery<ClassB>(conformQuery);

// => { 'id': 'u003', 'name': 'Hanako', 'age': -1 }
final conformedUser = ClassB.fromDict(db
    .collection("users")
    .raw[0]);

Speed

This package is an in-memory database, so it is generally fast.
There is usually no problem with around 100,000 records.
I recommend that you test it in an actual environment using speed_test.dart in the test folder.
However, since it consumes RAM capacity according to the amount of data,
if you need an extremely large database, consider using a general database.
For reference, below are the results of a speed test (test/speed_test.dart) run on a slightly
older PC equipped with a Ryzen 3600 CPU.
The test conditions were chosen to take a sufficiently long time, but I think it will rarely cause
any problems in practical use.

speed test for 100000 records                                                                                                                                                                              
start add
end add: 167 ms
start getAll (with object convert)
end getAll: 276 ms
returnsLength:100000
start save (with json string convert)
end save: 323 ms
start load (with json string convert)
end load: 246 ms
start search (with object convert)
end search: 474 ms
returnsLength:100000
start search paging, half limit pre search (with object convert)
end search paging: 371 ms
returnsLength:50000
start search paging by obj (with object convert)
end search paging by obj: 378 ms
returnsLength:50000
start search paging by offset (with object convert)
end search paging by offset: 351 ms
returnsLength:50000
start update at half index and last index object
end update: 57 ms
start updateOne of half index object
end updateOne: 9 ms
start conformToTemplate
end conformToTemplate: 72 ms
start delete half object (with object convert)
end delete: 219 ms
returnsLength:50000

Future plans

It is possible to speed up the database, but this is a low priority, so I think that improving usability and creating peripheral tools will take priority.

Support

There is essentially no support at this time, but bugs will likely be fixed.
If you find any issues, please open an issue on Github.

About version control

The C part will be changed at the time of version upgrade.
However, versions less than 1.0.0 may change the file structure regardless of the following rules.

• Changes such as adding variables, structure change that cause problems when reading previous files.
  • C.X.X
• Adding methods, etc.
  • X.C.X
• Minor changes and bug fixes.
  • X.X.C

License

This software is released under the Apache-2.0 License, see LICENSE file.

Copyright 2025 Masahide Mori

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright notice

The “Dart” name and “Flutter” name are trademarks of Google LLC.
*The developer of this package is not Google LLC.