topspeed 0.2.0
topspeed: ^0.2.0 copied to clipboard
Pure Dart reader and writer for Clarion TopSpeed (.tps) database files: tables, keys, MEMO/BLOB, code pages and encrypted files. No native code.
topspeed #
Read and write Clarion TopSpeed (.tps) database files in pure Dart.
It needs no native code, no ODBC driver and no Clarion runtime, and it
runs on the Dart VM, in Flutter (mobile and desktop) and on the web.
import 'dart:io';
import 'package:topspeed/topspeed.dart';
void main() {
final file = TpsFile.parse(File('CUSTOMER.TPS').readAsBytesSync());
final customers = file.table();
for (final row in customers.rows) {
print('${row.recordNumber}: ${row['NAME']} ${row['BALANCE']}');
}
}
Features #
| Area | Support |
|---|---|
| File structure | header, B+tree pages, RLE page compression, several tables per file |
| Field types | BYTE SHORT USHORT LONG ULONG SREAL REAL DECIMAL DATE TIME STRING CSTRING PSTRING PICTURE GROUP, DIM arrays |
| Unknown types | returned as raw bytes; the rest of the file still loads |
| MEMO / BLOB | read (row.memo, row.blob); kept on update, removed on delete |
| Text | windows1252 (default), cp850, cp437, or any Encoding |
| Encryption | files with an OWNER password: read, and written back encrypted |
| Keys | DUP, OPT, NOCASE, descending parts; rebuilt on every change |
| Writing | insert, update, delete, then toBytes() |
| Damaged files | recovery mode scans leaf pages when the page tree is broken (read-only) |
| Large files | TpsReader reads a few pages at a time and skips index pages (read-only) |
Installing #
dart pub add topspeed
# or, in a Flutter app
flutter pub add topspeed
Reading #
final file = TpsFile.parse(bytes);
// Most files hold one table; files with several need a name.
final table = file.tables.length == 1 ? file.table() : file.table('ORDERS');
// The layout.
for (final f in table.definition.valueFields) {
print('${f.label}: ${f.type.name}, ${f.length} bytes');
}
// One row by record number, or all of them.
final row = table.row(42);
print(row?['NAME']); // a field by label ('NAME') or full name ('CUS:NAME')
print(row?.values); // every field: {NAME: ..., BALANCE: ...}
Values come back as these Dart types:
| Clarion | Dart |
|---|---|
BYTE, SHORT, USHORT, LONG, ULONG |
int |
SREAL, REAL |
double |
DECIMAL |
String, such as "-12.50" (no precision loss) |
STRING, PICTURE |
String, trailing spaces removed |
CSTRING, PSTRING |
String |
DATE |
DateTime? (null when blank) |
TIME |
Duration? (null when blank) |
DIM(n) |
List of the element type |
| unknown type | Uint8List (raw bytes) |
Dates stored in LONG fields #
Many Clarion programs keep dates and times in LONG fields as "standard"
dates (days since 28 December 1800) and times (hundredths of a second
since midnight, plus one). Convert them like this:
final shipped = clarionDateToDateTime(row['SHIPDATE'] as int); // DateTime?
final at = clarionTimeToDuration(row['SHIPTIME'] as int); // Duration?
table.update(row.recordNumber, {'SHIPDATE': dateTimeToClarionDate(DateTime.now())});
MEMO and BLOB #
final notes = row.memo('NOTES'); // String?
final photo = row.blob('PHOTO'); // Uint8List?
print(row.memos); // {NOTES: ..., PHOTO: ...}
Code pages #
Clarion stores text in the ANSI code page of the PC that wrote the file. The default is Windows-1252 (Western Europe and the Americas). Older DOS programs often used CP850 or CP437:
final file = TpsFile.parse(bytes, encoding: cp850);
Any Encoding from dart:convert works too (latin1, utf8, ...).
Encrypted files #
final file = TpsFile.parse(bytes, password: 'owner-password');
Without the password, parse throws a TpsException that says the file
may be encrypted. toBytes() encrypts again with the same password.
Large files #
TpsFile.parse decodes the whole file, including every index entry. For
large files, use TpsReader. It reads only the pages a call needs and
keeps a small cache of recent pages:
import 'package:topspeed/topspeed_io.dart';
final reader = TpsReader(TpsFileSource.open(r'C:\data\INV.TPS'));
try {
final inv = reader.table();
print(inv.rowCount); // from the table's metadata: instant
print(inv.row(50740)?['DESCR']); // one lookup: a few page reads
for (final row in inv.rows(fromRecord: 1000)) {
// rows are read as you go
}
} finally {
reader.close();
}
On a 52 MB file with 100,960 rows and 28 keys, a full scan takes under a
second and about 50 MB of memory. TpsFile.parse needs about 580 MB for
the same file.
topspeed_io.dartneedsdart:io. On the web, pass the bytes to aTpsMemorySource, or implementTpsByteSourceyourself.- The reader is read-only. To change rows, use
TpsFile. - Reads are synchronous. In Flutter, run full scans in
Isolate.runso the UI keeps drawing. Singlerow(n)lookups are fast enough on the UI isolate. - A Clarion program may keep writing to the file while you read it. If
reader.changedOnDiskis true, open a new reader. - A damaged page tree makes the reader throw.
TpsFile.parsecan still read such files in recovery mode.
Writing #
final file = TpsFile.parse(await File(path).readAsBytes());
final table = file.table();
// Only write when the library reproduces this file's keys exactly.
if (!table.indexesVerified) throw StateError('keys not verified');
final n = table.insert({'CODE': 10, 'NAME': 'New customer', 'BALANCE': '0.00'});
table.update(n, {'NAME': 'Renamed'});
table.delete(7);
await File(path).writeAsBytes(file.toBytes());
- Changes stay in memory until you call
toBytes(). That call returns the whole file with a freshly built page tree and a higher change counter. - Each change rebuilds the table's keys and checks unique keys. A
duplicate throws
TpsException. indexesVerifiedcompares the keys the library rebuilds with the keys stored in the file. If they differ, the library does not encode that file's keys the way TopSpeed does, and you should not write it.- MEMO and BLOB values cannot be written yet.
updatekeeps them, anddeleteremoves them with the row.
Safety checklist for production data #
- Never write while a Clarion program has the file open. This library does not take TopSpeed's locks.
- Keep a backup. Write the new bytes, read them back, and compare.
- Check
indexesVerifiedbefore writing. - Do not write a file opened in recovery mode (
file.recovered).toBytes()refuses to do so.
Checking your own files #
The repository has a command-line checker. It parses every .tps file in
a folder, decodes all values and memos, verifies the keys, and does a full
rewrite in memory. It never changes the files.
dart run tool/tps_check.dart path/to/folder [maxBytes] [--password=x] [--encoding=cp850]
dart run tool/tps_dump.dart path/to/FILE.TPS [rows] [--password=x]
dart run tool/tps_stream_bench.dart path/to/FILE.TPS [--compare] [--password=x]
tps_stream_bench times TpsReader on one file. With --compare, it
also checks every row against TpsFile.parse.
The result is a tally such as {OK: 120}. Any other status names the file
and the problem: INDEX_MISMATCH, REWRITE_MISMATCH, RECOVERED,
ERROR or SKIP(...). Please report such files (without private data)
as issues.
Limitations #
- MEMO/BLOB writing is not supported.
- MEMOs, BLOBs and keys kept in external files (
externalFileis not empty) are not read. TpsFile.parseloads the whole file into memory.TpsReaderdoes not, but it cannot write.- The format is not publicly documented. This library follows the
behaviour of real files and of tps-parse.
Before writing, always check
indexesVerified.
License #
MIT, see LICENSE. The encryption code is derived from tps-parse (Apache 2.0); see NOTICE.
Releasing (maintainers) #
Every push or merge to main runs the checks in .github/workflows/ci.yml.
If pubspec.yaml holds a version that is not on pub.dev yet, release.yml
creates the tag v<version>, and that tag starts publish.yml, which
publishes to pub.dev.
To release a new version:
- Raise
versioninpubspec.yaml(plainX.Y.Z). - Add a
## X.Y.Zsection toCHANGELOG.md. - Merge or push to
main.
One-time setup:
- Publish the first version by hand:
dart pub publish. - On pub.dev, open the package's Admin tab and enable Automated
publishing from GitHub Actions. Use repository
Isaac80536/topspeedand tag patternv{{version}}, and require the GitHub environmentpub.dev. - On GitHub, create the environment
pub.dev(Settings > Environments). - Create a fine-grained personal access token with Contents: Read and
write on this repository only, and save it as the Actions secret
RELEASE_TOKEN. Tags pushed with the default token do not start other workflows, so the tag must be pushed with this token.