dictosaurus 0.0.1-beta.4

Extensions on String that provide dictionary and thesaurus functions (**PRE-RELEASE**).

Extensions on String that provide dictionary and thesaurus functions.

THIS PACKAGE IS PRE-RELEASE, IN ACTIVE DEVELOPMENT AND SUBJECT TO DAILY BREAKING CHANGES.

Skip to section:

• Overview
• Usage
• API
• Definitions
• References
• Issues

Overview

The dictosaurus package provides language reference utilities used in information retrieval systems. It relies on three key indexes:

• a definitions index (see definitions) that maps the words in a language (vocabulary) to their definitions (meanings);
• a synonyms index that maps the vocabulary to a collection of synonyms, which may be empty; and
• a k-gram index that maps k-grams to the vocabulary.

Three utility classes provide dictionary and thesaurus functions:

• the Dictionary class exposes the Future function, looking up the meaning of the term in a vocabulary;
• the Thesaurus class exposes the Future<Set function, looking up the synonyms of the term in a synonyms index; and
• the AutoCorrect class exposes the Future<Map<String, List function that returns a set of unique alternative spellings for term by converting the term to k-grams and then finding the best matches for the (misspelt) term from the k-gram index, ordered in descending order of relevance (i.e. best match first).

The DictoSaurus composition class leverages a Dictionary, Thesaurus and AutoCorrect which it uses to expose the Future, Future<Set and Future<Map<String, List functions.

The DictoSaurus also exposes the Future<List function that looks up the term in the Dictionary, Thesaurus and AutoCorrect classes to return a term-expansion in descending order of relevance (best match first). If the term is found in the Dictionary it will appear as the first element of the returned list. If it is not found in the Dictionary it will not be in the returned list as it is likely to be misspelt.

The DictoSaurus.english static const instance uses the included vocabulary, synonymsIndex and kGramIndex hashmaps. For other languages or a custom implementation, initialize the DictoSaurus using the DictoSaurus.async factory constructor whichuses asynchronous callbacks to vocabulary, synonymsIndex and kGramIndex APIs. The DictoSaurus.async factory constructor has a named, required parameter TextAnalyzerConfiguration configuration. The optional named parameter int k (the k-gram length) defaults to 3 (tri-gram).

If the DictoSaurus is used as a term expander in an information retrieval system, the DictoSaurus.configuration must use the same tokenizing algorithm as the index.

Refer to the references to learn more about information retrieval systems.

Usage

In the pubspec.yaml of your flutter project, add the following dependency:

dependencies:
  dictosaurus: <latest_version>

In your code file add the following import:

import 'package:dictosaurus/dictosaurus.dart';

TODO: describe usage.

API

The API exposes

We use an interface > implementation mixin > base-class > implementation class pattern:

• the interface is an abstract class that exposes fields and methods but contains no implementation code. The interface may expose a factory constructor that returns an implementation class instance;
• the implementation mixin implements the interface class methods, but not the input fields;
• the base-class is an abstract class with the implementation mixin and exposes a default, unnamed generative const constructor for sub-classes. The intention is that implementation classes extend the base class, overriding the interface input fields with final properties passed in via a const generative constructor; and
• the class naming convention for this pattern is "Interface" > "InterfaceMixin" > "InterfaceBase".

Definitions

The following definitions are used throughout the documentation:

• corpus- the collection of documents for which an index is maintained.
• character filter - filters characters from text in preparation of tokenization.
• dictionary - is a hash of terms (vocabulary) to the frequency of occurence in the corpus documents.
• document - a record in the corpus, that has a unique identifier (docId) in the corpus's primary key and that contains one or more text zones/fields that are indexed.
• index - an inverted index used to look up document references from the corpus against a vocabulary of terms. k-gram - a sequence of (any) k consecutive characters from a term. A k-gram can start with "$", dentoting the start of the term, and end with "$", denoting the end of the term. The 3-grams for "castle" are { $ca, cas, ast, stl, tle, le$ }.
• lemmatizer - lemmatisation (or lemmatization) in linguistics is the process of grouping together the inflected forms of a word so they can be analysed as a single item, identified by the word's lemma, or dictionary form (from Wikipedia (2)).
• postings - a separate index that records which documents the vocabulary occurs in. In this implementation we also record the positions of each term in the text to create a positional inverted index.
• postings list - a record of the positions of a term in a document. A position of a term refers to the index of the term in an array that contains all the terms in the text.
• synonym - a word, morpheme, or phrase that means exactly or nearly the same as another word, morpheme, or phrase in a given language (from Wikipedia (4)).
• stemmer - stemming is the process of reducing inflected (or sometimes derived) words to their word stem, base or root form—generally a written word form (from Wikipedia (3)).
• stopwords - common words in a language that are excluded from indexing.
• term - a word or phrase that is indexed from the corpus. The term may differ from the actual word used in the corpus depending on the tokenizer used.
• term filter - filters unwanted terms from a collection of terms (e.g. stopwords), breaks compound terms into separate terms and / or manipulates terms by invoking a stemmer and / or lemmatizer.
• term frequency (Ft) is the frequency of a term in an index or indexed object.
• term position is the zero-based index of a term in an ordered array of terms tokenized from the corpus.
• text - the indexable content of a document.
• token - representation of a term in a text source returned by a tokenizer. The token may include information about the term position in the source text.
• token filter - returns a subset of tokens from the tokenizer output.
• tokenizer - a function that returns a collection of tokens from the terms in a text source after applying a character filter and term filter.
• vocabulary - the collection of terms indexed from the corpus or the words in a language.

References

• Manning, Raghavan and Schütze, "Introduction to Information Retrieval", Cambridge University Press, 2008
• University of Cambridge, 2016 "Information Retrieval", course notes, Dr Ronan Cummins, 2016
• Wikipedia (1), "Inverted Index", from Wikipedia, the free encyclopedia
• Wikipedia (2), "Lemmatisation", from Wikipedia, the free encyclopedia
• Wikipedia (3), "Stemming", from Wikipedia, the free encyclopedia
• Wikipedia (4), "Synonym", from Wikipedia, the free encyclopedia
• Wikipedia (5), "Jaccard Index", from Wikipedia, the free encyclopedia

Issues

If you find a bug please fill an issue.

This project is a supporting package for a revenue project that has priority call on resources, so please be patient if we don't respond immediately to issues or pull requests.