Augnito Flutter SDK

Overview

Use the Augnito Flutter SDK to enable Text To Speech and voice commands into a Flutter application.

Supported Platforms

• Android
• iOS

Requirements

Microphone access/recording permission is required in order to work.

Android

Add the following line to AndroidManifest.xml

<uses-permission android:name="android.permission.RECORD_AUDIO"/>

iOS

Add the following to Info.plist:

<key>NSMicrophoneUsageDescription</key>
<string>Microphone access required</string>

Getting Started

The majority of the functionality goes through the DictationManager class via the use of callbacks.

To instantiate a DictationManager object an AugnitoConfig must be provided with valid authorization and keys provided.

late DictationManager _dictationManager;
final AugnitoConfig _config = AugnitoConfig(
    "<your server>",
    "<your accountcode>",
    "<your accesskey>",
    "<your lmid>",
    "<your usertag>",
    sourceApp: "<your sourceapp>");

And on an initialization method

_dictationManager = DictationManager(_config,
    onConnected: _onConnected,
    onDisconnected: _onDisconnected,
    onError: _onError,
    onPartialResult: _onPartialResult,
    onFinalResult: _onFinalResult,
    onCommandResult: _onCommand,
    enableLogs: true);

Alternatively it can be initialized with a provided connection URL:

_dictationManager = DictationManager.fromCustomSpeechURL(
    "custom speech URL",
    onConnected: _onConnected,
    onDisconnected: _onDisconnected,
    onError: _onError,
    onPartialResult: _onPartialResult,
    onFinalResult: _onFinalResult,
    onCommandResult: _onCommand,
    enableLogs: true);

---

Dictation Manager

The DictationManager offers methods to initialize, receive, and stop communication with Augnito's server to provide Speech To Text and Commands support into a Flutter app.

The way it works is via methods and callbacks.

Logging

Logging can be enabled to have more insights on what may be causing errors via the enableLogs parameter at the moment of instantiation (constructor).

Dictation Manager Methods

method	notes
toggleDictation	Turns on or off the audio processing and server communication.
dispose	disposes and releases resources used by the SDK

Disposing

The dispose method should be called when there is no more need for the DictationManager:

example on a widget:

@override
void dispose() {
  _dictationManager.dispose();
  super.dispose();
}

Dictation Manager Callbacks

Callbacks are used to receive the output of the SDK as well as errors that may occur. The callbacks are provided via the DictationManager constructor as named optional parameters.

callback	notes
onConnected	Invoked when the connection to the server has been established. This does not guarantees the audio stream has started.
onDisconnected	Invoked when the server closes connection. It can be triggered on purpose by the DictationManager under certain conditions such as being unable to start audio streaming.
onError	Invoked when an error occurs. Not all errors invoke a connection termination.
onPartialResult	Invoked when an hypothesis or non processed final text has been obtained.
onFinalResult	Invoked when a final text output (not command) has been processed.
onCommandResult	Invoked when a command has been processed.

Example usage

Future<void> initPlatformState() async {
  if (!mounted) return;
  _dictationManager = DictationManager(_config,
      onConnected: _onConnected,
      onDisconnected: _onDisconnected,
      onError: _onError,
      onPartialResult: _onPartialResult,
      onFinalResult: _onFinalResult,
      onCommandResult: _onCommand,
      enableLogs: true);
}

void _onPartialResult(String hypothesis) {
  print("partial result: $hypothesis");
}

void _onFinalResult(String transcription) {
  print("final result: $transcription");
}

void _onCommand(ActionRecipe actionRecipe) {
  print("command: ${actionRecipe.name}");
  print(actionRecipe.toString());
}

void _onConnected() {
  setState(() {
    _status = "online";
  });
}

void _onDisconnected() {
  setState(() {
    _status = "offline";
  });
}

void _onError(DictationError error) {
  print("error ${error.errorMessage}");
}

Dictation Manager Error Types

Dictation Manager will report back on errors via the onError callback. Besides the method itself an object with further details is provided.

The errorType property is the best way to determine what could be happening underneath. Possible types:

enum DictationErrorType {
  noNetworkConnection,
  lowBandwidth,
  serviceDown,
  noDictationStopMic,
  invalidAuthCredentials,
  socketDisconnect,
  socketConnectionError,
  audioRecorder,
  audioRecorderCouldNotInitializePermission,
  audioRecorderCouldNotInitializeGeneric,
  unknown
}

Even further information will be provided on the errorMessage property when possible.

A note on noDictationStopMic: this will not terminate the connection but it's and indicator the microphone is open but idle. The consumer app may want to warn the user or terminate the session if this keeps happening.

---

Commands

Commands are represented by the ActionRecipe object. It includes the information required to be processed by the consumer app.

Commands are returned by the SDK via the onCommandResult callback (DictationManager).

ActionRecipe

As stated the result of a command (either Static or Dynamic) will be an instance of an ActionRecipe.

When a command is returned by the SDK the consumer needs to analyze it and determine what to do based on the properties of the ActionRecipe instance.

For example, a Static Command resulting ActionRecipe will always have it's isStaticCommand property set as true.

Other fields such as searchText, chooseNumber or selectFor are used to determine what to do with the command and in what context. For instance a Replace X with Y dynamic command will produce an ActionRecipe like:

// replace oranges with apples
{
  name: 'replace', // AugnitoCommands.replace entry
  isCommand: true,
  searchText: 'oranges',
  selectFor: 'apples'
  ...
}

ActionRecipe fields

property	notes
name	Command name. Can be matched against one of the entries of AugnitoCommands class.
action
chooseNumber	Unit: on commands that affect items it indicates how many.
searchText	On most select/action commands this represents the item to be searched. For example paragraph, line, word, etc. On some commands this indicates what is being searched for.
selectFor	Usually used to determine an action on most commands. For example delete, underline, etc.
isCommand	Indicates the ActionRecipe is a command.
isStaticCommand	Indicates this ActionRecipe is a static command.
nextPrevious	Direction on how this command should affect the items.
receivedText	Original received text.
receivedTextWithoutSpace	Original text without spaces.

Using Commands

Commands are meant to be used on their on and not in conjunction with regular dictation.

For example:

Dictating Select next 3 words will produce a command

Dictating Patient presents with fever and chills select last 2 words will only produce a transcription

Matching commands

Commands can be matched with entries on the AugnitoCommands dictionary. In the previous example the name equals to AugnitoCommands.replace.

Static Commands

Commands that have not effect on specific units or items are denominated as static commands.

Microphone Control Static Commands
stopMic

Selection Static Commands
selectAll
selectChar
selectWord
selectLine
selectNextLine
selectPreviousLine
selectParagraph

Lists Static Commands
startNumberList
startBulletList
stopBulletList
stopNumberList
stopList

Action Static Commands
undoIt
redoIt

Text Modification Commands
startBoldText
stopBoldText
startBulletText
stopBulletText
startItalicText
stoptItalicText

Modification Commands
deleteIt
pasteIt
copyIt
cutIt
headerIt
underlineIt
boldIt
italicizeIt
capitalizeIt
bulletIt

Navigation Static Commands
moveUp
moveDown
moveLeft
moveRight
goToLineStart
goToLineEnd
giveSpace
backspace
goToDocumentEnd
goToDocumentStart
goToNextPage
goToPreviousPage
goToNextParagraph
goTo

Field Navigation Static Commands
nextField
previousField

Dynamic Commands

Dynamic commands are more complex commands built based on the speech input and may contain extra information (always within the ActionRecipe itself).

Line and Paragraph number

Commands that affect a line or paragraph number via an action command such as select, bold, delete, etc.

^(select|choose|copy text|copytext|cut text|cuttext|correct|bold|underline|delete|header|capitalize|unbold|debold|dbold|uncapitalize|remove|capitalise|dcapitalise|decapitalicize|decapitalize|uncapitalize|Uncap|d capitalise|d capitalize|d underline|dunderline|deunderline|ununderline|goto|moveto|move|italicize|italicise|unitalicise|unitalicize)(the)?(line|para|paragraph|\n\n|\n)(number[ed]?)(.*?)(\.)?$

Try online

A correctly parsed Line and Paragraph number will have a structure similar to:

{
  name: AugnitoCommands.goToLineNumber // or AugnitoCommands.goToParaNumber 
  isCommand: true,
  chooseNumber: 10,
  searchText: "line",
  selectFor: "goToStart"
  ...
}

Example voice command

• choose line number 10
• delete paragraph number 5
• bold the para number 1

---

Go To New Line or New Paragraph

Simple Go To new line or new paragraph command.

{
  name: AugnitoCommands.selectLine // or AugnitoCommands.selectParagraph 
  isCommand: true,
  searchText: "line",
  selectFor: "gotoend"
  ...
}

Example voice command

• go to new line
• go to new paragraph

---

Active X Dynamic Command

Action against the active line, paragraph, space, etc.

^(select|choose|copy text|copytext|cut text|cuttext|correct|bold|underline|delete|header|capitalize|unbold|debold|dbold|uncapitalize|remove|capitalise|dcapitalise|decapitalicize|decapitalize|uncapitalize|Uncap|d capitalise|d capitalize|d underline|dunderline|deunderline|ununderline|goto|moveto|move|italicize|italicise|unitalicise|unitalicize)(the)?(line|para|paragraph|\n\n|\n)(number[ed]?)(.*?)(\.)?$

Try online

Example voice command

• select active paragraph
• delete the current line

{
  name: AugnitoCommands.selectActiveChar
  isCommand: true,
  ...
}

{
  name: AugnitoCommands.selectActiveWord
  isCommand: true,
  ...
}

{
  name: AugnitoCommands.selectActiveLine
  isCommand: true,
  ...
}

{
  name: AugnitoCommands.selectActiveParagraph
  isCommand: true,
  ...
}

---

Navigation Without Go To Dynamic Command

Processes a direction plus a combination of item and distance.

^(last|previous|next|down)(.*?)(word[s]?|line[s]?|sentence[s]?|paragraph[s]?|para[s]?|char[s]?|character[s]?|space|\n\n|\n)$

Try online

Example voice commands

• next 3 lines
• last 5 paras

---

Go To X Dynamic Command

Processor to identify movement with direction and unit.

^(go|goto|gotothe|move|moveto|movethe)(last|previous|next|down)(.*?)(word[s]?|line[s]?|sentence[s]?|paragraph[s]?|para[s]?|char[s]?|character[s]?|space|\n\n|\n)$

Try online

Example voice commands

• go last 4 word
• go to last space
• move the previous 4 words

// go last 4 word
{
  name: AugnitoCommands.selectWord
  isCommand: true,
  chooseNumber: 4,
  nextPrevious: 'last',
  selectFor: 'gotostart'
  ...
}

// go to last space
{
  name: AugnitoCommands.selectChar
  isCommand: true,
  chooseNumber: 0,
  nextPrevious: 'last',
  selectFor: 'gotostart'
  ...
}

// move the previous 4 words
{
  name: AugnitoCommands.selectWord
  isCommand: true,
  chooseNumber: 4,
  nextPrevious: 'previous',
  selectFor: 'gotostart'
  ...
}

Select / Action on Item with Unit and Direction Dynamic Command

Identifies an action on an item that includes direction and unit amount.

^(select|choose|copy text|copytext|cut text|cuttext|correct|bold|underline|delete|header|capitalize|unbold|debold|dbold|uncapitalize|remove|capitalise|dcapitalise|decapitalicize|decapitalize|uncapitalize|Uncap|d capitalise|d capitalize|d underline|dunderline|deunderline|ununderline|goto|moveto|move|italicize|italicise|unitalicise|unitalicize)(the)?(last|previous|next)(.*?)(word[s]?|line[s]?|sentence[s]?|paragraph[s]?|para[s]?|char[s]?|character[s]?|space|\n\n|\n\ns|\n|\ns)$

Try online

Example voice commands

• choose the next 10 characters
• underline previous new line
• capitalize the next 5 words
• delete previous 5 paragraphs

// choose the next 10 characters
{
  name: AugnitoCommands.selectChar
  isCommand: true,
  chooseNumber: 10,
  nextPrevious: 'next',
  searchText: 'characters',
  selectFor: ''
  ...
}

// underline previous new line
{
  name: AugnitoCommands.selectLine
  isCommand: true,
  chooseNumber: 1,
  nextPrevious: 'previous',
  searchText: '\n',
  selectFor: 'underline'
  ...
}

// capitalize the next 5 words
{
  name: AugnitoCommands.selectWord
  isCommand: true,
  chooseNumber: 5,
  nextPrevious: 'next',
  searchText: 'words',
  selectFor: 'capitalize'
  ...
}

// delete previous 5 paragraphs
{
  name: AugnitoCommands.selectParagraph
  isCommand: true,
  chooseNumber: 5,
  nextPrevious: 'previous',
  searchText: '\n\n',
  selectFor: 'delete'
  ...
}

Replace X with Y Dynamic Command

Identifies the replace operation between two elements.

replace[d]?([A-Z a-z 0-9]+)with([A-Z a-z 0-9]+)

Try online

Example voice commands

• replace oranges with apples
• replace blue with red
• replace cars with buses

// replace oranges with apples
{
  name: AugnitoCommands.replace
  isCommand: true,
  searchText: 'oranges',
  selectFor: 'apples'
  ...
}

Select/Action Group Dynamic Command

Identifies a Select/Action command.

^(select|choose|copy text|copytext|cut text|cuttext|correct|bold|underline|delete|header|capitalize|unbold|debold|dbold|uncapitalize|remove|capitalise|dcapitalise|decapitalicize|decapitalize|uncapitalize|Uncap|d capitalise|d capitalize|d underline|dunderline|deunderline|ununderline|goto|moveto|move|italicize|italicise|unitalicise|unitalicize)(\sthe)?\s?(.*?)$

Try online

Example voice commands

• delete mistaken
• bold firstname
• capitalize last name

// delete mistaken
{
  name: AugnitoCommands.select
  isCommand: true,
  searchText: 'mistaken',
  selectFor: 'delete'
  ...
}

---

Utils and Helpers

TextFieldProcessor

The SDK includes a processor to handle basic operations on the TextField widget (via the TextEditingController).

The methods included for the processor are:

• addContent
• selectLastLines
• selectLatWords
• deleteSelection

The usage of these methods is shown on the example app:

 void _processSelectLine(ActionRecipe actionRecipe) {
    TextFieldProcessor.selectLastLines(focusedEditor!.textEditingController,
        offset: actionRecipe.chooseNumber);

    if (actionRecipe.selectFor == Commands.Delete) {
      TextFieldProcessor.deleteSelection(focusedEditor!.textEditingController);
    }
  }

  void _processSelectWord(ActionRecipe actionRecipe) {
    TextFieldProcessor.selectLastWords(focusedEditor!.textEditingController,
        offset: actionRecipe.chooseNumber);
    if (actionRecipe.selectFor == Commands.Delete) {
      TextFieldProcessor.deleteSelection(focusedEditor!.textEditingController);
    }
  }

iOS and background UI modes

In order to keep the app working on the background the info.plist requires the following entries:

<key>UIBackgroundModes</key>
<array>
  <string>audio</string>
  <string>fetch</string>
</array>