Augnito Flutter SDK
Overview
Use the Augnito Flutter SDK to enable Text To Speech and voice commands into a Flutter application.
Supported Platforms
- Android
- iOS
Requirements
Microphone access/recording permission is required in order to work.
Android
Add the following line to AndroidManifest.xml
<uses-permission android:name="android.permission.RECORD_AUDIO"/>
iOS
Add the following to Info.plist:
<key>NSMicrophoneUsageDescription</key>
<string>Microphone access required</string>
Getting Started
The majority of the functionality goes through the DictationManager class via the use of callbacks.
To instantiate a DictationManager object an AugnitoConfig must be provided with valid authorization and keys provided.
late DictationManager _dictationManager;
final AugnitoConfig _config = AugnitoConfig(
"<your server>",
"<your accountcode>",
"<your accesskey>",
"<your lmid>",
"<your usertag>",
sourceApp: "<your sourceapp>");
And on an initialization method
_dictationManager = DictationManager(_config,
onConnected: _onConnected,
onDisconnected: _onDisconnected,
onError: _onError,
onPartialResult: _onPartialResult,
onFinalResult: _onFinalResult,
onCommandResult: _onCommand,
enableLogs: true);
Alternatively it can be initialized with a provided connection URL:
_dictationManager = DictationManager.fromCustomSpeechURL(
"custom speech URL",
onConnected: _onConnected,
onDisconnected: _onDisconnected,
onError: _onError,
onPartialResult: _onPartialResult,
onFinalResult: _onFinalResult,
onCommandResult: _onCommand,
enableLogs: true);
Dictation Manager
The DictationManager offers methods to initialize, receive, and stop communication with Augnito's server to provide Speech To Text and Commands support into a Flutter app.
The way it works is via methods and callbacks.
Logging
Logging can be enabled to have more insights on what may be causing errors via the enableLogs parameter at the moment of instantiation (constructor).
Dictation Manager Methods
method | notes |
---|---|
toggleDictation | Turns on or off the audio processing and server communication. |
dispose | disposes and releases resources used by the SDK |
Disposing
The dispose method should be called when there is no more need for the DictationManager:
example on a widget:
@override
void dispose() {
_dictationManager.dispose();
super.dispose();
}
Dictation Manager Callbacks
Callbacks are used to receive the output of the SDK as well as errors that may occur. The callbacks are provided via the DictationManager constructor as named optional parameters.
callback | notes |
---|---|
onConnected | Invoked when the connection to the server has been established. This does not guarantees the audio stream has started. |
onDisconnected | Invoked when the server closes connection. It can be triggered on purpose by the DictationManager under certain conditions such as being unable to start audio streaming. |
onError | Invoked when an error occurs. Not all errors invoke a connection termination. |
onPartialResult | Invoked when an hypothesis or non processed final text has been obtained. |
onFinalResult | Invoked when a final text output (not command) has been processed. |
onCommandResult | Invoked when a command has been processed. |
Example usage
Future<void> initPlatformState() async {
if (!mounted) return;
_dictationManager = DictationManager(_config,
onConnected: _onConnected,
onDisconnected: _onDisconnected,
onError: _onError,
onPartialResult: _onPartialResult,
onFinalResult: _onFinalResult,
onCommandResult: _onCommand,
enableLogs: true);
}
void _onPartialResult(String hypothesis) {
print("partial result: $hypothesis");
}
void _onFinalResult(String transcription) {
print("final result: $transcription");
}
void _onCommand(ActionRecipe actionRecipe) {
print("command: ${actionRecipe.name}");
print(actionRecipe.toString());
}
void _onConnected() {
setState(() {
_status = "online";
});
}
void _onDisconnected() {
setState(() {
_status = "offline";
});
}
void _onError(DictationError error) {
print("error ${error.errorMessage}");
}
Dictation Manager Error Types
Dictation Manager will report back on errors via the onError callback. Besides the method itself an object with further details is provided.
The errorType property is the best way to determine what could be happening underneath. Possible types:
enum DictationErrorType {
noNetworkConnection,
lowBandwidth,
serviceDown,
noDictationStopMic,
invalidAuthCredentials,
socketDisconnect,
socketConnectionError,
audioRecorder,
audioRecorderCouldNotInitializePermission,
audioRecorderCouldNotInitializeGeneric,
unknown
}
Even further information will be provided on the errorMessage property when possible.
A note on noDictationStopMic: this will not terminate the connection but it's and indicator the microphone is open but idle. The consumer app may want to warn the user or terminate the session if this keeps happening.
Commands
Commands are represented by the ActionRecipe object. It includes the information required to be processed by the consumer app.
Commands are returned by the SDK via the onCommandResult callback (DictationManager).
ActionRecipe
As stated the result of a command (either Static or Dynamic) will be an instance of an ActionRecipe.
When a command is returned by the SDK the consumer needs to analyze it and determine what to do based on the properties of the ActionRecipe instance.
For example, a Static Command resulting ActionRecipe will always have it's isStaticCommand property set as true.
Other fields such as searchText, chooseNumber or selectFor are used to determine what to do with the command and in what context. For instance a Replace X with Y dynamic command will produce an ActionRecipe like:
// replace oranges with apples
{
name: 'replace', // AugnitoCommands.replace entry
isCommand: true,
searchText: 'oranges',
selectFor: 'apples'
...
}
ActionRecipe fields
property | notes |
---|---|
name | Command name. Can be matched against one of the entries of AugnitoCommands class. |
action | |
chooseNumber | Unit: on commands that affect items it indicates how many. |
searchText | On most select/action commands this represents the item to be searched. For example paragraph, line, word, etc. On some commands this indicates what is being searched for. |
selectFor | Usually used to determine an action on most commands. For example delete, underline, etc. |
isCommand | Indicates the ActionRecipe is a command. |
isStaticCommand | Indicates this ActionRecipe is a static command. |
nextPrevious | Direction on how this command should affect the items. |
receivedText | Original received text. |
receivedTextWithoutSpace | Original text without spaces. |
Using Commands
Commands are meant to be used on their on and not in conjunction with regular dictation.
For example:
Dictating Select next 3 words will produce a command
Dictating Patient presents with fever and chills select last 2 words will only produce a transcription
Matching commands
Commands can be matched with entries on the AugnitoCommands dictionary. In the previous example the name equals to AugnitoCommands.replace.
Static Commands
Commands that have not effect on specific units or items are denominated as static commands.
Microphone Control Static Commands |
---|
stopMic |
Selection Static Commands |
---|
selectAll |
selectChar |
selectWord |
selectLine |
selectNextLine |
selectPreviousLine |
selectParagraph |
Lists Static Commands |
---|
startNumberList |
startBulletList |
stopBulletList |
stopNumberList |
stopList |
Action Static Commands |
---|
undoIt |
redoIt |
Text Modification Commands |
---|
startBoldText |
stopBoldText |
startBulletText |
stopBulletText |
startItalicText |
stoptItalicText |
Modification Commands |
---|
deleteIt |
pasteIt |
copyIt |
cutIt |
headerIt |
underlineIt |
boldIt |
italicizeIt |
capitalizeIt |
bulletIt |
Navigation Static Commands |
---|
moveUp |
moveDown |
moveLeft |
moveRight |
goToLineStart |
goToLineEnd |
giveSpace |
backspace |
goToDocumentEnd |
goToDocumentStart |
goToNextPage |
goToPreviousPage |
goToNextParagraph |
goTo |
Field Navigation Static Commands |
---|
nextField |
previousField |
Dynamic Commands
Dynamic commands are more complex commands built based on the speech input and may contain extra information (always within the ActionRecipe itself).
Line and Paragraph number
Commands that affect a line or paragraph number via an action command such as select, bold, delete, etc.
^(select|choose|copy text|copytext|cut text|cuttext|correct|bold|underline|delete|header|capitalize|unbold|debold|dbold|uncapitalize|remove|capitalise|dcapitalise|decapitalicize|decapitalize|uncapitalize|Uncap|d capitalise|d capitalize|d underline|dunderline|deunderline|ununderline|goto|moveto|move|italicize|italicise|unitalicise|unitalicize)(the)?(line|para|paragraph|\n\n|\n)(number[ed]?)(.*?)(\.)?$
A correctly parsed Line and Paragraph number will have a structure similar to:
{
name: AugnitoCommands.goToLineNumber // or AugnitoCommands.goToParaNumber
isCommand: true,
chooseNumber: 10,
searchText: "line",
selectFor: "goToStart"
...
}
Example voice command
- choose line number 10
- delete paragraph number 5
- bold the para number 1
Go To New Line or New Paragraph
Simple Go To new line or new paragraph command.
{
name: AugnitoCommands.selectLine // or AugnitoCommands.selectParagraph
isCommand: true,
searchText: "line",
selectFor: "gotoend"
...
}
Example voice command
- go to new line
- go to new paragraph
Active X Dynamic Command
Action against the active line, paragraph, space, etc.
^(select|choose|copy text|copytext|cut text|cuttext|correct|bold|underline|delete|header|capitalize|unbold|debold|dbold|uncapitalize|remove|capitalise|dcapitalise|decapitalicize|decapitalize|uncapitalize|Uncap|d capitalise|d capitalize|d underline|dunderline|deunderline|ununderline|goto|moveto|move|italicize|italicise|unitalicise|unitalicize)(the)?(line|para|paragraph|\n\n|\n)(number[ed]?)(.*?)(\.)?$
Example voice command
- select active paragraph
- delete the current line
{
name: AugnitoCommands.selectActiveChar
isCommand: true,
...
}
{
name: AugnitoCommands.selectActiveWord
isCommand: true,
...
}
{
name: AugnitoCommands.selectActiveLine
isCommand: true,
...
}
{
name: AugnitoCommands.selectActiveParagraph
isCommand: true,
...
}
Navigation Without Go To Dynamic Command
Processes a direction plus a combination of item and distance.
^(last|previous|next|down)(.*?)(word[s]?|line[s]?|sentence[s]?|paragraph[s]?|para[s]?|char[s]?|character[s]?|space|\n\n|\n)$
Example voice commands
- next 3 lines
- last 5 paras
Go To X Dynamic Command
Processor to identify movement with direction and unit.
^(go|goto|gotothe|move|moveto|movethe)(last|previous|next|down)(.*?)(word[s]?|line[s]?|sentence[s]?|paragraph[s]?|para[s]?|char[s]?|character[s]?|space|\n\n|\n)$
Example voice commands
- go last 4 word
- go to last space
- move the previous 4 words
// go last 4 word
{
name: AugnitoCommands.selectWord
isCommand: true,
chooseNumber: 4,
nextPrevious: 'last',
selectFor: 'gotostart'
...
}
// go to last space
{
name: AugnitoCommands.selectChar
isCommand: true,
chooseNumber: 0,
nextPrevious: 'last',
selectFor: 'gotostart'
...
}
// move the previous 4 words
{
name: AugnitoCommands.selectWord
isCommand: true,
chooseNumber: 4,
nextPrevious: 'previous',
selectFor: 'gotostart'
...
}
Select / Action on Item with Unit and Direction Dynamic Command
Identifies an action on an item that includes direction and unit amount.
^(select|choose|copy text|copytext|cut text|cuttext|correct|bold|underline|delete|header|capitalize|unbold|debold|dbold|uncapitalize|remove|capitalise|dcapitalise|decapitalicize|decapitalize|uncapitalize|Uncap|d capitalise|d capitalize|d underline|dunderline|deunderline|ununderline|goto|moveto|move|italicize|italicise|unitalicise|unitalicize)(the)?(last|previous|next)(.*?)(word[s]?|line[s]?|sentence[s]?|paragraph[s]?|para[s]?|char[s]?|character[s]?|space|\n\n|\n\ns|\n|\ns)$
Example voice commands
- choose the next 10 characters
- underline previous new line
- capitalize the next 5 words
- delete previous 5 paragraphs
// choose the next 10 characters
{
name: AugnitoCommands.selectChar
isCommand: true,
chooseNumber: 10,
nextPrevious: 'next',
searchText: 'characters',
selectFor: ''
...
}
// underline previous new line
{
name: AugnitoCommands.selectLine
isCommand: true,
chooseNumber: 1,
nextPrevious: 'previous',
searchText: '\n',
selectFor: 'underline'
...
}
// capitalize the next 5 words
{
name: AugnitoCommands.selectWord
isCommand: true,
chooseNumber: 5,
nextPrevious: 'next',
searchText: 'words',
selectFor: 'capitalize'
...
}
// delete previous 5 paragraphs
{
name: AugnitoCommands.selectParagraph
isCommand: true,
chooseNumber: 5,
nextPrevious: 'previous',
searchText: '\n\n',
selectFor: 'delete'
...
}
Replace X with Y Dynamic Command
Identifies the replace operation between two elements.
replace[d]?([A-Z a-z 0-9]+)with([A-Z a-z 0-9]+)
Example voice commands
- replace oranges with apples
- replace blue with red
- replace cars with buses
// replace oranges with apples
{
name: AugnitoCommands.replace
isCommand: true,
searchText: 'oranges',
selectFor: 'apples'
...
}
Select/Action Group Dynamic Command
Identifies a Select/Action command.
^(select|choose|copy text|copytext|cut text|cuttext|correct|bold|underline|delete|header|capitalize|unbold|debold|dbold|uncapitalize|remove|capitalise|dcapitalise|decapitalicize|decapitalize|uncapitalize|Uncap|d capitalise|d capitalize|d underline|dunderline|deunderline|ununderline|goto|moveto|move|italicize|italicise|unitalicise|unitalicize)(\sthe)?\s?(.*?)$
Example voice commands
- delete mistaken
- bold firstname
- capitalize last name
// delete mistaken
{
name: AugnitoCommands.select
isCommand: true,
searchText: 'mistaken',
selectFor: 'delete'
...
}
Utils and Helpers
TextFieldProcessor
The SDK includes a processor to handle basic operations on the TextField widget (via the TextEditingController).
The methods included for the processor are:
- addContent
- selectLastLines
- selectLatWords
- deleteSelection
The usage of these methods is shown on the example app:
void _processSelectLine(ActionRecipe actionRecipe) {
TextFieldProcessor.selectLastLines(focusedEditor!.textEditingController,
offset: actionRecipe.chooseNumber);
if (actionRecipe.selectFor == Commands.Delete) {
TextFieldProcessor.deleteSelection(focusedEditor!.textEditingController);
}
}
void _processSelectWord(ActionRecipe actionRecipe) {
TextFieldProcessor.selectLastWords(focusedEditor!.textEditingController,
offset: actionRecipe.chooseNumber);
if (actionRecipe.selectFor == Commands.Delete) {
TextFieldProcessor.deleteSelection(focusedEditor!.textEditingController);
}
}
iOS and background UI modes
In order to keep the app working on the background the info.plist requires the following entries:
<key>UIBackgroundModes</key>
<array>
<string>audio</string>
<string>fetch</string>
</array>
Libraries
- action_recipe
- active_x_command_processor
- augnito_api_server
- augnito_audio_stream
- augnito_command_builder
- augnito_commands
- augnito_config
- augnito_dynamic_command_processor
- augnito_event_types
- augnito_flutter_sdk
- augnito_mic_stream
- augnito_print_logger
- augnito_server_response
- augnito_source
- augnito_static_command_processor
- augnito_web_client
- base_regex_command_processor
- command_utils
- dictation_error
- dictation_manager
- domain_utils
- generic_goto_processor
- goto_moveto_command_processor
- goto_moveto_paragraph_command_processor
- goto_x_command_processor
- line_paragraph_number_command_processor
- meta_event_response
- number_command_processor
- replace_command_processor
- result_response
- sdk_config
- select_group_processor
- select_item_direction_processor
- speech_to_text_output
- text_field_processor
- word_to_integer_utils