voxera_media_client 1.6.2

Voxera Media Client — real-time audio and video for Flutter (iOS, Android, Web).

CHANGELOG

1.6.2

• README: every code sample now compiles. localParticipant is nullable, roomOptions goes in the Room constructor, and local tracks mute with mute() and unmute().

1.6.1

• The package homepage points to voxerastream.com.

1.6.0

• Android: the native WebRTC engine no longer floods logcat. Left alone it writes several lines a second for every ICE connectivity check (connection.cc: Sent STUN BINDING request ...), which buried the app's own logs. The engine now logs errors only. VoxeraMediaClient.setNativeLogLevel(VoxeraNativeLogLevel.info) (or verbose) brings the detail back while debugging a connection; nativeLogLevel reads it. The plugin filters the engine's lines in the process logger from the moment it attaches, before WebRTC loads, so this covers every room and call client in the app. Android 11 and later; on older versions the engine keeps its own level. (WebRTC 150 no longer lets Java lower that level: Logging.enableLogToDebugOutput changes a setting that stopped gating the output.)

1.5.0

• VoxeraPictureInPicture.changes: the Android activity's Picture-in-Picture state, read by the plugin (now ActivityAware) whenever the activity pauses, resumes, starts or stops, and checked again shortly after each. The room kits listen to it, so an app no longer forwards onPictureInPictureModeChanged from its MainActivity. Every listener gets the current state first. Empty on other platforms.

1.4.0

• VoxeraKit.configure(appId:, appKey:, tokenBaseUrl:, engineBaseUrl:) stores the app's credentials and hosts once. The audio and live room kits read them, so their widgets no longer need those arguments. VoxeraKit.settings reads them back.
• VoxeraMediaClient.isEngineInitialized and VoxeraMediaClient.engineProfile (roomVoice, media, call or null) let an app check the engine without reaching into flutter_webrtc.

1.3.4

• device_info_plus accepts 12.3 and up, and 13. 13 changed build tooling only (win32 6, iOS 13), so a host app on it no longer needs a dependency_overrides entry.

1.3.3

• Track.enable() / disable() no longer raise an uncaught PlatformException(MediaStreamTrack has been disposed.) when the native track was already disposed (its peer connection closed, or the remote track removed). flutter_webrtc's enabled setter does not await its native call, so the old try/catch never saw the failure and apps reported it as a fatal crash; it is now logged as a warning. The room-music track's setter handles the same failure.

1.3.2

• iOS: noise cancellation (AudioCaptureOptions.noiseCancellation, LocalAudioTrack.setNoiseCancellation) now runs on iOS too; before, iOS ignored it and isNoiseCancellationSupported() returned false.
  • The same DeepFilterNet 3 denoiser (android/src/main/cpp/vx_denoise.c, int8 weights) and voice focus gate as Android, in flutter_webrtc's capture post-processing ahead of the room mix (NoiseCancellationProcessor.swift, NearVoiceGate.swift). Only 48 kHz capture is processed; a Bluetooth HFP microphone passes through. Too slow for the device: it passes audio through, delayed by the same 30 ms.
  • The sources are built through ios/Classes/noise_cancellation and the weights are embedded in the framework (about 2.3 MB). The pod's C code is now built with -O3 in debug builds too.
  • VoxeraMediaClient.noiseCancellationStats() reads iOS as well.

1.3.1

• iOS: the pod now builds. 1.3.0 listed ../cpp/music_opus.cc and the Opus sources outside ios/, which CocoaPods never compiles, so MusicFrameEncryptor.mm failed with 'music_opus.h' file not found (and would not have linked without Opus).
  • The sources are compiled through ios/Classes/forwarders, one file per source that #includes the original under cpp/ or third_party/opus/; Android keeps building the same files. Regenerate with dart scripts/gen_ios_forwarders.dart.
  • MusicFrameEncryptor.mm imports <voxera_media_client/voxera_media_client-Swift.h> when the app uses use_frameworks! (the quoted import is kept as the fallback).

1.3.0

• Room music as an aux mix (RoomMusicAux), off unless used. See doc/room-music-aux-mix.md.
  • The song plays on this phone's speaker as before (the echo-cancellation reference) and is added to the published microphone after AEC3, noise suppression, AGC, noise cancellation and the silence gate, which never touch it. The room hears one track; there is no room-music track.
  • setMicMuted(true) puts the voice at gain 0 (10 ms ramp) while the song keeps going out; start(micMuted:) sets it from the first frame. Publish volume and local playout volume are separate. start(loop:) repeats the file.
  • While a song plays the microphone's Opus bitrate is raised to musicBitrate (48 kbps by default) with RTCRtpSender.setParameters, no republish; DTX and RED stay as published.
  • The phone announces its aux state (song live, voice muted, voice active measured before the mix) in a one-byte lossy data packet on vx.aux: on every change and every 2 s while a song is live; a state lasts 5 s without a repeat. The packets are consumed and never reach DataReceivedEvent. The attributes vx_music=aux / vx_mic=0 are also set as a best effort (a seat's runtime permissions can refuse them).
  • Participant.isMicrophoneEnabled() and isMuted report a participant whose voice is muted under a song as muted (the local one from local state); a change is announced as ParticipantAttributesChanged with the vx_music / vx_mic keys. The active-speaker list and isSpeaking of a participant with a live song follow its voice bit, so the song alone never lights a seat.
  • Native: Android RoomPlaybackMixer reads an aux song on the capture thread from its own ring reader, started at the speaker's position and kept within 200 ms of it; new calls playbackAuxMicGain, playbackAuxStats, event auxVoiceChanged. iOS: the same in RoomPlaybackMixer.swift, not compiled or tested yet.
• RoomMusicPublisher.keepTrackOnRemoteMute (default false): a server mute of the room-music track only mutes it instead of stopping it, and restoreAfterSeatUnmute() unmutes it after a seat unmute (LiveKit refuses a server-side unmute).
• RoomPlaybackMix.route(aux:) and RoomPlaybackController.start(aux:) (default false: unchanged).

1.2.0

• New RoomOptions.resilientReconnect (default false). When true:
  • After a full reconnect, a local track that fails to publish again is unpublished (LocalTrackUnpublishedEvent; stopped as with stopLocalTrackOnUnpublish) and the other tracks still go out; RoomReconnectedEvent is emitted either way. Before, the first failure ended the EngineRestartedEvent handler: the tracks after it were not published and RoomReconnectedEvent never came, so a kit stayed in reconnecting with the media session back. Seen on phones and the emulator after network cuts of about 100 s: the server had freed the user's seat, answered the microphone's AddTrack with NOT_ALLOWED, and the publish timed out.
  • While a reconnect waits for the network (networkIsAvailable() false), it also checks the network every 2 s instead of waiting only for a connectivity change. Android sends no change when an app's own network access comes back (data saver, firewall), and the wait then ran its full timeouts.connection * 10 (70 s with the room kits' 7 s) before the next attempt. When the engine is closed during that wait (the user left), the wait ends with an error and no reconnect attempt follows.
• LocalParticipant.rePublishAllTracks takes dropFailed (default false) for the first point.
• New AudioCaptureOptions.noiseCancellation (NoiseCancellationMode.off by default, standard or voiceFocus), Android only: AI noise cancellation of the published microphone, on the device.
  • The model is DeepFilterNet 3 (Hendrik Schröter, MIT / Apache-2.0) run by our own streaming C engine (vx_denoise.c) with int8 weights: no ONNX Runtime and no download. libvoxera_nc.so is 2.4 MB per ABI (the original model alone is 8.6 MB, ONNX Runtime 17.6 MB more). It matches the reference (libDF 0.5.6, deep-filter thresholds) within int8 rounding.
  • standard removes noise (fan, AC, street, keyboard) and keeps every voice, including a TV or people talking nearby. voiceFocus then lowers everything but the voice close to the phone by 20 dB: a gate learns the talker's level (90th percentile of the last minute of speech, 300-3400 Hz) and passes audio within 18 dB of it, 6 dB hysteresis, 600 ms hold, look-ahead from the two newer input frames (no extra delay). A closed gate lowers, it does not mute, so a soft word it misses stays audible.
  • Order in WebRTC capture post-processing: noise cancellation, then the silence gate, then the room playback mix (gift sounds and room music are never denoised). WebRTC's own AEC3, noise suppression and gain control stay as they are; the model runs after them.
  • Cost on a Redmi Note 12 (Snapdragon 685): 2.0 ms per 10 ms frame on a 2.8 GHz core, 5.7 ms on a 1.9 GHz efficiency core, 2.7-3.1 ms on the live capture thread. 30 ms of extra delay (10 ms STFT overlap, 20 ms model lookahead). Twenty frames in a row over 8 ms and it passes audio through (bypassed in the stats).
  • On that phone in a room with a fan and a TV, the owner talking: the quiet moments went from -43 dBFS to -80 (standard) and -103 (an earlier muting gate). Ten minutes with the mic open and speech half the time, voiceFocus against off: uplink 22.66 -> 14.96 MB/h, 49.9 -> 34.5 packets/s (Opus DTX in the gaps), 2.7 ms per 10 ms on the capture thread, 0.5 % of frames over 8 ms (one burst at a CPU temperature peak, no audible effect at the receiver), skin temperature at most 1.6 C higher.
  • LocalAudioTrack.setNoiseCancellation switches it on a live microphone without restarting the capture. VoxeraMediaClient.isNoiseCancellationSupported() tells whether this device and build can run it (it loads the library); VoxeraMediaClient.noiseCancellationStats() returns NoiseCancellationStats for debugging.
  • Size: libvoxera_nc.so is built into every app that depends on this version, whether it turns noise cancellation on or not: about 2.4 MB per ABI, so about 2.4 MB more per device from an app bundle and about 7 MB more in a fat APK (arm64-v8a, armeabi-v7a, x86_64). An app that never uses it can leave the library out of its APK (packaging { jniLibs { excludes += ["**/libvoxera_nc.so"] } }); support then reads false.
  • off attaches nothing, loads no library and makes no native call; only a track that asks for noise cancellation checks support.
  • The silent-recorder watch reads the input of noise cancellation while it processes, so its silence never triggers a recorder restart.
  • Robustness: a frame with NaN or Inf passes through untouched; a non-finite value inside the model resets it (nativeResets in the stats); audio that resumes after digital silence starts the pipeline over, so no stale audio lands before the next word; a capture rate other than 48 kHz (a 16 kHz Bluetooth headset) passes through unprocessed and the model starts over when 48 kHz is back. Too slow (20 frames in a row over 8 ms, or an average over 6 ms for 2 s): it passes the microphone through, delayed by the same 30 ms and cross-faded, so the switch drops nothing. A failed native start is caught. Consumer R8 rules keep NoiseCancellationNative's native methods.
  • The model removes music as noise: use it with speech. iOS ignores the option for now.

1.1.0

• Android consumer R8 rules keep the classes shared_preferences_android 2.5.x reaches through JNI. 2.5.0 moved to Pigeon JNI and ships no keep rules, so in a host app's release build R8 renamed or stripped kotlin.coroutines.Continuation and kotlin.Result, and every SharedPreferences call failed (NoSuchMethodError on getAll, then ClassNotFoundException). The room kits catch that error, so the device id changed on every launch and the connect budget and RTC prewarm hints were never saved. Host apps no longer need their own proguard-rules.pro for this.
• New AudioCaptureOptions.silenceGate (SilenceGateMode.off by default, or standard), Android only. With standard, a published microphone hands the Opus encoder digital silence instead of room noise whenever nobody talks, so DTX sends about one packet per 400 ms instead of 50 per second. An open but silent phone mic in a normal room otherwise keeps sending ~23 kbps (about 22 MB/h up, and the same down for every listener).
  • The gate is a capture post-processor (after AEC3, noise suppression and gain control, before the encoder). It opens on voice, not on any sound: the 300-3400 Hz band must be above a threshold (the measured noise floor, the quietest moment of the last 1.6 s, plus 9 dB on the loudspeaker or 7 dB on a headset, never below -45 dBFS / -48 dBFS on a headset) and carry a pitch (70-400 Hz) for 20 ms, or be at least 6 dB over that threshold for 20 ms. A fan next to the phone, wind on the microphone, typing and footsteps do not open it. The output lags by 50 ms so the start of a word goes out (40 ms before the confirming frames); every voiced frame keeps it open for another 400 ms; it fades out over 60 ms. A closed gate writes exact zeros.
  • Measured on a phone in a real room (fan on, balcony, the owner moving): an open mic sent 49 packets/s and 20.96 MB/h up with the gate off; with the gate on the room's quiet moments went out as DTX and it opened only on voice in the room (57 of 67 openings voiced): 21.6 packets/s, 9.71 MB/h. On a recording of the same phone in a quiet room the gate is open 1-2 % of the time; the owner's speech passed whole (100 % of voiced frames, 0 ms lost at the start of sentences).
  • off attaches nothing: the capture path is the stock one and the media client makes no extra native call. The first audio route change creates the gate's objects (no audio runs through them).
  • Gift sounds and room music mixed into the microphone (RoomPlaybackMixer publish gain) run after the gate and are never cut. They are not delayed either, so in the published stream the voice comes 50 ms after them.
  • The gate is process-wide and stays on while any published microphone asks for it. If WebRTC was not ready when it was asked, it is retried on the next stats tick, and it follows a new WebRTC engine. A non-finite sample (NaN, infinity) restarts it instead of leaving it broken.
  • CPU: about 0.6 % of one core on average, about 1.25 % while the gate is open on loud speech.
  • The silent-recorder watch reads the gate's input energy while the gate is processing, so the gate's zeros never trigger a recorder restart; a recorder that really delivers zeros is still restarted.
  • VoxeraMediaClient.silenceGateStats() returns SilenceGateStats for debugging: open share, opens, noise floor, open/close thresholds, route, sample rate, failed.
  • Use it with DTX on (speech). iOS ignores the option for now.
• LocalParticipant.setMicrophoneEnabled takes an optional audioPublishOptions, used when the call publishes a new microphone track. Options that are fixed per publication, such as RED and DTX, can then be chosen at each publish (the room kits pick RED from the loss the device measured). Null keeps RoomOptions.defaultAudioPublishOptions, as before. The options are ignored while a microphone publication exists (an unmute keeps the options it was published with; LocalAudioTrack.lastPublishOptions shows them).

1.0.12

• The primary PeerConnection wait during Room.connect uses Timeouts.peerConnection instead of Timeouts.connection. Kits shorten connection so a dead signal edge fails fast, which also cut ICE to the same short budget and failed joins on slow or TURN-only networks. The defaults are both 10s, so apps that keep the defaults see no change.
• VoxeraMediaSessionCoordinator.claim waits at most releaseTimeout (3s) for the previous owner's teardown, and a teardown that fails no longer fails the new claimant. The old session keeps releasing in the background.
• New VoxeraInstallId: a process-wide single-flight the room kits share for their device_id, so two kits can no longer pick different ids on a fresh install.

1.0.11

• A published Android microphone keeps its silent-recorder check when Track.statsMonitorEnabled is off. The check ran on the stats tick, so turning stats off also turned off the recorder restart. Only that one track polls; remote and video tracks stay quiet, and no stats events are emitted while stats are off.
• iOS room mixer: the render callback reads each output channel once per callback instead of once per sample, and reuses one PCM buffer instead of allocating one per source every 10 ms. Output is unchanged.
• Android room mixer: the 20 ms room-music publish pull reuses its buffers per encoder thread, and the render callback no longer allocates a list on every call.

1.0.10

• Drop digital silence in room music to save bandwidth during pauses, song transitions, and silent intros. Opus prediction is cleanly reset on audio resume.
• Automatically rebind the music frame encryptor after full room reconnects, preventing microphone audio leak on republished music tracks.
• Clean up data channel state stream subscriptions and track ended listeners to prevent memory and event listener leaks.
• Fall back to candidate-pair stats for round trip time when outbound-rtp lacks RTT.
• Optimize Android JNI thread attachment during native PCM pulls using pthread thread-local storage.

1.0.9

• Fix the Android arm64 crash when room music starts. WebRTC's arm64 build calls the music frame encryptor through a relative vtable, so that file is now compiled with the same ABI.

1.0.8

• Encode room music at 48 kbps with DTX off, and drop a short read instead of sending a silence frame.
• A personal music mute disables the remote music publication, so the server stops sending that track.

1.0.7

• Room music is published as mono. A stereo track rebuilt the live voice device and aborted the process on some phones when play was tapped.
• Open music files through a file descriptor, and do not start a 48 kHz side player while the room is running at another rate.
• Keep the music audio source alive for the life of its track, and ignore mixer errors on the audio thread instead of killing the process.

1.0.6

• Add native Opus frame encryption and music publisher support for Android and iOS.
• Report device temperature, audio-mode changes, and route changes in the room diagnostics dump, including iOS.
• Turn on the microphone high-pass filter by default so low-frequency rumble does not unsettle echo cancellation.
• Turn off Android's platform noise suppressor when a room keeps hardware echo cancellation off. That suppressor was running on the microphone before WebRTC's echo canceller.
• Keep gift and game audio on the echo canceller's playback path while a microphone is publishing, instead of playing it on a separate speaker output.
• Add LocalTrack.detachCaptureWhileMuted(). It takes a muted track off its sender so WebRTC stops recording and audio processing, and unmute() puts it back, including a server-driven unmute.
• Add Track.statsMonitorEnabled to turn off the per-track getStats() poll that runs every 2 seconds for each started track.

1.0.5

• Add native authoritative private audio route detection (hasPrivateAudioRoute) on Android and iOS to accurately handle Bluetooth and wired headsets even with non-standard accessory labels.
• Prevent audio leakage to phone loudspeaker during publisher seat transitions before microphone capture starts.
• Reassert audio routing immediately upon Android audio device changes and Bluetooth disconnects to prevent audio trapping on the call earpiece.
• Reapply speakerphone fallback only when confirmed that no private routes are available.

1.0.4

• Make iOS/macOS room playback safe when WebRTC changes between mono and stereo during an app background/foreground transition.
• Synchronize native playback gain and pause state read by the real-time audio callback.
• Align the Apple WebRTC-SDK pod with flutter_webrtc 1.6.2 (150.7871.01).

1.0.3

• In-room HTML5 game audio is captured from the game WebView and played through the room mixer, so the player hears it and echo cancellation keeps it out of the microphone.
• Gift and music playback falls back to the speaker directly when WebRTC is not rendering audio.

1.0.2

• Publish the current media client so the room kits can depend on this version.

1.0.0

• Initial release of Voxera Media Client — real-time audio and video SDK for Flutter (iOS, Android, Web, macOS, Windows, Linux).

2.8.5

• Uplink audio quality was invisible: every speaker looked perfect. packetsLost, jitter and roundTripTime are not fields of outbound-rtp — they are what the REMOTE end reports about what it received, and live in the linked remote-inbound-rtp entry. The audio sender read all three off outbound-rtp, where they are always null, and the caller turned that null into a zero. Measured on production: 802,417 quality reports with round-trip time null in ALL of them and loss reported as 0.0000 throughout — a speaker losing a fifth of their packets was indistinguishable from a flawless one. Now resolved through remoteId, the same way the video sender already did. With no linked report yet the values stay NULL, so "not known" stays distinguishable from "no loss".

2.8.4

• A replaced subscriber data channel no longer keeps delivering. When the server opened a data channel for a label it had already opened (a subscriber renegotiation), the field was overwritten but the previous channel stayed OPEN with its message handler attached — so from then on every message on that label was handled once per still-attached channel, silently duplicating the room's own state updates and anything an app sends, for the rest of the session. The replaced channel is now detached and closed.
• Fixed an unbounded listener leak on the same path. The subscriber channel's state-change handler registered ANOTHER listener on the publisher channel every time it fired, so both the listener count and the emitted SubscriberDataChannelStateUpdatedEvents grew without bound. It now subscribes once, reports the subscriber channel's own state (it previously reported the publisher's), and the subscription is cancelled when the channel is replaced or the connection is torn down.

2.8.3

• Loud failure when the audio profile can no longer be applied. The native engine is built once per process and flutter_webrtc builds it implicitly (call profile, no options) on the first native call of any kind — device enumeration, a video renderer, a speakerphone toggle, another kit. A VoxeraMediaClient.initialize(mediaAudioProfile: true) that arrives after that was silently a no-op: the room then ran in the call profile for the rest of the process, with no runtime way back. initialize now detects it and logs a SEVERE, unmistakable message instead; VoxeraMediaClient.engineHasMediaAudioProfile exposes the applied profile so a kit can assert it. Rule for integrators: the kit that owns the audio profile must be the first thing in the app to touch the media engine.

2.8.2

• Android: a listener's first join no longer plays through the earpiece. Under the media profile (mediaAudioProfile / bypassVoiceProcessing) the audio device module was still built with WebRTC's default USAGE_VOICE_COMMUNICATION playout attributes. Android routes that usage with the phone strategy, so under MODE_NORMAL — with nothing having forced the speaker — playout went to the earpiece on a large share of devices until something activated device routing (which only happens once a local audio track exists). Leaving and re-entering "fixed" it, which is exactly how it was reported. initialize now passes media attributes (USAGE_MEDIA / CONTENT_TYPE_SPEECH) so playout routes like music — speaker, A2DP, wired — by platform policy from the first frame. No API change; the call-profile initialize is untouched.

2.8.1

• Fix compile error on older Dart front-ends: publishOptions accesses inside the video negotiate() closure relied on flow-analysis promotion that older SDKs reject after an await ("could not be promoted due to an 'await' or 'yield'"). Read the options once into a non-nullable local instead.

2.8.0

• Add mediaAudioProfile to VoxeraMediaClient.initialize: keeps the OS audio session in the media profile (Android MODE_NORMAL + music stream on connect, no Apple voice-processing IO unit — same native layer as bypassVoiceProcessing) while the software audio processing (echoCancellation / noiseSuppression / autoGainControl) keeps following AudioCaptureOptions instead of being forced off. Fixes degraded speaker audio (no AEC/NS/AGC at all) in media-mode rooms. bypassVoiceProcessing semantics are unchanged: media profile AND raw, unprocessed microphone.
• Fix AudioCaptureOptions.toMediaConstraintsMap(): the voiceIsolation constraint was populated from noiseSuppression instead of the voiceIsolation field.

2.7.1

• Fix iOS/macOS build failure: ios/Classes and macos/Classes Swift sources were published as symlink path stubs (plain-text files containing ../../shared_swift/...) instead of real code, breaking pod install builds with "Expressions are not allowed at the top level". Sources are now real files.

2.7.0

• Realtime audio, video and data channels for Flutter across iOS, Android, Web, macOS, Windows and Linux.
• Adaptive stream and dynacast for efficient bandwidth usage.
• Screen sharing on all supported platforms.
• End-to-end encryption for audio/video.
• Declarative, event-driven room and participant APIs.