flutter_automate 1.1.1

A multi-language Android automation framework. Control UI, gestures, apps via Accessibility Service. Supports JavaScript scripting via WASM runtime. No Auto.js dependency.

Flutter Automate

A multi-language automation framework for Android. Supports JavaScript, Python, and other scripting languages via WASM runtime.

中文文档

Features

• 🚀 Multi-language Support - JavaScript, Python, Lua (via WASM)
• 📱 Complete Automation API - UI selectors, gestures, app management, device control
• 📸 Screen Capture - MediaProjection screenshot support (Android 10+ foreground service)
• 🔧 Pure Kotlin Implementation - No NDK required, no AutoJS dependency
• 🎯 Chainable API - Fluent API design
• 🔒 Secure - Scripts run in WASM sandbox

Installation

dependencies:
  flutter_automate:
    git:
      url: https://github.com/jiusanzhou/flutter_automate.git

Quick Start

1. Request Permissions

import 'package:flutter_automate/flutter_automate.dart';

final automate = FlutterAutomate.instance;

// Check and request accessibility service
final hasAccessibility = await automate.checkAccessibilityPermission();
if (!hasAccessibility) {
  await automate.requestAccessibilityPermission(wait: true, timeout: 30000);
}

// Check and request screenshot permission
final hasCapture = await automate.permissions.hasMediaProjection();
if (!hasCapture) {
  await automate.permissions.requestMediaProjection();
}

// Other permissions
await automate.permissions.requestStorage();           // Storage
await automate.permissions.requestManageStorage();     // All files access (Android 11+)
await automate.permissions.requestBatteryOptimizationExemption(); // Battery optimization
await automate.permissions.requestNotificationListener(); // Notification listener

2. Screen Capture

// Requires screenshot permission first
// Handle onActivityResult in MainActivity:
// ScreenCapture.onActivityResult(this, resultCode, data)

// Capture screen
final imageData = await automate.capture.capture();
if (imageData != null) {
  // imageData is Uint8List (PNG format)
  Image.memory(imageData);
}

// Capture and save to file
final success = await automate.capture.captureToFile(
  '/sdcard/Download/screenshot.png',
  quality: 90,
);

// Release resources
await automate.capture.release();

3. UI Automation

// Find element
final button = await automate.text("Login").findOne();

// Click
await automate.text("Login").click();

// Set text
await automate.id("username").setText("[e-mail]");

// Wait for element
final element = await automate.textContains("Success").waitFor(timeout: 5000);

// Chainable selector
final result = await automate
    .selector()
    .className("Button")
    .clickable()
    .findAll();

4. Gesture Operations

// Click at coordinates
await automate.click(500, 800);

// Long press
await automate.longClick(500, 800, duration: 1000);

// Swipe
await automate.swipe(100, 500, 100, 1500, duration: 300);

// Quick swipe
await automate.swipeUp();
await automate.swipeDown();

5. Global Actions

await automate.back();
await automate.home();
await automate.recents();
await automate.openNotifications();
await automate.takeScreenshot();

6. App Management

// Launch app
await automate.app.launch("com.example.app");
await automate.app.launchByName("WhatsApp");

// Get current app
final currentPkg = await automate.app.currentPackage();

// Force stop
await automate.app.forceStop("com.example.app");

// Get installed apps
final apps = await automate.app.getInstalled();

7. Device Info

// Device info
final info = await automate.device.info();
print("Model: ${info.model}");
print("Screen: ${info.screenWidth}x${info.screenHeight}");

// Clipboard
final text = await automate.device.getClipboard();
await automate.device.setClipboard("Hello");

// Vibrate
await automate.device.vibrate(duration: 100);

// Battery
final battery = await automate.device.getBattery();

8. Execute Scripts

// JavaScript
final execution = await automate.execute('''
  console.log("Hello from JS!");
  click(text("Login"));
  sleep(1000);
  swipeUp();
''', language: 'js');

// Python (coming soon)
await automate.execute('''
import automate
automate.click(text("Login"))
''', language: 'python');

Android Configuration

MainActivity

Screenshot feature requires handling permission callback in MainActivity:

// MainActivity.kt
package your.package.name

import android.content.Intent
import io.flutter.embedding.android.FlutterActivity
import im.zoe.flutter_automate.core.ScreenCapture

class MainActivity : FlutterActivity() {
    
    override fun onActivityResult(requestCode: Int, resultCode: Int, data: Intent?) {
        if (requestCode == ScreenCapture.REQUEST_CODE) {
            ScreenCapture.onActivityResult(this, resultCode, data)
            return
        }
        super.onActivityResult(requestCode, resultCode, data)
    }
}

AndroidManifest.xml

<!-- Basic permissions -->
<uses-permission android:name="android.permission.SYSTEM_ALERT_WINDOW" />
<uses-permission android:name="android.permission.FOREGROUND_SERVICE" />
<uses-permission android:name="android.permission.FOREGROUND_SERVICE_SPECIAL_USE" />
<uses-permission android:name="android.permission.FOREGROUND_SERVICE_MEDIA_PROJECTION" />

<!-- Storage permissions -->
<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
<uses-permission android:name="android.permission.MANAGE_EXTERNAL_STORAGE" />

<!-- Other -->
<uses-permission android:name="android.permission.VIBRATE" />
<uses-permission android:name="android.permission.REQUEST_IGNORE_BATTERY_OPTIMIZATIONS" />

Permissions Reference

Permission	Purpose	API
Accessibility Service	UI control, gesture execution	permissions.requestAccessibility()
Overlay	Display floating control panel	FloatwingPlugin
Screen Capture	MediaProjection screenshot	permissions.requestMediaProjection()
Storage	Read/write files	permissions.requestStorage()
All Files Access	Android 11+ access all files	permissions.requestManageStorage()
Battery Optimization	Background keep-alive	permissions.requestBatteryOptimizationExemption()
Notification Listener	Read system notifications	permissions.requestNotificationListener()

Architecture

flutter_automate/
├── lib/
│   └── flutter_automate.dart    # Flutter/Dart API
├── android/
│   └── src/main/kotlin/
│       ├── core/                 # Core automation modules
│       │   ├── AutomateAccessibilityService.kt
│       │   ├── UiSelector.kt
│       │   ├── UiObject.kt
│       │   ├── GestureEngine.kt
│       │   ├── ScreenCapture.kt
│       │   ├── ScreenCaptureService.kt
│       │   ├── AppUtils.kt
│       │   └── DeviceUtils.kt
│       ├── wasm/                 # WASM runtime
│       │   ├── ScriptEngineManager.kt
│       │   └── QuickJSEngine.kt
│       └── FlutterAutomatePlugin.kt
└── example/                      # Example app

License

MIT License

Contributing

Issues and Pull Requests are welcome!