UFO³ : Weaving the Digital Agent Galaxy

From Single Device Agent to Multi-Device Galaxy

📖 Language / 语言: English | 中文

📚 Quick Links: 🌌 UFO³ README • 🖥️ UFO² README • 📖 Full Documentation

🎯 Choose Your Path

🎬 See UFO³ Galaxy in Action

Watch how UFO³ Galaxy orchestrates complex workflows across multiple devices:

🎥 Click to watch: Cross-device task orchestration with UFO³ Galaxy

🌟 What's New in UFO³?

Evolution Timeline

%%{init: {'theme':'base', 'themeVariables': { 'primaryColor':'#E8F4F8','primaryTextColor':'#1A1A1A','primaryBorderColor':'#7CB9E8','lineColor':'#A8D5E2','secondaryColor':'#B8E6F0','tertiaryColor':'#D4F1F4','fontSize':'16px','fontFamily':'Segoe UI, Arial, sans-serif'}}}%%
graph LR
    A["<b>🎈 UFO</b><br/><span style='font-size:14px'>February 2024</span><br/><span style='font-size:13px; color:#666'><i>GUI Agent for Windows</i></span>"] 
    B["<b>🖥️ UFO²</b><br/><span style='font-size:14px'>April 2025</span><br/><span style='font-size:13px; color:#666'><i>Desktop AgentOS</i></span>"]
    C["<b>🌌 UFO³ Galaxy</b><br/><span style='font-size:14px'>November 2025</span><br/><span style='font-size:13px; color:#666'><i>Multi-Device Orchestration</i></span>"]
    
    A -->|Evolve| B
    B -->|Scale| C
    
    style A fill:#E8F4F8,stroke:#7CB9E8,stroke-width:2.5px,color:#1A1A1A,rx:15,ry:15
    style B fill:#C5E8F5,stroke:#5BA8D0,stroke-width:2.5px,color:#1A1A1A,rx:15,ry:15
    style C fill:#A4DBF0,stroke:#3D96BE,stroke-width:2.5px,color:#1A1A1A,rx:15,ry:15

🚀 UFO³ = Galaxy (Multi-Device Orchestration) + UFO² (Device Agent)

UFO³ introduces Galaxy, a revolutionary multi-device orchestration framework that coordinates intelligent agents across heterogeneous platforms. Built on five tightly integrated design principles:

1. 🌟 Declarative Decomposition into Dynamic DAG - Requests decomposed into structured DAG with TaskStars and dependencies for automated scheduling and runtime rewriting

2. 🔄 Continuous Result-Driven Graph Evolution - Living constellation that adapts to execution feedback through controlled rewrites and dynamic adjustments

3. ⚡ Heterogeneous, Asynchronous & Safe Orchestration - Capability-based device matching with async execution, safe locking, and formally verified correctness

4. 🔌 Unified Agent Interaction Protocol (AIP) - WebSocket-based secure coordination layer with fault tolerance and automatic reconnection

5. 🛠️ Template-Driven MCP-Empowered Device Agents - Lightweight toolkit for rapid agent development with MCP integration for tool augmentation

🎓 Migration Path

For UFO² Users:

1. ✅ Keep using UFO² – Fully supported, actively maintained
2. 🔄 Gradual adoption – Galaxy can use UFO² as Windows device agent
3. 📈 Scale up – Move to Galaxy when you need multi-device capabilities
4. 📚 Learning resources – Migration Guide

✨ Capabilities at a Glance

🌌 Galaxy Framework – What's Different?

User Request
     ↓
ConstellationAgent
     ↓
  [Task DAG]
   /   |   \
Task1 Task2 Task3
(Win) (Linux)(Mac)

Selection Criteria
  • Platform
  • Resource
  • Task requirements
  • Performance history
        ↓
  Auto-Assignment
        ↓
  Optimal Devices

Task1 → Running  ✅
Task2 → Pending  ⏸️
Task3 → Running  🔄
        ↓
   Completion
        ↓
   Final Report

🪟 UFO² Desktop AgentOS – Core Strengths

UFO² serves dual roles: standalone Windows automation and Galaxy device agent for Windows platforms.

As Galaxy Device Agent:

• Receives tasks from ConstellationAgent via Galaxy orchestration layer
• Executes Windows-specific operations using proven UFO² capabilities
• Reports status and results back to TaskOrchestrator
• Participates in cross-device workflows seamlessly

🚀 Quick Start Guide

Choose your path and follow the detailed setup guide:

# 1. Install
pip install -r requirements.txt

# 2. Configure ConstellationAgent
copy config\galaxy\agent.yaml.template config\galaxy\agent.yaml
# Edit and add your API keys

# 3. Configure devices
# Edit config\galaxy\devices.yaml to register your devices

# 4. Start device agents (with platform flags)
# Windows: Start server + client
# Linux: Start server + MCP servers + client  
# Mobile (Android): Start server + MCP servers + client
# See platform-specific guides for detailed setup

# 5. Launch Galaxy
python -m galaxy --interactive

# 1. Install
pip install -r requirements.txt

# 2. Configure
copy config\ufo\agents.yaml.template config\ufo\agents.yaml
# Edit and add your API keys

# 3. Run
python -m ufo --task <task_name>

📋 Common Configuration

Both frameworks require LLM API configuration. Choose your provider:

CONSTELLATION_AGENT:
  REASONING_MODEL: false
  API_TYPE: "openai"
  API_BASE: "https://api.openai.com/v1/chat/completions"
  API_KEY: "sk-your-key-here"
  API_MODEL: "gpt-4o"

VISUAL_MODE: True
API_TYPE: "openai"
API_BASE: "https://api.openai.com/v1/chat/completions"
API_KEY: "sk-your-key-here"
API_MODEL: "gpt-4o"

CONSTELLATION_AGENT:
  REASONING_MODEL: false
  API_TYPE: "aoai"
  API_BASE: "https://YOUR-RESOURCE.openai.azure.com"
  API_KEY: "your-azure-key"
  API_MODEL: "gpt-4o"
  API_DEPLOYMENT_ID: "your-deployment-id"

VISUAL_MODE: True
API_TYPE: "aoai"
API_BASE: "https://YOUR-RESOURCE.openai.azure.com"
API_KEY: "your-azure-key"
API_MODEL: "gpt-4o"
API_DEPLOYMENT_ID: "your-deployment-id"

💡 More LLM Options: See Model Configuration Guide for Qwen, Gemini, Claude, and more.

📚 Documentation Structure

📢 Latest Updates

2025-11 – UFO³ Galaxy Framework Released 🌌

Major Research Breakthrough: Multi-Device Orchestration System

• 🌟 Declarative DAG Decomposition: TaskConstellation structure for workflow logic and dependencies
• 🔄 Dynamic Graph Evolution: Living constellation that adapts through controlled rewrites
• 🎯 Heterogeneous Orchestration: Safe, asynchronous execution with capability-based device matching
• 🔌 Unified AIP Protocol: WebSocket-based secure agent coordination with fault tolerance
• 🛠️ MCP-Empowered Agent Framework: Template-driven toolkit for rapid device agent development
• 📄 Research Paper: UFO³: Weaving the Digital Agent Galaxy

Key Features:

• First multi-device orchestration framework for GUI agents
• Result-driven adaptive execution instead of rigid workflows
• Model Context Protocol (MCP) integration for tool augmentation
• Formally verified correctness and concurrency safety guarantees

2025-04 – UFO² v2.0.0

• 📅 UFO² Desktop AgentOS released
• 🏗️ Enhanced architecture with AgentOS concept
• 📄 Technical Report published
• ✅ Entered Long-Term Support (LTS) status

2024-02 – Original UFO

• 🎈 First UFO release - UI-Focused agent for Windows
• 📄 Original Paper
• 🌍 Wide media coverage and adoption

📚 Citation

If you use UFO³ Galaxy or UFO² in your research, please cite the relevant papers:

UFO³ Galaxy Framework (2025)

@article{zhang2025ufo3,
  title={UFO$^3$: Weaving the Digital Agent Galaxy}, 
  author = {Zhang, Chaoyun and Li, Liqun and Huang, He and Ni, Chiming and Qiao, Bo and Qin, Si and Kang, Yu and Ma, Minghua and Lin, Qingwei and Rajmohan, Saravan and Zhang, Dongmei},
  journal = {arXiv preprint arXiv:2511.11332},
  year    = {2025},
}

UFO² Desktop AgentOS (2025)

@article{zhang2025ufo2,
  title   = {{UFO2: The Desktop AgentOS}},
  author  = {Zhang, Chaoyun and Huang, He and Ni, Chiming and Mu, Jian and Qin, Si and He, Shilin and Wang, Lu and Yang, Fangkai and Zhao, Pu and Du, Chao and Li, Liqun and Kang, Yu and Jiang, Zhao and Zheng, Suzhen and Wang, Rujia and Qian, Jiaxu and Ma, Minghua and Lou, Jian-Guang and Lin, Qingwei and Rajmohan, Saravan and Zhang, Dongmei},
  journal = {arXiv preprint arXiv:2504.14603},
  year    = {2025}
}

Original UFO (2024)

@article{zhang2024ufo,
  title   = {{UFO: A UI-Focused Agent for Windows OS Interaction}},
  author  = {Zhang, Chaoyun and Li, Liqun and He, Shilin and Zhang, Xu and Qiao, Bo and Qin, Si and Ma, Minghua and Kang, Yu and Lin, Qingwei and Rajmohan, Saravan and Zhang, Dongmei and Zhang, Qi},
  journal = {arXiv preprint arXiv:2402.07939},
  year    = {2024}
}

🌐 Media & Community

Media Coverage:

• 微软正式开源UFO²，Windows桌面迈入「AgentOS 时代」
• Microsoft's UFO: Smarter Windows Experience
• 下一代Windows系统曝光
• More coverage →

Community:

• 💬 GitHub Discussions
• 🐛 Issue Tracker
• 📧 Email: ufo-agent@microsoft.com
• 📺 YouTube Channel

🎨 Related Projects & Research

Microsoft Research:

• TaskWeaver – Code-first LLM agent framework for data analytics and task automation

GUI Agent Research:

• LLM-Brained GUI Agents Survey – Comprehensive survey of GUI automation agents
• Interactive Survey Site – Explore latest GUI agent research and developments

Multi-Agent Systems:

• UFO³ Galaxy represents a novel approach to multi-device orchestration, introducing the Constellation framework for coordinating heterogeneous agents across platforms
• Builds on multi-agent coordination research while addressing unique challenges of cross-device GUI automation

Benchmarks:

• Windows Agent Arena (WAA) – Evaluation benchmark for Windows automation agents
• OSWorld – Cross-application task evaluation suite

💡 FAQ

⚠️ Disclaimer & License

Disclaimer: By using this software, you acknowledge and agree to the terms in DISCLAIMER.md.

License: This project is licensed under the MIT License.

Trademarks: Use of Microsoft trademarks follows Microsoft's Trademark Guidelines.

🚀 Ready to Get Started?

_{© Microsoft 2025 | UFO³ is an open-source research project}

_{⭐ Star us on GitHub | 🤝 Contribute | 📖 Read the docs | 💬 Join discussions}

From Single Agent to Digital Galaxy
UFO³ - Weaving the Future of Intelligent Automation