AzureFeeds

Project Overview

The system replicates a developer’s workflow by autonomously launching VS Code, configuring the environment, and generating code via GitHub Copilot Agent mode. This automation is ideal for scenarios where:

1. Codebase is not hosted on GitHub – GitHub Agents on github.com are not applicable.


2. GitHub Codespaces cannot access the code – VS Code in the browser is ruled out.


3. Desktop automation is required – Must work with VS Code desktop client on a local computer.

Key Innovation: CUA Model + PyAutoGUI Integration

The project demonstrates the synergy between CUA model and PyAutoGUI:

• PyAutoGUI: Executes desktop actions (launching apps, typing, clicking).


• CUA Model: Analyzes screenshots and determines next actions.


• Deterministic Outcomes: Enables autonomous detection and correction at each step.

Complete Autonomous Workflow

Phase 1: Environment Setup

1. Launch VS Code desktop application.


2. Navigate to the project workspace directory.


3. Maximize VS Code window.


4. Open integrated PowerShell terminal.

Phase 2: Intelligent Package Management

1. Read requirements.txt.


2. Execute pip install -r requirements.txt.


3. Monitor progress:
   
   
   
   • Capture screenshots.
   
   
   • CUA analyzes terminal output.
   
   
   • Detects “in_progress” vs “complete” states.
   
   
   • PyAutoGUI waits for CUA confirmation.
   
   
   
   

Phase 3: GitHub Copilot Integration

1. Open GitHub Copilot chat panel (Ctrl+Shift+I).


2. Switch to Agent mode.


3. Submit developer prompt.


4. Monitor status:
   
   
   
   • Capture screenshots every 3 seconds.
   
   
   • CUA checks “Keep” button status.
   
   
   • Accept generated code when ready.
   
   
   
   

CUA Model Decision Points

Package Installation Monitoring

• In Progress: Detects progress bars, downloads.


• Complete: Identifies empty prompt ready for input.

Code Generation Monitoring

• Disabled: Code generation ongoing.


• Enabled: Ready for acceptance

Automation Flow in Action

The following are the sequence of steps that are automated in the flow

1. Launch VS Code → Open project workspace.


2. Open terminal → Execute pip install.


3. CUA monitors → Detects completion.


4. Open Copilot → Activate Agent mode.


5. Submit prompt → Begin code generation.


6. Monitor “Keep” button → Detect enabled state.


7. Accept code → Automation complete.

Real-World Applications

Development Scenarios

• Clone repositories.


• Create branches.


• Generate code.


• Run tests and validations.

Regression Testing Use Cases

• Routine code generation.


• Test case creation.


• Documentation updates.


• Refactoring tasks.

Note: Requires a dedicated computer to run VS Code desktop. Use responsibly with safeguards and monitoring.

Refer to the GitHub Repo here. Refer to the video of this sample in action below:

Limitations:
With pyautogui, the App on which the automation needs to be done has to be on the foreground. It cannot choose a particular app from amongst other apps on the Desktop to monitor the visual state. There are other options to consider for that, which I will cover in another article later