Skip to content

Conversation

JagjeevanAK
Copy link
Contributor

@JagjeevanAK JagjeevanAK commented Sep 22, 2025

Signed-off-by: Jagjeevan Kashid [email protected]

What

This PR implements a comprehensive Computer.tracing API that solve's the issue #299 by providing flexible session recording capabilities for all Computer usage patterns, not just ComputerAgent.

Key Features:

  1. Flexible Recording: Works with direct Computer API calls, ComputerAgent, and custom agent implementations.
  2. Configurable Options: Record screenshots, API calls, accessibility trees, and custom metadata based on needs.
  3. Standardized Output: Consistent trace format with JSON metadata and organized file structure.
  4. Multiple Use Cases: Supports training data collection, RPA debugging, human-in-the-loop workflows, and compliance recording.

Implementation Details:

  • ComputerTracing class with start/stop methods and configurable recording options.
  • TracingInterfaceWrapper that intercepts interface calls to record API interactions.
  • Integration with existing Computer class through new property.
  • Support for both zip archives and directory output formats.
  • Privacy-conscious design (clipboard content length only, optional screenshot recording)

Usage Example:

# Start tracing with custom configuration
await computer.tracing.start({
    'screenshots': True,
    'api_calls': True,
    'accessibility_tree': True,
    'metadata': True
})

# Perform operations (automatically recorded)
await computer.interface.left_click(100, 200)
await computer.tracing.add_metadata('task_type', 'automation')

# Save trace
trace_path = await computer.tracing.stop({'format': 'zip'})

Benefits:

  1. Enables training data collection with rich context
  2. Improves debugging capabilities for automation workflows
  3. Supports mixed human/agent session recording
  4. Provides audit trails for compliance requirements
  5. Works seamlessly with existing ComputerAgent trajectory recording
  6. The API is designed to be extensible for future enhancements like video recording and real-time streaming.

@JagjeevanAK
Copy link
Contributor Author

Hey @jamesmurdza and @f-trycua this pull request is Ready review.

@jamesmurdza
Copy link
Collaborator

@JagjeevanAK Thank you. We will merge after testing.

@JagjeevanAK
Copy link
Contributor Author

Hey @jamesmurdza no issues just ping me if there is any issue.

@jamesmurdza
Copy link
Collaborator

It looks really good, by the way!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants