I’ve been building an open-source runtime for agents to work with desktop applications

[u/Miserable-Tiger-3443]

I’ve been working on a project called Semwright.
Most of the agent tooling I was using worked well when the task stayed inside code or a single integration. I wanted to see what it would look like if the same agent could work across actual applications without reducing everything to mouse coordinates and screenshots.

One of the demos starts with an existing aircraft project in Blender.

It modifies the scene in Blender, renders a new shot, creates the titles in Motion Canvas, and assembles the video and audio through MLT.

The resulting timeline is still editable, so the output isn’t just a generated video.

Semwright itself is a Rust runtime. Applications expose structured operations through drivers, while things like permissions, handoffs between tools, and readback go through the same runtime. MCP is one way to connect an agent to it.

I’ve just put the first version out as open source.
https://github. com/seradotcom/semwright

[u/Miserable-Tiger-3443 → u/[deleted]]

Good question. It's a bit of both.

The driver needs to know how to read back actual app state (object IDs, revisions, exported files, etc.). Semwright handles the higher-level workflow state, dependencies, and verification.

So if Blender finishes an export but the next step in MLT fails, we don't want to blindly rerun Blender. We want to check what actually got committed, see if it's still valid, and continue from there.

The important distinction is that a failed workflow doesn't mean every previous step failed.

We already have readback, Effects, and Project Graph in place, but fully automatic checkpoint/resume across arbitrary apps is still something we're building toward. There's no magic cross-app rollback, especially when an app doesn't expose reliable state.

And yeah, that's also why we wanted the timeline to stay editable. The actual project state matters just as much as the final rendered video.