Claude Did Poorly. Why?

Claude has been doing really well for me, until today. This is what went wrong, and why I think it went wrong.

(I used AI assistance writing this post — why)

The tool

Zepsec is a CVE management tool I’m developing for the Zephyr project, with the goal of it being usable elsewhere. It has a dashboard (two, as it turns out, which matters later) that shows the things that need to be done. Those things are a mix of processing steps implemented in Python and skill invocations. Up to now I’ve been cutting and pasting them into a shell or a Claude Code session.

That works, but clumsily. Many of the steps don’t show up on the dashboard until the previous one has completed, so I end up going back and forth between the dashboard and the terminal more than I’d like.

What I asked for

I wanted to automate this a bit, so I gave Mythos (probably overkill) this prompt:

I’d like to plan out an additional feature. Right now my daily steps are largely driven by the dashboard, but in a clumsy way, especially as many don’t show up there until the previous step is completed.

A typical flow might be:

• Do prepublish-audit on the pending GHSAs, publishing, or handling what is needed.
• file cve-publish for those published.
• cve reserve
• embargo notify
• people
• collaborators
• review-request
• notate
• backport-sync
• /zepsec-analyze [N]
• Process backports that need help (for this, I rely on notes for ones that look like they need an action on my part, but I’d really prefer to have an explicit marker on them, probably also with a note, saying this is on someone else to do something about it).

Many steps require a ghsa sync to make the next step possible. Some backport steps require a backport-sync.

Several things, I pass –yes to, because, for example, being asked about each email is tedious.

Please propose an idea of how to implement this.

What I got

Mythos made a plan, Opus implemented it, and it went wrong in several ways.

The most important was that it implemented the wrong list. There are two task boards in the app, which is something I need to reconcile, and it built on the one I don’t use.

It also really liked confirmation prompts. Each step was presented like this:

[6/9] notate — 1 advisory                        outward: github-advisory · batch
      then: refreshes mirror
  [r]un  [p]review  [s]kip  [q]uit >

That looks reasonable enough. But choosing run didn’t run anything. It ran the commands with a dry-run flag, then asked whether it was OK to proceed, and then each command prompted again on its own. My prompt had specifically said I pass --yes to things because being asked repeatedly is tedious, and the result asked me three times.

None of this was hard to fix. But I had to give a lot of detail about what I actually wanted, and it took about ten rounds of back and forth to get the code doing it.

Why it went wrong

Around the same time, I watched The Photoshop Clone That Exposed the Truth About Vibe Coding, and realized it described pretty much what had just happened.

The video is about a developer who used Claude to build open-source clones of Photoshop, Word, Excel and several others in about a week. The headline version is that he “one-shotted Photoshop.” The video argues that this is wrong. The apps that work best were built under heavy automated testing: the Photoshop clone has over 1,700 tests, and was checked by round-tripping large collections of real files. The model runs the tests, reads the failures, fixes the code, and repeats, thousands of times. Cloning is easy mode, because the answer key already exists: documented file formats, and real applications to compare against. Where the answer key is missing, the clones wobble. In the video’s words, “a test that nobody wrote is invisible.”

Most of what I use Claude for has an answer key. My embedded work, most of the security work, Zephyr itself: there is a well-defined notion of what “correct” is, or at least a way to tell a right answer from a wrong one. Tests pass or they don’t. The bootloader accepts the image or it doesn’t. That is something an agent can converge toward.

This task didn’t have that. There was no correct workflow runner waiting to be found, only the one I had in my head, and my prompt described the steps without describing the shape of the thing. Which task board? How much confirmation, and where? What should happen after a step finishes? Claude had to guess, and it guessed the way a careful engineer would guess about a tool that sends email and modifies public advisories: be cautious, show a preview, ask before doing anything. That isn’t an unreasonable instinct. It just wasn’t what I wanted.

The end result still works. It cost far more of my attention than the tasks where Claude has something definite to aim at.

What I learned

When I ask for a task, I should first ask myself whether I have a good idea that the model has a clearly defined “correct” solution to work toward.

When that isn’t the case, I don’t have to just go make the prompt better. I can iterate with Claude on the design first, and only then have it implement. That design conversation is how the answer key gets written. But I have to ask it to do this explicitly. Otherwise it just goes off and implements something.