What makes a good AI workflow?

Note

This is an automated translation of Was macht einen guten AI-Workflow aus? (Teil 1) which i wrote in german initially.

This is the first part of a multi-part series of blog posts about how AI and agentic coding are changing - or rather complementing - the previously “optimal workflow”. I believe that, fundamentally, not much has changed. The details, however, are quite interesting, and the weighting of the individual sections - both in terms of effort and of importance - is shifting.

🤔 Why?

AI is changing the way we developers think. The problem is that it’s all too easy to give in to the temptations of AI and equate what it eases for us in terms of transpiration (i.e. hard work, meaning the actual writing of code) with inspiration (i.e. the thinking you put into using AI) - and then hand the latter over to the AI as well. Coding? The AI can do that. So, architecture? The AI can do that too - surprisingly well. Testing! The AI can do that, and the tedious drudgery is finally over. Documentation perhaps? The AI can do that - often considerably better than the human. Defining acceptance criteria. The AI can do that too.

Essentially, the AI can handle pretty much every step of software development. But should it? Or does the classic saying apply once again: “Just because you could doesn’t mean you should.”

Personally, I’m actually of the opinion that the AI should take over as much as possible. But - with one caveat.

👨‍🏭 How?

The question is - what can the human do that the AI can’t (yet)?

The answer is simple: break rules. Including their own. Object, and generally just question everything.

I’m also quite convinced that the AI won’t be able to do this for a long time, because the AI lacks the drive - the motivation - the contrarian spirit.

AI, on the other hand, is very good at following rules. That’s what it was trained for. AI is also excellent at programming - as one of the few things where the verification of the result is absolutely automatable during training, programming itself is also one of the first things the AI will more or less perfect.

So how can we turn these circumstances to our advantage and apply them in a way that squeezes the maximum productivity out of them?

Even with a perfect workflow, we therefore need a human-in-the-loop. But - and that’s the crucial question - where? Where is the sweet spot, and what do you need to take into account?

Psychology plays a role here too - if a human only gets asked a few questions every half hour or so, they certainly won’t engage with the topic and the problem as deeply as if they had worked on it for that entire hour. All the more so when - in order to use that time sensibly while the AI works in the background - those questions come from all sorts of different projects and thus have a completely different context. I don’t have an answer to these questions either, but I will try to shed some light on a few rather technical circumstances.

Another problem is the speed at which the AI world is currently turning. One example: no sooner had I arrived at a fairly satisfactory workflow (here and here and here) that worked quite reliably (with the model selection needing constant adjustment, but I’ve solved that in the meantime too - there will probably be a post on that later) - along comes TypeSafe AI with Jev.

With Jev - or with decision models in general - completely new possibilities for workflows open up. I’m planning to rework my workflow soon, using Jev or a similar decision model. There will be a post about that as well.

Or LiteLLM - which I use for routing¹ - and which, seemingly every week, adds a big new feature that actually needs exploring and putting to use. After all, I’m not doing all this just for fun, but also to broaden my skillset.

We are still a long way from the end of AI tooling, and that’s why it’s important to think about the fundamentals - of course, that’s only possible with the knowledge we have now, but if we can accept that our view of things today may soon change again, we can already go quite a few meters.

🚧 Execution

If we approach the topic systematically, we first have to answer a few questions:

1. Which spots/steps in development can we even identify?
2. At which spots do we MUST deploy the human, at a minimum?
3. At which spots do we MUST deploy the AI, at a minimum?
4. Which spots can we identify where it doesn’t matter whether the human or the machine does it?

After that, we can plug the answers from 2-4 into the answer from 1 and at least have a first approximation.

Which spots/steps in development can we identify?

Approaches to answering this question we’ve actually already seen in classic development. When it comes to translating the customer’s requirements - which are often vague, contradictory and incomplete - into concrete tasks for development, that’s a very similar process. Only now we need to generalize it and apply it to the entire development process.

What are the prerequisites?

• There is someone who knows what the outcome of the development process should be
  The translation and completion of the customer requirement has already been carried out, and a requirements specification - in whatever form - has been worked out.

• There is a human-machine interface that can perform the translation of the requirements specification into concrete technical parts
  The aforementioned requirements specification must be translatable into concrete technical steps. In my opinion, this is still the job of a human, because this step is particularly prone to overlooking implicit assumptions. I would also keep it logically separate from the requirements specification, even though the temptation to merge the two is great - but the target audience here is already a machine, unlike the requirements specification, so a separation makes sense.

• There is a human who can judge whether a solution is good or bad
  The human responsible for the acceptance of proposed solutions has the competence to judge that.
  This is quite a critical point, one that is perhaps sometimes overlooked by overly enthusiastic vibe coders.

• There is a flexible system with which we can technically implement the concrete solutions we find
  One or more AI harnesses with breakpoints (for human interaction), which can be linked together in such a way that the solution we’re working on here is actually implementable.
  Example: A harness that only works with models from a single vendor is, among other things for this reason, not a suitable tool. We want to have control - not give it away.

An attempt at an overview

This list is for understanding - concrete work steps follow later and are still partially summarized here.

Description	Actor	Summary
Complete capture of the task	Human	Capture the customer requirement and translate it into a requirements specification - just like before.
Analysis of the task and the environment	AI	Important: not just the task itself, but also things like runtime, external requirements, etc.
Loop: follow-up questions + answers after analysis	AI and human	To close gaps that remained open during the complete capture of the task.
Implementation plan, acceptance criteria and test plan	AI and human	Creation: AI, acceptance: human
Splitting the task into sensible milestones	AI	As small as possible, but large enough that sensible e2e or integration tests are feasible2 - architecture level
Verification of the completeness of the milestones	AI or human	Can be done by human or AI - depending on how suspicious or conscientious the responsible human is, or how much one trusts the model. As a general rule, though, it’s probably not a bad idea to check as a human, once more, whether the chosen path is the right one.
Splitting the milestones into implementable tasks	AI	As large as possible, while still allowing sensible unit tests to be written3 - technical planning level
Implementation	AI	Work through the tasks of the milestones one by one. After each task, run all unit tests. After each milestone, run all implemented integration tests.
Troubleshooting (loop)	AI, with escalation	On failed unit or integration tests: analysis and fix by the AI. After several failed attempts (or on suspicious behavior), escalate to the human.
Integration tests (artifact)	AI and pipeline	“Belong” to the milestones and are meant to check and ensure their functionality.4 Hence execution after every milestone.
Unit tests (artifact)	AI and pipeline	“Belong” to the tasks - classic unit tests
Verification AI	AI	Acceptance, at defined points, of implemented things by the AI. For tasks, that would be, e.g., a classic code review. For milestones, rather acceptance tests with functional verification, a check of the acceptance criteria, etc. - variable depending on the size of the task
Documentation (code, tasks, milestones)	AI	Creation by the AI in parallel with the implementation - review by the human.
Acceptance human	Human	Sensibly, this is the final acceptance and not something done earlier. This could be, for example, a test deployment with QA testing. At this point, in any case, the human is in charge
Release / deployment to production	Pipeline, approval: human	After successful final acceptance. Here too, the approval must come from humans - the responsibility always remains with a human.

One brief aside, generally about the use of AI in the development process: I personally have absolutely nothing against using AI very extensively. AI can boost productivity enormously. But it is a tool, and the responsibility for the result of a tool lies with the one who uses it. A developer who starts making sloppy code reviews of AI code and then at some point misses something very critical is definitely responsible for the bug, just as if they had made it themselves. “The AI did it, not my fault!” will be a common excuse that superiors should hopefully nip in the bud very quickly and very clearly.

Back to the topic: so let’s stick with my thesis from the introduction: fundamentally, not much has changed - we still translate vague requirements into tasks, have them implemented, tested and accept the result. Only the division of roles is a different one, and that’s exactly what now needs to be negotiated concretely. In the second part, I’ll answer questions 2-4, show where the breakpoints for the human make sense, and how the table above becomes a workflow you can actually run as-is. And why “MUST”, in that context, might be the most important word of the entire post.

Footnotes

1. Another topic for a blog post: a complete comparison of the model proxies / routers I’ve tested so far (LiteLLM, Bifrost, OmniRoute). Soon. I’m only human.

2. That is definitely just an opinion. One could also use other criteria; the only important thing is that it’s a manageable, comprehensible but defined size.

3. That’s difficult enough. The point here really is the unit test, which means the task can’t really get all that large.

4. Whether “integration test” is the correct term is certainly debatable.