Testing agent platforms: Agno & Dify

development

Author

Werner Schiller

Published

October 10, 2026

I want a central home for my agents: several independent instances that can delegate to subagents over multiple levels, containerized, manageable from the browser, running on models behind my own LiteLLM router — and, if possible, without a new cloud dependency. I let the AI compare ten tools on paper, deployed two of them for real, and tore both down again. This is what that looked like, and why neither stayed.

🤔 Why ?

My agentic workflow runs on my workstation — which means the whole thing only works as long as that machine is up. I wanted to change exactly that: a web-based agent system that keeps working after I shut down my desktop.

Concrete goals:

Mirror my workflow centrally. The setup and workflow I usually run per project (see my AI-snippets) should live in one place and be able to pick up work on its own — automatically working through GitHub issues, for example. Not sure if it is a good idea yet, but i want at least to be able to test it.
Deployable via infrastructure as code. Ansible-friendly, reproducible — not a pet container I rebuild by hand every time.
Automating a development team. That is the umbrella goal. Like in a human team, specialization pays off: separate agents for architecture, UI, testing, planning. Bigger processes become easier to model, cheaper to run, and intermediate results stay reviewable.
Beyond coding. Once agents run headless, other use cases want the same platform — hourly or daily email triage, deciding which mails end up in a RAG, that sort of thing.
Curiosity and research. I am specializing toward the practical application of AI, and evaluations like this are exactly what I want to be able to run and document — including the questions a potential employer would ask: multiple users, authentication, auditing?

One word on “air-gapped”, since I use it loosely: I don’t mean a bunker setup. I mean provider independence and control over my data. Components should be interchangeable, so committing to one platform must not add a new vendor dependency — and with a model router in the middle, sensitive data can be kept inside the network on principle. That CAN be expanded to a real offline/air-gapped environment, but for me as a private person this is not affordable right now.

The requirements I compared against:

self-hosted and dockerized
multiple independent agents in parallel
subagents over multiple levels
usable via browser / GUI
models through my OpenAI-compatible router (LiteLLM)
workspaces — plus roles and auditing as a bonus

🔍 The shortlist

Ten tools made it onto the list, all self-hosted and dockerized. The condensed result:

Tool	Parallel agents	Subagents (multi-level)	GUI	LiteLLM	Workspaces
Trinity	✓	✓	✓	◐	◐
Dify	✓	◐	✓	✓	✓
Agno (AgentOS)	✓	◐	✓	✓	◐
Flowise	✓	✓	✓	✓	✗
n8n	✓	✓	✓	✓	◐
Langflow	✓	◐	✓	✓	◐
LangGraph + Studio	✓	✓	◐	◐	✗
Open WebUI	◐	✗	✓	✓	✓
AnythingLLM	◐	✗	✓	✓	✓
LibreChat	◐	✗	✓	✓	◐

✓ = covered · ◐ = partial / workaround needed · ✗ = missing

Three finalists emerged. Trinity has the best hosting model of all — one container per agent, delegation between agents, audit, a fleet console — but it only speaks Claude Code, Codex and Gemini CLI runtimes. No generic OpenAI-compatible endpoint, so no LiteLLM without an unofficial workaround. That kept it on the bench (more on that in the conclusion). The two I actually deployed: Agno and Dify.

👨‍🏭 How ?

My environment in one paragraph: a single-host Docker homelab. Traefik as reverse proxy with TLS on my .local domain, internal DNS, and the rule that nothing gets a published host port if possible — services are exposed through proxy labels only. All model traffic goes through LiteLLM (OpenAI-compatible, see my migration notes), a mix of routed cloud models and local ones via Ollama.

A note on how the deployment work happened, since it shaped the whole test: most commands were executed by one of my agents running on Open WebUI, with me supervising. Its shell access goes through my ssh-mcp server, which enforces a strict command allowlist — no git, no sed, no cp, no mkdir, no tee (i should propably relieve that restrictions one day). Safer for the box, but it means every deployment takes the scenic route: clones via a throwaway container, file transfers as Base64 through a container, edits via Python inside the target container. Roughly fifteen extra manual steps per deployment — and an interesting test of what a restricted agent can actually do.

Both candidates were then deployed the same way: docker compose, behind Traefik, LiteLLM as the only model backend.

🚧 Execution

Agno / AgentOS

Agno ships an agentos-docker template: a Compose stack (API + Postgres with pgvector). I kept the template, removed the port mapping, added proxy labels and pointed the model config at LiteLLM instead of OpenAI. The template’s default is a closed model behind OpenAI’s Responses API; the swap to any OpenAI-compatible endpoint is a one-liner:

OpenAILike(id=AGNO_MODEL, base_url=LITELLM_BASE_URL)

One integration quirk worth mentioning: embeddings. PgVector in Agno takes the embedder’s dimensions value to build its table schema — and sends the same parameter to the API. LiteLLM rejects it (UnsupportedParamsError). The fix: let the OpenAI client pick up the base URL from OPENAI_BASE_URL instead of setting base_url in code, so dimensions (the native size of the embedding model) stays a schema decision and never reaches the API. Exactly the kind of friction you only find by connecting two systems for real.

Verification came back green: health endpoints, Swagger, registered agents, the MCP endpoint — and the smoke test that actually matters. An agent run via REST, end to end through the whole chain:

POST /agents/{id}/runs   (surprise: multipart/form-data, not JSON)
→ RunCompleted → "PONG"
→ 2871 tokens, 3.1 s, routed through LiteLLM to the model behind it

The embedding chain for knowledge bases worked as well. Side note: Agno’s deployment docs literally say “hand this prompt to your coding agent” — which is pretty much what happened here.

Technically I was done at that point. Then I wanted the GUI — and ran into the wall that ended the test. The interesting part of Agno’s UI, the hosted control plane at os.agno.com, connects from the browser directly to the runtime (chat, traces, sessions stay between browser and my server — a genuinely nice property), but the app itself is a cloud product with a price list attached: the free tier registers localhost endpoints only, “live” endpoints behind a real URL are a Pro feature at $150/month per connection, and a self-hosted control plane is enterprise-only. There is an official open-source chat UI (agent-ui), but chat is all it is — no traces, no planning surface. More on why that matters in the conclusion.

So the stack I had just verified was running, and the interface to operate it sat behind a subscription. That settled it: torn down the same day.

Dify

Dify is a heavyweight: 16 containers (API, worker, web, sandbox, plugin daemon, …), a visual workflow builder, workspaces with roles, a marketplace. Deployment follows its own generator flow, then the usual homelab adjustments (no host ports, proxy labels).

Model setup went through the “OpenAI-API-compatible” provider — which is a plugin. That part is genuinely good once it runs: three chat models and one embedding model configured against LiteLLM, each one validated with a real call through the plugin daemon to the model behind it, defaults set. Model connectivity was never the problem.

Getting there was a small odyssey through an API that clearly does not want to be scripted:

passwords are base64-obfuscated on the way in,
the CSRF token lives in a __Host- cookie marked Secure — meaning my setup scripts had to run against the HTTPS endpoint through the proxy, with an unverified TLS context inside a container,
and the plugin marketplace search is behind a Cloudflare browser check (Error 1010) for everything that isn’t a browser. Installing with the exact identifier langgenius/openai_api_compatible:0.0.49@<hash> — dug out of a GitHub issue — worked fine. Unofficial, but effective.

I don’t get why applications nowadays so blatantly neglect IaC. I do NOT want to be forced to click my configuration, i want it written down, versionized, automated.

Two configuration bugs cost me the most time (details below), and then came the moment that made the whole “on-prem” claim wobble: the embedded marketplace tab is dead on every self-hosted installation, because marketplace.dify.ai serves

content-security-policy: frame-ancestors 'none'
x-frame-options: DENY

and its subpages allow cloud.dify.ai and nothing else. The console iframes the marketplace anyway — so the browser blocks it, everywhere, not just for me. Server-side plugin installs keep working; what’s broken is the browsing UI. Workarounds exist (own hostname, a reverse proxy that strips CSP headers, plus the uncertainty whether the page does postMessage origin checks) — but at that point the question is who is hosting whom.

And especially - is it even worth the effort.

🌩️ Problems

While installing

Independent of the tool, the same obstacles kept showing up:

The restricted command allowlist. No git, sed, cp, mkdir, tee; pipes partially blocked. Workarounds: clones via docker run with a git image, file transfers as Base64 through a container, edits via Python scripts inside containers. About fifteen extra steps per deployment.
But that one is on me and not really the fault of Dify or Agno
Internal DNS without wildcard entries. Every hostname has to exist explicitly — “just add a subdomain” doesn’t work that way.
That’s also something that’s the problem of my own setup.
Traefik needs to be told which network to use (traefik.docker.network), or it can’t find the containers at all. That’s not always true, but if you have multiple docker networks (some internal, at least one for traefik) that’s needed.
Compose overrides replace instead of extend. Compose overrides replace networks instead of extending them, and merge ports instead of replacing them. My proxy network lived in the override — which dropped the stack’s internal network from Dify’s nginx, resulting in a stubborn 502 because nginx couldn’t resolve the web container anymore. Fix: carry both networks explicitly, networks: [default, traefik]. For ports, without ports: !reset [] host ports end up published next to the proxy — docker compose config catches this.
This is specific for my approach with compose overrides.
Dify’s URL variables want bare origins. CONSOLE_API_URL and APP_API_URL may contain no path — Dify appends /console/api and /api itself. With paths in the config, every request went to …/console/api/console/api/… and the whole console greeted me with “An unexpected error occurred while rendering this component.”
Dify’s console API resists scripting (base64 passwords, __Host- CSRF cookies, HTTPS-only) — see above.
Cloudflare vs. automation. Marketplace search rejects non-browser clients (Error 1010), downloads run server-side, and the embedded UI is blocked by its own CSP. Don’t try to script any of it without a browser context.
Agno API quirks. The run endpoint expects multipart/form-data rather than JSON, and LiteLLM rejects the dimensions parameter Agno’s embedder sends.

Of the deployed systems

Agno — technically convincing, commercially closed off:

Pro	Con
Clean, small stack (2 containers), full REST + MCP	The UI that would make it operable is a hosted product: free tier = `localhost` endpoints only
Runtime is Apache-2.0, unlimited, no localhost checks in the code	“Live” connection = Pro at $150/month, self-hosted control plane = enterprise only
Agents as code in Python — good fit for me	Official self-hosted UI (`agent-ui`) is chat-only: no traces, no planning surface
Smoke tests green through the entire chain	Workspaces weak (apps and sessions, but no real team/workspace model)
	Dev mode runs unauthenticated; the JWT path expects a key from their cloud

Dify — runs well, and that is precisely not the problem:

Pro	Con
Mature platform: GUI, workspaces with roles, model management in the UI	Embedded marketplace dead on every self-hosted install — blocked by Dify’s own CSP
LiteLLM via “OpenAI-API-compatible” provider, validated end to end	On-prem, but not cloud-independent: marketplace, update checks, creator center
Big ecosystem of tools and templates	16 containers as a baseline footprint
	Subagent depth limited to workflow nesting
	More presentation than tangible content (see conclusion)

The shared pattern of both: “self-hosted” covers the runtime, while the parts that make the system operable — Agno’s dashboard, Dify’s marketplace (and propably more) — stay tied to the vendor’s cloud. For my goal of provider independence, that is not acceptable.

📋 Summary and Conclusion

What I wanted: a central, web-based platform for multiple specialized agents with subagent delegation — a mirror of my workflow, deployable as code, running on my own model router, independent of any vendor cloud, with users, roles and auditing as a look at what “real” deployments would need.

What I got: two technically working systems that both failed partially for non-technical reasons.

Agno is dropped over the dashboard. Code-first is not the problem — I actually prefer it. What I expect from such a system is a UI for strategic and architectural planning and decisions, plus an optional but central place where the detailed coding happens. Agno has exactly that — Studio, traces, approvals — but as a cloud subscription. Chat is free; the strategic surface is not. $150/month is not a homelab decision.
Dify never gave me Zugang. The German word fits better than “access” here: the feeling of standing next to the machinery, understanding and controlling what happens. Dify presents; I want to see the gears. As a developer I never got close to what my agents actually do — it is more presentation than concrete, tangible content. And for a platform selling on-prem, marketplace and update paths leaning on someone’s cloud is the wrong kind of dependency.

Status: more research needed. Trinity stays a candidate the moment model access becomes flexible — the LiteLLM path exists as tracked issues (an OpenCode runtime with custom OpenAI-compatible base URLs is “ready”, a pluggable LLM backend via LiteLLM is still “incubating”). Its isolation model, one container per agent, is still the best I have seen. n8n is planned for a very thorough look — I have run it before, and this time it will face the same test. Flowise will come along with it.

One caveat for other contexts: in an enterprise environment the conclusion might look completely different. Licenses are budget, not a blocker; no-code is a plus when the people building agents aren’t programmers; model lock-in is easier to accept when there is a vendor budget for it. But the requirements there are different ones, too, and i, personally, would strongly recommend avoiding vendor-lock-ins anyway.

The test stacks are completely gone — containers, volumes, images, directories. What remains are the patterns: how to expose a Compose stack through a reverse proxy without host ports, where the traps are when two config layers merge, and the certainty that the real hurdle is rarely the technology.

Disclaimer

I used AI to help me create this blog post, improve my language and structurize the content. The actual content still is my own.