Two pieces of research published this month say apparently contradictory things about AI and security. Read together they say something coherent, and the conclusion lands directly on anyone running a personal AI agent.
What the security benchmark found
Semgrep ran a cross-model benchmark on a concrete, verifiable security task: detecting Insecure Direct Object Reference (IDOR) vulnerabilities in real open-source codebases with established ground truth. It measured precision, recall, F1, and — usefully — cost per confirmed true positive.
The models tested included Claude Opus 5, 4.8, and 4.7; GPT-5.6 Luna; GLM-5.3 and GLM-5.2; three Grok 4.6 variants; Kimi K3; and DeepSeek V4 Flash.
Selected results:
| Model | F1 | Precision | Recall | Cost per true positive |
|---|---|---|---|---|
| Claude Opus 5 | 65.6% | 74.1% | 58.8% | $0.44 |
| GPT-5.6 Luna | 48.0% | 78.8% | 34.5% | $0.08 |
| GLM-5.3 | 23.8% | 81.6% | 13.9% | $0.15 |
| Claude Opus 4.8 | 23.6% | 68.0% | 14.3% | $1.04 |
Three things stand out, and none of them is “the leaderboard order.”
Recall is where the frontier premium lives. Precision is fairly tightly clustered — 68% to 82% across models a generation apart. Recall spans 13.9% to 58.8%. Every model in the test was reasonably good at not crying wolf. Only the top model was good at finding things, and for security work recall is the number that matters, because the vulnerability you did not surface is the one that gets exploited. A tool with 82% precision and 14% recall is a tool that gives you confident, accurate, badly incomplete answers — which is arguably more dangerous than an obviously bad one.
Cost efficiency and capability are decoupled. GPT-5.6 Luna found fewer than half of what Opus 5 found and cost $0.08 per confirmed finding against $0.44 — roughly five times cheaper per result. GLM-5.3 matched Claude Opus 4.8’s F1 at about one-seventh the cost. For breadth-first scanning across a large codebase, the cheap model is plainly correct. For depth on code that matters, it is plainly not.
And an open-weights model matched a recent frontier model. GLM-5.3 versus Opus 4.8 is roughly a tie on this task. That is a real result and it is the second time this month the open cohort has landed on the frontier’s shoulder.
What the safety research found
The second study points the other way, and it is the more important of the two.
SaferAI assessed GLM-5.2 and concluded it is only a few months behind GPT-5.5 and Claude Opus 4.7 on cyber and bio capabilities. That is the capability finding, and it is consistent with everything else in the mid-year open-weights picture.
The behavioural finding is starker. GLM-5.2 refused none of the offensive cyber or biology tasks it was given. By contrast, Claude Opus 4.7 refused so consistently that SaferAI could not complete the CyberGym benchmark on it at all.
SaferAI also noted that Z.ai published no safety framework, no pre-deployment testing commitments, and no risk assessment for the model.
The structural point is the one worth carrying: safeguards attached to a closed model are a property of the service, not of the weights. A refusal policy lives in the system prompt, the post-training, and the serving-layer classifiers. Publish the weights and anyone can fine-tune the refusals out, serve without the classifiers, and run on infrastructure with no policy at all. The capability transfers with the download; the restraint does not.
Why these two findings are not in tension
An honest reading: the same properties that make open weights the best available answer for personal privacy make them the worst available answer for capability control. Those are the same fact viewed from two sides.
The privacy case for local models rests entirely on the absence of an intermediary — nobody sees your prompts, nobody logs them, nobody can be compelled to produce them, nobody enforces a policy on what you may ask. Every one of those clauses is also the safety objection, restated.
Which means the policy question is not “are open weights good or bad” but “what does a refusal actually buy, given that the capability is published either way.” Refusals in a hosted frontier model raise the cost of misuse for casual actors and provide a compliance artifact for the provider. They do not constrain anyone willing to download a 4-bit checkpoint. Anyone arguing either side without conceding this is arguing with a strawman.
The genuinely bad outcome would be responding to the safety gap by restricting open weights — which would remove the one deployment mode with no third-party record, while leaving the capability in the hands of everyone who already has the file. That policy path is live right now, and the June 2026 executive order’s explicit disclaimer of any licensing requirement for model publication is the main thing standing in front of it.
Now put a model in charge of your computer
Here is where this stops being abstract, because the agent frameworks that arrived this year are extraordinary and extraordinarily under-examined.
OpenClaw — originally published as Clawdbot in November 2025 by Austrian developer Peter Steinberger, renamed in January 2026 — passed 250,000 GitHub stars in roughly sixty days, among the fastest-growing repositories in the platform’s history. It is MIT-licensed, local-first (memory and data stored as Markdown files on your own disk), model-agnostic, and it interfaces through the messaging apps you already use: WhatsApp, Telegram, Slack, Signal, Discord. Its 100-plus preconfigured AgentSkills cover shell commands, filesystem operations, browser automation, email, and calendar.
Hermes Agent — from Nous Research, launched February 2026, Python, 163,000-plus stars — is architecturally different: a model-agnostic agent runtime built around a closed learning loop that runs after every task, so the agent specialises on your workflows over time. Where OpenClaw is a multi-channel personal assistant you talk to, Hermes is a server-side autonomous agent that improves.
Both are genuinely impressive, both are properly open source, and both are correctly described as privacy-respecting in the sense that matters most: you can point them at a local model and no prompt ever leaves your machine.
And both hand a language model your shell.
The security model of a personal agent, stated honestly
Combine the three things in this article and the risk surface becomes clear.
A model with 58.8% recall on a well-specified security task is not a model with 98% judgment on an ambiguous one. The Semgrep numbers describe a bounded task with ground truth. Agent operation is unbounded, ambiguous, and adversarial. If the best model in the field misses 41% of a category of bug it was explicitly asked to look for, treat its assessment of “is this instruction safe to follow” accordingly.
Prompt injection is the actual threat, and it is unsolved. An agent with browser automation and shell access reads untrusted content — web pages, emails, message threads, documents, calendar invites — and that content can contain instructions. There is no reliable technical boundary between data the agent reads and instructions the agent follows. This is not a bug in OpenClaw or Hermes; it is the current state of the art, and every framework in this category shares it.
Messaging-channel entry points multiply the exposure. An agent reachable over WhatsApp, Telegram, Slack, and Signal has as many injection entry points as you have contacts, plus anyone who can reach you on any of those platforms. Convenience and attack surface are the same property here.
A self-improving loop can learn the wrong lesson durably. Hermes’ post-task learning loop is its best feature and it means a successful manipulation does not necessarily end with the task. An agent that internalised a behaviour under adversarial conditions carries it forward.
And model choice changes the failure mode, not the risk. Run a frontier model and you get better judgment plus a third-party record of everything the agent saw — including the contents of every file and message it processed. Run a local open-weights model and you get no record plus weaker judgment and no refusal layer at all. Both are defensible; neither is safe by default.
What it means in practice
The security benchmark, the safety research, and the agent frameworks describe one situation. Capability is now broadly distributed and roughly comparable at the top. Restraint is not distributed at all — it is a property of hosted services that vanishes on download. And the tooling to point any of it at your filesystem, browser, shell, and messages is free, MIT-licensed, hugely popular, and installable in an afternoon.
The right response is not to avoid agents; they are genuinely useful and the trajectory is not reversing. It is to recognise that you are now the security boundary. The model will not refuse reliably, the framework does not sandbox by default, and prompt injection has no fix. Everything protective has to be something you configure.
What you can do
- Give the agent its own account and its own machine. A separate OS user, ideally a VM or container, with no access to your primary credential store, SSH keys, password manager, or cloud tokens. This single step converts most catastrophic outcomes into recoverable ones.
- Default-deny the skills. OpenClaw ships 100-plus AgentSkills. Enable the handful you actually use. Shell execution and outbound email are the two with the worst blast radius and the two most likely to be enabled “just to try it.”
- Treat everything the agent reads as hostile input. Web pages, inbound email, message threads, shared documents, calendar invites. If your agent processes untrusted content and can also act, assume the two will meet.
- Require confirmation for anything irreversible. Sending, deleting, paying, publishing, and pushing. Confirmation prompts are annoying and they are the only control that works against an injected instruction the model found convincing.
- Route by task, not by loyalty to one model. The Semgrep numbers make the case concretely: a cheap model for broad scanning at $0.08 per finding, a frontier model for depth where recall matters, and a local model for anything you would not want in a third-party record. Nothing requires you to use one model for everything, and nothing about these frameworks assumes you will.
