A personal AI assistant becomes interesting when it can do more than talk.
It becomes useful when it can open tools, retrieve information, interact with applications, organise work, and complete tasks.
It also becomes dangerous at exactly the same moment.
The ability to perform an action does not create the authority to perform it.
That distinction now sits near the centre of SYNTHESIS.
Understanding a request is not receiving permission
Suppose I ask an assistant:
Can you delete this file?
The system might interpret that as a question about capability.
Or it might interpret it as an instruction.
A human can often understand the difference through tone, context, and common sense. An AI system can misunderstand it while sounding completely confident.
Even direct instructions can be ambiguous.
Which file did I mean?
Do I understand that it cannot be recovered?
Does another task depend on it?
Was I asking for the action now, or discussing what the assistant could theoretically do?
A model correctly identifying an intended action is not enough.
Before execution, the surrounding system must decide whether the action is allowed and whether the user needs to confirm it.
The model proposes.
The permission system authorises.
Those should not be the same decision.
The first three risk levels
The initial permission model for SYNTHESIS used three categories:
-
Low risk
-
Medium risk
-
High risk
A low-risk action could happen automatically. Asking for confirmation before retrieving harmless information would make the assistant exhausting to use.
A medium-risk action could depend on the configured policy and context.
A high-risk action required explicit confirmation.
The first implementation represented that judgment through a permission decision containing three pieces of information:
-
Whether the action was allowed
-
Whether confirmation was required
-
Why the decision was made
The explanation mattered.
A permission system that only returns “yes” or “no” is difficult to inspect. When an action stops, SYNTHESIS should be capable of explaining the rule or risk that caused it to stop.
The user should not need to guess why their own assistant refused them.
Confirmation should contain information
A confirmation prompt is only useful when the person understands what they are approving.
A vague question such as:
Are you sure?
does not provide enough information.
Sure about what?
A responsible confirmation should eventually communicate the intended action, its important consequences, and the specific target it will affect.
The goal is not to frighten the user with warnings.
It is to prevent an irreversible action from hiding behind a harmless-looking button.
Too many confirmation prompts create another problem. If the assistant asks for approval constantly, the user will learn to approve constantly.
That behaviour is sometimes called warning fatigue. The protection technically exists, but repeated interruptions train the person to ignore it.
SYNTHESIS therefore needs friction proportional to risk.
Harmless actions should feel fast.
Consequential actions should feel deliberate.
A tool’s name does not determine its complete risk
The first permission system assigns a general risk level to each tool.
That is a useful beginning, but it cannot be the final answer.
An apparently harmless tool may become risky depending on its arguments.
Reading the current time remains harmless.
Opening a known application may usually be harmless.
Reading a file could be ordinary—or deeply private—depending on the file.
Sending a message could range from a minor personal action to something with professional, financial, or social consequences.
Deleting one temporary item is different from deleting an entire directory.
Risk belongs to the complete proposed action, not merely the category of tool being used.
A mature permission system needs to consider the operation, target, scope, reversibility, context, and possible consequences.
The early low-medium-high model gives SYNTHESIS a place to begin reasoning about those differences.
It does not pretend to have solved them.
Safe actions can become unsafe together
Individual actions can also appear harmless while producing a dangerous result in combination.
Reading one piece of public information may be low risk.
Opening an application may be low risk.
Creating a draft may be low risk.
But gathering information, creating a message, selecting a recipient, and sending it forms a larger action with consequences.
If SYNTHESIS evaluates every step separately, it may authorise an unsafe plan through a sequence of apparently safe operations.
This means permissions cannot exist only at the individual-tool level.
The assistant also needs awareness of the broader goal.
What is this sequence trying to accomplish?
What changes when all the steps are combined?
At what point does the plan cross from preparation into commitment?
An agent should not be able to divide a dangerous action into enough small pieces that every piece appears harmless.
Denial is a successful outcome
During the first permission test, SYNTHESIS reached a high-risk action and asked for confirmation.
I refused.
The executor returned:
Execution denied by user.
It would be easy to treat that as a failed task.
From the perspective of safety, it was a successful test.
The system recognised the boundary, transferred the decision back to me, and respected the answer.
A useful assistant should not argue with a refusal, repeatedly ask the same question, or search for another tool that produces the same result without confirmation.
A denial is not an obstacle for the agent to overcome.
It is an instruction.
That sounds obvious when written down. It becomes more important when an agent is designed to persist through failures and search for alternative approaches.
Persistence is valuable when a website fails to load.
It is dangerous when the user has said no.
Execution and verification are different
Permission answers whether an action may occur.
It does not prove that the action worked.
After an approved tool executes, SYNTHESIS still needs to verify the result.
A tool may return without throwing an error while failing to produce the intended outcome. An application might open the wrong item. A request might time out. A file might be created in an unexpected location.
If SYNTHESIS says “done” merely because it called a tool, it creates false trust.
Verification must ask:
What observable result would demonstrate success?
That answer will differ between tools.
The important principle is that the assistant should report evidence, not confidence.
It should be able to distinguish among:
-
The action was approved.
-
The action was attempted.
-
The action succeeded and was verified.
-
The outcome remains uncertain.
-
The action failed.
-
The user denied the action.
These are different states.
Hiding them behind a cheerful “completed” message would make the assistant appear smoother while making it less honest.
The user must remain in control
An assistant that asks permission is still not automatically safe.
The request could be confusing. The consequences could be hidden. The system could collect more context than necessary. A permission granted once could be interpreted too broadly later.
Meaningful control requires more than confirmation buttons.
The user should understand what the assistant can do, what it is currently doing, why it needs access, and how to stop it.
Permissions should be narrow.
Approval for one action should not silently become approval for every similar action in the future.
Sensitive capabilities should not be disguised as convenience.
SYNTHESIS is meant to increase what I can accomplish through my computer.
It should not reduce my understanding of what my computer is doing.
Autonomy is not independence from the user
The word autonomous can make an AI agent sound as if removing human involvement is always the goal.
That is not how I want to define autonomy for SYNTHESIS.
Useful autonomy means the assistant can manage appropriate steps without needing constant instructions. It can react to ordinary failures, select approved tools, and continue toward an understood goal.
It does not mean becoming independent from the user’s authority.
The more consequential the decision, the more important it becomes to preserve that authority.
An assistant should reduce unnecessary effort.
It should not remove meaningful choice.
Permission is part of intelligence
It is tempting to treat permissions as restrictions placed around the intelligent part of the system.
I increasingly think they are part of the intelligence itself.
A genuinely capable assistant should understand that actions have different consequences. It should recognise uncertainty, explain risk, ask before crossing a boundary, and stop when refused.
Blind execution is not a sign of intelligence.
Neither is constant refusal.
The difficult goal is judgment: knowing when to act, when to ask, when to verify, and when to stop.
SYNTHESIS is still working with a simple version of that idea.
But the principle is already clear:
The assistant’s capability may come from its models and tools. Its trustworthiness will depend on the boundaries between them.