A useful personal assistant needs to understand more than the sentence directly in front of it.
If I ask:
What is happening on my computer?
a normal chatbot has no grounded answer.
It can explain how I might check CPU usage, available memory, running processes, or the operating system. It can provide general troubleshooting advice.
But unless I manually give it the relevant information, it does not know what is happening on my computer.
SYNTHESIS needed a way to build that context.
After completing the first tool and permission tests, the next milestone was creating a context builder: a layer responsible for gathering relevant information and presenting it to the system in a structured form.
It was the first small step from conversation toward awareness.
Why the latest message is not enough
Language models usually receive a collection of text and generate a response based on that text.
If the input contains only:
What is happening on my computer?
then the model has the question but not the evidence required to answer it.
It might respond cautiously and ask for more information.
It might provide generic instructions.
Or, if the system is poorly designed, it might produce a confident answer based on assumptions.
None of those outcomes creates the experience I want from SYNTHESIS.
The assistant should be able to connect a request to approved, relevant observations from its environment.
That means constructing a richer input containing more than the latest message.
Building a context package
The first context builder assembled several categories of information:
- The user’s current request
- Relevant conversation history
- Basic operating-system information
- Current processor usage
- Memory usage and availability
- The number of running processes
- A limited view of processes currently using significant resources
The information was organised into a readable context rather than passed around as unrelated values.
When the test ran, the terminal printed a structured snapshot containing the question, conversation, system state, and process observations.
For the first time, SYNTHESIS had enough grounded information to answer a basic question about the environment in which it was running.
It was still a test.
But it connected language to reality.
Data is not understanding
It would be easy to overstate what this achieved.
SYNTHESIS did not truly understand my computer because it could read system statistics.
It had observations.
Understanding would require interpreting those observations correctly in relation to my request.
High memory usage might be relevant when an application is freezing. The same number may be unimportant during ordinary use.
A large number of processes does not automatically mean something is wrong. One application may appear several times because of how it separates its work.
Raw data can be accurate while the conclusion drawn from it is misleading.
That means context collection and reasoning must remain separate concerns.
The context builder gathers evidence.
The reasoning layer decides which evidence matters.
Neither should pretend to perform the other’s job.
More context is not always better
Once a system can gather information, the temptation is to gather everything.
More data appears to mean greater intelligence.
But sending every available observation into every request creates several problems.
Irrelevant information can distract the model. Large context packages consume more computing resources. Constant collection can slow the system down.
Most importantly, unnecessary context creates unnecessary privacy risk.
If I ask for the time, SYNTHESIS does not need a list of running applications.
If I ask why the computer feels slow, system resources may be relevant, but private document contents are not.
If I ask about a particular window, the assistant should not automatically inspect everything else visible on the machine.
Context should be selected according to the request.
A personal assistant does not become trustworthy by knowing everything it can possibly access.
It becomes trustworthy by knowing what it needs—and leaving the rest alone.
Observation requires permission too
Permission is often discussed in relation to actions.
Deleting, sending, purchasing, or changing something obviously requires control.
But observation can also be sensitive.
A screen may contain private conversations, financial information, passwords, photographs, personal documents, or information belonging to someone else.
A microphone can capture people who never agreed to interact with the assistant.
A process list can reveal which applications are running.
Memory can preserve information long after it stops being relevant.
Therefore, perception cannot become an invisible background ability.
The user needs to understand what SYNTHESIS is observing, why that context is required, and whether any part of it will be stored.
Read access is still access.
An assistant that never changes a file can still violate privacy through careless observation.
Context needs structure
A useful context builder should not simply dump information into one large paragraph.
Different observations come from different sources and carry different levels of reliability.
The user’s direct instruction is different from an inferred preference.
A current system measurement is different from an older memory.
A verified tool result is different from something mentioned casually in conversation.
If all of those are blended together, the model may treat them as equally reliable.
The first version used clear sections such as the current request, conversation, system state, and process information.
That structure makes the context easier to inspect during development. It also establishes the idea that SYNTHESIS should know where information came from.
Eventually, useful context needs more than content.
It needs provenance, relevance, freshness, and appropriate boundaries.
The terminal showed the truth
The context test ran inside the terminal and printed everything it had assembled.
Visually, it was not impressive.
There was no animated interface translating the system state into beautiful graphics. There was no voice explaining what was happening.
But the terminal made the process inspectable.
I could see whether the user request appeared correctly. I could check whether conversation history was included. I could inspect the system observations and notice whether anything unnecessary or incorrect had entered the context.
During development, that transparency matters more than presentation.
If SYNTHESIS eventually produces a polished answer, I still need a way to understand which context shaped it.
Otherwise, a confident response could hide a faulty observation underneath.
This is not where SYNTHESIS will live
Seeing another successful terminal test raised an obvious question for me:
Will I always need to communicate with SYNTHESIS like this?
No.
The terminal is the development environment, not the final experience.
I want to speak and write naturally. I want the assistant to understand ordinary language rather than require commands shaped around the code.
Eventually, interaction may happen through voice, a dedicated interface, shortcuts, or other forms.
But those interfaces should be added after the underlying context is trustworthy.
A voice does not make an assistant more aware.
It only makes the existing system easier to communicate with.
If the context underneath is irrelevant, invasive, or wrong, a natural voice will merely present the mistake more convincingly.
From system state toward perception
The first context builder handled structured system information.
That is only one form of perception.
A more capable assistant may eventually need to understand approved parts of the screen, spoken input, open applications, ongoing tasks, and changes occurring over time.
Each new source of context creates new possibilities.
Each also creates new questions:
What is relevant?
What requires permission?
What can remain temporary?
What deserves to become memory?
How does the assistant distinguish observation from interpretation?
How can the user correct it?
The context builder did not solve all of those problems.
It gave SYNTHESIS a place to begin solving them.
Awareness should remain accountable
The milestone was small but important.
SYNTHESIS could now assemble a limited picture of its environment instead of reasoning from the user’s sentence alone.
That made it more capable of answering grounded questions.
It also made the privacy problem more concrete.
An assistant with no context is limited.
An assistant with unlimited context is dangerous.
The challenge is building something between those extremes: a system that can notice what matters, explain where its information came from, and avoid observing more than the task requires.
The first context test taught SYNTHESIS how to gather facts about its environment.
The harder work is teaching it when those facts should—and should not—enter the conversation.