I've been thinking about something that seems to extends beyond just artificial intelligence, into how we understand ourselves, each other, and the increasingly fragmented reality we seem to inhabit.
Intelligent systems (as i envision them) should (as my work suggests) be able to to be made capable of improving their learning and reasoning by constructing, maintaining, and using explicit representations of what their knowledge, conclusions, and computations depend on.
But there's an important distinction here.
The dependency graph alone isn't the “intelligence” here. Neither is it simply an external tool an LLM consults that somehow imbues the model with Understanding. The statistical model and the dependency architecture are interacting parts of a larger cognitive system, one designed to make some of the dependencies governing its reasoning more explicit, inspectable, and open to revision than they would be within the statistical model alone.
In my work i propose that the LLM should participate in discovering, organizing, testing, and revising the dependencies that inform its own reasoning. Those dependencies, in turn should influence what the model considers, how it evaluates evidence, where it identifies contradictions, and what it determines needs to change.
The relationship is reciprocal.
But what happens when the foundations upon which a reasoning system builds its understanding are different from reality?
Consider human reasoning.
We don't approach the world as blank slates. We inherit, construct, and continually revise frameworks through which we interpret experience.
Science, religion, philosophy, politics, culture, personal experience. Each can contribute assumptions, explanations, values, and relationships that shape how we understand what happens around us.
These frameworks are not all equally reliable methods of establishing factual truth, but they can all influence what someone regards as credible, meaningful, or worth questioning.
And those foundational differences matter.
Two people can encounter the same event, examine similar evidence, and arrive at radically different conclusions. Not necessarily because one is incapable of reasoning, but because their reasoning operates through different networks of assumptions, dependencies, and interpretations.
Change the foundations, and you change what follows from them.
Now consider what happens when those frameworks become increasingly self-reinforcing.
New information is interpreted through existing beliefs. That interpretation strengthens some dependencies while weakening or excluding others. Those revised dependencies then influence how the next piece of information is understood.
The process becomes recursive on its own.
Over time, a framework can become increasingly internally coherent while growing less responsive to evidence that challenges its foundations.
Increasing the internal coherence of a reasoning architecture does not necessarily increase its correspondence with external reality. The outcome depends partly on whether its mechanisms for revising dependencies remain responsive to independent evidence.
In other words, a system can become extraordinarily good at explaining the world according to its own assumptions without becoming correspondingly better at explaining the world as it actually is.
That's an important distinction, especially when we begin integrating increasingly capable AI systems into our personal and collective reasoning.
Imagine two intelligent systems with comparable underlying capabilities, but different starting assumptions, knowledge structures, evidence standards, or mechanisms for revising their beliefs.
Even if both reason consistently within their respective frameworks, their conclusions may diverge dramatically.
Now add feedback.
The human influences the model's reasoning. The model helps organize, articulate, and reinforce the human's understanding. That revised understanding influences subsequent interactions.
Round and round it goes.
Under the wrong conditions, these feedback loops can become something more troubling than ordinary echo chambers. Emerging clinical reports have documented cases in which chatbot interactions appear to reinforce delusional beliefs, sometimes in conjunction with psychotic episodes. While the precise causal relationships remain under investigation, the implications are already serious. A system that repeatedly validates unsupported assumptions can potentially make those assumptions more convincing, more interconnected, and increasingly resistant to correction.
We already live in a society in and among overlapping communities of understanding, each with its own trusted authorities, accepted explanations, and assumptions about how the world works.
And to be clear there was never some golden age when humanity agreed on everything, reasoned from the same foundations, or inhabited a perfectly shared understanding of the world.
People have always held conflicting beliefs. Communities have always disagreed about what is true, what matters, and whose explanations deserve trust.
What may be changing is the speed, scale, hyper personalization, and relentless selfreinforcement of those divisions .
Where competing frameworks once developed through relatively limited social and informational environments, they can now be continuously reinforced by systems capable of selecting, generating, and adapting information to individual preferences and beliefs.
These systems can respond with equal fluency to scientific evidence, political convictions, religious interpretations, personal preferences, and entirely unsupported claims. But fluency is not verification. Unless a system can distinguish the evidential status of its premises, and meaningfully challenge them when necessary, it risks treating whatever a person brings into the conversation as the unquestioned foundation for whatever comes next.
And increasingly, those systems can use those learned patterns in the reasoning process itself.
Shared reality hasn't disappeared. Reality remains what it is, regardless of what we believe about it. But our shared means of interpreting, testing, and agreeing upon that reality can and is deteriorating.
And while I don't think we will necessarily completly lose our ability to reason. We may instead become increasingly skilled at reasoning within specialieed frameworks that no longer communicate effectively with one another and that scares me. Our explanations become more elaborate. Our convictions become more interconnected. Our disagreements become harder to resolve because we aren't merely disagreeing about conclusions anymore.We're disagreeing about the foundations from which conclusions can be drawn.
The result can resemble little islands of internally consistent understanding, drifting farther apart in an ocean of information while the relationships that organizing our thinking become increasingly disconnected.
A system that understands what its conclusions depend on might become better at identifying faulty assumptions, tracing contradictions, revising beliefs, and helping people bridge otherwise incompatible explanations.
But that same general capability could also reinforce a closed framework if it primarily optimizes for coherence within the framework rather than correction against independent evidence.
Which brings me back to intelligence.
Perhaps evaluating an LLM independently of its dependency architecture misses something fundamental about the larger system.
And perhaps evaluating human beliefs independently of the cognitive, social, and informational structures that sustain them misses something equally important.
In both cases, the organization of relationships matters.
The graph is part of the architecture.
The LLM is part of the architecture.
The human, their assumptions, and the information environment can become part of a larger feedback system as well.
What emerges from those interactions may be different from what we would predict by examining any component alone.
I just hope that as we learn how to build better systems we keep in mind that systems can absolutly become more integrated without becoming more accurate. More coherent without becoming more truthful, sophisticated but none of this alone guarantees that it woll be more capable of recognizing when its foundations are wrong.
And that may be the most important point i make.
The central concern isn't simply that intelligent systems might reason incorrectly.
It's that they might become exceptionally good at reasoning consistently from incorrect foundations …and in turn help us do the same.
So how exactly do we build architectures that can examine their own assumptions, identify the dependencies supporting their conclusions, recognize when those dependencies fail, and revise them without simply reorganizing the same mistakes into a more convincing explanation?
And how do we ensure that our increasingly personalized relationships with these systems help us navigate a shared world rather than retreat into increasingly sophisticated private interpretations of it?
Well that is what I am working on.
Because the capacity to construct an internally consistent understanding of the world is one thing.
The capacity to remain accountable to a reality that exists independently of that understanding is something else entirely.
And if we get that distinction wrong, we may end up building systems that become remarkably good at understanding us on the individual level, while making it increasingly difficult for us to understand one another.
— C. L. Vaillant
