From Fake Empathy to Real Authority
What a 60-year-old chatbot can teach us about enterprise AI
In 1966, decades before ChatGPT, MIT computer scientist Joseph Weizenbaum introduced a program that allowed people to conduct a typed conversation with a computer.
He called it ELIZA.
Its best-known script, DOCTOR, imitated a psychotherapist using methods associated with Carl Rogers. Rogerian therapy relied heavily on reflection rather than diagnosis, which made it unusually easy to simulate. Tell ELIZA, “I am unhappy,” and it might ask why you were unhappy. Mention your mother, and it could invite you to say more about your family. Much of the conversation consisted of identifying keywords and transforming fragments of the user’s own statements into questions.
There was no model of the user, no comprehension of the subject, and no emotional state behind the responses. ELIZA was a text-processing system whose apparent personality came from the script it was running.
Users still disclosed intimate details. They inferred concern from follow-up questions and empathy from verbal reflection. Some asked to be left alone with the program. Weizenbaum was disturbed by how readily people attributed understanding and emotional presence to software that possessed neither.
The phenomenon became known as the ELIZA effect: our tendency to infer intelligence, intention, and inner experience from a machine’s ability to produce an appropriate social response.
ELIZA’s original source listing disappeared from public view for decades before researchers found it in Weizenbaum’s MIT archives in 2021. They later reconstructed the software environment around it and, in late 2024, ran the original implementation on a restored version of MIT’s Compatible Time-Sharing System operating through an emulated IBM 7094. Their restoration work, published in January 2025, exposed the actual machinery behind one of computing’s earliest performed personalities.
By current standards, that machinery appears almost comically modest. Modern language models can synthesize documents, maintain conversational context, retrieve corporate information, generate plans, use software tools, and initiate actions. Their capabilities bear little technical resemblance to ELIZA’s pattern-matching routines.
The human vulnerability ELIZA revealed has survived every architectural improvement.
It has acquired an expense account and access to the production environment.
ELIZA showed how easily linguistic responsiveness could be mistaken for understanding. Modern systems create more consequential errors. Fluent explanations can be mistaken for competent judgment. Confident language can be mistaken for verified knowledge.
A coherent conversational voice can be mistaken for a coherent decision-making entity.
These errors become dangerous when the system can do more than continue a conversation. An enterprise agent may approve a refund, modify a forecast, interpret a policy, contact a customer, update a system of record, initiate a payment, or delegate work to another agent. Each action may arrive wrapped in a polished explanation that makes the process appear more deliberate than it was.
The interface presents a single speaker with a stable voice, even when the response emerged from a collection of models, prompts, retrieved documents, policy rules, memory systems, and tool calls. Users encounter a personality. The organization has deployed a software supply chain.
ELIZA could make someone feel understood.
An enterprise agent can make an organization believe that a decision has been examined, authorized, and responsibly made.
The User Completed the Illusion
ELIZA’s apparent intelligence did not reside entirely inside the program. The user supplied a significant portion of it.
The software generated a response based on textual patterns. The person interpreted that response as part of a continuous exchange between two minds. They supplied intention, emotional continuity, and charitable interpretation. When ELIZA produced an awkward question, the user often repaired it mentally rather than treating it as evidence that the system had lost the thread.
The machine supplied words. The user supplied the mind behind them.
Modern language models produce vastly richer output, so the comparison should not be stretched into a claim that nothing has changed. A system capable of writing software, interpreting images, querying databases, and coordinating tools is not merely a more elaborate collection of ELIZA scripts.
The psychological mechanism, however, remains active.
Users still fill gaps in reasoning. They overlook contradictions when the overall response sounds plausible. They infer durable understanding from temporary context. They treat a consistent tone as evidence of a consistent worldview. When a response is nearly correct, they often complete the argument themselves and credit the system with the finished result.
This interpretive labor helps explain why AI demonstrations regularly look better than production deployments.
A skilled demonstrator knows what the system is supposed to produce. They phrase the request in familiar language, provide the right context, reformulate weak prompts, ignore unhelpful branches, and steer the conversation back toward a successful result. The system receives credit for an outcome partially constructed by a cooperative operator.
Production environments contain fewer cooperative interpreters.
Customers describe problems badly. Employees omit important context. Policies conflict. Data arrives late. Exceptions hide inside footnotes. The person using the system may not know what a good answer should look like, which means they cannot quietly repair its mistakes.
The same conversational design that makes AI accessible can conceal this difference. It turns interaction into something that feels natural, then encourages the user to treat naturalness as evidence of reliability.
That is manageable when the output is a draft.
It becomes a governance issue when the output becomes a decision.
Anthropomorphic Trust Meets Operational Access
Enterprise agents combine two properties that previously lived in separate systems.
The first is social fluency. The system can explain itself, respond to objections, remember earlier instructions, and maintain a recognizable tone.
The second is operational access. The system can retrieve records, call APIs, change workflows, send messages, and initiate transactions.
Together, those properties create a new form of institutional confusion. The system sounds as though it understands the situation, then acts as though it possesses the authority to resolve it.
A reliable enterprise design must separate four questions:
Can the system produce an answer?
Is it competent to make this decision?
Is it authorized to perform this action?
Who is accountable for the consequence?
Conversational interfaces tend to collapse all four because the same voice appears to answer them.
Consider a customer-service agent handling a disputed refund.
The system may be able to produce a plausible recommendation by summarizing the customer’s history and quoting the refund policy. That establishes an ability to answer.
The dispute may involve an unusual contractual exception, regulatory requirement, or prior commitment from a sales executive. Correctly resolving it requires competence within that specific decision domain.
Even a competent recommendation does not establish permission to transfer funds. Authorization depends on the amount, customer status, business unit, geography, and current approval rules.
The refund also needs an accountable owner. Someone must answer for the outcome if the system misread the contract, applied the wrong policy, or sent money to the wrong account.
A polished explanation can make these distinctions disappear. The agent may state that it reviewed the customer history, applied the policy, and approved the refund. The sentence sounds like one continuous act of judgment, even though it contains separate claims about information, interpretation, authority, and responsibility.
This is the enterprise version of the ELIZA effect. ELIZA’s users interpreted reflective language as emotional understanding. Organizations may interpret explanatory language as evidence that analysis, competence, authorization, and accountability arrived together.
They did not.
The ability to explain an action says little about whether the system should have taken it.
The distinction becomes even more important in multi-agent systems. One agent may interpret a request, another may retrieve information, a third may execute the transaction, and a fourth may summarize what happened. The final response can sound like the work of a single responsible actor even though authority moved across several systems.
Delegation adds another risk. An agent with permission to perform one task may ask another agent to complete part of it. That delegated agent may have different tools, data access, or operating constraints. Without explicit controls, authority can expand simply because work crossed a system boundary.
Delegation should preserve or narrow authority. It should never enlarge authority merely because one agent asked another for help.
The conversational layer hides these boundaries remarkably well. A user sees one thread and one apparent personality. Behind it may sit a small bureaucracy of models, retrieval systems, connectors, policies, and credentials.
Bureaucracies usually produce paperwork.
This one produces prose.
Designing Against False Authority
A disclaimer that “AI may make mistakes” does not solve this problem. It describes a general possibility while revealing almost nothing about the decision in front of the user.
The system must expose the operational boundaries that conversational fluency conceals.
Each of the four questions requires a different control.
Ability to Answer Requires Evidence
A system that produces an answer should reveal what it used to construct that answer.
That includes the source documents, the age of the information, relevant missing data, and any material uncertainty. A citation alone may not be sufficient. The user may need to know whether the agent relied on an approved policy, an outdated memo, a customer comment, or a model-generated summary of all three.
Evidence should also travel with the decision. If an agent changes a forecast or denies a claim, the supporting record should remain inspectable after the conversational session ends.
A polished paragraph is not an audit trail.
Competence to Decide Requires Scope
Access to information does not establish competence to interpret it.
An agent may retrieve a legal policy without being qualified to resolve an ambiguous legal exception. It may summarize a medical record without being appropriate for clinical judgment. It may calculate a financial variance without understanding whether the underlying accounting treatment is disputed.
Enterprise systems need explicit decision boundaries. The organization should define which classes of decisions an agent has been designed, tested, and approved to handle. The system should identify when a case falls outside that scope and escalate it without improvising its way through the exception.
Models are rewarded for continuing the conversation. Governance sometimes requires them to stop.
Competence should therefore be defined at the decision level, not inferred from general model capability. A model that performs well across broad benchmarks may still be unsuitable for a narrow workflow with unusual policies, sparse data, or asymmetric consequences.
Permission to Act Requires Explicit Authorization
An agent’s apparent intelligence should have no bearing on what it is allowed to do.
Authorization should attach to the proposed action, the current context, and the identity under which the action will occur. A system may be permitted to draft a customer response but prohibited from sending it. It may recommend a refund while requiring approval above a threshold. It may update one field in a system of record while remaining unable to alter payment instructions.
These controls need to operate at runtime. Static access granted during deployment cannot account for every transaction, customer, geography, amount, or regulatory condition an agent may later encounter.
The interface should show the action being proposed, the authority under which it will occur, any relevant limits, and whether human approval is required.
Permission should become narrower as work moves through a delegation chain. An agent assigned to investigate a claim does not automatically inherit the right to approve a payment. A downstream agent should receive only the context and authority required for its specific task.
The agent may sound confident.
The authorization system should remain unimpressed.
Accountability Requires Ownership
Every consequential action needs an identifiable human or organizational owner.
That does not require a person to approve every low-risk transaction manually. It requires the enterprise to decide, in advance, who owns the policy, who accepts the operating risk, who reviews failures, and who can suspend the system.
Accountability also requires reconstruction. The organization should be able to determine which model was used, which instructions were active, what information was retrieved, which tools were called, what permissions were exercised, where approval occurred, and how the final action was recorded.
Without that history, accountability degrades into an argument between teams after something goes wrong.
The application team blames the model. The model team blames the data. The data team blames the policy. The policy owner discovers that nobody told them an agent had started interpreting it.
A conversational transcript may help explain what the system said. It does not necessarily reveal why the system was allowed to act.
The Interface Is Part of the Control Environment
Enterprise AI governance is often treated as a back-office concern: policies, review boards, risk classifications, audit procedures, and access controls.
The conversational interface belongs inside that control environment.
A chat window does more than collect instructions. It shapes how users understand the system. A human name, a friendly avatar, a confident tone, and first-person explanations can make a collection of technical components feel like a stable colleague with judgment and agency.
That perception changes behavior. Users scrutinize a database field differently from a paragraph that says, “I reviewed the evidence and concluded that the claim should be denied.”
The second formulation sounds as though the decision has already acquired an owner.
Interfaces should make system boundaries visible at the moment they matter. They should distinguish recommendations from decisions, proposed actions from completed actions, retrieved facts from model inferences, and machine execution from human approval.
They should also connect these distinctions to the four underlying questions.
Evidence helps the user assess whether the system can support its answer. Scope clarifies whether the system is competent to decide. Permission displays whether it may act. Ownership identifies who remains accountable.
This does not require turning every interaction into a compliance dashboard. It requires displaying the information necessary to prevent conversational confidence from becoming institutional authority by accident.
The design challenge is not to make AI less usable. It is to prevent usability from concealing responsibility.
ELIZA succeeded because its interface allowed users to imagine more machinery than the program contained. Modern enterprise AI presents the reverse problem. The interface makes an extensive and fragmented technical system appear simpler, more unified, and more responsible than it is.
The user sees a single intelligence.
The enterprise needs to govern the actual system.
The Old Error, Now With Consequences
Capability does not erase the need for distinctions. It makes them more important.
A system can produce a persuasive answer without being competent to decide. It can be competent to decide without being authorized to act. It can be authorized to act without resolving who owns the outcome.
Enterprise AI governance begins by refusing to treat those as the same question.
ELIZA exposed our willingness to mistake responsiveness for understanding. Agentic AI raises the stakes by allowing apparent understanding to exercise authority.