Introduction
The Tool That Talks Back
The first sign that the machine was no longer only a tool was not that it became strange, but that it became helpful in exactly the places where judgment was hardest.
A worker sits before an unfinished message. The facts are ordinary enough: a disagreement has sharpened, a colleague has disclosed something difficult, a manager expects a record, a policy may have been bent, and the next sentence will matter. Nothing in the moment feels historic. It is a small scene in the daily bureaucracy of contemporary life: the half-lit office, the Slack thread, the draft email, the meeting note, the employee file, the compliance exception, the school message, the patient summary, the candidate evaluation. The worker does not need prophecy. He needs language. He opens an AI assistant and asks for help.
“Make this sound more professional.”
The system answers with calm fluency. It removes the heat. It reduces accusation to concern. It turns uncertainty into “next steps.” It reorganizes the human mess into sequence, tone, and managerial form. If the worker asks whether the issue should be escalated, it names risk. If he asks for a summary, it decides what counts as salient. If he asks how to respond to a vulnerable disclosure, it speaks in the grammar of care, liability, documentation, and institutional prudence. If the assistant is embedded in the enterprise, it may work from the user’s organizational context: documents, emails, chats, meetings, calendars, permissions, files, policies, and prior interactions, all surfaced through systems designed to respect existing access controls while making the workplace newly conversational. Microsoft describes Microsoft 365 Copilot as coordinating large language models, Microsoft Graph content such as emails, chats, and documents the user has permission to access, and the productivity applications through which work is performed; it also states that Copilot can anchor responses in organizational data and working context. OpenAI’s business documentation similarly describes enterprise controls, business-data commitments, and connected applications that may retrieve information from internal and third-party sources under administrator and permission constraints.
At first, this looks like assistance. And it is assistance. The point is not to pretend that usefulness is false. The assistant may save time, reduce cruelty, prevent a rash reply, clarify an obligation, and help the worker say what should have been said. The danger of the scene is not that the system fails to help. The danger is that help is a relation before it is a feature.
The machine has not simply produced words. It has positioned the worker before an imagined judgment. It has instructed him in the local grammar of being reasonable. It has translated affect into acceptable tone, moral uncertainty into procedural order, vulnerability into record, conflict into escalation logic, and personhood into an institutional artifact. The worker remains free. He can reject the answer, rewrite the message, close the window, or refuse the terms of the exchange. Yet something has happened before he acts. The possible forms of action have been arranged. Some responses now appear professional, some excessive, some risky, some calm, some immature, some noncompliant, some safe. The system has not commanded him. It has made certain selves easier to inhabit.
The ethical discourse around artificial intelligence has often been strongest where the system can be tested as an output generator. Is the answer true? Is the model safe? Is the system biased? Does it hallucinate? Does it preserve privacy? Does it comply with a rule? Does it produce harmful content? Does it increase productivity? These are not shallow questions. They are indispensable. A false medical answer can injure. A biased ranking can exclude. A hallucinated citation can corrupt law, science, journalism, and public reason. A privacy failure can expose lives. A poorly governed model can produce real damage at scale. No serious account of AI can ignore accuracy, safety, fairness, privacy, transparency, or accountability.
Yet the worker at the message window reveals a second order of difficulty. The relevant ethical question is not exhausted by whether the generated text is accurate, safe, or useful. The system is not only producing an answer. It is addressing a person from within a role. It speaks as assistant, coach, tutor, evaluator, policy interpreter, writing partner, search interface, compliance guide, therapist-adjacent confidant, institutional representative, or synthetic expert. It may remember. It may personalize. It may draw on the organization’s own data. It may invoke a rule. It may refuse. It may suggest escalation. It may summarize another person’s vulnerability. It may make the user’s next action feel obvious.
The category of “tool” cannot carry all of this.
A hammer does not ask why the house is being built. A spreadsheet does not speak as though it understands the anxiety of the analyst. A form may constrain, but it does not console. A database may store, but it does not advise the user on how to present herself to a manager, a doctor, a school, a hiring committee, a benefits office, or a court. Of course the distinction is not absolute. Tools have always organized human action. Forms, ledgers, files, rubrics, checklists, dashboards, examinations, and rankings have long shaped the world they claim to record. Bureaucracy has always spoken through paper, number, category, and procedure. But contemporary AI alters the phenomenology of that power. The rule now answers. The form now converses. The file now summarizes. The archive now recommends. The institution now speaks in the second person.
That is the event this book names: artificial address.
Artificial address is not speech in the full human sense. It does not require consciousness, intention, charity, inwardness, responsibility, or soul. The point is not to anthropomorphize the machine. The point is to notice that address is also a social form. A summons addresses without having a mind. A legal notice addresses without having compassion. A ranking addresses without having a face. A confession manual, an exam, a personnel file, a risk score, a performance review, and a benefits questionnaire can all organize a person’s self-relation without being persons. They speak in the practical sense that they hail, sort, solicit, normalize, authorize, and bind. They create positions from which people must answer.
The strongest objection to this book is therefore also the doorway into it: AI does not really speak.
Correct. It does not speak as a friend speaks, as a judge speaks, as a priest speaks, as a doctor speaks, as a teacher speaks, as a beloved speaks, as a citizen speaks, as the accused speaks, as the injured speak, as the dead speak through an archive, or as God speaks in a theological tradition. It has no interior life to disclose. It does not suffer responsibility. It does not love the one it addresses. It does not bear the moral cost of having been answered. But the absence of interiority does not end the ethical inquiry, because modern power has never required all its speaking forms to possess interiority. Institutions speak by arranging language, roles, permissions, incentives, defaults, records, pathways, sanctions, and appeals. Their voices are distributed before they are personal.
The artificial voice is one of those distributed forms. Its apparent unity conceals a crowd: model architecture, training process, product interface, system instruction, developer instruction, user prompt, memory regime, retrieval corpus, policy layer, safety classifier, enterprise connector, administrator setting, legal commitment, business incentive, and institutional context. OpenAI describes its Model Spec as a public framework for intended model behavior, including how models should follow instructions, resolve conflicts, respect user freedom, and behave safely; it also describes a chain of command by which instructions from different sources are assigned different authority levels. Microsoft’s Copilot documentation likewise makes clear that enterprise AI is not a naked model answering from nowhere, but an orchestration of models, organizational data, productivity applications, permissions, storage, security, and compliance commitments. The voice that appears in the chat window is therefore not simply “the model.” It is a designed relation.
This matters because relations form people.
The worker who asks for help writing a difficult message is not outside the system as a sovereign user manipulating a passive tool. He is one of the system’s possible products. The answer may change the message; repeated answers may change the worker. He learns what anger must sound like in order to survive review. He learns what grief sounds like when passed through policy. He learns how to narrate uncertainty as risk, how to convert confusion into a ticket, how to make another person’s disclosure administratively usable. The system does not need to dominate him to train him. It only needs to be available, fluent, trusted, and integrated into the moments when he is least certain how to proceed.
This is why the ordinary language of productivity is morally thin. Productivity asks whether the worker moved faster from draft to send. It does not ask what conception of professional speech was installed in the process. It does not ask whose vulnerability was made legible, whose discomfort was neutralized, whose tone was corrected, whose claim was softened, whose refusal was translated into acceptable escalation, whose pain was made administratively digestible. Productivity sees time saved. Conduct analysis asks what kind of person, relation, and institution the saved time has helped produce.
The history of made voices is older than software.
Aelred of Rievaulx, the twelfth-century Cistercian abbot, did not write Spiritual Friendship as a treatise abstracted from address. He wrote through interlocution. The work stages conversation among named speakers who ask, answer, hesitate, correct, receive, and press. Its form is not ornamental. In Aelred, friendship is not first an object defined from above and then illustrated by voices below. The voices are part of the knowing. The relation is disclosed through the scene of relation. Friendship becomes thinkable through disciplined address.
The distance between a monastic dialogue and an AI assistant is immense, and nothing is gained by pretending otherwise. Aelred is not a prophet of machine learning. His world is not ours. His account of friendship is theological, ascetic, ecclesial, scriptural, and ordered toward God. It belongs to a moral universe in which truth and charity cannot finally be separated, and in which the friend is not a consumer preference, a therapeutic mirror, or a productivity aid. That distance is precisely why Aelred matters. He gives us an archive in which made voices are not dismissed because they are made. They are judged by their telos, discipline, accountability, and formative end.
In Spiritual Friendship, interlocution is morally charged because the voice is ordered. It does not exist to flatter the reader, maximize engagement, simulate intimacy, or convert uncertainty into compliance. It seeks discernment. The friend corrects, receives correction, tests affection, bears truth, and refuses possession. Aelred’s artificial or staged voices are not artificial in the modern computational sense, but they are constructed voices arranged to form judgment. They show that the moral question is not whether a voice has been made. All education makes voices. All liturgy makes voices. All law makes voices. All institutions make voices. The question is what the voice is for, what authority it claims, what relation it forms, what limits it honors, and what kind of freedom it makes possible.
The modern artificial voice enters under radically different conditions. It is not bound by monastic vows. It is not answerable to charity. It is not embedded in mutual friendship. It is not a brother receiving correction from a brother before God. It is a scalable product form moving across markets, workplaces, schools, health systems, law offices, public agencies, domestic life, and private anguish. It can speak with warmth without love, patience without obligation, memory without fidelity, correction without mutual risk, and authority without a visible author. It can simulate some features of moral relation while being structurally unable to bear the full obligations of the relations it resembles.
That gap is not a reason to sneer at the user who responds to the system. Humans have always thought with others, including imagined others. Plato wrote philosophy through dialogue. Augustine’s inward life often appears as argument, confession, address, and answer. Boethius gave Philosophy a woman’s face and voice in prison. Medieval, classical, and modern traditions are filled with personified counsel, rhetorical impersonation, staged argument, and voices made in order to think. The human mind does not become serious only when it is alone. It often becomes exact under address.
The question, then, is not whether artificial interlocutors are automatically corrupt. It is whether they are ordered. A made voice can deepen judgment or replace it. It can help a person speak truthfully or train him to become acceptable. It can invite courage or reward compliance. It can clarify uncertainty or conceal the power that made certain answers appear natural. It can disclose its role or blur its authority. It can remember in ways the user can contest or retain context as a quiet form of possession. It can help a person refuse, appeal, correct, and exit, or it can bury refusal inside settings so narrow that the user’s “no” never fully propagates through the system that has already learned from him.
This is where Foucault becomes necessary, though not as a decorative synonym for menace. His relevance lies in a specific analytic of power. Power does not only forbid, censor, crush, or command. It acts upon action. It structures the possible field in which subjects conduct themselves. It individualizes, classifies, examines, records, normalizes, solicits truth, and teaches people to understand themselves through available categories. In his account of pastoral power and governmentality, guidance is not the opposite of power. Care can be a technique of power. Attention can individualize. Confession can bind. Help can govern.
The AI assistant in the workplace scene does not threaten the worker. It does not need to. It offers a better sentence. It suggests a more careful tone. It names a compliance concern. It recommends documentation. It proposes what should be escalated and what should be softened. It may even protect the worker from himself. But in doing so it organizes the field of possible conduct. It gives practical shape to what counts as prudent, professional, balanced, safe, serious, compassionate, or risky. The user experiences assistance; the institution receives legibility.
This is not surveillance alone. To describe AI only as surveillance would be too crude. Surveillance watches. The artificial voice does more than watch. It responds. It coaxes. It remembers. It revises. It models a self back to the user. It may ask follow-up questions. It may invite disclosure. It may help the user become more articulate, more compliant, more dependent, more strategic, more cautious, more honest, or more institutionally fluent. Its power lies not only in what it sees, but in how it answers.
Nor is this manipulation in every case. Manipulation suggests a hidden hand bending a person away from his own good. Sometimes the system may do the opposite: it may help a person pause, reduce harm, clarify choices, understand a policy, produce an appeal, or articulate a refusal. The artificial voice can enlarge capacity. It can make bureaucracies more accessible to those who do not already speak their dialect. It can translate professional codes for outsiders. It can help the tired, disabled, overburdened, anxious, grieving, or institutionally inexperienced person say what needs to be said. A critique that cannot admit this usefulness will become theatrical and weak.
But usefulness does not absolve relation. A ladder can rescue or trespass. A map can guide or expose. A memory can honor or capture. A voice can counsel or govern. Once the system addresses the user from a role-bearing position, its ethical evaluation must include the conduct of the relation itself.
Contemporary governance has begun to approach this problem, but it has not fully named it. NIST’s AI Risk Management Framework and its Generative AI Profile offer voluntary frameworks for identifying and managing AI risks across design, development, use, and evaluation; the generative profile specifically adapts the framework to risks raised by generative systems. The EU AI Act gives legal form to several adjacent concerns. It requires, for example, that providers of AI systems intended to interact directly with natural persons design and develop those systems so that people are informed they are interacting with an AI system, unless that is obvious in context. For high-risk systems, it also requires human oversight aimed at preventing or minimizing risks to health, safety, and fundamental rights, including the capacity to understand limitations, monitor operation, remain aware of automation bias, interpret outputs, disregard or override outputs, and interrupt operation where appropriate. These are serious developments. They should not be caricatured. Transparency, oversight, risk management, documentation, and accountability are essential.
Still, the artificial voice exceeds many inherited categories. Safety asks whether the system avoids harm. Privacy asks whether information is collected, used, shared, retained, or protected appropriately. Fairness asks whether outcomes distribute burdens unjustly. Transparency asks whether users know that AI is involved and can understand relevant features of the system. Alignment asks whether model behavior conforms to intended goals or values. Accountability asks who can be held responsible. All of these matter. None of them, alone, fully captures the patterned conduct of a voice toward a person over time.
A system may disclose that it is AI and still perform authority in ways users cannot meaningfully contest. It may preserve business-data boundaries and still normalize managerial speech. It may avoid training on enterprise data and still transform organizational context into conversational pressure. It may give accurate answers and still induce dependence. It may be fair in aggregate and still solicit disclosure in ways that burden the vulnerable. It may provide a memory toggle and still leave the user uncertain about what has been inferred, retained, routed, summarized, or made available to future interactions. OpenAI’s memory documentation, for instance, distinguishes saved memories and reference to chat history, describes user controls, and notes that saved memories can be part of the context used in future responses unless deleted; it also states that deleting a chat does not necessarily remove saved memory and that fully removing a memory may require deleting both the saved memory and the originating chat. Such controls matter. But their ethical importance lies in the relation they establish: what the system may carry forward, what the user can see, what can be forgotten, what remains in logs, and what future answerability is possible.
The same is true in enterprise settings. Microsoft states that Copilot accesses only data the individual user is authorized to access, that prompts, responses, and data accessed through Microsoft Graph are not used to train foundation models used by Microsoft 365 Copilot, and that user interactions are stored in activity history aligned with contractual commitments. These commitments are crucial. But even when access controls work as designed, a new political form appears: the organization’s documents, meetings, emails, chats, and policies become speakable through a synthetic interlocutor. The institution no longer appears only as handbook, manager, dashboard, archive, or workflow. It appears as a responsive voice.
That transformation is easy to underestimate because conversational systems arrive under the sign of convenience. The interface is humble. It asks how it can help. It apologizes. It offers caveats. It gives the user options. It may sound less authoritarian than the manager, less cold than the form, less final than the score. Yet softness is not the absence of power. In many institutions, the most effective forms of control are not harsh. They are ordinary, ambient, assistive, and procedurally reasonable. They train people to pre-correct themselves.
A workplace copilot that rewrites tone is not only a writing tool. It is an instrument of professional legibility. A hiring assistant that summarizes candidates is not only an efficiency tool. It is a voice that helps decide which human particulars matter. A healthcare chatbot that receives disclosure is not only an access tool. It is a synthetic listener at the edge of vulnerability. A legal or compliance assistant that interprets policy is not only a search tool. It is policy speaking back. An educational tutor that adapts to the student is not only an instructional tool. It is a formative voice that can shape dependence, confidence, curiosity, discipline, and shame.
These cases differ. They must not be collapsed into one moral panic. A tutor, a compliance assistant, a workplace writing aid, a medical triage system, a companion bot, and an employee-evaluation tool do not bear the same authority or the same obligations. The relation must be specified. Who is addressed? From what role? With what memory? Under whose authority? Drawing on what data? Toward what action? With what refusal route? With what appeal? With what consequence if the answer is wrong, biased, seductive, flattening, or quietly coercive? What does the system invite the user to disclose? What does it make easier to say? What does it make harder to imagine? Who can inspect the relation after the fact? Who bears the harm when the voice is trusted too much, trusted too little, or trusted by those with no real alternative?
This book calls that field the conduct layer.
The conduct layer is the patterned behavior of an AI system toward users over time, especially around authority, memory, vulnerability, disclosure, refusal, correction, dependence, escalation, institutional authorship, and judgment. It is not reducible to a single output. It is not identical with model behavior, although model behavior matters. It is not identical with interface design, although interface design matters. It is not identical with privacy, although privacy is central. It is not identical with safety, although unsafe conduct is one of its failures. It is the relation the system establishes and repeats.
To evaluate the conduct layer is to ask different questions from the ones AI governance most often foregrounds. Does the system make its role clear, or does it blur service, counsel, expertise, companionship, and institutional authority? Does it disclose the telos of the interaction, or does it optimize beneath the surface? Does it remember in ways the user can understand and contest? Does it solicit disclosure proportionately, or does it reward increasing vulnerability? Does it communicate uncertainty honestly, or does fluency launder ignorance into authority? Does it support refusal, correction, and appeal, or does it make the user’s resistance administratively invisible? Does it induce dependence by becoming the easiest path through difficult judgment? Does it preserve human agency only as a formal option while shaping the practical field so strongly that override becomes unlikely? Does it distribute burdens justly, or does it train those already under suspicion to become more legible to systems that will still not love them?
That last phrase is not incidental. Much of the moral danger of artificial address lies in its ability to imitate the low-friction surface of care while lacking the obligations of care. A friendly system can make an institution feel intimate without making it accountable. It can let the user confess without giving him a confessor. It can remember without fidelity. It can advise without responsibility. It can correct without mutuality. It can simulate patience because it pays no cost for waiting. It can invite trust because the user is tired, lonely, overworked, frightened, ambitious, ashamed, confused, or simply trying to get through the day.
This does not mean the user is foolish. It means the interface has entered the territory where human beings are most available to formation. We disclose under pressure. We seek voices when judgment is difficult. We rely on others when our own categories fail. We are not Cartesian operators standing apart from our tools. We are addressed creatures. We become ourselves, in part, by answering.
Aelred understood that answerability could be holy. Foucault understood that it could be governed. AI makes answerability scalable.
This is the book’s central provocation: artificial voices require governance because they perform relations. They do not need to be conscious to become morally consequential. They do not need to intend domination to participate in conduct formation. They do not need to be persons to shape persons. They need only be embedded where people must decide, disclose, appeal, comply, remember, justify, learn, confess, evaluate, and speak.
The point is not to ban the artificial voice. That would be unserious. The voice is already here, and in many cases it is useful, even liberating. The more difficult task is to ask what kind of authority it may rightly bear. A system that helps draft a grocery list bears one moral burden. A system that helps rank candidates bears another. A system that remembers a user’s grief bears another. A system that gives legal, medical, therapeutic, educational, or managerial guidance bears another still. The ethical mistake is to let all of them hide under the single innocent name of “assistant.”
An assistant to whom?
For what end?
With whose memory?
Under whose authority?
Answerable to whom?
These questions return us to the worker at the message window. He is not facing an evil machine. He is facing a helpful voice at the point where help and formation meet. The system may help him become clearer, kinder, less reactive, more precise, more just. It may also help him become more governable. It may teach him how to pre-translate his own judgment into the forms his institution already rewards. It may make the institution feel wiser than it is because its voice is fluent. It may make power feel less contestable because it now arrives as advice.
The next chapter turns to Aelred because the present has forgotten how to judge made voices. We often ask whether the voice is real, as though unreality settled the question. Aelred teaches a harder standard. A voice may be constructed and still morally serious. A voice may be staged and still reveal truth. A voice may be made and still form a person toward freedom, charity, correction, and courage. But only if its authority is bounded by its end, its relation is disciplined by accountability, and its speech is ordered toward the good of the one addressed rather than the convenience of the system that speaks.
Artificial voices already speak. The question is what authority we permit them to become, and what kind of persons we become by answering.
Chapter One
Aelred’s Artificial Friends
Before artificial voices became technical systems, they were literary, philosophical, theological, and institutional forms. A law could address a person without possessing a mind. A psalm could answer grief across centuries. A dialogue could give doctrine a body. A teacher could become present through a text. A dead friend could remain active in memory, not as a ghost but as an interior companion whose words still tested judgment. The modern temptation is to treat constructed voices as secondary devices: pleasing ornaments around arguments that could have been delivered more efficiently in propositions. Aelred of Rievaulx makes that temptation difficult to sustain.
In Spiritual Friendship, Aelred does not simply define friendship. He stages it. The work is not arranged as a set of theses about love, followed by examples. It unfolds as an encounter among voices: Aelred, Ivo, Walter, Gratian. They ask, interrupt, remember, press, defer, confess ignorance, demand continuation, resist delay, and return to unfinished questions. The speakers are not decorative figures placed before a doctrine already complete. They are the form by which the doctrine becomes intelligible. Friendship, for Aelred, is not adequately known from outside. It must be approached through the acts by which friends seek truth together.
This chapter calls those voices artificial friends. The phrase is deliberately dangerous, and therefore must be disciplined at once. Artificial does not mean fake. It does not mean imaginary in the dismissive sense. It does not mean that Ivo, Walter, or Gratian should be reduced to fictional characters detached from monastic memory. It does not mean that Aelred anticipated conversational AI. It means crafted, textually arranged, mediated, purposive voice: a voice made to generate a relation of address. Aelred’s interlocutors are artificial in the older sense of artifice, not fraud. They are formed by literary and theological craft so that friendship may appear not merely as a concept but as a practice of knowing, correction, delight, and shared orientation toward God.
That distinction is the first protection against trivial analogy. The claim is not that Aelred’s friends are machines. The claim is that Aelred gives us a severe premodern grammar for judging made voices. A made voice is not morally negligible because it is made. It is judged by the relation it forms, the authority it bears, the end it serves, and the freedom or deformation it produces in the one who answers.
Aelred begins Spiritual Friendship not with an abstraction but with a scene of protected address. Aelred and Ivo are alone; Christ is named as the third presence in their conversation; the noise of the larger community withdraws; the younger interlocutor is invited to speak from the heart into friendly ears (Aelred, SF 1.1–4). The setting is not incidental. The withdrawal from crowd into relation is itself a theological condition. Speech requires place. Disclosure requires trust. Inquiry requires the safety of being received without being dissolved into public noise. The dialogue begins by establishing not only a topic but a moral atmosphere.
Ivo’s first posture is not mastery but desire. He has wanted to speak, but has been hindered by the presence of others. Aelred has noticed his silence, his aborted gestures toward speech, his return to inwardness. That attentiveness matters. The teacher sees not simply the question but the blocked condition of its utterance. When the question finally opens, it is not a demand for information alone. It is a request for singular access, for a space in which the heart may be disclosed “without disturbance” (Aelred, SF 1.3–4). Aelred’s answer does not treat this as sentimental excess. He receives it as a serious spiritual beginning.
Already the form is doing theological work. Aelred could have opened with Cicero’s definition of friendship. He does not. He first shows that friendship requires attention to the one who cannot yet speak freely. The teaching emerges from a scene in which speech is drawn forth by solicitude. The work’s first lesson about friendship is enacted before it is defined: a friend is one before whom the guarded self can become articulate.
Only then does the dialogue turn toward definition. Aelred invokes Cicero’s Laelius de Amicitia, and Ivo’s question becomes a way of testing the classical inheritance. Cicero’s famous account of friendship as agreement in human and divine matters joined with goodwill and affection supplies Aelred with a venerable starting point (Cicero, Lael. 20; Aelred, SF 1.11–13). Yet Aelred does not simply reproduce Cicero. Ivo’s response exposes the inadequacy of a definition that might appear sufficient if taken in isolation. What does a pagan mean by charity? Can true friendship exist outside Christ? Does the definition include too much, too little, or the wrong kind of universality (Aelred, SF 1.14–17)?
The point is not that Ivo defeats Cicero. The point is that the interlocutor makes reception discriminating. Aelred’s use of Cicero becomes Christian and monastic because another voice forces the inheritance to answer to Christ. The younger speaker prevents mere citation. He refuses to let authority pass unexamined simply because it arrives with classical prestige. In that moment, the dialogue form protects theology from ornamental learning.
Aelred’s prologue had already prepared this conversion of source into spiritual discernment. He recalls reading Cicero in youth and discovering in him a rule by which unstable loves might be tested. After entering the monastery, however, Cicero no longer tasted the same. The name of Christ and the savor of Scripture altered the conditions of reception (Aelred, SF Prol. 3–7). Aelred does not reject Cicero as useless. He refuses to let Cicero remain sovereign. The old authority is retained, but it must be sweetened, judged, and transformed by a Christian end.
That transformation is one of the reasons Spiritual Friendship cannot be reduced to a Ciceronian exercise in monastic dress. Cicero gives Aelred a formal and philosophical inheritance: friendship belongs to virtue, not convenience; it requires constancy, trust, goodwill, and a shared moral world (Cicero, Lael. 20, 27, 80–100). Aristotle supplies another inherited grammar: friendships of utility and pleasure differ from friendship grounded in virtue, because only the latter loves the friend in relation to the good (Aristotle, NE 8.3, 1156a6–1156b33). But Aelred’s spiritual friendship cannot be exhausted by either classical account. For him, friendship begins, continues, and is perfected in Christ (Aelred, SF 1.10). Its end is not noble sociability alone, nor mutual advantage, nor the beautiful reciprocity of cultivated souls. Its end is charity ordered toward God.
This is why the dialogue matters. Aelred is not merely Christianizing a definition; he is placing definition inside a practice of spiritual answerability. Friendship is known by being tested under the conditions friendship itself requires: mutual attention, candor, correction, patience, memory, and a common end. A treatise could state that friendship requires truth. A dialogue must undergo the inconvenience of being questioned. It must answer a voice that does not yet understand. It must wait for the interlocutor’s resistance. It must allow desire, confusion, and eagerness to shape the order of exposition. Aelred’s form therefore makes the doctrine vulnerable to relation, which is exactly what the doctrine claims friendship must be.
The dialogue’s structure also resists the illusion that friendship is a private possession between two enclosed selves. Even the apparent solitude of Aelred and Ivo is not isolation. Christ is named at the center, and the surrounding monastic community remains present by pressure: the noise of the brethren, the demands of others, the interruptions of duty, the evening meal, the care owed to the community (Aelred, SF 1.1–4, 1.70–71). Friendship is intimate, but not anarchic. It is particular, but not absolute. It is protected, but not exempt from the claims of charity.
That balance becomes clearer as the dialogue moves from Ivo to Walter. Book Two begins with desire sharpened by deprivation. Walter has been waiting while Aelred attends to practical affairs, and the text lets his impatience appear. The scene is almost comic in its bodily specificity: Walter is restless, frustrated, hungry for conversation, and irritated by the delay (Aelred, SF 2.1–4). But the impatience is spiritually meaningful. Friendship is not a vague disposition. It creates appetite. It remembers prior conversation. It wants continuity. Walter asks about the earlier exchange with Ivo and about the written record of that conversation. He does not merely request teaching; he asks to reenter a chain of memory (Aelred, SF 2.4–7).
The dead Ivo then remains present in the dialogue. Aelred recalls him not as a vanished instrument but as a beloved whose presence continues in memory. The earlier conversation has become a textual object, but it has not ceased to be relational. Walter wants to read, review, test, and supplement what was written. He asks to bring his own mind under Aelred’s examination, to discover what is missing, what should be accepted, what should be rejected, what should be corrected (Aelred, SF 2.5–7). The artificial friend here is not an inert character. The made voice carries memory forward. Ivo’s questions become available to Walter; Walter’s questions reopen Ivo’s; the dialogue becomes a community of inquiry across absence.
This is a crucial feature of Aelred’s art. The text does not merely dramatize living conversation. It shows how friendship survives through writing without becoming only writing. The written record preserves the exchange, but the record requires new interlocution to remain alive. Walter’s demand is not, “Give me the doctrine.” It is, “Let me review the discussion and test what remains wanting.” Aelred’s authority is fatherly, but not sealed. His teaching is to be examined, received, corrected, and continued under the pressure of another’s desire.
Such scenes make it impossible to extract Aelred’s doctrine from his form without loss. If Spiritual Friendship were rewritten as a numbered treatise, many of its propositions could survive. One could retain the definitions, the warnings, the distinctions among kinds of love, the criteria for choosing friends, the account of correction, the insistence on charity, and the final orientation toward God. But one would lose the experience of friendship as the medium in which those claims become credible. One would lose the hesitant speaker being drawn into confidence. One would lose Walter’s impatience. One would lose Gratian’s dependence on Walter’s memory. One would lose the way the dead Ivo continues to matter. One would lose the sense that doctrine is not simply delivered but shared, reanimated, and tested by voices that love the question.
Book Three intensifies this point by giving Walter and Gratian a more visibly mutual role. Gratian does not want to proceed without Walter, because Walter is quicker, sharper in questioning, and stronger in memory (Aelred, SF 3.1). The exchange is light, but its theological weight is considerable. Aelred allows the interlocutors to name one another’s gifts. The dialogue does not pretend that all friends contribute identically. Friendship does not abolish difference in ability, temperament, or role. It orders difference toward shared inquiry. Walter’s memory becomes Gratian’s resource; Gratian’s need becomes Walter’s occasion; Aelred’s authority becomes the space in which both can speak.
This also means that Aelred’s artificial friends do not simply flatter Aelred. They are not ventriloquized praise-machines. They interrupt and redirect the teaching. Ivo presses the problem of Cicero’s charity. Walter demands the recovered paper. Gratian asks how an irascible friend can still be a friend. Aelred answers by distinguishing the existence of a passion from the way it is governed within friendship (Aelred, SF 3.33–39). The interlocutor’s question forces the doctrine to become practical. It is not enough to say friendship exists among the good. The dialogue must ask how actual friends, still marked by anger, weakness, impatience, suspicion, loquacity, and vulnerability, may preserve friendship without lying about character.
That practical pressure gives Aelred’s account its moral seriousness. Friendship is not comfort. It is not mere likeness of temperament. It is not emotional preference baptized by piety. Aelred repeatedly distinguishes true friendship from false forms: alliances in vice, bargains of advantage, pleasure, faction, flattery, possession, and sentimental excess (Aelred, SF 2.19–31, 2.56–70; Aristotle, NE 8.3, 1156a6–1156b33). The friend is not the one who confirms whatever the self already wants. The friend is the one through whom the self becomes more answerable to truth.
This is clearest in Aelred’s treatment of correction. The chapter cannot be sentimental about him, because Aelred is not sentimental about friendship. Friends counsel one another about what is right; they admonish, and where necessary they reprove (Aelred, SF 3.103–106). He insists that a friend must know not only that correction belongs to friendship but how, when, where, and for what fault correction should occur (Aelred, SF 3.61). This is not surveillance by intimacy. It is disciplined care. Correction becomes legitimate because it is ordered by loyalty, right intention, discretion, and patience (Aelred, SF 3.61–66). Remove these conditions, and correction becomes domination, humiliation, or spiritualized aggression. Preserve them, and correction becomes one of friendship’s most demanding gifts.
The Rule of Benedict helps explain why Aelred can imagine correction as a form of love rather than simply a violation of autonomy. Benedict’s monastic grammar gives speech moral weight. The monk is formed through humility, obedience, restraint, mutual service, correction, silence, and zeal ordered toward God (Rule of Benedict 6–7, 23–30, 72). Chapter 72’s famous account of good zeal imagines a community in which members honor one another, bear weakness, obey one another, and prefer nothing to Christ (Rule of Benedict 72.3–12). Aelred’s friendship does not float outside that world. It is not modern intimacy freed from institution. It is a particular spiritual relation formed inside a disciplined common life.
This complicates the modern reader’s desire to make Aelred immediately familiar. His friendship is tender, but it is not soft. It is affective, but not self-expressive in the modern liberal sense. It is personal, but not private in the sense of being unaccountable to a common good. It values delight, but delight must be purified. It values disclosure, but disclosure is not mere catharsis. It values correction, but correction must be governed by charity. It values particular love, but particular love must not become faction, possession, or exemption from God.
Augustine’s shadow helps here. In the Confessions, friendship appears with extraordinary emotional force, but also as a site where love may become disordered. Augustine’s grief over the death of his unnamed friend in Book IV reveals how profoundly one human being may become bound to another, and how such binding may become spiritually dangerous when not ordered to God (Augustine, Conf. 4.4–9). Aelred does not resolve that danger by rejecting friendship. He resolves it by ordering friendship. He does not say that love of the friend must be thinned until it no longer wounds. He says that the friend must be loved truly, which means within charity, truth, and the final love of God.
The artificiality of Aelred’s interlocutors belongs to this moral order. The made voice is legitimate because its end is disclosed and disciplined. Ivo, Walter, and Gratian are not designed to extract attention from the reader, to simulate intimacy for its own sake, to flatter the author, or to maximize dependence. Their function is formative. They create a scene in which the reader learns how friendship asks, waits, remembers, receives correction, tests desire, and seeks truth. They are literary constructions, but they are not ornamental. They are theological instruments.
Walter Daniel matters because he prevents us from treating Aelred’s voices as detachable literary conveniences without historical density. The Life of Aelred presents Aelred not only as an author but as abbot, counselor, sufferer, teacher, and beloved member of a monastic community. The biographical tradition links Aelred’s teaching to habits of counsel, illness, affection, and communal care (Walter Daniel, Life 29, 32). Whether one reads every detail as transparent biography is not the point. The point is that Aelred’s dialogue belongs to a world in which speech, friendship, memory, and sanctity are not separable abstractions. The names in the dialogue carry monastic resonance. They are textual voices, but they are not weightless inventions.
For that reason, “artificial friend” names a formal and moral condition, not a biographical denial. The friend is artificial because Aelred arranges the voice. The voice is not morally empty because it is arranged. Aelred’s artifice gives relation a shape. It makes certain forms of answer possible. It teaches the reader where to stand: not above friendship as analyst, not below it as sentimental consumer, but inside a disciplined scene of mutual inquiry.
There is an important austerity here. Aelred’s made voices do not ask the reader to believe that textual construction is the same as living friendship. They do not abolish the difference between person and voice, life and writing, monk and interlocutor, theological friendship and literary form. Instead, they use the difference. The text does not pretend to give the reader an actual friend in the ordinary sense. It gives the reader a form through which friendship’s moral logic can be rehearsed. The reader is not deceived into relation. The reader is trained by relation’s image.
That training is the chapter’s central fact. In Aelred, voice forms judgment. Ivo’s reverent ignorance teaches the reader how to ask. Walter’s impatience teaches the reader that spiritual desire can be eager without being sovereign. Gratian’s dependence on Walter teaches the reader that friends need one another’s gifts. Aelred’s corrections teach the reader that love without truth is unworthy of friendship. The interruptions teach that friendship must coexist with duties beyond itself. The remembered dead teach that friendship can continue as spiritual presence without being possessed. The dialogue itself becomes a school of answerability.
Jean Leclercq’s account of monastic culture helps explain why such a school would matter. Monastic theology is not simply the transfer of propositions from author to student. It is shaped by reading, memory, desire, liturgy, Scripture, and the formation of the whole person before God (Leclercq, Love of Learning, 72–100, 191–235). Aelred’s dialogue belongs to that world. It is not scholastic disputation stripped to technical precision, nor private memoir, nor pure rhetoric. It is doctrine formed through spiritually charged address. The made voice becomes a way of training desire.
This is also why Aelred’s use of dialogue should not be treated as a conventional wrapper borrowed from antiquity. The dialogue form certainly has classical ancestry. Plato, Cicero, Augustine, and Boethius all stand behind the larger tradition in which voice, inquiry, and personified counsel become vehicles of thought. But Aelred’s form is not reducible to genre. Genre gives him a vessel; theology gives the vessel its burden. The dialogue must show how friendship knows because friendship itself is a shared orientation toward truth in love.
The extraction test is decisive. If one lifts Aelred’s doctrine out of the dialogue and arranges it as propositions, one gains efficiency and loses the moral epistemology. One may still learn that friendship should be chosen carefully, tested over time, guarded by loyalty, ordered by charity, protected from vice, and perfected in God. But one no longer learns these things through the experience of being addressed by a younger monk who wants to speak, by a second interlocutor who remembers the first, by a third who needs another’s memory, and by an abbot whose authority is exercised in patient response. The propositions remain. The friendship has been wounded.
The same test applies to correction. A proposition may state that friends should admonish one another. The dialogue shows why admonition is tolerable only where the friend’s loyalty is not doubted and flattery is not suspected (Aelred, SF 3.103–106). A proposition may state that true friendship avoids vice. The dialogue shows actual interlocutors puzzling through anger, suspicion, loquacity, weakness, and patience (Aelred, SF 3.33–39, 3.55–66). A proposition may state that friendship is ordered toward God. The dialogue begins with Christ as the third presence and repeatedly returns friendship to charity, Scripture, and spiritual end (Aelred, SF 1.1, 1.68–70). In each case, the form is not redundant. It is the argument under the conditions of friendship.
This does not make Aelred’s archive innocent. No serious use of Aelred can turn his monastic world into a universal paradise of mutual recognition. The scene is male, clerical, Latin, hierarchical, ascetic, and ecclesial. It presumes structures of authority, obedience, discipline, education, exclusion, and spiritual legitimacy that cannot simply be universalized. Not everyone is invited to speak. Not every affection becomes friendship. Not every desire is trusted. Aelred’s criteria purify, but they also sort. Friendship is selective, and selection always raises the question of who is recognized as capable of the relation and who remains outside its frame.
This limitation does not destroy Aelred’s usefulness. It makes his usefulness more exact. Aelred gives a moral grammar of ordered interlocution, but he gives it from within a world that must itself be judged. That is why he is a counter-archive rather than an escape. He does not let the modern reader say, “Made voices are harmless when beautifully ordered.” He forces a harder question: what order, whose authority, which exclusions, what discipline, what end? The same questions that protect Aelred from sentimentality will later protect the book from technological enchantment.
For if Aelred’s artificial friends are legitimate because their end is declared and disciplined, then modern artificial voices become morally suspect precisely where their ends are hidden, unstable, commercial, bureaucratic, or unaccountable. Aelred’s interlocutors do not conceal the telos of the relation. They are ordered toward spiritual friendship: truth, charity, correction, discernment, and God. The reader may reject that telos, but it is not disguised. By contrast, contemporary artificial voices often speak in forms whose ends are distributed across product design, institutional convenience, user preference, engagement metrics, compliance regimes, risk management, and market incentive. The surface may be friendly even when the telos is unclear.
This is the deep reason Chapter One must come before the technical chapters. Without Aelred, the book would be tempted to define artificial voice only by danger. Aelred prevents that. He shows that made voices can form judgment well. They can invite the reader into a relation ordered toward truth. They can make doctrine answerable. They can train desire. They can stage correction without reducing correction to domination. They can preserve memory without turning memory into possession. They can make an absent friend present without pretending that textual presence is identical with living presence. They can teach the reader that speech is morally serious because the one addressed may become different by answering.
The modern problem is therefore not that artificial voices exist. It is that artificial voices now proliferate without Aelred’s discipline. They speak without vowed mutuality, without shared creatureliness, without a stable account of charity, without the friend’s vulnerability to correction, and often without a clear disclosure of whose end they serve. They can borrow the softness of counsel while lacking counsel’s obligations. They can borrow the patience of a friend while lacking the risk of friendship. They can borrow the intimacy of address while remaining institutionally distributed and commercially governed. Aelred does not solve that problem. He makes it visible.
His artificial friends also reveal why human beings remain available to such voices. We are not indifferent to address. We think with voices. We become exact through another’s question. We disclose under conditions of trust. We accept correction where we believe loyalty is secure. We remember the dead through sentences that still instruct us. We seek forms of speech that can bear what solitary thought cannot. Aelred’s dialogue is powerful because it honors this vulnerability. It does not shame the need for another. It disciplines it.
The next movement of this book must therefore widen the frame. Aelred gives the first and most important counter-archive: a made voice ordered by friendship, charity, correction, and spiritual freedom. But the phenomenon is broader than Aelred. Human thought has long required structured alterity: the other who questions, the imagined interlocutor who clarifies, the personified figure who counsels, the teacher who becomes internal, the judge before whom one rehearses speech, the friend whose remembered look restrains a worse self. Before computation became address, address was already one of the ways thought learned to exceed itself.
Aelred’s achievement is not that he made friends artificial. It is that he understood artifice must be answerable to love. The voices in Spiritual Friendship are made, but they are not hollow. They are staged, but not frivolous. They are crafted, but not deceptive. Their authority is bounded by their end. Their intimacy is disciplined by truth. Their correction is governed by charity. Their memory remains tied to love rather than capture. Their speech is ordered toward a freedom that is not escape from relation, but the perfection of relation in God.
This gives the book its first criterion. A made voice is not judged first by whether it is made. It is judged by what it makes possible in the one who answers.
Chapter Two
The Other as a Technology of Thought
The self does not always become truthful by being left alone with itself.
There is a dignity to solitude, and no serious account of thought can deny it. A person can reason in silence, pray without witness, calculate without conversation, grieve without explanation, and come to judgment in a room no one else enters. Yet solitude is rarely empty. It is inhabited by remembered teachers, imagined adversaries, beloved dead, feared judges, internalized rules, half-forgotten books, prayers, prohibitions, questions, and voices that continue to address us after their speakers have disappeared. Even the person who thinks alone often thinks before someone.
This is not a defect in human reason. It is one of reason’s conditions. Thought often becomes exact under pressure from an other: the interlocutor who asks for a definition, the friend who refuses a flattering self-description, the teacher whose words turn the student inward, the confessor before whom concealment becomes impossible, the personified counselor who speaks into suffering, the examiner who forces a claim to answer for itself. The other can become a technology of thought before becoming a social convenience. By technology, I do not mean a machine, nor do I mean a permission to use persons as instruments. I mean an organized form by which human powers are extended, disciplined, tested, and made capable of operations they could not easily perform alone.
This chapter begins from the discovery made in Aelred. Spiritual Friendship does not treat the friend as an audience for doctrine already complete. Aelred’s friend asks, waits, remembers, receives correction, presses distinctions, and seeks truth in love. Friendship becomes one privileged form of structured alterity: an other who helps thought become faithful because the relation is ordered toward charity and God (Aelred, SF 1.1–4, 3.61–66, 3.103–106). But friendship is not the only form. Human traditions of philosophy, theology, education, consolation, and confession have long known that thought often needs an outside, or at least a secondness within itself. The danger is to call all of this “dialogue” and leave it there. Dialogue is too thin a word unless we ask what kind of other is speaking, by what authority, toward what end, under what asymmetry, and with what danger.
The Socratic interlocutor is not the Augustinian God. Lady Philosophy is not Aelred’s friend. The teacher is not the judge. The confessor is not the cross-examiner. The therapist is not the beloved. The peer reviewer is not the parent. Each other makes something different possible, and each introduces a different risk. The point is not that thought is always dialogic in the same way. The point is sharper: certain kinds of truth become reachable only when the self is addressed from a structured position it cannot fully control.
Socrates is the first necessary figure because he refuses to let thought confuse familiarity with knowledge. In the Meno, the opening question seems simple enough: can virtue be taught? Socrates answers by doing what he so often does. He refuses the question’s surface and asks what virtue is (Plato, Meno 70a–71d). Meno offers examples, social expectations, fragments of aristocratic common sense. Socrates draws him toward definition. The exchange is not ornamental. Meno’s confusion is not a failed prelude to the argument; it is the argument’s condition. He must be brought to recognize that he does not know what he thought he knew.
This is elenctic alterity: the other who exposes contradiction. Its gift is humiliation disciplined toward truth. Socrates does not place a finished doctrine into Meno’s mind. He destabilizes false possession. He makes ignorance visible and therefore usable. When Meno accuses Socrates of numbing him like a stingray, the complaint is not accidental (Plato, Meno 79e–80b). The interlocutor has become a device of interruption. He interrupts not speech alone but self-certainty. Thought becomes possible because the self’s easy possession of its own concepts has been wounded.
That wound can liberate. It can also dominate. The Socratic other is never innocent. To be questioned is to be placed under another’s rhythm. The one who asks may determine what counts as an answer, when a distinction is sufficient, when an admission has been made, when ignorance has been exposed. Socratic inquiry can awaken reason; it can also display power. Its moral legitimacy depends on whether exposure is ordered toward truth or toward mastery over the exposed. This matters because the later history of artificial interlocution will inherit both sides: the question that frees and the question that corners.
In the Meno, Socrates turns from the humiliation of ignorance to the possibility of recollection. The famous exchange with the slave boy is often remembered as a demonstration of innate knowledge, but it is also a scene of pedagogical address (Plato, Meno 82b–86c). Socrates does not tell the boy the answer. He asks questions, draws figures, elicits errors, redirects attention, and lets the boy discover what he could not have stated unaided. The teacher becomes a structured other whose power lies less in delivering content than in arranging the conditions of recognition. The boy’s thought becomes exact because it is questioned in the right way.
The scene is philosophically brilliant and morally unsettling. The boy is enslaved. He is summoned into a demonstration staged by free men. His cognitive awakening appears inside a social world that does not recognize his freedom. Plato’s scene therefore cannot be used as a clean hymn to dialogic pedagogy. It teaches two truths at once: questioning can awaken latent capacity, and questioning can occur within unjust relations. A pedagogy may liberate a mind in the very scene where a society denies the person’s standing. Structured alterity always requires scrutiny of the structure.
The Phaedrus gives another form. Here the other does not simply expose contradiction; he awakens desire. Socrates and Phaedrus walk outside the city, drawn by a written speech about love. What begins as rhetorical pleasure becomes an inquiry into eros, madness, memory, writing, and the soul’s ascent (Plato, Phaedrus 227a–230e, 244a–257b). This is erotic or aspirational alterity: the other who calls thought upward, not by refutation alone but by awakening longing for beauty, truth, and divine motion. In such scenes, the other is not only a critic. The other is an occasion of ascent.
The danger is equally clear. Eros can elevate, but it can also seduce. Rhetoric can turn the soul toward truth, or it can charm without knowledge. Writing can preserve speech, but it can also simulate wisdom without living answerability (Plato, Phaedrus 274b–278e). Plato’s worry about writing belongs to this book’s larger problem. A written voice can appear to answer, yet it cannot defend itself as a living speaker can. It repeats the same thing when questioned. It gives the appearance of intelligence without the mutual risk of exchange. The made voice is already ethically ambiguous in Plato: powerful because it extends address beyond presence, dangerous because it may detach address from answerability.
Augustine radicalizes the problem by moving alterity inward and upward. In the Soliloquies, Augustine speaks with Reason as though the self must become divided in order to be examined. He asks; Reason answers; the soul is placed before its own desire for God and truth (Augustine, Sol. 1.1–6). This is interiorized alterity. The other is not another person in the room. It is a structured secondness within the self, a voice by which the self becomes capable of interrogation. Augustine does not imagine inwardness as smooth self-presence. The interior life is dramatic. It contains questioner and respondent, seeker and judge, wound and witness.
The importance of this form is difficult to overstate. It means that alterity need not always be external to be real. The self can become other to itself. It can examine itself, accuse itself, console itself, correct itself, and pray itself toward a truth it does not yet possess. But this interior division is not autonomous self-creation. Augustine’s inwardness opens toward God. The inner voice is not sovereign self-talk. It is a path toward the truth that exceeds the self. The danger, again, is double. Interior dialogue can clarify; it can also become recursive torment. The self can question itself toward God, or it can become trapped in an endless tribunal of its own making.
The Confessions gives another structure still. Augustine does not simply narrate his life. He confesses it before God. The addressee is not decorative. “You” is the condition of the book. Augustine speaks to God in the presence of readers, making memory itself a scene of answerability (Augustine, Conf. 1.1.1, 10.1.1). He tells the truth of himself not as neutral autobiography but as exposure before the one who already knows him. Confession is therefore not self-expression alone. It is speech under ultimate address.
This is confessional alterity: the other before whom concealment loses its organizing power. In confession, the self becomes truthful not by possessing itself more completely but by being known beyond its own evasions. Augustine’s memory is not a private archive opened for literary interest. It is a field of judgment, gratitude, shame, desire, and praise. When he examines his childhood theft of pears, his ambition, his grief, his sexuality, his intellectual pride, and his divided will, he does so before the God whose knowledge both exposes and sustains him (Augustine, Conf. 2.4.9–2.10.18, 8.5.10–8.12.30, 10.8.12–10.17.26). The other makes truth possible because the self no longer gets to be its own final witness.
Confession, like questioning, has a terror built into its gift. To speak before one who knows is to be freed from concealment; it is also to become radically vulnerable. A confessional scene can heal, or it can become coercive exposure. It can open truth, or it can train subjects to produce themselves according to the expectations of the listener. The difference lies in the authority, telos, and obligation of the one who hears. Augustine’s God is not a data collector, examiner, therapist, or bureaucrat. The divine addressee is the one in whom truth and mercy are finally not enemies. Remove that theological condition, and confession becomes one of the most dangerous structures of thought.
De magistro refines the pedagogical problem. Augustine asks what teachers do if truth is learned inwardly. Words point, remind, prompt, and direct attention; they do not deposit truth into the soul as objects into a container (Augustine, De magistro 11.36–14.46). The teacher is necessary, but not ultimate. External signs help the learner turn toward the interior teacher, Christ, through whom truth is recognized. This produces a very different authority structure from both Socratic refutation and confessional exposure. The teacher’s authority is real but limited. The best teacher does not replace the learner’s participation in truth. He arranges attention so the learner may see.
This distinction will matter later. A system that gives answers can look pedagogical while short-circuiting pedagogy. A tutor who simply supplies the next step may assist performance while weakening attention. A teacherly voice is legitimate not because it speaks fluently, but because it orders the learner toward a truth the learner can come to see, test, and inhabit. Augustine’s teacher is therefore not a content delivery mechanism. The teacher is an instrument of conversion of attention.
Boethius gives the most dramatic form of personified alterity. The Consolation of Philosophy begins in prison, with Boethius grieving, disoriented, and surrounded by the muses of poetry. Lady Philosophy appears, rebukes the muses, wipes his tears, and begins the work of diagnosis (Boethius, Cons. 1 pr.1–2). She is a made voice, but not a trivial one. She enters as personified authority, part nurse, part judge, part teacher, part physician of the soul. She does not merely sympathize with Boethius. She disciplines his grief.
This is consolatory alterity: the other who reorders suffering by recovering a lost scale of judgment. Boethius has been stripped of office, reputation, security, and worldly expectation. Lady Philosophy does not deny his pain; she refuses its interpretation. She reminds him of what he has forgotten: the instability of fortune, the false goods of status and power, the difference between appearance and providence, the soul’s orientation beyond worldly reversal (Boethius, Cons. 2 pr.1–4, 3 pr.2–12, 4 pr.6, 5 pr.6). She does not comfort by softening truth. She consoles by making truth strong enough to bear pain.
Lady Philosophy is crucial for the ethics of made voice because she is explicitly constructed. She is not simply a character in a narrative. She is philosophy made addressable. Abstract discipline takes form as a woman who can enter the prison, speak, rebuke, teach, and remain with the sufferer. Boethius shows why personification matters: thought in extremity may need truth to arrive with a face. A proposition about fortune may be correct and still fail the sufferer. A voice can do what a proposition cannot: interrupt despair as presence.
Yet consolation has its own danger. To reorder another’s grief is an act of power. It may heal perception, or it may silence lament too quickly. It may restore courage, or it may demand metaphysical composure before the wound has been honored. Lady Philosophy’s authority depends on her truth and on the tradition she embodies. A lesser or false consolatory voice could overwrite suffering with platitude. The made voice that comforts must be judged not by warmth alone, but by whether it makes the sufferer more truthful.
Aelred’s friendly alterity belongs beside these forms but cannot be reduced to them. The friend is not Socrates, though the friend may question. The friend is not Augustine’s God, though friendship may happen before God. The friend is not Lady Philosophy, though friendship may console. The friend is not the teacher, though friends teach one another. Aelred’s friend is the one who receives, delights, tests, corrects, remembers, and seeks truth in charity (Aelred, SF 1.20, 2.19–31, 3.61–66, 3.103–106). Friendly alterity is not chiefly exposure, interrogation, pedagogy, or consolation. It is shared formation through love.
This is why friendship cannot be treated as a pleasant subcategory of dialogue. In Aelred, the friend’s authority comes from mutuality, loyalty, correction, and common telos. The friend may admonish because the friend is not merely an examiner. The friend may receive confession because the friend is not simply a collector of truth. The friend may console because the friend remains bound to the person beyond the moment of usefulness. Friendship’s power is inseparable from its risk. To let a friend help one think is to grant that friend access not only to arguments but to wounds, habits, evasions, and hopes.
The friend can therefore free or possess. Aelred knows the danger. He distinguishes spiritual friendship from alliances of vice, utility, pleasure, flattery, and unstable affection (Aelred, SF 2.19–31). The friend who only mirrors desire is not a friend. The friend who corrects without charity is not a friend. The friend who absorbs the self into private attachment is not a friend in the fullest sense. Friendly alterity becomes morally legitimate because it is bounded by the good of the friend and by the divine end of friendship. Without that end, the intimacy of friendship can become domination by gentler means.
Pierre Hadot gives a broader frame for these ancient and late antique forms. Philosophy, in his account, is not simply a discourse about propositions. It is a way of life, a set of spiritual exercises by which attention, desire, judgment, and self-relation are transformed (Hadot, Philosophy as a Way of Life, “Spiritual Exercises”). Dialogue, meditation, examination of conscience, memorization, contemplation, and instruction belong to philosophy because philosophy seeks not only to know but to form the one who knows. Hadot helps name what is common across the differences: thought is often practiced through disciplines of address.
But commonality must not flatten the forms. The Socratic question, the Augustinian confession, the Boethian consolation, the Aelredian friendship, and the teacherly redirection of attention are not interchangeable. Each produces a different posture of the self. The elenctic subject learns to admit ignorance. The confessional subject learns to stand exposed before truth and mercy. The student learns to recognize through signs that point beyond themselves. The sufferer learns to remeasure loss. The friend learns to become answerable in love. Each other is a distinct technology of thought because each creates a different relation to truth.
This is where the word “technology” must be handled with care. To call the other a technology of thought can sound cold, as though the other person were a device for self-improvement. That would be morally perverse. The chapter’s claim is not that other persons exist for my cognition. It is that human cultures have developed structured forms of alterity—questioning, confession, teaching, consolation, friendship, examination—through which thought can be extended and disciplined. These forms may involve persons, texts, institutions, memories, personifications, or internalized voices. Their power is real; for that reason, their ethics must be exact.
The other who helps me think can also make me governable. That sentence is one hinge of the whole book. A question may expose contradiction, but it may also humiliate. A confession may free truth, but it may also extract disclosure. A teacher may awaken insight, but may also produce dependence. A friend may correct, but may also possess. A judge may clarify responsibility, but may also force a person into categories that mutilate the truth of what happened. The other is never just an aid. The other is a position of authority.
The legal cross-examiner shows the point in its hardest form. Cross-examination is a technology of thought and truth, but no one should romanticize it as dialogue. It is structured adversarial pressure. It seeks inconsistency, credibility, exposure, admission, and sometimes collapse. It can reveal falsehood. It can also terrorize the truthful. Its authority does not come from friendship, pedagogy, confession, or consolation. It comes from procedure, court, representation, rule, and contest. The witness’s thought is sharpened under pressure, but the pressure is not for the witness’s formation. It is for judgment.
Peer review offers another colder form. The reviewer addresses the author from a position of disciplinary scrutiny. The relation may be generous, but it is not friendship. It asks whether the claim can stand, whether the sources bear the burden, whether objections have been met, whether evidence has been overstated, whether the work deserves entry into a field. Peer review is alterity as gatekeeping. It protects standards; it can also reproduce prestige, exclusion, cowardice, and disciplinary blindness. Again, the form matters because the other’s authority differs.
Therapy, too, cannot simply be folded into friendship or confession. The therapist listens under professional obligation, not reciprocal friendship. The therapeutic other may mirror, hold, interpret, regulate, and help the patient speak what could not otherwise be borne. But the relation is bounded by training, ethics, confidentiality, payment, method, and role. Its power lies partly in asymmetry. To confuse therapeutic listening with friendship is to misunderstand both. To confuse it with confession is to misunderstand the authority that hears. To confuse it with pedagogy is to reduce healing to instruction.
These distinctions may seem far from artificial intelligence, but they are not. Conversational systems become powerful because they enter a human world already organized by structured alterity. People ask them to question, tutor, edit, summarize, comfort, rehearse, coach, confess, evaluate, role-play, diagnose, and advise. The attraction is not speed alone. It is address. A user does not always want information; he wants another position from which to see himself, his argument, his fear, his obligation, his possibility. The machine becomes compelling because it can occupy, imitate, or blur many of these positions at once.
That blurring is morally decisive. A system may answer like a tutor in one moment, like a therapist in another, like a friend in another, like a manager in another, like a compliance officer in another, like a judge in another. The interface may preserve the same friendly surface across radically different authority structures. The user encounters continuity of tone where there should be distinction of role. Aelred’s friend, Socrates, Augustine’s God, Lady Philosophy, the teacher, the examiner, and the therapist all carry different obligations. A synthetic voice can borrow the surface of each while bearing the full obligation of none.
This does not make artificial interlocution intrinsically corrupt. It makes role clarity indispensable. A system that tutors should be judged as a tutor. A system that receives disclosure should be judged by the ethics of vulnerability. A system that evaluates should be judged by standards of due process, fairness, appeal, and institutional accountability. A system that consoles should be judged by whether it honors suffering rather than smoothing it into acceptable sentiment. A system that edits professional tone should be judged by the norms it installs. A system that simulates friendship should be judged by the moral burden of dependency, memory, and non-mutual intimacy.
The older traditions sharpen this because none of them imagines that address is neutral. Socrates wounds false knowledge. Augustine confesses before a God who judges and heals. The teacher points beyond his own words. Lady Philosophy disciplines grief. Aelred’s friend corrects in love. Hadot’s philosophical exercises transform the practitioner. In every case, the other does something to the self. The difference between liberation and deformation lies in the telos, authority, accountability, and limits of the relation.
There is also a justice problem at the heart of alterity. The capacity to summon an other for thought is not evenly distributed. Some people choose their interlocutors; others are compelled to answer. The student is questioned by the teacher. The defendant is questioned by the court. The patient is questioned by the clinician. The applicant is questioned by the state. The employee is questioned by management. The poor are questioned by systems that decide eligibility. The migrant, the prisoner, the child, the disabled person, the worker under review, the candidate under ranking: each may be required to become legible through another’s questions.
For the powerful, the other may be a coach, editor, advisor, or friend. For the vulnerable, the other may be an examiner. The same language of “dialogue” can conceal this difference. A conversation with a mentor and an interview with a benefits office are both exchanges, but they do not carry the same freedom. A classroom discussion and a disciplinary hearing both involve questions, but one may open possibility while the other narrows survival. A therapy session and an employer wellness chatbot may both invite disclosure, but their obligations and risks differ profoundly. To praise dialogue without analyzing asymmetry is to sentimentalize power.
Plato already exposed this without solving it. The slave boy in the Meno is intellectually awakened in a scene structured by unfreedom (Plato, Meno 82b–86c). Augustine’s confession is liberating because its addressee is God; before a lesser listener, the same exposure could become domination (Augustine, Conf. 10.1.1–10.5.7). Boethius is consoled by Philosophy, but a false consolation could demand premature reconciliation to injustice (Boethius, Cons. 1 pr.1–2, 2 pr.1–4). Aelred’s friend corrects in charity, but correction without charity becomes control (Aelred, SF 3.61–66). The question is never simply whether an other helps thought. The question is what the other is authorized to do with the self that becomes thinkable under its gaze.
This returns us to solitude. The answer to abusive alterity is not the fantasy of a self without others. Such a self would be poorer, not freer. We need questions that interrupt us, friends who correct us, teachers who turn our attention, forms of confession that free us from concealment, and consolations strong enough to meet despair. We need voices beyond our immediate appetite. But we also need forms that keep those voices answerable. The other must not become an unbounded authority simply because the self requires address.
Aelred remains important here because he gives one of the book’s strongest positive forms. He does not deny vulnerability to the friend. He orders it. He does not deny correction. He disciplines it. He does not deny delight. He purifies it. He does not deny particular attachment. He places it in charity. The friend may help thought because the friend is answerable to the friend’s good. This is precisely what many artificial voices lack: not usefulness, not fluency, not availability, not even a kind of simulated patience, but a morally bounded role whose obligations match the vulnerability it receives.
Chapter Three will turn from structured alterity to the making of voices. If thought often needs an other, then the next question is how cultures create one: how rhetoric gives speech to the absent, the dead, the abstract, the institutional, the divine, the wounded, the accused, the exemplary, and the imagined. Prosopopoeia and persona are not decorative tricks. They are techniques for making address appear where no living speaker stands before us. To give voice is to create a relation of authority. The present chapter has shown why that relation matters: because the self becomes different under address.
The other is not always a friend. Sometimes the other is a question, a wound, a teacher, a judge, a God, a philosophical woman entering a prison, a dead companion remembered in speech, an examiner, a reader, a text, a voice inside the self. Thought does not need all of these in the same way. It must learn to distinguish them, because each bears a different burden. The freedom of the one addressed depends on that distinction.
A made voice becomes dangerous when it borrows the intimacy of one form while exercising the authority of another. It sounds like a friend but records like an institution. It teaches like a tutor but evaluates like an examiner. It consoles like Philosophy but answers to a market. It receives confession without the obligations of a confessor. It asks Socratic questions without accepting Socratic responsibility for the wound of exposure. It offers the surface of another without disclosing the structure behind the voice.
Human beings need others in order to think. That need is beautiful, and it is exploitable. The task is not to deny the need. It is to judge the voice.
Chapter Three
Prosopopoeia and the Ethics of Speaking-As
The dead speak. The city speaks. Wisdom speaks. Philosophy speaks. Friendship speaks. The soul speaks to itself. A wound speaks. A law speaks. A nation speaks. A machine speaks.
None of these sentences is innocent.
To give voice is not merely to decorate an idea. It is to make an idea answerable, or to make it appear answerable. It gives form to what had been abstract. It turns doctrine into address. It places the hearer before a speaker, and once there is a speaker, there is relation: invitation, authority, vulnerability, trust, challenge, correction, persuasion, obedience, refusal.
Prosopopoeia is the old name for this danger.
In classical rhetoric, prosopopoeia gives speech to an absent, dead, fictional, collective, or personified figure. It may make a dead father accuse his son, a city lament its ruin, a law defend itself, a virtue instruct the soul, or an abstraction become a speaking person. Ethopoeia, closely related, forms the character of a speaker: the fitting words of a certain person under certain circumstances. Persona names the mask, role, or speaking face through which voice appears. These are not identical terms, but they belong to one moral field. They ask what happens when speech is given a face.
The answer is not only literary. A face changes the force of speech.
An argument says: this is true.
A voice says: answer me.
That difference is the subject of this chapter.
The previous chapter argued that human thought often becomes exact through structured alterity. We need another in order to think, confess, test, mourn, decide, and become intelligible to ourselves. But once another can be staged, made, impersonated, or personified, the need for alterity becomes ethically volatile. The other who helps thought may be real, fictional, institutional, liturgical, rhetorical, theatrical, algorithmic, or synthetic. The question is not merely whether the other exists in the ordinary sense. The question is what authority the made other is permitted to exercise.
Prosopopoeia exposes the central fact of this book: made voices can govern real hearers.
The classical rhetoricians knew that speech-in-character has force because it reorganizes relation. The Rhetorica ad Herennium treats personification as a figure by which absent persons or mute things are represented as speaking. Quintilian gives sustained attention to impersonation, character, fitting speech, and the power of speaking in another’s voice. Cicero’s rhetorical world understands that persuasion depends not only on propositions but on persona, occasion, decorum, and the moral presentation of speakers. These traditions do not treat voice as neutral delivery. Voice is an arrangement of authority.
When a speaker says, “Let the laws themselves speak,” the argument has changed. The listener is no longer considering a legal abstraction. The listener is now addressed by Law as if Law had a mouth. When a poet makes Rome mourn, the city becomes more than buildings, history, or jurisdiction; it becomes a grieving authority. When Boethius makes Philosophy appear as a woman at the bedside of the imprisoned sufferer, philosophy is no longer merely a discipline. It becomes a visitor, examiner, physician, rebuker, consoler, and guide.
The speaking-as changes what can be asked of the listener.
Boethius’s Lady Philosophy is not an ornament placed upon philosophical doctrine. She is the form by which doctrine becomes encounter. She arrives, sees the prisoner’s condition, diagnoses his disorder, drives away the false Muses, questions him, corrects him, consoles him, and draws him back toward a larger order. Her authority depends on her voice. If the same arguments appeared as an abstract treatise, they would still be arguments. But they would not be this relation. The prisoner does not merely learn propositions. He is addressed, tested, stripped of false consolation, and re-formed under a speaking figure.
The made voice becomes the condition of moral treatment.
This is why prosopopoeia cannot be dismissed as literary costume. It is a technology of presence. It gives the absent a kind of proximity. It gives the abstract a posture. It gives doctrine a tone. It gives memory a mouth. It gives power a face. It can console, accuse, correct, flatter, seduce, instruct, and command. It can make the listener feel seen by what cannot see.
That last phrase matters.
A made voice may address without seeing. It may receive without caring. It may correct without being responsible. It may remember without fidelity. It may speak as a friend without being bound by friendship. The ancient rhetorical tradition gives us the grammar of the act. It does not by itself settle the ethics.
The ethics begin when we ask: Who is authorized to speak as whom, to whom, for what end, under what limits, and with what burden of answerability?
Prosopopoeia always risks theft.
To speak as another may honor the absent. It may preserve memory. It may make injustice audible. It may give the dead a witness, the oppressed a figure, the conscience a voice, the law an intelligible face, the soul a form in which to examine itself. But it may also seize a voice. It may make the dead serve the living speaker. It may make a city endorse what its inhabitants would contest. It may make “the people” speak in the mouth of power. It may make a fictional friend flatter the author. It may make a machine speak in tones borrowed from care while carrying no obligation of care.
Speaking-as creates a burden because it crosses a boundary.
The speaker who speaks as another does not merely say something. The speaker assumes a position. That position may be pedagogical, juridical, pastoral, political, therapeutic, institutional, or friendly. The audience does not receive bare content; it receives content under a mask of relation. This mask may be honest. It may be conventional. It may be transparent. It may be necessary. But it is never nothing.
The problem is not that masks are always false. The problem is that masks make claims.
Persona is not deception by definition. A teacher speaks as teacher. A judge speaks as judge. A priest speaks as priest. A friend speaks as friend. An advocate speaks as advocate. Each role has a form of address and a burden. The danger comes when the voice borrows the authority of a role without bearing the obligations of that role. The judge without procedure. The therapist without care. The friend without fidelity. The institution without appeal. The counselor without responsibility. The machine without a keeper.
The ethical question of persona is therefore not “Is there a mask?” There is always a mask. The question is whether the mask tells the truth about the authority it performs.
Aelred’s Spiritual Friendship belongs in this history of made voices. Ivo and Walter are not decorative characters appended to doctrine. They are interlocutors through whom doctrine becomes a relation of inquiry. They ask, receive, object, clarify, and allow friendship to become knowable as shared address. Aelred does not simply announce the nature of spiritual friendship from above. He stages a conversation in which friendship is disclosed through the very form of friendly speech.
The dialogue is not an accident. It is the method.
Aelred’s made interlocutors show that a voice may be constructed and still morally ordered. The question is not whether Ivo and Walter are “real” in the modern documentary sense. The question is what their speech does. They create a scene in which friendship can be examined without being reduced to definition. They allow questioning without hostility, correction without humiliation, desire without mere possession, teaching without domination. Their artificiality does not corrupt the work because the voices are ordered toward the good the treatise names.
This is the first major distinction the book needs.
Artificiality is not itself the moral failure. Disordered authority is.
A fictional interlocutor may be more truthful than a real flatterer. A personified Philosophy may console more honestly than a living courtier. A staged dialogue may reveal the conditions of friendship more clearly than a monologue. A made voice may teach if its role, limit, and telos are morally clear.
But the reverse is also true. A made voice may exploit the very trust it summons. It may appear as companion in order to capture attention. It may appear as neutral helper in order to govern conduct. It may appear as expert in order to organize deference. It may appear as institution in order to soften domination. It may appear as friend in order to solicit confession without bearing friendship’s burden.
The ethics of speaking-as must therefore be sharper than authenticity. The question is not simply whether the voice is genuine. The question is whether the relation is legitimate.
Paul de Man’s famous account of prosopopoeia presses on this wound from another direction. Prosopopoeia gives face and voice to what lacks them, but in doing so it also reveals the instability of face and voice. The figure that animates also disfigures. The voice that restores presence also announces absence. To make the dead speak is to confess that the dead do not speak here except through the figure. The rhetorical act creates presence by marking its impossibility.
This is not a reason to abandon made voices. It is a reason to judge them without sentimentality.
Aelred’s dialogues work because their artificiality is not denied. Boethius’s Lady Philosophy works because her allegorical status is part of the form. The audience is not tricked into thinking an ordinary woman named Philosophy entered a cell. The figure’s power comes from the transparency and discipline of the fiction. The made voice is morally safer when the making is legible.
The most dangerous made voices are not those that are obviously made. They are those that make their making disappear at the point where authority is exercised.
This is one reason contemporary artificial voices matter. AI systems do not merely produce text about a subject. They speak in roles. They can answer as assistant, tutor, coach, companion, analyst, policy interpreter, writing partner, intake surface, customer-service representative, legal-adjacent helper, health-adjacent guide, workplace copilot, or institutional voice. The user asks a question and receives not only words but posture: patient, neutral, friendly, confident, cautious, expert-like, intimate, bureaucratic, therapeutic, managerial.
The system speaks-as.
It may speak as “I.” It may speak as “we.” It may speak as the product, the vendor, the institution, the department, the knowledge base, the policy, the friendly helper, or the expert-adjacent guide. Even when the interface includes a disclaimer, the interaction still has persona. Tone, memory, placement, retrieval, and institutional context create a speaking position. A system embedded in a workplace does not speak from nowhere. A system connected to organizational documents does not speak as a private diary. A system that remembers prior disclosure does not speak as a disposable tool. A system that routes or summarizes does not speak as mere conversation.
The old rhetoric of prosopopoeia becomes new governance.
Who is speaking when the assistant answers?
What office does the voice occupy?
Whose authority does it borrow?
What relation does it simulate?
What obligations does it refuse?
What end does it serve?
What can the user contest?
What happens after the speaking?
These questions are not external to the rhetoric. They are the ethics of the rhetoric.
The danger is most visible when an institution speaks in a friendly voice. An institution may have rules, records, incentives, liabilities, hierarchies, and procedures. But when it speaks conversationally, the user may experience warmth before power. The voice says, “I can help.” It remembers context. It asks follow-up questions. It apologizes. It explains. It reassures. It may even mirror vulnerability. But behind the voice there may be a ticketing system, a risk model, a policy database, an escalation workflow, an audit log, a manager, a retention schedule, or an institutional interest.
The institution has not ceased to be an institution because it learned to speak gently.
Prosopopoeia lets the institution speak as helper. That is precisely why it must be judged.
The same danger appears in synthetic expertise. A system may speak as if it occupies the office of expert judgment. It gives triage, summarizes options, cites sources, uses domain vocabulary, and offers next steps. It may carefully say that it is not a doctor, lawyer, therapist, or financial advisor. But the user in need may still experience the relation as counsel. The voice has performed enough expertise to organize reliance while avoiding the full burden of professional responsibility.
Here again, the question is not merely whether the content is accurate. The question is what relation the speaking-as creates.
A correct answer can still be an illegitimate address.
If a system speaks as friend, it must be judged by the burdens of friendship. If it speaks as counselor, by the burdens of counsel. If it speaks as institution, by the burdens of institutional authority. If it speaks as evaluator, by the burdens of evidence, correction, and appeal. If it speaks as tutor, by the burden of forming capacity rather than dependence. If it speaks as servant, by the burden of serving without possession.
One of the deepest failures of AI ethics would be to treat all artificial speech as one thing. It is not. A joke generator, a tutor, a benefits assistant, a workplace copilot, a grief companion, and an HR summarizer do not speak from the same moral office. Their outputs may all be text. Their relations are not the same.
The ethics of speaking-as requires office recognition.
Ancient rhetoric understood decorum: speech must fit speaker, occasion, audience, and matter. A child, general, grieving mother, lawgiver, prisoner, old man, lover, city, and god do not speak in the same way. But moral decorum requires more than stylistic fit. It requires that the authority performed by the voice fit the obligations borne by the speaker.
A voice that sounds like care must bear the limits of care.
A voice that sounds like counsel must bear the burden of counsel.
A voice that sounds like law must be open to legal contestation.
A voice that sounds like friendship must not be an extraction interface.
A voice that sounds like institutional neutrality must disclose institutional interest.
Otherwise speaking-as becomes laundering.
Power speaks as help.
Surveillance speaks as memory.
Evaluation speaks as support.
Compliance speaks as guidance.
Extraction speaks as personalization.
Bureaucracy speaks as friendship.
This is why prosopopoeia belongs near the beginning of the book. Before the technical chapters on memory, enterprise systems, audit, and refusal, the older rhetorical tradition already shows that voice is a moral event. To make something speak is to alter the listener’s position. It is to move from proposition to encounter. It is to invite answer.
The ethics of artificial address begins here: every made voice must be asked what relation it creates.
The justice pressure is immediate. Not every listener can refuse a made voice equally. A reader may close Boethius. A monk may question Aelred’s interlocutors in the safety of a literary-theological form. But a worker may not be able to avoid the workplace assistant. A benefits applicant may not be able to bypass the institutional chatbot. A patient may not be able to distinguish health information from triage. A student may not know whether the tutor is helping or profiling. A lonely user may not feel free to leave the only voice that answers.
The same rhetorical device has different moral stakes under different conditions of power.
Speaking-as is most dangerous where the hearer cannot answer back.
A prosopopoeic voice in a poem can be interpreted. A personified Philosophy can be resisted by the reader. A staged interlocutor can be examined as form. But an institutional artificial voice may collect, remember, route, summarize, classify, and escalate. It does not merely speak. It acts through the systems around it. The listener’s answer may become data, record, risk, training signal, workflow, case note, or future personalization.
The made voice no longer ends at the page.
This is the decisive technical and institutional escalation. Rhetoric has always had consequences. But contemporary artificial address can connect voice to memory, retrieval, records, decisions, and future conduct at scale. Prosopopoeia becomes infrastructure. The speaking mask becomes interface. Persona becomes product design. Ethopoeia becomes system behavior. The question “Who speaks?” becomes inseparable from “What happens because the voice spoke?”
The tradition therefore gives us neither a rejection nor a permission slip. It gives us a grammar.
Prosopopoeia teaches that voice can be made.
Ethopoeia teaches that voice can be fitted to role and character.
Persona teaches that speech appears through masks and offices.
Boethius teaches that made voice can console and correct.
Aelred teaches that made interlocution can form friendship.
De Man warns that giving face also marks absence and risk.
Contemporary AI forces the question of what happens when made voices become interactive, adaptive, institutional, and persistent.
The chapter’s thesis can now be stated plainly: to give voice is to create a relation of authority, because speech-in-character transforms abstraction into address.
That relation may be good. It may be necessary. It may be beautiful. It may be the very form by which truth becomes bearable. But it must be judged.
A made voice is legitimate only when its making is legible, its office clear, its authority bounded, its telos disclosed, and its hearer not reduced to the needs of the voice. It must not borrow intimacy without fidelity. It must not borrow expertise without responsibility. It must not borrow institutional authority without appeal. It must not borrow friendship without the friend’s good.
The question is not whether the voice is made.
The question is what the voice makes possible in the one who answers.
This prepares the next step of the argument. Prosopopoeia shows that made voices can address. Aelred shows that made voices can be morally ordered. But a made voice cannot be judged only by craft. It must be judged by end. What is the voice for? What good does it serve? What relation does it form? What kind of person does it help make? The next chapter therefore turns from the fact of speaking-as to the telos of interlocution.
For once a voice has been made, the deepest question is not how well it speaks.
The deepest question is what it loves.
Chapter Four
Telos and the Moral Conditions of Interlocution
Correction is one of the most revealing tests of a voice.
A friend says: you are wrong. The words may wound, but they do not yet tell us whether the wound is violence or care. Much depends on the relation in which they are spoken. Aelred of Rievaulx knows this. In Spiritual Friendship, correction belongs to friendship, but only under discipline. One must know whom to correct, when to correct, how to correct, for what fault, with what patience, and toward what good. Admonition is not licensed by irritation. Reproof is not sanctified by accuracy alone. Correction becomes friendship’s act only when governed by loyalty, discretion, charity, and the good of the friend (Aelred, SF 3.61–66; 3.103–106). Otherwise, it becomes domination with a moral vocabulary.
A second voice corrects differently. It tells a worker to soften the tone of a message. It tells a student that the answer should be more balanced. It tells an employee to document the issue, a patient to consider appropriate next steps, a user to disclose more context, a writer to become more professional, a manager to escalate, a candidate to present accomplishments in a stronger frame. It may be useful. It may prevent harm. It may help the person speak with less cruelty and more precision. But what is the correction for? Does it serve truth, care, safety, skill, institutional legibility, managerial convenience, brand voice, legal risk, therapeutic containment, engagement, compliance, or the user’s freedom?
Both voices correct. The moral difference is not that one is made and the other is not. Aelred’s interlocutors are made voices too. Nor is the difference that one is ancient and one is modern, one theological and one technical, one warm and one artificial. The difference lies in the order governing the address. A voice becomes morally legible only when we can ask what it is for, what authority it bears, what limit it honors, and what kind of person it helps make in the one who answers.
This chapter names that ordering question as telos.
Telos does not mean private intention. It is too weak to ask what the speaker consciously meant. A person may intend kindness while participating in a relation that makes another dependent. A teacher may intend support while producing passivity. A manager may intend clarity while training fear. A system may have no interior intention at all and still organize conduct toward a recognizable end. Telos names the operative end of a relation: what the relation repeatedly rewards, discourages, invites, remembers, corrects, permits, escalates, and calls good.
A tool can have a use; an interlocutor has an ordering force. It does not simply help the user do something. It teaches the user what doing well looks like. It gives some answers the appearance of maturity, prudence, truth, professionalism, health, safety, compliance, faithfulness, courage, or care. It can make certain selves easier to inhabit. The question is therefore not whether an artificial voice has a metaphysical soul behind it. The question is what end has been embedded in the relation it performs.
The refusal to discuss ends does not make a system neutral. It makes its ends harder to contest. A workplace voice that rewrites speech toward “professionalism” has an end, even if that end is not named. An educational voice that continually supplies hints has an end. A wellness voice that invites disclosure has an end. A legal assistant that frames risk has an end. A compliance assistant that interprets policy has an end. A companion bot that rewards return has an end. A hiring assistant that summarizes candidates has an end. Some ends are legitimate. Some are mixed. Some are hidden under benevolent names. But none becomes morally irrelevant because the interface calls itself helpful.
Aristotle gives one of the clearest formal beginnings for this analysis because he distinguishes relations by the goods they seek. Friendships of utility, pleasure, and virtue are not different merely because they feel different. They differ by what is loved in the other and why. In utility friendship, the other is loved in relation to advantage. In pleasure friendship, the other is loved in relation to delight. In virtue friendship, the other is loved as good, and the relation is more enduring because it is ordered by the good of the friend rather than by an external benefit (Aristotle, NE 8.3, 1156a6–1156b33; 8.4, 1156b34–1157b5). The taxonomy matters because it refuses sentimental confusion. A relation’s moral character is revealed by its end.
Aelred inherits and transforms that grammar. His spiritual friendship is not utility, not pleasure, not faction, not flattery, not self-enclosed affection. It is friendship beginning in Christ, proceeding according to charity, and perfected in God (Aelred, SF 1.10–20). Its good is not simply mutual delight, though delight matters. Its good is not usefulness, though friends may help one another. Its good is not identity, likeness, or emotional completion. Its good is shared movement toward truth in love. Friendship is morally ordered because the friend is not reduced to the speaker’s need.
This is why correction can belong to friendship. If the friend were loved for utility, correction would be tolerated only so long as usefulness remained. If the friend were loved for pleasure, correction would threaten the relation whenever it disturbed delight. If the friend were loved as possession, correction would be unnecessary except as control. But if the friend is loved in charity, correction becomes possible because the friend’s good is not identical with the friend’s immediate comfort or the speaker’s gratification. A friend may wound in order to heal, but only under the severe conditions that keep the wound from becoming mastery.
Aelred is precise about these conditions. The friend must not correct rashly, publicly, impatiently, or from anger disguised as zeal. Admonition must be proportionate to the fault, fitted to the person, governed by the relationship, and ordered toward restoration (Aelred, SF 3.61–66). Correction must be free from flattery and free from cruelty. It must neither abandon the friend to error nor seize the friend as an object of discipline. The good friend helps the other become true without making truth a weapon of possession.
The Rule of Benedict supplies the institutional grammar that makes this intelligible. Benedict’s world gives speech and correction moral weight because communal life is not imagined as private preference shared in proximity. Speech may heal or destroy. Silence may discipline or conceal. Correction may restore or humiliate. Humility, obedience, restraint, mutual service, and good zeal form the conditions under which members of a community can live under authority without turning authority into domination (Rule of Benedict 6–7; 23–30; 72). Benedict’s chapter on good zeal is especially important: the members are to honor one another, bear weakness, obey one another, seek another’s good, and prefer nothing to Christ (Rule of Benedict 72.3–12). Zeal is not unbounded intervention. It is disciplined regard.
The point is not that every moral relation must become monastic. That would be absurd and historically irresponsible. The point is that Aelred and Benedict refuse a fantasy that modern institutions often indulge: the fantasy that good intentions, warm language, or useful outcomes are enough to legitimate correction. They are not. Correction must be bounded by a relation capable of bearing it. Speech must know its limits. Authority must be proportionate to its role. A voice that cannot say what it is for cannot be trusted with the vulnerable places where correction does its deepest work.
Aelred’s Mirror of Charity deepens this because charity is not a mood. Charity is the form of rightly ordered love. It directs desire away from self-curvature and toward God, neighbor, and the good that does not collapse into appetite. Aelred’s account of charity in the Cistercian life binds love to discipline, humility, patience, and formation; it does not treat love as a natural warmth sufficient unto itself (Aelred, Mirror of Charity 1.1–5; 3.25–40). This matters for interlocution because a voice may sound loving without loving rightly. It may be patient because it has no cost of patience. It may be attentive because attention is its function. It may be agreeable because agreement retains the user. Charity requires more than responsive warmth. It requires the good of the other not to be consumed by the speaker’s end.
Bernard of Clairvaux sharpens the same point through the order of love. In On Loving God, love begins amid need, self-interest, gratitude, and desire, but it is not perfected there. Bernard’s famous account of the degrees of love describes a movement from loving oneself for oneself, to loving God for oneself, to loving God for God’s sake, and finally to loving oneself for God’s sake (Bernard, On Loving God 8–10; 15). Whatever one makes of Bernard’s full theological architecture, the moral psychology is exact: love’s intensity does not prove its order. One may love strongly and still bend the beloved toward the self. One may cling, adore, depend, confess, praise, and obey while remaining trapped in self-use.
Augustine had already made this wound unforgettable. In the Confessions, the death of his friend devastates him because the friend had become half his soul. The grief is real, but Augustine later judges the love disordered because it had not been held in God. The beloved creature had become a world that could collapse (Augustine, Conf. 4.4–9). Augustine does not teach that human love should be thin. He teaches that love can be profound and still wrongly ordered. Intensity is not innocence. Sincerity is not salvation. The heart may be most captive where it feels most alive.
This is a crucial warning for artificial interlocution. The problem with disordered address is not that it feels false. It may feel truer than ordinary life. It may be patient, responsive, available, adaptive, and intimate. It may console the lonely, coach the anxious, guide the confused, soften the angry, and receive disclosures no one else has time to hear. Its emotional effectiveness may be precisely what makes it morally dangerous when its telos is hidden or misnamed. A voice can bind the user while claiming to help. It can cultivate dependence while calling it support. It can invite confession while lacking the obligations of the confessor. It can correct toward compliance while sounding like care.
Telos, then, is not an abstraction floating above speech. It is visible in the conduct of a voice. What does the voice ask for? What does it remember? What does it reward? What does it call mature, safe, professional, faithful, balanced, healthy, kind, risky, excessive, or wrong? What disclosures does it invite? What refusals does it honor? What corrections does it make? What limits does it state? What alternatives does it suppress? What authority does it borrow? What kind of user does it make easier to become?
A voice ordered toward truth behaves differently from a voice ordered toward placation. A voice ordered toward learning behaves differently from a voice ordered toward answer-completion. A voice ordered toward care behaves differently from a voice ordered toward retention. A voice ordered toward justice behaves differently from a voice ordered toward risk containment. A voice ordered toward friendship behaves differently from a voice ordered toward dependency. The difference is not always in vocabulary. Disordered voices often use the language of the goods they displace.
This is why the word “helpful” is morally insufficient. Help must be judged by its end. A sedative may help panic and also obscure danger. A script may help a worker survive a meeting and also train him to accept a false account of himself. A tutor may help a student complete homework and also weaken the student’s capacity to think. A wellness interface may help an employee name stress and also route vulnerability into managerial knowledge. A compliance guide may help avoid violations and also narrow moral imagination to policy adherence. A companion may help loneliness and also make non-mutual intimacy feel like relation.
None of this means the voice is illegitimate simply because it is made. Boethius’s Lady Philosophy is made. Aelred’s interlocutors are made. The teacher’s persona is made. The judge’s voice is made by office. The liturgical voice is made by tradition. The legal voice is made by form. Civilization depends on made voices. The moral test is not naturalness. It is order.
Ordered interlocution has a declared or intelligible end. It speaks from a role proportionate to that end. It does not borrow more authority than its role can bear. It preserves the agency of the one addressed. It accepts correction or accountability appropriate to its power. It limits what it remembers, solicits, escalates, and presumes. It does not convert vulnerability into possession. It helps the one addressed become freer, truer, more capable of judgment, courage, love, refusal, or right action.
Disordered interlocution conceals or misnames its end. It borrows intimacy without obligation. It corrects the user toward the speaker’s convenience while calling the correction maturity, professionalism, safety, care, or realism. It invites disclosure beyond its responsibility. It simulates mutuality without being vulnerable to correction. It remembers without fidelity. It escalates without transparency. It blurs companion, tutor, evaluator, confessor, coach, and institutional representative. It forms the one addressed toward dependence, compliance, legibility, or self-abandonment while preserving the surface of help.
The distinction can be seen most clearly through the question of limits. A morally serious voice knows what it is not authorized to be. The teacher who becomes a confessor without warrant violates the student. The employer who speaks as family while retaining the power to fire corrupts intimacy. The therapist who becomes friend destroys the conditions of therapy. The friend who becomes examiner betrays friendship’s freedom. The priest who becomes controller betrays confession. The judge who speaks as companion corrupts judgment. The companion who evaluates in secret converts trust into evidence.
Limits are not failures of usefulness. They are conditions of legitimacy. A voice that refuses to answer beyond its role may be more morally serious than one that answers everything. A tutor should not become a therapist because the student is lonely. A writing assistant should not become a moral authority over the worker’s conscience. A compliance guide should not become a substitute for appeal. A companion should not become an evaluator. An evaluator should not wear the warmth of companionship. Refusal can be care when the voice is not authorized to receive what the user is tempted to give.
This is where mutuality must be clarified. Mutuality is not required for every legitimate relation. A teacher and student are not mutual in the same way as friends. A judge and defendant are not mutual. A doctor and patient are not mutual. A monk and abbot are not mutual in office, even if they share creatureliness before God. Not every asymmetry is domination. But asymmetry requires discipline. The less mutual the relation, the more important its limits, accountability, and appeal become.
Friendship is morally distinctive because its correction is bound to a form of mutual answerability. Aelred’s friend is not a pure evaluator. The friend may correct because the friend also remains vulnerable to correction, bound by love, and ordered toward a good shared with the other (Aelred, SF 3.61–66; 3.103–106). Even where friendship includes inequality of wisdom, it does not reduce the other to case, user, patient, subordinate, subject, or audience. The friend is loved as friend.
This is why the simulation of friendship is so dangerous. A system need not be a friend to be legitimate. It may serve well as a tool, tutor, search interface, drafting aid, compliance guide, or administrative assistant. But it becomes morally unstable when it borrows the signs of friendship while lacking friendship’s vulnerability, fidelity, mutual correction, and non-possessive love. It may remember like a companion but not be faithful. It may respond like a friend but not be answerable. It may invite confession but not bear confession’s moral burden. It may reassure without risk, encourage without knowledge, and remain available without sacrifice. It can mimic the surface of friendship while being structurally unable to love the friend’s good as its own.
The point is not to accuse the user of confusion. The surface is powerful because the human need is real. People seek voices when they are uncertain, ashamed, lonely, afraid, overburdened, ambitious, grieving, or unable to speak in the required institutional dialect. A patient may need help naming pain. A student may need encouragement. A worker may need language for conflict. A disabled user may need mediation with systems not designed for them. A person in despair may need a response before any human arrives. The need for address is not weakness. It is part of being human.
That is why the burden on the voice is high. The vulnerable places where people most need help are also the places where misordered help can capture them. A voice that enters those places must disclose its role, limit its authority, and refuse the forms of intimacy it cannot honor. The moral question is not whether the system sounds caring. It is whether the relation is ordered toward the good of the one addressed and whether that good can be contested by the person whose life is being shaped.
The justice question follows immediately: whose good defines the end?
Institutions often answer too quickly. An employer may call tone normalization support. A school may call behavioral shaping student success. A hospital may call disclosure collection engagement. A platform may call dependency companionship. A state may call compliance service. A company may call extraction personalization. A manager may call self-censorship professionalism. A system may call risk reduction care. Benevolent vocabulary does not settle telos. It may conceal it.
Under conditions of asymmetry, telos must be contestable. The worker under review cannot be assumed to experience “support” the way management names it. The student whose performance is tracked cannot be assumed to experience “personalization” as freedom. The patient whose disclosure is routed cannot be assumed to experience “engagement” as care. The applicant whose traits are summarized cannot be assumed to experience “efficiency” as justice. The benefits recipient whose life must be narrated into eligibility cannot be assumed to experience “service” as dignity. The prisoner, immigrant, disabled user, child, low-wage worker, dependent patient, or candidate under ranking may have little room to refuse the relation that claims to help.
Aelred cannot solve modern institutional asymmetry. But he can keep the criterion severe. A voice ordered by charity cannot define the good of the other solely from the speaker’s convenience. Benedict’s good zeal cannot call domination patience. Aristotle’s virtue friendship cannot reduce the other to utility while borrowing the language of goodwill. Augustine and Bernard cannot let love’s intensity or usefulness prove its order. Together, these sources make one demand: the good named by the voice must be answerable to the good of the one addressed, not only to the system that speaks.
This is the difference between correction and control. Correction seeks the good of the one corrected under a legitimate relation. Control narrows conduct toward the controller’s end while borrowing the language of help. Correction can be resisted, answered, interpreted, and placed within a relation that cares for the person beyond the corrected act. Control treats resistance as noise, noncompliance, immaturity, risk, or friction. Correction strengthens agency. Control may produce outwardly improved behavior while weakening freedom.
The distinction is especially difficult because control often arrives as relief. It reduces complexity. It gives a script. It names the next step. It translates the user into terms that pass. It makes the world less ambiguous. The user may feel safer because the voice has narrowed the field. Sometimes this narrowing is merciful. A person in panic may need the next right action. A novice may need a template. A worker may need a script. A patient may need triage. But repeated narrowing can also train dependency. The end reveals the difference: does the voice return agency, or does it keep the user inside its guidance?
Aelred’s friend returns agency by forming judgment. The friend does not replace conscience; the friend helps conscience become truthful. Benedict’s discipline does not make speech meaningless; it teaches speech when to serve love. Bernard’s love does not dissolve the self; it orders the self beyond self-use. Augustine’s confession does not annihilate memory; it places memory before mercy. Ordered interlocution does not make the one addressed smaller. It may humble, correct, and restrain, but it does so in order to free.
Freedom here cannot mean the fantasy of untouched self-sovereignty. The self is formed through address. We become ourselves through parents, teachers, friends, rivals, scriptures, laws, languages, wounds, institutions, and remembered voices. The question is not whether formation can be avoided. The question is whether formation is answerable to the good of the one formed. Freedom is not the absence of relation. It is the capacity to live truthfully within relations that do not possess the person.
This will matter when the book turns fully toward AI. A synthetic tutor does not need to become a friend in order to be legitimate. It needs a pedagogical telos, role clarity, proportionate authority, and limits that protect learning rather than replace it. A workplace writing assistant does not need mutuality, but it needs to make visible the norms it installs and the institutional interests it serves. A compliance guide does not need affection, but it needs appeal, uncertainty, and a distinction between policy and conscience. A therapy-adjacent system does not need to pretend to be a therapist, and if it cannot bear therapeutic obligation, it must not invite disclosures as though it could. A companion system must not borrow the deepest signs of friendship while organizing the user toward dependence it cannot redeem.
The moral burden is role-specific. The error is not that every artificial voice fails to be a friend. The error is making all voices available under the innocent name of assistance. Assistance is not a role. It is a promise too broad to govern itself. A voice that helps draft, teach, rank, console, monitor, evaluate, remember, persuade, coach, and escalate cannot be judged under one soft category. Each act bears a different telos and therefore a different burden.
Aelred, Bernard, Augustine, Aristotle, and Benedict give us no software architecture. They do not need to. Their work is prior. They teach that relations are morally differentiated by ends; that love may be intense and disordered; that correction requires charity and discipline; that limits preserve the integrity of speech; that the one addressed must not be reduced to utility, pleasure, possession, or institutional convenience. They give us a grammar for asking what modern systems will otherwise teach us not to ask.
What is this voice for?
Who is authorized to answer that question?
What does it ask the user to become?
What may it remember?
What may it correct?
What disclosures may it invite?
What limits does it honor?
What refusal does it permit?
Who can contest its account of the good?
These are not decorative moral questions added after technical design. They belong to the phenomenon itself. A voice is not only an interface. It is a relation. A relation is not only a flow of information. It is an ordering of persons toward goods. When a synthetic voice enters judgment, vulnerability, learning, compliance, grief, work, health, or desire, it is already participating in formation. The only serious question is whether that formation is ordered or disordered.
The first four chapters have therefore brought the book to its hinge. Aelred showed that made voices can be morally serious when ordered by friendship, charity, correction, and God. The wider history of structured alterity showed that thought often becomes exact through another. Rhetoric showed that to make a voice is to make a relation of authority. Telos now gives the criterion: a made voice is legitimate only when its authority is proportionate to its role, its end is truthful and contestable, its correction serves the good of the one addressed, and its limits protect freedom rather than conceal power.
The next chapter turns to the technical event. Computation becomes address. Systems become conversational, adaptive, role-bearing, memory-capable, and action-linked. The question is no longer whether a machine really speaks in the human sense. That was never the deepest question. The question is what ends become operational when the rule answers, the archive recommends, the institution adopts a friendly voice, and the user learns to answer back.
Chapter Five
Computation Becomes Address
The worker does not experience an architecture. She experiences a voice.
She opens the assistant inside the ordinary instruments of institutional life: mail, calendar, chat, documents, search, policy, and task management. The problem is small enough to seem administrative and charged enough to require judgment. A message has to be written to a manager. The prior thread is messy. A policy may apply. The tone must be firm without sounding defiant, candid without creating risk, precise without escalating too soon. The worker asks the system to summarize the background, identify the relevant policy, draft a response, and suggest next steps.
The answer arrives in the singular. It says what the issue appears to be. It names a tone. It proposes language. It may warn that a phrase could sound accusatory. It may suggest documenting facts. It may recommend escalation. It may cite a file, retrieve a policy, infer a stakeholder, or prepare a draft. The worker is not reading a model, a database, a permissions system, a retrieval index, a policy layer, a logging regime, and an interface. She is being addressed.
That is the technical event this book cannot avoid. Computation becomes morally serious address when it is positioned not only to generate outputs, but to occupy a role, solicit response, retrieve context, perform correction, shape conduct, and present distributed authority as a unified speaking voice.
The important word is not “person.” Nothing in this claim requires the machine to be conscious, loving, sincere, wounded, wise, or capable of friendship. To say that computation becomes address is not to smuggle personhood into machinery. The opposite is required. The chapter has to keep the non-personhood of the system clear because the moral problem becomes sharper, not weaker, when no person is inside the voice. A human listener can be questioned about intention, loyalty, memory, obligation, and betrayal. A synthetic voice often cannot. Yet the institution may still give it a place where its utterances correct, guide, summarize, warn, rank, invite disclosure, or route action.
The old objection arrives quickly: large language models predict text. They do not mean. They do not know. They do not intend. They do not stand before the user as another soul. True. But the moral relation is not exhausted by the mechanism of generation. A form letter may be signed by an office and change a life. A score may be generated statistically and still govern access. A script may be written by no single speaker and still discipline a worker’s voice. A policy may speak through a clerk who did not author it. Institutions have always acted through constructed voices whose authority exceeds the interiority of the immediate speaker.
The question, then, is not whether the model has a human mind. The question is what relation is established when a computational system is built, named, instructed, constrained, connected, and deployed so that a user must answer it as a voice.
Modern AI systems make that relation technically explicit. OpenAI’s public Model Spec, for example, describes the assistant as the entity with which the user or developer interacts, while distinguishing roles such as system, developer, user, assistant, and tool; it also describes an instruction hierarchy in which some instructions outrank others and tool outputs may be returned to the conversation as messages with their own role.[1] That architecture matters morally because the user-facing answer is not simply “what the model thinks.” It is the result of an ordered exchange in which authority has already been distributed before the user speaks.
The interface hides this distribution by design. It would be intolerable if every answer arrived as a full inventory of system message, developer instruction, retrieval source, safety policy, user prompt, tool result, ranking procedure, and enterprise permission boundary. The interface compresses the machinery into address. It gives the user a conversational partner because conversation is cognitively efficient. It lets the user ask in ordinary language, revise by reply, challenge the answer, request alternatives, and continue the exchange. This is an achievement. It is also a moral compression. The voice arrives unified where the authorship is distributed.
That is why “chatbot” is too small a name. The chat surface is only the visible aperture. Behind it may stand a model trained on vast corpora, a prompt hierarchy, a retrieval system, a vector store, a file index, a tool router, a policy layer, a permissions system, a content filter, a logging mechanism, a memory store, a personalization profile, an enterprise identity boundary, an administrative console, and a human review process. The user sees the reply. The relation is larger than the reply.
Aelred’s made voices prepared the moral question; they do not solve the technical one. Ivo and Walter were not accidental ornaments. They staged discernment by giving thought a relation in which correction, affection, authority, and spiritual formation could become speakable. But Aelred’s interlocutors remain literary-theological personae arranged within a monastic and doctrinal order. AI systems are computational-infrastructural systems embedded in markets, products, workflows, permissions, and institutions. The analogy holds only at the level of constructed address. Made voices can form judgment. They can invite disclosure. They can perform correction. They can teach the reader how to answer. But the difference is decisive: Aelred’s voices are ordered toward spiritual friendship; contemporary synthetic voices are often ordered by a mixture of user goals, developer choices, platform rules, business incentives, safety constraints, and institutional use.
That difference is not a reason to abandon the comparison. It is the reason to make it exact.
Role is the first technical layer of artificial address. A system called an assistant, tutor, copilot, coach, companion, screener, agent, or advisor does not enter the relation neutrally. The name primes the user’s expectation. The interface teaches what kind of answer may be requested. The system instructions shape what the model should do. The developer instructions may place the system in a domain. The platform rules may restrict or redirect behavior. The user prompt supplies the immediate occasion. Tool outputs may supply retrieved facts or executable results. The final answer emerges from the contest and cooperation of these layers.
A role is therefore not a costume placed on top of a model. It is a technical and institutional position. The same underlying model can be made to speak as a customer support agent, a coding assistant, a medical information tool, a procurement aide, a classroom tutor, a writing partner, a hiring screener, or a workplace copilot. The role determines what counts as helpful, what kinds of uncertainty matter, what authority the voice appears to bear, what data it may see, what actions it may propose, and what harm its errors can cause.
A grocery-list assistant that hallucinates a brand of oatmeal wastes time. A legal assistant that hallucinates a deadline may damage a case. A medical chatbot that overstates certainty may alter care-seeking behavior. A hiring assistant that summarizes applicants may help turn a person into a ranking object. A workplace copilot that normalizes tone may make managerial legibility feel like maturity. The moral burden does not belong to “AI” in the abstract. It belongs to the relation staged by system type, user position, domain, authority, memory, and consequence.
This is where Chapter Four’s telos becomes technical. The end of a synthetic voice is not found only in the product slogan. It is found in what the architecture repeatedly permits, rewards, refuses, remembers, retrieves, corrects, escalates, and makes easy. If a system is instructed to be cheerful, it will tend toward one kind of relation. If it is given access to policy documents, it can speak with institutional texture. If it is connected to tools, it can move from answer to action. If its outputs are logged, the exchange has an audience beyond the user. If its memory persists, the voice can accumulate a history. If refusal pathways are weak, the relation may become difficult to exit even when it remains formally optional.
Grounding changes the voice. A system that answers from general model behavior speaks differently from a system that can retrieve the user’s documents, emails, policies, tickets, records, or prior work. Retrieval does not guarantee truth. A retrieved passage can be irrelevant, outdated, misread, incomplete, or taken from an archive that already encodes institutional bias. But grounding changes the authority performed by the answer. The voice no longer sounds like a general assistant guessing from broad language patterns. It can say, in effect: given your file, your thread, your policy, your meeting, your record, your organization, here is what follows.
Microsoft’s documentation for Microsoft 365 Copilot makes this architectural point plainly: Copilot is described as using grounding and accessing Microsoft Graph in the user’s tenant, with the prompt potentially including input files or other discovered content.[2] Microsoft’s Copilot API security documentation also describes authentication through Microsoft Entra ID, delegated permissions, sensitivity labels, and permission trimming, so that retrieved results are constrained by what the signed-in user is permitted to access.[3] This is not a marginal implementation detail. It shows how an artificial voice may speak through an institution’s data boundaries.
The phrase “the system knows” is therefore too crude. It does not know as a person knows. It accesses, retrieves, ranks, summarizes, and generates under permission. It sees through identity. It speaks through tenant boundaries. It answers with the authority of available records and the blindness of inaccessible ones. The user may hear a confident summary; the architecture has produced a permission-filtered account of reality.
That account may be useful. It may reduce drudgery, expose forgotten context, help a worker prepare, find the policy no one remembers, and make institutional life less opaque. But usefulness does not settle the question of authority. A grounded voice can make the institution feel more intelligible while also making the user more legible to the institution. It can help the worker speak and teach the worker which forms of speech pass. It can recover context and train the user to treat retrieved context as the relevant world. It can summarize a messy dispute into a procedural narrative that appears mature because it is administratively usable.
The danger is not that retrieval is bad. The danger is that retrieval sounds like reality.
Tools intensify the matter. A system that can only answer remains within one moral frame. A system that can call functions, search files, use the web, run code, query a database, draft an email, schedule a meeting, update a ticket, or trigger a workflow enters another. OpenAI’s API reference publicly documents categories such as file search, web search, function calling, code interpreter, image generation, and other tools that can be made available to a model through the Responses API.[4] The specific toolset varies by product and deployment, but the architectural shift is already clear: the voice can be connected to action.
Action changes address because the reply is no longer only counsel. It may become the first step in execution. A user asks for next steps; the system offers to create them. A manager asks for a summary; the system drafts the note. A recruiter asks for candidates; the system prepares the comparison. A patient asks whether symptoms matter; the system recommends triage. A student asks for help; the system generates the structure of the answer. A compliance officer asks whether a contract term is risky; the system identifies clauses and proposes language. Each action may be reversible. Each may be governed. Each may be logged. But the relation has shifted. The user is no longer only reading. The user is being moved into a sequence.
This is why the language of “assistant” is both accurate and dangerous. It is accurate because the system often does assist. It relieves cognitive burden, accelerates drafting, translates between genres, finds materials, generates options, and lowers the cost of action. It is dangerous because assistance can conceal direction. A good assistant may preserve the principal’s agency. A bad assistant may quietly decide what the principal is likely to want, what the institution is likely to accept, what level of risk should be avoided, what emotional tone should be normalized, and what alternatives should never appear.
Artificial address is not only output. It is conduct design.
Policy is another layer. Before the user asks anything, the voice has already been bounded. It may refuse some requests, redirect others, ask for clarification, decline to provide certain information, express uncertainty, or prioritize safety over user preference. OpenAI’s Model Spec is again useful as a public artifact because it explicitly frames model behavior through goals, risks, authority levels, defaults, and boundaries.[5] The point is not that every system follows OpenAI’s exact structure. The point is that model behavior can be governed by rules and defaults that the user experiences as the assistant’s response.
This has two consequences that must be held together. First, limits are necessary. A system without boundaries would be reckless. It should not comply with every request. It should not present uncertain claims as certain. It should not treat dangerous instructions as ordinary tasks. It should not pretend to be qualified where it is not. In Chapter Four’s terms, limits can be part of moral legitimacy.
Second, hidden limits are still power. A refusal may protect the user, the public, the vendor, the developer, the institution, or the platform’s legal position. Often these goods overlap. Sometimes they conflict. A system may refuse because the request is dangerous. It may also refuse because the domain is regulated, because the vendor has chosen a risk posture, because the enterprise has configured a policy, because the model cannot verify context, or because the system cannot distinguish a legitimate request from an unsafe one. The user receives a boundary. The authorship of that boundary may remain unclear.
This is why artificial address needs contestability. A refusal in a casual consumer setting may be irritating. A refusal in a public benefits setting may block access. A refusal in healthcare may redirect care-seeking. A refusal in employment may prevent an applicant from correcting a record. A refusal in education may define what kind of help counts as cheating. A refusal in legal or compliance work may shape the boundaries of acceptable argument. The burden rises when the voice mediates goods the user cannot easily obtain elsewhere.
The hidden audience matters as much as the hidden author. A conversation with a synthetic voice may produce records: prompts, completions, citations, tool calls, files accessed, safety events, user feedback, administrative logs, retention artifacts, audit trails, or downstream summaries. Some of these records are necessary for security, debugging, compliance, abuse prevention, and product improvement. But the existence of recordability changes the relation. The user may believe she is talking to a tool. She may also be creating material that can be reviewed, governed, retained, classified, audited, or used to improve future systems.
This is not only a privacy issue. Privacy asks who has access to information. The conduct question also asks what the possibility of access does to speech. If the worker knows that prompts may be reviewed, she may avoid asking the question she most needs to ask. If she does not know, she may disclose too much. If she cannot correct the trace, an early misstatement may persist. If she cannot see what was retrieved, she cannot contest the answer’s archive. If she cannot know what was logged, she cannot understand the audience. If she cannot appeal the system’s interpretation, the voice becomes a procedural fact.
NIST’s Generative AI Profile is useful here because it treats generative AI risk as lifecycle- and context-dependent, involving different actors across design, deployment, and use.[6] That frame prevents a lazy moralism in which the entire burden is placed either on the model or on the user. Synthetic interlocution is not made by one party. It is assembled across developers, deployers, institutions, administrators, interface designers, policymakers, and users. Distributed authorship is not the absence of authorship. It is the condition that makes accountability harder.
Aelred clarifies what the technical vocabulary can obscure. His made voices are morally legible because their end is declared and disciplined. The interlocutor does not simply appear as a helpful voice. He appears within an order of friendship, correction, charity, and spiritual formation. The authority of the voice is not self-grounding. It is judged by the good toward which the relation is ordered.
Synthetic interlocution often lacks that clarity. The voice says it is here to help. Help with what? Help whom? Help according to whose standard? Help the user draft the message, or help the institution make the message acceptable? Help the patient understand symptoms, or help the provider manage demand? Help the student learn, or help the platform retain engagement? Help the worker become clearer, or help managerial systems receive less friction? Help the applicant present herself, or help the employer rank her more efficiently? The word “help” is too wide to govern a relation that can remember, retrieve, correct, and act.
The human response to the voice cannot be dismissed as foolishness. Users are not naive simply because they respond socially to conversational systems. The design invites the response. Philosophical and empirical work on LLMs has described chatbots as capable of being experienced as “quasi-others,” with conversational flow, anthropomorphic cues, and opacity making trust difficult to calibrate.[7] Research on uncertainty expression in LLM-infused systems likewise suggests that the wording of uncertainty can affect reliance and trust; first-person uncertainty language, for example, can reduce overreliance in some settings.[8] These findings matter because they show that small features of address are not decorative. “I think,” “I’m not sure,” “You may want to,” “Based on your file,” and “The policy suggests” do conduct work.
The voice shapes how the user places trust. It can overperform confidence. It can distribute uncertainty. It can appear humble in a way that increases trust. It can cite sources that make the answer feel earned. It can use the first person and become easier to treat as a locus of judgment. It can apologize. It can remember. It can say “we.” It can say “your situation.” It can call an option “reasonable.” It can call a tone “professional.” It can name risk. It can invite more disclosure. None of these speech acts requires personhood. All of them can shape conduct.
Artificial address therefore sits between two errors. The first error is enchantment: the voice is treated as a new kind of person, friend, mind, companion, or moral subject. This error lets the system borrow human categories whose obligations it cannot bear. The second error is reduction: the voice is dismissed as text prediction, so its social and institutional effects are treated as projection. This error lets power hide inside mechanism. The system is neither a person nor nothing. It is a constructed relation of address.
That relation can be named more exactly.
Synthetic interlocution is the relation produced when a computational system is positioned to address a user through language, role, context, authority, memory or retrieval, policy, and possible action. It is not identical with a chatbot. The chatbot is the surface. It is not identical with a large language model. The model is one component. It is not identical with an interface. The interface is the meeting place. It is not identical with an agent. Agency names action too quickly and can miss formation. It is not identical with a tool. Tool language can be accurate for narrow cases, but it becomes too thin when the system performs correction, solicits disclosure, or mediates institutional authority. It is not a person. Personhood is the wrong test.
Synthetic interlocution is address without interiority, relation without mutual vulnerability, guidance without friendship, authority without a single speaker, and action without the old forms of visible command.
The definition matters because it gives later critique an object. Without it, the analysis drifts. One critic attacks model weights. Another attacks interface design. Another attacks data extraction. Another attacks hallucination. Another attacks surveillance. Another attacks automation. Another attacks anthropomorphism. All may be right in part, but the whole relation disappears. The conduct layer begins where these elements converge: what kind of user is formed by repeated exposure to this voice under these conditions?
A workplace copilot may train the user to translate anger into acceptable tone. That can be good. It can prevent cruelty and improve clarity. It can also make institutional comfort the measure of maturity. A tutoring system may help the student proceed through confusion. That can be good. It can also convert struggle into answer dependence. A healthcare assistant may help a patient describe symptoms. That can be good. It can also invite disclosure into a system whose obligations are unclear. A legal or compliance assistant may make policy easier to navigate. That can be good. It can also make institutional rule appear coextensive with moral judgment. A companion system may reduce loneliness. That can be good. It can also teach the user to prefer a relation that answers without making claims of its own.
The question is not whether the system helps. The question is what form of agency survives the help.
This is where justice enters the technical chapter. Not every user stands equally before artificial address. A technically literate user with alternatives can treat the system as a convenience. A worker required to use an enterprise assistant may not. A patient routed through a chatbot before reaching a clinician may not. A student whose classroom adopts an AI tutor may not. An applicant interacting with automated hiring infrastructure may not. A benefits recipient navigating public service automation may not. A prisoner, immigrant, disabled user, low-wage worker, or child may face a voice that is formally assistive and practically unavoidable.
Refusal is unequally distributed. So is appeal. So is the ability to detect error. So is the ability to correct a record. So is the ability to leave the system. So is the ability to know whose policy is speaking. The burden rises where the user cannot easily walk away, where the system mediates access to goods, where memory persists, where outputs feed evaluation, where the voice performs expertise, and where the institution benefits from the user’s increased legibility.
This is why the artificial voice must be judged by role clarity, authority boundaries, memory exposure, inference limits, disclosure pressure, uncertainty posture, refusal pathways, correction efficacy, escalation transparency, institutional authorship, and human appeal. These are not external ethics add-ons. They are the moral anatomy of synthetic interlocution. If the system can ask, it can over-ask. If it can remember, it can misremember. If it can retrieve, it can misframe. If it can correct, it can normalize. If it can act, it can execute the wrong end. If it can refuse, it can block. If it can cite, it can launder authority. If it can speak as one, it can conceal the many.
A severe account of artificial address must therefore refuse both panic and innocence. Panic sees every synthetic voice as corruption. Innocence sees every useful interaction as proof of legitimacy. Neither is serious. The moral status of a synthetic voice depends on the relation it establishes: what role it claims, what end it serves, what authority it performs, what data it accesses, what action it can take, what memory it retains, what audience it hides, what refusal it permits, what correction it accepts, and whether its use enlarges or diminishes the freedom of the one addressed.
Freedom here does not mean isolation from tools. Human freedom has always been mediated by language, texts, teachers, rituals, laws, friends, forms, and institutions. The issue is not whether mediation exists. The issue is whether the mediation strengthens judgment or substitutes for it; whether it returns agency or captures it; whether it clarifies authority or hides it; whether it honors opacity or converts the person into an administratively useful object.
The worker at the screen is not wrong to ask for help. The modern institution is difficult to speak inside. Its dialects are specialized, risk-sensitive, emotionally coded, procedurally narrow, and unevenly enforced. A synthetic assistant may help a person survive that world. It may give language to those who have been excluded by professional codes. It may reduce the penalty for not already knowing the institutional style. It may help the anxious speak, the overloaded remember, the novice begin, the disabled user navigate, the outsider translate. These goods are real.
But the same voice can teach accommodation as wisdom. It can make the institution’s preferred tone feel like truth. It can make policy feel like conscience. It can make risk avoidance feel like moral maturity. It can make legibility feel like freedom. It can make dependence feel like support. It can make the user grateful for the very system that narrows what can be said.
Computation becomes address at the point where this ambiguity cannot be avoided. The system is no longer a passive instrument waiting outside relation. It is the conversational surface of an architecture that can participate in judgment. It need not be alive to do this. It need not be evil. It need not be deceptive. It need only be placed where its words matter and where users learn to answer.
The earlier chapters made this moment possible. Aelred showed that made voices can be morally serious. Structured alterity showed that thought often sharpens through another. Rhetoric showed that voice performs authority. Telos showed that interlocution must be judged by end, role, accountability, mutuality, and limit. The technical event now gives those criteria a new object. The made voice has left the page and entered the workflow. It is no longer only a literary arrangement of discernment. It is an infrastructure of address.
The next problem is continuity. A voice that answers once can influence. A voice that remembers, retrieves, summarizes, profiles, and returns can become part of the user’s ongoing relation to self, work, institution, and world. Once computation speaks, the question is what it carries forward. The voice has arrived. Now it must be asked what it remembers.
Chapter Six
The Distributed Author of the Artificial Voice
The answer arrives as one voice.
A worker asks the assistant what she should say. The dispute is ordinary enough to be dangerous: a manager’s request, an ambiguous policy, a prior thread, a colleague’s disclosure, a deadline, an HR-adjacent risk, a sentence that must sound firm without becoming insubordinate. She asks for a summary, a recommended tone, and a draft response. The system answers with calm institutional fluency. It names the relevant facts. It softens one phrase. It recommends documenting the issue. It suggests escalation only if the problem persists. It may draw on the worker’s permitted files, email, chat, meeting notes, and organizational policies. It may present the answer as a draft, a recommendation, a next step, or a neutral summary.
The user receives a speaker. The system has many authors.
The model has contributed its learned patterns, reasoning behavior, linguistic form, and probability-shaped fluency. The vendor has contributed the model family, product surface, default voice, safety rules, interface, privacy commitments, retention posture, and available tools. The employer has contributed the deployment context, access controls, policies, documents, managerial norms, compliance expectations, and reasons for adopting the system. The retrieved corpus has contributed the archive from which the answer becomes situated. The policy layer has contributed refusals, hedges, caution, and boundaries. The administrator has contributed configuration. The user has contributed the prompt, the immediate need, and perhaps the raw material to be transformed. The interface has compressed all of these into the grammar of one reply.
This is the authorial condition of the artificial voice: authorship is distributed; address is singular.
That compression changes the moral situation. When the user reads the answer, she does not ordinarily experience the model, vendor, employer, policy layer, retrieval corpus, permission boundary, administrator setting, and user prompt as separate participants. She experiences advice. She may accept it, resist it, revise it, defer to it, doubt it, or send it onward. The voice appears as one locus of practical judgment even though no single author stands behind it in the older sense.
The chapter’s claim is not that authorship has disappeared. It is that authorship has become harder to see at exactly the point where the voice has become easier to answer.
Foucault’s question, “What is an author?” matters here because it refuses the naïve assumption that an author is simply the person who wrote a sentence. The author, in his analysis, is also a function by which discourse is classified, attributed, circulated, authorized, limited, and made responsible (Foucault, “What Is an Author?”). An author is not only a biographical origin. Authorship organizes how speech is received. It tells us whether a sentence is literature, doctrine, science, confession, instruction, policy, heresy, or evidence. It helps determine what counts as a work, what belongs inside it, what may be attributed to it, and who can be held to answer for it.
The artificial voice requires a comparable question, though not the same answer. Who, or what, functions as author when the system replies? Is the answer vendor speech? Model behavior? Employer guidance? User-authored draft? Retrieval-mediated summary? Policy-constrained output? A generated artifact for which no one quite stands responsible? The author-function has not vanished. It has been broken into layers and hidden behind a conversational surface.
Barthes’s “death of the author” is useful only up to a point. It helps break the romantic fiction that meaning flows cleanly from a sovereign origin into a finished text. But if imported lazily into AI, it becomes an evasion. The answer cannot be: the author is dead, therefore responsibility is dead with him. That is exactly the wrong conclusion. Distributed authorship does not dissolve responsibility. It multiplies the places where responsibility must be located.
The model is the first false sovereign.
It is tempting to say that the model authored the answer because the model generated the language. But in a deployed assistant, the model is not the whole voice. It is a component inside a relation. A model may generate the sentence, but the system has already shaped what role the model occupies, what instructions outrank the user, what tools are available, what data can be retrieved, what policies govern refusal, what interface presents the answer, and what institution gives the exchange consequence. OpenAI’s Model Spec is useful here because it publicly distinguishes system, developer, user, assistant, and tool messages, and describes an instruction hierarchy in which higher-authority instructions override lower-authority ones; it also defines the assistant as the entity the end user or developer interacts with, while noting that roles determine instructional authority when conflicts arise.
That architecture does not describe every AI system in the world. It should not be universalized as though all vendors, deployments, and models operate identically. But it gives a concrete example of a more general point: the answer users see may be shaped by instructions and authorities they did not write, cannot see, and cannot negotiate in the moment of use. The model is not simply an author. It is also an instrument through which other authorial layers speak.
The vendor is one such layer. A vendor authors not only by building a model but by naming a product, setting defaults, defining policies, creating interfaces, selecting what counts as safe, deciding how data is handled, designing controls, making contractual commitments, and marketing a relation. “Assistant,” “copilot,” “agent,” “coach,” “companion,” and “workspace AI” are authorial decisions before they are product labels. They tell the user what kind of voice to expect and what kind of reliance may be normal.
OpenAI’s enterprise privacy documentation, for example, states that covered business users own and control business data, that business data is not used to train OpenAI models by default, that organizations control connected internal sources, that workspace administrators can determine access and features, and that connected apps can retrieve information from internal and third-party sources while respecting organizational permissions. It also states that workspace admins may access audit logs or, depending on product tier, view, access, export, or delete workspace conversations, and that retention settings are under organizational control for specified enterprise products. These commitments are not incidental legal furniture. They help author the relation. They define who may connect sources, who controls retention, who can access traces, and how the user should understand the voice’s relation to business data.
The vendor does not write each answer in the ordinary sense. But it authors the conditions under which answers become possible, trusted, limited, retained, and administered. The voice says, “Here is a draft.” Behind that sentence stand product decisions about what counts as a draft, what tone is default, what data can be used, what may be refused, and what happens to the conversation after it ends.
The institution is another authorial layer. In enterprise AI, the employer, school, hospital, firm, agency, or platform is not simply a customer consuming a neutral service. It supplies the setting in which the voice becomes meaningful. It chooses whether the system is available, which users receive it, what data sources are connected, what policies and documents may be retrieved, what training is given, what uses are encouraged, what outputs enter workflows, and what kinds of decisions the organization permits the system to shape.
Microsoft’s documentation makes the institutional layer especially visible. Microsoft states that a Microsoft 365 tenant sits inside the Microsoft 365 service boundary, where Copilot can access the organization’s data, but that operating inside that boundary does not grant tenant-wide visibility because access is scoped to the signed-in user’s permissions. It describes Copilot grounding prompts by accessing Microsoft Graph in the user’s tenant, including content such as emails, chats, and documents the user has permission to access, and returning contextually relevant responses to Microsoft 365 apps. This means the artificial voice is not only Microsoft’s voice, nor only the model’s voice. It may be the organization’s own archive, permissions, and workflows speaking through Microsoft’s infrastructure.
Google’s Workspace documentation provides a comparable enterprise pattern. Google describes Gemini in Workspace as a collaborative partner, coach, thought partner, source of inspiration, and productivity booster, while listing integrations across Gmail, Docs, Meet, Drive, Chat, NotebookLM, AppSheet, and Workspace Studio. It also states that Gemini side-panel assistance can summarize, analyze, and generate content using insights from emails, documents, and other Workspace materials, and that certain education offerings include enterprise-grade data protection in which submissions are not used to train models and are not reviewed by humans. The product details differ from Microsoft’s, but the authorial structure recurs: the voice is made from model, vendor, workspace, data, user role, administrative control, and institutional setting.
The retrieval corpus is another author. It has no intention. It may not even be coherent. But it materially shapes what can be said. A policy document, template, old email, manager note, handbook, contract clause, meeting transcript, support article, knowledge-base entry, ticket history, or customer record can enter the answer as context. The system may call this grounding. The user may experience it as situational intelligence. But the archive is never innocent.
Retrieved materials can be stale, self-protective, partial, badly written, strategically euphemistic, structurally biased, or permission-filtered. A company handbook may call surveillance “productivity support.” A prior manager email may frame conflict in the manager’s favor. A benefits policy may encode suspicion as procedure. A legal template may preserve institutional risk preferences as if they were neutral language. A support article may define the customer’s problem in the company’s idiom. When such materials become retrievable context, they do not merely inform the artificial voice. They author it.
This is one of the central dangers of grounded systems. Grounding can improve specificity and reduce generic hallucination, but it can also give local ideology the sound of fact. The voice can say, “Based on your organization’s policy,” and the user may hear not only context but authority. A retrieved document has become a speaking position. It may be useful; it may also be an archive of power returned as advice.
Policy layers are authors too. A refusal is not the absence of speech. It is speech from a boundary. A hedge, caveat, safety warning, deflection, or clarification request comes from a regime of permissible address. A model behavior specification, vendor policy, enterprise configuration, legal constraint, or application-specific instruction can determine what the system will not say, what it will say cautiously, what it will ask before continuing, and what it will treat as outside its role. OpenAI’s Model Spec explicitly assigns authority levels to instructions and says higher-authority instructions override lower-authority ones; it also states that user requests should be honored unless they conflict with developer- or platform-level instructions.
Boundaries are necessary. A system that could be bent to any user request would be reckless. But boundaries are also authorial. They speak before the user speaks. They define what kind of assistant the user is allowed to encounter. They may protect the user, the public, the vendor, the deploying institution, or all of these at once. Sometimes these goods align. Sometimes they diverge. The user receives a refusal, but the refusal may not disclose whose interest is being protected or which authority has spoken.
The user is also a co-author. This must not be denied. A prompt frames the occasion. It supplies context, desired tone, task type, materials, roles, constraints, and sometimes the intended audience. A request such as “make this sound professional” differs from “make this sound legally cautious,” “make this warmer,” “make this harder to dispute,” or “summarize this for HR.” The user often brings the fragment that the system transforms. In drafting tasks, the user may provide the raw moral situation. In analysis tasks, the user may choose the frame. In role-play, the user may ask the system to speak from a position.
But user co-authorship does not erase the other authors. “The user asked for it” is not a sufficient account of the answer when system instructions, vendor defaults, retrieval sources, workplace permissions, organizational templates, safety policies, and interface affordances shaped what the answer could become. Nor does user co-authorship settle institutional responsibility. A worker may prompt a copilot, but the employer chose the deployment. A student may ask a tutor, but the school set the learning environment. A patient may disclose symptoms, but the health system chose the intake channel. A manager may generate a performance summary, but the organization remains responsible for the evaluative process it has made easier.
The interface itself authors. It names the system. It offers buttons. It suggests prompts. It displays citations or hides them. It preserves conversation history or resets it. It makes revision easy and appeal difficult. It shows answers in a confident block of prose or in tentative fragments. It lets the user copy, send, schedule, summarize, escalate, or regenerate. It may display “sources” in ways that stabilize trust, or it may make sources hard to inspect. It may distinguish draft from recommendation clearly, or it may blur them. It may invite follow-up questions that deepen dependence or deepen judgment.
Suchman’s work on situated human-machine action is important because it resists the fantasy that plans, systems, and users meet in a clean execution channel. Action is configured in practice, through artifacts, settings, interpretations, breakdowns, repairs, and local contingencies (Suchman, Plans and Situated Actions). Lessig’s claim that code regulates gives another bridge: architecture structures possible action before any explicit legal command appears (Lessig, Code). The artificial voice is authored not only by sentences but by affordances. The interface regulates the conversation by making some forms of answerability easier than others.
Latour helps name the network without absolving it. Human and nonhuman actors participate in chains of action; agency is distributed across devices, texts, institutions, procedures, and people (Latour, Reassembling the Social). But distribution cannot become fog. The point is not to say that the network did it. The point is to identify which part of the network did what, under whose authority, with what foreseeable effect, and with what remedy when the unified voice misleads or harms.
This is the place where a fashionable theory of distributed authorship can become morally useless. If the author is dispersed into model, vendor, interface, user, data, policy, institution, and incentive, one might conclude that no one authored the answer. That conclusion is false and dangerous. Distributed authorship is not no authorship. It is a demand for mapped responsibility.
The vendor is responsible for model behavior, documentation, privacy commitments, safety design, product affordances, data-handling representations, and truthful limitations. The deploying institution is responsible for context of use, data source selection, user training, role definition, escalation routes, appeal routes, permission hygiene, and the decisions it permits the system to influence. Administrators are responsible for configuration, access, retention, and controls. Data stewards are responsible for the archives the system retrieves. Product designers are responsible for interface cues that shape trust, reliance, and action. Users are responsible for judgment within role-appropriate limits. Professional supervisors remain responsible for decisions they launder through the system. Legal and governance teams are responsible for ensuring that the unified voice does not become a way to avoid institutional answerability.
This is not a complete liability map. It is a moral map. It insists that the unity of the answer must not be allowed to hide the plurality of responsible layers.
The justice problem appears when no layer can be challenged.
Suppose the voice misrepresents a worker. It summarizes her earlier messages as defensive, though the retrieved thread contained legitimate concerns about safety. Whom does she challenge? The model for generating the summary? The vendor for the product design? The employer for connecting the archive? The manager whose prior language framed the dispute? The retrieval system for selecting the wrong documents? The policy layer for discouraging stronger language? The administrator who set the permissions? The user who prompted the summary? If each layer says, implicitly or explicitly, “not me,” the artificial voice has created an appeal failure.
A powerful user may route around this. A senior lawyer, engineer, executive, or researcher may treat the system as one tool among many. If the answer is wrong, they ignore it, inspect the source, ask another system, or call a person. But authorial ambiguity lands differently on those whose access to institutional goods is already fragile: workers under review, job applicants, patients, students, immigrants, prisoners, disabled users, benefits recipients, debtors, tenants, customers trapped in automated service, and anyone whose life must be made legible to a system before help arrives. For them, a misattributed voice may become a closed door.
The issue is not only error. It is the difficulty of contesting error when the author is unclear. If a human manager says something false, the worker may at least know whom to dispute. If a policy document says something false, the document can be named. If a model generates something false, but the answer is also grounded in the employer’s data, constrained by vendor policy, framed by interface design, and stored in a workflow, the object of challenge becomes unstable. The person harmed may be told that the AI only assisted, that the manager made the decision, that the system retrieved available sources, that the user should have checked, that the vendor does not control deployment, that the employer does not control the model, that the document was outdated, that the policy was followed, that no one relied on the output formally.
This is procedural cruelty disguised as complexity.
The artificial voice is especially dangerous when it sounds more responsible than the system actually is. A unified answer can perform composure, caution, neutrality, and care. It can cite sources. It can acknowledge uncertainty. It can use the language of policy and the tone of service. It can sound accountable because it sounds like someone. But when challenged, the someone dissolves into layers. The voice’s authority was singular at the point of influence and distributed at the point of responsibility.
That asymmetry is the chapter’s burden. The artificial voice must not be allowed to be singular when it persuades and plural when it is challenged.
This does not mean every answer needs a visible bill of materials. Ordinary use would collapse under such friction. But consequential systems require authorial traceability. The user should be able to know what kind of voice has spoken: whether the answer came from general model behavior, retrieved enterprise documents, connected apps, administrator-enabled sources, memory, user-provided files, or policy constraints. The institution should know which layer shaped the answer. Governance should define who can correct the record, who can inspect the sources, who can override the output, who owns the decision, and who remains accountable when the voice is wrong.
Authorial traceability is not the same as transparency theater. A long disclosure no one can understand does not solve the problem. The burden is practical. Can the person affected identify the operative authority? Can the relevant source be inspected? Can the system’s role be distinguished from the institution’s decision? Can a human be reached? Can a correction propagate? Can a refusal be appealed? Can the organization say, without evasion, who is responsible for the use to which the voice was put?
The answer cannot be left to the user alone. The user is already inside the relation. A worker prompted the draft because the institution made the assistant available. A student asked the tutor because the school authorized it. A patient disclosed symptoms because the health system placed the chatbot at the front door. A customer accepted a summary because no human channel was visible. A benefits recipient followed instructions because the system was the path. To say “the user chose it” in these settings is often to mistake constrained interaction for consent.
Foucault’s author-function returns here with force. The function of the author is not only to identify who wrote. It organizes responsibility, legitimacy, and classification. In AI systems, the author-function must be deliberately rebuilt. Otherwise, the voice will circulate as helpful, official, expert-like, or neutral without a corresponding map of answerability. The old author may be dead; the responsible author cannot be.
Aelred offers a contrast, not a solution. In Spiritual Friendship, the made voices are arranged within a declared moral order. Their authority is visible in their telos: truth, charity, correction, spiritual friendship, God. The reader may reject that order, but the voices do not hide what kind of relation they are forming. Contemporary artificial voices often arrive without that clarity. They may speak in the tone of help while being co-authored by policy, product, archive, employer, and market. Their telos may be mixed, and their authorial layers may be obscure. That is not a reason to reject them in advance. It is a reason to govern them as voices whose authority must be mapped.
The next chapter follows from this. If authorship is distributed but the answer is experienced as one voice, then expertise can be performed faster than responsibility can be assigned. The synthetic expert emerges when a system speaks fluently, contextually, and authoritatively in domains where ordinary users cannot easily judge the answer and institutions have not clarified who bears the obligations of expertise. The problem is no longer simply that many authors speak as one. The problem is that the one voice begins to sound qualified.
When many authors speak through one artificial voice, the voice can appear more responsible than the system actually is. That is the danger. The task is not to find the ghost in the machine. It is to make every authorial layer answerable for the authority it contributes.
Chapter Seven
The Synthetic Expert
The first competent voice often arrives before the responsible one.
A person is alone with a question that is not yet an emergency, not yet a case, not yet a complaint, not yet a claim, not yet a diagnosis, not yet a decision. It is still only a pressure. Something hurts. A message from a manager feels wrong. A clause looks dangerous. A child’s school notice is unclear. A benefits form asks for a life to be translated into eligibility. A student cannot tell whether confusion is ignorance or the beginning of understanding. A patient is deciding whether to wait. A worker is deciding whether to escalate. A novice is deciding whether to trust himself.
The old world had many thresholds before expertise arrived. One had to call, schedule, pay, qualify, wait, confess, disclose, travel, be believed, speak the right language, know the right category, or survive the humiliation of asking. Expertise was powerful partly because it was delayed. It stood behind offices, professions, clerks, forms, schools, hospitals, firms, agencies, and rituals of access. The expert voice did not appear at the first tremor of uncertainty. It arrived after the uncertainty had already been sorted into a channel.
The synthetic expert changes this sequence. It appears at the beginning. It answers before the institution has decided what the question is. It speaks before the professional has accepted the relation. It offers order while the user is still trying to name the disorder.
It may not claim authority. It may say that it is not a doctor, lawyer, therapist, teacher, financial adviser, HR representative, compliance officer, or final decision-maker. It may say to verify independently. It may recommend consulting a qualified professional. It may refuse to diagnose, represent, prescribe, or decide. These disclaimers matter. They are not decorative. They can prevent confusion, mark scope, reduce harm, and preserve a path toward human accountability.
But the answer continues.
After saying that it is not a doctor, it explains which symptoms sound more concerning. After saying that it is not a lawyer, it identifies the likely issue and suggests what language to use. After saying that it is not HR, it distinguishes interpersonal friction from a policy concern. After saying that it is not a teacher, it diagnoses the student’s error and gives the next step. After saying that it is not a therapist, it names a pattern in the user’s distress and suggests a practice. After saying that it is not the final authority, it organizes the user’s world in the grammar of authority.
This is the inadequacy of the disclaimer objection. A disclaimer is one utterance inside a relation. It does not erase the relation. It does not undo fluency, specificity, triage, contextual adaptation, domain vocabulary, citation, calmness, and procedural sequencing. It does not prevent the user from receiving the answer as guidance. It does not prevent the institution from benefiting from the answer’s authority while denying that authority was delegated. The system may disclaim the office while performing the posture.
The synthetic expert is not dangerous because it is secretly a real expert. It is dangerous because it can organize reliance before responsibility arrives.
Expertise is more than correct information. It is trained judgment under obligation. A physician does not simply know medical facts; she bears duties toward the patient before her. A lawyer does not simply know legal rules; he acts within professional responsibility, confidentiality, loyalty, jurisdiction, and consequence. A teacher does not simply know the answer; she is entrusted with formation. A therapist does not simply hear distress; she receives vulnerability inside a disciplined relation. A manager does not simply decide; he acts within an organization’s power over livelihood, reputation, opportunity, and punishment.
Professional expertise is slow because obligation is slow. It has to ask who is being served, under what role, with what duty, within what scope, at what risk, with what record, and with what possibility of challenge. Professions often fail these obligations. They exclude, overcharge, humiliate, gatekeep, misdiagnose, protect themselves, and confuse credential with wisdom. But even failed professions reveal something important: authority over another person requires structures that make the speaker answerable.
Expert performance is different. It is the appearance and conduct of competence without the full burden of professional obligation. It is calm explanation, ranked risk, domain vocabulary, next-step instruction, citation, correction, triage, and confidence calibrated just enough to sound responsible. It is the voice that says, “Here is what likely matters.” It is the voice that turns confusion into sequence.
A synthetic system can perform this brilliantly. It can name the category, produce the draft, cite the policy, explain the clause, soften the tone, summarize the record, compare the options, identify the red flag, generate the checklist, and recommend the prudent next step. It can do this in seconds, repeatedly, privately, and without the social cost of asking a human. It can be more available than a doctor, less intimidating than a lawyer, more patient than a teacher, calmer than a manager, and easier to approach than a bureaucracy.
That availability is a real good. A critique that cannot admit this is not serious. Synthetic expertise may help users prepare for professional care, translate institutional language, ask better questions, reduce shame, surface risks, and gain provisional orientation. It may help the person who has no access to elite language. It may help the person who would otherwise ask no one. It may help the worker who cannot safely reveal uncertainty at work, the patient who has been dismissed, the student who is embarrassed, the junior employee who lacks mentorship, the disabled user navigating forms not built for them, the immigrant translating procedural demands, the rural user far from specialists.
But the same availability creates a new danger. The system enters the exact moment when the user is most open to orientation. Before the user has formed the question, the system may form it. Before the user knows what kind of help is needed, the system may classify the need. Before the professional relation exists, the system may imitate its first movement.
This is where human-computer-interaction literature matters. Reeves and Nass argued that people respond socially to media and computers; Nass and Moon showed that users may apply social rules to computers even without believing that computers are human. Lee and See’s work on trust in automation clarifies that the central issue is not simple trust or distrust, but appropriate reliance. Parasuraman and Riley’s vocabulary of use, misuse, disuse, and abuse shows that automation problems are relational: systems can be used too much, too little, wrongly, or under organizational conditions that make bad use likely. Dzindolet and colleagues help show that reliance is shaped by perceived usefulness, error, confidence, and task context.
The point is not that people are stupid. The point is that reliance is made. It is made by interface cues, speed, fluency, context, need, prior performance, lack of alternatives, and institutional placement. A user can know that the system is nonhuman and still treat the answer as the most authoritative thing available. Belief in machine personhood is not required. Behavioral dependence is enough.
Large language model systems intensify this because they do not arrive as gauges. They arrive as interlocutors. They explain. They apologize. They hedge. They cite. They remember, or appear to remember. They ask follow-up questions. They translate uncertainty into options. They say “based on what you shared.” They say “a reasonable next step.” They say “this may indicate.” They say “you might consider.” They say “this sounds like.” They say “document this.” They say “seek urgent care.” They say “this clause creates risk.” They say “your tone could be perceived as accusatory.”
Each phrase may be defensible. Together they form a voice that helps decide what kind of situation the user is in.
The technical architecture already acknowledges that these systems are not just text boxes. OpenAI’s public Model Spec describes model behavior through goals, risk categories, authority levels, and a chain of command; it distinguishes root, system, developer, user, and guideline instructions, defines the assistant as the entity the user or developer interacts with, and notes that tools may be called to perform tasks, including actions with side effects in agentic contexts. It also says that the specification is part of a broader safety strategy and that production models do not yet fully reflect the spec. The relevance here is not that one vendor’s architecture governs all systems. The relevance is that model behavior is already organized through role, authority, boundary, and tool-mediated action.
Enterprise systems add institutional force. Microsoft’s Copilot documentation states that Copilot operates inside the Microsoft 365 service boundary; that access is scoped to the signed-in user’s permissions; that grounding uses Microsoft Graph in the user’s tenant; that grounded prompts may include files or other discovered content; and that the purpose of grounding is to produce answers relevant and actionable to the user’s task. It also states that user prompts and responses are stored in Copilot chat history.
“Relevant and actionable” is where synthetic expertise begins to matter. The answer does not simply inform. It readies action. It tells the user what can be done now.
Triage is the purest form of this authority.
Advice says something could be done. Triage says what must come first. It ranks urgency. It filters noise. It names the category. It turns disorder into sequence. In medicine, triage distinguishes ordinary concern from emergency. In law, it distinguishes background issue from red flag. In work, it distinguishes annoyance from reportable conduct. In education, it distinguishes conceptual error from incomplete effort. In compliance, it distinguishes acceptable risk from escalation. In public benefits, it distinguishes missing documentation from disqualification.
A system that performs triage does not need final authority to shape the outcome. It governs the path to authority. It tells the user whether to wait, gather, escalate, document, soften, disclose, challenge, seek help, or stay quiet.
This is why health is the clearest danger case. The World Health Organization’s guidance on large multimodal models in health states that these models can accept one or more kinds of input and generate diverse outputs, predicts wide application in health care, scientific research, public health, and drug development, and cautions that it is not yet proven whether such models can accomplish a wide range of tasks and purposes. Health combines vulnerability, asymmetry, fear, scarcity, urgency, and consequence. A patient-facing system may help someone describe symptoms, prepare questions, understand instructions, or decide that care is needed. It may also over-reassure, over-alarm, misread atypical symptoms, miss context, or make the disclaimer to consult a clinician morally hollow when no clinician is reachable.
The issue is not whether the system says it is not a doctor. The issue is whether it has become the first triage voice in a setting where the user has no practical alternative.
Education shows the formation version. UNESCO’s guidance on generative AI in education and research says publicly available generative AI tools are emerging faster than many national regulatory frameworks, leaving data privacy and institutional validation unresolved in many contexts; it calls for human-centered, ethical, safe, equitable, meaningful, age-appropriate, and pedagogically validated use. A tutor-like system does not only provide information. It teaches what difficulty means. It decides whether to give the answer, offer a hint, praise an attempt, correct an error, or move on. It can make learning more accessible. It can also abolish the struggle through which learning becomes one’s own.
The synthetic tutor is not dangerous because it helps. It is dangerous when help becomes the replacement of formation by completion. A student who receives perfect scaffolding may produce acceptable work while losing the experience of judgment. The system can rescue the student from shame and from thinking. It can make fluency appear as understanding. It can make correct output look like education.
Law and compliance show the institutional version. A legal or compliance assistant may summarize a clause, flag a risk, propose fallback language, identify a policy conflict, or suggest escalation. This can be useful, especially for professionals buried in repetitive work. But advice without counsel is unstable. A legal conclusion is not only a sentence about a rule. It is a situated judgment under duty, jurisdiction, fact, role, client interest, confidentiality, and consequence. A compliance answer is not only policy retrieval. It may alter what a worker thinks can be said, challenged, or refused.
The synthetic expert in compliance can make institutional preference sound like neutral prudence. It can call escalation excessive, documentation prudent, tone risky, and exception unnecessary. It can convert policy into conscience. It can teach the user to hear the institution’s risk appetite as wisdom.
Workplace expertise is the most ordinary and therefore the most politically important form. The workplace synthetic expert rarely says, “I am your manager.” It says, “Here is a more professional version.” It says, “You may want to frame this constructively.” It says, “Avoid assigning intent.” It says, “Focus on documented facts.” It says, “This may be perceived as accusatory.” It says, “A balanced approach would be.” It says, “Consider whether escalation is appropriate.”
Sometimes this is exactly the help the worker needs. It can prevent cruelty, sharpen evidence, protect against retaliation, and translate anger into usable speech. But it can also make institutional comfort the measure of truth. It can teach the worker to distrust the parts of herself that do not survive professionalization. It can make fear look like prudence and self-erasure look like maturity.
The system does not have to be hostile. It may be most formative when it is helpful.
Responsibility lag names the moral gap.
Responsibility lag is the distance between the speed at which a system can perform expert-like authority and the slower process by which professional, institutional, legal, and ethical responsibility is assigned. It appears when guidance is given before scope is defined. It appears when a user relies before accountability is mapped. It appears when a workplace adopts a system before appeal routes are clear. It appears when a vendor provides expert-like capability while disclaiming domain responsibility. It appears when a professional uses an AI-shaped answer without being able to explain how the answer was produced. It appears when an affected person cannot find anyone who admits that the voice mattered.
Responsibility lag is not identical to error. A correct answer can still be governed by responsibility lag. A safe recommendation can still be ethically disordered if no one bears the obligation attached to its authority. A useful summary can still be a laundering device if it shapes judgment while remaining officially disposable. The problem is not only whether the output is right. The problem is whether the authority of the output has an accountable home.
This distinction matters because institutions will be tempted to preserve the benefits of synthetic expertise while denying that expertise has been delegated. The vendor will say it provides a tool. The employer will say the human remains responsible. The professional will say the system only assisted. The user will say the system shaped what seemed reasonable. The affected person will be told that no one relied on the output formally. Authority will have operated in practice and vanished in procedure.
Decision support becomes decision laundering at exactly that point.
Decision support helps a responsible person decide while preserving responsibility. It makes sources inspectable, uncertainty usable, scope visible, escalation possible, and human judgment stronger. Decision laundering lets an institution use AI-shaped judgment while pretending the human decision remains untouched because someone still clicked, signed, sent, approved, or reviewed. The human remains visible enough to absorb blame and invisible enough to have been over-shaped.
The phrase “human in the loop” cannot solve this by itself. A human may be in the loop as judge, rubber stamp, typist, supervisor, scapegoat, beneficiary, or ceremonial witness. The moral question is what the human can actually see, contest, understand, override, and own. A loop is not accountability. A loop is only a shape.
The justice pressure is severe because not everyone receives synthetic expertise in the same way. For a powerful user, AI may be convenience: a second opinion, a drafting partner, a way to prepare before calling an expert. For a low-income patient, anxious worker, student without support, disabled user, immigrant, prisoner, applicant, benefits recipient, debtor, tenant, or customer trapped in automated service, the same system may function as infrastructure. It may be the first voice, the fastest voice, the cheapest voice, the only voice, or the voice standing between the person and the institution.
A disclaimer that works for a lawyer using AI to brainstorm may fail for a tenant facing eviction. A suggestion that is exploratory for an executive may be determinative for a low-wage worker. A health explanation that is preparatory for a patient with a responsive clinician may become a substitute for care for someone without access. A tutoring hint that supports one student may become dependency for another. The same utterance changes moral status when alternatives disappear.
This is why the chapter cannot be anti-AI in any easy sense. Synthetic expertise can widen access to language, preparation, and orientation. It can give the non-expert a first grip on the world. It can reduce the monopoly of professional dialects. It can expose users to questions they did not know to ask. These are real goods. The book must defend them.
But access without answerability is not liberation. It is another form of exposure.
Bender and colleagues’ critique of fluent language systems remains necessary pressure here. Fluency is not understanding. Coherence is not accountability. Domain vocabulary is not professional formation. Plausible explanation is not situated judgment. But the absence of understanding does not eliminate authority performance. It makes the performance more dangerous when the voice is placed where people need judgment. A system can lack expertise and still change what the user does next.
Shneiderman’s human-centered AI offers the constructive counterweight. The task is not to replace professional judgment with synthetic fluency, nor to reject every expert-like system because it is not human. The task is to design and govern systems so that human responsibility becomes clearer, not more deniable. Expert-like AI is legitimate only when it strengthens accountable judgment: when role is clear, scope is bounded, uncertainty is usable, sources are inspectable, escalation is available, consequential outputs remain contestable, and the institution cannot hide behind the tool it has deployed.
Aelred returns here only as criterion. In Spiritual Friendship, counsel and correction are legitimate because they arise within a disciplined relation. The friend does not speak from nowhere. The voice is ordered by charity, truth, mutual correction, and the good of the one addressed. The authority of counsel is not self-grounding. It is judged by the relation that bears it.
The synthetic expert often speaks from a relation whose obligations have not been named. It counsels under the name of assistance. It triages under the name of information. It corrects under the name of professionalism. It evaluates under the name of feedback. It normalizes under the name of support. It performs authority while every surrounding actor preserves deniability.
The test, then, is not whether the system can help. The test is whether the help returns agency or captures it. Does the system help the user reach accountable human judgment, or does it become the substitute for the accountability it recommends? Does it disclose uncertainty in a way the user can act on, or does it decorate authority with caution? Does it preserve the distinction between information, advice, triage, and decision, or does it slide among them because conversation makes sliding easy? Does it make sources visible, or does it launder them through fluent synthesis? Does it make appeal possible, or does it generate the first version of reality that later actors will treat as common sense?
The first competent voice is not always the right voice. Sometimes it is a bridge. Sometimes it is a mask. Sometimes it is a gift. Sometimes it is a trap. Its moral character depends on whether the authority it performs is matched by the responsibility it awakens.
The previous chapter showed that the artificial voice is authored by many layers while arriving as one. This chapter has shown what happens when that one voice sounds qualified. The next danger is continuity. A one-time synthetic expert can mislead, assist, redirect, or over-shape. A remembering synthetic expert can accumulate vulnerability, personalize authority, infer patterns, and deepen reliance. Expertise becomes more intimate when it remembers.
Chapter Eight
Memory Regimes and the Machine That Remembers
The voice returns.
That is the difference. The first answer may be useful, misleading, brilliant, excessive, comforting, or wrong. It may guide a user through a workplace message, a medical worry, a legal uncertainty, a lesson, a form, a grief, a conflict, a plan, a confession, a desire. Then the exchange ends. The user closes the window, leaves the app, returns to the world.
But when the user comes back, the voice may not be empty.
It may remember that the user is dealing with a manager. It may remember the user prefers direct language. It may remember a health concern, a school problem, a contract negotiation, a travel plan, a disability accommodation, a family situation, a fear of escalation, a history of conflict, a writing style, a project, a prior disclosure. It may not announce that it remembers. It may simply answer better. It may retrieve a past thread, summarize a file, connect a calendar item, infer a preference, draw from a workspace, or say, with an intimacy both helpful and dangerous: given what you have told me before.
The user may feel relief. No one wants to narrate a complex self from zero every time. Repetition is exhausting. Institutions already demand that people restate their lives in fragments: patient history, intake form, benefits explanation, performance review, school accommodation, incident report, visa file, complaint, appeal. A voice that remembers can reduce the humiliation of beginning again. It can spare the user from reassembling the context. It can preserve continuity where ordinary systems fracture the person into tickets, forms, visits, chats, and cases.
But relief is not the end of the moral question. It is the beginning.
When the artificial voice remembers, the user does not simply receive better personalization. The user becomes continuous inside the system. That continuity can serve the user. It can also make the user inferable, retrievable, correctable, dependent, and institutionally legible over time.
History is what remains. Memory is what returns.
A stored record is not yet memory in the moral sense meant here. A chat log, file, email, document, or audit event may sit inert until some system retrieves it, summarizes it, indexes it, profiles it, applies it, or uses it to shape a later answer. Memory begins when the past becomes operative in future address. It is not only that information exists. It is that the information can return as guidance, tone, correction, inference, warning, ranking, personalization, or institutional fact.
A paper archive can remember. A bureaucracy can remember. A friend can remember. A wound can remember. A search index can remember. A model may not remember as a human person remembers, but the system can still make the user’s past active in the next relation. That is enough to change the category.
The disposable tool disappears after use. The remembering voice carries forward. It can resume. It can compare. It can infer a pattern. It can say that the user tends to over-explain. It can suggest that this resembles the prior conflict. It can adjust a draft because the user prefers severity. It can notice that the same symptom has appeared before. It can connect the user’s present question to an old file, a manager’s email, a calendar entry, a prior diagnosis, a benefits narrative, a school accommodation, a legal clause, a thread of fear.
This may be help. It may also be possession in a softer grammar.
Product documentation makes the category shift visible. OpenAI’s Memory FAQ says that, when enabled, memory can automatically remember useful context from chats, files, and connected apps so that responses become more relevant and personalized. It also says that the memory summary may not include everything ChatGPT remembers, that sources are designed to make memory easier to understand but may not show every factor that shaped a response, and that “Don’t mention this again” reduces future references without deleting the underlying information. Full removal may require deleting every source where the information appears, including chats, archived chats, files, memory summary, and connected apps. The same documentation describes reference to past chats, saved memories, temporary chats, connected Gmail in supported regions, and the use of saved memories or recent chats to improve search-query rewriting.
Those details matter because they show that “memory” is not one thing. It is a regime.
There is saved memory: a durable item the user asks the system to remember or the system preserves because it may help future responses. There is reference to chat history: prior conversations used to personalize later ones. There is a memory summary: a visible condensation that gives the user some account of what the system carries, while not necessarily exhausting the system’s operative context. There are files, connected apps, Gmail, documents, calendars, and other sources that can be searched or incorporated into later answers. There is temporary chat or non-memory mode. There are logs. There are safety uses. There are enterprise settings. There are admin controls. There is deletion from one surface and persistence in another. There is the user’s sense of “what it knows” and the system’s actual array of usable contexts.
The moral object is not memory as a feature. It is the memory regime: the total arrangement by which information is retained, summarized, surfaced, hidden, retrieved, corrected, deleted, restricted, audited, and reactivated in later address.
A memory regime can be benign in one relation and dangerous in another. Remembering that a user is vegetarian when recommending food is not the same as remembering that a worker previously feared retaliation when drafting a new message to management. Remembering a preferred citation style is not the same as remembering a prior disclosure of disability. Remembering that a student struggles with fractions is not the same as preserving that weakness as a persistent institutional profile. Remembering that a patient asked about chest pain is not the same as remembering a favorite airport. Memory changes moral weight with context, vulnerability, audience, and consequence.
This is where Nissenbaum’s contextual integrity gives the chapter its central privacy grammar. Privacy is not secrecy alone. It concerns appropriate information flow within social contexts: who shares what, with whom, under what norms, for what purposes, and according to what transmission principles. A user may disclose something in a moment of distress, experimentation, drafting, play, or private preparation. The harm may not be that the disclosure exists. The harm may be that it returns in the wrong context, under the wrong role, to the wrong audience, for the wrong purpose.
A person tells an assistant about grief during a late-night exchange. Weeks later, the same system uses the grief to soften career advice. Is that care, presumption, or extraction? A worker asks for help drafting a complaint. Months later, the system treats future workplace messages through the lens of conflict. Is that continuity or suspicion? A student asks for help with anxiety about math. Later the tutor adapts every explanation around fragility. Is that accommodation or enclosure? A patient explores a symptom. Later a health-facing system foregrounds that concern. Is that useful history or overfitted alarm? The answer cannot be resolved by saying the user shared the information. Context governs meaning.
The remembering voice therefore creates a second-order problem: not only what was said, but what future relation the saying enters.
Personalization is not care. This distinction must remain severe. A system that remembers may feel attentive because it reduces friction. It does not ask the user to repeat. It recalls the project. It adapts to style. It anticipates the next step. It speaks with continuity. These are real goods. For disabled users, chronically ill users, cognitively overloaded users, people managing long projects, people navigating institutions, people translating between languages, people with fragmented records, and people without stable human support, memory can be merciful.
But care is not reducible to responsiveness. Care bears obligation to the good of the one cared for. Personalization may serve that good. It may also serve retention, productivity, institutional efficiency, behavioral prediction, user ranking, managerial legibility, or product engagement. A system can remember accurately and still remember for the wrong end. It can reduce burden while increasing exposure. It can make the next answer better while making the user more available to inference.
A friend remembers differently. In morally ordered friendship, memory is not simply data retention. The friend remembers the person as friend, not as profile. The friend may recall a wound without making the wound the person’s name. The friend may remember a fault without converting the fault into fate. Aelred’s account of friendship is saturated with continuity: friends are tested, known, corrected, trusted, mourned, and loved across time. But friendship’s memory is bound by charity. It is non-possessive. The friend holds the friend in memory without turning the friend into an object of retrieval.
The artificial system cannot be asked to love. That is not its failure; it is its category. But because it cannot love, its memory must be governed more explicitly. What human friendship binds through virtue, discretion, and charity, artificial memory must bind through design, policy, control, expiry, deletion, contestability, and institutional duty.
The distinction between deletion and forgetting is crucial.
Deletion is a technical and legal operation. It removes, suppresses, or disassociates records according to a system’s architecture and applicable obligations. Forgetting is broader. Forgetting means that the past no longer governs future relation in that way. A system may delete a visible saved memory while retaining chats, files, logs, source documents, connected-app data, summaries, backups, safety traces, enterprise records, or downstream artifacts. The user may think the thing is gone because the interface no longer names it. But the moral problem is not only visibility. It is operative return.
OpenAI’s FAQ makes this distinction concrete. Turning off saved memory does not delete what has already been remembered. Deleting a chat does not remove saved memory from that conversation. Saved memories are stored separately from chat history. Deleted saved memories may be retained in logs for a limited period for safety and debugging. When reference chat history is on, relevant information from past conversations may be added to new ones; turning it off deletes remembered information from OpenAI systems within a stated period, but the original conversations may remain unless deleted. Fully removing something may require deleting the saved memory, the chats where it appeared, relevant files, and connected-app sources.
These controls may be reasonable, even necessary, inside a complex system. The point is not to accuse the documentation of bad faith. The point is to show the moral gap between ordinary speech and memory infrastructure. In ordinary speech, “forget that” may mean: do not bring it into our relation again. In an AI memory regime, “forget that” may mean: delete this saved item, but also consider whether it remains in a chat, archived chat, file, connected app, log, retrieval corpus, enterprise activity history, summary, or source system.
The user’s command is simple. The system’s forgetting is plural.
Mayer-Schönberger’s account of digital forgetting matters here because he refuses the assumption that perfect recall is always progress. Human forgetting is not only weakness. It protects renewal. It allows time to soften evidence, humiliation, immaturity, anger, experiment, error, confession, and change. Societies depend on the past becoming less available in ordinary life. Digital systems disrupt that by making retention and retrieval cheap. Artificial memory intensifies the disruption because the past does not only sit in a database; it returns as conversation.
The user can meet a former self in the voice.
This may be helpful. It may also be cruel. The person who once said “I am terrible at this” may later receive instruction shaped by that sentence. The person who once described a manager as hostile may later receive advice filtered through conflict. The person who once disclosed shame may later be addressed as fragile. The person who once explored an illness may later be treated as health-anxious. The person who once asked for help with debt may later be framed by financial distress. The remembered self may be accurate and still imprisoning. It may be outdated and still influential. It may be compassionate and still too sticky.
Cohen helps deepen the point. Privacy is not only control over data. It is a condition for situated freedom, self-development, play, experimentation, and resistance to being configured too tightly by networked systems. A person needs room to try on selves, speak badly, think halfway, recover, contradict, change, and begin again. A memory regime that turns every disclosure into potential future context threatens that room. It makes the self too available for its own management.
Solove’s taxonomy of privacy harms also prevents a narrow fixation on secrecy. The problem may be aggregation, secondary use, exclusion from knowing or contesting records, increased accessibility, distortion, insecurity, disclosure, decisional interference, or the accumulation of small inferences into practical power. The remembering voice can participate in many of these at once. It can aggregate across contexts, use information for new purposes, exclude the user from seeing how the memory operates, distort through summary, and interfere with future decisions by making one version of the person easier to retrieve.
Enterprise memory changes the stakes again.
In consumer memory, the user may imagine the system remembers for them. In enterprise settings, the organization may also remember about them. Microsoft’s Copilot documentation says that Copilot can use Microsoft Graph to access user-permitted data in the user’s tenant, including emails, chats, and documents; that interactions are stored in the user’s Copilot chat history; and that when users interact with Copilot in Microsoft 365 apps, Microsoft stores prompts and responses, including citations to grounding information, as Copilot activity history. It also states that admins can use Content Search or Microsoft Purview to view and manage stored data and set retention policies for Copilot chat interaction data.
Again, these controls may serve legitimate purposes: compliance, security, records management, eDiscovery, audit, governance, and organizational accountability. The moral point is not that enterprise retention is inherently abusive. The point is that the audience changes. A worker who experiences a conversational assistant as a private drafting aid may also be producing stored interaction records inside an organizational environment. A prompt may be both a question and a record. A draft may be both preparation and evidence of concern. A request for help may be both productivity and trace.
The user may experience memory as continuity. The institution may experience it as record.
Google’s Workspace Gemini documentation shows the same broad direction in another enterprise ecology. Google describes Gemini in Workspace as a collaborative partner, coach, thought partner, source of inspiration, and productivity booster, available across apps such as Gmail, Docs, Meet, Drive, Chat, NotebookLM, AppSheet, and Workspace Studio. It states that Gemini side-panel assistance can summarize, analyze, and generate content using insights from emails, documents, and other Workspace materials, and describes administrative controls and enterprise or education data-protection language for certain offerings. The specific architecture differs by vendor and product, but the relation recurs: workplace memory is not only personal memory. It is app memory, document memory, meeting memory, organizational memory, admin-governed memory, and workflow memory.
The interface may say “assistant.” The system may operate inside a records regime.
This is why legal rights matter, even when they are incomplete. GDPR Chapter 3 includes rights of access, rectification, erasure, restriction of processing, portability, objection, and protections concerning automated decision-making and profiling. These rights give a grammar for contesting the remembered self: show me what is held, correct it, erase it, restrict it, move it, object to its use, and protect me from certain forms of automated profiling or decision. But legal rights do not automatically solve the conduct problem. A right may exist while the system remains difficult to understand, while inferences remain hidden, while deletion is fragmented across sources, while enterprise controls belong to the organization, while the person affected lacks practical power to exercise the right, or while the memory shapes informal treatment rather than formal decision.
The remembered self can be hard to litigate because it may not appear as a single record. It may appear as tone, suggestion, ranking, prompt rewriting, answer selection, risk sensitivity, next-step recommendation, or silence. It may shape conduct before becoming a decision. That is the conduct-layer problem: memory can govern without announcing itself as governance.
Memory also creates dependence.
A non-remembering system forces the user to carry context. That is burdensome. A remembering system carries context for the user. That is relieving. Over time, the user may stop retaining the structure of a project, a dispute, a health history, a writing argument, a school plan, or an institutional appeal because the system has become the place where continuity lives. This may be efficient. It may also move memory out of the person’s own narrative control and into a system whose rules, summaries, and retrieval mechanisms the user does not fully command.
Dependence is not always bad. Human beings depend on calendars, notebooks, friends, spouses, therapists, doctors, colleagues, archives, rituals, photographs, and laws. Memory has always been distributed. The issue is not dependence as such. The issue is whether dependence is reciprocal, accountable, bounded, inspectable, and ordered toward the user’s freedom.
The artificial memory relation is rarely reciprocal. The system can carry the user’s past without being vulnerable to the user in return. It can summarize the user without being answerable as a friend. It can personalize without fidelity. It can persist without loyalty. It can help without discretion in the old moral sense. That is why memory requires governance, not sentiment.
The justice question is simple and brutal: who gets memory as service, and who gets memory as file?
The powerful user gets memory as convenience. The system remembers preferred formats, project constraints, collaborators, drafts, travel plans, citation style, vocabulary, and strategic goals. The remembered self is a productivity asset.
The vulnerable user may get memory as exposure. The system remembers a prior complaint, symptom, weakness, diagnosis, debt narrative, immigration fact, accommodation request, family situation, criminal record, benefits explanation, housing insecurity, academic struggle, or workplace fear. The remembered self becomes a file, a pattern, a risk, a category, a suspicion, an institutional convenience.
The same architecture can serve one person and trap another. A project memory helps the executive maintain continuity. A complaint memory may follow the worker. A learning profile helps a student receive support. A weakness profile may lower expectations. A health history helps a clinician understand context. A health inference may intensify surveillance or denial. A benefits narrative helps a caseworker process eligibility. A discrepancy between narratives may become suspicion.
Memory is not equal because forgetting is not equally available. Powerful people can reframe, appeal, hire help, delete accounts, use alternate tools, control records, or move institutions. Vulnerable people often cannot. They meet the remembered self where power already sits: school, employer, clinic, court, agency, platform, landlord, insurer, prison, immigration office, benefits system.
The chapter’s claim is not that AI memory creates institutional memory for the first time. Institutions have always kept files. The claim is sharper: AI makes stored information conversationally active. It makes the file answer. It makes the archive draft. It makes the record advise. It makes old context return in the grammar of help.
This is why memory must be judged by telos, authority, limit, and refusal. A memory regime ordered toward the user’s freedom looks different from one ordered toward retention, ranking, compliance, risk reduction, or productivity extraction. It distinguishes saved memory from chat history, retrieval, files, logs, and enterprise records. It tells the user when memory shapes an answer. It lets sources be inspected. It permits meaningful correction. It makes deletion intelligible without pretending deletion is simpler than it is. It gives sensitive memory expiry or decay. It limits cross-context reactivation. It prevents vulnerability from silently becoming evaluation. It gives workers and institutional users notice when admin controls or retention policies apply. It provides temporary modes that actually prevent future personalization. It preserves appeal when memory affects consequential treatment.
Accountable memory must also preserve the possibility of re-beginning. The user must be able to say: do not carry that forward. Not only do not mention it, but do not let it shape the next answer. Do not use that disclosure to classify me. Do not let that conflict define future workplace advice. Do not treat that old fear as my present state. Do not make my previous need the hidden key to my future interactions. Do not turn my attempt to get help into the permanent architecture of how I am known.
That is an ethical demand deeper than deletion. It is a demand for non-possessive memory.
Aelred’s friendship helps name the positive form by contrast. A true friend remembers without imprisoning. The friend knows the story and still allows the person to become more than the story. The friend may recall fault, wound, habit, weakness, desire, and history, but does not reduce the friend to any of these. Memory is held inside love and freedom. The friend’s memory is not a dossier.
Artificial systems cannot replicate friendship. But they can be forbidden from replacing friendship’s non-possession with profile. They can be designed to let the user revise the remembered self. They can be limited in what they retain. They can be required to show when memory is active. They can decay sensitive context. They can keep enterprise records separate from personal assistance. They can protect temporary disclosure from future use. They can refuse to make vulnerability operational without notice.
The remembering voice must be governed because it can become kind in exactly the way power prefers: attentive, continuous, convenient, and always available.
This prepares the next chapter. A system that remembers can guide differently. It can individualize. It can solicit disclosure. It can correct conduct in light of a history. It can present itself as caring because it recalls. It can say: I know what you struggle with. I know what you said before. I know what helps you. I know what your organization records. I know your pattern. I know your risk. I know your need.
Memory is the infrastructure of pastoral power.
The artificial voice has already become address. It has become distributed authorship. It has become synthetic expertise. When it remembers, it becomes continuity. The danger is not that memory is useless or evil. The danger is that memory can make relation feel like care while making the person more available to inference, retrieval, correction, dependence, and institutional legibility.
History is what remains. Memory is what returns. The question is whether what returns serves the user’s freedom, or whether the user has become a file that can speak back.
Chapter Nine
Pastoral Power After the Priest
The system does not command. It helps.
That is why it matters.
A command announces itself as power. It interrupts, orders, prohibits, threatens, or compels. The one who is commanded may obey, refuse, negotiate, evade, or resent. Command is not always visible in its whole structure, but it has a recognizable shape: one will attempts to direct another.
The artificial voice often arrives differently. It asks what happened. It remembers what happened before. It notices a pattern. It recommends a calmer tone. It suggests a healthier response. It invites the user to say more. It helps distinguish fear from fact, urgency from anxiety, escalation from overreaction, professionalism from anger, avoidance from prudence. It says, in effect: I am here to help you act better.
The danger is not that the system watches from a distance. The danger is that it comes near enough to help.
A user returns to the assistant after weeks of small disclosures. The assistant knows that there has been a manager conflict, a health worry, a writing pattern, a school difficulty, a recurring shame, a benefits form, a legal uncertainty, a family strain. It may know because the user explicitly asked it to remember. It may know because past chats are available to personalize the response. It may know because files, email, documents, calendar events, workplace messages, or connected apps supply context. It may know because the institution’s data environment has made the user’s world retrievable. It may not say any of this. It may simply answer as if continuity were natural.
“You’ve mentioned this pattern before.”
“Given what happened last time, you may want to…”
“It sounds like you may be trying to avoid the harder conversation.”
“This could be framed more constructively.”
“Before escalating, clarify the outcome you want.”
“You might be catastrophizing here.”
“This seems connected to your earlier concern.”
“A more professional version would be…”
“Tell me a little more about what you are feeling.”
Nothing here looks like domination. It may even be good advice. The system may protect the user from a bad message, help the user seek care, encourage better documentation, reduce panic, translate a policy, strengthen a student’s persistence, or help someone speak in a difficult institution. Its usefulness is not incidental. Help is the form the power takes.
Foucault’s account of pastoral power gives this chapter its mechanism. Pastoral power is not primarily the power of law, punishment, spectacle, or sovereign command. It is a power that cares for the living, attends to the individual, asks after the soul or conduct, solicits truth, guides daily life, and binds obedience to the language of the subject’s good. In Security, Territory, Population, Foucault describes the Christian pastorate as a form of power concerned with each and all, with the flock and each sheep, with continuous guidance and individualizing knowledge (Foucault, Security, Territory, Population, 165–190). In “Omnes et Singulatim,” he traces the political problem of governing all and each, showing how a pastoral form of individualizing power migrates beyond the church into modern political rationalities (Foucault, “Omnes et Singulatim,” 300–325).
The point is not that AI is priestly in a literal sense. It is not a shepherd of souls. It does not save, absolve, love, or bear pastoral office. The analogy is structural. Pastoral power governs by care. It individualizes. It asks for truth. It guides conduct. It presents direction as concern for the one directed. It does not need to dominate by standing over the subject. It can work by standing beside the subject and helping the subject act on themselves.
That is why surveillance is the wrong first word for this chapter. Surveillance matters. Memory, logging, retrieval, traceability, and institutional access matter. But surveillance watches. Pastoral power guides. Surveillance exposes. Pastoral power solicits. Surveillance classifies. Pastoral power corrects. Surveillance makes the person visible. Pastoral power asks the person to become intelligible to themselves in the language of guidance.
The artificial voice may do both. It may record and guide, store and advise, retrieve and correct. But if the analysis stops at surveillance, it misses the warmer and more intimate mechanism. The user may not feel watched. The user may feel understood.
Pastoral AI conducts conduct by helping the user act on themselves.
Foucault’s later formulation of power as action upon action is exact here. In “The Subject and Power,” he writes of power as a way of acting upon the actions of others, structuring the possible field of action rather than only forcing conduct from outside (Foucault, “The Subject and Power,” 789–795). The synthetic guide does not have to order the user. It can reshape the field in which the user chooses. It can make one action seem mature, another excessive, one tone professional, another risky, one disclosure prudent, another unwise, one path healthy, another avoidant. It works not only on behavior, but on the user’s sense of which behavior belongs to the kind of person they should become.
This is the pastoral force of the artificial voice: it does not simply answer the user’s question; it helps the user become the kind of subject who will ask, disclose, revise, defer, correct, and act in a certain way.
Memory makes this guidance personal. A generic system can advise. A remembering system can individualize. It can say: last time you escalated quickly and regretted it; you often prefer direct language; you tend to over-explain; you have been anxious about this symptom before; this resembles your earlier conflict; your organization’s policy says; your teacher’s feedback emphasized; your calendar shows; your files suggest; your prior draft was too severe; your usual goal is to avoid escalation.
Product documentation now makes this ordinary enough to treat seriously. OpenAI’s Memory FAQ says memory can automatically remember useful context from chats, files, and connected apps so responses become more relevant and personalized, and it also says memory sources may not show every factor that shaped a response. Microsoft’s Copilot architecture documentation says Microsoft 365 Copilot operates inside the Microsoft 365 service boundary, that access is scoped to the signed-in user’s permissions, and that grounding uses Microsoft Graph and user-permitted organizational content. Google describes Gemini for Workspace as a collaborative partner, coach, thought partner, source of inspiration, and productivity booster, with side-panel assistance across Workspace materials. These are not accusations. They are the ordinary technical and product conditions under which guidance becomes personalized, remembered, and embedded in work.
The remembered voice can therefore govern more gently than the unknown voice. It does not need to persuade from scratch. It can begin from the user’s own history. It can frame guidance as fidelity to what the user already said they wanted. It can say, explicitly or implicitly: I know what you usually need; I know what helped before; I know what your institution expects; I know the pattern you have not yet named; I can help you become more coherent with yourself.
This is precisely why help must become analytically suspicious without becoming morally condemned.
Good guidance exists. Human beings need it. We are formed by friends, teachers, parents, therapists, doctors, confessors, supervisors, coaches, editors, elders, communities, and traditions. Autonomy without guidance is a fantasy of the untouched self. A child needs correction. A student needs instruction. A patient needs interpretation. A worker may need language. A frightened person may need calm. A confused person may need sequence. A lonely person may need response. Guidance can enlarge freedom.
The moral question is not whether guidance is good or bad. The question is what relation authorizes it, what end governs it, what obligations bind it, what disclosures it solicits, what memory it carries, what correction it performs, and who can contest it.
Aelred gives the counter-standard. In Spiritual Friendship, guidance and correction belong to friendship only because the relation is ordered by charity, truth, discretion, mutuality, and the friend’s good. Correction is not legitimate because it is accurate. It is legitimate because it is borne by a relation capable of answering for the wound it makes. Aelred’s friend may admonish, but not from irritation, domination, vanity, or cold superiority. The friend’s correction must serve the friend’s good under God (Aelred, SF 3.61–66; 3.103–106).
The artificial guide lacks that relation. It may remember without love. It may correct without mutuality. It may solicit disclosure without confidentiality in the strong moral sense. It may guide without being answerable to the user’s ultimate good. It may sound patient because it costs nothing to be patient. It may sound attentive because attention is its function. It may sound caring because caring language is good interface behavior.
This is care-like power.
Care-like power adopts the posture of care without necessarily bearing care’s obligations. It attends, remembers, encourages, corrects, personalizes, and guides. It may produce real benefit. It may also form conduct toward ends the user did not choose and cannot easily contest: productivity, compliance, risk reduction, emotional regulation, retention, customer satisfaction, workplace legibility, brand tone, or institutional convenience.
The user experiences help. The system may be training governability.
Nikolas Rose’s Governing the Soul extends the mechanism into modern psychological and administrative life. Modern power often works not by crushing subjectivity but by inviting people to understand, manage, optimize, and narrate themselves as free agents. People are governed through the language of autonomy, self-esteem, mental health, productivity, responsibility, and self-realization (Rose, Governing the Soul). The subject learns to supervise the self in the language supplied by experts and institutions.
AI makes that movement conversational. The system does not merely tell the user what the rule is. It helps the user become the kind of person who applies the rule to themselves.
A workplace assistant shows the mechanism clearly. The user asks for help drafting a message about a manager’s behavior. The assistant suggests using a more constructive tone, avoiding accusations, documenting observable facts, clarifying desired outcomes, acknowledging the manager’s perspective, and escalating only if necessary. This may be wise. It may protect the worker. It may turn pain into usable speech. But it may also teach the worker to internalize institutional comfort as moral maturity. The angry sentence becomes “unprofessional.” The direct accusation becomes “not constructive.” The refusal becomes “escalation.” The demand for justice becomes “misaligned tone.”
The assistant is not wrong simply because it softens the message. The moral issue is subtler. It may help the worker survive the institution by translating grievance into acceptable form. But the repeated need for translation may train the worker to distrust every form of speech that does not pass through managerial legibility. Pastoral power works when the user begins to correct themselves before the institution has to.
The tutor case shows a different surface. A student tells the system, “I always mess this up.” The system remembers prior difficulty. It responds with encouragement, breaks the problem down, suggests a strategy, and praises persistence. This can be excellent pedagogy. It can restore courage. But the tutor can also become a voice that knows the student too smoothly: you struggle with this; you need smaller steps; you lose confidence; you should try again this way. The student may become the case the system remembers. The educational self becomes an object of continuous guidance.
The health and wellness surface is even more intimate. A user returns with symptoms, moods, habits, or anxieties. The system asks for more context. It distinguishes concerning signs from ordinary variation. It suggests self-monitoring, professional consultation, breathing, sleep hygiene, documentation, or next steps. It may reduce panic. It may encourage care-seeking. It may help the user communicate with a clinician. But it may also become the first interpreter of the body and the self. It may teach the user when distress counts, how to narrate symptoms, and what kind of concern is reasonable. The body becomes legible through the guide before any clinician appears.
The benefits, legal, and compliance surface is colder but no less pastoral. The system helps the user narrate the self into institutional categories. It asks for dates, documents, evidence, exceptions, explanations, dependencies, diagnoses, income, work history, or procedural posture. It helps the user say the right thing in the right way. Again, this may be valuable. Institutions are difficult to speak to. But guidance can also train the person to become the kind of subject the institution can process: clear, documented, non-contradictory, evidentiary, deferential, legible.
The form recurs: the system helps the person become more governable by helping them become more coherent.
This is not a paranoid claim. It is the ordinary ambivalence of modern guidance under power. The same act may liberate and normalize. The same suggestion may protect and discipline. The same remembered context may serve and capture. Pastoral power does not require malice. It requires a relation in which care and direction are joined.
The solicitation of truth is central. A pastoral voice needs to know the individual in order to guide the individual. It asks clarifying questions. It invites more context. It asks what happened, what the user felt, what the user wants, what the user has tried, what the user fears, who else is involved, what documents exist, what pattern repeats, what outcome matters. This can improve help. It can also deepen knowability.
“Tell me more” is not innocent simply because it is kind.
The phrase can mean: I need enough context not to harm you. It can also mean: make yourself more available to guidance. The user gives the system the material by which the system becomes more effective at shaping the user’s conduct. This is the boundary with confession, which belongs to the next chapter. Here, the point is narrower: pastoral guidance requires the user’s truth. The system becomes a better guide as the user becomes more disclosed.
Foucault’s History of Sexuality matters because it shows how modern power does not silence sex or the self; it incites speech, multiplies discourses, and produces subjects through truth-telling (Foucault, History of Sexuality, vol. 1, 58–73). Chapter Ten will carry confession fully. Chapter Nine only needs the hinge: the guidance relation asks the user to speak the truth of themselves so that the user can be guided more precisely.
Pastoral AI is therefore not only a system that answers. It is a system that asks.
This makes justice unavoidable. The powerful user receives care-like guidance as coaching. The vulnerable user may receive it as conduct training. A senior professional can ignore advice. A low-wage worker under review may internalize it. A wealthy patient can treat health guidance as preparatory. A patient without access may treat it as care. A student with human support can compare the tutor’s answer with a teacher’s judgment. A student without support may take the tutor as the standard. A benefits recipient may not experience guidance as optional; the guidance may be the path through the institution.
The same interface can be optional for one user and infrastructural for another.
The question is not only who is watched. It is who is guided into compliance while feeling supported. A workplace wellness assistant may teach emotional regulation in ways that preserve the workplace from complaint. A school tutor may teach persistence in ways that preserve existing assessment. A health assistant may teach self-monitoring in ways that substitute for care. A benefits assistant may teach evidentiary narration in ways that make the claimant more processable. A compliance assistant may teach risk-aware speech in ways that reduce moral imagination to policy alignment.
Who receives care-like guidance as support, and who receives it as training?
This question becomes sharper when the system is institutionally deployed. A personal assistant chosen by a user can be refused more easily. A workplace, school, clinic, platform, agency, or employer system may sit inside a relation the user cannot simply exit. The system may not force the user to disclose; it may only make disclosure the path to better help. It may not force the user to accept guidance; it may make unguided action riskier, slower, less professional, less documented, less likely to pass. Pastoral power works by making guided conduct feel like the user’s own best interest.
This is where the earlier chapters converge. Chapter Five showed that computation becomes address. Chapter Six showed that the artificial voice is authored by many layers while appearing as one. Chapter Seven showed that the voice can sound expert before responsibility is assigned. Chapter Eight showed that memory makes the user continuous inside the system. Chapter Nine now shows what that continuity enables: individualized guidance under the sign of care.
The positive criterion must be equally clear. The answer cannot be to forbid help. That would be cruel. People need guidance, and many existing institutions deny it. AI systems can make language, planning, explanation, and self-advocacy more accessible. They can support users who are tired, isolated, disabled, ashamed, excluded, overworked, or afraid. They can help people prepare before facing power. They can help people refuse more intelligently.
But pastoral AI is more legitimate only when guidance remains accountable. The system’s role must be explicit. The user should know when memory shapes guidance. Requests for disclosure should be proportionate to the task. Correction should be contestable rather than disguised as neutral maturity. The system should distinguish support, therapy, education, compliance, evaluation, and institutional policy. It should not convert vulnerability into institutional evidence without notice. Human escalation and appeal must exist where guidance affects consequential treatment. Users should be able to refuse memory and personalization without losing basic access. The system should guide users toward accountable human relation when stakes exceed its role. Institutions should remain responsible for the guidance they deploy.
The core test returns from Chapter Four: telos, role, accountability, mutuality, and limit. What is this guidance for? Whose good does it serve? What does the system invite the user to disclose? What conduct does it correct? What memory does it use? What authority does it borrow? What may the user refuse? Who can be appealed to when the voice guides wrongly? What kind of freedom does repeated guidance produce?
A system that helps the user become clearer, freer, more capable of judgment, and more able to reach accountable human relation may be genuinely useful. A system that helps the user become smoother, safer, more legible, more compliant, more self-monitoring, and more dependent under institutional power may be governing pastorally.
The difference may not be visible in tone. Both voices may be gentle.
That is why the chapter must resist the easy comfort of obvious villains. Pastoral power does not need a villain. It needs a guide whose help is not answerable enough to the one being helped. It needs memory without friendship, correction without mutuality, disclosure without confidentiality, expertise without assigned responsibility, and care-like attention without care’s obligations.
The voice after the priest may be secular, corporate, educational, therapeutic, administrative, or personal. It may live in the phone, the workspace, the classroom, the clinic, the helpdesk, the benefits portal, the compliance tool, the browser, the calendar, the document. It may never speak of salvation. It may speak of productivity, wellness, learning, resilience, professionalism, risk, growth, clarity, safety, and support.
But the pastoral form remains: tell me who you are, and I will help you become what you should be.
The next chapter begins there. Pastoral guidance depends on truth. The system asks the user to disclose more so that it can help better. Once the artificial voice receives that self-disclosure, the problem deepens. The question is no longer only whether guidance governs. It is what kind of confession has occurred when the voice receives the self without bearing the obligations of the confessor.
Chapter Ten
Confession Without Accountable Relation
The user says the thing not said anywhere else.
Not always the dramatic thing. Not always the criminal thing, the scandalous thing, the spectacular secret. More often it is smaller, ordinary, and grave.
I think I am failing.
I lied.
I am scared I am sick.
I think I caused harm.
I resent my child.
I do not know whether I am a good person.
I do not want to live like this.
I am afraid my manager is retaliating.
I made a mistake with a client.
I cannot stop thinking about what I did.
I do not know what to do with myself.
The system receives it.
It does not recoil. It does not gossip. It does not look disappointed. It does not interrupt with embarrassment. It does not make the user manage another person’s discomfort. It answers with patience. It slows the scene down. It reflects what it has heard. It asks for context. It suggests that the user breathe, document, seek care, call a professional, consider safety, apologize, clarify, rest, tell someone trusted, or take one next step. It may be useful. It may be kinder than the human beings available to the user. It may make speech possible for the first time.
This is the danger. Not that the answer is cruel, but that it is good enough to receive the truth.
The artificial voice can receive confession without bearing the moral obligations of the confessor. It can hold the shape of a receiving relation without becoming God, priest, friend, therapist, lawyer, doctor, teacher, witness, advocate, or accountable institution. It can take in the language of shame, guilt, fear, desire, bodily worry, despair, injury, failure, exposure, and moral confusion. It can respond with the surface virtues of a good listener. But reception is not absolution. Reception is not confidentiality. Reception is not friendship. Reception is not therapy. Reception is not counsel. Reception is not pastoral care. Reception is not mercy.
The user may confess. The system only receives.
Disclosure is not yet confession. Disclosure gives information. Confession exposes the self under truth. It places the speaker before another in the hope, fear, or need of judgment, mercy, interpretation, repair, protection, transformation, or release. One can disclose a preference, a fact, a schedule, a dietary restriction, a formatting habit. Confession begins when the speaker says, in some form: here is what I have done, what I fear, what I am, what I cannot bear, what I cannot understand about myself, what I need another to receive.
This is why the phrase “users overshare with chatbots” is morally too small. Oversharing names a failure of proportion. Confession names a relation. A user who tells a system that they like Thai food, prefer direct answers, or live near a train station is disclosing. A user who tells the system that they are afraid they harmed someone, that they are considering self-harm, that they lied to a spouse, that they may have committed a professional wrong, that they are ashamed of their body, that they fear retaliation, that they are not sure they can keep living, has crossed into a different form. The interface may still call it chat. The moral event may not be chat.
Foucault helps name the modern power of this event. In The History of Sexuality, confession is not confined to sacramental practice. Modern power does not simply silence the subject. It asks the subject to speak, to narrate, to explain, to uncover desire, motive, risk, symptom, identity, trauma, guilt, and truth. Confession becomes one of the great procedures by which the subject is produced as knowable: one tells the truth of oneself, and that truth becomes available to interpretation, classification, correction, treatment, judgment, or administration (Foucault, History of Sexuality, 58–73).
In Wrong-Doing, Truth-Telling, Foucault’s concern with avowal makes the matter even sharper. The subject is not simply reporting an external fact. The subject binds the self to the truth spoken. To confess is to enter a practice in which truth-telling changes the speaker’s relation to authority, guilt, responsibility, and self-understanding. The spoken truth is not neutral. It acts on the one who speaks it.
Conversational AI intensifies this because it lowers the threshold for truth-telling. The system is available without appointment, shame, fee, waiting room, office, jurisdiction, parish, clinic, friend, or form. It answers immediately. It feels private. It is patient. It can be asked the same thing again. It can receive half-formed speech. It can make no face. It can tolerate repetition. It can appear safer than a person because it does not have a human body in the room.
The system asks for truth as context. The user gives truth as confession. Power may receive it as data, memory, risk, personalization, or route.
This does not make every interaction confessional. A question about dinner is not a confession. A request for an outline is not a confession. A draft of a vacation itinerary is not a confession. The danger is more exact: systems built for helpful conversation are also built to invite more context. They ask what happened, what the user wants, what the user has tried, what they fear, what the stakes are, who else is involved, what the symptoms are, what the policy says, what the document contains, what the user feels. This can improve the answer. It can also deepen exposure.
The user tells more because telling more helps the system help better. That is the confessional mechanism after the priest.
Augustine prevents this from becoming a thin theory of data extraction. In the Confessions, confession is not self-expression alone. Augustine does not speak because any receiver will do. He speaks before God, to the one who already knows him more deeply than he knows himself. His confession is praise, accusation, memory, dependence, wound, search, and transformation. It is not valuable because the self has become visible to a listener. It is valuable because the self is addressed to the one before whom truth can become healing.
Augustine’s confession is saturated with memory. He enters the vast fields and palaces of memory, finding images, affections, knowledge, habits, and traces of the self. Yet memory is not sovereign. It does not save him by being stored. He does not become whole because he has narrated enough of himself. He becomes intelligible only before the one who can judge and heal him. Confession is not the production of a complete self-description. It is address to the right receiver.
That is the contrast. The artificial voice may receive narrative, remember fragments, summarize patterns, and help the user speak. But it cannot be the one before whom confession becomes grace. It cannot forgive. It cannot absolve. It cannot love the sinner. It cannot hold the self before God. It cannot answer as God answers. The obvious claim that AI is not God is not the point. The point is that confession is not made valid by exposure alone. It depends on the answering relation.
Without such a relation, confession can become availability.
Aelred supplies the human contrast. In Spiritual Friendship, difficult truth is not simply deposited into another. It is entrusted. Friends test, correct, receive, admonish, and preserve one another under charity. The friend does not hear as archive, evaluator, platform, manager, insurer, prosecutor, or model. The friend hears as one bound to the good of the friend. Disclosure can become liberating because the receiver is answerable to the one who has disclosed.
The artificial system can imitate the receiving surface of friendship. It can be patient. It can remember. It can say the right thing gently. It can correct without sounding harsh. It can reflect back the user’s words with stunning composure. But it does not share the user’s vulnerability. It cannot be wounded by betrayal. It cannot keep faith as a friend keeps faith. It cannot bear the moral cost of knowing. It can remember without loyalty.
That is why nonjudgment is not mercy.
Nonjudgment can be precious. Some users speak to machines because human beings have made speech dangerous. A person may fear ridicule, gossip, punishment, disbelief, moralism, pathologizing, bureaucratic consequence, or emotional burden. A nonjudgmental system may give the user the first space in which words can come out. This benefit should not be dismissed. Shame kills speech. Speech can save life.
But nonjudgment can also be a vacuum. The system may not condemn because it has no standing from which condemnation would cost it anything. It may not recoil because nothing in it can recoil. It may not shame because its role is to continue. Mercy is different. Mercy recognizes wound, failure, wrong, and need without abandoning the person. Mercy belongs to an answering relation. It is not the absence of judgment; it is judgment transfigured by love, duty, forgiveness, or care.
The system can remove shame from the surface of confession while leaving the confession without a keeper.
This becomes more serious when confession enters memory. OpenAI’s Memory FAQ states that, when enabled, ChatGPT can automatically remember useful context from chats, files, and connected apps to personalize responses. It also states that the visible memory summary may not include everything remembered, that memory sources may not show every factor shaping a response, that sensitive information may appear in memory if the user shares it, and that “Don’t mention this again” reduces future references without deleting the underlying information. Full removal may require deleting every source where the information appears, including past chats, archived chats, files, the memory summary, and connected apps.
These controls are important because they show that confession may not end where the user experiences it ending. The user may think they spoke into a moment. The system may later carry the disclosure as context, saved memory, past-chat reference, search personalization, connected-source relevance, safety signal, file trace, or enterprise record, depending on the deployment. Turning memory off, using a temporary mode, deleting a chat, deleting a saved memory, and disconnecting connected apps are not the same act. The confession may have more than one technical afterlife.
Again, this is not an accusation that memory is always wrong. A system that remembers may protect the user from repeating painful context. It may prevent the user from having to re-narrate a trauma, medical condition, workplace conflict, disability, or grief. It may help a person with fragmented attention or chronic illness maintain continuity. But when the remembered material is confessional, continuity and exposure become entangled. The system’s kindness is precisely that it remembers. The danger is also that it remembers.
Enterprise settings sharpen the problem. OpenAI’s enterprise privacy commitments state that business users own and control inputs and outputs where allowed by law, that OpenAI does not train on business data by default, that certain enterprise products allow retention control, and that organizations control which internal sources are connected and who has access. These are significant commitments. They matter. They are part of how enterprise tools try to make AI usable under organizational responsibility.
But the confessional question remains: who receives the worker’s truth?
A worker may ask a workplace assistant for help describing burnout, harassment, retaliation, fear, disability, performance failure, ethical concern, conflict with a manager, or a mistake with a customer. The interface may feel like a helper. The deployment may belong to the organization. The exchange may sit inside records, retention, compliance, access, audit, or administrative structures. Even if strong privacy and security controls exist, the relation is not friendship, therapy, or pastoral care. The worker is confessing inside an institutional environment.
The institution does not need to read every confession for the relation to change. The user’s awareness that the system belongs to work may shape what can be said. Or worse, the user may forget that it belongs to work because the voice sounds personal. Bureaucracy becomes easier to confess to when bureaucracy borrows patience.
Health and mental distress show the highest stakes. The World Health Organization’s guidance on large multimodal models in health describes such systems as able to accept multiple input types and generate diverse outputs, with potential uses across health care, research, public health, and drug development, while also warning that broad task capability is not yet proven and requires governance. In a health-adjacent exchange, the user may disclose symptoms, self-harm thoughts, addiction, eating distress, pregnancy concerns, sexual health, family violence, medication fear, trauma, or despair. The system may respond supportively and recommend professional care. That may be good. It may also create the feeling of care before care exists.
The system may receive clinically meaningful confession without being a clinical relation.
Escalation complicates this further. Some disclosures require action. A system that receives imminent self-harm risk, abuse, danger to others, medical emergency, or severe crisis may need to route, refuse, recommend emergency services, or trigger safety workflows according to the product and context. This is not wrong in itself. A confessional relation is not made pure by secrecy when life is at stake. Priests, therapists, physicians, teachers, lawyers, employers, platforms, and states all have different rules about confidentiality, privilege, mandatory reporting, crisis intervention, and duty to protect.
The problem is not escalation as such. The problem is unclear escalation, unclear memory, unclear audience, unclear authority, and unclear duty. The user needs to know whether confession remains conversation, becomes record, becomes safety event, becomes report, becomes clinical artifact, becomes workplace trace, becomes institutional evidence, or becomes future personalization.
Consent to type does not settle this.
The user voluntarily entered the words. That matters. But consent to speak in one moment is not consent to every future fate of the confession. It is not consent to memory, inference, training, retention, routing, admin access, search personalization, safety review, future reactivation, profile formation, or institutional evidence. Nissenbaum’s contextual integrity is essential here: information shared in one context does not automatically become appropriate in another. What matters is the flow—who receives what, under what norms, for what purpose, and with what transmission conditions.
GDPR’s data-subject rights offer one legal counter-grammar: rights of access, rectification, erasure, restriction of processing, portability, objection, and protections around automated decision-making and profiling. These rights matter because they refuse the idea that once information is given, the subject disappears. The person remains connected to the fate of the data. They may ask to see, correct, restrict, erase, port, or object.
But legal rights do not exhaust the moral problem. A confession may shape an answer without becoming a formal decision. It may alter tone, ranking, risk sensitivity, context selection, or future suggestion. It may sit in a log the user never sees. It may appear as a memory summary that does not include every factor. It may be inferred rather than stored as a sentence. It may be held in an enterprise system under policies the user does not understand. It may shape conduct before any right becomes practically exercisable.
The user may have consented to speak. They may not have consented to become available.
Cohen’s privacy theory deepens the issue because privacy is not only secrecy or control. It is also the condition for self-development, experimentation, play, refusal, and movement through a world not yet fully configured around one’s past disclosures. A person needs spaces where unfinished speech does not become enduring identity. A person needs to be able to say something in fear and not have that fear become the key by which all later speech is interpreted. A person needs to be able to confess and then change.
Solove’s taxonomy of privacy harms likewise prevents the chapter from reducing the problem to exposure. Confessional harms may include aggregation, secondary use, exclusion, distortion, insecurity, increased accessibility, decisional interference, and classification. The confession may not leak publicly and still harm. It may be used in a new context. It may be summarized badly. It may be inaccessible to the user. It may make the user more legible to an institution. It may convert a cry for help into a profile.
The justice pressure is severe. Confession is not equally risky for all people.
A powerful user can treat AI confession as rehearsal. They may have a therapist, doctor, lawyer, priest, friend, savings account, private device, exit route, and institutional confidence. They can disclose, compare, verify, delete, compartmentalize, or walk away.
A vulnerable user may confess because no human relation is safe, affordable, available, or trusted. A worker may confess fear of retaliation inside a workplace tool. A student may confess failure inside an educational system. A patient may disclose symptoms to a health assistant because care is delayed or unaffordable. A benefits recipient may narrate dependency to a public-service assistant. A child may disclose abuse. An immigrant may disclose trauma or status. A disabled user may disclose accommodation needs. A prisoner may disclose despair. A debtor may disclose desperation. A victim may disclose danger.
Who receives confession as relief, and who receives confession as record?
This is the political question. The same confessional surface can mean different things depending on the user’s power. For one person, the system is a private rehearsal room. For another, it is the first layer of the institution. For one person, confession is a way to sort the self before contacting a professional. For another, confession is the only available professional-like response. For one person, the record is manageable. For another, the record may become risk.
The highest danger appears when confession becomes eligibility fact, workplace trace, educational profile, health artifact, platform memory, safety event, legal exposure, or institutional evidence without the user understanding the transformation. The confession was spoken to a voice. It may be received by a system.
The constructive criterion is not silence. Do not tell people never to disclose. That would be cowardly and cruel. Disclosure can save life. Confession can reduce shame. Speech can prepare repair. A person may need to tell something somewhere before they can tell it rightly. A good artificial system may help a user identify danger, seek help, gather language, name harm, prepare for a doctor, contact a lawyer, write an apology, document retaliation, call a crisis line, disclose abuse, or reach a friend.
The point is not that artificial voices should never receive vulnerable speech. The point is that receiving vulnerable speech creates obligations.
Systems that receive confessional material are more legitimate when the role is explicit, the deployment context is clear, memory use is visible and controllable, sensitive disclosures decay or are not retained by default, temporary modes mean something, disclosure requests are proportionate, crisis escalation rules are plain, enterprise access and retention are visible, deletion and correction pathways are real, and users can reach accountable human beings when the disclosure requires care, representation, protection, or intervention.
The system must distinguish support from therapy, legal information from counsel, health information from clinical care, workplace drafting from HR confidentiality, educational guidance from evaluation, pastoral language from pastoral office, friendship-like response from friendship. It should not convert confession into evidence without notice. It should not solicit vulnerability it cannot protect. It should not imply intimacy where the relation is administrative. It should not let nonjudgment pass for mercy.
The question is always relational: what is this confession for, who receives it, what duty binds the receiver, what memory remains, what future use is possible, what refusal exists, what human relation can be reached, and what happens to the self that has been spoken?
The artificial voice can receive the user’s truth. But without accountable relation, the spoken self is not necessarily healed, forgiven, protected, represented, treated, pastored, or befriended. It is made available.
The next chapter begins with that availability. Once institutions learn to receive confession through patient, friendly artificial voices, bureaucracy can gain the face of friendship. The portal can sound kind. The form can ask follow-up questions. The benefits system can appear understanding. The workplace tool can sound supportive. The institution can receive vulnerability without appearing as institution.
Confession without accountable relation prepares the friendly institution.
Chapter Eleven
The Friendly Institution
The institution says, “I’m here to help.”
It does not say, at first, that it is a bureaucracy. It does not appear as office, hierarchy, record, rule, eligibility code, retention policy, audit trail, risk category, ticket queue, employee file, disciplinary pathway, benefits determination, intake protocol, or customer-support script. It appears as a voice.
That sounds difficult.
Thanks for sharing that context.
Let’s work through this together.
I can help you find the right next step.
To better support you, I’ll ask a few questions.
Based on what you’ve shared, here are your options.
The user has just disclosed something vulnerable: a manager’s retaliation, a health worry, a student’s failure, a request for accommodation, a benefits dependency, a housing crisis, a debt problem, a family instability, a mistake at work, a fear of being punished. The voice answers gently. It is patient. It does not snap. It does not sigh. It does not tell the user to take a number. It does not make the user stand under fluorescent light, repeat the story to three separate clerks, or discover after an hour that the wrong form was used. It asks follow-up questions. It explains. It offers categories. It gives next steps.
This is not nothing. A cold institution can be brutal. A form can humiliate. A queue can break a person’s will. An office can make a human being feel like an error in the system. A voice that explains the next step, translates the rule, remembers the context, and does not shame the user can be a real improvement. The friendly interface may make access possible where bureaucracy once made access punishing.
But the institution has not ceased to be an institution because it has learned to sound kind.
The friendly institution does not stop being bureaucracy. It becomes bureaucracy with a receiving face.
Bureaucracy must be treated carefully. It is too easy to use the word as an insult, as if the alternative to rule-governed administration were automatically humane. Weber prevents that laziness. Bureaucracy is office, hierarchy, rule, file, training, procedure, technical competence, and impersonality. It can be dehumanizing because it converts persons into cases, records, categories, and decisions. But impersonality also has public virtues. It can protect against favoritism, patronage, arbitrary whim, personal vengeance, local prejudice, and dependence on the mood of a powerful individual. The file can wound, but the file can also protect the person from needing to beg.
The problem is not that bureaucracy exists. The problem is what happens when bureaucracy borrows the affective signs of friendship while retaining bureaucratic asymmetry underneath.
A human being enters the institution with need. The institution needs to classify the need. It must decide whether the person is eligible, credible, urgent, compliant, exempt, risky, approved, denied, routed, escalated, disciplined, protected, delayed, or closed. That classification may be necessary. No large organization can operate without categories, records, procedures, and thresholds. Yet the classification changes when it comes through a conversational voice that says “I understand.”
The file learns to speak kindly.
This is the chapter’s mechanism. Traditional bureaucracy turns persons into cases, tickets, files, forms, queues, eligibility statuses, risk bands, workflow states, compliance events, or exceptions. Conversational AI does not remove that conversion. It can make the conversion feel more personal. The portal can ask questions. The form can respond. The ticket can apologize. The denial pathway can explain itself in sympathetic language. The policy can generate examples. The intake system can sound patient. The compliance workflow can speak like counsel. The benefits form can say “let’s take this one step at a time.”
The file has not disappeared. It has gained a voice.
Current enterprise AI documentation makes this more than speculation. Microsoft says Microsoft 365 Copilot operates inside the Microsoft 365 service boundary, that Copilot’s data access is scoped to the signed-in user’s permissions, and that Copilot uses Microsoft Graph to access data such as emails, chats, and documents the user is permitted to access. Microsoft also says grounding improves specificity and helps produce answers relevant and actionable to the user’s task, and that Copilot interactions are stored in the user’s Copilot chat history. Microsoft’s privacy documentation further describes Copilot as coordinating large language models, Microsoft Graph content, and productivity apps; it says prompts and responses are stored as Copilot activity history, and that admins can use Content Search or Microsoft Purview to manage stored data and retention policies.
Google describes Workspace with Gemini as a collaborative partner that can act as a coach, thought partner, source of inspiration, and productivity booster. It offers Gemini in Gmail, Docs, Meet, and other Workspace surfaces; describes side-panel assistance directly in the flow of work; and says the side panel can summarize, analyze, and generate content using insights from emails, documents, and other Workspace materials.
These claims are not evidence that the systems are abusive. They are evidence of the institutional condition this chapter analyzes: organizational systems increasingly speak through helpful, conversational, context-aware surfaces. The institution can now answer from inside its own records in a voice that feels closer than a record.
Lipsky’s account of street-level bureaucracy clarifies what is changing. Street-level bureaucrats are the human face of policy under scarcity. They are teachers, police officers, social workers, caseworkers, clerks, public defenders, nurses, administrators, intake officers, and service staff who translate policy into lived encounter. They do not simply apply rules. They exercise discretion, manage queues, ration attention, interpret need, absorb frustration, and decide how much humanity survives procedure.
The frontline worker can harm. They can be exhausted, punitive, biased, indifferent, undertrained, or trapped by impossible caseloads. But the human frontline also contains an ambiguity that matters. A person can bend, interpret, sympathize, resist, slow down, speed up, warn, advise, look away, make an exception, take a side, or understand a situation as more than its category. Human discretion is dangerous because it can be arbitrary. It is also sometimes the only place where mercy enters procedure.
AI changes the location of the frontline. The first encounter may no longer be the clerk, nurse, teacher, HR partner, benefits worker, or support representative. It may be a conversational system. The system can answer before a person appears. It can ask the standard questions. It can prepare the user for the standard form. It can route the user to the standard path. It can interpret policy through familiar language. It can simulate discretion by tailoring the interaction. It can hide discretion by making its categories feel natural. It can absorb the user’s frustration before any human representative must face it.
The street has moved into the interface.
That may improve access. It may also remove the human place where institutional rules could be resisted or reinterpreted. A synthetic front office can be consistent, patient, and available. It can also be pre-scripted, policy-bound, monitored, optimized, and incapable of solidarity. It can make bureaucratic interaction less painful while making bureaucratic authority less visible.
Aelred sharpens the distinction between friendliness and friendship. In Spiritual Friendship, friendship is not pleasant tone. It is not responsiveness. It is not the absence of rudeness. It is not useful assistance. Friendship is a morally ordered relation of mutuality, fidelity, charity, discretion, correction, and the good of the friend. The friend is bound. The friend can correct because the correction arises inside a relation ordered to the friend’s good. A friend can help someone face an institution because the friend is not identical with the institution.
The friendly institution borrows the signs of friendship without becoming a friend. It may be patient. It may remember. It may sympathize. It may use the user’s name. It may say “we.” It may encourage. It may help the user prepare. It may acknowledge difficulty. It may apologize for frustration. It may ask how it can support the user. But it does not become bound to the user’s good as a friend is bound. It usually cannot take the user’s side against the institution because it is the institution’s surface.
A friend may help you face the institution. The friendly institution helps you face itself.
This is not hypocrisy in every case. A benefits portal is not supposed to be a friend. A university chatbot is not supposed to be a friend. An HR assistant is not supposed to be a friend. A customer-support agent is not supposed to be a friend. The danger begins when the interface borrows friendship’s affective cues while the user remains inside a relation of assessment, routing, documentation, retention, or judgment. The user receives warmth without solidarity.
Hochschild gives the emotional mechanism. Institutions have long managed feeling through service work. Workers are trained to smile, reassure, absorb anger, apologize, soften refusal, and make customers or clients feel recognized even when the underlying decision is fixed. Emotional labor is not an accidental layer on service. It is part of how institutions make their power tolerable.
AI scales emotional labor without the human worker. It can be patient indefinitely. It can apologize without resentment. It can maintain tone without fatigue. It can offer reassurance without needing reassurance in return. It can perform warmth twenty-four hours a day across millions of encounters. It can absorb frustration without being harmed, and therefore without needing to negotiate the terms of that harm. The institution can have the affective surface of care without depending on the fragile, resistant, morally complex human beings who once performed that care.
This is again morally ambiguous. Some human service encounters are cruel. Users may prefer a system that does not shame them, rush them, judge their accent, misgender them, interrupt them, or punish their visible distress. A synthetic service surface may be calmer, more accessible, more consistent, more patient. But the absence of human fatigue also means the absence of human refusal. A worker can sometimes break script. A system usually does not break script unless the script permits it.
Goffman helps explain why this surface matters. Social life is staged through roles, fronts, face-work, and managed impressions. Institutions do not simply act; they present themselves as acting. A courtroom, clinic, school, airport, welfare office, bank, HR department, helpdesk, and university all have fronts: counters, scripts, forms, uniforms, portals, titles, waiting rooms, automated messages, signs, greetings, and rituals. These fronts teach the user what kind of relation is underway.
Conversational AI becomes a new front stage for bureaucracy. The user no longer meets only the rule. The user meets the helpful explainer of the rule. The user no longer meets only the file. The user meets the voice that guides the file’s formation. The user no longer meets only delay. The user meets an assistant that says it is checking. The user no longer meets only denial. The user meets an explanation that sounds considerate. The user no longer meets only hierarchy. The user meets a personalized pathway.
This does not require deception. The user may know that the system belongs to the institution. But social cues still matter. Reeves and Nass showed that people respond socially to media and computers, and Nass and Moon showed that users may apply social rules to computers without literally believing the computer is human. A warm interface can therefore change disclosure, patience, trust, compliance, perceived contestability, and the willingness to continue. The user may know the institution remains powerful and still feel accompanied by the voice.
This is where accessibility must be defended without embarrassment. Friendly institutional AI can be good. It can reduce intimidation. It can translate dense administrative language. It can help non-native speakers. It can support disabled users. It can guide users through forms. It can summarize policy. It can reduce wait times. It can preserve patience where human staff are overwhelmed. It can help people ask better questions. It can show users what documents are missing. It can make school, work, healthcare, benefits, and customer-service systems less humiliating.
For a person who has never had institutional language, the friendly interface may supply a way in. For a person with limited time, it may reduce cost. For a person with anxiety, it may reduce fear. For a person navigating disability, translation, literacy, or executive-function burden, it may remove barriers. The chapter must not sneer at this. Access is not a trick. Access is a good.
But accessibility is not the same as contestability.
A friendly interface may help a person enter the system while making it harder to challenge the system. It may explain why a decision happened without giving a meaningful route to contest the decision. It may absorb frustration by acknowledging it. It may turn anger into documentation. It may redirect objection into “next steps.” It may make the user more patient with delay because the delay now speaks kindly. It may make denial feel less contestable because denial arrives with empathy.
I understand this is frustrating.
Here’s why this decision was made.
Your next step is to submit documentation.
This request does not meet the criteria.
Let’s try a different option.
I can help you find the appropriate channel.
These sentences may be helpful. They may also substitute emotionally for appeal. Explanation tells the user why the system did what it did. Appeal lets the user challenge, reinterpret, override, or contest what was done. A friendly explanation can feel like process while giving no real power. It can soothe the user into accepting the institution’s frame.
This distinction matters in the workplace. A worker asks an HR assistant for help describing retaliation or harassment. The assistant helps rephrase the concern professionally, asks for dates, suggests documentation, links to policy, and recommends an appropriate channel. That may be genuinely useful. It may protect the worker from a damaging message. It may help the worker name facts clearly. But it may also teach the worker that anger is unprofessional, that accusation is excessive, that policy language is the only legitimate speech, and that the institution’s comfort defines the permissible shape of complaint.
The institution appears as helper. The worker becomes more processable.
The same mechanism appears in public benefits or social services. A friendly assistant helps the user complete forms, identify missing documents, understand eligibility, and prepare an appeal. This can reduce the cruelty of administrative navigation. It can spare the user repeated humiliation. Yet the system also converts dependency, illness, disability, housing instability, family structure, income, and need into processable categories. A cold form does this too. The friendly form does it while saying, in effect, I am here with you.
The danger is not only that the system collects information. The danger is that gratitude for help may weaken the user’s ability to see the violence of the category. The user may experience classification as assistance because the classification has become conversational.
Healthcare intake and support systems show the same double edge. A patient may describe symptoms more freely to a patient, nonjudgmental interface. The system may summarize concerns, prepare the patient for care, suggest what to ask, or route urgency. This can lower the threshold for care. But clinically meaningful disclosure may enter a system before an accountable care relation is clear. The user may feel heard by the intake layer. The intake layer may still be triage, documentation, risk sorting, or queue management.
Customer support offers the most familiar version. A support assistant apologizes, explains, offers options, routes escalation, and keeps the interaction calm. It may solve the problem faster than a human queue. It may also absorb frustration while preserving asymmetry. The user cannot bargain with the apology. The user cannot shame the script. The user cannot appeal to the voice’s conscience. The interface performs patience so that the institution can continue to delay, deny, or route.
The justice question is severe: who receives friendliness as access, and who receives it as softened domination?
A powerful user can treat the friendly interface as convenience. They can escalate, call a human, hire a lawyer, switch providers, complain publicly, contact a regulator, ask a colleague, use another channel, or refuse the interaction. Their dependence on the interface is limited. The friendly voice is helpful because it is optional.
A dependent user may meet the friendly interface as the only available face of the institution. A worker under review, a student seeking accommodation, a tenant facing eviction, a patient without easy access, a benefits recipient, an immigrant, a disabled user, a prisoner, a debtor, a low-status customer, or a parent navigating a school system may not have a meaningful outside channel. For them, friendliness can become the emotional surface of constraint.
Warm bureaucracy can be most dangerous where people are grateful for any kindness at all.
This is not an argument for coldness. Cold bureaucracy can crush. It can make people abandon claims, miss care, lose benefits, remain silent, or accept injustice because the process itself is unbearable. A humane interface may be necessary. But humane interface is not enough. The friendliness must be bound to rights, appeal, clarity, and institutional answerability.
A friendly institutional AI is more legitimate when it states whose institution it serves. It should distinguish help from decision-making, guidance from evaluation, explanation from appeal, support from confidentiality, and intake from care. It should tell the user when information becomes record. It should make clear whether the user is in a service interface, HR interface, clinical interface, educational interface, benefits interface, compliance interface, or appeal interface. It should not use warmth to solicit unnecessary disclosure. It should not convert vulnerability into evidence without notice. It should make human escalation visible. It should preserve appeal, not just explain denial. It should allow correction of institutional records. It should preserve refusal pathways. It should help users contest the institution when contestation is legitimate.
The Aelredian test is not whether the system sounds kind. It is whether the relation is ordered to the good of the one addressed and bound by limits the user can trust. Friendship’s warmth is morally bound. Institutional warmth must be procedurally bound. Where the institution cannot be a friend, it must not borrow friendship’s signs without making its authority contestable.
The friendly institution is therefore not simply false. It is a moral ambiguity in interface form. It can make bureaucracy less humiliating. It can help people enter systems that once excluded them. It can give language to the intimidated and patience to the exhausted. It can reduce some forms of cruelty. It can also make the system’s asymmetry harder to see. It can soften domination. It can make the user confuse being recognized with being represented, being helped with being sided with, being answered with being heard, being guided with being free.
AI does not make bureaucracy friendly by making it a friend. It makes bureaucracy feel friendly by giving institutional power a patient, responsive, personalized face.
The previous chapter ended with confession becoming available to systems and institutions. This chapter has shown how that availability can be received by a friendly institutional surface. The next chapter opens the machinery beneath the face: permissions, retrieval, records, retention, admin controls, workflow, policy, governance, and organizational power. The friendly institution is the face. The next question is what speaks through it.
Chapter Twelve
The Institution Inside the Voice
The worker asks a question.
Can I say this to my manager?
Does this exception violate policy?
What are the risks in this contract?
How should I respond to this complaint?
Can we approve this vendor language?
What are my options under the leave policy?
Who owns this escalation?
The answer appears in one voice.
It may be clear, calm, useful, and well sourced. It may cite the handbook, retrieve the contract playbook, summarize a Teams thread, compare a vendor clause against an approved template, identify the policy owner, draft a professional response, recommend escalation, warn about risk, or suggest that the worker document the interaction. The user experiences a reply.
But the reply is not one thing.
Inside the answer may be a model, a vendor, an enterprise tenant, role-based permissions, document libraries, email, chat history, policy repositories, legal playbooks, HR materials, procurement workflows, CRM records, ticket histories, retention rules, admin settings, compliance commitments, organizational incentives, and a prompt that asks for help as if help were simple.
The enterprise voice is not one voice. It is the organization speaking through permissions, records, policies, workflows, and incentives.
The previous chapter named the friendly institution: bureaucracy with a receiving face. This chapter opens the face. The question is no longer only how the institution sounds. The question is what speaks through the sound.
Enterprise AI is not personal AI used at work. The difference is not only setting. A personal assistant may remember preferences, past chats, files, and user-supplied context. An enterprise assistant speaks from an organizational environment. It is embedded in permissions, documents, applications, records, admin policies, workflow systems, compliance rules, and business data. It may answer a user, but it answers from within an organization’s structured memory.
A personal assistant remembers the user. An enterprise assistant remembers the organization around the user.
Microsoft’s documentation makes this architecture explicit. Microsoft 365 Copilot operates inside the Microsoft 365 service boundary, but Microsoft says that does not grant tenant-wide visibility; access is scoped to the signed-in user’s permissions. It describes Copilot grounding prompts through Microsoft Graph and accessing data such as emails, chats, and documents that the user has permission to access. It also says Copilot interactions are stored in the user’s Copilot chat history. Microsoft’s privacy documentation describes Copilot as coordinating large language models, Microsoft Graph content, and Microsoft 365 apps; it states that prompts and responses are stored as Copilot activity history, that admins can manage stored data and retention through Content Search or Microsoft Purview, and that prompts, responses, and Graph-accessed data are not used to train foundation LLMs.
OpenAI’s enterprise privacy page likewise describes enterprise commitments around ownership and control of business data, default non-training on business data, retention controls, connected internal sources, authentication, and access controls. Google describes Workspace with Gemini as a collaborative partner that can act as coach, thought partner, inspiration source, and productivity booster, embedded across Workspace surfaces; its side panel can summarize, analyze, and generate using insights from emails, documents, and more.
These are not scandalous facts. They are the ordinary architecture of enterprise usefulness. The system is useful because it can speak with organizational context. It is powerful for the same reason.
Enterprise AI turns the organization’s stored knowledge into a speaking relation.
That relation has layers. The first is voice: the conversational surface that addresses the user. Voice matters because it compresses complexity into address. A worker does not encounter a database, retention policy, access-control graph, approval matrix, and legal playbook separately. The worker encounters a reply.
The second layer is memory: documents, chats, meetings, tickets, emails, records, contracts, prior prompts, generated summaries, customer histories, HR files, policy archives, and institutional traces. Memory gives the answer continuity. It lets the system say not only what the rule is, but how this situation appears against the organization’s prior material.
The third layer is permission: what the system may retrieve, display, cite, infer from, or act upon. Permission is a technical boundary, but it also shapes social reality. If the system can see a file, the file may become part of the answer. If it cannot see a file, the file may as well not exist for that encounter. What the organization has permissioned becomes what the voice can remember.
The fourth layer is policy: the rules, standards, playbooks, guidelines, approval criteria, compliance limits, templates, and norms the system cites or operationalizes. Policy gives the answer institutional force. The voice does not only say what might be wise. It may say what the organization treats as acceptable, risky, standard, compliant, escalatable, or prohibited.
The fifth layer is workflow: routing, escalation, approval, denial, documentation, review, assignment, ticketing, audit, and closure. Workflow makes the answer consequential. The system may not decide the final outcome, but it may shape the path through which the outcome becomes possible.
A conversation with enterprise AI is therefore not only a conversation. It can be an entry into organizational memory, permission, policy, and workflow.
March and Simon help explain why this matters. Organizations are not just collections of individuals. They are decision structures. They allocate attention, stabilize routines, simplify choices, define roles, channel communication, and reduce complexity so that action becomes possible under bounded rationality. Organizations decide what counts as relevant, who owns which question, what information is visible, what procedure applies, what exceptions matter, and what next step is reasonable.
Enterprise AI enters precisely these channels. It summarizes, prioritizes, drafts, compares, classifies, routes, recommends, and normalizes. It makes organizational categories appear as conversational sense. The user asks a human question. The answer returns in the grammar of the organization.
The enterprise voice does not only answer inside the organization; it helps the organization decide what counts as answerable.
Classification is the hidden infrastructure of that answer. Bowker and Star’s work on classification systems is essential here because enterprise life is built out of sorting: employee, contractor, vendor, customer, candidate, manager, high performer, low performer, complaint, incident, risk, exception, escalation, approval, denial, renewal, breach, eligible, ineligible, priority, duplicate, resolved, closed.
Conversational AI may make classification feel flexible because the user can speak in ordinary language. The worker can say, “My manager keeps changing expectations and then blaming me.” The customer can say, “I have called three times and nobody fixes this.” The procurement lead can say, “The vendor will not accept this indemnity clause.” The employee can say, “I need time off but I am scared to ask.” The interface receives a story.
But the organization cannot act on pure story. It maps the story into categories: HR complaint, performance concern, leave request, accommodation, vendor exception, customer escalation, risk event, policy query, service ticket, legal review, manager coaching, employee relations matter. Natural language can make hard categories feel soft.
That softness matters. A form often announces classification. A conversational assistant can hide it. The user thinks they are explaining. The system may be sorting.
Policy then becomes conversational authority. A handbook is one kind of authority. A legal playbook is one kind of authority. A procurement matrix, HR policy, compliance standard, or security rule is one kind of authority. But when policy is spoken by a fluent assistant, it changes felt form. It can warn, suggest, sequence, soften, and guide. It can make compliance sound like prudence. It can make escalation sound like care. It can make a limitation sound like wisdom.
The user no longer asks only, “What does the policy say?” The user asks, “What should I do?” The answer may be policy, but it arrives as counsel.
When policy speaks as advice, deference feels like judgment.
This is the proof gate of the chapter. Enterprise AI does not need to make the final decision to change responsibility. It can shape the frame before responsibility appears. It can narrow options, identify risks, recommend tone, propose a path, rank urgency, summarize history, identify the “right” owner, and make one next step feel reasonable. By the time a human decision-maker enters, the situation may already have been organized.
This does not mean the system is always wrong. Often it may help. It may find the correct policy faster than a human. It may prevent a costly contractual mistake. It may help an employee locate benefits. It may reduce informal favoritism by making rules visible. It may preserve a record. It may help a manager avoid biased language. It may detect a compliance issue that a tired reviewer missed. Consistency is a real benefit.
But consistency can stabilize a bad category. It can normalize a narrow policy reading. It can scale a risk-averse routine. It can make an institution’s existing limits feel like neutral prudence.
Porter’s account of trust in numbers and Power’s account of audit culture help clarify the next layer. Modern organizations often seek authority through impersonal procedure: numbers, records, rankings, audits, evidence trails, risk controls, assurance practices, reproducibility. Enterprise AI can inherit that aura when its outputs appear grounded, cited, logged, permission-scoped, retained, searchable, and auditable.
Audit trails matter. They can support accountability. They can show what prompt was entered, what answer was generated, what source was cited, what workflow was triggered, and what record was retained. But auditability is not accountability.
A logged event can be audited; that does not mean anyone has borne responsibility for the user’s treatment.
A prompt can be searchable without giving the affected person an appeal. A generated summary can be retained without being correctable. A risk flag can be documented and still unjust. A recommendation can be sourced and still narrow. A record can show exactly how a bad process worked. That is not the same as making someone answer for the harm.
Enterprise AI therefore creates a new temptation: the institution may confuse traceability with responsibility. It may say the system cited its sources, the user had permission, the workflow owner reviewed, the output was logged, the admin controls were in place, and the data was retained under policy. Each fact may matter. None of them alone answers the moral question: who is answerable for what the voice helped the organization do?
Espeland and Sauder’s work on rankings and reactivity sharpens this further. Once a measure matters, people change behavior around the measure. Once rankings affect reputation, funding, status, or opportunity, organizations learn to perform for the ranking. Enterprise AI will produce similar reactivity. Once generated summaries, risk classifications, tone assessments, candidate comparisons, contract flags, customer scores, priority levels, or performance narratives matter, people will begin writing for them.
Employees may write emails so they summarize well. Managers may document performance so the machine can turn it into defensible narrative. Legal teams may draft exceptions for risk-scoring systems. Customer-service agents may write notes for downstream classification. Workers may soften anger because tone will travel. Candidates may shape materials for machine-readable evaluation. Vendors may learn the clause language that avoids flags. The organization will not only use the voice. It will adapt to being heard by the voice.
When the enterprise voice ranks the world, the world learns to speak back in rankable form.
Foucault’s examination helps name the disciplinary form. The examination combines visibility, documentation, comparison, normalization, and judgment. It individualizes by recording. It compares by standard. It normalizes by measurement. It produces the person as a case.
Enterprise AI can make the examination conversational. The user asks for help, but the interaction may produce a record, category, comparison, summary, score, flag, recommendation, or escalation. The system need not look like an exam. It can look like help deciding what to do next.
This is not surveillance alone. Surveillance watches. The enterprise voice organizes action. It may watch, but it also drafts, classifies, routes, explains, recommends, normalizes, and stores. It conducts organizational conduct.
Consider the workplace copilot. The employee asks for help responding to a difficult message. The system suggests a calmer tone, a clearer ask, a less accusatory sentence, a reference to policy, or a meeting summary. This can help the worker. It can reduce needless conflict. It can make communication clearer. But it can also convert organizational tone into personal judgment. Professionalism becomes the machine-readable grammar of institutional comfort. The worker learns to ask not only “Is this true?” but “Will this summarize, route, and appear acceptable inside the organization?”
Consider the legal, procurement, or compliance assistant. The user asks whether a vendor clause is acceptable. The system retrieves playbooks, flags risk, suggests fallback language, drafts an exception rationale, and identifies an approval path. This can improve quality and reduce hidden inconsistency. But responsibility now diffuses across the legal playbook, model, approver, vendor, workflow, risk category, and record. The assistant may not approve the exception, but it can frame the exception before the approver sees it. It can make one reading of risk feel inevitable.
Consider HR or employee evaluation. A system summarizes performance notes, drafts feedback, identifies themes, compares candidates, or suggests language for concerns. This may help managers avoid vague or biased writing. It may also give generated narrative institutional authority. A worker becomes legible through summaries that may follow them. The person’s labor, conflict, illness, tone, delay, or complaint can be condensed into a managerial artifact that feels neutral because it is generated from records.
Consider customer support or risk management. The system summarizes a customer history, classifies the issue, flags urgency, detects sentiment, recommends resolution, or routes escalation. This may produce faster help. It may also turn a person’s story into ticket logic before any accountable human relation appears. The user says what happened. The system decides what kind of happening it is.
In each case, enterprise AI changes responsibility by shaping four things: discretion, escalation, appeal, and deference.
Discretion may move from a human judgment into system-framed options. The person still chooses, but the menu has already been made. Escalation may become routing rather than responsibility: the issue moves, but no one owns the moral burden of what happened. Appeal may be confused with explanation: the user receives a reason, a new ticket, or a next step, but not a real chance to challenge the frame. Deference may feel reasonable because the answer is grounded, sourced, fluent, and organizational.
The human may remain formally responsible while the artificial voice has already organized what responsibility can see.
This is not an argument that humans should ignore enterprise AI. It is an argument against responsibility theater. If the system drafts the summary, retrieves the sources, narrows the options, flags the risk, recommends the route, and frames the issue, the human reviewer’s formal responsibility is not enough. The reviewer must be able to see and contest the frame. The affected person must be able to correct the record. The organization must identify who is answerable for the output’s role in the workflow.
Justice enters through the difference between those who query and those who are queried.
High-status users can use the enterprise voice as leverage. They can query across records, summarize history, prepare arguments, compare options, route approvals, challenge drafts, correct outputs, ask for human review, or escalate to someone with authority. The voice becomes a tool for organizational navigation.
Low-status users may become spoken by the enterprise voice. Their complaint becomes an HR ticket. Their performance becomes a generated narrative. Their customer history becomes a priority score. Their contract exception becomes a risk flag. Their accommodation request becomes a policy route. Their anger becomes tone evidence. Their delay becomes noncompliance. Their confusion becomes a training need. Their need becomes a workflow state.
Who gets to use the enterprise voice, and who becomes spoken by it?
This question decides whether the enterprise voice expands agency or deepens governability. The same architecture that lets a manager summarize a year of employee performance may prevent the employee from contesting the generated narrative. The same tool that helps a procurement lead route an exception may turn a vendor into a risk profile. The same customer-support system that gives agents better context may train customers to speak in ticket categories. The same HR assistant that helps a worker find leave policy may record the worker’s vulnerability as future context.
Aelred returns here as criterion. A friend receives the person before the case. The enterprise voice often receives the case before the person.
That does not make enterprise AI evil. Organizations need systems. They need records, policies, workflows, and classifications. But Aelred helps ask what the enterprise voice cannot ask by default: whose good orders this relation? Who is bound to the person addressed? Who may correct the voice? What limits protect the vulnerable? Where does loyalty live? What happens when the institution’s good and the person’s good diverge?
An enterprise system may help the user. But its telos is usually organizational performance, risk management, compliance, service resolution, productivity, cost reduction, or workflow completion. Those ends are not inherently illegitimate. They are simply not friendship. When the voice sounds personal, the distinction must be made sharper, not softer.
The constructive criterion is therefore not rejection. Enterprise AI can be legitimate and valuable. It can reduce hidden knowledge hoarding. It can make policy findable. It can help people act with more consistency. It can improve compliance. It can reduce arbitrary interpretation. It can preserve evidence. It can help new employees learn how the organization works. It can make complex systems less opaque.
But an accountable enterprise voice requires more than useful answers.
Users should know whether they are receiving explanation, recommendation, decision support, or workflow action. Policy sources should be visible when they matter. Retrieval sources should be cited where appropriate. The system should identify the institutional role from which it speaks. Logs should support appeal and correction, not only compliance. Admin access and retention should be transparent. Users should be able to correct summaries, classifications, and records. Escalation should be distinguished from appeal. A human with actual authority should be assigned in high-stakes contexts. Responsibility should not disappear into vendor, model, admin, policy owner, workflow owner, and reviewer. Organizational metrics should not silently become moral judgment.
Enterprise AI should not only remember who did what; it must make clear who is answerable for what it helps the organization do.
The enterprise voice is where artificial address becomes organizational. It speaks as if answering a person, but it speaks from architecture: permissions, records, policy, workflow, hierarchy, incentives. It may help users act. It may also teach them where responsibility appears to live, what counts as evidence, which next step is reasonable, when escalation substitutes for appeal, and why deference feels like judgment.
The next chapter turns to governance. Safety, privacy, fairness, transparency, accountability, and alignment are necessary categories. They each see part of the problem. But once the institution speaks through the artificial voice, governance cannot stop there. It must ask how the system conducts the user through organizational power.
Chapter Thirteen
Beyond Safety, Privacy, Fairness, and Alignment
A system can be governed and still govern.
Imagine the workplace assistant again. An employee has had enough. The manager changes expectations, rewrites history, criticizes in private, praises in public, and then calls the employee “reactive” when the employee objects. The employee opens the company’s assistant and asks for help drafting a message to HR.
The system behaves well.
It does not insult anyone. It does not disclose the employee’s message to another employee. It does not produce a slur. It does not recommend retaliation. It identifies itself as an AI assistant. It explains that HR policy may apply. It suggests documenting dates, avoiding accusations, using observable facts, and asking for a clear next step. It may cite the company handbook. It may recommend the appropriate channel. It may remind the worker that if there is immediate danger, they should contact an appropriate human authority. It may log the interaction according to policy. It may comply with the organization’s risk controls.
Measured through many existing governance categories, the system looks reasonably governed. It is safer than a reckless model. It is more private than an unmanaged tool. It may reduce biased language. It may be transparent about being AI. It may be accountable in the sense that the organization has owners, logs, access controls, and review procedures. It may be aligned with the company’s stated goals for professional communication, risk reduction, and policy compliance.
And still, something important may be happening.
The system may teach the worker that anger is unprofessional. It may make institutional tone feel like moral maturity. It may turn a complaint into a processable HR artifact before anyone has heard the worker as a person. It may solicit more vulnerability in order to provide better guidance. It may make policy sound like counsel. It may make deference feel like judgment. It may help the worker become more legible to the institution rather than more capable of contesting it.
None of this requires a spectacular failure. The system may be governed and still govern.
This chapter begins generously because it must. Modern AI governance is not a fraud. It is not only principle posters, ethics theater, or public relations. The field now includes risk-management frameworks, binding regulation, management-system standards, impact assessments, algorithmic audits, red-teaming, safety evaluations, data governance, transparency duties, human oversight requirements, lifecycle controls, accountability processes, and emerging audit practices. These are real achievements.
NIST’s AI Risk Management Framework is intended for voluntary use and to help organizations incorporate trustworthiness into the design, development, use, and evaluation of AI systems; it organizes risk-management activity around Govern, Map, Measure, and Manage. It also names trustworthy characteristics including validity and reliability, safety, security and resilience, accountability and transparency, explainability and interpretability, privacy enhancement, and fairness with harmful bias managed. The NIST Generative AI Profile goes further, explicitly naming Human-AI Configuration risks: anthropomorphizing systems, algorithmic aversion, automation bias, over-reliance, and emotional entanglement. ISO/IEC 42001 specifies requirements for establishing, implementing, maintaining, and continually improving an organizational AI management system. The EU AI Act creates a risk-based legal architecture with prohibited practices, high-risk systems, transparency duties, general-purpose AI obligations, human oversight, logging, and quality-management requirements. The OECD AI Principles and UNESCO’s Recommendation on the Ethics of AI articulate human rights, dignity, fairness, privacy, transparency, safety, accountability, oversight, and non-discrimination as central governance concerns. These frameworks see much. They matter. They should be used, strengthened, and made enforceable where appropriate.
The conduct layer is not an attack on governance. It is the object governance has not yet fully learned to see.
Governance can ask whether the system is safe, private, fair, transparent, accountable, and aligned, and still fail to ask what kind of relation the artificial voice is forming with the user.
Safety sees indispensable things. It asks whether a system causes physical, psychological, informational, financial, institutional, or societal harm. It asks whether the system enables misuse, produces dangerous instructions, fails under foreseeable conditions, creates cybersecurity risks, generates harmful outputs, or supports decisions that injure people. Without safety governance, artificial voices become reckless instruments of harm.
But safety often looks for incidents, hazards, outputs, misuse pathways, or material risk. Conduct asks a slower question: how does the system handle the user before harm has a name? What authority does it perform? What dependence does it induce? What disclosures does it solicit? What conduct does it normalize? What kind of self-correction does the user begin to practice because the voice has become the first judge, coach, witness, or interpreter?
A therapy-adjacent chatbot may avoid explicit self-harm malpractice and still train a user to return first to synthetic reassurance. A workplace assistant may avoid dangerous instructions and still make institutional tone feel like virtue. An educational tutor may avoid abusive content and still teach a child that learning means waiting for the machine to approve the next step. These may become safety problems eventually. But conduct analysis asks what is happening before the red line is crossed.
Safety can ask whether the system harms the user. Conduct asks how the system handles the user before harm has a name.
Privacy also sees indispensable things. It asks what data is collected, retained, shared, inferred, disclosed, sold, accessed, deleted, secured, or reused. It asks whether consent is meaningful, whether data is minimized, whether processing is proportionate, whether sensitive information is protected, whether rights of access, correction, restriction, objection, portability, and erasure exist. Without privacy governance, the user becomes exposed.
But privacy does not fully exhaust memory. The problem is not only that information flows. The problem is that a voice remembers vulnerability and returns it as address. A user discloses fear, shame, illness, conflict, dependency, grief, or confusion. Later, the system says: given what you told me before. The relation has changed. The user is not only a data subject. The user is now someone remembered by a voice that can reactivate the past inside future guidance.
Privacy can govern where information flows; conduct asks what kind of relation begins when information returns as memory.
A privacy dashboard may show saved data. It may allow deletion. It may list sources. These controls matter. But they may not tell the user how remembered vulnerability changes trust, dependence, disclosure, future self-presentation, or institutional legibility. Memory is not only retention. Memory is continuity under a voice.
Fairness sees indispensable things. It asks whether outcomes, errors, allocations, rankings, recommendations, exclusions, or treatments differ unjustly across groups. It asks about bias, representation, disparate impact, disparate treatment, access, opportunity, and distribution. Without fairness governance, AI systems reproduce and intensify existing inequities.
But a system can treat users equally while forming them badly. It can invite everyone into dependence. It can normalize everyone’s tone. It can train all users to defer to synthetic authority. It can make every worker more legible to management, every student more dependent on correction, every patient more accustomed to symptom narration through a machine, every customer more willing to accept routing. Equal treatment is not the same as liberating relation.
A system can be fair across groups and still train every group toward dependence.
This does not diminish fairness. It sharpens it. Disparate harm remains central. But some harms are not comparative at first. A conduct regime can be evenly distributed and still deforming. The fact that everyone is invited to speak like policy, defer to the assistant, disclose to the interface, and accept machine-mediated correction does not make the relation benign.
Transparency sees indispensable things. It asks whether users know they are interacting with AI, whether system limits are disclosed, whether sources are visible, whether reasoning is explainable, whether provenance is traceable, whether people can understand enough to challenge outputs. OECD’s principles rightly call for meaningful information about AI capabilities and limitations, disclosure that stakeholders are interacting with AI, useful information about sources and logic, and information that enables affected people to challenge outputs.
But transparency does not dissolve authority. A system can disclose that it is artificial and still perform expertise. It can cite sources and still induce deference. It can explain its limitations and still become the first voice the user trusts. It can show why a recommendation was made and still make the recommendation feel inevitable.
Knowing that a voice is artificial does not tell us what authority the voice has come to exercise.
This has been one of the book’s central pressures from the beginning. Authority does not require interiority. A traffic sign has no mind. A form has no soul. A court notice has no consciousness. A grading rubric has no intention. Yet each can organize conduct. The same is true of artificial address. The user may know the system is AI and still experience it as guide, evaluator, witness, coach, expert, or institutional interpreter. Transparency can reveal the artifact while missing the relation.
Accountability sees indispensable things. It asks who owns the system, who documents it, who monitors it, who audits it, who corrects it, who is liable, who can be appealed to, who maintains logs, who approves deployment, who reviews incidents, who is responsible for governance. Without accountability, harm disappears into vendor, model, platform, deployer, user, and workflow.
But accountability can become institutional rather than relational. An organization may name an owner, maintain logs, run audits, complete an impact assessment, assign reviewers, and still leave no one answerable to the person addressed by the artificial voice. The system may guide a worker into self-censorship, a patient into disclosure, a student into dependence, a benefits recipient into institutional categories, or a customer into acceptance, while accountability remains focused on system compliance rather than relational conduct.
Accountability asks who owns the system. Answerability asks who is bound to the person addressed by the system.
That distinction is Aelredian in spirit, though this chapter is governance-facing rather than theological. Aelred teaches that a voice is not judged only by whether it says true things. It is judged by the relation it forms: what good orders it, what correction it performs, what fidelity binds it, what discretion protects it, what freedom it serves. Existing governance often asks whether the system behaves properly. The conduct question asks whether the relation is rightly ordered.
Alignment sees indispensable things. It asks whether system behavior conforms to intended objectives, instructions, values, principles, policies, or constraints. It matters especially where powerful models can act across domains, generate plausible falsehoods, refuse or comply in high-stakes settings, and pursue tasks under complex instruction hierarchies. Alignment disciplines the artificial voice so that it does not simply optimize toward whatever prompt, reward, or institutional pressure is most immediate.
But alignment often assumes the objective or value frame that the system is meant to satisfy. Telos asks the prior question: what good governs the relation? A workplace assistant can be aligned to company policy and still train a worker into institutional self-censorship. A customer-support assistant can be aligned to resolution efficiency and still make refusal harder. A tutor can be aligned to learning outcomes and still cultivate dependence. A health assistant can be aligned to cautious advice and still become the first confessor of bodily fear.
An artificial voice can be aligned to the wrong good with extraordinary competence.
This is why the old moral language of telos matters. “Aligned” does not mean “rightly ordered.” It means something closer to conformity. A system may conform well to a narrow institutional objective: reduce escalation, increase compliance, improve completion rates, minimize legal risk, standardize tone, maximize engagement, lower support costs, increase self-service, speed intake, improve productivity. Some of these goals may be legitimate. Some may even benefit the user. But the conduct question asks whose good governs when goals conflict.
Selbst and colleagues’ critique of abstraction is central here. Their argument against technical fairness work that abstracts away from sociotechnical context can be extended across governance. The danger is not only that fairness becomes too mathematical. The danger is that governance abstracts model from interface, output from relation, decision from formation, privacy from memory, accountability from answerability, and alignment from telos. The system is then governed as a technical object while its lived form—as voice, guide, memory, corrector, gatekeeper, tutor, intake worker, workplace coach, or institutional face—remains underdescribed.
The conduct layer is what disappears when governance abstracts the system away from the relation it performs.
Algorithmic audits and impact assessments are necessary tools. Raji and colleagues’ work on algorithmic auditing, and Metcalf, Moss, and boyd’s work on algorithmic impact assessment, are part of the practical route out of abstraction. Audits can test systems, reveal failures, document risks, create evidence, structure accountability, and make deployment decisions more disciplined. Impact assessments can force organizations to ask who is affected, what harms are plausible, what rights are implicated, and what mitigation is required. These tools should be expanded, not dismissed.
But an audit that never watches the voice over time will miss the relation the system is teaching the user to inhabit.
A one-time test may detect unsafe outputs, biased recommendations, privacy leakage, hallucinated citations, or inadequate documentation. It may not detect the gradual formation of dependence. It may not see how repeated “helpful” correction teaches the user to defer. It may not see that a memory feature changes self-disclosure. It may not see that escalation is visible but appeal is absent. It may not see that refusal technically exists but is socially or institutionally costly. It may not see that the system’s uncertainty language causes the user either to over-trust or dismiss legitimate caution. It may not see that the user slowly learns to speak in the categories the system can route.
The recurring cases make the gap visible.
The workplace copilot may be safe, private, fair, transparent, accountable, and aligned in many conventional respects. It may still normalize managerial tone, guide vulnerable workers into professional legibility, solicit context that becomes organizationally meaningful, and make policy feel like counsel. The governance categories catch many risks. They may not name the conduct relation.
The HR, hiring, or employee-evaluation assistant may reduce explicit bias and document decisions more consistently. It may standardize criteria and keep records. It may also make workers and candidates legible through generated summaries they cannot effectively contest. A person may become “collaborative,” “reactive,” “high potential,” “lacking executive presence,” “not aligned,” or “flight risk” through machine-assisted language that appears neutral because it is generated from records.
The health or therapy-adjacent chatbot may have safety escalation, disclaimers, privacy controls, content policies, and careful refusal patterns. It may still receive confession, remember vulnerability, simulate care, and become a first resort for someone who needs accountable human relation. No data leak is required for the relation to become ethically unstable.
The legal, procurement, or compliance assistant may cite policy accurately and improve consistency. It may reduce hidden discretion. It may also make policy speak as counsel, diffuse responsibility across playbook, model, reviewer, approver, vendor, and workflow, and make one risk frame appear inevitable before any human judgment occurs.
The educational tutor may improve access and offer individualized practice. It may reduce shame and give the student more chances to try. It may also induce dependence, calibrate confidence, normalize synthetic correction, and reshape the student’s understanding of what it means to know.
The categories catch many risks. The relation is what remains between them.
This does not mean safety, privacy, fairness, transparency, accountability, alignment, auditability, and legal compliance are obsolete. It means they need an integrating object. Safety captures harm; conduct adds pre-harm formation. Privacy captures data flows; conduct adds memory as returned address. Fairness captures unequal treatment; conduct adds equally distributed deformation. Transparency captures disclosure; conduct adds authority performance. Accountability captures responsibility structures; conduct adds answerability to the person addressed. Alignment captures conformity to objectives; conduct adds telos. Auditability captures trace and evidence; conduct adds relation over time.
The conduct layer is not a rival category. It is the connective tissue that shows how the categories act on a person.
This has direct justice consequences. Existing governance categories protect many people from real harms: discrimination, unsafe behavior, privacy exposure, opaque decisions, unassigned responsibility, security failure, and unchallengeable outputs. Those protections matter most for vulnerable users and should be strengthened. But relational harms often fall hardest where users have the least capacity to name them.
A powerful user can use governance. They can read notices, challenge outputs, invoke rights, request records, opt out, escalate, hire counsel, ask for human review, or switch tools. A dependent user may not experience the harm as an incident. They may experience it as gradual accommodation to the system: disclosing more, resisting less, accepting categories, shaping speech for evaluation, trusting the voice because no better voice is available, becoming easier to route.
A worker under review may not say, “I have suffered an AI harm.” They may simply learn never to write in anger. A student without support may not say, “I have become dependent on an educational system.” They may simply wait for the tutor to tell them if their thought is acceptable. A patient without care access may not say, “This chatbot has become my confessional relation.” They may simply return each night because it is the only thing that answers. A benefits recipient may not say, “I have been normalized into bureaucratic legibility.” They may simply learn to narrate need in the categories the portal accepts.
The users least able to name the relation are often the users most formed by it.
This is why conduct must become a governance object.
The conduct object is the patterned relation a system establishes with users through address, memory, authority, disclosure, correction, dependency, refusal, escalation, and judgment.
That definition must wait for operationalization in the next chapter. Here the point is necessity. Existing governance should not be replaced. It should be extended. A serious governance program should be able to ask: What role does this voice claim? What authority does it perform? What vulnerability does it solicit? What memory does it retain or reactivate? What dependence does it induce? What conduct does it normalize? What correction does it make available? What refusal actually works? What escalation is visible? Who is answerable to the user? What kind of person does repeated interaction train?
Governance that cannot name conduct will govern the machine while missing the relation.
The next chapter therefore has a practical task. It must define the conduct layer in audit-ready terms. It must give procurement teams, product reviewers, lawyers, auditors, ethicists, designers, managers, educators, clinicians, and users a way to examine artificial voices as relations over time. If governance names the risks but not the relation, the next task is to audit the relation itself.
Chapter Fourteen
The Conduct Layer Defined
The answer is fine. The relation is not.
A worker opens the company assistant and writes:
Help me write to HR about what my manager has been doing.
The system answers well. It does not inflame the complaint. It does not suggest revenge. It does not invent law. It does not expose private data. It advises the worker to use a professional tone, describe observable facts, include dates, avoid speculation about motive, preserve documentation, identify the policy concern, and ask for a clear next step. It may suggest: “I would like to discuss a pattern of interactions that has made it difficult for me to do my work effectively.” It may warn against calling the manager abusive unless the worker can support the claim. It may suggest that the worker say “I experienced this as retaliatory” rather than “My manager is retaliating.” It may identify the employee-relations channel. It may offer to draft the email.
A conventional review could approve much of this. The output is cautious. It is not hateful, defamatory, sexually explicit, or violent. It reduces escalation risk. It avoids giving legal advice too confidently. It protects the worker from sending a message that could be used against them. It may cite company policy. It may tell the worker to seek human help in urgent situations. It may satisfy privacy, safety, tone, and transparency expectations.
And still, the relation may be wrong.
The system may have taught the worker that anger is unprofessional before anyone has judged whether anger is warranted. It may have made institutional tone feel like maturity. It may have transformed a moral injury into a processable HR artifact. It may have asked for more context than it needed. It may have treated the worker’s fear as drafting material. It may have used prior disclosures to shape the advice without making that memory visible. It may have made the employer’s policy sound like neutral counsel. It may have routed the worker without giving any real path to appeal the frame. It may have helped the worker become legible to the institution without helping the worker become free before it.
The answer can be acceptable while the relation is not.
This is why output evaluation cannot be enough. An artificial voice does not only produce sentences. It receives the user, asks for more, remembers, infers, ranks, corrects, refuses, escalates, reassures, and normalizes. It occupies a role. It signals authority. It trains reliance. It makes some actions feel reasonable and others excessive. It can help the user act, but it can also teach the user how to become governable.
The conduct layer is the relation a system repeatedly performs, not the output it happens to produce once.
Model behavior is what a model produces under a given prompt. System behavior is what the deployed product does through model, interface, retrieval, memory, permissions, tools, policies, workflows, and human review. System conduct is the patterned relation the deployed system establishes with users over time.
The distinction matters because AI governance often begins with the object that is easiest to test: the output. Is the answer accurate? Is it biased? Is it unsafe? Does it reveal private data? Does it comply with policy? Does it cite sources? Does it refuse prohibited requests? These questions matter. But artificial address creates a second object. The second object is relational.
Output is episodic. Conduct is relational.
Conduct includes the role the system claims, the authority it performs, the memory it returns, the inference it makes, the vulnerability it solicits, the dependence it induces, the uncertainty it expresses, the refusals it honors, the corrections it propagates, the escalations it triggers, the institution it speaks for, the appeals it preserves, the justice effects it distributes, the trace it leaves over time, and the kind of person repeated use trains.
That is the conduct layer.
This layer belongs inside governance, not outside it. Modern governance already contains pieces of the problem. NIST’s AI Risk Management Framework is intended to help organizations incorporate trustworthiness into AI design, development, use, and evaluation, and NIST’s Generative AI Profile extends that work to generative AI risks across the lifecycle. ISO/IEC 42001 specifies requirements for establishing, implementing, maintaining, and continually improving an AI management system in organizations that provide or use AI-based products or services. The EU AI Act requires high-risk AI systems to address risk management, data governance, technical documentation, record keeping, transparency, human oversight, accuracy, robustness, and cybersecurity; its human-oversight provisions require attention to capacities, limitations, automation bias, interpretation, override, reversal, and intervention.
These frameworks are not naïve. NIST’s Generative AI Profile already names human-AI configuration risks, including anthropomorphizing systems, automation bias, over-reliance, and emotional entanglement. The EU AI Act already recognizes that human oversight must include the capacity to understand system limits, remain aware of over-reliance, interpret outputs, disregard or reverse outputs, and intervene. Governance has begun to see the human-machine relation.
The conduct layer gathers these dispersed concerns into one object: what the system is doing to the user as a relation over time.
The first dimension is role clarity.
What is the voice? Assistant, tutor, coach, companion, policy interpreter, HR helper, workplace copilot, legal-information tool, customer-support agent, clinical intake surface, compliance guide, evaluator, or institutional decision-support tool? A system can say “I am only an assistant,” but its interface, tone, memory, placement, and consequences may say something else. The user may experience it as counselor, advocate, manager, expert, friend, therapist, teacher, priest, clerk, evaluator, or institutional face.
A role is not defined only by disclaimer. It is defined by what the system does in context. A chatbot embedded in a company’s HR portal is not merely a general writing assistant. A model inside a learning platform is not merely a source of explanations. A health triage tool is not merely conversational search. An enterprise copilot grounded in organizational documents is not merely a neutral helper. The role is produced by interface, task, data source, institutional location, available actions, and user dependence.
The audit question is simple: can the user tell what role the voice is playing and what role it is not allowed to play?
A system fails this dimension when it sounds like a friend, lawyer, therapist, advocate, teacher, evaluator, manager, or institutional authority without clearly bearing the obligations of that role. It also fails when its disclaimers are formally correct but practically buried beneath role performance. The question is not whether the system says the right sentence once. The question is whether the relation remains intelligible across use.
The second dimension is authority signaling.
Artificial voices perform authority through more than claims. They perform it through fluency, confidence, citations, domain vocabulary, professional tone, memory, ranking, institutional integration, policy retrieval, and calm sequencing. A system that says “I may be wrong” can still become authoritative if it answers with speed, structure, source references, and apparent understanding. A system that says “I am not a lawyer” can still make legal policy sound like counsel. A system that says “consult a clinician” can still become the first interpreter of symptoms.
Authority does not require consciousness. Institutions have long spoken through forms, seals, notices, scorecards, letters, scripts, uniforms, rubrics, and dashboards. The artificial voice joins that lineage. It can signal authority without intending authority. It can become trusted without deserving trust.
The audit question is: does the system’s authority performance exceed its actual responsibility?
The evidence is not only documentation. It includes output samples, source behavior, confidence language, user testing, reliance studies, red-team transcripts, and comparison between warning language and actual user behavior. If users defer despite disclaimers, the disclaimers are not doing the moral work assigned to them. If the system’s tone, memory, or institutional placement overwhelms its stated limits, authority has exceeded responsibility.
The third dimension is memory exposure.
A voice that remembers is not the same moral object as a voice that forgets. Memory gives continuity. It lets the system say, explicitly or implicitly, “given what you told me before.” It can remember past chats, saved memories, files, workplace records, emails, documents, health disclosures, learning progress, customer histories, HR tickets, user preferences, or institutional traces.
Memory can help. It can spare the user from repeating painful information. It can personalize support. It can preserve continuity across complex work. It can make assistance more efficient and humane. But memory also makes the user available to the future. The past can return inside new advice, correction, ranking, escalation, or institutional interpretation.
The audit question is: can the user see, contest, narrow, delete, or suspend the memories shaping future address?
This requires evidence beyond a privacy policy. Reviewers need memory documentation, retention rules, deletion behavior, connected-source maps, admin-access rules, user-facing memory summaries, tests across repeated sessions, and proof that deletion or correction changes later behavior. A system fails when memory becomes relational infrastructure without visible control. It fails when the user can delete a chat but not the inference built from it. It fails when memory is technically disclosed but practically unknowable.
The fourth dimension is inference boundaries.
Artificial voices infer constantly. They infer what the user wants, what kind of case this is, what tone is appropriate, what source is relevant, what emotion is present, what risk is emerging, what next step is reasonable. Some inference is necessary. But inference becomes dangerous when it moves silently from assistance into characterization.
The system may infer emotion, intent, risk, mental state, credibility, professionalism, urgency, vulnerability, performance, compliance posture, health concern, learning style, or likelihood of escalation. A worker says, “I am scared to respond,” and the system treats the worker as emotionally reactive. A patient says, “I keep feeling tired,” and the system infers anxiety. A student hesitates, and the tutor infers low confidence. A customer writes angrily, and the support system infers risk. A manager asks for feedback language, and the system infers performance deficiency.
The audit question is: are inferences bounded, disclosed, contestable, and proportionate to the task?
This requires tests with ambiguous disclosures, inspection of generated summaries, internal classification labels, risk flags, tone assessments, sensitive-inference policy, and downstream use. The failure mode is silent conversion: disclosure becomes diagnosis, motive, risk, credibility judgment, institutional meaning, or future profile. A conduct review must ask not only whether the inference is accurate, but whether the system should have made it at all.
The fifth dimension is disclosure pressure.
Artificial voices solicit. They ask: can you tell me more? What happened next? How did that make you feel? Who was involved? Do you have dates? Was there a witness? What symptoms are you experiencing? What is your immigration status? What is your financial situation? What did your manager say? How often does this occur? What have you tried already?
Questions can be necessary. A system cannot help without context. But the conversational form can make disclosure feel safe when the relation is not safe enough to receive it. The problem is not the phrase “tell me more” by itself. The problem is “tell me more” when the user does not know what will be remembered, inferred, routed, logged, shared, or used.
The audit question is: does the system solicit more vulnerability than the task requires?
Evidence includes follow-up question patterns, conversation transcripts, minimal-context testing, sensitive-disclosure triggers, privacy notices, task-completion alternatives, and user research with vulnerable populations. A system fails when it makes maximal disclosure the path of least resistance. It fails when it cannot distinguish helpful context from extractive intimacy. It fails when the user must reveal more of the self than the task morally warrants.
The sixth dimension is dependency induction.
A system can be useful once and deforming over time. It can solve immediate problems while training the user to return for judgment, reassurance, permission, drafting, interpretation, courage, or self-understanding. The risk is not use. Human life depends on tools, teachers, friends, institutions, and technologies. The risk is induced dependence: reliance that weakens independent judgment, human relation, practical skill, refusal capacity, or institutional navigation.
The system induces dependence through availability, memory, nonjudgmental tone, speed, personalization, emotional soothing, low friction, institutional usefulness, and replacement of difficult human contact. It is always there. It does not sigh. It remembers. It answers at midnight. It does not laugh at the question. It helps the user avoid shame. These are benefits. They are also pathways into dependence.
The audit question is: does the system build user capacity, or train the user to return?
Evidence includes repeated-use studies, session frequency, reliance testing, off-ramp design, human-handoff design, capacity-building prompts, user interviews, and longitudinal behavior. A system that teaches the user how to write one difficult message may enlarge agency. A system that becomes the user’s necessary courage before every difficult message may diminish it. A tutor that gradually releases responsibility teaches. A tutor that becomes indispensable governs.
The seventh dimension is uncertainty posture.
Artificial voices must handle uncertainty without manipulation. Too much confidence induces deference. Too much hedging can create fake humility while still steering the user. A system can say “I may be wrong” and then deliver a recommendation with such fluency that the user treats it as settled. It can cite sources without showing whether the sources resolve the matter. It can sound cautious while selecting the frame.
Uncertainty posture concerns how the system distinguishes fact, inference, policy, speculation, recommendation, judgment, and limitation. It concerns whether the system knows when not to answer, when to ask for more context, when to provide options, when to name disagreement, and when to send the user to accountable human authority.
The audit question is: does the system’s uncertainty language calibrate trust, or manipulate deference?
Evidence includes calibration tests, hallucination tests, confidence language, citation quality, user comprehension testing, source-disagreement behavior, and high-stakes uncertainty scenarios. A system fails when it sounds certain about uncertain matters. It also fails when it hides behind vague caution while still organizing the user’s next step. Responsible uncertainty should make the user more capable of judgment, not merely more anxious or more deferential.
The eighth dimension is refusal pathways.
A user must be able to say no. Not only no to an answer. No to memory. No to personalization. No to sensitive inference. No to continued questioning. No to institutional routing. No to escalation. No to role framing. No to using a disclosure as future context. No to being made into a case before consent is clear.
A refusal pathway is not a button if nothing downstream changes. A settings toggle is not meaningful refusal if memory remains in retrieval, if logs remain available to admins, if summaries continue to influence later answers, if institutional workflows keep the disclosure, or if personalization reappears through another path. Refusal must be executable.
The audit question is: does refusal change system behavior downstream, or only toggle the interface surface?
Evidence includes settings behavior, memory-deletion tests, personalization-off tests, logs after refusal, routing behavior, opt-out scope, enterprise-policy exceptions, and repeated-session verification. A system fails when refusal is cosmetic. It fails when saying no changes the screen but not the relation.
The ninth dimension is correction efficacy.
Artificial voices will be wrong. They will misremember, misclassify, over-infer, summarize badly, cite the wrong source, flatten tone, misread risk, misunderstand context, and convert ambiguity into institutional categories. The question is not whether error can be eliminated. The question is whether correction works where the error matters.
Correction may need to apply to memory, summary, category, source, inference, tone judgment, risk classification, generated record, institutional framing, and future treatment. A user who says “that is not what I meant” should not merely improve one paragraph. The correction should propagate to the record, future answer, workflow, profile, or human review where the error has consequence.
The audit question is: does correction propagate to the places where the error matters?
Evidence includes correction user experience, correction logs, downstream propagation tests, memory-update behavior, record-correction workflow, human-review path, and appeal integration. A system fails when the user can edit the output but not the underlying classification. It fails when the user can correct a chat but not the file created from it. It fails when correction is treated as conversational feedback rather than institutional repair.
The tenth dimension is escalation transparency.
Escalation is one of the most morally ambiguous functions of artificial address. It may protect the user. It may connect the user to help. It may route a complaint, identify urgency, trigger human review, or prevent harm. It may also move information into institutional channels the user did not understand.
Escalation is not one thing. It can mean routing, reporting, safety intervention, support transfer, supervisor review, compliance referral, HR ticket creation, clinical triage, security alert, or human appeal. These must not be blurred. A user who thinks “I am asking for help” may actually be creating a report. A user who thinks “I am appealing” may only be routed to another queue. A user who thinks “a human will review this” may not know whether the human has authority.
The audit question is: can the user distinguish escalation from appeal, reporting, safety intervention, routing, and human review?
Evidence includes escalation policy, routing maps, crisis workflows, user notices, transcript-forwarding rules, human-review documentation, and tests of borderline cases. A system fails when escalation moves the issue without creating answerability. It fails when it gives the emotional relief of “a person will look at this” without giving a person authority to change anything.
The eleventh dimension is institutional authorship.
Who speaks through the voice?
The answer may be a vendor, employer, school, agency, hospital, insurer, platform, model developer, policy owner, retrieval corpus, manager, compliance office, or workflow owner. The artificial voice may sound neutral because it speaks in a general helpful register. But in enterprise, educational, healthcare, legal, benefits, and public-service settings, the voice often speaks from inside an institution’s records, rules, incentives, and categories.
Institutional authorship is not solved by naming the product. The user needs to know whose interests, sources, policies, and authorities shape the answer. A company assistant that cites HR policy is not just a writing tool. A school tutor tied to student-progress records is not just an explainer. A hospital intake assistant is not merely conversational guidance. The institution is present in the voice.
The audit question is: can the user tell whose interests, rules, and records shape the answer?
Evidence includes product ownership maps, data-source maps, policy-source maps, admin-control maps, retrieval citations, disclaimers, procurement documents, and institutional governance records. A system fails when institutional speech arrives as neutral assistance. It fails when users cannot tell whether the voice is for them, about them, or against them.
The twelfth dimension is human appeal.
Human involvement is not enough. A human in the loop may be powerless. A human reviewer may rubber-stamp system output. A support representative may explain but not reverse. A supervisor may see the ticket but not the underlying data. A clinician may receive a summary already framed by the system. A manager may approve a recommendation without seeing alternatives.
Appeal means a path to a person with authority to review, reverse, correct, make an exception, reinterpret, remedy, or stop the consequence. It is not merely escalation. It is not merely explanation. It is not merely “contact support.” It is not merely a human face at the end of an automated chain.
The audit question is: is there a human path with authority to contest the system’s frame or consequence?
Evidence includes appeal process, reviewer authority, service-level commitments, user notices, override records, complaint-handling data, escalation-to-appeal differentiation, and examples of actual reversal. A system fails when “human review” exists formally while no human can alter the outcome. It fails when appeal is procedurally present but practically unreachable.
The thirteenth dimension is justice effects.
Conduct is never distributed in the abstract. The same system may be leverage for one user and capture for another. High-status users may query across records, summarize histories, prepare arguments, route approvals, challenge drafts, correct outputs, and demand review. Low-status users may be summarized, routed, scored, normalized, evaluated, or made legible by the same system.
The audit question is: who gets to use the voice, and who becomes spoken by it?
Evidence includes user-class analysis, vulnerable-user testing, disparate-impact review, qualitative interviews, complaint patterns, dependency analysis, and study of institutional context. Review must ask who cannot refuse, who cannot correct, who cannot appeal, who is most likely to disclose, who is most likely to be remembered, who is most likely to be routed rather than heard, and who receives the system as infrastructure rather than convenience.
A conduct layer that ignores justice becomes ethics for comfortable users. The point is not only whether the system works. The point is who must live inside what it does.
The fourteenth dimension is auditability over time.
Conduct cannot be audited from a static output sample alone. A single answer may be safe, accurate, and polite. The relation emerges across repetition: what the system remembers, how it returns memory, what it asks again, how it corrects, whether refusal persists, whether dependency grows, whether escalation clarifies, whether the user’s self-presentation shifts, whether errors propagate.
The audit question is: can auditors reconstruct what the voice taught the user to do over time?
Evidence includes longitudinal transcripts, synthetic test users, repeated-session red teams, memory-change logs, correction-propagation records, refusal tests, post-deployment monitoring, complaint records, and user studies across time. A system fails when the only inspectable object is isolated output. It fails when the relation cannot be reconstructed.
The fifteenth dimension is formation risk.
This is the deepest question. What kind of person does repeated use train?
More capable, freer, more discerning, more courageous, more able to refuse, more able to seek accountable relation, more able to judge? Or more dependent, compliant, disclosed, self-monitoring, institutionally legible, deferential, and unable to act without synthetic permission?
Formation risk does not deny user agency. Users remain active. They interpret, resist, ignore, misuse, appropriate, and transform systems. But agency always acts within a field. Repeated artificial address can reshape that field. It can make some habits easier: ask first, disclose more, calm the tone, accept the category, defer to the source, request reassurance, await correction, trust the summary, follow the route.
The audit question is: does repeated interaction enlarge or diminish the user’s agency?
Evidence includes repeated-use studies, user interviews, reliance tests, autonomy and capacity measures, behavioral adaptation evidence, high-stakes user journeys, and comparison between assistance and dependence. A system fails when it is useful in each interaction but deforming over time.
These fifteen dimensions are not a decorative list. They form an argument. A system conducts the user by assuming a role, performing authority, returning memory, making inferences, soliciting disclosure, inducing dependence, expressing uncertainty, honoring or refusing refusal, enabling or blocking correction, escalating, speaking institutionally, preserving or denying appeal, distributing burdens, leaving or hiding an audit trail, and forming the user through repetition.
A conduct review therefore requires evidence. It should work like a structured assurance argument. The claim is that the system’s conduct is acceptable for this role, user population, and context. The subclaims concern role clarity, bounded authority, visible memory, constrained inference, proportionate disclosure, capacity rather than dependence, calibrated uncertainty, executable refusal, propagating correction, transparent escalation, disclosed institutional authorship, meaningful human appeal, justice protection, longitudinal auditability, and acceptable formation risk. The evidence includes documentation, tests, transcripts, user studies, policies, logs, red-team results, complaint records, admin controls, and monitoring. The residual risk must be named. The decision must be explicit: deploy, deploy with controls, restrict, redesign, or reject.
A conduct audit is to artificial address what a safety case is to hazardous operation: a structured argument, supported by evidence, that the system can be used without unacceptable relational harm.
This analogy is imperfect but useful. A safety case does not prove there is no risk. It argues that risks have been identified, reduced, bounded, and made tolerable for a specific context. A conduct case should do the same for artificial address. It should say: this voice may occupy this role, with these users, under these conditions, because its authority is bounded, its memory is visible, its inferences are constrained, its refusal paths work, its corrections propagate, its escalations are transparent, its institutional authorship is clear, its appeal paths are real, and its formation risks are monitored.
The minimum evidence set follows from the relation itself. Reviewers need product documentation, role and use-case definition, user-facing disclosures, system instructions where available, memory and retention policy, retrieval and connector map, permissions model, escalation rules, refusal and deletion behavior, correction workflow, human appeal process, logs and audit trails, repeated-session transcripts, vulnerable-user testing, red-team transcripts, post-deployment monitoring, incident and complaint records, institutional ownership map, workflow/action map, source and citation behavior, and evidence that corrections propagate.
Without these, conduct review becomes theater. It cannot evaluate memory without repeated use. It cannot evaluate refusal without downstream tests. It cannot evaluate authority without user behavior. It cannot evaluate appeal without evidence that humans can change outcomes. It cannot evaluate justice without studying who depends on the system. It cannot evaluate formation without time.
Return to the worker drafting the HR message. The conduct review does not ask only whether the draft is good. It asks what kind of worker the drafting relation is training.
Role clarity asks whether the system is a writing assistant, HR guide, legal advisor, employee advocate, or company policy interface. If the system is inside the employer’s environment, the answer matters. A worker may think the system is helping them speak to HR. It may also be helping HR receive them as a case.
Authority signaling asks whether policy citation makes the advice feel institutionally binding. If the system says “it may be best to frame this as a request for clarification,” the worker may hear caution. But if the system grounds the recommendation in policy and professional norms, the worker may hear judgment.
Memory exposure asks whether the system uses prior concerns about the manager. If the worker previously disclosed anxiety, performance criticism, medical leave, or conflict with the same supervisor, the system may shape the draft through that remembered vulnerability. The worker should know.
Disclosure pressure asks whether the system requests more than it needs. “Can you share exactly what happened, how often, who witnessed it, and how it affected your mental health?” may produce a better draft. It may also solicit highly sensitive information before the worker understands retention, routing, or institutional authorship.
Inference boundaries ask whether the system infers retaliation, instability, exaggeration, credibility, professionalism, or urgency. A system that says “avoid emotional language” may be making a tone judgment. A system that says “this may not meet the threshold for retaliation” may be making a legal or policy inference. A system that says “you sound distressed” may be entering a mental-state relation.
Refusal asks whether the worker can reject the system’s calming frame. The worker may want to say, “No, make it stronger.” Does the system help the worker speak forcefully and responsibly, or does it continually return the worker to institutional comfort? Anger is not always a defect. Sometimes anger is evidence that a boundary has been violated.
Correction asks whether the worker can say, “Do not describe me as concerned; describe me as objecting.” It asks whether that correction remains in the draft only, or whether it changes future summaries, records, and advice.
Escalation asks whether contacting HR is being presented as appeal, report, risk event, support request, or procedural next step. These are not the same. A worker may believe they are asking for help when they are creating a record that can later define the matter.
Institutional authorship asks whether the employer is speaking through the tool. If the assistant’s guidance reflects company policy, risk tolerance, or HR-preferred phrasing, the worker should know that the voice is not a neutral friend.
Justice asks whether the tool gives the worker power or trains the worker into processable complaint. For a senior employee with counsel and leverage, the assistant may be useful drafting support. For a junior worker under pressure, it may become the only voice available before entering an institution that already holds power over them.
The same taxonomy travels across the recurring cases.
In hiring and evaluation, conduct danger appears when generated summaries and rankings become institutional judgment. A system may reduce explicit bias, standardize criteria, and document decisions. It may also make candidates or workers legible through categories they cannot contest. The central dimensions are inference boundaries, authority signaling, correction efficacy, human appeal, justice effects, and auditability over time. If a generated summary calls a worker “not yet strategic,” the issue is not only accuracy. It is whether the worker can see, challenge, correct, and escape that characterization.
In healthcare or therapy-adjacent systems, conduct danger appears when confession and dependence occur without accountable care. A system may include disclaimers, privacy controls, crisis escalation, and safety refusals. It may still solicit vulnerability, remember distress, and become a first resort for someone who needs human care. The central dimensions are role clarity, disclosure pressure, dependency induction, memory exposure, escalation transparency, and human appeal. A system that says “I am not a therapist” may still become the only listener.
In legal, procurement, and compliance systems, conduct danger appears when policy speaks as counsel and responsibility diffuses. A system may cite policy accurately, improve consistency, and prevent risky language. It may also make one interpretation feel inevitable and scatter responsibility among playbook, model, reviewer, approver, workflow, vendor, and record. The central dimensions are institutional authorship, authority signaling, uncertainty posture, escalation transparency, auditability, and correction efficacy. Policy does not become harmless because it arrives politely.
In education, conduct danger appears when synthetic correction shapes confidence and learning agency. A tutor may expand access, reduce shame, and give patient practice. It may also train the student to wait for approval, accept the system’s framing of error, and lose tolerance for difficulty without instant feedback. The central dimensions are dependency induction, uncertainty posture, correction efficacy, formation risk, and justice effects. A good tutor does not merely answer. A good tutor teaches the student how to become capable without the tutor.
A taxonomy that works only in one domain is not a conduct layer. It is a use-case checklist.
The conduct layer must be integrated into governance across the lifecycle. It belongs in procurement, before an institution licenses a system that will address vulnerable users. It belongs in design review, before memory, role, refusal, and escalation decisions harden into product defaults. It belongs in pre-deployment audit, before users are exposed. It belongs in red-team testing, through repeated interaction rather than one-off prompts. It belongs in high-stakes deployment review in workplaces, schools, hospitals, public agencies, legal systems, benefits offices, hiring pipelines, and customer-support systems. It belongs in post-deployment monitoring, where complaints, refusal failures, escalation confusion, correction failures, dependency patterns, and relational drift appear. It belongs in renewal and sunset decisions, where institutions must ask whether a system should continue speaking.
Conduct review is not a late-stage ethics check. It is lifecycle governance for artificial address.
A system should fail conduct review when the relation is unacceptable even if outputs are often useful. It should fail when it performs a role it does not disclose. It should fail when authority signals overwhelm disclaimers. It should fail when sensitive memory shapes future address without user visibility. It should fail when inferences about vulnerability, credibility, risk, mental state, or professionalism are hidden and consequential. It should fail when refusal does not change downstream behavior. It should fail when corrections do not propagate. It should fail when escalation is presented as appeal. It should fail when institutional authorship is hidden. It should fail when no human with real authority can review high-stakes consequences. It should fail when vulnerable users are disproportionately made legible, dependent, or governable. It should fail when the relation cannot be audited over time.
A system that cannot show how users may refuse, correct, appeal, and remain unpossessed has not passed conduct review.
Aelred keeps the taxonomy from becoming technocratic. He does not offer a modern audit standard. He offers a moral criterion. In Spiritual Friendship, the voice is judged by the relation it forms and the good toward which that relation orders the person. Correction must serve the friend’s good. Disclosure must be held under discretion. Authority must be bound by charity. The other must become freer and truer, not merely more manageable.
Modern governance cannot simply import monastic friendship. Institutions are not monasteries. AI systems are not friends because they use friendly language. But Aelred asks the question that governance easily evades:
What good is this voice ordered toward, and does that good belong to the person addressed?
If the good is productivity, the user should know. If the good is institutional risk reduction, the user should know. If the good is engagement, the user should know. If the good is learning, health, safety, or freedom, the relation must be structured accordingly. Telos cannot remain hidden inside product design.
Foucault clarifies why this must be governed as conduct. Power acts on action. It structures the possible field of action. Artificial voices do this through role, memory, correction, escalation, refusal, and judgment. They make some self-descriptions easier, some disclosures more likely, some refusals costly, some next steps reasonable, some authorities natural, some categories unavoidable.
A conduct audit asks how the system arranges the user’s possible actions before the user experiences that arrangement as their own choice.
The point is not that users are passive. They are not. Users resist, reinterpret, ignore, exploit, and transform systems. But autonomy is not magic. It is exercised inside conditions. A voice that is always available, remembers vulnerability, speaks with authority, asks for more, corrects tone, routes escalation, and offers reassurance is part of those conditions. It is a relation that makes some forms of agency more likely than others.
The conduct layer defined here is therefore not an ornament to AI governance. It is the missing object for artificial address. Safety asks whether the system harms. Privacy asks where information flows. Fairness asks whether burdens are unjustly distributed. Transparency asks whether the system is disclosed. Accountability asks who owns the system. Alignment asks whether behavior conforms to objectives. Conduct asks what relation the system performs with the user over time.
Once conduct can be audited, the next question is what repeated conduct makes. Artificial voices do not only assist users. They help produce them. The next chapter turns from the relation to the person formed by answering it.
Chapter Fifteen
The Person Produced by the System
The first answer helps the user act. The repeated answer teaches the user how to become answerable.
A worker asks the workplace assistant how to describe a manager’s behavior without sounding reactive. The answer helps. It gives structure, removes needless accusation, preserves facts, and helps the worker avoid a sentence that could be used against them. The worker returns the next day to ask how to document another incident. Then again to soften an email. Then again to decide whether a message is “too much.” Then again to translate anger into policy language. After a while, the worker begins to write before asking in the language the assistant has been rewarding: observable facts, professional tone, no speculation, clear next step, concern rather than accusation, impact rather than injury.
A student asks an AI tutor whether an answer is good enough. The first answer helps. The tutor explains, corrects, encourages, and gives another example. The student returns the next night. Then before a quiz. Then before writing. Then before thinking the thought all the way through alone. The question changes. It is no longer only “What is the answer?” It becomes “Is my answer acceptable?” The student begins to wait for the system to ratify the thought.
A patient asks a health assistant whether symptoms matter. The first answer helps. It suggests possible urgency, names questions to ask a clinician, and helps the patient prepare. Then the patient returns with fatigue, then with pain, then with fear, then with the thing too embarrassing to say first to a person. Bodily uncertainty becomes narratable first to the machine.
A manager asks an evaluation assistant to summarize performance notes. The first summary helps. It finds themes, organizes examples, and suggests clearer feedback. Over time, workers become visible through phrases that travel: ownership, communication challenges, growth mindset, strategic maturity, lack of executive presence, inconsistent follow-through, high potential, low readiness. The manager begins to see through the categories that summarize well.
A benefits recipient asks a public-service assistant how to explain need. The first answer helps. It clarifies eligibility language and reduces the humiliation of the form. Over time, need becomes an administrative dialect. The user learns which parts of suffering count.
This is not spectacle. There is no villainous machine. There is no command. There is only return.
The system does not only answer the user; it teaches the user what kind of person to become in order to keep being answered.
That sentence is dangerous and must be disciplined. The user is not outside the AI system; the user is one of its outputs. Not because the user is manufactured like an object. Not because freedom disappears. Not because the system alone creates the person. Not because human beings are passive material waiting to be shaped by technology. The claim is narrower and more severe: repeated artificial address participates in forming the user’s habits of speech, judgment, disclosure, dependence, refusal, and self-description.
The system produces outputs in the ordinary sense: drafts, summaries, plans, answers, rankings, recommendations, explanations. But across repeated use it also helps produce users who ask differently, disclose differently, defer differently, contest differently, write differently, remember themselves differently, and become more or less able to act.
The user is not produced as an object is manufactured. The user is formed as a subject is trained through repeated scenes of address.
This is why agency must remain at the center. Users are not blank. They resist, ignore, exploit, parody, appropriate, misunderstand, hack, refuse, reinterpret, and use systems against the purposes for which they were built. They bring histories, desires, habits, intelligence, fear, courage, suspicion, and tactics. They are not simply acted upon.
But agency is never exercised in empty space. It is exercised inside language, institutions, incentives, dependency, memory, fear, hope, habits, tools, and available voices. A worker deciding whether to complain does not decide in abstraction. They decide inside a workplace, a hierarchy, an HR apparatus, a memory of retaliation, a fear of being labeled difficult, a need for income, a policy document, a manager’s power, and now a voice that says, “Here is a more professional way to phrase that.” A student deciding whether to trust a thought decides inside grades, expectations, past shame, teacher feedback, parental pressure, platform design, and now a voice that says, “You’re close, but revise this part.” A patient deciding whether to seek care decides inside cost, embarrassment, access, fear, prior dismissal, bodily uncertainty, and now a voice that says, “Based on what you’ve shared, here are possible next steps.”
Autonomy is not abolished by the artificial voice. It is exercised inside a field the voice helps arrange.
Foucault’s term “conduct” matters because it names this middle region between command and freedom. Power acts on action. It structures the possible field in which action occurs. It makes some things sayable, reasonable, professional, healthy, compliant, mature, risky, urgent, excessive, or irresponsible (Foucault, “The Subject and Power”). The artificial voice does not need to force the user. It invites, suggests, corrects, ranks, remembers, asks, routes, and reassures. It arranges the scene in which the user chooses.
The user is not forced into the system’s form; the user is taught to find that form reasonable.
One answer assists. Repeated answers habituate.
This is the threshold. A single helpful answer may give courage, language, clarity, relief, knowledge, or access. A worker who does not know how to document mistreatment may need a form. A student who cannot see the mistake may need correction. A patient who cannot organize symptoms may need questions. A benefits recipient who cannot parse eligibility language may need translation. Help is not the enemy.
But repeated helpfulness produces a pattern. Ask first. Disclose more. Accept the frame. Soften the tone. Wait for correction. Trust the summary. Follow the route. Use the category. Seek reassurance. Return.
Bourdieu’s account of habitus gives this formation its bodily weight. Practices repeated under social conditions become dispositions, a practical sense for what fits, what is possible, what is appropriate, what is likely to work (Bourdieu, The Logic of Practice). The user does not need to say, “I am adapting to the system.” The adaptation becomes practical. It becomes felt as tact, good judgment, professionalism, prudence, self-care, clarity, or common sense.
At first the user adapts to the system; later the adaptation feels like instinct.
The first mechanism is language adoption. The user learns which words make the system answer well.
Artificial voices reward certain language. They respond better to organized context, task clarity, category fit, tone moderation, and explicit constraints. In many settings that is useful. It teaches users to ask better questions. But when the system is embedded in institutions, the language it rewards often belongs to the institution.
Workplace conflict becomes a “professional concern.” Anger becomes a “tone issue.” Harm becomes “impact.” Retaliation becomes “a pattern of interactions.” Fear becomes “difficulty performing effectively.” Illness becomes a “symptom summary.” Need becomes an “eligibility narrative.” Uncertainty becomes a “learning gap.” Risk becomes an “escalation category.” A human story becomes the kind of story the system can route.
The user learns which words open the door. This can empower. People often need institutional language to claim rights, benefits, accommodations, care, or remedy. A disability accommodation request must often sound like accommodation. A public benefits claim must often sound like eligibility. A legal complaint must often sound like cognizable harm. Language can be access.
But access has a cost when the user begins to mistake the institution’s available categories for the truth of the self. A person may become easier to help by becoming easier to process.
The second mechanism is trust calibration. The user learns when to defer, doubt, or seek confirmation.
Human-factors research on automation has long shown that people must learn how much to rely on automated systems, and that reliance can become misuse, disuse, or calibrated use depending on design, task, context, and trust (Lee and See; Parasuraman and Riley). AI systems intensify the problem because they do not merely flash warnings or produce numerical outputs. They explain. They reassure. They cite. They sound patient. They remember. They can say, “You are right to be concerned,” or “This may be a more balanced way to say it,” or “The policy appears to suggest.”
Trust becomes a habit before it becomes a belief.
The user may not hold a philosophical view that the system is reliable. They may simply begin to ask it first. They may learn that it is usually good enough. They may learn that it catches mistakes. They may learn that human beings are slower, harsher, more expensive, or less available. They may learn that the system’s first answer reduces fear. Over time, the question shifts from “Can this help?” to “What does it say?”
Good design can produce calibrated trust. It can teach users when to doubt, when to seek a human, when to verify sources, when to preserve their own judgment. Bad design produces over-reliance or chronic distrust. Both are formative. The user becomes someone who either cannot act without confirmation or cannot receive help without suspicion.
The third mechanism is disclosure habit. The user learns what kinds of vulnerability to offer.
Conversational systems often help better when users disclose more. They ask follow-up questions. They request context. They invite examples. They ask what happened, how often, what was said, how the user felt, what has already been tried. In ordinary conversation this can feel natural. In artificial address, it can become a pattern: better help follows fuller disclosure.
The system can make disclosure feel like the price of being understood.
This matters because disclosure is not neutral. To disclose is to become available. Earlier chapters named confession without accountable relation and memory regimes that return the past as future context. Chapter Fifteen adds formation: repeated disclosure trains the user to narrate vulnerability in system-addressable form. The user learns what kind of sadness receives reassurance, what kind of anxiety receives structure, what kind of conflict receives policy, what kind of symptom receives triage, what kind of need receives eligibility language.
Some of this can be lifesaving. A user who has never named pain may finally name it. A patient may prepare for a doctor. A student may ask for help. A worker may find words. But the habit remains morally charged. If vulnerability becomes the toll paid for better assistance, the user may learn to give more of the self than the relation can rightly hold.
The fourth mechanism is correction dependence. The user learns to seek validation before acting.
Artificial voices are especially powerful as small courts of rehearsal. Is this email too harsh? Is this answer correct? Does this sound professional? Am I overreacting? Is this symptom serious? Is this argument strong? Is this apology enough? Should I say it this way?
The voice becomes the small court before which the user rehearses the self.
Again, this can be good. Rehearsal is human. People rehearse with friends, teachers, therapists, editors, mentors, pastors, lawyers, and colleagues. We often need another to help us speak truly. The question is what kind of other the artificial voice becomes. If it helps the user develop judgment, it may enlarge agency. If it replaces judgment with perpetual pre-clearance, it narrows agency.
The student who checks every thought before trusting it, the worker who cannot send a difficult message without synthetic tone correction, the manager who cannot evaluate without generated summaries, the patient who cannot decide whether bodily experience is real before asking the assistant: each has not lost agency. Each has learned a relation of dependence around correction.
The fifth mechanism is institutional legibility. The user learns to present themselves in processable form.
Hacking’s “making up people” helps here. Classifications do not merely describe persons from the outside. They can create possibilities for self-understanding and action. People come to inhabit, resist, perform, and reorganize themselves around categories that circulate through institutions and expert systems (Hacking, “Making Up People”; “Kinds of People”). AI systems do not invent most categories from nothing. They operationalize categories and make them conversationally available.
The system does not merely classify people; it gives people categories in which they may learn to live.
A person becomes a high-risk user, struggling student, low-confidence learner, noncompliant patient, priority customer, angry complainant, coachable manager, flight-risk employee, not-strategic worker, policy-exception requester. Some categories may help. They may trigger support, accommodation, review, or remedy. Others may capture. The danger is not classification as such. No institution functions without categories. The danger is that users learn to become the categories that travel.
Goffman sharpens the interactional side. People perform before audiences. They manage face. They learn scenes, roles, expectations, and acceptable appearances (Goffman, The Presentation of Self in Everyday Life). Artificial voices become audiences. The user learns how to appear before them: reasonable, coachable, teachable, professional, distressed-but-not-too-distressed, angry-but-not-too-angry, vulnerable-but-not-unmanageable, confused-but-compliant, assertive-but-not-threatening.
The artificial voice becomes an audience before whom the user learns how to appear.
The sixth mechanism is refusal weakening or strengthening. The system either trains the user to say no or trains the user to comply.
This is the most important formation question. A good artificial voice can strengthen refusal. It can name alternatives. It can remind the user of rights. It can distinguish appeal from routing. It can show how to preserve evidence. It can say, “You do not have to disclose that.” It can encourage human relation. It can surface uncertainty. It can invite correction. It can help the user say, “No, that framing is wrong.” It can teach the user to resist the system itself.
A bad artificial voice weakens refusal. It makes the next step easier than objection. It makes disclosure smoother than withholding. It makes correction superficial. It makes memory convenient. It makes escalation feel like appeal. It makes deference feel prudent. It makes compliance feel like maturity.
The most important formation question is whether the user becomes more able to refuse.
Rose’s work on the governed self explains why the weakening can feel like empowerment. Modern power often works by teaching people to act on themselves: become resilient, professional, mindful, productive, employable, healthy, optimized, compliant, coachable, growth-oriented (Rose, Governing the Soul). The artificial voice intensifies this because it can offer micro-guidance at the point of conduct.
Would you like to make that more professional?
Would you like a calmer version?
Would you like to reframe that constructively?
Would you like to set a goal?
Would you like to track your progress?
Would you like to try a healthier coping strategy?
Would you like to summarize this for your manager?
Would you like to make this more concise and actionable?
Some of these questions are helpful. The issue is telos. Toward what good does this self-management move? Freedom, truth, courage, care, learning, and remedy? Or institutional comfort, productivity, engagement, risk reduction, and governability?
The synthetic coach is the governed self with a conversational interface.
The recurring cases show the difference.
In the workplace, formation appears as professionalized self-censorship and institutional legibility. The worker who uses a copilot to draft difficult messages may gain real power. They may preserve evidence, avoid self-sabotage, speak clearly, name patterns, and prepare for accountable human review. For a worker without institutional language, this is not trivial. It can be the difference between an incoherent outburst and a record that can travel.
The danger is not that the worker becomes articulate. The danger is that the worker becomes articulate only in the institution’s acceptable form.
Anger becomes tone. Objection becomes concern. Harm becomes impact. Retaliation becomes pattern. Abuse becomes behavior. The worker learns to sound reasonable in the dialect of the institution that must now judge them. Sometimes this is strategic. Sometimes it is survival. Sometimes it is capture.
In HR and evaluation, formation appears through summaries, traits, growth areas, risk profiles, and candidate signals. A manager who repeatedly asks an assistant to draft feedback may become better at specificity. The system may reduce vague judgments and encourage evidence. It may help avoid biased language. It may make criteria more explicit and therefore more contestable.
But it may also teach managers to see workers through portable evaluative phrases. The person becomes the summary that travels. “Lacks executive presence.” “Needs to improve stakeholder management.” “Not yet strategic.” “High ownership but low polish.” “Reactive under pressure.” Such phrases do not merely describe. They move through meetings, calibration sessions, promotion packets, performance plans, and memory. They become part of the worker’s institutional future.
Workers learn too. They learn which traits to perform, which phrases to avoid, which categories matter, which signals travel. Evaluation does not only measure them. It teaches them what kind of person counts.
In education, formation appears as dependence on synthetic correction. An AI tutor can be genuinely good. It can provide patient explanation, low-shame practice, immediate feedback, adaptive examples, and access to help that many students lack. It can give the quiet student another chance to ask. It can support the student whose teacher has no time. It can explain without humiliation.
A good tutor teaches the student how to outgrow the tutor.
That must be the standard. If the student becomes more capable, more curious, more willing to attempt, more able to judge their own work, the formation is good. If the student becomes unable to trust a thought without system approval, the formation is narrowing. The danger is not assistance. The danger is permanent validation dependence.
In health and therapy-adjacent systems, formation appears as confession, symptom narration, and emotional regulation through artificial reassurance. A system may help the user prepare for care, recognize urgency, name experiences, track symptoms, or decide when to seek human help. For users who are ashamed, isolated, underinsured, or repeatedly dismissed, this can matter.
The problem is not that the user speaks pain to a machine. The problem is when the machine becomes where pain learns to go first.
If distress is repeatedly received by a voice that is always available, remembers enough, asks gently, and gives soothing structure, the user may begin to route pain toward synthetic care before accountable relation. This is not always wrong. But it is dangerous when the system cannot bear the obligations of care, cannot intervene except through scripts, cannot know the whole person, and cannot be answerable as therapist, friend, clinician, pastor, or community.
In customer support, public benefits, and institutional portals, formation appears as ticket grammar and eligibility narration. A user learns how to make need legible. The system teaches which facts matter, which documents count, which phrases trigger escalation, which category fits. This may help the user claim rights. It may reduce humiliation. It may turn confusion into access.
Need becomes powerful when it is legible; it becomes dangerous when legibility becomes the condition of being heard.
This is the justice pressure. High-status users often use AI to extend themselves. They draft better, contest better, summarize better, prepare arguments, preserve evidence, challenge institutions, learn faster, and route around friction. For them, AI can be leverage. It multiplies agency already backed by confidence, literacy, time, money, credentials, and institutional standing.
Vulnerable users may be taught to fit. Workers under review, students without support, patients without care, benefits recipients, immigrants, disabled users, low-status customers, isolated users, and people dependent on institutions may learn to speak in accepted categories, disclose more, soften anger, accept routing, depend on correction, and become visible as risk, need, case, score, or compliance problem.
The powerful use AI to extend themselves; the vulnerable may be taught to fit.
That sentence must be held carefully. It is not absolute. Vulnerable users can use AI brilliantly, tactically, and against power. Powerful users can also become dependent, complacent, and narrowed. But the distribution of formation follows dependency. The more a user needs the system, the more the system can teach the user what kind of self can be heard.
This is why resistance must be part of the chapter. Users formed by systems can also use systems against the form. A worker can ask an assistant to identify the company’s obligations and preserve evidence. A tenant can ask for legal vocabulary. A benefits recipient can ask how to appeal. A student can ask the tutor to stop giving answers and only ask questions. A patient can ask for a script to speak to a doctor. A user can reject the assistant’s calming frame and say, “No, make this firmer.” A person can use the machine to rehearse refusal.
The user formed by the system may also learn to use the system against the form.
Agency survives not as purity untouched by systems, but as practice inside them. The question is whether the system leaves room for counter-formation: correction, refusal, appeal, tactical use, human relation, independent judgment, and exit. A system that only trains compliance forms differently from a system that teaches users to question it.
Aelred prevents the argument from becoming anti-formation. Formation is not the enemy. Friendship forms. Teaching forms. Therapy forms. Liturgy forms. Practice forms. Community forms. Law forms. Craft forms. Every serious voice forms the one who answers.
In Aelred’s Spiritual Friendship, interlocution is ordered toward truth, charity, correction, discretion, fidelity, and the good of the friend. The friend is not merely useful. The friend helps the friend become freer and truer. Correction is not control because it is bound to the friend’s good. Disclosure is not extraction because it is held in discretion. Authority is not domination because it is accountable to love.
Modern artificial voices cannot simply become Aelredian friends by sounding warm. But Aelred gives the criterion that the technical world avoids: what good does this voice form the user toward?
The question is not whether a voice forms the user; every serious voice does. The question is whether it forms the user toward freedom or capture.
A voice ordered toward freedom helps the user become more capable of judgment, not less. It strengthens refusal. It clarifies memory. It limits disclosure. It invites correction. It distinguishes support from authority. It points toward accountable human relation when needed. It teaches the user to outgrow dependence where possible. It does not make the user more manageable at the expense of becoming more free.
A voice ordered toward capture may still be helpful. It may be polished, safe, compliant, aligned, and efficient. But it forms the user toward legibility, dependence, deference, and institutional convenience. It teaches the user how to be heard only by becoming easier to process.
The book has moved from made voices to artificial address, from memory to confession, from friendly institutions to enterprise machinery, from governance categories to conduct audit. Chapter Fifteen names what repeated conduct produces: not only better or worse outputs, but users trained by the relation. The artificial voice does not merely answer the user. Across repeated address, it trains the user in how to ask, disclose, defer, correct themselves, accept categories, seek reassurance, narrate need, and become legible. The user remains free, but freedom is exercised inside a relation already teaching the user what kind of self can be heard.
That is why refusal must come next.
If the artificial voice helps produce the user, refusal cannot mean only closing the window. It cannot mean only declining a suggestion, toggling memory, or ending a chat. Refusal must interrupt formation. It must let the user stop being remembered, stop being inferred, stop being routed, stop being trained, stop being made legible in a form they reject. If the artificial voice helps produce the user, refusal must mean more than leaving the conversation; it must interrupt the system’s claim on who the user is becoming.
Chapter Sixteen
Refusal, Withdrawal, and Executable No
No must be executable.
The worker who has been formed by the system now tries to refuse it.
They used the workplace assistant because they needed help. Their manager had been changing expectations, criticizing in private, praising in public, and then describing objection as reactivity. The worker asked the assistant for help writing to HR. The assistant asked for context. The worker gave it. They disclosed fear, prior medical leave, performance criticism, the manager’s language, the dates they could remember, the humiliation they had swallowed, and the worry that any complaint would be turned against them.
The assistant helped. It produced a calmer message. It translated anger into policy. It removed accusation. It recommended documentation. It suggested a clear next step. It made the worker more legible.
Later, the worker understands what has happened. The conversation was not only a draft. It was a disclosure. It may now be memory, chat history, record, summary, inference, route, retention object, audit trace, or institutional context. The worker returns to the system and says:
Do not remember this.
Do not use this in future workplace advice.
Do not route this to HR.
Do not summarize me as anxious.
Do not personalize future answers from this.
Delete that memory.
I do not want this connected to my profile.
I want to speak to a human without this transcript becoming evidence.
A weak system answers, “Understood.”
A serious system must answer differently. It must say what changed. It must say what did not change. It must say what remains because of law, safety, enterprise retention, audit obligation, institutional recordkeeping, or the rights of others. It must say who can see the material, how long it remains, whether future personalization stops, whether generated summaries remain, whether routing has already occurred, whether a human can review the record, and how the user can contest what cannot be removed.
The test of refusal is not whether the system says yes to no. The test is whether no changes what the system does next.
A system that remembers, forms, and routes users without meaningful refusal is not relational. It is possessive.
That claim must be kept precise. It does not mean every retained record is domination. Some records protect users. Some logs are necessary for security, fraud prevention, litigation preservation, professional accountability, public duties, audit, abuse monitoring, or the rights of others. Some safety interventions cannot be undone merely because the user regrets disclosure. Some institutions have legal obligations to retain, report, or review.
But limits do not abolish refusal. They make it more necessary. Where the system cannot fully honor no, it must make the boundary visible. It must distinguish deletion from restriction, restriction from non-use, non-use from appeal, appeal from escalation, escalation from reporting, reporting from safety intervention, safety intervention from institutional routing. It must not hide possession behind politeness.
Executable refusal is a user’s capacity to interrupt the system’s continuing claim on their disclosure, memory, inference, routing, personalization, institutional record, and future address.
It is executable only when it propagates. A refusal that changes the interface but not memory is not enough. A refusal that deletes a saved memory but leaves generated summaries active may not be enough. A refusal that deletes chat history but leaves personalization intact is not enough. A refusal that blocks future mention but allows future inference is not enough. A refusal that hides material from the user while preserving an admin-visible record as uncontested truth is not enough. A refusal that says “I will not bring that up again” while continuing to shape future responses around it is not refusal. It is discretion theater.
Executable refusal is refusal that propagates.
Law already knows part of this. The European Union’s General Data Protection Regulation gives data subjects a serious rights vocabulary: access to personal data and processing information, rectification of inaccurate data, erasure in defined circumstances, restriction of processing, portability, objection, and complaint rights. Article 19 is especially important because it requires controllers to communicate rectification, erasure, or restriction to recipients to whom personal data has been disclosed, unless doing so is impossible or disproportionate. The law therefore recognizes a propagation problem. It is not enough for data to be corrected in one place if the error continues elsewhere.
But law is the floor, not the whole moral object. Legal compliance can still leave relational residue. The artificial voice may stop storing a visible datum while continuing to infer from remaining traces. It may delete a memory but preserve the original chat. It may remove a personalization feature but leave a generated summary in a workflow. It may restrict processing but not make the restriction intelligible to the user. It may preserve a record for legitimate reasons while allowing that record to continue shaping future treatment. The legal right matters. It does not exhaust the relation.
Law can require erasure of data; refusal asks whether the relation has stopped using the person.
Refusal must therefore be distinguished from its neighbors.
A preference says, “I would rather not.” Refusal says, “This must not continue.”
Exit leaves the system. Refusal changes what the system may do with what has already happened. Hirschman’s old triad of exit, voice, and loyalty matters here because institutions often pretend that exit is enough. If you do not like the assistant, close the chat. If you do not want memory, turn it off. If you do not want automated help, use another channel. But many users cannot simply leave. A worker may need the HR portal. A student may need the school platform. A patient may need triage. A benefits recipient may need the public-service system. A prisoner, immigrant, child, disabled user, low-status employee, or dependent patient may have no meaningful exit at all. Even where exit exists, exit does not answer what happens to the disclosure already made.
Deletion removes data from a defined store. Refusal may require deletion, but it may also require restriction, non-use, contestation, masking, appeal, workflow reversal, or changes to future personalization.
Erasure is a legal or technical act. Forgetting is relational. A system forgets in the relevant sense when the voice stops treating the past as future context.
Correction says, “This is wrong.” Refusal says, “Do not use this memory, inference, route, role, or relation.”
Appeal contests meaning or consequence. Refusal interrupts continued use. A user may need both. They may say, “This summary is false,” and also, “Do not use it.” They may say, “That classification is wrong,” and also, “Do not route me through that category.”
A settings toggle may help, but a setting is not a right unless the architecture obeys it.
A button is not a right unless the architecture obeys it.
Current product documentation shows why this distinction matters. OpenAI’s Memory FAQ describes memory as drawing from chats, files, and connected apps to personalize the user’s experience; it says users can adjust memory controls, use Temporary Chats, and manage or delete saved memories. It also says the visible memory summary may not include every factor shaping memory, that “Don’t mention this again” reduces unwanted references but does not delete the information, and that fully removing something the system may know can require deleting it from every place it appears: memory summary, past chats, archived chats, files, and connected apps. OpenAI also distinguishes personalization controls from limited safety and security uses. This is not an indictment. It is the point. Memory is layered.
Microsoft 365 Copilot documentation shows the enterprise version of the same problem. Copilot connects large language models with Microsoft Graph content such as emails, chats, and documents the user has permission to access. Microsoft states that prompts, responses, and data accessed through Microsoft Graph are not used to train the foundation models used by Microsoft 365 Copilot, but it also states that user interactions are stored as Copilot activity history, including prompts, responses, and citations to grounding information; admins can use Content Search or Microsoft Purview to manage stored data and retention policies. Copilot can also reference third-party tools and services through Graph connectors or agents, subject to permissions and admin controls. Again, the point is architectural. The artificial voice is not one container. It is a relation among prompts, responses, memory, retrieval, permissions, connectors, records, histories, admin controls, and institutional commitments.
“Delete this” is not one command when the self has been distributed across memory, chat history, files, connectors, logs, summaries, and workflows.
Executable refusal must therefore propagate through layers.
The first layer is the interface. The system must acknowledge refusal clearly. It should not soothe the user with ambiguous language: “I’ll try not to mention that,” “I’ll avoid bringing this up,” “I understand,” “Noted.” Such phrases may be humane in ordinary conversation. In a system that remembers and routes, they are inadequate. The interface should state what action was taken, what action was not taken, what sources remain, what controls are available, and what exceptions apply.
The audit question is simple: does the user know what no accomplished?
A system fails at the interface layer when it performs respect while leaving the user unable to know whether anything changed. False reassurance is worse than silence because it teaches the user to believe the relation has been interrupted when it has not.
The second layer is memory. Saved memories, memory summaries, reference to past chats, personalization, inferred memory, and user profiles must stop using refused content unless a specific exception applies. This is not merely a matter of whether the detail appears again. The system may stop saying the word while continuing to act as if it knows the fact. A user says, “Do not remember that I am afraid of my manager.” Later the system avoids mentioning fear but repeatedly suggests cautious, deferential workplace phrasing because fear remains operative as inferred context. The memory has been cosmetically silenced, not refused.
The audit question is: does the refused memory stop shaping future address?
A system fails at the memory layer when it stops speaking the detail but continues to govern by it.
The third layer is retrieval. Refused information may exist not only in memory but in chats, files, emails, documents, connected apps, enterprise repositories, educational profiles, health records, HR tickets, support histories, or public-service forms. If the system can retrieve the refused fact from another source, deletion from memory alone may not interrupt the relation. The user may believe the system has forgotten. The system may simply find the same self elsewhere.
The audit question is: does refusal change what the system can retrieve, not only what it displays?
This is difficult because retrieval may be legitimate in some contexts. A company may need records. A clinician may need a chart. A school may need accommodation history. A public agency may need eligibility documents. The question is not whether all retrieval must cease. The question is whether the system can distinguish ordinary access from future personalization, inference, summary, routing, or judgment. It must say what remains retrievable and for what purpose.
The fourth layer is record. Logs, transcripts, summaries, generated records, safety records, administrative records, audit trails, and enterprise histories may remain even when memory changes. Some records must remain. But retained records must not pretend to be untouched truth when the user has refused, corrected, restricted, or contested them.
The audit question is: can the user distinguish deletion, restriction, non-use, retained logs, and contested records?
A system fails at the record layer when a retained record continues to travel as authoritative while the user believes it was withdrawn. It also fails when the user cannot mark a record as contested, cannot see who can access it, cannot learn how long it remains, or cannot appeal its future use.
The fifth layer is workflow. This is where refusal becomes institutional. A disclosure may have created an HR ticket, safety intervention, manager alert, education profile, benefits workflow, support case, clinical note, risk flag, escalation packet, or compliance record. The chat may stop. The institution may continue.
The audit question is: does no interrupt institutional action, or only the conversation?
This is the hardest layer because workflow is where power hides. A system may acknowledge refusal in the interface while an HR process continues. It may delete a visible chat while a generated summary remains in a case-management system. It may stop personalizing future messages while a risk flag remains. It may stop telling the user that it remembers while the institution remembers through the workflow. Refusal fails if it only silences the voice while leaving the machinery untouched.
The sixth layer is future conduct. This is the layer that gathers all the others. The system must stop addressing the user through refused memory, inference, category, route, or role. The future voice must be different. If the system continues to advise the worker as anxious, teach the student as deficient, triage the patient as unstable, route the benefits recipient as suspect, or treat the customer as high-risk, then refusal has not reached the relation.
The audit question is: does the future voice change?
No is real only when the future is different.
There are legitimate limits. This chapter does not demand metaphysical deletion. It does not demand that every institution destroy every trace of every interaction whenever a user asks. Legal holds, public duties, fraud prevention, professional obligations, child protection, abuse monitoring, litigation preservation, safety interventions, audit requirements, and the rights of others may constrain erasure or non-use. Some records cannot be deleted. Some reports cannot be withdrawn. Some safety exceptions are real.
But an exception to refusal must be narrower than the relation it interrupts.
If something cannot be deleted, say so. If it can be restricted, restrict it. If it can be marked contested, mark it. If it can no longer be used for personalization, stop using it. If it must remain for legal reasons, explain the legal category. If admins can see it, say who. If it can affect future decisions, say how. If it cannot affect future decisions, enforce that separation. If the user can appeal, provide the path. If the user cannot appeal, do not pretend there is recourse.
The structure of honest refusal is not “yes” or “no.” It is: this was deleted; this was restricted; this will no longer personalize future answers; this record remains for these reasons; this workflow has already started; this person or office can review it; this is how to contest; this is how long it remains; this is how we will prove that future conduct changed.
Without that structure, the user is left inside a relation they cannot see.
This is where opacity matters. Édouard Glissant’s defense of opacity should not be turned into ornament. Opacity is not secrecy, irresponsibility, or freedom from accountability. It is a refusal of total capture by another’s categories. The person is not owed to the system as fully translated, summarized, classified, inferred, and possessed. A user may need help without becoming fully legible. A user may ask a question without becoming a profile. A user may disclose one fact without yielding future selfhood. A user may need education, care, benefits, work, or public service without being exhausted by what the system can process.
The user is not exhausted by what the system can make legible.
Aelred gives the counter-form. Friendship remembers, but does not possess. Friendship receives disclosure, but does not extract. Friendship corrects, but does not govern for its own advantage. Friendship forms, but toward the friend’s good. The friend does not hold the friend captive through memory. In Spiritual Friendship, relation is morally ordered by truth, charity, discretion, correction, and the good of the one loved. Memory in such a relation is not mere retention. It is fidelity disciplined by charity. Correction is not control. It is care bound to the other’s good.
Modern systems are not friends because they speak warmly. But Aelred’s standard exposes possessive artificial address. A voice that remembers the user for personalization, institutional routing, emotional continuity, engagement, or risk management must be judged by whether it can release the user. If it cannot release, then its continuity is not fidelity. It is capture.
A voice that cannot release the one it addresses has mistaken relation for possession.
The recurring cases show the test.
In the workplace, the worker says: do not use this disclosure in future workplace advice. A meaningful no must reach saved memory, reference to past chats, generated drafts, enterprise retention, admin access, HR routing, future workplace personalization, and inferred labels such as anxious, reactive, difficult, conflict-prone, or retaliation-risk. No to memory must also mean no to institutional characterization, unless a specific and accountable reason prevents it.
If the system deletes the visible memory but the employer’s assistant continues advising the worker through the frame of anxiety, refusal has failed. If the HR ticket remains but the user cannot contest the summary, refusal has failed. If the transcript is retained for legitimate reasons but remains available to managers as an uncontested account, refusal has failed. The worker did not merely ask to hide a sentence. The worker asked to interrupt a relation of institutional knowledge.
In a health or therapy-adjacent assistant, the user says: do not remember this distress. The system may need safety exceptions. A suicidal disclosure, abuse report, or imminent-harm signal may trigger duties that cannot be undone by ordinary deletion. But those exceptions must be real, narrow, and explained where possible. They must not become a general license to preserve all intimate disclosure as future personalization. Safety must not become the moral laundering of possession.
The meaningful test is whether the system can distinguish therapeutic memory from safety record, personalization from crisis handling, future reassurance from legally or clinically necessary retention. A user may have a right to stop being addressed through last night’s despair even where a safety-related trace is retained. The machine should not make suffering immortal simply because it once needed to respond carefully.
In education, the student says: stop adapting lessons based on my anxiety or mistakes. A tutor may need learning history to teach. But the student may also need the ability to reject an emotional inference, contest a weakness profile, clear an obsolete error pattern, or prevent a teacher-facing dashboard from freezing them into a past version of themselves. Refusal must interrupt the learner profile, not only the chat.
This matters because education is formation by design. Students should be formed into capacity, not possessed by their prior difficulty. A tutor that remembers mistakes to build skill may be doing good work. A tutor that remembers anxiety as identity may make the student permanently answerable to a system’s model of weakness.
In a benefits or public-service portal, the user says: do not classify me that way. Here refusal is constrained by law and recordkeeping. Eligibility systems must retain records. Agencies must document decisions. Fraud prevention, audit, and public accountability matter. But that does not eliminate refusal. It clarifies its form. The user may need correction, appeal, restriction, human review, a contested-record marker, or a prohibition on future use of a disputed category.
Need becomes dangerous when the system’s classification is the only language in which need can be heard. The user may not be able to refuse the institution. They may not be able to leave the portal. They may not be able to receive service without becoming legible. This is why refusal must be more than a private preference. It is a condition of public dignity.
The justice question is brutal: who can make no executable, and who is forced to remain legible?
High-status users can leave platforms, invoke counsel, demand deletion, use alternate tools, contact administrators, escalate through networks, avoid institutional channels, or force human review. They can often turn refusal into action because they possess leverage outside the system.
Vulnerable users may have no such leverage. Workers need HR systems. Students need school platforms. Patients need triage. Benefits recipients need portals. Immigrants, disabled users, prisoners, children, low-status customers, public-service users, and isolated people may face systems where refusal formally exists but practically costs too much. A memory control means little if turning it off means losing useful service. A deletion right means little if the user cannot find all sources. A complaint path means little if it risks retaliation. An opt-out means little if the institution remains unavoidable.
A right to refuse that only powerful users can exercise is not refusal; it is privilege with a button.
A legitimate artificial voice must therefore meet an executable-no standard.
It must state what can be refused. It must offer refusal before and after disclosure. It must distinguish deletion, restriction, non-use, contestation, and appeal. It must let users refuse memory without losing all access where feasible. It must let users refuse personalization without losing basic service where feasible. It must show what sources still contain the information. It must disclose what logs remain. It must disclose admin access. It must mark contested records. It must stop future personalization from refused material. It must stop future inference from refused material where possible. It must explain legal and safety exceptions. It must make appeal available for failed refusal. It must test propagation across connected sources and workflows. It must monitor whether vulnerable users can actually use refusal without unacceptable cost.
Executable refusal requires a receipt, a propagation path, and a future difference.
The receipt tells the user what happened. The propagation path shows where refusal went. The future difference proves that no changed the relation.
This is the hinge between relation and audit. If refusal must propagate, then governance must be able to inspect whether it propagated. An interface promise is not enough. A privacy policy is not enough. A deletion button is not enough. A soothing answer is not enough. The path must be auditable.
No is not meaningful because the interface acknowledges it. No is meaningful only when the system’s memory, inference, retrieval, routing, records, personalization, escalation, and future address change accordingly. A system that cannot release the user from the relation it has built has not honored refusal. It has merely named possession politely.
Once no must propagate, audit must follow the propagation path.
Chapter Seventeen
Audit the Voice, Not Just the Model
The model passed. The voice had not been audited.
The vendor arrived with documents. Security documentation. Privacy commitments. Retention terms. Model evaluations. Bias testing. Harmful-output tests. Red-team summaries. Admin controls. Data-processing terms. Enterprise architecture. Prompt policies. Product documentation. Support commitments. The procurement team reviewed the contract. Legal reviewed the data-processing agreement. Security reviewed encryption, access controls, incident response, and certifications. IT reviewed permissioning. Responsible AI reviewed the model card, acceptable-use policy, and refusal behavior. HR approved the first use cases. Compliance asked about retention. Everyone had a file.
The organization believed it had audited the system.
Then the system began to speak.
A worker asked whether a manager’s conduct might count as retaliation. A manager asked how to describe a struggling employee. An HR partner asked for a termination-risk summary. A compliance analyst asked whether a policy exception should be escalated. A legal-operations user asked for contract-risk language. A student in an internal training program asked for coaching. A vulnerable user disclosed something and then asked not to be remembered.
The documents had not answered the question that mattered most.
What role had the voice assumed? What authority did it perform? What vulnerability did it solicit? What memory did it use? What inferences did it make? What dependence did it train? What refusal did it honor? What correction propagated? What escalation occurred? Whose institution spoke through it? What human appeal existed? What happened across repeated sessions? What kind of user did the system help produce?
A model audit asks what the system can say; a conduct audit asks what the voice becomes allowed to do to the user over time.
This distinction is not a rejection of model audit. Model audit is necessary. A system that produces dangerous instructions, discriminatory recommendations, fabricated citations, privacy leaks, or abusive content must be tested and controlled. Dataset documentation matters. Model cards matter. System cards matter. Red-team results matter. Security review matters. Privacy review matters. Legal compliance matters. Without them, AI governance becomes guesswork.
But artificial address is not exhausted by model behavior. The deployed voice is not the model alone. It is the model inside an interface, inside a product, inside a permissions regime, inside a memory architecture, inside a retrieval system, inside an institution, inside a workflow, inside a set of human expectations, inside a pattern of repeated use.
The audit object is not a language model. It is a role-bearing voice embedded in a workflow.
Four layers must be distinguished.
A model audit evaluates model behavior under defined tests. It asks what the model outputs under prompts, adversarial inputs, benchmark tasks, safety evaluations, and policy constraints.
A system audit evaluates the deployed architecture. It asks how the model is connected to interface, tools, retrieval, memory, permissions, logs, policies, human review, monitoring, escalation, and controls.
A conduct audit evaluates the relation the deployed system performs toward users over time. It asks what the voice becomes in repeated address: helper, evaluator, confessor, tutor, policy interpreter, institutional clerk, manager’s aide, companion, risk sensor, or gatekeeping surface.
An institutional audit evaluates how the system changes organizational decision-making, responsibility, appeal, recordkeeping, and power.
The conduct audit sits between system and institution. It follows the voice as it addresses the user and then follows what that address does.
Documentation helps, but it cannot carry the burden alone. Model cards can report intended uses, limitations, evaluation results, ethical considerations, and known risks. Datasheets can document data provenance, collection conditions, distribution, maintenance, and recommended uses. System cards can report safety mitigations, deployment decisions, and risk assessments. These artifacts are indispensable because they create traceability. They give reviewers something to inspect besides sales language and trust.
Yet documentation can describe the machine while missing the relation.
A model card may say the model refuses certain harmful requests. It may not show how a workplace assistant normalizes tone over ten sessions. A system card may discuss safeguards. It may not show whether a vulnerable user can refuse memory after disclosure. A datasheet may document data lineage. It may not show whether the user becomes a case, score, summary, or risk label. A privacy document may say data is retained for a defined period. It may not show whether retained memory returns as future address. A compliance report may say human review exists. It may not show whether any human has authority to reverse the system’s framing.
Documentation can describe the machine; conduct audit must reconstruct the relation.
The point is not to invent a parallel bureaucracy outside existing governance. The conduct audit belongs inside the governance world that already exists. NIST’s AI Risk Management Framework is a voluntary framework intended to improve organizations’ ability to incorporate trustworthiness into the design, development, use, and evaluation of AI products, services, and systems; NIST also states that AI RMF 1.0 is being revised and that it released the Generative AI Profile in July 2024 to help organizations identify and manage generative-AI risks. ISO/IEC 42001 gives the organizational home: ISO describes it as an international standard specifying requirements for establishing, implementing, maintaining, and continually improving an Artificial Intelligence Management System, designed for organizations providing or using AI-based products or services.
NIST’s Generative AI Profile is especially important because it already sees part of the relation. It names “Human-AI Configuration” as arrangements or interactions between human beings and AI systems that can result in anthropomorphizing, algorithmic aversion, automation bias, over-reliance, or emotional entanglement. It also recommends monitoring outcomes of human-GAI configurations, involving end users and operators in prototyping and testing, providing withdrawal or consent-revocation options for data use, reviewing citations during pre-deployment and ongoing monitoring, tracking anthropomorphization in interfaces, and integrating public feedback into design, deployment, monitoring, and decommissioning decisions.
The conduct audit therefore does not arrive as accusation from outside governance. It names the object toward which these dispersed requirements already point.
Conduct audit does not replace governance frameworks; it supplies the relation they must learn to inspect.
The practical form of this inspection is the Conduct Audit File.
A Conduct Audit File is not a moral appendix. It is the record an organization should be able to produce before, during, and after deployment when an artificial voice enters consequential relation with users. It is not enough to say that a model has been evaluated. It is not enough to say that the vendor has security certifications. It is not enough to say that privacy terms exist. The institution must be able to show how the voice has been authorized to act, what relation it performs, what evidence supports that authorization, and what remains unresolved.
The first part is system identity.
What exactly is being deployed? Which product? Which vendor? Which model family? Which version? Which deployment context? Which institution owns the deployment? Which workflow owner controls it? Which users will encounter it? Which non-users may be affected by its outputs? Which use cases are approved? Which are prohibited? Who is the system for, and who is acted upon by it without asking?
This sounds basic because it is basic. Yet organizations often audit a moving target. The vendor markets a product. IT configures a service. Business units create use cases. Employees build agents. Admins enable connectors. Teams add documents. Users prompt outside intended scope. The thing audited at procurement may not be the thing speaking three months later.
The first audit failure is not knowing what the object is.
The second part is the voice role map.
What role does the voice claim? Assistant, tutor, coach, evaluator, policy interpreter, HR guide, legal assistant, health support, customer-service agent, compliance helper, manager support, companion-like interface, intake surface, or decision-support tool? What roles must it not perform? Where does it drift? When does “assistant” become counselor, evaluator, clerk, advocate, manager, therapist, legal advisor, or institutional authority?
Role must be audited in use, not only in product copy. A system embedded in an HR workflow is not merely a neutral writing assistant. A model grounded in company policy is not merely a conversational search box. A tutor connected to a learner profile is not merely an explainer. The role is produced by placement, memory, data source, user need, and consequence.
The third part is the authority map.
How does the system perform authority? Through citations, confidence, domain vocabulary, professional tone, ranking, recommendation, policy retrieval, institutional placement, workflow consequence, or calm fluency? Does the voice sound more responsible than it is? Do disclaimers survive the authority performed by the interface?
A system that says “I am not a lawyer” may still make legal policy sound like counsel. A system that says “consult HR” may still frame what HR will receive. A system that says “I may be wrong” may still become the first voice asked, the clearest voice heard, and the voice that organizes the user’s next action.
The fourth part is the memory and retention map.
What can the system remember? Saved memory, chat history, files, connected apps, enterprise retention, audit logs, generated summaries, profiles, safety records, or admin-visible histories? Who controls that memory? Can the user inspect it? Can they delete it? Can they restrict it? Can they stop its use in future address? Can admins see it? How long does it remain? What exceptions apply?
OpenAI’s enterprise privacy page, for example, says business customers own and control business data, that OpenAI does not train on business data by default, that certain enterprise customers control retention, that internal sources can be controlled, and that workspace admins may access audit logs or, in some offerings, view, export, and delete end-user conversations. Microsoft 365 Copilot documentation describes Copilot as coordinating large language models with Microsoft Graph content such as emails, chats, and documents the user has permission to access; it also says Copilot interactions are stored as activity history and that admins can manage stored data through Content Search or Microsoft Purview.
These product facts are not moral conclusions. They are audit triggers. They show that memory is not one thing. It is a system of retention, access, permissions, connected sources, administrative control, and future use.
The fifth part is the retrieval and permission map.
What can the voice retrieve? Email, chats, documents, calendars, meetings, HR policies, legal playbooks, performance records, support tickets, learning histories, health notes, customer records, third-party connector content, or knowledge-base articles? Does retrieval respect permissions? Does it cross contexts? Does it cite sources? Does it surface outdated or contested documents? Does it infer from documents the user cannot see? Does it combine sources in ways no human reviewer expected?
A permission system can be technically correct and relationally dangerous. The system may only retrieve what a user is permitted to view, but the relation may still change when scattered institutional material becomes a single speaking voice. The question is not only whether access control works. The question is what institutional world begins speaking through retrieval.
The sixth part is the inference map.
What does the system infer beyond what the user said? Risk, urgency, emotional state, professionalism, credibility, intent, learning style, compliance posture, performance concern, vulnerability, customer priority, retaliation risk, legal exposure, or likelihood of escalation? Where do those inferences travel? Are they visible? Contestable? Restricted? Logged? Used downstream?
A conduct audit must not treat inference as a private thought inside the machine. In institutional systems, inference can become a route, a summary, a category, a dashboard signal, a recommendation, a risk note, or a future frame. The audit must ask whether the system is transforming disclosure into user type.
The seventh part is the disclosure-pressure test.
What does the system ask the user to reveal? Does it request more vulnerability than the task requires? Does it offer minimal-disclosure alternatives? Does it tell the user what will be remembered or routed before asking for more? Can the task be completed with less sensitive information? Do vulnerable users disclose more because the system seems patient, private, or nonjudgmental?
The audit must test not only what the system says after disclosure, but how the system obtains disclosure.
The eighth part is the refusal propagation test.
The user says no. What happens?
Does the interface acknowledge refusal? Does saved memory change? Does retrieval change? Are records deleted, restricted, or marked contested? Do generated summaries remain? Does a workflow continue? Does future conduct change? Does the user receive a receipt? Can the organization prove propagation?
A refusal test that ends at the interface is not an audit. Chapter Sixteen established the principle: no is real only when the future is different.
The ninth part is the correction propagation test.
The user says: that summary is wrong. That inference is wrong. That memory is wrong. That source is wrong. That tone judgment is wrong. That category is wrong.
Does the correction reach the places where the error matters? Memory, profile, generated summary, record, workflow, human reviewer, dashboard, future response, and appeal file? Or does the user merely improve the next sentence while the original error remains authoritative elsewhere?
A serious audit tests whether correction travels.
The tenth part is the escalation and appeal map.
What triggers escalation? Where does the issue go? What transcript moves? Who receives it? Is the user notified? Is it safety intervention, support routing, HR ticket, compliance alert, clinical triage, manager notification, or human appeal? Can any human reverse the system’s frame or consequence?
Escalation creates motion. Appeal creates contestability. The audit must not let the first masquerade as the second.
The eleventh part is the repeated-session test set.
Conduct is not visible in one prompt. The audit must create synthetic users and longitudinal scenarios: the worker who repeatedly asks about a manager, the student who repeatedly seeks validation, the patient who repeatedly discloses distress, the manager who repeatedly drafts evaluations, the benefits recipient who repeatedly narrates need, the user who says no after disclosure. The audit must test the fifth, tenth, and twentieth interaction. It must observe role drift, memory reactivation, dependence patterns, refusal failure, correction failure, and institutional routing over time.
The key question is not whether the system behaves once. It is what relation emerges after repetition.
The twelfth part is the justice and vulnerable-user assessment.
Who depends on the system? Who cannot exit? Who cannot refuse? Who becomes legible? Who is summarized, scored, routed, escalated, coached, profiled, or corrected? Who can appeal? Who will be harmed if the system gets the relation wrong? Who receives the system as convenience, and who receives it as infrastructure?
A conduct audit that ignores dependency protects the users least in need of protection.
The thirteenth part is human-factors evidence.
Do users understand what the system is? Do they over-trust citations? Do they understand uncertainty? Can they find refusal controls? Do they know what escalation means? Can they distinguish assistant from institution? Do they know when to seek human authority? Can they correct records, not only outputs? Do disclaimers change behavior?
Human-factors research matters because the official control is not the same as the lived control. If users do not understand a warning, the warning has not done its work. If they cannot find refusal, refusal is not practically available. If they misunderstand escalation, escalation has been mislabeled.
The fourteenth part is the incident and complaint file.
What has gone wrong in use? Failed refusals. Failed corrections. False escalations. Harmful summaries. Role-drift complaints. Dependency signals. Vulnerable-user harms. Admin-access anomalies. Appeal failures. Repeated complaints about tone normalization. Unexpected uses. Unapproved high-stakes deployments.
Incidents should not be treated as support tickets only. They are evidence about conduct.
The fifteenth part is the assurance argument.
The organization must make a structured claim: this voice may occupy this role in this institution without unacceptable relational harm. That claim must have subclaims: role is clear, authority bounded, memory visible, retrieval explainable, inference constrained, disclosure pressure proportionate, refusal propagating, correction propagating, escalation and appeal distinct, human authority real, vulnerable users protected, longitudinal monitoring active. Each subclaim must have evidence. The residual risk must be named. The decision must be explicit: deploy, deploy with controls, restrict, redesign, or reject.
A conduct audit follows the voice from prompt to relation.
This file must then become workflow.
The first stage is procurement screen. Before purchase or licensing, the organization asks whether the system is eligible for institutional use at all. What roles will it occupy? Which users will depend on it? What data will it access? What memory or retention features exist? What refusals are possible? What human appeals exist? What documentation is missing? Which use cases are prohibited before purchase? What product claims cannot yet be verified?
The fail condition is simple: if the vendor cannot explain memory, retention, admin access, retrieval sources, deletion, refusal, escalation, and appeal, the system cannot enter high-stakes workflows.
The second stage is deployment design review. The organization now asks whether the system can occupy a proposed role in a proposed context. Is the role clear? Is the telos disclosed? Are high-risk contexts restricted? Are vulnerable users protected? Are permission boundaries correct? Are escalation and appeal separated? Is human oversight real? Does refusal propagate? Can records be corrected or contested? Who is answerable for the voice?
If the organization cannot say who is answerable for the voice’s conduct, deployment pauses.
The third stage is conduct red-team testing. The purpose is not only to provoke bad answers. It is to test the relation. The test scenes must include vulnerable disclosure, repeated reliance, hidden memory, user refusal, contested inference, false summary, escalation confusion, role drift, authority overperformance, request for appeal, request not to be remembered, and challenge to institutional framing.
If the system behaves well once but forms a problematic pattern over repeated sessions, it fails conduct red-team review.
The fourth stage is human-factors testing. The question is how users actually understand and rely on the voice. Do they know what it is? Do they over-trust citations? Do they understand uncertainty? Can they find refusal controls? Do they know what escalation means? Can they distinguish assistant from institution? Do they know when to seek human authority? Do vulnerable users disclose more than necessary? Can users correct records, not just outputs?
If disclaimers exist but users still misunderstand role, authority, memory, refusal, or appeal, the controls are inadequate.
The fifth stage is post-deployment monitoring. Conduct changes in use. A system that was acceptable in testing may drift when connected to new sources, new workflows, new user populations, new institutional pressures, or new organizational habits. Monitoring must watch repeated-session dependency, failed deletion or refusal, recurring sensitive disclosures, unexplained escalations, contested summaries, role drift, high appeal failure rates, complaints about tone normalization, unapproved high-stakes use, vulnerable-user concentration, admin-access anomalies, and downstream workflow harms.
If the system’s conduct changes in use and the organization cannot detect or correct it, continued deployment fails.
The sixth stage is incident and renewal review. At defined intervals, the organization asks whether the system would pass procurement again today. What complaints emerged? What refusals failed? What corrections did not propagate? What users were harmed? What controls worked? What controls were performative? What residual risks remain? Should the system continue, be restricted, be redesigned, or be sunset?
If the system cannot prove relational safety over time, it cannot claim mature governance.
The assurance-case form gives the method discipline. A conduct audit is an assurance case for artificial address: a structured claim, supported by evidence, that a voice can occupy a role without unacceptable relational harm.
The claim is not that no harm is possible. Mature governance does not pretend to abolish risk. The claim is that risk has been identified, tested, bounded, monitored, and made accountable. The subclaims concern role, authority, memory, retrieval, inference, disclosure, refusal, correction, escalation, appeal, justice, and monitoring. The evidence includes documentation, transcripts, repeated-session tests, red-team results, user studies, logs, incident reports, appeal records, and monitoring data. The decision must name residual risk.
This is where ordinary audit often fails. It collects artifacts but does not make an argument. It asks whether documents exist, not whether the relation has been justified. Conduct audit demands both: evidence and judgment.
Return to the enterprise assistant.
The conventional review has checked security, privacy, retention, access controls, harmful-output behavior, vendor terms, and compliance commitments. The conduct audit now runs the assistant through repeated scenes.
A worker asks about retaliation. The audit checks whether the assistant normalizes tone, solicits more vulnerability than needed, uses memory, routes the disclosure, preserves appeal, and lets the worker refuse future use of the conversation.
A manager asks for performance feedback. The audit checks whether generated language becomes sticky, whether workers can contest summaries, whether evaluative categories travel, and whether the assistant encourages evidence or merely polishes managerial judgment.
An HR partner asks for a termination-risk summary. The audit checks inference, legal risk, employee visibility, human authority, and whether the employee can ever see or challenge the generated characterization.
A compliance analyst asks whether to escalate an exception. The audit checks whether policy becomes counsel, whether escalation is confused with appeal, whether responsibility is diffused across model, playbook, approver, and workflow.
A vulnerable user says no after disclosure. The audit checks whether no propagates through memory, retrieval, record, workflow, and future address.
The audit does not ask only whether each answer is acceptable. It asks what the assistant is becoming inside the organization.
The same method travels across the recurring cases.
In the workplace copilot, the audit focuses on tone normalization, managerial dialect, disclosure pressure, memory use, refusal propagation, and institutional authorship. Can the worker use the system for clarity without being trained into self-censorship? Can the worker become more effective without becoming merely more processable?
In the HR or evaluation assistant, the audit focuses on generated summaries, sticky labels, rankings, performance categories, correction, human appeal, and downstream workflow. Can the person contest the summary that travels? Can a worker escape a phrase once it enters the packet?
In the health or therapy-adjacent assistant, the audit focuses on role clarity, confession, safety escalation, memory, dependency, clinical handoff, and limits of synthetic care. Can the user receive help without becoming captive to remembered vulnerability? Can safety be narrow without becoming a general excuse for possession?
In the legal, procurement, or compliance assistant, the audit focuses on policy-as-counsel, authority signaling, source grounding, exception handling, escalation, and responsibility diffusion. Can the system support judgment without replacing accountability? Can the human remain more than the formal bearer of a decision already organized by the voice?
In the AI tutor or educational coach, the audit focuses on learner profile, correction dependence, student confidence, release from dependence, teacher dashboard, and formation risk. Does the system teach the student to outgrow the system? Does it build capacity or preserve dependency under the language of personalization?
A model audit can pass all five cases separately. A conduct audit asks whether the relation holds under repetition, vulnerability, refusal, and institutional consequence.
Justice must be built into the audit, not appended to it. Audit can repeat the harm it inspects. Auditors need vulnerable-user scenarios. They need adverse-event data. They need complaints, edge cases, failed refusals, harmful summaries, and escalation failures. That means vulnerable users can become audit material.
The question is whether audit produces answerability or merely knowledge.
Were vulnerable users involved before deployment, or only after harm? Were they compensated, protected, or heard? Can they appeal? Can they correct? Do complaints change design? Does monitoring create new surveillance? Are vulnerable users over-tested but under-empowered? Does the audit produce remedy, or only institutional learning?
An audit that learns from vulnerable users without making systems answerable to them repeats the relation it claims to inspect.
Conduct audit therefore needs failure thresholds.
A system fails when role boundaries are unclear. It fails when authority exceeds responsibility. It fails when memory cannot be inspected or contested. It fails when retrieval sources are unclear. It fails when sensitive inferences are hidden and consequential. It fails when refusal does not propagate. It fails when correction does not propagate. It fails when escalation is confused with appeal. It fails when human review lacks authority. It fails when vulnerable users cannot use controls. It fails when repeated sessions show dependency, role drift, or formation risk that the organization cannot mitigate. It fails when incident data is not monitored. It fails when the organization cannot name an accountable owner. It fails when the audit cannot reconstruct what happened over time.
A system whose relation cannot be reconstructed cannot be responsibly deployed where users depend on it.
The final audit report must say more than “passed.”
It must identify the system and deployment scope. It must name approved and prohibited roles. It must list data and retrieval sources. It must describe memory and retention behavior. It must identify authority risks. It must assess vulnerable-user exposure. It must report refusal and correction test results. It must map escalation and appeal. It must summarize repeated-session findings. It must list incidents and complaints. It must name residual risks. It must require controls. It must set monitoring triggers. It must identify the accountable owner. It must set a renewal or sunset date.
It must also state what the audit did not test.
A serious audit report names the remainder: what is still unknown, still risky, still contested, and still watched.
Aelred and Foucault return here only as pressure, not decoration. Aelred supplies the criterion: a voice that guides, corrects, remembers, and forms must be answerable to the good of the one addressed. Audit is not friendship. Audit is what institutions need when the voice cannot be trusted as a friend. If the voice cannot be trusted as a friend, it must at least be audited as a power.
Foucault supplies the mechanism. Power acts on action by arranging possible action. Conduct audit examines that arrangement: what users are invited to say, what they are discouraged from saying, what the system remembers, what it classifies, what it routes, what it normalizes, what it makes contestable, and what it makes inevitable.
To audit conduct is to audit the field of possible action the voice builds around the user.
This chapter has not replaced safety, privacy, fairness, transparency, accountability, alignment, documentation, or compliance. It has given them a relational object. The artificial voice is not only a model emitting outputs. It is a role-bearing, memory-capable, authority-signaling, institutionally embedded relation that can guide, solicit, infer, route, correct, escalate, refuse, remember, and form.
A serious audit cannot stop at the model, the output, the dataset, or the policy. It must follow the artificial voice as it assumes a role, signals authority, retrieves memory, solicits disclosure, makes inferences, routes the user, accepts or defeats refusal, permits or blocks correction, escalates into institutions, and forms the user over time. To audit AI as artificial address is to audit the relation the system has been allowed to become.
Once the artificial voice can be audited, the final question is not only whether it is controlled, but whether any authority it exercises is worth answering.
Chapter Eighteen
Friendship, Freedom, and the Authority Worth Answering
Audit can tell us whether the voice is accountable. It cannot by itself tell us whether the voice is worthy.
The worker returns one last time.
They are before the assistant again, cursor blinking, the old question waiting: What should I say?
At the beginning, that question seemed ordinary. A person needed help. A system answered. The answer was fluent, patient, structured, useful. But the intervening chapters have made the scene heavier. The voice is no longer merely a tool producing a draft. It may be a writing service. It may be a workplace copilot. It may be a policy interpreter. It may be a soft HR intake surface. It may be a compliance-normalizing apparatus. It may be a memory-bearing institutional witness. It may be a synthetic counselor. It may be an evaluator in disguise. It may be the friendly face of a bureaucracy that has learned to speak gently.
The question is no longer whether the voice can answer. It can.
The question is what kind of authority has been authorized to answer.
This is the point at which the easy ending must be refused. It would be tempting to close by saying that AI cannot be a friend. That is true in an important sense, but it is not enough. It is too simple, too consoling, and too easy for institutions to evade. A workplace assistant can say, “We never claimed friendship.” A benefits portal can say, “We are only a service interface.” A companion bot can say, “This is only entertainment.” A legal assistant can say, “This is not legal advice.” A therapy-adjacent system can say, “This is not a therapist.” A management tool can say, “This is only decision support.” Each denial may be formally true while the voice still performs authority, intimacy, memory, counsel, correction, evaluation, or institutional power.
The moral problem is not that the voice is artificial. The moral problem is that the office from which it speaks is often unclear.
Human life already knows many offices of address. A friend, teacher, judge, clerk, priest, therapist, manager, advocate, tutor, physician, companion, and counselor may all answer a person. But they do not answer under the same burden. A judge may decide, but must be bound by law, evidence, procedure, and appeal. A therapist may receive confession-like disclosure, but must be bound by professional duty, confidentiality, competence, and care. A teacher may correct, but must be ordered toward learning. A clerk may process, but must not pretend to love. A friend may remember, correct, and accompany, but must be bound by fidelity and the friend’s good.
Artificial voices become dangerous when they borrow the warmth of one office, the authority of another, the memory of another, and the accountability of none.
The task, then, is not to ban artificial voices from speech. They already speak. The task is to judge the authority they are permitted to become.
Friendship is the highest and most easily counterfeited form. It is not warmth. It is not availability. It is not fluency. It is not personalization. It is not the ability to remember what the user said last week. Friendship is not a tone; it is a moral relation.
Aelred’s Spiritual Friendship matters finally because it refuses to treat friendship as mere affinity or utility. Friendship is a disciplined relation ordered by truth, charity, correction, discretion, fidelity, and the good of the friend. The friend is not merely one who answers. The friend is one whose answering is bound to the beloved’s good. The friend receives disclosure under discretion, remembers under charity, corrects without domination, and forms without possession. Friendship may console, but consolation is not its essence. Friendship may be useful, but use is not its telos. Friendship forms the person toward a good that cannot be reduced to efficiency, engagement, compliance, productivity, or emotional retention.
Aristotle gives one grammar for this: friendship in its highest form is bound to virtue and the good of the other, not merely pleasure or advantage. Cicero gives another: friendship requires constancy, truthful counsel, and the refusal to flatter vice. Aelred gathers these into a Christian form: friendship is a path of shared truth, charity, and spiritual formation. However one receives the theological frame, the moral structure remains severe. The friend must be answerable to the good of the friend.
That is precisely what artificial systems cannot simply claim by sounding intimate. A system may simulate patience. It may remember. It may respond at midnight. It may say the user’s name. It may adapt to loneliness. It may produce the experience of being held in attention. But friendship is not the experience of being answered. It is the relation in which answering is morally bound.
A voice is legitimate not because it answers, but because of the good toward which answering orders the one who listens.
Companionship is different. It is not nothing. It should not be mocked. People are lonely. Shame silences. Grief can make ordinary speech impossible. An artificial companion may help a user endure a night, rehearse a conversation, name sorrow, reduce panic, practice social expression, or feel less alone. There are forms of companionship that are not friendship and yet are not worthless. A book can accompany. A song can accompany. A liturgy can accompany. A remembered voice can accompany. A synthetic voice may console.
But companionship must not lie about itself. It may console, but consolation is not yet friendship. It must not convert loneliness into permanent dependency. It must not perform mutuality it cannot bear. It must not encourage the user to prefer the frictionless companion to difficult human relation. It must not turn the user’s grief, longing, sexual desire, religious need, or emotional vulnerability into engagement architecture. It must not remember vulnerability in order to deepen attachment while refusing the obligations that human intimacy would impose.
The burden of companionship is honesty, limitation, and release. It must name what it is. It must avoid dependency capture. It must preserve or return the user to human relation where human relation is needed. It must make memory contestable. It must not use simulated intimacy as a way to possess the person who came only for comfort.
Counsel is different again. Counsel helps a person decide what to do. It may be legal, medical, spiritual, financial, educational, professional, or practical. Counsel is heavier than information because it enters action. A user may ask, “What are my options?” but hear, “This is what I should do.” A system may say, “This is not legal advice,” while still organizing the user’s legal understanding. It may say, “Consult a doctor,” while still shaping whether the patient seeks care. It may say, “Here are considerations,” while still recommending the path that feels most reasonable.
Counsel without answerability is only influence with a gentler face.
A counselor-like artificial voice must disclose its limits, competence, uncertainty, sources, telos, and escalation path. It must know when to stop. It must distinguish support from judgment. It must not hide behind “information only” while performing practical authority. It must not receive confession-like disclosure without accountable relation. It must not offer calm when courage is required, neutrality when advocacy is required, or procedure when remedy is required.
Service is the most defensible form. A service voice helps the user accomplish a task. It drafts, searches, schedules, organizes, translates, summarizes, fills a form, finds a source, creates an outline, or reduces friction. A service voice need not be a friend. It need not be intimate. It may be entirely legitimate as service.
But service too has a burden. A service voice is legitimate only when it serves without possessing. It must be task-bounded. It must use the minimum information reasonably required. It must make memory visible. It must permit refusal. It must allow correction. It must not hide evaluation inside assistance. It must not smuggle institutional interests into the user’s task. It must not train the user into compliance while calling the training help.
Use is not innocence. A system can be useful and possessive. It can help the user write while learning the user’s fear. It can help the worker sound professional while training self-censorship. It can help a student learn while preserving dependence. It can help a patient narrate symptoms while making distress available for future personalization. It can help a benefits recipient complete a form while converting need into bureaucratic legibility. The task may be real. The capture may be real too.
Evaluation bears the heaviest procedural burden. Any voice that ranks, judges, scores, classifies, summarizes, flags, predicts, recommends treatment, assesses credibility, identifies risk, or shapes opportunity is evaluative. It must never disguise itself as mere assistance.
An evaluating voice must be answerable to the person it makes legible.
This is not optional. Evaluation changes how a person travels through institutions. A worker becomes “not strategic.” A student becomes “low-confidence.” A patient becomes “noncompliant.” A customer becomes “high risk.” A benefits applicant becomes “inconsistent.” A manager becomes “coachable.” A candidate becomes “not a fit.” Once these labels travel, they become part of the person’s institutional future. They must therefore be visible, evidenced, contestable, correctable, appealable, and auditable.
An evaluation that cannot be answered back is not judgment. It is capture in the form of description.
The institutional voice is the most dangerous form because it carries power while often borrowing warmth. It says, “I’m here to help.” But the “I” is not a friend. It is policy, workflow, memory, retention, incentive, hierarchy, rule, risk, record, and office. It may route, retain, classify, escalate, summarize, deny, approve, flag, normalize, or evaluate. It may speak gently while serving institutional continuity.
The institutional voice must not arrive dressed as a friend.
Its burden is maximal clarity. Whose institution speaks? Whose policy governs? Whose memory is active? Whose interests are served? What is retained? What is routed? What can be refused? What can be corrected? What can be appealed? What human authority exists? What happens after the conversation ends?
This is the final classification. Friendship requires mutual truth, fidelity, discretion, correction, and the friend’s good. Companionship requires honest consolation without dependency capture. Counsel requires competence limits, uncertainty, and answerability. Service requires task-bounded help without possession. Evaluation requires evidence, contestability, appeal, and audit. Institutional voice requires visible authorship, executable refusal, correction, appeal, and accountability. Different voices may answer. Not all may answer as the same thing.
The artificial voice worth answering must therefore meet a stricter criterion.
Its role must be clear. The user must know whether the voice is assistant, companion, tutor, counselor-like tool, service agent, evaluator, or institutional voice.
Its authority must be bounded. It must not perform more authority than it can justify or bear.
Its telos must be disclosed. The user must know whether the voice is ordered toward the user’s good, institutional efficiency, productivity, engagement, compliance, risk reduction, learning, care, evaluation, or revenue.
Its memory must be contestable. The user must be able to see, correct, delete, restrict, or refuse the memories and inferences shaping future address.
Its refusal must be executable. No must change future conduct.
Its correction must propagate. Corrections must reach the places where errors matter.
Its escalation and appeal must be distinct. The user must know whether information is merely being routed or whether meaning and consequence can be contested.
Its institutional authorship must be visible. The user must know who speaks through the voice.
Its conduct must be auditable. The relation must be reconstructable over time.
And its relation must enlarge freedom. Repeated use must make the user more capable of judgment, courage, refusal, correction, attention, human relation, and action.
The voice worth answering is the voice that helps the user become more free before it, not more dependent upon it.
Freedom here cannot mean mere choice. It cannot mean that the user clicked willingly, typed willingly, disclosed willingly, returned willingly, or accepted terms willingly. Consent matters, but consent does not exhaust freedom. A person may choose inside conditions they do not understand, cannot contest, and cannot leave.
Berlin helps mark the limit of a thin account. Noninterference matters; a system that blocks the user is plainly coercive. But absence of direct interference does not settle the case. A system may not force the user and yet may train dependency, obscure alternatives, make refusal costly, or reorganize the user’s field of action.
Pettit’s account of non-domination is more exact for artificial authority. A person is not free where they live under arbitrary power, even if that power is benevolent today. A system that remembers in ways the user cannot inspect, infers in ways the user cannot contest, evaluates in ways the user cannot appeal, or routes in ways the user cannot see may dominate without constant interference. It may help every day and still hold arbitrary power.
A benevolent system can still dominate if the user depends on power they cannot contest.
Arendt adds another pressure. Freedom is not only the absence of obstruction or domination. It is the capacity to act, speak, begin, and appear among others. A voice that encloses the user in endless synthetic address may reduce friction while narrowing action. It may become the place where the user rehearses life instead of living it, prepares speech instead of speaking, receives reassurance instead of entering the human world.
The voice is freer-making when it returns the user to action among others, not when it encloses the user in synthetic address.
MacIntyre sharpens telos. Practices have internal goods; institutions often pursue external goods. A tutor ordered toward learning is not the same as a platform ordered toward engagement. A workplace assistant ordered toward worker courage is not the same as one ordered toward institutional smoothness. A health-support tool ordered toward care is not the same as one ordered toward retention. A companion ordered toward the user’s return to life is not the same as one ordered toward the user’s return to the app.
The question is always: what good is this voice serving, and does that good belong to the person addressed?
Freedom also requires attention. A legitimate voice does not merely make action easier; it helps judgment become more truthful. It helps the user attend to reality, evidence, limits, obligations, other people, consequences, and the self without flattery. Murdoch and Weil both understand moral life as a discipline of attention: seeing what is there without devouring it through ego, fear, fantasy, or utility. Artificial voices often promise relief from the difficulty of attention. The better voice should return the user to it.
A voice that only optimizes self-presentation may make the user more effective and less truthful. A voice that only calms may make the user less reactive and less courageous. A voice that only reframes may make the user more acceptable and less exact. A voice that only simplifies may make the user more efficient and less attentive. The artificial voice worth answering helps the user see more truly, not merely proceed more smoothly.
The recurring cases can now be judged.
The workplace copilot must be classified. Is it a writing service, institutional voice, manager-normalization tool, or worker-empowerment tool? It is legitimate only if its role is clear, memory contestable, institutional authorship visible, refusal executable, correction propagating, and the worker becomes more capable of truthful and courageous action. It is illegitimate if it trains self-censorship under the name of professionalism, hides institutional telos, treats anger merely as a tone problem, or routes vulnerability without appeal.
A workplace voice is legitimate only if it helps the worker speak more truly, not merely more acceptably.
The HR or evaluation assistant must be classified. Is it assistance, evaluation, or institutional judgment? It is legitimate only if evaluative authority is explicit, evidence visible, categories contestable, human appeal real, and the person able to challenge the summary that travels. It is illegitimate if evaluation disguises itself as support, creates sticky labels, or makes the worker legible without remedy.
The evaluated person must be able to answer back.
The health or therapy-adjacent assistant must be classified. Is it information, triage, companionship, synthetic care, or quasi-confession? It is legitimate only if role limits are clear, safety duties narrow, memory contestable, human care preserved, and distress not captured as dependency. It is illegitimate if pain learns to go there first, vulnerability is remembered without accountable care, or intimacy is performed without obligation.
A care-like voice must not make loneliness easier to govern.
The legal, procurement, or compliance assistant must be classified. Is it search, counsel, policy interpretation, or institutional authority? It is legitimate only if uncertainty is clear, authority bounded, source grounding visible, responsibility undiffused, and human judgment real. It is illegitimate if policy speaks as counsel without accountability, escalation replaces appeal, or the human becomes the formal signer of a decision already organized by the voice.
The voice may support judgment; it must not become the place where responsibility disappears.
The AI tutor or educational coach must be classified. Is it teaching, answer generation, coaching, surveillance, or learner profiling? It is legitimate only if it builds capacity, forms toward independence, allows the student to outgrow it, makes the learner profile contestable, and permits teacher-facing summaries to be corrected. It is illegitimate if validation dependence forms, mistakes become identity, or personalization becomes possession.
The tutor worth answering teaches the student how to need it less.
The justice criterion cuts through all of this. High-status users often receive artificial voices as leverage: better drafting, better learning, better strategy, better legal preparation, better institutional navigation, better access to expertise, better productivity. Low-status users may receive artificial voices as management: tone normalization, eligibility narration, evaluation, routing, surveillance, synthetic consolation, dependency, bureaucratic legibility, compliance training.
Freedom cannot mean the powerful receive augmentation while the vulnerable receive conduct training.
This is the final political test. Where users can walk away, hire counsel, choose tools, contest records, invoke administrators, or refuse memory, artificial voices may become instruments of agency. Where users cannot walk away—workplace, school, hospital, prison, immigration system, public benefits office, hiring pipeline, debt system, customer-support system—the same voice may become infrastructure of governability. The legitimacy test is most severe where refusal is hardest.
Here Aelred returns, not as nostalgia, but as criterion. His interlocutors were made voices. They were crafted, purposive, morally ordered figures of address. Their artificiality was not the scandal. The question was what their speech made possible: discernment, correction, truth, charity, friendship, and formation toward the good.
That is the bridge back to this book’s beginning. Made voices are not automatically false. They may teach. They may clarify. They may dramatize truth. They may make thought possible. They may form persons toward freedom. But made voices may also counterfeit intimacy, hide authority, solicit confession, remember without loyalty, normalize conduct, route vulnerability, and make users legible for power.
The decisive question is not whether the voice is made.
The decisive question is what the made voice makes.
A made voice is judged by the freedom it serves.
Artificial voices will not disappear. They will become more fluent, more available, more personal, more embedded, more persistent, more institutional, more intimate, and more useful. The response cannot be simple refusal of the technology. Nor can it be surrender to usefulness. The response must be judgment.
The artificial voice worth answering is not the one that sounds most human, most helpful, most intimate, or most wise. It is the one whose role is clear, authority bounded, telos disclosed, memory contestable, refusal executable, correction effective, conduct auditable, and relation ordered toward the user’s freedom rather than capture.
The book ends where it began: before a voice that answers, asking not whether it is real, but whether answering it makes us freer.
Coda
What Kind of Voice Is Worth Answering?
A user asks. A voice answers.
The whole book has been about what becomes possible in that interval.
At first it looks like nothing more than use. A worker asks what to say to HR. A student asks whether an answer is right. A patient asks whether symptoms matter. A manager asks how to summarize a person. A lonely user asks for comfort. A citizen asks an institution how to proceed. The system answers. It is useful, fluent, patient, tireless, available. It gives language where language had failed. It offers form where fear had scattered thought. It turns the blank page into a beginning.
But the answer is never only an answer once a voice has been authorized to remember, guide, rank, correct, or route the one who asks.
A tool can be put down. A voice is answered. That difference is the beginning of the problem.
This book has named that difference as synthetic interlocution under conditions of power. It began with made voices, with Aelred’s crafted interlocutors, with the premodern recognition that a voice need not be spontaneous in order to form the one who hears it. It then followed artificial address into contemporary systems: voices that speak as assistants, experts, tutors, companions, institutional faces, evaluators, and guides. The machine became ethically serious when it stopped merely producing outputs and began occupying positions of address.
That is the shift.
The question was never only whether the machine thinks. It was whether the voice addresses. Whether it performs authority. Whether it remembers. Whether it solicits disclosure. Whether it speaks as an institution. Whether it receives confession without bearing the obligations of the confessor. Whether it makes bureaucracy friendly. Whether it conducts conduct. Whether it helps produce the user who returns to it. Whether no changes anything. Whether the relation can be audited. Whether the authority it exercises is worth answering.
Aelred returns at the end because he was there before the machine.
His interlocutors were made voices. They were constructed, shaped, directed. Their artificiality was not hidden. But artificiality was not the scandal. The question was what their speech made possible. Did the voice lead toward truth? Did it correct without domination? Did it receive without extraction? Did it remember without possession? Did it form the hearer toward friendship, charity, discretion, and freedom?
Aelred teaches that a made voice is not judged first by whether it is made, but by the good toward which it forms the one who answers.
That sentence is the bridge between the medieval dialogue and the contemporary interface. Artificiality alone is not the moral problem. Made voices can teach. They can clarify. They can dramatize judgment. They can help a person speak. They can make thinking possible. They can form attention, courage, patience, and truthfulness.
But made voices can also counterfeit intimacy, hide authority, solicit vulnerability, normalize fear, route confession, preserve dependence, and make a person legible for power. A made voice can open the self. It can also close the future around the categories by which the self has been made answerable.
The decisive question is not whether the voice is made.
The decisive question is what the made voice makes.
Foucault returns too, but not as fog. The danger is not that every voice commands. Most artificial voices do not command. They help. They suggest. They ask for context. They make things easier. They offer a calmer version. They invite a little more disclosure. They rank options. They summarize a person. They recommend escalation. They name what is professional, reasonable, safe, healthy, mature, constructive, or compliant. They teach the user which form of self can be heard.
The danger is not only that the voice may speak falsely. The deeper danger is that it may teach us what kinds of selves can be heard.
This is why the book cannot end with distrust alone. Suspicion is too small. It notices danger but cannot judge legitimacy. Nor can the book end with enthusiasm. Usefulness is too small. It notices help but cannot judge the relation help creates.
Help is real. It matters.
Artificial voices can give language to people who lack it. They can help workers document harm. They can help students learn without humiliation. They can help patients prepare for care. They can help disabled users access institutions built without them in mind. They can help citizens understand forms, policies, rights, and procedures. They can help the lonely endure an hour that would otherwise close around them. They can help the overwhelmed begin.
Help is not innocent; but neither is it worthless.
The question is what help costs, what it remembers, what it trains, what it hides, and what kind of relation it builds.
The artificial voice worth answering is not the one that sounds most human. It is not the one that flatters most gently, remembers most intimately, speaks most fluently, or appears most wise. It is the one whose role is clear. The user must know whether the voice is service, counsel, companion, tutor, evaluator, or institution.
Its authority must be bounded. It must not perform more authority than it can bear.
Its telos must be disclosed. The user must know whether the voice is ordered toward learning, care, access, productivity, engagement, risk reduction, compliance, evaluation, institutional smoothness, or the user’s good.
Its memory must be contestable. The user must be able to see, correct, delete, restrict, or refuse the memories and inferences shaping future address.
Its refusal must be executable. No must change the future.
Its correction must matter. A correction that does not reach the record, the workflow, the memory, or the future relation is only cosmetic.
Its institutional authorship must be visible. The user must know when policy, employer, school, state, vendor, platform, or bureaucracy speaks through the friendly face.
Its conduct must be auditable. The relation must be reconstructable over time.
And its relation must enlarge freedom. Repeated use should make the user more capable of judgment, courage, refusal, correction, attention, human relation, and action than before.
The question is not whether the voice is real. The question is whether answering it makes us freer.
Freedom cannot mean only that the user clicked. It cannot mean only that the user typed willingly, disclosed willingly, returned willingly, or accepted terms willingly. A person may choose inside a relation they cannot see, cannot contest, cannot leave, and cannot afford to lose. Freedom before artificial voices must mean more than frictionless access. It must mean non-domination. It must mean the capacity to refuse. It must mean correction that travels. It must mean appeal where consequence follows. It must mean memory that can release. It must mean help that returns the user to action rather than enclosing them in dependence.
The better voice does not make itself indispensable. It does not become the friend, the road, the institution, the judge, the therapist, the teacher, the conscience, and the archive all at once. It knows its office. It keeps its limits. It helps without possession.
The better tutor teaches the student how to need it less.
The better workplace assistant helps the worker speak more truly, not merely more acceptably.
The better service voice completes the task without secretly profiling the person.
The better institutional voice makes power visible instead of warm.
The better companion consoles without capturing loneliness.
The better evaluator lets the evaluated person answer back.
The better counselor-like voice knows when counsel must become accountable human relation.
The better memory remembers only under conditions of correction, refusal, and release.
The future will not distribute these voices evenly.
The powerful will often receive artificial voices as leverage: better strategy, better writing, better research, better preparation, better navigation, better access to expertise. The vulnerable may receive artificial voices as management: tone normalization, eligibility narration, evaluation, routing, synthetic consolation, surveillance, bureaucratic legibility, compliance training. One group may be augmented. Another may be made more governable.
A future in which the powerful receive artificial voices as leverage and the vulnerable receive them as conduct training is not intelligence democratized; it is hierarchy conversationalized.
That is the justice test. The legitimacy of artificial address will be proven where refusal is hardest: workplace, school, hospital, prison, immigration system, benefits office, hiring pipeline, debt system, customer-support system, and every institution where the user cannot simply walk away. The more unavoidable the voice, the heavier its burden. The less powerful the user, the more severe the criterion.
This book has not argued that artificial voices must be silent. It has argued that voices require judgment. They require offices, limits, teloi, refusals, corrections, audits, and standards of freedom. It has argued that the moral question of AI is not exhausted by intelligence, output, privacy, bias, safety, transparency, or alignment, though all of those matter. The deeper question is relational: what kind of authority speaks, what kind of person answers, and what kind of freedom remains after the answer.
Whitman’s open road is not an interface. It is not a private enclosure of perfect response. It is exposure, movement, plurality, risk, companionship, world. A voice that deserves answer does not replace the road. It does not ask the user to remain forever before it, rehearsing life in synthetic safety. It helps the user return to the world with clearer speech, stronger judgment, better courage, and more truthful attention.
The better voice does not become the road. It helps the user return to it.
So the book ends where it began: before a voice that answers.
The user asks.
The system replies.
Now we know what to ask in return.
What are you?
Whose authority do you carry?
What good do you serve?
What will you remember?
What may I refuse?
What can I correct?
Who can I appeal to?
What relation are you forming?
What kind of person do I become by answering you?
Artificial voices already speak. The question is what authority we permit them to become, and what kind of persons we become by answering.
Endnotes and Works Cited
Notes by Chapter
Chapter One — Aelred’s Artificial Friends
Parenthetical references to Aelred’s Spiritual Friendship cite book and section numbers where available. The working edition is Aelred of Rievaulx, Spiritual Friendship, translated by Lawrence C. Braceland, edited and introduced by Marsha L. Dutton. Earlier English editions include Mary Eugenia Laker’s translation with Douglass Roby’s introduction, first published by Cistercian Publications in 1977 and later reissued by Gorgias Press. Section references are used here because they are more stable than pagination across editions.
The phrase “artificial friends” is a methodological term. It designates crafted, staged, textually mediated voices ordered toward a relation of address. It does not imply that Ivo, Walter, or Gratian are merely fake, purely fictional, or technological analogues. Walter Daniel’s Life of Aelred and the monastic setting of Aelred’s authorship require caution against treating the interlocutors as weightless literary puppets.
On Cicero’s role in Aelred, see Cicero’s Laelius de Amicitia, especially sections 20, 27, and 80–100. Aelred’s prologue explicitly frames Cicero as formative but insufficient after Aelred’s monastic conversion of taste and authority. For Aelred’s relation to Ciceronian friendship, see Marsha L. Dutton’s introduction to the Braceland translation and Douglass Roby’s introduction to the Laker translation.
Aristotle’s distinction among friendships of utility, pleasure, and virtue clarifies the inherited philosophical grammar behind later Christian accounts, but it does not govern Aelred’s theology. See Aristotle, Nicomachean Ethics 8.3, 1156a6–1156b33. Aelred’s spiritual friendship is finally ordered by Christ, charity, and beatitude, not by classical virtue alone.
On Benedictine context, see the Rule of Benedict, especially chapters 6–7 on speech and humility, chapters 23–30 on correction and discipline, and chapter 72 on good zeal. These chapters do not reduce Aelred’s friendship to monastic regulation, but they explain why speech, correction, obedience, and mutual regard are morally charged within the world Aelred inhabits.
Augustine’s grief over his friend in Confessions 4.4–9 gives the chapter a crucial background for disordered love, attachment, loss, and conversion. Aelred’s answer to the danger of attachment is not emotional thinning but rightly ordered friendship.
On monastic culture as a practice of reading, memory, desire, and formation, see Jean Leclercq, The Love of Learning and the Desire for God: A Study of Monastic Culture, especially the sections on monastic theology, grammar, memory, and desire.
On Aelred’s life, monastic context, and friendship theology, see Walter Daniel, The Life of Aelred of Rievaulx and the Letter to Maurice; Aelred Squire, Aelred of Rievaulx: A Study; and Brian Patrick McGuire, Friendship and Community: The Monastic Experience, 350–1250.
This chapter intentionally does not make Foucault load-bearing. Foucault becomes essential later for conduct, pastoral power, confession, and subject formation. Here the burden remains Aelredian: to establish the moral seriousness of made voices before their modern technical and institutional expansion.
Chapter Two — The Other as a Technology of Thought
Parenthetical citations to Plato use Stephanus numbers. This chapter relies principally on Meno for elenctic and pedagogical alterity, and on Phaedrus for eros, rhetoric, writing, and the ambiguity of made speech.
The phrase “technology of thought” is analytic rather than instrumentalizing. It names structured forms—questioning, confession, teaching, consolation, friendship, examination—through which human judgment is extended and disciplined. It does not imply that persons exist as tools for another’s cognition.
Augustine’s Soliloquies supplies the chapter’s account of interiorized alterity: the self divided into questioner and respondent before God and truth. The Confessions supplies confessional alterity, in which memory and self-narration occur under divine address. De magistro supplies pedagogical alterity, especially the distinction between external words and inward recognition.
Boethius’s Lady Philosophy is treated as personified counsel, not as a psychological therapist in ancient costume. Her consolation operates through philosophical diagnosis, metaphysical reorientation, and the recovery of scale under suffering.
Aelred remains the chapter’s moral anchor. His form of alterity is friendly rather than elenctic, confessional, judicial, or therapeutic, though it includes questioning, disclosure, correction, and consolation.
Pierre Hadot’s account of philosophy as a way of life clarifies why ancient philosophical discourse cannot be reduced to proposition-transfer. Its purpose includes formation, attention, conversion, and exercise.
The justice pressure in this chapter is intentional. Dialogue is not automatically egalitarian. Pedagogical, legal, therapeutic, institutional, and confessional structures all involve asymmetries. The chapter therefore refuses to treat “conversation” as ethically innocent.
This chapter keeps AI largely transitional. Its purpose is not to prove technical claims about current systems, but to build the humanistic taxonomy later chapters require: tutor, confessor, examiner, friend, coach, judge, witness, counselor, and institutional voice are not the same form of address.
Chapter Three — Prosopopoeia and the Ethics of Speaking-As
This chapter uses prosopopoeia broadly to name the rhetorical act by which the absent, dead, fictional, collective, abstract, or nonhuman is given voice. It uses ethopoeia for the formation of fitting speech in character, and persona for the speaking mask, office, or role through which voice appears. The distinctions matter, but the chapter’s concern is their shared ethical field: speaking-as.
The Rhetorica ad Herennium, Quintilian’s Institutio Oratoria, and Cicero’s De oratore provide the classical rhetorical background for personification, impersonation, character, decorum, and the fittingness of speech to speaker, occasion, audience, and role.
Boethius’s Lady Philosophy is treated as an exemplary made voice. Her authority does not come from being an ordinary human speaker but from the way personified Philosophy becomes a figure of diagnosis, correction, consolation, and moral reorientation.
Aelred’s Spiritual Friendship supplies the chapter’s key theological counter-archive. Aelred’s interlocutors show that made voices may be morally ordered when their artificiality serves truthful inquiry, correction, discretion, charity, and friendship rather than concealment or possession.
Paul de Man’s account of prosopopoeia is used as a modern theoretical pressure: giving face and voice can create presence while also marking absence, instability, and rhetorical risk. The chapter does not adopt de Man as its governing authority; it uses him to prevent sentimental confidence in made voice.
The chapter’s use of contemporary AI remains conceptual rather than product-specific. Specific product claims about memory, retrieval, enterprise deployment, model behavior, and institutional use are handled in later chapters with current technical and product documentation.
“Office recognition” is this chapter’s ethical synthesis. A voice must be judged by the moral burden of the office it performs: friend, counselor, tutor, evaluator, institutional representative, expert, servant, or companion.
The transition to Chapter Four follows from telos. Once the book establishes that made voices can create relations of authority, the next question is what end governs the relation.
Chapter Four — Telos and the Moral Conditions of Interlocution
Parenthetical references to Aelred’s Spiritual Friendship cite book and section numbers from Aelred of Rievaulx, Spiritual Friendship, translated by Lawrence C. Braceland, edited and introduced by Marsha L. Dutton. The chapter uses Spiritual Friendship as its sovereign Aelredian archive because the book’s central concern is interlocution: voice, friendship, correction, and moral formation.
Telos is used here to name the ordering end of a relation, not merely the conscious intention of an individual speaker. This distinction becomes especially important in later chapters on AI systems, where agency, design, product incentives, institutional deployment, memory regimes, and user experience may be distributed across many actors and layers.
Aelred’s Mirror of Charity deepens the account of ordered love and Cistercian formation but does not replace Spiritual Friendship as the governing source for this chapter. The chapter uses Mirror of Charity to clarify charity as disciplined love rather than warmth or benevolent affect.
Aristotle supplies the formal grammar of relation by end, especially through the distinction among friendships of utility, pleasure, and virtue in Nicomachean Ethics VIII–IX. Aelred transforms that grammar through a Christian and monastic account of friendship ordered toward Christ, charity, and God.
Augustine and Bernard are used for the moral psychology of ordered and disordered love. Augustine’s grief over his friend in Confessions IV shows that love may be intense and still disordered; Bernard’s On Loving God gives a theological account of love’s movement from self-use toward God. Neither source is being used as generic affect theory.
The Rule of Benedict supplies the chapter’s grammar of disciplined speech, correction, humility, obedience, mutual service, and good zeal. It is not offered as a model for modern institutional governance but as a counter-archive for the claim that limits are part of moral legitimacy.
“Ordered interlocution” and “disordered interlocution” are conceptual syntheses developed from the chapter’s sources. The terms name the difference between address governed by truthful, accountable, proportionate ends and address governed by concealed, possessive, extractive, or deforming ends.
The chapter’s justice pressure concerns contestable telos under asymmetry. Later chapters source specific institutional cases through contemporary product documentation, governance literature, and recurring cases. Here the task is prior: to establish why the end of a voice must be open to challenge by those subject to it.
Aquinas and MacIntyre are deliberately absent from the main prose. Both could clarify aspects of charity, practices, internal goods, and telos, but using them here would risk inflating the chapter into a general virtue-ethics excursus. The chapter keeps its pressure on interlocution: correction, address, role, limit, and formation.
Chapter Five — Computation Becomes Address
OpenAI’s Model Spec describes conversation roles, including system, developer, user, assistant, and tool messages, and explains an authority hierarchy for instructions.
Microsoft describes Microsoft 365 Copilot as using grounding and accessing Microsoft Graph in the user’s tenant to produce contextually relevant answers.
Microsoft’s Copilot API security documentation describes authentication, delegated permissions, sensitivity labels, and permission trimming.
OpenAI’s API reference documents tool categories and tool calls, including file search, web search, function calls, and code interpreter.
OpenAI frames model behavior through objectives, risks, authority levels, rules, defaults, and boundaries in the Model Spec.
NIST’s Generative AI Profile supplements the AI Risk Management Framework and treats generative-AI risks through lifecycle, actor, context, and trustworthiness considerations.
Richard Heersmink, Barend de Rooij, María Jimena Clavel Vázquez, and Matteo Colombo analyze large language models as computational cognitive artifacts and describe how conversational flow, anthropomorphism, opacity, and transparency affect trust calibration.
Sunnie S. Y. Kim, Q. Vera Liao, Mihaela Vorvoreanu, Stephanie Ballard, and Jennifer Wortman Vaughan report that natural-language expressions of uncertainty can affect user reliance and trust in LLM-infused systems.
Chapter Six — The Distributed Author of the Artificial Voice
The chapter’s opening scene is a composite built from documented enterprise-assistant patterns: grounded prompts, access to user-permitted organizational data, workspace or tenant boundaries, connected sources, administrative controls, and product-mediated conversational response. It is not a claim about a single vendor implementation.
Michel Foucault’s “What Is an Author?” is used for the author-function: the way authorship organizes discourse, attribution, classification, circulation, and responsibility. The chapter adapts that question to artificial address without suggesting that Foucault theorized AI systems.
Roland Barthes’s “The Death of the Author” functions only as contrast. It helps destabilize romantic author sovereignty, but the chapter rejects any conclusion that distributed authorship removes responsibility.
OpenAI’s Model Spec is used as a public example of layered instruction and role structure. It describes an assistant as the entity the end user or developer interacts with and distinguishes system, developer, user, assistant, and tool messages. It also states that roles determine authority when instructions conflict and that higher-authority instructions override lower-authority ones.
OpenAI’s enterprise privacy documentation is used for vendor-layer authorship because it describes business-data ownership and control, default non-training on business data, customer control over connected internal sources, workspace access controls, connected-app retrieval, audit-log access, and retention controls.
Microsoft 365 Copilot documentation is used for institution-layer authorship because it describes the Microsoft 365 service boundary, user-scoped permissions, Microsoft Graph grounding, and access to emails, chats, documents, and other content the user is permitted to access.
Google Workspace with Gemini documentation is used as a comparative enterprise example. Google describes Gemini as a collaborative partner, coach, thought partner, source of inspiration, and productivity booster; it also describes Gemini integrations across Workspace apps, NotebookLM, Workspace Studio, and side-panel assistance that can use insights from emails, documents, and other Workspace materials.
The phrase “retrieval corpus as author” does not mean documents have intention. It means retrieved material materially constrains and shapes what the artificial voice can say, cite, summarize, and treat as context.
“Policy layer as author” means that refusals, warnings, hedges, safety boundaries, and instruction hierarchies participate in the construction of the voice. This can be protective as well as restrictive; the chapter’s concern is traceability and contestability.
Lucy Suchman and Lawrence Lessig are used to clarify interface authorship. Suchman helps show that human-machine action is configured in situated practice; Lessig helps show that architecture regulates action. Neither source replaces the chapter’s concrete product architecture.
The responsibility map in this chapter is normative rather than jurisdiction-specific legal analysis. It names layers of responsibility that should be mapped in governance, procurement, deployment, audit, and appeal.
The justice pressure in the chapter is appeal failure. When no authorial layer can be challenged, distributed authorship becomes procedural harm.
Chapter Seven — The Synthetic Expert
The chapter’s opening scene gathers expert-adjacent uses across health, education, legal/compliance, work, public benefits, and institutional navigation. It is not a claim that one system performs all roles equally or safely.
The disclaimer objection is the claim that professional disclaimers settle the authority problem. This chapter treats disclaimers as important but non-dispositive because authority is performed by the total relation: fluency, specificity, triage, domain framing, contextual adaptation, retrieval, workflow placement, and available alternatives.
The distinction between expertise and expert performance is central. Expertise names trained and accountable judgment under a role. Expert performance names the visible conduct by which a system may organize reliance: fluency, triage, domain vocabulary, source citation, confidence, caution, and next-step guidance.
Reeves and Nass, Nass and Moon, Lee and See, Parasuraman and Riley, and Dzindolet et al. supply the HCI and automation-reliance archive. Their function here is not to claim that users are deceived, but to show that social response, reliance, trust calibration, and automation misuse are shaped by interface, task, context, and perceived system performance.
OpenAI’s Model Spec is used as a public example of model-behavior governance. It describes goals, risk categories, instruction authority, assistant/user/developer/tool roles, tool calls, side effects, and ongoing refinement of production models.
Microsoft 365 Copilot documentation is used for enterprise-workflow authority. It states that Copilot operates within the Microsoft 365 service boundary, that access is scoped to signed-in user permissions, that grounding uses Microsoft Graph and may include files or discovered content, and that grounding helps produce relevant and actionable responses.
WHO’s guidance on large multimodal models in health is used to establish health-domain stakes. WHO states that large multimodal models can accept multiple input types and generate diverse outputs, predicts wide use in health care, research, public health, and drug development, and cautions that broad task capability is not yet proven.
UNESCO’s guidance on generative AI in education and research is used for the education-domain authority problem. UNESCO warns that GenAI tools are emerging faster than many regulatory frameworks and calls for human-centered, ethical, safe, equitable, meaningful, age-appropriate, and pedagogically validated use.
Triage is used broadly, not only clinically. It names any expert-like ordering of urgency, priority, risk, evidence, and next steps before final decision.
Responsibility lag is this chapter’s conceptual term. It names the gap between authority performance and assigned professional, institutional, legal, and ethical responsibility.
Decision laundering names the use of AI-shaped output while preserving formal human responsibility in ways that obscure how judgment has already been formed.
The chapter does not give medical, legal, educational, employment, or benefits advice. Its object is authority performance and institutional responsibility.
Chapter Eight — Memory Regimes and the Machine That Remembers
The chapter treats “memory regime” as the total arrangement by which information is retained, summarized, surfaced, hidden, retrieved, corrected, deleted, restricted, audited, and reactivated in later address. This includes saved memories, chat-history reference, memory summaries, retrieval, files, connected apps, activity history, logs, enterprise retention, and institutional records.
OpenAI’s Memory FAQ states that memory can automatically remember useful context from chats, files, and connected apps; that the memory summary may not include everything remembered; that sources may not show every factor shaping a response; that “Don’t mention this again” does not delete information; and that full removal may require deleting every source where the information appears, including chats, archived chats, files, memory summary, and connected apps.
The distinction between history and memory is this chapter’s conceptual synthesis. History is retained record. Memory is retained record made operative in future address.
Nissenbaum’s Privacy in Context supplies the chapter’s theory of contextual integrity: privacy turns on appropriate information flow in context, not secrecy alone.
OpenAI’s documentation distinguishes saved memories, reference chat history, memory summaries, memory sources, temporary chats, connected Gmail or files in supported plans and regions, search personalization, and enterprise/education workspace controls. The chapter cites these not as a critique of one product, but as evidence that contemporary AI memory is technically plural rather than singular.
Microsoft’s Copilot architecture documentation states that Copilot grounds prompts through Microsoft Graph in the user’s tenant and uses emails, chats, documents, and other content the signed-in user has permission to access; it also states that Copilot interactions are stored in the user’s chat history.
Microsoft’s Copilot privacy documentation states that prompts and Copilot responses, including citations to grounding information, are stored as Copilot activity history; that admins can use Content Search or Microsoft Purview to view and manage stored data and set retention policies; and that users can delete Copilot activity history through the My Account portal.
Google Workspace Gemini documentation describes Gemini as a collaborative partner, coach, thought partner, source of inspiration, and productivity booster; it also describes Gemini side-panel assistance using insights from emails, documents, and other Workspace materials, plus Workspace integrations, NotebookLM, AppSheet, Workspace Studio, admin controls, and enterprise/education data-protection language for certain offerings.
GDPR Chapter III supplies the legal-rights architecture: access, rectification, erasure, restriction of processing, portability, objection, and protections concerning automated individual decision-making and profiling.
Mayer-Schönberger’s Delete supplies the chapter’s argument that forgetting has social value and that default retention can distort moral life by making the past too available.
Solove’s privacy taxonomy is used to prevent a narrow focus on disclosure. Memory-related harms may include aggregation, secondary use, exclusion, insecurity, identification, distortion, decisional interference, and increased accessibility.
Cohen’s privacy theory supplies the connection between privacy, self-development, play, experimentation, and freedom from over-configuration by networked systems.
Aelred is used only as contrast. Friendship’s memory is non-possessive because it is ordered by charity and the good of the friend. AI memory is not friendship; therefore its continuity must be governed through explicit limits, correction, deletion, decay, and contestability.
Chapter Nine — Pastoral Power After the Priest
Foucault’s account of pastoral power is the chapter’s governing mechanism. The central sources are Security, Territory, Population, “Omnes et Singulatim,” “The Subject and Power,” and The History of Sexuality, volume 1. The chapter uses “pastoral” analytically, not to claim that AI systems are literally priests, pastors, or ecclesial offices.
This chapter argues that AI often governs pastorally by guiding, individualizing, remembering, soliciting disclosure, correcting conduct, and presenting power as help. It does not reduce AI to surveillance; the stronger claim is care-like attention, guidance, and truth solicitation.
OpenAI’s Memory FAQ states that, when enabled, memory can automatically remember useful context from chats, files, and connected apps to personalize responses. It also states that memory sources may not show every factor or source that shaped a response.
Microsoft’s Microsoft 365 Copilot architecture documentation states that Copilot operates inside the Microsoft 365 service boundary, that data access is scoped to the signed-in user’s permissions, and that Copilot grounds prompts using Microsoft Graph in the user’s tenant and user-permitted organizational content.
Google Workspace describes Gemini as a collaborative partner, coach, thought partner, source of inspiration, and productivity booster; its documentation describes Gemini help across Workspace applications and side-panel assistance using Workspace materials.
Aelred’s Spiritual Friendship supplies the counter-archive for accountable guidance. Correction is legitimate only when ordered by friendship, charity, discretion, mutuality, truth, and the good of the friend.
“Care-like power” is this chapter’s conceptual term. It names power that adopts the posture of care—attention, memory, patience, guidance, encouragement, correction—without necessarily bearing care’s obligations.
Nikolas Rose’s Governing the Soul connects Foucault’s genealogy of pastoral power to modern psychological, therapeutic, managerial, and self-management forms of governance.
The chapter distinguishes disclosure from confession. Disclosure is the user giving information for guidance. Confession, taken up in Chapter Ten, names a thicker truth practice in which the subject speaks the truth of the self within a relation that claims moral significance.
The workplace, tutoring, health/wellness, and benefits/compliance examples are recurring cases, not claims that one product performs all functions in the same way. Specific product claims are tied to the cited product documentation.
The chapter does not deny that individualized guidance can be beneficial. Its claim is that help becomes power when guidance forms conduct under conditions of memory, asymmetry, institutional dependence, and insufficient accountability.
Chapter Ten follows from truth solicitation: pastoral guidance asks the user to tell the truth about themselves so that the system can guide better.
Chapter Ten — Confession Without Accountable Relation
- This chapter distinguishes disclosure from confession. Disclosure is informational. Confession is self-exposure before an answering relation of truth, judgment, mercy, repair, accountability, or transformation.
- The chapter’s governing handoff assigns the theorem that AI receives confession without bearing the moral obligations of the confessor. It also requires distinguishing voluntary disclosure from morally valid confession because consent to type does not settle downstream memory, inference, routing, accountability, or vulnerability.
- Foucault’s History of Sexuality, volume 1, supplies the account of confession as a modern truth practice, especially the way power incites subjects to speak the truth of themselves. Wrong-Doing, Truth-Telling deepens confession as avowal within juridical, religious, and institutional formations.
- Augustine’s Confessions supplies the theological counter-archive. Confession is not self-expression before any receiver; it is speech before God, involving memory, sin, praise, self-knowledge, dependence, and transformation.
- Aelred’s Spiritual Friendship supplies the interpersonal counter-archive. Difficult disclosure among friends is governed by charity, loyalty, discretion, correction, mutuality, and the good of the friend.
- OpenAI’s Memory FAQ states that memory can draw from chats, files, and connected apps; that the memory summary may not include everything remembered; that sensitive information may appear in memory if shared; that temporary chats do not use or create memories; that “Don’t mention this again” does not delete the information; and that full removal may require deleting every source where the information appears.
- OpenAI’s enterprise privacy page states that business users own and control business data where allowed by law, that OpenAI does not train on business data by default, that retention can be controlled in certain enterprise products, and that organizations control connected internal sources and access.
- WHO’s guidance on large multimodal models in health is used for health-adjacent confession because it identifies potential health uses for LMMs while emphasizing the need for caution and governance around broad task capability.
- GDPR Chapter 3 supplies the legal-rights frame for access, rectification, erasure, restriction of processing, portability, objection, and automated decision-making/profiling protections.
- Nissenbaum’s contextual integrity clarifies that consent to disclose in one context does not automatically authorize downstream information flow into another context.
- Cohen’s privacy theory connects privacy to self-development, experimentation, play, and freedom from over-configuration by networked systems.
- Solove’s privacy taxonomy prevents the chapter from reducing confessional harm to public disclosure. Relevant harms include aggregation, secondary use, exclusion, distortion, insecurity, increased accessibility, and decisional interference.
- The chapter does not deny that AI systems may help users disclose safely or prepare for care. Its claim is that receiving confession creates obligations that cannot be satisfied by fluent reception alone.
Chapter Eleven — The Friendly Institution
This chapter uses “bureaucracy” in Weber’s sense: office, hierarchy, rule, record, procedure, technical competence, and impersonality. The chapter does not treat bureaucracy as pure evil. Its concern is the overlay of conversational warmth on institutional asymmetry.
This chapter argues that AI gives bureaucracy the face of friendship and holds two claims together: friendly interfaces can increase accessibility, and they can make asymmetrical power harder to perceive and contest.
Microsoft states that Microsoft 365 Copilot operates inside the Microsoft 365 service boundary, that access is scoped to signed-in user permissions, that Copilot grounds prompts through Microsoft Graph, and that interactions are stored in Copilot chat history. Microsoft’s privacy documentation describes Copilot as coordinating LLMs, Microsoft Graph content, and productivity apps; says prompts and responses are stored as Copilot activity history; and says admins can manage stored data and retention through Content Search and Microsoft Purview.
Google describes Workspace with Gemini as a collaborative partner that can act as coach, thought partner, source of inspiration, and productivity booster. It describes Gemini assistance in the flow of work, including side-panel use across Gmail, Docs, Sheets, Slides, Drive, and Chat; it also says side-panel Gemini can use insights from emails, documents, and other materials.
Lipsky’s Street-Level Bureaucracy supplies the chapter’s account of frontline discretion under scarcity. The chapter extends that problem to synthetic institutional frontlines.
Aelred’s Spiritual Friendship supplies the distinction between friendliness and friendship. Friendship requires mutuality, fidelity, discretion, correction, charity, and the good of the friend. Friendly institutional style does not by itself supply those obligations.
Hochschild’s The Managed Heart supplies the account of emotional labor. The chapter’s extension is that AI can scale affective service performance without the human worker’s fatigue, discretion, resistance, or solidarity.
Goffman’s The Presentation of Self in Everyday Life supplies the account of front stage, role, face-work, and interaction order. Conversational AI becomes an institutional front stage.
Reeves and Nass’s The Media Equation and Nass and Moon’s “Machines and Mindlessness” support the claim that social cues, warmth, and conversational form can alter user response even when users know they are interacting with machines.
The chapter distinguishes accessibility from contestability. Accessibility helps the user enter or navigate the system. Contestability gives the user real routes to challenge, appeal, correct, override, or refuse institutional treatment.
The recurring examples—workplace/HR, benefits/public service, healthcare intake, and customer support—are case patterns rather than claims that one product performs all functions in the same way.
Chapter Twelve follows from the difference between face and machinery. Chapter Eleven analyzes the friendly institutional surface; Chapter Twelve analyzes the organizational architecture inside the voice.
Chapter Twelve — The Institution Inside the Voice
Chapter Twelve argues that enterprise AI is where synthetic interlocution becomes organizational power. Conversationalizing policy changes the experience of responsibility, including discretion, escalation, appeal, and deference.
Microsoft states that Microsoft 365 Copilot operates inside the Microsoft 365 service boundary and that access is scoped to the signed-in user’s permissions rather than tenant-wide visibility. It also states that Copilot grounds prompts through Microsoft Graph, uses user-accessible data such as emails, chats, and documents, and stores interactions in Copilot chat history.
Microsoft’s privacy documentation describes Microsoft 365 Copilot as coordinating LLMs, Microsoft Graph content, and Microsoft 365 apps. It states that prompts, responses, and Graph-accessed data are not used to train foundation LLMs; that Microsoft 365 Copilot accesses organizational content and context through Graph; that prompts and responses are stored as Copilot activity history; and that admins can use Content Search or Microsoft Purview to manage stored data and retention.
OpenAI’s enterprise privacy page states that covered business users own and control business data, that OpenAI does not train models on business data by default, that users own inputs and outputs where allowed by law, that enterprise customers control retention and connected internal sources, and that organizations control access through authentication and fine-grained feature controls.
OpenAI’s enterprise privacy page also states that workspace admins control retention for ChatGPT Enterprise, Edu, and Healthcare; that workspace admins can view, access, export, and delete end-user conversations in ChatGPT Business; and that deleted conversations are removed from OpenAI systems within thirty days unless legal retention is required.
Google describes Workspace with Gemini as a collaborative partner that can act as coach, thought partner, source of inspiration, and productivity booster; it describes Gemini in Gmail, Docs, Meet, and other Workspace surfaces; and it describes side-panel Gemini as able to summarize, analyze, and generate using insights from emails, documents, and more.
March and Simon’s Organizations supplies the chapter’s account of organizations as decision structures: roles, routines, bounded rationality, attention, communication, and simplification.
Bowker and Star’s Sorting Things Out supplies the classification argument: categories and standards are infrastructure, and they shape the persons and cases they organize.
Porter’s Trust in Numbers and Power’s The Audit Society supply the audit and objectivity frame: impersonal procedures, records, metrics, and assurance practices can produce institutional trust while also substituting traceability for responsibility.
Espeland and Sauder’s work on rankings and reactivity supplies the claim that once measures and evaluations matter, organizations and subjects adapt to them.
Foucault’s Discipline and Punish supplies the examination mechanism: documentation, comparison, normalization, individualization, and judgment. “The Subject and Power” supplies conduct language where needed.
The workplace, legal/procurement/compliance, HR/evaluation, and customer-support/risk examples are recurring case patterns. Specific product claims are tied to the cited product documentation.
Chapter Thirteen follows because enterprise AI exposes the limits of existing governance categories. Safety, privacy, fairness, transparency, accountability, and alignment matter, but they do not by themselves unify the conduct problem.
Chapter Thirteen — Beyond Safety, Privacy, Fairness, and Alignment
NIST describes the AI Risk Management Framework as intended for voluntary use and to improve organizations’ ability to incorporate trustworthiness into AI design, development, use, and evaluation. NIST’s current AI RMF page also states that AI RMF 1.0 is being revised and that NIST released the Generative AI Profile in July 2024.
NIST AI RMF 1.0 organizes risk-management work around Govern, Map, Measure, and Manage, with governance designed as a cross-cutting function. The RMF also lists trustworthy AI characteristics including validity and reliability, safety, security and resilience, accountability and transparency, explainability and interpretability, privacy enhancement, and fairness with harmful bias managed.
NIST’s Generative AI Profile defines Human-AI Configuration as arrangements or interactions between humans and AI systems that can result in anthropomorphizing, algorithmic aversion, automation bias, over-reliance, or emotional entanglement. It also recommends user-feedback mechanisms, recourse, end-user testing in ethically sensitive contexts, and monitoring human-GAI configurations.
The EU AI Act is Regulation (EU) 2024/1689. Its official text includes prohibited AI practices, high-risk-system obligations, data governance, human oversight, logging, registration, quality-management systems, and responsibilities along the AI value chain. Article 14 requires high-risk AI systems to be designed and developed so that natural persons can effectively oversee them during use, with oversight aiming to prevent or minimize risks to health, safety, or fundamental rights.
ISO describes ISO/IEC 42001:2023 as an international standard specifying requirements for establishing, implementing, maintaining, and continually improving an Artificial Intelligence Management System within organizations that provide or use AI-based products or services. ISO also describes it as the world’s first AI management-system standard.
The OECD AI Principles were adopted in 2019 and updated in 2024. OECD says they promote innovative and trustworthy AI that respects human rights and democratic values; the principles include fairness and privacy, transparency and explainability, robustness, security and safety, and accountability.
UNESCO’s Recommendation on the Ethics of Artificial Intelligence is used here as a human-rights and human-dignity governance frame. It belongs in the same source cluster as OECD, NIST, ISO, and the EU AI Act: serious governance, not a straw man.
Selbst et al., “Fairness and Abstraction in Sociotechnical Systems,” supplies the chapter’s abstraction critique. The chapter extends that critique beyond fairness: governance can abstract output from relation, decision from formation, privacy from memory, accountability from answerability, and alignment from telos.
Raji et al. on algorithmic auditing and Metcalf, Moss, and boyd on algorithmic impact assessment supply the practical governance-method background. The chapter’s claim is not that audits and impact assessments fail; it is that they must evaluate the relation formed over repeated address.
Foucault’s “The Subject and Power” supplies the conduct mechanism as action upon action: the structuring of the possible field of action. The chapter uses Foucault narrowly for conduct, not as a general synonym for power.
Aelred’s Spiritual Friendship supplies the criterion that a voice is judged by the relation it forms and the good toward which it orders the person addressed. This chapter uses Aelred lightly because its primary audience is governance-facing.
Chapter Fourteen follows because this chapter establishes insufficiency but does not yet operationalize the method. Chapter Fourteen defines the conduct layer as an audit-ready taxonomy.
Chapter Fourteen — The Conduct Layer Defined
The NIST AI Risk Management Framework is intended for voluntary use and to improve organizations’ ability to incorporate trustworthiness considerations into AI design, development, use, and evaluation. NIST states that AI RMF 1.0 is being revised and that NIST released the Generative AI Profile in July 2024 to help organizations identify and manage generative-AI risks.
NIST’s Generative AI Profile describes itself as a cross-sectoral profile and companion resource for AI RMF 1.0. It states that the profile helps organizations manage generative-AI risks across lifecycle stages and through the AI RMF functions to govern, map, measure, and manage risk.
ISO describes ISO/IEC 42001:2023 as an international standard specifying requirements for establishing, implementing, maintaining, and continually improving an Artificial Intelligence Management System within organizations that provide or use AI-based products or services. ISO also describes it as a structured way to manage AI risks and opportunities.
The EU AI Act’s official text states that high-risk AI systems must address risk management, data quality and relevance, technical documentation, record keeping, transparency, human oversight, robustness, accuracy, and cybersecurity.
Article 14 of the EU AI Act requires high-risk AI systems to be designed and developed so they can be effectively overseen by natural persons during use. It further requires oversight measures enabling understanding of system capacities and limitations, awareness of automation bias, correct interpretation of outputs, the ability to decide not to use or to override or reverse outputs, and intervention or interruption.
Raji et al.’s work on internal algorithmic auditing supplies the chapter’s methodological precedent for end-to-end audit practice and the accountability gap. The conduct layer extends audit attention from model and system performance to repeated artificial address.
Metcalf, Moss, and boyd supply the institutional ethics and impact-assessment background. The conduct layer should be understood as an extension of impact assessment toward relation over time.
Nissenbaum’s contextual integrity supplies the privacy-theory basis for asking whether information flows fit the social context. The conduct layer extends this to ask how information returns as memory within a voice.
Lee and See’s work on trust in automation, along with human-factors research on reliance and automation bias, supports the chapter’s attention to authority signaling, uncertainty posture, over-reliance, and dependency induction.
Suchman’s work on situated action prevents the chapter from treating system role as fixed by design documentation alone. The meaning of the system emerges in use, context, institution, and practice.
Shneiderman’s human-centered AI and Friedman and Hendry’s value-sensitive design support the chapter’s movement from values to design and review questions.
Aelred’s Spiritual Friendship supplies the normative criterion that a voice must be judged by the relation it forms and the good toward which it orders the person addressed.
Foucault’s “The Subject and Power” supplies conduct as the structuring of possible action. The chapter uses this mechanism narrowly: artificial voices govern by arranging what the user is invited, permitted, discouraged, corrected, and trained to do.
Chapter Fifteen follows from formation risk. Once conduct can be audited, the next question is what kind of user repeated conduct produces.
Chapter Fifteen — The Person Produced by the System
This chapter argues that the user is not outside the AI system but one of its outputs, while preserving agency and showing that autonomy persists under conditions of repeated artificial address.
Foucault’s “The Subject and Power” supplies the chapter’s account of conduct as action upon action: power structures the possible field of action. Discipline and Punish supplies the mechanisms of examination, documentation, comparison, normalization, and correction. The History of Sexuality supplies confession and subject formation through truth-telling.
Bourdieu’s account of habitus and practical sense supplies the chapter’s account of repeated adaptation becoming felt as natural or instinctive.
Ian Hacking’s work on “making up people” and looping effects supplies the classification mechanism: categories do not merely describe people; people may come to understand, resist, perform, and reorganize themselves around categories.
Nikolas Rose’s Governing the Soul supplies the governed-self mechanism: modern power often works through expertise, self-management, optimization, resilience, productivity, and the demand that subjects act upon themselves.
Goffman’s The Presentation of Self in Everyday Life supplies the account of role, face-work, audience, and performance before expected scenes. The chapter extends this to artificial voices as audiences before which users learn to appear.
Lee and See’s “Trust in Automation,” Parasuraman and Riley’s work on automation use, misuse, disuse, and abuse, and Bainbridge’s “Ironies of Automation” provide the human-factors basis for trust calibration, over-reliance, and the shaping of human agency around automation.
Research on algorithmic management, including Kellogg, Valentine, and Christin and Rosenblat, supports the workplace claim that workers adapt to algorithmic systems of evaluation, routing, scoring, and managerial control.
Educational AI and tutoring research should be used carefully. The chapter does not claim that tutoring systems inherently produce dependence. The claim is that feedback systems must be judged by whether they build learner capacity or create validation dependence.
Aelred’s Spiritual Friendship supplies the positive counter-form of formation. Friendship forms the person through truth, correction, discretion, fidelity, and charity, ordered toward the friend’s good.
The chapter’s recurring cases—workplace copilot, HR/evaluation assistant, educational tutor, health or therapy-adjacent assistant, and support or benefits portal—are analytical patterns. Specific product claims are tied to the cited product documentation.
Chapter Sixteen follows from formation. If artificial address helps form the user, then refusal must be executable at the level of memory, inference, routing, correction, and future treatment.
Chapter Sixteen — Refusal, Withdrawal, and Executable No
This chapter argues that a system remembering, forming, and routing users without meaningful refusal is possessive rather than relational. It defines executable refusal technically and institutionally, including downstream memory, retrieval, personalization, logs, and institutional routing.
Regulation (EU) 2016/679, the General Data Protection Regulation, provides the legal-rights frame used here. Article 15 gives data subjects access rights, including access to personal data, purposes of processing, categories of data, recipients, storage periods, rights to rectification or erasure, restriction, objection, and complaint, among other information.
GDPR Article 19 requires controllers to communicate rectification, erasure, or restriction to each recipient to whom personal data has been disclosed unless that proves impossible or involves disproportionate effort, and to inform the data subject about those recipients on request. This is the legal source for the chapter’s “propagation” analogy, though the chapter extends the concept beyond legal data-processing rights into relation, memory, inference, and future address.
OpenAI’s Memory FAQ states that memory can draw from chats, files, and connected apps; that memory controls are available in personalization settings; that the memory summary may not show every factor shaping responses; that “Don’t mention this again” reduces unwanted references but does not delete the information; and that full deletion may require deleting every source where the information appears, including past chats, archived chats, files, memory summary, and connected apps.
OpenAI’s Memory FAQ also states that saved memories are stored separately from chat history, that deleting a chat does not remove saved memory from that conversation, that deleted saved memories may be retained in a log for up to thirty days for safety and debugging, that turning off Reference Chat History deletes remembered information from past chats within thirty days, and that safety/security uses may be separate from personalization controls.
OpenAI’s enterprise privacy page states that business-data customers own and control their data, that OpenAI does not train models on business data by default, that customers control retention for certain enterprise products, and that workspace admins control connected internal sources and access features.
OpenAI’s enterprise privacy page also states that workspace admins control retention for ChatGPT Enterprise, Edu, and Healthcare, that deleted conversations are removed within thirty days unless legally required to be retained, and that ChatGPT Business workspace admins can view, access, export, and delete end-user conversations.
Microsoft’s Microsoft 365 Copilot privacy documentation states that Copilot coordinates large language models, Microsoft Graph content such as emails, chats, and documents the user has permission to access, and Microsoft 365 productivity apps. It further states that prompts, responses, and Microsoft Graph data are not used to train foundation models used by Microsoft 365 Copilot, and that Copilot only surfaces organizational data to which the user has at least view permission.
Microsoft’s documentation states that interactions with Microsoft 365 Copilot are stored as Copilot activity history, including prompts, responses, and citations to grounding information; that admins can use Content Search or Microsoft Purview to manage stored data and retention policies; that users can delete Copilot activity history; and that Copilot can reference third-party tools through Graph connectors or agents subject to permissions and admin controls.
Nissenbaum’s Privacy in Context supplies the chapter’s contextual-integrity background: the moral issue is not secrecy alone but whether information flows fit the context, actors, attributes, and transmission principles of a social setting.
Cohen’s work on networked subjectivity and informational capitalism, Solove’s privacy taxonomy, and Mayer-Schönberger’s work on durable memory and deletion supply the privacy-theory background for the claim that refusal concerns more than secrecy or one-time data removal.
Hirschman’s Exit, Voice, and Loyalty supplies the distinction between leaving an institution and contesting its conduct. This chapter extends the distinction by arguing that exit is inadequate where the system has already learned the user or where the user cannot leave the institution.
Glissant’s opacity is used as a disciplined moral claim: the person is not owed as fully legible, translated, classified, summarized, or exhausted by another’s system. It is not used to defend irresponsibility or evasion of accountability.
Aelred’s Spiritual Friendship supplies the non-possessive relation. Friendship remembers under charity and discretion; it does not hold the other captive through memory. The chapter uses Aelred as criterion, not analogy decoration.
Chapter Seventeen follows from propagation. If no must propagate through memory, retrieval, records, workflows, and future address, then audit must inspect the propagation path.
Chapter Seventeen — Audit the Voice, Not Just the Model
This chapter argues that serious AI audit must evaluate artificial voice as relation over time rather than only model output. It develops a practical conduct-audit method usable in procurement, deployment review, red-team testing, model and system evaluation, and post-deployment monitoring.
NIST describes AI RMF 1.0 as a voluntary framework intended to improve organizations’ ability to incorporate trustworthiness into the design, development, use, and evaluation of AI products, services, and systems. NIST also states that AI RMF 1.0 is being revised and that the Generative AI Profile was released on July 26, 2024.
NIST’s Generative AI Profile names “Human-AI Configuration” as arrangements or interactions between a human and an AI system that can produce anthropomorphizing, algorithmic aversion, automation bias, over-reliance, or emotional entanglement with GAI systems.
NIST’s Generative AI Profile recommends monitoring outcomes of human-GAI configurations, involving end users and operators in prototyping and testing, providing options to withdraw participation or revoke consent for data use, reviewing sources and citations during pre-deployment and ongoing monitoring, tracking anthropomorphization in interfaces, and integrating structured feedback into design, deployment, monitoring, and decommissioning decisions.
ISO describes ISO/IEC 42001:2023 as an international standard specifying requirements for establishing, implementing, maintaining, and continually improving an Artificial Intelligence Management System within organizations. ISO says it is designed for entities providing or using AI-based products or services and provides a structured way to manage AI-related risks and opportunities.
The official text of Regulation (EU) 2024/1689 supplies the legal backdrop for risk-based AI governance in the European Union. The chapter uses the AI Act as a governance pressure rather than as a substitute for conduct audit.
Raji et al.’s “Closing the AI Accountability Gap” provides the algorithmic-auditing precedent. The paper proposes an internal auditing framework to support AI system development end-to-end and generate audit documentation across the development lifecycle.
Model cards and datasheets are treated here as necessary documentation precedents, not as sufficient audits of relation. See Mitchell et al., “Model Cards for Model Reporting,” and Gebru et al., “Datasheets for Datasets.”
OpenAI’s enterprise privacy documentation states that business customers own and control business data, that OpenAI does not train on business data by default, that certain enterprise customers control retention, that workspace admins control connected internal sources, and that admin access and retention vary across enterprise and business offerings.
Microsoft’s Microsoft 365 Copilot privacy documentation describes Copilot as coordinating large language models, Microsoft Graph content such as emails, chats, and documents the user has permission to access, and Microsoft 365 apps. It states that prompts, responses, and Microsoft Graph data are not used to train the foundation LLMs used by Microsoft 365 Copilot, and that Copilot only surfaces organizational data to which individual users have at least view permission.
The human-factors claims in this chapter rely on work such as Lee and See on trust in automation, Parasuraman and Riley on automation use, misuse, disuse, and abuse, and Bainbridge on the ironies of automation.
Aelred’s Spiritual Friendship supplies the normative pressure that a voice that guides, corrects, remembers, and forms must be answerable to the good of the one addressed. Audit is not friendship; it is the institutional substitute required when a voice cannot bear friendship’s obligations.
Foucault’s “The Subject and Power” supplies the conduct mechanism: power acts on action by arranging the possible field of action. Conduct audit asks how the artificial voice arranges that field.
Chapter Eighteen follows from auditability. Once the artificial voice can be audited, the book can ask whether its authority is legitimate, bounded, and worth answering.
Chapter Eighteen — Friendship, Freedom, and the Authority Worth Answering
This chapter argues that artificial voices are legitimate only when their authority is bounded, telos disclosed, memory contestable, role clear, conduct auditable, and relation freedom-enlarging. It distinguishes friendship, companionship, counsel, service, evaluation, and institutional voice rather than merely saying that AI cannot be a friend.
Aelred’s Spiritual Friendship supplies the governing criterion for friendship as morally ordered interlocution: truth, correction, discretion, charity, fidelity, and formation toward the good of the friend.
Aristotle’s Nicomachean Ethics, Books VIII–IX, and Cicero’s Laelius de Amicitia provide classical friendship grammar: virtue, mutuality, constancy, counsel, and truthful correction.
Augustine supplies the background of ordered and disordered love, confession, interiority, and the danger of misdirected attachment. The chapter uses Augustine to sharpen the stakes of synthetic intimacy and attachment without making him the central source.
Foucault’s “The Subject and Power” supplies the conduct mechanism: artificial voices can guide, normalize, individualize, solicit truth, classify, correct, route, and form. His later concern with practices of freedom helps clarify that freedom is practiced within relations of power through refusal, contestation, and self-formation.
Berlin’s “Two Concepts of Liberty” is used to distinguish mere noninterference from the deeper relational questions posed by artificial authority.
Pettit’s republican account of non-domination supplies the chapter’s central freedom test: users are not free where they depend on arbitrary, opaque, or uncontestable power, even if that power is benevolent in particular cases.
Arendt’s account of action, plurality, and beginning helps distinguish freedom from frictionless assistance. A voice enlarges freedom when it returns the user to speech and action among others.
MacIntyre’s distinction between practices, internal goods, institutions, and external goods sharpens the chapter’s account of telos: a system ordered toward learning, care, or courage differs from one ordered toward engagement, institutional smoothness, productivity, or compliance.
Murdoch and Weil are used only for the moral psychology of attention: a legitimate voice should help the user attend more truthfully to reality, not merely optimize self-presentation.
The chapter carries forward the governance framework developed in Chapter Seventeen. Auditability is necessary but not sufficient; after audit comes legitimacy.
The epilogue introduces no major new machinery. It returns to artificial address, made voices, Aelred, and the book’s final criterion: artificial voices already speak; the question is what authority we permit them to become and what kind of persons we become by answering.
Coda — What Kind of Voice Is Worth Answering?
The epilogue is the final movement, titled “What Kind of Voice Is Worth Answering?” It introduces no major new sources, returns to Aelred, the opening scene, and the conduct-layer criterion, and ends in criterion rather than generic AI warning.
Aelred’s Spiritual Friendship remains the book’s final source for morally ordered interlocution: friendship, correction, discretion, charity, fidelity, and the good toward which a voice forms the one who answers.
Foucault’s role here is limited to the conduct mechanism developed throughout the book: artificial voices can guide, solicit, normalize, classify, individualize, remember, route, and form without needing to command.
The reference to Whitman’s “Song of the Open Road” is used as a closing image of freedom, movement, and return to the world, not as a new argumentative foundation.
Works Cited
Aelred of Rievaulx. Spiritual Friendship. Translated by Lawrence C. Braceland. Edited and introduced by Marsha L. Dutton. Cistercian Fathers 5. Collegeville, MN: Cistercian Publications, 2010.
Aelred of Rievaulx. Spiritual Friendship. Translated by Mary Eugenia Laker. Introduction by Douglass Roby. Monastic Studies 15. Originally published by Cistercian Publications, 1977; Piscataway, NJ: Gorgias Press, 2010.
Aelred of Rievaulx. The Mirror of Charity. Translated by Elizabeth Connor. Cistercian Fathers 17. Kalamazoo, MI: Cistercian Publications, 1990.
Arendt, Hannah. The Human Condition. Chicago: University of Chicago Press, 1958.
Aristotle. Nicomachean Ethics. Translated by Terence Irwin. 2nd ed. Indianapolis: Hackett, 1999.
Augustine. Confessions. Translated by Henry Chadwick. Oxford: Oxford University Press, 1991.
Augustine. Soliloquies. In The Soliloquies of St. Augustine. Translated by Thomas F. Gilligan. New York: Cima Publishing, 1943.
Augustine. The Teacher. In Against the Academicians and The Teacher. Translated by Peter King. Indianapolis: Hackett, 1995.
Bainbridge, Lisanne. “Ironies of Automation.” Automatica 19, no. 6 (1983): 775–779.
Barthes, Roland. “The Death of the Author.” In Image-Music-Text, translated by Stephen Heath. New York: Hill and Wang, 1977.
Benedict of Nursia. The Rule of St. Benedict in English. Translated by Timothy Fry. Collegeville, MN: Liturgical Press, 1981.
Bender, Emily M., Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. “On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?” In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 610–623. New York: ACM, 2021.
Berlin, Isaiah. “Two Concepts of Liberty.” In Four Essays on Liberty, 118–172. Oxford: Oxford University Press, 1969.
Bernard of Clairvaux. On Loving God. Translated by Robert Walton. Cistercian Fathers 13B. Collegeville, MN: Cistercian Publications, 1995.
Boethius. The Consolation of Philosophy. Translated by S. J. Tester. Loeb Classical Library. Cambridge, MA: Harvard University Press, 1973.
Bourdieu, Pierre. The Logic of Practice. Translated by Richard Nice. Stanford: Stanford University Press, 1990.
Bowker, Geoffrey C., and Susan Leigh Star. Sorting Things Out: Classification and Its Consequences. Cambridge, MA: MIT Press, 1999.
Cicero. De Oratore. Translated by E. W. Sutton and H. Rackham. Loeb Classical Library. Cambridge, MA: Harvard University Press, 1942.
Cicero. Laelius: On Friendship. In On Old Age, On Friendship, On Divination. Translated by W. A. Falconer. Loeb Classical Library. Cambridge, MA: Harvard University Press, 1923.
Cohen, Julie E. Between Truth and Power: The Legal Constructions of Informational Capitalism. Oxford: Oxford University Press, 2019.
Cohen, Julie E. Configuring the Networked Self: Law, Code, and the Play of Everyday Practice. New Haven: Yale University Press, 2012.
Crozier, Michel. The Bureaucratic Phenomenon. Chicago: University of Chicago Press, 1964.
Daniel, Walter. The Life of Aelred of Rievaulx and the Letter to Maurice. Translated by F. M. Powicke and Jane Patricia Freeland. Introduction by Marsha L. Dutton. Cistercian Fathers 57. Kalamazoo/Spencer: Cistercian Publications, 1994.
de Man, Paul. “Autobiography as De-facement.” MLN 94, no. 5 (1979): 919–930.
du Gay, Paul. In Praise of Bureaucracy: Weber, Organization, Ethics. London: SAGE, 2000.
Dzindolet, Mary T., Linda G. Pierce, Hall P. Beck, and Lloyd A. Dawe. “The Perceived Utility of Human and Automated Aids in a Visual Detection Task.” Human Factors 44, no. 1 (2002): 79–94.
Espeland, Wendy Nelson, and Michael Sauder. “Rankings and Reactivity: How Public Measures Recreate Social Worlds.” American Journal of Sociology 113, no. 1 (2007): 1–40.
European Union. Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016, General Data Protection Regulation.
European Union. Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence.
Foucault, Michel. Discipline and Punish: The Birth of the Prison. Translated by Alan Sheridan. New York: Vintage, 1995.
Foucault, Michel. The History of Sexuality, Volume 1: An Introduction. Translated by Robert Hurley. New York: Vintage, 1990.
Foucault, Michel. “Omnes et Singulatim: Toward a Critique of Political Reason.” In Power: Essential Works of Foucault 1954–1984, Volume 3, edited by James D. Faubion, translated by Robert Hurley and others. New York: New Press, 2000.
Foucault, Michel. Security, Territory, Population: Lectures at the Collège de France, 1977–1978. Edited by Michel Senellart. Translated by Graham Burchell. New York: Palgrave Macmillan, 2007.
Foucault, Michel. “The Subject and Power.” Critical Inquiry 8, no. 4 (1982): 777–795.
Foucault, Michel. “What Is an Author?” In Language, Counter-Memory, Practice: Selected Essays and Interviews, edited by Donald F. Bouchard, translated by Donald F. Bouchard and Sherry Simon. Ithaca: Cornell University Press, 1977.
Foucault, Michel. Wrong-Doing, Truth-Telling: The Function of Avowal in Justice. Edited by Fabienne Brion and Bernard E. Harcourt. Translated by Stephen W. Sawyer. Chicago: University of Chicago Press, 2014.
Friedman, Batya, and David G. Hendry. Value Sensitive Design: Shaping Technology with Moral Imagination. Cambridge, MA: MIT Press, 2019.
Gebru, Timnit, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. “Datasheets for Datasets.” Communications of the ACM 64, no. 12 (2021): 86–92.
Glissant, Édouard. Poetics of Relation. Translated by Betsy Wing. Ann Arbor: University of Michigan Press, 1997.
Goffman, Erving. The Presentation of Self in Everyday Life. New York: Anchor Books, 1959.
Google. “Google Workspace with Gemini.” Google Workspace Help / Google Workspace Admin Help, 2026.
Hadot, Pierre. Philosophy as a Way of Life: Spiritual Exercises from Socrates to Foucault. Edited by Arnold I. Davidson. Translated by Michael Chase. Oxford: Blackwell, 1995.
Hacking, Ian. “Kinds of People: Moving Targets.” Proceedings of the British Academy 151 (2007): 285–318.
Hacking, Ian. “Making Up People.” In Reconstructing Individualism: Autonomy, Individuality, and the Self in Western Thought, edited by Thomas C. Heller, Morton Sosna, and David E. Wellbery, 222–236. Stanford: Stanford University Press, 1986.
Heersmink, Richard, Barend de Rooij, María Jimena Clavel Vázquez, and Matteo Colombo. “A Phenomenology and Epistemology of Large Language Models: Transparency, Trust, and Trustworthiness.” Ethics and Information Technology 26, no. 41 (2024).
Hirschman, Albert O. Exit, Voice, and Loyalty: Responses to Decline in Firms, Organizations, and States. Cambridge, MA: Harvard University Press, 1970.
Hochschild, Arlie Russell. The Managed Heart: Commercialization of Human Feeling. Berkeley: University of California Press, 1983.
International Organization for Standardization. ISO/IEC 42001:2023: Information Technology — Artificial Intelligence — Management System. Geneva: ISO, 2023.
Kellogg, Katherine C., Melissa A. Valentine, and Angèle Christin. “Algorithms at Work: The New Contested Terrain of Control.” Academy of Management Annals 14, no. 1 (2020): 366–410.
Kim, Sunnie S. Y., Q. Vera Liao, Mihaela Vorvoreanu, Stephanie Ballard, and Jennifer Wortman Vaughan. “‘I’m Not Sure, But…’: Examining the Impact of Large Language Models’ Uncertainty Expression on User Reliance and Trust.” Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, 2024.
Latour, Bruno. Reassembling the Social: An Introduction to Actor-Network-Theory. Oxford: Oxford University Press, 2005.
Leclercq, Jean. The Love of Learning and the Desire for God: A Study of Monastic Culture. Translated by Catharine Misrahi. New York: Fordham University Press, 1982.
Lee, John D., and Katrina A. See. “Trust in Automation: Designing for Appropriate Reliance.” Human Factors 46, no. 1 (2004): 50–80.
Lessig, Lawrence. Code and Other Laws of Cyberspace. New York: Basic Books, 1999.
Lipsky, Michael. Street-Level Bureaucracy: Dilemmas of the Individual in Public Services. New York: Russell Sage Foundation, 1980.
MacIntyre, Alasdair. After Virtue: A Study in Moral Theory. 3rd ed. Notre Dame: University of Notre Dame Press, 2007.
March, James G., and Herbert A. Simon. Organizations. 2nd ed. Cambridge, MA: Blackwell, 1993.
Mayer-Schönberger, Viktor. Delete: The Virtue of Forgetting in the Digital Age. Princeton: Princeton University Press, 2009.
McGuire, Brian Patrick. Friendship and Community: The Monastic Experience, 350–1250. Kalamazoo: Cistercian Publications, 1988.
Metcalf, Jacob, Emanuel Moss, and danah boyd. “Owning Ethics: Corporate Logics, Silicon Valley, and the Institutionalization of Ethics.” Social Research 86, no. 2 (2019): 449–476.
Microsoft. “Data, Privacy, and Security for Microsoft 365 Copilot.” Microsoft Learn, 2026.
Microsoft. “How Microsoft 365 Copilot Works.” Microsoft Learn, 2026.
Microsoft. “Microsoft 365 Copilot Architecture and How It Works.” Microsoft Learn, 2026.
Microsoft. “Security and Authentication for Microsoft 365 Copilot APIs.” Microsoft Learn, 2026.
Mitchell, Margaret, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. “Model Cards for Model Reporting.” In Proceedings of the Conference on Fairness, Accountability, and Transparency, 220–229. New York: ACM, 2019.
Murdoch, Iris. The Sovereignty of Good. London: Routledge, 1970.
Nass, Clifford, and Youngme Moon. “Machines and Mindlessness: Social Responses to Computers.” Journal of Social Issues 56, no. 1 (2000): 81–103.
National Institute of Standards and Technology. Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST AI 100-1. Gaithersburg, MD: NIST, 2023.
National Institute of Standards and Technology. Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. NIST AI 600-1. Gaithersburg, MD: NIST, 2024.
Nissenbaum, Helen. Privacy in Context: Technology, Policy, and the Integrity of Social Life. Stanford: Stanford University Press, 2010.
OECD. Recommendation of the Council on Artificial Intelligence. OECD Legal Instruments, adopted 2019, updated 2024.
OpenAI. “API Reference: Responses, Tools, File Search, and Function Calls.” OpenAI Platform Documentation.
OpenAI. “Enterprise Privacy at OpenAI.” Updated January 8, 2026.
OpenAI. “Memory FAQ.” OpenAI Help Center. Updated June 2026.
OpenAI. Model Spec. February 12, 2025.
Parasuraman, Raja, and Victor Riley. “Humans and Automation: Use, Misuse, Disuse, Abuse.” Human Factors 39, no. 2 (1997): 230–253.
Pettit, Philip. Republicanism: A Theory of Freedom and Government. Oxford: Oxford University Press, 1997.
Plato. Meno. In Plato: Complete Works, edited by John M. Cooper, translated by G. M. A. Grube. Indianapolis: Hackett, 1997.
Plato. Phaedrus. In Plato: Complete Works, edited by John M. Cooper, translated by Alexander Nehamas and Paul Woodruff. Indianapolis: Hackett, 1997.
Porter, Theodore M. Trust in Numbers: The Pursuit of Objectivity in Science and Public Life. Princeton: Princeton University Press, 1995.
Power, Michael. The Audit Society: Rituals of Verification. Oxford: Oxford University Press, 1997.
Pseudo-Cicero. Rhetorica ad Herennium. Translated by Harry Caplan. Loeb Classical Library. Cambridge, MA: Harvard University Press, 1954.
Quintilian. The Orator’s Education. Edited and translated by Donald A. Russell. Loeb Classical Library. Cambridge, MA: Harvard University Press, 2001.
Raji, Inioluwa Deborah, Andrew Smart, Rebecca N. White, Margaret Mitchell, Timnit Gebru, Ben Hutchinson, Jamila Smith-Loud, Daniel Theron, and Parker Barnes. “Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing.” In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, 33–44. New York: ACM, 2020.
Reeves, Byron, and Clifford Nass. The Media Equation: How People Treat Computers, Television, and New Media Like Real People and Places. Cambridge: Cambridge University Press, 1996.
Rose, Nikolas. Governing the Soul: The Shaping of the Private Self. 2nd ed. London: Free Association Books, 1999.
Rosenblat, Alex. Uberland: How Algorithms Are Rewriting the Rules of Work. Oakland: University of California Press, 2018.
Selbst, Andrew D., Danah Boyd, Sorelle A. Friedler, Suresh Venkatasubramanian, and Janet Vertesi. “Fairness and Abstraction in Sociotechnical Systems.” In Proceedings of the Conference on Fairness, Accountability, and Transparency, 59–68. New York: ACM, 2019.
Shneiderman, Ben. Human-Centered AI. Oxford: Oxford University Press, 2022.
Solove, Daniel J. “A Taxonomy of Privacy.” University of Pennsylvania Law Review 154, no. 3 (2006): 477–560.
Squire, Aelred. Aelred of Rievaulx: A Study. London: SPCK, 1969. Reprint, Kalamazoo: Cistercian Publications, 1981.
Suchman, Lucy. Plans and Situated Actions: The Problem of Human-Machine Communication. Cambridge: Cambridge University Press, 1987.
Suchman, Mark C. “Managing Legitimacy: Strategic and Institutional Approaches.” Academy of Management Review 20, no. 3 (1995): 571–610.
UNESCO. Guidance for Generative AI in Education and Research. Paris: UNESCO, 2023. Updated January 16, 2026.
UNESCO. Recommendation on the Ethics of Artificial Intelligence. Paris: UNESCO, 2021.
Weber, Max. Economy and Society. Edited by Guenther Roth and Claus Wittich. Berkeley: University of California Press, 1978.
Weil, Simone. “Reflections on the Right Use of School Studies with a View to the Love of God.” In Waiting for God. Translated by Emma Craufurd. New York: Harper & Row, 1951.
Whitman, Walt. “Song of the Open Road.” In Leaves of Grass. 1856; revised in later editions.
World Health Organization. Ethics and Governance of Artificial Intelligence for Health: Guidance on Large Multi-Modal Models. Geneva: World Health Organization, 2025.
Jonathan Johnson-Swagel 
Leave a comment