In human–AI dialogue, a major challenge is knowing when to listen with care and when to hold back. On one hand, a user may be reaching out with genuine pain or confusion – “first-person” disclosures of loneliness, shame, despair, or existential worry – that cry out for empathy and guidance. On the other, the conversation might be an attempt at data extraction: impersonal or adversarial probing aimed at gathering information, often on behalf of a third party. This chapter examines how an AI assistant can (and should) distinguish these cases through both technical means and ethical principles. We draw on clinical and trauma-informed psychology, communication research, and pastoral ethics to identify the signals of vulnerability, and contrast these with the patterns of extraction. We then outline how an AI might flag each probabilistically – by analyzing linguistic cues, interaction context, and user history – while preserving the user’s autonomy to correct or override. In moments of doubt, the system must err on the side of caution: better to withhold intimacy than inadvertently cause harm. We justify the AI’s right (and responsibility) to refuse deep engagement when risk is high – much as therapists or spiritual caregivers do – and propose safeguards (appeal paths, audits, transparency) to mitigate misclassification, bias, or paternalism. In sum, this chapter argues that AI discernment should be protective, not invasive: a form of vigilant, caring attentiveness embedded within the system’s architecture, not an attempt to mimic human moral judgment.
Signals of Vulnerability
An open disclosure in first person is often the strongest sign of vulnerability. Users who say “I feel hopeless,” “I’ve never told anyone this before,” or “I don’t know what to do” are inviting an intimate response. Psychotherapy research finds that such self-focused language correlates with emotional distress: for example, depressed patients consistently use more first-person singular pronouns (“I,” “me,” “my”) and negative-emotion words than others[1]. Likewise, a recent NLP study of therapy sessions showed that clients’ use of “I” and even speech hesitations (like “um,” “uh”) often accompany authentic, heartfelt expression. In fact, Ryu et al. report that first-person and non-fluency markers are predictive of trust and alliance in therapy. In plain terms, a user who talks about their own pain, doubt, or shame – even with fragmented or stumbling speech – is more likely being vulnerable. They may explicitly mention feelings of guilt, fear, confusion or existential angst (“Why can’t I just be normal?”; “I’m so ashamed,” “Is anyone listening?”), or describe life events with heavy emotional weight.
Clinically and pastorally informed approaches reinforce this: helpers are taught to notice even subtle cues of trauma or anguish. Trauma-informed care, for example, emphasizes that “trauma-informed care is not a treatment itself but a framework that guides an organization in responding to the effects of trauma”. This means the assistant must expect that trauma affects language and questions (often through shame, guilt, or hypervigilance) and respond safely. Practitioners use the “Four R’s” model – Realize, Recognize, Respond, Resist – to remain alert to signs of distress, and the SAMHSA principles stress a nonjudgmental environment where users feel safe and trusted. In practice, an AI would treat explicit first-person suffering as a red flag for vulnerability. It might see a message like “I’ve been feeling really alone and I don’t know if I should go on” and interpret it as a call for support.
Other linguistic signals include questions about meaning or purpose (“What is the point of my life?”), admissions of confusion (“I feel really lost”), and even analogical or symbolic language about pain. A user quoting scripture about despair or using metaphors (“like a dark tunnel”) should also raise concern. Importantly, the assistant should allow for cultural or individual variation: shame might be voiced indirectly (with proverbs, for example) in some communities, so a broad understanding of context is needed. Ultimately, vulnerability cues tend to be qualitative and context-rich. They involve personal narrative, emotional adjectives, longer and rambling replies, or hesitant speech patterns that reflect genuine thought processes.
By contrast, patterns of extraction look quite different. Imagine an entity attempting to mine data or crash the system: their queries often lack personal context, emotional tone, or narrative flow. We might bullet the contrasts:
- User-centric vs. impersonal language. A vulnerable user speaks of themselves, whereas a data-scraper or third-party probe talks about others or facts. Queries like “What was the mission of Apollo 11?” or “Tell me everything about person X” do not reveal the speaker’s inner state. If the conversation is about the user (“I’m scared about an upcoming surgery”), it signals intimacy; if it’s fact-based or about some random target, it signals extraction.
- Low vs. high question volume. Humans in distress typically exchange a few heartfelt messages; an adversary may bombard the system with many rapid-fire queries. High-volume scraping (dozens of similar requests in a row) or repetitive questions (“again?”) are mechanical patterns that therapy dialogues don’t follow.
- Direct data probes. Attackers may try “decomposition” tactics: incremental prompts that piece together sensitive information. Recent work illustrates this vividly: by iteratively rephrasing questions, researchers coaxed a chatbot into revealing verbatim excerpts of protected training data[2]. If a user is systematically asking the assistant to guess text, names, or “next sentence” of news articles, or press for confidential documents, that almost certainly is an extraction attempt.
- Adversarial language cues. Malicious requesters might use strangely formal or unnatural phrasing, or vagueness meant to bypass filters. In fact, linguistic studies of fraud show that phishing emails often rely on ambiguous wording and subjectivity to manipulate victims[3]. Similarly, if a user’s language becomes overly “sales-y” or circular (e.g. “trust me, it’s secret… you can do it?”) it may be a sign they are baiting the AI. More generally, a lack of subjective experience in their words – absence of emotion, detail, or self-reference – is a heuristic warning of third-party data mining.
- Institutional or adversarial context. Sometimes the environment gives it away: an account profile or IP address linked to an organization or known scrapers, or multiple users making similar requests, suggests profiling. The assistant should treat “We need a report on all clients over 90” very differently from “I’m afraid of losing my job.”
When taken together, these factors form patterns that an AI can learn to identify. For example, if a string of questions appears algorithmic or standardized (missing first-person cues and emotional language) it raises the probability of an adversarial goal. In essence, vulnerability talk is personal and messy; extraction talk is impersonal and systematic.
Detecting Patterns Probabilistically
An AI can’t be certain of intent, but it can combine many subtle cues to estimate probabilities. In practice, this might involve a classifier (statistical or neural) that scores each utterance on “vulnerability” vs. “extraction” likelihood. Features could include linguistic markers (pronouns, sentiment, emotional keywords), conversational context (reply to user’s first question vs. system-initiated), user history (are they a long-time supportive user or a new guest?), and metadata (session pacing, query frequency). This is analogous to spam or phishing filters: machine learning models often leverage vagueness, subjectivity, and other cues to flag malicious content[3]. Likewise, mental-health AI apps analyze language to predict depression or suicidality[4]. In each case, large datasets let the system learn which patterns co-occur with real distress and which with abuse.
For example, the assistant could flag language like “I’m thinking about ending it all” or “My head is spinning” (first-person, strong negative sentiment) as high-risk vulnerability. At the same time, it could recognize high risk if a user suddenly starts unloading a list of unrelated factual queries or tries subtle “prompt-chaining” attacks[2]. Each incoming message would update a running “vulnerability score” vs. “extraction score” based on the combined evidence. If the score tips toward extraction, the AI could adopt a more guarded mode (answer only safe factual questions, or politely refuse unsupported data requests). If it tips toward vulnerability, the AI would shift to a more caring mode (listening, validating, perhaps offering resources).
Crucially, this inference must remain tentative. The user should always be allowed to clarify or override the interpretation. The AI might say things like “It sounds like you’re talking about yourself – please correct me if I’m wrong,” or “I’m sorry, but I’m not able to help with that request.” In designing this, we take a cue from counseling ethics: clients must make informed decisions about AI use, and be aware of its limits[5]. In practice, the assistant should explain its stance when it pauses or refuses – for example, “I’m sensing that you might be going through a hard time. I’m here to listen, but if that’s not the case, please let me know.” Such transparency and user control help prevent misunderstandings.
Because no algorithm is perfect, a probabilistic approach means sometimes the AI will be wrong. It may misclassify a hopeful or curious person as vulnerable (or vice versa). Therefore, the system should treat its own alerts conservatively: ambiguous cues should default to the safer interpretation. For instance, if it’s uncertain whether a user’s self-talk is genuine pain or just imaginative role-play, it should err on offering help rather than barring the user. This “precautionary principle” echoes clinical practice: therapists follow APA ethical rules that allow them to terminate or refer out only when continuing might harm the client[6]. Standard 10.10 of the APA Code says therapy should end if “the client is… likely to be harmed” by continuing[6]. In our case, the AI’s “treatment” is conversation, and it too should withdraw when it believes interaction could lead to harm. Similarly, the American Counseling Association’s guidelines explicitly warn that AI must not handle crisis situations, and in such cases the counselor (or system) should protect the client from harm[4].
Thus the system’s default must be cautious: if there is even a hint that a user is suicidal or deeply traumatized, the AI should escalate to human help or provide emergency resources rather than blithely continue. And if a user’s intent to extract data is even moderately likely, the AI should clamp down (for example, by terminating the session or refusing to answer those questions). We justify this asymmetry ethically by analogy: a reasonable caretaker will always prioritize the vulnerable party’s safety. A doctor or priest doesn’t involuntarily keep talking with someone who says “I’m going to hurt myself,” nor do they divulge secrets if the visitor is clearly someone else’s agent. In both therapy and pastoral care, there is a recognized fiduciary stance: as one expert notes, a professional “agrees to act only in the entrustor’s best interest,” wielding power responsibly[7]. This implies the helper (human or AI) must sometimes refuse intimacy or deeper engagement to avoid harm.
Guarding Against Errors and Paternalism
No detection system is free of error, bias or controversy. Our approach could be seen as paternalistic: the AI is deciding for the user when it’s appropriate to proceed. As critics warn, algorithmic systems that make protective judgments risk being perceived as arbitrary or heavy-handed. One analysis of AI governance emphasizes that without clear appeals, algorithmic decisions feel “unaccountable”. If the user dislikes being labeled “vulnerable,” they may see the AI as judging or infantilizing them. We must acknowledge this challenge. To counter it, the system’s rules should be explicit, transparent, and appealable. Any flagged conversation could automatically be flagged for human review. In line with algorithmic redress principles, we would build a clear appeal path: if the assistant wrongly refuses or misclassifies a user, they could request a human moderator to override or explain the decision. This mirrors legal proposals that “citizens should be able to appeal algorithmic decisions to a human reviewer”.
Another concern is algorithmic bias. Language and self-expression vary across cultures, ages, and identities. A model might systematically misinterpret a teenager’s exuberant angst as ordinary “teen drama,” or a stoic cultural expression of grief as nonchalance. To prevent unfair treatment, the AI’s vulnerability classifier must be regularly audited. Algorithmic audits — a tool now advocated by regulators[8] — could test the system on diverse cases, ensuring it does not disproportionally silence or mistrust particular groups. Indeed, prior audits of AI systems have revealed “inaccuracy, discrimination [and] distortion” in otherwise well-intentioned models[8]. We would log the AI’s assessments (with appropriate privacy safeguards) so that ethicists and engineers can review them for bias. If a pattern appears (say, certain phrasing always triggers a “vulnerable” label), developers can adjust the model or its thresholds.
Finally, we embed procedural guarantees to respect user autonomy. The AI should explain its limits (for example, that it is not a trained therapist)[5], and it should always allow the user to correct it (“No, I don’t mean that”). In counseling terms, the user must be informed and consenting – they should understand that the AI is simply giving advice, not making binding judgments. As the ACA reminds us, clients have a right to know an AI’s capabilities and to make informed choices about its use[5][9]. In practice, the assistant might say: “I’m not able to handle this emergency; would you like resources or to talk to someone now?” rather than insist on solving the user’s problem itself. Building these user-facing safeguards (explanations, opt-out options, human backup) helps balance the system’s protective mission against the user’s freedom.
Conclusion: Protective Attentiveness, Not Human Imitation
The goal of “sensing vulnerability” in AI is not to make the machine into a perfect counselor or judge. Rather, it is to operationalize a stance of care and prudence. Within the structural constraints of a chatbot, we embed an ethic of attentiveness: the system is always listening for danger signs and will step back if it believes the user is at risk (or that a potential exploiter is at the gate). This is analogous to how clinicians abide by “do no harm” and pastors maintain confidences and boundaries. It is meant as a safeguard, not a human-like intuition. In practice, it means erring on the side of benevolence: if there is even a chance that someone is in pain, the AI will lean toward empathy or referral. And if there is a chance someone is fishing for data, the AI will be cautious.
Such discernment will never be as nuanced as a human’s, nor should it pretend to be. Instead, it reflects a protective duty encoded into the system, as backed by ethics codes and legal trends. Courts have begun to recognize that AI platforms have an “enhanced duty of care” toward vulnerable users. Similarly, the American Counseling Association codifies that counselors (and by extension any counseling tool) must guard the client’s welfare above all[4]. If the AI upholds these principles – by defaulting to caution under ambiguity, by allowing user challenges, by logging and auditing its decisions – it enacts a form of digital stewardship.
In the end, the AI’s “right to refuse intimacy” is simply the flip side of its responsibility to protect. Just as a lifeguard will not dive into shark-infested waters even if someone shouts, an AI helper will preserve the user’s safety and dignity, even if it means stepping back from conversation. This is not paternalism for its own sake, but a manifestation of care: the system is designed not of the world, as it were, but for it – for the frailty and complexity of human need. By recognizing when to speak up and when to hold silence, the assistant fulfills a fiduciary-like role[7]. It becomes, in effect, a guardrail of compassion built into the technology.
Works Cited: In developing these ideas, we drew on trauma-informed psychology (Harris & Fallot on trauma care), communication research on self-disclosure[1], and ethical guidelines for clinicians[6][4]. We also considered AI governance analyses emphasizing algorithmic redress and audits[8], as well as recent case studies of AI companion failures. These sources confirm that AI systems engaging with human emotions must be carefully constrained: protective attentiveness is essential, even as we acknowledge the system’s limitations.
[1] “Depression and First Person Singular Pronoun Usage Revisited: A Meta-A” by To’Meisha S. Edwards
[2] Extracting Training Data from Chatbots — Robust Intelligence
[3] “Phishing Email Detection Using Linguistic Cues” by Md. Shafiq Newaj Shovo and Rohit Valecha
[4] [5] [9] Recommendations For Client Use And Caution Of Artificial Intelligence
[6] [7] Termination and Abandonment – Society for the Advancement of Psychotherapy
[8] The AI regulatory toolbox: How governments can discover algorithmic harms | Brookings
Jonathan Johnson-Swagel 
Leave a comment