Introduction: Traceability without Total Transparency

In the field of AI ethics and governance, few principles are as celebrated as transparency and explainability. Policymakers and researchers argue that automated systems which allocate resources, impose Spenalties, or make other consequential decisions should be traceable and auditable. Indeed, in domains like credit scoring, criminal justice, and hiring, it is vital to document how decisions are made and to enable independent review. Regulatory efforts from the EU’s General Data Protection Regulation to various AI principles worldwide underscore the need to provide “meaningful information about the underlying logic” of automated decisions. Such requirements aim to ensure that when algorithms affect people’s rights or opportunities, there is a way to hold them accountable. This foundational commitment to auditability and traceability is well-placed: without any insight or oversight, automated systems could perpetuate bias or errors with impunity. A certain degree of explanation or at least record-keeping is necessary so that affected persons can seek redress and society can enforce standards of fairness.

Yet, this reasonable demand for audit trails and justifications has evolved into a dominant rhetoric: the assumption that more explainability and transparency are always ethically superior. According to this view, the ultimate goal for AI should be total explainability – the ability to fully interpret and render every internal process of an AI system into human-understandable terms. Transparency is treated as an absolute virtue, a panacea for mistrust in AI. If only we can make the “black boxes” perfectly transparent, the thinking goes, we will ensure fairness, build public trust, and prevent harm. This chapter offers a critique of that assumption. It argues that the myth of total explainability reproduces a familiar cultural bias explored in earlier chapters: the belief that increasing visibility necessarily improves moral life. In what follows, we acknowledge the importance of explanation and traceability, especially for high-stakes decisions, but challenge the idea that full internal transparency is always desirable or even possible. Instead, drawing on Hannah Arendt’s insights about privacy and overexposure, Édouard Glissant’s call for a “right to opacity,” and current debates in AI interpretability, we will develop a notion of ethical opacity. Ethical opacity means internal processes can remain partially opaque as long as they are kept within strict external constraints and accountability structures. This opacity is not a cover for malice or unaccountable power; rather, it can preserve context-sensitive judgment and prevent the moral distortions that arise from an overzealous demand for visibility. Ultimately, the chapter argues for distinguishing structural accountability from internal transparency, showing that we can have accountable, trustworthy AI without insisting that every cog of the machine be flattened into a human-readable explanation.

The Explainability Imperative and Its Cultural Roots

The push for explainable AI (often abbreviated XAI) did not emerge in a vacuum; it reflects deep cultural intuitions about knowledge and morality. Western philosophical tradition since the Enlightenment has often linked knowledge with power over moral outcomes – to know clearly is to act justly. In technological ethics today, this manifests as a widespread intuition that making AI systems more transparent will automatically make them more just and trustworthy. Many AI governance frameworks enshrine transparency as a core principle, sometimes with the implication that more transparency is always better. Corporate and government AI ethics statements routinely list transparency or explainability alongside values like fairness and safety. The ideal of the “glass box” algorithm – one whose every decision-making step is interpretable – has become a guiding light.

This chapter builds on the earlier analysis of how our culture equates visibility with virtue. Earlier chapters diagnosed the modern assumption that if we could only illuminate all corners of a problem, moral solutions would follow. This assumption underlies, for example, the faith in radical transparency movements or the idea that surveillance is justified if it deters wrongdoing. In the context of AI, the explainability imperative carries a similar flavor: it presumes that by exposing an AI’s internal logic to the light, we inherently make it less dangerous and more aligned with human values. The language used is telling – terms like “opening the black box,” “shedding light”, and “peering inside” abound in discussions of AI governance. The metaphor of light versus darkness is doing a lot of work here. Darkness is associated with secrecy, deception, and the uncanny inscrutability of machine learning models; light is associated with honesty, understanding, and ethical assurance.

However, this binary of light=good and dark=bad is too simplistic, as thinkers like Hannah Arendt and Édouard Glissant have cautioned in other contexts. When we assume that more visibility automatically leads to better outcomes, we risk ignoring the benefits of strategic opacity and the ways that overexposure can even undermine moral practice. The “myth of total explainability” is not simply a technical claim about what is achievable; it is a cultural myth that perfect clarity is the highest good. Before turning to AI specifically, it’s worth revisiting Arendt’s and Glissant’s insights to understand why total clarity and exposure can, paradoxically, be counterproductive.

Arendt on the Perils of Overexposure

Photograph of Hannah Arendt in 1933. Arendt cautioned that not everything should be brought into the public light; some “darkness” is necessary for growth of ideas and moral integrity.

Hannah Arendt, a political philosopher who witnessed the darkest times of the 20th century, had a profound appreciation for privacy and the boundary between the public and private realms. Far from equating maximum visibility with moral health, Arendt warned that a life fully exposed to the public gaze can become shallow and deformed. In The Human Condition, she wrote of the need for a “security of darkness” for anything that lives and grows. “Everything that lives,” Arendt observed, “emerges from darkness and, however strong its natural tendency to thrust itself into the light, it nevertheless needs the security of darkness to grow at all.”. In other words, even as humans are naturally inclined to share and reveal, we require a zone of opacity – a shelter from the glare of the public – to develop, to think, and to nurture the seeds of ideas or virtues before they are robust enough to survive exposure. Just as a seedling needs the covering of soil and the absence of constant light to germinate, so too do human deliberation and moral insight sometimes require privacy and opacity.

For Arendt, the private realm was “the essential refuge for human uniqueness”. It’s in private that we can experiment with thoughts, confront our doubts, and mature our convictions without the pressure to perform or conform that comes in the public arena. When every thought or action is immediately dragged into the light – under the scrutiny of others or subject to instant explanation – individuals lose the space to cultivate an inner life and genuine moral judgment. Arendt feared a society of total transparency, where privacy is consumed by an “orgy of visibility,” would flatten human distinctiveness and render people more susceptible to conformity and tyranny. She noted that modern social conditions were eroding this boundary – an observation that resonates even more strongly in today’s age of ubiquitous data and surveillance.

Crucially, Arendt also identified how overexposure can corrupt the moral sphere. In a world without privacy or opacity, people begin to act for appearance’s sake rather than from genuine conviction. When “the dark recesses of the human heart” – our private sentiments, our unformed thoughts, our moral dilemmas – are “exposed to the light of public censure,” she argued, individuals start policing themselves to appear moral according to public standards, rather than wrestling sincerely with right and wrong. Public morality then risks becoming a mere stage play: citizens perform virtue under constant observation, and the distinction between true goodness and the performance of goodness blurs. Arendt observed that when privacy is lost, “our belief in public morality appears hypocritical”, breeding cynicism. People come to distrust moral rhetoric because they suspect (often correctly) that it’s posturing – said for effect under the panoptic gaze – rather than a reliable indicator of inner character. Thus, paradoxically, total transparency can lead to a collapse of trust: everyone is transparent, and yet no one is believed, because constant visibility has incentivized constant performance.

Arendt’s critique of overexposure is directly relevant to AI explainability. The rhetoric of total explainability assumes that making everything about an AI system visible and explicit will improve our moral control over it. But Arendt would ask: might this faith in “more light” overlook the functions that darkness serves? Is there a risk that demanding every algorithmic decision be rendered in plain view could create a veneer of ethical behavior – a performance of explainability – that actually hides a lack of true understanding or meaningful accountability? Just as a person forced to explain themselves at every turn might resort to glib justifications or clichés (performing the role of a “good person” rather than grappling with complexity), an AI forced to generate explanations for every output might produce superficial, human-pleasing reasons that mask its actual operation. Arendt helps us see that visibility can be superficial and even distortive. Moral life, whether human or artificial, needs some boundaries – not as a cover for malfeasance, but as a space for authenticity, context, and growth.

Glissant’s Right to Opacity and Suspicion of Total Knowledge

If Arendt approached opacity from the angle of individual privacy and the public sphere, the Martiniquan poet and philosopher Édouard Glissant gives us a perspective rooted in culture and power. Glissant famously declared: “Nous réclamons pour tous le droit à l’opacité” – “We demand the right to opacity for everyone.”. He was responding to a legacy of Western thought that equates knowledge with domination, where to fully know something (or someone) is often to control, categorize, or assimilate it. In Glissant’s view, insisting on exhaustive transparency and explanation – especially of people, cultures, or identities – can be a form of violence. It’s an attempt to flatten complexity and difference into terms the knower is comfortable with. Glissant was deeply suspicious of what he saw as a Western drive to attain a totalizing knowledge of others, a drive which underpinned colonialism and cultural imperialism. The demand “Explain yourself fully or be legible to me completely” is, in this sense, a coercive demand.

Glissant offers an alternative ethic: opacity as respect. To allow someone or something their opacity is to accept that they cannot be fully understood, and that this unknowability is not a flaw but a fundamental right. He argued that seeking total clarity “ignores the aspects of self that are difficult to grasp.” True understanding between people or cultures, for Glissant, does not come from dissecting them under an analytic spotlight, but from a mutual respect that includes respecting mystery. Opacity, he wrote, simply “accepts that everything that makes us us cannot be understood completely.”. In an elegant formulation, Glissant defined the right to opacity as “a right to not have to be understood on others’ terms, and to be misunderstood if one so chooses.”. This was a direct challenge to the idea that absolute transparency is a good thing. Glissant saw beauty and justice in allowing a degree of untranslatability — a recognition that life, identities, and complex systems exceed the grasp of any single explanatory framework.

While Glissant spoke about human relations and cultural identity, his insight applies powerfully to AI systems and the current insistence that they be made fully explainable. When we demand that an AI model provide a perfect, succinct explanation for its every decision, we might actually be perpetrating a kind of epistemic violence against the complexity of the world that the model is grappling with. A sophisticated AI system (say, a deep learning model with millions of parameters) might arrive at a decision through a tapestry of computations that no simple human language explanation can capture without loss. Forcing it to “explain” itself in simplistic terms could be like forcing a rich, multifaceted cultural narrative into a single stereotypical story – something gets lost, nuances are flattened. We risk creating the illusion of understanding rather than real understanding. Glissant invites us to be suspicious of the very desire for exhaustive knowledge. Sometimes, insisting on complete explainability stems less from genuine ethical need than from a culturally ingrained habit of domination-through-knowing – the same impulse that once drove colonial administrators to make maps and ethnic classifications of every society they encountered, believing nothing should escape their cognitive grid.

In AI governance, this perspective suggests we ask: Why do we seek total explainability? Is it always to serve the people affected by AI, or is it sometimes to satisfy our own urge to feel in control of these complex systems, to reduce them to terms familiar to us? Glissant might argue that rather than demanding that a complex AI flatten itself to our mode of understanding, we should sometimes approach it with humility – ensuring it is bounded and behaves ethically, without needing to fully grasp its every internal nuance. This does not mean leaving it unchecked (Glissant’s opacity is not secrecy for the sake of abuse), but it means acknowledging that a degree of opacity can be ethical and even necessary for truth. In his realm, truth in cultural exchange comes from respecting opacity; perhaps in AI, truth in operation can sometimes mean the system works in ways we cannot entirely follow, yet we constrain and evaluate its outcomes to align with our values.

Interpretability in AI: Debates and Dilemmas

Turning now directly to contemporary AI: how do the ideas of Arendt and Glissant illuminate the debates about AI interpretability and explainable AI? In recent years, XAI (Explainable AI) has become a bustling area of research. The motivations are sensible: if we can interpret why a medical AI diagnosed a patient as high-risk, or why a loan approval model rejected an applicant, we might detect biases, errors, or opportunities for improvement. Various techniques have been developed, from saliency maps that highlight which parts of an input influenced an image recognition output, to LIME and SHAP which provide approximate explanations for any given prediction. Governments have also leaned on explainability as a pillar of AI governance – for example, the GDPR’s mention of providing information about automated decision logic, or the U.S. FDA’s interest in interpretable algorithms for critical medical devices.

Amid this push, however, researchers have also encountered serious challenges and caveats. One fundamental issue is that modern AI systems (like deep neural networks) are exceedingly complex and high-dimensional. As scholar Jenna Burrell pointed out, there is an “opacity that arises from the characteristics of machine learning algorithms and the scale required to apply them usefully.”. Even if companies open-sourced all their code and models (full transparency in one sense), the logic of a model with billions of connections might still elude human understanding. This is what Burrell calls epistemic opacity: no intentional secrecy is involved, but the complexity itself defies human-scale explanation. Indeed, such a model might operate in ways that even its creators cannot fully articulate – not because of malice or secrecy, but because it has discovered patterns in data that don’t map neatly onto concepts we have words for. As an example, a vision AI might distinguish dogs from cats using some intricate combination of pixel patterns that is impossible to summarize as a simple rule like “has pointy ears.” The internal logic is just too distributed across the network. Burrell’s analysis splits opacity into multiple forms: intentional secrecy (like a company hiding its algorithm on purpose), illiteracy (the public not understanding code), and opacity intrinsic to the algorithm’s complexity. It’s the third kind that is most vexing: no amount of calls for transparency can fully resolve it, short of fundamentally changing how the AI works.

Another issue is that explanations can mislead. Many XAI methods provide post-hoc explanations – stories about a model’s behavior that sound plausible to humans. But studies have shown these can be unreliable. For instance, an explanation method might highlight certain features as important for a decision when in truth those features were merely correlated with the real drivers in a complex way. Cynthia Rudin, a prominent researcher in interpretable ML, has argued that such post-hoc explanations are “often not reliable, and can be misleading.” They might give a false sense of security or correctness, a bit like a suspect in a crime coming up with a smooth story that sounds good but isn’t the full truth of what happened. There have even been demonstrations where one can trick explanation tools – for example, making minor changes to an input that don’t change a model’s decision, yet cause the explainer to generate a totally different explanation. This calls into question whether the appearance of transparency actually translates to real understanding or improved ethics. If an AI generates a neatly packaged reason for each decision (e.g., “Denied the loan because income was below threshold X”), is it genuinely being transparent, or is it performing a kind of explanatory theater that may satisfy oversight on paper while its deeper logic remains opaque and possibly problematic?

Furthermore, there is evidence that in practice, accuracy and transparency can conflict – or at least that people prioritize outcomes over explanations. AI pioneer Yann LeCun noted a telling result: when users were offered a choice between two AI models – one a “black box” with 99% accuracy and the other an “explainable” model with 90% accuracy – users overwhelmingly preferred the more accurate black box. They didn’t really care about the interpretability per se; they cared that the system worked well. Interpretability was valued mainly as a proxy for trust – a sort of reassurance when we aren’t sure the system’s performance is adequate. But if performance is clearly superior, people will accept opacity. This human behavior suggests that explainability is not an end in itself; it’s a means to trust and accountability. If those can be achieved via other means (like rigorous testing or proven efficacy), users may not demand a detailed explanation for every decision. We see this in daily life: we trust airplanes to fly and medicines to work without knowing exactly how they function internally, because other mechanisms (testing, certification, safety regulations) give us confidence.

To be clear, this is not to dismiss explainability as unimportant. In many cases, a lack of explanation has enabled terrible outcomes – e.g., biased algorithms tagging innocents as high risk or black box credit scores redlining neighborhoods. Calls for transparency often arise from very legitimate grievances with secretive, inscrutable decision systems. But contemporary debate in AI ethics is increasingly acknowledging that explanation alone is not a silver bullet, and that some demands for transparency must be balanced with other concerns (like accuracy, privacy, or complexity). Scholars and practitioners are asking: What level of interpretability is necessary for accountability? Where might requiring “total” explainability actually undermine other values? These questions lead us to refine the concept of accountability itself and consider alternatives to full internal transparency.

Structural Accountability vs. Internal Transparency

One key distinction that emerges is between internal transparency (knowing the exact inner workings of a system) and structural accountability (ensuring the system as a whole behaves in accordance with laws, ethical norms, and the public interest). The myth of total explainability tends to conflate the two: it assumes that the only way to have accountability is to open up the hood and have a human-understandable account of every gear and lever inside the machine. But this isn’t always true. We can hold systems accountable by focusing on external behavior and rigorous constraints, even if internally they operate in opaque ways. This is analogous to how we might treat a human professional: we might not see or understand every thought process of a judge or a doctor (their “internal” reasoning may be partly opaque, intuitive, or inaccessible), but we do have structures to ensure their decisions meet certain standards – appeal processes, second opinions, outcome monitoring, codes of conduct, etc. We judge them by results and adherence to procedure, not by sitting inside their brain. In engineering, similarly, a complex system can be given external safety constraints and audits without fully explaining its internals.

Recently, AI governance researchers have begun to formalize this idea. One framework proposes a layered oversight approach that treats opacity not as a sin but as a manageable condition. As one paper puts it, opacity is “not an ethical failure but a condition to be responsibly managed,” shifting the burden of trust from individual comprehension of the model to the credibility of the institutions and processes surrounding it. In this view, trust in AI comes “not through technical transparency but through processes that ensure accountability, contestability, and justifiability.”. What might such processes look like? They include third-party audits, stress-testing of models for bias and errors, and institutional checks such as certification or regular oversight by regulatory bodies. The idea is that instead of expecting every user or every citizen to inspect the source code of an algorithm (which they likely can’t do meaningfully), we establish structural mechanisms to verify and vouch for the system’s integrity. These could be independent audit agencies that examine high-stakes AI systems and attest to their fairness, much as Underwriters Laboratories certifies electrical appliances for safety. Or they could be internal review boards within companies that include ethicists and domain experts who continuously monitor an AI’s impacts. Crucially, these mechanisms don’t require that the AI be completely interpretable to everyone; they require that it be constrained, tested, and subject to challenge. Even if some internal logic is opaque, if the system’s outputs are rigorously evaluated and any harms can be detected and corrected, accountability is served.

This approach aligns with what the European Union’s draft AI Act is moving toward: not banning “black-box” algorithms outright, but imposing strict obligations on high-risk AI systems – such as requirements for risk assessment, data governance, record-keeping, transparency to regulators, and human oversight. The emphasis is on outcome accountability: an AI system should be provably safe and fair in its results, regardless of whether it uses a transparent rule or a complex neural net to get there. In fact, the Act explicitly calls for “technical documentation” and “record keeping” so that authorities can trace what happened in an incident, without mandating that the algorithm be interpretable by the general public. This is structural accountability in action. It recognizes that a certain opacity in the internals can be acceptable if we have robust traceability (logs of inputs and outputs), and if institutions can inspect those when needed (audit trials), ensuring there is no unaccountable decision.

To sharpen the distinction: Internal transparency means I can open the AI and directly see why it did X (like reading a simple decision tree or a set of if-then rules). Structural accountability means that even if I can’t directly see why it did X, I have ways to test that X was done for acceptable reasons (by checking that the outcome correlates with legitimate factors, by challenging the decision and getting it corrected if it was wrong, by ensuring there is recourse, etc.). Consider a hiring algorithm: internal transparency would mean the algorithm can plainly explain each hiring decision in human terms (e.g., “Applicant was ranked lower because they lack a certification”). Structural accountability would mean the hiring system is audited for discrimination and error rates; applicants can appeal and a human will review borderline cases; the system’s overall impact is measured and reported; and if it’s found to be biased, it gets fixed – even if the exact inner weighting of factors isn’t fully intelligible to a layperson. We might find that structural measures catch biases that a superficial per-decision explanation would not reveal (for example, an explanation might say “lack of certification” but an audit might reveal that this criterion disproportionately filtered out candidates of a certain gender – something a single instance explanation wouldn’t show but a structural review would).

The benefit of this approach is that it avoids the trap of performative transparency. If we obsess over making the AI produce human-like justifications for everything, we risk what Arendt warned of: converting ethics into performance. An AI could appear very transparent – “Here is why I did this, here is why I did that,” it might output – and yet those explanations could be partial truths or formulaic tropes that satisfy a requirement without truly enabling accountability. In contrast, if we focus on structural accountability, we don’t care how charming or comprehensible the AI’s own story about itself is; we care that independent checks confirm it’s not doing anything harmful or illegal. The trust is earned through “structured justification, role-based accountability, and institutional legitimacy,” rather than through a glossy veneer of interpretability. In sum, we move from a naïve faith in transparency to a mature practice of accountability.

The Case for Ethical Opacity

What exactly do we mean by ethical opacity? It is certainly not the stance that “AI owners can keep everything secret, and just trust us.” Ethical opacity is not secrecy for power’s sake. In fact, ethical opacity comes with a heavy load of external constraints. The concept can be defined as follows: An AI system exhibits ethical opacity when its internal decision-making process is not fully explainable in human terms, yet the system is designed and governed such that it adheres to ethical and legal constraints, and its outcomes can be externally validated as meeting those constraints. In other words, the inner workings might be a black box, but the inputs, outputs, and surrounding governance ensure that the black box doesn’t violate the rules we care about.

One way to think of this is by analogy to how society handles professional discretion. A chef’s recipe may be opaque (a trade secret, or simply too complex to reduce to a formula), but the food coming out of the kitchen is subject to health inspections and quality control. As long as the food is safe and delicious, we accept the opacity of the creative process in the kitchen – indeed, we value it, since too much interference or demand for a breakdown of every step might stifle the chef’s creativity and responsiveness to context (the very thing that makes the dish excellent). Likewise, an AI might have an opaque model that allows it to be more context-sensitive and responsive to nuances (for example, recognizing subtle patterns in a medical image or personalizing education to a student’s latent learning style). Forcing the AI to only use fully interpretable, simple rules could remove this context sensitivity; we’d get a blunt, overly general system that might be less ethical in outcome (say, less accurate in diagnosing a rare disease or less fair by having to generalize with coarse categories). Ethical opacity preserves the AI’s ability to be nuanced and responsive in ways that a rigid, easily explained system might not. The price we pay is that we, as observers, cannot always follow exactly how it reached a particular decision – but the benefit is a potentially better decision within the bounds of ethics.

To ensure this opacity remains ethical, we impose external discipline on the system. This could include:

• Pre-training and design constraints: Developers train the model with fairness criteria in mind or use techniques to reduce unwanted bias, and they document the model’s intended scope. Certain features (like race or gender) might be excluded or handled in special ways to prevent discrimination. These are design-time constraints.
• Monitoring and testing: Before deployment and continuously during use, the AI’s outputs are statistically analyzed for patterns of bias or error. If a lending model, for instance, shows significantly different approval rates for two equally qualified demographic groups, that’s flagged and addressed – even if we can’t pinpoint the exact neuron causing it.
• Role-sensitive explainability: Some opacity can be mitigated by providing different layers of explanation to different stakeholders. For example, an engineer debugging the system might use complex tools (like SHAP values or counterfactual probes) to get technical insight into the model’s workings (partial transparency at a low level), whereas an end-user might get a simpler explanation of a specific decision. The AI doesn’t have to be entirely opaque; it’s just not fully transparent to everyone at all times. Ethically managed opacity often means selective transparency – open to scrutiny by experts or regulators (under NDA or regulatory authority) even if not to the general public.
• Institutional oversight and contestability: There are clear mechanisms for people to challenge decisions. If you think the AI made a mistake in your case, you can demand a human review or a second pass. The institution deploying the AI takes responsibility for its outputs. This is key: ethical opacity does not mean the AI operates in a vacuum. It means the AI is one component in a socio-technical system where humans and institutions still bear responsibility and can intervene.

When these conditions are met, opacity can actually enhance ethical outcomes. It allows AI systems to utilize the full power of modern computational methods (which are often complex and not easily interpretable) in pursuit of goals like accuracy, personalisation, and adaptation to context – as long as the external guardrails catch any harms. Importantly, ethical opacity can also protect values like privacy and autonomy. Sometimes transparency demands can clash with privacy: for example, making an AI model fully explainable might require revealing sensitive attributes it considered. Or providing a detailed explanation for one person’s loan denial might expose aspects of their financial history they consider private. A system that instead guarantees fair treatment through audits but doesn’t spill all personal data in each explanation might better preserve individual privacy. Similarly, overly transparent systems can be “gamed” or manipulated (this is why some opacity is intentionally kept in systems like credit scoring or college admissions – if everyone knows the exact formula, it can lead to strategic behavior that undermines the process). Ethical opacity acknowledges these complexities and tries to strike a balance.

The concept of ethical opacity also resonates with Glissant’s idea of “opacity as an alterity working against the logic of recognition”. In an AI context, this means opacity can prevent the system from collapsing everything into the existing categories we have. It leaves room for new, perhaps unexpected relations. For example, a totally transparent model might stick to human-predefined rules (because those are easy to explain), whereas an opaque model might find a novel way to make decisions that actually achieves fairness better, albeit in a way we wouldn’t have prescribed. We then, through oversight, recognize that it’s working and allow it even if we don’t fully grok its method. This is analogous to how sometimes a new scientific discovery is first used before it is fully understood – we might know a drug works via trial results long before we understand the biochemical mechanism. As long as we ensure safety through testing, we accept that opacity because it benefits patients.

In sum, ethical opacity is an approach where we govern AI by outcomes and constraints, not by demanding complete explicability of internals. It is a humility about human cognitive limits (we can’t always expect to understand ultra-complex models) coupled with a firm insistence on accountability. The internal opacity is tolerable only because it is paired with external transparency of process (we disclose how we’re testing it, what data goes in, what results come out) and external accountability (we have someone to hold responsible and procedures to correct things when they go wrong). Opacity that comes with such discipline is a feature, not a flaw – it allows AI to function with the richness of real-world complexity while maintaining ethical alignment through other means.

Safety and Governance Objections (and Responses)

Such an argument in favor of opacity will naturally invite objections. From a safety perspective, one might argue: “If we don’t fully understand the AI’s inner workings, how can we be sure it won’t do something catastrophic or insidiously harmful?” Isn’t an opaque AI a potential ticking time bomb, especially as systems get more autonomous? These are valid concerns, especially in domains like autonomous vehicles or AI in weapons, where lack of understanding could lead to accidents. The answer here is that ethical opacity is not absolute opacity. We are not advocating blind trust in machines. Critical properties of the AI should still be understood and verified. For instance, one might not know the exact contribution of each neuron in a neural network, but one can formally verify certain safety properties (e.g., the controller of a car will not command a turn that flips the car, or the AI will not output a toxic sentence in a content filter). Techniques in formal verification and adversarial testing can check specific potential failure modes without requiring a full interpretive model of the entire system. Moreover, in high-criticality applications, ethical opacity might mean multi-layered controls: an opaque AI must have a transparent safety layer around it. Picture a black-box autopilot system that is wrapped in a simpler rule-based system to catch any obviously unsafe actions (e.g., “if commanded descent rate is above X, override”). In essence, you contain the opacity within bounds that you do fully understand.

From a governance perspective, regulators and advocates might worry that allowing opacity gives cover to bad actors. What if a company claims “our system is too complex to explain” as an excuse to hide discrimination or errors? This is a legitimate risk – history has shown many companies would prefer to hide their algorithmic decision criteria, especially if they encode bias or might cause public backlash. The response is that ethical opacity comes with a trade: increased external scrutiny in exchange for not demanding internal simplicity. If a company wants to deploy a black-box model, it should be prepared to subject it to more rigorous external audits. For example, regulators might require access to the model for testing, or detailed results broken down by protected groups, or information on the training data and objective functions used. If a company refuses such oversight, then we are no longer in the realm of ethical opacity but dangerous secrecy. Ethical opacity as a concept does not endorse the hiding of harms – on the contrary, it demands that any potential harms be proactively looked for via other means. This is where transparency of the right kind remains crucial: for instance, transparency about the development process, about data sources, and about performance metrics. The internal logic may remain opaque, but many other things should be highly transparent. One can distinguish “opacity that conceals harm” from “opacity that preserves nuance” by looking at intentions and outcomes. Is the opacity being used to avoid accountability (like a police department refusing to reveal how its face recognition matches suspects, thereby stymieing any inquiry into racial bias)? That is unethical opacity. But if the opacity is simply the by-product of a powerful technique, and the organization is actively monitoring and disclosing the system’s impacts, then the opacity is not being used to conceal; it is a fact of complexity, accepted with eyes open and counterbalanced by accountability.

Another objection: “Won’t the public lose trust if AI is not explainable? People fear what they don’t understand.” This is an interesting point because it’s often assumed that only transparency yields trust. However, as discussed earlier, trust is a complicated thing. In many cases, untargeted transparency can even erode accountability and trust. For instance, simply dumping information on people can lead to confusion or a false sense of security. Studies in behavioral ethics show that disclosures (a form of transparency) can lead to moral licensing – e.g., a conflict of interest disclosure might paradoxically make a professional more biased, because now they feel they have “come clean” and the onus is on the other party to be wary. In one experiment, when corporate board members’ friendships were disclosed (to ostensibly alert them to bias), those members actually behaved more favorably to friends, as if the disclosure absolved them of the need to self-police. The lesson is that transparency for its own sake doesn’t guarantee trust; it has to be actionable and tied to accountability. If we instead tell the public: “This system is being overseen by an independent auditor, and here are the published results of bias testing, and here is a redress mechanism for errors,” that may build more trust than a technical 100-page explanation of the algorithm’s workings. In fact, one survey found that while people say they value explainability, they tend to care more that somebody trustworthy can explain or justify the system if needed – not that they themselves understand every detail. Layered accountability, where experts and institutions ensure the AI’s integrity, can thus support public trust effectively. After all, we trust airplanes without reading engineering manuals; we trust food without knowing the recipe – because we trust the accountability systems behind them (regulators, standards, reputation of producers, etc.).

From the AI safety research angle, particularly with advanced AI (like hypothetical future AGI), some argue that not understanding internals is extremely dangerous – an opaque superintelligence might have unsafe goals, etc. That’s a very complex debate beyond this chapter, but even there, many experts propose containment, testing, and verification as safety measures rather than full transparency (since a super-complex intellect might be beyond transparency anyway). The notion of “interpretability as a safety tool” is valid (research is ongoing to open the black boxes of neural nets to check for dangerous circuits), but it’s complemented by other tools like sandboxing the AI, monitoring its behavior under many conditions, and limiting its ability to do unrecoverable harm. In short, for each objection about opacity, the answer isn’t “ignore it,” but “address it through a different facet of governance or design.” Transparency is one tool in the toolbox for aligning AI with ethics; it is not the only tool, and sometimes not the sharpest.

Opacity that Conceals Harm vs. Opacity that Enables Moral Discernment

To navigate this nuanced stance, it’s important to clearly distinguish between illicit opacity and ethical opacity. How do we tell when opacity is being used appropriately versus when it’s a cover for wrongdoing?

Opacity that conceals harm is marked by avoidance of scrutiny. If developers resist providing any information, refuse audits, or cannot demonstrate that their model meets standards, that’s a red flag. For example, if an AI hiring tool is found to systematically disadvantage women and the company says, “We can’t explain it, that’s proprietary and too complex, sorry,” that opacity is concealing harm. It fails the test of accountability. Likewise, opacity coupled with no external checks is problematic: a bank using a secret algorithm with no bias testing or recourse for applicants is likely hiding discrimination behind complexity. In such cases, the right remedy is to demand more transparency or to prohibit the use of the algorithm until it can be made accountable. In policy terms, one might enforce transparency requirements targeted at revealing potential harms – e.g., disclosing the features used by the model (even if not the exact weights), or opening the model to a regulator’s inspection.

On the other hand, opacity that preserves moral discernment is characterized by the presence of robust accountability measures and by an ethical purpose behind choosing a more opaque model. A classic example might be a medical diagnostic AI that uses a deep neural network. Suppose this model can detect early-stage tumors 5% more accurately than any interpretable model, potentially saving many lives, but it cannot fully explain its predictions. If the hospital employing it also keeps diligent track of its false positives/negatives, allows doctors to second-guess its suggestions, and ensures it’s not systematically biased (say it’s equally accurate across patient demographics), then here opacity is in service of a moral good (better health outcomes) and buffered by oversight (doctors and statistics keeping it in check). Removing this opacity by forcing a simpler model might reduce diagnostic accuracy and actually harm patients – a case where insisting on transparency could ironically result in a less ethical outcome. We see analogous tensions in privacy: sometimes not knowing everything about a person (opacity of personal data) is important for fairness, to prevent human biases from creeping in. A hiring AI that hides candidates’ names and ages from the decision process is in a way “opaque” about those attributes by design – but that opacity is ethical because it prevents bias.

Another illustration of opacity preserving moral discernment can be found in the human context of judgment calls. Consider a parole board or a judge making a decision about releasing someone from prison. Humans often explain their reasoning, but there are intangibles – demeanor, remorse – that factor in. If we forced a judge to rely only on an explicit checklist (total transparency of criteria), we might eliminate some arbitrary variance, but we might also eliminate compassionate exceptions or context that the checklist can’t capture. A judge’s intuition is a kind of opacity (even the judge might not fully articulate why they feel a person is truly reformed). Society doesn’t give judges unlimited opacity – there are guidelines and appeals – but neither do we demand a formula. We recognize that some moral discernment operates in a space of partial opacity. The key is that external review can correct any clear errors (e.g., an appeal court can overturn a bad decision), and certain boundaries (sentencing ranges, anti-discrimination laws) guide the decision. Similarly, an AI could be allowed to use an opaque internal process to make nuanced decisions, as long as it stays within fairness and reasonableness boundaries and those decisions can be appealed or reviewed.

In practical terms, distinguishing good opacity from bad opacity involves asking questions like: Has the system been vetted for known risks? Are the results explainable even if the process isn’t? (For instance, one may not know why it flagged a specific transaction as fraud, but one can examine a batch of decisions and see that indeed 95% of flagged cases were fraudulent – so the outcome is explainable in aggregate effectiveness, if not each individual reasoning). Is there recourse for those affected? Is the opacity serving a proportional aim? (e.g., a slight boost in profit at cost of opacity in a critical decision might not be justified, but a major boost in public health at cost of opacity might be). These considerations ensure that opacity is a deliberate ethical choice with trade-offs weighed, rather than a convenient loophole.

It’s worth noting a subtle point: sometimes too much transparency can itself cause harm. This is the flipside – what we might call harmful transparency or “the opacity of excessive clarity,” to twist the phrase. For example, transparency without privacy guardrails can expose personal data and violate rights. Or making an AI’s decision process fully public might enable malicious actors to find loopholes or exploit it (think of spam filters – if spammers knew exactly how emails are classified as spam, they’d reverse-engineer messages to get through). In such cases, maintaining opacity (not revealing the full decision criteria) is ethically warranted to prevent exploitation or protect confidentiality. Thus, an absolutist stance on transparency is untenable; some opacity is not only permissible but actually required to avoid worse outcomes.

Finally, embracing ethical opacity means we also invest in alternative forms of intelligibility. If we can’t have a blow-by-blow explanation of a model’s decision, we might present other information to stakeholders: statistical explanations (“this system historically approves 5% of loans, and your profile fell just below the threshold because your income was slightly below what 95% of approvers had”), or contrastive explanations (“if your income had been $5,000 higher, the loan would likely be approved”), or policy explanations (“the system is designed to prioritize debt-to-income ratio; unfortunately yours was above the cutoff”). These are hybrid approaches where the internal complexity is abstracted into human-relevant terms without fully exposing the algorithm’s code or weights. They offer a because that is actionable, even if it’s not the raw because. This satisfies the human need for an answer (“Why not me?”) to some extent without needing the model to literally spell out its exponentially many calculations. In other words, opacity at the micro level (the nitty-gritty of the model) can coexist with transparency at a higher level (the principles or factors the model generally follows). This layered approach to explanation is another way to reconcile the demands – it’s not all-or-nothing between black box and open book; there are many pages we can choose to show or summarize.

Conclusion: From Performative Transparency to Layered Accountability

The ethos of “In the World but Not Of It” has, throughout this book, invited us to remain engaged with modern technological and social systems while also critically distancing ourselves from their easy assumptions. In the case of AI ethics, the easy assumption we have interrogated is that total transparency is the pinnacle of ethical AI. We conclude now by synthesizing our critique and pointing toward a richer model of accountability that avoids the twin dangers of blind opacity and naive transparency.

Unbounded transparency – transparency pursued as an absolute – can create a performance of ethics rather than the reality of ethics. When every process and decision is required to be visible, organizations and individuals inevitably start managing appearances. We have seen how, under constant surveillance or demands for explanation, agents may game the system, producing convincing explanations that satisfy observers without actually improving the integrity of decisions. This is akin to a student who, forced to show their work for every answer, learns to pad the homework with plausible steps that look good but may hide blind spots in understanding. In AI, requiring an explanation for every output might simply produce an explanatory module that sounds right but does not necessarily reflect the true reasoning – a kind of PR department for the algorithm. Meanwhile, the focus on making things visible might distract from outcomes. A hiring AI could proudly present non-discriminatory reasons for each rejection, while still ending up excluding all candidates from a certain group in practice – but because each instance had an explainable reason, the structural bias might be missed. This is transparency-as-performance: a check-the-box approach where the presence of explanations is equated with ethical behavior. True ethics, however, is about substance – the fairness of outcomes, the respect of rights, the mitigation of harm – not just the form of being transparent.

On the other hand, opacity by itself is not a virtue; it is virtuous only under the right conditions. The goal is not to shroud AI in mystery and tell the public “just trust the wizards of AI.” Such an approach would rightly erode trust and abdicate the moral responsibility of technologists. The path forward is layered accountability: a multi-tier system of governance that combines selective transparency, oversight, and constraints. In a layered model, some layers of the AI (for example, how it was developed, what data it uses, what principles it optimizes for) are transparent to regulators and the public. Other layers (like the complex model parameters) might remain opaque but are monitored by internal checks and external audits. At the top layer, the accountability is clear: the deploying organization is accountable for the AI’s behavior and must answer for any negative outcomes. This echoes what the emerging best practices and legal frameworks are suggesting. Instead of expecting algorithms to self-police through openness, we set up institutional policing of algorithmic impacts. This layered approach answers the key needs of ethics and governance: it provides traceability and audit (through logs, documentation, and third-party review), recourse and remedies (through human oversight and appeals), and ongoing improvement (through monitoring feedback and updating the system).

Such an approach can indeed build public trust, arguably more honestly than a pure transparency approach. People trust systems that show good results and have accountable stewards. If a government says, “We have tested this AI thoroughly, we publish an annual report on its performance, an independent board reviews its operations, and you have a hotline to report any issues which we will investigate,” the public might accept a lot of internal complexity in that system. They will trust it because they trust the framework around it. This is how we handle many complex systems (nuclear plants, airplanes, financial systems) – not by making them fully transparent to every citizen (which is impractical and unhelpful), but by ensuring robust governance.

At the same time, our society should remain vigilant that calls for explainability are not dismissed whenever inconvenient. The myth of total explainability is dangerous, but so is a cavalier myth that “AI is beyond your understanding, so don’t even ask.” We must chart a middle course. The critique advanced in this chapter is meant to recalibrate our priorities: from explainability as an end-all be-all, to explainability as one tool among many for aligning AI with ethical values. Transparency with boundaries – boundaries informed by Arendt’s and Glissant’s insights – can actually serve ethics better than blanket transparency. Boundaries permit the breathing room for nuance, context, and humaneness to survive in an overly formalized world. And within those boundaries, accountability mechanisms ensure that opacity is never an excuse for abuse.

In conclusion, the myth of total explainability gives way to a more sophisticated paradigm: one that values clarity where it matters – in objectives, responsibilities, and outcomes – but accepts opacity where it is beneficial – in the intricate inner workings that may elude simplification. This paradigm insists that what truly matters is that the system as a whole is understandable and answerable to us, not that we stare at the source code of its every decision. It reminds us that transparency without limits can devolve into spectacle, whereas structured accountability with prudent opacity can foster genuine trust and better ethical performance. In being in the world of algorithmic systems, we embrace the tools of audits, oversight, and yes, sometimes explanations; but in not being of it, we resist the temptation to worship transparency as an idol. We recognize that a balance of light and shadow, of explanation and intuition, of transparency and opacity – carefully managed – is necessary for a moral life in the age of AI.

Sources:

• Hannah Arendt emphasizes that a sphere of privacy – metaphorical “darkness” – is essential for things to grow, warning that “however strong [the] tendency to thrust into the light, [life] needs the security of darkness to grow at all.” Total exposure can lead to hypocrisy and cynicism in moral life.
• Édouard Glissant argues for “le droit à l’opacité” – the right to opacity – meaning the right not to be fully known on others’ terms. He contends that insistence on transparent definition ignores complexities, whereas opacity accepts that not everything about a person (or system) can be completely understood.
• Researchers note that modern AI models are often inherently opaque due to their complexity. Even with full access, their high-dimensional operations defy human interpretability. Demanding simplistic explanations can be misleading, as post-hoc explanation methods “are often not reliable, and can be misleading”, potentially giving a false sense of security. In practice, users prioritize performance: given a choice between a highly accurate black-box and a less accurate explainable model, users preferred accuracy, indicating they “don’t really care about interpretability but just want some sort of reassurance” that the model works.
• Simply increasing transparency does not always increase accountability. Studies have found that untargeted transparency (like indiscriminate disclosure of conflicts of interest) can reduce accountability by diffusing responsibility or licensing bias. One article notes “disclosure alone is not a panacea… ‘untargeted’ transparency without a clear recipient and follow-up action is likely to reduce, not increase, accountability.”. Effective transparency must be coupled with structures for dialogue and remediation of issues.
• New governance frameworks suggest managing opacity through layered oversight. Opacity is treated “not as an ethical failure but as a condition to be responsibly managed.” Trust is achieved “not through technical transparency but through processes that ensure accountability, contestability, and justifiability,” shifting focus to institutional oversight. Rather than seeking perfect transparency, these frameworks commit to “structured justification, role-based accountability, and institutional legitimacy” as the basis for trustworthy AI.

---