A few weeks ago I sat across from the head of clinical governance at a mid-size aged care provider. We were thirty minutes into a product walkthrough when she leaned forward and asked the question she had actually come to ask. “When something goes wrong, can my team explain to a regulator what this thing did, and why?”
I've been designing digital products for 25 years, and that question, in some form, is the one I keep hearing in every procurement room, security review, and legal sign-off across the sector. It is rarely about the model. It is almost never about whether the AI is clever. It is about whether the organisation can stand behind what the system is doing on a Tuesday afternoon when the clinical lead is on leave and the manager on duty has never seen this screen before.
This is the layer I think about first now, before I think about features or flows or screens. I call it the organisational trust layer. It is the part of an AI-native experience that procurement, security, legal and operations evaluate, sometimes openly, sometimes by quietly slowing the deal down. When the design holds, deployments move. When it does not, the demo wins the meeting and the pilot stalls in the second month.
Honestly, this is where I've had to unlearn the most. Most of what I knew about enterprise software said the boring layers, admin, audit, settings, were the layers nobody had to design for. With AI, they are the product.
Here is how I think about designing for them.
Predictable boundaries and clear accountability
AI is not going to be perfectly predictable, and pretending otherwise undermines the system. What can be made predictable, and must be, is the shape of what the system can do. Users need to know what it is allowed to touch, what requires approval, what it remembers, what it logs, and what it forgets. When boundaries are unclear, organisations compensate by wrapping every interaction in a manual process, and manual processes kill the adoption they were designed to protect.
The design job is to make those boundaries visible without making them noisy. A care coordinator should be able to see, at a glance, which actions are routine, which need sign-off from a quality lead, and which are off the table altogether. A reviewer should be able to look at any action and see what was inside the boundary and what was not. A compliance officer should be able to read those boundaries as policy, not as code.
The other half of the same problem is accountability. The instinct in early AI design is to build “one assistant that does everything”, and it is the worst pattern enterprises encounter. When something goes wrong, the organisation has to be able to say which part of the system made which decision, what data influenced it, and who was responsible for what. A monolithic assistant is almost impossible to audit, bound or fix. A system of named, scoped agents with clear responsibilities can be designed to be auditable, bounded and improvable without rewriting itself, with every action belonging to a known part of the system and every decision tied to an owner.
Safety as a feature, not a friction
Demos that optimise for speed are the ones that stall in pilot. The team in the room loves the four-second response. The compliance team in the next meeting wants to know what happened in those four seconds, what was sent where, and how to get it back if it goes wrong. If the experience cannot answer those questions, the deployment slows to a halt.
The pattern that works is to design safety as a visible part of the experience. A preview before any consequential action. A simulation before an execution that touches a person's record. Clear undo paths after the fact. A kill switch that works instantly and is obvious where it is. Before the system does anything that affects a person's plan, their record, or their funding, the people responsible should be able to see what it is about to do, what it will affect, and how to stop it.
Explainability sits beside safety, and the two reinforce each other. An answer on its own is not enough. A decision needs reasons. If the system cannot explain why it did something, the people in the loop become rubber stamps, approving things they do not really understand, until something breaks and trust collapses.
What works is showing the inputs, the trade-offs, and the alternatives the system considered. The user does not need a research paper. They need enough to feel the decision is comprehensible, and to know where to push back. Learnability is part of the same design, because the experience is teaching the user how the system thinks while they use it. A team that can predict what the system will do is a team that adopts it.
Governance and observability as the product
Enterprises do not buy features. They buy governance. Who can do what, who can approve what, what is auditable, and how the audit trail gets exported. If those answers are not visible and easy to understand, procurement and legal slow everything down, sometimes for months, and the deal the demo won quietly stops moving.
The implication for design is uncomfortable. Governance has to be a first-class part of the product surface, not an admin panel buried three menus deep. The roles and permissions structure should be readable by someone who is not technical. The audit trail should be navigable rather than a CSV dump. The controls that matter most to compliance should be discoverable from the same screens the operational team uses, because the people running the audit and the people running the work are usually closer than the org chart suggests.
Observability lives in the same neighbourhood. If only the engineers can understand the system, the system never scales beyond a small group. Care providers do not just want the platform to work, they want to operate it. That means readable activity history, understandable health indicators, drift monitors that surface in plain language, and validation states that are obvious from the surface. The IT team, the clinical team, and the operations team should each be able to look at the system and answer the question “is this working today?” without filing a ticket.
This is what distinguishes AI-native enterprise design from consumer AI design. Consumer experiences hide the machinery. Enterprise experiences expose the parts of the machinery the organisation has to operate, and put them in a form the operator can read.
Adoption as a confidence journey
Adoption in this category is not a tutorial. It is a journey, with stages, and the design either supports those stages or accidentally fights them.
The early stage is risk reduction. The team needs sensible defaults, a sandbox to try things in, a clear escalation path when something does not look right, and training that explicitly de-risks the system rather than evangelising it. The first month is for fear to be replaced by familiarity, and the right tone is reassurance, not enthusiasm.
The middle stage is competence. The team starts to know what the system is good at, where to push back, and where to take its output and run with it. This is where the explainability work pays off, because the team is interpreting reasons rather than just approving outcomes. Confidence, at this stage, is a function of how legible the system's behaviour has become.
The later stage is operating depth. The team is using the system to surface patterns they would not otherwise see, and starting to ask the kinds of questions that need governance and observability to answer. Has a behaviour drifted? Does a control need to change? Can a workflow be expanded? This is when the experience repays the investment in audit trails and health indicators, because the people asking those questions are not engineers.
What I check, designing each layer, is whether the team a year from now will be able to operate the system without us in the room. If the answer is no, the design has not finished its job.
The loop check
Agentic AI does not run in steps the way a form or a workflow runs in steps. It runs in loops, where memory shapes decisions, decisions shape plans, plans shape actions, and actions feed back into memory. The design question is not really about screens. It is whether the organisation can reconstruct what happened across a loop, end to end, trace a decision back to its source, prove compliance to a regulator, and recover when something goes wrong.
This is the test I keep coming back to. Can the people who have to answer for the system answer for it. Can the team running the audit run the audit. Can the operator stop the loop and pick up the pieces. Can the next person to inherit this platform read what happened before they arrived.
The head of clinical governance I started with did not need a smarter AI. She needed an experience her team could stand behind. From the design seat, that is the whole job. The intelligence is the surface. The trust is the structure.