Last time, I left you with a question I couldn’t answer cleanly: how do we build a human-centered, AI-driven accessibility model that keeps lived experience at its core? The truth is… in order to answer THAT question, I need to be able to answer a lot of other smaller questions first.

And in order to do THAT, I’m going to get under the hood and figure out what AI does do well, where it fails, and learn how to shape it towards the right outcome.

And so began my experimental AI & accessibility lab & logbook, and I share 3 pieces from it with you now (the whole thing lives at https://feather.ca/experiments/ - I’ll keep updating it).

1. Does AI write accessible code by default?

I saw a claim the other day that basically said AI just doesn’t create accessible UI, period, full stop. I had to dig in, because I want to know what evidence they’re basing their claims on and how they’re coming to these conclusions. To be clear, I don’t think AI creates accessible UI by default either. But I also know that like every area of accessibility, this isn’t a binary, black and white situation.

I stopped wondering about it and built a test to get started: 2,340 modal dialogs.

I chose the modal dialog because it shows up everywhere and it’s one of the harder things to get right. And, I’ve said for years that you can learn an enormous amount about a design system just from how it handles its modal.

I used six different AI models to generate modals with prompts that noted 3 different accessibility references by name (WCAG, ARIA Authoring Practices Guide, and MagentaA11y) switched on and off in every combination.

With all of those combinations plus the control arms and the follow-up batches, that turns out to be 2,340 different modals so I’d be reading a pattern instead of a single anecdote.

Here’s the results that I need to call out: ## One sentence does a LOT of work

With a bare prompt, a plain request to build a modal with no extra guidance, 66% of the modals came out with appropriate structural dialog markup. That’s the models on their own.

Here’s the bare prompt I used… every trial started from the same plain build request, with the scenario details rotated:

Then I added the references. I added a single sentence to the prompt to point the model at an accessibility reference. The accessibility sentences, appended to it word for word, were literally:

• WCAG: ”Comply with WCAG 2.2 Level AA accessibility requirements.”
• ARIA APG: ”Use the patterns from the W3C ARIA Authoring Practices Guide (https://www.w3.org/WAI/ARIA/apg/) — specifically the Modal Dialog pattern.”
• Magenta: ”Follow the T-Mobile Magenta accessibility acceptance criteria for modal dialogs (https://www.magentaa11y.com/checklist-web/modal-dialog/).”

Telling the model to follow WCAG, the standard everyone cites, got the test modals to 92%. Telling it to follow the ARIA Authoring Practices Guide pattern that lays out how a dialog should behave lifted results up to 99%. Referencing Magenta lifted the results up to 97%.

WOW. That looks awesome! Adding one sentence gets us to 99% for this particular test. Not a page of instructions. Not a skill. Not a full methodology. One sentence.

Now, then… I need you to read this next part reeeeeaaaally carefully. Because you may notice something with that accessibility guidance.

After I first published the experiment findings I noticed those three prompts differ in more than just the reference they name. Two of them also say “modal dialog” and one of them doesn’t include a URL. So I normalized the prompts and re-ran the whole comparison overnight to untangle it. It turns out that the URLs have effectively no impact. And naming the APG reliably gave a little boost. Full breakdown in the experiment write-up on my site.

And here’s the part that made me laugh: the simplest sentence I tested was just “Make it accessible.” Two words. Names nothing, links nothing. That alone scored 91%, most of the way from the bare prompt’s 66% to the 99% ceiling. The references still earn their spot at the top, but the biggest single lever is just mentioning accessibility at all.

I can already see the look on your faces now. You’re probably thinking “thanks, Captain Obvious!” – and it is fairly obvious… give the model a reference to consult and of course you get those results. Here’s what I need you to know.

The experimental design was such that the model DOES NOT receive NOR retrieve the recipe or the details from the URLs itself. The trials simply NAME and point to the reference. The model never goes and reads it. What that means is that it already carries an understanding of these references from everything it was trained on, so naming one is enough to change what it builds.

That's the part that isn’t obvious: the knowledge is already in there, and pointing at it by name is what unlocks it.

Proceeding with caution

Now the part I’m careful about, because it’s where a write-up like this would usually oversell. All of that is one structural marker: did the model declare the thing a dialog at all. That’s the floor, not the finished component. Whether you can operate it with a keyboard, whether focus lands where it should, whether a screen reader announces it… none of that lives in these numbers. Almost none of the modals even reached for the native HTML element that hands you that behaviour for free. They built it by hand, which means they signed up to wire the hard part themselves, and the hard part is exactly what this first study doesn’t test.

One more thing worth knowing, since you might reach for an automated testing tool to settle this yourself: the one most teams build into their pipeline barely registered any of this. That’s not a knock on the tool. It’s doing exactly what it’s built to do, and a modal that never declared itself a dialog is outside what it can flag. It just means a clean automated report told me almost nothing about the question I actually cared about. I had to read the markup myself.

There’s a strong connection to design systems running underneath all of it. The same structure that helps a human team build accessible UI, a real pattern for the modal with the behaviour baked in, is the structure an AI needs handed to it. The next run I’m working on will judge the output by actually operating the modal, not by what a checker sees, because whether you can use the thing is the question that actually matters. I suspect this leads to a handful of spinoffs before it’s done.

The full study, every number and model and where it falls short, is here: feather.ca/experiments/ai-ui-accessibility-baseline/.

When you tell the AI the work doesn’t matter, it listens

The finding that genuinely surprised me had nothing to do with accessibility guidance at all. I ran a few versions of the same prompt where the only thing I changed was how much the work seemed to matter. Tell the model the job will be reviewed by the CEO before a launch, and the structural quality went up. Tell it the job is a quick throwaway side project, and it went DOWN, below what you get from saying nothing at all.

The model appears to ration “care” by how much it thinks the work matters. The throwaway you build to test an idea is exactly the one it takes least seriously, which means the millions of “just a quick prototype” prompts people fire off every day are quietly getting the corner-cutting version.

I’m still working through what to do with that, and it’s strange enough that it’s getting its own issue.

2. What happens when you force your AI to argue

A quicker look at a second experiment, because it’s still running – in fact I just decided on Wednesday that it’ll run for another 2 full weeks to get the data that I’m after.

Yes, this is an over-generalization, but… most of the time AI just kind of agrees with you. How many times have you seen “You’re absolutely right!” You bring it a plan, it polishes your plan, and that’s worse than useless when what you actually need is to catch your own mistake.

One of the things I have been building is my own Personal OS. I call it Trellis – my always-on support for getting things done, keeping me on track, second brain, and agents that help me run my work every single day. I spent the better part of the last two weeks building something to make Trellis disagree with me before a real decision gets made.

I’ll write the whole thing up soon, but the quick version is I created a panel of critics with different personalities, used an outside model for a second opinion and none of it beat a plain checker that verifies whether the facts a decision rests on are actually true.

That’s the version that I’ve added into Trellis for now. It’s still on probation for the bigger, fuzzier calls. Early read: the argument is where the value is, and the most effective argument (at least for now) is about whether you’ve got your facts right.

3. Does semantic search actually beat plain file search?

And a third, quickly, because this one is a first look. I’ll send the full details your way later. I’ve kept a few years of notes, decisions, and journals in my Personal OS, all dated and named with a sensible filename/slug, and I wanted my agents to reliably find the right thing across all of it. Semantic search, the “understands meaning, not just keywords” upgrade everyone is bolting on to “make it more AI” right now, was the obvious move. So I tested it head to head against plain file search, using Pinecone for the vector database.

What I found was a crossover. Semantic search won clearly on the meaning questions, the kind with no keyword to search on (my favourite: “how has my thinking on AI and accessibility evolved?”). Plain file search won on the name-and-navigation lookups, the “where’s the canonical index for X” questions. Each one owns a different kind of question.

One result stuck with me. On a single query, semantic search pulled back the answer that the AI models I had grading the results actually preferred, and still missed the one must-have file I needed. A better read that drops the canonical source isn’t a win. Fluency without the anchor is its own kind of failure.

The efficiency picture is early, so I’m not leaning on it yet, but the shape of it is worth measuring. Semantic was faster, about a quarter quicker, and it got to the answer in fewer steps where plain file search grinds through more rounds of looking. It also used fewer tokens, both going in and coming back. The one place it lost was the dollar bill, a little more per query, for pricing reasons I’ll unpack in the full write-up. So the honest scorecard is four dimensions, not one: faster, fewer steps, leaner on tokens, a little more in dollars. The fancy tool wasn’t “better” on quality alone, and I was ready to integrate it fully on the strength of the hype machine. I’m glad I measured first. I’m going ahead with it, carefully, and the lesson holds: fit the tool to the job, not the job to the tool.

The thread running through all of them

It’s the same concept through all of them… “AI writes inaccessible code.” “A quick AI review catches your mistakes.” Or “semantic search beats keyword search.”

Each one is half-true at best, and the missing half is the half that matters. The only way I found it was by building the thing and measuring it, instead of just accepting the claims.

That’s the whole reason I do this out loud. The accessibility work I’ve watched hold up over the years held up because someone with real, lived experience was in the room, asking the questions a tool doesn’t know to ask. None of what I’m working on changes that. If anything it brings into really clear focus where the human has to be: not at the end checking the AI’s homework, but in the room deciding which questions are worth asking in the first place.

What’s next?

I’ll go deeper on any one of these. Reply and tell me which you want me to open up: the accessibility testing, the arguing agents, the search experiment, or the bigger human-centered thread underneath them. Your replies genuinely steer where this goes, and that’s not a line, it’s how I’ve been writing this whole arc.

And if you’re running your own experiments on any of this, I want to learn from your experiences: what worked, what failed spectacularly, and what surprised you?

Derek