The most dangerous thing my AI ever did was nothing
I gave my AI assistant the keys to my family's documents, our calendar, our email, the home server — and then I felt good about it.
That's the part worth talking about. Not the access. The feeling good.
I'd been careful, or told myself I had. I built the thing; I knew what it could reach. And because I knew, I stopped asking the harder question underneath — what it could *do* with all that, on a day I wasn't watching.
---
## The number that stopped me
A recent study from Cequence and EMA surveyed a couple hundred enterprise security leaders. Ninety-four percent were confident their AI agents didn't have more access than they needed. Thirty-three percent had actually set it up that way.
I felt the cold little recognition of being described. I was in the ninety-four. If you'd made me go and check — trace every action my assistant could take without me in the room — I'd have found the same gap.
These are companies with security teams. The confidence still ran ahead of the control. And if they can be wrong about their own systems, the owner running an AI off a subscription and a good feeling doesn't stand a chance — not from carelessness, but because *feeling* you've scoped it right is identical to having done it. There's no alarm that tells the two apart.
---
## Software breaks loudly. Agents don't.
Ordinary software fails in a way you notice. It crashes, it throws an error, it stops.
An agent fails *confidently*. It takes a wrong action with the same calm as a right one — deletes the wrong file, emails the wrong person, misreads what you asked — and reports back in the same reassuring tone either way.
So "does it seem fine?" is a useless question. It always seems fine. The real question is colder: what would it take for me to actually *know*?
---
## Don't walk into the kitchen — unless it's your child eating
There's a saying: if you want to enjoy your meal, don't go into the kitchen. Most of the time that's right. You don't want to see how the sausage is made.
But if you know the food is going to your baby, you walk in — with a magnifying glass.
That's the shift. Using AI to draft an email is eating out; you don't need to watch the kitchen. AI touching your business records, your customers, your family's data — that's your child's plate. Now you want to see everything.
The trouble is most agentic AI won't let you into the kitchen. It's a black box: it acts, and if you want to know what it did, you go digging through logs nobody reads. That's the opposite of confidence.
When I rebuilt my own system — Tangerine — I went the other way. Two rules. A **wall**: a short list of things it can never do alone — move money, delete permanently, send anything outside the house — that wait for a human yes, because the cost of one confident mistake there is unrecoverable. And a **window**: I can see what it knows and why it acted. Its memory is legible to me. I don't trust it because it's caged; I trust it because I can look.
A gate you can't see through is just a black box with a lock on it.
---
## The agent you didn't approve
Everything so far assumes you know what your AI can do. In a company, you often don't.
Someone in accounts plugs a clever tool into the invoicing system. Someone in sales wires an assistant into the customer database to save an hour a day. Not reckless — resourceful. But now their agent acts under *their* login. When something moves, the log says the employee did it. It can't tell you the employee's AI did it, on its own, while they were at lunch. The badge is the same.
And then the sharper version. In 2023, Samsung let its chip engineers use ChatGPT. Within twenty days, three of them — separately — pasted in proprietary source code and a confidential meeting transcript, just trying to do their jobs. It couldn't be taken back; under the tool's terms, the data was already gone. These were expert engineers who knew they were using an outside service. Knowing wasn't the same as being stopped.
For a small business, that's the whole game. You won't out-monitor it. You make one question normal before anything gets connected: *if this got it wrong, what's the worst it could do — and who's in the loop before that happens?* A twenty-person shop can afford to ask. Not asking is what gets expensive.
---
I still trust my AI. It runs through my whole household. But not the old way — the easy, unexamined way that felt like diligence and was really just comfort.
The most dangerous thing it ever did was nothing: it sat there, capable of more than I'd admitted, waiting for a day I wasn't looking. That day never came. But I can only say so because I stopped asking whether it seemed fine, and started walking into the kitchen.