For months, my community app ran an automated test suite that tried to break its own permissions. It ran against the live system, created test data, checked the rules, and cleaned up after itself. Every run reported green, with zero residue left in the database.
It had also been sending push notifications to real members’ phones the whole time.
Nobody noticed until a member sent a bug report with a screenshot of their lock screen, showing a test message from a test account. The database was clean. The phones weren’t.
The database isn’t the whole world
That’s the lesson I keep relearning: a cleanup proves the database is clean. It never proves the outside world is untouched.
Most of what an app does stays inside it. You can edit it, delete it, restore it from a backup. But some things leave:
- Push notifications
- Emails and text messages
- Payments and refunds
- Webhooks to other services
- Anything posted somewhere public
The moment those leave, they’re someone else’s. A deleted row can’t un-ring a phone. A refund isn’t the same as never charging.
Why AI-built apps leak outward
When you ask an AI to add a feature, it wires up the side effects too. That’s helpful: the notification sends, the email goes out. But it means a test, a mistake or a retry can reach real people as easily as the real thing does, unless someone decides otherwise.
And while you build, you’re usually the only user, so every email and push lands on your own phone. Everything looks fine until the audience is real.
Three more times something got out
The apology that was swallowed. After the test-notification bug, I added a guard: anything marked as a test never sends a push. Then I wrote the team a message explaining what had happened. It mentioned the test marker by name, so the guard treated my explanation as a test and blocked its notification. Most of the people I was writing to never got the alert.
Cancel versus delete. Removing an event from the calendar sounds like one action. It’s two. Deleting something made by mistake should be silent. Canceling something people planned around should tell them, with a reason. If the app only has one button, it either spams people over a typo or silently strands people who were planning to show up.
The confirm dialog that wasn’t true. The first version of a delete confirmation warned that it would permanently erase sign-ups and check-ins. The code actually just hid the event, and nothing was erased. A confirm dialog is a factual claim, read at the exact moment someone can’t afford to be misled. Mine was wrong, and I only caught it by checking the database after a real delete.
Five habits for the part you can’t undo
1. List everything that leaves
Before launch, get the complete list. Most people are surprised by how long it is.
List everything this app sends outside itself: emails, push notifications, text messages, payments, webhooks and anything posted publicly. For each one, what triggers it, and who receives it?
2. Put the guard at the door, not in the test
Don’t rely on every test remembering to be careful. Put one rule where messages leave the system: anything marked as a test never goes out. That covers tests nobody has written yet. Then make sure your real messages can’t accidentally look like tests.
Add one check at the point where this app sends notifications and emails, so anything created by a test account or marked as a test is never delivered. Then show me how I’d confirm it works.
3. Use test modes and test accounts
Payment providers have test modes and test cards. Email services have sandboxes. Use them, and keep test accounts separate from real people, so a test can’t reach anyone real even when something goes wrong.
4. Design the undo, or design the warning
For anything that leaves, decide what happens if it was a mistake. Can it be canceled, followed up or corrected? If it can’t, the action deserves a clear warning, and the warning has to be true.
For each action in this app that sends something or charges money, what happens if the person did it by mistake? Show me the undo, cancel or follow-up for each one, and check that every confirmation message matches what the code actually does.
5. Send it to yourself first
Before anything goes to a real audience, send it to yourself on a real device. Read the email in an actual inbox, including the spam folder. Look at the push on a locked phone. That’s how most people will see it, and it’s where problems show.
The design lesson
What leaves your app is the part of your product people experience when they aren’t using it: a buzz in a pocket, a line in an inbox, a charge on a card. That’s design territory. Who should hear about this, when, in what words, and what happens if we’re wrong?
AI will wire it all up for you, and quickly. Deciding what’s allowed out the door is still your job.
Want to check your own app? “What leaves your app” is item 7 of the Vibe Coding Checklist. The free Fix Kit has a prompt to list everything your app sends, and a way to test it without reaching a real person.
Stuck on a fix? I do a few 1:1 sessions where you share your screen and I teach you how to check and fix it yourself.