How to judge a free security scanner for your AI-built app

Free scanners are everywhere right now, and many of them are aimed at apps built with AI tools. A scan can be a useful first look. It can't tell you your app is safe, and some scanners ask for things you should never hand over. This guide covers how we'd judge one, without naming or rating any of them.

1. Know what an outside scanner can see

Most free scanners work like a visitor. OWASP describes web vulnerability scanners as tools that test apps "normally from the outside" (OWASP: Vulnerability Scanning Tools). So a scanner sees roughly what a browser sees: your pages, your JavaScript bundle, your network requests and your response headers.

That's enough to catch some real problems:

• Keys shipped to the frontend. Anything bundled into a web page can be read by anyone. Supabase notes that keys in web pages, mobile apps and scripts are public "because anyone can retrieve the key from the source code or build artifacts" (Supabase: API keys). If a secret key ended up in your bundle, an outside scan can find it.
• Surface-level settings, such as missing security headers or an insecure server configuration (OWASP: Vulnerability Scanning Tools).

A publishable key showing up in your frontend is not, by itself, a problem. Supabase says it's "Safe to expose online", because it only reaches what Row Level Security allows (Supabase: API keys). A scanner that flags it as a leaked secret is telling you something about the scanner, not about your app.

A clean scan means “nothing obvious from the outside”, not “secure”.

2. Know what it usually can't see

Your most important rules live on the server, where an outside scanner has no view. In Supabase, Row Level Security policies are "authorization rules that run inside the database" (Supabase: RLS). A visitor can't read your policies. At most, a scanner can try requests and guess from the answers, and it can only try the requests it thinks of.

That matters because access control is the classic weak spot. OWASP's Top 10 puts broken access control first, with examples such as viewing or editing someone else's account "by providing its unique identifier" (OWASP Top 10: A01 Broken Access Control). Spotting that kind of gap usually depends on knowing who is supposed to see what, which a generic tool doesn't know. OWASP's testing guide says automated tools are useful for breadth, but "often lack the application-specific context needed to detect deeper flaws" (OWASP Web Security Testing Guide).

So a clean scan means "nothing obvious from the outside", not "secure".

3. Never paste secrets into a scanner

Some tools offer a "deeper" scan if you paste in more access. Don't give them:

• your Supabase secret key or legacy service_role key,
• your database password or connection string,
• any other private API key or token.

Supabase is direct about this. A secret key "bypasses every Row Level Security policy you have" (Supabase: API keys). It says secret keys should only be used on computers you fully own or control, and should never travel over chat or email, or in a URL (Supabase: API keys). A web form run by someone else fits none of those. OWASP's secrets guidance makes the same point more generally. Every person and system that can read a secret is another place "the secret can now leak" (OWASP: Secrets Management Cheat Sheet).

If you've already pasted a secret key somewhere, treat it as leaked. Supabase's steps are to fix the cause, create a new secret key, replace the old one everywhere, confirm, and then retire the old one (Supabase: API keys).

4. What a trustworthy scanner asks for, and what it doesn't

From the sources above, here's what we'd look for. These are our criteria, not an official standard.

Reasonable to ask for:

• Your app's public URL.
• Your public (publishable) key, if anything. It's already in your frontend and designed to be public (Supabase: API keys).
• Confirmation that you own the app or are allowed to test it.

Red flags:

• Asking for a secret or service_role key, a database password, or admin login details (Supabase: API keys).
• Asking you to turn off protections such as RLS "so the scan works".
• Scanning sites without asking whether you own them.
• Scary results with no explanation. A good report shows what it tried, what it saw, and how to check it yourself.
• Calling a clean result a guarantee. Even OWASP says scanners have "their own strengths and weaknesses" (OWASP: Vulnerability Scanning Tools).

5. Check the findings yourself

Treat every finding, good or bad, as a lead to verify. Two checks cover most of what matters for a Supabase-backed app.

The two-account test. Create two test accounts in your own app. Signed in as user A, try to read, change or delete user B's data, for example by changing an ID in a request. Then try again while signed out. You should get nothing back. This is a hands-on version of the "someone else's account" problem OWASP describes (OWASP Top 10: A01). Only test apps you own.

The RLS check. In your Supabase project:

• Confirm RLS is enabled on every table in an exposed schema. Supabase warns that a table without it "is readable and writable by any role with a grant on it" (Supabase: RLS).
• Make sure each table has separate policies for select, insert, update and delete, and that each one names its role (Supabase: RLS).
• Run the Security Advisor. Supabase's advisors flag issues such as "misconfigured RLS policies" (Supabase: Advisors). Some findings may be intentional, so check them against how your app should work (Supabase: Advisors).

Our Supabase RLS guide walks through all of this step by step, including the common errors and tested SQL.

Quick checklist

• The scanner only needs your public URL (and, at most, your publishable key)
• You never paste a secret, service_role key or database credentials into it
• You read "no issues found" as "nothing obvious from outside"
• You checked each finding with the two-account test
• RLS is on for every exposed table, with a policy per operation (our RLS guide)
• Security Advisor findings have been reviewed

Your turn

What's the most confusing scanner result you've had on an AI-built app, and how did you work out whether it was real? Tell us. We'll use the answers to shape a future guide.