What to do if a dependency in your AI app's backend gets poisoned

Many AI app backends are Python: an API server, an agent, a script that calls a model provider. They pull in dozens of packages, and any of them can ship a bad release. This guide covers one pattern seen in a recent incident: a popular Python package published a poisoned release. Installing it dropped two files into site-packages. One was a .pth file that started a credential stealer whenever Python started. The other was a sitecustomize.py file that started a backdoor. The app didn't have to import anything for either to run.

If you think you installed a poisoned release, the short version:

1. Don't keep using the affected machine or image for anything sensitive. Every Python process on it may run the bad code.
2. Check for unexpected startup files (commands below).
3. Rebuild the environment or image from a known-good, pinned state. Don't just delete files.
4. Rotate every credential the machine could reach, working from a clean machine.
5. Pin with hashes, lock, delay new releases, and pin images by digest so the next bad release is less likely to reach you.

1. How a package can run code without being imported

Python's site module runs automatically at startup and reads configuration files in each site-packages folder (Python: site):

• .pth files normally list extra folders to add to the import path. But "lines starting with import (followed by space or tab) are executed" (Python: site).
• .start files are new in Python 3.15. They name a pkg.mod:callable entry point that's called at startup (Python: Startup entry points, PEP 829).
• sitecustomize is a module Python tries to import after setting up paths, and it "can perform arbitrary site-specific customizations". usercustomize does the same from the user's site folder, if the user site is enabled (Python: site).

Python's docs say these executable lines "are always run at Python startup (unless -S is given...)", whether or not the package is used (Python: site). Python 3.15 deprecates import lines in .pth files, but they still run during the deprecation period (Python: site).

Verified (Python 3.13 and 3.15): a harmless .pth file with one import line, which only wrote a marker file, ran when we started python -c 'pass'. It also ran when we ran python -m pip --version, so pip itself triggers it. A harmless sitecustomize.py ran the same way, and so did a usercustomize.py in a user site folder. On 3.15, a .start file ran too. With python -S, none of them ran.

2. Check site-packages for unexpected startup files

Starting the suspect Python runs its startup hooks, so these checks use find and grep instead. Add every Python environment you use to DIRS: your project's .venv, a virtualenv folder, or a conda env.

DIRS="/usr/lib/python3* /usr/local/lib/python3* $HOME/.local/lib/python3* ./.venv"

# 1. Every startup file Python might run
find $DIRS \( -name '*.pth' -o -name '*.start' -o -name 'sitecustomize.py' -o -name 'usercustomize.py' \) 2>/dev/null

# 2. The lines Python will execute from .pth files
find $DIRS -name '*.pth' -exec grep -HnE '^import[[:space:]]' {} \; 2>/dev/null

On macOS, use the same commands with these folders. They are the python.org installer's framework (Python: Using Python on macOS), Homebrew's shared site-packages (Homebrew: Python), and the user site folder for framework builds (Python: site). Virtual environments use .venv/lib/python3.X/site-packages, the same as on Linux.

DIRS="/Library/Frameworks/Python.framework/Versions/*/lib/python3*/site-packages $(brew --prefix 2>/dev/null)/lib/python3*/site-packages $HOME/Library/Python/*/lib/python/site-packages $HOME/.local/lib/python3* ./.venv"

On Windows, in PowerShell. The folders are the user site folder %APPDATA%\Python (Python: site), the default install folders %LocalAppData%\Programs\Python and %ProgramFiles%\Python X.Y (Python: Using Python on Windows), and %LocalAppData%\Python, where the Python install manager keeps its commands by default (Python: Using Python on Windows). On Windows, a venv's packages are in .venv\Lib\site-packages.

$dirs = @("$env:LOCALAPPDATA\Python", "$env:LOCALAPPDATA\Programs\Python", "$env:ProgramFiles\Python*", "$env:APPDATA\Python", ".\.venv") | Where-Object { $_ -and (Test-Path $_) }

# 1. Every startup file Python might run
Get-ChildItem -Path $dirs -Recurse -File -Include *.pth, *.start, sitecustomize.py, usercustomize.py -ErrorAction SilentlyContinue | Select-Object -ExpandProperty FullName

# 2. The lines Python will execute from .pth files
Get-ChildItem -Path $dirs -Recurse -File -Include *.pth -ErrorAction SilentlyContinue | Select-String -Pattern '^import[ \t]'

Add any other Python you use, such as a conda or version-manager install, to DIRS or $dirs.

Docs-only for the macOS and Windows folder locations: we didn't run these on a Mac or a Windows PC. We did run both command blocks on Linux against fake folders laid out like a Mac and a Windows PC: the macOS line in bash, and the PowerShell commands in PowerShell 7.6.7. Both found our planted .pth, .start, sitecustomize.py and usercustomize.py files and printed the import lines.

To list the site-packages folders, Python's own command is python -c 'import site; print(site.getsitepackages())' (Python: site). On a machine you suspect, that starts Python and runs the hooks. On Python 3.14 and later, python -S -c 'import site; print(site.getsitepackages())' shows the right folders without running them. In our test on Python 3.13, the -S version run from a virtual environment listed the system folders instead of the venv's, which matches the 3.14 change noted in the docs.

Legitimate startup files exist. Don't panic at every hit:

• distutils-precedence.pth is installed by setuptools, and it contains an import line.
• Editable installs (pip install -e) create __editable__.*.pth files. Setuptools says it creates "a special .pth file" for them, and some of these install an import hook (setuptools: Development mode, pip: Local project installs).
• Environments created with the virtualenv tool can contain _virtualenv.pth, which is a single import line. We found one on our own test box.
• Some Linux distributions ship their own sitecustomize.py with the system Python.

So an import line tells you what to read, not what's bad. For each file with an import line, and for any sitecustomize.py, usercustomize.py or .start file, check:

1. Which package installed it. Each installed package has a RECORD file that "holds the list of installed files" (Python Packaging: Recording installed projects). In the site-packages folder, run grep -l '^NAME.pth,' *.dist-info/RECORD (replace NAME.pth with the file name). For files outside site-packages on Debian or Ubuntu, dpkg -S /path/to/file names the system package.
2. Whether that package should have it. A file owned by setuptools or by your own editable project is expected. A file owned by an unrelated package you just upgraded is not, and neither is one that no package lists.
3. What it does. Read it. Be wary of encoded or compressed blobs passed to exec, network calls, or anything that reads keys, tokens or environment variables.

Being listed in RECORD doesn't prove a file is safe. A poisoned release lists its own files too. Compare against the previous version, or against the package's public source.

Verified (Python 3.13 and 3.15): on a fresh venv with setuptools and an editable install, the commands found distutils-precedence.pth (one import line, owned by setuptools) and __editable__.myproj-0.1.pth (owned by our test project). Our planted fakes (zz_fake.pth, sitecustomize.py) were found, and the RECORD check reported them as not listed by any package. On the box's Debian-based system Python 3.13, an OS-owned /usr/lib/python3.13/sitecustomize.py sits earlier on the import path, so it ran instead of our planted one. Python imports only the first sitecustomize it finds, which is why the find step lists all of them.

3. Check your Docker images

Check the image without starting its Python. docker create makes a container "without starting it" (Docker: container create), and docker export writes its filesystem "as a tar archive" (Docker: container export):

IMAGE=your-image:tag
cid=$(docker create "$IMAGE")
docker export "$cid" | tar -tf - | grep -E '\.pth$|\.start$|(site|user)customize\.py$'
# Read one file without running anything:
# docker export "$cid" | tar -xOf - usr/local/lib/python3.X/site-packages/NAME.pth
docker rm "$cid"

If the image has find, you can also run it instead of Python, with networking off: docker run --rm --network none --entrypoint find "$IMAGE" / -xdev \( -name '*.pth' -o -name '*.start' -o -name '*customize.py' \).

Check every image built while the bad release was available, not just the latest one.

Docs-only for the Docker commands: Docker isn't installed on our test box. We tested the tar -tf | grep step on a tar archive laid out like an exported image, and it listed our planted .pth and sitecustomize.py files.

4. Rebuild, don't clean up by hand

If you find something, deleting the file isn't enough. A backdoor may have downloaded more code, and the stealer may already have run. Rebuild the virtual environment or image from scratch, from a lockfile or requirements file pinned to a known-good version, and redeploy. Treat the old machine or container as untrusted until it's rebuilt.

5. Rotate credentials, from a clean machine

Assume anything the affected machine or container could read is exposed. Do the rotation from a different, clean computer. Signing in to dashboards or creating new keys on the affected machine could hand the new keys over too.

Rotate, in roughly this order:

• LLM provider API keys your app or tools use. Create a new key, deploy it, then revoke the old one. Provider-specific steps are in our AI provider key rotation fix.
• Cloud credentials: access keys, service account keys, and any role credentials the machine could get from the cloud's metadata service. See secrets and IAM in your own cloud.
• Database and backend keys, for example a Supabase secret key. See rotate a Supabase key without downtime.
• SSH keys. Generate a new key pair on the clean machine (OpenSSH: ssh-keygen), add the new public key where it's needed, and remove the old public key from every server and Git host.
• Tokens your AI tools use: coding assistant sign-ins, MCP server configs, package-registry and Git hosting tokens, CI/CD secrets, and anything in the shell history or .env files on that machine.

Then check each provider's usage and audit logs for activity you don't recognize.

Docs-only: we didn't rotate real keys for this guide.

6. Pin versions and require hashes

Pinning a version (==) stops surprise upgrades. Adding a hash means pip installs only the exact file you approved. pip says hashes protect "against remote tampering and network issues", and that once hash checking is on, "Hashes are required for all requirements" and for all dependencies (pip: Secure installs).

# requirements.txt: every package pinned, with a hash (example names)
somepackage==1.2.3 \
    --hash=sha256:<hash of the file you reviewed>

python -m pip hash path/to/somepackage-1.2.3-py3-none-any.whl   # prints --hash=sha256:...
python -m pip install --require-hashes -r requirements.txt

pip hash computes the value for a downloaded file (pip: pip hash). If you use uv, uv pip compile --generate-hashes writes a requirements file with hashes for every package (uv: CLI reference). A hash pins a file you already chose. It doesn't stop you from choosing a bad release in the first place, which is what sections 7 and 8 help with.

Verified (pip 25.1): with the correct hash, the install succeeded. With a wrong hash, pip refused: "THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE", and it exited with code 1. With --require-hashes and no hashes, pip refused and printed the hash it saw. uv pip compile --generate-hashes produced a pinned line with two --hash=sha256: entries.

7. Commit a lockfile and install from it

If you use uv projects, uv lock writes uv.lock, which holds "the exact resolved versions", and uv says it "should be checked into version control" (uv: Project structure). In CI and deploys, install with --locked. If the lockfile is out of date, "uv will raise an error instead of updating the lockfile" (uv: Locking and syncing).

uv lock                 # after you change dependencies, on purpose
uv sync --locked        # in CI and deploys: fail instead of silently re-resolving

Verified (uv 0.13.1): uv.lock recorded sha256 hashes for the locked files. After we added a dependency to pyproject.toml without relocking, uv sync --locked failed with "The lockfile at uv.lock needs to be updated, but --locked was provided."

8. Wait before taking brand-new releases

In the incident this guide is based on, the poisoned release was taken down within hours. uv can ignore releases newer than a set age, which it calls a dependency "cooldown". It describes this as "a good way to improve security posture by delaying package updates until the community has had the opportunity to vet new versions" (uv: Resolution). The setting is exclude-newer. It takes a date or a duration such as 24 hours, 1 week or P7D (uv: Settings):

# pyproject.toml
[tool.uv]
exclude-newer = "7 days"
# Exempt one package if you need its security fix today:
# exclude-newer-package = { somepackage = false }

Two caveats from uv's docs: the cutoff is compared with each file's upload time, and when you use a duration with uv.lock, the cutoff is recalculated only on a new resolution, for example with --upgrade (uv: Resolution). A cooldown also delays security fixes, so decide how you'll make exceptions.

Verified (uv 0.13.1): with exclude-newer = "2020-01-01T00:00:00Z", uv lock picked an old release of a small test package. After we switched to "7 days", a plain uv lock kept the existing resolution, and uv lock --upgrade re-resolved to a current release. uv.lock recorded the window as exclude-newer-span = "P7D".

9. Pin Docker base images by digest

Image tags can move. Docker says "Image tags are mutable", and that pinning to a digest guarantees "the same image version, even if a publisher replaces the tag" (Docker: Building best practices):

FROM python:3.13-slim@sha256:<digest>

docker pull prints the digest (Digest: sha256:...), and you can pull by digest too (Docker: image pull). The trade-off is that you have to update the digest yourself to get base-image security patches, so schedule it.

Docs-only: Docker isn't installed on our test box.

Your turn

Which of these do you already do: hashes, a committed lockfile, a release cooldown, or digest-pinned images? Tell us what made it stick, or what got in the way.