Check before you send: NebuRedaction, an experiment in keeping secrets out of logs

Read this first. NebuRedaction is an experiment. It did not reach the accuracy target we set for it, we do not recommend it for production, and we have no plans to improve or maintain it. We are publishing the idea and the code, under the MIT licence, in the hope that someone else will take it further.

How NebuRedaction started

NebuLog is our lightweight framework for distributed logging. Its job is to collect log entries from many places and send them over the network to be stored and searched.

While discussing how to deploy it, our AI coding assistant, Claude Code, raised a point that any logging system lives or dies by: never put a secret in a log message. No passwords, no API keys, no tokens. The assistant called it a strict engineering discipline.

It is also a discipline that nothing enforced. NebuLog is a technical-interest project, and it had no mechanism to check that the rule was followed. Anyone writing a log line could slip a key in by accident, and because NebuLog's logs travel across a network, that mistake would not stay on one machine.

Frank's first idea was to let an AI model check each log entry. Two candidates were on the table, and neither fitted.

• Jev, from TypeSafe, is a decision model offered only as a proprietary cloud service. It cannot run locally, so checking a log line would mean sending that line to someone else's server. A check that has to happen before data leaves cannot itself require the data to leave.
• Laya, from Convai Innovations, is the open counterpart: a 421-million-parameter model under the Apache-2.0 licence that runs on your own machine. But it is a general-purpose decision engine, not trained for redaction, and it is heavy. According to a practical write-up on deploying it, each checkpoint's weights are around 840 MB and take roughly 1.7 GB of memory to load, and a single query takes tens of milliseconds on a GPU or an Apple chip. That is fine for a workstation, and far too much for a logging library that lives inside every application.

So the idea changed. Instead of running Laya in the logging path, use Laya as a teacher, and train a small student model built only for log redaction. Following imady's habit of naming projects "Nebu" plus their purpose, the project became NebuRedaction.

Secrets are now a target in their own right

A developer's keys usually live on their own computer: in environment variables, configuration files, command-line tools. That is risky enough. But keys also leave the computer in ways nobody intends. They end up in log lines, and increasingly they end up in conversations with AI tools. A developer pastes a failing command or a configuration file into an AI assistant to ask what went wrong, and the key goes with it.

Those conversations do not travel directly to the AI provider. They often pass through layers of software in between, such as a gateway like LiteLLM, an open-source library that lets applications reach many AI providers through one interface. A gateway sits exactly where prompts and API keys pass, and it usually holds the keys for every provider it talks to. A poisoned release of such a package runs with the same access as the application around it: it can read the keys stored on that machine and quietly send them elsewhere.

Frank's concern was not without reason. LiteLLM has been poisoned twice this year:

• March 2026. Malicious versions 1.82.7 and 1.82.8 were published. Researchers cited by Help Net Security have since linked data stolen in that campaign to roughly 2,500 organisations.
• 21 October 2026. A malicious version 1.107.0 was published to PyPI and to LiteLLM's Docker images. It was online for about two hours. Researchers report that it collected cloud credentials, AI service API keys, Git and SSH keys, saved browser logins and shell history, along with any upstream provider keys stored in a LiteLLM proxy's database, and sent them to an attacker-controlled address.

In both documented cases the malware did not read conversations. It ran as soon as Python started and took keys from where they were stored: configuration files, credential stores, shell history and the gateway's own database. The lesson is still the same: anything that sits in the path of your data, and anywhere a key has been copied to, is a place it can be stolen from. Every log line or prompt that carries a key adds one more such place.

The consequences of losing a key are rarely limited to one system:

• Attackers use it quickly. In the October incident, researchers at Wiz, as reported by Help Net Security, observed the attacker testing stolen cloud credentials, and checking whether they gave access to an AI service, within about four and a half hours.
• Someone else spends your money. A stolen cloud or AI key can run up large bills on your account before anyone notices.
• One key opens the next door. Keys to code repositories, databases or deployment pipelines let an attacker move further into a business, or poison software that other people depend on, which is how supply-chain attacks spread.
• Cleaning up is slow. Every key that might have been exposed has to be found, rotated, and checked for misuse.

The idea: check locally, before the data leaves

Most secret scanners run afterwards, over a repository or a log archive. For a system like NebuLog, afterwards is too late. By the time a scan finds a key, the log line has already crossed the network, been stored, and perhaps been forwarded to other systems. The copies exist, and each one has to be tracked down.

So the idea is to check in the program, on the same machine, just before the data leaves, and to mask anything that looks like a credential. A masked value looks like ***#a1b2. The four characters after the # are a short fingerprint of the original, so you can still tell whether two log lines involve the same token, but you cannot recover the token from it.

The same reasoning applies to anything else that leaves a program, including text on its way to an AI model.

Why the usual defences fall short

There are three common ways to keep secrets out of logs. Each helps, and each has a gap.

1. Engineering discipline. Train people never to log secrets, and review code for it. This is essential, but it relies on everyone getting it right every time. Secrets rarely get in on purpose. They arrive inside something else: a web address with a token in it, a configuration object printed while debugging, a full set of request headers.
2. Pattern rules, such as regular expressions. Match key names like password or token, or known formats such as a cloud provider's key prefix. Tools like gitleaks work this way. They are precise when the format is known, but they flag harmless values like passwordPolicy=strict, and they miss any secret that has no fixed shape. In our evaluation, a typical rule set found about 73% of the secrets in our synthetic test set, while also flagging about 11% of harmless values.
3. Entropy checks. Measure how random a string looks, and flag anything that looks like gibberish, the approach popularised by tools such as detect-secrets. It catches keys that have no known format, but randomness alone is a blunt signal. A 32-character hexadecimal string might be an API key, or it might be the checksum of a file, and an entropy check cannot tell the two apart.

How NebuRedaction works, briefly

NebuRedaction combines those signals in a small machine-learning model. Each log attribute becomes a handful of numbers describing its key name, the shape of its value (length, character mix, how random it looks) and the words around it. A set of small decision trees (a technique called gradient boosting [Friedman 2001; Ke et al. 2017]) scores those numbers and produces a probability that the value is a secret.

Context is what lets it do better than any one signal. A 32-character hex string under the key DATADOG_API_KEY scores high and is masked. The same string under the key digest scores low and is left alone.

The model learned its judgement from Laya through knowledge distillation. Laya labelled a large set of synthetic examples, and the small model was trained to imitate it. The technique is well established. What we added is a complete, measurable workflow around it.

The result is about 89 KB, runs offline with no dependencies, and judges one attribute in roughly 4 to 19 microseconds, small and fast enough to sit inside a logging library.

Where it applies, and where it does not

What it is built for. NebuRedaction checks structured log attributes, a key and a value with some context, inside .NET or Python programs. It is meant as one extra layer of protection, behind good discipline and good rules, never as the only one.

What we did not test. We built and measured it on log attributes, not on free text such as AI prompts or chat messages. Using it on prompts is a natural extension of the idea, but it is not something we have measured.

What it does not do. It does not detect personal information such as names, addresses or card numbers, and it does not look for secrets in source code. It cannot protect a key from malware already running on the same machine. The defences against that are the familiar ones: pin your dependency versions, delay upgrades by a few days, keep secrets off developer machines where you can, and rotate quickly when something goes wrong.

How well it works. We set the pass mark before training anything: find 10 percentage points more secrets than rules alone at a 0.1% false-alarm rate. We did not reach it. The best model scored 0.804 against a target of 0.834. More importantly, the model learned the formats our own data generator produces, not the general idea of what a secret is:

• on formats it had been trained on, it raised false alarms 0.075% of the time;
• on formats we deliberately held back from training, the false-alarm rate rose to roughly 49% for the released model;
• on a public dataset of credentials taken from real open-source code, it found only about 26% of them.

There were bright spots. On tricky cases, such as a key called password holding a harmless value, the small model raised far fewer false alarms than the larger model that taught it, and the .NET and Python implementations agreed with each other to within rounding error. The most reusable result may be the testing method itself: holding out whole families of formats is what exposed the weakness.

Why we are sharing it

NebuRedaction was a learning and research project. We built it to find out whether a model small enough to sit inside a logging library could learn to recognise secrets, and to work through the whole process, from generating data to measuring the result honestly. The answer, for now, is "not well enough".

We are not going to continue it. We have no plans to tune the model, add formats or maintain the packages. We are publishing it to share the idea and to start a conversation, in the hope that someone else can take it further. A few directions look worth trying:

• training on more varied, real-world data instead of generated data;
• testing the approach on prompts and chat messages, not only on log attributes;
• combining the model with rules, so each covers the other's blind spots;
• building a better held-out test than we managed.

The code, the technical report and the model are all there to be reused, forked or ignored.

Credits, licensing and sources

Licence. The NebuRedaction code, synthetic data generator, model weights and documentation are released under the MIT licence, copyright 2026 Frank Shen. The source is at github.com/imadyTech/NebuRedaction, and a pre-release package, imady.NebuRedaction, is available on NuGet. The licence file and a full list of third-party notices are in the repository.

Credits.

• Laya (convaiinnovations/laya, Apache-2.0) served as the teacher model whose judgements were distilled into the small model. Only synthetic and our own samples were shown to it, and its weights are not redistributed.
• CredData by Samsung (github.com/Samsung/CredData, Apache-2.0, with individual files under their original projects' licences) was used for evaluation only. It was not used for training, and none of its data is redistributed.
• gitleaks (MIT), detect-secrets by Yelp (Apache-2.0) and CredSweeper by Samsung (MIT) were used as comparison baselines and tools. Two gitleaks rule patterns are ported and attributed in the repository.

Product and company names mentioned here, including the secret formats of cloud vendors, are the trademarks of their owners. They are used only to describe formats or products. This project has no affiliation with, and no endorsement from, any of them.

Sources.

• Help Net Security: LiteLLM hit by second supply chain attack this year, users urged to rotate credentials (22 October 2026)
• Datadog Security Labs: LiteLLM supply chain compromise: what you need to know
• Junjian (军舰): Laya local deployment and integration best practices, Zhihu, 28 September 2026 (in Chinese). The Laya and Jev figures above are quoted from this article, not measured by us.
• NebuRedaction user guide and technical report
• Friedman J H. Greedy function approximation: a gradient boosting machine. The Annals of Statistics, 2001, 29(5): 1189-1232.
• Ke G, Meng Q, Finley T, et al. LightGBM: a highly efficient gradient boosting decision tree. Advances in Neural Information Processing Systems 30, 2017: 3146-3154.