Inspired by Andrej Karpathy's microgpt implementation, which I thing does an awesome job explaining the very foundations of LLMs making it super understandable, I wanted to see if fi could use the same philosophy to understand LLM output watermarking. Scott Aaronson's work on keyed Gavel sampling provided just the elegant way needed to understand 'watermarking'. Since an LLM generates text by sampling the next token from a probability distribution, the randomness used in that sampling can instead be derived from a secret key. The model and its underlying token probabilities remain the same, but across many token choices a statistical relationship begins to emerge.
I kept microgpt as-is and added a minimal implementation of the keyed sampler and statistical detector without external dependencies.
This is a deliberate educational experiment not production watermarking. I would be interested in feedback particularly on the implementation and statistical part of the experiment.
Inspired by Andrej Karpathy's microgpt implementation, which I thing does an awesome job explaining the very foundations of LLMs making it super understandable, I wanted to see if fi could use the same philosophy to understand LLM output watermarking. Scott Aaronson's work on keyed Gavel sampling provided just the elegant way needed to understand 'watermarking'. Since an LLM generates text by sampling the next token from a probability distribution, the randomness used in that sampling can instead be derived from a secret key. The model and its underlying token probabilities remain the same, but across many token choices a statistical relationship begins to emerge. I kept microgpt as-is and added a minimal implementation of the keyed sampler and statistical detector without external dependencies. This is a deliberate educational experiment not production watermarking. I would be interested in feedback particularly on the implementation and statistical part of the experiment.