PEEK: The Paper, Explained

A beginner-friendly guide to PEEK — a fast, lightweight way to pick the few most useful frames from a video before an AI describes it. Every technical term is defined. Every concept is grounded in analogy.

Paper by Killian Steunou, Anas Filali Razzouki, Khalil Guetari, Mounîm A. El-Yacoubi & Yannis Tevissen (Télécom SudParis / Institut Polytechnique de Paris & Moments Lab, 2026) • Explainer published June 2026

The Big Picture

Imagine you want an AI to watch a two-minute video and write a one-sentence description of it — this task is called video captioningThe task of automatically writing a short natural-language description of what happens in a video, e.g. “A man plays the bagpipes in front of a crowd.”. There's a catch: the AI models that do this, called vision–language modelsA vision–language model (VLM) is an AI that takes in images (or video frames) together with text, and can reason about both — for example, looking at frames and producing a caption. Examples in this paper: Qwen2.5-VL and SmolVLM2., can only look at a small fixed number of framesA single still image pulled from a video. A two-minute clip contains thousands of frames; the model only gets to see a few of them. at once — often just one, two, four, or eight. A video has thousands of frames. So before the AI ever sees the video, something has to choose which few frames to hand it. That choice is the frame budgetThe fixed number of frames (k) a vision–language model is allowed to process for one video. Small budgets (k = 1, 2, 4, 8) are common because frames are expensive to process. problem, and it turns out to matter a lot.

The standard trick is uniform samplingSplitting the video into k equal-length time chunks and taking one frame (usually the middle one) from each. Simple and cheap, but blind to what's actually in the frames.: chop the video into equal time slices and grab the middle frame of each. It's cheap and surprisingly hard to beat — but it's content-blind. If the one moment that matters (a goal being scored, an instrument appearing) happens between the slices it grabs, uniform sampling simply misses it. The video below shows exactly this failure, using a real example from the paper:

Smarter, content-aware selectors do exist, but they come with their own problems. The problems PEEK sets out to solve:

1. Uniform sampling is content-blind. It treats a clip where the key event lasts a single instant exactly like one where evidence is spread evenly. Sometimes it works; sometimes it grabs nothing useful.
2. Existing smart selectors are slow. Methods that score frames by usefulness run a big neural network over many densely-sampled frames. Some add 65% to 212% to the total captioning time — expensive enough to cancel out the point of using few frames.
3. For captioning, you can't use the caption to choose. The most accurate way to find relevant frames is to compare them against the text you're looking for. But in captioning, the caption is exactly what you're trying to produce — you don't have it yet. So any method that needs the text up front is unusable at selection time.

Background Concepts

PEEK stacks together several standard ideas from modern AI. Here's each one, in plain language, before we see how they fit together.

Embeddings & dual encoders

Neural networks don't work with raw pixels or letters — they work with embeddingsA list of numbers (a vector) that represents the meaning of something — an image or a sentence. Things with similar meaning get vectors that point in similar directions.: lists of numbers (vectors) that capture meaning. A dual encoderA model with two separate encoders — one for images, one for text — trained so that an image and a matching caption land near each other in the same embedding space. CLIP and SigLIP are the famous examples. like CLIPContrastive Language–Image Pre-training (OpenAI, 2021): a dual encoder trained on millions of image–caption pairs so that matching images and texts get similar embeddings. The blueprint for SigLIP. or SigLIPA CLIP-style dual encoder from Google that uses a sigmoid training loss. SigLIP 2 is the version PEEK uses as its “teacher.” has two of these encoders — one for images, one for text — trained so that a picture and its matching caption produce vectors that point in nearly the same direction. That shared space is what lets you measure how well an image matches a sentence.

Cosine similarity

Once an image and a caption are both vectors, how do you measure their match? With cosine similarityA score from −1 to 1 measuring the angle between two vectors. 1 means they point the same way (very similar), 0 means unrelated. It ignores length and looks only at direction.: the cosine of the angle between them. A small angle (vectors pointing the same way) means high similarity; a wide angle means low similarity. Because it looks only at direction, not length, the vectors are first L2-normalizedRescaling a vector so its length is exactly 1, so only its direction matters. Standard before computing cosine similarity. (rescaled to length 1). This is how PEEK's expert scores each frame against the caption:

Knowledge distillation (teacher & student)

Knowledge distillationA training technique where a small “student” model is trained to copy the outputs of a larger, more capable “teacher” model — ending up nearly as good but far cheaper to run. is how you compress the skill of a big, expensive model into a small, cheap one. The big model (the teacher) produces outputs on training data; the small model (the student) is trained to reproduce them. Crucially, the student can be built to need less information than the teacher — which is the whole trick in PEEK: the teacher gets to see the caption, the student never does.

The Transformer & self-attention

PEEK's student is a small TransformerThe neural-network architecture behind almost all modern AI. It processes a whole sequence at once and lets every element “look at” every other element via self-attention. — the same architecture behind ChatGPT and Claude (we have a full explainer on the Transformer if you want the deep dive). The key feature it relies on is self-attentionA mechanism that lets each element in a sequence directly look at every other element and decide how relevant each one is. It's what lets PEEK judge a frame in the context of the whole video.: every frame in the video can directly look at every other frame and judge its importance in context. A frame isn't scored in isolation — it's scored relative to what surrounds it.

Zero-shot transfer

Zero-shot transferUsing a model on a new dataset or task it was never trained on, with no extra training. A strong test of whether the model learned something general rather than memorizing its training set. means taking a model trained on one dataset and testing it on a completely different one without any retraining. If it still works, it learned something general. PEEK is trained on one video dataset and then tested, untouched, on a second — a key check that it learned a real notion of “relevant frame” rather than memorizing quirks of its training videos.

How PEEK Works

At run time, PEEK is a simple pipeline that sits in front of any captioning model. Frames go in; a tiny scorer rates each one; a coverage rule picks the best few; those frames go to the captioner.

But the interesting part isn't the run-time pipeline — it's how that little scorer was taught. PEEK is built in two stages: an expert teacher that's too slow and too privileged to deploy, and a cheap student distilled from it.

Stage 1 — the Oracle teacher

The teacher is a frozen SigLIP 2 dual encoder. For each training video it does something the deployed system never can: it looks at the ground-truth caption. It embeds the caption with its text encoder, embeds every candidate frame with its vision encoder, and scores each frame by cosine similarity to the caption. The result is a relevance score for every frame — high for frames that visually match the caption, low for the rest. The scores are then min–max rescaledLinearly rescaling a set of numbers so the smallest becomes 0 and the largest becomes 1, while keeping their order intact. Used here to standardize the teacher's targets. to the range 0–1.

This teacher is called an OracleA model given privileged information unavailable at deployment (here, the correct caption). It's used to define an upper bound on performance and to generate training targets — never to run live. because it cheats: it knows the answer. You could never use it for real captioning, because the caption is the thing you're trying to generate. Its only job is to produce a teaching signal — a high-quality opinion about which frames matter.

Stage 2 — the query-free student

The student is what actually gets deployed. Each frame is first turned into a 512-number embedding by a frozen, lightweight MobileCLIP2A small, fast image encoder designed to run cheaply (even on phones). PEEK uses it to turn frames into embeddings; it stays frozen and is never trained. encoder. A small Transformer then reads the whole sequence of frame embeddings and outputs one logitA raw, unbounded score a model produces before it's turned into a probability or ranking. Higher means “more relevant” here. — a raw relevance score — per frame. It uses a fixed positional encodingExtra numbers added to each frame's embedding that encode its position in time, so the model knows the order of frames. PEEK uses fixed sine/cosine patterns. so it knows the order of frames in time.

The student is deliberately tiny: about 1.7 million trained parameters (13.1 million counting the frozen encoder it sits on). And it is query-freeNeeds no text query or caption at run time — it scores frames from pixels alone. The opposite of query-dependent methods that must be told what to look for.: it never sees any text. It learned its sense of “relevant” once, during training, by copying the teacher — and now applies it from pixels alone.

How It's Trained

Because both encoders are frozen, training is cheap: you precompute everything once, then train only the small scorer. Three stages:

Only the third box involves any learning, and only the ~1.7M-parameter scorer is updated. The expensive encoders are run once and their outputs cached, so training never has to touch a heavy vision model again.

How It Picks Frames

At run time the scorer rates every candidate frame. But you can't just keep the top-scoring frames overall — if the most relevant moment is one big peak, the top few frames will all be crowded around that single instant, giving the captioner four near-identical pictures and no sense of the rest of the video. PEEK fixes this with a rule called stratified argmaxSplit the video into k equal time-bins, then keep the single highest-scoring frame in each bin. Combines “pick the best frame” with “cover the whole timeline.”: split the timeline into k equal bins, and keep the best frame in each bin. That blends two instincts — pick content-rich frames, but also spread them across the video.

The selected frames are then put back in time order and handed to the captioner. The paper's ablation confirms this matters: stratified argmax beats raw top-k selection on every metric and every budget tested. For a budget of one frame, stratified argmax simply reduces to picking the single best-scoring frame in the whole video.

The Data

PEEK is trained on one dataset and evaluated on two — the second purely to test whether it generalizes.

Results

The headline: when the frame budget is tight, PEEK is the best content-aware selector that doesn't need text — clearly so at one frame. It's measured mostly with CIDErConsensus-based Image Description Evaluation: the standard 0–100+ score for captioning, rewarding captions that match human reference captions on the important, distinctive words. Higher is better., the standard captioning score, alongside BLEU-4A captioning/translation metric measuring overlap of 4-word sequences with reference captions. Higher is better., METEORA captioning metric that rewards matching words including synonyms and word-stems, not just exact matches. Higher is better., and ROUGE-LA metric based on the longest common word sequence shared with the reference caption. Higher is better..

One-frame captioning on ActivityNet (CIDEr)

With a budget of a single frame — the hardest, most selection-sensitive case — PEEK beats plain uniform sampling across every captioner. The Oracle column is the cheating teacher, shown only as an upper bound; it can't be deployed.

On the zero-shot MSR-VTT test — a dataset PEEK never trained on — it is again the best text-free selector for all four captioners at one frame, improving CIDEr by up to +2.68 points. That's the strongest evidence that it learned a transferable idea of relevance, not a memorized one.

It's cheap — that's the point

Accuracy isn't worth much if selecting frames costs more than captioning them. This is where PEEK separates from other content-aware methods. Measured over the full ActivityNet evaluation set:

PEEK is roughly 12× faster than CSTA and 40× faster than MaxInfo per segment, while beating both at low budgets. The other learned baselines optimize visual diversity (MaxInfo) or summarization importance (CSTA) — and the paper shows neither is the same thing as caption relevance, especially when you only get one frame.

An honest limitation

PEEK is not a universal replacement for uniform sampling. Its advantage is largest at one or two frames and shrinks as the budget grows: at four or eight frames, simply spreading frames evenly often catches the whole short clip just as well, and uniform or diversity-based methods sometimes edge ahead. Learned relevance selection helps most exactly when it's hardest — when you can afford only a frame or two. (See the bagpipes example in the opening video for the kind of case where it wins decisively.)

Final Quiz

Check your understanding of the key ideas.

Why It Matters

For people building things

PEEK is a drop-in front-end for any video captioner. It's text-free, so it works for captioning (where you have no query). It's independent of the downstream model, so the same checkpoint helps a 2-billion-parameter captioner and a 7-billion one alike. And it's so cheap — about 5% added time, a third of a second per clip — that there's little reason not to use it when frames are scarce. The authors point to obvious extensions: picking thumbnails, preview frames, or representative stills for any video product. For anyone running video AI at scale, “which frames do we even send?” is a real cost and quality lever, and PEEK answers it without a heavy model.

For the research community

The paper makes a clean, useful point: a lot of a privileged “cheating” signal can be distilled into a model that doesn't get the privilege. The Oracle, by reading the caption, measures how much caption-aware frame selection could help; PEEK then shows how much of that a pixels-only model can actually recover at deployment time. Two design choices turn out to matter and are worth carrying forward: training to reproduce a ranking (not exact scores), and enforcing temporal coverage at selection time. Just as important, the paper is honest about the boundary — learned relevance beats uniform sampling mainly in the low-budget regime, and uniform remains a strong baseline once you can afford several frames. That clarity about when a method helps is as valuable as the method.

The bigger picture

As AI moves from images to long video, the bottleneck shifts. Models can't watch everything, so the question becomes what to look at — and that decision is increasingly a learned one. PEEK is a small, sharp example of a pattern that keeps recurring in modern AI: use an expensive, privileged model once, offline, to teach a cheap one that you can run everywhere. The expert is the scaffolding; the student is the product. As video understanding spreads into search, moderation, robotics, and assistants, lightweight learned front-ends that decide where to spend a model's limited attention — without needing to know the question in advance — are likely to become standard plumbing.

Want the primary source? Read the full paper on arXiv or browse the code and pretrained checkpoint.