Lookups vs. subsearches, finally

I reached for a subsearch out of habit for years. Then I actually measured it, and the habit mostly went away.

The rule of thumb

A subsearch runs first, in full, and hands its result back as a filter. That is fine when the result is small and cheap to compute, and painful when it is neither.

• Lookup — the reference set is stable and reused. Load it once, join at search time.
• Subsearch — the reference set is small, dynamic, and only needed for this one search.

What the numbers said

The search that pushed me over was an allow-list join. As a subsearch it hit the default 10,000-row limit and silently truncated:

index=web [ search index=allowlist | fields ip ]

The same thing as a lookup was both correct and faster, because the reference set never had to be recomputed per run:

index=web | lookup allowlist ip OUTPUT owner

A truncated subsearch does not error — it just returns the wrong answer quietly. That alone is reason enough to prefer a lookup once the set stops being tiny.

Where subsearches still win

For a genuinely dynamic filter — “the ten hosts that errored in the last hour” — a subsearch is exactly right. Compute it fresh, use it once, move on.