Harnessing the Unseen: The Hidden Influence of Intrinsic Knowledge in Long-Context Language Models

[Download Paper] [Source Code]

Long-context evaluation usually asks whether a model can retrieve information from the prompt. That misses a second skill: recalling facts stored in parameters and combining them with the context. We show that parametric knowledge matters more as context grows, and that stronger extrinsic retrieval can actually suppress parametric recall. Hybrid Needle-in-a-Haystack tests both abilities together. On that test, Qwen-2.5 models combine the two much more effectively than Llama-3.1 models of similar or larger size.