上下文窗口

在生成式人工智慧的領域中，上下文窗口（context window）是指大型语言模型（LLM）或是小型語言模型（SLM）中，自然语言处理單次所能處理的最大輸入長度。在這類语言模型的開發過程中，擴充上下文窗口一直是主要目標之一^[1]^[2]。上下文窗口的長度會用token為單位來衡量。以Gemini LLM為例，在2025年時，其上下文窗口可以到二百萬個token^[3]。然而，隨著技術競爭，部分開源模型（如 Llama 4 Scout）已宣稱能支援更長的序列。若語言模型是以Transformer架构為基礎的人工神经网络，上下文窗口會影響語言模型的性能以及能力。

有些模型的上下文窗口大小會受到訓練過程中輸入序列的長度所限制^[4]。透過採用特定的注意力機制，可以讓LLM可解釋的序列長度遠長於訓練時所見的長度^[5]。

參考資料

^ Ratner, Nir; Levine, Yoav; Belinkov, Yonatan; Ram, Ori; Magar, Inbal; Abend, Omri; Karpas, Ehud; Shashua, Amnon; Leyton-Brown, Kevin; Shoham, Yoav. Parallel Context Windows for Large Language Models. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023: 6383–6402. doi:10.18653/v1/2023.acl-long.352.
^ Dong, Zican; Li, Junyi; Men, Xin; Zhao, Wayne Xin; Wang, Bingning; Tian, Zhen; Chen, Weipeng; Wen, Ji-Rong. Exploring context window of large language models via decomposed positional vectors. Proceedings of the 38th International Conference on Neural Information Processing Systems (Curran Associates Inc.). 10 December 2024, 37: 10320–10347.
^ Yeung, Ken. Google announces Gemini 1.5 Flash, a rapid multimodal model with a 1M context window. VentureBeat. 2024-05-14 [2025-08-26]. （原始内容存档于2025-01-28）（美国英语）.
^ Wu, S. BloombergGPT: A Large Language Model for Finance. 2023. arXiv:2303.17564  [LG].
^ Press, Ofir. Train short, test long: Attention with linear biases enables input length extrapolation. 2021. arXiv:2108.12409  [LG].

[1] Ratner, Nir; Levine, Yoav; Belinkov, Yonatan; Ram, Ori; Magar, Inbal; Abend, Omri; Karpas, Ehud; Shashua, Amnon; Leyton-Brown, Kevin; Shoham, Yoav. Parallel Context Windows for Large Language Models. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023: 6383–6402. doi:10.18653/v1/2023.acl-long.352.

[2] Dong, Zican; Li, Junyi; Men, Xin; Zhao, Wayne Xin; Wang, Bingning; Tian, Zhen; Chen, Weipeng; Wen, Ji-Rong. Exploring context window of large language models via decomposed positional vectors. Proceedings of the 38th International Conference on Neural Information Processing Systems (Curran Associates Inc.). 10 December 2024, 37: 10320–10347.

[3] Yeung, Ken. Google announces Gemini 1.5 Flash, a rapid multimodal model with a 1M context window. VentureBeat. 2024-05-14 [2025-08-26]. （原始内容存档于2025-01-28）（美国英语）.

[4] Wu, S. BloombergGPT: A Large Language Model for Finance. 2023. arXiv:2303.17564  [LG].

[5] Press, Ofir. Train short, test long: Attention with linear biases enables input length extrapolation. 2021. arXiv:2108.12409  [LG].

[1]

[2]

[3]

[4]

[5]