Adapting language models with continual learning for temporal drifts

Large Language Models (LLMs) struggle to keep up with the fast-changing nature of real-world information, as their pre-trained knowledge quickly becomes outdated. This work addresses the challenge of keeping LLMs up to date with factual knowledge (adaptation) while avoiding forgetting the relevant existing knowledge. Leveraging temporally-aligned Wikipedia and Wikidata dumps, we extract a continuous data stream and evaluate the performance of an incrementally trained GPT-2 across different time periods. Additionally, we extend our analysis to real-world news data using the RealTimeData dataset, examining how LLMs respond to novel facts, such as the COVID-19 pandemic. Our methodology includes synthetic data generation and SmartReview, a continual learning strategy that avoids forgetting by rehearsing on a carefully selected subset of the old data. Experimental results highlight that pretrained models require continual learning and demonstrate the effectiveness of replay-based approaches in mitigating forgetting. In particular, SmartReview provides a strong replay-based baseline that limits forgetting and enhances adaptation. This work advances the study of continual learning in LLMs, offering insights into the development of more temporally-aware and reliable AI systems.