Inferring Cognitive Workload From Symbolic Gaze Sequences Using Large Language Models

This work proposes a new method for real-time assessment of cognitive workload by combining symbolic representation of eye movement behavior with natural language inference through Large Language Models (LLMs). While prior gaze-based workload estimators typically rely on handcrafted numerical features, domain-specific deep networks, or feature-to-text serializations, they do not treat gaze trajectories themselves as symbolic linguistic inputs for LLM reasoning; this work addresses that gap by introducing an interpretable, language-native representation of eye-movement behavior for workload inference. Raw eye-tracking signals are encoded into symbolic sequences that capture fixations, saccades, regressions, blinks, and pupil variations. These sequences are then embedded into LLM prompts, allowing the model to determine the workload level while simultaneously providing an explanatory description in natural language. The pipeline operates online and in real time and does not require specialized hardware. It combines established segmentation techniques with interpretable symbolic encoding and prompt-based reasoning. We validate our method on the COLET dataset, where the zero-shot LLM-based classifier achieves 70% accuracy, a macro F1-score of 0.40, and a Cohen's κ of 0.23, outperforming both a traditional random forest baseline and a multilayer perceptron (MLP) classifier trained on the same handcrafted gaze features, while preserving temporal responsiveness and human-readable output. The system accommodates both cloud-hosted and local LLMs, incorporates a comprehensive logging system, and provides real-time visualization capabilities. Its modular architecture facilitates smooth integration into adaptive user interfaces and cognition-aware environments.