Exploring Alternative Language Model Architectures through HPC
Language models use information from earlier text to predict what comes next. How this information is retained and updated remains an important research question in artificial intelligence, particularly as researchers explore alternatives to conventional language-model architectures.
This proof of concept (PoC) examines Transitor, a language model developed by Felix Koba, which learns transitions between internal representations of text. The model maintains a compact state based on transition matrices, using them to predict the next representation and updating them as new information becomes available. Training and evaluation were carried out using computing resources available through HPC Montenegro.
CHALLENGE
Developing alternative language-model architectures requires investigating how information can be efficiently represented, retained, and updated during text processing. Beyond proposing a new mechanism, researchers must determine whether it can be trained at a meaningful scale and how its performance compares with established approaches.
The main challenge was to evaluate Transitor’s transition-based memory mechanism through large-scale training and standardized language benchmarks. This required substantial computational resources and a systematic assessment of the architecture’s strengths and limitations.
SOLUTION
Transitor uses several small transition memories operating in parallel. The model adjusts their contributions and combines their outputs with a separate representation of the current text. This enables it to use both information carried in memory and information available at the current processing step.
A 12-layer model with 398.8 million parameters was trained from scratch on 15 billion tokens from FineWeb-Edu, using an NVIDIA L40 GPU.
The trained model was evaluated across eight language benchmarks. Its performance was compared with published results for Transformer++, Mamba2, DeltaNet, and Gated DeltaNet at a similar training scale. The reference results were taken from the Titans study (Table 6), covering models with approximately 400 million parameters trained on 15 billion tokens.
RESULTS
Transitor achieved results close to the reference models on PIQA, SocialIQA, and BoolQ, while showing lower performance on tasks including LAMBADA, HellaSwag, and ARC. These findings provide an initial assessment of the proposed architecture and help identify areas where further improvements are needed.
Figure shows benchmark results for Transitor compared with published results for Transformer++, Mamba2, DeltaNet, and Gated DeltaNet. Sources: Transitor evaluations and Titans, Table 6 (400M/15B).
The trained model was also tested through text generation. The following example illustrates how it continues a short prompt:
Prompt: High-performance computing (HPC) can help researchers in Montenegro by
Model continuation:
High-performance computing (HPC) can help researchers in Montenegro by providing a full range of support services to help facilitate research and educational activities, including the development of new software tools and new hardware.
The model’s continuation is reproduced without editing.
BENEFITS
- Access to HPC Montenegro’s computing resources enabled training and evaluation of an alternative language-model architecture at a substantial computational scale.
- The project produced a trained model with 398.8 million parameters, using 15 billion training tokens.
- Benchmark comparisons provided insights into the architecture’s performance relative to established language-model approaches.
- The experiments identified strengths and limitations that can guide further architectural improvements and research.
- The work demonstrates how HPC infrastructure can support computationally demanding AI research and the exploration of new machine-learning approaches.
This proof of concept illustrates the role of HPC Montenegro in supporting advanced AI research, enabling researchers to develop, train, and evaluate new model architectures using high-performance computing resources.