I focus on developing open-source LLMs that are transparent and compliant with current legal frameworks. This work provides a foundation for society to build trustworthy AI products and services while enabling researchers to better understand the benefits and risks of LLM-based systems. Key technical challenges include quantifying the impact of training data governance decisions, understanding memorisation risks in large-scale models, and developing infrastructure that democratises access to language modelling research across different computational scales.

I advance neural architecture research through fast weight programmers such as the DeltaNet, which I invented together with Kazuki Irie and Jürgen Schmidhuber, and which contributes to the most significant architectural innovation since the rise of the Transformer. Similar to linear RNNs, like Mamba or RWKV, it offers enhanced efficiency and generality compared to attention-based architectures. Recently, DeltaNet became a core component of major releases from some of the leading AI labs in the world: Alibaba's Qwen3-Next, Qwen3.5, Qwen3.6, and Qwen3.8 models use Gated DeltaNet, and Moonshot AI's Kimi Linear and Kimi K3 use Kimi Delta Attention, a DeltaNet variant. A close DeltaNet variant is also used in OLMo Hybrid from the Allen Institute for AI. This research explores self-referential weight matrices that learn to modify themselves, block-recurrent mechanisms that maintain global context while enabling parallel training, and novel activation functions derived through mathematical integration principles.

I investigate fundamental questions around LLM scaling and generalisation. In particular, how to train these systems more efficiently and enable them to generalise beyond their current limitations. This includes exploring self-modifying neural networks as a pathway toward more general AI systems. Technical focus areas include developing compute-optimal training strategies through adaptive scaling laws, investigating the role of tokenisation and language imbalance in cross-lingual transfer, and creating hybrid neuro-symbolic approaches that augment classical algorithms with neural components for robust generalisation.