BiRNA-BERT: Efficient RNA language modeling with adaptive tokenization
Date:
Invited talk at Mila Multiomics Reading Group, Montreal, Canada
I presented BiRNA-BERT at Mila’s Multiomics Reading Group in February 2025. The talk is a practical tour of the modeling choice that matters most here: tokenization.
RNA models inherit a transformer habit from NLP — treat the sequence as a list of tokens, then hope the context window is long enough. For RNA that fails in two directions at once. Nucleotide-level tasks (secondary structure, torsion angles) want character resolution. Full-length non-coding RNAs can be far longer than a BERT window, so the usual fix is truncation, which throws away distal context the biology actually uses.
BiRNA-BERT’s answer is an adaptive dual tokenizer. Short or structure-sensitive inputs stay at nucleotide resolution. Long inputs switch to BPE so the same 117M-parameter encoder can run without chopping the sequence. Pretraining on 36 million coding and non-coding RNAs is enough to match much larger models on short tasks and to do well on long-sequence downstream work, at a small fraction of the pretraining compute.
The paper is now in Nature Communications Biology. Code and weights are on GitHub.