BiRNA-BERT Allows Efficient RNA Language Modeling with Adaptive Tokenization

[Download Paper] [Source Code]

RNA language models usually have to truncate long sequences, and NLP-style tokenization does not transfer cleanly to nucleotide-level tasks. BiRNA-BERT is a 117M-parameter encoder trained on 36 million RNA sequences with an adaptive dual tokenizer: nucleotide-level encoding for short, structure-sensitive inputs and BPE for long sequences. That lets the model handle arbitrarily long RNAs without truncation, match much larger models on short-sequence tasks, and set strong results on long-sequence downstream work at a fraction of the pretraining compute.