Skip to content

HyenaDNA — Nguyen et al. (Stanford Hazy Research + Harvard + Mila)

One-line description — A decoder-only causal genomic LM built on Hyena implicit-convolution operators (replacing self-attention with sub-quadratic long convolutions) that handles context lengths up to 1 million single-nucleotide tokens — a 500× context jump over Transformer-based DNA models, with the first demonstration of in-context learning in genomics.

What it does

HyenaDNA is a decoder-only causal language model for DNA that abandons the transformer's quadratic-attention bottleneck in favour of Hyena operators — a stack of implicit long convolutions parameterised by small MLPs, combined with data-controlled gating. The headline result is operating on 1,000,000 single-nucleotide tokens in a single forward pass, with full global context at every layer, at ~160× the throughput of a Transformer at matched length. Outputs are next-token logits for autoregressive sequence modelling and fixed-size pooled embeddings for downstream classification. It was the first genomic FM to demonstrate in-context learning (soft-prompt tuning, k-shot pattern induction) — the methodological seed that later motivated Evo and the JHU "genomic ICL" workshop paper at ICLR 2026.

How it works

Architecture. Decoder-only causal sequence model. Each layer is a Hyena operator: an implicit convolution where the kernel is generated by a small feedforward network conditioned on positional features (not stored as fixed weights). The operator combines (1) data-controlled gating (multiplicative interactions between input projections, similar to GLU), (2) an implicit long convolution computed in O(N log N) via FFT, and (3) shortcut connections. This replaces self-attention entirely — no attention layers in the model — and is the key to sub-quadratic scaling.

Tokenisation. Single-nucleotide tokens — vocabulary size 12 (A, C, G, T, N, plus special tokens for padding, masking, BOS, EOS, and reverse-complement markers). This is the strongest possible resolution for SNV-effect prediction: no information is lost at any base.

Pretraining corpus. Human reference genome HG38, approximately 3.2 billion base pairs (one human genome). This is dramatically less data than NT (174 Gb multispecies) or DNABERT-2 (32 Gb multispecies) — the parameter / data efficiency is part of the methodological argument.

Pretraining objective. Causal next-token prediction (standard autoregressive LM). No masking, no bidirectional context. This unlocks generation and in-context learning that NT/DNABERT-2 cannot natively support.

Context length variants. Six checkpoints released, scaling depth (2–8 layers), width (128–256 hidden dim), and pretraining sequence length: - tiny-1k-seqlen — 1,024-token context, smallest (~0.4M params) - tiny-1k-seqlen-d256 — 1,024 tokens, wider hidden dim - small-32k-seqlen — 32,768-token context - medium-160k-seqlen — 163,840 tokens - medium-450k-seqlen — ~450k tokens - large-1m-seqlen — 1,000,000-token context — the headline variant; trained on A100-80GB GPUs

Parameter counts across the family span roughly 0.4M – 6.6M (the largest configuration is still under 7M params — orders of magnitude smaller than NT and Evo). The dnaGrinder follow-up paper documents the exact configuration table.

Pretraining compute. Headline efficiency claim: 1500× fewer parameters and 3200× less pretraining data than NT-2.5B (1.6M params vs 2.5B; 1 human ref genome vs 3,202 in 1000G). Training throughput is up to 160× faster than a same-context Transformer for long sequences. Total pretraining wall-clock for large-1m-seqlen is single-digit A100-days (per repo training logs).

Downstream tasks. Evaluated on (1) the 18-task Nucleotide Transformer benchmark suite (histone marks, regulatory annotation), (2) the GenomicBenchmarks suite of 8 short-context classification tasks, and (3) a novel species classification at very long context task that requires >100 kb — designed to stress-test long-range modelling.

Headline benchmarks

  • GenomicBenchmarks (8 tasks): HyenaDNA exceeds prior SOTA on 7 of 8 datasets, average +10 absolute accuracy points over the previous best (a DNABERT-1 / CNN baseline).
  • Nucleotide Transformer 18-task suite: HyenaDNA-32kb reaches SOTA on 12 of 18 tasks — competitive with NT-500M but loses to NT-2.5B-multispecies on most regulatory tasks where multispecies-pretraining diversity matters more than long context. This is the canonical "single human genome, 1 Mb context" vs "850 species, 6 kb context" trade-off datapoint.
  • Species classification at 100 kb – 1 Mb: HyenaDNA-large-1m succeeds where Transformer baselines fail to even fit the input — a "context-as-capability" demonstration rather than a head-to-head comparison.
  • In-context learning (soft-prompt tuning): First demonstration of k-shot pattern induction in genomic LMs — a small soft-prompt trained on as few as 16 examples generalises to held-out classification — the methodological seed that motivated the JHU Sci4DL paper on ICL in Evo2 at ICLR 2026 (Breslow et al.).
  • vs DNABERT-2: DNABERT-2 wins on short-range classification (promoter, splice-site, TFBS — GUE benchmark). HyenaDNA wins on tasks that genuinely require >10 kb. Confirms the two-axis design space (tokenisation × context).
  • vs AlphaGenome: HyenaDNA is a generative causal LM; AlphaGenome is a supervised sequence-to-track regressor. They are not direct competitors on variant-effect prediction — AlphaGenome dominates regulatory-variant scoring, but HyenaDNA can sample novel sequences and serve as an ICL backbone, which AlphaGenome cannot.

Why it matters

HyenaDNA established three results that shaped the 2024–2026 genomic-FM landscape:

  1. Sub-quadratic operators are sufficient for DNA. Hyena (and later Mamba / SSM derivatives like Caduceus) replaced attention as the default long-context primitive in DNA. Evo and Evo2 (Arc Institute) directly inherit the Hyena-style implicit-convolution backbone — Evo2's 1 Mb context at 40B params is HyenaDNA's recipe at scale.
  2. Single-nucleotide tokenisation works. Before HyenaDNA, the field assumed k-mer / BPE compression was necessary; HyenaDNA proved a vocabulary of 12 is enough if the operator is the right shape. AlphaGenome's per-base decoder, Evo's single-base tokenisation, and the NTv3 single-base variant all trace back to this argument.
  3. In-context learning is possible in DNA models. This is the methodological hinge that converts genomic FMs from "fine-tuning backbones" into "queryable knowledge bases" — and is what the JHU group is now systematically probing in Evo2 at ICLR 2026.

Known limitations

  • Single-species pretraining. HG38-only pretraining underperforms multi-species models (NT-2.5B-multispecies) on regulatory tasks where evolutionary conservation is the signal. The Caduceus follow-up (NeurIPS 2024) and Evo (Arc Institute 2024) both fix this by adding multi-species data.
  • No bidirectional context. Causal LM means each token sees only its left context. For variant-effect scoring this requires either dual-direction inference (forward + reverse-complement) or post-hoc symmetrisation; HyenaDNA does ship reverse-complement augmentation but it's not as principled as bidirectional MLM (NT / DNABERT-2) or the U-Net-with-transformer-bottleneck of AlphaGenome.
  • Small parameter count. At ~6.6M parameters maximum, HyenaDNA has limited capacity for storing complex regulatory grammar. Evo (7B) and Evo2 (40B) scale the same Hyena recipe ~10,000× larger and dominate HyenaDNA on most downstream tasks accordingly.
  • Hyena kernel is data-dependent in a subtle way. The implicit-convolution kernels are parameterised by positional features and can struggle with highly periodic / repetitive sequences (centromeres, satellite DNA) — a known failure mode of long convolutions.
  • In-context learning is unstable at small scale. The ICL demonstration in the paper uses soft-prompt tuning rather than zero-shot prompting; the JHU ICLR 2026 work shows that organically emergent ICL only appears at Evo2-scale (40B params).
  • Memory for 1M-token inference. Even with FlashFFT-style optimisations, forward-pass memory at 1 Mb context requires A100-40GB minimum.

Sources