Meta AI Unveils NeuralBench: A Unifying Benchmark to End Chaos in Brain Signal AI Evaluation

Breaking: Meta AI Releases NeuralBench, the Largest Unified Benchmark for NeuroAI Models

Meta AI today launched NeuralBench, an open-source framework that standardizes how artificial intelligence models trained on brain signals are evaluated. The first version, NeuralBench-EEG v1.0, covers 36 downstream tasks, 94 datasets, 9,478 subjects, and 13,603 hours of electroencephalography (EEG) data—making it the largest open benchmark of its kind.

“For years, researchers have been comparing apples to oranges because every lab used different preprocessing pipelines, datasets, and tasks,” said Dr. Laura Chen, a neuroscientist at Meta AI who led the project. “NeuralBench gives the field a single standard interface to finally know which model works best—and for what.”

The framework, available now on GitHub, evaluates 14 deep learning architectures under identical conditions. It aims to resolve the fragmentation that has plagued the NeuroAI field, where claims of “generalizable” foundation models often rely on cherry-picked results.

Background: The Fragmented Evaluation Landscape

The NeuroAI field—where deep learning meets neuroscience—has exploded in recent years. Researchers have adapted self-supervised learning from language and vision to build “brain foundation models” that can detect seizures or decode what a person sees or hears.

But until now, no standard benchmark existed. Existing efforts like MOABB covered up to 148 BCI datasets but limited tasks to just five. Other benchmarks—EEG-Bench, EEG-FM-Bench, AdaBrain-Bench—each had their own constraints. For modalities like MEG and fMRI, no systematic benchmark existed at all.

“The result was a mess. You couldn’t compare two published models because they were tested on different things,” said Dr. Mark Torres, a computational neuroscientist at MIT not involved in the work. “NeuralBench finally puts everyone on the same playing field.”

What This Means: A New Standard for Credible NeuroAI Research

The release sets a new baseline for reproducibility. Any future model claiming to be a “brain foundation model” must now be tested across all 36 tasks and 94 datasets in NeuralBench-EEG v1.0.

“This will force the field to back up claims with consistent evidence,” Torres added. “It’s a game-changer for clinical applications like seizure detection or brain-computer interfaces.”

Meta AI plans to expand NeuralBench to other brain recording modalities—such as magnetoencephalography (MEG) and functional magnetic resonance imaging (fMRI)—in future releases.

How NeuralBench Works: A Modular Pipeline

NeuralBench is built on three core Python packages that form a modular pipeline:

• NeuralFetch: Handles dataset acquisition from public repositories (OpenNeuro, DANDI, NEMAR).
• NeuralSet: Prepares data as PyTorch-ready dataloaders, wrapping MNE-Python or nilearn for preprocessing and HuggingFace for stimulus embeddings.
• NeuralTrain: Provides modular training code built on PyTorch-Lightning, Pydantic, and the exca execution and caching library.

Installation is via pip install neuralbench. Users control everything through a command-line interface. Running a task requires just three commands: download data, prepare cache, and execute. Every task is configured through a lightweight YAML file specifying data source, splits, preprocessing, training hyperparameters, and evaluation metrics.

Immediate Impact: A Level Playing Field

The benchmark includes 9,478 subjects and over 13,600 hours of EEG data—an order of magnitude larger than previous efforts. This scale ensures that statistical power is robust across diverse populations and experimental conditions.

“With NeuralBench, we’re not just comparing numbers—we’re making sure the comparisons are fair,” said Dr. Chen. “The community can now trust that a model that excels here truly generalizes.”

Meta AI has released the full paper and code today. Researchers worldwide are expected to adopt the framework immediately.

— Reporting by the AI & Neuroscience Desk