Benchmarks module

This module provides popular continual learning benchmarks and generic facilities to build custom benchmarks.
  • Popular benchmarks (like SplitMNIST, PermutedMNIST, SplitCIFAR, …) are contained in the classic sub-module.

  • Dataset implementations are available in the datasets sub-module.

  • One can create new benchmarks by using the utilities found in the generators sub-module.

  • Avalanche uses custom dataset and dataloader implementations contained in the utils sub-module. More info can be found in this couple of How-Tos here and here.


Continual Learning Scenarios

Generic definitions for scenarios, streams and experiences. All the continual learning benchmarks are specific instantiations of these concepts.



Continual Learning benchmark.

OnlineCLScenario(original_streams[, ...])

ExModelCLScenario(original_benchmark, ...)

Ex-Model CL Scenario.

NCScenario(train_dataset, test_dataset, ...)

This class defines a "New Classes" scenario.

NIScenario(train_dataset, test_dataset, ...)

This class defines a "New Instance" scenario.


CLStream(name, exps_iter[, benchmark, ...])

A CL stream is a named iterator of experiences.

EagerCLStream(name, exps[, benchmark, ...])

A CL stream which is a named list of experiences.

ClassificationStream(name, benchmark, *[, ...])


CLExperience([current_experience, origin_stream])

Base Experience.

ClassificationExperience(*args, **kwargs)

Definition of a classification experience.

NCExperience(origin_stream, current_experience)

Defines a "New Classes" experience.

NIExperience(origin_stream, current_experience)

Defines a "New Instances" experience.

OnlineCLExperience([current_experience, ...])

Online CL (OCL) Experience.

ExModelExperience(expert_model[, ...])

Ex-Model CL Experience.

ExperienceAttribute(value[, use_in_train, ...])

Experience attributes are used to define data belonging to an experience which may only be available at train or eval time.

Classic Benchmarks

The classic benchmarks sub-module covers all mainstream benchmarks. Expect this list to grow over time!

Benchmarks based on the CORe50 dataset.

CORe50(*[, scenario, run, object_lvl, mini, ...])

Creates a CL benchmark for CORe50.

Benchmarks based on the CIFAR-10 and CIFAR-100 datasets.

SplitCIFAR10(n_experiences, *[, ...])

Creates a CL benchmark using the CIFAR10 dataset.

SplitCIFAR100(n_experiences, *[, ...])

Creates a CL benchmark using the CIFAR100 dataset.

SplitCIFAR110(n_experiences, *[, seed, ...])

Creates a CL benchmark using both the CIFAR100 and CIFAR10 datasets.

Benchmarks based on the Caltech-UCSD Birds 200 dataset.

SplitCUB200([n_experiences, ...])

Creates a CL benchmark using the Cub-200 dataset.

Benchmarks based on the EndlessCLSim derived datasets.

EndlessCLSim(*[, scenario, patch_size, ...])

Creates a CL scenario for the Endless-Continual-Learning Simulator's derived datasets, or custom datasets created from the Endless-Continual-Learning-Simulator's `standalone application <>`__.

Benchmarks based on the Fashion MNIST dataset.

SplitFMNIST(n_experiences, *[, ...])

Creates a CL benchmark using the Fashion MNIST dataset.

Benchmarks based on the ImageNet ILSVRC-2012 dataset.

SplitImageNet(dataset_root, *[, ...])

Creates a CL benchmark using the ImageNet dataset.

SplitTinyImageNet([n_experiences, ...])

Creates a CL benchmark using the Tiny ImageNet dataset.

Benchmarks based on the iNaturalist-2018 dataset.

SplitInaturalist(*[, super_categories, ...])

Creates a CL benchmark using the iNaturalist2018 dataset.

Benchmarks based on the MNIST dataset.

SplitMNIST(n_experiences, *[, ...])

Creates a CL benchmark using the MNIST dataset.

PermutedMNIST(n_experiences, *[, seed, ...])

Creates a Permuted MNIST benchmark.

RotatedMNIST(n_experiences, *[, seed, ...])

Creates a Rotated MNIST benchmark.

Benchmarks based on the Omniglot dataset.

SplitOmniglot(n_experiences, *[, ...])

Creates a CL benchmark using the OMNIGLOT dataset.

PermutedOmniglot(n_experiences, *[, seed, ...])

Creates a Permuted Omniglot benchmark.

RotatedOmniglot(n_experiences, *[, seed, ...])

Creates a Rotated Omniglot benchmark.

Benchmarks based on the OpenLORIS dataset.

OpenLORIS(*[, factor, train_transform, ...])

Creates a CL benchmark for OpenLORIS.

Benchmarks based on the Stream-51, dataset.

CLStream51(*[, scenario, seed, eval_num, ...])

Creates a CL benchmark for Stream-51.

Benchmarks based on the CLEAR dataset.

CLEAR(*[, data_name, evaluation_protocol, ...])

Creates a Domain-Incremental benchmark for CLEAR 10 & 100 with 10 & 100 illustrative classes and an n+1 th background class.

Benchmarks for learning from pretrained models or multi-agent continual learning scenarios. Based on the Ex-Model paper. Pretrained models are downloaded automatically.

ExMLMNIST([scenario, run_id])

ExML scenario on MNIST data.

ExMLCoRE50([scenario, run_id])

ExML scenario on CoRE50.

ExMLCIFAR10([scenario, run_id])

ExML scenario on CIFAR10.


The datasets sub-module provides PyTorch dataset implementations for datasets missing from the torchvision/audio/* libraries. These datasets can also be used in a standalone way!

CORe50Dataset(root, ~pathlib.Path]] = None, *)

CORe50 Pytorch Dataset

CUB200(root, ~pathlib.Path]] = None, *[, ...])

Basic CUB200 PathsDataset to be used as a standard PyTorch Dataset.

EndlessCLSimDataset([root, scenario, ...])

Endless Continual Leanring Simulator Dataset

INATURALIST2018([root, split, transform, ...])

INATURALIST Pytorch Dataset

MiniImageNetDataset(imagenet_path, ...[, loader])

The MiniImageNet dataset.

Omniglot(root[, train, transform, ...])

Custom class used to adapt Omniglot (from Torchvision) and make it compatible with the Avalanche API.

OpenLORIS(root, ~pathlib.Path]] = None, *[, ...])

OpenLORIS Pytorch Dataset

Stream51(root, ~pathlib.Path]] = None, *[, ...])

Stream-51 Pytorch Dataset

TinyImagenet(root, ~pathlib.Path]] = None, ...)

Tiny Imagenet Pytorch Dataset

CLEARDataset([root, data_name, download, ...])

CLEAR Base Dataset for downloading / loading metadata

Benchmark Generators

The generators sub-module provides a lot of functions that can be used to create a new benchmark.
This set of functions tries to cover most common use cases (Class/Task-Incremental, Domain-Incremental, …) but it also allows for the creation of entirely custom benchmarks (based on lists of tensors, on file lists, …).

nc_benchmark(train_dataset, test_dataset, ...)

This is the high-level benchmark instances generator for the "New Classes" (NC) case.

ni_benchmark(train_dataset, test_dataset, ...)

This is the high-level benchmark instances generator for the "New Instances" (NI) case.

dataset_benchmark(train_datasets, ...[, ...])

Creates a benchmark instance given a list of datasets.

filelist_benchmark(root, train_file_lists, ...)

Creates a benchmark instance given a list of filelists and the respective task labels.

paths_benchmark(train_lists_of_files, ...[, ...])

Creates a benchmark instance given a sequence of lists of files.

tensors_benchmark(train_tensors, test_tensors, *)

Creates a benchmark instance given lists of Tensors.

Avalanche offers utilities to adapt a previously instantiated benchmark object.
More utilities to come!

data_incremental_benchmark(...[, shuffle, ...])

High-level benchmark generator for a Data Incremental setup.

benchmark_with_validation_stream(...[, ...])

Helper that can be used to obtain a benchmark with a validation stream.

Utils (Data Loading and AvalancheDataset)

The custom dataset and dataloader implementations contained in this sub-module are described in more detailed in the How-Tos about “data loading and replay” <> and “Avalanche Dataset” <>.

TaskBalancedDataLoader(data, ...[, ...])

Task-balanced data loader for Avalanche's datasets.

GroupBalancedDataLoader(datasets, ...[, ...])

Data loader that balances data from multiple datasets.

ReplayDataLoader(data, memory, ...[, ...])

Custom data loader for rehearsal/replay strategies.

GroupBalancedInfiniteDataLoader(datasets[, ...])

Data loader that balances data from multiple datasets emitting an infinite stream.

AvalancheDataset(dataset, *[, transform, ...])

The Dataset used as the base implementation for Avalanche.

AvalancheSubset(dataset[, indices, ...])

A Dataset that behaves like a PyTorch

AvalancheTensorDataset(*dataset_tensors[, ...])

A Dataset that wraps existing ndarrays, Tensors, lists.

AvalancheConcatDataset(datasets, *[, ...])

A Dataset that behaves like a PyTorch