Benchmarks module

This module provides popular continual learning benchmarks and generic facilities to build custom benchmarks.

Popular benchmarks (like SplitMNIST, PermutedMNIST, SplitCIFAR, …) are contained in the classic sub-module.
Dataset implementations are available in the datasets sub-module.
One can create new benchmarks by using the utilities found in the generators sub-module.
Avalanche uses custom dataset and dataloader implementations contained in the utils sub-module. More info can be found in this couple of How-Tos here and here.

avalanche.benchmarks

Classic Benchmarks 

The classic benchmarks sub-module covers all mainstream benchmarks. Expect this list to grow over time!

CORe50-based benchmarks 

Benchmarks based on the CORe50 dataset.

CORe50(*[, scenario, run, object_lvl, mini, ...])

Creates a CL benchmark for CORe50.

CIFAR-based benchmarks 

Benchmarks based on the CIFAR-10 and CIFAR-100 datasets.

`SplitCIFAR10`(n_experiences, *[, ...])	Creates a CL benchmark using the CIFAR10 dataset.
`SplitCIFAR100`(n_experiences, *[, ...])	Creates a CL benchmark using the CIFAR100 dataset.
`SplitCIFAR110`(n_experiences, *[, seed, ...])	Creates a CL benchmark using both the CIFAR100 and CIFAR10 datasets.

CUB200-based benchmarks 

Benchmarks based on the Caltech-UCSD Birds 200 dataset.

SplitCUB200([n_experiences, ...])

Creates a CL benchmark using the Cub-200 dataset.

EndlessCLSim-based benchmarks 

Benchmarks based on the EndlessCLSim derived datasets.

EndlessCLSim(*[, scenario, patch_size, ...])

Creates a CL scenario for the Endless-Continual-Learning Simulator's derived datasets, which are available at: https://zenodo.org/record/4899267, or custom datasets created from the Endless-Continual-Learning-Simulator's standalone application, available at: https://zenodo.org/record/4899294. Both are part of the publication of `A Procedural World Generation Framework for Systematic Evaluation of Continual Learning (https://arxiv.org/abs/2106.02585).

FashionMNIST-based benchmarks 

Benchmarks based on the Fashion MNIST dataset.

SplitFMNIST(n_experiences, *[, ...])

Creates a CL benchmark using the Fashion MNIST dataset.

ImageNet-based benchmarks 

Benchmarks based on the ImageNet ILSVRC-2012 dataset.

`SplitImageNet`(dataset_root, *[, ...])	Creates a CL benchmark using the ImageNet dataset.
`SplitTinyImageNet`([n_experiences, ...])	Creates a CL benchmark using the Tiny ImageNet dataset.

iNaturalist-based benchmarks 

Benchmarks based on the iNaturalist-2018 dataset.

SplitInaturalist(*[, super_categories, ...])

Creates a CL benchmark using the iNaturalist2018 dataset.

MNIST-based benchmarks 

Benchmarks based on the MNIST dataset.

`SplitMNIST`(n_experiences, *[, ...])	Creates a CL benchmark using the MNIST dataset.
`PermutedMNIST`(n_experiences, *[, seed, ...])	Creates a Permuted MNIST benchmark.
`RotatedMNIST`(n_experiences, *[, seed, ...])	Creates a Rotated MNIST benchmark.

Omniglot-based benchmarks 

Benchmarks based on the Omniglot dataset.

`SplitOmniglot`(n_experiences, *[, ...])	Creates a CL benchmark using the OMNIGLOT dataset.
`PermutedOmniglot`(n_experiences, *[, seed, ...])	Creates a Permuted Omniglot benchmark.
`RotatedOmniglot`(n_experiences, *[, seed, ...])	Creates a Rotated Omniglot benchmark.

OpenLORIS-based benchmarks 

Benchmarks based on the OpenLORIS dataset.

OpenLORIS(*[, factor, train_transform, ...])

Creates a CL benchmark for OpenLORIS.

Stream51-based benchmarks 

Benchmarks based on the Stream-51, dataset.

CLStream51(*[, scenario, seed, eval_num, ...])

Creates a CL benchmark for Stream-51.

Datasets 

The datasets sub-module provides PyTorch dataset implementations for datasets missing from the torchvision/audio/* libraries. These datasets can also be used in a standalone way!

`CORe50Dataset`(root, pathlib.Path]] = None, *)	CORe50 Pytorch Dataset
`CUB200`(root, pathlib.Path]] = None, *[, ...])	Basic CUB200 PathsDataset to be used as a standard PyTorch Dataset.
`EndlessCLSimDataset`([root, scenario, ...])	Endless Continual Leanring Simulator Dataset
`INATURALIST2018`([root, split, transform, ...])	INATURALIST Pytorch Dataset
`MiniImageNetDataset`(imagenet_path, ...[, loader])	The MiniImageNet dataset.
`Omniglot`(root[, train, transform, ...])	Custom class used to adapt Omniglot (from Torchvision) and make it compatible with the Avalanche API.
`OpenLORIS`(root, pathlib.Path]] = None, *[, ...])	OpenLORIS Pytorch Dataset
`Stream51`(root, pathlib.Path]] = None, *[, ...])	Stream-51 Pytorch Dataset
`TinyImagenet`(root, pathlib.Path]] = None, *, ...)	Tiny Imagenet Pytorch Dataset

Benchmark Generators 

The generators sub-module provides a lot of functions that can be used to create a new benchmark.

This set of functions tries to cover most common use cases (Class/Task-Incremental, Domain-Incremental, …) but it also allows for the creation of entirely custom benchmarks (based on lists of tensors, on file lists, …).

Generators for Class/Task/Domain-incremental benchmarks 

`nc_benchmark`(train_dataset, test_dataset, ...)	This is the high-level benchmark instances generator for the "New Classes" (NC) case.
`ni_benchmark`(train_dataset, test_dataset, ...)	This is the high-level benchmark instances generator for the "New Instances" (NI) case.

Starting from tensor lists, file lists, PyTorch datasets 

`dataset_benchmark`(train_datasets, ...[, ...])	Creates a benchmark instance given a list of datasets.
`filelist_benchmark`(root, train_file_lists, ...)	Creates a benchmark instance given a list of filelists and the respective task labels.
`paths_benchmark`(train_lists_of_files, ...[, ...])	Creates a benchmark instance given a sequence of lists of files.
`tensors_benchmark`(train_tensors, test_tensors, *)	Creates a benchmark instance given lists of Tensors.

Misc (make data-incremental, add a validation stream, …)

Avalanche offers utilities to adapt a previously instantiated benchmark object.

More utilities to come!

`data_incremental_benchmark`(...[, shuffle, ...])	High-level benchmark generator for a Data Incremental setup.
`benchmark_with_validation_stream`(...[, ...])	Helper that can be used to obtain a benchmark with a validation stream.

Utils (Data Loading and AvalancheDataset)

The custom dataset and dataloader implementations contained in this sub-module are described in more detailed in the How-Tos here and here.

Data Loaders 

`TaskBalancedDataLoader`(data, ...[, ...])	Task-balanced data loader for Avalanche's datasets.
`GroupBalancedDataLoader`(datasets, ...[, ...])	Data loader that balances data from multiple datasets.
`ReplayDataLoader`(data, memory, ...[, ...])	Custom data loader for rehearsal/replay strategies.
`GroupBalancedInfiniteDataLoader`(datasets[, ...])	Data loader that balances data from multiple datasets emitting an infinite stream.

AvalancheDataset 

`AvalancheDataset`(dataset, *[, transform, ...])	The Dataset used as the base implementation for Avalanche.
`AvalancheSubset`(dataset[, indices, ...])	A Dataset that behaves like a PyTorch `torch.utils.data.Subset`.
`AvalancheTensorDataset`(*dataset_tensors[, ...])	A Dataset that wraps existing ndarrays, Tensors, lists.
`AvalancheConcatDataset`(datasets, *[, ...])	A Dataset that behaves like a PyTorch `torch.utils.data.ConcatDataset`.