avalanche.benchmarks

The benchmarks module provides a set of utilities that can be used for handling and generating your continual learning data stream. In the datasets module, basic PyTorch Datasets are provided. In the classic module instead, classic benchmarks (already proposed in the CL literature) generated from the datasets are provided. Finally, in generators basic utilities to generate new benchmarks on-the-fly are made available.

Subpackages

Package Contents

Classes

Experience

Definition of an experience. An experience contains a set of patterns

ScenarioStream

A scenario stream describes a sequence of incremental experiences.

StreamUserDef

Typed version of namedtuple.

StreamDef

Typed version of namedtuple.

GenericCLScenario

Base implementation of a Continual Learning benchmark instance.

GenericScenarioStream

A scenario stream describes a sequence of incremental experiences.

AbstractExperience

Definition of a learning experience. A learning experience contains a set of

GenericExperience

Definition of a learning experience based on a GenericCLScenario

NCScenario

This class defines a "New Classes" scenario. Once created, an instance

NCExperience

Defines a "New Classes" experience. It defines fields to obtain the current

NIScenario

This class defines a "New Instance" scenario.

NIExperience

Defines a "New Instances" experience. It defines fields to obtain the

Stream51

Stream-51 Pytorch Dataset

Functions

create_multi_dataset_generic_scenario(train_dataset_list: Sequence[SupportedDataset], test_dataset_list: Sequence[SupportedDataset], task_labels: Sequence[int], complete_test_set_only: bool = False, train_transform=None, train_target_transform=None, eval_transform=None, eval_target_transform=None, dataset_type: AvalancheDatasetType = None) → GenericCLScenario

This helper function is DEPRECATED in favor of

create_generic_scenario_from_filelists(root: Union[str, Path], train_file_lists: Sequence[Union[str, Path]], test_file_lists: Union[Union[str, Path], Sequence[Union[str, Path]]], task_labels: Sequence[int], complete_test_set_only: bool = False, train_transform=None, train_target_transform=None, eval_transform=None, eval_target_transform=None) → GenericCLScenario

This helper function is DEPRECATED in favor of

create_generic_scenario_from_paths(train_list_of_files: Sequence[Sequence[FileAndLabel]], test_list_of_files: Union[Sequence[FileAndLabel], Sequence[Sequence[FileAndLabel]]], task_labels: Sequence[int], complete_test_set_only: bool = False, train_transform=None, train_target_transform=None, eval_transform=None, eval_target_transform=None, dataset_type: AvalancheDatasetType = AvalancheDatasetType.UNDEFINED) → GenericCLScenario

This helper function is DEPRECATED in favor of

create_generic_scenario_from_tensor_lists(train_tensors: Sequence[Sequence[Any]], test_tensors: Sequence[Sequence[Any]], task_labels: Sequence[int], *, complete_test_set_only: bool = False, train_transform=None, train_target_transform=None, eval_transform=None, eval_target_transform=None, dataset_type: AvalancheDatasetType = None) → GenericCLScenario

This helper function is DEPRECATED in favor of

create_generic_scenario_from_tensors(train_data_x: Sequence[Any], train_data_y: Sequence[Sequence[SupportsInt]], test_data_x: Union[Any, Sequence[Any]], test_data_y: Union[Any, Sequence[Sequence[SupportsInt]]], task_labels: Sequence[int], complete_test_set_only: bool = False, train_transform=None, train_target_transform=None, eval_transform=None, eval_target_transform=None, dataset_type: AvalancheDatasetType = AvalancheDatasetType.UNDEFINED) → GenericCLScenario

This helper function is DEPRECATED in favor of

nc_scenario(train_dataset: Union[Sequence[SupportedDataset], SupportedDataset], test_dataset: Union[Sequence[SupportedDataset], SupportedDataset], n_experiences: int, task_labels: bool, *, shuffle: bool = True, seed: Optional[int] = None, fixed_class_order: Sequence[int] = None, per_exp_classes: Dict[int, int] = None, class_ids_from_zero_from_first_exp: bool = False, class_ids_from_zero_in_each_exp: bool = False, one_dataset_per_exp: bool = False, reproducibility_data: Dict[str, Any] = None) → NCScenario

This helper function is DEPRECATED in favor of nc_benchmark.

ni_scenario(train_dataset: Union[Sequence[SupportedDataset], SupportedDataset], test_dataset: Union[Sequence[SupportedDataset], SupportedDataset], n_experiences: int, *, task_labels: bool = False, shuffle: bool = True, seed: Optional[int] = None, balance_experiences: bool = False, min_class_patterns_in_exp: int = 0, fixed_exp_assignment: Optional[Sequence[Sequence[int]]] = None, reproducibility_data: Optional[Dict[str, Any]] = None) → NIScenario

This helper function is DEPRECATED in favor of ni_benchmark.

dataset_scenario(train_dataset_list: Sequence[SupportedDataset], test_dataset_list: Sequence[SupportedDataset], task_labels: Sequence[int], *, complete_test_set_only: bool = False, dataset_type: AvalancheDatasetType = AvalancheDatasetType.UNDEFINED) → GenericCLScenario

This helper function is DEPRECATED in favor of dataset_benchmark.

filelist_scenario(root: Union[str, Path], train_file_lists: Sequence[Union[str, Path]], test_file_lists: Union[Union[str, Path], Sequence[Union[str, Path]]], task_labels: Sequence[int], *, complete_test_set_only: bool = False, train_transform=None, train_target_transform=None, eval_transform=None, eval_target_transform=None) → GenericCLScenario

This helper function is DEPRECATED in favor of filelist_benchmark.

paths_scenario(train_list_of_files: Sequence[Sequence[FileAndLabel]], test_list_of_files: Union[Sequence[FileAndLabel], Sequence[Sequence[FileAndLabel]]], task_labels: Sequence[int], *, complete_test_set_only: bool = False, train_transform=None, train_target_transform=None, eval_transform=None, eval_target_transform=None, dataset_type: AvalancheDatasetType = AvalancheDatasetType.UNDEFINED) → GenericCLScenario

This helper function is DEPRECATED in favor of paths_benchmark.

tensors_scenario(train_tensors: Sequence[Sequence[Any]], test_tensors: Sequence[Sequence[Any]], task_labels: Sequence[int], *, complete_test_set_only: bool = False, train_transform=None, train_target_transform=None, eval_transform=None, eval_target_transform=None, dataset_type: AvalancheDatasetType = AvalancheDatasetType.UNDEFINED) → GenericCLScenario

This helper function is DEPRECATED in favor of tensors_benchmark.

tensor_scenario(train_data_x: Sequence[Any], train_data_y: Sequence[Sequence[SupportsInt]], test_data_x: Union[Any, Sequence[Any]], test_data_y: Union[Any, Sequence[Sequence[SupportsInt]]], task_labels: Sequence[int], *, complete_test_set_only: bool = False, train_transform=None, train_target_transform=None, eval_transform=None, eval_target_transform=None, dataset_type: AvalancheDatasetType = AvalancheDatasetType.UNDEFINED) → GenericCLScenario

This helper function is DEPRECATED in favor of tensors_benchmark.

nc_benchmark(train_dataset: Union[Sequence[SupportedDataset], SupportedDataset], test_dataset: Union[Sequence[SupportedDataset], SupportedDataset], n_experiences: int, task_labels: bool, *, shuffle: bool = True, seed: Optional[int] = None, fixed_class_order: Sequence[int] = None, per_exp_classes: Dict[int, int] = None, class_ids_from_zero_from_first_exp: bool = False, class_ids_from_zero_in_each_exp: bool = False, one_dataset_per_exp: bool = False, train_transform=None, eval_transform=None, reproducibility_data: Dict[str, Any] = None) → NCScenario

This is the high-level benchmark instances generator for the

ni_benchmark(train_dataset: Union[Sequence[SupportedDataset], SupportedDataset], test_dataset: Union[Sequence[SupportedDataset], SupportedDataset], n_experiences: int, *, task_labels: bool = False, shuffle: bool = True, seed: Optional[int] = None, balance_experiences: bool = False, min_class_patterns_in_exp: int = 0, fixed_exp_assignment: Optional[Sequence[Sequence[int]]] = None, train_transform=None, eval_transform=None, reproducibility_data: Optional[Dict[str, Any]] = None) → NIScenario

This is the high-level benchmark instances generator for the

data_incremental_benchmark(benchmark_instance: GenericCLScenario, experience_size: int, shuffle: bool = False, drop_last: bool = False, split_streams: Sequence[str] = ('train', ), custom_split_strategy: Callable[[Experience], Sequence[AvalancheDataset]] = None, experience_factory: Callable[[GenericScenarioStream, int], Experience] = None)

High-level benchmark generator for a Data Incremental setup.

benchmark_with_validation_stream(benchmark_instance: GenericCLScenario, validation_size: Union[int, float], shuffle: bool = False, input_stream: str = 'train', output_stream: str = 'valid', custom_split_strategy: Callable[[Experience], Tuple[AvalancheDataset, AvalancheDataset]] = None, *, experience_factory: Callable[[GenericScenarioStream, int], Experience] = None, lazy_splitting: bool = None)

Helper that can be used to obtain a benchmark with a validation stream.

CORe50(*, scenario: str = 'nicv2_391', run: int = 0, object_lvl: bool = True, mini: bool = False, train_transform: Optional[Any] = _default_train_transform, eval_transform: Optional[Any] = _default_eval_transform, dataset_root: Union[str, Path] = None)

Creates a CL benchmark for CORe50.

SplitCIFAR10(n_experiences: int, *, first_exp_with_half_classes: bool = False, return_task_id=False, seed: Optional[int] = None, fixed_class_order: Optional[Sequence[int]] = None, shuffle: bool = True, train_transform: Optional[Any] = _default_cifar10_train_transform, eval_transform: Optional[Any] = _default_cifar10_eval_transform, dataset_root: Union[str, Path] = None) → NCScenario

Creates a CL benchmark using the CIFAR10 dataset.

SplitCIFAR100(n_experiences: int, *, first_exp_with_half_classes: bool = False, return_task_id=False, seed: Optional[int] = None, fixed_class_order: Optional[Sequence[int]] = None, shuffle: bool = True, train_transform: Optional[Any] = _default_cifar100_train_transform, eval_transform: Optional[Any] = _default_cifar100_eval_transform, dataset_root: Union[str, Path] = None)

Creates a CL benchmark using the CIFAR100 dataset.

SplitCIFAR110(n_experiences: int, *, seed: Optional[int] = None, fixed_class_order: Optional[Sequence[int]] = None, train_transform: Optional[Any] = _default_cifar100_train_transform, eval_transform: Optional[Any] = _default_cifar100_eval_transform, dataset_root_cifar10: Union[str, Path] = None, dataset_root_cifar100: Union[str, Path] = None) → NCScenario

Creates a CL benchmark using both the CIFAR100 and CIFAR10 datasets.

SplitCUB200(n_experiences=11, *, classes_first_batch=100, return_task_id=False, seed=0, fixed_class_order=None, shuffle=False, train_transform: Optional[Any] = _default_train_transform, eval_transform: Optional[Any] = _default_eval_transform, dataset_root: Union[str, Path] = None)

Creates a CL benchmark using the Cub-200 dataset.

SplitFMNIST(n_experiences: int, *, first_batch_with_half_classes: bool = False, return_task_id=False, seed: Optional[int] = None, fixed_class_order: Optional[Sequence[int]] = None, shuffle: bool = True, train_transform: Optional[Any] = _default_fmnist_train_transform, eval_transform: Optional[Any] = _default_fmnist_eval_transform, dataset_root: Union[str, Path] = None)

Creates a CL benchmark using the Fashion MNIST dataset.

SplitImageNet(dataset_root: Union[str, Path], *, n_experiences=10, per_exp_classes=None, return_task_id=False, seed=0, fixed_class_order=None, shuffle: bool = True, train_transform: Optional[Any] = _default_train_transform, eval_transform: Optional[Any] = _default_eval_transform)

Creates a CL benchmark using the ImageNet dataset.

SplitMNIST(n_experiences: int, *, return_task_id=False, seed: Optional[int] = None, fixed_class_order: Optional[Sequence[int]] = None, shuffle: bool = True, train_transform: Optional[Any] = _default_mnist_train_transform, eval_transform: Optional[Any] = _default_mnist_eval_transform, dataset_root: Union[str, Path] = None)

Creates a CL benchmark using the MNIST dataset.

PermutedMNIST(n_experiences: int, *, seed: Optional[int] = None, train_transform: Optional[Any] = _default_mnist_train_transform, eval_transform: Optional[Any] = _default_mnist_eval_transform, dataset_root: Union[str, Path] = None) → NCScenario

Creates a Permuted MNIST benchmark.

RotatedMNIST(n_experiences: int, *, seed: Optional[int] = None, rotations_list: Optional[Sequence[int]] = None, train_transform: Optional[Any] = _default_mnist_train_transform, eval_transform: Optional[Any] = _default_mnist_eval_transform, dataset_root: Union[str, Path] = None) → NCScenario

Creates a Rotated MNIST benchmark.

SplitOmniglot(n_experiences: int, *, return_task_id=False, seed: Optional[int] = None, fixed_class_order: Optional[Sequence[int]] = None, shuffle: bool = True, train_transform: Optional[Any] = _default_omniglot_train_transform, eval_transform: Optional[Any] = _default_omniglot_eval_transform, dataset_root: Union[str, Path] = None)

Creates a CL benchmark using the OMNIGLOT dataset.

PermutedOmniglot(n_experiences: int, *, seed: Optional[int] = None, train_transform: Optional[Any] = _default_omniglot_train_transform, eval_transform: Optional[Any] = _default_omniglot_eval_transform, dataset_root: Union[str, Path] = None) → NCScenario

Creates a Permuted Omniglot benchmark.

RotatedOmniglot(n_experiences: int, *, seed: Optional[int] = None, rotations_list: Optional[Sequence[int]] = None, train_transform: Optional[Any] = _default_omniglot_train_transform, eval_transform: Optional[Any] = _default_omniglot_eval_transform, dataset_root: Union[str, Path] = None) → NCScenario

Creates a Rotated Omniglot benchmark.

SplitTinyImageNet(n_experiences=10, *, return_task_id=False, seed=0, fixed_class_order=None, shuffle: bool = True, train_transform: Optional[Any] = _default_train_transform, eval_transform: Optional[Any] = _default_eval_transform, dataset_root: Union[str, Path] = None)

Creates a CL benchmark using the Tiny ImageNet dataset.

OpenLORIS(*, factor: Literal['clutter', 'illumination', 'occlusion', 'pixel', 'mixture-iros'] = 'clutter', train_transform: Optional[Any] = None, eval_transform: Optional[Any] = None, dataset_root: Union[str, Path] = None)

Creates a CL benchmark for OpenLORIS.

class avalanche.benchmarks.Experience[source]

Bases: Protocol[TScenario, TScenarioStream]

Definition of an experience. An experience contains a set of patterns which has become available at a particular time instant. The content and size of an Experience is defined by the specific benchmark that creates the IExperience instance.

For instance, an experience of a New Classes scenario will contain all patterns belonging to a subset of classes of the original training set. An experience of a New Instance scenario will contain patterns from previously seen classes.

Experiences of Single Incremental Task (a.k.a. task-free) scenarios are usually called “batches” while in Multi Task scenarios an Experience is usually associated to a “task”. Finally, in a Multi Incremental Task scenario the Experience may be composed by patterns from different tasks.

origin_stream :TScenarioStream

A reference to the original stream from which this experience was obtained.

benchmark :TScenario

A reference to the benchmark.

current_experience :int

This is an incremental, 0-indexed, value used to keep track of the position of current experience in the original stream.

Beware that this value only describes the experience position in the original stream and may be unrelated to the order in which the strategy will encounter experiences.

dataset :AvalancheDataset

The dataset containing the patterns available in this experience.

property task_labels(self) List[int]

This list will contain the unique task labels of the patterns contained in this experience. In the most common scenarios this will be a list with a single value. Note: for scenarios that don’t produce task labels, a placeholder task label value like 0 is usually set to each pattern (see the description of the originating scenario for details).

property task_label(self) int

The task label. This value will never have value “None”. However, for scenarios that don’t produce task labels a placeholder value like 0 is usually set. Beware that this field is meant as a shortcut to obtain a unique task label: it assumes that only patterns labeled with a single task label are present. If this experience contains patterns from multiple tasks, accessing this property will result in an exception.

property scenario(self) TScenario

This property is DEPRECATED, use self.benchmark instead.

avalanche.benchmarks.TExperience
avalanche.benchmarks.TScenario
class avalanche.benchmarks.ScenarioStream[source]

Bases: Protocol[TScenario, TExperience]

A scenario stream describes a sequence of incremental experiences. Experiences are described as IExperience instances. They contain a set of patterns which has become available at a particular time instant along with any optional, scenario specific, metadata.

Most scenario expose two different streams: the training stream and the test stream.

name :str

The name of the stream.

benchmark :TScenario

A reference to the scenario this stream belongs to.

property scenario(self) TScenario

This property is DEPRECATED, use self.benchmark instead.

__getitem__(self: TScenarioStream, experience_idx: Union[int, slice, Iterable[int]]) Union[TExperience, TScenarioStream][source]

Gets an experience given its experience index (or a stream slice given the experience order).

Parameters

experience_idx – An int describing the experience index or an iterable/slice object describing a slice of this stream.

Returns

The Experience instance associated to the given experience index or a sliced stream instance.

__len__(self) int[source]

Used to get the length of this stream (the amount of experiences).

Returns

The amount of experiences in this stream.

avalanche.benchmarks.TScenarioStream
class avalanche.benchmarks.StreamUserDef[source]

Bases: typing.NamedTuple

Typed version of namedtuple.

Usage in Python versions >= 3.6:

class Employee(NamedTuple):
    name: str
    id: int

This is equivalent to:

Employee = collections.namedtuple('Employee', ['name', 'id'])

The resulting class has an extra __annotations__ attribute, giving a dict that maps field names to types. (The field names are also in the _fields attribute, which is part of the namedtuple API.) Alternative equivalent keyword syntax is also accepted:

Employee = NamedTuple('Employee', name=str, id=int)

In Python versions <= 3.5 use:

Employee = NamedTuple('Employee', [('name', str), ('id', int)])

Create and return a new object. See help(type) for accurate signature.

exps_data :TStreamDataOrigin
exps_task_labels :TStreamTaskLabels
origin_dataset :TOriginDataset
is_lazy :Optional[bool]
avalanche.benchmarks.TStreamUserDef
avalanche.benchmarks.TStreamsUserDict
class avalanche.benchmarks.StreamDef[source]

Bases: typing.NamedTuple

Typed version of namedtuple.

Usage in Python versions >= 3.6:

class Employee(NamedTuple):
    name: str
    id: int

This is equivalent to:

Employee = collections.namedtuple('Employee', ['name', 'id'])

The resulting class has an extra __annotations__ attribute, giving a dict that maps field names to types. (The field names are also in the _fields attribute, which is part of the namedtuple API.) Alternative equivalent keyword syntax is also accepted:

Employee = NamedTuple('Employee', name=str, id=int)

In Python versions <= 3.5 use:

Employee = NamedTuple('Employee', [('name', str), ('id', int)])

Create and return a new object. See help(type) for accurate signature.

exps_data :LazyDatasetSequence
exps_task_labels :Sequence[Set[int]]
origin_dataset :TOriginDataset
is_lazy :bool
avalanche.benchmarks.TStreamsDict
avalanche.benchmarks.TGenericCLScenario
class avalanche.benchmarks.GenericCLScenario(: TGenericCLScenario, *, stream_definitions: TStreamsUserDict, complete_test_set_only: bool = False, experience_factory: Callable[['GenericScenarioStream', int], TExperience] = None)[source]

Bases: Generic[TExperience]

Base implementation of a Continual Learning benchmark instance. A Continual Learning benchmark instance is defined by a set of streams of experiences (batches or tasks depending on the terminology). Each experience contains the training (or test, or validation, …) data that becomes available at a certain time instant.

Experiences are usually defined in children classes, with this class serving as the more general implementation. This class handles the most simple type of assignment: each stream is defined as a list of experiences, each experience is defined by a dataset.

Defining the “train” and “test” streams is mandatory. This class supports custom streams as well. Custom streams can be accessed by using the streamname_stream field of the created instance.

The name of custom streams can only contain letters, numbers or the “_” character and must not start with a number.

Creates an instance of a Continual Learning benchmark instance.

The benchmark instance is defined by a stream definition dictionary, which describes the content of each stream. The “train” and “test” stream are mandatory. Any other custom stream can be added.

There is no constraint on the amount of experiences in each stream (excluding the case in which complete_test_set_only is set).

Parameters
  • stream_definitions – The stream definitions dictionary. Must be a dictionary where the key is the stream name and the value is the definition of that stream. “train” and “test” streams are mandatory. This class supports custom streams as well. The name of custom streams can only contain letters, numbers and the “_” character and must not start with a number. Streams can be defined is two ways: static and lazy. In the static case, the stream must be a tuple containing 1, 2 or 3 elements: - The first element must be a list containing the datasets describing each experience. Datasets must be instances of AvalancheDataset. - The second element is optional and must be a list containing the task labels of each experience (as an int or a set of ints). If the stream definition tuple contains only one element (the list of datasets), then the task labels for each experience will be obtained by inspecting the content of the datasets. - The third element is optional and must be a reference to the originating dataset (if applicable). For instance, for SplitMNIST this may be a reference to the whole MNIST dataset. If the stream definition tuple contains less than 3 elements, then the reference to the original dataset will be set to None. In the lazy case, the stream must be defined as a tuple with 2 elements: - The first element must be a tuple containing the dataset generator (one for each experience) and the number of experiences in that stream. - The second element must be a list containing the task labels of each experience (as an int or a set of ints).

  • complete_test_set_only – If True, the test stream will contain a single experience containing the complete test set. This also means that the definition for the test stream must contain the definition for a single experience.

  • experience_factory – If not None, a callable that, given the benchmark instance and the experience ID, returns a experience instance. This parameter is usually used in subclasses (when invoking the super constructor) to specialize the experience class. Defaults to None, which means that the GenericExperience constructor will be used.

stream_definitions

A structure containing the definition of the streams.

original_train_dataset :Optional[Dataset]

The original training set. May be None.

original_test_dataset :Optional[Dataset]

The original test set. May be None.

train_stream :GenericScenarioStream[TExperience, TGenericCLScenario]

The stream used to obtain the training experiences. This stream can be sliced in order to obtain a subset of this stream.

test_stream :GenericScenarioStream[TExperience, TGenericCLScenario]

The stream used to obtain the test experiences. This stream can be sliced in order to obtain a subset of this stream.

Beware that, in certain scenarios, this stream may contain a single element. Check the complete_test_set_only field for more details.

complete_test_set_only :bool

If True, only the complete test set will be returned from experience instances.

This flag is usually set to True in scenarios where having one separate test set aligned to each training experience is impossible or doesn’t make sense from a semantic point of view.

property streams(self) Dict[str, 'GenericScenarioStream[TExperience, TGenericCLScenario]']
property n_experiences(self) int

The number of incremental training experiences contained in the train stream.

property task_labels(self) Sequence[List[int]]

The task label of each training experience.

get_reproducibility_data(self) Dict[str, Any][source]

Gets the data needed to reproduce this experiment.

This data can be stored using the pickle module or some other mechanism. It can then be loaded by passing it as the reproducibility_data parameter in the constructor.

Child classes should create their own reproducibility dictionary. This means that the implementation found in GenericCLScenario will return an empty dictionary, which is meaningless.

In order to obtain the same benchmark instance, the reproducibility data must be passed to the constructor along with the exact same input datasets.

Returns

A dictionary containing the data needed to reproduce the experiment.

property classes_in_experience(self) Mapping[str, Sequence[Optional[Set[int]]]]

A dictionary mapping each stream (by name) to a list.

Each element of the list is a set describing the classes included in that experience (identified by its index).

In previous releases this field contained the list of sets for the training stream (that is, there was no way to obtain the list for other streams). That behavior is deprecated and support for that usage way will be removed in the future.

get_classes_timeline(self, current_experience: int, stream: str = 'train')[source]

Returns the classes timeline given the ID of a experience.

Given a experience ID, this method returns the classes in that experience, previously seen classes, the cumulative class list and a list of classes that will be encountered in next experiences of the same stream.

Beware that by default this will obtain the timeline of an experience of the training stream. Use the stream parameter to select another stream.

Parameters
  • current_experience – The reference experience ID.

  • stream – The stream name.

Returns

A tuple composed of four lists: the first list contains the IDs of classes in this experience, the second contains IDs of classes seen in previous experiences, the third returns a cumulative list of classes (that is, the union of the first two list) while the last one returns a list of classes that will be encountered in next experiences. Beware that each of these elements can be None when the benchmark is initialized by using a lazy generator.

class avalanche.benchmarks.GenericScenarioStream(: TGenericScenarioStream, name: str, benchmark: TGenericCLScenario, *, slice_ids: List[int] = None)[source]

Bases: Generic[TExperience, TGenericCLScenario], ScenarioStream[TGenericCLScenario, TExperience], Sequence[TExperience]

A scenario stream describes a sequence of incremental experiences. Experiences are described as IExperience instances. They contain a set of patterns which has become available at a particular time instant along with any optional, scenario specific, metadata.

Most scenario expose two different streams: the training stream and the test stream.

Initialize self. See help(type(self)) for accurate signature.

slice_ids :Optional[List[int]]

Describes which experiences are contained in the current stream slice. Can be None, which means that this object is the original stream.

name :str

The name of the stream (for instance: “train”, “test”, “valid”, …).

benchmark

A reference to the benchmark.

__len__(self) int[source]

Gets the number of experiences this stream it’s made of.

Returns

The number of experiences in this stream.

__getitem__(self, exp_idx: Union[int, slice, Iterable[int]]) Union[TExperience, TScenarioStream][source]

Gets a experience given its experience index (or a stream slice given the experience order).

Parameters

exp_idx – An int describing the experience index or an iterable/slice object describing a slice of this stream.

Returns

The experience instance associated to the given experience index or a sliced stream instance.

drop_previous_experiences(self, to_exp: int) None[source]

Drop the reference to experiences up to a certain experience ID (inclusive).

This means that any reference to experiences with ID [0, from_exp] will be released. By dropping the reference to previous experiences, the memory associated with them can be freed, especially the one occupied by the dataset. However, if external references to the experience or the dataset still exist, dropping previous experiences at the stream level will have little to no impact on the memory usage.

To make sure that the underlying dataset can be freed, make sure that: - No reference to previous datasets or experiences are kept in you code; - The replay implementation doesn’t keep a reference to previous

datasets (in which case, is better to store a copy of the raw tensors instead);

  • The benchmark is being generated using a lazy initializer.

By dropping previous experiences, those experiences will no longer be available in the stream. Trying to access them will result in an exception.

Parameters

to_exp – The ID of the last exp to drop (inclusive). Can be a negative number, in which case this method doesn’t have any effect. Can be greater or equal to the stream length, in which case all currently loaded experiences will be dropped.

Returns

None

class avalanche.benchmarks.AbstractExperience(: TExperience, origin_stream: TScenarioStream, current_experience: int, classes_in_this_exp: Sequence[int], previous_classes: Sequence[int], classes_seen_so_far: Sequence[int], future_classes: Optional[Sequence[int]])[source]

Bases: Experience[TScenario, TScenarioStream], abc.ABC

Definition of a learning experience. A learning experience contains a set of patterns which has become available at a particular time instant. The content and size of an Experience is defined by the specific benchmark that creates the experience.

For instance, an experience of a New Classes scenario will contain all patterns belonging to a subset of classes of the original training set. An experience of a New Instance scenario will contain patterns from previously seen classes.

Creates an instance of the abstract experience given the benchmark stream, the current experience ID and data about the classes timeline.

Parameters
  • origin_stream – The stream from which this experience was obtained.

  • current_experience – The current experience ID, as an integer.

  • classes_in_this_exp – The list of classes in this experience.

  • previous_classes – The list of classes in previous experiences.

  • classes_seen_so_far – List of classes of current and previous experiences.

  • future_classes – The list of classes of next experiences.

classes_in_this_experience :Sequence[int]

The list of classes in this experience

previous_classes :Sequence[int]

The list of classes in previous experiences

classes_seen_so_far :Sequence[int]

List of classes of current and previous experiences

future_classes :Optional[Sequence[int]]

The list of classes of next experiences

property task_label(self) int

The task label. This value will never have value “None”. However, for scenarios that don’t produce task labels a placeholder value like 0 is usually set. Beware that this field is meant as a shortcut to obtain a unique task label: it assumes that only patterns labeled with a single task label are present. If this experience contains patterns from multiple tasks, accessing this property will result in an exception.

class avalanche.benchmarks.GenericExperience(: TGenericExperience, origin_stream: GenericScenarioStream[TGenericExperience, TGenericCLScenario], current_experience: int)[source]

Bases: AbstractExperience[TGenericCLScenario, GenericScenarioStream[TGenericExperience, TGenericCLScenario]]

Definition of a learning experience based on a GenericCLScenario instance.

This experience implementation uses the generic experience-patterns assignment defined in the GenericCLScenario instance. Instances of this class are usually obtained from a benchmark stream.

Creates an instance of a generic experience given the stream from this experience was taken and and the current experience ID.

Parameters
  • origin_stream – The stream from which this experience was obtained.

  • current_experience – The current experience ID, as an integer.

property task_labels(self) List[int]
avalanche.benchmarks.create_multi_dataset_generic_scenario(train_dataset_list: Sequence[SupportedDataset], test_dataset_list: Sequence[SupportedDataset], task_labels: Sequence[int], complete_test_set_only: bool = False, train_transform=None, train_target_transform=None, eval_transform=None, eval_target_transform=None, dataset_type: AvalancheDatasetType = None) GenericCLScenario[source]

This helper function is DEPRECATED in favor of create_multi_dataset_generic_benchmark.

Creates a generic scenario given a list of datasets and the respective task labels. Each training dataset will be considered as a separate training experience. Contents of the datasets will not be changed, including the targets.

When loading the datasets from a set of fixed filelist, consider using the create_generic_scenario_from_filelists() helper method instead.

In its base form, this function accepts a list of test datsets that must contain the same amount of datasets of the training list. Those pairs are then used to create the “past”, “cumulative” (a.k.a. growing) and “future” test sets. However, in certain Continual Learning scenarios only the concept of “complete” test set makes sense. In that case, the complete_test_set_only should be set to True (see the parameter description for more info).

Beware that pattern transformations must already be included in the datasets (when needed).

Parameters
  • train_dataset_list – A list of training datasets.

  • test_dataset_list – A list of test datasets.

  • task_labels – A list of task labels. Must contain the same amount of elements of the train_dataset_list parameter. For Single-Incremental-Task (a.k.a. Task-Free) scenarios, this is usually a list of zeros. For Multi Task scenario, this is usually a list of ascending task labels (starting from 0).

  • complete_test_set_only – If True, only the complete test set will be returned by the scenario. This means that the test_dataset_list parameter must be list with a single element (the complete test set). Defaults to False, which means that train_dataset_list and test_dataset_list must contain the same amount of datasets.

  • train_transform – The transformation to apply to the training data, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). Defaults to None.

  • train_target_transform – The transformation to apply to training patterns targets. Defaults to None.

  • eval_transform – The transformation to apply to the test data, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). Defaults to None.

  • eval_target_transform – The transformation to apply to test patterns targets. Defaults to None.

  • dataset_type – The type of the dataset. Defaults to None, which means that the type will be obtained from the input datasets. If input datasets are not instances of AvalancheDataset, the type UNDEFINED will be used.

Returns

A GenericCLScenario instance.

avalanche.benchmarks.create_generic_scenario_from_filelists(root: Union[str, Path], train_file_lists: Sequence[Union[str, Path]], test_file_lists: Union[Union[str, Path], Sequence[Union[str, Path]]], task_labels: Sequence[int], complete_test_set_only: bool = False, train_transform=None, train_target_transform=None, eval_transform=None, eval_target_transform=None) GenericCLScenario[source]

This helper function is DEPRECATED in favor of create_generic_benchmark_from_filelists.

Creates a generic scenario given a list of filelists and the respective task labels. A separate dataset will be created for each filelist and each of those training datasets will be considered a separate training experience.

In its base form, this function accepts a list of filelists for the test datsets that must contain the same amount of elements of the training list. Those pairs of datasets are then used to create the “past”, “cumulative” (a.k.a. growing) and “future” test sets. However, in certain Continual Learning scenarios only the concept of “complete” test set makes sense. In that case, the complete_test_set_only should be set to True (see the parameter description for more info).

This helper functions is the best shot when loading Caffe-style dataset based on filelists.

The resulting benchmark instance and the intermediate datasets used to populate it will be of type CLASSIFICATION.

Parameters
  • root – The root path of the dataset.

  • train_file_lists – A list of filelists describing the paths of the training patterns for each experience.

  • test_file_lists – A list of filelists describing the paths of the test patterns for each experience.

  • task_labels – A list of task labels. Must contain the same amount of elements of the train_file_lists parameter. For Single-Incremental-Task (a.k.a. Task-Free) scenarios, this is usually a list of zeros. For Multi Task scenario, this is usually a list of ascending task labels (starting from 0).

  • complete_test_set_only – If True, only the complete test set will be returned by the scenario. This means that the test_file_lists parameter must be list with a single element (the complete test set). Alternatively, can be a plain string or Path object. Defaults to False, which means that train_file_lists and test_file_lists must contain the same amount of filelists paths.

  • train_transform – The transformation to apply to the training data, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). Defaults to None.

  • train_target_transform – The transformation to apply to training patterns targets. Defaults to None.

  • eval_transform – The transformation to apply to the test data, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). Defaults to None.

  • eval_target_transform – The transformation to apply to test patterns targets. Defaults to None.

Returns

A GenericCLScenario instance.

avalanche.benchmarks.create_generic_scenario_from_paths(train_list_of_files: Sequence[Sequence[FileAndLabel]], test_list_of_files: Union[Sequence[FileAndLabel], Sequence[Sequence[FileAndLabel]]], task_labels: Sequence[int], complete_test_set_only: bool = False, train_transform=None, train_target_transform=None, eval_transform=None, eval_target_transform=None, dataset_type: AvalancheDatasetType = AvalancheDatasetType.UNDEFINED) GenericCLScenario[source]

This helper function is DEPRECATED in favor of create_generic_benchmark_from_paths.

Creates a generic scenario given a sequence of lists of files. A separate dataset will be created for each list. Each of those training datasets will be considered a separate training experience.

This is very similar to create_generic_scenario_from_filelists, with the main difference being that create_generic_scenario_from_filelists accepts, for each experience, a file list formatted in Caffe-style. On the contrary, this accepts a list of tuples where each tuple contains two elements: the full path to the pattern and its label. Optionally, the tuple may contain a third element describing the bounding box of the element to crop. This last bounding box may be useful when trying to extract the part of the image depicting the desired element.

In its base form, this function accepts a list for the test datasets that must contain the same amount of elements of the training list. Those pairs of datasets are then used to create the “past”, “cumulative” (a.k.a. growing) and “future” test sets. However, in certain Continual Learning scenarios only the concept of “complete” test set makes sense. In that case, the complete_test_set_only should be set to True (see the parameter description for more info).

The label of each pattern doesn’t have to be an int.

Parameters
  • train_list_of_files – A list of lists. Each list describes the paths and labels of patterns to include in that training experience, as tuples. Each tuple must contain two elements: the full path to the pattern and its class label. Optionally, the tuple may contain a third element describing the bounding box to use for cropping (top, left, height, width).

  • test_list_of_files – A list of lists. Each list describes the paths and labels of patterns to include in that test experience, as tuples. Each tuple must contain two elements: the full path to the pattern and its class label. Optionally, the tuple may contain a third element describing the bounding box to use for cropping (top, left, height, width).

  • task_labels – A list of task labels. Must contain the same amount of elements of the train_file_lists parameter. For Single-Incremental-Task (a.k.a. Task-Free) scenarios, this is usually a list of zeros. For Multi Task scenario, this is usually a list of ascending task labels (starting from 0).

  • complete_test_set_only – If True, only the complete test set will be returned by the scenario. This means that the test_list_of_files parameter must define a single experience (the complete test set). Defaults to False, which means that train_list_of_files and test_list_of_files must contain the same amount of paths.

  • train_transform – The transformation to apply to the training data, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). Defaults to None.

  • train_target_transform – The transformation to apply to training patterns targets. Defaults to None.

  • eval_transform – The transformation to apply to the test data, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). Defaults to None.

  • eval_target_transform – The transformation to apply to test patterns targets. Defaults to None.

  • dataset_type – The type of the dataset. Defaults to UNDEFINED.

Returns

A GenericCLScenario instance.

avalanche.benchmarks.create_generic_scenario_from_tensor_lists(train_tensors: Sequence[Sequence[Any]], test_tensors: Sequence[Sequence[Any]], task_labels: Sequence[int], *, complete_test_set_only: bool = False, train_transform=None, train_target_transform=None, eval_transform=None, eval_target_transform=None, dataset_type: AvalancheDatasetType = None) GenericCLScenario[source]

This helper function is DEPRECATED in favor of create_generic_benchmark_from_tensor_lists.

Creates a generic scenario given lists of Tensors. A separate dataset will be created from each Tensor tuple (x, y, z, …) and each of those training datasets will be considered a separate training experience. Using this helper function is the lowest-level way to create a Continual Learning scenario. When possible, consider using higher level helpers.

Experiences are defined by passing lists of tensors as the train_tensors and test_tensors parameter. Those parameters must be lists containing sub-lists of tensors, one for each experience. Each tensor defines the value of a feature (“x”, “y”, “z”, …) for all patterns of that experience.

By default the second tensor of each experience will be used to fill the targets value (label of each pattern).

In its base form, the test lists must contain the same amount of elements of the training lists. Those pairs of datasets are then used to create the “past”, “cumulative” (a.k.a. growing) and “future” test sets. However, in certain Continual Learning scenarios only the concept of “complete” test set makes sense. In that case, the complete_test_set_only should be set to True (see the parameter description for more info).

Parameters
  • train_tensors – A list of lists. The first list must contain the tensors for the first training experience (one tensor per feature), the second list must contain the tensors for the second training experience, and so on.

  • test_tensors – A list of lists. The first list must contain the tensors for the first test experience (one tensor per feature), the second list must contain the tensors for the second test experience, and so on. When using complete_test_set_only, this parameter must be a list containing a single sub-list for the single test experience.

  • task_labels – A list of task labels. Must contain a task label for each experience. For Single-Incremental-Task (a.k.a. Task-Free) scenarios, this is usually a list of zeros. For Multi Task scenario, this is usually a list of ascending task labels (starting from 0).

  • complete_test_set_only – If True, only the complete test set will be returned by the scenario. This means that test_tensors must define a single experience. Defaults to False, which means that train_tensors and test_tensors must define the same amount of experiences.

  • train_transform – The transformation to apply to the training data, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). Defaults to None.

  • train_target_transform – The transformation to apply to training patterns targets. Defaults to None.

  • eval_transform – The transformation to apply to the test data, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). Defaults to None.

  • eval_target_transform – The transformation to apply to test patterns targets. Defaults to None.

  • dataset_type – The type of the dataset. Defaults to None, which means that the type will be obtained from the input datasets. If input datasets are not instances of AvalancheDataset, the type UNDEFINED will be used.

Returns

A GenericCLScenario instance.

avalanche.benchmarks.create_generic_scenario_from_tensors(train_data_x: Sequence[Any], train_data_y: Sequence[Sequence[SupportsInt]], test_data_x: Union[Any, Sequence[Any]], test_data_y: Union[Any, Sequence[Sequence[SupportsInt]]], task_labels: Sequence[int], complete_test_set_only: bool = False, train_transform=None, train_target_transform=None, eval_transform=None, eval_target_transform=None, dataset_type: AvalancheDatasetType = AvalancheDatasetType.UNDEFINED) GenericCLScenario[source]

This helper function is DEPRECATED in favor of create_generic_benchmark_from_tensor_lists.

Please consider using create_generic_scenario_from_tensor_lists() instead. When switching to the new function, please keep in mind that the format of the parameters is completely different!

Creates a generic scenario given lists of Tensors and the respective task labels. A separate dataset will be created from each Tensor pair (x + y) and each of those training datasets will be considered a separate training experience. Contents of the datasets will not be changed, including the targets. Using this helper function is the lower level way to create a Continual Learning scenario. When possible, consider using higher level helpers.

By default the second tensor of each experience will be used to fill the targets value (label of each pattern).

In its base form, the test lists must contain the same amount of elements of the training lists. Those pairs of datasets are then used to create the “past”, “cumulative” (a.k.a. growing) and “future” test sets. However, in certain Continual Learning scenarios only the concept of “complete” test set makes sense. In that case, the complete_test_set_only should be set to True (see the parameter description for more info).

Parameters
  • train_data_x – A list of Tensors (one per experience) containing the patterns of the training sets.

  • train_data_y – A list of Tensors or int lists containing the labels of the patterns of the training sets. Must contain the same number of elements of train_datasets_x.

  • test_data_x – A Tensor or a list of Tensors (one per experience) containing the patterns of the test sets.

  • test_data_y – A Tensor or a list of Tensors or int lists containing the labels of the patterns of the test sets. Must contain the same number of elements of test_datasets_x.

  • task_labels – A list of task labels. Must contain the same amount of elements of the train_datasets_x parameter. For Single-Incremental-Task (a.k.a. Task-Free) scenarios, this is usually a list of zeros. For Multi Task scenario, this is usually a list of ascending task labels (starting from 0).

  • complete_test_set_only – If True, only the complete test set will be returned by the scenario. This means that test_data_x and test_data_y must define a single experience. Defaults to False, which means that train_data_* and test_data_* must define the same amount of experiences.

  • train_transform – The transformation to apply to the training data, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). Defaults to None.

  • train_target_transform – The transformation to apply to training patterns targets. Defaults to None.

  • eval_transform – The transformation to apply to the test data, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). Defaults to None.

  • eval_target_transform – The transformation to apply to test patterns targets. Defaults to None.

  • dataset_type – The type of the dataset. Defaults to UNDEFINED.

Returns

A GenericCLScenario instance.

class avalanche.benchmarks.NCScenario(train_dataset: AvalancheDataset, test_dataset: AvalancheDataset, n_experiences: int, task_labels: bool, shuffle: bool = True, seed: Optional[int] = None, fixed_class_order: Optional[Sequence[int]] = None, per_experience_classes: Optional[Dict[int, int]] = None, class_ids_from_zero_from_first_exp: bool = False, class_ids_from_zero_in_each_exp: bool = False, reproducibility_data: Optional[Dict[str, Any]] = None)[source]

Bases: GenericCLScenario['NCExperience']

This class defines a “New Classes” scenario. Once created, an instance of this class can be iterated in order to obtain the experience sequence under the form of instances of NCExperience.

This class can be used directly. However, we recommend using facilities like avalanche.benchmarks.generators.nc_benchmark().

Creates a NCGenericScenario instance given the training and test Datasets and the number of experiences.

By default, the number of classes will be automatically detected by looking at the training Dataset targets field. Classes will be uniformly distributed across n_experiences unless a per_experience_classes argument is specified.

The number of classes must be divisible without remainder by the number of experiences. This also applies when the per_experience_classes argument is not None.

Parameters
  • train_dataset – The training dataset. The dataset must be a subclass of AvalancheDataset. For instance, one can use the datasets from the torchvision package like that: train_dataset=AvalancheDataset(torchvision_dataset).

  • test_dataset – The test dataset. The dataset must be a subclass of AvalancheDataset. For instance, one can use the datasets from the torchvision package like that: test_dataset=AvalancheDataset(torchvision_dataset).

  • n_experiences – The number of experiences.

  • task_labels – If True, each experience will have an ascending task label. If False, the task label will be 0 for all the experiences.

  • shuffle – If True, the class order will be shuffled. Defaults to True.

  • seed – If shuffle is True and seed is not None, the class order will be shuffled according to the seed. When None, the current PyTorch random number generator state will be used. Defaults to None.

  • fixed_class_order – If not None, the class order to use (overrides the shuffle argument). Very useful for enhancing reproducibility. Defaults to None.

  • per_experience_classes – Is not None, a dictionary whose keys are (0-indexed) experience IDs and their values are the number of classes to include in the respective experiences. The dictionary doesn’t have to contain a key for each experience! All the remaining experiences will contain an equal amount of the remaining classes. The remaining number of classes must be divisible without remainder by the remaining number of experiences. For instance, if you want to include 50 classes in the first experience while equally distributing remaining classes across remaining experiences, just pass the “{0: 50}” dictionary as the per_experience_classes parameter. Defaults to None.

  • class_ids_from_zero_from_first_exp – If True, original class IDs will be remapped so that they will appear as having an ascending order. For instance, if the resulting class order after shuffling (or defined by fixed_class_order) is [23, 34, 11, 7, 6, …] and class_ids_from_zero_from_first_exp is True, then all the patterns belonging to class 23 will appear as belonging to class “0”, class “34” will be mapped to “1”, class “11” to “2” and so on. This is very useful when drawing confusion matrices and when dealing with algorithms with dynamic head expansion. Defaults to False. Mutually exclusive with the class_ids_from_zero_in_each_exp parameter.

  • class_ids_from_zero_in_each_exp – If True, original class IDs will be mapped to range [0, n_classes_in_exp) for each experience. Defaults to False. Mutually exclusive with the class_ids_from_zero_from_first_exp parameter.

  • reproducibility_data – If not None, overrides all the other scenario definition options. This is usually a dictionary containing data used to reproduce a specific experiment. One can use the get_reproducibility_data method to get (and even distribute) the experiment setup so that it can be loaded by passing it as this parameter. In this way one can be sure that the same specific experimental setup is being used (for reproducibility purposes). Beware that, in order to reproduce an experiment, the same train and test datasets must be used. Defaults to None.

classes_order :List[int] = []

Stores the class order (remapped class IDs).

classes_order_original_ids :List[int]

Stores the class order (original class IDs)

class_mapping :List[int] = []

class_mapping stores the class mapping so that mapped_class_id = class_mapping[original_class_id].

If the benchmark is created with an amount of classes which is less than the amount of all classes in the dataset, then class_mapping will contain some -1 values corresponding to ignored classes. This can happen when passing a fixed class order to the constructor.

n_classes_per_exp :List[int] = []

A list that, for each experience (identified by its index/ID), stores the number of classes assigned to that experience.

original_classes_in_exp :List[Set[int]] = []

A list that, for each experience (identified by its index/ID), stores a set of the original IDs of classes assigned to that experience. This field applies to both train and test streams.

class_ids_from_zero_from_first_exp :bool

If True the class IDs have been remapped to start from zero.

class_ids_from_zero_in_each_exp :bool

If True the class IDs have been remapped to start from zero in each experience

n_classes :int

The number of classes

train_exps_patterns_assignment

A list containing which training instances are assigned to each experience in the train stream. Instances are identified by their id w.r.t. the dataset found in the original_train_dataset field.

test_exps_patterns_assignment

A list containing which test instances are assigned to each experience in the test stream. Instances are identified by their id w.r.t. the dataset found in the original_test_dataset field.

get_reproducibility_data(self)[source]

Gets the data needed to reproduce this experiment.

This data can be stored using the pickle module or some other mechanism. It can then be loaded by passing it as the reproducibility_data parameter in the constructor.

Child classes should create their own reproducibility dictionary. This means that the implementation found in GenericCLScenario will return an empty dictionary, which is meaningless.

In order to obtain the same benchmark instance, the reproducibility data must be passed to the constructor along with the exact same input datasets.

Returns

A dictionary containing the data needed to reproduce the experiment.

classes_in_exp_range(self, exp_start: int, exp_end: Optional[int] = None) List[int][source]

Gets a list of classes contained in the given experiences. The experiences are defined by range. This means that only the classes in range [exp_start, exp_end) will be included.

Parameters
  • exp_start – The starting experience ID.

  • exp_end – The final experience ID. Can be None, which means that all the remaining experiences will be taken.

Returns

The classes contained in the required experience range.

class avalanche.benchmarks.NCExperience(origin_stream: GenericScenarioStream['NCExperience', NCScenario], current_experience: int)[source]

Bases: GenericExperience[NCScenario, GenericScenarioStream['NCExperience', NCScenario]]

Defines a “New Classes” experience. It defines fields to obtain the current dataset and the associated task label. It also keeps a reference to the stream from which this experience was taken.

Creates a NCExperience instance given the stream from this experience was taken and and the current experience ID.

Parameters
  • origin_stream – The stream from which this experience was obtained.

  • current_experience – The current experience ID, as an integer.

class avalanche.benchmarks.NIScenario(train_dataset: AvalancheDataset, test_dataset: AvalancheDataset, n_experiences: int, task_labels: bool = False, shuffle: bool = True, seed: Optional[int] = None, balance_experiences: bool = False, min_class_patterns_in_exp: int = 0, fixed_exp_assignment: Optional[Sequence[Sequence[int]]] = None, reproducibility_data: Optional[Dict[str, Any]] = None)[source]

Bases: GenericCLScenario['NIExperience']

This class defines a “New Instance” scenario. Once created, an instance of this class can be iterated in order to obtain the experience sequence under the form of instances of NIExperience.

Instances of this class can be created using the constructor directly. However, we recommend using facilities like avalanche.benchmarks.generators.ni_scenario().

Consider that every method from NIExperience used to retrieve parts of the test set (past, current, future, cumulative) always return the complete test set. That is, they behave as the getter for the complete test set.

Creates a NIScenario instance given the training and test Datasets and the number of experiences.

Parameters
  • train_dataset – The training dataset. The dataset must be an instance of AvalancheDataset. For instance, one can use the datasets from the torchvision package like that: train_dataset=AvalancheDataset(torchvision_dataset).

  • test_dataset – The test dataset. The dataset must be a subclass of AvalancheDataset. For instance, one can use the datasets from the torchvision package like that: test_dataset=AvalancheDataset(torchvision_dataset).

  • n_experiences – The number of experiences.

  • task_labels – If True, each experience will have an ascending task label. If False, the task label will be 0 for all the experiences. Defaults to False.

  • shuffle – If True, the patterns order will be shuffled. Defaults to True.

  • seed – If shuffle is True and seed is not None, the class order will be shuffled according to the seed. When None, the current PyTorch random number generator state will be used. Defaults to None.

  • balance_experiences – If True, pattern of each class will be equally spread across all experiences. If False, patterns will be assigned to experiences in a complete random way. Defaults to False.

  • min_class_patterns_in_exp – The minimum amount of patterns of every class that must be assigned to every experience. Compatible with the balance_experiences parameter. An exception will be raised if this constraint can’t be satisfied. Defaults to 0.

  • fixed_exp_assignment – If not None, the pattern assignment to use. It must be a list with an entry for each experience. Each entry is a list that contains the indexes of patterns belonging to that experience. Overrides the shuffle, balance_experiences and min_class_patterns_in_exp parameters.

  • reproducibility_data – If not None, overrides all the other scenario definition options, including fixed_exp_assignment. This is usually a dictionary containing data used to reproduce a specific experiment. One can use the get_reproducibility_data method to get (and even distribute) the experiment setup so that it can be loaded by passing it as this parameter. In this way one can be sure that the same specific experimental setup is being used (for reproducibility purposes). Beware that, in order to reproduce an experiment, the same train and test datasets must be used. Defaults to None.

n_classes :int

The amount of classes in the original training set.

n_patterns_per_class :List[int]

The amount of patterns for each class in the original training set.

n_patterns_per_experience :List[int] = []

The number of patterns in each experience.

exp_structure :List[List[int]] = []

This field contains, for each training experience, the number of instances of each class assigned to that experience.

train_exps_patterns_assignment

A list containing which training instances are assigned to each experience in the train stream. Instances are identified by their id w.r.t. the dataset found in the original_train_dataset field.

get_reproducibility_data(self) Dict[str, Any][source]

Gets the data needed to reproduce this experiment.

This data can be stored using the pickle module or some other mechanism. It can then be loaded by passing it as the reproducibility_data parameter in the constructor.

Child classes should create their own reproducibility dictionary. This means that the implementation found in GenericCLScenario will return an empty dictionary, which is meaningless.

In order to obtain the same benchmark instance, the reproducibility data must be passed to the constructor along with the exact same input datasets.

Returns

A dictionary containing the data needed to reproduce the experiment.

class avalanche.benchmarks.NIExperience(origin_stream: GenericScenarioStream['NIExperience', NIScenario], current_experience: int)[source]

Bases: GenericExperience[NIScenario, GenericScenarioStream['NIExperience', NIScenario]]

Defines a “New Instances” experience. It defines fields to obtain the current dataset and the associated task label. It also keeps a reference to the stream from which this experience was taken.

Creates a NIExperience instance given the stream from this experience was taken and and the current experience ID.

Parameters
  • origin_stream – The stream from which this experience was obtained.

  • current_experience – The current experience ID, as an integer.

avalanche.benchmarks.nc_scenario(train_dataset: Union[Sequence[SupportedDataset], SupportedDataset], test_dataset: Union[Sequence[SupportedDataset], SupportedDataset], n_experiences: int, task_labels: bool, *, shuffle: bool = True, seed: Optional[int] = None, fixed_class_order: Sequence[int] = None, per_exp_classes: Dict[int, int] = None, class_ids_from_zero_from_first_exp: bool = False, class_ids_from_zero_in_each_exp: bool = False, one_dataset_per_exp: bool = False, reproducibility_data: Dict[str, Any] = None) NCScenario[source]

This helper function is DEPRECATED in favor of nc_benchmark.

This method is the high-level specific scenario generator for the “New Classes” (NC) case. Given a sequence of train and test datasets creates the continual stream of data as a series of experiences. Each experience will contain all the patterns belonging to a certain set of classes and a class won’t be assigned to more than one experience.

The task_labels parameter determines if each incremental experience has an increasing task label or if, at the contrary, a default task label “0” has to be assigned to all experiences. This can be useful when differentiating between Single-Incremental-Task and Multi-Task scenarios.

There are other important parameters that can be specified in order to tweak the behaviour of the resulting scenario. Please take a few minutes to read and understand them as they may save you a lot of work.

This generator features a integrated reproducibility mechanism that allows the user to store and later re-load a scenario. For more info see the reproducibility_data parameter.

Parameters
  • train_dataset – A list of training datasets, or a single dataset.

  • test_dataset – A list of test datasets, or a single test dataset.

  • n_experiences – The number of incremental experience. This is not used when using multiple train/test datasets with the one_dataset_per_exp parameter set to True.

  • task_labels – If True, each experience will have an ascending task label. If False, the task label will be 0 for all the experiences.

  • shuffle – If True, the class (or experience) order will be shuffled. Defaults to True.

  • seed – If shuffle is True and seed is not None, the class (or experience) order will be shuffled according to the seed. When None, the current PyTorch random number generator state will be used. Defaults to None.

  • fixed_class_order – If not None, the class order to use (overrides the shuffle argument). Very useful for enhancing reproducibility. Defaults to None.

  • per_exp_classes – Is not None, a dictionary whose keys are (0-indexed) experience IDs and their values are the number of classes to include in the respective experiences. The dictionary doesn’t have to contain a key for each experience! All the remaining experiences will contain an equal amount of the remaining classes. The remaining number of classes must be divisible without remainder by the remaining number of experiences. For instance, if you want to include 50 classes in the first experience while equally distributing remaining classes across remaining experiences, just pass the “{0: 50}” dictionary as the per_experience_classes parameter. Defaults to None.

  • class_ids_from_zero_from_first_exp – If True, original class IDs will be remapped so that they will appear as having an ascending order. For instance, if the resulting class order after shuffling (or defined by fixed_class_order) is [23, 34, 11, 7, 6, …] and class_ids_from_zero_from_first_exp is True, then all the patterns belonging to class 23 will appear as belonging to class “0”, class “34” will be mapped to “1”, class “11” to “2” and so on. This is very useful when drawing confusion matrices and when dealing with algorithms with dynamic head expansion. Defaults to False. Mutually exclusive with the class_ids_from_zero_in_each_exp parameter.

  • class_ids_from_zero_in_each_exp – If True, original class IDs will be mapped to range [0, n_classes_in_exp) for each experience. Defaults to False. Mutually exclusive with the class_ids_from_zero_from_first_exp parameter.

  • one_dataset_per_exp – available only when multiple train-test datasets are provided. If True, each dataset will be treated as a experience. Mutually exclusive with the per_experience_classes and fixed_class_order parameters. Overrides the n_experiences parameter. Defaults to False.

  • reproducibility_data – If not None, overrides all the other scenario definition options. This is usually a dictionary containing data used to reproduce a specific experiment. One can use the get_reproducibility_data method to get (and even distribute) the experiment setup so that it can be loaded by passing it as this parameter. In this way one can be sure that the same specific experimental setup is being used (for reproducibility purposes). Beware that, in order to reproduce an experiment, the same train and test datasets must be used. Defaults to None.

Returns

A properly initialized NCScenario instance.

avalanche.benchmarks.ni_scenario(train_dataset: Union[Sequence[SupportedDataset], SupportedDataset], test_dataset: Union[Sequence[SupportedDataset], SupportedDataset], n_experiences: int, *, task_labels: bool = False, shuffle: bool = True, seed: Optional[int] = None, balance_experiences: bool = False, min_class_patterns_in_exp: int = 0, fixed_exp_assignment: Optional[Sequence[Sequence[int]]] = None, reproducibility_data: Optional[Dict[str, Any]] = None) NIScenario[source]

This helper function is DEPRECATED in favor of ni_benchmark.

This method is the high-level specific scenario generator for the “New Instances” (NI) case. Given a sequence of train and test datasets creates the continual stream of data as a series of experiences. Each experience will contain patterns belonging to the same classes.

The task_labels parameter determines if each incremental experience has an increasing task label or if, at the contrary, a default task label “0” has to be assigned to all experiences. This can be useful when differentiating between Single-Incremental-Task and Multi-Task scenarios.

There are other important parameters that can be specified in order to tweak the behaviour of the resulting scenario. Please take a few minutes to read and understand them as they may save you a lot of work.

This generator features an integrated reproducibility mechanism that allows the user to store and later re-load a scenario. For more info see the reproducibility_data parameter.

Parameters
  • train_dataset – A list of training datasets, or a single dataset.

  • test_dataset – A list of test datasets, or a single test dataset.

  • n_experiences – The number of experiences.

  • task_labels – If True, each experience will have an ascending task label. If False, the task label will be 0 for all the experiences.

  • shuffle – If True, patterns order will be shuffled.

  • seed – A valid int used to initialize the random number generator. Can be None.

  • balance_experiences – If True, pattern of each class will be equally spread across all experiences. If False, patterns will be assigned to experiences in a complete random way. Defaults to False.

  • min_class_patterns_in_exp – The minimum amount of patterns of every class that must be assigned to every experience. Compatible with the balance_experiences parameter. An exception will be raised if this constraint can’t be satisfied. Defaults to 0.

  • fixed_exp_assignment – If not None, the pattern assignment to use. It must be a list with an entry for each experience. Each entry is a list that contains the indexes of patterns belonging to that experience. Overrides the shuffle, balance_experiences and min_class_patterns_in_exp parameters.

  • reproducibility_data – If not None, overrides all the other scenario definition options, including fixed_exp_assignment. This is usually a dictionary containing data used to reproduce a specific experiment. One can use the get_reproducibility_data method to get (and even distribute) the experiment setup so that it can be loaded by passing it as this parameter. In this way one can be sure that the same specific experimental setup is being used (for reproducibility purposes). Beware that, in order to reproduce an experiment, the same train and test datasets must be used. Defaults to None.

Returns

A properly initialized NIScenario instance.

avalanche.benchmarks.dataset_scenario(train_dataset_list: Sequence[SupportedDataset], test_dataset_list: Sequence[SupportedDataset], task_labels: Sequence[int], *, complete_test_set_only: bool = False, dataset_type: AvalancheDatasetType = AvalancheDatasetType.UNDEFINED) GenericCLScenario[source]

This helper function is DEPRECATED in favor of dataset_benchmark.

Creates a generic scenario given a list of datasets and the respective task labels. Each training dataset will be considered as a separate training experience. Contents of the datasets will not be changed, including the targets.

When loading the datasets from a set of fixed file lists, consider using the filelist_scenario() helper method instead. Also, loading from a list of paths is supported through the paths_scenario() helper.

In its base form, this function accepts a list of test datasets that must contain the same amount of datasets of the training list. Those pairs are then used to create the “past”, “cumulative” (a.k.a. growing) and “future” test sets. However, in certain Continual Learning scenarios only the concept of “complete” test set makes sense. In that case, the complete_test_set_only parameter should be set to True (see the parameter description for more info).

Beware that pattern transformations must already be included in the datasets (when needed).

Parameters
  • train_dataset_list – A list of training datasets.

  • test_dataset_list – A list of test datasets.

  • task_labels – A list of task labels. Must contain the same amount of elements of the train_dataset_list parameter. For Single-Incremental-Task (a.k.a. Task-Free) scenarios, this is usually a list of zeros. For Multi Task scenario, this is usually a list of ascending task labels (starting from 0).

  • complete_test_set_only – If True, only the complete test set will be returned by the scenario. This means that the test_dataset_list parameter must be list with a single element (the complete test set). Defaults to False, which means that train_dataset_list and test_dataset_list must contain the same amount of datasets.

  • dataset_type – The type of the dataset. Defaults to None, which means that the type will be obtained from the input datasets. If input datasets are not instances of AvalancheDataset, the type UNDEFINED will be used.

Returns

A properly initialized GenericCLScenario instance.

avalanche.benchmarks.filelist_scenario(root: Union[str, Path], train_file_lists: Sequence[Union[str, Path]], test_file_lists: Union[Union[str, Path], Sequence[Union[str, Path]]], task_labels: Sequence[int], *, complete_test_set_only: bool = False, train_transform=None, train_target_transform=None, eval_transform=None, eval_target_transform=None) GenericCLScenario[source]

This helper function is DEPRECATED in favor of filelist_benchmark.

Creates a generic scenario given a list of filelists and the respective task labels. A separate dataset will be created for each filelist and each of those training datasets will be considered a separate training experience.

In its base form, this function accepts a list of filelists for the test datsets that must contain the same amount of elements of the training list. Those pairs of datasets are then used to create the “past”, “cumulative” (a.k.a. growing) and “future” test sets. However, in certain Continual Learning scenarios only the concept of “complete” test set makes sense. In that case, the complete_test_set_only should be set to True (see the parameter description for more info).

This helper functions is the best shot when loading Caffe-style dataset based on filelists.

The resulting benchmark instance and the intermediate datasets used to populate it will be of type CLASSIFICATION.

Parameters
  • root – The root path of the dataset.

  • train_file_lists – A list of filelists describing the paths of the training patterns for each experience.

  • test_file_lists – A list of filelists describing the paths of the test patterns for each experience.

  • task_labels – A list of task labels. Must contain the same amount of elements of the train_file_lists parameter. For Single-Incremental-Task (a.k.a. Task-Free) scenarios, this is usually a list of zeros. For Multi Task scenario, this is usually a list of ascending task labels (starting from 0).

  • complete_test_set_only – If True, only the complete test set will be returned by the scenario. This means that the test_file_lists parameter must be list with a single element (the complete test set). Alternatively, can be a plain string or Path object. Defaults to False, which means that train_file_lists and test_file_lists must contain the same amount of filelists paths.

  • train_transform – The transformation to apply to the training data, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). Defaults to None.

  • train_target_transform – The transformation to apply to training patterns targets. Defaults to None.

  • eval_transform – The transformation to apply to the test data, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). Defaults to None.

  • eval_target_transform – The transformation to apply to test patterns targets. Defaults to None.

Returns

A properly initialized GenericCLScenario instance.

avalanche.benchmarks.paths_scenario(train_list_of_files: Sequence[Sequence[FileAndLabel]], test_list_of_files: Union[Sequence[FileAndLabel], Sequence[Sequence[FileAndLabel]]], task_labels: Sequence[int], *, complete_test_set_only: bool = False, train_transform=None, train_target_transform=None, eval_transform=None, eval_target_transform=None, dataset_type: AvalancheDatasetType = AvalancheDatasetType.UNDEFINED) GenericCLScenario[source]

This helper function is DEPRECATED in favor of paths_benchmark.

Creates a generic scenario given a list of files and class labels. A separate dataset will be created for each list and each of those training datasets will be considered a separate training experience.

This is very similar to filelist_scenario, with the main difference being that filelist_scenario accepts, for each experience, a file list formatted in Caffe-style. On the contrary, this accepts a list of tuples where each tuple contains two elements: the full path to the pattern and its label. Optionally, the tuple may contain a third element describing the bounding box of the element to crop. This last bounding box may be useful when trying to extract the part of the image depicting the desired element.

In its base form, this function accepts a list of lists of tuples for the test datsets that must contain the same amount of lists of the training list. Those pairs of datasets are then used to create the “past”, “cumulative” (a.k.a. growing) and “future” test sets. However, in certain Continual Learning scenarios only the concept of “complete” test set makes sense. In that case, the complete_test_set_only should be set to True (see the parameter description for more info).

The label of each pattern doesn’t have to be an int.

Parameters
  • train_list_of_files – A list of lists. Each list describes the paths and labels of patterns to include in that training experience as tuples. Each tuple must contain two elements: the full path to the pattern and its class label. Optionally, the tuple may contain a third element describing the bounding box to use for cropping (top, left, height, width).

  • test_list_of_files – A list of lists. Each list describes the paths and labels of patterns to include in that test experience as tuples. Each tuple must contain two elements: the full path to the pattern and its class label. Optionally, the tuple may contain a third element describing the bounding box to use for cropping (top, left, height, width).

  • task_labels – A list of task labels. Must contain the same amount of elements of the train_file_lists parameter. For Single-Incremental-Task (a.k.a. Task-Free) scenarios, this is usually a list of zeros. For Multi Task scenario, this is usually a list of ascending task labels (starting from 0).

  • complete_test_set_only – If True, only the complete test set will be returned by the scenario. This means that the test_file_lists parameter must be list with a single element (the complete test set). Alternatively, can be a plain string or Path object. Defaults to False, which means that train_file_lists and test_file_lists must contain the same amount of filelists paths.

  • train_transform – The transformation to apply to the training data, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). Defaults to None.

  • train_target_transform – The transformation to apply to training patterns targets. Defaults to None.

  • eval_transform – The transformation to apply to the test data, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). Defaults to None.

  • eval_target_transform – The transformation to apply to test patterns targets. Defaults to None.

  • dataset_type – The type of the dataset. Defaults to UNDEFINED.

Returns

A properly initialized GenericCLScenario instance.

avalanche.benchmarks.tensors_scenario(train_tensors: Sequence[Sequence[Any]], test_tensors: Sequence[Sequence[Any]], task_labels: Sequence[int], *, complete_test_set_only: bool = False, train_transform=None, train_target_transform=None, eval_transform=None, eval_target_transform=None, dataset_type: AvalancheDatasetType = AvalancheDatasetType.UNDEFINED) GenericCLScenario[source]

This helper function is DEPRECATED in favor of tensors_benchmark.

Creates a generic scenario given lists of Tensors and the respective task labels. A separate dataset will be created from each Tensor tuple (x, y, …) and each of those training datasets will be considered a separate training experience. Using this helper function is the lowest-level way to create a Continual Learning scenario. When possible, consider using higher level helpers.

Experiences are defined by passing lists of tensors as the train_tensors and test_tensors parameter. Those parameters must be lists containing sub-lists of tensors, one for each experience. Each tensor defines the value of a feature (“x”, “y”, “z”, …) for all patterns of that experience.

By default the second tensor of each experience will be used to fill the targets value (label of each pattern).

In its base form, the test lists must contain the same amount of elements of the training lists. Those pairs of datasets are then used to create the “past”, “cumulative” (a.k.a. growing) and “future” test sets. However, in certain Continual Learning scenarios only the concept of “complete” test set makes sense. In that case, the complete_test_set_only should be set to True (see the parameter description for more info).

Parameters
  • train_tensors – A list of lists. The first list must contain the tensors for the first training experience (one tensor per feature), the second list must contain the tensors for the second training experience, and so on.

  • test_tensors – A list of lists. The first list must contain the tensors for the first test experience (one tensor per feature), the second list must contain the tensors for the second test experience, and so on.

  • task_labels – A list of task labels. Must contain a task label for each experience. For Single-Incremental-Task (a.k.a. Task-Free) scenarios, this is usually a list of zeros. For Multi Task scenario, this is usually a list of ascending task labels (starting from 0).

  • complete_test_set_only – If True, only the complete test set will be returned by the scenario. This means that test_tensors must define a single experience. Defaults to False, which means that train_tensors and test_tensors must define the same amount of experiences.

  • train_transform – The transformation to apply to the training data, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). Defaults to None.

  • train_target_transform – The transformation to apply to training patterns targets. Defaults to None.

  • eval_transform – The transformation to apply to the test data, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). Defaults to None.

  • eval_target_transform – The transformation to apply to test patterns targets. Defaults to None.

  • dataset_type – The type of the dataset. Defaults to UNDEFINED.

Returns

A properly initialized GenericCLScenario instance.

avalanche.benchmarks.tensor_scenario(train_data_x: Sequence[Any], train_data_y: Sequence[Sequence[SupportsInt]], test_data_x: Union[Any, Sequence[Any]], test_data_y: Union[Any, Sequence[Sequence[SupportsInt]]], task_labels: Sequence[int], *, complete_test_set_only: bool = False, train_transform=None, train_target_transform=None, eval_transform=None, eval_target_transform=None, dataset_type: AvalancheDatasetType = AvalancheDatasetType.UNDEFINED) GenericCLScenario[source]

This helper function is DEPRECATED in favor of tensors_benchmark.

Please consider using tensors_benchmark() instead. When switching to the new function, please keep in mind that the format of the parameters is completely different!

Creates a generic scenario given lists of Tensors and the respective task labels. A separate dataset will be created from each Tensor pair (x + y) and each of those training datasets will be considered a separate training experience. Contents of the datasets will not be changed, including the targets. Using this helper function is the lower level way to create a Continual Learning scenario. When possible, consider using higher level helpers.

By default the second tensor of each experience will be used to fill the targets value (label of each pattern).

In its base form, the test lists must contain the same amount of elements of the training lists. Those pairs of datasets are then used to create the “past”, “cumulative” (a.k.a. growing) and “future” test sets. However, in certain Continual Learning scenarios only the concept of “complete” test set makes sense. In that case, the complete_test_set_only should be set to True (see the parameter description for more info).

Parameters
  • train_data_x – A list of Tensors (one per experience) containing the patterns of the training sets.

  • train_data_y – A list of Tensors or int lists containing the labels of the patterns of the training sets. Must contain the same number of elements of train_datasets_x.

  • test_data_x – A Tensor or a list of Tensors (one per experience) containing the patterns of the test sets.

  • test_data_y – A Tensor or a list of Tensors or int lists containing the labels of the patterns of the test sets. Must contain the same number of elements of test_datasets_x.

  • task_labels – A list of task labels. Must contain the same amount of elements of the train_datasets_x parameter. For Single-Incremental-Task (a.k.a. Task-Free) scenarios, this is usually a list of zeros. For Multi Task scenario, this is usually a list of ascending task labels (starting from 0).

  • complete_test_set_only – If True, only the complete test set will be returned by the scenario. This means that the test_datasets_x and test_datasets_y parameters must be lists with a single element (the complete test set). Defaults to False, which means that train_file_lists and test_file_lists must contain the same amount of filelists paths.

  • train_transform – The transformation to apply to the training data, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). Defaults to None.

  • train_target_transform – The transformation to apply to training patterns targets. Defaults to None.

  • eval_transform – The transformation to apply to the test data, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). Defaults to None.

  • eval_target_transform – The transformation to apply to test patterns targets. Defaults to None.

  • dataset_type – The type of the dataset. Defaults to UNDEFINED.

Returns

A properly initialized GenericCLScenario instance.

avalanche.benchmarks.nc_benchmark(train_dataset: Union[Sequence[SupportedDataset], SupportedDataset], test_dataset: Union[Sequence[SupportedDataset], SupportedDataset], n_experiences: int, task_labels: bool, *, shuffle: bool = True, seed: Optional[int] = None, fixed_class_order: Sequence[int] = None, per_exp_classes: Dict[int, int] = None, class_ids_from_zero_from_first_exp: bool = False, class_ids_from_zero_in_each_exp: bool = False, one_dataset_per_exp: bool = False, train_transform=None, eval_transform=None, reproducibility_data: Dict[str, Any] = None) NCScenario[source]

This is the high-level benchmark instances generator for the “New Classes” (NC) case. Given a sequence of train and test datasets creates the continual stream of data as a series of experiences. Each experience will contain all the instances belonging to a certain set of classes and a class won’t be assigned to more than one experience.

This is the reference helper function for creating instances of Class- or Task-Incremental benchmarks.

The task_labels parameter determines if each incremental experience has an increasing task label or if, at the contrary, a default task label “0” has to be assigned to all experiences. This can be useful when differentiating between Single-Incremental-Task and Multi-Task scenarios.

There are other important parameters that can be specified in order to tweak the behaviour of the resulting benchmark. Please take a few minutes to read and understand them as they may save you a lot of work.

This generator features a integrated reproducibility mechanism that allows the user to store and later re-load a benchmark. For more info see the reproducibility_data parameter.

Parameters
  • train_dataset – A list of training datasets, or a single dataset.

  • test_dataset – A list of test datasets, or a single test dataset.

  • n_experiences – The number of incremental experience. This is not used when using multiple train/test datasets with the one_dataset_per_exp parameter set to True.

  • task_labels – If True, each experience will have an ascending task label. If False, the task label will be 0 for all the experiences.

  • shuffle – If True, the class (or experience) order will be shuffled. Defaults to True.

  • seed – If shuffle is True and seed is not None, the class (or experience) order will be shuffled according to the seed. When None, the current PyTorch random number generator state will be used. Defaults to None.

  • fixed_class_order – If not None, the class order to use (overrides the shuffle argument). Very useful for enhancing reproducibility. Defaults to None.

  • per_exp_classes – Is not None, a dictionary whose keys are (0-indexed) experience IDs and their values are the number of classes to include in the respective experiences. The dictionary doesn’t have to contain a key for each experience! All the remaining experiences will contain an equal amount of the remaining classes. The remaining number of classes must be divisible without remainder by the remaining number of experiences. For instance, if you want to include 50 classes in the first experience while equally distributing remaining classes across remaining experiences, just pass the “{0: 50}” dictionary as the per_experience_classes parameter. Defaults to None.

  • class_ids_from_zero_from_first_exp – If True, original class IDs will be remapped so that they will appear as having an ascending order. For instance, if the resulting class order after shuffling (or defined by fixed_class_order) is [23, 34, 11, 7, 6, …] and class_ids_from_zero_from_first_exp is True, then all the patterns belonging to class 23 will appear as belonging to class “0”, class “34” will be mapped to “1”, class “11” to “2” and so on. This is very useful when drawing confusion matrices and when dealing with algorithms with dynamic head expansion. Defaults to False. Mutually exclusive with the class_ids_from_zero_in_each_exp parameter.

  • class_ids_from_zero_in_each_exp – If True, original class IDs will be mapped to range [0, n_classes_in_exp) for each experience. Defaults to False. Mutually exclusive with the class_ids_from_zero_from_first_exp parameter.

  • one_dataset_per_exp – available only when multiple train-test datasets are provided. If True, each dataset will be treated as a experience. Mutually exclusive with the per_experience_classes and fixed_class_order parameters. Overrides the n_experiences parameter. Defaults to False.

  • train_transform – The transformation to apply to the training data, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). Defaults to None.

  • eval_transform – The transformation to apply to the test data, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). Defaults to None.

  • reproducibility_data – If not None, overrides all the other benchmark definition options. This is usually a dictionary containing data used to reproduce a specific experiment. One can use the get_reproducibility_data method to get (and even distribute) the experiment setup so that it can be loaded by passing it as this parameter. In this way one can be sure that the same specific experimental setup is being used (for reproducibility purposes). Beware that, in order to reproduce an experiment, the same train and test datasets must be used. Defaults to None.

Returns

A properly initialized NCScenario instance.

avalanche.benchmarks.ni_benchmark(train_dataset: Union[Sequence[SupportedDataset], SupportedDataset], test_dataset: Union[Sequence[SupportedDataset], SupportedDataset], n_experiences: int, *, task_labels: bool = False, shuffle: bool = True, seed: Optional[int] = None, balance_experiences: bool = False, min_class_patterns_in_exp: int = 0, fixed_exp_assignment: Optional[Sequence[Sequence[int]]] = None, train_transform=None, eval_transform=None, reproducibility_data: Optional[Dict[str, Any]] = None) NIScenario[source]

This is the high-level benchmark instances generator for the “New Instances” (NI) case. Given a sequence of train and test datasets creates the continual stream of data as a series of experiences.

This is the reference helper function for creating instances of Domain-Incremental benchmarks.

The task_labels parameter determines if each incremental experience has an increasing task label or if, at the contrary, a default task label “0” has to be assigned to all experiences. This can be useful when differentiating between Single-Incremental-Task and Multi-Task scenarios.

There are other important parameters that can be specified in order to tweak the behaviour of the resulting benchmark. Please take a few minutes to read and understand them as they may save you a lot of work.

This generator features an integrated reproducibility mechanism that allows the user to store and later re-load a benchmark. For more info see the reproducibility_data parameter.

Parameters
  • train_dataset – A list of training datasets, or a single dataset.

  • test_dataset – A list of test datasets, or a single test dataset.

  • n_experiences – The number of experiences.

  • task_labels – If True, each experience will have an ascending task label. If False, the task label will be 0 for all the experiences.

  • shuffle – If True, patterns order will be shuffled.

  • seed – A valid int used to initialize the random number generator. Can be None.

  • balance_experiences – If True, pattern of each class will be equally spread across all experiences. If False, patterns will be assigned to experiences in a complete random way. Defaults to False.

  • min_class_patterns_in_exp – The minimum amount of patterns of every class that must be assigned to every experience. Compatible with the balance_experiences parameter. An exception will be raised if this constraint can’t be satisfied. Defaults to 0.

  • fixed_exp_assignment – If not None, the pattern assignment to use. It must be a list with an entry for each experience. Each entry is a list that contains the indexes of patterns belonging to that experience. Overrides the shuffle, balance_experiences and min_class_patterns_in_exp parameters.

  • train_transform – The transformation to apply to the training data, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). Defaults to None.

  • eval_transform – The transformation to apply to the test data, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). Defaults to None.

  • reproducibility_data – If not None, overrides all the other benchmark definition options, including fixed_exp_assignment. This is usually a dictionary containing data used to reproduce a specific experiment. One can use the get_reproducibility_data method to get (and even distribute) the experiment setup so that it can be loaded by passing it as this parameter. In this way one can be sure that the same specific experimental setup is being used (for reproducibility purposes). Beware that, in order to reproduce an experiment, the same train and test datasets must be used. Defaults to None.

Returns

A properly initialized NIScenario instance.

avalanche.benchmarks.dataset_benchmark
avalanche.benchmarks.filelist_benchmark
avalanche.benchmarks.paths_benchmark
avalanche.benchmarks.tensors_benchmark
avalanche.benchmarks.data_incremental_benchmark(benchmark_instance: GenericCLScenario, experience_size: int, shuffle: bool = False, drop_last: bool = False, split_streams: Sequence[str] = ('train',), custom_split_strategy: Callable[[Experience], Sequence[AvalancheDataset]] = None, experience_factory: Callable[[GenericScenarioStream, int], Experience] = None)[source]

High-level benchmark generator for a Data Incremental setup.

This generator accepts an existing benchmark instance and returns a version of it in which experiences have been split in order to produce a Data Incremental stream.

In its base form this generator will split train experiences in experiences of a fixed, configurable, size. The split can be also performed on other streams (like the test one) if needed.

The custom_split_strategy parameter can be used if a more specific splitting is required.

Beware that experience splitting is NOT executed in a lazy way. This means that the splitting process takes place immediately. Consider optimizing the split process for speed when using a custom splitting strategy.

Please note that each mini-experience will have a task labels field equal to the one of the originating experience.

The complete_test_set_only field of the resulting benchmark instance will be True only if the same field of original benchmark instance is True and if the resulting test stream contains exactly one experience.

Parameters
  • benchmark_instance – The benchmark to split.

  • experience_size – The size of the experience, as an int. Ignored if custom_split_strategy is used.

  • shuffle – If True, experiences will be split by first shuffling instances in each experience. This will use the default PyTorch random number generator at its current state. Defaults to False. Ignored if custom_split_strategy is used.

  • drop_last – If True, if the last experience doesn’t contain experience_size instances, then the last experience will be dropped. Defaults to False. Ignored if custom_split_strategy is used.

  • split_streams – The list of streams to split. By default only the “train” stream will be split.

  • custom_split_strategy – A function that implements a custom splitting strategy. The function must accept an experience and return a list of datasets each describing an experience. Defaults to None, which means that the standard splitting strategy will be used (which creates experiences of size experience_size). A good starting to understand the mechanism is to look at the implementation of the standard splitting function fixed_size_experience_split_strategy().

  • experience_factory – The experience factory. Defaults to GenericExperience.

Returns

The Data Incremental benchmark instance.

avalanche.benchmarks.benchmark_with_validation_stream(benchmark_instance: GenericCLScenario, validation_size: Union[int, float], shuffle: bool = False, input_stream: str = 'train', output_stream: str = 'valid', custom_split_strategy: Callable[[Experience], Tuple[AvalancheDataset, AvalancheDataset]] = None, *, experience_factory: Callable[[GenericScenarioStream, int], Experience] = None, lazy_splitting: bool = None)[source]

Helper that can be used to obtain a benchmark with a validation stream.

This generator accepts an existing benchmark instance and returns a version of it in which a validation stream has been added.

In its base form this generator will split train experiences to extract validation experiences of a fixed (by number of instances or relative size), configurable, size. The split can be also performed on other streams if needed and the name of the resulting validation stream can be configured too.

Each validation experience will be extracted directly from a single training experience. Patterns selected for the validation experience will be removed from the training one.

If shuffle is True, the validation stream will be created randomly. Beware that no kind of class balancing is done.

The custom_split_strategy parameter can be used if a more specific splitting is required.

Please note that the resulting experiences will have a task labels field equal to the one of the originating experience.

Experience splitting can be executed in a lazy way. This behavior can be controlled using the lazy_splitting parameter. By default, experiences are split in a lazy way only when the input stream is lazily generated.

Parameters
  • benchmark_instance – The benchmark to split.

  • validation_size – The size of the validation experience, as an int or a float between 0 and 1. Ignored if custom_split_strategy is used.

  • shuffle – If True, patterns will be allocated to the validation stream randomly. This will use the default PyTorch random number generator at its current state. Defaults to False. Ignored if custom_split_strategy is used. If False, the first instances will be allocated to the training dataset by leaving the last ones to the validation dataset.

  • input_stream – The name of the input stream. Defaults to ‘train’.

  • output_stream – The name of the output stream. Defaults to ‘valid’.

  • custom_split_strategy – A function that implements a custom splitting strategy. The function must accept an experience and return a tuple containing the new train and validation dataset. Defaults to None, which means that the standard splitting strategy will be used (which creates experiences according to validation_size and shuffle). A good starting to understand the mechanism is to look at the implementation of the standard splitting function random_validation_split_strategy().

  • experience_factory – The experience factory. Defaults to GenericExperience.

  • lazy_splitting – If True, the stream will be split in a lazy way. If False, the stream will be split immediately. Defaults to None, which means that the stream will be split in a lazy or non-lazy way depending on the laziness of the input_stream.

Returns

A benchmark instance in which the validation stream has been added.

avalanche.benchmarks.CORe50(*, scenario: str = 'nicv2_391', run: int = 0, object_lvl: bool = True, mini: bool = False, train_transform: Optional[Any] = _default_train_transform, eval_transform: Optional[Any] = _default_eval_transform, dataset_root: Union[str, Path] = None)[source]

Creates a CL benchmark for CORe50.

If the dataset is not present in the computer, this method will automatically download and store it.

This generator can be used to obtain the NI, NC, NIC and NICv2-* scenarios.

The benchmark instance returned by this method will have two fields, train_stream and test_stream, which can be iterated to obtain training and test Experience. Each Experience contains the dataset and the associated task label.

The task label “0” will be assigned to each experience.

The benchmark API is quite simple and is uniform across all benchmark generators. It is recommended to check the tutorial of the “benchmark” API, which contains usage examples ranging from “basic” to “advanced”.

Parameters
  • scenario – CORe50 main scenario. It can be chosen between ‘ni’, ‘nc’, ‘nic’, ‘nicv2_79’, ‘nicv2_196’ or ‘nicv2_391.’

  • run – number of run for the benchmark. Each run defines a different ordering. Must be a number between 0 and 9.

  • object_lvl – True for a 50-way classification at the object level. False if you want to use the categories as classes. Default to True.

  • mini – True for processing reduced 32x32 images instead of the original 128x128. Default to False.

  • train_transform – The transformation to apply to the training data, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). Defaults to None.

  • eval_transform – The transformation to apply to the test data, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). Defaults to None.

  • dataset_root – Absolute path indicating where to store the dataset and related metadata. Defaults to None, which means that the default location for ‘core50’ will be used.

Returns

a properly initialized GenericCLScenario instance.

avalanche.benchmarks.SplitCIFAR10(n_experiences: int, *, first_exp_with_half_classes: bool = False, return_task_id=False, seed: Optional[int] = None, fixed_class_order: Optional[Sequence[int]] = None, shuffle: bool = True, train_transform: Optional[Any] = _default_cifar10_train_transform, eval_transform: Optional[Any] = _default_cifar10_eval_transform, dataset_root: Union[str, Path] = None) NCScenario[source]

Creates a CL benchmark using the CIFAR10 dataset.

If the dataset is not present in the computer, this method will automatically download and store it.

The returned benchmark will return experiences containing all patterns of a subset of classes, which means that each class is only seen “once”. This is one of the most common scenarios in the Continual Learning literature. Common names used in literature to describe this kind of scenario are “Class Incremental”, “New Classes”, etc. By default, an equal amount of classes will be assigned to each experience.

This generator doesn’t force a choice on the availability of task labels, a choice that is left to the user (see the return_task_id parameter for more info on task labels).

The benchmark instance returned by this method will have two fields, train_stream and test_stream, which can be iterated to obtain training and test Experience. Each Experience contains the dataset and the associated task label.

The benchmark API is quite simple and is uniform across all benchmark generators. It is recommended to check the tutorial of the “benchmark” API, which contains usage examples ranging from “basic” to “advanced”.

Parameters
  • n_experiences – The number of experiences in the current benchmark. The value of this parameter should be a divisor of 10 if first_task_with_half_classes is False, a divisor of 5 otherwise.

  • first_exp_with_half_classes – A boolean value that indicates if a first pretraining step containing half of the classes should be used. If it’s True, the first experience will use half of the classes (5 for cifar10). If this parameter is False, no pretraining step will be used and the dataset is simply split into a the number of experiences defined by the parameter n_experiences. Defaults to False.

  • return_task_id – if True, a progressive task id is returned for every experience. If False, all experiences will have a task ID of 0.

  • seed – A valid int used to initialize the random number generator. Can be None.

  • fixed_class_order – A list of class IDs used to define the class order. If None, value of seed will be used to define the class order. If not None, the seed parameter will be ignored. Defaults to None.

  • shuffle – If true, the class order in the incremental experiences is randomly shuffled. Default to false.

  • train_transform – The transformation to apply to the training data, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). If no transformation is passed, the default train transformation will be used.

  • eval_transform – The transformation to apply to the test data, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). If no transformation is passed, the default eval transformation will be used.

  • dataset_root – The root path of the dataset. Defaults to None, which means that the default location for ‘cifar10’ will be used.

Returns

A properly initialized NCScenario instance.

avalanche.benchmarks.SplitCIFAR100(n_experiences: int, *, first_exp_with_half_classes: bool = False, return_task_id=False, seed: Optional[int] = None, fixed_class_order: Optional[Sequence[int]] = None, shuffle: bool = True, train_transform: Optional[Any] = _default_cifar100_train_transform, eval_transform: Optional[Any] = _default_cifar100_eval_transform, dataset_root: Union[str, Path] = None)[source]

Creates a CL benchmark using the CIFAR100 dataset.

If the dataset is not present in the computer, this method will automatically download and store it.

The returned benchmark will return experiences containing all patterns of a subset of classes, which means that each class is only seen “once”. This is one of the most common scenarios in the Continual Learning literature. Common names used in literature to describe this kind of scenario are “Class Incremental”, “New Classes”, etc. By default, an equal amount of classes will be assigned to each experience.

This generator doesn’t force a choice on the availability of task labels, a choice that is left to the user (see the return_task_id parameter for more info on task labels).

The benchmark instance returned by this method will have two fields, train_stream and test_stream, which can be iterated to obtain training and test Experience. Each Experience contains the dataset and the associated task label.

The benchmark API is quite simple and is uniform across all benchmark generators. It is recommended to check the tutorial of the “benchmark” API, which contains usage examples ranging from “basic” to “advanced”.

Parameters
  • n_experiences – The number of incremental experiences in the current benchmark. The value of this parameter should be a divisor of 100 if first_task_with_half_classes is False, a divisor of 50 otherwise.

  • first_exp_with_half_classes – A boolean value that indicates if a first pretraining batch containing half of the classes should be used. If it’s True, a pretraining experience with half of the classes (50 for cifar100) is used. If this parameter is False no pretraining task will be used, and the dataset is simply split into a the number of experiences defined by the parameter n_experiences. Default to False.

  • return_task_id – if True, a progressive task id is returned for every experience. If False, all experiences will have a task ID of 0.

  • seed – A valid int used to initialize the random number generator. Can be None.

  • fixed_class_order – A list of class IDs used to define the class order. If None, value of seed will be used to define the class order. If non-None, seed parameter will be ignored. Defaults to None.

  • shuffle – If true, the class order in the incremental experiences is randomly shuffled. Default to false.

  • train_transform – The transformation to apply to the training data, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). If no transformation is passed, the default train transformation will be used.

  • eval_transform – The transformation to apply to the test data, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). If no transformation is passed, the default test transformation will be used.

  • dataset_root – The root path of the dataset. Defaults to None, which means that the default location for ‘cifar100’ will be used.

Returns

A properly initialized NCScenario instance.

avalanche.benchmarks.SplitCIFAR110(n_experiences: int, *, seed: Optional[int] = None, fixed_class_order: Optional[Sequence[int]] = None, train_transform: Optional[Any] = _default_cifar100_train_transform, eval_transform: Optional[Any] = _default_cifar100_eval_transform, dataset_root_cifar10: Union[str, Path] = None, dataset_root_cifar100: Union[str, Path] = None) NCScenario[source]

Creates a CL benchmark using both the CIFAR100 and CIFAR10 datasets.

If the datasets are not present in the computer, this method will automatically download and store them in the data folder.

The CIFAR10 dataset is used to create the first experience, while the remaining n_experiences-1 experiences will be created from CIFAR100.

The returned benchmark will return experiences containing all patterns of a subset of classes, which means that each class is only seen “once”. This is one of the most common scenarios in the Continual Learning literature. Common names used in literature to describe this kind of scenario are “Class Incremental”, “New Classes”, etc. By default, an equal amount of classes will be assigned to each experience.

This generator will apply a task label “0” to all experiences.

The benchmark instance returned by this method will have two fields, train_stream and test_stream, which can be iterated to obtain training and test Experience. Each Experience contains the dataset and the associated task label (always “0” for this specific benchmark).

The benchmark API is quite simple and is uniform across all benchmark generators. It is recommended to check the tutorial of the “benchmark” API, which contains usage examples ranging from “basic” to “advanced”.

Parameters
  • n_experiences – The number of experiences for the entire benchmark. The first experience will contain the entire CIFAR10 dataset, while the other n-1 experiences will be obtained from CIFAR100.

  • seed – A valid int used to initialize the random number generator. Can be None.

  • fixed_class_order – A list of class IDs used to define the class order ONLY for the incremental part, which is based on cifar100. The classes must be in range 0-99. If None, value of seed will be used to define the class order for the incremental batches on cifar100. If non-None, seed parameter will be ignored. Defaults to None.

  • train_transform – The transformation to apply to the training data, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). If no transformation is passed, the default train transformation will be used.

  • eval_transform – The transformation to apply to the test data, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). If no transformation is passed, the default test transformation will be used.

  • dataset_root_cifar10 – The root path of the CIFAR-10 dataset. Defaults to None, which means that the default location for ‘cifar10’ will be used.

  • dataset_root_cifar100 – The root path of the CIFAR-100 dataset. Defaults to None, which means that the default location for ‘cifar100’ will be used.

Returns

A properly initialized NCScenario instance.

avalanche.benchmarks.SplitCUB200(n_experiences=11, *, classes_first_batch=100, return_task_id=False, seed=0, fixed_class_order=None, shuffle=False, train_transform: Optional[Any] = _default_train_transform, eval_transform: Optional[Any] = _default_eval_transform, dataset_root: Union[str, Path] = None)[source]

Creates a CL benchmark using the Cub-200 dataset.

If the dataset is not present in the computer, this method will NOT be able automatically download and store it.

The returned benchmark will return experiences containing all patterns of a subset of classes, which means that each class is only seen “once”. This is one of the most common scenarios in the Continual Learning literature. Common names used in literature to describe this kind of scenario are “Class Incremental”, “New Classes”, etc. By default, an equal amount of classes will be assigned to each experience.

This generator doesn’t force a choice on the availability of task labels, a choice that is left to the user (see the return_task_id parameter for more info on task labels).

The benchmark instance returned by this method will have two fields, train_stream and test_stream, which can be iterated to obtain training and test Experience. Each Experience contains the dataset and the associated task label.

The benchmark API is quite simple and is uniform across all benchmark generators. It is recommended to check the tutorial of the “benchmark” API, which contains usage examples ranging from “basic” to “advanced”.

Parameters
  • n_experiences – The number of experiences in the current benchmark. Defaults to 11.

  • classes_first_batch – Number of classes in the first batch. Usually this is set to 500. Defaults to 100.

  • return_task_id – if True, a progressive task id is returned for every experience. If False, all experiences will have a task ID of 0.

  • seed – A valid int used to initialize the random number generator. Can be None.

  • fixed_class_order – A list of class IDs used to define the class order. If None, value of seed will be used to define the class order. If non-None, seed parameter will be ignored. Defaults to None.

  • shuffle – If true, the class order in the incremental experiences is randomly shuffled. Default to false.

  • train_transform – The transformation to apply to the training data, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). If no transformation is passed, the default train transformation will be used.

  • eval_transform – The transformation to apply to the test data, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). If no transformation is passed, the default test transformation will be used.

  • dataset_root – The root path of the dataset. Defaults to None, which means that the default location for ‘CUB_200_2011’ will be used.

Returns

A properly initialized NCScenario instance.

avalanche.benchmarks.SplitFMNIST(n_experiences: int, *, first_batch_with_half_classes: bool = False, return_task_id=False, seed: Optional[int] = None, fixed_class_order: Optional[Sequence[int]] = None, shuffle: bool = True, train_transform: Optional[Any] = _default_fmnist_train_transform, eval_transform: Optional[Any] = _default_fmnist_eval_transform, dataset_root: Union[str, Path] = None)[source]

Creates a CL benchmark using the Fashion MNIST dataset.

If the dataset is not present in the computer, this method will automatically download and store it.

The returned benchmark will return experiences containing all patterns of a subset of classes, which means that each class is only seen “once”. This is one of the most common scenarios in the Continual Learning literature. Common names used in literature to describe this kind of scenario are “Class Incremental”, “New Classes”, etc. By default, an equal amount of classes will be assigned to each experience.

This generator doesn’t force a choice on the availability of task labels, a choice that is left to the user (see the return_task_id parameter for more info on task labels).

The benchmark instance returned by this method will have two fields, train_stream and test_stream, which can be iterated to obtain training and test Experience. Each Experience contains the dataset and the associated task label.

The benchmark API is quite simple and is uniform across all benchmark generators. It is recommended to check the tutorial of the “benchmark” API, which contains usage examples ranging from “basic” to “advanced”.

Parameters
  • n_experiences – The number of experiences in the current benchmark. If the first experience is a “pretraining” step and it contains half of the classes. The value of this parameter should be a divisor of 10 if first_task_with_half_classes if false, a divisor of 5 otherwise.

  • first_batch_with_half_classes – A boolean value that indicates if a first pretraining batch containing half of the classes should be used. If it’s True, a pretraining batch with half of the classes (5 for cifar100) is used. If this parameter is False no pretraining task will be used, and the dataset is simply split into a the number of experiences defined by the parameter n_experiences. Default to False.

  • return_task_id – if True, a progressive task id is returned for every experience. If False, all experiences will have a task ID of 0.

  • seed – A valid int used to initialize the random number generator. Can be None.

  • fixed_class_order – A list of class IDs used to define the class order. If None, value of seed will be used to define the class order. If non-None, seed parameter will be ignored. Defaults to None.

  • shuffle – If true, the class order in the incremental experiences is randomly shuffled. Default to false.

  • train_transform – The transformation to apply to the training data, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). If no transformation is passed, the default train transformation will be used.

  • eval_transform – The transformation to apply to the test data, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). If no transformation is passed, the default test transformation will be used.

  • dataset_root – The root path of the dataset. Defaults to None, which means that the default location for ‘fashionmnist’ will be used.

Returns

A properly initialized NCScenario instance.

avalanche.benchmarks.SplitImageNet(dataset_root: Union[str, Path], *, n_experiences=10, per_exp_classes=None, return_task_id=False, seed=0, fixed_class_order=None, shuffle: bool = True, train_transform: Optional[Any] = _default_train_transform, eval_transform: Optional[Any] = _default_eval_transform)[source]

Creates a CL benchmark using the ImageNet dataset.

If the dataset is not present in the computer, this method will NOT be able automatically download and store it.

The returned benchmark will return experiences containing all patterns of a subset of classes, which means that each class is only seen “once”. This is one of the most common scenarios in the Continual Learning literature. Common names used in literature to describe this kind of scenario are “Class Incremental”, “New Classes”, etc. By default, an equal amount of classes will be assigned to each experience.

This generator doesn’t force a choice on the availability of task labels, a choice that is left to the user (see the return_task_id parameter for more info on task labels).

The benchmark instance returned by this method will have two fields, train_stream and test_stream, which can be iterated to obtain training and test Experience. Each Experience contains the dataset and the associated task label.

The benchmark API is quite simple and is uniform across all benchmark generators. It is recommended to check the tutorial of the “benchmark” API, which contains usage examples ranging from “basic” to “advanced”.

Parameters
  • dataset_root – Base path where Imagenet data is stored.

  • n_experiences – The number of experiences in the current benchmark.

  • per_exp_classes – Is not None, a dictionary whose keys are (0-indexed) experience IDs and their values are the number of classes to include in the respective experiences. The dictionary doesn’t have to contain a key for each experience! All the remaining exps will contain an equal amount of the remaining classes. The remaining number of classes must be divisible without remainder by the remaining number of experiences. For instance, if you want to include 50 classes in the first experience while equally distributing remaining classes across remaining experiences, just pass the “{0: 50}” dictionary as the per_experience_classes parameter. Defaults to None.

  • return_task_id – if True, a progressive task id is returned for every experience. If False, all experiences will have a task ID of 0.

  • seed – A valid int used to initialize the random number generator. Can be None.

  • fixed_class_order – A list of class IDs used to define the class order. If None, value of seed will be used to define the class order. If non-None, seed parameter will be ignored. Defaults to None.

  • shuffle – If true, the class order in the incremental experiences is randomly shuffled. Default to false.

  • train_transform – The transformation to apply to the training data, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). If no transformation is passed, the default train transformation will be used.

  • eval_transform – The transformation to apply to the test data, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). If no transformation is passed, the default test transformation will be used.

Returns

A properly initialized NCScenario instance.

avalanche.benchmarks.SplitMNIST(n_experiences: int, *, return_task_id=False, seed: Optional[int] = None, fixed_class_order: Optional[Sequence[int]] = None, shuffle: bool = True, train_transform: Optional[Any] = _default_mnist_train_transform, eval_transform: Optional[Any] = _default_mnist_eval_transform, dataset_root: Union[str, Path] = None)[source]

Creates a CL benchmark using the MNIST dataset.

If the dataset is not present in the computer, this method will automatically download and store it.

The returned benchmark will return experiences containing all patterns of a subset of classes, which means that each class is only seen “once”. This is one of the most common scenarios in the Continual Learning literature. Common names used in literature to describe this kind of scenario are “Class Incremental”, “New Classes”, etc. By default, an equal amount of classes will be assigned to each experience.

This generator doesn’t force a choice on the availability of task labels, a choice that is left to the user (see the return_task_id parameter for more info on task labels).

The benchmark instance returned by this method will have two fields, train_stream and test_stream, which can be iterated to obtain training and test Experience. Each Experience contains the dataset and the associated task label.

The benchmark API is quite simple and is uniform across all benchmark generators. It is recommended to check the tutorial of the “benchmark” API, which contains usage examples ranging from “basic” to “advanced”.

Parameters
  • n_experiences – The number of incremental experiences in the current benchmark. The value of this parameter should be a divisor of 10.

  • return_task_id – if True, a progressive task id is returned for every experience. If False, all experiences will have a task ID of 0.

  • seed – A valid int used to initialize the random number generator. Can be None.

  • fixed_class_order – A list of class IDs used to define the class order. If None, value of seed will be used to define the class order. If non-None, seed parameter will be ignored. Defaults to None.

  • shuffle – If true, the class order in the incremental experiences is randomly shuffled. Default to false.

  • train_transform – The transformation to apply to the training data, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). If no transformation is passed, the default train transformation will be used.

  • eval_transform – The transformation to apply to the test data, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). If no transformation is passed, the default test transformation will be used.

  • dataset_root – The root path of the dataset. Defaults to None, which means that the default location for ‘mnist’ will be used.

Returns

A properly initialized NCScenario instance.

avalanche.benchmarks.PermutedMNIST(n_experiences: int, *, seed: Optional[int] = None, train_transform: Optional[Any] = _default_mnist_train_transform, eval_transform: Optional[Any] = _default_mnist_eval_transform, dataset_root: Union[str, Path] = None) NCScenario[source]

Creates a Permuted MNIST benchmark.

If the dataset is not present in the computer, this method will automatically download and store it.

Random pixel permutations are used to permute the MNIST images in n_experiences different manners. This means that each experience is composed of all the original 10 MNIST classes, but the pixel in the images are permuted in a different way.

The benchmark instance returned by this method will have two fields, train_stream and test_stream, which can be iterated to obtain training and test Experience. Each Experience contains the dataset and the associated task label.

A progressive task label, starting from “0”, is applied to each experience.

The benchmark API is quite simple and is uniform across all benchmark generators. It is recommended to check the tutorial of the “benchmark” API, which contains usage examples ranging from “basic” to “advanced”.

Parameters
  • n_experiences – The number of experiences (tasks) in the current benchmark. It indicates how many different permutations of the MNIST dataset have to be created. The value of this parameter should be a divisor of 10.

  • seed – A valid int used to initialize the random number generator. Can be None.

  • train_transform – The transformation to apply to the training data before the random permutation, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). If no transformation is passed, the default train transformation will be used.

  • eval_transform – The transformation to apply to the test data before the random permutation, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). If no transformation is passed, the default test transformation will be used.

  • dataset_root – The root path of the dataset. Defaults to None, which means that the default location for ‘mnist’ will be used.

Returns

A properly initialized NCScenario instance.

avalanche.benchmarks.RotatedMNIST(n_experiences: int, *, seed: Optional[int] = None, rotations_list: Optional[Sequence[int]] = None, train_transform: Optional[Any] = _default_mnist_train_transform, eval_transform: Optional[Any] = _default_mnist_eval_transform, dataset_root: Union[str, Path] = None) NCScenario[source]

Creates a Rotated MNIST benchmark.

If the dataset is not present in the computer, this method will automatically download and store it.

Random angles are used to rotate the MNIST images in n_experiences different manners. This means that each experience is composed of all the original 10 MNIST classes, but each image is rotated in a different way.

The benchmark instance returned by this method will have two fields, train_stream and test_stream, which can be iterated to obtain training and test Experience. Each Experience contains the dataset and the associated task label.

A progressive task label, starting from “0”, is applied to each experience.

The benchmark API is quite simple and is uniform across all benchmark generators. It is recommended to check the tutorial of the “benchmark” API, which contains usage examples ranging from “basic” to “advanced”.

Parameters
  • n_experiences – The number of experiences (tasks) in the current benchmark. It indicates how many different rotations of the MNIST dataset have to be created. The value of this parameter should be a divisor of 10.

  • seed – A valid int used to initialize the random number generator. Can be None.

  • rotations_list – A list of rotations values in degrees (from -180 to 180) used to define the rotations. The rotation specified in position 0 of the list will be applied to the task 0, the rotation specified in position 1 will be applied to task 1 and so on. If None, value of seed will be used to define the rotations. If non-None, seed parameter will be ignored. Defaults to None.

  • train_transform – The transformation to apply to the training data after the random rotation, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). If no transformation is passed, the default train transformation will be used.

  • eval_transform – The transformation to apply to the test data after the random rotation, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). If no transformation is passed, the default test transformation will be used.

  • dataset_root – The root path of the dataset. Defaults to None, which means that the default location for ‘mnist’ will be used.

Returns

A properly initialized NCScenario instance.

avalanche.benchmarks.SplitOmniglot(n_experiences: int, *, return_task_id=False, seed: Optional[int] = None, fixed_class_order: Optional[Sequence[int]] = None, shuffle: bool = True, train_transform: Optional[Any] = _default_omniglot_train_transform, eval_transform: Optional[Any] = _default_omniglot_eval_transform, dataset_root: Union[str, Path] = None)[source]

Creates a CL benchmark using the OMNIGLOT dataset.

If the dataset is not present in the computer, this method will automatically download and store it.

The returned benchmark will return experiences containing all patterns of a subset of classes, which means that each class is only seen “once”. This is one of the most common scenarios in the Continual Learning literature. Common names used in literature to describe this kind of scenario are “Class Incremental”, “New Classes”, etc.

By default, an equal amount of classes will be assigned to each experience. OMNIGLOT consists of 964 classes, which means that the number of experiences can be 1, 2, 4, 241, 482, 964.

This generator doesn’t force a choice on the availability of task labels, a choice that is left to the user (see the return_task_id parameter for more info on task labels).

The benchmark instance returned by this method will have two fields, train_stream and test_stream, which can be iterated to obtain training and test Experience. Each Experience contains the dataset and the associated task label.

The benchmark API is quite simple and is uniform across all benchmark generators. It is recommended to check the tutorial of the “benchmark” API, which contains usage examples ranging from “basic” to “advanced”.

Parameters
  • n_experiences – The number of incremental experiences in the current benchmark. The value of this parameter should be a divisor of 10.

  • return_task_id – if True, a progressive task id is returned for every experience. If False, all experiences will have a task ID of 0.

  • seed – A valid int used to initialize the random number generator. Can be None.

  • fixed_class_order – A list of class IDs used to define the class order. If None, value of seed will be used to define the class order. If non-None, seed parameter will be ignored. Defaults to None.

  • shuffle – If true, the class order in the incremental experiences is randomly shuffled. Default to false.

  • train_transform – The transformation to apply to the training data, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). If no transformation is passed, the default train transformation will be used.

  • eval_transform – The transformation to apply to the test data, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). If no transformation is passed, the default test transformation will be used.

  • dataset_root – The root path of the dataset. Defaults to None, which means that the default location for ‘omniglot’ will be used.

Returns

A properly initialized NCScenario instance.

avalanche.benchmarks.PermutedOmniglot(n_experiences: int, *, seed: Optional[int] = None, train_transform: Optional[Any] = _default_omniglot_train_transform, eval_transform: Optional[Any] = _default_omniglot_eval_transform, dataset_root: Union[str, Path] = None) NCScenario[source]

Creates a Permuted Omniglot benchmark.

If the dataset is not present in the computer, this method will automatically download and store it.

Random pixel permutations are used to permute the Omniglot images in n_experiences different manners. This means that each experience is composed of all the original 964 Omniglot classes, but the pixel in the images are permuted in a different way.

The benchmark instance returned by this method will have two fields, train_stream and test_stream, which can be iterated to obtain training and test Experience. Each Experience contains the dataset and the associated task label.

A progressive task label, starting from “0”, is applied to each experience.

The benchmark API is quite simple and is uniform across all benchmark generators. It is recommended to check the tutorial of the “benchmark” API, which contains usage examples ranging from “basic” to “advanced”.

Parameters
  • n_experiences – The number of experiences (tasks) in the current benchmark. It indicates how many different permutations of the Omniglot dataset have to be created.

  • seed – A valid int used to initialize the random number generator. Can be None.

  • train_transform – The transformation to apply to the training data before the random permutation, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). If no transformation is passed, the default train transformation will be used.

  • eval_transform – The transformation to apply to the test data before the random permutation, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). If no transformation is passed, the default test transformation will be used.

  • dataset_root – The root path of the dataset. Defaults to None, which means that the default location for ‘omniglot’ will be used.

Returns

A properly initialized NCScenario instance.

avalanche.benchmarks.RotatedOmniglot(n_experiences: int, *, seed: Optional[int] = None, rotations_list: Optional[Sequence[int]] = None, train_transform: Optional[Any] = _default_omniglot_train_transform, eval_transform: Optional[Any] = _default_omniglot_eval_transform, dataset_root: Union[str, Path] = None) NCScenario[source]

Creates a Rotated Omniglot benchmark.

If the dataset is not present in the computer, this method will automatically download and store it.

Random angles are used to rotate the Omniglot images in n_experiences different manners. This means that each experience is composed of all the original 964 Omniglot classes, but each image is rotated in a different way.

The benchmark instance returned by this method will have two fields, train_stream and test_stream, which can be iterated to obtain training and test Experience. Each Experience contains the dataset and the associated task label.

A progressive task label, starting from “0”, is applied to each experience.

The benchmark API is quite simple and is uniform across all benchmark generators. It is recommended to check the tutorial of the “benchmark” API, which contains usage examples ranging from “basic” to “advanced”.

Parameters
  • n_experiences – The number of experiences (tasks) in the current benchmark. It indicates how many different rotations of the Omniglot dataset have to be created.

  • seed – A valid int used to initialize the random number generator. Can be None.

  • rotations_list – A list of rotations values in degrees (from -180 to 180) used to define the rotations. The rotation specified in position 0 of the list will be applied to the task 0, the rotation specified in position 1 will be applied to task 1 and so on. If None, value of seed will be used to define the rotations. If non-None, seed parameter will be ignored. Defaults to None.

  • train_transform – The transformation to apply to the training data after the random rotation, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). If no transformation is passed, the default train transformation will be used.

  • eval_transform – The transformation to apply to the test data after the random rotation, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). If no transformation is passed, the default test transformation will be used.

  • dataset_root – The root path of the dataset. Defaults to None, which means that the default location for ‘omniglot’ will be used.

Returns

A properly initialized NCScenario instance.

avalanche.benchmarks.SplitTinyImageNet(n_experiences=10, *, return_task_id=False, seed=0, fixed_class_order=None, shuffle: bool = True, train_transform: Optional[Any] = _default_train_transform, eval_transform: Optional[Any] = _default_eval_transform, dataset_root: Union[str, Path] = None)[source]

Creates a CL benchmark using the Tiny ImageNet dataset.

If the dataset is not present in the computer, this method will automatically download and store it.

The returned benchmark will return experiences containing all patterns of a subset of classes, which means that each class is only seen “once”. This is one of the most common scenarios in the Continual Learning literature. Common names used in literature to describe this kind of scenario are “Class Incremental”, “New Classes”, etc. By default, an equal amount of classes will be assigned to each experience.

This generator doesn’t force a choice on the availability of task labels, a choice that is left to the user (see the return_task_id parameter for more info on task labels).

The benchmark instance returned by this method will have two fields, train_stream and test_stream, which can be iterated to obtain training and test Experience. Each Experience contains the dataset and the associated task label.

The benchmark API is quite simple and is uniform across all benchmark generators. It is recommended to check the tutorial of the “benchmark” API, which contains usage examples ranging from “basic” to “advanced”.

Parameters
  • n_experiences – The number of experiences in the current benchmark.

  • return_task_id – if True, a progressive task id is returned for every experience. If False, all experiences will have a task ID of 0.

  • seed – A valid int used to initialize the random number generator. Can be None.

  • fixed_class_order – A list of class IDs used to define the class order. If None, value of seed will be used to define the class order. If non-None, seed parameter will be ignored. Defaults to None.

  • shuffle – If true, the class order in the incremental experiences is randomly shuffled. Default to false.

  • train_transform – The transformation to apply to the training data, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). If no transformation is passed, the default train transformation will be used.

  • eval_transform – The transformation to apply to the test data, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). If no transformation is passed, the default test transformation will be used.

  • dataset_root – The root path of the dataset. Defaults to None, which means that the default location for ‘tinyimagenet’ will be used.

Returns

A properly initialized NCScenario instance.

avalanche.benchmarks.OpenLORIS(*, factor: Literal['clutter', 'illumination', 'occlusion', 'pixel', 'mixture-iros'] = 'clutter', train_transform: Optional[Any] = None, eval_transform: Optional[Any] = None, dataset_root: Union[str, Path] = None)[source]

Creates a CL benchmark for OpenLORIS.

If the dataset is not present in the computer, this method will NOT be able automatically download and store it.

This generator can be used to obtain scenarios based on different “factors”. Valid factors include ‘clutter’, ‘illumination’, ‘occlusion’, ‘pixel’, or ‘mixture-iros’.

The benchmark instance returned by this method will have two fields, train_stream and test_stream, which can be iterated to obtain training and test Experience. Each Experience contains the dataset and the associated task label.

The task label “0” will be assigned to each experience.

The benchmark API is quite simple and is uniform across all benchmark generators. It is recommended to check the tutorial of the “benchmark” API, which contains usage examples ranging from “basic” to “advanced”.

Parameters
  • factor – OpenLORIS main factors, indicating different environmental variations. It can be chosen between ‘clutter’, ‘illumination’, ‘occlusion’, ‘pixel’, or ‘mixture-iros’. The first three factors are included in the ICRA 2020 paper and the last factor (mixture-iros) is the benchmark setting for IROS 2019 Lifelong robotic vision competition.

  • train_transform – The transformation to apply to the training data, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). Defaults to None.

  • eval_transform – The transformation to apply to the test data, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). Defaults to None.

  • dataset_root – The root path of the dataset. Defaults to None, which means that the default location for ‘openloris’ will be used.

Returns

a properly initialized GenericCLScenario instance.

class avalanche.benchmarks.Stream51(root: Union[str, Path] = None, *, train=True, transform=None, target_transform=None, loader=default_loader, download=True)[source]

Bases: avalanche.benchmarks.datasets.DownloadableDataset

Stream-51 Pytorch Dataset

Creates an instance of the Stream-51 dataset.

Parameters
  • root – The directory where the dataset can be found or downloaded. Defaults to None, which means that the default location for ‘stream51’ will be used.

  • train – If True, the training set will be returned. If False, the test set will be returned.

  • transform – The transformations to apply to the X values.

  • target_transform – The transformations to apply to the Y values.

  • loader – The image loader to use.

  • download – If True, the dataset will be downloaded if needed.

static make_dataset(data_list, ordering='class_instance', seed=666)[source]

data_list for train: [class_id, clip_num, video_num, frame_num, bbox, file_loc] for test: [class_id, bbox, file_loc]

__getitem__(self, index)[source]
Args:

index (int): Index

Returns:

tuple: (sample, target) where target is class_index of the target class.

__len__(self)[source]
__repr__(self)[source]

Return repr(self).