avalanche.benchmarks.generators.dataset_benchmark

avalanche.benchmarks.generators.dataset_benchmark(train_datasets: Sequence[Union[IDatasetWithTargets, ITensorDataset, Subset, ConcatDataset]], test_datasets: Sequence[Union[IDatasetWithTargets, ITensorDataset, Subset, ConcatDataset]], *, other_streams_datasets: Optional[Dict[str, Sequence[Union[IDatasetWithTargets, ITensorDataset, Subset, ConcatDataset]]]] = None, complete_test_set_only: bool = False, train_transform=None, train_target_transform=None, eval_transform=None, eval_target_transform=None, other_streams_transforms: Optional[Dict[str, Tuple[Any, Any]]] = None, dataset_type: Optional[AvalancheDatasetType] = None) GenericCLScenario

Creates a benchmark instance given a list of datasets. Each dataset will be considered as a separate experience.

Contents of the datasets must already be set, including task labels. Transformations will be applied if defined.

This function allows for the creation of custom streams as well. While “train” and “test” datasets must always be set, the experience list for other streams can be defined by using the other_streams_datasets parameter.

If transformations are defined, they will be applied to the datasets of the related stream.

Parameters
  • train_datasets – A list of training datasets.

  • test_datasets – A list of test datasets.

  • other_streams_datasets – A dictionary describing the content of custom streams. Keys must be valid stream names (letters and numbers, not starting with a number) while the value must be a list of dataset. If this dictionary contains the definition for “train” or “test” streams then those definition will override the train_datasets and test_datasets parameters.

  • complete_test_set_only – If True, only the complete test set will be returned by the benchmark. This means that the test_dataset_list parameter must be list with a single element (the complete test set). Defaults to False.

  • train_transform – The transformation to apply to the training data, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). Defaults to None.

  • train_target_transform – The transformation to apply to training patterns targets. Defaults to None.

  • eval_transform – The transformation to apply to the test data, e.g. a random crop, a normalization or a concatenation of different transformations (see torchvision.transform documentation for a comprehensive list of possible transformations). Defaults to None.

  • eval_target_transform – The transformation to apply to test patterns targets. Defaults to None.

  • other_streams_transforms – Transformations to apply to custom streams. If no transformations are defined for a custom stream, then “train” transformations will be used. This parameter must be a dictionary mapping stream names to transformations. The transformations must be a two elements tuple where the first element defines the X transformation while the second element is the Y transformation. Those elements can be None. If this dictionary contains the transformations for “train” or “test” streams then those transformations will override the train_transform, train_target_transform, eval_transform and eval_target_transform parameters.

  • dataset_type – The type of the dataset. Defaults to None, which means that the type will be obtained from the input datasets. If input datasets are not instances of AvalancheDataset, the type UNDEFINED will be used.

Returns

A GenericCLScenario instance.