avalanche.benchmarks.scenarios.benchmark_with_validation_stream

avalanche.benchmarks.scenarios.benchmark_with_validation_stream(benchmark: CLScenario, validation_size: int | float = 0.5, shuffle: bool = False, seed: int | None = None, split_strategy: Callable[[AvalancheDataset], Tuple[AvalancheDataset, AvalancheDataset]] | None = None) → CLScenario[source]

Helper to obtain a benchmark with a validation stream.

This generator accepts an existing benchmark instance and returns a version of it in which the train stream has been split into training and validation streams.

Each train/validation experience will be by splitting the original training experiences. Patterns selected for the validation experience will be removed from the training experiences.

The default splitting strategy is a random split as implemented by split_validation_random. If you want to use class balancing you can use split_validation_class_balanced, or use a custom split_strategy, as shown in the following example:

validation_size = 0.2
foo = lambda exp: split_dataset_class_balanced(validation_size, exp)
bm = benchmark_with_validation_stream(bm, custom_split_strategy=foo)
Parameters:

  • benchmark – The benchmark to split.

  • validation_size – The size of the validation experience, as an int or a float between 0 and 1. Ignored if custom_split_strategy is used.

  • shuffle – If True, patterns will be allocated to the validation stream randomly. This will use the default PyTorch random number generator at its current state. Defaults to False. Ignored if custom_split_strategy is used. If False, the first instances will be allocated to the training dataset by leaving the last ones to the validation dataset.

  • split_strategy – A function that implements a custom splitting strategy. The function must accept an AvalancheDataset and return a tuple containing the new train and validation dataset. By default, the splitting strategy will split the data according to validation_size and shuffle). A good starting to understand the mechanism is to look at the implementation of the standard splitting function random_validation_split_strategy().

Returns:

A benchmark instance in which the validation stream has been added.