avalanche.training.plugins.EvaluationPlugin

class avalanche.training.plugins.EvaluationPlugin(*metrics: Union[PluginMetric, Sequence[PluginMetric]], loggers: Union[StrategyLogger, Sequence[StrategyLogger]] = None, collect_all=True, benchmark=None, strict_checks=False, suppress_warnings=False)[source]

Manager for logging and metrics.

An evaluation plugin that obtains relevant data from the training and eval loops of the strategy through callbacks. The plugin keeps a dictionary with the last recorded value for each metric. The dictionary will be returned by the train and eval methods of the strategies. It is also possible to keep a dictionary with all recorded metrics by specifying collect_all=True. The dictionary can be retrieved via the get_all_metrics method.

This plugin also logs metrics using the provided loggers.

__init__(*metrics: Union[PluginMetric, Sequence[PluginMetric]], loggers: Union[StrategyLogger, Sequence[StrategyLogger]] = None, collect_all=True, benchmark=None, strict_checks=False, suppress_warnings=False)[source]

Creates an instance of the evaluation plugin.

Parameters
  • metrics – The metrics to compute.

  • loggers – The loggers to be used to log the metric values.

  • collect_all – if True, collect in a separate dictionary all metric curves values. This dictionary is accessible with get_all_metrics method.

  • benchmark – continual learning benchmark needed to check stream completeness during evaluation or other kind of properties. If None, no check will be conducted and the plugin will emit a warning to signal this fact.

  • strict_checks – if True, benchmark has to be provided. In this case, only full evaluation streams are admitted when calling eval. An error will be raised otherwise. When False, benchmark can be None and only warnings will be raised.

  • suppress_warnings – if True, warnings and errors will never be raised from the plugin. If False, warnings and errors will be raised following benchmark and strict_checks behavior.

Methods

__init__(*metrics[, loggers, collect_all, ...])

Creates an instance of the evaluation plugin.

after_backward(strategy, **kwargs)

Called after criterion.backward() by the BaseStrategy.

after_eval(strategy, **kwargs)

Called after eval by the BaseStrategy.

after_eval_dataset_adaptation(strategy, **kwargs)

Called after eval_dataset_adaptation by the BaseStrategy.

after_eval_exp(strategy, **kwargs)

Called after eval_exp by the BaseStrategy.

after_eval_forward(strategy, **kwargs)

Called after model.forward() by the BaseStrategy.

after_eval_iteration(strategy, **kwargs)

Called after the end of an iteration by the BaseStrategy.

after_forward(strategy, **kwargs)

Called after model.forward() by the BaseStrategy.

after_train_dataset_adaptation(strategy, ...)

Called after train_dataset_adapatation by the BaseStrategy.

after_training(strategy, **kwargs)

Called after train by the BaseStrategy.

after_training_epoch(strategy, **kwargs)

Called after train_epoch by the BaseStrategy.

after_training_exp(strategy, **kwargs)

Called after train_exp by the BaseStrategy.

after_training_iteration(strategy, **kwargs)

Called after the end of a training iteration by the BaseStrategy.

after_update(strategy, **kwargs)

Called after optimizer.update() by the BaseStrategy.

before_backward(strategy, **kwargs)

Called before criterion.backward() by the BaseStrategy.

before_eval(strategy, **kwargs)

Called before eval by the BaseStrategy.

before_eval_dataset_adaptation(strategy, ...)

Called before eval_dataset_adaptation by the BaseStrategy.

before_eval_exp(strategy, **kwargs)

Called before eval_exp by the BaseStrategy.

before_eval_forward(strategy, **kwargs)

Called before model.forward() by the BaseStrategy.

before_eval_iteration(strategy, **kwargs)

Called before the start of a training iteration by the BaseStrategy.

before_forward(strategy, **kwargs)

Called before model.forward() by the BaseStrategy.

before_train_dataset_adaptation(strategy, ...)

Called before train_dataset_adapatation by the BaseStrategy.

before_training(strategy, **kwargs)

Called before train by the BaseStrategy.

before_training_epoch(strategy, **kwargs)

Called before train_epoch by the BaseStrategy.

before_training_exp(strategy, **kwargs)

Called before train_exp by the BaseStrategy.

before_training_iteration(strategy, **kwargs)

Called before the start of a training iteration by the BaseStrategy.

before_update(strategy, **kwargs)

Called before optimizer.update() by the BaseStrategy.

get_all_metrics()

Return the dictionary of all collected metrics.

get_last_metrics()

Return a shallow copy of dictionary with metric names as keys and last metrics value as values.

reset_last_metrics()

Set the dictionary storing last value for each metric to be empty dict.

Attributes

active