avalanche.evaluation

The Evaluation module provides tools and interfaces to easily define new metrics to monitor experiments. In addition, it also provides a set of already implemented metrics. These metrics can be used by passing them to the EvaluationPlugin.

Subpackages

Submodules

Package Contents

Classes

Metric

Definition of a standalone metric.

PluginMetric

A metric that can be used together with EvaluationPlugin.

GenericPluginMetric

This class provides a generic implementation of a Plugin Metric.

class avalanche.evaluation.Metric[source]

Bases: Protocol[TResult]

Definition of a standalone metric.

A standalone metric exposes methods to reset its internal state and to emit a result. Emitting a result does not automatically cause a reset in the internal state.

The specific metric implementation exposes ways to update the internal state. Usually, standalone metrics like Sum, Mean, Accuracy, … expose an update method.

The Metric class can be used as a standalone metric by directly calling its methods. In order to automatically integrate the metric with the training and evaluation flows, you can use PluginMetric class. The class receives events directly from the EvaluationPlugin and can emits values on each callback. Usually, an instance of Metric is created within PluginMetric, which is then responsible for its update and results. See PluginMetric for more details.

result(self, **kwargs) Optional[TResult][source]

Obtains the value of the metric.

Returns

The value of the metric.

reset(self, **kwargs) None[source]

Resets the metric internal state.

Returns

None.

class avalanche.evaluation.PluginMetric[source]

Bases: Metric[TResult], StrategyCallbacks['MetricResult'], abc.ABC

A metric that can be used together with EvaluationPlugin.

This class leaves the implementation of the result and reset methods to child classes while providing an empty implementation of the callbacks invoked by the EvaluationPlugin. Subclasses should implement the result, reset and the desired callbacks to compute the specific metric.

Remember to call the super() method when overriding after_train_iteration or after_eval_iteration.

An instance of this class usually leverages a Metric instance to update, reset and emit metric results at appropriate times (during specific callbacks).

Creates an instance of a plugin metric.

Child classes can safely invoke this (super) constructor as the first experience.

abstract result(self, **kwargs) Optional[TResult][source]

Obtains the value of the metric.

Returns

The value of the metric.

abstract reset(self, **kwargs) None[source]

Resets the metric internal state.

Returns

None.

before_training(self, strategy: BaseStrategy) 'MetricResult'[source]

Called before train by the BaseStrategy.

before_training_exp(self, strategy: BaseStrategy) 'MetricResult'[source]

Called before train_exp by the BaseStrategy.

before_train_dataset_adaptation(self, strategy: BaseStrategy) 'MetricResult'[source]

Called before train_dataset_adapatation by the BaseStrategy.

after_train_dataset_adaptation(self, strategy: BaseStrategy) 'MetricResult'[source]

Called after train_dataset_adapatation by the BaseStrategy.

before_training_epoch(self, strategy: BaseStrategy) 'MetricResult'[source]

Called before train_epoch by the BaseStrategy.

before_training_iteration(self, strategy: BaseStrategy) 'MetricResult'[source]

Called before the start of a training iteration by the BaseStrategy.

before_forward(self, strategy: BaseStrategy) 'MetricResult'[source]

Called before model.forward() by the BaseStrategy.

after_forward(self, strategy: BaseStrategy) 'MetricResult'[source]

Called after model.forward() by the BaseStrategy.

before_backward(self, strategy: BaseStrategy) 'MetricResult'[source]

Called before criterion.backward() by the BaseStrategy.

after_backward(self, strategy: BaseStrategy) 'MetricResult'[source]

Called after criterion.backward() by the BaseStrategy.

after_training_iteration(self, strategy: BaseStrategy) 'MetricResult'[source]

Called after the end of a training iteration by the BaseStrategy.

before_update(self, strategy: BaseStrategy) 'MetricResult'[source]

Called before optimizer.update() by the BaseStrategy.

after_update(self, strategy: BaseStrategy) 'MetricResult'[source]

Called after optimizer.update() by the BaseStrategy.

after_training_epoch(self, strategy: BaseStrategy) 'MetricResult'[source]

Called after train_epoch by the BaseStrategy.

after_training_exp(self, strategy: BaseStrategy) 'MetricResult'[source]

Called after train_exp by the BaseStrategy.

after_training(self, strategy: BaseStrategy) 'MetricResult'[source]

Called after train by the BaseStrategy.

before_eval(self, strategy: BaseStrategy) 'MetricResult'[source]

Called before eval by the BaseStrategy.

before_eval_dataset_adaptation(self, strategy: BaseStrategy) 'MetricResult'[source]

Called before eval_dataset_adaptation by the BaseStrategy.

after_eval_dataset_adaptation(self, strategy: BaseStrategy) 'MetricResult'[source]

Called after eval_dataset_adaptation by the BaseStrategy.

before_eval_exp(self, strategy: BaseStrategy) 'MetricResult'[source]

Called before eval_exp by the BaseStrategy.

after_eval_exp(self, strategy: BaseStrategy) 'MetricResult'[source]

Called after eval_exp by the BaseStrategy.

after_eval(self, strategy: BaseStrategy) 'MetricResult'[source]

Called after eval by the BaseStrategy.

before_eval_iteration(self, strategy: BaseStrategy) 'MetricResult'[source]

Called before the start of a training iteration by the BaseStrategy.

before_eval_forward(self, strategy: BaseStrategy) 'MetricResult'[source]

Called before model.forward() by the BaseStrategy.

after_eval_forward(self, strategy: BaseStrategy) 'MetricResult'[source]

Called after model.forward() by the BaseStrategy.

after_eval_iteration(self, strategy: BaseStrategy) 'MetricResult'[source]

Called after the end of an iteration by the BaseStrategy.

class avalanche.evaluation.GenericPluginMetric(metric, reset_at='experience', emit_at='experience', mode='eval')[source]

Bases: PluginMetric[TResult]

This class provides a generic implementation of a Plugin Metric. The user can subclass this class to easily implement custom plugin metrics.

Creates an instance of a plugin metric.

Child classes can safely invoke this (super) constructor as the first experience.

reset(self, strategy) None[source]
result(self, strategy)[source]
update(self, strategy)[source]
before_training(self, strategy: BaseStrategy)[source]

Called before train by the BaseStrategy.

before_training_exp(self, strategy: BaseStrategy)[source]

Called before train_exp by the BaseStrategy.

before_training_epoch(self, strategy: BaseStrategy)[source]

Called before train_epoch by the BaseStrategy.

before_training_iteration(self, strategy: BaseStrategy)[source]

Called before the start of a training iteration by the BaseStrategy.

after_training_iteration(self, strategy: BaseStrategy) None[source]

Called after the end of a training iteration by the BaseStrategy.

after_training_epoch(self, strategy: BaseStrategy)[source]

Called after train_epoch by the BaseStrategy.

after_training_exp(self, strategy: BaseStrategy)[source]

Called after train_exp by the BaseStrategy.

after_training(self, strategy: BaseStrategy)[source]

Called after train by the BaseStrategy.

before_eval(self, strategy: BaseStrategy)[source]

Called before eval by the BaseStrategy.

before_eval_exp(self, strategy: BaseStrategy)[source]

Called before eval_exp by the BaseStrategy.

after_eval_exp(self, strategy: BaseStrategy)[source]

Called after eval_exp by the BaseStrategy.

after_eval(self, strategy: BaseStrategy)[source]

Called after eval by the BaseStrategy.

after_eval_iteration(self, strategy: BaseStrategy)[source]

Called after the end of an iteration by the BaseStrategy.

before_eval_iteration(self, strategy: BaseStrategy)[source]

Called before the start of a training iteration by the BaseStrategy.