avalanche.evaluation.metrics.AverageMeanClassAccuracy

class avalanche.evaluation.metrics.AverageMeanClassAccuracy(classes: Optional[Union[Dict[int, Iterable[int]], Iterable[int]]] = None)[source]

The Average Mean Class Accuracy (AMCA) metric. This is a standalone metric used to compute more specific ones.

Instances of this metric keeps the running average accuracy over multiple <prediction, target> pairs of Tensors, provided incrementally.

Beware that this class does not provide mechanisms to separate scores based on the originating data stream. For this, please refer to MultiStreamAMCA.

The “prediction” and “target” tensors may contain plain labels or one-hot/logit vectors.

Each time result is called, this metric emits the average mean accuracy as the average accuracy of all previous experiences (also considering the accuracy in the current experience). The metric expects that the next_experience() method will be called after each experience. This is needed to consolidate the current mean accuracy. After calling next_experience(), a new experience with accuracy 0.0 is immediately started. If you need to obtain the AMCA up to experience t-1, obtain the result() before calling next_experience().

The set of classes to be tracked can be reduced (please refer to the constructor parameters).

The reset method will bring the metric to its initial state (tracked classes will be kept). By default, this metric in its initial state will return a {task_id -> amca} dictionary in which all AMCAs are set to 0 (that is, the reset method will hardly be useful when using this metric).

__init__(classes: Optional[Union[Dict[int, Iterable[int]], Iterable[int]]] = None)[source]

Creates an instance of the standalone AMCA metric.

By default, this metric in its initial state will return an empty dictionary. The metric can be updated by using the update method while the running AMCA can be retrieved using the result method.

By using the classes parameter, one can restrict the list of classes to be tracked and in addition will initialize the accuracy for that class to 0.0.

Setting the classes parameter is very important, as the mean class accuracy may vary based on this! If the test set is fixed and contains at least a sample for each class, then it is safe to leave classes to None.

Parameters

classes – The classes to keep track of. If None (default), all classes seen are tracked. Otherwise, it can be a dict of classes to be tracked (as “task-id” -> “list of class ids”) or, if running a task-free benchmark (with only task 0), a simple list of class ids. By passing this parameter, the list of classes to be considered is created immediately. This will ensure that the mean class accuracy is correctly computed. In addition, this can be used to restrict the classes that should be considered when computing the mean class accuracy.

Methods

__init__([classes])

Creates an instance of the standalone AMCA metric.

next_experience()

Moves to the next experience.

reset()

Resets the metric.

result()

Retrieves the running AMCA for each task.

update(predicted_y, true_y, task_labels)

Update the running accuracy given the true and predicted labels for each class.