Evaluation module
PluginMetric
class, which provides all the callbacks needed to include custom metric logic in specific points of the continual learning workflow.evaluation.metrics
Metrics helper functions
EvaluationPlugin
).
|
Helper method that can be used to obtain the desired set of plugin metrics. |
|
Helper method that can be used to obtain the desired set of plugin metrics. |
|
Helper method that can be used to obtain the desired set of plugin metrics. |
|
Helper method that can be used to obtain the desired set of plugin metrics. |
|
Helper method that can be used to obtain the desired set of plugin metrics. |
|
Helper method that can be used to obtain the desired set of plugin metrics. |
|
Helper method that can be used to obtain the desired set of plugin metrics. |
|
Helper method that can be used to obtain the desired set of standalone metrics. |
|
Helper method that can be used to obtain the desired set of plugin metrics. |
|
Helper method that can be used to obtain the desired set of plugin metrics. |
|
Helper method that can be used to obtain the desired set of plugin metrics. |
|
Helper method that can be used to obtain the desired set of plugin metrics. |
|
Create plugins to monitor the labels repartition. |
|
Helper to create plugins to show the scores of the true class, averaged by |
Stream Metrics
At the end of the entire stream of experiences, this plugin metric reports the average accuracy over all patterns seen in all experiences. |
|
At the end of each experience, this plugin metric reports the average accuracy for only the experiences that the model has been trained on so far. |
|
At the end of the entire stream of experiences, this metric reports the average loss over all patterns seen in all experiences. |
|
The StreamBWT metric, emitting the average BWT across all experiences encountered during training. |
|
The StreamForgetting metric, describing the average evaluation accuracy loss detected over all experiences observed during training. |
|
The Forward Transfer averaged over all the evaluation experiences. |
|
|
The Stream Confusion Matrix metric. |
|
Confusion Matrix metric compatible with Weights and Biases logger. |
The average stream CPU usage metric. |
|
|
The average stream Disk usage metric. |
The stream time metric. |
|
|
The Stream Max RAM metric. |
|
The Stream Max GPU metric. |
|
Plugin to show the scores of the true class during evaluation, averaged by |
Experience Metrics
At the end of each experience, this plugin metric reports the average accuracy over all patterns seen in that experience. |
|
At the end of each experience, this metric reports the average loss over all patterns seen in that experience. |
|
The Experience Backward Transfer metric. |
|
The ExperienceForgetting metric, describing the accuracy loss detected for a certain experience. |
|
The Forward Transfer computed on each experience separately. |
|
The average experience CPU usage metric. |
|
|
The average experience Disk usage metric. |
The experience time metric. |
|
At the end of each experience, this metric reports the MAC computed on a single pattern. |
|
|
The Experience Max RAM metric. |
|
The Experience Max GPU metric. |
Epoch Metrics
The average accuracy over a single training epoch. |
|
The average loss over a single training epoch. |
|
The Epoch CPU usage metric. |
|
|
The Epoch Disk usage metric. |
The epoch elapsed time metric. |
|
|
The MAC at the end of each epoch computed on a single pattern. |
|
The Epoch Max RAM metric. |
|
The Epoch Max GPU metric. |
|
Plugin to show the scores of the true class during the lasts training |
RunningEpoch Metrics
The average accuracy across all minibatches up to the current epoch iteration. |
|
The average loss across all minibatches up to the current epoch iteration. |
|
The running epoch CPU usage metric. |
|
The running epoch time metric. |
Minibatch Metrics
The minibatch plugin accuracy metric. |
|
The minibatch loss metric. |
|
The minibatch CPU usage metric. |
|
|
The minibatch Disk usage metric. |
The minibatch time metric. |
|
The minibatch MAC metric. |
|
|
The Minibatch Max RAM metric. |
|
The Minibatch Max GPU metric. |
evaluation.metric_definitions
General interfaces on which metrics are built.
|
Definition of a standalone metric. |
A metric that can be used together with |
|
|
This class provides a generic implementation of a Plugin Metric. |