> For the complete documentation index, see [llms.txt](https://linzhiqiu.gitbook.io/the-clear-benchmark/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://linzhiqiu.gitbook.io/the-clear-benchmark/introduction/evaluation-protocol-on-clear.md).

# Evaluation Protocol on CLEAR

How to faithfully evaluate a continual learning algorithm on CLEAR?

For bucket 1st to 10th that each comes with an annotated labeled trainset, we also release **a held-out testset** over the same timespan (now downloadable [here](/the-clear-benchmark/documentation/download-clear-10-clear-100.md)). 

Evaluating on CLEAR is the same as on any other continual learning benchmarks: we measure the performance of a model per timestamp on all the 10 testsets. This produces a *10x10 accuracy matrix*.

Different parts of the accuracy matrix focus on different aspects of the performance of a model. For example, the diagonal entries represent performance on a testset that is sampled from the same distribution as the current bucket (assuming each bucket is a locally iid distribution). Lower triangular part of the matrix instead focus on test performance on previously seen buckets, which is the focus of most state-of-the-art works combatting the forgetting issue. Therefore, we introduce 4 simplified evaluation metrics to summarize the accuracy matrix:

![A visual illustration of the 4 evaluation metrics on a 4x4 accuracy matrix.](https://2411580087-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FiPLWAhemH9JTpCCJxZ3p%2Fuploads%2FLzYZhtHY8Soyu2KTH2UZ%2Fcontent_metrics.png?alt=media\&token=0578c2c8-9211-4a66-93b3-36665a1f93f4)

1. `In-Domain Accuracy:` The average of diagonal entries, i.e., test performance *within the same* *domain* of current bucket.
2. `Next-Domain Accuracy:` The average of super-diagonal entries, i.e., test performance on the *immediate next domain*.
3. `Backward Transfer:` The average of lower triangular entries, i.e., test performance on *previously seen domains*.
4. `Forward Transfer:` The average of upper triangular entries, i.e., test performance on *future unseen domains*.

We hope that our proposed metrics can simplify evaluation on CLEAR and similar domain-incremental benchmarks; nonetheless, CLEAR can also be repurposed for task-/class-incremental scenarios, which could be exciting future works.