> For the complete documentation index, see [llms.txt](https://linzhiqiu.gitbook.io/the-clear-benchmark/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://linzhiqiu.gitbook.io/the-clear-benchmark/introduction/motivation-of-clear-benchmark.md).

# Motivation of CLEAR Benchmark

A continual/lifelong learning benchmark capturing natural distribution shifts of Internet imagery over a decade

## Why Continual Learning?

Most of the successes in nowadays vision and learning community are achieved on **static benchmarks** that did not change ever since they are released:

![ImageNet (2010) and COCO (2015) are the modern test stones of visual recognition and detection algorithms. However, they did not model the temporal dynamic aspect of real world.](https://2411580087-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FiPLWAhemH9JTpCCJxZ3p%2Fuploads%2FEWLUVQ2oN1HHIE4Lur4G%2Fimagenet_coco.png?alt=media\&token=9e0bbe81-338d-4fe1-b7a5-46c2926b3382)

In real world, such "IID" assumption does not usually hold. Therefore, researchers have put efforts in the field of **continual (or incremental/lifelong) learning**, aiming for learning systems that are **more robust under distribution shifts**. 

Yet, most of the existing works focus on combatting the **catastrophic forgetting** nature of neural networks, a phenomenon commonly observed on popular continual benchmarks with **extreme distribution shifts between tasks** such as "*Permuted-MNIST*", "*Split-CIFAR*", "*Incremental-ImageNet*", and so on..

![Popular continual learning benchmarks that do not align with practical applications.](https://2411580087-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FiPLWAhemH9JTpCCJxZ3p%2Fuploads%2FUuj3N4lodv13cuuGvNqD%2Fmnist_cifar.png?alt=media\&token=e22269d2-7740-48b9-9007-b67162312a24)

Made from existing vision datasets, these benchmarks usually contain **synthetic distribution shifts** via randomly shuffling pixels, or splitting labels into disjoint subsets. Instead, we posit that **a more practical continual learning benchmark should reflect how the real world is changing**, such as when AVs moving to a new city, and when seeing brand new car models:

![Examples of real world distribution shifts for AVs.](https://2411580087-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FiPLWAhemH9JTpCCJxZ3p%2Fuploads%2Fccwr2uwDlue5joYBw8SW%2Freal_world.png?alt=media\&token=ccb4a5b0-7903-4c12-beed-0b4173fc3832)

## Temporal Evolution of Visual Concepts

In the context of visual recognition, we observe that a lot of visual concepts in Internet imagery are evolving over time, i.e., temporal evolution of visual concepts.

![The visual concept of "computer" naturally evolved from 2004 to 2014 as laptops became more popular than bulky desktops.](https://2411580087-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FiPLWAhemH9JTpCCJxZ3p%2Fuploads%2Fg4rYhP1hkTus39gaT4m1%2Fevolve.png?alt=media\&token=65f2bfe1-a1dd-409d-bb51-a8a91b84b83e)

Therefore, we propose to make the CLEAR benchmark featuring such natural continual learning scenarios. We select dynamic visual concepts that are common in Internet image collections to form the label space of [CLEAR-10](/the-clear-benchmark/documentation/download-clear-10-clear-100.md#clear-10-s3-download-links) and [CLEAR-100](/the-clear-benchmark/documentation/download-clear-10-clear-100.md#clear-100-s3-download-links).

![Label space of CLEAR-10 and CLEAR-100.](https://2411580087-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FiPLWAhemH9JTpCCJxZ3p%2Fuploads%2FSZc8VGVuMgO57xuAch39%2Fvisual_concepts.png?alt=media\&token=dfe54d1c-f93a-4b56-9864-170095c15833)

We will discuss next how we curate the CLEAR benchmark with an efficient visio-linguisitic dataset curation approach, as well as some of the valuable assets made available for the vision\&learning community.