Measuring the progress of learned image compression

The Challenge on Learned Image Compression (CLIC) is an annual competition of image and video compression methods which has been ongoing since 2018. Its 7th iteration concluded this week with a workshop held at the Picture Coding Symposium in Aachen, Germany. Each year, teams submit decoders alongside compressed media files, and their ranking is decided based on subjective tests. Mabyduck was used to conduct the evaluation of image compression methods.

Surprisingly, we found no significant improvement in the perceptual quality of compressed images compared to last year. Does this mean that progress in learned image compression has stalled? In this article, we take a closer look.

Each year’s challenge is divided into a validation phase and a test phase. Usefully, the validation set of this year’s image track was the same as the test set of CLIC 2024. This offered us a unique opportunity to evaluate the progress in the field of learned image compression over the past year.

We conducted pairwise comparison experiments to compare all methods of the CLIC 2024 and 2025 image tracks. In total, we collected 46,546 pairwise comparisons of compressed images. We used Elo scores to measure the performance of different methods. (See below for additional information.)

We found that the best method of CLIC 2024 has nearly identical perceptual quality as the best method of CLIC 2025. We also found that this year’s methods were much closer in performance than last year’s, with most methods falling between the second and first placed methods of CLIC 2024.

A potential explanation for these observations is that this year, a new rule was introduced making the 25% slowest decoders ineligible for prizes. This was done to encourage participants to keep the decoder complexity low.

Some support for this hypothesis can be found if we look at decoding times. The best performing decoder of last year’s competition took 4.9 seconds per image at 0.075 bpp, while the best performing decoder of this year’s competition took 0.8 seconds per image. (Both decoders were run on an L4 GPU and otherwise similar but not identical hardware). As another point of comparison, the winning decoder of the first CLIC in 2018 took several minutes to decode a single image. This suggests a trend towards more practical neural compression methods.

Advances in more practical methods also become apparent when we compare the submissions of Cool-Chic from 2024 and 2025. At 8.4 MB, this decoder is significantly smaller than any other submitted neural compression method. Compared to last year, the gap to the best performing compression methods was significantly reduced.

Despite not seeing an improvement in the perceptual quality at CLIC, it would be premature to conclude that no advances have been made in the field of learned compression as a whole. Methods based on diffusion and similar powerful generative image models were noticeably absent from the competition, despite being an active area of research. Given the popularity of these methods in other application domains of generative AI, and the pressure to make them more cost efficient, it should only be a matter of time before we see these methods appear at CLIC as well.

Additional information

Pairwise comparisons were collected across multiple experiments which were later combined to compute a combined ranking. 29,520 comparisons were between conditions included in the CLIC 2025 challenge and using the test set of CLIC 2025 (30 images). The remaining comparisons were collected on the validation set (32 images), which corresponds to the test set of CLIC 2024. In each experiment, raters were able to flip between two compressed images, and were also shown an uncompressed reference image. The total number of conditions across all methods and bit-rates was 61 (20 methods, 3 bit-rates, and 1 reference).