Lightly Files

Highlights

LightlySSL now trains latent world models, starting with LeWM (#2032). LeWM predicts the embedding of the next frame from past frames and actions. Try it in the tutorial Train a World Model and Plan with It (#2076). The world model modules are experimental.

New method: CAPI, with examples, docs and an ImageNet benchmark (#1993, [#1994], [#1995], [#1996], [#1998]).

The MAE and Pixio TIMM decoders are deprecated in favor of the new MaskedVisionTransformerDecoder (#1978, [#2006]).

Breaking changes

• The API client is removed. If you still need ApiWorkflowClient, pin lightly<1.6 (#1939). These modules and commands are gone:

  :::text lightly.api, lightly.openapi_generated, lightly.active_learning lightly-download, lightly-serve

• Deprecated collate functions, model wrappers and arguments are removed, and the PR has a migration table (#2021). The removals are:

  :::text lightly.data.collate (the whole module) lightly.data.imagenet_normalize lightly.models.{BarlowTwins, BYOL, MoCo, NNCLR, SimCLR, SimSiam} lightly.loss.SymNegCosineSimilarityLoss MSNLoss(me_max_weight=...) GaussianBlur(kernel_size=..., scale=...) MemoryBankModule(size=<int>) trainer.weights_summary (CLI config)

• Python 3.7 is no longer supported (#2028).

• I-JEPA predictor checkpoints from earlier releases no longer load with strict=True, because the predictor's submodules were renamed (#1978).

Fixes

• LeJEPALoss now includes all views in SIGReg and the invariance term, so its loss values differ from earlier releases (#2023).
• WMSELoss no longer raises RuntimeError on construction (#2045).

What's Changed

• Add MaskedVisionTransformerDecoder by @lorinczszabolcs in https://github.com/lightly-ai/lightly/pull/1978
• Migrate model module tests to pytest style by @I-am-Uchenna in https://github.com/lightly-ai/lightly/pull/1999
• Migrate core tests to pytest style by @Jayashanker-Padishala in https://github.com/lightly-ai/lightly/pull/1990
• Migrate Utils tests from unittest.TestCase to pytest style (#1987) by @adityaanikam in https://github.com/lightly-ai/lightly/pull/1991
• Migrate transforms tests to pytest style by @Jayashanker-Padishala in https://github.com/lightly-ai/lightly/pull/1989
• test(models): migrate model tests from unittest to pytest style by @ealeonraz in https://github.com/lightly-ai/lightly/pull/2001
• Migrate embedding tests to pytest style by @AnayGarodia in https://github.com/lightly-ai/lightly/pull/1988
• ci: run required checks for merge groups by @gabrielfruet in https://github.com/lightly-ai/lightly/pull/2007
• test: add shared gloo process pool for distributed loss tests by @saud5150 in https://github.com/lightly-ai/lightly/pull/1983
• feat: add CAPI loss by @lorinczszabolcs in https://github.com/lightly-ai/lightly/pull/1993
• feat: add CAPI transform by @lorinczszabolcs in https://github.com/lightly-ai/lightly/pull/1994
• fix(benchmarks): use ViT-S/16 for Pixio benchmark by @gabrielfruet in https://github.com/lightly-ai/lightly/pull/2014
• feat(benchmarks): add VISReg to the ImageNet vitb16 benchmark by @saud5150 in https://github.com/lightly-ai/lightly/pull/1997
• Update CLAUDE.md by @imnaur in https://github.com/lightly-ai/lightly/pull/2013
• ci: run distributed pool tests in a dedicated gloo job by @saud5150 in https://github.com/lightly-ai/lightly/pull/2011
• docs: interpret custom augmentation embeddings by @iamdhrv in https://github.com/lightly-ai/lightly/pull/2008
• feat: add CAPI clustering head and cross-attention predictor by @lorinczszabolcs in https://github.com/lightly-ai/lightly/pull/1995
• feat: add CAPI examples and docs by @lorinczszabolcs in https://github.com/lightly-ai/lightly/pull/1996
• Migrate data tests to pytest style by @Jayashanker-Padishala in https://github.com/lightly-ai/lightly/pull/1992
• build: add make target to run distributed tests locally by @saud5150 in https://github.com/lightly-ai/lightly/pull/2017
• Use InterpolationMode.BICUBIC in transforms by @alloutflo in https://github.com/lightly-ai/lightly/pull/2018
• Deprecate mae pixio decoder timm by @lorinczszabolcs in https://github.com/lightly-ai/lightly/pull/2006
• Move IJEPAMaskCollator to its own module, keep the old import working by @gabrielfruet in https://github.com/lightly-ai/lightly/pull/2020
• fix(loss): include all views in LeJEPA SIGReg and invariance loss (#2002) by @harmehak0173 in https://github.com/lightly-ai/lightly/pull/2023
• test: finish the unittest→pytest migration of model-module tests (#1987) by @lorinczszabolcs in https://github.com/lightly-ai/lightly/pull/2027
• fix: examples + benchmarks: weight init in mae+pixio by @lorinczszabolcs in https://github.com/lightly-ai/lightly/pull/2026
• added types for online_linear_classifier.py by @rayyanshuda in https://github.com/lightly-ai/lightly/pull/2024
• Remove API client from LightlySSL by @gabrielfruet in https://github.com/lightly-ai/lightly/pull/1939
• docs: use local colab badge SVG asset to prevent privacy ad-blocker filtering (#1705) by @harmehak0173 in https://github.com/lightly-ai/lightly/pull/2029
• fix(models): handle zero masked patch count in random_inverse_block_mask (#2025) by @harmehak0173 in https://github.com/lightly-ai/lightly/pull/2041
• feat: add CAPI imagenet benchmark by @lorinczszabolcs in https://github.com/lightly-ai/lightly/pull/1998
• fix: MaskedCausalVisionTransformer build on all supported timm versions by @lorinczszabolcs in https://github.com/lightly-ai/lightly/pull/2043
• fix: correct torch.linalg.solve_triangular availability check in WMSELoss by @lorinczszabolcs in https://github.com/lightly-ai/lightly/pull/2045
• Migrate development workflow to uv lockfile by @wmcv in https://github.com/lightly-ai/lightly/pull/2016
• build: drop deprecated numpy mypy plugin by @gabrielfruet in https://github.com/lightly-ai/lightly/pull/2052
• Raise minimum Python version to 3.8 by @gabrielfruet in https://github.com/lightly-ai/lightly/pull/2028
• Fix masked attention with gradient checkpointing by @Purin1410 in https://github.com/lightly-ai/lightly/pull/2050
• Remove deprecated collate functions, model wrappers, and arguments by @gabrielfruet in https://github.com/lightly-ai/lightly/pull/2021
• Add types for crop_image_by_bounding_boxes.py by @RishikeshVepura in https://github.com/lightly-ai/lightly/pull/2060
• feat: add LeWM, an action-conditioned latent world model trained with SIGReg by @gabrielfruet in https://github.com/lightly-ai/lightly/pull/2032
• fix(models): honor is_causal in MaskedCausalVisionTransformer by @harmehak0173 in https://github.com/lightly-ai/lightly/pull/2068
• feat(benchmarks): resume ImageNet runs in plac by @karttikjangid in https://github.com/lightly-ai/lightly/pull/2069
• Add types for knn_classifier.py by @atique-ahmad-01 in https://github.com/lightly-ai/lightly/pull/2067
• fix(models): keep block grads under reentrant checkpointing in causal ViT by @gabrielfruet in https://github.com/lightly-ai/lightly/pull/2087
• fix: match LeWMProjectionHead to the official le-wm MLP by @gabrielfruet in https://github.com/lightly-ai/lightly/pull/2075
• docs: add LeWM world model tutorial by @gabrielfruet in https://github.com/lightly-ai/lightly/pull/2076
• Add ViT gradient checkpointing by @jaredrainsha in https://github.com/lightly-ai/lightly/pull/2058
• Remove stale mypy exclusions by @laclance in https://github.com/lightly-ai/lightly/pull/2070
• Add types for test_multi_view_collate.py by @Bhavyashah20 in https://github.com/lightly-ai/lightly/pull/2079
• docs: make world models discoverable in README and docs by @gabrielfruet in https://github.com/lightly-ai/lightly/pull/2089
• Remove stale test mypy exclusions by @laclance in https://github.com/lightly-ai/lightly/pull/2071
• Fix docstrings that name parameters the classes do not have by @VenishPaneliya in https://github.com/lightly-ai/lightly/pull/2074
• fix: point removed API client message at lightly<1.6 by @gabrielfruet in https://github.com/lightly-ai/lightly/pull/2091
• Pre-release v1.6.0 by @gabrielfruet in https://github.com/lightly-ai/lightly/pull/2090

New Contributors

• @I-am-Uchenna made their first contribution in https://github.com/lightly-ai/lightly/pull/1999
• @Jayashanker-Padishala made their first contribution in https://github.com/lightly-ai/lightly/pull/1990
• @adityaanikam made their first contribution in https://github.com/lightly-ai/lightly/pull/1991
• @ealeonraz made their first contribution in https://github.com/lightly-ai/lightly/pull/2001
• @AnayGarodia made their first contribution in https://github.com/lightly-ai/lightly/pull/1988
• @imnaur made their first contribution in https://github.com/lightly-ai/lightly/pull/2013
• @iamdhrv made their first contribution in https://github.com/lightly-ai/lightly/pull/2008
• @alloutflo made their first contribution in https://github.com/lightly-ai/lightly/pull/2018
• @harmehak0173 made their first contribution in https://github.com/lightly-ai/lightly/pull/2023
• @rayyanshuda made their first contribution in https://github.com/lightly-ai/lightly/pull/2024
• @wmcv made their first contribution in https://github.com/lightly-ai/lightly/pull/2016
• @Purin1410 made their first contribution in https://github.com/lightly-ai/lightly/pull/2050
• @RishikeshVepura made their first contribution in https://github.com/lightly-ai/lightly/pull/2060
• @karttikjangid made their first contribution in https://github.com/lightly-ai/lightly/pull/2069
• @atique-ahmad-01 made their first contribution in https://github.com/lightly-ai/lightly/pull/2067
• @jaredrainsha made their first contribution in https://github.com/lightly-ai/lightly/pull/2058
• @laclance made their first contribution in https://github.com/lightly-ai/lightly/pull/2070
• @Bhavyashah20 made their first contribution in https://github.com/lightly-ai/lightly/pull/2079
• @VenishPaneliya made their first contribution in https://github.com/lightly-ai/lightly/pull/2074

Full Changelog: https://github.com/lightly-ai/lightly/compare/v1.5.26...v1.6.0