Visual Diffing

Visual Diffing

Lastest offers three comparison engines and extensive configuration for reducing false positives.

---

Diff Engines

pixelmatch (default)

Pixel-perfect comparison. Fast and deterministic. Best for most use cases.

SSIM (Structural Similarity Index)

Measures structural similarity between images. More tolerant of minor rendering differences while catching meaningful layout changes.

Butteraugli

Human-perception-aligned comparison. Models how the human visual system perceives differences. Best for catching changes that users would actually notice.

Configure the default engine in Settings > Diff Sensitivity.

---

Text-Region-Aware Diffing

OCR-based two-pass comparison with separate thresholds for text vs non-text regions:

1. First pass: OCR detects text regions in the screenshot
2. Second pass: Text regions are compared with a more tolerant threshold (cross-OS font rendering varies)
3. Non-text regions use the standard threshold

This dramatically reduces false positives from dynamic text and font rendering differences across operating systems.

Enable in Settings > Diff Sensitivity.

---

Diff Sensitivity Configuration

Setting	Description
Diff engine	pixelmatch, SSIM, or Butteraugli
Text-region-aware	Enable OCR-based two-pass comparison
Pixel threshold	Number of different pixels to trigger a change
Percentage threshold	Percentage of different pixels to trigger a change
Classification	Unchanged / Flaky / Changed thresholds

---

Diff Classifications

Classification	Meaning
Unchanged	No meaningful difference detected
Flaky	Minor differences within the flaky threshold (likely noise)
Changed	Meaningful visual difference that needs review

---

AI Diff Analysis

When configured, AI can automatically classify visual diffs:

• Insignificant -- rendering noise, minor font differences
• Meaningful -- actual UI changes that need review
• With confidence scores and change category labels

Configure a separate AI provider for diff analysis in AI Configuration.

---

Ignore Regions (per-step)

Mask dynamic areas from comparison:

• Draw ignore regions directly on the screenshot
• Regions are now scoped per step (per-screenshot), not per-test -- a region you draw on the "Dashboard" screenshot won't bleed into the "Settings" screenshot of the same test
• Excluded from diff calculation
• Useful for timestamps, ads, counters, avatars, etc.
• Configurable mask styles: solid-color or placeholder-text

Migration note: existing per-test ignore regions are migrated to a step_label of null (apply-to-all). New regions are bound to the screenshot they were drawn on.

---

DOM-Based Diff Fallback

When the pixel diff is inconclusive (e.g. very small differences clustered around inserted/removed nodes), Lastest can run a structural DOM diff in addition to the pixel comparison. The DOM diff is computed from the captured DOM snapshot at screenshot time and surfaces the affected node tree alongside the visual diff. Toggle in Settings > Diff Sensitivity.

---

Planned Screenshots

Compare test results against design files:

1. Upload a design file (Figma export, PNG, etc.)
2. Run the test
3. See a separate planned vs actual diff alongside the baseline diff

---

Page Shift Detection

Detect vertical content shifts (inserted/deleted rows):

• Fuzzy row matching identifies shifted content
• Distinguishes between actual changes and simple insertions/deletions
• Avoids flagging every pixel below an inserted row as "changed"

---

Diff Engine Benchmarks

Built-in benchmark framework to compare all three engines:

• Synthetic test scenarios with known differences
• Timing and accuracy metrics
• Helps you choose the best engine for your use case