Project

General

Profile

Feature #47497

mgr/dashboard: add visual regression testing

Added by Ernesto Puerta about 1 year ago. Updated 6 days ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
UI
Target version:
% Done:

100%

Source:
Tags:
testing
Backport:
pacific
Reviewed:
Affected Versions:
Pull request ID:

Description

Currently cypress framework supports visual regression testing (comparing screenshots of a webpage/subpart under test with a given reference screenshot).

This will help reduce issues with styling/brandind changes and reduce the needs of manual testing & validation.

Downsides of this approach:
  • As already noted by Tiago, is that most pages are data driven, and hence dynamically changing. That makes hard to compare to fixed screenshots.
    • We can either test isolated/static elements of webpages (login page, about box, navigation, forms, ...)
    • Provide a given threshold for the matching algorithm (this would require manual tuning and hence is subject to making the test flapping)
    • Test isolated elements without back-end data: allowing dashboard to instantiate components (blank table, static landing page, ...)
    • Ensure the backend provides 'deterministic data' (feasible?)
    • Hide changing data by tagging those with some class (.dynamic) and making those invisible or deterministic.

Subtasks

Tasks #51124: mgr/dashboard: exploration/research/state of the artResolved

Tasks #51125: mgr/dashboard: identify & start prototyping on an existing e2e test to cover with screenshotsResolvedNizamudeen A

Tasks #51126: mgr/dashboard: final demo (proposal)Resolved

Bug #52282: mgr/dashboard: False positives in visual regression tests of dashboard componentResolved

Documentation #52370: mgr/dashboard: Add Documentation for Visual Regression TestingResolved


Related issues

Copied to Dashboard - Backport #52021: pacific: mgr/dashboard: add visual regression testing Resolved

History

#1 Updated by Ernesto Puerta about 1 year ago

  • Description updated (diff)

#2 Updated by Ernesto Puerta 6 months ago

  • Project changed from mgr to Dashboard
  • Category changed from 165 to UI

#3 Updated by Aaryan Porwal 5 months ago

  • Assignee set to Aaryan Porwal

#4 Updated by Aaryan Porwal 3 months ago

  • Pull request ID set to 42328

#5 Updated by Ernesto Puerta 3 months ago

  • Tracker changed from Tasks to Feature
  • Status changed from New to Pending Backport
  • Backport set to pacific

#6 Updated by Backport Bot 3 months ago

  • Copied to Backport #52021: pacific: mgr/dashboard: add visual regression testing added

#7 Updated by Ernesto Puerta 3 months ago

This follow-up PR is required: https://github.com/ceph/ceph/pull/42605

#8 Updated by Ernesto Puerta 3 months ago

Aarya, this is the feedback from your demo today:
  • Ilya: Number of runs included in the Open Source plan?
  • Josh: How many visual bugs vs. functional ones?

#9 Updated by Aaryan Porwal 2 months ago

Ernesto Puerta wrote:

Aarya, this is the feedback from your demo today:
  • Ilya: Number of runs included in the Open Source plan?
  • Josh: How many visual bugs vs. functional ones?

We're using Applitools open source plan, which currently provides us with 10,000 checkpoint screenshots per month. Everything is enlisted here: https://applitools.com/open-source-terms-of-use/

#10 Updated by Ernesto Puerta about 1 month ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF