Project regression testing
At this point, it seems necessary to setup a collection of Kdenlive project files that could be automatically rendered by a script, which would compare the renders with a reference rendering. This would help us prevent regressions, and these tests should pass before a release.
Steps needed:
-
Make it possible to automatically open a project and render it from a command line (#1615) -
Make it possible to automatically open a project and save it under another name from the command line -
Upload various assets, if possible not too large, that can be used in the projects. These should include images, videos (interlaced, progressive, horizontal and vertical orientation, various pixel formats) -
Create project files using those assets in various categories: projects with just clip cuts (no effects, no composition), projects with mixes, projects with effects, projects with multiple timelines, etc. -
Create a script that can render all project files in a folder -
Create a script that can compare the result with a reference render (based on @erjiang's work: https://invent.kde.org/erjiang/kdenlive-backcompat) -
Create a script that automatically renders and tests all projects in a subfolder -
Create a script that opens a project, save it and reopens it to compare the timelines hash (ensuring no data loss).
Related to: #1615
Maybe need to https://invent.kde.org/multimedia/kdenlive/-/wikis/dev/ideas/Project-File-Refactoring
Edited by Jean-Baptiste Mardelle