Untangling Ruby tests
The CI has test failures for ktexteditor:
- 5 on FreeBSD
- 1 on openSUSE w/ Qt 5.12
- 3 on openSUSE w/ Qt 5.15
- 10 on Windows
I wanted to see what was going on, so I ran the tests locally on my FreeBSD machine: only 1 test failure. It was testRuby. I checked, and the same test fails on all three CI platforms in the same way:
--- /usr/home/jenkins/workspace/Frameworks/ktexteditor/kf5-qt5 FreeBSDQt5.15/autotests/input/indent/ruby/regexp1/expected 2020-08-16 01:20:54.464834000 +0200
+++ /usr/home/jenkins/workspace/Frameworks/ktexteditor/kf5-qt5 FreeBSDQt5.15/autotests/input/indent/ruby/regexp1/actual 2020-08-16 01:22:19.009634000 +0200
@@ -1,2 +1,2 @@
rx =~ /^hello/
- foo
+ foo
FAIL! : IndentTest::testRuby(regexp1) Compared values are not the same
Actual (diff.exitCode()): 1
Expected (EXIT_SUCCESS) : 0
Loc: [/usr/home/jenkins/workspace/Frameworks/ktexteditor/kf5-qt5 FreeBSDQt5.15/autotests/src/script_test_base.cpp(153)]
See, e.g. the FreeBSD log and the openSUSE Qt 5.12 log. In both, search for rx =
.
This would suggest the test should be marked XFAIL.
But actually doing the test (open ruby file with that text, go to end of line, hit enter) in a running kate in a Plasma Desktop session, indents the text as expected (i.e. foo
aligns with rx
rather than being indented). So that tells me that the test doesn't accurately reflect what kate / ktexteditor does, and is invalid.
Summing up:
- the test produces consistent results across at least 3 UNIXy platforms
- the test results all match, and all do not match expected test result
- the test results do not match what the application actually does