Commit 4eec6b8d authored by Harald Sitter's avatar Harald Sitter

enable sysrq + add special sysrq debugging when stuck on plymouth

this is special error handling.

we have persistent problems that reboots get stuck. previous debugging
attempts via journald didn't give useful results because according to the
journal the system is shutting down as expected. It's clearly stuck on
something though.

so... to debug this we'll try to get as much out of the kernel as possible
and then crash the system for good measure.

additionally we'll now enable sysrq all the time since ubuntu's default
security would disable most of the debugging commands. since we have no
tests relying on the security being present we may as well just disable it
parent 9fb252b0
......@@ -24,3 +24,9 @@ puts "#{$0} Letting systemd-journald log to ttyS1."
system 'sed -i "s%.*ForwardToConsole=.*%ForwardToConsole=yes%g" /etc/systemd/journald.conf' || raise
system 'sed -i "s%.*TTYPath=.*%TTYPath=/dev/ttyS1%g" /etc/systemd/journald.conf' || raise
system 'systemctl restart systemd-journald' || raise
# Ubuntu by default is hardened. To ease debugging we'll want full sysrq access.
# This is done in here since this is the only helpe rurn by both regular tests
# and live tests.
puts "#{$0} Enabling sysrq."
File.write('/proc/sys/kernel/sysrq', '1')
......@@ -34,6 +34,39 @@ sub new {
}
sub post_fail_hook {
# Special handler for plymouth.
# We continue to have problems with systems rebooting. According to the
# journal systemd reached the shutdown state but the screen is stuck on
# plymouth. There's a range of options of what is going wrong ranging from
# kvm/qemu bug to systemd bug to openqa bug. Without additional data its
# hard to say what's going on. To get that data we'll attempt to get data
# out of the kernel. Should that turn out unresponsive as well we at least
# know it's not a problem in the guest.
# This ends in a system crash. We'll not be able to retrieve anything useful
# anyway as the system is in a garbage state if plymouth gets stuck.
if (check_screen('plymouth', 4)) {
send_key 'sysrq-r'; # raw kbd mode
send_key 'sysrq-9'; # log level
send_key 'sysrq-k'; # kill programs on VT (plymouth I hope)
sleep 1;
save_screenshot;
send_key 'sysrq-l'; # backtrace cpus
sleep 1;
save_screenshot;
send_key 'sysrq-m'; # memory info
sleep 1;
save_screenshot;
send_key 'sysrq-t'; # all task info
sleep 1;
save_screenshot;
send_key 'sysrq-w'; # list blocked tasks
sleep 1;
save_screenshot;
send_key 'sysrq-c'; # crash the system
sleep 1;
save_screenshot;
}
if (check_screen('drkonqi-notification', 4)) {
assert_and_click('drkonqi-notification');
record_soft_failure 'not implemented drkonqi opening';
......
{
"area": [
{
"height": 265,
"match": 95,
"type": "match",
"width": 1024,
"xpos": 0,
"ypos": 0
},
{
"height": 152,
"match": 95,
"type": "match",
"width": 173,
"xpos": 418,
"ypos": 305
},
{
"height": 30,
"match": 95,
"type": "match",
"width": 117,
"xpos": 410,
"ypos": 461
},
{
"height": 768,
"match": 95,
"type": "match",
"width": 307,
"xpos": 717,
"ypos": 0
},
{
"height": 768,
"match": 95,
"type": "match",
"width": 272,
"xpos": 0,
"ypos": 0
}
],
"properties": [],
"tags": [
"plymouth"
]
}
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment