Skip to content

Fix webengine related issues when running in headless mode to fetch html content on unix like OS

Ralf Habacker requested to merge habacker/alkimia:fix-docker-webengine into master

depends on !35 (merged), !36 (merged), !37 (merged), !39 (merged)

The problem

In alkimia, QWebEngine is used in headless mode to retrieve online offers from websites that are generated with Javascript and are therefore not available with a simple retrieval of the static HTML page.

Due to the current structure of alkimia, this query takes place in a synchronous call, which leads to problems when using WebEngine that do not occur with WebKit.

Recognized issues

  • freeze by callback not called when using toHtml() at least in docker environments -> tbaumgart recognized this issue also on a linux desktop

  • QDEBUG : AlkDownloadEngineTest::testDownloadJavaScriptEngineFinished() Error creating openGL context

  • Warning: "Release of profile requested but WebEnginePage still not deleted. Expect troubles !"

  • Crash on destruction

#0  0x00007f1f95978278 in QWebEnginePagePrivate::~QWebEnginePagePrivate() () from /usr/lib64/libQt5WebEngineWidgets.so.5
#1  0x00007f1f95978479 in QWebEnginePagePrivate::~QWebEnginePagePrivate() () from /usr/lib64/libQt5WebEngineWidgets.so.5
#2  0x00007f1f9596ed6a in QWebEnginePagePrivate::releaseProfile() () from /usr/lib64/libQt5WebEngineWidgets.so.5
#3  0x00007f1f8cdf86ed in QtWebEngineCore::ProfileAdapter::~ProfileAdapter() () from /usr/lib64/libQt5WebEngineCore.so.5
#4  0x00007f1f8cdf8b41 in QtWebEngineCore::ProfileAdapter::~ProfileAdapter() () from /usr/lib64/libQt5WebEngineCore.so.5
#5  0x00007f1f8cd8aae1 in QtWebEngineCore::BrowserMainPartsQt::PostMainMessageLoopRun() () from /usr/lib64/libQt5WebEngineCore.so.5
#6  0x00007f1f914bda71 in content::BrowserMainLoop::ShutdownThreadsAndCleanUp() () from /usr/lib64/libQt5WebEngineCore.so.5
#7  0x00007f1f914c09fd in content::BrowserMainRunnerImpl::Shutdown() () from /usr/lib64/libQt5WebEngineCore.so.5
#8  0x00007f1f914c0b65 in content::BrowserMainRunnerImpl::~BrowserMainRunnerImpl() () from /usr/lib64/libQt5WebEngineCore.so.5
  • crash with QT_LOGGING_RULES="*=true" QTWEBENGINE_DISABLE_SANDBOX=1 QT_WEBENGINE_DISABLE_GPU=1
Thread 23 "QNetworkAccessM" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fd513fff700 (LWP 24757)]
0x00007fd53dbfc539 in QApplicationPrivate::notify_helper(QObject*, QEvent*) () from /usr/lib64/libQt5Widgets.so.5
(gdb) bt
#0  0x00007fd53dbfc539 in QApplicationPrivate::notify_helper(QObject*, QEvent*) () from /usr/lib64/libQt5Widgets.so.5
#1  0x00007fd53dc032ff in QApplication::notify(QObject*, QEvent*) () from /usr/lib64/libQt5Widgets.so.5
#2  0x00007fd53d045013 in QCoreApplication::notifyInternal2(QObject*, QEvent*) () from /usr/lib64/libQt5Core.so.5
#3  0x00007fd53d07febc in QObjectPrivate::setParent_helper(QObject*) () from /usr/lib64/libQt5Core.so.5
#4  0x00007fd53bfa488c in ?? () from /usr/lib64/libQt5Network.so.5
#5  0x00007fd53c045525 in ?? () from /usr/lib64/libQt5Network.so.5
#6  0x00007fd53d0794ab in QObject::event(QEvent*) () from /usr/lib64/libQt5Core.so.5
#7  0x00007fd53dbfc53c in QApplicationPrivate::notify_helper(QObject*, QEvent*) () from /usr/lib64/libQt5Widgets.so.5
#8  0x00007fd53dc032ff in QApplication::notify(QObject*, QEvent*) () from /usr/lib64/libQt5Widgets.so.5
#9  0x00007fd53d045013 in QCoreApplication::notifyInternal2(QObject*, QEvent*) () from /usr/lib64/libQt5Core.so.5
#10 0x00007fd53d047a31 in QCoreApplicationPrivate::sendPostedEvents(QObject*, int, QThreadData*) () from /usr/lib64/libQt5Core.so.5
#11 0x00007fd53d0a82b3 in ?? () from /usr/lib64/libQt5Core.so.5
#12 0x00007fd53adf282b in g_main_context_dispatch () from /usr/lib64/libglib-2.0.so.0
#13 0x00007fd53adf2bd0 in ?? () from /usr/lib64/libglib-2.0.so.0
#14 0x00007fd53adf2c5c in g_main_context_iteration () from /usr/lib64/libglib-2.0.so.0
#15 0x00007fd53d0a795c in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib64/libQt5Core.so.5
#16 0x00007fd53d04387a in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib64/libQt5Core.so.5
#17 0x00007fd53ce4a497 in QThread::exec() () from /usr/lib64/libQt5Core.so.5
#18 0x00007fd53ce4b9fc in ?? () from /usr/lib64/libQt5Core.so.5
#19 0x00007fd53c8f46ea in start_thread () from /lib64/libpthread.so.0
#20 0x00007fd53ca2650f in clone () from /lib64/libc.so.6
  • crash on destruction of alkonlinequotes-webengine-test
7: 0x00007f786509edc8 in vtable for mojo::core::WatcherDispatcher () from /usr/lib64/libQt5WebEngineCore.so.5
  • inotify issue
[6520:6520:0328/225236.857054:ERROR:proxy_config_service_linux.cc(607)] inotify_init failed: Zu viele offene Dateien (24)
[6520:6520:0328/225236.919947:ERROR:proxy_config_service_linux.cc(607)] inotify_init failed: Zu viele offene Dateien (24)

Knowledge base

  • QWebEngine: Some Methods Now Return Their Result Asynchronously [1]
  • Commit where QWebEnginePage::toHtml() has been added [2]
  • Thread related
>~"  Id   Target Id                                          Frame \n"
>~"* 1    Thread 0x7fffe4ad0280 (LWP 8940) \"alkwebpage-webe\" AlkWebPageTest::testToHtml (this=0x7fffffffd910) at /home/ralf/src/alkimia-master/autotests/alkwebpagetest.cpp:108\n"
>~"  2    Thread 0x7fffd59ff700 (LWP 8943) \"QDBusConnection\" 0x00007fffed1bfa89 in poll () from /lib64/libc.so.6\n"
>~"  3    Thread 0x7fffd51fe700 (LWP 8944) \"QXcbEventQueue\"  0x00007fffed1bfa89 in poll () from /lib64/libc.so.6\n"

After creating an QWebEnginePage instance 14 additional threads are visible

>~"  4    Thread 0x7fffccf30700 (LWP 8980) \"sandbox_ipc_thr\" 0x00007fffed1bfa89 in poll () from /lib64/libc.so.6\n"
>~"  5    Thread 0x7fffc55f9700 (LWP 8986) \"alkwebpage-webe\" 0x00007fffed195c1f in wait4 () from /lib64/libc.so.6\n"
>~"  6    Thread 0x7fffc4df8700 (LWP 8987) \"ThreadPoolServi\" 0x00007fffed1cc97f in epoll_wait () from /lib64/libc.so.6\n"
>~"  7    Thread 0x7fffb7fff700 (LWP 8988) \"ThreadPoolForeg\" 0x00007fffed0a1a5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0\n"
>~"  8    Thread 0x7fffb77fe700 (LWP 8989) \"Chrome_IOThread\" 0x00007fffed1cc97f in epoll_wait () from /lib64/libc.so.6\n"
>~"  9    Thread 0x7fffb6ffd700 (LWP 8990) \"ThreadPoolForeg\" 0x00007fffed0a1a5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0\n"
>~"  10   Thread 0x7fffb67fc700 (LWP 8991) \"ThreadPoolForeg\" 0x00007fffed0a1a5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0\n"
>~"  11   Thread 0x7fffb5ffb700 (LWP 8992) \"inotify_reader\"  0x00007fffed1c2137 in select () from /lib64/libc.so.6\n"
>~"  12   Thread 0x7fffb57fa700 (LWP 8993) \"ThreadPoolForeg\" 0x00007fffed0a1a5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0\n"
>~"  13   Thread 0x7fffb4ff9700 (LWP 8994) \"ThreadPoolForeg\" 0x00007fffed0a1a5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0\n"
>~"  14   Thread 0x7fff97fff700 (LWP 8995) \"CompositorTileW\" 0x00007fffed0a170c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0\n"
>~"  15   Thread 0x7fff977fe700 (LWP 8996) \"Chrome_InProcGp\" 0x00007fffed1baf53 in open64 () from /lib64/libc.so.6\n"
>~"  16   Thread 0x7fff96ffd700 (LWP 8997) \"Chrome_ChildIOT\" 0x00007fffed1cc97f in epoll_wait () from /lib64/libc.so.6\n"
>~"  17   Thread 0x7fff967fc700 (LWP 8998) \"VideoCaptureThr\" 0x00007fffed0a170c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0\n"

or like this

(gdb) info threads
  Id   Target Id                                           Frame
* 1    Thread 0x7fd52fcaecc0 (LWP 24718) "alkdownloadengi" 0x00007fd53ca19a09 in poll () from /lib64/libc.so.6
  2    Thread 0x7fd522573700 (LWP 24721) "Qt bearer threa" 0x00007fd53ca19a09 in poll () from /lib64/libc.so.6
  3    Thread 0x7fd521d72700 (LWP 24722) "QDBusConnection" 0x00007fd53ca19a09 in poll () from /lib64/libc.so.6
  5    Thread 0x7fd520d70700 (LWP 24724) "QLibProxyWrappe" 0x00007fd53ca1f7c9 in syscall () from /lib64/libc.so.6
  6    Thread 0x7fd513fff700 (LWP 24725) "Thread (pooled)" 0x00007fd53c8fba5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  7    Thread 0x7fd521571700 (LWP 24726) "sandbox_ipc_thr" 0x00007fd53ca19a09 in poll () from /lib64/libc.so.6
  8    Thread 0x7fd5137fe700 (LWP 24729) "ThreadPoolServi" 0x00007fd53ca268ff in epoll_wait () from /lib64/libc.so.6
  9    Thread 0x7fd512ffd700 (LWP 24730) "ThreadPoolForeg" 0x00007fd53c8fba5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  10   Thread 0x7fd5127fc700 (LWP 24731) "Chrome_IOThread" 0x00007fd53ca268ff in epoll_wait () from /lib64/libc.so.6
  11   Thread 0x7fd511ffb700 (LWP 24732) "ThreadPoolForeg" 0x00007fd53c8fba5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  12   Thread 0x7fd5117fa700 (LWP 24733) "ThreadPoolForeg" 0x00007fd53c8fba5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  13   Thread 0x7fd510ff9700 (LWP 24734) "ThreadPoolForeg" 0x00007fd53c8fba5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  14   Thread 0x7fd4f3fff700 (LWP 24735) "inotify_reader"  0x00007fd53ca1c0b7 in select () from /lib64/libc.so.6
  15   Thread 0x7fd4f37fe700 (LWP 24736) "ThreadPoolForeg" 0x00007fd53c8fba5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  16   Thread 0x7fd4f2ffd700 (LWP 24737) "CompositorTileW" 0x00007fd53c8fb70c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  17   Thread 0x7fd4f27fc700 (LWP 24738) "VizCompositorTh" 0x00007fd53c8fb70c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  18   Thread 0x7fd4f1ffb700 (LWP 24739) "VideoCaptureThr" 0x00007fd53c8fb70c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  19   Thread 0x7fd4f17fa700 (LWP 24740) "NetworkService"  0x00007fd53ca268ff in epoll_wait () from /lib64/libc.so.6
  20   Thread 0x7fd4f0ff9700 (LWP 24741) "ThreadPoolSingl" 0x00007fd53c8fb70c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  21   Thread 0x7fd4d3fff700 (LWP 24748) "MemoryInfra"     0x00007fd53c8fb70c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  22   Thread 0x7fd4d37fe700 (LWP 24749) "ThreadPoolSingl" 0x00007fd53c8fb70c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

Ideas on how to solve the problem:

  • Possibly using Chromium or the environment variables provided by qtwebengine may resolve the issues
  • Run qtwebengine instance in another process
  • AI-generated solution, researched by tbaumgart - This has reduced the number of crashes I see, but has not yet solved the problem (because the data is read out before the lambda function is called).
#include <QWebEnginePage>
#include <QDebug>

class MyWebEnginePage : public QWebEnginePage {
   Q_OBJECT
public:
   MyWebEnginePage(QObject *parent = nullptr) : QWebEnginePage(parent) {
       // Constructor code here
   }

   void getHtml() {
       // Call toHtml with a member function callback
       connect(this, &MyWebEnginePage::htmlReceived, this, &MyWebEnginePage::handleHtmlContent);
       toHtml([this](const QString &html) {
           emit htmlReceived(html);
       });
   }

signals:
   void htmlReceived(const QString &html);

private:
   void handleHtmlContent(const QString &html) {
       // Process the HTML content here
       qDebug() << html;
   }
};

// Usage
// MyWebEnginePage *page = new MyWebEnginePage(parent);
// page->getHtml();

What has been already done

  • application freeze prevented by adding timeout support
Edited by Ralf Habacker

Merge request reports