Potentially blocking codepaths in KIO/Dolphin
By Kai from: https://github.com/blue-systems/plasma-issues/issues/201
Over the past months every once in a while I've been poking KIO and Dolphin to see where it freezes when accessing slow or broken mounts. Most of the APIs are designed with the assumption that local file means fast, which is usually fine if you use KIO but falls apart once you have network shares mounted locally through NFS and CIFS.
This is a list of places I have found that potentially block with slow and unresponsive mounts.
There's basically four ways to deal with those issues:
1. Rewrite them to use asynchronous KIO APIs, even for local files. Sometimes a class already uses async but has a "fast path" or some prior checks that could be dropped or rearranged. 2. When it is an obscure feature nobody needs, remove it 3. At least delay doing the work and/or don't do it for the common case 4. It's a very difficult to fix design decision which we cannot really solve in the current architecture/KF5 cycle
I used sshfs (which you can easily break by turning off the network) and slowfs which easily lets you set up a slow file system for testing and basically tested four scenarios:
1. Entering a folder which has a slow mount inside (e.g. entering /home/foo with /home/foo/Stuff being a slow network share) 2. Entering a slow folder (e.g. entering /home/foo/Stuff which is a slow network share) 3. Clicking a file in a slow folder to open it (e.g. clicking /home/foo/Stuff/somefile.txt) 4. Right-clicking a file in a slow folder to reveal its context menu
Here's all that I found! :)
KIO Core
KCoreDirLister
* `KDirListerCache::listDir` will use `QFileInfo` to resolve symlinks which blocks when e.g. using back button in Dolphin, see [Bug 213799](https://bugs.kde.org/show_bug.cgi?id=213799) * `KCoreDirListerCache::forgetDirs` checks whether a folder was "mounted manually" to prevent a dir watch on its contents preventing unmounting the device. (I recall the `KDirWatch` itself also blocking doing a stat to find out whether the destination is a folder but I couldn't reproduce just now). There is a `TODO` comment in the code suggesting to port that to Solid but I don't think that will help us, if anything, will take forever to load a tonne of backends and make blocking DBus calls to UDisks instead. * `KCoreDirListerCache::slotEntries` reads a `.hidden` file inside the folder with a list of files to hide in the folder. This could be alleviated by waiting for the ".hidden" file to be listed and only then reading it. This could be a nice optimization in general as it won't try reading a `.hidden` file which usually isn't present.
KFileItem
* `KFileItem::mimeComment` (shown as file type in Dolphin's sidebar, read on hovering, which means easy oppoertunity to freeze the UI) reads `Comment` of a `.desktop` file to show instead of the actual file type. Not sure how useful that is. The code has a comment about "This cannot move to kio_file (with UDS_DISPLAY_TYPE) because it needs the mimetype to be determined, which is done here, and possibly delayed...". * `KFileItem::mimeComment` reads the `Comment` of the `.directory` file _inside_ the folder to show that as mime type instead. There's no UI to configure a folder comment and the only (questionable) usecase I could see for this is that e.g. a Pictures folder shows "Pictures folder" instead of "Folder", like Windows 98 did it. I tried [moving this to the KIO slave](https://phabricator.kde.org/D13757) but there's valid performance concerns about unconditionally reading it for every folder encountered. * `KFileItem::isReadable` (used in turn by `KFileItem::emblems` and `checkDesktopFile`) uses a `QFileInfo` as a last resort for figuring out whether something is readable. Likely so it catches inherited permissions and ACLs which the KIO slave itself might not know? * `KFileItem::isWritable` likewise.
Overall I find that concept of "delayed mime determination" somewhat questionable. We have an out of process aysnchronous KIO slave listing the directory, sending us information, and then we end up doing potentially expensive mime type determination and file icon resolving (which might involve reading bits of the file or traversing a subdirectory) in a blocking way on the client side.
Worse, the check for whether something is a "slow mount" (where we might skip some of the expensive stuff) might also block as it will
statvfs
the mount to figure out its file system type.KFileItemListProperties
* `KFileItemListPropertiesPrivate::setItems` checks whether the `KFileItem` `isWritable` (see above) and as such blocks. It is used for updating the "Paste" action and whether to enable stuff like "Create New" menu, i.e. it is called every time you navigate folders. * `KFileItemListPropertiesPrivate::setItems` also checks the parent directory whether it is writable to determine whether deleting an item is allowed. There is a `TODO` comment in the code "if we knew about the parent KFileItem, we could even do that for remote protocols too"
KFileWidget (file dialog)
* `KFileWidget::slotOk` does a synchronous `StatJob` on the selected URL when acecpting the dialog. The code has a comment "Called by KFileDialog", it is likely this must be synchronous so the dialog doesn't close/accept as long as the stat isn't done. * `KFileWidgetPrivate::setLocationText` (both overloads) calls `KIO::iconNameForUrl` to update the icon next to the address bar. This means it blocks every time you select a file
Generally the file dialog performs a lot better with slow mounts than Dolphin. It does block but not nearly as bad. Granted, it also doesn't have that many features :)
KFilePlacesView
KFilePlacesViewDelegate::paint
sidebar uses synchronousKDiskFreeSpaceInfo::freeSapceInfo
for painting the capacity bar. I optimized it to only do that when actually asked for (when hovering a folder) but the proper solution would be to use an asynchronous job.KPropertiesDialog
KFilePropsPlugin::KFilePropsPlugin
does mime type content determination to show both "File type" and "Contents", e.g. "File type" could be TAR but "Content" could be some app bundle. This could probably be changed to use a MimeTypeJob, not sure if that could be forced to only do content determination and not be "misguidable" by the file extension. This causes the file properties dialog to block when opening it for a slow file.KRun
* `KRun::KRunPrivate::isPromptNeeded` does `QMimeDatabase::mimeTypeForUrl` to figure out whether something is an executable and the "run this?" prompt should be shown. This casues a freeze when trying to open a file on a slow mount. `KRun` already does mime type determination later, so potentially this could be shuffled to do the check once the mime type has been determined. * `KRUn::init` has a "fast path" for local files which uses `QFileInfo::isReadable` to check whether the file being opened is readable. If all of this was using the mime job, that job would return us an appropriate error without the need for such an explicit check.
KMountPoint
KMountPoint::List::findByPath
usesQFileInfo
to resolve symlinks which blocks. So the check for checking whether something is a slow mount might also block :)PreviewJob
PreviewJobPrivate::startPreview
usesKFileItem::mimetype()
which blocks resolving mimetype.Basically preview job does all the heavy stuff in-process and uses the external
thumbnail:/
KIO slave merely to generate the acutal thumbnail. All the caching, mime type checking, plugin checking, is done in-process.Other
When the file slave blocks in
lstat
on a slow mount, no listings are processed until it gets unstuck. This means if you have a folder with one slow share inside (e.g. /home/foo with a slow /home/foo/Stuff inside), you won't get any results until that one slow folder has been processed. It only sends listed files every 300ms or 200 files, whichever comes first, but if it processed 10 files and then gets stuck on one file, you won't get any results until that 11th file is processed.Dolphin
DolphinTabWidget
DolphinTabWidget
usesKIO::iconNameForUrl
to set the appropriate tab icon potentially blocking. It does so, even if the tab bar isn't shown. An easy alleviation would be to not update the tab icon if the tab bar isn't visible. This won't get rid of the blocking but would fix it for the "common case". Ideally, the tab bar had access to the view's rootKFileItem
so it could use the icon from there instead of having to look it up manually every time.ViewProperties
ViewProperties
checks whether to write the view settings in the.directory
file inside the folder or to a a location in Dolphin's config folder. The check does aQFileInfo
on the directory and potential.directory
file inside, blocking a few times in a row. It then usesKSharedConfig
to read that.directory
file when it determined where to write it to.Other
It also suffers from freezes caused by "delayed mime determination",in
KFileItemModelRolesUpdater::slotGotPreview
andKFileItemModelRolesUpdater::applyResolvedRoles
e.g. it callsdetermineMimeType()
using some clever "only block for so long" and "prefer for the visible range" heuristic but that can obviously still freeze.There were some improvements to
KFileItem
recently so it can be used without causing stat calls, e.g. there now is aKFileItem::SkipMimeTypeFromContent
, which we could make use of more in Dolphin. Perhaps we could even go so far as to prefer file extension and only do a content lookup if none is found/invalid. I think that is how it does it already but it still always looks at the contents.For local folders Dolphin only shows folder contents once
KDirLister
has emittedlistingCompleted
. This is usually fine as it prevents it from re-sorting constantly. However, for slow locally mounted shares this can reduce the perceived performance a lot as no files are shown until it is completely done listing. There is a timer for updating the view every 2 seconds (mostly meant for thesearch
KIO slave) which can probably just be enabled for local folders as well.Other
Ark's
ExtractFileItemAction
uses a very elaborate mime type determination which blocks when right-clicking on a file. The code explains it as "detection-by-content does not work for compressed tar archives", see Bug 328815 and "Compressed tar-archives are detected as single compressed files when detecting by content"