Skip to content

Change Klipper's save format on disk

Oliver Hiorns requested to merge oshiorns/plasma-workspace:work/klipper-new-history-format into master

Edit: for info about testing, please see !3749 (comment 850434).

This MR aims to make major changes to how Klipper stores clipboard items on disk. The motivation is to make Klipper faster at saving the clipboard history data. When a lot of images are in the clipboard, this can eliminate delays of 10s+.

Old file format:

klipper/                [directory]
    history2.lst        [file]
        crc(data)
        data
            history_item_type, history_item_data,
            history_item_type, history_item_data,
            history_item_type, history_item_data,
            ...

New file format:

klipper/                [directory]
    history3.lst        [file]
        crc(data)
        data
            uuid1
            uuid2
            uuid3
            ...
    storage/            [directory]
        uuid1           [file]
            history_item_type, history_item_data
        uuid2           [file]
            history_item_type, history_item_data
        uuid3           [file]
            history_item_type, history_item_data
        ...

With the old format, any change to the clipboard history requires rewriting the entirety of history2.lst. This requires processing all data currently in the clipboard history and saving it to disk (potentially 10s of MBs). This also means we need to re-encode things in an appropriate format for saving. For images, this means re-encoding bitmaps as PNGs (slow).

With the new format, we do the minimum amount of work. For re-ordering, we only have to rewrite the UUID file (history3.lst) which is very small (a few 10s of KB) and fast since the UUIDs have already been computed and cached. For insertion/deletion/editing, we create/delete/rewrite one item in the storage/ directory - O(1) - and rewrite history3.lst (for insertion and deletion). In the old system, we would have to effectively rewrite every item - O(n).

As a bonus, this also makes it much easier to load only the data necessary. In future, this will be exploited to save on memory: by loading only a thumbnail at startup, images can stay on disk until they are actually requested for clipboard focus. If many images are in the clipboard, this can save hundreds of MB compared to storing all images in RAM as bitmaps.

This is the second stage of the larger MR !2502. The first stage (!3697 (merged)) has already gone through. The third stage will be lazy loading of image data to save RAM.

Edited by Oliver Hiorns

Merge request reports