copyjob: write a copy to a removable device before reporting success

A copy to a USB stick or a memory card reports success while the copied data is still in the page cache. Unplugging the device then loses it. A move is worse: the sources are deleted right after each copy, so the files are lost in both places.

The file worker already reports a copy to a removable device with the flush-destination metadata, and has a command that flushes such a folder with syncfs(). Since !2517 (merged), FileCopyJob keeps that metadata, so CopyJob receives it.

BUG: 281270

What this does

  1. A flush before success. When a file copy reports flush-destination, CopyJob sends the flush command for its destination folder before it reports success. One syncfs() covers every file of the copy. While it runs, the job shows "Writing the copied data to the device...". An error from the flush is the error of the job, because the data may not be on the device. A copy to any other folder sends no extra command.
  2. A move deletes its sources after the flush. A moved file whose copy or put reports flush-destination keeps its source. After the flush succeeds, CopyJob deletes these sources one by one, then the source folders. When the flush or a delete fails, the remaining sources stay in place and are not reported as removed.
  3. A source edited during the move stays. A source can now wait until the end of the job before its delete. CopyJob stats it first, and keeps a source whose mtime or size changed since its copy, with a warning.
  4. Undo and views follow the delete. A file whose source is kept is reported with copyingDone and FileMoved only once its source is deleted. A file whose source stays is not in the undo record, so undoing the move does not run into it.
  5. Errors and cancel. A copy that stops on an error flushes the device before it reports that error, and a move then deletes the unchanged sources of the files that it moved. A cancelled job cannot wait, so it starts a flush that runs on its own, and deletes those sources once the flush succeeds.
  6. A description while the sources are deleted. The job shows "Deleting" with the source being deleted, so a move from a remote source does not look stuck at 100%.
  7. Which destinations. The flush folder is the destination when it is a local file, which includes a desktop:/ folder through UDS_LOCAL_PATH, or otherwise its local UDS_TARGET_URL. Without a local folder, nothing is kept and nothing is flushed, as before.

Tests

copyFolderFlushesRemovableDestination, with the destination forced to be removable through KIO_FILE_FORCE_REMOVABLE, covers:

  • a copy to a removable and to a fixed destination: only the first one is flushed
  • a move from another partition to a removable and to a fixed destination: on the removable one, the sources are still there during the flush, and gone after it
  • a move whose destination folder disappears before the flush: the job fails with ERR_CANNOT_WRITE and every source stays
  • a move whose source is edited after its copy: that source stays, one warning is emitted, and the undo record holds only the other files

copyStoppedByErrorFlushesRemovableDestination stops a copy and a move on a conflict after a first folder: the device is flushed, the error of the conflict is the error of the job, and the move deleted the sources of the first folder. killedMoveFlushesRemovableDestination kills a move after three files: their sources are deleted once the flush that runs on its own succeeds.

moveToTargetUrlFlushesRemovableDestination moves a file to a test worker that maps its folder to a local one through UDS_TARGET_URL, and reports flush-destination from its put. The local folder is flushed, and the source stays until the flush.

The removable cases are skipped outside Linux, which has no syncfs().

On a USB stick

A USB stick with exfat, recognised as removable without KIO_FILE_FORCE_REMOVABLE. The sources were on the local disk. After each job, a sync -f of the stick measured what was left to write.

done after sync -f after done result
copy of 300 files of 32 KiB 32.5 s 3.7 ms 300 files on the stick
copy of a 64 MiB file 19.6 s 4.1 ms same content
move of 300 files of 32 KiB 17.4 s 2.8 ms 300 files on the stick, no source left
move cancelled 2 s into the flush all files on the stick, all sources deleted
copy-as of an 8 MiB file to a new name 0.54 s 6.6 ms same content
copy into the root of the stick 0.15 s 3.8 ms 20 files on the stick
copy stopped by a conflict after 50 files 0.28 s, error 112 2.7 ms flushed before the error
move stopped by a conflict after 50 files 11.9 s, error 112 2.8 ms flushed, then the 50 sources deleted
move with one source edited during the flush 16.5 s 3.1 ms that source kept with a warning, the 299 others deleted
move of 1000 files of 32 KiB 42.7 s 2.6 ms 1000 files on the stick, no source left
move of 1000 files, stick unplugged during the flush 11.6 s, error 129 all 1000 sources kept
copy to the local disk 0.06 s no flush

The job now reports done when the data is on the stick, so the time to done is the time the stick takes to write it. Before this change, these copies reported done after a fraction of a second.

Known limitations

  • The flush writes back the whole filesystem of the device, so it also waits for data that other programs wrote to it at the same time.
  • Only CopyJob flushes. A KIO::file_copy() or KIO::file_move() alone is not flushed.
  • A source that cannot be deleted after the flush ends the job with an error, without the per-file skip dialog. A failed source delete also ended the job before this change.
  • While the sources are deleted, the progress stays at 100%; only the description changes.
Edited by Méven Car

Merge request reports

Loading
Loading