copyjob: write a copy to a removable device before reporting success
A copy to a USB stick or a memory card reports success while the copied data is still in the page cache. Unplugging the device then loses it. A move is worse: the sources are deleted right after each copy, so the files are lost in both places.
The file worker already reports a copy to a removable device with the flush-destination
metadata, and has a command that flushes such a folder with syncfs(). Since !2517 (merged), FileCopyJob
keeps that metadata, so CopyJob receives it.
What this does
- A flush before success. When a file copy reports
flush-destination,CopyJobsends the flush command for its destination folder before it reports success. Onesyncfs()covers every file of the copy. While it runs, the job shows "Writing the copied data to the device...". An error from the flush is the error of the job, because the data may not be on the device. A copy to any other folder sends no extra command. - A move deletes its sources after the flush. A moved file whose copy or put reports
flush-destinationkeeps its source. After the flush succeeds,CopyJobdeletes these sources one by one, then the source folders. When the flush or a delete fails, the remaining sources stay in place and are not reported as removed. - A source edited during the move stays. A source can now wait until the end of the job before
its delete.
CopyJobstats it first, and keeps a source whose mtime or size changed since its copy, with a warning. - Undo and views follow the delete. A file whose source is kept is reported with
copyingDoneandFileMovedonly once its source is deleted. A file whose source stays is not in the undo record, so undoing the move does not run into it. - Errors and cancel. A copy that stops on an error flushes the device before it reports that error, and a move then deletes the unchanged sources of the files that it moved. A cancelled job cannot wait, so it starts a flush that runs on its own, and deletes those sources once the flush succeeds.
- A description while the sources are deleted. The job shows "Deleting" with the source being deleted, so a move from a remote source does not look stuck at 100%.
- Which destinations. The flush folder is the destination when it is a local file, which
includes a
desktop:/folder throughUDS_LOCAL_PATH, or otherwise its localUDS_TARGET_URL. Without a local folder, nothing is kept and nothing is flushed, as before.
Tests
copyFolderFlushesRemovableDestination, with the destination forced to be removable through
KIO_FILE_FORCE_REMOVABLE, covers:
- a copy to a removable and to a fixed destination: only the first one is flushed
- a move from another partition to a removable and to a fixed destination: on the removable one, the sources are still there during the flush, and gone after it
- a move whose destination folder disappears before the flush: the job fails with
ERR_CANNOT_WRITEand every source stays - a move whose source is edited after its copy: that source stays, one warning is emitted, and the undo record holds only the other files
copyStoppedByErrorFlushesRemovableDestination stops a copy and a move on a conflict after a first
folder: the device is flushed, the error of the conflict is the error of the job, and the move
deleted the sources of the first folder. killedMoveFlushesRemovableDestination kills a move after
three files: their sources are deleted once the flush that runs on its own succeeds.
moveToTargetUrlFlushesRemovableDestination moves a file to a test worker that maps its folder
to a local one through UDS_TARGET_URL, and reports flush-destination from its put. The local
folder is flushed, and the source stays until the flush.
The removable cases are skipped outside Linux, which has no syncfs().
On a USB stick
A USB stick with exfat, recognised as removable without KIO_FILE_FORCE_REMOVABLE. The sources
were on the local disk. After each job, a sync -f of the stick measured what was left to write.
| done after | sync -f after done |
result | |
|---|---|---|---|
| copy of 300 files of 32 KiB | 32.5 s | 3.7 ms | 300 files on the stick |
| copy of a 64 MiB file | 19.6 s | 4.1 ms | same content |
| move of 300 files of 32 KiB | 17.4 s | 2.8 ms | 300 files on the stick, no source left |
| move cancelled 2 s into the flush | all files on the stick, all sources deleted | ||
| copy-as of an 8 MiB file to a new name | 0.54 s | 6.6 ms | same content |
| copy into the root of the stick | 0.15 s | 3.8 ms | 20 files on the stick |
| copy stopped by a conflict after 50 files | 0.28 s, error 112 | 2.7 ms | flushed before the error |
| move stopped by a conflict after 50 files | 11.9 s, error 112 | 2.8 ms | flushed, then the 50 sources deleted |
| move with one source edited during the flush | 16.5 s | 3.1 ms | that source kept with a warning, the 299 others deleted |
| move of 1000 files of 32 KiB | 42.7 s | 2.6 ms | 1000 files on the stick, no source left |
| move of 1000 files, stick unplugged during the flush | 11.6 s, error 129 | all 1000 sources kept | |
| copy to the local disk | 0.06 s | no flush |
The job now reports done when the data is on the stick, so the time to done is the time the stick takes to write it. Before this change, these copies reported done after a fraction of a second.
Known limitations
- The flush writes back the whole filesystem of the device, so it also waits for data that other programs wrote to it at the same time.
- Only
CopyJobflushes. AKIO::file_copy()orKIO::file_move()alone is not flushed. - A source that cannot be deleted after the flush ends the job with an error, without the per-file skip dialog. A failed source delete also ended the job before this change.
- While the sources are deleted, the progress stays at 100%; only the description changes.