Skip to content
  • Robert Krawitz's avatar
    Further perf improvements for loading and correct autostacking behavior. · fdbfad64
    Robert Krawitz authored
    This involves several changes:
    
    1) Insert EXIF data for each loaded image as it's loaded, but within a
       transaction.  This may reduce I/O for big loads over batching up
       all the loads at the end because it avoids a second pass over the
       files.
    
    2) Add an image hash to the XMLDB, to allow fast lookup by name.  This
       enables the new image loader to add new files in while looking for
       duplicates and stack candidates.  The newly added images are kept
       in a pending list until the load is complete.
    
       This also improves correctness by making image load be a transaction.
       It also allows for full autostacking during load, which wasn't
       working before.
    
    3) Add a filename->MD5 map in addition to MD5->filename map to allow
       duplicate detection without having to compute the MD5 checksum
       on possible duplicate files, which is expensive in computation and
       very expensive in I/O.
    
    4) Add image scout thread during load, to read files in from disk
       ahead of the loader and reducing or eliminating time the new
       image loader waits on I/O.  It limits its readahead to 20 images
       to reduce the risk of blowing out RAM and making problems worse.
       This appears to usually benefit if I/O is needed, and I haven't
       found cases where it makes things worse (the most likely scenario
       would be big images with slow I/O and low memory).
    
    In addition, print the progress bar after the final image is loaded.
    fdbfad64