Fix high memory consumption of SAM2

Fixes: #1973 (closed)

Official AsyncVideoFrameLoader loads all frames into memory which prevents it for being used clips longer than a few seconds. This introduces our own version of AsyncVideoFrameLoader which doesn't cache all images.

Check out the comment #1973 (comment 1199934) for more details.

Didn't bother to create a PR for the official Facebook repo. Based on outstanding open PRs and official activity on that repo its not a community project. Need to fix this on our side unfortunately.

Its basically as three line change as mentioned in the comment but needed to create our custom SAM2VideoPredictor which delegates to the official SAM2VideoPredictorOfficial except loading the images in init in order to fix it (Wanted to avoid forking SAM2 repo so we don't have another repo to maintain...).

I intend to work a bit more on the SAM integration and added a few TODOs for myself. Will clean up this code and fix the TODOs in future MRs.

Also, while testing the feature looks like preview mode is somewhat broken (preview seems to work only for the first frame atm).

Merge request reports

Loading