Using Vulkan for buffer allocations
Currently we're using gbm for buffer allocation, which is nice because it's pretty simple. All you need for allocating a buffer is a gbm device, size, format, modifier list and if the buffer will be used for scanout or not, and you don't have to mess with any graphics APIs to do all that.
In practice however, this simplicity is also a problem when it comes to handling multiple GPUs, as we get very little information about the devices, very little control about how buffers are allocated and how the relevant APIs access them. With Vulkan, we get a lot more control:
- before using it, we know if a device is an integrated GPU, a dedicated GPU, or just a display device with software rendering
- we can choose how to allocate buffers, whether it should be in VRAM or RAM, if it needs to be CPU accessible, cached in RAM etc
- we can choose how to transfer buffer contents - usually with a separate transfer queue, you get to use the dma hardware of the GPU to copy from and to system memory without a lot of CPU overhead
This control in Vulkan also comes with a comparatively large amount of code, but the complexity for buffer allocations should be quite manageable and encapsulated. We allocate buffers with VK_EXT_image_drm_format_modifier
, export a dmabuf fd with VK_KHR_external_memory_fd
and import that into OpenGL as usual. After allocation, the only place where Vulkan is involved is deleting the buffer again once it's no longer used.
In order to additionally use Vulkan for the multi gpu transfers, the general idea is to allocate the buffer on the integrated GPU as host-visible with a format+modifier pair compatible with the dedicated GPU, so that it can be imported to Vulkan or EGL on the dedicated GPU, which then do the actual copy to or from the integrated GPU. With two dedicated GPUs there will need to be a staging buffer in system memory to copy to and from, but the rest stays the same - the buffer that is used for compositing is always device-local, to guarantee usable performance.
One thing needs more investigation before any work can realistically start for this: I haven't found out yet how to ensure an image allocated with Vulkan can be used for scanout. This doesn't seem to be covered in any related Vulkan extensions; it's possible that there is an implicit assumption that all images allocated with VK_EXT_image_drm_format_modifier
are scanout capable, but I would rather not rely on any implicit assumptions.