Fix buffer manager takedown error.

Prepare for the possibility to evict all buffers from vram / agp.
This will be used by the X server when, for example, switching vts.