build: Don't depend on libffi unless we're building libraries

When building just the scanner or docs it's not required.

This can reduce requirements for a host cross compiling wayland,
since only the scanner needs to be built natively.

Signed-off-by: Derek Foreman <derekf@osg.samsung.com>
Reviewed-by: Peter Hutterer <peter.hutterer@who-t.net>
Reviewed-by: Pekka Paalanen <pekka.paalanen@collabora.co.uk>