Fix recursive directory traversal and remove file skipping

Reverts 1c62b82d

The manual directory traversal was broken if someone used name filters. Besides that, we have "filters" and "exclude_patterns" to skip stuff, those should be used instead of us skipping things for people which may not be wanted in some cases.