How do I skip the "Preparing Working Directory" step?

That is the checkout phase. In recent agents, you can set BUILDKITE_REPO to a blank string in a pre-checkout hook and it will skip it.

Might also be worth checking out Shared git repository checkouts in the Agent.