I've been leaning on this direction too.
Prepare the release locally (or in Continuous Deployment server), build all the assets needed, installing all composer packages, checking git modules if any. Then just rsync the release to production.
It also removes the need of having GIT and private keys (eg: when having private composer packages) on the server. One less key to manage.