I have a similar setup, only I don't use SSH commands on the server. Instead, I have another part of my Phing file dedicated to the deployment tasks on the server and call Phing on the server via SSH.
The SSH task of Phing did not work right for me: It silently fails when the server is not in the known hosts list amd on some other commands (unzip in my case) it also silently fails. For SSH-related things I'd rather use exec. The Phing SCP task works, though.