Wanting to deploy to Azure using Continuous Delivery but not use the Visual Studio Hosted Build servers?

No problem; but you need to install the right version of things first!

I installed:

 

I then installed a couple of the assemblies from the Azure SDK Lib's into the GAC; these get installed into C:\Program Files\Microsoft SDKs\Windows Azure\.NET SDK\v2.0\ref, and I installed:
Microsoft.ServiceBus
Microsoft.WindowsAzure.Storage
Microsoft.WindowsAzure.Configuration

If you don't do this, you get an error during the deployment portion of the workflow.

If you try it, let me know how you get on!