I am posting this in case anyone else runs into the same problem. We implemented the EF6 TransactionHandler which is suppose to help transient errors on Sql Azure.
https://msdn.microsoft.com/en-us/library/dn630221(v=vs.113).aspx
We didn't realize the table does not get cleaned. Eventually the __transactionHistory table had thousands of records. Every time we wrote to SQL, because of this mechanism, it caused horrible 5-10 second delays.
The fix, we removed it. The implementation from MS should have auto cleaned that table. Ours did not.
Ef6 Azure Sql slow writes