Google Drive Android API and Cross Client Authoriz

I use Google Drive Android API for letting users upload files from Google Drive on their Android device to my servers. I would now like to change this setup to allow my servers to have direct access to the users Google Drive, using Cross Client Authorization, which means that I have to generate an ID Token using GoogleAuthUtil.getToken. And this is were everything go off the rails. In order to generate this token I need an Account object.

Currently, I'm setting up the GoogleApiClient like this:

mGoogleApiClient = new GoogleApiClient.Builder(getContext())
    .addApi(Drive.API)
    .addScope(Drive.SCOPE_FILE)
    .addConnectionCallbacks(this)
    .addOnConnectionFailedListener(this)
    .build();

This works great, every now and then it shows an account picker where the user can choose what account to connect with. All managed by the framework. But there is no way to access the Account the user chose to connect with!

Instead, I have to manage it myself, using:

Intent intent = AccountPicker.newChooseAccountIntent(null, null, new String[]{"com.google"},
    false, null, null, null, null);
startActivityForResult(intent, RC_ACCOUNT_PICKER);

And in onActivityResult I can get the selected account using:

Account account = new Account(intent.getExtras().getString(AccountManager.KEY_ACCOUNT_NAME),
    intent.getExtras().getString(AccountManager.KEY_ACCOUNT_TYPE));

And now, since I have an Account object, I can connect explicitly using that:

mGoogleApiClient = new GoogleApiClient.Builder(getContext())
    .addApi(Drive.API)
    .addScope(Drive.SCOPE_FILE)
    .setAccountName(mAccount.name) // Account name!
    .addConnectionCallbacks(this)
    .addOnConnectionFailedListener(this)
    .build();

Is this my only option, is there really no way to get the Account the user selected using my first solution?

Later on, when trying to generate my ID Token using:

String scope = String.format("oauth2:server:client_id:%s:api_scope:%s",
    "12345.apps.googleusercontent.com", 
    "https://www.googleapis.com/auth/drive.file");
String token = GoogleAuthUtil.getToken(getContext(), mAccount, scope);

I'm still getting an UserRecoverableAuthException where I need to be granted yet another permission from the user, even though I'm using the same account and the same auth scope as when I connect. When all that is taken care of, I can run the same code again and finally get my ID Token.

Is there something I've fundamentally misunderstood or this is really how it's supposed to work?

You shouldn't use GoogleAuthUtil.getToken() anymore as explained in this blog post as it is unsafe to pass those tokens to your server.

Instead, you should use Enable Server-Side Access via Google Sign In and use the requestServerAuthCode() method.

This avoids double prompting the user, does not use the GET_ACCOUNTS permission, and gives you a single user flow for logging in.