How do I use JWT to access Google Directory (Admin

I'm trying to make a server application that will add/remove users to my domain's groups. Note that it will not have any interaction with users, it is server-to-server application.

I registered my application in the Google API Console, downloaded the key and transformed it to .pem by issuing

openssl pkcs12 -in my_google_key.p12 -out my_google_key.pem -nocerts -nodes

Then I've been to the Domain Administration, Security -> Advanced Settings -> Authentication -> Manage OAuth Client access. There I added a record in Authorized API clients. I used the Client ID I got from the Service Account in Console and used scope:

https://www.googleapis.com/auth/admin.directory.group.

I installed googleapis for nodejs, using

npm install googleapis

And here's my code:

var googleapis = require('googleapis');

var SERVICE_ACCOUNT_EMAIL = 'My Service Account E-mail Address';
var SERVICE_ACCOUNT_KEY_FILE = 'my_google_key.pem'; // The .pem file is at the root of my application

var jwt = new googleapis.auth.JWT(
    SERVICE_ACCOUNT_EMAIL,
    SERVICE_ACCOUNT_KEY_FILE,
    null,
    ['https://www.googleapis.com/auth/admin.directory.group']
);

var client;

googleapis
.discover('admin', 'directory_v1')
.execute(function(err, data) {
    client = data;

    jwt.authorize(function(err, result) {
        console.log(jwt);
        client.admin.groups.list({
            "customer": "my_customer", // This is actually "my_customer"
            "domain": "domain.com" // The domain name I administer
        })
        .withAuthClient(jwt)
        .execute(function(err, result) {
            console.log(err);
            console.log(result);
        });
    });
});

And the result of running this code is:

{ errors: 
    [ { domain: 'global',
        reason: 'forbidden',
        message: 'Not Authorized to access this resource/api' } ],
    code: 403,
    message: 'Not Authorized to access this resource/api' }

What am I missing? How do I authorize my application with the Admin SDK?

1) Make sure you delegate domain wide authority to your service account.

2) Service Accounts must impersonate someone with access to the Admin SDK Directory API.

Include it when initializing:

var jwt = new googleapis.auth.JWT(
    SERVICE_ACCOUNT_EMAIL,
    SERVICE_ACCOUNT_KEY_FILE,
    null,
    ['https://www.googleapis.com/auth/admin.directory.group'],
    account_with_Admin_SDK_access_to_impersonate@example.com
);

This section of the docs explains both of these in detail: https://developers.google.com/admin-sdk/directory/v1/guides/delegation#delegate_domain-wide_authority_to_your_service_account