What API does VBoxManage.exe use?

VBoxManage.exe is an Oracle VirtualBox companion utility, which allows to control VMs via command line. It can do numerous operations, including start/stop and screen capturing.

I am interested, which API it uses?

How can I capture VM screen or send keyboard or mouse commands there without this heavy commandline utility? Which language is better? Is is possible to access this API with Java?

One of the advantages to using an open source project is supposed to be that you can answer such questions by looking at the source.

VBoxManage is located in the source repository under /src/VBox/Frontends/VBoxManage. The code you're looking for is in VBoxManageControlVM.cpp under the condition if (!strcmp(a->argv[1], "screenshotpng")):

ComPtr<IDisplay> pDisplay;
CHECK_ERROR_BREAK(console, COMGETTER(Display)(pDisplay.asOutParam()));

ULONG width, height, bpp;
CHECK_ERROR_BREAK(pDisplay, 
        GetScreenResolution(displayIdx, &width, &height, &bpp));

com::SafeArray<BYTE> saScreenshot;
CHECK_ERROR_BREAK(pDisplay, TakeScreenShotPNGToArray(displayIdx, 
        width, height, ComSafeArrayAsOutParam(saScreenshot)));

RTFILE pngFile = NIL_RTFILE;
vrc = RTFileOpen(&pngFile, a->argv[2], RTFILE_O_OPEN_CREATE | RTFILE_O_WRITE |
    RTFILE_O_TRUNCATE | RTFILE_O_DENY_ALL);

if (RT_FAILURE(vrc))
{
    RTMsgError("Failed to create file '%s'. rc=%Rrc", a->argv[2], vrc);
    rc = E_FAIL;
    break;
}
vrc = RTFileWrite(pngFile, saScreenshot.raw(), saScreenshot.size(), NULL);
if (RT_FAILURE(vrc))
{
    RTMsgError("Failed to write screenshot to file '%s'. rc=%Rrc",
        a->argv[2], vrc);
    rc = E_FAIL;
}
RTFileClose(pngFile);

So it's done via a COM API, and you can look at:

Is it possible to call a COM API from Java?

Googling for TakeScreenShotPNGToArray finds the display interface:

https://www.virtualbox.org/sdkref/interface_i_display.html

From there you can find the list of all the other things like mouse and keyboard:

https://www.virtualbox.org/sdkref/annotated.html