Last year I posted a similar question, but due to it not being very descriptive and overall a mess I decided to rewrite it completely.
I am making a simple 3D editor in JavaFX that has a camera movement similar to Blender or pretty much any other one. I added the rotation around the x-axis and then using trigonometric functions I calculated the other 2 rotations (y and z-axis). Both x and y-axis rotations are made relative to the pivot point, but the z rotation is always made relative to the screen/camera, not its pivot point.
For a better understanding of the problem, I sketched where the axis is and where I'd like to have them:
This is how angles are applied at the moment:
This is how I'd like them to be:
Here is the part of the code I use to calculate the angles (although from my observations it seems irrelevant to my problem):
//variables angleY and angleX are calculated according to mouse drags on the screen
rotateY.setAngle(angleY);
rotateX.setAngle(angleX*Math.cos(Math.toRadians(angleY)));
rotateZ.setAngle(angleX*Math.sin(Math.toRadians(angleY)));
To make testing easier, I have uploaded the code on Pastebin.
Edit: I could achieve this same effect by rotating everything, but the camera (which is exactly what the camera movement does anyway) and using x-axis for elevation, but I'm looking for a more elegant solution.