I'm trying to write my own window manager. One issue I faced is that I don't understand how to define in which order windows should be displayed. The only means I found is to use xcb_configure_window. But it looks very limited to me: it only allows either to rise the window on top of all, or put it to the very bottom (no notion of layers or something). What a limited functionality :(.
What I really would like to do is to define the window order and tell X about it. Or to define multiple layers of windows (e.g, normal, above all, below all). So I could rise (or lower) the window with respect to its layer. So windows from lower layers will not ever cover windows from upper layers.
So, is there any other library function to define the order of windows apart from xcb_configure_window? Or I need to live with it? That would mean I have to track the order of windows in my window manager.
Link to relevant XCB documentation: https://www.x.org/releases/X11R7.6/doc/libxcb/tutorial/index.html#winstack
X11 only gives you the absolute minimum. There are no layers or always-over or always-under windows. You implement all the fancy stuff yourself, typically in the window manager, which is what you are writing.
There is no raw protocol request to change the stacking order of windows at once. Xlib has a function XRestackWindows which does that, but it uses one configure request per window. The pseudocode is just
That's it. Nothing more fancy than that.
You do need to track the stacking order in your WM in order to implement layers, so that when a program tries to restack a top level window, you intercept the request and only restack it within its layer.
Another possible way to implement layers is to have several transparent full-screen windows and reparent users' top-level windows to those, instead of to the root window. I'm not totally sure it will always work though, and you need to deal with transparency in one way or another, which requires transparency supporting hardware or the composite extension or possibly both.
It is the responsibility of the window manager to order the windows. A window gives hints about which type it is (popup dialog, menu window, splashscreen, toolbox, etc), the window manager then decides how to display this on the screen. This includes keeping track of the order and deciding which gets priority.
There is a X11 call to change the order (XRestackWindows), but I don't know the Xcb equivalent.