C++ and FULLY dynamic functions

Question

I have a problem with detours. Detours, as you all know, can only move among 5 bytes of space (i.e a 'jmp' call and a 4 byte address). Because of this it is impossible to have the 'hook' function in a class (a method), you cannot supply the 'this' pointer because there is simply not enough space (here's the problem more thoroughly explained). So I've been brainstorming all day for a solution, and now I want your thoughts on the subject so I don't begin a 3-5 day project without knowing if it would be possible or not.

I had 3 goals initially, I wanted the 'hook' functions to be class methods, I wanted the whole approach to be object-oriented (no static functions or global objects) and, the worst/hardest part, to be completely dynamic. This is my (in theory) solution; with assembly one can modify functions at runtime (a perfect example is any detouring method). So since I can modify functions dynamically, shouldn't I also be able to create them dynamically? For example; I allocate memory for, let's say ~30 bytes (through malloc/new). Wouldn't it be possible to just replace all bytes with binary numbers corresponding to different assembly operators (like 0xE9 is 'jmp') and then call the address directly (since it would contain a function)?

NOTE: I know on beforehand the return value, and all the arguments to all functions that I want to detour, and since I'm using GCC, the thiscall convention is practically identical to the _cdecl one.

So this is my thought/soon-to-be implementation; I create a 'Function' class. This constructor takes a variadic amount of arguments (except the first argument, which describes the return value of the target function).

Each argument is a description of the arguments the hook will receive (the size, and whether it is a pointer or not). So let's say I want to create a Function class for a int * RandomClass::IntCheckNum(short arg1);. Then I would just have to do like this:Function func(Type(4, true), Type(4, true), Type(2, false));. Where 'Type' is defined as Type(uint size, bool pointer). Then through assembly I could dynamically create the function (note: this would all be using _cdecl calling convention) since I can calculate the number of arguments and total size.

EDIT: With the example, Type(4, true) is the return value (int*), the scondType(4, true) is the RandomClass 'this' pointer and Type(2, false) describes the first argument (short arg1).

With this implementation I could easily have class methods as callbacks, but it would require an extensive amount of assembly code (which I'm not even especially experienced at). In the end, the only non-dynamic thing would be the methods in my callback class (which also would require pre and post callbacks).

So I wanted to know; is this possible? How much work would it require, and am I way over my head here?

EDIT: I'm sorry if I presented everything a bit fuzzy, but if there is something you want more thoroughly explained, do ask!

EDIT2: I'd also like to know, if I can find the hex values for all assembly operators somewhere? A list would help a ton! And/or if it is possible to somehow 'save' the asm(""); code at a memory address (which I highly doubt).

Answer 1

What you describe is usually called "thunking", and is quite commonly implemented. Historically, the most common purpose has been mapping between 16-bit and 32-bit code (by autogenerating a new 32-bit function that calls an existing 16-bit one or vice versa). I believe some C++ compilers generate similar functions to adjust base class pointers to subclass pointers in multiple inheritance, also.

It certainly seems like a viable solution to your problem, and I don't foresee any huge issues. Just make sure you allocate the memory with any flags needed in your operating system to make sure the memory is executable (most modern OSs give out non-executable memory by default).

You may find this link helpful, particularly if working in Win32: http://www.codeproject.com/Articles/16785/Thunking-in-Win32-Simplifying-Callbacks-to-Non-sta

Regarding finding the hex values of assembly operations, the best reference I know of is the Appendix to the manual of the NASM assembler (and I don't just say that because I helped write it). There's a copy available here: http://www.posix.nl/linuxassembly/nasmdochtml/nasmdoca.html