... Or: How Hourglass can use the Windows debugger API to do better.
This write-up is going to get technical, there's no way around it, and no way for me write a tl;dr. You have been warned.
After a lot of problems with how Hourglass hooked into the WinAPI that I encountered when I tried to implement memory management, and several discussions on IRC etc, it was decided that the current hooking method was never going to work good enough. It was simply, too slow, several DLLs had completed their DllMain()'s before hooking even began, and this cause more and more headaches. Heaps were set up etc already before it could be managed.
The previous method of IAT patching was just not going to cut it anymore, so, a new method had to be found. After bouncing ideas with Nach and the #midipix IRC channel, among others, I was inspired to craft the solution presented here.
I am truly sorry but I forgot who else inspired me for this solution, so yell at me about it so I know! Credits are deserved where credits are due.
I was given some other potential solutions as well, which I need to point out, including Kernel level APCs (Async Procedure Calls). I found the chosen solution to be sufficient for my problem as it stands today.
The solution I went with is to behave more like a real debugger does, and depend more on debug events than before. Almost all the information needed are given in the debug events, and given much earlier than any injected DLL can even begin to execute. This is where I put my focus, how can I inject a DLL as early as the process space setup (which is before code executes)? There's this handy debug event for this that triggers when the process has just been created, and it also freezes the process at just the right moment by itself (being a debug event and all). There's a similar event for each DLL the process pulls in.
The missing information? Well, that's where in my own DLL the code is that I want the hooks to execute, the linker can help me with that part. I just had to tell it to generate map-files, which I then can parse, and know exactly what parts in my DLL I need to patch or refer to.
Here I started looking at "How do I get my DLL into the remote process space when I can't load it using an API call?" as there is no way to load a DLL into a process that hasn't executed a single instruction of it's code. Not with the APIs at least...
Enter the failed attempts:
> Load the DLL as a file and just write it in the process space. --- This does not work, the DLL does not have the same structure in memory as it has as a file. Caused a lot of headaches with memory access violations and pointers not pointing at anything sane in memory.
> Load the DLL into my own process and copy it to the process space. --- This does not work, the DLL does not handle this if the address in the remote memory is different from the local memory. Also came with a lot of issues regarding the copying itself, which usually never actually took place leaving me with uninitialized memory instead of a DLL to execute.
What I needed was my own way of loading a DLL as the API call LoadLibrary loads it (except for the execution of DllMain()). This sent me on a very interesting adventure among PE and COFF specifications. These specifications define the layout of executable files for Windows. In order to load the DLL correctly, I had to learn how to parse these things. And that I did.
I apologize in advance for the not very exciting picture (click to zoom):
Here we see Hourglass running my little demo program that does nothing except create a window. In the background is where the exciting stuff is going on as that is the call stacks and debug messages exported by Hourglass when it's captured by Visual Studio. Pay attention at the top of the log where it says "MyCreateWindowExW: Hello from the hook!", this is not part of the demo program, but the hook I attached to the API call that creates the window, CreateWindowExW.
I also need to specifically thank ais523 and Masterjun who gave me invaluable help when I broke the assembly patching of API calls: THANK YOU!
Code is here for those who want to admire it, or be completely horrified by it:
https://github.com/Warepire/Hourglass-Resurrection/tree/new_hooking_poc