iczelion tut24
Tutorial 24: Windows Hooks
We will learn about Windows hooks in this tutorial. Windows hooks are very powerful. With them, you can poke inside other processes and sometimes alter their behaviors.
Download the example here.
Theory:
Windows hooks can be considered one of the most powerful features of Windows. With them, you can trap events that will occur, either in your own process or in other processes. By "hooking", you tell Windows about a function, filter function also called hook procedure, that will be called everytime an event you're interested in occurs. There are two types of them: local and remote hooks.- Local hooks trap events that will occur in your own process.
 - Remote hooks trap events that will occur in other process(es). There are two types of remote hooks 
- thread-specific traps events that will occur in a specific thread in other process. In short, you want to observe events in a specific thread in a specific process.
 - system-wide traps all events destined for all threads in all processes in the system.
 
 
You have to understand how a hook works before you can use it efficiently. When you create a hook, Windows creates a data structure in memory, containing information about the hook, and adds it to a linked list of existing hooks. New hook is added in front of old hooks. When an event occurs, if you install a local hook, the filter function in your process is called so it's rather straightforward. But if it's a remote hook, the system must inject the code for the hook procedure into the address space(s) of the other process(es). And the system can do that only if the function resides in a DLL. Thus , if you want to use a remote hook, your hook procedure must reside in a DLL. There is two exceptions to this rule: journal record and journal playback hooks. The hook procedures for those two hooks must reside in the thread that installs the hooks. The reason why it must be so is that: both hooks deal with the low-level interception of hardware input events. The input events must be recorded/playbacked in the order they appeared. If the code of those two hooks is in a DLL, the input events may scatter among several threads and it is impossible to know the order of them. So the solution: the hook procedure of those two hooks must be in a single thread only i.e. the thread that installs the hooks.
There are 14 types of hooks:
- WH_CALLWNDPROC called when SendMessage is called
 - WH_CALLWNDPROCRET called when SendMessage returns
 - WH_GETMESSAGE called when GetMessage or PeekMessage is called
 - WH_KEYBOARD called when GetMessage or PeekMessage retrieves WM_KEYUP or WM_KEYDOWN from the message queue
 - WH_MOUSE called when GetMessage or PeekMessage retrieves a mouse message from the message queue
 - WH_HARDWARE called when GetMessage or PeekMessage retrieves some hardware message that is not related to keyboard or mouse.
 - WH_MSGFILTER called when a dialog box, menu or scrollbar is about to process a message. This hook is local. It's specifically for those objects which have their own internal message loops.
 - WH_SYSMSGFILTER same as WH_MSGFILTER but system-wide
 - WH_JOURNALRECORD called when Windows retrieves message from the hardware input queue
 - WH_JOURNALPLAYBACK called when an event is requested from the system's hardware input queue.
 - WH_SHELL called when something interesting about the shell occurs such as when the task bar needs to redraw its button.
 - WH_CBT used specifically for computer-based training (CBT).
 - WH_FOREGROUNDIDLE used internally by Windows. Little use for general applications
 - WH_DEBUG used to debug the hooking procedure
 
To install a hook, you call SetWindowsHookEx which has the following syntax:
SetWindowsHookEx proto HookType:DWORD, pHookProc:DWORD, hInstance:DWORD, ThreadID:DWORDYou can uninstall a hook by calling UnhookWindowsHookEx which accepts only one parameter, the handle of the hook you want to uninstall. If the call succeeds, it returns a non-zero value in eax. Otherwise, it returns NULL.If the call is successful, it returns the hook handle in eax. If not, NULL is returned. You must save the hook handle for unhooking later.
- HookType is one of the values listed above, e.g., WH_MOUSE, WH_KEYBOARD
 - pHookProc is the address of the hook procedure that will be called to process the messages for the specified hook. If the hook is a remote one, it must reside in a DLL. If not, it must be in your process.
 - hInstance is the instance handle of the DLL in which the hook procedure resides. If the hook is a local one, this value must be NULL
 - ThreadID is the ID of the thread you want to install the hook to spy on. This parameter is the one that determines whether a hook is local or remote. If this parameter is NULL, Windows will interpret the hook as a system-wide remote hook that affects all threads in the system. If you specify the thread ID of a thread in your own process, this hook is a local one. If you specify the thread ID from other process, the hook is a thread-specific remote one. There are two exceptions to this rule: WH_JOURNALRECORD and WH_JOURNALPLAYBACK are always local system-wide hooks that are not required to be in a DLL. And WH_SYSMSGFILTER is always a system-wide remote hook. Actually it is identical to WH_MSGFILTER hook with ThreadID==0.
 
Now that you know how to install/uninstall hooks, we can examine the hook procedure.
The hook procedure will be called whenever an event that is associated with the type of hook you have installed occurs. For example, if you install WH_MOUSE hook, when a mouse event occurs, your hook procedure will be called. Regardless of the type of hook you installed, the hook procedure always has the following prototype:
- HookProc proto nCode:DWORD, wParam:DWORD, lParam:DWORD 
 - nCode specifies the hook code.
 - wParam and lParam contain additional information about the event
 
WH_CALLWNDPROCThe bottom line is: you must consult your win32 api reference for details about the meanings of the parameters and return value of the hook you want to install.WH_MOUSE
- nCode can be only HC_ACTION which means there is a message sent to a window
 - wParam contains the message being sent, if it's not zero
 - lParam points to a CWPSTRUCT structure
 - return value: not used, return zero
 
- nCode can be HC_ACTION or HC_NOREMOVE
 - wParam contains the mouse message
 - lParam points to a MOUSEHOOKSTRUCT structure
 - return value: zero if the message should be processed. 1 if the message should be discarded.
 
Now there is a little catch about the hook procedure. Remember that the hooks are chained in a linked list with the most recently installed hook at the head of the list. When an event occurs, Windows will call only the first hook in the chain. It's your hook procedure's responsibility to call the next hook in the chain. You can choose not to call the next hook but you'd better know what you're doing. Most of the time, it's a good practice to call the next procedure so other hooks can have a shot at the event. You can call the next hook by calling CallNextHookEx which has the following prototype:
CallNextHookEx proto hHook:DWORD, nCode:DWORD, wParam:DWORD, lParam:DWORDAn important note about remote hooks: the hook procedure must reside in a DLL which will be mapped into other processes. When Windows maps the DLL into other processes, it will not map the data section(s) into the other processes. In short, all processes share a single copy of code but they will have their own private copy of the DLL's data section! This can be a big surprise to the unwary. You may think that when you store a value into a variable in the data section of a DLL, that value will be shared among all processes that load the DLL into their process address space. It's simply not true. In normal situation, this behavior is desirable since it provides the illusion that each process has its own copy of the DLL. But not when Windows hook is concerned. We want the DLL to be identical in all processes, including the data. The solution: you must mark the data section as shared. You can do this by specifying the section(s) attribute in the linker switch. For MASM, you need to use this switch:
- hHook is your own hook handle. The function uses this handle to traverse the linked list and search for the hook procedure it should call next.
 - nCode, wParam and lParam you can just pass those three values you receive from Windows to CallNextHookEx.
 
/SECTION:<section name>, SThe name of the initialized data section is .data and the uninitialized data is .bss. For example if you want to assemble a DLL which contains a hook procedure and you want the uninitialized data section to be shared amoung processes, you must use the following line:
link /section:.bss,S /DLL /SUBSYSTEM:WINDOWS ..........S attribute marks the section as shared.
Example:
There are two modules: one is the main program which will do the GUI part and the other is the DLL that will install/uninstall the hook.;--------------------------------------------- This is the source code of the main program -------------------------------------- 
.386 
.model flat,stdcall 
option casemap:none 
include masm32includewindows.inc 
include masm32includeuser32.inc 
include masm32includekernel32.inc 
include mousehook.inc 
includelib mousehook.lib 
includelib masm32libuser32.lib 
includelib masm32libkernel32.lib 
wsprintfA proto C :DWORD,:DWORD,:VARARG 
wsprintf TEXTEQU <wsprintfA> 
.const 
IDD_MAINDLG                   equ 101 
IDC_CLASSNAME              equ 1000 
IDC_HANDLE                     equ 1001 
IDC_WNDPROC                 equ 1002 
IDC_HOOK                         equ 1004 
IDC_EXIT                           equ 1005 
WM_MOUSEHOOK             equ WM_USER+6 
DlgFunc PROTO :DWORD,:DWORD,:DWORD,:DWORD
.data 
HookFlag dd FALSE 
HookText db "&Hook",0 
UnhookText db "&Unhook",0 
template db "%lx",0 
.data? 
hInstance dd ? 
hHook dd ? 
.code 
start: 
    invoke GetModuleHandle,NULL 
    mov hInstance,eax 
    invoke DialogBoxParam,hInstance,IDD_MAINDLG,NULL,addr DlgFunc,NULL 
    invoke ExitProcess,NULL 
DlgFunc proc hDlg:DWORD,uMsg:DWORD,wParam:DWORD,lParam:DWORD 
    LOCAL hLib:DWORD 
    LOCAL buffer[128]:byte 
    LOCAL buffer1[128]:byte 
    LOCAL rect:RECT 
    .if uMsg==WM_CLOSE 
        .if HookFlag==TRUE 
            invoke UninstallHook 
        .endif 
        invoke EndDialog,hDlg,NULL 
    .elseif uMsg==WM_INITDIALOG 
        invoke GetWindowRect,hDlg,addr rect 
        invoke SetWindowPos, hDlg, HWND_TOPMOST, rect.left, rect.top, rect.right, rect.bottom, SWP_SHOWWINDOW 
    .elseif uMsg==WM_MOUSEHOOK 
        invoke GetDlgItemText,hDlg,IDC_HANDLE,addr buffer1,128 
        invoke wsprintf,addr buffer,addr template,wParam 
        invoke lstrcmpi,addr buffer,addr buffer1 
        .if eax!=0 
            invoke SetDlgItemText,hDlg,IDC_HANDLE,addr buffer 
        .endif 
        invoke GetDlgItemText,hDlg,IDC_CLASSNAME,addr buffer1,128 
        invoke GetClassName,wParam,addr buffer,128 
        invoke lstrcmpi,addr buffer,addr buffer1 
        .if eax!=0 
            invoke SetDlgItemText,hDlg,IDC_CLASSNAME,addr buffer 
        .endif 
        invoke GetDlgItemText,hDlg,IDC_WNDPROC,addr buffer1,128 
        invoke GetClassLong,wParam,GCL_WNDPROC 
        invoke wsprintf,addr buffer,addr template,eax 
        invoke lstrcmpi,addr buffer,addr buffer1 
        .if eax!=0 
            invoke SetDlgItemText,hDlg,IDC_WNDPROC,addr buffer 
        .endif 
    .elseif uMsg==WM_COMMAND 
        .if lParam!=0 
            mov eax,wParam 
            mov edx,eax 
            shr edx,16 
            .if dx==BN_CLICKED 
                .if ax==IDC_EXIT 
                    invoke SendMessage,hDlg,WM_CLOSE,0,0 
                .else 
                    .if HookFlag==FALSE 
                        invoke InstallHook,hDlg 
                        .if eax!=NULL 
                            mov HookFlag,TRUE 
                            invoke SetDlgItemText,hDlg,IDC_HOOK,addr UnhookText 
                        .endif 
                    .else 
                        invoke UninstallHook 
                        invoke SetDlgItemText,hDlg,IDC_HOOK,addr HookText 
                        mov HookFlag,FALSE 
                        invoke SetDlgItemText,hDlg,IDC_CLASSNAME,NULL 
                        invoke SetDlgItemText,hDlg,IDC_HANDLE,NULL 
                        invoke SetDlgItemText,hDlg,IDC_WNDPROC,NULL 
                    .endif 
                .endif 
            .endif 
        .endif 
    .else 
        mov eax,FALSE 
        ret 
    .endif 
    mov eax,TRUE 
    ret 
DlgFunc endp 
end start
;----------------------------------------------------- This is the source code of the DLL -------------------------------------- 
.386 
.model flat,stdcall 
option casemap:none 
include masm32includewindows.inc 
include masm32includekernel32.inc 
includelib masm32libkernel32.lib 
include masm32includeuser32.inc 
includelib masm32libuser32.lib 
.const 
WM_MOUSEHOOK equ WM_USER+6 
.data 
hInstance dd 0 
.data? 
hHook dd ? 
hWnd dd ? 
.code 
DllEntry proc hInst:HINSTANCE, reason:DWORD, reserved1:DWORD 
    .if reason==DLL_PROCESS_ATTACH 
        push hInst 
        pop hInstance 
    .endif 
    mov  eax,TRUE 
    ret 
DllEntry Endp 
MouseProc proc nCode:DWORD,wParam:DWORD,lParam:DWORD 
    invoke CallNextHookEx,hHook,nCode,wParam,lParam 
    mov edx,lParam 
    assume edx:PTR MOUSEHOOKSTRUCT 
    invoke WindowFromPoint,[edx].pt.x,[edx].pt.y 
    invoke PostMessage,hWnd,WM_MOUSEHOOK,eax,0 
    assume edx:nothing 
    xor eax,eax 
    ret 
MouseProc endp 
InstallHook proc hwnd:DWORD 
    push hwnd 
    pop hWnd 
    invoke SetWindowsHookEx,WH_MOUSE,addr MouseProc,hInstance,NULL 
    mov hHook,eax 
    ret 
InstallHook endp 
UninstallHook proc 
    invoke UnhookWindowsHookEx,hHook 
    ret 
UninstallHook endp 
End DllEntry
;---------------------------------------------- This is the makefile of the DLL ----------------------------------------------
NAME=mousehook 
$(NAME).dll: $(NAME).obj 
        Link /SECTION:.bss,S  /DLL /DEF:$(NAME).def /SUBSYSTEM:WINDOWS /LIBPATH:c:masmlib $(NAME).obj 
$(NAME).obj: $(NAME).asm 
        ml /c /coff /Cp $(NAME).asm 
  
Analysis:
The example will display a dialog box with three edit controls that will be filled with the class name, window handle and the address of the window procedure associated with the window under the mouse cursor. There are two buttons, Hook and Exit. When you press the Hook button, the program hooks the mouse input and the text on the button changes to Unhook. When you move the mouse cursor over a window, the info about that window will be displayed in the main window of the example. When you press Unhook button, the program removes the mouse hook.The main program uses a dialog box as its main window. It defines a custom message, WM_MOUSEHOOK which will be used between the main program and the hook DLL. When the main program receives this message, wParam contains the handle of the window that the mouse cursor is on. Of course, this is an arbitrary arrangement. I decide to send the handle in wParam for the sake of simplicity. You can choose your own method of communication between the main program and the hook DLL.
                    .if HookFlag==FALSE 
                        invoke InstallHook,hDlg 
                        .if eax!=NULL 
                            mov HookFlag,TRUE 
                            invoke SetDlgItemText,hDlg,IDC_HOOK,addr UnhookText 
                        .endif 
The program maintains a flag, HookFlag, to monitor the state of the hook. It's FALSE if the hook is not installed and TRUE if the hook is installed. 
When the user presses Hook button, the program checks if the hook is already installed. If it is not, it call InstallHook function in the hook DLL to install it. Note that we pass the handle of the main dialog as the parameter of the function so the hook DLL can send the WM_MOUSEHOOK messages to the right window i.e. our own. 
When the program is loaded, the hook DLL is loaded too. Actually, DLLs are loaded immediately after the program is in memory. The DLL entrypoint function is called before the first instruction in the main program is execute even. So when the main program executes the DLL(s) is/are initialized. We put the following code in the DLL entrypoint function of the hook DLL: 
    .if reason==DLL_PROCESS_ATTACH 
        push hInst 
        pop hInstance 
    .endif 
The code just saves the instance handle of the hook DLL itself to a global variable named hInstance for use within the InstallHook function. Since the DLL entrypoint function is called before other functions in the DLL are called , hInstance is always valid. We put hInstance in .data section so that this value is kept on per-process basis. Since when the mouse cursor hovers over a window, the hook DLL is mapped into the process. Imagine that there is already a DLL that occupies the intended load address of the hook DLL, the hook DLL would be remapped to another address. The value of hInstance will be updated to those of the new load address. When the user presses Unhook button and then Hook button, SetWindowsHookEx will be called again. However, this time, it will use the new load address as the instance handle which will be wrong because in the example process, the hook DLL's load address hasn't been changed. The hook will be a local one where you can hook only the mouse events that occur in your own window. Hardly desirable.
InstallHook proc hwnd:DWORD 
    push hwnd 
    pop hWnd 
    invoke SetWindowsHookEx,WH_MOUSE,addr MouseProc,hInstance,NULL 
    mov hHook,eax 
    ret 
InstallHook endp 
The InstallHook function itself is very simple. It saves the window handle passed as its parameter to a global variable named hWnd for future use. It then calls SetWindowsHookEx to install a mouse hook. The return value of SetWindowsHookEx is stored in a global variable named hHook for use with UnhookWindowsHookEx. 
After SetWindowsHookEx is called, the mouse hook is functional. Whenever a mouse event occurs in the system, MouseProc ( your hook procedure) is called. 
MouseProc proc nCode:DWORD,wParam:DWORD,lParam:DWORD 
    invoke CallNextHookEx,hHook,nCode,wParam,lParam 
    mov edx,lParam 
    assume edx:PTR MOUSEHOOKSTRUCT 
    invoke WindowFromPoint,[edx].pt.x,[edx].pt.y 
    invoke PostMessage,hWnd,WM_MOUSEHOOK,eax,0 
    assume edx:nothing 
    xor eax,eax 
    ret 
MouseProc endp 
The first thing it does is to call CallNextHookEx to give the other hooks the chance to process the mouse event. After that, it calls WindowFromPoint function to retrieve the handle of the window at the specified screen coordinate. Note that we use the POINT structure in the MOUSEHOOKSTRUCT structure pointed to by lParam as the current mouse coordinate. After that we send the window handle to the main program via PostMessage with WM_MOUSEHOOK message. One thing you should remember is that: you should not use SendMessage inside the hook procedure, it can cause message deadlock. PostMessage is recommended. The MOUSEHOOKSTRUCT structure is defined below:
MOUSEHOOKSTRUCT STRUCT DWORD 
  pt            POINT <> 
  hwnd          DWORD      ? 
  wHitTestCode  DWORD      ? 
  dwExtraInfo   DWORD      ? 
MOUSEHOOKSTRUCT ENDS 
  
- pt is the current screen coordinate of the mouse cursor
 - hwnd is the handle of the window that will receive the mouse message. It's usually the window under the mouse cursor but not always. If a window calls SetCapture, the mouse input will be redirected to that window instead. Because of this reason, I don't use the hwnd member of this structure but choose to call WindowFromPoint instead.
 - wHitTestCode specifies the hit-test value. The hit-test value gives more information about the current mouse cursor position. It specifies on what part of window the mouse cursor is. For complete list, check your win32 api reference under WM_NCHITTEST message.
 - dwExtraInfo contains the extra information associated with the message. Normally this value is set by calling mouse_event and retrieved by calling GetMessageExtraInfo.
 
    .elseif uMsg==WM_MOUSEHOOK 
        invoke GetDlgItemText,hDlg,IDC_HANDLE,addr buffer1,128 
        invoke wsprintf,addr buffer,addr template,wParam 
        invoke lstrcmpi,addr buffer,addr buffer1 
        .if eax!=0 
            invoke SetDlgItemText,hDlg,IDC_HANDLE,addr buffer 
        .endif 
        invoke GetDlgItemText,hDlg,IDC_CLASSNAME,addr buffer1,128 
        invoke GetClassName,wParam,addr buffer,128 
        invoke lstrcmpi,addr buffer,addr buffer1 
        .if eax!=0 
            invoke SetDlgItemText,hDlg,IDC_CLASSNAME,addr buffer 
        .endif 
        invoke GetDlgItemText,hDlg,IDC_WNDPROC,addr buffer1,128 
        invoke GetClassLong,wParam,GCL_WNDPROC 
        invoke wsprintf,addr buffer,addr template,eax 
        invoke lstrcmpi,addr buffer,addr buffer1 
        .if eax!=0 
            invoke SetDlgItemText,hDlg,IDC_WNDPROC,addr buffer 
        .endif 
To avoid flickers, we check the text already in the edit controls and the text we will put into them if they are identical. If they are, we skip them. 
We retrieve the class name by calling GetClassName, the address of the window procedure by calling GetClassLong with GCL_WNDPROC and then format them into strings and put them into the appropriate edit controls. 
                        invoke UninstallHook 
                        invoke SetDlgItemText,hDlg,IDC_HOOK,addr HookText 
                        mov HookFlag,FALSE 
                        invoke SetDlgItemText,hDlg,IDC_CLASSNAME,NULL 
                        invoke SetDlgItemText,hDlg,IDC_HANDLE,NULL 
                        invoke SetDlgItemText,hDlg,IDC_WNDPROC,NULL 
When the user presses Unhook button, the program calls UninstallHook function in the hook DLL. UninstallHook just calls UnhookWindowsHookEx. After that, it changes the text of the button back to "Hook", HookFlag to FALSE and clears the content of the edit controls. 
Note the linker switch in the makefile. 
Link /SECTION:.bss,S /DLL /DEF:$(NAME).def /SUBSYSTEM:WINDOWS
It specifies .bss section as a shared section to make all processes share the same uninitialized data section of the hook DLL. Without this switch, your hook DLL will not function correctly.
