iczelion tut30

Tutorial 30: Win32 Debug API part 3

---

In this tutorial, we continue the exploration of win32 debug api. Specifically, we will learn how to trace the debuggee.
Theory:

If you have used a debugger before, you would be familiar with tracing. When you "trace" a program, the program stops after executing each instruction, giving you the chance to examine the values of registers/memory. Single-stepping is the official name of tracing.
The single-step feature is provided by the CPU itself. The 8th bit of the flag register is called trap flag. If this flag(bit) is set, the CPU executes in single-step mode. The CPU will generate a debug exception after each instruction. After the debug exception is generated, the trap flag is cleared automatically.
We can also single-step the debuggee, using win32 debug api. The steps are as follows:

1. Call GetThreadContext, specifying CONTEXT_CONTROL in ContextFlags, to obtain the value of the flag register.
2. Set the trap bit in regFlag member of the CONTEXT structure
3. call SetThreadContext
4. Wait for the debug events as usual. The debuggee will execute in single-step mode. After it executes each instruction, we will get EXCEPTION_DEBUG_EVENT with EXCEPTION_SINGLE_STEP value in u.Exception.pExceptionRecord.ExceptionCode
5. If you need to trace the next instruction, you need to set the trap bit again.

Example:

.386
.model flat,stdcall
option casemap:none
include masm32includewindows.inc
include masm32includekernel32.inc
include masm32includecomdlg32.inc
include masm32includeuser32.inc
includelib masm32libkernel32.lib
includelib masm32libcomdlg32.lib
includelib masm32libuser32.lib

.data
AppName db "Win32 Debug Example no.4",0
ofn OPENFILENAME <>
FilterString db "Executable Files",0,"*.exe",0
             db "All Files",0,"*.*",0,0
ExitProc db "The debuggee exits",0Dh,0Ah
         db "Total Instructions executed : %lu",0
TotalInstruction dd 0

.data?
buffer db 512 dup(?)
startinfo STARTUPINFO <>
pi PROCESS_INFORMATION <>
DBEvent DEBUG_EVENT <>
context CONTEXT <>

.code
start:
mov ofn.lStructSize,SIZEOF ofn
mov ofn.lpstrFilter, OFFSET FilterString
mov ofn.lpstrFile, OFFSET buffer
mov ofn.nMaxFile,512
mov ofn.Flags, OFN_FILEMUSTEXIST or OFN_PATHMUSTEXIST or OFN_LONGNAMES or OFN_EXPLORER or OFN_HIDEREADONLY
invoke GetOpenFileName, ADDR ofn
.if eax==TRUE
    invoke GetStartupInfo,addr startinfo
    invoke CreateProcess, addr buffer, NULL, NULL, NULL, FALSE, DEBUG_PROCESS+ DEBUG_ONLY_THIS_PROCESS, NULL, NULL, addr startinfo, addr pi
    .while TRUE
       invoke WaitForDebugEvent, addr DBEvent, INFINITE
       .if DBEvent.dwDebugEventCode==EXIT_PROCESS_DEBUG_EVENT
          invoke wsprintf, addr buffer, addr ExitProc, TotalInstruction
          invoke MessageBox, 0, addr buffer, addr AppName, MB_OK+MB_ICONINFORMATION
          .break
       .elseif DBEvent.dwDebugEventCode==EXCEPTION_DEBUG_EVENT           .if DBEvent.u.Exception.pExceptionRecord.ExceptionCode==EXCEPTION_BREAKPOINT
             mov context.ContextFlags, CONTEXT_CONTROL
             invoke GetThreadContext, pi.hThread, addr context
             or context.regFlag,100h
             invoke SetThreadContext,pi.hThread, addr context
             invoke ContinueDebugEvent, DBEvent.dwProcessId, DBEvent.dwThreadId, DBG_CONTINUE
             .continue
          .elseif DBEvent.u.Exception.pExceptionRecord.ExceptionCode==EXCEPTION_SINGLE_STEP
             inc TotalInstruction
             invoke GetThreadContext,pi.hThread,addr context or context.regFlag,100h
             invoke SetThreadContext,pi.hThread, addr context
             invoke ContinueDebugEvent, DBEvent.dwProcessId, DBEvent.dwThreadId,DBG_CONTINUE
             .continue
          .endif
       .endif
       invoke ContinueDebugEvent, DBEvent.dwProcessId, DBEvent.dwThreadId, DBG_EXCEPTION_NOT_HANDLED
    .endw
.endif
invoke CloseHandle,pi.hProcess
invoke CloseHandle,pi.hThread
invoke ExitProcess, 0
end start

Analysis:

The program shows the openfile dialog box. When the user chooses an executable file, it executes the program in single-step mode, couting the number of instructions executed until the debuggee exits.

       .elseif DBEvent.dwDebugEventCode==EXCEPTION_DEBUG_EVENT           .if DBEvent.u.Exception.pExceptionRecord.ExceptionCode==EXCEPTION_BREAKPOINT

We take this opportunity to set the debuggee into single-step mode. Remember that Windows sends an EXCEPTION_BREAKPOINT just before it executes the first instruction of the debuggee.

             mov context.ContextFlags, CONTEXT_CONTROL
             invoke GetThreadContext, pi.hThread, addr context

We call GetThreadContext to fill the CONTEXT structure with the current values in the registers of the debuggee. More specifically, we need the current value of the flag register.

             or context.regFlag,100h

We set the trap bit (8th bit) in the flag register image.

             invoke SetThreadContext,pi.hThread, addr context
             invoke ContinueDebugEvent, DBEvent.dwProcessId, DBEvent.dwThreadId, DBG_CONTINUE
             .continue

Then we call SetThreadContext to overwrite the values in the CONTEXT structure with the new one(s) and call ContinueDebugEvent with DBG_CONTINUE flag to resume the debuggee.

          .elseif DBEvent.u.Exception.pExceptionRecord.ExceptionCode==EXCEPTION_SINGLE_STEP
             inc TotalInstruction

When an instruction is executed in the debuggee, we receive an EXCEPTION_DEBUG_EVENT. We must examine the value of u.Exception.pExceptionRecord.ExceptionCode. If the value is EXCEPTION_SINGLE_STEP, then this debug event is generated because of the single-step mode. In this case, we can increment the variable TotalInstruction by one because we know that exactly one instruction was executed in the debuggee.

             invoke GetThreadContext,pi.hThread,addr context or context.regFlag,100h
             invoke SetThreadContext,pi.hThread, addr context
             invoke ContinueDebugEvent, DBEvent.dwProcessId, DBEvent.dwThreadId,DBG_CONTINUE
             .continue

Since the trap flag is cleared after the debug exception is generated, we must set the trap flag again if we want to continue in single-step mode.
Warning: Don't use the example in this tutorial with a large program: tracing is SLOW. You may have to wait for ten minutes before you can close the debuggee.