Analyze -v

I had an interesting crash on a system running Process Monitor from Sysinternals that implicated the Process Monitor driver in a NULL pointer dereference:

Here we see what appears to be an image load callback (as indicated by the call to PsCallImageNotifyRoutines on the stack) that is dereferencing a NULL pointer value contained in EDI.

Zooming into the assembly of the routine in question, we can see that the bad pointer value of EDI is retrieved a couple of lines up from EBP+8. Since we also can see that this is a standard EBP frame in the prolog, we can assume at this point that the NULL pointer deref occurred due to a failure to check the first parameter for NULL:

At this point I went back to the PsSetLoadImageNotifyRoutine documentation to check the prototype of the callback function, hoping to found out what the first parameter of this routine was and if it was documented to be NULL. According to the documentation, the following is the prototype of the function:

VOID
(*PLOAD_IMAGE_NOTIFY_ROUTINE) (
IN PUNICODE_STRING  FullImageName,
IN HANDLE  ProcessId, // where image is mapped
IN PIMAGE_INFO  ImageInfo
);

Note that the first parameter is indicated to be a pointer to a UNICODE_STRING structure. This fits with the assembly lines listed above:

mov edi, dword ptr [ebp+8]

xor eax, eax

cmp word ptr [edi], ax

Since that assembly listing would match the following C code due to the fact that the first field of a UNICODE_STRING structure is an USHORT Length value:

if (FullImageName->Length == 0)

Thus, this routine assumes that if there is no image name available for the module being loaded, the first parameter to this routine will be a pointer to an empty UNICODE_STRING. Clearly in my case the first parameter passed in was NULL, thus I had to dig further back to determine if I was dealing with a corruption or if Process Monitor’s image load callback had a genuine bug.

Moving up a couple of frames, I located the code that gathers the name for the image being loaded and calls the dispatcher routine. Below is the highlighted assembly path that is of interest to us:

The highlighted sections correspond to the following four steps that ultimately lead to the crash:

1) Set the value of EBX to be zero, or NULL

2) Call an internal memory manager routine, MmGetFileNameForSection. While I’m not familiar with this routine, I suspect that it tries to get the file name of a section Upon return, compare the return value of the routine (EAX) with EBX, which we know to be zero.

3) Based on the compare, we will “jump if less than zero.” This assembly pattern basically checks a value to determine if it is less than zero and, if it is, jumps to a new location. Failure codes in the kernel are negative values, thus this is a fairly common pattern that corresponds to the following C statement:

status = SomeFunction();

if (!NT_SUCCESS(status)) {

}

Thus, if MmGetFileNameForSection returns a negative value (i.e. a failure code) we will jump.

4) The final highlighted section is the failure path that we take if MmGetFileNameForSection returns a failure code. In that case, a push of EBX is performed. This means that we will pass NULL as the first parameter to PsCallImageNotifyRoutines and not a pointer to an empty UNICODE_STRING structure.

So, after all of that digging, the result is that this is a legitimate bug in the Process Monitor image load callback. The system that this crash occurred on had several other third party drivers present including two security products, thus I suspect that someone failed the attempt to get the image name for some reason or another. This triggered an untested path in the Process Monitor driver and led to the crash.

Note: If you try to follow the above assembly on your own system you might find different routines in play here as this code was rewritten for Vista and later. However, the bug is still real as the first parameter to the image load callback may be NULL on those platforms as well.

Driver Verifier has a Special Pool option, which causes your pool allocations to come out of special pool and get all sorts of added checking. This includes things such as guard pages at the end of your allocations to avoid buffer overruns, checks against accessing buffers after you free them, etc. Unfortunately the !pool command in WinDBG appears to be broken when given a special pool address on Windows XP. I haven’t had the chance to investigate further on newer O/S platforms, so it’s possible that there are also some issues there.

The extension command appears to have two issues:

1) The size shown as the allocation size is really the allocation size minus 8. Take for example a verified driver that does the following:

	a = ExAllocatePoolWithTag(NonPagedPool, 4, 'xxxx');
       	b = ExAllocatePoolWithTag(NonPagedPool, 8, 'xxxx');
       	c = ExAllocatePoolWithTag(NonPagedPool, 16, 'xxxx');

A !pool on a, b, and c shows the following (respectively):

1: kd> !pool 0x82a5aff8
Pool page 82a5aff8 region is Special pool
*82a5b000 size: fffffffc non-paged special pool, Tag is xxxx
	Owning component : Unknown (update pooltag.txt)
1: kd> !pool 0x824eeff8
Pool page 824eeff8 region is Special pool
*824ef000 size:    0 non-paged special pool, Tag is xxxx
	Owning component : Unknown (update pooltag.txt)
1: kd> !pool 0x82940ff0
Pool page 82940ff0 region is Special pool
*82940ff8 size:    8 non-paged special pool, Tag is xxxx
	Owning component : Unknown (update pooltag.txt)

2) The pool header addresses are incorrect. Take a in the example above:

1: kd> !pool 0x82a5aff8
Pool page 82a5aff8 region is Special pool
*82a5b000

That address given as the pool header is actually the guard page and not the header:

1: kd> dt nt!_pool_header 82a5b000
	+0x000 PreviousSize     : ??
	+0x000 PoolIndex        : ??
	+0x002 BlockSize        : ??
	+0x002 PoolType         : ??
	+0x000 Ulong1           : ??
	+0x004 ProcessBilled    : ????
	+0x004 PoolTag          : ??
	+0x004 AllocatorBackTraceIndex : ??
	+0x006 PoolTagHash      : ??
	Memory read error 82a5b006

The pool header in this case is the page rounded down to page size, not up to:

1: kd> dt nt!_pool_header 0x82a5a000
	+0x000 PreviousSize     : 0y000000100 (0x4)
	+0x000 PoolIndex        : 0y0100000 (0x20)
	+0x002 BlockSize        : 0y001010001 (0x51)
	+0x002 PoolType         : 0y0000000 (0)
	+0x000 Ulong1           : 0x514004
	+0x004 ProcessBilled    : 0x78787878 _EPROCESS
	+0x004 PoolTag          : 0x78787878
	+0x004 AllocatorBackTraceIndex : 0x7878
	+0x006 PoolTagHash      : 0x7878

As you can see, the tag is the correct “xxxx” that we specified in the allocation:

1: kd> .formats 0x78787878
	Evaluate expression:
	Hex:     78787878
	Decimal: 2021161080
	Octal:   17036074170
	Binary:  01111000 01111000 01111000 01111000
	Chars:   xxxx
	Time:    Tue Jan 17 20:38:00 2034
	Float:   low 2.01583e+034 high 0
	Double:  9.98586e-315

One day I wanted to demonstrate the !pte command on a page that had been paged out to a paging file. The platform in question was an XP SP3 dump and I was slightly on the spot, so I had to quickly think of a way to find a page that was actually paged out.

I immediately thought to crawl through the paged pool range in the kernel virtual address space, running !pte on each page. I could do this with something like .for or simply manually walk every 4K until I hit a paged out page.

Because I was feeling lucky (and didn’t want to look up the .for syntax), I decided to try manually walking but quickly found that tedious and I didn’t get the results that I was looking for. Everything I hit was paged in, as indicated by the valid bit being set:

So, I needed something else that might provide a quicker solution. Then I thought about paged code sections in a driver image. By default, all driver code (and data) is non-pageable. However, a driver writer is able to use a pragma directive to put code into the PAGE section of its image.

Code in the PAGE section of a driver is subject to being paged out. In addition, the memory manager pages this code to a paging file instead of retrieving it again from the image file (like in most user mode application scenarios). You can see an example of the pragma usage in the FASTFAT source from the WDK:

Being reminded of this, I decided to grab a Microsoft supplied driver and check for a PAGE section in the PE header, since MS supplied drivers often utilize this to decrease their memory footprint in the system. For no particular reason, I went with fltmgr.sys, which is the binary for Filter Manager. This did indeed reveal a code section named PAGE in the image:

This gave me a new, smaller space to search for a page paged out to a paging file. So, I gave it a shot and lucked out immediately:

You might have noticed that WinDBG sometimes “knows” where to find the PDB for your driver even if you have never set your Symbol Search Path to point to your object directory. Spooky, and it certainly begs the question of how WinDBG knows where to look for your PDB.

The answer is in the debug directory of your driver’s PE image. The debug directory can contain, well, debug information, or can point off to a file containing debug information. This is how WinDBG can so easily find your PDB, by default the fully qualified path (FQP) to the PDB is found in the debug directory. If you want to see the path, you can do so with the !dh command, which will parse the PE header of the image and show you the intepreted results. Just execute !dh modulename in the debugger and scroll down to the Debug Directories section:

You can even leverage this to find information about drivers that you don’t own. I just checked the loaded module list on my current machine and had a driver loaded that I didn’t recognize: NETw4x32.sys. Doing a !dh on that module tells me that the FQP is:

So, it’s the free build of an NDIS 5.x miniport driver, which isn’t bad for information gleaned from two seconds of effort.

The problem with those FQPs is that they can contain information that you don’t want to share with the world. Build machine names, developer names, etc. may show up in the path and you might not want people outside your company to have access to that info. Luckily, there is a way to strip the path information out of your image files and that is exactly what Microsoft does with their binaries. For example, take the kernel on my system using the info from !dh nt:

The secret to making this work is the binplace tool, which exists mostly to move binary outputs of the build process to various locations. However, it has the added features of generating public PDB files as well as stripping the path infromation from your released image files.

Getting binplace integrated to your build is another post for another day, but details can be found on MSDN if you’re interested: http://msdn.microsoft.com/en-us/library/ms791453.aspx