Analyze -v

Happy Valentine’s Day! And what’s more romantic than a post about analyzing an x64 crash dump? If you haven’t picked up a card already, feel free to print this out and hand it to your significant other.

Way back in December, we started looking at the fundamentals of x64 crash analysis so that we could work up to analyzing an actual x64 crash. If you haven’t already, I suggest that you read them in order since we’ll put all of those posts in practice here:

x64 Trap Frames

x64 Calling Convention

x64 Stack Frame layout

Reconstructing parameters from x64 crash dumps

With that out of the way, we can start our analysis the way we always do with any crash by running !analyze -v:

The bugcheck code in this case is PAGE_FAULT_IN_NONPAGED_AREA (0×50). In order to solve this crash we should probably cover what exactly this crash code is indicating.

The kernel virtual address space in Windows contains lots of different memory regions that serve different purposes. Regardless of the purpose of the region, one characteristic that all kernel memory has is whether or not the memory is pageable. When a memory region is pageable, the Memory Manager (Mm) is free to take the contents of the physical page of memory, write it out to disk, and then invalidate the virtual address. The next time someone tries to read the contents of that address, a page fault occurs and the contents are brought back into memory from disk. Once the memory is again resident, the Mm fixes the virtual address pointer and resumes the thread. When a memory region is non-pageable, it means that the Mm promises to never page out the memory and invalidate the virtual address in this manner.

Having non-pageable memory is important in Windows because it is the only kind of memory that you are allowed to access at IRQL DISPATCH_LEVEL or above (see my previous post here for more on IRQL). The reason for this is that you are not allowed to perform any wait operations at IRQL DISPATCH_LEVEL or above and by using pageable memory you’re implicitly stating that you can wait if the memory that you’re trying to access is not resident in memory.

With that out of the way, we can understand what the particular bugcheck code means. These non-pageable regions are only guaranteed to be valid if you have a valid outstanding resource allocation from the Mm. Take, for example, non-paged pool, which is the kernel equivalent to the user mode heap with the exception that memory allocated from this pool is guaranteed to never be paged out to disk. However, that does not mean that every address within the non-paged pool area is valid at all times. The Mm may delay programming a particular virtual address in this region until he is going to return a pool allocation to a particular caller, or mark the virtual address as invalid when someone frees a valid pool allocation. If someone tries to access one of these invalid addresses, a page fault will occur and the Mm will inspect the invalid virtual address to decide what needs to be done with it. If this virtual address corresponds to a region that is guaranteed to not page fault when valid, the Mm calls KeBugCheck with a bugcheck code of PAGE_FAULT_IN_NONPAGED_AREA. If you think about it, this is the only reasonable thing that the Mm can do since there is no solution to this state (you can argue that there are things that could be done, but that’s in the realm of fault tolerant systems and not relevant to the discussion).

We can now break down the text associated with the bugcheck code and understand a bit more of what it means:

Invalid system memory was referenced.  This cannot be protected by try-except, it must be protected by a Probe.  Typically the address is just plain bad or it is pointing at freed memory.

Invalid system memory was referenced.

This bugcheck is always the result of dereferencing a bad kernel virtual address.

This cannot be protected by try-except, it must be protected by a Probe.

If you dereference a bad user virtual address, a structured exception is raised that your driver can catch in a structured exception handling (SEH) block. When an invalid kernel address is accessed, there is no structured exception raised and the system simply bugchecks. The comment here about being protected by a probe is misleading. There is no way to validate a kernel address other than dereferencing it and hoping for the best. The idea is that kernel callers are trusted, thus if a kernel component hands you a kernel virtual address you must assume that it is valid. What the comment here is referring to is that you should not be touching kernel virtual addresses that originated from user mode. You can avoid this situation by calling ProbeForRead or ProbeForWrite on any address handed to you from user mode, which will raise an exception if the address is a kernel virtual address. This is only useful if you’re performing METHOD_NEITHER I/O and is not relevant to our conversation.

Typically the address is just plain bad or it is pointing at freed memory.

This is the short version of what we’ve been talking about up until now. If you have page faulted on an address in the non-paged area it means that you have dereferenced something that is not a valid memory allocation. Generally this means that it’s a garbage value (e.g. uninitialized pointer), freed memory, or some kind of corruption (e.g. a pointer value from a corrupted data structure).

Now we should have a much better idea as to what we’re looking at and the kind of bug that we’re looking for. According to the !analyze output, the invalid address that caused the wreck was 0xffffba80`07122a88 and we were attempting to write to the address (parameter 2 from the bugcheck information). Let’s look at the trap frame output and attempt to identify the invalid address in the faulting instruction:

If you haven’t read my previous post about x64 trap frames, the output above will likely be confusing. There are two pointer dereferences in the above output, the instruction pointer RIP and RBX. Neither one of these values is 0xffffba80`07122a88 and, in fact, this looks to be a NULL pointer dereference since RBX is zero. However, as we know, the trap frames on the x64 do not contain non-volatile register state and RBX is a non-volatile register. So, in order to get the value of RBX at the time of the crash back we’ll need to scroll back through the assembly and find another volatile register that either shadows RBX in this frame or we can use to derive RBX.

The first step to getting this information will be to execute the .trap command and get ourselves into the correct trap frame. We don’t need to find the trap frame address ourselves since it was already present in the !analyze output:

Now that our registers are back, we can go back through the disassembly and figure out where RBX came from prior to the dump. I generally do this by bringing up the disassembly window (Alt+7) since I find that a bit more convenient in this situation than trying to use the keyboard shortcuts to navigate the assembly. Bringing up the window and scrolling back a bit shows RBX coming from RDX a few instructions earlier:

If we go back and view the RDX register value contents, we’ll see that they match the pointer value from the first parameter to the bugcheck:

We could have intuited the value of RBX based on the bugcheck information and our knowledge of x64 trap frames, though I like to take this extra step to make sure that I understand these things and also get my bearings with the dump. Also, in this case I know that the RBX value came from RDX, which is likely the second parameter to the function. Thus, if I can find a function prototype I can know what the type of the structure should be. This all provides me with greater context for the dump and a greater chance that I’ll have some success in analyzing it.

Let’s review what we have up to this point:

1) Someone has tried to write to an invalid system address

2) The address was passed as the second parameter to this routine

The next logical step that we need to explore is, “what kind of address is this supposed to be and why is this address invalid?”

For the first part of this question, we can typically use the !address extension and figure out what region this address lies in. Unfortunately, at the time of this writing that command does not work on Windows 7 and determining this without that extension is beyond the scope of this article. There is one thing we can quickly check though and that is whether or not this address lies in one of the various executive pools, which we can do with the !pool command:

Based on that output, it’s likely that this is not a pool allocation. The fact that it states that it is corrupt or free pool doesn’t really mean much, it should really say, “I have no idea what this is, but it doesn’t look like pool to me.” While that doesn’t provide us much positive information, we at least know that it’s not likely a bad pool address due to being used after it was freed. This at least removes a class of bugs and narrows our search a bit.

Since it’s not pool and !address doesn’t work, the best we can now is inspect the PTE contents and see if there’s anything interesting there:

According to the !pte output, this address is not only bad it’s really bad. At no level of the page translation process does this address have valid information. To me, this screams that this address is the result of an uninitialized pointer reference or a data corruption. Since the crash occurs in a Microsoft supplied component, the fact that this crash would come from an uninitialized pointer is very unlikely and so my sights are set firmly on some type of corruption. But what kind?

When I first analyzed this dump, I stayed at this point for almost 24 hours (luckily not straight!). I just couldn’t see what the corruption was or where it came from. I spent the time to go through every other thread in the system and even searched the kernel address space for other references to this address hoping for some light to shine on a clue. Since this was a Filter Manager structure that was corrupt, I also checked all of the current Filter Manager mini-filters with the !fltkd.filters command and was bummed to find only in box Microsoft supplied filters running.

At this point I took the advice that I give to all students: I walked away from the dump. Anyone who claims that crash dump analysis isn’t difficult is either lying or doesn’t get presented with many challenging dumps. Sometimes walking away from the dump gives you the fresh perspective and eyes that you need to spot something miniscule, such as a missing bit.

Before calling it quits on the dump, I gave it one last look and noticed something curious about the faulting pointer value when compared to the other values in the trap frame: only the high four bytes at the top of the address were 0xf, not the high five bytes like the other registers.

I felt like Archimedes in the bathtub, though I expressed my excitement in a slightly less dramatic fashion (and without all of the nudity). What if this was a single bit flip error? It’s possible that due to some sort of cosmic error there was a single bit in this address that should have come back as 1 but instead read as 0. So, I flipped the bit and found what was an entirely plausible pool address that indicated it was a valid Filter Manager allocation:

This gave me a plausible explanation for the dump: hardware failure. I wanted to collect at least one other piece of information to support this, so I decided to check to see if there were any physical pages marked as bad in this system. This this wouldn’t provide any type of solution, it would at least be another indicator that this machine was having hardware related issues. Thus, I inspected the state of the Page Frame Database in this machine using the !memusage command and did indeed find ~7MB of bad pages:

We’ve been working on a series of x64 related crashes so that we can work up to analyzing a crash dump from a real world system. In this post I’m going to demonstrate the techniques you can use to find the actual parameters passed to a function when working with an x64 crash. As mentioned previously, this can be tricky due to the x64 compiler’s use of volatile registers for parameter passing. So we’re going to have to rely on the parameters being stored into other registers or on the execution stack.

Let’s use the dump that we’re eventually going to analyze and get the parameters to the faulting subroutine back. Here’s a shot of the faulting instruction in the faulting module:

The faulting intruction is the second arrow and the first arrow shows the bad pointer value being loaded from RDX into RBX. The analysis will later show that this bugcheck is a result of a bad pointer dereference via the RBX register. As we know from our discussion of calling conventions, RDX is the second parameter to this function thus presumably this function was passed a bad pointer value as the second parameter.

Googling the function name gives us the function prototype for RtlDeleteNoSplay:

VOID
  RtlDeleteNoSplay(
    IN PRTL_SPLAY_LINKS  Links,
    IN OUT PRTL_SPLAY_LINKS *Root
    ); 

What I would like to do now is reconstruct the two parameters to the function so that I can perform further analysis. Because this can range from a simple task to something quite annoying, I’m going to demonstrate reconstructing the first parameter to this routine using three different levels of difficultly. This is always going to of course be very situation specific, but hopefully the three techniques shown here will come in handy.

EASY

In this routine I get lucky in finding the first parameter, as RCX is loaded into the R11 register before the crash:

Since R11 is a volatile register, I can trust the value of R11 in the trap frame that I have so I can inspect the first parameter by looking in R11:

 But, what if RCX hadn’t been stored into R11 before the crash? Then we’d have to dig a little deeper and try to find the parameter in a previous frame. I like to do this as a practice exercise when I can, I already have the result that I know I have to get to so I can easily validate my result from going the hard way.

MEDIUM

So, how do I find the first parameter to this function without using R11? Well, I know that the previous function in the stack must have loaded a value into RCX to set up for this frame. So, I’ll start by walking up the stack to the previous function, which in this case is TreeUnlinkNoBalance:

You’ll note here that RCX is stored in RBX before calling into RtlDeleteNoSplay. Now I know that upon entry to RtlDeleteNoSplay RCX was equal to RBX, so that gives me another register I can track inside RtlDeleteNoSplay in the hopes of recontructing the first parameter. So, let’s look at RtlDeleteNoSplay again:

Notice the first instruction of RtlDeleteNoSplay, it’s a push of RBX onto the stack! That means that the value of the first parameter to this routine was preserved on the execution stack. The RSP value in the trap frame will be that of the stack pointer at the time of the crash, thus in order to get the value on the stack back I will need to unwind any stack manipulations done after the save of the value on the stack. In this case, I simply have to add 0×20 to the stack pointer to undo the subtraction of 0×20 from RSP after the push:

HARD

But what if we had to go even further up in the stack to find a parameter? Would we be able to get a parameter to a function back? Let’s give it a shot!

For this one I’m doing to use a different routine with different parameters, since it’s further up in the stack and that will complicate our analysis.

Earlier in the stack PurgeStreamNameCache calls DeleteNameCacheNodes:

Note that the routine sets up RCX by loading it with the value of RSI. Thus, if I want the first parameter back when I’m inside DeleteNameCacheNodes I can look for either RCX or RSI being loaded into another register or, even better, saved on the stack:

Immediately at the beginning of the function we see DeleteNameCacheNodes storing RSI into the R9 home space on the stack. This is great news for us, as it means that we can get this parameter back by plucking it off of the stack location. But, how can we figure out what stack pointer value to use?

This is the key trick to reconstructing parameters on the x64. It’s quite simple actually, all we need to do is execute the k command and get the Child-SP value for the frame that we’re interested in:

That is the value the stack pointer will be upon return from the current call that subroutine is making. That’s a bit confusing, but what I mean is that in this case we see DeleteNameCacheNodes calling TreeUnlinkMulti. The Child-SP indicates what RSP will be when that call returns. This makes it very easy to find anything on the stack that we want in DeleteNameCacheNodes since all we need to do is start unwinding the stack changes that were made in the prolog until we hit the value that we want:

(Note: Picture above fixed on 1/15 based on comments from Anonymous. See comments section of the post.)

Now that we have the correct values, we can just dump out that memory location with the dq command:

As a final tip, in this can I could have quickly found this value by leveraging the fact that the Args to Child in the kv output shows the four home space values from the stack. Since RSI was put in the R9 home space (RSP+0×20), I could have just done a kv and pulled the fourth value from the resulting output in the DeleteNameCacheNodes entry.

Just posted a nifty new WinDBG extension on OSR Online. It provides quick access to all of the WinDBG commands that you’ve used in your current debug session:

http://www.osronline.com/article.cfm?article=547

It’s something that I’ve wanted for a while, so I finally decided to just shut up and code it!

I was over on the Sysinternals forums and analyzed a crash dump for a member there. The dump turned out to be fascinating and also reminded me just how painful debugging x64 crashes can be…

I was so interested by the crash that I was going to just write up an analysis and dump it here (no pun intended), but I think I’m going to take a bit more of a scientific approach and tackle a couple of important x64 crash dump details/techniques before diving in to the crash. Hopefully that will lead to a more meaningful analysis and help everyone out there that has to work with x64 crashes (which will be all of you eventually!).