Analyze -v

It’s half-time in the New England Patriots game so now’s a good a time as ever to stack about the x64 compiler’s usage of the execution stack.

The x64 compiler on Windows performs all stack manipulation during the function prolog. This includes saving non-volatile registers on the stack, making room for local variables, and even making room for parameters passed to the functions that it calls (remember that the first four parameters are passed in registers with any remaining parameters passed on the stack).

This leads to another interesting feature of the x64 compiler’s use of the stack. All subroutines are guaranteed to have a “spill” or “home” space allocated for them on the stack by the caller. This space is large enough to house the first four parameters passed to the routine via registers: RCX, RDX, R8, and R9. In addition, this space comes before the space where the fifth parameter can be found.

A picture probably clears this up. Upon entry to a subroutine, RSP points to a location that contains the return address of the routine. RSP+8 points to the RCX home space, RSP+10 is the RDX home space, etc:

It is important to realize however that these locations on the stack are not reserved to just hold RCX, RDX, R8, and R9. In some cases, the prolog of the function may “home” all of the parameter registers into these locations. In other cases, the prolog may “home” a subset. In other cases, the prolog may store completely unrelated register values here. Or the compiler may choose to ignore this space and not store anything here. It is effectively just a scratch space for the subroutine to use in whatever way it sees fit.

We’re going to extensively utilize the scatch space in any x64 crash that we analyze since it’s a great place to find parameters passed to functions and non-volatile registers that aren’t present in trap frames. Next up we’ll see just how we can use the space to easily reconstruct the parameter passed to a function.

(Just finished typing in time for kick off!)

Update

Anonymous makes a really interesting point in the comments:

It’s important to get this right – even though stack manipulations are performed during prolog, RSP itself _can_ change during execution of function body. For example, due to “_alloca” call (other cases, anybody?).

This isn’t a common programming practice in kernel mode, so I failed to mention that. If anyone else knows of any cases please let us know!

This entry was posted on Sunday, December 20th, 2009 at 3:37 pm and is filed under x64. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.