x64 Calling Convention

We’re working up to analyzing an interesting crash by learning more about working with the x64…

In order to work with x64 dumps we’re going to need to understand the calling convention used, that is going to allow us to do things such as identify the parameters passed to a particular function.

The basic rule here is that the first four parameters to a function are passed in registers, with the remaining parameters to the routine passed on the stack. The typical registers used here are:

Parameter 1 - RCX

Parameter2 - RDX

Parameter 3 - R8

Parameter 4 - R9

(NOTE: There are special rules when it comes to things such as floating point operations. Full details for those scenarios can be found here.)

For example, if a routine begins by accessing the contents of the RDX register, we can know that this routine is accessing the second parameter. We can also imply from that the fact that the caller must have loaded RDX with a meaningful value in a previous frame:

rdx

There is a major issue with this convention however. And that is the fact that all four of these parameters are treated as volatile by the compiler, meaning that their contents do not need to be saved across subroutine calls. Thus it is entirely possible and in fact quite likely that the compiler will overwrite the contents of these registers with unrelated values over the course of the subroutine. This makes reconstructing parameter values quite difficult on the x64.

The next x64 post will talk about the unique way that the x64 compiler utilizes the execution stack, which will then lead to a more detailed discussion on how we can utilize the stack to get parameter information back when we need it.

2 Responses to “x64 Calling Convention”

  1. Analyze -v says:

    [...] x64 Calling Convention [...]

Leave a Reply