Implementing breakpoints in Rust

In this post I will continue working on writing a debugger crate in Rust called rusty_trap. We’ll be implementing support for breakpoints by writing INT 3 instructions into the inferior. I’ll also explore building up a proper API for the debugger crate. This will all mirror the work done implementing breakpoints in C.

This post is part my larger series on writing a debugger in C and Rust. If you haven’t read the earlier posts you may want to go back and start from the beginning.

Also, this post is shorter than posts I’ve written in the past. I’m going to experiment with writing shorter posts more often. So expect to see a post on “A System Programming Blog” once a week rather than twice a month.

Debugger API in Rust

For the C version of the debugger library we established a simple API for managing inferior processes and setting breakpoints. This inciuded a callback used when breakpoints are triggered. We drove development of this API using a test. We’ll do the same in Rust. I’ve made up a test that uses an API I would like to have:

#[test]
fn it_can_set_breakpoints () {
    let mut bp: rusty_trap::TrapBreakpoint = 0;
    let mut breakpoint_count: i32 = 0;
    let mut inferior = 0;

    rusty_trap::trap_set_breakpoint_callback(|passed_inferior, passed_bp| {
        assert_eq!(passed_inferior, inferior);
        assert_eq!(passed_bp, bp);
        breakpoint_count += 1;
    }).unwrap();

    let inferior = rusty_trap::trap_inferior_exec("./target/debug/twelve", &[]).unwrap();
    bp = rusty_trap::trap_inferior_set_breakpoint(inferior, "main");
    rusty_trap::trap_inferior_continue(inferior);

    assert_eq!(breakpoint_count, 1);
}

I like this test as an API proposal; I like the closure used as the breakpoint callback. But, I need to understand how Rust closures really works. Specifically, how does ownership for breakpoint_count work in this case, if at all? I took some time to read through the chapter on closures in the Rust Book. The book doesn’t really address the case I have so I think some experimentation is in order.

First I write the trap_set_breakpoint_callback function, or at least a stub it estabilish the type signature:

pub fn trap_set_breakpoint_callback(callback: &Fn(TrapInferior, TrapBreakpoint) -> ()) -> Result<(), i32> {
    Ok(())
}

When trying to compile the tests one of the errors I get is:

tests/lib.rs:18:9: 18:30. error: cannot assign to data in a captured outer variable in an `Fn` closure
tests/lib.rs:18         breakpoint_count += 1;
                        ^~~~~~~~~~~~~~~~~~~~~
note: in expansion of closure expansion
tests/lib.rs:15:47: 19:6 note: expansion site
tests/lib.rs:15:47: 19:6 help: consider changing this closure to take self by mutable reference

So, I need to make the closure mutable. I can’t fathom the syntax to do that when passing a closure literal to trap_set_breakpoint_callback so maybe I need to bind the closure to a mutable variable like this:

let mut callback = |passed_inferior, passed_bp| {
    assert_eq!(passed_inferior, inferior);
    assert_eq!(passed_bp, bp);
    breakpoint_count += 1;
};

rusty_trap::trap_set_breakpoint_callback(&callback).unwrap();

This still fails to compile:

tests/lib.rs:21:46: 21:55 error: the trait `core::ops::Fn<(i32, i32)>` is not implemented for the type `[closure tests/lib.rs:15:24: 19:6]` [E0277]
tests/lib.rs:21     rusty_trap::trap_set_breakpoint_callback(&callback).unwrap();
                                                             ^~~~~~~~~
error: aborting due to previous error

Interestingly, if I comment out the increment of breakpoint_count like this:

let mut callback = |passed_inferior : rusty_trap::TrapInferior, passed_bp : rusty_trap::TrapBreakpoint| {
      assert_eq!(passed_inferior, inferior);
      assert_eq!(passed_bp, bp);
      //breakpoint_count += 1;
  };

then the test compiles. This was my original concern, that I have to give ownership of breakpoint_count to rusty_trap. After some trial and error and some reading I realized that the type used in the signature of trap_set_breakpoint_callback is wrong. To make this work I need to use FnMut instead of Fn like this:

callback: &FnMut(TrapInferior, TrapBreakpoint) -> ()

I’ve also made a change to get rid of trap_set_breakpoint_callback and instead I plan to pass the callback to trap_inferior_continue. I think this better matches the expected lifetime for the callback and reflects the period in which breakpoint_count will be borrowed by rusty_trap. So now, the whole test looks like this:

#[test]
fn it_can_set_breakpoints () {
    let mut breakpoint_count: i32 = 0;

    let inferior = rusty_trap::trap_inferior_exec("./target/debug/twelve", &[]).unwrap();
    let bp = rusty_trap::trap_inferior_set_breakpoint(inferior, "main");
    rusty_trap::trap_inferior_continue(inferior, &|passed_inferior, passed_bp| {
        assert_eq!(passed_inferior, inferior);
        assert_eq!(passed_bp, bp);
        breakpoint_count += 1;
    });

    assert_eq!(breakpoint_count, 1);
}

Now, this change breaks our previous test as it now needs to pass a callback to trap_inferior_continue. So we update it:

#[test]
fn it_can_exec () {
    let inferior = rusty_trap::trap_inferior_exec("./target/debug/twelve", &[]).unwrap();
    assert_eq!(12, rusty_trap::trap_inferior_continue(inferior, &|_, _| {}));
}

Implementation

At this point the interfaces and tests are coming together. Next, I need to implement trap_inferior_set_breakpoint and add support to trigger the callback from trap_inferior_continue.

I updated trap_inferior_continue to loop and call the calback:

pub fn trap_inferior_continue(inferior: TrapInferior,
                              callback: &FnMut(TrapInferior, TrapBreakpoint) -> ()) -> i8 {
    let pid = inferior;

    loop {
        ptrace(PTRACE_CONT, pid, ptr::null_mut(), ptr::null_mut())
            .ok()
            .expect("Failed PTRACE_CONTINUE");

        match waitpid(pid, None) {
            Ok(WaitStatus::Exited(_pid, code)) => return code,
            Ok(WaitStatus::Stopped(_pid, signal::SIGTRAP)) => callback(inferior, 0),
            Ok(_) => panic!("Unexpected stop in trap_inferior_continue"),
            Err(_) => panic!("Unhandled error in trap_inferior_continue")
        }
    }
}

but this won’t compile:

src/lib.rs:67:63: 67:71 error: cannot borrow immutable borrowed content `*callback` as mutable
src/lib.rs:67             Ok(WaitStatus::Stopped(_pid, signal::SIGTRAP)) => callback(inferior, 0),
                                                                            ^~~~~~~~
error: aborting due to previous error
src/lib.rs:67:63: 67:71 error: cannot borrow immutable borrowed content `*callback` as mutable
src/lib.rs:67Build failed, waiting for other jobs to finish...
             Ok(WaitStatus::Stopped(_pid, signal::SIGTRAP)) => callback(inferior, 0),
                                                                            ^~~~~~~~
error: aborting due to previous error
Could not compile `rusty_trap`.

I understand the concept behind the error, if the original content is immutable I can’t try to mutate it. But, why is my borrow mutable in the first place? I’m just calling the closure, isn’t that what it’s for? After experimenting a bit and reading up on stack overflow I arrive at this:

pub fn trap_inferior_continue<F>(inferior: TrapInferior, mut callback: F) -> i8
    where F: FnMut(TrapInferior, TrapBreakpoint) -> ();

The semantic differences are that callback is mutable, this is the biggest piece I was missing, and it is no longer a reference. Syntactically, I also switched to using where.

Peeking and Poking

I write up the following implemenation of trap_inferior_set_breakpoint which is similar to the C version:

pub fn trap_inferior_set_breakpoint(inferior: TrapInferior, location: usize) -> TrapBreakpoint {
    let target_address = location as * mut c_void;
    let aligned_address = location & !0x7usize;

    let original = ptrace(PTRACE_PEEKTEXT, inferior, aligned_address as * mut c_void, ptr::null_mut())
        .ok()
        .expect("Failed PTRACE_PEEKTEXT");

    let shift = (location - aligned_address) * 8;
    let mut modified = original as usize;
    modified &= !0xFFusize << shift;
    modified |= 0xCCusize << shift;

    ptrace(PTRACE_POKETEXT, inferior, target_address, modified as * mut c_void)
        .ok()
        .expect("Failed PTRACE_POKETEXT");

    0
}

The goal here is to write 0xCC which is the opcode for INT 3 into the inferior at location. Because PTRACE_PEEKTEXT and PTRACE_POKETEXT only read and write aligned words from the inferio so we have to fiddle the bits around to put the INT 3 at the right spot.

I needed to update the test with the right value for location. I’ve replaced "main" with 0x5040:

#[test]
fn it_can_set_breakpoints () {
    let mut breakpoint_count: i32 = 0;

    let inferior = rusty_trap::trap_inferior_exec("./target/debug/twelve", &[]).unwrap();
    let bp = rusty_trap::trap_inferior_set_breakpoint(inferior, 0x5040);
    rusty_trap::trap_inferior_continue(inferior, |passed_inferior, passed_bp| {
        assert_eq!(passed_inferior, inferior);
        assert_eq!(passed_bp, bp);
        breakpoint_count += 1;
    });

    assert_eq!(breakpoint_count, 1);
}

The number 0x5040 comes from:

$ objdump -x target/debug/twelve | grep main
0000000000005000 l     F .text	0000000000000033              _ZN4main20hce5d2c3b1c46c004faaE
0000000000000000       F *UND*	0000000000000000              __libc_start_main@@GLIBC_2.2.5
0000000000005040 g     F .text	0000000000000154              main

When I ran the test it failed the PTRACE_PEEKTEXT request with error EIO. One meaing for error EIO is that the address is bad. So I loaded up the twelve program in gdb:

$ rust-gdb target/debug/twelve
(gdb) b main
Breakpoint 1 at 0x5040
(gdb) r
Starting program: /home/joseph/src/rust/rusty_trap/target/debug/twelve
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Breakpoint 1, 0x0000555555559040 in main ()

So, the executable has been relocated and the addresses don’t match. For our simple tests I just updated the address in the test to 0x0000555555559040:

#[test]
fn it_can_set_breakpoints () {
    let mut breakpoint_count: i32 = 0;

    let inferior = rusty_trap::trap_inferior_exec("./target/debug/twelve", &[]).unwrap();
    let bp = rusty_trap::trap_inferior_set_breakpoint(inferior, 0x0000555555559040);
    rusty_trap::trap_inferior_continue(inferior, |passed_inferior, passed_bp| {
        assert_eq!(passed_inferior, inferior);
        assert_eq!(passed_bp, bp);
        breakpoint_count += 1;
    });

    assert_eq!(breakpoint_count, 1);
}

But this still failed with EIO!

The Linux Kernel has a feature called Address Space Layout Randomization (ASLR) which causes memory mappings to be allocated random address space ranges for security purposes. This means that main will be located at a different address every time we run the inferior. This feature breaks our ability to use these hardcoded addresses in our tests. With the C version of our inferior this wasn’t a problem because our executable was not relocatable (only the libraries it uses were). This is also not an issue for gdb because it disables the randomization when debugging – rusty_trap should do the same.

Disabling ASLR

I wrote up quick-and-dirty code to disable ASLR using the personality Linux system call. I dumped this code directly into rusty_trap for now. Eventually, I should add write a pull request to add personality support to nix.

mod ffi {
    use libc::{c_int, c_long};

    extern {
        pub fn personality(persona: c_long) -> c_int;
    }
}

fn disable_address_space_layout_randomization() -> () {
    unsafe {
        let old = ffi::personality(0xffffffff);
        ffi::personality((old | 0x0040000) as i64);
    }
}

disable_address_space_layout_randomization queries the current personality (using 0xffffffff) and then sets the ADDR_NO_RANDOMIZE bit (0x0040000). I should add some constants for these values. But anyway, I call this from exec_inferior before calling execve. This successfull disables the ramdomization and allows the PTRACE_PEEKTEXT and PTRACE_POKETEXT requests to work!

However, the test still fails:

     Running target/debug/lib-8caa31b4833c2b69

running 2 tests

thread '<main>' has overflowed its stack
test it_can_set_breakpoints ... FAILED
test it_can_exec ... ok

failures:

---- it_can_set_breakpoints stdout ----
	thread 'it_can_set_breakpoints' panicked at 'Unexpected stop in trap_inferior_continue', src/lib.rs:80



failures:
    it_can_set_breakpoints

test result: FAILED. 1 passed; 1 failed; 0 ignored; 0 measured

thread '<main>' panicked at 'Some tests failed', /home/rustbuild/src/rust-buildbot/slave/stable-dist-rustc-linux/build/src/libtest/lib.rs:255

Still working through the types

The first thing I wanted to do was to create a new function, handle_breakpoint, to uninstall the breakpoint and to call the callback. This meant passing the callback from trap_inferior_continue to handle_breakpoint. I struggled with this for some time in getting the type right for the callback. This was confusing to me, it seemed to me that it should match the type of the callback passed into trap_inferior_continue and it should. But I eventually realized that I had the wrong type for trap_inferior_continue. I really needed a mutable reference to the callback which I eventually figured out is written like this:

pub fn trap_inferior_continue<F>(inferior: TrapInferior, callback: &mut F) -> i8
    where F: FnMut(TrapInferior, TrapBreakpoint) -> ();

So now I have:

pub fn trap_inferior_continue<F>(inferior: TrapInferior, callback: &mut F) -> i8
    where F: FnMut(TrapInferior, TrapBreakpoint) -> () {

    let pid = inferior;

    loop {
        ptrace(PTRACE_CONT, pid, ptr::null_mut(), ptr::null_mut())
            .ok()
            .expect("Failed PTRACE_CONTINUE");

        match waitpid(pid, None) {
            Ok(WaitStatus::Exited(_pid, code)) => return code,
            Ok(WaitStatus::Stopped(_pid, signal::SIGTRAP)) =>
                handle_breakpoint(inferior, callback),
            Ok(_) => panic!("Unexpected stop in trap_inferior_continue"),
            Err(_) => panic!("Unhandled error in trap_inferior_continue")
        }
    }
}

fn handle_breakpoint<F>(inferior: TrapInferior,  mut callback: &mut F) -> ()
    where F: FnMut(TrapInferior, TrapBreakpoint) -> () {

    callback(inferior, 0)
}

Now, I just need to add the code to uninstall the breakpoint.

Uninstalling the breakpoint

Uninstalling the breakpoint means using PTRACE_POKETEXT to replace the original data at the site of the breakpoint. But this means collecting the original data in trap_inferior_set_breakpoint and using it in handle_breakpoint. Since the calls are separated it’s hard to pass data around. In C I used a global variable to store the data with the promise to clean it up later. For now, I’ll do the same in Rust, however mutable global variables stand out even more in Rust than they do in C – the require unsafe blocks:

At the end of trap_inferior_set_breakpoint I added:

unsafe { original_breakpoint_word = original as i64; }

Then, I use the original_breakpoint_word in handle_breakpoint

let original = unsafe { original_breakpoint_word };

But I also need the target address of the breakpoint. The inferior is stopped at the breakpoint so we need to look up its instruction pointer. To do this, I need to port over ptrace_util_get_instruction_pointer from the C version. However, it depends on the user struct. It is possible to import this struct in Rust but I eventually realized that ptrace_util_get_instruction_pointer actually depends on the offset of fields in the user struct rather than the struct itself. So, I’ve added the following constants and implemented my functions:

pub mod user {
    pub mod regs {
        pub const R15: i64 = 0 * 8;
        pub const R14: i64 = 1 * 8;
        pub const R13: i64 = 2 * 8;
        pub const R12: i64 = 3 * 8;
        pub const RBP: i64 = 4 * 8;
        pub const RBX: i64 = 5 * 8;
        pub const R11: i64 = 6 * 8;
        pub const R10: i64 = 7 * 8;
        pub const R9:  i64 = 8 * 8;
        pub const R8:  i64 = 9 * 8;
        pub const RAX: i64 = 10 * 8;
        pub const RCX: i64 = 11 * 8;
        pub const RDX: i64 = 12 * 8;
        pub const RSI: i64 = 13 * 8;
        pub const RDI: i64 = 14 * 8;
        pub const ORIG_RAX: i64 = 15 * 8;
        pub const RIP: i64 = 16 * 8;
        pub const CS:  i64 = 17 * 8;
        pub const EFLAGS: i64 = 18 * 8;
        pub const RSP: i64 = 19 * 8;
        pub const SS:  i64 = 20 * 8;
        pub const FS_BASE: i64 = 21 * 8;
        pub const GS_BASE: i64 = 22 * 8;
        pub const DS:  i64 = 23 * 8;
        pub const ES:  i64 = 24 * 8;
        pub const FS:  i64 = 25 * 8;
        pub const GS:  i64 = 26 * 8;
    }
}

fn ptrace_util_get_instruction_pointer(pid: pid_t) -> usize {
    return ptrace(PTRACE_PEEKUSER, pid, user::regs::RIP as * mut c_void, ptr::null_mut())
        .ok()
        .expect("Failed PTRACE_PEEKUSER") as usize;
}

fn ptrace_util_set_instruction_pointer(pid: pid_t, ip: usize) -> () {
    ptrace(PTRACE_POKEUSER, pid, user::regs::RIP as * mut c_void, ip as * mut c_void)
        .ok()
        .expect("Failed PTRACE_POKEUSER");
}

Then I put these new functions to use.

fn handle_breakpoint<F>(inferior: TrapInferior,  mut callback: &mut F) -> ()
    where F: FnMut(TrapInferior, TrapBreakpoint) -> () {

    let original = unsafe { original_breakpoint_word };
    let target_address = ptrace_util_get_instruction_pointer(inferior) - 1;

    ptrace(PTRACE_POKETEXT, inferior, target_address as * mut c_void, original as * mut c_void)
        .ok()
        .expect("Failed PTRACE_POKETEXT");

    callback(inferior, 0);

    ptrace_util_set_instruction_pointer(inferior, target_address);
}

As before, we start by looking up the original_breakpoint_word which is unsafe because it is global. Then, we lookup the breakpoint’s address using ptrace_util_get_instruction_pointer Then we replace the breakpoint’s INT 3 with the original word using PTRACE_POKETEXT

At this point, while the inferior is in its natural state we call the application’s callback.

Finally, we set the instruction pointer back to the start of the breakpoint so we can execute the original instruction. We return and let trap_inferior_continue take over control of the inferior’s life cycle.

And with that the tests pass!

     Running target/debug/lib-8caa31b4833c2b69

running 2 tests
test it_can_exec ... ok
test it_can_set_breakpoints ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured

Wrapping up

In this post we implemented rudimentary support for breakpoints. We modeled this on the C version but allowed Rust to guide some of our API decisions by using it’s native closures and passing the callback through a different function to better adhere to Rust’s lifetime standards.

While porting this code to rust I struggled with (and learned a lot about) the type system, especially regarding the callback.

In other parts of the port I found that I really like Rust’s treatment of global variables. In C they are hard to notice but in Rust the unsafe blocks really stand out. It creates a strong code smell and incentive to clean up those globals. I plan to do exactly that in a future post.

I’ll post again next week. The post will return to the C version and handle multiple breakpoints - both in terms of setting multiple distinct breakpoints as well as handling hitting the same breakpoint multiple times.