-
The Official Relaunch of "A System Programming Blog"
Wow—exactly 10 years ago, I launched this blog. And while it only lived for a few short months, the original intent still resonates.
Today, I’m relaunching it with the same goals at heart:
- To explore the Rust programming language
- To demonstrate how to build low-level systems projects in C
- To investigate emerging tools and technologies
- To write long-form technical series
- And to occasionally dive into smaller, focused topics
I’ve already written two new posts in which I get back up to speed on the debugger project and get the two projects building again.
And here’s what’s coming up:
- I’ll finish work on the how to write a debugger series, likely with a more streamlined scope.
- I’m launching a new series on Systems Thinking—these posts will appear between deeper technical dives.
- I’ll also begin a new series on building a dynamic loader, using it to understand how executable loading works—and maybe even build a full binary interpreter.
Whether you’re new here or remember the original launch, thanks for being part of this reboot. This time, I’m going the distance.
-
Question and Answer Thinking
In this post I want to describe one form of Systems Thinking that I apply in my work and honestly in many aspects of my life. This is what I call Question and Answer thinking.
Whenever I start investigating an issue, solving a problem, or thinking about a new system to build I start by asking myself a question. I keep a notes document and I write this question down. Then, I take that question as a heading and start trying to answer it.
The first question is usually very broad, extremely broad. If I’m looking a bug in an OpenGL driver implementation in which the wrong image is rendered I simply ask: why did it draw this? If I’m looking at a high latency issue in a component of an AV system I might ask how does this component spend its time?
Are these questions helpful? These questions are obvious: of course we don’t know why the wrong image is drawn or how the time is spent. Or maybe we do know why in which case the question is even less helpful, right?
I find these questions and writing down the questions to be helpful in setting the context of the investigation, or design, or whatever work I’m trying to do. Writing down the questions sets an implicit goal of trying to write down an answer. When we arrive at an answer, writing it down is very helpful in that it serves to
- Document something true, and apparently not obvious, about our system.
- Helps to maintain context of what we know about the system
- Directs further investigation
Ok, so we have a broad, obvious question written down which sets a goal to find an answer and document truth. Great, now what? Well, I usually start by asking a smaller question? Let’s continue with the example of looking at performance in an AV system.
- Q: How does this component spend its time?
- Q: What tools do I have for investigating how a component spends its time?
- A: I have a tracing tool to extract and visualize timing data.
- I’ve run the tracing tool and generated
mycomponent.trace.html.
- Q: What does the trace say?
- A: The trace shows a lot of time spent in function foo?
- Q: What tools do I have for investigating how a component spends its time?
So, this is an example refining questions and getting answers. I have questions and answers, and also maybe some other notes I took along the way explaining what I did or things that were unexpected. Now, I like to do this in a system that allows for folding. So I can fold closed questions. I’ve done this in org-mode and in Google docs. I also like to do a little curating of the answered questions by reordering the question and the answer. That is, when I have an answer I make the answer the headline and put the question under. If I do that I might end up with:
- A: The component spends a lot of time in function foo.
- Q: How does this component spend its time?
- A: I have a tracing tool to extract and visualize timing data.
- Q: What tools do I have for investigating how a component spends its time?
- I’ve run the tracing tool and generated
mycomponent.trace.html.
- A: The trace shows a lot of time spent in function foo?
- Q: What does the trace say?
Now, depending on the tools you are using this curating can be a little tedious. I’ll admit that I found it tedious just now while writing the example. That said, in a real investigation you might spend quite a bit of time, say, reading and interpreting the trace so maybe this tedium is broken up by real work and overhead is low.
The advantage of curating the answers to the top is that when folded it reads like this:
- A: The component spends a lot of time in function foo.
- A: I have a tracing tool to extract and visualize timing data.
- A: The trace shows a lot of time spent in function foo?
On one level, this is as I said before in that the answers document true and not so obvious facts about our system. But when we use this system of refinement, hide the questions, and just look at the answers we have not just a set of true statements but a true statement that answered our big broad question and a series of justifications of that fact. Further, we can open up any level to find notes on how we arrived at these conclusions.
This is very powerful.
Imaging you were working on a hard problem that’s going to take a while and you get pulled off of it to work on some high priority bug. When you come back to the hard problem you can review a series of true statements about the problem that summarize what you know so far organized as a line of reasoning that you believe is directed toward the solution to your problem.
One thing to note, just because I’ve answered the initial broad obvious question doesn’t mean I’m done. In our performance example we concluded “The component spends a lot of time in function foo.” but so what? This doesn’t fix the problem or even really tell us how to fix the problem. But, it might help us figure out how to fix the problem if we ask more questions given our new knowledge. On one hand maybe we should have asked an even broader question first, on the other it doesn’t really matter. I have not objection to adding more broad questions and the top level. I can continue with any or all of:
- How much time should we spend in the function
foo? - Are we hitting a particularly bad case in
foo? - How does the component spend its time in a well performing case?
Answering these new questions will give me a better understanding of the system and what went wrong in this particular issue.
Now, I’ve been doing this for… decades, I guess. And at this point I not only take notes this way but I think this way. This is how I approach problems: I ask a question and either answer it or break the question down into smaller questions until I know the answer or know how to find the answer. Finding the answer can mean
- Read some code
- Stop something in a debugger and inspect the state
- Ask someone
- Etc.
But if I know how to find the answer I do that. I write down what I found in the debugger or what my coworker told me in the notes and the set the answer. I continue this way until the problem is solved.
Now, this idea of “I ask a question and either answer it or break the question down into smaller questions until I know the answer” is very similar to the way I write software. We breakdown functionality using successive refinement making smaller and smaller functions until the function are easy to implement.
Similarly to writing this code this work strategy is good for large problems. It’s overkill for things that are very simple. Starting a doc, writing down questions, curating answer doesn’t make sense if you already know what is wrong. If you just wrote some new code and now it crashed and you can realize, “oops I passed the wrong argument here and it’s going to difference a NULL pointer in this case” then just fix it. Don’t write these notes and don’t refine any questions. But, if you have a large problem to work on that might take a few days, or longer, then this system can be very powerful and this approach to systems thinking can really help you dig into problems, navigate systems, and build amazing things.
-
Cleaning a rusty_trap.
In my last post I came back to my debugger project and got set up to build it and tried to get the tests to pass. In this post I’m doing the same with the rust version, rusty_trap.
I want to get in the habit of tracking where I’m starting these posts. So I’m at commit 6c22cba.
When I try to compile I get a few errors:
-*- mode: compilation; default-directory: "/plink:wsl:/home/jkain/projects/rusty_trap/" -*- Compilation started at Sun Jul 6 08:53:57 /home/jkain/.cargo/bin/cargo b warning: no edition set: defaulting to the 2015 edition while the latest is 2024 Compiling nix v0.3.9 (https://github.com/joekain/nix-rust.git?branch=WaitStatus#4ae37a8b) error[E0591]: can't transmute zero-sized type --> /home/jkain/.cargo/git/checkouts/nix-rust-9e641bdbdbe8194f/4ae37a8/src/sched.rs:168:20 | 168 | ... ffi::clone(mem::transmute(callback), ptr as *mut c_void, flags, &... | ^^^^^^^^^^^^^^ | = note: source type: for<'a> extern "C" fn(*mut Box<(dyn FnMut() -> isize + 'a)>) -> i32 {callback} = note: target type: *const for<'a> extern "C" fn(*const Box<(dyn FnMut() -> isize + 'a)>) -> i32 = help: cast with `as` to a pointer instead For more information about this error, try `rustc --explain E0591`. error: could not compile `nix` (lib) due to 1 previous error Compilation exited abnormally with code 101 at Sun Jul 6 08:53:58This is a problem building the nix crate which seems unexpected. I think I have some old version locked. Now that I think about it, everything must have some old version locked. But, looking at Cargo.toml I have:
[package] name = "rusty_trap" version = "0.1.0" authors = ["Joseph Kain <joekain@gmail.com>"] [dependencies] libc = "0.1.8" [dependencies.nix] git = "https://github.com/joekain/nix-rust.git" version = "0.3.8" branch = "WaitStatus" [lib] name = "rusty_trap" path = "src/lib.rs" [[bin]] name = "twelve" path = "tests/inferiors/twelve.rs" [[bin]] name = "loop" path = "tests/inferiors/loop.rs"So i have nix setup to track my repo with my
WaitStatusbranch, which is woefully out of date.Now, I think there might have been a change I made that I never created a pull request for in proper nix. But, let’s see what happens if I just list nix as a dependency and pick of the officilal version. And I’ll do a
cargo updateto get new versions of all the dependencies.@@ -4,12 +4,8 @@ version = "0.1.0" authors = ["Joseph Kain <joekain@gmail.com>"] [dependencies] -libc = "0.1.8" - -[dependencies.nix] -git = "https://github.com/joekain/nix-rust.git" -version = "0.3.8" -branch = "WaitStatus" +libc = "0.2.174" +nix = "0.30.1"Ok, this still doesn’t build but at least it’s complaining about problems in rusty_trap now.
-*- mode: compilation; default-directory: "/plink:wsl:/home/jkain/projects/rusty_trap/" -*- Compilation started at Sun Jul 6 09:08:55 /home/jkain/.cargo/bin/cargo b warning: no edition set: defaulting to the 2015 edition while the latest is 2024 Downloading crates ... Downloaded cfg_aliases v0.2.1 Downloaded nix v0.30.1 Compiling cfg_aliases v0.2.1 Compiling libc v0.2.174 Compiling cfg-if v1.0.1 Compiling bitflags v2.9.1 Compiling nix v0.30.1 Compiling rusty_trap v0.1.0 (/home/jkain/projects/rusty_trap) error[E0432]: unresolved import `nix::unistd::Fork` --> src/lib.rs:5:18 | 5 | use nix::unistd::Fork::*; | ^^^^ could not find `Fork` in `unistd` error[E0432]: unresolved import `nix::sys::wait` --> src/lib.rs:9:15 | 9 | use nix::sys::wait::*; | ^^^^ could not find `wait` in `sys` error[E0433]: failed to resolve: could not find `ptrace` in `sys` --> src/ptrace_util/mod.rs:3:15 | 3 | use nix::sys::ptrace::ptrace::*; | ^^^^^^ could not find `ptrace` in `sys` | note: found an item that was configured out --> /home/jkain/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/nix-0.30.1/src/sys/mod.rs:81:13 | 81 | pub mod ptrace; | ^^^^^^ note: the item is gated behind the `ptrace` feature --> /home/jkain/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/nix-0.30.1/src/sys/mod.rs:79:8 | 79 | #![feature = "ptrace"] | ^^^^^^^ error[E0432]: unresolved import `nix::sys::ptrace` --> src/ptrace_util/mod.rs:2:15 | 2 | use nix::sys::ptrace::*; | ^^^^^^ could not find `ptrace` in `sys` warning: unused import: `nix::unistd::*` --> src/lib.rs:4:5 | 4 | use nix::unistd::*; | ^^^^^^^^^^^^^^ | = note: `#[warn(unused_imports)]` on by default warning: unused import: `handle` --> src/lib.rs:22:18 | 22 | use breakpoint::{handle, TrapBreakpoint}; | ^^^^^^ warning: `extern` declarations without an explicit ABI are deprecated --> src/lib.rs:43:5 | 43 | extern { | ^^^^^^ help: explicitly specify the "C" ABI: `extern "C"` | = note: `#[warn(missing_abi)]` on by default error[E0599]: no variant or associated item named `Sys` found for enum `Errno` in the current scope --> src/lib.rs:82:24 | 82 | Err(Error::Sys(errno::EAGAIN)) => continue, | ^^^ variant or associated item not found in `Errno` warning: variable does not need to be mutable --> src/breakpoint/mod.rs:35:34 | 35 | pub fn handle<F>(inf: Inferior, mut callback: &mut F) -> InferiorState | ----^^^^^^^^ | | | help: remove this `mut` | = note: `#[warn(unused_mut)]` on by default Some errors have detailed explanations: E0432, E0433, E0599. For more information about an error, try `rustc --explain E0432`. warning: `rusty_trap` (lib) generated 4 warnings error: could not compile `rusty_trap` (lib) due to 5 previous errors; 4 warnings emitted Compilation exited abnormally with code 101 at Sun Jul 6 09:08:56So it seems like either my additions to nix for wait, and ptrace aren’t present or they moved. There is something about
forkas well so maybe things just moved. Let’s take a look at the nix docs.Fork, looks like this should be in lowercase. No idea why it was capitalized before but let’s fix that. Fixed.
Also, this use line gives us some hints about waitpid as well:
use nix::{sys::wait::waitpid,unistd::{fork, ForkResult, write}};The documentation also poits out
Available on crate feature process only.This is new, I guess, and also relevant beause just using these doesn’t work:
warning: no edition set: defaulting to the 2015 edition while the latest is 2024 Compiling rusty_trap v0.1.0 (/home/jkain/projects/rusty_trap) error[E0432]: unresolved imports `nix::sys::wait`, `nix::unistd::fork`, `nix::unistd::ForkResult`, `nix::unistd::errno` --> src/lib.rs:4:16 | 4 | ...sys::wait::waitpid,unistd::{fork, ForkResult, Error, errno}, sys::si... | ^^^^ ^^^^ ^^^^^^^^^^ ^^^^^ | | | | | | | | | no `errno` in `unistd` | | | | help: a similar name exists in the module: `Errno` | | | no `ForkResult` in `unistd` | | no `fork` in `unistd` | could not find `wait` in `sys` | note: found an item that was configured out --> /home/jkain/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/nix-0.30.1/src/sys/mod.rs:181:13 | 181 | pub mod wait; | ^^^^ note: the item is gated behind the `process` feature --> /home/jkain/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/nix-0.30.1/src/sys/mod.rs:180:8 | 180 | #![feature = "process"] | ^^^^^^^ note: found an item that was configured out --> /home/jkain/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/nix-0.30.1/src/unistd.rs:278:15 | 278 | pub unsafe fn fork() -> Result<ForkResult> { | ^^^^ note: the item is gated behind the `process` feature --> /home/jkain/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/nix-0.30.1/src/unistd.rs:159:4 | 159 | #![feature = "process"] | ^^^^^^^ note: found an item that was configured out --> /home/jkain/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/nix-0.30.1/src/unistd.rs:209:10 | 209 | pub enum ForkResult { | ^^^^^^^^^^ note: the item is gated behind the `process` feature --> /home/jkain/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/nix-0.30.1/src/unistd.rs:159:4 | 159 | #![feature = "process"] | ^^^^^^^ = help: consider importing this module instead: nix::errnoOh, I need to add this to the Cargo file on the nix dependency. Looks like I need something similar for ptrace. This is new to me and seems pretty cool that I can pick and choose pieces of nix (and other crates I assume) to include.
modified Cargo.toml @@ -5,7 +5,7 @@ authors = ["Joseph Kain <joekain@gmail.com>"] [dependencies] libc = "0.2.174" -nix = "0.30.1" +nix = {version = "0.30.1", features = ["process", "ptrace"]} [lib] name = "rusty_trap"Now, ptrace is interesting. It looks like someone had a similar idea to my idea to create separate functions for each of the different ptrace requests (PTRACE_CONT, PTRACE_PEEKTEXT, etc.) with proper type checking. That’s fantastic.
These are perfect and I’ll use them directly.
Well, in the short term I’ll implement ptrace_util on top of them which will
- Let me review the types and see how they compare to the types I choose.
- Leave the code that uses ptrace_util alone for now. I like to mimize changes until we get the tests running and passing again.
Hmm,
PEEKTEXTandPOKETEXTdon’t seem to have coresponding functions in the new nix ptrace. However,PEEKDATAandPOKEDATAdo have functions (read,write) and Linux doesn’t actually separate text and data so I should be able to use these instead.And ok, now it builds at least. No idea if it works but these seem pretty straightfoward so I hope they do.
modified Cargo.toml @@ -5,7 +5,7 @@ authors = ["Joseph Kain <joekain@gmail.com>"] [dependencies] libc = "0.2.174" -nix = "0.30.1" +nix = {version = "0.30.1", features = ["process", "ptrace"]} [lib] name = "rusty_trap" modified src/ptrace_util/mod.rs @@ -1,8 +1,6 @@ use libc::pid_t; -use nix::sys::ptrace::*; -use nix::sys::ptrace::ptrace::*; -use std::ptr; -use libc::c_void; +use nix::sys::ptrace; +use nix::unistd::Pid; use inferior::InferiorPointer; @@ -39,44 +37,49 @@ pub mod user { } pub fn trace_me() -> () { - ptrace(PTRACE_TRACEME, 0, ptr::null_mut(), ptr::null_mut()) + ptrace::traceme() .ok() .expect("Failed PTRACE_TRACEME"); } pub fn get_instruction_pointer(pid: pid_t) -> InferiorPointer { - let raw = ptrace(PTRACE_PEEKUSER, pid, user::regs::RIP as * mut c_void, ptr::null_mut()) + let raw = ptrace::read_user(Pid::from_raw(pid), user::regs::RIP as ptrace::AddressType) .ok() .expect("Failed PTRACE_PEEKUSER"); InferiorPointer(raw as u64) } pub fn set_instruction_pointer(pid: pid_t, ip: InferiorPointer) -> () { - ptrace(PTRACE_POKEUSER, pid, user::regs::RIP as * mut c_void, ip.as_voidptr()) + ptrace::write_user(Pid::from_raw(pid), user::regs::RIP as ptrace::AddressType, ip.as_i64()) .ok() .expect("Failed PTRACE_POKEUSER"); } pub fn cont(pid: pid_t) -> () { - ptrace(PTRACE_CONT, pid, ptr::null_mut(), ptr::null_mut()) + ptrace::cont(Pid::from_raw(pid), None) .ok() .expect("Failed PTRACE_CONTINUE"); } pub fn peek_text(pid: pid_t, address: InferiorPointer) -> i64 { - ptrace(PTRACE_PEEKTEXT, pid, address.as_voidptr(), ptr::null_mut()) - .ok() - .expect("Failed PTRACE_PEEKTEXT") + // From ptrace(2) regarding PTRACE_PEEKTEXT and PTRACE_PEEKDATA + // Linux does not have separate text and data address spaces, + // so these two operations are currently equivalent. + // So use ptrace::read which is ptrace(PTRACE_PEEKDATA, ...) + // An alterantive would be to use libc::ptrace. + ptrace::read(Pid::from_raw(pid), address.as_voidptr()) + .ok() + .expect("Failed PTRACE_PEEKTEXT") } pub fn poke_text(pid: pid_t, address: InferiorPointer, value: i64) -> () { - ptrace(PTRACE_POKETEXT, pid, address.as_voidptr(), value as * mut c_void) - .ok() - .expect("Failed PTRACE_POKETEXT"); + ptrace::write(Pid::from_raw(pid), address.as_voidptr(), value) + .ok() + .expect("Failed PTRACE_POKETEXT") } pub fn single_step(pid: pid_t) -> () { - ptrace(PTRACE_SINGLESTEP, pid, ptr::null_mut(), ptr::null_mut()) + ptrace::step(Pid::from_raw(pid), None) .ok() .expect("Failed PTRACE_SINGLESTEP"); }Now the compilation errors go back to lib.rs, I fixed the problems with
useearlier but now it’s looking at the code itself. On of the main issues I faced was thatexecvechanged the types it expects. I had a lot of trouble getting this right and I think part of this is I have forgotten almost all the rust I knew and I might be doing this wrong. But these type conversions to pass to nix::unistd::execve are too much IMHO.fn exec_inferior(filename: &Path, args: &[&str]) -> () { // let c_filename = &CStr::from_ptr(filename.to_str().unwrap().as_ptr() as *const i8); let cstring_filename = CString::new(filename.to_str() .expect("Failed to get string from filename")) .expect("Failed to create CString from filename"); let cstr_filename = CStr::from_ptr(cstring_filename.as_ptr()); disable_address_space_layout_randomization(); ptrace_util::trace_me(); execve::<CString, CString>(cstr_filename, &[], &[]) .ok() .expect("Failed execve"); unreachable!(); }I kind worked through this by brute force / trial and error and am not proud of it. But, let’s move on.
On the bright side, libc now includes personality and I can use that.
@@ -37,35 +34,32 @@ static mut global_breakpoint : Breakpoint = Breakpoint { }; static mut global_inferior : Inferior = Inferior { pid: 0, state: InferiorState::Stopped }; -mod ffi { - use libc::{c_int, c_long}; - - extern { - pub fn personality(persona: c_long) -> c_int; - } -} - fn disable_address_space_layout_randomization() -> () { unsafe { - let old = ffi::personality(0xffffffff); - ffi::personality((old | 0x0040000) as i64); + let old = libc::personality(0xffffffff); + libc::personality((old | libc::ADDR_NO_RANDOMIZE) as u64); } }Happy to be rid of the hard coded
0x0040000.The rest were some easier updates to the types. I do wonder if I should push
nix::unistd::Pidall the way up and be consistent about using it.modified src/lib.rs @@ -1,15 +1,12 @@ extern crate nix; extern crate libc; -use nix::unistd::*; -use nix::unistd::Fork::*; +use nix::{Error, sys::wait::waitpid,unistd::{execve, fork, ForkResult}, sys::signal}; use libc::pid_t; -use nix::Error; -use nix::errno; use nix::sys::wait::*; -use std::ffi::CString; +use nix::unistd::Pid; +use std::ffi::{CString, CStr}; use std::path::Path; -use nix::sys::signal; mod ptrace_util; @@ -19,7 +16,7 @@ use inferior::*; mod breakpoint; pub use self::breakpoint::trap_inferior_set_breakpoint; -use breakpoint::{handle, TrapBreakpoint}; +use breakpoint::{TrapBreakpoint}; #[derive(Copy, Clone)] struct Breakpoint { @@ -37,35 +34,32 @@ static mut global_breakpoint : Breakpoint = Breakpoint { }; static mut global_inferior : Inferior = Inferior { pid: 0, state: InferiorState::Stopped }; -mod ffi { - use libc::{c_int, c_long}; - - extern { - pub fn personality(persona: c_long) -> c_int; - } -} - fn disable_address_space_layout_randomization() -> () { unsafe { - let old = ffi::personality(0xffffffff); - ffi::personality((old | 0x0040000) as i64); + let old = libc::personality(0xffffffff); + libc::personality((old | libc::ADDR_NO_RANDOMIZE) as u64); } } fn exec_inferior(filename: &Path, args: &[&str]) -> () { - let c_filename = &CString::new(filename.to_str().unwrap()).unwrap(); + // let c_filename = &CStr::from_ptr(filename.to_str().unwrap().as_ptr() as *const i8); + let cstring_filename = CString::new(filename.to_str() + .expect("Failed to get string from filename")) + .expect("Failed to create CString from filename"); disable_address_space_layout_randomization(); ptrace_util::trace_me(); - execve(c_filename, &[], &[]) + let cstr_filename = unsafe { CStr::from_ptr(cstring_filename.as_ptr()) }; + execve::<CString, CString>(cstr_filename, &[], &[]) .ok() .expect("Failed execve"); unreachable!(); } -fn attach_inferior(pid: pid_t) -> Result<Inferior, Error> { - match waitpid(pid, None) { - Ok(WaitStatus::Stopped(pid, signal::SIGTRAP)) => - return Ok(Inferior {pid: pid, state: InferiorState::Running}), +fn attach_inferior(raw_pid: pid_t) -> Result<Inferior, Error> { + let nix_pid = Pid::from_raw(raw_pid); + match waitpid(nix_pid, None) { + Ok(WaitStatus::Stopped(pid, signal::Signal::SIGTRAP)) => + return Ok(Inferior {pid: pid.into(), state: InferiorState::Running}), Ok(_) => panic!("Unexpected stop in attach_inferior"), Err(e) => return Err(e) } @@ -73,25 +67,25 @@ fn attach_inferior(pid: pid_t) -> Result<Inferior, Error> { pub fn trap_inferior_exec(filename: &Path, args: &[&str]) -> Result<TrapInferior, Error> { loop { - match fork() { - Ok(Child) => exec_inferior(filename, args), - Ok(Parent(pid)) => { - unsafe { global_inferior = attach_inferior(pid).ok().unwrap() }; - return Ok(pid) + match unsafe { fork() } { + Ok(ForkResult::Child) => exec_inferior(filename, args), + Ok(ForkResult::Parent{child: pid}) => { + unsafe { global_inferior = attach_inferior(pid.into()).ok().unwrap() }; + return Ok(pid.into()) }, - Err(Error::Sys(errno::EAGAIN)) => continue, - Err(e) => return Err(e) + Err(Error::EAGAIN) => continue, + Err(e) => return Err(e) } } } -pub fn trap_inferior_continue<F>(inferior: TrapInferior, callback: &mut F) -> i8 +pub fn trap_inferior_continue<F>(inferior: TrapInferior, callback: &mut F) -> i32 where F: FnMut(TrapInferior, TrapBreakpoint) -> () { let mut inf = unsafe { global_inferior }; ptrace_util::cont(inf.pid); loop { - inf.state = match waitpid(inf.pid, None) { + inf.state = match waitpid(Pid::from_raw(inf.pid), None) { Ok(WaitStatus::Exited(_pid, code)) => return code, Ok(WaitStatus::Stopped(_pid, signal::SIGTRAP)) => breakpoint::handle(inf, callback),Now, it compiles. But the tests fail. Before even running the tests I get some loader issues:
Inconsistency detected by ld.so: ../sysdeps/x86_64/dl-machine.h: 541: elf_machine_rela_relative: Assertion `ELFW(R_TYPE) (reloc->r_info) == R_X86_64_RELATIVE' failed! Inconsistency detected by ld.so: ../sysdeps/x86_64/dl-machine.h: 541: elf_machine_rela_relative: Assertion `ELFW(R_TYPE) (reloc->r_info) == R_X86_64_RELATIVE' failed!These are weird, unless I
ptracepokeed something that I shouldn’t have. And, I am suspicious that the breakpoint offsets would have moved, so let’s look into that first.I used gdb to find the proper address of the breakpoints. And this elimited the inconsistencies detected by ld.so.
But the tests still don’t pass. Well, one of them doesn’t:
running 3 tests test it_can_exec ... ok test it_can_handle_a_breakpoint_more_than_once ... FAILED test it_can_set_breakpoints ... ok failures: ---- it_can_handle_a_breakpoint_more_than_once stdout ---- thread 'it_can_handle_a_breakpoint_more_than_once' panicked at /home/jkain/projects/rusty_trap/src/lib.rs:93:17: Unexpected stop on signal SIGSEGV in trap_inferior_continue. State: 0 note: run with `RUST_BACKTRACE=1` environment variable to display a backtraceOk, so
inferior_continuefound the inferior stopped onSIGSEGV. We could very well cause this by patching/restoring the breakpoint word incorrectly.Is this the first time the breakpoint triggered?
Using printouts it seems that we’ve handled the breakpoint twice already. But stopping on the breakpoint calls
breakpoint::handleonce and the single stepping calls it a second time. Let’s confirm with better prints.in breakpoint::handle inf.state = 0 in breakpoint::handle inf.state = 2Right, so 0 is
Runningand 2 isSingleStepping.So, in
breakpoint::handlewhen single stepping we would attempt to set the breakpoint again and then continue.Oh, this is the test that only has a breakpoint in main. Why did I think it was the other one?
Given that the test of a breakpoint multiple times works but the one with a single breakpoint occurance does not, it seems that either the address is wrong or something is wrong with trying to write this. Let me look at the main function disassembly in gdb:
I do see one bug and have fixed it but it isn’t the problem I’m looking for.
modified src/breakpoint/mod.rs @@ -27,7 +27,7 @@ fn step_over(inferior: TrapInferior, bp: Breakpoint) -> () { fn set(inferior: TrapInferior, bp: Breakpoint) -> () { let mut modified = bp.original_breakpoint_word; - modified &= !0xFFi64 << bp.shift; + modified &= !(0xFFi64 << bp.shift); modified |= 0xCCi64 << bp.shift; poke_text(inferior, bp.aligned_address, modified);I think this was never an issue because the target and aligned address were always the same (and therefor
shiftwas 0) becauze functions are usually aligned to 16 bytes. Either that or I don’t understand the operator precedence here. Eitehr way I think this is a good change but it doesn’t fix the test.At this point I took a diversion to setup a cleaner development system with a newer version of Ubuntu. And then came back to working on this on the new system. Strangely, the test that was failing changed: The basic breakpoint test worked but the multiple breakpoint test stopped working.
One issue is that again compilers and systems have changed so the breakpoint addresses have moved. Also, the addresses now have namespaces so the symbol name is
loop::fooand it gets C++-name-mangled. With this new name I’m able to find the address in gdb and plug it into the test.After that, I wanted to focus only on the one test that is failing. I commented out the other tests (there must a better way to do this). Then
it_can_handle_a_breakpoint_more_than_oncepassed!?!?I asked, are these tests run concurrently?
- Yes, concurrent use of rusty_trap isn’t going to work yet.
I can use
cargo test -- --test-threads=1to run them sequentially.Now this passes! Well, I learned something about cargo testing and we have new code smell to encourage us to make
rusty_trapthread safe. The biggest road block here is the global breakpoint.Also, from the same documentation I see that there is a better way to run a single test:
cargo test it_can_handle_a_breakpoint_more_than_onceHere is the pull request for all of the changes we made for this post.
There are a number of to do items we can carry forward with us.
TODO:
- Go over the warnings
- Consider
nix::unistd::Pidrather than pid_t. - Look at
ptrace::AddressTypeand consider replacingInferiorPointer. - Get rid of the global breakpoint and make the system thread safe.
- Handle multiple breakpoints.
We should be able to tackle some of these in the next post. I hope to see you there.
-
Trap - Back Up and Running
It’s been a long time and I’m finally coming back to this project and need to get things building again. I tried compiling the project and ran into the following errors:
-- Configuring done -- Generating done -- Build files have been written to: /home/jkain/projects/trap/_out Scanning dependencies of target trap [ 4%] Building C object src/CMakeFiles/trap.dir/inferior_load.c.o /home/jkain/projects/trap/src/inferior_load.c: In function ‘attach_to_inferior’: /home/jkain/projects/trap/src/inferior_load.c:24:49: error: ‘SIGTRAP’ undeclared (first use in this function) 24 | if (WIFSTOPPED(status) && WSTOPSIG(status) == SIGTRAP) { | ^~~~~~~ /home/jkain/projects/trap/src/inferior_load.c:24:49: note: each undeclared identifier is reported only once for each function it appears in /home/jkain/projects/trap/src/inferior_load.c: In function ‘trap_inferior_continue’: /home/jkain/projects/trap/src/inferior_load.c:71:51: error: ‘SIGTRAP’ undeclared (first use in this function) 71 | if (WIFSTOPPED(status) && WSTOPSIG(status) == SIGTRAP) { | ^~~~~~~ make[2]: *** [src/CMakeFiles/trap.dir/build.make:63: src/CMakeFiles/trap.dir/inferior_load.c.o] Error 1 make[1]: *** [CMakeFiles/Makefile2:149: src/CMakeFiles/trap.dir/all] Error 2 make: *** [Makefile:95: all] Error 2SIGTRAPisn’t defined? I guess I need to include signal.h but I don’t understand why I didn’t need this before.modified src/inferior_load.c @@ -5,6 +5,7 @@ #include <sys/wait.h> #include <assert.h> #include <errno.h> +#include <signal.h> #include <stdio.h> #include <stdlib.h>Now it builds, great! But the tests fail:
Test project /home/jkain/projects/trap/_out Start 1: test_exec 1/7 Test #1: test_exec ............................ Passed 0.00 sec Start 2: test_breakpoint 2/7 Test #2: test_breakpoint ......................Child aborted***Exception: 0.01 sec PTRACE_POKETEXT: : Input/output error Hello World! Start 3: test_multi_breakpoint 3/7 Test #3: test_multi_breakpoint ................Child aborted***Exception: 0.01 sec PTRACE_POKETEXT: : Input/output error Start 4: test_multiple_breakpoints 4/7 Test #4: test_multiple_breakpoints ............Child aborted***Exception: 0.01 sec PTRACE_POKETEXT: : Input/output error Start 5: test_delete_breakpoint 5/7 Test #5: test_delete_breakpoint ...............Child aborted***Exception: 0.01 sec PTRACE_POKETEXT: : Input/output error Start 6: test_delete_breakpoint_in_callback 6/7 Test #6: test_delete_breakpoint_in_callback ...Child aborted***Exception: 0.01 sec PTRACE_POKETEXT: : Input/output error Start 7: test_duplicate_breakpoint 7/7 Test #7: test_duplicate_breakpoint ............Child aborted***Exception: 0.01 sec PTRACE_POKETEXT: : Input/output error 14% tests passed, 6 tests failed out of 7 Total Test time (real) = 0.06 sec The following tests FAILED: 2 - test_breakpoint (Child aborted) 3 - test_multi_breakpoint (Child aborted) 4 - test_multiple_breakpoints (Child aborted) 5 - test_delete_breakpoint (Child aborted) 6 - test_delete_breakpoint_in_callback (Child aborted) 7 - test_duplicate_breakpoint (Child aborted) Errors while running CTestOh, this is because the address for the breakpoints has changed because it’s been 10 years and I’m using a different system, compiler, etc. now. This is a great reminder that we should implement symbol look up so that the breakpoints aren’t just hard coded addresses, but let’s get the basics working again before we take that on.
Fixing the offsets isn’t enough to fix this. There is probably address space randomization like we saw with Rust. I’ve implemented
personalitycalls to disable address randomization.@@ -12,6 +14,11 @@ static inferior_t g_inferior; static void setup_inferior(const char *path, char *const argv[]) { + unsigned long old = personality(0xFFFFFFFF); + if (personality(old | ADDR_NO_RANDOMIZE) < 0) { + perror("Failed to set personality:"); + } + ptrace_util_traceme(); execv(path, argv); }But this still doesn’t work. So, what is the address of main?
From gdb
main = 0x555555555169. And it is always this value, it’s not random. I am running on WSL, maybe that’s weird. Also does this binary have a start address set?- Yes, it does. The start address is 0x0000000000001080.
Which coresponds with the normal offset of .text
Sections: Idx Name Size VMA LMA File off Algn 15 .text 00000195 0000000000001080 0000000000001080 00001080 2**4 CONTENTS, ALLOC, LOAD, READONLY, CODEOk, so what’s going on? Maybe another look at gdb can help:
$ gdb inferiors/hello (No debugging symbols found in inferiors/hello) (gdb) b main Breakpoint 1 at 0x1169 (gdb)So gdb gets the basic address (note this is different than the 0x1149 we saw before because I added a printf call and string.).
Hang on, let’s run under gdb:
(gdb) r Starting program: /home/jkain/projects/trap/_out/test/inferiors/hello Breakpoint 1, 0x0000555555555169 in main () (gdb)Ok, so object is being loaded into a different address as it runs. And at the start gdb doesn’t know where it will be loaded (just like trap).
For now, I can just hard code this address, I guess. I’ve removed the printout from the inferior and use 0x555555555149 as the breakpoint address.
Now that I think about it, I saw the same thing in Rust when I last worked on it. This makes sense in that the binary wouldn’t necessarily get loaded at address 0. If I understand correctly, there are non-relocatable types of binaries and that must have been what I was building before and the newer tool chain allows for relocation. But, that’s just speculation on my part.
Even with these new addresses the I see an assertion failure in
breakpoint_resolve. I added a printout inbreakpoint_resolveto print the instruction pointer. I see two calls:breakpoint_resolve: ip = 0x555555555149 breakpoint_resolve: ip = 0x55555555514c test_breakpoint: /home/jkain/projects/trap/src/breakpoint.c:102: breakpoint_resolve: Assertion `result' failed.The first call to
breakpoint_resolvemust be when the breakpoint in main is reached. And the second call is 3 bytes later. I think this is trying to resolve the stop in execution that happens when we are in stateINFERIOR_SINGLE_STEPPINGto step over the breakpoint. And in this case the instruction pointer will have moved past the byte where we inserted the breakpoint – past the entire original instruction. Sobreakpoint_resolveshould fail as this address isn’t one of our breakpoints. That is,breakpoint_resolveassumes we executed a one byte instruction and subtracts 1 from the instruction pointer to find thetarget_address.This is not correct and wasn’t correct when I originally wrote it. I don’t know how this worked before.
Is this something I never wrote about (and presumably didn’t test properly)?
- It looks like this was added in the multiple-breakpoints post.
I think I need to back up to the previous post and, make these breakpoint address tweaks, and then retrace my steps through multiple-breakpoints. I also should get in the habit of announcing which commit I’m starting with in each post and have a PR at the end of each post.
I’ll take this work on in a separate post. For now, I’ve pushed the fixes as a PR and we should get started on getting the Rust version working again.
But, before, I go I have one question for you. At some point in the future I plan to port trap and rusty_trap to ARM. Do you have a favorite ARM board? I have a few: an old Raspberry Pi 2, an odroid XU4, and an Orange Pi Zero 3. I’m thinking about getting an Orange Pi 5. Let me know your thoughts in the comments.
-
Refactor fail
In the last post we implemented a state machine to manage breakpoints in rusty_trap. We also added support for hitting a single breakpoint multiple times. Now, I want to start adding support for multiple distinct breakpoints.
One way to approach adding a new feature is to refactor to make the code easily accept the new feature. That’s what I set out to do today, but it didn’t go the way I planned.
Read more...