GDB

even629

2025-06-24 (Updated: 2025-10-13)

programming

GNU, programming

Timeline

2025-06-24

init

2025-10-07

add gdb in emacs

2025-10-09

add complementary skills

2025-10-10

add multithread debugging

2025-10-13

add multiprocess debugging，remote debugging，coredump debugging，add memory problem check

This article introduces the configuration and various startup methods of the GDB debugger, discusses methods for debugging running programs and core dump files, and summarizes common debugging operation techniques such as source code viewing and breakpoint management.

GDB Configuration

Display Chaos in TUI Mode

Reference:

gdb tuiSolution for Display Chaos in Mode

https://blog.csdn.net/u012772347/article/details/142598998

In short, create a~/.config/gdb/gdbinitfile and write the following content

define c
continue
refresh
end

define n
next
refresh
end

set debuginfod enabled on
set print pretty

For C++ debugging, it is recommended to use the following extended commands for easier viewing of STL containers:

gdbinit_stl_views

http://www.yolinux.com/TUTORIALS/src/dbinit_stl_views-1.03.txt

The following commands are available:

std::vector<T> -- via pvector command
std::list<T> -- via plist or plist_member command
std::map<T,T> -- via pmap or pmap_member command
std::multimap<T,T> -- via pmap or pmap_member command
std::set<T> -- via pset command
std::multiset<T> -- via pset command
std::deque<T> -- via pdequeue command
std::stack<T> -- via pstack command
std::queue<T> -- via pqueue command
std::priority_queue<T> -- via ppqueue command
std::bitset<n> -- via pbitset command
std::string -- via pstring command
std::widestring -- via pwstring command

GDB Cannot See Program’s printf Output

This is because the output is buffered; use the following command

1	call fflush(stdout)

IO Input/Output Using Different Terminals

By default, GDB and the program share the same terminal for input/output. You can specify a separate terminal for the program by first opening a terminal and entering the tty command to get the current terminal name.
Then start:

1	gdb -tty /dev/pts/3 ./a.out

Using gdb in emacs

Mainly use gdb’s MI mode, refer to

my emacs configuration

https://even629.com/posts/28521/

gdb tui mode

In TUI mode, up to 3 windows are displayed, and the command window always exists

Open tui mode and open source code

1	(gdb) layout src

Breakpoints before line numbers in the source window

B indicates it has been hit, at least once
b indicates not yet hit
- indicates the breakpoint is enabled
- indicates the breakpoint is disabled

Display assembly window

1	(gdb) layout asm

Display register window

1	(gdb) layout reg

Split window

1	(gdb) layout split

Switch window focus

1 2	(gdb) focus src/asm/reg/cmd (gdb) fs asm

View the currently focused window

1	(gdb) info win

Exit window mode

Ctrl + x +a

Start gdb

Compilation phase: add debugging information

To allow GDB to see function names, variable names, and source line numbers, you must add the-gparameter:

1 2	gcc -g hello.c -o hello # For C programs g++ -g hello.cpp -o hello # For C++ programs

Otherwise, GDB can only see assembly and memory addresses, making source-level debugging impossible.

Startup method

Debug a program

1	gdb ./program

Set program runtime parameters

Set runtime parameters (e.g., command-line arguments)：
1
set args 10 20 30
View currently set parameters：
1
show args

Set runtime environment variables

Set program run path (for finding executable files)：
1
path /your/bin/dir
View run path settings：
1
show paths
Set environment variables(e.g., passed tomain()program’s environment):
1
set environment USER=yourname
View environment variables：
1
2
show environment
show environment USER

Set working directory

Setting the working directory refers to the current directory when the program runs

Change current directory(equivalent to shell’scd）：
1
cd /path/to/dir
View current directory：
1
pwd

Control program input and output

View terminal information bound to the program：
1
info terminal
Redirect output (e.g., save output to a file)：
1
run > output.txt
Specify the terminal device for program input and output：
1
tty /dev/pts/1

Debugging core dump files

Linux core dump：

Commonly referred to as core dump or kernel dump, collectively called dump files. It is a memory information snapshot of a process at a specific moment, containing the entire memory information and register data of that process when the dump file was generated. Dump files can be for a single process or the entire system. They can be generated while the process is alive or automatically when the process or system crashes.
Creating a core dump file for a live process can generally be done using gdb. After attaching the process with gdb, execute the generate-core-file or gcore command to generate the core dump file.
More often, we analyze core dump files generated from crashes.

Generating a core dump file for a live process

Generating a core dump file via gcore does not affect the program’s operation at all

1
2
3

gdb attach pid
(gdb) gcore test.core
(gdb) detach

A core dump is a dump after a program crash

1	gdb ./program core

Enabling Linux core file generation

Linux Core file generation is not enabled by default, meaning it will not be generated when a segmentation fault occurscore dumped. It can be enabled with the following commandcoreto generate the file:

1 2	# No limit on core file size ulimit -c unlimited

unlimitedmeans the systemdoes not limit the size of core files, as long as there is enough disk space, it will dump all the memory occupied by the program. If you need to limit the system-generatedcore size, you can use the following command:

1 2	# The maximum core file size limit is 409600 bytes ulimit -c 409600

You can configure the name of the generated coredump file to avoid overwriting the default coredump file

1	echo -e "%e-%p-%t" > /proc/sys/kernel/core_pattern

Disable Linux core file generation

To disable core dump functionality, simply set the limit size to0:

1	ulimit -c 0

Note: If you only enter the command “ulimit -c unlimited”, it will only take effect in the current terminal and will be invalid when you exit the terminal or open a new one.

Example:

Write a simple C program to artificially create aSegmentation faulterror:

#include <stdio.h>
#include <stdlib.h>

int main(int argc, char **argv)
{
  int *p = NULL;

  // Assigning a value to a NULL pointer will cause a Segmentation fault error
  *p = 100;

  return 0;
}

In the above code, a null pointer variable P is defined, and then a value is assigned to the null pointer P. Running the program willgenerate a segmentation fault。

After enablingcore dump, a corefile will be generated.

# Compile hello.c to generate the hello program
gcc -o hello hello.c -g

# Run the program
./hello

After running, we can seeSegmentation fault (core dumped)a prompt message, indicating that acore file has been generated in the current directory:

Debugging a running program

1	gdb ./program <PID>

Start GDB by directly specifying the PID(requires the executable program path):

1	gdb ./program <PID>

Attach to a PID within GDB：

1 2	(gdb) attach <PID> (gdb) detach # Detach

Common startup parameters

Parameter	Meaning
`-s`or`-symbols <file>`	Specify symbol table file
`-se <file>`	Specify symbol table file and associate with executable
`-c`or`-core <file>`	Specify core dump file for debugging
`-d`or`-directory <dir>`	Add source search path (default uses`$PATH`）

Exit by typing quit(q)

Running Shell commands in GDB

In GDB, you can directly run operating system commands by:

1 2	(gdb) shell <命令字符串> (gdb) !<命令字符串>

example

1 2	(gdb) shell ls -l (gdb) shell cat input.txt

This starts your system’s shell inside GDB (determined by environment variableSHELL), then executes the command you wrote.

GDB also has a built-in command:

1	(gdb) make <参数>

It is essentially equivalent to:

1	(gdb) shell make <参数>

That is, it will call the system’smaketool to recompile the program, which is very convenient for quickly rebuilding after modifying code during debugging.

1 2	(gdb) pip i locals \| grep test (gdb) \| thread apply all bt \| wc

Save gdb debug output log

# Enable log output, off to disable
(gdb) set logging on
# Set output file
(gdb) set logging file filename
# Overwrite output file, default is append
(gdb) set logging overwrite

Debug program

Source code

Display source code

Display source codelist or **l,**Default shows 10 lines
Set the number of lines to display each time:set listsize xx
View the code of a specified function:list test_fun
View code at a specified line in a specified file:list main.cpp:15

Search source code

searchRegular expression
forward-searchRegular expression
reverse-searchRegular expression

(After entering the search command, pressing Enter will continue to find the next match; use the list command to specify the starting search position)

Set source code search directory

directory path

Breakpoint

Set breakpoint

Set breakpoint by function name

1	break function

At the specified function’sentrystop.
For C++, it can be written as:
- break ClassName::Function
- break function(type1, type2)(if overloaded)

Set breakpoint by line number

break 42

At line42of the current source file, set a breakpoint.

Set breakpoint relative to current line

1 2	break +5 // 5 lines after current line break -3 // 3 lines before current line

Specify file + line number

1	break filename.c:42

Atfilename.cline 42, set a breakpoint.

Specify file + function name

1	break filename.c:func

Atfilename.cinfuncset a breakpoint at the entry of the function.

Set breakpoint by address

Commonly used in assembly debugging

1	break *0x4007d0

At program memory address0x4007d0set a breakpoint.

Set conditional breakpoint

1	break func if i == 100

When variablei == 100and execution reachesfuncfunction, then stop.

Set breakpoint at next statement (no arguments)

break

Set a breakpoint at the statement that will be executed next.

View breakpoints

View all breakpoints

1	info breakpoints

View breakpoint by specified number

1	info break 3

Delete breakpoint

Delete a specific breakpoint

1 2	delete <编号> del <编号>

For example:delete 1indicates deleting breakpoint number 1.

Delete multiple breakpoints

1	delete 1 2 3

Delete breakpoints 1, 2, and 3 simultaneously.

Delete all breakpoints

delete

Without parametersmeans delete all breakpoints. GDB will prompt you to confirm (entery）。

Add commands to a breakpoint

1
2
3

commands <bnum>
...gdb命令序列...
end

Example:

break foo if x > 0
commands
printf "x is %d\n", x
continue
end

Effect: When x > 0, the breakpoint is hit, prints and automatically continues, no need to manually pressc。

b 31
commands
>p *curr
>p prev
>end
i b

After setting commands for a breakpoint, when the program stops at that breakpoint, it automatically prints these two values;

The commands command can be followed directly by the breakpoint number

1 2	i b commands 2

Clear existing commands

1 2	commands <bnum> end

Ignore breakpoint count (ignore)

1	ignore <bnum> <count> # Ignore breakpoint number bnum for count triggers

For example:

1	ignore 2 3

Ignore the first three hits of breakpoint 2, and only break on the fourth hit.

example

When debugging issues in loops or large functions, it is recommended to use:
break <line> if i == 9999
or
ignore <bnum> 9998
After locating the bug, do not delete the breakpoint, directly:
1
disable <bnum> # Keep the breakpoint for later reuse
When you want to test changes in multiple variables:
1
2
watch a
watch b

When you want to automate debugging:

silent
printf "Reached here\n"
continue
end

Save breakpoints to a file and load them

1 2	(gdb) save breakpoints d.txt (gdb) !cat d.txt

After closing and reopening, apply the saved breakpoint information

1	(gdb) source d.txt

Watchpoint

A watchpoint is a special breakpoint that triggers when the value of an expression changes. The expression can be a variable’s value, or it can include one or more variables combined with operators, such as ‘a+b’. It is sometimes called a data breakpoint.

Set a watchpoint

`watch <expr>`

Purpose: When the expression or variableexpr’svalue is changedthe program will pause.
Example：
1
2
watch x
watch gdata+gdata2>10

When variablexpauses when its value changes.

The program stops when any thread satisfies gdata+gdata2>10

`rwatch <expr>`

Purpose: When expression or variableexprisread, the program pauses.
Example：
1
rwatch y

When variableyis read, pause the program.

`awatch <expr>`

Purpose: When expression or variableexprisread or written, the program pauses.
Example：
1
awatch z

When variablezis read or written, it pauses.

View current watchpoints

1	info watchpoints

Display all set watchpoints (similar toinfo breakpoints）。

Delete watchpoint

1	delete <编号>

Same as deleting breakpoints.

Notes

Watchpoints depend onwhether the target architecture supports hardware watchpoints(most do).
If not supported, GDB may not be able to setwatch、rwatchetc.
The number of watchpoints is limited, generally fewer than breakpoints (usually 4).

Catchpoint

A catchpoint is a special breakpoint. The command syntax is: catch event, meaning that when the event is caught, the program will stop.

Command format

1	catch <event>

You can also use a one-time catchpoint:

1	tcatch <event>

Common catchpoint types

Event type	Description
`throw`	Catch the location where a C++ program throws an exception.
`catch`	Catch the location where a C++ program catches an exception.
`exec`	Catch program call`exec()`System call (replace process image).
`fork`	Catch program call`fork()`System call (create child process).
`vfork`	catch`vfork()`call (special type of`fork()`）。
`load`	Catch dynamic-link library load events.
`unload`	Catch dynamic-link library unload events.

Example

1	catch throw

Break when C++ throws an exception.

1	catch fork

Break when the program callsfork().

1	tcatch exec

Set a one-time catchpoint, break when the program callsexec()system call, then automatically remove.

Command	Description
catch assert	Catch failed Ada assertions, when raised.
catch catch	Catch an exception, when caught.
catch exception	Catch Ada exceptions, when raised.
catch exec	Catch calls to exec.
catch fork	Catch calls to fork.
catch handlers	Catch Ada exceptions, when handled.
catch load	Catch loads of shared libraries.
catch rethrow	Catch an exception, when rethrown.
catch signal	Catch signals by their names and/or numbers.
catch syscall	Catch system calls by their names, groups and/or numbers.
catch throw	Catch an exception, when thrown.
catch unload	Catch unloads of shared libraries.
catch vfork	Catch calls to vfork.

Clear breakpoints

Clear breakpoints (clear)

clear                    # Clear all breakpoints at current location
clear <function>         # Clear all breakpoints on the function
clear <filename:function># Specify source file and function
clear <linenum>          # Clear breakpoint at a line in the current file
clear <filename:linenum> # Clear by specifying file and line number

Note:clearis based on “location” clearing, not by number.

Delete breakpoints (delete)

delete                   # Delete all breakpoints
delete <bnum>            # Delete breakpoint by specified number
delete <range>           # e.g., delete 3-5 deletes breakpoints numbered 3 to 5
d                        # Abbreviation for delete

Disable/Enable breakpoints (disable / enable)

disable                  # Disable all breakpoints
disable <bnum> [range]   # Disable specific breakpoint
dis                      # Abbreviation for disable

enable                   # Enable all breakpoints
enable <bnum>            # Enable a specific breakpoint
enable <bnum> once       # Automatically disable after one execution
enable <bnum> delete     # Automatically delete after one execution

It is recommended to usedisable/enableto manage debugging state, flexible without losing breakpoint information.

Set/modify breakpoint condition

Set conditional breakpoint (when setting)

1 2	break foo if x > 5 watch var if var == 0

Modify breakpoint condition (during maintenance)

1 2	condition <bnum> x > 100 # Modify condition for breakpoint number bnum condition <bnum> # Clear breakpoint condition

Debug program execution

Resume program execution (continue)

Command	Description
`continue`/`c`/`fg`	Continue running from the current breakpoint
`continue <ignore-count>`	Ignore the next`<count>`breakpoint hits
`run`/`r`	Restart the program (from the beginning)

Suitable when the program has just stopped and you want to skip some breakpoints or continue execution.

Step debugging (source code level)

Command	Description
`step`/`s`	Step execution, enters functions (Step Into)
`next`/`n`	Step execution, does not enter functions (Step Over)
`step <count>`/`next <count>`	Continue execution`<count>`Step

Used to view program logic line by line,stepwill enter the function,nextwill skip it.

Exit the current function (function-level jump out)

Command	Description
`finish`	Continue running until the current function returns, and print the return value and return address

Very practical, suitable for exiting a function after tracing it.

Jump out of loop body / block (until)

Command	Description
`until <location>`/`u`	Execute until a certain position or the end of the current block (suitable for exiting loops)

Example:

1 2	until 42 # Run to line 42 of the current file until main.c:100 # Run to line 100 of main.c

Used to quickly jump out of structural blocks like for/while loops.

Assembly-level single-stepping (instruction-level debugging)

Command	Description
`stepi`/`si`	Single-step one machine instruction (Step Into)
`nexti`/`ni`	Single-step one machine instruction (Step Over)

Used for low-level tracing, such as tracing system calls, libc internal logic, or boot code.

Assembly viewing suggestions:

1	display/i $pc # Display the currently executing instruction in real time

Set step-mode mode

Mainly used to control whether to step into unsymbolized functions

Command	Description
`set step-mode on`	Stop even without debug symbols (default off)
`set step-mode off`	Skip unsymbolized functions (default)

Useful when debugging assembly or library files with only partial symbols.

skip: skip single-step execution of a function

skip function

1	test_str(test.get_str());

If we don’t care about get_str() but want to see test_str(), executing the s command will first enter get_str()

(gdb) s
(gdb) finish

(gdb) skip test_c::get_str
(gdb) s

skip is not jump; although the function is skipped, it still executes, just bypassing debugging

skip file filename

1	(gdb) skip file test.cpp

skip -gfi wildcard

1 2	-gfi可以通过文件名通配符匹配的方式跳过 (gdb) skip -gfi common/.

jump command

Resume execution at a specified location; if a breakpoint exists, execution will stop at that location. If there is no breakpoint, it will not stop, so we usually set a breakpoint at the specified location. The jump command does not change any registers other than the current stack frame, stack pointer, and program counter

# Jumping forward or backward to skip certain lines is fine, but jumping to another function yields unpredictable results
(gdb) jump location
(gdb) j location

# Jumping can be achieved by changing the PC register
(gdb) i line 12
(gdb) p $pc=0x5400000

Core function: force the programto jump to a specified location for execution(can be any line number or address), directly “skipping” the intermediate execution steps

rn reverse execution

First, execute the record command

Must be in non-stop mode

(gdb) record
....
....
# Reverse execute one step
(gdb) rn
# Reverse execute to the beginning of this function
(gdb) reverse-fisnish
(gdb) record stop

Core function: Implementretrospection of program execution history, allowing the program to “run backwards” to investigate errors that have already occurred (e.g., when a variable was unexpectedly modified).
Stop mode (default)
When you executerunorcontinue,the entire program and all threads will pause/resume。
Features:
- Simple and easy to use
- Cannot control threads individually during thread debugging
The default is stop mode.

Non-stop mode

Each thread can be paused or resumed individually without affecting other threads.
Features:
- Allows pausing only a specific thread for debugging in multi-threaded programs
- Supports more flexible thread debugging
How to enable:
1
set non-stop on
Note:
- Does not support certain features, such asprocess record(execution recording) and some remote target features

View runtime data

When the program is paused, use the print command (abbreviated as p) or the synonymous command inspect to view the current program’s runtime data, in the format:

1 2	print <expr> print/<f> <expr>

An expression in the programming language being debugged
Refers to format, for example, outputting in hexadecimal is /x

print(p) output format

Generally, GDB outputs the value of a variable based on its type. However, you can also customize GDB’s output format. For example, you might want to output an integer in hexadecimal or binary to examine the bits of this integer
type variable. To do this, you can use GDB’s data display formats:

x Display variable in hexadecimal format. (hex)
d Display variable in decimal format. (decimal)
u Display unsigned integer in hexadecimal format. (unsigned hex)
o Display variable in octal format. (octal)
t Display variable in binary format. (two)
a Display variable in hexadecimal format. (address)
c Display variable in character format. (char)
f Display variable in floating-point format. (float)

(gdb) p i
$21 = 101
(gdb) p/a i
$22 = 0x65
(gdb) p/c i
$23 = 101 'e'
(gdb) p/f i
$24 = 1.41531145e-43 (gdb) p/x i
$25 = 0x65
(gdb) p/t i
$26 = 1100101

Expressions

Expressions can include const constants, variables, functions, etc., from the current program execution, but cannot be macros defined in the program

Program variables

In GDB, you can view the values of the following three types of variables at any time:

Global variable (visible to all files)
Static global variable (visible to the current file)
Local variable (visible to the current scope)

The value displayed by print will be the value of the local variable in the function. If you want to view the value of a global variable at this point, you can use the “::” operator:

1 2	file::variable function::variable

example

1	gdb) p 'f2.c'::x

Note: If your program is compiled with optimization options enabled, when debugging the optimized program with GDB, some variables may become inaccessible or return incorrect values. This is normal because the optimizer modifies your program, rearranges statement order, and removes meaningless variables. Therefore, when debugging such a program with GDB, the runtime instructions differ from the ones you wrote, leading to unexpected results. To handle this, you need to disable compilation optimization when compiling the program. Generally, almost all compilers support compilation optimization switches. For example, with GNU’s C/C++ compiler GCC, you can use the “-gstabs” option to address this issue.

Array

1	int array = (int ) malloc (len * sizeof (int));

During GDB debugging, you can display the values of this dynamic array with the following command:

1	p *array@len

The left side of @ is the value of the array’s starting address, which is what the variable array points to; the right side is the length of the data, stored in the variable len. The output will look something like this:

1
2

(gdb) p *array@len
$1 = {2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40} 如果是静态数组的话，可以直接用 print 数组名，就可以显示数组中所有数据的内容了。

If it is a static array, you can directly use print with the array name to display the contents of all data in the array.

examine (x) to view memory

Use the examine command (abbreviated as x) to view the value at a memory address. The syntax of the x command is as follows:

1	x/<n/f/u> <addr>

n: The number of units to display, starting from the memory address, display several units (default is 1).
f: Display format, for example:
- x hexadecimal
- d decimal
- t binary
- c character
- f floating point
- s string
- i instruction
u: size of the read unit, determines how many bytes to read each time:
- b = 1 byte
- h = 2 bytes (half word)
- w = 4 bytes (word, default)
- g = 8 bytes (giant/quad word)

view registers

To view register values, simply use info registers (i r)

# View register status (excluding floating-point registers).
info all-registers
# View all register status (including floating-point registers).
info registers <regname ...>

You can also use the print command to access the register status, just add a$symbol before the register name. For example:

p $eip。

View the status of the specified register.
Registers hold data during program execution, such as the current instruction address (ip), the current stack address (sp), and so on. You can also use the print command to access the register status, just add a$symbol before the register name. For example: p $eip.

Automatic Display

You can set some variables to be automatically displayed. When the program stops or when you single-step, these variables will be shown automatically. The relevant GDB command is display.

1
2
3

display <expr>
display/<fmt> <expr>
display/<fmt> <addr>

expr is an expression
fmt indicates the display format
addr indicates a memory address
After you set one or more expressions with display, whenever your program is stopped, GDB will automatically show the values of those expressions.

Formats i and s are also supported by display. A very useful command is:

1	display/i $pc

$pc is a GDB environment variable representing the instruction address, and /i indicates the output format as machine instruction code, i.e., assembly. Thus, when the program stops, the source code and machine instruction code will correspond.

Deleting Automatic Display

To delete automatic display, you can use the following command

1 2	undisplay <dnums...> delete display <dnums...>

dnums refers to the automatically displayed numbers that have been set.
To delete several at once, numbers can be separated by spaces; to delete a range of numbers, use a hyphen (e.g., 2-5).

Hide automatic display

1 2	(gdb) disable display <dnums...> (gdb) enable display <dnums...>

disable and enable do not delete the automatic display settings, but only deactivate and restore them.

View the set automatic display information

1	(gdb) info display

View the automatic display information set by display. GDB will output a table reporting how many automatic display settings are set during the current debugging session, including the setting number, expression, and whether it is enabled.

View function parameters

(gdb) info args
(gdb) i args

# View which functions exist
(gdb) info functions

View local variables

1 2	(gdb) info locals (gdb) i locals

Modify values during execution

Modify the value of a variable

1 2	(gdb) p test.age=28 (gdb) set {int}&test.gender = 110

Modify the value of a register

1 2	set var $pc=xxx p $rip=xxx

Can be used with the info command to view the address of a certain line or the address of a certain stack frame

info line 14
p $pc=0x5555555553a3

info f 0

View the function call stack

backtrace/btView stack backtrace information

frame nSwitch stack frame

info f nView stack frame information

View data type information

whatis

(gdb) whatis test1
(gdb) whatis test2
(gdb) whatis node
(gdb) whatis main
(gdb) whatis test3.test_fun2

ptype

1	(gdb) ptype test1

Can display member variables, functions, etc. of this class

ptype /m

1	(gdb) ptype /m test1

Do not display member functions

ptype /t

1	(gdb) ptype /t test3

Do not display typedef

ptype /o

1	(gdb) ptype /o node

View offset and size of struct

i variables

1	(gdb) i variables count

Will display definitions of all variables whose names contain ‘count’

set print object on

1 2	(gdb) set print object on (gdb) ptype test2

set print object on means to display the derived type of the class

Call internal and external functions

p expression

Evaluate the expression and display the result. The expression can include calls to functions in the program being debugged; even if the function returns void, it will be displayed

call expression

Evaluate the expression and display the result; if it is a function call and the return value is void, the void return value is not displayed

p sizeof(int)
p strlen(name)

call (int)getpid()

call malloc(10)
call strcpy($5,"soft")

Also strcmp, printf, etc.

Debugging multithreaded programs

Thread management commands

info threads

1 2	(gdb) info threads (gdb) i threads

View all thread information; an asterisk before the number indicates the current thread

Thread is followed by the thread address
LWP stands for Light Weight Process (you can use ps -aL in the command line to view all lightweight threads)
When naming threads, the process name is used by default

thread find

Find threads

1 2	(gdb) thread find multh (gdb) thread find 689519

The search scope includes thread, address, LWP, and thread name, which are the first three items of ‘i threads’

thread num

1
2
3

(gdb) bt
(gdb) thread 2
(gdb) bt

Switch threads; ‘bt’ only shows the call stack of the current thread

thread name

Set thread name; this sets the name of the current thread

1	(gdb) thread name main

b breakpoint thread id

Set breakpoints for individual threads

1	(gdb) b 15 thread 2

If using a regular ‘b 15’, then all threadsthat can execute to line 15will stop

thread apply

Execute commands for threads

(gdb) thread apply 3 i args
(gdb) thread apply 1 2 3 i args
(gdb) thread apply 1-3 5 i args
(gdb) thread apply all i locals

-q Do not display thread information

-s Do not display error messages

Both parameters must be placed after the thread ID

set scheduler-locking off|on|step

set scheduler-lockingBy setting a locking strategy, restrict the execution of other threads to ensure the debugging focus remains on the target thread

Parameter	Effect	Applicable Scenario
`off`	Locking disabled (default), the scheduler can freely switch between all threads for execution.	When observing multi-thread interaction (e.g., thread synchronization, communication), allow other threads to execute naturally.
`on`	Enable full locking,only the currently debugged thread can execute, other threads are paused.	Focus on debugging the logic of a single thread (e.g., function call chain, local variable changes) to avoid interference from other threads.
`step`	Step-by-step locking, only during single-step execution (`step`) lock other threads,`continue`Unlock at time.	During single-step debugging, other threads need to be isolated, but it is hoped that`continue`the program can run normally in multi-threaded mode afterwards.

# View current scheduler-locking configuration
(gdb) show scheduler-locking
scheduler-locking is off.

# Set to full lock (on) to ensure the currently debugged thread exclusively occupies the core
(gdb) set scheduler-locking on

# When single-stepping kernel code, set to step lock (step)
(gdb) set scheduler-locking step

# After debugging, restore to default (off)
(gdb) set scheduler-locking off

Print thread event information

1 2	(gdb) show print thread-events (gdb) set print thread-events off

Multi-threaded deadlock debugging

Deadlock conditions:

Mutual exclusion condition
Hold and wait condition
No preemption condition
Circular wait condition

Common commands:

thread 2
bt
f 2
p _mutex_2

Ways to resolve deadlocks:

Use locks in order
Control the scope of locks
Can use a timeout mechanism

Debugging multi-process programs

Basic concepts

inferior

GDB uses inferior to represent the state of a process being debugged. Typically, one inferior represents one process. This is an internal concept and object of GDB, and we can attach a running process to an inferior.

(gdb) i inferiors
(gdb) add-inferior
(gdb) remove-inferior 1


(gdb) !ps aux | grep rele
(gdb) attach 2959251

(gdb) i inferiors
(gdb) detach inferior 2

set schedule-multiple on/off

Allow multiple processes to execute simultaneously

1 2	(gdb) show schedule-multiple (gdb) set schedule-multiple on

Debug child processes

By default, child processes cannot be debugged

set follow-fork-mode child/parent

By default, follow-fork-mode is set to parent, meaning debugging follows the parent process. Only when set to child will it follow the child process.

1 2	(gdb) set follow-fork-mode child (gdb) show follow-fork-mode

set detach-on-fork on/off

By default, detach-on-fork is on, meaning when debugging a child process, the parent process continues executing and may finish directly, making it impossible to debug the parent. Therefore, to debug both the parent and child processes simultaneously, you need to

1	(gdb) set detach-on-fork off

info inferiors

View processes started by GDB, i.e., GDB’s child processes

inferior process_num

Switch to child process number 1

1	(gdb) inferior 1

Creating a debug release

The release requires a version without debug information, while also retaining a version with debug information for convenient debugging

Method 1: Two versions, one without the -g flag and one with the -g flag

Write two compilation tasks in the Makefile, one with the -g flag and one without the -g flag

Method 2: The strip command

strip -g debug_version.o release_version.o

# Debugging
gdb --symbol=debug_version.o -exec=release_version.o

Method 3: The objcopy command

# Generate a pure debug symbol file
objcopy --only-keep-debug debug_version.o debug.sym
# Debugging
gdb --symbol=debug.sym -exec=release_version.o

Directly editing the executable file

Add the --write parameter when starting gdb

1
2
3

(gdb) gdb --write test.o
# Display the assembly and machine code corresponding to the source code
(gdb) disassemble /mr check_some

What we need to modify is the machine code; exit directly after modification

1 2	(gdb) p {unsigned char}0x00000000000011b4=0x65 (gdb) q

Memory checking

Memory leak checking

call malloc_stats()

1	(gdb) call malloc_stats()

Arena0 represents the memory data used by the current thread
Total represents the memory data used by the entire process

call malloc_info(0, stdout)

1	(gdb) call malloc_info(0, stdout)

The output is in XML format, mainly focus onrest

gcc option -fsanitize=address

Check for memory leaks

Check for heap overflow

Check for stack overflow

Check for global memory overflow

Check for use-after-free

Remote debugging

Server side / debugged machine

Install gdbserver, start gdbserver

ifconfig

sudo apt-get install gdbserver

# This port can be chosen arbitrarily, but must not conflict with other programs and must not be blocked by the firewall
gdbserver 10.20.50.83:9988 ./test.o

# If the program is already running, first get the program PID via the ps command, then
gdbserver 10.20.50.83:9988 --attach pid

Client side / debugging machine

gdb remote connection and debugging

(gdb) target remote 10.20.50.83:9988
# Entering quit in the remote program does not cause it to terminate
(gdb) quit
# You can also use detach
(gdb) detach