From 1027cd52a2ee06a5a0e76cce9eab1f6ce2461ac0 Mon Sep 17 00:00:00 2001 From: Brandon Hurst Date: Mon, 13 Apr 2026 23:24:43 -0700 Subject: [PATCH] products/adsp: Add early kernel debug info Add a document for ADSP Linux showing how to debug the early kernel boot before even the early console can be up and running. This will assist developers who are working on crashing systems. Testing performed is exactly as described in the document. Signed-off-by: Brandon Hurst --- docs/products/adsp/debug-early-boot.rst | 257 ++++++++++++++++++++++++ docs/products/adsp/index.rst | 1 + 2 files changed, 258 insertions(+) create mode 100644 docs/products/adsp/debug-early-boot.rst diff --git a/docs/products/adsp/debug-early-boot.rst b/docs/products/adsp/debug-early-boot.rst new file mode 100644 index 00000000000..2070b6e02d2 --- /dev/null +++ b/docs/products/adsp/debug-early-boot.rst @@ -0,0 +1,257 @@ +.. _adsp debug-early-boot: + +Use JTAG to debug early Linux booting +======================================= + +Debugging early boot problems with Embedded Linux can be difficult, especially +when the system isn't able to boot far enough to provide console logging over +a serial or network connection. One way to attain more visibility for early +boot is to build the kernel with debug symbols, and use a JTAG debugger with +GDB to step through the boot process. + +Before proceeding with this guide, please make sure to follow the +:ref:`adsp setup` guide and one of the `ADSP Getting Started Guides +`_ for the given +target hardware. Ensure you have built a kernel and populated the SDK for your +build target. + +Background (Build System) +~~~~~~~~~~~~~~~~~~~~~~~~~ + +The build system should already output a ``vmlinux`` file, which is an +uncompressed and statically linked ELF (Executable and Linkable Format) file +with the Linux kernel binary. This file must include debug symbol information +to be used with GDB for early boot debugging. Many build systems also produce +files like ``vmlinuz``, ``Image``, ``zImage``, or ``fitImage``, which are +stripped of debug symbols and compressed for loading to the target hardware. + +To check whether a given ``vmlinux`` file contains debug symbols, use the +``file`` or ``readelf`` utilities. The following example shows the ``file`` output +for a modified :git-lnxdsp-adi-meta:` v5.0.1 Yocto release ` +with a :git-linux:`6.12 Linux release `, but the +instructions on this page will work for any release. + +.. code-block:: bash + + file -b tmp/work/adsp_sc589_mini-adi_glibc-linux-gnueabi/linux-adi/6.12/linux-adsp_sc589_mini-standard-build/vmlinux + ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked, BuildID[sha1]=d36ff404bbf4e711b3e6476f987c01747f7960fb, with debug_info, not stripped + +The ``with debug info, not stripped`` indicates this file contains debug symbols. + +To build a kernel with debug symbols and a debuggable configuration, some extra +debug symbols need to be set to "y" via Kconfig. Yocto provides many ways of +adding custom configuration which include, among others: + +1. Altering sources directly, cleaning the shared state cache and rebuilding +2. Providing a custom meta-layer for experimental changes or patches +3. Modifying a source via devtool (either temporarily or committing to a patch) +4. Using menuconfig, e.g. ``bitbake -c menuconfig linux-adi`` + +To demonstrate the simplest example, we will alter the sources directly. + +Build with a Debug Configuration +******************************** + +To start, follow one of the :ref:`adsp setup` guides to complete a kernel build +and populate an SDK. This will ensure that a kernel rebuild is expedited and +that an SDK is available to use OpenOCD and GDB for debugging with. + +Once a kernel build is in place, the following Kconfig symbols must be +enabled: + +- **CONFIG_DEBUG_KERNEL** to enable kernel debug info options +- **CONFIG_DEBUG_INFO_DWARF4** to set debug info format to DWARF4 +- **CONFIG_FRAME_POINTER** to keep frame pointers for backtraces + +Add the following file in the Yocto sources: + +.. code-block:: kconfig + :caption: sources/meta-adi/meta-adi-adsp-sc5xx/recipes-kernel/linux/linux-adi/feature/cfg/debug-symbols.cfg + + # Enable kernel debug symbols for JTAG/GDB debugging + # Use DWARF4 format and keeps frame pointers for stack backtraces + CONFIG_DEBUG_KERNEL=y + CONFIG_DEBUG_INFO_DWARF4=y + CONFIG_FRAME_POINTER=y + +Then modify the ``linux-adi_%.bb`` file to include the new configuration: + +.. code-block:: text + :emphasize-lines: 8,13 + :caption: sources/meta-adi/meta-adi-adsp-sc5xx/recipes-kernel/linux/linux-adi_%.bb + + # ... + LICENSE="GPL-2.0-only" + LIC_FILES_CHKSUM="file://COPYING;md5=6bc538ed5bd9a7fc9398086aedcd7e46" + + DEPENDS += "u-boot-mkimage-native dtc-native" + + # Include kernel configuration fragments + # NOTE: Added debug-symbols.cfg + SRC_URI:append="\ + file://feature/cfg/nfs.cfg \ + file://feature/cfg/crypto.cfg \ + file://feature/cfg/tracepoints.cfg \ + file://feature/cfg/debug-symbols.cfg + " + # ... + +Next, rebuild the kernel with the following steps: + +.. code-block:: bash + + # Clean Yocto's shared state cache for the linux-adi recipe, + bitbake -c cleansstate linux-adi + + # Rebuild linux-adi + bitbake linux-adi + + # Copy vmlinux file to known location + cp tmp/work/adsp_sc589_mini-adi_glibc-linux-gnueabi/linux-adi/6.12/linux-adsp_sc589_mini-standard-build/vmlinux \ + $HOME/workspace/sc589-mini-5.0.1/vmlinux-dbg + +This will store a kernel image with debug symbols to a known location. +This file will most likely be well over 100 MiB, compared to the stripped +kernel loaded to the target which will likely be 10-15 MiB depending on the +enabled features. + +Finally, rebuild the complete Linux image including the corresponding stripped +zImage file: + +.. code-block:: bash + + # Clean sstate cache for adsp-sc5xx-minimal, then rebuild + bitbake -c cleansstate adsp-sc5xx-minimal + bitbake adsp-sc5xx-minimal + + # Copy stripped loadable image to target boot media, e.g. http server + cp tmp/deploy/images/adsp-sc589-mini/fitImage ~/image-* + +Load the Image +************** + +Load the debug image to the target using your chosen boot media (SD, eMMC, +HTTP, etc.). When booting the target, stop the U-Boot bootloader by +pressing "Enter" before booting the kernel. + +Connect OpenOCD +*************** + +In one terminal, connect OpenOCD with the following commands (can be +placed into a script for ease of use): + +.. code-block:: bash + :caption: openocd-kernel-dbg.sh + + #!/bin/bash + export sdk_usr=/opt/adi-distro-glibc/5.0.1/sysroots/x86_64-adi_glibc_sdk-linux/usr/ + + # Use skip_reset command to avoid resetting the target + $sdk_usr/bin/openocd \ + -f $sdk_usr/share/openocd/scripts/interface/ice2000.cfg \ + -f $sdk_usr/share/openocd/scripts/target/adspsc58x.cfg \ + -c "skip_reset" + +Connect GDB +*********** + +In a separate terminal, connect GDB with the following command (can be +placed into a script for ease of use): + +.. code-block:: bash + + GDB="/opt/adi-distro-glibc/5.0.1/sysroots/x86_64-adi_glibc_sdk-linux/usr/bin/arm-adi_glibc-linux-gnueabi/arm-adi_glibc-linux-gnueabi-gdb" + VMLINUX="$HOME/workspace/sc589-mini-5.0.1/vmlinux-dbg" + LOAD_ADDR="0xC3000000" + + "$GDB" "$VMLINUX" \ + -ex "target extended-remote localhost:3333" \ + -ex "set print pretty on" \ + -ex "add-symbol-file $VMLINUX $LOAD_ADDR" + +A script can be used to simplify this step. + +.. code-block:: bash + :caption: gdb-kernel-dbg.sh + + #!/bin/bash + # GDB helper script for SC589-MINI kernel debugging + + GDB="/opt/adi-distro-glibc/5.0.1/sysroots/x86_64-adi_glibc_sdk-linux/usr/bin/arm-adi_glibc-linux-gnueabi/arm-adi_glibc-linux-gnueabi-gdb" + VMLINUX="$HOME/workspace/sc589-mini-5.0.1/vmlinux-dbg" + LOAD_ADDR="0xC3000000" + + if [ ! -f "$VMLINUX" ]; then + echo "Error: vmlinux not found at $VMLINUX" + echo "Please build the kernel first with: bitbake linux-adi" + exit 1 + fi + + if [ ! -f "$GDB" ]; then + echo "Error: GDB not found at $GDB" + exit 1 + fi + + echo "Starting GDB for SC589-MINI kernel debugging..." + echo "GDB: $GDB" + echo "vmlinux: $VMLINUX" + echo "Load address: $LOAD_ADDR" + echo "" + + "$GDB" "$VMLINUX" \ + -ex "target extended-remote localhost:3333" \ + -ex "set print pretty on" \ + -ex "add-symbol-file $VMLINUX $LOAD_ADDR" + +Notes on Using GDB to Debug Early Boot +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Now that a GDB connection has been established, please be aware that there are +a couple potential obstacles when debugging early boot: + +- The kernel boots using physical addresses until the MMU is enabled +- The vmlinux file contains virtual addresses, which may not easily translate + +To mitigate these issues, use hardware breakpoints (``hbreak`` in GDB) to +capture early execution e.g. ``hbreak start_kernel`` to break on the start_kernel +function. Single stepping across MMU enabling code should also be avoided. + +Useful GDB Commands +~~~~~~~~~~~~~~~~~~~ + +Below is an example of debugging the kernel at early boot time that shows some +useful commands to use. Within GDB, the "help" command can also be useful to +get more information about these commands. Other useful resources are the GDB +documentation and `ADI's GDB Command Reference `_. + +The below example sets a breakpount while the target is in U-Boot, which gets +triggered once U-Boot loads the kernel and starts booting: + +.. code-block:: bash + + (gdb) # Set kernel source code path + (gdb) set substitute-path /usr/src/kernel /build/tmp/work-shared/adsp-sc589-mini/kernel-source + (gdb) + (gdb) # hardware breakpoints at start_kernel + (gdb) hbreak start_kernel + Hardware assisted breakpoint 1 at 0xc0b00da0: file /usr/src/kernel/init/main.c, line 904. + (gdb) c + Continuing. + + Breakpoint 1, start_kernel () at /usr/src/kernel/init/main.c:905 + 905 char *command_line; + (gdb) l + 900 } + 901 + 902 asmlinkage __visible __init __no_sanitize_address __noreturn __no_stack_protector + 903 void start_kernel(void) + 904 { + 905 char *command_line; + 906 char *after_dashes; + 907 + 908 set_task_stack_end_magic(&init_task); + 909 smp_setup_processor_id(); + (gdb) bt + #0 start_kernel () at /usr/src/kernel/init/main.c:905 + #1 0x00000000 in ?? () + (gdb) diff --git a/docs/products/adsp/index.rst b/docs/products/adsp/index.rst index 4a0a87ddc8c..8cbb724e723 100644 --- a/docs/products/adsp/index.rst +++ b/docs/products/adsp/index.rst @@ -58,3 +58,4 @@ Cortex-A5 and Cortex-A55 can also run 32-bit or 64-bit Linux respectively. setup sc598-clock-tree use-cases + debug-early-boot