Merge feature/vnext-delta into develop#865
Merged
Conversation
* Update x509 doc * Add X.509 demo script with device-specific naming and DU config generation - 8-step validation, dual thumbprints, IoT Hub connectivity testing - File structure reporting and Ubuntu 20.04 disclaimer * Fixed shellcheck error * docs: add comprehensive X.509 testing framework with dependency management Enhance X.509 authentication documentation and demo script with smart dependency caching, service-aware testing, comprehensive validation, and extensive troubleshooting guidance. Includes build automation and compilation fixes for seamless development workflow. * fix: resolve sign-conversion warnings in command_helper and hash_utils - Fix safeReadSize type usage in command_helper.c - Add explicit cast for loop index in hash_utils.c - Improve type safety in buffer and array operations * feat: enhance X.509 authentication with dual certificates and mandatory IoT Hub validation - Add dual certificate generation (primary/secondary) with unique SHA1 thumbprints for X.509 authentication - Implement mandatory IoT Hub hostname validation for agent testing scenarios - Update demo-setup.sh with comprehensive interactive mode and enhanced certificate management - Add Microsoft package repository setup to resolve deviceupdate-agent dependency issues - Extend agent timeout from 15s to 60s for better testing and validation - Update configuration schema version to 1.2 with new X.509 connection format - Improve connection string validation logging and error messages - Add remote debugging configuration and version updates (1.2.1-private-preview) - Enhance X.509 documentation with production-ready certificate examples Major improvements: * Interactive certificate setup with device registration guidance * Dual certificate workflow supporting IoT Hub primary/secondary thumbprints * Smart dependency management with Microsoft repository integration * Enhanced validation with 8-step certificate verification process * Backward compatibility maintained through symlinks and configuration migration Fixed shellcheck issues: * Add -r flag to all read commands to prevent backslash interpretation * Replace 'cat file | sed' with 'sed < file' for better efficiency * Fix sudo redirect issue by using pipe to tee instead of direct redirection * Add shellcheck source directive for /etc/os-release inclusion * - Add comprehensive parameter change detection and connection testing - Fix output hub hostname * Update x509 demo script and document Remove test-device-twin.sh
* Add prelim doc for GetAduServiceStatus API * Add ADUC_ServiceStatus_Pausing * Update description of command handler and view state manager Clarified the explanation of how the CommandHandler interacts with the ViewStateManager and the ADU service status. * Add packaging, pkgcfg, versioning, ipc sections * update diagrams to include Paused state * fix so lib name i state flow diagram * Add apisvc get_state crossproc api * add utils timer * add idlePauseMilliseconds config * Add out-of-proc SDK wrapper API, docs, and examples * apiproto_utils * arm64 rpi4 integration fixes * updates to doc and unused cmake var * fix typo and apiproto hdr lnk * fix indent adu sdk hdr * rmv stale doc param * set viewstate to Failed on report json failure * put extern g_vsm in hdr * rev schemaVersion 1.2 du-config * add MAX_BUF_LEN macro comment * replace 3 sizeof u16 with MSG_HDR_LEN * add links to yocto repo for sample yocto integration via status_monitor * add missing file headers * add missing file hdr for viewstatemgr * log warn when not init'ed in getter * add RESP_MSG_READ_BUF_SIZE
* flush and wait before close resp fifo * further improvements for rpi4 * fix sdk examples zlog indirect dep and clangformat
* Fix example manufacturer and model fields and add IdlePauseMilliseconds to an example config
…kaging Features: - New cmake option: ADUC_BUILD_DELTA_HANDLER (default: OFF) - Integration with iot-hub-device-update-delta library - Separate Debian package: deviceupdate-agent-delta-handler - Component-based packaging fixes to generate 2 packages Changes: - Added FindAzureIotHubDeviceUpdateDelta.cmake finder module - Updated install-deps.sh with --install-delta support - Updated build.sh with --delta-handler flag - Fixed packages/CMakeLists.txt for proper component packaging - Added comprehensive documentation (850+ lines) - Changed work folder to .adu-tmp to avoid vcpkg conflicts - Added package maintainer scripts for handler lifecycle
Implements incremental build optimization for result.h generation by
adding SHA256 hash-based change detection.
Changes:
- CMakeLists.txt: Added hash-based caching for result.h generation
- Calculates SHA256 hashes of result_codes.json and generator script
- Stores hash in build directory (.result_h_generation_cache)
- Only regenerates result.h when source files actually change
- Added ADUC_FORCE_REGENERATE_RESULT_H option to bypass cache
- Detailed diagnostic messages showing regeneration reason
- .gitignore: Added cache file pattern and test directory exclusions
Benefits:
- Faster CMake reconfiguration when result_codes.json unchanged
- Avoids unnecessary file generation and timestamp updates
- Preserves configure-time generation behavior
- No structural changes to build system
Usage:
- Normal: result.h regenerated only when sources change
- Force: cmake -DADUC_FORCE_REGENERATE_RESULT_H=ON
Cache file location: ${CMAKE_BINARY_DIR}/.result_h_generation_cache
- result.h: Add SHA256 hash-based caching to skip regeneration when sources unchanged - build.sh: Skip CMake reconfiguration on incremental builds (only run on first build or with -c flag) - Faster back-to-back builds by avoiding unnecessary CMake overhead Cache stored in build dir (.result_h_generation_cache), cleaned with build output.
Removed 'debian:12' from the OS matrix in the Docker build workflow. We'll add debian:12 support once iot-hub-device-update-delta support debian:12
…ADUC_BUILD_UNIT_TESTS=ON during a cross-compile build. Without this check, CMake would fail at configure time trying to execute an ARM binary.
- Add adu-reboot-wrapper.sh that waits for agent lock file - Modify adu-shell to use wrapper instead of direct reboot - Add ADUC_Result_Apply_RebootPending (709) result code - Agent creates lock, completes reporting, then removes lock - Prevents lost telemetry and state corruption on update reboot
…ance logging New Features: - Add CacheSourceUpdate API for pre-reboot source file caching - Log bandwidth savings on successful delta reconstruction Cache Improvements: - Add hash verification before caching (prevents corruption) - Skip caching if file already exists with valid hash - Fix file copy fallback when rename fails across mount points - Add disk space check before copy operation - Create .info metadata files for cached updates - Add null pointer guards for FileEntity hash fields Logging/Observability: - Add [DELTA] prefix to all delta handler logs for filtering - Add [TIMING] prefix for performance metrics - Document log patterns and monitoring commands in README Documentation: - Add Delta Update Flow Overview diagrams - Add End-to-End Integration Guide - Expand troubleshooting sections
…odes for Steps Handler (microsoft/swupdate:2)
Includes fix for STRING_HANDLE cleanup in microsoft_delta_download_handler_utils.c
…S=ON, which is the project's standard flag for controlling unit test builds and is properly disabled during Yocto cross-compilation.
- install-deps.sh: Add SUDO to chmod for permission handling, fix all shellcheck warnings (quote variables, fix glob with -f) - sh-format.sh: Update shellcheck path to use ~/.adu-tmp instead of /tmp - system_utils.c: Fix feof() bug causing 0-byte files, add null pointer checks in cleanup - system_utils_ut.cpp: Add 8 regression tests for CopyFileToDir - swupdate_handler_v2_ut.cpp: Update expected output to include --workflow-id parameter
- Clone and bootstrap vcpkg directly instead of using setup_vcpkg.sh - Create x64-linux triplet programmatically if missing in community triplets - Use --classic flag in vcpkg install to avoid manifest mode conflicts - Fix apt-get command to list packages separately (not quoted string) - Add VCPKG_OVERLAY_TRIPLETS for community triplet discovery - Skip vcpkg stage in build.sh since dependencies installed manually Tested successfully in Docker debian:11 container simulating CI.
The install-deps.sh script now auto-detects the CPU architecture using 'uname -m' and selects the appropriate vcpkg triplet: - x86_64/amd64 -> x64-linux - aarch64/arm64 -> arm64-linux - armv7l/armhf -> arm-linux This fixes GitHub Actions CI failures on arm64 runners where the hardcoded x64-linux triplet would fail to build.
- Add open_fifo_with_timeout() helper using poll() for non-blocking I/O - Replace blocking FIFO open() calls with timeout-based operations - Prevents indefinite hangs when FIFO peer is not ready - Uses 10-second timeout for FIFO operations The crossproc tests were timing out (1500+ seconds) in Docker containers because blocking open() on FIFOs waits indefinitely for a peer. Now uses O_NONBLOCK with poll() to timeout gracefully. Test now completes in ~1.4 seconds.
Debian 12 (Bookworm) ships with GCC 12 and does not have gcc-10 in its repositories. This fix adds OS/version detection to the do_install_delta function to use the correct GCC version: - Debian 12: gcc-12/g++-12 - Debian 11, Ubuntu 20.04/22.04: gcc-10/g++-10 - Other distros: system default GCC This matches the existing logic in do_install_aduc_packages and fixes the Docker build failure on Debian 12 arm64: 'E: Unable to locate package gcc-10'
…ponse FIFO - Change g_api_svc_thread_running from bool to atomic_bool to prevent compiler/CPU caching and reordering issues between threads - Replace retry-count-based loop with absolute timeout (10 sec) using CLOCK_MONOTONIC for response FIFO open - Add stdatomic.h and time.h includes This fixes potential race conditions where: 1. The thread running flag could be cached in registers 2. Response FIFO open could block indefinitely if client dies
…the Debian 12 docker builds
- The cross-compiler cannot be upgraded to GCC 10+ easily - Ubuntu 20.04 reached end of standard support in April 2025 - The amd64 build of Ubuntu 20.04 still works (uses native GCC 10)
This reverts commit c7337b2.
…mprovements' into henrymorales/unit-test-updates-1
Several Unit Test Updates
Merged 26 commits from develop branch including: - Ubuntu 24.04 LTS / GCC 13 compatibility - Debian 13 (Trixie) support - Use curl handler as default content handler - Pass workflowData to PrepareScriptArguments - Overall agent logging improvements - Rootkey Validator Tool - Fix zlog memory leak - Report detailed error for failed workflows - Show ExtendedResultCode in IoTHub details - Fix wrong documentation (array bounds) - Extensive unit test additions across modules - Code coverage tooling Conflicts resolved favoring staging for delta update handler, service status API, and X.509 authentication features. Bug fixes and unit tests from develop preserved.
Based on official release notes from GitHub releases tag 1.2.0. Includes security bug fixes, other bug fixes, usage enhancements, and code enhancements.
Covers all major changes since 1.2.0 including: - X.509 client certificate authentication - CrossProc Query API (Service Status API) - Delta Download Handler with CacheSourceUpdate API - Ubuntu 24.04 and Debian 13 platform support - Curl as default content handler - 12+ memory leak fixes (Valgrind-verified) - Thread safety improvements - Extensive unit test coverage additions
- Rewrite docs/agent-reference/README.md: fix all 18 broken links, add all 14 previously unlinked docs, update features table for v1.3.0, add What's New section, fix stale prerequisites (remove EOL platforms) - Create architecture-overview.md: components, security, update lifecycle - Create quick-start.md: clone-to-running in 15 minutes - Create configuration-guide.md: du-config.json schema with all options - Update how-to-install-deb-pkg-on-rpi.md: remove Debian 10, add Ubuntu 24.04, Debian 11/12 for RPi - Update how-to-troubleshoot-guide.md: add 1.3.0 troubleshooting (X.509, delta handler, service status API), fix stale links
- Fix typo: stept_handlers → step_handlers in custom handler doc - Fix broken relative link to troubleshoot guide in X.509 doc - Fix schema version conflict: quick-start now uses schemaVersion 1.2 - Update 7 stale docs.microsoft.com URLs to learn.microsoft.com
Test fixes (all caused by bad merge conflict resolution): - agent_workflow_ut.cpp: restore deleted MockReportStateCallback_WithCounter function, add missing closing brace in SetReceivedC2D section - diagnostics_devicename_ut.cpp: remove broken DiagnosticsDeviceNameTestHelper class fragment duplicated before extern C block - config_utils_ut.cpp: add missing closing braces in apiRequestFifoPath and idlePauseMilliseconds sections, add config setup for GetAduShellTrustedUsers - operation_id_utils_ut.cpp: remove broken TEST_CASE wrapping includes, remove duplicate test sections and orphaned code fragments - system_utils_ut.cpp: restore 6 missing closing braces between TEST_CASEs - adushell_ut.cpp: update reboot test to match adu-reboot-wrapper.sh change Curl downloader improvement: - rootkeypackage_curl_download.cpp: add --connect-timeout (default 30s) to prevent long waits on unreachable hosts, add compile-time configurable timeout defines (ADUC_CURL_CONNECT_TIMEOUT_SECS, ADUC_CURL_MAX_TIME_SECS), add diagnostic logging for specific curl exit codes (timeout, DNS, connect) Build verified: 695/695 unit tests pass.
- Add --connect-timeout (default 30s) to prevent long waits on unreachable hosts or invalid URLs during root key package download - Add compile-time configurable timeout defines: ADUC_CURL_CONNECT_TIMEOUT_SECS (default 30) and ADUC_CURL_MAX_TIME_SECS (default 3600) - Add diagnostic logging for specific curl exit codes: code 6 (DNS resolution failure), code 7 (connection refused), code 28 (operation timed out), with timeout values in message
The 'main' branch of iot-hub-device-update-delta uses 'enum class algorithm : uint32_t' which GCC 14 (Debian 13, Ubuntu 24.04) rejects with 'elaborated-type-specifier for a scoped enum must not use the class keyword'. The feature/vnext-delta branch renames this to 'enum adu_algorithm' which compiles on all supported GCC versions. Changes: - Set default delta_ref to 'feature/vnext-delta' instead of 'main' - Simplify GCC compatibility override to redirect any 'main' ref to 'feature/vnext-delta' automatically - Remove Debian 12-only override (now covered by the default)
…ta build
The feature/vnext-delta branch of iot-hub-device-update-delta changed
from find_package() to pkg_check_modules() for bsdiff and zstd discovery
(for Yocto compatibility). However, the bsdiff vcpkg overlay port only
ships Findbsdiff.cmake — no .pc file — causing pkg_check_modules to fail.
Additionally, the delta CMakeLists uses bare -l flags (e.g. -lbsdiff,
-lzstd) from pkg-config without link_directories, so the linker cannot
locate libraries installed under vcpkg's prefix.
This commit:
- Generates a relocatable bsdiff.pc (using ${pcfiledir} pattern) with
all transitive dependencies (-lbsdiff -ldivsufsort -ldivsufsort64 -lbz2)
- Exports PKG_CONFIG_PATH pointing to vcpkg's pkgconfig directory
- Exports LIBRARY_PATH so the linker finds vcpkg-installed static libs
- Cleans up both env vars after the delta build completes
- Patch dumpextfs CMakeLists.txt to link com_err (required by static libext2fs) - Patch recompress CMakeLists.txt to link libconfig (required by static libconfig++) - Add dependency skip logic using stamp files to avoid rebuilding existing deps - Add git and safe.directory config to Docker CI prereqs
- install-deps.sh: Add shellcheck disable directives for SC2016 on sed lines that use CMake variables (not shell variables) in single quotes - sh-format.sh: Remove unused work_folder variable (SC2034) - adu-reboot-wrapper.sh: Quote $TIMEOUT variable to prevent globbing and word splitting (SC2086) Co-authored-by: Daniel Mirsky <danielmirsky@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…docs Merge "develop" back to vnext-delta
* fix: Fix CMAKE in rootkey workflow (#842) * fix: Fix segmentation fault (#844) * fix: Fix double-free in logging library (#843) * feat: ARM 32 compatibilty fixes (#841) * Fix ARM32 segfault in extension registration: use %ld for long instead of %lld The %lld format specifier consumes 8 bytes but long is 4 bytes on ARM32, causing subsequent varargs (hash pointer) to be misread and segfaulting in strlen(). Regression introduced in PR #633 when %d was changed to %lld without updating the variable type from long to long long. * Add Debian 11 ARM32 build job using QEMU emulation Adds arm32-container-job that builds on ubuntu-latest with QEMU ARM emulation via docker/setup-qemu-action. Runs the full build pipeline (install-deps, build, test) inside a debian:11 arm/v7 Docker container. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Prioritize dpkg for detecting userspace architecture Update determine_machine_architecture function to prefer dpkg for userspace architecture detection. * Replace CMake 3.20+ commands with CMake 3.5-compatible alternatives Replace cmake_path() and file(COPY_FILE) calls with get_filename_component() and configure_file(COPYONLY) respectively, to match the project's cmake_minimum_required(VERSION 3.5). Also add out*/ to .gitignore to cover build output directories with suffixes (e.g. out-debian11-arm32/). * support armhf * arm64 * remove arm32 buid --------- Co-authored-by: nox-msft <wewilair@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Nox-MSFT <55153324+Nox-MSFT@users.noreply.github.com> * fix: Fix Curl downloader default (#839) * fix group memberships * make choice * make choice multiline * back to defaults * default curl * content downloader message * fix: Fix coverage issues (#848) * change it back to CPACK_DEBIAN_UNSPECIFIED_PACKAGE_DEPENDS,CPACK_DEBIAN_UNSPECIFIED_PACKAGE_SUGGESTS * fix: Make DO recommended when built with DO support (#847) * fix: Make DO a suggestion when built with DO support * don't break * doc * exclued cmakefiles fom coverage * fix another bug * undo coverage changes * set rootkeypkg_curl to true --------- Co-authored-by: Chinonso Chukwuogor <nonsochukwuogor@gmail.com> Co-authored-by: nox-msft <wewilair@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Nox-MSFT <55153324+Nox-MSFT@users.noreply.github.com> Co-authored-by: yinaliu <yinaliu@microsoft.com>
Co-authored-by: yinaliu <yinaliu@microsoft.com>
…ags (#859) The ADUC_ENABLE_FILE_LOG and ADUC_ENABLE_CONSOLE_LOG CMake options (ON/OFF) were passed directly to add_definitions, producing -DADUC_ENABLE_FILE_LOG=ON. In C, #if ON evaluates the undefined identifier ON as 0, silently disabling file logging entirely. Convert to 1/0 so the preprocessor guards in src/logging/zlog/src/init.c work correctly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove file that accidentally added.
…der (#864) Merge develop into feature/vnext-delta — bug fixes, ARM32, V2 downloader Key changes: - Bug fixes: segfault, double-free, hash check, curl downloader default, cmake paths - ARM32 compatibility fixes - V2 content downloader contract with backward compatibility - Prevent unnecessary regeneration of swupdate_handler_result_codes.h Validation: Build (344 targets) ✅ | Tests (702/702) ✅ Co-authored-by: nox-msft wewilair@microsoft.com (mailto:wewilair@microsoft.com) Co-authored-by: Nox-MSFT 55153324+Nox-MSFT@users.noreply.github.com (mailto:55153324+Nox-MSFT@users.noreply.github.com) Co-authored-by: Chinonso Chukwuogor nonsochukwuogor@gmail.com (mailto:nonsochukwuogor@gmail.com) Co-authored-by: AndreRicardo-Zoetis 122284839+AndreRicardo-Zoetis@users.noreply.github.com ( mailto:122284839+AndreRicardo-Zoetis@users.noreply.github.com) Co-authored-by: Andre Ricardo andre.ricardo@zoetis.com (mailto:andre.ricardo@zoetis.com)
…sft/nox-msft/vnext-delta-to-develop # Conflicts: # CMakeLists.txt # packages/CMakeLists.txt # src/logging/zlog/src/init.c
chgennar
approved these changes
Apr 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
9 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Merges the feature/vnext-delta feature branch into develop, bringing in all major features developed on vnext-delta.
Key features
Improvements
1.2.0 and 1.3.0-rc1 changelogs)
24.04
Merge conflicts resolved
Same 3 conflicts as Phase 1, resolved identically:
Validation