[Bug] Kernel panic when resuming from a memory snapshot

[shomilj]

Describe the bug

When we resume from a memory snapshot, roughly 5% of the time, we are seeing a kernel panic. Logs are below ->

2025-07-21T22:11:28.564769760 [1f073339-55c6-4046-afc6-4fcb1b191134:main] Running Firecracker v1.12.0
2025-07-21T22:11:28.564843981 [1f073339-55c6-4046-afc6-4fcb1b191134:main] Listening on API socket ("/run/firecracker.socket").
2025-07-21T22:11:28.565178178 [1f073339-55c6-4046-afc6-4fcb1b191134:fc_api] API server started.
2025-07-21T22:11:28.622405469 [1f073339-55c6-4046-afc6-4fcb1b191134:fc_api] The API server received a Put request on "/logger" with body "{\"log_path\":\"run/firecracker.log\",\"level\":\"Debug\",\"show_level\":true,\"show_log_origin\":true}".
DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400
[ 942.463237] PKRU: 55555554
[ 942.463237] Call Trace:
[ 942.463238] <NMI>
[ 942.463240] ? show_regs+0x6d/0x80
[ 942.463247] ? nmi_cpu_backtrace+0x90/0x100
[ 942.463249] ? nmi_cpu_backtrace_handler+0x15/0x20
[ 942.463251] ? nmi_handle+0x60/0x160
[ 942.463256] ? default_do_nmi+0x45/0x120
[ 942.463258] ? exc_nmi+0xe9/0x110
[ 942.463258] ? end_repeat_nmi+0xf/0x53
[ 942.463265] ? __pv_queued_spin_lock_slowpath+0xdd/0x370
[ 942.463266] ? __pv_queued_spin_lock_slowpath+0xdd/0x370
[ 942.463267] ? __pv_queued_spin_lock_slowpath+0xdd/0x370
[ 942.463268] </NMI>
[ 942.463268] <IRQ>
[ 942.463269] _raw_spin_lock+0x2d/0x40
[ 942.463269] __queue_work.part.0+0xa9/0x320
[ 942.463281] delayed_work_timer_fn+0x40/0x90
[ 942.463282] ? __pfx_delayed_work_timer_fn+0x10/0x10
[ 942.463284] call_timer_fn+0x2c/0x130
[ 942.463286] ? __pfx_delayed_work_timer_fn+0x10/0x10
[ 942.463287] __run_timers+0x158/0x270
[ 942.463289] run_timer_softirq+0x7b/0xe0
[ 942.463290] handle_softirqs+0xc0/0x290
[ 942.463300] irq_exit_rcu+0x6c/0x90
[ 942.463301] sysvec_apic_timer_interrupt+0x7b/0x90
[ 942.463303] </IRQ>
[ 942.463303] <TASK>
[ 942.463303] asm_sysvec_apic_timer_interrupt+0x1f/0x30
[ 942.463304] RIP: 0010:default_idle+0xf/0x20
[ 942.463310] Code: 4c 01 c7 4c 29 c2 e9 72 ff ff ff 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa eb 07 0f 00 2d 33 5d 35 00 fb f4 <fa> c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90
[ 942.463315] RSP: 0018:ffffc900000b7e98 EFLAGS: 00000206
[ 942.463316] RAX: ffff88841fd00000 RBX: 0000000000000000 RCX: 0000000000000000
[ 942.463316] RDX: 0000000000000002 RSI: 0000000000000002 RDI: 000000000003b6c4
[ 942.463317] RBP: ffffc900000b7ea0 R08: 0000000000000000 R09: 0000000000000001
[ 942.463317] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000000
[ 942.463318] R13: ffff888100399700 R14: 0000000000000000 R15: 0000000000000000
[ 942.463319] ? arch_cpu_idle+0xd/0x20
[ 942.463327] default_idle_call+0x34/0xf0
[ 942.463328] cpuidle_idle_call+0x10d/0x120
[ 942.463329] ? tsc_verify_tsc_adjust+0x3b/0x100
[ 942.463332] do_idle+0x78/0xc0
[ 942.463332] cpu_startup_entry+0x30/0x40
[ 942.463333] start_secondary+0x118/0x140
[ 942.463335] common_startup_64+0x13b/0x148
[ 942.463337] </TASK>
[ 942.464156] rcu: rcu_preempt kthread timer wakeup didn't happen for 239726 jiffies! g-483 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402
[287358.388326] rcu: 	Possible timer handling issue on cpu=3 timer-softirq=101
[287358.388643] rcu: rcu_preempt kthread starved for 71843709 jiffies! g-483 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x200 ->cpu=3
[287358.389054] rcu: 	Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
[287358.389401] rcu: RCU grace-period kthread stack dump:
[287358.389614] task:rcu_preempt state:R stack:0 pid:17 tgid:17 ppid:2 flags:0x00004000
[287358.389998] Call Trace:
[287358.390115] <TASK>
[287358.390235] __schedule+0x259/0x780
[287358.390415] ? __mod_timer+0x27f/0x390
[287358.390623] ? __pfx_rcu_gp_kthread+0x10/0x10
[287358.390821] schedule+0x2e/0xa0
[287358.390948] schedule_timeout+0x90/0x160
[287358.391102] ? __pfx_process_timeout+0x10/0x10
[287358.391301] rcu_gp_fqs_loop+0x10c/0x4d0
[287358.391493] rcu_gp_kthread+0xd7/0x170
[287358.391648] kthread+0xe5/0x120
[287358.391783] ? __pfx_kthread+0x10/0x10
[287358.391936] ret_from_fork+0x3d/0x60
[287358.392081] ? __pfx_kthread+0x10/0x10
[287358.392240] ret_from_fork_asm+0x1a/0x30
[287358.392408] </TASK>
[287358.392514] rcu: Stack dump where RCU GP kthread last ran:
[287358.392746] CPU: 3 UID: 0 PID: 0 Comm: swapper/3 Not tainted 6.12.8+ #11
[287358.393023] RIP: 0010:default_idle+0xf/0x20
[287358.393204] Code: 4c 01 c7 4c 29 c2 e9 72 ff ff ff 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa eb 07 0f 00 2d 33 5d 35 00 fb f4 <fa> c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90
[287358.393945] RSP: 0018:ffffc900000bfe98 EFLAGS: 00000206
[287358.394153] RAX: ffff88841fd80000 RBX: 0000000000000000 RCX: 0000000000000000
[287358.394432] RDX: 0000000000000003 RSI: 0000000000000003 RDI: 0000000000008a54
[287358.394728] RBP: ffffc900000bfea0 R08: 0000000000000000 R09: 0000000000000001
[287358.395014] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000000
[287358.395303] R13: ffff88810039ae00 R14: 0000000000000000 R15: 0000000000000000
[287358.395619] FS: 0000000000000000(0000) GS:ffff88841fd80000(0000) knlGS:0000000000000000
[287358.395941] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[287358.396170] CR2: 00007ff16601f6c8 CR3: 0000000101e64006 CR4: 0000000000770eb0
[287358.396454] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[287358.396741] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400
[287358.397047] PKRU: 55555554
[287358.397165] Call Trace:
[287358.397266] <IRQ>
[287358.397352] ? show_regs+0x6d/0x80
[287358.397526] ? dump_cpu_task+0x6b/0x80
[287358.397696] ? rcu_check_gp_kthread_starvation+0x175/0x1c0
[287358.397924] ? print_other_cpu_stall+0x264/0x460
[287358.398118] ? check_cpu_stall+0x1cb/0x230
[287358.398282] ? rcu_sched_clock_irq+0xd7/0x450
[287358.398463] ? update_process_times+0x7d/0xd0
[287358.398670] ? tick_nohz_handler+0x97/0x140
[287358.398843] ? __pfx_tick_nohz_handler+0x10/0x10
[287358.399031] ? __hrtimer_run_queues+0x111/0x2a0
[287358.399212] ? hrtimer_interrupt+0x108/0x230
[287358.399392] ? __sysvec_apic_timer_interrupt+0x5a/0x120
[287358.399620] ? sysvec_apic_timer_interrupt+0x76/0x90
[287358.399831] </IRQ>
[287358.399926] <TASK>
[287358.400031] ? asm_sysvec_apic_timer_interrupt+0x1f/0x30
[287358.400296] ? default_idle+0xf/0x20
[287358.400476] ? arch_cpu_idle+0xd/0x20
[287358.400682] default_idle_call+0x34/0xf0
[287358.400904] cpuidle_idle_call+0x10d/0x120
[287358.401115] ? tsc_verify_tsc_adjust+0x3b/0x100
[287358.401305] do_idle+0x78/0xc0
[287358.401434] cpu_startup_entry+0x30/0x40
[287358.401612] start_secondary+0x118/0x140
[287358.401761] common_startup_64+0x13b/0x148
[287358.401924] </TASK>
[287358.406474] BUG: kernel NULL pointer dereference, address: 0000000000000051
[287358.406816] #PF: supervisor read access in kernel mode
[287358.407020] #PF: error_code(0x0000) - not-present page
[287358.407224] PGD 0 P4D 0
[287358.407346] Oops: Oops: 0000 [#1] PREEMPT SMP NOPTI
[287358.407563] CPU: 1 UID: 0 PID: 1285 Comm: containerd Not tainted 6.12.8+ #11
[287358.407893] RIP: 0010:pick_task_fair+0x38/0xf0
[287358.408042] random: crng reseeded due to virtual machine fork
[287358.408113] Code: 54 4c 8d a7 80 00 00 00 53 48 89 fb 48 83 ec 08 8b 8b 90 00 00 00 85 c9 0f 84 88 00 00 00 4c 89 e7 eb 1e 66 90 e8 88 79 ff ff <80> 78 51 00 75 58 48 85 c0 74 db 48 8b b8 a8 00 00 00 48 85 ff 74
[287358.409151] RSP: 0000:ffffc90001897d70 EFLAGS: 00010082
[287358.409370] RAX: 0000000000000000 RBX: ffff88841fcaf3c0 RCX: 00000000000172b9
[287358.409668] RDX: 8a96886e8fb64e06 RSI: 00000000000172b9 RDI: 0000000c3074f664
[287358.409941] RBP: ffffc90001897d88 R08: 0000000000010e1c R09: 00000000f5257d14
[287358.410229] R10: 0000000002dc718d R11: 0000000000000000 R12: ffff88841fcaf440
[287358.410507] R13: ffff88841fcaf3c0 R14: ffff8881053ec500 R15: ffff88841fcaf3c0
[287358.410801] FS: 00007f78f7fff6c0(0000) GS:ffff88841fc80000(0000) knlGS:0000000000000000
[287358.411271] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[287358.411612] CR2: 0000000000000051 CR3: 0000000101b82001 CR4: 0000000000770eb0
[287358.411958] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[287358.412267] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400
[287358.412596] PKRU: 55555554
[287358.412715] Call Trace:
[287358.412817] <TASK>
[287358.412909] ? show_regs+0x6d/0x80
[287358.413049] ? __die+0x29/0x70
[287358.413177] ? page_fault_oops+0x84/0x140
[287358.413341] ? psi_group_change+0x139/0x330
[287358.413528] ? do_user_addr_fault+0x2fc/0x5e0
[287358.413705] ? exc_page_fault+0x6d/0x150
[287358.413862] ? asm_exc_page_fault+0x2b/0x30
[287358.414038] ? pick_task_fair+0x38/0xf0
[287358.414193] pick_next_task_fair+0x4c/0x370
[287358.414378] __schedule+0x12a/0x780
[287358.414600] ? _raw_spin_unlock_irqrestore+0x12/0x40
[287358.414828] schedule+0x2e/0xa0
[287358.414970] kvm_async_pf_task_wait_schedule+0xf4/0x180
[287358.415232] __kvm_handle_async_pf+0x62/0xc0
[287358.415470] exc_page_fault+0x8f/0x150
[287358.415665] asm_exc_page_fault+0x2b/0x30
[287358.415872] RIP: 0033:0x563411fefbce
[287358.416070] Code: 24 28 44 8b 44 24 2c e9 70 ff ff ff cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc 49 89 f2 48 89 fa 48 89 ce 48 89 df 0f 05 <48> 3d 01 f0 ff ff 76 15 48 f7 d8 48 89 c1 48 c7 c0 ff ff ff ff 48
[287358.416962] RSP: 002b:00007f78f7ffe560 EFLAGS: 00010246
[287358.417173] RAX: 0000000000000000 RBX: 0000000000000004 RCX: 0000563411fefbce
[287358.417452] RDX: 0000000000000080 RSI: 00007f78f7ffe5f4 RDI: 0000000000000004
[287358.417725] RBP: 00007f78f7ffe5a8 R08: 0000000000000000 R09: 0000000000000000
[287358.418003] R10: 00000000000000fa R11: 0000000000000246 R12: 00007f78f7ffe638
[287358.418302] R13: 000000c0005b0e7a R14: 000000c000104e00 R15: 0000000000000000
[287358.418603] </TASK>
[287358.418704] Modules linked in:
[287358.418855] CR2: 0000000000000051
[287358.419011] ---[ end trace 0000000000000000 ]---
[287358.419233] RIP: 0010:pick_task_fair+0x38/0xf0
[287358.419413] Code: 54 4c 8d a7 80 00 00 00 53 48 89 fb 48 83 ec 08 8b 8b 90 00 00 00 85 c9 0f 84 88 00 00 00 4c 89 e7 eb 1e 66 90 e8 88 79 ff ff <80> 78 51 00 75 58 48 85 c0 74 db 48 8b b8 a8 00 00 00 48 85 ff 74
[287358.420115] RSP: 0000:ffffc90001897d70 EFLAGS: 00010082
[287358.420316] RAX: 0000000000000000 RBX: ffff88841fcaf3c0 RCX: 00000000000172b9
[287358.420595] RDX: 8a96886e8fb64e06 RSI: 00000000000172b9 RDI: 0000000c3074f664
[287358.420888] RBP: ffffc90001897d88 R08: 0000000000010e1c R09: 00000000f5257d14
[287358.421166] R10: 0000000002dc718d R11: 0000000000000000 R12: ffff88841fcaf440
[287358.421454] R13: ffff88841fcaf3c0 R14: ffff8881053ec500 R15: ffff88841fcaf3c0
[287358.421737] FS: 00007f78f7fff6c0(0000) GS:ffff88841fc80000(0000) knlGS:0000000000000000
[287358.422049] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[287358.422298] CR2: 0000000000000051 CR3: 0000000101b82001 CR4: 0000000000770eb0
[287358.422609] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[287358.422880] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400
[287358.423148] PKRU: 55555554
[287358.423264] note: containerd[1285] exited with irqs disabled
[287358.423503] Oops: general protection fault, probably for non-canonical address 0x2e6666000005ad11: 0000 [#2] PREEMPT SMP NOPTI
[287358.423928] CPU: 1 UID: 0 PID: 1285 Comm: containerd Tainted: G D 6.12.8+ #11
[287358.424275] Tainted: [D]=DIE
[287358.424404] RIP: 0010:kvm_async_pf_task_wake+0x58/0x170
[287358.424629] Code: fb 45 31 e4 49 c1 e6 04 4d 8d be e0 47 4f 83 4c 89 ff e8 8b 50 e4 00 49 8b 47 08 48 85 c0 75 0a eb 6f 48 8b 00 48 85 c0 74 67 <3b> 58 28 75 f3 48 8b 50 08 48 85 d2 74 1e 48 8b 08 48 89 0a 48 85
[287358.425366] RSP: 0000:ffffc900000f4e78 EFLAGS: 00010002
[287358.425584] RAX: 2e6666000005ace9 RBX: 0000000000007001 RCX: 000000000000080b
[287358.425859] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffffffff834f49a0
[287358.426164] RBP: ffffc900000f4ea0 R08: 0000000000000000 R09: 0000000000000000
[287358.426438] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[287358.426717] R13: 000000000000001c R14: 00000000000001c0 R15: ffffffff834f49a0
[287358.426991] FS: 00007f78f7fff6c0(0000) GS:ffff88841fc80000(0000) knlGS:0000000000000000
[287358.427300] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[287358.427527] CR2: 0000000000000051 CR3: 0000000101b82001 CR4: 0000000000770eb0
[287358.427804] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[287358.428067] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400
[287358.428339] PKRU: 55555554
[287358.428458] Call Trace:
[287358.428558] <IRQ>
[287358.428647] ? show_regs+0x6d/0x80
[287358.428786] ? die_addr+0x3c/0xa0
[287358.428922] ? exc_general_protection+0x1c5/0x3d0
[287358.429112] ? asm_exc_general_protection+0x2b/0x30
[287358.429308] ? kvm_async_pf_task_wake+0x58/0x170
[287358.429504] ? kvm_async_pf_task_wake+0x45/0x170
[287358.429694] __sysvec_kvm_asyncpf_interrupt+0x52/0xb0
[287358.429901] sysvec_kvm_asyncpf_interrupt+0x39/0x90
[287358.430100] asm_sysvec_kvm_asyncpf_interrupt+0x1f/0x30
[287358.430317] RIP: 0010:handle_softirqs+0x70/0x290
[287358.430519] Code: 44 0f b7 2d ca 4f f8 7e 65 81 05 9f 4f f8 7e 00 01 00 00 c7 45 c4 0a 00 00 00 31 c0 65 66 89 05 ae 4f f8 7e fb bb ff ff ff ff <49> c7 c2 c0 60 80 82 41 0f bc dd 83 c3 01 74 79 8d 43 ff 48 98 4d
[287358.431214] RSP: 0000:ffffc900000f4f98 EFLAGS: 00000246
[287358.431422] RAX: 0000000000000000 RBX: 00000000ffffffff RCX: ffff88841fcb4c60
[287358.431703] RDX: 0000000000000000 RSI: ffff8881053ec500 RDI: 0000000000000000
[287358.431974] RBP: ffffc900000f4fd8 R08: 0000000000000000 R09: 0000000000000000
[287358.432241] R10: 0000000000000000 R11: ffffc900000f4ff8 R12: ffffc90001897e88
[287358.432535] R13: 0000000000000010 R14: 0000000000000000 R15: 0000000000000000
[287358.432824] ? __flush_smp_call_function_queue+0x99/0x420
[287358.433057] irq_exit_rcu+0x6c/0x90
[287358.433208] sysvec_call_function+0x7b/0x90
[287358.433387] </IRQ>
[287358.433482] <TASK>
[287358.433574] asm_sysvec_call_function+0x1f/0x30
[287358.433757] RIP: 0010:make_task_dead+0x3e/0x140
[287358.433939] Code: 71 f8 7e 53 65 f7 05 15 71 f8 7e 00 ff ff 00 75 65 41 8b 84 24 c8 04 00 00 85 c0 74 65 89 fb 9c 58 f6 c4 02 0f 84 9f 00 00 00 <65> 8b 0d f3 70 f8 7e 81 e1 ff ff ff 7f 75 55 8b 35 15 8c 7a 01 b8
[287358.434650] RSP: 0000:ffffc90001897f38 EFLAGS: 00000246
[287358.434855] RAX: 0000000000000030 RBX: 0000000000000009 RCX: 0000000000000027
[287358.435131] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88841fc9c200
[287358.435405] RBP: ffffc90001897f48 R08: 0000000000000000 R09: ffffc90001897db0
[287358.435688] R10: ffffc90001897da8 R11: ffffffff82f59868 R12: ffff8881053ec500
[287358.435973] R13: 0000000000000051 R14: 0000000000000009 R15: 0000000000000046
[287358.436274] ? make_task_dead+0xf9/0x140
[287358.436447] rewind_stack_and_make_dead+0x16/0x20
[287358.436653] RIP: 0033:0x563411fefbce
[287358.436808] Code: 24 28 44 8b 44 24 2c e9 70 ff ff ff cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc 49 89 f2 48 89 fa 48 89 ce 48 89 df 0f 05 <48> 3d 01 f0 ff ff 76 15 48 f7 d8 48 89 c1 48 c7 c0 ff ff ff ff 48
[287358.437530] RSP: 002b:00007f78f7ffe560 EFLAGS: 00010246
[287358.437738] RAX: 0000000000000000 RBX: 0000000000000004 RCX: 0000563411fefbce
[287358.438012] RDX: 0000000000000080 RSI: 00007f78f7ffe5f4 RDI: 0000000000000004
[287358.438284] RBP: 00007f78f7ffe5a8 R08: 0000000000000000 R09: 0000000000000000
[287358.438560] R10: 00000000000000fa R11: 0000000000000246 R12: 00007f78f7ffe638
[287358.438839] R13: 000000c0005b0e7a R14: 000000c000104e00 R15: 0000000000000000
[287358.439118] </TASK>
[287358.439217] Modules linked in:
[287358.439346] ---[ end trace 0000000000000000 ]---
[287358.439544] RIP: 0010:pick_task_fair+0x38/0xf0
[287358.439730] Code: 54 4c 8d a7 80 00 00 00 53 48 89 fb 48 83 ec 08 8b 8b 90 00 00 00 85 c9 0f 84 88 00 00 00 4c 89 e7 eb 1e 66 90 e8 88 79 ff ff <80> 78 51 00 75 58 48 85 c0 74 db 48 8b b8 a8 00 00 00 48 85 ff 74
[287358.440428] RSP: 0000:ffffc90001897d70 EFLAGS: 00010082
[287358.440635] RAX: 0000000000000000 RBX: ffff88841fcaf3c0 RCX: 00000000000172b9
[287358.440909] RDX: 8a96886e8fb64e06 RSI: 00000000000172b9 RDI: 0000000c3074f664
[287358.441205] RBP: ffffc90001897d88 R08: 0000000000010e1c R09: 00000000f5257d14
[287358.441470] R10: 0000000002dc718d R11: 0000000000000000 R12: ffff88841fcaf440
[287358.441744] R13: ffff88841fcaf3c0 R14: ffff8881053ec500 R15: ffff88841fcaf3c0
[287358.442017] FS: 00007f78f7fff6c0(0000) GS:ffff88841fc80000(0000) knlGS:0000000000000000
[287358.442321] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[287358.442548] CR2: 0000000000000051 CR3: 0000000101b82001 CR4: 0000000000770eb0
[287358.442825] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[287358.443096] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400
[287358.443405] PKRU: 55555554
[287358.443532] Kernel panic - not syncing: Fatal exception in interrupt
[287359.473649] Shutting down cpus with NMI
[287359.474045] Kernel Offset: disabled
[287359.474175] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---

To Reproduce

Unclear how to reproduce - the cases

1. Start Firecracker VM based on Debian rootfs
2. Starts docker daemon on boot
3. Take full memory snapshot
4. Create container with http server workload
5. Take full memory snapshot & restore from that

Expected Behavior

Resuming from a memory snapshot results in a working VM.

Environment

• Firecracker version: 1.12.0
• Host and guest kernel versions: 6.8.0-1031-aws, 6.12.8+
• Rootfs used:
  • Description: Debian GNU/Linux 12 (bookworm)
  • Release: 12
  • Codename: bookworm
• Architecture: x86_64
• Any other relevant software versions:
  • Docker version 28.3.2, build e77ff99
  • Guest container is Ubuntu 24.04.1 LTS

Checks

• Have you searched the Firecracker Issues database for similar problems?
• Have you read the existing relevant Firecracker documentation?
• Are you certain the bug being reported is a Firecracker issue?

[zulinx86]

Hello @shomilj , thank you for reporting the issue.

We don't run into the issue in our CI. So we need your help and more info/resources!

We are internally testing the host kernel version using ubuntu AMI, but don't test the guest kernel version that you use. The guest kernel looks relatively new, so it could be a problem of the kernel. Could you try to reproduce with a supported guest kernel version (5.10 or 6.1)?

Also, could you provide us more info/resources to reproduce/debug on our side?

• the timing you take a snapshot (soon after uVM booted? after init process started?) since there is a known limitation of VMGenID device.
• minimal reproducer script with which you can reproduce the issue
• guest resources
  • kernel file
  • rootfs file
• host CPU model (or instance type)

Thanks,

[shomilj]

Switched to a supported kernel version (6.1 / amazonlinux). Now, we see this -> https://gist.github.com/shomilj/e31dfcca7046db3508f2724cd29c77ea

We take a snapshot well after the kernel has finished booting. We've been unable to get a repro script at this point.

Here's our kernel config ->

https://gist.github.com/shomilj/8c58b4cc818c45c626e39a836f917715

Thanks!