After "Extracting image" stage, i can't find tarball of root filesystem of the extracted Image, need help

[EvE-VeV]

Hi, when I created my docker image using the dockfile called Dockerfile.full, but then i run the following commands, i came across with the following problem:

`root@8309f862dc18:/workspaces/firmafl-repro# ./run_experiment.sh 9050
It seems experiment 9050 was not extracted, doing so now

• Starting PostgreSQL 9.5 database server [ OK ]
  [*] Waiting for postgresql database to become available ... done!
  [+] Cleaning up previously unpacked images
  rm: cannot remove './image_9050': No such file or directory
  rm: cannot remove './images/9050.tar.gz': No such file or directory
  [+] Extracting image

Database Image ID: 1

/workspaces/firmafl-repro/FirmAFL/firmware/DIR-815_FIRMWARE_1.01.ZIP

MD5: 1e941413c6489000c7ee6269e98e0a3f
Tag: 1
Temp: /tmp/tmp78m216a1
Status: Kernel: True, Rootfs: False, Do_Kernel: False, Do_Rootfs: True

Zip archive data, at least v1.0 to extract, name: dir815_FW_101/
Recursing into archive ...

/tmp/tmp78m216a1/_DIR-815_FIRMWARE_1.01.ZIP.extracted/dir815_FW_101/dir815_release_notes_101.docx
>> MD5: 9dc2fb97cc6fa74faf88331dea52abcf
>> Tag: 1
>> Temp: /tmp/tmpua724_ah
>> Status: Kernel: True, Rootfs: False, Do_Kernel: False, Do_Rootfs: True
>>>> Zip archive data, at least v2.0 to extract, compressed size: 358, uncompressed size: 1312, name: [Content_Types].xml
>> Recursing into archive ...

/tmp/tmpua724_ah/_dir815_release_notes_101.docx.extracted/[Content_Types].xml
>> MD5: ded1b06d92c5c6b15c0b12c176ff3355
>> Tag: 1
>> Temp: /tmp/tmp3pcgbb92
>> Status: Kernel: True, Rootfs: False, Do_Kernel: False, Do_Rootfs: True
>> Recursing into archive ...
>> Recursing into compressed ...
>> Cleaning up /tmp/tmp3pcgbb92...

/tmp/tmpua724_ah/_dir815_release_notes_101.docx.extracted/word/styles.xml
>> MD5: 5af58c20067e0dbfb7c4c04851428582
>> Tag: 1
>> Temp: /tmp/tmpmpl6bl34
>> Status: Kernel: True, Rootfs: False, Do_Kernel: False, Do_Rootfs: True
>> Recursing into archive ...
>> Recursing into compressed ...
>> Cleaning up /tmp/tmpmpl6bl34...

/tmp/tmpua724_ah/_dir815_release_notes_101.docx.extracted/word/document.xml
>> MD5: 8dbb759691b2c0b29020130fad4cc4a0
>> Tag: 1
>> Temp: /tmp/tmp304wfz52
>> Status: Kernel: True, Rootfs: False, Do_Kernel: False, Do_Rootfs: True
>> Recursing into archive ...
>> Recursing into compressed ...
>> Cleaning up /tmp/tmp304wfz52...

/tmp/tmpua724_ah/_dir815_release_notes_101.docx.extracted/word/settings.xml
>> MD5: 9592c157d91602694469155ed3e0efc3
>> Tag: 1
>> Temp: /tmp/tmpulaxoonw
>> Status: Kernel: True, Rootfs: False, Do_Kernel: False, Do_Rootfs: True
>> Recursing into archive ...
>> Recursing into compressed ...
>> Cleaning up /tmp/tmpulaxoonw...

/tmp/tmpua724_ah/_dir815_release_notes_101.docx.extracted/word/fontTable.xml
>> MD5: 1270dc29206e448125ee0536a4908a9b
>> Tag: 1
>> Temp: /tmp/tmp4hwm7emj
>> Status: Kernel: True, Rootfs: False, Do_Kernel: False, Do_Rootfs: True
>> Recursing into archive ...
>> Recursing into compressed ...
>> Cleaning up /tmp/tmp4hwm7emj...

/tmp/tmpua724_ah/_dir815_release_notes_101.docx.extracted/word/webSettings.xml
>> MD5: 15065d2de3eddbb09d84337a09fd7985
>> Tag: 1
>> Temp: /tmp/tmp2pagsj9p
>> Status: Kernel: True, Rootfs: False, Do_Kernel: False, Do_Rootfs: True
>> Recursing into archive ...
>> Recursing into compressed ...
>> Skipping: recursion breadth 5
>> Cleaning up /tmp/tmp2pagsj9p...
>> Skipping: completed!
>> Cleaning up /tmp/tmpua724_ah...

/tmp/tmp78m216a1/_DIR-815_FIRMWARE_1.01.ZIP.extracted/dir815_FW_101/DIR-815 FW 1.01b14_1.01b14.bin
>> MD5: cff3aa9820e35e21a146ce45f2155f42
>> Tag: 1
>> Temp: /tmp/tmpu2ujhr5i
>> Status: Kernel: True, Rootfs: False, Do_Kernel: False, Do_Rootfs: True
>> Recursing into archive ...

WARNING: Extractor.execute failed to run external extractor 'unsquashfs -d 'squashfs-root' '%e'': [Errno 2] No such file or directory: 'unsquashfs', 'unsquashfs -d 'squashfs-root' '%e'' might not be installed correctly

WARNING: Extractor.execute failed to run external extractor 'sasquatch -p 1 -le -d 'squashfs-root' '%e'': [Errno 2] No such file or directory: 'sasquatch', 'sasquatch -p 1 -le -d 'squashfs-root' '%e'' might not be installed correctly

WARNING: Extractor.execute failed to run external extractor 'sasquatch -p 1 -be -d 'squashfs-root' '%e'': [Errno 2] No such file or directory: 'sasquatch', 'sasquatch -p 1 -be -d 'squashfs-root' '%e'' might not be installed correctly
>>>> Squashfs filesystem, little endian, version 4.0, compression:lzma, size: 2808054 bytes, 1526 inodes, blocksize: 262144 bytes, created: 2011-05-12 14:14:40
>>>> LZMA compressed data, properties: 0x5D, dictionary size: 33554432 bytes, uncompressed size: 3017436 bytes
>> Recursing into compressed ...

/tmp/tmpu2ujhr5i/_DIR-815 FW 1.01b14_1.01b14.bin-0.extracted/6C
>> MD5: b9785d487cc4f1804ba0ff95c88f2c64
>> Tag: 1
>> Temp: /tmp/tmpgo3b95xe
>> Status: Kernel: True, Rootfs: False, Do_Kernel: False, Do_Rootfs: True
>> Recursing into archive ...
>>>> MySQL MISAM compressed data file Version 8
>> Recursing into compressed ...
>> Cleaning up /tmp/tmpgo3b95xe...
>> Cleaning up /tmp/tmpu2ujhr5i...

Recursing into compressed ...

/tmp/tmp78m216a1/_DIR-815_FIRMWARE_1.01.ZIP.extracted/dir815_FW_101/dir815_release_notes_101.docx
>> MD5: 9dc2fb97cc6fa74faf88331dea52abcf
>> Skipping: 9dc2fb97cc6fa74faf88331dea52abcf...

/tmp/tmp78m216a1/_DIR-815_FIRMWARE_1.01.ZIP.extracted/dir815_FW_101/DIR-815 FW 1.01b14_1.01b14.bin
>> MD5: cff3aa9820e35e21a146ce45f2155f42
>> Skipping: cff3aa9820e35e21a146ce45f2155f42...

Cleaning up /tmp/tmp78m216a1...
[+] Creating FirmAFL image
tar: /workspaces/firmafl-repro/FirmAFL/firmadyne/images/9050.tar.gz: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now
tar: /workspaces/firmafl-repro/FirmAFL/firmadyne/images/9050.tar.gz: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now
tar: /workspaces/firmafl-repro/FirmAFL/firmadyne/images/9050.tar.gz: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now
Querying database for architecture...
----Running----
----Copying Filesystem Tarball----
Error: Cannot find tarball of root filesystem for 9050!
Querying database for architecture...
Error: Unable to lookup architecture. Please specify {armel,mipseb,mipsel} as the second argument!
tee: 'cpu*/cpufreq/scaling_governor': No such file or directory
performance
/workspaces/firmafl-repro/FirmAFL
Traceback (most recent call last):
File "generate_run_full.py", line 55, in
generate_run_full(image_id, arch)
File "generate_run_full.py", line 7, in generate_run_full
file_src = open(script_src)
IOError: [Errno 2] No such file or directory: 'firmadyne/scratch/9050/run.sh'
cp: cannot stat 'firmadyne/scratch/9050/run_full.sh': No such file or directory
cp: cannot create regular file 'image_9050/': Not a directory
cp: cannot create regular file 'image_9050/': Not a directory
cp: cannot stat 'user_mode/mipsel-linux-user/qemu-mipsel': No such file or directory
cp: cannot create regular file 'image_9050/': Not a directory
cp: cannot create regular file 'image_9050/': Not a directory
cp: cannot create regular file 'image_9050/': Not a directory
cp: cannot create regular file 'image_9050/': Not a directory
cp: cannot create regular file 'image_9050/': Not a directory
cp: cannot stat 'firmadyne/scratch/9050/image.raw': No such file or directory
cp: cannot create regular file 'image_9050/': Not a directory
cp: cannot create regular file 'image_9050/': Not a directory
cp: cannot create regular file 'image_9050/': Not a directory
cp: cannot create regular file 'image_9050/': Not a directory
mkdir: cannot create directory ‘image_9050/inputs’: No such file or directory
cp: cannot create regular file 'image_9050/inputs/': No such file or directory
cp: cannot create regular file 'image_9050/': Not a directory
cp: cannot stat './image_9050/qemu-system-mipsel': No such file or directory
[+] Setting up experiment 9050 (image id: 9050)
cp: failed to access './image_9050/': Not a directory
cp: failed to access './image_9050/keywords_9050': Not a directory
cp: failed to access './image_9050/inputs/seed': Not a directory
[+] Adjusting time budget for 9050
sed: can't read /workspaces/firmafl-repro/FirmAFL/image_9050/start_full.sh: Not a directory
./run_experiment.sh: line 27: cd: /workspaces/firmafl-repro/FirmAFL/image_9050: Not a directory
./start_full.sh: line 11: ./run_full.sh: No such file or directory
^Z
[1]+ Stopped ./run_experiment.sh 9050
root@8309f862dc18:/workspaces/firmafl-repro# ./run_experiment.sh 9050
It seems experiment 9050 was not extracted, doing so now

• Starting PostgreSQL 9.5 database server [ OK ]
  [*] Waiting for postgresql database to become available ... done!
  [+] Cleaning up previously unpacked images
  rm: cannot remove './image_9050': No such file or directory
  rm: cannot remove './images/9050.tar.gz': No such file or directory
  [+] Extracting image

Database Image ID: 1

/workspaces/firmafl-repro/FirmAFL/firmware/DIR-815_FIRMWARE_1.01.ZIP

MD5: 1e941413c6489000c7ee6269e98e0a3f
Tag: 1
Temp: /tmp/tmp9_naothk
Status: Kernel: True, Rootfs: False, Do_Kernel: False, Do_Rootfs: True

Zip archive data, at least v1.0 to extract, name: dir815_FW_101/
Recursing into archive ...

/tmp/tmp9_naothk/_DIR-815_FIRMWARE_1.01.ZIP.extracted/dir815_FW_101/dir815_release_notes_101.docx
>> MD5: 9dc2fb97cc6fa74faf88331dea52abcf
>> Tag: 1
>> Temp: /tmp/tmp778r7en2
>> Status: Kernel: True, Rootfs: False, Do_Kernel: False, Do_Rootfs: True
>>>> Zip archive data, at least v2.0 to extract, compressed size: 358, uncompressed size: 1312, name: [Content_Types].xml
>> Recursing into archive ...

/tmp/tmp778r7en2/_dir815_release_notes_101.docx.extracted/[Content_Types].xml
>> MD5: ded1b06d92c5c6b15c0b12c176ff3355
>> Tag: 1
>> Temp: /tmp/tmpteq6tje1
>> Status: Kernel: True, Rootfs: False, Do_Kernel: False, Do_Rootfs: True
>> Recursing into archive ...
>> Recursing into compressed ...
>> Cleaning up /tmp/tmpteq6tje1...

/tmp/tmp778r7en2/_dir815_release_notes_101.docx.extracted/word/styles.xml
>> MD5: 5af58c20067e0dbfb7c4c04851428582
>> Tag: 1
>> Temp: /tmp/tmp_s_6qc3f
>> Status: Kernel: True, Rootfs: False, Do_Kernel: False, Do_Rootfs: True
>> Recursing into archive ...
>> Recursing into compressed ...
>> Cleaning up /tmp/tmp_s_6qc3f...

/tmp/tmp778r7en2/_dir815_release_notes_101.docx.extracted/word/document.xml
>> MD5: 8dbb759691b2c0b29020130fad4cc4a0
>> Tag: 1
>> Temp: /tmp/tmpeu4e6p45
>> Status: Kernel: True, Rootfs: False, Do_Kernel: False, Do_Rootfs: True
>> Recursing into archive ...
>> Recursing into compressed ...
>> Cleaning up /tmp/tmpeu4e6p45...

/tmp/tmp778r7en2/_dir815_release_notes_101.docx.extracted/word/settings.xml
>> MD5: 9592c157d91602694469155ed3e0efc3
>> Tag: 1
>> Temp: /tmp/tmp5dqmy064
>> Status: Kernel: True, Rootfs: False, Do_Kernel: False, Do_Rootfs: True
>> Recursing into archive ...
>> Recursing into compressed ...
>> Cleaning up /tmp/tmp5dqmy064...

/tmp/tmp778r7en2/_dir815_release_notes_101.docx.extracted/word/fontTable.xml
>> MD5: 1270dc29206e448125ee0536a4908a9b
>> Tag: 1
>> Temp: /tmp/tmpm_9gak3n
>> Status: Kernel: True, Rootfs: False, Do_Kernel: False, Do_Rootfs: True
>> Recursing into archive ...
>> Recursing into compressed ...
>> Cleaning up /tmp/tmpm_9gak3n...

/tmp/tmp778r7en2/_dir815_release_notes_101.docx.extracted/word/webSettings.xml
>> MD5: 15065d2de3eddbb09d84337a09fd7985
>> Tag: 1
>> Temp: /tmp/tmp202l0fdy
>> Status: Kernel: True, Rootfs: False, Do_Kernel: False, Do_Rootfs: True
>> Recursing into archive ...
>> Recursing into compressed ...
>> Skipping: recursion breadth 5
>> Cleaning up /tmp/tmp202l0fdy...
>> Skipping: completed!
>> Cleaning up /tmp/tmp778r7en2...

/tmp/tmp9_naothk/_DIR-815_FIRMWARE_1.01.ZIP.extracted/dir815_FW_101/DIR-815 FW 1.01b14_1.01b14.bin
>> MD5: cff3aa9820e35e21a146ce45f2155f42
>> Tag: 1
>> Temp: /tmp/tmps0izs9a7
>> Status: Kernel: True, Rootfs: False, Do_Kernel: False, Do_Rootfs: True
>> Recursing into archive ...

WARNING: Extractor.execute failed to run external extractor 'unsquashfs -d 'squashfs-root' '%e'': [Errno 2] No such file or directory: 'unsquashfs', 'unsquashfs -d 'squashfs-root' '%e'' might not be installed correctly

WARNING: Extractor.execute failed to run external extractor 'sasquatch -p 1 -le -d 'squashfs-root' '%e'': [Errno 2] No such file or directory: 'sasquatch', 'sasquatch -p 1 -le -d 'squashfs-root' '%e'' might not be installed correctly

WARNING: Extractor.execute failed to run external extractor 'sasquatch -p 1 -be -d 'squashfs-root' '%e'': [Errno 2] No such file or directory: 'sasquatch', 'sasquatch -p 1 -be -d 'squashfs-root' '%e'' might not be installed correctly
>>>> Squashfs filesystem, little endian, version 4.0, compression:lzma, size: 2808054 bytes, 1526 inodes, blocksize: 262144 bytes, created: 2011-05-12 14:14:40
>>>> LZMA compressed data, properties: 0x5D, dictionary size: 33554432 bytes, uncompressed size: 3017436 bytes
>> Recursing into compressed ...

/tmp/tmps0izs9a7/_DIR-815 FW 1.01b14_1.01b14.bin-0.extracted/6C
>> MD5: b9785d487cc4f1804ba0ff95c88f2c64
>> Tag: 1
>> Temp: /tmp/tmp93vtbyes
>> Status: Kernel: True, Rootfs: False, Do_Kernel: False, Do_Rootfs: True
>> Recursing into archive ...
>>>> MySQL MISAM compressed data file Version 8
>> Recursing into compressed ...
>> Cleaning up /tmp/tmp93vtbyes...
>> Cleaning up /tmp/tmps0izs9a7...

Recursing into compressed ...

/tmp/tmp9_naothk/_DIR-815_FIRMWARE_1.01.ZIP.extracted/dir815_FW_101/dir815_release_notes_101.docx
>> MD5: 9dc2fb97cc6fa74faf88331dea52abcf
>> Skipping: 9dc2fb97cc6fa74faf88331dea52abcf...

/tmp/tmp9_naothk/_DIR-815_FIRMWARE_1.01.ZIP.extracted/dir815_FW_101/DIR-815 FW 1.01b14_1.01b14.bin
>> MD5: cff3aa9820e35e21a146ce45f2155f42
>> Skipping: cff3aa9820e35e21a146ce45f2155f42...

Cleaning up /tmp/tmp9_naothk...
[+] Creating FirmAFL image
tar: /workspaces/firmafl-repro/FirmAFL/firmadyne/images/9050.tar.gz: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now
tar: /workspaces/firmafl-repro/FirmAFL/firmadyne/images/9050.tar.gz: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now
tar: /workspaces/firmafl-repro/FirmAFL/firmadyne/images/9050.tar.gz: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now
Querying database for architecture...
----Running----
----Copying Filesystem Tarball----
Error: Cannot find tarball of root filesystem for 9050!
Querying database for architecture...
Error: Unable to lookup architecture. Please specify {armel,mipseb,mipsel} as the second argument!
tee: 'cpu*/cpufreq/scaling_governor': No such file or directory
performance
/workspaces/firmafl-repro/FirmAFL
Traceback (most recent call last):
File "generate_run_full.py", line 55, in
generate_run_full(image_id, arch)
File "generate_run_full.py", line 7, in generate_run_full
file_src = open(script_src)
IOError: [Errno 2] No such file or directory: 'firmadyne/scratch/9050/run.sh'
cp: cannot stat 'firmadyne/scratch/9050/run_full.sh': No such file or directory
cp: cannot create regular file 'image_9050/': Not a directory
cp: cannot create regular file 'image_9050/': Not a directory
cp: cannot stat 'user_mode/mipsel-linux-user/qemu-mipsel': No such file or directory
cp: cannot create regular file 'image_9050/': Not a directory
cp: cannot create regular file 'image_9050/': Not a directory
cp: cannot create regular file 'image_9050/': Not a directory
cp: cannot create regular file 'image_9050/': Not a directory
cp: cannot create regular file 'image_9050/': Not a directory
cp: cannot stat 'firmadyne/scratch/9050/image.raw': No such file or directory
cp: cannot create regular file 'image_9050/': Not a directory
cp: cannot create regular file 'image_9050/': Not a directory
cp: cannot create regular file 'image_9050/': Not a directory
cp: cannot create regular file 'image_9050/': Not a directory
mkdir: cannot create directory ‘image_9050/inputs’: No such file or directory
cp: cannot create regular file 'image_9050/inputs/': No such file or directory
cp: cannot create regular file 'image_9050/': Not a directory
cp: cannot stat './image_9050/qemu-system-mipsel': No such file or directory
[+] Setting up experiment 9050 (image id: 9050)
cp: failed to access './image_9050/': Not a directory
cp: failed to access './image_9050/keywords_9050': Not a directory
cp: failed to access './image_9050/inputs/seed': Not a directory
[+] Adjusting time budget for 9050
sed: can't read /workspaces/firmafl-repro/FirmAFL/image_9050/start_full.sh: Not a directory
./run_experiment.sh: line 27: cd: /workspaces/firmafl-repro/FirmAFL/image_9050: Not a directory
./start_full.sh: line 11: ./run_full.sh: No such file or directory
python: can't open file 'test.py': [Errno 2] No such file or directory
python: can't open file 'test.py': [Errno 2] No such file or directory`

[mariusmue]

Hi,

Thank you for your interest. Unfortunately, we are unable to reproduce your issue; on what kind of system are you running docker? Over here, on a fresh Ubuntu 22.04 install, the dockerfile runs without issues.

The logs suggest that you either did not start docker with the --privileged flag or are running on a different host than linux.

Note that the setup is only tested with Linux and other OSs (Windows/MacOS) are not likely to work, as firmadyne (one if FirmAFL dependencies) uses linux-specific tooling to mount images.

[EvE-VeV]

oh, thanks a lot, and i tried with --privileged, now it is working now :) and thanks for sharing your work with us, it makes experiments much easier ( shaking hands )