Custom symbols not working correctly

[oxnan]

@ikelos As a follow up to my comments in #413

I tried putting the symbols in symbols/linux which worked! now it seems something else is wrong:

INFO     volatility3.framework.automagic: Running automagic: LayerStacker
Level 6  volatility3.framework: Importing from the following paths: /tmp/_MEI1ve994/volatility3/framework/layers
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Bash.kernel
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using LimeStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using Elf64Stacker
Level 6  volatility3.framework.layers.elf: Exception: Bad magic 0xf000ff53 at file offset 0x0
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using QemuStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using AVMLStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using WindowsCrashDumpStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using VmwareStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using LinuxIntelStacker
DEBUG    volatility3.framework.automagic.linux: Identified banner: b'Linux version 4.9.0-11-amd64 (debian-kernel@lists.debian.org) (gcc version 6.3.0 20170516 (Debian 6.3.0-18+deb9u1) ) #1 SMP Debian 4.9.189-3+deb9u1 (2019-09-20)\n'

INFO     volatility3.schemas: Dependency for validation unavailable: jsonschema
DEBUG    volatility3.schemas: All validations will report success, even with malformed input
DEBUG    volatility3.framework.symbols: Unresolved reference: LintelStacker1!dma_coherent_mem
DEBUG    volatility3.framework.symbols: Unresolved reference: LintelStacker1!fscrypt_info
DEBUG    volatility3.framework.symbols: Unresolved reference: LintelStacker1!netns_ipvs
DEBUG    volatility3.framework.symbols: Unresolved reference: LintelStacker1!fscrypt_operations
DEBUG    volatility3.framework.symbols: Unresolved reference: LintelStacker1!mtd_info
DEBUG    volatility3.framework.symbols: Unresolved reference: LintelStacker1!assoc_array_ptr
DEBUG    volatility3.framework.symbols: Unresolved reference: LintelStacker1!nf_ct_event_notifier
DEBUG    volatility3.framework.symbols: Unresolved reference: LintelStacker1!nf_exp_event_notifier
DEBUG    volatility3.framework.symbols: Unresolved reference: LintelStacker1!ip_conntrack_stat
DEBUG    volatility3.framework.symbols: Unresolved reference: LintelStacker1!xt_table
DEBUG    volatility3.framework.symbols: Unresolved reference: LintelStacker1!mpls_route
DEBUG    volatility3.framework.symbols: Unresolved reference: LintelStacker1!nft_af_info
DEBUG    volatility3.framework.symbols: Unresolved reference: LintelStacker1!sctp_mib
DEBUG    volatility3.framework.symbols: Unresolved reference: LintelStacker1!ebt_table
DEBUG    volatility3.framework.symbols: Unresolved reference: LintelStacker1!dn_dev
DEBUG    volatility3.framework.symbols: Unresolved reference: LintelStacker1!garp_port
DEBUG    volatility3.framework.symbols: Unresolved reference: LintelStacker1!mpls_dev
DEBUG    volatility3.framework.symbols: Unresolved reference: LintelStacker1!mrp_port
DEBUG    volatility3.framework.symbols: Unresolved reference: LintelStacker1!tipc_bearer
DEBUG    volatility3.framework.symbols: Unresolved reference: LintelStacker1!pcpu_dstats
DEBUG    volatility3.framework.symbols: Unresolved reference: LintelStacker1!pcpu_vstats
DEBUG    volatility3.framework.symbols: Unresolved reference: LintelStacker1!dn_route
DEBUG    volatility3.framework.symbols: Unresolved reference: LintelStacker1!cfg80211_conn
DEBUG    volatility3.framework.symbols: Unresolved reference: LintelStacker1!cfg80211_cached_keys
DEBUG    volatility3.framework.symbols: Unresolved reference: LintelStacker1!cfg80211_internal_bss
DEBUG    volatility3.framework.symbols: Unresolved reference: LintelStacker1!libipw_device
DEBUG    volatility3.framework.automagic.linux: Linux ASLR shift values determined: physical 213a00000 virtual b200000
DEBUG    volatility3.framework.automagic.linux: DTB was found at: 0x215608000
Level 8  volatility3.framework.automagic.stacker: Stacked IntelLayer using LinuxIntelStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using LimeStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using Elf64Stacker
Level 6  volatility3.framework.layers.elf: Exception: Offset 0x0 does not exist within the base layer
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using QemuStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using AVMLStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using WindowsCrashDumpStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using VmwareStacker
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Bash.kernel.layer_name
Level 6  volatility3.framework: Importing from the following paths: /tmp/_MEI1ve994/volatility3/framework/layers
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Bash.kernel
Level 6  volatility3.framework: Importing from the following paths: /tmp/_MEI1ve994/volatility3/framework/layers
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Bash.kernel
Level 6  volatility3.framework: Importing from the following paths: /tmp/_MEI1ve994/volatility3/framework/layers
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.Bash.kernel
Level 6  volatility3.framework: Importing from the following paths: /tmp/_MEI1ve994/volatility3/framework/layers
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Bash.kernel.layer_name
Level 6  volatility3.framework: Importing from the following paths: /tmp/_MEI1ve994/volatility3/framework/layers
Level 6  volatility3.framework: Importing from the following paths: /tmp/_MEI1ve994/volatility3/framework/layers
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Bash.kernel.layer_name.memory_layer
Level 6  volatility3.framework: Importing from the following paths: /tmp/_MEI1ve994/volatility3/framework/layers
Level 6  volatility3.framework: Importing from the following paths: /tmp/_MEI1ve994/volatility3/framework/layers
Level 6  volatility3.framework: Importing from the following paths: /tmp/_MEI1ve994/volatility3/framework/layers
Level 6  volatility3.framework: Importing from the following paths: /tmp/_MEI1ve994/volatility3/framework/layers
Level 6  volatility3.framework: Importing from the following paths: /tmp/_MEI1ve994/volatility3/framework/layers
Level 6  volatility3.framework: Importing from the following paths: /tmp/_MEI1ve994/volatility3/framework/layers
Level 6  volatility3.framework: Importing from the following paths: /tmp/_MEI1ve994/volatility3/framework/layers
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Bash.kernel.symbol_table_name
Level 6  volatility3.framework: Importing from the following paths: /tmp/_MEI1ve994/volatility3/framework/layers
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.Bash.kernel.symbol_table_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Bash.kernel.symbol_table_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.Bash.kernel
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Bash.kernel
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.Bash
Level 6  volatility3.framework: Importing from the following paths: /tmp/_MEI1ve994/volatility3/framework/layers
Level 6  volatility3.framework: Importing from the following paths: /tmp/_MEI1ve994/volatility3/framework/layers
DEBUG    volatility3.framework.automagic.stacker: Stacked layers: ['IntelLayer', 'FileLayer']
INFO     volatility3.framework.automagic: Running automagic: LinuxSymbolFinder
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Bash.kernel.symbol_table_name
DEBUG    volatility3.framework.automagic.symbol_finder: Identified banner: b'Linux version 4.9.0-11-amd64 (debian-kernel@lists.debian.org) (gcc version 6.3.0 20170516 (Debian 6.3.0-18+deb9u1) ) #1 SMP Debian 4.9.189-3+deb9u1 (2019-09-20)\n'
DEBUG    volatility3.framework.automagic.symbol_finder: Using symbol library: file:///temp/symbols/linux/test.json
INFO     volatility3.schemas: Dependency for validation unavailable: jsonschema
DEBUG    volatility3.schemas: All validations will report success, even with malformed input
INFO     volatility3.framework.automagic: Running automagic: KernelModule
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Bash.kernel

PID	Process	CommandTime	Command
Level 6  volatility3.framework.symbols.intermed: Searching for symbols in /temp/symbols, /usr/bin/symbols, /usr/share/Volatility3/symbols, /tmp/_MEI1ve994/volatility3/symbols, /tmp/_MEI1ve994/volatility3/framework/symbols
INFO     volatility3.schemas: Dependency for validation unavailable: jsonschema
DEBUG    volatility3.schemas: All validations will report success, even with malformed input
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!dma_coherent_mem
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!fscrypt_info
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!netns_ipvs
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!fscrypt_operations
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!mtd_info
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!assoc_array_ptr
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!nf_ct_event_notifier
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!nf_exp_event_notifier
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!ip_conntrack_stat
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!xt_table
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!mpls_route
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!nft_af_info
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!sctp_mib
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!ebt_table
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!dn_dev
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!garp_port
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!mpls_dev
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!mrp_port
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!tipc_bearer
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!pcpu_dstats
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!pcpu_vstats
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!dn_route
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!cfg80211_conn
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!cfg80211_cached_keys
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!cfg80211_internal_bss
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!libipw_device

Not entirely sure what the problem here is, but you might :)

[ikelos]

Hmmmm, well, it looks at least like it found and used your symbols this time? Unfortunately it looks like there's a lot of structures that it can't find. It usually can't find a few, but that many suggests that there's something missing in the symbols file. I'm not sure what would cause that unfortunately. You could check the output of isfinfo and make sure there seem to be the right number of symbols and types and so on. Otherwise try some other plugins and check if it's just pslist, or all linux plugins that aren't working?

[oxnan]

Seems to me like all linux plugins are not working. looking at these:
volatility3.framework.symbols: Unresolved reference: symbol_table_name1!dma_coherent_mem the dma_coherent_mem is actually present in the symbol table (numbers are from ripgrep not the actual file):

68051-        "dma_mem": {
68052-          "type": {
68053-            "kind": "pointer",
68054-            "subtype": {
68055-              "kind": "struct",
68056:              "name": "dma_coherent_mem"
68057-            }
68058-          },
68059-          "offset": 552
68060-

this is the info from the isfinfo

URI	Valid	Number of base_types	Number of types	Number of symbols	Number of enums	Windows info	Linux banner	Mac banner

file:///temp/symbols/linux/test.json	Unknown	17	6088	99118	940	-	Linux version 4.9.0-11-amd64 (debian-kernel@lists.debian.org) (gcc version 6.3.0 20170516 (Debian 6.3.0-18+deb9u1) ) #1 SMP Debian 4.9.189-3+deb9u1 (2019-09-20)

the numbers seem similar to the other symbol tables.

I think it is a bit weird that it is missing symbol_table_name1 but that is maybe expected naming?

[ikelos]

So that's actually a member (dma_mem) of some other type, which is supposed to be a pointer to something of type dma_coherent_mem, but that type presumably isn't present in the types dictionary, and so the failure you get. The number of symbols looks around right. As I say, even on windows, you occasionally get forward references to types that aren't actually defined in the file, so I'm not sure whether that's the root of the problem or just a side effect of lots of debugging info.

The real question is whether the pointer we use to get to the task list is intact, and whether the task list has items in it. Unfortunately we don't seem to have many debugging statements that would tell us that, so I'm not sure what's the easiest way to proceed. Depending on how comfortable you are with python, volshell might be the way to try and debug this, you can essentially do the same steps as the pslist plugin and see why it's not returning you any results?

[oxnan]

Sure thing, I will try to give volshell a try. Usually I would just add debugging statements to the code myself, but unfortunately I don't have access to that on the system I am using.

is there documentation for volshell somewhere?

[ikelos]

This is the best we've got at the moment: https://volatility3.readthedocs.io/en/latest/volshell.html

Hopefully it'll be enough to get started?

[oxnan]

Hopefully that will do. will update if I find a solution :)

[oxnan]

looking at the dump with volshell shows me that it does not even find any running processes:

        vmlinux = context.modules[vmlinux_module_name]

        init_task = vmlinux.object_from_symbol(symbol_name = "init_task")

        for task in init_task.tasks:
                 print(task)

(the output from this is empty)

I am not sure if this is because it can't find them, or if it just does not know what to do.

[ikelos]

Well, it's very unlikely the operating system didn't have any running tasks, so it suggests either that the symbol we've got for init_task is incorrect, or that there was smear or something at the time. I'd suggest looking at the memory right around the offset of init_task (once you construct the object you can use init_task.vol.offset) and make sure it looks like it matches what the init_task structure would look like. If it does and the pointer for the tasks list points somewhere (rather than null) then we can manually check that there's no entries, and try to fathom what's going wrong.

That should tell us whether the init_task is legit and the list of tasks is bad, or the entire init_task structure is bad. Once we know that we know whether to try to find the init_task offset a different way (and the symbol address is bad, and work out why) or whether the structure's working fine and linux just didn't populate it for some reason...

[oxnan]

holy shit what the fuck is your timezone. I will try to do what you suggested and return with my findings :)

[oxnan]

I think I see the problem...

(layer_name) >>> init_task.vol.offset
281473045304576

this offset is 256 TB

any ideas or just a broken symbol table?

[ikelos]

I find it best to represent things as hex, which in this case would be 0xffff8ce11500. If that gets masked by the address space (which it almost certainly will), it probably comes out at 0x8ce11500, which depending on how the kernel slices up memory is probably an in-kernel address (I think it's usually a 50/50 slice, so 0x80000000 would be the start of kernel memory). So that address may be just fine, I'd suggest trying to do a read on the virtual layer at that address and see what data it might give you?

[ikelos]

Volatility should automask the address, so you can lookup the full one and it ought to take you to the right place (if it exists). If it doesn't you should get a page fault, and it's entirely possible we try to smooth over such errors (even though I think that's a bad idea because it leads to confusions like this), but at least it'll give us the next step to check... 5:)

[oxnan]

Ah thanks, I will try to figure out if I can find out how to read from a specific address in the layer (unless you know the command by heart?)

[ikelos]

Sure, it's layer.read(<offset>, <length>), but you need to get the layer first which can be the more complex bit. 5:)

As to my timezone, it's supposed to be UK but I think I shifted a little towards the US in my college days and never shifted back, so probably somewhere midatlantic. I usually go to sleep at 2am... 5;P

[oxnan]

oh, not far from my timezone then. I will try to see if I can get the layer then. But that should be easy since you have such a vast documentation XD