Disable recursive targets in SANS_Triage

[respondersGY]

Description

Please include a summary of the change and (if applicable) which issue is fixed.

Checklist:

Please replace every instance of [ ] with [X] OR click on the checkboxes after you submit your PR

• I have generated a unique GUID for my Target(s)/Module(s)
• I have placed the Target(s)/Module(s) in an appropriate subfolder in Targets or Modules. If one doesn't exist, I have either added it to the Misc folder or created a relevant subfolder with justification
• I have set or updated the version of my Target(s)/Module(s)
• I have verified that KAPE parses the Target(s)/Module(s) successfully via kape.exe, using --tlist/--mlist and corrected any errors
• I have validated my Target(s)/Module(s) against test data and verified they are working as intended
• I have made an attempt to document the artifacts within the Target(s) or Module(s) I am submitting. If documentation doesn't exist, I have placed N/A underneath the Documentation header
• For Targets, I have consulted either the Target Guide, Target Template, Compound Target Guide, or Compound Target Template to ensure my Target(s) follow the same format
• For Modules, I have consulted either the Module Guide, Module Template, Compound Module Guide, or Compound Module Template to ensure my Module(s) follow the same format

If your submission involves an SQLite database, have you considered making an SQLECmd Map for the SQLite database? If you make a Map, please add the SQLite database to the SQLiteDatabases.tkape Compound Target.

Thank you for your submission and for contributing to the DFIR community!

[reece394]

This commit seems a bit heavy handed. For CloudStorage_Metadata.tkape for example all of those rules will exit if the folders don't exist apart from the rclone.conf one. I am planning on rewriting that one similar to the Advanced IP and Port Scanner Target with the caveat of letting the user know in order to cover all of rclone.conf paths recursive would be required.

FTPClients.tkape the only one that is a problem is WinSCP.tkape .ini which unfortunately we cannot rewrite as portable WinSCP can literally be anywhere on disk.

NetworkScanner.tkape I rewrote the Advanced IP and Port Scanner Targets already with only one recursive rule being in place for favorites due to the behaviour of that one being different. SoftPerfectNetscan.tkape cannot be rewritten either as that has been observed on many cases in different paths as well.

In summary the only ones I believe are a problem are WinSCP.tkape, SoftPerfectNetscan.tkape and RcloneConf.tkape. With Advanced IP and Port Scanner Targets telling the user to comment out the recursive favorites sections as required.

The reason all of these were added to the SANS_Triage rule FYI is they are covered in SANS material/ SANS posters

[EricZimmerman]

I'd like to see a comparison for time taken with and with our the recursion on as well. Is it worth it

[reece394]

I'd like to see a comparison for time taken with and with our the recursion on as well. Is it worth it

I will say this every recursion increases per the number of files on the disk. So on a small VM when I rewrote the Advanced IP/Port Scanner rules. It was 2-3 seconds with hard coded paths vs 80-90 seconds ish for 1 full disk recursive rule. On bigger machines it can scale exponentially with some triages going into hours. And the more recursive rules the time scales per number of recursive rules.

[EricZimmerman]

That is wholly the wrong approach then.

This is why you should just target a single file, the mft, and dump it, THEN look for files that may be anywhere on the disk.

To do it the either way is crazy

[reece394]

I believe in a lot of IR workflows people collect the triage once so I can understand why the rules were written that way as they wanted the kitchen sink approach. But yeah the WinSCP, Rclone and Netscan ones are very expensive and waste a lot of time on machines that don’t even have it.

I can optimise Rclone a lot more but WinSCP Portable and Netscan portable are not viable

[AndrewRathbun]

@respondersGY thank you for the PR but we should definitely approach this a different way before modifying the SANS_Triage Compound Target. Out of respect for the work @mark-hallman has done on this Target, and for its role in SANS courses, I'd want to ensure he's consulted before we do any major subtractions from the Target.

As always, the !Local folder is always available to make your own modifcations to these pre-existing Targets, compound or otherwise, so you can have your own workflow that works for you, but I know a lot of people rely on this Compound Target as is, so I'm very hesitant to make the changes as they're currently proposed without checking other options first, which were mentioned above. I will keep this open for now but this is probably better as an Issue than a PR at this point.

[respondersGY]

@EricZimmerman
The use of recursive targets makes SANS_Triage unusable for IR investigations. I stopped using SANS_Triage a while ago so I am luckily not impacted.

I will say this every recursion increases per the number of files on the disk. So on a small VM when I rewrote the Advanced IP/Port Scanner rules. It was 2-3 seconds with hard coded paths vs 80-90 seconds ish for 1 full disk recursive rule. On bigger machines it can scale exponentially with some triages going into hours. And the more recursive rules the time scales per number of recursive rules.

Testing on VMs is not the same as using KapeFiles on hundreds of different production systems. The previous commit causes collections taking hours instead of minutes, which has an impact on everybody that uses KapeFiles. @reece394 do you understand the impact of this?

Velociraptor has implemented a feature to disable these recursive targets due to issues like this: Velocidex/velociraptor#4205.

DropVerySlowRules: Some targets specify globs which need to examine every file on the disk.For example, DirectoryTraversal_AudioFiles has a glob similar to C:***.{3gp,aa,aac,act,aiff}.
This type of search is very slow as it needs to examine every file on disk. By default we disable these rules because they are too slow to be useful. If you really want them enabled, switch this setting off, but collection time will increase significantly.

I don't think that is a good sign.

Out of respect for the work @mark-hallman has done on this Target, and for its role in SANS courses, I'd want to ensure he's consulted before we do any major subtractions from the Target.

I don't understand why a change related to a SANS course is more important than the quality of the commit

@AndrewRathbun I am gonna leave this PR open for you to discuss this further, but I am not actively working on a different solution.

[EricZimmerman]

I wouldn't recommend using that in ir. Use it as an example. Make your own. Comment the ones out that don't work for you. There is no single thing that will do all things for all people.

I wouldn't be wading thru a file system for a random name. That's what the mft is for, and that should always be pulled

[respondersGY]

I understand that but there are many other IR consultants that blindly use SANS_Triage, that are impacted due to the scalability and speed of KapeFiles and other software packages that utilize KapeFiles.

[AndrewRathbun]

I've never used the SANS_Triage Target in production. I've always used KapeTriage and built up from there, if needed. I've never had any issues with that Target, personally. My personal take on this specific Target was that it was meant for SANS to use in class since it was curated by course authors. KapeTriage is geared towards IR, and BasicCollection is geared more for DF. Those are always good starting points and you can tack on whatever else in addition to those.

[reece394]

Testing on VMs is not the same as using KapeFiles on hundreds of different production systems. The previous commit causes collections taking hours instead of minutes, which has an impact on everybody that uses KapeFiles. @reece394 do you understand the impact of this?

If you fully read my comment you would realise I used the VM times as a comparator of how much faster hardcoded paths are vs recursive. Why would I have bothered rewriting Advanced IP/Port Scanner and Rclone rules and added comments regarding this inside the rules themselves otherwise? In the rules I am giving people the choice to disable them via commenting out the recursive ones. If I commented these out by default other projects such as Velociraptor Triage Artifacts repo wouldn't be able to convert them from comments to rules as far as I am aware. I made the rules comprehensive in case people want everything covered. It is not just IR teams that make use of this tool FYI.