MSTICPy Council 2023-02-29

[ianhelle]

MSTICPy Council 2023-02-29

Attendees:

• Thomas Roccia
• Ryan Cobb
• Mehmet Ergene
• Mark Kendrick
• Julien Touche
• Pete Bryan
• Ashwin Patil

Use of GPT with notebooks/msticpy:

• prompt engineering is key
  • MK - My favorite website right now:
    https://learnprompting.org/ (protip: prompting is everything)
  • PB - I also really liked this -
    https://github.com/dair-ai/Prompt-Engineering-Guide
  • MK - Also this is worth looking into:
    https://github.com/hwchase17/langchain
• TR - Here is the list of prompt I was experimenting on MSTICpy:
  GPT_MSTICpy

- Identify possible signs of a cyber attack in the following logs:  [insert log data here].
- Analyze this network traffic data [insert data here] and determine if there are any indicators of compromise.
- Given the following list of IP addresses [insert list here], identify any known malicious IP addresses and their associated threat
actors.
- Review the following email [insert email content here] for signs of phishing or other malicious content.
- Based on the following user behavior data [insert data here], identify any suspicious or anomalous activities that may suggest a security breach.
- Analyze the following domain names [insert domain list here] and identify any known malicious domains or those associated with threat actors.
- Given these file hashes [insert file hash list here], identify any known malware samples and their associated threat groups.
- Review the following incident report [insert report here] and suggest possible attack vectors that the threat actors may have used.
- Assess the security of this web application based on the following logs [insert log data here] and identify any potential vulnerabilities
or signs of exploitation.
- Based on the following threat intelligence report 

• TR to start dicsussion on MP Github

Current PRs/Release plans

• PRs for 2.4.0
  • We've pushed a bunch of 2.4.0 PRs to 2.5.0 because 2.3.2 got
    too big:
    • Items in 2.4.0
      • Pulsedive TI provider
      • Dynamic summaries (Sentinel), Az Authentication, Process
        Tree fixes.
    • Items for 2.5.0
      • MSTICPy extensibility
      • OSQuery and Velociraptor
        providers
        (also PR add LocalOsquery driver based on LocalData one #624 and Support extensibility with plugins (TI, Data, Notebooklets, etc.) #625)
      • Panel Tabulator data
        viewer
      • Multithreading support for additional
        connections
      • Bokeh heatmap visualization
      • Download Sentinel hunting/detection queries as MP
        queries
      • Carbon black data provider
      • Updated Splunk driver

OSQuery and Velociraptor data providers

• Julien supplied most of the code and expertise. We also
  made OSQuery driver auto-create queries based on event
  types in logs
• ToDo - check on VR logs to see if we can do something similar

MSTICPy Plugins

• Ian's PR allows authoring of local extensions (put in named folder
  somewhere)
• Ryan has follow-up code that lets you do installable extensions. Key
  points:
  • Can be authored with their own dependencies independent of MP
  • Installed as separate PIP packages
  • Cookie cutter allows authors to create boilerplate package
    • Would be nice to have templates for DataProvider and
      TIProvider
  • We could move some code (esp with separate dependencies) out
    into own repos/PyPI

Panel Dataviewer

• Uses panel tabulator
  • Auto filtering, sorting, selection
  • I've added some tweaks
    • Interactive column selection
    • Detail display in per-row drop down of selected columns
• Currently incompatibility with Bokeh 3.0
  • ME - panel will be compatible with bokeh 3.0 in v1 release I
    think. Initial Bokeh 3.0 compatibility holoviz/panel#4098
• RC - One thing we might need to consider: if the dataframes
  contain columns of python objects that are not automatically
  json serializable, you get some weird rendering where it does
  the classic [Object object] representation in the table. It is
  pretty easy to address, but it confused some folks when we
  starting migrating to it.

Multi-threaded extension for data providers

• Would like to provider multi-threading for splitting queries across
  multiple instances of same provider type
• Would like to add ability to split by date and run partial queries
  in parallel
• Has prompted work on moving from Kqlmagic to azure.monitor.query and
  azure.data.kusto so that we can support multi-threading
• RC - shouldn't mix async and threading constructs (sample code I
  posted in Discord chat)
• IH - Construct comes from official Python docs
• MK - has sample code for async data querying in notebooks.

ProcessTree display related non-process events

• Comes from internal request

• Plan is to join non-process data where event has a clear parent
  process link

• Display these in process tree as child events (need some formatting
  changes)

• ME and RC - should look at graph as alternative

• IH - Graphs are great but too dense to be useful when many nodes. I
  think we can do both - so display proctree subset as graph or tree.

• AP - Mermaid diagrams - mermaid diagrams could be another option
  from jupyter notebook perspective.. will have to frame data in a
  way.. may not be scalable on large dataframes About Mermaid |
  Mermaid

• ME - github.com/Cyb3r-Monk/forensic_as_code/blob/master/case_2_writeup.ipynb
  there is some interesting graphs at the bottom