v2.5.2: Hotfix for Holoviz panel compatibility

• Release is mainly to align bokeh version requirements with the new release of Holoviz panel.
  • moved bokeh from <3.0.0 to < 4.0.0
• Also fixes an issue with the MicrosoftSentinel attribute disappearing from msticpy

What's Changed

• Ianhelle/hotfix 2.5.2 2023 06 08 by @ianhelle in #676

Full Changelog: v2.5.1...v2.5.2

v2.5.1: Hotfix for import failure

Some minor fixes that address:

• importing msticpy without some non-default azure packages installed failed
• added more resiliency to query reader so that the whole thing does not fail if there is bad query file.
• removed initialization dependency on azure-resourcegraph in MicrosoftSentinel class.

What's Changed

• Hotfix for v2.5.1 by @ianhelle in #672

Full Changelog: v2.5.0...v2.5.1

v2.5.0

Summary of main changes

• New MS Sentinel and Azure Kusto drivers/data providers - these include support for multi-threaded parallel queries, proxies and user-defined query timeouts.
• Extensibility model for MSTICPy - you can create private data providers, TI and Context providers and load them into MSTICPy alongside the built-in providers.
• MS Sentinel repo query download - add current detection and hunting queries from the Sentinel repo as Sentinel queries runnable from MSTICPy/notebooks
• OSQuery data provider - makes it easy to import OS Query logs to dataframes to do additional processing/analysis on them.
• Panel tabulator now supported as default data viewer (a million times better than the one we built!)

More details on these changes below

Sentinel and Kusto provider new drivers

This change adds replacement drivers for the MSSentinel and Kusto data providers.
In place of Kqlmagic, these drivers use the azure-kusto-data and azure-monitor-query SDKs, respectively.

Currently these drivers are enabled alongside the existing versions - in a future version we will make these the defaults for Sentinel and Kusto.

Some of the main changes with these new versions:

• They use the provider names MSSentinel_New and Kusto_New when creating a QueryProvider instance.
• Both drivers support setting proxies for firewall-protected networks
• Both drivers support custom configuration of the server timeout via a timeout parameter
• Both drivers use integrated Azure authentication by default and support the auth_types and tenant_id parameters used elsewhere
  in MSTICPy
• Both drivers support threaded execution for parallelizing queries (across multiple workspaces/clusters or split by time) - this functionality, however, will be exposed in v2.6.0 via a separate feature.
• The MSSentinel_New driver allows you to execute the same query across multiple workspaces in parallel and returns the results as a combined dataframe.
• Some of the previous parameters have been deprecated:
  • mp_az_auth is replaced by auth_types (the former still works but will be removed in a future release).
  • mp_az_auth_tenant_id is replaced by tenant_id (the former is not supported in the new providers).

Note: in order to use these new versions you must have the azure-kusto-data and/or azure-monitor-query Python packages
installed. You can install these using pip install msticpy[azure] or install them separately using pip.

For more details on how to use these providers, see:

• Documentation for the new Sentinel provider
• Documentation for the new Kusto provider

Changes specific to the MS Sentinel provider

Connecting to multiple workspaces allows you to run queries across these workspaces and return the combined results as a single Pandas DataFrame. The workspaces must use common authentication credentials and should have the same data schema.

# use workspace names if these workspaces are configured in msticpyconfig.yaml
qry_prov.connect(workspaces=["Default", "MyOtherWorkspace"])

# or use a list of workspace IDs
qry_prov.connect(workspaces=["e6b4bc15-119b-45a2-8f3d-c39ed384ed37", "b17e0e5a...."])

# run query against connected workspaces
qry_prov.SecurityAlert.list_alerts()

Changes specific to the Kusto provider

• The settings format has changed (although the existing format is still supported albeit with some limited functionality).
  See the Kusto provider documentation for details.
• In the earlier implementation of driver you can specify a new cluster to connect to in when executing a query. This is no longer supported.
  Once the provider is connected to a cluster it will only execute queries against that cluster. (You can however, call the connect() function to
  connect the provider to a new cluster before running the query.)
• Filtering pre-defined queries by cluster. If you have MSTICPy query definitions for the Kusto provider, these will all be attached as methods
  of the QueryProvider, when it is created. However, as soon as you connect to a specific cluster, the queries will be filtered down to show
  only the queries that are intended to run on that cluster.
• New APIs (exposed via the query_provider):
  • get_database_names() - return list of databases for the connected cluster
  • get_database_schema() - return table schema for a database in the cluster
  • configured_clusters() - return a list of clusters configured in msticpyconfig.yaml
  • set_cluster() - switch connected to cluster to a different one (you can use the connect method to do this, which also lets you specify
    additional connection parameters).

Extend MSTICPy with Data provider, TI provider and Context provider plugins

This adds the ability to "side-load" data providers, TI providers and context providers. If you have a data/TI/context source that you want to use in MSTICPy you can write a provider (deriving from one of the base provider classes) and tell MSTICPy where to load it from.

In a future release we'll build on this framework to let you install plugins from external packages and provide some cookie-cutter templates to generate skelton provider classes.

Writing a TI provider or Context provider (partial example)

    class TIProviderHttpTest(HttpTIProvider):
        """Custom IT provider TI HTTP."""

        PROVIDER_NAME = "MyTIProvider"
        _BASE_URL = "https://api.service.com"
        _QUERIES = _QUERIES = {
            "ipv4": APILookupParams(path="/api/v1/indicators/IPv4/{observable}/general"),
            "ipv6": APILookupParams(path="/api/v1/indicators/IPv6/{observable}/general"),

Telling MSTICPy to load the plugins

Load on demand

    import msticpy as mp

    mp.load_plugins(plugin_paths="/my_modules")

    # or multiple paths
    mp.load_plugins(
        plugin_paths=["./my_modules", "./my_other_modules"]
    )

Or specify in msticpyconfig.yaml

        ...
        Custom:
            - "testdata"
    PluginFolders:
        - tests/testdata/plugins
    Azure:
        ...

See the new Extending Msticpy section in our docs.
If you want to contribute any of the drivers you write, also check out the new Development section in the MSTICPy docs.

OS Query Provider

Great contribution from @juju4 here (with a bit of collaboration with @ianhelle).
Create a MSTICPy QueryProvider with the data environment name "OSQueryLogs" and load forensic logs from OSQuery.

# specify one or more paths to folders where the dumped JSON OSQuery logs can be found
qry_prov = mp.QueryProvider("OSQueryLogs", data_paths=["~/logs1", "~/logs2"])
qry_prov.connect()
qry_prov.list_queries()

['osquery.acpi_tables',
'osquery.device_nodes',
'osquery.dns_resolvers',
'osquery.events',
'osquery.fim',
'osquery.last',
'osquery.listening_ports',
'osquery.logged_in_users',
'osquery.mounts',
'osquery.open_sockets',
...

Each event type is available as a separate function that returns a pandas DataFrame with the combined events from the logs for that type

qry_prov.osquery.processes()

Downloading Sentinel Detection and Hunting queries for the Sentinel Query Provider

We haven't finished documenting this or integrating it fully, so will leave the full announcement of this until the next release. If you want to play around with it look at the following module:

from msticpy.data.drivers.sentinel_query_reader import download_and_write_sentinel_queries

download_and_write_sentinel_queries(
    query_type="Hunting",    # or "Detections"
    yaml_output_folder="./sentinel_hunting",
)
qry_prov = mp.QueryProvider("Sentinel_New", query_paths=["./sentinel_hunting"])

Since there are lots of queries, the import might take a little while in its current form.

Panel Tabulator now available as a DataViewer control.

HoloViz Panel is a powerful Bokeh-based data exploration & web app framework for Python. It has an immense amount of functionality that you can read about at the Panel documentation site. You need to have panel installed for the Tabulator-based viewer to run (pip install panel).

Unfortunately, the documentation for our Tabulator view never made it into this release but most of the functionality should be obvious from the UI. There are some useful load-time parameters that you can use at startup for things like:

• selecting an initial column set.
• adding columns to a per-row expando pane - useful for viewing long column values such as command-line.

We also kept the column chooser widget from the previous data viewer so that you can interactively select which columns to display. The Tabulator MSTICPy initialization parameters are documented in the code.

Most of the Tabulator init parameters are also passed through to the underlying control - which give you an immense amount of control over the viewer. These are described in the Panel Tabulator documentation

Big thanks to our contributors in this release!

@juju4
@jannieli
@ianhelle
@Tatsuya-hasegawa
@FlorianBracq
@danielyates2
@petebryan
@ashwin-patil

What's Changed PR Reference

• Up...

v2.4.0

Main changes for this release

There are no huge changes in this release but a good variety of important updates and fixes.
We're also delighted to welcome 3 new contributors to the MSTICPy family:

• @ZeArioch
• @ctoma73
• @jllangley

Thanks so much!

New Threat Intel provider for Pulsedive from @fr0gger #609

This includes a standard MSTICPy TI provider (so you can include it in you collection of providers used for
regular TI checks on IPs, URLs, etc. This provider also contain a few custom methods that let to query
some other facets of the Pulsedive data. For example, the explore function that allows you to use
the pulsedive query language

pddetail = pdlookup.explore(query="ioc=pulsedive.com or threat=AgentTesla")
pddetail

You can also request a can on a domain or URL

pdscan = pdlookup.scan(observable= "alvoportas.com.br")
pdscan

To use any of the Pulsedive features you'll need an account and API key from Pulsedive
See more details of the usage in the Pulsedive notebook

Process tree updates #637

• @ZeArioch added Process Tree support for FireEye HX data so it should be automatically recognized and render correct
• We also added the ability to export a process tree as a text object - which is useful if you want to copy and paste
  a tree or part of it into a non-HTML document. See the Process Tree docs for more details

+--  Process: C:Program FilesMicrosoft Monitoring AgentAgentMonitoringHost.exe
   PID: 0x888
   Time: 1970-01-01 00:00:00+00:00
   Cmdline: nan
   Account: nan  LoginID: 0x3e7
   +--  Process: C:WindowsSystem32cscript.exe  PID: 0x364
      Time: 2019-01-15 04:15:26+00:00
      Cmdline: "C:Windowssystem32cscript.exe" /nologo
         "MonitorKnowledgeDiscovery.vbs"
      Account: WORKGROUPMSTICAlertsWin1$  LoginID: 0x3e7
   +--  Process: C:Program FilesMicrosoft Monitoring AgentAgentHealth Service
      StateCT_602681692NativeDSCDesiredStateConfigurationASMHost.exe  PID:
      0x1c4
      Time: 2019-01-15 04:16:24.007000+00:00
      Cmdline: "C:Program FilesMicrosoft Monitoring AgentAgentHealth
         Service
         StateCT_602681692NativeDSCDesiredStateConfigurationASMHost.exe"
         GetInventory "C:Program FilesMicrosoft Monitoring
         AgentAgentHealth Service
         StateCT_602681692workServiceStateServiceState.mof" "C:Program
         FilesMicrosoft Monitoring AgentAgentHealth Service
         StateCT_602681692workServiceState"
      Account: WORKGROUPMSTICAlertsWin1$  LoginID: 0x3e7

Miscellaneous fixes #644

This sounds like a small item but contain several important fixes:

• Azure authentication (az_connect) now avoids throwing exceptions if you ask it to use authentication types (e.g. clientsecret) where parameters are not passed (or available in environment variables). It will now just ignore those credential types and only throw an exception if no usable credential types remain.
• Updates to API documentation
• A new IPython magic "%save_to_cell" - this lets you save a Python object (e.g. a DataFrame to a base64-encoded blob in a new cell. The cell contains code to restore the original data. This is subject to the usual caveats about pickle - including the security ones. Do Not run a cell that unpickles some arbitrary data in notebooks that you do not trust.
• A bunch of changes/fixes to the Sentinel APIs
  • Most of these are fixes related to the newly-supported Sentinel Dynamic Summaries feature
  • Some minor fixes also to Sentinel core

Python Logging support #640

We should have had this from the beginning but it's never too late to start correcting your mistakes.
We've implemented a central logging module and started to instrument some of the code that is especially complex
and where people often get stuck with cryptic errors. E.g. the init_notebook function.
We also enabled in in the authentication modules (az_connect) in #644
Most of the time, this will be invisible. However, if you need it you can just do the following:

# import msticpy as mp   # if not already imported
mp.set_logging_level("INFO")

Then re-run the function that you are having trouble with again.
You can also use the MSTICPYLOGLEVEL variable to control this. And, if you want to log to a file, set the env variable MSTICPYLOGFILE to the path of your log file. (You'll need to restart the kernel/python session and reload MSTICPy for this to take effect).

Support for Bokeh 3.0 #630 #642 and #650

@ctoma73 did some awesome work to track down problems with compatibility with Bokeh 3.0 and fix all of them (a lot were tedious mypy/linting fixes due to some of the more dynamic nature of the Bokeh 3.0 object model).
You'll notice in #650 that we still have Bokeh 2.4.3 in the MSTICPy requirements. We're not going to change that just yet since we want compatibility with PyViz/HoloViz panel - you will likely see some panel-related features in the next minor release.
Despite this (and assuming you can ignore some pip warning about MSTICPy not being compatible with Bokeh 3.x) you can install Bokeh 3.0 after MSTICPy and enjoy the delights of the new release. All of our code should be compatible (tested with 3.0.0 and 3.1.0).

That's all for this release.
We'll likely be doing a follow-on 2.5.0 release that will include several contributions from our 2023 Hackmonth (which turned into a HackNMonths event).

What's Changed

• Add support for FireEye HX acquisition packages in process_tree by @ZeArioch in #616
• Adding Pulsedive as Threat Intel provider by @fr0gger in #609
• Fix error when latest version 3.0.3 of bokeh is installed by @ctoma73 in #630
• Adding logging and updating settings access by @ianhelle in #640
• ProcTree and init_notebook fixes by @ianhelle in #637
• Adding data query paths test for DEX support by @ianhelle in #638
• Fixing RangeTool with bokeh 3.1.0 not a GestureTool by @ctoma73 in #642
• Modified the upload_df method to split the data into batches of 10,00… by @jllangley in #633
• Misc updates for 2.3.2 release: by @ianhelle in #644
• Reverting to bokeh version 2.4.3 for default install by @ianhelle in #650

New Contributors

• @ZeArioch made their first contribution in #616
• @ctoma73 made their first contribution in #630
• @jllangley made their first contribution in #633

Full Changelog: v2.3.1...v2.4.0

MSTICPy Feb 2023 Fixes

This is minor release with mostly fixes.

Some higlights from the #631 PR

#629 - You can now suppress progress bar for Threat Intel lookups (useful to avoid screen mess
when running multiple lookups from other code)

  tilookup.lookup_iocs(data, progress=False)

#572 - We've had a long-running issue in Azure Machine Learning where the UI does not correctly
handle javascript written by the notebook. This results in JS code in the output cells. While we're waiting
for AML to re-adopt the latest Azure Notebooks package and get rid of this bug altogether we've
added a fix to suppress javascript text for out Kqlmagic data provider

• Fix to Azure ML use - automatic creation of msticpyconfig.yaml was writing the file to
  the wrong place, so users always got the message that no config file was found.

• We had a request (again for batch jobs) to remove automatic display of license information in the geoip module.

• Using MSTICPy offline or in isolated environment - it has always been our goal to support this but
  we recently discovered that we were running a check_version call from init_notebook. This function
  did not handle network failure and crashed the entire init_notebook process. This has been fixed
  so should be runnable offline or in air-gapped networks.

• Related to this we've also cleaned up remaining units tests that make outbound network requests.

Full Changelist

• Adding job to file issue if main build fails. by @ianhelle in #613
• Removing prospector from CI build by @ianhelle in #619
• Reverting PR #496 - Removing blank sub-id from resource graph list by @ianhelle in #621
• Resolved issues with nextLink following in Sentinel API calls by @petebryan in #617
• Fix MDE procschema by @rrevuelta in #626
• Bump sphinx-rtd-theme from 1.1.1 to 1.2.0 by @dependabot in #628
• Bump sphinx from 5.3.0 to 6.1.3 by @dependabot in #610
• Ianhelle/misc fixes 2023 02 17 by @ianhelle in #631

New Contributors

• @rrevuelta made their first contribution in #626

Full Changelog: v2.2.3...v2.3.1

v2.3.0

Some new data-related features in this release.

• Support for the new (still in preview at time of writing) Dynamic Summaries feature of MS Sentinel
• Added ability to create and use "ad-hoc" parameterized queries for data providers
• Simple search mechanism for finding queries
• Support for JSON queries for CyberReason

Support for Microsoft Sentinel Dynamic Summaries

Dynamic Summaries are a Sentinel feature that allow you to persist results of
query jobs in a summarized/serialized form. This might be useful for keeping
results of daily watch jobs, for example. We will be using it in MSTICPy notebooks
to publish more complex result sets from automated notebook runs.

MSTICPy operations available include:

• Retrieve list of current dynamic Summaries
• Retrieve a full dynamic summary
• Create a dynamic summary
• Delete a dynamic summary
• Update an existing dynamic summary

Examples:

# list dynamic summaries
sentinel.list_dynamic_summaries()

# create a dynamic summary in Sentinel
sentinel.connect()
sentinel.create_dynamic_summary(
    name="My_XYZ_Summary",
    description="Summarizing the running of the XYZ job.",
    data=summary_df,
    tactics=["discovery", "exploitation"],
    techniques=["T1064", "T1286"],
    search_key="host.domain.dom",
)

The MSTICPy support also includes a DynamicSummary class that lets you
manipulate dynamic summary objects more easily

  # can also import the class directly
  # from msticpy.context.azure.sentinel_dynamic import DynamicSummary
  # dyn_summary = DynamicSummary(....)
  # This example shows using the "factory" method - new_dynamic_summary
  dyn_summary = sentinel.new_dynamic_summary(
      summary_name="My new summary",
      summary_description="Description of summary",
      source_info={"TI Records": "misc"},
      summary_items=ti_summary_df,
  )
  # Add the local summary object to add to the Sentinel dynamic summaries.
  sentinel.create_dynamic_summary(dyn_summary)

# Retrieve a dynamic summary from Sentinel
dyn_summary = sentinel.get_dynamic_summary(
      summary_id="cea27320-829c-4654-bbf0-b14367483418"
)
# the return value is a DynamicSummary object
dyn_summary

  DynamicSummary(id=cea27320-829c-4654-bbf0-b14367483418, name=test2, items=0)

By default get_dynamic_summary returns the header data for the summary.

The next example shows how you can also fetch full data for the dynamic
summary (by adding summary_items=True). From the returned object,
you can convert the summary items to a pandas DataFrame.

Note: fetching summary items is done via the Sentinel QueryProvider
since the APIs do not support retrieving these.

    dyn_summary = sentinel.get_dynamic_summary(
        summary_id="cea27320-829c-4654-bbf0-b14367483418",
        summary_items=True
    )

dyn_summary.to_df()

index	Ioc	IocType	QuerySubtype	Provider	Result	Severity	Details	TimeGenerated
OTX	hXXp://38[.]75[.]37[.]1/static/encrypt.min.js	url		OTX	True	2	{‘pulse_count’: 3, ‘names’: [‘Underminer EK’	2022-12-15 01:55:15.135136+00:00
VirusTotal	hXXp://38[.]75[.]37[.]1/static/encrypt.min.js	url		VirusTotal	False	0	Request forbidden. Allowed query rate may ha	2022-12-15 01:55:15.135136+00:00
XForce	hXXp://38[.]75[.]37[.]1/static/encrypt.min.js	url		XForce

You can also create dynamic summaries from a DataFrame and append
DataFrame records to an existing dynamic summary.

Read the full documentation in MSTICPy Sentinel Dynamic Summaries doc

New QueryProvider API to dynamically add a parameterized query.

MSTICPy has always supported the ability to run ad hoc text queries for different providers
and return the results as a DataFrame. Using a static query string like this is quick and easy
if you only want to run a query once but what if you want to re-run with different time
range or host name? A lot of tedious editing or string search/replace!

Adding a full query template to MSTICPy, on the other hand, is overkill for this kind of thing.
Dynamic parameterized queries are especially suited for notebooks - you can create an
in-line parameterized query and have it update with the new parameters every time
you run the notebook.

To use dynamic queries - define the query with parameter placeholders (delimited
with curly braces "{" and "}"), then create parameter objects (these handle any special
formatting for datetimes, lists, etc.).
You add the list of parameter objects along with the replaceable parameter values
when you run the query, as shown below.

# intialize a query provider
qry_prov = mp.QueryProvider("MSSentinel")

# define a query
query = """
SecurityEvent
| where EventID == {event_id}
| where TimeGenerated between (datetime({start}) .. datetime({end}))
| where Computer has "{host_name}"
"""
# define the query parameters
qp_host = qry_prov.Param("host_name", "str", "Name of Host")
qp_start = qry_prov.Param("start", "datetime")
qp_end = qry_prov.Param("end", "datetime")
qp_evt = qry_prov.Param("event_id", "int", None, 4688)

# add the query
qry_prov.add_custom_query(
    name="get_host_events",
    query=query,
    family="Custom",
    parameters=[qp_host, qp_start, qp_end, qp_evt]
)

# query is now available as
qry_prov.Custom.get_host_events(host_name="MyPC"....)

See Dynamically Adding Queries in MSTICPy Docs

QueryProvider - Query Search

As the number of queries for some providers grows, it has become more difficult to quickly
find the right query. We've implemented a simple search capability that lets you search
over the names or properties of queries. It takes four parameters:

• search - search terms to look for in the
  query name, description, parameter names, table and query text.
• table - search terms to match on the target table of the query.
  (note: not all queries have the table parameter defined in their metadata)
• param - search terms to match on a parameter name
• case - boolean to force case-sensitive matching (default is case-sensitive).

The first three parameters can be a simple string or an iterable (e.g. list, tuple)
of search terms. The search terms are treated as regular expressions. This
means that a the search terms are treated as substrings (if no other
regular expression syntax is included).

Find all queries that have the term "syslog" in their properties

    qry_prov.search("syslog")
    # equivalent to qry_prov.search(search="syslog")

    ['LinuxSyslog.all_syslog',
    'LinuxSyslog.cron_activity',
    'LinuxSyslog.list_account_logon_failures',
    ...

See Search queries in MSTICPY Docs

Support for JSON queries in Data Providers

@FlorianBracq has updated the CyberReason data provider so that it supports JSON queries. The
mechanism that we used for KQL and SQL queries breaks JSON since it is a simple string substitution.
Other data sources that use JSON queries include Elastic - we are planning to leverage the same
mechanism to support parameterized Elastic queries in a future release.
Thanks @FlorianBracq!

What Else has Changed?

• Kql query formatting by @FlorianBracq in #595
• Fix minor linting issues in main by @petebryan in #604
• Updated M365D and MDE data connectors with correct scopes when using delegated auth. by @petebryan in #580
• Ianhelle/remove extranous nb 2022 11 28 by @ianhelle in #588
• Enable native JSON support for Data Providers + move Cybereason driver to native JSON by @FlorianBracq in #584
• Adding query search to data_providers.py by @ianhelle in #587
• Fix typo by @FlorianBracq in #606
• Ianhelle/mypy cache 2023 01 17 by @ianhelle in #608
• Added API to QueryProvider to add a custom query at runtime by @ianhelle in #586
• Bump sphinx from 5.3.0 to 6.1.3 by @dependabot in #605
• Bump httpx from 0.23.0 to 0.23.3 by @dependabot in #607
• Dynamic Summaries Sentinel API and DynamicSummary class. by @ianhelle in #593
• Update sentinel_analytics.py list_alert_rules API version. by @pensivepaddle in #592

Full Changelog: v2.2.0...v2.3.0

IoC Defanging, ServiceNow, GCC support for MDE, Python 3.11

Highlights

Re-architected context and TI providers

The biggest feature of this release is not directly visible but has involved a huge amount of work by @FlorianBracq.
Florian spotted that our HTTP TI provider (used for several TI services such as VirusTotal, OTX, XForce) could be used more generically, specifically for non-TI sources that provided valuable context, such as ServiceNow. So, he re-worked the TI providers sub-package to pull out generic context provider capabilities used by both TI and non-TI sources.
The immediate benefit of this is the next highlight

ServiceNow context provider

This is yet to be full documented but if you have a ServiceNow instance and want to hook up MSTICPy to query it try the following.

1. Add your ServiceNow configuration to msticpyconfig.yaml

ContextProviders:
  ServiceNow:
    Primary: True
    Args:
      TenantId: 8360dd21-0294-4240-9128-89611f415c53
      AuthKey: "authkey"
      AuthId: "authid"
    Provider: "ServiceNow"

Note: you can store the secrets in KeyVault in the same way as TI and other Providers - see the Key Vault Secrets section of MSTICPy Settings Editor

Import and instantiate a ContextProvider and look things up

from msticpy.context.contextlookup import ContextLookup

context_lookup = ContextLookup()
result = context_lookup.lookup_observable("10.0.0.1", providers=["ServiceNow"])
result2 = context_lookup.lookup_observable("user@some.dom", providers=["ServiceNow"])

Defanging support for IoCExtract and TI Providers

In threat reports, IoCs are often de-fanged to make IP addresses, URLs, etc, not clickable. An example
de-fanged IP address would look something like this 17[.]34[.]21[.]195

Previously these would not be matched by the IoCExtract patterns due to the "escaped" dots.
IoCExtract now supports common de-fanged markup such as

• "[.]" to escape dots in IP addresses and domains,
• "@" replaced by "AT"
• "http(s)" and "(s)ftp(s)" replaced by "hXXp(s)" and "(s)fXp(s)" respectively.

We have also added support for email address patterns to IoCExtract.

TI providers will also accept de-fanged IoCs, removing the de-fanging before submitting them to the provider for lookup.

We've also supplied a couple of utility functions defang_ioc and refang_ioc in msticpy.common.utility. These are not yet added as Pivot functions to IpAddress, Url, Dns, Account but will be added in a future release.

Added GCC support to MDE/M365 data providers

This allows customers working with government clouds to query the correct Defender endpoints.

Python 3.11 officially supported

Although there wasn't anything in our code that was a Py 3.11 blocker, some of our dependencies took
a little while to publish 3.11-compatible wheels. That was all done with SciPy, Statsmodels and ScikitLearn
and our build pipeline now in includes a full test pass on Python 3.11. Many thanks to @tonybaloney for
pushing us through this.

What's Changed

• Add base for Context Providers by @FlorianBracq in #511
• Adding skip and warning to test_vt_pivot.py by @ianhelle in #560
• Improved bug template getting rid of irrelevant sections by @ianhelle in #559
• Intsights endpoint update. by @FlorianBracq in #526
• Added support for GCC and Regional Clouds to MDE driver by @petebryan in #525
• Resourcegraph - Incomplete list returned by @pensivepaddle in #496
• Bump sphinx-rtd-theme from 1.0.0 to 1.1.0 by @dependabot in #553
• Sumologic driver: custom dtypes options+fix, add paging, remove days duration int casting by @juju4 in #481
• New mypy failures in kql_base, elastic_driver, splunk_driver, sumolog… by @ianhelle in #564
• Bump sphinx-rtd-theme from 1.1.0 to 1.1.1 by @dependabot in #563
• Add 3.11 to test matrix by @tonybaloney in #546
• Update dnspython requirement from <=2.0.0 to <3.0.0 by @dependabot in #289
• Inability to fetch "all" incidents, only 50 by @pensivepaddle in #565
• Add de-fanging support for iocextract and TI providers by @ianhelle in #536
• Implementing isort for context classes, adding missing docs by @ianhelle in #567
• Add support for context provider Service Now by @FlorianBracq in #556
• Added Sentinel TI integration features. by @petebryan in #532
• Ianhelle/pygeohash and exceptions 2022 11 11 by @ianhelle in #566
• Removing debug prints and duplicate code. by @petebryan in #570
• Moving ASN http lookup to execute at runtime, when whois lookup happens. by @ianhelle in #568
• Added a new set of Sentinel queries related to network activity using the CommonSecurityLog data source. by @petebryan in #524
• Fixed issues with dataprovider instances by @ianhelle in #549
• Adding AzureAuthentication.rst by @ianhelle in #578

Full Changelog: v2.1.5...v2.2.0

Bokeh, ipywidgets version restrictions

The main driver for this release is to restrict versions of bokeh, ipywidgets and pandas.

• Version 3.0.0 of bokeh plots has some breaking changes that prevent it working with MSTICPy
• Version 8.0.0 of ipywidgets has changes that prevent some of the MSTICPy compound widgets displaying correctly.

We also decided to start restricting versions of some of our other dependencies to the current major version - to prevent unexpected breaking changes stopping MSTICPy from working. We have included pandas in this list and will expand it to cover more packages in future. We will combine this with an automated build job that has no version restrictions so that we're aware of version changes that we need to address. The intent here is to have MSTICPy have as broad a version range as possible for its dependencies while still avoiding failures due to breaking changes.

Another small but important change is an update to the Process Tree viewer to allow process GUIDs as process IDs (rather than just hex or decimal format integers). Thanks to @nbareil for this change!

What's Changed

• process_tree: Accept GUID format for ProcessID and ParentProcessID by @nbareil in #542
• Bump sphinx from 5.1.1 to 5.3.0 by @dependabot in #540
• Bump readthedocs-sphinx-ext from 2.1.9 to 2.2.0 by @dependabot in #545
• Update AzureBlobStorage.rst by @garybushey in #539
• Adding upper version restrictions to bokeh, pandas and ipywidgets deps by @ianhelle in #552

New Contributors

• @garybushey made their first contribution in #539

Full Changelog: v2.1.4...v2.1.5

Fixes for MS Sentinel API and configuration

Some minor fixes and improvements:

• MicrosoftSentinel class now defaults to "Default" workspace or workspace name supplied as workspace parameter
  when connecting.

sentinel = MicrosoftSentinel()
sentinel.connect()  # connect to "Default" workspace
sentinel.connect(workspace="MyWorkspace")   # connect to named workspace

• Sentinel create_* APIs now return ID of new item (incident, bookmark, analytic, watchlist)
• init_notebook - now accepts config parameter to use custom msticpyconfig.yaml for notebook session (overrides enviromnent variable and other defaults

import msticpy as mp
mp.init_notebook(config="~/configs/all_ti_provs.yaml")   # use a custom msticpy config file.

• Sentinel configuration editor no longer throws an exception if named control not found
• Sentinel TI provider will not attempt lookups if ThreatIntelligenceIndicator table not found in the Sentinel data provider schema
• Support for Kusto/Azure Data explorer settings in Settings editor
• Added checked_kwargs decorator to utility/types.py

What's Changed

• Ianhelle/training hotfixes 2022 10 13 by @ianhelle in #543
• Updated ReadMe with Blackhat Arsenal Tag by @petebryan in #521

Full Changelog: v2.1.3...v2.1.4

Process Tree Viewer updates

Highlights

This is a minor release with some fixes and additions that enable broader functionality.
The biggest-impacting changes apply to the
Process Tree visualization.
These changes allow it to work with broader types of Windows or Linux process data:

• Removed the following columns that were previously required: host_name, logon_id, user_name, cmd_line.
• Added auto-coloring by level if no legend is supplied.
• Fixed process sorting so that tree and peer groups in the tree are sorted by level, then timestamp.
• Added ability to supply schema as dictionary to the process tree APIs.

The changes are described in more detail below.

We've also added support for a new MS Sentinel API to retrieve queries stored in a Sentinel workspace
and fixed some issues in IP WhoIs lookups.

Process Tree changes

Reduced required column set

This allows you to use the process tree visualization and utilities with a minimal set of data fields:

• process_id
• parent_id
• process_name
• time_stamp

    cust_schema = {
        "process_name": "ImageFileName",
        "process_id": "PID",
        "parent_id": "PPID",
        "time_stamp": "CreateTime",
    }
    df.mp_plot_process(schema=cust_schema)

Auto-coloring of tree plot

If you do not supply a legend_col parameter, the process objects will be
automatically colored by level in the hierarchy. This makes a basic tree more colorful and easier to navigate.

Processes are correctly sorted by process time

Previously, the code that builds the process tree left individual processes in an unintuitive order.
For a given level (e.g. parents) all of the processes will be displayed in time created order.

For example:

A \
   - A.1
   - A.2
B \
   - B.1
   - B.2

A will always have a timestamp less than or equal to B. All children of A (A.1, A.2...) and B will be shown in
time created order. However, across different levels and peer groups, there is no guarantee of time-ordering. In the above example, even though timestamp A is less than timestamp B, B.1 and B.2 could have timestamps earlier than either A.1 or A.2.

	path	ImageFileName	CreateTime
proc_key
registry|88|2021-04-01 05:04:54.000000	116/0	Registry	2021-04-01 05:04:54+00:00
system|4|2021-04-01 05:04:58.000000	117/1	System	2021-04-01 05:04:58+00:00
smss.exe|404|2021-04-01 05:04:58.000000	117/1/2	smss.exe	2021-04-01 05:04:58+00:00
csrss.exe|640|2021-04-01 05:05:00.000000	118/3	csrss.exe	2021-04-01 05:05:00+00:00
winlogon.exe|700|2021-04-01 05:05:00.000000	118/4	winlogon.exe	2021-04-01 05:05:00+00:00
dwm.exe|1028|2021-04-01 05:05:02.000000	118/4/17	dwm.exe	2021-04-01 05:05:02+00:00
logonui.exe|512|2021-04-01 05:05:02.000000	118/4/21	LogonUI.exe	2021-04-01 05:05:02+00:00
fontdrvhost.ex|960|2021-04-01 05:05:01.000000	118/4/7	fontdrvhost.ex	2021-04-01 05:05:01+00:00
wininit.exe|632|2021-04-01 05:05:00.000000	119/5	wininit.exe	2021-04-01 05:05:00+00:00
lsass.exe|776|2021-04-01 05:05:01.000000	119/5/10	lsass.exe	2021-04-01 05:05:01+00:00

mp_plot.process_tree and mp.build_process_tree support schema as dictionary

Previously these accessors and the underlying functions plot_process_tree and
build_process_tree would only accept msticpy.transform.process_tree_schema.ProcSchema
instances. These will now accept dictionaries with at least the minimum required
attributes as keys.

What's Changed

• Sentinel - Return all saved queries by @petebryan in #519
• Bump readthedocs-sphinx-ext from 2.1.8 to 2.1.9 by @dependabot in #507
• Bump respx from 0.19.2 to 0.20.0 by @dependabot in #512
• Allow process tree to work with more data sources. by @ianhelle in #513
• Fixed error in cell using non-existing column name by @ianhelle in #527
• Ianhelle/proc tree fixes 2022 09 16 by @ianhelle in #530
• Fixed issue with whois lookups on only local IPs by @petebryan in #506

Full Changelog: v2.1.2...v2.1.3