Skip to content

Releases: microsoft/msticpy

Hotfix - Azure authentication failure

14 Sep 19:54
@ianhelle ianhelle
v2.1.2
19e6431
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
Hotfix - Azure authentication failure

A last-minute change before release of 2.1.0 introduced a critical bug in azure_auth_core.py.
This caused all azure authentication to fail. It would also cause init_notebook() to fail if the user had any Key Vault secrets referenced in their msticpyconfig.yaml.

Thanks to @FlorianBracq for spotting this independently (and before us) and submitting a PR with the fix.
The PR below is essentially the same fix as Florian's with a subtle change to allow an EnvironmentCredential of None to appear in the list of creds sent to ChainedTokenCredential. This is to cover an edge case where EnvironmentCredential is requested but the required environment variables are not set.

What's Changed

  • [fix] bug in call to ChainTokenCredential breaks all authentication by @ianhelle in #505
  • Rolling back change on _build_chained_creds by @FlorianBracq in #504

Full Changelog: v2.1.1...v2.1.2

Contributors

ianhelle and FlorianBracq
Loading

Hotfix - missing beautifulsoup4 from requirements/dependencies

14 Sep 19:46
@ianhelle ianhelle
v2.1.1
cff0279
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
Hotfix - missing beautifulsoup4 from requirements/dependencies

We inadvertently took a hard dependency on beautifulsoup4 but didn't have it in our dependencies.
Unfortunately, since bs4 is in our test dependencies this passed all the tests so didn't spot it until later.

What's Changed

Full Changelog: v2.1.0...v2.1.1

Contributors

ianhelle
Loading

IpWhois, Malware Bazaar, Azure Auth, Azure Synapse

13 Sep 19:54
@ianhelle ianhelle
v2.1.0
98ffa22
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
IpWhois, Malware Bazaar, Azure Auth, Azure Synapse

Highlights

Replaced dependency on IPWhois with local code #479

The ipwhois package seems to be abandoned and was causing conflicts with dnspython. We've
created equivalent functionality in msticpy removing build warnings and (minutely) speeding
up install time. We've also added a MSTICPy pandas accessor df.mp.whois() so that you can
do bulk queries from a dataframe.

Malware Bazaar TI Provider #459

Many thanks to @fr0gger for this.
Check out the notebook MBLookup to
see how you can use this new provider.

Documentation on how to build a Data Provider #465

This was previously a blog post but we've added it to the official docs - Writing and Contributing a Data Provider

Updates to Azure authentication to support more authentication types #484

We've switched from using DefaultAzureCredential to supporting the native credential types.
This lets us support additional credential types such as Client Secret and Certification authentication.
You can also create your own custom AzureCredential and pass this to az_connect.

Updates to SQL2Kql converter #488

This was really prompted by @tonybaloney in helping us get a build working on Python 3.8-3.11. This
module had a dependency on a now-deprecated moz_sql_parser. We've updated to use mo_sql_parsing - many thanks to @klahnakoski for work on keeping this alive and well.
Our module also contains some fixes and enhancements from the original.

Builds and tests now running on Python 3.8, 3.9, 3.10 #476

We were previously only building on Python 3.8. Huge thanks to @tonybaloney for working on this and bringing us
into the modern era. We still have some issues with Python 3.11dev - although this is due to SciPy breaking with
the last 3.11 version we tried. As soon as this is sorted we will add 3.11 back.

Added support for msticpy notebooks in Azure Synapse pipelines #493

This is mostly work done to support MS Sentinel running unattended notebooks in Synapse pipelines.
We've extended the mp.init_notebook() function so that it can correctly configure msticpy (looking for
msticpyconfig.yaml on a mounted blob storage container and persisting cached data there),
use the linked Key Vault to store secrets and supply service principal credentials to msticpy.

Important fixes

  • Allow for missing columns in Folium map data frame #489
  • Updated M365D/MDE driver to pass query request with JSON encoding #498 Defender have always supported this
    but we were sending a JSON string, which they recently stopped supporting. This should be working again.
  • You can now see data query help before connecting to the data provider. It's also possible
    to dry run the query to see the full query with parameters replaced without needing to connect. #482

What's Changed

New Contributors

Full Changelog: v2.0.0...v2.1.0

Contributors

juju4, tonybaloney, and 7 other contributors
Loading

MSTICPy Version 2.0

07 Jul 18:23
@ianhelle ianhelle
v2.0.0
95a1d3c
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
MSTICPy Version 2.0

MSTICPy Release 2.0

A notebook containing some of the features of MSTICPy 2.0
is available at What's new in MSTICPy 2.0

If you are new to MSTICPy or just want to catch up and get a quick
overview check out our new MSTICPy Quickstart Guide.

Contents

  • Dropping Python 3.6 support
  • Package re-organization and module search
  • Simplifying imports in MSTICPy
  • Folium map update - single function, layers, custom icons
  • Threat Intelligence providers - async support
  • Time Series simplified - analysis and plotting
  • DataFrame to graph/network visualization
  • Pivots - easy initialization/dynamic data pivots
  • Consolidating Pandas accessors
  • MS Sentinel workspace configuration
  • MS Defender queries available in MS Sentinel QueryProvider
  • Microsoft Sentinel QueryProvider
  • New queries
  • Documentation Additions and Improvements
  • Miscellaneous improvements
  • Previous feature changes since MSTICPy 1.0

 

Dropping Python 3.6 support

As of this release we only officially support Python 3.8 and above.
We will try to support Python 3.6 if the fixes required are small
and contained but make no guarantees of it working completely on
Python prior to 3.8.

 

Package re-organization and module search

One of our main goals for V2.0.0 was to re-organize MSTICPy to be more logical and easier to
use and maintain. Several years of organic growth had seen modules created in places that
seemed like a good idea at the time but did not age well.

The discussion about the V2 structure can be found here #320.

Due to the re-organization, many features are no longer in places
where they used to be imported from!

We have tried to maintain compatibility with old locations by adding "glue" modules.
These allow import of many modules from their previous locations but will issue a
Deprecation warning if loaded from the old location.
The warning will contain the new location of the module -
so you should update your code to point to this new location.

This table gives a quick overview of the V2.0 structure

folder description
analysis data analysis functions - timeseries, anomalies, clustering
auth authentication and secrets management
common common used utilities and definitions (e.g. exceptions)
config configuration and settings UI
context enrichment modules geoip, ip_utils, domaintools, tiproviders, vtlookup
data data acquisition/queries/storage/uploaders
datamodel entities, soc objects
init package loading and initialization - nbinit, pivot modules
nbwidgets nb widgets modules
transform simple data processing - decoding, reformatting, schema change, process tree
vis visualization modules including browsers

Notable things that have moved:

  • most things from the sectools folder have migrated to context, transform or analysis
  • most things from the nbtools folder have migrated to:
    • msticpy.init - (not to be confused with __init__) - package initialization
    • msticpy.vis - visualization modules
  • pivot functionality has moved to msticpy.init

Module Search

If you are having trouble finding a module, we have added a simple search function:

    import msticpy as mp
    mp.search("riskiq")

Matches will be returned in a table with links to the module documentation

Modules matching 'riskiq'

ModuleHelp
msticpy.context.tiproviders.riskiq msticpy.context.tiproviders.riskiq

 

Simplifying imports in MSTICPy

The root module in MSTICPy now has several modules and
classes that can be directly accessed from it (rather than
having to import them individually).

We've also decided to adopt a new "house style" of importing
msticpy as the alias mp. Slavishly copying the idea from
some of admired packages that we use (pandas -> pd,
numpy -> np, networkx -> nx) we thought it would save
a bit of typing. You are free to adopt or ignore this style -
it obviously has no impact on the functionality.

    import msticpy as mp
    mp.init_notebook()
    qry_prov = mp.QueryProvider("MDE")
    ti = mp.TILookup()

Many commonly-used classes and functions are exposed as
attributes of msticpy (or mp).

Also a number of commonly-used classes are imported by default
by init_notebook, notably all of the entity classes.
This makes it easier to use pivot functions without any initialization
or import steps.

    import msticpy as mp
    mp.init_notebook()

    # IpAddress can be used without having to import it.
    IpAddress.whois("123.45.6.78")

init_notebook improvements

  • You no longer need to supply the namespace=globals() parameter when
    calling from a notebook. init_notebook will automatically obtain the
    notebook global namespace and populate imports into it.
  • The default verbosity of init_notebook is now 0, which produces
    minimal output - use verbosity=1 or verbosity=2 to get more
    detailed reporting.
  • The Pivot subsystem is automatically initialized in init_notebook.
  • All MSTICPy entities are imported automatically.
  • All MSTICPy magics are initialized here.
  • Most MSTICPy pandas accessors are initialized here (some, which
    require optional packages, such as the timeseries accessors are
    not initialized by default).
  • init_notebook supports a config parameter - you can use this to
    provide a custom path to a msticpyconfig.yaml overriding the usual
    defaults.
  • searching for a config.json file is only enabled if you are running
    MSTICPy in Azure Machine Learning.

 

Folium map update - single function, layers, custom icons

The Folium module in MSTICPy has always been a bit complex to use
since it normally required that you convert IP addresses to MSTICPy
IpAddress entities before adding them to the map.

You can now
plot maps with a single function call from a DataFrame containing
IP addresses or location coordinates. You can group the data
into folium layers, specify columns to populate popups and tooltips
and to customize the icons and coloring.

folium_layers

plot_map

A new plot_map function (in the msticpy.vis.foliummap module) that
lets you plot mapping points directly from a DataFrame. You can
specify either an ip_column or coordinates columns (lat_column and
long_column). In the former case, the geo location of the IP address
is looked up using the MaxMind GeoLiteLookup data.

You can also control the icons used for each marker with the
icon_column parameters. If you happen to have a column in your
data that contains names of FontAwesome or GlyphIcons icons
you can use that column directly.
More typically, you would combine the icon_column with the
icon_map parameter. You can specify either a dictionary or a
function. For a dictionary, the value of the row in icon_column
is used as a key - the value is a dictionary of icon parameters
passed to the Folium.Icon class. For a function, the icon_column
value is passed to the function as a single parameter and the return value
should be a dictionary of valid parameters for the Icon class.
You can read the documentation for this function in the
docs

plot_map pandas accessor

Plot maps from the comfort of your own DataFrame!
Using the msticpy mp_plot accessor you can plot maps directly
from a DataFrame containing IP or location information.
The folium_map function has the same syntax as plot_map
except that you omit the data parameter.

    df.mp_plot.folium_map(ip_column="ip", layer_column="CountryName")

pd_accessors

Layering, Tooltips and Clustering support

In plot_map and .mp_plot.folium_map you can specify
a layer_column parameter. This will group the data
by the values in that column and create an
individually selectable/displayable layer in Folium. For performance
and sanity reasons this should be a column with a relatively
small number of discrete values.

Clustering of markers in the same layer is also implemented by
default - this will collapse multiple closely located markers
into a cluster that you can expand by clicking or zooming.

You can also populate tooltips and popups with values
from one or more column names.

"Classic" interface

The original FoliumMap class is still there for more manual
control. This has also been
enhanced to support direct plotting from IP, coordinates or GeoHash
in addition to the existin...

Read more

Contributors

aeetos, juju4, and 6 other contributors
Loading
0 Join discussion

MSTICPy 2.0 - Pre-release 3

21 Jun 01:25
@ianhelle ianhelle
v2.0.0.rc3
6eed3ff
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
MSTICPy 2.0 - Pre-release 3 Pre-release
Pre-release

New Features

A notebook containing some of the features of MSTICPy 2.0
is available at What's new in MSTICPy 2.0

Dropping Python 3.6 Support

As of this release we only officially support Python 3.8 and above.
We will try to support Python 3.6 if the fixes required are small
and contained but make no guarantees of it working completely on
Python prior to 3.8

DataFrame to Graph/Network

You can convert a pandas DataFrame into a NetworkX graph or
plot directly as a graph using Bokeh interactive plotting.

You pass the functions the column names for the source and target nodes to build a basic graph. You can also name other columns to be node or edge attributes. When displayed these attributes are visible as popup details courtesy of Bokeh’s Hover tool.

    proc_df.head(100).mp_plot.network(
        source_col="SubjectUserName",
        target_col="Process",
        source_attrs=["SubjectDomainName", "SubjectLogonId"],
        target_attrs=["NewProcessName", "ParentProcessName", "CommandLine"],
        edge_attrs=["TimeGenerated"],
    )

Graph plot

Pivots without initialization/dynamic data query import

The pivot functionality has been overhauled - it is now initialized
automatically in init_notebook.

Previously queries from
data providers were added at initialization - meaning that you had
to create your query providers before starting pivot or re-initialize
pivot. Data providers now dynamically add relevant queries as pivot
functions when you authenticate. Also for some providers, such
as Azure Sentinel, that support multiple instances, pivot now
supports separate instance naming so that each Workspace has a
separate instance of a given pivot query.

The naming of the Threat Intelligence pivot functions has been
simplified considerably.

VirusTotal and RiskIQ relationships should now be available as
pivot functions (you need the VT 3 and PassiveTotal packages installed
respectively to enable this functionality).

Simplify imports in msticpy

The root module in msticpy now has several modules and
classes that can be directly accessed from it (rather than
having to import them)

    import msticpy as mp
    mp.init_notebook()
    qry_prov = mp.QueryProvider("MDE")
    ti = mp.TILookup()

Also a number of commonly-used classes are imported by default
by init_notebook, notable all of the entity classes.
This makes it easier to use pivot functions without any initialization
or import steps.

  • entities
    import msticpy as mp
    mp.init_notebook()

    IpAddress.whois("123.45.6.78")

Consolidation of Pandas accessors

Pandas accessors are extensions to DataFrames allowing you to
call custom functionality as a DataFrame method.

Almost all of the core MSTICPy functions previously available in
various accessors (plus a few new ones) are accessible in:

  • df.mp - analysis and transform functions
  • df.mp_plot - visualization functions
    df.mp.ioc_extract(...)
    df.mp.to_graph(...)
    df.mp.mask(...)

    df.mp_plot.timeline(...)
    df.mp_plot.timeline_values(...)
    df.mp_plot.process_tree(...)
    df.mp_plot.network(...)
    df.mp_plot.folium_map(...)

MS Defender Queries available to MS Sentinel Query Provider

Since Sentinel now has the ability to import Microsoft data, we've
made the Defender queries usable from the MS Sentinel provider.

Many of these queries are now available as Pivot functions.

ContiLeaks notebook added to MSTICPy Repo

We are privileged to host Thomas's awesome ContiLeaks notebook.
Thanks @fr0gger

New Queries added

Several new Sentinel and MS Defender queries have been added.

See the new built-in query list

Documentation Additions and Updates

The documentation for V2.0 is now live and available at https://msticpy.readthedocs.io
(Previous versions are still online and can be accessed through
the ReadTheDocs interface).

The API documentation has been split into separate modules to
make it easier to navigate. The API docs also now support "InterSphinx".
This means that MSTICPy references to objects in other packages (e.g. Python
standard library, pandas, Bokeh) have active links that will take you
to the native documentation for that item.

Also, the sample notebooks for most of these features have been updated
along the same lines. See MSTICPy Sample notebooks

Miscellaneous Improvements

  • The MS Sentinel provider now support a timeout parameter allowing you
    lengthen and shorten the default.
  • MSTICPy network requests use a custom User Agent header so that you
    can identify or track requests from MSTICPy/Notebooks.

Plus a lot more that I can't recall at the moment.

What's Changed - The gory detail of the PRs

Read more

Contributors

fr0gger, petebryan, and 3 other contributors
Loading

MSTICPy 2.0.0 pre-release 2

31 May 02:22
@ianhelle ianhelle
v2.0.0.rc2
fbf9915

Choose a tag to compare

View all tags
MSTICPy 2.0.0 pre-release 2 Pre-release
Pre-release

New Features

There are several new features in V 2.0.0 of MSTICPy. The major
items include:

  • Folium map update - plot a map using multiple layers, custom
    icons, colors and tooltips from a single function call.
  • Time Series - calculate and display a Time Series anomalies
    plot from a single function call.
  • Threat Intelligence lookups - individual providers run asynchronously
    (simultaneously) making it many times faster to perform lookups
    across providers. Lookup progress is also displayed with a progress
    bar

Pre-release documentation for v2.0.0 is on ReadtheDocs
Note: API documentation should be up-to-date but user-guides for new features
are still TBD.

Folium map update

The Folium module in MSTICPy has always been a bit complex to use
since it normally required that you convert IP addresses to MSTICPy
IpAddress entities before adding them to the map. You can now
plot maps with a single function call from a DataFrame containing
IP addresses or location coordinates. You can group the data
into folium layers, specify columns to populate popups and tooltips
and to customize the icons and coloring.

folium_layers

plot_map

A new plot_map function (in the msticpy.vis.foliummap module) that
lets you plot mapping points directly from a DataFrame. You can
specify either an ip_column or coordinates columns (lat_column and
long_column). In the former case, the geo location of the IP address
is looked up using the MaxMind GeoLiteLookup data.

You can also control the icons used for each marker with the
icon_column parameters. If you happen to have a column in your
data that contains names of FontAwesome or GlyphIcons icons
you can use that column directly.
More typically you would combine the icon_column with the
icon_map parameter. You can specify either a dictionary or a
function. For a dictionary, the value of the row in icon_column
is used as a key - the value is a dictionary of icon parameters
passed to the Folium.Icon class. For a function, the icon_column
value is passed to the function as a single parameter and the return value
should be a dictionary of valid parameters for the Icon class.
You can read the documentation for this function in the
docs

plot_map pandas accessor

Plot maps from the comfort of your own DataFrame!
Using the msticpy mp_plot accessor you can plot maps directly
from a DataFrame containing IP or location information.
The folium_map function has the same syntax as plot_map
except that you omit the data parameter.

    df.mp_plot.folium_map(ip_column="ip", layer_column="CountryName")

pd_accessors

Layering, Tooltips and Clustering support

In plot_map and .mp_plot.folium_map you can specify
a layer_column parameter. This will group the data
by the values in that column and create an
individually selectable/displayable layer in Folium. For performance
and sanity reasons this should be a column with a relatively
small number of discrete values.

Clustering of markers in the same layer is also implemented by
default - this will collapse multiple closely located markers
into a cluster that you can expand by clicking or zooming.

You can also populate tooltips and popups with values
from one or more column names.

Classic interface

The original FoliumMap class is still there for more manual
control. This has also been
enhanced to support direct plotting from IP, coordiates or GeoHash
in addition to the existing IpAddress and GeoLocation entities.
It also supports layering and clustering.

Threat Intelligence Providers - Async support

When you have configured more than one TI provider, MSTICPy will
execute requests to each of them asynchronously. This will bring big
performance benefits when querying IoCs from multiple providers.
Note: requests to individual providers are still executed synchronously
since we want to avoid swamping provider services with multiple
simultaneous requests.

We've also implemented progress bar tracking for TILookups, giving a visual
indication of progress when querying multiple IoCs.

Combining the progress tracking with asynchronous operation means
that not only is performing lookups for lots of observables faster
but ou will also less likely to be left guessing whether or not your kernel
has hung.

TI Providers are now also loaded on demand - i.e. only when you have
a configuration entry in your msticpyconfig.yaml for that provider.
This prevents loading of code (and possibly import errors) due to providers
which you are not intending to use.

Finally, we've added functions to enable and disable providers
after loading TILookup:

    from msticpy.context import TILookup
    ti_lookup = TILookup()

    iocs = ['162.244.80.235', '185.141.63.120', '82.118.21.1', '85.93.88.165']
    ti_lookup.lookup_iocs(iocs, providers=["OTX", "RiskIQ"])

ti_providers_async

Time Series pandas accessor

Although the Time Series functionality was relatively simple to
use, it previously required several disconnected steps to compute
the time series, plot the data, extract the anomaly periods. Each of
these needed a separate function import. Now you can do all of these
from a DataFrame via pandas accessors.
(currently there is a separate accessor df.mp_timeseries but we are
still working on consolidating our pandas accessors so this may change
before the final release.)

Because you typically still need these separate outputs, the accessor
has multiple methods:

  • df.mp_timeseries.analyze - takes a time-summarized DataFrame
    and returns the results of a time-series decomposition
  • df.mp_timeseries.plot - takes a decomposed time-series and
    plots the anomalies
  • df.mp_timeseries.anomaly_periods - extracts anomaly periods
    as a list of time ranges
  • df.mp_timeseries.anomaly_periods - extracts anomaly periods
    as a list of KQL query clauses
  • df.mp_timeseries.apply_threshold - applies a new anomaly
    threshold score and returns the results.

See documentation

Analyze data to produce time series.

    df = qry_prov.get_networkbytes_per_hour(...)
    ts_data = df.mp_timeseries.analyze()

Analyze and plot time series anomalies

    df = qry_prov.get_networkbytes_per_hour(...)
    ts_data = df.mp_timeseries.analyze().mp_timeseries.plot()

Analyze and retrieve anomaly time ranges

    df = qry_prov.get_networkbytes_per_hour(...)
    ts_data = df.mp_timeseries.analyze().mp_timeseries.anomaly_periods()

In next pre-release

Plot networks (graphs) directly from a DataFrame

One frequently-requested feature is the ability to easily plot
networks from data. For example you may want to view the interactions
between account names and IP addresses. This feature use
Networkx to build the graph and
Bokeh to plot the graph.

Note: The graph has the usual Bokeh interactivity - zoomin, panning, selecting,
hover-over tooltips. It does not allow you to move individual
nodes and interactively recalculate the layout. For the
latter, you can use this functionality to build a networkx graph
and plot using something like GraphViz or PyViz.

The network plot will give you two functions:

  • df.mp.to_graph to convert a DataFrame to a networkx graph
  • df.mp_plot.network create and plot the graph in a single step.

(There is also a separate function msticpy.vis.network_plot.plot_nx_graph
that will just do the NX -> plot operation)

You can specify the columns to use as source and target. An edge
is created between source and target when the two occur on
in the same row (or more than one row). You can also
specify columns to use as node and edge attributes.

To Do items

We intend to add the following before release:

  • allow you to specify the networkx layout algorithm to use
    (currently it uses the default spring_layout)
  • assign edge weight attribute based on number of rows contributing
    to an edge

MS Sentinel Workspaces API

Lets you query and resolve details for Sentinel workspaces.
This is integrated into the MpConfigEdit and MpConfigFile utilities
to let you lookup workspace details when you are editing your
settings:

  • paste in a URL from the Sentinel Azure portal to populate workspace settings
  • or resolve full details from partial workspace such as the workspace ID.

Other important fixes

The API details for most of the MSTICPy functions were not being
generated - this should now be fixed.

What's Changed (GitHub PR Summary)

Read more

Contributors

petebryan and ianhelle
Loading

Fixes for Linux auth, kql and nbinit initialization

23 May 17:26
@ianhelle ianhelle
v1.8.2
4ee68f0
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
Fixes for Linux auth, kql and nbinit initialization

Minor release fixing a few usability issues.

What's Changed

  • Adding full Delegated Auth support to all OData Drivers by @petebryan in #409
    This allows MDE and Graph users to use User-delegated authentication rather than app ID/secrets
  • Fixes for usability bugs in kql_driver, nbinit, user_config - added typing-extensions requirement by @ianhelle in #411
    • Kql driver will revert to Kqlmagic-based device authentication if Azure Authentication fails
    • Kql driver suppresses "missing PyGObject message" - a dependency that isn't required in this scenario
    • init_notebook produced spurious error message about Virus Total libraries not being available even if they were not used.
    • User config throws error if user has partial auto-load configuration in msticpy
  • Replace MSAL auth plaintext file cache with memory cache by @petebryan in #413
    • removed ability to use plaintext token cache because of security concerns
  • Update API version for list_alert_rules by @FlorianBracq in #399
  • Updating Dockerfile source to mcr anaconda by @ianhelle in #397
    Docker source switched to trusted anaconda source for supply chain security
  • Updated Tweet bot to include more context in the tweets by @petebryan in #403
  • Updated tweet action to include more detail in the tweets by @petebryan in #405
  • Adding Microsoft SECURITY.MD by @microsoft-github-policy-service in #407
  • Bump readthedocs-sphinx-ext from 2.1.5 to 2.1.6 by @dependabot in #400

Full Changelog: v1.8.1...v1.8.2

Contributors

petebryan, ianhelle, and 2 other contributors
Loading

MSTICPy V2.0.0 Pre-Release 1

10 May 19:44
@ianhelle ianhelle
v2.0.0.rc1
857835d
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
MSTICPy V2.0.0 Pre-Release 1 Pre-release
Pre-release

One of our main goals for V2.0.0 was to re-organize MSTICPy to be more logical and easier to
use and maintain. Several years of organic growth had seen modules created in places that
seemed like a good idea at the time but did not age well.

The discussion about the V2 structure can be found here #320.

This first pre-release of V2.0 has a small number of new features but is mostly
about testing compatibility. We'll be adding features to future pre-releases in
the next week or two.

**Due to the re-organization, many features are no longer in places where they used
to be imported from! **

We have tried to maintain compatibility with old locations by adding "glue" modules.
These allow import of many modules from their previous locations but will issue a Deprecation warning if loaded from here.
The warning will contain the new location of the module - so you should update your code to point to this new location.

This table gives a quick overview of the V2.0 structure

folder description
analysis data analysis functions - timeseries, anomalies, clustering
auth authentication and secrets management
common common used utilities and definitions (e.g. exceptions)
config configuration and settings UI
context enrichment modules geoip, ip_utils, domaintools, tiproviders, vtlookup
data data acquisition/queries/storage/uploaders
datamodel entities, soc objects
init package loading and initialization - nbinit, pivot modules
nbwidgets nb widgets modules
transform simple data processing - decoding, reformatting, schema change, process tree
vis visualization modules including browsers

Notable things that have moved:

  • most things from the sectools folder have migrated to context, transform or analysis
  • most things from the nbtools folder have migrated to:
    • init - (not to be confused with init) - package initialization
    • vis - visualization modules
  • pivot functionality has moved to init

If you are having trouble finding a module, we have added a simple search function

    import msticpy
    msticpy.search("riskiq")

Any matches will be returned in a table with links to the module documentation

Modules matching 'riskiq'

ModuleHelp
msticpy.context.tiproviders.riskiq msticpy.context.tiproviders.riskiq

The Help link is to the current release version of the documentation, so will not
map to the correct path on ReleaseTheDocs for the release documentation. You should be able to
map this to the pre-release doc version by adding v2.0.0.rc1 into the URL.

https://msticpy.readthedocs.io/en/v2.0.0rc1/api/msticpy.sectools.tiproviders.html

Other Features/Changes

Pandas accessors have been consolidated into two main ones:

  • df.mp.func - contains all of the transformation functions like base64 decoding, ioc searching, etc.
  • df.mp_plot.func - contains all of the visualization accessors (timeline, process tree, etc.)

We've also done a lot of work to clean up warnings, move some remaining Python unittest tests to pytest
format and random other things that you do when you're cleaning house. A fuller but
not especially readable list of changes follows.

Please let us know if you hit any issues with the re-jigged structure
or any other bugs. File them as issues or reach out to msticpy@microsoft.com

List of changes

Co-authored-by: Ian Hellen ianhelle@microsoft.com
Co-authored-by: Pete Bryan pebryan@microsoft.com

  • 857835d@Ianhelle/mpconfigedit fix from main 2022 05 22 (#396)* Fix for MpConfigEdit ValueError

  • Updating Dockerfile source to mcr anaconda

  • bd96d40@Ianhelle/v2 reorg directories 2 2022 04 12 (#377)* Merging changes from main for geoip.py, mp_config_edit, mp_config_file, pkg_config and kusto_driver

  • Some fixes to Kusto common_imports

    • now works with Kusto config entry without instance suffix
    • can now supply cluster ALIAS (instance name) instead of actual cluster name in connect or query
    • added explicit "database" key in query files - can be used instead of the more opaque "data_family.database" encoding
      in the data_families key.

  • Fixed documentation in DataProv-Kusto.rst to correct inaccuracies and update sections on query templates and configuration

  • Fixed bug and simplified/cleaned up code for GeoLiteLookup in geoip.py.

  • Fixed bug in mp_config_edit.py and mp_config_file.py where empty/new msticpyconfig.yaml didn't save any settings.

  • Reorganized logic for handling parameters and failing on invalid file path in config module.

  • Moved analysis.data to transform folder

  • Moved data.context to context

  • Moved auth and secrets modules to auth folder

  • Moved nbwidgets to new folder

  • Updated deprecation warning in glue modules to v2.2

  • Fix to URLs in README.md

  • Adding init folder - moved:

    • nbinit.py, user_config.py, pivot.py, azure_ml_tools.py to here
    • also moved vt_pivot.py and pivot_ti_provider.py to init/pivot_init

  • Renaming datamodel/pivots to datamodel/pivot

  • Moved azure_blob_storage.py to data/storage folder

  • Refactored query_container to data_types.py - to be separate types for queries (query_container.py) and pivots (pivot_container.py)

  • Moved browsers to vis folder

  • Updating API docs for moves

  • Moved all pivot functions to init folder.

  • Added functionality to pkg_config to delete and translate settings (for AzureSentinel->MSSentinel switch TBD)

  • Added automatic acquisition of globals() in nbinit.py

  • Fixed a couple of bugs in pivot_pipeline.py

  • Removed direct import of pivot into datamodel/pivot and added code to add them dynamically after init.pivot initiialization.

  • Added trap to timeline when supplied with no data.

  • Fixed incorrect escaping in regex in kql_driver.py

  • Notebook updates for errors and invalid links.

  • Added script to run all notebooks for testing

  • Changing the pattern for httpx timeout to default to Timeout(None). This can be overridden in settings and in the case of drivers and TILookup in runtime parameter (timeout=x). Other components use the default.

  • Updated typing rigor for pkg_config::get_http_timeout

  • Fixed bugs in test test_pkg_config.py and test_code_view.py

  • Fixed re-auth on query issue in KQL driver

  • Fixed kql_driver tests

  • Kusto provider

    • now works with Kusto config entry without instance suffix
    • can now supply cluster ALIAS (instance name) instead of actual cluster name in connect or query
    • added explicit "database" key in query files - can be used instead of the more opaque "data_family.database" encoding
      in the data_famiies key.

  • Fixed documentation in DataProv-Kusto.rst to correct inaccuracies and update sections on query templates and configuration

  • Fixed bug and simplified/cleaned up code for GeoLiteLookup in geoip.py.

  • Fixed bug in mp_config_edit.py and mp_config_file.py where empty/new msticpyconfig.yaml didn't save any settings.

  • Reorganized logic for handling parameters and failing on invalid file path in config module.

  • e2e743a@Ianhelle/main mergeback 2022 04 05 (#355)* Updated nbwidgets - GetText, QueryTime, GetEnvironmentKey to work with notebook parameters.

  • Fixed query_time widget so that you can reset time range from parameter

  • Added additional unit test for QueryTime setter

  • Simplified SelectAlert (in select_alert and nbdisplay) to remove title line. this was not updating so every alert selected would add another titlel line.

  • Changed structure and formatting of alert item display - removing CompromisedEntity and adding ProductName.

  • Updated TimeSpan class so it has more flexible constructor

  • Added account_id as a parameter for list_aad_signins_by_account query

  • Fix for kql_driver - reconnecting for each query loses original kwargs (including mp_az_auth) setting, so reverts
    to defaults. This can cause errors if the defaults are different to user-specified parameters.
    There is also a problem in azure_auth.py - if a user has AzureCLI settings, these override everything. I've removed this since we don't really want people configuring auth methods from these settings.

  • aa14784@moved list_sentinel_workspaces to AzureData

  • f3e82d9@New MSAL delegated auth methods added and support for this added to Graph providers. Added ability to pass tenant ID to KQL provider fixing issue 333. Minor fixes added incl merge from #352.

  • ba33c38@Ianhelle/pivot dataprov selfload 2022 03 15 (#343)* Query providers load pivots dynamically when created.

  • Renamed query paths and changed data_providers so that only env-specific queries are loaded.

  • Moving ensure_df_datetimes to common/data_utils.py to avoid circular imports

  • Consolidated data-related pandas accessors into single module.

  • Fixing circular dependency in iocextract

  • Fixing linting errors in data_providers.py, azure_resource.py, host.py, process.py, pivot_data_queries.py

  • Adding default timeout values to httpx calls.

  • Changing tor_exit_nodes.py Tor provider to defer download of tor list until first lookup

  • Fixing test for trig...

Read more

Contributors

petebryan and ianhelle
Loading

Hotfix release for MpConfigEdit

10 May 18:06
@ianhelle ianhelle
v1.8.1
b0ba1f4
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
Hotfix release for MpConfigEdit

What's Changed

  • MpConfigEdit throws exception when passed file name
  • MpConfigEdit throws error with invalid file path. by @ianhelle in #395

Full Changelog: v1.8.0...v1.8.1

Contributors

ianhelle
Loading

Sentinel Search API, Azure Authentication, Settings management, HTTPX timeouts

04 May 22:17
@ianhelle ianhelle
v1.8.0
5c4c7f6
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
Sentinel Search API, Azure Authentication, Settings management, HTTPX timeouts

Summary

There are some feature changes and fixes in this release:

  • MS Sentinel Search API support in the Sentinel package - allowing you to create, check status and delete automated search jobs.
  • Authentication updates to
    • support wider range of Azure authentication types (incl VSCode and Powershell)
    • specify tenantID at auth time (this was a specific issue for Azure Data explorer users)
    • lets you use MSAL token caching
  • Fixes to httpx timeouts (we recently switch from using requests to httpx and have
    changed the default timeout to be none (like requests). You can also set this in configuration
    and specify while calling a function that makes a network request, e.g. mde_prov.my_query(...params, timeout=30)
  • MpConfigEdit now behaves better when no existing msticpyconfig.yaml exists - making it easier to create a
    config file from scratch.

What's Changed

  • Ianhelle/mp config edit load fix 2022 03 28 by @ianhelle in #352
    Also fixes to multiple widgets for papermill/automation. Can now set QueryTime timespan programmatically.
    Fix to Sentinel data provider to prevent re-auth for every query.
  • Pebryan/2022 3 29 auth updates by @petebryan in #351
    • Re-implemented using DefaultCredential
    • Added support for MSAL token cache
  • Bump sphinx from 4.4.0 to 4.5.0 by @dependabot in #350
  • Fixes for GeoLiteLookup and MpConfigEdit by @ianhelle in #356
    • GeoIPLite no longer tries to update DB during initialization - only on first query
    • Fixes to MpConfigEdit and MpConfig file for msticpyconfig path handling.
  • Some fixes to Kusto common_imports by @ianhelle in #358
  • Changing the pattern for httpx timeout to default to Timeout(None). by @ianhelle in #378
    • config based setting for http timeouts
    • can be overridden in calls in several providers - OData (MSGraph, MDE) and HTTP-based TI providers.
  • Add Workflow to Tweet by @petebryan in #369
    • testing to automatically tweet status on PR completion
  • Fixed minor issues by @petebryan in #371
    • some issues with incident exploration having incomplete information
  • Fixing bug in local_data_driver.py if CSV with no TimeGenerated field by @ianhelle in #374
    • also added new Sentinel query list_logon_attempts_by_ip
  • Ianhelle/nb fixes 2022 04 20 by @ianhelle in #379
  • Added new Sentinel Search Features: by @petebryan in #376
    • Sentinel Search API - can create, delete and check status of an automated search
    • Also added feature to treat cases where you only have one Sentinel workspace configured - this will be treated as the default
      even if it is not marked as the default.

Full Changelog: v1.7.5...v1.8.0

Contributors

petebryan, ianhelle, and dependabot
Loading