Plugins for TI Providers

[ianhelle]

Two of our (recently) most requested features are:

• Supporting extensibility for some components such as TI and Context providers
• Adding support for a range of TI providers

This proposal addresses (at least partly) these two items.

Rationale for Config-based TI and Context providers

HTTP-based context providers have a very regular pattern:

1. global configuration: name, base API url and other metadata
2. URL paths for different IoC/observable types
3. Handling authentication by injecting API keys into the request
4. Parsing results to extract severity and summary information

1 and 2 can easily be specified in a configuration schema in JSON or YAML.
3 has limited range of possibilities (key in headers, query string, etc.) and we already handle these variations declaratively
4 is a bit more challenging but could be handled with a mini-parsing language expressed in yaml/json

We'd like to avoid consuming and exec-ing or eval-ing code from YAML files.

Proposal

1. Define TI provider in yaml/json
   Definitions can be:
   a. Included in the msticpy package
   - These would be stored as yaml files in mp/resources
   - A yaml-based TI providers would load/instantiate if there is an entry for it in msticpyconfig.yaml
   b. User-defined providers
   - YAML file definition
   - Config entry in msticpyconfig.yaml. The entry would contain either
   - a path to provider definition yaml file
   - a key to a provider definition stored inside of msticpyconfig.yaml
2. Load and Instantiate the provider
   • On load, create new TI class from definition
     • Validate definition on create
     • Create required parse_results function using logic from definition
     • Output dynamically-created class (subclass of HTTPProvider)
     • Instantiate the class and add to TILookup

A couple of things that we should add to this work:

• It should work with the generic context provider based on @FlorianBracq's PoC
• We should change TI provider classes to Singletons - we do internal caching of lookup results, which are no benefit if you create a new instance.
• Singletons should be argument-keyed. So if I create 3 instances TI(apikey=A), TI(apikey=B) and TI(apikey=A). I will get 2 instances. The second instantiation with the same args will just return the existing instance.

Definition Format

This is an overly wordy definition which includes a bunch of optional items that wouldn't be
needed in a typical definition. Just used to define parsing logic. There's a shorter definition
for OTX at the end of the doc.

Note - the entire summary section is optional.

name: OTX
description: AlienVault OTX
base_url: https://otx.alienvault.com/   # [OPT] if None - queries must contain absolute URL
required_parameters: [API_KEY]          # The parameters that must be available from config
                                        #    to instantiate the provider (usually API_KEY)
request_defaults:                       # The only required value in this section is is where to insert
                                        #    auth keys in the request
    headers:
        X-OTX-API-KEY: API_KEY          # Pass the API_KEY in headers
    params:                             #    or in params?
    data:                               #    or in data?
    auth_type: HTTPBasic                # [OPT] This is the default so not normally needed
    auth_str:                           # [OPT] This is needed if the auth string needs some special formatting
    verb: GET                           # [OPT] GET is default or POST
macros:                                 # [OPT] keys defined here will have their value substituted in queries 
    api_path: /api/v1/indicators
queries:                                # tokens with braces replaced by macro or runtime value
  ipv4:
    path: "{api_path}/IPv4/{observable}/general"
  ipv6:
    path: "{api_path}/IPv6/{observable}/general"
  ipv4-passivedns:
    path: "{api_path}/IPv4/{observable}/passive_dns"
  ipv6-passivedns:
    path: "{api_path}/IPv6/{observable}/passive_dns"
  ipv4-geo:
    path: "{api_path}/IPv4/{observable}/geo"
  ipv6-geo:
    path: "{api_path}/IPv6/{observable}/geo"
  dns:
    path: "{api_path}/domain/{observable}/general"
  dns-passivedns:
    path: "{api_path}/domain/{observable}/passive_dns"
  dns-geo:
    path: "{api_path}/domain/{observable}/geo"
  hostname:
    path: "{api_path}/hostname/{observable}/general"
  file_hash:
    path: "{api_path}/file/{observable}/general"
    aliases: [md5_hash, sha1_hash, sha256_hash]
  url:
    path: "{api_path}/url/{observable}/general"
result_processing:                      # Rules used to parse response
    severity:                               # Determine severity from response
        key: pulse_info.pulses              # default key for conditions to use
        conditions:                         # evaluate conditions (in order)
            high: len gt 1                      # simple condition 
            warning:
                and:                            # support ands and ors
                    - len gt 1
                    - len gt 3
            information: None                   # [OPT] default if no previous matches (optional)
        # other examples
        #   high:                               # Override section key for this condition
        #       pulse_info.pulses: len gt 1     # and specify condition for that key value
        #   warning:
        #       pulse_info.pulses:              # Override section key for this condition
        #           and: [len gt 1, len gt 3]   # with more complex condition
    summary:                            # [OPT] define which items from response to pull out
        key: pulse_info.pulses              # default key to use if not specified in subkey
        fields:                             # fields to create in summary dictionary
            pulse_count:
                action: count                   # count of contents
            sections_available:
                key: pulse_info.sections_available
                action: value                   # [OPT] add value at key (also default if no action specified)
            sections_available_k:               # list of keys at location
                key: pulse_info.sections
                action: keys
            names: 
                action:                         # get the list of values of the 'name' key
                    get_item: name              # e.g. [data[name] for data in key]
            tags: 
                key: pulse_info.pulses
                action:
                    get_item: tags              # get the list of values of the 'tags' key
                flatten: True                   # if the result is nested list, flatten
            
            references: 
                key: pulse_info.pulses
                action:
                    get_item: references
                flatten: True

Other items to consider

• Could include a flag to expand all of the summary fields to pandas columns
  (note: this would probably be different for each provider)

Appendix

More typical example AlienVault OTX

name: OTX
description: AlienVault OTX
base_url: https://otx.alienvault.com/
required_parameters: [API_KEY]
request_defaults:
    headers:
        X-OTX-API-KEY: API_KEY
macros:
    api_path: /api/v1/indicators
queries:
  ipv4:
    path: "{api_path}/IPv4/{observable}/general"
  ipv6:
    path: "{api_path}/IPv6/{observable}/general"
  ipv4-passivedns:
    path: "{api_path}/IPv4/{observable}/passive_dns"
  ipv6-passivedns:
    path: "{api_path}/IPv6/{observable}/passive_dns"
  ipv4-geo:
    path: "{api_path}/IPv4/{observable}/geo"
  ipv6-geo:
    path: "{api_path}/IPv6/{observable}/geo"
  dns:
    path: "{api_path}/domain/{observable}/general"
  dns-passivedns:
    path: "{api_path}/domain/{observable}/passive_dns"
  dns-geo:
    path: "{api_path}/domain/{observable}/geo"
  hostname:
    path: "{api_path}/hostname/{observable}/general"
  file_hash:
    path: "{api_path}/file/{observable}/general"
    aliases: [md5_hash, sha1_hash, sha256_hash]
  url:
    path: "{api_path}/url/{observable}/general"
result_processing:                      # Rules used to parse response
    severity:                               # Determine severity from response
        key: pulse_info.pulses              # default key for conditions to use
        conditions:                         # evaluate conditions (in order)
            high: len gt 1                      # simple condition 
            warning:
                and:                            # support ands and ors
                    - len gt 1
                    - len gt 3
            information: None                   # [OPT] default if no previous matches (optional)
    summary:                            # [OPT] define which items from response to pull out
        key: pulse_info.pulses              # default key to use if not specified in subkey
        fields:                             # fields to create in summary dictionary
            pulse_count:
                action: count                   # count of contents
            sections_available:
                key: pulse_info.sections_available
                action: value                   # [OPT] add value at key (also default if no action specified)
            names: 
                action:                         # get the list of values of the 'name' key
                    get_item: name              # e.g. [data[name] for data in key]
            tags: 
                key: pulse_info.pulses
                action:
                    get_item: tags              # get the list of values of the 'tags' key
                flatten: True                   # if the result is nested list, flatten
            references: 
                key: pulse_info.pulses
                action:
                    get_item: references
                flatten: True

[ianhelle]

Florian's work on generic provider model
https://github.com/FlorianBracq/msticpy/tree/context_provider

[ianhelle]

Currently this schema supports only JSON responses - so would not work with HTML or raw data responses.
If anyone has examples of TI providers that return results in formats other than JSON, please leave a comment/reference.

[ianhelle]

Another TODO item for this work - create a pub/sub mechanism so that TI providers can register to receive notification about settings being re-read. Currently they have no knowledge of this so stay with old settings. This means if you add a TI provider to your settings and refresh, it's not loaded unless you a) dig into the code a lot to find the pivot TI provider instance b) restart the kernel.