Code structure

Code explanation

Here are some explanations for the functions used in the code.

Project Structure and Key Components

The new_inventory_api.py file is a Python script designed for inventorying AWS services for a specified account across all available regions. It uses multi-threading to perform inventory operations concurrently. The key components are:

• Global Variables: Manages inventory results, progress counters, and threading configurations.
• Modules Imported: threading, boto3, json, os, sys, re, datetime, time, argparse, multiprocessing, concurrent.futures, tqdm, glob, utils (custom utility functions).
• Classes:
  • InventoryThread: A thread class for performing inventory tasks.
• Functions:
  • Utility Functions: write_log, transform_function_name, json_serial, is_empty.
  • Inventory Management Functions: get_all_regions, test_region_connectivity, create_services_structure, inventory_handling, list_used_services.
• The main function initiates the script execution, parses arguments, and manages the inventory process.

function detail_handling

def detail_handling(client, inventory, node_details, resource, key):

    """
    Handles the details of inventory items by calling specified detail functions on the client.

    Args:
        client (object): The client object used to call detail functions.
        inventory (dict): The inventory containing items to be processed.
        node_details (dict): The node in the inventory specifying details to be retrieved.
        resource (str): The type of resource being processed (e.g., 's3').
        account_id (str): The account ID used for certain resource types.

    Raises:
        ClientError: If an error occurs while calling the detail function on the client.

    Returns:
        None
    """

The detail_handling function in the new_inventory_api.py script is responsible for processing the details of inventory items by calling specified detail functions on the AWS client. This function is a crucial part of the inventory process as it enriches the basic inventory data with additional details that are necessary for a comprehensive understanding of the resources.

When the detail_handling function is called, it typically receives parameters such as the AWS client, the resource type, and the specific details that need to be retrieved. The function iterates over the inventory items and, for each item, it invokes the appropriate detail functions. These detail functions are designed to make additional API calls to AWS services to fetch more granular information about each resource.

For example, if the inventory includes EC2 instances, the detail_handling function might call detail functions to retrieve information about instance types, security groups, or attached volumes. This additional data is then appended to the inventory item, ensuring that the final inventory dataset is rich with detailed information.

The detail_handling function also includes error handling mechanisms to manage any exceptions that might occur during the API calls. This is important because AWS environments can be complex, and network issues or API rate limits can cause errors. By handling these exceptions gracefully, the function ensures that the inventory process can continue even if some detail retrievals fail.

function inventory_handling

def inventory_handling(category, region_name, resource, boto_resource_name, node_details, key, progress_callback):

    """
    Handles the inventory process for a given AWS resource in a specified region.

    Parameters:
        category (str): The category of the resource.
        region (str): The AWS region where the resource is located.
        resource (str): The type of AWS resource to inventory.
        boto_resource_name (str): The name used in boto3 (genrally the same, but you have surprises)
        node_details (dict): Details about the node, including the function to call and any additional details.
        key (str): A key used for logging or identification purposes.
        progress_callback (function): A callback function to report progress.
    
    Returns:
        None

    Raises:
        AttributeError: If there is an attribute error during the inventory process.
        ClientError: If there is a client error during the inventory process.
        EndpointConnectionError: If there is a connection error during the inventory process.
        Exception: For any other exceptions that occur during the inventory process.

    Notes:
    - This function logs the start and end of the inventory process.
    - It handles specific exceptions for EC2 resources that require a region even when global.
    - The function supports threading and reports progress through the progress_callback.
    - It constructs the inventory and handles empty results based on the provided arguments.
    - Detailed resource information can be retrieved if specified in node_details.
    """

The provided code is part of an AWS inventory script designed to perform an inventory of AWS resources across multiple regions using multithreading. The script imports various modules, including threading for threading capabilities, boto3 for AWS SDK interactions, and several utility modules for logging, JSON handling, and more. The script also defines global variables to store results, progress counters, and configuration settings.

The InventoryThread class extends the threading.Thread class to perform inventory tasks in parallel. It initializes with parameters such as category, region name, resource type, boto resource name, node details, key, and a progress callback function. The run method of this class calls the inventory_handling function to handle the inventory process for a specified AWS resource and region.

The inventory_handling function is the core of the script, responsible for managing the inventory process. It logs the start and end of the inventory process, handles specific exceptions for EC2 resources, and supports threading. The function iterates over the node_details to make API calls for each resource, handles exceptions, and constructs the inventory results. It also reports progress through the provided callback function.

The detail_handling function processes the details of inventory items by calling specified detail functions on the client. It iterates over the inventory items and retrieves additional details as specified in the node_details.

The resource_inventory function collects resource inventory for a specified AWS resource and region, updating progress and managing threads for concurrent execution. It initializes threads for each sub-task and adds them to a thread list.

The list_used_resources function orchestrates the overall inventory process. It retrieves all AWS regions, initializes the progress bar, manages threads using ThreadPoolExecutor, and logs the execution time and results. It also writes the results to a JSON file and returns the inventory results.

The script's main function handles command-line arguments, finds policy files, converts YAML policy files to JSON, and calls the list_used_resources function to perform the inventory and list used resources. The script is designed to be run with appropriate command-line arguments to perform the inventory, making it a comprehensive tool for AWS resource inventory management.

function list_used_resources

def list_used_resources(inventory_structure):

    """
    List used resources based on the provided YAML files.

    This function performs the following steps:
    1. Retrieves all AWS regions.
    2. Creates a structure of resources based on the provided IAM policy files.
    3. Initializes and manages threads to query both global and regional resources.
    4. Updates a progress bar to reflect the progress of the inventory process.
    5. Logs the execution time and results of the inventory process.
    6. Writes the results to a JSON file.

    Args:
        inventory_structure (list): A structure of resources based on the provided YAML file with all informations about the resources.

    Returns:
        dict: A dictionary of the used resources.
    """

The list_used_resources function in the new_inventory_api.py script is the main orchestrator for the AWS inventory process. This function is responsible for coordinating the retrieval and processing of AWS resource information across multiple regions, utilizing multithreading to enhance performance and efficiency.

When list_used_resources is called, it begins by retrieving a list of all available AWS regions using the get_all_regions function. This ensures that the inventory process covers all regions where resources might be deployed. The function then initializes a progress bar using the tqdm library to provide visual feedback on the progress of the inventory process.

Next, list_used_resources sets up a ThreadPoolExecutor to manage the concurrent execution of inventory tasks. It creates and submits tasks to the executor for each combination of resource type and region, leveraging the InventoryThread class to handle the actual inventory retrieval in parallel threads. This approach significantly speeds up the inventory process by utilizing multiple threads to perform tasks simultaneously.

As the threads execute, they call the inventory_handling function to retrieve and process resource information. The list_used_resources function monitors the progress of these threads, updating the progress bar and handling any exceptions that might occur. This ensures that the inventory process is robust and can handle errors gracefully.

Once all threads have completed their tasks, list_used_resources aggregates the results and logs the execution time and final inventory data. It writes the inventory results to a JSON file for further analysis and reporting. Finally, the function returns the comprehensive inventory results, which include detailed information about all AWS resources across the specified regions.

The list_used_resources function is a critical component of the AWS inventory script, providing a structured way to gather and process resource information across multiple regions. Its use of multithreading and robust error handling ensures that the inventory process is both fast and reliable.