Misc notes on config2py

[thorwhalen]

Just for miscellaneous, uncategorized notes.

[thorwhalen]

2023-09-27 meeting notes

• Might want to change the name config2py to something less x2y, and with less strong semantics. We're already noticing that our tool is both config2py and py2config, and that it goes beyond configs (though if we generalize all the way, we're back to general key-value interfaces (with is the domain of dol).
• Want to generalize the ConfigStore (or Configs) to be more than just ini/cfg files, but include other types of config files like py, json, csv, and (with dynamic dependencies) yaml, toml...
• Want this general ConfigStore to have an open-closed plugin architecture (go see pyckup grabber for an an example of such a thing.

Parsing py files

Specifying configurations in .py files is quite natural, and powerful (because our values can be any valid python object) in the python context, so we want to support that. The easiest, but unsafe way, is to import a module where these configuration objects are defined. But there is also a safe, no-need-to-execute-the-module, way: Parse the python code. This is more limited though.

Unsafe way

The easiest way to parse out key-values from a python module is to import it and extract what we want.
Note, though, that this executes the module, so it can be unsafe if you're getting your python file from someone you might not trust.

meshed has a function called objects_defined_in_module that does the job.

Note that one can use partial to set name and object filters.

Safe way

The builtin ast.parse can safely parse Python source code to create an Abstract Syntax Tree (AST), and it won't execute any code in the process. This can indeed be used to extract variable names and potentially their assigned values if they are literals or other simple expressions, but keep in mind, you won’t be able to retrieve the runtime values of more complex expressions.

This config2py.parse_assignments_from_py_source will do that job.

Caveats of the "save way"

1. Limitation on Values:

   • You can only safely get the values of simple literal expressions (like numbers, strings, lists, tuples, dictionaries).
   • You won't be able to get the runtime values of variables assigned with more complex expressions, function calls, or objects.

2. Global Scope:

   • This approach will only find the variables in the global scope of the module, not those defined inside functions, classes, or other scopes.

3. Code Safety:

   • Be cautious with ast.literal_eval as it can raise exceptions if it encounters unsupported expressions.

4. Dynamic Assignments:

   • Variables assigned dynamically (e.g., using exec or setattr) won’t be detected with this method.

5. AST Changes:

   • The structure of the AST can change between Python versions, so code relying on ast.parse may need updates for different Python versions.

[thorwhalen]

.config versus .cache

I read that the default folder for app data is .config, so I've been using the ~/.config/APP_NAME/configs folder to configurations of APP_NAME, and ~/.config/APP_NAME/data to hold data. Should I be using ~/.cache/APP_NAME instead of ~/.config/APP_NAME/data to store data for an application/package?

When developing a Python package for managing configuration and data files, understanding the appropriate directories for storing these files is crucial for both functionality and adherence to standard practices. The distinction between ~/.config and ~/.cache is particularly important in this context.

Standard Directory Usage

1. Configuration Files (~/.config):

   • The ~/.config directory is intended for storing configuration files. These are files that contain settings and preferences for an application. This includes user-specific configuration data that needs to be preserved between sessions.
   • For example, storing configuration information in ~/.config/APP_NAME/configs aligns with the convention and is appropriate.

2. Cache Files (~/.cache):

   • The ~/.cache directory is meant for non-essential data that can be regenerated or re-downloaded if necessary. This includes temporary files, cached data, and other ephemeral information that the application can function without.
   • If the data stored in ~/.config/APP_NAME/data is not critical for the operation of the application and can be recreated, it should be stored in ~/.cache/APP_NAME instead. This ensures that the user’s configuration directory is not cluttered with data that can be discarded.

Best Practices

To align with best practices, consider the nature of the data you are storing:

• Essential and Persistent Data:

  • Data that is crucial for the application’s operation and needs to persist across sessions should remain in ~/.config/APP_NAME.
  • This could include user profiles, application settings, and other critical data that the application needs to function correctly.

• Non-Essential and Temporary Data:

  • Data that can be safely deleted without affecting the application's ability to function should be stored in ~/.cache/APP_NAME.
  • Examples include cached results, temporary files, session-specific data, and any other data that can be regenerated.

Example Structure

Given the above considerations, an example structure for an application named APP_NAME could be:

• Configuration Files:

  • ~/.config/APP_NAME/configs: For storing configuration settings.
  • ~/.config/APP_NAME/preferences: For user preferences and settings.

• Cache Files:

  • ~/.cache/APP_NAME: For storing cached data that can be safely deleted.

Implementation in Python

For implementation within your Python package, consider using the pathlib module to manage paths:

from pathlib import Path
import os

def get_config_path(app_name: str) -> Path:
    return Path(os.path.expanduser(f"~/.config/{app_name}/configs"))

def get_cache_path(app_name: str) -> Path:
    return Path(os.path.expanduser(f"~/.cache/{app_name}"))

# Example usage
config_path = get_config_path('APP_NAME')
cache_path = get_cache_path('APP_NAME')

# Ensure directories exist
config_path.mkdir(parents=True, exist_ok=True)
cache_path.mkdir(parents=True, exist_ok=True)

This approach ensures that your application follows standard directory practices, keeping configuration and cache data organized and in appropriate locations. By adhering to these conventions, your application will integrate more seamlessly with user environments and other software, enhancing its usability and maintainability.