Logging refactor

[kbwestfall]

This refactors the logging infrastructure for PypeIt so that it now uses the standard python logging library (I don't know why I torture myself...). The new logger draws heavily from the ones implemented by astropy and sdsstools.

The python logging package gives the following levels, DEBUG, INFO, WARNING, ERROR, and CRITICAL. In their parlance, we basically consider all ERRORs as CRITICAL, so the log.critical function isn't really useful to us. For the scripts, I've mapped the verbosity to these levels as DEBUG = 2, INFO = 1, and WARNING = 0. I.e., if verbosity == 0, we only log warnings and errors, whereas verbosity == 1 adds info message, and verbosity == 2 adds debug messages. All the scripts now also allow us to set the level specifically for the file log (using the same integer mapping as the console verbosity), which can be different from the console log.

The key changes are:

• pypmsgs.py is effectively replaced by logger.py. There is some functionality loss (I think this is primarily limited to the fact that we will no longer catch Ctrl-c events and the QA html will not be automatically constructed when failures happen), so let me know if you think we're losing something critical
• instead of from pypeit import msgs, basically every file will need to from pypeit import log, PypeItError.
• anywhere you would have put msgs.error(...), you should instead raise PypeItError(...). All exceptions (and warnings) are now caught and processed by the PypeItLogger (not just PypeItError instances); so if you think its more appropriate to raise a different error type, go for it. The logging class has error and critical functions, but they're not that useful for pypeit.
• the warn function in the logging package has been deprecated, so all warnings should use log.warning instead of msgs.warn.
• we should now consider making more use of the debug message level. Anything that used to use msgs.work, I've now converted to log.debug, but there are some conceptual differences there.
• the msgs.newline() function no longer exists. If you want a new line in the log message, include the \n character directly. I was thinking of adding some line-wrapping tools for messages in the future, but formatted output that suits everyone's terminal width choices is tricky.
• all the scripts now add verbosity and log file options by default (see above)
• the default logging level is set to DEBUG, which displays all messages and adds the function file, name, line number information.
• the INFO level does not include the calling function information
• if the previous scripts had a default log file, I kept it, but now effectively any script can produce a log file with a user-specified name.
• The format of the console stream logging and the file logging are different
  • the calling function information is always included in the file log and
  • the file log includes date/time information.
• The logging level can be different between the console stream and the file stream. I.e., one could set the console to only display warnings and errors, but include all the messages in the file log (or vice versa).
• most of the changes are just find/replace, so the key files to review are pypeit/logger.py, pypeit/__init__.py, pypeit/tests/test_log.py.

I'm submitting this as a draft PR, but I think it's mostly done. But, because this PR basically touches everything, we should discuss when it gets merged. I expect it should at least come after #2007 and #1971 , and I will deal with the likely conflicts. So if you have another big PR in the works and don't want to deal with the conflicts, let's try to get those in before this one.

[ejeschke]

Good move to integrate with standard library logger. One cool feature is the use of the exc_info flag when you call logger.error() (or possibly other levels as well). If you call the logger in a except clause, and pass the keyword argument exc_info=True, then in addition to logging the error message it will also log a stack trace. Very handy.

[rcooke-ast]

I'm OK with this. It's great that you've brought this part of the code into the present times!

I only reviewed __init__.py, test_logger.py and logger.py.

I couldn't checkout a test drive of the code, just to see what it looks like as terminal printout. Could you please post a representative screenshot? Thanks!

[tbowers7]

This is a great advancement for PypeIt. There were several small things I noticed, but nothing major. Merging other PRs into this one will be challenging and will require patience. Also, there was a blanket addition of log and PypeItError to files that previously had msgs, but both were not needed all of the time. May be worth a fine-tooth comb to remove spurious imports, by maybe not?

The only request I have is to re-include the Ctrl+C catcher and provide a graceful shutdown. With the use of atexit.register(), it should be possible to close out QA's and logging gracefully before sys.exit()-ing.

[tbowers7]

Does this function mean that we log errors but still allow them to crash the code?

Is there a more graceful way to print useful traceback without a full-on crash?

[kbwestfall]

Right, the purpose of this function is basically to add the log entry before executing the same exception code that would have been executed without the bespoke logger.

Similar to a comment above, I'm not sure what you mean by exiting more gracefully, but let's discuss.

[kbwestfall]

Thanks for the detailed review, @tbowers7 ! I'd like to talk through how we exit because I don't understand the details well enough. Let's talk at the next development telecon about that and/or over Slack.

[kbwestfall]

I've never used the atexit package, but it seems pretty useful. I also don't have a good sense of why we should prefer to exit gracefully (I'm not sure what that would look like) when someone intentionally kills the program using Ctrl+C.

Let's discuss at the next telecon and/or over Slack.

[kbwestfall]

Right, the purpose of this function is basically to add the log entry before executing the same exception code that would have been executed without the bespoke logger.

Similar to a comment above, I'm not sure what you mean by exiting more gracefully, but let's discuss.

[Copilot]

Pull request overview

This PR refactors PypeIt's logging infrastructure to use Python's standard logging library instead of the custom pypmsgs module. The new implementation provides standard logging levels (DEBUG, INFO, WARNING, ERROR) with improved flexibility, including separate verbosity control for console and file outputs.

Key changes:

• Replace pypmsgs.py with logger.py using standard Python logging
• Add new exceptions.py module to centralize exception definitions
• Update all scripts to support verbosity and log file options by default

Reviewed changes

Copilot reviewed 300 out of 515 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
pypeit/init.py	Replaces pypmsgs import with new logger module and exception imports
pypeit/cache.py	Updates logging calls from msgs.warn/error to log.warning and raise PypeItError
pypeit/archive.py	Updates logging calls from msgs.info/error to log.info and raise PypeItError
pypeit/alignframe.py	Updates logging calls, replaces msgs.work with log.debug and msgs.error with raise PypeItError
doc/pypeit_par.rst	Updates parameter documentation with new default values and parameters
doc/conf.py	Adds sphinx configuration for type hints and additional intersphinx mappings
doc/api/*	Updates API documentation structure with new logger module and exceptions module
doc/help/*	Updates script help documentation with new verbosity and logging options
doc/include/*	Updates datamodel documentation versions and dependency tables

Comments suppressed due to low confidence (1)

pypeit/cache.py:1

• Multi-line error messages are missing line breaks between concatenated f-strings. Add \n characters to ensure proper formatting when displayed.

# -*- coding: utf-8 -*-

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[kbwestfall]

Dev-suite tests started.