Create repo for general-purpose utilities

[jnspitale]

We have a good deal of repeated code that could be turned into utility functions that would be easier to maintain and would make the code more readable. An example of where this would have been handy was in the migration to python3, where some repeated code got updated and some didn't, leading to intermittent problems until all of the instances were fixed.

In my own code I include a note "move to utilities" next to functions that should be moved to this new repo. I don't know if rms-tools is the right place or whether we need something like rms-utilities.

[astrocfi]

rms-tools isn't currently used for anything, so it's available. But could you give more examples of what you would want to include? Any software placed there would have to be vetted and automatically tested with the understanding that any future changes could break a lot of code. Also it would have to be installable as a pip package so that when people install our other modules they would also install this one because it's needed for support. So it would really have to be treated like a supported, documented, tested library.

[jnspitale]

#===============================================================================
def splitpath(path: str, string: str): ### move to utilities
"""Split a path at a given string.

Args:
    path (str, Path, or FCPath): Path to split.
    string (str): 
        Search string. The path is split at the first occurrence and the search 
        string is omitted.

Returns:
    NamedTuple (lines (list), lnum (int)): 
        lines: Lines comprising the output label.
        lnum: Line number in output label at which processing 
                    is to continue.
"""
parts = path.parts
i = parts.index(string)
return (FCPath('').joinpath(*parts[0:i]), FCPath('').joinpath(*parts[i+1:]))

#===============================================================================
def add_by_base(x_digits, y_digits, bases): ### move to utilities
"""Add numbers represented using the specified bases.

Args:
    x_digits (list): Digits (int) representing the first operand.
    y_digits (list): Digits (int) representing the second operand.
    bases (list): Bases (int) for each position.

Returns:
    list: Digits (int) representing the result.

"""
import math

result = [0]*(len(bases)+1)
for i, (x_digit, y_digit, base) in \
    enumerate(zip(reversed(x_digits), reversed(y_digits), reversed(bases))):
    result[i] += (x_digit + y_digit) % base
    result[i+1] += (x_digit + y_digit) // base
return list(reversed(result))

#===============================================================================
def expandvars(filespec): ### add to FCPath?
"""Expand environment variables in path.

Args:
    filespec (str, Path, or FCPath): Path to expand.

Returns:
    str, Path, or FCPath: Expanded path.

"""
if not isinstance(filespec, str):
    result = filespec.as_posix()

result = re.sub('://', '<<token>>', result)
result = os.path.expandvars(result)
result = re.sub('<<token>>', '://', result)

if isinstance(filespec, str):
    return result.as_posix()
if isinstance(filespec, FCPath):
    return FCPath(result)
return Path(result)

#===============================================================================
def read_txt_file(filespec, as_string=False, terminator='\r\n'): ### move to utilities
"""Read a text file, with some options.

Args:
    filespec (str, Path, or FCPath): 
        Path to the file to read.  Environment variables are expanded.
    as_string (bool, optional): 
        If True, the result is returned as a string using the specified 
        terminator.
    terminator (str): Terminator to use for string return.

Returns:
    list (as_string==False): Lines of the file with no terminators.
    str (as_string==True): Lines of the file concatenated using the specified
    terminator. 

"""

# Expand environment variables and resolve to absolute path
filespec = expandvars(filespec)

# Read the file
content = filespec.read_text(encoding='utf-8', newline=terminator)    
if as_string:
    return content

# Split into list of lines with no terminator
content = content.split('\n')
if content[-1] == '':
    content = content[:-1]
content = [c.rstrip('\r\n') for c in content]

return content

#===============================================================================
def write_txt_file(filespec, content, terminator='\r\n'): ### move to utilities
"""Write a text file, with some options.

Args:
    filespec (str, Path, or FCPath): Path to the file to write.
    content (str or list): 
        Text to write.  If list, each element is a line that will be terminated
        using the specified terminator.  If string, existing terminators are 
        replaced with the specified terminator.
    terminator (str): Desired line terminator.

Returns:
    None
"""

# Expand environment variables and resolve to absolute path
filespec = expandvars(filespec)

# Determine terminator
if terminator is None:
    if isinstance(content, list):
        crlf = content[0].endswith('\r\n')
    else:
        crlf = content.endswith('\r\n')
    terminator = '\r\n' if crlf else '\n'

# Split into list of lines with no terminator
if not isinstance(content, list):
    content = content.split('\n')
content = [c.rstrip('\r\n') for c in content]

# Reconstitute with correct terminator
content = terminator.join(content) + terminator

# Write file
filespec.write_text(content, encoding='utf-8')

#===============================================================================
def rebase(x, bases, ceil=False): ### move to utilities
"""Convert a decimal number to a different base.

Args:
    x (int): Number to convert.
    bases (list): Base (int) to use for each decimal place.

Returns:
    NamedTuple (digits (list), over (int)): 
        digits: Digits (int) in the new base.
        overflow: 
            Remaining quantity, if any, exceeding the maximum value that can be
            represented by the given bases.
"""
import math

digits = []
for base in reversed(bases):
    digit = x % base
    if not ceil:
        digit = int(digit)
    else:
        digit = math.ceil(digit)
    digits.append(digit)

    x //= base
return (list(reversed(digits)),x)

[jnspitale]

Agreed would have to be fully vetted. Could break a lot of code because it can fix a lot of code every time the rug get pulled out form under us with a new python version.

[rfrenchseti]

A few thoughts:

• splitpath If you really think it's useful we could add it to FCPath. Personally it makes me nervous to rely on splitting at a particular directory name, because there's no guarantee that the root doesn't have that same name. Like if you think the directory should look like .../top_level_holdings/holdings/volumes/stuff there's no guarantee the person doesn't put it in a directory called holdings so the full path is /mnt/holdings/top_level_holdings/holdings/volumes/stuff. Then splitting on holdings does the wrong thing.
• add_by_base should just be int(s1, base=base1) + int(s2, base=base2) (assuming you don't truly want to pass around your numbers as lists of digits and strings are OK). I don't think that needs to be a function.
• expandvars should be added as functionality inside FCPath. I'll create an issue.
• rebase Do you really need to support arbitrary bases? If not, Python already natively supports conversion to decimal, octal, and hex.
  But in general I really don't think these kinds of things warrant the creation and maintenance of an entire new Python package, but we can talk about it more. The kind of disruption we got from Python 2 to Python 3 is unlikely to happen again for a long time, and even if it did we don't know now what would be changing, so we can't prepare for it.

[jnspitale]

Good point about splitpath, I'll take a look at whether there's a better way to do that. Arbitrary bases are needed in dealing with spacecraft clocks, but maybe not more generally.

We could benefit from a layer of abstraction for some common operations, but many of them can probably be separate packages (or added to existing packages). For example, we shouldn't be worrying about line terminators every time we read or write a file (except at the point of reading and writing), so maybe we just need a package for simplifying / standardizing file I/O.

I find the argument that we hope python doesn't screw us (again) in the near future very compelling.

[rfrenchseti]

Anything having to do with reading a label or table should be in pdstable. Anything having to do with writing a label should be in pdstemplate. Perhaps we should add table writing to pdstable. If you aren't developing one of those packages, you should never need to worry about line terminations.

It would not be a bad idea to have a package for dealing with spacecraft clock counts. We actually already have a lot of code to deal with this in OPUS that we could separate out and put somewhere: https://github.com/SETI/rms-opus/blob/main/lib/opus_support.py

[jnspitale]

read_ and write_text_file are for reading and text files, not just labels. I like the idea of a package to deal with spacecraft clocks. A quick browse of the opus support code does suggest that there may be code in there that could be useful to have available more broadly so we don't keep having to reinvent the wheel.