Merge pull request #676 from python-cmd2/pipe_chaining

Pipe chaining

Author: kmvanbrunt

.travis.yml

@@ -46,7 +46,7 @@ before_script:
   # stop the build if there are Python syntax errors or undefined names
   # exit-zero treats all errors as warnings.  The GitHub editor is 127 chars wide
   if [[ $TOXENV == py37 ]]; then
-    flake8 . --count --ignore=E252 --max-complexity=31 --max-line-length=127 --show-source --statistics ;
+    flake8 . --count --ignore=E252,W503 --max-complexity=31 --max-line-length=127 --show-source --statistics ;
   fi
 
 script:

CHANGELOG.md

@@ -5,7 +5,11 @@
     * History now shows what was typed for macros and not the resolved value by default. This is consistent with
     the behavior of aliases. Use the `expanded` or `verbose` arguments to `history` to see the resolved value for
     the macro.
+    * Fixed parsing issue in case where output redirection appears before a pipe. In that case, the pipe was given
+    precedence even though it appeared later in the command.
+    * Fixed issue where quotes around redirection file paths were being lost in `Statement.expanded_command_line()`
 * Enhancements
+    * Added capability to chain pipe commands and redirect their output (e.g. !ls -l | grep user | wc -l > out.txt)
     * `pyscript` limits a command's stdout capture to the same period that redirection does.
       Therefore output from a command's postparsing and finalization hooks isn't saved in the StdSim object.
     * `StdSim.buffer.write()` now flushes when the wrapped stream uses line buffering and the bytes being written
@@ -18,6 +22,7 @@
 * Potentially breaking changes
     * Replaced `unquote_redirection_tokens()` with `unquote_specific_tokens()`. This was to support the fix
       that allows terminators in alias and macro values.
+    * Changed `Statement.pipe_to` to a string instead of a list 
 * **Python 3.4 EOL notice**
     * Python 3.4 reached its [end of life](https://www.python.org/dev/peps/pep-0429/) on March 18, 2019
     * This is the last release of `cmd2` which will support Python 3.4
@@ -87,7 +92,7 @@
     sorted the ``CompletionItem`` list. Otherwise it will be sorted using ``self.matches_sort_key``.
     * Removed support for bash completion since this feature had slow performance. Also it relied on
     ``AutoCompleter`` which has since developed a dependency on ``cmd2`` methods. 
-    * Removed ability to call commands in ``pyscript`` as if they were functions (e.g ``app.help()``) in favor
+    * Removed ability to call commands in ``pyscript`` as if they were functions (e.g. ``app.help()``) in favor
     of only supporting one ``pyscript`` interface. This simplifies future maintenance.
     * No longer supporting C-style comments. Hash (#) is the only valid comment marker.
     * No longer supporting comments embedded in a command. Only command line input where the first

cmd2/argparse_completer.py

@@ -999,9 +999,7 @@ def error(self, message: str) -> None:
             linum += 1
 
         self.print_usage(sys.stderr)
-        sys.stderr.write(Fore.LIGHTRED_EX + '{}\n'.format(formatted_message) + Fore.RESET)
-
-        sys.exit(1)
+        self.exit(2, Fore.LIGHTRED_EX + '{}\n\n'.format(formatted_message) + Fore.RESET)
 
     def format_help(self) -> str:
         """Copy of format_help() from argparse.ArgumentParser with tweaks to separately display required parameters"""
@@ -1051,7 +1049,7 @@ def format_help(self) -> str:
         formatter.add_text(self.epilog)
 
         # determine help from format above
-        return formatter.format_help()
+        return formatter.format_help() + '\n'
 
     def _get_nargs_pattern(self, action) -> str:
         # Override _get_nargs_pattern behavior to use the nargs ranges provided by AutoCompleter

cmd2/cmd2.py

@@ -1519,18 +1519,19 @@ def _complete_worker(self, text: str, state: int) -> Optional[str]:
                             # Check if any portion of the display matches appears in the tab completion
                             display_prefix = os.path.commonprefix(self.display_matches)
 
-                            # For delimited matches, we check what appears before the display
-                            # matches (common_prefix) as well as the display matches themselves.
-                            if (' ' in common_prefix) or (display_prefix and ' ' in ''.join(self.display_matches)):
+                            # For delimited matches, we check for a space in what appears before the display
+                            # matches (common_prefix) as well as in the display matches themselves.
+                            if ' ' in common_prefix or (display_prefix
+                                                        and any(' ' in match for match in self.display_matches)):
                                 add_quote = True
 
                         # If there is a tab completion and any match has a space, then add an opening quote
-                        elif common_prefix and ' ' in ''.join(self.completion_matches):
+                        elif common_prefix and any(' ' in match for match in self.completion_matches):
                             add_quote = True
 
                         if add_quote:
                             # Figure out what kind of quote to add and save it as the unclosed_quote
-                            if '"' in ''.join(self.completion_matches):
+                            if any('"' in match for match in self.completion_matches):
                                 unclosed_quote = "'"
                             else:
                                 unclosed_quote = '"'
@@ -1540,7 +1541,7 @@ def _complete_worker(self, text: str, state: int) -> Optional[str]:
                     # Check if we need to remove text from the beginning of tab completions
                     elif text_to_remove:
                         self.completion_matches = \
-                            [m.replace(text_to_remove, '', 1) for m in self.completion_matches]
+                            [match.replace(text_to_remove, '', 1) for match in self.completion_matches]
 
                     # Check if we need to restore a shortcut in the tab completions
                     # so it doesn't get erased from the command line
@@ -2027,35 +2028,44 @@ def _redirect_output(self, statement: Statement) -> Tuple[bool, utils.Redirectio
             subproc_stdin = io.open(read_fd, 'r')
             new_stdout = io.open(write_fd, 'w')
 
-            # We want Popen to raise an exception if it fails to open the process.  Thus we don't set shell to True.
+            # Set options to not forward signals to the pipe process. If a Ctrl-C event occurs,
+            # our sigint handler will forward it only to the most recent pipe process. This makes
+            # sure pipe processes close in the right order (most recent first).
+            if sys.platform == 'win32':
+                creationflags = subprocess.CREATE_NEW_PROCESS_GROUP
+                start_new_session = False
+            else:
+                creationflags = 0
+                start_new_session = True
+
+            # For any stream that is a StdSim, we will use a pipe so we can capture its output
+            proc = subprocess.Popen(statement.pipe_to,
+                                    stdin=subproc_stdin,
+                                    stdout=subprocess.PIPE if isinstance(self.stdout, utils.StdSim) else self.stdout,
+                                    stderr=subprocess.PIPE if isinstance(sys.stderr, utils.StdSim) else sys.stderr,
+                                    creationflags=creationflags,
+                                    start_new_session=start_new_session,
+                                    shell=True)
+
+            # Popen was called with shell=True so the user can chain pipe commands and redirect their output
+            # like: !ls -l | grep user | wc -l > out.txt. But this makes it difficult to know if the pipe process
+            # started OK, since the shell itself always starts. Therefore, we will wait a short time and check
+            # if the pipe process is still running.
             try:
-                # Set options to not forward signals to the pipe process. If a Ctrl-C event occurs,
-                # our sigint handler will forward it only to the most recent pipe process. This makes
-                # sure pipe processes close in the right order (most recent first).
-                if sys.platform == 'win32':
-                    creationflags = subprocess.CREATE_NEW_PROCESS_GROUP
-                    start_new_session = False
-                else:
-                    creationflags = 0
-                    start_new_session = True
-
-                # For any stream that is a StdSim, we will use a pipe so we can capture its output
-                proc = \
-                    subprocess.Popen(statement.pipe_to,
-                                     stdin=subproc_stdin,
-                                     stdout=subprocess.PIPE if isinstance(self.stdout, utils.StdSim) else self.stdout,
-                                     stderr=subprocess.PIPE if isinstance(sys.stderr, utils.StdSim) else sys.stderr,
-                                     creationflags=creationflags,
-                                     start_new_session=start_new_session)
+                proc.wait(0.2)
+            except subprocess.TimeoutExpired:
+                pass
 
-                saved_state.redirecting = True
-                saved_state.pipe_proc_reader = utils.ProcReader(proc, self.stdout, sys.stderr)
-                sys.stdout = self.stdout = new_stdout
-            except Exception as ex:
-                self.perror('Failed to open pipe because - {}'.format(ex), traceback_war=False)
+            # Check if the pipe process already exited
+            if proc.returncode is not None:
+                self.perror('Pipe process exited with code {} before command could run'.format(proc.returncode))
                 subproc_stdin.close()
                 new_stdout.close()
                 redir_error = True
+            else:
+                saved_state.redirecting = True
+                saved_state.pipe_proc_reader = utils.ProcReader(proc, self.stdout, sys.stderr)
+                sys.stdout = self.stdout = new_stdout
 
         elif statement.output:
             import tempfile
@@ -2072,7 +2082,7 @@ def _redirect_output(self, statement: Statement) -> Tuple[bool, utils.Redirectio
                 if statement.output == constants.REDIRECTION_APPEND:
                     mode = 'a'
                 try:
-                    new_stdout = open(statement.output_to, mode)
+                    new_stdout = open(utils.strip_quotes(statement.output_to), mode)
                     saved_state.redirecting = True
                     sys.stdout = self.stdout = new_stdout
                 except OSError as ex:
@@ -3021,21 +3031,8 @@ def do_shell(self, args: argparse.Namespace) -> None:
         # Create a list of arguments to shell
         tokens = [args.command] + args.command_args
 
-        # Support expanding ~ in quoted paths
-        for index, _ in enumerate(tokens):
-            if tokens[index]:
-                # Check if the token is quoted. Since parsing already passed, there isn't
-                # an unclosed quote. So we only need to check the first character.
-                first_char = tokens[index][0]
-                if first_char in constants.QUOTES:
-                    tokens[index] = utils.strip_quotes(tokens[index])
-
-                tokens[index] = os.path.expanduser(tokens[index])
-
-                # Restore the quotes
-                if first_char in constants.QUOTES:
-                    tokens[index] = first_char + tokens[index] + first_char
-
+        # Expand ~ where needed
+        utils.expand_user_in_tokens(tokens)
         expanded_command = ' '.join(tokens)
 
         # Prevent KeyboardInterrupts while in the shell process. The shell process will
@@ -3334,18 +3331,21 @@ def load_ipy(app):
                                               help='output commands to a script file, implies -s'),
             ACTION_ARG_CHOICES, ('path_complete',))
     setattr(history_action_group.add_argument('-t', '--transcript',
-                                              help='output commands and results to a transcript file, implies -s'),
+                                              help='output commands and results to a transcript file,\n'
+                                                   'implies -s'),
             ACTION_ARG_CHOICES, ('path_complete',))
     history_action_group.add_argument('-c', '--clear', action='store_true', help='clear all history')
 
     history_format_group = history_parser.add_argument_group(title='formatting')
-    history_script_help = 'output commands in script format, i.e. without command numbers'
-    history_format_group.add_argument('-s', '--script', action='store_true', help=history_script_help)
-    history_expand_help = 'output expanded commands instead of entered command'
-    history_format_group.add_argument('-x', '--expanded', action='store_true', help=history_expand_help)
+    history_format_group.add_argument('-s', '--script', action='store_true',
+                                      help='output commands in script format, i.e. without command\n'
+                                           'numbers')
+    history_format_group.add_argument('-x', '--expanded', action='store_true',
+                                      help='output fully parsed commands with any aliases and\n'
+                                           'macros expanded, instead of typed commands')
     history_format_group.add_argument('-v', '--verbose', action='store_true',
-                                      help='display history and include expanded commands if they'
-                                           ' differ from the typed command')
+                                      help='display history and include expanded commands if they\n'
+                                           'differ from the typed command')
 
     history_arg_help = ("empty               all history items\n"
                         "a                   one history item by number\n"

cmd2/parsing.py

@@ -2,7 +2,6 @@
 # -*- coding: utf-8 -*-
 """Statement parsing classes for cmd2"""
 
-import os
 import re
 import shlex
 from typing import Dict, Iterable, List, Optional, Tuple, Union
@@ -160,13 +159,13 @@ def do_mycommand(stmt):
     # characters appearing after the terminator but before output redirection, if any
     suffix = attr.ib(default='', validator=attr.validators.instance_of(str))
 
-    # if output was piped to a shell command, the shell command as a list of tokens
-    pipe_to = attr.ib(default=attr.Factory(list), validator=attr.validators.instance_of(list))
+    # if output was piped to a shell command, the shell command as a string
+    pipe_to = attr.ib(default='', validator=attr.validators.instance_of(str))
 
     # if output was redirected, the redirection token, i.e. '>>'
     output = attr.ib(default='', validator=attr.validators.instance_of(str))
 
-    # if output was redirected, the destination file
+    # if output was redirected, the destination file token (quotes preserved)
     output_to = attr.ib(default='', validator=attr.validators.instance_of(str))
 
     def __new__(cls, value: object, *pos_args, **kw_args):
@@ -208,7 +207,7 @@ def post_command(self) -> str:
             rtn += ' ' + self.suffix
 
         if self.pipe_to:
-            rtn += ' | ' + ' '.join(self.pipe_to)
+            rtn += ' | ' + self.pipe_to
 
         if self.output:
             rtn += ' ' + self.output
@@ -453,56 +452,56 @@ def parse(self, line: str, expand: bool = True) -> Statement:
                 arg_list = tokens[1:]
                 tokens = []
 
-        # check for a pipe to a shell process
-        # if there is a pipe, everything after the pipe needs to be passed
-        # to the shell, even redirected output
-        # this allows '(Cmd) say hello | wc > countit.txt'
-        try:
-            # find the first pipe if it exists
-            pipe_pos = tokens.index(constants.REDIRECTION_PIPE)
-            # save everything after the first pipe as tokens
-            pipe_to = tokens[pipe_pos + 1:]
+        pipe_to = ''
+        output = ''
+        output_to = ''
 
-            for pos, cur_token in enumerate(pipe_to):
-                unquoted_token = utils.strip_quotes(cur_token)
-                pipe_to[pos] = os.path.expanduser(unquoted_token)
+        # Find which redirector character appears first in the command
+        try:
+            pipe_index = tokens.index(constants.REDIRECTION_PIPE)
+        except ValueError:
+            pipe_index = len(tokens)
 
-            # remove all the tokens after the pipe
-            tokens = tokens[:pipe_pos]
+        try:
+            redir_index = tokens.index(constants.REDIRECTION_OUTPUT)
         except ValueError:
-            # no pipe in the tokens
-            pipe_to = []
+            redir_index = len(tokens)
 
-        # check for output redirect
-        output = ''
-        output_to = ''
         try:
-            output_pos = tokens.index(constants.REDIRECTION_OUTPUT)
-            output = constants.REDIRECTION_OUTPUT
+            append_index = tokens.index(constants.REDIRECTION_APPEND)
+        except ValueError:
+            append_index = len(tokens)
 
-            # Check if we are redirecting to a file
-            if len(tokens) > output_pos + 1:
-                unquoted_path = utils.strip_quotes(tokens[output_pos + 1])
-                output_to = os.path.expanduser(unquoted_path)
+        # Check if output should be piped to a shell command
+        if pipe_index < redir_index and pipe_index < append_index:
 
-            # remove all the tokens after the output redirect
-            tokens = tokens[:output_pos]
-        except ValueError:
-            pass
+            # Get the tokens for the pipe command and expand ~ where needed
+            pipe_to_tokens = tokens[pipe_index + 1:]
+            utils.expand_user_in_tokens(pipe_to_tokens)
 
-        try:
-            output_pos = tokens.index(constants.REDIRECTION_APPEND)
-            output = constants.REDIRECTION_APPEND
+            # Build the pipe command line string
+            pipe_to = ' '.join(pipe_to_tokens)
+
+            # remove all the tokens after the pipe
+            tokens = tokens[:pipe_index]
+
+        # Check for output redirect/append
+        elif redir_index != append_index:
+            if redir_index < append_index:
+                output = constants.REDIRECTION_OUTPUT
+                output_index = redir_index
+            else:
+                output = constants.REDIRECTION_APPEND
+                output_index = append_index
 
             # Check if we are redirecting to a file
-            if len(tokens) > output_pos + 1:
-                unquoted_path = utils.strip_quotes(tokens[output_pos + 1])
-                output_to = os.path.expanduser(unquoted_path)
+            if len(tokens) > output_index + 1:
+                unquoted_path = utils.strip_quotes(tokens[output_index + 1])
+                if unquoted_path:
+                    output_to = utils.expand_user(tokens[output_index + 1])
 
-            # remove all tokens after the output redirect
-            tokens = tokens[:output_pos]
-        except ValueError:
-            pass
+            # remove all the tokens after the output redirect
+            tokens = tokens[:output_index]
 
         if terminator:
             # whatever is left is the suffix