Store and Access Logfile Data

[samrose]

We need a way capture the stdin/out/err from runs, and store it in either local logfile, or to S3 compatible object store. Later on this will be used to show this data to users through the UI. For now, we just need to store and organize it by the id if a run in our db/datetime

[garrettmichaelgeorge]

@samrose would we want to pull out the stdin/out/err and store it separately from the rest of the run data structure, or keep it nested inside of it?

Currently the data structure of a run looks something like the following, with the command output captured inside of each %Step{} (of course pipelines should be a graph and not a list – this uses the old code):

iex(1)> Mix.Tasks.Ei.run config: "ei.json"

 ...
{:ok,
 %ExIntegrate.Core.Run{
   active_pipelines: [],
   pipelines: [
     %ExIntegrate.Core.Pipeline{
       completed_steps: [
         %ExIntegrate.Core.Step{
           args: ["compile"],
           command: "mix",
           err: "warning: variable \"config\" is unused (if the variable is not meant to be used, prefix it with an underscore)\n  lib/ex_integrate/pipeline_runner.ex:64: ExIntegrate.Boundary.PipelineRunner.handle_info/2\n\nwarning: variable \"config\" is unused (if the variable is not meant to be used, prefix it with an underscore)\n  lib/ex_integrate/pipeline_runner.ex:70: ExIntegrate.Boundary.PipelineRunner.handle_info/2\n\nwarning: ExIntegrate.Boundary.PipelineRunner.run_pipeline/1 is deprecated. It runs pipelines using an old, naive, sequential implementation. Use Elixir.ExIntegrate.Boundary.PipelineRunner.start_link/1 instead for the recommended concurrent implementation.\n\n  lib/ex_integrate.ex:43: ExIntegrate.run_pipelines/1\n\n",
           name: "compile",
           out: "Compiling 12 files (.ex)\nGenerated ex_integrate app\n",
           status_code: 0
         },
         %ExIntegrate.Core.Step{
           args: ["clean"],
           command: "mix",
           err: "",
           name: "clean",
           out: "",
           status_code: 0
         },
         %ExIntegrate.Core.Step{
           args: ["deps.get"],
           command: "mix",
           err: "",
           name: "get deps",
           out: "Resolving Hex dependencies...\nDependency resolution completed:\nUnchanged:\n  bunt 0.2.0\n  credo 1.6.1\n  file_system 0.2.10\n  jason 1.2.2\n  libgraph 0.13.3\n  rambo 0.3.4\nAll dependencies are up to date\n",
           status_code: 0
         }
       ],
       failed?: false,
       name: "exintegrate_build",
       steps: [
         %ExIntegrate.Core.Step{
           args: ["deps.get"],
           command: "mix",
           err: nil,
           name: "get deps",
           out: nil,
           status_code: nil
         },
         %ExIntegrate.Core.Step{
           args: ["clean"],
           command: "mix",
           err: nil,
           name: "clean",
           out: nil,
           status_code: nil
         },
         %ExIntegrate.Core.Step{
           args: ["compile"],
           command: "mix",
           err: nil,
           name: "compile",
           out: nil,
           status_code: nil
         }
       ]
     },
     %ExIntegrate.Core.Pipeline{
       completed_steps: [
         %ExIntegrate.Core.Step{
           args: ["test"],
           command: "mix",
           err: "warning: variable \"config\" is unused (if the variable is not meant to be used, prefix it with an underscore)\n  lib/ex_integrate/pipeline_runner.ex:64: ExIntegrate.Boundary.PipelineRunner.handle_info/2\n\nwarning: variable \"config\" is unused (if the variable is not meant to be used, prefix it with an underscore)\n  lib/ex_integrate/pipeline_runner.ex:70: ExIntegrate.Boundary.PipelineRunner.handle_info/2\n\nwarning: ExIntegrate.Boundary.PipelineRunner.run_pipeline/1 is deprecated. It runs pipelines using an old, naive, sequential implementation. Use Elixir.ExIntegrate.Boundary.PipelineRunner.start_link/1 instead for the recommended concurrent implementation.\n\n  lib/ex_integrate.ex:43: ExIntegrate.run_pipelines/1\n\nwarning: variable \"server\" is unused (if the variable is not meant to be used, prefix it with an underscore)\n  test/ex_integrate/run_manager_test.exs:22: ExIntegrate.RunManagerTest.\"test works\"/1\n\nwarning: variable \"second_pipeline\" is unused (if the variable is not meant to be used, prefix it with an underscore)\n  test/ex_integrate/run_test.exs:109: ExIntegrate.RunTest.\"test return the next pipelines to launch\"/1\n\nwarning: variable \"second_pipeline\" is unused (there is a variable with the same name in the context, use the pin operator (^) to match on it or prefix this variable with underscore if it is not meant to be used)\n  test/ex_integrate/run_test.exs:111: ExIntegrate.RunTest.\"test return the next pipelines to launch\"/1\n\nwarning: unused alias Pipeline\n  test/ex_integrate_test.exs:6\n\nwarning: unused alias Server\n  test/ex_integrate_test.exs:5\n\nwarning: unused alias Step\n  test/ex_integrate_test.exs:8\n\ncat: nonexistant: No such file or directory\n",
           name: "compile",
           out: "Compiling 12 files (.ex)\nGenerated ex_integrate app\n......................step 1\nstep 2\n.step 3\nstep 1\nstep 2\nstep 3\n.*....\n\nFinished in 0.2 seconds (0.1s async, 0.06s sync)\n29 tests, 0 failures, 1 skipped\n\nRandomized with seed 954778\n",
           status_code: 0
         }
       ],
       failed?: false,
       name: "exintegrate_test",
       steps: [
         %ExIntegrate.Core.Step{
           args: ["test"], 
           command: "mix",
           err: nil,
           name: "compile",
           out: nil,
           status_code: nil
         }
       ]
     }
   ]
 }}

[garrettmichaelgeorge]

This connects to what we discussed recently – that there are two potential sources of stdout/stderr output: Rambo's streamed output (side effect), and Rambo's function return.

Which one would be better to utilize for logging purposes?