rapids-5129

[amahussein]

Draft PR to share progress integrating Rapids with Spark-History-Server

[amahussein]

This is just a hack with minimum changes to the Profiler tool.

• When the Rapids-Tab gets loaded, a REST call pull the logLines and initialize a new AppInfo (see code-block)
• to avoid making changes to the Rapids tool, I created RapidsAppInfoLogProcessor which extends ApplicationInfo. The new class was created to separate between the initialization of the instance and the processing phase.
• Currently, the log will be processed everytime the tab is loaded.
• Compare mode is not supported yet.

ToDo:

• Initialize the EventProcessor in the Plugin initialization. This will allow the events to be processed during the replay of the log
• the above step requires some modifications in both ApplicationInfo and EventsProcessor to enable initialization of the EventProcessor before an App gets created.
• Accessing the output of the profile inside the HTML was not very trivial. ApplicationSummaryInfo is not friendly to read the values back. We will need to have a common interface Renderer that will support both txtFormat and Html/JS code generation
• The output of the profiler needs to be saved to the SHS store. This will guarantee that the state of the profiling gets saved correctly, and that the data will be shared across the pages.
• Profiler currently reads the arguments from command line. We can switch to look into getting the information from the configurations passed to the SHS.

[amahussein]

---

Below is the UI after adding JobInfo. Note that the empty columns are due to Started/Completed not profiled by the PRofiler tool yet.
AppIndex column is hidden. I will only enable it in the compare mode.
For JobInfo table, I explored grouping rows together. For example, instead of displaying each stage like [stage1, stage2] (which is the way SparkUI displays them). In this version, I display each stageID in a separate row.
There are lots of fancier way to display "grouping" but I did this simple form for a demo purpose.

[amahussein]

The compare table lists all the tables

---

The results are displayed in the results page