Monitoring RushTI #124
Description
I recently posted a question regarding “Python/RushTI hanging.”
We have now identified that this “hanging” was caused by a lock preventing RushTI from authenticating via CAM.
As a result, everything that runs after RushTI also hangs, because the TI process is waiting for RushTI to complete, while RushTI is waiting for the lock to be released.
@MariusWirtz suggested using ExecuteCommand and passing 0 as the second argument. This would allow the subsequent TI processes to continue running after RushTI. However, this approach does not work for us, because once RushTI has finished, we want to validate, via TI, the file passed through --Result (formerly rushti.csv).
In the past, we experienced similar issues with our own Python-based report bursting application, where Excel would occasionally hang without any obvious reason.
To address this, we implemented a parallel monitoring process that watches the report bursting log file. If the log file is not updated within a defined interval, the process terminates Excel so that subsequent requests can continue. It also sends an email notification to an administrator group.
I believe a similar mechanism could be useful for RushTI. If it "hangs", it could be terminated automatically, and an error file (similar to rushti.csv) could be written. This file could then be processed by TI once the deadlock has been resolved.
There may be more robust ways to detect whether RushTI is hanging than simply monitoring the log file. In our locking case, however, we consistently saw the entry “RushTI starts.” in rushti.log, but no further log entries were written afterward.
What are your thoughts on this?