NPD customPluginMonitor stuck at waiting for stdoutPipe/stdoutPipe to complete if the plugin process hung in D state

[wenzt]

The customer plugin may end up in the D (uninterruptible) state. NPD customPluginMonitor stuck at waiting for stdoutPipe/stdoutPipe and no more rules will be executed.

[googs1025]

/assign

maybe I can take a look this😄

[googs1025]

after checking code: we already have timeout config.
maybe you can try this

node-problem-detector/config/custom-plugin-monitor.json

Line 5 in 308b7cf

"timeout": "5s",

[jessehu]

hi @googs1025，we tested in local env to simulate a process hung in D state, which is launched by NPD custom plugins module. We found that NPD customPluginMonitor stuck at waiting for stdoutPipe/stdoutPipe to complete: https://github.com/kubernetes/node-problem-detector/blob/master/pkg/custompluginmonitor/plugin/plugin.go#L212-L223. The timeout you mentioned only triggerd util.Kill(cmd).

We figured out a solution like this:

1. Run wg.Wait() in a go routine, and use the plugin exection timeout to detect if it's timed out.
2. Run util.Kill(cmd) after timed out and removed the original logic for running util.Kill(cmd).

	wg.Add(2)
	go func() {
		defer wg.Done()
		stdout, stdoutErr = readFromReader(stdoutPipe, maxCustomPluginBufferBytes)
	}()
	go func() {
		defer wg.Done()
		stderr, stderrErr = readFromReader(stderrPipe, maxCustomPluginBufferBytes)
	}()
	// This will wait for the reads to complete. If the execution times out, the pipes
	// will be closed and the wait group unblocks.
	waitChan := make(chan struct{})
	go func() {
		wg.Wait()
		close(waitChan)
	}()
	select {
	case <-waitChan:
		break
	case <-time.After(timeoutDuration):
		glog.Errorf("Waiting for command output timed out when running plugin %q", rule.Path)
		isTimeout = true
		err := util.Kill(cmd)
		if err != nil {
			glog.Errorf("Error when killing process %d: %v", cmd.Process.Pid, err)
		}

		// Check if the process is in D state. If it is, the process is hung and can not be killed.
                // TODO Fix: If the sub process created by the plugin bash script is in D state, the cmd.Process.Pid is not in D state and can be killed successfully.
		if util.IsProcessInDState(cmd.Process.Pid) {
			glog.Errorf("Process %d is hung in D state", cmd.Process.Pid)
			isHung = true
		}
	}

	if isHung {
		return cpmtypes.Unknown, fmt.Sprintf("Process is hung when running plugin %s", rule.Path)
	}

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten