RejectedExecutionException: Task org.apache.accumulo.core.trace.TraceWrappedRunnable

[milleruntime]

Saw this while running a single RW Bulk test.

2022-05-02T16:11:49,182 [testing.randomwalk.Framework] ERROR: Error during random walk
java.lang.Exception: Error running node bulk.BulkPlusOne
        at org.apache.accumulo.testing.randomwalk.Module.visit(Module.java:364) ~[accumulo-testing-shaded.jar:2.1.0-SNAPSHOT]
        at org.apache.accumulo.testing.randomwalk.Framework.run(Framework.java:54) ~[accumulo-testing-shaded.jar:2.1.0-SNAPSHOT]
        at org.apache.accumulo.testing.randomwalk.Framework.main(Framework.java:98) ~[accumulo-testing-shaded.jar:2.1.0-SNAPSHOT]
Caused by: java.util.concurrent.RejectedExecutionException: Task org.apache.accumulo.core.trace.TraceWrappedRunnable@3c12d28d rejected from org.apache.accumulo.core.util.threads.ThreadPools$2@14bca5e3[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2446]
        at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2055) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:825) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1355) ~[?:?]
        at org.apache.accumulo.core.util.threads.ThreadPools$2.execute(ThreadPools.java:479) ~[accumulo-testing-shaded.jar:2.1.0-SNAPSHOT]
        at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118) ~[?:?]
        at org.apache.accumulo.core.util.threads.ThreadPools$2.submit(ThreadPools.java:499) ~[accumulo-testing-shaded.jar:2.1.0-SNAPSHOT]
        at org.apache.accumulo.testing.randomwalk.bulk.Setup.run(Setup.java:77) ~[accumulo-testing-shaded.jar:2.1.0-SNAPSHOT]
        at org.apache.accumulo.testing.randomwalk.bulk.BulkTest.visit(BulkTest.java:29) ~[accumulo-testing-shaded.jar:2.1.0-SNAPSHOT]
        at org.apache.accumulo.testing.randomwalk.bulk.BulkImportTest.visit(BulkImportTest.java:72) ~[accumulo-testing-shaded.jar:2.1.0-SNAPSHOT]
        at org.apache.accumulo.testing.randomwalk.Module.lambda$visit$0(Module.java:299) ~[accumulo-testing-shaded.jar:2.1.0-SNAPSHOT]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
        at org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52) ~[accumulo-testing-shaded.jar:2.1.0-SNAPSHOT]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
        at org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52) ~[accumulo-testing-shaded.jar:2.1.0-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
        at org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52) ~[accumulo-testing-shaded.jar:2.1.0-SNAPSHOT]
        at java.lang.Thread.run(Thread.java:829) ~[?:?]

[milleruntime]

@ctubbsii I think this is similar to the problem you fixed in Accumulo main in apache/accumulo#2442

[dlmarion]

Caused by: java.util.concurrent.RejectedExecutionException: Task org.apache.accumulo.core.trace.TraceWrappedRunnable@3c12d28d rejected from org.apache.accumulo.core.util.threads.ThreadPools$2@14bca5e3[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2446]

This looks to me like the task was added after the ExecutorService was shut down. I haven't looked at the code to see if that's actually possible.

[ctubbsii]

@ctubbsii I think this is similar to the problem you fixed in Accumulo main in apache/accumulo#2442

I don't think so. In that situation, we were making incorrect assumptions about the actual sub-class of Runnable inside the executor service's queues.

It looks like this is executing the Runnable (in this case, it's a TraceWrappedRunnable) inside an executor service that has terminated. The runnable seems to be the RW module, Module.java's FutureTask created on line 297.