Serializing an EXTERNALSXP can lead to an endless loop

[rihafilip]

When a WriteItem is called with an EXTERNALSXP that is not Code, Function or DispatchTable, it ends up in an endless loop.

WriteItem calls to an externalCodeWrite function pointer, which is set to serializeRir. Then it calls WriteItem with the result of rirDecompile.

Currently, the rirDecompile function only deserializes Code, Function and DispatchTable types. For all others it returns the SEXP unmodified.

Hence the second call to WriteItem is the same as the first one and it loops endlesly.

[rihafilip]

Reproducing this issue is a bit trickier, easiest way I found was with the recording tool: triggering a EnvStubMaterialized deopt, which has a LazyEnvironment as a trigger, creates a new event with this trigger saved and at the end of execution forces a LazyEnvironment to be serialized, which leads to an endless loop described above.

Smallest example I've been able to find:

RIR_RECORD_SERIALIZE=1 RIR_RECORD=/tmp/rec.rds RIR_RECORD_FILTER=Deopt R -e "library(ggplot2)"

By putting an assert(false) on line 18 in interpreter/decompile.cpp, the code fails faster on the incorrect inputs.