Deserialization error for some querysets

[Peder2911]

Me and @jimdale found an issue where viewser would raise a deserialization
error, while there was obviously at least partial Parquet bytes data in the
response:

DeserializationError: DeserializationError:

  Description:
                Could not deserialize as parquet:             "b'PAR1\x15\x04\
                x15\xe0D\x15\xf8?L\x15\xcc\x08\x15\x04\x12\x00\x00\x1f\x8b\x08
                \x00\x00\x00\x00\x00\x00\x03-Wi8Vk\x1b5e\x1e\xdei\x8f\xafY*\x9
                1\xc21'..."

This only seems to happen with certain querysets. The queryset that lead to this error was:

queryset = (Queryset("jim_fatalities_conflict_history_lag_tdecay", "priogrid_month")
 
            # target variable
            .with_column(Column("ln_ged_sb", from_table = "ged2_pgm", from_column = "ged_sb_best_sum_nokgi")
                         .transform.missing.fill()
                         .transform.ops.ln()
                        )
            
            # spatial-tree-lagged d^-2 target variable
             .with_column(Column("ln_ged_sb_treelag_2_th1_0", from_table = "ged2_pgm", from_column = "ged_sb_best_sum_nokgi")
                         .transform.missing.fill()
                         .transform.ops.ln()
                         .transform.spatial.treelag(thetacrit_tree,2)
                        )
            
            # 1 tlagged spatial-tree-lagged d^-2 target variable
             .with_column(Column("ln_ged_tlag_1_sb_treelag_2_th1_0", from_table = "ged2_pgm", from_column = "ged_sb_best_sum_nokgi")
                         .transform.missing.fill()
                         .transform.ops.ln()
                         .transform.spatial.treelag(thetacrit_tree,2)
                         .transform.temporal.tlag(1)
                         .transform.missing.fill()
                        )
            
            # spatial-tree-lagged d^-1 target variable
             .with_column(Column("ln_ged_sb_treelag_1_th1_0", from_table = "ged2_pgm", from_column = "ged_sb_best_sum_nokgi")
                         .transform.missing.fill()
                         .transform.ops.ln()
                         .transform.spatial.treelag(thetacrit_tree,1)
                        )
            
            # 1 tlagged spatial-tree-lagged d^-1 target variable
             .with_column(Column("ln_ged_tlag_1_sb_treelag_1_th1_0", from_table = "ged2_pgm", from_column = "ged_sb_best_sum_nokgi")
                         .transform.missing.fill()
                         .transform.ops.ln()
                         .transform.spatial.treelag(thetacrit_tree,1)
                         .transform.temporal.tlag(1)
                         .transform.missing.fill()
                        )
            
            # spatial-tree-lagged ln(1+d) target variable
             .with_column(Column("ln_ged_sb_treelag_0_th1_0", from_table = "ged2_pgm", from_column = "ged_sb_best_sum_nokgi")
                         .transform.missing.fill()
                         .transform.ops.ln()
                         .transform.spatial.treelag(thetacrit_tree,0)
                        )
            
            # 1 tlagged spatial-tree-lagged ln(1+d) target variable
             .with_column(Column("ln_ged_tlag_1_sb_treelag_0_th1_0", from_table = "ged2_pgm", from_column = "ged_sb_best_sum_nokgi")
                         .transform.missing.fill()
                         .transform.ops.ln()
                         .transform.spatial.treelag(thetacrit_tree,0)
                         .transform.temporal.tlag(1)
                         .transform.missing.fill()
                        )
             )

To begin diagnosing this, we need to write some tooling for dumping the
erroneous response data to figure out what is being returned that is not
deserializable. This will give us a clue about whether or not the issue is
being caused by something upstream, or is caused by some issue with
deserialization.

A clue is that there is no exception happening upstream, which means that the
data is written to parquet and sent away just fine. This hints towards there
being something wrong with viewser.

[Peder2911]

This seems to have something to do with our current network topology, as no issue has been found with viewser. We have inspected dumped data from when this issue occurs, and there is no exception, nor any deserialization malfunctions in viewser. The data is simply incomplete out of transfer, which again seems to suggest that something interrupts the connection mid-transfer.

It will be interesting to see if this issue persists with our new servers.