Skip to content

Appended numeric type validation #294

@james-cockayne

Description

@james-cockayne

Hi,

I'm fairly new to DuckDB, so please correct me if this request is not sensible.

We had an issue where we were seeing strange data corruption issues when processing our data, which was fairly tricky to pin down. In the end the fix seemed to be casting our .net decimals to floats before appending to our table (which expected a float). answerdigital/oxford-omop-data-mapper@f408c17

I'll admit I did see this warning banner beforehand, but the code seemed to work at the time. It turns out that mostly we insert nulls, so the problem hardly ever happens.

Image

I can now see that .net decimals are 8 bytes, and the float duckdb type is 4 bytes, so it would have never worked if I understand it correctly.

Is there a reason why we can't guard against this kind of fault? I'm happy to put a change in if you think this makes sense.

It looks as though NumericVectorDataWriter.AppendNumeric<T> has everything we would need to test the type of the candidate value against the column's type ColumnType property.

James

Metadata

Metadata

Assignees

Labels

No labels
No labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions