Add in-memory data support for all formats #24
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add In-Memory File Support
Summary
This PR adds support for opening tabular data files directly from memory, enabling processing of files from HTTP responses, embedded data, and other sources without writing to disk.
Motivation
Currently,
grateonly supports opening files from the filesystem usinggrate.Open(filename). This limitation requires users to:This PR addresses these limitations by adding in-memory support across all file formats.
Changes
New Public API
grate.OpenBytes(data []byte) (Source, error)Sourceinterface asOpen()grate.RegisterWithBytes(name, priority, opener, openerBytes)Register()Implementation Details
Core Package (
grate.go)OpenBytesFunctype for format handlersOpenBytes()andOpenReader()functionsXLS Package (
xls/xls.go,xls/cfb/interface.go)OpenBytes()functionopenWorkbook()helper to eliminate duplicationcfb.OpenBytes()for Compound File Binary supportXLSX Package (
xlsx/xlsx.go)OpenBytes()functionparseDocument()helper to eliminate duplicationClose()to handle nil file handleSimple Package (
simple/csv.go,simple/tsv.go)OpenCSVBytes()andOpenTSVBytes()functionsparseCSV()andparseTSV()helpersTesting
inmemory_test.go:TestOpenBytes- validates all formats from byte slicesexamples/example_inmemory.goDocumentation
OpenBytes()Usage Examples
From Byte Slice
From Embedded File
Breaking Changes
None. This PR is fully backward compatible:
grate.Open(filename)code works unchangedPerformance Considerations