Parallel load from a directory or a large file

[dhalperi]

Directory:

• some custom logic for each file (e.g., extract metadata from filename as new column). Should probably be just ability to append filename as column. E.g., NChilada data
• parallelize across files

File:

• need some ability to parallelize WITHIN file and implementation may vary based on file type and storage backend.
• specialization for files with fixed-width records

Remember: Often users want a counter for the records