Support automatic deduplication of data in copy() and move() #317
Description
Currently we just copy or move the data to the specified table in the specified data folder without handling deduplication of the copied/moved data. This is done due to issues with the merge method in Delta Lake, which caused the data to not be copied when using when_matched_delete and caused data ordering issues when using when_not_matched_insert.
We can either always deduplicate the data or add an argument that allows the user to control whether data should be deduplicated.