[CARBONDATA-4300] Clean files command supports specify segment ids

[jack86596]

Why is this PR needed?

Currently clean files command will delete all the Marked for Delete and Compacted segments after the number of theses segments reaches carbon.invisible.segments.preserve.count, this delete operation may take lots of time and user cannot decide to only delete some of these segments. It is better to enhance clean files command to allow specify the segments to be deleted.

What changes were proposed in this PR?

1. Clean files command supports specify segment ids, syntax is "clean files for table table_name options("segment_ids"="id1,id2,id3...")". If specified segment ids, then only the segment with these ids will be delete physically.
2. Refactoring lock taken: during clean files, take the tablestatus lock at the begining and release the lock at the end, and during lock taken period, only read tablestatus file one time(before there could be 10+) and all operations are done on it like change the visibility of segment, move visibility = false segment to tablestatus.history file.

Does this PR introduce any user interface change?

• Yes. One more option is added for clean files command: segment_ids. Value is the segment ids user wants to delete. Only Marked for Delete and Compacted segment ids are valid. If invalid ids are given, operation will fail directly. If segments are specified, force option will be ignored.
  CLEAN FILES FOR TABLE TABLE_NAME options('segment_ids'='0,1,2')

Is any new testcase added?

• Yes

[CarbonDataQA2]

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6042/

[CarbonDataQA2]

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/432/

[CarbonDataQA2]

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4298/

[CarbonDataQA2]

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6043/

[CarbonDataQA2]

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4299/

[CarbonDataQA2]

Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/433/

[jack86596]

retest this please

[CarbonDataQA2]

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6044/

[CarbonDataQA2]

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4300/

[CarbonDataQA2]

Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/434/

[akashrn5]

@jack86596 this is behavioral and functional change. So instead of directly raising PR with more code changes, better to first raise discussion in community and take the inputs and then do the changes accordingly.

[jack86596]

OK, i will raise a discussion in the mail list.