Need ComprehensiveTableOperationsIT_SimpleSuite.test_compact() functionality

[kevinrr888]

Is your feature request related to a problem? Please describe.
The test was originally written as shown in additional context but was deleted. This test needs to test compactions on all the system tables, but the approach below is either flawed or is demonstrating a bug with the Manager.

Describe the solution you'd like
Need to determine if the issue is with the test itself or the Manager. If the issue is with the test itself, need to fix the test below or rewrite the test while still testing basic functionality of compactions on all system tables. If the issue is with the Manager, need to resolve the issue and hopefully can use the same test below.

Describe alternatives you've considered
N/A

Additional context
See #5474 (comment) which details the issue seen in the below test.

  @Test
  public void test_compact() throws Exception {
    // compact for user tables is tested in various ITs. One example is CompactionIT. Ensure
    // test exists
    assertDoesNotThrow(() -> Class.forName(CompactionIT.class.getName()));
    // disable the GC to prevent automatic compactions on METADATA and ROOT tables
    getCluster().getClusterControl().stopAllServers(ServerType.GARBAGE_COLLECTOR);
    try {
      // test basic functionality for system tables
      userTable = getUniqueNames(1)[0];
      ops.create(userTable);

      // create some RFiles for the METADATA and ROOT tables by creating some data in the user
      // table, flushing that table, then the METADATA table, then the ROOT table
      for (int i = 0; i < 3; i++) {
        try (var bw = client.createBatchWriter(userTable)) {
          var mut = new Mutation("r" + i);
          mut.put("cf", "cq", "v");
          bw.addMutation(mut);
        }
        ops.flush(userTable, null, null, true);
        ops.flush(SystemTables.METADATA.tableName(), null, null, true);
        ops.flush(SystemTables.ROOT.tableName(), null, null, true);
      }

      for (var sysTable : SystemTables.tableNames()) {
        // create some RFiles for FATE and SCAN_REF tables
        if (sysTable == SystemTables.SCAN_REF) {
          createScanRefTableRow();
          ops.flush(SystemTables.SCAN_REF.tableName(), null, null, true);
        } else if (sysTable == SystemTables.FATE) {
          createFateTableRow(userTable);
          ops.flush(SystemTables.FATE.tableName(), null, null, true);
        }

        Set<StoredTabletFile> stfsBeforeCompact = getStoredTabFiles(sysTable);

        log.info("Compacting " + sysTable);
        ops.compact(sysTable.tableName(), null, null, true, true);
        log.info("Finished compacting " + sysTable);

        // RFiles resulting from a compaction begin with 'A'. Wait until we see an RFile beginning
        // with 'A' that was not present before the compaction.
        Wait.waitFor(() -> {
          var stfsAfterCompact = getStoredTabFiles(sysTable);
          String regex = "^A.*\\.rf$";
          var A_stfsBeforeCompaction = stfsBeforeCompact.stream()
              .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
          var A_stfsAfterCompaction = stfsAfterCompact.stream()
              .filter(stf -> stf.getFileName().matches(regex)).collect(Collectors.toSet());
          return !Sets.difference(A_stfsAfterCompaction, A_stfsBeforeCompaction).isEmpty();
        });
      }
    } finally {
      getCluster().getClusterControl().startAllServers(ServerType.GARBAGE_COLLECTOR);
    }
  }

[ArbaazKhan1]

I can take a look at this

[ArbaazKhan1]

I'm running into two issues while running test_compacton system tables.

1. In its current form the test uses a slow iterator, which gets stuck in CompcationDriver.isReadyrather than proceeding to call() as expected. This unexpectedly causes other system table compactions to hang as well.
2. writing the test without slow iterator or using a low sleep time it seems compactions complete but don't generate 'A' files for the FATE table, causing the test to time out in Wait.waitFor()

It seems the FATE table might have special compaction criteria that aren't being met in the test. What's the recommended approach here? Any guidance on properly testing system table compactions would be appreciated.

[kevinrr888]

1. Need to have data in the FATE table when the flush() occurs. This will write an RFile for the FATE table
2. Then, compact(SystemTables.FATE) should compact that file giving the 'A' file

Are you sure when you are using the slow iterator that data exists in the FATE table when the flush occurs?

[kevinrr888]

#5798 is merged and introduced a new way to have a slow running FATE operation in the Manager. I think this would help in this issue. Instead of attaching a slow iterator and compacting a user table, could use the new SlowFateSplitManager and add splits to a user table. This split operation will take >= 10sec to complete. Can flush FATE table while this is running.

This doesn't completely answer why compactions on other tables were affected by the slow compaction on the user table, but maybe that can be a new issue.

[kevinrr888]

This doesn't completely answer why compactions on other tables were affected by the slow compaction on the user table, but maybe that can be a new issue.

Test needed two compactors instead of the default of one. Resolved in #5881