MdfReader::ReadPartialData doesn't always read data when encountering DZ blocks

[chrisblaylock]

I need to determine the timespan of the data, which requires reading the first and last samples of each time channel. The first sample is fine, but calling ReadPartialData fails for the final sample as it lives in a DZ block, and ReadCache doesn't cope with skipping bytes in these blocks, as it assumes if file_buffer_ isn't empty, that it has been populated, so you just get 0 values. I have a WIP fix for this that I aim to create a PR for once I've finished the rest of my implementation.

[ihedvall]

@chrisblaylock
Thanks for finding bugs in the library.
Just for your information, the issue of finding the time range have been discussed earlier in
Reading start and end time.

This implementation of a MDF file reader, parses directly from the disc (NAS).
Some implementations read in the entire file into primary memory and then parses the MDF file.
The latter approach might be faster but fails with large files.

The MDF file have a very complex way of storing sample data (records).
In the MDF version 3 all samples where stored in one DT block.
In the MDF version 4, this DT block can be split into a list of DT blocks that in turn can be compressed into DZ blocks.
The Read Cache job is to keep track of where in the list your are.
The cache first read in the record ID which identify the CG block in use.
Then it either skip or read in the record data bytes.
When reading partial read the skipping of bytes is much faster than reading in the record bytes.
Well, at least for DT blocks but for compressed DZ block, the reader needs to uncompressed the DZ block why the skipping of bytes ain't going to be faster.

I have a use case when inserting MDF files into an ASAM ODS index database (odslib).
The database doesn't store the sample data but it should store min, max and average values for each channel.
The database also store so-called meta data as information about the test object.
There might be better to do a special observer function that just store the 3 above values.
For a master channel, the min and max values should be what you want.

Note, that fetching the last sample requires to read the entire data blob as you don't know where the last record is stored.
Call ReadPartial() function twice is most likely slower or similar than reading in all compressed samples.

I normally first makes a new unit test that detects the bug before I do any changes.
Then I do the changes and that verify the changes.
Bugs have a nasty habit of returning later on.
Please let me know if you want me to add a unit test.

[chrisblaylock]

Sorry for the delayed response, I've been working on my local implementation.
A unit test would be great - essentially if you create 2 DZ blocks e.g. [1,2,3,4], [5,6,7,8] then try to read the first sample from the first, then the last sample from the second, you'll end up with 0 as the value for the second (as it will have allocated the array, but not read from disk).

[ihedvall]

@chrisblaylock
I have added a unit test that reads the first and last time by using the ReadPartialData() function.
I scan through all of my (188) test files.
The unit test fails for a handful of the files and the common is that they have a single DZ block as data.
It fails on the "last time >= first time" test.

I noted that the test runs faster if I read in all time samples (ReadData()) instead of just the first and last.
I think that reading the last sample require to read pretty much the whole data block.

There are some issues if min and max index is equal. ReadPartialData(..,0,0) will fail.

  TEST_F(TestRead, TestPartial) {
  if (kMdfList.empty()) {
    GTEST_SKIP_("No MDF files.");
  }
  size_t read_count = 0;
  for (const auto& [name, test_file]: kMdfList) {
    MdfReader reader(test_file);
    const bool read_config = reader.ReadEverythingButData();
    ASSERT_TRUE(read_config) << name;

    const MdfFile* mf4_file = reader.GetFile();
    ASSERT_TRUE(mf4_file != nullptr);

    std::vector<IDataGroup*> dg_list;
    mf4_file->DataGroups(dg_list);
    for (IDataGroup* data_group : dg_list) {
      if (data_group == nullptr) {
        continue;
      }

      for (auto* channel_group : data_group->ChannelGroups()) {
        if (channel_group == nullptr) {
          continue;
        }
        ChannelObserverList observers;
        for (auto* channel : channel_group->Channels()) {
          if (channel == nullptr || channel->Type() != ChannelType::Master
            || channel->Sync() != ChannelSyncType::Time) {
            continue;
          }
          observers.emplace_back(CreateChannelObserver(*data_group, *channel_group, *channel));
        }
        if (observers.empty() || channel_group->NofSamples() == 0) {
          continue;
        }
        // Read out firat sample and last sample
        const uint64_t nof_samples = channel_group->NofSamples();
        // reader.ReadData(*data_group);
        reader.ReadPartialData(*data_group, 0, 1);
        reader.ReadPartialData(*data_group, nof_samples - 1, nof_samples);
        for (auto& observer : observers) {
          EXPECT_EQ(observer->NofSamples(), nof_samples) << name;
          double first = 0.0;
          double last = 0.0;
          bool valid = observer->GetEngValue(0,first);
          EXPECT_TRUE(valid) << "First: " << name;
          if (first > 1E10) {
            continue; // Most likely an invalid first time
          }
          valid = observer->GetEngValue(nof_samples -1,last);
          EXPECT_TRUE(valid) << "Last: " << name;
          EXPECT_GE(last, first) << "CG: " << channel_group->Name()
            << ", File: " << test_file;
          //std::cout << "File: " << name << ", Channel: " << observer->Name()
          //  << ", First [s] = " << first
          //  << ", Last [s] = " << last
          //<< std::endl;
        }
      }
    }

    ++read_count;
  }
  std::cout << "Tested " << read_count << " files." << std::endl;
}

[ihedvall]

@chrisblaylock
I have made a change so the DZ blocks can handle partial read correctly.
Error in the SkipByte(s) function.
Please check if this solves the partial read problem with your MDF files.