Add FetchDirectoryStream to save the extra GetTree call

[sluongng]

Currently in Remote Asset API, we have FetchDirectory with the following

  rpc FetchDirectory(FetchDirectoryRequest) returns (FetchDirectoryResponse) {
    option (google.api.http) = { post: "/v1/{instance_name=**}/assets:fetchDirectory" body: "*" };
  }

...

message FetchDirectoryResponse {
  google.rpc.Status status = 1;

  string uri = 2;

  repeated Qualifier qualifiers = 3;

  google.protobuf.Timestamp expires_at = 4;

  // The result of the fetch, if the status had code `OK`.
  // the root digest of a directory tree, suitable for fetching via
  // [ContentAddressableStorage.GetTree].
  build.bazel.remote.execution.v2.Digest root_directory_digest = 5;

  build.bazel.remote.execution.v2.DigestFunction.Value digest_function = 6;
}

Note how the client will have to call GetTree with the returned digest to get the whole tree, which is an additional round trip

  rpc GetTree(GetTreeRequest) returns (stream GetTreeResponse) {
    option (google.api.http) = { get: "/v2/{instance_name=**}/blobs/{root_digest.hash}/{root_digest.size_bytes}:getTree" };
  }

I wonder if we can simply return the GetTreeResponse, at least the first page, directly in FetchDirectoryResponse. That can simplify the call path for clients who are expecting a small and relatively shallow tree.

Otherwise, perhaps a FetchDirectoryStream variant that returns a stream of Directory messages of the whole tree would be nice to have as well.

[sluongng]

I know FetchDirectory is relatively unpopular, especially because Bazel does not use it today. But for those who are interested in/are using this API, I would love to hear your thoughts.

[juergbi]

If the tree is not yet cached locally, an extra round trip is also required for action results returned by Execute or GetActionResult. Do you see this as an issue for both, Remote Asset directories and action results? Or do you consider it an issue only for Remote Asset directories because FetchDirectory returns the root Directory digest and not a Tree digest?

If it's the former, should we try to come up with a solution that also works for the action cache?

If it's the latter, could it make sense to add an OutputDirectoryFormat field to FetchDirectoryRequest and a tree_digest to FetchDirectoryResponse, aligning with RE actions?