Don't use internal APIs for GPU memory allocation

[manopapad]

Lux is currently using Realm::get_runtime() and Realm::RuntimeImpl::get_memory_impl from realm/runtime_impl.h, and Realm::Cuda::GPUFBMemory::alloc_bytes and Realm::Cuda::GPUFBMemory::get_direct_ptr from realm/cuda/cuda_module.h to allocate GPU memory that needs to persist past the lifetime of the allocating task (e.g. see https://github.com/LuxGraph/Lux/blob/master/pagerank/pagerank_gpu.cu#L269-L277). These are apparently Realm-internal interfaces, and instead the API in realm/instance.h, or possibly the new Legion variable-size allocation API should be preferred.

[jiazhihao]

@manopapad Thanks for the suggestion! Are you referring to the create_instance calls in realm/instance.h? I didn't pay special attention to the new variable-size allocation calls in Legion. Do you have a pointer to related documents/discussions?

[manopapad]

Are you referring to the create_instance calls in realm/instance.h?

That's the one. Based on the discussion at StanfordLegion/legion#973, it sounds like that's the "officially supported" function for doing what you need. No idea how you're supposed to use it though :-)

I didn't pay special attention to the new variable-size allocation calls in Legion. Do you have a pointer to related documents/discussions?

I don't think there's much documentation available, but this test showcases the API. Unfortunately it looks like it's only available on the control_replication branch.

[streichler]

@jiazhihao can you confirm which Realm interface(s) you're using for this? MemoryImpl::alloc_bytes has been gone for over a year now, and a change that should be merging into the master branch shortly moves Realm::Cuda::GPUFBMemory to a different header file, so if you are still directly using Realm for device memory allocation, it needs to be going through the public interface.

[jiazhihao]

We use Realm::RegionInstance::create_instance for allocating GPU memory inside a task. Will that still work after the merge? The code snippet is as follows.

  Rect1 bounds(Point1(0), Point1(0));
  std::vector<size_t> field_sizes;
  field_sizes.push_back(model->config.simulator_work_space_size);
  Realm::RegionInstance::create_instance(simulatorInst,
      memory, bounds, field_sizes, 0, Realm::ProfilingRequestSet()).wait();
  base_ptr = (char*)simulatorInst.pointer_untyped(0, sizeof(char));
  capacity = model->config.simulator_work_space_size;

[streichler]

Yes, that should still work fine. Thanks for checking.