Add RequiredParam<T> and RequiredResult<T> to mark Object * arguments and return values as required

[dsnopek]

Implements proposal godotengine/godot-proposals#2241

The core problem that this is solving is that in some languages, different language constructs/features are used with nullable and non-nullable values, where the non-nullable ones are usually friendlier for the developer to use. This has come up numerous times specifically in the context of Rust, Swift and C#, but would also apply to Kotlin and probably others.

The goal of this PR is to mark various arguments and return values in extension_api.json as "required", which means:

1. Passing null for a required parameter is an error, which will prevent the method from doing its intended function, and so the GDExtension binding can attempt to prevent passing null into that argument using language features. (But this isn't required.)
2. Returning null for a required return value is an error, and the GDExtension language binding can use that as a signal to do whatever error behavior makes the most sense for that language, for example, throwing an exception (ex in Python, C#, etc) or panicking (ex Rust). (But this isn't required.)

How this is accomplished:

• Instead of using Object * or const Ref<T> & in method arguments, you can use const RequiredParam<T> &, which will get the parameter marked as "required" in ClassDB.
• Some new error macros are added, like EXTRACT_REQUIRED_PARAM_OR_FAIL(m_name, m_param) which acts just like ERR_FAILED_NULL(m_param), except it also assigns a local variable with the object, if it wasn't null. Using the macro is required in order to get the underlying value out of the parameter, which has two advantages:
  • It will help prevent contributors from using RequiredParam<T> when they don't really mean it, so (hopefully) these markers are less likely to become out-of-date
  • This will help contributors to remember to check for null (like we do currently do with the ERR_* macros) which is beneficial to engine development, even outside the context of GDExtension
• On return values, instead of using RequiredParam<T> there is RequiredResult<T>. This separate object is used because return values are more of a "soft" requirement and don't need a special process in order to extract the value
• In ClassDB, the meta for any of required parameters or return values will be METADATA_OBJECT_IS_REQUIRED, which will look like this in the extension_api.json:

  "arguments": [
    {
      "name": "node",
      "type": "Node",
      "meta": "required"
    }
  ]
  

• Just to test, this PR has a 2nd commit which marks a bunch of parameters and return values as required
• In the parameters, I've used a new naming convention of rp_* (rather than p_*) in order to minimize the diff when switching to RequiredParam<T>. This makes it so each change is just +/- 2 lines in the diff, and it's easy to review and see that there were no other functional changes
• When compiling a release build, the goal is for RequiredParam<T> and RequiredResult<T> to be optimized away, and generate basically the same binary as using a plain Object * or Ref<T>.

UPDATE (2024-12-10): The original version of this had this whole auto-vivifying thing that was terrible, and was replaced with an excellent suggestion from @Bromeon in this comment.

UPDATE (2025-10-17): The semantics of parameters and return values turned out to be different enough that this now has two templates (RequiredParam<T> and RequiredResult<T> instead of the single RequiredPtr<T>).

[AThousandShips]

One option would be to have this requirement be strict and crash on failure to fulfil it, I feel this is the area where that'd be appropriate, as if it's not so strict an error print should be used

At least I think such an option for this decoration feels appropriate, if not for the complete case

[dsnopek]

And code that works with RequiredPtr<T> will still need to use the error macros like ERR_FAIL_NULL(), because those will be able to make an error message with the file and line number of where the problem is. The error message when creating a RequiredPtr<T> has no idea what called it, again, because we don't have exceptions so we can't unwind the stack.

Actually, I think I may be wrong about this part!

With gcc/clang there are the backtrace() and backtrace_symbols() functions that may be able to get this information even when compiled with -fno-exceptions. We'd probably need to do this in a couple of different ways for different compilers and platforms, but it does seem to be possible.

EDIT: Heh, and we're even using them in Godot for the crash handler already. :-)

[Bromeon]

Very cool, and that was fast! 😎

---

Presently, it's marked by using METADATA_OBJECT_IS_REQUIRED in the same metadata field that holds things like METADATA_INT_IS_UINT32, which means it'll end up in the extension_api.json like this, for example:

"arguments": [
  {
    "name": "node",
    "type": "Node",
    "meta": "required"
  }
]

Could these metadata flags potentially overlap (maybe not now with the ints, but as we add more over time)?
If yes, how would they be represented in the JSON value for the meta key?

---

• If nullptr is passed into a RequiredPtr<T>, then it'll automatically create a new instance of the object in order to satisfy the "contract" (aka "auto-vivifying"), and print an error. I'm not sure this is the right thing to do. Since we don't have exceptions, we can't really do anything smarter at runtime. And code that works with RequiredPtr<T> will still need to use the error macros like ERR_FAIL_NULL(), because those will be able to make an error message with the file and line number of where the problem is. The error message when creating a RequiredPtr<T> has no idea what called it, again, because we don't have exceptions so we can't unwind the stack.

I'm probably missing something here, but if the expected usage is to always abort the function (i.e. print error + return null) after a "required" precondition fails, do we actually need to create a default instance?

Not too worried about performance in the error case, but some custom classes might have side effects in their constructors?

(Edit: since we're talking about the engine API here, there are probably no custom classes involved; side effects might still apply to some Godot classes though.)

[dsnopek]

@Bromeon:

Could these metadata flags potentially overlap (maybe not now with the ints, but as we add more over time)?
If yes, how would they be represented in the JSON value for the meta key?

I really have no idea what metadata could be added in the future. But like I said in the description, something more extensible like the the options you described in godotengine/godot-proposals#2550 would probably be better.

I'm probably missing something here, but if the expected usage is to always abort the function (i.e. print error + return null) after a "required" precondition fails, do we actually need to create a default instance?

Nope, you're not missing anything, we don't need to create a default instance. I added that because I was worried that engine contributors could mistakenly believe that RequiredPtr<T> somehow guarantees that the value is not null, and this would prevent crashes in that case. We could just trust that contributors will know they need to still check for null? Or perhaps (as @AThousandShips writes above) a crash is actually what we want is this case?

I'm really not sure what the right behavior is, so that's what I'm hoping to discuss here :-)

[AThousandShips]

I'd say it depends on how we interpret the argument limitation, is it shouldn't or can't be null?

From the proposal:

Add a way to hint specific method arguments as being nullable. In other words, it means that the method will behave correctly if the specified arguments are null. If this hint is not present, the argument is assumed to be non-nullable.

One use for this would be to remove checks, at least in non-debug builds, the annotation states that "the responsibility of ensuring this is non-null is on the caller" and no checks needs to be done on the callee side

In the same sense the check can be used before calling any extension methods or before calling anything from an extension, and exiting there, acting as a barrier

I'd prefer it to be a guarantee for the method implementer, a "I can safely skip doing any null checks in this method, someone else handles it for me", or "this can safely be optimized in some sense", akin to the LLVM nonnull, as an optimization hint

I'm also not familiar with what the other languages here do for this, so matching some norm there would make sense

[Bromeon]

I really have no idea what metadata could be added in the future. But like I said in the description, something more extensible like the the options you described in godotengine/godot-proposals#2550 would probably be better.

It's hard to plan ahead here. Most flexible would probably be a dictionary (JSON object), but it might also be overkill?

{
    // ...
    "metadata": {
        "exact_type": "float64",
        "angle": "radian",
        "nullable": true
    }
}

Also, we will need to keep the "meta" field around anyway, for backwards compatibility. If nullability and other meta types don't overlap, we could technically reuse that key. But are we sure that nullability isn't something we might at some point consider for non-object parameters, as well?

---

What I'm a bit worried is that (in a few years), this might devolve into:

{
    // ...
    "meta": "float64|angle=radian|nullable"
}

That is, we have a "protocol inside a protocol" -- which would also be breaking btw, since existing code likely checks for the entire value string and would not expect any separators.

---

In other words, we have multiple options:

1. reuse the meta field and either
   • hope potential values will stay exclusive, even in the future
   • keep using it only for those values which are exclusive, add new fields like angle (just an example) for specific metadata
2. keep using meta only for the existing meaning.
   add a new key nullable (or maybe rather required, since it's new) on the same level
3. design something more future-proof like a metadata object or array, meaning
   • we have to provide both representations due to backwards compatibility
   • if we choose object instead of array, technically it's similar to today's meta key, but with 1 extra level of nesting

I tend to favor 1) or 2), but maybe you have more thoughts on this?

[Bromeon]

I'm also not familiar with what the other languages here do for this, so matching some norm there would make sense

There are many examples of languages that didn't have a concept of nullability/optionality and retrofitted it late. Some I know:

1. Java: everything was nullable by default
   • @NonNull annotations to give stronger guarantees, @Nullable to confirm it's indeed nullable
   • Optional<T> which is a competing mechanism, more FP but semantically the same as the annotations
   • annoyingly, a simple T can still mean "possibly nullable or not, depending on API design"
2. PHP 7.1 introduced ?Type syntax
   • The good thing is that Type was strictly non-nullable before, so it was an easy addition
3. C++ with std::optional
   • This is probably the worst example... Because it came so late, everyone already had their own workarounds (null pointers, bool flags, hand-written optionals, unions, ...). Including the standard library.
   • Changing T to optional<T> is possible due to implicit conversions, but being C++, there's probably a ton of edge cases around type inference and name lookup where this means breaking code.
   • At least, C++ had a culture of "using values", so Java's billion-dollar-mistake doesn't really exist here. Being nullable is the exception rather than the rule.
4. C# 8.0 introduced string? in addition to string to denote explicit nullability
   • Since existing code has used string in the sense "possibly null", this needs an opt-in, allowing compatibility
   • Not being a heavy C# user myself, this looks quite elegant from the outside. It allows modern APIs to become more expressive without breaking code.

[Bromeon]

An idea I mentioned during today's GDExtension meeting is the type-state pattern, meaning that multiple types are used to represent different states of an object.

Concretely, we could split RequiredPtr<T> into two parts (names TBD):

• RequiredPtr<T> as the parameter type in the function signature
  • not yet verified if non-null
  • does not allow access to inner pointer/reference
• VerifiedPtr<T> inside the function
  • converted from RequiredPtr<T> via new macro similar to ERR_FAIL_NULL
  • can only be brought into existence if that macro succeeds
  • allows access to pointer and is always valid to dereference (no UB, no more checks)

The core idea here is that you cannot obtain VerifiedPtr, without going through one of the macros that verifies the non-null property. The macro returns from the current function if the check fails, making subsequent code safe.

[dsnopek]

@Bromeon I finally got around to reworking this PR based on your suggestion of using the "type-state pattern" from several months ago

This requires using new error macros to unpack the underlying pointer, which I think would also lead to encouraging more consistent use of the error macros for pointer parameters, which I think is something @AThousandShips would like per her earlier comments.

Please let me know what you think!

[Bromeon]

Thanks so much for this! ❤️

Added some first feedback.

It's also convenient that required meta is mutually exclusive with other metas, since it's an object and others are numbers 🙂

[dsnopek]

Alright, I think I've got these changes where I want them to be! I'd still like to do some more testing by marking more APIs with required and making sure everything still works, but I think this can come out of DRAFT and get some wider review :-)

Regarding the two CI failures:

• One is unrelated (it's in master too) and will be fixed by Core: Sidestep GCC false-positive #111771
• The other is because godot-cpp needs to be updated for this change, which will be fixed by Specifically handle each "meta" value, so new ones don't break code generation godot-cpp#1868

@Bromeon Can you do some testing with this using the Rust bindings?

[dsnopek]

CI is passing now!

[Bromeon]

Thanks a lot for the great work so far!
So, I implemented this in Rust: godot-rust/gdext#1383

Codegen works nicely! 👍

---

When testing the invocation, I ran into UB when calling Node::create_tween():

handle_crash: Program crashed with signal 11
Engine version: Godot Engine v4.6.dev.custom_build (9ccb329effa3aaac6d0e0ab3b157b6f02b099926)
Dumping the backtrace. Please include this when reporting the bug on: https://github.com/godotengine/godot/issues
[1] /lib64/libc.so.6(+0x1a070) [0x7f88079ef070] (??:0)
[2] List<Ref<Tweener>, DefaultAllocator>::front() (/.../godot/core/templates/list.h:269)
[3] List<Ref<Tweener>, DefaultAllocator>::clear() (/.../godot/core/templates/list.h:468 (discriminator 1))
[4] List<Ref<Tweener>, DefaultAllocator>::~List() (/.../godot/core/templates/list.h:702)
[5] void LocalVector<List<Ref<Tweener>, DefaultAllocator>, unsigned int, false, false>::_resize<true>(unsigned int) (/.../godot/core/templates/local_vector.h:57 (discriminator 3))
[6] LocalVector<List<Ref<Tweener>, DefaultAllocator>, unsigned int, false, false>::resize(unsigned int) (/.../godot/core/templates/local_vector.h:191)
[7] LocalVector<List<Ref<Tweener>, DefaultAllocator>, unsigned int, false, false>::clear() (/.../godot/core/templates/local_vector.h:153)
[8] Tween::clear() (/.../godot/scene/animation/tween.cpp:222)
[9] SceneTree::finalize() (/.../godot/scene/main/scene_tree.cpp:867 (discriminator 8))
[10] OS_LinuxBSD::run() (/.../godot/platform/linuxbsd/os_linuxbsd.cpp:996)
[11] ...

I could also reproduce this when calling godot-rust master with this revision, so it's not related to my changes.
There, I additionally get this error from Godot:

ERROR: Trying to unreference a SafeRefCount which is already zero is wrong and a symptom of it being misused.
Upon a SafeRefCount reaching zero any object whose lifetime is tied to it, as well as the ref count itself, must be destroyed.
Moreover, to guarantee that, no multiple threads should be racing to do the final unreferencing to zero.
   at: _check_unref_safety (./core/templates/safe_refcount.h:186)

================================================================
handle_crash: Program crashed with signal 4
Engine version: Godot Engine v4.6.dev.custom_build (9ccb329effa3aaac6d0e0ab3b157b6f02b099926)
Dumping the backtrace. Please include this when reporting the bug on: https://github.com/godotengine/godot/issues
[1] /lib64/libc.so.6(+0x1a070) [0x7f8e42ff3070] (??:0)
[2] RefCounted::unreference() (/.../godot/core/templates/safe_refcount.h:186 (discriminator 5))
[3] Ref<Tween>::unref() (/.../godot/core/object/ref_counted.h:207 (discriminator 1))
[4] Ref<Tween>::~Ref() (/.../godot/core/object/ref_counted.h:225)
[5] List<Ref<Tween>, DefaultAllocator>::Element::~Element() (/.../godot/core/templates/list.h:52)
[6] void memdelete_allocator<List<Ref<Tween>, DefaultAllocator>::Element, DefaultAllocator>(List<Ref<Tween>, DefaultAllocator>::Element*) (/.../godot/core/os/memory.h:154)
[7] List<Ref<Tween>, DefaultAllocator>::_Data::erase(List<Ref<Tween>, DefaultAllocator>::Element*) (/.../godot/core/templates/list.h:249)
[8] List<Ref<Tween>, DefaultAllocator>::erase(List<Ref<Tween>, DefaultAllocator>::Element*) (/.../godot/core/templates/list.h:436)
[9] SceneTree::process_tweens(double, bool) (/.../godot/scene/main/scene_tree.cpp:835)
[10] SceneTree::process(double) (/.../godot/scene/main/scene_tree.cpp:719)
[11] Main::iteration() (/.../godot/main/main.cpp:4748 (discriminator 3))
[12] OS_LinuxBSD::run() (/.../godot/platform/linuxbsd/os_linuxbsd.cpp:991 (discriminator 1))
[13] ...

RequiredValue<T> seems to behave differently than Ref<T> -- does it increment the refcount before returning the object to the caller? The above looks like UAF...

---

This further makes me think we may need separate RequiredParam<T> and RequiredReturn<T> (or RequiredValue<T>? naming is hard) classes...

Back to this: I would vote for calling it RequiredReturn, simply because it's quite possible that contributors -- especially when new to the "required" concept -- would declare parameters as RequiredValue<SomeType>. On the other hand, RequiredReturn in parameter position looks more like a mistake.

[dsnopek]

When testing the invocation, I ran into UB when calling Node::create_tween()

Thanks for testing and finding this issue! I'm not entirely sure how I didn't notice it in my testing originally :-/

I just pushed some changes that I think should fix it - please let me know if it works for you.

Back to this: I would vote for calling it RequiredReturn, simply because it's quite possible that contributors -- especially when new to the "required" concept -- would declare parameters as RequiredValue<SomeType>. On the other hand, RequiredReturn in parameter position looks more like a mistake.

Maybe we could call it RequiredReturnValue? Although, that's maybe getting a little long...

Personally, I just don't like RequiredReturn from a English language grammatical perspective. The word "return" in this usage isn't a noun, it's a verb - you don't "return a return", you "return a value", or to get extra specific, you "return a return value". The word "return" can be used as a noun (ex. "I'm awaiting his return") but that's a little old-timey and I don't know that I've ever seen it used this way in the context of programming; the noun that I've always seen used is "return value"

But, yeah, this sort of thing is always super subjective - I wouldn't be surprised to find another native English speaker from somewhere else that may feel differently about this :-) so perhaps it doesn't matter too much

[Bromeon]

Thanks for testing and finding this issue! I'm not entirely sure how I didn't notice it in my testing originally :-/

I think it's easy to miss, but I wonder why CI didn't catch it. Maybe it makes sense to add some smoke tests for known APIs? Either in core code (if that can trigger this sort of issue) or the godot-cpp CI.

In Rust I simply used this:

let mut parent = Node::new_alloc();
let child = Node::new_alloc();
parent.add_child(&child);

let tween: Gd<Tween> = parent.create_tween();
assert!(tween.is_instance_valid());
assert!(tween.to_string().contains("Tween"));

parent.free();

---

Personally, I just don't like RequiredReturn from a English language grammatical perspective.

RequiredResult maybe?

[dsnopek]

Maybe it makes sense to add some smoke tests for known APIs? Either in core code (if that can trigger this sort of issue) or the godot-cpp CI.

Makes sense to me! I've made a PR to add such a test to godot-cpp's tests:

godotengine/godot-cpp#1875

RequiredResult maybe?

That could work! Let's discuss at the GDExtension team meeting before I update this PR, though

[dsnopek]

Per our discussion at today's GDExtension meeting, I've renamed RequiredValue to RequiredResult

[Bromeon]

✅ Re-tested 17028c6 with godot-rust, all our tests pass.

This looks good from my side!