Skip to content

Conversation

@thalesmg
Copy link
Contributor

@thalesmg thalesmg commented Sep 4, 2025

Previously, when using maps with binary keys, flat unions and verification, unions would not work, even with binary map keys as expected.

For example, with the following definitions:

Src = """
syntax = "proto3";

message test {
  map<string, string> args = 1;
}

message union {
  oneof u {
    int32 a = 1;
    string b = 2;
  }
}
""".
{ok, Mod, Bin} = gpb_compile:string(aaa, Src, [binary, strings_as_binaries, {maps, true}, {maps_oneof, flat}, {maps_key_type, binary}, {verify, always}, {maps_unset_optional, omitted}]).
code:load_binary(Mod, "aaa.erl", Bin).

... normal maps work as expected:

7>   Encoded1 = Mod:encode_msg(#{<<"args">> => #{<<"hey">> => <<"hi">>}}, test).
 <<10,9,10,3,104,101,121,18,2,104,105>>
8>   Mod:decode_msg(Encoded1, test).
 #{<<"args">> => #{<<"hey">> => <<"hi">>}}

... but unions break, due to verifier failing to consider binary keys when encoding:

20>   %% Should produce <<8, 1>>.
      Mod:encode_msg(#{<<"a">> => 1}, union).
** exception error: {gpb_type_error,{{multiple_oneof_keys,[],u},
                                     [{value,#{<<"a">> => 1}},{path,"union.u"}]}}
     in function  aaa:mk_type_error/3 (aaa.erl, line 567)
     in call from aaa:encode_msg/3 (aaa.erl, line 79)

... and decoding due to incorrect generated code which produced <<(atom_to_list(SomeAtom))>> for the map key:

21>   Mod:decode_msg(<<8, 1>>, union).
** exception error: {gpb_error,{decoding_failure,{<<8,1>>,
                                                  union,
                                                  {error,badarg,
                                                         [{aaa,dfp_read_field_def_union,6,
                                                               [{file,"aaa.erl"},
                                                                {line,292},
                                                                {error_info,#{cause => {1,integer,type,"a"},
                                                                              function => format_bs_fail,module => erl_erts_errors}}]},
                                                          {aaa,decode_msg_1_catch,3,[{file,"aaa.erl"},{line,227}]},
                                                          {erl_eval,do_apply,7,[{file,"erl_eval.erl"},{line,919}]},
                                                          {shell,exprs,7,[{file,"shell.erl"},{line,916}]},
                                                          {shell,eval_exprs,7,[{file,"shell.erl"},{line,872}]},
                                                          {shell,restricted_eval_loop,5,
                                                                 [{file,"shell.erl"},{line,866}]}]}}}}
     in function  aaa:decode_msg_1_catch/3 (aaa.erl, line 231)

Previously, when using maps with binary keys, flat unions and verification,
unions would not work, even with binary map keys as expected.

For example, with the following definitions:

```erlang
Src = """
syntax = "proto3";

message test {
  map<string, string> args = 1;
}

message union {
  oneof u {
    int32 a = 1;
    string b = 2;
  }
}
""".
{ok, Mod, Bin} = gpb_compile:string(aaa, Src, [binary, strings_as_binaries, {maps, true}, {maps_oneof, flat}, {maps_key_type, binary}, {verify, always}, {maps_unset_optional, omitted}]).
code:load_binary(Mod, "aaa.erl", Bin).
```

... normal maps work as expected:

```erlang
7>   Encoded1 = Mod:encode_msg(#{<<"args">> => #{<<"hey">> => <<"hi">>}}, test).
 <<10,9,10,3,104,101,121,18,2,104,105>>
8>   Mod:decode_msg(Encoded1, test).
 #{<<"args">> => #{<<"hey">> => <<"hi">>}}
```

... but unions break, due to verifier failing to consider binary keys when encoding:

```erlang
20>   %% Should produce <<8, 1>>.
      Mod:encode_msg(#{<<"a">> => 1}, union).
** exception error: {gpb_type_error,{{multiple_oneof_keys,[],u},
                                     [{value,#{<<"a">> => 1}},{path,"union.u"}]}}
     in function  aaa:mk_type_error/3 (aaa.erl, line 567)
     in call from aaa:encode_msg/3 (aaa.erl, line 79)

```

... and decoding due to incorrect generated code which produced
`<<(atom_to_list(SomeAtom))>>` for the map key:

```erlang
21>   Mod:decode_msg(<<8, 1>>, union).
** exception error: {gpb_error,{decoding_failure,{<<8,1>>,
                                                  union,
                                                  {error,badarg,
                                                         [{aaa,dfp_read_field_def_union,6,
                                                               [{file,"aaa.erl"},
                                                                {line,292},
                                                                {error_info,#{cause => {1,integer,type,"a"},
                                                                              function => format_bs_fail,module => erl_erts_errors}}]},
                                                          {aaa,decode_msg_1_catch,3,[{file,"aaa.erl"},{line,227}]},
                                                          {erl_eval,do_apply,7,[{file,"erl_eval.erl"},{line,919}]},
                                                          {shell,exprs,7,[{file,"shell.erl"},{line,916}]},
                                                          {shell,eval_exprs,7,[{file,"shell.erl"},{line,872}]},
                                                          {shell,restricted_eval_loop,5,
                                                                 [{file,"shell.erl"},{line,866}]}]}}}}
     in function  aaa:decode_msg_1_catch/3 (aaa.erl, line 231)
```
@tomas-abrahamsson
Copy link
Owner

Thanks for a good report and fix. I'll merge it and make a new release.

@tomas-abrahamsson tomas-abrahamsson merged commit f0b2433 into tomas-abrahamsson:master Sep 14, 2025
5 checks passed
@tomas-abrahamsson
Copy link
Owner

Merged and included in 4.21.5.

@thalesmg thalesmg deleted the 20250904-fix-union-binary-map-keys-upstream branch September 15, 2025 00:15
@thalesmg
Copy link
Contributor Author

Thanks! 🍻

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants