Improve Access to View and Array Types

[josephbirkner]

Currently, accessing View and Array types is cumbersome due to the long templates. This also makes transitioning from the C++ 11 emitter hard, because where previously only one using-declarations per schema type were needed, now two are needed (one for the view and one for data struct). The most convenient solution (that I can see) would be

• to emit one using-declaration per used ArrayView instantiation.
• to emit one namespace per type (e.g. Position2D), which contains both Position2D::View and Position2D::Data.

[mikir]

Unfortunately, we are currently not able to figure out where exactly the problem is. Some examples would be a big help.

However, it might be that fixing issue #89 will help to solve this problem as well.

[mikir]

In the meantime, we have investigated the possibility of the Data-View concept redesign using one namespace which would contain both Data and View.

In short, such design is feasible and will not need a very big effort like several months. But it will have some syntax side effects that might be very annoying for users.

Schema example:

struct Field<T>
{
    T value;
    T arr[3];
};

Generated code snippet:

namespace Field
{
    template <typename T>
    struct Data
    {
        T value;
        ::zserio::Vector<T> arr;
    };

    template <typename T>
    using View = ::zserio::View<Data<T>>; // just trick for now
};

Note: If this design wins, we will implement clear solution by making View as a separate class in the Field namespace.

The problems with such a design:

1. Noisy syntax (Data will be everywhere):

Field::Data<zserio::Int32> data;

This will be annoying especially for users who do not want to work with Views. They might complain that this is verbose and that this does not reflect schema where template is defined as Field<int32>.
2. Template parameters will be at the end and not after Field. It would probably be more natural if the template parameters will be after the Field keyword:

Field<zserio::Int32>::Data data;

However in this case, Field cannot be namespace but class. Moreover, the template argument deduction in generic functions won’t work (deduction-issue). Example:

template <class T>
print(typename Field<T>::Data& field)

This function can't be called without template argument as one would expect:

print(someField)

It has to be called as

print<zserio::Int32>(someField) 

3. Namespace Field will be named with a capital F to avoid awkward clashes.

[josephbirkner]

Thanks Miki! This is exactly what I was imagining. There is also another possibility of having the view inside of Data, so e.g. Position2D is the data class and the view could be accessed through Position2D::View. That would of course be even cooler, and I heard that @MikeLankamp-TomTom was able to make it work.

[mikir]

We have found a different approach that solves some drawbacks.

Schema example:

struct Field<T>
{
    T value;
    T arr[3];
};

Generated code snippet:

template <typename T>
struct Field
{
    using Data = Field;
    using View = ::zserio::View<Field>;

    T value;
    ::zserio::Vector<T> arr;
};

This will solve the problem of noisy syntax together with the position of the template parameters:

Field<zserio::Int32> data;

However, it brings some new problems:

1. There will be two different possibilities how to use Data and View:

Field<zserio::Int32> data;
zserio::View<Field<zserio::Int32>> view;

or

Field<zserio::Int32>:Data data;
Field<zserio::Int32>:View view;

2. There will be a clash if Data or View keywords will be used in the schema:

struct Data<T>
{
    T value;
    T arr[3];
};

PS: We are working on simple PoCs. As soon as we are ready, we will attach them to this issue.

[mikir]

Thanks Miki! This is exactly what I was imagining. There is also another possibility of having the view inside of Data, so e.g. Position2D is the data class and the view could be accessed through Position2D::View. That would of course be even cooler, and I heard that @MikeLankamp-TomTom was able to make it work.

We have tried this as well (View inside Data) but we were not able to find out a good solution for template argument deduction so far:

template <class T>
print(Field<T>::View& view)

This function can't be called without template argument as one would expect:

print(someView);

It has to be called as

print<zserio::Int32>(someView);

Of course, it might be that we forgot about something and some workaround is available. Or it might be that this drawback will not be a big hassle for users.

[mikir]

I have attached here the PoC.zip for the following schema:

package struct_templated_field;

struct Field<T>
{
    T value;
    T arr[3];
};

struct Compound
{
    uint32 value;
};

struct StructTemplatedField
{
    Field<uint32>   uint32Field;
    Field<Compound> compoundField;
    Field<string>   stringField;
};

This PoC contains the followings:

• runtime/src: C++17 runtime library
• src/namespace: Redesigned generated sources using one namespace which would contain both Data and View
• src/original: Original sources generated by the latest C++17 generator 0.5.0
• src/using: Redesigned generated sources using keywords using Data and using View

Any feedback is always appreciated.

[reinco]

There is another option: make Data and View the top level namespace.
That way one can alias or use the Data or View namespace as one sees fit (see commented out lines in src/namespace_top/main.cpp):

namespace_top.zip

[reinco]

About the other options: I like the namespace more than the using solution.
Having to write print<zserio::UInt32>(stfv.uint32Field()); looks cumbersome.

However, the option namespace_top I brought up solves the problems I think that were mentioned in the comments. So maybe that is the better approach?:

1. No noisy syntax: In a nds compiler, you would not have to write Field::Data<zserio::Int32> data; One can just write using Data; Field<zserio:Int32> data;.
2. NDS compiler writers could even add a using Data in a top level header, and so then their compiler would only need very minor changes if they already implemented a compiler branch using the existing pre-release. The minor change is in the blob creation because there one needs a View, but in compilers that is only needed once per layer, so at a handful of places.
3. One can invoke print(stfv.uint32Field()); instead of print<zserio::UInt32>(stfv.uint32Field());
4. There will be no clash of names: Data, View are the top level namespaces. So even if someone would declare a struct named Data in zserio script, it is no issue, because the generated code would look like: namespace Data { struct Data {}; }.
5. Views would read natural like: View::struct_templated_field::StructTemplatedField stfv(stf); instead of zserio::View<StructTemplatedField> stfv(stf);. So it solves the original issue mentioned in this ticket.
6. A view is not a second-hand citizen: you can work with a View the same way as with Data. If you use Views more than Data in some cpp file, then one could do a using View;. Then, taking the example from bullet point 5, write struct_templated_field::StructTemplatedField stfv(stf);.

[mikir]

Thank you, Reinco! We have checked your option here and I can confirm that your option is feasible and workable solution.

I have just one comment that top level namespace does not look intuitive to me because it does not reflect the Zserio schema.

Moreover, some NDS users already claimed that there are interested only to Data part. They will be probably not very happy with this solution.

I agree that using Data in a top level header is a solution but this technique is not very preferable in C++ world (because it can unexpectedly change the meaning of code in any other files that include that header).

It might be that we can modify your solution slightly and use top level namespace only for View and Data leave as it is. This will solve at least one problem for users who only use Data.

I have attached here PoC2.zip which contains all 4 different approaches:

1. Solution with namespace with inner classes Data and View (subdirectory namespace)
2. Solution with class with inner using Data and View (subdirectory using)
3. Solution from Reinco (sudirectory top_namespace)
4. Modified solution from Reinco without using top level namespace Data (subdirectory top_view)

Please note that this PoC2.zip also contains the subdirectory original with C++17 sources generated by the current C++17 generator. So, if it turns out that no approach is better, we can still leave things as they are.

[reinco]

What we could also do:
Given package nds.struct_templated_field

1. generate data/struct_templated_field/Compound.h using namespace '<toplevelpackage.>data.struct_templated_field'
2. and generate view/struct_templated_field/Compound.h using namespace '<toplevelpackage.>view.struct_templated_field
   Note: toplevelpackage is an argument to the zserio compiler. The top level name 'nds' mentioned in the zs file is always ignored (even in the current emitter)!

The advantage:

1. people who want to work with Data only, do not see any of the View related classes and functions because they will end up in another header.
2. The directory structure will match the namespace hierarchy.

This archive shows the idea:
namespace_view_data.zip

So this is what you would get if you compile the following struct_templated_field.zs using
java -jar zserio -setTopLevelPackage nds -cpp17 cpp17 struct_templated_field.zs

package nds.struct_templated_field;

struct Field<T>
{
    T value;
    T arr[3];
};

struct Compound
{
    uint32 value;
};

struct StructTemplatedField
{
    Field<uint32>   uint32Field;
    Field<Compound> compoundField;
    Field<string>   stringField;
};

[mikir]

Thank you again! This will work as well of course.

My concerns are very similar as before, it does not look intuitive to me because it does not reflect the Zserio schema. And I am not sure if this solution would help somehow during the porting from the C++11 (I might be wrong because we don't have here some huge application).

I can even see a slightly different point of view. We are currently trying to adapt the quite intuitive generated C++17 code to something different just to ease porting from C++11. And what about newcomers? While they won't have any problems with the porting, they will likely start to complain about the cumbersome generated code and they would be right.