This is part of dc22dae issue. It's more difficult to validate values in wither methods than builder's setters.
So far I haven't find a good solution for fixing that, but I consider variant of composing with* methods to one big. It will fix the bug and broke user-friendly withers. Or a less terrible variant with adding attribute validation to serializer's methods