possible orientation desync issue

[Piano-Walrus]

so in my "YuGiOh Duel Cafe" world, i just recently put the latest release of this on every card present in the world upstairs, and apparently some people see cards flipped the opposite way but only while being held by another player. so as opposed to seeing the card's back, they might see the card's face (but only some/one user[s], not necessarily all of them). i've never submitted a github issue, not sure what the protocol is or anything, but, steps to reproduce:

1. have a synced object with very obvious different sides/faces
2. have a PC player pick up the object while a quest player is in the instance (they may not have to be a quest user, I'm just trying to give the exact scenario)
3. the object flips once picked up, then flips back once dropped, but only for some other players in the instance, and not the player picking up the object

also, terribly sorry if this is somehow just a me issue. and if it helps at all, the object in question is one of many similar/duplicate objects, each with its own SmartObjectSync script on it

[MMMaellon]

This is a known issue that I'm not sure I have a fix for.

The Quest user is seeing a fallback avatar which has different bone orientations than the PC user's real avatar. When a player picks up an object with SmartObjectSync, SmartObjectSync calculates the object's relative transform to the player's hand bone and uses that to move itself with the player's hand. Because that bone orientation is different for players seeing a fallback, the object aligns itself to the player's hand incorrectly on their screen.

VRChat is coming out with imposters soon and I'm hoping that will solve this issue. As far as I know, I don't have a way of detecting fallback avatars, so there's not much I can do to address the issue on my end.

[MMMaellon]

Thanks for the report though! Marking as not planned for now because I don't have a way to fix this issue.

[Piano-Walrus]

Ah, classic VRChat causing problems for creators, okay no worries!

I don't really know anything about bone tracking or anything like that, and I'll refrain from giving any other unsolicited suggestions, but maybe it would be possible to have an "orientation matters" toggle in Unity that could check "Networking.LocalPlayer == [the player who picked up the object]" and compare the rotation values of the object for each player, and if any values don't match up, just apply any necessary offsets to the rotation locally? I'm sure that's probably not possible for some reason, but just a thought.

Anyway, looking forward to when VRChat [hopefully] fixes this! And thank you so much for all your hard work on this (:

[Piano-Walrus]

So I'm unfortunately going to have to switch back to VRC_ObjectSync until this is fixed, BUT after looking through your code, at least to my understanding, you're getting the transform of the owner's avatar's hand bone (or head position/playspace when hand bones aren't available, etc.), then constantly tracking that bone's transform, while also keeping track of the held object's previous transform values. So for everyone who is not the owner of the object, you could note the object's rotation immediately OnPickup, then if the bone immediately rotates an unreasonable amount right after that (like say, more than 90 degrees the very first time you apply the bone's rotation, or just in an extremely short period of time if that wouldn't be possible), you might be able to apply that rotation offset locally for that playerId (or maybe an array of playerId's? or a split string of playerId's?) for the remainder of the duration that the object is held! Or maybe you wouldn't need to record playerId's; maybe you could just correct the initial rotation, then while the object is held, apply the same offsets that are applied in the owner's instance, to that corrected initial rotation in any other instances. And if the problem is somehow that the rotation over time might be inverted as the owner's bones move since the bone is inverted somehow, you could probably detect and correct for that by comparing two rotation points in both the owner's instance, and the instance of someone experiencing the issue. Or maybe you could simply compare initial rotations for that bone in each instance? Not too sure on that one, but I'd imagine some combination of all of those ideas should work?

I'm terribly sorry for dragging this on and trying to help when no one asked me to, but I'm a firm believer of "if there's a will, there's a way" so I really feel like there has to be some way to fix this.

[MMMaellon]

I've actually tried some of these fixes on a previous map of mine called The Hidden. It puts claws on players hands but I do the check for bone orientations and account for offsets locally. It worked ok-ish but because there's a lot of latency, people's claws always ended up crooked anyway.

This can be solved by syncing even more data, but I'll have to research how to do it efficiently and figure out if there's a way to only apply the fix if there's a fallback avatar. I'll leave the issue open for now

[Piano-Walrus]

Ah okay, makes sense. Well I wish you luck figuring everything out then!