🐛 : V3, useSharedValue is not working with V3

[rbayuokt]

What's happening?

hi Marc, I'm currently working on developing a vision-camera-pose-detection project with MLKit. I initially wrote the native code for iOS, and it connected perfectly. However, I encountered a problem when attempting to draw the overlay for the skeleton based on the keypoints.

In general, I used useSharedValue from Reanimated V3 to pass a value.
when I use useSharedValue I receive an error message like this:
Reading from '_value' directly is only possible on the UI runtime.

I have already wrapped my JS function into a worklet, but I still get an error.

If I pass the value into setState it works, but it doesn't behave as expected. there is a delay, as shown in this video. this is my friend who tested it

6b4e8199-3251-4d37-ac53-9990b8cdc49a.mov

"react-native-vision-camera": "^3.0.0", "react-native-worklets-core": "^0.2.0", "react-native-reanimated": "^3.4.2"

Reproduceable Code

PoseDetection.ts

import {VisionCameraProxy, Frame} from 'react-native-vision-camera';
import {IPoseDetection} from '../utils/frameProcessor/PoseDetection';

const plugin = VisionCameraProxy.getFrameProcessorPlugin('poseDetection');

export function poseDetection(frame: Frame): IPoseDetection {
  'worklet';

  if (plugin == null) {
    throw new Error('Failed to load Frame Processor Plugin "poseDetection"!');
  }

  return plugin.call(frame) as unknown as IPoseDetection;
}

Main Code:

...
// POSE

  const pose = useSharedValue<IPoseDetection>(defaultPose);

  const convertToWorklet = (poseObject: IPoseDetection, frame: Frame) => {
    const xFactor = dimensions.width / frame.width;
    const yFactor = dimensions.height / frame.height;

    const poseCopy: IPoseDetection = {
      leftShoulder: {x: 0, y: 0},
      rightShoulder: {x: 0, y: 0},
      leftElbow: {x: 0, y: 0},
      rightElbow: {x: 0, y: 0},
      leftWrist: {x: 0, y: 0},
      rightWrist: {x: 0, y: 0},
      leftHip: {x: 0, y: 0},
      rightHip: {x: 0, y: 0},
      leftKnee: {x: 0, y: 0},
      rightKnee: {x: 0, y: 0},
      leftAnkle: {x: 0, y: 0},
      rightAnkle: {x: 0, y: 0},
    };

    Object.keys(poseObject).forEach(v => {
      const key = v as keyof IPoseDetection;
      poseCopy[key] = {
        x: poseObject[key].x * xFactor,
        y: poseObject[key].y * yFactor,
      };
    });

    pose.value = poseCopy;
  };

  const setToWorklet = Worklets.createRunInJsFn(convertToWorklet);

  const frameProcessor = useFrameProcessor(
    frame => {
      'worklet';
      const poseObject = poseDetection(frame);
      console.log('pose data', poseObject);
      setToWorklet(poseObject, frame);
    },
    [pose],
  );

Relevant log output

Error: Reading from `_value` directly is only possible on the UI runtime, js engine: hermes

Camera Device

No response

Device

iPhone 11 ( 15.5 )

VisionCamera Version

3.0.0

Can you reproduce this issue in the VisionCamera Example app?

• I can reproduce the issue in the VisionCamera Example app.

Additional information

• I am using Expo
• I have enabled Frame Processors (react-native-worklets-core)
• I have read the Troubleshooting Guide
• I agree to follow this project's Code of Conduct
• I searched for similar issues in this repository and found none.

[emclain]

I think I've been able to fix this by using useSharedValue from react-native-worklets-core instead of reanimated.

Per #1466,

Reanimated is no longer used as a Worklet Runtime. Instead, VisionCamera now uses react-native-worklets-core.

[emclain]

However, this presents a problem if I want to use the shared state in an animation. Reanimated doesn't recognize when changes to the state occur, so they don't trigger animation update.

[mrousavy]

Yea useSharedValue is from RN Worklets Core, not from Reanimated.

[rbayuokt]

Yea useSharedValue is from RN Worklets Core, not from Reanimated.

mmm, it still doesn't work when I pass it into useAnimatedStyle. The line is not rendered. I was using react-native-svg and converted the Line into an AnimatedLine with react-native-reanimated's 'createAnimatedComponent'. I'm sorry, but is there any way to draw an overlay like an animated line?

[rbayuokt]

the native modules for Android and iOS is perfectly connected, it shows the values. I'm just missing the last step for drawing the overlay 😢 after it's finish I will publish this vision-camera frame processor library

[cardanoeconews]

I also have the same issue. I've successfully created my own version of vision-camera-ocr that does text recognition. Using latest react-native-vision-camera (v3).

The missing piece is how to get data from the worker in JS, out to my react component, and rendered.

When using "Worklets.createRunInJsFn" the app (iOS) crashes, when calling my backToReactThread function ^ It crashes in Native code, inside jsi::String JSCRuntime::createStringFromUtf8 calling: JSStringCreateWithUTF8CString

And using useShareValue (from Worklets) seems to have no affect. useAnimatedStyle ignores it..

How can we get some of the results/data from our useFrameProcessor javascript function, to the rest of react native so we can draw ui updates?

[mrousavy]

Hey! So VisionCamera V3 uses react-native-worklets-core, and react-native-reanimated uses it's own Worklets implementation, so the useSharedValue is not compatible with the one from Reanimated (I think we could add interop support for that though, wdyt @tomekzaw?)

What you can do (and what I would suggest you to do anyways regardless of this issue) is to draw the entire overlay using @Shopify/react-native-skia. This is much faster and flexible/powerful for drawing such overlays. Use the draw callback to draw, that should also work with useSharedValue. (afaik, right @chrfalch?)

[cardanoeconews]

Thank you @mrousavy ! Btw, is there a type-o in the offiical docs where it says:

"(Optional) If you want to use Frame Processors, you need to install react-native-worklets-core 1.0.0 or higher."

I think react-native-worklets-core only goes up to version 0.2.0.. I don't see 1.0.0 ?

---

Separately, I am still seeing the crash inside jsi::String JSCRuntime::createStringFromUtf8 calling: JSStringCreateWithUTF8CString . Do you have any tips or hints on where I can look for that? I wonder if it's because I have 0.2.0 and not 1.0.0 of react-native-worklets-core (which is no longer available on npm)?

[cardanoeconews]

@mrousavy I wonder if you could share a package.json from a working setup. Curious to see what version of react-native, react-native-worklets-core, and react-native-vision-camera are all working well together.

[mrousavy]

Package.json; the one here in example/ app.

And yea, you're right the docs should say 0.2.0 instead of 1.0.0.

I need a reproduceable example for that

[cardanoeconews]

@mrousavy thank you. that example was super helpful. I managed to get it building and running. In the end I had to:

• Upgrade Node !! This was the biggest source of my problems. I was on Node v16... Needed to go to 18+..
• I needed to install string-hash-64, I know you're aware since I saw you post in the other git repo about it
• I needed to apply the _closure --> __closure (double underline) patch from the other thread to prevent a crash on startup with the example app.

After that, I was good to go. Thank you so much for your help.

[mrousavy]

Oh interesting! Could you maybe send a PR to react-native-worklets-core to fix the _closure issue? Thanks! cc @chrfalch

I'll close this for now, let me know if this issue is still active and we can re-open it

[tomekzaw]

just fyi, in software-mansion/react-native-reanimated#4803 we renamed _closure to __closure for the purpose of consistency so that all fields start with a double underscore.