Turn OFF Video stabilization

[nDrew158]

For a proper camera tracking purposes is that possible to turn OFF any kind of video stabilization (digital, sensor and lens) in 360 video shooting mode?

[dnewman-gpsw]

it is mostly off as is, with the stabilization being a control in Player or Quik.

[dnewman-gpsw]

Two theories

1. look at GEOC metadata - "Geometry Calibrations". The 360 lens system is calibrated to know the orientation of one lens to the other, as there are never ideally 180° apart. In your case you altered this dramatically.

Example:

"DEVC": {
  "DVID": "GEOC",
  "DVNM": "Geometry Calibrations",
  "VERS": 3,
  "SHFX": -0.0574,
  "SHFY": 0.0543,
  "SHFZ": -34.2017,
  "ANGX": -0.0816,
  "ANGY": 179.9165,
  "ANGZ": -1.0757,
  "CALW": 4216,
  "CALH": 4216,
  "SCLH": 1.0000,
  "SCLV": 1.0000,
  "CAPW": 4216,
  "CAPH": 4216
},

See ANGX thru ANGZ - one axis ~1.1° off. So the projection of the EAC has corrected for this, if you don't compensate for this, you will see some remanent motion.

2. I do expect this is rolling shutter compensation being wrong for the new lens orientation. You can turn off ERS with *BERS=1, but that will get likely get you wobble in both lenses. The math will be very tricky, to reverse the ERS one lens. The best way to understand, is put the camera on a turn-table, so rotate one axis only. Vertical lines will be vertical in one lens, a angled in the other, the amount of angle is the amount remove. The amount to remove with related to the different in the CORI.

[silverqsy]

Two theories

1. look at GEOC metadata - "Geometry Calibrations". The 360 lens system is calibrated to know the orientation of one lens to the other, as there are never ideally 180° apart. In your case you altered this dramatically.

Example:

"DEVC": {
  "DVID": "GEOC",
  "DVNM": "Geometry Calibrations",
  "VERS": 3,
  "SHFX": -0.0574,
  "SHFY": 0.0543,
  "SHFZ": -34.2017,
  "ANGX": -0.0816,
  "ANGY": 179.9165,
  "ANGZ": -1.0757,
  "CALW": 4216,
  "CALH": 4216,
  "SCLH": 1.0000,
  "SCLV": 1.0000,
  "CAPW": 4216,
  "CAPH": 4216
},

See ANGX thru ANGZ - one axis ~1.1° off. So the projection of the EAC has corrected for this, if you don't compensate for this, you will see some remanent motion.

2. I do expect this is rolling shutter compensation being wrong for the new lens orientation. You can turn off ERS with *BERS=1, but that will get likely get you wobble in both lenses. The math will be very tricky, to reverse the ERS one lens. The best way to understand, is put the camera on a turn-table, so rotate one axis only. Vertical lines will be vertical in one lens, a angled in the other, the amount of angle is the amount remove. The amount to remove with related to the different in the CORI.

Hi David, these are really valueble insights, thanks! I do have some more questions.
Considering 1, the GEOC data, is the end result a static offse for the 2 lenses? or its effect are dynamic? cause if it's static it can easily be canceled out.

Regarding 2, I tried *BERS=1, but I'm not seeing any difference in the result as if it has no effect, I suspect somehow it does not work with Max 2 camera? Also I did a test shot when i quickly pan the camera, I see no slanted staight lines, but I still see some jello effect. and it is espscially bad at top/bottom. I suspect is has something to do with how i mod the camera. So i just rotate the sensor/lens module on the screen side along the vertical axis(basically did a yaw 180).

Do you have any more suggestions/comments at this point? I am trying to do my own RS correction to the problematic eye. seems there is a factor that is the sensor Readout time, do you have this info for the Max 2 sensor? it'll be very helpful, thank you!

[dnewman-gpsw]

GEOC is static. SROT is the sensor read out time.

[silverqsy]

GEOC is static. SROT is the sensor read out time.

Thanks for this infor David, that helps a lot. Meanwhile, could you confirm that *BERS=1 actually works on Max2? cause after many testing I don't see enabling it have any effect at all in the original .360 file or after gopro player export.

[silverqsy]

GEOC is static. SROT is the sensor read out time.

Hey David, I have confirmed that the EAC .360 file still have in-camera rolling shutter correction even with *BERS=1. I tried convert the EAC back to fisheye, using lens caliberation data from GEOC to get a image that close to raw fisheye output, then tried to apply 2x reverse rolling shutter correction to the tilt and roll axis, but at this point it's just too many variables and I cannot get the desiable result without knowing exactly what in-camera correciton is doing.
meanwhile, I found that the dualfish mode on Max2 with *BERS=1 does work and without in-camera RS correction, but it's only in 8bit and cannot use log.

So could you please take a look and implement either A: Make *BERS=1 work for EAC projection or B: Make dual-fisheye output work with 10bit.
OR even better, give an option to apply inversed RS correction on tilt and roll axis, for the front lens(the one on the other side of the screen/the one wihtout IMU sensor)
That will basiclaly solve my last road block. I went pretty far on this and already got almost everything else figured out. So I will not need any further assistance from GoPro other than this.
Thank you!

[dnewman-gpsw]

In 360 it seems the ERS is directly wired into the camera orientation and stabilization. If I zero out the gyro data going into the stabilization algorithms, it disables the ERS, but it also disables a camera orientation data. This means you would have to do your own stabilization based on GYRO not CORI.

[silverqsy]

In 360 it seems the ERS is directly wired into the camera orientation and stabilization. If I zero out the gyro data going into the stabilization algorithms, it disables the ERS, but it also disables a camera orientation data. This means you would have to do your own stabilization based on GYRO not CORI.

It would be great if these options could be modified using QR codes, or even reset to zero: "SHFX": -0.0574,

"SHFY": 0.0543,

"SHFZ": -34.2017,

"ANGX": -0.0816,

"ANGY": 179.9165,

"ANGZ": -1.0757.

If this could be achieved, it would undoubtedly be the best. Also, during EAC correction, are perfect spheres used for correction (i.e., equidistant cylindrical spheres)? Otherwise, even if the correction is done later using the above data, it still won't perfectly match the original spherical mesh.

of course not equidistant. GEOC metadata has parameters for the specific distortion for the lens MAX2 is using.

[suicaoya123]

In 360 it seems the ERS is directly wired into the camera orientation and stabilization. If I zero out the gyro data going into the stabilization algorithms, it disables the ERS, but it also disables a camera orientation data. This means you would have to do your own stabilization based on GYRO not CORI.

It would be great if these options could be modified using QR codes, or even reset to zero: "SHFX": -0.0574,
"SHFY": 0.0543,  “SHFY”：0.0543，
"SHFZ": -34.2017,  “SHFZ”：-34.2017，
"ANGX": -0.0816,  “ANGX”：-0.0816，
"ANGY": 179.9165,  “ANGY”：179.9165，
"ANGZ": -1.0757.  “ANGZ”：-1.0757。
If this could be achieved, it would undoubtedly be the best. Also, during EAC correction, are perfect spheres used for correction (i.e., equidistant cylindrical spheres)? Otherwise, even if the correction is done later using the above data, it still won't perfectly match the original spherical mesh.
of course not equidistant. GEOC metadata has parameters for the specific distortion for the lens MAX2 is using.当然不是等距离。

So I'm confirming if this is indeed the case.

And he's referring to the lens deviation angle, not anything else.

Because the process might be: --> Original image -> MAX2 lens mesh model -> Projecting a standard sphere and correcting --> Generating EAC

Because if the EAC isn't based on a standard equidistant sphere before generation, we can't obtain a truly accurate reverse effect. Even with highly accurate simulation data, it's impossible to perfectly fit an identical MAX2 mesh model. Therefore, we'll always get inconsistent stabilization results for the two eyes.

[silverqsy]

There

In 360 it seems the ERS is directly wired into the camera orientation and stabilization. If I zero out the gyro data going into the stabilization algorithms, it disables the ERS, but it also disables a camera orientation data. This means you would have to do your own stabilization based on GYRO not CORI.在 360 中，ERS 似乎直接接入了相机的朝向和防抖系统。如果我将放进防抖算法的陀螺仪数据归零，它会禁用 ERS，但也会禁用相机朝向数据。这意味着你必须基于陀螺仪而非 CORI 自行进行稳定。

It would be great if these options could be modified using QR codes, or even reset to zero: "SHFX": -0.0574,如果这些选项可以用二维码修改，甚至重置为零，那就太好了：“SHFX”： -0.0574，
"SHFY": 0.0543,  “SHFY”：0.0543，
"SHFZ": -34.2017,  “SHFZ”：-34.2017，
"ANGX": -0.0816,  “ANGX”：-0.0816，
"ANGY": 179.9165,  “ANGY”：179.9165，
"ANGZ": -1.0757.  “ANGZ”：-1.0757。
If this could be achieved, it would undoubtedly be the best. Also, during EAC correction, are perfect spheres used for correction (i.e., equidistant cylindrical spheres)? Otherwise, even if the correction is done later using the above data, it still won't perfectly match the original spherical mesh.如果能实现这一点，那无疑是最理想的。另外，在 EAC 校正过程中，是否使用完美球面进行校正（即等距圆柱形球面）？否则，即使之后用上述数据进行校正，也无法完全匹配原始球形网格。

of course not equidistant. GEOC metadata has parameters for the specific distortion for the lens MAX2 is using.当然不是等距离。GEOC 元数据包含 MAX2 所用透镜特定变真参数。

So I'm confirming if this is indeed the case.

And he's referring to the lens deviation angle, not anything else.

Because the process might be: --> Original image -> MAX2 lens mesh model -> Projecting a standard sphere and correcting --> Generating EAC

Because if the EAC isn't based on a standard equidistant sphere before generation, we can't obtain a truly accurate reverse effect. Even with highly accurate simulation data, it's impossible to perfectly fit an identical MAX2 mesh model. Therefore, we'll always get inconsistent stabilization results for the two eyes.

there are more data in GEOC metadata than David posted, including per-lens caliberation data. Just dig futhter. I am able to revserse from EAC to fisheye quite accurately.

[dnewman-gpsw]

This build uses *BERS=1 to disable all stabilization https://www.dropbox.com/scl/fi/s9kcz34shiw6nmxhom17e/LABS_MAX2_01_29_73_S3D_BERS.zip?rlkey=aivgn40sjcsqte1iyfsk04dn6&dl=1

[silverqsy]

This build uses *BERS=1 to disable all stabilization https://www.dropbox.com/scl/fi/s9kcz34shiw6nmxhom17e/LABS_MAX2_01_29_73_S3D_BERS.zip?rlkey=aivgn40sjcsqte1iyfsk04dn6&dl=1

Thank you David, trying to work with RAW Gyro data and it is indeed quite challenging.
At the same time, I think having the ability to edit GEOC might be a more efficient solution, From my thinking if we change the "ANGY": from 179.9165 to 0 or -0.0835, the algorithm will be able to apply in-camera RS in the correct direction and solve our issue.

[dnewman-gpsw]

That is very unlikely to work. The system was not designed for arbitrary lens configurations, so it's more likely to crash the camera than do what you expect

[dnewman-gpsw]

Gyro and CORI is what the camera does, so it is a reasonably straightforward transformation. But I do agree that quaternian math is not fun.

[silverqsy]

Gyro and CORI is what the camera does, so it is a reasonably straightforward transformation. But I do agree that quaternian math is not fun.

one random ask, is it possible to enable 5v out from the usbc port on max 2? thinking about powering an external fan via it

[dnewman-gpsw]

You might be able to use 3.3V with low current, but there is very limit support to powering peripherals. The camera does power media mod, and the audio adapter.

[silverqsy]

Finally getting some decent result, thank you very much David for your continuous support!

gyro.full.mov

I end up not using the all stab off firmware, but figured out how to apply 2x reverse ERS to the right eye in the tilt and roll axis properly. Cheers and thanks agian!