Fix integer overflow in texture size calculations for large images#13184
Open
xtfo wants to merge 3 commits intoobsproject:masterfrom
Open
Fix integer overflow in texture size calculations for large images#13184xtfo wants to merge 3 commits intoobsproject:masterfrom
xtfo wants to merge 3 commits intoobsproject:masterfrom
Conversation
The intermediate multiplication in BackupTexture and InitSRD can exceed UINT32_MAX for large textures, causing the value to wrap around. This results in truncated subresource data being passed to CreateTexture2D, which causes the driver to read out of bounds and crash. Cast to uint64_t before the multiply to prevent overflow.
The intermediate multiplication in upload_texture_2d can exceed UINT32_MAX for large textures, causing the value to wrap around. While the result is only used for compressed textures, fix the calculation for correctness.
The memory usage calculation in gs_image_file_init_internal can overflow uint32_t for large images. Cast to uint64_t to prevent the multiplication from wrapping around.
Member
|
Can confirm these changes allow me to properly load a 16000x9000 image in an image source. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fix integer overflow in texture size calculations across the D3D11 and OpenGL backends, and in image file memory tracking.
Motivation and Context
Loading large images within D3D11's resource limits crashes OBS due to a uint32_t overflow in the buffer size math. I also looked around a bit and found the same issue in the OpenGL and image-file code, so I've included those fixes here too.
Closes #11921
Supersedes #13183
How Has This Been Tested?
Loaded a large image that previously caused a crash. Image now loads and displays correctly. Verified in a debugger that the calculations in the OpenGL and image-file code paths produce correct values.
Types of changes
Checklist: