Skip to content

Conversation

@li-manssing
Copy link

Summary

Add support for capturing screenshots of multiple windows with hardware bitmap support

Issue:

  • Falcon library fails when processing hardware-accelerated bitmaps
    with error: Software rendering doesn't support hardware bitmaps.
  • The fallback method Utils.capture() is not capable of capturing
    screenshots of multiple window surfaces (dialogs, bottom sheets,
    Compose overlays).

Cause:

  • Falcon library uses Canvas-based rendering which requires software
    bitmaps, but modern Android apps use hardware-accelerated bitmaps
    for performance improvement, causing screenshot capture to fail.
  • The fallback approach used Utils.capture() which internally uses
    PixelCopy API that supports hardware bitmaps but could only capture
    the screenshot of the main activity window and not any additional
    windows layered on top (dialogs, bottom sheets, etc.).
  • Each window (activity, dialog, bottom sheet) has its own Surface in
    Android's rendering system, requiring separate capture operations.

Fix:

  • Created new MultiWindowScreenshotCapture.java utility class to
    handle hardware-accelerated bitmap capture of multiple windows using PixelCopy API.
  • Implemented 3-tier fallback strategy in Shaky.java:
    1. Attempt Falcon library for fast Canvas-based capture (works for
      traditional Views without hardware bitmaps).
    2. Use PixelCopy multi-window capture when Falcon fails (handles
      hardware bitmaps).
    3. Fallback to Utils.capture() for main activity window only (last resort).
  • Implemented getRootViews() to detect all active windows using
    reflection into WindowManager internals.
  • Added captureMultipleAsync() method with dual capture strategy:
    • Primary: Capture from Surface (via reflection on ViewRoot's mSurface)
      for per-window isolation.
    • Fallback: Capture from Window object when Surface unavailable.
  • Created ViewRootData class to encapsulate window hierarchy information:
    • View reference, window frame bounds, layout parameters, ViewRoot object.
    • Window type detection (activity vs dialog/bottom sheet).
    • Window association via tokens for parent-child relationships.
  • Updated CollectDataTask.java to handle multiple bitmaps:
    • First bitmap becomes the main screenshot (for UI preview).
    • Subsequent bitmaps (dialogs/bottom sheets) are added as attachments.

Testing done

Tested locally in integrating app and verified that the changes work as expected:

  • Compose bottom sheets captured successfully.
  • Traditional bottom sheets captured correctly.
  • Multiple screenshots saved as separate attachments.

…re bitmap support

Issue:
- Falcon library fails when processing hardware-accelerated bitmaps
  with error: Software rendering doesn't support hardware bitmaps.
- The fallback method Utils.capture() is not capable of capturing screenshots
  of multiple window surfaces (dialogs, bottom sheets, Compose overlays).

Cause:
- Falcon library uses Canvas-based rendering which requires software
  bitmaps, but modern Android apps use hardware-accelerated bitmaps
  for performance improvement, causing screenshot capture to fail.
- The fallback approach used Utils.capture() which internally uses
  PixelCopy API that supports hardware bitmaps but could only capture
  the screenshot of the main activity window and not any additional
  windows layered on top.
- Each window (activity, dialog, bottom sheet) has its own Surface in
  Android's rendering system, requiring separate capture operations.

Fix:
- Created new MultiWindowScreenshotCapture.java utility class to handle
  hardware-accelerated bitmap capture of multiple windows using PixelCopy API.
- Implemented 3-tier fallback strategy in Shaky.java:
  1. Attempt Falcon library for fast Canvas-based capture (works for
     traditional Views without hardware bitmaps).
  2. Use PixelCopy multi-window capture when Falcon fails (handles
     hardware bitmaps).
  3. Fallback to Utils.capture() for main activity window only (last resort).
- Implemented getRootViews() to detect all active windows using reflection
  into WindowManager internals.
- Added captureMultipleAsync() method with dual capture strategy:
  - Primary: Capture from Surface (via reflection on ViewRoot's mSurface)
    for per-window isolation.
  - Fallback: Capture from Window object when Surface unavailable.
- Created ViewRootData class to encapsulate window hierarchy information
  including view reference, window frame bounds, layout parameters,
  ViewRoot object, window type detection, and window association via tokens.
- Updated CollectDataTask.java to handle multiple bitmaps where first
  bitmap becomes the main screenshot for UI preview and subsequent
  bitmaps are added as attachments.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant