Skip to content

Conversation

@YasirAlibrahem
Copy link

Objective

Optimize buffering of non-seekable streams by replacing the manual chunk reading loop with shutil.copyfileobj(). The proposed method is more memory efficient with fewer buffer reallocations, all while maintaining the existing buffer growth pattern. Proposed code changes enhance code readability and maintainability.

Changes

  • In the convert.stream() method, replace the while True statement with shutil.copyfileobj(),
  • shutil.copyfileobj() uses a larger internal buffer size of 64KB instead of the current 4KB
  • shutil.copyfileobj() leverages a C-optimized implemented, reducing Python bytecode overhead

Example Performance Enhancement
For a 10MB non-seekable stream:

  • Before: ~2,560 loop iterations with 4KB chunks
  • After: ~160 iterations with 64KB chunks (16x fewer)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant