Skip to content

perf: multer memoryStorage buffers entire file in RAM #86

@laughable-9

Description

@laughable-9

Problem

File uploads use multer.memoryStorage() which buffers the entire file in Node.js process memory before streaming to Google Drive.

Affected files

  • srobackend/src/routes/activityRequestRoutes.js (line 13-16)
  • srobackend/src/routes/adminActivityRoutes.js (same pattern)
  • srobackend/src/routes/annualReportRoutes.js (same pattern)
  • srobackend/src/routes/orgApplicationRoutes.js (same pattern)

Current config

```javascript
const upload = multer({
storage: multer.memoryStorage(),
limits: { fileSize: 10 * 1024 * 1024 }, // 10MB max
});
```

Impact

  • Each concurrent 10MB upload holds 10MB in Node.js heap
  • 10 concurrent uploads = 100MB RAM consumed just for file buffers
  • Under heavy load (e.g., submission deadline), could cause OOM crashes or server slowdown
  • Current 10MB limit provides some protection, but memory usage is still proportional to concurrent uploads

Proposed fix

Option A: Switch to disk storage
```javascript
const upload = multer({
storage: multer.diskStorage({
destination: os.tmpdir(),
filename: (req, file, cb) => cb(null, `upload-${Date.now()}-${file.originalname}`),
}),
limits: { fileSize: 10 * 1024 * 1024 },
});
```
Then read from disk and stream to Google Drive using fs.createReadStream() instead of passing a buffer. Clean up temp files in a finally block.

Option B: Stream directly to Google Drive
Use multer's stream interface or busboy to pipe the upload stream directly to the Google Drive API without buffering. This is more complex but eliminates both memory and disk usage.

Why we deferred this

At UP Baguio's scale (~3,000 students), concurrent uploads during peak hours are unlikely to exceed 5-10 simultaneous requests. The 10MB file size limit caps worst-case memory usage at ~100MB, which is within typical server RAM.

The fix requires changing how file buffers are passed to uploadToGoogleDrive() across 4 route files, plus adding temp file cleanup logic. Low risk but moderate effort, and the current behavior is acceptable for the expected load.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions