2gb upload limit fix #352
2gb upload limit fix #352
Conversation
demiankatz
left a comment
demiankatz
left a comment
There was a problem hiding this comment.
Thanks, @Jason-Benson, great work -- this looks like a solid solution to the problem. See below for the usual nitpicks and suggestions, many of which are just about possibly removing log messages that are no longer needed. :-)
api/src/services/Fedora.ts
Outdated
| username: this.config.fedoraUsername, | ||
| password: this.config.fedoraPassword, | ||
| }; | ||
| const options = Object.assign({}, auth); |
There was a problem hiding this comment.
We use Object.assign in other parts of the code to combine multiple objects together, but since there is only one object here (auth), I think this call is unnecessary. You could just assign the values directly to an options variable and skip the assign step.
There was a problem hiding this comment.
My attempts to tinker with this very much broke ingest. How would you suggest changing this?
api/package.json
Outdated
| "mysql2": "^3.16", | ||
| "n3": "^2", | ||
| "nanoid": "^5.1.6", | ||
| "n3": "^1", |
There was a problem hiding this comment.
Not sure why these dependencies have been downgraded here; I suspect this may be some kind of merge issue and not intended as part of the PR.
demiankatz
left a comment
demiankatz
left a comment
There was a problem hiding this comment.
Thanks for the further progress, @Jason-Benson -- looking even better now. I had one more minor idea while reviewing, and it looks like there are still a couple of comments pending from the last review, but I've gone through and resolved all the threads that are finished to clarify what remains.
api/src/models/FedoraObject.ts
Outdated
| mimeType: "text/xml", | ||
| logMessage: "Initial Ingest addDatastream - MASTER-MD", | ||
| }; | ||
| console.log("Getting fits MasterMetadata for file:", filename); |
There was a problem hiding this comment.
My earlier comment was marked as resolved, but I'd still recommend changing any console.log calls in this file that you want to keep to this.log so they write to the permanent ingest log instead of the console; that will make them more readily available for troubleshooting real-life ingests once we launch this.
demiankatz
left a comment
demiankatz
left a comment
There was a problem hiding this comment.
See below for a couple of optional suggestions. We're definitely nearly done!
| const digestHeader = `md5=${md5}, sha-512=${sha512}`; | ||
| return digestHeader; |
There was a problem hiding this comment.
We could simplify to a direct return since the variable isn't really used for anything:
| const digestHeader = `md5=${md5}, sha-512=${sha512}`; | |
| return digestHeader; | |
| return `md5=${md5}, sha-512=${sha512}`; |
| return http(method, url, data, options); | ||
|
|
||
| return http(method, url, data, options).catch((err) => { | ||
| console.error(`Request failed for ${method.toUpperCase()} ${url}:`, err); |
There was a problem hiding this comment.
There are a number of console.error calls added to this file, and it looks like we have not console-logged errors in this layer in the past. I think it might make sense to remove them, since they seem to be associated with logic that is going to throw an exception anyway (which should lead to output elsewhere in the stack). If you think there's a strong reason to keep them around, though, I suspect they're harmless, since they should only show up in situations where things are going wrong.
2 participants
To get around node's 2gb limit on handling, I reworked the ingest and metadata handling to stream the files in chunks.