Rework mime type white list #2198

Tschuppi81 · 2025-11-10T15:20:08Z

Org: Ensure mime type validator on file upload fields in form code

TYPE: Feature
LINK: ogc-2738

linear · 2025-11-10T15:20:13Z

codecov · 2025-11-10T15:27:31Z

Codecov Report

❌ Patch coverage is 96.42857% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.39%. Comparing base (cadf75d) to head (00b1614).
⚠️ Report is 2 commits behind head on master.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
src/onegov/form/validators.py	93.33%	2 Missing ⚠️

Additional details and impacted files

Files with missing lines	Coverage Δ
src/onegov/agency/forms/agency.py	`97.26% <100.00%> (ø)`
src/onegov/election_day/forms/election.py	`98.14% <100.00%> (ø)`
src/onegov/election_day/forms/election_compound.py	`97.76% <100.00%> (ø)`
src/onegov/election_day/forms/upload/common.py	`100.00% <ø> (ø)`
src/onegov/election_day/forms/vote.py	`97.68% <100.00%> (ø)`
src/onegov/file/utils.py	`97.46% <ø> (ø)`
src/onegov/form/fields.py	`94.46% <100.00%> (+0.09%)`	⬆️
src/onegov/form/widgets.py	`85.46% <100.00%> (ø)`
src/onegov/landsgemeinde/forms/agenda.py	`47.66% <100.00%> (-0.27%)`	⬇️
src/onegov/landsgemeinde/forms/assembly.py	`94.91% <100.00%> (-0.09%)`	⬇️
... and 11 more

... and 3 files with indirect coverage changes

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cadf75d...00b1614. Read the comment docs.

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…load

src/onegov/form/validators.py

src/onegov/pas/forms/data_import.py

Tschuppi81 · 2025-12-04T12:05:07Z

I saw that files types are handled differently for view_upload_file_by_json in handle_file_upload. Basically all file types are allowed. Shall we keep this?

Tschuppi81 · 2025-12-04T12:16:23Z

Should I completely remove type application/octet-stream ? It is mostly used in conjunction with application/zip

Daverball · 2025-12-04T12:18:26Z

I saw that files types are handled differently for view_upload_file_by_json in handle_file_upload. Basically all file types are allowed. Shall we keep this?

We can make sure to set supported_content_types on GeneralFileCollection. That's the only one that would allow anything to be uploaded currently through those views.

Daverball

Looks good overall, but there's a couple of details we should iron out.

src/onegov/form/validators.py

src/onegov/org/forms/event.py

src/onegov/pas/forms/data_import.py

src/onegov/form/fields.py

Daverball · 2025-12-04T13:06:16Z

src/onegov/form/fields.py


    upload_field_class: type[UploadField] = UploadField
    upload_widget: Widget[UploadField] = UploadWidget()
+    validators = [WhitelistedMimeType()]


Same thing here

src/onegov/form/fields.py

Daverball · 2025-12-04T14:51:53Z

Should I completely remove type application/octet-stream ? It is mostly used in conjunction with application/zip

It's probably fine to remove it for now. There may however be the rare false positive for any files that cannot be identified correctly by libmagic. Generally pdfs, zips and any other binary file formats can end up as application/octet-stream, it's a generic catch-all content type for binary data if it couldn't be detected as anything else.

src/onegov/file/utils.py

Daverball · 2025-12-08T08:30:09Z

src/onegov/form/fields.py

            id=id,
            default=default,
            widget=widget,  # type:ignore[arg-type]
+            validators=[*(validators or ())],


I would still pass these to the unbound field, not the field list. There are still things like the file size validator that makes more sense on the individual field. It allows you to re-simplify the validators back to what they were before.

Without having it on the FieldList it does not work. On the unbound field the validators will be overwritten from the FieldList defaulted validators which results in the wrong mime type(s)

Are you sure this still happens? Before it made sense because you set a default validator on the class, now this should no longer happen, as long as you only pass the validators to the unbound field.

It's possible however that this also interacts badly with field dependencies, since there's no special casing for field lists. It might make sense to change how we implement field dependencies, since right now it is pretty fragile.

It fails when parsing a form from formcode, see test_parse_multiplefileinput.
I am looking into why it fails once I move the validators from the FieldList to the UnboundField.

My guess is that add_field skips something where as the new UploadMultipleField ctor does not

You're just looking for the validator in the wrong place in that test. The validator is on each field in the field list, not the field list itself. So I would get rid of this again and simplify the validators, otherwise the validator will run twice for each file. Once you move the validation from a validator in validators to post_validate you no longer will have to check for the presence of that validator anyways, since it's baked into the field itself.

Daverball · 2025-12-08T08:34:18Z

src/onegov/form/fields.py

+            if not any(isinstance(validator, WhitelistedMimeType)
+                for validator in validators
+            ):
+                validators.append(validator)


Make sure this logic still holds up when field dependencies are in play, I have a feeling it does not, since it will wrap everything in a conditional validator, so you probably need to detect that case, so you don't add an unconditional and potentially redundant mimetype validator.

It might be easier and more robust to store the mimetypes in an extra attribute and to override the post_validate hook, since validation_stopped is one of the parameters, which would be True if the dependency condition is unfulfilled.

So you can do something like the following in post_validate:

if not validation_stopped: WhitelistedMimeType(self.mimetypes)(form, self)

Actually nevermind, this has the same problem, StrictOptional only stops validation if there is no data. Although technically there should be no data, when the field is hidden, so it might not matter in the end.

I think it would still be a bit cleaner to move the validator into post_validate like in my code suggestion above, instead of trying to modify the given validator chain. If someone passes in a WhitelistValidator we can emit a warning or raise an exception to use the mimetypes parameter instead.

src/onegov/form/fields.py

Daverball · 2025-12-08T11:11:12Z

src/onegov/form/fields.py

        description: str = '',
        id: str | None = None,
-        default: Sequence[StrictFileDict] = (),
+        default: LaxFileDict | None = None,


Actually StrictFileDict was correct, just the sequence part was incorrect, I just remembered wrong which type was used here originally.

This still reads LaxFileDict instead of StrictFileDict, which is incorrect.

tests/onegov/form/test_fields.py

Co-authored-by: David Salvisberg <david.salvisberg@seantis.ch>

src/onegov/org/assets/js/upload.js

src/onegov/org/views/files.py

This reverts commit 0f7b210.

Tschuppi81 · 2025-12-18T14:21:01Z

src/onegov/org/assets/js/upload.js

-        var bar = $('<div class="progress"><span class="meter" style="width: 0%"></span></div>')
-            .attr('data-filename', file.name)
-            .prependTo(progress);
+        var bar = $(


Now the Unsupported Media Type error is visible

…load

Daverball

We're in a pretty good spot now. But we should clean up and refactor things a bit.

If mimetypes is an argument to upload fields, we no longer need to pass around instances of WhitelistedMimeType everywhere, we should use the new parameter and avoid inserting a validator object, instead let's rely on the post_validate method of Field, which we can override in UploadField (we don't need to override it in UploadMultipleField, but we should add an explicit mimetypes parameter, so we can easily access mimetypes from UploadMultipleField.mimetypes in addition to each subfield, this means this parameter should both be assigned to self.mimetypes and passed to the self.upload_field_class call.

Daverball · 2026-01-13T15:38:59Z

src/onegov/file/utils.py

-        'image/x-win-bitmap',
-        'image/x-pcx',
-        'image/x-portable-pixmap',
-        'image/x-tga'


Is there a good reason to change this list? Even if the current version of Pillow registers mime types for these, there's no guarantee a future version won't once again get rid of them, so I'd suggest leaving these, even if they're redundant.

Daverball · 2026-01-13T15:39:49Z

src/onegov/form/fields.py

        description: str = '',
        id: str | None = None,
-        default: Sequence[StrictFileDict] = (),
+        default: LaxFileDict | None = None,


This still reads LaxFileDict instead of StrictFileDict, which is incorrect.

Daverball · 2026-01-13T15:44:18Z

src/onegov/form/fields.py

+            if not any(isinstance(validator, WhitelistedMimeType)
+                for validator in validators
+            ):
+                validators.append(validator)


I think it would still be a bit cleaner to move the validator into post_validate like in my code suggestion above, instead of trying to modify the given validator chain. If someone passes in a WhitelistValidator we can emit a warning or raise an exception to use the mimetypes parameter instead.

Daverball · 2026-01-13T15:53:04Z

src/onegov/form/fields.py

            id=id,
            default=default,
            widget=widget,  # type:ignore[arg-type]
+            validators=[*(validators or ())],


You're just looking for the validator in the wrong place in that test. The validator is on each field in the field list, not the field list itself. So I would get rid of this again and simplify the validators, otherwise the validator will run twice for each file. Once you move the validation from a validator in validators to post_validate you no longer will have to check for the presence of that validator anyways, since it's baked into the field itself.

Daverball · 2026-01-13T15:54:16Z

src/onegov/form/validators.py

+        if isinstance(field.data, list):  # UploadMultipleField
+            for data in field.data:
+                if not data:
+                    continue  # in case of file deletion
+
+                self.validate_filesize(field, data)
+
+        else:
+            self.validate_filesize(field, field.data)
+
+    def validate_filesize(self, field: Field, data: dict[Any, Any]) -> None:
+        if data.get('size', 0) > self.max_bytes:


Suggested change

if isinstance(field.data, list): # UploadMultipleField

for data in field.data:

if not data:

continue # in case of file deletion

self.validate_filesize(field, data)

else:

self.validate_filesize(field, field.data)

def validate_filesize(self, field: Field, data: dict[Any, Any]) -> None:

if data.get('size', 0) > self.max_bytes:

if field.data.get('size', 0) > self.max_bytes:

Let's simplify this again, since it doesn't need to work on a FieldList, it's good enough if it works for each field in the list.

Daverball · 2026-01-13T16:01:33Z

tests/onegov/form/test_fields.py

+def assert_whitelisted_mimetype_validator(
+    field: UploadField | UploadMultipleField
+) -> None:
+
+    validator = find_validator(field, WhitelistedMimeType)
+    assert validator
+    assert validator.whitelist == WhitelistedMimeType.whitelist  # type:ignore[attr-defined]
+
+
+def find_validator(
+    field: Field | FileField,
+    cls: type
+) -> Validator[Any, Any] | None:
+    return next((v for v in field.validators if isinstance(v, cls)), None)


We can significantly simplify this when the whitelist is stored on the field (in fact you can probably get rid of this function and just replace it with the assertion):

Suggested change

def assert_whitelisted_mimetype_validator(

field: UploadField | UploadMultipleField

) -> None:

validator = find_validator(field, WhitelistedMimeType)

assert validator

assert validator.whitelist == WhitelistedMimeType.whitelist # type:ignore[attr-defined]

def find_validator(

field: Field | FileField,

cls: type

) -> Validator[Any, Any] | None:

return next((v for v in field.validators if isinstance(v, cls)), None)

def assert_whitelisted_mimetype_validator(

field: UploadField | UploadMultipleField

) -> None:

assert field.whitelist == WhitelistedMimeType.whitelist

Daverball · 2026-01-13T16:02:41Z

tests/onegov/form/test_parser.py

+    validator = find_validator(form['file'], WhitelistedMimeType)
+    assert validator
+    assert validator.whitelist == {  # type:ignore[attr-defined]
+        'application/msword', 'application/pdf'
+    }


Suggested change

validator = find_validator(form['file'], WhitelistedMimeType)

assert validator

assert validator.whitelist == { # type:ignore[attr-defined]

'application/msword', 'application/pdf'

}

assert form['file'].whitelist == { # type:ignore[attr-defined]

'application/msword', 'application/pdf'

}

Daverball · 2026-01-13T16:03:31Z

tests/onegov/form/test_parser.py

+    validator = find_validator(form['file'], WhitelistedMimeType)
+    assert validator
+    assert validator.whitelist == WhitelistedMimeType.whitelist  # type:ignore[attr-defined]


Suggested change

validator = find_validator(form['file'], WhitelistedMimeType)

assert validator

assert validator.whitelist == WhitelistedMimeType.whitelist # type:ignore[attr-defined]

assert form['file'].whitelist == WhitelistedMimeType.whitelist # type:ignore[attr-defined]

Daverball · 2026-01-13T16:04:34Z

tests/onegov/form/test_parser.py

+    validator = find_validator(form['files'], WhitelistedMimeType)
+    assert validator
+    assert validator.whitelist == {  # type:ignore[attr-defined]
+        'application/msword', 'application/pdf'
+    }


Suggested change

validator = find_validator(form['files'], WhitelistedMimeType)

assert validator

assert validator.whitelist == { # type:ignore[attr-defined]

'application/msword', 'application/pdf'

}

assert form['files'].whitelist == { # type:ignore[attr-defined]

'application/msword', 'application/pdf'

}

Daverball · 2026-01-13T16:04:58Z

tests/onegov/form/test_parser.py

+    validator = find_validator(form['my_files'], WhitelistedMimeType)
+    assert validator
+    assert validator.whitelist == WhitelistedMimeType.whitelist  # type:ignore[attr-defined]


Suggested change

validator = find_validator(form['my_files'], WhitelistedMimeType)

assert validator

assert validator.whitelist == WhitelistedMimeType.whitelist # type:ignore[attr-defined]

assert form['my_files'].whitelist == WhitelistedMimeType.whitelist # type:ignore[attr-defined]

Rework mime type white list

8699bc1

Tschuppi81 added 16 commits November 17, 2025 11:00

Ensure mime type validator for file fields in formcode

66ac404

Adds mime type validator by default

5ebe0fb

Merge branch 'master' into feature/ogc-2738-pentest-arbitrary-file-up…

9416abf

…load

Fix wrongly attached validators

7eed62a

Revert

4ab42df

improve tests and fix linter issues

cf252e2

Rework tests

6e9afd6

Fix file size validator and align

3f718d2

Set mime types for all upload fields

5572020

Merge branch 'master' into feature/ogc-2738-pentest-arbitrary-file-up…

a6bd385

…load

Fix missing validator

1293c8e

Cleanup unused import

af685f5

Add fixme

a50f6f0

Fix linting errors

458a82d

Extend test

8d99429

Fix more linter issues

c31d0e3

Tschuppi81 commented Dec 4, 2025

View reviewed changes

src/onegov/form/validators.py Show resolved Hide resolved

Tschuppi81 commented Dec 4, 2025

View reviewed changes

src/onegov/pas/forms/data_import.py Outdated Show resolved Hide resolved

Tschuppi81 requested a review from Daverball December 4, 2025 12:03

Daverball requested changes Dec 4, 2025

View reviewed changes

Tschuppi81 added 3 commits December 5, 2025 10:00

Remove validators from field list

eee980c

Add old ms office doc types

248819a

Remove non-standard svg type

5621994

Update supported image mime type

21274d1

Tschuppi81 commented Dec 5, 2025

View reviewed changes

src/onegov/file/utils.py Show resolved Hide resolved

Tschuppi81 added 3 commits December 5, 2025 11:16

Remove unused json validator

c4a1eef

Pass validators only to FieldList and introduce allowed mime types

e77bc49

Adjust tests

9a6a187

Daverball reviewed Dec 8, 2025

View reviewed changes

Tschuppi81 commented Dec 8, 2025

View reviewed changes

src/onegov/form/fields.py Show resolved Hide resolved

Tschuppi81 added 3 commits December 8, 2025 10:53

Fix wrong default value for UploadField

d122951

Fix linter issues

1e7b28b

Limit event import to excel kind of files

427563e

Daverball reviewed Dec 8, 2025

View reviewed changes

tests/onegov/form/test_fields.py Show resolved Hide resolved

Tschuppi81 and others added 4 commits December 8, 2025 07:50

Improve validator type

fa438e0

Co-authored-by: David Salvisberg <david.salvisberg@seantis.ch>

Verify file type for file collection upload

0f7b210

Adjust tests

e5552d6

Set default mime type white list for translator directory as well

a545477

Daverball reviewed Dec 16, 2025

View reviewed changes

src/onegov/org/assets/js/upload.js Outdated Show resolved Hide resolved

Daverball reviewed Dec 16, 2025

View reviewed changes

src/onegov/org/views/files.py Outdated Show resolved Hide resolved

Tschuppi81 added 3 commits December 18, 2025 09:40

Revert "Verify file type for file collection upload"

5f7bcb2

This reverts commit 0f7b210.

Make unsupported media type error visible on files view

5c54c06

Align upload columns with already uploaded files

f839024

Tschuppi81 commented Dec 18, 2025

View reviewed changes

Tschuppi81 added 5 commits December 29, 2025 14:33

Merge branch 'master' into feature/ogc-2738-pentest-arbitrary-file-up…

ef4b1fe

…load

Ensure required value error is shown for multiple upload widget

789e503

Extend tests with required upload field

f6f9d7c

Fix syntax error

c170834

Extend tests

00b1614

Tschuppi81 requested a review from Daverball January 13, 2026 14:41

Daverball requested changes Jan 13, 2026

View reviewed changes

Rework mime type white list #2198

Are you sure you want to change the base?

Rework mime type white list #2198

Uh oh!

Conversation

Tschuppi81 commented Nov 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

linear bot commented Nov 10, 2025

Uh oh!

codecov bot commented Nov 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Tschuppi81 commented Dec 4, 2025

Uh oh!

Tschuppi81 commented Dec 4, 2025

Uh oh!

Daverball commented Dec 4, 2025

Uh oh!

Daverball left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Daverball commented Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Tschuppi81 Dec 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Daverball Dec 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Tschuppi81 Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Daverball left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Tschuppi81 commented Nov 10, 2025 •

edited

Loading

codecov bot commented Nov 10, 2025 •

edited

Loading

Daverball commented Dec 4, 2025 •

edited

Loading

Tschuppi81 Dec 8, 2025 •

edited

Loading

Daverball Dec 8, 2025 •

edited

Loading

Tschuppi81 Dec 18, 2025 •

edited

Loading