Push v2 integration #262

MarcelGeo · 2025-08-15T14:14:00Z

Added ➕

integration to v2 push chunks mechanism - spliting some factory methods for UploadJob and methods in UploadQueueItem to v2 versions (we can discuss that, if it's ok)
ignore 409 error in push check_only to prevent stopping upload (?)
added retry mechanism for chunks - 2 attempts after 5 seconds
added retry mechanism for sync command - compatible with v1 and v2push version using checks for status codes, server codes and response texts. This retry is doing n attempts every 5 seconds. Retry is doing if
- 409 AntherUploadRunning, ProjectVersionExists
- 429 - reate limit
- for v1 if 400 and specific text in detail
introducing uploaded chunks cache which is used by v2 endpoints and cleaned by successful update
introducing LocalChanges class to handle dictionaries with changes (which nobody knows what's there) - it's used just in push methods to finalize push (time for improvement)
introducing server_features which will handle in future all feature flags from server
getting project info directly from v2 POST versions

Updated tests:

added remove only files test
added test for low level sync methods
added test for retry mechanism

❓

do we want to check if project is on the same version like a server before push? I added there DISCUSSION comment @wonder-sk - there so mutch faiing tests in that case
any suggestions guys on how to make proper tests for failing (409, 422) ?

❗

not sure what to do with progress bar in sync command, as it's not possible to update maximum size (length) of that on the fly.
some tests will fail, becasue we are there using v2 api which is not in app.dev yet

Chunks cache

Logging following to logs

2025-08-13 14:09:10,282 will upload 1 items with total size 92025-08-13 14:09:10,282 Starting upload chunks for project 459a8014-9c05-439d-9b6b-2687cfdfb4fa2025-08-13 14:09:10,283 Uploading /home/marcelkocisek/Documents/mm-workspaces/localhost/hejj7/hej3.txt part=02025-08-13 14:09:10,283 Chunk 76ade537-12f2-4e0e-b25b-c22929f10c5a already uploaded, skipping upload

Retry of sync_project

2025-08-15 13:36:32,558 Attempting sync process due to conflicts between server and local directory or another user is syncing.

Workflow

- attempts of failing chunks =- init for sync_project

…r local copy

- fixing update chunks for files with the same checksum - added minimal version of sync command

Introduce mapping based on chunk_id (generated by client) and server_chunk_id generated by server.

varmar05 · 2025-08-26T06:53:37Z

mergin/cli.py

+    upload_job = None
+    length = 1
+    try:
+        def on_progress(increment, push_job):


As a user I would probably like to see individual steps, each with progress bar rather than one global adaptive bar.

Yes, we can handle this progress bar is tricky to do... let me take a look at this after finish of logic

It's not possible to create addaptive bar in click

mergin/client.py

varmar05 · 2025-08-26T07:36:35Z

mergin/local_changes.py

+    version: Optional[str] = None
+    # some functions (MerginProject.compare_file_sets) are adding history dict to the change from project info
+    history: Optional[dict] = None
+    # some functions (MerginProject.compare_file_sets) are adding location dict to the change from project info


This is probably a bug, as it comes in server response but it should not.

Not sure here, as client is using some things from this dictinary.

sorry, I meant location key

mergin/local_changes.py

mergin/client_push.py

varmar05 · 2025-08-26T10:51:09Z

mergin/client_push.py

-        with open(self.file_path, "rb") as file_handle:
-            file_handle.seek(self.chunk_index * UPLOAD_CHUNK_SIZE)
-            data = file_handle.read(UPLOAD_CHUNK_SIZE)
+        for chunk_index, chunk_id in enumerate(change.chunks):


Do we want to rely here on chunks? Maybe we should take files and split to chunks here and not before.

We need chunk index and chunk id to send in v1 endpoints to push. It's coming from MerginProject.get_push_changes huge logic over chunks, where not just files are splitted, but also diffs. Not sure if we want to move it here.

mergin/client_push.py

+ black staff

mergin/client_push.py

mergin/common.py

@varmar05

- comment @varmar05

- +v1 is raising error just from push start (push_project_async), because there are not chunks caches

prevent conflcted copy fix

…r chunks cache on v1 supported server (#276)

MarcelGeo added 7 commits August 8, 2025 13:39

Initial dirty version of push v2

13b8ccb

tweak _do_upload running outside of class to prevent threads mix

216c3bd

add local changes

57c6bf4

- attempts of failing chunks =- init for sync_project

make tests runnable with checking current version - Please update you…

88f00ea

…r local copy

Cleanup and fix getting diff from dictionary to be more safe

a2cfe62

fixes for missing schema

054a818

- fixing update chunks for files with the same checksum - added minimal version of sync command

cleanup

7ba0971

MarcelGeo requested review from wonder-sk, varmar05 and harminius August 15, 2025 14:14

Retry sync job using

42c34a2

Introduce mapping based on chunk_id (generated by client) and server_chunk_id generated by server.

varmar05 requested changes Aug 26, 2025

View reviewed changes

tomasMizera mentioned this pull request Aug 27, 2025

Project sync fails when uploading a big file MerginMaps/qgis-plugin#782

Closed

MarcelGeo added 3 commits September 3, 2025 12:50

Address comstic comments from @varmar05 and some from @wonder-sk

0128a06

+ black staff

fix uploading 0 size files

df88376

handle project info directly from v2 projects versions

28aa757

tomasMizera reviewed Sep 3, 2025

View reviewed changes

mergin/client_push.py Outdated Show resolved Hide resolved

Add unit tests and tunning of retry mechanism

efd7ed6

varmar05 reviewed Sep 4, 2025

View reviewed changes

mergin/common.py Show resolved Hide resolved

MarcelGeo added 5 commits September 5, 2025 11:30

Update test calling count measurement

29292da

- comment @varmar05

add test also for v1

c6dafca

- +v1 is raising error just from push start (push_project_async), because there are not chunks caches

Drop versions and permissions check

e54bde8

get rid of project info from editor

57e854b

black

747ea46

prevent conflcted copy fix

MarcelGeo mentioned this pull request Sep 9, 2025

Create unified api interface for api integration #270

Open

MarcelGeo requested review from tomasMizera and varmar05 September 10, 2025 09:35

MarcelGeo added 3 commits September 10, 2025 16:08

Merge remote-tracking branch 'origin/master' into push-v2-integration

39cea8b

temp_dir fix

b6cfa3e

Close connection to make runnable test in new python and skip test fo…

77b82e9

…r chunks cache on v1 supported server (#276)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Push v2 integration #262

Push v2 integration #262

Uh oh!

MarcelGeo commented Aug 15, 2025 •

edited

Loading

Uh oh!

varmar05 Aug 26, 2025

Uh oh!

MarcelGeo Sep 3, 2025

Uh oh!

MarcelGeo Sep 3, 2025

Uh oh!

Uh oh!

Uh oh!

varmar05 Aug 26, 2025

Uh oh!

MarcelGeo Sep 3, 2025

Uh oh!

varmar05 Sep 4, 2025

Uh oh!

Uh oh!

Uh oh!

varmar05 Aug 26, 2025

Uh oh!

MarcelGeo Sep 3, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Push v2 integration #262

Are you sure you want to change the base?

Push v2 integration #262

Uh oh!

Conversation

MarcelGeo commented Aug 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MarcelGeo commented Aug 15, 2025 •

edited

Loading