Bugfix/unicode support by N64Core · Pull Request #77 · Emersont1/itchio

N64Core · 2023-03-24T23:38:38Z

Pertains to issue #74

This PR fixes a bug that occurs when downloading files that use UTF-8 characters, the characters they're meant to represent will get "smeared" into a different encoding format, the root cause I believe is this line in utils.py:

#filename_re = re.search(r'filename="(.+)"', cd)

What's going on here has something to do either with re.search returning an ascii formated byte string, or a weird consequence of passing the string through the assignment operator. My memory is a bit hazy on the specifics. in addition to fixing that issue the directory handling code was improved in other sections, using existing functions to replace blocks of code and making the variable names slightly more readable.

Also apologies for the weird duplicate commits on the branch, I'm a little rusty with git.

…racters. I don't remember exactly what the problem is, but I believe it had to do something with how the = operator and string objects don't play perfectly well together when it comes to unicode and need to be handled delicately. This fix is backported from a dead pull request from 2022 that I half remember working on.

…ode_support" This reverts commit a61f4c5, reversing changes made to 0c1e49d.

Emersont1

The code should be using pathlib, it makes it easier to use throughout the codebase

…d never be hit.

N64Core · 2023-03-30T04:00:35Z

The code should be using pathlib, it makes it easier to use throughout the codebase

Should be good now

itchiodl/game.py

itchiodl/utils.py

eibe1014 · 2023-03-31T14:49:03Z

I used this branch today (pip install https://github.com/N64Core/itchio/archive/refs/heads/bugfix/unicode_support.zip) and try to download my full lib and get into this issues:

When downloading:
Downloaded Moonlight on Roseville Beach | A Visitor's Guide to Cruisin! (180 of 6320) Traceback (most recent call last): File "C:\Users\*\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Users\*\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "C:\Users\*\AppData\Local\Programs\Python\Python310\lib\site-packages\itchiodl\downloader\__main__.py", line 75, in <module> main() File "C:\Users\*\AppData\Local\Programs\Python\Python310\lib\site-packages\itchiodl\downloader\__main__.py", line 71, in main lib.download_library(args.platform) File "C:\Users\*\AppData\Local\Programs\Python\Python310\lib\site-packages\itchiodl\library.py", line 108, in download_library for _ in r: File "C:\Users\*\AppData\Local\Programs\Python\Python310\lib\concurrent\futures\_base.py", line 609, in result_iterator yield fs.pop().result() File "C:\Users\*\AppData\Local\Programs\Python\Python310\lib\concurrent\futures\_base.py", line 439, in result return self.__get_result() File "C:\Users\*\AppData\Local\Programs\Python\Python310\lib\concurrent\futures\_base.py", line 391, in __get_result raise self._exception File "C:\Users\*\AppData\Local\Programs\Python\Python310\lib\concurrent\futures\thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "C:\Users\*\AppData\Local\Programs\Python\Python310\lib\site-packages\itchiodl\library.py", line 99, in dl g.download(self.login, platform) File "C:\Users\*\AppData\Local\Programs\Python\Python310\lib\site-packages\itchiodl\game.py", line 91, in download self.do_download(d, token) File "C:\Users\*\AppData\Local\Programs\Python\Python310\lib\site-packages\itchiodl\game.py", line 120, in do_download if md5 == d["md5_hash"]: KeyError: 'md5_hash'

When restart downloading (so it will use local files?):

Downloaded You're having a normal day and OH NO, GOBLINS! (8 of 6320) Traceback (most recent call last): File "C:\Users\*\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Users\*\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "C:\Users\*\AppData\Local\Programs\Python\Python310\lib\site-packages\itchiodl\downloader\__main__.py", line 75, in <module> main() File "C:\Users\*\AppData\Local\Programs\Python\Python310\lib\site-packages\itchiodl\downloader\__main__.py", line 71, in main lib.download_library(args.platform) File "C:\Users\*\AppData\Local\Programs\Python\Python310\lib\site-packages\itchiodl\library.py", line 108, in download_library for _ in r: File "C:\Users\*\AppData\Local\Programs\Python\Python310\lib\concurrent\futures\_base.py", line 609, in result_iterator yield fs.pop().result() File "C:\Users\*\AppData\Local\Programs\Python\Python310\lib\concurrent\futures\_base.py", line 446, in result return self.__get_result() File "C:\Users\*\AppData\Local\Programs\Python\Python310\lib\concurrent\futures\_base.py", line 391, in __get_result raise self._exception File "C:\Users\*\AppData\Local\Programs\Python\Python310\lib\concurrent\futures\thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "C:\Users\*\AppData\Local\Programs\Python\Python310\lib\site-packages\itchiodl\library.py", line 99, in dl g.download(self.login, platform) File "C:\Users\*\AppData\Local\Programs\Python\Python310\lib\site-packages\itchiodl\game.py", line 91, in download self.do_download(d, token) File "C:\Users\*\AppData\Local\Programs\Python\Python310\lib\site-packages\itchiodl\game.py", line 143, in do_download filename.rename(old_dir / f"{timestamp}-{filename}") AttributeError: 'str' object has no attribute 'rename'

…ing old files

N64Core · 2023-04-15T03:40:13Z

I used this branch today (pip install https://github.com/N64Core/itchio/archive/refs/heads/bugfix/unicode_support.zip) and try to download my full lib and get into this issues:

When downloading: Downloaded Moonlight on Roseville Beach | A Visitor's Guide to Cruisin! (180 of 6320) Traceback (most recent call last): File "C:\Users\*\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Users\*\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "C:\Users\*\AppData\Local\Programs\Python\Python310\lib\site-packages\itchiodl\downloader\__main__.py", line 75, in <module> main() File "C:\Users\*\AppData\Local\Programs\Python\Python310\lib\site-packages\itchiodl\downloader\__main__.py", line 71, in main lib.download_library(args.platform) File "C:\Users\*\AppData\Local\Programs\Python\Python310\lib\site-packages\itchiodl\library.py", line 108, in download_library for _ in r: File "C:\Users\*\AppData\Local\Programs\Python\Python310\lib\concurrent\futures\_base.py", line 609, in result_iterator yield fs.pop().result() File "C:\Users\*\AppData\Local\Programs\Python\Python310\lib\concurrent\futures\_base.py", line 439, in result return self.__get_result() File "C:\Users\*\AppData\Local\Programs\Python\Python310\lib\concurrent\futures\_base.py", line 391, in __get_result raise self._exception File "C:\Users\*\AppData\Local\Programs\Python\Python310\lib\concurrent\futures\thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "C:\Users\*\AppData\Local\Programs\Python\Python310\lib\site-packages\itchiodl\library.py", line 99, in dl g.download(self.login, platform) File "C:\Users\*\AppData\Local\Programs\Python\Python310\lib\site-packages\itchiodl\game.py", line 91, in download self.do_download(d, token) File "C:\Users\*\AppData\Local\Programs\Python\Python310\lib\site-packages\itchiodl\game.py", line 120, in do_download if md5 == d["md5_hash"]: KeyError: 'md5_hash'

When restart downloading (so it will use local files?):

Downloaded You're having a normal day and OH NO, GOBLINS! (8 of 6320) Traceback (most recent call last): File "C:\Users\*\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Users\*\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "C:\Users\*\AppData\Local\Programs\Python\Python310\lib\site-packages\itchiodl\downloader\__main__.py", line 75, in <module> main() File "C:\Users\*\AppData\Local\Programs\Python\Python310\lib\site-packages\itchiodl\downloader\__main__.py", line 71, in main lib.download_library(args.platform) File "C:\Users\*\AppData\Local\Programs\Python\Python310\lib\site-packages\itchiodl\library.py", line 108, in download_library for _ in r: File "C:\Users\*\AppData\Local\Programs\Python\Python310\lib\concurrent\futures\_base.py", line 609, in result_iterator yield fs.pop().result() File "C:\Users\*\AppData\Local\Programs\Python\Python310\lib\concurrent\futures\_base.py", line 446, in result return self.__get_result() File "C:\Users\*\AppData\Local\Programs\Python\Python310\lib\concurrent\futures\_base.py", line 391, in __get_result raise self._exception File "C:\Users\*\AppData\Local\Programs\Python\Python310\lib\concurrent\futures\thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "C:\Users\*\AppData\Local\Programs\Python\Python310\lib\site-packages\itchiodl\library.py", line 99, in dl g.download(self.login, platform) File "C:\Users\*\AppData\Local\Programs\Python\Python310\lib\site-packages\itchiodl\game.py", line 91, in download self.do_download(d, token) File "C:\Users\*\AppData\Local\Programs\Python\Python310\lib\site-packages\itchiodl\game.py", line 143, in do_download filename.rename(old_dir / f"{timestamp}-{filename}") AttributeError: 'str' object has no attribute 'rename'

I need to test this, but I think I understand what's going on here, it's trying to call rename on an object that still purely a string and not a Path

64Core and others added 13 commits March 18, 2023 21:18

Fixes an issue prevents writing paths that end in '...' or similar

c95b75b

Add Framework for human readable folder structures

2310f0f

Removed -h argument due to conflict

624eec9

Game Folders / Files now respond to --human-folders

4ddff68

Update Readme.md to include new argument

767895b

Remove -h from arg logic because it's not used and collides with help

ac09641

Remove redundant include

7c7a70d

Fix includes again

0c1e49d

Merge remote-tracking branch 'upstream/main' into bugfix/unicode_support

a61f4c5

Revert "Merge remote-tracking branch 'upstream/main' into bugfix/unic…

1eb2f69

…ode_support" This reverts commit a61f4c5, reversing changes made to 0c1e49d.

Fix linting conflicts

e7c28a8

Fix linting conflicts

c0a69ad

Emersont1 requested changes Mar 25, 2023

View reviewed changes

N64Core added 4 commits March 29, 2023 22:38

PR Emersont1#77 Convert path handling back to using pathlib

dbc42dd

PR Emersont1#77 Linting Fixes

85d7059

PR Emersont1#77 More Linting Fixes, Comment out dead branch that woul…

cd5f7bb

…d never be hit.

PR Emersont1#77 Tidying Pathlib code

6de39b6

N64Core requested a review from Emersont1 March 30, 2023 04:03

Emersont1 reviewed Mar 30, 2023

View reviewed changes

itchiodl/game.py Outdated Show resolved Hide resolved

Emersont1 reviewed Mar 30, 2023

View reviewed changes

itchiodl/game.py Outdated Show resolved Hide resolved

Emersont1 reviewed Mar 30, 2023

View reviewed changes

itchiodl/utils.py Outdated Show resolved Hide resolved

PR Emersont1#77 address second comment, potential bug regarding renam…

a5a09d1

…ing old files

N64Core requested a review from Emersont1 April 15, 2023 03:42

rheaplex added a commit to rheaplex/itchio that referenced this pull request Jan 25, 2026

Merge fixes from Emersont1#77

9052a33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bugfix/unicode support #77

Bugfix/unicode support #77
N64Core wants to merge 18 commits intoEmersont1:mainfrom
N64Core:bugfix/unicode_support

N64Core commented Mar 24, 2023

Uh oh!

Emersont1 left a comment

Uh oh!

N64Core commented Mar 30, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

eibe1014 commented Mar 31, 2023

Uh oh!

N64Core commented Apr 15, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

N64Core commented Mar 24, 2023

Uh oh!

Emersont1 left a comment

Choose a reason for hiding this comment

Uh oh!

N64Core commented Mar 30, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

eibe1014 commented Mar 31, 2023

Uh oh!

N64Core commented Apr 15, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants