Unicode Decode Errors

[matou]

I still have Unicode Decode Errors when there are non-ascii characters in filenames.

[cmichi]

I had these problems too, I think changing the code to this may help you:

 135     print "these files are the same: "
 136     for path in db:
 137         path = unicode(path[0]).encode('utf-8')

But actually it is strange that we have to encode the path back to utf-8 (after storing in in utf-8). I think there has to be a better solution than to convert everything back and forth (collations?).

[matou]

According to [0] the default type of text in a sqlite db ist unicode in python. I thought this should be enough. But apperantly it isn't. Maybe the "utf-8" part is missing. I don't know enough about encoding to be able to judge that. I will test your suggestion (and do further reading of [0]) as soon as I find time.

Thx for your help!

[0] http://docs.python.org/library/sqlite3.html#introduction

[matou]

Using python3 solved some problems. The current one appears when writing to the database (on an openBSD machine):

Traceback (most recent call last):
  File "/home/matou/duplicatefiles.py", line 104, in <module>
    db.execute("INSERT INTO files VALUES(?, ?)", (size, f))
UnicodeEncodeError: 'utf-8' codec can't encode character '\udcf1' in position 110: surrogates not allowed