你能用一个朴素的 try-except 块安全地读取 utf8 和 latin1 文件吗？

问：

我相信任何有效的 latin1 字符都会被 Python 的 utf8 编码器正确解释或抛出错误。因此，我声称，如果您只使用 utf8 文件或 latin1 文件，您可以安全地编写以下代码来读取这些文件，而不会以 Mojibake 结束：

from pathlib import Path

def read_utf8_or_latin1_text(path: Path, args, kwargs):
    try:
        return path.read_text(encoding="utf-8")
    except UnicodeDecodeError:
        return path.read_text(encoding="latin1")

我在这个大型字符数据集上测试了这个假设，发现它经得起推敲。情况总是如此吗？

输入：

import requests

insanely_many_characters = requests.get(
    "https://github.com/bits/UTF-8-Unicode-Test-Documents/raw/master/UTF-8_sequence_unseparated/utf8_sequence_0-0x10ffff_including-unassigned_including-unprintable-asis_unseparated.txt"
).text


print(
    f"\n=== test {len(insanely_many_characters)} utf-8 characters for same-same misinterpretations ==="
)
for char in insanely_many_characters:
    if (x := char.encode("utf-8").decode("utf-8")) != char:
        print(char, x)


print(
    f"\n=== test {len(insanely_many_characters)} latin1 characters for same-same misinterpretations ==="
)
latinable = []
nr = 0
for char in insanely_many_characters:
    try:
        if (x := char.encode("latin1").decode("latin1")) != char:
            print(char, x)
        latinable.append(char)
    except UnicodeEncodeError:
        nr += 1
if nr:
    print(f"{nr} characters not in latin1 set")

print('found the following valid latin1 characters: """\n' + "".join(latinable) + '\n"""')

print(
    f"\n=== test {len(latinable)} latin1 characters for utf-8 Mojibake ==="
)
for char in latinable:
    try:
        if (x := char.encode("latin1").decode("utf-8")) != char:
            print(char, x)
    except UnicodeDecodeError:
        pass

输出：

=== test 1111998 utf-8 characters for same-same misinterpretations ===

=== test 1111998 latin1 characters for same-same misinterpretations ===
1111742 characters not in latin1 set
found the following latin1 characters: """
    
 !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~€‚ƒ„…†‡ˆ‰Š‹ŒŽ‘’“”•–—˜™š›œžŸ ¡¢£¤¥¦§¨©ª«¬­®¯°±²³´µ¶·¸¹º»¼½¾¿ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõö÷øùúûüýþÿ
"""

=== test 256 latin1 characters for utf-8 Mojibake ===

补遗：

我看到我完全忘记了测试 latin1 字符的序列，而只测试了单个字符。通过添加此测试：

print(
    f"\n=== test {len(latinable)} latin1 sequences wrongly interoperable by utf-8 ==="
)
for char1 in latinable:
    for char2 in latinable:
        try:
            if (x := (char1 + char2).encode("latin1").decode("utf-8")) != char1 + char2:
                print(char1 + char2, x)
        except UnicodeDecodeError:
            pass

我最终生成了许多 utf-8 Mojibake（总共 1920 个实例），这是我假设的反例！

=== test 256 latin1 sequences wrongly interoperable by utf-8 ===
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
   
¡ ¡

 ⋮