partial garbled text

[yoliax]

How to detect partial garbled text?

部分乱码-1.pdf

[JorjMcKie]

This is not an issue but a question and thus belongs to the Forum or into the Discussions tab.

[JorjMcKie]

The default extraction flags return the glyph identifier if the glyph has no pointer to the originating Unicode. Simply take the default extraction flag and set off this flags bit.
For example flags=pymupdf.TEXTFLAGS_TEXT & ~pymupdf.TEXT_USE_CID_FOR_UNKNOWN_UNICODE. Or simply flags=0 - in most cases ok too.

Doing that will return � for those glyphs.

[JorjMcKie]

You can check the extracted text for instance via chr(0xfffd) in text. Or text.count(chr(0xfffd)).
The INVALID UNICODE value is 0xFFFD.

[yoliax]

@JorjMcKie
Besides extracting the text, I also need to get the coordinates. Since PyMuPDF can extract a document in the order of blocks → lines → words, I would like to ask: is it possible to determine if a specific line contains abnormal text?

If this can be determined, I could then use OCR results for that line.

[JorjMcKie]

This is certainly possible!
There is even an example script available which you can use for your purpose.
This requires Tesseract is installed. For your language you should also install the respective language support and specify this in the call.

[yoliax]

import time
import pymupdf

start_time = time.time()
INVALID_UNICODE = chr(0xFFFD) 
with pymupdf.open("/root/workdir/data/badcase/部分乱码-1.pdf") as doc:
    for page in doc:
        blocks = page.get_text("dict", flags=0)["blocks"]
        for b in blocks:
            for l in b["lines"]:
                for s in l["spans"]:
                    text = s["text"]
                    if INVALID_UNICODE in text:  # invalid characters encountered!
                        print(text)

I used the example you provided, but I found that this document does not detect INVALID_UNICODE = chr(0xFFFD).

[JorjMcKie]

You are right, the problems in your PDF are not of this kind:
Only if back-translation info is completely missing my recipe will work.
The fonts in your PDF however provide Unicode values for all glyphs under text extraction. In some cases however, the extracted Unicodes do not represent the visual appearance (i.e. your expectations).
There is no way to detect this, sorry.

[yoliax]

Thank you for the detailed explanation.

[yoliax]

@JorjMcKie
I have another question, why are the rendered images displayed correctly, but the extracted text appears garbled?

[JorjMcKie]

When you read a PDF using some PDF viewer you do not read Unicode values - never!
What you read are actually little vector graphics, called "glyphs". The small programs that write these vector graphics are the main part of any font (well, almost any font).
The glyph programs within a font are accessible by number within a font. During PDF creation, when you write text "ABC" using a certain font, the Unicode value for "A" (integer 65/0x41) is used to find the glyph program that produces the desired image. At this point, please note that the produced image need not look similar to an "A" at all ... it could be just any drawing you can imagine.

The code in the page object that is responsible for the visual appearance (/Contents object) contains something looking like this for each piece of text:

BT 0.016 Tc 11.83827 0 0 11.83827 255.4933 807.2249 Tm /C2 1 Tf <1f131f440fef1e5b0dbb13b8146a>
Tj 0 Tc ET BT 8.830189 0 0 8.830189 215.903 795.0422 Tm /Ty1 1 Tf (&) Tj ET

The stuff in brackets "<1f131f440fef1e5b0dbb13b8146a>" is the written text (7 glyphs in this case). It represents glyph numbers - not Unicode values, four bytes per glyph. What we read is what the glyph program with the number 1f13 draws.
The reverse mapping "glyph number to Unicode", which Unicode value was used to compute the value 1f13 is optional information: It may or may not be contained in the font, or it may simply be wrong, or it may not be a 1:1 relationship, etc., etc.
Whatever the situation: the page will look nice and be readable by the user.