Problem writing PNG

[dlobba]

Good morning,
I am experiencing an issue on writing a PNG from a PDF page.
I am on Ubuntu 17.10, I compiled manually MuPDF and install PyMuPDF as said in the wiki, and everything went good.
I am able to import fitz, access metadata and so on. But I cannot manage to export PNG, the result gets corrupted (grayscale lines on all the page, all images and text barely readable).
I think is a dependency issue, but I don't know how to solve it. I installed all the dependencies also on the repository to be sure, and I commented the external libraries in setup.py as the wiki said (and I also changed the branch according the mupdf version, 1.12.0).

the following code produce the result explained above:

>>> import fitz
>>> doc = fitz.open("my_fantastic_pdf.pdf")
>>> pix = doc[1].getPixmap()
>>> pix.writeImage("test.png")

The return value of the last method is 0 if can be useful.

Here is an example of the result:

[JorjMcKie]

We had this issue a few times. It usually indicated a mismatch between MuPDF's versions of the thirdparty imaging libraries (openjpeg, libjpeg, etc.) and any already installed on your system. It is essential that MuPDF is in sync with these libraries, so they should be installed in sync and exclusively used when building MuPDF.

[JorjMcKie]

I have tested the function in Ubuntu 17.10 x64: everything worked just fine.

[JorjMcKie]

Please also have another look at this (updated) Wiki.

[dlobba]

Thank you for the reply!
Unfortunately the problem still occurs. I've done the process several times, but still cannot figure out what the problem is.
While editing the setup.py I noticed that no folder named '/usr/local/thirdparty/zlib' exists when launching the setup program, although I managed to correctly build MuPDF as described in the wiki (downloading the source file, changing Makerules and so on).

Is it possible that another library is installed in the system and is causing the problem? Is there any way to check for that?
Thank you for the support

[JorjMcKie]

zlib is contained in MuPDF source. It is required by PyMuPDF directly, too, because I do compressions at various places.
In my Ubuntu, I had to copy this folder manually to the shown location. PyMuPDF generation will fail if it doesn't find the zlib headers (or maybe use one with the wrong version ...).

Being far from an Ubuntu expert, I think you are right when you suspect that some lib or include library on your system (outside MuPDF) is causing the issue. Maybe the repo owner @rk700 can better help you locating what and where it is. It must be a problem with some image library as mentioned above - but potentially also zlib?

When generating MuPDF: did you notice the new make parameter HAVE_GLUT?

[dlobba]

I noticed the HAVE_GLUT parameter, I copy-pasted the make command to be sure.
I also tried to move the thirdparty folder in /usr/local/include but again, no changes.

To be sure, everytime I cleaned the build folder in PyMuPDF so to perform a new generation of the library.

I write down the configurations and the steps I have done, in order to have a clear
view of the context.
My actual configuration (which shows the PNG problem) is:

File setup.py (branch is PyMuPDF 12.2):

if sys.platform.startswith('linux'):
    module = Extension('fitz._fitz', # name of the module
                       ['fitz/fitz_wrap.c'], # C source file
                       include_dirs=[  # we need the path of the MuPDF and zlib headers
                                     #'/usr/include/mupdf',
                                     '/usr/local/include/mupdf',
                                     '/usr/local/thirdparty/zlib',
                                    ],
                       library_dirs=['/usr/local/lib'],
                       libraries=[
                           'mupdf',
                           #'crypto', #openssl is required by mupdf on archlinux
                           'jbig2dec', 'openjp2', 'jpeg', 'freetype',
                           'mupdfthird',
                           ], # the libraries to link with
                      )

I actually tried also to comment out the line related to jbig2dec etc. but no changes.

File Makerules (MuPDF 12.0 downloaded as indicated in the wiki), starting from the top:

# Configuration for the Makefile

OS ?= $(shell uname)
OS := $(OS:MINGW%=MINGW)
OS := $(OS:MSYS%=MINGW)
OS := $(OS:Windows_NT=MINGW)
OS := $(OS:Darwin=MACOS)

HAVE_LIBDL ?= yes

CFLAGS += -Wall -Wdeclaration-after-statement -fPIC

Command issued to build and install MuPDF (always as root, so after doing sudo -i):

make HAVE_X11=no HAVE_GLFW=no HAVE_GLUT=no prefix=/usr/local
make HAVE_X11=no HAVE_GLFW=no HAVE_GLUT=no prefix=/usr/local install

Then I moved the thirdparty folder:

<mupdf-dir>/build/thirdparty/ -t /usr/local/include/

Then I ran setup.py:

python3 setup.py build
python3 setup.py install

This should be everything.
Again, thank you for the great help and support.

[rk700]

How about running ldd <path to _fitz.so>? It would show which libraries are used and maybe we can figure out some mismatch.

[dlobba]

Here it is, It seems that the bindings are done with the global libraries...

diego@lubuntu-virtualbox:/usr/local/lib/python3.6/dist-packages$ ldd fitz/_fitz.cpython-36m-x86_64-linux-gnu.so 
	linux-vdso.so.1 =>  (0x00007ffdb1b8b000)
	libjbig2dec.so.0 => /usr/lib/x86_64-linux-gnu/libjbig2dec.so.0 (0x00007f99d21f8000)
	libjpeg.so.9 => /usr/lib/x86_64-linux-gnu/libjpeg.so.9 (0x00007f99d1fbc000)
	libfreetype.so.6 => /usr/lib/x86_64-linux-gnu/libfreetype.so.6 (0x00007f99d1d09000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f99d1aea000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f99d170a000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f99d14ed000)
	libpng16.so.16 => /usr/lib/x86_64-linux-gnu/libpng16.so.16 (0x00007f99d12bb000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f99d48dd000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f99d0f65000)

I then tried to manually specify the include directory to look for each of the libraries.

if sys.platform.startswith('linux'):
    module = Extension('fitz._fitz', # name of the module
                       ['fitz/fitz_wrap.c'], # C source file
                       include_dirs=[  # we need the path of the MuPDF and zlib $
                                     #'/usr/include/mupdf',
                                     '/usr/local/include/mupdf',
                                     '/usr/local/thirdparty/zlib'
                                    ],
                       library_dirs=['/usr/local/lib', '/usr/local/include',
                                     '/usr/local/include/thirdparty/freetype',
                                     '/usr/local/include/thirdparty/harfbuzz',
                                     '/usr/local/include/thirdparty/jbig2dec',
                                     '/usr/local/include/thirdparty/libjpeg',
#                                    '/usr/local/thirdparty/mujs',
                                     '/usr/local/include/thirdparty/openjpeg'
                        ],
                       libraries=[
                           'mupdf',
                           #'crypto', #openssl is required by mupdf on archlinux
                           'jbig2dec', 'openjp2', 'jpeg', 'freetype',
                           'mupdfthird',
                           ], # the libraries to link with
                      )

But still it doesn't works...

[rk700]

Hmm, then what's the output of MuPDF itself by running mutool draw -o <PNG file> <PDF file> <page num>?

It may help to try on different PDF samples.

[JorjMcKie]

How about trying this ldd thing on mupdfthird, I still have the suspicion that something is going wrong already there ...

[dlobba]

Sorry for the delay.
So, after testing the command mutool draw the result was the expected page rendered, so at least mupdf seems to have been compiled successfully.

While on ldd command, I couldn't do it, since the generated library is static.

diego@lubuntu-virtualbox:/usr/local/lib$ ls
libmupdf.a  libmupdfthird.a  python2.7  python3.6
diego@lubuntu-virtualbox:/usr/local/lib$ ldd libmupdfthird.a 
	not a dynamic executable
diego@lubuntu-virtualbox:/usr/local/lib$ 

[joachim55]

according to your setup.py, you advise python in libraries= to use 'jbig2dec', 'openjp2', 'jpeg', 'freetype', which are system libraries, whereas in library_dirs you choose the libs from thirdparty folder.
In this way python uses the system libs as your ldd fitz/_fitz.cpython-36m-x86_64-linux-gnu.so shows.
If you uncomment the line 'jbig2dec', 'openjp2', 'jpeg', 'freetype', ldd output is

# ldd _fitz.cpython-35m-x86_64-linux-gnu.so 
        linux-vdso.so.1 =>  (0x00007ffd35bf4000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f141867e000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f14182b4000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f141ae03000)

and python uses the thirdparty libs for compiling PyMuPDF.
System I used is Linux Mint, so compatible to Ubuntu.

if sys.platform.startswith('linux'):
    module = Extension('fitz._fitz', # name of the module
                       ['fitz/fitz_wrap.c'], # C source file
                       include_dirs=[  # we need the path of the MuPDF and zlib $
                                     #'/usr/include/mupdf',
                                     '/usr/local/include/mupdf',
                                     '/usr/local/thirdparty/zlib'
                                    ],
                       library_dirs=['/usr/local/lib', '/usr/local/include',
                                     '/usr/local/include/thirdparty/freetype',
                                     '/usr/local/include/thirdparty/harfbuzz',
                                     '/usr/local/include/thirdparty/jbig2dec',
                                     '/usr/local/include/thirdparty/libjpeg',
#                                    '/usr/local/thirdparty/mujs',
                                     '/usr/local/include/thirdparty/openjpeg'
                        ],
                       libraries=[
                           'mupdf',
                           #'crypto', #openssl is required by mupdf on archlinux
                           'jbig2dec', 'openjp2', 'jpeg', 'freetype', **#uncomment or delete this line**
                           'mupdfthird',
                           ], # the libraries to link with
                      )

[dlobba]

Thank you for your reply @joachim55, but unfortunately I have already tried that out.
Still, I've the changed the setup.py and I've seen whether something has been changed, but no luck.

The output of the ldd command is exactly as yours, anyway.
Thank you for your patience, all of you.

[JorjMcKie]

As a non-Unix expert I am the last to profoundly contribute to this.
But because your output protocol suggests you are using a virtual machine for your system, just as I (sucessfully) did:
Why not set up a new, virgin one and install nothing in it except MuPDF and PyMuPDF? This is how it came to work in my case ...

[dlobba]

Here I am!! I have configured a new vm,
here is the most significant part of the history:

    3  wget https://mupdf.com/downloads/mupdf-1.12.0-source.tar.gz
    4  tar xvzf mupdf-1.12.0-source.tar.gz 
    6  cd mupdf-1.12.0-source/
    7  nano Makerules # added -fPIC here 
    9  apt-get install make
   15  apt-get install g++
   16  make HAVE_X11=no HAVE_GLFW=no HAVE_GLUT=no prefix=/usr/local
   17  make HAVE_X11=no HAVE_GLFW=no HAVE_GLUT=no prefix=/usr/local install
   19  cp -r build/release/thirdparty/ /usr/local/
   21  git clone -b 1.12.2 https://github.com/rk700/PyMuPDF.git
   22  cd PyMuPDF/
   23  nano setup.py 
   25  apt-get install python3-dev 
   27  apt-get install python3-pip
   39  apt-get install zlib1g-dev # needed to solve missing zlib.h error
   42  python3 setup.py build
   43  python3 setup.py install

still no luck. Import is fine but generate png is garbled.
In the history I removed my tries to change PyMuPDF version (setting to 12.2.0) to
see whether something would change.

I will a try installing a different distro (the current is lubuntu), maybe something changes
(I don't know what honestly).