Pubmed, bioRxiv, medRxiv and also local pdf files are not working

[ahyaha]

Pubmed, bioRxiv, medRxiv and also local pdf files are not working

example

https://www.biorxiv.org/content/10.1101/2024.09.23.614482v1.full.pdf

It shows this error

urllib.error.HTTPError: HTTP Error 403: Forbidden
Traceback:

File "/app/app.py", line 185, in
app.run()
File "/app/app.py", line 76, in run
output = self.build(url)
File "/usr/local/lib/python3.9/site-packages/streamlit/runtime/caching/cache_utils.py", line 174, in call
return self._cached_func(self._instance, *args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/streamlit/runtime/caching/cache_utils.py", line 218, in call
return self._get_or_create_cached_value(args, kwargs, spinner_message)
File "/usr/local/lib/python3.9/site-packages/streamlit/runtime/caching/cache_utils.py", line 260, in _get_or_create_cached_value
return self._handle_cache_miss(cache, value_key, func_args, func_kwargs)
File "/usr/local/lib/python3.9/site-packages/streamlit/runtime/caching/cache_utils.py", line 318, in _handle_cache_miss
computed_value = self._info.func(*func_args, **func_kwargs)
File "/app/app.py", line 145, in build
return _self.annotate(url)
File "/usr/local/lib/python3.9/site-packages/annotateai/annotate.py", line 60, in call
path = self.path(path)
File "/usr/local/lib/python3.9/site-packages/annotateai/annotate.py", line 98, in path
with contextlib.closing(urlopen(Request(path))) as connection:
File "/usr/local/lib/python3.9/urllib/request.py", line 214, in urlopen
return opener.open(url, data, timeout)
File "/usr/local/lib/python3.9/urllib/request.py", line 523, in open
response = meth(req, response)
File "/usr/local/lib/python3.9/urllib/request.py", line 632, in http_response
response = self.parent.error(
File "/usr/local/lib/python3.9/urllib/request.py", line 561, in error
return self._call_chain(*args)
File "/usr/local/lib/python3.9/urllib/request.py", line 494, in _call_chain
result = func(*args)
File "/usr/local/lib/python3.9/urllib/request.py", line 641, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)

[davidmezzetti]

This appears to be an error with the site blocking the request. The Textractor pipeline has a parameter to set the user agent but it's not exposed here. This would likely fix the issue.

What is the error when downloading the file and running locally?