Skip to content

Fix XML parsing: preserve attributes & handle UTF-16 BOM #57

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Fix XML parsing: preserve attributes & handle UTF-16 BOM #57

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 21 additions & 2 deletions man_spider/lib/parser/parser.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,26 @@ def extractous(self, file, pretty_filename):
if not self.match_magic(file):
return matches

text_content, metadata = self.extractor.extract_file_to_string(str(file))
# XML files: read raw to preserve attributes and handle encoding
if suffix == '.xml':
log.debug(f'Parsing raw XML for {pretty_filename}')
try:
# read raw bytes and detect BOM for encoding
with open(file, 'rb') as f:
raw_bytes = f.read()
if raw_bytes.startswith(b'\xff\xfe') or raw_bytes.startswith(b'\xfe\xff'):
text_content = raw_bytes.decode('utf-16', errors='ignore')
elif raw_bytes.startswith(b'\xef\xbb\bf'):
text_content = raw_bytes.decode('utf-8-sig', errors='ignore')
else:
text_content = raw_bytes.decode('utf-8', errors='ignore')
metadata = {}
except Exception as e:
log.warning(f"Error reading raw XML for {pretty_filename}: {e}")
return matches
else:
# non-XML: extract text via extractous
text_content, metadata = self.extractor.extract_file_to_string(str(file))

# try to convert to UTF-8 for grep-friendliness
try:
Expand All @@ -162,4 +181,4 @@ def extractous(self, file, pretty_filename):
if not self.quiet:
self.grep(binary_content, _filter.pattern)

return matches
return matches