Read issue with Signature ADCP dataset #392
Description
Describe the bug:
I get an OverflowError when reading a subset (using the nens argument) of a large (>10Gb) Signature 500 .ad2cp file. The adcp was measuring in a concurrent mode including the "burst and waves" + the echosounder option.
To Reproduce:
Provide a minimal working example that reproduces the error.
ds = dolfyn.read('my_sig500_file.ad2cp',nens=[2000,20000])
prompts
`File ~/miniconda3/lib/python3.10/site-packages/mhkit/dolfyn/io/api.py:114, in read(fname, userdata, nens, **kwargs)
112 func_map = dict(RDI=read_rdi, nortek=read_nortek, signature=read_signature)
113 func = func_map[file_type]
--> 114 return func(fname, userdata=userdata, nens=nens, **kwargs)
File ~/miniconda3/lib/python3.10/site-packages/mhkit/dolfyn/io/nortek2.py:79, in read_signature(filename, userdata, nens, rebuild_index, debug, dual_profile, **kwargs)
75 raise TypeError("nens must be: None (), int, or len 2")
77 userdata = _find_userdata(filename, userdata)
---> 79 rdr = _Ad2cpReader(
80 filename, rebuild_index=rebuild_index, debug=debug, dual_profile=dual_profile
81 )
82 d = rdr.readfile(nens[0], nens[1])
83 rdr.sci_data(d)
File ~/miniconda3/lib/python3.10/site-packages/mhkit/dolfyn/io/nortek2.py:157, in _Ad2cpReader.init(self, fname, endian, bufsize, rebuild_index, debug, dual_profile)
155 self.f.seek(0, 2) # Seek to end
156 self._eof = self.f.tell()
--> 157 self.start_pos = self._check_header()
158 self._index, self._dp = lib.get_index(
159 fname,
160 pos=self.start_pos,
(...)
164 dp=dual_profile,
165 )
166 self._reopen(bufsize)
File ~/miniconda3/lib/python3.10/site-packages/mhkit/dolfyn/io/nortek2.py:209, in _Ad2cpReader._check_header(self)
206 idx = s.find(c, idx + 1)
208 # Open the entire file
--> 209 self._reopen(self._eof)
210 pk = self.f.peek(1)
211 # Search for multiple saved headers
File ~/miniconda3/lib/python3.10/site-packages/mhkit/dolfyn/io/nortek2.py:226, in _Ad2cpReader._reopen(self, bufsize)
224 except AttributeError:
225 pass
--> 226 self.f = open(_abspath(self.fname), "rb", bufsize)
OverflowError: Python int too large to convert to C int`
Expected behavior:
Expected a subset of the file to be read
Screenshots:
Desktop (please complete the following information):
- OS: Ubuntu, Jammy
- MHKiT Version: 0.9.0