Dolfyn processing errors

[browniea]

Processing RiverPro ADCP data with dolfyn #169

When processing .PDO files collected using a Teledyne RiverPro ADCP several files came back with an error due to a repeated time step cannot reindex or align along dimension 'time' because the index has duplicate values case file along with a long list of unrecognized ID codes ex: Unrecognized ID code: FF01. Upon further investigation downloading the time data as a .txt file there where no repeated time steps when analyzing it using the unique function.

This ADCP was functioning under 'Auto' water mode and had 5 beams. The data files used can be found here: Data

[ssolson]

Hey @jmcvey3 I asked Emily to create an issue describing the current Dolfyn errors around processing her data using Dolfyn. From offline conversations it sounds like the issues may be caused by this being a newer model with 5 beams? Could you comment on the need to have Dolfyn correctly interface with data from this instrument and maybe what you would suggest as to the best course of action to resolve this issue/ similar ones?

[jmcvey3]

@browniea @ssolson DOLfYN has source code to read .PD0 files, which is TRDI's main binary format, from the Workhorse series of ADCPs. I'm not sure if DOLfYN has been tested with the Sentinel V and other 5 beam instruments, and judging by the Unrecognized ID codes, there are data blocks for the RiverPro ADCP that aren't taken into account by the read-TRDI-binary script (rdi.py).

To fix this issue, it would require knowing how to read the additional data blocks and where in the binary file they are located.

TRDI doesn't release integration guides, which makes it more difficult to decipher their format. I didn't get a chance to look through it, but they had an SDK on their github page called "PDDecoder" with C code to read in .PD0 files a couple months back, but has apparently since been taken down. May be worth emailing harvey.stoelinga@teledyne.com, the contact on TRDI's page, to see if that SDK is available if we want to go down this path.

[jmcvey3]

@lkilcher Of DOLfYN's readers, I'm the least familiar with rdi.py. Where did that original source code come from?

[browniea]

Just heard back from Teledyne. The downloads should be available on their website portal: https://tm-portal.force.com/TMsoftwareportal/s/

Once you have logged in, the PDDecoder files/SDK can be found in this folder:
Home -> Software -> PDDecoder Library

[ssolson]

@jmcvey3 what is the process to begin integration of the SDK with DOLfYN? Do you have thoughts on the difficulty level/ time rough expected time to complete if started?

[jmcvey3]

@jmcvey3 what is the process to begin integration of the SDK with DOLfYN? Do you have thoughts on the difficulty level/ time rough expected time to complete if started?

My rough guess is 1-4 weeks of time, but it's hard to tell because I don't know what the issue is or what needs to be added.

[ssolson]

Thanks, @jmcvey3! Emily has started to look into the issue. From our discussion today she indicated she could use your help understanding what needs to be done to integrate the SDK with DOLfYN. Could you potentially provide a high-level overview of what needs to be done... or is this more of a nonstandard addition that will be problem specific?

[jmcvey3]

@ssolson I can walk through how the current reader works with @browniea. Basically I'm expecting the SDK to contain key pairs consisting of an ID number and a code block, and I'm hoping all we need to do is add the missing ID-code bocks into dolfyn.

[jmcvey3]

Realized this is also related to lkilcher/dolfyn#92.

[lkilcher]

The original source for rdi.py was a reimplementation of Rich Pawlowicz’ matlab reader.

On Thu, Aug 4, 2022 at 8:21 AM ssolson ***@***.***> wrote: @jmcvey3 <https://github.com/jmcvey3> what is the process to begin integration of the SDK with DOLfYN? Do you have thoughts on the difficulty level/ time rough expected time to complete if started? — Reply to this email directly, view it on GitHub <#182 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AARLAUPDBATF5NN3AC6DHOTVXPUX7ANCNFSM54W4WA7A> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Levi Kilcher ~~~~ "The whole problem with the world is that fools and fanatics are always so certain of themselves, and wiser people so full of doubts." - Bertrand Russell

[jmcvey3]

Working on fix in lkilcher/dolfyn#109

[ssolson]

Fixed in #186

[jmcvey3]

It's not fixed with #186. I'm waiting to get test data from Emily (after it gets through UAF's release process) so I can cover the code additions.