Does mhkit work with the Nortek Aquadopp?

[ddileonardo]

Does mhkit work with the Nortek Aquadopp? Specifically, I have data from the legacy (pre-2024) aquadopp. It is an ADCP, but it isn't mentioned in the documentation anywhere.

If mhkit does support aquadopps, which output file extension should I be trying to read?

[jmcvey3]

@ddileonardo Looks like I forgot to update the reader docstring... It does support Aquadopps, but I've only had test data for the profiling versions. For those it's the .prf file, for the single point meters it is the .aqd filetype.

[ddileonardo]

Thanks! I have a .prf file. The instrument was set to profile.

I am getting a KeyError when I try to use dolfyn.read.

from mhkit import dolfyn

data_path = r'GS-LMR_CH2ADP_020526.prf'
ds = dolfyn.read(data_path)

KeyError                                  Traceback (most recent call last)
Cell In[3], line 2
      1 data_path = r'GS-LMR_CH2ADP_020526.prf'
----> 2 ds = dolfyn.read(data_path)

File c:\Users\ddileonardo\.conda\envs\qbay\Lib\site-packages\mhkit\dolfyn\io\api.py:114, in read(fname, userdata, nens, **kwargs)
    112     func_map = dict(RDI=read_rdi, nortek=read_nortek, signature=read_signature)
    113     func = func_map[file_type]
--> 114 return func(fname, userdata=userdata, nens=nens, **kwargs)

File c:\Users\ddileonardo\.conda\envs\qbay\Lib\site-packages\mhkit\dolfyn\io\nortek.py:60, in read_nortek(filename, userdata, debug, do_checksum, nens, **kwargs)
     51     logging.basicConfig(
     52         filename=str(logfile),
     53         filemode="w",
     54         level=logging.NOTSET,
     55         format="%(name)s - %(levelname)s - %(message)s",
     56     )
     58 userdata = base._find_userdata(filename, userdata)
---> 60 rdr = _NortekReader(filename, debug=debug, do_checksum=do_checksum, nens=nens)
     61 rdr.readfile()
     62 rdr.cleanup()

File c:\Users\ddileonardo\.conda\envs\qbay\Lib\site-packages\mhkit\dolfyn\io\nortek.py:230, in _NortekReader.__init__(self, fname, endian, debug, do_checksum, bufsize, nens)
    228     self.config["config_type"] = "ADV"
    229 # Initialize the instrument type:
--> 230 self._inst = self.config.pop("config_type")
    231 # This is the position after reading the 'hardware',
    232 # 'head', and 'user' configuration.
    233 pnow = self.pos

KeyError: 'config_type'

[jmcvey3]

@ddileonardo Can you rebuild your conda environment with the latest version of MHKiT we just released? Looks like you're running an older version.
If you open up your .prf file in a text editor, you will will see a few ascii characters that spell out the serial number at the very beginning of the file. MHKiT reads this serial number, which will start with "AQD" or "PRF" for Aquadopp profilers, to determine the correct reader to use. If yours doesn't start with either of those, please let me know.

[ddileonardo]

I used the method from the documentation to install
conda install -c conda-forge mhkit

Since it doesn't specify a version, I would have though it should be the most recent version? I installed in late July/early August.

When I try to update with conda, the only package that updates is openssl. I still get the same error.

I have tried creating a whole new environment and installing mhkit in the same way. I'm still getting the same error.

Is there a different conda channel that I need to use?

As far as the top of prf file goes, when I open in text editor I see a couple bits of binary and the AQD with a serial number so that seems right.

[jmcvey3]

Oh, please use pip install mhkit to install the full package, or pip install mhkit[dolfyn] for just the ADCP-related dependencies. We haven't yet updated the conda-forge package for the latest release.

[ddileonardo]

After the pip install I get an immediate import error for bottleneck when trying to import dolfyn or mhkit. Is there a specific python version I need or something else?

My install history in the environment goes:

• conda install pythong
• conda install pip
• pip install mhkit
• pip install ipykernel

So all the package version should be defined by pip.
pandas 3.0.5 pypi_0 pypi
numpy 2.5.1 pypi_0 pypi

ImportError                               Traceback (most recent call last)
Cell In[1], [line 1](vscode-notebook-cell:?execution_count=1&line=1)
----> [1](vscode-notebook-cell:?execution_count=1&line=1) from mhkit import dolfyn
      2 from mhkit.dolfyn.adp import api
      3 import numpy as np
      4 from matplotlib import pyplot as plt

File ~\.conda\envs\qbay-pip\Lib\site-packages\mhkit\__init__.py:5
      2 import importlib
      4 # Register datetime converter for a matplotlib plotting methods
----> [5](https://file+.vscode-resource.vscode-cdn.net/c%3A/Users/ddileonardo/Documents/GitHub/GS-LMR/~/.conda/envs/qbay-pip/Lib/site-packages/mhkit/__init__.py:5) from pandas.plotting import register_matplotlib_converters as _rmc
      7 _rmc()
      9 # Use targeted warning configuration

File ~\.conda\envs\qbay-pip\Lib\site-packages\pandas\__init__.py:58
     55 # let init-time option registration happen
     56 import pandas.core.config_init  # pyright: ignore[reportUnusedImport] # noqa: F401
---> [58](https://file+.vscode-resource.vscode-cdn.net/c%3A/Users/ddileonardo/Documents/GitHub/GS-LMR/~/.conda/envs/qbay-pip/Lib/site-packages/pandas/__init__.py:58) from pandas.core.api import (
     59     # dtype
     60     ArrowDtype,
     61     Int8Dtype,
     62     Int16Dtype,
     63     Int32Dtype,
     64     Int64Dtype,
     65     UInt8Dtype,
     66     UInt16Dtype,
     67     UInt32Dtype,
     68     UInt64Dtype,
     69     Float32Dtype,
     70     Float64Dtype,
     71     CategoricalDtype,
     72     PeriodDtype,
     73     IntervalDtype,
     74     DatetimeTZDtype,
     75     StringDtype,
     76     BooleanDtype,
     77     # missing
     78     NA,
     79     isna,
     80     isnull,
     81     notna,
     82     notnull,
     83     # indexes
     84     Index,
     85     CategoricalIndex,
     86     RangeIndex,
     87     MultiIndex,
     88     IntervalIndex,
     89     TimedeltaIndex,
     90     DatetimeIndex,
     91     PeriodIndex,
     92     IndexSlice,
     93     # tseries
     94     NaT,
     95     Period,
     96     period_range,
     97     Timedelta,
     98     timedelta_range,
     99     Timestamp,
    100     date_range,
    101     bdate_range,
    102     Interval,
    103     interval_range,
    104     DateOffset,
    105     # conversion
    106     to_numeric,
    107     to_datetime,
    108     to_timedelta,
    109     # misc
    110     Flags,
    111     Grouper,
    112     factorize,
    113     unique,
    114     NamedAgg,
    115     array,
    116     Categorical,
    117     set_eng_float_format,
    118     Series,
    119     DataFrame,
    120 )
    121 from pandas.core.col import col
    123 from pandas.core.dtypes.dtypes import SparseDtype

File ~\.conda\envs\qbay-pip\Lib\site-packages\pandas\core\api.py:[27](https://file+.vscode-resource.vscode-cdn.net/c%3A/Users/ddileonardo/Documents/GitHub/GS-LMR/~/.conda/envs/qbay-pip/Lib/site-packages/pandas/core/api.py:27)
     16 from pandas.core.dtypes.missing import (
     17     isna,
     18     isnull,
     19     notna,
     20     notnull,
     21 )
     23 from pandas.core.algorithms import (
     24     factorize,
     25     unique,
     26 )
---> 27 from pandas.core.arrays import Categorical
     28 from pandas.core.arrays.boolean import BooleanDtype
     29 from pandas.core.arrays.floating import (
     30     Float32Dtype,
     3[1](https://file+.vscode-resource.vscode-cdn.net/c%3A/Users/ddileonardo/Documents/GitHub/GS-LMR/~/.conda/envs/qbay-pip/Lib/site-packages/pandas/core/arrays/__init__.py:1)     Float64Dtype,
     32 )

File ~\.conda\envs\qbay-pip\Lib\site-packages\pandas\core\arrays\__init__.py:1
----> 1 from pandas.core.arrays.arrow import ArrowExtensionArray
      2 from pandas.core.arrays.base import (
      3     ExtensionArray,
      4     ExtensionOpsMixin,
      5     ExtensionScalarOpsMixin,
      6 )
      7 from pandas.core.arrays.boolean import BooleanArray

File ~\.conda\envs\qbay-pip\Lib\site-packages\pandas\core\arrays\arrow\__init__.py:[5](https://file+.vscode-resource.vscode-cdn.net/c%3A/Users/ddileonardo/Documents/GitHub/GS-LMR/~/.conda/envs/qbay-pip/Lib/site-packages/pandas/core/arrays/arrow/__init__.py:5)
      1 from pandas.core.arrays.arrow.accessors import (
      2     ListAccessor,
      3     StructAccessor,
      4 )
----> 5 from pandas.core.arrays.arrow.array import ArrowExtensionArray
      7 __all__ = ["ArrowExtensionArray", "ListAccessor", "StructAccessor"]

File ~\.conda\envs\qbay-pip\Lib\site-packages\pandas\core\arrays\arrow\array.py:78
     73 from pandas.core.arrays._utils import to_numpy_dtype_inference
     74 from pandas.core.arrays.base import (
     75     ExtensionArray,
     76     ExtensionArraySupportsAnyAll,
     77 )
---> [78](https://file+.vscode-resource.vscode-cdn.net/c%3A/Users/ddileonardo/Documents/GitHub/GS-LMR/~/.conda/envs/qbay-pip/Lib/site-packages/pandas/core/arrays/arrow/array.py:78) from pandas.core.arrays.masked import BaseMaskedArray
     79 from pandas.core.arrays.string_ import StringDtype
     80 import pandas.core.common as com

File ~\.conda\envs\qbay-pip\Lib\site-packages\pandas\core\arrays\masked.py:56
     45 from pandas.core.dtypes.dtypes import (
     46     ArrowDtype,
     47     BaseMaskedDtype,
     48 )
     49 from pandas.core.dtypes.missing import (
     50     array_equivalent,
     51     is_valid_na_for_dtype,
     52     isna,
     53     notna,
     54 )
---> [56](https://file+.vscode-resource.vscode-cdn.net/c%3A/Users/ddileonardo/Documents/GitHub/GS-LMR/~/.conda/envs/qbay-pip/Lib/site-packages/pandas/core/arrays/masked.py:56) from pandas.core import (
     57     algorithms as algos,
     58     arraylike,
     59     missing,
     60     nanops,
     61     ops,
     62 )
     63 from pandas.core.algorithms import (
     64     factorize_array,
     65     isin,
   (...)     68     take,
     69 )
     70 from pandas.core.array_algos import (
     71     masked_accumulations,
     72     masked_reductions,
     73 )

File ~\.conda\envs\qbay-pip\Lib\site-packages\pandas\core\nanops.py:[54](https://file+.vscode-resource.vscode-cdn.net/c%3A/Users/ddileonardo/Documents/GitHub/GS-LMR/~/.conda/envs/qbay-pip/Lib/site-packages/pandas/core/nanops.py:54)
     51 if TYPE_CHECKING:
     52     from collections.abc import Callable
---> 54 bn = import_optional_dependency("bottleneck", errors="warn")
     55 _BOTTLENECK_INSTALLED = bn is not None
     56 _USE_BOTTLENECK = False

File ~\.conda\envs\qbay-pip\Lib\site-packages\pandas\compat\_optional.py:173, in import_optional_dependency(name, extra, min_version, errors)
    171 minimum_version = min_version if min_version is not None else VERSIONS.get(parent)
    172 if minimum_version:
--> [173](https://file+.vscode-resource.vscode-cdn.net/c%3A/Users/ddileonardo/Documents/GitHub/GS-LMR/~/.conda/envs/qbay-pip/Lib/site-packages/pandas/compat/_optional.py:173)     version = get_version(module_to_get)
    174     if version and Version(version) < Version(minimum_version):
    175         msg = (
    176             f"Pandas requires version '{minimum_version}' or newer of '{parent}' "
    177             f"(version '{version}' currently installed)."
    178         )

File ~\.conda\envs\qbay-pip\Lib\site-packages\pandas\compat\_optional.py:80, in get_version(module)
     77 version = getattr(module, "__version__", None)
     79 if version is None:
---> [80](https://file+.vscode-resource.vscode-cdn.net/c%3A/Users/ddileonardo/Documents/GitHub/GS-LMR/~/.conda/envs/qbay-pip/Lib/site-packages/pandas/compat/_optional.py:80)     raise ImportError(f"Can't determine version for {module.__name__}")
     81 if module.__name__ == "psycopg2":
     82     # psycopg2 appends " (dt dec pq3 ext lo64)" to it's version
     83     version = version.split()[0]

ImportError: Can't determine version for bottleneck

[jmcvey3]

Huh, we haven't seen that pandas error before. What happens if you run pip install bottleneck?

[ddileonardo]

Looking at my package list, it seems like bottleneck wasn't installed before, for some reason. Installing it as above fixed the import issue.

Now though I am getting an error related to the beginning of the file:

WARNING:root:Unrecognized identifier: 0x9d

The first few characters look a bit like this when opened in notepad++

¥�� AQD11219

It looks like a bit of binary and then those ascii characters.

We are using legacy aquadopps, which might be useful information if the aquadopp workflow was developed with the newer gen 2 aquadopps.

[jmcvey3]

Noted, that should've been installed with the package.

Yes, that's the serial number. I've tested with an older instrument than yours (based on the serial number) and one of the new ones.

Can you run ds = dolfyn.read(data_path, debug=True) and upload the resulting log file here? There is no 0x9d ID, looking at Nortek's integrator guide

[ddileonardo]

Here is the log file

GS-LMR_CH2ADP_050726.dolfyn.log

I had a peek in there and it suggested to me that there was an issue with the data file. I tried a different data file and it read successfully.

Thank you very much for all your help.

[jmcvey3]

Is there an odd number of bins in the datafile with the issue? And an even number of bins in the datafile that ran successfully?

There is a note in the docs that states the former will include an extra "fill" byte, but we've never seen that to be true before. But it looks like that might be true for your case.

[ddileonardo]

Yes, there are 5 cells listed in the hdr file. There are 10 cells list in the hdr file of the file that works.

[jmcvey3]

Well this might be really easy. Can you pull PR #459 and try that?

You could also copy-paste the changes in https://github.com/MHKiT-Software/MHKiT-Python/pull/459/changes if that's faster for you

[ddileonardo]

I'm trying, but it's not quite working.

I've cloned the repo, but I couldn't figure out how to pull a PR, so I just made the changes in nortek.py. But when I tried to build it as in the documentation

pip install -e .["all"]

Something didn't quite work. I've got python 3.14, but that wasn't an issue before.

This is the pip error from the build

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for numcodecs
Building wheel for bitshuffle (pyproject.toml) ... error
error: subprocess-exited-with-error

× Building wheel for bitshuffle (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [18 lines of output]
Can't find hdf5 with pkg-config fallback to static config
Can't find hdf5 with pkg-config fallback to static config
Can't find hdf5 with pkg-config fallback to static config
Can't find hdf5 with pkg-config fallback to static config
Can't find hdf5 with pkg-config fallback to static config
running bdist_wheel
running build
running build_py
creating build\lib.win-amd64-cpython-314\bitshuffle
copying bitshuffle_init_.py -> build\lib.win-amd64-cpython-314\bitshuffle
running build_ext

  #################################
  # Compiling with OpenMP support #
  #################################
  
  building 'bitshuffle.ext' extension
  error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for bitshuffle
Successfully built mhkit
Failed to build cartopy aiohttp numcodecs bitshuffle
error: failed-wheel-build-for-install

× Failed to build installable wheels for some pyproject.toml based projects
╰─> cartopy, aiohttp, numcodecs, bitshuffle