Intro

MET_hobo

This module preforms basic QAQC on CSV files generated by Onset HOBO’s creating a directory of .hobo and CSV files formatted for easy import into the GCE Matlab toolbox.

See module documentation for use.

Data QA Performed:

  1. Columns that are empty or have erroneous text are removed,
  2. timezone is checked and converted to a user defined value,
  3. measurement units are checked and values converted to user defined units, and
  4. timestep is synced to a standardized interval; Exp: tstep=’5min’, 11:05:35 becomes 11:05:00.

File Management Performed

All files are removed from the source directory and filed for storage. File movement is tracked in log files.

Release Notes

  • 1.0- development finalized 11/22/21
    • Released after beta testing in production QA workflow for bird datasets.
    • What’s new:
      • Allow more flexible directory structures
      • Handle a wider range of input csv formats
      • Improve error checks and clarify explanations when warnings occur
        • extra checks for filepath errors
        • extra checks for header formatting
      • Allow ./config location to be specified at the command line
      • Allow command line override of ./config settings (time zone, units, etc.)
      • Adjust for package updates for final python 2.x states
  • 0.2A- Released 4/16/18
    • Beta version on development branch for integration in info management work flow
    • What’s new:
      • Added file management for processing large, multi-project, server-based directories
      • Add processing for hobos without light intensity
      • Add processing for different time steps
      • Add Sphinx docs
      • Add setting control through ./config file
  • BetaV0.2_hobo_only- Released 1/17/2018
    • Fork from verify repository and remove all modules that do not deal with hobo files
  • BetaV0.1_module- Released 1/17/2018
    • Last common commit with verify repository; verify module split into independent modules to load data from different sensors and preform sensor comparisons

Accepted File Format

There are many different export options for converting .hobo files to .csv in HOBOware, however not all formats can be processed by this module.

Format Requirements

  • U.S. Date formats only: MDY or YMD
  • U.S. Number formats only: -1,000.0 or -1000.0

Including GMT

A primary function of this module is correcting time zone. Because each sensor does not have it’s own clock and simply counts time from the Launcher or PC clock, daylight savings is often included or excluded inconsistently throughout the life of a sensor. For this reason, time zone (GMT -#) must be included in the header so that it can be converted to the user’s standard.

To export time zone from HOBOware:

  1. Go to Preferences>>General>>Export Settings
  2. DE-SELECT option, “No quotes or commas in headings, properties in parentheses”

Installation

pip and conda install are not yet supported.

Download the source code

You can directly download the repository by going to Bitbucket and selecting Downloads from the left-side menu.

Select a tab:

  1. Branches - which will contain the latest changes to master, development, and feature branches
  2. Tags - which will contain specific release versions. It is recommended that you use the latest release.

Download the .zip file for the version you want.

Clone the git repository

If you plan on using the program repeatedly, it’s a good idea to clone this repository to your desktop using Git. This will allow you to get the most recent changes to the program. It will also allow you to share any changes or improvements that you make with the rest of the community.

There are great tutorials available, but in brief:

  1. Install a current version of git or a GUI with git embedded [1]
  2. Open a command shell and navigate to a parent directory where you want to store the module
  3. Type: git clone https://<username>@bitbucket.org/hjandrews/met_hobo.git using your username without the angle brackets.

This creates a directory called MET_hobo that contains the module.

[1]: SourceTree or GitHub Desktop

Run the Program

Edit file_path.config

You must define the three file directories (source, working, final) and define the timestep of the data being processed. Take care to use \\ for Windows file paths or / which will work on any operating system. NEVER use \ which is a special character; example:

In[36]: print('c:\new\forest\temp\nc:\\new\\forest\\temp')
Out[36]: c:
        ew orest    emp
        c:\new\forest\temp

By default, this file should be located in the top directory of the repository. However, the user can provide a path to this file as a keyword variable in file_manager.FileHandling.manage().

Accessing the Methods

From Python 2.x

Import the module and access the individual classes and methods directly. This allows parameters to be set directly.

file_manager.FileHandling.manage() will run the process on all files in a directory from start to finish.

From Terminal

The entire batch process can be initiated from a terminal if Python is in the system path. file_manager Module executes file_manager.FileHandling.manage() when executed and parameters are set by config file.

rem batch execute HOBO QAQC from DOS
python file_manager.py

To manually control time step, units, or time zone from the terminal:

rem edit QAQC settings of batch from DOS
.\>python -c "from file_manager import FileHandling; FileHandling().manage(time_step='20min')"