file_manager Module

This module preforms QAQC methods in a batch. Methods were developed to process csv files created by HOBO sensors at meteorological sites on the HJ Andrews experimental forest. It also preforms other file storage and management functions. For a specified directory, it processes all files and creates a directory of new, processed csv files.

QAQC methods are imported from hobo_qaqc.HOBOdata.reformat_HOBO_csv().

When module is called FileHandling.manage() is executed.

This module is designed to minimize any read/write times by copying all files locally, preforming all processes, and then transferring files to final directories. This is ideal with external or network drives, but if all directories are local, it will create a final directory which duplicates file names from the source directory.

class file_manager.FileHandling(config='../file_path.config')

Processes all files in assigned directory for timezone, units, and timestep sync, and converts values where necessary. Contains methods for archiving using .zip, wiping directories after processing, and adding to existing directory structure: ./<FileArchive>/<Project>/<Site>.

Warning

Executes ./MET_hobo/file_path.config as Python file and saves variables to class object.

Todo

possible change from sys.platform to os.name to decrease package dependencies

possible change from shutil.rmtrees to os.remove os.rmdir

copy_processed_to_final_dir()

Copies processed (QC’d) files from local working directory to final directory using OS specifc DOS, bash, or shell command. Results are output log file.

Directory paths assigned to instance from file_path.config when instance is initialized.

cp <wdir/_processed> <dir_final_storage>

copy_selected_to_site_dir(file_list, subdir, loc)

Call OS specific system command to copy desired files from temporary working directory to final storage. Selects files by site using wildcard selection.

Example:

`cp <wdir/_processed/site*>  <dir_final_storage/proj_name/site_name/subdir>`

`cp //NewServer/hoboQA/_processed/RS12* //DataServer/REFSTANDS/RS12/_bulk_export_clean`

Warning

This method was modified per bitbucket issue issue #10 to create a simpler work flow

where file movement is more manually controlled. At 6ec103b it was superseded by copy_processed_to_final_dir, removing it from the workflow. It remains as a legacy method still in BETA testing.

Parameters:

• file_list – List of str to select files from. Example: [‘RS12’,’RS04’] copies files ‘RS12*’ and ‘RS04*’
• subdir – str. Destination subdirectory within final storage directory. Files are moved to here.
• loc – str. Directory where files are currently located.

Returns:

List of strings of each filename copied to the final directory

copy_src_to_wdir()

Copies source files to local working directory using OS specifc DOS, bash, or shell command. Results are output to log file.

Directory paths assigned to instance from file_path.config when instance is initialized. cp <dir_source_files> <wdir/_data>

del_files_frm_srcdir()

Wipe all files from the src_dir, defined in file_path.config as dir_source_files. All files and sub- folders in this directory will be wiped.

If source directory and final directory are the same, this process will abort.

Warning

This uses destructive methods which will erase any and all contents of the target directory and any sub- directories within.

shutil.rmtree()

Returns:List of strings of each filename wiped from the source directory

del_temp_folders()

This is to wipe temporary processing folders in the working directory. The convention maintained by this module is that all temp folders have the “_” prefix

If any files are still in _processed, and have not been copied to a final storage directory, deletion of this directory will be aborted.

Warning

This uses destructive methods which will erase any and all contents of the target directory and any sub- directories within.

shutil.rmtree()

index_files()

Identify files in source directory. Create list of .hobo, .csv, .log files, and any other file type encountered.

Identify site as any prefix to the left of “_” in filename and generate a list of unique sites.

manage(time_step=None, units='SI', tz=-8, final_subdirs=False)

Execute file managment.

1. Copy files to working directory (./_data).
2. Create list of .csv, .hobo, and .logs files in working directory.
3. Attempt to preform QAQC on all .csv files and transfer to ./_processed.
4. Create a .zip file for all .hobo files from each site. Disabled per bitbucket issue #10 .
5. Copy all files with .csv, .log, and unknown extension to final storage.
6. Delete temporary folders in working directory.
7. Wipe original source directory. This directory contains files where QAQC was not preformed. Disabled per bitbucket issue #10 .
8. Write log file.

3 keyword variables are defined to allow the user to alter format_QAQC_data() settings. units, and tz (time zone) are set to default values, SI units and PST (GMT-8). To change these values, manage() must be called directly, through the terminal, or through Python. time_step, is defined in the config file. This argument only needs to be defined here if the user wants to override the config file at the command line.

qaqc_csv(time_step=None, units='SI', tz=-8)

Attempt to QAQC all csv files for timezone, timestep sync, and units.

For list of .csv files generated by index_files(), call hobo_qaqc.HOBOdata.reformat_HOBO_csv().

Returns:list. strings of filenames processed with \n at end. Returns:int. number of csv files Returns:int. number of files processed

set_log_header()

Create header for log file. Assigns first items to list self.logs.

write_log()

Write log to file. <final storage directory>//logs//hobo_qaqc_<date>.log.

Log is a list of strings until this function is called.

zip_hobo_files()

Collect all files with .hobo extension and write to a zip file in the temp directory _processed.

Naming convetion is <site>_<today’s date>.zip, where site is any filename prefix to the left of “_”.

For list of .hobo files generated by index_files()

Returns:List of strings of each filename and it’s zipped filename with a \n at the end Returns:int. Count of hobo files Returns:int. Count of zipped files