lidarwind package

Submodules

lidarwind.data_attributes module

Module for wrinting the data attributes

class lidarwind.data_attributes.LoadAttributes(data)

Bases: object

Level 1 Attribute generator

It defines and writes the attributes from the Level 1

Parameters:

data (xarray.Dataset) – A xarray dataset containing the Level 1 data generated by the lidarwind

Returns:

data – A dataset with corrected attributes

Return type:

xarray.Dataset

variables_attrs()

Variable attributes definitions

All variables and coordinates attributes are defined here.

write_coords_attrs()

Coordinate attribute writer

It loops over the dataset coordinate and writes their attributes

write_global_attrs()

Global attribute writer

It loads and writes the global attributes from the dataset. The global attributes are defiened in the configuration file. See lidarwind_config.generate_conf documentation for information about generating the configuration file.

write_variables_attrs()

Variable attribute writer

It loops over the dataset variables and writes their attributes

lidarwind.data_operator module

Module to manage and prepare the data for other modules

class lidarwind.data_operator.DataOperations(data_paths, verbose=False)

Bases: object

Basic data manager

It performs some basic operations. For example: rounds the values from the azimuth coordinate and separates the vertical observations from the slanted observations.

It is helpful first use this class to process all data and then save the data as NetCDF files to speed up the derivation of wind properties later.

Examples

>>> merged_ds = lidarwind.DataOperations(file_list).merged_data
>>> merged_ds.to_netcdf(output_file_path)

Parameters:

data_paths (list) – List of paths of the original WindCube’s output.

Returns:

object – it returns an object containing an instance of the original files merged (.merged_data)

Return type:

object

elevation_filter()

It groups the data from the vertical and slanted observations and rounds the azimuth coordinate

get_merge_data()

It merges all readable data

rename_var_90()

It renames the vertical coordinate

class lidarwind.data_operator.DbsOperations(file_list, var_list)

Bases: object

DBS file manager

This class extracts the variables required to retrieve the wind information from the DBS files.

Parameters:

• file_list (list) – list of DBS files

• var_list (list) – list of variables to be extracted from the DBS files

Returns:

object – it returns an object containing an instance of the merged files (.merged_ds)

Return type:

object

add_mean_time(lidar_ds)

This method adds the mean time to each file from the DBS scan strategy.

Parameters:

lidar_ds (xarray.DataSet) – a dataset from a sequence of scans

mean_time_derivation(data)

merge_2_ds(file_to_merge, var_list)

This method merges the variables extracted from the single DBS file with the storage dataset (merged_ds).

Parameters:

• file_to_merge (xarray.DataSet) – a single file dataset

• var_list (list) – a list of variables to be merged

merge_data(file_list, var_list)

This method merges all files from a list of DBS files

Parameters:

• file_list (list) – list of files to be merged

• var_list (list) – list of variables to be merged

class lidarwind.data_operator.GetResampledData(xr_data_array, vert_coord='range', time_freq='15s', tolerance=10, time_coord='time')

Bases: object

Alternative basic data resample

This class is used to resample the data into a given temporal grid.

It mainly used internal processings of the package.

Parameters:

• xr_data_array (xr.DataArray) – varaiable that will be resampled

• vert_coord (str) – name of the vertical coordinate

• time_freq (str) – size of the window e.g.: ’15s’

• tolerance (int) – maximum separation from the reference

• time_coord (str) – name of the time coordinate

Returns:

data – time resampled variable

Return type:

xr.DataArray

calc_delt_grid(ref_grid, orig_grid)

Calculates the distance between the reference grid and the radar grid (time or range)

Parameters:

• ref_grid (numpy.array) – reference grid (array[n])

• radarGrid (numpy.array) – radar grid (array[m])

Returns:

delta_grid – distance between each element from the reference grid to each element from the radar grid

Return type:

numpy.array

convert_to_data_array()

It creates a DataArray of the resampled data.

get_nearest_index_method_2(delta_grid, tolerance)

Identify the index of the delta_grid that fulfil the resampling tolerance

Parameters:

• delta_grid (numpy.array) – output from calcRadarDeltaGrid

• tolerance (int) – tolerance distance for detecting the closest neighbour (time or range)

Returns:

grid_index – array of indexes that fulfil the resampling tolerance

Return type:

np.array

get_time_ref(date, time_freq='1s')

Genetates the time reference grid used for resampling the data

Parameters:

• date (Timestamp) – date for resampling (pandas Timestamp)

• dateFreq (str) – resolution of the reference grid (str, default=1s)

Returns:

time_ref – time reference grid (DatetimeIndex)

Return type:

DatetimeIndex

time_resample(data, time_index_array, vert_coord)

It resamples a given radar variable using the time and range index calculated by get_nearest_index_method_2

Parameters:

• var (str) – radar variable name to be resampled

• xrDataset (xarray.dataset) – xarray dataset containing the variables to be resampled

• timeIdexArray (np.array) – time resampling index (output from get_nearest_index_method_2)

• rangeIndexArray (np.array) – range resampling index (output from get_nearest_index_method_2)

Returns:

resampledArr – time/range resampled numpy array

Return type:

xarray.dataArray

class lidarwind.data_operator.GetRestructuredData(data, snr=False, status=True, n_prof=500, center=True, min_periods=30, n_std=2, check90=True)

Bases: object

Data re-structurer

It prepares the data structure for using the wind retrieval modules.

Examples

>>> wind_prop = lidarwind.GetRestructuredData(merged_data)

Parameters:

• data (xr.Dataset) – a xr.Dataset of pre-processed data

• snr (bool, int, optional) – if an interger is given it is used to as threshold to filter the data based on the signal to noise ratio

• status (bool, optional) – if true it filters the data using the status variable generated by the WindCube’s software

• sProf (int, optional) – number of profiles used to calculate the anomaly for partially filter the second trip echoes (IT GOES TO FILTER MODULE)

• center (bool, optional) – (IT GOES TO FILTER MODULE)

• min_periods (int, optional) – (IT GOES TO FILTER MODULE)

• n_std (int, optional) – size of the standard deviation window used to partially remove the second trip echoes (IT GOES TO FILTER MODULE)

• check90 (bool, optional) – If True, it checks if the vertical observations are available. If False, the verification is ignored.

Returns:

object – an instance of the prepared for the retrieval

Return type:

object

data_transform()

It creates an xr.DataArray from all slanted observations

data_transform_90()

It creates an xr.DataArray from all vertically pointing observations

get_coord_non_90()

It identifies and selects the slanted data

vertical_component_check(check90)

class lidarwind.data_operator.ReadProcessedData(file_list)

Bases: object

Pre-processed data reader

It reads all data pre-processed by data_operator.DbsOperations and merges them.

Examples

>>> merged_data = lidarwind.ReadProcessedData(file_list).merge_data()

Parameters:

file_list (list) – list of pre-processed NetCDF files

Returns:

object – an instance of all pre-processed data

Return type:

object

merge_data()

It merges all data from the file_list. It can choose between two different methods. One uses xr.open_mfdataset and the other uses xr.open_dataset.

merge_data_method_1()

It merges data using xr.open_mfdatset

merge_data_method_2()

It merges data using xr.open_dataset

lidarwind.data_operator.wc_fixed_preprocessing(ds, azimuth_resolution=1)

lidarwind.filters module

Module for keep all filtering functionalities

class lidarwind.filters.Filtering(data)

Bases: object

SNR and Status filter

It uses the carrier to noise ratio (SNR) and status variables available in the WindCube’s data. It is similar to the filter described in the manual

Parameters:

data (xrarray.Dataset) – Dataset containing the original WindCube’s data

get_radial_obs_comp(variable, azm, snr=False, status=True)

Slanted data filter

It uses the SNR and status variables to filter out the data from slanted observations

Parameters:

• variable (str) – name of the variable that will be filtered

• snr (bool, int, optional) – if an interger is given it is used to as threshold to filter the data based on the signal to noise ratio

• status (bool, optional) – if true it filters the data using the status variable generated by the WindCube’s software

Returns:

tmpaData – an instance of the variable filtered using SNR or status variable

Return type:

xarray.DataSet

get_vertical_obs_comp(variable, snr=False, status=True)

Vertical data filter

It uses the SNR and status variables to filter out the data from vertical observations

Parameters:

• variable (str) – name of the variable that will be filtered

• snr (bool, int, optional) – if an interger is given it is used to as threshold to filter the data based on the signal to noise ratio

• status (bool, optional) – if true it filters the data using the status variable generated by the WindCube’s software

Returns:

tmp_data – an instance of the variable filtered using SNR or status variable

Return type:

xarray.DataSet

class lidarwind.filters.SecondTripEchoFilter(data, n_prof=500, center=True, min_periods=30, n_std=2, str_h='09', end_h='16')

Bases: object

Boundary layer second trip echoes filter

This filter minimises the presence of second trip echoes (STE). This filter is based on the standard deviation of the anomaly of the observaions. It is applicable in regions where there is a contrast between the real data and the STE.

Parameters:

• data (object) – the object returned from the GetRestructuredData

• timeCloudMaks (xarray.DataArray) – it is a time series for indicating the presence of clouds above the maximum WinCube range. 1 indicates cloud and 0 indicates no cloud. (THIS MAKS IS NOT NEEDED NOW)

• n_prof (int) – number of profiles used to calculating the anomaly

• center (bool, optional) – it defines how the mean value for the anomaly will be calculated

• min_periods (int) – minimum number of profiles used for calculating the mean value

• n_std (int) – Multiplication factor for defining the size of the window to keep the data. The filter removes any anomaly larger than n_std * std

• str_h (str) – starting hour for calculating the anomaly

• end_h (str) – end hour for calculating the anomaly

Returns:

object – an object containing data filtered for STE

Return type:

object

cal_mean_and_anom_90()

It calculates the anomaly from the vertical observations

cal_mean_and_anom_slant()

It calculates the anomaly from the slanted observations

cleaning()

It removes the data that is larger than the n_std * anomaly from the slanted observations

cleaning90()

It removes the data that is larger than the n_std * anomaly from the vertical observations

get_time_edges(str_h='09', end_h='16')

It creates the time boundaries for the STD anomaly calculation

Parameters:

• str_h (str) – starting hour for calculating the anomaly

• end_h (str) – end hour for calculating the anomaly

class lidarwind.filters.WindCubeCloudRemoval(ceilo, lidar=None)

Bases: object

Above boundary layer second trip echoes filter

This filter reduces the second trip echoes contamination and clouds from regions above the boundary layer. It requires information from the ceilometer. IT IS STILL EXPERIMENTAL

Parameters:

• ceilo (xarray.Dataset) – A dataset of the CHM15k Nimbus ceilometer observations.

• lidar (xarray.Dataset) – An instance of the re-structured WindCube dataset

Returns:

object – an object containing an instance of the noise height interface and the re-structured dataset filtered for STE and clouds.

Return type:

object

get_height_interface()

It identifies the height of the separation between the noise and the non-noise data from the ceilometer backscattered signal

get_interp_interf_height()

It interpolates the noise height interface to the same temporal resolution from the windcube data

get_noise_free_beta()

It removes the noise from the backscattered signal

remove_cloud()

It removes from the windcube’s observation all data above the noise height interface

lidarwind.filters.filter_snr(ds, snr)

lidarwind.filters.filter_status(ds)

Filter dataset based on WindCube’s software

Parameters:

ds (xr.Dataset) – Dataset with LIDAR records

Returns:

A Dataset with records where status was true (1)

Return type:

xr.Dataset

lidarwind.io module

lidarwind.io.open_sweep(file_name)

Windcube’s data reader

It opens and reads the original NetCDF output from the Windcube lidar.

Parameters:

file_name (str) – path to the file that will be opened

Returns:

ds – a dataset from the original NetCDF files

Return type:

xarray.DataSet

lidarwind.lidar_code module

Module to open the lidar original output

class lidarwind.lidar_code.GetLidarData(file_name)

Bases: object

Windcube’s data reader

ATENTION, please move to io.open_sweep(). This GetLidarData will be eventually removed.

It opens and reads the original NetCDF output from the Windcube lidar

Parameters:

file_name (str) – name of the file that will be open

open_lidar_file()

Function to read the lidar NetCDF files

Returns:

ds – a dataset from the original NetCDF files

Return type:

xarray.DataSet

Note

Alias to io.open_sweep() so upper functions don’t break until we finished the transition to new I/O.

lidarwind.lidarwind_config module

Module for configuring the data global attribute

class lidarwind.lidarwind_config.Configurations(lidarwind=None)

Bases: object

Global attributes definition

This class defines all global attributes that will be writen on the dataset

Parameters:

lidarwind (object) – an instance of the lidarwind package

generate_conf()

It writes and saves all defined global attributes.

load_comments(comments='General comments')

It defines additional comments

Parameters:

comments (str) – additional comments

load_conf_file(file_path='config.json')

It loads the pre-defined global attributes from the config.json, if it exists.

Parameters:

file_path (str) – the path to the configuration file (config.json)

load_contact(contact='contact person')

It defines the author’s name

Parameters:

contact (str) – name of the contact person

load_email(email='contact email')

It defines the contacting email

Parameters:

email (str) – contact email

load_institution(institution='institution name')

It defines the institution affiliation name

Parameters:

institution (str) – institution name

load_instrument(instrument='instrument name')

It defines the instrument name

Parameters:

instrument (str) – name of the instrument used during the experiment

load_reference(reference='Generated by lidarwind version: {0}')

It loads the lidarwind’s version used for processing the data

Parameters:

reference (str) – lidarwind version used to process the data

load_site(site='site name')

It defines the name of the experimental site

Parameters:

site (str) – name of the experimental site

load_version(lidarwind)

It identifies the lidarwind version and writes it to the configuration file

Parameters:

lidarwind (object) – a instance of the lidarwind package

lidarwind.utilities module

Utilities module

class lidarwind.utilities.CloudMask(wc_data=None, ceilo_data=None, radar_data=None)

Bases: object

This class generates the time-height cloud mask and the temporal cloud mask using observations from lidar and ceilometer.

call_methods()

clean_ceilo()

clean_radar()

get_cloud_mask_2d()

get_time_mask(mask_type=None)

class lidarwind.utilities.Util

Bases: object

This class contains useful tools

data_filenames()

Sample file list

It searches for the sample files. If the files do not exist, it downloads them.

get_sample_data(file_type)

Downloading data

It downloads the sample needed for the examples.

get_time_bins(freq='10min')

Bins estimation

Creating time bins for a given day and time resolution

lidarwind.utilities.sample_data(key)

lidarwind.version module

lidarwind.visualization module

class lidarwind.visualization.PlotSettings(mpl, style='dark_background')

Bases: object

plot_setup()

update_settings()

class lidarwind.visualization.Visualizer(data)

Bases: object

plot_data(tmp_data, cmap='Spectral', vmin=-1, vmax=1, elv='90', azm='-', save=False, plot_id=None, fig_path=None, string_name=None, show=False, min_time=None, max_time=None)

plot_data_azm(data_non_90, cmap='Spectral', vmin=-1, vmax=1, fig_path=None, save=False, plot_id=None, show=False, min_time=None, max_time=None)

view_orig_var(var_name, cmap='Spectral', vmin=-1, vmax=1, elv='90', azm='-', save=False, plot_id=None, fig_path=None, name_prefix=None, show=False, min_time=None, max_time=None)

view_ret_var(var_name, cmap='Spectral', vmin=-1, vmax=1, elv='90', azm='-', save=False, plot_id=None, fig_path=None, name_prefix=None, show=False, min_time=None, max_time=None)

lidarwind.wind_prop_retrieval module

class lidarwind.wind_prop_retrieval.FourierTransfWindMethod(doppler_obs)

Bases: object

FFT wind retrieval method

It is a fft wind retrieval method. It was proposed by Ishwardat (2017). For more details see http://resolver.tudelft.nl/uuid:a659654b-e76a-4513-a656-ecad761bdbc8

Parameters:

doppler_obs (xarray.DataArray) – It should be a DataArray of slanted Doppler velocity observations as function of the azimuthal angle. It must have a coordinate called azm.

Returns:

wind_prop – A Dataset containing the horizontal wind speed and direction. It also includes the zonal and meridional wind components

Return type:

xarray.Dataset

get_azm_wind(azm)

Wind speed for a given azimuth

It retrieves the wind speed for a given azimuthal angle. It cam be used to calculate the meridional and zonal wind components

Parameters:

azm (float) – an azimuth for retrieving the wind

get_comp_amp()

First harmonic amplitude

It calculates the complex amplitudes from the first harmonic from the observations along the azm coordinate

get_hor_wind_speed()

Wind speed elevation correction

It corrects the magnitude of the wind speed using the elevation angle

get_phase()

First harmonic phase

It calculates the phase of the first harmonic from the complex amplitude used for retrieving wind direction, i.e. comp_amp.

get_rad_wind_speed()

Wind speed calculation

It calculates the wind speed using the first harmonic

get_wind_comp_u()

Zonal wind calculation

It retrives the zonal wind component the -1 multiplication is for comply with the conventions used in meteorology

get_wind_comp_v()

Meridional wind calculation

It retrieves the meridional wind component the -1 multiplication is for comply with the conventions used in meteorology

get_wind_dir()

Wind direction calculation

It calculates the wind direction based on the phase of the first harmonic

wind_prop()

Wind dataset

It creates and returnes a dataset containing the wind speed, direction, and components

class lidarwind.wind_prop_retrieval.GetWindProperties5Beam(data, status_filter=True, cnr=None, method='single_dbs', tolerance='8s')

Bases: object

DBS wind retrieval

This class caculates the wind speeed and direction using the 5bean dataset (DBS files) as input.

Parameters:

• data (xarray.Dataset) – merged xarray dataset (merged_ds) output from lidarwind.DbsOperations()

• status_filter (bolean) – Data filtering based on the wind lidar wind status variable. If True, all data with status not equal to 1 are removed. If False, no filtering is applied.

• cnr (int, optional) – Filter based on the carrier to noise ratio. If None, no filtering is applied. If a cnr value is given, all data smaller than the cnr is removed.

• method (str) – It can be ‘single_dbs’ (default) or ‘continuous’. If single_dbs: the wind information is retrieved from comple sets of DBS profiles. If continuos: it uses the nearest 4 observations to retrieve the wind information, a tolerance wind has to be specified.

• tolerance (str) – It defines the tolerance window that the method will use for identify the nearest profile. Example: ‘8s’ for 8 seconds.

Returns:

object – This class returns an object containing the derived wind speed (.hor_wind_speed) and direction (.hor_wind_dir).

Return type:

object

calc_hor_wind_comp_continuous()

Function to derive wind v and u components. It folows the same approach used by the lidar software.

calc_hor_wind_comp_single_dbs()

This method derives v and u components from the WindCube DBS files. The components are caculated from each individual DBS file. The mean time from each scan complete scan is used as identification tag.

calc_hor_wind_dir()

Function to derive wind direction. If folows the same approach used by the lidar sftware.

calc_hor_wind_speed()

Function to calculate the wind speed.

correct_vert_wind_comp()

This function replaces the original coordinate from the vertical wind component by the measurement_height.

correct_wind_comp(comp)

This function replaces the gate_index coordinate by the measurement_height. (For any component)

Parameters:

comp (xarray.DataArray) – a variable from the original dataset

Returns:

comp – updated variable

Return type:

xarray.DataArray

class lidarwind.wind_prop_retrieval.RetriveWindFFT(transfd_data)

Bases: object

6 beam wind retrieval

Wind retrieval based on the FFT method for the 6 beam observations.

Parameters:

transfd_data (object) – An instance of the re-structured data, it should be preferentially filtered for artefacts.

Returns:

object – An object containing an dataset of the retrieved wind speed, direction, wind components (meridional, zonal and vertical) and the relative beta

Return type:

object

get_beta()

It copies the raw beta from the vertical observations

load_attrs()

It loads the attributes from all variables into the dataset

ret_hor_wind_data()

It applies the FFT based method to retrieve the horizontal wind information

ret_vert_wind_data()

It copies the vertical wind from the observations

lidarwind.wind_prop_retrieval.first_harmonic_amplitude(ds, dim='azm')

First harmonic amplitude

It calculates the complex amplitudes from the first harmonic from the observations along the azm coordinate

ds doppler_obs

lidarwind.wind_prop_retrieval_6_beam module

Module for estimating turbulence

class lidarwind.wind_prop_retrieval_6_beam.SixBeamMethod(data, freq=10, freq90=10)

Bases: object

6 beam method

Implementation of the 6 beam method to retrieve the Reynolds stress tensor components based on the 6 Beam method developed by Sathe at all 2015. See: https://doi.org/10.5194/amt-8-729-2015

Parameters:

• data (object) – an instance of the object generated by the lidarwind.GetRestructuredData()

• freq (int) – number of profiles used to calculate the variance

• freq90 (int) – number of profiles used to calculate the variance

Returns:

var_comp_ds – a dataset of the eynolds stress tensor matrix elementes

Return type:

xarray.DataSet

calc_variances(data, freq, freq90)

get_m_matrix()

This method populates the coefficient matrix (M). Each element of M is one of the coefficients from equation 3 from Newman et. all 2016. The lines 0 to 4 in M are the radial velocities coefficients from the non 90 deg elevation and different azimuths. Line 6 in M has the coefficients from the radial velocity at 90 deg elevation. See: https://doi.org/10.5194/amt-9-1993-2016

M x SIGMA = S

get_m_matrix_inv()

This method calculates the inverse matrix of M.

get_s_matrix()

This method fills the observation variance matrix (S).

get_sigma()

This method calculates the components of the Reynolds stress tensor (SIGMA).

SIGMA = M^-1 x S

get_variance(data, freq=10, name='rVariance')

This method calculates the variance from the observed radial velocities within a time window. The default size of this window is 10 minutes.

Parameters:

• data (xarray.DataArray) – a dataarray of the slanted azimuthal observations

• freq (int) – number of profiles used to calculate the variance

get_variance_ds()

This method converts the SIGMA into a xarray dataset.

Module contents

Top-level package for lidarwind package