Using the IMOS User Code Library with R
Table of Contents
- 1. Introduction
- 2. General Features of the IMOS user code library
-
3. Dataset examples – Using the NetCDF Parser for Plotting
- 3.1 AATAMS – Animal Tagging and Monitoring - non QC'd data
- 3.2 DWM – Deep Water Mooring
- 3.3 ACORN – Ocean Radar - non QC'd data
- 3.4 ANFOG – Ocean Gliders - QC'd good data
- 3.5 ANMN – National Mooring Network - QC'd good data
- 3.6 AUV – Autonomous Underwater Vehicle - non QC'd data
- 3.7 Argo – Argo Floats Program - non QC'd data
- 3.8 FAIMMS – Wireless Sensor Networks - QC'd good data
- 3.9 SOOP – Ship Of Opportunities
- 3.10 SRS – Satellite Remote Sensing
This document describes how to use the NetCDF parser from the IMOS User Code Library to load IMOS NetCDF data files in an R environment, extract variables and produce a variety of plots including depth profiles, time-series, and raster plots.
Each dataset-specific example below can be found in the demos folder as a separate R script.
The code library can be checked out using a Git client, or be downloaded as a zip file : https://github.com/aodn/imos-user-code-library/archive/master.zip
The ncdf4 package needs to be downloaded and installed in R for use of the IMOS NetCDF parser function. It can be downloaded from: http://cirrus.ucsd.edu/~pierce/ncdf/
In order to find a dataset you are interested in, please refer to the portal help: http://help.aodn.org.au/help/?q=node/6. This is a how-to guide that can help users find an IMOS NetCDF file. When downloading your chosen dataset from the portal, choose one of the download options “List of URLs”, or “All source NETCDF files” to obtain netCDF files.
For users who are already familiar with IMOS facilities and datasets, IMOS NetCDF files are also directly accessible via an OPeNDAP catalog at : http://thredds.aodn.org.au/thredds/catalog/IMOS/catalog.html
Most of the examples in the following sections use the ‘Data URL’ of a dataset. If you have downloaded your dataset from the portal, the data URL is the file path to the file on your local machine. If you are using the THREDDS catelog, the file does not have to be downloaded to your local machine first – the OPeNDAP data URL can be parsed into R. The OPeNDAP data URL is found on the ‘OPeNDAP Dataset Access Form’ page (see http://help.aodn.org.au/help/?q=node/11), inside the box labelled ‘Data URL’ just above the ‘Global Attributes’ field.
Note: the list of URL’s generated by the IMOS portal when using that download option can be converted to a list of OPeNDAP data URL’s by replacing string http://data.aodn.org.au/IMOS/opendap with http://thredds.aodn.org.au/thredds/dodsC/IMOS.
To parse a NetCDF file in R, load the ncdf4 package and NetCDF parser function (‘ncParse.R’) and specify into your R environment the local or remote location of the NetCDF file. This is achieved using the following script.
Type in your R command window:
## Load the ncdf4 package and NetCDF parser function
library( ncdf4)
source( '/path/to/ncParse.R')
## Specify the location of the NetCDF file
# Remotely
file_URL <- 'http://thredds.aodn.org.au/thredds/dodsC/IMOS/eMII/demos/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/transect/2011/07/IMOS_SOOP-TMV_TSB_20110709T092744Z_VLST_FV02_transect-D2M_END-20110709T203700Z.nc'
dataset <- ncParse( file_URL)
# Locally
file <- '/path/to/NetCDFfileName.nc'
dataset <- ncParse( file)This creates a netCDF dataset object, through which you can access all the contents of the file.
The ncParse function produces a R list of three elements named metadata, dimensions, and variables. The ‘metadata’ list contains the NetCDF global attributes while the ‘dimensions’ and ‘variables’ lists respectively contain the dimension and variable attributes and their values. To examine the structure of the dataset parsed by the ncParse function, type the following code in the R console.
summary( dataset)To list all the global attributes of the NetCDF file, type dataset$metadata in the R console.
Then to extract the value of an element of the ‘dataset$metadata’ list, for instance ‘title’, type the following code in the R console.
dataset$metadata
title <- dataset$metadata$title
titleTo list all the variables of the NetCDF file, type the following text in the R console.
summary( dataset$variables)To list all the attributes of a variable, for instance TEMP, type:
str(dataset$variables$TEMP)All data values can be extracted by typing in the R console
dataset$variables$TEMP$dimensions
dataset$variables$TEMP$dataAll quality control flag values can be extracted by typing in the R console
dataset$variables$TEMP$flagsTo list all the dimensions of the NetCDF file and their attributes, type the following text in the R console.
summary( dataset$dimensions)All data values from the TIME dimension can be extracted by typing in the R console
dataset$dimensions$TIME$dataTo harvest only metadata from an IMOS NetCDF file (i.e. global attributes, along with dimension and variable attributes), type the following code in the R console.
dataset <- ncParse( file_URL, 'metadata')Parsing a subset of variables can be particularly useful when dealing with large dataset, such as gridded satellite products. To parse only data and metadata from certain variables, for instance ‘TEMP’ and ‘PSAL’, type the following code in the R console.
dataset <- ncParse( file_URL, 'all', c( 'PSAL', 'TEMP'))```
The Australian Animal Tagging And Monitoring System (AATAMS) is a coordinated marine animal tagging project. CTD Satellite Relay Data Loggers are used to explore how marine mammal behaviour relates to their oceanic environment.
NetCDF files can be found at : http://thredds.aodn.org.au/thredds/catalog/IMOS/AATAMS/marine_mammal_ctd-tag/catalog.html
In the example below, the ncParse function is used to extract temperature and salinity data. Those data are then used in conjunction with depth data to produce temperature and salinity profiles and time series plots.
To paste the code in your R environment, please copy it from : https://github.com/aodn/imos-user-code-library/blob/master/R/demos/aatams.R
To run those lines of commands, the ‘aqfig’ and ‘maps’ packages need to be installed and loaded.
Top. Profile and time series plots of sea water temperature recorded by a Southern elephant seal (tag number: ct64-M746-09). Bottom. Map showing the movements and sea surface salinity recorded during the migration of the same Southern elephant seal from Macquarie Island.
Water temperature and salinity depth profiles recorded by a Southern elephant seal (tag number: ct64-M746-09)
The Southern Ocean Time Series (SOTS) sub-facility provides high temporal resolution observations in sub-Antarctic waters. Observations are broad and include measurements of physical, chemical and biogeochemical parameters from multiple deep-water moorings.
NetCDF files can be found at : http://thredds.aodn.org.au/thredds/catalog/IMOS/DWM/SOTS/catalog.html
In the example below, the ncParse function is used to extract temperature data from a Pulse mooring instrument and then produce a temperature time series plot.
To paste the code in your R environment, please copy it from :
https://github.com/aodn/imos-user-code-library/blob/master/R/demos/dwm.R
Time series plot of sea water temperature.
The Australian Coastal Ocean Radar Network (ACORN) facility comprises a coordinated network of HF radars delivering real-time, non-quality controlled and delayed-mode, quality controlled surface current data into a national archive.
NetCDF files can be found at : http://thredds.aodn.org.au/thredds/catalog/IMOS/ACORN/catalog.html
Monthly aggregated files are also available in the following folders: monthly_gridded_1h-avg-current-map_QC monthly_gridded_1h-avg-current-map_non-QC
In the example below, the ncParse function is used to extract longitude, latitude, and velocity data for a single time value and then produce a raster plot of sea surface velocity on a longitude/latitude grid, with arrows indicating the direction and speed of sea surface currents.
To paste the code in your R environment, please copy it from : https://github.com/aodn/imos-user-code-library/blob/master/R/demos/acorn.R
To run those lines of commands, the ‘raster’ package needs to be installed and loaded.
Gridded plot showing the velocity and direction of sea surface currents.
The Australian National Facility for Ocean Gliders (ANFOG), with IMOS/NCRIS funding, deploys a fleet of eight gliders around Australia.
NetCDF files can be found at : http://thredds.aodn.org.au/thredds/catalog/IMOS/ANFOG/seaglider/catalog.html
In the example below, the ncParse function is used to extract depth, salinity, and time data and then produce a multiple time series plot. Only the data points with a Quality Control flag greater than 1 (which means 'good data', please refers to IMOS NetCDF User Manual for a description of the Quality Control, available at http://imos.org.au/facility_manuals.html)
To paste the code in your R environment, please copy it from :
https://github.com/aodn/imos-user-code-library/blob/master/R/demos/anfog.R
Time series plot of sea water salinity. Only data with good quality control flags are represented.
The Australian National Mooring Network Facility is a series of national reference stations and regional moorings designed to monitor particular oceanographic phenomena in Australian coastal ocean waters.
NetCDF files can be found at : http://thredds.aodn.org.au/thredds/catalog/IMOS/ANMN/catalog.html
In the example below, the ncParse function is used to extract depth, time and U current velocity data measured with an ADCP instrument (in Western Australia) and then produce a raster plot of U current velocity on a depth/time grid.
To paste the code in your R environment, please copy it from : https://github.com/aodn/imos-user-code-library/blob/master/R/demos/anmn_adcp.R
To run those lines of commands, the ‘ggplot2’ package needs to be installed and loaded.
Gridded plot showing the variations in sea water velocity with time at different depths, made using ADCP data.
The IMOS Autonomous Underwater Vehicle (AUV) Facility operates an ocean going AUV called Sirius capable of undertaking high resolution, geo-referenced survey work.
NetCDF files can be found at : http://thredds.aodn.org.au/thredds/catalog/IMOS/AUV/catalog.html
In the example below, the ncParse function is used to extract depth, temperature, and time data and then produce a multiple time series plot showing the variation of water temperature with depth and time during the robot’s dive.
To paste the code in your R environment, please copy it from : https://github.com/aodn/imos-user-code-library/blob/master/R/demos/auv.R
Time series plot of sea water temperature.
Argo floats have revolutionised our understanding of the broad scale structure of the oceans to 2000 m depth. In the past 10 years more high resolution hydrographic profiles have been provided by Argo floats then from the rest of the observing system put together. Each Argo float is identified by a unique identification number called a WMO ID.
NetCDF files can be found at : http://thredds.aodn.org.au/thredds/catalog/IMOS/Argo/aggregated_datasets/catalog.html
In the example below, the ncParse function is used to extract and then plot temperature and salinity data from an aggregated file (one file per year per basin: Atlantic, Indian, Pacific North, Pacific South), along with all the surface locations of all the Argo floats.
To paste the code in your R environment, please copy it from : https://github.com/aodn/imos-user-code-library/blob/master/R/demos/argo1.R
To run those lines of commands, the ‘maps’ package needs to be installed and loaded.
Locations of all Argo floats deployed in the South Pacific ocean in 2002.
Water temperature and salinity depth profiles made using Argo float ‘5900105’ data.
In the example below, the ncParse function is used to extract and then plot all temperature profiles recorded by an Argo float, along with its trajectory.
To paste the code in your R environment, please copy it from :
https://github.com/aodn/imos-user-code-library/blob/master/R/demos/argo2.R
Top. Gridded plot showing the variations in sea water temperature with time at different depths. Bottom. Trajectory of Argo float ‘5900106’.
The IMOS Facility for Intelligent Monitoring of Marine Systems is a sensor network established in the Great Barrier Reef off the coast of Queensland, Australia. A 'sensor network' is an array of small, wirelessly interconnected sensors that collectively stream sensor data to a central data aggregation point. Sensor networks can be used to provide spatially dense bio-physical measurements in real-time.
NetCDF files can be found at : http://thredds.aodn.org.au/thredds/catalog/IMOS/FAIMMS/catalog.html
In the example below, the ncParse function is used to extract temperature and time data and then produce a temperature time series plot. Only data points which have a flag value equal to 1 are used (which means 'good data', please refers to IMOS NetCDF User Manual for a description of the Quality Control, available at http://imos.org.au/facility_manuals.html).
To paste the code in your R environment, please copy it from : https://github.com/aodn/imos-user-code-library/blob/master/R/demos/faimms.R
Time series plot of sea water temperature. Only data with good quality control flags are represented.
IMOS Ship of Opportunity Underway Expandable Bathythermographs (XBT) group is a research and data collection project working within the IMOS Ship of Opportunity Multi-Disciplinary Underway Network sub-facility.
NetCDF files can be found at : http://thredds.aodn.org.au/thredds/catalog/IMOS/SOOP/SOOP-XBT/catalog.html
In the example below, the ncParse function is used to extract and then plot temperature and depth data from an XBT profile. Only the data points with a quality control flag of 1 or 2 are plotted (which means 'good data' and ‘probably good data’ respectively, see the IMOS NetCDF User Manual available at http://imos.org.au/facility_manuals.html).
To paste the code in your R environment, please copy it from : https://github.com/aodn/imos-user-code-library/blob/master/R/demos/soop_xbt.R
To run those lines of commands, the ‘raster’ package needs to be installed and loaded.
Water temperature depth profile.
Top. Gridded plot showing the variations in sea water temperature with time at different depths. Bottom. Trajectory of the SOOP XBT vessel.
The Sea Surface Temperature (SST) sub-facility aims to enable accurate, quality controlled, SST data to be supplied in near real-time (within 24 hours) from SOOPs and research vessels in the Australian region.
NetCDF files can be found at : http://thredds.aodn.org.au/thredds/catalog/IMOS/SOOP/SOOP-SST/catalog.html
In the example below, the ncParse function is used to extract and then plot temperature and time data from a SOOP SST cruise, along with the vessel trajectory. Only the data points which ‘have passed all tests’ according to BoM QC specifications are plotted.
To paste the code in your R environment, please copy it from :
https://github.com/aodn/imos-user-code-library/blob/master/R/demos/soop_sst.R
Top. Time series plot of sea water temperature. Bottom. Trajectory of the SOOP SST vessel. Only data which ‘have passed all tests’ are represented.
The bio-optical data base underpins the assessment of ocean colour products in the Australian region (e.g. chlorophyll a concentrations, phytoplankton species composition and primary production).
NetCDF files can be found at : http://thredds.aodn.org.au/thredds/catalog/IMOS/SRS/BioOptical/catalog.html
In the example below, the ncParse function is used to extract and then plot a Chlorophyll-a profile (High Performance Liquid Chromatography of pigments in discrete sea-water samples)
To paste the code in your R environment, please copy it from : https://github.com/aodn/imos-user-code-library/blob/master/R/demos/srs_BioOptical_pigment.R
Chlorophyll a concentration depth profile.
The bio-optical data base underpins the assessment of ocean colour products in the Australian region (e.g. chlorophyll a concentrations, phytoplankton species composition and primary production).
NetCDF files can be found at : http://thredds.aodn.org.au/thredds/catalog/IMOS/SRS/BioOptical/catalog.html
In the example below, we demonstrate how to use the netCDF4 module to plot (1) the variation of Absorption coefficients of CDOM (gilvin) in discrete sea-water samples at different wavelengths and (2) the variation of absorption coefficients of CDOM at different wavelengths and different depths.
To paste the code in your R environment, please copy it from : https://github.com/aodn/imos-user-code-library/blob/master/R/demos/srs_BioOptical_absorption.R
To run those lines of commands, the ‘raster’ package needs to be installed and loaded.
Gridded plot showing the variations in the absorption coefficient of CDOM at different wavelengths and depths.
Plot showing the variations in the absorption coefficient of CDOM at different wavelengths and discrete depths.
As part of IMOS, the Australian Bureau of Meteorology produces high-resolution satellite SST products over the Australian region. All these products follow the latest International Group for High Resolution Sea Surface Temperature (GHRSST: www.ghrsst.org) file formats. For more information about SRS SST products, please refer to the SRS product Help page : http://help.aodn.org.au/help/?q=node/67
NetCDF files can be found at : http://thredds.aodn.org.au/thredds/dodsC/IMOS/SRS/GHRSST-SSTsubskin/
In the example below, the ncParse function is used to extract longitude, latitude, and sea surface temperature data for a single time value and then produce a raster plot of sea surface temperature on a longitude/latitude grid.
Warning: this dataset can take a while to be loaded on your machine. To paste the code in your R environment, please copy it from : https://github.com/aodn/imos-user-code-library/blob/master/R/demos/srs_l3p_mosaic.R
To run those lines of commands, the ‘raster’ and ‘maps’ packages need to be installed and loaded. Note that to reduce memory usage, only the sea surface temperature variable is harvested from the NetCDF file, and it is advised to download a copy of the NetCDF file on your local machine using the HTTP server link on the Thredds server page.
Gridded plot showing the variations in sea surface temperature with latitude and longitude.
Please refer to the SRS product Help page: http://help.aodn.org.au/help/?q=node/67
NetCDF files can be found at : http://thredds.aodn.org.au/thredds/dodsC/IMOS/SRS/SRS-SST/L3C-01day/
In the example below, the ncParse function is used to extract longitude, latitude, and sea surface temperature data for a single time value and then produce a raster plot of sea surface temperature on a longitude/latitude grid.
Warning: this dataset can take a while to be loaded on your machine, or create some java memory heap space errors if your machine lacks off memory. To paste the code in your R environment, please copy it from : https://github.com/aodn/imos-user-code-library/blob/master/R/demos/srs_l3c_multi_swath.R
To run those lines of commands, the ‘raster’ and ‘maps’ packages need to be installed and loaded. Note that to reduce memory usage, only the sea surface temperature variable is harvested from the NetCDF file, and it is advised to download a copy of the NetCDF file on your local machine using the HTTP server link on the Thredds server page.
Example of Sea Surface Temperature plot from a L3C product
Please refer to the SRS product Help page: http://help.aodn.org.au/help/?q=node/67
NetCDF files can be found at : http://thredds.aodn.org.au/thredds/dodsC/IMOS/SRS/SRS-SST/L3S-01day/
In the example below, the ncParse function is used to extract longitude, latitude, and sea surface temperature data for a single time value and then produce a raster plot of sea surface temperature on a longitude/latitude grid.
Warning: this dataset can take a while to be loaded on your machine. To paste the code in your R environment, please copy it from : https://github.com/aodn/imos-user-code-library/blob/master/R/demos/srs_l3s_multi_swath.R
To run those lines of commands, the ‘raster’ and ‘maps’ packages need to be installed and loaded. Note that to reduce memory usage, only the sea surface temperature variable is harvested from the NetCDF file, and it is advised to download a copy of the NetCDF file on your local machine using the HTTP server link on the Thredds server page.
Example of Sea Surface Temperature plot from a L3S product