FIXME 20220519: begin again here , cross-checking against
.xml file to ensure all called out file names are being represented
with <para> tags by removing initial tabs and * from initial
level file/folder names.
1. Sample Collection
Water samples are collected at each monitoring site in triplicate
and filtered with a 0.7 µm glass fiber filter. N.b. Dates in the
analysis-ready data (i.e., those data available at the root level
of the data product in the AnnualAggregateFile-NH4-SRP-WY2017.csv
file) refer to the date of water sample collection. Dates of
instrument measurement may be found in the following files:
01_input/00a-202103-Proc/00_NH4-PO4/LTREB_NH4_PO4_AP2_2017_08_15_2017_09_28_RESULTS.xlsx;
01_input/00a-202103-Proc/00_NO3_DEPRECATED/LTREB_NO3_AP2_2017_08_15_2017_09_28_RESULTS.xlsx;
Further documentation on sample collection and filtration methods
is available in the following file:
02_protocol/00a-202103-Proc/00_NH4-PO4/SOPs/LTREB
Monitoring Field Standard Operating Procedure Utzman Ver 4.0 Final
March 2021.pdf
2. Measurement methods
Where more details are available, each description below is
cross-referenced with the brief methods caption provided in the
descriptions of the data table columns.
Flow Injection Analyzer: The Astoria Pacific
2 Segmented Flow Analyzer (AP2) instrument measures water samples
under lab conditions for concentrations of the nutrients nitrate
(N-NO3), ammonium (N-NH4), and soluble reactive phosphorus
(P-PO4). The instrument processes sample data on the proprietary
Flow Analyzer Software Package, FASPac 2 Version 2.20 (released
12/2012). The FASPac 2 program is designed to interface with a
continuous-flow analyzer and to perform analyses in batches of up
to 1000 samples. This software calculates concentrations from
absorbance readings and provides a copy of the run to store in
individual databases. See also
01_input/meta/Ancillary/LTREB_Software_Info.docx for more detail
on FASPac 2.
Further documentation on measurement methods and data management
is available in
02_protocol/00a-202103-Proc/00_NH4-PO4/SOPs/
in the following files:
Quality_Assurance_Quality_Control_Protocol_for_Nutrients.docx;
ValettLabAP2dataMgmtSOP_V3.pdf
3. Data processing methods
This data product is designed to provide the provenance of final
tables in a pipeline format representing phases of the scientific
workflow, to the degree that provenance is available. Raw data
files from instruments and manufacturer software are available in
the 01_Input.zip file. Data were preliminarily processed using the
computational Excel templates found in 02_protocol.zip file. The R
scripts and R markdown used for final processing of the data and
metadata are also available in the 02_Protocol.zip file.
Incremental files created in the course of data processing and
reflective of significant changes are available in the
03_Incremental.zip file. Ready-to-use analytical data are
available at the data package’s root level (on the dataset landing
page under the "Resources" section), and are described
in the full metadata. Contents of the .zip files are described
briefly in the subsections below.
The data file name schema used in this data product:
<Project>_<Analyte>_<Machine>_<Start Date
YYYY_MM_DD>_<End Date YYYY_MM_DD>_<Time>_<File
Type>
Schema key:
Project: LTREB;
Analyte: One or more of NO3N, NH4N, or PO4;
Machine: AP2 (Flow Injection Analyzer);
Start/End Date: Dates of sample collection;
Time: Time of sample collection, (if used) - Mountain
Standard Time; File Type: Data product (DP), metadata,
readme, results, sample, process, visualization, final
3.1. Contents of 01_input.zip
00a-202103-Proc/ - (where "Proc" =
"processed") - Directory containing initial
instrument-derived, FASPac2-processed data and simple
visualizations for all analytes run; files are further organized
by analyte as follows:
00_NH4-PO4/LTREB_NH4_PO4_AP2_2017_08_15_2017_09_28_RESULTS.xlsx
- Original ammonium and soluble reactive phosphate raw data xlsx
file containing results from run on AP2 machine. See
LTREB_FILETYPE_README.docx for column header details.
00_NH4-PO4/LTREB_NH4_PO4_AP2_2017_08_15_2017_09_28_VISUALIZATION.fpx
- fpx file - Faspac software must be downloaded on desktop to
access this file type. Original raw data file for ammonium and
soluble reactive phosphate identifies absorbance peaks and
calculates concentrations reported in mg/L. See
LTREB_FILETYPE_README.docx for details.
00_NO3_DEPRECATED/LTREB_NO3_AP2_2017_08_15_2017_09_28_RESULTS.xlsx
- Original nitrate raw data xlsx file containing results from run
on AP2 machine. See LTREB_FILETYPE_README.docx for column header
details.
00_NO3_DEPRECATED/LTREB_NO3_AP2_2017_08_15_2017_09_28_VISUALIZATION.fpx
- fpx file - Faspac software must be downloaded on desktop to
access this file type. Original raw data file for nitrate
identifies absorbance peaks and calculates concentrations reported
in mg/L. See LTREB_FILETYPE_README.docx for column header details.
00_NO3_DEPRECATED/LTREB_NH4_PO4_NO3_AP2_2017_08_15_2017_09_28_SAMPLE.xlsx
- Original ammonium, soluble reactive phosphate, and nitrate raw
data template xlsx file containing sample ID information. See
LTREB_FILETYPE_README.docx for column header details.
00_NO3_DEPRECATED/LTREB_FILETYPE_README.docx
- Original internal metadata file with file naming structure,
content types, and column header metadata (where relevant) for the
three content types found in 01_input: .
[projectName]_[analytes]_[machine]_[dateStart]_[dateEnd]_SAMPLE.xlsx
.
[projectName]_[analytes]_[machine]_[dateStart]_[dateEnd]_VISUALIZATION.fpx
.
[projectName]_[analytes]_[machine]_[dateStart]_[dateEnd]_RESULTS.xlsx
meta/ - Directory with project templates for
metadata.
Ancillary/ -Directory of original internal
narrative metadata files (for reference only, not thoroughly
verified)
2021 Site Table.xlsx - Microsoft Excel file
with authoritative project site list at the time of publication of
these data.
attributes.xlsx - Microsoft Excel file with
description of data table column attributes found in
analysis-ready data files.
people.xlsx - Excel file with tabular data on
people associated with the project, used for metadata generation.
3.2. Contents of 02_protocol.zip
Template .xlsx files included here are computational spreadsheets
representing AP2 run calibrant concentrations and run details, as
well as the sample QAQC processing calculations described in
corresponding narrative protocol documents in the SOPs directory.
R code in the protocol depends on R packages: tidyr, readtext,
knitr, readxl, EMLassemblyline, and xml2. Code also makes system
calls to the zip command using arguments compatible with Info-Zip
implementation 3.0, which appear to be mostly consistent with
linux zip arguments.
00a-202103-Proc/ - (where "Proc" =
"processed") - Directory containing computational
spreadsheet templates used in further processing (calibration
curves; etc) of initial instrument-derived FASPac2-processed data
(see 03_incremental files, parallel directory structure for
resulting processed data from this step). Directory is further
organized by analyte, whereunder are found computational
spreadsheets; and an SOPs directory containing narrative protocols
and standard operating procedures (SOPs) for field, lab, and data
management procedures; as follows:
00_NH4-PO4/ - Directory with analyte-specific
files for NH4 and PO4.
00_NH4-PO4/AP2_NH4N_SRP_Processing_Template.xlsx
- File: Computational spreadsheet that performs the sample
processing calculations for ammonium as described in corresponding
narrative protocol documents in the SOPs directory.
00_NH4-PO4/AP2_Sample_Table_Template_NH4N_SRP.xlsx
- File: Computational spreadsheets representing AP2 run calibrant
concentrations and run details for ammonium, as described in
corresponding narrative protocol documents in the SOPs directory.
00_NH4-PO4/LTREB_NH4_PO4_AP2_2017_08_15_2017_09_28_PROCESS.xlsx
- File: Processed ammonium and soluble reactive phosphate data
file inclusive of both computational equations and initial QAQC’d
results from run on AP2 machine.
00_NO3_DEPRECATED/ - Directory with
analyte-specific files for NO3.
00_NO3_DEPRECATED/00-README-LTREB_NO3N_AP2_2017_08_15_2017_09_28_RESULTS_INVALID.docx
- File: Declarative statement providing rationale for excluding
nitrate data from approved data release.
00_NO3_DEPRECATED/AP2_NO3N_Processing_Template.xlsx
- File; Computational spreadsheet that performs the sample
processing calculations for nitrate as described in corresponding
narrative protocol documents in the SOPs directory.
00_NO3_DEPRECATED/AP2_Sample_Table_Template_NO3N.xlsx
- File: Computational spreadsheets representing AP2 run calibrant
concentrations and run details for nitrate, as described in
corresponding narrative protocol documents in the SOPs directory.
00_NO3_DEPRECATED/LTREB_NO3_AP2_2017_08_15_2017_09_28_PROCESS.xlsx
- File: Processed nitrate data file inclusive of both
computational equations and initial QAQC’d results from run on AP2
machine.
SOPs-NH4-PO4-NO3/ - Directory with narrative
standard operating procedures (SOPs) used in the field and lab for
generating NH4, P04, and NO3 data.
SOPs-NH4-PO4-NO3/LTREB Monitoring Field Standard
Operating Procedure Utzman Ver 4.0 Final March 2021.pdf
- File: PDF version of procedures detailing field sampling,
filtering, measuring, cleaning, and storage protocols.
SOPs-NH4-PO4-NO3/Quality_Assurance_Quality_Control_Protocol_for_Nutrients.docx
- File: QAQC procedures to process measured samples including
calibration curve fitting to correct concentrations.
SOPs-NH4-PO4-NO3/ValettLabAP2dataMgmtSOP_V3.pdf
- File: This protocol serves to prepare, archive, and process data
generated by the Astoria Pacific 2 Segmented Flow Analyzer (AP2).
The AP2 is used to generate data for the nutrients: N-NO3, N-NH3,
P-PO4.
00b-202110-QAQC/ - Directory containing
computational spreadsheet templates used in further quality
assurance processing of initial instrument-derived
FASPac2-processed data (see 03_incremental files, parallel
directory structure for resulting processed data from this step).
Directory is further organized by analyte, whereunder are found
computational spreadsheets, as follows:
00b-202110-QAQC/00_NH4-PO4/ - Directory with
analyte-specific files for NH4 and PO4.
00b-202110-QAQC/00_NH4-PO4/LTREB_NH4_PO4_AP2_2017_08_15_2017_09_28_PROCESS.xlsx
- File: Re-processed and QA’d ammonium and soluble reactive
phosphate data file inclusive of both computational equations and
QA’d results.
00b-202110-QAQC/00_NO3_DEPRECATED/ -
Directory with analyte-specific files for NO3.
00b-202110-QAQC/00_NO3_DEPRECATED/00-README-LTREB_NO3N_AP2_2017_08_15_2017_09_28_RESULTS_INVALID.docx
- File: Declarative statement providing rationale for excluding
nitrate data from approved data release.
00b-202110-QAQC/00_NO3_DEPRECATED/LTREB_NO3_AP2_2017_08_15_2017_09_28_PROCESS.xlsx
- File: Re-processed and QA’d nitrate data file inclusive of both
computational equations and QA’d results.
00c-202204-validationQAQC/ - Directory
containing a statement on final confirmatory quality assurance
processing of previously QAQC’d data (see 03_incremental files,
parallel directory structure for resulting processed data).
00c-202204-validationQAQC/00-README-validationQAQC.txt
- File: Declarative statement documenting the chain of processing
and QAQC decisions.
.RProfile - An R profile file with R code
that establishes the appropriate working directory for the R
project used to generate this data product.
01_processing.Rmd - R markdown defining the
reformatting of the data tables and associated column metadata.
01_processing_compile.R - R script that
compiles the processing R markdown to an HTML document
(03_Incremental.zip - processing_notes.html)
10_metadata.Rmd - R markdown defining the
generation of the metadata file and the EML file based on EDI EML
assembly line.
10_metadata_compile.R - R script that
compiles the metadata R markdown to an HTML document
(metadata.html) and an EML file (eml.xml).
99_runall.R - Convenience R script for fully
regenerating the data product.
RStudioProject.Rproj - An RStudio project
file provided for convenience in working with the code generating
this data product with the appropriate working directory.
3.3. Contents of 03_incremental.zip
A combination of intermediary files generated at different stages
of data processing and curation.
00a-202103-Proc/ - (where "Proc" =
"processed") - Directory containing data tables
resulting from the computational spreadsheet templates used in
processing (calibration curves; etc) of initial instrument-derived
FASPac2-processed data (see 02_protocol files, parallel directory
structure for computational processes and SOPs generating these
data). Directory is further organized by analyte, whereunder are
found data at an incremental stage of processing.
00_NH4-PO4/ - Directory with analyte-specific
data files for NH4 and PO4.
00_NO3_DEPRECATED/ - Directory with
analyte-specific files for NO3.
00b-202110-QAQC/ - Directory containing data
tables resulting from the computational spreadsheet templates used
in 2nd stage quality assurance processing of initial
instrument-derived FASPac2-processed data (see 02_protocol files,
parallel directory structure for computational processes and SOPs
generating these data). Directory is further organized by analyte,
whereunder are found data at an incremental stage of processing.
An aggregate file of all measured nutrients (i.e., NH4, PO4, and
NO3) is included for convenience in two formats.
00_NH4-PO4/ - Directory with analyte-specific
data files for NH4 and PO4.
00_NO3_DEPRECATED/ - Directory with
analyte-specific data files for NO3.
00b-202110-QAQC/Annual aggregate file NH4 SRP NO3 WY
2017.csv - CSV File: Processed and QA’d ammonium,
soluble reactive phosphate, and nitrate data file. Deprecated
nitrate data represented here are in their most highly stewarded
form; no further versions are available for nitrate.
00b-202110-QAQC/Annual aggregate file NH4 SRP NO3 WY
2017.xlsx - XLSX File: Processed and QA’d ammonium,
soluble reactive phosphate, and nitrate data file. Deprecated
nitrate data represented here are in their most highly stewarded
form; no further versions are available for nitrate.
00c-202204-validationQAQC/ - Directory
containing final confirmatory quality assured data, prior to
curatorial processing (e.g., date formatting, siteID normalizing,
etc.) (see 02_protocol files, parallel directory structure for
processes leading to these data).
00c-202204-validationQAQC/AnnualAggregateFile-NH4-SRP-WY2017.csv
- Processed and QA’d ammonium and soluble reactive phosphate data
file. Deprecated nitrate data are excluded at this stage.
meta/ - Directory with narrative metadata
elements incorporated into metadata summary and EML
meta/abstract.md - A markdown text file
sourcing the data abstract.
meta/methods.md - [this file.] A markdown
text file sourcing the description of methods.
meta/purpose.txt - An unformatted text file
sourcing the description of purpose.
meta/intellectual_rights.md - A text file
sourcing the license for data reuse, used for metadata generation.
metadata_summary.html - [**Not included in
downloaded data product, however: R code included in this product
will generate this temporary file in the ./03_incremental folder.
This temporary file contains no unique data or metadata and thus
is not archived in the corresponding zip file, but may appear when
when new users run the R code on their local systems.] File descr:
Summary of metadata and the process of archiving the workflow
pipeline and compiling the associated metadata into EML.
processing_notes.html - File: Metadata
regarding the processing of the raw data files to the product’s
data tables and basic visualizations.
temp/ [**Not included in downloaded data
product, however: R code included in this product will generate
temporary files in the ./03_incremental/temp folder. These
temporary files contain no unique data or metadata and thus are
not archived in the corresponding zip file, but may appear when
when new users run the R code on their local systems.]
4. Data product development history
2021 December 28: First curated draft developed using dissolved
organic carbon 2019 water year as a template.
2022 May 19: EDI staging server-ready draft ready for upload to
EDI.