GHRSST-PP Validation Activities
The importance of properly validating GHRSST-PP
products is great. The value and utility of all observational data products depends
ultimately on the quality of their validation which is essential to building user confidence.
In this sense, the purpose of validation activities, embodied in the HR-DDS and the GHRSST-PP MDB
system together with field campaigns is to underpin aim and credibility of the GHRSST-PP itself.
SST product validation/verification should be based on inter-comparisons:
- With in situ observations made using
and buoys of SST - hence the need for a GHRSST-PP Match-up database (MDB)
- Between remotely sensed observations from independent sensors (DDS)
- With analysed SST and climatology (DDS)
- L4 analyses should be compared to input data sets and independent in situ observations
L4 verification facility
has been configured by the Medspiration Team for the Mediterranean Sea UHR-L4 analysis
The regular publication of GHRSST-PP performance metrics is considered essential to the success of the
GHRSST-PP in terms of generating confidence in the operation of the R/GTS framework.
The GHRSST-PP web space is the logical place to publish monthly metric reports together with a
monthly 'highlights' summary of important events. Once meterics are in production, these paghes will
be brought on-line.
Validation and metrics for the GHRSST-PP: Towards operational sustainability
Only by monitoring key activities
of the GHRSST-PP against identified criteria can
actions to mitigate identified problems be raised in a timely manner, progress be monitored
in a meaningful way and successes be captured and reported promptly. The
monitoring of theGHRSST-PP
will be a core activity of the GHRSST-PP Project Office and is an important
tool to aid the transition to sustained operations and funding. Metrics are essential
as they will provide the evidence of how well the present and post-GHRSST-PP operational
system is performing. Metrics must be realistic and to be effective they must not be
too easy or too hard to achieve requiring a consideration of their utility.
A Metric defines common quantities and diagnostics with given mathematical definitions.
Thus, given a specific metric, there is a need to absolutely classify numerical threshold thatdefines acceptable and unacceptable performance. Such a threshold may be called a
where just one or two target figures is all that is required rather than a range of "acceptability"
which should be avoided. Consider the following statement:
'RDAC L2P delivery/availability of data to the GDAC should be within 6 hours of the oldest data received'
A metric is then the percentage of data files that meet this 'target' for which
90% may be acceptable but 80% is not. Care must be taken to define appropriate scaling as perceptions are such
that extreme (high or low) numbers carry implied performance. In addition, consistency is required in terms of scaling
(i.e., low is bad and high is good or vice versa). In summary, metrics must provide a number
that shows how GHRSST is performing for a number of well-defined operations.
A basic philosophy used to define GHRSST-PP metrics was agreed at the 6th GHRSST-PP Science Team Meeting, Exeter UK, May 2005.
This is based on operational performance measures implemented for the day-to-day operation of Australia's
TERSS X-band satellite reception facility in Hobart, Tasmania where metrics have been carefully designed as
monetary bonuses are available to staff if certain metric targets are met each financial year.
GHRSST-PP v1.0 Metrics
The metrics required by the GHRSST-PP fall into several categories:
- The timely delivery of data to users, to GDAC, and to RDACs.
- The accuracy of SST fields (Number of in situ data points available for SST validation, rms, and bias).
- The number of satellite data sets (orbits, overpasses) available to GDAC.
- The extent to which all geographical areas covered.
- The production of DDS: areas covered, timely on-line access to DDSs.
- Access to in situ data via MDB records. Links to DDS.
Each metric must have meaningful measures and appropriate set targets. The results of these discussion are laid
out in a
which can be considered a Version 1.0 of GHRSST-PP metrics.