GHRSST-PP Validation Activities

The importance of properly validating GHRSST-PP products is great. The value and utility of all observational data products depends ultimately on the quality of their validation which is essential to building user confidence. In this sense, the purpose of validation activities, embodied in the HR-DDS and the GHRSST-PP MDB system together with field campaigns is to underpin aim and credibility of the GHRSST-PP itself. SST product validation/verification should be based on inter-comparisons:

• With in situ observations made using ships and buoys of SST - hence the need for a GHRSST-PP Match-up database (MDB)
• Between remotely sensed observations from independent sensors (DDS)
• With analysed SST and climatology (DDS)
• L4 analyses should be compared to input data sets and independent in situ observations

A L4 verification facility has been configured by the Medspiration Team for the Mediterranean Sea UHR-L4 analysis

The regular publication of GHRSST-PP performance metrics is considered essential to the success of the GHRSST-PP in terms of generating confidence in the operation of the R/GTS framework. The GHRSST-PP web space is the logical place to publish monthly metric reports together with a monthly 'highlights' summary of important events. Once meterics are in production, these paghes will be brought on-line.

Validation and metrics for the GHRSST-PP: Towards operational sustainability

Only by monitoring key activities of the GHRSST-PP against identified criteria can actions to mitigate identified problems be raised in a timely manner, progress be monitored in a meaningful way and successes be captured and reported promptly. The monitoring of theGHRSST-PP will be a core activity of the GHRSST-PP Project Office and is an important tool to aid the transition to sustained operations and funding. Metrics are essential as they will provide the evidence of how well the present and post-GHRSST-PP operational system is performing. Metrics must be realistic and to be effective they must not be too easy or too hard to achieve requiring a consideration of their utility.
A Metric defines common quantities and diagnostics with given mathematical definitions. Thus, given a specific metric, there is a need to absolutely classify numerical threshold thatdefines acceptable and unacceptable performance. Such a threshold may be called a target where just one or two target figures is all that is required rather than a range of "acceptability" which should be avoided. Consider the following statement:

'RDAC L2P delivery/availability of data to the GDAC should be within 6 hours of the oldest data received'

A metric is then the percentage of data files that meet this 'target' for which 90% may be acceptable but 80% is not. Care must be taken to define appropriate scaling as perceptions are such that extreme (high or low) numbers carry implied performance. In addition, consistency is required in terms of scaling (i.e., low is bad and high is good or vice versa). In summary, metrics must provide a number that shows how GHRSST is performing for a number of well-defined operations.
A basic philosophy used to define GHRSST-PP metrics was agreed at the 6th GHRSST-PP Science Team Meeting, Exeter UK, May 2005. This is based on operational performance measures implemented for the day-to-day operation of Australia's TERSS X-band satellite reception facility in Hobart, Tasmania where metrics have been carefully designed as monetary bonuses are available to staff if certain metric targets are met each financial year.

GHRSST-PP v1.0 Metrics

The metrics required by the GHRSST-PP fall into several categories:

1. The timely delivery of data to users, to GDAC, and to RDACs.
2. The accuracy of SST fields (Number of in situ data points available for SST validation, rms, and bias).
3. The number of satellite data sets (orbits, overpasses) available to GDAC.
4. The extent to which all geographical areas covered.
5. The production of DDS: areas covered, timely on-line access to DDSs.
6. Access to in situ data via MDB records. Links to DDS.

Each metric must have meaningful measures and appropriate set targets. The results of these discussion are laid out in a discussion document which can be considered a Version 1.0 of GHRSST-PP metrics.