1. Introduction
The National Metrology Institutes (NMIs) are responsible for the International System of Units (SI) which provides the foundation for measurement around seven base units and a system of coherent derived units. There are three key concepts underpinning how the desired multi-century stability and world-wide consistency of these units is achieved: uncertainty analysis, traceability and comparisons [
1].
Uncertainty analysis is the systematic review of all sources of uncertainty associated with a particular measurement and the formal propagation of uncertainties through methods defined by the Guide to the Expression of Uncertainty in Measurement (GUM) [
2]. Metrological traceability to a measurement unit of the International System of Units [
3] is the concept that links all metrological measurements to the SI through a series of calibrations or comparisons. Each step in this traceability chain has rigorous uncertainty analysis, usually peer reviewed or audited and always documented. Comparisons [
4] are the process of validating an uncertainty analysis by comparing the measurement of artefacts by different laboratories.
NMIs must participate in regular (usually every 10 years) formal comparisons. Each technical discipline defines a limited number of “key comparisons” and these provide evidence to support uncertainty analysis for a certain number of related quantities in a “Calibration and Measurement Capability Database”. For example, the Consultative Committee for Photometry and Radiometry (CCPR) has defined a key comparison for six key measurands (spectral irradiance, spectral responsivity, luminous intensity, luminous flux, spectral diffuse transmittance and spectral regular reflectance). There is no key comparison for spectral radiance, as it is assumed that reliable results (results that are consistent with declared uncertainties) in the spectral irradiance comparison together with results for the comparison of reflectance provide sufficient evidence for spectral radiance measurements as well.
In 2008, the Committee on Earth Observation Satellites (CEOS) established and endorsed the Quality Assurance Framework for Earth Observation (QA4EO) [
5] which set general principles for Earth Observation (EO) data quality assurance and which follows the same metrological principles of the NMIs. Although QA4EO does not explicitly require traceability to SI and allows “or [to] a community-agreed reference”, it does state “preferably [to] SI”. To apply these principles in practice a concept of Fiducial Reference Measurements (FRM) was established and defined as [
6]:
Fiducial Reference Measurements (FRM) are a suite of independent, fully characterized, and traceable ground measurements that follow the guidelines outlined by the Group on Earth Observations (GEO)/CEOS Quality Assurance framework for Earth Observation (QA4EO). These FRM provide the maximum Return On Investment (ROI) for a satellite mission by delivering, to users, the required confidence in data products, in the form of independent validation results and satellite measurement uncertainty estimation, over the entire end-to-end duration of a satellite mission [
7].
The Fiducial Reference Measurements for Satellite Ocean Colour (FRM4SOC) project was established and funded by the European Space Agency (ESA) to provide support for evaluating and improving the state of the art in satellite ocean colour validation through a series of comparisons under the auspices of the CEOS. The project makes a fundamental contribution to the Copernicus Earth Observation system, led by the European Commission, in partnership with ESA, by ensuring high quality ground-based measurements for ocean colour radiometry (OCR) for use in validation of ocean colour products from missions like Sentinel-3 Ocean Colour and Land Instrument (OLCI) [
8] and Sentinel-2 Multi Spectral Imager (MSI) [
9].
In the past, a dedicated program to support the quality of Sea-viewing Wide Field-of-view Sensor (SeaWIFS) [
10] products was conducted. The Seventh SeaWiFS Intercalibration Round-Robin Experiment (SIRREX) [
11] and the Second Intercomparison and Merger for Interdisciplinary Ocean Studies (SIMBIOS) [
12,
13,
14] included comparisons of irradiance and radiance sources as well as radiometers by the teams participating in validation activities of the SeaWIFS. For the Medium Resolution Imaging Spectrometer (MERIS) [
15] a comparison of in situ measurements was performed at the Acqua Alta Oceanographic Tower (AAOT) [
16] and showed the discrepancies between the measurements mostly explained by the combined uncertainties of the compared measurements with a few exceptions.
From the metrological point of view, it is important to repeat such comparison exercises at regular time frames; firstly to achieve the measurements agreement, then to ensure that the consistency between the organisations is held stable and, finally, to enable new participants to verify their measurements capability.
This paper presents results from the first step in the OCR measurement chain and includes a comparison intended to verify the performance of the irradiance and radiance sources used to calibrate ocean colour radiometers.
3. Results
This section presents the results of the comparisons. The irradiance values are reported at the following wavelengths (350, 360, 370, 380, 390, 400, 450, 500, 600, 700, 800, 900) nm. This selection of wavelengths was dictated by the wavelengths reported in the calibration certificates from the participants. Although currently there are no OC missions that provide data below 400 nm, we wanted to include the ultraviolet spectral region in the comparison. There is a scientific interest to cover shorter wavelengths and this is indeed planned for the Plankton Aerosol Cloud ocean Ecosystem (PACE) [
19] EO mission.
The radiance values are reported at the transfer radiometer spectral bands values (412, 443, 491, 510, 560, 667, 684) nm. We have chosen to present results anonymously to keep focus on community consistencies, rather than individual laboratory. Participants were informed which laboratory they were.
3.1. Irradiance
The overall summary result of the irradiance comparison is presented in
Figure 4. The data series for each lamp used in the comparison are marked as Lamp A to Lamp N. The black dotted line indicates the mean ratio to NPL.
The results show an agreement between all measured lamps as all data series above 400 nm lay within the 0.99–1.013 range. The spread in the results is higher for shorter wavelengths as expected due to the higher measurements uncertainty presented in the absolute radiometric calibration for this spectral range.
The uncertainty for the individual lamp ratio is expressed by Equation (
3) and was calculated by combining several uncertainty components. All components are relative, thus expressed in percentage form.
where the
is the uncertainty in the ratio of the irradiance values from the lamp calibration certificate to measurements performed at NPL, the
is uncertainty from the lamp calibration certificate, note that uncertainty value is divided by two to convert it to a standard uncertainty from a coverage factor
used in the certificate. The
is the NPL scale uncertainty and the additional components related to the measurements performed at NPL which included noise
, and the uncertainty contributions to the irradiance value caused by: wavelength setting accuracy
, room stray light
, the lamp current uncertainty
, ageing of the lamp
and the lamp alignment
. The typical values for the ratio uncertainty expressed in percent are given in
Table 3.
3.2. Radiance
The overall summary result of the radiance comparison is presented in
Figure 5 for the radiometer with serial number 051 and in
Figure 6 for the radiometer with serial number 110. The colour triangles represent the seven spectral bands of the radiometers and the participants/set-ups are marked as letters from A to M. Please note that we present here 13 entries to the summary results that came from 10 participating institutes. The number of entries is higher because some organisations provided results at two different distance settings between the lamp and the reflectance panel, or two different measurements set-up like radiometers alignment to the central channel versus alignment to each channel in turn.
The results for both instruments show the same trends. The difference for any individual participant/set-up from the mean value is up to 4%, which is slightly higher than expected. A clear split into two groups can be seen. The majority of the results forms a group with radiance calibration coefficient values below the mean value and a second group of 4 participants has radiance calibration coefficient values around 3% above the mean value.
Each participant was asked to provide an uncertainty budget for their calibration of the radiance calibration coefficient values of the radiometers. The uncertainty of radiance measurements is calculated according to Equations (
4) and (
5). The wavelength dependence is not included in the following equations, but all the values were derived for the central wavelength of each of the radiometer spectral channels and take a similar form for all spectral channels.
where
is the relative combined uncertainty of radiance calibration coefficient,
and
are the uncertainty of the radiometer reading during the radiance calibration, with the
and
indicating light and dark readings, respectively,
is the uncertainty contributor due to the room stray light, expressed as percentage of the radiometer signal after the dark reading subtraction and
is the relative uncertainty of the radiance source. The radiance source uncertainty components are listed in Equation (
5).
where
is lamp irradiance absolute calibration uncertainty converted to
from the certificate values,
is the relative uncertainty associated with the distance setting, note that this component has a sensitivity coefficient equal to 2 (from the inverse square law) and, hence, in Equation (
5) there is a term
just before it. In addition, for all measurements at distances different to the 500 mm, the participants were requested to include a filament-offset uncertainty component to account for the difference in the plane of the distance setting and actual lamp filament position. The
is the relative uncertainty of the reflectance standard calibration. Please note we use the uncertainty of the reflectance factor calibration at 0
:45
geometry at
; for the case where a diffuse reflectance calibration value is corrected to 0
:45
geometry, an additional uncertainty of that correction has to be included in the equation, NPL recommends a 0.5% value. The remaining terms provide the relative uncertainty associated with radiance due to lamp current uncertainty
, due to lamp ageing uncertainty,
, due to alignment
of the lamp and the reflectance panel at 0
:45
configuration and due to target illumination non-uniformity within the FOV
.
A few examples of participants’ radiance measurements uncertainty expressed in percent are presented in
Table 4.
4. Discussion
The irradiance comparison results showed a good agreement within the expected uncertainty range. The observed differences are at a similar level to those reported in SIRREX-7 exercise [
11], that states 1.3% for the comparisons on the FEL lamps irradiance calibration. However, the comparison presented here included more lamps with different designs and traceability routes. The SIRREX irradiance comparison was expressed as relative percent difference between the NIST and Optronic calibration of the same lamp. Higher differences were observed for wavelengths below 400 nm as this is a more challenging spectral region for radiometric measurements. This indicates that, for future satellite missions the absolute calibration will have higher uncertainty for these new wavelengths.
It is important to note that the same lamps (irradiance standards) used elsewhere in a different laboratory environment, using different power supply or being aligned less carefully, could produce different results. It is also important to note that the spread of results for irradiance lamps would be expected to be higher if the lamps are operated outside the controlled environmental and stray light conditions of a high-quality radiometric laboratory or operated with power supplies that are not as stable as those used at NPL. Therefore, for any measurement using the lamp as the transfer standard for the spectral irradiance it is essential to consider all the uncertainty components given in Equation (
3), and not just those quoted on the calibration certificate.
The radiance comparison result, showing a lower level of agreement as compared to the irradiance comparison, led NPL to perform further investigations to explain the cause of the observed difference. The similarly-structured SIMBIOS comparison had results with the absolute value for each participant typically higher than expected by 2% [
13,
14], but the results were generally within the expected uncertainty range, with few exceptions. The SIMBIOS comparison used a radiometer as a reference and determined the difference in radiance measured by the radiometer and that calculated by each participating laboratory. In our comparison we tried to identify common features for the smaller group of the results that agree well with each other, but are around 3% higher than the top of the “main group”. The first common feature for the small group is that those results correspond to a lamp-panel distance of 500 mm; although some of the results in the main group were also made at that lamp-diffuser distance. Thus, the distance setting is not the only cause of the difference. The second common feature was the size of the illuminated patch on the reflectance target from the lamp. This size was influenced by the choice of light shields and other baffles in a particular laboratory setup. For the laboratories that use a lamp in a housing enclosure, the illumination patch size changes with the distance between the lamp and the diffuser.
NPL repeated a set of measurements to accommodate various conditions that participants may have in their own labs. The additional investigation was performed using an 18" Spectralon panel that was illuminated by the lamp at a distance of 500 mm and 1300 mm. The second distance was chosen as this was the longest distance used by a participant during the comparison. The size of the illuminated patch on the panel was varied from the fully illuminated panel, via a patch size with the diameter of around 23 cm to the small patch size of around 15 cm. The top panel in
Figure 7a presents the photographs taken for the three different illumination patch sizes. The bottom panel in
Figure 7b presents the percentage difference in the calibration coefficients obtained from five scenarios plotted as the data series from A to E. The series A, B and C represent the measurements at 500 mm distance for the fully illuminated panel, patch size 23 cm and the patch size 15 cm, respectively. The series D and E were done at 1300 mm distance for the fully illuminated and 23 cm patch size. The series A is set as the reference in this data set, therefore the percentage difference for this series is 0%.
A clearly visible positive bias can be seen for the measurements performed at 500 mm with a smaller size of the source. In addition, a negative bias can be seen for the measurements performed at the larger distance. All participants from the small group had their measurements done at the 500 mm source-panel distance with a relatively small size of their radiance source.
The most likely explanation for this is a combination of the instrument FOV and the size-of-source effect. These will also be influenced, to a small extent, by the distance from the panel to the radiometer, though we believe this is a minor consideration compared to the lamp-panel distance and the baffling that defined the source size Although the radiometers have a FOV defined as 3
, this is a FHWM value and with a Gaussian shape to the FOV, rather than a top-hat. That 3
FOV, when plotted on a logarithmic scale rather than linear, (see
Figure 8) shows that there is still light detected at the level of
at 5
and this means that the instrument will see a wider area than expected, and perhaps, therefore, see a less uniform patch. The instrument was not viewing a non-illuminated patch, even for a 5
FOV, but was seeing a less uniform patch, and this would particularly be the case for a 500 mm lamp-panel distance. In addition, scattering on imperfections in the lens will lead to light “lost” from the central field of view and “regained” from the outer parts of the source. With a smaller source, less energy is regained and this can reduce the measured signal. This is known as the “size-of-source effect” [
20,
21].
Thus, the smaller size of the source leads to a reduced measured signal because of the size-of-source effect and a 500 mm lamp-diffuser distance leads to a smaller signal because of the reduced uniformity of the source across the FOV (the source is less bright away from the centre of the filament). When both effects are present, the measured signal is lower and therefore, from Equation (
1), the radiance calibration coefficient is higher, as seen in the comparison results. To confirm this hypothesis, we also analysed the data for all participants considering the sensitivity to distance. This effect is not as strong at longer distances, as can be seen in
Figure 7, series E, that has a smaller patch size but did not show a positive bias. Thus, here the effect of the size of the source is compensated by a negative bias introduced by the distance setting. We analysed the data of all participants according to their sensitivity to distance. The results of this analysis are presented in
Figure 9.
The four data series represent the averaged comparison results for different distances of 500 mm, 750 mm, 1000 mm and 1300 mm, where the 500 mm distance is set as a reference, thus the percentage difference for the 500 mm series is equal to zero. Please note that 750 mm and 1300 mm had one entry to the comparison thus these are not averaged. The 500 mm series contains only the results from the main group and was set as the reference distance for that exercise, thus this data series has 0% difference. A negative bias can be observed with the distance increase for the 1000 mm and 1300 mm distance. For the 750 mm this is not so obvious, however this might be due to the fact that this particular participant has the radiance calibration values provided in radiance units for the whole system, rather than calculated from a lamp irradiance calibrated at 500 nm and a reflectance factor.
The protocol for the comparison encouraged participants to add an uncertainty component to account for set-ups where the source-diffuser distance is different from the 500 mm distance at which the lamps were originally calibrated for spectral irradiance. However, participants did not include an explicit correction for a different lamp-diffuser distance. It is possible that some of the observed differences between the main group and the other group are due to a bias from the filament offset. We would recommend in the future for the filament offset to be evaluated and corrected for [
22]. This may also help to reduce the spread within the main group.
5. Conclusions
We presented here the results of the international irradiance and radiance sources comparisons that were run as a part of the ESA FRM4SOC study activities. Ten international organisations participated in that exercise.
The irradiance comparison was run at NPL, where all participating lamps were measured against NPL standards. The results of that comparison were reported as the difference between the spectral irradiance values measured by each participant and the mean spectral irradiance values measured by all participants to show the degree to which the individual measurements agree with one another, without introducing a bias toward NPL scale. The irradiance comparison values showed good agreements between all lamps.
The radiance comparison took a form of a round robin where two radiometers were sent in turn to each participant to obtain radiance calibration coefficients for the radiometer using their in-house facilities. The results were analysed to compare the calibration coefficients of each participant to that of the mean value of all participants. The results showed the discrepancies between the participants at the level of ±4% and two separate groups with the measurements agreement (see
Figure 5 and
Figure 6). Additional investigation showed that the reason for this difference was caused by a sensitivity to the size of the illuminated patch (instrument size-of-source effect) and partly because the instrument-effective FOV brought in non-uniform parts of the illumination for a shorter lamp-diffuser distance which affected the results of the smaller group. If these effects could be corrected for or the measurements repeated at different settings we would expect to see all measurements agreeing within ±2.5%, as this is the level of agreement in the results from the majority group. These results were obtained with a modified Satlantic OCR-200 that had a reduced FOV. The sensitivity to size-of-source would be expected to be larger for an unmodified instrument with a larger FOV.
The secondary objective of the comparison exercise was to increase the community awareness of measurement uncertainty evaluation using the GUM methodology. This was achieved via the training course that was provided for the participants being present at NPL during the irradiance comparison exercise week. The participants were given instruction on how to derive the uncertainty components related to their radiance measurements in-house and all the round robin radiance results were reported accompanied by the uncertainty budgets. The results have shown a discrepancy that is larger than the declared uncertainty. In part, this is due to uncorrected sensitivities (e.g., to lamp-panel distance and illuminated-source size) and in part due to the fact that some uncertainty components were not fully investigated. Had the measurements all been made at the same distance, the comparison may not have shown up this sensitivity to the size of the source, and source uniformity. The experience of this comparison provides an opportunity for all participants to improve their thinking of uncertainty analysis and shows the value of a systematic review of all effects during the development of an uncertainty budget.
Space Agencies continue to evolve the OCR constellation. For the first time, a fleet of European Copernicus Earth Observation satellites is now sustained in an operational manner. In the next few years two new Sentinel-2 and two new Sentinel-3 satellites will be launched and commissioned. As Copernicus evolves, the tools, methodology and collaboration developed by FRM4SOC will be applied and further refined to ensure that the best possible OCR measurements are available from space to the user community.