WP_1500.11 – EMM-CNR-EMN11-TN-002

Complete Data Fusion

a posteriori fusion of athmospheric level 2 products

Introduction

Complete Data Fusion (CDF) is a statistical method designed to optimally combine multiple independent atmospheric profile retrievals into a single, consistent, and information-rich product. It is particularly suited for satellite remote sensing applications, where different instruments provide complementary measurements of the same geophysical quantity (e.g., ozone, temperature, trace gases).

Within the EMM (Earth Moon Mars) research infrastructure, the CDF enables advanced scientific studies based on real atmospheric data. EMM provides the necessary computational resources, specialized software tools, and expert knowledge to support the application of CDF to optimized and multi-instrument satellite observations.

Basic Concepts

Possible CDF usages

A priori armonization and vertical regridding

Atmospheric profiles derived with optimal estimation are characterized by a vertical grid, an a priori profile, and an a priori variance covariance matrix (VCM) that characterizes the strength of the constraint. Profiles defined on different grids and expecially with different a priori information are often not directly comparable. The CDF allows these three elements to be consistently redefined, and this feature can make products defined on different grids and/or with different a priori values comparable.

A priori bias-free smart averages

Each atmospheric profile retrieved with optimal estimation is characterized by a priori profile, and products close in time and space typically share the same priori profile, often derived from climatology. If many profiles that share the same priori profile are averaged, this results in a bias. The CDF merges several profiles after removing the priori information from each of them. This information is added back to the fused product once and for all at the end of the process, thus creating a bias-free smart average.

Sinergies

It has been shown that under certain conditions, CDF is equivalent to simultaneous retrieval. Being a purely algebraic method, CDF is much simpler to implement than simultaneous retrieval, and its application is much less computationally demanding. Following the demonstration of the compatibility of the datasets available to the scientific community with CDF, it is of great interest to study the various possible synergies that may involve both different observation geometries and different measurement spectral bands.

Il CNR svolge un ruolo attivo e trasversale, partecipando attraverso sei Istituti (IFAC, IBE, INO, IEIIT, IAC, ISAC) afferenti a quattro Dipartimenti (DIITET, DISBA, DSFTM, DSSTTA). Oltre alla condivisione delle attività di coordinamento tecnico-scientifico e gestionale, il CNR è direttamente responsabile di specifiche linee progettuali, strategiche per l’avanzamento delle tecnologie e infrastrutture a supporto dell’esplorazione del sistema Terra. In particolare, il suo contributo si concentra su:

Sviluppo di prototipi e studi di fattibilità per strumentazione destinata alla misura dell’atmosfera e della superficie terrestre dalla Luna;
Partecipazione alla realizzazione del sistema di antenna per la ricezione di segnali provenienti dallo spazio profondo;
Progettazione e implementazione di un’infrastruttura hardware e software in grado di coprire l’intero ciclo operativo: dalla raccolta ed elaborazione dei dati osservativi per calibrazione e validazione di misure satellitari, alla generazione di modelli di trasferimento radiativo (diretti e inversi), algoritmi di fusione e retrieval sinergico, fino all’assimilazione dati e alla modellistica meteorologica e climatica.

Diplomazia spaziale

promuovendo collaborazioni internazionali basate su scambi scientifici e tecnologici

Economia spaziale

offrendo servizi osservativi unici e abilitando nuove opportunità di utilizzo e sviluppo delle risorse spaziali

Requirements for a Dedicated Software Infrastructure

The characteristics of a hardware infrastructure dedicated to the CDF, largely depend on application types and on the type of data to which the CDF applies. At the present development stage, it is very important both to list a first set of basic considerations on the implementation of the fusion, and to underline which information it is important to know for each type of data, to be able to collect them during the testing phase of the CDF on new datasets.

The maximum number of products to which the CDF can be applied depends on the application context:
1. if the CDF is used to collect the information embedded in different measurements of the same portion of atmosphere by one or several instruments, the maximum number of products to be fused in one is of the order of tenth or hundreds depending on the spatial-temporal coincidence criteria;
2. if the CDF is used to perform large spatial/temporal averages free from the a priori bias (Zoppetti et al. 2021) the maximum number of products to be fused at once can be of the order of hundreds of thousands or millions.
The computational costs (computation times and memory occupation) of the application of the CDF depend linearly on the number of fused products and for each of them in a linear to quadratic way on the number of elements of the state vector.
The single fusion of 10-100 atmospheric products having state vector size of 40-50 elements can be performed in thousandths/hundredths of a second in a standard laptop with poor code optimization using interpreted Python3 code.
In general, the chance to use the CDF for product synergism depends on the spatial and temporal distribution of each product. It will be important to collect all the pertinent information about all the datasets that pass the completeness tests.