The objective of this project is to extend recent developments in compressive sensing, statistical machine
learning, and data assimilation tools to design data reduction methods uniquely capable for the emerging US
Department of Energy (DOE) infrastructure being built for interconnected science and analysis of computa-
tional, experimental, and observational data. A critical feature of scientific data, in contrast to other types
of data, is that (possibly unknown) physical principles lie at the foundation. Hence the reduction methods
in this proposal seek to optimize for known physical properties as well as discover the underlying character-
istics of unknown physical properties. Importantly, while DOE scientific research is constantly growing in
size and scale, there are efforts to evolve into a new paradigm that considers more connected, collaborative,
autonomous, and real-time environments – creating exciting opportunities and challenges for data reduction
which the proposed research aims to address.

As they relate to the proposed work, key data reduction challenges facing the DOE include: (1) the in-
corporation of known or discovered scientific information into data reduction; (2) progressive data reduction
with tight error bounds near the point of generation taking advantage of similarities across the interconnected
infrastructure to optimize information flow; (3) uncertainty quantification for data with noise, error, or miss-
ing elements; and (4) the ability to effectively use new computing architecture, both centralized and at the
edge, in order to accelerate analysis through computation on reduced data. We propose the following three
research thrusts for developing methods in compressed sensing (CS), statistical/machine learning (ML), and
continuous data assimilation (CDA) for the data reduction challenges at the DOE.

Thrust 1: CS Methods for Data Reduction on Distributed Data in Scientific Ecosystems. We have devel-
oped new CS frameworks, adapted to the challenges of scientific data reduction, that can preserve structure
and known properties (prior information). We will derive new CS methods that can optimize compression
on distributed data across the DOE complex, providing progressively rank information and tight error es-
timation, that can be used to accelerate end-point analysis, optimize network communication, and order
data storage, and prioritize information for streaming.

Thrust 2: Statistical/ML Reduction Methods for Scientific Data. This team has developed statistical
interpolation/generative models and homotopy methods that can reduce data size and dimension while
preserving the statistical properties of original data. We will develop progressive statistical data reduction
(SDR) for the DOE challenges of storage/ transmission and accelerate machine learning and analysis.

Thrust 3: CDA to Reduce Required Amount of Simulation Data. We have developed temporal/spatial
CDA methods for data reduction of streaming scientific simulations. We will develop new CDA methods
that will allow local users to interact and analyze high resolution leadership computing facilities (LCFs)
simulations given their limited data transfer and computing budget.

This team consists of experts in these three thrust areas from Clemson University, Dartmouth College, New
York University, and Oak Ridge National Laboratory. The diverse aspects of scientific data will require
different approaches for reduction, and this proposal is designed to develop a variety of different reduction
methods for this purpose. We explain in this proposal, unique connections between thrust, where the methods
developed in each thrust can complement each other. We will demonstrate our methods in this proposal, with
applications that span the domain of scientific data, on neutron and X-ray light facility data, climate science
observational and simulation data, and LCF fluid/gas/plasma simulation data.