We develop a general framework for data-driven approximation of input-output maps between infinite-dimensional spaces. The proposed approach is motivated by the recent successes of neural networks and deep learning, in combination with ideas from model reduction. This combination results in a neural network approximation which, in principle, is defined on infinite-dimensional spaces and, in practice, is robust to the dimension of finite-dimensional approximations of these spaces required for computation. For a class of input-output maps, and suitably chosen probability measures on the inputs, we prove convergence of the proposed approximation methodology. We also include numerical experiments which demonstrate the effectiveness of the method, showing convergence and robustness of the approximation scheme with respect to the size of the discretization, and compare it with existing algorithms from the literature; our examples include the mapping from coefficient to solution in a divergence form elliptic partial differential equation (PDE) problem, and the solution operator for viscous Burgers’ equation.
Keywords: approximation theory, deep learning, model reduction, neural networks, partial differential equations.
CC-BY-NC-ND 4.0
@article{SMAI-JCM_2021__7__121_0,
author = {Kaushik Bhattacharya and Bamdad Hosseini and Nikola B. Kovachki and Andrew M. Stuart},
title = {Model {Reduction} {And} {Neural} {Networks} {For} {Parametric} {PDEs}},
journal = {The SMAI Journal of computational mathematics},
pages = {121--157},
publisher = {Soci\'et\'e de Math\'ematiques Appliqu\'ees et Industrielles},
volume = {7},
year = {2021},
doi = {10.5802/smai-jcm.74},
language = {en},
url = {https://smai-jcm.centre-mersenne.org/articles/10.5802/smai-jcm.74/}
}
TY - JOUR AU - Kaushik Bhattacharya AU - Bamdad Hosseini AU - Nikola B. Kovachki AU - Andrew M. Stuart TI - Model Reduction And Neural Networks For Parametric PDEs JO - The SMAI Journal of computational mathematics PY - 2021 SP - 121 EP - 157 VL - 7 PB - Société de Mathématiques Appliquées et Industrielles UR - https://smai-jcm.centre-mersenne.org/articles/10.5802/smai-jcm.74/ DO - 10.5802/smai-jcm.74 LA - en ID - SMAI-JCM_2021__7__121_0 ER -
%0 Journal Article %A Kaushik Bhattacharya %A Bamdad Hosseini %A Nikola B. Kovachki %A Andrew M. Stuart %T Model Reduction And Neural Networks For Parametric PDEs %J The SMAI Journal of computational mathematics %D 2021 %P 121-157 %V 7 %I Société de Mathématiques Appliquées et Industrielles %U https://smai-jcm.centre-mersenne.org/articles/10.5802/smai-jcm.74/ %R 10.5802/smai-jcm.74 %G en %F SMAI-JCM_2021__7__121_0
Kaushik Bhattacharya; Bamdad Hosseini; Nikola B. Kovachki; Andrew M. Stuart. Model Reduction And Neural Networks For Parametric PDEs. The SMAI Journal of computational mathematics, Volume 7 (2021), pp. 121-157. doi : 10.5802/smai-jcm.74. https://smai-jcm.centre-mersenne.org/articles/10.5802/smai-jcm.74/
[1] Solving ill-posed inverse problems using iterative deep neural networks, Inverse Probl., Volume 33 (2017) no. 12, 124007 | DOI | MR | Zbl
[2] Automatic choice of global shape functions in structural analysis, AIAA J., Volume 16 (1978) no. 5, pp. 525-528 | DOI
[3] An ‘empirical interpolation’ method: application to efficient reduced-basis discretization of partial differential equations, C. R. Math. Acad. Sci. Paris, Volume 339 (2004) no. 9, pp. 667-672 | DOI | MR | Zbl
[4] Gaussian Measures on Function Spaces, Am. J. Math., Volume 98 (1976) no. 4, pp. 891-952 | DOI | MR | Zbl
[5] Laplacian Eigenmaps for Dimensionality Reduction and Data Representation, Neural Computation, Volume 15 (2003) no. 6, pp. 1373-1396 | DOI | Zbl
[6] Operator inference for non-intrusive model reduction of systems with non-polynomial nonlinear terms, Comput. Methods Appl. Mech. Eng., Volume 372 (2020), p. 113433 | DOI | MR | Zbl
[7] Prediction of aerodynamic flow fields using convolutional neural networks, Comput. Mech. (2019), pp. 1-21 | DOI | MR | Zbl
[8] Data assimilation in reduced modeling, SIAM/ASA J. Uncertain. Quantif., Volume 5 (2017) no. 1, pp. 1-29 | DOI | MR | Zbl
[9] Statistical properties of kernel principal component analysis, Machine Learning, Volume 66 (2007) no. 2, pp. 259-294 | DOI | Zbl
[10] Reduced basis techniques for stochastic problems, Arch. Comput. Methods Eng., Volume 17 (2010) no. 4, pp. 435-454 | DOI | MR | Zbl
[11] DeepM&Mnet: Inferring the electroconvection multiphysics fields based on operator approximation by neural networks (2020) (https://arxiv.org/abs/2009.12935)
[12] Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems, IEEE Transactions on Neural Networks, Volume 6 (1995) no. 4, pp. 911-917 | DOI
[13] Regression Clustering for Improved Accuracy and Training Costs with Molecular-Orbital-Based Machine Learning, Journal of Chemical Theory and Computation, Volume 15 (2019) no. 12, pp. 6668-6677 | DOI
[14] Sparse adaptive Taylor approximation algorithms for parametric and stochastic elliptic PDEs, ESAIM, Math. Model. Numer. Anal., Volume 47 (2013) no. 1, pp. 253-280 | DOI | Numdam | MR | Zbl
[15] State Estimation–The Role of Reduced Models (2020) (https://arxiv.org/abs/2002.00220)
[16] Approximation of high-dimensional parametric PDEs, Acta Numer., Volume 24 (2015), pp. 1-159 | DOI | MR | Zbl
[17] Convergence Rates of Best N-term Galerkin Approximations for a Class of Elliptic SPDEs, Found. Comput. Math., Volume 10 (2010) no. 6, pp. 615-646 | DOI | MR | Zbl
[18] Analytic regularity and polynomial approximation of parametric and stochastic elliptic PDEs, Anal. Appl., Singap., Volume 09 (2011) no. 01, pp. 11-47 | DOI | MR
[19] Geometric diffusions as a tool for harmonic analysis and structure definition of data: Diffusion maps, Proceedings of the National Academy of Sciences, Volume 102 (2005) no. 21, pp. 7426-7431 | DOI | Zbl
[20] Besov priors for Bayesian inverse problems, Inverse Probl. Imaging, Volume 6 (2012), pp. 183-200 | DOI | MR | Zbl
[21] Nonlinear Approximation and (Deep) ReLU Networks (2019) (https://arxiv.org/abs/1905.02199)
[22] Nonlinear approximation, Acta Numer., Volume 7 (1998), pp. 51-150 | DOI | MR
[23] The Theoretical Foundation of Reduced Basis Methods, Model Reduction and Approximation, Society for Industrial and Applied Mathematics, 2014 | DOI
[24] A Discussion on Solving Partial Differential Equations using Neural Networks (2019) (https://arxiv.org/abs/1904.07200)
[25] The Deep Ritz Method: A Deep Learning-Based Numerical Algorithm for Solving Variational Problems, Communications in Mathematics and Statistics (2018) | DOI | MR | Zbl
[26] Partial differential equations, American Mathematical Society, 2010
[27] On a Theorem of Weyl Concerning Eigenvalues of Linear Transformations I, Proceedings of the National Academy of Sciences, Volume 35 (1949) no. 11, pp. 652-655 | DOI | MR
[28] A comprehensive deep learning-based approach to reduced order modeling of nonlinear time-dependent parametrized PDEs (2020) (https://arxiv.org/abs/2001.04001) | Zbl
[29] Numerical solution of the parametric diffusion equation by deep neural networks (2020) (https://arxiv.org/abs/2004.12131)
[30] , Proceedings of the 34th International Conference on Machine Learning (2017) (http://proceedings.mlr.press/v70/gilmer17a.html)
[31] Deep convolutional recurrent autoencoders for learning low-dimensional feature dynamics of fluid systems (2018) (https://arxiv.org/abs/1808.01346)
[32] Identification of distributed parameter systems: A neural net based approach, Computers & Chemical Engineering, Volume 22 (1998), p. S965-S968 | DOI
[33] Deep Learning, MIT Press, 2016 http://www.deeplearningbook.org | Zbl
[34] Stable architectures for deep neural networks, Inverse Probl., Volume 34 (2017) no. 1, p. 014004 | DOI | MR | Zbl
[35] Deep ReLU Neural Network Expression Rates for Data-to-QoI Maps in Bayesian PDE Inversion (2020) (https://www.sam.math.ethz.ch/sam_reports/reports_final/reports2020/2020-02.pdf)
[36] et al. Certified reduced basis methods for parametrized partial differential equations, SpringerBriefs in Mathematics, Springer, 2016 | DOI | Zbl
[37] Non-intrusive reduced order modeling of nonlinear problems using neural networks, J. Comput. Phys., Volume 363 (2018), pp. 55-78 | DOI | MR | Zbl
[38] Reducing the Dimensionality of Data with Neural Networks, Science, Volume 313 (2006) no. 5786, pp. 504-507 | DOI | MR | Zbl
[39] , AIAA Aviation 2019 Forum (2019), 3200 pages | DOI
[40] , International Conference on Learning Representations (2019) (https://openreview.net/forum?id=rklawn0qk7)
[41] Well-posed Bayesian geometric inverse problems arising in subsurface flow, Inverse Probl., Volume 30 (2014), p. 114001 | DOI | MR | Zbl
[42] Solving parametric PDE problems with artificial neural networks (2017) (https://arxiv.org/abs/1707.03351)
[43] Self-Normalizing Neural Networks, Advances in Neural Information Processing Systems 30 (I. Guyon; U. V. Luxburg; S. Bengio; H. Wallach; R. Fergus; S. Vishwanathan; R. Garnett, eds.), Curran Associates, 2017, pp. 971-980
[44] Model identification of a spatiotemporally varying catalytic reaction, AIChE J., Volume 39 (1993) no. 1, pp. 89-98 | DOI
[45] A theoretical analysis of deep neural networks and parametric PDEs (2019) (https://arxiv.org/abs/1904.00377)
[46] Efficient Approximation of Solutions of Parametric Linear Transport Equations by ReLU DNNs (2020) (https://arxiv.org/abs/2001.11441) | Zbl
[47] Artificial neural networks for solving ordinary and partial differential equations, IEEE Transactions on Neural Networks, Volume 9 (1998) no. 5, pp. 987-1000 | DOI
[48] Data Assimilation: A Mathematical Introduction, Texts in Applied Mathematics, 62, Springer, 2015 | DOI | Zbl
[49] Deep learning, Nature, Volume 521 (2015) no. 7553, pp. 436-444 | DOI
[50] Model reduction of dynamical systems on nonlinear manifolds using deep convolutional autoencoders, J. Comput. Phys., Volume 404 (2020) | MR | Zbl
[51] Neural Operator: Graph Kernel Networkfor Partial Differential Equations (2020) (https://arxiv.org/abs/2003.03485)
[52] Operator learning for predicting multiscale bubble growth dynamics (2020) (https://arxiv.org/abs/2012.12816)
[53] An introduction to computational stochastic PDEs, 50, Cambridge University Press, 2014 | DOI | Zbl
[54] DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators (2019) (https://arxiv.org/abs/1910.03193)
[55] Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators, Nature Machine Intelligence (2020)
[56] A parameterized-background data-weak approach to variational data assimilation: formulation, analysis, and application to acoustics, Int. J. Numer. Meth. Engng., Volume 102 (2015) no. 5, pp. 933-965 | DOI | MR | Zbl
[57] Lower Bounds for Approximation by MLP Neural Networks, Neurocomputing, Volume 25 (1999), pp. 81-91 | DOI | Zbl
[58] DeepM&Mnet for hypersonics: Predicting the coupled flow and finite-rate chemistry behind a normal shock using neural-network approximation of operators (2020) (https://arxiv.org/abs/2011.03349)
[59] Data-driven reduced-order models via regularized operator inference for a single-injector combustion process (2020) (https://arxiv.org/abs/2008.02862)
[60] Machine Learning: A Probabilistic Perspective, The MIT Press, 2012 https://www.cs.ubc.ca/~murphyk/mlbook/ | Zbl
[61] Modal representation of geometrically nonlinear behavior by the finite element method, Computers & Structures, Volume 10 (1979) no. 4, pp. 683-688 | DOI | Zbl
[62] On the Sum of the Largest Eigenvalues of a Symmetric Matrix, SIAM Journal of Matrix Analysis and Applications, Volume 13 (1992) no. 1, pp. 41-45 | DOI | MR | Zbl
[63] LIII. On lines and planes of closest fit to systems of points in space, The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, Volume 2 (1901) no. 11, pp. 559-572 | DOI | Zbl
[64] Sampling low-dimensional Markovian dynamics for pre-asymptotically recovering reduced models from data with operator inference (2019) (https://arxiv.org/abs/1908.11233)
[65] Data-driven operator inference for nonintrusive projection-based model reduction, Comput. Methods Appl. Mech. Eng., Volume 306 (2016), pp. 196-215 | DOI | MR | Zbl
[66] Lift & Learn: Physics-informed machine learning for large-scale nonlinear dynamical systems, Physica D: Nonlinear Phenomena, Volume 406 (2020), p. 132401 | DOI | MR
[67] Reduced basis methods for partial differential equations: an introduction, Springer, 2015 | DOI
[68] Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations, J. Comput. Phys., Volume 378 (2019), pp. 686-707 | DOI | MR | Zbl
[69] Probabilistic forecasting and Bayesian data assimilation, Cambridge University Press, 2015 | DOI | Zbl
[70] Deep Neural Networks Motivated by Partial Differential Equations, J. Math. Imaging Vis., Volume 62 (2019), pp. 352-364 | DOI | MR | Zbl
[71] Nonlinear Component Analysis as a Kernel Eigenvalue Problem, Neural Computation, Volume 10 (1998) no. 5, pp. 1299-1319 | DOI
[72] Deep learning in high dimension: Neural network expression rates for generalized polynomial chaos expansions in UQ, Anal. Appl., Singap., Volume 17 (2019) no. 01, pp. 19-55 | DOI | MR | Zbl
[73] On the eigenspectrum of the Gram matrix and its relationship to the operator eigenspectrum, International Conference on Algorithmic Learning Theory, Springer, 2002, pp. 23-40 | DOI | Zbl
[74] On the eigenspectrum of the Gram matrix and the generalization error of kernel-PCA, IEEE Transactions on Information Theory, Volume 51 (2005) no. 7, pp. 2510-2522 | DOI | MR | Zbl
[75] On the convergence and generalization of physics informed neural networks (2020) (https://arxiv.org/abs/2004.01806)
[76] EikoNet: Solving the Eikonal equation with Deep Neural Networks (2020) (https://arxiv.org/abs/2004.00361)
[77] Refined generalizations of the triangle inequality on Banach spaces, Math. Inequal. Appl., Volume 13 (2010) no. 4, pp. 733-741 | DOI | MR | Zbl
[78] Infinite-dimensional dynamical systems in mechanics and physics, 68, Springer, 2012
[79] Non-intrusive reduced order modeling of unsteady flows using artificial neural networks with application to a combustion problem, J. Comput. Phys., Volume 384 (2019), pp. 289-307 | DOI | MR | Zbl
[80] A proposal on machine learning via dynamical systems, Communications in Mathematics and Statistics, Volume 5 (2017) no. 1, pp. 1-11 | MR | Zbl
[81] Error bounds for approximations with deep ReLU networks, Neural Netw., Volume 94 (2017), pp. 103-114 | DOI | Zbl
[82] Applied Functional Analysis: Applications to Mathematical Physics, Springer, 2012
[83] Bayesian Deep Convolutional Encoder-Decoder Networks for Surrogate Modeling and Uncertainty Quantification, J. Comput. Phys., Volume 366 (2018) no. C, pp. 415-447 | DOI | MR | Zbl
Cited by Sources: