We develop a general framework for data-driven approximation of input-output maps between infinite-dimensional spaces. The proposed approach is motivated by the recent successes of neural networks and deep learning, in combination with ideas from model reduction. This combination results in a neural network approximation which, in principle, is defined on infinite-dimensional spaces and, in practice, is robust to the dimension of finite-dimensional approximations of these spaces required for computation. For a class of input-output maps, and suitably chosen probability measures on the inputs, we prove convergence of the proposed approximation methodology. We also include numerical experiments which demonstrate the effectiveness of the method, showing convergence and robustness of the approximation scheme with respect to the size of the discretization, and compare it with existing algorithms from the literature; our examples include the mapping from coefficient to solution in a divergence form elliptic partial differential equation (PDE) problem, and the solution operator for viscous Burgers’ equation.

Classification: 65N75, 62M45, 68T05, 60H30, 60H15

Keywords: approximation theory, deep learning, model reduction, neural networks, partial differential equations.

@article{SMAI-JCM_2021__7__121_0, author = {Kaushik Bhattacharya and Bamdad Hosseini and Nikola B. Kovachki and Andrew M. Stuart}, title = {Model {Reduction} {And} {Neural} {Networks} {For} {Parametric} {PDEs}}, journal = {The SMAI journal of computational mathematics}, pages = {121--157}, publisher = {Soci\'et\'e de Math\'ematiques Appliqu\'ees et Industrielles}, volume = {7}, year = {2021}, doi = {10.5802/smai-jcm.74}, language = {en}, url = {https://smai-jcm.centre-mersenne.org/articles/10.5802/smai-jcm.74/} }

Kaushik Bhattacharya; Bamdad Hosseini; Nikola B. Kovachki; Andrew M. Stuart. Model Reduction And Neural Networks For Parametric PDEs. The SMAI journal of computational mathematics, Volume 7 (2021) , pp. 121-157. doi : 10.5802/smai-jcm.74. https://smai-jcm.centre-mersenne.org/articles/10.5802/smai-jcm.74/

[1] Solving ill-posed inverse problems using iterative deep neural networks, Inverse Probl., Volume 33 (2017) no. 12, 124007 | Article | MR 3729789 | Zbl 1394.92070

[2] Automatic choice of global shape functions in structural analysis, AIAA J., Volume 16 (1978) no. 5, pp. 525-528 | Article

[3] An ‘empirical interpolation’ method: application to efficient reduced-basis discretization of partial differential equations, C. R. Math. Acad. Sci. Paris, Volume 339 (2004) no. 9, pp. 667-672 | Article | MR 2103208 | Zbl 1061.65118

[4] Gaussian Measures on Function Spaces, Am. J. Math., Volume 98 (1976) no. 4, pp. 891-952 | Article | MR 467809 | Zbl 0384.28011

[5] Laplacian Eigenmaps for Dimensionality Reduction and Data Representation, Neural Computation, Volume 15 (2003) no. 6, pp. 1373-1396 | Article | Zbl 1085.68119

[6] Operator inference for non-intrusive model reduction of systems with non-polynomial nonlinear terms, Comput. Methods Appl. Mech. Eng., Volume 372 (2020), p. 113433 | Article | MR 4159629 | Zbl 07337918

[7] Prediction of aerodynamic flow fields using convolutional neural networks, Comput. Mech. (2019), pp. 1-21 | Article | MR 3977168 | Zbl 07095679

[8] Data assimilation in reduced modeling, SIAM/ASA J. Uncertain. Quantif., Volume 5 (2017) no. 1, pp. 1-29 | Article | MR 3590658 | Zbl 06736493

[9] Statistical properties of kernel principal component analysis, Machine Learning, Volume 66 (2007) no. 2, pp. 259-294 | Article | Zbl 1078.68133

[10] Reduced basis techniques for stochastic problems, Arch. Comput. Methods Eng., Volume 17 (2010) no. 4, pp. 435-454 | Article | MR 2739947 | Zbl 1269.65005

[11] DeepM&Mnet: Inferring the electroconvection multiphysics fields based on operator approximation by neural networks (2020) (https://arxiv.org/abs/2009.12935)

[12] Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems, IEEE Transactions on Neural Networks, Volume 6 (1995) no. 4, pp. 911-917 | Article

[13] Regression Clustering for Improved Accuracy and Training Costs with Molecular-Orbital-Based Machine Learning, Journal of Chemical Theory and Computation, Volume 15 (2019) no. 12, pp. 6668-6677 | Article

[14] Sparse adaptive Taylor approximation algorithms for parametric and stochastic elliptic PDEs, ESAIM, Math. Model. Numer. Anal., Volume 47 (2013) no. 1, pp. 253-280 | Article | Numdam | MR 3800593 | Zbl 1273.65009

[15] State Estimation–The Role of Reduced Models (2020) (https://arxiv.org/abs/2002.00220)

[16] Approximation of high-dimensional parametric PDEs, Acta Numer., Volume 24 (2015), pp. 1-159 | Article | MR 3349307 | Zbl 1320.65016

[17] Convergence Rates of Best N-term Galerkin Approximations for a Class of Elliptic SPDEs, Found. Comput. Math., Volume 10 (2010) no. 6, pp. 615-646 | Article | MR 2728424 | Zbl 1206.60064

[18] Analytic regularity and polynomial approximation of parametric and stochastic elliptic PDEs, Anal. Appl., Singap., Volume 09 (2011) no. 01, pp. 11-47 | Article | MR 2763359

[19] Geometric diffusions as a tool for harmonic analysis and structure definition of data: Diffusion maps, Proceedings of the National Academy of Sciences, Volume 102 (2005) no. 21, pp. 7426-7431 | Article | Zbl 1405.42043

[20] Besov priors for Bayesian inverse problems, Inverse Probl. Imaging, Volume 6 (2012), pp. 183-200 | Article | MR 2942737 | Zbl 1243.62032

[21] Nonlinear Approximation and (Deep) ReLU Networks (2019) (https://arxiv.org/abs/1905.02199)

[22] Nonlinear approximation, Acta Numer., Volume 7 (1998), pp. 51-150 | Article | MR 1689432

[23] The Theoretical Foundation of Reduced Basis Methods, Model Reduction and Approximation, Society for Industrial and Applied Mathematics, 2014 | Article

[24] A Discussion on Solving Partial Differential Equations using Neural Networks (2019) (https://arxiv.org/abs/1904.07200)

[25] The Deep Ritz Method: A Deep Learning-Based Numerical Algorithm for Solving Variational Problems, Communications in Mathematics and Statistics (2018) | Article | MR 3767958 | Zbl 1392.35306

[26] Partial differential equations, American Mathematical Society, 2010

[27] On a Theorem of Weyl Concerning Eigenvalues of Linear Transformations I, Proceedings of the National Academy of Sciences, Volume 35 (1949) no. 11, pp. 652-655 | Article | MR 34519

[28] A comprehensive deep learning-based approach to reduced order modeling of nonlinear time-dependent parametrized PDEs (2020) (https://arxiv.org/abs/2001.04001) | Zbl 07347011

[29] Numerical solution of the parametric diffusion equation by deep neural networks (2020) (https://arxiv.org/abs/2004.12131)

[30] Neural message passing for quantum chemistry, Proceedings of the 34th International Conference on Machine Learning (2017) (http://proceedings.mlr.press/v70/gilmer17a.html)

[31] Deep convolutional recurrent autoencoders for learning low-dimensional feature dynamics of fluid systems (2018) (https://arxiv.org/abs/1808.01346)

[32] Identification of distributed parameter systems: A neural net based approach, Computers & Chemical Engineering, Volume 22 (1998), p. S965-S968 | Article

[33] Deep Learning, MIT Press, 2016 http://www.deeplearningbook.org | Zbl 1373.68009

[34] Stable architectures for deep neural networks, Inverse Probl., Volume 34 (2017) no. 1, p. 014004 | Article | MR 3742361 | Zbl 1426.68236

[35] Deep ReLU Neural Network Expression Rates for Data-to-QoI Maps in Bayesian PDE Inversion (2020) (https://www.sam.math.ethz.ch/sam_reports/reports_final/reports2020/2020-02.pdf)

[36] Certified reduced basis methods for parametrized partial differential equations, SpringerBriefs in Mathematics, Springer, 2016 | Article | Zbl 1329.65203

[37] Non-intrusive reduced order modeling of nonlinear problems using neural networks, J. Comput. Phys., Volume 363 (2018), pp. 55-78 | Article | MR 3784416 | Zbl 1398.65330

[38] Reducing the Dimensionality of Data with Neural Networks, Science, Volume 313 (2006) no. 5786, pp. 504-507 | Article | MR 2242509 | Zbl 1226.68083

[39] Field Inversion and Machine Learning With Embedded Neural Networks: Physics-Consistent Neural Network Training, AIAA Aviation 2019 Forum (2019), 3200 pages | Article

[40] Learning Neural PDE Solvers with Convergence Guarantees, International Conference on Learning Representations (2019) (https://openreview.net/forum?id=rklawn0qk7)

[41] Well-posed Bayesian geometric inverse problems arising in subsurface flow, Inverse Probl., Volume 30 (2014), p. 114001 | Article | MR 3274585 | Zbl 1304.35767

[42] Solving parametric PDE problems with artificial neural networks (2017) (https://arxiv.org/abs/1707.03351)

[43] Self-Normalizing Neural Networks, Advances in Neural Information Processing Systems 30 (I. Guyon; U. V. Luxburg; S. Bengio; H. Wallach; R. Fergus; S. Vishwanathan; R. Garnett, eds.), Curran Associates, 2017, pp. 971-980

[44] Model identification of a spatiotemporally varying catalytic reaction, AIChE J., Volume 39 (1993) no. 1, pp. 89-98 | Article

[45] A theoretical analysis of deep neural networks and parametric PDEs (2019) (https://arxiv.org/abs/1904.00377)

[46] Efficient Approximation of Solutions of Parametric Linear Transport Equations by ReLU DNNs (2020) (https://arxiv.org/abs/2001.11441) | Zbl 07349297

[47] Artificial neural networks for solving ordinary and partial differential equations, IEEE Transactions on Neural Networks, Volume 9 (1998) no. 5, pp. 987-1000 | Article

[48] Data Assimilation: A Mathematical Introduction, Texts in Applied Mathematics, 62, Springer, 2015 | Article | Zbl 1353.60002

[49] Deep learning, Nature, Volume 521 (2015) no. 7553, pp. 436-444 | Article

[50] Model reduction of dynamical systems on nonlinear manifolds using deep convolutional autoencoders, J. Comput. Phys., Volume 404 (2020) | MR 4043884 | Zbl 1454.65184

[51] Neural Operator: Graph Kernel Networkfor Partial Differential Equations (2020) (https://arxiv.org/abs/2003.03485)

[52] Operator learning for predicting multiscale bubble growth dynamics (2020) (https://arxiv.org/abs/2012.12816)

[53] An introduction to computational stochastic PDEs, 50, Cambridge University Press, 2014 | Article | Zbl 1327.60011

[54] DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators (2019) (https://arxiv.org/abs/1910.03193)

[55] Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators, Nature Machine Intelligence (2020)

[56] A parameterized-background data-weak approach to variational data assimilation: formulation, analysis, and application to acoustics, Int. J. Numer. Meth. Engng., Volume 102 (2015) no. 5, pp. 933-965 | Article | MR 3341243 | Zbl 1352.65529

[57] Lower Bounds for Approximation by MLP Neural Networks, Neurocomputing, Volume 25 (1999), pp. 81-91 | Article | Zbl 0931.68093

[58] DeepM&Mnet for hypersonics: Predicting the coupled flow and finite-rate chemistry behind a normal shock using neural-network approximation of operators (2020) (https://arxiv.org/abs/2011.03349)

[59] Data-driven reduced-order models via regularized operator inference for a single-injector combustion process (2020) (https://arxiv.org/abs/2008.02862)

[60] Machine Learning: A Probabilistic Perspective, The MIT Press, 2012 https://www.cs.ubc.ca/~murphyk/mlbook/ | Zbl 1295.68003

[61] Modal representation of geometrically nonlinear behavior by the finite element method, Computers & Structures, Volume 10 (1979) no. 4, pp. 683-688 | Article | Zbl 0406.73071

[62] On the Sum of the Largest Eigenvalues of a Symmetric Matrix, SIAM Journal of Matrix Analysis and Applications, Volume 13 (1992) no. 1, pp. 41-45 | Article | MR 1146651 | Zbl 0747.15005

[63] LIII. On lines and planes of closest fit to systems of points in space, The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, Volume 2 (1901) no. 11, pp. 559-572 | Article | Zbl 32.0246.07

[64] Sampling low-dimensional Markovian dynamics for pre-asymptotically recovering reduced models from data with operator inference (2019) (https://arxiv.org/abs/1908.11233)

[65] Data-driven operator inference for nonintrusive projection-based model reduction, Comput. Methods Appl. Mech. Eng., Volume 306 (2016), pp. 196-215 | Article | MR 3502565 | Zbl 1436.93062

[66] Lift & Learn: Physics-informed machine learning for large-scale nonlinear dynamical systems, Physica D: Nonlinear Phenomena, Volume 406 (2020), p. 132401 | Article | MR 4073541

[67] Reduced basis methods for partial differential equations: an introduction, Springer, 2015 | Article

[68] Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations, J. Comput. Phys., Volume 378 (2019), pp. 686-707 | Article | MR 3881695 | Zbl 1415.68175

[69] Probabilistic forecasting and Bayesian data assimilation, Cambridge University Press, 2015 | Article | Zbl 1314.62005

[70] Deep Neural Networks Motivated by Partial Differential Equations, J. Math. Imaging Vis., Volume 62 (2019), pp. 352-364 | Article | MR 4082375 | Zbl 1434.68522

[71] Nonlinear Component Analysis as a Kernel Eigenvalue Problem, Neural Computation, Volume 10 (1998) no. 5, pp. 1299-1319 | Article

[72] Deep learning in high dimension: Neural network expression rates for generalized polynomial chaos expansions in UQ, Anal. Appl., Singap., Volume 17 (2019) no. 01, pp. 19-55 | Article | MR 3894732 | Zbl 07007693

[73] On the eigenspectrum of the Gram matrix and its relationship to the operator eigenspectrum, International Conference on Algorithmic Learning Theory, Springer, 2002, pp. 23-40 | Article | Zbl 1024.68538

[74] On the eigenspectrum of the Gram matrix and the generalization error of kernel-PCA, IEEE Transactions on Information Theory, Volume 51 (2005) no. 7, pp. 2510-2522 | Article | MR 2246374 | Zbl 1310.15076

[75] On the convergence and generalization of physics informed neural networks (2020) (https://arxiv.org/abs/2004.01806)

[76] EikoNet: Solving the Eikonal equation with Deep Neural Networks (2020) (https://arxiv.org/abs/2004.00361)

[77] Refined generalizations of the triangle inequality on Banach spaces, Math. Inequal. Appl., Volume 13 (2010) no. 4, pp. 733-741 | Article | MR 2760496 | Zbl 1205.26034

[78] Infinite-dimensional dynamical systems in mechanics and physics, 68, Springer, 2012

[79] Non-intrusive reduced order modeling of unsteady flows using artificial neural networks with application to a combustion problem, J. Comput. Phys., Volume 384 (2019), pp. 289-307 | Article | MR 3920924 | Zbl 1459.76117

[80] A proposal on machine learning via dynamical systems, Communications in Mathematics and Statistics, Volume 5 (2017) no. 1, pp. 1-11 | MR 3627592 | Zbl 1380.37154

[81] Error bounds for approximations with deep ReLU networks, Neural Netw., Volume 94 (2017), pp. 103-114 | Article | Zbl 1429.68260

[82] Applied Functional Analysis: Applications to Mathematical Physics, Springer, 2012

[83] Bayesian Deep Convolutional Encoder-Decoder Networks for Surrogate Modeling and Uncertainty Quantification, J. Comput. Phys., Volume 366 (2018) no. C, pp. 415-447 | Article | MR 3800689 | Zbl 1407.62091