Model Reduction And Neural Networks For Parametric PDEs
The SMAI journal of computational mathematics, Volume 7 (2021) , pp. 121-157.

We develop a general framework for data-driven approximation of input-output maps between infinite-dimensional spaces. The proposed approach is motivated by the recent successes of neural networks and deep learning, in combination with ideas from model reduction. This combination results in a neural network approximation which, in principle, is defined on infinite-dimensional spaces and, in practice, is robust to the dimension of finite-dimensional approximations of these spaces required for computation. For a class of input-output maps, and suitably chosen probability measures on the inputs, we prove convergence of the proposed approximation methodology. We also include numerical experiments which demonstrate the effectiveness of the method, showing convergence and robustness of the approximation scheme with respect to the size of the discretization, and compare it with existing algorithms from the literature; our examples include the mapping from coefficient to solution in a divergence form elliptic partial differential equation (PDE) problem, and the solution operator for viscous Burgers’ equation.

Published online:
DOI: https://doi.org/10.5802/smai-jcm.74
Classification: 65N75,  62M45,  68T05,  60H30,  60H15
Keywords: approximation theory, deep learning, model reduction, neural networks, partial differential equations.
@article{SMAI-JCM_2021__7__121_0,
     author = {Kaushik Bhattacharya and Bamdad Hosseini and Nikola B. Kovachki and Andrew M. Stuart},
     title = {Model {Reduction} {And} {Neural} {Networks} {For} {Parametric} {PDEs}},
     journal = {The SMAI journal of computational mathematics},
     pages = {121--157},
     publisher = {Soci\'et\'e de Math\'ematiques Appliqu\'ees et Industrielles},
     volume = {7},
     year = {2021},
     doi = {10.5802/smai-jcm.74},
     language = {en},
     url = {https://smai-jcm.centre-mersenne.org/articles/10.5802/smai-jcm.74/}
}
Kaushik Bhattacharya; Bamdad Hosseini; Nikola B. Kovachki; Andrew M. Stuart. Model Reduction And Neural Networks For Parametric PDEs. The SMAI journal of computational mathematics, Volume 7 (2021) , pp. 121-157. doi : 10.5802/smai-jcm.74. https://smai-jcm.centre-mersenne.org/articles/10.5802/smai-jcm.74/

[1] J. Adler; O. Oktem Solving ill-posed inverse problems using iterative deep neural networks, Inverse Probl., Volume 33 (2017) no. 12, 124007 | Article | MR 3729789 | Zbl 1394.92070

[2] B. O. Almroth; P. Stern; F. A. Brogan Automatic choice of global shape functions in structural analysis, AIAA J., Volume 16 (1978) no. 5, pp. 525-528 | Article

[3] M. Barrault; Y. Maday; N. C. Nguyen; A. T. Patera An ‘empirical interpolation’ method: application to efficient reduced-basis discretization of partial differential equations, C. R. Math. Acad. Sci. Paris, Volume 339 (2004) no. 9, pp. 667-672 | Article | MR 2103208 | Zbl 1061.65118

[4] P. Baxendale Gaussian Measures on Function Spaces, Am. J. Math., Volume 98 (1976) no. 4, pp. 891-952 | Article | MR 467809 | Zbl 0384.28011

[5] M. Belkin; P. Niyogi Laplacian Eigenmaps for Dimensionality Reduction and Data Representation, Neural Computation, Volume 15 (2003) no. 6, pp. 1373-1396 | Article | Zbl 1085.68119

[6] P. Benner; P. Goyal; B. Kramer; B. Peherstorfer; K. Willcox Operator inference for non-intrusive model reduction of systems with non-polynomial nonlinear terms, Comput. Methods Appl. Mech. Eng., Volume 372 (2020), p. 113433 | Article | MR 4159629 | Zbl 07337918

[7] S. Bhatnagar; Y. Afshar; S. Pan; K. Duraisamy; S. Kaushik Prediction of aerodynamic flow fields using convolutional neural networks, Comput. Mech. (2019), pp. 1-21 | Article | MR 3977168 | Zbl 07095679

[8] P. Binev; A. Cohen; W. Dahmen; R. DeVore; G. Petrova; P. Wojtaszczyk Data assimilation in reduced modeling, SIAM/ASA J. Uncertain. Quantif., Volume 5 (2017) no. 1, pp. 1-29 | Article | MR 3590658 | Zbl 06736493

[9] G. Blanchard; O. Bousquet; L. Zwald Statistical properties of kernel principal component analysis, Machine Learning, Volume 66 (2007) no. 2, pp. 259-294 | Article | Zbl 1078.68133

[10] S. Boyaval; C. Le Bris; T. Lelievre; Y. Maday; N. C. Nguyen; A. T. Patera Reduced basis techniques for stochastic problems, Arch. Comput. Methods Eng., Volume 17 (2010) no. 4, pp. 435-454 | Article | MR 2739947 | Zbl 1269.65005

[11] S. Cai; Z. Wang; L. Lu; T. A Zaki; G. E. Karniadakis DeepM&Mnet: Inferring the electroconvection multiphysics fields based on operator approximation by neural networks (2020) (https://arxiv.org/abs/2009.12935)

[12] T. Chen; H. Chen Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems, IEEE Transactions on Neural Networks, Volume 6 (1995) no. 4, pp. 911-917 | Article

[13] L. Cheng; N. Kovachki; M. Welborn; T. F. Miller Regression Clustering for Improved Accuracy and Training Costs with Molecular-Orbital-Based Machine Learning, Journal of Chemical Theory and Computation, Volume 15 (2019) no. 12, pp. 6668-6677 | Article

[14] A. Chkifa; A. Cohen; R. DeVore; C. Schwab Sparse adaptive Taylor approximation algorithms for parametric and stochastic elliptic PDEs, ESAIM, Math. Model. Numer. Anal., Volume 47 (2013) no. 1, pp. 253-280 | Article | Numdam | MR 3800593 | Zbl 1273.65009

[15] A. Cohen; W. Dahmen; R. DeVore State Estimation–The Role of Reduced Models (2020) (https://arxiv.org/abs/2002.00220)

[16] A. Cohen; R. DeVore Approximation of high-dimensional parametric PDEs, Acta Numer., Volume 24 (2015), pp. 1-159 | Article | MR 3349307 | Zbl 1320.65016

[17] A. Cohen; R. DeVore; C. Schwab Convergence Rates of Best N-term Galerkin Approximations for a Class of Elliptic SPDEs, Found. Comput. Math., Volume 10 (2010) no. 6, pp. 615-646 | Article | MR 2728424 | Zbl 1206.60064

[18] A. Cohen; R. DeVore; C. Scwhab Analytic regularity and polynomial approximation of parametric and stochastic elliptic PDEs, Anal. Appl., Singap., Volume 09 (2011) no. 01, pp. 11-47 | Article | MR 2763359

[19] R. R. Coifman; S. Lafon; A. B. Lee; M. Maggioni; B. Nadler; F. Warner; S. W. Zucker Geometric diffusions as a tool for harmonic analysis and structure definition of data: Diffusion maps, Proceedings of the National Academy of Sciences, Volume 102 (2005) no. 21, pp. 7426-7431 | Article | Zbl 1405.42043

[20] M. Dashti; S. Harris; A. M. Stuart Besov priors for Bayesian inverse problems, Inverse Probl. Imaging, Volume 6 (2012), pp. 183-200 | Article | MR 2942737 | Zbl 1243.62032

[21] I. Daubechies; R. DeVore; S. Foucart; B. Hanin; G. Petrova Nonlinear Approximation and (Deep) ReLU Networks (2019) (https://arxiv.org/abs/1905.02199)

[22] R. DeVore Nonlinear approximation, Acta Numer., Volume 7 (1998), pp. 51-150 | Article | MR 1689432

[23] R. DeVore The Theoretical Foundation of Reduced Basis Methods, Model Reduction and Approximation, Society for Industrial and Applied Mathematics, 2014 | Article

[24] T. Dockhorn A Discussion on Solving Partial Differential Equations using Neural Networks (2019) (https://arxiv.org/abs/1904.07200)

[25] W. E; B. Yu The Deep Ritz Method: A Deep Learning-Based Numerical Algorithm for Solving Variational Problems, Communications in Mathematics and Statistics (2018) | Article | MR 3767958 | Zbl 1392.35306

[26] L. C. Evans Partial differential equations, American Mathematical Society, 2010

[27] K. Fan On a Theorem of Weyl Concerning Eigenvalues of Linear Transformations I, Proceedings of the National Academy of Sciences, Volume 35 (1949) no. 11, pp. 652-655 | Article | MR 34519

[28] S. Fresca; L. Dede; A. Manzoni A comprehensive deep learning-based approach to reduced order modeling of nonlinear time-dependent parametrized PDEs (2020) (https://arxiv.org/abs/2001.04001) | Zbl 07347011

[29] M. Geist; P. Petersen; M. Raslan; R. Schneider; G. Kutyniok Numerical solution of the parametric diffusion equation by deep neural networks (2020) (https://arxiv.org/abs/2004.12131)

[30] J. Gilmer; S. S Schoenholz; P. F. Riley; O. Vinyals; G. E. Dahl Neural message passing for quantum chemistry, Proceedings of the 34th International Conference on Machine Learning (2017) (http://proceedings.mlr.press/v70/gilmer17a.html)

[31] F. J. Gonzalez; M. Balajewicz Deep convolutional recurrent autoencoders for learning low-dimensional feature dynamics of fluid systems (2018) (https://arxiv.org/abs/1808.01346)

[32] R. Gonzalez-Garcia; R. Rico-Martínez; I. G. Kevrekidis Identification of distributed parameter systems: A neural net based approach, Computers & Chemical Engineering, Volume 22 (1998), p. S965-S968 | Article

[33] I. Goodfellow; Y. Bengio; A. Courville Deep Learning, MIT Press, 2016 http://www.deeplearningbook.org | Zbl 1373.68009

[34] E. Haber; L. Ruthotto Stable architectures for deep neural networks, Inverse Probl., Volume 34 (2017) no. 1, p. 014004 | Article | MR 3742361 | Zbl 1426.68236

[35] L. Herrmann; C. Schwab; J. Zech Deep ReLU Neural Network Expression Rates for Data-to-QoI Maps in Bayesian PDE Inversion (2020) (https://www.sam.math.ethz.ch/sam_reports/reports_final/reports2020/2020-02.pdf)

[36] J. S. Hesthaven; G. Rozza; B. Stamm; et al. Certified reduced basis methods for parametrized partial differential equations, SpringerBriefs in Mathematics, Springer, 2016 | Article | Zbl 1329.65203

[37] J. S. Hesthaven; S. Ubbiali Non-intrusive reduced order modeling of nonlinear problems using neural networks, J. Comput. Phys., Volume 363 (2018), pp. 55-78 | Article | MR 3784416 | Zbl 1398.65330

[38] G. Hinton; R. Salakhutdinov Reducing the Dimensionality of Data with Neural Networks, Science, Volume 313 (2006) no. 5786, pp. 504-507 | Article | MR 2242509 | Zbl 1226.68083

[39] J. R Holland; J. D Baeder; K. Duraisamy Field Inversion and Machine Learning With Embedded Neural Networks: Physics-Consistent Neural Network Training, AIAA Aviation 2019 Forum (2019), 3200 pages | Article

[40] J.-T. Hsieh; S. Zhao; S. Eismann; L. Mirabella; S. Ermon Learning Neural PDE Solvers with Convergence Guarantees, International Conference on Learning Representations (2019) (https://openreview.net/forum?id=rklawn0qk7)

[41] M. A. Iglesias; K. Lin; A. M. Stuart Well-posed Bayesian geometric inverse problems arising in subsurface flow, Inverse Probl., Volume 30 (2014), p. 114001 | Article | MR 3274585 | Zbl 1304.35767

[42] Y. Khoo; J. Lu; L. Ying Solving parametric PDE problems with artificial neural networks (2017) (https://arxiv.org/abs/1707.03351)

[43] G. Klambauer; T. Unterthiner; A. Mayr; S. Hochreiter Self-Normalizing Neural Networks, Advances in Neural Information Processing Systems 30 (I. Guyon; U. V. Luxburg; S. Bengio; H. Wallach; R. Fergus; S. Vishwanathan; R. Garnett, eds.), Curran Associates, 2017, pp. 971-980

[44] K. Krischer; R. Rico-Martínez; I. G. Kevrekidis; H. H. Rotermund; G. Ertl; J. L. Hudson Model identification of a spatiotemporally varying catalytic reaction, AIChE J., Volume 39 (1993) no. 1, pp. 89-98 | Article

[45] G. Kutyniok; P. Petersen; M. Raslan; R. Schneider A theoretical analysis of deep neural networks and parametric PDEs (2019) (https://arxiv.org/abs/1904.00377)

[46] F. Laakmann; P. Petersen Efficient Approximation of Solutions of Parametric Linear Transport Equations by ReLU DNNs (2020) (https://arxiv.org/abs/2001.11441) | Zbl 07349297

[47] I. E Lagaris; A. Likas; D. I Fotiadis Artificial neural networks for solving ordinary and partial differential equations, IEEE Transactions on Neural Networks, Volume 9 (1998) no. 5, pp. 987-1000 | Article

[48] K. Law; A. Stuart; K. Zygalakis Data Assimilation: A Mathematical Introduction, Texts in Applied Mathematics, 62, Springer, 2015 | Article | Zbl 1353.60002

[49] Y. LeCun; Y. Bengio; G. Hinton Deep learning, Nature, Volume 521 (2015) no. 7553, pp. 436-444 | Article

[50] K. Lee; K. T. Carlberg Model reduction of dynamical systems on nonlinear manifolds using deep convolutional autoencoders, J. Comput. Phys., Volume 404 (2020) | MR 4043884 | Zbl 1454.65184

[51] Z. Li; N. Kovachki; K. Azizzadenesheli; B. Liu; K. Bhattacharya; A. Stuart; A. Anandkumar Neural Operator: Graph Kernel Networkfor Partial Differential Equations (2020) (https://arxiv.org/abs/2003.03485)

[52] C. Lin; Z. Li; L. Lu; S. Cai; M. Maxey; G. E. Karniadakis Operator learning for predicting multiscale bubble growth dynamics (2020) (https://arxiv.org/abs/2012.12816)

[53] G. J. Lord; C. E. Powell; T. Shardlow An introduction to computational stochastic PDEs, 50, Cambridge University Press, 2014 | Article | Zbl 1327.60011

[54] L. Lu; P. Jin; G. E. Karniadakis DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators (2019) (https://arxiv.org/abs/1910.03193)

[55] L. Lu; P. Jin; G. Pang; Z. Zhang; G. E. Karniadakis Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators, Nature Machine Intelligence (2020)

[56] Y. Maday; A. T. Patera; J. D. Penn; M. Yano A parameterized-background data-weak approach to variational data assimilation: formulation, analysis, and application to acoustics, Int. J. Numer. Meth. Engng., Volume 102 (2015) no. 5, pp. 933-965 | Article | MR 3341243 | Zbl 1352.65529

[57] V. Maiorov; A. Pinkus Lower Bounds for Approximation by MLP Neural Networks, Neurocomputing, Volume 25 (1999), pp. 81-91 | Article | Zbl 0931.68093

[58] Z. Mao; L. Lu; O. Marxen; T. A Zaki; G. E. Karniadakis DeepM&Mnet for hypersonics: Predicting the coupled flow and finite-rate chemistry behind a normal shock using neural-network approximation of operators (2020) (https://arxiv.org/abs/2011.03349)

[59] S. A. McQuarrie; C. Huang; K. Willcox Data-driven reduced-order models via regularized operator inference for a single-injector combustion process (2020) (https://arxiv.org/abs/2008.02862)

[60] K. P. Murphy Machine Learning: A Probabilistic Perspective, The MIT Press, 2012 https://www.cs.ubc.ca/~murphyk/mlbook/ | Zbl 1295.68003

[61] D. A. Nagy Modal representation of geometrically nonlinear behavior by the finite element method, Computers & Structures, Volume 10 (1979) no. 4, pp. 683-688 | Article | Zbl 0406.73071

[62] M. L. Overton; R. S. Womersley On the Sum of the Largest Eigenvalues of a Symmetric Matrix, SIAM Journal of Matrix Analysis and Applications, Volume 13 (1992) no. 1, pp. 41-45 | Article | MR 1146651 | Zbl 0747.15005

[63] K. Pearson LIII. On lines and planes of closest fit to systems of points in space, The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, Volume 2 (1901) no. 11, pp. 559-572 | Article | Zbl 32.0246.07

[64] B. Peherstorfer Sampling low-dimensional Markovian dynamics for pre-asymptotically recovering reduced models from data with operator inference (2019) (https://arxiv.org/abs/1908.11233)

[65] B. Peherstorfer; K. Willcox Data-driven operator inference for nonintrusive projection-based model reduction, Comput. Methods Appl. Mech. Eng., Volume 306 (2016), pp. 196-215 | Article | MR 3502565 | Zbl 1436.93062

[66] E. Qian; B. Kramer; B. Peherstorfer; K. Willcox Lift & Learn: Physics-informed machine learning for large-scale nonlinear dynamical systems, Physica D: Nonlinear Phenomena, Volume 406 (2020), p. 132401 | Article | MR 4073541

[67] A. Quarteroni; A. Manzoni; F. Negri Reduced basis methods for partial differential equations: an introduction, Springer, 2015 | Article

[68] M. Raissi; P. Perdikaris; G. E. Karniadakis Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations, J. Comput. Phys., Volume 378 (2019), pp. 686-707 | Article | MR 3881695 | Zbl 1415.68175

[69] S. Reich; C. Cotter Probabilistic forecasting and Bayesian data assimilation, Cambridge University Press, 2015 | Article | Zbl 1314.62005

[70] L. Ruthotto; E. Haber Deep Neural Networks Motivated by Partial Differential Equations, J. Math. Imaging Vis., Volume 62 (2019), pp. 352-364 | Article | MR 4082375 | Zbl 1434.68522

[71] B. Schölkopf; A. Smola; K. Müller Nonlinear Component Analysis as a Kernel Eigenvalue Problem, Neural Computation, Volume 10 (1998) no. 5, pp. 1299-1319 | Article

[72] C. Schwab; J. Zech Deep learning in high dimension: Neural network expression rates for generalized polynomial chaos expansions in UQ, Anal. Appl., Singap., Volume 17 (2019) no. 01, pp. 19-55 | Article | MR 3894732 | Zbl 07007693

[73] J. Shawe-Taylor; C. Williams; N. Cristianini; J. Kandola On the eigenspectrum of the Gram matrix and its relationship to the operator eigenspectrum, International Conference on Algorithmic Learning Theory, Springer, 2002, pp. 23-40 | Article | Zbl 1024.68538

[74] J. Shawe-Taylor; C. Williams; N. Cristianini; J. Kandola On the eigenspectrum of the Gram matrix and the generalization error of kernel-PCA, IEEE Transactions on Information Theory, Volume 51 (2005) no. 7, pp. 2510-2522 | Article | MR 2246374 | Zbl 1310.15076

[75] Y. Shin; J. Darbon; G. E. Karniadakis On the convergence and generalization of physics informed neural networks (2020) (https://arxiv.org/abs/2004.01806)

[76] J. D. Smith; K. Azizzadenesheli; Z. E. Ross EikoNet: Solving the Eikonal equation with Deep Neural Networks (2020) (https://arxiv.org/abs/2004.00361)

[77] S.-E. Takahasi; J. M. Rassias; S. Saitoh; Y. Takahashi Refined generalizations of the triangle inequality on Banach spaces, Math. Inequal. Appl., Volume 13 (2010) no. 4, pp. 733-741 | Article | MR 2760496 | Zbl 1205.26034

[78] R. Temam Infinite-dimensional dynamical systems in mechanics and physics, 68, Springer, 2012

[79] Q. Wang; J. S. Hesthaven; D. Ray Non-intrusive reduced order modeling of unsteady flows using artificial neural networks with application to a combustion problem, J. Comput. Phys., Volume 384 (2019), pp. 289-307 | Article | MR 3920924 | Zbl 1459.76117

[80] E. Weinan A proposal on machine learning via dynamical systems, Communications in Mathematics and Statistics, Volume 5 (2017) no. 1, pp. 1-11 | MR 3627592 | Zbl 1380.37154

[81] D. Yarotsky Error bounds for approximations with deep ReLU networks, Neural Netw., Volume 94 (2017), pp. 103-114 | Article | Zbl 1429.68260

[82] E. Zeidler Applied Functional Analysis: Applications to Mathematical Physics, Springer, 2012

[83] Y. Zhu; N. Zabaras Bayesian Deep Convolutional Encoder-Decoder Networks for Surrogate Modeling and Uncertainty Quantification, J. Comput. Phys., Volume 366 (2018) no. C, pp. 415-447 | Article | MR 3800689 | Zbl 1407.62091