Model Reduction And Neural Networks For Parametric PDEs
The SMAI journal of computational mathematics, Volume 7 (2021), pp. 121-157.

We develop a general framework for data-driven approximation of input-output maps between infinite-dimensional spaces. The proposed approach is motivated by the recent successes of neural networks and deep learning, in combination with ideas from model reduction. This combination results in a neural network approximation which, in principle, is defined on infinite-dimensional spaces and, in practice, is robust to the dimension of finite-dimensional approximations of these spaces required for computation. For a class of input-output maps, and suitably chosen probability measures on the inputs, we prove convergence of the proposed approximation methodology. We also include numerical experiments which demonstrate the effectiveness of the method, showing convergence and robustness of the approximation scheme with respect to the size of the discretization, and compare it with existing algorithms from the literature; our examples include the mapping from coefficient to solution in a divergence form elliptic partial differential equation (PDE) problem, and the solution operator for viscous Burgers’ equation.

Published online:
DOI: 10.5802/smai-jcm.74
Classification: 65N75,  62M45,  68T05,  60H30,  60H15
Keywords: approximation theory, deep learning, model reduction, neural networks, partial differential equations.
Kaushik Bhattacharya 1; Bamdad Hosseini 2; Nikola B. Kovachki 2; Andrew M. Stuart 2

1 Mechanical and Civil Engineering, California Institute of Technology, Pasadena, CA, USA
2 Computing and Mathematical Sciences, California Institute of Technology, Pasadena, CA, USA
License: CC-BY-NC-ND
Copyrights: The authors retain unrestricted copyrights and publishing rights
@article{SMAI-JCM_2021__7__121_0,
     author = {Kaushik Bhattacharya and Bamdad Hosseini and Nikola B. Kovachki and Andrew M. Stuart},
     title = {Model {Reduction} {And} {Neural} {Networks} {For} {Parametric} {PDEs}},
     journal = {The SMAI journal of computational mathematics},
     pages = {121--157},
     publisher = {Soci\'et\'e de Math\'ematiques Appliqu\'ees et Industrielles},
     volume = {7},
     year = {2021},
     doi = {10.5802/smai-jcm.74},
     language = {en},
     url = {https://smai-jcm.centre-mersenne.org/articles/10.5802/smai-jcm.74/}
}
TY  - JOUR
TI  - Model Reduction And Neural Networks For Parametric PDEs
JO  - The SMAI journal of computational mathematics
PY  - 2021
DA  - 2021///
SP  - 121
EP  - 157
VL  - 7
PB  - Société de Mathématiques Appliquées et Industrielles
UR  - https://smai-jcm.centre-mersenne.org/articles/10.5802/smai-jcm.74/
UR  - https://doi.org/10.5802/smai-jcm.74
DO  - 10.5802/smai-jcm.74
LA  - en
ID  - SMAI-JCM_2021__7__121_0
ER  - 
%0 Journal Article
%T Model Reduction And Neural Networks For Parametric PDEs
%J The SMAI journal of computational mathematics
%D 2021
%P 121-157
%V 7
%I Société de Mathématiques Appliquées et Industrielles
%U https://doi.org/10.5802/smai-jcm.74
%R 10.5802/smai-jcm.74
%G en
%F SMAI-JCM_2021__7__121_0
Kaushik Bhattacharya; Bamdad Hosseini; Nikola B. Kovachki; Andrew M. Stuart. Model Reduction And Neural Networks For Parametric PDEs. The SMAI journal of computational mathematics, Volume 7 (2021), pp. 121-157. doi : 10.5802/smai-jcm.74. https://smai-jcm.centre-mersenne.org/articles/10.5802/smai-jcm.74/

[1] J. Adler; O. Oktem Solving ill-posed inverse problems using iterative deep neural networks, Inverse Probl., Volume 33 (2017) no. 12, 124007 | DOI | MR | Zbl

[2] B. O. Almroth; P. Stern; F. A. Brogan Automatic choice of global shape functions in structural analysis, AIAA J., Volume 16 (1978) no. 5, pp. 525-528 | DOI

[3] M. Barrault; Y. Maday; N. C. Nguyen; A. T. Patera An ‘empirical interpolation’ method: application to efficient reduced-basis discretization of partial differential equations, C. R. Math. Acad. Sci. Paris, Volume 339 (2004) no. 9, pp. 667-672 | DOI | MR | Zbl

[4] P. Baxendale Gaussian Measures on Function Spaces, Am. J. Math., Volume 98 (1976) no. 4, pp. 891-952 | DOI | MR | Zbl

[5] M. Belkin; P. Niyogi Laplacian Eigenmaps for Dimensionality Reduction and Data Representation, Neural Computation, Volume 15 (2003) no. 6, pp. 1373-1396 | DOI | Zbl

[6] P. Benner; P. Goyal; B. Kramer; B. Peherstorfer; K. Willcox Operator inference for non-intrusive model reduction of systems with non-polynomial nonlinear terms, Comput. Methods Appl. Mech. Eng., Volume 372 (2020), p. 113433 | DOI | MR | Zbl

[7] S. Bhatnagar; Y. Afshar; S. Pan; K. Duraisamy; S. Kaushik Prediction of aerodynamic flow fields using convolutional neural networks, Comput. Mech. (2019), pp. 1-21 | DOI | MR | Zbl

[8] P. Binev; A. Cohen; W. Dahmen; R. DeVore; G. Petrova; P. Wojtaszczyk Data assimilation in reduced modeling, SIAM/ASA J. Uncertain. Quantif., Volume 5 (2017) no. 1, pp. 1-29 | DOI | MR | Zbl

[9] G. Blanchard; O. Bousquet; L. Zwald Statistical properties of kernel principal component analysis, Machine Learning, Volume 66 (2007) no. 2, pp. 259-294 | DOI | Zbl

[10] S. Boyaval; C. Le Bris; T. Lelievre; Y. Maday; N. C. Nguyen; A. T. Patera Reduced basis techniques for stochastic problems, Arch. Comput. Methods Eng., Volume 17 (2010) no. 4, pp. 435-454 | DOI | MR | Zbl

[11] S. Cai; Z. Wang; L. Lu; T. A Zaki; G. E. Karniadakis DeepM&Mnet: Inferring the electroconvection multiphysics fields based on operator approximation by neural networks (2020) (https://arxiv.org/abs/2009.12935)

[12] T. Chen; H. Chen Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems, IEEE Transactions on Neural Networks, Volume 6 (1995) no. 4, pp. 911-917 | DOI

[13] L. Cheng; N. Kovachki; M. Welborn; T. F. Miller Regression Clustering for Improved Accuracy and Training Costs with Molecular-Orbital-Based Machine Learning, Journal of Chemical Theory and Computation, Volume 15 (2019) no. 12, pp. 6668-6677 | DOI

[14] A. Chkifa; A. Cohen; R. DeVore; C. Schwab Sparse adaptive Taylor approximation algorithms for parametric and stochastic elliptic PDEs, ESAIM, Math. Model. Numer. Anal., Volume 47 (2013) no. 1, pp. 253-280 | DOI | Numdam | MR | Zbl

[15] A. Cohen; W. Dahmen; R. DeVore State Estimation–The Role of Reduced Models (2020) (https://arxiv.org/abs/2002.00220)

[16] A. Cohen; R. DeVore Approximation of high-dimensional parametric PDEs, Acta Numer., Volume 24 (2015), pp. 1-159 | DOI | MR | Zbl

[17] A. Cohen; R. DeVore; C. Schwab Convergence Rates of Best N-term Galerkin Approximations for a Class of Elliptic SPDEs, Found. Comput. Math., Volume 10 (2010) no. 6, pp. 615-646 | DOI | MR | Zbl

[18] A. Cohen; R. DeVore; C. Scwhab Analytic regularity and polynomial approximation of parametric and stochastic elliptic PDEs, Anal. Appl., Singap., Volume 09 (2011) no. 01, pp. 11-47 | DOI | MR

[19] R. R. Coifman; S. Lafon; A. B. Lee; M. Maggioni; B. Nadler; F. Warner; S. W. Zucker Geometric diffusions as a tool for harmonic analysis and structure definition of data: Diffusion maps, Proceedings of the National Academy of Sciences, Volume 102 (2005) no. 21, pp. 7426-7431 | DOI | Zbl

[20] M. Dashti; S. Harris; A. M. Stuart Besov priors for Bayesian inverse problems, Inverse Probl. Imaging, Volume 6 (2012), pp. 183-200 | DOI | MR | Zbl

[21] I. Daubechies; R. DeVore; S. Foucart; B. Hanin; G. Petrova Nonlinear Approximation and (Deep) ReLU Networks (2019) (https://arxiv.org/abs/1905.02199)

[22] R. DeVore Nonlinear approximation, Acta Numer., Volume 7 (1998), pp. 51-150 | DOI | MR

[23] R. DeVore The Theoretical Foundation of Reduced Basis Methods, Model Reduction and Approximation, Society for Industrial and Applied Mathematics, 2014 | DOI

[24] T. Dockhorn A Discussion on Solving Partial Differential Equations using Neural Networks (2019) (https://arxiv.org/abs/1904.07200)

[25] W. E; B. Yu The Deep Ritz Method: A Deep Learning-Based Numerical Algorithm for Solving Variational Problems, Communications in Mathematics and Statistics (2018) | DOI | MR | Zbl

[26] L. C. Evans Partial differential equations, American Mathematical Society, 2010

[27] K. Fan On a Theorem of Weyl Concerning Eigenvalues of Linear Transformations I, Proceedings of the National Academy of Sciences, Volume 35 (1949) no. 11, pp. 652-655 | DOI | MR

[28] S. Fresca; L. Dede; A. Manzoni A comprehensive deep learning-based approach to reduced order modeling of nonlinear time-dependent parametrized PDEs (2020) (https://arxiv.org/abs/2001.04001) | Zbl

[29] M. Geist; P. Petersen; M. Raslan; R. Schneider; G. Kutyniok Numerical solution of the parametric diffusion equation by deep neural networks (2020) (https://arxiv.org/abs/2004.12131)

[30] J. Gilmer; S. S Schoenholz; P. F. Riley; O. Vinyals; G. E. Dahl Neural message passing for quantum chemistry, Proceedings of the 34th International Conference on Machine Learning (2017) (http://proceedings.mlr.press/v70/gilmer17a.html)

[31] F. J. Gonzalez; M. Balajewicz Deep convolutional recurrent autoencoders for learning low-dimensional feature dynamics of fluid systems (2018) (https://arxiv.org/abs/1808.01346)

[32] R. Gonzalez-Garcia; R. Rico-Martínez; I. G. Kevrekidis Identification of distributed parameter systems: A neural net based approach, Computers & Chemical Engineering, Volume 22 (1998), p. S965-S968 | DOI

[33] I. Goodfellow; Y. Bengio; A. Courville Deep Learning, MIT Press, 2016 http://www.deeplearningbook.org | Zbl

[34] E. Haber; L. Ruthotto Stable architectures for deep neural networks, Inverse Probl., Volume 34 (2017) no. 1, p. 014004 | DOI | MR | Zbl

[35] L. Herrmann; C. Schwab; J. Zech Deep ReLU Neural Network Expression Rates for Data-to-QoI Maps in Bayesian PDE Inversion (2020) (https://www.sam.math.ethz.ch/sam_reports/reports_final/reports2020/2020-02.pdf)

[36] J. S. Hesthaven; G. Rozza; B. Stamm et al. Certified reduced basis methods for parametrized partial differential equations, SpringerBriefs in Mathematics, Springer, 2016 | DOI | Zbl

[37] J. S. Hesthaven; S. Ubbiali Non-intrusive reduced order modeling of nonlinear problems using neural networks, J. Comput. Phys., Volume 363 (2018), pp. 55-78 | DOI | MR | Zbl

[38] G. Hinton; R. Salakhutdinov Reducing the Dimensionality of Data with Neural Networks, Science, Volume 313 (2006) no. 5786, pp. 504-507 | DOI | MR | Zbl

[39] J. R Holland; J. D Baeder; K. Duraisamy Field Inversion and Machine Learning With Embedded Neural Networks: Physics-Consistent Neural Network Training, AIAA Aviation 2019 Forum (2019), 3200 pages | DOI

[40] J.-T. Hsieh; S. Zhao; S. Eismann; L. Mirabella; S. Ermon Learning Neural PDE Solvers with Convergence Guarantees, International Conference on Learning Representations (2019) (https://openreview.net/forum?id=rklawn0qk7)

[41] M. A. Iglesias; K. Lin; A. M. Stuart Well-posed Bayesian geometric inverse problems arising in subsurface flow, Inverse Probl., Volume 30 (2014), p. 114001 | DOI | MR | Zbl

[42] Y. Khoo; J. Lu; L. Ying Solving parametric PDE problems with artificial neural networks (2017) (https://arxiv.org/abs/1707.03351)

[43] G. Klambauer; T. Unterthiner; A. Mayr; S. Hochreiter Self-Normalizing Neural Networks, Advances in Neural Information Processing Systems 30 (I. Guyon; U. V. Luxburg; S. Bengio; H. Wallach; R. Fergus; S. Vishwanathan; R. Garnett, eds.), Curran Associates, 2017, pp. 971-980

[44] K. Krischer; R. Rico-Martínez; I. G. Kevrekidis; H. H. Rotermund; G. Ertl; J. L. Hudson Model identification of a spatiotemporally varying catalytic reaction, AIChE J., Volume 39 (1993) no. 1, pp. 89-98 | DOI

[45] G. Kutyniok; P. Petersen; M. Raslan; R. Schneider A theoretical analysis of deep neural networks and parametric PDEs (2019) (https://arxiv.org/abs/1904.00377)

[46] F. Laakmann; P. Petersen Efficient Approximation of Solutions of Parametric Linear Transport Equations by ReLU DNNs (2020) (https://arxiv.org/abs/2001.11441) | Zbl

[47] I. E Lagaris; A. Likas; D. I Fotiadis Artificial neural networks for solving ordinary and partial differential equations, IEEE Transactions on Neural Networks, Volume 9 (1998) no. 5, pp. 987-1000 | DOI

[48] K. Law; A. Stuart; K. Zygalakis Data Assimilation: A Mathematical Introduction, Texts in Applied Mathematics, 62, Springer, 2015 | DOI | Zbl

[49] Y. LeCun; Y. Bengio; G. Hinton Deep learning, Nature, Volume 521 (2015) no. 7553, pp. 436-444 | DOI

[50] K. Lee; K. T. Carlberg Model reduction of dynamical systems on nonlinear manifolds using deep convolutional autoencoders, J. Comput. Phys., Volume 404 (2020) | MR | Zbl

[51] Z. Li; N. Kovachki; K. Azizzadenesheli; B. Liu; K. Bhattacharya; A. Stuart; A. Anandkumar Neural Operator: Graph Kernel Networkfor Partial Differential Equations (2020) (https://arxiv.org/abs/2003.03485)

[52] C. Lin; Z. Li; L. Lu; S. Cai; M. Maxey; G. E. Karniadakis Operator learning for predicting multiscale bubble growth dynamics (2020) (https://arxiv.org/abs/2012.12816)

[53] G. J. Lord; C. E. Powell; T. Shardlow An introduction to computational stochastic PDEs, 50, Cambridge University Press, 2014 | DOI | Zbl

[54] L. Lu; P. Jin; G. E. Karniadakis DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators (2019) (https://arxiv.org/abs/1910.03193)

[55] L. Lu; P. Jin; G. Pang; Z. Zhang; G. E. Karniadakis Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators, Nature Machine Intelligence (2020)

[56] Y. Maday; A. T. Patera; J. D. Penn; M. Yano A parameterized-background data-weak approach to variational data assimilation: formulation, analysis, and application to acoustics, Int. J. Numer. Meth. Engng., Volume 102 (2015) no. 5, pp. 933-965 | DOI | MR | Zbl

[57] V. Maiorov; A. Pinkus Lower Bounds for Approximation by MLP Neural Networks, Neurocomputing, Volume 25 (1999), pp. 81-91 | DOI | Zbl

[58] Z. Mao; L. Lu; O. Marxen; T. A Zaki; G. E. Karniadakis DeepM&Mnet for hypersonics: Predicting the coupled flow and finite-rate chemistry behind a normal shock using neural-network approximation of operators (2020) (https://arxiv.org/abs/2011.03349)

[59] S. A. McQuarrie; C. Huang; K. Willcox Data-driven reduced-order models via regularized operator inference for a single-injector combustion process (2020) (https://arxiv.org/abs/2008.02862)

[60] K. P. Murphy Machine Learning: A Probabilistic Perspective, The MIT Press, 2012 https://www.cs.ubc.ca/~murphyk/mlbook/ | Zbl

[61] D. A. Nagy Modal representation of geometrically nonlinear behavior by the finite element method, Computers & Structures, Volume 10 (1979) no. 4, pp. 683-688 | DOI | Zbl

[62] M. L. Overton; R. S. Womersley On the Sum of the Largest Eigenvalues of a Symmetric Matrix, SIAM Journal of Matrix Analysis and Applications, Volume 13 (1992) no. 1, pp. 41-45 | DOI | MR | Zbl

[63] K. Pearson LIII. On lines and planes of closest fit to systems of points in space, The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, Volume 2 (1901) no. 11, pp. 559-572 | DOI | Zbl

[64] B. Peherstorfer Sampling low-dimensional Markovian dynamics for pre-asymptotically recovering reduced models from data with operator inference (2019) (https://arxiv.org/abs/1908.11233)

[65] B. Peherstorfer; K. Willcox Data-driven operator inference for nonintrusive projection-based model reduction, Comput. Methods Appl. Mech. Eng., Volume 306 (2016), pp. 196-215 | DOI | MR | Zbl

[66] E. Qian; B. Kramer; B. Peherstorfer; K. Willcox Lift & Learn: Physics-informed machine learning for large-scale nonlinear dynamical systems, Physica D: Nonlinear Phenomena, Volume 406 (2020), p. 132401 | DOI | MR

[67] A. Quarteroni; A. Manzoni; F. Negri Reduced basis methods for partial differential equations: an introduction, Springer, 2015 | DOI

[68] M. Raissi; P. Perdikaris; G. E. Karniadakis Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations, J. Comput. Phys., Volume 378 (2019), pp. 686-707 | DOI | MR | Zbl

[69] S. Reich; C. Cotter Probabilistic forecasting and Bayesian data assimilation, Cambridge University Press, 2015 | DOI | Zbl

[70] L. Ruthotto; E. Haber Deep Neural Networks Motivated by Partial Differential Equations, J. Math. Imaging Vis., Volume 62 (2019), pp. 352-364 | DOI | MR | Zbl

[71] B. Schölkopf; A. Smola; K. Müller Nonlinear Component Analysis as a Kernel Eigenvalue Problem, Neural Computation, Volume 10 (1998) no. 5, pp. 1299-1319 | DOI

[72] C. Schwab; J. Zech Deep learning in high dimension: Neural network expression rates for generalized polynomial chaos expansions in UQ, Anal. Appl., Singap., Volume 17 (2019) no. 01, pp. 19-55 | DOI | MR | Zbl

[73] J. Shawe-Taylor; C. Williams; N. Cristianini; J. Kandola On the eigenspectrum of the Gram matrix and its relationship to the operator eigenspectrum, International Conference on Algorithmic Learning Theory, Springer, 2002, pp. 23-40 | DOI | Zbl

[74] J. Shawe-Taylor; C. Williams; N. Cristianini; J. Kandola On the eigenspectrum of the Gram matrix and the generalization error of kernel-PCA, IEEE Transactions on Information Theory, Volume 51 (2005) no. 7, pp. 2510-2522 | DOI | MR | Zbl

[75] Y. Shin; J. Darbon; G. E. Karniadakis On the convergence and generalization of physics informed neural networks (2020) (https://arxiv.org/abs/2004.01806)

[76] J. D. Smith; K. Azizzadenesheli; Z. E. Ross EikoNet: Solving the Eikonal equation with Deep Neural Networks (2020) (https://arxiv.org/abs/2004.00361)

[77] S.-E. Takahasi; J. M. Rassias; S. Saitoh; Y. Takahashi Refined generalizations of the triangle inequality on Banach spaces, Math. Inequal. Appl., Volume 13 (2010) no. 4, pp. 733-741 | DOI | MR | Zbl

[78] R. Temam Infinite-dimensional dynamical systems in mechanics and physics, 68, Springer, 2012

[79] Q. Wang; J. S. Hesthaven; D. Ray Non-intrusive reduced order modeling of unsteady flows using artificial neural networks with application to a combustion problem, J. Comput. Phys., Volume 384 (2019), pp. 289-307 | DOI | MR | Zbl

[80] E. Weinan A proposal on machine learning via dynamical systems, Communications in Mathematics and Statistics, Volume 5 (2017) no. 1, pp. 1-11 | MR | Zbl

[81] D. Yarotsky Error bounds for approximations with deep ReLU networks, Neural Netw., Volume 94 (2017), pp. 103-114 | DOI | Zbl

[82] E. Zeidler Applied Functional Analysis: Applications to Mathematical Physics, Springer, 2012

[83] Y. Zhu; N. Zabaras Bayesian Deep Convolutional Encoder-Decoder Networks for Surrogate Modeling and Uncertainty Quantification, J. Comput. Phys., Volume 366 (2018) no. C, pp. 415-447 | DOI | MR | Zbl

Cited by Sources: