In computational graphics, signals are often represented discretely and explicitly. For example, images are represented as discrete grids of pixels, and 3D shapes are described with polygonal mesh, voxels and point clouds. In contrast, implicit Neural Representation [16] (INRs) utilizes neural networks to approximate a continuous signal, as shown in Fig. 1. Parameterized signals are decoupled with spatial resolution and scale only with the complexity of the underlying signal. Commonly used methods include Signed Distance Functions [17] (SDF), Neural Distance Fields [18] (NDF), Occupancy Fields [19], etc. These continuous representations can be extended to coordinate-based networks. For 3D shapes with textured surfaces, the scene representation network proposed by Sitzmann et al. [20] encodes both geometry and appearance by learning SDFs and texture colors for each coordinate. This network can be trained end-to-end from 2D images and their camera poses without querying depth or shape information. Niemeyer et al. [21] introduce a differentiable rendering formulation for implicit representation of shape and texture, enabling optimization and learning of surface radiance. Saito et al. [22] propose a Pixel-aligned Implicit Function, which predicts surface occupancy and color for coordinates. Sitzmann et al. [23] suggest parameterizing room scale by employing sinusoidal activation functions (SIN) in INRs.

NeRF also falls into the category of INR, which trains coordinate-based networks to approximate the continuous radiance fields within the volume enclosed by the object’s surface. Once the training is completed, NeRF enables rendering 2D images from any new viewing directions. As one of the most significant breakthroughs in INRs, NeRF has attracted enormous interest since its inception. Mip-NeRF [24] addresses the issue of ignoring the volume and size of the observed region in NeRF by replacing rays with cones, allowing for sampling from a continuous conical frustum instead of discrete points. Instant-NGP [25] improves upon existing graphics structures and applies them to volumetric rendering frameworks by utilizing a sparsely parameterized voxel grid as a scene representation and optimizing both the scene and MLP (where one MLP serves as the decoder) based on gradients, thereby accelerating the transition from 2D to 3D reconstruction. PixelNeRF [26] incorporates spatial image features aligned with each pixel as input, leveraging convolutional networks to extract low-level features, which are then combined with the input to the NeRF grid to learn prior knowledge about the scene. This approach allows for generating new views with minimal input, even for unknown scenes. FiG-NeRF [27] interprets scenes as a geometrically consistent background and a deformable foreground representing object categories. By using only optical supervision and randomly captured object images, accurate 3D object category models can be learned.

Over the past few decades, significant progress has been made in the field of Computational Fluid Dynamics (CFD) (as shown in Fig. 2) [28] for solving compressible or incompressible flow problems [29] using mesh-based numerical methods such as finite volume methods and finite element methods [30]. Nevertheless, the generation of polygonal grids is a time-consuming task, and solving inverse problems is intractable. Deep learning seems to be a promising tool for data-driven computational mechanics [31,32] (Fig. 3). For example, a pioneering work can be seen in Milano et al. [33], who proposed a nonlinear neural network that significantly increased the prediction accuracy of near-wall velocity fields. However, different from the fields of object detection and tracking, where deep learning has been applied successfully, complex fluid flow possesses highly nonlinear and multiscale characteristics, and the data are often sparse and noisy, which poses great challenges to the pure data-driven paradigm of computing science.

Physics-Informed Neural Network [6,34] addressed the aforementioned issue by informing neural networks of the physical mechanisms of fluid dynamics. In addition to penalizing the deviation of neural network predictions from data, PINNs incorporate the residual of Navier-Stokes equations and boundary conditions into loss terms. Once trained, the PINN can quickly make predictions and thus can be used as a surrogate model. Furthermore, due to the differentiability of PINN, it is an ideal choice for solving inverse and optimization problems. It is noteworthy that classical CFD algorithms [35] still maintain their advantages in accuracy and efficiency when solving standard forward problems and meet the practical requirements of robustness and computational efficiency in many applications. Therefore, classical methods and deep neural network methods will coexist and complement each other in the foreseeable future.

Fluid flow patterns (e.g., color, density, and velocity) can be analyzed by visualizing and tracking the movement of passive tracers such as ink or dye in the fluids. Experimental fluid mechanics widely adopts Particle Image Velocimetry (PIV) [36], which is a non-intrusive laser optical measurement technique where the velocity field of an entire region within the flow is measured simultaneously. However, PIV requires cumbersome configuration of experimental devices and the accuracy has not matured to a satisfactory level in 3D. To alleviate the difficulty, Gregson et al. [37] reconstructed fluids from RGB images by combining fluid capture, simulation, and proximal methods [38] based on global priors. Zang et al. [39] extended the visible light tomographic imaging technique with sparse viewpoints to achieve fluid flow reshaping, utilizing interpolation techniques and two novel regularization methods. Qiu et al. [40] employed convolutional networks to simulate input scenes with flexible coupling effects based on the estimated velocity field, resulting in stable image effects. Chu et al. [11] propose the first method for combining NeRF and PINN to recover the density and velocity of fluids from sparse multi-view RGB video streams.

Uncertainty is ubiquitous in science and has a profound impact on engineering design and modeling [41–44]. Many approaches have been proposed for quantifying uncertainties of the prediction by deep neural networks, such as single deterministic methods [45], Bayesian methods [46], Test-time augmentation methods [47], and ensemble methods, as shown in Fig. 4 [48]. Single deterministic methods can be further categorized into internal uncertainty quantification methods and external uncertainty quantification methods, which involve modeling a single network and incorporating additional components on the network’s predictions to provide uncertainty estimation. Bayesian methods treat the weights of neural networks as random variables and make predictions based on multiple sets of weight distributions. Test-time augmentation methods apply data augmentation techniques to create multiple test samples for each test sample and compute the prediction distribution to measure uncertainty. Ensemble methods combine prediction results from multiple members of an ensemble to quantify uncertainty. While ensemble methods require high memory usage, they offer advantages in simplicity, scalability, and flexibility.

As a differentiable volume rendering approach, NeRF uses fully connected neural networks to construct a mapping from Cartesian coordinates and viewing directions to color intensities c and optical densities ρ in a 3D scene (Fig. 5):

(1)F(x,y,z,θ,ϕ)→(c,ρ)

where (x, y, z) denotes the coordinates in the three-dimensional Cartesian coordinate system, color c=(r,g,b) consists of three components associated with RBG channels, and (θ,ϕ) represents the azimuthal and polar angles of viewing directions. Supervised by the data of 2D images from different viewpoints, two Multilayer Perceptrons (MLPs) in NeRF, one coarse and one refined, are trained jointly to synthesize novel view images. It is noted that in Chu’s paper [11], the view direction variables (θ,ϕ) are neglected for simplicity, based on the assumption of Lambertian surfaces and isotropic scattering medium. By adding the time dimension, the model can be extended to the dynamics of fluid scenes in both spatial and temporal dimensions:

(2)Fs:(x,y,z)→(c,ρ)Fd:(x,y,z,t)→(c,ρ)

The aforementioned spatiotemporal neural radiance fields learn the information from neural scene flow [49]. By predicting 3D scene flow for both forward and backward frames, it represents the 3D displacement vectors of the x position at time i + 1 and i − 1.

(3)Fi=(Fi→i+1,Fi→i−1)

This is based on the hypothesis that the color of a pixel obtained by volume rendering at time i should be the same as the color of the pixel reached by the ray after moving to a new coordinate at time j.

Further, with the given volume density and color function, it is possible to use the volume rendering function to obtain the color C(r) for any camera ray r(t)=o+td, in which o represents the camera position, and d denotes the viewing direction, (4)C(r)=∫t2t1T(t)⋅ρ(r(t))⋅c(r(t),d)⋅dt

where T(t) represents the cumulative transmittance, which denotes the probability of a ray propagating from t1 to r without being intercepted. It can be calculated using the following equation:

(5)T(t)=exp⁡[−∫t1tρ(r(s))⋅ds]

By tracing camera rays C(r) passing through each pixel in the target synthesized image, novel views can be generated. This integration can be evaluated numerically with non-deterministic stratified sampling, where rays are partitioned into N equally spaced intervals, and samples are drawn uniformly from each interval. Then, Eq. (5) can be approximated as follows:

(6)C^(r)=∑i=1NαiTiciTi=exp⁡[−∑j=1i−1ρjδj]

where δi represents the distance from sample i to sample i+1, (ρi,ci) are the density and color evaluated at sampling point i along the given ray, respectively, which are computed by the neural network, and αi is given by the following equation:

(7)αi=1−exp⁡(ρiδi)

In general, the expected depth of a ray can be calculated by utilizing the cumulative transmittance:

(8)d(r)=∫t2t1T(t)⋅ρ(r(t))⋅t⋅dt

The above equation can be achieved by approximating Eqs. (3) and (5) in a similar way to Eq. (6). (9)D^(r)=∑i=1NαitiTi

For each pixel, the square error photometric loss is used to train neural network parameters. Over the whole image, this is given by the following equation:

(10)ℒ=∑r=R‖C^(r)−Cg(r)‖22

where ℒ is a batch of rays associated with the image to be synthesized, and Cg(r) is the true color of the training image pixel associated with R.

Position encoding is a commonly used technique in NeRF to improve the reconstruction of fine details in rendered views [9] (Fig. 6). The following position encoding γ is applied to each component of the scene coordinates x (normalized to [−1, 1]):

(11)γL(x)=(sin⁡(20x),cos⁡(20x),sin⁡(21x),cos⁡(21x),...,sin⁡(2L−1x),cos⁡(2L−1x))

where L is a user-determined coding dimension parameter, and in the original NeRF paper, x was set to L=10, and d was set to L=4.

A significant challenge faced by the basic NeRF structure is the tendency to overfit training data, thereby limiting their ability to interpret three-dimensional geometry in a multi-view manner. One possible reason is that position encoding, which extends the input location information to higher dimensions, prevents the NeRF from exploring low-frequency information in the early stages of neural network training. Based on the work of Chu et al. [11], we adopt frequency regularization proposed by Yang et al. [15] in FreeNeRF to optimize the reconstruction of sparse view inputs for smoke. This approach improves the position encoding by using a linearly growing frequency mask to modulate the visible spectrum based on training time steps. The formula is given as follows:

(12)γ′L(t,T;x)=γL(x)⊙α(t,T,L)with αi(t,T,L)={1t⋅LT−⌊t⋅LT⌋0ifi≤t⋅LT+3ift⋅LT+3<i≤t⋅LT+6ifi>t⋅LT+6

in which t and T represent the current training iteration and the final iteration, respectively. The frequency regularization mitigates the instability and vulnerability to high-frequency signals at the beginning of training and gradually provides NeRF with high-frequency information to avoid excessive smoothing.

PINNs encode physical laws represented by Partial Differential Equations (PDEs) in neural networks by transforming PDEs to an optimization model that minimizes a loss function containing the residuals (Fig. 7). In its most general form, a PDE can be expressed as follows:

(13)F(φ(z);γ)=f(z),in ΩB(φ(z))=g(z),on ∂Ω

where Ω⊂Rd is the domain on which the physical problem is defined, ∂Ω indicates the boundary of the domain, z:=[x1,...,xd−1;t] denotes the spatio-temporal coordinates, u represents the unknown physical fields, γ is a physical parameter, f is a function that identifies the problem data, and ℱ is the nonlinear differential operator. In the second equation, ℬ is expressed as an operator with arbitrary initial or boundary conditions associated with the problem, and g is taken to be a boundary function. Eq. (13) can be used in forward and backward problems. The goal of the forward problem is to find a function u for each z when γ is given. In the case of the inverse problem, γ must also be determined from the data.

To solve Eq. (13), ϕ(z) can be approximated with a neural network that is parameterized by a set of parameters θ, (14)φ(z)≈φ^θ(z)

where θ is obtained by minimizing a loss function that depends on the observed data and PDEs as well as the boundary conditions. (15)θ∗=arg⁡minθ⁡(wℱℒℱ(θ)+wℬℒℬ(θ)+wdℒdata(θ))

As can be seen from the above equation, PINN differs from traditional neural networks in that the residuals of PDEs are introduced into the loss function to enforce physical laws, whereby the accuracy and efficiency can be enhanced even with sparse training data.

Based on the model structure used in Chu’s paper [11], the architecture of our neural networks is shown in Fig. 8, which contains residual structure [50] and two-layer activation function from sf-PINN [51]. Residual networks address the issue of “degradation” in NNs by incorporating the original features into the subsequent training process, thereby avoiding feature loss during training. The definition of a residual network is very simple and is given by the following equation:

(16)f(x)=g(x)+x

Moreover, sf-PINN and its double-layer activation function have been proven to effectively increase gradient variability, preventing training stagnation and getting trapped in local minima. They have shown good performance in various forward and inverse modeling problems. The sin [23] and Swish functions [52] (as shown in Fig. 9) are defined by the following equation:

(17)f(x)=sin⁡(x)f(x)=x⋅η(x) with η(x)=11+e−x

The flow of incompressible fluid is governed by the Navier-Stokes equation [53]:

(18)∂u∂t+u∇⋅u=−ρ∇p+1Re∇2u+b (momentum equation)∇⋅u=0 (incompressibility)

where u is the velocity, which contains three components (ux,uy,uz), p is the pressure, Re is Reynolds number, and b is the body force.

The particles (passive scalar) in fluids are transferred through convection and diffusion processes, which are characterized by the convection-diffusion equation:

(19)∂c∂t+u⋅∇c=0

where c is the mass concentration of smoke particles, which is proportional to optical density ρ. In the remainder of the paper, we will not distinguish between c and ρ. Based on the particle density distribution obtained from NeRF, a PINN is trained to construct a mapping Fv spatio-temporal coordinates to velocity:

(20)Fv:(x,y,z,t)→u(ux,uy,uz)

which minimizes the following loss terms:

(21)ℒDcDt=(∂c∂t+u⋅∇c)2,ℒNSE=‖∂u∂t+u⋅∇u‖22+wdiv‖∇⋅u‖22

We take the partial derivatives of the velocity components obtained from the operations via the FFJORD tool [54] and encode them from the Navier-Stokes equations into the PDE loss function, which is added to each gradient backpropagation of the neural network. thus incorporating a physical prior in the hybrid model. The merit of PINN is that the approach does not require boundary conditions as input and can handle fluid within arbitrary spatial regions. In other words, we train a velocity network based on PINN.

In the field of computer vision, there are two main types of uncertainties: aleatoric uncertainty and epistemic uncertainty [14]. Aleatoric uncertainty arises from the inherent uncertainty in the input data, while epistemic uncertainty refers to the uncertainty in the model weights learned. Despite being one of the preferred methods for numerous computer vision modeling tasks, most existing models of NeRF are not able to accurately quantify their inherent model uncertainties (epistemic uncertainties).

Ensemble learning combines multiple base models to fulfill learning tasks. It is commonly perceived as a meta-algorithm and is rooted in intuitive thinking. it provides a representation of model uncertainty in predictions by quantifying the prediction distribution derived from multiple constituent members and evaluating the diversity among member predictions. This framework can be readily implemented and deployed without needing any alterations to the underlying NeRF and PINN architectures.

Since the primary application of NeRFs is to generate two-dimensional images of novel viewpoints, the uncertainty in color prediction is the issue that needs to be addressed first (Fig. 10). In accordance with the theory of ensemble learning, we train multiple NNs [55] {fθn}n=1...N. They are trained on the same dataset but initialized with different weights. The final prediction is obtained by averaging the predictions of all member networks. Hence, the expected color of r (ray) in the scene is written as:

(22)C¯(r)=1N∑n=1NCθn(r)

The average amount of variability is represented by the standard deviation (std) of individual members’ predictions as:

(23)σRGB(r)=1N∑n=1N(C¯(r)−Cθn(r))2

By employing this intuitively feasible approach, we obtain a color-density distribution field predicted by a neural network ensemble. These distributions are mutually independent and can be accessed independently in time. This ensemble learning-based framework can be easily extended and applied to NeRF-based models.

Based on the above equation, we can directly assess the use of C¯(r) and σRGB(r) to analyze uncertainty. However, the above equations are obtained for each individual color channel, to simplify the calculation and analysis, we calculate the average standard deviation of the three color channels:

(24)σ¯RGB(r)=13∑i∈{R,G,B}σRGB,i(r)

where σRGB,i(r) represents the standard deviation on each RGB color channel.

Although NeRF can manipulate the geometric shape, color, and density distribution of 3D objects or scenes, it fails to provide information on invisible physical quantities such as velocity and pressure, which are crucial for engineering design in fluid dynamics. Chu et al. [11] proposed to employ PINN to infer the velocity fields according to the densities obtained by NeRF. Because the fluid flow model is coupled with the particle transportation model, the accuracy of PINN in predicting velocity is closely linked with the uncertainty of the density distribution of the particles. With a similar stacked training approach, the distribution of the velocity field is evaluated by training multiple PINNs under the same conditions:

(25)σu,i(ρ)=1N∑n=1N(ui(ρ)−u¯i(ρ))2  i=x,y,z

where i represents the three components of velocity, and we adopt the mean of these three standard deviations as the final uncertainty term.

The work of PINN can also be accomplished using a multi-output approach, as shown in Fig. 11, which has been tested in experiments. In order to maintain consistency within the framework, an integrated training method is employed in subsequent discussions. Regardless of which method is used, we obtain the predicted distribution of the velocity field, which enables an in-depth analysis of it.

The present work adopts the ScalarFlow dataset [56], which comprises real-world smoke plumes captured by five fixed cameras positioned along a 120-degree arc. In this dataset, each fixed view comprises 120 frames with a duration of about 4 s. During training, the video frame is split into 600 images. In each training iteration, an image is selected randomly, and by utilizing the time anchor present in the image, the neural network predicts the attributes of the spatial point within the current scene. Although only 1024 pixel points were sampled at a time and fed into the neural network for training, the large number of training iterations (600 k) ensured that each pixel in the image was used at least once. The machine parameters used for training our model are outlined below:

Processor: Intel Core i9-10900x @ 3.70 GHz

We rendered the smoke plumes generated by the hybrid model into a 30-frame (1-s) RGB video, covering the entire process from smoke appearance to generation. Fig. 12 presents representative video frames extracted and arranged in chronological order (original dataset images on the left, rendered images on the right). It can be observed that the generated model not only achieves high fidelity in static images but also captures temporal dynamics features. In addition, some challenges arise in the rendering process. For example, spatial points that are close to white are rendered as pure white in the new view, as shown in Fig. 13. This is possibly due to flaws in the volume rendering itself as well as the repeated use of estimation in the calculation process. This necessitates more reasonable regularization penalty conditions in our future work.

As the velocity is invisible to the naked eye, we convert the acquired velocity and vorticity into the HSV space and render it as an RGB image, as shown in Fig. 14. Vorticity is used to describe the local rotation of a fluid parcel, which is the spin of an air microcluster in the atmosphere. As can be seen from the image, the velocity field obtained through PINN is non-linear and has been captured by our algorithm.

Common quality assessment metrics, such as PSNR, SSIM, and LPIPS, are often used. In this paper, we adopted PSNR [57] as the criterion for evaluating the quality of generated images with original training images, given by the following equation:

(26)PSNR(I)=10⋅log10(MAX(I)2MSE(I))

A comparison was made between our rendered results and Chu’s model [11] using the test set. The Fig. 15 below showcases the juxtaposition of six randomly selected images from the training data in chronological order with the test results obtained through PSNR evaluation. On the right side, the velocity loss data from the final 10k training iterations are presented. It is evident that our rendered results demonstrate certain enhancements in terms of fine details at the same number of iterations. Furthermore, the training process of the velocity network weights exhibits a higher level of stability.

We first examine the accuracy of color predictions for points in space along specific directions. A statistical analysis was performed on four sets of training samples collected over time (10, 100, 250, 500 k), as depicted in Fig. 16. Fig. 16 (left) features a line chart exhibiting the standard deviations for each batch of 1024 training points (sorted for improved clarity, as image oscillation was present), while Fig. 16 (right) showcases the distribution range of standard deviations. The figure illustrates that the maximum standard deviation predicted by the model is approximately 0.2, with the majority of values falling below 0.1. From Fig. 13, it can be seen that the values predicted by neural network for smoke particles are mostly approaching to 0. This implies that while the absolute value of uncertainty is not particularly high, there still exists a notable level of uncertainty relative to the mean prediction.

Furthermore, the pie chart (Fig. 17) also demonstrates that as the number of training iterations increases, there is an overall decrease in the model’s predicted uncertainty (the left corresponds to 100 k iterations, while the right corresponds to 600 k iterations). This indicates that although each training instance captures images at different instances of time, the color of a specific point in the dynamic fluid spatial domain could be similar or exhibit negligible changes over time.

To examine the accuracy of the aforementioned observation, we constructed a box plot for the standard deviations of the predicted results, ranging from 10 to 600 k with an increment of 100 k, as depicted in Fig. 18. The box plot is a statistical chart used to display the dispersion of a set of data. It is primarily used to reflect the distribution characteristics of raw data, and can also be employed to compare the distribution characteristics of multiple data sets. The observed trend from the graph indicates that there is no evident variation in the minimum and maximum values, with outliers close to 0.2 present even at the 600k-th training iteration. However, a declining trend in the overall standard deviation can be observed from the quartiles utilized to construct the box.

The optical density ρ at a specific spatial point obtained by NeRF indicates its opacity or high scattering characteristics of the object at that point (e.g., clouds or fog). The optical density is linearly proportional to the mass concentration of smoke particles, thus reflecting the transfer of particles in fluid flow. Due to the uncertainty in NeRF training, the number of density points in the fluid model will be different. In the simultaneously trained NeRFs, we first statistically analyze the density values as an initial quantification criterion for the density uncertainty. From Fig. 19, it can be observed that statistical analysis was performed on the density quantity six times, ranging from 100 to 600 k. The differences in each iteration are attributed to differences in scene selection.

As can be seen in Fig. 19, the values of densities at 200 and 600 k iterations are higher. We quantified the uncertainty (standard deviation) for these two instances separately. According to Fig. 20, we can infer that overall, density-related uncertainty has decreased. However, due to lack of supervision, there still remains a significant level of uncertainty even with an adequate number of training iterations.

Finally, we quantified velocity uncertainty, which stems from the uncertainty in neural network weight learning and the uncertainty in density distribution. As shown in Table 1, it is the mean value of the velocity we measured during training. From Fig. 21, it can be observed that the overall trend of velocity uncertainty increases as the number of iterations increases. Moreover, from Fig. 21b, it can be noted that since the mean velocity is only between 0.01–0.5, the uncertainty in velocity compared to color, which has clear dataset supervision, requires further extension of supervised training in future work.