The gradient propagation process of 3D Gaussian

## Definition

- Gaussians: $G_1, G_2, \dots, G_N$

- Definition of gaussians: 
    $$
    G_i(x) = \exp (-\frac{1}{2}(x-\mu_i^{3d})^T (\Sigma_i^{3d})^{-1}(x-\mu_i^{3d})) \quad(1)
    $$

- For each gaussian primitives, it also contains:

- Color: spherical harmonics $SH_i$
  - Opacity: $o_i$

## Rendering in 2D plane

- 2D Plane 
  $$
  G_i^{2d} = \exp{(-\frac{1}{2} (x-\mu_i)^T(\Sigma_i^{2d})^{-1}(x - \mu_i))} \quad(2)\
  $$

- alpha-blending:
  $$
  C(x) = \sum_{i=1}^N C_i \alpha_i \prod_{j=1}^{i-1} (1 - \alpha_j) \quad (3)
  $$
  where 
  $$
  \alpha_i = \sigma(o_i) \times G_i^{2d}(x) \quad (4)
  $$

## Gradient calculation

Assume that: The position of a projected gaussian to pixel-space, which is the mean value of its gaussian expression, is $\mu_i^k(\mu_{i,x}^k, \mu_{i,y}^k)$ under the $k-$th view. Its gradient whose loss is $L_k$ with totally $M$ views composing its gradient computation is:
$$
\nabla \mu_i L = \frac{\sum_{k=1}^M \|\frac{\partial L^k}{\partial \mu_i^k}\|}{M} = \frac{\sum_{k=1}^{M} \sqrt{(\frac{\partial L_k}{\partial \mu_{i,x}^{k}})^2 + \frac{\partial L_k}{\partial \mu_{i,y}^{k}})^2 }}{M} \quad(5)
$$
The condition for splitting a gaussian is:  $\nabla \mu_{i} L > \tau_p$ and $\Sigma_{i}^{3d} > \tau_s$

## The conflict of gradient directions

Note that: $ g_i $ is $ \frac{\partial L}{\partial \mu_i} $ under view $ k $ ，where $ g_{i,x} = \frac{\partial L}{\partial \mu_{i,x}} $，$ g_{i,y} = \frac{\partial L}{\partial \mu_{i,y}} $

For example, 
$$
g_{i,x} = \frac{\partial L}{\partial \mu_{i,x}} = \sum_{j=1}^{M} \frac{\partial L_j}{\partial \mu_{i,x}} \quad(6)
$$
where $L_j$ means that it is the $j-$th item of $L$ which is contributed by $j$-th pixel.

### Proof of anisotropy

Now, show that, $\frac{\partial L_j}{\partial \mu_{i,x}}$ is anisotropy (they have different directions means that their signs are different (either $+$ or $-$))

First we have, 
$$
\frac{\partial L_j}{\partial \mu_{i,x}} = \sum_{l=1}^3 \frac{\partial L_j}{\partial C_{l}^{j}} \times \frac{\partial C_l^{j}}{\partial \alpha_i} \times \frac{\partial \alpha_i}{\partial \mu_{i,x}} \quad (7)
$$
Evaluations for each item in $(7)$ are following:

#### Item 1

- $L_j$ is the loss. $L_j > 0$ always stands.
- $c_{l}^{j}$ is the color. $c_{l}^{j} > 0$ always stands.
- This item evaluate the difference between ground truth color and the rendered color.

#### Item 2

By $(3)$ ：
$$
\begin{aligned}
\frac{\partial c_{l}^{j}}{\partial \alpha_i} & = \frac{\partial (\sum_{i=1}^{N} c_i\alpha_i \prod_{j=1}^{i-1} \alpha_j)}{\partial \alpha_j} \\
& = \sum_{p=i}^N C_{l}^{p} \prod_{t=i+1}^{p-1} (1 - \alpha_t) + C_{l}^{i} \prod_{t=i+1}^{N} (1 - \alpha_t) \quad(8)
\end{aligned}
$$
In fact, the alpha-blending formula can be summarized as:

- transmittance: $T_{i-1} = \prod_{j=1}^{i - 1} (1 - \alpha_j)$  The rest light after previous $i - 1$ gaussians.
- weight: $w_i = \alpha_i T_{i-1}$

For the first item in $(8)$:

- $k = i$ : $w_i = \alpha_iT_{i-1} \Rightarrow \frac{\partial w_i}{\alpha_i} = T_{i-1}$

- $k > i$: $w_k = \alpha_{k}T_{k - 1}$ where $T_{k - 1} = T_{i-1}(1-\alpha_i)\prod_{p = i + 1}^{k - 1} (1 - \alpha_p)$. Therefore,
  $$
  \frac{\partial w_k}{\alpha_i} = \alpha_k \frac{\partial T_{k - 1}}{\partial \alpha_i} = -\alpha_k T_{k - 1} / (1 - \alpha_i)
  $$

**We can see that the two part have different signs. Therefore, the sign of this item us undetermined. **

#### Item 3

From $(4)$ :
$$
\frac{\partial \alpha_i}{\partial \mu_{i,x}} = \sigma(o_i) \times \frac{G_i^{2d}(x)}{\partial \mu_{i,x}}
$$
where $\sigma(x)$ is the sigmoid function.

Taking the derivative with respect to the central coordinate $\mu_{i,x}$ is equivalent to translating the Gaussian by an infinitesimal amount.
Let the current pixel coordinate be $x=(p_j,q_j)$. Taking the partial derivative with respect to $\mu_{i,x}$ gives
$$
\frac{\partial G_{i}^{2d}}{\partial\mu_{i,x}} = G_{i}^{2d}(x)\left[-\Sigma_{i}^{2d - 1}(x-\mu_{i})\right]_x=G_{i}^{2d}(x)\left[-\frac{1}{\sigma_{1}^{2}}(p_j - \mu_{i,x})\right]
$$
Pushing the negative sign out:
$$
\frac{\partial \alpha_i}{\partial \mu_{i,x}} = \frac{(\mu_{i,x} - p_j) G_{i}^{2d}(x)}{\sigma_i^2}
$$
Tips:

- Taking the derivative with respect to the Gaussian center = the exponential itself × a linear term.
- The coefficient of the linear term is provided with direction and scale by the inverse covariance matrix.

**Conclusion: the sign of this item depending on the relative position between $\mu_{i,x}$ and $p_j$ **