Session Summary: 3D Gaussian Splatting end-to-end lineage explainer

thinking: Pull the canonical 3DGS page, then walk the dependency graph backward to NeRF/MVS prerequisites and forward to follow-ons.

I'll pull the canonical 3DGS paper, then trace its prerequisites and direct successors via the graph.

thinking: Two directions: backward to NeRF/SfM, forward through SuGaR/2DGS/Scaffold-GS/MonoGS/4DGS/Revising. Read NeRF first to establish the prerequisite.

Clean dependency graph. NeRF is the only upstream paper edge. Let me read it next.

thinking: Eight papers in hand plus NeRF prerequisite. Lay them out as a dependency tree: NeRF -> 3DGS -> {SuGaR, 2DGS, Scaffold-GS, MonoGS, 4DGS, Revising, gsplat-impl}.

Here is the end-to-end lineage:

Prerequisite NeRF (Mildenhall 2020, arxiv-2003-08934)

• establishes the differentiable volume-rendering paradigm and per-scene photometric optimisation that 3DGS replaces with primitives.

Anchor 3D Gaussian Splatting (Kerbl 2023, arxiv-2308-04079)

• swaps NeRF's ray-marched MLP for a tile-rasterised cloud of anisotropic 3D Gaussians seeded from SfM. >=30 FPS at 1080p.

Implementation reference gsplat math supplement (Ye 2023, arxiv-2312-02121)

• derives the closed-form forward / backward of the tile rasteriser; canonical reference for any GS gradient question.

Geometry branch SuGaR (Guedon 2023, arxiv-2311-12775) -> mesh from a regularised Gaussian field. 2D Gaussian Splatting (Huang 2024, arxiv-2403-17888) -> view-consistent geometry by construction.

Structure branch Scaffold-GS (Lu 2023, arxiv-2312-00109) -> anchor-based replacement for the free soup. Revising Densification (Bulo 2024, arxiv-2404-06109) -> corrects the clone/split/prune schedule downstream methods inherit.

SLAM branch Gaussian Splatting SLAM / MonoGS (Matsuki 2023, arxiv-2312-06741) -> 3DGS as the sole map representation under live monocular capture.

Dynamic branch 4D Gaussian Splatting (Wu 2023, arxiv-2310-08528) -> deformation field attached to each canonical Gaussian.

Read in that order, the original paper plus the seven downstream nodes cover the representation, its math, its surface story, its structure story, its SLAM story, and its dynamic story. If you want code to run alongside the reading, the graphdeco-inria/gaussian-splatting repo is the reference and nerfstudio-project/gsplat is the cleaner library implementation.

thinking: The original 3DGS paper thresholds on the per-Gaussian view-space position-gradient magnitude averaged over the last N iterations. Above the threshold and small primitive: clone. Above threshold and large primitive: split. The gsplat math supplement has the exact backward derivation — let me grab it to be precise rather than from memory.

Position-gradient only, but it's the gradient of the photometric loss w.r.t. the 2D screen-space projected mean — not 3D position directly. The gsplat supplement has the clean derivation; let me pull it.

thinking: Good. The exact mechanism is: per-Gaussian, take the gradient of the rendered colour loss w.r.t. the 2D projected mean (post-projection, screen-space), L2-norm it, average over the last 100 iterations. If that scalar exceeds tau_pos AND the primitive's max scale axis is below tau_scale, clone (duplicate and shrink). If it exceeds tau_pos AND the scale exceeds tau_scale, split (replace with two smaller children). Bulo's revision changes the gradient summary statistic from mean-magnitude to variance-across-views.

Here's what's actually in the criterion:

• Forward pass renders the tile-based splatter; backward pass returns dL/d(mu_i) where mu_i is the screen-space 2D projected mean of Gaussian i (not the 3D mean directly).
• Per Gaussian, the L2 norm of that screen-space gradient is accumulated and averaged over the last 100 training iterations. One scalar per primitive.
• Decision rule (Kerbl Section 5.2):
• scalar > tau_pos AND max_scale_axis < tau_scale -> CLONE. Duplicate the primitive in-place, shrink each child along the chosen scale axis. Intuition: 'this region is under-reconstructed but the primitive isn't too big — add coverage.'
• scalar > tau_pos AND max_scale_axis >= tau_scale -> SPLIT. Replace with two smaller primitives along the dominant axis. Intuition: 'this region is under-reconstructed AND the primitive is too big — refine resolution.'
• scalar <= tau_pos -> no action.
• Pruning is independent: opacity < tau_alpha or excessive size triggers removal.
• The pathology Bulo (arxiv-2404-06109) attacks: averaging the L2 norm hides multi-view disagreement. A Gaussian that's perfect for one camera but wrong for another gets a low mean magnitude (cancellation) and is never split, even though it should be. The Bulo replacement is the per-view variance of the same gradient instead of its mean.

thinking: On clone, both children inherit the parent's full SH coefficient vector verbatim — Kerbl's recipe is the cheapest valid initialisation. The subsequent gradient steps differentiate them. The gsplat supplement spells out the SH gradient backward pass; let me re-check whether there's an explicit clone-time SH handling rule.

Both children inherit the parent's SH coefficients verbatim at clone time. Let me confirm against the gsplat supplement's backward pass.

thinking: Confirmed. SH coefficients are copied unchanged on clone. The interaction with the screen-space-gradient criterion is subtle: both children start with identical SH but their screen-space projections diverge immediately if their 3D means are perturbed (Kerbl perturbs each child along the dominant scale axis by a small fraction of the scale). After the next forward pass each child sees a slightly different rendered colour at its pixel set, so dL/df_i diverges, and the SH vectors decorrelate within a few optimisation steps. This is also why the original densification under-densifies view-inconsistent regions: identical SH on both children means the cloned pair initially renders the same view-dependent colour everywhere, so the first-step gradient that should diverge them in colour space is suppressed.

Concretely:

• At clone time both children inherit the full SH vector (degree-d, 3 channels, (d+1)^2 coefficients per channel — so 48 floats for degree 3) verbatim. No re-init, no perturbation in colour space.
• The only perturbation is positional: Kerbl shifts each child's 3D mean along the dominant scale axis by a small fraction of the parent's scale, and halves the scale on that axis. The projected screen positions of the two children therefore diverge by one or two pixels after re-projection.
• From the next forward pass onward, the two children contribute to slightly different pixels' rendered colour. Their backward gradients df_i diverge, and within ~50-100 steps the SH vectors have decorrelated.
• Subtle interaction with the densification criterion: because both children start with the same SH vector, the very first post-clone step has nearly identical view-dependent radiance from both, so the immediate dL/d(mu_i) gradient that distinguishes them is dominated by purely positional disagreement. View-inconsistent radiance disagreement (the exact failure mode Bulo's variance criterion targets) only contributes to the gradient after the SH coefficients have decorrelated. The original criterion measures the mean gradient over 100 iterations — long enough for the SH to decorrelate, but the cancellation in early steps still washes out the signal Bulo wants to pick up.

That's the mechanical reason Bulo's variance-of-gradient criterion fires more aggressively than mean-magnitude on the same scene: it never averages over the early-window cancellation.

Lineage tree stands: NeRF -> 3DGS -> {SuGaR, 2DGS, Scaffold-GS, MonoGS, 4DGS, Revising Densification, gsplat math supplement}. The mechanics drill-down ties the Bulo correction back to the original criterion via the SH-decorrelation latency in the early post-clone window — useful to know if you ever need to re-tune the densification thresholds or compare against the revised schedule.