Previous work

NeurIPS 2026

U-MVPEncode Locally, Decode Globally for Feed-Forward 3D Gaussian Splatting

Yonsei University*Equal contribution   †Corresponding author

32 posed input views
Rendered novel views

From 32 posed images of a DL3DV benchmark scene, U-MVP predicts 3D Gaussians in a single forward pass (0.12 s) and renders them along the captured camera path.

Encode locally

Frame-wise attention in the encoder keeps per-view detail at full resolution.

Decode globally

Multi-view attention in the decoder reasons about cross-view geometry.

16 to 256 views

Group-and-swap attention at the bottleneck keeps dense inputs affordable.

TL;DR

  1. Cross-view reasoning happens late, but compute goes early. Analyzing a multi-view 3DGS transformer (MVP), we find that early layers mostly build per-frame features while cross-view reasoning emerges in later layers, yet 52–66% of the inference time goes to frame-wise feature aggregation.
  2. So encode locally, decode globally. U-MVP uses a frame-wise encoder, a cross-view decoder, skip-connected fusion and a group-swap bottleneck. On DL3DV it achieves the highest PSNR among the compared feed-forward methods from 16 to 256 input views, and it is faster than MVP at every tested view count, now spending about 70% of its time in the transformer blocks.
Compute allocation, reversed. Share of inference time spent in the transformer blocks versus the multi-scale aggregation path (MVP's PFA module; U-MVP's skip fusion and transposed convolutions). Paper Table 8, single RTX PRO 6000.
Transformer blocksAggregation
16 views
MVP
U-MVP
32 views
MVP
U-MVP
64 views
MVP
U-MVP
128 views
MVP
U-MVP
192 views
MVP
U-MVP

Feed-forward vs. per-scene optimization · 16 and 32 input views

U-MVP against optimized 3D-GS

Both methods get the same 16 or 32 input frames and are rendered along the same camera path, following the scene's captured trajectory. 3D-GS is optimized on each scene for 30k iterations; U-MVP is one forward pass of a single checkpoint for both view counts. Drag the divider to compare.

Input views
3D-GS U-MVP (Ours)
0.0 s
PSNR (dB) · 32 input views 3D-GSU-MVP (Ours)

Inputs are the k-means 16 / 32 frames of the DL3DV benchmark protocol. PSNR is measured on the benchmark's standard test frames (every 8th frame, never an input): per scene for the scenes listed, and over all 140 scenes in the top row. The videos for 16 and 32 views share one camera path (the 32-view path) at 960×544 with the footage's focal length. 3D-GS uses the reference implementation with default hyperparameters.

Comparison · same 32 input views

Compare against the baselines

Held-out frames from ten other DL3DV benchmark scenes, rendered by every method from the same 32 input frames, next to the real footage. Long-LRM, MVP and U-MVP run at 960×544; tttLRM runs at its native 960×536 and is resized for display. Error maps show the mean absolute RGB difference on one colour scale per scene. Hover or drag on any image to zoom the same spot in all of them.

Frame

Geometry

The 3D Gaussians, as points

U-MVP's predicted Gaussian centres from one forward pass over 16 or 32 posed input views, each coloured with its own predicted colour. Pink frusta are the input cameras. A light filter removes centres that no other input view agrees with, flying pixels at sharp depth edges and the least opaque 20%. No ground-truth geometry is used.

Input views
drag: rotate · scroll / pinch: zoom · right- or shift-drag: pan
Loading point cloud…

Method

A U-shaped multi-view transformer

U-MVP architecture: four patchified input views enter frame-attention encoder stages with conv downsampling, a bottleneck of frame, group and global attention, decoder stages with group and frame attention fused with encoder features through residual convolution units, and a linear head producing 3D Gaussians, shown as a coloured point cloud of the scene.

Frame-wise encoder

Each view is processed on its own with frame attention across three resolution levels, extracting multi-resolution per-view features.

Cross-view decoder

Group and global attention bring in multi-view context while the decoder restores spatial resolution.

Skip fusion and Gaussian head

Encoder features rejoin the decoder through residual convolution units; a per-pixel linear head predicts the 3D Gaussians.

Features · 16 input views

Sharper features before the Gaussian head

PCA of the per-pixel features that enter the Gaussian head, for four of the sixteen input views. One PCA is fit over all input views for each model, and its top three components are shown as RGB, so colours are consistent across views. U-MVP's skip-connected decoder keeps fine structure such as shelf edges, leaves and carvings that MVP's features blur.

Quantitative results · DL3DV benchmark, 960×540

Quality keeps rising with more views

Method16 views32 views64 views128 views192 views256 views

PSNR (dB) from Tables 1 and 2 of the paper; 3D-GS is optimized for 30k iterations per scene. The released Long-LRM checkpoint produces blank frames at 192 and 256 views (N/A).

Reconstruction time ↓

Log-log scale. MVP and U-MVP: our timing in eager mode (dashed) and with torch.compile (solid). Other methods: paper Tables 1–2 (one B200).

Citation

@inproceedings{yang2026umvp,
  title     = {U-MVP: Encode Locally, Decode Globally for Feed-Forward 3D Gaussian Splatting},
  author    = {Yang, Seungkwon and Kang, Gyeongjin and Jeong, Hwasik and
               Kang, Byeongjin and Cho, Hyeongbhin and Park, Eunbyung},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026}
}