Adaptive Web-Based VR Streaming of Multi-LoD 3D Scenes via Author-Provided Relevance Scores
Research paper on adaptive, web-based VR streaming of 3D scenes, published at IEEE VRW 2021.
Published at the 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), Lisbon, Portugal, pages 488-489.
Authors: Hendrik Lievens, Maarten Wijnants, Mike Vandersanden, Peter Quax and Wim Lamotte.
My part was a web client with VR capabilities that uses MPEG-DASH to stream the resources of the virtual environment to the user, built in the summer of 2020.
Paper (IEEE)Summary
This paper proposes relevance-aware adaptive bitrate (ABR) scheduling with MPEG-DASH, tailored to VR in the web browser. Scene authors annotate the relative importance of scene assets, which the scheduler uses to decide what to stream first. The approach outperforms the state of the art, measured with MS-SSIM, across different scene complexities and network configurations.
How it works
Streaming a 3D scene differs from streaming video. The user can move and look around freely, so what to fetch next depends on where the user is and what is in view. The streamed scene is described in X3D, and every model is available in several levels of detail, so the client can start with a rough version and refine it. The scene is delivered with MPEG-DASH, and the client runs in the web browser with WebXR, so it works on any VR or AR headset with a browser.
Scheduling normally weighs an object’s size on screen against its distance to the user. Here the scene author can also give each model a semantic weight, stored in a separate heat-map file, to say which objects matter most. The weight never changes how an object looks. It only changes the order in which the client downloads objects.
My part was the web client, which streams the scene with MPEG-DASH and renders it in VR.
Left: a temple model at a low and a higher level of detail. Right: the city scene and, below it, a heat-map of the importance that the scene author gave to each model (blue is most important).
The city scene at frame 86 on a 25 Mbps link, with the baseline ABR logic (first) and the semantic ABR logic (second). The semantic one has already loaded the objects the author marked as important.
Results
The semantic scheduler was compared with an existing size-and-distance approach (the baseline) in three purpose-built scenes, on three network links.
| Scene | What it is | Weighted as important |
|---|---|---|
| City | A dense urban scene | Buildings and roads; trees count for less |
| Rural | Houses, farming machinery and a hangar | Roads and buildings; trees count for less |
| Temple | A temple with trees, in front of a mountain | The foreground, over the bigger background objects |
| Link | Bandwidth | Latency |
|---|---|---|
| Slow | 5 Mbps | 50 ms |
| Fast | 25 Mbps | 25 ms |
| Broadband | 100 Mbps | 10 ms |
Quality was measured with MS-SSIM, against a reference that has every object at full detail.
- Poor networks gain the most. The slower the link, the more it pays off to download the important objects first.
- Scene layout matters. In the temple scene, where small important objects stand in front of large unimportant ones, the gains are large. In the rural scene, where objects are similar in size, it behaves much like the existing approach.
- Wrong weights hurt. A wrong annotation gives a worse result than not annotating at all.
Cite
@inproceedings{lievens2021adaptive,
author = {Lievens, Hendrik and Wijnants, Maarten and Vandersanden, Mike and Quax, Peter and Lamotte, Wim},
title = {Adaptive Web-Based {VR} Streaming of Multi-{LoD} 3D Scenes via Author-Provided Relevance Scores},
booktitle = {2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)},
address = {Lisbon, Portugal},
pages = {488--489},
month = mar,
year = {2021},
publisher = {IEEE},
doi = {10.1109/VRW52623.2021.00126},
}
BibTeX
Tags
IEEE VR 2021
Research