This example displays the basic usage of configuring 360° video sources. WebXR and Cardboard mode support is added with special WebXR events for debugging.
360° feature for Flowplayer has been developed and improved over many years keeping up with changing browser apis , requirements and IOS problems introduced.
This feature has been updated to accomodate the new WebXR api with a polyfill for WebVR 1.0 and Cardboard view for Safari. WebXR is now the new standard for WebVR which Chrome currently supports. The new orientation sensor api available in Chrome browsers in Android has been added.
The rendering of the video makes use of WebGL in browsers that support it, utilising the graphics card available. NVidia, Radeon and Integrated GPU's will render better. Any Html5 video will be supported including Mpeg Dash and HLS Native and HLS.JS.
Cubemap Video Support
According to the investigations and work at Facebook https://code.facebook.com/posts/1638767863078802/under-the-hood-building-360-video/. Cubemap encoded videos help reduce wasted pixels and therefore bandwidth. The videos frames are tiled into a 3 x 2 formation.
Due to their discoveries equirectangle videos have wasted pixels around the edges and duplicated pixels where it wraps. Cubemaps have defined edges and therefore no redundant pixels are in the video. These are space and performance saving optimisations. Facebook have an Ffmpeg encoder filter called Transform360. Which will help transform Equirectangle sources to Cubemap video sources. The demo was produced using this filter.
Fisheye Camera Support
There is support for both Ricoh Theta dual 360 Fisheye and single 180 Fisheye support. When using 180 degree videos a skybox texture is supported.
360 Video Controls
Various user controls are supported for controlling the video camera view. Desktop rotation controls include keyboard key codes, mouse drag and pointer lock controls for rotating and panning, mouse wheel for zooming. An external player api is provide to also control rotation using scripting.
Mobile rotation controls include touch for rotation when not in fullscreen mode. In non VR fullscreen mode mobile orientation apis are used for controlling rotation with the deivce.
In VR headsets mode on Mobile and in WebXR enabled browsers on desktop. WebXR apis are used to control rotation using orientation apis of the VR Oculus or Samsung Gear headsets or the mobile device. On mobile in WebXR cardboard mode it will fallback to the device's orientation.
VR Controllers are supported to control a virtual video controlbar. When connected the GearVR and DayDream remote or Oculus and Vive gaming controllers will be enabled as a laser pointer.
360 video support is possible on Android 4.3 and above with Firefox or Chrome or Safari on IOS 8 and above that supports WebGL.
On Samsung S8 devices with Android 7, the default browser is Samsung Internet and has both WebGL and WebVR support.
With the Safari browser on IOS it will use the Cardboard view in VR mode having no WebVR / WebXR support.
IOS and Android apps can be built around Apache Cordova or IONIC Framework WebView based applications. The Android version of WebView uses Chrome so it's dependant on keeping the Chromium WebView up to date.
"pinned to Home Screen" standalone WebView apps work with VR.
Extensive bug fixes and work arounds are available for bugs in IOS Safari when streaming HLS formats, making live 360 video possible.
VR support for VR headsets like Oculus Rift, Vive, Samsung Gear VR, Google DayDream View, Microsoft Mixed Reality and Google Cardboard is possible with a VR stereo view of the video and autonomous rotating, panning and zooming control using either Cardboard webxr polyfill or native WebXR/WebVR support if enabled in the browser.
VR headset support is possible with WebXR/WebVR browser API's found in Chrome, Firefox and Edge browsers. This relies on the Oculus Runtime for Windows, therefore OSX support has been dropped for now. These API's provide controls and status from the headsets which can then be used for controlling the video camera view.
Microsoft Mixed Reality with the Mixed Reality Simulator and supported headsets. Provides WebVR support with the Microsoft Edge browser.
WebXR/WebVR is available on Android Chrome, Firefox and Samsung Gear VR Browser, this is now enabled by default.
Daydream View requires the Daydream application installed. And Chrome browser with WebVR enabled on Android.
VR Video Controls and subtitles text support is available in WebVR presentation mode.The video controls will display when clicking the VR display. And a reticle can be used for gaze control. Subtitles text will rotate with head tracking.
A flat 2D video Cinema screen mode in VR is supported. When launching from GearVR, Cardoard, or VR headets Occulus, Vive and Windows Mixed Reality. The video will display into a virtual Cinema screen. When subtitles are configured they will also be displayed over the Cinema screen.
Experimental support for 4 channel Ambisonics audio tracks in video is possible through an Ambisonics to Binaural audio decoder. Browsers firstly require mutli channel. The audio requires to be authored according to Google Spatial Media specification.
Amibsonics supports 2nd and 3rd order audio streams. The panning of the camera will pan the audio in headphones.
Producing 360 Videos
360° video can be produced using special surround cameras like ones supplied by https://www.360rize.com/.
The camera sources are required to be stitched together after into a panoromic flat spherical picture using special VR 360° software. Software like http://www.video-stitch.com/.
Live 360 is possible using a device like a Teradeck Sphere. This will publish to Wowza streaming servers which can then be packetised into Mpeg Dash and HLS.
Video Capture Support
Using the Snapshot Plugin, support is available to capture VR Video rendering.
VR Picture in picture support
Not normally supported by default. Features have been implemented to support picture in picture toggling of the VR rendering.
- 360° video support in any Html5 supported format including Mp4, Webm, Ogv, Mpeg Dash and HLS.
- Efficient , bandwidth and space saving Cubemap video support.
- 180 degree video rendering support. This will render in a half sphere with head turn rotation.
- Various user control functions for the video rotation including keyboard, mouse, tablet orientation and movements, tablet touch, external api controls and a movement pad area. In WebXR/WebVR mode orientation controls are provided by the headsets.
- VR headset support for Oculus Rift, Vive when used with a VR API enabled browser like Chrome and Firefox nightly.
- WebVR support for Microsoft Mixed Reality Simulator and supported headsets using Microsoft Edge browser.
- Support for Samsung Gear VR is through the Samsung Internet or Oculus VR Browser which also supports WebVR after being enabled.
- Google Daydream View support is available with Chrome for Android with the Daydream app installed.
- Cardboard mode support for non WebVR browsers is available using a customised reduced version of WebXR Polyfill. This is optional and loaded on demand.
- Full WebXR display events provided.
- Skeleton Cordova application supplied for IOS and Android based apps. Cordova WebView apps are required to work around IOS Iphone inline video limitations.
- Support for IOS "pinned to Home Screen" standalone WebView apps for inline playback support on Iphone.
- Pseudo WebVR support for Cardboard uses mobile orientation controls in VR mode or uses WebVR controls if enabled in Chrome and Firefox on Android.
- Stereoscopic preview mode for desktop viewing.
VR headset toggle button to go into VR fullscreen / VR display presentation mode and back to normal mode.
When in WebVR headset presentation mode. A status splash page will be displayed.
- Mobile landscape orientation screen lock support for browsers that support it in VR mode.
- Android and IOS wakelock support to prevent the screen going to sleep in VR mode.
- Orbital navigation dock area for not only controlling on the video display itself with a mouse.
- Hacks required for mouse controls to prevent the player toggling the video.
- proxy sources for Html5 non cross origin standards compliant browsers like Safari, IE and Android default browser.
- Dock button for switching to the proxy source of the VR video.
- Stereoscopic side by side video support for 2D picture.
- Playlist support between non VR and standard videos to switch between rendering views.
- Pseudo ratio scaling of the WebGL rendered video. VR mode will scale to fit the screen.
- Works with any Html5 based streaming including Mpeg Dash and HLS.JS.
- Ambisonics surround audio support for rotating with the video.
- Mouse pointer lock controls features to rotate the camera with a mouse pointer rather than mouse drag controls.
- HLS support and fixes for Safari.
- Flat 2D video in VR Cinema screen mode support for VR headsets and Cardboard.
- Subtitle text overlay support for Cinema screen and 360 mode.
- VR Video controlbar support in WebVR presentation mode.
- VR Controller support to interact with the virtual controlbar.
- Dual 360 Fisheye and Single 180 Fisheye support.
- Pseudo fullscreen for IOS in normal and VR cardboard mode due to IOS not supporting fullscreen.
- IOS 13 orientation api permissions check on user gesture.
- IOS 13 fullscreen and cardboard orientation.
- Using the Snapshot Plugin, support is available to capture VR Video rendering.
- Picture in picture toggling support for VR rendering.
- IOS 13 requires a gesture to accept orientation api permissions. This happens on first user gesture.
- IOS before 13 lacks html5 video fullscreen api. A pseudo fullscreen in normal and Cardboard VR mode has been provided.
- IOS lacks screen orientation lock api when going fullscreen or pseudo VR Cardboard mode
- IOS lacks WebXR/WebVR api and requires the Cardboard polyfill to be enabled.
- Multi channel audio support works on most browsers apart from IOS. Therefore IOS requires a stereo audio source instead.
- Safari and IE 11 does not support pointer lock browser api's therefore the pointer lock controls will not display.
- HLS rendering in IOS has been fixed in IOS 11.3 and above.
- Bug fixed with HLS rendering on IOS due to dimensions not reported on metadata. This might be unstable still.
- Ionic Framework
- Apache Cordova
- Samsung Gear VR
- Oculus Rift
- HTC Vive
- Mozilla VR
- Google Spatial Media
- Windows Mixed Reality