Capture Meta Presence Platform with LIV

Presence Platform Recording, particularly Passthrough, is designed for Professional  Production Studios.

Creating the final composite involves post-production tasks such as composition, background segmentation, and using third-party software.

Supported games must be Quest games specifically built to run on PCVR, which isn't available publicly.

You'll get a composition of the foreground over the real life camera. Background composition will need to be done in post production.

Introduction - Updated August 26, 2024

At a high level, the process for recording mixed reality capture is as follows;

1. Setup your Space in the Quest OS
2. Start Meta Quest Link. Doesn’t work with Virtual Desktop or Steam Link.
3. Ensure Meta Quest Link is set as the active OpenXR runtime.
4. In the Meta Quest Beta tab, ensure Developer Runtime Features, Passthrough over Meta Quest Link, and Spatial Data over Meta Quest Link is enabled.
5. Start LIV app on Steam, using the default branch.
6. Start your Quest game on PCVR with LIV 2.0 SDK
7. Start LIV Mixed Reality Capture and manually select the game
8. Start SteamVR
9. Capture

If you prefer, you can view a tutorial video.

Required and Recommended Software

• LIV App, which can be found on Steam
• Meta Quest Link app for PC
• Your favorite Capture software, such as OBS, ScreenApp, or Movavi Screen Recorder
• As of June 26, 2024, it's required to use OpenVR Space Calibrator, if you're wanting to use a tracked camera.

Required and Recommended Hardware

• Camera that can connect to your PC for capture
• If you're using a tracked cameras
  • It's highly recommended to use a Quest Pro controller as it self-tracks and won't lose the calibration as easily
  • A SteamVR/Lighthouse compatible tracker to attach to your camera
  • Some mechanism to securely attach the tracker to your camera

Setup your capture environment, equipment, and software

To capture Presence Platform content you will need to make sure you have the following ready:

Quest Setup

• On your Quest 3 headset, your guardian & floor height as well as the playspace center and direction need to be set.
• The LIV app does not have recording capabilities, rather we leave that function to your preferred software, for example OBS.  Set up your capture software accordingly.
• You need to use the Meta Quest Link desktop app to connect to your PC. Note that capture does not work with virtual Desktop or Steam Link at this time. 
• Within the Meta Quest Link App
  • Ensure Meta Quest Link is set as the active OpenXR runtime.
    • 

• • In the Beta tab, ensure Developer Runtime Features,Pass through over Meta Quest Link, and Spatial Data over Meta Quest Link is enabled.
    • 
• Although not necessary, it's recommended to disable the proximity sensor of your Quest.  You can do this step within Meta Quest Developer Hub each time you go through this process.  You could also cover the in-headset proximity sensor.

LIV Setup

1. Install the LIV app on Steam
2. Install the LIV SteamVR driver. 
3. Use the default branch of the LIV App.
4. The app you're using is currently supported with LIV.  You find the latest known supported games on liv.tv/games
5. Ensure your physical cameras are correctly set up.
   • Static: Place your camera in a fixed location to get the desired framing of both you and your background.
   • Tracked: Secure the tracker to the camera.
6. Complete camera setup based on your filming setup.

Static Camera

Setup

1. Start LIV. Ensure SteamVR is not yet started.
2. In the LIV app's Mixed Reality Capture tab, launch the compositor by clicking "Start PassThrough MR"
3. Launch your Game/App.
4. Launch SteamVR.
5. Set up your camera within LIV.  In the Camera tab in the LIV app, click "Add Camera"
   • • Enable Meta Quest Camera Offset will be automatically enabled when selecting "Start Passthrough MR"
       
6. Select Video Camera as camera type, choose your device, and then preferred resolution.
   1. Press "Begin Calibration" and follow the on-screen instructions to align the virtual controllers with your physical ones.  For more details, you can follow the steps in the calibration guide.
   2. If you're using a green screen, adjust the keying settings to ensure a clean separation between you and the background.  You may have to play around here to get things just right.
7. Navigate to the "Capture" tab in the LIV app and under "Manual," select your app's .exe in the target section. You should now see your game in the LIV Output window with your camera's view
   • 
8. For mixed reality apps with passthrough capabilities:
   1. In the LIV app, under the "Output" tab, enable the checkbox to hide the background.  When you enable background view, all content that the game developer has identified to be hidden in passthrough mode will be black.
      
      • 
   2. Under the "Capture" tab, set the "Effect" to "Dump + Presence Platform." You won't be able to see the passthrough effect, but the areas that will be masked out in post-production will appear as a solid color, making the process easier.
      
      • 
   3. Your output window will now show 4 quadrants that can be used to composite a final scene.  The top right quadrant will show the foreground alpha layer (Magenta) and the background alpha layer (Blue) packed together in a single view
      • 
9. You're now ready to start recording.  In your preferred capture software, such as OBS, capture the LIV Output window as your source to begin recording.

Tracked Camera Setup

A tracked camera enables dynamic, engaging shots for your viewers. However, it requires a more complex setup, including non-Quest tracking through SteamVR.

You will need:

• A SteamVR supported lighthouse-based tracking solution attached to your camera.
• We highly recommend using a Quest Pro controller because it has self-tracking capabilities. This reduces the risk of losing calibration when the controller moves out of the headset's field of view.
• OpenVR Space Calibrator to align the calibration coordinates between SteamVR and Quest.
• Physical clamps to securely fasten controllers

We are experiencing a calibration offset issue between Quest and Steam tracking center coordinates, causing misalignment between the game controllers and headset.

This error can be minimized by making sure Quest's guardian space is as square as possible and ensuring both Quest and Steam have the same forward vector. If you do notice the alignment drifting you'll need to recenter on the Quest and run through calibration on LIV again.

A solution is in progress and will be hot fixed into the LIV App shortly. A new SDK is not needed. You can follow along in our Discord for further updates.

Thank you for your patience during this open Beta period.

Setup

1. On your Quest 3 headset, set the guardian, scene setup, and floor height as well as the playspace center and direction.  Meta's guide can be found here.
2. Start the Meta Quest Link PC app and connect your headset to your PC, using AirLink or Link Cable.
3. Start SteamVR.
4. Verify that the SteamVR playspace center and direction match those set on the Quest 3. If they don't, use the Meta/Oculus button to reset. Note that you cannot use the Quest's recentering feature after this step without starting over.
5. Turn on both Lighthouse trackers and place one Quest Pro controller in a clamp, ensuring the controller’s cameras are unobstructed. It’s crucial that the Quest Pro controller remains stationary during calibration.
6. A second USB-attached Lighthouse tracker, mounted on a Manfrotto Super Clamp and positioned optimally within the Lighthouse tracked area, will help keep its position if the primary camera tracker is occluded. This second tracker and clamp will also securely lock the tracker and Quest Pro controller together during calibration in OpenVR Space Calibrator.
7. With the OpenVR Space Calibrator program open, use the "Slow" or "Very Slow" calibration process.
8. Hold the clamp so that both tracker and controller track optimally throughout the calibration process.
9. Perform Tai-Chi-like moves with the clamp, slow and steady, including rotations and vertical movement, while moving across a wide area of the playspace. It is crucial to maintain perfect tracking throughout this process; you can observe the tracking quality through the headset while performing this process.
10. Once complete, check calibration by moving the clamp around the space and looking for any significant drift. Place a controller against the camera tracker to confirm calibration. Repeat the calibration process if necessary.
11. Clamp the calibration tracker somewhere with optimal Lighthouse tracking, and leave it there until a recalibration is required. This will help maintain the Lighthouse playspace in case the camera tracker becomes momentarily occluded.
12. Start the LIV app on Steam
13. Calibrate the LIV camera by following all the steps in the calibration guide.
14. You're now ready to start recording. In your preferred capture software, such as OBS, capture the LIV Output window as your source to begin recording.

With tracked cameras, you'll eventually encounter 'drift' which is the calibration over time getting out sync.  It's unavoidable due to using the Quest and Steam tracking systems in tandem.  You can follow these best practices to reduce the frequency and effects.

With tracked cameras, you may experience 'drift,' where calibration gradually falls out of sync due to using Quest and Steam tracking systems together. While some drift is unavoidable, following these best practices can help reduce its frequency and impact.

• Do not recenter the playspace or have your device go to sleep after you have calibrated.  Recentering will change your coordinates and your calibration alignment won’t be accurate anymore.
• If you need to change your Quest space you will have to redo the entire process.