BETA - Capture Meta Presence Platform with LIV

Presence Platform Recording, specifically Passthrough, is currently in Beta and is intended for Professional studios at this time.  Creating the final composite requires post production  composition, background segmentation, and 3rd party software to create a composite.

 

At this point in the Beta, you'll get a composition of the foreground over the real life camera.  Background composition will need to be done in post production.

LIV-DumpPPeg-Dump.png

 

At a high level, the process for recording mixed reality capture is as follows;

  1. Setup your capture environment, equipment, and software
  2. Connect your Software to each other
  3. Calibrate your camera and refine tracking
  4. Record your capture with a 3rd party software

If you prefer, you can view a tutorial video.

 

Required and Recommended Software

Required and Recommended Hardware

  • Camera that can connect to your PC for capture
  • If you're using a tracked cameras
    • It's highly recommended to use a Quest Pro controller as it self-tracks and won't lose the calibration as easily
    • A SteamVR/Lighthouse compatible tracker to attach to your camera
    • Some mechanism to securely attach the tracker to your camera

 

Setup your capture environment, equipment, and software

To capture Presence Platform content you will need to make sure you have the following ready:

  • On your Quest 3 headset, your guardian & floor height as well as the playspace center and direction need to be set.
  • The LIV app does not have recording capabilities, rather we leave that function to your preferred software, for example OBS.  Set up your capture software accordingly.
  • You need to use the Meta Quest Link desktop app to connect to your PC. Note that capture does not work with virtual Desktop or Steam Link at this time. 
  • Within the Meta Quest Link App
    • Ensure Meta Quest Link is set as the active OpenXR runtime.
    • In the Beta tab, ensure Developer Runtime Features,Pass through over Meta Quest Link, and Spatial Data over Meta Quest Link is enabled.
  • Although not necessary, it's recommended to disable the proximity sensor of your Quest.  You can do this step within Meta Quest Developer Hub each time you go through this process.  You could also cover the in-headset proximity sensor.
  • Make sure you have downloaded and installed the LIV app on Steam
    • You have installed the LIV SteamVR driver
    • You're using the ‘public_beta_presence_platform’ branch.  Later in Q3 2024 we will release Presence Platform features to the main branch.  You can switch branches in Steam->LIV->Properties
  • The app you're using is currently supported with LIV.  You find the latest known supported games on liv.tv/games.
  • If using a static camera, place your physical cameras to correctly frame you and your background.  If using a tracked camera, ensure the trackers are attached to the camera.

 

 

The next step is dependent and your camera setup, whether you're using a static camera that isn't moving (ie: fixed on a tripod) or if you're using a tracked camera with dynamic movement.

 

Static Camera setup

  1. Start LIV.  Ensure SteamVR is not yet started.
  2. Via Mixed Reality Capture tab in the LIV app, launch the compositor by clicking Start PCVR Mixed Reality Capture
  3. Launch your app
  4. Launch SteamVR
  5. Set up your camera within LIV.  In the Camera tab in the LIV app, click "Add Camera."
  6. Select Video Camera as camera type, choose your device, and then preferred resolution.
    1. Press "Begin Calibration" and follow the on-screen instructions to align the virtual controllers with your physical ones.  For more details, you can follow the steps in the calibration guide
    2. If you're using a green screen, adjust the keying settings to ensure a clean separation between you and the background.  You may have to play around here to get things just right.
  7. Navigate to the "Capture" tab in the LIV app and under "Manual," select your app's .exe in the target section. You should now see your game in the LIV Output window with your camera's view.
  8. For mixed reality apps with passthrough capabilities:
    1. In the LIV app, under the "Output" tab, enable the checkbox to hide the background.  When you enable background view, all content that the game developer has identified to be hidden in passthrough mode will be black.
    2. Under the "Capture" tab, set the "Effect" to "Dump + Presence Platform." You won't be able to see the passthrough effect, but the areas that will be masked out in post-production will appear as a solid color, making the process easier.
    3. Your output window will now show 4 quadrants that can be used to composite a final scene.  The top right quadrant will show the foreground alpha layer (Magenta) and the background alpha layer (Blue) packed together in a single view.
  9. You're now ready to start recording.  In your preferred capture software, such as OBS, capture the LIV Output window as your source to begin recording.

 

Tracked Camera setup

A tracked camera allows you to create dynamic shots that may be more engaging for your viewers.  However, the more engaging capture does require a more complex set up and additional requirements due to requiring non-Quest tracking via SteamVR;

  • You will need a SteamVR supported lighthouse based tracking solution attached to your camera.
  • We strongly recommend using a Quest Pro controller as it's self-tracking, when moving the controller away from the field of view of the headset it reduces the chance of losing calibration.
  • You'll need to align SteamVR and Quest calibration coordinates with OpenVR Space Calibrator
  • Physical clamps to securely fasten controllers

As at June 26, 2024

We currently have a calibration offset issue between Quest and Steam tracking center coordinates. This offset results in the game controllers and headset not lining up exactly.

This error can be minimized by making sure Quest's guardian space is as square as possible and ensuring both Quest and Steam have the same forward vector. If you do notice the alignment drifting you'll need to recenter on the Quest and run through calibration on LIV again.

A solution is in progress and will be hot fixed into the LIV App shortly. A new SDK is not needed. You can follow along in our Discord for further updates.

Thank you for your patience during this open Beta period.

 

The process;

  1. On your Quest 3 headset, set the guardian, scene setup, and floor height as well as the playspace center and direction.  Meta's guide can be found here.
  2. Start the Meta Quest Link PC app and connect your headset to your PC, using AirLink or Link Cable.
  3. Then start SteamVR.
  4. Check that the SteamVR playspace center and direction are the same as set earlier in the Quest 3. Use the Meta/Oculus button to reset if not to fix this.  It's important to note that you can't use the Quest's recentering feature after this point without starting over.
  5. Turn both Lighthouse trackers on and put one Quest Pro controller in a clamp, making sure the controller's cameras are not occluded.  The key is ensuring the Quest Pro controller does not move during calibration.
  6. A second USB-attached Lighthouse based tracker mounted to a Manfrotto Super Clamp that is optimally positioned within the Lighthouse tracked area will help maintain its position in the event the camera tracker is occluded. This second tracker & clamp will also be used to securely lock the tracker and Quest Pro controller together when calibrating in OpenVR Space Calibrator.
  7. With the OpenVR Space Calibrator program open, use the "Slow" or "Very Slow" calibration process.
  8. Hold the clamp so that both tracker and controller track optimally throughout the calibration process.
  9. Perform Tai-Chi-like moves with the clamp, slow and steady, including rotations and vertical movement, while moving across a wide area of the playspace. It is crucial to maintain perfect tracking throughout this process; you can observe the tracking quality through the headset while performing this process.
  10. Once complete, check calibration by moving the clamp around the space and looking for any significant drift. Place a controller against the camera tracker to confirm calibration. Repeat the calibration process if necessary.
  11. Clamp the calibration tracker somewhere with optimal Lighthouse tracking, and leave it there until a recalibration is required. This will help maintain the Lighthouse playspace in case the camera tracker becomes momentarily occluded.
  12. Start the LIV app on Steam
  13. Calibrate the LIV camera by following all the steps in the calibration guide
  14. You're now ready to start recording.  In your preferred capture software, such as OBS, capture the LIV Output window as your source to begin recording.

With tracked cameras, you'll eventually encounter 'drift' which is the calibration over time getting out sync.  It's unavoidable due to using the Quest and Steam tracking systems in tandem.  You can follow these best practices to reduce the frequency and effects.

 

General notes

Do not recenter the playspace or have your device go to sleep after you have calibrated.  Recentering will change your coordinates and your calibration alignment won’t be accurate anymore.

 

If you need to change your Quest space you will have to redo the entire process.

 

 

Was this article helpful?
0 out of 0 found this helpful