In the summer of 2016 Valve software brought Punch Drunk on board the “VR Experience” at “The International 6” DOTA2 competition. The VR Experience featured 18 VR play areas where attendees could drop by for a 15-minute VR demonstration.
Five of these play areas were dressed with chroma-green backdrops, lights, and mixed-reality compositing systems that we designed, setup, and ran for the week. While players were in the VR world their friends and family could watch them in realtime on another screen and see the same virtual world. Check out the video to learn more about how we setup this massive multi-suite mixed reality VR experience:
Transcript of Jacob Stone’s interview:
“We explored a software fork and a hardware fork with how to do this mixed reality compositing. We ended up landing on this software fork that we developed; a custom workflow for it that does a real-time composite of the game player and the game they’re in.
To make that all play nice, we ended up writing a custom Unity application which does all the camera configuration. All of this took place in about a seven week time frame from the outset of the project to now, implementing this for a six day show run, eight to ten hours a day streaming mixed reality in five rooms on twitch.
In the VR experience tent here at TI6 for DOTA we’ve got 18 rooms of virtual reality. Five of those rooms are done up in green screen. The players come in and instead of a checkered background, they’re in a full green screen that we did lighting for like you would do in a film set for green screen.
Then we put a camera in each of those rooms that films the person in the room and then we’re able to do this live compositing where we insert the person into the game as it’s being played in real-time and then send that out to monitors.
When you walk by the room where somebody is playing in a mixed reality room, you actually see them in the game.
It’s very difficult to tell someone, “VR is cool. Trust me, you’ve got to try it.” If you just see a picture of somebody wearing a headset, that’s not a very compelling thing, but if you can see the person in the environment of the game and you can understand that that person is experiencing a completely new world, and as a third party you can watch them in that and you can watch them in real-time, you’re able to much more quickly much more fully engage with what this whole VR thing is.
We’re able to show that to people that are friends and family that are coming with the attendees and the players here, and then stream that online. It’s like a shortcut for anybody to understand, “That’s what VR is. That’s why it’s cool. That’s why it’s important.”
In addition to performing the real time composite of all these five green screen rooms and ISO recording that, we’re sending that to a series of 80 inch TVs that are located outside the tent so any passersby can see what’s happening in the tent.
For this particular job, we brought in a lot of different disciplines to make this work. At our core foundation we’ve got myself and my head engineer, Brian. We took the project and said, “From a video engineering standpoint what do we do?” The basics got figured out.
Then we brought in TJ who is just an amazing visual artist and a touring VJ and we said, “TJ, bring your thoughts to bear on how you would engineer a live software-based solution to do this compositing piece.”
We brought in Derek who is also a video engineer with a couple decades experience. We brought in a film gaffer, Michael. Michael came in and said, “Okay, here’s the best way that we can light this environment to give you bang for the buck and to give you a perfect key on everybody in the green screen.”
We’re drawing from a lot of different disciplines in Punch Drunk. On top of that, we’ve got a documentary team here shooting interviews and shooting behind the scenes of people in these games so that after the fact, anybody can take a look and see this is what it is. This is what it is to play a VR game.
We not only came together to execute what Valve was looking for, but we built a repeatable workflow. You can expand that scale into 10, 15, 50 because we built a solution not just for now, but a solution that can be used for all sorts of future events.
We at Punch Drunk are stoked to be part of this as it’s in its infancy. I absolutely see us staying part of VR and mixed reality as it continues to grow. From this experience I imagine we’re going to take a pretty active role in helping develop some of the technology that allows mixed reality and VR to be a common thing.”