Learn how the app creators translated cinematic moments from The Mandalorian Season 1 into interactive vignettes beamed directly to your surroundings.
The Child’s hovering pram is levitating inside my kitchen. With a tap of my finger, the Mandalorian unlocks the protective shell to reveal the bounty hiding within. And even though I know it’s augmented reality, the product of the new Google Pixel The Mandalorian AR Experience beaming this tender moment into my home, I can’t help myself. After examining the scene from every angle, I finally step between bounty hunter and asset to stare into the eyes of the Child for myself, his one small hand reaching out toward me through the screen.
A few minutes later, Grogu has toddled into a new scene just a few feet from my desk. Like a diminutive boss checking on my progress for the day, he stands gazing in my direction, then sips at the cup of bone broth clasped between his small green hands.
After all, there is work to be done. Recently, StarWars.com sat down with Johannes Tacskovics, senior manager, Disney Games and Interactive Experiences, and Matthieu Lorrain, the head of creative, AR Partnerships, for Google, to find out how the tech wizards behind the new app made it possible to step inside some of our favorite moments from the first season of the Disney+ series. Inside the experience, you get to play the part of a bounty hunter on the trail of the titular Mandalorian.
StarWars.com: Take us back to the beginning of this AR experience. What inspired you to tell this particular story, of a bounty hunter on the trail of the Mandalorian and the Child, witnessing different moments from Season 1?
Matthieu Lorrain: When we came with the early app concept, it was very important for us to give a narrative purpose to our fans. We wanted to give them the feeling that they were stepping into the world of The Mandalorian and were part of this universe. Pretty quickly we started to see a parallel between augmented reality and the hologram pucks used by bounty hunters in the show. This naturally led us to this idea that the users could actually be bounty hunters looking for the Child and Mando. It felt consistent with The Mandalorian story and added an interesting twist to the experience.
Johannes Tacskovics: We're huge fans of the show and wanted to think about how we could bring fans into the experience in a groundbreaking way. We thought that the idea of a "bounty hunt" would connect with fans and also give them clues to better enjoy all parts of the AR experience. We designed this experience to be fun, so we wanted it to be easy to create your own shareable photos. We wanted fans to revisit all these great moments from Season 1, so we thought if we imagined them as holograms, it would be an intuitive way to explore.
StarWars.com: Beyond the launch of Season 2, what made this year the right time to launch this experiment? Were there breakthroughs in AR or 5G that helped make this come together now as opposed to a year ago?
Matthieu Lorrain: We’ve used a combination of technologies that when brought together enabled this next-generation AR experience. That mix was truly unique as we used some technologies that were only a few months old.
We benefited from targeting only the latest generation of 5G smartphones and their usually more powerful GPU. By focusing on this smaller set of devices, we were able to push the limit of the graphics and scene composition with custom optimization. In addition, we leveraged the recent ARCore Depth API, which allowed us to better adapt the experience to the user’s environment with impressive effects such as occlusion. Finally, 5G is a bonus here allowing users to download the different AR drops faster than with a regular connection.
Johannes Tacskovics: It was awesome that we could focus on the latest mobile hardware to really show how far we could push the technology and interactive experiences. We wanted to set a high bar.
StarWars.com: Once you map your space, you can choose from five different vignettes, with a new one unlocked each week. How did you choose the moments you wanted to incorporate into this experience? Do you have a favorite?
Matthieu Lorrain: We spent a lot of time researching season 1 of The Mandalorian, watching countless times and studying the different chapters. Our goal was to identify the most iconic scenes that would both be great for augmented reality but also get fans excited about the past season. We even looked at online memes around the Child to better understand what resonated the most. Some moments were pretty obvious such as the Child slurping soup. Others are less expected but bring a very unique perspective. Something very important for us is also to think about the surroundings and how it would come to life. The Mandalorian AR Experience is indeed the first app experience. We wanted the fans to literally step into the world of Star Wars, not just bring it to life in their living room.
StarWars.com: Let's talk about the experiences. We started with "Bounty Tracking," in which you watch Mando tracking his prey on a snowy backdrop. I appreciate that you've incorporated the visual cues as well as sound effects, including the winds! Since AR incorporates the user’s actual surroundings, how do you strike the balance between the layers you're adding to make the AR feel tangible while still keeping enough reality in view to make it feel genuine?
Johannes Tacskovics: We strive to make the characters and environments as authentic as possible. Making sure we’ve got the details right like the reflection of the snowy sky in the Mandalorian’s helmet, or the type of ice he’s standing on are all elements that make the AR feel more tangible. Setting the correct lighting also contributes to a believable scene, so that the moment in AR is authentic to the original moment in the show. In terms of striking the balance, we worked closely with Google to blend the scene lights with dynamic lights from your own reality. This meant we could get the best from both worlds for that perfect photo. The AR environments themselves adapt and scale to your world, so if you’re in a hallway, or a backyard, what you see scales to fit your space and that’s another great example of how this can serve both worlds in a way to make a more genuine experience overall.
StarWars.com: Then of course, "the Child" shows up and we get to watch Mando meet Grogu for the first time all over again. Each one is interactive in its own way, but this one adds an element we didn't see in the series and puts you in control of making the hover pram float. When you're designing an experience like this, how do you identify these intuitive, organic moments where you think people will want to reach through the screen and interact with the experience by moving the story along?
Johannes Tacskovics: This is the first of the Google AR experience drops where we see “the Child,” and when we took our first pass in AR, it quickly became obvious that everyone walked straight to the hover pram to take a closer look [inside]. We used these natural moments of curiosity to push the interactivity forward, so when you move closer, the hover pram reacts and rises. After a user has experienced the main moment for each interactive experience, we use sound and other cues to encourage users to explore the environment where they can find delightful references to the original story.
StarWars.com: In "Sanctuary" we get to interact with the Child having a nice cup of bone broth. In the series, this moment comes after an intense battle between Mando and Cara Dune. How did you translate part of the scene for the experience and what makes it work so well in AR?
Johannes Tacskovics: The huge internet and fan reaction to this scene from Season 1 meant we knew this is something we needed to include. It has a very cute moment from the Child, a lush environment in the form of the planet Sorgan, and with AR we could try something new. AR is great for bringing fans into the experience in front of the camera, so we thought it would be fun to put the Child center stage and make it easy to create great photo moments. Everyone has their own “sipping tea” moment so it’s going to be fun to see where fans take this one.
StarWars.com: And "The Force" just dropped today. What can fans expect from this latest chapter and the final installment coming next week?
Johannes Tacskovics: “The Force” goes back to a big moment in Season 1, and is unique because there is more action to this installment than in any of the others. You’re going to see a lot more visual effects included. All about pushing the mobile and AR technology, and creating something very cool.
For the final installment of the AR experience, I would encourage them all to re-watch Chapter 8 of the show.
StarWars.com: The hyper-detailed models feel so realistic. What was the biggest challenge in translating these characters and locations that fans know so well by now into AR and making them feel like they're part of our real world?
Johannes Tacskovics: Getting the Child right was the hardest. We worked closely with Industrial Light & Magic for the models and with Lucasfilm on getting the character experience right. It was very important to ensure that the Child's personality came through and that what was captured for the AR experience matched what Star Wars fans saw in the show. We had so many creative sessions to make sure our artists and animators were capturing the Child's expressions and personality just right. I think if you get the character right, that’s how they feel part of the real world.
StarWars.com: Do you have a favorite?
Matthieu Lorrain: My favorite moment? It is a very difficult question as we all put so much love into every single experience. But if I really had to choose, I’d go with drop number four, “The Force.” It is packed with action and beautiful effects and a direct tribute to the Jedi powers.
Johannes Tacskovics: My personal favorite is drop number one, “Bounty Tracking.” The drops were developed in the order they appear, so this was the first one where the team had that collective "wow" moment seeing the experience for the first time on a mobile device. We knew we were on to something. We learned so much, including how to work collaboratively and quickly with Google. Getting the first one right can often be full of challenges. Solving them together is what makes it fun.
Check out The Mandalorian AR Experience App now.
