HoloLens & Mixed Reality Master Class
You will leave this workshop understanding:
Together, we’ll create an interactive recipe application that walks people through the process of cooking a good meal of spaghetti. The project will involve recipe steps, holographic indicators for ingredients’ locations, read-out-loud instructions, and more!
At the end of the experience, you’ll have the satisfaction of having created an amazing training-based HoloLens 2 application and as a bonus, you’ll also eat a good serving of Spaghetti!
*We will be adding small features of the cooking project each week and completing on the 6th week
In the first week of this course, we’ll be diving into how to get started with HoloLens 2 development. We’ll cover prerequisites, Unity and Visual Studio installation, setting up your HoloLens 2 and setting up your development environment. By the end of Week 1, you’ll build and deploy your first HoloLens 2 application using the OpenXR pathway within Unity.
We will also be covering Microsoft’s Mixed Reality Toolkit (MRTK.) The MRTK is Microsoft’s premiere toolkit for streamlining mixed reality development. The MRTK is a vast resource that needs several weeks to cover. However, we’ll be covering the top features over 2 weeks during this course. This week, we’ll be covering various input methods such as hand, voice, and eye tracking. We’ll also cover spatial awareness and touch on scene understanding. As part of learning these concepts, you will also begin building out what will eventually be the capstone project’s cooking application.
During Week 2, we’ll continue with the MRTK. We’ll cover MRTK’s Grid Object Collection, Tool Tips, and also show you how you can leverage the MRTK to add interactive objects such as videos into your scene. We will also touch on spatial audio and various solvers included with the MRTK.
Learn how to use Spatial Anchors. We’ll learn how to anchor the objects in mixed reality to experience the real world; you will also explore the various steps required to start and stop an Spatial Anchors session and create, upload, and download Spatial Anchors on a HoloLens 2.
This week will focus on using the Azure Speech SDK as part of mixed reality application development. The Azure Speech SDK leverages the power of cognitive services and artificial intelligence on the cloud for speech processing. There’s a wide array of possible capabilities that the SDK provides, including natural language processing, intent understanding, keyword recognition, speech-to-text services, text-to-speech services, translation, and more! This week, we’ll focus on text-to-speech services that the SDK offers in order to bring our cooking recipe instructions to life with a realistic voice that dictates the contents of the recipe cards.
During Week 5, students will be introduced to the next big version of MRTK, which is expected to be released at the end of 2022 or early 2023. MRTK3 integrates deeper with OpenXR, is more modular, and includes an enhanced set of UI and UX elements to streamline development.
During Week 6, students will dive deeper in the concepts and methods to begin creating the kitchen recipe HoloLens experience. Students will create tool-tips for ingredients and key locations, edit the labels of the tool-tips, and create a “pin” button to place spatial anchors for tool-tips. Students will place the finishing touches of the kitchen recipe Ho-loLens experience. Students will add cooking steps and timers, and create the ability for users to navigate through the recipe.