I vividly recall my first encounter with Microsoft Hololens, playing Roboraid in my living room. This wasn't just any game; it was an extraordinary fusion of the physical and digital worlds. The game mechanics, requiring players to gaze and point at alien robots seemingly emerging from the walls, introduced me to a new dimension of gaming - Mixed Reality. The way Hololens mapped the room's walls and spawned enemies created an illusion so real it was mesmerizing. This experience inspired me to explore the world of Hololens app development.

The announcement of a Hololens hackathon at Khalifa University by Microsoft UAE was a golden opportunity I couldn't miss. Joined by enthusiastic teams from across the UAE, we embarked on a 48-hour quest to develop a groundbreaking application. Our team, humorously named "Space Devs," was ready for this exhilarating challenge.

After brainstorming, our vision was clear: to bring the experience of zero gravity to earth. We aimed to create an educational experience that lets users feel like astronauts, understanding the peculiar behavior of objects in zero gravity. Our project included a puzzle game for assembling a space rover, a fluid simulation that allowed users to interact with floating liquids, and a fire simulation demonstrating how flames become spherical without gravity. The finale was a demonstration of how gravity varies across planets, shown through the impact on a car when objects are dropped from different heights.

Developing our prototype was a test of endurance and creativity. I still remember the awe I felt interacting with the simulated fluid in zero gravity - it was surreal. Our final level, showcasing gravity's varying effects on different planets, was particularly satisfying, especially watching the devastation caused by Jupiter's intense gravity.As the hackathon neared its end, our presentation emphasized the potential of mixed reality in education. We advocated for how this technology could provide students with experiences that are nearly impossible to recreate in real life. The tension during the award announcement was palpable. When "Space Devs" was declared the winner, it was a moment of disbelief and joy. Standing on the stage, and receiving our awards, was not just a triumph but a validation of our hard work and creativity.

This hackathon was more than a competition; it was a journey of discovery, learning, and inspiration. It solidified my passion for XR (Extended Reality) and motivated me to delve deeper into this fascinating field. The experience taught me that mixed reality is not just about technology; it's about opening doors to new worlds and possibilities, transforming how we learn, experience, and understand our world.

As we stepped into 2023, the landscape of virtual and augmented reality (VR/AR) was already burgeoning with potential. Numerous game engines had begun to support VR/AR development, and OpenXR had emerged as a universal standard, promising a seamless experience across various devices. Yet, the true epiphany for many, including myself, came with the discovery of WebXR - the gateway to an effortlessly immersive world of AR/VR on the web.

Skepticism towards 3D applications on the web was prevalent due to concerns about rendering capabilities and network latency. But the moment I delved into WebXR, my doubts dissipated. The ease of entering an AR/VR session - simply typing a URL into a browser - was nothing short of magical. At that moment, the world of augmented capabilities unfurled before me, and I realized the vast potential of this technology.

On a study abroad program in New York, I stumbled upon a life-changing opportunity: a WebXR hackathon organized by Meta and AWS in Seattle. Accompanied by my friend Utsav, we embarked on this journey, eager to explore the frontiers of WebXR. Meeting Rick from Meta, who spearheaded their WebXR efforts, was particularly enlightening. His enthusiasm and vision for how VR/AR could revolutionize fields like design and architecture were infectious. It was clear: we were on the cusp of a new era in computing.

The hackathon was an exhilarating whirlwind. Forming a team with the talented Simon and the adept product manager Rohan, we were tasked with creating a WebXR application using Meta's latest mixed reality features and AWS cloud services. Our idea? A mixed reality word puzzle game designed to aid dementia patients in exercising their cognitive skills and mobility. The development process was fraught with challenges, yet it was a journey of immense learning and collaboration.

As the event culminated, our team, brimming with nervous excitement, presented our prototype. The experience of sharing our vision, how our application could positively impact dementia patients, was profoundly rewarding. We may not have won, but we left with a profound sense of hope and determination.The hackathon was more than a competition; it was a glimpse into a future where WebXR technology is ubiquitous. The event not only showcased the remarkable capabilities of spatial computing but also ignited a passion in us to be at the forefront of this technological revolution.

In conclusion, the WebXR hackathon was not just an event; it was a pivotal moment that reshaped our perspective on AR/VR technology. It reinforced our belief that we are entering a new era of computing, where spatial web and Web 3.0 will transform how we interact with the world around us. This journey into WebXR was more than inspirational; it was a clarion call to be part of a future where reality is only the beginning.

During my summer research, I worked at the intersection of virtual reality (VR) and artificial intelligence (AI) to enhance communication in remote collaborative environments. The primary focus of my research was to explore how large language models (LLMs) can be leveraged to detect and interpret gaze cues from speech, aiming to improve the effectiveness of communication in virtual reality settings.

The objective of my research was to understand how natural conversation dynamics could be influenced by the integration of implicit gaze visualizations within a VR environment. By utilizing LLMs alongside shared gaze visualizations, we aimed to create more immersive interfaces that support seamless communication without relying on traditional methods such as detailed descriptive language, complex gestures, hand pointing, or sketching annotations. This approach holds the potential to significantly enhance collaboration by reducing the need for manual activation or rendering of communication cues, thereby streamlining the process of directing attention between remote users.

I was responsible for developing the entire software pipeline, from fine-tuning the LLM to suit our specific task, to developing a networked application using the Unity game engine. Initially, I applied natural language processing (NLP) techniques, such as entity recognition systems, to detect deictic language in conversations. However, this approach did not fully capture the context of communication, resulting in suboptimal outcomes.

To address this, I fine-tuned an LLM known as LLama-8B using parameter-efficient techniques like QLoRA. Due to the limited availability of datasets for detecting deictic language, the fine-tuned model was trained on a relatively small dataset and repurposed for binary classification. The model was trained to recognize phrases such as "Can you look at that?" or "Do you see that?" and subsequently activate visual cues to direct the attention of the other user.

In addition, I developed a networked multiplayer application using Unity's Netcode, enabling users to create rooms, join sessions, and interact as virtual avatars with head and hand tracking. I integrated OpenAI's Whisper model to convert speech to text, allowing the LLM to process the conversation. Whisper's voice activity detection feature automatically paused transcription when no activity was detected, enhancing the system's efficiency.

I chose the Oculus Quest Pro for its eye-tracking capabilities, enabling me to render a line from the user's eye to the object they were looking at. In the VR environment, if one user intended to direct another user's attention towards a specific object, the LLM would detect the deictic language used and activate the visual cue, effectively guiding the second user's focus.

To improve the accuracy of LLM predictions, I implemented a sentence tokenizer, ensuring that only complete sentences were passed to the model for processing. This approach minimized errors that could arise from incomplete sentence analysis. Additionally, the LLM was deployed locally within the Unity game engine to eliminate latency issues associated with cloud-based processing.

While the research is ongoing, with user studies yet to be conducted, our preliminary findings suggest that this integration of LLMs and gaze cues could facilitate more efficient and intuitive remote collaboration. By automating the transfer of communicative intent between users, we hope to enhance task efficiency and foster faster, more natural collaboration in mixed-reality environments.

Have you ever felt that rush of excitement when you see technology pushing the boundaries of what you thought was possible? That's exactly how I felt when I witnessed the release of Quest 3 and its passthrough and room scanning technology. Suddenly, the world of XR games and applications opened up before my eyes, and I knew I had to be a part of it. Every time I scrolled through LinkedIn, I found myself mesmerized by the innovative game mechanics and XR features being developed. It was like glimpsing into the future, and I was determined to be a part of shaping it. Despite the demands of university coursework, I made a bold decision: I would create my own XR game, even if it meant sacrificing a few grades along the way.

Armed with the Meta Quest SDK and fueled by my passion for XR technology, I embarked on an exhilarating journey of game development. With the SceneAPI providing the geometry of the physical room, all that was left was to weave together a gameplay experience that would showcase the full potential of XR features. And thus, my vision took shape: an XR space shooter game where players must destroy alien drones and bots hiding behind their own walls. The twist? These drones could shoot from behind the walls, forcing players to strategically make holes to reveal and destroy them. It was a concept that merged reality and virtuality in an immersive way, and I was determined to bring it to life. Utilizing stencils and shaders, I created the illusion of walls cracking as players blasted through them, revealing the virtual world behind. But I didn't stop there. I wanted to push the boundaries even further by experimenting with mixed reality enemies, like spider bots crawling menacingly along the walls, blurring the lines between the real and virtual worlds.

After weeks of relentless grinding, debugging, and polishing, my XR game was finally ready. The moment of truth arrived at our university's end-of-semester exhibition, where I proudly showcased my creation. With two walls set up to simulate a room, attendees were plunged into the heart of the action, battling waves of drones and spider bots in an adrenaline-fueled frenzy. The response was overwhelming. People were captivated by the immersive experience, and I was filled with pride seeing them locked into the game, pushing their skills to the limit. It was a small victory, but it fueled my determination to continue exploring the possibilities of XR technology.As I look back on this journey, I'm filled with excitement for the future. With XR technology evolving at a rapid pace, the possibilities are endless. And while this may have been just the beginning, I'm already looking forward to embarking on even bigger XR projects, pushing the boundaries of what's possible and shaping the future of gaming.