VR Chinese Learning with Speaking and Zombies
My CCS 1L Winter 2025 Project. A VR game to learn Chinese, except it's in a zombie survival game. Convince Duolingo to give you weapons by speaking in Chinese.
This is a project I've been thinking about for a while, but probably not possible until recently. I wanted to make a fully immersive Chinese learning game so that I could finally catch up on the 8 years since I've dropped out of Chinese school. It's a decision I still kinda regret today (really, my Chinese speaking is utterly embarrassing).
But with ChatGPT Realtime Conversations, this is totally possible! And, it'll be a totally novel game since the technology is so recent.
The Game
To be honest, I don't have the manpower to make it an open world game with all the features that I wanted (like bargaining in Chinese or asking for directions to the nearest shop). For my project proof-of-concept, I instead opted to give rewards to players when they correctly spoke good Chinese. For example, in a Zombie game, rewards could be ammo or new weapons.
You can see the demo below. At the beginning, when I asked for ammo, it actually clipped through the ground as it spawned in... so that's a little awkward.
ThePickleGawd
Building the Game
I spent a long time watching Youtube tutorials on Unity XR since that was all new to me. But if you've known me for a while and saw the other study game that I made back in 2021, you might recognize some of the assets. Actually, the zombies and city are literally the same.
You may be skeptical, but hey, I aced my English vocab quiz after playing this game for a few hours. And there were like 40 words on that test.
But anyways, after some slight modifications (and the Duolingo bird of course, though for a while we thought he was dead), We have a pretty basic scene.
The gun and player scripts are pretty standard so I'm not going to dive deep into them. I basically just ripped the same Gun, Health, Player script from all my other games and it worked just fine. The new feature was adding realtime speaking with ChatGPT.
Talking to ChatGPT
To give a high level overview of how this works, let's look at this amazing diagram I made for my presentation.
Basically, I have an intermediary Node.js server to handle communication from the VR game client and OpenAI servers (cuz writing C# code for their API sounds like a pain in the ass). It will start a WebSocket server and simultaneously connects to OpenAI via WebSockets.
- When user speaks, pipe the the audio to the server
- Server collects audio buffer and streams it to OpenAI, following their realtime API protocol.
- Server waits for OpenAI response. Then pipes audio back to VR client
- VR Client plays any audio it receives.
Here's an example:
Giving Player the Rewards
To actually reward the player in the game, we need some way for ChatGPT to communicate that it thinks the player deserves a reward. To do this, I relied on OpenAI Function Calling protocol. Basically, I define a function schema that accepts a "reward" parameter and I give it a description so ChatGPT knows when to call it.
On the Node.js server, I check if I receive a "function_call" from ChatGPT, and if so, I send it to my Unity client as a JSON message. I have similar code in C# to check whether the received WebSocket data is JSON or not.
Conclusion
And that's it! If I have more time, I really want to turn this into a stealth game, where you have to blend in with enemies by speaking flawless Chinese (otherwise they'll discover you and shoot you). That'll make the conversation aspect much more immersive and intense. It can also be personalized for based on what the player actually cares about learning, like speaking about business, for example. There is so much customizability since we're using AI NPCs.
Anyways, hope you enjoyed. You can try playing it if you have a Meta Quest (check GitHub link for .apk file).