Human Machine Teaming in Games

Team Size ( 12 - across Carnegie Mellon University and Georgia Institute of Technology )

Carnegie Mellon University - Producer, Programmers, Technical Game Designer, Artist, UI/UX Designer,

Game Subject Matter Expert.

Georgia Institute of Technology - AI researchers, AI Subject Matter Expert, Other Stakeholders.

Role

Research Assistant - Producer ( CMU Team )

Platforms

Unity 3D, WebGL, Photon Engine.

Tools

Google Docs, Google Sheets, Miro, Jira.

Duration

May 2023 - August 2023 (4 months) - Full Time

Jan 2024 - Present - Part Time

Challenge

To explore the future of Human-Machine Teaming (HMT) by developing natural, task-general human-guided machine learning capabilities for futuristic HMT scenarios in games - Through the creation of various online multiplayer cooperative games where teammates can include both humans and trainable AI agents.

Why does it matter?

Even though some AI systems are vastly more capable and often outperform humans in a variety of situations including games, they generally lack an understanding of forming teams with humans on complex tasks. By employing the Speculative Design lens and creating asymmetry in information to achieve a shared goal, we can explore if and how HMT works.

A detail breakdown can be found in the research paper.

Speculative Game Design of Asymmetric Cooperative Games to Study Human-Machine Teaming

What is Speculative Design in a nutshell?

It is best defined in the aforementioned paper as "A process of designing a collection of games, to probe the space of possible, plausible, and probable futures to identify those that are preferable."

Credits: "Speculative Game Design of Asymmetric Cooperative Games to
Study Human-Machine Teaming"

Here's another paper explaining speculative design in greater depth: https://dl.designresearchsociety.org/cgi/viewcontent.cgi?article=1253&context=drs-conference-papers

What did we make?

I collaborated closely with the game designer and games subject matter expert from the CMU team to brainstorm three distinct game concepts, each tailored to test Human-Machine Teaming (HMT) in varied sandbox environments.

Dice Adventure

( Turn-based adventure game )

Hackathon Page

Description

" Dice Adventure is a turn-based dungeon crawling adventure game, in which players take on the role of a dwarf, giant,
and human who must collaborate to navigate through maze-like dungeons to reach a goal. Each of the dungeons is filled with challenges such as traps and rocks that have to be broken and monsters that have to be fought by rolling dice. The different characters have asymmetry in their abilities to address each of these challenges (e.g. the giant’s available die cannot roll values capable of disarming traps, while the dwarf is similarly unable to break rocks) as well as their movement speed and ability to see through fog of war. Players must leverage their complementary abilities to navigate the dungeon and compensate for challenges that they are individually ill-suited for. "

Credits: "Speculative Game Design of Asymmetric Cooperative Games to
Study Human-Machine Teaming"

Dice Adventure Hackathon (Dates to be announced):

The Dice Adventure Hackathon is an opportunity for:

AI developers to train and submit their AI agents and win rewards based on its performance.
Other participants to play with other players across the globe and submitted AI agents

My Contributions as one of the hackathon organizer

Working closely with stakeholders to brainstorm and design competition logistics, leaderboard structure, and server architecture.
Collaborating with AI researchers to design an AI agent submission pipeline for the upcoming competition.

Cosmic Heist

( Physics-based Real-Time Strategy game )

Cosmic Heist Gameplay, where the Python GUI is used to send commands to a custom Web Socket-based thread-safe Unity interface. Both GUI and interface were developed by me.

Play Cosmic Heist on itch.io!

Description

Cosmic Heist" is a cooperative physics-based action game set in space, where players command a spaceship with the unique ability to tether to space asteroids. Divided control over navigation, translation, and tether management is spread across different rooms, providing precise spaceship operation. Collaboration is key as players team up to swing asteroids into enemy ships. However, caution is advised, as collisions with other asteroids can damage and ultimately destroy your spaceship.

Vertical Farm

( Puzzle game )

Vertical Farm is currently in the Brainstorming stage.

Description

Vertical Farming is a cooperative alien plant farming game with cascading effects on a 3D grid, creating complex game states meant for AI agents to analyze and create courses of action (COA) with. The COA generated by the agents are then passed to a human player who is overseeing the whole vertical farm, forming a team dynamic that plays on the strengths of AI and humans.

Design breakdown by Game Designer

How did I help?

( Contributions in developing all games and the project as a whole )

After some friendly introductions and chatting about our project goals and favorite places to eat in Pittsburgh, here's what I put into action:

1. Breaking The Ice, Goals, and Expectations Management

Contributions:

Organized 1 : 1's with each individual team member to understand their goal and expectations from the project.
Arranged team bonding meals and game jam sessions to foster communication, facilitate rapport building, and maintain momentum within the team.
Conducted Q&A-style meetings with stakeholders and subject matter experts to address any initial queries team members may have, ensuring clarity and alignment from the outset.

2. Literature Review

Contributions:

Conducted literature review sessions with the CMU team to collect and maintain research references on Speculative game design, Human-AI teaming frameworks, and AI in Games.
Collaborated with the stakeholders and subject matter experts to identify 3 pillars of focus:
- Actions and their consequences
  - Human and AI agents should learn the actions their game characters can execute in-game along with their consequences.
  - AI agents will know the actions to perform, it has to figure out when to perform each action individually and as a team to achieve the shared task.
- Communication and Coordination
  - Human and AI agents should As any cooperative game learn how to communicate in-game to achieve the shared task.
  - Any cooperative game requires a communication and coordination layer so this is a crucial requirement.
- Dealing with complex information in a changing environment
  - Human and AI agent should learn how to adapt to changing environment with complex information individually and as a team to achieve the shared task.

3. Game Design, Development and QA

Contributions:

Led collaborative brainstorming to develop game concepts and mechanics aligned with project goals, working closely with subject matter experts and designers, with decisions made through team consensus.
Created and managed multiple roadmaps with multiple milestones between CMU (Game Dev) and Georgia Tech (AI research) teams.
Created and managed Jira product backlog.
Facilitated weekly discussions between CMU and the Georgia Tech team to align design constraints, technical requirements, expert feedback, and feature prioritizations.
Worked collaboratively with the team to create comprehensive Technical documentation, Design documentation, Rulebooks, Game tutorials, and Demo videos.
Developed a custom Web Socket-based thread-safe Unity interface from scratch for AI-agent communication, collaborating with the lead AI researcher and lead game programmer.
Created a Python GUI for testing AI-agent communication with Unity, improving QA test time by 30%.
Organized multiple playtest and QA sessions with both the CMU and Georgia institute of tech team, followed by comprehensive bug triage and prioritization.

4. Development Plan and Meetings Structure

Contributions:

A high level view of the weekly development plan and meeting structure I executed to ensure alignment across the entire team.