Select Page
Helping engineers utilize augmented reality tools to assist with the design and assembly of spacecraft.
Services – Virtual Reality, Visual Design, UI/UX
My Involvement  – Lead UI Designer, UX Researcher, Lead Visual Designer
Date Completed – Late 2016
NASA JPL (Jet Propulsion Lab) Ops Laboratory is a national research facility that specializes in carrying out robotic space and Earth science missions. Engineers at JPL currently utilize augmented reality tools to assist with the design and assembly of spacecraft. These tools include Microsoft’s Hololens, a wireless augmented reality computing platform/headset developed by Microsoft, along with ProtoSpace, an augmented reality application developed by the JPL Ops Lab.

Below is a great example of the how of Engineers use Protospace as a tool to help examine spacecrafts before they are built

The Challenge: What can hierarchy look and feel like in ProtoSpace?

Use Case: A group of engineers are inspecting the arm of the Mars 2020 Rover model in ProtoSpace. They want to detach the arm and remove it from the rover so that they can better understand how to assemble it. One of the engineers uses the Manipulation tool to grab the arm but it only selects the Alpha Particle X-ray Spectrometer (APXS) because the rover arm is made up of many parts. Using some kind interface, the engineer needs to be able to move up the file hierarchy to select the entire arm. How will the engineer be able to do this by using the interaction controls? What can hierarchy look and feel like in ProtoSpace?

For this project, our team was given two challenges from the JPL client team:

– Design a way to navigate through CAD model file hierarchy in augmented reality
– Figure out how to communicate the design to the ProtoSpace team.

This is an example of just a portion of what the files in the CAD hierarchy look like.
My Role

When tasked with finding a solution to this problem I worked with a team of six, which consisted of a budding VR/AR expert (Yun), a software engineer (Joe), a psychologist (Lexus), and two marketing/communications professionals (Rose and Najma). I was mainly involved in the UX research and was responsible for the visual design of the user interface.

Because of the short, six-week time frame of the project, our team quickly cycled through various ideas, prototypes, and conducted several rounds of user testing before moving forward with our final prototype. The final prototype was demoed to Marijke and Matt from the JPL team and received positive feedback along with constructive criticism.

Constraints and Limitations: Protospace Application and Hololens Hardware
Application: Protospace
ProtoSpace is an augmented reality application that runs on Microsoft’s Hololens. ProtoSpace was developed at the Ops Lab at NASA Jet Propulsion Laboratory. The application gives spacecraft engineers (such as the Mars 2020 team) a way to collaboratively view and interact with their CAD models as holograms, before the spacecraft have been built. Seeing a holographic version of their model at scale allows teams to discuss and problem solve more efficiently and collaboratively.
Hardware: Microsoft Hololens
ProtoSpace runs on Microsoft’s Augmented Reality device Hololens. Hololens is a wireless headset that allows users to explore holographic imagery without additional camera installation. The field of view where you see the holograms is limited to a rectangular area, meaning the holograms cannot be seen all the way to your periphery.
Interaction and Gesture Controls: Hololens
Air-tap: This is generally used to select something. Holding your finger approximately a foot from your chest and within your field of view (FOV) hold your finger in the number 1 position and then bow it from your lowest knuckle.

Double air-tap: Same position and movement as the air tap but twice. This is a quick motion like double-clicking. This is used in ProtoSpace to open and close the models

Air-tap and drag: Air-tap but instead of lifting your finger back up you keep it down and drag it. This gesture is used for things like scrubbing through animations or drawing.

Gaze ray: Your head movement in ProtoSpace is marked by a small white dot inside the program. This is called your gaze ray. Your gaze ray acts as a cursor that is controlled by the movement of your head. To select things in Hololens, move your gaze ray to the thing you want to select and use the air-tap gesture to select it.

Clicker: Hololens ships with a handheld device that can replace the air-tap and double air-tap gesture. Like the air-tap gesture this tool works in conjunction with the gaze ray. Future versions of the clicker may include the ability to click and drag.

Protospace Features: Toolbar
Within ProtoSpace there are a series of tools that can be used to manipulate the model. They are the following:

Select Tool: Users can drop a shared “Point of Interest” indicator on a part of the spacecraft and hide and show selected parts.

Translation Tool: Users can move the spacecraft.
Scale Tool: Users can change the size of the spacecraft.

Rotate Tool: Users can rotate the spacecraft
Cut Plane: Users can manipulate a transparent plane to view select portions of the spacecraft.

Animation Tool: Users can play, loop, and scrub animated CAD models.

Manipulation Tool: Users can select and manipulate individual pieces of the spacecraft with 6 degrees of freedom.

Getting to know more about our users
The first challenge that we have is to get to know our target users, which are the mechanical engineers in NASA. Due to confidentiality concerns, we could not directly contact the NASA engineers, but we were able to get their insights directly from the design lead from the JPL team instead.

During this early stage in our process, our team felt like there were too many missing pieces for us to understand the workflow of mechanical engineers. To change this situation, the team proceeded to conduct interviews with engineers who are familiar with various 3D CAD software. So we conducted interviews with NYU robotics engineers who are a part of the Poly Lunabots. The robotic engineers showed us how they use various 3D CAD software (mainly Solidworks) by going through some basic workflows.

Interview Insights

Through communicating with clients and conducting interviews, we collected the following insights:

1. Identified use cases: From our clients, we got that there were two typical use cases. The first was to use ProtoSpace to support a collaborative discussion about the model with colleagues. The second was to use ProtoSpace as a presentation tool.

2. Identified users’ habits with CAD software: Observing users’ workflow helped us a lot in terms of understanding how the Hololens and the Protospace application could help improve such experiences.

After the first round of user research, we narrowed down the design challenge into the following pain points that we needed to address:

How can we quickly display parent-child relationships in the file hierarchy?

How is lengthy text displayed in 3D space?

What movements or gestures do users use to navigate the interface?

Ideation: Exploring Design Concepts

Once we had identified which issues to address, we began to brainstorm solutions. After our brainstorming session, we ended up with the three following design concepts for the user interface (the onion, the wheel, and the totem) that attempt to tackle our main pain points:
Concept One –

The ‘Onion’

Display parent-child relationship in the form of “Onion Rings”

Display text on “Onion Rings”

Move in space to control user interface

Concept Two –

The ‘Wheel’

Display parent-child relationship on the wheel

Display text on extended window

Air Tapping

Concept Three –

The ‘Totem’

Display parent-child relationship inside the totem disc

Display text below selected object

Air Tapping

Prototyping and Testing (First Round)

All initial prototyping for our concepts was done with paper and cardboard in order to get a feel for how users would interact with it in 3d space. Building low-fi prototypes allowed us to save time by quickly cycling through multiple iterations before deciding on a concept we felt more confident in moving forward with.

Moving forward with the ‘Totem’ concept
After much deliberation, we decided to move forward with the Totem concept based on the following reasons:
Relatively appropriate distance and size
Through testing, we quickly realized that we had to be mindful of the size of the user interface and the distance between users and the interface. For example, due to the limited field of view of the current version of HoloLens, the first design required users to constantly move their body and switch their views between the interface and view target(CAD Model), thus causing fatigue and unpleasant feelings. Therefore, we chose to move away from this design.
More in line with the use scenario of ProtoSpace
Based on previous interviews, ProtoSpace is mainly used in collaborative discussions or used as a presentation tool. In other words, it is used in a group environment that requires shared information. Comparing the second and the third design, we believed that the third design had better system visibility towards all participants in the ProtoSpace Session.


A visual representation of how an engineer would use the Totem feature. The storyboard below outlines a few of the basic functionalities of the Totem.

Step 1: Choose “Totem Select” tool from toolbar
Step 2: Air tap to open the first hierarchy of the Totem
Step 3: Click and drag to scrub through the objects in the first hierarchy (parent objects)
Step 4: Select the target child object
Prototyping and Usability Testing (Second Round)
To further test the usability of this system, Yun prototyped the user interface in Unity for HoloLens. Our team invited several users, including mechanical engineer students from the NASA Robotic team to conduct usability tests.
Scrub to select item
View from another HoloLens
Engineer student from NASA Robotics team joined us for usability testing
Analyze and plan for next steps

Through testing in HoloLens, we realized some other issues that came with AR UI design:

1. Interaction: The way to quickly jump between different hierarchy and the visibility of the system received positive feedback. Users could easily understand how to use the interface. However, users reported that using “click and drag” to scrub through objects could easily become a tiring gesture.

2. Visual: Users noticed that sometimes the UI would blend into the environment. This reminded us to be more careful of the typography, color, and transparency used in UI elements.

2. Sustainability/Scalability for extreme usage: We have to be more mindful of designing a more sustainable UI solution for more extreme users. i.e., when the totem expands to a point where the design can poentionally become severely compromised.

In our next step, we plan to explore deeper in the above areas to iterate a more efficient solution.

Team –
Project Managers  Dana Karwas, Najma Dawood
UX Research  Rose Bender, Joe Besilin, Lexus Aboloh, Yun Li
Unity Developer  Yun Li
UI/UX Designer  Ariana Ijaduola
Visual Designer  Ariana Ijaduola
Client Designer  NASA (Marijke Jorritsma and Matthew Clausen)
Next Project

Little Einstein