New AI will teach soldiers to dress wounds, fly helicopters in AR

DARPA (the Defense Advanced Research Projects Agency) is developing AI assistants to help military personnel perform complex tasks — by showing them step-by-step instructions in augmented reality (AR).

AI assistants: Depending on the day, military personnel might be tasked with bandaging battle wounds, repairing fighter jets, or flying them. To make it easier for them to perform such complex duties quickly and correctly, DARPA established the Perceptually-enabled Task Guidance (PTG) program in 2021.

The idea behind the program is that future soldiers will wear sensors, such as microphones and head-mounted cameras, that collect data on what they’re hearing, seeing, and saying. Instructions from an AI assistant on how to complete tasks could then be displayed to them through AR goggles.

“What we need in the middle is an assistant that can recognize what you are doing as you start a task, has the prerequisite know-how to accomplish that task, can provide step-by-step guidance, and can alert you to any mistakes you’re making,” said Bruce Draper, manager of the PTG program, at the time it was announced.

“The idea is partly for us to see their progress, but also for all of them to work together.”

Bruce Draper

What’s new? Since then, DARPA has awarded PTG contracts and grants to researchers from NYU, Northrop Grumman, and more. Recently, the groups came together at MIT’s Hacker Reactor to demonstrate the first versions of their AI assistants. 

“The idea is partly for us to see their progress, but also for all of them to work together so we get this sort of cross-stimulation of groups working with each other,” said Draper.

Order up: The first three types of tasks DARPA wants PTG systems to assist with are mechanical repair, battlefield medicine, and pilot guidance.

For this first demo, though, the teams weren’t expected to show how their AI assistants could help fly helicopters — instead, they demonstrated how the systems could help someone follow a cooking recipe.

“[Cooking is] visually quite complex,” said Draper. “There’s specialized terminology. There are specialized devices, and there’s a lot of different ways it can be accomplished, so it’s a really good practice domain for all kinds of other highly skilled tasks.”

“I’m really impressed at how quickly people are making progress toward the goals,” he added.

In addition to showing DARPA their in-development systems, the teams also had a chance to demo them for some of the military members likely to actually use the tech — this allowed the researchers to get feedback on what is or isn’t working from the target end user.

“At the same time, [the military members] hopefully get an idea of where the technology is going so they can bring that back to their leadership with regard to what future possibilities are,” said Draper.

Over-achieving AIs: It’s not clear when DARPA expects the military to be able to deploy PTG systems, but some of the teams are ahead of schedule, according to Draper, already demonstrating how their AIs could do battlefield medicine tasks.

It’s possible civilians might get a chance to use the AI assistants, too — other tech we use everyday, such as GPS and the internet, was developed for the military before it made its way to the consumer market.

“We believe that our system would not only be useful to DARPA and the military, but also to regular users and people who want to perform new or complex tasks,” said Ehsan Elhamifar, a professor at Northeastern University and one of the PTG grant recipients.

We’d love to hear from you! If you have a comment about this article or if you have a tip for a future Freethink story, please email us at [email protected].