The United States deployed operatives to Ukraine in the weeks before the Russian invasion in an effort to protect Kyiv’s critical networks and to learn more about the Kremlin’s hackers. The man in charge of these defensive cybermissions, known as hunt forward operations, is Maj. Gen. William J. Hartman.
In an interview with Recorded Future News’ podcast “Click Here,” Hartman says the cooperation between the US and foreign partners like Ukraine has become important in the effort to deter Russia’s cyberoperations.
Last week, Microsoft released new details about the barrage of cyberattacks that have been visited on Ukraine since the war began and said that a new threat actor — dubbed Cadet Blizzard — is the Russian military intelligence unit that has been leading the charge against Ukrainian networks.
Microsoft said the relatively new group has been linked to the wiper malware known as WhisperGate that showed up in Ukrainian government networks last January. When activated, the malware looked like your run-of-the-mill ransomware but was actually wiping master boot records (MBRs) on Ukrainian networks.
“Click Here” got access to half a dozen American cyberwarriors who were on the ground in Kyiv in the days before the invasion and helped discover and analyze the WhisperGate attack.
As part of that series of interviews, the “Click Here” podcast sat down with Hartman, the head of the US Cyber National Mission Force, to discuss not just the hunt forward operation in Ukraine but also the military’s effort to deploy cyberoperatives around the world.
Click Here: The Cyber National Mission Force hunt teams were in Estonia before the 2020 election and they deployed to Ukraine before the war. Why is it in the interest of the United States to deploy cyberoperatives in this way?
Gen. William J. Hartman: Well, it’s in our interest because what we’ve seen is that oftentimes the same nation-states that threaten us also threaten their neighbors. … And they will sometimes do so a little bit more aggressively than what we see in the United States. And so we send teams [to those countries] based on an invitation from the partner nation. We’re allowed to execute defensive hunt operations, and we’re able to observe adversaries operating in those foreign countries. We’re able to help our partner remediate their networks while also ensuring that we collect things like malicious software and other indicators of compromise. We bring that information with permission from the partner back to the United States and immediately share it with other US government partners, but also with US private industry. So if we want to talk about getting to scale from a defensive standpoint, gaining access to malicious software in a foreign nation, gaining access to indicators of compromise, sharing it with a private US cybersecurity company that may have millions of endpoints with sensors on their networks, it’s really beneficial to the defense of those things that we care about in the United States.
So in other words, if you catch it over there, it’s before it has a chance to come over here.
Absolutely. Absolutely.
What do you think people don’t really understand about a hunt forward operation?
The best time to build trust is before the Russians invade. Ukraine was a really important mission set for us — it was our fourth mission there. Each mission had gotten a little bit better, but I really do believe those previous missions set us up in order to move rapidly and be successful.Just from an authority standpoint, we have this unique hunt forward authority [that] allows us to deploy these teams to execute defensive hunt operations. We execute these fairly frequently, and we do think it puts us in a really good position to build resiliency and support our partners and allies around the world.
And so the number that’s been thrown around is 39 cyberteams, 2,000 military and civilian personnel. Are those numbers more or less accurate?
So those numbers are accurate. What we have done is organize our teams into jointly manned task forces. So we have six task forces that focus really against the major nation-state adversaries: Iran, China, Russia, North Korea. We have a task force that really looks at emerging threats, mostly ransomware threats to national security. And then we have a task force that really focuses on cyber access, weapons and tactics.
Can you walk me through a typical day for a hunt team that’s been deployed?
You know, the first thing that you really have to do is gain a real good understanding of the network, get a baseline, right? And then you’re going to identify anomalous activity — and the anomalous activity may be bad, it may be some misconfiguration, it may be bad user hygiene. You’re going to have a bunch of smart people sitting around a whiteboard, and they’re going to say, Hey, the No. 1 priority today is this. The team goes through the investigation and at the end of the day, they’re going to decide whether there’s a potentially malicious IP, or whether the malware that they found, they wanna know if it’s good or bad. That’s the No. 1 requirement. We found a piece of software that we don’t think should be here. Is it good or is it bad? And if it’s bad, how do we find more of it? That’s generally the drill. And if there’s deeper forensic analysis that’s required, we’ll work with the mission partner. We have the ability to send the data back [to the US], or the malware back here and do some deeper analysis to provide some more precise information. But generally that’s the way the team will operate. They’ll open investigations, they’ll attempt to close investigations, and then oftentimes we will find things on networks that are simply vulnerabilities — it’s not because there’s been adversary activity. And we will share that information with the mission partner. And our hope is that [the mission partner] will mediate any vulnerabilities on the network while the team is still there.
So a patch basically.
It might be a patch. It might be a port that’s open that shouldn’t be. It might be an admin password that was never changed. It might be a user on their network that’s operating in a way that they shouldn’t allow a user on their network to operate. I mean, there’s a host of things that could be wrong. And again, this isn’t a criticism of a foreign partner, we certainly have that occur on US networks.
So would most mornings start with a whiteboard? Your priority might be, let’s just say military communications and then the operators would swarm that?
So generally we will have different teams. And if at all possible, the way we like to organize a team in a US component and a mission partner component, and that team will investigate a case. And then we have host analysts, we have network analysts, we have malware analysts, and so there’s different work roles across the force.
Explain to me how the intelligence fits in.
The way the intelligence fits in for us is first and foremost we sit up here, at the campus of the National Security Agency, and we partner really, really closely with the Cybersecurity Directorate. We get access to, to information that the cybersecurity director has, about adversaries that target the United States or allies and partners. And so ultimately we want to execute an intelligence-driven mission. Because we have intel that tells us that an adversary that threatens us is also threatening one of these partners. But we also sit with the Cybersecurity Collaboration Center, which is the NSA element that really works with hundreds of industry partners, in order to exchange information. We have an organization called Under Advisement that sits out there. And again, the US private industry is extraordinarily powerful as it relates to the cyberindustry. And so information on the industry side can also help us, and we bring that intelligence [to the mission partner]. And it really helps us from a start point with the adversary to sort of know what to look for.
So, We’ve seen this kind of behavior, we’ve seen these kinds of malware. I know it’s basic, but is that the sort of thing that you’re talking about?
Sure. If we have seen particular infrastructure from the internet targeting the United States allies or partners, when we send a team overseas, I mean, they’re gonna look for those IP addresses. And if they’re seeing attempted inbound connections, that’s like an immediate trigger that hey, they can be concerned about that. There will be malicious software that has already been identified. And if it’s a mission like in Ukraine and it’s Russia-focused, there is gonna be a set of signatures that will be loaded up on the team’s kit of known malicious software that the adversaries we expect are gonna be using. And that’s the first thing you look for is known activity. And that’s what the team is able to bring with it. And some of that is because of the partnership with NSA. Some of it’s because of the partnership with private industry. Some of it is because we do these all over the world and we find malware and we make sure that we don’t do discovery learning. That each time we build on what we’ve done before and get better.
How do techniques that you see someplace like Ukraine differ from techniques you’d see used in the US? Is there more chalk on their cleats in Ukraine? Are [the Russians] trying bigger, bolder things?
So generally what we have seen in Ukraine is that in this case, an adversary like Russia will generally be a little bit more aggressive than they would in the United States. So in mid-January [2022] we started to see a series of destructive wiper attacks. And those are attacks that weren’t going to go unnoticed, that were ultimately going to be attributed. Certainly we have never seen the Russians operate in that manner in the United States or in one of our Western partners.
In other words, they try to hide their tracks as opposed to being so blatant. Is that the right way to understand it?
I think it’s a fair assessment.
When it comes to wiper malware, and that’s what we’ve heard coming out of Ukraine, what’s different about it?
So I think what’s different about it is the scale in which it was used. And so in mid-January we saw literally dozens of wiper attacks. We saw some attempt to obfuscate that it was, in fact, a nation-state executing a wiper attack by making it look like a ransomware attack with no ability to pay any kind of ransom or get your data unencrypted. And so to me it was just a fairly blatant attack that was pretty easy to attribute to the Russians in this case.
And had we seen wiper malware before and just it was the amount that was different or were there things about it that were new?
So I can’t really comment on the various categories of wiper malware. But this appeared to be somewhat unique to the operations there in the Ukraine. But certainly this type of malware isn’t something new or revolutionary.
How do you get adversaries out of networks? Does the partner do that? Do you help them do that?
We bring unclassified equipment. When we execute a defensive hunt operation we install that equipment on a partner’s network based on an agreement with that partner. And when we identify either malware or some type of misconfiguration on a network, we instruct the partner and the partner will take the remediation actions on their own network.
So you’ll say, OK, we found X. We’ve isolated X. We suggest you do Y to maybe get it gone. Or maybe they have their own system of doing that?
Generally we will make recommendations on changes that an organization should make to their network to make it more secure. And look, it’s generally based on our experience. It’s based on best business practices. It’s based on industry standards. And so at the end of the day, we’re generally not getting into these long debates about what’s the best way to remediate the network. And if possible, ultimately what we’re trying to do is to provide a mission partner advice on how they might remediate an adversary’s access to their network in a way that the adversary’s not able to easily come back again.
What’s different about this too is that, because of those Russian soldiers gathering on the border last year, there appears to have been a kind of time-pressure crunch that you don’t typically have. How did that change the tempo as you saw it from a command perspective?
It fundamentally changed it. We deployed a team in early December to execute what is really a pretty standard procedure for one of these defensive hunt operations. We deployed a team, led in this case by a young officer with about 12 years in our service. The team deploys, they link up with the mission partner, they exchange information, get a copy of network diagrams, get some access to some data. And then generally, we come back to the United States and we make a deliberate plan. We’d configure the equipment that we’re gonna use and then we’d deploy the team for a more deliberate defensive hunt operation.In this case, the team deployed early December 2021. The Russians [were] amassing 130,000 soldiers on the border with Ukraine. When the team arrived there, there was an immediate assessment by the team lead that the original plan probably was going to be insufficient. And so instead of executing the normal plan, the team lead immediately got on the phone and asked to deploy the rest of the team. And we immediately went into a hunt operation.
Did that change the way the team operated?
Certainly that changed the dynamic. And then relatively early in the mission we saw a series of destructive Russian wiper attacks, which fundamentally changed the mission. Now, in addition to the networks we were hunting on, we started a fourth line of effort to help the mission partner analyze the malware and to ensure, based on the [Ukrainians’] approval, that we could bring that malware back to the United States and share it broadly with the US government and private sector partners.If you remember, there were a lot of reports in the news at that point that we were expecting potential Russian attacks in the United States. And so there was a sense of urgency on the Ukrainian side, and there was a sense of urgency on the US side. And then to further complicate it, the team remained there until the end of February. And, again, if we remember the intelligence assessments, there was a wide belief that as soon as the Russians invaded they were going to be in Kyiv within a few days. We’ve never executed a hunt operation like that before.
Did you ever get a phone call in which they said, please don’t go yet, or please don’t go?
I never got a phone call from the Ukrainians saying please don’t go. I think I probably got a couple of phone calls from my team saying, please don’t make us go. And again, there is a human aspect of the mission that we don’t talk about here, right? The team deployed in early December. Their loved ones expected them home 10 days later. The loved ones didn’t expect them to go through Christmas and New Years. And so the team stayed there. The team focused on the mission, and our Joint Task Force there talked to the family members and explained to the extent they could what was going on. But the reality is, as it got closer, the desire from the team was to continue to stay there and work with the Ukrainian partners. And that was a fairly emotional order to say, Hey it’s time for everybody to leave the country. I can’t speak for the Ukrainians. But certainly for the US team, they still feel a connection with the Ukrainian partners to this day, and I expect they’ll continue to do so.
One of the things that’s really struck me is how many young people are at the pointy end of the spear. You basically have a cohort that’s been digitally literate since the crib. What role does that play in putting together a cyberforce?
I think the fact that they’re digitally literate is important, but the big piece here is leadership. I remember being a brigade commander a number of years ago when somebody asked me, Hey, what’s the No. 1 thing that you’re looking for in the cyberbranch? And it’s still leadership. And when we talk about what happened in the early stages of the mission in Ukraine, the team lead should be given all the credit. You know, we sent a team lead over there with a plan. [A US Marine Corps major in charge of the mission] decided that our plan really wasn’t suited to the environment and the needs that she saw there. And they made a series of decisions that were the right decisions to make, but ultimately the entire enterprise backed the leader up and supported them. So we have these highly technically trained young people that are great leaders, and we put them in positions where, quite honestly, their expertise is what leaders like me have to depend on. And these hunt missions are just a great example of the work that they do.
Do you think defensive cyber is at an inflection point? You know, offensive cyber always used to be the sort of sexy part of cyber. Is defensive cyber hunt teams changing the perception that defense is just whack-a-mole?
So I certainly think if you interviewed a broad number of cyber soldiers, sailors, airmen, Marines, and you asked them what might have been the most impactful mission that they had done, increasingly you would get a hunt forward with a mission partner in a part of the world that the United States has articulated as critically important to our national security. And so I think that that is really important.
Why do you think that is?
Well, we all join the service to do things that we believe we could closely link to protecting the United States. And I’ve been very fortunate to visit six or seven hunt forward missions in a number of locations. You walk in a room, and half of the people in that room are young Americans from the Cyber National Mission Force, and half the people are generally from a similar service in the mission partner. And their ability to work together to take defensive actions against an adversary that is attempting to threaten both their networks and our networks — it’s just a really powerful environment. And our service members and our civilians that participate generally derive a lot of personal satisfaction out of being able to do that.
How’s the hunt forward operation different now than it was two years ago?
I think the difference now is that we’ve done these enough times that we have a formula. One of the great things is I get a mission brief and approve all these things before I go out. You know, I’ll get a chart that’ll show me the unit members that are going on the mission. I always ask, Hey, how many people have been on one of these missions before? And generally it’s about 50/50. So we’re capitalizing from the previous missions, but we’ve improved the training we provide to the leadership of the team. We’ve improved the experience level. Before, if we did a 60-day hunt forward operation, we might get really smart at about day 50. Now, generally within the first couple weeks of an operation, I think we’re having meaningful conversations and meaningful mission outcomes for the teams.
So is there an attack that you stopped?
There are a number of things that we know that we’ve stopped that I’m not gonna be able to specifically share with you.
How about generally?
I think generally we have talked about before, during the SolarWinds attack based on our fantastic partnership with the Department of Homeland Security, we were able to get a compromised copy of the SolarWinds server from a US government agency. We were able to replicate that in our persistent training environment. We were then able to train and rehearse on finding that specific malware because we’re the military and we do that very well.And then we had a request to send a team to Europe in order to hunt for a suspected SVR (Russian Foreign Intelligence Service) compromise. And we knew exactly what to look for. We were able to censor up the network in question and we were able to: One, find the Russian malicious activity. Two, we were able to advise the mission partner on how to evict the adversary from the network. And three, we were able to watch the adversary attempt to unsuccessfully re-explain that network. And we were able to do that without the adversary having any idea that we were there.
And is there any example like that coming out of Ukraine that you think you stopped?
We initially sensored up three different networks in Ukraine. And while I’m not claiming credit for those networks not being attacked, certainly the networks we were on were not subject to any of the malicious wiper attacks that we saw in January [2022]. Prior to the team departing [Ukraine], we were able to get access to a significant amount of malware in Ukraine, really based on the partnership with Ukraine. And that’s the piece that I haven’t talked about yet that’s really important is, there’s nothing like being there. And so the team is on the ground in Ukraine. Ukraine starts to see a series of destructive attacks, we open up a fourth line of effort, but then we immediately start sharing information. And even when the team departed, we’ve continued to do that. And so between us and Ukrainian partners, I think we’ve shared over 6,000 indicators of compromise. And that’s stuff that we’ve been able to see from industry partnerships. That’s stuff that they’ve been able to see from activity on the ground. And that relationship continues to this day.
So a lot of people talk about Ukraine as the first hybrid war. Really a marriage of kinetic and cyber in a way we haven’t really seen live before. What do you think it’s taught us about the future of war?
So I think it’s reinforced something that we already knew. That this is really hard. I think the Russians had an intent to execute a series of destructive cyberattacks to take out ViaSat, to execute a fairly quick move and seizure of a large swath of Ukraine to include Kyiv. And it’s really much harder to execute a synchronized plan like that than it is to have an intention to do it. And so for us, we’ve got to continue to develop not just cybercapabilities, but ensure that what we’re doing is fully integrated with all our other plans. And for us, certainly on a daily basis it’s something that we work really closely with the Joint Force Headquarters-Cyber, that supports the combatant commands to ensure that anything we’re going to do is fully synchronized.
I think there was always this perception that in cyber, Russia was an 800-pound gorilla. Do we still think that?
So I think in Ukraine. … Look, the defender gets a vote. And so there should be credit given to Ukraine. 2014 didn’t go so well. We started sending teams over there in 2018, and Ukraine made a lot of decisions.They invested resources and at the end of the day, the Ukraine in 2021 and 2022 was much more resilient than we’d previously seen. And so I would tell you that if you go back, read the Microsoft report, look at history, Russia will have executed more cyberattacks in Ukraine than any nation-state has ever executed over a comparative period in the history of the world. But the Ukrainians have proved very resilient. They have gotten a lot of support. They have gotten some support from us. They’ve got some support from NATO partners. There are US corporations that have done a fantastic job. Moving data from on-prem to off-prem was, was really, really important here. But the lesson here is that the work that we do to ensure that our networks are prepared to operate in this high threat environment are really important. But the defender does get a vote. And if the defender does the right things, we can build resilient networks even in the face of something like hundreds of Russian cyberattacks.
Where are hunt forward operations now?
One of the things we don’t do is ever talk about ongoing hunt operations. And so at any one time we might have anywhere from six to 10 teams that are deployed and executed in operations. But we generally don’t acknowledge where those teams are until the missions are done. And then we only do it if the countries we’re operating in have agreed to a public announcement.
Is it safe to assume that there are some teams deployed around places that are close to Russia and close to Ukraine?
So I’ve been in command for almost four years, and it’s safe to assume there are always teams deployed in the European theater of command.
An earlier version of this story appeared on the “Click Here” podcast from The Recorded Future News. Additional reporting by Sean Powers, Will Jarvis, and Sarah Wyman. The interview has been edited for length and clarity.
Sign up for our daily newsletter