SolarWinds Hack: What Do You Do When the Sky is Falling? Hold Your Umbrella With One Hand, and Begin Troubleshooting With the Other
The SolarWinds hack last year affected nine federal agencies and hundreds of US companies. Just this week during a hearing of the US Senate's select committee on intelligence, executives from several of the impacted firms said that while the overall scope of the attack isn't yet known, what is known is that this was an operation "of stunning size." One of our biggest current clients, a large healthcare company in the Philadelphia area, was a customer of SolarWinds, and in fact JTM Technology helped setup a lot of monitoring through SolarWinds for them.
We faced a two-layer problem with our client. 1) We had to help them make sure their network was secure; and 2) We had to get monitoring of key systems in place as soon as possible.
At the time, many companies heard the news and were left wondering how they may have been affected. Knowing our client may or may not have been infected, the important task at hand was, getting this server off of the network as quickly as possible and assessing the situation.
We faced a two-layer problem with our client. 1) We had to help them make sure their network was secure; and 2) We had to get monitoring of key systems in place as soon as possible.
Of these two big challenges, the network security was the higher priority. That day, and the days that followed, we used all of the tools at hand, and reached out to trusted technology pals we have, many of whom were going through the same thing. It became an interesting gaggle of tech professionals, all working together (and apart) to find an intrusion if indeed one existed, and two, make sure nothing else would be launched from the possibly infected systems. Scary times indeed, but ... what do you do when the sky is falling? Hold your umbrella with one hand, and begin troubleshooting with the other.
For our client's situation, the SolarWinds server was a VM. So, weeks later, after determining we had indeed locked down the network from any an all intrusions, we convinced the client it was acceptable to take that SolarWinds VM, remove any and all network connections to it, and stand it up via a web console. Our crack database resource knew the backend tables of SolarWinds like the back of his hand, and in a short period of time we were able to pull important information about alert settings, network traps, etc., from the server, to help in setting up a new monitoring solution.
As for the new monitoring piece ... well, that's a blog for another day.