We got hit by ransomware — writing about the process and lessons learned

We had the same incident two years ago. Only difference: our backup wasn't clean.

Production stopped for three days. Now the backup copy sits separately and immutably. Was an expensive lesson.

I'd like to add a few points on the management side of incident response, because this side shapes the process just as much as the technical response.

First, who makes the call must be defined in advance. If it's unclear who to ask "should we stop the system" during an incident, hours are lost.

Second, communication plan. What to tell employees, when to inform customers, and how to fulfill legal notification obligations if needed should be written beforehand. If personal data is affected, consult regulations and your legal team for notification deadlines and procedures; these deadlines are short.

Third, the ransom payment decision. I want to emphasize that this is not a technical decision, but a top management and legal one. Payment doesn't guarantee data recovery and can create separate legal risks.

Fourth, the post-incident report. After the heat dies down, everyone goes back to normal and lessons aren't written down. An incident report not written within a week never gets written.

I want to poke at one point: you're saying "there were no vulnerabilities." How did you verify that?

I'm not asking out of malice. In most incidents, the first entry point found is assumed to be the correct one and the investigation stops there. But attackers usually leave behind multiple persistence methods.

Specific question: after the user logged in with a leaked password, did you check if any other accounts were created on the system, if scheduled tasks were added, or if remote access tools were installed? It's not uncommon for systems thought to be clean to be breached again three weeks later.

Fair question, let me answer.

We checked. An independent team investigated for two weeks. Two scheduled tasks and one locally created admin account created later were found. Both were missed during the initial cleanup.

That's why I want to add this: the decision to clean up shouldn't be made by the team experiencing the incident. The team, tired and in the middle of the incident, tends to be biased toward validating their own findings. If possible, have an outside perspective.

In the end, we reinstalled the affected machines from scratch. It was slower but a more certain path.

From an infrastructure perspective, let me suggest a concrete backup structure, because "keep it separate" is a bit abstract.

A common and effective setup is this: have at least three copies of the data, store them in at least two different environments, and keep at least one copy completely offsite. Additionally, making that offsite copy immutable after writing is crucial in a ransomware scenario.

Also, measure your recovery time. The statement "we have backups" is not as valuable as "we can be up and running in eight hours." If you don't know the latter, try it on a Saturday.

I'm a small business, let me explain from my side. Most time waste accumulates in tasks waiting for approval.

An automated scan report is not the same as a penetration test. Proven by experience.

Thanks for writing this, that's the right way. If it's your first time, start small; scaling comes later.

Correct me if I'm wrong.

You're right. If permission and scope aren't in writing, don't start that test.

Of course, it varies if your situation is different.

How did you solve this? An untested backup is not a backup.

I'm also curious if anyone does it differently.

This thread is archived.

Correct.

Id appreciate it if you shared the outcome.

My perspective changed after experiencing that... Payment information changes are never verified through the channel they came from.

That's all, sorry if I went on too long.

I've been down this road, let me tell you. When we decide without measuring, we always end up in the same place.

If you have questions, write them; I'll answer as best I can.

I felt relieved reading this answer, so it's not just me. Everything goes well for the first three months; problems arise in the fourth.

Correct me if I'm wrong.

im a small business let me explain from my side but everything goes well for the first three months; problems arise in the fourth.

the answer varies greatly by industry; there is no one-size-fits-all rule.

Just a heads-up. If permission and scope aren't in writing don't start that test.

Don't rely on a single measure; go layer by layer. If you have questions, write them; I'll answer as best I can.