Our recent online fireside chat “Continuous AI Penetration Testing: When Exploits Arrive Before CVEs” brought together a panel of industry experts to explore why point-in-time penetration tests and traditional vulnerability management are no longer sufficient in fast-moving environments.
This article highlights three significant takeaways, notable quotes from the industry experts on the panel, key points discussed, questions from the audience and how they were answered, then concludes with two call to actions for CISOs.
The Panel Included:
3 CISO Takeaways
- Optimize for exploitable attacker paths, not raw vulnerability counts.
- Build risk prioritization using exploitability, exposure (external surface), and impact to crown jewels.
- Continuous testing must be integrated into CI/CD and remediation workflows to avoid alert fatigue and ticket overload.
Notable Quotes
“Point-in-time testing tells you what’s true on a Tuesday, but attackers care about what became possible on Wednesday.” — Michael Rombout
“The closer you can get the microscope down to the actual systems and processes running on the device, the faster you can triage, the faster you can remediate, and you can focus your teams on what truly matters and reduce risk.” — Carrell Jackson
“The bottleneck moved from finding issues to validating what is truly exploitable.” — Ido Geffen
“It isn’t AI finding more vulnerabilities… it’s whether AI can reason across the findings… and act like an attacker.” — Michael Rombout
Key Points Discussed
- Point-in-time testing alone is insufficient for modern change cycles: Panelists explained that penetration tests provide a snapshot (“what’s true on a Tuesday”), while applications, identities, APIs, and integrations change continuously and the attackers operate on faster timelines.
- The attacker timeline starts before CVEs: Threat intelligence is shifting from “when was a CVE published” to “how is exploitation possible and by whom, even before reliable detection guidance exists.”
- Compliance vs. real security outcomes: Some teams use penetration testing primarily for audit/compliance sign-off. Continuous penetration testing is needed to better secure the business,not just satisfy reporting requirements.
- Risk prioritization must be exploitability + business context: Rather than treating all findings equally, organizations should prioritize based on likelihood/exploitability and the impact to “crown jewels,” including whether services are externally reachable and actually running/exploitable in runtime.
- AI changes the nature of offensive capability and the defense approach: AI/LLMs and agentic methods enable more sophisticated and parallelized attack workflows, including novel exploit patterns. This increases the urgency for continuous, more context-aware validation.
- The bottleneck moves from “finding issues” to “proving exploitable risk”: AI-driven testing can generate noise, but the key is reasoning across findings to produce evidence of true exploitation potential and business impact.
- Close the loop: testing must connect to remediation: as continuous testing alone can create ticket overload. Value increases when findings translate into actionable attack paths tied to owners, mitigations, and remediation workflows (including “zero minus one” testing before production exposure). Organizations must develop continuous mitigation and remediation capabilities alongside continuous testing. Otherwise, faster discovery simply creates a growing backlog of unresolved findings.
- Human oversight and guardrails for autonomy: Participants discussed increasing AI autonomy for discovery/analysis first, then moving toward control/validation with authorization boundaries, logging, kill switches, and human-in-the-loop for high-risk actions.
- Economics and scalability matter: AI model selection, token consumption, and predictable vendor pricing are critical to making continuous testing sustainable over time.
Questions from the Audience
Q: What is the safest way to do continuous testing when exploits could impact critical systems (e.g., OT)?
A: Use controlled environments and strict rules of engagement/guardrails; treat AI-based testing like human testers bound by contractual constraints. For safety-impact scenarios, testing should remain in controlled lab environments.
Q: How do you reduce noise to focus on signals when using AI for continuous penetration testing?
A: Don’t optimize for “more findings.” Focus on evidence of exploitable risk and contextual business impact; connect findings to exposed services, runtime conditions, and remediations.
Q: Will continuous penetration testing become a standard practice, and what must change?
A: Likely yes as software release cycles accelerate and customers expect assurance. The main practical question is affordability and integration into existing testing/validation/remediation processes.
Call-to-Action for CISOs
- Start the journey from external surface management (“black box” perspective) to identify what is truly exposed and exploitable from the internet.
- Reframe program goals from “discover vulnerabilities” to “eliminate attacker paths that matter most,” integrating threat intelligence, exposure management, penetration testing, and rapid remediation, with priorities determined by potential business impact..
Available to watch on-demand: “Continuous AI Penetration Testing: When Exploits Arrive Before CVEs”
