With Large Language Models (LLMs) rapidly becoming central to numerous applications, it’s critical to address the security vulnerabilities inherent in these complex systems. Recent research analyzing open-source LLM vulnerability scanners, including Garak, Giskard, PyRIT, and CyberSecEval, has identified current strengths and limitations in the tools available for identifying and mitigating LLM vulnerabilities. Here, we outline top security considerations for two common scenarios, developing LLMs and using foundation LLM models in applications, and offer a strategic recommendation for assessing third-party software with embedded LLMs. For critical infrastructure applications, additional security measures are essential to ensure safety and continuity.
1. Considerations for LLM Development
For organizations developing their own LLMs, customizable and accurate vulnerability testing is crucial. Tools like PyRIT and Giskard offer extensive customizability for crafting specific testing scenarios aligned with the model’s intended use. Multi-turn interaction testing, which PyRIT supports, is vital for conversational models, as vulnerabilities may emerge over extended dialogues rather than in single-turn responses.
If your LLM generates code, integrating specialized code security checks is paramount. CyberSecEval is particularly suitable for ensuring generated code aligns with secure coding practices, helping detect and mitigate risks associated with unsafe or insecure code outputs. This consideration is essential to prevent vulnerabilities that could propagate into software environments or end-user applications.
2. Considerations for Using Foundation LLMs
Organizations leveraging foundation models for custom applications should prioritize guardrails and layered security. Garak and Giskard support NeMo Guardrails integration, which provides protective layers to restrict potentially harmful responses or inadvertent disclosures. Additionally, contextual testing for scenario-specific vulnerabilities is essential for aligning model responses with application needs, such as customer service or healthcare requirements, where security concerns may differ.
Frequent scanner updates and monitoring are critical when using foundation LLMs, as these models evolve. Choosing scanners with active community support, such as Giskard and CyberSecEval, can ensure that your security tools adapt to address new threats as models are updated.
3. Strengthening Software Acquisition with AI Risk Assessment
For third-party software that includes LLM functionality, direct vulnerability scanning may be less feasible. Instead, organizations are advised to strengthen the software acquisition process with an AI-focused risk assessment. Evaluate vendor-provided security measures, including LLM risk assessments, certifications, and evidence of vulnerability testing. Establish clear criteria around the secure usage of embedded LLMs to ensure that any potential vulnerabilities are identified and mitigated by the vendor before deployment in your environment.
Additional Considerations for Critical Infrastructure
For critical infrastructure applications, where LLMs might interact with essential systems like utilities or healthcare, the stakes are even higher. Beyond the typical testing and acquisition recommendations, adhering to industry-specific compliance standards (such as NERC CIP for energy or HIPAA for healthcare) is crucial, incorporating these requirements into vulnerability testing and monitoring.
Real-time threat intelligence updates and ongoing monitoring are necessary for critical environments, where any delay in addressing emerging threats could compromise operational continuity. Additionally, robust adversarial testing is vital to ensure resilience against complex, context-based threats. Tools like PyRIT, with multi-turn testing, are valuable here to simulate sophisticated interactions that could reveal vulnerabilities under specific operational conditions.
By following these guidelines, organizations can strengthen their LLM implementations across various applications, reducing risks and ensuring safe, reliable deployment - an approach that aligns with the AISUF framework’s focus on building transparency and integrity into AI systems.