Today in Edworking News we want to talk about SAPwned: SAP AI vulnerabilities expose customers’ cloud environments and private AI artifacts. Wiz Research uncovers vulnerabilities in SAP AI Core, allowing malicious actors to take over the service and access customer data.
Executive Summary
Over the past few months, the Wiz Research Team delved deeply into tenant isolation within multiple AI service providers, discovering vulnerabilities in SAP AI Core. These vulnerabilities enable malicious actors to commandeer the service, gaining access to customer data and spreading to related environments. As AI infrastructures become ubiquitous in business environments, understanding these vulnerabilities is crucial for maintaining security. The research, dubbed "SAPwned," exposes how attackers could exploit these vulnerabilities and highlights the necessity for improved isolation and sandboxing standards when running AI models.
Image of SAP AI Core Network Setup
Description: Diagram illustrating the network setup of SAP AI Core
Introduction: The Research Begins
SAP AI Core is designed to let users develop, train, and run AI services utilizing SAP's cloud resources. However, running customer code in a shared environment is risky. Our research kicked off by exploiting basic permissions as an SAP customer to create AI projects. We used an Argo Workflow file to spawn a Kubernetes Pod, running our code within the Pod, bypassing the network restrictions enforced by an Istio proxy sidecar.
Bug #1: Bypassing Network Restrictions
Despite protections from an admission controller, we found configurations that were not blocked. Using the `shareProcessNamespace` and `runAsUser` configurations, we accessed Istio’s configuration, gaining an access token to the cluster's centralized Istiod server. This granted us network access, which we exploited to scan the Pod's internal network.
Bug #2: Loki Leaks AWS Tokens
An instance of Grafana Loki on the cluster exposed AWS secrets used for accessing S3, providing access to vast logs from AI Core services and customer Pods.
Bug #3: Unauthenticated EFS Shares
We discovered 6 AWS Elastic File System (EFS) instances configured publicly, allowing unauthorized access to AI data, including training datasets and code categorized by customer ID.
Bug #4: Unauthenticated Helm Server
The Helm server Tiller (version 2) was exposed without authentication, revealing secrets to SAP's Docker Registry and Artifactory server. Attackers could exploit these to read or modify internal images and cutomers' commercial secrets.
Bug #5: Full Cluster Takeover
The Helm server allowed both read and write operations, facilitating a complete cluster takeover. This exposure enabled attackers to access sensitive customer data, models, datasets, and more. Furthermore, customer secrets stored across AWS, SAP HANA, and Docker Hub were exposed and accessible.
Takeaways
Our research underscores several pivotal points:
The need for defense-in-depth strategies: Relying solely on perimeter defenses like Istio was insufficient once those defenses were bypassed.
The tenant isolation pitfalls in Kubernetes-managed services need addressing as they permit logical connections between control and data planes.
Guardrails are essential in AI model training to separate untrusted code from internal assets and other tenants.
Conclusion
All discovered vulnerabilities have been communicated to SAP, who promptly issued patches. Importantly, no customer data was compromised. Nonetheless, our findings stress the importance of robust, multifaceted security measures in managed AI platforms.
Description: A visual representation of AI Security Posture Management Framework
Further Engagement
Disclosure timeline:
- Jan. 25, 2024 – Wiz Research reports to SAP
- Jan. 27, 2024 – SAP responds and assigns a case number
- Feb. 16, 2024 – SAP fixes first vulnerability and rotates relevant secrets
- Feb. 28, 2024 – Wiz Research identifies new vulnerabilities, reports to SAP
- May 15, 2024 – SAP deploys fixes for all reported issues
- Jul. 17, 2024 – Public disclosure
Remember these 3 key ideas for your startup:
Prioritize Isolation and Sandboxing:
Ensure robust isolation and sandboxing standards in your AI R&D process to mitigate the risk of cross-tenant access and untrusted code execution. Check out SAP AI Core for how they address these challenges.Robust Defense-in-Depth Strategy:
Implement multifaceted security measures, including internal service hardening to minimize potential attack vectors. For instance, leveraging Istio for network security but be aware of its limitations.Stay Informed and Proactive:
Regularly update and patch vulnerabilities as soon as they are discovered. Engage with AI Security Posture Management tools for continuous visibility and proactive risk mitigation.
Edworking is the best and smartest decision for SMEs and startups to be more productive. Edworking is a FREE superapp of productivity that includes all you need for work powered by AI in the same superapp, connecting Task Management, Docs, Chat, Videocall, and File Management. Save money today by not paying for Slack, Trello, Dropbox, Zoom, and Notion.
For more details, see the original source.
