Microsoft and Nvidia Set Out Security Checks for Enterprise Open-Weight AI
Open-weight AI gives enterprises room to adapt models, but every change can create a gap between task performance and safety. That’s the central concern in new guidance from Microsoft and Nvidia: a model can keep producing useful results even as fine-tuning, quantization, or configuration changes weaken its safeguards.
The companies’ joint paper, published last week, identifies 30 risk types across five categories and proposes a process to evaluate, reduce, and monitor them. Its core advice is practical: verify where a model came from, assess the exact version and setup headed for production, and repeat checks after significant changes.
“Weight openness alone does not determine an AI system’s risk,” the paper said. “Risk depends on the model, use case, deployment, and safeguards.”
Safety can erode while performance holds
Open-weight models can be fine-tuned, combined with other models, modified, or repackaged. Those steps may preserve performance on ordinary tasks while changing how a model handles unsafe requests. Microsoft and Nvidia describe safety behavior as more fragile than task capability: changes to weights, fine-tuning, quantization, or configuration can “degrade safeguards while leaving task performance intact.”
For security teams, that means capability and safety need separate evaluations. Testing should cover the specific configuration an organization plans to deploy, with another assessment after material changes. The paper’s risk categories span cybersecurity, data protection, content, human-AI interaction, and bias. Cybersecurity examples include malicious code generation, unsafe tool calls, attempts to extract safety policies, excessive token generation, and behavior that departs from the system’s intended goal.
Behavioral tests also have limits. “No volume of behavioral testing establishes what an artifact is, only how it behaves,” the paper said. Microsoft and Nvidia recommend cryptographic signatures to confirm that a model matches its publisher’s original file. If signatures aren’t available, they advise recording the uncertainty as an open risk rather than treating test results as proof of integrity.
Gartner’s suggested vetting steps include checking model sources and file integrity, conducting independent adversarial testing, scanning model files, and verifying provenance both before and after deployment. It also recommends recording approved models in an enterprise registry.
Application controls matter as much as model tests
A model assessment alone can’t establish that an agent or application is safe. The paper calls on enterprises to examine permissions, data access, connected tools, runtime behavior, and operational controls alongside the model itself. Its guidance spans the model, runtime, harness, and application layers, with monitoring of inputs, outputs, and tool use after launch.
Gartner recommends routing open-weight model traffic through governed AI gateways that can enforce authentication, prompt-injection defenses, sensitive-data protections, and audit logging. For self-hosted systems, the firm advises isolating runtimes. Cloud API deployments may call for private network paths and customer-managed encryption keys.
Demand is rising. Gartner said client inquiries about open-weight models through its research inquiry service increased by more than 30% in the second quarter of 2026 compared with the first. It projects that these models will represent 50% of enterprise AI use cases by 2028, up from 20% at the time of its report.
Ownership must include people and processes
Self-hosting shifts operational work onto the enterprise, including the staff needed to maintain safeguards. Sanchit Vir Gogia, chief analyst at Greyhound Research, said, “Self-hosting is a staffing decision as much as an infrastructure one, and the people to carry it must move with the model.”
Gogia cautioned that self-hosting doesn’t automatically mean an organization must replace every protection offered by a managed service. One potential gap, he said, is the provider’s misuse-investigation function; customer responsibilities vary with the service and its configuration.
He sees the Microsoft-Nvidia paper as useful guidance, while pointing to unresolved questions around accountability. “The sharper question is who is contractually obliged to re-test and patch the model when a new stripping technique lands,” Gogia said. He advises CISOs to assess the model, deployment, and intended use before approving production workloads, and to keep control of their own risk registers.
The companies say future papers will extend their guidance across runtime, harness, and application risks. For enterprise teams, the immediate takeaway is to treat each model change as a reason to revisit provenance, safety testing, and ownership of the controls around deployment.
