I’ve been in procurement meetings where the promise of “best-in-class” AI models meets the cold reality of legal redlines and security assessments. Vendors talk about accuracy and throughput; security and legal teams ask about data flows, retention, and liability. Too often those conversations stall because the evaluation framework is fuzzy. Over the years I’ve developed a practical procurement checklist that helps security and legal teams verify whether a vendor’s AI model can be used without exposing customer or sensitive data.
Why this matters
AI models—especially large language models and image generation services—can introduce data leakage risks at multiple points: during ingestion, inference, storage, and in vendor support channels. A vendor might unintentionally store or surface sensitive customer inputs (think API logs, training data reuse, or model fine-tuning with customer content). As someone who’s worked at the intersection of engineering and product, I know that minimizing these risks isn’t only a technical exercise; it’s also about contracts, operational controls, and ongoing validation.
High-level framework I use
My approach separates four areas that both security and legal care about:
- Data flows: Where data goes (and doesn't go).
- Retention and reuse: Whether the vendor retains customer inputs or uses them to improve models.
- Technical controls: Encryption, isolation, and logging.
- Contractual & compliance controls: SLAs, liability, audits, and certifications.
Procurement checklist for security and legal teams
The checklist below covers the essential questions to ask vendors. I recommend embedding these into your RFP or security questionnaire and requiring attestation plus evidence where possible.
| Domain | Question | Why it matters | Evidence to request |
|---|---|---|---|
| Data Flows | Do you capture or persist customer inputs (prompts, files, metadata) on your systems? | Persistent storage can lead to inadvertent exposure or reuse. | Architectural diagram, data flow diagram, DNS/IP ranges. |
| Data Use & Reuse | Do you use customer data to train or fine-tune models (explicitly or implicitly)? | Training reuse can cause downstream leakage across tenants. | Written policy, opt-out mechanisms, technical separation proof. |
| Retention | What are default retention periods for logs, audit trails, and model checkpoints? | Long retention increases exposure window. | Retention policy, deletion certificates, demo of deletion API. |
| Isolation | How do you logically and physically isolate customer data from other tenants? | Multi-tenancy without proper isolation risks cross-customer leakage. | Tenant isolation diagrams, VPC/namespace details, encryption keys per tenant. |
| Encryption | Is data encrypted in transit and at rest? Who manages the keys? | Key management controls access to plaintext data. | Encryption standards, KMS particulars, BYOK support. |
| Access Controls | How is vendor staff access to customer data controlled and audited? | Human access is a frequent root cause of breaches. | RBAC policies, access logs, SOC reports. |
| Model Outputs | Can models inadvertently reproduce training data or customer inputs? | Risk of exposing PII or proprietary text/images in outputs. | Testing reports, red-teaming results, model memorization metrics. |
| Incident Response | What is your notification SLAs and remediation process on data exposure? | Timely detection and notification reduce downstream harm. | IR playbook, notification timelines, contact points. |
| Compliance | Which certs and audits do you maintain (SOC2, ISO27001, PCI, GDPR processes)? | Third-party assurance reduces verification burden. | Latest audit reports, certifications, gap remediation plans. |
| Legal | What contractual guarantees do you provide around data ownership and liability? | Contracts create legal recourse if data is misused. | Model contract language, data processing addendum (DPA). |
Practical verification steps I require
Asking questions is necessary but not sufficient. Here are pragmatic tests and proofs I ask vendors to perform or provide.
- Signed DPA with explicit clauses: Require an explicit data processing addendum that forbids model training on customer data (or provides an opt-out) and clarifies ownership and deletion rights.
- Demo deletion: Ask the vendor to delete a test record and provide a deletion certificate or API response showing removal from logs and backups.
- Pen-test and red-team reports: Obtain recent results that include tests against the inference API and storage systems.
- Reproducible privacy tests: Run prompt-injection and extraction tests. For instance, include harmless but unique tokens in inputs and verify they do not appear in other tenants’ outputs.
- Proof of isolation: Validate that your traffic is routed through dedicated VPCs or namespaces. For cloud-hosted vendors, confirm VPC peering or private endpoints (AWS PrivateLink, Azure Private Endpoint).
- BYOK or customer-managed keys: If available, prefer Bring Your Own Key (BYOK) so you control key revocation.
- Access attestations: Request a list of roles with access, human access justifications, and privilege review cadence.
Red flags that have stopped deals for me
From experience, vendors that show these traits are risky to onboard without significant mitigations:
- Ambiguous answers like “we don’t typically retain” without concrete retention windows, proofs or controls.
- Refusal to sign DPAs or to explicitly deny use of customer data for model training.
- Lack of third-party audits (no SOC 2 or ISO report) in contexts where sensitive data will be processed.
- No support for private endpoints or tenant isolation in multi-tenant platforms.
- No transparency on model updates or data provenance—if you can’t trace where behaviors come from, you can’t bound risk.
Vendor types and tailored checks
Different vendors require different emphasis:
- Cloud-hosted API providers (OpenAI, Anthropic, Cohere etc.): Focus on data-use policies, retention defaults, and enterprise features (private instances, customer-managed keys).
- On-prem or private deployment vendors: Verify the packaging and update mechanisms; ensure updates don’t phone home with local data.
- Model-integrators or ISVs using third-party models: Ask who is the data controller/processor—both the integrator and the model provider may bear responsibilities.
Operationalizing the checklist
Make this checklist part of your standard RFP and security questionnaire. I also recommend a three-strike review: legal must sign off on the DPA, security must validate the technical controls, and product/engineering must complete integration tests (including prompt extraction checks) before going live.
Finally, treat this as an ongoing program, not a one-time checkbox. Models change rapidly, vendors update policies, and new classes of leakage emerge. If a vendor upgrades their model or introduces a fine-tuning feature, re-run core verification steps and review contractual coverage. I make a habit of scheduling periodic reassessments—every 6–12 months for high-risk integrations.
