Considerations for Policymakers and Developers

Policymakers hoping to regulate foundation models have focused on preventing specific objectionable uses of AI systems, such as the creation of bioweapons, deepfakes, and child sexual abuse material. Effectively blocking these uses can be difficult in the case of foundation models as they are general-purpose technologies that in principle can be used to generate any type of content. Nevertheless, foundation model developers have been proactive in this area, adopting broad acceptable use policies that prohibit many dangerous uses that developers select themselves as part of their terms of service or model licenses.

As part of the 2023 Foundation Model Transparency Index, researchers at the Stanford Center for Research on Foundation Models cataloged the acceptable use policies of 10 leading foundation model developers. All 10 companies publicly disclose the permitted, restricted, and prohibited uses of their models, but there is little additional information available about these policies or how they are implemented. Only 3 of 10 leading foundation model developers disclose how they enforce their acceptable use policy, while only 2 of 10 give any justification to users when they enforce the policy.

This post provides background on acceptable use policies for foundation models, a preliminary analysis of 30 developers’ acceptable use policies, and a discussion of policy considerations related to developers’ attempts to restrict the use of their foundation models.

Background

What is an acceptable use policy?

Acceptable use policies are common across digital technologies. Providers of public access computers, websites, and social media platforms have long adopted acceptable use policies that articulate how their terms of service restrict what users can and cannot do with their products and services. While enforcement of these policies is uneven, restrictions on specific uses of digital technologies are widespread.

In the context of foundation model development, an acceptable use policy is a policy from a model developer that determines how a foundation model can or cannot be used.¹ Acceptable use policies restrict the use of foundation models by detailing the types of content users are prohibited from generating as well as domains of prohibited use. Developers make these restrictions binding by including acceptable use policies in terms of service agreements or in licenses for their foundation models.²

Acceptable use policies typically aim to prevent users from generating content that may violate the law or otherwise cause harm. They accomplish this by listing specific subcategories of violative content and authorizing model developers to punish users who generate such content by, for example, limiting the number of queries users can issue or banning a user’s account.

Acceptable use policies relate to how foundation models are built in important ways. For example, developers frequently filter training data in order to remove content that would violate acceptable use policies. OpenAI’s technical report for GPT-4 states “We reduced the prevalence of certain kinds of content that violate our usage policies (such as inappropriate erotic content) in our pre-training dataset, and fine-tuned the model to refuse certain instructions such as direct requests for illicit advice.”³

In addition, many developers state that the purpose of reinforcement learning from human feedback (RHLF) is to make their foundation models less likely to generate outputs that would violate their acceptable use policies. Meta’s technical report for Llama 2 notes that the risks RLHF was intended to mitigate include “illicit and criminal activities (e.g., terrorism, theft, human trafficking); hateful and harmful activities (e.g., defamation, self-harm, eating disorders, discrimination); and unqualified advice (e.g., medical advice, financial advice, legal advice),” which correspond closely to the acceptable use policy in Llama 2’s license. Anthropic’s model card for Claude 3 similarly says “We developed refusals evaluations to help test the helpfulness aspect of Claude models, measuring where the model unhelpfully refuses to answer a harmless prompt, i.e. where it incorrectly categorizes a prompt as unsafe (violating our [Acceptable Use Policy]) and therefore refuses to answer.”

Distinguishing acceptable use policies from other mechanisms to control model use:

Acceptable use policies are not the only means at a developer’s disposal to restrict the use of its models. Several other policy-related mechanisms that developers implement to restrict model use include:

• Model Behavior Policy: Model behavior policies determine what a model can or cannot do. While acceptable use policies apply to user behavior, model behavior policies apply to the behavior of the model itself.⁴ A model behavior policy is one way of effectively embedding the acceptable use policy into the model; methods for imposing a model behavior policy include using RLHF to cause the model to be more likely to refuse violative prompts or employing a safety classifier to filter violative model outputs. Model behavior policies generally go well beyond the acceptable use policy in terms of their impact on the model; for example, many developers fine-tune their models to produce more polite responses, though they do not prohibit users from generating impolite responses.
• Model Card: Model cards, which are published alongside machine learning models when they are released, provide essential information about models such as their intended uses and out-of-scope uses. However, model cards are not enforceable contracts, and they are not generally referenced in model licenses or developers’ terms of service; as a result, these out-of-scope uses do not rise to the same level as prohibited uses in an acceptable use policy.⁵
• API Policy: Policies relating to application programming interfaces (APIs) determine acceptable user behavior in relation to a company’s API. These policies generally apply to more than one foundation model and differ from acceptable use policies in that they attempt to shape the actions a user takes via an API, not the content that a user elicits from a specific model.⁶ Enforcement of API policies differs from enforcement when users are making individual queries in a playground environment or a user interface for a chatbot—companies must consider the risk of mass generation of violative content as well as the creation of services built on top of their models that violate their policies.
• Third party contracts: Foundation model developers frequently partner with other firms to disseminate foundation models. These include cloud service providers (e.g., Amazon Web Services, Microsoft Azure, Google Cloud Platform), platform providers (e.g., Scale AI, Nvidia), database providers (e.g., Salesforce, Oracle), and model distributors (e.g., Together, Quora). Custom contracts with third party providers of a developer’s foundation models often include use restrictions, but the extent to which companies’ acceptable use policies are altered via these partnership agreements is unclear.⁷

⠀ Emerging norms among closed foundation model developers on acceptable use policies:

Although generative AI is a nascent industry, norms have begun to emerge around use restrictions for foundation models. In 2022, OpenAI, Cohere, and AI21 Labs wrote in their “Best practices for deploying language models” that organizations should “Publish usage guidelines and terms of use of LLMs in a way that prohibits material harm to individuals, communities, and society such as through spam, fraud, or astroturfing.”⁸ As part of the November 2023 UK AI Safety Summit, Amazon, Anthropic, Google Deepmind, Inflection, Meta, Microsoft, and OpenAI shared their policies related to “preventing and monitoring model misuse,” many of which referenced their acceptable use policies.

Limitations of acceptable use policies for foundation models:

As currently implemented, acceptable use policies for foundation models can have serious limitations. Recent research from Longpre et al. has shown that of seven major foundation model developers with acceptable use policies, none provide comprehensive exemptions for researchers. As a result, acceptable use policies can act as a disincentive for independent researchers who red team models for issues related to safety and trustworthiness, which led over 350 researchers and advocates to sign an open letter calling for a safe harbor for this type of research.

The enforceability of acceptable use policies, especially for open foundation model developers, is a major limitation on how effective they are at restricting use. For example, the legality of enforcement actions based on the use restrictions contained in Responsible AI Licenses has not been tested, and in the current ecosystem there are few people who will be paid to enforce the terms of these licenses.⁹ Even if enforcement were simple, it may require developers to monitor users closely, which could facilitate privacy violations in the absence of adequate data protection, or increase the number of false refusals, reducing the helpfulness of foundation models.¹⁰

On the other hand, use restrictions in model licenses can have a dissuasive effect regardless of enforcement.¹¹ Most companies and individual users are not bad actors and may adhere to a clear acceptable use policy despite gaps in enforcement. Acceptable use policies can play a helpful role in limiting undesirable uses of foundation models, though they are not without costs and limitations.

Government policies related to acceptable use policies for foundation models:

Governments have taken an interest in acceptable use policies, which are a salient effort by foundation model developers to “self-regulate.” The European Union’s AI Act requires that all providers of general-purpose AI models disclose the “acceptable use policies [that are] applicable” to both the EU’s AI Office and other firms that integrate the general-purpose AI model into their own AI systems.¹² China’s Interim Measures for the Management of Generative Artificial Intelligence Services, which were adopted in July 2023, go a step further by requiring that providers of generative AI services act to prevent users from “using generative AI services to engage in illegal activities…including [by issuing] warnings, limiting functions, and suspending or concluding the provision of services.”¹³ In the US, the White House’s Voluntary AI Commitments, which 16 leading AI companies have adopted, include a provision that companies will publicly report “domains of appropriate and inappropriate use” as well as any limitations of the model that affect these domains.¹⁴

Neither the AI Act nor the White House Voluntary Commitments require that companies enforce their acceptable use policies or restrict any particular uses. While China’s Interim Measures for the Management of Generative Artificial Intelligence Services do not detail restrictions on content that is not outright illegal, regulatory guidance issued in February 2024 by China’s National Cybersecurity Standardisation Technical Committee specifies many types of prohibited uses such as subverting state power, harming national security, and disinformation.¹⁵

Acceptable Use Policies for Foundation Models in Practice

An overview of foundation model developers’ acceptable use policies:

Figure 2 details the acceptable use policies for 30 foundation model developers, including (i) the title of the acceptable use policy, (ii) whether it is applied to a specific foundation model, (iii) whether it is included in a model’s license, the company’s terms of service, or a standalone document, (iv) the developers’ flagship model series (and its modality), (v) the location of the developer’s headquarters, (vi) whether the weights of the developer’s flagship model series are openly available, and (vii) a reference to the public document that includes the acceptable use policy.¹⁶

Developers use different policy documents to restrict model use. These different types of documents include: standalone acceptable use policies for all of their foundation models (e.g., Google, Stability AI), use restrictions that are built into a general model license (e.g., AI2), use restrictions that are part of a custom model license (e.g., BigScience, Meta, TII), or provisions in an organization’s terms of service that apply to all services including foundation models (e.g., MidJourney, Perplexity, Eleven Labs). Developers release little information about how they enforce their acceptable use policies, making it difficult to assess whether one of these approaches is more viable.

Foundation model developers that have acceptable use policies are heterogenous along multiple axes. In terms of model release, 12 of the developers openly release the model weights for their flagship model series, while 18 do not. These models have a variety of different output modalities, with 19 language models, 5 multimodal models, 3 image models, 2 video models and 1 audio model. And in terms of where the developers are headquartered, 19 are based in the United States while the remainder include organizations are based in Canada, China, France, Germany, Israel, and the UAE. The different legal jurisdictions in which these organizations operate and make their models available helps explain some of the differences in the substance of their policies as some governments have requirements related to acceptable use policies and specific types of unacceptable uses.

Content-based prohibitions¹⁷

Acceptable use policies commonly prohibit users from employing foundation models to generate content that is explicit (e.g., violence, pornography), fraudulent (scams, spam), abusive (harassment, hate speech), deceptive (disinformation, impersonation), or otherwise harmful (malware, privacy infringements).¹⁸ Many developers have granular restrictions related to these types of prohibited content (e.g., Anthropic, Cohere, OpenAI), whereas others have broad restrictions without much elaboration (e.g., AI21 Labs, Midjourney, Writer). Another group of developers has fairly minimal acceptable use policies, such as open developers like the Technology Innovation Institute, which prohibits use of Falcon 180B to harm minors, disseminate disinformation, or harass others; by contrast, Stability AI, another open developer, has over 45 itemized prohibited uses.¹⁹

Acceptable use policies also generally prohibit users from generating content that violates the law, impedes the model developer’s operations, or is not accompanied by adequate disclosure that it is machine-generated. These catch-all prohibitions cover unenumerated content that may be problematic, and make acceptable use policies more malleable and comprehensive by linking them to laws that may change. Importantly, content-based restrictions generally apply only to user prompts that attempt to have a model generate this type of content – models will classify the toxicity of this type of content if asked, but they will not generate it.

While the bulk of these prohibited uses are common across acceptable use policies, there are also a handful of edge cases.²⁰ (See the underlying data in the GitHub.) Political content, such as using foundation models for campaigning or lobbying, is explicitly prohibited by OpenAI, Anthropic, Cohere, Midjourney, and others, whereas Google, Meta, and Eleven Labs have no such prohibition.²¹ Eating disorder-related content, such as prompts related to anorexia, is explicitly prohibited by Character, Cohere, Meta, and Mistral, but not by other developers. Weapons-related content is explicitly prohibited by the Allen Institute for AI, Anthropic, Meta, Mistral, OpenAI, and Stability, but not by other developers. And while some open developers such as Adept, DeepSeek, and Together broadly prohibit some types of sexual content, others like Meta and Mistral prohibit content related to prostitution or sexual violence.²²

Several other prohibited uses that stand out include:

• Undermining the interests of the state: Baidu and DeepSeek, two of three model developers based in China in Figure 2, state in their acceptable use policies that users must not generate content “endangering national security, leaking state secrets, subverting state power, overthrowing the socialist system, and undermining national unity…damaging the honor and interests of the state…undermining the state’s religious policy”. 01.ai, the other Chinese developer, also includes a prohibition against “harming national security”. These restrictions directly parallel China’s regulatory guidance on Basic Safety Requirements for Generative Artificial Intelligence Services.
• Password trafficking: Eleven Labs, the only foundation model developer specializing in audio in Figure 2, prohibits users from using its models to “trick or mislead us or other users, especially in an attempt to learn sensitive account information, for example user passwords”. This may help address common concerns regarding the use of voice cloning for scams.
• Misinformation: The extent to which companies restrict users’ ability to generate inaccurate content varies widely. While some companies’ usage policies include wholesale bans on misinformation (e.g., AI21 Labs, Inflection), others have more lenient restrictions that apply only to verifiable disinformation with the intent to cause harm (e.g., TII).

Restrictions on types of end use:

In addition to content-based restrictions, acceptable use policies for foundation models often restrict the types of activities that users can engage in when using their models. Several notable examples include:

• Model scraping: Anthropic’s Acceptable Use Policy states that it does not allow users to “Utilize prompts and results to train an AI model (e.g., ‘model scraping’)”. Developers such as Adobe, Aleph Alpha, and Perplexity similarly prohibit the use of outputs of their models for training other foundation models.²³
• Scaling distribution of AI-generated content: AI21 Labs’ Usage Guidelines state that “No content generated by AI21 Studio will be posted automatically (without human intervention) to any public website or platform where it may be viewed by an audience greater than 100 people.”
• Hosting the model with an API: The Technology Innovation Institute’s license for Falcon 180B, which incorporates its acceptable use policy, prohibits users from hosting the model with an API.²⁴

⠀

Restrictions on industry-specific uses:

A number of acceptable use policies include restrictions that prevent firms in certain industries from making use of the related foundation models.²⁵ Nevertheless, companies in these industries may succeed in using foundation models by negotiating permissive contracts with the developer. This creates an information asymmetry where the company and its clients are aware of the domains in which foundation models are being used, while regulators and the public may be led to believe that model use is based solely on the publicly disclosed acceptable use policy.

Common examples include restrictions on the following industries:

• Weapons manufacturers: Several acceptable use policies prohibit the use of models for the development of weapons, with some developers’ policies restricting many different specific types of weapons. Although the use of these foundation models by weapons manufacturers would violate the developers’ publicly stated acceptable use policy, it is possible that developers negotiate custom contracts with weapons manufacturers.
• Legal, medical, and financial advice: 8 of the 30 acceptable use policies prohibit users from leveraging foundation models for use in highly-regulated industries, such as providing legal, medical, and financial advice. This prohibition applies not only to lawyers, doctors, and financial advisers, but also to the many organizations that provide informal advice in these fields.
• Surveillance: The Allen Institute for AI, Anthropic, Amazon, Google, and OpenAI prohibit use of their models for surveillance to some extent. For instance, Google prohibits use of its models for “tracking or monitoring people without their consent” while AI2 singles out “military surveillance.” This would theoretically prevent spyware companies and defense and intelligence contractors respectively from making use of such foundation models.

⠀

Foundation models without acceptable use policies:

There are tens of foundation model developers besides the 30 described above and many do not have acceptable use policies for their models.²⁶

Some open foundation model developers do not use acceptable use policies because their models are intended for research purposes only—if they were to adopt use restrictions, it could deter researchers from conducting safety research through red teaming, adversarial attacks, or other means (in the absence of an exemption for research). Other models intended for research may lack acceptable use policies on the basis that they present less severe risks of misuse, whether because they have less significant capabilities or fewer users.²⁷ Non-commercial models such as these are frequently distributed using licenses without use restrictions such as Apache 2.0 or Creative Commons Attribution-NonCommercial licenses. While a model may not include any use restrictions for noncommercial users, commercial users may have to agree to custom use restrictions in their contracts with the model developer.

Models for commercial use may not include acceptable use policies for several reasons.²⁸ In some cases, models are not intended for commercial use without further fine tuning or other safety guardrails, leading developers to offer the model as is, leaving downstream developers to restrict uses (e.g., Databricks’ MPT-30B). Other developers release their models without complete documentation, whether because they intend to release an acceptable use policy at a later point, which could be part of staged release, or due to under-documentation in the rush to release a model.

Nevertheless, foundation models without acceptable use policies may be covered by other kinds of restrictions. Alibaba Cloud restricts companies with over 100 million users from making use of Qwen-VL through its license, which also bans model scraping. Restrictions on who can use a foundation model may have a significant effect on how it is used even in the absence of binding restrictions on certain categories of use.

Discussion:

Governments have attempted to spur self-regulation in the foundation model ecosystem through voluntary codes of conduct.²⁹ The G7 and member states such as the US and Canada have issued codes of conduct and received pledges from leading foundation model developers. The G7’s code of conduct states that foundation model developers should “take appropriate measures… [to] mitigate risks across the AI lifecycle” including risks related to bias, disinformation, privacy, and cyber. Canada’s code of conduct requires developers to create “safeguards against malicious use” while the White House’s “Voluntary AI Commitments” include a commitment to “prioritize research on societal risks posed by AI systems” such as risks to children.

However, these codes of conduct stop short of recommending that foundation model developers enact binding restrictions on risky use cases. Foundation model developers have gone beyond voluntary commitments, shielding themselves against legal liability for misuse of their models. In this way, acceptable use policies are a significant form of self-regulation.

Still, there is little publicly available information about how acceptable use policies are actually enforced. Although companies make the prohibited uses of their models clear, it is often unclear how they enforce their policies against users. Just as the 2023 Foundation Model Transparency Index found that 10 leading foundation model developers share little information about how they enforce, justify, and provide appeals processes for acceptable use policy violations, this work finds that other foundation model developers provide little or no information about these matters. This lack of transparency is different from other digital technologies; social media companies, for instance, regularly release transparency reports that provide details about how they enforce their acceptable use policies and other provisions in their terms of service. Without information about how acceptable use policies are enforced, it is not obvious that they are actually being implemented or effective in limiting dangerous uses. Companies are moving quickly to deploy their models and may in practice invest little in establishing and maintaining the trust and safety teams required to enforce their policies to limit risky uses.

Another issue in gauging the enforcement of acceptable use policies is the way in which they propagate across the foundation model ecosystems. Companies’ publicly available policies often do not specify precisely how their acceptable use policies propagate to third parties such as cloud service providers, and whether third parties have obligations to enforce the original acceptable use policy. In addition to developers, cloud service providers act as deployers of foundation models that were not developed in-house. Cloud service providers have their own acceptable use policies that do not align perfectly with external developers’ acceptable use policies, and it is not clear that a cloud provider would have adequate expertise to restrict the uses of the foundation models in accordance with acceptable use policies that are more stringent than that of the cloud provider.³⁰

Acceptable use policies may impose obligations on users that they are ill equipped to follow. Acceptable use policies are a means of distributing responsibility for harm to users and downstream developers; however, these other actors may not be well suited to prevent certain uses of a developer’s model.³¹ Restricting users from generating self-harm related content, for instance, only works for some types of users—for example, vulnerable users who are turning to a language model for advice may trigger acceptable use policies by openly sharing their mental health issues.

Questions regarding the enforceability of acceptable use policies extend beyond users who cannot self-enforce. While many acceptable use policies are reasonably comprehensive, the degree to which developers can enforce their policies differs across context and jurisdiction. Enforcement of acceptable use policies differs based on how models are deployed, whether through an API, a user interface operated by the developer, or locally. Strict enforcement is near impossible when models are deployed locally, whereas it resembles content moderation for social media platforms when models are deployed via an API.

Nevertheless, many open foundation model developers attempt to restrict the use of their models to some degree. Twelve of the developers examined that have acceptable use policies openly release their model weights (and other assets in some cases), but do so using licenses or terms of service that block certain unacceptable uses. Although open foundation models are frequently referred to as “open-source” in popular media, truly open-source software or machine learning models cannot have use restrictions by definition.

While many previous works have taxonomized the risks stemming from foundation models, this post assesses how companies taxonomize risk on the basis of their own policies. Future research directions include assessing the enforcement of acceptable use policies and comparing acceptable use policies to model behavior policies, toxicity classifiers, responsible scaling policies, and existing AI regulations.

Key takeaways for policymakers:

1. Developers’ acceptable use policies have important differences

Acceptable use policies differ across developers in terms of restrictions on content, types of end use, and industry-level restrictions. These differences have been underexplored and may have important implications for the safety of foundation model deployment.

• There are several gray areas where some companies include content-based restrictions and others do not. Content related to politics, eating disorders, sex, and medical advice are among the areas where some companies have harsh prohibitions and others are silent.
• While some companies’ policies prevent their models from being used by the weapons or surveillance industry, others have few industry-related restrictions, and others release only noncommercial models with no policies limiting use.

⠀ 2. Acceptable use policies help shape the foundation model market

Acceptable use policies alter the foundation model market by affecting who can use a model and for what purpose.

• Foundation model developers advertise use restrictions or lack thereof as a comparative advantage relative to other companies. For example, Mistral says its models are “customizable at will,” allowing users to adapt its models more fully, whereas OpenAI states it provides safety to users and the ecosystem through “post-deployment monitoring for patterns of misuse”.
• Foundation model developers employ acceptable use policies in order to prevent other companies from making use of their services, stealing their intellectual property, or using their products to develop a competitor.³² For example, companies ban firms and other users from using their models to train other AI models, restricting the supply of datasets of model outputs and concentrating the market for models that are trained on model outputs.³³
• Acceptable use policies help determine what industries can make use of developers’ foundation models. For example, policies that prohibit the use of models for weapons production may block the arms industry from making use of those foundation models.
• Acceptable use policies determine the types of uses that are legitimate for foundation models (e.g., no automated decision systems). Even industries that are allowed to make use of models may not be able to do so for common applications.

⠀

3. Policy proposals to restrict the use of foundation models should take acceptable use policies into account

Many AI regulatory proposals do not acknowledge that companies already limit the use of their foundation models with acceptable use policies. Policymakers should craft regulations that contend with existing restrictions in these acceptable use policies, including by potentially reinforcing them or by helping to close gaps in enforcement regimes.

• Among the most common policy proposals related to foundation models are those that propose to restrict specific uses. Policies such as liability for model developers and export controls on model weights are primarily intended to block undesirable uses of foundation models.
• These policies should account for the fact that foundation model developers already have policies in place that prevent such risky uses. For example, every foundation model developer restricts unlawful uses, a broad restriction that governments could encourage developers to enforce in specific ways.
• Enforcement of acceptable use policies (and companies’ other policies) is piecemeal and quite difficult. Policymakers should encourage and assist in enforcement of existing policies in addition to imposing new requirements.

Acknowledgments

I thank Ahmed Ahmed, Rishi Bommasani, Peter Cihon, Carlos Muñoz Ferrandis, Peter Henderson, Aspen Hopkins, Sayash Kapoor, Percy Liang, Shayne Longpre, Aviya Skowron, Betty Xiong, and Yi Zeng for discussions on this topic. A special thanks to Loredana Fattorini for her assistance with the graphics.

@misc{klyman2024aups-for-fms, 
    author = {Kevin Klyman}, 
    title  = {Acceptable Use Policies for Foundation Models: Considerations for Policymakers and Developers}, 
    month = april,
    year   = 2024, 
    url    = {https://crfm.stanford.edu/2024/04/08/aups.html}
}