In October 2023, we launched the Foundation Model Transparency Index (FMTI), defining 100 concrete transparency indicators to articulate how we conceptualize transparency and assessing 10 major foundation model developers against these indicators. In December 2023, we announced the advisory board for FMTI (Arvind Narayanan, Daniel E. Ho, Danielle Allen, Daron Acemoglu, Rumman Chowdhury). Today, we announce our plans for the v1.1 of the Index.

Introduction

Foundation models are transforming society and the economy: their broad impact necessitates transparency to the public. We created the Foundation Model Transparency Index (FMTI) to make progress on transparency in two ways. First, while transparency is often an amorphous buzzword, we concretize transparency by defining 100 transparency indicators that span how foundation models are built (e.g. data, labor, compute), the models themselves (e.g. capabilities, risks, evaluations), and their downstream use (e.g. distribution channels, usage policies, affected geographies). By providing arguments and citations for why these indicators are valuable, we make precise a well-defined set of information that foundation model developers could disclose to benefit a variety of stakeholders (e.g. deployers, end-users, researchers, governments). Second, while transparency is often a passive desideratum, we proactively score 10 major foundation model developers to establish what the overall status quo is for transparency. By scoring developers based on their current practices, we make precise what dimensions of foundation models are systemically opaque (e.g. labor practices, downstream usage) and what dimensions are more idiosyncratic (i.e. where some developers disclose information while others do not).

Much as transparency is an instrument for other societal objectives, the FMTI is also only one piece of the puzzle in creating a foundation model ecosystem that more resolutely advances the public interest. Thus far, we have seen extensive media coverage of the opacity of foundation models citing FMTI, bringing greater awareness to the need for greater transparency. Since the release of FMTI, US Representatives Donald Beyer and Anna Eshoo introduced the AI Foundation Model Transparency Act to “direct the Federal Trade Commission to establish standards for making publicly available information about the training data and algorithms used in artificial intelligence foundation models”. Further, improved transparency for foundation models features in policies across the US, EU, Canada, and the G7. This is also a priority of new AI governance bodies like the UK AI Safety Institute, US AI Safety Institute, and EU AI Office. As the societal impact of foundation models becomes more clear, the ecosystem becomes more complex, and our own understanding of transparency evolves, we will need to change the Index to make sure its operationalization continues to match its mission.

v1.1

As a follow up to the initial 2023 Foundation Model Transparency Index (v1.0), our plan for the next release is v1.1 in April 2024. Here, we detail the key changes for v1.1 relative to v1.0.

As a recap, here is a succinct description of our process in v1.0:

1. Companies for assessment. We selected 10 major foundation model developers and designated a flagship foundation model for each.
2. Information gathering. We systematically gathered information made publicly available by the developer for their flagship model.
3. Initial scoring. We scored each developer on the 100 transparency indicators by having two researchers provide initial scores, resolving disagreements, and then ensuring the same standard was upheld horizontally across all developers for a given indicator.
4. Developer rebuttals. We shared the initial scores with each developer, allowing them to contest the scores.
5. Final results. We reviewed the rebuttals, making changes where the rebuttal had merit and re-standardizing across developers.

In v1.1, we use the same 100 indicators and maintain a similar process, changing steps 1 and 2.

1. Companies for assessment (outreach). We expanded to consider 18 major commercial foundation model developers, adding 8 to the original 10 in v1.0, and reached out to their leadership between December-January. Of these, all 18 responded and, to date, 11 indicated they would likely/certainly participate in the FMTI process, 4 are looking into whether they have capacity, and 3 declined due to lack of capacity. Rather than having us designate the flagship foundation model, we instead allowed the developer to designate their flagship foundation model as of February 2024.
2. Information (self-assessment). Instead of gathering information that is already public, developers will instead assess themselves against the 100 indicators by disclosing the relevant information. As a result, developers may choose to provide information that is not already public, but that will become public through this process.

We have made these changes for four reasons, in part based on feedback to the first iteration of the Index. First, by allowing developers to determine their flagship model, we reduce disagreement on whether our choices are appropriate. Second, by having developers provide information, we eliminate the concern that information was publicly available but we missed it in our systematic search process. Third, by allowing developers to include information that is not yet public, we encourage greater deliberation on information disclosure and provide opportunities to disclose information that developers previously did not consider disclosing. Moving forward, we envision this process as the basis for standardized transparency reporting, in line with recommendations from the White House and the G7 for foundation model developers to provide transparency reports.

Timeline

Based on the changes we made for v1.1, we expect the overall FMTI iteration time to reduce, which is desirable given the fast-evolving nature of the foundation model ecosystem. Here is our projected timeline with public release for v1.1 aimed at April 2024, 6 months after the initial October 2023 release.

• Outreach (December 2023 - January 2024). We engaged the 17 foundation model developers to help them understand our effort, answer their questions, and address potential concerns/confusion.
• Self-assessment (mid February 2024). Developers submit their self-assessments for their flagship models.
• Initial scoring (early March 2024). The Index team will review all self-assessments, scoring them and ensuring horizontal standardization, and return the reviewed assessments to the developers.
• Developer rebuttal (mid March 2024). Developers rebut any changes.
• v1.1 launch (April 2024). The Index team will launch v1.1, which includes the final scores and underlying data, as well as the developer self-assessments.

Acknowledgements

We thank the FMTI advisory board for their guidance on the v1.1 plan.