In October 2023, we launched the Foundation Model Transparency Index (FMTI), defining 100 concrete transparency indicators to articulate how we conceptualize transparency and assessing 10 major foundation model developers against these indicators. In December 2023, we announced the advisory board for FMTI (Arvind Narayanan, Daniel E. Ho, Danielle Allen, Daron Acemoglu, Rumman Chowdhury). Today, we announce our plans for the v1.1 of the Index.
Foundation models are transforming society and the economy: their broad impact necessitates transparency to the public. We created the Foundation Model Transparency Index (FMTI) to make progress on transparency in two ways. First, while transparency is often an amorphous buzzword, we concretize transparency by defining 100 transparency indicators that span how foundation models are built (e.g. data, labor, compute), the models themselves (e.g. capabilities, risks, evaluations), and their downstream use (e.g. distribution channels, usage policies, affected geographies). By providing arguments and citations for why these indicators are valuable, we make precise a well-defined set of information that foundation model developers could disclose to benefit a variety of stakeholders (e.g. deployers, end-users, researchers, governments). Second, while transparency is often a passive desideratum, we proactively score 10 major foundation model developers to establish what the overall status quo is for transparency. By scoring developers based on their current practices, we make precise what dimensions of foundation models are systemically opaque (e.g. labor practices, downstream usage) and what dimensions are more idiosyncratic (i.e. where some developers disclose information while others do not).
Much as transparency is an instrument for other societal objectives, the FMTI is also only one piece of the puzzle in creating a foundation model ecosystem that more resolutely advances the public interest. Thus far, we have seen extensive media coverage of the opacity of foundation models citing FMTI, bringing greater awareness to the need for greater transparency. Since the release of FMTI, US Representatives Donald Beyer and Anna Eshoo introduced the AI Foundation Model Transparency Act to “direct the Federal Trade Commission to establish standards for making publicly available information about the training data and algorithms used in artificial intelligence foundation models”. Further, improved transparency for foundation models features in policies across the US, EU, Canada, and the G7. This is also a priority of new AI governance bodies like the UK AI Safety Institute, US AI Safety Institute, and EU AI Office. As the societal impact of foundation models becomes more clear, the ecosystem becomes more complex, and our own understanding of transparency evolves, we will need to change the Index to make sure its operationalization continues to match its mission.
As a follow up to the initial 2023 Foundation Model Transparency Index (v1.0), our plan for the next release is v1.1 in April 2024. Here, we detail the key changes for v1.1 relative to v1.0.
As a recap, here is a succinct description of our process in v1.0:
In v1.1, we use the same 100 indicators and maintain a similar process, changing steps 1 and 2.
We have made these changes for four reasons, in part based on feedback to the first iteration of the Index. First, by allowing developers to determine their flagship model, we reduce disagreement on whether our choices are appropriate. Second, by having developers provide information, we eliminate the concern that information was publicly available but we missed it in our systematic search process. Third, by allowing developers to include information that is not yet public, we encourage greater deliberation on information disclosure and provide opportunities to disclose information that developers previously did not consider disclosing. Moving forward, we envision this process as the basis for standardized transparency reporting, in line with recommendations from the White House and the G7 for foundation model developers to provide transparency reports.
Based on the changes we made for v1.1, we expect the overall FMTI iteration time to reduce, which is desirable given the fast-evolving nature of the foundation model ecosystem. Here is our projected timeline with public release for v1.1 aimed at April 2024, 6 months after the initial October 2023 release.
We thank the FMTI advisory board for their guidance on the v1.1 plan.