Initial Statement of the Facebook Data Transparency Advisory Group

In May 2018, Facebook shared its preliminary Community Standards Enforcement Report, which published a range of summary measures Facebook uses to assess its effectiveness in enforcing its Community Standards. These Community Standards set rules prohibiting certain types of content from being posted on Facebook. Facebook chartered the Data Transparency Advisory Group (“DTAG”) to:

• assess Facebook’s measurement approach underlying the report,
• provide recommendations for how to improve its measurement and reporting practices, and
• produce a public report on its efforts.

The DTAG is currently reviewing Facebook’s metrics for measuring and reporting violations of the Community Standards and how Facebook is applying them—i.e., whether it does so accurately, consistently, and effectively.

Process / What Was Provided

Facebook has provided detailed information on:

• how it measures the prevalence of content it has labeled as violating the Community Standards, and
• how often it takes action on this content.

Facebook has also provided significant access to the staff who developed these metrics so they could explain design choices to the DTAG.

Preliminary Findings & Recommendations

Overall, the DTAG notes the significant effort Facebook has put into enforcing its Community Standards and measuring its enforcement efforts. Deciding whether content violates the standards frequently involves difficult judgments about context and intent, as well as tradeoffs between competing values and significant statistical challenges given the volume of activity on the platform and apparent rarity of some kinds of violations. The DTAG has encouraged Facebook to be more open with users about these challenges.

In its preliminary assessment, the DTAG has found the formulas used for calculating prevalence metrics are appropriate given constraints such as the volume and variability of content. The formulation of ‘prevalence’ underlying Facebook measures comports with the broad public’s understanding of crime victimization and is an appropriate metric for assessing how significant the Community Standards enforcement challenge actually is for Facebook.

The DTAG has offered several recommendations for making Facebook’s published metrics more meaningful, for minimizing error, and for measuring additional outcomes. Facebook has indicated it intends to incorporate some of the DTAG’s recommendations into future versions of the Community Standards Enforcement Report. The roadmap for future measurement and reporting efforts laid out by Facebook is sensible. If followed over the next year it will provide significantly greater transparency for the public regarding both the extent of community standards violations and Facebook’s actions to address them.

Caveats

First, at present, the DTAG does not have the information needed to assess whether content which Facebook counts as violating the Community Standards is actually a violation of those standards as written. Facebook has committed to providing the information the DTAG needs to assess the accuracy and reliability of Facebook’s application of its own standards, and evaluate whether Facebook is employing best practices to ensure the accuracy and reliability of decisions about whether content violates the Community Standards. Additional details regarding the full assessment and associated recommendations will be included in the group’s final public report.

Second, the DTAG is evaluating Facebook’s enforcement of its Community Standards, as written, and the metrics Facebook has chosen for evaluating its enforcement efforts. Facebook’s policies with respect to collecting, storing, sharing, or selling users’ personal data are beyond the predetermined scope of the DTAG’s assessment. Facebook has not provided any information on its data collection and privacy practices, nor has it asked the DTAG for feedback on these.

Third, the DTAG’s assessment thus far has been limited to Facebook’s methods for calculating the metrics Facebook chose to include in its Community Standards Enforcement Report. The DTAG has not determined that the information in the Community Standards Enforcement Report is sufficient for Facebook to be transparent and accountable to the public. Nor has it determined that this information is the most meaningful or important information to share with users who are concerned about Facebook’s practices. The DTAG’s final report will include recommendations about what additional information Facebook would need to share, and what other measures FB should take, in order to be more transparent and accountable to users.

About the DTAG

The DTAG is led by The Justice Collaboratory at Yale Law School, directed by faculty members Tom L. Tyler and Tracey R. Meares. Its members include academics from several universities who are experts in measurement and the role that metrics play in building legitimate, accountable institutions.

Members of the DTAG are:

Ben Bradford, Professor, Department of Security and Crime Science, University College London

Florian Grisel, Research Fellow, French Centre National de la Recherche Scientifique and Reader in Transnational Law, King’s College London

Tracey L. Meares, Walton Hale Hamilton Professor of Law, Yale Law School

Emily Owens, Professor, Department of Criminology, Law and Society, and Department of Economics, University of California, Irvine

Baron L. Pineda, Professor of Anthropology and Latin American Studies, Oberlin College

Jacob N. Shapiro, Professor of Politics and International Affairs, Princeton University

Tom R. Tyler, Macklin Fleming Professor of Law and Professor of Psychology, Yale Law School