To improve the accuracy of decisions made upon secondary review, Meta should assess how its current review routing protocol impacts accuracy. The Board believes Meta would increase accuracy by sending secondary review jobs to different reviewers than those who previously assessed the content. The Board will consider this implemented when Meta publishes a decision, informed by research on the potential impact to accuracy, whether to adjust its secondary review routing.
Our commitment: We have ongoing monitoring mechanisms in place which assess how review routing protocol and enforcement decisions impacts accuracy across reviewers. We are continuously working to refine and improve how systems impact our full set of enforcement metrics, including accuracy.
Considerations: Meta has review protocols in place to ensure that secondary review is allocated to a different reviewer other than the initial reviewer to the most feasible extent. As shared in our response to
PAO on Meta’s Cross-Check Policies #31, we have an internal system called
Dynamic Multi Review (DMR) which enables us to review certain content multiple times, by different reviewers before making a final decision. This ensures that the quality and accuracy of human review is carefully considered upon secondary review, taking into account factors such as virality and potential for harm.
We have a dedicated global operations measurement team which monitors enforcement decisions across all content types. This team monitors the accuracy and quality of our review decisions to ultimately develop integrity metrics that validate our review processes. We do this through protocols such as audit validation which ensures that our accuracy metrics can be trusted on an ongoing basis and remain in alignment with the source of truth. Our operational measurement teams also conduct engagement with scaled reviewers to maintain validation across our metrics, monitor tooling and triaging to report and address malfunctions in our tools on an ongoing basis and generate insights to consistently improve review accuracy.
In practice, decisions that are escalated to secondary review are processed through channels that will ensure at least one additional reviewer other than the initial assessor may be allocated. Oftentimes more than one additional reviewer is allocated, depending on factors including the violation type, associated account tags and accumulated views. We have also enabled robust appeals processes which have recently enabled users to appeal decisions directly to the board per our response to
PAO on Meta’s Cross-Check Policies #25. Additionally, our appeals processes default to secondary review from reviewers who are different to those who previously assessed the content, unless determined otherwise by capacity constrained exceptions.
We are constantly iterating on our routing protocol and monitoring our accuracy metrics. We continue to strive towards ensuring that our secondary review routing systems strengthen our enforcement accuracy. We now consider this recommendation complete and will have no further updates on this work.