Pinterest: Pinterest automatically detects content that promotes self-harm or suicide and removes it. When a user searches for terms related to emotional distress or self-harm, the platform intercepts the search and shows wellbeing resources — such as breathing exercises and crisis hotline information — instead of or alongside regular results. The crisis search feature, called Compassionate Search, was developed with Stanford Medicine's Brainstorm Lab and crisis organizations, and launched in July 2019. | AI Trace
Content ModerationConsumer FacingVerified
Pinterest automatically detects content that promotes self-harm or suicide and removes it. When a user searches for terms related to emotional distress or self-harm, the platform intercepts the search and shows wellbeing resources — such as breathing exercises and crisis hotline information — instead of or alongside regular results. The crisis search feature, called Compassionate Search, was developed with Stanford Medicine's Brainstorm Lab and crisis organizations, and launched in July 2019.
Details
Machine learning classifiers identify self-harm and suicide-related content across the full Pin corpus. Compassionate Search uses a separate keyword-based and machine learning system to detect distress signals in search queries; over 4,600 terms and phrases related to self-harm are blocked from surfacing regular results. Pinterest has stated that interactions with wellbeing resources through this feature are deliberately not linked to user accounts and are not used for ad targeting. By H1 2023, quarterly removals for self-harm content ran into the tens of millions of Pins, with 95% of removed Pins having been seen by fewer than 10 users before removal.
Products affected
Pinterest searchPinterest home feedPinterest BoardsPinterest recommendations