Meta & Children's Online SafetyChat: [P]eople who are suffering from depression and self-harm go down IG rabbit holes, and explore functionality compounds this issue.
Thread View
████ name redacted. Different bars may be the same person or different people.
quoted in an excerpt in the archive.
Tuesday 21 April 2020
16:16████Hey product friends. We just chatted on the policy side about sensitive content control. We think it makes sense to draw the line at likely violating with the agreement that the control will *not* have any impact on entity-level filtering we have in place. This would mean that our account-level enforcement for SSI (e.g. you are not Explore eligible if you've been checkpointed in the prior 30 days for SSI, including for non-violating admission content) *should solve the SSI issue does this sound right from a product standpoint? Is there an additional "general SSI" filter we use today on Explore that I might not be aware of? That means the only remaining open Q would be holocaust denial, which as will mentioned in his prior email is more of an academic debate given we dont have enforcement for it today.
16:19████If this sounds right, we're going to socialize with Monika first to get her alignment
16:21████Does anyone have flags with the accuracy of the above? or other flags before we chat with monika?
16:23████we do have an SSI filter on Explore, I'm trying to confirm how significant it is
16:23████Thanks and it's a non-violating filter?
16:24████In general, I'm on board with likely violating being the line. I would appreciate your support in advocating for the iterative approach to get there vs. the cold turkey approach.
16:24████If you're going to respond on that thread
16:24████Yes, agree on the iterative approach we definitely can't cold turkey without proper violating classifiers
16:25████not going to respond on the thread yet first we're going to socialize with Monika
16:25████this line is more "north star" than "day one"
16:26████Ok. Please say that on the thread when someone responds bc i think the current convo is conflating the two.
16:27████One comment: I wonder if there is a better way to frame the entity level thing you are trying to get at.
I'm worried that it won't resonate with Adam bc I thinks we should have a higher bar for entity level stuff and give people more control be it's more of a blunt instrument. I see what you are trying to do but maybe there is a different way to frame it around Co actions vs. entity level stuff. For example, account tiering takes into account CO actions but also non-rec classifiers.
16:28████Yes, that's a good point. Are we going to consider allowing people to opt out of filtering based on tiering though?
16:28████I can say "entity level filtering based on CO actions"
16:28████to be more precise
16:28████but again, not responding on the thread yet to be clear
16:29████Under what policy does CO action these things? It does muddle the line a bit.
16:29████SSI Admission
16:30████This content has sensitivity screens on it right? If so, that could be another way to draw the line.
16:30████what we're saying is at the CONTENT level, you can opt into seeing non-rec stuff so long as it is NOT likely violating. However, the control does NOT allow you to opt in to see any content from entities that are being filtered at the account level
16:30████so the distinction is that the control applies strictly to content level filtering, not entity level filtering
16:31████I'm afraid it we get into details on the entity piece, we'll derail again. So if we can agree on this as the scope of the control today, we can revisit in the future if we want to bring entity level in scope and, if so, what lines we draw
16:31████Is the basis of the distinction that the entity filter is a strikes-based penalty. like how these accounts also can't go live?
16:32████I get that but the reason we do the entity filtering is to filter out more content so im not sure that argument holds up. since it's more of a blunt instrument too.
16:32████it's not really strikes based SSI admission deletions don't give strikes, but it's similar
16:32████The idea that we would do entity enforcement based on take downs of violating content seems fine to me bc that's the violating bar.
16:32████Ok, so if we don't think entity level is a good option, then we're going to need to discuss further
16:32████it's not a violating bar though
16:32████But it's the entity filtering that's at some higher bar that seems a bit tricky to rationalize
16:32████SSI Admission is not violating
16:32████that's the problem
16:33████Yeah. Is SSI Admission the only example here?
16:33████It's the one we're trying to solve for, yes
16:33████So entity level filtering based on CO action
16:33████could be the line
16:33████we don't allow you to opt in to seeing entities we've filtered based on a CO action
16:34████but if it's based on something else, like high volume of non rec content, theoretically, we'd allow opt out of that
16:34████I'm willing to go for it. Just telling you I see the hole in the argument. But might be the best one.
16:34████What is the hole if we use the above A
16:34████That the filtering by Co is done with a bar higher than the violating/likely violating bar.
16:36████Yes so we could say "we allow you to opt out of filtered content under our Rec policy so long as it is not likely violating NOR CO actioned (e.g. MAS or MAD screen). The same applies to entity-level filtering (e.g. if your account was recently SSI checkpointed for admission content, it will remain filtered):
16:36████Does this close the gap?
16:37████the line: you can't opt of action taken under our Community Guidelines
16:38████opt out of
16:40████I think that's a fine line to take. It's a bit complex when you dig in but not sure anyone will.
The other options are:
(1) Make the bar something a bit higher than "likely violating" more like "Borderline" where it's a bit higher than our violating line but not significantly higher like much of our rec guidelines and make the case that SSI admission is very close to the violating content in this category.
(2) Let people control SSI admission. Adam might be ok with that in a world where the default is that it's out and I think it's in the spirit of what we committed to.
16:41████borderline is what we'll be doing almost all filtering on borderline nudity sexually suggestive
16:41████so it we dont allow people to opt out of that then they essentially get to opt out of nothing today
16:42████Maybe that's not the right word to use in this context. In the nudity example I mean something that is so close to nudity but is cut off or covered up in the right place.
16:42████That'd mean we need to create a third policy line
16:42████and define it
16:42████and build classifiers for it
16:43████That's basically what our classifiers for violating would catch right?
16:43████?
16:43████So how is it different from "likely violating"?
16:43████And how do we explain it externally?
16:44████I guess in the case of SSI we would have to say admission is so close to violating that it falls into the "likely violating" bucket even if it's not violating.
16:44████It seems pretty straightforward to me to say: We filter things based on whether they are likely violating our Community Guidelines or have been actioned under our Community Guidelines. You cannot opt out of that filtering. We also filtering things based on this Rec policy. You can opt out of those.
16:45████This feels more straightford than the path I led us down.
16:45████I'm also a big fan of this phrasing
16:45████Can you point to somewhere in the public facing community guidelines that talks about SSI admission?
16:47████https://www.facebook.com/communitystandards/suicide_self_injury_violence
16:49████This is when we checkpoint an account: "We provide resources to people who post written or verbal admissions of engagement in self injury, including:
-Suicide -Euthanasia/assisted suicide -self-harm -Eating disorders" <
16:50████+1, I think this framing makes sense.
In terms of SSI risk if we were to do this, we do have a non-rec SSI filter in place on Explore today targeting depiction. It's an old classifier with low precision and low filter volume so unlikely to materially change ecosystem but may result in some SSI admission content appearing if we removed it
16:52████Given that we wouldn't go cold turkey, however, I'd expect we'd wait to remove this until we have likely violating mitigations in place
16:52████"we provide resource to people...and btw prevent their content from getting distribution"
16:54████ok so generally are people aligned with this approach? I'm going to let people vote via thumbs up on the options:
16:54████aligned
16:54████not aligned
16:54████haha
16:54████(they created a workchat polling option for this ^)
16:55████The more I think about this, the more I feel like the right approach is to just not filter SSI admission for people with the setting off. I'm not going to fight that battle but a pure version of the explanation is the only one that really lands well to me.
16:55Karina Lynn Newtoni am really not ok with that
16:55████I agree that the option of setting the bar higher than "likely violating" is the worst option
16:55Karina Lynn Newtonand this is my problem with describing this as about borderline content
16:57Karina Lynn Newtoni think the holocaust denial example is an academic example that applies to very little content
16:57████I feel like you're just making the case to not provide this control then. Which is an argument that also resonates with me. I'm just worried that we are rationalizing this strange middle ground that will be too hard to defend.
16:57████That being said, I will fully back your proposal outside of this thread.
16:58Karina Lynn Newtonha
16:59Karina Lynn Newtonmy issue is this: people who are suffering from depression and self-harm go down IG rabbit holes, and explore functionality compounds this issue. this is why adam published a national op-ed in the uk to say we won't recommend ssi content anymore, this isn't an academic example. and it's not responsible to give young people a way to opt out of this.
17:00████I'm also totally fine not providing this setting for people under 18 if that helps. Issues could be that we don't have age for many and could create incentives to lie about age.
17:00████I'd actually let you make the call there independent of this other convo
17:01████If you have an opinion
17:01████It's something ive brought up with
17:01Karina Lynn Newtonwe also only have age for 55ish percent of people
17:02Karina Lynn Newtonbut we will have new teen/non-teen classifiers we could use to gate this control
17:02Karina Lynn Newtonin july
17:03Karina Lynn Newtonbut i like something on age, despite the incentives to lie
17:03████Not sure we should do it on a classifier and make the classifier prediction so obvious in the product.
But totally fine with doing it on stated age
17:03████I totally agree with you. Do you think people who are struggling are so likely to want to go down this rabbit hole that they will see out the setting and turn it off?
17:04Karina Lynn Newtonyes, sadly
17:05████hmm, ok, if we feel so strongly about this one issue, i agree that proposal is the best one.
17:05████Shared an image
17:05████:)
17:05████To state on the record, I still remain more in the camp of either doing this and being ok with the implications being kind of extreme or not doing it at all.
17:07████I see everyone trying to water it down so much. Do you all want to actually fight for not doing it?
17:07████I dont actualyl think it's watering it down to say that the topics we cover in CG are not opt-out-able
17:07Karina Lynn Newtoni think it's a non-starter to argue back on feed
17:07Karina Lynn Newtonand i think why this gets tricky is because we're not doing separate controls
17:08Karina Lynn Newtonwhich isn't to say that wasn't the right call
17:08Karina Lynn Newtonbut it puts additional pressure
17:08████You will be able to opt out of a ton of stuff: borderline nudity, borderline drugs, borderline guns, borderline spam, etc
17:08████those things are all contentious
17:08████and we're giving a user the control
17:08████When I say not doing it at all, I mean making the control just about downranking in Home.
17:09████And most of our "shadowban" claims are about those topics, not about SSI admission...
17:12████BTW, I was wrong earlier.
helped confirm w/ eng on the SSI team and the SSI filter we're using is a violating classifier. So there would be no change in SSI classifiers on Explore if we took this approach
17:12████HOORAY
17:13████for future state, worth talking about how this would work for the future SSI borderline classifier? (it was planned for H2'20 might be pushed back due to COVID)
17:13████I'm fine trying to land 's proposal and the iterative approach (vs. cold turkey). If that doesn't land, let's reevaluate.
17:15Karina Lynn NewtonSo...
Y Option 1:
Ineligible for opt-out: Content actioned by CO or eligible to be actioned by CO [MAD, MAS, SSI, or determined by classifier to be likely violating our Community Guidelines] Option 2:
Option 1 + people under 18 don't get the control
17:16████I like Option 2
17:22Karina Lynn Newtoni think option 2 releases some policy pressure and is a place that can help guy get to comfortability without overcomplicating
17:25████the incentive to lie about age is the main concern I could see about restricting the control based on age.
any other concerns we can think of or is it a no-brainer?