Meta & Children's Online Safety

Documents / Teen EN Country iOS Take a Break Upsell Experiment Analysis -- Final Results

Teen EN Country iOS Take a Break Upsell Experiment Analysis -- Final Results

Internal document
Open source PDF (49 pages)

Transcribed text as recorded, including headers and formatting fragments.

Case 4:22-md-03047-YGR Document 2758-6 Filed 02/20/26 Page 1 of 49 AMENDED Exhibit 266 PLAINTIFFS' OMNIBUS OPPOSITION TO DEFENDANTS' MOTIONS FOR SUMMARY JUDGMENT Case No.: 4:22-md-03047-YGR MDL No. 3047 In Re: Social Media Adolescent Addiction/Personal Injury Products Liability Litigation t Case 4:22-md-03047-YGR Document 2758-6 Filed 02/20/26 DOCUMENT SLIPSHEET Bates Number: META3047MDL-035-00002791 Custodian: CASTANEDA_DIEGO;DOW_ALEX;META PLATFORMS;NEWTON_KARINA;OTWAY_STEPHANIE;ROTHSCHILD_MIKI Author: SHARED DRIVE (UNAVAILABLE) Filename: 2021-11-29 TEEN EN COUNTRY IOS TAKE A B 1BNFLZ- QIZ7A163N17ADGLE7DD3UW9VZEGZY1D-DBSVK.PPTX Create Date: 11-22-2021 12:00 AM Last Modified Date: 1-6-2023 12:00 AM TNAG BEGBATES: TNAG ENDBATES: MAAG BEGBATES: MAAG ENDBATES: depobook.com Page 2 of 49 EXHIBIT META- Castaneda-31 Case 4:22-md-03047-YGR Document 2758-6 Filed 02/20/26 Page 3 of 49 Teen EN Country iOS Take a Break Upsell Experiment Analysis -- Final Results Instagram Mental Well-Being Nov 29, 2021 A/c priv TAKE A BREAK MVP Case 4:22-md-03047-YGR Document 2758-6 Filed 02/20/26 Page 4 of 49 Upsell Experience Overview Teens with app sessions longer than 10 or 20 minutes (the two experiment variants) are eligible for the upsell. However, the upsell can only be shown in an eligible Explore chaining surface. The in-line upsell will prompt them to turn on their break reminders. If they select "turn on", we take them to the break reminder settings page where they have the option of setting 10, 20, or 30 minute app session duration break reminders. If they dismiss the upsell, we remind them that they could turn on reminders anytime and share the link to the "Your Activities" page. Similar Posts 9:41 < lil lapisla golden hours kenzoere Great Shot! View all 5 comments Add a comment இ 1 day ago Want a break? Regular breaks can help you reset. You can now turn on reminders to take breaks when it makes sense for you. jasohdesign Turn on Not now 9:41 < Similar Posts lif lapista golden hours kenzoore Great Shoth Want a break? Кош коштол saunders to like broeksist L Reminder to take breaks We'll remind you sa take a break when you spend this amount of time at once on instagram 30 minutes 20 minutes 10 minutes Off Done Explore upsell Edit break reminder 2 TAKE A BREAK MVP Case 4:22-md-03047-YGR Document 2758-6 Filed 02/20/26 Page 5 of 49 9:41 Break Experience Overview • If the reminders are turned on, after 10, 20, or 30 minutes of continuous use of the app, depending on the settings, we surface an immersive break reminder. The reminder includes tips for taking a break and a link to the break reminder settings. Time for a break? You've set reminders for every 10 minutes. Take a moment to reset by closing Instagram, Take a few deep breaths Write down what you're thinking Listen to your favorite song Do something on your to-do list Done Edit reminder 9:41 Instagram + photosbyean Sunday sunshine. #weekendvibes amethyst_gri View all 5 comments And a comment 1 day ago photosbyean You can adjust your break reminders anytime. Liked by kenzoere and omers Edit Break reminder After exiting break 3 Case 4:22-md-03047-YGR Document 2758-6 Filed 02/20/26 Page 6 of 49 Pre-Aligned Launch Criteria As a reminder, our team mission is to "Help teens become self-aware to take control over experiences that make them feel bad about themselves, and reduce these experiences on the platform overall." Specifically the product impact we're looking for here is behavioral signal for awareness and control. The launch criteria are: 1. See that we're helping people. Our recent research with users and experts suggests that interruptions could help with pulling users out of downward emotional spirals, especially when they are unable to recognize how the content they see is making them feel or unable to mitigate NAC experiences. 2. See meaningful usage over funnel. Our benchmarks here are Nudge / upsell CTR >= 0.5% and Nudge / upsell dismiss rate <= 20%. (Benchmarks are based on comparing with Update Following, Daisy controls, and voter registration.) 3. Balance ecosystem effects. We will disregard variants with material ecosystem regressions: Time spent: regressions < 0.2%, Sessions: regressions < 0.2%, Creator watch time: regressions < 0.5%. The best variant prioritizes trading off big reductions in ecosystem regressions for small reductions in adoption. 4. Complements external narrative. Judgement call that the product and business workstreams for proactive public comms complement each other explainably. Caveat: Outreach teams have already talked about this product with press and policy makers, so it would be hard to walk back what's shared. At this time, as agreed in the NAC measurement Communities review, we don't expect to use surveys or Negative Appearance Comparison-specific metrics in the launch criteria. 4 Case 4:22-md-03047-YGR Document 2758-6 Filed 02/20/26 Page 7 of 49 TL;DR Experiment was successful and we recommend launching with the +20 min variant (vs. +10 min variant): Upsell Adoption: 3.9% and 3.6% for the 20 vs 10 min variant; above 0.5% benchmark*. Upsell Dismissal: 7.3% and 6.8% for the 20 vs 10 min variant; below 20% benchmark. Break Usage: 14.6% and 15.8% for the 20 vs 10 min variants. Ecosystem: There are no stat-sig regressions in ecosystem metrics and we have sufficient power to rule out material regressions in all IG Critical Metrics except good user experiences (performance metric). We recommend proceeding with the launch and monitoring this metric. We also recommend monitoring IG Revenue, which, worst case, could regress by about half of the material regression threshold (-0.07% vs the -0.20% budget). Projected Reach: Based on the results, we predict that 1M teens will see the upsell and ~31k will adopt Take a Break in the first month if launched with the +20 min variant. Rationale & Considerations: We recommend moving forward with the +20 min variant because it has similar adoption/dismissal rates to the +10 min option without a big drop in reach (8.7% vs 11.28% of eligible teens in the 6 EN countries). As we evolve the product next half, through expansion of upsell to more surfaces and opting new teens in by default, going with this more conservative option allows us to evaluate and broaden our reach further via the trigger as needed. Notes: Upsell Adoption Rate: the percent of upsold teens that set and kept the break reminder Upsell Dismissal Rate: the percent of upsold teens that dismissed the upsell and never set the break reminder Break Usage Rate: percent of teens, after seeing the reminder, ended the session within 1 min of seeing the reminder reminder and didn't start a new session for another +10 mins We reviewed the following collections: FB Critical, IG Critical, IG Performance Metrics, Explore Shipping Calculus, and IG Sessions Commons 5 Case 4:22-md-03047-YGR Document 2758-6 Filed 02/20/26 Page 8 of 49 Upsell Adoption Rate Variants Exceed 0.5% Upsell Adoption Benchmark; +20 Min Variant Has Higher Adoption Rate (Stat-Sig) Adoption Rate % of Upsold Teens That Set and Kept the Break Reminder (Teen EN Country Take a Break Upsell Experiment, 2021-11-02 to 2021-11-29) 4.0% 3.5% 3.0% 2.5% 2.0% 1.5% 1.0% 0.5% 0.0% +10 Minutes +20 Minutes Test Variants Variant Adoption Rate Lower Cl Adoption Rate Adoption Rate Upper CI +10 Minutes 3.480 +20 Minutes 3.773 3.622 3.771 3.934 4.101 6 Case 4:22-md-03047-YGR Document 2758-6 Filed 02/20/26 Page 9 of 49 Upsell Dismissal Rate Variants Remain Below 20% Upsell Dismissal Benchmark; +20 Min Variant Has Higher Dismissal Rate (Stat-Sig) % of Upsold Teens That Dismissed Upsell and Never Set the Break Reminder (Teen EN Country Take a Break Upsell Experiment, 2021-11-02 to 2021-11-29) Dismissal Rate 7.0% 6.0% 5.0% 4.0% 3.0% 2.0% 1.0% I 0.0% +10 Minutes +20 Minutes Test Variants Variant Dismissal Rate Lower Cl Dismissal Rate Dismissal Rate Upper Cl +10 Minutes 6.63 +20 Minutes 7.05 6.83 7.03 7.27 7.49 7 Case 4:22-md-03047-YGR Document 2758-6 Filed 02/20/26 Page 10 of 49 Break Reminder Usage Rate Meaningful Break Usage Across Both Variants and All Break Reminder Settings. Usage Rate Break Usage Rate by Reminder Value (Teen EN Country Take a Break Upsell Experiment, 2021-11-02 to 2021-11-29) 20.0% 15.0% 10.0% 5.0% 0.0% +10 Minutes +20 Minutes Test Variants 10 Minutes 20 Minutes 30 Minutes Variant Break Reminder Value Break Usage Rate Lower CI Break Usage Rate Break Usage Rate Upper Cl +10 Minutes 10 Minutes 14.00 14.79 15.60 +10 Minutes 20 Minutes 17.35 18.94 20.64 +10 Minutes 30 Minutes 15.21 17.26 19.54 +20 Minutes 10 Minutes 13.27 14.00 14.77 +20 Minutes 20 Minutes 15.93 17.61 19.44 +20 Minutes 30 Minutes 12.64 14.40 16.37 8 Case 4:22-md-03047-YGR Document 2758-6 Filed 02/20/26 Page 11 of 49 # of Teens in 6 Step in Month 1 % of prior EN countries # of Teens in 6 EN countries % of prior 1 10 min 20 min First Month Projections 2 IG u18 MAU in 6 EN countries 3 Eligible for TaB upsell* 35M 9M 35M 26% 9M 26% 4 See TaB upsell** 1M 11.3% 790k 8.7% 5 Activate control 36.8k 3.6% 30.8k 3.9% Based on test data 6 See break experience >=1 time 18.8k 51% 15.9k 52% 7 Use break experience to leave app >=1 time 10.2k 54% 8.9k 56% 8 Use break experience to leave 3.3k 32% app >=3 times 2.7k 31% * Eligibility considerations: only stated age, adding English locale, iOS, and adding the app version constraint ** Seeing upsell depends on hitting trigger criteria while scrolling down Explore "similar posts" view 9 Case 4:22-md-03047-YGR Document 2758-6 Filed 02/20/26 Page 12 of 49 +20 Minute Variant IG Criticals Ecosystem Impact +20 Minute Variant Ecosystem Impact Materiality Threshold Description Good User Experiences Fully Qualified Name Pillar (Budget) Topline Impact Lower Bound Topline Impact Upper Bound Topline Impact Stat-Sig? Meets Budget? instagram:perf:good_user_experiences Product Foundation -0.20% -0.22% 0.06% No No Overall Objectionable Prevalence community_integrity:cep:viewer:ig_holistic_prevalence_world_combined Wellbeing 1.00% -0.77% 0.90% No Yes Foreground Sessions instagram:sessions:foreground_session_cnt:overall Feed & Relevance -0.20% -0.11% 0.05% No Yes Reels Intentional Plays instagram:clips_media:video_playback:intentional_play_count:overall Reels -0.75% -0.28% 0.12% No Yes Stories Impressions instagram:media_vpvd_impressions:stories:home Stories -0.30% -0.07% 0.21% No Yes Feed Impressions instagram:post_vpvd_impressions:feed:home Feed & Relevance -0.50% -0.14% 0.03% No Yes Direct Sends instagram:direct:sends:overall N/A -1.00% -0.18% 0.25% No Yes IG Time Spent instagram:time_spent:overall N/A -0.20% -0.12% 0.10% No Yes Infra - Power instagram:efficiency:normalized_power:overall Infra 0.40% -0.10% 0.03% No Yes Infra - TAO instagram:efficiency:tao_fbobj_get:overall Infra 0.40% -0.11% 0.03% No Yes IG DAP instagram:growth:instagram_classic:dap:overall Growth -0.20% -0.04% 0.01% No Yes Original Stories Creation instagram:creation: producer_17:original_media:stories Stories -0.50% -0.11% 0.11% No Yes Reels Producer DAP instagram:creation:clips_media_producer_dap:overall Reels -2.00% -1.51% 1.06% No Yes Creator Watch Time instagram:video_consumption:viewer:creator_watch_time_60s_capped:excluding_stories Creators -0.50% -0.18% 0.09% No Yes IG Revenue instagram:abp:ad_metrics:revenue_usd_cents:overall IG4B -0.20% -0.07% 0.20% No Yes IG DAU instagram:growth:instagram_classic:dau:overall Growth n/a -0.04% 0.01% No Yes Source query: https://fburl.com/deltoid3/u679t6ji Color Key: Reach: Doesn't violate topline threshold, stat-sig 8.70% Violates topline threshold, stat-sig No stat-sig difference between topline threshold and topline impact Source: https://docs.google.com/spreadsheets/d/11WEGaWnyozNTcU_ttklsPYTpnyPIK35ae1yeZwb ARs/edit#gid=1621687515 10 Case 4:22-md-03047-YGR Document 2758-6 Filed 02/20/26 Page 13 of 49 +10 Minute Variant IG Criticals Ecosystem Impact +10 Minute Variant Ecosystem Impact Materiality Threshold Topline Impact Description Good User Experiences Fully Qualified Name Pillar (Budget) Lower Bound Topline Impact Upper Bound Topline Impact Stat-Sig? Meets Budget? instagram:perf:good_user_experiences Product Foundation -0.20% -0.29% 0.07% No No Overall Objectionable Prevalence community_integrity:cep:viewer:ig_holistic_prevalence_world_combined Wellbeing 1.00% -1.00% 1.17% No No Foreground Sessions Reels Intentional Plays Stories Impressions Feed Impressions Direct Sends instagram:sessions:foreground_session_cnt:overall Feed & Relevance -0.20% -0.14% 0.06% No Yes instagram:clips_media:video_playback:intentional_play_count:overall Reels -0.75% -0.37% 0.16% No Yes instagram:media_vpvd_impressions:stories:home Stories -0.30% -0.09% 0.27% No Yes instagram:post_vpvd_impressions:feed:home Feed & Relevance -0.50% -0.18% 0.04% No Yes instagram:direct:sends:overall N/A -1.00% -0.23% 0.32% No Yes IG Time Spent Infra - Power Infra - TAO IG DAP Original Stories Creation Reels Producer DAP Creator Watch Time IG Revenue IG DAU instagram:growth:instagram_classic:dau:overall Source query: https://fburl.com/deltoid3/pxd9mtop instagram:time_spent:overall N/A -0.20% -0.15% 0.14% No Yes instagram:efficiency:normalized_power:overall Infra 0.40% -0.13% 0.04% No Yes instagram:efficiency:tao_fbobj_get:overall Infra 0.40% -0.14% 0.04% No Yes instagram:growth:instagram_classic:dap:overall Growth -0.20% -0.01% 0.01% No Yes instagram:creation:producer_17:original_media:stories Stories -0.50% -0.15% 0.15% No Yes instagram:creation:clips_media_producer_dap:overall Reels -2.00% -1.96% 1.37% No Yes instagram:video_consumption:viewer:creator_watch_time_60s_capped excluding_stories Creators -0.50% -0.24% 0.11% No Yes instagram:abp:ad_metrics:revenue_usd_cents:overall IG4B -0.20% -0.09% 0.26% No Yes Growth n/a -0.05% 0.02% No Yes Color Key: Reach: Doesn't violate topline threshold, stat-sig 11.28% Violates topline threshold, stat-sig No stat-sig difference between topline threshold and topline impact Source: https://docs.google.com/spreadsheets/d/11WEGWnyozNTcU_ttklsPYTpnyPIK35ae1yeZwb_ARs/edit?usp=sharing 11 Case 4:22-md-03047-YGR Document 2758-6 Filed 02/20/26 Page 14 of 49 Upsell Dismissal Rate TL;DR • The Upsell dismissal rate, defined as the percent of upsold teens that dismissed the upsell and never set the break reminder, in both the 10 min and 20 min variants remain below the 20% success benchmark overall: at about 6%. Above is true for teen boys and girls but not teens that use custom gender identities. For those teens, only 15 year olds in the 20 min variant, 16 year olds in both variants, and 17 year olds in the 10 min variant have dismissal rates below the 20% benchmark. However the sample size for these teens is small (~20 for 13 year olds and ~150-200 for +14). For both variants, dismissal rates are lower for teens with higher historical NAC prevalence but higher for teens that tend to have longer sessions. However differences between variants are mostly non-stat-sig. 26 Case 4:22-md-03047-YGR Document 2758-6 Filed 02/20/26 Page 15 of 49 Dismissal Rates Consistently Below 20% Benchmark Across EN Speaking Countries 12.0% % of Upsold Teens That Dismissed Upsell and Never Set the Break Reminder by User Country (Teen EN Country Take a Break Upsell Experiment, 2021-11-02 to 2021-11-19) +10 Minutes +20 Minutes Dismissal Rate 10.0% 8.0% 6.0% 4.0% 2.0% 0.0% AU CA GB IE NZ US User Country 34 Case 4:22-md-03047-YGR Document 2758-6 Filed 02/20/26 Page 16 of 49 Ignore Rates TL;DR Most teens ignore the upsell and never set the break reminder: 88-89% ignore it and don't set the reminder. No stat-sig differences between variants. After moderate jump post first day of upsell impressions, ignore rates decline steadily from 96-97% to 93% by the 15th day since first upsell impression. There are no stat-sig or practical differences in ignore rates by teen historical NAC prevalence, however teens that tend to have longer sessions ignore the upsell at lower rates. 36 Case 4:22-md-03047-YGR Document 2758-6 Filed 02/20/26 Page 17 of 49 Most Teens Ignore TaB Upsell; No Stat-sig Differences Between Variants % of Upsold Teens That Ignored Upsell and Never Set the Break Reminder (Teen EN Country Take a Break Upsell Experiment, 2021-11-02 to 2021-11-19) Ignore Rate 80.0% 60.0% 40.0% 20.0% 0.0% +10 Minutes +20 Minutes Test Variants Variant Ignore Rate Lower Cl Ignore Rate Ignore Rate Upper Cl +10 Minutes 88.9 89.18 89.45 +20 Minutes 88.2 88.51 88.81 37 Case 4:22-md-03047-YGR Document 2758-6 Filed 02/20/26 Page 18 of 49 Case 4:22-md-03047-YGR Document 2758-6 Filed 02/20/26 Page 19 of 49 Case 4:22-md-03047-YGR Document 2758-6 Filed 02/20/26 Page 20 of 49 Case 4:22-md-03047-YGR Document 2758-6 Filed 02/20/26 Page 21 of 49 Case 4:22-md-03047-YGR Document 2758-6 Filed 02/20/26 Page 22 of 49 No FB Critical Regressions in Delta Time Series View Case 4:22-md-03047-YGR Document 2758-6 Filed 02/20/26 Page 23 of 49 Case 4:22-md-03047-YGR Document 2758-6 Filed 02/20/26 Page 24 of 49 Case 4:22-md-03047-YGR Document 2758-6 Filed 02/20/26 Page 25 of 49 Case 4:22-md-03047-YGR Document 2758-6 Filed 02/20/26 Page 26 of 49 Case 4:22-md-03047-YGR Document 2758-6 Filed 02/20/26 Page 27 of 49 Case 4:22-md-03047-YGR Document 2758-6 Filed 02/20/26 Page 28 of 49 Case 4:22-md-03047-YGR Document 2758-6 Filed 02/20/26 Page 29 of 49 Case 4:22-md-03047-YGR Document 2758-6 Filed 02/20/26 Page 30 of 49 Case 4:22-md-03047-YGR Document 2758-6 Filed 02/20/26 Page 31 of 49 0 Case 4:22-md-03047-YGR Document 2758-6 Filed 02/20/26 Page 32 of 49 Case 4:22-md-03047-YGR Document 2758-6 Filed 02/20/26 Page 33 of 49 Case 4:22-md-03047-YGR Document 2758-6 Filed 02/20/26 Page 34 of 49 Case 4:22-md-03047-YGR Document 2758-6 Filed 02/20/26 Page 35 of 49 Case 4:22-md-03047-YGR Document 2758-6 Filed 02/20/26 Page 36 of 49 Case 4:22-md-03047-YGR Document 2758-6 Filed 02/20/26 Page 37 of 49 Case 4:22-md-03047-YGR Document 2758-6 Filed 02/20/26 Page 38 of 49 Case 4:22-md-03047-YGR Document 2758-6 Filed 02/20/26 Page 39 of 49 Case 4:22-md-03047-YGR Document 2758-6 Filed 02/20/26 Page 40 of 49 Case 4:22-md-03047-YGR Document 2758-6 Filed 02/20/26 Page 41 of 49 Case 4:22-md-03047-YGR Document 2758-6 Filed 02/20/26 Page 42 of 49 Case 4:22-md-03047-YGR Document 2758-6 Filed 02/20/26 Page 43 of 49 Case 4:22-md-03047-YGR Document 2758-6 Filed 02/20/26 Page 44 of 49 Case 4:22-md-03047-YGR Document 2758-6 Filed 02/20/26 Page 45 of 49 Case 4:22-md-03047-YGR Document 2758-6 Filed 02/20/26 Page 46 of 49 Case 4:22-md-03047-YGR Document 2758-6 Filed 02/20/26 Page 47 of 49 Case 4:22-md-03047-YGR Document 2758-6 Filed 02/20/26 Page 48 of 49 Case 4:22-md-03047-YGR Document 2758-6 Filed 02/20/26 Page 49 of 49