Meta & Children's Online SafetyIG Feed Relevance Working Group
Raw text
Case 4:22-md-03047-YGR IG Feed Relevance Working Group ton June 30, 2023 2:10 PM Document 3007-8 Filed 05/01/26 Page 1 of 2 #2023 H1 Teen-specific Core Ranking Improvements - Learning & Suggestions for Next Half ## TL;DR In 2023 H1 we scoped and worked on teen specific core ranking improvements ([scoping] (https://fb.workplace.com/groups/233247545227463/permalink/652569153295298/), eng updates [1](https://fb.workplace.com/groups/233247545227463/permalink/664031478815732/), [2] (https://fb.workplace.com/groups/233247545227463/permalink/667609008457979/), [3](https://fb.workplace.com/groups/233247545227463/permalink/676265877592292/), [4] (https://fb.workplace.com/groups/233247545227463/permalink/682184400333773/), [5](https://fb.workplace.com/groups/233247545227463/permalink/689313956287484/), [6] %VS (https://fb.workplace.com/groups/233247545227463/permalink/697499772135569/)). We want to share a summary of learnings and suggestions for next half in this post and share the more relevant findings here. * Given upsampled teen and overall data, we verified that (1) more teen data leads to consistent teen NE wins and wins for overall NE in most cases; (2) teen data is more efficient in improving teen NE compared with non-teen data; (3) non-teen data is indispensable for model performance for teens especially given its large volume - so it is always suggested to use all non-teen data as it improves teen NE as well. * Suggested next steps (1) we saw teen core and capped session wins for the secondary model tuning for teens with model trained on upsampled data and we are discussing launch paths to materialize the wins ([post] (https://fb.workplace.com/groups/233247545227463/permalink/723962446155968/)); (2) upsampling teens is benefit both teen NE and overall NE so given teen is more top priority and teen data is more efficient for teen NE we should prioritize upsampling teens; (3) [needs verification] use the upsampled data in top rank should lead to large NE wins given it is only using top post (so much smaller data volume); can do offline analysis to check this and if true suggest upsample top posts to 100%. * p_skip, p_external_share, and p_15s have worse NE for teens. Suggest (1) try different label definitions for teens for p_15s and p_skip as teen's CTR are different ([dashboard](https://fburl.com/unidash/vtrtslq0)) to see if these labels provide better value for predicting teen's engagement; (2) try techniques that handles the low CTR issue or simply use upsampled training data as p_external_share for teens is very low ( % for overall users). * We found that age breakdown and media_user_teen_non_teen key features in the production feature tag are not more important for teens. There could be an opportunity in digging in how the age and is_teen is defined in fireball e.g. modeled or stated age so we can improve these features. * We found top features are quite different between teens and overall users (e.g. about 20% features in top 300 for teens are not in top 300 for overall users). Suggest do a more thorough analysis on top features between teen and overall users based on tasks that are session drivers to decide on which types of features to author for teens. * Dedicated teen tasks shows mild wins - suggest trying small weights and adding tasks that showed promising teen NE wins only. ## Not promising directions * Direct up-weight teen losses * Feature selection based on teen data: reason being that when feature tag contains a good amount of features (say), the tag that is selected by teens does not differ too much from that by overall data. * Check top features that are selected by teens that are left out in MB. We don't expect much from this work but worth doing for every MB. ## Projects ## Project: Feed ranking mtml model performance for teens **Description**: We compare the model performance between teens versus overall users (as part of the teen early leaming series [feed analyzer](https://fb.workplace.com/notes/161423796602547), [engagement analysis] (https://fb.workplace.com/notes/164565316288395), [model performance](https://fb.workplace.com/notes/633002922162847/)) **Method**: In the first version of [model performance] (https://fb.workplace.com/notes/633002922162847/) we use NE to compare model performance. Later we learned that comparing NE between segments (e.g. teens vs. overall) where the CTR is different between the segments could be misleading as normalization is different between them (see Israel's comment in [post](https://fb.workplace.com/notes/633002922162847/)). In this updated version we add ROC and PRAUC metrics as the benchmark and include the results for top rank model predictors. **Results*** Model performance is worse for teens based on ALL three metrics (NE, ROC, PRAUC) for * skip * external share* comment reply * time spent next two posts | current not skipped * p15s * session seen five **Suggestions for next half**: * Prioritize improving those predictors that performed worse for teens and are more important based on [feed analyzer] (https://www.internalfb.com/intem/unidash/dashboard/feed_ranking_ig_feed_analyzer/predictors/?events=%7B%22907812383451683%22%3A%7B%22select_Start_Date%22%3A%7B%22data%22%3A%222023-06- 07%22%2C%22publisher_id%22%3A%22907812383451683%22%7D%2C%22907812383451683%22%3A%7B%22data%22%3A%222023-06- 07%22%2C%22publisher_id%22%3A%22907812383451683%22%7D%7D%2C%224941279285906967%22%3A%7B%22select_Date_Pie%22%3A%7B%22data%22%3A%222023-06- 25%22%2C%22publisher_id%22%3A%224941279285906967%22%7D%2C%224941279285906967%22%3A%7B%22data%22%3A%222023-06- 25%22%2C%22publisher_id%22%3A%224941279285906967%22%7D%7D%2C%22855646071779704%22%3A%7B%22select_Date_Hist%22%3A%7B%22data%22%3A%222023-06- 24%22%2C%22publisher_id%22%3A%22855646071779704%22%7D%2C%22855646071779704%22%3A%7B%22data%22%3A%222023-06- 24%22%2C%22publisher_id%22%3A%22855646071779704%22%7D%7D%2C%224793256447391864%22%3A%7B%22select_End_Date%22%3A%7B%22data%22%3A%222023-06- 24%22%2C%22publisher_id%22%3A%224793256447391864%22%7D%2C%224793256447391864%22%3A%7B%22data%22%3A%222023-06- 24%22%2C%22publisher_id%22%3A%224793256447391864%22%7D%7D%2C%22415228130082755%22%3A%7B%22select_Predictor%22%3A%7B%22data%22%3A%22bvi_subvmf%22%2C%22publisher_id%22%3A% than the skip (#1; 19.92%), external share (#2; 15.43%) and p15s (#5; 6.33%). (the predictor importance is based on overall users, we should make a dashboard for teens next half) * As teen's CTR for p_skip and p_15s are lower overall users ([dashboard](https://fburl.com/unidash/vtrtslq0)), we can try different label definitions for teens (e.g. p_10s) and see if it creates more value for predicting teen engagements. p_skip (top) & p_15s (bottom) CTR * As teen's CTR for p_external share is very low (half that of the overall user % VS %), we can try techniques that handles the low CTR issue or simply use upsampled training data as our past experience shows more data can largely improves model performance for low CTR predictors. p_external_share CTR ## Project: Upsample teen training data **Description**: Previously ([post1](https://fb.workplace. 1](https://fb.workplace.com/notes/667146475415158), [post2] (https://fb.workplace.com/groups/233247545227463/permalink/676265877592292/), [post3](https://fb.workplace.com/notes/686807013449104)) we used various approaches such as downsampling prod, longer look-back window, upsampling teens etc, to show that **(1)****more teen data leads to teen NE wins and mostly mild wins or neutral results for overall NE and (2) teen data is more efficient in improving teen NE compared with non-teen data. **We show some simple diagrams to summarize our findings here. **Method**: See below. **Results**:**(1) more teen data leads to consistent teen NE wins and mostly wins for overall NE** **Method**: we see consistent and decent NE wins across all tasks. * For overall users: * [left diagram] Adding teen data improves overall NE except for a couple of comer cases ). * [right diagram] We also tested the case of adding more overall data (i ) to check if we see NE wins for ALL tasks. It turns out there are still corner cases where we see overall NE regression (e.g. ). **(2) Teen data is more efficient in improving teen NE compared with non-teen data**. On the other hand, non-teen data is still instrumental for teen NE given its sheer volume. **Given our current sampling rate, we are far away from the saturation point where we have sufficient teen data to train an model without seeing benefits from using non-teen data**. Again we use two feature tags for more robustness. For teen users: * We see consistent teen NE wins for the models trained with more teen data (38%) across all tasks. This is a data point that evidences that teen data is more "efficient" in improving teen NE. * For overall users: * Results of overall NE are mixed comparing higher %) and lower%) teen data percentage i.e. it is not always that training with production level %) teen percentage results in better NE compared with higher %) teen percentage. The NE difference is not too drastic %) though. * **(3) Non-teen data is indispensable for model performance for teens given its large volume and we are far away from the saturation point where we have sufficient teen data to train an model alone without seeing benefits from using non- teen data** **Suggestions for next half*** We saw teen core and capped session wins for the secondary model tuning for teens with model trained on upsampled data and we are discussing launch paths to materialize the wins ([post](https://fb.workplace.com/groups/233247545227463/permalink/723962446155968/)) * Our guess is that when the proportion of teen data is not too high (we have seer % seems okay), then upsampling teens should benefit both teen NE and overall NE - so given teen is more top priority we should prioritize upsampling teens. * Feed ranking team plans for the next half to upsample overall data from % to % and upsample teen data from % to % ([post1](https://fb.workplace.com/groups/1196779007703460/permalink/1200448430669851/), [post2] (https://fb.workplace.com/groups/233247545227463/permalink/704005428151670/)). (1) Given that teen data is more efficient for model performance for teens, we can prioritize upsampling the teen part and checking along the way if it breaks the balance of teen proportion in the training data (the previous bullet point). (2) And given the usefulness of non-teen data for teen model performance, we should almost always use "ALL" data that is available even if we want to optimize teens only (which is also not the case). * Use the upsampled data in top rank should lead to large NE wins given it is only using top post (so much smaller data volume); can do offline analysis to check this and if true suggest upsample top posts to 100%.. ## **Project: Up-weight teen training losses** We have tried two approaches to up-weight teen training losses (1) direct up-weight and (2) add teen dedicated tasks. The advantage of the second approach is that it will preserve calibration (idea from [FB YA project](https://fb.workplace.com/notes/612300036689297)). ##**(1) Direct up-weight teen training losses**** Description**: Apply higher weight on teen training losses directly. ** However, this observation aligns with the results we obtained from "dedicated teen tasks" (below section and also results from [reels tab] (https://docs.google.com/document/d/1wi519nUxsV-NDKex0tetXVT4uwP8bhw9G_PFzSQgGA/edit)). We guess this may be due to that up-weight teen training losses to some extent offsets the learnings from "non-teen" data, which has been proved to be useful for model performance for teens. * For overall users (diagram left: primary organic; right: secondary organic): * **Suggestions for next half**: That said, this work does % not bok like a promising direction given the feed and stories results and we'd suggest minimal priority. ## **(2) Adding teen dedicated teen tasks****Description**: Add additional tasks for selected production tasks such that they only predict for teen data. This way, losses will be applied more for Teen data, resulting in more emphasis on teen model performance. This is essentially up-weight teen train loss without affecting calibration. These additional teen tasks are only applied at training time, and are turned off at inference time. **Method**: Add tasks that only train on teen data for both primary and secondary organic models. We trained two versions of model - one with 7 days and the other with 12 days of training data. **Results*****Initial test on 4 predictors in primary organic model showed very neutral NE results when teen dedicated tasks are added** (with loss weight = 1) e.g. teen NE improves a negligible and 0.01% for like and profile tap, while regresses % and % for comment and reshare. We did a more thorough tests training models with 7 days and 12 days of data with different loss weights (w=1, 3, 5) for the dedicated teen tasks. The case of "W_large" in below diagram represents a model trained only on teen data. * For teen users (diagram left: primary organic; right: secondary organic): * First, we think the 7D results for primary organic model is too good to be true (e.g. like % and comment %!) and it could be an outlier data point. We believe the 12D results for primary organic model could be more representative of the teen NE wins we expect (e.g. % for reshare, comment is still too good -0.93% though). * With the increase of loss weight of teen dedicated tasks, teen NE regressed. This is inline with the results from Reels tab ([doc](https://docs.google.com/document/d/1wi519nUxsV- NDKex0tetXVT4uwPi8bhw9G_PFzSQgGA/edit), [post](https://fb.workplace.com/groups/539507628209026/permalink/572840494875739/)). This is also inline with what we saw for direct up-weight teens (previous section). * For overall users (diagram left: primary organic; right: secondary organic): * The NE results for overall users are quite intuitive - NE worsens with increasing teen loss weights and in general NE is worse compared to not having the teen dedicated tasks added. * With smaller teen loss weight (e.g. w=1) we get roughly neutral overall NE. **Suggestions for next half*** This work is more promising compared to direct up-weight as (1) we see mild teen NE wins and neutral overall NE with smaller loss weight for teen dedicated tasks; (2) this will not affect model calibration as these dedicated tasks are turned off at inference time - so minimal maintenance cost. (3) this has also been launched for FB YA ([post] (https://fb.workplace.com/notes/612300036689297)). ***We can (1) explore smaller oss weights (ie. 0.5, 0.7 etc), and (2) try only add dedicated tasks for tasks showing promising NE wins (e.g. reshare, like, external share). Once we find a combination of tasks and loss weights that lead to teen NE wins with minimal overall NE regressions we can propose to next MB.** ## Project: Teen specific feature engineering ##**(1) Feature importance based on teen data****Description**: We want to understand if there is a difference of important features between teens and overall users. In addition, if there is difference, what kinds of features are more important to teens. We also want to check if age and key-user_is_teen sort of features are more important to teens. **Method**: Note that the feature importance analysis done here is based on the production feature tag which is selected by overall data. We check feature selection results based on teens in next section **Suggestions for next half**: * We found that age breakdown and media_user_teen_non_teen key features in the production feature tag are not more important for teens. There could be an opportunity in digging in how the age and is_teen is defined in fireball e.g. modeled or stated age so we can improve these features. * We found top features are quite different between teens and overall users (e.g. about 20% features in top 300 for teens are not in top 300 for overall users). Suggest do a more thorough analysis on top features between teen and overall users based on tasks that are session drivers to decide on which types of features to author for teens. ##**(2) Feature selection based on teen data ****Description**: We select dense features based on the teen data instead of the overall data to check if the models trained with this teen selected feature tag improves NE for teens. **Method**: HIGHLY CONFIDENTIAL (COMPETITOR) ΜΕΤΑ3047MDL-047-01202953 Case 4:22-md-03047-YGR Document 3007-8 Filed 05/01/26 Page 2 of 2 (details in [post](https://fb.workplace.com/notes/686807013449104), FS notebook: [N3352710](https://www.internalfb.com/intern/anp/view/?id=3352710)) **Results**: * We compare the NE% difference between models trained with features selected by teen data versus overall data for two scenarios (1) model trained on teen data and (2) model trained on overall data. Results show that **for both scenarios, features selected based on teen data does not significantly improve teen NE**. * We checked the feature tags selected by teen data and overall data and found that there are about features that are in the top for teens but not in top for the overall users. However, among these features there is only in top and in top based on the feature inmportance run based on the teen tag** This indicates that when feature tag contains a good amount of features (say ), the tag that is selected by teens does not differ too much from that by overall data.**** We didn't include in the diagram above for "external share" (due to scale), the NE regressed quite a lot (around %) for model trained with features selected by teen data. Our guess is that this is due to the low CTR for this label, and using teen data alone could not find good features for such sparse actions. **Suggestions for next half**: * We showed that the model performance based on the tags selected by teen data and by overall data are very neutral and we showed that it is due to that the features selected based on teens data is not very different ie, among the different features only very few of them are important, from those based on overall data and. We should check if this is true for other miml models e.g. the toprank model. * While the ecosystem changes over time and newly are registered we should check at a regular basis if there are features left out by MB, which selects features based on overall data, but are in fact important for teens. We did this this half based on per task feature importance for teens for important tasks likes, profile taps, and reshares and currently these proposed features are included in 2023 H1 MB. #thanks Special thanks to who spent time discussing results from these projects and giving me so many invaluable feedbacks #thanks for their work on data upsampling that facilitates the analysis #thanks for discussions of teen related features for constant check-ins and giving feedbacks on this work #thanks work #thanks Israel Nir for his advice on Swiss Cheese which makes this work much more easier W WWW Thanks Boton June 30, 2023 at 2:16 PM Thanks for thanking, about Thanks formatting, visit https://fburl.com/thanks-parsing. Ion June 30, 2023 at 2:17 PM #sharebot Relevance & Teens Insights Move Bot on June 30, 2023 at 2:17 PM Alrighty CC Di Liu and can see your thanks at fourl.com/thanks. Awesome! For more info I shared this post to Relevance & Teens Insights. Here is the shared post: https://fb.workplace.com/groups/526099235989340/permalink/607251707874092/ on June 30, 2023 at 2:18 PM on June 30, 2023 at 2:40 PM This is fantastic summary of H1, congrats ! Two things I've discussed with a couple of weeks ago: 1. We might benefit for not upsampling all "non-teens" uniformly. Since you've found that teen data is more efficient than non-teen data for teens, it makes sense that among non-teens we might want to over samples those that are "teen like". In the most basic form it means - sample more 20 year olds than 60 year olds. 2. Personally (and I'll need you guys' guidance here) - I'd be interested in taking the existing model and trying to fine tune it on teen data, just to see what happens. on July 10, 2023 at 2:57 PM Sorry for my late reply! These are great feedbacks! I thought about (1) but hasn't started any test, I think these can be done quickly on my side. For (2) let's take it offline. too I think we can take a try together. brought up this idea some time ago HIGHLY CONFIDENTIAL (COMPETITOR) ΜΕΤΑ3047MDL-047-01202954