<div class="sub-cat-page-holder"><div class="sub-cat-page-img"><img src="https://persona.ly/glossary/wp-content/uploads/2020/02/graph-bar.svg"/></div><div class="sub-cat-page-h-holder"><h2 class="sub-cat-page-title">Performance metrics</h2><h4 class="sub-cat-page-subtitle"><span class="sub-cat-page-subtitle">Creative Testing  &#8211; Multi-Armed Bandit vs. A/B Testing</span><span class="csr-page-counter"> -> Page <span class="max-for-title">1 of <span class="sub-cat-max-page"></span></span></span></h4></div></div><div class="search-sidebar-holder"><a href="/glossary/"><h3 class="search-sidebar-holder-title">UA Glossary</h3></a>[vc_widget_sidebar sidebar_id="single-page-sidebar" el_class="front-page-sidebar-holder"]<div class="search-sidebar-footer-holder"></div></div>{"id":1274,"date":"2021-03-29T11:47:17","date_gmt":"2021-03-29T11:47:17","guid":{"rendered":"https:\/\/persona.ly\/glossary\/?page_id=1274"},"modified":"2021-03-29T14:42:59","modified_gmt":"2021-03-29T14:42:59","slug":"creative-testing-multi-armed-bandit-vs-a-b-testing","status":"publish","type":"page","link":"https:\/\/persona.ly\/glossary\/performance-metrics\/creative-testing-multi-armed-bandit-vs-a-b-testing\/","title":{"rendered":"Creative Testing  &#8211; Multi-Armed Bandit vs. A\/B Testing"},"content":{"rendered":"<p>[vc_row el_class=&#8221;front-hero-section&#8221;][vc_column width=&#8221;1\/2&#8243; css=&#8221;.vc_custom_1582730807176{background-image: url(https:\/\/persona.ly\/glossary\/\/wp-content\/uploads\/2020\/02\/group-19.svg?id=58) !important;background-position: center !important;background-repeat: no-repeat !important;background-size: contain !important;}&#8221; el_class=&#8221;hero-first-section&#8221;][vc_column_text el_class=&#8221;page-heading&#8221; css=&#8221;.vc_custom_1593096796228{margin-top: 30px !important;}&#8221;]<\/p>\n<h1><a href=\"\/glossary\/\">UA <span class=\"color-ancient\">Glossary<\/span><\/a><\/h1>\n<p>[\/vc_column_text][vc_column_text]<p>Your source for everything mobile UA, from the basics to contentious standards, the glossary can help and inform both aspiring growth managers and experienced mobile app developers<\/p>\n[\/vc_column_text]<div class=\"ult-spacer spacer-69e939d376af7\" data-id=\"69e939d376af7\" data-height=\"105\" data-height-mobile=\"0\" data-height-tab=\"\" data-height-tab-portrait=\"\" data-height-mobile-landscape=\"0\" style=\"clear:both;display:block;\"><\/div>[\/vc_column][vc_column width=&#8221;1\/2&#8243;][vc_single_image image=&#8221;60&#8243; img_size=&#8221;full&#8221; alignment=&#8221;center&#8221; el_class=&#8221;hero-section-image&#8221;][\/vc_column][\/vc_row][vc_section css=&#8221;.vc_custom_1583429011820{background-image: url(https:\/\/persona.ly\/glossary\/\/wp-content\/uploads\/2020\/03\/bg_repeater_with_bg_svg.svg?id=320) !important;}&#8221; el_class=&#8221;post-content-section archive-page-custom single-subcat&#8221;][vc_row el_class=&#8221;covered-grace-font&#8221;][vc_column width=&#8221;2\/3&#8243; el_class=&#8221;gap-in-end-single&#8221;][vc_column_text][\/vc_column_text][vc_column_text]<style>.wp-show-posts-columns#wpsp-1276 {margin-left: -2em; }.wp-show-posts-columns#wpsp-1276 .wp-show-posts-inner {margin: 0 0 2em 2em; }<\/style><section id=\"wpsp-1276\" class=\" wp-show-posts\" style=\"\"><article class=\" wp-show-posts-single wpsp-clearfix post-1278 post type-post status-publish format-standard hentry category-creative-testing-multi-armed-bandit-vs-a-b-testing\" itemtype=\"http:\/\/schema.org\/CreativeWork\" itemscope><div class=\"wp-show-posts-inner\" style=\"\">\t\t\t\t\t\t<header class=\"wp-show-posts-entry-header\">\n\t\t\t\t\t\t\t<h1 class=\"wp-show-posts-entry-title\" itemprop=\"headline\"><a href=\"https:\/\/persona.ly\/glossary\/creative-testing-multi-armed-bandit-vs-a-b-testing\/\" rel=\"bookmark\">Creative Testing  &#8211; Multi-Armed Bandit vs. A\/B Testing<\/a><\/h1><div class=\"wp-show-posts-entry-meta wp-show-posts-entry-meta-below-title post-meta-inline\"><span class=\"wp-show-posts-posted-on wp-show-posts-meta\">\n\t\t\t\t\t<a href=\"https:\/\/persona.ly\/glossary\/creative-testing-multi-armed-bandit-vs-a-b-testing\/\" title=\"11:50 am\" rel=\"bookmark\"><time class=\"wp-show-posts-entry-date published\" datetime=\"2022-03-29T11:50:06+00:00\" itemprop=\"datePublished\">March 29, 2022<\/time><time class=\"wp-show-posts-updated\" datetime=\"2022-12-11T12:40:19+00:00\" itemprop=\"dateModified\">December 11, 2022<\/time><\/a>\n\t\t\t\t<\/span><\/div>\t\t\t\t\t\t<\/header><!-- .entry-header -->\n\t\t\t\t\t\t\t\t\t\t\t<div class=\"wp-show-posts-entry-content\" itemprop=\"text\">\n\t\t\t\t\t\t\t<p>[vc_row][vc_column el_class=&#8221;normal-text&#8221;][vc_column_text]<span style=\"font-weight: 400;\">Creative testing is the way marketers compare the performance of different creatives in a campaign in order to evaluate which creatives yield better results. We will cover two methods: <\/span><b>A\/B testing<\/b><span style=\"font-weight: 400;\">, which is a commonly used method and widely known method, and the <\/span><b>multi-armed bandit<\/b><span style=\"font-weight: 400;\">, which is less commonly known and used, usually due to its relative complexity.\u00a0<\/span>[\/vc_column_text][vc_column_text]<span style=\"font-weight: 400;\">These two methods offer very different approaches to creative testing, each with its own advantages and disadvantages. The aim here is to help marketers understand these differences and bring attention to some of the limitations of A\/B testing and help them make a more informed decision.<\/span>[\/vc_column_text][vc_custom_heading text=&#8221;A\/B Testing &#8221; use_theme_fonts=&#8221;yes&#8221; el_class=&#8221;styled-heading blue-back&#8221; link=&#8221;url:%23|||&#8221;][vc_single_image image=&#8221;1285&#8243; img_size=&#8221;full&#8221; alignment=&#8221;center&#8221; el_class=&#8221;single-post-image-back&#8221;][vc_column_text]<span style=\"font-weight: 400;\">An A\/B test (also known as an A\/B\/n test or a Split Test) is designed to try out different creatives in a <\/span><b>win\/lose scenario,<\/b><span style=\"font-weight: 400;\"> resulting in a \u201cChampion\u201d<\/span> <span style=\"font-weight: 400;\">(i.e, a single winning creative). These tests are executed by running different creatives for a set amount of users and a set amount of time. This initial stage of the campaign is referred to as the <\/span><i><span style=\"font-weight: 400;\">exploration stage<\/span><\/i><span style=\"font-weight: 400;\">. Then, the winning creative, i.e, the one yielding better results (be it clicks, conversions, or any other set goal) is the one chosen to run for the rest of the campaign. This second stage of the campaign is the <\/span><i><span style=\"font-weight: 400;\">exploit<\/span><\/i><span style=\"font-weight: 400;\"> portion.<\/span>[\/vc_column_text][vc_column_text]<!-- Widget Shortcode --><div id=\"custom_html-7\" class=\"widget_text widget widget_custom_html widget-shortcode area-sidebar-form-mobile \"><div class=\"textwidget custom-html-widget\"><div class=\"subscribe-footer-from-holder sidebar-form-intouch\">\r\n<h3>Looking for app growth opportunities?<\/h3>\r\n<p>Leave your email and we'll get \r\nback to you!<\/p>\r\n<form class=\"subscribe-footer-from\">\r\n\t<div class=\"subscribe-footer-from-email-holder\"><input type=\"email\" class=\"subscribe-footer-email\" placeholder=\"Enter your email\"><\/div>\r\n<button class=\"subscribe-footer-submit\">\r\n\tGet in Touch\r\n<\/button>\r\n<\/form>\r\n<\/div>\r\n<div class='subscribe-footer-from-thankyou gsifthy'>\r\n\t<img alt=\"\" src=\"\/glossary\/wp-content\/uploads\/2020\/08\/02.svg\">\r\n<h3>Thank you for your interest!<\/h3>\r\n<p>Our team will contact you with more details.<\/p>\r\n<\/div>\r\n<div class=\"search-sidebar-footer-holder gisf\"><\/div><\/div><\/div><!-- \/Widget Shortcode -->[\/vc_column_text][vc_column_text]<span style=\"font-weight: 400;\">This type of test is rather simple to understand and set up. It\u2019s useful for initial testing, and when there\u2019s a need to choose a single champion. This may be useful to test out different creatives with starkly different styles, or when there are just a few creatives (2-3 options).\u00a0<\/span>[\/vc_column_text]<div class=\"aio-icon-component   info-noty style_1\"><div id=\"Info-box-wrap-4446\" class=\"aio-icon-box left-icon\" style=\"\"  ><div class=\"aio-icon-left\"><div class=\"ult-just-icon-wrapper  \"><div class=\"align-icon\" style=\"text-align:center;\">\n<div class=\"aio-icon-img \" style=\"font-size:71px;background:rgba(255,255,255,0.01);display:inline-block;\" >\n\t<img class=\"img-icon\" alt=\"null\" src=\"https:\/\/persona.ly\/glossary\/wp-content\/uploads\/2020\/03\/combined-shape-1.svg\"\/>\n<\/div><\/div><\/div><\/div><div class=\"aio-ibd-block\"><div class=\"aio-icon-header\" ><h3 class=\"aio-icon-title ult-responsive\"  data-ultimate-target='#Info-box-wrap-4446 .aio-icon-title'  data-responsive-json-new='{\"font-size\":\"\",\"line-height\":\"\"}'  style=\"color:#191536;\">Considering the Audience <\/h3><\/div> <!-- header --><div class=\"aio-icon-description ult-responsive\"  data-ultimate-target='#Info-box-wrap-4446 .aio-icon-description'  data-responsive-json-new='{\"font-size\":\"\",\"line-height\":\"\"}'  style=\"color:#191536;\"><span style=\"font-weight: 400;\">Creatives may perform differently on different audiences (users in different locations, new vs. existing users, etc\u2019). If you\u2019re running the same campaign (promoting the same product, game, or feature) on different audiences, you should test out how the same creatives perform on each of the different audiences. What works on a Japanese audience, for example, might not work for an American audience.\u00a0<\/span><\/div> <!-- description --><\/div> <!-- aio-ibd-block --><\/div> <!-- aio-icon-box --><\/div> <!-- aio-icon-component -->[vc_custom_heading text=&#8221;The Explore-Exploit Dilemma &#8221; font_container=&#8221;tag:h3|text_align:left|color:%23191536&#8243; use_theme_fonts=&#8221;yes&#8221; el_class=&#8221;styled-subheading&#8221;][vc_column_text]<span style=\"font-weight: 400;\">Part of the reason that A\/B testing is easy to understand and execute is that it offers a clear distinction between the <\/span><i><span style=\"font-weight: 400;\">exploration <\/span><\/i><span style=\"font-weight: 400;\">period and the <\/span><i><span style=\"font-weight: 400;\">exploitation <\/span><\/i><span style=\"font-weight: 400;\">period of the campaign. There\u2019s a clear separation between the two stages.\u00a0<\/span>[\/vc_column_text][vc_column_text]<span style=\"font-weight: 400;\">The exploration period is the time in which <\/span><span style=\"text-decoration: underline;\"><span style=\"font-weight: 400;\">all of the creatives<\/span><\/span><span style=\"font-weight: 400;\"> are tested against one another on the same audience, and the exploitation period is when <\/span><span style=\"font-weight: 400;\"><span style=\"text-decoration: underline;\">the winning creative runs exclusivel<\/span>y<\/span><span style=\"font-weight: 400;\">. The name <\/span><i><span style=\"font-weight: 400;\">exploitation<\/span><\/i><span style=\"font-weight: 400;\"> refers to taking advantage of the gathered data from the <\/span><i><span style=\"font-weight: 400;\">exploration <\/span><\/i><span style=\"font-weight: 400;\">period.\u00a0<\/span>[\/vc_column_text][vc_column_text]<span style=\"font-weight: 400;\">Other than its relative simplicity, A\/B testing offers <\/span><i><span style=\"font-weight: 400;\">control <\/span><\/i><span style=\"font-weight: 400;\">during the exploration period since advertisers can set in advance the audience, as well as the running time, which means that they can limit the budget spent \u201cunoptimized\u201d.\u00a0<\/span>[\/vc_column_text][vc_custom_heading text=&#8221;The Caveats of A\/B Testing &#8221; font_container=&#8221;tag:h3|text_align:left|color:%23191536&#8243; use_theme_fonts=&#8221;yes&#8221; el_class=&#8221;styled-subheading&#8221;][vc_column_text]<span style=\"font-weight: 400;\">The simplicity of the test is inviting but there are some disadvantages that should be taken into consideration.\u00a0<\/span>[\/vc_column_text][vc_column_text]<\/p>\n<ol class=\"default-list\">\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Limited Testing<\/b><span style=\"font-weight: 400;\"> &#8211; since the aim is to shorten the exploration as much as possible (as to not waste budget on less effective creatives), the audience on which the creatives are tested is significantly smaller than the overall audience. Marketers would want to choose the smallest possible audience to run these different ads on before settling on a creative, which means their sample might not be truly representative of the wider audience.\u00a0<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Higher Overall Costs<\/b><span style=\"font-weight: 400;\"> &#8211; even if marketers are testing their ads on the smallest possible audience, budget is still equally spent on creatives that aren\u2019t working as well as the winning creative. They may be 1% worse than the winning creative, but they may also be 50% worse &#8211; and in both cases the spent on these creatives will be equal during the testing stage.\u00a0<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>May Be Arbitrary<\/b><span style=\"font-weight: 400;\"> &#8211; there are no set best practices or simple <\/span><i><span style=\"font-weight: 400;\">correct<\/span><\/i><span style=\"font-weight: 400;\"> settings for an A\/B test. It depends on the size of the overall audience, the budget, the creatives, and how much they differ. The results may not be as conclusive as you\u2019d like them to. You may get a very small margin separating between the winning creative and other tested creatives, which may be caused because the audience you\u2019ve tested was too small or that both creatives were just as good, and there was no distinctive winner &#8211; a true \u201cchampion\u201d creative.\u00a0\u00a0<\/span><\/li>\n<\/ol>\n<p>[\/vc_column_text]<div class=\"aio-icon-component   info-noty blue-noty style_1\"><div id=\"Info-box-wrap-1571\" class=\"aio-icon-box left-icon\" style=\"\"  ><div class=\"aio-icon-left\"><div class=\"ult-just-icon-wrapper  \"><div class=\"align-icon\" style=\"text-align:center;\">\n<div class=\"aio-icon-img \" style=\"font-size:71px;background:rgba(255,255,255,0.01);display:inline-block;\" >\n\t<img class=\"img-icon\" alt=\"Info Container\" src=\"https:\/\/persona.ly\/glossary\/wp-content\/uploads\/2020\/03\/combined-shape-2.svg\"\/>\n<\/div><\/div><\/div><\/div><div class=\"aio-ibd-block\"><div class=\"aio-icon-header\" ><h3 class=\"aio-icon-title ult-responsive\"  data-ultimate-target='#Info-box-wrap-1571 .aio-icon-title'  data-responsive-json-new='{\"font-size\":\"\",\"line-height\":\"\"}'  style=\"color:#191536;\">A Holistic Overview - CPI and Other Performance KPIs<\/h3><\/div> <!-- header --><div class=\"aio-icon-description ult-responsive\"  data-ultimate-target='#Info-box-wrap-1571 .aio-icon-description'  data-responsive-json-new='{\"font-size\":\"\",\"line-height\":\"\"}'  style=\"color:#191536;\"><span style=\"font-weight: 400;\">While creative testing helps determine which creative is the best-performing one, in and of itself it&#8217;s not the single indicator for the campaign&#8217;s success. Since, as you well know by now, CPI, while instrumental, does not necessarily indicate users&#8217; quality. As we&#8217;ve previously covered, performance is measured by more than a single KPI. In our entry <a class=\"custom-link-content\" href=\"https:\/\/persona.ly\/glossary\/performance-metrics\/app-growth-metrics-the-key-user-acquisition-metrics-to-follow\/\">App Growth Metrics <\/a><\/span>we take a look at pre-install, and post-install metrics and how they serve as early indicators of the campaign&#8217;s potential. In another entry, <a class=\"custom-link-content\" href=\"https:\/\/persona.ly\/glossary\/performance-metrics\/the-cornerstone-kpis-for-mobile-ua\/\">The Cornerstone KPIs for Mobile UA<\/a>, we look at value-based KPIs, which are usually harder to calculate but serve as well-based predictive indicators. In order to get a holistic overview of the campaign, all of these KPIs should be taken into consideration.<\/div> <!-- description --><\/div> <!-- aio-ibd-block --><\/div> <!-- aio-icon-box --><\/div> <!-- aio-icon-component -->[vc_column_text]While our focus here is on <span style=\"text-decoration: underline;\">creatives testing<\/span>, A\/B testing can be used in many different ways. When explaining their split test tool, <a class=\"custom-link-content\" href=\"https:\/\/support.appsflyer.com\/hc\/en-us\/articles\/360001523657-AppsFlyer-A-B-testing-guide\" target=\"_blank\" rel=\"noopener noreferrer\">AppsFlyer<\/a> mentions testing different <span style=\"text-decoration: underline;\">media sources, ad placements, and more.<\/span><\/p>\n<p><a class=\"custom-link-content\" href=\"https:\/\/www.adjust.com\/blog\/ab-testing-for-mobile-apps\/\" target=\"_blank\" rel=\"noopener noreferrer\">Adjust<\/a>, on the other hand, offers running an <span style=\"text-decoration: underline;\">in-app A\/B test, as well as a marketing campaign test<\/span>, and adds a few important best practices.[\/vc_column_text][vc_custom_heading text=&#8221;The Multi-Armed Bandit (MAB)&#8221; use_theme_fonts=&#8221;yes&#8221; el_class=&#8221;styled-heading purple-back&#8221; link=&#8221;url:%23|||&#8221;][vc_column_text]<span style=\"font-weight: 400;\">The name and concept of the multi-armed bandit are derived from slot machines. These machines are originally called one-armed bandits since their mechanics is that gamblers pull the lever, AKA the arm, and, in return, the machine \u2018robs\u2019 them.\u00a0<\/span>[\/vc_column_text][vc_column_text]<span style=\"font-weight: 400;\">The multi-armed bandit offers an alternative and more intricate way to test creatives. The MAB is a group of algorithms, based on the idea of the one-armed bandit. These algorithms offer different solutions for the theoretical problem a gambler faces in the casino &#8211; \u201cwhat is the least expensive and fastest way to test all of the slot machines at the casino, assuming they vary in performance\u201d.\u00a0<\/span>[\/vc_column_text][vc_column_text]<span style=\"font-weight: 400;\">The concept is \u201ctesting as you go\u201d, as opposed to running an A\/B test with a clear start and endpoint. In MAB, the test is ongoing, operating continuously throughout the campaign. In mobile marketing that would mean that the tested ads run in parallel, while the weight (i.e. percentage of the budget) given to each ad is decided by its performance.\u00a0\u00a0<\/span>[\/vc_column_text][vc_column_text]<span style=\"font-weight: 400;\">Unlike A\/B testing, in the different multi-armed bandit algorithms, the explore and exploit stages are intertwined.\u00a0<\/span>[\/vc_column_text][vc_single_image image=&#8221;1292&#8243; img_size=&#8221;full&#8221; alignment=&#8221;center&#8221; el_class=&#8221;single-post-image-back&#8221;][vc_column_text]<span style=\"font-weight: 400;\">This solution uses an algorithm to constantly change the <\/span><i><span style=\"font-weight: 400;\">spending <\/span><\/i><span style=\"font-weight: 400;\">of each creative to coincide with its performance. This means that the best performing creative gets more budget, while the least performing one still runs, just with a smaller budget, and anything in between is still being used, based on its performance. There\u2019s a range and the budget is relative.\u00a0<\/span>[\/vc_column_text]<div class=\"aio-icon-component   info-noty blue-noty style_1\"><div id=\"Info-box-wrap-7476\" class=\"aio-icon-box left-icon\" style=\"\"  ><div class=\"aio-icon-left\"><div class=\"ult-just-icon-wrapper  \"><div class=\"align-icon\" style=\"text-align:center;\">\n<div class=\"aio-icon-img \" style=\"font-size:71px;background:rgba(255,255,255,0.01);display:inline-block;\" >\n\t<img class=\"img-icon\" alt=\"Info Container\" src=\"https:\/\/persona.ly\/glossary\/wp-content\/uploads\/2020\/03\/combined-shape-2.svg\"\/>\n<\/div><\/div><\/div><\/div><div class=\"aio-ibd-block\"><div class=\"aio-icon-header\" ><h3 class=\"aio-icon-title ult-responsive\"  data-ultimate-target='#Info-box-wrap-7476 .aio-icon-title'  data-responsive-json-new='{\"font-size\":\"\",\"line-height\":\"\"}'  style=\"color:#191536;\">Data Trends<\/h3><\/div> <!-- header --><div class=\"aio-icon-description ult-responsive\"  data-ultimate-target='#Info-box-wrap-7476 .aio-icon-description'  data-responsive-json-new='{\"font-size\":\"\",\"line-height\":\"\"}'  style=\"color:#191536;\"><span style=\"font-weight: 400;\">The multi-armed bandit algorithms follow trends in the data. If a creative\u2019s performance starts to decrease, the budget follows, and if that creative suddenly starts to improve its performance, the budget for it will gradually increase. If the current champion starts to deteriorate, it will lose its \u201ccrown\u201d so to speak.\u00a0<\/span><\/div> <!-- description --><\/div> <!-- aio-ibd-block --><\/div> <!-- aio-icon-box --><\/div> <!-- aio-icon-component -->[vc_column_text]<span style=\"font-weight: 400;\">Since there\u2019s no distinction between the exploration period to the exploitation in MAB (the two stages happen simultaneously), <span style=\"text-decoration: underline;\">MAB produces <\/span><\/span><span style=\"text-decoration: underline;\"><span style=\"font-weight: 400;\">faster results and is widely proven to be more efficient than A\/B testing.\u00a0<\/span><\/span>[\/vc_column_text][vc_column_text]<span style=\"font-weight: 400;\">Though the different MAB models differ, generally speaking, MAB models start to \u201cmove\u201d traffic towards the winning variations as soon as they detect the slightest variation. This means that, unlike an A\/B test, in MAB there\u2019s no waiting for the end of the experiment, which means that MAB algorithms not only work faster, they also reach results at a lower cost, as long as the creatives actually perform differently from one another.\u00a0<\/span>[\/vc_column_text]<div class=\"aio-icon-component   error-noty style_1\"><div id=\"Info-box-wrap-3081\" class=\"aio-icon-box left-icon\" style=\"\"  ><div class=\"aio-icon-left\"><div class=\"ult-just-icon-wrapper  \"><div class=\"align-icon\" style=\"text-align:center;\">\n<div class=\"aio-icon-img \" style=\"font-size:71px;background:rgba(255,255,255,0.01);display:inline-block;\" >\n\t<img class=\"img-icon\" alt=\"null\" src=\"https:\/\/persona.ly\/glossary\/wp-content\/uploads\/2020\/03\/combined-shape.svg\"\/>\n<\/div><\/div><\/div><\/div><div class=\"aio-ibd-block\"><div class=\"aio-icon-header\" ><h3 class=\"aio-icon-title ult-responsive\"  data-ultimate-target='#Info-box-wrap-3081 .aio-icon-title'  data-responsive-json-new='{\"font-size\":\"\",\"line-height\":\"\"}'  style=\"color:#191536;\">No Backsies - The Cost of Regret<\/h3><\/div> <!-- header --><div class=\"aio-icon-description ult-responsive\"  data-ultimate-target='#Info-box-wrap-3081 .aio-icon-description'  data-responsive-json-new='{\"font-size\":\"\",\"line-height\":\"\"}'  style=\"color:#191536;\"><i><span style=\"font-weight: 400;\">Regret<\/span><\/i><span style=\"font-weight: 400;\"> is calculated by measuring the difference between the conversions reached during a test vs. the <\/span><i><span style=\"font-weight: 400;\">potential of conversions<\/span><\/i><span style=\"font-weight: 400;\">. The <\/span><i><span style=\"font-weight: 400;\">potential of conversions<\/span><\/i><span style=\"font-weight: 400;\"> is the predicted number of conversions the campaign would\u2019ve reached if the campaign had run with the winning creative. This means that, most likely, regret in an A\/B test is likely to be higher than it is in MAB, since MAB is constantly optimized.\u00a0<\/span><\/div> <!-- description --><\/div> <!-- aio-ibd-block --><\/div> <!-- aio-icon-box --><\/div> <!-- aio-icon-component -->[vc_custom_heading text=&#8221;A\/B Testing vs. MAB&#8221; font_container=&#8221;tag:h3|text_align:left|color:%23191536&#8243; use_theme_fonts=&#8221;yes&#8221; el_class=&#8221;styled-subheading&#8221;][vc_column_text]<\/p>\n<ul class=\"default-list\">\n<li aria-level=\"1\"><b>Trends in data &#8211; <span style=\"font-weight: 400;\">MAB results may change over time. One creative may become more popular than it was, initially. It may be impacted by varying factors, but it will be reflected in the data since the test runs continuously. A\/B, on the other hand, only reflects the results of the limited time it ran in.<\/span><\/b><\/li>\n<\/ul>\n<ul class=\"default-list\">\n<li aria-level=\"1\"><strong>Adding Creatives &#8211;<\/strong> <span style=\"font-weight: 400;\">since different creatives get different percentages of the budget, it\u2019s easy to add new creatives to the mix and test them alongside the champion. If they \u201cshine\u201d, they gradually get more and more volume.\u00a0\u00a0<\/span><\/li>\n<\/ul>\n<ul class=\"default-list\">\n<li aria-level=\"1\"><strong>Earning while learning &#8211;<\/strong> <span style=\"font-weight: 400;\">by combining the explore and exploit stages, MAB offers lower costs since the optimization starts <\/span><i><span style=\"font-weight: 400;\">while <\/span><\/i><span style=\"font-weight: 400;\">data is still being collected. The response is immediate and optimization kicks in faster..\u00a0<\/span><\/li>\n<\/ul>\n<ul class=\"default-list\">\n<li aria-level=\"1\"><strong>Automation &#8211;<\/strong> <span style=\"font-weight: 400;\">Bandits are a very effective way to automate creative selection optimization with machine learning, especially when considering different user features\u2014since correct A\/B tests are much more complicated in that situation. The MAB algorithm can be activated at very granular levels, and select the best creative for subsections of the targeted audience specifically, while other subsections would have different champions.<\/span><\/li>\n<\/ul>\n<p>[\/vc_column_text][vc_custom_heading text=&#8221;When Should You Use A\/B Testing in Mobile Campaigns?&#8221; font_container=&#8221;tag:h3|text_align:left|color:%23191536&#8243; use_theme_fonts=&#8221;yes&#8221; el_class=&#8221;styled-subheading&#8221;][vc_column_text]<span style=\"font-weight: 400;\">When comparing the two methods, it sounds like MAB is the way to go, and a way to save money and get quicker results. Nonetheless, there\u2019s a reason for A\/B testing\u2019s popularity.\u00a0<\/span>[\/vc_column_text][vc_column_text]<\/p>\n<ol class=\"default-list\">\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">If your app is new and you\u2019ve yet to settle on a creative line and you want to test out a couple of very distinctive and different creatives, A\/B testing is a way to get a conclusive yes\\no answer (and if the margin is small, maybe you should run another test with different creatives).\u00a0<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">If your creatives are limited, and not endless. If you have 2-3 variations and not 10, it\u2019s a simple way to decide between the versions without wasting your budget on multiple variations.\u00a0<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">The majority of acquisition channels don\u2019t allow users to conduct MAB-based optimizations, only A\/B testing. This option leaves marketers with a choice between A\/B or no testing at all. In this scenario, A\/B testing is better than nothing.\u00a0\u00a0<\/span><\/li>\n<\/ol>\n<p>[\/vc_column_text][vc_custom_heading text=&#8221;The Different Types of MAB Algorithms&#8221; use_theme_fonts=&#8221;yes&#8221; el_class=&#8221;styled-heading blue-back&#8221; link=&#8221;url:%23|||&#8221;][vc_column_text]<span style=\"font-weight: 400;\">Since it\u2019s a complex topic, the aim here is to give a simplified version of the most relevant MAB models for marketing campaigns in general, and creative testing specifically.\u00a0<\/span>[\/vc_column_text][vc_custom_heading text=&#8221;Epsilon Greedy&#8221; font_container=&#8221;tag:h3|text_align:left|color:%23191536&#8243; use_theme_fonts=&#8221;yes&#8221; el_class=&#8221;styled-subheading&#8221;][vc_column_text]<span style=\"font-weight: 400;\">Epsilon greedy, as the name indicates, is a MAB model that gives the largest part of the budget to the champion. In Epsilon Greedy, the person running the campaign decides on an epsilon (i.e, a percentage of the budget dedicated to the challengers) and the rest of the budget is allocated to the champion.\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Let\u2019s say the epsilon is set at 20%, this means that 80% of the budget of the campaign goes to running the champion, and the 20% is <\/span><span style=\"text-decoration: underline;\"><span style=\"font-weight: 400;\">equally divided between the rest of the creatives<\/span><\/span><span style=\"font-weight: 400;\">. This means that epsilon greedy somewhat resembles an A\/B test, but if there are changes in the performance during the campaign, a challenger may become the new champion.\u00a0\u00a0<\/span>[\/vc_column_text][vc_custom_heading text=&#8221;Thompson Sampling&#8221; font_container=&#8221;tag:h3|text_align:left|color:%23191536&#8243; use_theme_fonts=&#8221;yes&#8221; el_class=&#8221;styled-subheading&#8221;][vc_column_text]<span style=\"font-weight: 400;\">Thompson sampling, on the other hand, still utilizes the epsilon, but in its case, the challengers get budgeted in relation to their performance. Let\u2019s say, again, that the epsilon is set at 20%. This means that 80% of the budget goes to the champion, while the 20% are divided based on their relative performance (some creatives will get 8% of the budget, some might get just 1%, and as the performance shifts, so will the budget).\u00a0<\/span>[\/vc_column_text][vc_custom_heading text=&#8221;The Upper Confidence Bound (UCB)&#8221; font_container=&#8221;tag:h3|text_align:left|color:%23191536&#8243; use_theme_fonts=&#8221;yes&#8221; el_class=&#8221;styled-subheading&#8221;][vc_column_text]<span style=\"font-weight: 400;\">UCB may be referred to as the opposite of Thompson Sampling. UCB challenges the least explored creative to test if its low performance is related to the limited budget it\u2019s been getting. When testing creatives, it\u2019s least likely to be used, but when testing, for example, new inventory pockets with higher bids, it might yield interesting results.\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">If there are bid floors you\u2019ve been avoiding, since they\u2019re too high, it might be worth exploring, <\/span>since these pockets of inventory might yield higher LTV users.[\/vc_column_text][vc_custom_heading text=&#8221;Contextual Bandits &#8221; font_container=&#8221;tag:h3|text_align:left|color:%23191536&#8243; use_theme_fonts=&#8221;yes&#8221; el_class=&#8221;styled-subheading&#8221;][vc_column_text]<span style=\"font-weight: 400;\">Contextual bandits, as the name implies, are MAB algorithms that group around context. In the case of creative testing, contextual bandits may be grouped around gender, location, or other contextual relations and set a different champion and challengers for each group. This means that instead of a general competition between all of the creatives and the audience, the audience is segmented, and each segment gets its own champion.\u00a0<\/span>[\/vc_column_text]<div class=\"aio-icon-component   info-noty blue-noty style_1\"><div id=\"Info-box-wrap-7625\" class=\"aio-icon-box left-icon\" style=\"\"  ><div class=\"aio-icon-left\"><div class=\"ult-just-icon-wrapper  \"><div class=\"align-icon\" style=\"text-align:center;\">\n<div class=\"aio-icon-img \" style=\"font-size:71px;background:rgba(255,255,255,0.01);display:inline-block;\" >\n\t<img class=\"img-icon\" alt=\"Info Container\" src=\"https:\/\/persona.ly\/glossary\/wp-content\/uploads\/2020\/03\/combined-shape-2.svg\"\/>\n<\/div><\/div><\/div><\/div><div class=\"aio-ibd-block\"><div class=\"aio-icon-header\" ><h3 class=\"aio-icon-title ult-responsive\"  data-ultimate-target='#Info-box-wrap-7625 .aio-icon-title'  data-responsive-json-new='{\"font-size\":\"\",\"line-height\":\"\"}'  style=\"color:#191536;\">Creative Testing in iOS 14 <\/h3><\/div> <!-- header --><div class=\"aio-icon-description ult-responsive\"  data-ultimate-target='#Info-box-wrap-7625 .aio-icon-description'  data-responsive-json-new='{\"font-size\":\"\",\"line-height\":\"\"}'  style=\"color:#191536;\"><span style=\"font-weight: 400;\">With the deprecation of the IDFA in iOS 14, and the reliance on the data in the SDKAdNetwork (or SKAN, in short), creative testing is even more essential than it was before. Since optimizing based on aggregated data is virtually impossible, creatives will take center-stage again. Showing potential users an accurate and appealing ad showcasing the app or game is vital.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">You can read more about <a class=\"custom-link-content\" href=\"https:\/\/persona.ly\/glossary\/attribution\/user-acquisition-and-retargeting-in-ios-14\/\" target=\"_blank\" rel=\"noopener noreferrer\">UA and retargeting in iOS 14 <\/a><\/span>and review our <a class=\"custom-link-content\" href=\"https:\/\/persona.ly\/glossary\/attribution\/getting-ready-for-ios-14-5-an-extensive-guide-for-mobile-marketers\/\" target=\"_blank\" rel=\"noopener noreferrer\">extensive iOS 14 readiness guide<\/a> for more details about the upcoming changes and how to prepare for them.<\/div> <!-- description --><\/div> <!-- aio-ibd-block --><\/div> <!-- aio-icon-box --><\/div> <!-- aio-icon-component -->[\/vc_column][\/vc_row]<\/p>\n\t\t\t\t\t\t<\/div><!-- .entry-content -->\n\t\t\t\t\t<\/div><!-- wp-show-posts-inner --><div class=\"wpsp-clear\"><\/div><\/article><\/section><!-- .wp-show-posts -->[\/vc_column_text][\/vc_column][vc_column width=&#8221;1\/3&#8243; el_class=&#8221;gap-in-end-single single-subcat-widget&#8221;][vc_column_text][\/vc_column_text][vc_single_image image=&#8221;1297&#8243; img_size=&#8221;full&#8221; alignment=&#8221;center&#8221; onclick=&#8221;custom_link&#8221; el_class=&#8221;stick-img-holder-single hide-mobile&#8221; css=&#8221;.vc_custom_1617028964170{margin-bottom: 43px !important;}&#8221; link=&#8221;https:\/\/persona.ly\/context-distance-calculator?utm_source=post&amp;utm_medium=glossary&amp;utm_campaign=creatives&#8221;][vc_column_text]<!-- Widget Shortcode --><div id=\"custom_html-6\" class=\"widget_text widget widget_custom_html widget-shortcode area-sidebar-form \"><div class=\"textwidget custom-html-widget\"><div class=\"subscribe-footer-from-holder sidebar-form-intouch\">\r\n<h3>Looking for app growth opportunities?<\/h3>\r\n<p>Leave your email and we'll get \r\nback to you!<\/p>\r\n<form id=\"subscribe-footer-from\" class=\"subscribe-footer-from\">\r\n\t<div class=\"subscribe-footer-from-email-holder\"><input type=\"email\" class=\"subscribe-footer-email\" placeholder=\"Enter your email\"><\/div>\r\n<button data-sitekey=\"6Lcl6eIkAAAAAIJsyVC0Qmc2ZTAgfz3pqyAeJjYI\" data-callback=\"onSubscribeFooterSubmit\" data-action=\"submit\" class=\"g-recaptcha subscribe-footer-submit\">\r\n\tGet in Touch\r\n<\/button>\r\n<\/form>\r\n<\/div>\r\n<div class='subscribe-footer-from-thankyou gsifthy'>\r\n\t<img alt=\"\" src=\"\/glossary\/wp-content\/uploads\/2020\/08\/02.svg\">\r\n<h3>Thank you for your interest!<\/h3>\r\n<p>Our team will contact you with more details.<\/p>\r\n<\/div>\r\n<div class=\"search-sidebar-footer-holder gisf\"><\/div><\/div><\/div><!-- \/Widget Shortcode -->[\/vc_column_text][vc_single_image image=&#8221;1295&#8243; img_size=&#8221;full&#8221; alignment=&#8221;center&#8221; el_class=&#8221;stick-img-holder-single hide-mobile&#8221;][\/vc_column][\/vc_row][\/vc_section]<\/p>\n","protected":false},"excerpt":{"rendered":"<p>[vc_row el_class=&#8221;front-hero-section&#8221;][vc_column width=&#8221;1\/2&#8243; css=&#8221;.vc_custom_1582730807176{background-image: url(https:\/\/persona.ly\/glossary\/\/wp-content\/uploads\/2020\/02\/group-19.svg?id=58) !important;background-position: center !important;background-repeat: no-repeat !important;background-size: contain !important;}&#8221; el_class=&#8221;hero-first-section&#8221;][vc_column_text el_class=&#8221;page-heading&#8221; css=&#8221;.vc_custom_1593096796228{margin-top: 30px !important;}&#8221;] UA Glossary [\/vc_column_text][vc_column_text][\/vc_column_text][\/vc_column][vc_column width=&#8221;1\/2&#8243;][vc_single_image image=&#8221;60&#8243; img_size=&#8221;full&#8221; alignment=&#8221;center&#8221; el_class=&#8221;hero-section-image&#8221;][\/vc_column][\/vc_row][vc_section css=&#8221;.vc_custom_1583429011820{background-image: url(https:\/\/persona.ly\/glossary\/\/wp-content\/uploads\/2020\/03\/bg_repeater_with_bg_svg.svg?id=320) !important;}&#8221; el_class=&#8221;post-content-section archive-page-custom single-subcat&#8221;][vc_row el_class=&#8221;covered-grace-font&#8221;][vc_column width=&#8221;2\/3&#8243; el_class=&#8221;gap-in-end-single&#8221;][vc_column_text][\/vc_column_text][vc_column_text][\/vc_column_text][\/vc_column][vc_column width=&#8221;1\/3&#8243; el_class=&#8221;gap-in-end-single single-subcat-widget&#8221;][vc_column_text][\/vc_column_text][vc_single_image image=&#8221;1297&#8243; img_size=&#8221;full&#8221; alignment=&#8221;center&#8221; onclick=&#8221;custom_link&#8221; el_class=&#8221;stick-img-holder-single hide-mobile&#8221; css=&#8221;.vc_custom_1617028964170{margin-bottom: 43px !important;}&#8221; link=&#8221;https:\/\/persona.ly\/context-distance-calculator?utm_source=post&amp;utm_medium=glossary&amp;utm_campaign=creatives&#8221;][vc_column_text][\/vc_column_text][vc_single_image image=&#8221;1295&#8243; img_size=&#8221;full&#8221; alignment=&#8221;center&#8221; el_class=&#8221;stick-img-holder-single hide-mobile&#8221;][\/vc_column][\/vc_row][\/vc_section]<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":660,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v19.10 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Creative Testing - Multi-Armed Bandit vs. A\/B Testing - The UA Glossary - Persona.ly<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/persona.ly\/glossary\/performance-metrics\/creative-testing-multi-armed-bandit-vs-a-b-testing\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Creative Testing - Multi-Armed Bandit vs. A\/B Testing - The UA Glossary - Persona.ly\" \/>\n<meta property=\"og:description\" content=\"[vc_row el_class=&#8221;front-hero-section&#8221;][vc_column width=&#8221;1\/2&#8243; css=&#8221;.vc_custom_1582730807176{background-image: url(https:\/\/persona.ly\/glossary\/\/wp-content\/uploads\/2020\/02\/group-19.svg?id=58) !important;background-position: center !important;background-repeat: no-repeat !important;background-size: contain !important;}&#8221; el_class=&#8221;hero-first-section&#8221;][vc_column_text el_class=&#8221;page-heading&#8221; css=&#8221;.vc_custom_1593096796228{margin-top: 30px !important;}&#8221;] UA Glossary [\/vc_column_text][vc_column_text][\/vc_column_text][\/vc_column][vc_column width=&#8221;1\/2&#8243;][vc_single_image image=&#8221;60&#8243; img_size=&#8221;full&#8221; alignment=&#8221;center&#8221; el_class=&#8221;hero-section-image&#8221;][\/vc_column][\/vc_row][vc_section css=&#8221;.vc_custom_1583429011820{background-image: url(https:\/\/persona.ly\/glossary\/\/wp-content\/uploads\/2020\/03\/bg_repeater_with_bg_svg.svg?id=320) !important;}&#8221; el_class=&#8221;post-content-section archive-page-custom single-subcat&#8221;][vc_row el_class=&#8221;covered-grace-font&#8221;][vc_column width=&#8221;2\/3&#8243; el_class=&#8221;gap-in-end-single&#8221;][vc_column_text][\/vc_column_text][vc_column_text][\/vc_column_text][\/vc_column][vc_column width=&#8221;1\/3&#8243; el_class=&#8221;gap-in-end-single single-subcat-widget&#8221;][vc_column_text][\/vc_column_text][vc_single_image image=&#8221;1297&#8243; img_size=&#8221;full&#8221; alignment=&#8221;center&#8221; onclick=&#8221;custom_link&#8221; el_class=&#8221;stick-img-holder-single hide-mobile&#8221; css=&#8221;.vc_custom_1617028964170{margin-bottom: 43px !important;}&#8221; link=&#8221;https:\/\/persona.ly\/context-distance-calculator?utm_source=post&amp;utm_medium=glossary&amp;utm_campaign=creatives&#8221;][vc_column_text][\/vc_column_text][vc_single_image image=&#8221;1295&#8243; img_size=&#8221;full&#8221; alignment=&#8221;center&#8221; el_class=&#8221;stick-img-holder-single hide-mobile&#8221;][\/vc_column][\/vc_row][\/vc_section]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/persona.ly\/glossary\/performance-metrics\/creative-testing-multi-armed-bandit-vs-a-b-testing\/\" \/>\n<meta property=\"og:site_name\" content=\"The UA Glossary - Persona.ly\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/persona.ly.monetization\" \/>\n<meta property=\"article:modified_time\" content=\"2021-03-29T14:42:59+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/persona.ly\/glossary\/wp-content\/uploads\/2020\/06\/Metadata-Image-1.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"1200\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:site\" content=\"@persona_ly\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/persona.ly\/glossary\/performance-metrics\/creative-testing-multi-armed-bandit-vs-a-b-testing\/\",\"url\":\"https:\/\/persona.ly\/glossary\/performance-metrics\/creative-testing-multi-armed-bandit-vs-a-b-testing\/\",\"name\":\"Creative Testing - Multi-Armed Bandit vs. A\/B Testing - The UA Glossary - Persona.ly\",\"isPartOf\":{\"@id\":\"https:\/\/persona.ly\/glossary\/#website\"},\"datePublished\":\"2021-03-29T11:47:17+00:00\",\"dateModified\":\"2021-03-29T14:42:59+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/persona.ly\/glossary\/performance-metrics\/creative-testing-multi-armed-bandit-vs-a-b-testing\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/persona.ly\/glossary\/performance-metrics\/creative-testing-multi-armed-bandit-vs-a-b-testing\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/persona.ly\/glossary\/performance-metrics\/creative-testing-multi-armed-bandit-vs-a-b-testing\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/persona.ly\/glossary\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Performance Metrics\",\"item\":\"https:\/\/persona.ly\/glossary\/performance-metrics\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Creative Testing &#8211; Multi-Armed Bandit vs. A\/B Testing\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/persona.ly\/glossary\/#website\",\"url\":\"https:\/\/persona.ly\/glossary\/\",\"name\":\"The UA Glossary - Persona.ly\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\/\/persona.ly\/glossary\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/persona.ly\/glossary\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/persona.ly\/glossary\/#organization\",\"name\":\"The UA Glossary - Persona.ly\",\"url\":\"https:\/\/persona.ly\/glossary\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/persona.ly\/glossary\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/persona.ly\/glossary\/wp-content\/uploads\/2022\/10\/logo-personaly-black.svg\",\"contentUrl\":\"https:\/\/persona.ly\/glossary\/wp-content\/uploads\/2022\/10\/logo-personaly-black.svg\",\"width\":180,\"height\":37,\"caption\":\"The UA Glossary - Persona.ly\"},\"image\":{\"@id\":\"https:\/\/persona.ly\/glossary\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/persona.ly.monetization\",\"https:\/\/twitter.com\/persona_ly\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Creative Testing - Multi-Armed Bandit vs. A\/B Testing - The UA Glossary - Persona.ly","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/persona.ly\/glossary\/performance-metrics\/creative-testing-multi-armed-bandit-vs-a-b-testing\/","og_locale":"en_US","og_type":"article","og_title":"Creative Testing - Multi-Armed Bandit vs. A\/B Testing - The UA Glossary - Persona.ly","og_description":"[vc_row el_class=&#8221;front-hero-section&#8221;][vc_column width=&#8221;1\/2&#8243; css=&#8221;.vc_custom_1582730807176{background-image: url(https:\/\/persona.ly\/glossary\/\/wp-content\/uploads\/2020\/02\/group-19.svg?id=58) !important;background-position: center !important;background-repeat: no-repeat !important;background-size: contain !important;}&#8221; el_class=&#8221;hero-first-section&#8221;][vc_column_text el_class=&#8221;page-heading&#8221; css=&#8221;.vc_custom_1593096796228{margin-top: 30px !important;}&#8221;] UA Glossary [\/vc_column_text][vc_column_text][\/vc_column_text][\/vc_column][vc_column width=&#8221;1\/2&#8243;][vc_single_image image=&#8221;60&#8243; img_size=&#8221;full&#8221; alignment=&#8221;center&#8221; el_class=&#8221;hero-section-image&#8221;][\/vc_column][\/vc_row][vc_section css=&#8221;.vc_custom_1583429011820{background-image: url(https:\/\/persona.ly\/glossary\/\/wp-content\/uploads\/2020\/03\/bg_repeater_with_bg_svg.svg?id=320) !important;}&#8221; el_class=&#8221;post-content-section archive-page-custom single-subcat&#8221;][vc_row el_class=&#8221;covered-grace-font&#8221;][vc_column width=&#8221;2\/3&#8243; el_class=&#8221;gap-in-end-single&#8221;][vc_column_text][\/vc_column_text][vc_column_text][\/vc_column_text][\/vc_column][vc_column width=&#8221;1\/3&#8243; el_class=&#8221;gap-in-end-single single-subcat-widget&#8221;][vc_column_text][\/vc_column_text][vc_single_image image=&#8221;1297&#8243; img_size=&#8221;full&#8221; alignment=&#8221;center&#8221; onclick=&#8221;custom_link&#8221; el_class=&#8221;stick-img-holder-single hide-mobile&#8221; css=&#8221;.vc_custom_1617028964170{margin-bottom: 43px !important;}&#8221; link=&#8221;https:\/\/persona.ly\/context-distance-calculator?utm_source=post&amp;utm_medium=glossary&amp;utm_campaign=creatives&#8221;][vc_column_text][\/vc_column_text][vc_single_image image=&#8221;1295&#8243; img_size=&#8221;full&#8221; alignment=&#8221;center&#8221; el_class=&#8221;stick-img-holder-single hide-mobile&#8221;][\/vc_column][\/vc_row][\/vc_section]","og_url":"https:\/\/persona.ly\/glossary\/performance-metrics\/creative-testing-multi-armed-bandit-vs-a-b-testing\/","og_site_name":"The UA Glossary - Persona.ly","article_publisher":"https:\/\/www.facebook.com\/persona.ly.monetization","article_modified_time":"2021-03-29T14:42:59+00:00","og_image":[{"width":1200,"height":1200,"url":"https:\/\/persona.ly\/glossary\/wp-content\/uploads\/2020\/06\/Metadata-Image-1.png","type":"image\/png"}],"twitter_card":"summary_large_image","twitter_site":"@persona_ly","twitter_misc":{"Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/persona.ly\/glossary\/performance-metrics\/creative-testing-multi-armed-bandit-vs-a-b-testing\/","url":"https:\/\/persona.ly\/glossary\/performance-metrics\/creative-testing-multi-armed-bandit-vs-a-b-testing\/","name":"Creative Testing - Multi-Armed Bandit vs. A\/B Testing - The UA Glossary - Persona.ly","isPartOf":{"@id":"https:\/\/persona.ly\/glossary\/#website"},"datePublished":"2021-03-29T11:47:17+00:00","dateModified":"2021-03-29T14:42:59+00:00","breadcrumb":{"@id":"https:\/\/persona.ly\/glossary\/performance-metrics\/creative-testing-multi-armed-bandit-vs-a-b-testing\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/persona.ly\/glossary\/performance-metrics\/creative-testing-multi-armed-bandit-vs-a-b-testing\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/persona.ly\/glossary\/performance-metrics\/creative-testing-multi-armed-bandit-vs-a-b-testing\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/persona.ly\/glossary\/"},{"@type":"ListItem","position":2,"name":"Performance Metrics","item":"https:\/\/persona.ly\/glossary\/performance-metrics\/"},{"@type":"ListItem","position":3,"name":"Creative Testing &#8211; Multi-Armed Bandit vs. A\/B Testing"}]},{"@type":"WebSite","@id":"https:\/\/persona.ly\/glossary\/#website","url":"https:\/\/persona.ly\/glossary\/","name":"The UA Glossary - Persona.ly","description":"","publisher":{"@id":"https:\/\/persona.ly\/glossary\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/persona.ly\/glossary\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/persona.ly\/glossary\/#organization","name":"The UA Glossary - Persona.ly","url":"https:\/\/persona.ly\/glossary\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/persona.ly\/glossary\/#\/schema\/logo\/image\/","url":"https:\/\/persona.ly\/glossary\/wp-content\/uploads\/2022\/10\/logo-personaly-black.svg","contentUrl":"https:\/\/persona.ly\/glossary\/wp-content\/uploads\/2022\/10\/logo-personaly-black.svg","width":180,"height":37,"caption":"The UA Glossary - Persona.ly"},"image":{"@id":"https:\/\/persona.ly\/glossary\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/persona.ly.monetization","https:\/\/twitter.com\/persona_ly"]}]}},"_links":{"self":[{"href":"https:\/\/persona.ly\/glossary\/wp-json\/wp\/v2\/pages\/1274"}],"collection":[{"href":"https:\/\/persona.ly\/glossary\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/persona.ly\/glossary\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/persona.ly\/glossary\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/persona.ly\/glossary\/wp-json\/wp\/v2\/comments?post=1274"}],"version-history":[{"count":0,"href":"https:\/\/persona.ly\/glossary\/wp-json\/wp\/v2\/pages\/1274\/revisions"}],"up":[{"embeddable":true,"href":"https:\/\/persona.ly\/glossary\/wp-json\/wp\/v2\/pages\/660"}],"wp:attachment":[{"href":"https:\/\/persona.ly\/glossary\/wp-json\/wp\/v2\/media?parent=1274"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}