{"id":96,"date":"2022-03-29T11:50:06","date_gmt":"2022-03-29T11:50:06","guid":{"rendered":"https:\/\/persona.ly\/glossary\/performance-metrics\/creative-testing-multi-armed-bandit-vs-a-b-testing\/"},"modified":"2022-12-11T12:40:19","modified_gmt":"2022-12-11T12:40:19","slug":"creative-testing-multi-armed-bandit-vs-a-b-testing","status":"publish","type":"page","link":"https:\/\/persona.ly\/glossary\/performance-metrics\/creative-testing-multi-armed-bandit-vs-a-b-testing\/","title":{"rendered":"Creative Testing  &#8211; Multi-Armed Bandit vs. A\/B Testing"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">Creative testing is the way marketers compare the performance of different creatives in a campaign in order to evaluate which creatives yield better results. We will cover two methods: <strong>A\/B testing<\/strong>, which is a commonly used method and widely known method, and the <strong>multi-armed bandit<\/strong>, which is less commonly known and used, usually due to its relative complexity.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">These two methods offer very different approaches to creative testing, each with its own advantages and disadvantages. The aim here is to help marketers understand these differences and bring attention to some of the limitations of A\/B testing and help them make a more informed decision.<\/p>\n\n\n\n<h2 class=\"wp-block-heading is-style-tape-blue\">A\/B Testing<\/h2>\n\n\n\n<figure class=\"wp-block-image size-full is-style-framed\"><img loading=\"lazy\" decoding=\"async\" width=\"1382\" height=\"640\" src=\"https:\/\/persona.ly\/glossary\/wp-content\/uploads\/2021\/03\/b_testing_1a.png\" alt=\"A\/B testing AKA Split test\" class=\"wp-image-94\" srcset=\"https:\/\/persona.ly\/glossary\/wp-content\/uploads\/2021\/03\/b_testing_1a.png 1382w, https:\/\/persona.ly\/glossary\/wp-content\/uploads\/2021\/03\/b_testing_1a-300x139.png 300w, https:\/\/persona.ly\/glossary\/wp-content\/uploads\/2021\/03\/b_testing_1a-1024x474.png 1024w, https:\/\/persona.ly\/glossary\/wp-content\/uploads\/2021\/03\/b_testing_1a-768x356.png 768w\" sizes=\"auto, (max-width: 1382px) 100vw, 1382px\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">An A\/B test (also known as an A\/B\/n test or a Split Test) is designed to try out different creatives in a <strong>win\/lose scenario,<\/strong> resulting in a \u201cChampion\u201d (i.e, a single winning creative). These tests are executed by running different creatives for a set amount of users and a set amount of time. This initial stage of the campaign is referred to as the <em>exploration stage<\/em>. Then, the winning creative, i.e, the one yielding better results (be it clicks, conversions, or any other set goal) is the one chosen to run for the rest of the campaign. This second stage of the campaign is the <em>exploit<\/em> portion.<\/p>\n\n\n<aside class=\"inline-cta\">\t<div class=\"lead lead--inline\">\n\t\t<p class=\"lead__title\">Looking for app growth opportunities?<\/p>\n\t\t<p class=\"lead__text\">Leave your email and we\u2019ll get back to you!<\/p>\n\t\t<form class=\"lead-form\" novalidate data-placement=\"inline\">\n\t\t\t<label class=\"screen-reader-text\" for=\"lead-email-1\">Work email<\/label>\n\t\t\t<input class=\"lead-form__email\" id=\"lead-email-1\" type=\"email\" name=\"email\" autocomplete=\"email\" placeholder=\"Enter your email\" required>\n\t\t\t<button class=\"button button--green lead-form__submit\" type=\"submit\">Get in touch<\/button>\n\t\t\t<p class=\"lead-form__error\" role=\"alert\" hidden>Please enter a valid email address.<\/p>\n\t\t<\/form>\n\t\t<p class=\"lead-form__thanks\" role=\"status\" hidden>Thank you for your interest! Our team will contact you with more details.<\/p>\n\t<\/div>\n\t<\/aside>\n\n\n\n<p class=\"wp-block-paragraph\">This type of test is rather simple to understand and set up. It\u2019s useful for initial testing, and when there\u2019s a need to choose a single champion. This may be useful to test out different creatives with starkly different styles, or when there are just a few creatives (2-3 options).<\/p>\n\n\n\n<div class=\"wp-block-group is-style-note\"><div class=\"wp-block-group__inner-container is-layout-constrained wp-block-group-is-layout-constrained\">\n<p class=\"note-title wp-block-paragraph\"><strong>Considering the Audience<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Creatives may perform differently on different audiences (users in different locations, new vs. existing users, etc\u2019). If you\u2019re running the same campaign (promoting the same product, game, or feature) on different audiences, you should test out how the same creatives perform on each of the different audiences. What works on a Japanese audience, for example, might not work for an American audience.<\/p>\n<\/div><\/div>\n\n\n\n<h3 class=\"wp-block-heading\">The Explore-Exploit Dilemma<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Part of the reason that A\/B testing is easy to understand and execute is that it offers a clear distinction between the <em>exploration <\/em>period and the <em>exploitation <\/em>period of the campaign. There\u2019s a clear separation between the two stages.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The exploration period is the time in which all of the creatives are tested against one another on the same audience, and the exploitation period is when the winning creative runs exclusively. The name <em>exploitation<\/em> refers to taking advantage of the gathered data from the <em>exploration <\/em>period.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Other than its relative simplicity, A\/B testing offers <em>control <\/em>during the exploration period since advertisers can set in advance the audience, as well as the running time, which means that they can limit the budget spent \u201cunoptimized\u201d.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">The Caveats of A\/B Testing<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The simplicity of the test is inviting but there are some disadvantages that should be taken into consideration.<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>Limited Testing<\/strong> &#8211; since the aim is to shorten the exploration as much as possible (as to not waste budget on less effective creatives), the audience on which the creatives are tested is significantly smaller than the overall audience. Marketers would want to choose the smallest possible audience to run these different ads on before settling on a creative, which means their sample might not be truly representative of the wider audience.<\/li>\n\n\n<li><strong>Higher Overall Costs<\/strong> &#8211; even if marketers are testing their ads on the smallest possible audience, budget is still equally spent on creatives that aren\u2019t working as well as the winning creative. They may be 1% worse than the winning creative, but they may also be 50% worse &#8211; and in both cases the spent on these creatives will be equal during the testing stage.<\/li>\n\n\n<li><strong>May Be Arbitrary<\/strong> &#8211; there are no set best practices or simple <em>correct<\/em> settings for an A\/B test. It depends on the size of the overall audience, the budget, the creatives, and how much they differ. The results may not be as conclusive as you\u2019d like them to. You may get a very small margin separating between the winning creative and other tested creatives, which may be caused because the audience you\u2019ve tested was too small or that both creatives were just as good, and there was no distinctive winner &#8211; a true \u201cchampion\u201d creative.<\/li>\n<\/ol>\n\n\n\n<div class=\"wp-block-group is-style-note-blue\"><div class=\"wp-block-group__inner-container is-layout-constrained wp-block-group-is-layout-constrained\">\n<p class=\"note-title wp-block-paragraph\"><strong>A Holistic Overview &#8211; CPI and Other Performance KPIs<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">While creative testing helps determine which creative is the best-performing one, in and of itself it&#8217;s not the single indicator for the campaign&#8217;s success. Since, as you well know by now, CPI, while instrumental, does not necessarily indicate users&#8217; quality. As we&#8217;ve previously covered, performance is measured by more than a single KPI. In our entry <a href=\"https:\/\/persona.ly\/glossary\/performance-metrics\/app-growth-metrics-the-key-user-acquisition-metrics-to-follow\/\">App Growth Metrics <\/a>we take a look at pre-install, and post-install metrics and how they serve as early indicators of the campaign&#8217;s potential. In another entry, <a href=\"https:\/\/persona.ly\/glossary\/performance-metrics\/the-cornerstone-kpis-for-mobile-ua\/\">The Cornerstone KPIs for Mobile UA<\/a>, we look at value-based KPIs, which are usually harder to calculate but serve as well-based predictive indicators. In order to get a holistic overview of the campaign, all of these KPIs should be taken into consideration.<\/p>\n<\/div><\/div>\n\n\n\n<p class=\"wp-block-paragraph\">While our focus here is on creatives testing, A\/B testing can be used in many different ways. When explaining their split test tool, <a href=\"https:\/\/support.appsflyer.com\/hc\/en-us\/articles\/360001523657-AppsFlyer-A-B-testing-guide\" rel=\"noopener\" target=\"_blank\">AppsFlyer<\/a> mentions testing different media sources, ad placements, and more.<br><a href=\"https:\/\/www.adjust.com\/blog\/ab-testing-for-mobile-apps\/\" rel=\"noopener\" target=\"_blank\">Adjust<\/a>, on the other hand, offers running an in-app A\/B test, as well as a marketing campaign test, and adds a few important best practices.<\/p>\n\n\n\n<h2 class=\"wp-block-heading is-style-tape-purple\">The Multi-Armed Bandit (MAB)<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The name and concept of the multi-armed bandit are derived from slot machines. These machines are originally called one-armed bandits since their mechanics is that gamblers pull the lever, AKA the arm, and, in return, the machine \u2018robs\u2019 them.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The multi-armed bandit offers an alternative and more intricate way to test creatives. The MAB is a group of algorithms, based on the idea of the one-armed bandit. These algorithms offer different solutions for the theoretical problem a gambler faces in the casino &#8211; \u201cwhat is the least expensive and fastest way to test all of the slot machines at the casino, assuming they vary in performance\u201d.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The concept is \u201ctesting as you go\u201d, as opposed to running an A\/B test with a clear start and endpoint. In MAB, the test is ongoing, operating continuously throughout the campaign. In mobile marketing that would mean that the tested ads run in parallel, while the weight (i.e. percentage of the budget) given to each ad is decided by its performance.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Unlike A\/B testing, in the different multi-armed bandit algorithms, the explore and exploit stages are intertwined.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full is-style-framed\"><img loading=\"lazy\" decoding=\"async\" width=\"1382\" height=\"640\" src=\"https:\/\/persona.ly\/glossary\/wp-content\/uploads\/2021\/03\/b_vs_mab_1a.png\" alt=\"ab test vs mab\" class=\"wp-image-95\" srcset=\"https:\/\/persona.ly\/glossary\/wp-content\/uploads\/2021\/03\/b_vs_mab_1a.png 1382w, https:\/\/persona.ly\/glossary\/wp-content\/uploads\/2021\/03\/b_vs_mab_1a-300x139.png 300w, https:\/\/persona.ly\/glossary\/wp-content\/uploads\/2021\/03\/b_vs_mab_1a-1024x474.png 1024w, https:\/\/persona.ly\/glossary\/wp-content\/uploads\/2021\/03\/b_vs_mab_1a-768x356.png 768w\" sizes=\"auto, (max-width: 1382px) 100vw, 1382px\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">This solution uses an algorithm to constantly change the <em>spending <\/em>of each creative to coincide with its performance. This means that the best performing creative gets more budget, while the least performing one still runs, just with a smaller budget, and anything in between is still being used, based on its performance. There\u2019s a range and the budget is relative.<\/p>\n\n\n\n<div class=\"wp-block-group is-style-note-blue\"><div class=\"wp-block-group__inner-container is-layout-constrained wp-block-group-is-layout-constrained\">\n<p class=\"note-title wp-block-paragraph\"><strong>Data Trends<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The multi-armed bandit algorithms follow trends in the data. If a creative\u2019s performance starts to decrease, the budget follows, and if that creative suddenly starts to improve its performance, the budget for it will gradually increase. If the current champion starts to deteriorate, it will lose its \u201ccrown\u201d so to speak.<\/p>\n<\/div><\/div>\n\n\n\n<p class=\"wp-block-paragraph\">Since there\u2019s no distinction between the exploration period to the exploitation in MAB (the two stages happen simultaneously), MAB produces faster results and is widely proven to be more efficient than A\/B testing.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Though the different MAB models differ, generally speaking, MAB models start to \u201cmove\u201d traffic towards the winning variations as soon as they detect the slightest variation. This means that, unlike an A\/B test, in MAB there\u2019s no waiting for the end of the experiment, which means that MAB algorithms not only work faster, they also reach results at a lower cost, as long as the creatives actually perform differently from one another.<\/p>\n\n\n\n<div class=\"wp-block-group is-style-note-warning\"><div class=\"wp-block-group__inner-container is-layout-constrained wp-block-group-is-layout-constrained\">\n<p class=\"note-title wp-block-paragraph\"><strong>No Backsies &#8211; The Cost of Regret<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><em>Regret<\/em> is calculated by measuring the difference between the conversions reached during a test vs. the <em>potential of conversions<\/em>. The <em>potential of conversions<\/em> is the predicted number of conversions the campaign would\u2019ve reached if the campaign had run with the winning creative. This means that, most likely, regret in an A\/B test is likely to be higher than it is in MAB, since MAB is constantly optimized.<\/p>\n<\/div><\/div>\n\n\n\n<h3 class=\"wp-block-heading\">A\/B Testing vs. MAB<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Trends in data &#8211; MAB results may change over time. One creative may become more popular than it was, initially. It may be impacted by varying factors, but it will be reflected in the data since the test runs continuously. A\/B, on the other hand, only reflects the results of the limited time it ran in.<\/strong><\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Adding Creatives &#8211;<\/strong> since different creatives get different percentages of the budget, it\u2019s easy to add new creatives to the mix and test them alongside the champion. If they \u201cshine\u201d, they gradually get more and more volume.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Earning while learning &#8211;<\/strong> by combining the explore and exploit stages, MAB offers lower costs since the optimization starts <em>while <\/em>data is still being collected. The response is immediate and optimization kicks in faster..<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Automation &#8211;<\/strong> Bandits are a very effective way to automate creative selection optimization with machine learning, especially when considering different user features\u2014since correct A\/B tests are much more complicated in that situation. The MAB algorithm can be activated at very granular levels, and select the best creative for subsections of the targeted audience specifically, while other subsections would have different champions.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">When Should You Use A\/B Testing in Mobile Campaigns?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">When comparing the two methods, it sounds like MAB is the way to go, and a way to save money and get quicker results. Nonetheless, there\u2019s a reason for A\/B testing\u2019s popularity.<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>If your app is new and you\u2019ve yet to settle on a creative line and you want to test out a couple of very distinctive and different creatives, A\/B testing is a way to get a conclusive yes\\no answer (and if the margin is small, maybe you should run another test with different creatives).<\/li>\n\n\n<li>If your creatives are limited, and not endless. If you have 2-3 variations and not 10, it\u2019s a simple way to decide between the versions without wasting your budget on multiple variations.<\/li>\n\n\n<li>The majority of acquisition channels don\u2019t allow users to conduct MAB-based optimizations, only A\/B testing. This option leaves marketers with a choice between A\/B or no testing at all. In this scenario, A\/B testing is better than nothing.<\/li>\n<\/ol>\n\n\n\n<h2 class=\"wp-block-heading is-style-tape-blue\">The Different Types of MAB Algorithms<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Since it\u2019s a complex topic, the aim here is to give a simplified version of the most relevant MAB models for marketing campaigns in general, and creative testing specifically.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Epsilon Greedy<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Epsilon greedy, as the name indicates, is a MAB model that gives the largest part of the budget to the champion. In Epsilon Greedy, the person running the campaign decides on an epsilon (i.e, a percentage of the budget dedicated to the challengers) and the rest of the budget is allocated to the champion.\u00a0<br>Let\u2019s say the epsilon is set at 20%, this means that 80% of the budget of the campaign goes to running the champion, and the 20% is equally divided between the rest of the creatives. This means that epsilon greedy somewhat resembles an A\/B test, but if there are changes in the performance during the campaign, a challenger may become the new champion.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Thompson Sampling<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Thompson sampling, on the other hand, still utilizes the epsilon, but in its case, the challengers get budgeted in relation to their performance. Let\u2019s say, again, that the epsilon is set at 20%. This means that 80% of the budget goes to the champion, while the 20% are divided based on their relative performance (some creatives will get 8% of the budget, some might get just 1%, and as the performance shifts, so will the budget).<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">The Upper Confidence Bound (UCB)<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">UCB may be referred to as the opposite of Thompson Sampling. UCB challenges the least explored creative to test if its low performance is related to the limited budget it\u2019s been getting. When testing creatives, it\u2019s least likely to be used, but when testing, for example, new inventory pockets with higher bids, it might yield interesting results.\u00a0<br>If there are bid floors you\u2019ve been avoiding, since they\u2019re too high, it might be worth exploring, since these pockets of inventory might yield higher LTV users.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Contextual Bandits<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Contextual bandits, as the name implies, are MAB algorithms that group around context. In the case of creative testing, contextual bandits may be grouped around gender, location, or other contextual relations and set a different champion and challengers for each group. This means that instead of a general competition between all of the creatives and the audience, the audience is segmented, and each segment gets its own champion.<\/p>\n\n\n\n<div class=\"wp-block-group is-style-note-blue\"><div class=\"wp-block-group__inner-container is-layout-constrained wp-block-group-is-layout-constrained\">\n<p class=\"note-title wp-block-paragraph\"><strong>Creative Testing in iOS 14<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">With the deprecation of the IDFA in iOS 14, and the reliance on the data in the SDKAdNetwork (or SKAN, in short), creative testing is even more essential than it was before. Since optimizing based on aggregated data is virtually impossible, creatives will take center-stage again. Showing potential users an accurate and appealing ad showcasing the app or game is vital.<br>You can read more about <a href=\"https:\/\/persona.ly\/glossary\/attribution\/user-acquisition-and-retargeting-in-ios-14\/\" rel=\"noopener\" target=\"_blank\">UA and retargeting in iOS 14 <\/a>and review our <a href=\"https:\/\/persona.ly\/glossary\/attribution\/getting-ready-for-ios-14-5-an-extensive-guide-for-mobile-marketers\/\" rel=\"noopener\" target=\"_blank\">extensive iOS 14 readiness guide<\/a> for more details about the upcoming changes and how to prepare for them.<\/p>\n<\/div><\/div>\n","protected":false},"excerpt":{"rendered":"<p>Creative testing is the way marketers compare the performance of different creatives in a campaign in order to evaluate which creatives yield better results. We will cover two methods: A\/B testing, which is a commonly used method and widely known method, and the multi-armed bandit, which is less commonly known and used, usually due to [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":12,"menu_order":3,"comment_status":"closed","ping_status":"closed","template":"","meta":{"gl_intro":"","gl_hero_image":0,"gl_subtitle":"","gl_icon":0,"gl_bullet":0,"gl_terms":"multi-armed bandit, A\/B testing","gl_related":[],"gl_sidebar_items":"[{\"type\":\"image\",\"image\":97,\"frame\":\"polaroid\",\"caption\":\"\",\"url\":\"\"},{\"type\":\"cta\"}]","gl_autolink_off":false,"gl_hidden":false,"footnotes":""},"class_list":["post-96","page","type-page","status-publish","hentry"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.6 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Creative Testing - Multi-Armed Bandit vs. A\/B Testing - The UA Glossary - Persona.ly<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/persona.ly\/glossary\/performance-metrics\/creative-testing-multi-armed-bandit-vs-a-b-testing\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Creative Testing - Multi-Armed Bandit vs. A\/B Testing - The UA Glossary - Persona.ly\" \/>\n<meta property=\"og:description\" content=\"Creative testing is the way marketers compare the performance of different creatives in a campaign in order to evaluate which creatives yield better results. We will cover two methods: A\/B testing, which is a commonly used method and widely known method, and the multi-armed bandit, which is less commonly known and used, usually due to [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/persona.ly\/glossary\/performance-metrics\/creative-testing-multi-armed-bandit-vs-a-b-testing\/\" \/>\n<meta property=\"og:site_name\" content=\"The UA Glossary - Persona.ly\" \/>\n<meta property=\"article:modified_time\" content=\"2022-12-11T12:40:19+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/persona.ly\/glossary\/wp-content\/uploads\/2021\/03\/b_testing_1a.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1382\" \/>\n\t<meta property=\"og:image:height\" content=\"640\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"11 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/persona.ly\\\/glossary\\\/performance-metrics\\\/creative-testing-multi-armed-bandit-vs-a-b-testing\\\/\",\"url\":\"https:\\\/\\\/persona.ly\\\/glossary\\\/performance-metrics\\\/creative-testing-multi-armed-bandit-vs-a-b-testing\\\/\",\"name\":\"Creative Testing - Multi-Armed Bandit vs. A\\\/B Testing - The UA Glossary - Persona.ly\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/persona.ly\\\/glossary\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/persona.ly\\\/glossary\\\/performance-metrics\\\/creative-testing-multi-armed-bandit-vs-a-b-testing\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/persona.ly\\\/glossary\\\/performance-metrics\\\/creative-testing-multi-armed-bandit-vs-a-b-testing\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/persona.ly\\\/glossary\\\/wp-content\\\/uploads\\\/2021\\\/03\\\/b_testing_1a.png\",\"datePublished\":\"2022-03-29T11:50:06+00:00\",\"dateModified\":\"2022-12-11T12:40:19+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/persona.ly\\\/glossary\\\/performance-metrics\\\/creative-testing-multi-armed-bandit-vs-a-b-testing\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/persona.ly\\\/glossary\\\/performance-metrics\\\/creative-testing-multi-armed-bandit-vs-a-b-testing\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/persona.ly\\\/glossary\\\/performance-metrics\\\/creative-testing-multi-armed-bandit-vs-a-b-testing\\\/#primaryimage\",\"url\":\"https:\\\/\\\/persona.ly\\\/glossary\\\/wp-content\\\/uploads\\\/2021\\\/03\\\/b_testing_1a.png\",\"contentUrl\":\"https:\\\/\\\/persona.ly\\\/glossary\\\/wp-content\\\/uploads\\\/2021\\\/03\\\/b_testing_1a.png\",\"width\":1382,\"height\":640,\"caption\":\"A\\\/B testing AKA Split test\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/persona.ly\\\/glossary\\\/performance-metrics\\\/creative-testing-multi-armed-bandit-vs-a-b-testing\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/persona.ly\\\/glossary\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Performance Metrics\",\"item\":\"https:\\\/\\\/persona.ly\\\/glossary\\\/performance-metrics\\\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Creative Testing &#8211; Multi-Armed Bandit vs. A\\\/B Testing\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/persona.ly\\\/glossary\\\/#website\",\"url\":\"https:\\\/\\\/persona.ly\\\/glossary\\\/\",\"name\":\"Mobile UA & Retargeting Glossary | Persona.ly\",\"description\":\"Programmatic, attribution and retargeting, explained simply\",\"publisher\":{\"@id\":\"https:\\\/\\\/persona.ly\\\/glossary\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/persona.ly\\\/glossary\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/persona.ly\\\/glossary\\\/#organization\",\"name\":\"Persona.ly\",\"url\":\"https:\\\/\\\/persona.ly\\\/glossary\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/persona.ly\\\/glossary\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/persona.ly\\\/glossary\\\/wp-content\\\/uploads\\\/2020\\\/02\\\/logo.png\",\"contentUrl\":\"https:\\\/\\\/persona.ly\\\/glossary\\\/wp-content\\\/uploads\\\/2020\\\/02\\\/logo.png\",\"width\":32,\"height\":32,\"caption\":\"Persona.ly\"},\"image\":{\"@id\":\"https:\\\/\\\/persona.ly\\\/glossary\\\/#\\\/schema\\\/logo\\\/image\\\/\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Creative Testing - Multi-Armed Bandit vs. A\/B Testing - The UA Glossary - Persona.ly","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/persona.ly\/glossary\/performance-metrics\/creative-testing-multi-armed-bandit-vs-a-b-testing\/","og_locale":"en_US","og_type":"article","og_title":"Creative Testing - Multi-Armed Bandit vs. A\/B Testing - The UA Glossary - Persona.ly","og_description":"Creative testing is the way marketers compare the performance of different creatives in a campaign in order to evaluate which creatives yield better results. We will cover two methods: A\/B testing, which is a commonly used method and widely known method, and the multi-armed bandit, which is less commonly known and used, usually due to [&hellip;]","og_url":"https:\/\/persona.ly\/glossary\/performance-metrics\/creative-testing-multi-armed-bandit-vs-a-b-testing\/","og_site_name":"The UA Glossary - Persona.ly","article_modified_time":"2022-12-11T12:40:19+00:00","og_image":[{"width":1382,"height":640,"url":"https:\/\/persona.ly\/glossary\/wp-content\/uploads\/2021\/03\/b_testing_1a.png","type":"image\/png"}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"11 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/persona.ly\/glossary\/performance-metrics\/creative-testing-multi-armed-bandit-vs-a-b-testing\/","url":"https:\/\/persona.ly\/glossary\/performance-metrics\/creative-testing-multi-armed-bandit-vs-a-b-testing\/","name":"Creative Testing - Multi-Armed Bandit vs. A\/B Testing - The UA Glossary - Persona.ly","isPartOf":{"@id":"https:\/\/persona.ly\/glossary\/#website"},"primaryImageOfPage":{"@id":"https:\/\/persona.ly\/glossary\/performance-metrics\/creative-testing-multi-armed-bandit-vs-a-b-testing\/#primaryimage"},"image":{"@id":"https:\/\/persona.ly\/glossary\/performance-metrics\/creative-testing-multi-armed-bandit-vs-a-b-testing\/#primaryimage"},"thumbnailUrl":"https:\/\/persona.ly\/glossary\/wp-content\/uploads\/2021\/03\/b_testing_1a.png","datePublished":"2022-03-29T11:50:06+00:00","dateModified":"2022-12-11T12:40:19+00:00","breadcrumb":{"@id":"https:\/\/persona.ly\/glossary\/performance-metrics\/creative-testing-multi-armed-bandit-vs-a-b-testing\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/persona.ly\/glossary\/performance-metrics\/creative-testing-multi-armed-bandit-vs-a-b-testing\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/persona.ly\/glossary\/performance-metrics\/creative-testing-multi-armed-bandit-vs-a-b-testing\/#primaryimage","url":"https:\/\/persona.ly\/glossary\/wp-content\/uploads\/2021\/03\/b_testing_1a.png","contentUrl":"https:\/\/persona.ly\/glossary\/wp-content\/uploads\/2021\/03\/b_testing_1a.png","width":1382,"height":640,"caption":"A\/B testing AKA Split test"},{"@type":"BreadcrumbList","@id":"https:\/\/persona.ly\/glossary\/performance-metrics\/creative-testing-multi-armed-bandit-vs-a-b-testing\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/persona.ly\/glossary\/"},{"@type":"ListItem","position":2,"name":"Performance Metrics","item":"https:\/\/persona.ly\/glossary\/performance-metrics\/"},{"@type":"ListItem","position":3,"name":"Creative Testing &#8211; Multi-Armed Bandit vs. A\/B Testing"}]},{"@type":"WebSite","@id":"https:\/\/persona.ly\/glossary\/#website","url":"https:\/\/persona.ly\/glossary\/","name":"Mobile UA & Retargeting Glossary | Persona.ly","description":"Programmatic, attribution and retargeting, explained simply","publisher":{"@id":"https:\/\/persona.ly\/glossary\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/persona.ly\/glossary\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/persona.ly\/glossary\/#organization","name":"Persona.ly","url":"https:\/\/persona.ly\/glossary\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/persona.ly\/glossary\/#\/schema\/logo\/image\/","url":"https:\/\/persona.ly\/glossary\/wp-content\/uploads\/2020\/02\/logo.png","contentUrl":"https:\/\/persona.ly\/glossary\/wp-content\/uploads\/2020\/02\/logo.png","width":32,"height":32,"caption":"Persona.ly"},"image":{"@id":"https:\/\/persona.ly\/glossary\/#\/schema\/logo\/image\/"}}]}},"_links":{"self":[{"href":"https:\/\/persona.ly\/glossary\/wp-json\/wp\/v2\/pages\/96","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/persona.ly\/glossary\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/persona.ly\/glossary\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/persona.ly\/glossary\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/persona.ly\/glossary\/wp-json\/wp\/v2\/comments?post=96"}],"version-history":[{"count":0,"href":"https:\/\/persona.ly\/glossary\/wp-json\/wp\/v2\/pages\/96\/revisions"}],"up":[{"embeddable":true,"href":"https:\/\/persona.ly\/glossary\/wp-json\/wp\/v2\/pages\/12"}],"wp:attachment":[{"href":"https:\/\/persona.ly\/glossary\/wp-json\/wp\/v2\/media?parent=96"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}