Testing & Analysis INSIGHTS

INSIGHT #31 - Measuring Across Time

February 17, 2013


Our last Insight was about the philosophy of triggered lifecycle programs.

And now I've promised you a "How To"
session on measuring the incrementality of lifecycle marketing.

The philosophy in a nutshell:
Lifecycle triggered campaigns are about influencing lifetime value.

So even though triggered lifecycle marketing campaigns can get upwards of double the immediate direct response for open, click and conversion vs. a blast marketing campaign....

...don't stop at simple campaign response tracking!

Because that will cut short the real value of a triggered lifecycle program.


Triggered lifecycle marketing programs generate LIFETIME value in a way that blast marketing campaigns do not.


So how do you measure the incremental lifetime impact of a triggered program? Well, you measure it over the lifetime of the customer. Oops, that's too long! But the point is, you must measure it over a long period of time, maybe 6 months or a year.


WHY so long? Because the value of a triggered lifecycle program shows itself more and more over time.  You'll start with some set of lifecycle triggers, but as you measure the incremental lifetime value, you'll continue to develop more and more of these triggers along with corresponding targeting, creative and offers.


Here are 5 basic "how-to" steps:

  1. Set up an initial lifecycle segmentation and contact strategy.
    That's for another day. But for now...
    Just define some lifecycle segments.
    Base it on purchase recency, first purchase, birthday, category of purchase, channel of purchase... Pick some lifecycle segments.
    Whatever your segments, customer lifecycle is NOT about what your brand is planning.
    Customer lifecycle is about what the customer did or did not do
    as of a certain poitn in time.

  2. Split your segments into Test and Control Groups.
    How big?
    Start with 50/50 and you won’t go wrong.
    Don’t worry, some of these will come out of the Control and into the mail stream shortly.
    You can always reduce the Control Group after your analysis shows statistical significance; but you can never increase it (because you will have already tainted the comparison with that first lifecycle mailing).
  3. Suppress the Control Groups from any lifecycle marketing.
    How long?

    For a period of X months, your lifecycle Control Groups will receive whatever they would have received, but absolutely NO lifecycle campaigns. The Control Groups continue to be treated as normal for everything else, just no lifecycle campaigns.
  4. Implement lifecycle marketing for the Test Groups.
    What's that look like?
    Your lifecycle Test Groups will receive every lifecycle campaign they qualify for.
    For emails, the lifecycle campaign should pre-empt whatever else they would have received that day.
    For direct mail, the lifecycle campaign can be a version of what they would have received or it can be in place of what they would have received.
  5. Measure the growing net spending disparity between the Test and Control Groups ACROSS TIME.
    At the end of no less than 6 months (and at the end of every month until then), you’ll analyze the multi-channel net spending of the Control Groups and compare it to the multi-channel spending of the Test Groups.

This long-term analysis has nothing to do with campaign response or direct response. You're not filtering on promo codes or who got any specific campaigns - you're just comparing the Test and Control Groups with How frequently did they buy and much did they spend? Over a long period of time.

And ... over time ... you will see a growing disparity in spending between the Test and Control Groups.

The longer you hold out the Control, the greater will be the disparity. And THAT growing disparity between Test and Control is the incremental value of running a lifecycle trigger program.


At some point, you'll find sufficient statistical significance to go on and move some portion of your Control Groups into the lifecycle mail stream, reducing the size of your Control Groups.


And eventually, you'll decide that enough is enough, and you'll end the analysis and move all the Control into the lifecycle mail stream.


You'll likely only ever do this analysis once. Use the results as a benchmark for the scale of annual incremental value from  triggered lifecycle programs. You'll gain insight into the ongoing impact of this type marketing vs one-off blast campaigns. You'll muster the necessary buy-in to create more and more triggers, campaigns, and offers. And forever more, your marketing will grow more and more relevant and it will drive more and more loyalty and retention.


Please enter the code
* Required fields
There are no entries yet.

INSIGHT#30 - Tracking Across Time

Feb 3, 2013


If you’re only tracking ROI during the campaign response period, you’re missing it.


Especially for any triggered, lifecycle-driven contact strategy.



Think about the point of lifecycle marketing. “Lifecycle” implies two things:

1)      Something today

2)      That affects the future


Think about raising children. If it were all about how they behave today, we’d just bribe or cajole. Instead, we discipline to raise our children into law-abiding adults.


Think about dieting. If it were all about today, you’d throw caution to the wind and eat <fill in your blank>. Instead, we diet in February to fit into swimsuits this summer.


Think about exercise. Don’t know about you, but I hurt after exercise. For me, it’s all about a long and healthy future.


Now think about marketing. Is it all about making today’s plan?

Well let me have a hey ya'll and yee-haw!


But wait! That really is NOT what triggered, lifecycle campaigns are about.

“Lifecycle” implies two things:

1)      Something today >>> bellwether event coupled with a bellwether contact

2)      That affects the future >>> long-term impact


I love Wikipedia’s definition of bellwether: any entity in a given arena that serves to create or influence trends or to presage future happenings.


Lifecycle triggered campaigns are about influencing trends, future happenings.


So If you’re only tracking ROI during the campaign response period, you’re missing it.


Next up: how to track impact across time.


Please enter the code
* Required fields
There are no entries yet.

INSIGHT #29 - Needing Time

Jan 13, 2013

I haven’t sent you one of these little “Insights” since Dec 3. Between the holiday marketing frenzy and a couple of big projects, I was busy.


Also, my mother passed away, and I just needed time.


Needing time. Let’s divert from the mini-series you’ve probably forgotten about anyway. And let’s talk about NEEDING TIME.


Here’s the thing: If you’re only tracking ROI for your marketing campaigns during the campaign response period or through a promotion code, you’re missing it.


Everyone needs time. Your customers need time. And they don’t always respond on YOUR timeline.


If near-term ROI is all you’re measuring, you’re selling your marketing impact way short.


Your customers remember. Not every marketing campaign. But the direct mail piece you sent last summer may be the very reason they think to buy from you this January. Your continued, relevant messaging over time might be what keeps them engaged.


It wasn’t just what my mother said to me last month, but the impression she made my whole life, that impacts my thoughts about her right now. And it’s not that different with customers and brands.


Are you measuring the long-term impact of your marketing?


Next up, measuring long-term ROI. Meanwhile, check out Insight Outcomes #12 below; it's about measuring the long-term impact of a frequency test - over time.


Please enter the code
* Required fields
There are no entries yet.


INSIGHT #12 - Frequency Test

May 28, 2012


Last week, we looked at what REALLY matters in a frequency test – CUSTOMER metrics.

But this week I promised to look at CAMPAIGN metrics within the context of an email frequency test.


What’s the easiest way to set up the segments, the frequency cap codes and the reporting template?

Here goes:

ONE. Set up the test and control groups for various customer segments.

  • You'll want to optimize frequency for each segment so do some sort of segmentation, even if it’s just Active vs. Inactive
  • Leave out anyone who already has a frequency cap; include EVERYONE who doesn't


TWO. During the frequency test, add NEW SUBSCRIBERS to similar but separate frequency cap codes.

  • The biggest gap between the test and control will be amongst the new subscribers – so don’t leave them out of the test
  • Hint: How you onboard your subscribers, including frequency, will make a big difference in their future performance
  • But eventually, you’re likely to prove that it pays to start low, but ramp up to a higher frequency

THREE. Assign frequency codes that are distinct from any of your other frequency codes

  • Every existing subscriber will have a frequency code assigned
  • For example, let’s say you typically send 7 emails per week, if you’re testing 2, 4 and 6 times per week, depending on the size of your segments, your codes might look something like this:





Size of group







2 times a week

4 times a week

6 times a week

7 times a week

4% of total Active










2 times a week

4 times a week

6 times a week

7 times a week

4% of total Inactive










2 times a week

4 times a week

6 times a week

7 times a week

All new subs during week 1

All new subs during week 2

All new subs during week 3

All new subs during week 4ff




1 time a week

These are not in the test;

they were already capped


FOUR. Set up your email reporting tool to track these distinct frequency cap codes separately.

  • Since you assigned EVERYONE a frequency code, everyone will show up in the results
  • Campaign results are additive over time. Your email tool never attributes any response to more than one campaign, so you can use simple addition to sum up the volume metrics across time
  • If you’re multi-channel, you MUST take offline orders into account – if not, your frequency test will lead you astray, but if systems make that difficult you might plan to do the multi-channel analysis at the end of multi-month test, while constantly keeping your eye on the immediately available online results


FIVE. Set up a Frequency Test reporting template.

The Reporting Template needs to do the following:

  • Let you easily paste the results of each frequency cap group for each campaign
  • Sum up the volume metrics (total sends, opens, clicks, orders, revenues)
    across time for each frequency cap group
  • Calculate efficiency metrics (open rate, click rate, conversion, rev/sent)
    using the summed volume metrics across time
  • Display apples-to-apples Total Volume Metrics as if each of the test & control groups were of equal size
  • Display apples-to-apples Net Efficiency Metrics based on the ORIGINAL quantity in each group (to account for the impact of frequency on unsubs)

In the end you want to show which frequency drives the most profits from each customer segment over time, so your test must run its course over several months time; the longer the better.


Next week, let’s move on to something less technical.

Something fun from the “Profitable Programs” bag!



Please enter the code
* Required fields
There are no entries yet.

INSIGHT #11 - Campaigns vs Customers

May 21, 2012


Last week I promised we’d look at a specific example of testing, analyzing and optimizing various volume and efficiency scorecards.


One of my favorite tests is email frequency.


The VOLUME scores we’re usually after include:

  • Revenues, Orders, or Profits (money)
  • Clicks (site traffic)
  • Opens (interaction)


And the EFFICIENCY scores we’re usually watching are:

  • Revenues per Sent
  • Click Rate
  • Open Rate
  • Unsub Rate
  • Bounce Rate
  • Average Order Value


Costs are so low with email, but don’t forget hidden “costs” – like list burn-out, brand erosion, or leaving revenues on the table. Even if it were free, you’d still want to test for the frequency that drives the most profits over time.


With an email frequency test, none of the above measures are most important. Because these are all about how your CAMPAIGNS perform. When it comes to email frequency, what you’re really after is optimizing how your CUSTOMERS perform over time.


What’s the difference?


First, consider unsubs. Happy with your low unsub rate? Consider this: the more frequently you email, the lower will be your “unsub rate.” But those unsubs are additive, so a 0.1% unsub rate if you’re mailing daily is like a weekly mailer’s 0.7% unsub rate. Ask an Email Services Provider what a decent unsub rate is, and 9 times out of 10 they’ll give you a per-campaign rate and they won’t bother to put that answer within the context of how often you’re emailing. If frequent emails are driving your customers to unsubscribe, your leaky bucket may be getting the better of you, and your campaign metrics will not give you visibility to this.


Now, consider customer performance. If you’re mailing daily, you probably aren’t looking for your subscribers to open every email. You’d be thrilled if every subscriber just opened and responded to 1 email every month! Right?


Email frequency optimization is all about CUSTOMER performance, not CAMPAIGN performance. You need a way to track what percent of your list has ENGAGED (opened, clicked, purchased, subscribed, unsubscribed) in the last

  • 1 month
  • 2-3 months
  • 4-12 months
  • More than 12 months ago
  • Never… they never opened, never clicked, or never purchased

See my blog, Insight #2 – List KPI’s, for a graphic view of tracking Customer Engagement.



Customer performance metrics – not campaign performance metrics – are the most important metrics for optimizing frequency.




When you put these customer performance metrics into context with campaign performance, you’ll have yourself the makings for frequency optimization.


And if you do all of that within segments of your list, vs just for the whole list, you’ll really be on the road to frequency optimization.


Next week, we’ll look at tracking campaign metrics within the context of a frequency test. This week, I just had to emphasize the more important factor – Customer Performance.

Looking for a specific example? Ask for it here!


There are no entries yet.
Please enter the code
* Required fields

Insight #10 – Which Is It?

May 13, 2012


Retail is not like sports where a single score determines the winner.


Depending on your role, you're accountable to multiple "scorecards." And some scores may be in direct conflict with others!


So WHICH IS IT for you? Volume or Efficiency (ROI)? To optimize against both, establish a minimum acceptable ROI and go for as much volume as that permits.


Even then, the list of ROI metrics and Volume metrics goes on and on. So you must prioritize the various scorecards that are inherently present in any test.


For example, here are some volume metrics:

  • Top line revenues
  • Bottom line profits
  • Traffic
  • Awareness
  • Customer List Growth
  • Customer Satisfaction
  • Customer Retention
  • Customer Acquisition
  • Customer Value
  • And on and on...


And no fewer Efficiency scores:

  • Revenues or Profits per Campaign
  • Revenues or Profits per Send
  • Revenues or Profits per Customer
  • Revenues or Profits per Order
  • Visits per Campaign
  • Visits per Send
  • Visits per Open
  • Visits per Customer
  • Acquisitions per Campaign
  • Acquisitions per Send
  • Clicks per Open
  • INCREMENTAL (all of the above)
  • Cost per Customer
  • Cost per Acquisition
  • Cost per Order
  • Cost per Revenues or Profits


It comes down to this:


What are you aiming to increase? And how much decrease in efficiency is acceptable in exchange for that increase in volume?


By the way, these scores exist, whether you track them or not!


Next week, we'll look at a specific example of testing, analyzing and optimizing various volume and efficiency scorecards.

Your thoughts?


There are no entries yet.
Please enter the code
* Required fields

Blog #7 – Scraping the Bottom

April 16, 2012


In Blog #4 (see entry below), we talked about your List Growth and List Engagement reports fueling your segmentation and analysis strategy. Using right-brain pre-analysis for:

  • Seeking out any little program nuances that could improve the KPI metrics for your biggest segment

  • Turning a small, high-performing segment into a bigger piece of the pie to achieve critical volume

Today, let’s consider your LOWEST-PERFORMING SEGMENT. They appear disengaged. Non-recent interactions, non-recent opens, non-recent clicks, non-recent purchases. Very non-recent. As in, they’ve been around for more than a year and you haven’t detected a pulse in all that time.

Don’t just cut them off. Especially if you’re a multi-channel retailer, you can’t really be sure if some of them are shopping but your systems are just not capturing and matching back their purchases. Test!

  • Test with low-cost, interactive marketing, like email.
    • Low-cost, because ROI will be tight with this group
    • Interactive, because your primary objective is to just stir up a heartbeat

  • Test lowering the frequency of email. Run something like an 80/20 split for an extended period (as in, months, quarters). Measure their overall spending and your ROI over that period.

  • This is especially smart if it’s a big segment and you email frequently. Two reasons:
    • If it’s a big enough segment, it will actually impact your budget
    • If you stop stuffing their inbox, and send them more pointed, personalized subject lines, they might actually take notice and reactivate
  • Test personalized subject lines and offers. They’re not responding to your normal marketing lingo, so try something different. Test content using a 50/50 split for both the 80/20 groups.

  • Test using a survey – maybe a few will let you know a problem you can fix so you can bring the many back.

  • Test excluding them from direct mail except for that one piece that will be most compelling for specifically targeted segments of this group, based on such factors as:
    • Their last category, price-point, or time-of-year shopped
    • The last offer-type they responded to


Don’t just cut them off. Scrape them off the floor. If you’re spending anything on prospecting, THIS is the top priority group you should be prospecting! They’re going to give you better marketing ROI than going after prospects. Guaranteed!


Join the conversation. What can you share about what you've learned through testing your low-performing groups?



There are no entries yet.
Please enter the code
* Required fields

Blog #4 - Right-Brain Pre-Analysis

March 25, 2012


With your List Growth and List Engagement KPIs in place (Blog #3), now it's time for some right-brain pre-analysis.


Recall, we started with any readily accessible segmentation scheme for our List Growth and List Engagement KPIs – gender, source of collection, region, channel, etc. If we only have it for the TOTAL list – or if we don’t have weekly or monthly trend over time – then we’re all dressed up with nowhere to go.


First step > identify the biggest segment of your list. Is the segment growing or shrinking? Is engagement increasing or decreasing? How do metrics for this segment compare to metrics for the other segments? Now get your right brain in gear to identify key factors influencing these trends. Is it merchandise, marketing, promotion, price-point, distinctives, digital interactions… you get it. Look for little things that are broken and fix them. Look for positive dynamics and incent them more.


Any little thing you do to improve the biggest segment will pay off big. 


Here’s a short list of possible factors that are likely in your control if you’re reading this:

  • You’ve hidden your sign-up form
  • You’re sacrificing sign-up volume to a form that is too complicated
  • Your sign-up form or data feed has a glitch
  • Your competitors put new interactive features in place (tablet, mobile, ratings, increased search & affiliate, improved SEO) - features you haven't gotten around to yet


Here’s a short list of factors that are likely beyond your control but any Marketing workaround can bide time:

  • New competitors
  • Merchandise issues (inventory, fit, pricing)
  • Fulfillment issues (high shipping fees, slow delivery, poor store experience)
  • Changing society or demographics


Next step > Identify a smaller segment of your list that is already doing well. It’s growing and engagement metrics are strong and improving. Right brain: how can you make this segment a bigger piece of your list or customer base? If you can turn this segment into critical volume, that’s a win all the way around.


Some ways to grow a small segment:

  • Targeted banner ads, keywords, promotions, sweepstakes, and co-reg partnerships on other websites
  • Target some Refer-a-Friend campaigns and incentives to this segment
  • Drive them to specially designed sign-up forms; incent the new sign-ups with a short-life promo code; get them marketable, and thank them with a drive to purchase conversion
  • Test some in-store changes targeted at this segment


These explorations will start to fuel your segmentation and analysis strategy.


How have you been successful at identifying and steering opportunities in your segments?

Join the conversation!



There are no entries yet.
Please enter the code
* Required fields

Learn more about Insight Outcomes LLC:


About Dianne



Newsletter Signup