How to Measure the True Profitability of Free Shipping Promotions

Free shipping isn’t free. It’s paid for with margin.

If your average order is $75 and you’re covering $12 in shipping, that’s 16% of revenue gone before you even consider your cost of goods. If your gross margin is 35%, you’re working with $26.25 in profit. Subtract the $12 shipping cost, and you’re left with $14.25—or a 19% effective margin.

That shift has downstream consequences.

With a 35% gross margin, your breakeven ROAS is 2.86x. That is, your ad campaigns are profitable so long as every $1 you invest generates at least $2.86 in return. 

But once free shipping eats into that margin, your breakeven ROAS jumps to 5.26x — a much more difficult target to sustain in most verticals. The higher your ROAS target climbs, the harder it becomes to scale paid media efficiently. Your ability to spend declines, traffic slows, and customer acquisition suffers.

And when your marketing team learns that their ROAS targets have now doubled, you can assume that your advertising budgets will have to be cut in half (or more). Nobody wants this. 

So this isn’t just about protecting margin. It’s about protecting growth. Promotions like free shipping might help with conversion rates, but they can quietly undermine your ability to invest in traffic—and in doing so, put pressure on both top-line revenue and bottom-line profitability.

These are the kinds of tradeoffs that rarely get questioned. They're easy to justify as “standard offers” or “costs of doing business.” But over time, those assumptions add up. And eventually, you’ve got a marketing program optimized for short-term performance, while long-term profitability and acquisition efficiency both erode.

Earlier this week, Eric Seufert shared a GitHub repo from a member of Walmart’s marketing team that breaks this cycle. It outlines a clean, scalable framework for measuring the incremental impact of free shipping on customer revenue. Walmart ran this at massive scale, but the underlying model is simple enough for any brand to adopt—even if you’re doing less than $1M a month in revenue.

Here’s the link: https://github.com/banani29/DID_Implementation

What follows is a breakdown of how to set up the same kind of experiment using basic tools: Shopify, a spreadsheet, and a Python script.

What this model is designed to measure

This analysis doesn’t help prove whether free shipping helps boost conversion rate. All things equal, any discount should help boost conversion rate.

Instead, we’re attempting to measure incrementality—whether the promotion actually changed customer behavior, and whether that change created more revenue than it cost you to offer.

That’s the only thing that matters.

If free shipping results in a statistically significant lift in revenue, and that lift covers your cost, then the promo makes sense. If not, it’s a margin loss that looks like a growth tactic.

Who this is for

This specific test is ideal for any brand generating at least a few thousand orders over a 2–3 month window. That generally means monthly sales in the $250K–1MM range, depending on AOV. But even smaller brands can run the same test—they just need to extend the timeline until they hit the right sample size.

If you're running on Shopify and sending basic lifecycle emails, you already have everything you need to get started.

How the test works

You’ll run a 16-week randomized control trial:

• Weeks 1–8: Baseline period (no promotions)
  
• Weeks 9–16: Treatment period (free shipping for half your audience)
  

You’ll split your customers into two groups:

• Treatment: Receives free shipping promo
  
• Control: Receives nothing
  

You’ll then compare revenue over time across both groups. The model uses a difference-in-differences (DiD) regression to isolate the causal impact of the promotion, while controlling for natural revenue trends and customer-level differences.

Step-by-step breakdown

1. Build your customer pool
Start with recent customers—anyone who made a purchase in the last 60–90 days. Exclude customers who’ve previously received free shipping or heavy discount codes. Keep it to domestic customers for fulfillment consistency.

2. Randomly assign treatment vs. control
Split the list 50/50. You can do this in Excel. Tag each group in Shopify using customer tags like free_shipping_test_treatment and free_shipping_test_control.

3. Run your baseline period (Weeks 1–8)
No promotions. Just let customers shop as normal. This gives you a clean view of natural revenue behavior across both groups before any intervention.

4. Launch the promotion (Weeks 9–16)
Send an email or campaign to the treatment group offering free shipping. The control group receives nothing. Keep the rest of the marketing cadence identical across both groups.

5. Track and tag every order
Make sure every order includes:

• Customer group (treatment vs. control)
  
• Order date
  
• Order value
  
• Shipping cost
  
• Discount code used (if any)
  

You can automate this using Shopify Flow or tag orders manually during exports.

6. Run the analysis
Once the test is complete, export all order and customer data. Use the GitHub repo to run a difference-in-differences regression. The model will show you:

• Whether the treatment group spent more
  
• By how much
  
• Whether that lift was statistically significant
  
• And whether it covered the cost of shipping
  

What the results will tell you

If the promo resulted in a lift greater than the cost of shipping, then you’re creating incremental value. If not, you’re giving away margin without changing behavior.

The test can also reveal which segments responded best. You might find that free shipping worked well for first-time buyers but had no impact on repeat customers. Or that high-AOV customers didn’t need an incentive at all.

This is how smarter promotional strategies get built—by focusing on actual outcomes, not just assumed best practices.

Final thought

Marketing teams spend an enormous amount of time and budget optimizing media, tweaking creative, and chasing platform ROAS improvements. Meanwhile, no one questions whether their baseline offers—discounts, shipping perks, welcome codes—are actually helping the business.

Contribution margin is the most fragile part of your P&L. If you're giving it away, you better know what you're getting in return.

This model gives you a way to find out.

Start here: https://github.com/banani29/DID_Implementation

And if you’re already offering free shipping (or planning to), run the test. The data will tell you everything you need to know.