On 2026-11-18, Adswerve announced an expansion into predictive marketing consulting, including services for purchase propensity and predicted lifetime value. Company announcement.
This is a consulting announcement, not independently established performance. The proposed evaluation below applies regardless of which vendor or internal team builds the model.
Name the decision the score will change
A propensity score estimates how likely someone is to take an action. It does not tell you whether your message or discount changes that probability. Start with a single choice: for consenting repeat customers, should the business send a reminder, send an incentive, or do neither? Define the purchase window and who is eligible before examining scores. Agree on the maximum contact frequency and exclusions such as unresolved service complaints. A model should not override someone's communication preferences or turn an otherwise unnecessary offer into the default next action.
Rebuild the information available when the decision was made
For each historical customer, reconstruct the information that existed on the decision date. A return posted later, a conversion label copied into an activity field, or a final order status from the end of the month makes a backtest look better than it will be live. Create a dated input table and have a second person trace a small sample to the original records. Use older periods for training and a later, fully observed period for evaluation. When some targets have not had time to occur, do not code them as permanent negatives just to finish the report.

Compare the score with a rule someone can explain
A plain baseline could rank customers by time since their most recent order. Give both approaches the same eligible population and the same number of contacts. Compare who actually purchased in the evaluation window, and show whether observed purchase rates resemble the probabilities assigned to each band. For predicted customer value, set a definite horizon and say whether it means revenue or contribution after discounts, returns and variable costs. A complicated model that does not outperform the baseline where the decision changes is a weak reason to replace a working process.
Test intervention and prediction separately
Randomly keep a no-contact comparison group inside the same score bands as the customers receiving outreach. Make the unit of assignment stable and exclude simultaneous experiments that use the same population, or document the overlap. Compare outcomes by the group assigned, not merely the group that opened the message. Measure contribution per eligible customer and separately monitor opt-outs, complaints and return rates. Set the observation window, the minimum useful effect and the decision criteria before activation. A highly accurate score can coexist with an offer that costs more than it adds.
A hypothetical example: more orders, smaller margin
Imagine 500 eligible customers assigned to an offer and 500 to no contact. In a fixed window, the offer group places 70 orders and the comparison group 60. Each order contributes 30 units before incentives, and each offer-group order uses a discount of 8 units. The offer group contributes 1540 units after discounts, versus 1800 in the comparison, before any delivery cost for messages. The group had more orders but less contribution. These invented numbers illustrate the trade-off, not Adswerve's performance or a causal conclusion from a single sample. Check uncertainty and whether the accounting assumptions hold before acting.
Walk one decision all the way through before automating it
Treat the first deployment as a desk exercise. Take several historical cases with different starting conditions, including one with no recent purchase, one with a recent return and one with a contact preference that forbids the proposed channel. Ask a marketing operator to identify which records would be available on the decision date, what score band the customer would receive, and what the written policy would permit. Have an independent reviewer reproduce the steps. If those two people reach different decisions from the same input, fix the ambiguous policy before debating whether the model needs more features or a finer threshold.
Document who can see the prediction and who can act on it. A reporting analyst should be able to audit a recommendation without sending a message. A campaign manager may need authority to send an approved message to the assigned audience but not to change the scope of a test while it is running. A small, timestamped approval record is more useful than a blanket statement that everyone agreed. In particular, note when the score was produced and when someone actually authorized the action; a score can be valid on Monday and irrelevant by Friday if a customer has since bought, returned an item or unsubscribed.
Give marketing and customer service an agreed fallback for ordinary exceptions. An incomplete order record is reason to postpone an offer, not permission to infer that a purchase failed. If someone opts out after group assignment but before the send, honor the updated preference. Keep the original assignment for the analysis, mark the send as withheld for that reason, and report the number of withheld contacts rather than reassigning them to the control group. This protects the experimental comparison and also makes the operating policy visible. If these exceptions are common, fix the timing and audience data before increasing contact volume.
Keep score monitoring separate from campaign monitoring. On the model side, inspect whether the coverage and distribution of scores shift and whether predicted rates still resemble observed rates in mature data. On the campaign side, watch the actual sends, contact limits, voucher redemptions, returns and complaints. A broken event import can damage the model while messages keep going out; an overly aggressive offer can upset customers even when the model is properly calibrated. Different owners may be needed for those two failure modes. Define what each owner can stop, and do not resume automation until the source of the issue is understood.
Do not make the score a permanent label. If a retailer ranks customers weekly, it can retain the dated score needed to understand a prior decision, while showing operators only the current recommendation and its explanation. Avoid duplicating the underlying customer records in every experiment export. Separate the limited data reviewers need from contact information used by the sending system, apply access controls, and set a retention period suitable for reviewing the experiment. This also makes it easier to reconstruct what happened at a specific time without granting broad access to customer information or confusing later data with the original inputs.
Explain what would make the conclusion change
Start the result discussion by looking for differences in what could be observed. One group may have used vouchers that were easy to reconcile, while some purchases in the other group came through stores that were slower to report. In that case the treatment could change the chance that a transaction appears in the dataset, without changing the chance the transaction occurred. Use the same outcome sources for both groups wherever possible and explain what is missing. Show when the outcome table was last refreshed and how a late order or return will be reflected. Review data completeness before discussing small differences in purchase rates.
When the overall result is ambiguous, don't hunt through dozens of segments for one that looks positive. Say which segments were specified in advance, how many people and outcomes they contain, and which findings are exploratory. An attractive number from a small subgroup can suggest a next targeted test but should not automatically become a rule for all members of that subgroup. Ask whether the pattern is consistent across periods and channels, and whether it fits a plausible customer need. Confirm any new recommendation with an independent comparison before exposing a larger group or increasing the size of the incentive.
Give finance two transparent scenarios for the same result. The first should use costs confirmed in the accounting records, including redeemed discounts, variable fulfillment and returns already booked. The second can vary uncertain items such as late refunds and the value of staff time, using documented assumptions. Do not average these worlds together and call the result a precise profit number. The point is to see whether an apparently promising campaign depends on one favorable assumption. If a normal range of refund costs would reverse the sign, keep the offer limited or wait for additional observation rather than announcing success.
Translate the result into a decision table with three possible outcomes: keep the procedure, fix the test, or stop the intervention. Keep it only when the test is interpretable, the gain is large enough to justify the intervention and customer guardrails are intact. Fix the test when a change in eligibility, incomplete observation or overlapping campaigns makes the comparison unreliable. Stop when value is negative, when downside reaches the agreed threshold, or when the intervention violates the conditions for proper contact. These are different decisions; an inconclusive test is not proof that the model failed, while a harmful campaign is not a reason to train a more elaborate predictor and repeat it unchanged.
The handover should record which part of the work can be reused. Perhaps the team now has a trustworthy purchase label and a reviewable baseline, even though it will not keep the discount. Or the experiment may show that the operations team can implement randomized holdouts, while the model itself adds nothing. Either is useful. A repeatable record includes the inputs, group definitions, actual numbers, limitations, and a named date for the next decision. That makes future pilots cheaper to review and harder to overstate, regardless of who provided the marketing technology.
Put an expiry date on the decision
When a pilot ends, document the input version, who was eligible, both comparison groups, and which costs are missing. Keep a human owner for approval, stopping and follow-up. Set a date to review whether scores have drifted and whether the policy still adds value after incentives. If the model improves ranking but the intervention adds nothing, keep the learning without scaling the offer. If a simple rule produces the same business outcome, prefer the one the team can audit, sustain and explain.
When the next review comes due, start with the original promise, not the newest dashboard. Ask the owner to reconstruct which customer behavior the model was meant to predict, which action was tested and which billable or operational items were included in the cost. If the business changed its definition of an active customer in the meantime, keep the earlier result under its original definition and build a new comparison under the new one. Do not combine them into a single trend without explaining the break. An old experiment can remain useful as evidence of how a specific practice worked under known conditions, without becoming an automatic instruction for a different campaign with different people, costs and timing. Record why you kept or changed the decision so the next team sees the limits of the evidence, not just the report headline.

Comments
No published comments yet.