To know whether an automation pays off, measure the process before and after, including human time, errors, exceptions, supervision and recurring costs. The number of runs is not the same as savings. A useful evaluation starts from comparable data and tells apart observed benefits from hypotheses that have not yet been verified.
What should you measure before automating?
Record the real execution time, the frequency, the failures and the correction work. Define what a unit of work includes: resolving a ticket is not the same as sending an automatic reply. If samples are small, avoid generalising. The baseline must reflect normal days and less usual situations, not just the best scenario.
Which indicators show a genuine benefit?
Besides time per task, consider the percentage of correct operations, rework, open incidents, waiting times and customer or team satisfaction when it can be measured. A flow that reduces minutes but doubles errors is not necessarily an improvement. Select a few indicators linked to the specific goal before starting the pilot.
How do you calculate ROI without manipulating the data?
First define verifiable monetisable benefits and the total cost over an equivalent period. An indicative formula is (net attributable benefit minus total cost) divided by total cost, multiplied by one hundred. The challenge is attribution: telling apart what changed because of the automation from variations in demand, staffing or processes. If you cannot attribute it, report an operational improvement and avoid claiming ROI.
How do you compare before and after correctly?
Try to compare similar volume and complexity. If there was an exceptional campaign during the pilot or commercial policies changed, record the circumstance. Also observe the learning period: the first few days may need more intervention. Do not remove failed cases from the calculation just because they hurt the result.
What do you do if the results are mixed?
Separate resolved tasks from exceptions and look for bottlenecks. The automation may speed up classification but slow down human review for lack of context. Adjust the process and measure again. If there is no demonstrable benefit after reasonable iterations, document the decision to stop or reduce the scope.
How do you measure the result without attributing savings that do not exist?
The starting point is a baseline: define the task, observe several runs and record effective time, waiting, errors and corrections. A process may take two calendar days but need only a few minutes of human work. If you mix both measures, the announced saving will be misleading. Automation can reduce cycle time without freeing as many staff hours.
Calculate separately volume, minutes of work avoided and new supervision tasks. An indicative formula is: net hours = (manual minutes per case − supervision minutes per case) × valid cases / 60 − maintenance hours in the period. If any variable has not been measured, use an explicitly hypothetical scenario, never a result attributed to a real company.
The economic cost requires stating assumptions: hourly value, licences, maintenance, incidents and observed period. Do not automatically equate freed hours with payroll savings. It may be capacity available to respond faster, improve quality or take on more work without adding resources; that is a different effect from reducing an actual expense.
Control side effects. If speed increases but so does the correction rate, the net result may get worse. Compare requests of the same type and review seasonality, campaigns and team changes. When volume is low or the system has just been launched, present provisional results and set a date to measure again.
How to present results to management without overstating them
A results sheet should distinguish direct measurements, estimates and qualitative observations. In the first column, record how many cases were processed and how much human work they required. In another, the errors and corrections. In a third, the recurring costs observed. That separation avoids presenting a perception of improvement as a demonstrated financial saving.
When you compare periods, state what changed besides the automation. There may be more orders, a change in staff, new campaigns or different customers. If you cannot isolate the effects, describe a temporal association and not a guaranteed causality. Collect examples of exceptions to understand why certain tasks did not speed up, even if the averages look positive.
The final presentation should include the conditions under which the project would stop being worthwhile: too much supervision, usage cost that rises, repeated errors or tool changes. Defining that threshold is as useful as describing improvements. It lets you decide whether to expand, redesign or retire an automation without being trapped by a sunk investment.
How do you avoid a misleading measurement of savings?
Time freed does not automatically equal benefit. If a task takes less time but needs more reviews or creates incidents, the indicator is being measured incompletely. It helps to tell apart manual time avoided, added supervision time, incidents corrected and operating cost. The net improvement comes from looking at all those components.
To compare before and after, take samples of equivalent tasks: the same type of order, a similar mix of easy and complex cases and periods with comparable workload. Avoid measuring only the launch day or extrapolating an exceptional case to the whole year.
Add a quality metric tied to the goal: data errors, bookings actually confirmed or requests closed without being reopened. If the number of actions grows but customer satisfaction gets worse, the automation is not working as it should. A useful dashboard is for making decisions, including the decision to switch off a flow that does not pay off.
How we approach it at meiika
At meiika we distinguish a demonstration from a deployment with observed metrics. Our voice assistant S.A.R.A. is a demonstration case and does not allow attributing savings in hours, revenue or return on investment. To evaluate a real project we would first agree the variables and the comparison period.
Measuring is also part of the work. In Marketing, Growth and MarTech we work on analytics and data so that decisions rest on measurements and not on impressions.