Business Metrics, KPIs and AWS Infrastructure Cost Reporting
Common Business Metrics and Leading vs Lagging indicators
There are many drivers to have clarity around the performance of a business. Most directly they are internal for the executive, board and leadership team to be able set strategy and track performance to that. But they are also external such as compliance (e.g. regulatory oversight and accounting) or fund-raising (e.g. venture capital).
In order to do this efficiently there are a number of standardised metrics that are used. Where financially focused these will target either the cost side, the revenue side, or some ratio between the two. Common examples include:
- monthly or annual recurring revenue (MRR / ARR)
- gross / net profit margin
- the cost of acquiring a customer (CAC)
- the average duration it takes to “pay back” customer acquistion costs
- revenue per employee (RPE)
These well known metrics are preferred because there are industry expectations for each and having an easy method for comparison is highly desirable for understanding performance. These are all lagging indicators in that they are tracking historical record rather than making a prediction about the future that a leading indicator might show. Leading indicators may provide an opportunity to anticipate an outcome and act to influence that prior to it happening but are less reliable or open to intepretation so are inherently less trusted. Examples of leading indicators include:
- sales pipeline coverage (multipler over budget)
- customer usage of the service
- support activity
- percentage of business from referals
At some point these metrics converge with the work of the finance and accounting team where the viewpoint is focussed on OpEx/CapEx or from the cashflow statement:
- “above the line” costs: required to enable revenue (e.g. COGS)
- “below the line” costs: general and administrative costs
A practical example of how this translates to a SaaS business is the desire to understand:
- the cost of product development (including related I.T. and infrastructure costs)
- the incremental cost of delivering and then servicing a unit sale
The goal outcome for knowing these metrics is an ability to forecast costs given the various scenarios of sales and or other activity (e.g. product development) within the organisation. The outcome will be a blend of fixed costs that continue regardless of activity or variable costs that are dynamic and linked to what is happening in the business.
Cloud Infrastructure Cost Reporting
In modern businesses I.T. and in particular cloud infrastructure is (outside of labour) both the top cost and enabler for the business. Its ability to influence the success of the business means that having a handle on cost modelling is critical. In my previous article on delivery maturity I discussed an overview of cost reporting capabilities that I’ll touch on here.
| Cost Factor | Description |
|---|---|
| Attribution | All I.T. & infrastructure services are attributed to a cost centre |
| Owner | Cost centres have owners or related stakeholders who are informed of cost positions |
| Modelling | As described in this article an understanding exists of the key reporting dimensions relevant for the business |
| Reporting | Regular cost reports are presented to stakeholders in a way that aligns to the cost model and facilitates informed cross-functional decisions |
| Budgets & Forecasts | Cost Forecasts are made for the cost centres and actual costs are tracked against budgets over time |
| Anomaly Detection | Spikes in cost are detected in a timely way so action can be taken to prevent unintended blowouts |
Examples of fixed costs in a cloud infrastructure environment would be data storage and servers (for a stable production environment) where as variable costs might be analytical tasks or compute for temporary or ad hoc work. By tagging or having a methodology for putting the infrastructure costs in appropriate buckets the process of getting to these metrics can be significantly automated.
A good starting point for this process is to identify the ‘dimensions’ for analysis. I’ve written more on dimensions here. These reporting dimensions would typically include:
- time (e.g. daily, weekly, monthly)
- geography (e.g state, country)
- service type (perhaps related to the customer or sales categorisations)
- organisation (e.g team or business unit)
- cost profile: fixed, variable (as described above)
With these dimensions sorted it’s then necessary to ensure your cost drivers (such as AWS in the example I’ll continue with) can be identified in the cost reporting tools. Where these are not built into standard attributes of AWS these can be set with tags (although not retrospectively of course).
An AWS Cost Reporting Solution
A simple pattern I’ve developed for cost reporting in AWS is to extract the costs using a script and then store and model them in a BI Dashboard like PowerBI or Looker Studio.
The script uses the AWS boto3 Python SDK. Although the AWS Cost explorer is extremely useful for ad hoc cost reporting it has various constraints for ongoing cost tracking:
There is limited history in Cost Explorer although this has been extended recently. By extracting the data yourself you can manage it fully.
The filters & grouping capabilities of the Cost Explorer UI are powerful, but lack the ability to be combined (using boolean logic) for more advanced tracking. When using the API and simple object modelling this is quite easy to achieve.
The reporting and export to S3 capabilities of the Cost Explorer are tricky to work with. Yes, there are solutions that plug proper reporting platforms into this but you need to be prepared for the effort to setup and maintain these.
A fully automated model can be achieved using AWS Lambda to run the scripts
Other considerations with AWS Cost Tracking:
Use AWS accounts to delineate cost centres as much as possible. Being able to filter by account is a full-proof separator that will capture all costs. Note: a multi-account strategy is part of the AWS ‘well architected’ framework review
Invariably tagging & tracking services by name will locate most but not all of the costs in your bill. The gaps will be hard to allocate and where you aren’t using a multi-account approach as in point (1) these will be present. This script attempts resolve that by firstly determining the total cost for the month. After doing the filter based reporting, it then determines the gap to the total and uses that to scale up the results so they match. This approach is not 100% correct obviously, but is a low effort solution, and based on the % adjustment you can decide whether it’s acceptable. Tinkering with the filters can improve this outcome if that’s needed.
The dimensions defined above can be modelled in the cost_filters.py file linked with the script
By using a solution outside of AWS it’s easier to merge with the cost of other vendors. You can also further process the data such as apply exchange rates (whether that be at time of invoicing, or an agreed rate set for budgeting purposes).