Planning and estimation
Deploying workloads in the cloud requires an assessment of expected costs and potential value. This process includes exploring opportunities for automation, sustainability, and cost optimization.
Explore different cloud scenarios
Define the scope of the estimation.
Identify the information needed for the estimation.
Determine the model parameters for evaluating the future state.
Estimate the business value of each scenario
Use calculators and estimation tools to explore the relationship between cost, usage, and environmental impact.
Compare with similar applications and systems.
Extrapolate from past cost and technical demand metrics.
Document the expected impact of future plans.
Use trial runs and non-production environments for estimation.
Consider pricing, policies, carbon emission targets, shared services, and other support costs.
Plan the implementation
Plan a Proof of Concept (PoC).
Propose recommendations for tuning the forecast model.
Definition
Cloud services are numerous, frequently updated, and constantly emerging with new offerings and managed service layers. The models for building and migrating applications in the cloud are also diverse. This complexity requires a robust set of practices to estimate the long-term costs of a workload or system. Organizations also need to estimate and plan their resource consumption based on their sustainability goals. The scope of this estimation can range from a single service change to a full application migration to the cloud. Organizations often perform multiple estimations to compare the potential business value of different scenarios.
Engineers are primarily responsible for estimation, with support from the FinOps team. If an estimation is critical, has a significant impact, is complex, or requires an experimental budget, input from product, finance, or management teams may also be necessary.
Planning and estimation is closely related to forecast management. Estimations help you understand potential future costs in different scenarios or use cases, which aids in planning for migration, deployment, or modernization projects. The results of an estimation are used as input for forecast management. Forecast management creates and maintains a detailed forecast model for a change after it has been planned. It also shows the expected spending and value creation that engineering or product teams are expected to deliver.
In contrast, planning and estimation is exploratory. It is an upstream activity that precedes cloud cost forecasting and other activities. Estimations are often required to support activities in the cloud cost optimization realm, such as cloud architecture optimization and workload optimization, and even to support workload onboarding.
To estimate future cloud costs, an organization must clearly define the scenario. This includes understanding the services, architecture, or change plan, the technology deployment patterns the organization uses, and the estimation parameters for clear communication. Various scenarios might be set for a specific change. For example, an engineering team might estimate and compare the cost, effort, and impact of migrating a workload from virtual machines to a managed service, a Kubernetes environment, or a serverless computing model.
A range of methods can be used to estimate cloud costs, including the following:
Cost calculators: Most cloud service providers and some third-party vendors offer calculators to estimate the cost of expected services. These calculators are effective for estimating costs in simple service replacement scenarios but are less effective for estimating the cost of applications that do not yet exist in the organization's cloud environment.
Carbon emission calculators: You can use tools from cloud service providers and third parties to estimate the current carbon footprint.
Comparison with similar applications: This is suitable for organizations with a range of standardized architectures or applications. If a new system follows a common pattern, engineers can begin the estimation for a new change or workload by comparing the costs and performance of existing systems.
Extrapolation from historical costs: Extrapolating from historical costs simplifies the estimation of changes in an existing system. You can isolate the change or usage of a single service and infer future costs from current costs. This method is suitable for estimating isolated service changes within large systems.
Estimation based on future plans: The on-demand nature of cloud computing allows organizations to save costs because resources are deployed only when needed. Therefore, large-scale estimations for an entire system should consider the phased deployment of all required resources and environments. For example, to create a new system in the cloud, you might only need to deploy development and staging environments at the beginning. The production environment might be deployed months later as development nears completion. The estimation should then calculate the cost of this phased deployment based on the system's future plan, rather than estimating the cost of all environments at once.
Trial run estimation: Using automation tools such as infrastructure as code (IaC) scripts, you can quickly build and destroy cloud services. Therefore, an organization can accurately estimate costs by creating a staging environment on the cloud platform, running it for one to three days, and then destroying it. Although this method incurs a small fee from R&D or experimental budget funds, it provides a more realistic cost estimate and also validates the effectiveness of the IaC scripts.
In all methods, engineers should collaborate with the FinOps team to ensure the estimation aligns with cloud policies, such as policies that dictate where to create resources, which types of resources to use, and appropriate architectural models. This collaboration also ensures that cost estimates are reasonable, considering on-demand pricing, discount rates, and expected resource commitment levels. The estimation should also cover shared costs, platform adjustments, and other support costs and impacts. These impacts are not limited to financial costs but should also include other factors, such as the effect on sustainability and the operational impact of implementing the change.
Estimation scenarios can be used as input for the forecasting process or as feedback to the optimization process that triggered the estimation work. If a PoC budget is needed to perform the estimation or to obtain more specific cost information, the finance department must be involved.
Unfortunately, no single estimation method is perfect. The variability of cloud spending makes it inherently difficult to predict. Engineers can also often create environments and workloads at any time without going through a procurement process. Therefore, having a robust estimation capability is crucial. This requires clear parameters, scenario planning, tools, and documentation requirements.
Maturity assessment
Crawl
Simple or consistent application build patterns require fewer or more direct estimation techniques.
Estimates do not need to be very granular or accurate because the organization's cloud spending is low or immature.
Different stakeholders in the organization use different cloud cost data sources and tools for forecasting. Consistency is not required.
Estimates are often recorded manually or on an ad-hoc basis due to low usage or simple scenario planning.
The organization relies more on trend-based extrapolation than on complex mechanisms.
There is limited need for a consolidated view of overall estimates, perhaps only by business unit or cost center.
Fewer shared cost or add-on cost elements are needed to complete the estimation.
Engineering teams are involved in creating estimates, but there is little need to track the variance between estimated and actual values.
The organization uses cloud service provider tools and dashboards to measure the current carbon footprint.
The organization understands the correlation between cloud resource usage and carbon footprint.
Walk
A wider range of application build patterns or complex cloud usage requires more estimation techniques.
Estimates are more granular at the service level, and the organization requires higher accuracy for estimates.
Estimates consider cloud rate optimization, commitment-based discounts, and carbon emission targets.
Estimates are performed regularly with a unified recording method but are not yet automated.
Core and related FinOps roles, such as product, management, engineering, finance, and sustainability, have access to cloud cost estimation data.
The FinOps team regularly reviews estimation accuracy and estimation trends with relevant teams.
Run
Complex and large-scale build patterns and cloud usage require sophisticated estimation methods.
Estimates are extremely granular and include multiple types of shared cost elements, platform costs, support costs, sustainability, and policy compliance elements.
Estimations are created using multiple pricing models, including discount adjustments and amortized cloud usage pricing.
Estimation scenarios support forecast management capabilities for rolling, trend, and driver dimensions.
Optimization opportunities are included in the estimates.
Estimates align with the allocation structure used for reporting cloud costs within the organization.
Estimation granularity is refined based on organizational KPIs for business units, cost centers, teams, products, or services.
Core and related FinOps roles, such as product, management, engineering, finance, and sustainability, can view a single source of truth in real-time to understand how cloud resource usage affects estimates, forecasts, trends, the carbon footprint, and budgets.
The organization automates the integration and data stream between cloud cost forecast data, Environmental, Social, and Governance (ESG) data, and backend accounting systems to facilitate broader organizational reporting.
Functional activities
FinOps practitioner
Set estimation requirements and parameters for product and engineering teams.
Generate reasonable, accurate, and granular estimation results.
Help establish estimation KPIs that align with business goals.
Provide cloud cost estimation results that stakeholders can use to create forecast models.
Participate in implementation planning by estimating scenarios for pricing, policies, allocated costs, and other impacts.
Provide teams with granular reports on projected spending, broken down by different business dimensions.
Coordinate the roles and stakeholders involved in the entire estimation process.
Provide reports on estimation accuracy trends.
Product
Track product-centric Key Performance Indicators (KPIs) to measure the cost-effectiveness of achieving expected business outcomes.
Collaborate with the FinOps sustainability role to track unit carbon emission level (metric tons of CO2 equivalent, MTCO2e) KPIs for products, departments, or workloads to account for the carbon footprint.
Use these KPIs in collaboration with engineering teams to inform estimation scenarios.
Set estimation parameters that align with the product lines for which they are responsible.
Finance
Define requirements for estimation accuracy and granularity.
Help set estimation KPIs that align with business goals.
Provide cloud cost data that enables stakeholders to create forecasts, which helps in budget formulation.
Provide teams with detailed reports on forecasted spending by different business dimensions.
Provide reports comparing budget vs. actuals, forecasts, and estimates to identify trends and compare against variance KPIs.
Provide experimental funds when needed to produce more accurate estimation results.
Engineering
Monitor cloud spending forecasts for budget overruns that trigger estimation work.
Monitor whether optimization, migration, or architectural design triggers estimation work for new resources.
Collaborate with core and related FinOps roles, such as FinOps practitioners, product, and sustainability teams, to create valuable estimation scenarios as needed.
Understand all estimation scenarios that should be considered in various situations.
Obtain approval for planned changes that affect cloud spending forecasts or budgets.
Lead estimation efforts in their areas of responsibility.
When forecast models are available, understand when new estimates are needed and how these estimates affect the existing forecast models for which they are responsible.
Consider cost, optimization, and carbon emissions when designing and building infrastructure.
Management
Understand the process of estimating cloud costs and their impacts, and support engineering staff in this work.
Set estimation variance thresholds that align with business goals.
Manage priorities for conflicting project timelines when estimated spending affects the budgets of high-priority projects.
Facilitate communication between core and related FinOps roles to ensure that technology spending and ESG priorities align with business goals to create value.
Weigh and make planning decisions when different estimation scenarios need to be compared.
Sustainability
Provide actual and estimated carbon footprint values, enabling stakeholders to build ESG forecast models.
Collaborate with core FinOps roles and ESG teams to understand and establish the organization's sustainability goals.
Use cloud service provider tools to provide carbon footprint reports to the broader FinOps and ESG teams.
Success metrics and KPIs
Estimation models are built using discount-adjusted, amortized cloud usage data.
The variance between estimated cost and actual cost trends is within a set percentage threshold.
Estimates are performed quickly using appropriate mechanisms relevant to business goals.
Estimates consider shared costs, reasonable pricing metrics, environmental sustainability impacts, and other relevant elements.
A schedule for regular meetings is established.
Establish organization-specific unit cost and usage KPIs, such as Private Pricing Agreements (PPAs) and Enterprise Discount Programs (EDPs), to help create more accurate forecasts.
Inputs and outputs
Inputs
Estimations triggered by workload optimization
Estimations triggered by rate optimization
Estimations triggered by reporting and analytics
Estimations triggered by workload onboarding
Estimations triggered by forecast management
Details of the current system or workload
Functional and non-functional requirements for the system or workload in the target location or model
Details of the scenarios that need to be modeled
The desired state of the system or workload, or future change plans
Current architectural guidelines (cloud architecture optimization)
Required estimation parameters, such as time frame and precision
ESG regulations and standards for FinOps practitioners to follow
Data needed to validate accuracy
Outputs
Cost, carbon emission, and delivery estimates for a system or workload in a specific scenario
Recommendations based on estimates from multiple competing scenarios
Potential threats and confidence level
Data Quality scorecard for data validation, including KPIs, validation frequency, and more