Overview
What is important from a cost perspective is that usually, different combinations of services have different pricing schemes, and different elastic services could use at run-time different services with specific configurations, influencing the service cost. Involving different pricing schemes for different combinations of services, cost complexity also depends on the complexity of a cloud service’s deployment structure. Moreover, depending on the level of knowledge and particular interests of the user, some users might be interested in very complex and complete cost evaluation, while others.html might be content with an estimation over cost, even if not completely accurate.
To this end, we decided to provide two cost composition mechanisms, one using MELA’s composition rules for directly enriching monitoring information with a service’s cost based on simple cost schemes; and one providing a separate MELA Composable Cost Evaluation Service, using detailed complex cloud pricing schemes.
Cost of using cloud services to run elastic services is, in many cases, complex. Many existing cloud services can be configured and used in association, either mandatory or optional, with other services. For example, considering the Amazon EC2 IaaS services, a particular virtual machine (VM) service can be instantiated on different “availability regions”, the cost of the VM service depending on the chosen region. Many of the offered VM types can optionally use at run-time the Amazon Elastic Block Store storage service [AmazonEBS], and for additional cost, the VM can be instantiated as EBS Optimized. Moreover, some Amazon VM services, e.g., T2 series (t2.micro, t2.small, t2.medium), can be used only in conjunction with other services, in this case with an EBS storage. Thus, the cost of running T2 instances depends on the type of instance (i.e., instance resources), the availability region where the instance is deployed, and the cost of using the EBS storage. In turn, the cost of using EBS depends on the EBS storage option chosen, from General Purpose EBS at $0.10 per GB-month, to Amazon EBS Magnetic volumes costing $0.05 per GB-monthof provisioned storage and $0.05 per 1 million I/O requests. Evaluating cost even over a single IaaS service can be complex, due to different possible configurations of that service. For capturing this complexity, we extend the model for associating pricing schemes to cloud offered services, considering cloud offered services as service units, which can have Mandatory and Optional configuration options with respect to Cost, Quality, and Resources. Each of these entities, i.e., configuration options, or the service unit, can have associated different Cost Functions with different Cost Element instances. A Cost Element describes one unit of the entity’s cost, such as cost per using EBS, or cost per particular availability regions. Cost can be Periodic (e.g., hourly, monthly), or per Usage (e.g., per million I/O), is reported per periodicity or usage Unit(e.g., 1 month, 1 million I/O), over a service Metric (e.g., VM uptime, I/O). Cost can be defined from static cost per period/usage, to cost intervals, such as first million I/O free, next million $0.05. We capture such cost intervals using a costFunction specifying cost over different intervals of measured Units of the cost Metric. Finally, as cost can depend on the mandatory/optional association of the entity with other entities (service units, or quality, cost, resource options), an inAssociationWith property describes if the described cost should be applied when the entity is associated with certain other entities.
Using this model for associating cost to cloud services, we can capture from simple cost functions, such as fixed cost per used cloud service, to cost per service if used in conjunction with other services, with respect to the service usage so far (usage intervals). Using this cost composition model, the MELA Composable Cost Evaluation service computes in detail the cost for the whole service or individual units, using information about the cloud services used by the service, and structured monitoring information provided by the MELA Data Service.
Describing Cloud Pricing Schemes
To evaluate cost, first the pricing scheme of the supported cloud providers needs to be described in XML. One separate XML file must be added for each cloud provider, in the "./config/default/" folder of MELA-ComplexCostEvaluationService. To generate the cloud provider XML description we provide a fluent JAVA API. Examples of using the API to describe the pricing scheme of cloud services offered by Flexiant is available here and the generated XML is available here, and partial description of Amazon EC2 cost for VM, EBS and Network services can be found here and the generated XML is available here.
Expand for detailed explanation of the concepts captured in the fluent API for describing code schemes
-
First, we defined the cloud provider with a unique identifier (ID)
CloudProvider provider = new CloudProvider("Flexiant"); provider.setUuid(UUID.fromString("10000000-0000-0000-0000-000000000001")); Next, we define the storage cloud service, with category, subcategory, name, and unique ID
CloudOfferedService unit = new CloudOfferedService() .withCategory("IaaS") .withSubcategory("VM") .withName("CloudStorage") .withUuid(UUID.fromString("30000000-0000-0000-0000-000000000002"));Next, we define the cost functions for the service, with each cost element. For this service we have cost per disk size per hour, and cost per generated disk I/O
unit.withCostFunction(new CostFunction(unit.getName()) .withCostElement(new CostElement("diskSizeCost") .withCostMetric(new Metric("diskSize", "GB", Metric.MetricType.RESOURCE)) .withBillingCycle(CostElement.BillingCycle.HOUR) .withType(CostElement.Type.PERIODIC) //5 units per month => 5 /30 units per day per GB no mather how many GBs no mather how many hours .withBillingInterval(new MetricValue(Double.POSITIVE_INFINITY), 5d / 30 / 24 )) .withCostElement(new CostElement("diskUsageCost") .withCostMetric(new Metric("IODataSize", "GB", Metric.MetricType.RESOURCE)) .withType(CostElement.Type.USAGE) .withBillingInterval(new MetricValue(Double.POSITIVE_INFINITY), 2d)));Then we add the offered service to the provider.
provider.addCloudOfferedService(unit);We can easily generate the cloud provider XML by marshalling the defined cloud provider
JAXBContext elementContext = JAXBContext.newInstance(CloudProvider.class); //persist structure Marshaller m = elementContext.createMarshaller(); m.setProperty(Marshaller.JAXB_FORMATTED_OUTPUT, true); m.marshal(provider, new FileWriter("file"));
MELA service service specific configuration
Collecting required monitoring information
To enable cost evaluation, we must ensure we first collect the required monitoring information, according to the cost metrics defined in the cloud pricing scheme. To this end, MELA metric composition rules should be defined, depending on the configured monitoring data sources.
While can be written directly in XML, we also provide support for generating the composition rules from Jav
-
Example of defining a rule which retrieves from Ganglia the metric "dataOut" and "dataIn" expressed in bytes, converts them to GB, and sums them up over all VMs used by a service unit (for all service units, such as a Load Balancer, Data Node), creating a new "dataTransfer" metric.
//data transfer rule CompositionRule computePublicIPDataTransferForServiceUnit = new CompositionRule() .withTargetMonitoredElementLevel(MonitoredElement.MonitoredElementLevel.SERVICE_UNIT) .withResultingMetric(new Metric("dataTransfer", "GB", Metric.MetricType.RESOURCE)) .withOperation(new CompositionOperation() .withOperationType(CompositionOperationType.SUM) .withSubOperation(new CompositionOperation() .withOperationType(CompositionOperationType.DIV) .withMetricSourceMonitoredElementLevel(MonitoredElement.MonitoredElementLevel.VM) .withReferenceMetric(new Metric("dataIn", "byte", Metric.MetricType.RESOURCE)) .withValue("" + ((Double) Math.pow(1024, 3)).toString()) ) .withSubOperation(new CompositionOperation() .withOperationType(CompositionOperationType.DIV) .withMetricSourceMonitoredElementLevel(MonitoredElement.MonitoredElementLevel.VM) .withReferenceMetric(new Metric("dataOut", "byte", Metric.MetricType.RESOURCE)) .withValue("" + ((Double) Math.pow(1024, 3)).toString()) ) );
Specifying used cloud offered services
The next configuring step is specifying the service's configuration, with the used instances of cloud offered services. and submitting this structure to MELA Data Service
While the structure cab be written directly in XML, we also provide support for generating it from Jav
-
Example of service structure with used cloud offered services
MonitoredElement service = new MonitoredElement("Service").withLevel(MonitoredElement.MonitoredElementLevel.SERVICE) //with used network service .withCloudOfferedService(new UsedCloudOfferedService(provider.getUuid(), "Flexiant", UUID.fromString("30000000-0000-0000-0000-000000000001"), "PublicVLAN")) //local processing topology .withContainedElement(new MonitoredElement("LocalProcessingTopology").withLevel(MonitoredElement.MonitoredElementLevel.SERVICE_TOPOLOGY) //local processing unit .withContainedElement(new MonitoredElement("LocalProcessingUnit").withLevel(MonitoredElement.MonitoredElementLevel.SERVICE_UNIT) //used image storage .withCloudOfferedService(new UsedCloudOfferedService(provider.getUuid(), "Flexiant", UUID.fromString("40000000-0000-0000-0000-000000000001"), "ImageStorage")) //first unit instance using a VM service .withContainedElement(new MonitoredElement("10.99.0.40").withLevel(MonitoredElement.MonitoredElementLevel.VM) .withCloudOfferedService(new UsedCloudOfferedService(provider.getUuid(), "Flexiant", UUID.fromString("20000000-0000-0000-0000-000000000001"), "1CPU1") //the ID of the service instance .withInstanceUUID(UUID.randomUUID())) .withCloudOfferedService(new UsedCloudOfferedService(provider.getUuid(), "Flexiant", UUID.fromString("30000000-0000-0000-0000-000000000002"), "CloudStorage") //the ID of the service instance .withInstanceUUID(UUID.randomUUID())))) .withContainedElement(new MonitoredElement("10.99.0.23").withLevel(MonitoredElement.MonitoredElementLevel.VM) .withCloudOfferedService(new UsedCloudOfferedService(provider.getUuid(), "Flexiant", UUID.fromString("20000000-0000-0000-0000-000000000001"), "1CPU1") //the ID of the service instance .withInstanceUUID(UUID.randomUUID())) .withCloudOfferedService(new UsedCloudOfferedService(provider.getUuid(), "Flexiant", UUID.fromString("30000000-0000-0000-0000-000000000002"), "CloudStorage") //the ID of the service instance .withInstanceUUID(UUID.randomUUID()))) ) ) ;
MELA Composable Cost Evaluation Service RESTful API
MELA-ComposableCostEvaluationService exposes a RESTful API, in which the service ID or version is marked with the '{serviceID}' parameter. The API enables a user to list the managed cloud services (call 1), and retrieve their current structure, containing the used cloud offered services (calls 2-3). The complete decomposition of system total cost can be retrieved with calls 5-6. Decomposition of total and instant can be retrieved in different formats (tree, pie chart), and for specific moments in time using calls 7-18. The cost of a different configuration for an already monitored service can be analyzed by using call 19. Calls 20-27 provide in different formats information about the cost efficiency of the analyzed service, while call 28 returns complete historical cost decomposition in CSV.