Documentation and Guides

MELA Data Service

View project onGitHub

Overview

In general, elastic cloud services can re-configure individual units or the whole service at run-time, due to various elasticity requirements. Such run-time re-configuration can either focus on: (i) vertical scaling, adding/removing resources to existing units; or (ii) horizontal scaling, duplicating units/instantiating more instances of the same type. Due to individual horizontal scaling mechanisms, one service unit could “burst” by a controller, creating a new instance in unused virtual machines along other units, or in new machines. This generates two main issues. First, as units can dynamically appear/disappear on virtual machines, a monitoring system for elastic services must be able to dynamically start/stop collecting metrics on individual virtual machines. Secondly, as virtual machines themselves are created/destroyed dynamically at run-time, if monitoring information is associated only with each virtual machine, it will be lost during scale-in operations which destroy virtual machines. While the first issue can be solved using existing monitoring systems such as Ganglia or JCatascopia, for solving the Virtual Machine (VM) volatility issue, we must associate monitoring information with the service structure. Addressing this second issue is the initial objective of MELA Data Service.

Note: From here onwards we will refer to services and cloud applications with the same "services" term.

Service structure specification

Elastic Cloud Service representation model
Figure 1: Elastic Cloud Service representation model

MELA considers any cloud application as a service, and uses a representational model which divides Cloud Services in Service Topologies, which are composed of Service Units, which run in Virtual Machines members of Virtual Clusters. In Listing 1 we show an application description example, which describes a CloudService having two topologies, a DataEndServiceTopology and an EventProcessingServiceTopology. The topologies are just logical concepts, and have no physical equivalent. Each topology has Service Units. The service description must be submitted to MELA issuing a “PUT” request towards the MELA “/service” service.

Listing 1: Service Structure Description Example
<MonitoredElement id="CloudService" level="SERVICE">
                  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                  xsi:noNamespaceSchemaLocation="structure.xsd"
    <MonitoredElement id="DataEndServiceTopology" level="SERVICE_TOPOLOGY">
        <MonitoredElement id="DataControllerServiceUnit" level="SERVICE_UNIT">
            <MonitoredElement id="10.99.0.1" level="VM"/>
        </MonitoredElement>
        <MonitoredElement id="DataNodeServiceUnit" level="SERVICE_UNIT">
            <MonitoredElement id="10.99.0.2" level="VM"/>
        </MonitoredElement>
    </MonitoredElement>
    <MonitoredElement id="EventProcessingServiceTopology" level="SERVICE_TOPOLOGY">
        <MonitoredElement id="LoadBalancerServiceUnit" level="SERVICE_UNIT">
            <MonitoredElement id="10.99.0.3" level="VM"/>
        </MonitoredElement>
        <MonitoredElement id="EventProcessingServiceUnit" level="SERVICE_UNIT">
            <MonitoredElement id="10.99.0.4" level="VM"/>
        </MonitoredElement>
</MonitoredElement>
</MonitoredElement>

The XML schema for writing service structure specifications can be downloaded from here

Composition rules specification

The second important configuration step is the specification of metric composition rules, issuing a “PUT” request towards the MELA “/metricscompositionrules” service. In Listing 2 we show a composition rule example, which applies a SUM operation over the “numberOfVMs” metric collected from all the Virtual Machine instances belonging to any Service unit, and creates a new metric also called “numberOfVMs” which will be associated with the Service Unit for which is computed.

<CompositionRulesConfiguration>
                  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                  xsi:noNamespaceSchemaLocation="compositionRules.xsd"
    <MetricsCompositionRules>
        <CompositionRule TargetMonitoredElementLevel="SERVICE_UNIT">
            <ResultingMetric type="RESOURCE" measurementUnit="number" name="numberOfVMs"/>
            <Operation MetricSourceMonitoredElementLevel="VM" type="SUM">
                <ReferenceMetric type="RESOURCE" name="numberOfVMs"/>
            </Operation>
        </CompositionRule>
    </MetricsCompositionRules>
</CompositionRulesConfiguration>

The XML schema for writing metrics composition rules can be downloaded from here

A composition rule applies a set of composition operations over a set of operands.

Writing a Composition rule

  1. For a composition rule one must first specify the “ResultingMetric”, i.e. the type (RESOURCE, COST, QUALITY), name and measurement unit of the Metric resulting from the composition rule.

    Example: <ResultingMetric type="RESOURCE" measurementUnit="ms" name="avgResponseTime"/>

  2. The “TargetMonitoredElementLevel” must be specified, which defines at WHICH level in the application structure level (SERVICE, SERVICE_TOPOLOGY, SERVICE_UNIT, VIRTUAL_CLUSTER, and VM) the composition rule is applied. If ONLY the “TargetMonitoredElementLevel” is specified, the composition rule will be applied to all instances having the specified level (e.g., to all SERVICE_UNIT instances)

    Example: <CompositionRuleTargetMonitoredElementLevel="SERVICE_UNIT">

  3. If a list of "TargetMonitoredElementID" are specified, this indicates particular elements from the application structure on which the composition rule will be applied.

     Example: 
    <TargetMonitoredElementID>DataControllerServiceUnit</TargetMonitoredElementID>
    <TargetMonitoredElementID>LoadBalancerServiceUnit</TargetMonitoredElementID>
    

  4. The last element is the list of “CompositionOperation” specifications, where we define the composition operation to be applied at the specified “TargetMonitoredElementLevel” over the optionally specified “TargetMonitoredElementID”

  5. The metric resulting from applying the composition rule will be added to the monitoring data at the specified “TargetMonitoredElementLevel” and optionally specified “TargetMonitoredElementID”. If no optionally specified “TargetMonitoredElementID” are present, the metric will be added to all elements belonging to the specified level.

Composition Operation

A Composition Operation applies an operation over one or more operands, which can be values or metrics. Depending on the operation type, each operation is processed differently.

Operands

For a composition operation, an operand can be:

  • A “value”: any statically defined value Example: <Operation value="100" type="ADD">

  • A “ReferenceMetric” , which indicates that the operation will be applied on value of the ReferenceMetric.

    Example:
         <Operation MetricSourceMonitoredElementLevel="VM" type="AVG">
             <ReferenceMetric type="RESOURCE" name="cpuIdle"/>
         </Operation>
      

  • For an operation involving a “ReferenceMetric” , one must also specify the “MetricSourceMonitoredElementLevel”, which indicates at which application structure level (SERVICE, SERVICE_TOPOLOGY, SERVICE_UNIT, VIRTUAL_CLUSTER, and VM) the reference metric will be found UNDER the monitored element over which the rule is applied. The “MetricSourceMonitoredElementLevel” can equal the “TargetMonitoredElementLevel”, or indicate any level belonging to the children of the “TargetMonitoredElementLevel”.

    Example:
         <CompositionRule TargetMonitoredElementLevel="SERVICE_UNIT">
             <Operation MetricSourceMonitoredElementLevel="VM" type="AVG">
      
  • For defining which monitored element from the “MetricSourceMonitoredElementLevel“ is considered in extracting metric values, any number of “SourceMonitoredElementID” can be defined.

    
        Example:
         <Operation MetricSourceMonitoredElementLevel="SERVICE_UNIT" type="KEEP">
            <ReferenceMetric type="RESOURCE" measurementUnit="req/s" name="troughput"/>
            <SourceMonitoredElementID>LoadGenerator
                                    

Sub-Operations

An operation can contain sub-operations, and the values returned by those sub-operations are used by the current operation, in a hierarchical manner.

Example
     <Operation value="100" type="ADD">
       <Operation value="-1" type="MUL">
         <Operation MetricSourceMonitoredElementLevel="VM" type="AVG">
	   <ReferenceMetric type="RESOURCE" name="cpuIdle"/>
         </Operation>
       </Operation>
     </Operation>
  

Operation Type

For any operation, it is important to specify the type, as the operations are processed differently depending on their type

  • SUM:
    Intended to sum the values retrieved from a “ReferenceMetric”

    Example
         <CompositionRuleTargetMonitoredElementLevel="SERVICE_UNIT">
           <ResultingMetric type="RESOURCE" measurementUnit="numberOf" name="numberOfVMs"/>
           <Operation MetricSourceMonitoredElementLevel="VM" type="SUM">
             <ReferenceMetric type="RESOURCE" name="numberOfVMs"/>
           </Operation>
         </CompositionRule>
       

  • AVG:
    Applied similar to SUM, extracts the average.

  • MAX / MIN:
    Applied similar to SUM, extracts the maximum or minimum value.

  • DIV/MUL/ADD/SUB:
    Divide, Multiply, Subtract From, Add To, the value of the first operand the value of the second operand

    Example
         <Operation value="-1" type="MUL">
             <Operation MetricSourceMonitoredElementLevel="VM" type="AVG">
               <ReferenceMetric type="RESOURCE" name="cpuIdle"/>
             </Operation>
         </Operation> 
     

  • SET_VALUE
    Used in creating metric values

    Example
         <CompositionRuleTargetMonitoredElementLevel="VM">
         <ResultingMetric type="RESOURCE" measurementUnit="ms" name="numberOfVMs"/>
           <Operation value="1" type="SET_VALUE"/>
         </CompositionRule>
     

  • CONCAT
  • UNION
  • KEEP/ KEEP_LAST/ KEEP_FIRST

MELA-DataService RESTful API

MELA-DataService exposes a RESTful API (Table 11), in which the application ID or version is marked with the "{serviceID}" parameter. The API enables a user to submit or remove an application (1,2), add metrics composition rules (3), application requirements (4), and update the application structure after scaling actions (5). The updated structure of the application can be retrieved at each given time using call 6, and the available monitored metrics for the application\92s VMs using call 7. The application structured monitoring data can be retrieved either as JSON (8), or as XML (9). If monitoring data is required only for a certain application component, it can be retrieved using call (10), by issuing a POST containing as argument the monitored element for which data should be retrieved. Historical monitoring data can also be retrieved using calls 11-13, while 14 and 15 provide methods for accessing the metrics composition rules in JSON or XML for the target application. Methods 16-17 let a user add/remove executing actions on the application, used for display purposes. Method 18 allow a user to test if all metrics coming from a service have values >=0, indicating whether the application is operating properly, useful for determining after a scale action if the application is stable. Finally, call 19 allows a user to list all applications monitored by MELA-DataService.

No.

Method

URL

Input

Output

1.        

PUT

/service

application/xml

-

2.        

DELETE

/{serviceID}

-

-

3.        

PUT

/{serviceID}/metricscompositionrules

application/xml

-

4.        

PUT

/{serviceID}/requirements

application/xml

-

5.        

PUT

/{serviceID}/structure

application/xml

-

6.        

GET

/{serviceID}/structure

-

application/xml

7.        

GET

/{serviceID}/metrics

-

application/xml

8.        

GET

/{serviceID}/monitoringdata/json

-

application/json

9.        

GET

/{serviceID}/monitoringdata/xml

 

application/xml

10.     

POST

/{serviceID}/monitoringdata/xml

application/xml

application/xml

11.     

GET

/{serviceID}/historicalmonitoringdata/all/xml

-

application/xml

12.     

GET

/{serviceID}/historicalmonitoringdata/ininterval/xml

-

application/xml

13.     

GET

/{serviceID}/historicalmonitoringdata/lastX/xml

-

application/xml

14.     

GET

/{serviceID}/metriccompositionrules/json

-

application/ json

15.     

GET

/{serviceID}/metriccompositionrules/xml

-

application/xml

16.     

PUT

/{serviceID}/{targetEntityID}/executingaction/{action}

-

application/xml

17.     

DELETE

/{serviceID}/{targetEntityID}/executingaction/{action}

-

application/xml

18.     

GET

/{serviceID}/metricsGreaterEqualThanZero

-

Boolean

19.     

GET

/services

-

application/json

MELA-DataService REST API