2. Working with the API

Before introducing the API, some background on the CTL CDN provisioning architecture is in order. The figure below illustrates the high-level CDN provisioning architecture:

../_images/Provisioning_Arch.png

As shown, the CTL Media Portal exposes both an API and a UI for use in provisioning CDN services. Downstream of Media Portal is the service image DB, which is a geo-distributed noSQL platform that stores all service images. When a service image is promoted to production, the config generation system reads the service image and translates it into a platform configuration, which is then distributed out to the network. It is important to note that this is an asynchronous process, so a successful response to pushing a service image to the network does not mean that the configuration was successfully pushed to the network. The time from pushing a service image to production to when it is in place and running is typically less than 5 minutes.

2.1. Environments

The v2 provisioning system supports pushing service images to different environments. Customers are provided access to staging and production network environments. The staging environment is intended to allow for testing of new configurations exactly as they are expected to run in production. As such, access to the staging environment requires testers to force requests to specific IP addresses. This can be done by modifying the test environment’s local host file or by using the native capabilities of the test tool. For example, if the tester is using cURL and the staging network of 192.0.2.5, they can use the –resolve option.

>curl --resolve www.example.com:80:192.0.2.5 http://www.example.com/index.html

For a list of staging servers, a DNS query can be performed against cdn-staging.footprint.net.

A few important notes:

  1. Traffic sent to the staging environment is billed at the same rate as production traffic.
  2. The staging environment is not intended for load or performance testing.
  3. A service image must first be copied to staging then production. skipping the staging environment may result in an incomplete configuration on the network.

2.1.1. Status

When a service image is pushed to an environment, it is placed in a queue for processing. The API returns an indication of success (which means it was successfully placed in the queue) and a LVLT-TransactionId header containing a transactionID which can be used to track the status of the request. The transaction-queue endpoint can be used to determine the status of the request and whether the Service Image has been successfully processed and pushed to the network, or if any errors were encountered. The endpoint takes the transactionID as a path element and returns the following:

 {
   "transaction_id": uuid,
   "client_transaction_id": text,
   "subscriber_id": int,
   "transaction_queue_time": timestamp,
   "start_time": ,
   "complete_time": timestamp,
   "env": text,
   "env_version_id": int,
   "source_config_type": text, // stored|environment
   "source_config_name": text,
   "source_config_version_id": int,
   "type": text, //pcg
   "state": text, // QUEUED|STARTED|ERRORED|COMPLETED|CANCELED
   "message": text,
}

Where the key elements are defined as follows:

  • transaction_queue_time: The time the request was received
  • start_time: The time the system began processing the request
  • complete_time: The time the system completed processing the request and made the platform config available for distribution to the network.
  • state: The outcome of processing the service image.

2.2. Workflow

Service images are managed much in the same way as files in a software version control system. Service images are stored in a per customer workspace as a named service image and independently versioned. A named service image cannot be pushed/published to an active environment (e.g., staging or production). Rather, it must be copied from the customer’s workspace or copied from one active environment to another. (Moving from one environment to another does not constitute a version change.) The system tracks each environment’s history by subscriber, capturing each named service image and version copied to the environment. This is analogous to tagging a release.

Note that a service image must be copied to the staging environment before production.This is due to how the backend system builds platform configuration files for use by the caching software.

There are some basic restrictions on how a service image can be named. The following restrictions are in place:

  • Allowed Characters
    • Alphanumerics [0-9a-zA-Z]
    • Special characters “-“, “_”, and “.”
  • Maximum length of 128 Characters

When an update to an existing service image is posted to the workspace, the system increments the service image version number and inserts the new service image into the service image DB. The previous service image is retained with a version of (current version - 1).

As part of the service activation process for new customers, an initial service image is provided that can be edited and copied to one of the available environments.

Note that a newly named service image cannot be posted to the system. If a new service image is desired, a copy must first be made of an existing service image and then edited. This is analogous to creating a new branch that can evolve independently from its source file.

This workflow is illustrated in the figure below:

../_images/SI_Workflow.png

Since service images are named and versioned, there is no requirement to keep the staging and production environments in sync. One can easily compare the two based on the service image name and version number.

2.3. Schema

The service image is described by a JSON schema. The schema can be found on Media Portal Online Help (Media Portal Web Service APIs/API Descriptions/Http Delivery (Under the heading Http Delivery - Version 2.0). The link to the schema contains a zip file with all the relevant schema files and a brief README.txt describing the inheritance.

The schema represents all possible subscriber configuration attributes. Updates are performed using a Read/Modify/Replace operation. Partial updates to a service image are not supported.

Starting with the 19.5 release, two new keys are being introduced to the schema, schemaVersion and genSchemaVersion. schemaVersion is used to identify the version of the schema, genSchemaVersion identifies the schema version used to create the service image. schemaVersion is represented as follows:

"schemaVersion": {
  "type": "string",
  "description": "A constant string describing the major and minor release numbers of this version of schema.",
  "enum": ["1.0"]
}

The schemaVersion key will only appear in the schema and should never appear in actual JSON data (service image). This will be enforced by the schema by requiring the key NOT be present. Its purpose is to identify the version of the schema. The schemaVersion key will be changed manually in new software releases to indicate when changes have occurred.

genSchemaVersion is represented as follows:

"genSchemaVersion": {
  "type": "string",
  "description": "A constant string describing the major and minor release numbers of the version of schema used to generate this JSON data.",
}

The genSchemaVersion must be present in the Service Image and is used to indicate the schemaVersion used to create the Service Image.

2.3.1. Example JSON Service Image

{
  "subscriberId": 29,
  "reportingIds": [
    {
      "logEndpoints": [
        0
      ],
      "id": 136518,
      "name": "Reporting Group 1"
    },
    {
      "logEndpoints": [
        0
      ],
      "id": 192694,
      "name": "Reporting Group 2"
    }
  ],
  "originDefinitions": [
    {
      "id": 98,
      "description": "My Origin",
      "protocol": "HTTPS",
      "name": "Origin_1",
      "host": "owww.example.com",
      "originDNS": "www.example-origin.com",
      "port": 443,
      "webroot": "/test"
    }
  ],
  "aliasDefinitions": [
    {
      "reportingId": 98,
      "protocol": [
        "HTTP"
      ],
      "name": "980",
      "id": 980,
      "originId": 98,
      "aliases": [
        "www.example.com",
        "www.example.net",
        "www.foo.coom",
        "www.foo.net"
      ]
    }
  ],
  "matchGroups": [
    {
      "id": 0,
      "matchRules": [
        {
          "expression": "$id.minor == 98",
          "id": 98,
          "features": {
            "aeOverride": {
              "value": {
                "id": 980
              }
            },
            "logExtras": {
              "value": "r:$http_range"
            },
            "dct": {
              "value": {
                "id": 980
              }
            }
          }
        }
      ]
    },
    {
      "id": 98,
      "matchRules": [
        {
          "expression": "/lua",
          "id": 980,
          "features": {
            "caseSensitive": {
              "value": false
            },
            "qshMode": {
              "value": true
            },
            "lua": {
              "value": {
                "idList": [
                  983
                ]
              }
            }
          }
        },
        {
          "expression": "/testheaders",
          "id": 981,
          "features": {
            "caseSensitive": {
              "value": false
            },
            "respHeaders": {
              "value": {
                "idList": [
                  980
                ]
              }
            },
            "qshMode": {
              "value": true
            }
          }
        },
        {
          "expression": "^/testtoken",
          "id": 982,
          "features": {
            "caseSensitive": {
              "value": false
            },
            "token": {
              "value": {
                "ipWhitelist": [
                  980
                ],
                "denial": {
                  "action": "Error"
                },
                "id": 980
              }
            },
            "qshMode": {
              "value": true
            },
          }
        },
        {
          "expression": "/test_resp_hdr_cors",
          "id": 983,
          "features": {
            "caseSensitive": {
              "value": false
            },
            "respHeaders": {
              "value": {
                "idList": [
                  980
                ]
              }
            },
            "qshMode": {
              "value": true
            }
          }
        },
        {
          "expression": "/test_ip_blacklist",
          "id": 984,
          "features": {
            "caseSensitive": {
              "value": false
            },
            "ipBlock": {
              "value": {
                "denial": {
                  "action": "Error"
                },
                "actionType": "Deny",
                "idList": [
                  980
                ]
              }
            },
            "qshMode": {
              "value": true
            }
          }
        },
        {
         "expression": "/test_native_geo",
          "id": 9825,
          "features": {
            "caseSensitive": {
              "value": false
            },
            "geo": {
              "value": {
                "ipWhitelist": [
                  980
                ],
                "matchAnonymizers": true,
                "denial": {
                  "action": "Error"
                },
                "actionType": "Deny",
                "idList": [
                  980
                ]
              }
            },
            "qshMode": {
              "value": true
            }
          }
        }
      ]
    }
  ],
  "tokenDefinitions": [
    {
      "queryParameterControl": "Exclude",
      "hash": "hash",
      "name": "98-token",
      "sharedSecrets": [
        "test"
      ],
      "queryParameterNames": [
        ""
      ],
      "datePreference": "EPOCH",
      "nva": "",
      "id": 980,
      "nvb": ""
    }
  ],
  "geoDefinitions": [
    {
      "countryCodes": [
        "US"
      ],
      "id": 980,
      "name": "98-us"
    }
  ],
  "luaDefinitions": [
    {
      "heavy": false,
      "direction": "Request",
      "id": 980,
      "name": "98-add_header1_req",
      "script": "local leapfrog = leapfrog\nlocal fphttp = fphttp\nlocal fputil = fputil\n\nleapfrog.audit(1, \"cos 98 executing add_header1_req\")\n\nlocal header1 = \"custom-resp-hdr1: value1a\"\n\nleapfrog.set_note(txn,\"myheaders\", fputil.base64encode(header1))"
    }
  ],
  "responseHeaderDefinitions": [
    {
      "statusRedir": false,
      "name": "98-group1",
      "statusSuccess": false,
      "statusCliError": true,
      "headers": [
        {
          "name": "Access-Control-Allow-Origin",
          "value": "*"
        }
      ],
      "statusAll": false,
      "statusSrvError": false,
      "id": 980,
      "customStatusCodes": []
    }
  ],
  "defaults": {
    "logEndpoints": []
  },
  "logEndpoints": [],
  "requestHeaderDefinitions": [
    {
      "headers": [
        {
          "name": "test-req-header",
          "value": "test-value"
        }
      ],
      "id": 980,
      "name": "98-group1"
    }
  ],
  "ipDefinitions": [
    {
      "id": 980,
      "name": "98-whitelist",
      "ipCidrs": [
        "192.168.1.25"
      ]
    }
  ],
  "dctDefinitions": [
    {
      "fileExtensions": [
        "m3u8"
      ],
      "id": 980,
      "isDefault": false,
      "name": "98-0"
    }
  ],
  "aeOverrideDefinitions": [
    {
      "compressionTypes": [
        "gzip"
      ],
      "fileExtensions": [
        "m3u8"
      ],
      "id": 980,
      "isDefault": false,
      "name": "98-0"
    }
  ]
}