Profiles and constants

A profile can be viewed as a template for tasks, that can be parametrized by constants. Every task has a profile, and the basic thing you'll do when creating a task is to select a profile and adjust some constants.

Let's take some examples:

  • you want to run some Python script of yours on Qarnot's computing platform. After uploading your Python files and input data to a bucket, you will select one of our python profiles, and set in the PYTHON_SCRIPT constant the arguments you want to pass to the python command (e.g --some-opt=some-value).
  • If you want to run a container from the Docker hub, you'll select one of our docker profiles, and use the DOCKER_REPO and DOCKER_TAG constants to select the Docker image and tag to use. DOCKER_CMD will let you specify the command that you want to run in your containers.

Batch and worker profiles

There are two major kinds of profiles that you can choose from: batch profiles, and worker profiles. They differ in what they allow to do, and in the way they are invoiced when one of your instances is terminated before completion.

Termination before completion

Most of the time, Qarnot's computing nodes are located in one of our computing heaters, that are used to provide free domestic heat. We try to pay an equal attention to people we heat and to people that compute on our service, hence we try to meet temperature requests with the minimal possible impact on your computing tasks. Most of the time, this is handled by our scheduler that targets in priority the nodes that need the most heat, and then, during the task's execution, by regulating the frequency of the CPU. Sometimes however, a given computing node needs to completely stop heating. This happens mostly for tasks that run for a very long time. In those rare cases, your instance will be terminated and re-launched on another node. One of the main differences between batch and worker profiles is the way you are invoiced in such a case.

Batch profiles

A batch profile is made to execute a program on a set of inputs (a resource bucket) and produce a set of outputs (a result bucket). Examples of this would be a blender rendering, machine learning on a fixed data set, … With such a kind of task, an instance terminated before completion did not output any result, so it produced no value for you. Hence, it is not invoiced.

For instance, let's take the following situation:

  • you launched a blender rendering (only one instance for the sake of simplicity)
  • it computed 2 hours on a node, then we needed to terminate it
  • it was re-launched on another node, where it completed in 3 hours.

Your task has used 5 hours of computation time on our farm, 3 of which produced valuable output for you. You will only be invoiced 3 hours.

There is a limitation to benefit from such a behaviour: batch tasks do not have network access

Worker profiles

A worker profiles is designed to let you start instances running a worker, that pulls work from and pushes results to sources external to Qarnot's computing service.

For example, you could have several instances of your application that pull work items from an instance of RabbitMQ that you host, then pull associated data form an Amazon S3 bucket, perform some computation, upload results to S3 and notify your RabbitMQ that the task is completed.

Tasks based on a worker profile have full internet access. However, as we cannot distinguish "useful" from "useless" work, everything second of computation time will be invoiced.

Next steps