Developers
menu
menu

Buckets

A bucket is a storage unit, storing a collection of files. In addition to our SDKs, you can use any S3 compatible software to manage your buckets and files.

When connecting to our storage service you should use your email as the Access Key ID, and your token as the Secret Access Key, don't forget to set "storage.qarnot.com" as the REST Endpoint, and to use SSL.

They are used for two purposes in Qarnot's computing API: for storing input data for your tasks, and for storing result data generated by your tasks.

Resources

Every task can have zero, one, or several resource buckets.

The most usual case is to have one bucket in which you upload all the data needed by the computation (e.g: python files and input data, or blender file and textures, …).

However, there are cases where you don't need resources. For instance, some computing nodes fetching work and uploading results to one of you servers don't really need a resource bucket if your application is packaged as a Docker image available on the Docker Hub.

Another possibility is to have a list of several buckets as resources. This will behave the same way a union filesystem (like overlayfs) would, with all files from all buckets being a resource to your task. If two buckets contain files with the same name, the file from the bucket appearing last in the list will override the other.

For example, you can have one bucket containing your input data, and one containing your Python scripts to run.

In any case, the task's resources will be made available to your program in the $TASK_PATH directory (see environment for further details).

Results

If your computation produce result files (e.g: images rendered by Blender, or any other output file that you need), you have to provide a result bucket. Just have your program write those results in $TASK_PATH when it has finished (see environment for further details), and they will be automatically put in the result bucket. You can then download the content of this bucket from the API.

NOTE: resources and results are two use cases for buckets. Still, there is only one kind of objects: buckets. You can very well use the result bucket of some task as a resource for a subsequent task, or the same bucket as resource and result.

Next steps

  • Learn how to select and use the right profile.