Bucket

Module for bucket object.

class qarnot.bucket.Bucket(connection, name, create=True)[source]

Bases: qarnot.storage.Storage

Represents a resource/result bucket.

This class is the interface to manage resources or results from a qarnot.bucket.Bucket.

Note

Paths given as ‘remote’ arguments, (or as path arguments for Bucket.directory()) must be valid unix-like paths.

delete()[source]

Delete the bucket represented by this Bucket.

list_files()[source]

List files in the bucket

Return type:list(S3.ObjectSummary)
Returns:A list of ObjectSummary resources
directory(directory=’’)[source]

List files in a directory of the bucket according to prefix.

Return type:list(S3.ObjectSummary)
Returns:A list of ObjectSummary resources
sync_directory(directory, verbose=False, remote=None)[source]

Synchronize a local directory with the remote buckets.

Parameters:
  • directory (str) – The local directory to use for synchronization
  • verbose (bool) – Print information about synchronization operations
  • remote (str) – path of the directory on remote node (defaults to local)

Warning

Local changes are reflected on the server, a file present on the bucket but not in the local directory will be deleted from the bucket.

A file present in the directory but not in the bucket will be uploaded.

Note

The following parameters are used to determine whether synchronization is required :

  • name
  • size
  • sha1sum
sync_files(files, verbose=False, remote=None)[source]

Synchronize files with the remote buckets.

Parameters:
  • files (dict) – Dictionary of synchronized files
  • verbose (bool) – Print information about synchronization operations
  • remote (str) – path of the directory on remote node (defaults to local)

Dictionary key is the remote file path while value is the local file path.

Warning

Local changes are reflected on the server, a file present on the bucket but not in the local directory will be deleted from the bucket.

A file present in the directory but not in the bucket will be uploaded.

Note

The following parameters are used to determine whether synchronization is required :

  • name
  • size
  • sha1sum
add_file(local_or_file, remote=None)[source]

Add a local file or a Python File on the storage.

Note

You can also use object[remote] = local

Parameters:
  • local_or_file (str or File) – path of the local file or an opened Python File
  • remote (str) – name of the remote file (defaults to local_or_file)
get_all_files(output_dir, progress=None)[source]

Get all files from the storage.

Parameters:
  • output_dir (str) – local directory for the retrieved files.
  • progress (bool or function(float, float, str)) – can be a callback (read,total,filename) or True to display a progress bar
Raises:

Warning

Will override output_dir content.

get_file(remote, local=None, progress=None)[source]

Get a file from the storage. Create needed subfolders.

Parameters:
  • remote (str) – the name of the remote file
  • local (str) – local name of the retrieved file (defaults to remote)
  • progress (bool or function(float, float, str)) – can be a callback (read,total,filename) or True to display a progress bar
Return type:

str

Returns:

The name of the output file.

Raises:

ValueError – no such file

add_directory(local, remote=’’)[source]

Add a directory to the storage. Does not follow symlinks. File hierarchy is preserved.

Parameters:
  • local (str) – path of the local directory to add
  • remote (str) – path of the directory on remote node (defaults to local)
Raises:

IOError – not a valid directory

copy_file(source, dest)[source]

Create a copy of a file

Parameters:
  • source (str) – name of the existing file to duplicate
  • dest (str) – name of the created file
flush()[source]

Ensure all background uploads are complete

update(flush=False)[source]

Update object from remote endpoint

Parameters:flush (bool) – bypass cache
delete_file(remote)[source]

Delete a file from the storage.

Parameters:remote (str) – the name of the remote file
uuid

Bucket identifier

description

Bucket identifier