aiida.repository package

Module with resources dealing with the file repository.

class aiida.repository.AbstractRepositoryBackend

Bases: object

Class that defines the abstract interface for an object repository.

The repository backend only deals with raw bytes, both when creating new objects as well as when returning a stream or the content of an existing object. The encoding and decoding of the byte content should be done by the client upstream. The file repository backend is also not expected to keep any kind of file hierarchy but must be assumed to be a simple flat data store. When files are created in the file object repository, the implementation will return a string-based key with which the content of the stored object can be addressed. This key is guaranteed to be unique and persistent. Persisting the key or mapping it onto a virtual file hierarchy is again up to the client upstream.

__abstractmethods__ = frozenset({'_put_object_from_filelike', 'erase', 'has_object', 'initialise', 'is_initialised', 'uuid'})

__dict__ = mappingproxy({'__module__': 'aiida.repository.backend.abstract', '__doc__': 'Class that defines the abstract interface for an object repository.\n\n The repository backend only deals with raw bytes, both when creating new objects as well as when returning a stream\n or the content of an existing object. The encoding and decoding of the byte content should be done by the client\n upstream. The file repository backend is also not expected to keep any kind of file hierarchy but must be assumed\n to be a simple flat data store. When files are created in the file object repository, the implementation will return\n a string-based key with which the content of the stored object can be addressed. This key is guaranteed to be unique\n and persistent. Persisting the key or mapping it onto a virtual file hierarchy is again up to the client upstream.\n ', 'uuid': <abc.abstractproperty object>, 'initialise': <function AbstractRepositoryBackend.initialise>, 'is_initialised': <abc.abstractproperty object>, 'erase': <function AbstractRepositoryBackend.erase>, 'is_readable_byte_stream': <staticmethod object>, 'put_object_from_filelike': <function AbstractRepositoryBackend.put_object_from_filelike>, '_put_object_from_filelike': <function AbstractRepositoryBackend._put_object_from_filelike>, 'put_object_from_file': <function AbstractRepositoryBackend.put_object_from_file>, 'has_object': <function AbstractRepositoryBackend.has_object>, 'open': <function AbstractRepositoryBackend.open>, 'get_object_content': <function AbstractRepositoryBackend.get_object_content>, 'get_object_hash': <function AbstractRepositoryBackend.get_object_hash>, 'delete_object': <function AbstractRepositoryBackend.delete_object>, '__dict__': <attribute '__dict__' of 'AbstractRepositoryBackend' objects>, '__weakref__': <attribute '__weakref__' of 'AbstractRepositoryBackend' objects>, '__abstractmethods__': frozenset({'_put_object_from_filelike', 'initialise', 'erase', 'is_initialised', 'has_object', 'uuid'}), '_abc_impl': <_abc_data object>})

__module__ = 'aiida.repository.backend.abstract'

__weakref__

list of weak references to the object (if defined)

_abc_impl = <_abc_data object>

abstract _put_object_from_filelike(handle: BinaryIO) → str

delete_object(key: str)

Delete the object from the repository.

Parameters

key – fully qualified identifier for the object within the repository.

Raises

• FileNotFoundError – if the file does not exist.

• OSError – if the file could not be deleted.

abstract erase() → None

Delete the repository itself and all its contents.

Note

This should not merely delete the contents of the repository but any resources it created. For example, if the repository is essentially a folder on disk, the folder itself should also be deleted, not just its contents.

get_object_content(key: str) → bytes

Return the content of a object identified by key.

Parameters

key – fully qualified identifier for the object within the repository.

Raises

• FileNotFoundError – if the file does not exist.

• OSError – if the file could not be opened.

get_object_hash(key: str) → str

Return the SHA-256 hash of an object stored under the given key.

Important

A SHA-256 hash should always be returned, to ensure consistency across different repository implementations.

Parameters

key – fully qualified identifier for the object within the repository.

Raises

• FileNotFoundError – if the file does not exist.

• OSError – if the file could not be opened.

abstract has_object(key: str) → bool

Return whether the repository has an object with the given key.

Parameters

key – fully qualified identifier for the object within the repository.

Returns

True if the object exists, False otherwise.

abstract initialise(**kwargs) → None

Initialise the repository if it hasn’t already been initialised.

Parameters

kwargs – parameters for the initialisation.

abstract property is_initialised

Return whether the repository has been initialised.

static is_readable_byte_stream(handle) → bool

open(key: str) → Iterator[BinaryIO]

Open a file handle to an object stored under the given key.

Note

this should only be used to open a handle to read an existing file. To write a new file use the method put_object_from_filelike instead.

Parameters

key – fully qualified identifier for the object within the repository.

Returns

yield a byte stream object.

Raises

• FileNotFoundError – if the file does not exist.

• OSError – if the file could not be opened.

put_object_from_file(filepath: Union[str, pathlib.Path]) → str

Store a new object with contents of the file located at filepath on this file system.

Parameters

filepath – absolute path of file whose contents to copy to the repository.

Returns

the generated fully qualified identifier for the object within the repository.

Raises

TypeError – if the handle is not a byte stream.

put_object_from_filelike(handle: BinaryIO) → str

Store the byte contents of a file in the repository.

Parameters

handle – filelike object with the byte content to be stored.

Returns

the generated fully qualified identifier for the object within the repository.

Raises

TypeError – if the handle is not a byte stream.

abstract property uuid

Return the unique identifier of the repository.

class aiida.repository.DiskObjectStoreRepositoryBackend(container)

Bases: aiida.repository.backend.abstract.AbstractRepositoryBackend

Implementation of the AbstractRepositoryBackend using the disk-object-store as the backend.

__abstractmethods__ = frozenset({})

__init__(container)

Initialize self. See help(type(self)) for accurate signature.

__module__ = 'aiida.repository.backend.disk_object_store'

__str__() → str

Return the string representation of this repository.

_abc_impl = <_abc_data object>

_put_object_from_filelike(handle: BinaryIO) → str

Store the byte contents of a file in the repository.

Parameters

handle – filelike object with the byte content to be stored.

Returns

the generated fully qualified identifier for the object within the repository.

Raises

TypeError – if the handle is not a byte stream.

property container

delete_object(key: str)

Delete the object from the repository.

Parameters

key – fully qualified identifier for the object within the repository.

Raises

• FileNotFoundError – if the file does not exist.

• OSError – if the file could not be deleted.

erase()

Delete the repository itself and all its contents.

get_object_hash(key: str) → str

Return the SHA-256 hash of an object stored under the given key.

Important

A SHA-256 hash should always be returned, to ensure consistency across different repository implementations.

Parameters

key – fully qualified identifier for the object within the repository.

Raises

FileNotFoundError – if the file does not exist.

has_object(key: str) → bool

Return whether the repository has an object with the given key.

Parameters

key – fully qualified identifier for the object within the repository.

Returns

True if the object exists, False otherwise.

initialise(**kwargs) → None

Initialise the repository if it hasn’t already been initialised.

Parameters

kwargs – parameters for the initialisation.

property is_initialised

Return whether the repository has been initialised.

open(key: str) → Iterator[BinaryIO]

Open a file handle to an object stored under the given key.

Note

this should only be used to open a handle to read an existing file. To write a new file use the method put_object_from_filelike instead.

Parameters

key – fully qualified identifier for the object within the repository.

Returns

yield a byte stream object.

Raises

• FileNotFoundError – if the file does not exist.

• OSError – if the file could not be opened.

property uuid

Return the unique identifier of the repository.

class aiida.repository.File(name: str = '', file_type: aiida.repository.common.FileType = <FileType.DIRECTORY: 0>, key: Optional[str] = None, objects: Dict[str, File] = None)

Bases: object

Data class representing a file object.

__dict__ = mappingproxy({'__module__': 'aiida.repository.common', '__doc__': 'Data class representing a file object.', '__init__': <function File.__init__>, 'from_serialized': <classmethod object>, 'serialize': <function File.serialize>, 'name': <property object>, 'file_type': <property object>, 'key': <property object>, 'objects': <property object>, '__eq__': <function File.__eq__>, '__repr__': <function File.__repr__>, '__dict__': <attribute '__dict__' of 'File' objects>, '__weakref__': <attribute '__weakref__' of 'File' objects>, '__hash__': None})

__eq__(other) → bool

Return whether this instance is equal to another file object instance.

__hash__ = None

__init__(name: str = '', file_type: aiida.repository.common.FileType = <FileType.DIRECTORY: 0>, key: Optional[str] = None, objects: Dict[str, File] = None) → None

Construct a new instance.

Parameters

• name – The final element of the file path

• file_type – Identifies whether the File is a file or a directory

• key – A key to map the file to its contents in the backend repository (file only)

• objects – Mapping of child names to child Files (directory only)

Raises

ValueError – If a key is defined for a directory, or objects are defined for a file

__module__ = 'aiida.repository.common'

__repr__()

Return repr(self).

__weakref__

list of weak references to the object (if defined)

property file_type

Return the file type of the file object.

classmethod from_serialized(serialized: dict, name='') → aiida.repository.common.File

Construct a new instance from a serialized instance.

Parameters

serialized – the serialized instance.

Returns

the reconstructed file object.

property key

Return the key of the file object.

property name

Return the name of the file object.

property objects

Return the objects of the file object.

serialize() → dict

Serialize the metadata into a JSON-serializable format.

Note

the serialization format is optimized to reduce the size in bytes.

Returns

dictionary with the content metadata.

class aiida.repository.FileType(value)

Bases: enum.Enum

Enumeration to represent the type of a file object.

DIRECTORY = 0

FILE = 1

__module__ = 'aiida.repository.common'

class aiida.repository.Repository(backend: aiida.repository.backend.abstract.AbstractRepositoryBackend = None)

Bases: object

File repository.

This class provides an interface to a backend file repository instance, but unlike the backend repository, this class keeps a reference of the virtual file hierarchy. This means that through this interface, a client can create files and directories with a file hierarchy, just as they would on a local file system, except it is completely virtual as the files are stored by the backend which can store them in a completely flat structure. This also means that the internal virtual hierarchy of a Repository instance does not necessarily represent all the files that are stored by repository backend. The repository exposes a mere subset of all the file objects stored in the backend. This is why object deletion is also implemented as a soft delete, by default, where the files are just removed from the internal virtual hierarchy, but not in the actual backend. This is because those objects can be referenced by other instances.

__dict__ = mappingproxy({'__module__': 'aiida.repository.repository', '__doc__': 'File repository.\n\n This class provides an interface to a backend file repository instance, but unlike the backend repository, this\n class keeps a reference of the virtual file hierarchy. This means that through this interface, a client can create\n files and directories with a file hierarchy, just as they would on a local file system, except it is completely\n virtual as the files are stored by the backend which can store them in a completely flat structure. This also means\n that the internal virtual hierarchy of a ``Repository`` instance does not necessarily represent all the files that\n are stored by repository backend. The repository exposes a mere subset of all the file objects stored in the\n backend. This is why object deletion is also implemented as a soft delete, by default, where the files are just\n removed from the internal virtual hierarchy, but not in the actual backend. This is because those objects can be\n referenced by other instances.\n ', '_file_cls': <class 'aiida.repository.common.File'>, '__init__': <function Repository.__init__>, '__str__': <function Repository.__str__>, 'uuid': <property object>, 'initialise': <function Repository.initialise>, 'is_initialised': <property object>, 'from_serialized': <classmethod object>, 'reset': <function Repository.reset>, 'serialize': <function Repository.serialize>, 'hash': <function Repository.hash>, '_pre_process_path': <staticmethod object>, 'backend': <property object>, 'set_backend': <function Repository.set_backend>, '_insert_file': <function Repository._insert_file>, 'create_directory': <function Repository.create_directory>, 'get_file_keys': <function Repository.get_file_keys>, 'get_object': <function Repository.get_object>, 'get_directory': <function Repository.get_directory>, 'get_file': <function Repository.get_file>, 'list_objects': <function Repository.list_objects>, 'list_object_names': <function Repository.list_object_names>, 'put_object_from_filelike': <function Repository.put_object_from_filelike>, 'put_object_from_file': <function Repository.put_object_from_file>, 'put_object_from_tree': <function Repository.put_object_from_tree>, 'is_empty': <function Repository.is_empty>, 'has_object': <function Repository.has_object>, 'open': <function Repository.open>, 'get_object_content': <function Repository.get_object_content>, 'delete_object': <function Repository.delete_object>, 'delete': <function Repository.delete>, 'erase': <function Repository.erase>, 'clone': <function Repository.clone>, 'walk': <function Repository.walk>, '__dict__': <attribute '__dict__' of 'Repository' objects>, '__weakref__': <attribute '__weakref__' of 'Repository' objects>})

__init__(backend: aiida.repository.backend.abstract.AbstractRepositoryBackend = None)

Construct a new instance with empty metadata.

Parameters

backend – instance of repository backend to use to actually store the file objects. By default, an instance of the SandboxRepositoryBackend will be created.

__module__ = 'aiida.repository.repository'

__str__() → str

Return the string representation of this repository.

__weakref__

list of weak references to the object (if defined)

_file_cls

alias of aiida.repository.common.File

_insert_file(path: pathlib.PurePosixPath, key: str) → None

Insert a new file object in the object mapping.

Note

this assumes the path is a valid relative path, so should be checked by the caller.

Parameters

• path – the relative path where to store the object in the repository.

• key – fully qualified identifier for the object within the repository.

static _pre_process_path(path: Union[str, pathlib.PurePosixPath] = None) → pathlib.PurePosixPath

Validate and convert the path to instance of pathlib.PurePosixPath.

This should be called by every method of this class before doing anything, such that it can safely assume that the path is a pathlib.PurePosixPath object, which makes path manipulation a lot easier.

Parameters

path – the path as a pathlib.PurePosixPath object or None.

Raises

TypeError – if the type of path was not a str nor a pathlib.PurePosixPath instance.

property backend

Return the current repository backend.

Returns

the repository backend.

clone(source: aiida.repository.repository.Repository) → None

Clone the contents of another repository instance.

create_directory(path: Union[str, pathlib.PurePosixPath]) → aiida.repository.common.File

Create a new directory with the given path.

Parameters

path – the relative path of the directory.

Returns

the created directory.

Raises

TypeError – if the path is not a string or Path, or is an absolute path.

delete() → None

Delete the repository.

Important

This will not just delete the contents of the repository but also the repository itself and all of its assets. For example, if the repository is stored inside a folder on disk, the folder may be deleted.

delete_object(path: Union[str, pathlib.PurePosixPath], hard_delete: bool = False) → None

Soft delete the object from the repository.

Note

can only delete file objects, but not directories.

Parameters

• path – the relative path of the object within the repository.

• hard_delete – when true, not only remove the file from the internal mapping but also call through to the delete_object method of the actual repository backend.

Raises

• TypeError – if the path is not a string or Path, or is an absolute path.

• FileNotFoundError – if the file does not exist.

• IsADirectoryError – if the object is a directory and not a file.

• OSError – if the file could not be deleted.

erase() → None

Delete all objects from the repository.

classmethod from_serialized(backend: aiida.repository.backend.abstract.AbstractRepositoryBackend, serialized: Dict[str, Any]) → aiida.repository.repository.Repository

Construct an instance where the metadata is initialized from the serialized content.

Parameters

backend – instance of repository backend to use to actually store the file objects.

get_directory(path: Union[str, pathlib.PurePosixPath] = None) → aiida.repository.common.File

Return the directory object at the given path.

Parameters

path – the relative path of the directory.

Returns

the File representing the object located at the given relative path.

Raises

• TypeError – if the path is not a string or Path, or is an absolute path.

• FileNotFoundError – if no object exists for the given path.

• NotADirectoryError – if the object at the given path is not a directory.

get_file(path: Union[str, pathlib.PurePosixPath]) → aiida.repository.common.File

Return the file object at the given path.

Parameters

path – the relative path of the file object.

Returns

the File representing the object located at the given relative path.

Raises

• TypeError – if the path is not a string or Path, or is an absolute path.

• FileNotFoundError – if no object exists for the given path.

• IsADirectoryError – if the object at the given path is not a directory.

get_file_keys() → List[str]

Return the keys of all file objects contained within this repository.

Returns

list of keys, which map a file to its content in the backend repository.

get_object(path: Union[str, pathlib.PurePosixPath] = None) → aiida.repository.common.File

Return the object at the given path.

Parameters

path – the relative path where to store the object in the repository.

Returns

the File representing the object located at the given relative path.

Raises

• TypeError – if the path is not a string or Path, or is an absolute path.

• FileNotFoundError – if no object exists for the given path.

get_object_content(path: Union[str, pathlib.PurePosixPath]) → bytes

Return the content of a object identified by path.

Parameters

path – the relative path of the object within the repository.

Raises

• TypeError – if the path is not a string or Path, or is an absolute path.

• FileNotFoundError – if the file does not exist.

• IsADirectoryError – if the object is a directory and not a file.

• OSError – if the file could not be opened.

has_object(path: Union[str, pathlib.PurePosixPath]) → bool

Return whether the repository has an object with the given path.

Parameters

path – the relative path of the object within the repository.

Returns

True if the object exists, False otherwise.

Raises

TypeError – if the path is not a string or Path, or is an absolute path.

hash() → str

Generate a hash of the repository’s contents.

Warning

this will read the content of all file objects contained within the virtual hierarchy into memory.

Returns

the hash representing the contents of the repository.

initialise(**kwargs: Any) → None

Initialise the repository if it hasn’t already been initialised.

Parameters

kwargs – keyword argument that will be passed to the initialise call of the backend.

is_empty() → bool

Return whether the repository is empty.

Returns

True if the repository contains no file objects.

property is_initialised

Return whether the repository has been initialised.

list_object_names(path: Union[str, pathlib.PurePosixPath] = None) → List[str]

Return a sorted list of the object names contained in this repository, optionally in the given sub directory.

Parameters

path – the relative path of the directory.

Returns

a list of File named tuples representing the objects present in directory with the given path.

Raises

• TypeError – if the path is not a string or Path, or is an absolute path.

• FileNotFoundError – if no object exists for the given path.

• NotADirectoryError – if the object at the given path is not a directory.

list_objects(path: Union[str, pathlib.PurePosixPath] = None) → List[aiida.repository.common.File]

Return a list of the objects contained in this repository sorted by name, optionally in given sub directory.

Parameters

path – the relative path of the directory.

Returns

a list of File named tuples representing the objects present in directory with the given path.

Raises

• TypeError – if the path is not a string or Path, or is an absolute path.

• FileNotFoundError – if no object exists for the given path.

• NotADirectoryError – if the object at the given path is not a directory.

open(path: Union[str, pathlib.PurePosixPath]) → Iterator[BinaryIO]

Open a file handle to an object stored under the given path.

Note

this should only be used to open a handle to read an existing file. To write a new file use the method put_object_from_filelike instead.

Parameters

path – the relative path of the object within the repository.

Returns

yield a byte stream object.

Raises

• TypeError – if the path is not a string or Path, or is an absolute path.

• FileNotFoundError – if the file does not exist.

• IsADirectoryError – if the object is a directory and not a file.

• OSError – if the file could not be opened.

put_object_from_file(filepath: Union[str, pathlib.PurePosixPath], path: Union[str, pathlib.PurePosixPath]) → None

Store a new object under path with contents of the file located at filepath on the local file system.

Parameters

• filepath – absolute path of file whose contents to copy to the repository

• path – the relative path where to store the object in the repository.

Raises

TypeError – if the path is not a string and relative path, or the handle is not a byte stream.

put_object_from_filelike(handle: BinaryIO, path: Union[str, pathlib.PurePosixPath]) → None

Store the byte contents of a file in the repository.

Parameters

• handle – filelike object with the byte content to be stored.

• path – the relative path where to store the object in the repository.

Raises

TypeError – if the path is not a string or Path, or is an absolute path.

put_object_from_tree(filepath: Union[str, pathlib.PurePosixPath], path: Union[str, pathlib.PurePosixPath] = None) → None

Store the entire contents of filepath on the local file system in the repository with under given path.

Parameters

• filepath – absolute path of the directory whose contents to copy to the repository.

• path – the relative path where to store the objects in the repository.

Raises

• TypeError – if the filepath is not a string or Path, or is a relative path.

• TypeError – if the path is not a string or Path, or is an absolute path.

reset() → None

serialize() → Dict[str, Any]

Serialize the metadata into a JSON-serializable format.

Returns

dictionary with the content metadata.

set_backend(backend: aiida.repository.backend.abstract.AbstractRepositoryBackend) → None

Set the backend for this repository.

Parameters

backend – the repository backend.

Raises

TypeError – if the type of the backend is invalid.

property uuid

Return the unique identifier of the repository or None if it doesn’t have one.

walk(path: Union[str, pathlib.PurePosixPath] = None) → Iterable[Tuple[pathlib.PurePosixPath, List[str], List[str]]]

Walk over the directories and files contained within this repository.

Note

the order of the dirname and filename lists that are returned is not necessarily sorted. This is in line with the os.walk implementation where the order depends on the underlying file system used.

Parameters

path – the relative path of the directory within the repository whose contents to walk.

Returns

tuples of root, dirnames and filenames just like os.walk, with the exception that the root path is always relative with respect to the repository root, instead of an absolute path and it is an instance of pathlib.PurePosixPath instead of a normal string

class aiida.repository.SandboxRepositoryBackend

Bases: aiida.repository.backend.abstract.AbstractRepositoryBackend

Implementation of the AbstractRepositoryBackend using a sandbox folder on disk as the backend.

__abstractmethods__ = frozenset({})

__del__()

Delete the entire sandbox folder if it was instantiated and still exists.

__init__()

Initialize self. See help(type(self)) for accurate signature.

__module__ = 'aiida.repository.backend.sandbox'

__str__() → str

Return the string representation of this repository.

_abc_impl = <_abc_data object>

_put_object_from_filelike(handle: BinaryIO) → str

Store the byte contents of a file in the repository.

Parameters

handle – filelike object with the byte content to be stored.

Returns

the generated fully qualified identifier for the object within the repository.

Raises

TypeError – if the handle is not a byte stream.

delete_object(key: str)

Delete the object from the repository.

Parameters

key – fully qualified identifier for the object within the repository.

Raises

• FileNotFoundError – if the file does not exist.

• OSError – if the file could not be deleted.

erase()

Delete the repository itself and all its contents.

has_object(key: str) → bool

Return whether the repository has an object with the given key.

Parameters

key – fully qualified identifier for the object within the repository.

Returns

True if the object exists, False otherwise.

initialise(**kwargs) → None

Initialise the repository if it hasn’t already been initialised.

Parameters

kwargs – parameters for the initialisation.

property is_initialised

Return whether the repository has been initialised.

open(key: str) → Iterator[BinaryIO]

Open a file handle to an object stored under the given key.

Note

this should only be used to open a handle to read an existing file. To write a new file use the method put_object_from_filelike instead.

Parameters

key – fully qualified identifier for the object within the repository.

Returns

yield a byte stream object.

Raises

• FileNotFoundError – if the file does not exist.

• OSError – if the file could not be opened.

property sandbox

Return the sandbox instance of this repository.

property uuid

Return the unique identifier of the repository.

Note

A sandbox folder does not have the concept of a unique identifier and so always returns None.

Subpackages

• aiida.repository.backend package
  • Submodules

Submodules

Module with resources common to the repository.

class aiida.repository.common.File(name: str = '', file_type: aiida.repository.common.FileType = <FileType.DIRECTORY: 0>, key: Optional[str] = None, objects: Dict[str, File] = None)

Bases: object

Data class representing a file object.

__dict__ = mappingproxy({'__module__': 'aiida.repository.common', '__doc__': 'Data class representing a file object.', '__init__': <function File.__init__>, 'from_serialized': <classmethod object>, 'serialize': <function File.serialize>, 'name': <property object>, 'file_type': <property object>, 'key': <property object>, 'objects': <property object>, '__eq__': <function File.__eq__>, '__repr__': <function File.__repr__>, '__dict__': <attribute '__dict__' of 'File' objects>, '__weakref__': <attribute '__weakref__' of 'File' objects>, '__hash__': None})

__eq__(other) → bool

Return whether this instance is equal to another file object instance.

__hash__ = None

__init__(name: str = '', file_type: aiida.repository.common.FileType = <FileType.DIRECTORY: 0>, key: Optional[str] = None, objects: Dict[str, File] = None) → None

Construct a new instance.

Parameters

• name – The final element of the file path

• file_type – Identifies whether the File is a file or a directory

• key – A key to map the file to its contents in the backend repository (file only)

• objects – Mapping of child names to child Files (directory only)

Raises

ValueError – If a key is defined for a directory, or objects are defined for a file

__module__ = 'aiida.repository.common'

__repr__()

Return repr(self).

__weakref__

list of weak references to the object (if defined)

property file_type

Return the file type of the file object.

classmethod from_serialized(serialized: dict, name='') → aiida.repository.common.File

Construct a new instance from a serialized instance.

Parameters

serialized – the serialized instance.

Returns

the reconstructed file object.

property key

Return the key of the file object.

property name

Return the name of the file object.

property objects

Return the objects of the file object.

serialize() → dict

Serialize the metadata into a JSON-serializable format.

Note

the serialization format is optimized to reduce the size in bytes.

Returns

dictionary with the content metadata.

class aiida.repository.common.FileType(value)

Bases: enum.Enum

Enumeration to represent the type of a file object.

DIRECTORY = 0

FILE = 1

__module__ = 'aiida.repository.common'

Module for the implementation of a file repository.

class aiida.repository.repository.Repository(backend: aiida.repository.backend.abstract.AbstractRepositoryBackend = None)

Bases: object

File repository.

This class provides an interface to a backend file repository instance, but unlike the backend repository, this class keeps a reference of the virtual file hierarchy. This means that through this interface, a client can create files and directories with a file hierarchy, just as they would on a local file system, except it is completely virtual as the files are stored by the backend which can store them in a completely flat structure. This also means that the internal virtual hierarchy of a Repository instance does not necessarily represent all the files that are stored by repository backend. The repository exposes a mere subset of all the file objects stored in the backend. This is why object deletion is also implemented as a soft delete, by default, where the files are just removed from the internal virtual hierarchy, but not in the actual backend. This is because those objects can be referenced by other instances.

__dict__ = mappingproxy({'__module__': 'aiida.repository.repository', '__doc__': 'File repository.\n\n This class provides an interface to a backend file repository instance, but unlike the backend repository, this\n class keeps a reference of the virtual file hierarchy. This means that through this interface, a client can create\n files and directories with a file hierarchy, just as they would on a local file system, except it is completely\n virtual as the files are stored by the backend which can store them in a completely flat structure. This also means\n that the internal virtual hierarchy of a ``Repository`` instance does not necessarily represent all the files that\n are stored by repository backend. The repository exposes a mere subset of all the file objects stored in the\n backend. This is why object deletion is also implemented as a soft delete, by default, where the files are just\n removed from the internal virtual hierarchy, but not in the actual backend. This is because those objects can be\n referenced by other instances.\n ', '_file_cls': <class 'aiida.repository.common.File'>, '__init__': <function Repository.__init__>, '__str__': <function Repository.__str__>, 'uuid': <property object>, 'initialise': <function Repository.initialise>, 'is_initialised': <property object>, 'from_serialized': <classmethod object>, 'reset': <function Repository.reset>, 'serialize': <function Repository.serialize>, 'hash': <function Repository.hash>, '_pre_process_path': <staticmethod object>, 'backend': <property object>, 'set_backend': <function Repository.set_backend>, '_insert_file': <function Repository._insert_file>, 'create_directory': <function Repository.create_directory>, 'get_file_keys': <function Repository.get_file_keys>, 'get_object': <function Repository.get_object>, 'get_directory': <function Repository.get_directory>, 'get_file': <function Repository.get_file>, 'list_objects': <function Repository.list_objects>, 'list_object_names': <function Repository.list_object_names>, 'put_object_from_filelike': <function Repository.put_object_from_filelike>, 'put_object_from_file': <function Repository.put_object_from_file>, 'put_object_from_tree': <function Repository.put_object_from_tree>, 'is_empty': <function Repository.is_empty>, 'has_object': <function Repository.has_object>, 'open': <function Repository.open>, 'get_object_content': <function Repository.get_object_content>, 'delete_object': <function Repository.delete_object>, 'delete': <function Repository.delete>, 'erase': <function Repository.erase>, 'clone': <function Repository.clone>, 'walk': <function Repository.walk>, '__dict__': <attribute '__dict__' of 'Repository' objects>, '__weakref__': <attribute '__weakref__' of 'Repository' objects>})

__init__(backend: aiida.repository.backend.abstract.AbstractRepositoryBackend = None)

Construct a new instance with empty metadata.

Parameters

backend – instance of repository backend to use to actually store the file objects. By default, an instance of the SandboxRepositoryBackend will be created.

__module__ = 'aiida.repository.repository'

__str__() → str

Return the string representation of this repository.

__weakref__

list of weak references to the object (if defined)

_file_cls

alias of aiida.repository.common.File

_insert_file(path: pathlib.PurePosixPath, key: str) → None

Insert a new file object in the object mapping.

Note

this assumes the path is a valid relative path, so should be checked by the caller.

Parameters

• path – the relative path where to store the object in the repository.

• key – fully qualified identifier for the object within the repository.

static _pre_process_path(path: Union[str, pathlib.PurePosixPath] = None) → pathlib.PurePosixPath

Validate and convert the path to instance of pathlib.PurePosixPath.

This should be called by every method of this class before doing anything, such that it can safely assume that the path is a pathlib.PurePosixPath object, which makes path manipulation a lot easier.

Parameters

path – the path as a pathlib.PurePosixPath object or None.

Raises

TypeError – if the type of path was not a str nor a pathlib.PurePosixPath instance.

property backend

Return the current repository backend.

Returns

the repository backend.

clone(source: aiida.repository.repository.Repository) → None

Clone the contents of another repository instance.

create_directory(path: Union[str, pathlib.PurePosixPath]) → aiida.repository.common.File

Create a new directory with the given path.

Parameters

path – the relative path of the directory.

Returns

the created directory.

Raises

TypeError – if the path is not a string or Path, or is an absolute path.

delete() → None

Delete the repository.

Important

This will not just delete the contents of the repository but also the repository itself and all of its assets. For example, if the repository is stored inside a folder on disk, the folder may be deleted.

delete_object(path: Union[str, pathlib.PurePosixPath], hard_delete: bool = False) → None

Soft delete the object from the repository.

Note

can only delete file objects, but not directories.

Parameters

• path – the relative path of the object within the repository.

• hard_delete – when true, not only remove the file from the internal mapping but also call through to the delete_object method of the actual repository backend.

Raises

• TypeError – if the path is not a string or Path, or is an absolute path.

• FileNotFoundError – if the file does not exist.

• IsADirectoryError – if the object is a directory and not a file.

• OSError – if the file could not be deleted.

erase() → None

Delete all objects from the repository.

classmethod from_serialized(backend: aiida.repository.backend.abstract.AbstractRepositoryBackend, serialized: Dict[str, Any]) → aiida.repository.repository.Repository

Construct an instance where the metadata is initialized from the serialized content.

Parameters

backend – instance of repository backend to use to actually store the file objects.

get_directory(path: Union[str, pathlib.PurePosixPath] = None) → aiida.repository.common.File

Return the directory object at the given path.

Parameters

path – the relative path of the directory.

Returns

the File representing the object located at the given relative path.

Raises

• TypeError – if the path is not a string or Path, or is an absolute path.

• FileNotFoundError – if no object exists for the given path.

• NotADirectoryError – if the object at the given path is not a directory.

get_file(path: Union[str, pathlib.PurePosixPath]) → aiida.repository.common.File

Return the file object at the given path.

Parameters

path – the relative path of the file object.

Returns

the File representing the object located at the given relative path.

Raises

• TypeError – if the path is not a string or Path, or is an absolute path.

• FileNotFoundError – if no object exists for the given path.

• IsADirectoryError – if the object at the given path is not a directory.

get_file_keys() → List[str]

Return the keys of all file objects contained within this repository.

Returns

list of keys, which map a file to its content in the backend repository.

get_object(path: Union[str, pathlib.PurePosixPath] = None) → aiida.repository.common.File

Return the object at the given path.

Parameters

path – the relative path where to store the object in the repository.

Returns

the File representing the object located at the given relative path.

Raises

• TypeError – if the path is not a string or Path, or is an absolute path.

• FileNotFoundError – if no object exists for the given path.

get_object_content(path: Union[str, pathlib.PurePosixPath]) → bytes

Return the content of a object identified by path.

Parameters

path – the relative path of the object within the repository.

Raises

• TypeError – if the path is not a string or Path, or is an absolute path.

• FileNotFoundError – if the file does not exist.

• IsADirectoryError – if the object is a directory and not a file.

• OSError – if the file could not be opened.

has_object(path: Union[str, pathlib.PurePosixPath]) → bool

Return whether the repository has an object with the given path.

Parameters

path – the relative path of the object within the repository.

Returns

True if the object exists, False otherwise.

Raises

TypeError – if the path is not a string or Path, or is an absolute path.

hash() → str

Generate a hash of the repository’s contents.

Warning

this will read the content of all file objects contained within the virtual hierarchy into memory.

Returns

the hash representing the contents of the repository.

initialise(**kwargs: Any) → None

Initialise the repository if it hasn’t already been initialised.

Parameters

kwargs – keyword argument that will be passed to the initialise call of the backend.

is_empty() → bool

Return whether the repository is empty.

Returns

True if the repository contains no file objects.

property is_initialised

Return whether the repository has been initialised.

list_object_names(path: Union[str, pathlib.PurePosixPath] = None) → List[str]

Return a sorted list of the object names contained in this repository, optionally in the given sub directory.

Parameters

path – the relative path of the directory.

Returns

a list of File named tuples representing the objects present in directory with the given path.

Raises

• TypeError – if the path is not a string or Path, or is an absolute path.

• FileNotFoundError – if no object exists for the given path.

• NotADirectoryError – if the object at the given path is not a directory.

list_objects(path: Union[str, pathlib.PurePosixPath] = None) → List[aiida.repository.common.File]

Return a list of the objects contained in this repository sorted by name, optionally in given sub directory.

Parameters

path – the relative path of the directory.

Returns

a list of File named tuples representing the objects present in directory with the given path.

Raises

• TypeError – if the path is not a string or Path, or is an absolute path.

• FileNotFoundError – if no object exists for the given path.

• NotADirectoryError – if the object at the given path is not a directory.

open(path: Union[str, pathlib.PurePosixPath]) → Iterator[BinaryIO]

Open a file handle to an object stored under the given path.

Note

this should only be used to open a handle to read an existing file. To write a new file use the method put_object_from_filelike instead.

Parameters

path – the relative path of the object within the repository.

Returns

yield a byte stream object.

Raises

• TypeError – if the path is not a string or Path, or is an absolute path.

• FileNotFoundError – if the file does not exist.

• IsADirectoryError – if the object is a directory and not a file.

• OSError – if the file could not be opened.

put_object_from_file(filepath: Union[str, pathlib.PurePosixPath], path: Union[str, pathlib.PurePosixPath]) → None

Store a new object under path with contents of the file located at filepath on the local file system.

Parameters

• filepath – absolute path of file whose contents to copy to the repository

• path – the relative path where to store the object in the repository.

Raises

TypeError – if the path is not a string and relative path, or the handle is not a byte stream.

put_object_from_filelike(handle: BinaryIO, path: Union[str, pathlib.PurePosixPath]) → None

Store the byte contents of a file in the repository.

Parameters

• handle – filelike object with the byte content to be stored.

• path – the relative path where to store the object in the repository.

Raises

TypeError – if the path is not a string or Path, or is an absolute path.

put_object_from_tree(filepath: Union[str, pathlib.PurePosixPath], path: Union[str, pathlib.PurePosixPath] = None) → None

Store the entire contents of filepath on the local file system in the repository with under given path.

Parameters

• filepath – absolute path of the directory whose contents to copy to the repository.

• path – the relative path where to store the objects in the repository.

Raises

• TypeError – if the filepath is not a string or Path, or is a relative path.

• TypeError – if the path is not a string or Path, or is an absolute path.

reset() → None

serialize() → Dict[str, Any]

Serialize the metadata into a JSON-serializable format.

Returns

dictionary with the content metadata.

set_backend(backend: aiida.repository.backend.abstract.AbstractRepositoryBackend) → None

Set the backend for this repository.

Parameters

backend – the repository backend.

Raises

TypeError – if the type of the backend is invalid.

property uuid

Return the unique identifier of the repository or None if it doesn’t have one.

walk(path: Union[str, pathlib.PurePosixPath] = None) → Iterable[Tuple[pathlib.PurePosixPath, List[str], List[str]]]

Walk over the directories and files contained within this repository.

Note

the order of the dirname and filename lists that are returned is not necessarily sorted. This is in line with the os.walk implementation where the order depends on the underlying file system used.

Parameters

path – the relative path of the directory within the repository whose contents to walk.

Returns

tuples of root, dirnames and filenames just like os.walk, with the exception that the root path is always relative with respect to the repository root, instead of an absolute path and it is an instance of pathlib.PurePosixPath instead of a normal string

aiida.plugins package aiida.repository.backend package