mirror of https://github.com/TREX-CoE/trexio.git synced 2025-04-27 19:04:45 +02:00

q-posev 2911e91941 make dim an optional argument in read_ functions

if dim is None - the function read all necessary dimensions from the TREXIO file

2021-08-24 12:51:43 +03:00

88 KiB

Raw Blame History

Front end API

Coding conventions
- Memory allocation
Front end
Templates for front end
Fortran helper/wrapper functions

Coding conventions

integer types will be defined using types given in stdint.h
pointers are always initialized to NULL
when memory is freed, the pointer is set to NULL
assert.h should be used extensively
variable names are in lower case
#define constants are in upper case
structs are suffixed by _s
types are suffixed by _t
API calls return trexio_exit_code (except for trexio_open function)

Memory allocation

Memory allocation of structures can be facilitated by using the following macro, which ensures that the size of the allocated object is the same as the size of the data type pointed by the pointer.

#define MALLOC(T) (T*) malloc (sizeof(T))
#define CALLOC(N,T) (T*) calloc ( (N) , sizeof(T) )

When a pointer is freed, it should be set to NULL. This can be facilitated by the use of the following macro:

#define FREE(X) { free(X) ; (X)=NULL; }

The maximum string size for the filenames is 4096 characters.

#define TREXIO_MAX_FILENAME_LENGTH 4096

Front end

All calls to TREXIO are thread-safe. TREXIO front end is modular, which simplifies implementation of new back ends.

Error handling

Macro	Code	Description
`TREXIO_FAILURE`	-1	'Unknown failure'
`TREXIO_SUCCESS`	0	'Success'
`TREXIO_INVALID_ARG_1`	1	'Invalid argument 1'
`TREXIO_INVALID_ARG_2`	2	'Invalid argument 2'
`TREXIO_INVALID_ARG_3`	3	'Invalid argument 3'
`TREXIO_INVALID_ARG_4`	4	'Invalid argument 4'
`TREXIO_INVALID_ARG_5`	5	'Invalid argument 5'
`TREXIO_END`	6	'End of file'
`TREXIO_READONLY`	7	'Read-only file'
`TREXIO_ERRNO`	8	strerror(errno)
`TREXIO_INVALID_ID`	9	'Invalid ID'
`TREXIO_ALLOCATION_FAILED`	10	'Allocation failed'
`TREXIO_HAS_NOT`	11	'Element absent'
`TREXIO_INVALID_NUM`	12	'Invalid dimensions'
`TREXIO_ATTR_ALREADY_EXISTS`	13	'Attribute (num/str) already exists'
`TREXIO_DSET_ALREADY_EXISTS`	14	'Dataset already exists'
`TREXIO_OPEN_ERROR`	15	'Error opening file'
`TREXIO_LOCK_ERROR`	16	'Error locking file'
`TREXIO_UNLOCK_ERROR`	17	'Error unlocking file'
`TREXIO_FILE_ERROR`	18	'Invalid file handle'
`TREXIO_GROUP_READ_ERROR`	19	'Error reading group'
`TREXIO_GROUP_WRITE_ERROR`	20	'Error writing group'
`TREXIO_ELEM_READ_ERROR`	21	'Error reading element'
`TREXIO_ELEM_WRITE_ERROR`	22	'Error writing element'
`TREXIO_UNSAFE_ARRAY_DIM`	23	'Access to memory beyond allocated'
`TREXIO_INVALID_STR_LEN`	30	'Invalid max_str_len'

""" This script generates the C and Fortran constants for the error
 codes from the org-mode table.
"""

result = [ "#+begin_src c :tangle prefix_front.h :exports none" ]
for (text, code,_) in table:
text=text.replace("~","")
result += [ f"#define {text:30s} ((trexio_exit_code) {code:d})" ]
result += [ "#+end_src" ]

result += [ "" ]

result += [ "#+begin_src f90 :tangle prefix_fortran.f90 :exports none" ]
for (text, code,_) in table:
text=text.replace("~","")
result += [ f"   integer(trexio_exit_code), parameter :: {text:30s} = {code:d}" ]
result += [ "#+end_src" ]

result += [ "" ]

result += [ "#+begin_src python :tangle prefix_python.py :exports none" ]
result += [ "# define TREXIO exit codes" ]
for (text, code,_) in table:
text=text.replace("~","")
result += [ f"{text:30s} = {code:d}" ]
result += [ "#+end_src" ]

return '\n'.join(result)

Decoding errors

The trexio_string_of_error converts an exit code into a string. The string is assumed to be large enough to contain the error message (typically 128 characters).

To decode the error messages, trexio_string_of_error converts an error code into a string.

The text strings are extracted from the previous table.

#+NAME:cases

C source code

const char*
trexio_string_of_error (const trexio_exit_code error)
{
switch (error) {
<<cases()>>
}
return "Unknown error";
}

void
trexio_string_of_error_f (const trexio_exit_code error, char result[<<MAX_STRING_LENGTH()>>])
{
strncpy(result, trexio_string_of_error(error), <<MAX_STRING_LENGTH()>>);
}

Fortran interface

interface
 subroutine trexio_string_of_error (error, string) bind(C, name='trexio_string_of_error_f')
   use, intrinsic :: iso_c_binding
   import
   integer (trexio_exit_code), intent(in), value :: error
   character, intent(out) :: string(<<MAX_STRING_LENGTH()>>)
 end subroutine trexio_string_of_error
end interface

Python interface

def string_of_error(return_code: int) -> str:
  """Decode the TREXIO exit code.

  Argument is an integer return code that correspond to one of the TREXIO errors.

  Returns string that contains description of TREXIO ~return_code~.
  """


  try:
      error_str = trexio_string_of_error(trexio_return_code)
  except:
      raise 

  return error_str

Back ends

TREXIO has several back ends:

TREXIO_HDF5 relies on extensive use of the HDF5 library and the associated file format. The HDF5 file is binary and tailored to high-performance I/O. This back end is the default one. HDF5 can be compiled with MPI for parallel I/O. Note, that HDF5 has to be downloaded and installed independently of TREXIO, which may cause some obstacles, especially when the user is not allowed to install external software. The produced files usually have .h5 extension.
TREXIO_TEXT relies on basic file I/O in C, namely fopen, fclose, fprintf, fscanf etc. from stdio.h library. This back end is not optimized for performance. It is supposed to be used for debug purposes or, for example, when the user wants to modify some data manually within the file. This back end is supposed to work "out-of-the-box" since there are no external dependencies, which might be useful for users that do not have access to HDF5 library. The produced files usually have .txt extension.

Additional back ends can be implemented thanks to the modular nature of the front end. This can be achieved by adding a new case (corresponding to the desired back end) in the front-end switch. Then the corresponding back-end has/read/write functions has to be implemented. For example, see the commented lines that correspond to the TREXIO_JSON back end (not implemented yet).

Note: It is important to increment the value of TREXIO_INVALID_BACK_END when a new back end is added. Otherwise, it will not be available.

C

typedef int32_t back_end_t;

#define TREXIO_HDF5             ( (back_end_t) 0 )
#define TREXIO_TEXT             ( (back_end_t) 1 )
/*#define TREXIO_JSON             ( (back_end_t) 2 )*/
#define TREXIO_INVALID_BACK_END ( (back_end_t) 2 )

#define TREXIO_DELIM            "\n"

Fortran

integer(trexio_backend), parameter :: TREXIO_HDF5 = 0
integer(trexio_backend), parameter :: TREXIO_TEXT = 1
! integer(trexio_backend), parameter :: TREXIO_JSON = 2
integer(trexio_backend), parameter :: TREXIO_INVALID_BACK_END = 2

Python

# define TREXIO back ends
TREXIO_HDF5 = 0
TREXIO_TEXT = 1
#TREXIO_JSON = 2
TREXIO_INVALID_BACK_END = 2

Read/write behavior

Every time a reading function is called, the data is read from the disk. If data needs to be cached, this is left to the user of the library.

Writing to TREXIO files is done with transactions (all-or-nothing effect) in a per-group fashion. File writes are attempted by calling explicitly the write (TREXIO_HDF5) or flush (TREXIO_TEXT) function, or when the TREXIO file is closed. If writing is impossible because the data is not valid, no data is written.

The order in which the data is written is not necessarily consistent with the order in which the function calls were made.

The TREXIO files are supposed to be opened by only one program at a time: if the same TREXIO file is modified simultaneously by multiple concurrent programs, the behavior is not specified.

TREXIO file type

trexio_s is the the main type for TREXIO files, visible to the users of the library. This type is kept opaque, and all modifications to the files will be necessarily done through the use of functions, taking such a type as argument.

File creation and opening functions will return TREXIO file handles, namely pointers to trexio_s types. All functions accessing to the TREXIO files will have as a first argument the TREXIO file handle.

typedef struct trexio_s trexio_t;

struct trexio_s {
char              file_name[TREXIO_MAX_FILENAME_LENGTH];
char              version[16];
pthread_mutex_t   thread_lock;
back_end_t        back_end;
char              mode;
bool              one_based;
char              padding[6];   /* Ensures the proper alignment of back ends */
};

Polymorphism of the file handle

Polymorphism of the trexio_t type is handled by ensuring that the corresponding types for all back ends can be safely casted to trexio_t. This is done by making the back-end structs start with struct trexio_s:

struct trexio_back_end_s {
trexio_t     parent ;
/* add below specific back-end data */
}

File opening

trexio_open creates a new TREXIO file or opens existing one.

input parameters:

file_name - string containing file name
mode - character containing open mode (see below)
- 'w' - (write) creates a new file as READWRITE (overwrite existing file)
- 'r' - (read) opens existing file as READONLY
back_end - integer number (or the corresponding global parameter) specifying the back end
- TREXIO_HDF5 - for HDF5 back end (integer alternative: 0)
- TREXIO_TEXT - for TEXT back end (integer alternative: 1)

output: trexio_t file handle

Note: the file_name in TEXT back end actually corresponds to the name of the directory where .txt data files are stored. The actual name of each .txt file corresponds to the group name provided in trex.config (e.g. nucleus.txt for nuclei-related data). These names are populated by the generator.py (i.e. they are hard-coded), which is why the user should tend to avoid renaming the .txt data files.

C

trexio_t*
trexio_open(const char* file_name, const char mode,
  const back_end_t back_end)
{

if (file_name == NULL) return NULL;
if (file_name[0] == '\0') return NULL;
/* Check overflow in file_name */

if (back_end <  0) return NULL;
if (back_end >= TREXIO_INVALID_BACK_END) return NULL;

if (mode != 'r' && mode != 'w') return NULL;

trexio_t* result = NULL;
void* result_tmp = NULL;

/* Allocate data structures */
switch (back_end) {

case TREXIO_TEXT:
 result_tmp = malloc(sizeof(trexio_text_t));
 break;

case TREXIO_HDF5:
 result_tmp = malloc(sizeof(trexio_hdf5_t));
 break;
/*
case TREXIO_JSON:
 result = (trexio_t*) malloc (sizeof(trexio_json_t));
 break;
,*/
}
result = (trexio_t*) result_tmp;

assert (result != NULL);    /* TODO: Error handling */


/* Data for the parent type */

strncpy(result->file_name, file_name, TREXIO_MAX_FILENAME_LENGTH-1);
if (result->file_name[TREXIO_MAX_FILENAME_LENGTH-1] != '\0') {
 free(result);
 return NULL;
}

strncpy(result->version, PACKAGE_VERSION, 16);
if (result->version[15] != '\0') {
 free(result);
 return NULL;
}

result->back_end    = back_end;
result->mode        = mode;
result->one_based   = false;  // Need to be flipped in Fortran interface
int irc = pthread_mutex_init ( &(result->thread_lock), NULL);
if (irc != 0) {
 free(result);
 return NULL;
}
assert (irc == 0);

trexio_exit_code rc;

/* Back end initialization */

rc = TREXIO_OPEN_ERROR;

switch (back_end) {

case TREXIO_TEXT:
 rc = trexio_text_init(result);
 break;

case TREXIO_HDF5:
 rc = trexio_hdf5_init(result);
 break;
/*
case TREXIO_JSON:
 rc = trexio_json_init(result);
 break;
,*/
}

if (rc != TREXIO_SUCCESS) {
 free(result);
 return NULL;
}

rc = trexio_has_metadata_package_version(result);
if (rc == TREXIO_FAILURE) {
 free(result);
 return NULL;
}

if (rc == TREXIO_HAS_NOT) {
 switch (back_end) {

 case TREXIO_TEXT:
   rc = trexio_text_write_metadata_package_version(result, PACKAGE_VERSION);
   break;

 case TREXIO_HDF5:
   rc = trexio_hdf5_write_metadata_package_version(result, PACKAGE_VERSION);
   break;
/*
 case TREXIO_JSON:
   rc = trexio_json_write_metadata_package_version(result, PACKAGE_VERSION);
   break;
,*/
 }
}

if (rc != TREXIO_SUCCESS) {
 free(result);
 return NULL;
}


/* File locking */

rc = TREXIO_LOCK_ERROR;

switch (back_end) {

case TREXIO_TEXT:
 rc = trexio_text_lock(result);
 break;
/* HDF5 v.>=1.10 has file locking activated by default */
case TREXIO_HDF5:
 rc = TREXIO_SUCCESS;
 break;
/*
case TREXIO_JSON:
 rc = trexio_json_lock(result);
 break;
,*/
}

if (rc != TREXIO_SUCCESS) {
 free(result);
 return NULL;
}

return result;
}

Fortran

interface
integer(8) function trexio_open_c (filename, mode, backend) bind(C, name="trexio_open")
  use, intrinsic :: iso_c_binding
  import
  character(kind=c_char), dimension(*)       :: filename
  character, intent(in), value               :: mode
  integer(trexio_backend), intent(in), value :: backend
end function trexio_open_c
end interface

Python

def open(file_name: str, mode: str, back_end: int):
 """Create TREXIO file or open existing one.

 Parameters:

 file_name: str
     Name of the TREXIO file

 mode: str
     One of the currently supported ~open~ modes (e.g. 'w', 'r')

 back_end: int
     One of the currently supported TREXIO back ends (e.g. TREXIO_HDF5, TREXIO_TEXT)    

 Return: 
     SWIG object of type trexio_s.

 Examples:
 >>> from trexio import open as tr_open
 >>> trex_file = tr_open("example.h5", "w", TREXIO_HDF5)    
 """


 try:
     trexio_file = trexio_open(file_name, mode, back_end)
 except:
     raise 

 return trexio_file

Zero-based versus one-based arrays of indices

Because arrays are zero-based in Fortran, we need to set a flag to know if we need to shift by 1 arrays of indices.

trexio_exit_code trexio_set_one_based(trexio_t* file)
{
if (file == NULL)
 return TREXIO_FILE_ERROR;

file->one_based = true;

return TREXIO_SUCCESS;
}

interface
integer function trexio_set_one_based(trex_file) bind(C)
  use, intrinsic :: iso_c_binding
  integer(8), intent(in), value :: trex_file
end function trexio_set_one_based
end interface

File closing

trexio_close closes an existing trexio_t file.

input parameters: file – TREXIO file handle.

output: trexio_exit_code exit code.

C

trexio_exit_code
trexio_close (trexio_t* file)
{

if (file == NULL) return TREXIO_FILE_ERROR;

trexio_exit_code rc = TREXIO_FAILURE;

assert(file->back_end < TREXIO_INVALID_BACK_END);

/* Terminate the back end */
switch (file->back_end) {

case TREXIO_TEXT:
rc = trexio_text_deinit(file);
break;

case TREXIO_HDF5:
rc = trexio_hdf5_deinit(file);
break;
/*
case TREXIO_JSON:
rc = trexio_json_deinit(file);
break;
,*/
}

if (rc != TREXIO_SUCCESS) {
FREE(file);
return rc;
}

/* File unlocking */

rc = TREXIO_UNLOCK_ERROR;

switch (file->back_end) {

case TREXIO_TEXT:
rc = trexio_text_unlock(file);
break;

case TREXIO_HDF5:
rc = TREXIO_SUCCESS;
break;
/*
case TREXIO_JSON:
rc = trexio_json_unlock(file);
break;
,*/
}

/* Terminate front end */

int irc = pthread_mutex_destroy( &(file->thread_lock) );

free(file);

if (irc != 0) return TREXIO_ERRNO;
if (rc != TREXIO_SUCCESS) return rc;

return TREXIO_SUCCESS;
}

Fortran

interface
integer function trexio_close (trex_file) bind(C)
 use, intrinsic :: iso_c_binding
 integer(8), intent(in), value :: trex_file
end function trexio_close
end interface

Python

def close(trexio_file):
 """Close TREXIO file.

 Parameter is a ~trexio_file~ object that has been created by a call to ~open~ function.
 """


 try:
     rc = trexio_close(trexio_file)
     assert rc==TREXIO_SUCCESS
 except AssertionError:
     raise Exception(trexio_string_of_error(rc))

Templates for front end

Description

Consider the following block of trex.json:

{
 "nucleus": {
"num"                : [ "int"  , [                     ] ]
   , "charge"             : [ "float", [ "nucleus.num"       ] ]
   , "coord"              : [ "float", [ "nucleus.num", "3"  ] ]
   , "label"              : [ "str" ,  [ "nucleus.num"       ] ]
 }
}

TREXIO is generated automatically by the generator.py Python script based on the tree-like configuration provided in the trex.json file. Because of that, generalized templates can be implemented and re-used. This approach minimizes the number of bugs as compared with manual copy-paste-modify scheme.

All templates presented below use the $var$ notation to indicate the variable, which will be replaced by the generator.py. Sometimes the upper case is used, i.e. $VAR$ (for example, in #define statements). More detailed description of each variable can be found below:

Template variable	Description	Example
$group$	Name of the group	`nucleus`
$group_num$	Name of the dimensioning variable (scalar)	`nucleus_num`
$group_dset$	Name of the dataset (vector/matrix/tensor)	`nucleus_coord`
$group_dset_rank$	Rank of the dataset	`2`
$group_dset_dim$	Selected dimension of the dataset	`nucleus_num`
$group_dset_dim_list$	All dimensions of the dataset	`{nucleus_num, 3}`
$group_dset_dtype$	Basic type of the dataset (int/float/char)	`float`
$group_dset_h5_dtype$	Type of the dataset in HDF5	`double`
$group_dset_std_dtype_in$	Input type of the dataset in TEXT [fscanf]	`%lf`
$group_dset_std_dtype_out$	Output type of the dataset in TEXT [fprintf]	`%24.16e`
$group_dset_dtype_default$	Default datatype of the dataset [C]	`double/int32_t`
$group_dset_dtype_single$	Single precision datatype of the dataset [C]	`float/int32_t`
$group_dset_dtype_double$	Double precision datatype of the dataset [C]	`double/int64_t`
$default_prec$	Default precision for read/write without suffix [C]	`64/32`
$group_dset_f_dtype_default$	Default datatype of the dataset [Fortran]	`real(8)/integer(4)`
$group_dset_f_dtype_single$	Single precision datatype of the dataset [Fortran]	`real(4)/integer(4)`
$group_dset_f_dtype_double$	Double precision datatype of the dataset [Fortran]	`real(8)/integer(8)`
$group_dset_f_dims$	Dimensions in Fortran	`(:,:)`

Note: parent group name is always added to the child objects upon construction of TREXIO (e.g. num of nucleus group becomes nucleus_num and should be accessed accordingly within TREXIO).

TREXIO generator parses the trex.json file. TREXIO operates with names of variables based on the 1-st (parent group) and 2-nd (child object) levels of trex.json . The parsed data is divided in 2 parts:

Dimensioning variables (contain num in their names). These are always scalar integers.
Datasets. These can be vectors, matrices or tensors. The types are indicated in trex.json. Currently supported types: int, float and strings.

For each of the aforementioned objects, TREXIO provides has, read and write functionality. TREXIO supports I/O with single or double precision for integer and floating point numbers.

Templates for front end has/read/write a single dimensioning variable

This section concerns API calls related to dimensioning variables.

Function name	Description	Precision
`trexio_has_$group_num$`	Check if a dimensioning variable exists in a file	—
`trexio_read_$group_num$`	Read a dimensioning variable	Single
`trexio_write_$group_num$`	Write a dimensioning variable	Single
`trexio_read_$group_num$_32`	Read a dimensioning variable	Single
`trexio_write_$group_num$_32`	Write a dimensioning variable	Single
`trexio_read_$group_num$_64`	Read a dimensioning variable	Double
`trexio_write_$group_num$_64`	Write a dimensioning variable	Double

C templates for front end

The C templates that correspond to each of the abovementioned functions can be found below. First parameter is the TREXIO file handle. Second parameter is the variable to be written/read to/from the TREXIO file (except for trexio_has_ functions). Suffixes _32 and _64 correspond to API calls dealing with single and double precision, respectively. The basic (non-suffixed) API call on dimensioning variables deals with single precision (see Table above).

trexio_exit_code
trexio_read_$group_num$_64 (trexio_t* const file, int64_t* const num)
{
if (file == NULL) return TREXIO_INVALID_ARG_1;

uint64_t u_num = 0;
trexio_exit_code rc = TREXIO_GROUP_READ_ERROR;

switch (file->back_end) {

case TREXIO_TEXT:
rc = trexio_text_read_$group_num$(file, &u_num);
break;

case TREXIO_HDF5:
rc = trexio_hdf5_read_$group_num$(file, &u_num);
break;
/*
case TREXIO_JSON:
rc =trexio_json_read_$group_num$(file, &u_num);
break;
,*/
}

if (rc != TREXIO_SUCCESS) return rc;

*num = (int64_t) u_num;
return TREXIO_SUCCESS;
}

trexio_exit_code
trexio_write_$group_num$_64 (trexio_t* const file, const int64_t num)
{
if (file == NULL) return TREXIO_INVALID_ARG_1;
if (num  <  0   ) return TREXIO_INVALID_ARG_2;
if (trexio_has_$group_num$(file) == TREXIO_SUCCESS) return TREXIO_ATTR_ALREADY_EXISTS;

switch (file->back_end) {

case TREXIO_TEXT:
return trexio_text_write_$group_num$(file, (int64_t) num);
break;

case TREXIO_HDF5:
return trexio_hdf5_write_$group_num$(file, (int64_t) num);
break;
/*
case TREXIO_JSON:
return trexio_json_write_$group_num$(file, (int64_t) num);
break;
,*/
}

return TREXIO_FAILURE;
}

trexio_exit_code
trexio_read_$group_num$_32 (trexio_t* const file, int32_t* const num)
{
if (file == NULL) return TREXIO_INVALID_ARG_1;

uint64_t u_num = 0;
trexio_exit_code rc = TREXIO_GROUP_READ_ERROR;

switch (file->back_end) {

case TREXIO_TEXT:
rc = trexio_text_read_$group_num$(file, &u_num);
break;

case TREXIO_HDF5:
rc = trexio_hdf5_read_$group_num$(file, &u_num);
break;
/*
case TREXIO_JSON:
rc =trexio_json_read_$group_num$(file, &u_num);
break;
,*/
}

if (rc != TREXIO_SUCCESS) return rc;

*num = (int32_t) u_num;
return TREXIO_SUCCESS;
}

trexio_exit_code
trexio_write_$group_num$_32 (trexio_t* const file, const int32_t num)
{

if (file == NULL) return TREXIO_INVALID_ARG_1;
if (num  <  0   ) return TREXIO_INVALID_ARG_2;
if (trexio_has_$group_num$(file) == TREXIO_SUCCESS) return TREXIO_ATTR_ALREADY_EXISTS;

switch (file->back_end) {

case TREXIO_TEXT:
return trexio_text_write_$group_num$(file, (int64_t) num);
break;

case TREXIO_HDF5:
return trexio_hdf5_write_$group_num$(file, (int64_t) num);
break;
/*
case TREXIO_JSON:
return trexio_json_write_$group_num$(file, (int64_t) num);
break;
,*/
}

return TREXIO_FAILURE;
}

trexio_exit_code
trexio_read_$group_num$ (trexio_t* const file, int32_t* const num)
{
return trexio_read_$group_num$_32(file, num);
}

trexio_exit_code
trexio_write_$group_num$ (trexio_t* const file, const int32_t num)
{
return trexio_write_$group_num$_32(file, num);
}

trexio_exit_code
trexio_has_$group_num$ (trexio_t* const file)
{

if (file == NULL) return TREXIO_INVALID_ARG_1;

assert(file->back_end < TREXIO_INVALID_BACK_END);

switch (file->back_end) {

case TREXIO_TEXT:
return trexio_text_has_$group_num$(file);
break;

case TREXIO_HDF5:
return trexio_hdf5_has_$group_num$(file);
break;
/*
case TREXIO_JSON:
return trexio_json_has_$group_num$(file);
break;
,*/
}

return TREXIO_FAILURE;
}

Fortran templates for front end

The Fortran templates that provide an access to the C API calls from Fortran. These templates are based on the use of iso_c_binding. Pointers have to be passed by value.

interface
integer function trexio_write_$group_num$_64 (trex_file, num) bind(C)
use, intrinsic :: iso_c_binding
integer(8), intent(in), value :: trex_file
integer(8), intent(in), value :: num
end function trexio_write_$group_num$_64
end interface

interface
integer function trexio_read_$group_num$_64 (trex_file, num) bind(C)
use, intrinsic :: iso_c_binding
integer(8), intent(in), value :: trex_file
integer(8), intent(out) :: num
end function trexio_read_$group_num$_64
end interface

interface
integer function trexio_write_$group_num$_32 (trex_file, num) bind(C)
use, intrinsic :: iso_c_binding
integer(8), intent(in), value :: trex_file
integer(4), intent(in), value :: num
end function trexio_write_$group_num$_32
end interface

interface
integer function trexio_read_$group_num$_32 (trex_file, num) bind(C)
use, intrinsic :: iso_c_binding
integer(8), intent(in), value :: trex_file
integer(4), intent(out) :: num
end function trexio_read_$group_num$_32
end interface

interface
integer function trexio_write_$group_num$ (trex_file, num) bind(C)
use, intrinsic :: iso_c_binding
integer(8), intent(in), value :: trex_file
integer(4), intent(in), value :: num
end function trexio_write_$group_num$
end interface

interface
integer function trexio_read_$group_num$ (trex_file, num) bind(C)
use, intrinsic :: iso_c_binding
integer(8), intent(in), value :: trex_file
integer(4), intent(out) :: num
end function trexio_read_$group_num$
end interface

interface
integer function trexio_has_$group_num$ (trex_file) bind(C)
use, intrinsic :: iso_c_binding
integer(8), intent(in), value :: trex_file
end function trexio_has_$group_num$
end interface

Python templates for front end

def write_$group_num$(trexio_file, num_w: int) -> None: 
"""Write the $group_num$ variable in the TREXIO file.

Parameters:

trexio_file:
   TREXIO file handle.

num_w: int
   Value of the $group_num$ variable to be written.

Raises:
   - Exception from AssertionError if TREXIO return code ~rc~ is different from TREXIO_SUCCESS and prints the error message using trexio_string_of_error.
   - Exception from some other error (e.g. RuntimeError).
"""


try:
   rc = trexio_write_$group_num$(trexio_file, num_w)
   assert rc==TREXIO_SUCCESS
except AssertionError:
   raise Exception(trexio_string_of_error(rc))
except:
   raise

def read_$group_num$(trexio_file) -> int: 
"""Read the $group_num$ variable from the TREXIO file.

Parameter is a ~trexio_file~ object that has been created by a call to ~open~ function.

Returns:
   ~num_r~: int
   Integer value of $group_num$ variable read from ~trexio_file~.

Raises:
   - Exception from AssertionError if TREXIO return code ~rc~ is different from TREXIO_SUCCESS and prints the error message using trexio_string_of_error.
   - Exception from some other error (e.g. RuntimeError).
"""


try:
   rc, num_r = trexio_read_$group_num$(trexio_file)
   assert rc==TREXIO_SUCCESS
except AssertionError:
   raise Exception(trexio_string_of_error(rc))
except:
   raise    

return num_r

Templates for front end has/read/write a dataset of numerical data

This section concerns API calls related to datasets.

Function name	Description	Precision
`trexio_has_$group_dset$`	Check if a dataset exists in a file	—
`trexio_read_$group_dset$`	Read a dataset in default precision	Double/Single for float/int
`trexio_write_$group_dset$`	Write a dataset in default precision	Double/Single for float/int
`trexio_read_safe_$group_dset$`	Read a bounded dataset	Double
`trexio_write_safe_$group_dset$`	Write a bounded dataset	Double
`trexio_read_$group_dset$_32`	Read a dataset in single precision	Single
`trexio_write_$group_dset$_32`	Write a dataset in single precision	Single
`trexio_read_$group_dset$_64`	Read a dataset in double precision	Double
`trexio_write_$group_dset$_64`	Write a dataset in double precision	Double

C templates for front end

The C templates that correspond to each of the abovementioned functions can be found below. First parameter is the TREXIO file handle. Second parameter is the variable to be written/read to/from the TREXIO file (except for trexio_has_ functions). Suffixes _32 and _64 correspond to API calls dealing with single and double precision, respectively. The basic (non-suffixed) API call on datasets deals with double precision (see Table above).

Function declarations

Source code for double precision functions

trexio_exit_code
trexio_read_$group_dset$_64 (trexio_t* const file, $group_dset_dtype_double$* const $group_dset$)
{

if (file  == NULL) return TREXIO_INVALID_ARG_1;
if ($group_dset$ == NULL) return TREXIO_INVALID_ARG_2;

trexio_exit_code rc;
int64_t $group_dset_dim$ = 0;

/* Error handling for this call is added by the generator */
rc = trexio_read_$group_dset_dim$_64(file, &($group_dset_dim$));

if ($group_dset_dim$ == 0L) return TREXIO_INVALID_NUM;

uint32_t rank = $group_dset_rank$;
uint64_t dims[$group_dset_rank$] = {$group_dset_dim_list$};

assert(file->back_end < TREXIO_INVALID_BACK_END);

rc = TREXIO_FAILURE;
switch (file->back_end) {

case TREXIO_TEXT:
rc = trexio_text_read_$group_dset$(file, $group_dset$, rank, dims);
break;

case TREXIO_HDF5:
rc = trexio_hdf5_read_$group_dset$(file, $group_dset$, rank, dims);
break;
/*
case TREXIO_JSON:
rc = trexio_json_read_$group_dset$(file, $group_dset$, rank, dims);
break;
,*/
}
if (rc != TREXIO_SUCCESS) return rc;

/* Handle index type */
if ($is_index$) {
uint64_t dim_size = 1;
for (uint32_t i=0; i<rank; ++i){
   dim_size *= dims[i];
}
for (uint64_t i=0; i<dim_size; ++i){
  $group_dset$[i] += ($group_dset_dtype_single$) 1;
}
}

return TREXIO_SUCCESS;
}

trexio_exit_code
trexio_write_$group_dset$_64 (trexio_t* const file, const $group_dset_dtype_double$* $group_dset$)
{

if (file  == NULL) return TREXIO_INVALID_ARG_1;
if ($group_dset$ == NULL) return TREXIO_INVALID_ARG_2;
if (trexio_has_$group_dset$(file) == TREXIO_SUCCESS) return TREXIO_DSET_ALREADY_EXISTS;

trexio_exit_code rc;
int64_t $group_dset_dim$ = 0;

/* Error handling for this call is added by the generator */
rc = trexio_read_$group_dset_dim$_64(file, &($group_dset_dim$));

if ($group_dset_dim$ == 0L) return TREXIO_INVALID_NUM;

uint32_t rank = $group_dset_rank$;
uint64_t dims[$group_dset_rank$] = {$group_dset_dim_list$};

$group_dset_dtype_double$* $group_dset$_p = ($group_dset_dtype_double$*) $group_dset$;

/* Handle index type */
if ($is_index$) {

uint64_t dim_size = 1;
for (uint32_t i=0; i<rank; ++i){
   dim_size *= dims[i];
}

$group_dset$_p = CALLOC(dim_size, $group_dset_dtype_double$);
if ($group_dset$_p == NULL) return TREXIO_ALLOCATION_FAILED;

for (uint64_t i=0; i<dim_size; ++i){
  $group_dset$_p[i] = $group_dset$_p[i] - ($group_dset_dtype_single$) 1;
}
}

assert(file->back_end < TREXIO_INVALID_BACK_END);

rc = TREXIO_FAILURE;
switch (file->back_end) {

case TREXIO_TEXT:
rc = trexio_text_write_$group_dset$(file, $group_dset$_p, rank, dims);
break;

case TREXIO_HDF5:
rc = trexio_hdf5_write_$group_dset$(file, $group_dset$_p, rank, dims);
break;
/*
case TREXIO_JSON:
rc = trexio_json_write_$group_dset$(file, $group_dset$_p, rank, dims);
break;
,*/
}

/* Handle index type */
if ($is_index$) {
FREE($group_dset$_p);
}

return rc;
}

Source code for single precision functions

trexio_exit_code
trexio_read_$group_dset$_32 (trexio_t* const file, $group_dset_dtype_single$* const $group_dset$)
{

if (file  == NULL) return TREXIO_INVALID_ARG_1;
if ($group_dset$ == NULL) return TREXIO_INVALID_ARG_2;

trexio_exit_code rc;
int64_t $group_dset_dim$ = 0;

/* Error handling for this call is added by the generator */
rc = trexio_read_$group_dset_dim$_64(file, &($group_dset_dim$));

if ($group_dset_dim$ == 0L) return TREXIO_INVALID_NUM;

uint32_t rank = $group_dset_rank$;
uint64_t dims[$group_dset_rank$] = {$group_dset_dim_list$};

uint64_t dim_size = 1;
for (uint32_t i=0; i<rank; ++i){
dim_size *= dims[i];
}

$group_dset_dtype_double$* $group_dset$_64 = CALLOC(dim_size, $group_dset_dtype_double$);
if ($group_dset$_64 == NULL) return TREXIO_ALLOCATION_FAILED;

assert(file->back_end < TREXIO_INVALID_BACK_END);

rc = TREXIO_FAILURE;

switch (file->back_end) {

case TREXIO_TEXT:
rc = trexio_text_read_$group_dset$(file, $group_dset$_64, rank, dims);
break;

case TREXIO_HDF5:
rc = trexio_hdf5_read_$group_dset$(file, $group_dset$_64, rank, dims);
break;
/*
case TREXIO_JSON:
rc = trexio_json_read_$group_dset$(file, $group_dset$_64, rank, dims);
break;
,*/
}

if (rc != TREXIO_SUCCESS){
FREE($group_dset$_64);
return rc;
}

if ($is_index$) {
for (uint64_t i=0; i<dim_size; ++i){
 $group_dset$[i] = ($group_dset_dtype_single$) $group_dset$_64[i] + ($group_dset_dtype_single$) 1;
}
} else {
for (uint64_t i=0; i<dim_size; ++i){
 $group_dset$[i] = ($group_dset_dtype_single$) $group_dset$_64[i];
}
}

FREE($group_dset$_64);
return TREXIO_SUCCESS;
}

trexio_exit_code
trexio_write_$group_dset$_32 (trexio_t* const file, const $group_dset_dtype_single$* $group_dset$)
{

if (file  == NULL) return TREXIO_INVALID_ARG_1;
if ($group_dset$ == NULL) return TREXIO_INVALID_ARG_2;
if (trexio_has_$group_dset$(file) == TREXIO_SUCCESS) return TREXIO_DSET_ALREADY_EXISTS;

trexio_exit_code rc;
int64_t $group_dset_dim$ = 0;

/* Error handling for this call is added by the generator */
rc = trexio_read_$group_dset_dim$_64(file, &($group_dset_dim$));

if ($group_dset_dim$ == 0L) return TREXIO_INVALID_NUM;

uint32_t rank = $group_dset_rank$;
uint64_t dims[$group_dset_rank$] = {$group_dset_dim_list$};

uint64_t dim_size = 1;
for (uint32_t i=0; i<rank; ++i){
dim_size *= dims[i];
}

$group_dset_dtype_double$* $group_dset$_64 = CALLOC(dim_size, $group_dset_dtype_double$);
if ($group_dset$_64 == NULL) return TREXIO_ALLOCATION_FAILED;

/* A type conversion from single precision to double reqired since back end only accepts 64-bit data */
if ($is_index$) {
for (uint64_t i=0; i<dim_size; ++i){
 $group_dset$_64[i] = ($group_dset_dtype_double$) $group_dset$[i] - ($group_dset_dtype_double$) 1;
}
} else {
for (uint64_t i=0; i<dim_size; ++i){
 $group_dset$_64[i] = ($group_dset_dtype_double$) $group_dset$[i];
}
}

assert(file->back_end < TREXIO_INVALID_BACK_END);

rc = TREXIO_FAILURE;
switch (file->back_end) {

case TREXIO_TEXT:
rc = trexio_text_write_$group_dset$(file, $group_dset$_64, rank, dims);
break;

case TREXIO_HDF5:
rc = trexio_hdf5_write_$group_dset$(file, $group_dset$_64, rank, dims);
break;
/*
case TREXIO_JSON:
rc = trexio_json_write_$group_dset$(file, $group_dset$_64, rank, dims);
break;
,*/
}

FREE($group_dset$_64);

if (rc != TREXIO_SUCCESS) return rc;

return TREXIO_SUCCESS;
}

Source code for memory-safe functions

trexio_exit_code
trexio_read_safe_$group_dset$ (trexio_t* const file, $group_dset_dtype_default$* const dset_out, const int64_t dim_out)
{

if (file  == NULL) return TREXIO_INVALID_ARG_1;
if (dset_out == NULL) return TREXIO_INVALID_ARG_2;

trexio_exit_code rc;
int64_t $group_dset_dim$ = 0;

/* Error handling for this call is added by the generator */
rc = trexio_read_$group_dset_dim$_64(file, &($group_dset_dim$));

if ($group_dset_dim$ == 0L) return TREXIO_INVALID_NUM;

uint32_t rank = $group_dset_rank$;
uint64_t dims[$group_dset_rank$] = {$group_dset_dim_list$};

/* The block below is specific to safe API as it checks the boundaries */
uint64_t dim_size = 1;
for (uint32_t i=0; i<rank; ++i){
 dim_size *= dims[i];
}

if (dim_out > (int64_t) dim_size) return TREXIO_UNSAFE_ARRAY_DIM;
/* */

return trexio_read_$group_dset$_$default_prec$(file, dset_out);
}

trexio_exit_code
trexio_write_safe_$group_dset$ (trexio_t* const file, const $group_dset_dtype_default$* dset_in, const int64_t dim_in)
{

if (file  == NULL) return TREXIO_INVALID_ARG_1;
if (dset_in == NULL) return TREXIO_INVALID_ARG_2;
if (trexio_has_$group_dset$(file) == TREXIO_SUCCESS) return TREXIO_DSET_ALREADY_EXISTS;

trexio_exit_code rc;
int64_t $group_dset_dim$ = 0;

/* Error handling for this call is added by the generator */
rc = trexio_read_$group_dset_dim$_64(file, &($group_dset_dim$));

if ($group_dset_dim$ == 0L) return TREXIO_INVALID_NUM;

uint32_t rank = $group_dset_rank$;
uint64_t dims[$group_dset_rank$] = {$group_dset_dim_list$};

/* The block below is specific to safe API as it checks the boundaries */
uint64_t dim_size = 1;
for (uint32_t i=0; i<rank; ++i){
 dim_size *= dims[i];
}

if (dim_in > (int64_t) dim_size) return TREXIO_UNSAFE_ARRAY_DIM;
/* */

return trexio_write_$group_dset$_$default_prec$(file, dset_in);

}

Source code for default functions

trexio_exit_code
trexio_read_$group_dset$ (trexio_t* const file, $group_dset_dtype_default$* const $group_dset$)
{
return trexio_read_$group_dset$_$default_prec$(file, $group_dset$);
}

trexio_exit_code
trexio_write_$group_dset$ (trexio_t* const file, const $group_dset_dtype_default$* $group_dset$)
{
return trexio_write_$group_dset$_$default_prec$(file, $group_dset$);
}

trexio_exit_code
trexio_has_$group_dset$ (trexio_t* const file)
{

if (file  == NULL) return TREXIO_INVALID_ARG_1;

assert(file->back_end < TREXIO_INVALID_BACK_END);

switch (file->back_end) {

case TREXIO_TEXT:
return trexio_text_has_$group_dset$(file);
break;

case TREXIO_HDF5:
return trexio_hdf5_has_$group_dset$(file);
break;
/*
case TREXIO_JSON:
return trexio_json_has_$group_dset$(file);
break;
,*/
}
return TREXIO_FAILURE;
}

Fortran templates for front end

The Fortran templates that provide an access to the C API calls from Fortran. These templates are based on the use of iso_c_binding. Pointers have to be passed by value.

interface
integer function trexio_write_$group_dset$_64 (trex_file, dset) bind(C)
use, intrinsic :: iso_c_binding
integer(8), intent(in), value :: trex_file
$group_dset_f_dtype_double$, intent(in) :: dset$group_dset_f_dims$
end function trexio_write_$group_dset$_64
end interface

interface
integer function trexio_read_$group_dset$_64 (trex_file, dset) bind(C)
use, intrinsic :: iso_c_binding
integer(8), intent(in), value :: trex_file
$group_dset_f_dtype_double$, intent(out) :: dset$group_dset_f_dims$
end function trexio_read_$group_dset$_64
end interface

interface
integer function trexio_write_$group_dset$_32 (trex_file, dset) bind(C)
use, intrinsic :: iso_c_binding
integer(8), intent(in), value :: trex_file
$group_dset_f_dtype_single$, intent(in) :: dset$group_dset_f_dims$
end function trexio_write_$group_dset$_32
end interface

interface
integer function trexio_read_$group_dset$_32 (trex_file, dset) bind(C)
use, intrinsic :: iso_c_binding
integer(8), intent(in), value :: trex_file
$group_dset_f_dtype_single$, intent(out) :: dset$group_dset_f_dims$
end function trexio_read_$group_dset$_32
end interface

interface
integer function trexio_write_$group_dset$ (trex_file, dset) bind(C)
use, intrinsic :: iso_c_binding
integer(8), intent(in), value :: trex_file
$group_dset_f_dtype_default$, intent(in) :: dset$group_dset_f_dims$
end function trexio_write_$group_dset$
end interface

interface
integer function trexio_read_$group_dset$ (trex_file, dset) bind(C)
use, intrinsic :: iso_c_binding
integer(8), intent(in), value :: trex_file
$group_dset_f_dtype_default$, intent(out) :: dset$group_dset_f_dims$
end function trexio_read_$group_dset$
end interface

interface
integer function trexio_has_$group_dset$ (trex_file) bind(C)
use, intrinsic :: iso_c_binding
integer(8), intent(in), value :: trex_file
end function trexio_has_$group_dset$
end interface

Python templates for front end

def write_$group_dset$(trexio_file, dset_w) -> None: 
"""Write the $group_dset$ array of numbers in the TREXIO file.

Parameters:

trexio_file:
   TREXIO file handle.

dset_w: list OR numpy.ndarray
   Array of $group_dset$ values to be written.

Raises:
   - Exception from AssertionError if TREXIO return code ~rc~ is different from TREXIO_SUCCESS and prints the error message using trexio_string_of_error.
   - Exception from some other error (e.g. RuntimeError).
"""


try:
   rc = trexio_write_safe_$group_dset$(trexio_file, dset_w)
   assert rc==TREXIO_SUCCESS
except AssertionError:
   raise Exception(trexio_string_of_error(rc))
except:
   raise

def read_$group_dset$(trexio_file, dim = None): 
"""Read the $group_dset$ array of numbers from the TREXIO file.

Parameters:

trexio_file:
   TREXIO file handle.

dim (Optional): int
   Size of the block to be read from the file (i.e. how many items of $group_dset$ will be returned)
   If None, the function will read all necessary array dimensions from the file.

Returns:
   ~dset_r~: numpy.ndarray
   1D NumPy array with ~dim~ elements corresponding to $group_dset$ values read from the TREXIO file.

Raises:
   - Exception from AssertionError if TREXIO return code ~rc~ is different from TREXIO_SUCCESS and prints the error message using trexio_string_of_error.
   - Exception from some other error (e.g. RuntimeError).
"""


# if dim is not specified, read dimensions from the TREXIO file
if dim is None:
   $group_dset_dim$ = read_$group_dset_dim$(trexio_file) 
   
   dims_list = [$group_dset_dim_list$]
   dim = 1
   for i in range($group_dset_rank$):
       dim *= dims_list[i]


try:
   rc, dset_r = trexio_read_safe_$group_dset$(trexio_file, dim)
   assert rc==TREXIO_SUCCESS
except AssertionError:
   raise Exception(trexio_string_of_error(rc))
except:
   raise    

# additional assert can be added here to check that read_safe functions returns numpy array of proper dimension

return dset_r

Sparse data structures

Sparse data structures are used typically for large tensors such as two-electron integrals. For example, in the trex.json file sparse arrays appear as for the eri :

"ao_2e_int"  : {
  "eri_num"  : [ "int", [  ] ]
  "eri"      : [ "float sparse", [ "ao.num", "ao.num", "ao.num", "ao.num" ] ]
}

The electron repulsion integral $\langle ij | kl \rangle$ is represented as a quartet of integers $(i,j,k,l)$ and a floating point value.

To store $N$ integrals in the file, we store

An array of quartets of integers
An array of values (floats)

Both arrays have the same size, $N$, the number of non-zero integrals. Knowing the maximum dimensions allows to check that the integers are in a valid range, and also lets the library choose the smallest integer representation to compress the storage.

Fortran uses 1-based array indexing, while C uses 0-based indexing. Internally, we use a 0-based representation but the Fortran binding does the appropriate conversion when reading or writing.

As the number of integrals to store can be prohibitively large, we provide the possibility to read/write the integrals in chunks. So the functions take two extra parameters:

offset : the index of the 1st integral we want to read. An offset of zero implies to read the first integral.
num : the number of integrals to read.

We provide a function to read a chunk of indices, and a function to read a chunk of values, because some users might want to read only the values of the integrals, or only the indices.

Here is an example for the indices:

trexio_exit_code
trexio_read_chunk_ao_2e_int_eri_index_32(trexio_t* const file,
                                       const int64_t offset,
                                       const int64_t num,
                                       int32_t* buffer)
{
if (file  == NULL) return TREXIO_INVALID_ARG_1;
if (offset   < 0L) return TREXIO_INVALID_ARG_2;
if (num      < 0L) return TREXIO_INVALID_ARG_3;

const uint32_t rank = 4;  // To be set by generator : number of indices

int64_t nmax;             // Max number of integrals
trexio_exit_code rc;

rc = trexio_read_ao_2e_int_eri_num(const file, &nmax);
if (rc != TREXIO_SUCCESS) return rc;

switch (file->back_end) {

case TREXIO_TEXT:
  return trexio_text_read_chunk_ao_2e_int_eri_index(file, buffer, offset, num, rank, nmax);
  break;

case TREXIO_HDF5:
  return trexio_hdf5_read_chunk_ao_2e_int_eri_index(file, buffer, offset, num, rank, nmax);
  break;

default:
  return TREXIO_FAILURE;  /* Impossible case */
}
}

For the values,

trexio_exit_code
trexio_read_chunk_ao_2e_int_eri_value_64(trexio_t* const file,
                                       const int64_t offset,
                                       const int64_t num,
                                       double* buffer)
{
if (file  == NULL) return TREXIO_INVALID_ARG_1;
if (offset   < 0L) return TREXIO_INVALID_ARG_2;
if (num      < 0L) return TREXIO_INVALID_ARG_3;

int64_t nmax;             // Max number of integrals
trexio_exit_code rc;

rc = trexio_read_ao_2e_int_eri_num(const file, &nmax);
if (rc != TREXIO_SUCCESS) return rc;

switch (file->back_end) {

case TREXIO_TEXT:
  return trexio_text_read_chunk_ao_2e_int_eri_value(file, buffer, offset, num, nmax);
  break;

case TREXIO_HDF5:
  return trexio_hdf5_read_chunk_ao_2e_int_eri_index(file, buffer, offset, num, nmax);
  break;

default:
  return TREXIO_FAILURE;  /* Impossible case */
}
}

Templates for front end has/read/write a dataset of strings

Introduction

This section concerns API calls related to datasets of strings.

Function name	Description
`trexio_has_$group_dset$`	Check if a dataset exists in a file
`trexio_read_$group_dset$`	Read a dataset
`trexio_write_$group_dset$`	Write a dataset

C templates for front end

First parameter is the TREXIO file handle. Second parameter is the variable to be written/read to/from the TREXIO file (except for trexio_has_ functions).

trexio_exit_code
trexio_read_$group_dset$_low (trexio_t* const file, char* dset_out, const uint32_t max_str_len)
{

if (file  == NULL) return TREXIO_INVALID_ARG_1;
if (dset_out == NULL) return TREXIO_INVALID_ARG_2;
if (max_str_len <= 0) return TREXIO_INVALID_ARG_3;

trexio_exit_code rc;
int64_t $group_dset_dim$ = 0;

/* Error handling for this call is added by the generator */
rc = trexio_read_$group_dset_dim$_64(file, &($group_dset_dim$));

if ($group_dset_dim$ == 0L) return TREXIO_INVALID_NUM;

uint32_t rank = $group_dset_rank$;
uint64_t dims[$group_dset_rank$] = {$group_dset_dim_list$};

assert(file->back_end < TREXIO_INVALID_BACK_END);
switch (file->back_end) {

case TREXIO_TEXT:
return trexio_text_read_$group_dset$(file, dset_out, rank, dims, max_str_len);
break;

case TREXIO_HDF5:
return trexio_hdf5_read_$group_dset$(file, dset_out, rank, dims, max_str_len);
break;
/*
case TREXIO_JSON:
rc = trexio_json_read_$group_dset$(file, dset_out, rank, dims);
break;
,*/
}

return TREXIO_FAILURE;
}

trexio_exit_code
trexio_read_$group_dset$ (trexio_t* const file, char** dset_out, const uint32_t max_str_len)
{

if (file  == NULL) return TREXIO_INVALID_ARG_1;
if (dset_out == NULL) return TREXIO_INVALID_ARG_2;
if (max_str_len <= 0) return TREXIO_INVALID_ARG_3;

assert(file->back_end < TREXIO_INVALID_BACK_END);

trexio_exit_code rc;
int64_t dset_dim = 0;

/* Error handling for this call is added by the generator */
rc = trexio_read_$group_dset_dim$_64(file, &(dset_dim));

if (dset_dim == 0L) return TREXIO_INVALID_NUM;

char* str_compiled = CALLOC(dset_dim*(max_str_len+1) + 1, char);
if (str_compiled == NULL) return TREXIO_ALLOCATION_FAILED;

rc = trexio_read_$group_dset$_low(file, str_compiled, max_str_len);
if (rc != TREXIO_SUCCESS) {
FREE(str_compiled);
return rc;
}

char * pch;
for (uint64_t i=0; i < (uint64_t) dset_dim; i++) {

pch = i == 0 ? strtok(str_compiled, TREXIO_DELIM) : strtok(NULL, TREXIO_DELIM) ;
if (pch == NULL) {
 FREE(str_compiled);
 return TREXIO_FAILURE;
}

strcpy(dset_out[i], "");
strcat(dset_out[i], pch);

}

FREE(str_compiled);
return TREXIO_SUCCESS;

}

trexio_exit_code
trexio_write_$group_dset$_low (trexio_t* const file, const char* dset_in, const uint32_t max_str_len)
{

if (file  == NULL) return TREXIO_INVALID_ARG_1;
if (dset_in == NULL) return TREXIO_INVALID_ARG_2;
if (max_str_len <= 0) return TREXIO_INVALID_ARG_3;
if (trexio_has_$group_dset$(file) == TREXIO_SUCCESS) return TREXIO_DSET_ALREADY_EXISTS;

trexio_exit_code rc;
int64_t $group_dset_dim$ = 0;

/* Error handling for this call is added by the generator */
rc = trexio_read_$group_dset_dim$_64(file, &($group_dset_dim$));

if ($group_dset_dim$ == 0L) return TREXIO_INVALID_NUM;

uint32_t rank = $group_dset_rank$;
uint64_t dims[$group_dset_rank$] = {$group_dset_dim_list$};

assert(file->back_end < TREXIO_INVALID_BACK_END);

char*  tmp_str  = CALLOC(dims[0]*(max_str_len+1), char);
if (tmp_str == NULL) return TREXIO_ALLOCATION_FAILED;
char** dset_str = CALLOC(dims[0], char*);
if (dset_str == NULL) return TREXIO_ALLOCATION_FAILED;

char* pch;
size_t pch_len;
/* parse the string using strtok */
for(uint64_t i=0; i<dims[0]; i++) {

pch = i == 0 ? strtok( (char*) dset_in, TREXIO_DELIM) : strtok(NULL, TREXIO_DELIM) ;

if (pch == NULL) {
 FREE(dset_str[0]);
 FREE(dset_str);
 return TREXIO_FAILURE;
}

pch_len = strlen(pch) + 1;

if (pch_len > max_str_len) {
 FREE(dset_str[0]);
 FREE(dset_str);
 return TREXIO_INVALID_STR_LEN;
}

dset_str[i] = tmp_str;
strncpy(tmp_str, pch, pch_len);
tmp_str += pch_len + 1;
}

rc = TREXIO_FAILURE;  
switch (file->back_end) {

case TREXIO_TEXT:
rc = trexio_text_write_$group_dset$(file, (const char**) dset_str, rank, dims);
break;

case TREXIO_HDF5:
rc = trexio_hdf5_write_$group_dset$(file, (const char**) dset_str, rank, dims);
break;
/*
case TREXIO_JSON:
rc = trexio_json_write_$group_dset$(file, dset, rank, dims);
break;
,*/
}

FREE(dset_str[0]);
FREE(dset_str);

return rc;

}

trexio_exit_code
trexio_write_$group_dset$ (trexio_t* const file, const char** dset_in, const uint32_t max_str_len)
{

if (file  == NULL) return TREXIO_INVALID_ARG_1;
if (dset_in == NULL) return TREXIO_INVALID_ARG_2;
if (max_str_len <= 0) return TREXIO_INVALID_ARG_3;
if (trexio_has_$group_dset$(file) == TREXIO_SUCCESS) return TREXIO_DSET_ALREADY_EXISTS;

assert(file->back_end < TREXIO_INVALID_BACK_END);

trexio_exit_code rc;
int64_t dset_dim = 0;

/* Error handling for this call is added by the generator */
rc = trexio_read_$group_dset_dim$_64(file, &(dset_dim));

if (dset_dim == 0L) return TREXIO_INVALID_NUM;

char* str_compiled = CALLOC(dset_dim*max_str_len + 1, char);
if (str_compiled == NULL) return TREXIO_ALLOCATION_FAILED;

strcpy(str_compiled, "");
for (uint64_t i=0; i < (uint64_t) dset_dim; i++) {
strcat(str_compiled, dset_in[i]);
strcat(str_compiled, TREXIO_DELIM);
}

rc = trexio_write_$group_dset$_low(file, str_compiled, max_str_len);

FREE(str_compiled);

return rc;
}

trexio_exit_code
trexio_has_$group_dset$ (trexio_t* const file)
{

if (file  == NULL) return TREXIO_INVALID_ARG_1;

assert(file->back_end < TREXIO_INVALID_BACK_END);

switch (file->back_end) {

case TREXIO_TEXT:
return trexio_text_has_$group_dset$(file);
break;

case TREXIO_HDF5:
return trexio_hdf5_has_$group_dset$(file);
break;
/*
case TREXIO_JSON:
return trexio_json_has_$group_dset$(file);
break;
,*/
}

return TREXIO_FAILURE;
}

Fortran templates for front end

The Fortran templates that provide an access to the C API calls from Fortran. These templates are based on the use of iso_c_binding. Pointers have to be passed by value.

interface
integer function trexio_write_$group_dset$_low (trex_file, dset, max_str_len) bind(C)
use, intrinsic :: iso_c_binding
integer(8), intent(in), value :: trex_file
character, intent(in) :: dset(*)
integer(4), intent(in), value :: max_str_len
end function trexio_write_$group_dset$_low
end interface

interface
integer function trexio_read_$group_dset$_low (trex_file, dset, max_str_len) bind(C)
use, intrinsic :: iso_c_binding
integer(8), intent(in), value :: trex_file
character, intent(out) :: dset(*)
integer(4), intent(in), value :: max_str_len
end function trexio_read_$group_dset$_low
end interface

interface
integer function trexio_has_$group_dset$ (trex_file) bind(C)
use, intrinsic :: iso_c_binding
integer(8), intent(in), value :: trex_file
end function trexio_has_$group_dset$
end interface

 integer function trexio_read_$group_dset$ (trex_file, dset, max_str_len)
   implicit none
   integer(8), intent(in), value :: trex_file
   integer(4), intent(in), value :: max_str_len
   character(len=*), intent(inout) :: dset(*)

   character, allocatable :: str_compiled(:)
   integer(8) :: $group_dset_dim$
   integer :: rc
   
   rc = trexio_read_$group_dset_dim$_64(trex_file, $group_dset_dim$)
   if (rc /= TREXIO_SUCCESS) trexio_read_$group_dset$ = rc

   allocate(str_compiled($group_dset_dim$*(max_str_len+1)+1))
   
   rc = trexio_read_$group_dset$_low(trex_file, str_compiled, max_str_len)
   if (rc /= TREXIO_SUCCESS) then 
     deallocate(str_compiled)
     trexio_read_$group_dset$ = rc
   else
     call trexio_str2strarray(str_compiled, $group_dset_dim$, max_str_len, dset)
     deallocate(str_compiled)
     trexio_read_$group_dset$ = TREXIO_SUCCESS
   endif

 end function trexio_read_$group_dset$

 integer function trexio_write_$group_dset$ (trex_file, dset, max_str_len)
   implicit none
   integer(8), intent(in), value :: trex_file
   integer(4), intent(in), value :: max_str_len
   character(len=*), intent(in) :: dset(*)

   character(len=:), allocatable :: str_compiled
   integer(8) :: $group_dset_dim$
   integer :: rc
   
   rc = trexio_read_$group_dset_dim$_64(trex_file, $group_dset_dim$)
   if (rc /= TREXIO_SUCCESS) then
     trexio_write_$group_dset$ = rc
   else
     call trexio_strarray2str(dset, $group_dset_dim$, max_str_len, str_compiled)
     trexio_write_$group_dset$ = trexio_write_$group_dset$_low(trex_file, str_compiled, max_str_len)
   endif

 end function trexio_write_$group_dset$

Python templates for front end

def write_$group_dset$(trexio_file, dset_w: list) -> None: 
"""Write the $group_dset$ array of strings in the TREXIO file.

Parameters:

trexio_file:
   TREXIO file handle.

dset_w: list
   Array of $group_dset$ strings to be written.

Raises:
   - Exception from AssertionError if TREXIO return code ~rc~ is different from TREXIO_SUCCESS and prints the error message using trexio_string_of_error.
   - Exception from some other error (e.g. RuntimeError).
"""


max_str_length = len(max(dset_w, key=len)) + 1

try:
   rc = trexio_write_$group_dset$(trexio_file, dset_w, max_str_length)
   assert rc==TREXIO_SUCCESS
except AssertionError:
   raise Exception(trexio_string_of_error(rc))
except:
   raise

def read_$group_dset$(trexio_file, dim = None) -> list: 
"""Read the $group_dset$ array of strings from the TREXIO file.

Parameters:

trexio_file:
   TREXIO file handle.

dim (Optional): int
   Size of the block to be read from the file (i.e. how many items of $group_dset$ will be returned)
   If None, the function will read all necessary array dimensions from the file.

Returns:
   ~dset_r~: list
   1D list with ~dim~ elements corresponding to $group_dset$ strings read from the TREXIO file.

Raises:
   - Exception from AssertionError if TREXIO return code ~rc~ is different from TREXIO_SUCCESS and prints the error message using trexio_string_of_error.
   - Exception from some other error (e.g. RuntimeError).
"""


# if dim is not specified, read dimensions from the TREXIO file
if dim is None:
   $group_dset_dim$ = read_$group_dset_dim$(trexio_file) 
   
   dims_list = [$group_dset_dim_list$]
   dim = 1
   for i in range($group_dset_rank$):
       dim *= dims_list[i]


try:
   rc, dset_1d_r = trexio_read_$group_dset$_low(trexio_file, PYTREXIO_MAX_STR_LENGTH)
   assert rc==TREXIO_SUCCESS
except AssertionError:
   raise Exception(trexio_string_of_error(rc))
except:
   raise    


try:
   dset_full = dset_1d_r.split(TREXIO_DELIM)
   dset_2d_r = [dset_full[i] for i in range(dim) if dset_full[i]]
   assert dset_2d_r
except AssertionError:
   raise TypeError(f"Output of {read_$group_dset$.__name__} function cannot be an empty list.")

return dset_2d_r

Templates for front end has/read/write a single string attribute

Introduction

This section concerns API calls related to string attributes.

Function name	Description
`trexio_has_$group_str$`	Check if a string attribute exists in a file
`trexio_read_$group_str$`	Read a string attribute
`trexio_write_$group_str$`	Write a string attribute

C templates for front end

trexio_exit_code
trexio_read_$group_str$ (trexio_t* const file, char* const str_out, const uint32_t max_str_len)
{

if (file == NULL) return TREXIO_INVALID_ARG_1;
if (str_out  == NULL) return TREXIO_INVALID_ARG_2;
if (max_str_len <= 0) return TREXIO_INVALID_ARG_3;

switch (file->back_end) {

case TREXIO_TEXT:
return trexio_text_read_$group_str$(file, str_out, max_str_len);
break;

case TREXIO_HDF5:
return trexio_hdf5_read_$group_str$(file, str_out, max_str_len);
break;
/*
case TREXIO_JSON:
return trexio_json_read_$group_str$(file, str);
break;
,*/
}

return TREXIO_FAILURE;
}

trexio_exit_code
trexio_write_$group_str$ (trexio_t* const file, const char* str, const uint32_t max_str_len)
{

if (file == NULL) return TREXIO_INVALID_ARG_1;
if (str  == NULL) return TREXIO_INVALID_ARG_2;
if (max_str_len <= 0) return TREXIO_INVALID_ARG_3;
if (trexio_has_$group_str$(file) == TREXIO_SUCCESS) return TREXIO_ATTR_ALREADY_EXISTS;

size_t len_write = strlen(str);
if (max_str_len < len_write) return TREXIO_INVALID_STR_LEN;  

switch (file->back_end) {

case TREXIO_TEXT:
return trexio_text_write_$group_str$(file, str);
break;

case TREXIO_HDF5:
return trexio_hdf5_write_$group_str$(file, str);
break;
/*
case TREXIO_JSON:
return trexio_json_write_$group_str$(file, str);
break;
,*/
}

return TREXIO_FAILURE;
}

trexio_exit_code
trexio_has_$group_str$ (trexio_t* const file)
{

if (file == NULL) return TREXIO_INVALID_ARG_1;

assert(file->back_end < TREXIO_INVALID_BACK_END);

switch (file->back_end) {

case TREXIO_TEXT:
return trexio_text_has_$group_str$(file);
break;

case TREXIO_HDF5:
return trexio_hdf5_has_$group_str$(file);
break;
/*
case TREXIO_JSON:
return trexio_json_has_$group_str$(file);
break;
,*/
}

return TREXIO_FAILURE;
}

Fortran templates for front end

The Fortran templates that provide an access to the C API calls from Fortran. These templates are based on the use of iso_c_binding. Pointers have to be passed by value.

interface
integer function trexio_write_$group_str$_c (trex_file, str, max_str_len) &
      bind(C, name="trexio_write_$group_str$")
use, intrinsic :: iso_c_binding
integer(8), intent(in), value :: trex_file
character, intent(in) :: str(*)
integer(4), intent(in), value :: max_str_len
end function trexio_write_$group_str$_c
end interface

interface
integer function trexio_read_$group_str$_c (trex_file, str, max_str_len) &
      bind(C, name="trexio_read_$group_str$")
use, intrinsic :: iso_c_binding
integer(8), intent(in), value :: trex_file
character, intent(out) :: str(*)
integer(4), intent(in), value :: max_str_len
end function trexio_read_$group_str$_c
end interface

interface
integer function trexio_has_$group_str$ (trex_file) bind(C)
use, intrinsic :: iso_c_binding
integer(8), intent(in), value :: trex_file
end function trexio_has_$group_str$
end interface

 integer function trexio_read_$group_str$ (trex_file, str, max_str_len)
   implicit none
   integer(8), intent(in), value :: trex_file
   integer(4), intent(in), value :: max_str_len
   character, intent(out) :: str(*)

   trexio_read_$group_str$ = trexio_read_$group_str$_c(trex_file, str, max_str_len)

 end function trexio_read_$group_str$

 integer function trexio_write_$group_str$ (trex_file, str, max_str_len)
   use, intrinsic :: iso_c_binding, only : c_null_char
   implicit none
   integer(8), intent(in), value :: trex_file
   integer(4), intent(in), value :: max_str_len
   character(len=*), intent(in) :: str

   character(len=len_trim(str)+1) :: str_c

   str_c = trim(str) // c_null_char

   trexio_write_$group_str$ = trexio_write_$group_str$_c(trex_file, str_c, max_str_len)

 end function trexio_write_$group_str$

Python templates for front end

def write_$group_str$(trexio_file, str_w: str) -> None: 
"""Write the $group_str$ variable in the TREXIO file.

Parameters:

trexio_file:
   TREXIO file handle.

str_w: str
   String corresponding to the $group_str$ variable to be written.

Raises:
   - Exception from AssertionError if TREXIO return code ~rc~ is different from TREXIO_SUCCESS and prints the error message using trexio_string_of_error.
   - Exception from some other error (e.g. RuntimeError).
"""


max_str_length = len(str_w) + 1

try:
   rc = trexio_write_$group_str$(trexio_file, str_w, max_str_length)
   assert rc==TREXIO_SUCCESS
except AssertionError:
   raise Exception(trexio_string_of_error(rc))
except:
   raise

def read_$group_str$(trexio_file) -> str: 
"""Read the $group_str$ variable from the TREXIO file.

Parameter is a ~trexio_file~ object that has been created by a call to ~open~ function.

Returns:
   ~str_r~: str
   String corresponding to the $group_str$ variable read from ~trexio_file~.

Raises:
   - Exception from AssertionError if TREXIO return code ~rc~ is different from TREXIO_SUCCESS and prints the error message using trexio_string_of_error.
   - Exception from some other error (e.g. RuntimeError).
"""


try:
   rc, str_r = trexio_read_$group_str$(trexio_file, PYTREXIO_MAX_STR_LENGTH)
   assert rc==TREXIO_SUCCESS
except AssertionError:
   raise Exception(trexio_string_of_error(rc))
except:
   raise    

return str_r

Fortran helper/wrapper functions

The function below adapts the original C-based trexio_open for Fortran. This is needed due to the fact that strings in C are terminated by NULL character \0 unlike strings in Fortran. Note, that Fortran interface calls the main TREXIO API, which is written in C.

contains
 integer(8) function trexio_open (filename, mode, backend)
   use, intrinsic :: iso_c_binding, only : c_null_char
   implicit none
   character(len=*), intent(in)                 :: filename
   character, intent(in), value                 :: mode
   integer(trexio_backend), intent(in), value   :: backend
   character(len=len_trim(filename)+1) :: filename_c
   integer :: rc

   filename_c = trim(filename) // c_null_char
   trexio_open = trexio_open_c(filename_c, mode, backend)
   if (trexio_open == 0_8) then
     return
   endif
   rc = trexio_set_one_based(trexio_open)
   if (rc /= TREXIO_SUCCESS) then
      rc = trexio_close(trexio_open)
      trexio_open = 0_8
   endif
 end function trexio_open

The subroutine below transforms an array of Fortran strings into one big string using TREXIO_DELIM symbol as a delimeter and adds NULL character in the end in order to properly pass the desired string to C API. This is needed due to the fact that strings in C are terminated by NULL character \0.

 subroutine trexio_strarray2str(str_array, max_num_str, max_len_str, str_res)
   use, intrinsic :: iso_c_binding, only : c_null_char
   implicit none

   integer(8), intent(in), value   :: max_num_str  ! number of elements in strign array
   integer, intent(in), value   :: max_len_str  ! maximum length of a string in an array
   character(len=*), intent(in)  :: str_array(*)
   character(len=:), allocatable, intent(out) :: str_res
   integer :: i

   str_res = ''
   do i = 1, max_num_str
     str_res = str_res // trim(str_array(i)) // TREXIO_DELIM
   enddo
   str_res = str_res // c_null_char

 end subroutine trexio_strarray2str

The subroutine below does the reverse tranformation from one big string with delimeters into an array of Fortran strings.

 subroutine trexio_str2strarray(str_flat, max_num_str, max_len_str, str_array)
   implicit none

   integer(8), intent(in), value   :: max_num_str  ! number of elements in strign array
   integer, intent(in), value   :: max_len_str  ! maximum length of a string in an array
   character, intent(in) :: str_flat(*)
   character(len=*), intent(inout)  :: str_array(*)
   
   character(len=max_len_str)  :: tmp_str
   integer :: i, j, k, ind, offset
   integer(8) :: len_flat

   len_flat = (max_len_str+1)*max_num_str + 1
  
   ind=1
   offset=1
   do i=1,max_num_str
     k = 1
     tmp_str=''
     do j=ind,len_flat
       if (str_flat(j) == TREXIO_DELIM) then
         ind=j+1
         exit
       endif
       tmp_str(k:k) = str_flat(j)
       k = k + 1
     enddo
     str_array(i)=tmp_str
     offset=ind
   enddo

 end subroutine trexio_str2strarray

The subroutine is a Fortran analogue of assert in C. It check that the the return code of the TREXIO API call is equal to a given return code. It can optionally print a success message if the two code are identical, i.e. if the assert statement pass.

 subroutine trexio_assert(trexio_rc, check_rc, success_message)
   implicit none

   integer, intent(in), value   :: trexio_rc
   integer, intent(in), value   :: check_rc
   character(len=*), intent(in), optional  :: success_message

   character*(128) :: str

   if (trexio_rc == check_rc) then
     if (present(success_message)) write(*,*) success_message
   else
     call trexio_string_of_error(trexio_rc, str)
     print *, trim(str)
     call exit(1)
   endif

 end subroutine trexio_assert

88 KiB Raw Blame History Unescape Escape

Front end API

Coding conventions

Memory allocation

Front end

Error handling

Decoding errors

C source code

Fortran interface

Python interface

Back ends

C

Fortran

Python

Read/write behavior

TREXIO file type

Polymorphism of the file handle

File opening

C

Fortran

Python

Zero-based versus one-based arrays of indices

File closing

C

Fortran

Python

Templates for front end

Description

Templates for front end has/read/write a single dimensioning variable

C templates for front end

Fortran templates for front end

Python templates for front end

Templates for front end has/read/write a dataset of numerical data

C templates for front end

Function declarations

Source code for double precision functions

Source code for single precision functions

Source code for memory-safe functions

Source code for default functions

Fortran templates for front end

Python templates for front end

Sparse data structures

Templates for front end has/read/write a dataset of strings

Introduction

C templates for front end

Fortran templates for front end

Python templates for front end

Templates for front end has/read/write a single string attribute

Introduction

C templates for front end

Fortran templates for front end

Python templates for front end

Fortran helper/wrapper functions

88 KiB

Raw Blame History