From 79f233e00d4c3552f3e303d76c8575ba27cec38c Mon Sep 17 00:00:00 2001 From: q-posev Date: Thu, 16 Sep 2021 16:40:29 +0200 Subject: [PATCH] completed README --- docker/README | 45 ------------------------------ docker/README.md | 71 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 71 insertions(+), 45 deletions(-) delete mode 100644 docker/README create mode 100644 docker/README.md diff --git a/docker/README b/docker/README deleted file mode 100644 index 31a6300..0000000 --- a/docker/README +++ /dev/null @@ -1,45 +0,0 @@ -First, make sure to place the source code distribution (suffixed with .tar.gz) of the TREXIO Python API in this directory. - -TODO: the scripts have to be adapted for an arbitrary version of TREXIO and Python ! - -Outside Docker image: - -# Build containers with hdf5 inside: - -# for manylinux2014_x86_64 -sudo docker build -t hdf5_1_12_on_2014_x86_64 . -f Dockerfile_2014_x86_64 - -# for manylinux_2_24_x86_64 -sudo docker build -t hdf5_1_12_on_2_24_x86_64 . -f Dockerfile_2_24_x86_64 - -# (create an image using HDF5 containers, see https://github.com/h5py/hdf5-manylinux) -# -t hdf5_1_12_on_${PLATFORM} builds an image with a custom name (hdf5_1_12_1) - -# Run one of the produced containers in the interactive mode - -# for manylinux2014_x86_64 -sudo docker run --rm -it -e PLAT=manylinux2014_x86_64 -v `pwd`:/tmp hdf5_1_12_on_2014_x86_64 /bin/bash - -# for manylinux_2_24_x86_64 -sudo docker run --rm -it -e PLAT=manylinux_2_24_x86_64 -v `pwd`:/tmp hdf5_1_12_on_2_24_x86_64 /bin/bash - -#(PLAT specifies the base platform tag for manilinux) -#-i (run docker container interactively) -#-t f8e4232fa208 (run an image using ID if the image is not named) -#/bin/bash (a binary to run inside a container environment) - -In the Docker image: - -cd tmp && ./build_manylinux_wheels_py_36_37_38.sh -# (creates virtual environments, installs things with pip, produces conventional wheels and repairs them with auditwheel) - -# auditwheel repair trexio-0.1.0/dist/trexio-0.1.0-cp37-cp37m-linux_x86_64.whl -# (repairs wheel and produces manylinux one) - -After Docker container execution: - -# the produced manylinux wheels are in trexio-0.1.0/wheelhouse directory - -# ALTERNATIVELY: one can copy the produced manylinux wheel from the container back to the host system -sudo docker cp $CONTAINER_ID:/tmp/trexio-0.1.0/wheelhouse/MANYLINUX_NAME.whl directory-on-the-user-machine/ - diff --git a/docker/README.md b/docker/README.md new file mode 100644 index 0000000..8a1996e --- /dev/null +++ b/docker/README.md @@ -0,0 +1,71 @@ +# Docker build + +Containerized builds related to TREXIO library. + +## Producing manylinux wheels for Python API + +Distributing binary wheels for Python packages that rely on C extensions is a non-trivial task. +For more details, see the associated +[PEP 425](https://www.python.org/dev/peps/pep-0425/) and +[PEP 600](https://www.python.org/dev/peps/pep-0600/). + +This issue is particularly tricky on Linux platforms. However, PyPA provides `Docker` containers +that can be used as an isolated environment to produce binary wheels that are compatible with most Linux distributions. +Such wheels should contain `manylinux` in their platform tag +(e.g. `...-manylinux_2_24_x86_64.whl` on 64-bit Intel with `glibc >= 2.24`). +More technical details can be found in the corresponding [GitHub repository](https://github.com/pypa/manylinux). + +This section summarizes the steps needed to build `manylinux`-compatible wheels using the aforementioned containers. + + +### Building the Docker containers + +The primary TREXIO back end at the moment is based on the [HDF5](https://portal.hdfgroup.org/display/HDF5/HDF5) library. +Because of this, one has to build new Docker images from the PyPA ones. +These images will contain the HDF5 library and will be used to produce wheels for the Python API. + +To build the Docker images on x86_64 platform, execute the following: + +`cd hdf5-on-docker/ && ./build_images.sh` + +This should produce several images tagged with `hdf5_1_12_` prefix and `_x86_64` suffix. +These images inherit from the PyPA manylinux containers. +All available images can be listed using `docker image list` command. + + +### Building the manylinux wheels + +First, make sure that you have the source code distribution +(e.g. `trexio-0.2.0.tar.gz`) of the Python API in the current directory. +Then run one of the previously produced containers in the interactive mode using the following command: + +``docker run --rm -it -e PLAT=manylinux2014_x86_64 -v `pwd`:/tmp hdf5_1_12_on_2014_x86_64 /bin/bash`` + +where `2014_x86_64` can be replaced with any other available platform suffix (e.g. `2010_x86_64` or `2_24_x86_64`) +The `docker run` command line arguments used here: + + - `-i` (run the docker container in interactive mode) + - `--rm` (remove the container upon exit) + - ``-v `pwd`:/tmp`` (mount current directory into the `tmp` directory on the container + - `-e` (set the environment variables according to the provided list) + - `hdf5_1_12_on_2014_x86_64` (name of the Docker image to run) + - `/bin/bash` (which binary the container should execute) + +Once the Docker container is launched and the corresponding prompt is switched to the container, run the following (example for release 0.2.0): + +`cd tmp/ && ./build_manylinux_wheels.sh trexio-0.2.0.tar.gz` + +This script creates virtual envirionments for several versions of `CPython` (e.g. 3.7, 3.8, 3.9), +installs all requirements, +produces `trexio` wheels with conventional `linux_x86_64` tag and +repairs them to be tagged with `manylinux` using the `auditwheel` tool from PyPA + +The produced wheels with the `manylinux` platfrom tag can be found in `trexio-0.2.0/wheelhouse` directory. + + +### Exporting the wheels + +You may want to produce wheels for several versions of `glibc`. +Prior to running a new docker container, make sure to rename/move the `wheelhouse` directory. +Otherwise, the `build_manylinux_wheels.sh` script will crush. +