-
Notifications
You must be signed in to change notification settings - Fork 8
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Taweret/OpenBTMixing Building & Distribution #83
Comments
Let's begin by noting that Assuming the Build
|
@ominusliticus Interesting. I've never heard of a package building in a dependency but refusing to install it. That does sound like something we might need. As long as the error message is helpful and points them to clear docs, it seems acceptable. It seems like much of your workflow is in accord with my requirements. In other words, it could be a solution that satisfies the requirements. Do you agree? I've realized that while we are trying to evolve the package distribution portion of OpenBT as you point out, we are at the same time updating the overall Taweret software architecture in the case that OpenBTMixing is a dependency so that the architecture is compatible with an acceptable build/distribution strategy. We've seen this in the work that John is doing and my next question is related. There is the "homebrew-style" architecture where the OpenBT/C++ CLTs and the OpenBTMixing Python package are independent layers in the SW architecture. In such case, the Python package is a pure Python package. However, there is a "Python source build" architecture where the OpenBTMixing Python package is a true wrapper of the OpenBT/C++ CLTs. I believe that this style implies that the package must have its own build system and users would likely always have to build from source. Is one of these what you have in mind for your workflow? |
In my own view of things, I am more inclined towards the figure on the right. The picture on the left is something we have to fit into the "fast-paced data science in python" pipeline. If we can enable this for users, we ought to. Regarding the discussion of the relevance of OpenBT infrastructure for Taweret, I agree. Though the results of our findings/conclusion will probably be documents in the BAND software guidelines, which will be useful for future developers. |
In regard to Anaconda, here are the things I have found out so far:
Update:
|
The simplest means for users to install
Taweret
would be by issuingpip install Taweret
with that command installingopenbtmixing
automatically from PyPI. However, we need to determine if such a scheme is feasible givenopenbtmixing
's dependence on MPI. As part of this, we can try to determine all the ways in which users can install and use both of these packages.Possible Requirements
openbtmixing
’s build system shall allows users to use the software on laptops, desktops, nodes, clusters, or supercomputers. The build system shall be capable for building, installing, and using on macOS, *nix, Windows/Ubuntu, and Windows/Powershell.openbtmixing
’s build system shall allow for CI testing across the Cartesian product of common setups{All OS} x {GCC, Intel, …} x {OpenMPI, MPICH, MVAPICH, etc.} x {debug, production}
openbtmixing
command line tools (CLT) and libraries shall be made available through at least one compatible package manager (e.g., homebrew, apt-get, spack, etc.).openbtmixing
C++ should be installed by itself and then the Python package just looks for it or finds it in the path. Is this what we want? Should the Python package build its own internal version of the tools as it presently does?openbtmixing
be installable with and without integrated CLT?openbtmixing
with the compiler suite and MPI implementation of their choice. This includes using suites and implementations installed by experts and that are optimized for the associated platform.openbtmixing
shall be pip installable. Will this be a source-only distribution with automatic building integrated? Can we distribute prebuilt wheels and satisfy our MPI-based requirements?Taweret
shall be pip installable. If MPI is isolated inopenbtmixing
, thenTaweret
is a pure python package and users shall be able to pip install from PyPI via source distribution or universal binary wheel. Should we account for users that want to run through a git clone? Should we allow for users/developers to install via a clone in editable/developer mode (i.e.,pip install -e
)?openbtmixing
Python package shall be listed as an external dependence ofTaweret
.pip install Taweret
would try to installopenbtmixing
from PyPI and therefore with the defaultopenbtmixing
install scheme. Feasible? Good idea?openbtmixing
distribution includes an MPI implementation, then the integration of that implementation in the package shall be such that other implementations in a user's system cannot be used withopenbtmixing
at execution by accident and such that our MPI implementation cannot accidentally insert itself in a different software’s stack.openbtmixing
andTaweret
in a regular Python installation or in an anaconda installationopenbtmixing
openbtmixing
Python package know what compilers to use to build the CLT/libraries and where to find the MPI implementation?Taweret
on macOS in a general Python installation isbrew install open-mpi
(or mpich and also a particular version using which compiler suite?)brew install eigen
(header only => no compilation here)brew install openbtmixing
(C++ CLT and libraries built with your homebrew MPI installation & matching compiler suite using your eigen)pip install Taweret
(this should automatically installopenbtmixing
Python package based on your homebrewopenbtmixing
/MPI)The text was updated successfully, but these errors were encountered: