Free Thoughts: May 2015

Rationale

I've been trying to figure out how to deploy a consistent set of scientific python applications and modules across different operating systems. I wanted to try to satisfy the following goals:

Use rpm for package deployment
Build directly from Fedora packages
Re-use as much of the base OS environment as possible

What I'm testing out now is building a base python virtual environment that will be the location that all other packages built will install into. Here's how it works:

Base environment package

I'm calling my base package 'pyvenv-nwra'. This indicates that it is a python virtualenv, and specifies the name (NWRA). The spec file is:

%global envname nwra

Name:           pyvenv-%{envname}
Version:        1.0
Release:        2%{?dist}
Summary:        NWRA python environment
License:        GPLv3+

BuildRequires:  python-virtualenv
%if 0%{?rhel} && 0%{?rhel} == 6
Requires:       environment-modules
%else
Requires:       environment(modules)
%endif

%description
NWRA python environment.

%package devel
Summary:        NWRA python environment development files
Requires:       %{name}%{?_isa} = %{version}-%{release}

%description devel
NWRA python environment development files.

%install
mkdir -p %{buildroot}/opt/pyvenv/%{envname}-%{version}
%if 0%{?rhel} && 0%{?rhel} == 6
source /opt/rh/python27/enable
%endif
virtualenv --system-site-packages --no-setuptools %{buildroot}/opt/pyvenv/%{envname}-%{version}
virtualenv --relocatable %{buildroot}/opt/pyvenv/%{envname}-%{version}
# Fixup buildroot
find %{buildroot} -type f -exec sed -i -e 's|%{buildroot}||g' '{}' +

# Needs to be in /etc to override EL python macros
mkdir -p %{buildroot}/etc/rpm
cat > %{buildroot}/etc/rpm/macros.zz-nwra-pyvenv << EOF
%%__python2 /opt/pyvenv/%{envname}-%{version}/bin/python2
%%pyvenv_name_prefix pyvenv-%{envname}-
EOF

mkdir -p %{buildroot}/usr/share/Modules/modulefiles/pyvenv/%{envname}
cat > %{buildroot}/usr/share/Modules/modulefiles/pyvenv/%{envname}/%{version} << EOF
#%Module 1.0

set prefix /opt/pyvenv/%{envname}-%{version}

prepend-path    PATH    $$prefix/bin
prepend-path    PYTHONPATH    $$prefix/lib/python2.7/site-packages
EOF


%files
/usr/share/Modules/modulefiles/pyvenv/%{envname}/%{version}
/opt/pyvenv/%{envname}-%{version}

%files devel
/etc/rpm/macros.zz-nwra-pyvenv

We install a virtual environment with --system-site-packages enabled and then strip out the buildroot paths. We also create a -devel package with a rpm macro file to override the %{__python2} macro for when we build dependent packages. Finally, we create an environment module file so our users can do:

module load pyvenv/nwra

to load the python environment.

Mock configuration

We use mock to build our packages. In order to have our environment in place we add to our config:

config_opts['chroot_setup_cmd'] = 'install @buildsys-build pyvenv-nwra-devel'

so we automatically pull in pyvenv-nwra-devel.

If you use COPR, you can add it to the "Additional chroot packages" list.

Building dependent packages

We want to modify the packages we build so that they use the "pyvenv-nwra-" namespace. This involves prefixing the old package name with "pyvenv-nwra-". We also need to change the name of any dependencies we have already built.

This does involve more modifications than I would have liked, but without it system packages and newer pyvenv-nwra- (or other pyvenv-*) packages could not co-exist. Using conditional macros though allows to the packages to build nomally in the normal build environments.

Example diff:

--- a/python-tornado.spec
+++ b/python-tornado.spec
@@ -4,7 +4,7 @@

 %global pkgname tornado

-Name:           python-%{pkgname}
+Name:           %{?pyvenv_name_prefix}python-%{pkgname}
 Version:        4.1
 Release:        1%{?dist}
 Summary:        Scalable, non-blocking web server and tools
@@ -38,16 +38,16 @@ ideal for real-time web services.
 %package doc
 Summary:        Examples for python-tornado
 Group:          Documentation
-Requires:       python-tornado = %{version}-%{release}
+Requires:       %{?pyvenv_name_prefix}python-tornado = %{version}-%{release}

 %description doc
 Tornado is an open source version of the scalable, non-blocking web
 server and and tools. This package contains some example applications.

 %if 0%{?with_python3}
-%package -n python3-tornado
+%package -n %{?pyvenv_name_prefix}python3-tornado
 Summary:        Scalable, non-blocking web server and tools
-%description -n python3-tornado
+%description -n %{?pyvenv_name_prefix}python3-tornado
 Tornado is an open source version of the scalable, non-blocking web
 server and tools.

@@ -57,12 +57,12 @@ reasonably fast. Because it is non-blocking and uses epoll, it can
 handle thousands of simultaneous standing connections, which means it is
 ideal for real-time web services.

-%package -n python3-tornado-doc
+%package -n %{?pyvenv_name_prefix}python3-tornado-doc
 Summary:        Examples for python-tornado
 Group:          Documentation
-Requires:       python3-tornado = %{version}-%{release}
+Requires:       %{?pyvenv_name_prefix}python3-tornado = %{version}-%{release}

-%description -n python3-tornado-doc
+%description -n %{?pyvenv_name_prefix}python3-tornado-doc
 Tornado is an open source version of the scalable, non-blocking web
 server and and tools. This package contains some example applications.

@@ -132,13 +132,13 @@ popd
 %doc python2/demos

 %if 0%{?with_python3}
-%files -n python3-tornado
+%files -n %{?pyvenv_name_prefix}python3-tornado
 %doc python3/README.rst python3/PKG-INFO

 %{python3_sitearch}/%{pkgname}/
 %{python3_sitearch}/%{pkgname}-%{version}-*.egg-info

-%files -n python3-tornado-doc
+%files -n %{?pyvenv_name_prefix}python3-tornado-doc
 %doc python3/demos
 %endif

I'm hoping this can be kept fairly easy in a separate git branch.

SCL rats nest

I took a bad turn trying to use the python27 SCL for EL6, since ipython requires it. However, this completely isolates the python install from the system one and so completely changes the rpm namespace. Everything needs to get built for the new python, and every rpm requires needs to get changed. The solution there may be the quick and dirty approach below:

Quick and dirty packages

I started out with the following approach, but then decided that I wanted something more modular. But if you just want to build up a quick pyvenv you could do something like the following. The major drawback to this approach is that you must do everything in one go - you can't build other packages later on top of this.

%global envname ipython

Name:           pyvenv-%{envname}
Version:        3.1.0
Release:        1%{?dist}
Summary:        An enhanced interactive Python shell

License:        (BSD and MIT and Python) and GPLv2+
URL:            http://ipython.org/

BuildRequires:  python-virtualenv

%install
mkdir -p %{buildroot}/opt/pyvenv/%{envname}-%{version}
virtualenv -v --system-site-packages %{buildroot}/opt/pyvenv/%{envname}-%{version}
source %{buildroot}/opt/pyvenv/%{envname}-%{version}/bin/activate
pip install --upgrade pip
pip install -v --no-use-wheel '%{envname}[all]'
echo y | pip -q uninstall setuptools
echo y | pip -q uninstall pip
deactivate
virtualenv --relocatable %{buildroot}/opt/pyvenv/%{envname}-%{version}
# Fixup buildroot
find %{buildroot} -type f -exec sed -i -e 's|%{buildroot}||g' '{}' +

%files
/opt/pyvenv/%{envname}-%{version}

and if you really wanted to try to leverage system packages for requirements you could add:

BuildRequires:  gcc-c++
BuildRequires:  python-jinja2
BuildRequires:  python-jsonschema >= 2.0
BuildRequires:  python-mistune >= 0.5
BuildRequires:  python-mock
BuildRequires:  python-nose >= 0.10.1
%if 0%{?fedora}
BuildRequires:  python-numpydoc
%endif
BuildRequires:  python-pygments
BuildRequires:  python-requests
BuildRequires:  python-sphinx >= 1.1
# Not in Fedora
#BuildRequires:  python-terminado >= 0.3.3
%if 0%{?fedora} >= 23
BuildRequires:  python-tornado >= 4.0
%endif
BuildRequires:  python-zmq >= 13
# Need to specify requires satisfied by the system
Requires:       python-jinja2
Requires:       python-jsonschema >= 2.0
Requires:       python-mistune >= 0.5
Requires:       python-mock
Requires:       python-nose >= 0.10.1
%if 0%{?fedora}
Requires:       python-numpydoc
%endif
Requires:       python-pygments
Requires:       python-requests
Requires:       python-sphinx >= 1.1
# Not in Fedora
#Requires:       python-terminado >= 0.3.3
%if 0%{?fedora} >= 23
Requires:       python-tornado >= 4.0
%endif
Requires:       python-zmq >= 13

Free Thoughts

Tuesday, May 12, 2015

Compiling numpy 1.9.2 with Intel compiler on EL7

Thursday, May 7, 2015

Deploying A Scientific Python Environment