craneium.net

brain matters . . .

  • Increase font size
  • Default font size
  • Decrease font size
Home Python Programming
Python Programming

Installing OpenCV 2.0 for Python Fun

E-mail Print PDF

OpenCV has a lot of nifty stuff built into it.  The install is pretty easy but has a few small hitches worth mentioning.

First download the .tar.bz from here: link

Install cmake under Ubuntu/Debian, untar the directory, cd in, and run cmake.


$sudo apt-get install cmake
$tar xfj OpenCV-2.0.0.tar.gz
$cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local -D BUILD_PYTHON_SUPPORT=ON .

Don't omit the dot at the end!  Next:


$make
$make install

From there I ran into the small problem that it installs to the depreciated "site-packages" instead of "dist-packages" so:


$cd /usr/local/lib/python2.6/site-packages
$sudo cp cv.so ../dist-packages

From there everything worked wonderfully!

 

 

Last Updated on Monday, 08 March 2010 17:44
 

Compiling PyTables, parallel iPython, and Trilinos 10 for PyTrilinos LOCALLY From Almost Nothing

E-mail Print PDF

(Update iPython parallel is now very different: See Here For More)

 

This is instructions for how to do a LOCAL install of parallel ipython, pyTables, and Trilinos 10 for use with pytrilinos on a SGI Altix HPC with only the following available:

gccgfortranblaslapackfftw, tar, wget

My parallel n-dimensional fluid dynamics solver relies on pyTables, parallel ipython, and now pyTrilinos so here is how to download/compile/build/install everything needed!  This is a rough chart of the dependencies running around:

General notes:

These instructions are designed for a local install, meaning you don't have root access to the machine in question.  I use the path of "/home/u27/mgoodman/POST" but any directory that you have write access to will do.

I assume that after wget'ing a particular package, you can untar and change into the resulting directory, so I omit this instruction in almost all steps.

The make steps can be accelerated by the addition of the "-j" flag on computers with more than one core.  This is especially key during the Trilinos make step (which can take over a couple hours and then fail)

 

Installing zlib with shared libraries:

$wget http://www.zlib.net/zlib-1.2.3.tar.gz

$./configure --prefix=/home/u27/mgoodman/POST 

$make

$make install

Now we need to make the shared libraries!

$make clean

$./configure -s --prefix=/home/u27/mgoodman/POST

$make

$make install

Installing bz2 including shared objects:

$wget http://www.bzip.org/1.0.5/bzip2-1.0.5.tar.gz

$./configure

$make

$make -f Makefile-libbz2_so

$make install PREFIX=/home/u27/mgoodman/POST

At this point I added the following lines to my ".profile" file:

PATH=/home/u27/mgoodman/POST/bin:$PATH

INCLUDE=/home/u27/mgoodman/POST/include:$INCLUDE

LD_LIBRARY_PATH=/home/u27/mgoodman/POST/lib:$LD_LIBRARY_PATH

Restart your shell session to ensure the libraries, includes and binaries we create are preferred to any (probably wrong) ones that are already installed

Compiling and Installing python2.6:

$wget http://python.org/ftp/python/2.6.4/Python-2.6.4.tar.bz2

$./configure --prefix=/home/u27/mgoodman/POST

$make

$make install

Huzzah!  Not even halfway there though!

 

Installing setuptools:

$wget http://pypi.python.org/packages/source/s/setuptools/setuptools-0.6c11.tar.gz

$python setup.py config

$python setup.py build

$python setup.py install

 

Installing ipython dependancies for parallel computing:

$easy_install zope.Interface
$easy_install Twisted
$easy_install nose
$easy_install sphinx
$easy_install Foolscap
$easy_install pexpect

 

Installing OpenSSL for pyOpenSSL

$wget http://pypi.python.org/packages/source/p/pyOpenSSL/pyOpenSSL-0.10.tar.gz

$./config --prefix=/home/u27/mgoodman/POST

$make

$make install

$easy_install pyOpenSSL

 

Compiling and Installing IPython:

$wget http://ipython.scipy.org/dist/0.10/ipython-0.10.tar.gz

$python setup.py config

This should produce output that looks like:

BUILDING IPYTHON

python: 2.6.4 (r264, Jan 17 2010, 17:21:54)  [GCC 4.4.0]

platform: linux2

OPTIONAL DEPENDENCIES

Zope.Interface: yes

Twisted: 9.0.0

Foolscap: 0.4.2

OpenSSL: 0.10

sphinx: 0.6.4

pygments: 1.2.2

nose: 0.11.1

pexpect: 2.4

If any of the optional modules are missing the parallel computing functionalities will not be present!  Now build and install:

$python setup.py build

$python setup.py install

 

Installing Numpy:

$wget http://sourceforge.net/projects/numpy/files/NumPy/1.4.0/numpy-1.4.0.tar.gz/download
$python setup.py config
(make sure it finds BLAS/LAPACK)
$python setup.py build
$python setup.py install

 

Installing Swig:

$wget http://prdownloads.sourceforge.net/swig/swig-1.3.40.tar.gz

$./configure

$make

$make install

Installing SuiteSparse (contains UMFPACK)

$wget http://www.cise.ufl.edu/research/sparse/SuiteSparse/current/SuiteSparse.tar.gz

in "UFconfig/UFconfig.mk"
Replace
"CHOLMOD_CONFIG ="
with
"CHOLMOD_CONFIG = -DNPARTITION"
And replace
"CFLAGS ="
with
"CFLAGS = -O3 -fexceptions -ffast-math -funroll-loops -fPIC"

If you need to do direct solution of large matrices (1M+ DOF) consider adding the -DL64 flag (this will slow smaller matrix solutions though and at this size you should use the Trilinos iterative solvers)

In the main directory of SuiteSparse:

$make

(It failed to build the "demos" but everything else seemed to work fine)

$make install PREFIX=/home/u27/mgoodman/POST

 

Compiling and Installing Scipy:

Scipy is not strictly necessary for trilinos, but if it would be a strange situation where you are using one and not the other!

$wget http://sourceforge.net/projects/scipy/files/scipy/0.7.1/scipy-0.7.1.tar.gz/download

Untar and edit the site.cfg.example and save as site.cfg

The section regarding fftw, umfpack, and amd need to be edited.  The lines changed for me look like the following:

[amd]

library_dirs = /home/u27/mgoodman/local-install/SuiteSparse/AMD/Lib

include_dirs = /home/u27/mgoodman/local-install/SuiteSparse/AMD/Include/

amd_libs = amd

 

[umfpack]

library_dirs = /home/u27/mgoodman/local-install/SuiteSparse/UMFPACK/Lib/

include_dirs = /home/u27/mgoodman/local-install/SuiteSparse/UMFPACK/Include/

umfpack_libs = umfpack

Run:

$python setup.py config

Make sure it finds UMFPACK includes and libraries.  The output should look something like this:

lapack_info:

libraries lapack not found in /home/u27/mgoodman/POST/lib

libraries lapack not found in /usr/local/lib64

FOUND:

libraries = ['lapack']

library_dirs = ['/usr/local/lib']

language = f77

 

FOUND:

libraries = ['lapack', 'blas']

library_dirs = ['/usr/local/lib']

define_macros = [('NO_ATLAS_INFO', 1)]

language = f77

 

umfpack_info:

amd_info:

FOUND:

libraries = ['amd']

library_dirs = ['/home/u27/mgoodman/local-install/SuiteSparse/AMD/Lib']

swig_opts = ['-I/home/u27/mgoodman/local-install/SuiteSparse/AMD/Include']

define_macros = [('SCIPY_AMD_H', None)]

include_dirs = ['/home/u27/mgoodman/local-install/SuiteSparse/AMD/Include']

 

FOUND:

libraries = ['umfpack', 'amd']

library_dirs = ['/home/u27/mgoodman/local-install/SuiteSparse/UMFPACK/Lib/', '/home/u27/mgoodman/local-install/SuiteSparse/AMD/Lib']

swig_opts = ['-I/home/u27/mgoodman/local-install/SuiteSparse/UMFPACK/Include', '-I/home/u27/mgoodman/local-install/SuiteSparse/AMD/Include']

define_macros = [('SCIPY_UMFPACK_H', None), ('SCIPY_AMD_H', None)]

include_dirs = ['/home/u27/mgoodman/local-install/SuiteSparse/UMFPACK/Include', '/home/u27/mgoodman/local-install/SuiteSparse/AMD/Include']

If that looks good proceed with:

$python setup.py build

(this will take a while)

$python setup.py install

Compiling and Installing hdf5 libraries:

$wget http://www.hdfgroup.org/ftp/HDF5/current16/src/hdf5-1.6.10.tar.gz

$./configure --prefix=/home/u27/mgoodman/POST

$make

$make install

 

Compiling and Installing CMake:

$wget http://www.cmake.org/files/v2.8/cmake-2.8.0.tar.gz

untar and enter directory

$./configure --prefix=/home/u27/mgoodman/POST

$make

$make install

 

Compiling and Installing pytables:

For whatever reason this particular installer ignores the standard INCLUDE and LD_LIBRARY_PATH environment vars, so we need to do some hacking if we want a local install to work.

Open the setup.py and make some changes . . .

In the section starting with:

if os.name == 'posix':

default_header_dirs = ['/usr/include', '/usr/local/include', '/home/u27/mgoodman/POST/include']

default_library_dirs = [

os.path.join(_tree, _arch)

for _tree in ('/', '/usr', '/usr/local')

for _arch in ('lib64', 'lib') ]

default_library_dirs.append('/home/u27/mgoodman/POST/lib')

default_runtime_dirs = default_library_dirs

 

Now we can build install properly:

$python setup.py config --cflags="-I/home/u27/mgoodman/POST/include/" --lflags="-L/home/u27/mgoodman/POST/lib"

$python setup.py build --cflags="-I/home/u27/mgoodman/POST/include/" --lflags="-L/home/u27/mgoodman/POST/lib"

$python setup.py install --cflags="-I/home/u27/mgoodman/POST/include/" --lflags="-L/home/u27/mgoodman/POST/lib" --prefix=/home/u27/mgoodman/POST/

 

Compiling and Installing Trilinos 10:

$wget http://trilinos.sandia.gov/download/files/trilinos-10.0.4-Source.tar.gz

Make sure you have a good amount of disk space, this build takes on the order of 1.5Gb, so I had to make some disk space arrangements on my allocation.
Create a file named config-trilinos.sh containing:

cmake \

-D BUILD_SHARED_LIBS=ON   \

-D CMAKE_INSTALL_PREFIX=/home/u27/mgoodman/POST \

-D CMAKE_BUILD_TYPE:STRING=DEBUG \

-D TPL_ENABLE_MPI:BOOL=ON \

-D Trilinos_ENABLE_DEFAULT_PACKAGES:BOOL=OFF \

-D Trilinos_ENABLE_PyTrilinos:BOOL=ON \

-D Trilinos_ENABLE_Meros:BOOL=ON \

-D Trilinos_ENABLE_Epetra:BOOL=ON \

-D Trilinos_ENABLE_EpetraExt:BOOL=ON \

-D Trilinos_ENABLE_Belos:BOOL=ON \

-D Trilinos_ENABLE_Ifpack:BOOL=ON \

-D Trilinos_ENABLE_Teuchos:BOOL=ON \

-D Trilinos_ENABLE_Amesos:BOOL=ON \

-D Trilinos_ENABLE_Gallery:BOOL=ON \

-D Trilinos_ENABLE_ML:BOOL=ON \

-D Trilinos_ENABLE_ForTrilinos:BOOL=ON \

-D Trilinos_ENABLE_TESTS:BOOL=ON \

-D Teuchos_ENABLE_MPI:BOOL=ON \

-D Teuchos_ENABLE_ABC:BOOL=ON \

-D Teuchos_ENABLE_COMPLEX:BOOL=ON \

-D Teuchos_ENABLE_EXTENDED:BOOL=ON \

-D DART_TESTING_TIMEOUT:STRING=600 \

-D Trilinos_EXTRA_LINK_FLAGS:STRING="$FORTRAN_LIBRARIES" \

-D CMAKE_VERBOSE_MAKEFILE:BOOL=TRUE       \

-D Trilinos_VERBOSE_CONFIGURE:BOOL=TRUE   \

.

So finally

$bash config-trilinos.sh

$make

$make install

Now go get some real work done!
Last Updated on Thursday, 19 May 2011 18:09
 

Python's __del__ Function

E-mail Print PDF

I had seen people use Python's __del__ method in classes, but upon using it myself, I discovered it is very broken.  See the following:

from scipy import *

import time

class Bork:

def __init__(self):

print "I'm Alive"

def __del__(self):

print "I'm Dead"

raise ValueError("WTF")

 

b = Bork()

del b

print "I am undead"

Which results in the following:
meawoppl@leeloo:~/ndsolver/ndsolver$ python what.py
I'm Alive
I'm Dead
Exception ValueError: ValueError('WTF',) in <bound method Bork.__del__ of 
<__main__.Bork instance at 0x1db2878>> ignored
I am undead
Exceptions in the desctuctor just get printed to the standard error?  Alright, although this is documented it is a pretty serious departure from typical Python described here:

 

Due to the precarious circumstances under which __del__() methods are invoked, exceptions that occur during their execution are ignored, and a warning is printed to sys.stderrinstead. Also, when __del__() is invoked in response to a module being deleted (e.g., when execution of the program is done), other globals referenced by the __del__() method may already have been deleted or in the process of being torn down (e.g. the import machinery shutting down). For this reason, __del__() methods should do the absolute minimum needed to maintain external invariants. Starting with version 1.5, Python guarantees that globals whose name begins with a single underscore are deleted from their module before other globals are deleted; if no other references to such globals exist, this may help in assuring that imported modules are still available at the time when the __del__() method is called.

So in summary:

1.) __del__ may  or may not have access to methods of the class.

2.) __del__ methods have an implicit try/except/print so exceptions can not be handled externally.

3.) Don't use this method if you can avoid it!

Last Updated on Friday, 15 January 2010 16:21
 

Compiling PyTrilinos for i7 with MPI Support

E-mail Print PDF

I used the following config to get pytrilinos working on the new office computer.  Next up HPC.

cmake \
-D CMAKE_BUILD_TYPE:STRING=DEBUG \
-D BUILD_SHARED_LIBS=ON   \
-D TPL_ENABLE_MPI:BOOL=ON \
-D Trilinos_ENABLE_DEFAULT_PACKAGES:BOOL=OFF \
-D Trilinos_ENABLE_PyTrilinos:BOOL=ON \
-D Trilinos_ENABLE_Meros:BOOL=ON \
-D Trilinos_ENABLE_Epetra:BOOL=ON \
-D Trilinos_ENABLE_EpetraExt:BOOL=ON \
-D Trilinos_ENABLE_Belos:BOOL=ON \
-D Trilinos_ENABLE_Ifpack:BOOL=ON \
-D Trilinos_ENABLE_Teuchos:BOOL=ON \
-D Trilinos_ENABLE_Amesos:BOOL=ON \
-D Trilinos_ENABLE_Gallery:BOOL=ON \
-D Trilinos_ENABLE_ML:BOOL=ON \
-D Trilinos_ENABLE_ForTrilinos:BOOL=ON \
-D Trilinos_ENABLE_TESTS:BOOL=ON \
-D Teuchos_ENABLE_MPI:BOOL=ON \
-D Teuchos_ENABLE_ABC:BOOL=ON \
-D Teuchos_ENABLE_Boost:BOOL=ON \
-D Teuchos_ENABLE_COMPLEX:BOOL=ON \
-D Teuchos_ENABLE_EXTENDED:BOOL=ON \
-D DART_TESTING_TIMEOUT:STRING=600 \
-D Trilinos_EXTRA_LINK_FLAGS:STRING="$FORTRAN_LIBRARIES" \
-D CMAKE_VERBOSE_MAKEFILE:BOOL=TRUE       \
-D Trilinos_VERBOSE_CONFIGURE:BOOL=TRUE   \
.

Last Updated on Friday, 15 January 2010 16:23
 

Compiling ipython for Cloud Computing Fun

E-mail Print PDF

Ipython now has a powerful cloud computing environment built into it. Python is becoming a very common cloud computing platform. It is seen in Amazon’s compute cloud, and the Google App Engine is purely Python.

Also notably, it seems that future versions of Ubuntu will ship with such capabilities!

This post will hopefully tell you everything you need to install similar capabilities and build a cloud of your own using ipython!

Installation on Ubuntu Gusty/Hardy/Intrepid:

First you need to get the various code bits that python modules will depend on:

$sudo apt-get install libssl-dev python-setuptools

This will install both the openssl crpytography libraries and the “easy_install” tool, that makes getting and updating python modules really effortless. There is an ongoing battle between this utility and “pip“, but that discussion is well beyond the scope of our post today. I perfer easy_install becuase it is exactly what is advertises.

Next up, we “easy_install” everything we need that ipython’s parallel capabilities will depend on:

$sudo easy_install foolscap nose pexpect pyopenssl sphinx

Depending on what you already have installed, this may take some time.

There are a lot of dependencies to ipython, so to make sure we got them all right, we grab the current ipython source:

$mkdir ipython_build
$cd ipython_build
$wget http://ipython.scipy.org/dist/ipython-0.9.1.tar.gz
$tar xfz ipython-0.9.1.tar.gz
$cd ipython-0.9.1
$./setup.py config

At this point ought to see something like:

Zope.Interface: yes
Twisted: 8.1.0
Foolscap: 0.3.2
OpenSSL: 0.8
sphinx: 0.5.1
pygments: 1.0
nose: 0.10.4
pexpect: 2.1

If you are missing something essential to ipython’s build, it will let you know :p Now that we know we have everything necessary we can go ahead with either the typical build/install whatnot from this source directory, or simply:

$cd ../..
$rm -rf ipython_build
$sudo easy_install -U ipython

And you should be all set to go!

Under *nix distros the following is a good test as to whether everything is functioning properly:

$ipcluster -n 4

Based on what I have seen the OSX / Windows install ought to be fairly similar. You might have to grab the openssl source and compile pyopenssl on you own, pointing the linker to the right directories. If anyone has any luck doing this in OSX / Windows, let me know and I’ll post an update. Some links that might be helpful ( foolscap nose pexpect pyopenssl sphinx openssl-source)

 

Last Updated on Wednesday, 13 January 2010 14:25
 


Page 2 of 3