Blog moved!

This blog has been moved to: https://iscinumpy.gitlab.io

Announcing GooFit 2.0

The next version of the CUDA/OpenMP fitting program for HEP analysis, GooFit 2.0, has been released. GooFit is now easy to build on a wide variety of Unix systems, and supports debuggers and IDEs. GooFit is faster, has unit tests, and working examples. More PDFs and examples have been added, as well as newly released example datasets that are downloaded automatically. GooFit now has built in support for MPI, and can use that to deploy to multiple graphics cards on the same machine. A new command line parser (CLI11) and drastically improved logging and errors have made code easier to write and debug. Usage of GooFit specific terminology is now reduced, using standard Thrust or CUDA terms when possible, lowering the barrier for new developers. A new Python script has been added to assist users converting from pre 2.0 code.

Announcing CLI11 Version 1.0

CLI11, a powerful library for writing command line interfaces in C++11, has just been released. There are no requirements beyond C++11 support (and even <regex> support not required). It works on Mac, Linux, and Windows, and has 100% test coverage on all three systems. You can simply drop in a single header file (CLI11.hpp available in releases) to use CLI11 in your own application. Other ways to integrate it into a build system are listed in the README.

Perfect forwarding for methods

I often see perfect forwarding listed for constructor arguments, but not usually for functions with a return or methods. Here is my solution for an method method of class Cls

template<typename ...Args>
static auto method(Cls* cls, Args &&  ...args)
  -> typename std::result_of<decltype(&Cls::method)(Cls, Args...)>::type {
    return cls->method(std::forward<Args>(args)...);
}

This is useful if you want to call protected classes from a “helper” friend class, for example, to expose them to tests without having to require GoogleTest/GoogleMock to be avaible for regular users.

Lua Environment Modules

This is a guide to setting up Lmod (lua environment modules) on a CentOS system. I’ve used a similar procedure to set them up on a Mac, as well, so this is still a useful guide to the workings of Lmod if you use a different system; mostly paths will change. On a Mac, you’ll want to install Lmod from the science tap in brew.

There are several good pages covering environment modules (TCL version), but not many that use the newer Lua syntax. This document aims to fill that roll.

Setting up environment modules

Note: please use Lmod instead. It is a newer project that is backward compatible, but also makes some nice improvements. It is used on many supercomputer systems. See my next article.

The environment modules project is an ideal way to set up (albeit mostly manually) your environment for multiple packages. This is a quick guide on setting it up.

First, install the environment-modules (CentOS) package with yum. You’ll also need to initialize it in the bashrc file, I chose to add source /usr/share/Modules/init/bash (and optionally bash_completion) to /etc/bashrc instead of running /usr/share/Modules/bin/add.modules, but if you only want it locally, that’s also an option.

To set up a module file, you want something like this, for example /usr/share/Modules/modulefiles/cuda/8.0:

#%Module1.0
proc ModulesHelp { } {
        global version prefix name
        puts stderr "$name/$version - loads the environment for $name, in $prefix"
}

set     name      cuda
set     version   8.0

module-whatis   "loads the $name environment"

set prefix /usr/local/cuda-$version

prepend-path     LD_LIBRARY_PATH     $prefix/lib64
prepend-path     PATH                $prefix/bin

You can generate the meat of this file by doing, for example for ROOT:

/usr/share/Modules/bin/createmodule.py /opt/root-6.08.02/bin/thisroot.sh > /usr/share/Modules/modulefiles/root/6.08.02

You can remove most or all the default modules - and yes you’ll need to make modules for each package. The “spider” search does not seem to be in the standard modules package.

Note: I used http://dillinger.io to generate the this page.

GoogleTest and CMake

This is a quick recipe for setting up CMake to use googletest in your projects. First, make a tests folder in the root of your project. Then, add add_subdirectory(tests) to your CMakeLists.txt, after you've finished adding the libraries in your project. Note that the way I've written this probably requires CMake 3.4+.

The CMakeLists.txt file in tests should look like this:

set(THREADS_PREFER_PTHREAD_FLAG ON)
find_package(Threads REQUIRED)

This adds the Threads::Threads target that we can link to, to enable the threading support that GTest requires. On some systems, it is important to use the -pthread flag, so this does that if necessary.

include(ExternalProject)

ExternalProject_Add(
    gtest
    URL http://googletest.googlecode.com/files/gtest-1.7.0.zip
    PREFIX ${CMAKE_CURRENT_BINARY_DIR}/gtest
    URL_MD5 2d6ec8ccdf5c46b05ba54a9fd1d130d7
    INSTALL_COMMAND ""
)

ExternalProject_Get_Property(gtest source_dir binary_dir)

We have to add an external property, to get CMake to download and build GTest for us. We also need to get the source directory and binary directory for use in linking.

add_library(libgtest INTERFACE)
add_dependencies(libgtest gtest)
target_link_libraries(libgtest
    INTERFACE Threads::Threads
              "${binary_dir}/libgtest_main.a"
              "${binary_dir}/libgtest.a")
target_include_directories(libgtest INTERFACE "${source_dir}/include")

Hopefully these lines are familiar to you; they are setting up a special target that we aren't "building", but are using. The target libgtest is simply an interface (no building), and is dependent on gtest (That has to be built first). The link and include commands set up the dependencies so that future target_link_libraries commands only need this target, and will inherit everything else!

enable_testing()

This prepares CTest to handle the tests. You can either run the binaries, or use "make test" to run the tests through CTest's runner program.

file(GLOB test_cases *.cpp)

Or however you want to collect your test cases.

foreach(case_file ${test_cases})
    get_filename_component( case_name ${case_file} NAME_WE )
    set (case_name test_${case_name})
    add_executable(${case_name} ${case_file})
    target_link_libraries(${case_name} libgtest MyLibrary)
    add_test(NAME ${case_name}
             COMMAND ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/${case_name}
             WORKING_DIRECTORY
             ${PROJECT_BINARY_DIR}
             )

endforeach()

Here, we make the tests, doing two things. Setting up the CTest integration is the bulk of the commands above; the main command is the target_link_libraries, which should have your library target (MyLibrary in this example) and the libgtest target. That gets all the includes and links (and defs, if you have those) set on the test targets. That's it!

A simple introduction to Asyncio

This is a simple explanation of the asyncio module and new supporting language features in Python 3.5. Even though the new keywords async and await are new language constructs, they are mostly* useless without an event loop, and that is supplied in the standard library as asyncio. Also, you need awaitable functions, which are only supplied by asyncio (or in the growing set of async libraries, like asyncssh, quamash etc.).

Note: My previous post uses these without the asyncio event loop. But you should not normally do that.

A little example of how Asyncio works

This is a simple example to show how Asyncio works without using Asyncio itself, instead using a basic and poorly written event loop. This is only meant to give a flavor of what Asyncio does behind the curtains. I'm avoiding most details of the library design, like callbacks, just to keep this simple. Since this is written as an illustration, rather than real code, I'm going to dispense with trying to keep it 2.7 compatible.

Feynman Diagrams in Tikz

There is a package for making Feynman diagrams in LaTeX. Unfortunately, it is old and dvi latex only. If you are using pdflatex or lualatex, as you should be, it does not work. Even in regular LaTeX, it's a bit of a pain. Why is there not a new package for pdflatex? Turns out, you don't need one. Due to the powerful drawing library Tikz, you can create any diagram easily, and can customize it completely. For example:

Including CRY cosmic ray generator in CMake

I realized that CRY did not have a CMake based install option, so including it in a GEANT4 cmake project might not be obvious. This is how you would do it in your CMakeLists.txt:

GTest Submodule

Note: There is a better way to do this described here.

If you've ever tried apt-get or brew to try to install gtest, you arer probably familiar with the fact that gtest is not "recommend" for global install on your system. As an alternitive, the recommendation is that you make it part of your project. The process for making gtest part of your project, however, is not well documented, at least for modern git projects. What follows is the procedure I used to do so.

Slots in Python

Slots seem to be poorly documented. What they do is simple, but whether they are used is tricky. This is a little mini-post on slots.

Basics of Metaclasses

This is a quick tutorial over the basics of what metaclasses do.

The Metaclass¶

Metaclasses, while seemingly a complex topic, really just do something very simple. They control what happens when you have code that turns into a class object. The normal place they are executed is right after the class statement. Let's see that in action by using print as our metaclass.

Note: this post uses Python 3 metaclass notation. Python 2 uses assignment to a special __metaclass__ attribute to set the metaclass. Also, Python 2 requires explicit, 2 argument super() calls.

In [13]:

class WillNotBeAClass(object, metaclass=print):
    x=1

WillNotBeAClass (<class 'object'>,) {'x': 1, '__module__': '__main__', '__qualname__': 'WillNotBeAClass'}

Here, we have replaced the metaclass (type) with print, just to investigate how it works. This is quite useless, of course, but does show that the metaclass gets called with three arguments when a class is created. The first is the name of the class to be created, the second is a tuple of base classes, and the third is a dictionary that has the namespace of the body of a class, with a few extra special values added.

Factory Classmethods in Python

I haven't seen a great deal of practical documentation about using classmethods as factories in Python (which is arguably the most important use of a classmethod, IMO). This post hopes to fill in that gap.

Making a Plumbum Autoload Extension for IPython

I recently decided to try my hand at making an auto-load extension for Python and plumbum. I was planning to suggest it as a new feature, then I thought it might be an experimental feature, and now it's just a blog post. But it was an interesting idea and didn't seem to be well documented process on the web. So, here it is.

The plan was to make commands like this:

>>> from plumbum.cmd import echo
>>> echo("This is echoed!")

or

>>> from plumbum import local
>>> echo = local['echo']
>>> echo("This is echoed!")

into this:

>>> echo("This is echoed!")

Thereby making Plumbum even more like Bash for fast scripting.

Uncertainty Extension for IPython

Wouldn't it be nice if we had uncertainty with a nice notation in IPython? The current method would be to use raw Python,

In [1]:

from uncertainties import ufloat
print(ufloat(12.34,.01))

12.340+/-0.010

Let's use the handy infix library to make the notation easier. We'll define |pm| to mean +/-.

Note: this is a very simple library that is less than a page long. Feel free to write the code yourself (as I do later in this notebook).

Plumbum Color

I've been working on a color addition to Plumbum for a little while, and I'd like to share the basics of using it with you now. This library was originally built around a special str subclass, but now is built on the new Styles representation and is far more powerful than the first implementation. It safely does nothing if you do not have a color-compatible systems (posix + tty currently), but can be forced if need be. It is included with Plumbum, so you don't have to add a requirement for your scripts that is non-essential (as color often is). It is integrated with plumbum.cli, too. Also, I've managed to accelerate the color selection algorithims about 8x, allowing near game-like speeds. (see the fullcolor.py example).

University of Texas Doctoral Thesis Template

I have created a thesis class file for a UT Thesis in LaTeX. It has already been used for at least one passing thesis, so it does meet the current UT guidelines. (Please let me know if there are any issues!)

Since I use Bitbucket for all my private repositories (like my thesis itself), the code is in a Bitbucket repository rather than GitHub. Here is the link if you want to create a pull request or want to compile the class and documentation from the source .dtx file.

If all you want is a download of a working version, and if you don't want to compile the code but just want the class file, here are downloadable packages including the class file.

Plumbum Scripting

Scripting in Bash is a pain. Bash can do almost anything, and is unbeatable for small scripts, but it struggles when scaling up to doing anything close to a real world scripting problem. Python is a natural choice, especially for the scientist who already is using it for analysis. But, it's much harder to do basic tasks in Python. So you are left with scripts starting out as Bash scripts, and then becoming a mess, then being (usually poorly) ported to Python, or even worse, being run by a Python script. I've seen countless Python scripts that run Bash scripts that run real programs. I've even written one or two. It's not pretty.

I recently came (back) across a really powerful library for doing efficient command line scripts in Python. It contains a set of tools that makes the four (five with color) main tasks of command line scripts simple and powerful. I will also go over the one main drawback of the library (and the possible enhancement!).