tag:blogger.com,1999:blog-83038931185978996542024-02-19T04:20:36.196-08:00iSciNumPyBlog moved to : https://iscinumpy.gitlab.ioAnonymoushttp://www.blogger.com/profile/10752018806658508906noreply@blogger.comBlogger22125tag:blogger.com,1999:blog-8303893118597899654.post-69644190362749228722017-11-10T13:44:00.004-08:002017-11-10T13:45:23.749-08:00Blog moved!<h4>
This blog has been moved to: <a href="https://iscinumpy.gitlab.io/">https://iscinumpy.gitlab.io</a></h4>
Anonymoushttp://www.blogger.com/profile/10752018806658508906noreply@blogger.com0tag:blogger.com,1999:blog-8303893118597899654.post-50288224751589876962017-06-08T17:54:00.001-07:002017-06-09T13:41:40.325-07:00Announcing GooFit 2.0<span style="background-color: white; font-family: helvetica, arial, sans-serif; font-size: 14px;">The next version of the CUDA/OpenMP fitting program for HEP analysis, GooFit 2.0, </span><a href="file:///Users/henryiii/Dropbox/UC/notes/GooFit" style="color: #4183c4; font-family: helvetica, arial, sans-serif; font-size: 14px;">has been released</a><span style="background-color: white; font-family: helvetica, arial, sans-serif; font-size: 14px;">. GooFit is now easy to build on a wide variety of Unix systems, and supports debuggers and IDEs. GooFit is faster, has unit tests, and working examples. More PDFs and examples have been added, as well as newly released example datasets that are downloaded automatically. GooFit now has built in support for MPI, and can use that to deploy to multiple graphics cards on the same machine. A new command line parser (</span><a href="https://github.com/CLIUtils/CLI11" style="color: #4183c4; font-family: helvetica, arial, sans-serif; font-size: 14px;">CLI11</a><span style="background-color: white; font-family: helvetica, arial, sans-serif; font-size: 14px;">) and drastically improved logging and errors have made code easier to write and debug. Usage of GooFit specific terminology is now reduced, using standard Thrust or CUDA terms when possible, lowering the barrier for new developers. A new Python script has been added to assist users converting from pre 2.0 code.</span><br /><a name='more'></a><br />
<div style="-webkit-print-color-adjust: exact; background-color: white; font-family: Helvetica, arial, sans-serif; font-size: 14px; margin-bottom: 15px; margin-top: 15px;">
The file structure of GooFit and the build system have been completely revamped. The fake <code style="-webkit-print-color-adjust: exact; background-color: #f8f8f8; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(234, 234, 234); margin: 0px 2px; padding: 0px 5px; white-space: nowrap;">nvcc</code> features have been removed, as have the <code style="-webkit-print-color-adjust: exact; background-color: #f8f8f8; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(234, 234, 234); margin: 0px 2px; padding: 0px 5px; white-space: nowrap;">rootstuff</code> copies of ROOT classes. PDFs are now organized by type and compile and link separately. Multiple PDF caching support has improved. The build system now uses CMake and manages external libraries.</div>
<div style="-webkit-print-color-adjust: exact; background-color: white; font-family: Helvetica, arial, sans-serif; font-size: 14px; margin-bottom: 15px; margin-top: 15px;">
A new feature of the CMake build system is GooFit Packages, which are complete packages that can be added to GooFit and built, allowing analysis code to live in a separate location from GooFit, rather than the old method of simply forking GooFit and adding your analysis manually. A GooFit Package can be made into an example trivially. See <a href="https://github.com/maddocbf/goofit_KKPiPi" style="-webkit-print-color-adjust: exact; color: #4183c4;">this package</a> for an example.</div>
<div style="-webkit-print-color-adjust: exact; background-color: white; font-family: Helvetica, arial, sans-serif; font-size: 14px; margin-bottom: 15px; margin-top: 15px;">
GooFit 2.0 will receive continuing support while development on GooFit 2.1 presses on.</div>
<h4 id="toc_1" style="-webkit-font-smoothing: antialiased; -webkit-print-color-adjust: exact; background-color: white; cursor: text; font-family: Helvetica, arial, sans-serif; font-size: 16px; margin: 20px 0px 10px; padding: 0px; position: relative;">
Links:</h4>
<div style="-webkit-print-color-adjust: exact; background-color: white; font-family: Helvetica, arial, sans-serif; font-size: 14px; margin-bottom: 15px; margin-top: 15px;">
<a href="https://github.com/GooFit/GooFit" style="-webkit-print-color-adjust: exact; color: #4183c4;">GooFit on GitHub</a> • <a href="https://goofit.github.io/" style="-webkit-print-color-adjust: exact; color: #4183c4;">GooFit webpage</a> • <a href="https://goofit.github.io/GooFit" style="-webkit-print-color-adjust: exact; color: #4183c4;">API documentation</a></div>
<h4 id="toc_2" style="-webkit-font-smoothing: antialiased; -webkit-print-color-adjust: exact; background-color: white; cursor: text; font-family: Helvetica, arial, sans-serif; font-size: 16px; margin: 20px 0px 10px; padding: 0px; position: relative;">
Key features:</h4>
<ul style="-webkit-print-color-adjust: exact; background-color: white; font-family: Helvetica, arial, sans-serif; font-size: 14px; margin: 15px 0px; padding-left: 30px;">
<li style="-webkit-print-color-adjust: exact; margin: 0px;">New 4-body Dalitz plot support and example</li>
<li style="-webkit-print-color-adjust: exact; margin: 0px;">New 4 body signal generation example and time-dependent Dalitz plot generation example powered by <a href="https://github.com/MultithreadCorner/MCBooster" style="-webkit-print-color-adjust: exact; color: #4183c4; margin-top: 0px;">MCBooster</a></li>
<li style="-webkit-print-color-adjust: exact; margin: 0px;">Restructured files with script to aid in renaming includes, available for assisting existing projects in converting to 2.0</li>
<li style="-webkit-print-color-adjust: exact; margin: 0px;">CMake build system: See <a href="https://github.com/GooFit/GooFit/issues/22" style="-webkit-print-color-adjust: exact; color: #4183c4; margin-top: 0px;">Issue 22</a> and <a href="https://github.com/GooFit/GooFit/pull/23" style="-webkit-print-color-adjust: exact; color: #4183c4;">PR 23</a>.<ul style="-webkit-print-color-adjust: exact; margin: 15px 0px; padding-left: 30px;">
<li style="-webkit-print-color-adjust: exact; margin: 0px;">Auto compute capability detection based on detected GPUs</li>
<li style="-webkit-print-color-adjust: exact; margin: 0px;">Auto CUDA/OMP selection </li>
<li style="-webkit-print-color-adjust: exact; margin: 0px;">Added CPP single threaded backend, support for macOS and IDEs/debuggers</li>
<li style="-webkit-print-color-adjust: exact; margin: 0px;">Separable compilation for PDFs</li>
<li style="-webkit-print-color-adjust: exact; margin: 0px;">Support for more compilers, such as Clang and Intel</li>
<li style="-webkit-print-color-adjust: exact; margin: 0px;">Macros for <code style="-webkit-print-color-adjust: exact; background-color: #f8f8f8; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(234, 234, 234); margin: 0px 2px; padding: 0px 5px; white-space: nowrap;">CMakeLists.txt</code> for adding a new package in 2-3 lines</li>
<li style="-webkit-print-color-adjust: exact; margin: 0px;">Auto linking for build directory</li>
<li style="-webkit-print-color-adjust: exact; margin: 0px;">Auto download of dependencies through git submodules and CMake</li>
<li style="-webkit-print-color-adjust: exact; margin: 0px;">Data for examples automatically downloaded</li>
</ul>
</li>
<li style="-webkit-print-color-adjust: exact; margin: 0px;">CUDA 7.0+ required, C++11 compliant compiler required. Large portions of the code have been moved to cleaner C++11 syntax, both by hand and with the <code style="-webkit-print-color-adjust: exact; background-color: #f8f8f8; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(234, 234, 234); margin: 0px 2px; padding: 0px 5px; white-space: nowrap;">clang-tidy</code> tool (<a href="https://github.com/GooFit/GooFit/pull/88" style="-webkit-print-color-adjust: exact; color: #4183c4;">PR 86</a>and <a href="https://github.com/GooFit/GooFit/pull/88" style="-webkit-print-color-adjust: exact; color: #4183c4;">PR 88</a>).</li>
<li style="-webkit-print-color-adjust: exact; margin: 0px;">ROOT 6 recommended, but no longer required</li>
<li style="-webkit-print-color-adjust: exact; margin: 0px;">Fixes for building examples, nicer warnings with incorrect command line parameters</li>
<li style="-webkit-print-color-adjust: exact; margin: 0px;">Rootstuff, fakecuda, and other hacks removed (<a href="https://github.com/GooFit/GooFit/pull/56" style="-webkit-print-color-adjust: exact; color: #4183c4; margin-top: 0px;">PR 56</a>)</li>
<li style="-webkit-print-color-adjust: exact; margin: 0px;">Examples have a script that run all of them with timing info</li>
<li style="-webkit-print-color-adjust: exact; margin: 0px;">Travis CI builds (<a href="https://github.com/GooFit/GooFit/pull/32" style="-webkit-print-color-adjust: exact; color: #4183c4; margin-top: 0px;">PR 32</a>)</li>
<li style="-webkit-print-color-adjust: exact; margin: 0px;">Improved documentation, automatically builds on changes to master</li>
<li style="-webkit-print-color-adjust: exact; margin: 0px;">Added optional <code style="-webkit-print-color-adjust: exact; background-color: #f8f8f8; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(234, 234, 234); margin: 0px 2px; padding: 0px 5px; white-space: nowrap;">GooFit::Application</code>, based on <a href="https://github.com/CLIUtils/CLI11" style="-webkit-print-color-adjust: exact; color: #4183c4;">CLI11</a>, with standard GooFit options and logging output, fully user extendable for new options. See <a href="https://github.com/GooFit/GooFit/pull/36" style="-webkit-print-color-adjust: exact; color: #4183c4;">PR 26</a> and <a href="https://github.com/GooFit/GooFit/issues/33" style="-webkit-print-color-adjust: exact; color: #4183c4;">Issue 33</a>.</li>
<li style="-webkit-print-color-adjust: exact; margin: 0px;">Better naming to match CUDA (<a href="https://github.com/GooFit/GooFit/pull/61" style="-webkit-print-color-adjust: exact; color: #4183c4; margin-top: 0px;">PR 61</a>)</li>
<li style="-webkit-print-color-adjust: exact; margin: 0px;">Added the GooFit namespace to all GooFit classes and variables.</li>
<li style="-webkit-print-color-adjust: exact; margin: 0px;">Better Variable based caching with multi-pdf support (<a href="https://github.com/GooFit/GooFit/pull/65" style="-webkit-print-color-adjust: exact; color: #4183c4; margin-top: 0px;">PR 65</a> and <a href="https://github.com/GooFit/GooFit/pull/68" style="-webkit-print-color-adjust: exact; color: #4183c4;">PR 68</a>)</li>
<li style="-webkit-print-color-adjust: exact; margin: 0px;">Logging and formatting support, cleanup of old commented code (<a href="file:///GooFit/GooFit/pull/66" style="-webkit-print-color-adjust: exact; color: #4183c4; margin-top: 0px;">PR 66</a>)</li>
<li style="-webkit-print-color-adjust: exact; margin: 0px;">Support for Minuit2 (default and available without ROOT) joins Minuit1, rebuilt fitters provide better output and automatic timing information</li>
<li style="-webkit-print-color-adjust: exact; margin: 0px;">CountingVariable added for MPI ready event numbers</li>
<li style="-webkit-print-color-adjust: exact; margin: 0px;">Added MPI support in <a href="https://github.com/GooFit/GooFit/pull/51" style="-webkit-print-color-adjust: exact; color: #4183c4; margin-top: 0px;">PR 51</a>, supporting multiple GPUs per node and multiple nodes</li>
<li style="-webkit-print-color-adjust: exact; margin: 0px;">Added preliminary Python bindings using <a href="http://pybind11.readthedocs.io/en/master/" style="-webkit-print-color-adjust: exact; color: #4183c4; margin-top: 0px;">PyBind11</a></li>
<li style="-webkit-print-color-adjust: exact; margin: 0px;">Started a new tutorial series, <a href="https://henryiii.gitbooks.io/goofit/content/" style="-webkit-print-color-adjust: exact; color: #4183c4; margin-top: 0px;">GooFit 2torial</a>, to replace <a href="https://github.com/GooFit/GooTorial" style="-webkit-print-color-adjust: exact; color: #4183c4;">GooTorial</a></li>
<li style="-webkit-print-color-adjust: exact; margin: 0px;">Added a changelog and version information</li>
</ul>
Anonymoushttp://www.blogger.com/profile/10752018806658508906noreply@blogger.com0tag:blogger.com,1999:blog-8303893118597899654.post-26612138257823625582017-06-01T14:05:00.000-07:002017-06-02T10:20:16.558-07:00Announcing CLI11 Version 1.0<div style="-webkit-print-color-adjust: exact; background-color: white; font-family: Helvetica, arial, sans-serif; font-size: 14px; margin-bottom: 15px; margin-top: 15px;">
CLI11, a powerful library for writing command line interfaces in C++11, has just been released. There are no requirements beyond C++11 support (and even <code style="-webkit-print-color-adjust: exact; background-color: #f8f8f8; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(234, 234, 234); margin: 0px 2px; padding: 0px 5px; white-space: nowrap;"><regex></code> support not required). It works on Mac, Linux, and Windows, and has 100% test coverage on all three systems. You can simply drop in a single header file (<code style="-webkit-print-color-adjust: exact; background-color: #f8f8f8; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(234, 234, 234); margin: 0px 2px; padding: 0px 5px; white-space: nowrap;">CLI11.hpp</code> available in <a href="https://github.com/CLIUtils/CLI11/releases" style="-webkit-print-color-adjust: exact; color: #4183c4;">releases</a>) to use CLI11 in your own application. Other ways to integrate it into a build system are listed in the <a href="https://github.com/CLIUtils/CLI11/blob/master/README.md" style="-webkit-print-color-adjust: exact; color: #4183c4;">README</a>.</div>
<a name='more'></a>
<div style="-webkit-print-color-adjust: exact; background-color: white; font-family: Helvetica, arial, sans-serif; font-size: 14px; margin-bottom: 15px; margin-top: 15px;">
The library was inspired the Python libraries <a href="http://plumbum.readthedocs.io/en/latest/" style="-webkit-print-color-adjust: exact; color: #4183c4;">Plumbum</a> and <a href="http://click.pocoo.org/5/" style="-webkit-print-color-adjust: exact; color: #4183c4;">Click</a>, and incorporates many of their user friendly features. The library is extensively documented, with a <a href="https://github.com/CLIUtils/CLI11/blob/master/README.md" style="-webkit-print-color-adjust: exact; color: #4183c4;">friendly introduction</a>, a tutorial filled (in progress) <a href="https://henryiii.gitbooks.io/cli11/content/" style="-webkit-print-color-adjust: exact; color: #4183c4;">GitBook</a>, and more technical <a href="https://cliutils.github.io/CLI11" style="-webkit-print-color-adjust: exact; color: #4183c4;">API docs</a>.</div>
<div style="-webkit-print-color-adjust: exact; background-color: white; font-family: Helvetica, arial, sans-serif; font-size: 14px; margin-bottom: 15px; margin-top: 15px;">
The syntax is simple and scales from a basic application to a massive physics analysis with multiple models and many parameters and switches. For example, this is a simple program that has an optional parameter that defaults to 1:</div>
<div style="-webkit-print-color-adjust: exact; background-color: white; font-family: Helvetica, arial, sans-serif; font-size: 14px;">
<pre class=" language-none" style="-webkit-hyphens: none; -webkit-print-color-adjust: exact; background-color: #f5f2f0; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(204, 204, 204); font-family: Consolas, Monaco, 'Andale Mono', 'Ubuntu Mono', monospace; font-size: 13px; line-height: 1.5; margin-bottom: 0.5em; margin-top: 0.5em; overflow: auto; padding: 1em; tab-size: 4; text-shadow: white 0px 1px; word-break: normal; word-spacing: normal; word-wrap: normal;"><code class=" language-none" style="-webkit-hyphens: none; -webkit-print-color-adjust: exact; background-image: none; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: none; font-family: Consolas, Monaco, 'Andale Mono', 'Ubuntu Mono', monospace; line-height: 1.5; margin: 0px; padding: 0px; tab-size: 4; text-shadow: white 0px 1px; word-break: normal; word-spacing: normal; word-wrap: normal;">./a.out
Parameter value: 0
./a.out -p 4
Parameter value: 4
./a.out --help
App description
Usage: ./a.out [OPTIONS]
Options:
-h,--help Print this help message and exit
-p INT Parameter</code></pre>
</div>
<div style="-webkit-print-color-adjust: exact; background-color: white; font-family: Helvetica, arial, sans-serif; font-size: 14px; margin-bottom: 15px; margin-top: 15px;">
Like any good command line application, help is provided. This program can be implemented in under 15 lines:</div>
<div style="-webkit-print-color-adjust: exact; background-color: white; font-family: Helvetica, arial, sans-serif; font-size: 14px;">
<pre class=" language-cpp" style="-webkit-hyphens: none; -webkit-print-color-adjust: exact; background-color: #f5f2f0; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(204, 204, 204); font-family: Consolas, Monaco, 'Andale Mono', 'Ubuntu Mono', monospace; font-size: 13px; line-height: 1.5; margin-bottom: 0.5em; margin-top: 0.5em; overflow: auto; padding: 1em; tab-size: 4; text-shadow: white 0px 1px; word-break: normal; word-spacing: normal; word-wrap: normal;"><code class=" language-cpp" style="-webkit-hyphens: none; -webkit-print-color-adjust: exact; background-image: none; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: none; font-family: Consolas, Monaco, 'Andale Mono', 'Ubuntu Mono', monospace; line-height: 1.5; margin: 0px; padding: 0px; tab-size: 4; text-shadow: white 0px 1px; word-break: normal; word-spacing: normal; word-wrap: normal;"><span class="token macro property" style="-webkit-print-color-adjust: exact; color: #990055;">#<span class="token directive keyword" style="-webkit-print-color-adjust: exact; color: #0077aa;">include</span> <span class="token string" style="-webkit-print-color-adjust: exact; color: #669900;">"CLI11.hpp"</span></span>
<span class="token macro property" style="-webkit-print-color-adjust: exact; color: #990055;">#<span class="token directive keyword" style="-webkit-print-color-adjust: exact; color: #0077aa;">include</span> <span class="token string" style="-webkit-print-color-adjust: exact; color: #669900;"><iostream></span></span>
<span class="token keyword" style="-webkit-print-color-adjust: exact; color: #0077aa;">int</span> <span class="token function" style="-webkit-print-color-adjust: exact; color: #dd4a68;">main</span><span class="token punctuation" style="-webkit-print-color-adjust: exact; color: #999999;">(</span><span class="token keyword" style="-webkit-print-color-adjust: exact; color: #0077aa;">int</span> argc<span class="token punctuation" style="-webkit-print-color-adjust: exact; color: #999999;">,</span> <span class="token keyword" style="-webkit-print-color-adjust: exact; color: #0077aa;">char</span> <span class="token operator" style="-webkit-print-color-adjust: exact; background-color: rgba(255, 255, 255, 0.498039); background-position: initial initial; background-repeat: initial initial; color: #a67f59;">*</span><span class="token operator" style="-webkit-print-color-adjust: exact; background-color: rgba(255, 255, 255, 0.498039); background-position: initial initial; background-repeat: initial initial; color: #a67f59;">*</span>argv<span class="token punctuation" style="-webkit-print-color-adjust: exact; color: #999999;">)</span> <span class="token punctuation" style="-webkit-print-color-adjust: exact; color: #999999;">{</span>
CLI<span class="token operator" style="-webkit-print-color-adjust: exact; background-color: rgba(255, 255, 255, 0.498039); background-position: initial initial; background-repeat: initial initial; color: #a67f59;">::</span>App app<span class="token punctuation" style="-webkit-print-color-adjust: exact; color: #999999;">{</span><span class="token string" style="-webkit-print-color-adjust: exact; color: #669900;">"App description"</span><span class="token punctuation" style="-webkit-print-color-adjust: exact; color: #999999;">}</span><span class="token punctuation" style="-webkit-print-color-adjust: exact; color: #999999;">;</span>
<span class="token comment" spellcheck="true" style="-webkit-print-color-adjust: exact; color: slategrey;">// Define options</span>
<span class="token keyword" style="-webkit-print-color-adjust: exact; color: #0077aa;">int</span> p <span class="token operator" style="-webkit-print-color-adjust: exact; background-color: rgba(255, 255, 255, 0.498039); background-position: initial initial; background-repeat: initial initial; color: #a67f59;">=</span> <span class="token number" style="-webkit-print-color-adjust: exact; color: #990055;">0</span><span class="token punctuation" style="-webkit-print-color-adjust: exact; color: #999999;">;</span>
app<span class="token punctuation" style="-webkit-print-color-adjust: exact; color: #999999;">.</span><span class="token function" style="-webkit-print-color-adjust: exact; color: #dd4a68;">add_option</span><span class="token punctuation" style="-webkit-print-color-adjust: exact; color: #999999;">(</span><span class="token string" style="-webkit-print-color-adjust: exact; color: #669900;">"-p"</span><span class="token punctuation" style="-webkit-print-color-adjust: exact; color: #999999;">,</span> p<span class="token punctuation" style="-webkit-print-color-adjust: exact; color: #999999;">,</span> <span class="token string" style="-webkit-print-color-adjust: exact; color: #669900;">"Parameter"</span><span class="token punctuation" style="-webkit-print-color-adjust: exact; color: #999999;">)</span><span class="token punctuation" style="-webkit-print-color-adjust: exact; color: #999999;">;</span>
<span class="token comment" spellcheck="true" style="-webkit-print-color-adjust: exact; color: slategrey;">// Standard parsing lines (copy and paste in)</span>
<span class="token keyword" style="-webkit-print-color-adjust: exact; color: #0077aa;">try</span> <span class="token punctuation" style="-webkit-print-color-adjust: exact; color: #999999;">{</span>
app<span class="token punctuation" style="-webkit-print-color-adjust: exact; color: #999999;">.</span><span class="token function" style="-webkit-print-color-adjust: exact; color: #dd4a68;">parse</span><span class="token punctuation" style="-webkit-print-color-adjust: exact; color: #999999;">(</span>argc<span class="token punctuation" style="-webkit-print-color-adjust: exact; color: #999999;">,</span> argv<span class="token punctuation" style="-webkit-print-color-adjust: exact; color: #999999;">)</span><span class="token punctuation" style="-webkit-print-color-adjust: exact; color: #999999;">;</span>
<span class="token punctuation" style="-webkit-print-color-adjust: exact; color: #999999;">}</span> <span class="token keyword" style="-webkit-print-color-adjust: exact; color: #0077aa;">catch</span> <span class="token punctuation" style="-webkit-print-color-adjust: exact; color: #999999;">(</span><span class="token keyword" style="-webkit-print-color-adjust: exact; color: #0077aa;">const</span> CLI<span class="token operator" style="-webkit-print-color-adjust: exact; background-color: rgba(255, 255, 255, 0.498039); background-position: initial initial; background-repeat: initial initial; color: #a67f59;">::</span>ParseError <span class="token operator" style="-webkit-print-color-adjust: exact; background-color: rgba(255, 255, 255, 0.498039); background-position: initial initial; background-repeat: initial initial; color: #a67f59;">&</span>e<span class="token punctuation" style="-webkit-print-color-adjust: exact; color: #999999;">)</span> <span class="token punctuation" style="-webkit-print-color-adjust: exact; color: #999999;">{</span>
<span class="token keyword" style="-webkit-print-color-adjust: exact; color: #0077aa;">return</span> app<span class="token punctuation" style="-webkit-print-color-adjust: exact; color: #999999;">.</span><span class="token function" style="-webkit-print-color-adjust: exact; color: #dd4a68;">exit</span><span class="token punctuation" style="-webkit-print-color-adjust: exact; color: #999999;">(</span>e<span class="token punctuation" style="-webkit-print-color-adjust: exact; color: #999999;">)</span><span class="token punctuation" style="-webkit-print-color-adjust: exact; color: #999999;">;</span>
<span class="token punctuation" style="-webkit-print-color-adjust: exact; color: #999999;">}</span>
std<span class="token operator" style="-webkit-print-color-adjust: exact; background-color: rgba(255, 255, 255, 0.498039); background-position: initial initial; background-repeat: initial initial; color: #a67f59;">::</span>cout <span class="token operator" style="-webkit-print-color-adjust: exact; background-color: rgba(255, 255, 255, 0.498039); background-position: initial initial; background-repeat: initial initial; color: #a67f59;"><<</span> <span class="token string" style="-webkit-print-color-adjust: exact; color: #669900;">"Parameter value: "</span> <span class="token operator" style="-webkit-print-color-adjust: exact; background-color: rgba(255, 255, 255, 0.498039); background-position: initial initial; background-repeat: initial initial; color: #a67f59;"><<</span> p <span class="token operator" style="-webkit-print-color-adjust: exact; background-color: rgba(255, 255, 255, 0.498039); background-position: initial initial; background-repeat: initial initial; color: #a67f59;"><<</span> std<span class="token operator" style="-webkit-print-color-adjust: exact; background-color: rgba(255, 255, 255, 0.498039); background-position: initial initial; background-repeat: initial initial; color: #a67f59;">::</span>endl<span class="token punctuation" style="-webkit-print-color-adjust: exact; color: #999999;">;</span>
<span class="token keyword" style="-webkit-print-color-adjust: exact; color: #0077aa;">return</span> <span class="token number" style="-webkit-print-color-adjust: exact; color: #990055;">0</span><span class="token punctuation" style="-webkit-print-color-adjust: exact; color: #999999;">;</span>
<span class="token punctuation" style="-webkit-print-color-adjust: exact; color: #999999;">}</span></code></pre>
</div>
<div style="-webkit-print-color-adjust: exact; background-color: white; font-family: Helvetica, arial, sans-serif; font-size: 14px; margin-bottom: 15px; margin-top: 15px;">
Unlike some other libraries, this is enough to exit correctly and cleanly if help is requested or if incorrect arguments are passed. You can try this example out for yourself. To compile with GCC:</div>
<div style="-webkit-print-color-adjust: exact; background-color: white; font-family: Helvetica, arial, sans-serif; font-size: 14px;">
<pre class=" language-bash" style="-webkit-hyphens: none; -webkit-print-color-adjust: exact; background-color: #f5f2f0; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(204, 204, 204); font-family: Consolas, Monaco, 'Andale Mono', 'Ubuntu Mono', monospace; font-size: 13px; line-height: 1.5; margin-bottom: 0.5em; margin-top: 0.5em; overflow: auto; padding: 1em; tab-size: 4; text-shadow: white 0px 1px; word-break: normal; word-spacing: normal; word-wrap: normal;"><code class=" language-bash" style="-webkit-hyphens: none; -webkit-print-color-adjust: exact; background-image: none; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: none; font-family: Consolas, Monaco, 'Andale Mono', 'Ubuntu Mono', monospace; line-height: 1.5; margin: 0px; padding: 0px; tab-size: 4; text-shadow: white 0px 1px; word-break: normal; word-spacing: normal; word-wrap: normal;">g++ -std<span class="token operator" style="-webkit-print-color-adjust: exact; background-color: rgba(255, 255, 255, 0.498039); background-position: initial initial; background-repeat: initial initial; color: #a67f59;">=</span>c++11 main.cpp </code></pre>
</div>
<div style="-webkit-print-color-adjust: exact; background-color: white; font-family: Helvetica, arial, sans-serif; font-size: 14px; margin-bottom: 15px; margin-top: 15px;">
Much more complicated options are handled elegantly:</div>
<div style="-webkit-print-color-adjust: exact; background-color: white; font-family: Helvetica, arial, sans-serif; font-size: 14px;">
<pre class=" language-cpp" style="-webkit-hyphens: none; -webkit-print-color-adjust: exact; background-color: #f5f2f0; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(204, 204, 204); font-family: Consolas, Monaco, 'Andale Mono', 'Ubuntu Mono', monospace; font-size: 13px; line-height: 1.5; margin-bottom: 0.5em; margin-top: 0.5em; overflow: auto; padding: 1em; tab-size: 4; text-shadow: white 0px 1px; word-break: normal; word-spacing: normal; word-wrap: normal;"><code class=" language-cpp" style="-webkit-hyphens: none; -webkit-print-color-adjust: exact; background-image: none; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: none; font-family: Consolas, Monaco, 'Andale Mono', 'Ubuntu Mono', monospace; line-height: 1.5; margin: 0px; padding: 0px; tab-size: 4; text-shadow: white 0px 1px; word-break: normal; word-spacing: normal; word-wrap: normal;">std<span class="token operator" style="-webkit-print-color-adjust: exact; background-color: rgba(255, 255, 255, 0.498039); background-position: initial initial; background-repeat: initial initial; color: #a67f59;">::</span>string req_real_file<span class="token punctuation" style="-webkit-print-color-adjust: exact; color: #999999;">;</span>
app<span class="token punctuation" style="-webkit-print-color-adjust: exact; color: #999999;">.</span><span class="token function" style="-webkit-print-color-adjust: exact; color: #dd4a68;">add_option</span><span class="token punctuation" style="-webkit-print-color-adjust: exact; color: #999999;">(</span><span class="token string" style="-webkit-print-color-adjust: exact; color: #669900;">"-f,--file"</span><span class="token punctuation" style="-webkit-print-color-adjust: exact; color: #999999;">,</span> file<span class="token punctuation" style="-webkit-print-color-adjust: exact; color: #999999;">,</span> <span class="token string" style="-webkit-print-color-adjust: exact; color: #669900;">"Require an existing file"</span><span class="token punctuation" style="-webkit-print-color-adjust: exact; color: #999999;">)</span>
<span class="token operator" style="-webkit-print-color-adjust: exact; background-color: rgba(255, 255, 255, 0.498039); background-position: initial initial; background-repeat: initial initial; color: #a67f59;">-</span><span class="token operator" style="-webkit-print-color-adjust: exact; background-color: rgba(255, 255, 255, 0.498039); background-position: initial initial; background-repeat: initial initial; color: #a67f59;">></span><span class="token function" style="-webkit-print-color-adjust: exact; color: #dd4a68;">required</span><span class="token punctuation" style="-webkit-print-color-adjust: exact; color: #999999;">(</span><span class="token punctuation" style="-webkit-print-color-adjust: exact; color: #999999;">)</span>
<span class="token operator" style="-webkit-print-color-adjust: exact; background-color: rgba(255, 255, 255, 0.498039); background-position: initial initial; background-repeat: initial initial; color: #a67f59;">-</span><span class="token operator" style="-webkit-print-color-adjust: exact; background-color: rgba(255, 255, 255, 0.498039); background-position: initial initial; background-repeat: initial initial; color: #a67f59;">></span><span class="token function" style="-webkit-print-color-adjust: exact; color: #dd4a68;">check</span><span class="token punctuation" style="-webkit-print-color-adjust: exact; color: #999999;">(</span>CLI<span class="token operator" style="-webkit-print-color-adjust: exact; background-color: rgba(255, 255, 255, 0.498039); background-position: initial initial; background-repeat: initial initial; color: #a67f59;">::</span>ExistingFile<span class="token punctuation" style="-webkit-print-color-adjust: exact; color: #999999;">)</span><span class="token punctuation" style="-webkit-print-color-adjust: exact; color: #999999;">;</span></code></pre>
</div>
<div style="-webkit-print-color-adjust: exact; background-color: white; font-family: Helvetica, arial, sans-serif; font-size: 14px; margin-bottom: 15px; margin-top: 15px;">
You can use any valid type; the above example could have used a <code style="-webkit-print-color-adjust: exact; background-color: #f8f8f8; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(234, 234, 234); margin: 0px 2px; padding: 0px 5px; white-space: nowrap;">boost::file_system</code> file instead of a <code style="-webkit-print-color-adjust: exact; background-color: #f8f8f8; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(234, 234, 234); margin: 0px 2px; padding: 0px 5px; white-space: nowrap;">std::string</code>. The value is a real value and does not require any special lookups to access. You do not have to risk typos by repeating the values after parsing like some libraries require. The library also handles positional arguments, flags, fixed or unlimited repeating options, interdependent options, flags, custom validators, help groups, and more.</div>
<div style="-webkit-print-color-adjust: exact; background-color: white; font-family: Helvetica, arial, sans-serif; font-size: 14px; margin-bottom: 15px; margin-top: 15px;">
You can use subcommands, as well. Subcommands support callback lambda functions when parsed, or they can be checked later. You can infinitely nest subcommands, and each is a full <code style="-webkit-print-color-adjust: exact; background-color: #f8f8f8; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(234, 234, 234); margin: 0px 2px; padding: 0px 5px; white-space: nowrap;">App</code> instance, supporting everything listed above.</div>
<div style="-webkit-print-color-adjust: exact; background-color: white; font-family: Helvetica, arial, sans-serif; font-size: 14px; margin-bottom: 15px; margin-top: 15px;">
Reading/producing <code style="-webkit-print-color-adjust: exact; background-color: #f8f8f8; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(234, 234, 234); margin: 0px 2px; padding: 0px 5px; white-space: nowrap;">.ini</code> files for configuration is also supported, as is using environment variables as input. The base <code style="-webkit-print-color-adjust: exact; background-color: #f8f8f8; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(234, 234, 234); margin: 0px 2px; padding: 0px 5px; white-space: nowrap;">App</code> can be subclassed and customized for use in a toolkit (like <a href="https://github.com/GooFit/GooFit" style="-webkit-print-color-adjust: exact; color: #4183c4;">GooFit</a>). All the standard shell idioms, like <code style="-webkit-print-color-adjust: exact; background-color: #f8f8f8; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(234, 234, 234); margin: 0px 2px; padding: 0px 5px; white-space: nowrap;">--</code>, work as well.</div>
<div style="-webkit-print-color-adjust: exact; background-color: white; font-family: Helvetica, arial, sans-serif; font-size: 14px; margin-bottom: 15px; margin-top: 15px;">
CLI11 was developed at the <a href="http://www.uc.edu/" style="-webkit-print-color-adjust: exact; color: #4183c4;">University of Cincinnati</a> to support of the <a href="https://github.com/GooFit/GooFit" style="-webkit-print-color-adjust: exact; color: #4183c4;">GooFit</a> library under <a href="https://nsf.gov/awardsearch/showAward?AWD_ID=1414736" style="-webkit-print-color-adjust: exact; color: #4183c4;">NSF Award 1414736</a>. It was recently featured in a <a href="http://diana-hep.org/" style="-webkit-print-color-adjust: exact; color: #4183c4;">DIANA/HEP</a> meeting at CERN. Please give it a try! Feedback is always welcome.</div>
Anonymoushttp://www.blogger.com/profile/10752018806658508906noreply@blogger.com0tag:blogger.com,1999:blog-8303893118597899654.post-80773100861937632882017-03-17T10:43:00.000-07:002017-03-17T10:43:01.614-07:00Perfect forwarding for methods<p>I often see perfect forwarding listed for constructor arguments, but not usually for functions with a return or methods. Here is my solution for an method <code>method</code> of class <code>Cls</code></p>
<pre><code class="language-cpp">template<typename ...Args>
static auto method(Cls* cls, Args && ...args)
-> typename std::result_of<decltype(&Cls::method)(Cls, Args...)>::type {
return cls->method(std::forward<Args>(args)...);
}
</code></pre>
<p>This is useful if you want to call protected classes from a “helper” friend class, for example, to expose them to tests without having to require <code>GoogleTest</code>/<code>GoogleMock</code> to be avaible for regular users.</p>
Anonymoushttp://www.blogger.com/profile/10752018806658508906noreply@blogger.com0tag:blogger.com,1999:blog-8303893118597899654.post-17629611323449442742017-01-11T06:08:00.000-08:002017-01-11T08:02:26.972-08:00Lua Environment Modules<div style="box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; font-size: 14px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
This is a guide to setting up Lmod (lua environment modules) on a CentOS system. I’ve used a similar procedure to set them up on a Mac, as well, so this is still a useful guide to the workings of Lmod if you use a different system; mostly paths will change. On a Mac, you’ll want to install <code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">Lmod</code> from the <code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">science</code> tap in <code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">brew</code>.</div>
<div style="box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; font-size: 14px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
There are several good pages covering environment modules (TCL version), but not many that use the newer Lua syntax. This document aims to fill that roll.<br />
<br />
<a name='more'></a></div>
<h1 style="box-sizing: border-box; color: #373d49; font-family: 'Source Sans Pro', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-feature-settings: 'dlig' 1, 'liga' 1, 'lnum' 1, 'kern' 1; font-size: 2.0571429rem; line-height: 3rem; margin: 0px 0px 0.21999rem; padding-top: 0.78001rem;">
<a href="https://www.blogger.com/null" id="Installation_4" style="background-position: 0px 0px; background-repeat: initial initial; box-sizing: border-box; color: #a0aabf; cursor: pointer; text-decoration: underline;"></a>Installation</h1>
<div style="box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; font-size: 14px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
To install Lmod, you should add the epel repository and then do <code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">yum install Lmod</code> to set it up. It will add an init script to <code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">/etc/profile.d</code> to activate the <code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">module</code> command automatically for all users. If you need to customize anything, like the paths included in the <code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">MODULEPATH</code>, the scripts added here are <code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">00-modulepath.*</code> and <code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">z00_lmod.*</code>. You might find that removing the last entry in the modulepath init script (<code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">&& export MODULEPATH=/etc/modulefiles:/usr/share/modulefiles</code>) is useful, as the <code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">MODULEPATH</code> is better set by the profile script already sourced.<br />
<br />
If you follow the instructions <a href="https://lmod.readthedocs.io/en/latest/070_standard_modules.html">here</a>, then you can set up default module lists for all users, as well as assist them in auto-loading their default modules. It basically consists of adding a <code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">StdEnv</code> module file, as well as two init scripts to <code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">profile.d.</code></div>
<h1 style="box-sizing: border-box; color: #373d49; font-family: 'Source Sans Pro', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-feature-settings: 'dlig' 1, 'liga' 1, 'lnum' 1, 'kern' 1; font-size: 2.0571429rem; line-height: 3rem; margin: 0px 0px 0.21999rem; padding-top: 0.78001rem;">
<a href="https://www.blogger.com/null" id="Setting_up_the_directories_Core_8" style="background-position: 0px 0px; background-repeat: initial initial; box-sizing: border-box; color: #a0aabf; cursor: pointer; text-decoration: underline;"></a>Setting up the directories: Core</h1>
<div style="box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; font-size: 14px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
I’ll be showing you how to set up the directories in <code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">/usr/share/modulepath/</code>, since that is already prepared for us. We’ll make two directories, <code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">Core</code> and <code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">Compiler</code>. <code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">/usr/share/modulepath/Core</code> should be the only one in your module path; I perfer not to have <code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">/usr/share/modulepath</code> in the <code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">MODULEPATH</code>, as that will add clutter when looking at the available modules (and it is already available as <code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">MODULEPATH_ROOT</code>).</div>
<div style="box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; font-size: 14px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
Inside <code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">Core</code>, you’ll make directories for your compilers, such as <code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">gcc</code>. Inside <code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">gcc</code>, we’ll set up the default 4.8 compiler by creating a file called <code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">4.8.lua</code>. This is what it should look like:</div>
<pre style="background-image: linear-gradient(rgb(255, 255, 255) 0px, rgb(255, 255, 255) 0.75rem, rgb(245, 247, 250) 0.75rem, rgb(245, 247, 250) 2.75rem, rgb(255, 255, 255) 2.75rem, rgb(255, 255, 255) 4rem); background-size: 100% 4rem; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; border: 1px solid rgb(211, 218, 234); box-sizing: border-box; color: #333333; font-family: monospace, monospace; font-size: 14px; line-height: 2rem; margin-bottom: 1.33999rem; overflow: auto; padding: 0.66001rem 9.5px 9.5px; word-break: break-all; word-wrap: break-word;"><code style="border-bottom-left-radius: 0px; border-bottom-right-radius: 0px; border-top-left-radius: 0px; border-top-right-radius: 0px; box-sizing: border-box; color: inherit; font-family: monospace, monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">help([[
This is the module file for the GCC compiler.
]])
local version = "4.8"
whatis("Name: GCC compiler (system default)")
whatis("Version: " .. version)
whatis("Keywords: System, Compiler")
whatis("URL: http://www.gnu.org/")
whatis("Description: GNU compiler family")
family("compiler")
local prefix = "/usr/bin"
setenv("CC", pathJoin(prefix, "gcc"))
setenv("CXX", pathJoin(prefix, "g++"))
setenv("FC", pathJoin(prefix, "fc"))
setenv("C77", pathJoin(prefix, "fc"))
local mroot = os.getenv("MODULEPATH_ROOT")
local mdir = pathJoin(mroot, "Compiler", "gcc", version)
prepend_path("MODULEPATH", mdir)
</code></pre>
<div style="box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; font-size: 14px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
This has a lot more than is needed, but provides an example of everything you need. This is in Lua syntax, so the items like the multiline string and the local statement for defining local variables might be a little unfamiliar to Python users, but otherwise it’s a lot like Python.</div>
<div style="box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; font-size: 14px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
The important features are:</div>
<ul style="box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-size: 14px; margin-bottom: 0.83999rem; padding-top: 0.16001rem;">
<li style="box-sizing: border-box; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; margin-left: 1rem;">The <code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">family</code> command, which means only one module marked with this string can be loaded at a time.</li>
<li style="box-sizing: border-box; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; margin-left: 1rem;">The <code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">setenv</code> commands, which set environment variables and forget the previous setting<ul style="box-sizing: border-box; margin-bottom: 0px; padding-top: 0.16001rem;">
<li style="box-sizing: border-box; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; margin-left: 1rem;">Another option is <code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">pushenv</code>, which remember the previous setting for unloading</li>
<li style="box-sizing: border-box; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; margin-left: 1rem;">Another similar command is <code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">unsetenv</code>, which clears a variable on loading</li>
</ul>
</li>
<li style="box-sizing: border-box; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; margin-left: 1rem;">The <code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">prepend_path</code> command , which adds a path to the beginning of an environment variable, and removes it on unloading<ul style="box-sizing: border-box; margin-bottom: 0px; padding-top: 0.16001rem;">
<li style="box-sizing: border-box; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; margin-left: 1rem;">There is also a matching <code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">append_path</code> command</li>
</ul>
</li>
<li style="box-sizing: border-box; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; margin-left: 1rem;">The fact that I’m changing <code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">MODULEPATH</code> in a module is special to Lmod; it will also unload all modules in the added path when it is removed!</li>
</ul>
<div style="box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; font-size: 14px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
I’ve also chosen to use a prefix and a version to make this easy to change. The <code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">PathJoin</code> commmand is from one of the optional Lua libraries that Lmod includes automatically.</div>
<div style="box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; font-size: 14px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
Feel free to add more compilers to <code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">Core</code> if needed.</div>
<h1 style="box-sizing: border-box; color: #373d49; font-family: 'Source Sans Pro', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-feature-settings: 'dlig' 1, 'liga' 1, 'lnum' 1, 'kern' 1; font-size: 2.0571429rem; line-height: 3rem; margin: 0px 0px 0.21999rem; padding-top: 0.78001rem;">
<a href="https://www.blogger.com/null" id="Setting_up_the_directories_Compiler_56" style="background-position: 0px 0px; background-repeat: initial initial; box-sizing: border-box; color: #a0aabf; cursor: pointer; text-decoration: underline;"></a>Setting up the directories: Compiler</h1>
<div style="box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; font-size: 14px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
You’ll notice that my compiler loaded it’s matching directory in Compiler; that’s where packages that are compiled with this compiler live. Matching names in multiple compiler directories will correctly swap if you swap compiliers! Inside a <code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">/Compiler/gcc/4.8/</code> directory, add the modules that you want to use.</div>
<div style="box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; font-size: 14px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
If you want the default module to be something other than the latest version number, make a <code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">default.lua</code>symbolic link in the directory pointing to the file you want; if you want to alias a simpler version number, like 6 instead of 6.08.02, you can also make a similar symbolic link.</div>
<h1 style="box-sizing: border-box; color: #373d49; font-family: 'Source Sans Pro', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-feature-settings: 'dlig' 1, 'liga' 1, 'lnum' 1, 'kern' 1; font-size: 2.0571429rem; line-height: 3rem; margin: 0px 0px 0.21999rem; padding-top: 0.78001rem;">
<a href="https://www.blogger.com/null" id="Converting_shell_scripts_62" style="background-position: 0px 0px; background-repeat: initial initial; box-sizing: border-box; color: #a0aabf; cursor: pointer; text-decoration: underline;"></a>Converting shell scripts</h1>
<div style="box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; font-size: 14px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
Many packages use a shell script to set up the environment. Lmod can capture those changes and write (most) of the <code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">.lua</code> file for you. To run it, load the included <code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">lmod</code> package and then run <code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">sh_to_modulefile mysetupscript.sh -o mymodulefile.lua</code> and it will write a modulefile for you. You can use <code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">--help</code> to see the options. If you like the prefix suggestion I used above, I have written my own version of this script that includes this, available in a collection of useful files <a href="https://github.com/henryiii/lmod_useful" style="background-position: 0px 0px; box-sizing: border-box; color: #a0aabf; cursor: pointer;">here</a>. It also can save an environment to a file, then load and compare, creating the <code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">.lua</code> file that describes the changes.</div>
<h1 style="box-sizing: border-box; color: #373d49; font-family: 'Source Sans Pro', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-feature-settings: 'dlig' 1, 'liga' 1, 'lnum' 1, 'kern' 1; font-size: 2.0571429rem; line-height: 3rem; margin: 0px 0px 0.21999rem; padding-top: 0.78001rem;">
<a href="https://www.blogger.com/null" id="Using_modules_66" style="background-position: 0px 0px; background-repeat: initial initial; box-sizing: border-box; color: #a0aabf; cursor: pointer; text-decoration: underline;"></a>Using modules</h1>
<div style="box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; font-size: 14px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
Some common commands you’ll want are:</div>
<ul style="box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-size: 14px; margin-bottom: 0.83999rem; padding-top: 0.16001rem;">
<li style="box-sizing: border-box; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; margin-left: 1rem;"><code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">module avail</code> Shows all currently loadable modules</li>
<li style="box-sizing: border-box; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; margin-left: 1rem;"><code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">module load gcc</code> Prepares a compiler, makes the packages built with that compiler available</li>
<li style="box-sizing: border-box; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; margin-left: 1rem;"><code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">module load package</code> Loads a package from the current compiler</li>
<li style="box-sizing: border-box; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; margin-left: 1rem;"><code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">module list</code> Shows all the currently used modules</li>
<li style="box-sizing: border-box; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; margin-left: 1rem;"><code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">module unload package</code> Unloads a package</li>
<li style="box-sizing: border-box; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; margin-left: 1rem;"><code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">module save</code> Saves the current list of packages (can save to a name, or without a name is your personal default)</li>
<li style="box-sizing: border-box; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; margin-left: 1rem;"><code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">module restore</code> Loads a (named or default) package list - note the system default is <code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">system</code></li>
<li style="box-sizing: border-box; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; margin-left: 1rem;"><code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">module</code> Describes the available subcommands</li>
</ul>
<blockquote style="box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-size: 14px; margin: 0px;">
<div style="border-left-color: rgb(160, 170, 191); border-left-style: solid; border-left-width: 3px; box-sizing: border-box; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; font-size: 1rem; font-style: italic; margin-bottom: 0.33999rem; padding: 0.66001rem 1rem 1rem;">
Note: a timesaver is available as the <code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">ml</code> command; it is short for <code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">module load</code>, but it works with any of the other module subcommands too. Without a package or subcommand it will list the in-use packages. It also supports unloading as <code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">ml -package</code> too!</div>
</blockquote>
<div style="box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; font-size: 14px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
Created with <a href="http://dillinger.io/" style="background-position: 0px 0px; box-sizing: border-box; color: #a0aabf; cursor: pointer;">Dillinger.io</a>.</div>
Anonymoushttp://www.blogger.com/profile/10752018806658508906noreply@blogger.com0tag:blogger.com,1999:blog-8303893118597899654.post-35786734141759023282016-12-15T08:54:00.001-08:002017-01-11T05:21:07.470-08:00Setting up environment modules<div style="box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; font-size: 14px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
Note: please use Lmod instead. It is a newer project that is backward compatible, but also makes some nice improvements. It is used on many supercomputer systems. See my next article.<br />
<br />
The environment modules project is an ideal way to set up (albeit mostly manually) your environment for multiple packages. This is a quick guide on setting it up.</div>
<div style="box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; font-size: 14px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
First, install the environment-modules (CentOS) package with yum. You’ll also need to initialize it in the bashrc file, I chose to add source <code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">/usr/share/Modules/init/bash</code> (and optionally <code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">bash_completion</code>) to <code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">/etc/bashrc</code> instead of running <code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">/usr/share/Modules/bin/add.modules</code>, but if you only want it locally, that’s also an option.</div>
<div style="box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; font-size: 14px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
To set up a module file, you want something like this, for example <code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">/usr/share/Modules/modulefiles/cuda/8.0</code>:</div>
<pre style="background-image: linear-gradient(rgb(255, 255, 255) 0px, rgb(255, 255, 255) 0.75rem, rgb(245, 247, 250) 0.75rem, rgb(245, 247, 250) 2.75rem, rgb(255, 255, 255) 2.75rem, rgb(255, 255, 255) 4rem); background-size: 100% 4rem; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; border: 1px solid rgb(211, 218, 234); box-sizing: border-box; color: #333333; font-family: monospace, monospace; font-size: 14px; line-height: 2rem; margin-bottom: 1.33999rem; overflow: auto; padding: 0.66001rem 9.5px 9.5px; word-break: break-all; word-wrap: break-word;"><code style="border-bottom-left-radius: 0px; border-bottom-right-radius: 0px; border-top-left-radius: 0px; border-top-right-radius: 0px; box-sizing: border-box; color: inherit; font-family: monospace, monospace; font-size: inherit; padding: 0px; white-space: pre-wrap;">#%Module1.0
proc ModulesHelp { } {
global version prefix name
puts stderr "$name/$version - loads the environment for $name, in $prefix"
}
set name cuda
set version 8.0
module-whatis "loads the $name environment"
set prefix /usr/local/cuda-$version
prepend-path LD_LIBRARY_PATH $prefix/lib64
prepend-path PATH $prefix/bin
</code></pre>
<div style="box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; font-size: 14px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
You can generate the meat of this file by doing, for example for ROOT:</div>
<div style="box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; font-size: 14px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
<code style="background-color: #f9f2f4; border-bottom-left-radius: 4px; border-bottom-right-radius: 4px; border-top-left-radius: 4px; border-top-right-radius: 4px; box-sizing: border-box; color: #c7254e; font-family: monospace, monospace; font-size: 1em; padding: 2px 4px;">/usr/share/Modules/bin/createmodule.py /opt/root-6.08.02/bin/thisroot.sh > /usr/share/Modules/modulefiles/root/6.08.02</code></div>
<div style="box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; font-size: 14px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
You can remove most or all the default modules - and yes you’ll need to make modules for each package. The “spider” search does not seem to be in the standard modules package.</div>
<div style="box-sizing: border-box; color: #373d49; font-family: Georgia, Cambria, serif; font-feature-settings: 'kern' 1, 'onum' 1, 'liga' 1; font-size: 14px; margin-bottom: 1.33999rem; padding-top: 0.66001rem;">
Note: I used http://dillinger.io to generate the this page.</div>
Anonymoushttp://www.blogger.com/profile/10752018806658508906noreply@blogger.com0tag:blogger.com,1999:blog-8303893118597899654.post-85646401692300806282016-03-25T16:12:00.000-07:002016-03-27T10:42:25.906-07:00GoogleTest and CMake <div tabindex="-1" id="notebook" class="border-box-sizing">
<div class="container" id="notebook-container">
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>This is a quick recipe for setting up CMake to use googletest in your projects. First, make a <code>tests</code> folder in the root of your project. Then, add <code>add_subdirectory(tests)</code> to your <code>CMakeLists.txt</code>, after you've finished adding the libraries in your project. Note that the way I've written this probably requires CMake 3.4+.</p>
<p>The <code>CMakeLists.txt</code> file in <code>tests</code> should look like this:</p>
<div class="highlight"><pre><span class="nb">set</span><span class="p">(</span><span class="s">THREADS_PREFER_PTHREAD_FLAG</span> <span class="s">ON</span><span class="p">)</span>
<span class="nb">find_package</span><span class="p">(</span><span class="s">Threads</span> <span class="s">REQUIRED</span><span class="p">)</span>
</pre></div>
<p>This adds the <code>Threads::Threads</code> target that we can link to, to enable the threading support that GTest requires. On some systems, it is important to use the <code>-pthread</code> flag, so this does that if necessary.</p>
<div class="highlight"><pre><span class="nb">include</span><span class="p">(</span><span class="s">ExternalProject</span><span class="p">)</span>
<span class="nb">ExternalProject_Add</span><span class="p">(</span>
<span class="s">gtest</span>
<span class="s">URL</span> <span class="s">http://googletest.googlecode.com/files/gtest-1.7.0.zip</span>
<span class="s">PREFIX</span> <span class="o">${</span><span class="nv">CMAKE_CURRENT_BINARY_DIR</span><span class="o">}</span><span class="s">/gtest</span>
<span class="s">URL_MD5</span> <span class="s">2d6ec8ccdf5c46b05ba54a9fd1d130d7</span>
<span class="s">INSTALL_COMMAND</span> <span class="s2">""</span>
<span class="p">)</span>
<span class="nb">ExternalProject_Get_Property</span><span class="p">(</span><span class="s">gtest</span> <span class="s">source_dir</span> <span class="s">binary_dir</span><span class="p">)</span>
</pre></div>
<p>We have to add an external property, to get CMake to download and build GTest for us. We also need to get the source directory and binary directory for use in linking.</p>
<div class="highlight"><pre><span class="nb">add_library</span><span class="p">(</span><span class="s">libgtest</span> <span class="s">INTERFACE</span><span class="p">)</span>
<span class="nb">add_dependencies</span><span class="p">(</span><span class="s">libgtest</span> <span class="s">gtest</span><span class="p">)</span>
<span class="nb">target_link_libraries</span><span class="p">(</span><span class="s">libgtest</span>
<span class="s">INTERFACE</span> <span class="s">Threads::Threads</span>
<span class="s2">"${binary_dir}/libgtest_main.a"</span>
<span class="s2">"${binary_dir}/libgtest.a"</span><span class="p">)</span>
<span class="nb">target_include_directories</span><span class="p">(</span><span class="s">libgtest</span> <span class="s">INTERFACE</span> <span class="s2">"${source_dir}/include"</span><span class="p">)</span>
</pre></div>
<p>Hopefully these lines are familiar to you; they are setting up a special target that we aren't "building", but are using. The target <code>libgtest</code> is simply an interface (no building), and is dependent on <code>gtest</code> (That has to be built first). The link and include commands set up the dependencies so that future target_link_libraries commands only need this target, and will inherit everything else!</p>
<div class="highlight"><pre><span class="nb">enable_testing</span><span class="p">()</span>
</pre></div>
<p>This prepares CTest to handle the tests. You can either run the binaries, or use "make test" to run the tests through CTest's runner program.</p>
<div class="highlight"><pre><span class="nb">file</span><span class="p">(</span><span class="s">GLOB</span> <span class="s">test_cases</span> <span class="s">*.cpp</span><span class="p">)</span>
</pre></div>
<p>Or however you want to collect your test cases.</p>
<div class="highlight"><pre><span class="nb">foreach</span><span class="p">(</span><span class="s">case_file</span> <span class="o">${</span><span class="nv">test_cases</span><span class="o">}</span><span class="p">)</span>
<span class="nb">get_filename_component</span><span class="p">(</span> <span class="s">case_name</span> <span class="o">${</span><span class="nv">case_file</span><span class="o">}</span> <span class="s">NAME_WE</span> <span class="p">)</span>
<span class="nb">set</span> <span class="p">(</span><span class="s">case_name</span> <span class="s">test_</span><span class="o">${</span><span class="nv">case_name</span><span class="o">}</span><span class="p">)</span>
<span class="nb">add_executable</span><span class="p">(</span><span class="o">${</span><span class="nv">case_name</span><span class="o">}</span> <span class="o">${</span><span class="nv">case_file</span><span class="o">}</span><span class="p">)</span>
<span class="nb">target_link_libraries</span><span class="p">(</span><span class="o">${</span><span class="nv">case_name</span><span class="o">}</span> <span class="s">libgtest</span> <span class="s">MyLibrary</span><span class="p">)</span>
<span class="nb">add_test</span><span class="p">(</span><span class="s">NAME</span> <span class="o">${</span><span class="nv">case_name</span><span class="o">}</span>
<span class="s">COMMAND</span> <span class="o">${</span><span class="nv">CMAKE_RUNTIME_OUTPUT_DIRECTORY</span><span class="o">}</span><span class="s">/</span><span class="o">${</span><span class="nv">case_name</span><span class="o">}</span>
<span class="s">WORKING_DIRECTORY</span>
<span class="o">${</span><span class="nv">PROJECT_BINARY_DIR</span><span class="o">}</span>
<span class="p">)</span>
<span class="nb">endforeach</span><span class="p">()</span>
</pre></div>
<p>Here, we make the tests, doing two things. Setting up the CTest integration is the bulk of the commands above; the main command is the target_link_libraries, which should have your library target (<code>MyLibrary</code> in this example) and the <code>libgtest</code> target. That gets all the includes and links (and defs, if you have those) set on the test targets. That's it!</p>
</div>
</div>
</div><a name='more'></a>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="Using-an-IDE">Using an IDE<a class="anchor-link" href="#Using-an-IDE">¶</a></h3><p>I've found this has some issues with IDE's that support multiple build configurations. I'm looking into a better solution, but for now, these lines injected after the <code>Get_Property</code> call will allow you to build directly in the IDE again:</p>
<div class="highlight"><pre><span class="nb">if</span><span class="p">(</span><span class="s">CMAKE_GENERATOR</span> <span class="s">STREQUAL</span> <span class="s">Xcode</span><span class="p">)</span>
<span class="nb">set</span><span class="p">(</span><span class="s">binary_dir</span> <span class="s2">"${binary_dir}/Debug"</span><span class="p">)</span>
<span class="nb">endif</span><span class="p">()</span>
</pre></div>
</div>
</div>
</div>
</div>
</div>Anonymoushttp://www.blogger.com/profile/10752018806658508906noreply@blogger.com0tag:blogger.com,1999:blog-8303893118597899654.post-86677120878373352562015-11-19T09:46:00.001-08:002015-11-19T09:46:18.666-08:00A simple introduction to Asyncio <div tabindex="-1" id="notebook" class="border-box-sizing">
<div class="container" id="notebook-container">
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>This is a simple explanation of the <code>asyncio</code> module and new supporting language features in Python 3.5. Even though the new keywords <code>async</code> and <code>await</code> are new language constructs, they are mostly* useless without an event loop, and that is supplied in the standard library as <code>asyncio</code>. Also, you need awaitable functions, which are only supplied by <code>asyncio</code> (or in the growing set of async libraries, like <code>asyncssh</code>, <code>quamash</code> etc.).</p>
<blockquote><p>Note: My previous post uses these without the asyncio event loop. But you should not normally do that.</p>
</blockquote>
</div>
</div>
</div><a name='more'></a>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [2]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="kn">import</span> <span class="nn">asyncio</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [3]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">async</span> <span class="k">def</span> <span class="nf">awaitable</span><span class="p">(</span><span class="n">x</span><span class="p">):</span>
<span class="nb">print</span><span class="p">(</span><span class="s">'Awaiting for {} seconds...'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">x</span><span class="p">),</span> <span class="n">flush</span><span class="o">=</span><span class="k">True</span><span class="p">)</span>
<span class="n">await</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s">'Done awaiting for {} seconds.'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">x</span><span class="p">),</span> <span class="n">flush</span><span class="o">=</span><span class="k">True</span><span class="p">)</span>
<span class="k">return</span> <span class="s">'Returned from {}'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>This was our first async function. Unlike a normal generator function, you can't just include an await statement to get it to change from a normal function to an async function; you must also use <code>async def</code> instead of <code>def</code>. This was done partially because <code>await</code> is (still) a valid function name, and there is a lot more code out there when <code>yield</code> was changed to be a keyword.</p>
<blockquote><p>Note: this is using the new syntax. Exactly the same behavior can be accieved in Python 3.4 or even 3.5 using <code>@asyncio.coroutine</code> decorators in place of <code>async</code> and <code>yield from</code> in place of <code>await</code>. <code>async for</code> and <code>async with</code> are new constructs, though.</p>
</blockquote>
<p>Under the covers, this is still just a special generator that will behave in a special way when iterated against. Let's see this first in it's natural setting, an event loop:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [4]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">loop</span> <span class="o">=</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">get_event_loop</span><span class="p">()</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [5]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">tasks</span> <span class="o">=</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">gather</span><span class="p">(</span><span class="o">*</span><span class="p">(</span><span class="n">awaitable</span><span class="p">(</span><span class="n">i</span><span class="p">)</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">3</span><span class="p">)))</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [6]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="o">%%</span><span class="k">time</span>
loop.run_until_complete(tasks)
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>Awaiting for 0 seconds...
Awaiting for 1 seconds...
Awaiting for 2 seconds...
Done awaiting for 0 seconds.
Done awaiting for 1 seconds.
Done awaiting for 2 seconds.
CPU times: user 12.9 ms, sys: 2.79 ms, total: 15.6 ms
Wall time: 2.01 s
</pre>
</div>
</div>
<div class="output_area"><div class="prompt output_prompt">Out[6]:</div>
<div class="output_text output_subarea output_execute_result">
<pre>['Returned from 0', 'Returned from 1', 'Returned from 2']</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Closing the loop will close it forever. You cannot restart a closed loop. (see the method later if you need to get the global loop working again)</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [7]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">loop</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Even though there are no new threads, the global interpreter lock is still in place, etc, we see that these are completing in parallel! This is the core of async programming; the code is make of resumable functions (coroutines) that can keep yielding a "not ready" object when they are not complete.</p>
<p>Let's look at making our own loop without touching the main loop. We will have to use the new loop with any awaiting functions, too:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [8]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">async</span> <span class="k">def</span> <span class="nf">awaitable</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="o">*</span><span class="p">,</span> <span class="n">loop</span><span class="o">=</span><span class="k">None</span><span class="p">):</span>
<span class="nb">print</span><span class="p">(</span><span class="s">'Awaiting for {} seconds...'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">x</span><span class="p">),</span> <span class="n">flush</span><span class="o">=</span><span class="k">True</span><span class="p">)</span>
<span class="n">await</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">loop</span><span class="o">=</span><span class="n">loop</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s">'Done awaiting for {} seconds.'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">x</span><span class="p">),</span> <span class="n">flush</span><span class="o">=</span><span class="k">True</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [9]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">newloop</span> <span class="o">=</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">new_event_loop</span><span class="p">()</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [10]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">tasks</span> <span class="o">=</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">gather</span><span class="p">(</span><span class="o">*</span><span class="p">(</span><span class="n">awaitable</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">loop</span><span class="o">=</span><span class="n">newloop</span><span class="p">)</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">3</span><span class="p">)),</span>
<span class="n">loop</span><span class="o">=</span><span class="n">newloop</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [11]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="o">%%</span><span class="k">time</span>
newloop.run_until_complete(tasks)
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>Awaiting for 1 seconds...
Awaiting for 2 seconds...
Awaiting for 0 seconds...
Done awaiting for 0 seconds.
Done awaiting for 1 seconds.
Done awaiting for 2 seconds.
CPU times: user 10.6 ms, sys: 2.37 ms, total: 13 ms
Wall time: 2.01 s
</pre>
</div>
</div>
<div class="output_area"><div class="prompt output_prompt">Out[11]:</div>
<div class="output_text output_subarea output_execute_result">
<pre>[None, None, None]</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Notice how we had to use the new loop everywhere? It had to be specified in the wait function, and in the gather function. Those default to registering into the main loop unless another is specified (by keyword only). The await using the loop to produce the delays. The gather function actually registers the tasks into the loop. Maybe this would be more obvious if we used run_forever instead:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [12]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">async</span> <span class="k">def</span> <span class="nf">stopit</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">loop</span><span class="p">):</span>
<span class="nb">print</span><span class="p">(</span><span class="s">'Stopping in {} seconds...'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">x</span><span class="p">),</span> <span class="n">flush</span><span class="o">=</span><span class="k">True</span><span class="p">)</span>
<span class="n">await</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">loop</span><span class="o">=</span><span class="n">loop</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s">'Stopping loop.'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">x</span><span class="p">),</span> <span class="n">flush</span><span class="o">=</span><span class="k">True</span><span class="p">)</span>
<span class="n">loop</span><span class="o">.</span><span class="n">stop</span><span class="p">()</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [13]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">asyncio</span><span class="o">.</span><span class="n">gather</span><span class="p">(</span><span class="n">awaitable</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="n">loop</span><span class="o">=</span><span class="n">newloop</span><span class="p">),</span> <span class="n">stopit</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="n">newloop</span><span class="p">),</span> <span class="n">loop</span><span class="o">=</span><span class="n">newloop</span><span class="p">);</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [14]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">newloop</span><span class="o">.</span><span class="n">run_forever</span><span class="p">()</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>Awaiting for 2 seconds...
Stopping in 3 seconds...
Done awaiting for 2 seconds.
Stopping loop.
</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>We can see that gather did, in fact, add the tasks to the event loop (other methods can do this too, like <code>asyncio.ensure_future</code> or <code>loop.create_task</code>). The loop will run forever until stop is called, and obviously it must be called from something that is running in the loop (since the run_forever call blocks).</p>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Since all these calls requiring the loop are a little ugly, you can also use <code>asyncio.set_event_loop</code> to set the global event loop. You can use other event loops that follow the same design principles; the QT event loop has been patched to work with asyncio in the <code>quamash</code> library, for example.</p>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>To use global loop code again, we can combine the previous loop creation and set the global event loop with a new loop:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [15]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">asyncio</span><span class="o">.</span><span class="n">set_event_loop</span><span class="p">(</span><span class="n">asyncio</span><span class="o">.</span><span class="n">new_event_loop</span><span class="p">())</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>We can also restart the global event loop simply by resetting the global event loop policy:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [16]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">asyncio</span><span class="o">.</span><span class="n">set_event_loop_policy</span><span class="p">(</span><span class="n">asyncio</span><span class="o">.</span><span class="n">DefaultEventLoopPolicy</span><span class="p">())</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Here's a quick example modified from the <code>asyncssh</code> readme, to show using ssh with the async syntax:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [17]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="kn">import</span> <span class="nn">asyncio</span><span class="o">,</span> <span class="nn">asyncssh</span><span class="o">,</span> <span class="nn">sys</span>
<span class="n">async</span> <span class="k">def</span> <span class="nf">run_client</span><span class="p">():</span>
<span class="k">with</span> <span class="p">(</span><span class="n">await</span> <span class="n">asyncssh</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="s">'localhost'</span><span class="p">))</span> <span class="k">as</span> <span class="n">conn</span><span class="p">:</span>
<span class="n">stdin</span><span class="p">,</span> <span class="n">stdout</span><span class="p">,</span> <span class="n">stderr</span> <span class="o">=</span> <span class="n">await</span> <span class="n">conn</span><span class="o">.</span><span class="n">open_session</span><span class="p">(</span><span class="s">'echo "Hello!"'</span><span class="p">)</span>
<span class="n">output</span> <span class="o">=</span> <span class="n">await</span> <span class="n">stdout</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
<span class="nb">print</span><span class="p">(</span><span class="n">output</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="s">''</span><span class="p">)</span>
<span class="n">status</span> <span class="o">=</span> <span class="n">stdout</span><span class="o">.</span><span class="n">channel</span><span class="o">.</span><span class="n">get_exit_status</span><span class="p">()</span>
<span class="k">if</span> <span class="n">status</span><span class="p">:</span>
<span class="nb">print</span><span class="p">(</span><span class="s">'Program exited with status {}'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">status</span><span class="p">),</span> <span class="n">file</span><span class="o">=</span><span class="n">sys</span><span class="o">.</span><span class="n">stderr</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="nb">print</span><span class="p">(</span><span class="s">'Program exited successfully'</span><span class="p">)</span>
<span class="n">asyncio</span><span class="o">.</span><span class="n">get_event_loop</span><span class="p">()</span><span class="o">.</span><span class="n">run_until_complete</span><span class="p">(</span><span class="n">run_client</span><span class="p">())</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>Hello!
Program exited successfully
</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<blockquote><p>Note that Cryptography 1.1 is required by the latest version of AsyncSSH, and that is not yet supported by Conda, and since Conda uses its own version of openssh, it is hard to install a new cryptography with pip. I would use <code>pip install asyncssh==1.3.0</code> until Conda releases a new cryptography release. Both libraries are under very active development.</p>
</blockquote>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h1 id="Bonus">Bonus<a class="anchor-link" href="#Bonus">¶</a></h1>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>As you just saw, to quickly run an asyncio function as a synchronous function, just do</p>
<div class="highlight"><pre><span class="n">asyncio</span><span class="o">.</span><span class="n">get_event_loop</span><span class="p">()</span><span class="o">.</span><span class="n">run_until_complete</span><span class="p">(</span><span class="n">function</span><span class="p">())</span>
</pre></div>
<p>If you really do not want scheduled global code running, you can always use a new loop (but then make sure the async functions use it) or you can temporarily change the global event loop. For example, first using the global event loop, then using a new temporary global loop:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [18]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="o">%%</span><span class="k">time</span>
asyncio.get_event_loop().run_until_complete(asyncio.sleep(3))
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>CPU times: user 2.65 ms, sys: 1.16 ms, total: 3.81 ms
Wall time: 3.01 s
</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [19]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="kn">from</span> <span class="nn">contextlib</span> <span class="k">import</span> <span class="n">contextmanager</span>
<span class="nd">@contextmanager</span>
<span class="k">def</span> <span class="nf">temp_eventloop</span><span class="p">():</span>
<span class="n">oldeventloop</span> <span class="o">=</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">get_event_loop</span><span class="p">()</span>
<span class="n">asyncio</span><span class="o">.</span><span class="n">set_event_loop</span><span class="p">(</span><span class="n">asyncio</span><span class="o">.</span><span class="n">new_event_loop</span><span class="p">())</span>
<span class="k">try</span><span class="p">:</span>
<span class="k">yield</span>
<span class="n">asyncio</span><span class="o">.</span><span class="n">get_event_loop</span><span class="p">()</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
<span class="k">finally</span><span class="p">:</span>
<span class="n">asyncio</span><span class="o">.</span><span class="n">set_event_loop</span><span class="p">(</span><span class="n">oldeventloop</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [20]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">main_loop</span> <span class="o">=</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">get_event_loop</span><span class="p">()</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [21]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="o">%%</span><span class="k">time</span>
with temp_eventloop() as loop:
asyncio.get_event_loop().run_until_complete(asyncio.sleep(3))
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>CPU times: user 3.55 ms, sys: 1.43 ms, total: 4.98 ms
Wall time: 3.01 s
</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [22]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">main_loop</span> <span class="ow">is</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">get_event_loop</span><span class="p">()</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt output_prompt">Out[22]:</div>
<div class="output_text output_subarea output_execute_result">
<pre>True</pre>
</div>
</div>
</div>
</div>
</div>
</div>
</div>Anonymoushttp://www.blogger.com/profile/10752018806658508906noreply@blogger.com0tag:blogger.com,1999:blog-8303893118597899654.post-71797954852421536352015-11-19T09:43:00.002-08:002015-11-19T09:43:57.973-08:00A little example of how Asyncio works <div tabindex="-1" id="notebook" class="border-box-sizing">
<div class="container" id="notebook-container">
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>This is a simple example to show how Asyncio works without using Asyncio itself, instead using a basic and poorly written event loop. This is only meant to give a flavor of what Asyncio does behind the curtains. I'm avoiding most details of the library design, like callbacks, just to keep this simple. Since this is written as an illustration, rather than real code, I'm going to dispense with trying to keep it 2.7 compatible.</p>
</div>
</div>
</div><a name='more'></a>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [23]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="kn">import</span> <span class="nn">time</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>This loop will be based on async functions that 'guess' how long they need before being able to complete. It would be an inficient way to wait for unknown items, like input, but works great for timers (which I'll be using). In Asyncio, I would return a Future, and this would wait until <code>.done()</code> was true. And the loop would use callbacks and select to wait on things like file discriptors to be ready. That is only needed to make the wait time variable, ending when work needs to be done, to maximize speed and effeciency, which is not needed for this example.</p>
<p>I'm going to make new version of a float that I can specifically test for using <code>isinstance</code>, but otherwise is a float. The size of the float is a rough guess of how long to sleep.</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [24]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="k">class</span> <span class="nc">NotReady</span><span class="p">(</span><span class="nb">float</span><span class="p">):</span>
<span class="k">pass</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Now, here's an event loop that runs through a list of tasks (created by create_task) and then sleeps if no thread is ready (based on the minimum recommended time by each incomplete task):</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [25]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="k">class</span> <span class="nc">EventLoop</span><span class="p">:</span>
<span class="k">def</span> <span class="nf">run_until_complete</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">tasks</span><span class="p">):</span>
<span class="n">results</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">while</span> <span class="n">tasks</span><span class="p">:</span> <span class="c"># Stops when all tasks are done, unlike asyncio run_forever</span>
<span class="n">waits</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">task</span> <span class="ow">in</span> <span class="n">tasks</span><span class="p">:</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">res</span> <span class="o">=</span> <span class="n">task</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="k">None</span><span class="p">)</span> <span class="c"># async function runs here</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">res</span><span class="p">,</span> <span class="n">NotReady</span><span class="p">):</span>
<span class="n">waits</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">res</span><span class="p">)</span> <span class="c"># Build up all the requested waits</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">waits</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="c"># Don't wait if a task completed</span>
<span class="n">results</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">res</span><span class="p">)</span> <span class="c"># Gather results</span>
<span class="k">except</span> <span class="ne">StopIteration</span><span class="p">:</span>
<span class="n">tasks</span><span class="o">.</span><span class="n">remove</span><span class="p">(</span><span class="n">task</span><span class="p">)</span> <span class="c"># Task done, remove from tasks</span>
<span class="k">if</span> <span class="n">waits</span><span class="p">:</span>
<span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="nb">min</span><span class="p">(</span><span class="n">waits</span><span class="p">))</span> <span class="c"># Wait for the shortest requested wait</span>
<span class="k">return</span> <span class="n">results</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>It is easy to then write a sleep function for this eventloop:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [26]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="k">def</span> <span class="nf">sleep</span><span class="p">(</span><span class="n">t</span><span class="p">):</span>
<span class="n">endtime</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">()</span> <span class="o">+</span> <span class="n">t</span>
<span class="k">while</span> <span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">()</span> <span class="o"><</span> <span class="n">endtime</span><span class="p">:</span>
<span class="k">yield</span> <span class="n">NotReady</span><span class="p">(</span><span class="n">endtime</span> <span class="o">-</span> <span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">())</span>
<span class="k">yield</span> <span class="s">'Sleep {} over'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">t</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Now we can print the results, making sure it only takes 4 seconds and the tasks completed in the correct order:</p>
<blockquote><p>Note: For this purpose, I order the output by completion order, rather than input order, unlike <code>asyncio</code>. Later you will also notice that I gather all outputs, rather than just the last one.</p>
</blockquote>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [28]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="o">%%</span><span class="k">time</span>
myloop = EventLoop()
print(myloop.run_until_complete([sleep(3),
sleep(2),
sleep(1),
sleep(4)]))
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>['Sleep 1 over', 'Sleep 2 over', 'Sleep 3 over', 'Sleep 4 over']
CPU times: user 567 µs, sys: 523 µs, total: 1.09 ms
Wall time: 4 s
</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Now the pieces are in place to see why <code>yield from</code> is needed:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [34]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="k">def</span> <span class="nf">print_every_period</span><span class="p">(</span><span class="n">msg</span><span class="p">,</span> <span class="n">period</span><span class="p">):</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="nb">int</span><span class="p">(</span><span class="mi">3</span><span class="o">/</span><span class="n">period</span><span class="p">)):</span>
<span class="k">yield from</span> <span class="n">sleep</span><span class="p">(</span><span class="n">period</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">msg</span><span class="p">,</span> <span class="n">flush</span><span class="o">=</span><span class="k">True</span><span class="p">)</span>
<span class="k">return</span> <span class="s">'Print {} over'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">period</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [35]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">myloop</span> <span class="o">=</span> <span class="n">EventLoop</span><span class="p">()</span>
<span class="nb">print</span><span class="p">(</span><span class="n">myloop</span><span class="o">.</span><span class="n">run_until_complete</span><span class="p">([</span><span class="n">print_every_period</span><span class="p">(</span><span class="s">"First message!"</span><span class="p">,</span><span class="mi">1</span><span class="p">),</span>
<span class="n">print_every_period</span><span class="p">(</span><span class="s">"Second message!"</span><span class="p">,</span><span class="mi">1</span><span class="p">),</span>
<span class="n">print_every_period</span><span class="p">(</span><span class="s">"..."</span><span class="p">,</span><span class="o">.</span><span class="mi">25</span><span class="p">)]))</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>...
...
...
First message!
Second message!
...
...
...
...
First message!
Second message!
...
...
...
...
First message!
Second message!
...
['Sleep 0.25 over', 'Sleep 0.25 over', 'Sleep 0.25 over', 'Sleep 1 over', 'Sleep 1 over', 'Sleep 0.25 over', 'Sleep 0.25 over', 'Sleep 0.25 over', 'Sleep 0.25 over', 'Sleep 1 over', 'Sleep 1 over', 'Sleep 0.25 over', 'Sleep 0.25 over', 'Sleep 0.25 over', 'Sleep 0.25 over', 'Sleep 1 over', 'Sleep 1 over', 'Sleep 0.25 over']
</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>We have created a working async system, with event loop and a real async sleep function, using only generators and a normal sleep function. This is incredibly simplified, but should have given a hint as to the real system Python uses to implement generator based Asyncio. I've avoiding using an OS level select loop, callbacks, etc.</p>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Just to show this is similar to asyncio:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [1]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="kn">import</span> <span class="nn">asyncio</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [36]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="nd">@asyncio</span><span class="o">.</span><span class="n">coroutine</span>
<span class="k">def</span> <span class="nf">as_print_every_period</span><span class="p">(</span><span class="n">msg</span><span class="p">,</span> <span class="n">period</span><span class="p">):</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="nb">int</span><span class="p">(</span><span class="mi">3</span><span class="o">/</span><span class="n">period</span><span class="p">)):</span>
<span class="k">yield from</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="n">period</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">msg</span><span class="p">,</span> <span class="n">flush</span><span class="o">=</span><span class="k">True</span><span class="p">)</span>
<span class="k">return</span> <span class="s">'Print {} over'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">period</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [37]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">loop</span> <span class="o">=</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">get_event_loop</span><span class="p">()</span>
<span class="n">loop</span><span class="o">.</span><span class="n">run_until_complete</span><span class="p">(</span><span class="n">asyncio</span><span class="o">.</span><span class="n">gather</span><span class="p">(</span>
<span class="n">as_print_every_period</span><span class="p">(</span><span class="s">"First message!"</span><span class="p">,</span><span class="mi">1</span><span class="p">),</span>
<span class="n">as_print_every_period</span><span class="p">(</span><span class="s">"Second message!"</span><span class="p">,</span><span class="mi">1</span><span class="p">),</span>
<span class="n">as_print_every_period</span><span class="p">(</span><span class="s">"..."</span><span class="p">,</span><span class="o">.</span><span class="mi">25</span><span class="p">)))</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>...
...
...
Second message!
First message!
...
...
...
...
Second message!
First message!
...
...
...
...
Second message!
First message!
...
</pre>
</div>
</div>
<div class="output_area"><div class="prompt output_prompt">Out[37]:</div>
<div class="output_text output_subarea output_execute_result">
<pre>['Print 1 over', 'Print 1 over', 'Print 0.25 over']</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>The only change was the addition of <code>gather</code>, since a coroutine must be turned into a <code>Task</code>, which is a feature my system did not need.</p>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h1 id="Bonus:-awaitable">Bonus: awaitable<a class="anchor-link" href="#Bonus:-awaitable">¶</a></h1><p>Let's modify the previous example to do the unthinkable: use the await/async syntax without using asyncio. First, this is a simple wrapper to change our current normal generator into an awaitable:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [99]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="k">class</span> <span class="nc">asleep</span><span class="p">:</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">t</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">t</span> <span class="o">=</span> <span class="n">t</span>
<span class="k">def</span> <span class="nf">__await__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="n">sleep</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">t</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>We can then use the new syntax with <code>asleep</code>:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [95]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">async</span> <span class="k">def</span> <span class="nf">print_every_period</span><span class="p">(</span><span class="n">msg</span><span class="p">,</span> <span class="n">period</span><span class="p">):</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="nb">int</span><span class="p">(</span><span class="mi">3</span><span class="o">/</span><span class="n">period</span><span class="p">)):</span>
<span class="n">await</span> <span class="n">asleep</span><span class="p">(</span><span class="n">period</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">msg</span><span class="p">,</span> <span class="n">flush</span><span class="o">=</span><span class="k">True</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>The event loop needs only tiny changes; there is no need to call iter, and we will need to explicitly close the awaitable before letting it be garbage collected:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [129]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="k">class</span> <span class="nc">EventLoop</span><span class="p">:</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">tasks</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">def</span> <span class="nf">create_task</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">task</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">tasks</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">task</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">run_forever</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">results</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">while</span> <span class="bp">self</span><span class="o">.</span><span class="n">tasks</span><span class="p">:</span>
<span class="n">waits</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">task</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">tasks</span><span class="p">:</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">res</span> <span class="o">=</span> <span class="n">task</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="k">None</span><span class="p">)</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">res</span><span class="p">,</span> <span class="n">NotReady</span><span class="p">):</span>
<span class="n">waits</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">res</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">waits</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
<span class="n">results</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">res</span><span class="p">)</span>
<span class="k">except</span> <span class="ne">StopIteration</span><span class="p">:</span>
<span class="n">task</span><span class="o">.</span><span class="n">close</span><span class="p">()</span> <span class="c"># Needed to avoid "never awaited"</span>
<span class="bp">self</span><span class="o">.</span><span class="n">tasks</span><span class="o">.</span><span class="n">remove</span><span class="p">(</span><span class="n">task</span><span class="p">)</span>
<span class="k">if</span> <span class="n">waits</span><span class="p">:</span>
<span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="nb">min</span><span class="p">(</span><span class="n">waits</span><span class="p">))</span>
<span class="k">return</span> <span class="n">results</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [130]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">myloop</span> <span class="o">=</span> <span class="n">EventLoop</span><span class="p">()</span>
<span class="n">myloop</span><span class="o">.</span><span class="n">create_task</span><span class="p">(</span><span class="n">print_every_period</span><span class="p">(</span><span class="s">"First message!"</span><span class="p">,</span><span class="mi">1</span><span class="p">))</span>
<span class="n">myloop</span><span class="o">.</span><span class="n">create_task</span><span class="p">(</span><span class="n">print_every_period</span><span class="p">(</span><span class="s">"Second message!"</span><span class="p">,</span><span class="mi">1</span><span class="p">))</span>
<span class="n">myloop</span><span class="o">.</span><span class="n">create_task</span><span class="p">(</span><span class="n">print_every_period</span><span class="p">(</span><span class="s">"..."</span><span class="p">,</span><span class="o">.</span><span class="mi">25</span><span class="p">))</span>
<span class="nb">print</span><span class="p">(</span><span class="n">myloop</span><span class="o">.</span><span class="n">run_forever</span><span class="p">())</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>...
...
...
First message!
Second message!
...
...
...
...
First message!
Second message!
...
...
...
...
First message!
Second message!
...
['Sleep 0.25 over', 'Sleep 0.25 over', 'Sleep 0.25 over', 'Sleep 1 over', 'Sleep 1 over', 'Sleep 0.25 over', 'Sleep 0.25 over', 'Sleep 0.25 over', 'Sleep 0.25 over', 'Sleep 1 over', 'Sleep 1 over', 'Sleep 0.25 over', 'Sleep 0.25 over', 'Sleep 0.25 over', 'Sleep 0.25 over', 'Sleep 1 over', 'Sleep 1 over', 'Sleep 0.25 over']
</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>This is a working usage of the new syntax using only standard library tools. Note that I had to call <code>.close</code>, so that the task would not display a warning that it was never awaited on when it was garbage collected.</p>
</div>
</div>
</div>
</div>
</div>Anonymoushttp://www.blogger.com/profile/10752018806658508906noreply@blogger.com0tag:blogger.com,1999:blog-8303893118597899654.post-25314489659103903602015-10-30T07:02:00.002-07:002015-10-30T07:02:57.709-07:00Feynman Diagrams in Tikz <div tabindex="-1" id="notebook" class="border-box-sizing">
<div class="container" id="notebook-container">
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>There is a package for making Feynman diagrams in LaTeX. Unfortunately, it is old and <code>dvi</code> latex only. If you are using pdflatex or lualatex, as you should be, it does not work. Even in regular LaTeX, it's a bit of a pain. Why is there not a new package for pdflatex? Turns out, you don't need one. Due to the powerful drawing library Tikz, you can create any diagram easily, and can customize it completely. For example:</p>
</div>
</div>
</div><div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgUYE__QP4Yh9sZEhr6lCMTrR6Qn71aDDcajP0bW6xh3EMIcehIghWIPJn__M8-YBIT8MCPndU6ph0MzSRu8rnG3dFPpIgXUucDgqsXGWH-zUl4OeGy26BWhaTta-C_GxfbMAZPOXD0xPk/s1600/fd_color.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="305" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgUYE__QP4Yh9sZEhr6lCMTrR6Qn71aDDcajP0bW6xh3EMIcehIghWIPJn__M8-YBIT8MCPndU6ph0MzSRu8rnG3dFPpIgXUucDgqsXGWH-zUl4OeGy26BWhaTta-C_GxfbMAZPOXD0xPk/s400/fd_color.png" width="400" /></a></div><a name='more'></a>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>For the following, I highly recommend using LyX, even if you plan to use the finished diagram in LaTeX. As you'll see, you can get near real time previews of the diagrams.</p>
<p>First, you'll want to have something like this in your preamble (or in Preferences->Document Settings -> Latex Preamble in LyX).</p>
<div class="highlight"><pre><span class="k">\usepackage</span><span class="nb">{</span>tikz<span class="nb">}</span>
<span class="k">\usetikzlibrary</span><span class="nb">{</span>decorations.pathmorphing<span class="nb">}</span>
<span class="k">\usetikzlibrary</span><span class="nb">{</span>decorations.markings<span class="nb">}</span>
<span class="k">\usetikzlibrary</span><span class="nb">{</span>positioning, shapes, snakes, arrows<span class="nb">}</span>
<span class="k">\tikzset</span><span class="nb">{</span>
quark/.style=<span class="nb">{</span>postaction=<span class="nb">{</span>decorate<span class="nb">}</span>,
decoration=<span class="nb">{</span>markings,mark=at position .5 with <span class="nb">{</span><span class="k">\arrow</span><span class="na">[#1]</span><span class="nb">{</span>latex<span class="nb">}}}}</span>,
scalar/.style=<span class="nb">{</span>dashed,postaction=<span class="nb">{</span>decorate<span class="nb">}</span>,
decoration=<span class="nb">{</span>markings,mark=at position .5 with <span class="nb">{</span><span class="k">\arrow</span><span class="na">[#1]</span><span class="nb">{</span>latex<span class="nb">}}}}</span>,
gluon/.style=<span class="nb">{</span>decorate,
decoration=<span class="nb">{</span>coil,amplitude=2pt, segment length=2pt, pre length=.1cm, post length=.1cm<span class="nb">}}</span>,
boson/.style=<span class="nb">{</span>-latex,decorate, decoration=<span class="nb">{</span>snake, segment length=4pt, amplitude=1.8pt, pre length=.1cm, post length=.25cm<span class="nb">}}</span>,
photon/.style=<span class="nb">{</span>decorate, decoration=<span class="nb">{</span>snake, segment length=4pt, amplitude=1.8pt, pre length=.1cm, post length=.1cm<span class="nb">}}</span>,
dphoton/.style=<span class="nb">{</span>decorate, decoration=<span class="nb">{</span>snake, segment length=4pt, amplitude=1.8pt, pre length=.1cm, post length=.25cm<span class="nb">}</span>,-latex<span class="nb">}</span>
<span class="nb">}</span>
</pre></div>
<blockquote><p>Aside:
Although not necessary, you'll probably want the slashed package also if you are working with QFT and are writing equations. You can add it using a LyX macro and a replacement, such as <code>\s{#1}:=\slashed#1 \acute{#1}</code> or similar), allowing it to show up in LyX Math Mode.</p>
</blockquote>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="T-channel-diagram">T channel diagram<a class="anchor-link" href="#T-channel-diagram">¶</a></h2>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<div class="highlight"><pre><span class="k">\begin</span><span class="nb">{</span>tikzpicture<span class="nb">}</span>[scale=1.5]
</pre></div>
<p>To draw a figure, you'll need to define the verticies first. Tikz uses a standard coordinate system, starting from (0,0) and automatically sets the bounding box to fit the contents (though you can set one manually). Don't worry too much about the units, you can always scale the picture after you created it with the scale option.</p>
<div class="highlight"><pre><span class="k">\def\wid</span><span class="nb">{</span>.7<span class="nb">}</span>;
<span class="k">\def\iwid</span><span class="nb">{</span>.5<span class="nb">}</span>;
</pre></div>
<p>Tikz allows you to define constants (it is still LaTeX, after all), so I'm going to define a couple of values here (these are actually going to be used as heights, despite my name choice). I can easily move the coordinates symmetrically with these. I don't have to end these lines with a simicolon, but Tikz lines do end in semicolons, so I'm being consistant.</p>
<div class="highlight"><pre><span class="k">\node</span> (mu) at (-1,<span class="k">\wid</span>) <span class="nb">{</span><span class="s">$</span><span class="nv">\mu</span><span class="nb">^{</span><span class="o">+</span><span class="nb">} </span><span class="o">(</span><span class="nb">p</span><span class="o">)</span><span class="s">$</span><span class="nb">}</span>;
<span class="k">\node</span> (e) at (-1,-<span class="k">\wid</span>) <span class="nb">{</span><span class="s">$</span><span class="nb">e^{</span><span class="o">-</span><span class="nb">} </span><span class="o">(</span><span class="nb">q</span><span class="o">)</span><span class="s">$</span><span class="nb">}</span>;
<span class="k">\coordinate</span> (Wu) at (0,<span class="k">\iwid</span>);
<span class="k">\coordinate</span> (Wd) at (0,-<span class="k">\iwid</span>);
<span class="k">\node</span> (numu) at (1,<span class="k">\wid</span>) <span class="nb">{</span><span class="s">$</span><span class="nv">\bar\nu</span><span class="nb">_</span><span class="nv">\mu</span><span class="nb"> </span><span class="o">(</span><span class="nb">p'</span><span class="o">)</span><span class="s">$</span><span class="nb">}</span>;
<span class="k">\node</span> (nue) at (1,-<span class="k">\wid</span>) <span class="nb">{</span><span class="s">$</span><span class="nv">\nu</span><span class="nb">_e </span><span class="o">(</span><span class="nb">q'</span><span class="o">)</span><span class="s">$</span><span class="nb">}</span>;
</pre></div>
<p>Now, I'm setting up the vertices. There are two kinds; <code>\coordinate</code> is simply a named coordinate, with a name given in parentheses and at a location. Nothing visually is added here at this point.</p>
<p><code>\node</code> is the same, but it also has a label given in curly brackets. If you wanted to give options, such as a direction to offset the point, those would come in square brackets. You usually do not need separate nodes for labels, the ones you use for lines should work too.</p>
<p>The coordinates are specified as <code>(x,y)</code>, though <code>(radius:angle)</code> is also possible.</p>
<p>Notice that a line of Tikz starts with a backslash, and ends with a semicolon.</p>
<div class="highlight"><pre><span class="k">\draw</span> [fill=black] (Wu) circle (.04) node [above] <span class="nb">{</span><span class="s">$</span><span class="nv">\alpha</span><span class="s">$</span><span class="nb">}</span>;
<span class="k">\draw</span> [fill=black] (Wd) circle (.04) node [below] <span class="nb">{</span><span class="s">$</span><span class="nv">\beta</span><span class="s">$</span><span class="nb">}</span>;
</pre></div>
<p>This creates the "dot" at the vertices for the interaction. I'm chaining commands; note that chained commands don't start with a backslash, and they use the location of the previous command. The proper way to read the first line would be: draw, with a black fill, at the point Wu, a circle with radius 4, and put a node label of $\alpha$ above.</p>
<pre><code>\draw [boson] (Wu) -- (Wd) node [midway, right] {$W^{+}$};
\draw [quark] (mu) -- (Wu);
\draw [quark] (Wu) -- (numu);
\draw [quark] (e) -- (Wd);
\draw [quark] (Wd) -- (nue);</code></pre>
<p>Here we are actually drawing the diagram. We are using the previous style definitions to decorate our paths, which makes them look like the correct particle. The <code>--</code> command draws a straight line from one point to another. The <code>[midway]</code> option for nodes puts the node in the middle of the previous two points. You can still use offsets, too. We have named all of our coordinates, though we could directly enter points if needed.</p>
<p>We can chain coordinates together as we draw lines (use <code>\path</code> to avoid drawing the line). Prepending a <code>+</code> will make a coordinate relative, and <code>++</code> will make it relative and not move the cursor to the new position. Combined with polar coordinates, this makes even complex diagrams easy. You can also do math with nodes, though you'll need <code>\usetikzlibrary{calc}</code> to do that.</p>
<div class="highlight"><pre><span class="k">\end</span><span class="nb">{</span>tikzpicture<span class="nb">}</span>
</pre></div>
<p>And that's it! The result:</p>
</div>
</div>
</div><div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjXfghjNSxkLeERniTCNi9xuPfCjP6RSVMNlRAAOvueALfUa-LXQjOM61Z_d7JZBXiPftkgxaTuI2l5ToA-WApp2uWqqnRKV-qM7t0SmEsEvq7pabRh7c1z_2O-gxCd26HDRKq7naYjWgg/s1600/td_example.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjXfghjNSxkLeERniTCNi9xuPfCjP6RSVMNlRAAOvueALfUa-LXQjOM61Z_d7JZBXiPftkgxaTuI2l5ToA-WApp2uWqqnRKV-qM7t0SmEsEvq7pabRh7c1z_2O-gxCd26HDRKq7naYjWgg/s320/td_example.png" /></a></div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>For futher information, see</p>
<ul>
<li><a href="http://mirrors.ctan.org/graphics/pgf/base/doc/pgfmanual.pdf">The Tikz Manual</a></li>
<li><a href="http://www.texample.net/tikz/examples/">The Tikz Examples Page</a></li>
</ul>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="QCD-Process">QCD Process<a class="anchor-link" href="#QCD-Process">¶</a></h2><p>This is the cover image of the article; one of the diagrams for a QCD process of the form
$$
q\left(p_{1}\right)+g\left(k\right)\rightarrow q\left(p_{2}\right)+Q\left(q_{1}\right)+\bar{Q}\left(q_{2}\right).
$$</p>
<p>The code to create it is as follows:</p>
<div class="highlight"><pre><span class="k">\begin</span><span class="nb">{</span>tikzpicture<span class="nb">}</span>[scale=1.5]
<span class="k">\node</span> (g) at (-1,1) <span class="nb">{</span><span class="s">$</span><span class="nb">g^j</span><span class="nv">\left</span><span class="o">(</span><span class="nb"> k </span><span class="nv">\right</span><span class="o">)</span><span class="nb"> </span><span class="nv">\epsilon</span><span class="nb">^</span><span class="nv">\mu</span><span class="s">$</span><span class="nb">}</span>;
<span class="k">\node</span> (qi) at (-1,-1) <span class="nb">{</span><span class="s">$</span><span class="nb">q^i</span><span class="nv">\left</span><span class="o">(</span><span class="nb"> p_</span><span class="m">1</span><span class="nb"> </span><span class="nv">\right</span><span class="o">)</span><span class="s">$</span><span class="nb">}</span>;
<span class="k">\coordinate</span> (mu) at (0,0);
<span class="k">\coordinate</span> (b) at (1.5,0);
<span class="k">\coordinate</span> (a) at (1.5,1);
<span class="k">\node</span> (qf) at (2.5,-1) <span class="nb">{</span><span class="s">$</span><span class="nb">q^k</span><span class="nv">\left</span><span class="o">(</span><span class="nb"> p_</span><span class="m">2</span><span class="nb"> </span><span class="nv">\right</span><span class="o">)</span><span class="s">$</span><span class="nb">}</span>;
<span class="k">\node</span> (Q1) at (2.5,0) <span class="nb">{</span><span class="s">$</span><span class="nb">Q^l </span><span class="nv">\left</span><span class="o">(</span><span class="nb"> q_</span><span class="m">1</span><span class="nb"> </span><span class="nv">\right</span><span class="o">)</span><span class="s">$</span><span class="nb">}</span>;
<span class="k">\node</span> (Q2) at (2.5,2) <span class="nb">{</span><span class="s">$</span><span class="nv">\bar</span><span class="nb"> Q^m </span><span class="nv">\left</span><span class="o">(</span><span class="nb"> q_</span><span class="m">2</span><span class="nb"> </span><span class="nv">\right</span><span class="o">)</span><span class="s">$</span><span class="nb">}</span>;
<span class="k">\draw</span> [gluon] (g) -- (mu);
<span class="k">\draw</span> [quark] (qi) -- (mu);
<span class="k">\draw</span> [quark, magenta] (mu) -- (b) node [midway, below] <span class="nb">{</span><span class="k">\tiny</span> <span class="s">$</span><span class="nb">r</span><span class="s">$</span><span class="nb">}</span>
node [midway, above] <span class="nb">{</span><span class="k">\tiny</span> <span class="s">$</span><span class="nv">\left</span><span class="o">(</span><span class="nb"> p_</span><span class="m">1</span><span class="o">+</span><span class="nb">k </span><span class="nv">\right</span><span class="o">)</span><span class="nb"> </span><span class="s">$</span><span class="nb">}</span>;
<span class="k">\draw</span> [quark] (b) -- (qf);
<span class="k">\draw</span> [gluon, cyan] (b) -- (a) node [pos=.5, right] <span class="nb">{</span><span class="k">\tiny</span> <span class="s">$</span><span class="nb">b</span><span class="s">$</span><span class="nb">}</span>
node [pos=.5,left] <span class="nb">{</span><span class="k">\tiny</span> <span class="s">$</span><span class="nv">\left</span><span class="o">(</span><span class="nb"> q_</span><span class="m">1</span><span class="o">+</span><span class="nb">q_</span><span class="m">2</span><span class="nb"> </span><span class="nv">\right</span><span class="o">)</span><span class="nb"> </span><span class="s">$</span><span class="nb">}</span>;
<span class="k">\draw</span> [quark] (Q2) -- (a);
<span class="k">\draw</span> [quark] (a) -- (Q1);
<span class="k">\draw</span> [fill,red] (mu) circle (.04) node [below right] <span class="nb">{</span><span class="s">$</span><span class="nv">\mu</span><span class="s">$</span><span class="nb">}</span>;
<span class="k">\draw</span> [fill,green] (b) circle (.04) node [below left] <span class="nb">{</span><span class="s">$</span><span class="nv">\beta</span><span class="s">$</span><span class="nb">}</span>;
<span class="k">\draw</span> [fill,blue] (a) circle (.04) node [above left] <span class="nb">{</span><span class="s">$</span><span class="nv">\alpha</span><span class="s">$</span><span class="nb">}</span>;
<span class="k">\end</span><span class="nb">{</span>tikzpicture<span class="nb">}</span>
</pre></div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h1 id="Bonus:">Bonus:<a class="anchor-link" href="#Bonus:">¶</a></h1><p>To easily make standalone figures, use the standalone class. You can even use the convert option and <code>--shell-escape</code> to have it create other formats, like <code>.png</code> for web embedding.</p>
</div>
</div>
</div>
</div>
</div>Anonymoushttp://www.blogger.com/profile/10752018806658508906noreply@blogger.com0tag:blogger.com,1999:blog-8303893118597899654.post-24494160090529136092015-10-19T09:07:00.002-07:002015-11-19T09:47:52.181-08:00Including CRY cosmic ray generator in CMakeI realized that CRY did not have a CMake based install option, so including it in a GEANT4 cmake project might not be obvious. This is how you would do it in your CMakeLists.txt:<br />
<div>
<a name='more'></a># CRY</div>
<div>
<pre><code>include(ExternalProject)
ExternalProject_Add(
CRY-1.7
URL http://nuclear.llnl.gov/simulation/cry_v1.7.tar.gz
URL_MD5 85f240bebe81fe0b257e92ef1d390a83
CONFIGURE_COMMAND ""
INSTALL_COMMAND ""
BUILD_IN_SOURCE 1
TEST_AFTER_INSTALL 1
)
ExternalProject_Get_Property(CRY-1.7 source_dir)
set(CRY_INCLUDE_DIRS "${source_dir}/src")
set(CRY_LIBRARY_DIR "${source_dir}/lib")
set(CRY_LIBRARIES "libCRY.a")
include_directories(${CRY_INCLUDE_DIRS})
link_directories(${CRY_LIBRARY_DIR})
set(LINK_LIBS ${CRY_LIBRARIES} ${LINK_LIBS})
</code>
</pre>
</div>
<div>
<br /></div>
<div>
Hope that helps! As this is a Python blog, here's a CMake based version of CRY with Python bindings: <a href="https://bitbucket.org/mayamuon/cry">https://bitbucket.org/mayamuon/cry</a>.</div>
Anonymoushttp://www.blogger.com/profile/10752018806658508906noreply@blogger.com0tag:blogger.com,1999:blog-8303893118597899654.post-67503739220551346242015-10-07T06:43:00.000-07:002016-03-25T16:17:38.884-07:00GTest Submodule <div tabindex="-1" id="notebook" class="border-box-sizing">
<div class="container" id="notebook-container">
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<blockquote><p>Note: There is a better way to do this described <a href="http://iscinumpy.blogspot.com/2016/03/googletest-and-cmake.html">here</a>.</p>
</blockquote>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>If you've ever tried <code>apt-get</code> or <code>brew</code> to try to install gtest, you arer probably familiar with the fact that gtest is not "recommend" for global install on your system. As an alternitive, the recommendation is that you make it part of your project. The process for making gtest part of your project, however, is not well documented, at least for modern git projects. What follows is the procedure I used to do so.</p>
</div>
</div>
</div><a name='more'></a>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Now that gtest is a git repository, you'll want to check it out as a submodule for your project. The command is</p>
<div class="highlight"><pre><span class="nv">$ </span>git submodule add https://github.com/google/googletest.git
<span class="nv">$ </span><span class="nb">cd </span>googletest
<span class="nv">$ </span>git checkout release-1.7.0
<span class="nv">$ </span><span class="nb">cd</span> ..
<span class="nv">$ </span>git add googletest
</pre></div>
<p>Here, we've added googletest as a submodule, then changed the reference to the 1.7.0 tag (assuming you don't want some random commit in the middle of a release cycle).</p>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Now, to set up cmake, add the following to your main <code>CMakeLists.txt</code> right before your <code>add_subdirectory(tests)</code> line, so that it looks like this:</p>
<pre><code>add_subdirectory(googletest)
enable_testing()
add_subdirectory(tests)</code></pre>
<p>Now, your <code>tests/CMakeLists.txt</code> should look something like this:</p>
<pre><code>include_directories(SYSTEM
${gtest_SOURCE_DIR}
${gtest_SOURCE_DIR}/include)
file(GLOB_RECURSE test_cases *.cpp)
foreach(case_file ${test_cases})
get_filename_component( case_name ${case_file} NAME_WE )
set (case_name test_${case_name})
add_executable(${case_name} ${case_file})
target_link_libraries(${case_name} ${PROJECT_NAME}
${LINK_LIBS} gtest_main pthread)
add_test(NAME ${case_name}
COMMAND ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/${case_name}
WORKING_DIRECTORY
${PROJECT_BINARY_DIR})
endforeach()</code></pre>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Of course, you may need to edit this a little, like if you don't want to have <code>test_</code> appended to each test binary name.</p>
</div>
</div>
</div>
</div>
</div>Anonymoushttp://www.blogger.com/profile/10752018806658508906noreply@blogger.com0tag:blogger.com,1999:blog-8303893118597899654.post-42031001067048907662015-08-06T16:01:00.003-07:002015-08-06T16:01:56.443-07:00Slots in Python <div tabindex="-1" id="notebook" class="border-box-sizing">
<div class="container" id="notebook-container">
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Slots seem to be poorly documented. What they do is simple, but whether they are used is tricky. This is a little mini-post on slots.</p>
</div>
</div>
</div><a name='more'></a>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Immubable-list-of-instance-attributes">Immubable list of instance attributes<a class="anchor-link" href="#Immubable-list-of-instance-attributes">¶</a></h2>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Slots replace the mutible dictionary of <em>instance</em> level attributes (<code>__dict__</code>) that most subclasses have with an immutible one. For example:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [3]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="k">class</span> <span class="nc">NoSlots</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">pass</span>
<span class="k">class</span> <span class="nc">Slots</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="n">__slots__</span> <span class="o">=</span> <span class="p">(</span><span class="s">'x'</span><span class="p">,)</span>
<span class="k">pass</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>We need an instance, as class level attributes (shared by all instances of a class) are not affected by <code>__slots__</code>.</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [4]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">noslots</span> <span class="o">=</span> <span class="n">NoSlots</span><span class="p">()</span>
<span class="n">slots</span> <span class="o">=</span> <span class="n">Slots</span><span class="p">()</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [5]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">noslots</span><span class="o">.</span><span class="n">x</span> <span class="o">=</span> <span class="mi">2</span>
<span class="n">slots</span><span class="o">.</span><span class="n">x</span> <span class="o">=</span> <span class="mi">2</span>
<span class="n">noslots</span><span class="o">.</span><span class="n">y</span> <span class="o">=</span> <span class="mi">3</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [6]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">slots</span><span class="o">.</span><span class="n">y</span> <span class="o">=</span> <span class="mi">3</span> <span class="c"># Fails</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_text output_error">
<pre>
<span class="ansired">---------------------------------------------------------------------------</span>
<span class="ansired">AttributeError</span> Traceback (most recent call last)
<span class="ansigreen"><ipython-input-6-db7b0acdb722></span> in <span class="ansicyan"><module></span><span class="ansiblue">()</span>
<span class="ansigreen">----> 1</span><span class="ansiyellow"> </span>slots<span class="ansiyellow">.</span>y <span class="ansiyellow">=</span> <span class="ansicyan">3</span> <span class="ansired"># Fails</span><span class="ansiyellow"></span>
<span class="ansired">AttributeError</span>: 'Slots' object has no attribute 'y'</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Since there is no <code>__dict__</code> to store the instance variable <code>.y</code> in, this is error. This makes <code>__slots__</code> a lot like the list of member variables in other languages, and lets you see all the possible instance variables in one place, instead of checking all the methods.</p>
<p>This feature was actually intended, however, to save memory and performance, since each instance now has a specific set of slots they can put variables in, instead of each one tracking the possible variables. However, it doesn't make much of a difference unless you have a very large number of instances. Python 3.3 reduced the difference even further with key sharing dictionaries. I would argue the previous reason is more important; I've caught bugs with misnamed variables when adding slots.</p>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="When-slots-count">When slots count<a class="anchor-link" href="#When-slots-count">¶</a></h2><p>Slots only work in a special case: all parent classes have to have <code>__slots__</code> too (that is, none of them can have an instance <code>__dict__</code>). Most or all the builtins do this, so you just need to be careful when subclassing your own or third party library types.</p>
<p><code>__slots__</code> combine in an odd way, but it makes sense: A class has it's own and all parent classes <code>__slots__</code> combined. So, for example, look at the following subclass:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [6]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="k">class</span> <span class="nc">Vector2D</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="n">__slots__</span> <span class="o">=</span> <span class="p">(</span><span class="s">'x'</span><span class="p">,</span><span class="s">'y'</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">Vector3D</span><span class="p">(</span><span class="n">Vector2D</span><span class="p">):</span>
<span class="n">__slots__</span> <span class="o">=</span> <span class="p">(</span><span class="s">'z'</span><span class="p">,)</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Here, <code>Vector3D</code> has three instance variables, x, y, and z; since it inherits the Vector2D slots. You can't remove slots (as that would break the class anyway), and you keep the same slots by setting slots to an empty tuple (or similar). If you igore slots, the class becomes a <code>__dict__</code> class.</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [7]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="k">class</span> <span class="nc">Also2Slots</span><span class="p">(</span><span class="n">Vector2D</span><span class="p">):</span>
<span class="n">__slots__</span><span class="o">=</span> <span class="p">()</span>
<span class="k">class</span> <span class="nc">NoSlots</span><span class="p">(</span><span class="n">Vector2D</span><span class="p">):</span>
<span class="k">pass</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [10]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">Also2Slots</span><span class="p">()</span><span class="o">.</span><span class="n">z</span> <span class="o">=</span> <span class="mi">2</span> <span class="c"># Won't work, slots are x and y</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_text output_error">
<pre>
<span class="ansired">---------------------------------------------------------------------------</span>
<span class="ansired">AttributeError</span> Traceback (most recent call last)
<span class="ansigreen"><ipython-input-10-862813c200ee></span> in <span class="ansicyan"><module></span><span class="ansiblue">()</span>
<span class="ansigreen">----> 1</span><span class="ansiyellow"> </span>Also2Slots<span class="ansiyellow">(</span><span class="ansiyellow">)</span><span class="ansiyellow">.</span>z <span class="ansiyellow">=</span> <span class="ansicyan">2</span> <span class="ansired"># Won't work, slots are x and y</span><span class="ansiyellow"></span>
<span class="ansired">AttributeError</span>: 'Also2Slots' object has no attribute 'z'</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [9]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">NoSlots</span><span class="p">()</span><span class="o">.</span><span class="n">z</span> <span class="o">=</span> <span class="mi">2</span> <span class="c"># No error!</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Interesting-consequence">Interesting consequence<a class="anchor-link" href="#Interesting-consequence">¶</a></h2><p>Have you ever wondered why this doesn't work:</p>
<div class="highlight"><pre><span class="n">thing</span> <span class="o">=</span> <span class="nb">object</span><span class="p">()</span>
<span class="n">thing</span><span class="o">.</span><span class="n">x</span> <span class="o">=</span> <span class="mi">2</span> <span class="c"># Error!</span>
</pre></div>
<p>But this does:</p>
<div class="highlight"><pre><span class="k">class</span> <span class="nc">SubObject</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">pass</span>
<span class="n">thing</span> <span class="o">=</span> <span class="n">SubObject</span><span class="p">()</span>
<span class="n">thing</span><span class="o">.</span><span class="n">x</span> <span class="o">=</span> <span class="mi">2</span> <span class="c"># Works!</span>
</pre></div>
<p>Now you know, <code>object</code> has <code>__slots__</code>, but <code>SubObject</code> doesn't.</p>
<blockquote><p>Side note: In Python 3.3+, the <code>SimpleNamespace</code> object is a lot like <code>object</code> without the <code>__slots__</code> (and a fancy constructor).</p>
</blockquote>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [ ]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre>
</pre></div>
</div>
</div>
</div>
</div>
</div>
</div>Anonymoushttp://www.blogger.com/profile/10752018806658508906noreply@blogger.com0tag:blogger.com,1999:blog-8303893118597899654.post-82116105593197801992015-08-06T15:59:00.000-07:002015-08-06T15:59:16.506-07:00Basics of Metaclasses <div tabindex="-1" id="notebook" class="border-box-sizing">
<div class="container" id="notebook-container">
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>This is a quick tutorial over the basics of what metaclasses do.</p>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h1 id="The-Metaclass">The Metaclass<a class="anchor-link" href="#The-Metaclass">¶</a></h1><p>Metaclasses, while seemingly a complex topic, really just do something very simple. They control what happens when you have code that turns into a class object. The normal place they are executed is right after the class statement. Let's see that in action by using print as our metaclass.</p>
<blockquote><p>Note: this post uses Python 3 metaclass notation. Python 2 uses assignment to a special <code>__metaclass__</code> attribute to set the metaclass. Also, Python 2 requires explicit, 2 argument <code>super()</code> calls.</p>
</blockquote>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [13]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="k">class</span> <span class="nc">WillNotBeAClass</span><span class="p">(</span><span class="nb">object</span><span class="p">,</span> <span class="n">metaclass</span><span class="o">=</span><span class="nb">print</span><span class="p">):</span>
<span class="n">x</span><span class="o">=</span><span class="mi">1</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>WillNotBeAClass (<class 'object'>,) {'x': 1, '__module__': '__main__', '__qualname__': 'WillNotBeAClass'}
</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Here, we have replaced the metaclass (<code>type</code>) with <code>print</code>, just to investigate how it works. This is quite useless, of course, but does show that the metaclass gets called with three arguments when a class is created.
The first is the name of the class to be created, the second is a tuple of base classes, and the third is a dictionary that has the namespace of the body of a class, with a few extra special values added.</p>
</div>
</div>
</div><a name='more'></a>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Given this, we know see show to make this into a class using type: (I will not bother to add <code>__module__</code> and <code>__qualname__</code> for now, they are not needed)</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [16]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="k">class</span> <span class="nc">WillBeAClass</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="n">x</span><span class="o">=</span><span class="mi">1</span>
<span class="n">WillBeAClass</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt output_prompt">Out[16]:</div>
<div class="output_text output_subarea output_execute_result">
<pre>__main__.WillBeAClass</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [17]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">WillAlsoBeAClass</span> <span class="o">=</span> <span class="nb">type</span><span class="p">(</span><span class="s">'WillAlsoBeAClass'</span><span class="p">,</span> <span class="p">(</span><span class="nb">object</span><span class="p">,),</span> <span class="p">{</span><span class="s">'x'</span><span class="p">:</span><span class="mi">1</span><span class="p">})</span>
<span class="n">WillAlsoBeAClass</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt output_prompt">Out[17]:</div>
<div class="output_text output_subarea output_execute_result">
<pre>__main__.WillAlsoBeAClass</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>These two objects, <code>WillBeAClass</code> and <code>WillAlsoBeAClass</code>, are basically the same thing. The second method is exactly what the class statement does (with <code>__module__</code> and <code>__qualname__</code> added).</p>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h1 id="The-type">The type<a class="anchor-link" href="#The-type">¶</a></h1><p>So, we are done with metaclasses, that's all there is to know. However, to actually make useful classes, you probably want to make normal classes, just with some sort of modification. For that, you need to understand <code>type</code>, and how it works, and how to subclass it.</p>
<p>First, let's pretend we can just patch type and ignore subclassing. You probably already see how to do that:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [24]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="k">def</span> <span class="nf">newtype</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">):</span>
<span class="nb">print</span><span class="p">(</span><span class="s">"I'm sort of a new type, but I have a problem!"</span><span class="p">)</span>
<span class="k">return</span> <span class="nb">type</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [27]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="k">class</span> <span class="nc">NewClass</span><span class="p">(</span><span class="nb">object</span><span class="p">,</span> <span class="n">metaclass</span><span class="o">=</span><span class="n">newtype</span><span class="p">):</span>
<span class="n">x</span> <span class="o">=</span> <span class="mi">1</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>I'm sort of a new type, but I have a problem!
</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [28]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="k">class</span> <span class="nc">NewNewClass</span><span class="p">(</span><span class="n">NewClass</span><span class="p">):</span>
<span class="n">y</span> <span class="o">=</span> <span class="mi">2</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>All was fine and well, until we subclassed <code>NewClass</code>. The metaclass did not come along for the ride! That's because <code>type</code> adds a reference to itself when it creates a class:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [30]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">NewClass</span><span class="o">.</span><span class="n">__class__</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt output_prompt">Out[30]:</div>
<div class="output_text output_subarea output_execute_result">
<pre>type</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<blockquote><p>Note: the standard way to check the class of an object is to call <code>type(NewClass)</code>, however, since that is an unrelated use of <code>type</code> that is there for historical reasons, I've avoided using it here)</p>
</blockquote>
<h1 id="How-type-works">How type works<a class="anchor-link" href="#How-type-works">¶</a></h1><p>We must subclass type to get a metaclass that actually works on subclasses, too: (Here I'm overriding all the used parameters, so that you can see where each gets called)</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [41]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="k">class</span> <span class="nc">NewType</span><span class="p">(</span><span class="nb">type</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">__new__</span><span class="p">(</span><span class="n">cls</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kargs</span><span class="p">):</span>
<span class="nb">print</span><span class="p">(</span><span class="s">"I'm a new type! __new__"</span><span class="p">)</span>
<span class="k">return</span> <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">__new__</span><span class="p">(</span><span class="n">cls</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kargs</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kargs</span><span class="p">):</span>
<span class="nb">print</span><span class="p">(</span><span class="s">"I'm a new type! __init__"</span><span class="p">)</span>
<span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kargs</span><span class="p">)</span>
<span class="nd">@classmethod</span>
<span class="k">def</span> <span class="nf">__prepare__</span><span class="p">(</span><span class="n">cls</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kargs</span><span class="p">):</span>
<span class="nb">print</span><span class="p">(</span><span class="s">"I'm new in Python 3! __prepare__"</span><span class="p">)</span>
<span class="k">return</span> <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">__prepare__</span><span class="p">(</span><span class="n">cls</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kargs</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">__call__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kargs</span><span class="p">):</span>
<span class="nb">print</span><span class="p">(</span><span class="s">"I'm a new type! __call__"</span><span class="p">)</span>
<span class="k">return</span> <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">__call__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kargs</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [42]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="k">class</span> <span class="nc">NewClass</span><span class="p">(</span><span class="nb">object</span><span class="p">,</span> <span class="n">metaclass</span><span class="o">=</span><span class="n">NewType</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="nb">print</span><span class="p">(</span><span class="s">"I'm init in the class"</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">__new__</span><span class="p">(</span><span class="n">cls</span><span class="p">):</span>
<span class="nb">print</span><span class="p">(</span><span class="s">"I'm new"</span><span class="p">)</span>
<span class="k">return</span> <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">__new__</span><span class="p">(</span><span class="n">cls</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>I'm new in Python 3! __prepare__
I'm a new type! __new__
I'm a new type! __init__
</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [17]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="k">class</span> <span class="nc">NewNewClass</span><span class="p">(</span><span class="n">NewClass</span><span class="p">):</span>
<span class="n">y</span> <span class="o">=</span> <span class="mi">2</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>I'm new in Python 3! __prepare__
I'm a new type! __new__
I'm a new type! __init__
</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [18]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">instance</span> <span class="o">=</span> <span class="n">NewClass</span><span class="p">()</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>I'm a new type! __call__
I'm new
I'm init in the class
</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Notice how <code>__init__</code> was used, too? This gives us a peek at one more feature of metaclasses: the <code>__class__</code> parameter of a class is used to create instances. The super part of <code>__call__</code> actually puts together the class, it's where <code>__new__</code> and <code>__init__</code> are called, etc.</p>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Python-3-only">Python 3 only<a class="anchor-link" href="#Python-3-only">¶</a></h2><p>As you already have seen, the <code>__prepare__</code> method is only in Python 3, and allows you to customize the <code>__dict__</code> before <code>__new__</code>, however, as a reminder, in CPython the dict for a class is written in C and is not customizable (ie, can't be ordered, etc). So you'll have to manage that yourself, but <code>__prepare__</code> helps. It returns a dictionary-like object that then collects the namespace, then gets passed to <code>__new__</code>.</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [40]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="kn">from</span> <span class="nn">collections</span> <span class="k">import</span> <span class="n">OrderedDict</span>
<span class="k">class</span> <span class="nc">PrepareMeta</span><span class="p">(</span><span class="nb">type</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">__new__</span><span class="p">(</span><span class="n">cls</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">bases</span><span class="p">,</span> <span class="n">ns</span><span class="p">):</span>
<span class="nb">print</span><span class="p">(</span><span class="n">ns</span><span class="p">)</span>
<span class="k">return</span> <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">__new__</span><span class="p">(</span><span class="n">cls</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">bases</span><span class="p">,</span> <span class="n">ns</span><span class="p">)</span>
<span class="nd">@classmethod</span>
<span class="k">def</span> <span class="nf">__prepare__</span><span class="p">(</span><span class="n">cls</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kargs</span><span class="p">):</span>
<span class="k">return</span> <span class="n">OrderedDict</span><span class="p">()</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [29]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="k">class</span> <span class="nc">PrepareClass</span><span class="p">(</span><span class="n">metaclass</span><span class="o">=</span><span class="n">PrepareMeta</span><span class="p">):</span>
<span class="n">y</span> <span class="o">=</span> <span class="mi">2</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>OrderedDict([('__module__', '__main__'), ('__qualname__', 'PrepareClass'), ('y', 2)])
</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>We only get that one look at the dict, since it becomes the special C <code>mappingproxy</code> once the class is created.</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [30]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">PrepareClass</span><span class="o">.</span><span class="n">__dict__</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt output_prompt">Out[30]:</div>
<div class="output_text output_subarea output_execute_result">
<pre>mappingproxy({'__doc__': None, '__weakref__': <attribute '__weakref__' of 'PrepareClass' objects>, '__module__': '__main__', 'y': 2, '__dict__': <attribute '__dict__' of 'PrepareClass' objects>})</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Another Python 3 only feature is class level arguments. You can do things like this:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [39]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="k">class</span> <span class="nc">ArgMeta</span><span class="p">(</span><span class="nb">type</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">__new__</span><span class="p">(</span><span class="n">cls</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kargs</span><span class="p">):</span>
<span class="nb">print</span><span class="p">(</span><span class="n">kargs</span><span class="p">)</span>
<span class="k">return</span> <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">__new__</span><span class="p">(</span><span class="n">cls</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kargs</span><span class="p">):</span>
<span class="k">return</span> <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">ArgClass</span><span class="p">(</span><span class="n">metaclass</span><span class="o">=</span><span class="n">ArgMeta</span><span class="p">,</span> <span class="n">kwarg</span> <span class="o">=</span> <span class="mi">2</span><span class="p">):</span>
<span class="n">y</span> <span class="o">=</span> <span class="mi">2</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>{'kwarg': 2}
</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Example">Example<a class="anchor-link" href="#Example">¶</a></h2><p>A dictionary can be make using the following ugly hack:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [44]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="k">class</span> <span class="nc">a_dictionary</span><span class="p">(</span><span class="n">metaclass</span><span class="o">=</span><span class="k">lambda</span> <span class="n">name</span><span class="p">,</span> <span class="n">bases</span><span class="p">,</span> <span class="n">ns</span><span class="p">:</span> <span class="p">{</span><span class="n">n</span><span class="p">:</span><span class="n">ns</span><span class="p">[</span><span class="n">n</span><span class="p">]</span> <span class="k">for</span> <span class="n">n</span> <span class="ow">in</span> <span class="n">ns</span> <span class="k">if</span> <span class="s">'__'</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">n</span><span class="p">}):</span>
<span class="n">one</span> <span class="o">=</span> <span class="mi">1</span>
<span class="n">two</span> <span class="o">=</span> <span class="mi">2</span>
<span class="n">three</span> <span class="o">=</span> <span class="mi">3</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [46]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="nb">print</span><span class="p">(</span><span class="n">a_dictionary</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>{'three': 3, 'one': 1, 'two': 2}
</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>An ordered class can be make using the <code>__prepare__</code> method (Python 3 only):</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [49]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="k">class</span> <span class="nc">OrderedMeta</span><span class="p">(</span><span class="nb">type</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">__new__</span><span class="p">(</span><span class="n">cls</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">bases</span><span class="p">,</span> <span class="n">ns</span><span class="p">):</span>
<span class="n">ns</span><span class="p">[</span><span class="s">'orderednames'</span><span class="p">]</span><span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">ns</span><span class="p">)</span>
<span class="k">return</span> <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">__new__</span><span class="p">(</span><span class="n">cls</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">bases</span><span class="p">,</span> <span class="n">ns</span><span class="p">)</span>
<span class="nd">@classmethod</span>
<span class="k">def</span> <span class="nf">__prepare__</span><span class="p">(</span><span class="n">cls</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kargs</span><span class="p">):</span>
<span class="k">return</span> <span class="n">OrderedDict</span><span class="p">()</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [55]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="k">class</span> <span class="nc">Ordered</span><span class="p">(</span><span class="n">metaclass</span><span class="o">=</span><span class="n">OrderedMeta</span><span class="p">):</span>
<span class="n">one</span> <span class="o">=</span> <span class="mi">1</span>
<span class="n">two</span> <span class="o">=</span> <span class="mi">2</span>
<span class="n">three</span> <span class="o">=</span> <span class="mi">3</span>
<span class="n">four</span> <span class="o">=</span> <span class="mi">4</span>
<span class="n">five</span> <span class="o">=</span> <span class="mi">5</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [57]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="nb">print</span><span class="p">(</span><span class="n">Ordered</span><span class="o">.</span><span class="n">__dict__</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>{'one': 1, 'three': 3, 'orderednames': ['__module__', '__qualname__', 'one', 'two', 'three', 'four', 'five'], '__weakref__': <attribute '__weakref__' of 'Ordered' objects>, '__dict__': <attribute '__dict__' of 'Ordered' objects>, '__doc__': None, 'five': 5, 'two': 2, 'four': 4, '__module__': '__main__'}
['__module__', '__qualname__', 'one', 'two', 'three', 'four', 'five']
</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Here we can see that the list <code>orderednames</code> is ordered correctly.</p>
</div>
</div>
</div>
</div>
</div>Anonymoushttp://www.blogger.com/profile/10752018806658508906noreply@blogger.com0tag:blogger.com,1999:blog-8303893118597899654.post-50856606178722683342015-07-29T09:13:00.000-07:002015-07-29T09:42:07.417-07:00Factory Classmethods in Python
<div tabindex="-1" id="notebook" class="border-box-sizing">
<div class="container" id="notebook-container">
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>I haven't seen a great deal of practical documentation about using classmethods as factories in Python (which is arguably the most important use of a classmethod, IMO). This post hopes to fill in that gap.</p>
</div>
</div>
</div>
<a name='more'></a>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Simple-Example-of-classmethod-Factory">Simple Example of <code>classmethod</code> Factory<a class="anchor-link" href="#Simple-Example-of-classmethod-Factory">¶</a></h2><p>This is not to hard to find <a href="https://julien.danjou.info/blog/2013/guide-python-static-class-abstract-methods">a good example of</a>, but here is a simple example of a class method being used for a generator. Let's say you have the following vector class, and you want to be able to make a new vector using <code>Vector(x,y,z)</code> or <code>Vector.from_cyl(ρ,θ,z)</code> (as usual, I'm exploiting Python 3's unicode variable names, change these to plain text if you are using Python 2). Here is how we would do that, with a <code>repr</code> added to make the display easier to see:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [9]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="kn">from</span> <span class="nn">math</span> <span class="k">import</span> <span class="n">sin</span><span class="p">,</span> <span class="n">cos</span><span class="p">,</span> <span class="n">pi</span> <span class="k">as</span> <span class="n">π</span>
<span class="k">class</span> <span class="nc">Vector</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">z</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">x</span> <span class="o">=</span> <span class="n">x</span>
<span class="bp">self</span><span class="o">.</span><span class="n">y</span> <span class="o">=</span> <span class="n">y</span>
<span class="bp">self</span><span class="o">.</span><span class="n">z</span> <span class="o">=</span> <span class="n">z</span>
<span class="nd">@classmethod</span>
<span class="k">def</span> <span class="nf">from_cyl</span><span class="p">(</span><span class="n">cls</span><span class="p">,</span> <span class="n">ρ</span><span class="p">,</span> <span class="n">θ</span><span class="p">,</span> <span class="n">z</span><span class="p">):</span>
<span class="k">return</span> <span class="n">cls</span><span class="p">(</span><span class="n">ρ</span><span class="o">*</span><span class="n">cos</span><span class="p">(</span><span class="n">θ</span><span class="p">),</span> <span class="n">ρ</span><span class="o">*</span><span class="n">sin</span><span class="p">(</span><span class="n">θ</span><span class="p">),</span> <span class="n">z</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">__repr__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="s">"<{0.__class__.__name__}({0.x}, {0.y}, {0.z})>"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [10]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">Vector</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt output_prompt">Out[10]:</div>
<div class="output_text output_subarea output_execute_result">
<pre><Vector(1, 2, 3)></pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [11]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">Vector</span><span class="o">.</span><span class="n">from_cyl</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="n">π</span><span class="o">/</span><span class="mi">4</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt output_prompt">Out[11]:</div>
<div class="output_text output_subarea output_execute_result">
<pre><Vector(0.7071067811865476, 0.7071067811865475, 1)></pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>A few take away points:</p>
<ul>
<li>Class methods take the class as the first argument, traditionally <code>cls</code>.</li>
<li>Class methods that are factories call <code>cls()</code>, and return the new instance they created.</li>
<li>Both <code>__new__</code> and <code>__init__</code> ran when <code>cls()</code> was called.</li>
</ul>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Avoiding-__init__">Avoiding <code>__init__</code><a class="anchor-link" href="#Avoiding-__init__">¶</a></h2><p>This is fine, but there are two more features that we often need. First, we might not want to call the <code>__init__</code> function at all; this is fine in the above example, but we might not happen to be able to make the transformation to the original parameters. If we can't do that, it is actually possible in Python to create a new instance of a class without calling <code>__init__</code> using:</p>
<div class="highlight"><pre><span class="n">cls</span><span class="o">.</span><span class="n">__new__</span><span class="p">(</span><span class="n">cls</span><span class="p">,</span> <span class="o">...</span><span class="p">)</span>
</pre></div>
<p>where the ellipsis are the parameters for the new function (often this is simply <code>cls.__new__(cls)</code>).</p>
<p>Take the following example, where we've reversed the class, so that cylindrical is now default:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [14]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="kn">from</span> <span class="nn">math</span> <span class="k">import</span> <span class="n">sin</span><span class="p">,</span> <span class="n">cos</span><span class="p">,</span> <span class="n">pi</span> <span class="k">as</span> <span class="n">π</span>
<span class="k">class</span> <span class="nc">Vector</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">ρ</span><span class="p">,</span> <span class="n">θ</span><span class="p">,</span> <span class="n">z</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">x</span> <span class="o">=</span> <span class="n">ρ</span><span class="o">*</span><span class="n">cos</span><span class="p">(</span><span class="n">θ</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">y</span> <span class="o">=</span> <span class="n">ρ</span><span class="o">*</span><span class="n">sin</span><span class="p">(</span><span class="n">θ</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">z</span> <span class="o">=</span> <span class="n">z</span>
<span class="nd">@classmethod</span>
<span class="k">def</span> <span class="nf">from_xyz</span><span class="p">(</span><span class="n">cls</span><span class="p">,</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">z</span><span class="p">):</span>
<span class="n">ob</span> <span class="o">=</span> <span class="n">cls</span><span class="o">.</span><span class="n">__new__</span><span class="p">(</span><span class="n">cls</span><span class="p">)</span>
<span class="n">ob</span><span class="o">.</span><span class="n">x</span> <span class="o">=</span> <span class="n">x</span>
<span class="n">ob</span><span class="o">.</span><span class="n">y</span> <span class="o">=</span> <span class="n">y</span>
<span class="n">ob</span><span class="o">.</span><span class="n">z</span> <span class="o">=</span> <span class="n">z</span>
<span class="k">return</span> <span class="n">ob</span>
<span class="k">def</span> <span class="nf">__repr__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="s">"<{0.__class__.__name__}({0.x}, {0.y}, {0.z})>"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [15]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">Vector</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="n">π</span><span class="o">/</span><span class="mi">4</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt output_prompt">Out[15]:</div>
<div class="output_text output_subarea output_execute_result">
<pre><Vector(0.7071067811865476, 0.7071067811865475, 1)></pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [16]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">Vector</span><span class="o">.</span><span class="n">from_xyz</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt output_prompt">Out[16]:</div>
<div class="output_text output_subarea output_execute_result">
<pre><Vector(1, 2, 3)></pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Though it should be obvious now, I'll point out that each class method using this technique is responible for doing anything that the <code>__init__</code> function normally does. In this case, <code>self.x</code>, <code>.y</code>, and <code>.z</code> all must be manually set. If we forgot to set <code>self.z</code>, for example, bad things might happen when we call other methods.</p>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h1 id="Auto-generator">Auto generator<a class="anchor-link" href="#Auto-generator">¶</a></h1><h2 id="__init__-Method"><code>__init__</code> Method<a class="anchor-link" href="#__init__-Method">¶</a></h2><p>This is something I do a lot that is still is difficult with above trick. Let's say you want to have an auto-selecting factory function. For example, if you have <code>.from_csv(filename)</code> and <code>.from_excel(filename)</code> functions, you might want to make a <code>.from_any</code> function that bases it's choise of loading function on the extenstion of <code>filename</code>. This is easy to do with normal factory functions unless you decide that you'd like your <code>__init__</code> function to act like <code>.from_any</code>. Then, you'll need to write two functions for each factory function (this only uses one <code>from_</code> method pair for clarity):</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [23]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="k">class</span> <span class="nc">FromFile</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">filename</span><span class="p">):</span>
<span class="k">if</span> <span class="n">filename</span><span class="p">[</span><span class="o">-</span><span class="mi">3</span><span class="p">:]</span> <span class="o">==</span> <span class="s">'csv'</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_from_csv</span><span class="p">(</span><span class="n">filename</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">file</span> <span class="o">=</span> <span class="s">'Not a valid format'</span>
<span class="nd">@classmethod</span>
<span class="k">def</span> <span class="nf">from_csv</span><span class="p">(</span><span class="n">cls</span><span class="p">,</span> <span class="n">filename</span><span class="p">):</span>
<span class="bp">self</span> <span class="o">=</span> <span class="n">cls</span><span class="o">.</span><span class="n">__new__</span><span class="p">(</span><span class="n">cls</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_from_csv</span><span class="p">(</span><span class="n">filename</span><span class="p">)</span>
<span class="k">return</span> <span class="bp">self</span>
<span class="k">def</span> <span class="nf">_from_csv</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">filename</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">file</span> <span class="o">=</span> <span class="s">'I got {0} from csv!'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">filename</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">__repr__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="s">"<{0.__class__.__name__}({0.file})>"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [20]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">FromFile</span><span class="p">(</span><span class="s">'this.file'</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt output_prompt">Out[20]:</div>
<div class="output_text output_subarea output_execute_result">
<pre><FromFile(Not a valid format)></pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [21]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">FromFile</span><span class="p">(</span><span class="s">"this.csv"</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt output_prompt">Out[21]:</div>
<div class="output_text output_subarea output_execute_result">
<pre><FromFile(I got this.csv from csv!)></pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [25]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">FromFile</span><span class="o">.</span><span class="n">from_csv</span><span class="p">(</span><span class="s">"this.is"</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt output_prompt">Out[25]:</div>
<div class="output_text output_subarea output_execute_result">
<pre><FromFile(I got this.is from csv!)></pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="__new__-Method"><code>__new__</code> Method<a class="anchor-link" href="#__new__-Method">¶</a></h2><p>One other way we could do this would be to change <code>__new__</code> itself. This is technically what we want anyway; <code>__new__</code> is where new instances of a class get created.</p>
<p>The downside to this method is that we are limited in our use of <code>__init__</code> with arguments (if you have an <code>__init__</code>, you need to accept everything that <code>__new__</code> accepts). <code>__init__</code> will run only when we call the class normally, so it may or may not run when a factory function is called, forcing you to put any common init code in a seperate function, and then manually calling it from each factory function, and <code>__new__</code> too, if necessary.</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [62]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="k">class</span> <span class="nc">FromFile</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">__new__</span><span class="p">(</span><span class="n">cls</span><span class="p">,</span> <span class="n">filename</span><span class="p">):</span>
<span class="k">if</span> <span class="n">filename</span> <span class="ow">is</span> <span class="k">None</span><span class="p">:</span>
<span class="k">return</span> <span class="nb">super</span><span class="p">(</span><span class="n">FromFile</span><span class="p">,</span><span class="n">cls</span><span class="p">)</span><span class="o">.</span><span class="n">__new__</span><span class="p">(</span><span class="n">cls</span><span class="p">)</span>
<span class="k">elif</span> <span class="n">filename</span><span class="p">[</span><span class="o">-</span><span class="mi">3</span><span class="p">:]</span> <span class="o">==</span> <span class="s">'csv'</span><span class="p">:</span>
<span class="k">return</span> <span class="n">cls</span><span class="o">.</span><span class="n">from_csv</span><span class="p">(</span><span class="n">filename</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="bp">self</span> <span class="o">=</span> <span class="nb">super</span><span class="p">(</span><span class="n">FromFile</span><span class="p">,</span><span class="n">cls</span><span class="p">)</span><span class="o">.</span><span class="n">__new__</span><span class="p">(</span><span class="n">cls</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">file</span> <span class="o">=</span> <span class="s">'Not a valid format'</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_common_init_code</span><span class="p">()</span>
<span class="k">return</span> <span class="bp">self</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kargs</span><span class="p">):</span>
<span class="nb">print</span><span class="p">(</span><span class="s">"This was called directly with"</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">_common_init_code</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="nb">print</span><span class="p">(</span><span class="s">"All correct creations of this class will print this line!"</span><span class="p">)</span>
<span class="nd">@classmethod</span>
<span class="k">def</span> <span class="nf">from_csv</span><span class="p">(</span><span class="n">cls</span><span class="p">,</span> <span class="n">filename</span><span class="p">):</span>
<span class="bp">self</span> <span class="o">=</span> <span class="nb">super</span><span class="p">(</span><span class="n">FromFile</span><span class="p">,</span><span class="n">cls</span><span class="p">)</span><span class="o">.</span><span class="n">__new__</span><span class="p">(</span><span class="n">cls</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">file</span> <span class="o">=</span> <span class="s">'I got {0} from csv!'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">filename</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_common_init_code</span><span class="p">()</span>
<span class="k">return</span> <span class="bp">self</span>
<span class="k">def</span> <span class="nf">__repr__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="s">"<{0.__class__.__name__}({0.file})>"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [63]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">FromFile</span><span class="p">(</span><span class="s">'this.file'</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>All correct creations of this class will print this line!
This was called directly with this.file
</pre>
</div>
</div>
<div class="output_area"><div class="prompt output_prompt">Out[63]:</div>
<div class="output_text output_subarea output_execute_result">
<pre><FromFile(Not a valid format)></pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [64]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">FromFile</span><span class="p">(</span><span class="s">"this.csv"</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>All correct creations of this class will print this line!
This was called directly with this.csv
</pre>
</div>
</div>
<div class="output_area"><div class="prompt output_prompt">Out[64]:</div>
<div class="output_text output_subarea output_execute_result">
<pre><FromFile(I got this.csv from csv!)></pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [65]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">FromFile</span><span class="o">.</span><span class="n">from_csv</span><span class="p">(</span><span class="s">"this.is"</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>All correct creations of this class will print this line!
</pre>
</div>
</div>
<div class="output_area"><div class="prompt output_prompt">Out[65]:</div>
<div class="output_text output_subarea output_execute_result">
<pre><FromFile(I got this.is from csv!)></pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>There are other ways to generate objects of certain classes; <a href="http://www.informit.com/articles/article.aspx?p=2131418">subclassing</a> is a valid <a href="http://python-3-patterns-idioms-test.readthedocs.org/en/latest/Factory.html">method</a>, or <a href="http://stackoverflow.com/questions/456672/class-factory-in-python">using a factory function</a>, or even <a href="http://sahandsaba.com/python-classes-metaclasses.html">metaclasses</a>. (For metaclasses, <a href="http://blog.ionelmc.ro/2015/02/09/understanding-python-metaclasses/">this article</a> is hard to beat.) Several of these methods cause the type of the object not match the object used to create it (like <code>numpy.array()</code> is a <code>numpy.ndarray</code>), but still is commonly used.</p>
</div>
</div>
</div>
</div>
</div>Anonymoushttp://www.blogger.com/profile/10752018806658508906noreply@blogger.com0tag:blogger.com,1999:blog-8303893118597899654.post-70984003047548341682015-07-24T07:06:00.000-07:002015-07-24T07:06:32.077-07:00Making a Plumbum Autoload Extension for IPython <div tabindex="-1" id="notebook" class="border-box-sizing">
<div class="container" id="notebook-container">
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>I recently decided to try my hand at making an auto-load extension for Python and plumbum.
I was planning to suggest it as a new feature, then I thought it might be an experimental feature, and now it's just a blog post. But it was an interesting idea and didn't seem to be well documented process on the web. So, here it is.</p>
<p>The plan was to make commands like this:</p>
<div class="highlight"><pre><span class="o">>>></span> <span class="kn">from</span> <span class="nn">plumbum.cmd</span> <span class="kn">import</span> <span class="n">echo</span>
<span class="o">>>></span> <span class="n">echo</span><span class="p">(</span><span class="s">"This is echoed!"</span><span class="p">)</span>
</pre></div>
<p>or</p>
<div class="highlight"><pre><span class="o">>>></span> <span class="kn">from</span> <span class="nn">plumbum</span> <span class="kn">import</span> <span class="n">local</span>
<span class="o">>>></span> <span class="n">echo</span> <span class="o">=</span> <span class="n">local</span><span class="p">[</span><span class="s">'echo'</span><span class="p">]</span>
<span class="o">>>></span> <span class="n">echo</span><span class="p">(</span><span class="s">"This is echoed!"</span><span class="p">)</span>
</pre></div>
<p>into this:</p>
<div class="highlight"><pre><span class="o">>>></span> <span class="n">echo</span><span class="p">(</span><span class="s">"This is echoed!"</span><span class="p">)</span>
</pre></div>
<p>Thereby making Plumbum even more like Bash for fast scripting.</p>
<a name='more'></a>
<h1 id="First-attempt:-Magic">First attempt: Magic<a class="anchor-link" href="#First-attempt:-Magic">¶</a></h1><p>I originally thought it would be a magic:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [41]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="o">%%</span><span class="k">writefile</span> local_magic.py
try:
from IPython.core.magic import (Magics, magics_class,
cell_magic, needs_local_scope)
except ImportError:
print("IPython required for the IPython extension to be loaded.")
raise
import ast
from plumbum import local, CommandNotFound
try:
import builtins
except ImportError: # Python 2
import __builtins__ as builtins
@magics_class
class AutoMagics(Magics):
@needs_local_scope
@cell_magic
def autocmd(self, line, cell, local_ns=None):
mod = ast.parse(cell)
calls = [c for c in ast.walk(mod) if isinstance(c,ast.Call) or isinstance(c, ast.Subscript)]
for call in calls:
name = call.func.id if isinstance(call, ast.Call) else call.value.id
if name not in self.shell.user_ns and name not in dir(builtins):
try:
self.shell.user_ns[name] = local[name]
except CommandNotFound:
pass
exec(cell, self.shell.user_ns, local_ns)
def load_ipython_extension(ipython):
ipython.register_magics(AutoMagics)
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>Writing local_magic.py
</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Here, I grab the contents of a cell, which is still a string, and run it through the python AST. I look at the nodes produced, and for every Call or Subscript node, I check and see if it's in the current namespace. If it's not, I make one attepmt to load it from <code>plumbum.local</code>, and pass if I can't. Then, I exec the cell as normal.</p>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Now, let's see if it works:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [1]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="o">%</span><span class="k">load_ext</span> local_magic
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [2]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="nb">print</span><span class="p">(</span><span class="n">pwd</span><span class="p">())</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_text output_error">
<pre>
<span class="ansired">---------------------------------------------------------------------------</span>
<span class="ansired">NameError</span> Traceback (most recent call last)
<span class="ansigreen"><ipython-input-2-43fe98b08815></span> in <span class="ansicyan"><module></span><span class="ansiblue">()</span>
<span class="ansigreen">----> 1</span><span class="ansiyellow"> </span>print<span class="ansiyellow">(</span>pwd<span class="ansiyellow">(</span><span class="ansiyellow">)</span><span class="ansiyellow">)</span><span class="ansiyellow"></span>
<span class="ansired">NameError</span>: name 'pwd' is not defined</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [3]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="o">%%</span><span class="k">autocmd</span>
print(pwd())
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>/home/henryiii/Dropbox/Personal/notebooks
</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [4]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">pwd</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt output_prompt">Out[4]:</div>
<div class="output_text output_subarea output_execute_result">
<pre>LocalCommand(<LocalPath /bin/pwd>)</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Note that the tracebacks are pretty ugly, as is normal for IPython magics:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [5]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="o">%%</span><span class="k">autocmd</span>
print(ThisIsNotAProgram())
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_text output_error">
<pre>
<span class="ansired">---------------------------------------------------------------------------</span>
<span class="ansired">NameError</span> Traceback (most recent call last)
<span class="ansigreen"><ipython-input-5-b2c7811ed05a></span> in <span class="ansicyan"><module></span><span class="ansiblue">()</span>
<span class="ansigreen">----> 1</span><span class="ansiyellow"> </span>get_ipython<span class="ansiyellow">(</span><span class="ansiyellow">)</span><span class="ansiyellow">.</span>run_cell_magic<span class="ansiyellow">(</span><span class="ansiblue">'autocmd'</span><span class="ansiyellow">,</span> <span class="ansiblue">''</span><span class="ansiyellow">,</span> <span class="ansiblue">'print(ThisIsNotAProgram())'</span><span class="ansiyellow">)</span><span class="ansiyellow"></span>
<span class="ansigreen">/usr/local/lib/python3.4/dist-packages/IPython/core/interactiveshell.py</span> in <span class="ansicyan">run_cell_magic</span><span class="ansiblue">(self, magic_name, line, cell)</span>
<span class="ansigreen"> 2262</span> magic_arg_s <span class="ansiyellow">=</span> self<span class="ansiyellow">.</span>var_expand<span class="ansiyellow">(</span>line<span class="ansiyellow">,</span> stack_depth<span class="ansiyellow">)</span><span class="ansiyellow"></span>
<span class="ansigreen"> 2263</span> <span class="ansigreen">with</span> self<span class="ansiyellow">.</span>builtin_trap<span class="ansiyellow">:</span><span class="ansiyellow"></span>
<span class="ansigreen">-> 2264</span><span class="ansiyellow"> </span>result <span class="ansiyellow">=</span> fn<span class="ansiyellow">(</span>magic_arg_s<span class="ansiyellow">,</span> cell<span class="ansiyellow">)</span><span class="ansiyellow"></span>
<span class="ansigreen"> 2265</span> <span class="ansigreen">return</span> result<span class="ansiyellow"></span>
<span class="ansigreen"> 2266</span> <span class="ansiyellow"></span>
<span class="ansigreen">/home/henryiii/Dropbox/Personal/notebooks/local_magic.py</span> in <span class="ansicyan">autocmd</span><span class="ansiblue">(self, line, cell, local_ns)</span>
<span class="ansigreen">/usr/local/lib/python3.4/dist-packages/IPython/core/magic.py</span> in <span class="ansicyan"><lambda></span><span class="ansiblue">(f, *a, **k)</span>
<span class="ansigreen"> 191</span> <span class="ansired"># but it's overkill for just that one bit of state.</span><span class="ansiyellow"></span><span class="ansiyellow"></span>
<span class="ansigreen"> 192</span> <span class="ansigreen">def</span> magic_deco<span class="ansiyellow">(</span>arg<span class="ansiyellow">)</span><span class="ansiyellow">:</span><span class="ansiyellow"></span>
<span class="ansigreen">--> 193</span><span class="ansiyellow"> </span>call <span class="ansiyellow">=</span> <span class="ansigreen">lambda</span> f<span class="ansiyellow">,</span> <span class="ansiyellow">*</span>a<span class="ansiyellow">,</span> <span class="ansiyellow">**</span>k<span class="ansiyellow">:</span> f<span class="ansiyellow">(</span><span class="ansiyellow">*</span>a<span class="ansiyellow">,</span> <span class="ansiyellow">**</span>k<span class="ansiyellow">)</span><span class="ansiyellow"></span>
<span class="ansigreen"> 194</span> <span class="ansiyellow"></span>
<span class="ansigreen"> 195</span> <span class="ansigreen">if</span> callable<span class="ansiyellow">(</span>arg<span class="ansiyellow">)</span><span class="ansiyellow">:</span><span class="ansiyellow"></span>
<span class="ansigreen">/home/henryiii/Dropbox/Personal/notebooks/local_magic.py</span> in <span class="ansicyan">autocmd</span><span class="ansiblue">(self, line, cell, local_ns)</span>
<span class="ansigreen"> 28</span> <span class="ansigreen">except</span> CommandNotFound<span class="ansiyellow">:</span><span class="ansiyellow"></span>
<span class="ansigreen"> 29</span> <span class="ansigreen">pass</span><span class="ansiyellow"></span>
<span class="ansigreen">---> 30</span><span class="ansiyellow"> </span>exec<span class="ansiyellow">(</span>cell<span class="ansiyellow">,</span> self<span class="ansiyellow">.</span>shell<span class="ansiyellow">.</span>user_ns<span class="ansiyellow">,</span> local_ns<span class="ansiyellow">)</span><span class="ansiyellow"></span>
<span class="ansigreen"> 31</span> <span class="ansiyellow"></span>
<span class="ansigreen"> 32</span> <span class="ansigreen">def</span> load_ipython_extension<span class="ansiyellow">(</span>ipython<span class="ansiyellow">)</span><span class="ansiyellow">:</span><span class="ansiyellow"></span>
<span class="ansigreen"><string></span> in <span class="ansicyan"><module></span><span class="ansiblue">()</span>
<span class="ansired">NameError</span>: name 'ThisIsNotAProgram' is not defined</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h1 id="Better-Than-Magic">Better Than Magic<a class="anchor-link" href="#Better-Than-Magic">¶</a></h1><p>We didn't really gain much, though. We still have to type the extra line, we just don't have to repeat the name of the function we are importing.</p>
<p>How about removing the silly requirement that we stick a magic in front of every command? Easy, we can use a <a href="https://ipython.org/ipython-doc/stable/config/inputtransforms.html">hook</a> in IPython to get the AST transform:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [8]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="o">%%</span><span class="k">writefile</span> local_ext.py
import IPython, ast, builtins
from plumbum import local, CommandNotFound, FG, BG, TF, RETCODE
class NameLoader(ast.NodeTransformer):
"""Direct calls to functions not defined are imported from local if possible."""
def visit_Call(self, node):
ipython = IPython.get_ipython()
if isinstance(node.func, ast.Name): # Will ignore names with .
name = node.func.id
if name not in ipython.user_ns and name not in dir(builtins):
try:
ipython.user_ns[name] = local[name]
except CommandNotFound:
pass
self.generic_visit(node) # The rest of the nodes should be visited
return node
def visit_Subscript(self, node):
"""Bracket syntax too."""
ipython = IPython.get_ipython()
if isinstance(node.value, ast.Name): # Will ignore names with .
name = node.value.id
if name not in ipython.user_ns and name not in dir(builtins):
try:
ipython.user_ns[name] = local[name]
except CommandNotFound:
pass
self.generic_visit(node) # The rest of the nodes should be visited
return node
ipython = IPython.get_ipython()
nl = NameLoader()
def load_ipython_extension(ipython):
ipython.ast_transformers.append(nl)
ipython.user_ns['local'] = local
ipython.user_ns['BG'] = BG
ipython.user_ns['FG'] = FG
ipython.user_ns['TF'] = TF
ipython.user_ns['RETCODE'] = RETCODE
def unload_ipython_extension(ipython):
ipython.ast_transformers.remove(nl)
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>Overwriting local_ext.py
</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Here, ast_transformers will allow us to run on the AST tree after IPython has converted IPython syntax out (which is nicer than before, as IPython code still works), and it a bit cleaner. All tracebacks should be normal now. We also go ahead an populate the namespace of IPython, so local and some modifiers are available.</p>
<p>We load it, causing all commands that are not found to be loaded from local if possible:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [6]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="o">%</span><span class="k">load_ext</span> local_ext
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [7]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">echo</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt output_prompt">Out[7]:</div>
<div class="output_text output_subarea output_execute_result">
<pre>'2\n'</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [8]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="nb">print</span><span class="p">(</span><span class="n">pandoc</span><span class="p">[</span><span class="s">'-h'</span><span class="p">]())</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>pandoc [OPTIONS] [FILES]
Input formats: docbook, haddock, html, json, latex, markdown, markdown_github,
markdown_mmd, markdown_phpextra, markdown_strict, mediawiki,
native, opml, org, rst, textile
Output formats: asciidoc, beamer, context, docbook, docx, dzslides, epub, epub3,
fb2, html, html5, icml, json, latex, man, markdown,
markdown_github, markdown_mmd, markdown_phpextra,
markdown_strict, mediawiki, native, odt, opendocument, opml,
org, pdf*, plain, revealjs, rst, rtf, s5, slideous, slidy,
texinfo, textile
[*for pdf output, use latex or beamer and -o FILENAME.pdf]
Options:
-f FORMAT, -r FORMAT --from=FORMAT, --read=FORMAT
-t FORMAT, -w FORMAT --to=FORMAT, --write=FORMAT
-o FILENAME --output=FILENAME
--data-dir=DIRECTORY
--strict
-R --parse-raw
-S --smart
--old-dashes
--base-header-level=NUMBER
--indented-code-classes=STRING
-F PROGRAM --filter=PROGRAM
--normalize
-p --preserve-tabs
--tab-stop=NUMBER
-s --standalone
--template=FILENAME
-M KEY[:VALUE] --metadata=KEY[:VALUE]
-V KEY[:VALUE] --variable=KEY[:VALUE]
-D FORMAT --print-default-template=FORMAT
--print-default-data-file=FILE
--no-wrap
--columns=NUMBER
--toc, --table-of-contents
--toc-depth=NUMBER
--no-highlight
--highlight-style=STYLE
-H FILENAME --include-in-header=FILENAME
-B FILENAME --include-before-body=FILENAME
-A FILENAME --include-after-body=FILENAME
--self-contained
--offline
-5 --html5
--html-q-tags
--ascii
--reference-links
--atx-headers
--chapters
-N --number-sections
--number-offset=NUMBERS
--no-tex-ligatures
--listings
-i --incremental
--slide-level=NUMBER
--section-divs
--default-image-extension=extension
--email-obfuscation=none|javascript|references
--id-prefix=STRING
-T STRING --title-prefix=STRING
-c URL --css=URL
--reference-odt=FILENAME
--reference-docx=FILENAME
--epub-stylesheet=FILENAME
--epub-cover-image=FILENAME
--epub-metadata=FILENAME
--epub-embed-font=FILE
--epub-chapter-level=NUMBER
--latex-engine=PROGRAM
--bibliography=FILE
--csl=FILE
--citation-abbreviations=FILE
--natbib
--biblatex
-m[URL] --latexmathml[=URL], --asciimathml[=URL]
--mathml[=URL]
--mimetex[=URL]
--webtex[=URL]
--jsmath[=URL]
--mathjax[=URL]
--gladtex
--trace
--dump-args
--ignore-args
-v --version
-h --help
</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [9]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="nb">print</span><span class="p">(</span><span class="n">DontHaveThisProgramEither</span><span class="p">[</span><span class="s">'-h'</span><span class="p">]())</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_text output_error">
<pre>
<span class="ansired">---------------------------------------------------------------------------</span>
<span class="ansired">NameError</span> Traceback (most recent call last)
<span class="ansigreen"><ipython-input-9-0b276825516c></span> in <span class="ansicyan"><module></span><span class="ansiblue">()</span>
<span class="ansigreen">----> 1</span><span class="ansiyellow"> </span>print<span class="ansiyellow">(</span>DontHaveThisProgramEither<span class="ansiyellow">[</span><span class="ansiblue">'-h'</span><span class="ansiyellow">]</span><span class="ansiyellow">(</span><span class="ansiyellow">)</span><span class="ansiyellow">)</span><span class="ansiyellow"></span>
<span class="ansired">NameError</span>: name 'DontHaveThisProgramEither' is not defined</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>This allows us to write code very quickly in a shell like environment by simply putting:</p>
<div class="highlight"><pre><span class="c">#!/usr/bin/env ipython</span>
<span class="o">%</span><span class="n">load_ext</span> <span class="n">local_ext</span>
</pre></div>
<p>At the top of our scripts. (Assuming this is in the IPython path) Once the script is done, you can go import the programs used manually and then remove the IPython parts.</p>
</div>
</div>
</div>
</div>
</div>Anonymoushttp://www.blogger.com/profile/10752018806658508906noreply@blogger.com0tag:blogger.com,1999:blog-8303893118597899654.post-2255670065373230302015-07-24T06:45:00.000-07:002015-07-24T07:07:37.027-07:00Uncertainty Extension for IPython <div tabindex="-1" id="notebook" class="border-box-sizing">
<div class="container" id="notebook-container">
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Wouldn't it be nice if we had uncertainty with a nice notation in IPython? The current method would be to use raw Python,</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [1]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="kn">from</span> <span class="nn">uncertainties</span> <span class="k">import</span> <span class="n">ufloat</span>
<span class="nb">print</span><span class="p">(</span><span class="n">ufloat</span><span class="p">(</span><span class="mf">12.34</span><span class="p">,</span><span class="o">.</span><span class="mi">01</span><span class="p">))</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>12.340+/-0.010
</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Let's use the handy infix library to make the notation easier. We'll define <code>|pm|</code> to mean <code>+/-</code>.</p>
<blockquote><p>Note: this is a very simple library that is less than a page long.
Feel free to write the code yourself (as I do later in this notebook).</p>
</blockquote>
</div>
</div>
</div>
<a name='more'></a>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [19]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="kn">from</span> <span class="nn">infix</span> <span class="k">import</span> <span class="n">or_infix</span><span class="p">,</span> <span class="n">custom_infix</span><span class="p">,</span> <span class="n">base_infix</span>
<span class="n">pm</span> <span class="o">=</span> <span class="n">or_infix</span><span class="p">(</span><span class="n">ufloat</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [3]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="nb">print</span><span class="p">(</span><span class="mf">12.34</span> <span class="o">|</span><span class="n">pm</span><span class="o">|</span> <span class="o">.</span><span class="mi">01</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>12.340+/-0.010
</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Aside">Aside<a class="anchor-link" href="#Aside">¶</a></h2><p>Our <code>|</code> operator has very low precedence. So, things like this are not very useful:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [6]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="mi">2</span> <span class="o">*</span> <span class="mf">12.34</span> <span class="o">|</span><span class="n">pm</span><span class="o">|</span> <span class="o">.</span><span class="mi">01</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt output_prompt">Out[6]:</div>
<div class="output_text output_subarea output_execute_result">
<pre>24.68+/-0.01</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>The highest order we can do with the infix library is mul,</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [12]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">pm</span> <span class="o">=</span> <span class="n">custom_infix</span><span class="p">(</span><span class="s">'__rmul__'</span><span class="p">,</span><span class="s">'__mul__'</span><span class="p">)(</span><span class="n">ufloat</span><span class="p">)</span>
<span class="mi">2</span> <span class="o">*</span> <span class="mf">12.34</span> <span class="o">*</span><span class="n">pm</span><span class="o">*</span> <span class="o">.</span><span class="mi">01</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt output_prompt">Out[12]:</div>
<div class="output_text output_subarea output_execute_result">
<pre>24.68+/-0.01</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>The library doesn't have support for python's only right-assosiative operator, <code>**</code>. We can add that easily, though:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [23]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="k">class</span> <span class="nc">pow_infix</span><span class="p">(</span><span class="n">base_infix</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">right</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">right</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">func</span><span class="p">(</span><span class="n">right</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">__infix__</span><span class="p">)</span>
<span class="n">__pow__</span> <span class="o">=</span> <span class="n">base_infix</span><span class="o">.</span><span class="n">left</span>
<span class="n">__rpow__</span> <span class="o">=</span> <span class="n">right</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [24]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">pm</span> <span class="o">=</span> <span class="n">pow_infix</span><span class="p">(</span><span class="n">ufloat</span><span class="p">)</span>
<span class="mi">2</span> <span class="o">*</span> <span class="mf">12.34</span> <span class="o">**</span><span class="n">pm</span><span class="o">**</span> <span class="o">.</span><span class="mi">01</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt output_prompt">Out[24]:</div>
<div class="output_text output_subarea output_execute_result">
<pre>24.68+/-0.02</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h1 id="Extending-IPython">Extending IPython<a class="anchor-link" href="#Extending-IPython">¶</a></h1>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>We've made an improvement, but this is ugly notation. Let's patch IPython to find the notation <code>+/-</code> or <code>±</code> and replace it with our infix operator before passing it to the underlying python kernel:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [34]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="kn">from</span> <span class="nn">IPython.core.inputtransformer</span> <span class="k">import</span> <span class="n">StatelessInputTransformer</span>
<span class="nd">@StatelessInputTransformer</span><span class="o">.</span><span class="n">wrap</span>
<span class="k">def</span> <span class="nf">my_special_commands</span><span class="p">(</span><span class="n">line</span><span class="p">):</span>
<span class="k">return</span> <span class="n">line</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s">'+/-'</span><span class="p">,</span><span class="s">'**pm**'</span><span class="p">)</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s">'±'</span><span class="p">,</span><span class="s">'**pm**'</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>We only needed a line transformer, and there was no need to even handle storing a state. Thankfully, we didn't need something like the AST this time, since it's a simple replacement only; that's the beauty of starting with a valid python construct for infix operations.</p>
<p>Now, let's use our transformer in the current IPython instance:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [35]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="kn">import</span> <span class="nn">IPython</span>
<span class="n">ip</span> <span class="o">=</span> <span class="n">IPython</span><span class="o">.</span><span class="n">get_ipython</span><span class="p">()</span>
<span class="n">ip</span><span class="o">.</span><span class="n">input_splitter</span><span class="o">.</span><span class="n">logical_line_transforms</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">my_special_commands</span><span class="p">())</span>
<span class="n">ip</span><span class="o">.</span><span class="n">input_transformer_manager</span><span class="o">.</span><span class="n">logical_line_transforms</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">my_special_commands</span><span class="p">())</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [36]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="nb">print</span><span class="p">(</span><span class="mi">112</span> <span class="o">+/-</span> <span class="mi">2</span> <span class="o">-</span> <span class="mi">1321</span> <span class="o">+/-</span> <span class="mi">2</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>-1209.0+/-2.8
</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [37]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="nb">print</span><span class="p">(</span><span class="mi">112</span> <span class="err">±</span> <span class="mi">2</span> <span class="o">-</span> <span class="mi">1321</span> <span class="err">±</span> <span class="mi">2</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>-1209.0+/-2.8
</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h1 id="Loadable-extension">Loadable extension<a class="anchor-link" href="#Loadable-extension">¶</a></h1>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Let's make a IPython extension that makes all of this automatic. I'm removing the need for the infix library (since it's only a few lines).</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [16]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="o">%%</span><span class="k">writefile</span> uncert_ipython.py
from uncertainties import ufloat
import IPython
from IPython.core.inputtransformer import StatelessInputTransformer
class pow_infix(object):
def __init__(self, func):
self.func = func
def __pow__(self, left):
self.__infix__ = left
return self
def __rpow__(self, right):
return self.func(right, self.__infix__)
pm = pow_infix(ufloat)
@StatelessInputTransformer.wrap
def my_special_commands(line):
return line.replace('+/-','**pm**').replace('±','**pm**')
comm1 = my_special_commands()
comm2 = my_special_commands()
def load_ipython_extension(ipython):
ipython.input_splitter.logical_line_transforms.append(comm1)
ipython.input_transformer_manager.logical_line_transforms.append(comm2)
ipython.user_ns['pm'] = pm
def unload_ipython_extension(ipython):
ipython.input_splitter.logical_line_transforms.remove(comm1)
ipython.input_transformer_manager.logical_line_transforms.remove(comm2)
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>Overwriting uncert_ipython.py
</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>After restarting the kernel, we can test this:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [17]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="o">%</span><span class="k">load_ext</span> uncert_ipython
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [23]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="mi">1</span> <span class="o">+/-</span> <span class="o">.</span><span class="mi">1</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt output_prompt">Out[23]:</div>
<div class="output_text output_subarea output_execute_result">
<pre>1.0+/-0.1</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [24]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="o">%</span><span class="k">unload_ext</span> uncert_ipython
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [25]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="mi">1</span> <span class="o">+/-</span> <span class="o">.</span><span class="mi">1</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_text output_error">
<pre>
<span class="ansicyan"> File </span><span class="ansigreen">"<ipython-input-25-5d2725f7d570>"</span><span class="ansicyan">, line </span><span class="ansigreen">1</span>
<span class="ansiyellow"> 1 +/- .1</span>
<span class="ansigrey"> ^</span>
<span class="ansired">SyntaxError</span><span class="ansired">:</span> invalid syntax
</pre>
</div>
</div>
</div>
</div>
</div>
</div>
</div>Anonymoushttp://www.blogger.com/profile/10752018806658508906noreply@blogger.com0tag:blogger.com,1999:blog-8303893118597899654.post-11039777887896766582015-07-22T12:25:00.001-07:002015-10-29T18:40:04.095-07:00Plumbum Color <div tabindex="-1" id="notebook" class="border-box-sizing">
<div class="container" id="notebook-container">
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>I've been working on a color addition to Plumbum for a little while, and I'd like to share the basics of using it with you now. This library was originally built around a special <code>str</code> subclass, but now is built on the new <code>Styles</code> representation and is far more powerful than the first implementation. It safely does nothing if you do not have a color-compatible systems (posix + tty currently), but can be forced if need be. It is included with Plumbum, so you don't have to add a requirement for your scripts that is non-essential (as color often is). It is integrated with <code>plumbum.cli</code>, too. Also, I've managed to accelerate the color selection algorithims about 8x, allowing near game-like speeds. (see the <code>fullcolor.py</code> example).</p>
</div>
</div>
</div><a name='more'></a>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<blockquote><p>Note: The <code>plumbum.colors</code> library was written for the terminal and ANSI escape sequences. However, the following article was written in the IPython notebook, so it cannot show ANSI escapes. Due to the fact that the <code>plumbum.colors</code> library uses a flexible <code>Styles</code> based representation, HTML is easy to implement as a Styles subclass and is available as htmlcolors (see docs for an example). With the <code>plumbum.colorlib</code> IPython extension, IPython loads the <code>%%to html</code> magic and makes <code>plumbum.colorlib.htmlcolors</code> available as <code>colors</code>. <code>htmlcolors</code> does not support <code>colors.reset</code> (and therefore using <code>colors</code> directly as a context manager, as well), but otherwise it is very similar. If we are constructing strings, we'll need to remember to include HTML line breaks, so we'll redefine print too.</p>
</blockquote>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [2]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="o">%</span><span class="k">load_ext</span> plumbum.colorlib
<span class="kn">from</span> <span class="nn">functools</span> <span class="k">import</span> <span class="n">partial</span>
<span class="nb">print</span> <span class="o">=</span> <span class="n">partial</span><span class="p">(</span><span class="nb">print</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="s">'<br/></span><span class="se">\n</span><span class="s">'</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h1 id="Plumbum.colors">Plumbum.colors<a class="anchor-link" href="#Plumbum.colors">¶</a></h1>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Safe-color-manipulations-made-easy">Safe color manipulations made easy<a class="anchor-link" href="#Safe-color-manipulations-made-easy">¶</a></h2><p>This is a quick overview and tutorial on the <code>plumbum.colors</code> library that is being proposed for Plumbum. It allows you to do things like this:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [3]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="o">%%</span><span class="k">to</span> html
with colors.blue:
print("This is in Blue.")
print("But this is not.")
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_html rendered_html output_subarea ">
<font color="#0000C0">This is in Blue.<br/>
</font>But this is not.<br/>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>It works through the COLOR object, which gives you access to styles, and the terminal colors through the forground and background objects. You can wrap a string with the <code>|</code> operator:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [4]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="o">%%</span><span class="k">to</span> html
print(colors.red | "This is in red", '... but not this')
print("You can change the background too!" | colors.bg.light_yellow)
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_html rendered_html output_subarea ">
<font color="#C00000">This is in red</font> ... but not this<br/>
<span style="background-color: #FFFF00">You can change the background too!</span><br/>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>The color can go on either side of the string you are wrapping.</p>
<p>Styles are available, too:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [5]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="o">%%</span><span class="k">to</span> html
with colors.green:
print(colors.underline | "This is underlined", "and this is still green!")
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_html rendered_html output_subarea ">
<font color="#00C000"><span style="text-decoration: underline;">This is underlined</span> and this is still green!<br/>
</font>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>You can combine styles, and the result is still a valid style:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [6]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="o">%%</span><span class="k">to</span> html
mix = colors.bold & colors.italics & colors.red & colors.bg.light_green
print(mix | "This is a muddle of styles!")
with (colors.strikeout & colors.red):
print("Twin styles")
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_html rendered_html output_subarea ">
<span style="background-color: #00FF00"><font color="#C00000"><b><i>This is a muddle of styles!</i></b></font></span><br/>
<font color="#C00000"><s>Twin styles<br/>
</s></font>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>All the major ANSI represetations are supported, include Basic (the first 8 colors), Simple (the first 16 colors), Full (256 colors using three parameter color codes), and True (24 bit color, using 5 parameter color codes). You can even find the closest color in a lower representation if you need to.</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [7]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="o">%%</span><span class="k">to</span> html
print(colors.dark_blue | 'This is from the extended color set.')
print(colors['LIGHT_SEA_GREEN'] | 'And another one.')
print(colors.rgb(193,41,210) | 'This supports all colors!')
print(colors["#3AB227"] | 'Hex notation, too.')
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_html rendered_html output_subarea ">
<font color="#000087">This is from the extended color set.</font><br/>
<font color="#00AFAF">And another one.</font><br/>
<font color="#C129D2">This supports all colors!</font><br/>
<font color="#3AB227">Hex notation, too.</font><br/>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>The full list is on the <code>plumbum.colors</code> ReadTheDocs page.</p>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>As a quick shortcut, you use <code>.print</code> directly on color (<code>.print_</code> if you are using the classic print statment in Python 2):</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [10]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="o">%%</span><span class="k">to</span> html
colors.orchid.print("This is in orchid.")
colors.bg.magenta.print("This is on a magenta background.")
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_html rendered_html output_subarea ">
<font color="#D75FD7">This is in orchid.</font><br/>
<span style="background-color: #C000C0">This is on a magenta background.</span><br/>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>The colors can be iterated and sliced:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [12]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="o">%%</span><span class="k">to</span> html
for color in colors.fg[:16]:
print(color | "This is color:", color.fg.name.upper())
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_html rendered_html output_subarea ">
<font color="#000000">This is color:</font> BLACK<br/>
<font color="#C00000">This is color:</font> RED<br/>
<font color="#00C000">This is color:</font> GREEN<br/>
<font color="#C0C000">This is color:</font> YELLOW<br/>
<font color="#0000C0">This is color:</font> BLUE<br/>
<font color="#C000C0">This is color:</font> MAGENTA<br/>
<font color="#00C0C0">This is color:</font> CYAN<br/>
<font color="#C0C0C0">This is color:</font> LIGHT_GRAY<br/>
<font color="#808080">This is color:</font> DARK_GRAY<br/>
<font color="#FF0000">This is color:</font> LIGHT_RED<br/>
<font color="#00FF00">This is color:</font> LIGHT_GREEN<br/>
<font color="#FFFF00">This is color:</font> LIGHT_YELLOW<br/>
<font color="#0000FF">This is color:</font> LIGHT_BLUE<br/>
<font color="#FF00FF">This is color:</font> LIGHT_MAGENTA<br/>
<font color="#00FFFF">This is color:</font> LIGHT_CYAN<br/>
<font color="#FFFFFF">This is color:</font> WHITE<br/>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [14]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="o">%%</span><span class="k">to</span> html
for color in colors:
color.print("&#x25a0;", end=' ')
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_html rendered_html output_subarea ">
<font color="#000000">■</font> <font color="#C00000">■</font> <font color="#00C000">■</font> <font color="#C0C000">■</font> <font color="#0000C0">■</font> <font color="#C000C0">■</font> <font color="#00C0C0">■</font> <font color="#C0C0C0">■</font> <font color="#808080">■</font> <font color="#FF0000">■</font> <font color="#00FF00">■</font> <font color="#FFFF00">■</font> <font color="#0000FF">■</font> <font color="#FF00FF">■</font> <font color="#00FFFF">■</font> <font color="#FFFFFF">■</font> <font color="#000000">■</font> <font color="#00005F">■</font> <font color="#000087">■</font> <font color="#0000AF">■</font> <font color="#0000D7">■</font> <font color="#0000FF">■</font> <font color="#005F00">■</font> <font color="#005F5F">■</font> <font color="#005F87">■</font> <font color="#005FAF">■</font> <font color="#005FD7">■</font> <font color="#005FFF">■</font> <font color="#008700">■</font> <font color="#00875F">■</font> <font color="#008787">■</font> <font color="#0087AF">■</font> <font color="#0087D7">■</font> <font color="#0087FF">■</font> <font color="#00AF00">■</font> <font color="#00AF5F">■</font> <font color="#00AF87">■</font> <font color="#00AFAF">■</font> <font color="#00AFD7">■</font> <font color="#00AFFF">■</font> <font color="#00D700">■</font> <font color="#00D75F">■</font> <font color="#00D787">■</font> <font color="#00D7AF">■</font> <font color="#00D7D7">■</font> <font color="#00D7FF">■</font> <font color="#00FF00">■</font> <font color="#00FF5F">■</font> <font color="#00FF87">■</font> <font color="#00FFAF">■</font> <font color="#00FFD7">■</font> <font color="#00FFFF">■</font> <font color="#5F0000">■</font> <font color="#5F005F">■</font> <font color="#5F0087">■</font> <font color="#5F00AF">■</font> <font color="#5F00D7">■</font> <font color="#5F00FF">■</font> <font color="#5F5F00">■</font> <font color="#5F5F5F">■</font> <font color="#5F5F87">■</font> <font color="#5F5FAF">■</font> <font color="#5F5FD7">■</font> <font color="#5F5FFF">■</font> <font color="#5F8700">■</font> <font color="#5F875F">■</font> <font color="#5F8787">■</font> <font color="#5F87AF">■</font> <font color="#5F87D7">■</font> <font color="#5F87FF">■</font> <font color="#5FAF00">■</font> <font color="#5FAF5F">■</font> <font color="#5FAF87">■</font> <font color="#5FAFAF">■</font> <font color="#5FAFD7">■</font> <font color="#5FAFFF">■</font> <font color="#5FD700">■</font> <font color="#5FD75F">■</font> <font color="#5FD787">■</font> <font color="#5FD7AF">■</font> <font color="#5FD7D7">■</font> <font color="#5FD7FF">■</font> <font color="#5FFF00">■</font> <font color="#5FFF5F">■</font> <font color="#5FFF87">■</font> <font color="#5FFFAF">■</font> <font color="#5FFFD7">■</font> <font color="#5FFFFF">■</font> <font color="#870000">■</font> <font color="#87005F">■</font> <font color="#870087">■</font> <font color="#8700AF">■</font> <font color="#8700D7">■</font> <font color="#8700FF">■</font> <font color="#875F00">■</font> <font color="#875F5F">■</font> <font color="#875F87">■</font> <font color="#875FAF">■</font> <font color="#875FD7">■</font> <font color="#875FFF">■</font> <font color="#878700">■</font> <font color="#87875F">■</font> <font color="#878787">■</font> <font color="#8787AF">■</font> <font color="#8787D7">■</font> <font color="#8787FF">■</font> <font color="#87AF00">■</font> <font color="#87AF5F">■</font> <font color="#87AF87">■</font> <font color="#87AFAF">■</font> <font color="#87AFD7">■</font> <font color="#87AFFF">■</font> <font color="#87D700">■</font> <font color="#87D75F">■</font> <font color="#87D787">■</font> <font color="#87D7AF">■</font> <font color="#87D7D7">■</font> <font color="#87D7FF">■</font> <font color="#87FF00">■</font> <font color="#87FF5F">■</font> <font color="#87FF87">■</font> <font color="#87FFAF">■</font> <font color="#87FFD7">■</font> <font color="#87FFFF">■</font> <font color="#AF0000">■</font> <font color="#AF005F">■</font> <font color="#AF0087">■</font> <font color="#AF00AF">■</font> <font color="#AF00D7">■</font> <font color="#AF00FF">■</font> <font color="#AF5F00">■</font> <font color="#AF5F5F">■</font> <font color="#AF5F87">■</font> <font color="#AF5FAF">■</font> <font color="#AF5FD7">■</font> <font color="#AF5FFF">■</font> <font color="#AF8700">■</font> <font color="#AF875F">■</font> <font color="#AF8787">■</font> <font color="#AF87AF">■</font> <font color="#AF87D7">■</font> <font color="#AF87FF">■</font> <font color="#AFAF00">■</font> <font color="#AFAF5F">■</font> <font color="#AFAF87">■</font> <font color="#AFAFAF">■</font> <font color="#AFAFD7">■</font> <font color="#AFAFFF">■</font> <font color="#AFD700">■</font> <font color="#AFD75F">■</font> <font color="#AFD787">■</font> <font color="#AFD7AF">■</font> <font color="#AFD7D7">■</font> <font color="#AFD7FF">■</font> <font color="#AFFF00">■</font> <font color="#AFFF5F">■</font> <font color="#AFFF87">■</font> <font color="#AFFFAF">■</font> <font color="#AFFFD7">■</font> <font color="#AFFFFF">■</font> <font color="#D70000">■</font> <font color="#D7005F">■</font> <font color="#D70087">■</font> <font color="#D700AF">■</font> <font color="#D700D7">■</font> <font color="#D700FF">■</font> <font color="#D75F00">■</font> <font color="#D75F5F">■</font> <font color="#D75F87">■</font> <font color="#D75FAF">■</font> <font color="#D75FD7">■</font> <font color="#D75FFF">■</font> <font color="#D78700">■</font> <font color="#D7875F">■</font> <font color="#D78787">■</font> <font color="#D787AF">■</font> <font color="#D787D7">■</font> <font color="#D787FF">■</font> <font color="#D7AF00">■</font> <font color="#D7AF5F">■</font> <font color="#D7AF87">■</font> <font color="#D7AFAF">■</font> <font color="#D7AFD7">■</font> <font color="#D7AFFF">■</font> <font color="#D7D700">■</font> <font color="#D7D75F">■</font> <font color="#D7D787">■</font> <font color="#D7D7AF">■</font> <font color="#D7D7D7">■</font> <font color="#D7D7FF">■</font> <font color="#D7FF00">■</font> <font color="#D7FF5F">■</font> <font color="#D7FF87">■</font> <font color="#D7FFAF">■</font> <font color="#D7FFD7">■</font> <font color="#D7FFFF">■</font> <font color="#FF0000">■</font> <font color="#FF005F">■</font> <font color="#FF0087">■</font> <font color="#FF00AF">■</font> <font color="#FF00D7">■</font> <font color="#FF00FF">■</font> <font color="#FF5F00">■</font> <font color="#FF5F5F">■</font> <font color="#FF5F87">■</font> <font color="#FF5FAF">■</font> <font color="#FF5FD7">■</font> <font color="#FF5FFF">■</font> <font color="#FF8700">■</font> <font color="#FF875F">■</font> <font color="#FF8787">■</font> <font color="#FF87AF">■</font> <font color="#FF87D7">■</font> <font color="#FF87FF">■</font> <font color="#FFAF00">■</font> <font color="#FFAF5F">■</font> <font color="#FFAF87">■</font> <font color="#FFAFAF">■</font> <font color="#FFAFD7">■</font> <font color="#FFAFFF">■</font> <font color="#FFD700">■</font> <font color="#FFD75F">■</font> <font color="#FFD787">■</font> <font color="#FFD7AF">■</font> <font color="#FFD7D7">■</font> <font color="#FFD7FF">■</font> <font color="#FFFF00">■</font> <font color="#FFFF5F">■</font> <font color="#FFFF87">■</font> <font color="#FFFFAF">■</font> <font color="#FFFFD7">■</font> <font color="#FFFFFF">■</font> <font color="#080808">■</font> <font color="#121212">■</font> <font color="#1C1C1C">■</font> <font color="#262626">■</font> <font color="#303030">■</font> <font color="#3A3A3A">■</font> <font color="#444444">■</font> <font color="#4E4E4E">■</font> <font color="#585858">■</font> <font color="#626262">■</font> <font color="#6C6C6C">■</font> <font color="#767676">■</font> <font color="#808080">■</font> <font color="#8A8A8A">■</font> <font color="#949494">■</font> <font color="#9E9E9E">■</font> <font color="#A8A8A8">■</font> <font color="#B2B2B2">■</font> <font color="#BCBCBC">■</font> <font color="#C6C6C6">■</font> <font color="#D0D0D0">■</font> <font color="#DADADA">■</font> <font color="#E4E4E4">■</font> <font color="#EEEEEE">■</font>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Finally, you can also use <code>[]</code> notation to wrap a color (less convenient, but similar to other methods):</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [19]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="o">%%</span><span class="k">to</span> html
print(colors.blue['This is wrapped in blue'])
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_html rendered_html output_subarea ">
<font color="#0000C0">This is wrapped in blue</font><br/>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Unsafe-color-manipulations,-too">Unsafe color manipulations, too<a class="anchor-link" href="#Unsafe-color-manipulations,-too">¶</a></h2>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Sometimes, you will find unsafe manipulations faster than wrapping
every string. This can be done with <code>plumbum.colors</code>, too.</p>
<p>If you are planning unsafe manipulations, you can wrap your code in a context manager that restores color to your terminal. For example,</p>
<div class="highlight"><pre><span class="k">with</span> <span class="n">colors</span><span class="p">:</span>
<span class="o">...</span><span class="n">unsafe</span> <span class="n">color</span> <span class="n">operations</span><span class="o">...</span>
</pre></div>
<p>All styles will be restored on leaving the manager. The color will automatically be reset when Python quits, as well, even on an exception, so this is not necessary, but is useful in local code.</p>
<p>Also, you can restore or set color instantly using the emergency restore from a terminal:</p>
<div class="highlight"><pre><span class="nv">$ </span>python -m plumbum.colorlib
</pre></div>
<p>This takes any string that you can use in colors, and without a string, it restores color.</p>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>The string representation of a color is the ANSI sequence that would produce the color. If you want to instantly write a color to the terminal, you can call the color without arguments. So, either of the following would change the color to blue without restoring it:</p>
<div class="highlight"><pre><span class="k">print</span><span class="p">(</span><span class="n">colors</span><span class="o">.</span><span class="n">blue</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="s">''</span><span class="p">)</span>
<span class="n">colors</span><span class="o">.</span><span class="n">blue</span><span class="p">()</span>
</pre></div>
<p>To get the reset color, you can either use the <code>.reset</code> property on a factory or a style, or you can use <code>~</code> (inversion). So, this would be a manual color safe wrapping from unsafe components:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [21]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="o">%%</span><span class="k">to</span> html
print("Before " + colors.red + "Middle" + ~colors.red + " After")
print("Before " + colors.blue + "Middle" + ~colors.fg + " After")
print("Before " + colors.green + "Middle" + colors.fg.reset + " After")
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_html rendered_html output_subarea ">
Before <font color="#C00000">Middle</font> After<br/>
Before <font color="#0000C0">Middle</font> After<br/>
Before <font color="#00C000">Middle</font> After<br/>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Details-of-the-Style-Factories:">Details of the Style Factories:<a class="anchor-link" href="#Details-of-the-Style-Factories:">¶</a></h2><p>Let's look at the contents of a <code>colors.fg</code> or <code>colors.bg</code> object:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [22]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="p">{</span><span class="n">x</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="nb">dir</span><span class="p">(</span><span class="n">colors</span><span class="o">.</span><span class="n">bg</span><span class="p">)</span> <span class="k">if</span> <span class="n">x</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">!=</span> <span class="s">'_'</span><span class="p">}</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt output_prompt">Out[22]:</div>
<div class="output_text output_subarea output_execute_result">
<pre>{'ansi',
'black',
'blue',
'cyan',
'dark_gray',
'full',
'green',
'hex',
'light_blue',
'light_cyan',
'light_gray',
'light_green',
'light_magenta',
'light_red',
'light_yellow',
'magenta',
'red',
'reset',
'rgb',
'simple',
'white',
'yellow'}</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Notice that the extended colors are not listed, to make completion easier. Also, color access is not case sensitive and ignores underscores.</p>
<p>Since the <code>colors</code> object looks like a <code>fg</code> object, let's only look at the unique contents (<code>.reset</code> has a different meaning for <code>colors</code>, as it resets the terminal completly instead of just the foreground color, so let's remove it from the <code>fg</code> set):</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [23]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">fg</span> <span class="o">=</span> <span class="p">{</span><span class="n">x</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="nb">dir</span><span class="p">(</span><span class="n">colors</span><span class="o">.</span><span class="n">bg</span><span class="p">)</span> <span class="k">if</span> <span class="n">x</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">!=</span> <span class="s">'_'</span> <span class="ow">and</span> <span class="n">x</span><span class="o">!=</span><span class="s">'reset'</span><span class="p">}</span>
<span class="n">col</span> <span class="o">=</span> <span class="p">{</span><span class="n">x</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="nb">dir</span><span class="p">(</span><span class="n">colors</span><span class="p">)</span> <span class="k">if</span> <span class="n">x</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">!=</span> <span class="s">'_'</span><span class="p">}</span>
<span class="n">col</span> <span class="o">-</span> <span class="n">fg</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt output_prompt">Out[23]:</div>
<div class="output_text output_subarea output_execute_result">
<pre>{'bg',
'bold',
'code',
'contains_colors',
'do_nothing',
'em',
'extract',
'fatal',
'fg',
'filter',
'from_ansi',
'get_colors_from_string',
'highlight',
'info',
'italics',
'li',
'load_stylesheet',
'ol',
'reset',
'stdout',
'strikeout',
'success',
'title',
'underline',
'use_color',
'warn'}</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Note that the properties are generated based on the attributes allowed for a style, so HTML has some slight differences here vs. ANSI.</p>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Stylesheets">Stylesheets<a class="anchor-link" href="#Stylesheets">¶</a></h2><p>A recent addition to colors is stylesheets. Stylesheets allow you to use and create styles based on usage. The default sheet is the following:</p>
<div class="highlight"><pre><span class="n">default_styles</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">(</span>
<span class="n">warn</span><span class="o">=</span><span class="s">"fg red"</span><span class="p">,</span>
<span class="n">title</span><span class="o">=</span><span class="s">"fg cyan underline bold"</span><span class="p">,</span>
<span class="n">fatal</span><span class="o">=</span><span class="s">"fg red bold"</span><span class="p">,</span>
<span class="n">highlight</span><span class="o">=</span><span class="s">"bg yellow"</span><span class="p">,</span>
<span class="n">info</span><span class="o">=</span><span class="s">"fg blue"</span><span class="p">,</span>
<span class="n">success</span><span class="o">=</span><span class="s">"fg green"</span><span class="p">,</span>
<span class="p">)</span>
</pre></div>
<p>You can load a new sheet or a changed sheet with <code>colors.load_stylesheet(default_styles)</code>. The new and changed styles will be accessable just like any normal color.</p>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h1 id="Bonus">Bonus<a class="anchor-link" href="#Bonus">¶</a></h1><p>The cell magic we've been using is actually a slight upgrade on the following example:</p>
<p>We are going to make a quick cell magic for IPython to capture output and render html from it. It's really not hard to make a cell magic in IPython:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [29]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="kn">from</span> <span class="nn">io</span> <span class="k">import</span> <span class="n">StringIO</span>
<span class="kn">from</span> <span class="nn">IPython.display</span> <span class="k">import</span> <span class="n">display_html</span>
<span class="kn">from</span> <span class="nn">contextlib</span> <span class="k">import</span> <span class="n">redirect_stdout</span>
<span class="kn">from</span> <span class="nn">IPython.core.magic</span> <span class="k">import</span> <span class="n">register_cell_magic</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [28]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="nd">@register_cell_magic</span>
<span class="k">def</span> <span class="nf">output_html</span><span class="p">(</span><span class="n">line</span><span class="p">,</span> <span class="n">cell</span><span class="p">):</span>
<span class="s">"Captures stdout and renders it in the notebook as html."</span>
<span class="n">out</span> <span class="o">=</span> <span class="n">StringIO</span><span class="p">()</span>
<span class="k">with</span> <span class="n">redirect_stdout</span><span class="p">(</span><span class="n">out</span><span class="p">):</span>
<span class="n">exec</span><span class="p">(</span><span class="n">cell</span><span class="p">)</span>
<span class="n">out</span><span class="o">.</span><span class="n">seek</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
<span class="n">display_html</span><span class="p">(</span><span class="n">out</span><span class="o">.</span><span class="n">getvalue</span><span class="p">(),</span> <span class="n">raw</span><span class="o">=</span><span class="k">True</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Let's test this to make sure it works:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [30]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="o">%%</span><span class="k">output_html</span>
print("<p>Wow!</p>")
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_html rendered_html output_subarea ">
<p>Wow!</p><br/>
</div>
</div>
</div>
</div>
</div>
</div>
</div>Anonymoushttp://www.blogger.com/profile/10752018806658508906noreply@blogger.com0tag:blogger.com,1999:blog-8303893118597899654.post-76792617358025490702015-07-12T17:05:00.000-07:002015-07-18T19:44:11.681-07:00University of Texas Doctoral Thesis TemplateI have created a thesis class file for a UT Thesis in LaTeX. It has already been used for at least one passing thesis, so it does meet the current UT guidelines. (Please let me know if there are any issues!)<br />
<br />
Since I use Bitbucket for all my private repositories (like my thesis itself), the code is in a Bitbucket repository rather than GitHub. <a href="https://bitbucket.org/henryiii/texutthesis">Here is the link</a> if you want to create a pull request or want to compile the class and documentation from the source .dtx file.<br />
<br />
If all you want is a download of a working version, and if you don't want to compile the code but just want the class file, <a href="https://bitbucket.org/henryiii/texutthesis/downloads">here are downloadable packages including the class file</a>.<br />
<br />
<a name='more'></a>
<h3>
Original email</h3>
This was originally released in the following email to the UT Gradlist:<br />
<br />
<span style="color: #134f5c;">Due to the pitiful lack of vaguely resent or usable resources for a UT thesis in LaTeX, I have created a thesis template in LaTeX to match the UT guidelines. The repository for the source can be found on BitBucket here, and a downloadable package is here, as texutthesis.zip. Please let me know if you have improvements.</span><br />
<span style="color: #134f5c;"><br /></span>
<span style="color: #134f5c;">To use the class in the download, put the utthesis.cls file in your folder with your thesis (recommended), or put it in a LaTeX searchable location. If you have the source, run "pdflatex utthesis.dtx" to generate the .cls file (and a helpful pdf guide). I've included a sample thesis too.</span><br />
<span style="color: #134f5c;"><br /></span>
<span style="color: #134f5c;">Most of the time, this works as expected, modifying and theming the standard commands. It does add a few custom commands, like UTabstract, when something special is needed. I followed the UT guideline as closely as possible. Internally, it uses Memoir to provide precise formatting control.</span><br />
<span style="color: #134f5c;"><br /></span>
<span style="color: #134f5c;">If you don't have time to look through the sample, here is an abbreviated sample to whet your appetite: </span><br />
<code><span style="color: #134f5c;">
<span style="font-family: Times New Roman;"><br /></span>
\documentclass{utthesis}<br />
<br />
\UTyear=2015<br />
<br />
\begin{document}<br />
<br />
\author{John Fredrick Doe}<br />
\title{Very Important Title For a Very Important Thesis}<br />
\date{Revised: \today}<br />
<br />
\UTcopyrightlegend<br />
<br />
\begin{UTcommittee}<br />
\UTaddsupervisor{George Washington}<br />
\UTaddcosupervisor{Chuck Norris}<br />
\UTaddcommittee{John Adams}<br />
\UTaddcommittee{Thomas Jefferson}<br />
\end{UTcommittee}<br />
<br />
\UTtitlepage{B.S.}{May}<br />
<br />
\frontmatter<br />
<br />
\begin{UTabstract}[s]{George Washington, Chuck Norris}<br />
I did some cool research.<br />
\end{UTabstract}<br />
<br />
...<br />
<br />
\end{document}</span></code><br />
<span style="color: #134f5c;"><code><br />
</code>
Hopefully some of you will find this useful.</span><br />
<span style="color: #134f5c;"><br /></span>
<span style="color: #134f5c;">Henry Schreiner III</span>Anonymoushttp://www.blogger.com/profile/10752018806658508906noreply@blogger.com2tag:blogger.com,1999:blog-8303893118597899654.post-34904076178281547872015-07-12T14:07:00.000-07:002015-10-29T18:41:50.183-07:00Plumbum Scripting <div tabindex="-1" id="notebook" class="border-box-sizing">
<div class="container" id="notebook-container">
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Scripting in Bash is a pain. Bash can do almost anything, and is unbeatable for small scripts, but it struggles when scaling up to doing anything close to a real world scripting problem. Python is a natural choice, especially for the scientist who already is using it for analysis. But, it's much harder to do basic tasks in Python. So you are left with scripts starting out as Bash scripts, and then becoming a mess, then being (usually poorly) ported to Python, or even worse, being run by a Python script. I've seen countless Python scripts that run Bash scripts that run real programs. I've even written one or two. It's not pretty.</p>
<p>I recently came (back) across a really powerful library for doing efficient command line scripts in Python. It contains a set of tools that makes the four (five with color) main tasks of command line scripts simple and powerful. I will also go over the one main drawback of the library (and the possible enhancement!).</p>
</div>
</div>
</div><a name='more'></a>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<blockquote><p><strong>Note:</strong> The colors module is new to Plumbum in 1.6.0.</p>
</blockquote>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h1 id="Local-commands">Local commands<a class="anchor-link" href="#Local-commands">¶</a></h1><p>The first and foremost part of the library is a replacement for popen, subprocess, etc. of Python. I'll compare the "correct, current" Python standard library method and Plumbum's method.</p>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Basic-commands">Basic commands<a class="anchor-link" href="#Basic-commands">¶</a></h2><p>Our first task will simply be to get our feet wet with a simple command. Let's run <code>ls</code> to see the contents of the current directory. This is easy with <code>subprocess.call</code>:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [4]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="kn">import</span> <span class="nn">subprocess</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [2]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">subprocess</span><span class="o">.</span><span class="n">call</span><span class="p">([</span><span class="s">"echo"</span><span class="p">,</span> <span class="s">"I am a string"</span><span class="p">])</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt output_prompt">Out[2]:</div>
<div class="output_text output_subarea output_execute_result">
<pre>0</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>What just happened? The result, zero, was the return code of the call. The output of the call went to stdout, so if we were in a terminal, we would have seen it output (and in IPython notebook, it will show up in the terminal that started the notebook). This might be what we want, but probably we wanted the value of the output. That would be <code>subprocess.check_output</code>:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [3]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">subprocess</span><span class="o">.</span><span class="n">check_output</span><span class="p">([</span><span class="s">"echo"</span><span class="p">,</span> <span class="s">"I am a string"</span><span class="p">])</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt output_prompt">Out[3]:</div>
<div class="output_text output_subarea output_execute_result">
<pre>b'I am a string\n'</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>As you can already see, this not only requires different calls for different situations, but it even gave a bytes string (which is technically correct, but almost never what you want for a shell script). The reason for the different calls is because they are shortcuts to the actual subprocess Popen object. So we really need:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [10]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">p</span> <span class="o">=</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">Popen</span><span class="p">([</span><span class="s">"echo"</span><span class="p">,</span><span class="s">"I am a string"</span><span class="p">],</span>
<span class="n">shell</span><span class="o">=</span><span class="k">False</span><span class="p">,</span> <span class="n">bufsize</span><span class="o">=</span><span class="mi">512</span><span class="p">,</span>
<span class="n">stdin</span><span class="o">=</span><span class="n">subprocess</span><span class="o">.</span><span class="n">PIPE</span><span class="p">,</span>
<span class="n">stdout</span><span class="o">=</span><span class="n">subprocess</span><span class="o">.</span><span class="n">PIPE</span><span class="p">,</span>
<span class="n">stderr</span><span class="o">=</span><span class="n">subprocess</span><span class="o">.</span><span class="n">PIPE</span><span class="p">)</span>
<span class="n">outs</span><span class="p">,</span> <span class="n">errs</span> <span class="o">=</span> <span class="n">p</span><span class="o">.</span><span class="n">communicate</span><span class="p">()</span>
<span class="n">outs</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt output_prompt">Out[10]:</div>
<div class="output_text output_subarea output_execute_result">
<pre>b'I am a string\n'</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>As you can guess, this is only a small smattering of the options you can pass (not all were needed for this call), but it gives you an idea of what is needed to work with subprocess.</p>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Let's look at Plumbum. First, let's see the fastest method to get a command:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [7]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="kn">from</span> <span class="nn">plumbum</span> <span class="k">import</span> <span class="n">local</span><span class="p">,</span> <span class="n">FG</span><span class="p">,</span> <span class="n">BG</span><span class="p">,</span> <span class="n">TF</span><span class="p">,</span> <span class="n">RETCODE</span>
<span class="n">echo</span> <span class="o">=</span> <span class="n">local</span><span class="p">[</span><span class="s">'echo'</span><span class="p">]</span>
<span class="n">echo</span><span class="p">(</span><span class="s">"I am a string"</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt output_prompt">Out[7]:</div>
<div class="output_text output_subarea output_execute_result">
<pre>'I am a string\n'</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Here, we have a local object, which represents the computer. It acts like a dictionary; if you put in a key, you get the command that would be called if you run that command in a terminal. Let's look at the object we get:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [5]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">echo</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt output_prompt">Out[5]:</div>
<div class="output_text output_subarea output_execute_result">
<pre>LocalCommand(<LocalPath /bin/echo>)</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Now this is a working python object and can be called like any Python function! In fact, it can access most all of the details and power of the Popen object we saw earlier. If you don't like to repeat yourself, there is a magic shortcut for getting commands:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [6]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="kn">from</span> <span class="nn">plumbum.cmd</span> <span class="k">import</span> <span class="n">echo</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>There is no <code>echo</code> command in a <code>cmd.py</code> file somewhere; this dynamically does exactly what we did, calling <code>['echo']</code> on the local object. This is quicker and simpler, but it is good to know what happens behind the scenes!</p>
<p>Plumbum also allows you to add arguments to a command without running the command; as you will soon see, this allows you to build complex commands just like bash. If you use square brackets instead of parenthesis, the command doesn't run yet (Haskal users: this is currying; Pythonistas will know it as <code>partial</code>)</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [7]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">echo</span><span class="p">[</span><span class="s">"I am a string"</span><span class="p">]</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt output_prompt">Out[7]:</div>
<div class="output_text output_subarea output_execute_result">
<pre>BoundCommand(LocalCommand(<LocalPath /bin/echo>), ['I am a string'])</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>When you are ready, you can call it:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [8]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">echo</span><span class="p">[</span><span class="s">"I am a string"</span><span class="p">]()</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt output_prompt">Out[8]:</div>
<div class="output_text output_subarea output_execute_result">
<pre>'I am a string\n'</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Or, you can run it in the forground, so that the output is sent to the current terminal <strong>as it runs</strong> (this is the <code>subprocess.call</code> equivilent from the beginning, although non-zero return values are not handled in the same way):</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [8]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="kn">from</span> <span class="nn">plumbum</span> <span class="k">import</span> <span class="n">FG</span>
<span class="n">echo</span><span class="p">[</span><span class="s">"I am a string"</span><span class="p">]</span> <span class="o">&</span> <span class="n">FG</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Complex-commands-(piping)">Complex commands (piping)<a class="anchor-link" href="#Complex-commands-(piping)">¶</a></h2>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="Stdin">Stdin<a class="anchor-link" href="#Stdin">¶</a></h3><p>Now, how about input a python text string to a command? As an example, let's use the unix <code>dc</code> command. It is a desktop calculator, with reverse polish notation syntax.</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [10]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="kn">from</span> <span class="nn">plumbum.cmd</span> <span class="k">import</span> <span class="n">dc</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>We can call it using the <code>-e</code> flag followed by the calculation we want to preform, like <code>1 + 2</code>. We already know how to do that,</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [11]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">dc</span><span class="p">(</span><span class="s">'-e'</span><span class="p">,</span> <span class="s">'1 2 + p'</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt output_prompt">Out[11]:</div>
<div class="output_text output_subarea output_execute_result">
<pre>'3\n'</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>But, it also can be run without this flag. If we do that, we can then type (or pipe) text in from the bash shell.</p>
</div>
</div>
</div><div class="splitbox"><div class="left">
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>In <code>subprocess</code>, we don't have a shortcut, so we have to use <code>Popen</code>, manually setting the <code>stdin</code> and <code>stdout</code> to a <code>subprocess</code> <code>PIPE</code>, and then communicate in bytes.</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [9]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">proc</span> <span class="o">=</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">Popen</span><span class="p">([</span><span class="s">'dc'</span><span class="p">],</span>
<span class="n">stdin</span><span class="o">=</span><span class="n">subprocess</span><span class="o">.</span><span class="n">PIPE</span><span class="p">,</span>
<span class="n">stdout</span><span class="o">=</span><span class="n">subprocess</span><span class="o">.</span><span class="n">PIPE</span><span class="p">)</span>
<span class="n">outs</span><span class="p">,</span> <span class="n">errs</span> <span class="o">=</span> <span class="n">proc</span><span class="o">.</span><span class="n">communicate</span><span class="p">(</span><span class="s">'1 2 + p'</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s">'ascii'</span><span class="p">))</span>
<span class="n">outs</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt output_prompt">Out[9]:</div>
<div class="output_text output_subarea output_execute_result">
<pre>b'3\n'</pre>
</div>
</div>
</div>
</div>
</div></div><div class="right">
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Compare that to Plumbum:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [13]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="p">(</span><span class="n">dc</span> <span class="o"><<</span> <span class="s">'1 2 + p'</span><span class="p">)()</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt output_prompt">Out[13]:</div>
<div class="output_text output_subarea output_execute_result">
<pre>'3\n'</pre>
</div>
</div>
</div>
</div>
</div></div><div style="clear:both"></div></div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="Piping">Piping<a class="anchor-link" href="#Piping">¶</a></h3><p>Of course, in bash we can pipe text from one command to another. Let's compare that (not going to even try the subprocess call here).</p>
</div>
</div>
</div><div class="splitbox"><div class="left">
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Since I'm using IPython, prepending a line with ! will cause it to run in bash. So, in Bash:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [21]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="o">!</span><span class="nb">echo</span> <span class="s2">"1 2 + p"</span> <span class="p">|</span> dc
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>3
</pre>
</div>
</div>
</div>
</div>
</div></div><div class="right">
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>In Plumbum:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [17]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="p">(</span><span class="n">echo</span><span class="p">[</span><span class="s">"1 2 + p"</span><span class="p">]</span> <span class="o">|</span> <span class="n">dc</span><span class="p">)()</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt output_prompt">Out[17]:</div>
<div class="output_text output_subarea output_execute_result">
<pre>'3\n'</pre>
</div>
</div>
</div>
</div>
</div></div><div style="clear:both"></div></div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>If we wanted to see what that command would look like in bash, then we can call print on the unevaluated object:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [18]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="nb">print</span><span class="p">(</span><span class="n">echo</span><span class="p">[</span><span class="s">"1 2 + p"</span><span class="p">]</span> <span class="o">|</span> <span class="n">dc</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>/bin/echo '1 2 + p' | /usr/bin/dc
</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="Background-execution">Background execution<a class="anchor-link" href="#Background-execution">¶</a></h3><p>One of the great things about Bash is the ease of "simple" multithreading; you can start a command in the background using the <code>&</code> character. To test this, we need a long running command that returns a value. In bash, we can make this using the following function:</p>
<div class="highlight"><pre><span class="nv">$ </span>fun <span class="o">()</span> <span class="o">{</span> sleep 3<span class="p">;</span> <span class="nb">echo</span> <span class="s1">'finished'</span><span class="p">;</span> <span class="o">}</span>
<span class="nv">$ </span>fun
finished
<span class="nv">$ </span>fun <span class="p">&</span>
<span class="o">[</span>1<span class="o">]</span> 6210
<span class="nv">$finished</span>
<span class="o">[</span>1<span class="o">]</span>+ Done fun
</pre></div>
<p>Here, when we ran it in the foreground, it held up our terminal until it finished. The second time we ran it, it gave us back our terminal, but we were interrupt ed three seconds later with text from the process. If we wanted to interact with the process, or wait for it to finish, etc, we could do <code>$!</code> to get the pid of the last spawned process, and then use <code>wait</code> to wait on the pid. (see <a href="https://gist.github.com/henryiii/5841984">git-all.bash</a> for an example).</p>
<p>This simplicity is not usually something that is easy to emulate in a programing language. Let's see it in plumbum. Here, I'm piping sleep (which doesn't print anything) to echo, just so I can get a slow running command, and I'm using IPython's time magic to measure the time taken:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [19]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="o">%%</span><span class="k">time</span>
sleep = local['sleep']
sleep_and_print = sleep['3'] | echo['hi']
print(sleep_and_print())
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>hi
CPU times: user 5.75 ms, sys: 9.25 ms, total: 15 ms
Wall time: 3.01 s
</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [26]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="o">%%</span><span class="k">time</span>
bg = sleep_and_print & BG
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>CPU times: user 3.94 ms, sys: 7.45 ms, total: 11.4 ms
Wall time: 20.4 ms
</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Now, bg is a <code>Future</code> object that is attached to the background process. We can call <code>.poll()</code> on it to see if it's done or <code>.wait()</code> to wait until it returns. Then, we can access the stdout and stderr of the command. (<code>stdout</code>, etc. will automatically <code>wait()</code> for you, so you can use them directly.)</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [27]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="o">%%</span><span class="k">time</span>
print(bg.stdout)
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>hi
CPU times: user 2.14 ms, sys: 1.76 ms, total: 3.9 ms
Wall time: 2.73 s
</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h1 id="Remote-commands">Remote commands<a class="anchor-link" href="#Remote-commands">¶</a></h1><p>Besides local commands, Plumbum provides a remote class for working with remote machines via SSH in a platform independent manner. It works much like the local object, and will use the best system, including Paramiko, to do the processes. I haven't moved my scripts from pure Paramiko to Plumbum yet, but only having to learn one procedure for both local and remote machines is a huge plus (and Paramiko is fairly ugly to program in, like subprocess).</p>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h1 id="Command-Line-Applications">Command Line Applications<a class="anchor-link" href="#Command-Line-Applications">¶</a></h1><p>Command line applications on Python already have one of the best toolkits available, argparse (C++'s Boost Program Options library is a close second). However, after seeing the highly pythonic Plumbum <code>cli</code> module, it feels repetitive and antiquated.</p>
<p>Let's look at a command line application that takes a couple of options.
In argparse, we would need to do the following:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [25]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="o">%%</span><span class="k">writefile</span> color_argparse.py
import argparse
def main():
parser = argparse.ArgumentParser(description='Echo a command in color.')
parser.add_argument('-c','--color', type=str,
help='Color to print')
parser.add_argument('echo',
help='The item to print in color')
args = parser.parse_args()
print('I should print', args.echo, 'in', args.color, "- but I'm lazy.")
if __name__ == '__main__':
main()
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>Overwriting color_argparse.py
</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [26]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="o">%</span><span class="k">run</span> color_argparse.py -c red item
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>I should print item in red - but I'm lazy.
</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>As you can tell from the <a href="https://docs.python.org/3/library/argparse.html">documentation</a>, the programs quickly grow as you try to do more advanced commands, grouping, or subcommands. Now compare to Plumbum:</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [4]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="o">%%</span><span class="k">writefile</span> color_plumbum.py
from plumbum import cli
class ColorApp(cli.Application):
color = cli.SwitchAttr(['-c','--color'], help='Color to print')
def main(self, echo):
print('I should print', echo, 'in', self.color, "- but I'm lazy.")
if __name__ == '__main__':
ColorApp.run()
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>Overwriting color_plumbum.py
</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [5]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="o">%</span><span class="k">run</span> color_plumbum.py -c red item
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>I should print item in red - but I'm lazy.
</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Here, we see a more natural mapping of class -> program, and also we have a lot more control over the items this way, as well. For example, if we want to add a validator, say to check existing files or to ensure a number in a range or a word in a set, we can do that on each switch. Switches can also be full fledged functions that run when the switch is set. And, we can easily extend this process to subcommands (see <a href="https://gist.github.com/e194abcae39eff6ef123.git">git-all.py</a>) and it remains readable and avoids duplication.</p>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h1 id="Path-manipulations">Path manipulations<a class="anchor-link" href="#Path-manipulations">¶</a></h1><p>Path manipulations using <code>os.path</code> functions are messy and can become involved quickly. Things that should be simple require several functions chained to get anywhere. The situation was bad enough to warrant adding an additional module to Python 3.4+, the provisional <a href="https://docs.python.org/3/library/pathlib.html">pathlib</a> module. Now this is not a bad module, but you have to install a separate library on Python 2.7 or 3.3 to get it, and it has a couple of missing features. Plumbum provides a similar construct, and it is automatically available if you are already using Plumbum, and it corrects two of the three missing features. The features I'm mentioning are:</p>
<ul>
<li>No support for manipulation of multiple extensions, like <code>.tar.gz</code><ul>
<li>Plumbum supports an additional argument to <code>.with_suffix()</code>, default matches pathlib</li>
</ul>
</li>
<li>No support for home directories<ul>
<li>Plumbum provides the <code>local.env.home</code> path</li>
</ul>
</li>
<li>No support for using <code>open(path)</code> without wrapping in a <code>str()</code> call<ul>
<li>Can't be fixed unless path subclasses str (not likely for either library, see unipath), or pathlib support added to the system open function (any Python devs reading? Please?)</li>
</ul>
</li>
</ul>
<p>I would love to see the <code>pathlib</code> module adapt the <code>.with_suffix()</code> addition that Plumbum has, and add some sort of home directory expansion or path, as well.</p>
<p>Plumbum also has the unique little trick that <code>//</code> automatically calls glob, making path composition even simpler. I doubt we'll get this added to pathlib, but I can always hope (at least, until someone removes the provisional status).</p>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h1 id="Color-support-(NEW)">Color support (NEW)<a class="anchor-link" href="#Color-support-(NEW)">¶</a></h1><p>I've been working on a new color library for Plumbum. <code>git-all.py</code> has been converted to use it.</p>
<p>Colors are used through the Styles generated by the colors object. You can get colors and atributes like this:</p>
<div class="highlight"><pre><span class="kn">from</span> <span class="nn">plumbum</span> <span class="kn">import</span> <span class="n">colors</span>
<span class="n">red</span> <span class="o">=</span> <span class="n">colors</span><span class="o">.</span><span class="n">fg</span><span class="o">.</span><span class="n">red</span> <span class="c"># Red forground color</span>
<span class="n">other_color</span> <span class="o">=</span> <span class="n">colors</span><span class="o">.</span><span class="n">bg</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span> <span class="c"># The second background color</span>
<span class="n">bold</span> <span class="o">=</span> <span class="n">colors</span><span class="o">.</span><span class="n">cold</span>
<span class="n">reset</span> <span class="o">=</span> <span class="n">colors</span><span class="o">.</span><span class="n">reset</span>
</pre></div>
<p>You can directly access <code>colors</code> as if it was the <code>fg</code> object. Standard term colors can be accessed with <code>()</code>, and the 256 extended colors can be accessed with <code>[]</code> by number, name (camel case or underscore), or html code. All objects support with statements, which restores normal font (for a single <code>Style</code>, it will reset only the necessary component if possible, like <code>bold</code> or <code>fg</code> color). You can manually take the inverse (<code>~</code>) to get the undo-ing action. Calling a <code>Style</code> without any parameters will send it to <code>stdout</code>. Using <code>|</code> will wrap a string with a style (as will <code>[]</code> notation). Styles otherwise act just like normal text, so they can be added to text, etc (they are <code>str</code> subclasses, after all).</p>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>For the following demo, I'll be using the HTMLCOLOR, and a with statement to capture ouput in IPython and display it as HTML. (See my upcoming post for a more elegant IPython display technique.) Also note <code>redirect_stdout</code> is new in Python 3.4, but is easy to implement in other versions if needed.</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [1]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="kn">from</span> <span class="nn">plumbum.colorlib</span> <span class="k">import</span> <span class="n">htmlcolors</span> <span class="k">as</span> <span class="n">colors</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [2]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="kn">from</span> <span class="nn">IPython.display</span> <span class="k">import</span> <span class="n">display_html</span>
<span class="kn">from</span> <span class="nn">contextlib</span> <span class="k">import</span> <span class="n">contextmanager</span><span class="p">,</span> <span class="n">redirect_stdout</span>
<span class="kn">from</span> <span class="nn">io</span> <span class="k">import</span> <span class="n">StringIO</span> <span class="c"># Python3 name</span>
<span class="nd">@contextmanager</span>
<span class="k">def</span> <span class="nf">show_html</span><span class="p">():</span>
<span class="n">out</span><span class="o">=</span><span class="n">StringIO</span><span class="p">()</span>
<span class="k">with</span> <span class="n">redirect_stdout</span><span class="p">(</span><span class="n">out</span><span class="p">):</span>
<span class="k">yield</span>
<span class="n">display_html</span><span class="p">(</span><span class="n">out</span><span class="o">.</span><span class="n">getvalue</span><span class="p">(),</span> <span class="n">raw</span><span class="o">=</span><span class="k">True</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Now, inside the capture context manager, we can use COLOR just like on a terminal (save for needing to use <code></br></code> to break lines if we don't take advantage of the build in htmlstyle print command, and having to be careful not to use un-reset Styles).</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [6]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="k">with</span> <span class="n">show_html</span><span class="p">():</span>
<span class="n">colors</span><span class="o">.</span><span class="n">green</span><span class="o">.</span><span class="n">print</span><span class="p">(</span><span class="s">"This is in red!"</span><span class="p">)</span>
<span class="p">(</span><span class="n">colors</span><span class="o">.</span><span class="n">bold</span> <span class="o">&</span> <span class="n">colors</span><span class="o">.</span><span class="n">blue</span><span class="p">)</span><span class="o">.</span><span class="n">print</span><span class="p">(</span><span class="s">"This is in bold blue!"</span><span class="p">)</span>
<span class="n">colors</span><span class="o">.</span><span class="n">bg</span><span class="p">[</span><span class="s">'LightYellow'</span><span class="p">]</span><span class="o">.</span><span class="n">print</span><span class="p">(</span><span class="s">"This is on the background!"</span><span class="p">)</span>
<span class="n">colors</span><span class="p">[</span><span class="s">'LightBlue'</span><span class="p">]</span><span class="o">.</span><span class="n">print</span><span class="p">(</span><span class="s">"This is also from the extented color set"</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s">"This is {colors.em}emphasized{colors.em.reset}! (reset was needed)"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">colors</span><span class="o">=</span><span class="n">colors</span><span class="p">),</span> <span class="n">end</span><span class="o">=</span><span class="s">'<br/>'</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s">"This is normal"</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_html rendered_html output_subarea ">
<font color="#00C000">This is in red!</font><br/>
<font color="#0000C0"><b>This is in bold blue!</b></font><br/>
<span style="background-color: #FFFF00">This is on the background!</span><br/>
<font color="#0000FF">This is also from the extented color set</font><br/>
This is <em>emphasized</em>! (reset was needed)<br/>This is normal
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h1 id="Putting-it-together-in-an-example:-git-all">Putting it together in an example: git-all<a class="anchor-link" href="#Putting-it-together-in-an-example:-git-all">¶</a></h1>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Now, let's look at a real world example previously mentioned: <a href="https://gist.github.com/henryiii/5841984">git-all.bash</a>. This is a script I wrote some time ago for checking a large number of repositories in a common folder. Due to the clever way git subcommands work, simply naming this <code>git-all</code> and putting it in your path gives your a <code>git all</code> command. It is written in very reasonable bash, IMO, and works well.</p>
<h2 id="Directory-manipulation">Directory manipulation<a class="anchor-link" href="#Directory-manipulation">¶</a></h2><p>Let's look at this piece by piece and see what would be required to convert it to Python. First, this script is in one of the repo's, so we need the current directory, up one.</p>
</div>
</div>
</div><div class="splitbox"><div class="left">
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>In Bash that's:</p>
<div class="highlight"><pre><span class="nb">unset </span>CDPATH
<span class="nv">SOURCE</span><span class="o">=</span><span class="s2">"</span><span class="si">${</span><span class="nv">BASH_SOURCE</span><span class="p">[0]</span><span class="si">}</span><span class="s2">"</span>
<span class="k">while</span> <span class="o">[</span> -h <span class="s2">"</span><span class="nv">$SOURCE</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">do</span>
<span class="nv">DIR</span><span class="o">=</span><span class="s2">"</span><span class="k">$(</span> <span class="nb">cd</span> -P <span class="s2">"</span><span class="k">$(</span> dirname <span class="s2">"</span><span class="nv">$SOURCE</span><span class="s2">"</span> <span class="k">)</span><span class="s2">"</span> <span class="o">&&</span> <span class="nb">pwd</span> <span class="k">)</span><span class="s2">"</span>
<span class="nv">SOURCE</span><span class="o">=</span><span class="s2">"</span><span class="k">$(</span>readlink <span class="s2">"</span><span class="nv">$SOURCE</span><span class="s2">"</span><span class="k">)</span><span class="s2">"</span>
<span class="o">[[</span> <span class="nv">$SOURCE</span> !<span class="o">=</span> /* <span class="o">]]</span> <span class="o">&&</span> <span class="nv">SOURCE</span><span class="o">=</span><span class="s2">"</span><span class="nv">$DIR</span><span class="s2">/</span><span class="nv">$SOURCE</span><span class="s2">"</span>
<span class="k">done</span>
<span class="nv">DIR</span><span class="o">=</span><span class="s2">"</span><span class="k">$(</span> <span class="nb">cd</span> -P <span class="s2">"</span><span class="k">$(</span> dirname <span class="s2">"</span><span class="nv">$SOURCE</span><span class="s2">"</span> <span class="k">)</span><span class="s2">"</span> <span class="o">&&</span> <span class="nb">pwd</span> <span class="k">)</span><span class="s2">"</span>
<span class="nv">REPOLOC</span><span class="o">=</span><span class="nv">$DIR</span>/..
</pre></div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>(Sorry for the awful highlighting by IPython, it hates the $ in strings for Bash.)</p>
</div>
</div>
</div></div><div class="right">
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Converted to python,</p>
<div class="highlight"><pre><span class="n">REPOLOC</span> <span class="o">=</span> <span class="n">local</span><span class="o">.</span><span class="n">path</span><span class="p">(</span><span class="n">__file__</span><span class="p">)</span> <span class="o">/</span> <span class="s">'..'</span>
</pre></div>
</div>
</div>
</div></div><div style="clear:both"></div></div><div class="splitbox"><div class="left">
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>We can find the directories that are valid repos:</p>
<div class="highlight"><pre><span class="k">for</span> file in <span class="k">$(</span>ls<span class="k">)</span><span class="p">;</span> <span class="k">do</span>
<span class="k">if</span> <span class="o">[[</span> -d <span class="nv">$REPOLOC</span>/<span class="nv">$file</span>/.git <span class="o">]]</span><span class="p">;</span> <span class="k">then</span>
...
<span class="k">done</span>
</pre></div>
<p>And code goes here.</p>
</div>
</div>
</div></div><div class="right">
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>In Python, lists are easy to use:</p>
<div class="highlight"><pre><span class="n">valid_repos</span> <span class="o">=</span> <span class="p">[</span><span class="n">d</span> <span class="o">/</span> <span class="s">'../..'</span> <span class="k">for</span> <span class="n">d</span> <span class="ow">in</span> <span class="n">local</span><span class="o">.</span><span class="n">cwd</span> <span class="o">//</span> <span class="s">'*/.git/config'</span><span class="p">]</span>
</pre></div>
</div>
</div>
</div></div><div style="clear:both"></div></div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>The multiple ugly loops over all repos can easily translate into a generator:</p>
<div class="highlight"><pre><span class="k">def</span> <span class="nf">git_on_all</span><span class="p">(</span><span class="n">bold</span><span class="o">=</span><span class="bp">False</span><span class="p">):</span>
<span class="k">for</span> <span class="n">n</span><span class="p">,</span><span class="n">repo</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">valid_repos</span><span class="p">):</span>
<span class="k">with</span> <span class="n">local</span><span class="o">.</span><span class="n">cwd</span><span class="p">(</span><span class="n">repo</span><span class="p">):</span>
<span class="k">with</span> <span class="n">color_change</span><span class="p">(</span><span class="n">n</span><span class="p">):</span>
<span class="k">yield</span> <span class="n">repo</span><span class="o">.</span><span class="n">basename</span>
</pre></div>
<p>To use it, simply loop over <code>git_on_all()</code>:</p>
<div class="highlight"><pre><span class="k">for</span> <span class="n">repo_name</span> <span class="ow">in</span> <span class="n">git_on_all</span><span class="p">():</span>
<span class="k">print</span><span class="p">(</span><span class="s">'The current working directory is in the'</span><span class="p">,</span> <span class="n">repo_name</span><span class="p">,</span> <span class="s">'repo!'</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Command-line-arguments">Command line arguments<a class="anchor-link" href="#Command-line-arguments">¶</a></h2><p>We don't have a nice <code>cli</code> tool in Bash, so we have to build long if statements. We can separate each command in Python, and let the help file be built for us:</p>
<div class="highlight"><pre><span class="nd">@GitAll.subcommand</span><span class="p">(</span><span class="s">"pull"</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">Pull</span><span class="p">(</span><span class="n">cli</span><span class="o">.</span><span class="n">Application</span><span class="p">):</span>
<span class="s">'Pulls all repos in the folder, not threaded.'</span>
<span class="k">def</span> <span class="nf">main</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">for</span> <span class="n">repo</span> <span class="ow">in</span> <span class="n">git_on_all</span><span class="p">():</span>
<span class="n">git</span><span class="p">[</span><span class="s">'pull'</span><span class="p">]</span> <span class="o">&</span> <span class="n">FG</span>
</pre></div>
<p>This is <code>git all pull</code>, clean and seperated from the ugly loops in Bash.</p>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Multithreading">Multithreading<a class="anchor-link" href="#Multithreading">¶</a></h2>
</div>
</div>
</div><div class="splitbox"><div class="left">
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>The fetch loop, one of the strong points of the Bash script, looks like this:</p>
<div class="highlight"><pre><span class="k">if</span> <span class="o">[[</span> <span class="nv">$1</span> <span class="o">==</span> qfetch <span class="o">]]</span>
<span class="o">||</span> <span class="o">[[</span> <span class="nv">$1</span> <span class="o">==</span> fetch <span class="o">]]</span>
<span class="o">||</span> <span class="o">[[</span> <span class="nv">$1</span> <span class="o">==</span> status <span class="o">]]</span><span class="p">;</span> <span class="k">then</span>
<span class="k">for</span> file in <span class="k">$(</span>ls<span class="k">)</span><span class="p">;</span> <span class="k">do</span>
<span class="k">if</span> <span class="o">[[</span> -d <span class="nv">$REPOLOC</span>/<span class="nv">$file</span>/.git <span class="o">]]</span><span class="p">;</span> <span class="k">then</span>
<span class="nb">cd</span> <span class="nv">$REPOLOC</span>/<span class="nv">$file</span>
git fetch -q <span class="p">&</span>
spawned+<span class="o">=(</span><span class="nv">$!</span><span class="o">)</span>
<span class="k">fi</span>
<span class="k">done</span>
<span class="nb">echo</span> -n <span class="s2">"Waiting for all repos to report: "</span>
<span class="k">for</span> pid in <span class="si">${</span><span class="nv">spawned</span><span class="p">[@]</span><span class="si">}</span><span class="p">;</span> <span class="k">do</span>
<span class="nb">wait</span> <span class="nv">$pid</span>
<span class="k">done</span>
<span class="nb">echo</span> <span class="s2">"done"</span>
<span class="k">fi</span>
</pre></div>
</div>
</div>
</div></div><div class="right">
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>This does a normally advanced multithreading process in a few, simple lines. In Python, we have</p>
<div class="highlight"><pre><span class="k">def</span> <span class="nf">fetch</span><span class="p">():</span>
<span class="n">bg</span> <span class="o">=</span> <span class="p">[(</span><span class="n">git</span><span class="p">[</span><span class="s">'fetch'</span><span class="p">,</span><span class="s">'-q'</span><span class="p">]</span> <span class="o">&</span> <span class="n">BG</span> <span class="p">)</span>
<span class="k">for</span> <span class="n">repo</span> <span class="ow">in</span> <span class="n">git_on_all</span><span class="p">()]</span>
<span class="n">printf</span><span class="p">(</span><span class="s">'Waiting for the repos to report: '</span><span class="p">)</span>
<span class="k">for</span> <span class="n">fut</span> <span class="ow">in</span> <span class="n">bg</span><span class="p">:</span>
<span class="n">fut</span><span class="o">.</span><span class="n">wait</span><span class="p">()</span>
<span class="k">print</span><span class="p">(</span><span class="s">'done'</span><span class="p">)</span>
</pre></div>
<p>This is just as readable, if not more so, and doesn't need the if loop to check the input, since that's now part of the <code>cli</code> interface. The actual version in the script also can report errors in the fetch, which the Bash version can not.</p>
</div>
</div>
</div></div><div style="clear:both"></div></div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Colors-(classic-tput-method)">Colors (classic tput method)<a class="anchor-link" href="#Colors-(classic-tput-method)">¶</a></h2><p>We would like to toggle colors, so each repo is in a different cyclic color. My final Bash solution was elegant:</p>
</div>
</div>
</div><div class="splitbox"><div class="left">
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Bash (will need to run <code>echo -n</code> on these):</p>
<div class="highlight"><pre><span class="nv">txtreset</span><span class="o">=</span><span class="k">$(</span>tput sgr0<span class="k">)</span>
<span class="nv">txtbold</span><span class="o">=</span><span class="k">$(</span>tput bold<span class="k">)</span>
</pre></div>
</div>
</div>
</div></div><div class="right">
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Python would be able to do the same thing (will only need to run these in the foreground, with <code>& FG</code> ):</p>
<div class="highlight"><pre><span class="n">txtreset</span><span class="o">=</span><span class="n">tput</span><span class="p">[</span><span class="s">'sgr0'</span><span class="p">]</span>
<span class="n">txtbold</span><span class="o">=</span><span class="n">tput</span><span class="p">[</span><span class="s">'bold'</span><span class="p">]</span>
</pre></div>
<p>Though with the <code>plumbum.color</code> library, we don't have to.</p>
</div>
</div>
</div></div><div style="clear:both"></div></div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Color changing is easy to implement with a Python context manager:</p>
<div class="highlight"><pre><span class="nd">@contextmanager</span>
<span class="k">def</span> <span class="nf">color_change</span><span class="p">(</span><span class="n">color</span><span class="p">):</span>
<span class="n">txtreset</span> <span class="o">&</span> <span class="n">FG</span>
<span class="n">txtbold</span> <span class="o">&</span> <span class="n">FG</span>
<span class="n">tput</span><span class="p">[</span><span class="s">'setaf'</span><span class="p">,</span><span class="n">color</span><span class="o">%</span><span class="mi">6</span><span class="o">+</span><span class="mi">1</span><span class="p">]</span> <span class="o">&</span> <span class="n">FG</span>
<span class="k">try</span><span class="p">:</span>
<span class="k">yield</span>
<span class="k">finally</span><span class="p">:</span>
<span class="n">txtrst</span> <span class="o">&</span> <span class="n">FG</span>
</pre></div>
<p>The try/finally block allows this to restore our color, even if it throws an exception! This is tremendously better than the Bash version, which leaves the color on the terminal if you make a mistake. A nice example of context managers can be found <a href="http://preshing.com/20110920/the-python-with-statement-by-example/">on Jeff Preshing's blog</a>.</p>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>You can use it to wrap parts of the code that print in a color:</p>
<div class="highlight"><pre><span class="k">with</span> <span class="n">colorchange</span><span class="p">(</span><span class="n">tput</span><span class="p">(</span><span class="s">'setaf'</span><span class="p">,</span><span class="mi">2</span><span class="p">),</span> <span class="n">bold</span><span class="o">=</span><span class="bp">True</span><span class="p">):</span>
<span class="k">print</span><span class="p">(</span><span class="s">'This will be in color number 2'</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Colors-(new-method)">Colors (new method)<a class="anchor-link" href="#Colors-(new-method)">¶</a></h2><p>Plumbum has a new colors tool, and this is how you would use in in this script.</p>
<div class="highlight"><pre><span class="kn">from</span> <span class="nn">plumbum</span> <span class="kn">import</span> <span class="n">colors</span>
</pre></div>
<p>Colors can be generated cyclically by number, and combinations of color and attributes can be put in a with statement, too:</p>
<div class="highlight"><pre><span class="k">with</span><span class="p">(</span><span class="n">colors</span><span class="o">.</span><span class="n">fg</span><span class="p">[</span><span class="mi">1</span><span class="p">:</span><span class="mi">7</span><span class="p">][</span><span class="n">n</span><span class="o">%</span><span class="mi">6</span><span class="p">]</span> <span class="o">&</span> <span class="n">colors</span><span class="o">.</span><span class="n">bold</span><span class="p">):</span>
</pre></div>
<p>And, we can simply unbold:</p>
<div class="highlight"><pre><span class="n">colors</span><span class="o">.</span><span class="n">bold</span><span class="o">.</span><span class="n">reset</span><span class="o">.</span><span class="k">print</span><span class="p">(</span><span class="n">git</span><span class="p">(</span><span class="s">'diff-files'</span><span class="p">,</span> <span class="s">'--name-status'</span><span class="p">,</span> <span class="s">'-r'</span><span class="p">,</span> <span class="s">'--ignore-submodules'</span><span class="p">,</span> <span class="s">'--'</span><span class="p">))</span>
</pre></div>
<p>And that's it! All the benefits we had from before are here.</p>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Final-Comparison">Final Comparison<a class="anchor-link" href="#Final-Comparison">¶</a></h2><p>I'll be using functions in the Python version to make it clear what each git call does, and making the Python version cleaner in a few ways that I could also apply to the Bash script. So this is not meant to be a 1:1 comparison. In my defense, Bash users tend to avoid functions or other clean programming practices.</p>
<p>Most of the extra lines are from the Python functions. Also, I've improved a couple of commands for git for current best practices. I've also avoided using FG for the print commands, so that I can control the color and the long-output paging (If you change <code>print()</code> for <code>& FG</code>, the output would match the Bash script). Here is the script: <a href="https://gist.github.com/henryiii/e194abcae39eff6ef123">git-all.py</a>.</p>
<blockquote><p><strong>Note:</strong> You might want to look at the history of that script, as I'll probaby update it occasionally as I start using it.</p>
</blockquote>
<p>Notice that it is very clear what each part of the <code>cli</code> part of the script, and it's easy to add a feature or extend it. The long for loops are nicely abstracted into iterators.</p>
<p>Also, there may be bugs for a few days while I start using this instead of my bash script. Also, it must be renamed to <code>git-all</code> with no extension for <code>git all status</code> etc. to work.</p>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h1 id="Bonus:-Possible-improvement:-argcomplete-support">Bonus: Possible improvement: argcomplete support<a class="anchor-link" href="#Bonus:-Possible-improvement:-argcomplete-support">¶</a></h1>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>One last thing: The one drawback to Plumbum over argparse is due to one enhancement package for argparse. I think a great addition to the Plumbum library would be <a href="https://github.com/kislyuk/argcomplete">argcomlete</a> support. If you've never seen argcomplete, it's a bash completion extension for argparse. Argcomplete allows you to add validators, and will suggest completions when pressing tab on the command line.</p>
<p>Adding support to Plumbum would not be that hard, and probably wouldn't even require argcomplete to be installed. The Argcomplete API requires three things:</p>
<ul>
<li><code>#ARGCOMPLETE_OK</code> near the top of a script</li>
<li>Special output piped to several channels (8 and 9, I believe) of the terminal when the <code>_ARGCOMPLETE</code> special environment variable is set, and then exits before calling <code>.main()</code>.</li>
<li>The ability to predict the next completion</li>
</ul>
<p>The first one is easy, and wouldn't require anything from Plumbum. The second would be a simple addition to a new method <code>cli.Application.argcomplete(self)</code> that could be overridden to remove or customize argcomplete support. The final one is the hard one, the prediction of the possible completions. If that can be done, support could be added.</p>
<p>Because support would be added into Plumbum itself, you wouldn't have to use the monkey patching that argcomplete has to use to inject itself into argparse. You would still use the same bash hooks that argcomplete uses, so it would work along side it, being called in the same way.</p>
</div>
</div>
</div>
</div>
</div>Anonymoushttp://www.blogger.com/profile/10752018806658508906noreply@blogger.com0tag:blogger.com,1999:blog-8303893118597899654.post-41958764551513300042015-07-07T11:38:00.000-07:002015-07-18T19:44:53.240-07:00Setting up Blogger for IPython notebooksI found that setting up Blogger to show IPython HTML was not very well covered for IPython 3.0. To get it to work, I added the following to my template's HTML, right after the head tag:
<a name='more'></a>
<div class="highlight">
<code>
<!-- Load mathjax --><br />
<script src="https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS_HTML"></script><br />
<!-- MathJax configuration --><br />
<script type="text/x-mathjax-config"><br />
MathJax.Hub.Config({<br />
tex2jax: {<br />
inlineMath: [ ['$','$'], ["\\(","\\)"] ],<br />
displayMath: [ ['$$','$$'], ["\\[","\\]"] ],<br />
processEscapes: true,<br />
processEnvironments: true<br />
},<br />
// Center justify equations in code and markdown cells. Elsewhere<br />
// we use CSS to left justify single line equations in code cells.<br />
displayAlign: 'center',<br />
"HTML-CSS": {<br />
styles: {'.MathJax_Display': {"margin": 0}},<br />
linebreaks: { automatic: true }<br />
}<br />
});<br />
</script><br />
<!-- End of mathjax configuration --><br />
</code>
</div>
<br />
And, I added to the custom css under customize -> advanced:<br />
<div class="highlight">
<code>
.highlight{background: #f8f8f8;<br />
overflow:auto;width:auto;border:solid gray;<br />
border-width:.1em .1em .1em .1em;padding:0em .5em;<br />
border-radius: 4dppx; margin:1em 0em 0em 0em;}<br />
<br />
.code_cell{background: #c8c8c8}<br />
.output_stderr{background: #e8a8a8; border-width:.1em .1em .1em .1em;border-radius: 4px; border:solid gray}<br />
<br />
.highlight pre {margin: .2em 0em;}<br />
.output pre {margin: .2em .2em;}<br />
<br />
.highlight .hll { background-color: #ffffcc }<br />
.highlight { background: #f8f8f8; }<br />
.highlight .c { color: #408080; font-style: italic } /* Comment */<br />
.highlight .err { border: 1px solid #FF0000 } /* Error */<br />
.highlight .k { color: #008000; font-weight: bold } /* Keyword */<br />
.highlight .o { color: #666666 } /* Operator */<br />
.highlight .cm { color: #408080; font-style: italic } /* Comment.Multiline */<br />
.highlight .cp { color: #BC7A00 } /* Comment.Preproc */<br />
.highlight .c1 { color: #408080; font-style: italic } /* Comment.Single */<br />
.highlight .cs { color: #408080; font-style: italic } /* Comment.Special */<br />
.highlight .gd { color: #A00000 } /* Generic.Deleted */<br />
.highlight .ge { font-style: italic } /* Generic.Emph */<br />
.highlight .gr { color: #FF0000 } /* Generic.Error */<br />
.highlight .gh { color: #000080; font-weight: bold } /* Generic.Heading */<br />
.highlight .gi { color: #00A000 } /* Generic.Inserted */<br />
.highlight .go { color: #888888 } /* Generic.Output */<br />
.highlight .gp { color: #000080; font-weight: bold } /* Generic.Prompt */<br />
.highlight .gs { font-weight: bold } /* Generic.Strong */<br />
.highlight .gu { color: #800080; font-weight: bold } /* Generic.Subheading */<br />
.highlight .gt { color: #0044DD } /* Generic.Traceback */<br />
.highlight .kc { color: #008000; font-weight: bold } /* Keyword.Constant */<br />
.highlight .kd { color: #008000; font-weight: bold } /* Keyword.Declaration */<br />
.highlight .kn { color: #008000; font-weight: bold } /* Keyword.Namespace */<br />
.highlight .kp { color: #008000 } /* Keyword.Pseudo */<br />
.highlight .kr { color: #008000; font-weight: bold } /* Keyword.Reserved */<br />
.highlight .kt { color: #B00040 } /* Keyword.Type */<br />
.highlight .m { color: #666666 } /* Literal.Number */<br />
.highlight .s { color: #BA2121 } /* Literal.String */<br />
.highlight .na { color: #7D9029 } /* Name.Attribute */<br />
.highlight .nb { color: #008000 } /* Name.Builtin */<br />
.highlight .nc { color: #0000FF; font-weight: bold } /* Name.Class */<br />
.highlight .no { color: #880000 } /* Name.Constant */<br />
.highlight .nd { color: #AA22FF } /* Name.Decorator */<br />
.highlight .ni { color: #999999; font-weight: bold } /* Name.Entity */<br />
.highlight .ne { color: #D2413A; font-weight: bold } /* Name.Exception */<br />
.highlight .nf { color: #0000FF } /* Name.Function */<br />
.highlight .nl { color: #A0A000 } /* Name.Label */<br />
.highlight .nn { color: #0000FF; font-weight: bold } /* Name.Namespace */<br />
.highlight .nt { color: #008000; font-weight: bold } /* Name.Tag */<br />
.highlight .nv { color: #19177C } /* Name.Variable */<br />
.highlight .ow { color: #AA22FF; font-weight: bold } /* Operator.Word */<br />
.highlight .w { color: #bbbbbb } /* Text.Whitespace */<br />
.highlight .mb { color: #666666 } /* Literal.Number.Bin */<br />
.highlight .mf { color: #666666 } /* Literal.Number.Float */<br />
.highlight .mh { color: #666666 } /* Literal.Number.Hex */<br />
.highlight .mi { color: #666666 } /* Literal.Number.Integer */<br />
.highlight .mo { color: #666666 } /* Literal.Number.Oct */<br />
.highlight .sb { color: #BA2121 } /* Literal.String.Backtick */<br />
.highlight .sc { color: #BA2121 } /* Literal.String.Char */<br />
.highlight .sd { color: #BA2121; font-style: italic } /* Literal.String.Doc */<br />
.highlight .s2 { color: #BA2121 } /* Literal.String.Double */<br />
.highlight .se { color: #BB6622; font-weight: bold } /* Literal.String.Escape */<br />
.highlight .sh { color: #BA2121 } /* Literal.String.Heredoc */<br />
.highlight .si { color: #BB6688; font-weight: bold } /* Literal.String.Interpol */<br />
.highlight .sx { color: #008000 } /* Literal.String.Other */<br />
.highlight .sr { color: #BB6688 } /* Literal.String.Regex */<br />
.highlight .s1 { color: #BA2121 } /* Literal.String.Single */<br />
.highlight .ss { color: #19177C } /* Literal.String.Symbol */<br />
.highlight .bp { color: #008000 } /* Name.Builtin.Pseudo */<br />
.highlight .vc { color: #19177C } /* Name.Variable.Class */<br />
.highlight .vg { color: #19177C } /* Name.Variable.Global */<br />
.highlight .vi { color: #19177C } /* Name.Variable.Instance */<br />
.highlight .il { color: #666666 } /* Literal.Number.Integer.Long */<br />
.prompt {display:none;}<br />
<br />
/* IPython split style */<br />
div.splitbox {width:100%; overflow:auto;}<br />
<br />
div.splitbox div.left {<br />
width:48%;<br />
display:inline-block;<br />
float:left;}<br />
div.splitbox div.right {<br />
width:48%;<br />
display:inline-block;<br />
float:right;}<br />
<br />
@media only screen and (max-device-width: 480px) {<br />
div.splitbox div.left {<br />
width:100%;<br />
display:inline-block;<br />
float:left;}<br />
div.splitbox div.right {<br />
width:100%;<br />
display:inline-block;<br />
float:left;}<br />
}<br />
</code>
</div>
<br />
<p>
I had to choose a non-dynamic template to get it to work. I also set the mobile template to custom.
</p>
<p>
To make a post, I simply get the HTML from IPython (print preview, download as HTML, or nbconvert simple), copy the part inside the body tags, and paste it into the HTML view for Blogger.
</p>Anonymoushttp://www.blogger.com/profile/10752018806658508906noreply@blogger.com0tag:blogger.com,1999:blog-8303893118597899654.post-64596577910053777452015-07-07T09:45:00.000-07:002015-07-18T19:45:43.571-07:00Simple Overloading in Python
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>This is intended as an example to demonstrate the use of overloading in object oriented programming. This was written as a Jupyter notebook (aka IPython) in Python 3. To run in Python 2, simply rename the variables that have unicode names, and replace <code>truediv</code> with <code>div</code>.</p>
<p>While there are several nice Python libraries that support uncertainty (for example, the powerful <a href="https://pypi.python.org/pypi/uncertainties/">uncertainties</a> package and the related units and uncertainties package <a href="http://pint.readthedocs.org/en/0.6/">pint</a>), they usually use standard error combination rules. For a beginning physics class, often 'maximum error' combination is used. Here, instead of using a standard deviation based error and using combination rules based on uncorrelated statistical distributions, we assume a simple maximum error and simply add errors.</p>
<p>To implement this, let's build a Python class and use overloading to implement algebraic operations.</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [1]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="kn">import</span> <span class="nn">unittest</span>
<span class="kn">import</span> <span class="nn">math</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>The main rules are listed below.</p>
<p>If $C=A+B$ or $C=A-B$, then
$$
\delta C = \delta A + \delta B. \tag{1}
$$</p>
<p>If $C=AB$ or $C=A/B$, then
$$
\delta C = C_0 \left( \frac{\delta A}{A_0} + \frac{\delta B}{B_0} \right). \tag{2}
$$</p>
<p>Following from that, if $C=A^n$ then
$$
\delta C = A_0^n \left( n \frac{\delta A}{A_0} \right). \tag{3}
$$</p>
</div>
</div>
</div>
<a name='more'></a>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [2]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="k">class</span> <span class="nc">LabUnc</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="nd">@staticmethod</span>
<span class="k">def</span> <span class="nf">combine</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">):</span>
<span class="k">return</span> <span class="n">a</span> <span class="o">+</span> <span class="n">b</span>
<span class="n">rounding_rule</span> <span class="o">=</span> <span class="mi">1</span>
<span class="s">'This is the number to round at for display, lab rule is 1, particle physics uses 3.54'</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">number</span><span class="p">,</span> <span class="n">uncertainty</span><span class="o">=</span><span class="mi">0</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">n</span> <span class="o">=</span> <span class="n">number</span>
<span class="bp">self</span><span class="o">.</span><span class="n">s</span> <span class="o">=</span> <span class="nb">abs</span><span class="p">(</span><span class="n">uncertainty</span><span class="p">)</span>
<span class="nd">@property</span>
<span class="k">def</span> <span class="nf">ndigits</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">v</span> <span class="o">=</span> <span class="n">math</span><span class="o">.</span><span class="n">ceil</span><span class="p">(</span><span class="o">-</span><span class="n">math</span><span class="o">.</span><span class="n">log10</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">s</span><span class="p">)</span> <span class="o">+</span> <span class="n">math</span><span class="o">.</span><span class="n">log10</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">rounding_rule</span><span class="p">))</span>
<span class="k">return</span> <span class="n">v</span> <span class="k">if</span> <span class="n">v</span> <span class="o">></span> <span class="mi">0</span> <span class="k">else</span> <span class="mi">0</span>
<span class="nd">@property</span>
<span class="k">def</span> <span class="nf">max</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">n</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">s</span>
<span class="nd">@property</span>
<span class="k">def</span> <span class="nf">min</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">n</span> <span class="o">-</span> <span class="bp">self</span><span class="o">.</span><span class="n">s</span>
<span class="k">def</span> <span class="nf">__repr__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="s">'{0}({1.n}, {1.s})'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="nb">type</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__name__</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">__str__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="p">(</span><span class="nb">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">n</span><span class="p">,</span><span class="s">'0.'</span><span class="o">+</span><span class="nb">str</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">ndigits</span><span class="p">)</span> <span class="o">+</span> <span class="s">'f'</span><span class="p">)</span>
<span class="o">+</span> <span class="s">' ± '</span> <span class="o">+</span> <span class="nb">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">s</span><span class="p">,</span><span class="s">'0.'</span><span class="o">+</span><span class="nb">str</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">ndigits</span><span class="p">)</span> <span class="o">+</span> <span class="s">'f'</span><span class="p">))</span>
<span class="k">def</span> <span class="nf">_repr_html_</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="nb">str</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">__eq__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span><span class="n">other</span><span class="p">):</span>
<span class="k">return</span> <span class="nb">abs</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">n</span> <span class="o">-</span> <span class="n">other</span><span class="o">.</span><span class="n">n</span><span class="p">)</span><span class="o"><.</span><span class="mi">0000001</span> <span class="ow">and</span> <span class="nb">abs</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">s</span> <span class="o">-</span> <span class="n">other</span><span class="o">.</span><span class="n">s</span><span class="p">)</span><span class="o"><.</span><span class="mi">0000001</span>
<span class="k">def</span> <span class="nf">__add__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">other</span><span class="p">):</span> <span class="c"># (1)</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">__class__</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">n</span><span class="o">+</span><span class="n">other</span><span class="o">.</span><span class="n">n</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">combine</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">s</span><span class="p">,</span> <span class="n">other</span><span class="o">.</span><span class="n">s</span><span class="p">))</span>
<span class="k">def</span> <span class="nf">__sub__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">other</span><span class="p">):</span> <span class="c"># (1)</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">__class__</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">n</span><span class="o">-</span><span class="n">other</span><span class="o">.</span><span class="n">n</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">combine</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">s</span><span class="p">,</span> <span class="n">other</span><span class="o">.</span><span class="n">s</span><span class="p">))</span>
<span class="k">def</span> <span class="nf">__mul__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">other</span><span class="p">):</span> <span class="c"># (2)</span>
<span class="n">C</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">n</span> <span class="o">*</span> <span class="n">other</span><span class="o">.</span><span class="n">n</span>
<span class="n">δC</span> <span class="o">=</span> <span class="n">C</span> <span class="o">*</span> <span class="bp">self</span><span class="o">.</span><span class="n">combine</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">s</span><span class="o">/</span><span class="bp">self</span><span class="o">.</span><span class="n">n</span><span class="p">,</span> <span class="n">other</span><span class="o">.</span><span class="n">s</span><span class="o">/</span><span class="n">other</span><span class="o">.</span><span class="n">n</span><span class="p">)</span>
<span class="k">return</span> <span class="nb">type</span><span class="p">(</span><span class="bp">self</span><span class="p">)(</span><span class="n">C</span><span class="p">,</span> <span class="n">δC</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">__truediv__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">other</span><span class="p">):</span> <span class="c"># (2)</span>
<span class="n">C</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">n</span> <span class="o">/</span> <span class="n">other</span><span class="o">.</span><span class="n">n</span>
<span class="n">δC</span> <span class="o">=</span> <span class="n">C</span> <span class="o">*</span> <span class="bp">self</span><span class="o">.</span><span class="n">combine</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">s</span><span class="o">/</span><span class="bp">self</span><span class="o">.</span><span class="n">n</span><span class="p">,</span> <span class="n">other</span><span class="o">.</span><span class="n">s</span><span class="o">/</span><span class="n">other</span><span class="o">.</span><span class="n">n</span><span class="p">)</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">__class__</span><span class="p">(</span><span class="n">C</span><span class="p">,</span> <span class="n">δC</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">__pow__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">power</span><span class="p">):</span> <span class="c"># (3)</span>
<span class="n">C</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">n</span><span class="o">**</span><span class="n">power</span>
<span class="n">δC</span> <span class="o">=</span> <span class="n">C</span> <span class="o">*</span> <span class="p">(</span><span class="n">power</span> <span class="o">*</span> <span class="bp">self</span><span class="o">.</span><span class="n">s</span><span class="o">/</span><span class="bp">self</span><span class="o">.</span><span class="n">n</span><span class="p">)</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">__class__</span><span class="p">(</span><span class="n">C</span><span class="p">,</span> <span class="n">δC</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Now that we have a (hopefully!) working class, let's put together a quick unittest to see if it does what we expect.</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [3]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="k">class</span> <span class="nc">TestLabUncCombine</span><span class="p">(</span><span class="n">unittest</span><span class="o">.</span><span class="n">TestCase</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">testAdd</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">LabUnc</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span> <span class="o">+</span> <span class="n">LabUnc</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span><span class="mi">2</span><span class="p">),</span> <span class="n">LabUnc</span><span class="p">(</span><span class="mi">20</span><span class="p">,</span><span class="mi">3</span><span class="p">))</span>
<span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">LabUnc</span><span class="p">(</span><span class="mi">20</span><span class="p">,</span><span class="o">.</span><span class="mi">1</span><span class="p">)</span> <span class="o">+</span> <span class="n">LabUnc</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span><span class="o">.</span><span class="mi">1</span><span class="p">),</span> <span class="n">LabUnc</span><span class="p">(</span><span class="mi">30</span><span class="p">,</span><span class="o">.</span><span class="mi">2</span><span class="p">))</span>
<span class="k">def</span> <span class="nf">testSub</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">LabUnc</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span> <span class="o">-</span> <span class="n">LabUnc</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span><span class="mi">2</span><span class="p">),</span> <span class="n">LabUnc</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span><span class="mi">3</span><span class="p">))</span>
<span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">LabUnc</span><span class="p">(</span><span class="mi">20</span><span class="p">,</span><span class="o">.</span><span class="mi">1</span><span class="p">)</span> <span class="o">-</span> <span class="n">LabUnc</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span><span class="o">.</span><span class="mi">1</span><span class="p">),</span> <span class="n">LabUnc</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span><span class="o">.</span><span class="mi">2</span><span class="p">))</span>
<span class="k">def</span> <span class="nf">testMul</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">LabUnc</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span> <span class="o">*</span> <span class="n">LabUnc</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span><span class="mi">2</span><span class="p">),</span> <span class="n">LabUnc</span><span class="p">(</span><span class="mi">100</span><span class="p">,</span><span class="mi">30</span><span class="p">))</span>
<span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">LabUnc</span><span class="p">(</span><span class="mi">20</span><span class="p">,</span><span class="o">.</span><span class="mi">1</span><span class="p">)</span> <span class="o">*</span> <span class="n">LabUnc</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span><span class="o">.</span><span class="mi">1</span><span class="p">),</span> <span class="n">LabUnc</span><span class="p">(</span><span class="mi">200</span><span class="p">,</span><span class="mi">3</span><span class="p">))</span>
<span class="k">def</span> <span class="nf">testTrueDiv</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">LabUnc</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span> <span class="o">/</span> <span class="n">LabUnc</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span><span class="mi">2</span><span class="p">),</span> <span class="n">LabUnc</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="o">.</span><span class="mi">3</span><span class="p">))</span>
<span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">LabUnc</span><span class="p">(</span><span class="mi">20</span><span class="p">,</span><span class="o">.</span><span class="mi">1</span><span class="p">)</span> <span class="o">/</span> <span class="n">LabUnc</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span><span class="o">.</span><span class="mi">1</span><span class="p">),</span> <span class="n">LabUnc</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span><span class="o">.</span><span class="mi">03</span><span class="p">))</span>
<span class="k">def</span> <span class="nf">testPow</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">LabUnc</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">,</span> <span class="n">LabUnc</span><span class="p">(</span><span class="mi">100</span><span class="p">,</span><span class="mi">20</span><span class="p">))</span>
<span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">LabUnc</span><span class="p">(</span><span class="mi">20</span><span class="p">,</span><span class="o">.</span><span class="mi">1</span><span class="p">)</span><span class="o">**</span><span class="mi">3</span><span class="p">,</span> <span class="n">LabUnc</span><span class="p">(</span><span class="mi">8000</span><span class="p">,</span><span class="mi">120</span><span class="p">))</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Since we are putting our TestCase in a IPython notebook, we'll need to run it manually. This would be simpler if it was in a separate testing file.</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [4]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">suite</span> <span class="o">=</span> <span class="n">unittest</span><span class="o">.</span><span class="n">TestLoader</span><span class="p">()</span><span class="o">.</span><span class="n">loadTestsFromTestCase</span><span class="p">(</span><span class="n">TestLabUncCombine</span><span class="p">)</span>
<span class="n">unittest</span><span class="o">.</span><span class="n">TextTestRunner</span><span class="p">(</span><span class="n">verbosity</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">suite</span><span class="p">);</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_stream output_stderr output_text">
<pre>testAdd (__main__.TestLabUncCombine) ... ok
testMul (__main__.TestLabUncCombine) ... ok
testPow (__main__.TestLabUncCombine) ... ok
testSub (__main__.TestLabUncCombine) ... ok
testTrueDiv (__main__.TestLabUncCombine) ... ok
----------------------------------------------------------------------
Ran 5 tests in 0.010s
OK
</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>We can now say that our class works! Now, let's do some example calculations.</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [5]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">A</span> <span class="o">=</span> <span class="n">LabUnc</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="o">.</span><span class="mi">005</span><span class="p">)</span>
<span class="n">B</span> <span class="o">=</span> <span class="n">LabUnc</span><span class="p">(</span><span class="mf">1.11</span><span class="p">,</span> <span class="o">.</span><span class="mi">05</span><span class="p">)</span>
<span class="n">C</span> <span class="o">=</span> <span class="n">LabUnc</span><span class="p">(</span><span class="o">.</span><span class="mi">004</span><span class="p">,</span> <span class="o">.</span><span class="mi">001</span><span class="p">)</span>
<span class="n">D</span> <span class="o">=</span> <span class="n">LabUnc</span><span class="p">(</span><span class="mf">2.02</span><span class="p">,</span> <span class="o">.</span><span class="mi">08</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [6]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="nb">print</span><span class="p">(</span><span class="s">'A × B ='</span><span class="p">,</span> <span class="n">A</span> <span class="o">*</span> <span class="n">B</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s">'A × C ='</span><span class="p">,</span> <span class="n">A</span> <span class="o">*</span> <span class="n">C</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s">'A / B ='</span><span class="p">,</span> <span class="n">A</span> <span class="o">/</span> <span class="n">B</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s">'A / D ='</span><span class="p">,</span> <span class="n">A</span> <span class="o">/</span> <span class="n">D</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s">'A⁵ ='</span><span class="p">,</span> <span class="n">A</span><span class="o">**</span><span class="mi">5</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>A × B = 3.3 ± 0.2
A × C = 0.012 ± 0.003
A / B = 2.7 ± 0.1
A / D = 1.49 ± 0.06
A⁵ = 243 ± 2
</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [7]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">a</span> <span class="o">=</span> <span class="n">LabUnc</span><span class="p">(</span><span class="mi">12</span><span class="p">,</span><span class="mi">2</span><span class="p">)</span>
<span class="n">b</span> <span class="o">=</span> <span class="n">LabUnc</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span><span class="o">.</span><span class="mi">5</span><span class="p">)</span>
<span class="n">c</span> <span class="o">=</span> <span class="n">LabUnc</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span><span class="o">.</span><span class="mi">2</span><span class="p">)</span>
<span class="n">d</span> <span class="o">=</span> <span class="p">(</span><span class="n">a</span><span class="o">/</span><span class="n">b</span><span class="p">)</span> <span class="o">-</span> <span class="n">c</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [8]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="nb">print</span><span class="p">(</span><span class="s">'d₀ ='</span><span class="p">,</span> <span class="n">d</span><span class="o">.</span><span class="n">n</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s">'d_max ='</span><span class="p">,</span> <span class="n">a</span><span class="o">.</span><span class="n">max</span> <span class="o">/</span> <span class="n">b</span><span class="o">.</span><span class="n">min</span> <span class="o">-</span> <span class="n">c</span><span class="o">.</span><span class="n">min</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s">'d_min ='</span><span class="p">,</span> <span class="n">a</span><span class="o">.</span><span class="n">min</span> <span class="o">/</span> <span class="n">b</span><span class="o">.</span><span class="n">max</span> <span class="o">-</span> <span class="n">c</span><span class="o">.</span><span class="n">max</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s">'δd ='</span><span class="p">,</span> <span class="n">d</span><span class="o">.</span><span class="n">s</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>d₀ = 2.0
d_max = 3.8
d_min = 0.657142857142857
δd = 1.5333333333333332
</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [9]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="nb">print</span><span class="p">(</span><span class="s">'d₀ + δd ='</span><span class="p">,</span> <span class="n">d</span><span class="o">.</span><span class="n">max</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s">'d₀ - δd ='</span><span class="p">,</span> <span class="n">d</span><span class="o">.</span><span class="n">min</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_stream output_stdout output_text">
<pre>d₀ + δd = 3.533333333333333
d₀ - δd = 0.4666666666666668
</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Bonus:-inheritance">Bonus: inheritance<a class="anchor-link" href="#Bonus:-inheritance">¶</a></h2><p>As a quick example of inheritance, let's set up the traditional uncertainty method using our class and inheritance. You may have noticed that the previous class was built with a combine method rather than simply adding uncertainties. Let's use the standard deviation rules to replace this combine with the one from traditional error analysis.</p>
<p>If $C=A+B$ or $C=A-B$, then
$$
\delta C = \sqrt{\delta A^2 + \delta B^2}. \tag{1}
$$</p>
<p>If $C=AB$ or $C=A/B$, then
$$
\delta C = C_0 \sqrt{ \left( \frac{\delta A}{A_0}\right)^2 + \left(\frac{\delta B}{B_0}\right)^2 }. \tag{2}
$$</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [10]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="k">class</span> <span class="nc">StdUnc</span><span class="p">(</span><span class="n">LabUnc</span><span class="p">):</span>
<span class="nd">@staticmethod</span>
<span class="k">def</span> <span class="nf">combine</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">):</span>
<span class="k">return</span> <span class="n">math</span><span class="o">.</span><span class="n">sqrt</span><span class="p">(</span><span class="n">a</span><span class="o">**</span><span class="mi">2</span> <span class="o">+</span> <span class="n">b</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Now, we can test this; since there are libraries that handle uncertainty this way, we can compare to those. (I'll be using slightly more advanced testing methods here.) Notice that I used <code>n</code> and <code>s</code> for the value and uncertainty; this was to allow duck typing for the comparison method for the <code>uncertainties</code> library classes.</p>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [11]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="kn">from</span> <span class="nn">uncertainties</span> <span class="k">import</span> <span class="n">ufloat</span>
<span class="kn">import</span> <span class="nn">itertools</span>
<span class="kn">from</span> <span class="nn">operator</span> <span class="k">import</span> <span class="n">add</span><span class="p">,</span> <span class="n">sub</span><span class="p">,</span> <span class="n">mul</span><span class="p">,</span> <span class="n">truediv</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [12]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="k">class</span> <span class="nc">TestStdUncCombine</span><span class="p">(</span><span class="n">unittest</span><span class="o">.</span><span class="n">TestCase</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">setUp</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">cases</span> <span class="o">=</span> <span class="p">((</span><span class="mi">10</span><span class="p">,</span><span class="mi">1</span><span class="p">),(</span><span class="mi">10</span><span class="p">,</span><span class="mi">2</span><span class="p">),(</span><span class="mi">20</span><span class="p">,</span><span class="o">.</span><span class="mi">1</span><span class="p">),(</span><span class="mi">10</span><span class="p">,</span><span class="o">.</span><span class="mi">1</span><span class="p">),(</span><span class="o">.</span><span class="mi">1234</span><span class="p">,</span><span class="o">.</span><span class="mi">02</span><span class="p">))</span>
<span class="bp">self</span><span class="o">.</span><span class="n">pairs</span> <span class="o">=</span> <span class="n">itertools</span><span class="o">.</span><span class="n">permutations</span><span class="p">(</span><span class="n">cases</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">run_operation_on_each</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">operation</span><span class="p">):</span>
<span class="k">for</span> <span class="n">a</span><span class="p">,</span><span class="n">b</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">pairs</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">operation</span><span class="p">(</span><span class="n">StdUnc</span><span class="p">(</span><span class="o">*</span><span class="n">a</span><span class="p">),</span><span class="n">StdUnc</span><span class="p">(</span><span class="o">*</span><span class="n">b</span><span class="p">)),</span>
<span class="n">operation</span><span class="p">(</span><span class="n">ufloat</span><span class="p">(</span><span class="o">*</span><span class="n">a</span><span class="p">),</span><span class="n">ufloat</span><span class="p">(</span><span class="o">*</span><span class="n">b</span><span class="p">)))</span>
<span class="k">def</span> <span class="nf">testAdd</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">run_operation_on_each</span><span class="p">(</span><span class="n">add</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">testSub</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">run_operation_on_each</span><span class="p">(</span><span class="n">sub</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">testMul</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">run_operation_on_each</span><span class="p">(</span><span class="n">mul</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">testTrueDiv</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">run_operation_on_each</span><span class="p">(</span><span class="n">truediv</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In [13]:</div>
<div class="inner_cell">
<div class="input_area">
<div class=" highlight hl-ipython3"><pre><span class="n">suite</span> <span class="o">=</span> <span class="n">unittest</span><span class="o">.</span><span class="n">TestLoader</span><span class="p">()</span><span class="o">.</span><span class="n">loadTestsFromTestCase</span><span class="p">(</span><span class="n">TestStdUncCombine</span><span class="p">)</span>
<span class="n">unittest</span><span class="o">.</span><span class="n">TextTestRunner</span><span class="p">(</span><span class="n">verbosity</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">suite</span><span class="p">);</span>
</pre></div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_stream output_stderr output_text">
<pre>testAdd (__main__.TestStdUncCombine) ... ok
testMul (__main__.TestStdUncCombine) ... ok
testSub (__main__.TestStdUncCombine) ... ok
testTrueDiv (__main__.TestStdUncCombine) ... ok
----------------------------------------------------------------------
Ran 4 tests in 0.049s
OK
</pre>
</div>
</div>
</div>
</div>
</div>Anonymoushttp://www.blogger.com/profile/10752018806658508906noreply@blogger.com0