The early registration deadline is rapidly approaching for SciPy2013, and all our decisions on speakers are out. This year was really tough as we had more submission than ever at a quality that exceeded all my expectations. If you submitted something and haven’t heard from us, let me know.

If you missed the deadline for proposals, I want to elaborate on a number of ways you can still participate. While the material for the official program is closed, we have ample opportunities for more spontaneous contributions for all attendees.

• Join the community
• Speak at Lightning talks
• Hack at the sprints
• Participate in BoFs

Join the community

This year we have a number of opportunities to help get acquainted with learning how to contribute. Starting with two days of beginner tutorials to teach the basic tools of our community and a tutorial on contributing to NumPy, the core of almost all scientific python projects. If you are new to the ecosystem or an old hand needing to know the ends and outs of contribution the tutorials are there to help.

We have teamed up with the Austin PyLadies to present some material on how and why to contribute to open source communities. It’s well known that open source communities can be difficult to navigate and folks are not always polite. We hope to show some perspectives on contributing and what to expect.

Lightning Talks

While we don’t all have a story of printing NumPy arrays to QR codes and scanning them to get around bureaucracy, lightning talks remain one of the most fun parts of the conference. Tell us about your new venture, your craziest hack, your nerdiest tech rant, or whatever you feel best fills your five minutes.

Hack at Sprints

The sprints start Friday and we hope to have a large number of the core projects all come out. In the next week we should have sign ups on the website. Sprints are not only a good time to focus on your project for a few days, it also is a place where the experts in the scientific python will be in the room. Any quick bugs or small details to ask can have a much quicker iteration cycle.

This year we are also encouraging folks to not wait until Friday to start hacking, and I know some of you wouldn’t wait anyways. I’m looking at you Fernando, Jake, Wes, …. But this year we will have about 6 break out rooms just right for 4 to 8 people to brainstorm and hack in between things.

Birds of Feathers

This year we have nine time spaces open for Birds of a Feather sessions. I personally find these session the best to discuss ideas with the conference attendees. We will have a sign up on the web soon as well. Some ideas I have floated include: state of the project BoFs discuss the future of many projects, evolution of the publication in the state of open code and data, and best ways for teaching scientist python.

The post reiterating that the biggest problem in teaching software carpentry (SWC) is installation, got me all in a tissy. I even signed up for their mailing list!

My first impression with SWC was a wonderful community that in encouraging folks to learn computing skills. After witnessing the TeachScheme movement, I was skeptical that two day bootcamps could really have any impact, but then again I’m usually wrong. Now coming off two and a half years of supporting users on supercomputers, I too see the frustration that started SWC in the first place. Scientists waste quite a lot of time not using the tools effectively, but then again I waste too much time futzing with tools. The current frustration is over the view of what education of these tools actually is.

What is a university education?

Some of my best friends in life were made in the classrooms of my university education. While learning the essence of science, math and philosophy, something else was happening as well. I was participating in a community of learning. These friends were the ones who helped me struggle through problems, understand new topics, and motivate my continued education.

While one can go through Bloom’s Taxonomy and be as theoretical as one likes, I have yet to see many people actually discuss the building of a community around your classroom. For most open source developers, this is the major draw, to work on hard problems with a set of intellectually challenging peers. I will say that some tools are being developed to help this, such as Piazza. At the end of the day, MOOCs, bootcamps, and plain-ol-lectures that don’t give students the ability to actively engage an topic on their own terms isn’t worth its salt. Developing communities around learning is another method for continuing the education process.

Perhaps the only thing that a University is providing that other methods of teaching hasn’t touched is this community of learning. To me that’s the biggest problem in alternative university education, there is no community of learning after the day ends.

(On that note I have an unfunded grant proposal to turn textbooks into community initiators. Perhaps I’ll get my act together and pursue it again.)

Make your software useful. That is the point without usefulness there is no need for existence.

If you write a function that is not useful, scrub it from your text editors as not to offend the blessed pixels. Do not fear throwing away non-useful parts of your code, better to cast away these parts rather than sacrifice the usefulness of the whole.

If it is useful to repeat yourself, do so. Repeating yourself in subtly different ways teaches your novices, which must be done well. If there is a common pattern for your task use it, until it is not useful anymore. Write your tests as soon as they will be useful, tests are always useful before giving your code to others. Separate your code into useful partitions. If the partitions are too big or too small the usefulness will be clouded as it is difficult to track the function and interplay between the partitions.

When the program is done, let it stand on its usefulness. If others do not find it useful, seek out more to judge your craft. If none count themselves blessed to have witnessed your skill, throw out the program. Better had the program not been born than to prop it up with false claims of usefulness or many words about its beauty, adherence to principles, or features that none have found useful.

Each programmer must look from within to find the metric of usefulness. Some programs are only useful to the programmer, others must be useful to a multiplicity of programmers, and few hold themselves to such high esteem as being useful to the unwashed non-programming masses. Do not make the program raise malcontent by seeking affection in those it cannot help.

The API can hold back the usefulness of your code. Write many useful programs with the API and minimize their length. Large APIs are weaker than small APIs, but both can be useful. Do not be afraid to erase parts of the API which were only scaffolding to find the better API, the masterpiece could not have been made without the scaffolding, but if it is left standing it will obstruct.

When the performance guru tells you speed is more important, ask them why they don’t make their users write in assembly, why BLAS doesn’t require you to pack your own matrices, and why the CMS doesn’t make you connect to your own sockets. To discourage performance hacks give the programmer five lashes as these hacks are the enemy of abstraction, which can be useful. If the programmer persist in their hacks, let them continue as clearly the speed is useful.

When the enterprise architect recommends yet another abstract factory manager class, ask them why they started programming and at what point they decided to make others despise reading their code. Separate them from the strong, eager, useful programmers and burn their resumes so that others do not confuse their ways as useful.

If the functional zealot tells you your side effects destroy the formal analysis, explain to him that monads rhymes with gonads which are quite useful. Ask them to travel discalced to the far reaches of the empty quarter so they do not harm useful programmers ears. Remember these prophets ramblings as even they are sometimes useful. Teach them to the young, but always remind them of the first rule.

When your sales representative tells you to remove the useful parts of your code, give them wind and ask them to sell the liberated air. When your project manager wants features that are not useful, give them a shovel and ask them to remove dirt from the parched earth only to have them place it back again for they have forgotten the first rule and should be left to sisyphean tasks. When other programmers check in code that is not useful, give them the rule and ask them if they are on the path to manager or guru.

There are two major hurdles to Python disrupting the entire HPC work stack: packaging and dynamic loading. Today I want to discuss the packaging issue. While there are many people working on these hurdles, it is my opinion that the community needs to seek out methods to solve these hurdles together in a satisfactory way. I see this problem taking shape in many different communities, but the HPC version of the problem is probably the most difficult, thus by solving it, we can provide solutions for many other communities as well.

To this end, I helped host a conference call among the young, enthusiastic NumFOCUS community. This call seemed much more of a get to know the problem rather than a listing of solutions. We are working on editting the call and hope to have it published as a InSCIght podcast soon. The call included several companies that produces packaged Python solutions, university folks from around the globe, and industrial users of Python. The interest in the call was so great that we had to switch mediums at the last moment and lost out on some interactions with other great folks. I hope to have another call in the future and a discussion at the upcoming SuperComputing 2012 conference.

What is wrong with Python Packaging

Perhaps the place to start is defining the problem. When you have a piece of code in a pure Python setting, one easily adds the top source directory to PYTHONPATH and happily imports away. This model works quite well usually. For the first years of Google App engine, code could only be installed in this manner.

The issue is that in HPC one must depend upon libraries that are highly tuned for specifics of the architecture where the code is run. These highly tuned libraries often include assembly and usually only include compiled static or shared libraries. Since, HPC often uses specific compilers dedicated to their hardware, these libraries depend on the formats compatible with those compilers. To further enhance the problem, if these libraries are distributed via MPI, then the libraries depend on the MPI implementation (which depends upon the compiler). At TACC we have a hierarchical module system to keep this all straight. You load the compiler, then the MPI implementation, and finally any libraries. If you switch to a different compiler, the entire stack is reloaded. Actually a big part of my job is to make sure these stacks are all compiled in a way that they can be switched out, something that is a feat with biology codes. In a nutshell, we have a huge dependency chain that is binary and machine dependent.

Python isn’t supporting this long list of dependencies very well. For HPCers this is not new as we are use to having to hack build and package systems. In fact in the eight years I’ve been around the FEniCS community, I’ve seen four package systems come and go. It took me three weeks to get the Trilinos code installed on my system back in 2006, at the time folks opined that picking that the right Trilinos configure was NP-Hard. It is much better today. In both cases though, CMake has turned out to be the tool that has stabilized everything.

Why can’t Python support this jumble of dependencies? The reasons are many, but as David Cournapeau has pointed out, Python fundamentally mixes both build and packaging together in a way that makes it impossibly hard. I don’t know David’s history, but he seems pretty critical of the community about not understanding the issue. Rather than rehash all the arguments, I’m going to outline my view of the solution, separating build and packaging.

The build process

The difficult part of the build process is not calling the compiler, rather configuring the environment and compiler flags to produce the desired binary.

To give a simple example, if you are compiling a threaded library, say using OpenMP, you need to know that gcc uses -fopenmp, icc uses -openmp, pgi -mp, something else on xl, and clang doesn’t even support it. Thank you Apple for making my job that much harder by using a default compiler that doesn’t support OpenMP.

Okay that’s actually not hard to code, but it gets better. Because multicore threading is still an infant technology, if you compose several threaded libraries you can see a real performance degradation (despite @dbeaz thinking threading is easy). Thus most vendors provide a threaded version of BLAS and a sequential version, the idea being if you have a big matrix to work with make BLAS be the multithreaded portion of the code, but if you have lots of smaller matrices let the application be multithreaded. This means that each application could be built with 4 compilers X 2 BLAS threading modes, we also need to throw in debugging and optimized modes as yes -O3 versus -g -O0 really do matter.

In this trivial example, we already need to test 16 different builds of a library, not to mention figuring out how many threads to run (which is often a runtime decision with OpenMP). If you want to drive your graduate students to madness get them to install Scalapack on your cluster.

At this point, you might say “16 versions of the library, quit your crying, baby!”. It should be easy to script except Python distribution tools really don’t give a way to manage these different compiler flags. In many cases the tools just slaps your code in some place on the path and runs all sorts of bootstrapping patching hacks to install at all. Virtualenv give a bit of sanity as one can isolate different Python builds, but it doesn’t handle any of the system libraries at all. It is my understanding that when EPD was built, they initially tried to fix this problem but quickly switched to a single binary only distribution model.

To tackle all these burdens, we need to start using real configure and build tools. They need to work on all platforms, yes Windows matters in HPC. They need to be free and open source. And they actually already exist, the community just needs to start adopting them and that will take some effort. The two tools I’ve seen used well are Bento produced by David and CMake produced by Kitware. I’m not going to have a bake-off on the pros and cons of each, I have far more experience with CMake but people I trust tell me Bento is far easier to use. If NumFOCUS and the PSF really wants to help the community today, it would start helping people augment distutils to use one of these tools.

The package selection process

Now that we’ve discussed the build process, we need to package up those builds and allow developers to use them. I was really glad the Yaroslav, one of the Debian developers, and Samuel, homebrew’s Python guy, was able to join us on the call. Open source platforms have created great packaging tools, but it only gets us code monkeys about halfway. I can install the basic libraries that aren’t cutting edge and don’t need fine grain configuration system wide. Then I just keep the jumbled mess in my developer space. But what if a developer needs to fix a bug with one of those system installed libraries that isn’t the version installed? I’ve spend more time futzing with the packaging system installer than fixing the bug.

The truth of the matter is we really want the jumbled mess to be managed better too. Just like on my supercomputer, I can issue a single command and have a whole new compiler stack working, why can’t I do that with my current development tree. Here enters HashDist. HashDist is Dag and Ondrej’s plan for fixing this problem. The idea is to build libraries and stash them some place, recording the necessary details in a small distribution file that is hashed and put in a database store. Now when you need to build different versions of a library, one can simply refer to the different hashes of the other built libraries. At this point, HashDist is vaporware that has some funding. I have been working with these guys trying to fund this project for about a year and a half.

Let me be frank. If this tool existed, reporting bugs could come with a hashdist number that would set your machine up in the bug state immediately, i.e., no more futzing with installers just working. For HPC systems this would be huge. Hell it is even a good idea for a SAAS startup company.

This is very similar to the module system that we use at TACC, lmod, but it should build into the development cycle not just running a supercomputer. For what its worth, supercomputers use module systems because when you have 5000 users on a machine, you have to keep the kids from stepping on each others toes.

Why this affects developers outside of HPC

Hopefully, I’ve explained the problem well enough and pointed to a few promising fixes. While these problems are accentuated in HPC systems, they exist everywhere. Every single company, I have consulted with has this problem. They solve it different ways, but they all manage their own build and packaging system. Every single open source scientific Python team, has this problems. They usually solve it the old fashioned way, indentured servants, err graduate students. But perhaps more importantly, as PyPy and other python distributions threaten CPython for popularity, even Python core is having these problems. The wheel proposals are a good first step, but its gonna take more than changing package names to fix it.

NumFOCUS was founded to help encourage businesses and institutions to put funding together to solve science software problems. This is a big problem and one that prevents a large number of people from effectively using our ecosystem. I would like to see more funds and community resources put forward to create working solutions. Finally a homework for everyone who has read this far. I invite you to send me your versions of how you fix Python packaging for your work so I can collect the best ideas.

For SciPy2012 I was given the privilege of serving as the co-chair of the program committee. The experience was exciting and rewarding as I was able to affect what I view as one of the most important conferences for the scientific world. Now, that’s a pretty bold statement and I’m not going to dwell on it long here, but essentially many conferences people go to show off; SciPy is a conference where people come to learn best practices. And when young scientists are able to see science rock stars like Joshua Bloom lift up the covers and show off how his scientific process works, it changes their entire workflow creating more efficient, better scientists. With that said, I would like to focus more on my opinions about the state of the conference, its successes and failures, and how I believe the conference should be transformed.

The state of the SciPy Community

It was a bit strange being the co-chair of the program committee at SciPy2012, because prior to 2012 I had only attended one SciPy conference. Of course, I have known about the conference for years and have tried to go many times but life has always gotten in the way. Additionally, I came to the scientific Python community after most everything had evolved to a pretty high level of quality. Sure there are bits and pieces that could use some polish, but largely I feel that my contributions have all been cosmetic (although the GSOC students I have mentored have done phenomenal work). This puts me at a bit of a disadvantage to extol any long history of the SciPy community, but I’m sure someone will let me know where I get things right and wrong.

There are some interesting divides in the community. For example, SciPy can refer to many different things:

1. The US SciPy conference,
2. The international SciPy conference series,
3. The Python package named scipy,
4. The basic scientific computing stack used by most pythonistas, and/or
5. Any piece of code slightly related to science and Python

For this article, when I say SciPy I am talking about the US SciPy conference (since if I usually can’t afford to make it to SciPy in Austin, TX, I sure can’t afford to go to Brussels or India) and a mix of the other topics. SciPy was originally a set of friends who got together in a room at Caltech to chat and hack about the scipy package. They use to have these talks from the core devs where each gave a summary of the package and largely discuss technical issues around these packages. As of late, the conference has turned into much more of a place where any scientist or technical computing professional can go and hear about a very wide variety of topics on science and Python.

This change in community is both a good and a bad thing. It means that at coffee, there are a lot of strange faces and let’s be honest if we were the most social crowd we wouldn’t have spent a non-trivial amount of our lives pouring over code and mathematics. Second we seem to have lost some of the core devs attention. For example, in the months preceding the conference the numpy mailing list had turned into a pretty nasty flamewar. At the conference, I only saw one of the major contributors to the numpy code base. I find the fact that SciPy isn’t the most important event on the calendar for these devs kind of surprising. After all this is where you core devs are the gurus everyone has come to be inspired by.

The good news is that we are able to see a more accurate view of the penetration of Python in science. But not only science, pretty much all technical computing areas of industry, academia, and government were represented. The conference had a record number of attendees and submissions, in fact this was the first SciPy that we knew of where people received rejections.

Successes and Failures of SciPy 2012

In many ways SciPy is a great conference, but in other ways it is lacking. I want to explore a few of these ideas, get feedback from the community, and put the consensus into action.

Success

If it ain’t broke don’t fix it ~ Anonymous

Topics

The past two years, there have been two special topics. Last year was Big Data and Core Issues; this year we did High Performance Computing and Visualization. The truth was that we picked these topics because a few motivated folks were excited about them. This is true about the domain session this year as well. Lauren and I chatted, I gave names and she went calling people down to get the right set of talks. Largely I think this works, but I don’t like it.

I don’t like it because the community wasn’t really involved and when you have a couple of people picking the overall direction of a community, it largely becomes a clique. If we want to keep a vibrant community, I think these topics are really important as they are aimed at bringing in people from outside our community.

With that said, my proposal this year would be:

• Machine Learning, and
• Geographic Information Systems

Both topics that came out during the surveys and are a broad cutting technologies. Machine Learning should be a no-brainer as the Big Data world needs it more than ever, but GIS is a bit harder for me to predict being a break out success. Of course last year I thought HPC would be the small topic and Vis a bigger draw, but I was dead wrong. We could of had the whole conference on PyHPC (if you want sort of thing, check out the SuperComputing Conference pyHPC workshop).

Fabulous plenaries

This year I believe the plenaries were really off the chart. I was actually recruited to help out after the plenaries had already been chosen, so this was really Stéfan and Warren’s doing. I really like the format as well. A talk about a well-known package, a science talk about the things people are able to accomplish with the tools, and a look into industry and how our community relates.

I have quite a few ideas about how to replicate this formula, but I believe we should probably have some discussion from the community.

Poster session

The poster session was great. There was a lot of interaction between people at the session. I’m somewhat beside myself on the session though. I thought it would be better during the reception, but we couldn’t get the posters in the reception room and be visible during the entire conference. I really liked the poster session and hope to see it in the future. One fact of science conferences is that usually it is harder to get funding if you don’t have something to present. Posters are a great way to increase participation, although I do wonder if we need to go to three parallel tracks in the future.

Domain science symposiums

The symposiums at the end of the day were much sparser than the day. This was expected, but they play an important role. As the main session includes a large number of folks that have quite different domains, these sessions allowed for topic specifics to be discussed. More and more domain conferences are including sessions on Python, but SciPy allows for sessions to grab the attention of Python experts who are often able to give fresh advice.

Failures

Develop success from failures. Discouragement and failure are two of the surest stepping stones to success. ~ Dale Carnegie

Failure is probably too harsh a word for what I discuss in this section. It’s more adequately, “Things that disappointed Andy.” And as most people who know me know, I’m not afraid to tell you why I’m disappointed (only I usually use harsher words). At any rate, I believe in admitting failure and want to discuss a few things I would like to rectify.

Community involvement

For the most part, the conference is put on by about four people. Sure there is a longer list on the website, but the brunt of the work falls on a few. I would like to see this change. For example, look at how PyCon is run. There are more volunteers than you can count. While Enthought is an amazing institutional sponsor of the event, they only have a small number of resources to put into the event. My impression is that these resources are over exhausted.

It is time we as a community come together to setup structures for more community participation. For example, the websites need work, the process of reviewing papers needs work, there needs to be more logistical help at the meeting, and so on. My guess is that most people don’t even know that they could be helping. To even start though we need to build a volunteer network and appropriate web presence to give volunteers things to do.

Diversity

Matt Davis brought this up on Twitter, but I want to echo it louder. There was a discouraging lack of diversity at SciPy. I believe there were 100 males to every female and very few minorities. When talking to a few people about this, the general view was that there just aren’t women coders around. I think this is really wrong, I think that women coders either don’t know about SciPy or don’t care.

Which brings up how do you attract female coders? One metric given to me is that conferences with women plenary speakers and/or women on the executive staff have 50% more women participants. I look at the places we advertised and we didn’t focus on any female or minority organizations. Closing this gap will only make our community more vibrant.

Industry/Academia interactions

SciPy is a place where there are a few industrial partners but there could be many more. The problem I saw on the program committee was that anyone who submitted work on closed source software were negatively impacted. We are an open source community but hundreds (thousands?) of companies use our tools. Industrial partners that come back to the community and talk about how to interact more can only be a good thing. For one it will give us a since of the impact that SciPy has beyond the halls of academia. It will heighten our profile among young developers as they are able to come to SciPy to meet potential employers.

My suggestion here would be to have vendor booths and a recruitment event. I would even go as far to suggest hosting a few sponsor speaking sessions. Open source is often supported by industry, just as the scipy stack has been supported by Enthought for years. Inviting industry to our meeting will make it more visible.

Social Media

The lack of social media at the conference is quite surprising. Are we all old fuddies who don’t know how to use social media to discuss? It is quite exciting to go to a conference where the social media around the conference is working well. It allows for more dialog among the participants and thus a greater shared experience.

With that said, I would like to see a greater use of Twitter, G+ and Facebook throughout the event, including planning. Connecting our community may even heal all those fights we have on mailing lists.

Lack of Core Developers

I’ve already touched on this, but one last failure is the lack of core project developers at the conference. Sure people have lots competing with their time, but this is the conference dedicated to the code we spend so much free time on. Conversations at SciPy are still having an effect on the directions of codes (in fact today scipy.users had a long conversation about the scipy domain which was hotly discussed at the sprints).

The few ideas I have on bringing the coders back include:

1. More sprinting (space every night all night)
2. Sessions dedicated to core projects
3. More organization around BoFs
4. Conference sponsorship for devs to attend conference

Vision

Okay 2000 words in and I’m sure you are getting bored. But I’ve said this before and I’ll say it again. There is no reason SciPy shouldn’t be as large as PyCon. While there are certainly more users of the Python language, our community is always one of the featured groups when Python users are discussed. The suggestions I outlined above are only a small set of comments I really have, but they are the ones I feel probably need a broader discussion.

For me the vision of SciPy should be simple. SciPy is the fuel of the scientific Python movement. It is where producers and consumers of scientific and technical code come to learn from each other and make each other better scientists and coders. SciPy as a conference should be inclusive of all technical disciplines and help mentor novice programmers. Finally, it should be the place where we inject new life into our projects, hack the wild ideas we only dream about, and submit pull requests that change the face of science.

Getting to know Python is a pretty hairy task, as is diving into any language. Fortunately Python is considered an easy language to learn and the potential for using it in HPC settings is pretty large. Below I give an introduction to learning, speeding, and scaling Python. Each topic can be the subject of its own book, so take these talking points and notes as only hints to what one can look at. For a very detailed view of this topic for scientists I recommend the following three books:

• A Primer on Scientific Programming with Python
• Python Scripting for Computational Science
• Python for Data Analysis

Learning Python

There are a very large number of resources for learning the Python language. The issue of course is finding a resource that is attractive to the correct mindset. Very often something that a programmer is able to learn from has little resonance with an artist. For this reason, I am going to list off a couple of places that I find particularly good, but of course I have been programming for over a decade.

While learning the language is important, the real value of Python is the numerous libraries and the community around the tools. With that said finding what your particular community is doing is pretty important, but there are a core set of tools that are used by most Pythonistas. I list off a few of these tools for your consumption.

Also, because we like to see what people are doing with Python, I list off a few shining examples of Python in the wide world of scientific computing. I have to admit that there are a huge number of possibilities but few in the realm of HPC. At the very least, a HPC person should know Python for its scripting abilities but as the other sections of this document underscore, Python is a good candidate for HPC codes as well.

Python Tutorials

• Python Doc Tutorial: This is the tutorial written by the developers of Python, best for programmers.
• Think Python: A book for non-programmers.
• SciPy Lectures: A series of lectures written by scientists geared towards scientists.
• Py4Science Starter Kit: A version of this page written in 2008. Lots of good material but a bit less selective.

Python Tools

• Anaconda Pro: A distribution of common Python packages for Big Data and Cluster programming. Includes disco (a Python map-reduce) and mpi4py which is of more interest to HPCers.
• Enthought Python Distribution: A distribution of the most commonly used tools by the community (free for academics).
• NumPy:
• SciPy: A collection of scientific libraries.
• MatPlotLib: A highly customizable 2D plotting library
• IPython: An interactive Python shell and parallel code manager. The IPython notebook has become very popular and allows users to use an interface similar to Mathematica on a supercomputer.

Python Stories

• SciPy Conferences: The series of conferences associated with the scientific Python community see recent videos at PyVideo.
• Python in Astronomy: Joshua Bloom from UC Berkeley gave a keynote talk at SciPy 2012 on “Python as Super Glue for the Modern Scientific Workflow”
• NumFocus User Stories: A foundation for scientific computing tools with a growing number of user stories.
• PyCLAW: A petascale application written in Python

Performance

Despite all the great features outlined above, the (mis)perception is that Python is too slow for HPC Development. While it is true that Python might not be the best language to write your tight loop and expect a high percentage of peak flop rate, it turns out that Python has a number of tools to help switch to those lower-level languages.

To discuss performance I outline three sets of tools: profiling, speeding up the Python code via C/Fortran, and speeding up Python via Python. It is my view that Python has some of the best tools for looking at what your code’s performance is then drilling down to the actual bottle necks. Speeding up code without profiling is about like trying to kill a deer with an uzi.

Python tools for profiling

• profile and cProfile modules: These modules will give you your standard run time analysis and function call stack. It is pretty nice to save their statistics and using the pstats module you can look at the data in a number of ways.
• kernprof: This tool puts together many routines for doing things like line by line code timing
• memory_profiler: This tool produces line by line memory foot print of your code.
• IPython timers: The timeit function is quite nice for seeing the differences in functions in a quick interactive way.

Speeding up Python

• Cython: Cython is the quickest way to take a few functions in Python and get faster code. You can decorate the function with the cython variant of Python and it generates c code. This is very maintainable and can also link to other hand written code in c/c++/fortran quite easily. It is by far the preferred tool today.
• ctypes: Ctypes will allow you to write your functions in c and then wrap them quickly with its simple decoration of the code. It handles all the pain of casting from PyObjects and managing the gil to call the c function.
• FWrap and f2py: Fortran wrapping tools. FWrap is a newer tool for F90 and integrates with the rest of cython, and f2py is the older more standard tool. Unfortunately, neither of these libraries are well maintained but are used in numerous projects.

Other approaches exist for writing your code in C but they are all somewhat more for taking a C/C++ library and wrapping it in Python.

Python-only approaches

If you want to stay inside Python mostly, my advice is to figure out what data you are using and picking correct data types for implementing your algorithms. It has been my experience that you will usually get much farther by optimizing your data structures then any low level c hack. For example:

• numpy: A contiguous array very fast for strided operations of arrays
• numexpr: A numpy array expression optimizer. It allows for multithreading numpy array expressions and also gets rid of the numerous temporaries numpy makes because of restrictions of the Python interpreter.
• blist: A b-tree implementation of a list, very fast for inserting, indexing, and moving the internal nodes of a list
• pandas: Data frames (or tables) very fast analytics on the arrays.
• pytables: Fast structured hierarchical tables (like hdf5), especially good for out of core calculations and queries to large data.

Scaling Python

Right now there are a few distributed Python tools but the list is growing rapidly. Here I just give a list the tools and some domain tools that are used in HPC that provide a Python interface.

Distributed computing libraries

• mpi4py: Fastest most complete mpi Python wrapper.
• disco: Python Hadoop-like framework.
• IPython Parallel: A mpi or zero-mq based parallel Python.
• pathos: Framework for heterogeneous computing
• Global Arrays: A shared memory interface to distributed computing

Domain specific libraries

• petsc4py: Python bindings for PETSc, the Portable, Extensible Toolkit for Scientific Computation.
• slepc4py: Python bindings for SLEPc, the Scalable Library for Eigenvalue Problem Computations.
• tao4py: Python bindings for TAO, the Toolkit for Advanced Optimization.
• pyTrilinos: Trilinos wrappers

---

Changelog

2012-09-27 Andy R. Terrel

* Initial version

2012-09-30 Andy R. Terrel

* Added links to Fernando Perez's 2008 version of this document
* Added link to Anaconda Pro
* Added link to Gael Varoquaux's scipy lectures
* Added link to Jeff Daily's Global Array Python bindings
* Added link to FWrap and f2py
* Fixed some grammatical mistakes