Codematician

NumFOCUS 2025: A Year of Transition and Renewal

2026-01-12T00:00:00-08:00

When I rejoined the NumFOCUS Admin Board in late 2024, I knew we had significant work ahead. What I didn't fully anticipate was just how transformative 2025 would become. This was a year defined by leadership transitions, governance improvements, and laying the groundwork for NumFOCUS's next chapter. As Secretary this year, I had a front-row seat to all of it.

Leadership Transition

The biggest story of 2025 was our executive leadership transition. Leah Silen, our founding Executive Director who had been with NumFOCUS since 2012, announced her departure. Leah built this organization from nothing into an entity that now supports nearly 100 scientific software projects. The community owes her an enormous debt of gratitude for her tireless work.

Our first order of business in January was establishing a hiring committee. I proposed the resolution and Julie Hollek took the lead as committee chair. We engaged Perrett-Laver as our search firm and cast a wide net. Jim Weiss stepped up as interim ED in March, keeping operations steady while we conducted the search. By July, the board accepted the hiring committee's recommendation, and Rachel Kerestes joined us as our new Executive Director by September.

Rachel hit the ground running. By December, she was already auditing internal policies, working with legal specialists on contracting procedures, and planning strategic sessions for 2026. The transition was smoother than I had dared hope.

NumFOCUS Governance and Transparency

One resolution I'm particularly proud of came in our very first meeting of 2025: we voted to make board minutes public by default. Previously, minutes took months to publish and required approval from a quorum. Under the new policy, minutes go public unless a board member specifically moves to redact sensitive information. You can see all our 2025 minutes at numfocus.github.io/numfocus-board. This aligns with the community standards of transparency that our open source projects embody.

We also modernized our bylaws to address board member accountability. The new amendment codifies that directors who miss three consecutive meetings without valid excuse may be deemed to have resigned. This wasn't theoretical—we had to apply it during the year when a board member became unreachable.

Financial Health

The 2025 audit came back clean. In the auditor's words, our financial statements and practices are in good order, there's no evidence of fraud or theft, and we can be trusted with people's money. We couldn't have hoped for a better outcome.

The board also approved a comprehensive Financial Policies and Procedures Manual in June, bringing more rigor to our operations. We approved the 2026 budget in December and established quarterly financial reporting to keep the board informed throughout the year.

Key Resolutions

Looking back at the year's board resolutions tells the story of our priorities:

January: Public minutes by default, ED hiring committee established
March: Bylaws amendment for attendance, Jim Weiss appointed interim ED
April: $10K budget for PyCon booth presence
June: Financial Policies Manual approved
July: New ED hire accepted, PyData Global established as a committee, Small Development Grant fund restrictions formalized
September: EU Cooperation Agreement with NumFOCUS France
December: 2025 election results certified (welcoming James, Francesca, and Noor), 2026 budget approved

Looking Ahead

As I write this, we're onboarding three new board members and kicking off a strategic planning process. Rachel is recruiting a Chief Financial and Operations Officer. We're moving to monthly board meetings instead of the quarterly schedule I was skeptical about last year—turns out my instincts were right on that one.

The Technical Steering Committee is finalizing its charter. Projects are pushing back on some process changes, which I actually see as a healthy sign that they care about how NumFOCUS operates. We're also exploring domain-specific events in areas like biotech, climate, and AI as smaller, more focused gatherings.

2025 was a year of foundations. We navigated a major leadership change, improved our governance, and maintained financial health. The work continues, but I feel the organization is pointed in the right direction. NumFOCUS exists to help scientists build better tools for discovery, and that mission remains as vital as ever.

The new NumFOCUS Admin Board

2025-01-10T00:00:00-08:00

The new NumFOCUS admin board met for the first time this week and I can say that I am truly honored to serve the community in this capacity. NumFOCUS has been a leading example of how to build open source culture and provide services to our scientific tool makers. With almost 100 scientific software projects representing fields from Astrophysics, Biology, Economics, Engineering, Statistics, Medical Imaging, Data Science, Computer Science, and so on, I feel the impact of our work will be felt for generations. With all that said, the admin board has some important work to get done this term and we set forth doing it right away.

First and formost, in keeping with the value of Transparency, the admin board’s first resolution was to make the notes become default public. This allows us to post minutes much faster than previous boards which required approval from a quorum to post minutes. You can see the board minutes at our site. Under the new resolution, a board member will have to propose a discussion item be voted private to be removed. I believe this was an important step to transition to steer the organization to a more radically transparent operations. While NumFOCUS has won awards for transparency from non-profit watchdogs, many in the community have asked for more visibility in the discussions of these board meetings. Our community standards of transparency have evolved in the decade we have been operating and this is an opportunity to re-evaluate all NumFOCUS practices.

Perhaps the second most important action is a resolution to create a hiring committee for our Executive Director (ED) position. As has been announced in the previous board term, our faithful founding ED, Leah Silen, has stepped down. She will continue to serve in this role for a few more months, which means we need to hire ASAP. Leah has been such an inspiring leader for the community and will leave a huge impression for the community going forward. The community owes Leah a debt of graditude for her hard work.

Finally our last resolution was to install a temporary set of officers which I was nominated for the secretary position. We focused on temporary because of two reasons. First only 4 of the 6 members of the current board were able to attend this special session. Second, under the new structure, the technical steering committee has the right to appoint a voting admin board member, but that has not happened yet. Considering we only have a few months to start a hiring committee, find a search firm, interview candidates, and bring the community to a consensus on the new ED, I certainly felt we should get a full set of officers now.

Well it was a productive meeting and I am super excited to be along for the journey. I can say this admin board feels charged to help the community and keep building the future that NumFOCUS promises, i.e. more open scientific tools for better scientific progress.

Things to do in 2025

2025-01-05T00:00:00-08:00

I seem to be waking up with purpose every morning so I felt it good to get a list of things I would like to do in 2025. Yeah this list is too long but gives my future self a direction.

Daily General Life

The heavens declare the glory; the skies proclaim the work.

Serve my family,
Read a holy scripture,
Find meaning in my work, and
Exercise.

OSS Software

The wicked borrow and do not repay, but the righteous give generously.

Find a way to commit code every day
Launch a pip prototype with gpu aware metadata semantics, follow at wheel-next
Build Pure Python versions of AI algorithms that perform e.g. llm.py
Explore the boundaries between languages and libraries for gpu computing. See numba-cuda and triton-lang
Contribute to my favorite libraries and tools:
- numba
- CPython
- pixi
- dask
- bokeh
- DISCOVER-Cookbook
- LFortran/LPython
Help raise the standards of governance in the PyData projects and the NumFOCUS projects
Help guide the NumFOCUS community hire a new executive director

Hobbies

Satisfy your desires with good things so that your youth is renewed like the eagle’s

Redesign my website and tinker with a few applications
Complete my woodworking todo list
- bunk beds for kids
- end grain cutting boards and pens for gifts
- handtool chest from hand tool school
Read 12 books (currently on the list, need suggestions for non-western male authors):
- Life by Pope Francis
- The Godfather by Mario Puzo
- The intelligent investor by Benjamin Graham
- Something by Ted Chiang
- Stormlight Archive book 5 (Wind and Truth) by Brandon Sanderson
Set up a family wildlife preserve

Photo attribution

By Bradaen Collum. Unsplash License via Unsplash\

Looking back at 2024

2025-01-04T00:00:00-08:00

You accomplish less in one year than you want more more in 10 than you ever expect.

These words of wisdom seem attributed to Bill Gates. This year of any, I started to see their meaning. The past year represented a pretty stark departure from the last decade of my life. It is a return to helping build tools for computation. A return to my life’s work in academia, before started working on AI based startup companies. Let’s just say I’ve had some catching up to do, but having people quote my papers in meetings was somewhat surreal.

To back up a minute, in 2023, I joined NVIDIA as the product lead for CUDA Python. I left my academic post back in 2012 to join the founding team at Anaconda. In academia I was building compilers for scientific computing applications, e.g. ways to build hurricane simulations with hundreds of lines of code over millions using GPUs and other accelerators. I was literally told that my work had a “snowball’s chance in hell” at getting me a tenure track position, so I left. At Anaconda, I kept working with academics to produce better big data tooling for the Python data ecosystem. I was the PI on a few DARPA grants that funded some of our beloved PyData tools, e.g. conda, dask, numba, and bokeh. Making tools for academics and government was nice, but had the same feel of my previous work, i.e. not making significant impact.

From 2015 to 2023, I helped organizations use computing tools to build new and interesting technologies. This included virtual clothing sizing tools at Boldmetrics ¹, a fully online real estate system at REX Homes ², a cloud manufacturing system at Xometry ³, and an early film and television script evaluation system at StoryFit ⁴. I also kept helping with NumFOCUS which was a vestige of the last decade of work, i.e. building tools for scientists, but one that I felt uniquely suited to do. I did also run a side gig for CTO advisement for teams wear I helped folks like KindHealth build modern Health Insurance, Korbit.ai build AI education systems, and Sapling Learning build better math education. All this work had immediate impact to customers.

Near the end of 2023, a friend reached out about joining NVIDIA, right when I was looking at joining another startup. It represented an opportunity to take back a decade of learning to building tools for computation. I jumped at the opportunity even though it meant going from an executive role to an individual contributor. One year in it has probably been one of the most productive years of my life, but to see all that you need to come meet me at this year’s GTC where we are launching numerous Python products ⁵.

In 2010-2015, it was a different era of computing. Big data was the main hurdle to technical innovation at the largest companies. Python had already taken off as a great language for gluing and building computational pipelines, but during this time we sealed it as the data scientists preferred toolbox. Now we are focused on Generative AI systems and many of the same challenges exist. The community is more willing to accept compilers like numba and triton-lang. In the big data world, only the largest systems really needed to compile their computational kernels to something that would run 10-100x faster over the entire dataset, but in GenAI there is no small or medium datasets. Thus today we have the big data tools and compilers we wrote back in those early days that have enabled GenAI to move quickly and build just amazing models.

At NVIDIA, I’ve been pushing on numerous initiatives that combine the decade of learning in startups with the new computational of the GenAI stack. It is work I would not have been able to do without the previous decades of work. If there were any words I could whisper in my ears back in 2010, it would be these we started with.

https://boldmetrics.com/ - I drank from the fire hose building this system and deploying it at Men’s Wearhouse as their online fit tool ↩
REX Real Estate - We took the company all the way to an IPO when COVID hit and the bottom of the real estate market fell out. We created a lot of great tech that made it possible to launch in around 30 major cities. ↩
https://www.xometry.com/ - I joined to build the seller marketplace but it was shut down the first quarter I was there. Helped build out a growth team and streamline the processes of the data and algorithms teams. Probably the best demonstration of a company transition that I saw where romans and greek culture clashing. Great product and wonderful operations culture, definitely recommend anyone wanting to get into AI operational systems. ↩
StoryFit - Probably one of my most fun roles as working with creative entertainment folks was most definitely a left turn for me. The research team had thousands of models and we were able to turn it into a self service platform. Ultimately the market was in turmoil as during this period I saw the writer’s strike put all Hollywood on a standstill, the chatGPT moment that shocked the tech world, and the failure of Silcon Valley Bank that wiped out debt equity loans for many startups, including ours. ↩
If you are a company using Python and GPUs definitely contact me. I do have the 20% coupon I can give you but more importantly, I would love to understand your use case and see if there is a match with any of the things we are launching. ↩

When should your board meet

2024-12-28T00:00:00-08:00

Seems the first duty as a new director at NumFOCUS is to determine when we should be meeting. Simple question deserves a blog post with far too much reflection, but hey I promised more transparency in decision making to numerous NumFOCUS constituents. In my experience through many boards at NumFOCUS, sporting non-profit boards, property boards, and corporate boards, I can confidently say IT DEPENDS!

Fine, Andy, WHAT does it depend on?

Why Board Meet

I posit there are really only three reasons or actions for which a boards meet:

A decision needs to be made,
A status report needs to be filed,
A collaboration to accomplish a concrete task, i.e. drafting a position statement.

Now each of these reasons have other dimensions that govern their meeting frequency, time, and place. These include, but not limited to, some of the following:

The deadline for the needed action, i.e. decision, report, or collaboration,
The participants needed for the action
The prerequisites for the action, e.g. the report is finialized for a status report, a discussion has finished for a decision, or items purchased for a collaboration.

Okay seems simple we can use the needed action and the governing dimensions to determine if a meeting should happen. EXCEPT!!!! Different boards have different types of collaborations and different cadences for delivery. Now we could give some pretty generic advice, like meet every day or only when absolutely necessary, but I have been stung in the past by generic advice ¹. Let’s explore different cadences in reference to the various goals of the board.

When do boards meet?

Every single board I have ever been a part of has an oversight role for something, that is usually defined by some charter. For example the NumFOCUS infrastructure board has to provide oversight for IT decisions and my youth baseball league has to provide oversite for baseball games. IT decisions happen almost all the time, but in reality NumFOCUS uses managed services for most everything and so there is not a huge rush warranting a daily meeting. Baseball games happen every blessed hour (M-F 5pm-10 and weekends at 8am - 10pm) during the season, and that makes the occasion for oversight very frequent. Okay the frequency of your deadlines will determine a natural cadence, but the second dimension comes into play, i.e. who is needed for the decision.

Not every single board member will be committed to meet every day, week, or even month. If your deadlines are coming more quickly than your decision points, there will be frustration from your members and many will question your goals. To make decisions faster you need to pair down the number of people needed to make the decision. For example the frequency of youth baseball games has a warranted two boards (executive and full) with board members on duty who can determine when to call an executive meeting at any moment². Most software teams have deliverables every single day and will need to meet almost daily to even get code out the door later than intended. Their manager doesn’t need to provide oversight every line of code thus we meet every few weeks to look over the sprint’s outcomes. To sum up, you need to understand how frequently your board members are willing to meet and let that guide your cadence and board structure.

Another aspect of meetings is that it may take multiple meetings to make decision. In my experience, I rarely see multi-faceted decisions being made in less than three meetings. The first meeting presents the problem that requires a decision. If it was a decision that needs no discussion, then it probably would not need the board to decide. Not all board members are fully aware of the problem and will want to time to investigate and determine their position, so a decision is pushed to a second meeting. The second meeting generally fleshes out the various elements of the decision. This is the chance for board members to bring new perspectives. New perspectives requires more investigation. That is why we arrive at three meetings³. Of course this is just a minimum number of meetings, often these investigations result in new information that just keeps pushing more meetings. Now combine the meeting cadence with this rule of three meetings and you have how quickly a board can make a decision. Meet weekly, then expect 1 month for a decision; meet daily, expect a week. Thus you determine the speed your board makes decisions by the cadence of meetings⁴. One hack I use with many teams I run is to have a status report at the cadence I expect decisions to be made and schedule ad hoc meetings for when decisions need to be sped up due to short deadlines.

Okay we have actions and dimensions of an action that determine when meetings can happen. We have a cadence of oversight that determines how often reports need to be made. We even have governance structures being determined by the frequency of actions and the minimum number of people being needed for particular actions. How does this apply to NumFOCUS? Well first we need to talk about the structure and actions of NumFOCUS governance. This blog is already over 1000 words so we will save that for the next posting.

I use to say, “you are not a team if you don’t meet every day”, and I still believe that to an extent. Once a CTO at a company I advised came to me with his resignation complaining that he wasn’t part of the “Executive Team” and used my saying as proof. Oh boy, was I in hot water, here is a company I’m supposed to be helping achieve their goals and a simple off hand truism is causing a rift so wide the team is breaking up! That CTO was clearly frustrated with the other executives on the team but it wasn’t really the meeting frequency that was the issue, it was being out of the loop of the decision process. Today, I do think if you are not meeting regularly, you aren’t working together, which can be fine, but I hold my tongue on demanding daily meetings. ↩
I’ll leave it to the reader to determine when a decision is beyond the charter of a particular board. Once in a youth baseball duty I had two divorced parents showing up and screaming at the umps and coaches about visitation rights. The mother even had a thirty page document which she claimed showed the father was not legally allowed to be there. Here was a case where I needed help from the executive board as there was no right answer. Both the mom and dad where violating our no verbal abuse policy, but kick the dad out you violate his rights as a father, kick the mom out you take the kid away from baseball (which was our whole goal of being there) or don’t do anything and violate the rest of the teams rights to watch the game. In the end we just separated the parents (luckily the foul poles on a baseball field are a long ways apart and yell there is fine) and told the mom to appeal to the local police for determining if the father was breaking a court order. We accomplished our goals but because the decision required different levels of governance we used a separation of concerns to dictate our escalations. ↩
There are ways to speed up the number of meetings, e.g. using email to flesh out details asynchronously. Unfortunately, many boards of filled with busy people who don’t have time to read all their email...I have over 13K emails in my inboxes at the moment. So I think my rule of 3 meetings generally holds. ↩
One of my favorite demonstrations of the principle is the State of Texas legislative session. If the meeting cadence determines the amount of work a board can do, it will also be a limiter to the amount of damage a board can do. The State of Texas lawmakers only meet every other year. I have no idea how to judge if it is a successful strategy but think about how slowing down decisions can have a purpose as well. ↩

Rejoining the NumFOCUS Board

2024-12-15T00:00:00-08:00

A few weeks ago, I was elected to a new term on the NumFOCUS board. This time it is the Admin board while a newly created Technical Steering council has also been formed. One can see some of the differences of the two group on the election webpage. This group is different than before where being on the board really meant, "Make sure the org is working." With so many different responsibilities there is a greater need to split the tasks.

Why re-join the board

I've been thinking why I joined the board again. I had been on the board from 2012 - 2021 and felt at the end that I had spent all I could on the cause. Since that time I've seen several businesses I had been working with fail and took a "normal" job at NVIDIA. NVIDIA has a lot of great perks, but the one that seems to have brought more spark to my life is the adage, "Do your life's work." Well helping scientists make tools for discovery had been my life's work up until I left Anaconda to pursue various applied ML startups. Even my work at NVIDIA seems less meaningful when I'm not focused on helping scientist use our technology. I don't know why but science has always been a calling.

It would be naive to say that was the only reason I wanted to rejoin. There has also been a bit of a challenge with watching the board from the side. Various events have led me to believe that the organization has lost touch with the community that put so much time and effort to build it. The classic story of serving more than one master. In this case it is very hard to know exactly what went wrong, but it was clear with Jupyter's exit that projects have started to feel less happy with the arrangement. My hope is that I will be able to bring back the spirit of the early days, but the truth is that it may be that the world has just moved on.

First week back

The first surprising thing to me is that the new board is only supposed to meet once a quarter. I'm skeptical that this will work out super well. The first task is to define strategic direction and the second is to profide fiscal oversight. It will be very difficult to know how the organization can structure itself strategically if reports are only once a quarter. I've been part of the Advisory Council which has a nebulous duty of giving advice but gain no right of a director, or be required to be heard (see bylaws). To be frank, the advisory board seems to be a thing where a bunch of folks are updated on things going on but I don't think there has ever been time for a lot of discussion.

The board met as a joint group for the first time where we had a few tasks, one was to elect officers and another to approve the code of conduct working group members. It was unclear to me how we would do either of those tasks as this was the first time the group had ever gotten together. I proposed we find a single temporary chair to handle the Technical Steering committee officers, as the Admin Board already has elected officers (not sure if those are changing). It was mentioned that the board will have a retreat in a few months, so a few of us volunteered to help organize that. If both the Admin Board and the Technical Steering Committee are incharge of setting strategic direction, the retreat seems a good way to get folks working on the same page.

Okay I'll keep it short for now. I would like to start blogging more about NumFOCUS and the community. We'll see how that goes.

Thoughts On Board Member Duties

2024-05-03T00:00:00-07:00

Last weekend I was working at the concession stand. I’m a board member of the local recreation baseball league and the assigned team to work the stand didn’t show up. For those new to the economics of kid sports, the concession stand is where flexible spending comes from and if it isn’t open, then the league can only afford the bare bones, e.g. umpires, grounds maintenance, and uniforms. The concession stand is where we make money to fix things like bathrooms and offer scholarships to families that even the bare bones price is hard.

While working the stand, with one volunteer and my son who played in the league for 6 years, I had two complaints about the women’s restroom. I called for help over the board communication channel, but everyone else was busy. At this point I had to make a decision. Do we stop serving concessions and take care of the bathroom? Do I leave my son who is capable but maybe not world knowledgeable enough to handle problems that come up? Do I ignore the bathroom and just keep working, after all bathrooms are supposed to be nasty at ballparks?

Then I remember the issues the league had last year being sued because of a sink failure. One of our little bundles of joy decided to jump up and down on the sink until it broke off. The ceramic crashed and broke, cutting the boy and sending him to the urgent care facility. For months after that the board was hounded with a lawsuit from the parents, which was eventually dismissed. The point is that operating a bathroom for a public event has certain duties and if you have two complaints about unsanitary conditions, you better check it out.

At this point, I did the thing that was one of the hardest things I’ve done at a baseball field, and I’ve kicked out drunks, confronted vagrants, and many of the unfortunate duties of a baseball director who is trying to keep a nice place for kids to play baseball. As I approached the line of ladies in their baseball mom shirts hopping up and down wanting to get back to the action, I could smell the bathroom. A quick peek in between occupants and the worst was true, a urine soaked floor with tons of trash everywhere. I politely told the ladies that I deemed the bathroom unsafe, closed the bathroom, and pointed to the portapotty on the other side of the ballpark. Thank goodness stares don’t actually kill.

Rushing back to the concession stand my plan was to clean it when we get relief from the next set of volunteers. Over a dozen women came and asked when the restroom would be opened. I heard people say silly things about the league not using its money wisely. I didn’t check but I expect someone posted mean things on social media, as the song goes “if good news sold checkin your phone would never get old”. The rumor mill doesn’t take a back seat, it fires up and starts spreading falsehoods as quickly as it can.

I did stop to explain to one group about the liabilities of the situation. We had complaints about an unsafe bathroom, and we are liable if someone gets sick. We sell food and if a health inspector shows up, we risk being shut down, and we risk losing the budget for non-critical repairs. Finally, once we had the bathroom cleaned and it no longer smelled, it had a wet floor, so we would be liable if a person fell and broke their hip. While I wanted to make sure our fans had a nice restroom, we were not liable if they had an emergency in their pants. The death stares did not go away.

What is your duty as a board member

I often get asked, what is the duty of the board. Having been a board member for a decade at NumFOCUS and now for several years at the baseball league, I can tell you simply, “Accomplish the mission”. The decisions that are made will be based on the best information you have at the time balancing the risks and rewards of those decisions. Furthermore, expect no one to understand or care about the person making those decisions. The number of people who complained about the restroom didn’t stop and thank me, a volunteer who is taking his free time to ensure their players had a ball field.

Ultimately, the board has to rise above the complaints and help the team come together to solve the next challenge. Expect to be called “incompetent”, “opportunistic”, and even “mediocre”. Nobody will take the time to understand the economics of your organization. Expect that when you serve people around the world, cultural differences and your governing policies will collide. Sometimes having a meeting will be considered hostile, sometimes not having a meeting will be considered hostile. Stepping into those shoes doesn’t come with a plan and certainly not a list of things that are “the right things to do”. Expect to lose nights of sleep, wake up to the fear that you are the problem and not the solution. Do not expect anyone to understand the job and thank you for doing it. It’s a hard job and don’t let anyone tell you otherwise.

If you as a board member don’t take the time to learn about the entire system you are helping manage, you will have a very hard time changing that institution. Having first hand seen board members not come to board meetings and they lay out complaints of having no effect. I have seen board members work very hard and still not build enough consensus from stakeholders. Some of these board members burned out because their own initiatives did not get accepted. I have seen board members try to drive from the back seat and numerous board members get very upset. Each board is different and that board has to learn to work together to solve problems.

But I posit you have one duty above all as a board member, and after you have been in a position of extreme pressure you will understand, you have the duty to support your fellow board members. I’ve seen this from the best leaders who reach out when you see a fellow member struggle. I remember a kind phone call from a more experienced COO once who saw how the CEO and I were not communicating with one another effectively. The result was a better proposal that still included my points and moved the organization in a better direction.

A Personal Apology to Jeremy Howard and the Community

2020-11-01T00:00:00-07:00

I, Andy R. Terrel, issue a personal apology to Jeremy Howard and the Community for the handling of the JupyterCon 2020 Code of Conduct reports. As the ranking board member on the NumFOCUS Code of Conduct Enforcement Committee (COCEC), it was my duty to oversee a fair process and that did not happen. I take full responsibility for the incident and recuse myself from any duty that the NumFOCUS Board of Directors feel I am unfit.

To Jeremy Howard, I'm sorry. I did not recognize you appropriately in responding to you in the call. I did not intend to make any mannerism that was laughter. When I heard the reading of the complaints and the way it was perceived I wanted to clarify and did so badly. I'm sorry.

To the JupyterCon community, I'm sorry. I recognize that having months of hard work invalidated by my failures is devastating.

To the NumFOCUS community, I'm sorry. I did not represent you well and now your work to create diverse, inclusive events is being called into question.

To the NumFOCUS staff, I'm sorry. My failure to oversee the process put you in a stressful situation in which you were not prepared. This new style of conference required so much more of you than was expected.

To the wider software community, I'm sorry. I have advocated for Code of Conducts for over 8 years now. In a few weeks, my actions have caused more damage than I could ever realize. I'm committed to working to undo that damage in whatever capacity I am able.

To all, I want it to be known that I hold Jeremy Howard's account to be honest and fair. I have seen several instances of negativity towards Jeremy and I ask that it please stop.

Prior to this incident I had admired Jeremy's analysis on COVID masks. After reading his post and seeing all the other advocacy he has done I realize how much Jeremy truly gives to the world. While so many political leaders are acting in ways that cause more harm, Jeremy has been showing positive solutions. Jeremy Howard's advocacy is saving lives and he deserves all the support we can give him.

I hope one day to be able to meet Jeremy, apologize face to face, and work together to build a better community so people like him can keep creating a better world.

Sincerely,

Andy R. Terrel

Personal history of NumFOCUS

2016-06-14T00:00:00-07:00

A NumFOCUS History: Sustaining Open Source Scientific Codes

After a few of the last board meetings and discussions on board votes, I’ve been pondering a number of questions. Questions about who is NumFOCUS and why we exist. Where we are going and what problems in the world we would like to solve. I feel that our small organization has started to make a big turn in our journey. We are receiving more attention from the organizations we serve and the wider community. The PyData conference has begun to become a profitable enterprise that allows us to see a sustainable path but also requires a great amount of our time. At the same time we need to support our projects and programs in a functional way that promotes our mission.

Why NumFOCUS

In 2011 a group of scientific python community members got together to find a way to sustain the tools the community was building. Two of the gentlemen, Travis Oliphant and John Hunter, the creators of NumPy and MatPlotLib, respectively, had left academia and used their tools in industry.

It was clear that many parts of finance, oil and gas, and analytics fields were using their tools for much of the business. While many organizations would have liked to give back it was not an easy thing to do. The communities were divided up at many universities, government labs, and companies. Who would take money for the project and who would own the results?

While Travis and John brought a strong passion to help corporations to give back to the community through donations that would help. The other three founding members, Perry Greenfield, Fernando Perez and Jarrod Millman, were still in academia seeing first hand how hard it had become to receive awards to develop software critical to the success of scientific innovation. For so many the academic software business was the thing you did on the side. I remember sneaking my software by the university’s technological commercialization program so the work wouldn’t be hidden.

In that initial set of founders, we had several project leaders for AstroPy, IPython, NumPy, SciPy, and MatPlotLib. Each of these projects are free and open source built in a bazaar model of development. Meaning everything was done in the open, accepting patches and bugs from anyone who gave enough attention to make them. These projects also strived to not form any restrictions on their use, as seen by the base layer for much of the NumFOCUS ecosystem. While this may seem like a minor point, it becomes a bit of a defining feature of the tools. While many frameworks will take over a program and prevent portability, these libraries were intended to be used by anyone. Notably the licenses for these projects were also quite liberal preferring BSD-like licenses. These decisions made the projects viable options to the enterprise thus are critical to their sustainability.

The initial board was formed and Leah Silen was hired as our Executive Director to help us organize the non profit and daily actions. I don’t have much to say about those early days as the organization tried to form itself. Looking back it was clear there wasn’t as straight a path forward as the initial fervor of the five founders might have liked. Additionally tragedy struck as John suddenly passed away mid 2012. And Travis left his position at Enthought to form Continuum Analytics.

Getting started

The initial struggles of the organization were clear. The board was too busy to be a working board at the level needed. Thus the board was grown to 9 (which included myself) with a very small quorum of 3. This policy didn’t change until 2016. The board built the organization just like any open source project with Github accounts, mailing lists and many discussions about the minutia of running a non profit. Needless to say this was not going to be your typical non-profit organization.

In those early years, we sustained from three major corporate donations. Continuum Analytics, the newly formed company by Travis, gave us office space and paid Leah’s salary. JP Morgan Chase gave us a generous donation to help build our diversity program and Microsoft gave a donation to the IPython team. Additionally, we started a small conference series called PyData.

It’s not hard to see how overwhelmed we quickly became. With nine board members, one full time staffer and now projects, conferences, and a diversity program to run. We struggled to communicate all we did effectively much less get it started. In hindsight we should have been a bit more cautious about diving in head first. Leah got to know the night cleaners at the office and really poured more dedication into our effort than I can ever describe.

It is worth a moment to pause and discuss why these particular activities. The simple answer is that it takes much more for a community to survive than just coders committing to a repository. Just like all humans we have to have time to gather, discuss, and collaborate. Additionally, at that time there were two very important topics our community was grappling with, diversity and data science.

NumFOCUS Mission

I think the first surprise to most folks is that our mission is not to write software. In fact I question any person whose mission is to write software. It’s a bit like meeting a delivery driver whose mission is to consume gasoline. While certainly a side effect of making deliveries, not the essence of the activity.

Additionally, when we started we knew we wanted to support the ecosystem of numerical tools that scientists used. NumFOCUS was an acronym of this idea “Numerical Foundation for Open Code and Usable Science”. This has also become a sticking point over the years. What about tools that are not numerical? How about those that are important but have a very small user base? Our mission became much more about promotion, education, and fiscal sponsorship. From our webpage (June 2016):

The mission of NumFOCUS is to promote sustainable high-level programming languages, open code development, and reproducible scientific research. We accomplish this mission through our educational programs and events as well as through fiscal sponsorship of open source data science projects. We aim to increase collaboration and communication within the scientific computing community.

With that as our defining goal let’s review our activities and how they pertain to that goal.

Fiscal sponsorship program

The initial idea and structure of the organization was to provide a fiscal home to projects that fit. This is a bit of a foreign concept to most open source codes so I’ll try to explain a bit. Let’s look a somewhat standard progression of open source scientific software.

A group of scientists determine a need in the community, usually at a conference, and pursue a grant from somewhere like the NSF or NIH. The project starts a shared repository and contributors from several different organizations start committing code. A license is added to the repo, now-a-days mandated by the granting institution, and it becomes usable by those the license stipulates. If the project is successful, a user base pops up and folks start putting things on mailing lists. Decisions start to be made that impact a larger group of people and thus some form of governance will emerge. The grant ends, as they are usually only for a few years, and there is a pile of code with the starting of a community.

I usually liken this to the building of a house. While the house may be built and be sound, it is far from done. Every week the house is cleaned and the grounds are attended. Every month requires minor maintenance and at least once a year major maintenance. New additions are built, rooms are rearranged, and furniture comes and goes. Even the people in the house grow and change. Eventually, people move out and new people move in. Some houses get turned into businesses, others host many overnight guests, but most serve as homes to the people who own them. Until an end of life event, such as being torn down for the progress of another more up to date home or even a restructuring of the entire city. But during all these years of a house’s life, it must be cleaned, attended, and maintained.

Just like a house, a software community has the same burdens of maintenance for sustainability. Communities also come and go. Few people use Numeric or NumArray the two libraries that NumPy was based upon. LinPACK was replaced by BLAS and LAPACK. Since the first lines of FORTRAN, the university and government lab has been a place to house scientists who are building these communities. During the last 10 years our academies no longer employ most of the scientific software writers. I have seen reports of as low as 12 percent of graduate students stay in academia.

Thus as graduate students no longer become professors, they move to industry and have taken their tools with them. It is hard to find a software project in technical fields that don’t use some open source software that comes from the academy. But without a home at universities or labs, the scientific community needs a place to hold its assets. Just like a grounds keeper of a house owns a hoe to weed, software developers gather tools and computers for maintaining their code. Project need websites for gathering information about the development. Additionally, they need meetings to gather and discuss the future of their projects.

The NumFOCUS Fiscal Sponsorship program was designed to help communities achieve these goals. We help projects to promote and organize themselves. We help them seek funding from foundations and corporations. We provide infrastructure for websites and code maintenance. We do this because we believe that giving scientists access to better tools helps everyone.

Not every project fits well with our fiscal sponsorship. For example if you are at a single institution, then that institution should be the sponsor. They provide the overhead to keep your team working and usually want you to pay your share when you do raise money. But as we have all seen it is sometimes quite hard to work with universities and labs whose goals do not meet the projects. We have an additional program, our Affiliate Project, for projects who for whatever reason cannot be a fiscal sponsored program. Being an Affiliate Project means that the board has approved your project for applying for grants from our general funds, but it subject to board approval.

There are a lot more legal details in our sponsorship projects, but we think it fills a gap for our community. As of this writing we have 15 sponsored projects and have given dozens of grants to sponsored and affiliate projects.

PyData and Conferences

In my story above about the creation of scientific software communities, one will notice that conferences and meetings come up quite a bit. It is my belief that these conferences are the most important part of our ecosystem. They allow us to promote and teach our tools. They recruit young aspiring scientists to our projects. Furthermore, they allow an open forum of thought that helps guide our future.

With this notion several people started the PyData community, notably Travis Oliphant, Peter Wang, Julie Steele, and Lynn Bender. Initially it was a one day workshop after Strata San Jose where a group of scientists got together with Guido van Rossum. The idea was to help Guido understand our use case of the Python programming language and how the core dev team could help us, such as providing better support to non standard CPython builds. The result of that meeting was a kind of admission by Guido that our community should make separate solutions as the Python community would not be focusing our needs.

The PyData Silicon Valley event was repeated in New York City, another place we knew our tools were being used in the finance sector. Van Lindberg, chair of the Python Software Foundation (PSF), gave a keynote and gave the suggestion that PyData become a conference to support the NumFOCUS foundation like PyCon does for the PSF. The conference was sold out and became an instant hit, so the notion became appealing.

Currently we have 12 PyData conferences throughout North America and Europe. One of the keys to the success has been to leave a PyData Meetup group in all the cities we visit. Some have worked and those, such as London, have created a thriving community. It gives the local community a way to keep engaged year round.

PyData is only one conference we support. We have supported many such as JuliaCon, SciPy, EuroSciPy, SciPy Latin America, Scientific Software Days, AstroPy, PyCon, and Scikit Learn sprints. These meetings provide us the opportunity to teach the world how to use our tools. As most sessions have tutorials for beginners. They also give you an opportunity to sit down with the developers of a project and understand the problems they solve. So to me they have been a huge part of our organization.

Diversity programs

Our field, science and code, has come to admit a major diversity problem in the recent years and NumFOCUS has been trying to do our part in helping fix the problem. I, like many in the field, was blindsided by the issue. I remember distinctly the moment I saw the problem in its full breadth. Matt Davis tweeted at the SciPy 2012 Conference, where I was program chair, “More men have walked on the moon than women are at the conference.” I went back to our registration and count a whole 3 of 200 participants were female.

In the years since, NumFOCUS has given a fellowship to the SciPy Conference. At each conference we held a diversity event where people talked about ways to increase the diversity of our community. I think the number one take away for me at those events was to attract a diverse community, one has to invite a diverse set of individuals. I have always been more interested in events that I felt invited too and the scientific computing community wasn’t the most inviting bunch.

To that end, NumFOCUS has put on many diversity workshops and training programs. I think my favorite that is still ambling along is the training week for high school aged females. One of our board members Cindee Madison started the program and helped lead several workshops. Its definitely one of the programs that is currently struggling as volunteer help is always needed. I believe the program was going to spin out on its own but lost a bit of steam when Cindee had some family medical problems.

Other programs

One of the best parts of being the President of NumFOCUS these past two years has been to work with some of the brightest minds on the planet try to tackle the issues facing our community. Each year our board comes up with new challenges to try and address. Some programs take off and others die.

One notable program that I didn’t speak much about is the Technical Fellowship Program. We were able to give a graduate student, Olga Botvinnik, time to work on the seaborn visualization library while working in biology. An experience that is difficult to get while pursuing a PhD. I think Olga was a great candidate but in the years since we have failed to come together and decide how we want the program to evolve. While giving a student time to develop their technical skill, it seems that money could also go to developers who work full time on the project. I’m sure we will revamp the program in the future.

We’ve given out grants to some interesting projects that are cross cutting in their concerns. One example is PyPy in a Box, which the PyPy jit accelerated Python implementation was built to work with scientific codes. Ultimately it didn’t take off. Another ongoing project is to help make Microsoft Windows a first class citizen in our development community by building a tool chain with functioning Fortran compilers and compatibility with system libraries. Yes, we still us a lot of Fortran code and you are welcome to come and try to rewrite it. Many have fallen to this task.

Sustaining Our Future

Now I’ve spent the better part of 3K words to get to the part I wanted to write about. One has to know the history to the places we are ready to go, but I think folks only ever see one part of the organization. We have many lives at NumFOCUS these days. Part conference organizers, part grant administrators, part project governance advisors, and always under funded. Our staff has grown from a single Executive Director to a team of four.

Our main source of income comes from the PyData conference series. The conference is designed to be cheap enough that students can afford to go but nice enough to fly in good keynote speakers. Without the conference series we would not be able to do the work we do.

We also receive some money from the fiscal sponsored projects. Initially, we had a very low overhead fee as we wanted to give as much to the projects as possible. We still only charge about 10% which is still absurdly low in the non profit space, but we found there is more work than we could possibly manage without more funding.

But for the most part, this is just the beginning. We need to flesh out our corporate donations program more and start applying for institutional funding for our own programs.

For now we have some basic income and can start thinking again about better ways to support our community. I would like to see us able to hire folks to help maintain projects better. For example run the infrastructure that is necessary and provide workshops for scientists to sustain their projects.

Diversity at SciPy2014

2014-07-17T00:00:00-07:00

SciPy2014 is over, but there is so much more to do. I expect to be reviving my blog with a set of prose on my experience. Today I want to talk about a topic that was dear to my heart when I started this blog, diversity in the scientific python ecosystem. I'm focusing on gender diversity, but we also need to be aware of racial and other diversities, as well. I hope helping one will lead to helping others.

For the record, I am writing this post from my personal perspective. It is not representative of any company or community that has graciously chosen me as a representative.

SciPy, we have a diversity problem and it's not anyone else's job to fix it. We've been trying to fix this problem, but this has uncovered other problems with our methodologies. It is my sincere hope that this amazingly brave post by April Wright and follow up will help shape the future of our community. Thank you, April, your courageous anecdote has been hugely impactful ... if the community listens.

Working on diversity at SciPy

First let me outline some of the efforts that have been taken in the past three years that I have been involved with organizing the SciPy Conference. Even getting these seemingly trivial changes has been a challenge I never expected. I've had community members tell me I'm wasting my time, others tell me not enough is being done, and the larger part of the community a bit uninterested in the topic. Folks, SciPy is a community conference, it is on us to work hard to make it the community we want and I don't want to be in an exclusive white male club any longer.

First, in 2012, Matt Davis noted via twitter that more men had walked on the moon than women attending SciPy. How dare he say such slander! ... wait let's count how many women are here ... I only see 3 ... très ... ¡Qué hostias!

The first thought on how to change was to find more women to be on our organizing committee and keynotes. Becoming chair in 2013, I asked every woman I knew in the community to help out. We increased to 8 out of 58 in our organizers. Jonathan and I asked a half dozen women to keynote, but were roundly rejected. Not only do we have a problem with diverse groups coming to the conference, they have to be heavily persuaded to be a part of the organization at all. I could send one or two emails to community men I knew and get a avid contributor. This is the hill we must climb.

In 2013, we also had a social funded by NumFOCUS and the PSF. The result was about 20 women of a variety of races coming together to talk about the issues. The message the organizers took home was cast a wider net. There is amazing quality from women in scientific computing, but they will not be looking at conferences that are primarily white men. We are also a very small growing conference (150 in 2012 to 460 in 2014) so people outside our clique aren't going to come for prestige. Also the overwhelming opinion was, a social is nice, but less alcohol more food.

This year, 2014, we did similar things as last. Increase women on the organizing committee (19 of 84), added a diversity committee, asked my uber-prof friend to give one of the best SciPy keynotes ever and host an event for diversity. We also worked with NumFOCUS to fund scholarships explicitly for women in technology through a generous donation from JP Morgan. This funding helped 5 more women attend the conference, in addition to the 15 other students (which included 3 women) supported to attend. We estimated 15% of attendees were women, not great but better than that original 1.5% in 2012.

The conference also had several discussions on the topic during the conference.

Katy Huff's wonderful assessment of the situation in the poorly named "You win or you SciPy" Birds of a Feather (ignore the fat white guy before her!)
Cayci Gorlitsky lightning talk on Diversity in Tech

Additionally, I opened up the mailing list for the conference organization with the hope of encouraging more participation from the community. The efforts to take the conference as open as possible is one other effort to build the community we want. Being open isn’t sufficient to be diverse, it’s a necessary but not even close to sufficient condition.

Needed Changes

First, April hit the nail on the head. Who cares if you do all the things above if your conference is not welcoming to everyone. Below, I outline a few specific actions the SciPy2015 community can take.

Welcome to SciPy track in all event categories
Better social events to get to know each other
Cast wider nets via smaller user groups
Teach the community how to run events better
Have a welcome committee to help people know who we are.

I've discussed this with many many community members and SciPy2015 will have a "Welcome to SciPy" arc to help folks know who we are, what tools we use, and why. The comment that it is a great conference if you already know all the things to succeed with SciPy has been repeated often.

Additionally, we have discussed quite a bit about having social events that are not alcohol or food related. There have always been smaller events not organized by the conference but we need to encourage this activity more broadly. This year if you watched twitter, you would have found folks swimming, running, and climbing, but having this in a program would be a big help.

As Katy pointed out we need to advertise our activities more. Some folks are chatting about starting meetups around the SciPy community. A small way to make sure folks are seeing local faces more often.

Sheila Miguez pointed out the incredible in-person event handbook from Shauna G. of Open Hatch. I think taking up the principles in this handbook is really needed. We have not made welcoming, goal setting, and clarifying structures a priority at events.

In the past, I have held events for other orgs where we made sure that a small set of folks were explicitly tasked with shaking hands and talking to people who were alone. It seems foolish to some, but making folks break from the clique and talk to others only helps build the community.

We Need You

Finally, SciPy, we need you. Today all these events happen because of a few concerned volunteers. We need more people helping with our events, advocating for the minority, and building the community.

Community building doesn't fit nicely on a resume nor does it always feel good, but it is critical to everyone. You will be depressed after you read a blog showing how far our community has to go. You will have colleagues tell you that you are wasting your time. You will miss deadlines that affect your funding or day job. Without community we will not sustain.

When I sat down with folks to encourage them to join our executive committee, I typically made the argument that if we don't build the community we want, no one else will. There are a lot of challenges we need to face, e.g. promoting code as a real scholarly work and promoting an open sharing culture among coders in scientific computing. We cannot do this without a diverse functional community.

Please go sign up for the mailing list and start making this a better place. http://mail.scipy.org/mailman/listinfo/scipy-organizers

Acknowledgements

Thank you to: April Wright, Kelsey Jordahl, Anthony Scopatz, Matthew Turk, Dan Katz, Sheila Miguez, and Kyle Mandli for reading an early version and edits to this document.

Joining the Big Data Brain Drain

2013-11-21T00:00:00-08:00

So long academia and thanks for all the debates!

Today is my last day of a wonderful three years at the University of Texas at Austin. Starting Monday, following Jake Vanderplas' terminology, I join the big data brain drain. I will be joining Continuum Analytics a their Chief Computational Scientist, with a focus of helping them join HPC technologies to the Big Data world. But that is another story.

My story

I have wanted to be a scientist since I was quite young, okay I really wanted to be a jet pilot but my genes killed that one. I remember the day that I decided to become a computational scientist. It was at the Red Raider Minisymposium on Biological Systems. I heard an excellent discussions by James Keener and Charles Taylor on the role of computational mathematics and understanding cardiovascular systems. Additionally I was basically offered a position at the prestigious University of Chicago on the spot. The idea that I could contribute to treating the medical ailment that has claimed the lives of many in my family. To this end, I was elated when center for biomedical computing at Simula started using the software I helped with in graduate school.

After graduate school, I've had the amazing opportunity to collaborate with scientists from a diverse set of applications and backgrounds. I've traveled to four continents to present my work and collaborate. I've released several coding projects that have pushed the boundaries of computational mathematics field in numerics on accelerators, adaptive meshes, and domain specific languages. Some of these discoveries have even pushed into PETSc, the premier HPC platform produced by the Argonne Lab. Additionally, I have had the privileged of helping with the scientific python community taking the role of advocating and organizing the SciPy conference and the NumFOCUS foundation. In the coming months I will be an editor for the Scientific Software Days issue in CiSE magazine. Its been a jam packed three years and there are only a few things I would do differently.

Despite my deep respect for academia, I have come to terms that it is fundamentally not the right fit. I've had on and off moments where I start something in industry and come back, but this time is much different. After finally getting a rather large grant from the US Army, I had thought that I would finally have some time to determine my own direction. Unfortunately, due to no part of mine, the grant was cut in half. My own deliverables were slipping so the rest of the money was on a slippery slope. The government sequestration and shutdown meant that my money would definitely be out before it could be renewed. Just like that, I went from well funded PI to soft money beggar.

This presented with a choice of bouncing around on other PI's grants and hoping to get a more permanent position or moving to industry. I shopped around my CV to see if I had a shot on the academic job market, but was told that I was not competitive by several friendly collaborators. Most hires, even in computer science and applied mathematics, are people who have been out of graduate school for five or more years. On top of this, the general perception was I am "just the coder" for the work. So when my friends at Continuum Analytics gave me a call to do some of the most ambitious work I've seen, I didn't hesitate.

Call for sustaining career paths for scientific coders

Could I have salvaged my academic career? Absolutely. But the events above makes me stop and ask if I wanted to. Having turned down offers that tripled my salary for years, I've stayed with science to make an impact on the wider world. I think academia does make the world a much better place. Unfortunately, I also see my role as the scientific coder being diminished year after year.

I can point to half a dozen folks I know not getting tenure. People in post doctoral roles for a decade. Families living in different countries trying to keep their academic careers alive. And almost no academic I've ever met has a good work life balance. I'm willing to give up a lot to be an academic, but now that I have two children and one on the way, I'm not interested in giving up as much as it takes.

Even with academia being a hard place to make a career, you add to the fray that the field doesn't acknowledge code as a scholarly work and I'm out. At Supercomputing 2013 I saw three different presentations (WSSSPE, Saul Perlmutter's plenary, and a BoF on Sequencing and Accelerators ) all calling for better coders in science. But as pointed out again and again, the motivating factors are not in favor of better code in science. Without a sustainable career path for those who are motivated to be "just the coder", science will continue to not compete.

One of the more persistent complaints I and many of my colleagues have had is being the lone coder in a larger group. One astrophysicist turned startup monkey once told me, "I'm tired of being the only one in my group that knows what an expression tree is." We need more computer science skills for our scientific staff. Unfortunately, computer science departments across the country are trying to move scientific computing out of their department. With only a few scientific computing departments in the entire US, there is a lack of options for scientific computing professionals.

I have volunteered as much time as I can afford to promoting and training folks like me with Software Carpentry and SciPy conferences. The recent news of the Moore Foundation and Sloan Foundation to further the careers of folks like me is a welcome message. With the increase of science relying on advanced technologies and the increased rate of retracted publications due to software errors, academia must adapt. As a person who advocated for better software at an NSF Supercomputing facility, I can only wonder how long until the NSF makes a center for open sustainable software. Money for the hardware is just not enough to advance the cyber-infrastructure of the US.

Future Advocate

While the door to academia never truly closes, I will have an opportunity to work with an amazing team that is using much of the scientific technologies I've grown to love. A scientist at heart, I plan to continue helping with SciPy and NumFOCUS, but as a industrial scientist with a team of very talented coders. My hope is to keep working with folks in academia to take their ideas to new levels of impact on the world they are passionate about helping.

As the great @dabeaz did while I was a student in his class, I give the 21 gun salute to my academic career. So long and thanks for all the great debates.

Photo attribution

By U.S. Navy photo by Photographer’s Mate 2nd Class Denise Martin. [Public domain], via Wikimedia Commons

Women in Scientific Computing, SciPy 2013

2013-07-18T16:00:00-07:00

This post written by Kristen M. Thyng about an event we both worked to organize at SciPy2013.

As in other computing fields, women make up a small percentage of the people in scientific computing. In order to bring together the women at the SciPy conference to aid in building community and to discuss related issues, a mixer was held during one of the evening Birds of a Feather sessions, moderated by SciPy Conference co-chair Andy Terrel and attendee Kristen Thyng. The mixer was co-sponsored by generous support from NumFOCUS, the Python Software Foundation, and the SciPy conference.

Approximately three women attended SciPy 2012. In 2013, there was a substantially higher representation of women, with about 35 women or 10% of the participants between the tutorials, the conference, and the sprints. However, this number is still low, and more elucidating is how the numbers break down by participation activity. More detailed numbers for female participation are as follows, noting that some of these numbers are exact and some are rough best estimates since demographic information was not gathered from conference participants:

1/18 active program committee members
1 of about 10 lightning talks
1/5 awardees for the plotting competition
1 of about 20 tutorial leaders
2/13 sponsored students
3/19 conference co-chairs
Authors or co-authors on at least 13 of 120 submissions
At least 3 of 27 poster presenters
At least 4 of 70 speakers

Soon after arriving at the event, people became acquainted with each other and started chatting. There was a good number of people at the mixer to meet new people but still have a chance to meet everyone. There were, all together, 16 people at the mixer: 13 women and 3 men.

Eventually, the group formed a circle in order to have a chance to have a discussion. The discussion covered many topics. Specific special interest groups for women in computing were brought up and women attending the mixer had positive things to say about the groups. In fact, many of them were attending the conference due to encouragement from PyLadies. They found that PyLadies was a less intimidating way to meet and talk and learn about Python than a regular Meetup group. In particular, some women found the Learn Python group to be helpful.

For helping to get women (and men) who are already interested in computing to learn more and therefore be able to participate more in a conference like SciPy, it was suggested to have a separate tutorials and talk track for introductory material, and have proper tools for self-assessment to make sure participants can correctly choose their path.

When asked whether a separate women's track at the conference would be desirable, participants said that they would not like this because they would rather be able to attend whatever track best fit their interests, and would not like to be singled out when it came to the research material itself.

Mentorship at the next SciPy between women (and possibly men) who have previously attended a SciPy (or related) conference was suggested to help new participants feel more comfortable in what might be a new community.

It was brought up that few iPython (and other package) contributors are women. Possible reasons suggested to explain this included that women may not feel comfortable contributing when men are aggressive and blunt. This is something that could be addressed by increased understanding and awareness on all sides.

The conference Code of Conduct was brought up as was how prominent to make the statement and how best to preempt any potential problems. Participants agreed that having previously established rules for how to deal with problems was worth any time that was spent on it.

It was unanimous that people at the SciPy conference had been accepting and friendly to everyone.

A question brought up was how to get more women involved with conference planning, which could then in turn help the conference be more inclusive and diverse in the future, keeping in mind that sometimes women are cautious not to be stuck "just" planning and organizing, which could be considered a traditionally female task.

To get more women to attend SciPy and submit abstracts, people thought that helping with the process could be beneficial. This sort of support would be most logical to come from a group like PyLadies.

Various organizations were suggested as being helpful to seek out, including:

NumFOCUS (has funds for scholarships and women intiatives along with resources)
Ada Foundation
PyLadies
Outreach Program for Women (OPW) (Internships for women in computing)
GNOME Women
Women in Engineering (UT program to encourage and provide support for female students in the engineering program. Similar programs are sometimes available at other universities.)
Software Carpentry (teaching/learning materials are available online)
Girls Who Code

A general suggestion for getting more people to SciPy was to have differentiated funding for people coming from companies versus those who are paying their own way (and are not academics).

Overall, a good time was had by all and it was very enjoyable meeting and discussing with a small group of like-minded individuals. Having these sorts of discussions could be helpful in having a more diverse set of scientific coders over time! For more information about this event, contact Andy Terrel at andy.terrel [at] gmail.com or Kristen Thyng at kthyng [at] gmail.com.

NumPy SciPy Benchmarks

2013-07-07T21:08:00-07:00

At PyCon2013, a group of about 30 developers got together to discuss NumPy and SciPy. The room mostly included folks that use NumPy and SciPy extensively, such as myself, a few who were just learning, and several core contributors (if not currently in the past), such as Eric Jones, Travis Oliphant, Anthony Scopatz, and Jake Vanderplas. It was an impromptu meeting, as most birds of a feather sessions are, so I don't think we even got the full number of interested participants at PyCon2013. The scientific large support at PyCon was surprising, since most of us scientists think the wider Python ecosystem leaves us behind in many of their activities.

There were three topics we discussed: briefly about the development and maintenance of the libraries, about the use of newer technologies, e.g. Cython, PyPy, Copperhead, Numba, and finally about having a benchmark for new implementations of NumPy to see how complete they are. Since there were not very many core developers in the room, it was hard to really push a topic on the development side but the other two topics were the most discussed.

To briefly summarize the maintenance and future plans section, I asked about how the community was doing since I've been organizing this whole conference thing and haven't been on the mailing lists in a while. Travis mentioned that NumFOCUS was trying to raise money for the projects, especially around the topic of paying a maintainer.

Miniature Applications

Next we turn to the topic of benchmarks. At the end we decided that there are at least two types of benchmarks that are really needed by the community. First, is a set of miniature applications that show off the tools well. Since there is such a wide variety of tools available right now, it is difficult to know what to invest in as a developer. Ian Osvald mentioned his book that includes one of the only major comparisons I've seen in the tools. I suggested we look at how the Mantevo project implemented its mini-apps for the US DOE. Another suggestion was to organize the applications in the seven dwarfs of scientific computing suggested by Phil Collela. I believe these have been renamed to motifs and extended to eleven by the Berkeley crowd.

The idea moving forward will be to collect a number of applications, give a reference implementation, and accept contributions showing how to implement them in a Python runnable way. The idea is not to make everyone use Python for everything but to show off the capabilities of Python as an integration (and implementation) platform. One participant brought up that we could have a site like the Computer Language Benchmarks, while we agreed in spirit enough of us do not like that benchmark because of its purist ideas that applications should stick to a single language. To be friendly to vendor solutions, I'm willing to accept closed runtimes but since the point is to highlight the code, all code must be free and open, preferably BSD. I am sure we will refine the rules as we go.

NumPy Benchmarks

Matti, a contributor to NumPyPy, brought up the idea of creating a suite of benchmarks to compare different implementations of NumPy. The idea here is once again not to compete ruthlessly for fame and glory, but rather provide a baseline for developer to hit when implementing alternatives and users to know how complete and single implementation is. Right now I know of several NumPy implementations (or close to implementations), e.g., NumPyPy, Numba, Copperhead, and a benchmark will be a welcome addition to the tools.

The plan to start off will be to grab out some tests from the NumPy regression tests and grab a benchmark runner. Matti suggested trying to port the PyPy runners for our needs. Once we get something up and going we can try to put it up somewhere for people to make contributions.

There were many concerns about the benchmarks not being productive and bringing out too much competition. My view is that we should try to be as open and welcoming as the rest of the Python ecosystem. That is let the numbers speak for themselves and make these benchmarks something anyone can contribute to.

A final aspect of the benchmark would be to show off that NumPy is useful for more then numeric algorithms. Many people see "Num" and assume there is no use for their specific needs, but in actuality, NumPy should be thought of as more a typed array. This allows fast access to contiguous blocks (although the element access is often slower than lists).

Going Forward

This post is intended to start a wider discussion among the community. I'm posting here because there are enough places in the community and I wanted a single place rather than four or five mailing lists. The hope is we can start a repository and get some tests rolling.

SciPy2013 Recap

2013-07-05T12:28:00-07:00

It's been a week since the end of SciPy2013, where I was co-chair, and I'm just now catching my breath. It was an amazing event with so many innovative and forward looking trends that I won't be able to give them enough credence, ever. When I took on the role as co-chair, my guiding goal was to provide a home to scientific researcher engineers, whether they are in government labs, industry, or academia. Just as William Schroeder said in his keynote, we are the new scientific publishers and are the driving force behind much of the technology that is changing the world.

Favorite Moments

Let me give a list of the best moments for me at SciPy2013, I'm sure I will be commenting on them throughout the year.

The SciPy Proceedings have moved to a fully open peer review process for more details see Stefan van der Walt's blog
There were a huge number of commits to 18 projects. Including these highlights (mostly gleaned from #scipy2013.)
- Matplotlib had over a dozen contributors (including a 10K LOC patch review).
- Scikit-Image now runs on Python 3 and 2 from single code base (without 2to3).
- IPython developers sequestered themselves to focus on the ever-promised 1.0 released.
- Contributers to over 16 projects came together to discuss commonalities with the compiler technology that is being reproduced in each other's code.
- Please let me know of any other major achievements at SciPy2013 sprints. We need to collect these stories to help convince sponsors.
We had >350 people attend, >500 people tweeting, social events almost every night, 139 videos including 12 amazing tutorials diving into every aspect of science, and over 6% of the conference either named Matt or from France.
Hosted a gathering of Women in Science with a wide array of participants with the help of PyLadies, The Python Software Foundation, and NumFOCUS.
Over 10 BoFs that provided for a much more participatory discussion on major topics facing our community.
And yet another completely ridiculous lightning talk from Matt Terry.

Once again I want to give a shout out to all our wonderful sponsors! Without them there would be no SciPy conference (or at least one with less BBQ).

Other's words

Also a list of other people discussing the conference:

Travis Oliphant
Aaron Meurer
Software Carpentry
Ondrej Certik
Bruce Berriman
Hairy Code, 2, 3, 4/5,
Eric Jones
Kitware: Pat Marion, Matt McCormick, Will Schroeder
Again please let me know of any other postings and I will add them.

Participate at SciPy2013

2013-04-24T18:41:00-07:00

The early registration deadline is rapidly approaching for SciPy2013, and all our decisions on speakers are out. This year was really tough as we had more submission than ever at a quality that exceeded all my expectations. If you submitted something and haven't heard from us, let me know.

If you missed the deadline for proposals, I want to elaborate on a number of ways you can still participate. While the material for the official program is closed, we have ample opportunities for more spontaneous contributions for all attendees.

Join the community
Speak at Lightning talks
Hack at the sprints
Participate in BoFs

Join the community

This year we have a number of opportunities to help get acquainted with learning how to contribute. Starting with two days of beginner tutorials to teach the basic tools of our community and a tutorial on contributing to NumPy, the core of almost all scientific python projects. If you are new to the ecosystem or an old hand needing to know the ends and outs of contribution the tutorials are there to help.

We have teamed up with the Austin PyLadies to present some material on how and why to contribute to open source communities. It's well known that open source communities can be difficult to navigate and folks are not always polite. We hope to show some perspectives on contributing and what to expect.

Lightning Talks

While we don't all have a story of printing NumPy arrays to QR codes and scanning them to get around bureaucracy, lightning talks remain one of the most fun parts of the conference. Tell us about your new venture, your craziest hack, your nerdiest tech rant, or whatever you feel best fills your five minutes.

Hack at Sprints

The sprints start Friday and we hope to have a large number of the core projects all come out. In the next week we should have sign ups on the website. Sprints are not only a good time to focus on your project for a few days, it also is a place where the experts in the scientific python will be in the room. Any quick bugs or small details to ask can have a much quicker iteration cycle.

This year we are also encouraging folks to not wait until Friday to start hacking, and I know some of you wouldn't wait anyways. I'm looking at you Fernando, Jake, Wes, .... But this year we will have about 6 break out rooms just right for 4 to 8 people to brainstorm and hack in between things.

Birds of Feathers

This year we have nine time spaces open for Birds of a Feather sessions. I personally find these session the best to discuss ideas with the conference attendees. We will have a sign up on the web soon as well. Some ideas I have floated include: state of the project BoFs discuss the future of many projects, evolution of the publication in the state of open code and data, and best ways for teaching scientist python.

Usefulness of a University

2013-04-08T21:08:00-07:00

The post reiterating that the biggest problem in teaching software carpentry (SWC) is installation, got me all in a tissy. I even signed up for their mailing list!

My first impression with SWC was a wonderful community that in encouraging folks to learn computing skills. After witnessing the TeachScheme movement, I was skeptical that two day bootcamps could really have any impact, but then again I'm usually wrong. Now coming off two and a half years of supporting users on supercomputers, I too see the frustration that started SWC in the first place. Scientists waste quite a lot of time not using the tools effectively, but then again I waste too much time futzing with tools. The current frustration is over the view of what education of these tools actually is.

What is a university education?

Some of my best friends in life were made in the classrooms of my university education. While learning the essence of science, math and philosophy, something else was happening as well. I was participating in a community of learning. These friends were the ones who helped me struggle through problems, understand new topics, and motivate my continued education.

While one can go through Bloom's Taxonomy and be as theoretical as one likes, I have yet to see many people actually discuss the building of a community around your classroom. For most open source developers, this is the major draw, to work on hard problems with a set of intellectually challenging peers. I will say that some tools are being developed to help this, such as Piazza. At the end of the day, MOOCs, bootcamps, and plain-ol-lectures that don't give students the ability to actively engage an topic on their own terms isn't worth its salt. Developing communities around learning is another method for continuing the education process.

Perhaps the only thing that a University is providing that other methods of teaching hasn't touched is this community of learning. To me that's the biggest problem in alternative university education, there is no community of learning after the day ends.

(On that note I have an unfunded grant proposal to turn textbooks into community initiators. Perhaps I'll get my act together and pursue it again.)

A Rule of Software on Usabilty

2012-11-01T10:00:00-07:00

Make your software useful. That is the point without usefulness there is no need for existence.

If you write a function that is not useful, scrub it from your text editors as not to offend the blessed pixels. Do not fear throwing away non-useful parts of your code, better to cast away these parts rather than sacrifice the usefulness of the whole.

If it is useful to repeat yourself, do so. Repeating yourself in subtly different ways teaches your novices, which must be done well. If there is a common pattern for your task use it, until it is not useful anymore. Write your tests as soon as they will be useful, tests are always useful before giving your code to others. Separate your code into useful partitions. If the partitions are too big or too small the usefulness will be clouded as it is difficult to track the function and interplay between the partitions.

When the program is done, let it stand on its usefulness. If others do not find it useful, seek out more to judge your craft. If none count themselves blessed to have witnessed your skill, throw out the program. Better had the program not been born than to prop it up with false claims of usefulness or many words about its beauty, adherence to principles, or features that none have found useful.

Each programmer must look from within to find the metric of usefulness. Some programs are only useful to the programmer, others must be useful to a multiplicity of programmers, and few hold themselves to such high esteem as being useful to the unwashed non-programming masses. Do not make the program raise malcontent by seeking affection in those it cannot help.

The API can hold back the usefulness of your code. Write many useful programs with the API and minimize their length. Large APIs are weaker than small APIs, but both can be useful. Do not be afraid to erase parts of the API which were only scaffolding to find the better API, the masterpiece could not have been made without the scaffolding, but if it is left standing it will obstruct.

When the performance guru tells you speed is more important, ask them why they don't make their users write in assembly, why BLAS doesn't require you to pack your own matrices, and why the CMS doesn't make you connect to your own sockets. To discourage performance hacks give the programmer five lashes as these hacks are the enemy of abstraction, which can be useful. If the programmer persist in their hacks, let them continue as clearly the speed is useful.

When the enterprise architect recommends yet another abstract factory manager class, ask them why they started programming and at what point they decided to make others despise reading their code. Separate them from the strong, eager, useful programmers and burn their resumes so that others do not confuse their ways as useful.

If the functional zealot tells you your side effects destroy the formal analysis, explain to him that monads rhymes with gonads which are quite useful. Ask them to travel discalced to the far reaches of the empty quarter so they do not harm useful programmers ears. Remember these prophets ramblings as even they are sometimes useful. Teach them to the young, but always remind them of the first rule.

When your sales representative tells you to remove the useful parts of your code, give them wind and ask them to sell the liberated air. When your project manager wants features that are not useful, give them a shovel and ask them to remove dirt from the parched earth only to have them place it back again for they have forgotten the first rule and should be left to sisyphean tasks. When other programmers check in code that is not useful, give them the rule and ask them if they are on the path to manager or guru.

Python Packaging

2012-10-28T20:08:00-07:00

There are two major hurdles to Python disrupting the entire HPC work stack: packaging and dynamic loading. Today I want to discuss the packaging issue. While there are many people working on these hurdles, it is my opinion that the community needs to seek out methods to solve these hurdles together in a satisfactory way. I see this problem taking shape in many different communities, but the HPC version of the problem is probably the most difficult, thus by solving it, we can provide solutions for many other communities as well.

To this end, I helped host a conference call among the young, enthusiastic NumFOCUS community. This call seemed much more of a get to know the problem rather than a listing of solutions. We are working on editting the call and hope to have it published as a InSCIght podcast soon. The call included several companies that produces packaged Python solutions, university folks from around the globe, and industrial users of Python. The interest in the call was so great that we had to switch mediums at the last moment and lost out on some interactions with other great folks. I hope to have another call in the future and a discussion at the upcoming SuperComputing 2012 conference.

What is wrong with Python Packaging

Perhaps the place to start is defining the problem. When you have a piece of code in a pure Python setting, one easily adds the top source directory to PYTHONPATH and happily imports away. This model works quite well usually. For the first years of Google App engine, code could only be installed in this manner.

The issue is that in HPC one must depend upon libraries that are highly tuned for specifics of the architecture where the code is run. These highly tuned libraries often include assembly and usually only include compiled static or shared libraries. Since, HPC often uses specific compilers dedicated to their hardware, these libraries depend on the formats compatible with those compilers. To further enhance the problem, if these libraries are distributed via MPI, then the libraries depend on the MPI implementation (which depends upon the compiler). At TACC we have a hierarchical module system to keep this all straight. You load the compiler, then the MPI implementation, and finally any libraries. If you switch to a different compiler, the entire stack is reloaded. Actually a big part of my job is to make sure these stacks are all compiled in a way that they can be switched out, something that is a feat with biology codes. In a nutshell, we have a huge dependency chain that is binary and machine dependent.

Python isn't supporting this long list of dependencies very well. For HPCers this is not new as we are use to having to hack build and package systems. In fact in the eight years I've been around the FEniCS community, I've seen four package systems come and go. It took me three weeks to get the Trilinos code installed on my system back in 2006, at the time folks opined that picking that the right Trilinos configure was NP-Hard. It is much better today. In both cases though, CMake has turned out to be the tool that has stabilized everything.

Why can't Python support this jumble of dependencies? The reasons are many, but as David Cournapeau has pointed out, Python fundamentally mixes both build and packaging together in a way that makes it impossibly hard. I don't know David's history, but he seems pretty critical of the community about not understanding the issue. Rather than rehash all the arguments, I'm going to outline my view of the solution, separating build and packaging.

The build process

The difficult part of the build process is not calling the compiler, rather configuring the environment and compiler flags to produce the desired binary.

To give a simple example, if you are compiling a threaded library, say using OpenMP, you need to know that gcc uses -fopenmp, icc uses -openmp, pgi -mp, something else on xl, and clang doesn't even support it. Thank you Apple for making my job that much harder by using a default compiler that doesn't support OpenMP.

Okay that's actually not hard to code, but it gets better. Because multicore threading is still an infant technology, if you compose several threaded libraries you can see a real performance degradation (despite @dbeaz thinking threading is easy). Thus most vendors provide a threaded version of BLAS and a sequential version, the idea being if you have a big matrix to work with make BLAS be the multithreaded portion of the code, but if you have lots of smaller matrices let the application be multithreaded. This means that each application could be built with 4 compilers X 2 BLAS threading modes, we also need to throw in debugging and optimized modes as yes -O3 versus -g -O0 really do matter.

In this trivial example, we already need to test 16 different builds of a library, not to mention figuring out how many threads to run (which is often a runtime decision with OpenMP). If you want to drive your graduate students to madness get them to install Scalapack on your cluster.

At this point, you might say "16 versions of the library, quit your crying, baby!". It should be easy to script except Python distribution tools really don't give a way to manage these different compiler flags. In many cases the tools just slaps your code in some place on the path and runs all sorts of bootstrapping patching hacks to install at all. Virtualenv give a bit of sanity as one can isolate different Python builds, but it doesn't handle any of the system libraries at all. It is my understanding that when EPD was built, they initially tried to fix this problem but quickly switched to a single binary only distribution model.

To tackle all these burdens, we need to start using real configure and build tools. They need to work on all platforms, yes Windows matters in HPC. They need to be free and open source. And they actually already exist, the community just needs to start adopting them and that will take some effort. The two tools I've seen used well are Bento produced by David and CMake produced by Kitware. I'm not going to have a bake-off on the pros and cons of each, I have far more experience with CMake but people I trust tell me Bento is far easier to use. If NumFOCUS and the PSF really wants to help the community today, it would start helping people augment distutils to use one of these tools.

The package selection process

Now that we've discussed the build process, we need to package up those builds and allow developers to use them. I was really glad the Yaroslav, one of the Debian developers, and Samuel, homebrew's Python guy, was able to join us on the call. Open source platforms have created great packaging tools, but it only gets us code monkeys about halfway. I can install the basic libraries that aren't cutting edge and don't need fine grain configuration system wide. Then I just keep the jumbled mess in my developer space. But what if a developer needs to fix a bug with one of those system installed libraries that isn't the version installed? I've spend more time futzing with the packaging system installer than fixing the bug.

The truth of the matter is we really want the jumbled mess to be managed better too. Just like on my supercomputer, I can issue a single command and have a whole new compiler stack working, why can't I do that with my current development tree. Here enters HashDist. HashDist is Dag and Ondrej's plan for fixing this problem. The idea is to build libraries and stash them some place, recording the necessary details in a small distribution file that is hashed and put in a database store. Now when you need to build different versions of a library, one can simply refer to the different hashes of the other built libraries. At this point, HashDist is vaporware that has some funding. I have been working with these guys trying to fund this project for about a year and a half.

Let me be frank. If this tool existed, reporting bugs could come with a hashdist number that would set your machine up in the bug state immediately, i.e., no more futzing with installers just working. For HPC systems this would be huge. Hell it is even a good idea for a SAAS startup company.

This is very similar to the module system that we use at TACC, lmod, but it should build into the development cycle not just running a supercomputer. For what its worth, supercomputers use module systems because when you have 5000 users on a machine, you have to keep the kids from stepping on each others toes.

Why this affects developers outside of HPC

Hopefully, I've explained the problem well enough and pointed to a few promising fixes. While these problems are accentuated in HPC systems, they exist everywhere. Every single company, I have consulted with has this problem. They solve it different ways, but they all manage their own build and packaging system. Every single open source scientific Python team, has this problems. They usually solve it the old fashioned way, indentured servants, err graduate students. But perhaps more importantly, as PyPy and other python distributions threaten CPython for popularity, even Python core is having these problems. The wheel proposals are a good first step, but its gonna take more than changing package names to fix it.

NumFOCUS was founded to help encourage businesses and institutions to put funding together to solve science software problems. This is a big problem and one that prevents a large number of people from effectively using our ecosystem. I would like to see more funds and community resources put forward to create working solutions. Finally a homework for everyone who has read this far. I invite you to send me your versions of how you fix Python packaging for your work so I can collect the best ideas.

Thoughts on the SciPy Conference

2012-10-08T01:24:00-07:00

For SciPy2012 I was given the privilege of serving as the co-chair of the program committee. The experience was exciting and rewarding as I was able to affect what I view as one of the most important conferences for the scientific world. Now, that's a pretty bold statement and I'm not going to dwell on it long here, but essentially many conferences people go to show off; SciPy is a conference where people come to learn best practices. And when young scientists are able to see science rock stars like Joshua Bloom lift up the covers and show off how his scientific process works, it changes their entire workflow creating more efficient, better scientists. With that said, I would like to focus more on my opinions about the state of the conference, its successes and failures, and how I believe the conference should be transformed.

The state of the SciPy Community

It was a bit strange being the co-chair of the program committee at SciPy2012, because prior to 2012 I had only attended one SciPy conference. Of course, I have known about the conference for years and have tried to go many times but life has always gotten in the way. Additionally, I came to the scientific Python community after most everything had evolved to a pretty high level of quality. Sure there are bits and pieces that could use some polish, but largely I feel that my contributions have all been cosmetic (although the GSOC students I have mentored have done phenomenal work). This puts me at a bit of a disadvantage to extol any long history of the SciPy community, but I'm sure someone will let me know where I get things right and wrong.

There are some interesting divides in the community. For example, SciPy can refer to many different things:

The US SciPy conference,
The international SciPy conference series,
The Python package named scipy,
The basic scientific computing stack used by most pythonistas, and/or
Any piece of code slightly related to science and Python

For this article, when I say SciPy I am talking about the US SciPy conference (since if I usually can't afford to make it to SciPy in Austin, TX, I sure can't afford to go to Brussels or India) and a mix of the other topics. SciPy was originally a set of friends who got together in a room at Caltech to chat and hack about the scipy package. They use to have these talks from the core devs where each gave a summary of the package and largely discuss technical issues around these packages. As of late, the conference has turned into much more of a place where any scientist or technical computing professional can go and hear about a very wide variety of topics on science and Python.

This change in community is both a good and a bad thing. It means that at coffee, there are a lot of strange faces and let's be honest if we were the most social crowd we wouldn't have spent a non-trivial amount of our lives pouring over code and mathematics. Second we seem to have lost some of the core devs attention. For example, in the months preceding the conference the numpy mailing list had turned into a pretty nasty flamewar. At the conference, I only saw one of the major contributors to the numpy code base. I find the fact that SciPy isn't the most important event on the calendar for these devs kind of surprising. After all this is where you core devs are the gurus everyone has come to be inspired by.

The good news is that we are able to see a more accurate view of the penetration of Python in science. But not only science, pretty much all technical computing areas of industry, academia, and government were represented. The conference had a record number of attendees and submissions, in fact this was the first SciPy that we knew of where people received rejections.

Successes and Failures of SciPy 2012

In many ways SciPy is a great conference, but in other ways it is lacking. I want to explore a few of these ideas, get feedback from the community, and put the consensus into action.

Success

If it ain't broke don't fix it ~ Anonymous

Topics

The past two years, there have been two special topics. Last year was Big Data and Core Issues; this year we did High Performance Computing and Visualization. The truth was that we picked these topics because a few motivated folks were excited about them. This is true about the domain session this year as well. Lauren and I chatted, I gave names and she went calling people down to get the right set of talks. Largely I think this works, but I don't like it.

I don't like it because the community wasn't really involved and when you have a couple of people picking the overall direction of a community, it largely becomes a clique. If we want to keep a vibrant community, I think these topics are really important as they are aimed at bringing in people from outside our community.

With that said, my proposal this year would be:

Machine Learning, and
Geographic Information Systems

Both topics that came out during the surveys and are a broad cutting technologies. Machine Learning should be a no-brainer as the Big Data world needs it more than ever, but GIS is a bit harder for me to predict being a break out success. Of course last year I thought HPC would be the small topic and Vis a bigger draw, but I was dead wrong. We could of had the whole conference on PyHPC (if you want sort of thing, check out the SuperComputing Conference pyHPC workshop).

Fabulous plenaries

This year I believe the plenaries were really off the chart. I was actually recruited to help out after the plenaries had already been chosen, so this was really Stéfan and Warren's doing. I really like the format as well. A talk about a well-known package, a science talk about the things people are able to accomplish with the tools, and a look into industry and how our community relates.

I have quite a few ideas about how to replicate this formula, but I believe we should probably have some discussion from the community.

Poster session

The poster session was great. There was a lot of interaction between people at the session. I'm somewhat beside myself on the session though. I thought it would be better during the reception, but we couldn't get the posters in the reception room and be visible during the entire conference. I really liked the poster session and hope to see it in the future. One fact of science conferences is that usually it is harder to get funding if you don't have something to present. Posters are a great way to increase participation, although I do wonder if we need to go to three parallel tracks in the future.

Domain science symposiums

The symposiums at the end of the day were much sparser than the day. This was expected, but they play an important role. As the main session includes a large number of folks that have quite different domains, these sessions allowed for topic specifics to be discussed. More and more domain conferences are including sessions on Python, but SciPy allows for sessions to grab the attention of Python experts who are often able to give fresh advice.

Failures

Develop success from failures. Discouragement and failure are two of the surest stepping stones to success. ~ Dale Carnegie

Failure is probably too harsh a word for what I discuss in this section. It's more adequately, "Things that disappointed Andy." And as most people who know me know, I'm not afraid to tell you why I'm disappointed (only I usually use harsher words). At any rate, I believe in admitting failure and want to discuss a few things I would like to rectify.

Community involvement

For the most part, the conference is put on by about four people. Sure there is a longer list on the website, but the brunt of the work falls on a few. I would like to see this change. For example, look at how PyCon is run. There are more volunteers than you can count. While Enthought is an amazing institutional sponsor of the event, they only have a small number of resources to put into the event. My impression is that these resources are over exhausted.

It is time we as a community come together to setup structures for more community participation. For example, the websites need work, the process of reviewing papers needs work, there needs to be more logistical help at the meeting, and so on. My guess is that most people don't even know that they could be helping. To even start though we need to build a volunteer network and appropriate web presence to give volunteers things to do.

Diversity

Matt Davis brought this up on Twitter, but I want to echo it louder. There was a discouraging lack of diversity at SciPy. I believe there were 100 males to every female and very few minorities. When talking to a few people about this, the general view was that there just aren't women coders around. I think this is really wrong, I think that women coders either don't know about SciPy or don't care.

Which brings up how do you attract female coders? One metric given to me is that conferences with women plenary speakers and/or women on the executive staff have 50% more women participants. I look at the places we advertised and we didn't focus on any female or minority organizations. Closing this gap will only make our community more vibrant.

Industry/Academia interactions

SciPy is a place where there are a few industrial partners but there could be many more. The problem I saw on the program committee was that anyone who submitted work on closed source software were negatively impacted. We are an open source community but hundreds (thousands?) of companies use our tools. Industrial partners that come back to the community and talk about how to interact more can only be a good thing. For one it will give us a since of the impact that SciPy has beyond the halls of academia. It will heighten our profile among young developers as they are able to come to SciPy to meet potential employers.

My suggestion here would be to have vendor booths and a recruitment event. I would even go as far to suggest hosting a few sponsor speaking sessions. Open source is often supported by industry, just as the scipy stack has been supported by Enthought for years. Inviting industry to our meeting will make it more visible.

Social Media

The lack of social media at the conference is quite surprising. Are we all old fuddies who don't know how to use social media to discuss? It is quite exciting to go to a conference where the social media around the conference is working well. It allows for more dialog among the participants and thus a greater shared experience.

With that said, I would like to see a greater use of Twitter, G+ and Facebook throughout the event, including planning. Connecting our community may even heal all those fights we have on mailing lists.

Lack of Core Developers

I've already touched on this, but one last failure is the lack of core project developers at the conference. Sure people have lots competing with their time, but this is the conference dedicated to the code we spend so much free time on. Conversations at SciPy are still having an effect on the directions of codes (in fact today scipy.users had a long conversation about the scipy domain which was hotly discussed at the sprints).

The few ideas I have on bringing the coders back include:

More sprinting (space every night all night)
Sessions dedicated to core projects
More organization around BoFs
Conference sponsorship for devs to attend conference

Vision

Okay 2000 words in and I'm sure you are getting bored. But I've said this before and I'll say it again. There is no reason SciPy shouldn't be as large as PyCon. While there are certainly more users of the Python language, our community is always one of the featured groups when Python users are discussed. The suggestions I outlined above are only a small set of comments I really have, but they are the ones I feel probably need a broader discussion.

For me the vision of SciPy should be simple. SciPy is the fuel of the scientific Python movement. It is where producers and consumers of scientific and technical code come to learn from each other and make each other better scientists and coders. SciPy as a conference should be inclusive of all technical disciplines and help mentor novice programmers. Finally, it should be the place where we inject new life into our projects, hack the wild ideas we only dream about, and submit pull requests that change the face of science.

Getting started with Python in HPC

2012-09-27T15:57:00-07:00

Getting to know Python is a pretty hairy task, as is diving into any language. Fortunately Python is considered an easy language to learn and the potential for using it in HPC settings is pretty large. Below I give an introduction to learning, speeding, and scaling Python. Each topic can be the subject of its own book, so take these talking points and notes as only hints to what one can look at. For a very detailed view of this topic for scientists I recommend the following three books:

Learning Python

There are a very large number of resources for learning the Python language. The issue of course is finding a resource that is attractive to the correct mindset. Very often something that a programmer is able to learn from has little resonance with an artist. For this reason, I am going to list off a couple of places that I find particularly good, but of course I have been programming for over a decade.

While learning the language is important, the real value of Python is the numerous libraries and the community around the tools. With that said finding what your particular community is doing is pretty important, but there are a core set of tools that are used by most Pythonistas. I list off a few of these tools for your consumption.

Also, because we like to see what people are doing with Python, I list off a few shining examples of Python in the wide world of scientific computing. I have to admit that there are a huge number of possibilities but few in the realm of HPC. At the very least, a HPC person should know Python for its scripting abilities but as the other sections of this document underscore, Python is a good candidate for HPC codes as well.

Python Tutorials

Python Doc Tutorial: This is the tutorial written by the developers of Python, best for programmers.
Think Python: A book for non-programmers.
SciPy Lectures: A series of lectures written by scientists geared towards scientists.
Py4Science Starter Kit: A version of this page written in 2008. Lots of good material but a bit less selective.

Python Tools

Anaconda Pro: A distribution of common Python packages for Big Data and Cluster programming. Includes disco (a Python map-reduce) and mpi4py which is of more interest to HPCers.
Enthought Python Distribution: A distribution of the most commonly used tools by the community (free for academics).
NumPy:
SciPy: A collection of scientific libraries.
MatPlotLib: A highly customizable 2D plotting library
IPython: An interactive Python shell and parallel code manager. The IPython notebook has become very popular and allows users to use an interface similar to Mathematica on a supercomputer.

Python Stories

SciPy Conferences: The series of conferences associated with the scientific Python community see recent videos at PyVideo.
Python in Astronomy: Joshua Bloom from UC Berkeley gave a keynote talk at SciPy 2012 on "Python as Super Glue for the Modern Scientific Workflow"
NumFocus User Stories: A foundation for scientific computing tools with a growing number of user stories.
PyCLAW: A petascale application written in Python

Performance

Despite all the great features outlined above, the (mis)perception is that Python is too slow for HPC Development. While it is true that Python might not be the best language to write your tight loop and expect a high percentage of peak flop rate, it turns out that Python has a number of tools to help switch to those lower-level languages.

To discuss performance I outline three sets of tools: profiling, speeding up the Python code via C/Fortran, and speeding up Python via Python. It is my view that Python has some of the best tools for looking at what your code's performance is then drilling down to the actual bottle necks. Speeding up code without profiling is about like trying to kill a deer with an uzi.

Python tools for profiling

profile and cProfile modules: These modules will give you your standard run time analysis and function call stack. It is pretty nice to save their statistics and using the pstats module you can look at the data in a number of ways.
kernprof: This tool puts together many routines for doing things like line by line code timing
memory_profiler: This tool produces line by line memory foot print of your code.
IPython timers: The timeit function is quite nice for seeing the differences in functions in a quick interactive way.

Speeding up Python

Cython: Cython is the quickest way to take a few functions in Python and get faster code. You can decorate the function with the cython variant of Python and it generates c code. This is very maintainable and can also link to other hand written code in c/c++/fortran quite easily. It is by far the preferred tool today.
ctypes: Ctypes will allow you to write your functions in c and then wrap them quickly with its simple decoration of the code. It handles all the pain of casting from PyObjects and managing the gil to call the c function.
FWrap and f2py: Fortran wrapping tools. FWrap is a newer tool for F90 and integrates with the rest of cython, and f2py is the older more standard tool. Unfortunately, neither of these libraries are well maintained but are used in numerous projects.

Other approaches exist for writing your code in C but they are all somewhat more for taking a C/C++ library and wrapping it in Python.

Python-only approaches

If you want to stay inside Python mostly, my advice is to figure out what data you are using and picking correct data types for implementing your algorithms. It has been my experience that you will usually get much farther by optimizing your data structures then any low level c hack. For example:

numpy: A contiguous array very fast for strided operations of arrays
numexpr: A numpy array expression optimizer. It allows for multithreading numpy array expressions and also gets rid of the numerous temporaries numpy makes because of restrictions of the Python interpreter.
blist: A b-tree implementation of a list, very fast for inserting, indexing, and moving the internal nodes of a list
pandas: Data frames (or tables) very fast analytics on the arrays.
pytables: Fast structured hierarchical tables (like hdf5), especially good for out of core calculations and queries to large data.

Scaling Python

Right now there are a few distributed Python tools but the list is growing rapidly. Here I just give a list the tools and some domain tools that are used in HPC that provide a Python interface.

Distributed computing libraries

mpi4py: Fastest most complete mpi Python wrapper.
disco: Python Hadoop-like framework.
IPython Parallel: A mpi or zero-mq based parallel Python.
pathos: Framework for heterogeneous computing
Global Arrays: A shared memory interface to distributed computing

Domain specific libraries

petsc4py: Python bindings for PETSc, the Portable, Extensible Toolkit for Scientific Computation.
slepc4py: Python bindings for SLEPc, the Scalable Library for Eigenvalue Problem Computations.
tao4py: Python bindings for TAO, the Toolkit for Advanced Optimization.
pyTrilinos: Trilinos wrappers

Changelog

2012-09-27 Andy R. Terrel * Initial version 2012-09-30 Andy R. Terrel * Added links to Fernando Perez's 2008 version of this document * Added link to Anaconda Pro * Added link to Gael Varoquaux's scipy lectures * Added link to Jeff Daily's Global Array Python bindings * Added link to FWrap and f2py * Fixed some grammatical mistakes