The most important Flock Talk

I was just voting on FLOCK talks and happened upon this talk proposal:

What does Red Hat Want?

It’s no secret that many Fedora participants work for Red Hat. Or that Red Hat provides funding for the Fedora Infrastructure. There have been many conspiracy theories over the years centering on what, exactly, does Red Hat want out of Fedora in return. This talk, by the Red Hat VP who runs the RHEL engineering team, proposes to address that eternal question. What does Red Hat want? Join Denise Dumas to learn what Red Hat is working on next and how we would like to work with the Fedora community

In my time working for Red Hat on Fedora I often found that the Fedora Community was operating in a vacuum.  We wanted to run a Linux Distribution that we had a stake in and a chance to modify to our needs.  We also knew that Red Hat invested a considerable amount of money into Fedora to support our being able to do that.  But what we were in the dark about was what Red Hat expected to get out of this partnership and what they wanted us to do to justify their continued investment.  Although over time we did get our hands dirty maintaining more of the packages that made up the distribution, in a lot of ways we never graduated beyond mricon’s 2004 tongue-in-cheek posting about Red Hat’s relationship to its community (and its own internal divisions at the time).

In the last few years, Red Hat’s portfolio of products and future directions have greatly expanded.  No longer just a producer of a Linux distribution, Red Hat is pursuing revenue sources in application middleware, both IaaS and PaaS pieces of the cloud, and containers.  They also have engineers working on a multitude of open source solutions that enhance these basic products, adding flesh to the framework they set up.  But where does the Fedora Community fit into this expanded roster of technologies?  The Fedora Product has been very focused on “A Linux Distro” for a number of years but the Fedora Community is very broad and multi-talented.  I’m hoping that Denise’s talk will provide an entrypoint for Fedora Contributors to start talking about what new directions they can take the Project in that would align with Red Hat’s needs.  There’s a number of difficulties to work out (for instance, how does Fedora keep its identity while at the same time doing more work on these things that have traditionally been “Upstream Projects”) but we can’t even begin to solve those problems until we understand where our partner in this endeavour wants to go.

Crafting the best compromise possible

This is something I’ve been noticing for a while and am finally getting around to blogging.

In the first days of FESCo, Thorsten Leemhuis was the chairman. One of the quirks of his time was that we’d encounter a topic where we voted on a solution and found that a majority agreed with one sentiment but it wasn’t unanimous. When that happened, Thorsten would be sure to ask if there was anything we could do to make the solution more acceptable to the dissenters even if they still wouldn’t vote for the proposal.

This sometimes lead to discussions of a proposal that had been approved with margins like 7 to 2 and after the discussion and changes, the vote was still 7 to 2. So from an external standpoint, this might be seen as unproductive. Why don’t we just get a decision made and move on?

But over the years I’ve watched a lot of other split decisions be made on several committees from both the inside and the outside and it’s struck me that, perhaps, we don’t do nearly enough of this sort of examination. Making changes after it was clear that a majority agreed with the basic proposal had several beneficial effects:

  1. It made the proposals more palatable to more people by getting rid of at least some issues that had made their way in to the final drafts.
  2. It forced dissenters to figure out what specific things they wanted to be changed in the proposal rather than simply being able to say “I hate this whole thing”.
  3. It made more people a part of the decision– whether or not they voted for it, if some of their ideas were in it, they felt some ownership for having help craft it.
  4. And perhaps most importantly, it let everyone know that the door of communication still worked. People found that their ideas were still valued by the other members even if they didn’t agree with each other on the overall picture.

So what can we do with this? Maybe it’s too much to ask that we look over every little decision we make where there’s disagreement and attempt to find every last bit of common ground that we can (There were certainly times when it seemed to take forever to make a decision) but what about decisions that are close votes? What about decisions that have days-long threads as part of their backstory? In these cases, consider the proposal that the majority agrees on to be a strawman. A starting point from which to start chipping away to see what changes can be made that are still acceptable to the majority while addressing many of the issues that the minority has. Remember that the goal is to craft a compromise that addresses as many concerns as possible.

Get your board member Now!

Are you on IRC? Are you at FUDCon? Do you have a project that you want to happen in Fedora?

The Fedora Board is working on choosing goals that each individual member wants to champion and bring to fruition this year in Fedora. So if you have some idea that you think a Board member’s help will make work better, grab your nearest Board member[*] and ask them to bring it up on their Sunday meeting. One of them may take it up as something they think they can work on and help accomplish in the coming year.

[*] Board members you may see wandering around at FUDCon:

  • Jared Smith
  • Toshio Kuratomi
  • David Nalley
  • Peter Robinson
  • Jon Stanley
  • Christoph Wickert

Brought to you by a cast of thousands (well, tens, at least)

The past few weeks I’ve been coordinating a new release of the Fedora Account System(FAS). Since FAS is something used within Fedora but not a whole lot of other places, development is usually driven by a relatively small handful of people: Ricky Zhou, Mike Mcgrath, and I. This release saw a large number of other contributors which has been very good as the three of us have been increasingly pulled into other projects so our time for FAS has steadily decreased.

  • Adam M. Dutko fixed several long standing bugs and feature requests
  • Luis Bazán updated several pieces of the UI
  • Sijis Aviles switched us from signing the CLA to the new FPCA
  • Pierre-Yves Chibon created a new captcha to replace the universally hated one that we were employing
  • Jun Chen added a means to clear a user’s ssh key
  • Nick Bebout started the work of removing copyright phrase-ology that we no longer want to use (“All rights reserved”) and tracking down which people needed to agree that we could switch licenses from GPLv2-only to GPLv2-or-later
  • Jim Lieb contributed code to make handling of languages easier and made FAS more configurable for use in other sites.
  • and for the first time in several releases we coordinated our release with the Fedora Translation Team on transifex so that the translations they contributed could go out with the first release instead of when a subsequent bugfix was released.

So let’s take a brief tour of some of these new features.

More Translations

Although we’ve been using transifex to manage translations for FAS for a while now, I hadn’t really understood how to leverage the full power of the Fedora Translation Team to get translations.  Thanks to some prodding by pingou, I got in touch with the translation team this time around and arranged for a string freeze before release during which they worked hard to translate FAS into their native languages. Thanks to transifex, I can show you this nice graph of their hard work:

Top translations: fas » faspot Full graph of translation stats

Clearing SSH Keys

When Fedora Infrastructure recently made the decision to invalidate public ssh keys because we had no way to tell which users might have hosted their ssh private keys on other projects servers which had been attacked and infiltrated, one of the options was for a user who didn’t actually need to use ssh to simply remove their ssh. Unfortunately, the web interface didn’t include the ability to do that so user’s who wanted to go this route had to contact one of the admins and have them remove it for them manually. Thanks to Jun Chen, users can now perform this step for themselves:
ssh clear key button

New Captcha!

There have always been many times more accounts in FAS than there were active contributors to Fedora. In itself, this wasn’t a problem. However, at some point, spammers started signing their bots up for Fedora accounts as they found that with that, they could write to the Fedora Wiki. To combat this, we added a captcha to the signup process. However, we quickly found that the captcha we added was too hard. Many people came to us to complain that they could not answer the captcha successfully. Thanks to pingou, we have a new captcha which displays a simple math equation in a much less distorted image. Writing the correct answer to the equation is all you need to do.
New captcha

These are just some of the more user visible changes. If you’re interested in the more behind the scenes changes (SELinux fixes from ricky, password strength checking, and more), check out the changes in FAS’s git repository.

Best Lightning Talk

At about 8:30 into this video, you’ll see a great lightning talk: PyCon 2011: Friday Afternoon Lightning Talks
It struck me on several levels.  It was told by an excellent straight man.  It had a great story.  It was gently deprecating.   In short it was very entertaining.
But beyond that, it also highlighted one of the core philosophies of open source that sometimes gets lost in mailing list threads about user base and popularity.  Open source is powered by people wanting to scratch their own itches.  Having a popular product is fun but people will work on something that only has a small amount of users as long as it is a good base for them to do what they want.

Python3 porting organization

Last week, a few people crawled out of the wordwork and decided we wanted to start porting third party python modules to python3. We need a bit of structure for this since some of the time, we have people who are packagers for individual Linux distributions (or even, just at the one company that they work for) doing the job of porting something they need over. The work that is done in that one place then doesn’t get upstreamed for some reason (upstream is dead, upstream only wants to work on python2 problems at the moment, you got busy and forgot about it). Having a central place to coordinate these efforts would make for a nice way to make sure that we’re working on things that no one else has done instead of everybody duplicating efforts.

With that in mind, we’ve decided that we should collaborate on the python porting mailing list to try to organize what we’re doing. If you’re interested in doing some of this python3 porting work, join up and see how you can help!


Dennis, Jared, and I flew to FUDCon this week. We all encountered delays in our flights but luckily, we all made it in the end.

I’ve already learned something a bit surprising in its magnitude but that made sense after a little thought. Jayme Ayres mentioned that the language barrier is more of an issue when using email then when speaking. When speaking to each other, there is feedback (body language, immediate questions, etc) that make misunderstandings easier to resolve. When using email, there is a lot of miscommunication that can’t be ironed out in the same way because there is no body language and because the communication lacks… the participation of both parties.

In a face-to-face conversation where the speakers are not fluent in each other’s language, both parties need to help express one person’s idea… starting with a phrase in English and in Spanish/Portuguese and trying to create an English phrase that more accurately describes what the person means. What generally seems to happen in email is one person writes what they mean to communicate as best they can. Then the second participant interprets it and assumes that their interpretation is correct. This leads to problems as often the interpretation was faulty in small details and sometimes in the larger picture as well. Jayme went on to say that the fear of misunderstanding is high enough that at least some people do not write an email expressing their postions in the first place.

Figuring out how to change this is hard but seems necessary. Jared Smith had a few ideas such as

  • encouraging people to send messages that include what they want to say in their native language as well as their translation of it.
  • having people who can speak multiple languages helping to clarify when messages like this come across the wire.
  • having a packager language sig where people can go for help translating things they want to say to each other.

I think a deeper, culture change might be needed as well. Using Active listening strategies when responding to email could help us to communicate in a more welcoming and more comprehensible manner when we’re trying to deal with language barriers (as well as simply in some of the highly charged emotional battles that seem to spring up in Fedora from time to time.)

Mailman, Fedora infrastructure, and involving non-software developers in open source (Part I)

Mizmo came to #fedora-admin yesterday to see about getting drupal with a specific plugin that puts a more web-forum type interface on top of mailman. This spawned a big discussion about a wide array of things. I’m posting a bit about it here to get it more exposure and also to try and separate out the different threads that ran through it.

Part 1: Fedora Infrastructure

First, something that has become very apparent within Fedora Infrastructure but isn’t so apparent to people outside.  Infrastructure is starved for people.  And unfortunately, simply throwing more people at infrastructure doesn’t help as much as in other parts of the project.  Here’s what happens:  Within infrastructure, we have a very few people who are trusted to do work on all of the infrastructure boxes (the so called, sysadmin-main group).  These people can log in to all but one of the servers, make changes as needed, access the database, vote on change requests during release freeze, and basically have rights to fix any problems that may crop up.  With great power comes great responsibility and new members to this group need to be present in the general Fedora sysadmin community for a good while doing a lot of things before they come to be in this group.

Outside of this group we have several others that have varying degrees of power over varyious critical items.  We have the sysadmin-noc group that monitors all the servers and has a limited ability to diagnose and help fix routine issues that may crop up (although they often need to call on someone with more access to perform actual fixes), the sysadmin-hosted group that can work on the servers related to keeping fedorahosted up and running, sysadmin-web group that can work on the main app servers that make infrastructure services go, syadmin-cvs that deals with the cvs server for fedora packages, sysadmin-db (which in practice is the same as sysadmin-main due to having access to sensitive information).  We also have satellite groups in the form of committers to the applications that we write (the packagedb, fas, python-fedora, bodhi, elections, and mirrormanager committers).  These applications are written by infrastructure coders to meet needs identified by the infrastructure group for Fedora. In addition, there are a few groups that interact very closely with infrastructure — the releng team deploys the release and needs to coordinate closely with infrastructure on mirror space and times that we can make changes, some of the groups that develop applications that we run (members of the transifex, fedoracommunity/moksha, and zikula community) work with us to help solve issues and bugs in our deployments and to greater and lesser extents, help us to maintain the apps.

So, where’s the bottleneck in infrastructure getting new people?  There’s actually three places, two of which are related:

  1. We need more people involved who want to solve specific coding problems for infrastructure.  These people would need to be willing to be a jack-of-all-trades.  They need to be members of infrastructure that get involved with upstream projects.  Sometimes there might be a performance issue that we need to have addressed.  Sometimes we might identify a security problem and need to get a fix out quickly.  Other times we might identify a high value feature that would help fedora contributors and need someone to develop it.  The people doing any of this work would need to be able to sit down and involve themselves with both infrastructure and any upstreams to get commit rights or, at least, be trusted enough to get their patches looked at and added.  They’d need to be able to dive into unfamiliar code, get an idea of how it works, and produce working patches.
  2. We need more people to maintain (not just deploy) applications.  In many ways these people end up doing the same sorts of jobs as the people in #1.  However, the emphasis is slightly different.  Where the first set of people are primarily coders, these people are primarily system administrators.  Now, in Fedora Infrastructure we do have a lot of system administrators who come by to be a part of the team.  Where we end up with problems is that most of them aren’t able to commit to being part of the team over a long period.  This is difficult for us because we end up being able to deploy more projects but as we do, the people who are committed to maintaining the services get more and more stretched.  For a non-sysadmin, a question that often comes up here is — well, but isn’t deploying the application the hard part?  After it’s deployed, it should just work, right?  The answer to that is almost always no.  All software has bugs — so there’s always going to be the need to do updates.  Updates are not always backwards compatible so there’s always going to be the need to test updates and update configuration and code that we’ve built to help us manage the software when we do them.  Critical bugs (often, security related) do get discovered after a piece of software has been deployed which makes for some late nights rushing around fixing an issue that must be applied to the production instance ASAP.  As the service gets used more (or other software running on the same hsot gets used more) we can run into scaling issues that weren’t apparent when we first deployed.  All of these things contribute to the maintainance burden of deploying a new service to be used and all of them are helped immensely by having someone who can maintain the software as a long term commitment.
  3. We need to get more people who can gateway changes to many things.  These are the members of the core teams, sysadmin-web, sysadmin-db, cvsadmins, and ultimately, sysadmin-main.  This ties in heavily with #2.  in order to be sponsored into one of these groups you need to build up trust within the sysadmin community.  You’ll be given access to services that can bring down Fedora in any number of ways from simply making mistakes that cause outages to maliciously causing problems for core services so we have to trust that you’ll do the right thing with your responsibilities.  Building trust is not an instant thing.  It takes many man-hours of hard work, being in the right place to do something helpful, and generally showing that you are not just someone with valuable talents, but also someone that is responible enough to use their talents for the benefit of everyone and not just a few.

Over the past couple years we’ve tried a variety of things to alleviate these issues, none with a great deal of success.  Commitment is hard when you have mouths to feed.  It’s hard to be effective at working on issues when you don’t have access to deploy your apps in a production environment.  Feel free to bring some suggestions (the best suggestions come with prototypes! 🙂 to #fedora-admin or the infrastructure@fp.o mailing list and we’ll see if next year we can look back and say this was the year we figured out how to grow infrastructure at a sustainable rate.