Mailman, Fedora infrastructure, and involving non-software developers in open source (Part I)

Mizmo came to #fedora-admin yesterday to see about getting drupal with a specific plugin that puts a more web-forum type interface on top of mailman. This spawned a big discussion about a wide array of things. I’m posting a bit about it here to get it more exposure and also to try and separate out the different threads that ran through it.

Part 1: Fedora Infrastructure

First, something that has become very apparent within Fedora Infrastructure but isn’t so apparent to people outside.  Infrastructure is starved for people.  And unfortunately, simply throwing more people at infrastructure doesn’t help as much as in other parts of the project.  Here’s what happens:  Within infrastructure, we have a very few people who are trusted to do work on all of the infrastructure boxes (the so called, sysadmin-main group).  These people can log in to all but one of the servers, make changes as needed, access the database, vote on change requests during release freeze, and basically have rights to fix any problems that may crop up.  With great power comes great responsibility and new members to this group need to be present in the general Fedora sysadmin community for a good while doing a lot of things before they come to be in this group.

Outside of this group we have several others that have varying degrees of power over varyious critical items.  We have the sysadmin-noc group that monitors all the servers and has a limited ability to diagnose and help fix routine issues that may crop up (although they often need to call on someone with more access to perform actual fixes), the sysadmin-hosted group that can work on the servers related to keeping fedorahosted up and running, sysadmin-web group that can work on the main app servers that make infrastructure services go, syadmin-cvs that deals with the cvs server for fedora packages, sysadmin-db (which in practice is the same as sysadmin-main due to having access to sensitive information).  We also have satellite groups in the form of committers to the applications that we write (the packagedb, fas, python-fedora, bodhi, elections, and mirrormanager committers).  These applications are written by infrastructure coders to meet needs identified by the infrastructure group for Fedora. In addition, there are a few groups that interact very closely with infrastructure — the releng team deploys the release and needs to coordinate closely with infrastructure on mirror space and times that we can make changes, some of the groups that develop applications that we run (members of the transifex, fedoracommunity/moksha, and zikula community) work with us to help solve issues and bugs in our deployments and to greater and lesser extents, help us to maintain the apps.

So, where’s the bottleneck in infrastructure getting new people?  There’s actually three places, two of which are related:

1. We need more people involved who want to solve specific coding problems for infrastructure.  These people would need to be willing to be a jack-of-all-trades.  They need to be members of infrastructure that get involved with upstream projects.  Sometimes there might be a performance issue that we need to have addressed.  Sometimes we might identify a security problem and need to get a fix out quickly.  Other times we might identify a high value feature that would help fedora contributors and need someone to develop it.  The people doing any of this work would need to be able to sit down and involve themselves with both infrastructure and any upstreams to get commit rights or, at least, be trusted enough to get their patches looked at and added.  They’d need to be able to dive into unfamiliar code, get an idea of how it works, and produce working patches.
2. We need more people to maintain (not just deploy) applications.  In many ways these people end up doing the same sorts of jobs as the people in #1.  However, the emphasis is slightly different.  Where the first set of people are primarily coders, these people are primarily system administrators.  Now, in Fedora Infrastructure we do have a lot of system administrators who come by to be a part of the team.  Where we end up with problems is that most of them aren’t able to commit to being part of the team over a long period.  This is difficult for us because we end up being able to deploy more projects but as we do, the people who are committed to maintaining the services get more and more stretched.  For a non-sysadmin, a question that often comes up here is — well, but isn’t deploying the application the hard part?  After it’s deployed, it should just work, right?  The answer to that is almost always no.  All software has bugs — so there’s always going to be the need to do updates.  Updates are not always backwards compatible so there’s always going to be the need to test updates and update configuration and code that we’ve built to help us manage the software when we do them.  Critical bugs (often, security related) do get discovered after a piece of software has been deployed which makes for some late nights rushing around fixing an issue that must be applied to the production instance ASAP.  As the service gets used more (or other software running on the same hsot gets used more) we can run into scaling issues that weren’t apparent when we first deployed.  All of these things contribute to the maintainance burden of deploying a new service to be used and all of them are helped immensely by having someone who can maintain the software as a long term commitment.
3. We need to get more people who can gateway changes to many things.  These are the members of the core teams, sysadmin-web, sysadmin-db, cvsadmins, and ultimately, sysadmin-main.  This ties in heavily with #2.  in order to be sponsored into one of these groups you need to build up trust within the sysadmin community.  You’ll be given access to services that can bring down Fedora in any number of ways from simply making mistakes that cause outages to maliciously causing problems for core services so we have to trust that you’ll do the right thing with your responsibilities.  Building trust is not an instant thing.  It takes many man-hours of hard work, being in the right place to do something helpful, and generally showing that you are not just someone with valuable talents, but also someone that is responible enough to use their talents for the benefit of everyone and not just a few.

Over the past couple years we’ve tried a variety of things to alleviate these issues, none with a great deal of success.  Commitment is hard when you have mouths to feed.  It’s hard to be effective at working on issues when you don’t have access to deploy your apps in a production environment.  Feel free to bring some suggestions (the best suggestions come with prototypes! :-) to #fedora-admin or the infrastructure@fp.o mailing list and we’ll see if next year we can look back and say this was the year we figured out how to grow infrastructure at a sustainable rate.

Why I haven’t given up on Fedora (but won’t run for election)

I’ve been asked by several people to run for the Fedora Board and after a lot of wrangling with the issue, I’ve decided running for election to the Board or FESCo is the last thing I should do with my Fedora presence right now.  Since I’ve been asked to run, I feel I owe those people an explanation of why I won’t.  Luckily for me, a few messages have crossed the mailing list recently that express a large part of why it’s a bad idea for me to participate in this way.

At first, I told people that I’d consider running if we had a slate of like minded people running for election because, as I put it, I wouldn’t want to just be another Kevin Kofler — marginalized because I disagree and increasingly frustrated because people don’t feel the need to come to any sort of compromises about the issues that I raise.  Jef Spaleta and Kevin Kofler both echo the same idea of running with a slate of candidates with a similar platform, for instance this quote:

But I think your entirely correct about an organized slate of candidates being the correct path to take to address the expressed general concern. This is typically how changes in direction in brick and mortar general election scenarios are carried forward. An organized slate of candidates with a stated agenda.

– Jef Spaleta, May 3, 2010

Kevin Kofler’s Open Letter: Why I, Kevin Kofler, am not rerunning for FESCo, message has a lot of the platform that I would run under:

Any attempts to discuss those issues where everyone was against me went nowhere. In most cases, people rushed out a vote without even considering the real issue at hand and then shot down any discussion with “we already voted, we want to move on”. In those few cases where there actually was a discussion, my position was always dismissed as being ridiculous and not even worth considering, my arguments, no matter how strong, were entirely ignored.

– Kevin Kofler, bullet three, sub-bullet two

• FESCo currently values expediting decisions over coming as close to consensus as possible.  I would reverse those values.  Valuing decisions that work hard to satisfy people is one of the things that I really liked about decisions made when Thorsten Leemhuis was the FESCo chair.  Everybody was heard to the extent that we could state what the differences in viewpoints were and we often made changes to proposals to make them more palatable to people — even if those changes didn’t affect the final vote.  You can’t make all of the people happy all of the time but you can work to address as many of the individual concerns as possible.  The cloture rule that FESCo recently passed is detrimental in my view.  Instead, if a discussion is contentious, points and counter points should be written down in the wiki so we don’t keep rehashing old arguments but we do address as many of the issues as we can before a final policy is passed.

• There’s currently too much Label and Dismiss going on.  Saying something is “ridiculous” or saying that what someone says is “rehashing old arguments” is ignoring whatever it is that they think they’re adding to the discussion.  Wiki documentation of contentious issues would help with the rehashing portion of this and realizing that label and dismiss is destructive to constructive discourse would help people to listen to what others are saying.

Maintainers are continuously being distrusted.

– Kevin Kofler, bullet four, sub-bullet one

• I would emphasize a slightly different aspect of this in my platform:  We should be worrying more about education of contributors  than enforcement of rules upon them.  Although one of the values to contributors of Fedora is that they can use the distribution they produced a much bigger value is that they learn a lot from contributing.  Learning to package, for instance, helps you to build software from source, learn how to write good build scripts, think from a system administrator viewpoint when you write software, understand the way pieces of a typical Linux system interact, understand the importance of API and ABI stability, and more.  We should be teaching people about the tradeoffs in the decisions they make rather than enforcing a single decision upon them.

All the power in Fedora is being centralized into 2 major committees: the Board and FESCo. FESCo is responsible for a lot of things all taking up meeting time, leading to lengthy meetings and little time for discussion. Many of those things could be handled better in a more decentralized way. Power should be delegated to SIGs and technical committees wherever possible, FESCo should only handle issues where no reponsible subcommittee can be found or where there is disagreement among affected committees.
– Kevin Kofler, bullet four, sub-bullet two

• Why do I contribute to Fedora and not Ubuntu/OpenSuSE/etc?  It’s because Fedora positioned itself as being more open to contributors influencing where the distribution will go.  Decentralizing most decisions and striving to concentrate FESCo and the Board’s efforts on conflict resolution puts more power back into the hands of contributors who are doing the work.

The prevailing opinion of the electorate of Fedora contributors keeps getting ignored. Feedback on the Fedora devel mailing list is never seen as in any way binding, it’s often dismissed as noise or “trolling”. The predominant opinion in FESCo is “you voted for us, now we get to do whatever we want”, which is flawed in many ways:

• It assumes there were true alternatives to vote for instead. This
  assumption does not look true to me.
• It assumes the voters were aware of the positions of all the candidates.
  I’m fairly sure this was not the case. While I appreciate what has been
  done in an attempt to solve this issue (questionnaire, townhalls), this
  has proven by far insufficient to build an opinion on the candidates. I
  think there’s a reason representative democracies normally work with
  parties/factions and I think something like that might help a lot,
  depending on what kind of factions would show up.
• It assumes representative democracy is a well-working model in the first
  place, especially in its most hardcore form (“now we get to do whatever we
  want”). I believe elected representatives should really REPRESENT the
  people who voted them. I realize politicians aren’t doing that, but are
  they really a good model to follow?

I believe listening more to the feedback on the devel ML and taking it into
account during decision-making would reduce frustration with FESCo a lot.
– Kevin Kofler, bullet four, sub-bullet three

• In the last two FESCo elections combined, I voted for only seven people.  Of the five people who got in those elections, only two turned out to agree with my platform after I elected them.  So, for me, the first two bullets were accurate.
• The last bullet is more of a judgement of what is expected of someone elected to FESCo and it’s not what I think the expectation should be.  They should be consistent with their stated platforms, not necessarily with the will of who elected them.
• However, I also agree with the opening and closing paragraphs of Kevin’s point.  FESCo and the Board are not currently responding to the desires expressed by people not in the decision making bodies.  I go a little farther than Kevin in saying that the desires of the majority of contributors is not the only thing being missed here, but also the desires of important minorities (“important” defined as, if the minority group were to stop contributing to Fedora, how much would we suffer?) Three solid ideas that could help are the wiki consolidation of contentious points mentioned earlier, delegation of powers as Kevin mentioned earlier, and non-binding referendum for Board level decisions which would help the Board evaluate how many contributors are negatively impacted by a decision.

Now, with all this said about what a platform would look like, here’s why I’ve changed my mind and decided running for the Board or FESCo, even as part of a group of people working for changing the status quo would be a mistake.  Jef has a quote that sums up my feelings nicely:

It’s difficult being cast into the role of loyal opposition (whether by choice or by calculation.) especially when you don’t enjoy that role and being in that position burns you out.

– Jef Spaleta, May 3 2010

This is the heart of the matter.  I have put a great deal of myself into making Fedora a good place to contribute.  I’m loyal to Fedora because it’s one of my children; a project that I’ve worked to shape, nurture, and grow from before the time that it even had a name.  However, I am in opposition to many of the recent decisions made for Fedora by the Board and by FESCo.  Even though I’m not currently a member of these groups, I’m feeling burnout from discussing the points of dissention with them.  I can’t attend a FESCo meeting or read the minutes of the Board without feeling my heart rate speed up, my jaw tighten, and a nervous tenseness build up that demands to be released.  This level of trauma is bad in multiple ways:

1. It is bad for my well being
2. It makes me want to stop contributing to Fedora
3. I can’t make good decisions about what is mandatory to get changed, what is important to get changed, and what, even if I disagree with, doesn’t cause too much damage to the overall picture.

So until I stop feeling this badly about where Fedora is going, I’m going to stay away from the decision making process in Fedora and just deal with the portions that are not causing me this kind of grief: coding things that help Fedora contributors get their jobs done, working on problems being experienced in Fedora Infrastructure, packaging and reviewing when I have a need to.

One further note, specifically about running for the Board.  FESCo is a body 100% elected by the participants in the project.   As such, it may take a concerted effort by people wanting a different vision than the current members to work on building a platform and slate of candidates but effecting change only requires you to fill half of the available seats with people who agree with you.  The Board, on the other hand, is only 50% +1 elected.  The remaining seats are appointed by the Fedora Project Leader.   Additionally, the Fedora Project Leader is the leader of the Fedora Board and is able to add his opinion at the Board Meetings.  This makes for a large hurdle for anyone looking to shift the Board’s vision in a direction that is radically at odds with the FPL’s.  People who agree with your platform have to win election to all of the available seats in two election cycles to even have an equal number of voices on the Board.