Why upstream Ansible stopped shipping rpms

The upstream Ansible Project used to ship rpms and tarballs on their server, releases.ansible.com. For Ansible-2.10+, they’ve (I’m part of the project although not the originator of this decision) decided not to ship rpms anymore and to push people to use pypi.python.org as the official source of the tarballs. This came up in a recent thread on twitter with a question of whether this meant that Ansible was forgetting who it was meant to serve (sysadmins) as sysadmins want to get their software in their platforms’ native packaging format rather than the language packaging format.

I don’t think this decision, in and of itself, means that.

For tarballs, I’m not sure of the rationale but it doesn’t seem like a problem to me. Most python software packaged in Fedora has pypi as the canonical source of the tarballs.  Pypi serves as a hosting service for the source code rather than the point from which most Fedora users are going to install the software.

The lack of upstream rpms seems to be what triggered a nerve for some people. I was present for those discussions and I think the reasons make a lot of sense for end users.  A few of those reasons:

  • Fedora and EPEL have been building superior rpms since forever. There’s a few reasons that the Fedora and EPEL rpms are better in my mind:
    • They know the distro and the distro users that they’re targeting so they can do more specific things. For instance, they include a dependency on python-jmespath to make a popular filter plugin work out of the box. The upstream rpm did not as python-jmespath is available in EPEL but not in RHEL.
    • The Fedora rpms were able to switch over to Python3 when the distro did while the upstream rpm had to wait while other rpm-based distros caught up (the spec file was shared) and, with no specific roadmap for dropping python2 support, hesitated over whether switching the rpms would inconvenience its users.
    • The upstream version was an automated build. Not to say that all automated builds are always bad, but our build was really geared towards putting our software into an rpm rather than creating a high quality rpm in and of itself. Although we weren’t quite as bad as simply dropping a binary blob onto the filesystem, we did shortchange things like the rpm changelog because that wasn’t necessary to get the build to work.
  • Our rpms weren’t all that popular. We attributed that to most people using the rpms that came with Fedora or EPEL which we thought were much better overall, anyway.
  • It felt like our rpms were confusing the situation for end users. if you were a RHEL or CentOS user, you could get ansible rpms from three places: A Red Hat channel, EPEL, or releases.ansible.com. All three of these sources shipped different rpms. Dependencies (as noted above) were different, the directories that end-users and other packages could install things into were present in some of the rpms and not others, docs might be built or not or not even shipped in some cases. An end user who installed from releases.ansible.com (the least popular of these sources) could be confused as to why certain things didn’t seem to work the way a blog post or article (written by someone with the EPEL rpm) implied it should.

Getting back to the fear that removing rpms from releases.ansible.com was an indication that ansible is forgetting that it is a tool for sysadmins and needs to be shipped in ways that sysadmins will find palatable…. I don’t think that the removal of rpms and tarballs is an indication as the above rationale seems like it will make things better for sysadmins in the end. However, ansible-2.10 is a big realignment of how ansible is developed and shipped and I think those changes are going to have costs for sysadmins [2]_, [3]_. nirik (Kevin Fenzi, the Fedora/EPEL ansible package maintainer) and I have been talking on and off about how the Fedora/EPEL ansible rpm should be adapted to minimize those costs but it is a large change and changes are often both hard in the transition and, after the transition is over, may be better in many areas but worse in some others. Ideas about how we can smooth out the things that are worse while taking advantage of the things that are better is appreciated!

Footnote [2]:

The problems driving upstream to make the major changes that are present in 2.10: https://www.ansible.com/blog/thoughts-on-restructuring-the-ansible-project

Footnote [3]:

A newer document, focused on the implementation of the changes proposed above and how they affect end users: https://github.com/ansible-collections/overview/blob/master/README.rst

The most important Flock Talk

I was just voting on FLOCK talks and happened upon this talk proposal:

What does Red Hat Want?

It’s no secret that many Fedora participants work for Red Hat. Or that Red Hat provides funding for the Fedora Infrastructure. There have been many conspiracy theories over the years centering on what, exactly, does Red Hat want out of Fedora in return. This talk, by the Red Hat VP who runs the RHEL engineering team, proposes to address that eternal question. What does Red Hat want? Join Denise Dumas to learn what Red Hat is working on next and how we would like to work with the Fedora community

In my time working for Red Hat on Fedora I often found that the Fedora Community was operating in a vacuum.  We wanted to run a Linux Distribution that we had a stake in and a chance to modify to our needs.  We also knew that Red Hat invested a considerable amount of money into Fedora to support our being able to do that.  But what we were in the dark about was what Red Hat expected to get out of this partnership and what they wanted us to do to justify their continued investment.  Although over time we did get our hands dirty maintaining more of the packages that made up the distribution, in a lot of ways we never graduated beyond mricon’s 2004 tongue-in-cheek posting about Red Hat’s relationship to its community (and its own internal divisions at the time).

In the last few years, Red Hat’s portfolio of products and future directions have greatly expanded.  No longer just a producer of a Linux distribution, Red Hat is pursuing revenue sources in application middleware, both IaaS and PaaS pieces of the cloud, and containers.  They also have engineers working on a multitude of open source solutions that enhance these basic products, adding flesh to the framework they set up.  But where does the Fedora Community fit into this expanded roster of technologies?  The Fedora Product has been very focused on “A Linux Distro” for a number of years but the Fedora Community is very broad and multi-talented.  I’m hoping that Denise’s talk will provide an entrypoint for Fedora Contributors to start talking about what new directions they can take the Project in that would align with Red Hat’s needs.  There’s a number of difficulties to work out (for instance, how does Fedora keep its identity while at the same time doing more work on these things that have traditionally been “Upstream Projects”) but we can’t even begin to solve those problems until we understand where our partner in this endeavour wants to go.

Flock Ansible Talk

Hey Fedorans, I’m trying to come up with an Ansible Talk Proposal for FLOCK in Rochester. What would you like to hear about?

Some ideas:

* An intro to using ansible
* An intro to ansible-playbook
* Managing docker containers via ansible
* Using Ansible to do X ( help me choose a value for X 😉
* How to write your own ansible module
* How does ansible transform a playbook task into a python script on the remote system

Let me know what you’re interested in 🙂

Hacky, hacky playbook to upgrade several Fedora machines

Normally I like to polish my code a bit before publishing but seeing as Fedora 21 is relatively new and a vacation is coming up which people might use as an opportunity to upgrade their home networks, I thought I’d throw this extremely *unpolished and kludgey* ansible playbook out there for others to experiment with:

https://bitbucket.org/toshio/local-playbooks/src/3d3ae76a56034784ab60fcfa1129221c59a40f3b/provisioning/fedora-upgrade.yml?at=master

When I recently looked at updating all of my systems to Fedora 21 I decided to try to be a little lighter on network bandwidth than I usually am (I’m on slow DSL and I have three kids all trying to stream video at the same time as I’m downloading packages). So I decided that I’d use a squid caching proxy to cache the packages that I was going to be installing since many of the packages would end up on all of my machines. I found a page on caching packages for use with mock and saw that there were a bunch of steps that I probably wouldn’t remember the next time I wanted to do this. So I opened up vim, translated the steps into an ansible playbook, and tried to run it.

First several times, it failed because there were unsolved dependencies in my packageset (packages I’d built locally with outdated dependencies, packages that were no longer available in Fedora 21, etc). Eventually I set the fedup steps to ignore errors so that the playbook would clean up all the configuration and I could fix the package dependency problems and then re-run the playbook immediately afterwards. I’ve now got it to the point where it will successfully run fedup in my environment and will cache many packages (something’s still using mirrors instead of the baseurl I specified sometimes but I haven’t tracked that down yet. Those packages are getting cached more than once).

Anyhow, feel free to take a look, modify it to suit your needs, and let me know of any “bugs” that you find 🙂

Things I’ll probably do to it for when I update to F22:

Quick question for Fedora Planet

Hey Fedora Planet, quick survey: I’ve been doing some blogging on general python programming and on learning to use Ansible. Are either of these topics that you’d be interested in showing up on planet? Right now the feed I’m sending to planet only includes Fedora-specific posts, linux-specific, or “free software political” but I could easily change the feed to include the ansible or python programming posts if either of those are interesting to people.

Leave me a comment or touch base with me on IRC: abadger1999 on freenode.

A going away present

For those who haven’t heard through Flock or the rumor mill, today is my last day at Red Hat and also the beginning of a hiatus from working on Fedora. Since I’ve been asked this many times in the past few weeks, this is because I’ve become a bit burnt out having worked on Fedora as both my day job and my hobby for the past seven years. It’s time for me to pull back, let fresh faces fill the roles I held, and do something else for a while to add some spice and variety. I may come back to Fedora or to Red Hat in the future but at the moment I’m only looking far enough ahead to see that I need to go forth and have some new experiences.

I do want to say thank you to all the wonderful people who have worked not just to make the Fedora distribution a solid piece of software but also filled Fedora with friendly faces and kind words. Truly, although I’m physically far removed from the rest of you, you are my neighbors, my community, and my friends. Even though I’m stepping away from working on Fedora, I hope to keep in touch with you via IRC for many many years.

I’d also like to announce that I woke up this morning to find that I’d been made the gatekeeper for a new Fedora Badge. As the badge submitter describes it:

Dancing with ToshioI dream of a future where Toshio could fully express his techniques with the complicity and trust of many dance partners, responding to his moves and being pushed forward by him in the arts of dancing; exchanging, learning, growing as a vibrant community.

-Aurelien Bompard

Taking away the specifics of dancing and myself, this is my hope for everyone who participates in Fedora: to be able to grow in sympathy with a larger community.

With that in mind, if we’ve danced together and you would like this badge, please contact me (abadger1999 on IRC, toshio.fedoraproject.org via email). I can’t remember everyone’s FAS usernames but I’m extremely happy to award you the badge if you remind me what of what it is 🙂

Rest, my friend, the next five years are ours to pass along your wisdom

Just installed a new system and was having ssh connections timeout. Then I remembered talking about this same issue last year on IRC. The anecdote is amusing so I figured I would post the logs:

[Mon April 22 2013] * abadger1999 wishes he knew why his ssh connections to infra keep on hanging.
[Mon April 22 2013] <abadger1999> it’s a timeout of some sort… I just don’t know what.
[Mon April 22 2013] <skvidal> abadger1999: did you reinstall recently?
[Mon April 22 2013] <abadger1999> skvidal: nope
[Mon April 22 2013] <abadger1999> skvidal: would that help?
[Mon April 22 2013] * abadger1999 still on f17
[Mon April 22 2013] <skvidal> I have found I often need to set
[Mon April 22 2013] <skvidal> net.ipv4.tcp_keepalive_time = 300
[Mon April 22 2013] <skvidal> in /etc/sysctl.conf
[Mon April 22 2013] <skvidal> to not get timeouts
[Mon April 22 2013] <abadger1999> Thanks. I’ll try that .

[…]

[Wed April 24 2013] <abadger1999> skvidal: btw, your sysctl recipe seems to have fixd my ssh timeout issues. Thanks!
[Wed April 24 2013] <skvidal> abadger1999: 🙂
[Wed April 24 2013] <skvidal> abadger1999: last time it happened to me I had to google for the solution
[Wed April 24 2013] <skvidal> abadger1999: and I found a post from myself from 5yrs earlier
[Wed April 24 2013] <skvidal> abadger1999: _that_ is kinda freaky
[Wed April 24 2013] <pingou> isn’t that what blog are for? 🙂
[Wed April 24 2013] <dwa> nice
[Wed April 24 2013] <abadger1999> Cool 🙂
[Wed April 24 2013] <skvidal> “wow, this dude knew what was going on…. but he sure writes like he’s an ass”
[Wed April 24 2013] <skvidal> “oh….. wait”

https://lists.dulug.duke.edu/pipermail/dulug/2007-July/010956.html

https://lists.dulug.duke.edu/pipermail/dulug/2003-August/007359.html


Picture of Seth from 2005 looking into the distance

Seth, you were more of a teddy bear than an ass.

git commit doesn’t commit? (GitPython bug)

Mostly posting this to remind myself of the fix the next time I run into this but htis might help some other people as well.

Every once in a while I’ll be working on a git repo in the fedora packages repository and when I git commit -a it, I’ll end up with an empty commit and the files with changes aren’t actually committed. Other intuitive variations of this like git add FILE && git commit have the same buggy behaviour.

The reason this is occurring has something to do with the GitPython library which is used by fedpkg to add some changes to your clone of the git repo when you add new source files. It’s somehow changing the index in a way that causes this behaviour. To get out of this there’s a few simple but non-intuitive things you can try:

git reset FILE && git add FILE

git stash && git stash pop

After running one of those pairs of commands you should once more be able to git commit -a.

Details in this GitPython bug report

Security FAD

All packed up and waiting for my plane to Raleigh. Going there to work on enabling two-factor authentication for the hosts that give shell access inside of Fedora’s Infrastructure. For the first round, I think we’re planning on going for simple and minimal to show what we can do. Briefly, the simplest and minimalist is:

* Server to verify a one time password (we already have one for yubikeys)
* CGI to take a username, password, and otp to verify in fas and the otp server
* pam module for sudo that verifies the user via the cgi
* database to store the secret keys for the otp generation and associate them with the fas username

We’re hoping to go a little beyond the minimal at the FAD:

* Have a web frontend to configure the secret keys that are stored for an account.
* Presently we’re thinking that this is a FAS frontend but we may end up re-evaluating this depending on what we decide to do for web apps and what to require for changing an auth source.
* Allow both yubikey and google-authenticator as otp

I’m also hoping that since we’ll have most of the sysadmin side of infrastructure present that we’ll get a chance to discuss and write down a few OTP policies for the future:

* Do we want to make two-factor optional for some people and required for others?
* How many auth sources do we require in order to change a separate auth source (email address, password, secret for otp generation, phone number, gpg key, etc)?

If we manage to get through all of that work, there’s a few other things we could work on as well:

* Design and implement OTP for our web apps