Rest, my friend, the next five years are ours to pass along your wisdom

Just installed a new system and was having ssh connections timeout. Then I remembered talking about this same issue last year on IRC. The anecdote is amusing so I figured I would post the logs:

[Mon April 22 2013] * abadger1999 wishes he knew why his ssh connections to infra keep on hanging.
[Mon April 22 2013] <abadger1999> it’s a timeout of some sort… I just don’t know what.
[Mon April 22 2013] <skvidal> abadger1999: did you reinstall recently?
[Mon April 22 2013] <abadger1999> skvidal: nope
[Mon April 22 2013] <abadger1999> skvidal: would that help?
[Mon April 22 2013] * abadger1999 still on f17
[Mon April 22 2013] <skvidal> I have found I often need to set
[Mon April 22 2013] <skvidal> net.ipv4.tcp_keepalive_time = 300
[Mon April 22 2013] <skvidal> in /etc/sysctl.conf
[Mon April 22 2013] <skvidal> to not get timeouts
[Mon April 22 2013] <abadger1999> Thanks. I’ll try that .


[Wed April 24 2013] <abadger1999> skvidal: btw, your sysctl recipe seems to have fixd my ssh timeout issues. Thanks!
[Wed April 24 2013] <skvidal> abadger1999: :)
[Wed April 24 2013] <skvidal> abadger1999: last time it happened to me I had to google for the solution
[Wed April 24 2013] <skvidal> abadger1999: and I found a post from myself from 5yrs earlier
[Wed April 24 2013] <skvidal> abadger1999: _that_ is kinda freaky
[Wed April 24 2013] <pingou> isn’t that what blog are for? :)
[Wed April 24 2013] <dwa> nice
[Wed April 24 2013] <abadger1999> Cool :-)
[Wed April 24 2013] <skvidal> “wow, this dude knew what was going on…. but he sure writes like he’s an ass”
[Wed April 24 2013] <skvidal> “oh….. wait”

Picture of Seth from 2005 looking into the distance

Seth, you were more of a teddy bear than an ass.

git commit doesn’t commit? (GitPython bug)

Mostly posting this to remind myself of the fix the next time I run into this but htis might help some other people as well.

Every once in a while I’ll be working on a git repo in the fedora packages repository and when I git commit -a it, I’ll end up with an empty commit and the files with changes aren’t actually committed. Other intuitive variations of this like git add FILE && git commit have the same buggy behaviour.

The reason this is occurring has something to do with the GitPython library which is used by fedpkg to add some changes to your clone of the git repo when you add new source files. It’s somehow changing the index in a way that causes this behaviour. To get out of this there’s a few simple but non-intuitive things you can try:

git reset FILE && git add FILE

git stash && git stash pop

After running one of those pairs of commands you should once more be able to git commit -a.

Details in this GitPython bug report

Learning “new” features of python

I started learning python with python-2.1. Since that time a lot of features have been added. Many of those changes were well publicized: decorator syntax, context-managers, generator statements, unification of try-except-finally…. Reading the major sections of the What’s new in Python document is sufficient to call all of those out. However, there’s often other very useful changes that only get a few sentences and a bullet in the rather large list of “Other Language Changes” a the tail end of the document. It’s very easy to either miss those changes or forget about them if you don’t immediately have a use for them.

Today bochecha pointed me at one of those tiny little changes that be very useful.

I was dealing with sorting of records returned from multiple database queries. I wanted to sort the records based on date, time, and name in that order. In SQL, that would be a simple order by date, time, name but this code needed to use separate queries so I had to operate on the python list instead. No problem! Pull out the handy .sort() function and we’re all set. My code started off a bit like this:

def _get_mtg_sort_key(mtg):
    return (, mtg.time,

# meetings contains an unsorted list of database records.

That’s pretty simple but bochecha asked why I didn’t use operator.attrgetter() instead. In my naivete I pointed out that I needed to get the values of three fields so I had to use a custom function. bochecha quickly responded that since python2.5, operator.attrgetter could handle multiple fields:

import operator

meetings.sort(key=operator.attrgetter('date', 'time', 'name'))

Although the changes here are minimal, they allow us to write something very standardized instead of an ad hoc function. That will have savings down the road when someone else needs to read the code and figure out what’s going on.

Now, this left me thinking, every new release, the python language and standard lib adds a large number of these little changes. Things like python-2.1 adding support for a tuple of type information or python-2.6 adding the httponly attribute to Cookie.Morsel objects. Anyone else have a tiny change like this that they’d like to share how they now use it to save time or make more readable code?

Security FAD

All packed up and waiting for my plane to Raleigh. Going there to work on enabling two-factor authentication for the hosts that give shell access inside of Fedora’s Infrastructure. For the first round, I think we’re planning on going for simple and minimal to show what we can do. Briefly, the simplest and minimalist is:

* Server to verify a one time password (we already have one for yubikeys)
* CGI to take a username, password, and otp to verify in fas and the otp server
* pam module for sudo that verifies the user via the cgi
* database to store the secret keys for the otp generation and associate them with the fas username

We’re hoping to go a little beyond the minimal at the FAD:

* Have a web frontend to configure the secret keys that are stored for an account.
* Presently we’re thinking that this is a FAS frontend but we may end up re-evaluating this depending on what we decide to do for web apps and what to require for changing an auth source.
* Allow both yubikey and google-authenticator as otp

I’m also hoping that since we’ll have most of the sysadmin side of infrastructure present that we’ll get a chance to discuss and write down a few OTP policies for the future:

* Do we want to make two-factor optional for some people and required for others?
* How many auth sources do we require in order to change a separate auth source (email address, password, secret for otp generation, phone number, gpg key, etc)?

If we manage to get through all of that work, there’s a few other things we could work on as well:

* Design and implement OTP for our web apps

Porting Kitchen to Python3: Part 1 — Detecting string types

I’ve spent a good part of the last week working on the python3 port of kitchen. It’s now to the point where I’ve reviewed all of the code and got the unittests passing. I still need to add some deprecation warnings and a gettext object that mirrors the python3 API instead of the python2 API. Then it’ll be ready for an alpha release. Still a lot of work to do before a final release. Most of the documentation will need to be updated to change from unicode + str to str + bytes and the best practices sections will need a major overhaul since a lot of the problems with python2 and unicode have either been fixed, mitigated, or moved to a different level.

It was both an easy and hard undertaking. The easy part was that kitchen is largely a collection of dependent but unrelated functions. So it’s reasonably easy to pick a set of functions, figure out that they don’t depend on anything else in kitchen, and then port them one by one.

The hard part is that a lot of those functions deal with things that are explicitly unicode and things that are explicitly byte strings; an area that has both changed dramatically in python3 and that 2to3 doesn’t handle very well. Here’s a couple of things I ended up doing to help out:

Detecting String Types

Kitchen has several places that need to know whether an object it’s been given is a byte string, unicode string, or a generic string. The python2 idioms for this are:

if isinstance(obj, basestring):
    pass # object is any of the string types
    if isinstance(obj, str):
        pass # object is a byte string
    elif isinstance(obj, unicode):
        pass # object is a unicode string
    pass # object was not a string type

In python3, a couple things have changed.

  • There’s no longer a basestring type as byte strings and unicode strings are no longer meant to be related types.
  • Byte strings now have an immutable (bytes) and mutable (bytearray) type.

With these changes, the python3 idioms equivalent to the python2 ones look something like this:

if isinstance(obj, str) or isinstance(obj, bytes) or isinstance(obj, bytearray):
    pass # any string type
    if isinstance(obj, bytes) or isinstance(obj, bytearray):
        pass # byte string
    elif isinstance(obj, str):
        pass # unicode string

There’s two issues with these changes:

  • code that needs to do this needs to be manually ported when moving from python2 to python3. 2to3 can correctly change all occurrences of isinstance(obj, unicode) to isinstance(obj, str) but occurrences of isinstance(obj, basestring) and isinstance(obj, str) will also be rendered as isinstance(obj, str) in the 2to3 output. This is correct for a lot of normal python2 code that is trying to separate strings from ints, floats, or other types but it is incorrect for code that’s trying to explicitly separate bytes from unicode. So you’ll need to hand-audit and fix your code wherever these idioms are being used.
  • This is more prolix and tedious to write than the python2 version and if your code has to do this sort of differentiation in many places you’ll soon get bored of it.

For kitchen, I added a few helper functions into kitchen.text.misc that encapsulate the python2 and python3 idioms. For instance:

def isbasestring(obj):
    if isinstance(obj, str) or isinstance(obj, bytes) or isinstance(obj, bytearray):
        return True
    return False

and similar for isunicodestring() and isbytestring(). [In case you're curious, I broke with PEP8 style for these function names because of the long history of is* functions and methods in python and other programming languages.] By pushing these into functions, I can use if isbasetring(obj): on both python2 and python3. I only have to change the implementation of the is*string() functions in a single place when porting from python2 to python3.

Now let’s mention some of the caveats to using this:

  • In python, calling a function (isbasestring()) is somewhat expensive. So if you use this in any hot inner loops, you may want to benchmark with the function and with the expanded version to see whether you take a noticable loss of speed.
  • Not every piece of code is going to want to define “string” in the same way. For instance, bytearrays are mutable so maybe your code shouldn’t include those with the “normal” string types.
  • Maybe your code can be changed to only deal with unicode strings (str). In python3 byte strings are not as ubiquitous as they were in python2 so maybe your code can be changed to stop checking for the type of the object altogether or reduced to a single isinstance(obj, str). The language has evolved so when possible, evolve your code to adapt as well.

Next time: Literals

My first python3 script

I’ve been hacking on other people’s python3 code for a while doing porting and bugfixes but so far my own code has been tied to python2 because of dependencies. Yesterday I ported my first personal script from python2 to python3. This was just a simple, one file script that hacks together a way to track how long my kids are using the computer and log them off after they’ve hit a quota. The kind of thing that many a home sysadmin has probably hacked together to automate just a little bit of their routine. For that use, it seemed very straightforward to make the switch. There were only three changes in the language that I encountered when making the transition:

  • octal values. I use octal for setting file permissions. The syntax for octal values has changed from "0755" to "0o755"
  • exception catching. No longer can you do: except Exception, exc. The new syntax is: except Exception as exc.
  • print function. In python2, print is a keyword so you do this: print 'hello world'. In python3, it’s a function so you write it this way: print('hello world')
  • The strict separation of bytes and string types. Required me to specify that one subprocess function should return string instead of bytes to me

When I’ve worked on porting libraries that needed to maintain some form of compat between python2 (older versions… no nice shiny python-2.7 for you!) and python3 these concerns were harder to address as there needed to be two versions of the code (usually, maintained via automatic build-time invocation of 2to3). With this application/script, throwing out python2 compatibility was possible so switching over was just a matter of getting an error when the code executed and switching the syntax over.

This script also didn’t use any modules that had either not ported, been dropped, or been restructured in the switch from python2 to python3. Unlike my day job where urllib’s restructuring would affect many of the things that we’ve written and lack of ported third-party libraries would prevent even more things from being ported, this script (and many other of my simple-home-use scripts) didn’t require any changes due to library changes.

Verdict? Within these constraints, porting to python3 was as painless as porting between some python2.x releases has been. I don’t see any reason I won’t use python3 for new programming tasks like this. I’ll probably port other existing scripts as I need to enhance them.