select's Exceptional Conditions

I’m writing some software that interfaces with the GPIOs on a Raspberry Pi. I found myself wanting to test this software on my Mac while I was developing it, meaning I’d need to make “mock GPIO files” to test against. The trick was that I needed something I could send to select(2) and signal an “exceptional condition,” since that’s how Linux indicates an interrupt on a GPIO. This ended up being an education for me on both TCP out-of-band data and, surprisingly, pseudo terminals.

GPIO Interrupts

I’m taking advantage of Linux’s support for GPIO interrupts to detect when a switch opens or closes. This involves opening the GPIO’s sysfs file and using select on it with the GPIO file descriptor in the exceptfds list.¹ To clear the interrupt you just seek back to the beginning of the sysfs file and read from it. (I will be reading the GPIO value using the bcm2835 library, so I don’t care to read its value from its sysfs file.)

To make a mock GPIO file I’d either need to:

1. Make it possible to swap out select in my main loop with something I can control in testing. Then I could send in whatever object I want for the FD sets, and return whatever subset of them I want, whenever I want.

2. Find some other way to make a “waitable object”² that I can generate an “exceptional condition” on.

In retrospect, #1 might have been more sane, but I chose #2 instead. Perhaps I preferred to have to modify as little of my code as possible to accommodate testing. Or perhaps I was dissuaded from trying to make a “semi-mock” select that could faithfully operate both on whatever dummy objects I made up as well as real network sockets (also in use in my software).

Out-of-Band Data

Perhaps the best-known “exceptional condition” for select is TCP’s “out-of-band” or “urgent” data. “Best-known” here is probably not saying much.

It’s pretty easy to send out-of-band data, just set MSG_OOB on the flags for e.g. send(2). However, there were a couple interesting facts I didn’t know about urgent data, having never touched it before:

• You only get one byte of it. If you send more than one byte with MSG_OOB set, only the last will be received as urgent data. All preceding bytes before the last will be sent as regular data. If you receive more than one byte of OOB data without reading it, you will only be able to receive the last (most recent) OOB byte received.³

• You have to read both the OOB data followed by at least one byte of regular (not OOB) data to clear the “exceptional” condition from the socket.

Here’s a short Python program to demonstrate these conditions:

import select
import socket
import fcntl
import os
import errno
import pprint


server = socket.socket()
server.bind(("", 0))
server.listen(1)

client = socket.socket()
client.setblocking(0)
try:
    client.connect(server.getsockname())
except socket.error, ex:
    assert ex.errno == errno.EINPROGRESS

server_to_client = server.accept()[0]
server_to_client.sendall("123", socket.MSG_OOB)

print "First select:"
pprint.pprint(select.select([client], [], [client]))
print "Regular data:"
print repr(client.recv(10))
print "Second select:"
print select.select([client], [], [client])
print "OOB data:"
print repr(client.recv(10, socket.MSG_OOB))
print "Select after all data read:"
print select.select([client], [], [client])

Here’s the output from that program when run on my OS X machine under Python 2.7:

First select:
([<socket._socketobject object at 0x10666b280>],
 [],
 [<socket._socketobject object at 0x10666b280>])
Regular data:
'12'
Second select:
([], [], [<socket._socketobject object at 0x10666b280>])
OOB data:
'3'
Select after all data read:
([], [], [<socket._socketobject object at 0x10666b280>])

Notice:

• I sent three bytes, '123' with MSG_OOB set, but the regular read on the other end of the connection returns just '12'. Only the last byte, '3', was returned as OOB data. If I wrote some more OOB data before the client read '3', the '3' would be lost, overwritten by the new OOB byte.

• select keeps saying that the socket has an exceptional condition, even though I’ve read all the OOB data.

This behavior is weird, but I can work around it. The code under test that waits for interrupts (as well as incoming network traffic) will look something like:

def main_loop():
    rlist, wlist, xlist = select.select(rlist, wlist, xlist, timeout)
    for selectable in xlist:
        owner_of_selectable[selectable].handle_exceptional_cond(selectable)

class SwitchController(object):
    def handle_exceptional_cond(self, gpio_file):
        # Clear interrupt.
        gpio_file.seek(0)
        gpio_file.read()
        # Do something in response to the interrupt here.

I can wrap a socket in something that provides seek, read, and fileno methods, so that’s not a big deal. To fulfill the requirement to read some regular data in order to clear the exceptional condition on the socket I can put some normal data on the socket and add an extra recv into read. Alternatively, it seems that I can set SO_OOBINLINE, at which point reading the single OOB byte will clear both the “readable” and the “exceptional” conditions on the socket. (I’m throwing away the data being read, as you can see above, so sending any single OOB byte is sufficient. I just need to set the exceptional condition on the socket.)

Exceptional Conditions with Pseudo Terminals

Problem solved: I can use a TCP socket as a fake GPIO sysfs file, and make a fake interrupt by sending a single OOB byte into it.

Before I settled on using a TCP socket, though, I looked about for alternatives that I could send into select and trigger an exceptional condition. The rest of this post will discuss that search.

The only other (cross-platform) construct I found to fit the bill was the master end of a pseudo terminal:

import pty
import fcntl
import termios
import os
import select


# Note that we're never going use the slave, but it needs to stay
# open so that we can write to the master end.
master_fd, slave_fd = pty.openpty()
fcntl.ioctl(master_fd, termios.TIOCPKT, "\x01")
master_obj = os.fdopen(master_fd, "r+")
# We'll block on the read, below, if we don't set this non-blocking first.
flags = fcntl.fcntl(master_obj, fcntl.F_GETFL)
fcntl.fcntl(master_obj, fcntl.F_SETFL, flags | os.O_NONBLOCK)
# \x13 == ^S, the terminal's default STOP key.
master_obj.write("\x13")
master_obj.flush()
# Now send master_obj into our main loop as a fake GPIO sysfs file.
# The following code simulates that main loop's basic behavior.
print select.select([master_obj], [], [master_obj])
# That prints:
#     ([<open file ...>], [], [<open file ...>])
# Exceptional condition: achieved.
#
# Our interrupt handler now needs to clear the exceptional condition.
master_obj.seek(0)
master_obj.read()
print select.select([], [], [master_obj], 0.1)
# That prints:
#     ([], [], [])
# Exceptional condition clearing: achieved.

TIOCPKT is the trick here. It puts the pseudo terminal into “packet mode.”⁴ Packet mode makes it so that every read from the master end of the pseudo terminal will begin with either a zero byte, which is then followed by regular output from the terminal, or else a single non-zero byte to indicate some condition such as a flow control stop having been received. I trigger that stop by sending control-S. Then, when a control byte such as TIOCPKT_STOP is available to be read, select will signal an exceptional condition on the master end. The next read from the FD will clear the exceptional condition.

Other Exceptional Conditions

I got a little too interested in other possible ways to signal an exceptional condition in select and I tried to look for more in the XNU source code (OS X’s kernel). Unfortunately the sources were harder to understand than my limited time allowed, but it doesn’t seem easy to grep for users of the exceptional condition.

I think I found where the exceptional condition is implemented for the master side of a pseudo terminal in bsd/kern/tty_ptmx.c. Excerpted:

FREE_BSDSTATIC int
ptmx_select(dev_t dev, int rw, void *wql, proc_t p)
{
    /* ... */
    switch (rw) {
    case FREAD:
        /*
         * Need to block timeouts (ttrstart).
         */
        if ((tp->t_state&TS_ISOPEN) &&
             tp->t_outq.c_cc && (tp->t_state&TS_TTSTOP) == 0) {
            retval = tp->t_outq.c_cc;
            break;
        }
        /* FALLTHROUGH */

    case 0:                 /* exceptional */
        if ((tp->t_state&TS_ISOPEN) &&
            ((pti->pt_flags & PF_PKT && pti->pt_send) ||
             (pti->pt_flags & PF_UCNTL && pti->pt_ucntl))) {
            retval = 1;
            break;
        }
        selrecord(p, &pti->pt_selr, wql);
        break;

    case FWRITE:
        if (tp->t_state&TS_ISOPEN) {
            /* ... */
        }
        selrecord(p, &pti->pt_selw, wql);
        break;

    }
out:
    tty_unlock(tp);
    return (retval);
}

I’m guessing that this function will be called once for each select set (readable, writable, exceptional) that a master pseudo terminal FD is in, with rw being given as FREAD, FWRITE, and zero, respectively. You can see in the “exceptional” case where it checks if packet mode is enabled (PF_PKT) and if there is also a control byte to send (pti->pt_send). I’m guessing any non-zero return value from this function for a given rw value means that select should return this FD in the corresponding FD result set.

I did not try grepping for 0 in the kernel sources to find other users of select’s exceptional conditions, as I thought it might generate rather too many false positives.