Sunday, February 28, 2016

Way back in the early 90s, in the DOS days, we used to work primarily in text mode. There was an early version of Windows, but it was ugly and slow, and even when I made the switch to Linux I often worked at the text console because X11 was ugly and slow. I did miss (and still do) the speedy, plain text mode user interfaces of programs like Borland C compiler and Norton Utilities. These were the best you could get on a 80x25 characters screen, and you know what, there was nothing wrong with it. Fore one thing, I have yet to find a front-end to gdb or lldb that looks as good and works as nicely as Borlands Turbo Debugger once did.
Anyway, on Linux (and Mac) we can recreate these text interfaces using a library named curses. Back in August 2012 I wrote an introduction titled Crash course in ncurses. This time I want to show a couple of tricks that make working with curses a little easier.

Curses in practice: part 0. The curses API argument order

This is a no-brainer that every curses programmer loves to hate. To create a window, call newwin(h, w, y, x) which is totally backwards. What you want is: create_window(x, y, w, h). Some say you just have to stick with it, but it’s easy to create wrapper functions anyway.

Curses in practice: part 1. Keyboard input

Getting keystrokes from curses is done with getch(). That’s easy enough, once you’ve put the terminal into raw() mode. The getch() function will return an integer key code. This is nice when you are programming in C, and it doesn’t matter much if all you want to do is catch the cursor keys. But in Python, when I press ‘a’, I would really like to get 'a', which is a string. If we make our own getch() function that returns a string, then we need to do some more trickery to be able to return cursor key codes or keys combined with Ctrl.

def getch():
    '''Returns keystroke as string'''

    # this is a good moment to render any screen updates
    curses.doupdate()

    key = STDSCR.getch()

    # in debug mode, Ctrl-Q is hardwired to quit
    if DEBUG and key == 17:
        # terminate() resets curses and calls endwin()
        terminate()
        sys.exit(-1)

    if key == curses.KEY_RESIZE:
        resize_event()

    elif key >= ord(' ') and key <= ord('~'):
        # ASCII key
        return chr(key)

    elif key >= 1 and key <= 26:
        # Ctrl-A to Ctrl-Z
        return 'Ctrl-' + chr(ord('@') + key)

    # special key : use a lookup table
    skey = '0x%02x' % key
    if skey in KEY_TABLE:
        return KEY_TABLE[skey]

    # unknown key; just return as hex string
    return skey

As you can see, we return all keys as a string value. This is a lot easier to work with when interpreting keystrokes, because now the character ‘a’ will be a Python string 'a'.
Alternatively, you might do a hybrid function that returns strings for ASCII values, but integers for special or modified keys. In the above example I use a key table however. The key table looks like this:

KEY_TABLE = {'0x1b': 'ESC', '0x0a': 'RETURN',
             '0x09': 'TAB', '0x161': 'BTAB',
             '0x102': 'DOWN', '0x103': 'UP',
             '0x104': 'LEFT', '0x105': 'RIGHT',
             '0x152': 'PAGEDOWN', '0x153': 'PAGEUP',
             '0x106': 'HOME', '0x168': 'END',
             '0x14a': 'DEL', '0x107': 'BS', '0x7f': 'BS'}

A note about the Esc key: curses imposes a timeout on this key stroke, making the key slow to react. The reason for this timeout is that in the UNIX terminal special keys like the cursor keys, Function keys, etcetera are all translated to escape sequences. You can fiddle with this timeout at the risk of breaking the program’s sense of input. In practice it is safe however to set the timeout as low as 25 milliseconds. C programs may call set_escdelay(). In Python, we set an environment variable:

# set ESC delay before calling initscr()
os.environ['ESCDELAY'] = '25'
STDSCR = curses.initscr()

A note about Function keys: F1 .. F10 are absolutely evil on the Mac. They are supposed to work as-is when holding down the special Fn key, but they don’t. Sorry, but the Mac is just broken like that. This is incredibly sad. Therefore the F-keys are off limits; just stay away from them.

Curses in practice: part 2. Color management

curses has the weirdest color management the world has ever seen. You would expect to be able to set foreground and background color. Well, you can … but you can’t set them independently. In curses, you create a color_pair which is exactly that: a pair of colors that stick together. For a text color you might combine white on black, and for a window title you might want to choose yellow on blue, in bold. (The bold attribute is not part of the color pair).

Working with color pairs is annoying if you are not used to it. You can set up a bunch of color pairs at program initialization, and use those. Personally, I like having a more dynamic system. So I came up with having a color cache, which caches combinations of colors.

COLOR_CACHE = {}
COLOR_PAIR_IDX = 0

def get_color(fg, bg, bold=True):
    '''Returns the curses color attribute'''

    global COLOR_PAIR_IDX

    pair = '%x:%x' % (fg, bg)
    if pair not in COLOR_CACHE:
        assert COLOR_PAIR_IDX < curses.COLOR_PAIRS

        # make new color pair
        COLOR_PAIR_IDX += 1
        curses.init_pair(COLOR_PAIR_IDX, fg, bg)
        COLOR_CACHE[pair] = COLOR_PAIR_IDX

    color = curses.color_pair(COLOR_CACHE[pair])
    if bold:
        color |= curses.A_BOLD
    return color

def init():
    '''initialize'''

    # get any colors at all
    curses.start_colors()

You could also add support for reverse video with curses.A_REVERSE.

The reason for curses having color pairs is that the number of colors that a terminal can simultaneously display may be severely limited. In practice however, that holds true for very ancient displays only. To make it perfect, we should check if the terminal has_colors() too.

Curses in practice: part 3. Screen emulation

So, you have your curses windows up and running, and it looks really nice. But next you want to add a shadow to the window, and find out that it doesn’t work because curses won’t allow you to draw outside the window, which actually makes sense.
We can make the shadow by writing into the STDSCR. This is a gross hack, and won’t work correctly once you start stacking windows — either with or without curses.panels.
I tried making a shadow by having a larger curses window so that the shadow would fall inside, but there is no easy way to draw a border that is smaller than the curses window.
So curses is letting us down here. We can fix it, but we are going to have to take a leap and leave the standard way of doing things behind.

The IBM PC text mode VGA screen worked like this: there was a frame buffer at segment address B800 that held the screen. You could poke into that buffer and directly manipulate what was on the screen. One byte represented a character, and the next was its color: 3 bits of foreground, 1 bold/bright bit, 3 bits of background color, 1 bit for blinking. What I really want is something like this. It’s easier though to choose two planes: one character buffer, and one color buffer.

You can see where this is going; we won’t use any curses windows at all. There is only one STDSCR, and in the STDSCR we can draw whatever we want, where we want, and make windows with shadows. We can have the windows clip and not throw an exception (unlike curses). We can easily save the background and restore it when a window closes. The stretch of code that does all that is too large to post here, but I will show the basics:

class ScreenBuffer(object):
    '''represents a screen buffer'''

    def __init__(self, x, y, w, h):
        '''initialize'''

        # TODO assert x >= 0 etc.

        self.w = w
        self.h = h
        self.textbuf = bytearray(w * h)
        self.colorbuf = bytearray(w * h)
        self.current_color = 0

    def putch(self, x, y, ch, color=-1):
        '''put character at x, y'''

        # TODO do clipping for x,y

        if color == -1:
            color = self.current_color

        if isinstance(ch, str):
            ch = ord(ch)

        offset = self.w * y + x
        self.textbuf[offset] = ch & 0x7f
        if ch > 0x400000:
            self.textbuf[offset] |= 0x80
        self.colorbuf[offset] = color
        # update curses screen, too
        STDSCR.addch(y, x, ch, curses_color(color))

There are two things to note here. First, I used the VGA color coding scheme that works with bits and bit shifts. These colors need to be translated again to curses color pairs. It’s not shown here, but it’s best to keep a current color member for both the VGA and the curses color. Secondly, when curses draws lines for borders, it uses character codes that are large integers: above 0x400000. You can’t fit those into a single byte. What I do is take the 7-bit ASCII value, and set the 8th bit if it was a special curses character. Knowing that there was a special character there is important when saving and restoring window backgrounds.

This code looks horribly inefficient (and in a way, it is) but we can make optimized versions for drawing horizontal and vertical lines, rectangles, and outputting messages on the screen.
Because curses doesn’t do its screen update magic until doupdate() gets called by our getch() routine, this homemade VGA-like screen emulation thing works surprisingly well.

This cool trick changes to awesome when overloading the index operator for the ScreenBuffer class with __getitem__() and __setitem__(). This enables writing code like:

screen[x, y] = (ch, color)

curses just got a whole lot easier. It’s a large library with lots of functions, but ultimately I’m only interested in getting a colored character on-screen at an arbitrary position. That’s all I need.

It’s too bad, but there are some things that curses just can not do; Novell Netware’s menu’s had double borders; Borland’s buttons had half-height shadows; Norton Utilities redefined the PC font to have pretty radio buttons, checkboxes, and extra wide window borders. Emulating these may only be possible by installing a special font file, or going full blown graphics mode and not using curses at all.