.grsec patch size_overflow

You probably know the Linux kernel hardening patches from grsecurity. Even hardening patches can introduce bugs, and this one is a classic: a signed value quietly turning into an unsigned one.

The bug

I originally found this through a tweet about the size overflow in the grsec patch. That tweet has since been deleted.

The problem line is this one:

room = N_TTY_BUF_SIZE - (ldata->read_head - tail);

When you change a variable from int (signed) to size_t (unsigned), you have to be sure you only ever store non-negative values in it. So what happens if the right-hand side goes negative?

A tiny demo

Here’s the simplest possible example:

$ cat test.c

#include <stdio.h>

int main(void) {
        size_t a = -15;
        printf("%lu\n", a);
        return 0;
}

$ gcc -Wall test.c -o test
$ ./test
18446744073709551601

A small negative value like -15 becomes a gigantic unsigned one. The bits don’t change, only how they are read:

The fix

Check that the value isn’t negative before using it. A ternary operator does the job as a quick fix:

room = ((N_TTY_BUF_SIZE - (ldata->read_head - tail)) > 0) ? N_TTY_BUF_SIZE - (ldata->read_head - tail) : 0;

Lessons learned

• Changing int to size_t is not a cosmetic change. Every place that can produce a negative value becomes a potential overflow.
• size_t a = -15; compiles fine, even with -Wall. The compiler won’t save you here.
• Validate the sign before the assignment, not after it.