When the most recent input is itself a token, passed up from some lower-level lexer, we know that the following characters aren't part of this token, so we can examine the current text in the same way as if it were at the end of the input.