### Regular Expressions

TADS 3 provides several string functions which use "regular expressions." A regular expression is a similar to a "wildcard" search string, but regular expressions are much more powerful than simple wildcards. Regular expressions are used to search for strings within text.

If you're a seasoned veteran of regular expressions in other programming languages or applications, you'll probably want to skip ahead to the summary. You might also want to read about the RexPattern intrinsic class, which stores compiled patterns; and about the specific regular expression functions, rexMatch, rexSearch, rexReplace, and rexGroup.

A regular expression is often called a "pattern," because it doesn't usually specify a literal string of characters to find, but rather specifies several possible formats that the text of interest could take. What makes regular expressions powerful is that you can specify a whole range of possibilities without actually listing all of the possibilities individually. In fact, you can easily specify a pattern that matches an infinite number of possibilities (and do so quite compactly).

Regular expressions can look a little daunting at first, because the pattern language uses punctuation marks to carry much of the meaning. Don't worry about that; once you learn what a few symbols mean and how they combine, you'll find most expressions are not that difficult to understand.

The simplest kind of regular expression is simply a string of literal text. For example, this is a valid regular expression:

       abc

This simply matches the string "abc", because the pattern consists entirely of "ordinary characters," and each ordinary character of the regular expression is matched literally to a character of the string to be searched.

An "ordinary character" is any character that doesn't have some other meaning in the regular expression language. All of the alphabetic characters (including accented characters), all of the digits, and space characters of all kinds are ordinary characters. The following punctuation marks have special meanings:

       % < > + . * ? [ ^ $ | ( ) {

Everything else is an ordinary character.

You can use almost all of these special characters as though they were ordinary characters by putting a percent sign ("%") in front of them. So, to search for the letters "abc" enclosed in parentheses, we could write this:

       %(abc%)

By putting the percent signs in front of the parentheses, you remove the special meaning from the parentheses and turn them into ordinary characters that literally match the search text.

Note one pair of exceptions to the "%" rule: the sequences "%<" and "%>" have special meanings of their own, so you can't use "%<" to match a less-than sign, and you can't use "%>" to match a greater-than sign. To match these characters, you must use a range expression or a named character expression; we'll see these in more detail later, but for now just note that we can match angle brackets like this:

       <langle>abc<rangle>

The "<langle>" part matches a single left angle bracket ("<"), and the "<rangle>" part matches a single right angle bracket (">"). Thus, the pattern above matches "<abc>".

The meanings of all of the special characters, and how they combine, are explained in the sections that follow.

### Concatenation

Even the simple string above uses one of the construction principles that lets you build complex search patterns. The string above consists of three ordinary characters that are concatenated together to form a longer string. When you concatenate a regular expression element to a regular expression, you get a new regular expression that matches what the first one matches, plus what the new element matches. This is pretty obvious for simple cases like "abc", because if we add a new element – say the letter "d" – we get a new regular expression which matches a longer literal string:

       abcd

### Alternation

Another construction principle that lets you combine expressions is alternation. With alternation, you specify that the pattern matches one regular expression or another regular expression. You specify alternation with the character "|" (the vertical bar).

We know that the expression "abc" matches the literal string "abc", and the expression "def" matches the literal string "def". So, we could combine these with alternation to make a new regular expression that matches either "abc" or "def":

       abc|def

### Named Characters

Each of the special characters (except angle brackets) can be entered literally in an expression using a percent sign ("%") to quote the character, as shown above. In addition, these characters, plus a few others, can be entered by name, by enclosing the character's name in angle brackets ("< >"). To search for an asterisk, for example, you could write this expression:

       <star>

The named character notation is equivalent to using "%" to quote the special characters. The named notation is provided as an alternative because it often results in expressions that are easier to read than the "%" equivalents.

The special character names and the corresponding characters are:

         <lparen>    (

         <rparen>    )

         <lsquare>   [

         <rsquare>   ]

         <lbrace>    {

         <rbrace>    }

         <langle>    <

         <rangle>    >

         <vbar>      |

         <caret>     ^

         <period>    .

         <dot>       .

         <squote>    '

         <dquote>    "

         <star>      *

         <plus>      +

         <percent>   %

         <question>  ?

         <dollar>    $

         <backslash> \

         <return>    carriage return (character code 0x000D)

         <linefeed>  line feed (character code 0x000A)

         <tab>       tab (character code 0x0009)

         <nul>       null character (character code 0x0000)

         <null>      null character (character code 0x0000)

These names are not sensitive to case, so <LANGLE> and <LAngle> are equivalent to <langle>.

Note that the named characters are simply "character classes," which we'll get to in a moment. This means that you can combine named characters with the "|" symbol, and reverse the sense of the match with the "^" symbol, just as with character classes. For example, the following matches a period, plus sign, or asterisk:

       <period|plus|star>

and the following matches anything except a question mark:

       <^question>

### Wildcards, Ranges, and Classes

If you've ever used an operating system like DOS or Unix, you're probably familiar with "wildcard" characters for file directory listings. A wildcard is a character that matches any other character.

Regular expressions have a wildcard character, too, but it's not what you might expect if you're thinking about filename wildcards from DOS or Unix. The regular expression wildcard character is the period ("."). This simply matches any single character. So, if we wanted to match the word "the" followed by a space followed by any three characters, we'd write this:

       the ...

Regular expressions don't stop at simple wildcards, though: they let you get much more specific. First, you can use "ranges," which let you match one of a selected group of specific characters. For example, if you want to match any single character that is a vowel, you could write a range like this:

       [aeiouAEIOU]

Note that, by default, regular expressions are case-sensitive, which is why we wrote the vowels in both upper- and lower-case. You can, however, control the case-sensitivity of a search, so you don't always have to write your expressions this way.

You can use a range expression in an expression wherever an ordinary character can go. So, to write a pattern that matches "button", followed by a space, followed by a digit from 0 to 9, you could write this:

       button [0123456789]

Ranges can also specify that you want to exclude characters. An "exclusive" range works just the opposite of a regular range: it matches anything that's not listed in the range. You specify an exclusive range by putting a caret ("^") as the first character inside the brackets of the range. So, to match any single character that isn't a vowel, you'd write this:

       [^aeiouAEIOU]

Note that exclusive ranges match anything that's not in the range, so the range above will match anything that isn't a vowel, including digits, spaces, and punctuation characters.

You can also use a range to specify contiguous portions of the Unicode character set simply by giving the endpoints of the portion. Do this by listing the ends of the range, separated by a hyphen ("-"). For example, to match any letter in the Roman alphabet, not including any accented characters, you'd write this:

       [a-zA-Z]

This matches any character whose Unicode character code value is between "a" and "z" inclusive, or between "A" and "Z" inclusive. (The Unicode character set includes the ASCII character set as a subset, assigning the same character code values as ASCII does to the ASCII characters, so if you're familiar with Unix-style regular expression ranges, you will find Unicode ranges end up working exactly the same way.)

You can use exclusion with subset ranges as well:

       [^a-zA-Z]

This matches any single character that is not in the Roman alphabet.

If you want to include the character "^" in a range expression, you can do so, as long as it's not the first character – if the "^" appears as the first character, it's taken to indicate an exclusive range. So, to specify a match for either an ampersand or a caret, you'd have to write the range expression like this:

       [&^]

Similarly, note that, if you want to include a hyphen character in a range expression, it must be the first character in the range list. If a hyphen appears anywhere else, it's taken as a subset specifier. So, to write a range that matches a pound sign or a hyphen, you'd have to write this:

       [-#]

In addition, if you want to include a right square bracket in a search string, it must be the first character in an inclusive range, or the first character after "^" in an exclusive range.

Combining all of the rules above, if we wanted to write an inclusive search for all of the special range characters – hyphen, caret, and right square bracket – we'd have to write this:

       []-^]

And to write a search that excludes all of these characters:

       [^]-^]

The two examples above are the exact orders needed for these special situations. If you want to write these ranges and include additional characters, insert them at the end of the range, just before the closing square bracket. If you don't want to include all of the special characters, take out the ones you don't want from the example above, leaving the remaining ones in the same order.

Note that, other than the three special range characters ("^", "-", and "]"), all of the characters that are special elsewhere in a pattern lose their special meaning within a range. So, the following range expression matches a period, a star, or a percent sign:

       [.*%]

Ranges are useful for matching a specific group of characters, but it's harder to write a good range expression for more complex character sets, such as any alphabetic character or any digit. Unicode has so many different groups of alphabetic characters, since it includes support for so many different languages, that it would take a lot of work to list all of the different alphabetic ranges. Fortunately, TADS regular expressions provide a shorthand notation for certain important character sets, called "character classes."

Each character class is written as a name enclosed in angle brackets ("<" and ">"). Each class matches a single character. The classes are:

- <Alpha> - matches any single alphabetic character. This class matches anything that the Unicode character database classifies as alphabetic, so this includes accented characters and characters from non-Roman alphabets.
- <Upper> - matches any single upper-case alphabetic character. Like <Alpha>, this class uses Unicode classifications, so it matches accented upper-case characters.
- <Lower> - matches any single lower-case alphabetic character.
- <Digit> - matches any single digit character.
- <AlphaNum> - matches any single alphanumeric character.
- <Space> - matches any whitespace character.
- <Punct> - matches any punctuation mark character.
- <Newline> - matches any single newline character (carriage return [0x000D], line feed [0x000A], or Unicode line separator [0x2028]).

Note that the class names are not case-sensitive (regardless of whether or not the search itself is), so <Alpha>, <alpha>, and <ALPHA> are equivalent.

You can use a character class in place of an ordinary character. So, to search for a five-letter word starting with an upper-case letter followed by four lower-case letters, we could write this:

       <Upper><lower><lower><lower><lower>

Character classes can be combined using "|" to separate class names. For example, if you want to write an expression that matches any upper-case letter or any digit, you could write this:

       <Upper|Digit>

This is equivalent to writing "(<upper>|<digit>)", but is a little more concise.

Character classes can be complemented using "^" as the first character inside the angle brackets. For example, to match any character other than an upper-case letter, you could write this:

       <^Upper>

You can complement combined character classes as well. This matches any character except a space or punctuation mark:

       <^Space|Punct>

Note that the "^" applies to the entire class expression, not just to the first element, so the example above does not match punctuation marks.

You can combine character classes and literal characters in a single angle-bracket expression. For example, suppose you want to match the characters of a C++ identifier. The first character of an identifier in C++ must be an alphabetic character or an underscore ("_"), and subsequent characters can be letters, numbers, or underscores. You could use this expression:

       <Alpha|_><Alpha|Digit|_>*

You can also use literal ranges, just like in square bracket expressions, and combine these with individual literals or classes. For example, to match any upper-case letter, but only lower-case "a" through "m", you could write this:

       <Upper|a-m>

You might have noticed by now that angle bracket expressions using literals are very similar to square bracket expressions. However, there's one crucial difference in the syntax: inside angle brackets, each individual literal character or literal range must be separated from others by bars ("|"). For example, consider this square-bracket range:

       [13579a-f]

To write the same expression with angle brackets, you must separate each digit from the next by a bar:

       <1|3|5|7|9|a-f>

The bars are required because the regular expression compiler would otherwise not be able to tell for sure what something like this means:

       <Alpha>

Were it not for the rule about separating literals with vertical bars, this could either mean any alphabetic character or any of the characters "A", "l", "p", "h", or "a". Thanks to the rule that literals must be separated with bars, there is no ambiguity: <Alpha> can only mean any alphabetic character. You might think that you should be allowed to get away without the bars when the literals don't spell a character class name or the name of a special character. However, if this were allowed, it could create problems in the future: what you think is a meaningless string of literals now could take on a new meaning in a future version. So, to avoid any confusion or future compatibility problems, the bars are required for all literals.

All of the named characters (<lparan>, <period>, and so on) are essentially just very narrow character classes, so you can freely mix them with literals and character classes in angle-bracketed patterns. For example, to match any upper-case letter or an open or close parenthesis, you could use this expression:

       <Upper|lparen|rparen>

### Closures and Optionality

If you've used filename patterns on DOS or Unix, you're probably wondering by now how you match a variable-length string, the way the "*" character does for filename matches on these systems. Regular expressions let you do this, but in a different and more powerful way than filename patterns do.

There are three ways of specifying variable-length regular expression matches. The first is the "optionality" operator, which specifies that the immediately preceding expression character is optional – specifically, that the preceding character can be present zero or one times in the match string. The optionality operator is the question mark, "?", and immediately follows the character to be made optional. So, to search for either "you" or "your", we could write this:

       your?

The second variable-length operator is the one-or-more "closure." This operator is the plus sign, "+", and specifies that the immediately preceding character is to be repeated once or more – any number of times, as long as it appears at least once. So, to match a string of any number of copies of the letter "A", we'd write this:

       A+

This matches "A", "AA", "AAA", and so on without limit.

The third variable-length operator is almost the same: it's the zero-or-more closure. This operator is the asterisk, "*". This specifies that the preceding character is to match any number of times, and furthermore that it need not be present at all.

       abcd*

This matches "abc", or "abcd", or "abcdd", or "abcddd", and so on.

You can apply the closure operators to more complex expressions than a single ordinary character. For example, to search for one or more digits, you could write this:

       <digit>+

To search for any word of any length written with an upper-case initial letter and lower-case letters following, you'd write this:

       <upper><lower>*

To search for any number of repetitions of an arithmetic operator character, we could write this amusing sequence of punctuation marks:

       [-+*/]*

Closures normally match as much text as they possibly can, but you can change this using the shortest-match modifier, as we'll see a little later.

(In case you're wondering why "*" and "+" are called "closures": this term comes from the set mathematics from which the concept of regular expressions descends. A set is said to be "closed" under an operator if, for every element of the set, applying the operator to the element yields another element of the set. The operator of interest in this case is concatenation. Suppose we wished to form a set of strings closed under the concatenation operator: we could start with a string X, but then we'd have to include XX (X concatenated with itself) in the set, plus XXX (X concatenated with this new member XX), XXXX, and so on, forever forming longer strings. The set wouldn't be closed under concatenation until we've added a string of X's of every possible length. This set would obviously be inconvenient to write out in research papers without some kind of trick, which is where "*" and "+" come in: these symbols give us a way to express a set that's closed under concatenation – hence infinite if non-trivial – with a finite notation, and thereby provide closure to the notation. Glad you asked? Happily, this level of formalism is not needed in the course of using regular expressions.)

### Intervals (Counted Repetitions)

It is often useful to match a certain number of repetitions of a given character. The obvious way to express a repetition is with ordinary concatenation; for example, if we wanted to find a string of five A's, we could write this:

       AAAAA

This type of thing is less convenient when we want to find something that takes up more space than a single letter, though; for example, a pattern matching any five lower-case letters is cumbersome when written with concatenation:

       <lower><lower><lower><lower><lower>

The regular expression language has a more convenient way: the interval operator. This operator immediately follows an expression character, and specifies that the preceding character is to be repeated a given number of times. The interval operator is written by placing a number if curly braces ("{ }"). We can use this operator to re-write our example more concisely:

       <lower>{5}

The interval operator can also be used to specify a range of repetitions. You can put two numbers between the braces, separating the two by a comma. The first number is the minimum repeat count, and the second is the maximum count. For example, to match a string of three, four, five, six, or seven digits, we could write this:

       <digit>{3,7}

Finally, the interval operator can specify an unlimited maximum count. To do this, include the comma, but omit the upper bound. For example, to match at least five digits in a row, we would write:

       <digit>{5,}

### Grouping

Each construction rule has a default grouping. For example, the alternation operator ("|") considers everything to the left of the "|" to be one complete regular expression, and everything to the right to be another complete expression: the pattern "abc|def" thus matches "abc" or "def". Sometimes, however, you will want to change the default grouping, to extend or limit the extent to which an operator applies. You can do this by putting a portion of the expression in parentheses ("(" and ")").

For example, suppose we wanted to construct an expression that matches either "the red ball" or "the blue ball". We might first attempt something like this:

       the red|blue ball

However, this wouldn't work the way we want: the "|" operator applies to everything to its left and right, so what this expression actually matches is "the red" or "blue ball". This is where parentheses come in handy: we can enclose in parentheses the part of the expression to which we want to apply the "|" operator:

       the (red|blue) ball

You can also use parentheses to achieve the opposite effect with the closure operators. Using parentheses, you can make the closure operators apply to more than just the single character preceding the closure. For example, to match any number of repetitions of the word "the" followed by a space, you could write this:

       (the )+

Similarly, you can use parentheses to control the reach of the interval operator:

       (the ){2,3}

You can use parentheses within parentheses for more complex grouping. For example, to search for the word "the" followed by any number of repetitions of "ball", and then repeating the whole thing any number of times, we'd write this:

       (the (ball )+)+

### Group Matches

Parenthesized groups have another use besides controlling operator grouping. Each time you use parentheses, the regular expression matcher automatically assigns a "group number" to the expression contained within the parentheses. The group numbers start at 1, and increase each time the parser encounters an open parenthesis. (Nesting doesn't matter for numbering – the order of appearance of the open parentheses establishes the group numbering.)

The regular expression functions let you look at the exact text that matched a particular group after a search. For example, suppose you defined a search like this:

       say "(.*)" to (<alphanum>*)

This expression has two groups. Group number 1 is the part within the quote marks. Group number two is the part after "to". Now, suppose we match this string:

       say "hello there" to Mark

If we ask the regular expression matcher for group number 1, it will give us the string "hello there" (no quotes – the group is inside the quotes, so the quotes won't be part of the group string). Similarly, group number 2 is the string "Mark".

Groups can also be used within an expression. If you write the sequence "%1" in an expression, it specifies a match to the same thing that group number 1 already matched in the same string. Similarly, "%2" matches the same text as group number 2, and so on, up to "%9" for group 9. This allows you to look for repeated sequences that are separated from one another. For example:

       (<alphanum>*) is %1

This will match any string of the form "word is word", where the two words are the same. So, it will match "red is red" and "blue is blue", but it won't match "blue is red".

### Non-Capturing Groups

As we've seen above, the grouping syntax (putting a portion of the regular expression in parentheses) has two uses: first, to control the reach of an operator such as "*" or "|"; second, to capture part of the matching text, for use in a group match (such as "%1") or for replacement or extraction.

When you're only grouping part of the expression to control the reach of an operator, the text-capture feature is sometimes undesirable. In particular, because groups are numbered by position, adding a new group into an existing expression requires that you adjust the numbering for any other groups. For example, suppose we defined an expression like this:

       to (.*) %1

Now, suppose we wanted to change this slightly by making it recognize "of" as well as "to". We'd change it to something like this:

       (to|of) (.*) %1

Unfortunately, we've forgotten something: the group that was originally number 1 is now number 2, because we've added another group before it.
This example is so simple that renumbering its groups wouldn't pose much of a challenge, but we might simply forget; and for a complicated expression, this could become a real maintenance problem.

Fortunately, the regular expression language has a feature that lets you mark a group as non-capturing. A non-capturing group still has the same operator grouping effects as a normal group, but it doesn't capture its matching text, and it doesn't affect the numbering for any other groups.

To make a non-capturing group, add the sequence "?:" immediately after the group's opening parenthesis. (This might seem like a strange bit of syntax, because the question mark character is normally used as a closure operator. However, the "?" closure operator is always a postfix operator – it has to follow the sub-expression that it modifies. When a "?" appears immediately following an open parenthesis, it's simply not meaningful as a closure operator, because there's no sub-expression that it can modify in this position. The regular expression language thus can assign this separate meaning to a question mark that immediately follows an open parenthesis.)

So, let's re-write our example with a non-capturing group:

       (?:to|of) (.*) %1

### Special Matches

The regular expression matcher provides a number of special match types.

The "^" character specifies a match to the very beginning of the search string. If specified, this has to be the first character in the pattern (or the first character within a parenthesized group at a top-level alternation). The "^" character doesn't match any characters – it simply matches if the search position is the very start of the string.

The "$" character specifies a match to the very end of the string. This must be the last character in the pattern or within a parenthesized group at a top-level alternation.

The sequence "%<" matches the start of a word, which is defined as a position where the preceding character is not a word character, and the following character is. A word character is any alphanumeric character. "%<" doesn't actually match any characters – it just requires that the current position is the start of a word.

The sequence "%>" matches the end of a word.

The sequence "%w" matches any word character, which is defined as an alphanumeric character. This is equivalent to "<AlphaNum>", but is shorter to type in.

The sequence "%W" matches any non-word character.

The sequence "%b" matches any word boundary, which is either the beginning or ending of a word.

The sequence "%B" matches anywhere that is not a word boundary.

### Case Sensitivity

By default, searches are sensitive to case, which means that an upper-case letter in the search pattern will match only the identical upper-case letter in the string being searched. You can, however, make a search insensitive to case. To do this, add the <NoCase> flag to the search pattern. There's also a <Case> flag to make the case sensitivity explicit, but this is the default, so you won't usually need to specify it.

The <Case> and <NoCase> flags don't match anything themselves; they're just flag sequences that control the overall search mode. You can put these anywhere in the search, but normally you'd just want to put them at the start of the search string to avoid confusion. Note that these flags are global, which means that the entire search is case-sensitive or case-insensitive; you can't make part of your search string sensitive to case and another part insensitive. If the flags appear more than once, only the last one that appears is obeyed.

For example, to search for a match to "abc", ignoring case, we'd write this:

       <NoCase>abc

When you use <NoCase>, the case of the letters in your pattern is mostly irrelevant, since the pattern matcher will match "A" or "a" to a pattern character "a", and will likewise match "A" or "a" to a pattern character "A". However, there are some cases involving non-English languages where the case of the pattern characters might be significant. In particular, when the matcher encounters an alphabetic character in case-insensitive mode, it will first convert the string character to the case of the corresponding pattern character, and it will then perform the comparison. In some languages, a few characters have ambiguous translations from upper-case to lower-case or vice versa. In these languages, you can control how the matcher performs the translation by using the correct unambiguous case in the pattern string. For most languages whose writing systems are based on the Roman alphabet, there is no ambiguity, so you won't have to worry about this.

### Shortest-Match Closures

Normally, closures will attempt to match as much text as possible. Sometimes, however, this is undesirable. Consider this example:

    tell%>(.*)%<to%>(.*)

This works fine for a string like this:

    tell him to call me

The first group matches " him ", and the second group matches " call me" (note that we could eliminate the spaces in the match groups by adding <space>* patterns outside the groups, but we left them out in this example to keep it simple). However, the pattern doesn't work so well if we type something like this:

    tell him to go to the store

The match here gives us " him to go " for the first group, and " the store" for the second group. This probably isn't what we wanted.

The problem with this example is that the first group (the ".*" closure) tries to match as much text as it can. So, instead of stopping at the first "to" in the string, the regular expression matcher notices that there's another "to" later in the string, and keeps going. The first "to" simply becomes part of the match for the first group.

Fortunately, there's a solution. We can modify the first closure so that, instead of finding the longest match it possibly can, it instead finds the shortest match. To modify the closure, add a question mark after the asterisk:

    tell%>(.*?)%<to%>(.*)

If we try the string ("tell him to go to the store") again with this new expression, we match " him " for the first group and " to to the store" for the second group.

You can use the shortest-match modifier with all of the closures ("*", "?", and "+"), as well as with the interval operator ("{ }").

### Look-Ahead Assertions

Sometimes, it's desirable to look for a sub-pattern without including it in the full result match. The regular expression language lets you do this using a positive look-ahead assertion. An "assertion" is a sub-pattern that requires that some condition be met, but doesn't actually match anything in the search text; we've already seen a couple of these before, namely the "%<" and "%>" special match sequences. The important thing about an assertion is that it doesn't match any text – it simply imposes a requirement on the text at a certain point.

The syntax for a positive look-ahead assertion is a special variation on the group syntax. Put the sub-expression within parentheses, but insert the sequence "?=" immediately after the open parenthesis.

A positive look-ahead assertion requires that the search text match a sub-expression; but since this is an assertion, the text matching the sub-expression doesn't actually become part of the result. For example, suppose we had some strings giving people's names, and we want to find the first name of each person whose last name is Smith – but we don't care about anyone with a different surname. We could do this using a positive look-ahead assertion:

       ^<alpha>+(?=<space>+Smith$)

This matches a sequence of alphabetic characters from the beginning of the string, but only matches when the sequence is immediately followed by one or more spaces, the word "Smith", and the end of the string. So, this will match "John" in "John Smith", but won't match anything in "John Jones". Note that the length of the match for "John Smith" is only four characters – the part that matches the assertion doesn't count in the match.

The regular expression language also provides negative look-ahead assertions. These have the opposite effect of positive assertions: a negative assertion requires that the search text at a given point does not match the sub-pattern.

The negative assertion syntax is similar to the positive assertion syntax, but uses "?!" as the special sequence after the open parenthesis.

For example, suppose we wanted to change the example above so that we find first names for anyone whose last name isn't Smith. We could do this by changing our example to use a negative assertion for the last name:

       ^<alpha>+(?!.*<space>+Smith$)

Note that we had to insert an extra ".*" at the beginning of the negative assertion. If we hadn't done this, the assertion would have given us a match for "Joh" in "John Smith", because the assertion would have been satisfied by the substring "n Smith" (it doesn't start with a space, after all). The extra ".*" ensures that the assertion will fail for any string that ends with one or more spaces followed by "Smith".

Negative assertions are sometimes useful within larger expressions as well. For example, suppose we wanted to create a group that matched one or more words, but stopped as soon as it got to the words "to" or "of". We could do this like so:

       ((?!%<(to|of)%>).)+

This is a little complicated, so let's consider it piece by piece, starting at the outside and working in. At the outermost level, we have a parenthesized group that's repeated one or more times. Inside this, we have a negative assertion, followed by a "." wildcard character. So, the overall expression is simply the wildcard character, repeated one or more times – with the additional requirement that the wildcard character obey the negative assertion each time it's repeated. Inside the assertion, we have this:

       %<(to|of)%>

This matches "to" or "of" as stand-alone words – they must not be immediately preceded or followed by word characters.

So, our entire expression means: repeat a wildcard one or more times, but each wildcard must obey the requirement that it is not at the start of "to" or "of" as a separate word. Consider this search text:

      he will go to the store

This will match the substring "he will go ": everything up to the word "to" will match, but the word "to" ends the match because of the negative assertion.

Note that, while look-ahead assertions look like groups (because they're in parentheses), they don't have group numbers – you can't refer back to an assertion with a group-match sequence (such as "%1"). Assertions are effectively always non-capturing groups.

# Warning: Assertions and Closures Don't Mix

Note that you must never apply an unbounded closure (*, +, or {n} without an upper bound) to a sub-expression that contains only assertions. It's okay to apply a closure to a mixture of assertions and ordinary pattern characters, but you have to be sure that you don't try to apply a closure to an assertion alone or a group that contains only assertions.

The reason is a bit subtle. Since an assertion doesn't consume any input text, matching it once is no different than matching it twice, or indeed any number of times. A closure forces the matcher to ask: how many times can I match this successfully? The answer, in the case of a successful assertion, is always: an infinite number of times. Thus, if you apply a closure to an assertion, and the assertion is successful, the matcher will get into an infinite loop, successfully matching the assertion over and over and over until you intervene by killing the process.

The regular expression parser is smart enough to detect and ignore obvious cases. In particular, if you apply any closure directly to an assertion, the parser will simply ignore the closure. However, the parser isn't smart enough to catch more subtle cases; you can fool it by enclosing one or more assertions in a parenthesized group that contains nothing else, and then applying a closure to the group. It's highly unlikely that you'll create such an expression accidentally, but it's something to be aware of. If you're finding that one of your patterns seems to get stuck in an infinite loop when being matched, this is a possible cause to consider.

### First Beginning/Ending, Longest/Shortest Match

As we saw above in the examples for the shortest-match closure modifier, there are times when a particular expression can match a string in several different ways. For example, consider this pattern:

       say (.*) to (.*)

For many strings, there will be only one way to match this. In some cases, though, we could type a string that could be interpreted different ways. For example:

       say time to go to Bob

This could match in several different ways. We could end up with group 1 as "time to go" and group 2 as "Bob". We could also have group 1 as "time" and group 2 as "go to Bob". We could also have group 1 as "time" and group 2 as "go", or even an empty group 2 – ".*" can match zero characters, after all.

Normally, the matcher will give us the longest match that begins earliest in the search string. The matcher will furthermore give the earliest groups in the string the longest matches. So, of all of the choices above, the matcher will normally pick the one where group 1 is longest and group 2 is longest given that group 1 is already longest – thus, group 1 is "time to go" and group 2 is "Bob".

You can, however, control this behavior.

Two flags control whether the matcher picks the longest or shortest match for a string. If you put the <Max> flag somewhere in your expression (it's a global flag, so it doesn't matter where it goes), the parser will always choose the longest string it can for each subexpression, giving precedence to the earliest expression. This is the default behavior. If you use the <Min> flag, though, the matcher will use the shortest match that it possibly can for the overall match. Thus, consider this new expression:

       <Min>say (.*) to (.*)

Now if match this to "say time to go to Bob", we'll get "time" for group 1, and an empty group 2.

Note that the matcher still always tries to give the earliest groups the longest matches, but this is only after figuring out which is the shortest overall match. Consider this example

       tell (.*) to (.*)

If we type in something like "tell Bob to eat my shorts", there's no ambiguity. But if we try a string like "tell Bob to go to the store", the parser matches group 1 as "Bob to go" and group 2 as "the store", which isn't what we want. How do we solve this?

Unfortunately, <Min> doesn't help us much with a situation like this, because the second group is free to match nothing at all. So, if we try this:

       <Min>tell (.*) to (.*)

and we try "tell Bob to go to the store", we'll have "Bob" for group 1, as we want, but now we'll have an empty group 2 – the shortest match to the string is simply "tell bob to ", since the second group can match nothing. We could change the expression like so:

       <Min>tell (.*) to (.*)$

This forces the expression to match to the end of the string. But this still doesn't do what we want, because now the first group will be "Bob to go" and the second will be "the store" – so we're back where we started. The reason that <Min> doesn't help us here is that <Min> affects only the length of the complete match, and doesn't affect the matcher's preference for putting the longer string in the earlier group in case of ambiguity.

You can solve this kind of problem in many cases using the shortest-match modifier. In some cases, though, you might want even more control. A good approach in these cases is to use two separate regular expressions applied in sequence. For the first, we eliminate the second anything-goes wildcard sequence, and end the expression at the "to":

       tell (.*) to<space>

Now, this reduces the ambiguity of the expression, but it still doesn't do what we want – when we match "tell Bob to go to the store", we again find that group 1 is "Bob to go", since the parser by default matches the longest sequence it can. However, we finally have a situation where the <Min> flag solves our problem:

       <Min>tell (.*) to<space>

This gives us what we want – group 1 is simply "Bob", since the shortest possible string that matches the complete pattern is now "tell Bob to ". We can finish by using the match length for the overall expression to learn what's left in the rest of the string, which gives us what we formerly tried to get from the second group.

You can also specify whether the matcher finds the matching string that begins first or ends first. By default, the matcher finds a string that begins earliest in the search string. However, there are times when you might want to find the string that ends earliest. To do this, use the <FirstEnd> flag, which you can also write as simply <FE>. The default flag, <FirstBegin> or <FB>, finds the string that begins earliest.

### Summary of Special Characters and Sequences

| | Alternation operator
( ) | Grouping operator
+ | Repeat preceding expression one or more times
+? | Repeat expression one or more times, taking shortest match when ambiguous
* | Repeat preceding expression zero or more times
*? | Repeat zero or more times, taking shortest match when ambiguous
? | Repeat preceding expression zero or one times
?? | Repeat zero or one times, taking shorter match when ambiguous
{n} | Match exactly n repetitions of the preceding expression
{n,m} | Match at least n and at most m repetitions of the preceding expression
{n,} | Match at least n repetitions of the preceding expression
{ }? | Match repetitions, taking shortest match when ambiguous
. | Match any single character
^ | Match only at beginning of string
$ | Match only at end of string
[ ] | Character range
[^ ] | Exclusive character range
<Alpha> | Any single alphabetic character
<Upper> | Any single upper-case alphabetic character
<Lower> | Any single lower-case alphabetic character
<Digit> | Any single digit character
<AlphaNum> | Any single alphabetic or digit character
<Space> | Any single space character
<Punct> | Any single punctuation mark character
<Newline> | Any single newline character: carriage return (0x000D), line feed (0x000A), or Unicode line separator (0x2028)
<x> | The literal character "x"
<a-z> | Any character in the range a-z (equivalent to [a-z])
<x|y|z> | "x", "y", or "z" (equivalent to [xyz])
<a-f|w-z> | Any character in "a" through "f" or "w" through "z" (equivalent to [a-fw-z]
    <Digit|Upper>
| Any character matching <Digit> or <Upper>
    <Digit|x|y>
| Any digit character, or either "x" or "y"
<^Digit> | Any character except a digit
%1 | Match the same text that the first parenthesized group matched
%2 | Match the same text as the second parenthesized group
%9 | Match the same text as the ninth parenthesized group
%< | Match only at the beginning of a word
%> | Match only at the end of a word
%w | Match any single word character
%W | Match any single non-word character
%b | Match at any word boundary
%B | Match only at a non-word boundary
<Case> | Make the match case-sensitive (default)
<NoCase> | Make the match insensitive to case
<FirstBegin> | Find the match that begins earliest in the search text (default)
<FB> | Same as <FirstBegin>
<FirstEnd> | Find the match that ends earliest in the search text
<FE> | Same as <FirstEnd>
<Max> | Find the longest match (default)
<Min> | Find the shortest match
% | Quote the following special character (except "<" and ">")
(?: ) | Non-capturing group
(?= ) | Positive look-ahead assertion
(?! ) | Negative look-ahead assertion
<langle> | The character "<"
<rangle> | The character ">"
<lsquare> | The character "["
<rsquare> | The character "]"
<lparen> | The character "("
<rparen> | The character ")"
<lbrace> | The character "{"
<rbrace> | The character "}"
<vbar> | The character "|"
<caret> | The character "^"
<period> | The character "."
<dot> | The character "."
<squote> | The character "'" (a single quote)
<dquote> | The character '"' (a double quote)
<star> | The character "*"
<plus> | The character "+"
<percent> | The character "%"
<dollar> | The character "$"
<backslash> | The character "\"
<return> | The carriage return (CR) character (code 0x000D)
<linefeed> | The line feed (LF) character (code 0x000A)
<tab> | The tab character (code 0x0009)
<nul> | The null character (code 0x0000)
<null> | The null character (code 0x0000)

### Examples

If we were writing a C++ compiler, we'd want to write regular expressions for the lexical tokens that make up the language. For example, a symbol is any string of characters starting with a letter or an underscore, followed by any number of letters, digits, or underscores; C++ symbols are limited to the ASCII character set, so we can use range expressions rather than the <alpha> class and the like:

       <nocase>[a-z_][a-z_0-9]*

A C-style comment (/* … */) is a little tricky. At first glance, we might try something simple like this:

       /%*.*%*/

This won't do quite what we want, though: suppose we tried matching something like this:

       a /* destination */ = 1 /* value */;

Our expression match would give us everything from the first slash to the final slash – the one before the semicolon. This is incorrect, because we've mistaken the part between the two comments as part of one big comment. To rectify this, we can use the shortest-match modifier with the ".*" part of the expression:

       /%*.*?%*/

Character strings are even trickier. We can start with a similar sort of expression:

       ".*?"

This almost does the trick, but misses one important case: in C++, a string can contain a quotation mark, if it's preceded by a backslash character. To handle this case, we can handle backslashes specially: we'll treat a backslash and the immediately following character as a group, and then alternatively handle anything that isn't a backslash:

       "([^\]|\.)*?"

(Note that, if you were coding this expression within a string in your source code, you'd have to double the backslashes, because the TADS compiler considers them significant in the same way a C++ compiler would.)

Moving on to other types of patterns, here's an expression that matches a North American telephone number, with optional area code:

       (%([0-9][0-9][0-9]%))?<space>*[0-9][0-9][0-9]-[0-9][0-9][0-9][0-9]

Or, more compactly using intervals:

       (%([0-9]{3}%))?<space>*[0-9]{3}-[0-9]{4}

The next expression matches a C-style floating point number. These numbers start with an optional sign character, then have either a string of digits, a decimal point, and a string of zero or more digits; or a decimal point followed by one or more digits. After this is an optional exponent, written with the letter "E" (capital or small) followed by an optional sign followed by one or more digits.

       [-+]?([0-9]+%.?|[0-9]*%.[0-9]+)([eE][-+]?[0-9]+)?

Note the way we constructed the alternation that gives us the mantissa (the part before the exponent). We use the alternation to gives us one of two expressions:

       [0-9]+%.?

       [0-9]*%.[0-9]+

The first expression matches a string of one or more digits, followed by an optional decimal point. This matches numbers that have no decimal point at all, as well as numbers that end in a decimal point. The second expression matches zero or more digits, a decimal point, and then one or more digits. One might wonder why we didn't write the expression more simply like this:

       [0-9]*%.?[0-9]*

In other words, as zero or more digits, an optional decimal, and zero or more digits. The reason we didn't write the expression this way is that everything in this expression is optional – this one would match an empty string. It would also match a period, without any digits on either side. Obviously, we don't want to consider either an empty string or simply a period as a valid floating point number, so this simpler form of the expression is a little too general. The alternation solves these problems, because it allows for starting with a decimal, ending with a decimal, or containing an embedded decimal, but there must always be one or more digits on one side or the other of the decimal.