Extended Regex Syntax in Python#

This file provides a concise reference for Matthew Barnett’s regex package, a popular extension of the standard re module that adds desperately-needed regex features (both long-awaited and brand-new).

Some of the additions – say, better support for emojis and non-roman script – likely seem arcane to the typical regex user, but I just can’t stress enough how powerful this package really is:

…and on and on!

This document started as a simple reorganization of Matthew’s fantastic README.md, but I have since added some brief elaborations, cross-references, & links to https://regular-expressions.info in order to make it more useful to both beginners and experts. Any allusions to “we” refer to Matthew and his fellow maintainers.

If you’re new to regex in general, I recommend you start by reading python’s quick tutorial, skimming regular-expression.info’s much more in-depth tutorials, and/or consulting python’s documentation for the re module directly. For completeness, I have included brief quotes from the re documentation to set the stage for many of the extensions – look out for “See Also” notes.

1. Flags#

(regular-expressions.info lesson)

Basic Syntax

See Python’s standard-library re documentation for the basic, global flag syntax; the sections below cover the scoped and extended flags added by the regex module.

1.1. Scoped Flags#

Scoped flags can apply to only part of a pattern and can be turned on or off.

1.1.1. Encoding#

1.1.1.1. (?u) UNICODE#

The default encoding of a regex string, matching everything according to international Unicode standards.

1.1.1.2. (?a) ASCII#

Makes \w, \W, \b, \B, \d, \D, \s and \S match only ASCII characters.

1.1.1.3. (?L) LOCALE#

Makes \w, \W, \b, \B, \d, \D, \s and \S match according to the current locale settings.

Caution

This flag is intended for legacy code and has limited support. We recommended you use UNICODE instead.

1.1.2. Case#

1.1.2.1. (?i) IGNORECASE#

Upper and lower case alphabet characters are matched as if they were identical.

1.1.2.2. (?f) FULLCASE#

When combined with (?i), enables “full” case-folding of Unicode text, which is critical if dealing with non-romance languages.

The flag is off by default in V0, and on by default in V1.

regex.match(r"(?iV1)strasse", "stra\N{LATIN SMALL LETTER SHARP S}e").span() # -> (0, 6)
regex.match(r"(?iV1)stra\N{LATIN SMALL LETTER SHARP S}e", "STRASSE").span() # -> (0, 7)

1.1.3. Whitespace#

1.1.3.1. (?m) MULTILINE#

(regular-expressions.info lesson)

The ^ and $ literals now match the beginnings and ends of lines, rather than of the whole string/file.

1.1.3.2. (?s) DOTALL#

(regular-expressions.info lesson)

The catch-all literal . now also catches line separators.

1.1.3.3. (?x) VERBOSE (i.e. Extended)#

(regular-expressions.info lesson)

Ignores all raw whitespace characters in the following pattern, allowing the user to include whitespace between components for clarity. Also allows comments, which begin with an “#” and continue until the end of the line.

To match whitespace in an extended expression, wrap it in a character set (e.g. ? -> [ ]?).

1.1.3.4. (?w) WORD#

Changes the definition of a ‘word boundary’ (\b/\B) to that of a default Unicode word boundary, a better choice for a variety of non-romance languages.

It also affects line separators (and, in turn, (?s) and (?m)):

  • Without this flag, the only line separator is \n (\x0A),

  • With this flag, \x0D\x0A, \x0A, \x0B, \x0C and \x0D are valid line separators, plus \x85, \u2028 and \u2029 when working with Unicode.

1.2. Global Flags#

Global flags apply to the entire pattern and can only be turned on – if these patterns are present anywhere in a given expression, they apply to the whole thing. They cannot be disabled.

1.2.1. General#

1.2.1.1. (?p) Posix#

Enables POSIX (leftmost longest) matching.

Note that it will take longer to find matches because when it finds a match at a certain position, it won’t return that immediately, but will keep looking to see if there’s another longer match there.

regex.search(r'Mr|Mrs', 'Mrs') # -> Mr
regex.search(r'(?p)Mr|Mrs', 'Mrs') # -> Mrs

regex.search(r'one(self)?(selfsufficient)?', 'oneselfsufficient') # -> oneself
regex.search(r'(?p)one(self)?(selfsufficient)?', 'oneselfsufficient') # -> oneselfsufficient
1.2.1.2. (?r) Reverse#

Enables reverse matching (from the end of the string to the beginning).

regex.findall(r".", "abc") # -> ['a', 'b', 'c']
regex.findall(r"(?r).", "abc") # -> ['c', 'b', 'a']

Note that the result of a reverse search is not necessarily the reverse of a forward search:

regex.findall(r"..", "abcde") # -> ['ab', 'cd']
regex.findall(r"(?r)..", "abcde") # -> ['de', 'bc']

1.2.2. Version#

1.2.2.1. (?V0) Version0#

Enables version 0 behaviour (old behaviour, compatible with the re module).

1.2.2.2. (?V1) Version1#

Enables version 1 behaviour (new behaviour, possibly different from the re module).

1.2.3. Fuzzy Match Modes#

1.2.3.1. (?b) Best Match#

Enables fuzzy matching search for the best match instead of the next match.

1.2.3.2. (?e) Enhance Match#

Enables fuzzy matching to attempt to improve the fit of the next match that it finds.

2. Sets#

2.1. Simple vs. Expanded Sets#

In Version 0, only simple sets are supported. For example, the pattern [[a-z]--[aeiou]] is mangled and interpreted as:

  • Set containing a literal “[” and the letters “a” to “z”

  • Literal “–”

  • Set containing letters “a”, “e”, “i”, “o”, “u”

  • Literal “]”

In version 1, the same pattern ([[a-z]--[aeiou]]) is a single set that uses a set operator to match all the lowercase letters from a to z except for the vowels (a, e, i, o, u).

2.2. Set operators#

(regular-expressions.info lesson 1, lesson 2)

Version 1’s set operators allow a set ([...]) to be composed of smaller sets. The operators, in order of increasing precedence, are:

Syntax

Name

Boolean

Example

||

Union

OR

[\w||[:punct:]] matches word and punctuation characters.

~~

Symmetric Difference

XOR

[\w~~[:punct:]] matches words or punctuation, but not _.

&&

Intersection

AND

[\w&&[:punct:]] matches only _.

--

Difference

SUB

[\w--[:punct:]] matches all word characters except _.

Note

Implicit union, ie, simple juxtaposition like in [ab], has the highest precedence. Thus, [ab&&cd] is the same as [[a||b]&&[c||d]].

Examples:

r'[ab]' # Matches 'a' or 'b'
r'[a-z]' # Matches all lowercase letters (from 'a' to 'z')
r'[[a-z]--[qw]]' # Set containing 'a' .. 'z', but not 'q' or 'w'
r'[a-z--qw]' # Same as above
r'[\p{L}--QW]' # Set containing all letters except 'Q' and 'W'
r'[\p{N}--[0-9]]' # Set containing all numbers except '0' .. '9'
r'[\p{ASCII}&&\p{Letter}]' # Set containing all characters which are ASCII and letter

3. Groups#

(regular-expressions.info lesson)

3.1. Subroutines#

(regular-expressions.info lesson)

All unnamed capturing groups ((...)) are assigned a group number, starting from 1. Groups with the same group name will have the same group number, and groups with a different group name will have a different group number.

The same name can be used by more than one group, with later captures ‘overwriting’ earlier captures. All the captures of the group will be available from the captures method of the match object.

3.1.1. Substituting & Invoking Subroutines#

Subroutines are useful for two things:

  1. “Substitution”/”Backreferencing” of their results: the exact text that the subroutine last matched will be sought out again. The quintessential usecase is matching quoted text: (["\'])(\w+)(\1).

  2. “Invocation” of their pattern: the subroutine itself is re-run at the current location, entirely separate from any previous uses. Like functions do for regular imperative programming languages, subroutines are so incredibly useful that it’s impossible to pick any one quintessential usecase – composition, recursion, and more are all possible.

Although the former is supported by re, the latter requires the full regex package. Many different syntaxes are used across different languages, so for convenience, here is a table clarifying exactly which syntaxes are supported by regex for each of these functions:

Group Type

Action Type

Supported

Unsupported

Numbered

Definition

(...)

Numbered

Substitution

\1, \g<1>, (?P=1)

Numbered

Invocation

(?1), (?R)

(?-1), (?+1)

Named

Definition

(?P<name>...)

(?<name>...), (?P'name'...), (?P"name"...)

Named

Substitution

(?P=name), \g<name>

\k<name>, \k'name', \k{name}, \g{name}, (?<name>), (?'name')

Named

Invocation

(?&name), (?P>name), (?P&name)

When substituting, the result of the most recent invocation is used; sadly, relative backreferences are not (yet?) supported. To get around this, wrap your subroutine with a new name (e.g. (?P<important_quote>(?P>quote))) for the calls whose results you want to substitute later on.

Note that you can only invoke a group if there is only one unique group with that name – else, an "ambiguous group reference" exception will be raised.

# Basic backreference
regex.match(r"(Tarzan|Jane) loves (?1)", "Tarzan loves Jane").groups() # -> ('Tarzan',)
regex.match(r"(Tarzan|Jane) loves (?1)", "Jane loves Tarzan").groups() # -> ('Jane',)

# Recursive
m = regex.search(r"(\w)(?:(?R)|(\w?))\1", "kayak")
m.group(0, 1, 2) # -> ('kayak', 'k', None)

# By number
regex.match(r'(["\'])(\w+)(\g<1>)', '"abc\' "def"') # -> "def"
regex.match(r'(?P<quote>["\'])(\w+)(\g<1>)', "'abc\" 'def'") # -> 'def'

# By name
regex.match(r'(?P<quote>["\'])(\w+)(\g<quote>)', '"abc\' "def"') # -> "def"
regex.match(r'(?P<quote>["\'])(\w+)(?P=quote)', "'abc\" 'def'") # -> 'def'

3.1.2. Predefined Subroutines ((?(DEFINE)...))#

This special group can be placed at the start of a complex pattern to define subroutines that can be invoked later on in the pattern, but that will not themselves be matched against the string. The normal rules for numbering groups still apply.

Caution

If you define a subroutine that shares the name DEFINE, this section will break.

regex.search(r'(?(DEFINE)(?P<quant>\d+)(?P<item>\w+))(?&quant) (?&item)', '5 elephants') # -> 5 elephants

3.2. Conditional Groups ((?(1)then|else))#

(regular-expressions.info lesson)

3.2.1. Lookarounds in Conditionals#

The test of a conditional pattern can be a lookaround:

regex.match(r'(?(?=\d)\d+|\w+)', '123abc') # -> 123
regex.match(r'(?(?=\d)\d+|\w+)', 'abc123') # -> abc123

This is not quite the same as putting a lookaround in the first branch of a pair of alternatives:

regex.match(r'(?:(?=\d)\d+\b|\w+)', '123abc') # -> 123abc
regex.match(r'(?(?=\d)\d+\b|\w+)', '123abc') # -> None

In the first example, the lookaround matched, but the remainder of the first branch failed to match, and so the second branch was attempted, whereas in the second example, the lookaround matched, and the first branch failed to match, but the second branch was not attempted.

3.3. Branch Reset Groups ((?|...))#

(regular-expressions.info lesson)

The special “branch reset” group syntax (e.g. (?|(first)|(second))) allows group numbers to be reused across different branches (the given example has only group 1). If groups have different group names then they will still have different group numbers.

Group numbers will be reused across the alternatives, but groups with different names will have different group numbers.

In the regex (\s+)(?|(?P<foo>[A-Z]+)|(\w+) (?P<foo>[0-9]+) there are 2 groups:

  • (\s+) is group 1.

  • (?P<foo>[A-Z]+) is group 2, also called “foo”.

  • (\w+) is group 2 because of the branch reset.

  • (?P<foo>[0-9]+) is group 2 because it’s called “foo”.

# Note that there is only one group:
regex.match(r"(?|(first)|(second))", "first").groups() # -> ('first',)
regex.match(r"(?|(first)|(second))", "second").groups() # -> ('second',)

3.4. Variable-length lookbehind ((?<=^.*))#

(regular-expressions.info lesson)

A lookbehind can match a variable-length string.

4. Other Literals#

4.1. Unicode Properties (\p{property})#

(regular-expressions.info lesson)

Unicode properties allow you to more flexibly match non-standard characters, and as such are highly recommended for production applications. For example, [A-Z] will not match Ö as expected, but [\p{Lu}] and [[:upper:]] will.

Four syntaxes are accepted:

  1. \p{value} matches characters with the given property.

  2. \P{value} matches characters without the given property.

  3. \p{property=value} matches characters whose property property does have value value.

  4. \P{property=value} matches characters whose property property does not have value value.

There are four types of Unicode properties, which are checked in order:

  1. General_Category properties describe all sorts of arbitrary information (e.g. Letter, Mark, Number, Punctuation, Symbol, Separator, Other)

  2. Script properties describe which real-life writing system the character belongs to (e.g. Latin, Cyrillic, Han).

  3. Block properties describe which of the (somewhat-arbitrary) Unicode Blocks the character was published within (e.g. Basic Latin, Greek and Coptic, CJK Unified Ideographs).

  4. Finally, all other binary properties are checked.

A short form starting with Is indicates a script or binary property (e.g. \p{IsLatin} or \p{IsAlphabetic}), while a short form starting with In indicates a block property (e.g. \p{InBasicLatin}).

In addition to the usual properties, you can also use:

  • \p{Horiz_Space}/\p{H} matches horizontal whitespace.

  • \p{Vert_Space}/\p{V} matches vertical whitespace.

4.2. POSIX Character Classes ([[:class:]])#

(regular-expressions.info lesson)

POSIX character classes are supported, e.g. [[:alpha:]] (matches alphabet characters) and its inverse [[:^alpha:]].

These are normally treated as an alternative form of \p{...}, except for alnum, digit, punct and xdigit, whose definitions are different from those of Unicode:

POSIX Class

Equivalent Unicode Property

[[:alnum:]]

\p{posix_alnum}

[[:digit:]]

\p{posix_digit}

[[:punct:]]

\p{posix_punct}

[[:xdigit:]]

\p{posix_xdigit}

4.3. Search Anchors (\G)#

(regular-expressions.info lesson)

The search anchor matches at the position where each search started/continued from.

# Can be used to find contiguous matches:
regex.findall(r"\w{2}", "abcd ef") # -> ['ab', 'cd', 'ef'], vs.
regex.findall(r"\G\w{2}", "abcd ef") # -> ['ab', 'cd']

# Or in negative variable-length lookbehinds to limit how far back the lookbehind goes
regex.findall(r"(?<!X.*)\w+", "aXa bXb") # -> ['aXa']
regex.findall(r"(?<!\G.*X.*)\w+", "aXa bXb") # -> ['aXa', 'bXb']

4.4. Word boundaries (\m\M\b\B)#

(regular-expressions.info lesson)

\m matches at the start of a word, and \M matches at the end of a word.

The definition of a ‘word’ character (\w) has also been expanded to conform to the Unicode specification at http://www.unicode.org/reports/tr29 – see regular-expressions.info’s discussion for details.

4.5. Named characters (\N{name})#

Named characters are supported. Note that only those known by Python’s Unicode database will be recognised.

4.6. A single grapheme (\X)#

(regular-expressions.info lesson)

The grapheme matcher is supported. It conforms to the Unicode specification at http://www.unicode.org/reports/tr29/.

5. Optimization#

5.1. Atomic grouping (?>...)#

(regular-expressions.info lesson)

If the following pattern subsequently fails, then the subpattern as a whole will fail.

5.2. Possessive quantifiers (.*+)#

(regular-expressions.info lesson)

(?:...)?+ ; (?:...)*+ ; (?:...)++ ; (?:...){min,max}+

The subpattern is matched up to ‘max’ times. If the following pattern subsequently fails, then all the repeated subpatterns will fail as a whole. For example, (?:...)++ is equivalent to (?>(?:...)+).

5.3. Backtracking Control Verbs ((*VERB))#

(regular-expressions.info lesson)

We support 3 of the 7 control verbs (excluding ACCEPT, COMMIT, MARK, and THEN):

  1. (*PRUNE) discards the backtracking info up to that point. When used in an atomic group or a lookaround, it won’t affect the enclosing pattern.

  2. (*SKIP) is similar to (*PRUNE), except that it also sets where in the text the next attempt to match will start. When used in an atomic group or a lookaround, it won’t affect the enclosing pattern.

  3. (*FAIL)/(*F) causes immediate backtracking.

5.4. Keep (\K)#

(regular-expressions.info lesson)

Keeps the part of the entire match after the position where \K occurred; the part before it is discarded. It does not affect what groups return.

m = regex.search(r'(\w\w\K\w\w\w)', 'abcdef')
m[0] # -> cde
m[1] # -> abcde

m = regex.search(r'(?r)(\w\w\K\w\w\w)', 'abcdef')
m[0] # -> bc
m[1] # -> bcdef

5.5. Named Lists (\L<name>)#

There are occasions where you may want to include a list (actually, a set) of options in a regex.

One way is to build the pattern like this:

p = regex.compile(r"first|second|third|fourth|fifth")

…but if the list is large, parsing the resulting regex can take considerable time, and care must also be taken that the strings are properly escaped and properly ordered, for example, “cats” before “cat”.

The new alternative is to use a named list:

option_set = ["first", "second", "third", "fourth", "fifth"]
p = regex.compile(r"\L<options>", options=option_set)

The order of the items is irrelevant, they are treated as a set. The named lists are available as the .named_lists attribute of the pattern object:

print(p.named_lists)
# -> {'options': frozenset({'third', 'first', 'fifth', 'fourth', 'second'})}

If there are any unused keyword arguments, ValueError will be raised unless you tell it otherwise:


option_set = ["first", "second", "third", "fourth", "fifth"]
p = regex.compile(r"\L<options>", options=option_set, other_options=[])
# -> ValueError: unused keyword argument 'other_options'

6. Python API#

6.1. Environment#

  • Python 2 is not supported.

  • This module is targeted at CPython.

  • Threading is supported IF strings don’t change during matching.

  • This module supports Unicode 17.0.0 and full Unicode case-folding.

6.2. Repeated Matches#

6.2.1. Captures & Spans#

A match object has additional methods which return information on all the successful matches of a repeated group. These methods are:

Method

Description

Singular Counterpart

.captures([group1, ...])

The strings matched for one group, or a list of lists for multiple.

.group([group1, ...])

.starts([group])

The start positions for this group.

.start([group])

.ends([group])

The end positions for this group.

.end([group])

.spans([group])

The spans for this group.

.span([group])

.allcaptures

All the captures of all the groups.

N/A

.allspans

All the spans of the all captures of all the groups.

N/A

 
m = regex.search(r"(\w{3})+", "123456789")
m.group(1) # -> '789'
m.captures(1) # -> ['123', '456', '789']
m.start(1) # -> 6
m.starts(1) # -> [0, 3, 6]
m.end(1) # -> 9
m.ends(1) # -> [3, 6, 9]
m.span(1) # -> (6, 9)
m.spans(1) # -> [(0, 3), (3, 6), (6, 9)]

m = regex.match(r"(?:(?P<word>\w+) (?P<digits>\d+)\n)+", "one 1\ntwo 2\nthree 3\n")
m.allcaptures() # -> (['one 1\ntwo 2\nthree 3\n'], ['one', 'two', 'three'], ['1', '2', '3'])
m.allspans() # -> ([(0, 20)], [(0, 3), (6, 9), (12, 17)], [(4, 5), (10, 11), (18, 19)])

6.2.2. Capture Mappings#

capturesdict returns a dict of the named groups and lists of all the captures of those groups. It is a combination of groupdict and captures:

  • groupdict returns a dict of the named groups and the last capture of those groups.

  • captures returns a list of all the captures of a group

m = regex.match(r'(?:(?P<word>\w+) (?P<digits>\d+)\n)+', "one 1\ntwo 2\nthree 3\n")
m.groupdict() # -> {'word': 'three', 'digits': '3'}
m.captures("word") # -> ['one', 'two', 'three']
m.captures("digits") # -> ['1', '2', '3']
m.capturesdict() # -> {'word': ['one', 'two', 'three'], 'digits': ['1', '2', '3']}

Group names can be duplicated:

# ---------------
# OPTIONAL GROUPS
# ---------------
# Both groups capture, the second capture 'overwriting' the first.
m = regex.match(r"(?P<item>\w+)? or (?P<item>\w+)?", "first or second")
m.group("item") # -> 'second'
m.captures("item") # -> ['first', 'second']

# Only the second group captures.
m = regex.match(r"(?P<item>\w+)? or (?P<item>\w+)?", " or second")
m.group("item") # -> 'second'
m.captures("item") # -> ['second']

# Only the first group captures.
m = regex.match(r"(?P<item>\w+)? or (?P<item>\w+)?", "first or ")
m.group("item") # -> 'first'
m.captures("item") # -> ['first']

# ----------------
# MANDATORY GROUPS
# ----------------
# Both groups capture, the second capture 'overwriting' the first.
m = regex.match(r"(?P<item>\w\*) or (?P<item>\w\*)?", "first or second")
m.group("item") # -> 'second'
m.captures("item") # -> ['first', 'second']

# Again, both groups capture, the second capture 'overwriting' the first.
m = regex.match(r"(?P<item>\w\*) or (?P<item>\w\*)", " or second")
m.group("item") # -> 'second'
m.captures("item") # -> ['', 'second']

# And yet again, both groups capture, the second capture 'overwriting' the first.
m = regex.match(r"(?P<item>\w\*) or (?P<item>\w\*)", "first or ")
m.group("item") # -> ''
m.captures("item") # -> ['first', '']

6.2.3. Match Subscripting (match[0])#

Match objects now allow access to their results via subscripting and slicing:

m = regex.search(r"(?P<before>.*?)(?P<num>\d+)(?P<after>.*)", "pqr123stu")
print(m["before"]) # -> pqr
print(len(m)) # -> 4
print(m[:]) # -> ('pqr123stu', 'pqr', '123', 'stu')

You can also use subscripting to get the captures of a repeated group for expandf and subf:


m = regex.match(r'(\w)+', 'abc')
m.expandf('{1}') # -> c
m.expandf('{1[0]} {1[1]} {1[2]}') # -> a b c
m.expandf('{1[-1]} {1[-2]} {1[-3]}') # -> c b a'

m = regex.match(r'(?P<letter>\w)+', 'abc')
m.expandf('{letter}') # -> c
m.expandf('{letter[0]} {letter[1]} {letter[2]}') # -> a b c
m.expandf('{letter[-1]} {letter[-2]} {letter[-3]}') # -> c b a

6.3. New Functions#

6.3.1. splititer()#

regex.splititer has been added. It’s a generator equivalent of regex.split.

6.3.2. fullmatch()#

fullmatch behaves like match, except that it must match all of the string.


print(regex.fullmatch(r"abc", "abc").span()) # -> (0, 3)
print(regex.fullmatch(r"abc", "abcx")) # -> None
print(regex.fullmatch(r"abc", "abcx", endpos=3).span()) # -> (0, 3)
print(regex.fullmatch(r"abc", "xabcy", pos=1, endpos=4).span()) # -> (1, 4)

regex.match(r"a.*?", "abcd").group(0) # -> 'a'
regex.fullmatch(r"a.*?", "abcd").group(0) # -> 'abcd'

6.3.3. subf() & subfn()#

subf and subfn are alternatives to sub and subn respectively. When passed a replacement string, they treat it as a format string.


regex.subf(r"(\w+) (\w+)", "{0} => {2} {1}", "foo bar") # -> 'foo bar => bar foo'
regex.subf(r"(?P<word1>\w+) (?P<word2>\w+)", "{word2} {word1}", "foo bar") # -> 'bar foo'

6.3.4. expandf()#

expandf is an alternative to expand. When passed a replacement string, it treats it as a format string.

m = regex.match(r"(\w+) (\w+)", "foo bar")
m.expandf("{0} => {2} {1}") # -> 'foo bar => bar foo'

m = regex.match(r"(?P<word1>\w+) (?P<word2>\w+)", "foo bar")
m.expandf("{word2} {word1}") # -> 'bar foo'

6.3.5. detach_string()#

A match object contains a reference to the string that was searched, via its string attribute. The detach_string method will ‘detach’ that string, making it available for garbage collection, which might save valuable memory if that string is very large.

m = regex.search(r"\w+", "Hello world")
print(m.group()) # -> Hello
print(m.string) # -> Hello world

m.detach_string()
print(m.group()) # -> Hello
print(m.string) # -> None

6.4. New Arguments#

6.4.1. Partial Matches#

A partial match is one that matches up to the end of string, but that string has been truncated and you want to know whether a complete match could be possible if the string had not been truncated.

Partial matches are supported by match, search, fullmatch and finditer with the partial keyword argument.

Match objects have a partial attribute, which is True if it’s a partial match.

For example, if you wanted a user to enter a 4-digit number and check it character by character as it was being entered:

pattern = regex.compile(r'\d{4}')

# Initially, nothing has been entered:
print(pattern.fullmatch('', partial=True))
# -> <regex.Match object; span=(0, 0), match='', partial=True>
# (An empty string is OK, but it's only a partial match.)

# The user enters a letter:
print(pattern.fullmatch('a', partial=True))
# -> None
# (It'll never match.)

# The user deletes that and enters a digit:
print(pattern.fullmatch('1', partial=True))
# -> <regex.Match object; span=(0, 1), match='1', partial=True>
# (It matches this far, but it's only a partial match.)

# The user enters 2 more digits:
print(pattern.fullmatch('123', partial=True))
# -> <regex.Match object; span=(0, 3), match='123', partial=True>
# (It matches this far, but it's only a partial match.)

# The user enters another digit:
print(pattern.fullmatch('1234', partial=True))
# -> <regex.Match object; span=(0, 4), match='1234'>
# (It's a complete match.)

# If the user enters another digit:
print(pattern.fullmatch('12345', partial=True))
# -> None
# (It's no longer a match.)

pattern.match('123', partial=True).partial # -> True
pattern.match('1233', partial=True).partial # -> False

6.4.2. Special Escapes#

regex.escape has an additional keyword parameter special_only. When True, only ‘special’ regex characters, such as ‘?’, are escaped.

regex.escape("foo!?", special_only=False) # -> 'foo\!\?'
regex.escape("foo!?", special_only=True) # -> 'foo!\?'

regex.escape("foo bar!?", literal_spaces=False) # -> 'foo\ bar!\?'
regex.escape("foo bar!?", literal_spaces=True) # -> 'foo bar!\?'

6.4.3. Boundaries#

regex.sub and regex.subn support ‘pos’ and ‘endpos’ arguments.

6.4.4. Overlaps#

regex.findall and regex.finditer support an ‘overlapped’ argument which permits overlapped matches.

6.4.5. Flags#

regex.split, regex.sub and regex.subn support a ‘flags’ argument.

6.4.6. Timeouts#

The matching methods and functions support timeouts. The timeout (in seconds) applies to the entire operation:

from time import sleep

def fast_replace(m):
    return 'X'

def slow_replace(m):
    sleep(0.5)
    return 'X'

regex.sub(r'[a-z]', fast_replace, 'abcde', timeout=2) # -> 'XXXXX'
regex.sub(r'[a-z]', slow_replace, 'abcde', timeout=2) # -> TimeoutError: regex timed out

7. Fuzzy Matching#

Regex usually attempts an exact match, but sometimes an approximate, or “fuzzy”, match is needed, for those cases where the text being searched may contain errors in the form of inserted, deleted or substituted characters.

7.1. Basics#

A fuzzy regex specifies A) which types of errors are permitted, and, optionally, B) either the minimum and maximum or only the maximum permitted number of each type. The 3 types of error are: Insertion (i), Deletion (d), Substitution (s), or any (e). You can also use “<” instead of “<=” if you want an exclusive minimum or maximum. You cannot specify only a minimum.

The fuzziness of a regex item is specified between { and } after the item:

  • foo match “foo” exactly

  • (?:foo){i} match “foo”, permitting insertions

  • (?:foo){d} match “foo”, permitting deletions

  • (?:foo){s} match “foo”, permitting substitutions

  • (?:foo){i,s} match “foo”, permitting insertions and substitutions

  • (?:foo){e} match “foo”, permitting errors

If a certain type of error is specified, then any type not specified will not be permitted:

  • (...){d<=3} permit at most 3 deletions, but no other types

  • (...){i<=1,s<=2} permit at most 1 insertion and at most 2 substitutions, but no deletions

  • (...){1<=e<=3} permit at least 1 and at most 3 errors

  • (...){i<3,d<=2,e<4} permit at most 2 insertions, at most 2 deletions, at most 3 errors in total, but no substitutions

7.2. Costs & Budgets#

It’s also possible to state the costs of each type of error and the maximum permitted total cost:

  • (...){2i+2d+1s<=4} each insertion costs 2, each deletion costs 2, each substitution costs 1, the total cost must not exceed 4

  • (...){i<=1,d<=1,s<=1,2i+2d+1s<=4} at most 1 insertion, at most 1 deletion, at most 1 substitution; each insertion costs 2, each deletion costs 2, each substitution costs 1, the total cost must not exceed 4

7.3. Tests#

You can add a test to perform on a character that’s substituted or inserted.

Examples:

  • (...){s<=2:[a-z]} at most 2 substitutions, which must be in the character set [a-z].

  • (...){s<=2,i<=3:\d} at most 2 substitutions, at most 3 insertions, which must be digits.

7.4. Flags#

By default, fuzzy matching searches for the first match that meets the given constraints. The ENHANCEMATCH flag will cause it to attempt to improve the fit (i.e. reduce the number of errors) of the match that it has found. The BESTMATCH flag will make it search for the best match instead.

Examples:

  • regex.search("(dog){e}", "cat and dog")[1] returns "cat" because that matches "dog" with 3 errors (an unlimited number of errors is permitted).

  • regex.search("(dog){e<=1}", "cat and dog")[1] returns " dog" (with a leading space) because that matches "dog" with 1 error, which is within the limit.

  • regex.search("(?e)(dog){e<=1}", "cat and dog")[1] returns "dog" (without a leading space) because the fuzzy search matches " dog" with 1 error, which is within the limit, and the (?e) then it attempts a better fit.

In the first two examples there are perfect matches later in the string, but in neither case is it the first possible match.

The match object has an attribute fuzzy_counts which gives the total number of substitutions, insertions and deletions:

# A 'raw' fuzzy match:
regex.fullmatch(r"(?:cats|cat){e<=1}", "cat").fuzzy_counts # -> (0, 0, 1)
# 0 substitutions, 0 insertions, 1 deletion.

# A better match might be possible if the ENHANCEMATCH flag used:
regex.fullmatch(r"(?e)(?:cats|cat){e<=1}", "cat").fuzzy_counts # -> (0, 0, 0)
# 0 substitutions, 0 insertions, 0 deletions.

The match object also has an attribute fuzzy_changes which gives a tuple of the positions of the substitutions, insertions and deletions:

m = regex.search('(fuu){i<=2,d<=2,e<=5}', 'anaconda foo bar')
# -> <regex.Match object; span=(7, 10), match='a f', fuzzy_counts=(0, 2, 2)>
m.fuzzy_changes # -> ([], [7, 8], [10, 11])

What this means is that if the matched part of the string had been:

'anacondfuuoo bar'

…it would’ve been an exact match.

However, there were insertions at positions 7 and 8:

'anaconda fuuoo bar'
#       ^^

…and deletions at positions 10 and 11:

'anaconda f~~oo bar'
#          ^^

So the actual string was:

'anaconda foo bar'

8. Known Issues / Complexities#

8.1. * operator not working correctly with sub()#

Sometimes it’s not clear how zero-width matches should be handled. For example, should .* match 0 characters directly after matching >0 characters?

regex.sub('.*', 'x', 'test') # -> xx
regex.sub('.*?', '|', 'test') # -> |||||||||