Buffer: A Performant, Mutable Text Buffer#

class my.types.Buffer.Buffer(*, text: list[str] = [''], uid: str = '', fences: ~typing.Annotated[~numpy._typing._array_like.NDArray[~numpy.int64], ~pydantic.types.GetPydanticSchema(get_pydantic_core_schema=~my.utils.SyntaxUtils.SyntaxUtils.pyd_schemify.<locals>.<lambda>, get_pydantic_json_schema=None)] = <factory>, fence_rgxs: list[str] = [], fence_rgx: ~typing.Annotated[~_regex.Pattern, ~pydantic.types.GetPydanticSchema(get_pydantic_core_schema=~my.utils.SyntaxUtils.SyntaxUtils.pyd_schemify.<locals>.<lambda>, get_pydantic_json_schema=None)] | None = None)#

A mutable text container optimized for iterative string modification.

Unlike immutable Python strings, buffers support in-place regex replacement while iterating over matches via functions such as rgx_iterator(). This enables complex text transformations that would otherwise require multiple passes or awkward index tracking.

Buffers also implement “fencing” to exclude certain regions from regex matching. Fences are defined by patterns (like code blocks in markdown) and stored as numpy arrays of span pairs for efficient lookup. When text is modified, the fence positions intersecting and/or following the changed span are efficiently updated, allowing for worry-free use by the caller. The typical usecase is for fenced code blocks in markdown or wikitext, which is where they get their name.

Finally, the class provides pair-matching functionality for finding balanced pairs of delimiters as simple as parens or as complex as HTML tags, handling nesting and self-closing delimiters along the way.

Some functionality is also included for identifying “[hanging] chads”, which refer to unmatched pair delimiters that are assumed to represent syntax errors in the original content.

Examples

Create a buffer and modify it in place:

>>> from my import Buffer
>>> buf = Buffer.new('hello world')
>>> buf.replace('world', 'there')
Buffer("hello there", len=11)
>>> str(buf)
'hello there'

Replace matches while iterating over them:

>>> buf = Buffer.new('x1 x2 x3')
>>> for match in buf.rgx_iterator(r'x(\d)'):
...     _ = buf.replace(match.span(), f'y{match[1]}!')
>>> str(buf)
'y1! y2! y3!'

Fence off backticked regions so pair matching skips them:

>>> buf = Buffer.new('a `b` c', fence_rgxs=['bactic'])
>>> list(buf.fence_spans)
[Span(2, 5)]

I Initial Methods#

Buffer.serialize() → str#

Serialize the Buffer instance to a string.

classmethod Buffer.new(text: list[str] | str | Self | None = None, uid: str = '', fence_rgxs: list[str] | None = None, no_fence: bool = False) → Self#

Create a new Buffer, flexibly coercing the given arguments.

The fence_rgxs parameter allows the caller to control exactly what content – if any – will be ignored while iterating through this buffer. For ease of use, 6 prewritten patterns can be identified by name:

  1. bactic: `` ... ``

  2. parens: (...)

  3. arrays: [...]

  4. braces: {...}

  5. blocks: {{...}}

  6. nowiki: <nowiki>...</nowiki>

If you’re creating lots of Buffers with the same fence_rgx`s, it may make sense to define a partial constructor, e.g.: `functools.partial(Buffer.new, fence_rgxs=['arrays']).

Parameters:

  • text – The string to coerce into the new buffer’s initial value.

  • uid – An optional identifier for the new buffer.

  • fence_rgxs – A list of regex patterns (or names of default patterns) to use as fences.

  • no_fence – If set to True, disables fence calculation even if fence_rgxs is given.

Returns:

A new Buffer instance (with fences calculated if possible).

Examples

Create a plain buffer, and a fenced one:

>>> from my import Buffer
>>> Buffer.new('hello world')
Buffer("hello world", len=11)
>>> Buffer.new('a `b` c', fence_rgxs=['bactic']).has_fences
True
Buffer.memcopy() → Self#

Create a deep copy of this Buffer instance without recalculating fences.

II Private Methods#

Buffer.update_fences(start: int, len_old: int, delta: int, diff: int = 0) → None#

(Re-)calculates the fence spans for the given region.

Handles both fences that are completely internal to the region, and fences that cross one of the boundaries. Sometimes, new cross-boundary fences can form where none existed before.

No changes are needed for fences that contain the region entirely; the normal index shifting after a replacement handles that case on its own.

Parameters:

  • start – The start position of the replaced region.

  • len_old – The length of the old text that was replaced.

  • delta – The change in length (new length - old length).

  • diff – When set, indicates that any fences found within the old text are still there, but have simply moved by this static amount.

III Primary Methods#

Buffer.raw_pair_iterator(rgx: Pattern, mode: 'all' | 'roots' | 'leaves' = 'all', b0: int = 0, b1: int = -1, strict: bool = True) → Iterator[tuple[Span, Span]]#

Find all the “pairs” of text delimiters matching the given regex, handling edge cases.

See pair_iterator() for full usage details.

Parameters:

  • rgx – The regex pattern defining the pair delimiters.

  • mode – 'all' by default, 'roots' to exclude nested pairs, or 'leaves' for the opposite.

  • b0 – The positive, inclusive start bound for searching.

  • b1 – The positive, exclusive end bound for searching, or -1 to search the whole text.

  • strict – If True, raise ValueError on unmatched starts; if False, silently ignore them.

Yields:

Tuples of spans representing the start and end delimiters.

IV Public Methods#

property Buffer.lines: list[str]#

The buffer’s text, split into a list of lines.

property Buffer.has_fences: bool#

Whether the buffer has any fences at the moment.

To see if fences are configured, use bool(buffer.fence_rgx) instead.

property Buffer.fence_spans: Iterator[Span]#

An iterator over the current fence spans.

Buffer.slice(start: int, end: int = -1) → str#

Convenience getter for slicing the buffer’s text.

Buffer.replace(old: Span | tuple[int, int] | str | Pattern, new: str, count: int = 0, diff: int = 0) → Self#

Replace the specified text with the new text, updating internal trackers as necessary.

Tip

Prefer to pass a precalculated span if you have it, to prevent rework.

Pattern replacements preserve matches beginning inside configured fences. Use rgx_iterator() directly when fenced matches must be processed deliberately.

Parameters:

  • old – The substring, span, or regex pattern to replace.

  • new – The new text to insert.

  • count – The maximum number of replacements to make (0 for all). NOOP for spans.

  • diff – When set, indicates that any fences found within the old text are still there, but have simply moved by this static amount.

Returns:

The modified Buffer instance (for convenient access and/or builder patterns).

Examples

Replace by substring, by span, or by regex:

>>> from my import Buffer
>>> import regex as re
>>> str(Buffer.new('a1 b2').replace(re.compile(r'\d'), '#'))
'a# b#'
>>> str(Buffer.new('hello world').replace((0, 5), 'goodbye'))
'goodbye world'
Buffer.insert(pos: int, new: str) → None#

Convenience setter for replacing an empty span with new text.

Parameters:

  • pos – The position to insert the new text at.

  • new – The new text to insert.

Buffer.drop(old: str | Span | tuple[int, int]) → Self#

Convenience setter for removing the specified text from the buffer.

Parameters:

old – The substring or span to remove.

Buffer.set(text: str) → Self#

Convenience setter for replacing the entire content of the buffer at once.

Parameters:

text – The new text to set the buffer to.

Buffer.clear() → Self#

Convenience setter for clearing the entire content of the buffer at once.

Buffer.strip(chars: str = '') → Self#

Performantly strip the buffer of leading/trailing characters.

Parameters:

chars – The characters to strip. Defaults to whitespace.

Returns:

The same (modified) Buffer instance.

Buffer.match(rgx: Pattern, *args, **kwargs) → Match | None#

Wrapper for regex.match().

Buffer.search(rgx: Pattern, *args, **kwargs) → Match | None#

Wrapper for regex.search().

Buffer.findall(rgx: Pattern, *args, **kwargs) → list[tuple[str, ...]]#

Wrapper for regex.findall().

Buffer.sub(rgx: Pattern, new: str, count: int = 0) → None#

Direct wrapper for _replace_regex(), defined to match the base regex interface.

Parameters:

  • rgx – The regex pattern to replace.

  • new – The new text to insert.

  • count – The maximum number of replacements to make (0 for all).

Buffer.apply(*functions: Callable[[str], str], b0: int = 0, b1: int = -1) → Self#

Iteratively maps one or more functions to the text in this buffer, modifying it.

Parameters:

  • functions – One or more functions that take a string and return a string.

  • b0 – The positive, inclusive start bound.

  • b1 – The positive, exclusive end bound, or -1 to apply up to the end of the text.

Returns:

The modified Buffer instance (for convenient access and/or builder patterns).

Examples

Apply a transformation to the whole buffer:

>>> from my import Buffer
>>> str(Buffer.new('abc').apply(str.upper))
'ABC'
Buffer.linespan(pos: int) → Span#

Find the start and end of the line on which pos sits.

Examples

Locate the line containing position 5:

>>> from my import Buffer
>>> Buffer.new('one\ntwo\nthree').linespan(5)
Span(4, 7)
Buffer.dedent() → Self#

Dedents all text evenly, so that the line with the fewest spaces starts at column 0.

Buffer.write(span: Span, text: str | Iterable[str], spacing: int = 0, diff: int = 0) → None#

Wrap a given string output in preparation for it to be inserted into the buffer.

Offers three distinct spacing modes, each of which also serves as the separator when text is an iterable of strings to join:

  • 0: Ensures a space (or other whitespace/punctuation) exists before and after

  • 1: Ensures a newline exists before and after

  • 2: Ensures two newlines (i.e. an empty line) exist before and after

Use replace() when no surrounding spacing should be added.

Parameters:

  • span – The span of text to replace.

  • text – The text to insert, either as a single string or an iterable of strings to join.

  • spacing – The spacing mode to use (0-2).

  • diff – When set, indicates that any fences found within the old text are still there, but have simply moved by this static amount.

Examples

Write into a buffer, ensuring space separation:

>>> from my import Buffer, Span
>>> buf = Buffer.new('ab')
>>> buf.write(Span(1, 2), 'c', spacing=0)
>>> str(buf)
'a c'
Buffer.find_chads(rgx: Pattern, b0: int = 0, b1: int = -1) → tuple[list[Span], list[Span]]#

Search for one or more “hanging chads” – unmatched delimiters of the given regex pair.

See find_pair_match() for details on pairs in general.

Parameters:

  • rgx – The regex pattern with the named groups ‘start’ and ‘end’.

  • b0 – The positive, inclusive start bound for searching.

  • b1 – The positive, exclusive end bound for searching, or -1 to search the whole text.

Returns:

(unmatched_starts, unmatched_ends)

Return type:

A tuple of two lists

Examples

Find the unmatched opening paren:

>>> from my import Buffer
>>> import regex as re
>>> pair_rgx = re.compile(r'(?P<start>\()|(?P<end>\))')
>>> Buffer.new('a (b (c d) e').find_chads(pair_rgx)
([Span(2, 3)], [])
Buffer.find_pair_match(rgx: Pattern, pos: int, b0: int = 0, b1: int = -1) → tuple[Span, str, str, str] | None#

Find the given substring’s “partner”, handling nesting & other edge cases.

Parameters:

  • rgx – The regex pattern with the named groups ‘start’ and ‘end’.

  • pos – The position of the known delimiter.

  • b0 – The start bound for searching.

  • b1 – The end bound for searching.

Returns:

(full_span, start_text, body_text, end_text) if a match is found, else None.

Examples

Find the pair enclosing position 6:

>>> from my import Buffer
>>> import regex as re
>>> pair_rgx = re.compile(r'(?P<start>\()|(?P<end>\))')
>>> Buffer.new('a (b (c) d) e').find_pair_match(pair_rgx, 6)
(Span(5, 8), '(', 'c', ')')
Buffer.pair_iterator(rgx: Pattern, mode: 'all' | 'roots' | 'leaves' = 'all', b0: int = 0, b1: int = -1) → Iterator[tuple[Span, str, str, str]]#

Iterate through matching “pairs” of delimiters, handling nesting and other edge cases.

Like the other Buffer iterators, this method supports a read+write paradigm where callers modify the buffer’s text while they iterate over it. To make this possible, it is assumed that the caller will only ever modify the last-yielded span of text during each iteration.

It also respects the ‘fence’ spans specified during initialization, such as code blocks in Markdown files or character sets in regular expressions.

Parameters:

  • rgx – The regex pattern with the named groups ‘start’ and ‘end’.

  • mode – ‘all’ by default, ‘roots’ to exclude nested pairs, or ‘leaves’ for the opposite.

  • b0 – The positive, inclusive start bound for searching.

  • b1 – The positive, exclusive end bound for searching, or -1 to search the whole text.

Yields:

(full_span, start_text, body_text, end_text) tuples, innermost pairs first.

Examples

Iterate over nested paren pairs:

>>> from my import Buffer
>>> import regex as re
>>> pair_rgx = re.compile(r'(?P<start>\()|(?P<end>\))')
>>> buf = Buffer.new('a (b (c) d) e')
>>> for span, start, body, end in buf.pair_iterator(pair_rgx):
...     print(span, repr(body))
5-7 'c'
2-10 'b (c) d'
Buffer.pair_list(rgx: Pattern, mode: 'all' | 'roots' | 'leaves' = 'all', b0: int = 0, b1: int = -1) → list[tuple[Span, str, str, str]]#

Materialized, cached version of pair_iterator() for read-only passes.

Returns the full list of (full_span, start_text, body_text, end_text) tuples. Results are cached per (rgx identity, mode, buffer version) and invalidated on any text mutation. Use this instead of pair_iterator() when the caller does not modify the buffer during iteration – it avoids re-running the regex scan on repeated calls with the same delimiter pattern.

Parameters:

  • rgx – The regex pattern with the named groups ‘start’ and ‘end’.

  • mode – ‘all’ by default, ‘roots’ to exclude nested pairs, or ‘leaves’ for the opposite.

  • b0 – The positive, inclusive start bound for searching.

  • b1 – The positive, exclusive end bound for searching, or -1 to search the whole text.

Returns:

A list of (full_span, start_text, body_text, end_text) tuples.

Examples

List only the outermost (root) pairs:

>>> from my import Buffer
>>> import regex as re
>>> pair_rgx = re.compile(r'(?P<start>\()|(?P<end>\))')
>>> pairs = Buffer.new('a (b (c) d) e').pair_list(pair_rgx, 'roots')
>>> [pair[0] for pair in pairs]
[Span(2, 11)]
Buffer.rgx_iterator(rgx: Pattern | str, recursive: bool = False, b0: int = 0, b1: int = -1, skip_fenced: bool = False) → Iterator[Match]#

Iterate over all matches of the given regex pattern in the buffer.

Like the other Buffer iterators, this method supports a read+write paradigm where callers modify the buffer’s text while they iterate over it. To make this possible, it is assumed that the caller will only ever modify the last-yielded match of text during each iteration.

By default, every match is yielded, including matches inside configured fences. This historical behavior lets callers deliberately process or remove the text defining a fence. Set skip_fenced to protect matches that begin inside fences, as the pair iterators do.

Parameters:

  • rgx – The regex pattern to search for (compiled or as a plain string).

  • recursive – If set, overlapping matches are also found.

  • b0 – The positive, inclusive start bound for searching.

  • b1 – The positive, exclusive end bound for searching, or -1 to search the whole text.

  • skip_fenced – If set, suppress matches that begin inside a configured fence.

Yields:

Match objects for each occurrence, adjusted for any in-flight modifications.

Examples

Rewrite each match while iterating:

>>> from my import Buffer
>>> buf = Buffer.new('x1 x2 x3')
>>> for match in buf.rgx_iterator(r'x(\d)'):
...     _ = buf.replace(match.span(), f'y{match[1]}!')
>>> str(buf)
'y1! y2! y3!'