Text¶
Constructors¶
create¶
An empty Text. len is a capacity hint passed through to the
underlying String. Returns ephemeral; bind to whichever cap the
context requires (most often val, matching Pony's String
constructor convention).
Parameters¶
- len: USize val = 0
Returns¶
- Text ref^
from_string¶
Wrap a String val after validating it as UTF-8. Raises if the
input is ill-formed.
If indexed is true, also builds the grapheme-start bitmap index
so subsequent size_graphemes() / size_codepoints() are O(1)
and random grapheme lookups are ~64× faster (see §3.5 of the
design notes). Pay-for-what-you-use — pass-through workloads that
never index pay nothing.
Parameters¶
Returns¶
- Text val^ ?
from_array¶
Wrap a byte array after validating it as UTF-8. Raises on
ill-formed input. The bytes are copied into the Text's internal
String ref buffer. See from_string for the indexed flag.
Parameters¶
Returns¶
- Text val^ ?
from_iso_string¶
Zero-byte-copy adoption of an iso String when indexed is false.
When indexed is true, builds the optional bitmap index — this
requires viewing the bytes via a val alias, which costs one extra
byte-copy (the opt-in price of indexing).
Validates UTF-8 in place. Raises on ill-formed input. See
from_string for the indexed flag.
Parameters¶
Returns¶
- Text iso^ ?
from_iso_array¶
Zero-byte-copy adoption of an iso byte array when indexed is
false. See from_iso_string for the indexed tradeoff.
Parameters¶
Returns¶
- Text iso^ ?
Public Functions¶
size_bytes¶
Number of UTF-8 bytes in this Text. O(1).
Returns¶
- USize val
utf8_bytes¶
Return a fresh String iso^ containing a copy of this Text's
UTF-8 bytes. The caller owns the returned String and can mutate,
consume, or recover to any cap. One byte-copy per call — for
callers that just want a read-only view, the underlying String
can be reached via viewpoint adaptation in the package's own
code; external callers always pay the copy.
Returns¶
- String iso^
graphemes¶
Iterate over the UAX #29 extended grapheme clusters in this Text.
Each yielded String val is a zero-byte-copy slice of the
underlying UTF-8 buffer (one small String wrapper allocation
per yield).
For zero-allocation iteration on large texts, use
grapheme_ranges().
Returns¶
grapheme_ranges¶
Iterate over grapheme clusters as (start_byte, end_byte_exclusive)
pairs. No per-yield allocation. Pair with utf8_bytes() to
materialize specific clusters lazily.
Returns¶
words¶
Iterate over UAX #29 word segments as zero-byte-copy String val
slices. UAX #29 defines boundaries, not "wordness" — punctuation
runs and whitespace runs each show up as their own segment. Use
Codepoints.is_letter / is_digit on the first cp of a segment
to filter to words-as-text yourself.
Returns¶
word_ranges¶
Iterate over word segments as (start_byte, end_byte_exclusive)
byte pairs. No per-yield allocation.
Returns¶
sentences¶
Iterate over UAX #29 sentence segments as zero-byte-copy
String val slices.
Returns¶
sentence_ranges¶
Iterate over sentence segments as (start_byte, end_byte_exclusive)
byte pairs.
Returns¶
lines¶
Iterate over UAX #14 line-break opportunities, yielding each line
as a zero-byte-copy String val slice. Each yielded slice ends
where a line break would occur (either a mandatory break — LF, CR,
NEL — or a soft opportunity).
Returns¶
line_ranges¶
Iterate over line segments as (start_byte, end_byte_exclusive)
byte pairs.
Returns¶
scripts¶
The set of script(cp) values for every codepoint in this Text.
Includes ScriptCommon (digits, punctuation) and
ScriptInherited (combining marks) when present; call
.resolved() on the result to drop those.
Returns¶
- ScriptSet val
dominant_script¶
The most-frequent non-Common, non-Inherited script in this Text.
Ties broken by first-seen. Returns ScriptCommon for an empty
or all-Common/Inherited Text.
Returns¶
contains_unassigned_codepoint¶
True iff this Text contains at least one codepoint with
General_Category = Cn (unassigned). Useful as an
identifier-input filter: unassigned codepoints in user input are
usually a sign of bad data or an attempt to exploit Unicode
version skew.
Returns¶
- Bool val
size_graphemes¶
Number of extended grapheme clusters in this Text. O(1) on an indexed Text (cached count); O(n) otherwise (walks the bytes).
Returns¶
- USize val
size_codepoints¶
Number of Unicode codepoints in this Text. O(1) on an indexed Text; O(n) otherwise (walks the UTF-8 bytes counting lead bytes).
Returns¶
- USize val
is_indexed¶
True iff this Text carries the optional bitmap index. Use
with_index() / without_index() to obtain a copy with the
index flipped.
Returns¶
- Bool val
with_index¶
Return a copy of this Text carrying the optional bitmap index.
The byte buffer is cloned and the index is rebuilt. Partial only
because from_string is partial — the bytes here are guaranteed
valid so the error path is unreachable.
Returns¶
- Text val ?
without_index¶
Return an index-free copy of this Text. The byte buffer is
cloned; partial for the same reason as with_index().
Returns¶
- Text val ?
byte_index¶
Construct a ByteIndex pointing at byte position n (or the
one-past-end position size_bytes()). Returns OutOfRange for
values larger than size_bytes().
Parameters¶
- n: USize val
Returns¶
- (ByteIndex | OutOfRange val)
codepoint_index¶
Construct a CodepointIndex pointing at the n-th codepoint (or
the one-past-end position). Range-checked against
size_codepoints() — O(n) until M4d's indexed Text caches the
count.
Parameters¶
- n: USize val
Returns¶
- (CodepointIndex | OutOfRange val)
grapheme_index¶
Construct a GraphemeIndex pointing at the n-th grapheme cluster
(or the one-past-end position). Range-checked against
size_graphemes().
Parameters¶
- n: USize val
Returns¶
- (GraphemeIndex | OutOfRange val)
byte_at¶
The raw UTF-8 byte at position i. Returns OutOfRange if i
is at or past the end.
Parameters¶
- i: ByteIndex
Returns¶
- (U8 val | OutOfRange val)
codepoint_at¶
The Codepoint val at position i. Walks the UTF-8 bytes from
the start; O(n) until M4d adds an indexed fast path.
Parameters¶
Returns¶
- (Codepoint val | OutOfRange val)
grapheme_at¶
The grapheme cluster at position i, returned as a String val
slice of the underlying UTF-8 buffer (zero byte-copy). Walks the
bytes; O(n) until M4d. Receiver is val because the returned
slice shares the val view of _utf8.
Parameters¶
Returns¶
- (String val | OutOfRange val)
slice_codepoints¶
A new Text containing codepoints in the half-open range
[start, end_idx). The result is unindexed; chain with_index()
if you need O(1) sizes on the slice.
Returns OutOfRange if start > end_idx or end_idx exceeds
size_codepoints(). An empty range yields an empty Text.
fun val slice_codepoints(
start: CodepointIndex,
end_idx: CodepointIndex)
: (Text val | OutOfRange val)
Parameters¶
- start: CodepointIndex
- end_idx: CodepointIndex
Returns¶
- (Text val | OutOfRange val)
slice_graphemes¶
A new Text containing grapheme clusters in the half-open range
[start, end_idx). The result is unindexed.
Returns OutOfRange if start > end_idx or end_idx exceeds
size_graphemes(). An empty range yields an empty Text.
fun val slice_graphemes(
start: GraphemeIndex,
end_idx: GraphemeIndex)
: (Text val | OutOfRange val)
Parameters¶
- start: GraphemeIndex
- end_idx: GraphemeIndex
Returns¶
- (Text val | OutOfRange val)
codepoint_byte_index¶
The ByteIndex of the first UTF-8 byte of codepoint c. If c
equals size_codepoints(), returns the one-past-end ByteIndex.
Parameters¶
Returns¶
- (ByteIndex | OutOfRange val)
grapheme_byte_index¶
The ByteIndex of the first UTF-8 byte of grapheme cluster g.
If g equals size_graphemes(), returns the one-past-end
ByteIndex.
Parameters¶
Returns¶
- (ByteIndex | OutOfRange val)