Strings and Paths

Rust's String/str maintains valid UTF-8; operating system paths, however, are not necessarily representable as UTF-8. You must first determine whether the data represents text, raw bytes, or a system path to choose types and operations that do not lose information.

Ownership and Borrowing are the First Axis

&str can borrow a portion of a String; it does not only point to string literals, nor does it automatically have a 'static lifetime. String's safe APIs maintain the UTF-8 invariant; when constructed or modified using unsafe, the caller must continue to maintain it. The compiler does not automatically verify all memory operations. String

Bytes, char, and Visible Characters are Different

fn main() {
    let text = "e\u{301}中";
    assert_eq!(text.len(), 6);
    assert_eq!(text.chars().count(), 3);
    assert_eq!(text.char_indices().collect::<Vec<_>>(),
               [(0, 'e'), (1, '\u{301}'), (3, '中')]);
    assert_eq!(text.get(1..3), Some("\u{301}"));
    assert_eq!(text.get(2..3), None); // 2 is in the middle of the UTF-8 encoding
}

len counts bytes, while chars iterates over Unicode scalar values. The term "code point" in Unicode also includes values like surrogates that cannot be represented as Rust char; the concept of "scalar value" should be used for precise discussion.

A character perceived by the user may consist of multiple scalar values, and emoji sequences do not necessarily consist of a single 4-byte scalar. Tasks such as editor cursor movement or truncating user nicknames usually require grapheme cluster segmentation rules; display column width is yet another issue. Unicode Text Segmentation

Indexing Requires Selecting a Unit First

Rust does not provide a text[n] interface that returns the "n-th character," because this term might refer to bytes, scalar values, or grapheme clusters, each with different costs and semantics.

• Read the n-th byte: text.as_bytes().get(n).
• Read the n-th scalar value: text.chars().nth(n), which usually requires scanning preceding encodings.
• Borrow text by byte range: text.get(a..b), requiring the range and UTF-8 boundaries to be valid.
• For grapheme clusters: Use a library implementing the corresponding Unicode segmentation rules, and be aware of its Unicode version.

Direct indexing &text[a..b] will panic if the range or boundary is invalid; get uses Option to express failure. Repeatedly calling chars().nth(i) in a loop may result in quadratic complexity due to repeated scanning; it is usually better to complete the work in a single chars or char_indices iteration.

Valid UTF-8 does not mean the text is already Unicode normalized: pre-composed é and e plus combining accent are typically not considered equal in standard string equality comparisons. The normalization strategy required for searching or identity matching should be explicitly decided by the business logic.

Finding Valid Boundaries Forward When Byte Budget is Limited

fn prefix_bytes(text: &str, budget: usize) -> &str {
    let mut end = budget.min(text.len());
    while !text.is_char_boundary(end) {
        end -= 1;
    }
    &text[..end]
}
fn main() {
    assert_eq!(prefix_bytes("中ab", 2), "");
    assert_eq!(prefix_bytes("中ab", 4), "中a");
    assert_eq!(prefix_bytes("中ab", 100), "中ab");
    assert_eq!(prefix_bytes("e\u{301}", 1), "e");
}

This satisfies the contract of "not exceeding the specified UTF-8 byte count and resulting in valid text." The last assertion also exposes its boundary: it can cut off the combining accent from the letter. This might be suitable for byte limits on network fields, but for user-visible nicknames, truncation points should be decided according to grapheme cluster rules. The return value is a borrow of the original text, with no new String created; consequently, the source string cannot be arbitrarily modified while the borrow is still in use.

Text Equality Does Not Automatically Perform Normalization

fn main() {
    let composed = "\u{e9}";
    let decomposed = "e\u{301}";
    assert_ne!(composed, decomposed);
    assert_eq!(composed.len(), 2);
    assert_eq!(decomposed.len(), 3);
    assert_eq!("a\nb\n".lines().collect::<Vec<_>>(), ["a", "b"]);
    assert_eq!("a\nb\n".split('\n').collect::<Vec<_>>(), ["a", "b", ""]);
}

Choosing between lines and split also depends on the protocol: the former handles line semantics, while the latter preserves trailing empty segments based on the delimiter character. When empty fields need to be preserved or exact raw newlines are required, do not arbitrarily replace one with the other for the sake of brevity. Normalization, case handling, and whitespace stripping are also data interpretation strategies that should be defined at the input boundary, preserving the original representation needed for display or auditing.

UTF-8 boundaries differ from visible-character boundaries

Byte-budget truncation asks two questions: preserve valid UTF-8, then decide whether splitting a user-visible character is acceptable. Budgets 2 or 3 cut inside 中; budget 5 is valid UTF-8 but removes e’s accent. These boundaries need different policies.

Converting Bytes to Text: Clarify Whether Loss is Allowed

String::from_utf8 checks the owned byte vector and returns the original bytes in the error on failure; str::from_utf8 checks borrowed slices. from_utf8_lossy displays invalid sequences using replacement characters, which is suitable for some logs but not for protocols or filenames requiring lossless round-tripping.

String does not guarantee a trailing C-style NUL byte. Use CString/CStr for interoperability with C strings, and consider internal NULs and external buffer lifetimes. See C ABI and bindgen.

Distinguish Between Missing Subsequent Bytes and Definitively Invalid Encodings

fn main() {
    let mut bytes = vec![b'A', 0xe4, 0xb8];
    let error = std::str::from_utf8(&bytes).unwrap_err();
    assert_eq!(error.valid_up_to(), 1);
    assert_eq!(error.error_len(), None);
    bytes.extend([0xad, b'B']);
    assert_eq!(std::str::from_utf8(&bytes), Ok("A中B"));

    let bad = vec![b'A', 0xff, b'B'];
    let error = String::from_utf8(bad).unwrap_err();
    assert_eq!(error.utf8_error().valid_up_to(), 1);
    assert_eq!(error.utf8_error().error_len(), Some(1));
    assert_eq!(error.into_bytes(), [b'A', 0xff, b'B']);
}

When reading the first chunk, the trailing E4 B8 might just be the character '中' spanning chunks; it cannot be immediately discarded or replaced. If the protocol specifies a UTF-8 stream, you can output the confirmed prefix, retain the incomplete tail, and continue checking with the next chunk; if the message or stream has ended, you must report a truncation or execute a specific replacement strategy. Applying from_utf8_lossy to each read chunk directly would turn originally valid cross-chunk characters into replacement characters. Utf8Error

This example stores all bytes in a Vec to demonstrate boundaries. A real streaming implementation must also limit the cumulative buffer size and avoid re-validating the long confirmed prefix from scratch on every append.

OsStr Does Not Promise a Specific Persistable Internal Encoding

Unix filename components can contain non-UTF-8 bytes but cannot contain NUL, and slashes are path separators. Windows' system wide-character interface uses 16-bit encoding units, which may contain content that cannot be treated as valid Unicode scalar sequences; the Windows filesystem native format cannot simply be called WTF-8.

Rust's internal Windows representation has an implementation relationship with WTF-8, but the cross-platform OsStr abstraction does not allow applications to rely on a specific internal encoding as a persistence protocol. On Unix, you can use OsStrExt::as_bytes or similar platform interfaces for lossless access; on Windows, you can use encode_wide to get wide character units. OsStr, Windows OsStrExt

#[cfg(unix)]
fn main() {
    use std::ffi::OsString;
    use std::os::unix::ffi::OsStringExt;
    use std::path::PathBuf;
    let path = PathBuf::from(OsString::from_vec(vec![b'f', 0xff]));
    assert!(path.to_str().is_none());
    assert_eq!(path.into_os_string().into_vec(), [b'f', 0xff]);
}
#[cfg(not(unix))]
fn main() {}

Path::display and to_string_lossy are suitable for human reading and should not be treated as the original path identity. For file operations, try to pass &Path/PathBuf throughout to avoid UTF-8 conversion round-trips.

Path Operations Do Not Automatically Access or Isolate the Filesystem

join, parent, extension, components, etc., primarily handle platform path syntax. If the right-hand side of join is an absolute path, it replaces the preceding base path; Windows prefixes and root paths also have platform-specific rules. Path

use std::path::Path;
fn main() {
    let path = Path::new("reports").join("2026").join("summary.txt");
    assert_eq!(path.file_name().unwrap(), "summary.txt");
    assert_eq!(path.extension().unwrap(), "txt");
}

components does not arbitrarily fold a/../b because symbolic links might change the meaning. canonicalize accesses the filesystem and resolves the path, but the result does not prove that the directory and links haven't changed during subsequent access.

Therefore, PathBuf::join is the correct tool for expressing path concatenation, but it is not a "safe boundary that guarantees user input always stays within a certain directory." If the task requires limiting access to a root directory, you must use appropriate directory handles and resolution constraints according to the target system; a simple string prefix check is insufficient.

Path Prefixes are Compared by Component, Still Not Representing Filesystem Containment

use std::path::{Component, Path};
fn normal_relative(path: &Path) -> bool {
    !path.as_os_str().is_empty()
        && path.components().all(|c| matches!(c, Component::Normal(_)))
}
fn main() {
    assert!(normal_relative(Path::new("reports/summary.txt")));
    assert!(!normal_relative(Path::new("../outside")));
    assert!(!normal_relative(Path::new("")));
    assert!(!Path::new("reports-old/a").starts_with("reports"));
    assert!(Path::new("reports/../outside").starts_with("reports"));
    assert!(!normal_relative(Path::new("reports/../outside")));
}

The function only expresses a lexical input strategy: non-empty, and all parsed components are Normal. components performs limited syntactic normalization; original spelling information like repeated separators may not be preserved; different platforms use their own separators and prefix rules. It does not check if the path exists, nor can it prevent symbolic links within reports from pointing outside.

Path::starts_with is more aligned with component semantics than string prefixes, so reports-old is not considered a sub-path of reports; but paths with .. can still pass this component prefix check. Even if .. is rejected, symbolic links and concurrent directory changes still need to be handled during the actual opening phase; normal_relative cannot be treated as a complete file access isolation solution. Path Components and Prefixes

Separate Display, Storage, and Operation Interfaces

When file APIs receive paths, prioritize having callers pass AsRef<Path> or &Path, and perform path operations internally; if you force receiving a String, you exclude some valid system paths at the entry point. Conversely, resource names in network protocols that look like paths should first be interpreted according to the protocol specifications; do not directly apply the current OS's Path rules.