DSL Design

Crafting refined vocabulary and syntax for a specific domain—from internal DSLs (extensions of the host syntax with free type checking) to external DSLs (independent parser + codegen). The choice depends on whether domain concepts can be expressed naturally in the host language.

Overview

General-Purpose Languages (GPLs) can write anything, but expressing certain domains in a GPL creates significant noise. A Domain-Specific Language (DSL) crafts a refined vocabulary and syntax for a specific domain—SQL for data manipulation, regular expressions for text matching, Terraform for describing infrastructure. For compilation techniques, there are two fundamentally different implementation paths for DSLs: Internal DSLs (extensions of the host language's syntax, no new parser by the compiler) and External DSLs (independent syntax + parser + code generation). This article covers the mechanisms of both paths, criteria for selection, and the effort required for the compiler frontend when designing an external DSL from scratch.

Internal DSLs: Extensions of the Host Language

Internal DSLs parasitize the host language—leveraging the host language's syntactic features (operator overloading, chained calls, macros, trailing closures) to make code read like a specially designed language. There is no new parser; compilation is handled by the host compiler.

Fluent API (Chained Calls)

// A fluent DSL for building SQL queries
let results = query::select(&[name, age])
    .from(users)
    .where_(
        condition::eq(department, "Engineering")
            .and(condition::gt(salary, 50000)))
    .order_by(name, Desc)
    .execute(db)?;

Mechanism: Each method (select, from, where_, order_by) returns a QueryBuilder, carrying accumulated AST fragments; .execute() translates the AST into an SQL string and executes it. The host compiler performs type checking at compile time—if the field type in where_ does not match the value type, the host compiler reports an error. A core advantage of internal DSLs: type checking is free.

Operator Overloading DSL

Leverage operator overloading to make expressions look more mathematical:

// C++ Eigen (Linear Algebra DSL)
MatrixXd A = B * C + D.transpose();
VectorXd x = A.lu().solve(b);

Under the hood, each */+/.transpose() is a C++ function call, but it does not perform actual matrix multiplication—it returns an expression template, which is a compile-time AST. When the assignment x = ... occurs, the expression template is expanded into optimal loop code (no temporary matrix allocation at compile time). Zero runtime overhead—this is the key distinction between operator overloading DSLs and "writing an interpreter".

Macro DSL

Rust's procedural macros allow translating custom syntax into the host language's AST at compile time:

#[derive(Serialize)]          ← DSL: Declares serialization rules
struct User {
    name: String,
    age: u32,
}

#[derive(Serialize)] expands at compile time into impl Serialize for User, generating field-by-field serialization code for the serialize() method. serde and diesel are two representative examples of Rust macro DSLs—serde's DSL translates struct definitions into serialization logic, while diesel's DSL maps schema definitions to database tables at compile time.

Advantages of Macro DSLs: Syntax is fully expanded at compile time—incorrect DSL usage (e.g., a field type that is not serializable) results in a compile-time error rather than a runtime error.

External DSLs: Independent Language + Full Compilation Pipeline

When a domain's syntax cannot be embedded in the host language (e.g., regular expressions, SQL, Terraform HCL), an external DSL is required—it has its own lexical/syntactic/semantic rules and compiles into a target representation (bytecode, machine code, or translated back to the host language).

External DSL Compilation Pipeline

Source Code (DSL Syntax) → Lexical Analysis → Syntax Analysis (typically Pratt/LR) → Semantic Analysis → Code Generation
                                         ↓
                                  AST (DSL-specific nodes)

It is exactly the same as a general compiler, except:

• The language is smaller (dozens of grammar rules rather than hundreds).
• The target is usually not machine code, but source code in another language (transpilation) or bytecode.
• Error messages are the user interface of this DSL, more important than error messages in general compilers—users working in a specific domain should receive error messages using domain terminology.

Code Generation Targets: Transpile vs. Compile

• Transpile: DSL source code → Source code in another language. Terraform HCL → Internal plan JSON; Protobuf .proto → .go/.rs/.py. Advantages: Output is readable and debuggable; downstream compilers perform optimizations.
• Compile: DSL source code → IR → Machine code/Bytecode. Terra (a mix of Lua and low-level languages) takes this path—certain parts of the DSL are JIT-compiled to machine code. Advantages: Best performance; Cost: Complex implementation (requires IR → backend).

The choice mainly depends on "whether the DSL users prioritize performance or debuggability."

Selecting an External DSL: When is it Worth Starting a New Language?

Ask three questions:

1. Can domain vocabulary be expressed naturally in the host syntax? SQL's SELECT...FROM...WHERE can be made close to natural using a fluent API in a general language, but regular expressions like ^a.*b$ cannot be made close to natural in general syntax.
2. Is the usage frequency sufficient? A configuration file format used a few times does not justify writing a parser; thousands of queries within a system do.
3. Do error messages require domain knowledge? "Unmatched parenthesis at position 15" from a regex compiler and "SyntaxError: unexpected token" from a general compiler are not of the same quality of experience—DSLs need to understand domain concepts to report good errors.

Criterion: If a concept "always requires comments to explain intent" in the host language, it is a signal for a DSL.

Progressive Evolution: From Internal DSL to External DSL

Many successful DSLs start as internal DSLs. When the limitations of the host syntax become unacceptable, an external parser is added:

Phase 1: Fluent API + String Templates                    (Internal DSL)
Phase 2: Macro DSL (Compile-time AST construction + Host type checking)   (Internal DSL, Compile-time)
Phase 3: External Parser + Transpilation to Host Language               (External DSL, Transpilation)

Serde evolved from Phase 1 (manual Serialize impl) to Phase 2 (procedural macro #[derive(Serialize)]), and currently stopping at Phase 2 is entirely sufficient. SQL stops at Phase 1–2 in most ORMs, but the SQL parser inside database engines is Phase 3.

References

• Martin Fowler: "Domain-Specific Languages" — DSL design patterns and selection guide
• Holden: "Build Your Own Lisp" — Writing a Lisp DSL interpreter from scratch, including parser + codegen
• Serde: https://serde.rs — The quintessential example of a Rust macro DSL
• Terra: https://terralang.org — Representative of low-level DSLs (JIT compilation)

Keywords: DSL, domain-specific language, internal DSL, external DSL, fluent API, builder pattern, operator overloading DSL, expression template, macro DSL, proc macro, transpile, code generation, embedded DSL, language design, Serde, diesel