So, in Rust, dyn Trait means $\exists s. \text{Trait}(s) \land s$ , and fn f() -> impl Trait means $f: \exists s. \text{Trait}(s) \land (() \rightarrow s)$ , but what if I want an existential over something other than the Self parameter? Like, what if I have Trait<B> and I want an $\exists s, b. \text{Trait}(s, b) \land s$ ?

That is a lot of fancy math symbols, but I hope by the end of this you will understand both what they mean and my answer to this question!

What Is An Existential Quantifier?

A hint can be found in the first part of her question:

dyn Trait means $\exists s. \text{Trait}(s) \land s$

Let's think about what it means to have a value of type dyn Trait¹. We know there is some underlying type, but we don't have a way of getting at that type. All we can do is call methods from Trait.

That's pretty much exactly what that formula says! "There exists ( $\exists$ ) some type s, such that ( $.$ ) s conforms to Trait, and ( $\land$ ) you can use s right now". The scary math symbols are just a concise way of writing that. Math also helps us formalize the fact that, given just this $\exists$ statement, we have no way of knowing what s will be ahead of time, so in constructing a "proof" that the program is well-typed, we can only assume things about s that come from it conforming to Trait².

Alice also points out another place Rust has something that looks like an existential quantifier:

fn f() -> impl Trait means $f: \exists s. \text{Trait}(s) \land (() \rightarrow s)$

This "return-position impl trait" syntax is indeed another way of specifying a type such that callers of f only know about the fact that the return type conforms to Trait. Unlike with dyn Trait, these are fixed at compile time, but type-theoretically they're somewhat similar.

So! Now that we have a bit of understanding, let's get started on the real question:

What if I have Trait<B> and I want an $\exists s,b. \text{Trait}(s, b) \land s$ ?

That is, is it possible to have an existentially quantified type S that itself has another existentially quantified type variable B, such that S: Trait<B>?

Attempt 1: For-Exists Conversion

A classic way of dealing with any existentials is something I call "for-exists conversion". Basically, it relies on the following identity:

$((\exists x.P(x)) \rightarrow Q) \Longleftrightarrow (\forall x.(P(x) \rightarrow Q))$

In Rust, this can be stated as:

fn f_left(x: impl P) -> Q {
    x.foobar()
}

fn f_right<X: P>(x: X) -> Q {
    x.foobar()
}

"But PolyWolf!" I hear you say, "You've just written the same function twice??" And that, dear reader, is exactly what's so sneaky about this identity. It seems obvious, but it's actually saying something pretty profound.

f_left really does use an existential quantifier for its argument (all you know about x is that it has a type conforming to P), while f_right first quantifies over all possible types for X, then restricts to those that satisfy the trait P. In Rust, you might think the former is "just sugar for" the latter, but that doesn't change the fact these are different mathematical statements.

To see this more clearly, let's think about what happens when we use those argument types as return types instead:

fn g1() -> impl P { SpecificX::proof() }

fn g2<X: P>() -> X { X::proof() }

Here, we can more clearly see that impl P and X: P are saying different things. The former, like $\exists$ , only requires you to demonstrate a SpecificX that satisfies the trait P, while the latter, like $\forall$ , requires you to produce valid values for all possible X: P.

So! Getting back to our original problem, we want to describe a type of the form:

$\exists s, b. \text{Trait}(s, b) \land s$ For the rest of this post, we're going to assume that $\text{Trait(s, b)}$ is some S: Generic<B> (renamed to avoid confusion w/ other traits introduced later), and that we can only build new types/traits that reference it, never modifying it or the following impls:

pub trait Generic<B> {
    fn name(&self) -> &'static str;
}
impl Generic<f32> for i32 {
    fn name(&self) -> &'static str { "f32" }
}
impl Generic<f64> for i32 {
    fn name(&self) -> &'static str { "f64" }
}

Restricting ourselves first to functions that take in such a type, we get:

$(\exists s, b. \text{Trait}(s, b) \land s) \rightarrow ()$

Then, using our identity, we can rewrite it as:

$\forall s, b. (\text{Trait}(s, b) \land s) \rightarrow ()$

This results in the following function (playground link):

fn use_generic<B, S: Generic<B>>(s: &S) -> &'static str {
    s.name()
}

Easy peasy! This is a very standard way of dealing with the problem, and imo the most clean. However, Alice was not satisfied:

You couldn't e.g. put this one in a data structure (behind a Box) though, could you?

And that's entirely true! Due to the way we rephrased the problem in terms of $\forall$ , once we return a Box<dyn Generic<B>>, we'd have to keep a $\forall b$ clause around on everything that uses it, which sort of defeats the purpose of having $\exists$ types in the first place (playground link):

fn put_in_box<B, S: Generic<B> + 'static>(s: S) -> Box<dyn Generic<B>> {
    Box::new(s)
}

fn use_box<B>(s: &Box<dyn Generic<B>>) -> &'static str {
    s.name()
}

This solution may be satisfactory for some usecases, but we'll have to try a bit harder to get things to only use $\exists$ .

Attempt 2: Associated Types

My next idea was to "smuggle" the $\exists s, b.$ quantifier behind a single $\exists t.$ quantifier that Rust gives when using trait objects. A first crack at it might look something like (playground link):

trait SingleAssociated {
    type B;
}

impl<B, S: Generic<B>> SingleAssociated for S {
    type B = B;
}

However, we quickly run into a problem:

error[E0207]: the type parameter `B` is not constrained by the impl trait, self type, or predicates
  --> src/main.rs:15:6
   |
15 | impl<B, S: Generic<B>> SingleAssociated for S {
   |      ^ unconstrained type parameter

The E0207 docs do an excellent job explaining this (you should try rustc --explain if you haven't already!), but the short version is, whenever we're implementing traits, Rust wants there to be at most one concrete trait implementation for every concrete type. This restriction is called "coherence", because it would be "incoherent" if we could choose between multiple possible trait implementations for a single type⁴. When we write impl<B, S: Generic<B>>, Rust considers all possible B and S that fit the criteria before moving onto the rest of the code. However, "all possible B" could result in more than one SingleAssociated for S implementations, so we are not allowed to do that.

To get around this, we'll have to put B in either the trait or type, to "constrain" it. We can't put it in the SingleAssociated trait definition (that would make it the same as Generic<B>!), and we can't put it in S (it's already generic), so what can we possibly do? PhantomData :)

PhantomData is a zero-sized type that lets us to implement the SingleAssociated trait for something that, at runtime, is basically the same as a normal S, just with extra type-level information:

use std::marker::PhantomData;

impl<B, S: Generic<B>> SingleAssociated for (S, PhantomData<B>) {
    type B = B;
}

fn mk_existential<B, S: Generic<B>>(s: S) -> impl SingleAssociated {
    (s, PhantomData)
}

Great, mission accomplished, right? Except wait... how do we actually use it?? (playground link)

fn use_existential(_s: &impl SingleAssociated) -> &'static str {
    todo!()
}

The whole point of our existential type is to carry a proof about how we can do stuff with the Generic trait. But on its own, the SingleAssociated trait doesn't let us get at anything that implements Generic<B>. We know that it is only implemented for our special tuple type, but Rust doesn't, so we have to help it along by adding a second associated type + a function to get at it:

trait DoubleAssociated {
    type S: Generic<Self::B>;
    type B;
    fn as_ref(&self) -> &Self::S;
}

impl<S, B> DoubleAssociated for (S, PhantomData<B>)
where
    S: Generic<B>,
{
    type S = S;
    type B = B;
    fn as_ref(&self) -> &Self::S { &self.0 }
}

This lets us use functions from Generic<B>⁵ (playground link):

fn mk_existential<B, S: Generic<B>>(s: S) -> impl DoubleAssociated {
    (s, PhantomData)
}

fn use_existential(s: &impl DoubleAssociated) -> &'static str {
    s.as_ref().name()
}

Great! If we were satisfied with using $\exists$ types as returned from functions, we could stop here.

...However, to satisfy Alice's request, we'd also like to be able to "erase" that type by putting it in a Box<dyn Trait>, giving it a consistent representation; otherwise, two functions that both return impl DoubleAssociated could actually be returning different types, and e.g. couldn't be put inside the same Vec. We should able to do Box<dyn DoubleAssociated>, so let's try that, shall we? (playground link):

fn mk_box<B: 'static, S: Generic<B> + 'static>(s: S) -> Box<dyn DoubleAssociated<S=S, B=B>> {
    Box::new((s, PhantomData))
}

fn use_box(s: &Box<dyn DoubleAssociated>) -> &'static str {
    DoubleAssociated::as_ref(Box::as_ref(s)).name()
}

Whuh oh

error[E0191]: the value of the associated types `S` and `B` in `DoubleAssociated` must be specified
  --> src/main.rs:44:24
   |
14 |     type S: Generic<Self::B>;
   |     ------------------------ `S` defined here
15 |     type B;
   |     ------ `B` defined here
...
44 | fn use_box(s: &Box<dyn DoubleAssociated>) -> &'static str {
   |                        ^^^^^^^^^^^^^^^^
   |
help: specify the associated types
   |
44 | fn use_box(s: &Box<dyn DoubleAssociated<S = /* Type */, B = /* Type */>>) -> &'static str {
   |                                        ++++++++++++++++++++++++++++++++

It wants us to use $\forall$ again... to be honest, I totally did not expect this. I thought the fact that Rust enforces "at most one concrete trait implementation per concrete type" meant associated types could be stored in the trait object vtable somehow, but my intuition was way wrong on that.

Thinking on it further, the fact that we return things type Associated::S means that, at runtime, different DoubleAssociated impls will necessarily have different vtables, because return types will have different layouts, so the trait objects with different associated types MUST be different. Rust 1 – Me 0

So, what can we take away from this? That it's impossible to fully automatically erase multiple existential bounds in Rust in the same type, because it only supports one at a time with dyn Trait? Unfortunately yes, that is my takeaway here :(

Still, if we're willing to put in a little bit more legwork, it is possible to manually collapse multiple existential bounds into one.

Attempt 3: Manual Erasing

Forget trying to use functions from Generic<B> directly, let's instead create a trait that manually, for each function we care about, passes thru to the function from the trait:

trait Erased {
    fn name(&self) -> &'static str;
}

impl<B, S: Generic<B>> Erased for (S, PhantomData<B>) {
    fn name(&self) -> &'static str {
        self.0.name()
    }
}

This works!! We can even box it, hooray (playground link):

fn mk_box<B: 'static, S: Generic<B> + 'static>(s: S) -> Box<dyn Erased> {
    Box::new((s, PhantomData))
}

fn use_box(b: &Box<dyn Erased>) -> &'static str {
    b.name()
}

Alice has one last request for us here:

hmmm but now this one doesn't work if the trait actually uses the B parameter right? (because you can't use it in Erased)

And that is true: Erased intentionally has no way for it to vary what types it cares about, everything needs to be concrete so it can be put inside a dyn without issues. If we want to deal with multiple types, we'll have to use std::any::Any to handle them at runtime. Here's an example (playground link):

use std::any::Any;
use std::marker::PhantomData;

trait Generic<B> {
    fn input(b: &B);
    fn output(&self) -> B;
}
impl Generic<f32> for i32 {
    fn input(b: &f32) { println!("f32: {b}") }
    fn output(&self) -> f32 { *self as f32 }
}
impl Generic<f64> for i32 {
    fn input(b: &f64) { println!("f64: {b}") }
    fn output(&self) -> f64 { *self as f64 }
}

trait Erased {
    fn input(&self, b: &Box<dyn Any>);
    fn output(&self) -> Box<dyn Any>;
}

impl<B: 'static, S: Generic<B>> Erased for (S, PhantomData<B>) {
    fn input(&self, b: &Box<dyn Any>) {
        S::input(b.downcast_ref().unwrap())
    }
    fn output(&self) -> Box<dyn Any> {
        Box::new(self.0.output())
    }
}

This compiles & runs successfully! Unfortunately, as you can see, manual erasure has a few drawbacks besides just dyn-compatibility.

This process, as I've outlined above, is very manual: you have to rewrite every function signature to include Box<dyn Any> instead of B, and make a decision for each about whether to .unwrap() or propagate the error up somehow. If there are enough functions to do this conversion on, you could write a macro_rules! to define a Generic<B> and Erased trait at the same time, but that seems very finnicky and I have been nerd-sniped enough already so I will not attempt that here :P

Anyways that's it, thanks for reading, hope you learned a little something in the process!

Problem Statement

Given:

• A finite set of structs/enums (henceforth referred to as "items"), possibly generic

Create:

• A way to, for each item in the set, recursively transform any other item, including inner generic parameters, if possible.

Let's take a look at a simple example:

struct ListNode<T> {
    val: Val<T>,
    next: Option<Box<ListNode<T>>>
}
struct Val<T>(T);

Then, given a ListNode<T>, we'd like to be able to turn it into a ListNode<U>, just by specifying how to turn Val<T> into Val<U>.

Approach 1: Write a traversal manually

impl<T> ListNode<T> {
    fn transform<U>(self, f: impl FnMut(Val<T>) -> Val<U>) -> ListNode<U> {
        ListNode {
            val: f(self.val),
            next: self.next.map(|n| Box::new((*n).transform(f))),
        }
    }
}

Great! Nice & simple. However, manually mapping through the Option and Box is a bit tedious, especially if we want to write other, similar transforms. Let's see if we can solve that.

Approach 2: Functor time!!

In my last post on this subject, I covered this crazy wacky trait I called Functor, invented specifically to do these sorts of transforms. Here's the trait definition¹:

trait Functor<Output> {
    type Input;
    type Mapped;
    fn fmap(self, f: impl FnMut(Self::Input) -> Output) -> Self::Mapped;
}

And here's how it's used (playground link), with doc comments explaining what each part means:

/// There are 4 types involved in any Functor implementation: {Input, Output} x {Inner, Container}.
/// What a Functor implementation means is, given a function InputInner -> OutputInner, you can write a function InputContainer -> OutputContainer.
/// Writing the full name of each type is a bit hard, so we have shorthand:
/// + InputInner = Input (Val<T>)
/// + OutputInner = Output (Val<U>)
/// + InputContainer = Self (ListNode<T>)
/// + OutputContainer = Mapped (ListNode<U>)
impl<T, U> Functor<Val<U>> for ListNode<T> {
    type Input = Val<T>;
    type Mapped = ListNode<U>;
    fn fmap(self, mut f: impl FnMut(Val<T>) -> Val<U>) -> ListNode<U> {
        ListNode {
            val: f(self.val),
            next: self.next.fmap(&mut f),
        }
    }
}
/// Given all types T that can be mapped over, allow Option<T> to be mapped over in the same way, automatically un/re-wrapping the Some case as required
impl<T, Output> Functor<Output> for Option<T>
where
    T: Functor<Output>
{
	/// Input is the type given as input to the internal transform. In this case, because we are transparent, we use T's input
    type Input = T::Input;
    /// Mapped is the external type after the transform has been done. Because we are wrapping T as input, we also wrap T's mapped type as output.
    type Mapped = Option<T::Mapped>;
    fn fmap(
        self, //: Option<T>
        f: impl FnMut(T::Input) -> Output,
    ) -> Option<T::Mapped> {
        Option::map(self, |x| x.fmap(f))
    }
}

/// The implementation for Box<T> is very similar to that for Option<T>, included for completeness.
impl<T, Output> Functor<Output> for Box<T>
where
    T: Functor<Output>,
{
    type Input = T::Input;
    type Mapped = Box<T::Mapped>;
    fn fmap(
        self,
        f: impl FnMut(Self::Input) -> Output,
    ) -> Box<T::Mapped> {
        Box::new((*self).fmap(f))
    }
}

Neat! This way, if we're writing a lot of transforms, they'll be composable with existing container types like Option & Box, as well as any other container types we might try.

However, I'm still a bit sad about having to write these transforms by hand. Especially given the fact that, while Functor is nice to implement for wrapper types like Option and Box, it isn't as nice for "AST-like" types...

Problems With AST-like Types

Consider what would happen if we had the following set of items:

struct A {
    b: B,
}
struct B {
    c: C,
}
struct C(i32);

Naturally, we'd want to call a.fmap(|c: C| ...). But! There is no item C in any direct fields of A, only indirectly thru b: B. If we only cared about this one fmap call, we could write:

impl Functor<C> for A {
	type Input = C;
	type Mapped = A;
	fn fmap(self, f: impl FnMut(C) -> C) -> A {
		A { b: self.fmap(f) }
	}
}
impl Functor<C> for B {
	type Input = C;
	type Mapped = B;
	fn fmap(self, mut f: impl FnMut(C) -> C) -> B {
		B { c: f(self.c) }
	}
}

Which is fine. But, you might eventually add an item D that you also want to map over from A. Then you'd have to write:

impl Functor<D> for A { ... }
impl Functor<D> for B { ... }
impl Functor<D> for C { ... }

Apparently, the number of Functor implementations you have to write scales linearly with both the depth of the tree and the number of things you want to map over, so quadratically in total, which is quite bad.

Ideally, you'd like to just scale linearly with the size of the tree. Here's one approach for this:

/// These "base impls" can all be easily generated with a macro
impl Functor<A> for A {
	type Input = A;
	type Mapped = A;
	fn fmap(self, mut f: impl FnMut(A) -> A) -> A {
		f(self)
	}
}
impl Functor<B> for B { ... }
impl Functor<C> for C { ... }

/// These "recursive impl"s are written manually, but that's OK, we only need one per item!
impl<T> Functor<T> for A where B: Functor<T, Mapped=B> {
    type Input = <B as Functor<T>>::Input;
    type Mapped = A;
    fn fmap(self, f: impl FnMut(Self::Input) -> T) -> A {
        Self {
            b: self.b.fmap(f),
        }
    }
}
impl<T> Functor<T> for B where C: Functor<T, Mapped=C> { ... }

Seasoned Rust veterans may be able to spot the problem: impl<A> Functor<A> for A and impl<T> Functor<T> for A conflict! Never mind that where clause, the Rust compiler is sometimes very particular about preventing future conflicts too², just in case B: Functor<A, Mapped=B> somehow.

If only there were a way to automatically write all those quadratic dumb-but-non-conflicting implementations... then it wouldn't matter that there's so many...³

Approach 3: Use A proc_macro

Turns out, there's enough information in the item definitions alone to generate the traversals automatically. All we really need is:

1. A list of all items considered a part of the AST
2. The type of each field in each item
3. A way of determining valid transforms to generate for each item
4. A way of generating each valid transform

We need (1) because of the problem presented before: it's not possible to generate Functor impls for arbitrary types, so we must explicitly choose the ones we want to map over.

(2) and (4) are also fairly straightforward given Rust's de-facto introspection libraries, syn & quote⁴. There are some subtleties when it comes to actually extracting the information we want (the Rust type grammar is so much...), but overall "get the type of a field" and "get the shape of an item" is not too hard. Same with generating the transform; once you have the information of "what type am I transforming" and "what fields should I transform for that type", emitting the code is fairly straightforward. You can take a look at my code if you're curious, but I promise it's really boring.

(3), on the other hand, is much more difficult + interesting, and is a big reason I kept delaying this piece. Turns out re-creating the Rust trait implementation coherence semantics is pretty involved!!

What Even Is A Valid Transformation?

Let's start out with the most basic criteria. Any valid implementation of Functor<B> for A comes about because:

1. A has a field with type B, where "with type" looks inside any wrapper types like Option/Box, OR
2. A has a field with type C, where Functor<B> for C is a valid transform.

We get the information for (1) when reading in the items initially. To get (2), we must turn to the horror that is graph algorithms in Rust. Get ready for some math!

1: The Lattice Step

I call this a "lattice" because it encodes a "higher-than" relationship that's useful for what I'm doing. It's not a true lattice by many definitions of the word, but im the programmer so i get to name things badly ok

What this step looks like is, taking all edges matching Functor<C> for A (denoted as A -> C) + C -> B, and collecting them into a new edge A -> B. This is so we can get all the possible implementations that would be covered by case (2) above. There are a few subtleties when it comes to unifying generic contexts, but like the mathematician I'm cosplaying as, I'll leave those for the appendix⁵.

However! We're not done yet. Creating all these new edges gives us all the possible implementations, but not all the valid ones. To do this, we need to filter out all invalid edges, so let's look at some ways an implementation (as defined by an edge) can be invalid.

2: Filter Out Conflicting Generic Instantiations

Consider the following:

struct A1<T, U> {
    bt: B<T>,
    bu: B<U>,
}
struct B<T>(T);

If we try list out all the possible implementations, we get:

impl<T, U> Functor<B<T>> for A1<T, U> { ... }
impl<T, U> Functor<B<U>> for A1<T, U> { ... }

Note we have two implementations, because B<T> and B<U> are different types. But clearly, these implementations conflict! This is because Functor<B<T>> and Functor<B<U>> are not different traits. Without any preference for which implementation to keep, we discard both.

There is a similar conflict if we instead have:

struct A2<T> {
    bt: B<T>,
    bi: B<i32>,
}

Because Functor<B<T>> could be the same trait as Functor<B<i32>>, and without a "type inequality" bound (not possible to express in Rust iirc), we can't generate these. In this case, we just prefer the Functor<B<i32>> implementation that only transforms bi, because without generic parameters, implementations can never conflict.

The algorithm for deciding if a pair of instantiations conflict is as follows:

1. For each pair of arguments in order:
   1. If both are generic, they must be the same generic. Otherwise, there's a conflict.
   2. If only one is generic, there's a conflict
   3. If neither are generic, there's no conflict.

3: Restrict Which Generic Parameters Are Transformed

This is a bit of a case of me being burned by making my Functor too complex, but screw it I did want that complexity. The situation is, if you have something like:

struct A<T> {
    b: B<T>,
    c: C<T>,
}
struct B<T> {
    c: C<T>,
}
struct C<T>(T);

And we try to write the "full" implementation for A -> B that transforms its generic parameter:

impl<T, U> Functor<B<U>> for A<T> {
    type Input = B<T>;
    type Mapped = A<U>;
    fn fmap(self, f: impl FnMut(B<T>) -> B<U>) -> A<U> {
        A {
            b: f(self.b),
            c: self.c, // ERROR: expected C<U>, got C<T>
        }
    }
}

Oh no type error!! The reason it happens is because we can't transform the generic type of c even though we need to. There are a couple parts to this:

1. How do we know we need to transform c?
   • Because the generic context used by field b, and therefore the generic context transformed by b, is also used by field c.
2. How do we know we cannot transform c?
   • We look at all the outgoing edges from item C, and if there are none that would satisfy the transformation C -> B (for the same instantiated item B as is being considered for field b), then field c cannot be transformed.

Anyways, this is all just to check if we can transform the generic parameter T to U. If we can't, no worries, we can just generate a Functor implementation that doesn't transform T :) Like any sane person would do :)

What Is The Correct Order For These Steps?

If you've been paying attention (no worries if not, it's a long one, you're almost there!), you may have noticed something: step 3 above look at outgoing edges for all nodes one level "below" the current. However, step 2 explicitly removes some edges from the graph. So! We must be smart about the order in which we look at nodes for step 3, so we don't end up keeping an edge A -> B only to later remove the edge C -> B that made keeping it possible.

I claim the correct traversal order is "reverse topological sort order". (Whoopee, more graph algorithms in Rust!) Basically, we should start with nodes that don't point to anything (the "bottom"), then nodes that only point to previously explored nodes, and so on. This ensures we don't end up in that bad scenario.

However! The topological sort order is only defined for directed graphs with no loops. Our graph can have loops, and indeed, that's a big feature for this macro in general, being able to support trees & linked lists & such, which all have edges that look like A -> A or even A -> B -> C -> A. Are we out of luck?

Not yet! By first finding the strongly connected components (omg more graph algorithms!!), we can do a topological sort on those, which are guaranteed to not have loops between them, and our traversal order will be good.

"But what about the order within a strongly connected component?", you may ask. Well, thanks to our lattice step earlier, every loop A -> B -> C -> A in the graph will become a clique, so if A, B, and C are in the same strongly connected component, A -> B & C -> B will always exist. Wow it's almost like I'm actually doing math!!

So, the correct order of steps is:

1. Lattice step
2. Filter out conflicting instantiations
3. Topological sort
4. Restrict generic parameters

And that's all there is to it! ;)

Conclusion

At long last, we have reached the end of everything I know about automatically deriving a Functor implementation. These rules have worked out for me in practice, for two very large ASTs. There are more things I know, like deriving TryFunctor and Foldable and Visit, but those all use these same basic principles too. There are also many more things I do not know, like "what if a field contains a generic type but doesn't have one of the designated items?" or "how can I prevent mapping invalid field types, like fn(T) -> U" or "am I using panic! to handle errors too often", but those are all left for later/never. I do not recommend anyone else attempt to use this code btw, it is simply a dangerous toy for my own amusement, and it will blow up in ur face.

Anyways, thanks for reading, see u next time!! :3

macro_rules! node {
    (
        $name:ident [ $subnode:ident ]
    ) => {
        pub struct $name(Vec<$subnode>);
    };
    (
        $name:ident ( $(
            * $field:ident : $ty:ty
        )+ )
    ) => {
        pub struct $name { $(
            $field: $ty,
        )+ }
    };
    (
        $name:ident ( $(
            + $ty:ident
        )+ )
    ) => {
        pub enum $name { $(
            $ty($ty),
        )+ }
    };
}

Then, what you might want to do next is generate all these all in one macro invocation, so you don't have to write node!() over and over again. You might think, "darn, looks like I'll have to write all these exact branches again in order to capture them in an outer macro". But fear not, weary traveler! For I bear great news:

Neat Trick 1: Use tt to capture arguments that you pass to other macro invocations.

macro_rules! nodes {
    ( $(
        $name:ident $tt:tt ;
    )* ) => {
        $(
            node!($name $tt);
        )*
    }
}

tt, or TokenTree, represents a pair of braces (round (), square [], or curly {})², with anything inside them. Rust is really good about enforcing braces to be in pairs even at the lex level, and this is represented in macros, which do just take in an arbitrary list of tokens.

So now we have a macro that we can use like this:

mod ast {
    nodes!(
        Program[Function];
        Function(
            *open_brace: OpenBrace
            *body: Body
            *close_brace: CloseBrace
        );
        Body[BlockItem];
        BlockItem( +Statement +Declaration );
        // ...
    );
}

But what if that's not good enough? What if we need more?

More?

To keep things short: in my compiler project, I have a proc_macro that wants to read in a whole bunch of struct/enum definitions at once, by being an attribute on a module. However! There is a dilemma: the simplest approach doesn't work.

#[my_proc_macro]
mod ast {
    nodes!(/* ... */);
}

This is because of the order in which Rust runs macros. #[my_proc_macro] runs first, because it could transform the output to be literally anything (see the #[cfg] attribute macros, which wholesale remove the thing they're applied to). What is this means, however, is that when #[my_proc_macro] runs, all it sees is a module with a single macro invocation, which it doesn't like, so it moves on without doing any work :(. Only then does nodes!() run and generate all the struct/enum definitions, too late.

Our goal is to have #[my_proc_macro] operate on a module with expanded struct/enum definitions. Knowing how Rust evaluates macros, we have two options:

1. Have #[my_proc_macro] eagerly expand all macro invocations inside it.
2. Have nodes!() generate everything in one shot, applying #[my_proc_macro] to the finished product.

(1) is not possible unless we're The Rust Compiler/willing to resort to dirty hacks, so we're left with (2).

Generating Everything In One Pass

Neat Trick 2: Inside a repetition $()*, you can use $()? like individual macro_rules! cases, and the results will be the same, so long as you use an appropriate delimiter³.

macro_rules! nodes_prime {
    (
        $(
            $name:ident
            $(
                // Case 1
                [ $a_subnode:ident ]
            )?
            $(
                // Case 2
                ( $(
                    * $b_field:ident : $b_ty:ty
                )+ )
            )?
            $(
                // Case 3
                ( $(
                    + $c_ty:ident
                )+ )
            )?
            // Delimiter
            ;
        )*
    ) => {
        #[my_proc_macro]
        mod ast {
            use super::*;

        $(
            $(
                // Case 1
                pub struct $name(Vec<$a_subnode>);
            )?
            $(
                // Case 2
                pub struct $name { $(
                    $b_field: $b_ty,
                )+ }
            )?
            $(
                // Case 3
                pub enum $name { $(
                    $c_ty($c_ty),
                )+ }
            )?
        )*

        }
    };
}

Whoa! That's a big macro, huh? Hopefully each part makes sense given the previous two macros.

How this trick works is, whenever Rust encounters a series of $()?, it checks the next token to see which branch matches. Assuming exactly one does, it takes that branch and emits that case within the repetition. Neat! However, there are some downsides:

1. The criteria for making $()? cases distinct is stricter than making macro_rules! cases distinct, due to the eager one-token lookahead.
2. You must make the variable names in each case distinct, which either leads to visual clutter (prefixes) or confusion (no prefixes).
3. The compiler will yell at you real good if you mess up at either of the above.

Still, at least for me, the benefit of generating everything in one shot & being compatible with #[my_proc_macro] easily outweighs the cost. I hope to have a blog post soon about what exactly I'm doing in #[my_proc_macro] that makes it so cool (whoops I hinted at such a post months ago already, oh well). Anyways, take care, see you next time!!

trait Functor<Inner> {
    type Input;
    type Output;
    type Mapped;
    fn fmap(self, f: &mut impl FnMut(Self::Input) -> Self::Output) -> Self::Mapped;
}

I called it a "specialized Functor" because I didn't quite understand what exactly a Functor was, but I knew that, at the very least, what I had was a little different.

Upon reading this, Alice of welltypedwit.ch, who actually knows what she's talking about when it comes to functional programming, pointed me towards a Haskell package called uniplate, noting that what I was doing sounded like its transformBi operation.

To fully understand what she meant by this, let's break down the type signature:

transformBi

In Haskell:

transformBi :: Biplate from to => (to -> to) -> from -> from

What is this saying? Even if you understand Haskell, it can sometimes still be tricky to get what certain operations mean. In this case, I think about it like:

Given a type From, which is a container type like the root of an AST, and a type To, which is like an inner node of an AST, transform From by applying an operation each time we encounter a node of type To in the tree.

Indeed, that is very similar to what I was trying to do previously! So why did I call my trait Functor, when this description almost exactly matches? Well mostly out of ignorance, yes, but also, when I later looked up the equivalent Rust crate uniplate, its type signature looked something more like¹:

trait Biplate<To> {
    fn transform_bi(self, op: fn(To) -> To) -> Self;
}

Quite different from what I ended up with! Note that this is homogenous in the types it maps over; it transforms ASTs to be ones of the same type, whereas I was interested in transforming ASTs to have different types, because the type of the AST can enforce specific invariants. So that's why I was, and still am, interested in something like fmap instead:

fmap

In Haskell:

fmap :: Functor f => (a -> b) -> f a -> f b

I think about this like:

Given a container type F over homogenous elements A, you can transform each A into a B in order to get a new container F over homogenous elements B.

And that's why I thought I wanted a Functor! Initially, I had only parameterized my AST on how it represented locations in memory. I had one type (A) for HardwareRegister | StackAddress | PseudoRegister, and another (B) for just HardwareRegister | StackAddress, in order to guarantee I could never emit assembly code that contained pseudo-registers. A regalloc² pass would get rid of all the pseudo-registers, and I would run that pass with a specially-crafted fmap call.

I was inspired by how the Rust crate fmap translated the Haskell signature³:

trait Functor<B> {
    type Inner;  // a.k.a. A
    type Mapped; // a.k.a. F<B>
    fn fmap(self, op: &mut impl FnMut(Self::Inner) -> B) -> Self::Mapped;
}

Now you can hopefully understand how I ended up where I did! My version of Functor is exactly this, just with an explict associated parameter for B so writing generic implementations is easier⁴.

Going beyond?

While playing around with this Functor trait, I realized something interesting. Unlike Haskell's Functor typeclass, where f is necessarily a type with a generic parameter, and fmap necessarily transforms that generic parameter, Rust's Functor is a lot more like Biplate, in that you can specialize the transform to only be over certain inner elements!

When I realized this, I started trying to shoehorn everything into it. I didn't have to just write one impl<L: Location> Functor<L> for Program<L> implementation, I could write multiple⁵!

struct Location<L>(L); // Needs to be a newtype so we can avoid potentially conflicting implementations
impl<Input, Output> Functor<Location<Output>> for Program<Input> {
    type Input = Location<Input>;
    type Output = Location<Output>;
    type Mapped = Program<Output>;
    fn fmap(self, f: &mut impl FnMut(Self::Input) -> Self::Output) -> Self::Mapped { ... }
}
impl<I, O> Functor<Expression<O>> for Program<I> { ... }
impl<I, O> Functor<Statement<O>> for Program<I> { ... }

The restrictions of Rust's trait system, and needing to specify explicit Input/Output/Mapped types, actually makes Functor more flexible than its Haskell counterpart! I'm using it to create a class of functions that look like:

fmapBi :: Triplate from to => (to a -> to b) -> from a -> from b

and even this, despite being a pretty neat extension of both Functor and Biplate in Haskell, still doesn't capture the full generality of the Rust trait. Still, for my purposes, this "best of both worlds" is all I need: the ability to transform only specific inner nodes (from Biplate), and the ability to change the parameterization of the container type (from Functor).

I'd like to write a paper on this. The authors of uniplate wrote one explaining their approach, as did the author of multiplate (which I don't quite understand fully, TODO read it i guess). I'm not 100% sure I have a novel idea on my hands, but at the very least, I think it's a new useful way of looking at things that I'll be happy to include in my toy compiler.

In case you're wondering, "PolyWolf, why has it taken you so long to get this post out? Surely you haven't spent over two weeks just thinking?", yes I have been thinking about this for a while, but mostly in the context of writing a proper Rust proc_macro that creates these implementations for me, just given the AST definition. It's unfortunately much much more involved than writing a macro_rules!⁶, especially trying to handle all the edge-cases properly. I don't know why I waited to publish this blog post until I finished writing this new macro; I didn't even need to explain anything about it in this post lol. Leaving that for a future blog post, maybe once I publish the crate perhaps !

As always, you can check out the code at github:p0lyw0lf/pwcc; it sort of has documentation right now, if u squint

My solution for Rust was to write a "specialized Functor" trait like so¹:

trait Functor<Inner> {
    type Input;
    type Output;
    type Mapped;
    fn fmap(self, f: &mut impl FnMut(Self::Input) -> Self::Output) -> Self::Mapped;
}

// example of how it's implemented:
impl<T, Inner, A, B> Functor<Inner> for Vec<T>
where
    T: Functor<Inner, Input=A, Output=B>,
{
    type Input = A;
    type Output = B;
    type Mapped = Vec<T::Mapped>;
    fn fmap(self, f: &mut impl FnMut(Self::Input) -> Self::Output) -> Self::Mapped {
        self.into_iter().map(&mut |x: T| x.fmap(f)).collect()
    }
}

I call this a "specialized Functor" because, unlike the Haskell version, it allows for implementations on arbitrary container types (e.x. parent AST nodes), for specific inner types (e.x. child AST nodes). To give an extremely simplified example:

struct Function {
    name: String,
    body: Vec<Statement>,
}

enum Statement {
    Return { val: isize },
}

A hand-written Functor implementation for isize would look like

impl Functor<isize> for isize {
    type Input = isize;
    type Output = isize;
    type Mapped = isize;
    fn fmap(self, f: &mut impl FnMut(Self::Input) -> Self::Output) -> Self::Mapped {
        f(self)
    }
}

impl Functor<isize> for Statement {
    type Input = isize;
    type Output = isize;
    type Mapped = Statement;
    fn fmap(self, f: &mut impl FnMut(Self::Input) -> Self::Output) -> Self::Mapped {
        match self {
            Statement::Return { val } => Statement::Return {
                val: val.fmap(f),
            },
        }
    }
}

impl Functor<isize> for Program {
    type Input = isize;
    type Output = isize;
    type Mapped = Program;
    fn fmap(self, f: &mut impl FnMut(Self::Input) -> Self::Output) -> Self::Mapped {
        Program {
            name: self.name,
            body: self.body.fmap(f),
        }
    }
}

#[test]
fn map_isize() {
    let x = Program {
        name: "main".into(),
        body: [Statement::Return { val: 1 }].into(),
    };
    // Pretend we derived Clone
    let y = Functor::<isize>::fmap(x.clone(), |i| i + 1);
    // Pretend we derived PartialEq
    assert_eq!(y, Program {
        name: "main".into(),
        body: [Statement::Return { val: 2 }].into(),
    });
}

That is, we re-construct the value in-place, recursively calling fmap to transform the appropriate fields².

Now, because the entire point of this is for me to be really lazy, I did not want to have to write out all these Functor implementations manually. So naturally, I wrote a macro³ that does most of the work for me. All I have to do now is specify, for a given inner type, for each AST node above it in the tree, what fields I'd like to map over.

Unfortunately, that's still too much work!!! I have a lot of AST nodes and inner types I want to map over, so what I'd really like is a way to define something like this:

func fmap<Container, Inner>(over c: Container, f: (Inner) -> Inner) -> Container {
  // somehow `fmap` over all of `c`'s fields
  return c;
}

func fmap<Inner>(over v: Inner, f (Inner) -> Inner) -> Inner {
  return f(v);
}

Basically: if we're at the target node type, just call the function (definition 2), otherwise, recursively call into all fields (definition 1). Easy as pie!

Wait a minute... that syntax highlighting... yep, this "pseudocode" is in fact, valid Swift! It even does exactly we want!!! See?

print(fmap(1, { $0 + 1 })); // 2
print(fmap("hello", { $0 + 1 })); // "hello"
print(fmap("hello", { $0 + "_world" })); // "hello_world"

This kind of definition is impossible to do in Rust thanks to a little thing called Coherence, a.k.a. No Specialization. What Specialization allows us to do is write down those 2 definitions for fmap, and the compiler will automatically choose the "least generic" one according to some rules.

Great! Now all I have to do is implement some compile-time introspection to get it to map over the appropriate fields. How hard could that be?

[2 hours of trying to read Swift documentation and finding mostly AI-generated blog posts. Thank goodness for StackOverflow at least]

Upon looking further, it seems the built-in read-only reflection with Mirror only happens at runtime, and the Runtime read-write reflection library is same way (aptly-named). Also, there doesn't seem to be a way to dynamically call fmap based on the runtime type; instead, all recursive calls result in an Any type for Container, since that's all that can be resolved statically. By passing in types as arguments, I can get all the way down to "cannot convert value of type 'any Any.Type' to expected argument type '(any Any).Type'", which really illustrates the predicament.

I might be able to hack something with the dynamic keyword or like Protocols or whatever, but I'm still not pleased about runtime-only reflection. I am willing to pay the slight extra binary cost for all these definitions, and Swift doesn't let me do that. Unfortunate that this was a dead end, but still pretty fun to learn something new!

Think I'll try Haskell next? :P