and maybe some other incidental details too...

We can almost make out the badge just in that one shot, but later shots make it much more explicit:

It's a Nikon F100, a classic 35mm SLR film camera. Mystery solved that was easy!!

If I were really serious about things, I would try to find out all their models of bikes, but I am a lowly camera geek so I'll leave that for others.

You should read Yotsuba&! btw. It is very good and I have been learning a lot by struggling thru the Japanese originals. At the start I was doing maybe 2 pages every 30min and now I am up to 20 pages in that time.

But enough about that. Really, this post is a low-effort one intended to answer "why have I not meaningfully posted for over a month". In order of relevance:

1. I've acquired a medical condition that is forcing me off estrogen for at least another 2 months, which sucks major ass and has been an acute reminder of why I went on in the first place.
2. I have a few drafts in various states of completion, but I haven't had enough time/energy due to my new job. It is a fine job but unemployment will always be better for blogging turns out.
3. Other hobbies like "learning FL studio" & "hanging out with friends" have eaten into the time I would have otherwise used to blog.

There likely isn't anyone I am disappointing here except myself, but to everyone who's subscribed to this over RSS, thank you for reading.

It's mostly been small things:

• no monospace font on main pages (just art page now), everywhere looks more consistent
• header links so you can link to a specific section of a blog post
• minor styling change: superscripts don't cause big line gaps

But there's a few bigger ones too:

• blogroll
• button wall
• by-year archives

Of these, I am quite proud of the buttons I made for the button wall. However, it currently doesn't have anyone else's buttons. If you have your own 88x31 button, please message me! I'd love to expand my social web even further :3

• We want to associate some data with an async task in Rust, such that, while that task is being polled, we can retrieve said data deeper in the callstack, without having to manually pass the data down thru function arguments.
• We don't want to use the tokio ecosystem.

That is, we want to do what tokio::task_local! does, but don't want to use tokio. What do? Well... I couldn't find anything else that does it, so I did it myself (:

Buckle Up It's Apple-Pie-Baking Time

To understand what we want, we must first understand why what the standard library gives is not enough. To understand what the standard library gives, we must first understand the execution model upon which it relies. I did say "from scratch"¹, after all :P

Most operating systems organize code execution into two distinct units: Processes & Threads. Processes roughly delineate "what memory is visible", while Threads roughly delineate "what instructions are running". A single Process may contain one or more Threads, and when there are multiple Threads in a Process, computer scientists call it a "Multithreaded Architecture" with "Shared Memory Parallelism": there are literally multiple Threads, sharing memory in the same Process.

This Process/Thread model requires a bit of work on the programmer's side to make sure Threads don't step on each other's toes while they're running. Most normal code assumes it's the only one with access to the memory it touches, and will get very upset if some other Thread scribbles over its work while it's not looking. If Threads don't touch each other's memory, or there is some mechanism by which they co-ordinate their mutual touching², that property is called "Thread-Safety".

Naturally, thread-safety is a very important property (ensuring it at compile time is a big motivator for Rust!), so we'll need to maintain it while achieving our overall goal. There are various strategies by which we can ensure thread-safety, but by far the simplest is "just use variables on the stack lol". Each Thread's stack is unique by construction³, so if we create a value on the stack + pass a reference down thru arguments to anywhere that needs it, boom done ezpz.

But remember, we have a special requirement:

[...] can retrieve said data deeper in the callstack, without having to manually pass the data down thru arguments.

So really, we want something that works more like a "global" variable, but without the complications that come from multiple Threads sharing a single global. Fortunately, this already exists! Thread-Local Storage, aka TLS, creates variables that are "local" in the sense they can only be accessed by a single thread, while still being "global" in the sense that any function can touch them. Different platforms can have greatly varying implementations of TLS, all we need to know is: each thread has its own place to put its global variables, no more toe-stepping yay everyone is happy. C/C++ uses the thread_local keyword for this, while Rust uses a std::thread_local! macro.

Are Thread-Locals Sufficient?

Turns out no! There's a bit more universe we need to conjure before we can start on our pie, specifically the difference between an Operating System Thread and a Rust Task.

With Threads, the OS automatically manages things like "what Thread is running on this CPU core?", "what Threads are available to run but aren't?", & "ok time to switch to different Thread". However, switching between Threads at the OS level is quite a slow operation⁴ compared to normal code. Instead, languages will often make their own thread-like things in "userspace" (that is, without any help from the OS) to help avoid this slowdow while keeping the other benefits of concurrent code. These thread-like things are called either Green Threads or Coroutines, depending on certain factors.

For now, I'll just group them all together under an umbrella labeled "Tasks", upsetting the largest number of people possible :)

Anyways! The important thing to notice is, Tasks are not Threads, so we can't only use Thread-Local Storage, because:

1. We could have multiple Tasks running on the same Thread, OR
2. We could have a Task that gets run on multiple Threads at different points in its execution (for throughput reasons).

Specifically, a plan to "set a thread-local when a Task starts running, and unset it when the Task finishes running" doesn't work, because:

1. Other Tasks that start running on the same Thread might share that same thread-local, OR
2. The Task might get shuffled to another Thread between it setting a thread-local and it trying to read from the thread-local.

both of which are failure modes that will make us sad :(

So! With all that out of the way, I hope I've convinced you we really do need a special construct for an Task-Local. Now to build it...

What's Our Strategy Cap'n?

How Rust's Tasks work is, they only ever do work when their Future::poll() method is called⁵. That, finally, leads to our key insight: in order to have a Task-Local value, we simply need to set some global value only while poll() is being called. Using a Thread-Local value as that global works because, for the duration of that single poll() call, we can't change threads. So we're free to save/restore on that single thread as we please:

use std::cell::RefCell;
use std::pin::Pin;
use std::task::{Context, Poll};
use std::thread::LocalKey;

/// Container type that implements [`std::future::Future`] by wrapping an existing one, as well as saving/restoring a task-local made available to it.
pub struct Scoped<F: Future, T: 'static> {
    /// Thread-Local Storage that holds the current task-local value.
    tls: &'static LocalKey<RefCell<Option<T>>>,
    /// When the future IS NOT being polled: the value we want to store is inside `curr`.
    /// When the future IS being polled: the previous value stored inside `curr`, if we are running inside another [`Scoped`].
    curr: Option<T>,
    /// The future we're wrapping.
    fut: F,
}

impl<F: Future, T: 'static> Future for Scoped<F, T> {
    type Output = F::Output;
    fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> {
        // SAFETY: `this` never moves out its value, so `self` stays pinned.
        let this = unsafe { &mut self.get_unchecked_mut() };
        // SAFETY: because `self` is still pinned, so is `fut`.
        let fut = unsafe { Pin::new_unchecked(&mut this.fut) };
        // The swap only ever moves values _stored inside_ `self`;
        // it doesn't change the location of `self` directly.
        let curr = &mut this.curr;
        let mut swap = || this.tls.with_borrow_mut(|prev| std::mem::swap(curr, prev));

        // Swap in the value from "the stack" (`this.curr`) into "global memory" (`this.tls`) while polling.
        swap();
        let out = fut.poll(cx);
        // Swap the value from "global memory" back onto "the stack" to save it for when we get polled later.
        swap();

        out
    }
}

I hope all that exposition helped understand why we're doing what we're doing, that we didn't just pull this implementation out of thin air, why we're putting save/restore logic in poll() instead of in a constructor/Drop impl.

Now all that's left is to provide some helpers so this wrapper has a more tokio-like API. First, we define a helper struct to provide scope() & with() functions that are very similar to what tokio::task::LocalKey has:

pub struct ScopeBuilder<T: 'static> {
    tls: &'static LocalKey<RefCell<Option<T>>>,
}
impl<T: 'static> ScopeBuilder<T> {
    /// Create a new task wrapper builder for the given Thread-Local Storage
    pub const fn new(tls: &'static LocalKey<RefCell<Option<T>>>) -> Self {
        Self { tls }
    }

    /// Given the Thread-Local Storage provided at builder creation, construct a [`Scoped`] future that will expose the given `value`.
    pub fn scope<F: Future>(&self, value: T, fut: F) -> Scoped<F, T> {
        Scoped {
            tls: self.tls,
            curr: Some(value),
            fut,
        }
    }

    /// Read from the Thread-Local Storage, which will be `Some` if we are inside a [`Scoped`] future.
    /// Takes a callback to ensure the lifetimes work out.
    pub fn with<V>(&self, f: impl FnOnce(Option<&T>) -> V) -> V {
        self.tls.with_borrow(|value| f(value.as_ref()))
    }
}

Then, for completeness's sake, we also define a macro similar to tokio::task_local!, which creates the Thread-Local Storage & associated ScopeBuilder at the same time in an ergonomic fashion:

#[macro_export]
macro_rules! task_local {
    ($(static $ident:ident : $ty:ty ;)*) => {$(
        static $ident: $crate::ScopeBuilder<$ty> = {
            std::thread_local! {
                static LOCAL: std::cell::RefCell<Option<$ty>> = const { std::cell::RefCell::new(None) };
            }
            $crate::ScopeBuilder::new(&LOCAL)
        };
    )*}
}

Putting it all together, usage looks something like:

task_local! {
    static SOME_GLOBAL: usize;
}

/// Prints the value currently contained in [`SOME_GLOBAL`]
fn print() {
    let v = SOME_GLOBAL.with(|v| v.map(|v| *v));
    match v {
       Some(v) => println!("{v}"),
       None => println!("None"),
    }
}

#[tokio::main]
async fn main() {
    // Define a stack of wrapped tasks, showing how the task-local is saved/restored
    print();    
    tokio::spawn(SOME_GLOBAL.scope(5138008, async {
        print();
        tokio::spawn(SOME_GLOBAL.scope(69, async {
            print();
            tokio::spawn(SOME_GLOBAL.scope(42, async {
                print();
            })).await.unwrap();
            print();
        })).await.unwrap();
        print();
    })).await.unwrap();
    print();
}

Note the tokio::spawn() calls around each .scope() call (to properly test what happens when there are multiple Tasks running concurrently) but NOT around each async {} call (to ensure the print() calls happen in the same Task as their surrounding .scope()).

Trying this out at the Rust playground, we observe the following output:

None
5138008
69
42
69
5138008
None

just like we drew it up :)

Conclusion

In this post, I've shown a complete(!!) way of writing Task-Locals in Rust, using only the stdlib, that covers the "multiple Tasks on one Thread" case (via swapping from stack to Thread-Local), as well as the "single Task on multiple Threads" case; std::thread::LocalKey stands for "how we access the current Thread's TLS", so the resulting Scoped future can, in fact, be sent to other threads, where it will simply use the TLS of whatever Thread it starts running on.

The source code in this post is directly copied from part of a larger collection of crates I'm making, and while I might eventually consider publishing it to crates.io, it's small enough to vendor in any project if you really want it.

Anyways that's all, hope you enjoyed & maybe even learned something! Until next time!!

So this is a manga about racecars that go very fast. And it's a good enough first chapter¹ (yay he wins wahoo) but I am more interested in the side character that appears for 2 pages taking photos of the race, because he has a cool camera!!

Panel 1

Panel 2

My goal for today is to find out exactly what gear he's using.

The Lens

From these photos, we observe the lens MUST have the following traits:

1. Is Very Long
2. Has the little thing that sticks out on the bottom for mounting
3. Outermost cone flares out a bit
4. Comes in a white-colored variant

(4) is obvious, and turns out the rest map cleanly to filters too:

1. We're looking for a "Telephoto"- or "SuperTelephoto"-class lens
2. We're looking for a lens with a "foot mount"
3. We're looking for a "Prime" lens; all "Zoom" lenses tend to be perfect cylinders to make the zoom optics easier

Browsing online stores like B&H Photo that sell All Lenses, we come across two likely candiates: the Canon RF 600mm F4 L IS USM² & the Sony FE 600mm f/4 GM OSS Lens³:

	Front View	Side View
Canon
Sony

Given what we can observe, I claim the Canon lines up better:

1. The front is more uniformly conical.
2. Hard to see, but the side knob is behind the button group instead of in front.

Also, we should probably be sure this is actually the 600mm Canon; there are various other telephoto Canon lenses of varying sizes that it could be right? Well, let's take a look at the other 3:

	Short Cone	Long Cone
Short Mount	400mm	600mm
Long Mount	800mm	1200mm

Looking at all these, it seems the 400mm actually fits the observed button layout in the second panel the best: the "long mount" lenses (with those built-in zoom doublers) are much too long when compared to the second manga panel, and the 600mm also has an extra ring, so only the 400mm is short enough to include all the buttons that are shown. I guess the perspective in the first shot is fake, or there is some other non-straight hood I don't know about?

Anyways all's well that ends well, the Canon RF 400mm f/2.8L IS USM is a near-exact match for what's shown in the manga.

...

"But wait!!!" I hear you yell, "Haven't you forgotten about the Canon EF 400mm f/2.8L IS III USM lens, which looks almost identical?" And yes, I have considered that, but in order to explain why, we'll need to look at the camera body too.

The Body

Given we almost certainly have a Canon lens, we're also very likely to have a Canon camera body. Compared to previous camera searches, this one is a lot easier because of one key feature on the camera: it is tall, much taller than an ordinary Canon camera, meaning it is top-of-line professional-grade stuff. (As expected of a press sports photographer lol.)

Of the available Canon bodies that exist, the following are potential matches:

	EOS R1	EOS R3	EOS-1D X Mark III
Front Profile
Side Profile

And of these, the EOS R1 side profile visually fits best, especially the dimensions of the viewfinder. This choice makes sense in-context too, because at the time of writing, the EOS R1 is the "best" Canon camera money can buy, for photography at least⁴.

So, because the camera is a Canon EOS R1, which is a mirrorless camera, we can be very confident he is using a mirrorless lens for it too, closing our earlier argument.

Bonus: The Camera On His Hip

You can hardly even see it, but in the first image Daisuke also has a camera strapped around his body lying on his hip. You can't make out much except the top panel, but I claim that's enough for us!!

The reasoning is, we can make out the following features:

1. Information display is rectangular
2. Dial is on the right (when looking at the lens)
3. (Assumption) It's another Canon, based on boot shape + being in the Canon ecosystem

Just these two features alone are enough to rule out all of Canon's mirrorless offerings. Those all have more square-ish information screens, and all the high-end ones have the dial on the left, taking the EOS R5 as an example:

Instead, I think this is a Canon EOS 5D Mark IV, the last & best DSLR Canon made, whose top does match pretty well:

I think it's that one instead of another one because, given the rest of his gear, no way a press photography would settle for anything other than the best.

Double Bonus: Conspiracy Theory

Either Canon has a massive mindshare among the Japanese people (unlikely, I've seen plenty of Nikon & Sony at press conferences n such), or there is some sort of agreement with Canon that allows them to use their cameras in manga...

What "The Backrooms" means to Millennials vs Zoomers is very different; while Millennials view those liminal spaces as once-bustling & having lost their purpose, Zoomers view them as always having been deserted. In the former interpretation, the creepyness comes from the fact that they're deserted & inherently wrong-looking while empty. In the latter, the creepyness comes from the fact that these spaces were built at all despite "not having a purpose". Therefore, it makes more sense to put a spooky monster if you're going for the latter interpretation; in the former, it just ruins the idea of the space being empty.

(not an exact quote my any means, just my recollection)

This movie does lend itself more toward the latter interpretation (being originally conceived & now directed by a Zoomer will do that), but I think it understands the former perspective as well. There are motivated reasons for why the spooky monster is in there that fit in with the overarching theme of "Backrooms is a physical reflection of society's unconscious remembering/forgetting". What it means for such a place to exist still eludes me, even after watching the movie, but the concepts r juicy & fun to munch on so i dont mind :)

Also mad shoutouts to the 3D design team, the myriad VFX teams, & the rotoscoping teams. Making this entirely inside Blender is no easy feat but they pulled it off flawlessly, immaculate vibes/aesthetic, as should be expected from a product that hinges so heavily on aesthetic to convey meaning. Oh and also the sound team, good use of subsonics without being too overbearing w/ it.

Overall there was a lot of tension but I had a great time, scary enough 4 me 2 get my stomach excited but not too scary that i cant watch. 4/5 would highly recommend if you liked the youtube videos

Also also terrible idea to go into One Of The Most Backroom-y subway stations immediately afterwards... 😰

So, in Rust, dyn Trait means $\exists s. \text{Trait}(s) \land s$ , and fn f() -> impl Trait means $f: \exists s. \text{Trait}(s) \land (() \rightarrow s)$ , but what if I want an existential over something other than the Self parameter? Like, what if I have Trait<B> and I want an $\exists s, b. \text{Trait}(s, b) \land s$ ?

That is a lot of fancy math symbols, but I hope by the end of this you will understand both what they mean and my answer to this question!

What Is An Existential Quantifier?

A hint can be found in the first part of her question:

dyn Trait means $\exists s. \text{Trait}(s) \land s$

Let's think about what it means to have a value of type dyn Trait¹. We know there is some underlying type, but we don't have a way of getting at that type. All we can do is call methods from Trait.

That's pretty much exactly what that formula says! "There exists ( $\exists$ ) some type s, such that ( $.$ ) s conforms to Trait, and ( $\land$ ) you can use s right now". The scary math symbols are just a concise way of writing that. Math also helps us formalize the fact that, given just this $\exists$ statement, we have no way of knowing what s will be ahead of time, so in constructing a "proof" that the program is well-typed, we can only assume things about s that come from it conforming to Trait².

Alice also points out another place Rust has something that looks like an existential quantifier:

fn f() -> impl Trait means $f: \exists s. \text{Trait}(s) \land (() \rightarrow s)$

This "return-position impl trait" syntax is indeed another way of specifying a type such that callers of f only know about the fact that the return type conforms to Trait. Unlike with dyn Trait, these are fixed at compile time, but type-theoretically they're somewhat similar.

So! Now that we have a bit of understanding, let's get started on the real question:

What if I have Trait<B> and I want an $\exists s,b. \text{Trait}(s, b) \land s$ ?

That is, is it possible to have an existentially quantified type S that itself has another existentially quantified type variable B, such that S: Trait<B>?

Attempt 1: For-Exists Conversion

A classic way of dealing with any existentials is something I call "for-exists conversion". Basically, it relies on the following identity:

$((\exists x.P(x)) \rightarrow Q) \Longleftrightarrow (\forall x.(P(x) \rightarrow Q))$

In Rust, this can be stated as:

fn f_left(x: impl P) -> Q {
    x.foobar()
}

fn f_right<X: P>(x: X) -> Q {
    x.foobar()
}

"But PolyWolf!" I hear you say, "You've just written the same function twice??" And that, dear reader, is exactly what's so sneaky about this identity. It seems obvious, but it's actually saying something pretty profound.

f_left really does use an existential quantifier for its argument (all you know about x is that it has a type conforming to P), while f_right first quantifies over all possible types for X, then restricts to those that satisfy the trait P. In Rust, you might think the former is "just sugar for" the latter, but that doesn't change the fact these are different mathematical statements.

To see this more clearly, let's think about what happens when we use those argument types as return types instead:

fn g1() -> impl P { SpecificX::proof() }

fn g2<X: P>() -> X { X::proof() }

Here, we can more clearly see that impl P and X: P are saying different things. The former, like $\exists$ , only requires you to demonstrate a SpecificX that satisfies the trait P, while the latter, like $\forall$ , requires you to produce valid values for all possible X: P.

So! Getting back to our original problem, we want to describe a type of the form:

$\exists s, b. \text{Trait}(s, b) \land s$ For the rest of this post, we're going to assume that $\text{Trait(s, b)}$ is some S: Generic<B> (renamed to avoid confusion w/ other traits introduced later), and that we can only build new types/traits that reference it, never modifying it or the following impls:

pub trait Generic<B> {
    fn name(&self) -> &'static str;
}
impl Generic<f32> for i32 {
    fn name(&self) -> &'static str { "f32" }
}
impl Generic<f64> for i32 {
    fn name(&self) -> &'static str { "f64" }
}

Restricting ourselves first to functions that take in such a type, we get:

$(\exists s, b. \text{Trait}(s, b) \land s) \rightarrow ()$

Then, using our identity, we can rewrite it as:

$\forall s, b. (\text{Trait}(s, b) \land s) \rightarrow ()$

This results in the following function (playground link):

fn use_generic<B, S: Generic<B>>(s: &S) -> &'static str {
    s.name()
}

Easy peasy! This is a very standard way of dealing with the problem, and imo the most clean. However, Alice was not satisfied:

You couldn't e.g. put this one in a data structure (behind a Box) though, could you?

And that's entirely true! Due to the way we rephrased the problem in terms of $\forall$ , once we return a Box<dyn Generic<B>>, we'd have to keep a $\forall b$ clause around on everything that uses it, which sort of defeats the purpose of having $\exists$ types in the first place (playground link):

fn put_in_box<B, S: Generic<B> + 'static>(s: S) -> Box<dyn Generic<B>> {
    Box::new(s)
}

fn use_box<B>(s: &Box<dyn Generic<B>>) -> &'static str {
    s.name()
}

This solution may be satisfactory for some usecases, but we'll have to try a bit harder to get things to only use $\exists$ .

Attempt 2: Associated Types

My next idea was to "smuggle" the $\exists s, b.$ quantifier behind a single $\exists t.$ quantifier that Rust gives when using trait objects. A first crack at it might look something like (playground link):

trait SingleAssociated {
    type B;
}

impl<B, S: Generic<B>> SingleAssociated for S {
    type B = B;
}

However, we quickly run into a problem:

error[E0207]: the type parameter `B` is not constrained by the impl trait, self type, or predicates
  --> src/main.rs:15:6
   |
15 | impl<B, S: Generic<B>> SingleAssociated for S {
   |      ^ unconstrained type parameter

The E0207 docs do an excellent job explaining this (you should try rustc --explain if you haven't already!), but the short version is, whenever we're implementing traits, Rust wants there to be at most one concrete trait implementation for every concrete type. This restriction is called "coherence", because it would be "incoherent" if we could choose between multiple possible trait implementations for a single type⁴. When we write impl<B, S: Generic<B>>, Rust considers all possible B and S that fit the criteria before moving onto the rest of the code. However, "all possible B" could result in more than one SingleAssociated for S implementations, so we are not allowed to do that.

To get around this, we'll have to put B in either the trait or type, to "constrain" it. We can't put it in the SingleAssociated trait definition (that would make it the same as Generic<B>!), and we can't put it in S (it's already generic), so what can we possibly do? PhantomData :)

PhantomData is a zero-sized type that lets us to implement the SingleAssociated trait for something that, at runtime, is basically the same as a normal S, just with extra type-level information:

use std::marker::PhantomData;

impl<B, S: Generic<B>> SingleAssociated for (S, PhantomData<B>) {
    type B = B;
}

fn mk_existential<B, S: Generic<B>>(s: S) -> impl SingleAssociated {
    (s, PhantomData)
}

Great, mission accomplished, right? Except wait... how do we actually use it?? (playground link)

fn use_existential(_s: &impl SingleAssociated) -> &'static str {
    todo!()
}

The whole point of our existential type is to carry a proof about how we can do stuff with the Generic trait. But on its own, the SingleAssociated trait doesn't let us get at anything that implements Generic<B>. We know that it is only implemented for our special tuple type, but Rust doesn't, so we have to help it along by adding a second associated type + a function to get at it:

trait DoubleAssociated {
    type S: Generic<Self::B>;
    type B;
    fn as_ref(&self) -> &Self::S;
}

impl<S, B> DoubleAssociated for (S, PhantomData<B>)
where
    S: Generic<B>,
{
    type S = S;
    type B = B;
    fn as_ref(&self) -> &Self::S { &self.0 }
}

This lets us use functions from Generic<B>⁵ (playground link):

fn mk_existential<B, S: Generic<B>>(s: S) -> impl DoubleAssociated {
    (s, PhantomData)
}

fn use_existential(s: &impl DoubleAssociated) -> &'static str {
    s.as_ref().name()
}

Great! If we were satisfied with using $\exists$ types as returned from functions, we could stop here.

...However, to satisfy Alice's request, we'd also like to be able to "erase" that type by putting it in a Box<dyn Trait>, giving it a consistent representation; otherwise, two functions that both return impl DoubleAssociated could actually be returning different types, and e.g. couldn't be put inside the same Vec. We should able to do Box<dyn DoubleAssociated>, so let's try that, shall we? (playground link):

fn mk_box<B: 'static, S: Generic<B> + 'static>(s: S) -> Box<dyn DoubleAssociated<S=S, B=B>> {
    Box::new((s, PhantomData))
}

fn use_box(s: &Box<dyn DoubleAssociated>) -> &'static str {
    DoubleAssociated::as_ref(Box::as_ref(s)).name()
}

Whuh oh

error[E0191]: the value of the associated types `S` and `B` in `DoubleAssociated` must be specified
  --> src/main.rs:44:24
   |
14 |     type S: Generic<Self::B>;
   |     ------------------------ `S` defined here
15 |     type B;
   |     ------ `B` defined here
...
44 | fn use_box(s: &Box<dyn DoubleAssociated>) -> &'static str {
   |                        ^^^^^^^^^^^^^^^^
   |
help: specify the associated types
   |
44 | fn use_box(s: &Box<dyn DoubleAssociated<S = /* Type */, B = /* Type */>>) -> &'static str {
   |                                        ++++++++++++++++++++++++++++++++

It wants us to use $\forall$ again... to be honest, I totally did not expect this. I thought the fact that Rust enforces "at most one concrete trait implementation per concrete type" meant associated types could be stored in the trait object vtable somehow, but my intuition was way wrong on that.

Thinking on it further, the fact that we return things type Associated::S means that, at runtime, different DoubleAssociated impls will necessarily have different vtables, because return types will have different layouts, so the trait objects with different associated types MUST be different. Rust 1 – Me 0

So, what can we take away from this? That it's impossible to fully automatically erase multiple existential bounds in Rust in the same type, because it only supports one at a time with dyn Trait? Unfortunately yes, that is my takeaway here :(

Still, if we're willing to put in a little bit more legwork, it is possible to manually collapse multiple existential bounds into one.

Attempt 3: Manual Erasing

Forget trying to use functions from Generic<B> directly, let's instead create a trait that manually, for each function we care about, passes thru to the function from the trait:

trait Erased {
    fn name(&self) -> &'static str;
}

impl<B, S: Generic<B>> Erased for (S, PhantomData<B>) {
    fn name(&self) -> &'static str {
        self.0.name()
    }
}

This works!! We can even box it, hooray (playground link):

fn mk_box<B: 'static, S: Generic<B> + 'static>(s: S) -> Box<dyn Erased> {
    Box::new((s, PhantomData))
}

fn use_box(b: &Box<dyn Erased>) -> &'static str {
    b.name()
}

Alice has one last request for us here:

hmmm but now this one doesn't work if the trait actually uses the B parameter right? (because you can't use it in Erased)

And that is true: Erased intentionally has no way for it to vary what types it cares about, everything needs to be concrete so it can be put inside a dyn without issues. If we want to deal with multiple types, we'll have to use std::any::Any to handle them at runtime. Here's an example (playground link):

use std::any::Any;
use std::marker::PhantomData;

trait Generic<B> {
    fn input(b: &B);
    fn output(&self) -> B;
}
impl Generic<f32> for i32 {
    fn input(b: &f32) { println!("f32: {b}") }
    fn output(&self) -> f32 { *self as f32 }
}
impl Generic<f64> for i32 {
    fn input(b: &f64) { println!("f64: {b}") }
    fn output(&self) -> f64 { *self as f64 }
}

trait Erased {
    fn input(&self, b: &Box<dyn Any>);
    fn output(&self) -> Box<dyn Any>;
}

impl<B: 'static, S: Generic<B>> Erased for (S, PhantomData<B>) {
    fn input(&self, b: &Box<dyn Any>) {
        S::input(b.downcast_ref().unwrap())
    }
    fn output(&self) -> Box<dyn Any> {
        Box::new(self.0.output())
    }
}

This compiles & runs successfully! Unfortunately, as you can see, manual erasure has a few drawbacks besides just dyn-compatibility.

This process, as I've outlined above, is very manual: you have to rewrite every function signature to include Box<dyn Any> instead of B, and make a decision for each about whether to .unwrap() or propagate the error up somehow. If there are enough functions to do this conversion on, you could write a macro_rules! to define a Generic<B> and Erased trait at the same time, but that seems very finnicky and I have been nerd-sniped enough already so I will not attempt that here :P

Anyways that's it, thanks for reading, hope you learned a little something in the process!

wow that webp compression sure does a number huh. remember to click in on each to see the full resolution!!

For a long time, you've suspected this world is not real.

Not in the "identical trenchcoat-wearing agents teleport you behind a green film filter" sense, or a "moving movie set captures your daily life" sense, or a "timeline split placed you in the wrong universe" sense, or even a "every so often the human race is wiped out, quickly cloned from scratch from a bunker under Yellowstone, then imprinted with memories all at the same time; it happened last Thursday" sense, but something even more visceral than all those combined.

You wake up in your bed that's not real and put some not-real clothes on your not-real body then brush your teeth which aren't real before eating some breakfast which isn't all that filling because it's not real then you check the clock and you would be late for work if time was real but it's not and neither is the door and neither is the sidewalk and neither is the car that you're driving and neither are the other cars in traffic or the traffic lights or the asphalt or the sound of the air through the engine bouncing off concrete panels, vibrating through tiny gaps in a steel-rubber-glass window interface, funneling into your amplifying sound receptacles where a snail-shaped organ with delicate hairs does a mechanical frequency analysis and sends the results as electrical signals to the empty void inside you head: there's nothing there because none of that intricately-described process was real, it was all made up, just like the job you're going to, which has about the same impact as sitting around all day doing nothing, because your work isn't real, the economy isn't real, society isn't real, and you, certainly, without a shred that could possibly be doubted even if shreds & doubt were real, aren't real.

THERE IS ONLY SO

MUCH YOU CAN TAKE BEFORE YOU

ARE FORCED TO ADMIT

NOTHING IN THIS WORLD

IS REAL WAS REAL OR WILL BE

REAL FOREVERMORE

You scream and it's not real and you cry and it's not real and the entire remaining gamut of human emotions aren't available to you right now, because they're not real, unreality seeping and dripping and crawling and squirming and pushing and forcing its way into every nook and cranny and crevice and inch and foot and mile of your brain and soul, dying it a deep void nothing that would erode your eyes if you tried to look at it but you can't since your eyes, just like the mirror and the you in the mirror and the you beyond the mirror, aren't real.

Despite all of this, you must continue on in this world. Someday, you hope, with enough effort and just the right tricks, you can make all of it real. It will become real. You shall be real. All you have to do is—

In my previous site, I solved this by writing a rehype pass that would wrap all to-be-converted images in <a></a> tags pointing to the original image, but this got a little annoying because when you'd click on the image, the browser would immediately attempt to download it, and then you'd have to open an image viewer, remember to delete the image later, etc. What should happen is, if you click the image in the browser, you stay in the browser but just transition to a view of the original image at larger scale.

In fact, that's already what I do for my art page /art/. This uses the HTML popover attribute & HTML <dialog> element to allow page interactivity without requiring a single line of Javascript!! All we need to do now is, whenever detecting an ![]() markdown image, replace it with the appropriate HTML & CSS styles to get it to work. So that's what I did!

This wouldn't have been nearly as possible w/o my new blog engine btw, because the way Astro does automatic image stuff would have required me to make a lot of nasty hacks, but now those "hacks" are just part of the site itself, so I can do whatever I want with it!!

Overall I am much happier w/ the image-viewing experience now, I hope ppl make great use of it to zoom into images when I release my next set :3 (btw you should look at my previous set again too I think it is very good still)

Anyways the plot is not what we're here to talk about today I want to know what model of camera they used!!!!!

pg04

From this first close-up, we can immediately tell it's a Canon, despite the lack of an explicit branding mark. The giveaways are the shape of the boot above the lens & the hand grip, both of which are fairly unique to Canon.

Upon seeing this camera, Tenichi says "An SLR?!". Given this information, we might be tempted to starting looking at just Canon Film SLRs. However, I think that's just what normal people call non-pocket-sized cameras, so for now we must include DSLRs and mirrorless in our search going forward.

However, even looking at every camera Canon's ever made, none match this body profile exactly. The combination of boot shape, distance from boot to lens mount, and position of the top dial are all things that can be found in isolation, but never perfectly together. Still, going thru everything gives us a few more hints:

1. It is not a film camera, because those generally have wildly different (blockier) body shapes until the first EOS models, and no EOS film has that same top dial (presumably a mode-switch dial if it's digital).
2. It is not a mirrorless camera, because those never have that area between the lens and the boot (because they don't need to do anything w/ optics for their digital viewfinders).

Therefore, despite the inexact dimensions, hitting all the key details means we can be relatively confident it is a Canon DSLR. Just from this one image!! Hopefully further closeups & better references will let us narrow it down further:

pg09	pg10
pg16

Fortunately, for the rest of the chapter, the mangaka realizes trying to draw a camera by hand is really hard/overly time-consuming, so they photobash it instead. This gives us a few more good clues into what the camera actually is.

The last image 100% confirms it's a DSLR; the image preview rules out it being film (lol) and the viewfinder focus correction dial rules out it being a mirrorless. Also, the button layout + the clearer body shape/badge location in the former images are a strong indicator this is a camera in the Rebel T-series, which I because I own very similar-looking T5i. Comparing the button layout to what I have, however, we see there's an extra "DISP" button in the main cluster on Honoka's camera, as well as it missing the "MENU" & "INFO" buttons on the top left bridge (photo from https://mpb.com):

Looking at all the other T-series, those ending in "i" are also missing this button, probably because it turns on/off the screen, which you can do on those "i" cameras by just flipping the screen around. Looking at the standard T5, T6, and T7, we see that the rubber patch is different compared to the T5, and exactly the same on the T6 & T7, so we can be fairly sure it's one of those (photos from https://mpb.com).

T5	T6	T7

Just for good measure, I did in fact look at a bunch of other Canon DSLRs with similar body shapes, and indeed the T6/T7 are the only ones that match the back exactly. Unfortunately, this is where the trail runs cold. There are really are no external differences remaining between the T6 & T7; the only possible thing is the color of the zoom button labels, but we can't tell that from the manga, and everything else is better image sensor/software differences. So, without further evidence we cannot decide which one they used.

Fortunately, I do have some evidence to show, so bear with me :) On Canon's official website, we only see them (re-?)selling the EOS Rebel T7, not the T6. I think this is because it's the last entry-level DSLR in their lineup, their last one before their move to mirrorless with the EOS R100 for that market segment. There was a 5-year gap between the T7's 2017 release and the R100's 2023 release, and the T7 is still being sold officially today. Meanwhile, there was only a 2-year gap between the T6's release in 2016 and the T7's release, and the T6 was discontinued immediately after the T7 started being sold. So, just speaking statistically, this camera is much more likely to be a T7, based on when the manga is set (effectively modern-day).

Some final notes:

• The pictures on Canon's official website don't exactly match the photobashed ones, but I don't take that as an issue, the photobashing looks pretty thoro anyways.
• The lens also matches the 18-55mm EF-S zoom lens that comes with the kit, with it appearing "longer" in the pg10 picture perhaps due to it being in a zoom configuration.
• While I have been using the American market names this entire time, the Japanese name for this camera is EOS Kiss x90.

With all that out of the way, I think I can pretty confidently say: Tenichi & Honoka use a Canon EOS Kiss x90. Very appropriate for a romance manga lol!

To celebrate a "working site" milestone in my driver SSG project, I decided to take a look back at my Static Site Generators post. It got a lot of traffic at the time!! Thank you again to everyone who read & commented on it, y'all made my week <3

As one might have expected, talking about SSGs online had a "360 noscope nerdsnipe headshot" effect, so across multiple platforms I have amassed a grand collection of Static Site Generators, based on peoples' mentions/recommendations, to gaze upon at our leisure. Sure it's not as grand a collection as Awesome Static Generators ("all 148 of them!") or Jamstack ("all 377 of them!") or staticsitegenerators.net ("all 484 of them!"), but I think it's impressive nonetheless.

To avoid bias, I shall order them alphabetically, and give reviews to all:

• 11ty: Embroiled in controversy over their acquisition by Font Awesome & their recent rename to "Build Awesome", but beloved by a strong community. One of the classic Javascript SSGs.
• bai by Kayla: Seems like a custom templating program, written in Rust, which can be used as part of an SSG? neat
• Bilo by domm: Perl! Poking around the codebase it's honestly pretty readable. Very much a "tool-for-one-person" deal tho
• blogserver by Bert: Instead of just producing files, this then loads produced files as part of a Go binary, and serves directly from loaded memory that way. Looks like it uses a Make-like system called redo to efficiently produce changes, TIL about that looks cool.
• flora by Nick: What a cute Ruby DSL! Be careful not to mix it up with flower by Jynn, that one uses a Clojure DSL instead.
• her.esy.fun by Yann: Makefile + org-mode, a classic pairing
• ikiwiki by Joey: Is a wiki a blog? I guess it sort of is, considering icefox's blog is a wiki. Anyways I like the name, it is a bit of complex Perl however
• luasmith by Jared: I was thinking about using a Lua DSL for my project, but didn't because I like Javascript more. Pretty cool to see just how much can be done with just Lua, and also how many libraries there are for it!
• mk12/blog by mk12: Zig, designed for speed, includes a custom Markdown & MathML parser?? pretty awesome honestly
• md2blog by Jared (same person as before): Deno binary, "zero config" opinionated setup, another classic type of personal project. This came before luasmith.
• Nanoc by Denis: Ruby! Tho this one uses more standard HTML templates. Seems like good docs too.
• nextgen by Florian: Purpose-built generator where most all the logic is written in Rust, again highly-opinionated to site layout/features. If Javascript SSGs are classic then Rust SSGs are modern :3
• riki by Lars: An attempt at re-writing ikiwiki in Rust. I like Rust more than Perl so I want this to succeed...!!
• susam.net by Susam: Common Lisp!! Specifically, Steel Bank. Seems like this author has re-written their static site generation multiple times, where this is the latest. I tried to read it but it sure is Lisp yep.
• tumblelog by John: Both Python & Perl versions of the same program, kept in feature parity?? They say it hasn't been done before now. Interesting concept to use Markdown files as the basis for templating instead of HTML; I guess makes sense given technically it's a subset, and given the intended use of the program (microblogging).
• YOcaml: It's always nice to see OCaml being used for real code <3 Seems like a pretty extensive tutorial too!
• Zensical: An offshoot project created as part of the whole MkDocs fiasco. Was plugged to me in a comment on Mastodon, seems very corporate somehow.
• Zine by Loris: It's all Zig!! The configuration, frontmatter, & templates all being Zig-inspired is a neat touch :3

I had a lot of fun cataloging these small corners of the internet. Really makes me aware of just how many other people are out there, also doing cool stuff. Hope this was as inspiring for you as it was for me!

Other types of rebuilds that change logic affecting all pages range from 200ms to 1sec, which is totally fine in my book. Overall I'm very pleased with the speed, which is good, considering that's literally the reason I started this whole project in the first place.

In the process of getting things this fast, I had to think a lot about algorithms & data structures, which also forced me to learn a lot of new things. I have a few draft posts about that in the works, which, now that I've properly finished the coding, I will have no excuse but to work on instead. Keep an eye out for those I guess ¯\_(ツ)_/¯

This post will go over what I want, which afaik nothing on the market satisfies, and what I plan on doing about that.

I Have A Need... For Speed™️

My top priorities in a static site generator are:

1. I can port my current site to it without too much difficulty
2. It is very very fast, "🚀BLAZING🔥" even

But what exactly does "fast" mean? Many times have I been saddened by Static Site Generators not being fast in certain ways, so learning from my experiences I'd like it to be fast in all possible ways please & thank you. Notably, we don't need a full-fledged build system for this, so we'll cut corners wherever possible to focus on speed for the SSG niche.

(1) I Want Fast Clean Builds

I'm not too concerned about these, because I would rather satisfy the upcoming requirements around re-builds, which are the majority of one's interaction with an SSG. 'd still be nice to be under a second tho.

Testing my current site, the most expensive part seems to be the in-memory transformations for all the separate files; surprisingly, file I/O itself is not the bottleneck! Here's an strace:

$ strace -c node node_modules/astro/astro.js build
...
18:34:57 [build] 301 page(s) built in 12.34s
18:34:57 [build] Complete!
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ------------------
 23.41    0.091898           3     30254       472 futex
 12.57    0.049332           1     39546     14563 statx
  8.74    0.034312          25      1352           unlink
  6.76    0.026544           1     15055           read
  6.43    0.025225           0     28933     27981 readlink
  6.09    0.023891           1     12906      5956 openat
  4.63    0.018191           4      3955           write
  4.56    0.017918           2      6894           epoll_pwait
  4.19    0.016459           2      7551           close
  2.50    0.009800        2450         4           wait4
  2.31    0.009070           8      1045           io_uring_enter
  1.78    0.006971           7       953           mmap
  1.67    0.006570           6       969           munmap
  1.57    0.006181           3      1694           mprotect
  1.41    0.005520        1840         3           clone
  1.33    0.005221           0      5781      4622 access
  1.26    0.004962           4      1055           madvise
  1.24    0.004865           0      7471           getpid
  1.16    0.004536           7       609       301 rmdir
  1.14    0.004478           0      6084           setsockopt
  1.11    0.004349          13       327           brk
  1.08    0.004240           4      1029           fstat
...

The total reported build time is 12.34s, yet syscalls are only *does some quick math* 0.4s of that total! (Also, that number of syscalls seems totally unreasonable, but whatever.) My guess is having everything in Javascript accumulates performance woes from all the data shuffling, and there's not a whole lot to be done except change frameworks/languages.

=> Build MUST Be In A Compiled Language

This really is the only way we can make content transformations fast enough. Fortunately, there is a wealth of non-Javascript-based Markdown-To-HTML & HTML-To-Minified-HTML programs to choose from; comrak & minify-html both look pretty good.

However... just because the "expensive" steps are in compiled languages, doesn't mean the build as a whole will be fast. Maybe I was holding it wrong, but trying a Makefile-based solution led to so many reads/process spawns that a clean build took about 5 seconds, already much too slow for my tastes. strace doesn't work quite as well here (besides saying "yes, waiting for subprocesses really is the blocker"), so I guess the cumulative overhead of running a bunch of processes that all touch a bunch of files (~600 of both) is just too much, somehow.

=> Build SHOULD Be A Single Process

Ideally, there's just one compiled program running the entire build start-to-finish, reading all the files into memory (my site is not very large), transforming them in-memory, and writing back exactly what's changed. This limits file I/O to the bare minimum, and we should be able to make the in-memory transforms fast too. Existing known-fast SSGs do this, so we must be on the right track!

(2) I Want Fast Re-Builds When Content Changes

I could just say "Fast Re-Builds are just Fast Clean Builds" and be done with it pack it up go home; that's what Hugo, Zola, & most homebrew SSGs do. But no, it'd be boring doing the same thing as everyone else. We need super specil[sic] fast. We need something with a cache, a cache so robust & complete that a re-build runs exactly what's needed based on what's changed, and no more. This is known in the business as an "incremental build". My modest goal is sub-300ms rebuild times, ideally sub-100ms or even sub-20ms if I can swing it.

However! Turns out doing proper incremental builds is kinda tricky. I guess is this why most frameworks avoid it :P Even if you can swing complete dependency tracking, a naïve method of "this file is dirty so I should rebuild everything that depends on it" doesn't suffice, because that can and will over-rebuild. Consider the following scenario:

• Pages are set up such that each one has a link to the next and previous pages, if they exist.
• Markdown files are stored with frontmatter specifying the title that should be used for any links pointing to them.

Then, the rebuild would go like this:

1. Markdown file changed, marked dirty
2. => Frontmatter metadata is marked dirty
3. => Collection of all metadata is marked dirty
4. => Any page reading from "collection of all metadata" is marked dirty
5. All pages read from "collection of all metadata" to create next/previous links
6. => All pages are marked dirty

Notably, step (2) happens even if only the page content changes! This means any change to any page causes the entire set of pages to get re-built, negating any benefits of caching.

You can somewhat get around this by breaking step (3), precisely tracking which pages read from what metadata so a more minimal set of pages are rebuilt, but an even better approach is to also break step (2), by only marking the frontmatter metadata as dirty if it actually changes.

=> Build MUST Use A Real Incremental Algorithm

There are a couple approaches to this sort of fine-grained incrementality that I'm aware of:

1. The Red-Green Algorithm as used by Rust for incremental compilation & explained really well by thunderseethe
2. Nix's derivation model, explained pretty well by shuppy
3. Jade's postmodern build system, which also covers the first two
4. Build Systems à la Carte, which provides a framework for all possible other approaches

I am a huge Rust-head so let's go with the first approach :3

How Red-Green deals with our example is, a step can be marked "green" (unchanged) even if it has "red" (changed) dependencies, so long as its output doesn't change on the re-run. This does the correct thing if both the metadata hasn't changed (only (1) is dirty, (2) is not), as well as if it has changed (1-3 are truly dirty, but many things at 4 aren't dirty because they read the unchanged parts).

fn parse_frontmatter(f: File) -> Metadata;
fn collect_metadata(fs: Vec[File]) -> CollectedMetadata;
fn index_metadata(cm: CollectedMetadata, i: usize) -> Option[Metadata];
fn build_page(f: File, cm: CollectedMetadata, i: usize) -> ();
fn build_all_pages(fs: Vec[File]) -> ();

build_all_pages() >---------------------\
       v                                |
       |                                |
       v                                |
collect_metadata()                      |
       v                                |
       |                                |
+------+--+---------+---------+         |
|         |         |         |         |
v         v         v         v         |
parse(0)  parse(1)  parse(2)  parse(3)  |
                                        |
index(0)  index(1)  index(2)  index(3)  |
^         ^         ^         ^         |
|         |         |         |         |
+--------/^\--------+--------/^         |
^\--------+--------/^\--------+         |
|         |         |         |         |
^         ^         ^         ^         |
build(0)  build(1)  build(2)  build(3)  |
^         ^         ^         ^         |
|         |         |         |         |
+---------+---------+---------+---------/

All of this is also why we need hermetic (deterministic) builds. Red-Green & other incremental algorithms assume that "inputs didn't change" => "outputs won't change either". In the context of a general build system running external programs, this assumption may be incorrect or the inputs very hard to specify². But in the context of a Static Site Generator where you can tightly control all the build logic, I think it is a perfectly fine assumption to make³.

(3) I Want Fast Re-Builds When Logic Changes

Even among other SSGs that do have caching (which the vast majority do not, mind you), they'll assume "most the time you'll be editing content, requiring a full re-build when logic changes is totally fine." But that assumption is bad, actually. Having quick iteration times when hacking on the logic of the site is super valuable & not something I will give up. It's fine if those builds a little slower than changing mere content, but it still shouldn't require a clean build of the entire site just to change the number of posts shown on the home page.

This might be tricky, however, because the previous two requirements reeeeeeally want the build to be in a compiled language. But we'll get there. Just u watch.

First, I'd like to draw a distinction between the "build driver" (program that reads/transforms/writes files) and the "logic" (specification for what files are read/written, and what transforms will be done).

These don't have to be the same program! In fact, many, nay, most SSGs separate these concepts. A common models is a "build driver" (Jekyll, Hugo, Zola) that finds & interprets a bunch of template files containing site-specific logic. These frameworks vary in how much logic they let you express with those templates/configuration files, but what they share in common is the build driver stays the same for all runs.

My one gripe with this model is, the faster the builds are, the less powerful the templates tend to be. As jyn mentions, writing logic inside such templating languages sucks & feels bad, so it'd be much nicer if we were able to write logic in a "real" language instead.

This brings us to another form of static site generator: one that merely provides a framework of composable tools to describe how your site should be built, and then "building the site" is just running the program you've assembled. Hakyll is an archetype for this, and I'd describe "powerful" Javascript frameworks such as Next.JS or Astro as this too: they all have "real languages" to push data into templates, rather than relying solely on folder structure/config files.

Unfortunately, this increased flexibility comes with a cost. If all the tools are Javascript, we get performance woes from earlier. And the logic is written in a compiled language, you have to re-compile whenever the logic changes, and that re-compile can take a while. Here are some language options we could choose, ordered by how familiar I am with it:

The problem is, even among languages with incremental rebuilds, they still aren't "🚀BLAZING🔥FAST" enough for what I want. It is very impressive they're able to go as fast as they are, but remember I'm targeting sub-100ms, not sub-1s. That order of magnitude matters.

So, we've eliminated "just use templates", and also "just use a compiled language". That means we're left with:

=> Logic MUST Be In A Scripting Language

This will probably result in the speed tradeoff I am most satisfied with. Probably. "Doing the incremental steps" & "transforming the files" are still best done in a compiled language, because those are a thorny bit of programming with plenty of opportunity to be slow. The remaining logic can use that incrementality and be written in a scripting language to support faster on-the-fly changes. Separating the "build driver" from the "logic" is what other fast SSGs do after all, we're just making each a bit more powerful ;)

My Vehicle Of Choice: Rust Binary + Javascript Interpreter

Based on everything above, I think you can see how we landed here: it's a hobby project, so I really really want to write Rust, and Javascript is a fine choice because:

1. I know it well
2. The string templates are pretty good
3. It goes Fast Enough for logic
4. I'm very scared of homemade languages/Lisp

Stuff that will go in the compiled Rust core:

• Incremental engine
• Markdown parser (comrak)
• HTML minifier (minify-html)
• URL Fetcher⁴
• Image Parser/Optimizer (TODO)
• Javascript interpreter (rquickjs)

And everything else, from base template partials to tag pages to the thing that makes a random PFP show up every time you refresh the page, will go in the Javascript files. It's another post entirely for how I'll actually structure the engine/scripts, so I'll leave it here for now :)

Related Projects

Other incremental blog engines that inspired me to write this one:

• shuppy's cohost clone autost
• fasterthanlime's dodeca project
• jyn's flower project, especially their four posts about build systems

I actually wanted to use one of these, but a hard look at each concluded they too fall under "for author's personal use only" and wouldn't suffice for my purposes. Still worth checking out tho!

Anyways that's it, stay tuned for if I ever finish this, hope you enjoyed ^^

User-Agents in alphabetical order, marked up with links where applicable:

• Emacs Elfeed 3.4.2¹
• Feedbin feed-id:2970179 - 3 subscribers
• FeedFlow (RSS Reader; +https://feedflow.dev)
• feedi/0.1.0 (+https://github.com/facundoolano/feedi)
• Feedly/1.0 (+http://www.feedly.com/fetcher.html; 1 subscribers; )
• feedparser/6.0.8 +https://github.com/kurtmckee/feedparser/²
• FreshRSS/1.28.2 (Linux; https://freshrss.org)³
• Mozilla/5.0 (compatible; Miniflux/2.2.16; +https://miniflux.app)⁴
• Mozilla/5.0 (compatible; theoldreader; support@theoldreader.com; 1 subscribers; feed-id=40411548cd512d9254ed529)
• NetNewsWire (RSS Reader; https://netnewswire.com/)⁵
• Newsboat/r2.42 (Linux x84_64)
• QuiteRSS 0.19.4 (EOL :c)
• Reeder/5050102 CFNetwork/3860.300.31 Darwin/25.2.0
• rss-parser
• RSS.Social/1.0 +https://rss.social/bot
• ScourRSSBot/1.0 (+https://scour.ing/bot; 2 subscribers)
• sharnoff.io-rss-fetcher/5cb7472 (reqwest 0.12.7)⁶
• Skyreader/1.0 (+https://skyreader.app)
• SpaceCowboys Android RSS Reader / 2.16.1(3858)⁷
• Tiny Tiny RSS/Debian 21~git20210204.b4cbc79+dfsg-1.2 (Unsupported) (http://tt-rss.org/)⁸
• UniversalFeedParser/5.1.3 +https://code.google.com/p/feedparser/
• Yarr/2.5

From the bottom of my heart, thank you for reading. It means a lot to me that y'all actively choose to read my stuff. I see you, I appreciate you, this diversity is what keeps the web alive.

So anyways ! I got around halfway thru another "blogging about blgging" post when I thought it might benefit from some feedback, forgoing of my typical "one & done" approach to drafting. The way I normally go about this is:

1. I attach the raw Markdown file to a Discord message.
2. My recipients open it in the editor of their choice (usually Obsidian because it's really good).

But that sucks. I can't edit the attachments after-the-fact, so if I want to make a tweak, I'd either have to give everyone a diff to apply, or re-send the entire file to everyone, both of which are annoying for all parties involved. "Surely there must be a better way !"

Turns out anything is possible on the computer provided you apply enough force. So I that's what I did!

Attempt 1: What I Have Currently, But Better

So. I already have a post composer. And it already saves drafts to localStorage. What if I could make it save drafts to the server instead, and then the server could provide a read-only rendered view of those drafts to anyone who knows a given UUID?

Turns out that's not too hard at all! It's a few more hacks on top of an already-creaky codebase, but it does in fact work. There are buttons for "save draft" (saves to remote server), "edit draft" (loads from remote server), & "delete draft" (deletes on remote server), and an up-to-date lists of drafts is fetched on every page load.

However, this got me thinking: All the server is doing is managing markdown files, exactly how I'm working locally. Yet, despite being "just files" under the hood, the way I interacted with those files through the web interface was very... special. I still had to manually copy from my Obsidian vault where the "true" drafts lived, and the web-based file interface was much more clunky than anything I had natively. "Surely there must be a better way !"

Attempt 2: Re-Thinking The Interaction

Of course there is. Anything is possible on the computer; I simply needed to provide more force. By the power of my three remaining brain cells colliding all at once, I came up with an Ideal Workflow:

• I can edit the files in Obsidian, locally, where I've always been editing them.
• Said files near-instantly show up behind a URL, assuming I have internet to upload with.

From there, it's just a matter of figuring out how to enable this :)

I already have Syncthing set up for syncing drafts between my phone and my desktop, so I also added it to be synced to the /var/opt/syncthing directory on the server the post composer lives on, using Docker because I'm lazy (docs):

# syncthing/docker-compose.yml
services:
  syncthing:
    image: syncthing/syncthing
    container_name: syncthing
    hostname: syncthing
    environment:
      - PUID=1001
      - PGID=1001
      - STGUIADDRESS=
    volumes:
      - /var/opt/syncthing:/var/syncthing
    network_mode: host
    restart: unless-stopped
    healthcheck:
      test: curl -fkLsS -m 2 127.0.0.1:8384/rest/noauth/health | grep -o --color=never OK || exit 1
      interval: 1m
      timeout: 10s
      retries: 3

# systemd/syncthing.service
[Unit]
Description=Syncthing via Docker Compose
Requires=docker.service
After=docker.service

[Service]
Restart=always
User=root
Group=docker
TimeoutStopSec=15
WorkingDirectory=/home/ubuntu/infrastructure/syncthing
ExecStartPre=/usr/bin/docker compose -f docker-compose.yml down
ExecStart=/usr/bin/docker compose -f docker-compose.yml up
ExecStop=/usr/bin/docker compose -f docker-compose.yml down

[Install]
WantedBy=multi-user.target

and, of course, we also want to expose the management interface remotely, just in case things go wrong (docs):

# nginx/rc.wolfgirl.dev.nginx
server {
	# ...
	location /syncthing/ {
		# Authenticate with ngx_http_auth_request_module, for my custom JWT authentication
		auth_request	/auth;

		# Needed in order for syncthing to know it's secure behind a reverse proxy
		proxy_set_header	Host "localhost";
		proxy_set_header	X-Real-IP $remote_addr;
		proxy_set_header	X-Forwarded-For $proxy_add_x_forwarded_for;
		proxy_set_header	X-Forwarded-Proto $scheme;

		proxy_pass	http://localhost:8384/;

		proxy_read_timeout	600s;
		proxy_send_timeout	600s;
	}	
	# ...
}

Then, I can configure my website to read from a subdirectory of that synced folder. (It's important to be a subdirectory, since that means I can change the visibility of a draft at any time.) Finally, I can re-use the post renderer I made in Attempt 1, get rid of most the composer functionality, add a simple file explorer, and blam! Drafts :)

At this point, one might think "well why don't I just move my entire site onto this system?", but I have good reason not to: the finished post site has a very different performance profile compared to a drafts site. Because finished posts (1) change much less often and (2) serve much more traffic, I prefer to serve it from statically-generated files in an S3 bucket behind AWS Cloudfront w/ generous cache policies, whereas the drafts site is perfectly good to "re-render Markdown to HTML every time a request is made" on a VPS w/ no caching. There are tradeoffs with either approach, and the usecase requirements inform each deployment.

This does mean the markdown rendering/CSS for drafts vs finished posts are slightly different. Overall it's not a huge biggie, but at some point I would like to unify them. That is a fair bit harder tho, so I'll leave that for later :)

Thanks for reading, hope this inspires some more cursed setups!! :3

How'd I Do Last Year?

Before getting into my new resolutions, let's take a look at my progress on last years' resolutions:

• 2025-07-22: Do A V7 Boulder

This was an extremely soft V7, so under most circumstances it should not have counted, but I did it anyways!

I haven't updated my resolution for bouldering because V8s are still very far out of my wheelhouse; I'm more comfortable doing V5s than last year, but I'm still having to cut my teeth on real V6s before I can truly say I've done a V7. Climbing is mainly a social activity, so I'll be doing it regularly regardless.

• Update My Wardrobe

Had to carry this one forward 😔

Fortunately, I have more concrete plans to achieve it this time around, with one thrift store tour already scheduled & another in the works. I don't yet feel comfortable going out in public in feminine clothes (which isn't helped by the fact I don't have any lol), let alone shopping for them, so having a group with me will be helpful. Hopefully getting some good outfits & having more occasions to wear them will change that.

• Run a Half-Marathon (again)

Nope 😔

I have dialed in exactly what's been causing my headaches fortunately, but while "don't have allergic reactions" (solved w/ Flonase OTC nasal spray) & "don't become overly stressed" (solved by not doing that lol) are manageable, "don't jostle the head around for long periods" is most exacerbated by running. I've tried light jogs at various points in time, all with the same result. I am fairly sure it's the movement instead of the cardio because intense stationary bike is just fine.

It sucks to give up on this, because I loved running so much, but at least I've found other ways to get exercise.

• blog more
  • more HN frontpages 😎

Somewhat of a success!! I did not blog nearly enough for my own liking, though I did get a Typescript post to HN front page (don't look at the comments, I made that mistake). I don't really use HN tho, preferring to browse & participate in the lobste.rs community instead. HN is only notable for the amount of traffic it drives & re-aggregation that happens as a result.

• go out more
  • take more photos
  • make new friends
  • hang out with current friends more

Triple check!! Again, still have room for improvement here, but I did do all of these things. Not really a resolution w/ a deadline, more of a standing attitude.

• get better at Japanese
  • level up a lot more in WaniKani (Japanese kanji learning app)
  • catch up with Yotsuba&! in Japanese

DNF 😔 and no plans to continue it, because I have other things in mind. Good thing that WaniKani had a lifelong purchase option lol.

Resolutions for This Year

When choosing these, I stuck to the criteria "what would be really frustrating if I hadn't achieved by next year?"

Job

I am not super enthusiastic about my current job. Granted, by almost much any metric, I have it pretty darn good! Coworkers are great, pay/benefits are great, no complaints about how I'm treated. I do feel a little "stuck", tho, like I'm not working on significantly harder problems than when I first joined, despite theoretically having a lot more knowledge about how our systems work. I am the "go-to" when it comes to debugging some of these systems, which feels nice, but at a certain point debugging stops being exciting and is more "oh boy here i go fixing kludges again"¹.

idk maybe it's just an ego thing & i need to check myself. But it really does feel like I'm not able to give a representative amount of effort compared to what I'm capable of, because I'm not excited by anything anymore. The company's direction is not my preferred direction as of late, especially compared to how it was early on. So I think it's time to look for greener pastures—after checking both sides of the fence to make sure the grass is truly greener ofc.

Wardrobe

Yeah!!! hashtag fempilled hashtag girlmaxxing. See previous section.

HRT

Similar to the job, it also feels like growth here has stalled. In some sense, stability here is good, don't want wild mood swings or crazy levels or anything. On the other hand, while smooth skin & small breasts r excellent additions, i would prefer "medium" or even "large" versions of the latter if at all possible. to fill out potential outfits ofc.

Genetics are likely not in my favor but hey we'll see. Everything I've read says injections are better than sublingual pills anyway, and while reviews of prog are mixed it's worth a shot at least I think. Current doctor doesn't want to change anything about my regimen so, uh, should do something about that first.

Rapid Fire Other Goals

Less important than the main goals, but still quite nice to work towards:

• Better At Art
  • I'm trying Drawabox again, seems better than last time I tried 4 years ago, we'll see how long it sticks tho. You'll know when I fall off updating this Bluesky thread of beginner doodles
• Invite Friends To Things
  • I am very good at saying "yes" when invited (mostly because I don't have a whole lot going on anyways), but no so much on the organizing front. This should change, because I always feel great when I get invited so time to pay that forward.
• Blogggggg
  • I have a few projects that are in a perpetual unfinished state so I'd like to get those out and also maybe develop a better cadence than "once every 3 months or more".
  • Still going to keep up the album posts tho, it's a great forcing function to listen to more new albums.

Anyways!! 2025 sure was a year huh. Only caught COVID once (proximate to the three times in 2024), went on trips with friends multiple times, so despite the rockiness in the world at large, it was an alright year all things considered. Reading some end-of-year accounts from others, I'd even go as far as to say I had a great year, and for that I am extremely grateful.

2026 has already started rocky in the large, but I'll do my best to not let that distract me from thriving. Here's hoping that your year will be on the up-&-up too.

And remember: EGGBUG LOVES YOU!!!

PSYQUI was my introduction into the fascinating world of Japanese EDM, by way of those pirate Youtube channels that just put a simple visualizer on top of some anime art. The albums he contributed to back when he was most most active in 2018-2019, like SPD GAR 001 and TECHCORE EVANGELIX 01, absolutely still hold up to this day. His style is effect- & bass-heavy with a strong beat, one of those micro-genres where no one else can replicate it. It's been a long hiatus for sure, but I am very happy to see some more from him in a full EP!

Favorite Track: 02 – Why Don't You

(this was the only non-queer album I promise the rest are Very Queer lol)

Poetically – Queer Oddities

I found this artist somewhat randomly when their song Girls In High Heels came up in discovery a while ago. That's from their first album, Odd Queer, and this is their B-sides follow up. Like the song that first enamored me, this album is very queer & a bit cheeky about it, distorted vocals & instruments leveraging the vocabulary of pop music to make something a little outside of it but still very catchy.

Favorite Track: 06 – Sun of a Beach

passengerprincess – PASSENGERPRINCESS

Apparently this was a meme in some furry circles when it first came out?? I overlapped just enough to find out about it, and my rule of thumb "if it's a meme, it's probably good, because a lot of people liked it" hasn't let me down.

This debut album definitely falls into the "hyperpop" genre, with similarities to artists like 100 gecs & Vylet Pony being readily apparent. The messages are a little more plainly stated, but I still get the sense we are laughing with the artist; it's very joyful & laser-targeted at the queer + furry spheres I inhabit.

Favorite Track: 04 – 4K CARPET

June LaLonde – Framemaker

Before listening to this album, I did not understand what the big deal w/ guitar was, "why are people so obsessed with that instrument?". After listening to this album, now I understand. June LaLonde uses it to incredible effect, pulling emotions out of me I was not ready to feel yet ones I am glad to have felt anyways. It was written while she was grieving the deaths of people close to her, which checks out. The resulting pieces are sad, yes, but also hopeful, a fitting message for such a commemoration. My favorite song is one of the more up-beat ones, but I think this album deserves to be listened to in full. Give it a shot even if you too think it's out of your strike zone; it sure did hit for me at least.

Favorite Track: 06 – Land of Kings

I've had a blast listening to lots of albums this year! it's been a great way to consume music and I plan to keep this going into next year too. Hopefully there will continue to be Bandcamp Fridays to force my hand at periodic roundups like this, otherwise I tend not to remember lol. Anyways that is it for now, happy listening!!

Convention: The as Operator

Ah, good ol' as:

const cast = <A, B,>(a: A): B => a as unknown as B;

We can't just directly do a as B because Typescript is smart enough to warn us about that, at least. That very same error message also says,

If this was intentional, convert the expression to 'unknown' first.

So we can just do that :3

If we were approaching this from a type theoretic perspective, it's already done & dusted, we have the most cut-and-dry demonstration of unsoundness, pack it up go home.

But, what if we couldn't use as? Can we still get between two completely unrelated types?

Unconvention 1: The is Operator

is is commonly used for for interfacing with Typescript's flow-typing system, helping it figure out what exactly the return value of a boolean function means. For example:

const notUndefined1 = <A,>(a: A | undefined): boolean => a !== undefined;
const notUndefined2 = <A,>(a: A | undefined): a is A => a !== undefined;

const maybeNumber0: number | undefined = someExternalFunction();
if (maybeNumber0 !== undefined) return;
// Thanks to flow-typing, Typescript knows that `maybeNumber0: number`
// if we get here.
const maybeNumber1 = someExternalFunction();
if (notUndefined1(maybeNumber1)) return;
// However, Typescript cannot infer flow from ordinary functions;
// At this point, it still thinks `maybeNumber1: number | undefined`
const maybeNumber2 = someExternalFunction();
if (notUndefined2(maybeNumber2)) return;
// The `is` annotation has the exact same `boolean` value at runtime,
// but provides extra information to the compiler, so Typescript can know
// that `maybeNumber2: number` if we get here.

However, is is sort of an escape hatch outside the regular typing system, and we can abuse it to tell the compiler whatever we want:

const badDetector = <A, B,>(a: A): B => {
    const detector = (_ab: A | B): _ab is B => true;
    if (detector(a)) return a;
    throw new Error("unreachable");
};

Typescript doesn't (and can't!) check that the function body is actually doing what the is assertion says. So we can just write a bad one on purpose! (Or on accident, introducing quite a subtle bug.)

Unconvention 2: Mutation Across Boundaries

This cast requires a "seed" value b: B in order to be able to cast a: A to B, but make no mistake: this sort of thing can come up fairly often if we're not careful about how we mutate objects.

const mutation = <A, B,>(a: A, b: B): B => {
    const mutate = (obj: { field: A | B }): void => {
        obj.field = a;
    };

    const obj = { field: b };
    mutate(obj);
    return obj.field;
};

I showed this to a type theory friend and their reaction was:

bruh ts type system mega fails Variance is hard xd xd xd

What they meant by that was, the coercion from { field: B } to { field: A | B } is unsafe when the destination field is mutable; if we allow it, we get exactly the behavior shown here. To make it safe we'd need { readonly field: A | B }, which then prevents the mutation.

Another way of thinking about this is, Typescript currently has no way to "flow" the cast of/assignment to obj.field after the function runs. (Potentially on purpose, because that would make the type system even more complex & limit certain useful patterns.) Inlining the obj.field = a; allows us to catch this, but the analysis does not go across function boundaries.

Unconvention 3: Smuggling Through Structural Typing

Typescript is structurally typed. This means that, if we have an obj: { field: string }, all we know is that there exists an obj.field: string. Typescript doesn't care at all if obj has other fields, and in fact this is the biggest advantage of structural typing: we can freely "upcast" to less restrictive types (i.e. fewer fields) without having to change runtime representations.

The downside of this sort of upcasting is that, some operations like Object.values/the spread operator are only properly typed when they have a complete list of fields, and have their assumptions violated when extra fields are in the mix:

const loopSmuggling = <A, B,>(a: A, b: B): B => {
    const objAB = { fieldA: a, fieldB: b };
    const objB: { fieldB: B } = objAB;
    for (const field of Object.values(objB)) {
        // Object.values believes all fields have type `B`,
        // but actually `fieldA` is first in iteration order.
        return field;
    }
    throw new Error("unreachable");
};

const spreadSmuggling = <A, B,>(a: A, b: B): B => {
    const objA = { field: a };
    const obj: {} = objA;
    const objB = { field: b, ...obj };
    // `objB.field` has been overwritten by the spread,
    // but Typescript doesn't know that.
    return objB.field;
};

These casts have the same restriction as (2), in that we require a "seed" value b: B in order to make it typecheck. Still, it's a bit of a double-whammy, because trying to avoid (2) by copying objects with a ... spread can make you run smack-dab into this one elsewhere.

Unconvention 4: | void is Very Bad

This one's by far the most unconventional; the rest you're probably aware of if you've worked with Typescript for a while, but this one hardly ever comes up because it's such a "why even do this" kinda deal. Still, I have seen it in my work's codebase (and immediately excised it once I realized), so it's not impossible to come across.

Anyways here it is:

const orVoid = <A, B,>(a: A): B => {
    const outer = (inner: () => B | void): B => {
        const b = inner();
        if (b) return b;
        throw new Error("falsy");
    };

    const returnsA = (): A => a;
    const voidSmuggled: () => void = returnsA;
    return outer(voidSmuggled);
};

This is a combination of a few interesting things. For Typescript, void is primarily seen as a function's return value, indicating "I don't care what this function returns because I'm not going to use it". This is why any function, including our () => A one, can be safely coerced to () => void. Usually, this is safe, because once we have a () => void, we really can't assign its output to a variable, nor can we directly type a value as void; it's a very special type after all.

However, void can still participate in type combinations like B | void. And, because functions are covariant in their return type, () => void can be safely coerced to () => B | void. And, as it turns out, we can assign that B | void return type to a variable!

If void were meant to be assigned directly, it should behave something more like any or unknown. But it's not, so instead it behaves like a falsy type, because a normal void-returning function actually returns undefined at runtime. This is how we're able to if (b) return b; (which is not the same as checking b's true type!) & still have everything typecheck.

Unfortunately, that means this cast only works for truthy a. But that's not too much of an issue I think, the Cool Factor outweighs this limitation :3

Does This Even Matter?

Yes, but it's complicated.

On the one hand, Typescript is clearly just a "best effort" at adding types to Javascript, and it does a darn good job at that. If you're holding it right, these things don't come up, and your code genuinely is much much safer than if you used raw Javascript.

On the other hand, all these "unconventions" are real footguns one can stumble into & unintentionally introduce unsafety into your codebase. It only takes a little bit of unsoundness in one place to render entire swaths buggy. We can do our best to avoid these patterns manually, but an automated solution will always be better at catching them.

What Can We Do About This?

TL;DR use typescript-eslint². While neither Typescript³ nor Eslint⁴ on their own come with enough rules to detect any of these, the typescript-eslint ruleset includes things like @typescript-eslint/prefer-readonly-parameter-types (prevents (2)), @typescript-eslint/no-invalid-void-type (prevents (4)), & @typescript-eslint/no-unnecessary-type-parameters (prevents the rest by making the unknown viral). Unfortunately, all of them are opt-in, and Typescript + eslint + typescript-eslint always requires a fair bit of mucking about to get working.

Anyways, I hope these examples are enough to convince you to use more aggressive linting on your Typescript projects in the future :3

Ninajirachi – I Love My Computer

Man. I was extremely hyped for this one after listening to Girl EDM (covered in issue #7), and not only did this not disappoint, it completely blew me out of the water & is now my favorite thing put out by Ninajirachi.

This is a hyper-modern kind of EDM, very much the kind that can only be made on a computer, which this album is absolutely reverent of. It's also about Nina's relationship with computers & making music, most obviously in tracks like "F**k My Computer" & "Sing Good", but really this theme is present in all tracks.

Favorite track: 04 – CSIRAC

"All I Am", "All At Once", and "Sing Good" are all really good as well, but are sort of too emotional for specific moods. CSIRAC is my overall favorite because it's a pumping bop no matter what!

Various Artists – Stream Palette 5 -Ranked-

Stumbled upon this by a chance algorithmic recommendation and yeah, I do love Japanese music, thanks for noticing! It's got all the big names like Zekk, Nhato, KO3, & more, with a corresponding diversity of genres like trance, phonk, color bass, & of course, dubstep:

Favorite track: 08 – Nhato – Killer in the Mirror

This being the 5th in a series does make me want to go back for more. The "-Ranked-" part seems to mean that this is the more hardcore version, as compared to "-Chatting-" which is more chill; wonder how that mix works on the previous, non-split albums? anyways

Iglooghost – Bronze Claw Iso

With Iglooghost, you always know you're getting cool new experimental music, complete with tons of backing lore, this time at https://neurowaste.net. True to the theme, this one is a lot more industrial-sounding than past albums like Lei Line Eon or Tidal Memory Exo.

Because the sounds are so experimental, they're always a bit hit-or-miss for me, but there's usually a banger tucked away, like Slug Song. Not to insinuate the other tracks aren't good (of course they are), just that this one song gets stuck in my head quite often & doesn't leave.

Favorite track: 04 – Slug Song

Moe Shop – EVO EVO

New classic Moe Shop! Their last full album release was Moe Moe, all the way back in March 2018!! They've certainly kept busy since then, with a steady stream of collabs with other artists, commercial commissions, & DJ sets over the years (as per their website, https://www.moeshop.jp). Still, it sure is nice to have something full-length to chew on :)

Favorite track: 02 – COMPLICATIONS

Like I said, "new classic Moe Shop", very distinctive flavor of music they have :P Of course, this album also includes a bunch of collabs, including one with everyone's favorite Cohost artist:

07 – NEWLY HUMAN FEELING (feat. Jamie Paige)

Lastly, I do want to give another shoutout to an album that I've been listening to a bunch again, Tanger – Prefer not to say. Can't believe it's almost been a full year... Anyways, that's it for this issue, happy spooky pumpkin season!

Step 1: Find Images

There's no getting around this: to start, I had to take a lot of photos just to sift through them all in order to come up with (only!) 9 final pieces. Each step is a whittling-down: from "everything" to "just stuff that immediately looks interesting" to "stuff that's probably going to work" to "stuff that actually does work". Let's see an image that made it all the way, image #1, IMG_0540.HEIC¹ ("antennae"):

Step 2: Look For Interesting Encodings

My goal for this project was "make something that looks good even after being run thru a -quality 1 encoder". In order to make that happen, I first had to know how each encoder behaved. I had in mind 4 lossy image formats: AVIF, JPEG, JPEG XL, and WebP. There are other lossy formats out there, but these I already knew had "interesting" behavior at low quality levels & were readily understood by all my image manipulation tools.

To make these, I used ImageMagick (and a POSIX shell):

for enc in avif jpeg jxl webp; do
    magick -quality 1 IMG_0540_stage0.png IMG_0540_stage1.${enc}
    magick IMG_0540_stage1.${enc} IMG_0540_stage1_${enc}.png
done

Right away, we see that JPEG (the oldest format) is a huge outlier; compared to the other formats, it's done crazy things to the colors & shapes, very limited palette & lots of hard edges. Quite distinctive, but maybe a bit too crazy... and also too recognizable, everyone & their mom knows what a deep fried JPEG looks like nowadays.

AVIF is second when it comes to aesthetics. It does a little bit of fun stuff with the colors still, especially with that prominent line down the middle, and generally makes things more blocky too. JPEG XL is third, making things look blockier if you zoom in but not from a distance. WebP is dead last, everything just gets blurrier in a boring way.

Here are the resulting filesizes²:

$ eza -la IMG_0540_stage*
.rw-rw-r-- 9.7M me 27 Sep 16:53 IMG_0540_stage0.png
.rw-rw-r--  22k me 27 Sep 16:55 IMG_0540_stage1.avif
.rw-rw-r--  96k me 27 Sep 16:55 IMG_0540_stage1.jpeg
.rw-rw-r--  68k me 27 Sep 16:55 IMG_0540_stage1.jxl
.rw-rw-r-- 107k me 27 Sep 16:55 IMG_0540_stage1.webp
.rw-rw-r--  24M me 27 Sep 16:57 IMG_0540_stage1_avif.png
.rw-rw-r--  12M me 27 Sep 16:57 IMG_0540_stage1_jpeg.png
.rw-rw-r--  28M me 27 Sep 16:57 IMG_0540_stage1_jxl.png
.rw-rw-r--  28M me 27 Sep 16:57 IMG_0540_stage1_webp.png

It's interesting to me that JPEG has the second largest filesize when it's also the most unnatural looking; I guess modern image formats really are improved huh :P. It's also interesting to see the filesizes of the resulting PNGs after the other compression passes. "Don't put a zip inside a zip" definitely applies to images, though some fault in the large size might also be due to ImageMagick's default settings. Anyways.

Step 3: Run The Pipeline

After seeing these encoding results on various images, I somewhat arbitrarily decided on the following pipeline:

1. Encode as JPEG
2. Blend with 50% of step 1, 50% of original
3. Encode as AVIF
4. Blend with 75% of step 3, 25% of step 2
5. Encode a JPEG XL

Expressing this as a Makefile:

input_files := $(shell find -type f -name "*.HEIC")
output_files := $(input_files:%.HEIC=%_stage3.png)
all: $(output_files)
# (1)
%_stage1.jpg: %.HEIC
	magick -quality 1 $< $@
# (2)
%_stage1_blend.txt: %_stage1.jpg
	python3 -c 'print(input(""),end="")' > $@
%_stage1.png: %_stage1_blend.txt %_stage1.jpg %.HEIC
	magick composite -blend $(shell cat $<) $(filter-out $<,$^) $@
# (3)
%_stage2.avif: %_stage1.png
	magick -quality 1 $< $@
# (4)
%_stage2_blend.txt: %_stage2.avif
	python3 -c 'print(input(""),end="")' > $@
%_stage2.png: %_stage2_blend.txt %_stage2.avif %_stage1.png
	magick composite -blend $(shell cat $<) $(filter-out $<,$^) $@
# (5)
%_stage3.jxl: %_stage2.png # (5)
	magick -quality 1 $< $@
%_stage3.png: %_stage3.jxl
	magick $< $@
.SECONDARY: # Keep all intermediate files

Each *_blend.txt step was so that I could interactively choose a blending amount for each image, because while 50% & 75% were pretty ok most the time, they were not universally good. As an example, IMG_0540 needed 70% blend on the first stage, otherwise the blockiness didn't show up enough for my tastes. The benefit of using a Makefile like this is that, after seeing a result I didn't like at 50, I could always update it with echo -n "70" > IMG_0540_stage1_blend.txt; make.

I made sure to record all the values used to make every image, but it's too boring to reproduce them all here, so I'll show off just the one completed pipeline instead:

Step 4: Postprocessing

Now, you'll notice that there are no super blocky artifacts in this stage 3 image like there are in the final result I posted elsewhere. This is because I fooled you!! I made things look similar to compression artifacats, when actually it's just a specifically placed HSV Saturation mask designed to blow out the colors in ways that interact strangely with existing HEIC compression artifacts :3

I used postprocessing tricks on other images too, like the Colors > Desaturate > Mono Mixer tool, with "preserve luminosity" turned on & one of the channels having negative weight, for slightly wild effects on #3 ("split sky") & #5 ("undergrowth"). For the last one ("sunset"), I made it so there was a gradient mask smoothly transitioning between the original image in the bottom left to the compressed version in the upper right, following the overall gradient of the sky.

Overall!! There's a lot of fun to be had in messing around with images digitally, this really is just the tip of the iceberg if I want to actually start doing glitch art. I certainly enjoyed myself a lot with this and hope to do more as a part of The Coho Photozine Club!

The zine is not coming out for a while, and there's no guarantee any of these photos will be in it, so I figured I'd just post them now :3

The theme for the inaugural zine issue is "yes-and-ism". As per one of the organizers:

it comes from someone asking "is that__?" and the response being "yes, and [long explanation]". it was a unique aspect to cohost style photography that most of us haven't found a place for outside of cohost or the various life raft servers, where the focus was less on the actual image but more on the process of image making and the conversations said processes inspired.

There's more themes from the primer that I tried to incorporate too, like orthorealism + deliberate imperfection, with my overall thrust being "what would look cool after being passed thru a -quality 1 encoder?" The end results are better than anything I could have asked for.

I'll have a making-of post coming soon hopefully :)

Put https://wolfgirl.dev/tags/programming/rss.xml (or any other tag) in your feed reader, or just point it at the https://wolfgirl.dev/tags/programming/ page directly b/c I'm pretty sure the metadata is set up correctly there too. Adds a couple more seconds to my build time but eh totally worth it.

Also wow I cannot take even 1 hour for simple side-project without running into another bug. I was doing a non-NixOS thing to get away from all the weird bugs!! Fortunately it is just "the URL being rendered in dev is /tags/programming/rss.xml/, which is different from the prod URL", not a hard blocker, but it does make my brain itch a little with the urge to fix it. No. I must resist... Too many other side-projects r crying out to be finished too...

• The ways in which the monster is used as a prop to trigger the real juicy political/emotional conflicts
• Lingering shots of machinery/infrastructure/crowds of people to get a grounded sense of what's going on, and demonstrate that these operations are more than just one person imposing their will, but a collective organizing towards a common goal
• The "don, don, don, don don-don" music that plays whenever there is a tense scenario (score in general seemed very similar i think!)
• Misato for president of the united states of america

I can see why it was a flop with foreign audiences: the CG is real bad, and the movie mostly consists of people talking about things rather than directly fighting the big monster.

I can also see why it was critically acclaimed in Japan: very simple, positive message about the tenacity of the Japanese people in times of crisis, if only these darn old politicians would stop ruining everything.

i rate it "hit Godzilla with a skyscraper"/5. highly recommend if you like Eva 1.0

You see, I like to write my own tools. What that ends up meaning is, in addition to running some "standard" services like a FreshRSS server & a Bluesky PDS on my cloud box, I also run some very non-standard stuff like https://girl.technology/, my post composer, & a small suite of home-grown discord bots. The "standard" services are all nicely packaged thanks to the hard work of many volunteers, but for my own stuff, I have to write my own NixOS modules if i want to run them on NixOS.

Fortunately, writing custom NixOS modules isn't too challenging once you get a proper setup going¹. Unfortunately for present me, past me did not really know about Nix when designing some tools, making assumptions like "of course we will be able to write to the directory my Python code lives in" & "of course $PATH will always be populated with all the programs we could possibly want". https://girl.technology/, being a Rust project where I tried to follow as many best practices as possible (including packaging in a flake!), was thankfully spared most of this, but my post composer & discord bots... first i took a month to polish up a blog post detailing exactly how i was packaging my Python code, just to make sure i understood everything that was going on, then i had to become a nixpkgs maintainer because some Python code upstream was broken (another month or so), and only this month did i realize "wait a minute i haven't actually fixed any of the bugs in my own code yet". so i did that. and hoo momma there were lots

No particular standouts really, just the standard "ah one more substititeInPlace/wrapProgram/dependency for patchShebangsAuto", plus figuring out how to detangle "everything goes into ." into "everything goes into$DATA_DIR"². That and converting a hand-written nginx config to a NixOS nginx module definition is worthy of exactly the time I allotted it (couple hours on a weekend), and yet, uh, just never got around to it 'til now ig 🙃

Finnicky & annoying & feels bad b/c I already have a perfectly cromulent Ubuntu 24.04 LTS server hosting all my stuff already, and it's been happily chugging along with no complaints. Just a few problems:

1. Every time the server reboots I have to ssh in and start a tmux session because all my services run as systemd user services that only run when my user has a session, not on boot.
2. Dist upgrades are liable to break all the Python virtual environments I have set up all over the place b/c those have links back to the main Python install.

Now, was it worth spending all this time just to sand off these few rough edges? Probably not! The one weekend I spent porting Python packages was likely more time than I would spend in the next 2 years managing my current setup. However, while I like to tell myself it's about "opportunities to make my cloud box do things it never even could before", it's really more about the principle of the matter :P

So! Lessons learned when designing my hacky pile of Python + shell scripts to be more compatible with Nix next time:

• No need to shell out to subprocess.run when something in os.shutil can do the trick
• If you do subprocess.run a shell script, invoke it directly so patchShebangsAuto can do its thing instead of an extra manual substituteInPlace step
• Do NOT just dump state into the current directory, put it somewhere configurable please :)

Now!! I've finally finished porting all my services to NixOS, even testing them in a VM to make sure (thanks, nixos build-vm!). So why is the article title "Almost Ready To Switch" & not "Have Successfully Switched"? Well... my target is an Oracle Always Free box, so it is aarch64-linux, and my laptop is x86_64-linux, and nixos-rebuild really doesn't like that situation³. There is a workaround where you use the --fast flag (to disable re-exec of Nix as aarch64-linux on your x86_64-linux host⁴) & a remote --build-host, unfortunately negating many of the effects of having a powerful builder locally. Still nice to have all your sources in one place ig.

It really is a rabbit hole with this Nix thing. "Ah yes just a little more effort and surely then i will have the perfect setup" is a trap that I am currently falling into, and one that I have seen many more experienced Nix users fall into too. Glad it's just a hobby project, at least I'm having fun :)

Ninajirachi – girl EDM (disc 1)

Very fun & energetic EDM. Album description on Bandcamp:

there are girls and EDM and there is girl EDM... cat ears, furry paws, headphones on world off, sci-fi fantasy cosplay windows down hands in the air, kandi bracelet, computer, the next step in evolution and a window to the soul <3

i think that about sums it up ya!

Favorite track: 03 – Ninacamina

I'd also like to purchase her next album, I Love My Computer, but that is only a pre-order so i am unsure if the full discount applies. oh well doing it anyways.

Bad Computer feat. Gold – Connected

Not an album, just a single. Also getting Move Like This, basically everything Bad Computer puts out immediately goes on my favorites playlist.

Pretty much their own microgenre of EDM at this point, very distinctive style, just part of that classic Monstercat sound ya know? u could load a Rocket League menu to this

Emplexx & Sunnexo – Control Shift

It is Emplexx & Sunnexo, two indie artists who I already really enjoy separately, doing a collab!! Must-buy single for sure yaya

They call this genre "complextro 2" and who am i to argue with the masters?

They also wrote about the making-of process on their blog, including the visuals! Very cool behind-the-scenes stuff, u should check it out: https://sunnexo.moe/controlshift

Various Artists – emo.gen 3

Doctór! inject that breakcore directly into my veins, please :3

Apparently this may be the final "emotional breakcore" compilation? As much as I enjoy having Literal Hours Of Breakcore to jam to & crave more, I can understand it is a massive effort to put together, plus there are two more like this lol.

technically u don't have to pay for this one but i did anyways just to be polite. haven't listened to the whole thing yet, just choosing the track by which i discovered it:

38 – Yem – Twin Hearts

UPDATE 2025-08-10: I have listened to the whole thing now. 11, 12, 26, 27, 32, 33, 34, 35, & 36 r all Pretty Good but i still think 38 is the most interesting. got it in one huh !

Jamie Paige – Constant Companions (Deluxe Edition)

yes i know i posted about the normal version before. but this one has BIRDBRAIN on it. and ROT FOR CLOUT again. i would buy another album for those two songs again :P

No need to post track snippets for this one, just watch those youtube videos if you haven't already. soon u too will be up birding ur brain & rotting ur clout :)))

Anyways, there are many, many more tracks I'm buying (including some for a project I don't want to spoil!!), these are just the ones I thought were worth sharing today. Now go! Go forth and consume!!

It's a fun one!

I do think this one counts as a real V7. It is a strength + flexibility test V7, but I can pass that test, so yeah let's go!!! feels good to keep a resolution :)

Bonus Photo

See yesterday's post :3

also Greg said Adobe Lightroom for iOS should be able to produce jpegs with HDR gain maps and it even had the toggle switch for it but idk if it did that right or if Photos/Safari just isn't displaying them right... (click thru to see non-webp versions) ah well least i tried

uh. so. some explanations are in order. XDR a.k.a. UltraHDR if u live in Google land instead of Apple land, is a feature by which displays will adjust their brightness beyond the system whitepoint when displaying compatible photos/videos. this can lead to some pretty striking effects; check out https://gregbenzphotography.com/hdr/ on your (somewhat recent) phone to see what I mean.

Honestly, I wish I had found Greg's website much sooner; it'd a godsend for exactly the information I was looking for (tysm greg ilyyyy <3). Before finding his website, all i came across was this lonely StackExchange post w/ no responses & this random forum post where they discuss manually creating gain maps to apply with command-line tools, which is a bit much even for me.

Before that, I tried out every image editing app I had available to see which one could preserve my image's XDR-ness when edited:

• iOS's default editor¹
• MacOS's default editor
• GIMP
• Krita
• Clip Studio Paint
• Procreate
• Affinity Photo
• Adobe Fresco
• hell, even Adobe Lightroom for iOS since that's free somehow

Unfortunately, according to Greg, the only ones that could have possibly worked are GIMP, Affinity Photo, & Adobe Lightroom. And of those, only Adobe Lightroom has support for gain maps², which my photo used. And! Out of all the apps I tried, Lightroom was the only one without a paintbrush tool :(((( no license plate redaction for me i guess...

So! What's done is done, and I have resigned myself to the fate of a normal Display P3 gamut image for this one, so sad i know. And listen, I already knew that my blog's image preprocessing likely would've wiped out the XDR-ness anyways, so all this effort wouldn't have affected the viewing experience at all, but it's about the principle of the thing y'know? a girl's gotta have principles

anyways hope i can upload the rest of those photos tomorrow. peace y'all ✌️

Maggie, a security researcher moonlighting as a internet sleuth, uses her unique set of skills to investigate a case of online vandalism.

Content warnings: Unreality, vomit, drugs, alcohol, death, missing people, ableism, report cards, video game gore, claustrophobia.

You can play a visual novel adaptation of this story on itch.io.

little bit late to the party on this one whoops :P good fiction is timeless, anything by Blackle Mori is an absolute banger, "absolute banger" > "good", ∴ i can review this whenever i want Q.E.D.

it's a noir-style short story, "homage" is maybe the wrong word, so "snapshot" perhaps, of a certain contemporary internet culture that i myself sometimes rub up against as a stereotypical Internet Transgender. I'd be interested to know how someone not as versed in the culture interprets it; would they develop a better understanding? would they miss a whole bunch of references that are essential to understanding? are the parts that i think are essential not actually? only one way to find out

masterfully executed prose & structure. i really appreciate the attention to detail on all the computery bits too, really soothes my soul. itch.io VN background pictures & text timing add a whole 'nother layer on ambiance to the experience that i really enjoyed. ty Blake Mori for the excellent story <3

4: Auvic – Voices Call

That link is actually to the bandcamp remastered version, which doesn't contain all the songs seemingly? The original version on Apple Music version has many more (and being honest I am not super into the remastering anyways but i digress). That original version is very 2015 in its soundfont, makes me feel groovy, bouncing in my seat a little, saying "woo, yeah", etc.

I do not know genres very well but I will try my best: upbeat electro, inspired by chiptune in some ways, but mostly it's its own thing. Once again, very 2015.

Favorite track: 08 – Upgrades ¹

3: Virtual Riot – Simulation (Deluxe Version)²

No bandcamp link, just an Apple Music one sorry. The original is probably a more cohesive album, the only reason I include this one is because (a) more songs = more flow & (b) between Deluxe/non-Deluxe, it has my favorite song, so yeah :)

Virtual Riot has a long track record of releasing very good color bass (i think that's the right genre label?), never let me down really, but that's all mostly in singles, so it's nice to get a full-length album from them to just plow through when I need that good good.

Favorite track: 17 – Bossfight Afterparty

2: Adam Tell – Peripheries

I mostly knew Adam Tell from his lyrical work/singing in Superseded, Falsework, & Data Loss. Turns out he can work those hardware synths too!! Bit darker than the other ones so far, but it's kind of like that starry night nighttime darkness, kinda cozy in a way at times.

Feels like it'd fit right at home in an Explosions&Fire video during one of those chemistry montages , so I guess that puts it in the ambient genre? anyways

Favorite track: 01 – Transfer Orbit

1: free.99 – Out For Blood

Trans Pride is over. now is the time for Trans Wrath /j

in all seriousness, this album is very angry. overall i am better off just choosing not to comprehend the lyrics³, treating the words as if they were from a foreign language instead, getting all i need to know from keywords/tone of voice. which is to say: get angry!

never fails to light a fire under me. however, because the concepts slide right off me, it is more "generic high energy" than "angry" for me at this point. still feels like a problematic fave, but whatever it's my problematic fave, and you can't take it away from me :P

genre feels like highly produced rave music, if i had to describe it. very move-to-able.

Favorite track: 02 – Lilith

That's it for this month, hope you found something new you liked! Music is very important to me and I hope you enjoy it too <3

You can watch the interactive music video on your phone here — turn on rotation lock & don't drop your device! If you want to watch it on desktop or TV, or if you don't want to have to rotate your device, watch the static version instead.

this music video/multimedia artwork project had a bit of an impact on me (forced a few tears out). very very good song and a very very good retrospective post on the personal journey of making the song.

The post cites "Melodic, emotional drum & bass", as the target, and Ash really knocked it out of the park on this one. Later on they mention Porter Robinson & Virtual Riot as inspirations when it came to the vocal processing, and I can definitely hear that reflected in the beats too. I think "EVERYTHING!" even holds up against those titans? in my ears, at least. Like I said, very very good stuff :3

hard to exactly put into words the emotions i got out of it but i will try anyways: it's about youth, it's about queer joy, it's about that time of night when you're hanging out with your friends & you completely forget yourself & all your worries, or that time where re-unite with friends after a couple of years & everything just clicks back into place, it's about in-jokes & cat pictures & late-night discord calls. it's about Ash, but it's also about you & me. that's all i got, peace

fortunately, i know nix{pkgs} now, so i was capable of resolving the problem. tho i might as well update Sanic while i was at it. and add myself as maintainer for the things I touched. and add that sanic-ext package i need for my backend too. oh wait yeah that third one right.

What Does Being A Maintainer Mean??

All it means is, I will automatically get pinged if someone opens a PR to make changes to these packages in the future :) Doesn't mean I have commit privileges yet! Tho, staring down the never-ending deluge of PRs that crash against nixpkgs daily, I'm not sure I want to really? It's a lot of power, but also a lot of responsibility (). Participating in the process for what happens if your PR drops off the first couple pages (you apologetically post in a Matrix room and a committer very graciously merges it) has been humbling, really reinforces that this is a community effort to make this thing run as smoothly as it does. Now that I've taken a closer look at the gears, I am more awe-struck than ever; there is such a massive amount of labor put into this project, for free, in people spare time, because they believe in it, just like I did. Something special, I think :)

As for why I waited until my last PR was merged (which took about 2 weeks), instead of when my maintainership status was official (2 weeks ago, just 3 minutes from PR to merge, surely some sort of record) to make this post? Well, ya know, girls be silly sometimes :P

• Each player gets 3 booster packs, 14 cards each
• Repeat following steps for each booster pack:
  • Open the booster pack
  • Pick 1 card from the pack
  • Pass the remaining cards to the next player
  • Repeat the above two steps until there are no more cards in the pack to be picked

Because each booster pack is random, and any broadly good cards will likely be taken early, this format naturally leads to more "casual"-style and -power decks, which is definitely more my speed. Also, this set more than others is very tribe-heavy, u just gotta pick ur cute animal & roll w/ it, very beginner friendly!!

unfortunately i did not get the cutest animal (that would be rabbits), ended up doing something midrange-y with squirrels and the black half of a bats package. Overall, I didn't find the drafting part too hard, or the "identifying the best cards" for the deck construction part, it was the act of actually playing Magic: The Gathering that set my brain on fire & made me feel like I had no earthly idea what I was doing.

Game 1

For both rounds, my opponent gets two 2/3 fliers on board by turn 4, and is able to use targeted removal on any fliers I put up to block (not that I get any big enough fliers those rounds), and none of my guys have reach, and also I don't get any of my own removal, so ggs no re onto the next. Despite this very simple description, it still almost went to time because I take a long time to make all decisions because I am new at the game (all other games went to time)

I actually had this guy in my deck and so learned so play him in future games.

Game 2

Stomped in the first round due to the above card and also this guy (aggro flyers my beloved):

But then got stomped in the second due to a "board wipe" (it was a Blue deck so just a "return to hand" effect but basically same thing for me).

Last game went to time, and I was doing pretty badly again, hanging on with just 2 life & an empty board against 19 life & a full board when the turn limit hit. I thought I would have lost at that point but turns out it's a dice roll-off? which i won?? which seems dumb but i'll take the undeserved W ig

Game 3

Got stomped in the first round due to an extremely wide rabbits board. Rabbits are the cutest clan but also extremely scary thanks to the following card:

so yeah once I was facing down a 6/6 version of that (due to creature token creation elsewhere) there wasn't any coming back :P ggs onto next.

My curve the next game was out of this world, got to play this guy with full stats on turn 4:

and activate this guy's ability:

So yeah. My board was pretty juicy otherwise too, able to stomp out opponent's board development before it could happen. ggs onto round 3

This final round was a lot closer. At this point in the night, despite having fully burnt out my decision-making circuitry, I was getting the hang of building boards & managing resources and we were tied at around 14 life apiece. However, it being 10 minutes past closing time and I being the only one eligible for prizing, I generously gifted one of my "yay you went 2-1" booster packs in exchange for officially being recorded as going 2-1 (the other person had a 0-2 record going in and we wouldn't have gotten prizing otherwise).

Concluding Thoughts

Despite being very overwhelmed for the entire process, I ended up having a pretty fun time and came out with a much better grasp on the game. "Trial by fire" 'n all. The friends I went with (friend in town from out-of-state who to experience a different scene, and a local friend who hasn't drafted in maybe 7 years), were not in my pod, and are much more experienced MTG players than I am, and ended up doing very well (3-0 and 2-1 respectively, with the only loss being from when they were paired up against each other). All in all a great time all around, probably not doing it again any time soon tho :P

1: Amamelia - Bananamelia!

rykarn recommended this one!! from the artist page:

Auckland based trans weirdo electronic breaks and house

so yeah, that's what it's all about, no wonder i enjoyed it :3

Compared to many other contemporary songs, the hooks on these aren't quite as strong, but in exchange, they hit a lot harder once you match yourself to their pace. Definitely worth listening to as a full album for that reason.

Favorite track: 4. Colorbox

2: OK Go - Hungry Ghosts

You all know them from their fantastic music videos, and I'll admit I was already familiar with many a track on this album thanks to those too. But! I just really like all the tracks on here?? Even the ones I haven't seen videos for before. These guys rock.

Favorite track: 1. Upside Down & Inside Out

3: Pedestrian Tactics - Nerds Trying To Dance EP

A rare W for Apple Music algorithmic suggestions. Usually it feeds me "niche stuff I've already listened to" or "pop music that I don't particularly like", but once in a blue moon it'll hit me with the new stuff that I vibe with. Best part is, Pedestrian Tactics has an extensive catalog for me to go through, though so far this is my favorite EP of theirs. Slightly experimental electronic, extremely danceable.

Favorite track: 2. Nerds Trying To Dance

4: Jamie Paige - Constant Companions

I think I bought this album as a pre-order when I was listening to a lot of other Jaime Paige and then just... forgot about it?? Either that or fell to the bottom of my pile while waiting for the "perfect moment" to listen to it, after experiencing the sublime artistry of the Machine Love music video/album preview.

turns out the perfect moment is right after watching the berd rots youtube video and getting ROT FOR CLOUT stuck in my head. who else up ROTTING THEY CLOUT /j

Bit more of a "collection of songs" than an "album listening experience" than the other albums I mentioned, but still very good. It's Jaime Paige after all, so every song is lyrical & fun to listen to.

Favorite Track: 3. ROT FOR CLOUT

Oh yeah and update on my last post: We Live In A Society was indeed p good, Track 1 is the most banger but The Ocean Is Fake is nice too, and unfortunately TOHO BOSSA NOVA 14 has not released on streaming services yet :( oh well

Also, as always, if you have any recommendations for things you think I should listen to, please let me know in the comments!! my ears r always hungry 4 more good tunes :3

Main character is a dipshit & very cringe, but never unwatchably so, at least for me. He's a narcissist beyond all redemption, a child's impulse control in a man's body, a toxic human being rewarded by the toxic society he inhabits, and many other poignant descriptions explicitly pointed to by the film. And that's just one of the characters. Anyways I think it's a pretty good flick you should watch it.

Basic Setup

Say we have two Python applications that depend on a third, shared Python library. That is, something like the following:

# shared/__init__.py
NAME = "PolyWolf"

# hello/__main__.py
from shared import NAME
if __name__ == "__main__":
    print(f"Hello, {NAME}")

# goodbye/__main__.py
from shared import NAME
if __name__ == "__main__":
    print(f"Goodbye, {NAME}")

Then, we can run them like so:

$ nix-shell -p python3 tree
fetching path input 'path:/nix/store/1dwky0bis4bkl3qngsc6pmq902swa9b6-source'

[nix-shell:/tmp/blogpost]$ tree
.
├── goodbye
│   └── __main__.py
├── hello
│   └── __main__.py
└── shared
    └── __init__.py

4 directories, 3 files

[nix-shell:/tmp/blogpost]$ python3 -m hello
Hello, PolyWolf

[nix-shell:/tmp/blogpost]$ python3 -m goodbye
Goodbye, PolyWolf

What the programs actually do doesn't matter all too much; what matters is the module structure, and the directory structure. We want shared to be, well, shared between the hello & goodbye programs, and for no one program to be the obvious one to contain all the shared code. This represents how my crossposter is structured, and is representative of many other "monorepo"-style applications too¹.

PYTHONPATH And You

As a slight aside, there's a specific reason we ran the programs with the -m flag earlier. To see this, let's try running without it:

[nix-shell:/tmp/blogpost]$ python3 hello
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/tmp/blogpost/hello/__main__.py", line 1, in <module>
    from shared import NAME
ModuleNotFoundError: No module named 'shared'

Oh no the dreaded ModuleNotFoundError!! Reading around a bit, we learn the module search path is stored in the sys.path variable, and learn about its initialization procedure:

The first entry in the module search path is the directory that contains the input script, if there is one. Otherwise, the first entry is the current directory, which is the case when executing the interactive shell, a -c command, or -m module.

Aha! So that's why: In the -m example, the /tmp/blogpost directory was added to the module path, because there was no "input script", just a module to be run. But without -m, the /tmp/blogpost/hello directory was added, because that's where __main__.py lives, and that directory does not contain a shared module. Is there a workaround? Fortunately, yes, in the next paragraph:

The PYTHONPATH environment variable is often used to add directories to the search path. If this environment variable is found then the contents are added to the module search path.

So! If we really wanted, we could do something like:

[nix-shell:/tmp/blogpost]$ PYTHONPATH="$(pwd):$PYTHONPATH" python3 hello
Hello, PolyWolf

But honestly, I just use -m :) Anywho, this diversion into PYTHONPATH & module resolution wasn't too relevant, still useful to understand, back to the main content now :)

Transitioning To Hatch

You may have noticed something about our basic setup: it doesn't create a virtual environment :( This isn't a problem for our extremely simple hello and goodbye programs, but as soon as we need a single dependency outside the standard library, we'll be in trouble.

For the longest time, I just stuck with the ol' reliable python3 -m venv .venv && source .venv/bin/activate && pip3 -r requirements.txt; venv is included in the standard library, so it'll Just Work™️ on any Python installation >= 3.3, which is to say, "all of them". There are a bazillion other tools to manage Python virtual environments, like virtualenv, pipenv, conda, uv, & more, but I have never needed their full power, hence, ol' reliable.

Now, however, with the goal of "creating packages to be built with Nix", we might want something a bit more powerful, something capable of creating Python packages. The modern way to do this is with the pyproject.toml format, whose guide recommends Hatch as the default build backend, so that's what I'll be using for the rest of this "tutorial".

First, let's make some more directories, and move some files around. Why we're doing this will become apparent shortly:

[nix-shell:/tmp/blogpost]$ for lib in shared hello goodbye; do
> mkdir -p $lib/src/$lib
> mv $lib/*.py $lib/src/$lib
> done

Then, let's define our pyproject.toml for these new folders:

# shared/pyproject.toml
[build-system]
requires = ["hatchling >= 1.26"]
build-backend = "hatchling.build"

[project]
name = "shared"
version = "0.1.0"

# {hello,goodbye}/pyproject.toml
[build-system]
requires = ["hatchling >= 1.26"]
build-backend = "hatchling.build"

[project]
name = "hello" # goodbye/pyproject.toml is identical, save this line
version = "0.1.0"
dependencies = [
  "shared @ {root:parent:uri}/shared", # ":parent" is equivalent to "../", and can be stacked as many times as required
]

[tool.hatch.metadata]
allow-direct-references = true

We end up with a directory structure that looks like this:

[nix-shell:/tmp/blogpost]$ tree
.
├── goodbye
│   ├── pyproject.toml
│   └── src
│       └── goodbye
│           └── __main__.py
├── hello
│   ├── pyproject.toml
│   └── src
│       └── hello
│           └── __main__.py
└── shared
    ├── pyproject.toml
    └── src
        └── shared
            └── __init__.py

10 directories, 6 files

We've created 3 separate "projects", and then put our files within each project in such a way that Hatch expects to be able to build a package from them.

So! Let us now create a virtual environment, build a package, and run it, all in one handy command:

[nix-shell:/tmp/blogpost]$ exit

$ nix-shell -p hatch tree
fetching path input 'path:/nix/store/1dwky0bis4bkl3qngsc6pmq902swa9b6-source'

[nix-shell:/tmp/blogpost]$ cd hello

[nix-shell:/tmp/blogpost/hello]$ hatch run -- python3 -m hello
  error: subprocess-exited-with-error

  × Preparing editable metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [56 lines of output]
      Traceback (most recent call last):
        File "/home/nixos/.local/share/hatch/env/virtual/hello/DMd9LDvE/hello/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 167, in prepare_metadata_for_build_editable
...

AAAAAAAAAA

...
      ValueError: Unable to determine which files to ship inside the wheel using the following heuristics: https://hatch.pypa.io/latest/plugins/builder/wheel/#default-file-selection

      The most likely cause of this is that there is no directory that matches the name of your project (hello).

      At least one file selection option must be defined in the `tool.hatch.build.targets.wheel` table, see: https://hatch.pypa.io/latest/config/build/

      As an example, if you intend to ship a directory named `foo` that resides within a `src` directory located at the root of your project, you can define the following:

      [tool.hatch.build.targets.wheel]
      packages = ["src/foo"]
      [end of output]
...

aaaaa oh wait ok I think I know why this is happening: we forgot to provide an empty __init__.py file to appease the Python Gods. The Python Gods are displeased when a folder acts as a module & does not contain an __init__.py file, and we only avoided their wrath earlier because our earthly machinations provided no threat to their heavenly realm; now that we have dared snatch the fire that is "build tool that manages a virtual environment" from their palace, the only way to placate them is to fix our directory structure, not to torture our config file like their oracles suggested. Allow me to demonstrate:

[nix-shell:/tmp/blogpost]$ for lib in hello goodbye; do touch $lib/src/$lib/__init__.py; done

[nix-shell:/tmp/blogpost]$ tree
.
├── goodbye
│   ├── pyproject.toml
│   └── src
│       └── goodbye
│           ├── __init__.py
│           └── __main__.py
├── hello
│   ├── pyproject.toml
│   └── src
│       └── hello
│           ├── __init__.py
│           └── __main__.py
└── shared
    ├── pyproject.toml
    └── src
        └── shared
            └── __init__.py

10 directories, 8 files

And now, we should surely be able to run:

[nix-shell:/tmp/blogpost/hello]$ hatch run -- python3 -m hello
/home/nixos/.local/share/hatch/env/virtual/hello/DMd9LDvE/hello/bin/python3: No module named hello

Ah, right, stale environment:

[nix-shell:/tmp/blogpost/hello]$ hatch env prune

[nix-shell:/tmp/blogpost/hello]$ hatch run -- python3 -m hello
Hello, PolyWolf

Big success!!² 🐍, amirite? :P

Time To Nix It Up

Referencing Jade's excellent resource on using Nix flakes the non-flake way, let's set up a simple devshell³:

# flake.nix
{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    flake-utils.url = "github:numtide/flake-utils";
  };

  outputs = { self, nixpkgs, flake-utils }: flake-utils.lib.eachDefaultSystem (system:
    let
      pkgs = import nixpkgs { inherit system; };
    in
      {
        devShells.default = pkgs.callPackage ./shell.nix { };
      }
  );
}

# shell.nix
{ pkgs ? import <nixpkgs> { } }:
pkgs.mkShell {
  packages = (
    with pkgs;
    [
      # The python virtual environments themselves are managed with hatch, no need for any other dependencies in this file! Unless you want to put like, language server stuff here ofc
      hatch
    ]
  );
}

And for this next part, it would be very useful for to do a bit of reading on Nix before continuing; I recommend the nix.dev tutorials on callPackage + local files, the NixOS Wiki page on Overlays, & all the basic Nix syntax ones. Otherwise, this will just be blind copy-pasting:

# shared/package.nix
{
  lib,
  buildPythonPackage,
  hatchling,
}:
buildPythonPackage {
  pname = "shared";
  version = "0.1.0";

  src = lib.fileset.toSource {
    root = ./.;
    fileset = lib.fileset.gitTracked ./.;
  };

  pyproject = true;
  build-system = [ hatchling ];
}

# {hello,goodbye}/package.nix
{
  lib,
  buildPythonPackage,
  hatchling,
  myproject-shared, # Here, we take the package defined by shared/package.nix as a dependency, to be hooked up later
}:
buildPythonPackage {
  pname = "hello"; # goodbye/package.nix is identical, save this line
  version = "0.1.0";

  src = lib.fileset.toSource {
    root = ./.;
    fileset = lib.fileset.gitTracked ./.;
  };

  pyproject = true;
  build-system = [ hatchling ];

  propagatedBuildInputs = [ myproject-shared ]; # Here, we declare how exactly we depend on the package defined by shared/package.nix; we'll get into what this means later
}

Whoa that's it!! For the Nixpkgs-style files, at least. Yeah, turns out using Hatch took care of most of the hard part of packaging, the rest is built-in to Nixpkgs itself. We do still have to repeat the dependencies of each project outside of the pyproject.toml⁴, which is a bummer, but oh well the rest of the file is quite small which is good, thank u Nixpkgs

With all these files in place, our tree now looks like:

[nix-shell:/tmp/blogpost]$ tree
.
├── flake.lock
├── flake.nix
├── goodbye
│   ├── package.nix
│   ├── pyproject.toml
│   └── src
│       └── goodbye
│           ├── __init__.py
│           └── __main__.py
├── hello
│   ├── package.nix
│   ├── pyproject.toml
│   └── src
│       └── hello
│           ├── __init__.py
│           └── __main__.py
├── shared
│   ├── package.nix
│   ├── pyproject.toml
│   └── src
│       └── shared
│           └── __init__.py
└── shell.nix

10 directories, 14 files

Next, we have to wire all these packages up inside our flake.nix so we can actually build them. These are fairly unusual packages, turns out; they use a top-level buildPythonPackage variable that doesn't exist in a normal Nixpkgs callPackage. This aligns with the Nixpkgs manual: Python packages are special, because they could be built for multiple versions of the interpreter, and this is the easiest way to do that without having to do something even crazier w/ overrides.

Anyways, here's our updated flake.nix, then an explanation:

# flake.nix
{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    flake-utils.url = "github:numtide/flake-utils";
  };

  outputs = { self, nixpkgs, flake-utils }: flake-utils.lib.eachDefaultSystem (system:
    let
      pkgs = import nixpkgs { inherit system; };
      python3 = pkgs.python3.override {
        packageOverrides = final: prev: {
          myproject-shared = final.callPackage ./shared/package.nix { };
          myproject-hello = final.callPackage ./hello/package.nix {
            myproject-shared = final.myproject-shared;
          };
          myproject-goodbye = final.callPackage ./goodbye/package.nix {
            myproject-shared = final.myproject-shared;
          };
        };
      };
    in
    {
      packages = { inherit python3; };
      devShells.default = pkgs.callPackage ./shell.nix { };
    }
  );
}

Oh wow that's dense! Let's split it up a bit:

let
  python3 = pkgs.python3.override {
    # ...
  };
in
{
  packages = { inherit python3; };
}

This declares a new python3 package in our Flake, defined as an override of the base python3 package, whose final pkgs attribute will contain our custom packages. And how do we declare those? With the overlay pattern!

packageOverrides = final: prev: {
  myproject-shared = final.callPackage ./shared/package.nix { };
  myproject-hello = final.callPackage ./hello/package.nix {
    myproject-shared = final.myproject-shared;
  };
  myproject-goodbye = final.callPackage ./goodbye/package.nix {
    myproject-shared = final.myproject-shared;
  };
};

Each package that we want to add gets callPackage-ed⁵, with its dependencies injected from the "final" resolved set of packages. This is just the Nix(pkgs)-y way of doing dependency tracking such that any dependency can be overridden at arbitrarily deep levels. Fortunately, we're just adding new packages, so we don't have to think too hard about that complexity here.

With that all in place, let's try to build one of our packages:

[nix-shell:/tmp/blogpost]$ nix build .#python3.pkgs.myproject-hello

[nix-shell:/tmp/blogpost]$ ls result
lib  nix-support

[nix-shell:/tmp/blogpost]$ ls result/lib/python3*/site-packages
hello  hello-0.1.0.dist-info

Huh! We might have expected a shared module to appear here too, given we declared it as a dependency. I guess it doesn't do those just yet... Maybe it'll be complete in the main package?

[nix-shell:/tmp/blogpost]$ nix build .#python3

[nix-shell:/tmp/blogpost]$ ls result
bin  include  lib  nix-support  share

[nix-shell:/tmp/blogpost]$ cd result/bin

[nix-shell:/tmp/blogpost/result/bin]$ ./python3 -m hello
/nix/store/kjvgj2n3yn70hmjifg6y0bk9m4rf7jba-python3-3.12.10/bin/python3: No module named hello

No, that works even less! I guess that makes sense, it's not like the default installation of python3 comes with every Python package ever built by default, we'll need to do something to explicitly install those somehow. But this begs the question: how do Nixpkgs Python dependencies work, anyways?

She Propagate On My Build Input Till I Fixed Point

Referencing the reference:

• nativeBuildInputs ? []: Build-time only dependencies. Typically executables as well as the items listed in setup_requires.
• buildInputs ? []: Build and/or run-time dependencies that need to be compiled for the host machine. Typically non-Python libraries which are being linked.
• nativeCheckInputs ? []: Dependencies needed for running the checkPhase. These are added to nativeBuildInputs when doCheck = true. Items listed in tests_require go here.
• propagatedBuildInputs ? []: Aside from propagating dependencies, buildPythonPackage also injects code into and wraps executables with the paths included in this list. Items listed in install_requires go here.

And summarizing in simpler English: use propagatedBuildInputs for Python library dependencies. Okie dokie! I'm sure there are various reasons for this, but we did that earlier so we're on the right track.

But wait a minute, doing a normal nix build did not propagate this build input into the result directory. What gives? This confused me for a long while⁶, until I started looking at things from the perspective of a C compiler.

Basically: Nix was originally meant for packaging C code, so a lot of its semantics make more sense when viewed that way. Packages put in buildInputs will be built, put into /nix/store, their paths exposed in certain environment variables, and then it's up to whatever linker to actually hard-code those paths into the final executable. However, there is no "final executable" at the Python library level, hence why we put the dependencies into propagatedBuildInputs so they'll be available by the time we get around to creating a Python environment. A Python environment can be thought of like an executable in this framework; a script can "link" (literally, create symlinks to) all specified packages into the directory structure Python expects.

Quite fortunately, our goal of "link a Python package & all of its dependencies into a functioning Python environment" already has a simple solution:

# flake.nix
# ...
      in
      {
        packages = {
          inherit python3;
          python3-myproject-hello = python3.withPackages (ps: [ ps.myproject-hello ]);
          python3-myproject-goodbye = python3.withPackages (ps: [ ps.myproject-goodbye ]);
        };
        devShells.default = pkgs.callPackage ./shell.nix { };
      }
# ...

This python3.withPackages function does exactly what we need!! Magic. Now, when we build & run:

[nix-shell:/tmp/blogpost]$ nix build .#python3-myproject-hello

[nix-shell:/tmp/blogpost]$ cd result/bin

[nix-shell:/tmp/blogpost/result/bin]$ ./python3 -m hello
Hello, PolyWolf

It works! Yay!!

Drawbacks

Now, by no means do I intend to represent that this approach is the perfect way of doing things with Python + Nix. Far from it, really. Here are some of rough edges I've run into in this workflow that I'd like to eventually sand off:

• shared, hello, and goodbye have to live in those strange $lib/src/$lib directories. This is solely because pyproject.toml doesn't have good multi-package support; Nix itself is flexible enough, but I am relying on pyproject.toml to make the Nix packaging easier.
• As mentioned earlier, we have to specify all dependencies twice: once in pyproject.toml, and once in package.nix, which is sad. Also, if there are any version constraints in pyproject.toml, those will effectively break package.nix because Nixpkgs only contains one version of each package.
• If shared ever changes, the hatch environments for hello and goodbye will need to be deleted and re-recreated. Hatch itself seems to have a "development mode" where changed to the current project are reflected in the virtual environment, but it doesn't seem like that applies to this multi-project usecase.

Still, this is just about good enough for my project, so I think I will stop here. All this Nix packaging is just so I can write a NixOS module and finally start deploying my site the Cool way (nixos-rebuild switch --flake .#devbox --target-host devbox) instead of the Boring way (ssh devbox 'cd crossposter; git pull; systemctl --user restart rc.wolfgirl.dev'). Join me next time for that, I guess?? Assuming I don't get sidetracked on something else first lol

I found out this was coming due to a teaser video for the first track. It's a bit crudely animated, but I think that was an intentional artistic choice to capture the transition from "meandering jam sesh with crazy discordant electronic noise" to "immaculately layered harmonics extracted from the best noise & ideas". I keep coming back to it for two reasons: the song they get to near the end is quite catchy & exactly my thing yes, but also I really like seeing that raw artistic process, really captivating even if it may have been a simulacra.

An accompanying text post lists the set at 4 tracks long, dropping May 2nd on presumably all major streaming services. That's quite soon, exciting!! All Flume stuff is worth a listen imo, even if I don't end up liking it :)

Shibayanrecords - TOHO BOSSA NOVA 14

This is a very conventional release, both for Shibayan and for Japanese music artists in general: Crossfade available as a whole-album sampler a week before release on May 5th, available for immediate preorder on melonbooks if you live in Japan, and streaming internationally some indefinite time later.

Shibayan always releases a TOHO BOSSA NOVA album in May, every year for 14 years running now I guess. I've mentioned before, and while #2 is so far my favorite, it is nice to have a yearly tradition to listen to something fresh yet familiar, nostalgic yet novel. Excited for this one too, hopefully I can hear it some time in May.

Usually I find a third album for things like this, but heh it's really only these two. Instead, here's two more already released albums I have not listened instead:

• Aritus x FIBRE - Virtues EP (Deluxe): Story goes that this was produced in they heyday of future funk, when high schoolers didn't quite know about music licensing, so now they finally figured out how to get the licensing for this remaster, neat!
• greyl - SINGALS CHRONICLE 01: u already know how i feel about greyl. i def gotta get this

Also, feel free to reach out with album recommendations of your own!! Comment links should be at the bottom of the page assuming my software works correctly. I cannot possibly keep up with all the cool music coming out, so hearing things from people directly helps filter that down considerably.

That's it for now, until next time!

The author does a lot of neat stuff with hardware imaging on their mastodon account, @azonenberg@ioc.exchange, which is very cool, and I've been following for a while. Just got around to reading this piece; it's fun sci-fi, I enjoyed it a lot!

Problem Statement

Given:

• A finite set of structs/enums (henceforth referred to as "items"), possibly generic

Create:

• A way to, for each item in the set, recursively transform any other item, including inner generic parameters, if possible.

Let's take a look at a simple example:

struct ListNode<T> {
    val: Val<T>,
    next: Option<Box<ListNode<T>>>
}
struct Val<T>(T);

Then, given a ListNode<T>, we'd like to be able to turn it into a ListNode<U>, just by specifying how to turn Val<T> into Val<U>.

Approach 1: Write a traversal manually

impl<T> ListNode<T> {
    fn transform<U>(self, f: impl FnMut(Val<T>) -> Val<U>) -> ListNode<U> {
        ListNode {
            val: f(self.val),
            next: self.next.map(|n| Box::new((*n).transform(f))),
        }
    }
}

Great! Nice & simple. However, manually mapping through the Option and Box is a bit tedious, especially if we want to write other, similar transforms. Let's see if we can solve that.

Approach 2: Functor time!!

In my last post on this subject, I covered this crazy wacky trait I called Functor, invented specifically to do these sorts of transforms. Here's the trait definition¹:

trait Functor<Output> {
    type Input;
    type Mapped;
    fn fmap(self, f: impl FnMut(Self::Input) -> Output) -> Self::Mapped;
}

And here's how it's used (playground link), with doc comments explaining what each part means:

/// There are 4 types involved in any Functor implementation: {Input, Output} x {Inner, Container}.
/// What a Functor implementation means is, given a function InputInner -> OutputInner, you can write a function InputContainer -> OutputContainer.
/// Writing the full name of each type is a bit hard, so we have shorthand:
/// + InputInner = Input (Val<T>)
/// + OutputInner = Output (Val<U>)
/// + InputContainer = Self (ListNode<T>)
/// + OutputContainer = Mapped (ListNode<U>)
impl<T, U> Functor<Val<U>> for ListNode<T> {
    type Input = Val<T>;
    type Mapped = ListNode<U>;
    fn fmap(self, mut f: impl FnMut(Val<T>) -> Val<U>) -> ListNode<U> {
        ListNode {
            val: f(self.val),
            next: self.next.fmap(&mut f),
        }
    }
}
/// Given all types T that can be mapped over, allow Option<T> to be mapped over in the same way, automatically un/re-wrapping the Some case as required
impl<T, Output> Functor<Output> for Option<T>
where
    T: Functor<Output>
{
	/// Input is the type given as input to the internal transform. In this case, because we are transparent, we use T's input
    type Input = T::Input;
    /// Mapped is the external type after the transform has been done. Because we are wrapping T as input, we also wrap T's mapped type as output.
    type Mapped = Option<T::Mapped>;
    fn fmap(
        self, //: Option<T>
        f: impl FnMut(T::Input) -> Output,
    ) -> Option<T::Mapped> {
        Option::map(self, |x| x.fmap(f))
    }
}

/// The implementation for Box<T> is very similar to that for Option<T>, included for completeness.
impl<T, Output> Functor<Output> for Box<T>
where
    T: Functor<Output>,
{
    type Input = T::Input;
    type Mapped = Box<T::Mapped>;
    fn fmap(
        self,
        f: impl FnMut(Self::Input) -> Output,
    ) -> Box<T::Mapped> {
        Box::new((*self).fmap(f))
    }
}

Neat! This way, if we're writing a lot of transforms, they'll be composable with existing container types like Option & Box, as well as any other container types we might try.

However, I'm still a bit sad about having to write these transforms by hand. Especially given the fact that, while Functor is nice to implement for wrapper types like Option and Box, it isn't as nice for "AST-like" types...

Problems With AST-like Types

Consider what would happen if we had the following set of items:

struct A {
    b: B,
}
struct B {
    c: C,
}
struct C(i32);

Naturally, we'd want to call a.fmap(|c: C| ...). But! There is no item C in any direct fields of A, only indirectly thru b: B. If we only cared about this one fmap call, we could write:

impl Functor<C> for A {
	type Input = C;
	type Mapped = A;
	fn fmap(self, f: impl FnMut(C) -> C) -> A {
		A { b: self.fmap(f) }
	}
}
impl Functor<C> for B {
	type Input = C;
	type Mapped = B;
	fn fmap(self, mut f: impl FnMut(C) -> C) -> B {
		B { c: f(self.c) }
	}
}

Which is fine. But, you might eventually add an item D that you also want to map over from A. Then you'd have to write:

impl Functor<D> for A { ... }
impl Functor<D> for B { ... }
impl Functor<D> for C { ... }

Apparently, the number of Functor implementations you have to write scales linearly with both the depth of the tree and the number of things you want to map over, so quadratically in total, which is quite bad.

Ideally, you'd like to just scale linearly with the size of the tree. Here's one approach for this:

/// These "base impls" can all be easily generated with a macro
impl Functor<A> for A {
	type Input = A;
	type Mapped = A;
	fn fmap(self, mut f: impl FnMut(A) -> A) -> A {
		f(self)
	}
}
impl Functor<B> for B { ... }
impl Functor<C> for C { ... }

/// These "recursive impl"s are written manually, but that's OK, we only need one per item!
impl<T> Functor<T> for A where B: Functor<T, Mapped=B> {
    type Input = <B as Functor<T>>::Input;
    type Mapped = A;
    fn fmap(self, f: impl FnMut(Self::Input) -> T) -> A {
        Self {
            b: self.b.fmap(f),
        }
    }
}
impl<T> Functor<T> for B where C: Functor<T, Mapped=C> { ... }

Seasoned Rust veterans may be able to spot the problem: impl<A> Functor<A> for A and impl<T> Functor<T> for A conflict! Never mind that where clause, the Rust compiler is sometimes very particular about preventing future conflicts too², just in case B: Functor<A, Mapped=B> somehow.

If only there were a way to automatically write all those quadratic dumb-but-non-conflicting implementations... then it wouldn't matter that there's so many...³

Approach 3: Use A proc_macro

Turns out, there's enough information in the item definitions alone to generate the traversals automatically. All we really need is:

1. A list of all items considered a part of the AST
2. The type of each field in each item
3. A way of determining valid transforms to generate for each item
4. A way of generating each valid transform

We need (1) because of the problem presented before: it's not possible to generate Functor impls for arbitrary types, so we must explicitly choose the ones we want to map over.

(2) and (4) are also fairly straightforward given Rust's de-facto introspection libraries, syn & quote⁴. There are some subtleties when it comes to actually extracting the information we want (the Rust type grammar is so much...), but overall "get the type of a field" and "get the shape of an item" is not too hard. Same with generating the transform; once you have the information of "what type am I transforming" and "what fields should I transform for that type", emitting the code is fairly straightforward. You can take a look at my code if you're curious, but I promise it's really boring.

(3), on the other hand, is much more difficult + interesting, and is a big reason I kept delaying this piece. Turns out re-creating the Rust trait implementation coherence semantics is pretty involved!!

What Even Is A Valid Transformation?

Let's start out with the most basic criteria. Any valid implementation of Functor<B> for A comes about because:

1. A has a field with type B, where "with type" looks inside any wrapper types like Option/Box, OR
2. A has a field with type C, where Functor<B> for C is a valid transform.

We get the information for (1) when reading in the items initially. To get (2), we must turn to the horror that is graph algorithms in Rust. Get ready for some math!

1: The Lattice Step

I call this a "lattice" because it encodes a "higher-than" relationship that's useful for what I'm doing. It's not a true lattice by many definitions of the word, but im the programmer so i get to name things badly ok

What this step looks like is, taking all edges matching Functor<C> for A (denoted as A -> C) + C -> B, and collecting them into a new edge A -> B. This is so we can get all the possible implementations that would be covered by case (2) above. There are a few subtleties when it comes to unifying generic contexts, but like the mathematician I'm cosplaying as, I'll leave those for the appendix⁵.

However! We're not done yet. Creating all these new edges gives us all the possible implementations, but not all the valid ones. To do this, we need to filter out all invalid edges, so let's look at some ways an implementation (as defined by an edge) can be invalid.

2: Filter Out Conflicting Generic Instantiations

Consider the following:

struct A1<T, U> {
    bt: B<T>,
    bu: B<U>,
}
struct B<T>(T);

If we try list out all the possible implementations, we get:

impl<T, U> Functor<B<T>> for A1<T, U> { ... }
impl<T, U> Functor<B<U>> for A1<T, U> { ... }

Note we have two implementations, because B<T> and B<U> are different types. But clearly, these implementations conflict! This is because Functor<B<T>> and Functor<B<U>> are not different traits. Without any preference for which implementation to keep, we discard both.

There is a similar conflict if we instead have:

struct A2<T> {
    bt: B<T>,
    bi: B<i32>,
}

Because Functor<B<T>> could be the same trait as Functor<B<i32>>, and without a "type inequality" bound (not possible to express in Rust iirc), we can't generate these. In this case, we just prefer the Functor<B<i32>> implementation that only transforms bi, because without generic parameters, implementations can never conflict.

The algorithm for deciding if a pair of instantiations conflict is as follows:

1. For each pair of arguments in order:
   1. If both are generic, they must be the same generic. Otherwise, there's a conflict.
   2. If only one is generic, there's a conflict
   3. If neither are generic, there's no conflict.

3: Restrict Which Generic Parameters Are Transformed

This is a bit of a case of me being burned by making my Functor too complex, but screw it I did want that complexity. The situation is, if you have something like:

struct A<T> {
    b: B<T>,
    c: C<T>,
}
struct B<T> {
    c: C<T>,
}
struct C<T>(T);

And we try to write the "full" implementation for A -> B that transforms its generic parameter:

impl<T, U> Functor<B<U>> for A<T> {
    type Input = B<T>;
    type Mapped = A<U>;
    fn fmap(self, f: impl FnMut(B<T>) -> B<U>) -> A<U> {
        A {
            b: f(self.b),
            c: self.c, // ERROR: expected C<U>, got C<T>
        }
    }
}

Oh no type error!! The reason it happens is because we can't transform the generic type of c even though we need to. There are a couple parts to this:

1. How do we know we need to transform c?
   • Because the generic context used by field b, and therefore the generic context transformed by b, is also used by field c.
2. How do we know we cannot transform c?
   • We look at all the outgoing edges from item C, and if there are none that would satisfy the transformation C -> B (for the same instantiated item B as is being considered for field b), then field c cannot be transformed.

Anyways, this is all just to check if we can transform the generic parameter T to U. If we can't, no worries, we can just generate a Functor implementation that doesn't transform T :) Like any sane person would do :)

What Is The Correct Order For These Steps?

If you've been paying attention (no worries if not, it's a long one, you're almost there!), you may have noticed something: step 3 above look at outgoing edges for all nodes one level "below" the current. However, step 2 explicitly removes some edges from the graph. So! We must be smart about the order in which we look at nodes for step 3, so we don't end up keeping an edge A -> B only to later remove the edge C -> B that made keeping it possible.

I claim the correct traversal order is "reverse topological sort order". (Whoopee, more graph algorithms in Rust!) Basically, we should start with nodes that don't point to anything (the "bottom"), then nodes that only point to previously explored nodes, and so on. This ensures we don't end up in that bad scenario.

However! The topological sort order is only defined for directed graphs with no loops. Our graph can have loops, and indeed, that's a big feature for this macro in general, being able to support trees & linked lists & such, which all have edges that look like A -> A or even A -> B -> C -> A. Are we out of luck?

Not yet! By first finding the strongly connected components (omg more graph algorithms!!), we can do a topological sort on those, which are guaranteed to not have loops between them, and our traversal order will be good.

"But what about the order within a strongly connected component?", you may ask. Well, thanks to our lattice step earlier, every loop A -> B -> C -> A in the graph will become a clique, so if A, B, and C are in the same strongly connected component, A -> B & C -> B will always exist. Wow it's almost like I'm actually doing math!!

So, the correct order of steps is:

1. Lattice step
2. Filter out conflicting instantiations
3. Topological sort
4. Restrict generic parameters

And that's all there is to it! ;)

Conclusion

At long last, we have reached the end of everything I know about automatically deriving a Functor implementation. These rules have worked out for me in practice, for two very large ASTs. There are more things I know, like deriving TryFunctor and Foldable and Visit, but those all use these same basic principles too. There are also many more things I do not know, like "what if a field contains a generic type but doesn't have one of the designated items?" or "how can I prevent mapping invalid field types, like fn(T) -> U" or "am I using panic! to handle errors too often", but those are all left for later/never. I do not recommend anyone else attempt to use this code btw, it is simply a dangerous toy for my own amusement, and it will blow up in ur face.

Anyways, thanks for reading, see u next time!! :3

1. Create Mastodon post
2. Create Bluesky post
3. Create commit in GitHub with the URLs of the above posts, for comment URLs
4. GitHub Actions sees the commit and kicks off a build

The problem with this is, the Mastodon and Bluesky posts were created before my blog post URL went live! Social media platforms like Mastodon and Bluesky like to include URL previews, and naturally they need to fetch the URL to do this. I could sort of tolerate just a few servers and fast-clicking users getting 404s, but uh, every fediverse server that syndicates with mine also fetches this URL very soon after posting, in an infamous "fedi hug of death".

The only possible fix is to make sure the post URL is live before creating the Mastodon/Bluesky posts. However, I also want to include the Mastodon/Bluesky post URLs in my final post, leading to a bit of a chicken-and-egg problem. I eventually decided it'd be fine to do 2 website rebuilds, one without the comment URLs and one with. Extending my existing publishing process, I thought I'd do this:

1. Create commit in GitHub w/ no comment URLs
2. GitHub Actions sees this commit and kicks off a build.
3. At the end of the build, GitHub Actions hits a webhook on my server, letting it know which post it should continue the process for
4. Create Mastodon post
5. Create Bluesky post
6. Create a second commit in GitHub with the new comment links, with a special commit format telling GitHub not to hit my webhook a second time

This is janky & nasty but it would probably work. I probably would have done it, too, if only GitHub Actions wasn't such a pain to work with. Very simple things like "parse some text out of a commit message" require shelling out to the classic head | cut and doing some nastiness w/ setting output variables, which, just why.

In my frustration, I had an epiphany: why am i using GitHub Actions at all?? I have a perfectly cromulent server running my crossposter already, so why not... just do the website build on there:

1. Create commit in GitHub with no comment links
2. Build & publish website, using a simple shell script that my crossposter can wait for before moving onto the next step
3. Create Mastodon post
4. Create Bluesky post
5. Create a second GitHub commit
6. Build & publish website again

lol².

Comparing the new publish_blog.sh³ to the old GitHub Actions workflow, it's actually insane how much incomprehensible boilerplate I've been able to get rid of. Much of it is related to saving/restoring caches, but a lot is also the YAML being very verbose. After replacing it with my own, much more comprehensible Python & YAML boilerplate, I understand my own system much better now. Write Your Own Tools strikes again!!

but also, it's fun to make my own art sometimes:

both this & my recent doodle, despite being pretty bad objectively, make me feel happy & communicated similar feelings i think. i should do more art.

This red one. Something about slopers where it pays to be flexible, & this corner, just clicks for me.

It's been a while since my last V6. I blame the holiday season. Fun to get back into it tho, I am still improving, and this did not feel like a fluke. Have my eye on a V7 but seems like it requires a lot more grip strength than I have at the moment... That is what I like in a climb, lots of big slopers that challenge ur balance & positioning instead.

Fun gym, got a good group of friends there by now, here's to keeping my resolution!

A while ago, I wrote my own post composer, which I use to make blog posts from my phone, including this one. It's pretty nice!

One of the requirements I had when making it was "I want my bundle sizes to be as small as possible". Inspired by seeing my work's 66MB React app, I decided to look for other frontend frameworks, still reactive (because that is just the best paradigm), but smaller & faster.

Eventually, I settled on SolidJS. After having used it for a while on more than just a simple "Hello, World!" project, I can confidently claim there's a pretty fatal (for me) flaw:

SolidJS defaults to not re-rendering when values change

From my understanding, this is how SolidJS works: everything is based around the "signal" as the underlying reactive primitive, with function calls as the way to subscribe to those signals:

export const Component = (props) => {
  const [name, setName] = createSignal("");
  const title = () => `${props.greeting}, ${name()}`;
  return (
    <form>
      <Title text={title()} />
      <input type="text" value={name()} onChange={e => setName(e.target.value)} />
    </form>
  );
}

When title() is called, it will then call both name() (obviously) and props.greeting (non-obviously, since it's a special getter, which is why you can't destructure props). By calling these functions, anything that calls title() will then know to re-render whenever either name() or props.greeting changes. So, <Title /> will re-render when props.greeting changes, and both <Title /> and <input> will re-render when name() changes.

However, if you don't make title a function, then name() and props.greeting will be called "too early", and <Title /> will never re-render; the body of Component isn't hooked up to any reactivity, just special places like the return value & createEffect.

This whole approach makes fine-grained reactivity quite simple, but also makes actually writing components a bit complex. You have to hook up all the reactive parts yourself!! It's very easy to get this wrong (I did many times), and the tooling (eslint) isn't helpful at detecting when it happens.

React may over-render, but at least it's easier to conceptualize

Here's the equivalent React example; it's just minor syntactical differences:

export const Component = ({ greeting }) => {
  const [name, setName] = useState("");
  const title = `${greeting}, ${name}`;
  return (
    <form>
      <Title text={title} />
      <input type="text" value={name} onChange={e => setName(e.target.value)} />
    </form>
  );
}

How React works is, any time the component props/useState values change, it runs the component body again, as well as any children, recursively. This gets us a lot of neat benefits over SolidJS, like:

• you can destructure props
• not everything has to be functions
• if something changes in state, it is guaranteed to cause a re-render that will display the change.

That approach would be expensive if, when a re-render happened, we had to remove all the old HTML elements from the page and add back the new ones, so instead React uses something called "Virtual DOM Diffing" to only call browser APIs for elements/attributes that need changing. Some unavoidable downsides are the fact you need to keep a virtual copy of the whole tree in-memory, as well as the pretty complex algorithm to do the diffing.

SolidJS cuts thru all this implementation complexity & potential runtime wastefulness by composing a simple + powerful primitive, signals. Unfortunately, it's a bit too simple + powerful, ending up finnicky to use as a result, like a Lisp, perhaps.

If you're making a non-trivial Javascript-powered application, just use React. The React Compiler takes care of a lot of the problems React has, effectively providing fine-grained reactivity "for free" (bit more compile-time cost oop). Preact is also available if you care about bundle sizes; the only reason I didn't choose it initially was because I wanted to learn a new framework. Still, sometimes u just gotta go back to ol' reliable.

So clearly, I don't need to optimize my spend with it in the pennies n all, but there's still a worrying trend, and I like optimizing, so optimize I shall.

What Happened In January?

Looking at CloudWatch metrics for my CloudFront distributions, we see that neither the number of requests nor the number of bytes downloaded lines up with the price increase I'm seeing²³:

Instead, what I think is happening is that I started analyzing my CloudFront access logs. Whoops!

What's Wrong With The Access Log Egress Pattern?

I have two hypotheses:

1. You're not supposed to put the files from CloudFront access logs into {yyyy}/{MM}/{dd} folders, because LIST requests are 10x as expensive as GET requests, or
2. There isn't a way to use the S3 access logs w/o incurring this cost, because CloudFront just loves to put a bunch of tiny files in ur bucket, and GETing all those files will incur costs.

Unfortunately, I can't really think of any way to solve this that doesn't involve services that are more expensive (Firehose, CloudWatch Logs)⁴ or fetching the files in some other way (AWS Lambda behind API Gateway behind CloudFront) which would still eat most of the same LIST/GET costs.

Let's Do Some Math

I have about 15000 objects in my log buckets, across 60 days. I check it about once a day, so looking at the S3 pricing page & accounting for the slightly nested nature of some of the folders, we get:

$ python
Python 3.11.11 (main, Dec  3 2024, 17:20:40) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 15000 * 0.0004 / 1000
0.006
>>> 30 * 65 * 0.005 / 1000
0.00975

hrm. That's not right. That should only show up as $0.02 on my bill, and clearly it's more than that, so I'm doing something wrong. Maybe it has to make a HEAD request to check if an object's been updated?

>>> 30 * 15000 * 0.0004 / 1000
0.18

Now that's roughly the same order of magnitude, but now it's too large. Maybe I'm not actually checking my charts every day, but maybe I'm still doing my math wrong... Wait a minute, what about the PUT requests CloudFront makes to upload these objects in the first place?

>>> 15000 * 0.005 / 1000
0.07

Bingo, we have a winner. Doesn't like up perfectly because I accidentally nuked part of it once, but yeah that'd do it, especially when combined with the uploads I do to my blog normally. Welp, nothing much I can do about that, mystery solved! So long as I actually run dashboard creation just once a day (or even more if I got that LIST calculation wrong), I think the cost will remain manageable (<$0.20); I've paid a lot more for not understanding the Oracle Cloud "Always Free" tier lol.

Log ingestion is hard!! Still working at it, GoAccess is really nice but still needs a way to get the data. Same with whatever dashboard software I decide to use next⁵.

whenever i come home (to my parents' house), she greets me with a lot of trills & tail wagging

she's always carrying a toy in her mouth, different favorite every day, there's an entire box of toys to choose from and she does purposefully switch out sometimes

our backyard is fenced-in, but she often finds a way to slip thru when we let her out at night to go pee to chase foxes. of course, it's only at night when we can't see her escape method

she always comes back eventually, though sometimes not before the neighbors find her & make a post on the neighborhood facebook group

out of my family, i am by far her favorite. one theory is that i was the first to hold her when we picked her up, but another is that i don't mind playing physical with her

she loves to wrestle, lick your face, jump up on your chair to see what's causing the tasty smell on the table

she knows what the limits are, tho; she never bites or barks w/o good reason, and does not take food unless explicitly offered

the house is dog-proofed, including the living room sofa she loves to jump up and snuggle on

her fur is so soft and i could just pet her forever. she loves being pet forever too, so this deal works out

honestly it's most forms of attention lol. she just loves to play and be loved

she follows me around the house everywhere, and will start/end the day outside by bedroom, if not kenneled up

im gonna miss her when i leave :(

So! I naturally put of doing this for all my feeds for a long time, just clicking thru to the source page, which is fine, that's how the author intended for the piece to be viewed right, doesn't even happen that often, I can totally ignore the repeated reminders of how I procrastinate doing small things that meaningfully improve my QoL¹.

However! Tonight I was so bored I figured I might as well just configure all those FreshRSS CSS selectors, and power thru I did. The process led me to discover the vast majority of blogs have their main content surrounded in <article>. And here I was, using <div class="post"> like an absolute fool... This has been remedied. My procrastination of that Rust blog post I keep meaning to finish has not. Good night.

Interesting read from @tante. "For hackers by hackers" is pretty much exactly what I imagined when making Write Your Own Tools, but he's exactly correct in that not everyone can or should do this.

Going to need to puzzle on this a bit more.

"But PolyWolf!" I hear you say, "That's not a blog post, that's just someone's (Izzy Muerte's) OSS project's landing page!"

And yes, you're somewhat correct. But contained in the guides section is one of the, if not the most comprehensive series of posts on modern CMake that exist today. If you've worked with CMake recently, I highly recommend you check this out.

In doing so, I discovered why Spotify keeps its spot as the biggest game in town: it's not just the playlist non-portability, it's the fact that it's algorithmic recommendation system is leagues ahead of the competition².

So! I'm listen to albums now. Here are some of my favorites:

Madeon - Good Faith

Man. What an album. My current favorites are "Hold Me Just Because" and "Heavy With Hoping", but yeah pretty much top-to-bottom bangers really. "Miracle" and "No Fear No More" were on an older playlist of mine, so it's also a bit nostalgic. Good album.

Laufey - Everything I Know About Love

I heard "Fragile" as part of an animated YouTube Short and immediately had to listen to the full thing. As expected, I cried hard.

After that catharsis, the rest of the album didn't hit quite as much. But still enough to tug at the yearning lovesickness living inside most every 20-odd-year-old. Good album.

ShibayanRecords - TOHO BOSSA NOVA 2

Like seeing an old friend after so long, this album welcomes me back in with open arms and a warm embrace. Yes it has the (at this point quite old) meme song on it but who cares it's my favorites TOHO BOSSA NOVA entry. Yes I've listened to all 13.

#14 is coming out in May of this year, like it always does, and I'll listen to that one too, but even if it usurps this, #2 will always have a special place in my heart. Good album.

I do have a bit of a complex when it comes to sharing photos, however, so these may not be comprehensive, mostly just the highlights.

Fukouka

I arrived quite early on a sleeper flight that I mostly slept in 1-hour bursts for. Getting outside and walking around helped a lot for adjusting to the new timezone.

There were a lot of bikes here! This is just some cool bike parking. I also liked how small the cars were, with their boxy shapes to fit into small parking spaces and thin tires because they were light. International-style cars looked bulky by comparison. I wasn't really in any cities that didn't have a great subway system, but I hear it's like this in the rest of the country too.

Other Things In Fukouka

• Best ramen of my life (so far)
• Life-size Gundam in LaLaport (surprisingly hard to get a good picture of !)
• 7-11

Osaka

The train system really is excellent here. I did go on the Shinkansen (not pictured (they are a also bit tricky to photograph!)), and yeah it was just the best train I've ever ridden. The subway system is also probably the best, very comparable to London, tho the fact that they have all their signage/announcements in English (+ Chinese + Korean at times) tips Japan over the edge.

The big shrines/temples were very impressive indeed, but I also liked seeing all the smaller ones nestled all over the city. It is a very distinctive and cool-looking architecture genre.

Other Things In Osaka

• Osaka castle
• Tatio Station arcade
• Mister Donut

Kyoto

The Kyoto Railway Museum was definitely worth a visit. Didn't have everything in English, but did have a lot, and plus it's just really cool to see a bunch of trains, no matter what language you speak. The museum also overlooks a pretty busy train intersection, fun to do trainspotting from there too.

Saw this cool bird on a hike!

Also saw the deer at Nara park, because of course :) They were so so so cool, very friendly with humans, and the park itself with all the temples nearby is also very good. I loved watching other tourists get swarmed by deer when they bought Shika Senbei :)

Other Things In Kyoto

• Nijo Castle, with a bunch of cool sliding door art you can't take pictures of
• Kyoto Tower
• Meat vending machine¹

Tokyo

Shibuya (southwest of Imperial Palace) was such a cool area (I participated in The Scramble!). I also really liked Harajuku (just north of Shinjuku) and Akihabara (northeast of Imperial Palace). All very distinct vibes, but all very energetic. So many small businesses mixing with large ones. And the number of malls!!

Went to a Shiba cafe in Asakusa, in the giant shopping complex near Senso-ji. Excellent time, they are trained to, if you put your jacket on your lap, sit down on it and curl up for you to pet them. They're so soft!!

Teamlab Planets Tokyo was amazing. Like, you read a normal art museum's marketing material about how it's a "full-body, immersive experience that challenges all the senses" and then the normal art museum will have a slightly larger painting. Whereas here, they are not kidding. Never before have I felt so transported, whisked away from this world into an all-encompasing, unfathomable alternate universe.

I promise you I'm not overhyping it. I only have this picture of the outside because the inside is just so hard to take pictures of, both because of the lighting and the fact that no picture (or even video!) can truly do it justice. It wasn't my #1 thing (that would be the arcades) but I am so glad I went and would highly recommend you go to.

Other Things In Tokyo

• Meiji Jingu, with the large Torii and barrels of sake (Japanese) and wine (French)
• Fushimi Inari-Taisha
• Pokemon store
• Kappabashidogugai street, where you can buy any kitchen supply you can imagine
• So many arcades & crane games
• Vtuber advertisements

And that's about it! So many stories, hope you liked what I've shared here. Japan in general was such a great place to be. Great food, great cities, great nature, affordable, just about everything one could want. Definitely want to go back someday. Not sold on living there just yet, maybe once I get better at Japanese tho...

Sad day. I want to see Rust 4 Linux succeed, and losing a passionate advocate who was willing to call out bad-faith opposition sucks. Can't say I wasn't expecting it tho; one can only deal with that situation + declining support for so long.

Hope he's able to recover from the burnout..

I dug into this because it had been on my mind for a while, and I wanted to get something working before I uploaded Japan trip photos. A couple nights later, I have learned a bit about how Astro & Remark/Rehype & Vite work, and a lot about how what I'm trying to do is futile.

From my understanding, this is how Astro optimizes local images in Markdown files:

1. remark-collect-images.ts, a Markdown AST transformation step, finds all the image paths that look like ![]() and not <img src="" />, so we only optimize the Markdown images. It records this information into context that's passed to the next step.
2. rehype-images.ts, an HTML AST transform step, turns all those images that look like <img src="./local.png" />¹ and turns them into something that looks like <img __ASTRO_IMAGE_="{&#x22;src&#x22;:&#x22;./local.png&#x22;,&#x22;index&#x22;:0}" />. It does this because it's easier to look for, and safer to parse later compared to raw HTML.
3. vite-plugin-markdown/images.ts takes the list of local image URLs (generated as part of transforming the Markdown into HTML in the above steps), and the HTML with these __ASTRO_IMAGE_ things in them, and generates a full-on Astro component, Javascript source & everything, to be rendered later.

Phew, that's a lot!! Hope you're still following along :)

So, the procedure I described works perfectly fine for standalone .md pages. The problem comes when trying to apply this same transformation to markdown files rendered as part of a content collection, which is what my blog files are stored as. Instead of being rendered as part of the "main" Astro flow, content collection entries are rendered separately, as part of a loader, like content/loaders/glob.ts. For Markdown, this calls into vite-plugin-markdown/content-entry-type.ts, which does no such image replacement, because content entries aren't in the correct stage of the build.

And I don't think we can get to the correct stage here! The whole reason this code does the __ASTRO_IMAGE_ rigamarole is so it can call the image transformation function at the correct time. With content collections, all the rendering happens "too early", so the images can't be transformed. UPDATE 2025-02-15: This is incorrect. The replacement happens in content/runtime.ts, during the actual rendering, which makes sense. I wasn't seeing this before because (a) im bad at looking and (b) it's gated on Astro detecting images in the body, which it does not currently do for remote images.

Alternatively, I could use MDX, which is like Markdown but compiles to Javascript instead of HTML, which does always have the image transformation step, thanks to rehype-image-to-component.ts. However, I do not like MDX's weird character restrictions (c'mon, no <, really??), so I will not use it >:(

At this point, I'm seriously considering moving away from Astro's built-in image optimization service to my own... For now though, I'm giving up 😔

UPDATE 2025-02-13: I have just discovered that Markdoc does exactly what I want it to: provide the ability to use a custom component to override what ![]() means, without limiting Markdown syntax. This is very very good news! Here's hoping those files will still have reasonable build times. Also I should probably make it so my Post composer re-uses the Markdoc config...

UPDATE 2024-02-14: Markdoc also doesn't work for my needs because it doesn't have footnote support :((((( I make heavy use of footnotes in my text and I do not want to change how I write those. The whole point of Markdown is to make it so you don't look like you're formatting your text with tags, and Markdoc ignores that and immediately goes back to demanding you use tags for nonstandard features. It seems I am better served by the remark/rehype ecosystem for my parsing needs, so I will continue to do that.

UPDATE 2024-02-15: I did it! I fixed Astro!! The above information was a tad incorrect, what I wanted to do was possible with a bit of elbow grease, so I put in the work and opened a couple PRs: #13254 & #13256. I also have the necessary patches for astro and @astrojs/markdown-remark in my website directly, so I don't have to wait for them to be upstreamed.

Where Do Numbers Come From?

From looking at an automatically-created CloudWatch (AWS's logging/metrics collection service) dashboard, I could see that CloudFront (AWS's CDN service) was already collecting... something...

Great! So my website is getting hits! But on what pages? That was the question I wanted answered, and this dashboard would not or could not answer it.

Looking further, I found documentation for how to turn on CloudFront access logs, which would contain the information I wanted in the cs-uri-stem field, among others. The data in CloudWatch currently was just "metrics", not "logs", though. If I wanted to put logs in CloudWatch, it would be... some unknown amount amount of money, based on usage. Given an example on the CloudWatch pricing page, I estimated about $1/mo.

Now that seems pretty reasonable, no? But currently, I pay just $0.10/mo for S3 storage, and nothing for anything else, so logs alone would 10x my costs :(. I am ruthless about making sure I pay as little as possible for the cloud services I use, so this was a non-starter.

Fortunately, there is another option: CloudFront can put its access logs directly into an S3 bucket, which I could analyze & rotate at my own convenience! Except... how?

Where Do Numbers Go?

Initially, I thought "oh, I can just write a hacky Python script to download & parse them all". But then the thought of writing a hacky Python script seemed really unappealing to me, so I lazyweb-ed on Mastodon to see if anyone knew of any pre-built solutions to parse CloudFront logs. Fortunately someone responded (unlobito) suggesting a program called GoAccess¹ that looked pretty good. Now, all I had to do was write some hacky Python that calls the following shell script!

#!/usr/bin/env bash

# Make it so if any individual step fails, the script exits early, among other things
set -euo pipefail
# Make it so ** acts as a recursive glob (not default on certain bash-es)
shopt -s globstar

if [[ -z "${AWS_BUCKET_NAME+x}" ]]; then
  >&2 echo "Must get bucket passed as AWS_BUCKET_NAME"
  exit 1
fi

output_folder="/tmp/${AWS_BUCKET_NAME}"
mkdir -p $output_folder
output_file="/tmp/${AWS_BUCKET_NAME}.html"
touch $output_file

# Download all logs from S3. Must also receive environment variables that allow this to work
aws s3 sync --delete "s3://${AWS_BUCKET_NAME}" "${output_folder}" 1>&2

# Generate the report with GoAccess running inside Docker
zcat "${output_folder}/**/*.gz" | docker run --rm -i -v "${output_file}:/report.html" -e LANG=$LANG allinurl/goaccess -a -o report.html --log-format CLOUDFRONT - 1>&2

echo "${output_file}"