<?xml version="1.0" encoding="utf-8"?>
    <feed xmlns="http://www.w3.org/2005/Atom">
      <id>https://wolfgirl.dev/blog/</id>
      <title>PolyWolf's Blog - #python</title>
      <subtitle
        >all posts tagged "python" on a blog written by PolyWolf</subtitle
      >
      <icon>/apple-touch-icon.png</icon>
      <link href="/tags/python/rss.xml" rel="self" />
      <link href="https://wolfgirl.dev/blog/" />
      <updated>2025-05-15T11:05:37Z</updated>
      
    <entry>
      <title><![CDATA[Packaging Multiple Dependent Python Modules Using Hatch & Nix]]></title>
      <link rel="alternate" href="https://wolfgirl.dev/blog/2025-05-15-packaging-multiple-dependent-python-modules-using-hatch-nix/" />
      <id>https://wolfgirl.dev/blog/2025-05-15-packaging-multiple-dependent-python-modules-using-hatch-nix/</id>
      <published>2025-05-15T11:05:37Z</published>
      <updated>2025-05-15T11:05:37Z</updated>
      <summary><![CDATA[Alternative title: "Overcomplicating My Website Backend Using Hatch & Nix". For the life for me, I could not figure out how to string all...]]></summary>
      <content type="html"><![CDATA[<p>Alternative title: "Overcomplicating My Website Backend Using Hatch & Nix". For the life for me, I could not figure out how to string all of this together until after pouring over the <a href=https://ryantm.github.io/nixpkgs/languages-frameworks/python/>nixpkgs manual</a> & <a href=https://nixos.wiki/wiki/Packaging/Python>NixOS Wiki</a> about 20 times each; no other resource applied cleanly to my situation. Here's hoping these instructions will help someone else going thru the same thing I did.</p><h2 id=heading-basic-setup>Basic Setup<a aria-label="Link to heading 'Basic Setup'" data-heading-content="Basic Setup" class=anchor href=#basic-setup></a></h2><p>Say we have two Python applications that depend on a third, shared Python library. That is, something like the following:</p><pre class=syntax-highlighting><code class=language-python><span class=comment># shared/__init__.py</span>
<span class=constant>NAME</span> <span class=operator>=</span> <span class=string>"PolyWolf"</span></code></pre><pre class=syntax-highlighting><code class=language-python><span class=comment># hello/__main__.py</span>
<span class=keyword>from</span> <span class=variable>shared</span> <span class=keyword>import</span> <span class=constant>NAME</span>
<span class=keyword>if</span> <span class=variable>__name__</span> <span class=operator>==</span> <span class=string>"__main__"</span>:
    <span class=function>print</span>(<span class=string>f"Hello, </span><span class=punctuation>{</span><span class=constant>NAME</span><span class=punctuation>}</span><span class=string>"</span>)</code></pre><pre class=syntax-highlighting><code class=language-python><span class=comment># goodbye/__main__.py</span>
<span class=keyword>from</span> <span class=variable>shared</span> <span class=keyword>import</span> <span class=constant>NAME</span>
<span class=keyword>if</span> <span class=variable>__name__</span> <span class=operator>==</span> <span class=string>"__main__"</span>:
    <span class=function>print</span>(<span class=string>f"Goodbye, </span><span class=punctuation>{</span><span class=constant>NAME</span><span class=punctuation>}</span><span class=string>"</span>)</code></pre><p>Then, we can run them like so:</p><pre class=syntax-highlighting><code>$ nix-shell -p python3 tree
fetching path input 'path:/nix/store/1dwky0bis4bkl3qngsc6pmq902swa9b6-source'

[nix-shell:/tmp/blogpost]$ tree
.
├── goodbye
│   └── __main__.py
├── hello
│   └── __main__.py
└── shared
    └── __init__.py

4 directories, 3 files

[nix-shell:/tmp/blogpost]$ python3 -m hello
Hello, PolyWolf

[nix-shell:/tmp/blogpost]$ python3 -m goodbye
Goodbye, PolyWolf
</code></pre><p>What the programs actually do doesn't matter all too much; what matters is the module structure, and the directory structure. We want <code>shared</code> to be, well, <em>shared</em> between the <code>hello</code> & <code>goodbye</code> programs, and for no one program to be the obvious one to contain all the shared code. This represents how my <a href=https://github.com/p0lyw0lf/crossposter>crossposter</a> is structured, and is representative of many other "monorepo"-style applications too<sup class=footnote-ref><a data-footnote-ref href=#fn-python-vs-js id=fnref-python-vs-js>1</a></sup>.</p><h3 id=heading-pythonpath-and-you><code>PYTHONPATH</code> And You<a aria-label="Link to heading 'PYTHONPATH And You'" data-heading-content="PYTHONPATH And You" class=anchor href=#pythonpath-and-you></a></h3><p>As a slight aside, there's a specific reason we ran the programs with the <code>-m</code> flag earlier. To see this, let's try running without it:</p><pre class=syntax-highlighting><code>[nix-shell:/tmp/blogpost]$ python3 hello
Traceback (most recent call last):
  File "&lt;frozen runpy>", line 198, in _run_module_as_main
  File "&lt;frozen runpy>", line 88, in _run_code
  File "/tmp/blogpost/hello/__main__.py", line 1, in &lt;module>
    from shared import NAME
ModuleNotFoundError: No module named 'shared'
</code></pre><p>Oh no the dreaded <code>ModuleNotFoundError</code>!! Reading around a bit, we learn the module search path is stored in the <code>sys.path</code> variable, and learn about <a href=https://docs.python.org/3/library/sys_path_init.html>its initialization procedure</a>:</p><blockquote><p>The first entry in the module search path is the directory that contains the input script, if there is one. Otherwise, the first entry is the current directory, which is the case when executing the interactive shell, a -c command, or -m module.</p></blockquote><p>Aha! So that's why: In the <code>-m</code> example, the <code>/tmp/blogpost</code> directory was added to the module path, because there was no "input script", just a module to be run. But without <code>-m</code>, the <code>/tmp/blogpost/hello</code> directory was added, because that's where <code>__main__.py</code> lives, and that directory does <em>not</em> contain a <code>shared</code> module. Is there a workaround? Fortunately, yes, in the next paragraph:</p><blockquote><p>The <code>PYTHONPATH</code> environment variable is often used to add directories to the search path. If this environment variable is found then the contents are added to the module search path.</p></blockquote><p>So! If we really wanted, we could do something like:</p><pre class=syntax-highlighting><code>[nix-shell:/tmp/blogpost]$ PYTHONPATH="$(pwd):$PYTHONPATH" python3 hello
Hello, PolyWolf
</code></pre><p>But honestly, I just use <code>-m</code> :) Anywho, this diversion into <code>PYTHONPATH</code> & module resolution wasn't too relevant, still useful to understand, back to the main content now :)</p><h2 id=heading-transitioning-to-hatch>Transitioning To Hatch<a aria-label="Link to heading 'Transitioning To Hatch'" data-heading-content="Transitioning To Hatch" class=anchor href=#transitioning-to-hatch></a></h2><p>You may have noticed something about our basic setup: it doesn't create a virtual environment :( This isn't a problem for our extremely simple <code>hello</code> and <code>goodbye</code> programs, but as soon as we need a single dependency outside the standard library, we'll be in trouble.</p><p>For the longest time, I just stuck with the ol' reliable <code>python3 -m venv .venv && source .venv/bin/activate && pip3 -r requirements.txt</code>; <a href=https://docs.python.org/3/library/venv.html>venv</a> is included in the standard library, so it'll Just Work™️ on any Python installation >= 3.3, which is to say, "all of them". There are a bazillion other tools to manage Python virtual environments, like <a href=https://virtualenv.pypa.io/en/latest/user_guide.html>virtualenv</a>, <a href=https://pipenv.pypa.io/en/latest/>pipenv</a>, <a href=https://www.anaconda.com/docs/getting-started/miniconda/main>conda</a>, <a href=https://docs.astral.sh/uv/>uv</a>, & more, but I have never needed their full power, hence, ol' reliable.</p><p>Now, however, with the goal of "creating packages to be built with Nix", we might want something a bit more powerful, something capable of creating Python packages. The modern way to do this is with the <a href=https://packaging.python.org/en/latest/guides/writing-pyproject-toml/>pyproject.toml</a> format, whose guide recommends <a href=https://hatch.pypa.io/latest/>Hatch</a> as the default build backend, so that's what I'll be using for the rest of this "tutorial".</p><p>First, let's make some more directories, and move some files around. Why we're doing this will become apparent shortly:</p><pre class=syntax-highlighting><code>[nix-shell:/tmp/blogpost]$ for lib in shared hello goodbye; do
> mkdir -p $lib/src/$lib
> mv $lib/*.py $lib/src/$lib
> done
</code></pre><p>Then, let's define our pyproject.toml for these new folders:</p><pre class=syntax-highlighting><code class=language-toml><span class=comment># shared/pyproject.toml</span>
<span class=punctuation>[</span><span class=property>build-system</span><span class=punctuation>]</span>
<span class=property>requires</span> <span class=operator>=</span> <span class=punctuation>[</span><span class=string>"hatchling >= 1.26"</span><span class=punctuation>]</span>
<span class=property>build-backend</span> <span class=operator>=</span> <span class=string>"hatchling.build"</span>

<span class=punctuation>[</span><span class=property>project</span><span class=punctuation>]</span>
<span class=property>name</span> <span class=operator>=</span> <span class=string>"shared"</span>
<span class=property>version</span> <span class=operator>=</span> <span class=string>"0.1.0"</span></code></pre><pre class=syntax-highlighting><code class=language-toml><span class=comment># {hello,goodbye}/pyproject.toml</span>
<span class=punctuation>[</span><span class=property>build-system</span><span class=punctuation>]</span>
<span class=property>requires</span> <span class=operator>=</span> <span class=punctuation>[</span><span class=string>"hatchling >= 1.26"</span><span class=punctuation>]</span>
<span class=property>build-backend</span> <span class=operator>=</span> <span class=string>"hatchling.build"</span>

<span class=punctuation>[</span><span class=property>project</span><span class=punctuation>]</span>
<span class=property>name</span> <span class=operator>=</span> <span class=string>"hello"</span> <span class=comment># goodbye/pyproject.toml is identical, save this line</span>
<span class=property>version</span> <span class=operator>=</span> <span class=string>"0.1.0"</span>
<span class=property>dependencies</span> <span class=operator>=</span> <span class=punctuation>[</span>
  <span class=string>"shared @ {root:parent:uri}/shared"</span><span class=punctuation>,</span> <span class=comment># ":parent" is equivalent to "../", and can be stacked as many times as required</span>
<span class=punctuation>]</span>

<span class=punctuation>[</span><span class=property>tool</span><span class=punctuation>.</span><span class=property>hatch</span><span class=punctuation>.</span><span class=property>metadata</span><span class=punctuation>]</span>
<span class=property>allow-direct-references</span> <span class=operator>=</span> <span class=constant>true</span></code></pre><p>We end up with a directory structure that looks like this:</p><pre class=syntax-highlighting><code>[nix-shell:/tmp/blogpost]$ tree
.
├── goodbye
│   ├── pyproject.toml
│   └── src
│       └── goodbye
│           └── __main__.py
├── hello
│   ├── pyproject.toml
│   └── src
│       └── hello
│           └── __main__.py
└── shared
    ├── pyproject.toml
    └── src
        └── shared
            └── __init__.py

10 directories, 6 files
</code></pre><p>We've created 3 separate "projects", and then put our files within each project in such a way that Hatch expects to be able to build a package from them.</p><p>So! Let us now create a virtual environment, build a package, and run it, all in one handy command:</p><pre class=syntax-highlighting><code>[nix-shell:/tmp/blogpost]$ exit

$ nix-shell -p hatch tree
fetching path input 'path:/nix/store/1dwky0bis4bkl3qngsc6pmq902swa9b6-source'

[nix-shell:/tmp/blogpost]$ cd hello

[nix-shell:/tmp/blogpost/hello]$ hatch run -- python3 -m hello
  error: subprocess-exited-with-error

  × Preparing editable metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [56 lines of output]
      Traceback (most recent call last):
        File "/home/nixos/.local/share/hatch/env/virtual/hello/DMd9LDvE/hello/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 167, in prepare_metadata_for_build_editable
...
</code></pre><p><em><strong>AAAAAAAAAA</strong></em></p><pre class=syntax-highlighting><code>...
      ValueError: Unable to determine which files to ship inside the wheel using the following heuristics: https://hatch.pypa.io/latest/plugins/builder/wheel/#default-file-selection

      The most likely cause of this is that there is no directory that matches the name of your project (hello).

      At least one file selection option must be defined in the `tool.hatch.build.targets.wheel` table, see: https://hatch.pypa.io/latest/config/build/

      As an example, if you intend to ship a directory named `foo` that resides within a `src` directory located at the root of your project, you can define the following:

      [tool.hatch.build.targets.wheel]
      packages = ["src/foo"]
      [end of output]
...
</code></pre><p>aaaaa oh wait ok I think I know why this is happening: we forgot to provide an empty <code>__init__.py</code> file to appease the Python Gods. The Python Gods are displeased when a folder acts as a module & does not contain an <code>__init__.py</code> file, and we only avoided their wrath earlier because our earthly machinations provided no threat to their heavenly realm; now that we have dared snatch the fire that is "build tool that manages a virtual environment" from their palace, the only way to placate them is to fix our directory structure, <em>not</em> to torture our config file like their oracles suggested. Allow me to demonstrate:</p><pre class=syntax-highlighting><code>[nix-shell:/tmp/blogpost]$ for lib in hello goodbye; do touch $lib/src/$lib/__init__.py; done

[nix-shell:/tmp/blogpost]$ tree
.
├── goodbye
│   ├── pyproject.toml
│   └── src
│       └── goodbye
│           ├── __init__.py
│           └── __main__.py
├── hello
│   ├── pyproject.toml
│   └── src
│       └── hello
│           ├── __init__.py
│           └── __main__.py
└── shared
    ├── pyproject.toml
    └── src
        └── shared
            └── __init__.py

10 directories, 8 files
</code></pre><p>And now, we should surely be able to run:</p><pre class=syntax-highlighting><code>[nix-shell:/tmp/blogpost/hello]$ hatch run -- python3 -m hello
/home/nixos/.local/share/hatch/env/virtual/hello/DMd9LDvE/hello/bin/python3: No module named hello
</code></pre><p>Ah, right, stale environment:</p><pre class=syntax-highlighting><code>[nix-shell:/tmp/blogpost/hello]$ hatch env prune

[nix-shell:/tmp/blogpost/hello]$ hatch run -- python3 -m hello
Hello, PolyWolf
</code></pre><p>Big success!!<sup class=footnote-ref><a data-footnote-ref href=#fn-errors id=fnref-errors>2</a></sup> 🐍, amirite? :P</p><h2 id=heading-time-to-nix-it-up>Time To Nix It Up<a aria-label="Link to heading 'Time To Nix It Up'" data-heading-content="Time To Nix It Up" class=anchor href=#time-to-nix-it-up></a></h2><p>Referencing <a href=https://hachyderm.io/@leftpaddotpy>Jade</a>'s excellent resource on <a href=https://jade.fyi/blog/flakes-arent-real/>using Nix flakes the non-flake way</a>, let's set up a simple devshell<sup class=footnote-ref><a data-footnote-ref href=#fn-more-nix-shells id=fnref-more-nix-shells>3</a></sup>:</p><pre class=syntax-highlighting><code class=language-nix><span class=comment># flake.nix</span>
<span class=punctuation>{</span>
  <span class=variable>inputs</span> <span class=punctuation>=</span> <span class=punctuation>{</span>
    <span class=variable>nixpkgs</span><span class=punctuation>.</span><span class=variable>url</span> <span class=punctuation>=</span> <span class=string>"github:NixOS/nixpkgs/nixos-unstable"</span><span class=punctuation>;</span>
    <span class=variable>flake-utils</span><span class=punctuation>.</span><span class=variable>url</span> <span class=punctuation>=</span> <span class=string>"github:numtide/flake-utils"</span><span class=punctuation>;</span>
  <span class=punctuation>};</span>

  <span class=variable>outputs</span> <span class=punctuation>=</span> <span class=punctuation>{</span> <span class=variable>self</span><span class=punctuation>,</span> <span class=variable>nixpkgs</span><span class=punctuation>,</span> <span class=variable>flake-utils</span> <span class=punctuation>}</span>: <span class=variable>flake-utils</span><span class=punctuation>.</span><span class=variable>lib</span><span class=punctuation>.</span><span class=variable>eachDefaultSystem</span> <span class=punctuation>(</span><span class=variable>system</span>:
    <span class=keyword>let</span>
      <span class=variable>pkgs</span> <span class=punctuation>=</span> <span class=variable>import</span> <span class=variable>nixpkgs</span> <span class=punctuation>{</span> <span class=keyword>inherit</span> <span class=variable>system</span><span class=punctuation>;</span> <span class=punctuation>};</span>
    <span class=keyword>in</span>
      <span class=punctuation>{</span>
        <span class=variable>devShells</span><span class=punctuation>.</span><span class=variable>default</span> <span class=punctuation>=</span> <span class=variable>pkgs</span><span class=punctuation>.</span><span class=variable>callPackage</span> <span class=string>./shell.nix</span> <span class=punctuation>{</span> <span class=punctuation>};</span>
      <span class=punctuation>}</span>
  <span class=punctuation>);</span>
<span class=punctuation>}</span></code></pre><pre class=syntax-highlighting><code class=language-nix><span class=comment># shell.nix</span>
<span class=punctuation>{</span> <span class=variable>pkgs</span> <span class=punctuation>?</span> <span class=variable>import</span> <span class=string>&lt;nixpkgs></span> <span class=punctuation>{</span> <span class=punctuation>}</span> <span class=punctuation>}</span>:
<span class=variable>pkgs</span><span class=punctuation>.</span><span class=variable>mkShell</span> <span class=punctuation>{</span>
  <span class=variable>packages</span> <span class=punctuation>=</span> <span class=punctuation>(</span>
    <span class=keyword>with</span> <span class=variable>pkgs</span><span class=punctuation>;</span>
    <span class=punctuation>[</span>
      <span class=comment># The python virtual environments themselves are managed with hatch, no need for any other dependencies in this file! Unless you want to put like, language server stuff here ofc</span>
      <span class=variable>hatch</span>
    <span class=punctuation>]</span>
  <span class=punctuation>);</span>
<span class=punctuation>}</span></code></pre><p>And for this next part, it would be very useful for to do a bit of reading on Nix before continuing; I recommend the nix.dev tutorials on <a href=https://nix.dev/tutorials/callpackage><code>callPackage</code></a> + <a href=https://nix.dev/tutorials/working-with-local-files>local files</a>, the NixOS Wiki page on <a href=https://wiki.nixos.org/wiki/Overlays>Overlays</a>, & all the basic Nix syntax ones. Otherwise, this will just be blind copy-pasting:</p><pre class=syntax-highlighting><code class=language-nix><span class=comment># shared/package.nix</span>
<span class=punctuation>{</span>
  <span class=variable>lib</span><span class=punctuation>,</span>
  <span class=variable>buildPythonPackage</span><span class=punctuation>,</span>
  <span class=variable>hatchling</span><span class=punctuation>,</span><span class=variable>
</span><span class=punctuation>}</span><span class=variable>:
</span><span class=variable>buildPythonPackage</span><span class=variable> </span><span class=punctuation>{</span><span class=variable>
  </span><span class=variable>pname</span><span class=variable> </span><span class=punctuation>=</span><span class=variable> </span><span class=string>"shared"</span><span class=punctuation>;</span><span class=variable>
  </span><span class=variable>version</span><span class=variable> </span><span class=punctuation>=</span><span class=variable> </span><span class=string>"0.1.0"</span><span class=punctuation>;</span><span class=variable>

  </span><span class=variable>src</span><span class=variable> </span><span class=punctuation>=</span><span class=variable> </span><span class=variable>lib</span><span class=punctuation>.</span><span class=variable>fileset</span><span class=punctuation>.</span><span class=variable>toSource</span><span class=variable> </span><span class=punctuation>{</span><span class=variable>
    </span><span class=variable>root</span><span class=variable> </span><span class=punctuation>=</span><span class=variable> </span><span class=string>./.</span><span class=punctuation>;</span><span class=variable>
    </span><span class=variable>fileset</span><span class=variable> </span><span class=punctuation>=</span><span class=variable> </span><span class=variable>lib</span><span class=punctuation>.</span><span class=variable>fileset</span><span class=punctuation>.</span><span class=variable>gitTracked</span><span class=variable> </span><span class=string>./.</span><span class=punctuation>;</span><span class=variable>
  </span><span class=punctuation>};</span><span class=variable>

  </span><span class=variable>pyproject</span><span class=variable> </span><span class=punctuation>=</span><span class=variable> </span><span class=variable>true</span><span class=punctuation>;</span><span class=variable>
  </span><span class=variable>build-system</span><span class=variable> </span><span class=punctuation>=</span><span class=variable> </span><span class=punctuation>[</span><span class=variable> </span><span class=variable>hatchling</span><span class=variable> </span><span class=punctuation>];</span><span class=variable>
</span><span class=punctuation>}</span></code></pre><pre class=syntax-highlighting><code class=language-nix><span class=comment># {hello,goodbye}/package.nix</span>
<span class=punctuation>{</span>
  <span class=variable>lib</span><span class=punctuation>,</span>
  <span class=variable>buildPythonPackage</span><span class=punctuation>,</span>
  <span class=variable>hatchling</span><span class=punctuation>,</span>
  <span class=variable>myproject-shared</span><span class=punctuation>,</span> <span class=comment># Here, we take the package defined by shared/package.nix as a dependency, to be hooked up later</span><span class=variable>
</span><span class=punctuation>}</span><span class=variable>:
</span><span class=variable>buildPythonPackage</span><span class=variable> </span><span class=punctuation>{</span><span class=variable>
  </span><span class=variable>pname</span><span class=variable> </span><span class=punctuation>=</span><span class=variable> </span><span class=string>"hello"</span><span class=punctuation>;</span><span class=variable> </span><span class=comment># goodbye/package.nix is identical, save this line</span><span class=variable>
  </span><span class=variable>version</span><span class=variable> </span><span class=punctuation>=</span><span class=variable> </span><span class=string>"0.1.0"</span><span class=punctuation>;</span><span class=variable>

  </span><span class=variable>src</span><span class=variable> </span><span class=punctuation>=</span><span class=variable> </span><span class=variable>lib</span><span class=punctuation>.</span><span class=variable>fileset</span><span class=punctuation>.</span><span class=variable>toSource</span><span class=variable> </span><span class=punctuation>{</span><span class=variable>
    </span><span class=variable>root</span><span class=variable> </span><span class=punctuation>=</span><span class=variable> </span><span class=string>./.</span><span class=punctuation>;</span><span class=variable>
    </span><span class=variable>fileset</span><span class=variable> </span><span class=punctuation>=</span><span class=variable> </span><span class=variable>lib</span><span class=punctuation>.</span><span class=variable>fileset</span><span class=punctuation>.</span><span class=variable>gitTracked</span><span class=variable> </span><span class=string>./.</span><span class=punctuation>;</span><span class=variable>
  </span><span class=punctuation>};</span><span class=variable>

  </span><span class=variable>pyproject</span><span class=variable> </span><span class=punctuation>=</span><span class=variable> </span><span class=variable>true</span><span class=punctuation>;</span><span class=variable>
  </span><span class=variable>build-system</span><span class=variable> </span><span class=punctuation>=</span><span class=variable> </span><span class=punctuation>[</span><span class=variable> </span><span class=variable>hatchling</span><span class=variable> </span><span class=punctuation>];</span><span class=variable>

  </span><span class=variable>propagatedBuildInputs</span><span class=variable> </span><span class=punctuation>=</span><span class=variable> </span><span class=punctuation>[</span><span class=variable> </span><span class=variable>myproject-shared</span><span class=variable> </span><span class=punctuation>];</span><span class=variable> </span><span class=comment># Here, we declare how exactly we depend on the package defined by shared/package.nix; we'll get into what this means later</span><span class=variable>
</span><span class=punctuation>}</span></code></pre><p>Whoa that's it!! For the Nixpkgs-style files, at least. Yeah, turns out using Hatch took care of most of the hard part of packaging, the rest is built-in to Nixpkgs itself. We do still have to repeat the dependencies of each project outside of the pyproject.toml<sup class=footnote-ref><a data-footnote-ref href=#fn-repeat-dependencies id=fnref-repeat-dependencies>4</a></sup>, which is a bummer, but oh well the rest of the file is quite small which is good, thank u Nixpkgs</p><p>With all these files in place, our tree now looks like:</p><pre class=syntax-highlighting><code>[nix-shell:/tmp/blogpost]$ tree
.
├── flake.lock
├── flake.nix
├── goodbye
│   ├── package.nix
│   ├── pyproject.toml
│   └── src
│       └── goodbye
│           ├── __init__.py
│           └── __main__.py
├── hello
│   ├── package.nix
│   ├── pyproject.toml
│   └── src
│       └── hello
│           ├── __init__.py
│           └── __main__.py
├── shared
│   ├── package.nix
│   ├── pyproject.toml
│   └── src
│       └── shared
│           └── __init__.py
└── shell.nix

10 directories, 14 files
</code></pre><p>Next, we have to wire all these packages up inside our <code>flake.nix</code> so we can actually build them. These are fairly unusual packages, turns out; they use a top-level <code>buildPythonPackage</code> variable that doesn't exist in a normal Nixpkgs <code>callPackage</code>. This aligns with the <a href=https://ryantm.github.io/nixpkgs/languages-frameworks/python/>Nixpkgs manual</a>: Python packages are special, because they could be built for multiple versions of the interpreter, and this is the easiest way to do that without having to do something even crazier w/ overrides.</p><p>Anyways, here's our updated <code>flake.nix</code>, then an explanation:</p><pre class=syntax-highlighting><code class=language-nix><span class=comment># flake.nix</span>
<span class=punctuation>{</span>
  <span class=variable>inputs</span> <span class=punctuation>=</span> <span class=punctuation>{</span>
    <span class=variable>nixpkgs</span><span class=punctuation>.</span><span class=variable>url</span> <span class=punctuation>=</span> <span class=string>"github:NixOS/nixpkgs/nixos-unstable"</span><span class=punctuation>;</span>
    <span class=variable>flake-utils</span><span class=punctuation>.</span><span class=variable>url</span> <span class=punctuation>=</span> <span class=string>"github:numtide/flake-utils"</span><span class=punctuation>;</span>
  <span class=punctuation>};</span>

  <span class=variable>outputs</span> <span class=punctuation>=</span> <span class=punctuation>{</span> <span class=variable>self</span><span class=punctuation>,</span> <span class=variable>nixpkgs</span><span class=punctuation>,</span> <span class=variable>flake-utils</span> <span class=punctuation>}</span>: <span class=variable>flake-utils</span><span class=punctuation>.</span><span class=variable>lib</span><span class=punctuation>.</span><span class=variable>eachDefaultSystem</span> <span class=punctuation>(</span><span class=variable>system</span>:
    <span class=keyword>let</span>
      <span class=variable>pkgs</span> <span class=punctuation>=</span> <span class=variable>import</span> <span class=variable>nixpkgs</span> <span class=punctuation>{</span> <span class=keyword>inherit</span> <span class=variable>system</span><span class=punctuation>;</span> <span class=punctuation>};</span>
      <span class=variable>python3</span> <span class=punctuation>=</span> <span class=variable>pkgs</span><span class=punctuation>.</span><span class=variable>python3</span><span class=punctuation>.</span><span class=variable>override</span> <span class=punctuation>{</span>
        <span class=variable>packageOverrides</span> <span class=punctuation>=</span> <span class=variable>final</span>: <span class=variable>prev</span>: <span class=punctuation>{</span>
          <span class=variable>myproject-shared</span> <span class=punctuation>=</span> <span class=variable>final</span><span class=punctuation>.</span><span class=variable>callPackage</span> <span class=string>./shared/package.nix</span> <span class=punctuation>{</span> <span class=punctuation>};</span>
          <span class=variable>myproject-hello</span> <span class=punctuation>=</span> <span class=variable>final</span><span class=punctuation>.</span><span class=variable>callPackage</span> <span class=string>./hello/package.nix</span> <span class=punctuation>{</span>
            <span class=variable>myproject-shared</span> <span class=punctuation>=</span> <span class=variable>final</span><span class=punctuation>.</span><span class=variable>myproject-shared</span><span class=punctuation>;</span>
          <span class=punctuation>};</span>
          <span class=variable>myproject-goodbye</span> <span class=punctuation>=</span> <span class=variable>final</span><span class=punctuation>.</span><span class=variable>callPackage</span> <span class=string>./goodbye/package.nix</span> <span class=punctuation>{</span>
            <span class=variable>myproject-shared</span> <span class=punctuation>=</span> <span class=variable>final</span><span class=punctuation>.</span><span class=variable>myproject-shared</span><span class=punctuation>;</span>
          <span class=punctuation>};</span>
        <span class=punctuation>};</span>
      <span class=punctuation>};</span>
    <span class=keyword>in</span>
    <span class=punctuation>{</span>
      <span class=variable>packages</span> <span class=punctuation>=</span> <span class=punctuation>{</span> <span class=keyword>inherit</span> <span class=variable>python3</span><span class=punctuation>;</span> <span class=punctuation>};</span>
      <span class=variable>devShells</span><span class=punctuation>.</span><span class=variable>default</span> <span class=punctuation>=</span> <span class=variable>pkgs</span><span class=punctuation>.</span><span class=variable>callPackage</span> <span class=string>./shell.nix</span> <span class=punctuation>{</span> <span class=punctuation>};</span>
    <span class=punctuation>}</span>
  <span class=punctuation>);</span>
<span class=punctuation>}</span></code></pre><p>Oh wow that's dense! Let's split it up a bit:</p><pre class=syntax-highlighting><code class=language-nix><span class=keyword>let</span>
  <span class=variable>python3</span> <span class=punctuation>=</span> <span class=variable>pkgs</span><span class=punctuation>.</span><span class=variable>python3</span><span class=punctuation>.</span><span class=variable>override</span> <span class=punctuation>{</span>
    <span class=comment># ...</span>
  <span class=punctuation>};</span>
<span class=keyword>in</span>
<span class=punctuation>{</span>
  <span class=variable>packages</span> <span class=punctuation>=</span> <span class=punctuation>{</span> <span class=keyword>inherit</span> <span class=variable>python3</span><span class=punctuation>;</span> <span class=punctuation>};</span>
<span class=punctuation>}</span></code></pre><p>This declares a new <code>python3</code> package in our Flake, defined as an override of the base <code>python3</code> package, whose final <code>pkgs</code> attribute will contain our custom packages. And how do we declare those? With the <a href=https://nixos.wiki/wiki/Overlays>overlay</a> pattern!</p><pre class=syntax-highlighting><code class=language-nix><span class=variable>packageOverrides</span> <span class=punctuation>=</span> <span class=variable>final</span>: <span class=variable>prev</span>: <span class=punctuation>{</span>
  <span class=variable>myproject-shared</span> <span class=punctuation>=</span> <span class=variable>final</span><span class=punctuation>.</span><span class=variable>callPackage</span> <span class=string>./shared/package.nix</span> <span class=punctuation>{</span> <span class=punctuation>};</span>
  <span class=variable>myproject-hello</span> <span class=punctuation>=</span> <span class=variable>final</span><span class=punctuation>.</span><span class=variable>callPackage</span> <span class=string>./hello/package.nix</span> <span class=punctuation>{</span>
    <span class=variable>myproject-shared</span> <span class=punctuation>=</span> <span class=variable>final</span><span class=punctuation>.</span><span class=variable>myproject-shared</span><span class=punctuation>;</span>
  <span class=punctuation>};</span>
  <span class=variable>myproject-goodbye</span> <span class=punctuation>=</span> <span class=variable>final</span><span class=punctuation>.</span><span class=variable>callPackage</span> <span class=string>./goodbye/package.nix</span> <span class=punctuation>{</span>
    <span class=variable>myproject-shared</span> <span class=punctuation>=</span> <span class=variable>final</span><span class=punctuation>.</span><span class=variable>myproject-shared</span><span class=punctuation>;</span>
  <span class=punctuation>};</span>
<span class=punctuation>};</span></code></pre><p>Each package that we want to add gets <code>callPackage</code>-ed<sup class=footnote-ref><a data-footnote-ref href=#fn-final-callPackage id=fnref-final-callPackage>5</a></sup>, with its dependencies injected from the "final" resolved set of packages. This is just the Nix(pkgs)-y way of doing dependency tracking such that any dependency can be overridden at arbitrarily deep levels. Fortunately, we're just adding new packages, so we don't have to think <em>too</em> hard about that complexity here.</p><p>With that all in place, let's try to build one of our packages:</p><pre class=syntax-highlighting><code>[nix-shell:/tmp/blogpost]$ nix build .#python3.pkgs.myproject-hello

[nix-shell:/tmp/blogpost]$ ls result
lib  nix-support

[nix-shell:/tmp/blogpost]$ ls result/lib/python3*/site-packages
hello  hello-0.1.0.dist-info
</code></pre><p>Huh! We might have expected a <code>shared</code> module to appear here too, given we declared it as a dependency. I guess it doesn't do those just yet... Maybe it'll be complete in the main package?</p><pre class=syntax-highlighting><code>[nix-shell:/tmp/blogpost]$ nix build .#python3

[nix-shell:/tmp/blogpost]$ ls result
bin  include  lib  nix-support  share

[nix-shell:/tmp/blogpost]$ cd result/bin

[nix-shell:/tmp/blogpost/result/bin]$ ./python3 -m hello
/nix/store/kjvgj2n3yn70hmjifg6y0bk9m4rf7jba-python3-3.12.10/bin/python3: No module named hello
</code></pre><p>No, that works even less! I guess that makes sense, it's not like the default installation of python3 comes with <em>every Python package ever built</em> by default, we'll need to do something to explicitly install those somehow. But this begs the question: how <em>do</em> Nixpkgs Python dependencies work, anyways?</p><h3 id=heading-she-propagate-on-my-build-input-till-i-fixed-point>She Propagate On My Build Input Till I Fixed Point<a aria-label="Link to heading 'She Propagate On My Build Input Till I Fixed Point'" data-heading-content="She Propagate On My Build Input Till I Fixed Point" class=anchor href=#she-propagate-on-my-build-input-till-i-fixed-point></a></h3><p>Referencing <a href=https://ryantm.github.io/nixpkgs/languages-frameworks/python/#buildpythonpackage-parameters>the reference</a>:</p><blockquote><ul><li><code>nativeBuildInputs ? []</code>: Build-time only dependencies. Typically executables as well as the items listed in <code>setup_requires</code>.</li><li><code>buildInputs ? []</code>: Build and/or run-time dependencies that need to be compiled for the host machine. Typically non-Python libraries which are being linked.</li><li><code>nativeCheckInputs ? []</code>: Dependencies needed for running the <code>checkPhase</code>. These are added to <code>nativeBuildInputs</code> when <code>doCheck = true</code>. Items listed in <code>tests_require</code> go here.</li><li><code>propagatedBuildInputs ? []</code>: Aside from propagating dependencies, <code>buildPythonPackage</code> also injects code into and wraps executables with the paths included in this list. Items listed in <code>install_requires</code> go here.</li></ul></blockquote><p>And summarizing in simpler English: use <code>propagatedBuildInputs</code> for Python library dependencies. Okie dokie! I'm sure there are various reasons for this, but we did that earlier so we're on the right track.</p><p>But wait a minute, doing a normal <code>nix build</code> did <em>not</em> propagate this build input into the <code>result</code> directory. What gives? This confused me for a long while<sup class=footnote-ref><a data-footnote-ref href=#fn-nixpkgs-confusing id=fnref-nixpkgs-confusing>6</a></sup>, until I started looking at things from the perspective of a C compiler.</p><p>Basically: Nix was originally meant for packaging C code, so a lot of its semantics make more sense when viewed that way. Packages put in <code>buildInputs</code> will be built, put into <code>/nix/store</code>, their paths exposed in certain environment variables, and then it's up to whatever linker to actually hard-code those paths into the final executable. However, there is no "final executable" at the Python library level, hence why we put the dependencies into <code>propagatedBuildInputs</code> so they'll be available by the time we get around to creating a Python environment. A Python environment can be thought of like an executable in this framework; a script can "link" (literally, create symlinks to) all specified packages into the directory structure Python expects.</p><p>Quite fortunately, our goal of "link a Python package & all of its dependencies into a functioning Python environment" already has a simple solution:</p><pre class=syntax-highlighting><code class=language-nix><span class=comment># flake.nix</span>
<span class=comment># ...</span>
      in
      <span class=punctuation>{</span>
        <span class=variable>packages</span> <span class=punctuation>=</span> <span class=punctuation>{</span>
          <span class=keyword>inherit</span> <span class=variable>python3</span><span class=punctuation>;</span>
          <span class=variable>python3-myproject-hello</span> <span class=punctuation>=</span> <span class=variable>python3</span><span class=punctuation>.</span><span class=variable>withPackages</span> <span class=punctuation>(</span><span class=variable>ps</span>: <span class=punctuation>[</span> <span class=variable>ps</span><span class=punctuation>.</span><span class=variable>myproject-hello</span> <span class=punctuation>]);</span>
          <span class=variable>python3-myproject-goodbye</span> <span class=punctuation>=</span> <span class=variable>python3</span><span class=punctuation>.</span><span class=variable>withPackages</span> <span class=punctuation>(</span><span class=variable>ps</span>: <span class=punctuation>[</span> <span class=variable>ps</span><span class=punctuation>.</span><span class=variable>myproject-goodbye</span> <span class=punctuation>]);</span>
        <span class=punctuation>};</span>
        <span class=variable>devShells</span><span class=punctuation>.</span><span class=variable>default</span> <span class=punctuation>=</span> <span class=variable>pkgs</span><span class=punctuation>.</span><span class=variable>callPackage</span> <span class=string>./shell.nix</span> <span class=punctuation>{</span> <span class=punctuation>};</span>
      <span class=punctuation>}</span>
<span class=comment># ...</span></code></pre><p>This <code>python3.withPackages</code> function does exactly what we need!! Magic. Now, when we build & run:</p><pre class=syntax-highlighting><code>[nix-shell:/tmp/blogpost]$ nix build .#python3-myproject-hello

[nix-shell:/tmp/blogpost]$ cd result/bin

[nix-shell:/tmp/blogpost/result/bin]$ ./python3 -m hello
Hello, PolyWolf
</code></pre><p>It works! Yay!!</p><h2 id=heading-drawbacks>Drawbacks<a aria-label="Link to heading 'Drawbacks'" class=anchor data-heading-content=Drawbacks href=#drawbacks></a></h2><p>Now, by no means do I intend to represent that this approach is the perfect way of doing things with Python + Nix. Far from it, really. Here are some of rough edges I've run into in this workflow that I'd like to eventually sand off:</p><ul><li><code>shared</code>, <code>hello</code>, and <code>goodbye</code> have to live in those strange <code>$lib/src/$lib</code> directories. This is solely because pyproject.toml doesn't have good multi-package support; Nix itself is flexible enough, but I am relying on pyproject.toml to make the Nix packaging easier.</li><li>As mentioned earlier, we have to specify all dependencies twice: once in <code>pyproject.toml</code>, and once in <code>package.nix</code>, which is sad. Also, if there are any version constraints in <code>pyproject.toml</code>, those will effectively break <code>package.nix</code> because Nixpkgs only contains one version of each package.</li><li>If <code>shared</code> ever changes, the hatch environments for <code>hello</code> and <code>goodbye</code> will need to be deleted and re-recreated. Hatch itself seems to have a "development mode" where changed to the current project are reflected in the virtual environment, but it doesn't seem like that applies to this multi-project usecase.</li></ul><p>Still, this is just about good enough for my project, so I think I will stop here. All this Nix packaging is just so I can write a <a href=https://github.com/p0lyw0lf/infrastructure/tree/d430367a2e3f9aa95783314451d1509446ea5e16/nixos>NixOS module</a> and finally start deploying my site the Cool way (<code>nixos-rebuild switch --flake .#devbox --target-host devbox</code>) instead of the Boring way (<code>ssh devbox 'cd crossposter; git pull; systemctl --user restart rc.wolfgirl.dev'</code>). Join me next time for that, I guess?? Assuming I don't get sidetracked on something else first lol</p><section class=footnotes data-footnotes><ol><li id=fn-python-vs-js><p>From what I understand, this is a much more common pattern in the Javascript & Rust worlds, where the multi-module build tooling is actually somewhat decent. No matter, we press onwards regardless of our fate here. <a aria-label="Back to reference 1" class=footnote-backref data-footnote-backref data-footnote-backref-idx=1 href=#fnref-python-vs-js>↩</a></p></li><li id=fn-errors><p>I hope showing these errors is useful, by the way. I know most tutorials just show the happy path, but I think it's also important to show the common error paths too, and ways off of them back onto the happy path. <a aria-label="Back to reference 2" class=footnote-backref data-footnote-backref data-footnote-backref-idx=2 href=#fnref-errors>↩</a></p></li><li id=fn-more-nix-shells><p>For longer reads on Nix devshells that go way too into the details (/pos) I highly recommend fasterthanlime's post series on the <a href=https://fasterthanli.me/series/building-a-rust-service-with-nix/part-9>non-flake way</a> and the <a href=https://fasterthanli.me/series/building-a-rust-service-with-nix/part-10>flake way</a>. These are more targeted at Rust devshells and also maybe slightly out-of-date but that's ok the basics are still very pertinent & Amos is a very good writer. <a aria-label="Back to reference 3" class=footnote-backref data-footnote-backref data-footnote-backref-idx=3 href=#fnref-more-nix-shells>↩</a></p></li><li id=fn-repeat-dependencies><p>I'm sure a sufficiently motivated nix-er could figure out a way to parse the pyproject.toml at evaluation time, but I am not that. Oh well! <a aria-label="Back to reference 4" class=footnote-backref data-footnote-backref data-footnote-backref-idx=4 href=#fnref-repeat-dependencies>↩</a></p></li><li id=fn-final-callPackage><p>Expert Nix users will note the <code>final.callPackage</code> calls are a bit strange, when usually overlays use <code>prev.callPackage</code> for various reasons beyond my ken. For another reason even further beyond my ken, <code>prev.callPackage</code> doesn't exist in this place, so we use <code>final.callPackage</code> instead ya <a aria-label="Back to reference 5" class=footnote-backref data-footnote-backref data-footnote-backref-idx=5 href=#fnref-final-callPackage>↩</a></p></li><li id=fn-nixpkgs-confusing><p>Pro tip: Don't read the <a href=https://nixos.org/manual/nixpkgs/stable/#ssec-stdenv-dependencies-propagated>relevant section of the nixpkgs manual</a> when first learning how dependencies work. Only served to confuse me more tbh <a aria-label="Back to reference 6" class=footnote-backref data-footnote-backref data-footnote-backref-idx=6 href=#fnref-nixpkgs-confusing>↩</a></p></li></ol></section>]]></content>
      <author>
        <name>PolyWolf</name>
        <uri>https://wolf.girl.technology/</uri>
      </author>
    </entry>
  
    <entry>
      <title><![CDATA[Weekend Project Complete]]></title>
      <link rel="alternate" href="https://wolfgirl.dev/blog/3587148/" />
      <id>https://wolfgirl.dev/blog/3587148/</id>
      <published>2023-11-19T15:42:55Z</published>
      <updated>2023-11-19T15:42:55Z</updated>
      
      <content type="html"><![CDATA[<p>Originally published on Cohost.</p><hr><p><a href=https://github.com/p0lyw0lf/crossposter>https://github.com/p0lyw0lf/crossposter</a></p><p>For a while, I've had a tradition of, whenever I would find a cool computer security-related article online, I would post it to a channel in a long-inactive Discord. I joined this Discord in my Freshman year of college, and I've been keeping up the tradition ever since, so I think I'm approaching 4 years of it now? Wild to think about.</p><p>The Problem with this is, my audience is limited. There are like <em>mayyyybe</em> 4 other people watching that channel, likely only 1. Plus, Discord is not a reliable store for information. So at one point, I decided to start backing things up <a href=https://wolfgirl.dev/cybersec/>on my blog</a>. Unfortunately, this was a very manual process, because interacting with Git is a lot harder than just pasting a link and some text into Discord, so the friction meant I very rarely updated that page.</p><p>During the initial Fediverse craze, one of my Solutions was to create an account, <a href=https://infosec.exchange/@PolyWolf>@PolyWolf@infosec.exchange</a>, that would take the place of my Discord channel. Problem was, that somehow ended up having even less of an audience because lol bootstrapping problem<sup class=footnote-ref><a data-footnote-ref href=#fn-1 id=fnref-1>1</a></sup>. I was using that account more for following other tech accounts, than I was for posting myself.</p><p>So finally, while I was talking with the 1 friend who I knows follow the channel, I had an idea: why not write a little bot to automatically crosspost from my cozy Discord channel to these other platforms? That way, I could have the maximum audience with the least amount of effort.</p><p>The only thing stopping me was the good 'ol "I programmed too much at work this week so now I want to enjoy my weekend doing anything else but programming." Fortunately, this week was light, so I found the spoons Saturday, banging thru the full implementation in 8 hours<sup class=footnote-ref><a data-footnote-ref href=#fn-2 id=fnref-2>2</a></sup>, and spent a bit more time this morning setting up a t4g.micro instance on AWS to host it. And bam, it's done!</p><p>I have created a new Discord server for this bot, and will likely post my links there in the future. If this sounds interesting and u want the server invite, DM me @polywolf on Discord. Otherwise, once the new Cohost API rolls out, I'm probably also going to add a crossposting target for it under a new page.</p><p>In summary, I am excited I still have the ability to smash things together and have them work out. That's all for now, thanks for reading!</p><section class=footnotes data-footnotes><ol><li id=fn-1><p>I somehow overcame the bootstrapping problem for the second time in my life ever on Co Host Dot Org thanks to @fullmoon 's initial list from Twitter (now X) and also "being a techy trans furry on the techy trans furry website". My first time was on Scratch Dot MIT Dot EDU thanks to "being a kid making simple and fun games on the simple and fun games made by kids website". <a aria-label="Back to reference 1" class=footnote-backref data-footnote-backref data-footnote-backref-idx=1 href=#fnref-1>↩</a></p></li><li id=fn-2><p>Why Python? For this exact reason. Unparalleled when it comes to gluing web things together. It was my first real programming language and I love it so. Like, i can just do <a href=https://github.com/p0lyw0lf/crossposter/blob/main/poster/template.py>stuff like this</a> and it just works because lol! This repo is definitely not a paragon of good Python style, but it sure is My Python style :) <a aria-label="Back to reference 2" class=footnote-backref data-footnote-backref data-footnote-backref-idx=2 href=#fnref-2>↩</a></p></li></ol></section>]]></content>
      <author>
        <name>PolyWolf</name>
        <uri>https://wolf.girl.technology/</uri>
      </author>
    </entry>
  
    </feed>
  