---
title: Reproducibility in functional package management
date: 2025-4-25
date-format: long
lightbox: true
logo: telecom.png
margin-top: "0px"
author:
  - name:
      given: Julien
      family: Malka
    url: https://luj.fr
    email: julien.malka@telecom-paris.fr
    orcid: 0009-0008-9845-6300
    roles:
      - conceptualization
      - investigation
      - writing – original draft
    affiliations:
      - id: telecom
        name: Télécom Paris, Institut Polytechnique de Paris
fig-align: center
code-overflow: wrap
code-line-numbers: false
css: styles.css

format:
  metropolis-beamer-revealjs:
    theme: slide.scss
    toc: false
    toc-title: Plan
    toc-depth: 2
    slide-level: 2
    slide-number: true
---



## Research topics

**Main topics:** Cybersecurity & Software engineering

*How can one trust the software installed on one’s system is not malicious?* 

- What if we make the assumption that the software is **open source**?


**Research question**: How to increase trust in the Open Source Software Supply Chain with *functional package managers* and *reproducible builds*?





## Functional package managers {auto-animate="true"}


New software deployment model (from which **Nix** has been the first example).


```nix 
{ stdenv, lib, fetchFromGitHub }:
```

## Functional package managers {auto-animate="true"}


New software deployment model (from which **Nix** has been the first example).

```nix 
{ stdenv, lib, fetchFromGitHub }:

stdenv.mkDerivation rec {
   version = "1.3.7";
   pname = "htpdate";

   src = fetchFromGitHub {
      owner = "twekkel";
      repo = pname;
      rev = "v${version}";
      sha256 = "sha256-X7r95Uc4oGB0eVum5D7pC4tebZIyyz73g6Q/D0cjuFM=";
   };

```

## Functional package managers {auto-animate="true"}


New software deployment model (from which **Nix** has been the first example).

```nix 
{ stdenv, lib, fetchFromGitHub }:

stdenv.mkDerivation rec {
   version = "1.3.7";
   pname = "htpdate";

   src = fetchFromGitHub {
      owner = "twekkel";
      repo = pname;
      rev = "v${version}";
      sha256 = "sha256-X7r95Uc4oGB0eVum5D7pC4tebZIyyz73g6Q/D0cjuFM=";
   };

   makeFlags = [
    "prefix=$(out)"
  ];

```

## Functional package managers {auto-animate="true"}

::: {.r-fit-text}

New software deployment model (from which **Nix** has been the first example).

```nix 
{ stdenv, lib, fetchFromGitHub }:

stdenv.mkDerivation rec {
   version = "1.3.7";
   pname = "htpdate";

   src = fetchFromGitHub {
      owner = "twekkel";
      repo = pname;
      rev = "v${version}";
      sha256 = "sha256-X7r95Uc4oGB0eVum5D7pC4tebZIyyz73g6Q/D0cjuFM=";
   };

   makeFlags = [
    "prefix=$(out)"
  ];

  meta = with lib; {
    description = "Utility to fetch time and set the system clock over HTTP";
    platforms = platforms.linux;
    license = licenses.gpl2Plus;
    maintainers = with maintainers; [ julienmalka ];
  };
}

```

:::


## Evaluation->Build pipeline


::: {#fig-eval-build}
![](./eval-build.png){.r-stretch}

Eval-build pipeline

:::


## Functional package managers for SSC security

Functional package managers also have interesting properties for software supply chain security (which are of interest for us):

- Builds from source;
- Sandboxed compilation.



## Functional package managers for SSC security

- Installed packages create a static graph structure (a Merkle tree) that can be analysed in order to find known vulnerability in the dependencies.


::: {#fig-eval-build}
![](./graph.png){.r-stretch}

Example of a package dependency graph

:::

## The binary distribution problem

- It is not always reasonable to compile all the software a user wants to install on their own machine: creates the necessity of binary caches ;
- **But** binary caches make us lose some of the interesting security properties of functional package managers.




## Reproducible builds {.lol}

![](rb.png){height='4em' fig-align="center"}

A build is **reproducible** if given the same source code, build environment and build instructions, any party can recreate bit-by-bit identical copies of all specified artifacts.


## Why is build reproducibility important?

:::: {.columns}
::: {.column width="40%"}
- Reproducible builds can be useful to improve the **security** of the **binary distribution problem**;
- Instead of having to trust a single third party to provide them the correct binary, they can divide the trust between multiple parties. 
:::
::: {.column width="60%"}
::: {#fig-rb}
![](rb-verif.png){height='13em'}

Leveraging reproducible-builds to increase trust in distributed artifacts.
:::
:::
::::







## Research questions {auto-animate="true"}


**How reproducible is software in the functional package management model?**

- Is Nix evaluation reproducible? Can we reproduce *build environments* of Nix packages? 
- Do functional package management enable **bitwise build reproducibility**?



## Reproducibility of build environments {auto-animate="true"}


- Is Nix evaluation reproducible? Can we reproduce *build environments* of Nix packages? 

:::: {.columns}

::: {.column width="50%"}

![](./icse.jpeg){height='14em'}

:::

::: {.column width="50%"}

**"Reproducibility of Build Environments through Space and Time"**, ICSE 2024 (New Ideas and Emerging Results track), *J. Malka, S. Zacchiroli, T. Zimmermann*.


:::

::::




## Reproducibility of build environments


We say that two build environments are identical if they contain the **exact same set of executables, up to their specific versions**.

![](./env.png)


## Reproducibility in Space  


::: {#fig-rb}
![](./space.png)

Reproducibility of build environments in Space

:::

## Reproducibility in Time



::: {#fig-rb}
![](./time.png){.r-stretch}

Reproducibility of build environments in Time

:::


## Research questions


::: {.incremental}
- **RQ1:** Is space and time reproducibility of build environments achievable with Nix ?
- **RQ2:** Does it allow rebuilding of past software versions ?
:::

## Experimental protocol 

::: {.incremental}

1) Sample 200 revisions of the Nix software repository, picked from 2017 to 2023;
2) For each sampled revision, perform the **evaluation** of each package and compare with the historical truth (historical CI results);
3) For the *oldest revision* of our samples, perform the **build** of each package and compare with the historical truth.

:::


## Results {auto-animate="true"}


**RQ1:** *Reproducibility of build environments*

- We were able to **reproduce the build environment of 99.99% of the packages** we tested;
- Discrepancies we found were due to the (unfortunate) use of some of Nix’s impure builtins.

## Results {auto-animate="true"}


**RQ1:** *Reproducibility of build environments*

- We were able to **reproduce the build environment of 99.99% of the packages** we tested;
- Discrepancies we found were due to the (unfortunate) use of some of Nix’s impure builtins.

**RQ2:** *Rebuilding past software versions*

- We were able to **build successfully 14233 out of the 14242 (99.94%) packages that were built successfully by CI in 2017**;
- Discrepancies we found were due to leakages of the Nix build sandbox, that we wish to investigate further.




## Research questions {auto-animate="true"}


**How reproducible is software in the functional package management model?**

- Is Nix evaluation reproducible? Can we reproduce *build environments* of Nix packages? 
- Do functional package management enable **bitwise build reproducibility**?



## Build reproducibility {auto-animate="true"}


- Do functional package management enable **bitwise build reproducibility**?

:::: {.columns}

::: {.column width="50%"}

![](./msr.jpeg){height='14em'}

:::

::: {.column width="50%"}

**"Does Functional Package Management Enable Reproducible Builds at Scale? Yes."**, MSR 2025, *J. Malka, S. Zacchiroli, T. Zimmermann*.


:::

::::


## Motivations 

Reproducible builds are good for software supply chain security, however...

- **Achieving build reproducibility is difficult**: there exist a lot of non- determinisms in compilers that have effects on produced artifacts;
- Uncertainty among reproducibility experts on the feasibility to achieve build reproducibility **at scale** (even though good players like Debian already achieve impressive performance) [^1]

[^1]: M. Fourné, D. Wermke, W. Enck, S. Fahl and Y. Acar, "It’s like flossing your teeth: On the Importance and Challenges of Reproducible Builds for Software Supply Chain Security," 2023 IEEE Symposium on Security and Privacy

## Research questions

::: {.incremental}
- **RQ1:** What is the evolution of bitwise reproducible packages in `nixpkgs` between 2017 and 2023?
- **RQ2:** What are the unreproducible packages?
- **RQ3:** Why are packages unreproducible?
- **RQ4:** How are unreproducibilities fixed? *(Not covered in this talk)*
:::


## Research methodology

::: {#fig-methodology}
![](pipelinev2.png){fig-align="center"}

Pipeline summarizing our research methodology.
:::


## A few figures

::::{.columns}

::: {.column width="60%"}



::: {#fig-diffoscope}
![](./ecosystems.png)

Evolution of the size of the nine most popular software ecosystems in `nixpkgs`.

:::

:::



::: {.column width="40%"}
- 709 816 packages built;
- 14 296 total build hours;
- 548 390 tracked by name and corresponding to <span style="white-space: nowrap;">59 103</span> unique packages associated to a specific software ecossytem .

:::

::::



## RQ1: Evolution of bitwise reproducible packages

::: {#fig-overall}

![](reproducibility-overall-relative.png){.r-stretch fig-align="center"}

Proportion of reproducible, rebuildable and non-rebuildable packages over time.

:::


## RQ1: Evolution of bitwise reproducible packages

::: {#fig-overall2}

![](reproducibility-overall-absolute.png){.r-stretch fig-align="center"}

Absolute numbers of reproducible, rebuildable and non-rebuildable packages over time.

:::



## RQ1: Evolution of bitwise reproducible packages


::: {#fig-overall2-reg}


![](reproducibility-overall-absolute-reg.png){.r-stretch fig-align="center"}

Reproducibility regression around June 2020.

:::





## RQ2: What are the unreproducible packages?

::: {#fig-diff}

![](./reproducibility-ecosystems.png){.r-stretch fig-align="center"}

Proportion of reproducible packages belonging to the three most popular ecosystems and the base namespace of nixpkgs.
:::



## RQ3: Why are packages unreproducible?

::: {#fig-diffoscope}

![](diffoscope.png)

Example of a diffoscope.

:::



## RQ3: Why are packages unreproducible?


::: {#fig-diff}

![](evolution-heuristics.png){.r-stretch fig-align="center"}

Evolution of the number of packages that are matched by each of our heuristics, over time.
:::


## Conclusion

- Bitwise reproducibility in `nixpkgs` as of 2023: 91%;
- This justifies investing resources/conducting research on distributed cache solutions relying on build reproducibility.


## Thank you for your attention!


<h3><ins>My socials:</ins></h3>


<div style="margin-bottom: 40px;"></div>
 {{< bi mastodon >}} luj@chaos.social

 {{< bi envelope >}} julien.malka@telecom-paris.fr


:::: {.columns}

::: {.column width="50%"}


![](qr-icse.png){height='11em'}
:::


::: {.column width="50%"}

![](qr-msr.png){height='11em'}
:::

::::

## RQ1: Evolution of bitwise reproducible packages


::: {#fig-overall2-reg}


![](./sankey-average.png){fig-align="center"}

Sankey graph of the average flow of packages between two revisions, excluding the revision from June 2020, considered as an outlier.

:::



