535 lines
11 KiB
Text
535 lines
11 KiB
Text
---
|
||
title: Reproducibility in functional package management
|
||
date: 2025-4-25
|
||
date-format: long
|
||
lightbox: true
|
||
logo: telecom.png
|
||
margin-top: "0px"
|
||
author:
|
||
- name:
|
||
given: Julien
|
||
family: Malka
|
||
url: https://luj.fr
|
||
email: julien.malka@telecom-paris.fr
|
||
orcid: 0009-0008-9845-6300
|
||
roles:
|
||
- conceptualization
|
||
- investigation
|
||
- writing – original draft
|
||
affiliations:
|
||
- id: telecom
|
||
name: Télécom Paris, Institut Polytechnique de Paris
|
||
fig-align: center
|
||
code-overflow: wrap
|
||
code-line-numbers: false
|
||
css: styles.css
|
||
|
||
format:
|
||
metropolis-beamer-revealjs:
|
||
theme: slide.scss
|
||
toc: false
|
||
toc-title: Plan
|
||
toc-depth: 2
|
||
slide-level: 2
|
||
slide-number: true
|
||
---
|
||
|
||
|
||
|
||
## Research topics
|
||
|
||
**Main topics:** Cybersecurity & Software engineering
|
||
|
||
*How can one trust the software installed on one’s system is not malicious?*
|
||
|
||
- What if we make the assumption that the software is **open source**?
|
||
|
||
|
||
**Research question**: How to increase trust in the Open Source Software Supply Chain with *functional package managers* and *reproducible builds*?
|
||
|
||
|
||
|
||
|
||
|
||
## Functional package managers {auto-animate="true"}
|
||
|
||
|
||
New software deployment model (from which **Nix** has been the first example).
|
||
|
||
|
||
```nix
|
||
{ stdenv, lib, fetchFromGitHub }:
|
||
```
|
||
|
||
## Functional package managers {auto-animate="true"}
|
||
|
||
|
||
New software deployment model (from which **Nix** has been the first example).
|
||
|
||
```nix
|
||
{ stdenv, lib, fetchFromGitHub }:
|
||
|
||
stdenv.mkDerivation rec {
|
||
version = "1.3.7";
|
||
pname = "htpdate";
|
||
|
||
src = fetchFromGitHub {
|
||
owner = "twekkel";
|
||
repo = pname;
|
||
rev = "v${version}";
|
||
sha256 = "sha256-X7r95Uc4oGB0eVum5D7pC4tebZIyyz73g6Q/D0cjuFM=";
|
||
};
|
||
|
||
```
|
||
|
||
## Functional package managers {auto-animate="true"}
|
||
|
||
|
||
New software deployment model (from which **Nix** has been the first example).
|
||
|
||
```nix
|
||
{ stdenv, lib, fetchFromGitHub }:
|
||
|
||
stdenv.mkDerivation rec {
|
||
version = "1.3.7";
|
||
pname = "htpdate";
|
||
|
||
src = fetchFromGitHub {
|
||
owner = "twekkel";
|
||
repo = pname;
|
||
rev = "v${version}";
|
||
sha256 = "sha256-X7r95Uc4oGB0eVum5D7pC4tebZIyyz73g6Q/D0cjuFM=";
|
||
};
|
||
|
||
makeFlags = [
|
||
"prefix=$(out)"
|
||
];
|
||
|
||
```
|
||
|
||
## Functional package managers {auto-animate="true"}
|
||
|
||
::: {.r-fit-text}
|
||
|
||
New software deployment model (from which **Nix** has been the first example).
|
||
|
||
```nix
|
||
{ stdenv, lib, fetchFromGitHub }:
|
||
|
||
stdenv.mkDerivation rec {
|
||
version = "1.3.7";
|
||
pname = "htpdate";
|
||
|
||
src = fetchFromGitHub {
|
||
owner = "twekkel";
|
||
repo = pname;
|
||
rev = "v${version}";
|
||
sha256 = "sha256-X7r95Uc4oGB0eVum5D7pC4tebZIyyz73g6Q/D0cjuFM=";
|
||
};
|
||
|
||
makeFlags = [
|
||
"prefix=$(out)"
|
||
];
|
||
|
||
meta = with lib; {
|
||
description = "Utility to fetch time and set the system clock over HTTP";
|
||
platforms = platforms.linux;
|
||
license = licenses.gpl2Plus;
|
||
maintainers = with maintainers; [ julienmalka ];
|
||
};
|
||
}
|
||
|
||
```
|
||
|
||
:::
|
||
|
||
|
||
## Evaluation->Build pipeline
|
||
|
||
|
||
::: {#fig-eval-build}
|
||
{.r-stretch}
|
||
|
||
Eval-build pipeline
|
||
|
||
:::
|
||
|
||
|
||
## Functional package managers for SSC security
|
||
|
||
Functional package managers also have interesting properties for software supply chain security (which are of interest for us):
|
||
|
||
- Builds from source;
|
||
- Sandboxed compilation.
|
||
|
||
|
||
|
||
## Functional package managers for SSC security
|
||
|
||
- Installed packages create a static graph structure (a Merkle tree) that can be analysed in order to find known vulnerability in the dependencies.
|
||
|
||
|
||
::: {#fig-eval-build}
|
||
{.r-stretch}
|
||
|
||
Example of a package dependency graph
|
||
|
||
:::
|
||
|
||
## The binary distribution problem
|
||
|
||
- It is not always reasonable to compile all the software a user wants to install on their own machine: creates the necessity of binary caches ;
|
||
- **But** binary caches make us lose some of the interesting security properties of functional package managers.
|
||
|
||
|
||
|
||
|
||
## Reproducible builds {.lol}
|
||
|
||
{height='4em' fig-align="center"}
|
||
|
||
A build is **reproducible** if given the same source code, build environment and build instructions, any party can recreate bit-by-bit identical copies of all specified artifacts.
|
||
|
||
|
||
## Why is build reproducibility important?
|
||
|
||
:::: {.columns}
|
||
::: {.column width="40%"}
|
||
- Reproducible builds can be useful to improve the **security** of the **binary distribution problem**;
|
||
- Instead of having to trust a single third party to provide them the correct binary, they can divide the trust between multiple parties.
|
||
:::
|
||
::: {.column width="60%"}
|
||
::: {#fig-rb}
|
||
{height='13em'}
|
||
|
||
Leveraging reproducible-builds to increase trust in distributed artifacts.
|
||
:::
|
||
:::
|
||
::::
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
## Research questions {auto-animate="true"}
|
||
|
||
|
||
**How reproducible is software in the functional package management model?**
|
||
|
||
- Is Nix evaluation reproducible? Can we reproduce *build environments* of Nix packages?
|
||
- Do functional package management enable **bitwise build reproducibility**?
|
||
|
||
|
||
|
||
## Reproducibility of build environments {auto-animate="true"}
|
||
|
||
|
||
- Is Nix evaluation reproducible? Can we reproduce *build environments* of Nix packages?
|
||
|
||
:::: {.columns}
|
||
|
||
::: {.column width="50%"}
|
||
|
||
{height='14em'}
|
||
|
||
:::
|
||
|
||
::: {.column width="50%"}
|
||
|
||
**"Reproducibility of Build Environments through Space and Time"**, ICSE 2024 (New Ideas and Emerging Results track), *J. Malka, S. Zacchiroli, T. Zimmermann*.
|
||
|
||
|
||
:::
|
||
|
||
::::
|
||
|
||
|
||
|
||
|
||
## Reproducibility of build environments
|
||
|
||
|
||
We say that two build environments are identical if they contain the **exact same set of executables, up to their specific versions**.
|
||
|
||
|
||
|
||
|
||
## Reproducibility in Space
|
||
|
||
|
||
::: {#fig-rb}
|
||
|
||
|
||
Reproducibility of build environments in Space
|
||
|
||
:::
|
||
|
||
## Reproducibility in Time
|
||
|
||
|
||
|
||
::: {#fig-rb}
|
||
{.r-stretch}
|
||
|
||
Reproducibility of build environments in Time
|
||
|
||
:::
|
||
|
||
|
||
## Research questions
|
||
|
||
|
||
::: {.incremental}
|
||
- **RQ1:** Is space and time reproducibility of build environments achievable with Nix ?
|
||
- **RQ2:** Does it allow rebuilding of past software versions ?
|
||
:::
|
||
|
||
## Experimental protocol
|
||
|
||
::: {.incremental}
|
||
|
||
1) Sample 200 revisions of the Nix software repository, picked from 2017 to 2023;
|
||
2) For each sampled revision, perform the **evaluation** of each package and compare with the historical truth (historical CI results);
|
||
3) For the *oldest revision* of our samples, perform the **build** of each package and compare with the historical truth.
|
||
|
||
:::
|
||
|
||
|
||
## Results {auto-animate="true"}
|
||
|
||
|
||
**RQ1:** *Reproducibility of build environments*
|
||
|
||
- We were able to **reproduce the build environment of 99.99% of the packages** we tested;
|
||
- Discrepancies we found were due to the (unfortunate) use of some of Nix’s impure builtins.
|
||
|
||
## Results {auto-animate="true"}
|
||
|
||
|
||
**RQ1:** *Reproducibility of build environments*
|
||
|
||
- We were able to **reproduce the build environment of 99.99% of the packages** we tested;
|
||
- Discrepancies we found were due to the (unfortunate) use of some of Nix’s impure builtins.
|
||
|
||
**RQ2:** *Rebuilding past software versions*
|
||
|
||
- We were able to **build successfully 14233 out of the 14242 (99.94%) packages that were built successfully by CI in 2017**;
|
||
- Discrepancies we found were due to leakages of the Nix build sandbox, that we wish to investigate further.
|
||
|
||
|
||
|
||
|
||
## Research questions {auto-animate="true"}
|
||
|
||
|
||
**How reproducible is software in the functional package management model?**
|
||
|
||
- Is Nix evaluation reproducible? Can we reproduce *build environments* of Nix packages?
|
||
- Do functional package management enable **bitwise build reproducibility**?
|
||
|
||
|
||
|
||
## Build reproducibility {auto-animate="true"}
|
||
|
||
|
||
- Do functional package management enable **bitwise build reproducibility**?
|
||
|
||
:::: {.columns}
|
||
|
||
::: {.column width="50%"}
|
||
|
||
{height='14em'}
|
||
|
||
:::
|
||
|
||
::: {.column width="50%"}
|
||
|
||
**"Does Functional Package Management Enable Reproducible Builds at Scale? Yes."**, MSR 2025, *J. Malka, S. Zacchiroli, T. Zimmermann*.
|
||
|
||
|
||
:::
|
||
|
||
::::
|
||
|
||
|
||
## Motivations
|
||
|
||
Reproducible builds are good for software supply chain security, however...
|
||
|
||
- **Achieving build reproducibility is difficult**: there exist a lot of non- determinisms in compilers that have effects on produced artifacts;
|
||
- Uncertainty among reproducibility experts on the feasibility to achieve build reproducibility **at scale** (even though good players like Debian already achieve impressive performance) [^1]
|
||
|
||
[^1]: M. Fourné, D. Wermke, W. Enck, S. Fahl and Y. Acar, "It’s like flossing your teeth: On the Importance and Challenges of Reproducible Builds for Software Supply Chain Security," 2023 IEEE Symposium on Security and Privacy
|
||
|
||
## Research questions
|
||
|
||
::: {.incremental}
|
||
- **RQ1:** What is the evolution of bitwise reproducible packages in `nixpkgs` between 2017 and 2023?
|
||
- **RQ2:** What are the unreproducible packages?
|
||
- **RQ3:** Why are packages unreproducible?
|
||
- **RQ4:** How are unreproducibilities fixed? *(Not covered in this talk)*
|
||
:::
|
||
|
||
|
||
## Research methodology
|
||
|
||
::: {#fig-methodology}
|
||
{fig-align="center"}
|
||
|
||
Pipeline summarizing our research methodology.
|
||
:::
|
||
|
||
|
||
## A few figures
|
||
|
||
::::{.columns}
|
||
|
||
::: {.column width="60%"}
|
||
|
||
|
||
|
||
::: {#fig-diffoscope}
|
||
|
||
|
||
Evolution of the size of the nine most popular software ecosystems in `nixpkgs`.
|
||
|
||
:::
|
||
|
||
:::
|
||
|
||
|
||
|
||
::: {.column width="40%"}
|
||
- 709 816 packages built;
|
||
- 14 296 total build hours;
|
||
- 548 390 tracked by name and corresponding to <span style="white-space: nowrap;">59 103</span> unique packages associated to a specific software ecossytem .
|
||
|
||
:::
|
||
|
||
::::
|
||
|
||
|
||
|
||
## RQ1: Evolution of bitwise reproducible packages
|
||
|
||
::: {#fig-overall}
|
||
|
||
{.r-stretch fig-align="center"}
|
||
|
||
Proportion of reproducible, rebuildable and non-rebuildable packages over time.
|
||
|
||
:::
|
||
|
||
|
||
## RQ1: Evolution of bitwise reproducible packages
|
||
|
||
::: {#fig-overall2}
|
||
|
||
{.r-stretch fig-align="center"}
|
||
|
||
Absolute numbers of reproducible, rebuildable and non-rebuildable packages over time.
|
||
|
||
:::
|
||
|
||
|
||
|
||
## RQ1: Evolution of bitwise reproducible packages
|
||
|
||
|
||
::: {#fig-overall2-reg}
|
||
|
||
|
||
{.r-stretch fig-align="center"}
|
||
|
||
Reproducibility regression around June 2020.
|
||
|
||
:::
|
||
|
||
|
||
|
||
|
||
|
||
## RQ2: What are the unreproducible packages?
|
||
|
||
::: {#fig-diff}
|
||
|
||
{.r-stretch fig-align="center"}
|
||
|
||
Proportion of reproducible packages belonging to the three most popular ecosystems and the base namespace of nixpkgs.
|
||
:::
|
||
|
||
|
||
|
||
## RQ3: Why are packages unreproducible?
|
||
|
||
::: {#fig-diffoscope}
|
||
|
||
|
||
|
||
Example of a diffoscope.
|
||
|
||
:::
|
||
|
||
|
||
|
||
## RQ3: Why are packages unreproducible?
|
||
|
||
|
||
::: {#fig-diff}
|
||
|
||
{.r-stretch fig-align="center"}
|
||
|
||
Evolution of the number of packages that are matched by each of our heuristics, over time.
|
||
:::
|
||
|
||
|
||
## Conclusion
|
||
|
||
- Bitwise reproducibility in `nixpkgs` as of 2023: 91%;
|
||
- This justifies investing resources/conducting research on distributed cache solutions relying on build reproducibility.
|
||
|
||
|
||
## Thank you for your attention!
|
||
|
||
|
||
<h3><ins>My socials:</ins></h3>
|
||
|
||
|
||
<div style="margin-bottom: 40px;"></div>
|
||
{{< bi mastodon >}} luj@chaos.social
|
||
|
||
{{< bi envelope >}} julien.malka@telecom-paris.fr
|
||
|
||
|
||
:::: {.columns}
|
||
|
||
::: {.column width="50%"}
|
||
|
||
|
||
{height='11em'}
|
||
:::
|
||
|
||
|
||
::: {.column width="50%"}
|
||
|
||
{height='11em'}
|
||
:::
|
||
|
||
::::
|
||
|
||
## RQ1: Evolution of bitwise reproducible packages
|
||
|
||
|
||
::: {#fig-overall2-reg}
|
||
|
||
|
||
{fig-align="center"}
|
||
|
||
Sankey graph of the average flow of packages between two revisions, excluding the revision from June 2020, considered as an outlier.
|
||
|
||
:::
|
||
|
||
|
||
|