Homelab on JJGadgets

CVE-2023-43809: My experience with my first CVE

Wed, 15 Nov 2023 10:00:00 +0800

Disclaimer

Everything in this post is my perspective, opinions and experience, and does not represent anyone else but myself. Also, I will be using Markdown URL texts for non-vulnerability-related links, but please, always check what link you’re about to click on or visit, as part of general internet safety practices.

About Soft Serve (my opinions)

Simple but very effective self-hosted Git server (and now local browser!), with an amazing TUI and CLI over SSH.

I selfhost Soft Serve on my home Kubernetes cluster, for private projects.

Vulnerability Info

Soft Serve Public Key Authentication Bypass Vulnerability when Keyboard-Interactive SSH Authentication is Enabled

My tl;dr: if public key needs client-side verification, and allow-keyless is enabled which turns on keyboard-interactive, public key can match an account but bypass (fail) client-side validation and successfully login on Soft Serve servers.

Find out more on the GHSA page https://github.com/advisories/GHSA-mc97-99j4-vm2v or the MITRE page https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-43809 or my GitHub issue on it https://github.com/charmbracelet/soft-serve/issues/389.

Experience with Developers (Good!)

The Charm team have been professional about handling this with minimal friction during the disclosure process, with regular progress updates and swift responses as they confirmed the vulnerability and worked to find the root cause and fix it.

My POV

So here comes what I consider the most unexpected way to find a vulnerability ever. I won’t be surprised if you think it’s a fake story, but I know for sure it was real. ~~Else I wouldn’t have got my lazy ass to write a post about it, would I?~~

I am aboard a public bus, and my destination is only a few stops away, within 10 minutes.

I tapped my Android phone, which has a virtual wallet card registered to the NFC, to board the bus. Found a seat, opened Termux.

“Hmm, what should I do?” Undecided, I fiddle with k9s to look at the list of pods, wondering which app to mess with. “Ah, I got it, I need to reconfigure my Soft Serve after screwing around with making sure the PostgreSQL support was fully working on the latest release.”

“Damn, HelmRelease didn’t update, lemme fix that… okay, good to go.” ssh softserve “Now, need my YubiKey to authenticate Soft Serve’s SSH with my YubiKey’s PGP SSH key…”

“Shit, it’s my stop, doors are about to close!” I literally jump up from my seat with one foot launching me forward, one hand grabbing onto my YubiKey, and one hand holding my Android phone with OpenKeychain prompt open.

And, the moment: taps phone’s NFC card to alight bus, foreground app switches from Termux with OpenKeychain prompt to virtual wallet app, alights and switches back to Termux “Wait… why am I logged in? I don’t remember even plugging the YubiKey in…”

I kill the SSH session, ssh softserve again, and when OpenKeychain prompt came up this time, I intentionally clicked “Cancel”. To my shock, I was logged in again, YubiKey having never been plugged in. This was no accident nor was I seeing things.

Side note: Funny enough, I had filed an issue on the bus ride back home for not being able to change allow-keyless and anon-access settings when PostgreSQL was used, which led to 0.6.1 being released, and opted to properly test and report this vulnerability when I got home. I didn’t know at the time that it would be the very setting that would mitigate this vulnerability if Soft Serve users couldn’t yet update to a patched version of 0.6.2 and above. So if you look at it a certain way, I was basically the reason for all the patch versions of the 0.6.x version family being released. All done from an Android phone. Oops!

Timeline (GMT+8)

12 September 2023: I found the potential vulnerability.

15 September 2023: I reported the vulnerability in a thread on the Charm Discord server with a description and listing the environments that I used to reproduce the vulnerability. Devs acknowledged and stated they would look into it.

16 September 2023: I screen recorded a PoC video using my Android, and reported my further discovery that turning allow-keyless off seemed to mitigate the vulnerability, amongst other details.

17-22 September 2023: Further communications between me and dev to identify the root cause.

27 September 2023: PR with patched code opened and merged.

28 September 2023: I tested that the nightly build with the patch PR merged does fix the issue, and opened the GitHub issue for public transparency.

3 October 2023: v0.6.2 patch version released with verified fix, GitHub Security Advisory (GHSA) filed and I accepted credit. GitHub issue closed as completed.

5 October 2023: CVE-2023-43809 was published.

Flux Repo Structure

Wed, 03 May 2023 00:00:00 +0000

Introduction

In this post, I go over the repository structure that I use for my Kubernetes homelab, powered with GitOps by FluxCD.

Remember that there is no “right way” or “one size fits all” to repo structuring, define and be clear on your goals before you structure your repo.

NOTE: From here on out, all files will be in bold and is prefixed by ./ while all folders will be in bold and suffixed by / e.g. folder/ and ./file.yaml

Goals

Define desired configuration of apps and services deployed in Kubernetes cluster.
Version control all configuration changes with Git.
Automate all the things! (as much as possible)
Establish high security baseline of both Kubernetes cluster’s components, and GitOps components.
Allow for basic multi-cluster environments (production and non-production) while using monorepo
- Cluster-specific configurations using variable substitution in deploy manifests, and cluster-specific secrets.
Allow for opting-in which apps/services/components to deploy per cluster, to optimize resource allocation (CPU, memory, storage, network etc).
- I don’t really need a Minecraft Java server on both prod and staging if I don’t change any of its configuration, do I?
Minimize human errors or forgetfulness by keeping duplicated code to an absolute minimum.

Repo Structure (kube/)

bootstrap/
- flux/
  - ./kustomization.yaml
    - Installs Flux from the kustomization.yaml located in ./manifests/install of fluxcd/flux2 remote GitHub repo.
clusters/
- cluster-name/
  - Cluster specific folder
  - Any name is fine, e.g. prod/ and dev/
  - distro/
    - Cluster distribution’s configuration
    - e.g. Talos via talhelper (talos/)
  - config/
    - All Flux manifests for cluster-specific configuration state goes here
    - ./flux-install.yaml
      - Flux SourceRepository pointed to Flux’s manifests OCI repo, scoped to version branch
      - Fluxtomization to deploy and control Flux’s components (takes over control of Flux components from Flux bootstrap)
    - ./flux-repo.yaml
      - Flux SourceRepository pointed to user’s repo (e.g. JJGadgets/Biohazard, onedr0p/home-ops, 0dragosh/homelab etc), use SSH key to clone (and optionally push if configured)
      - “Master” Fluxtomization to deploy and control cluster-specific configuration (path points to cluster folder) and all other deployments (via cluster folder’s ./kustomization.yaml)
        
        Includes configuration and patches for variable substitution ( ${VARIABLE} ) and SOPS secret decryption of all other deployments
        
        Includes patches for DRYing configs
        
        For the truly lazy, patching a Fluxtomization with patches for other resources controlled by said Fluxtomization (e.g. HelmRelease) is possible
    - ./kustomization.yaml
      - Native Kubernetes Kustomization to control what resources are deployed, either by Fluxtomization or kubectl apply -k
      - Used here for opt-in deployment of all cluster-specific configuration manifests in cluster folder.
      - Used here for opt-in deployment of which apps folders to deploy to cluster (from kube/deploy/).
        
        References the apps folders to deploy like so: ../../../deploy/apps/jellyfin
        
        Each app folder will then have its own kustomization.yaml, which opts-in deployment of the namespaces needed as well as the app’s Fluxtomization (ks.yaml).
        
        Indirectly the “master” Fluxtomization also controls the namespaces deployed (since there’s no Fluxtomizations in between the chain of kustomization.yamls).
        
        This allows the app’s Fluxtomization to add “master” Fluxtomization as dependency (via dependsOn).
        
        Ensures that namespaces are always deployed before the resources are deployed within the namespaces.
        
        NOTE: (about namespacing)
        
        I choose to group apps in their own namespace, unless an app requires either multiple containers (e.g. server and web client, or microservices architecture), or 2 different apps must be in the same namespace to share Kubernetes resources or other reasons.
        
        My structure is a janky ghetto way to implement “multi-cluster” with a very specific purpose of having production and non-production (dev/test/staging/UAT, whichever is appropriate) clusters, where the differences in the clusters mostly come down to cluster-specific variables/secrets and control of which apps are deployed, allowing the non-prod cluster(s) to be scaled smaller than the prod cluster (e.g. I don’t really need Jellyfin on both prod and staging if I don’t change any of its configuration)
        
        Other folks like onedr0p, bjw-s, 0dragosh etc will have the “master” Fluxtomization deploy a separate “apps” Fluxtomization that points to kubernetes/apps which has kubernetes/apps/namespace/kustomization.yaml control both namespace and app Fluxtomizations (kubernetes/apps/namespace/app/ks.yaml) to deploy, which isn’t a wrong way to structure for their needs, however this means that there is no easy way to control which cluster should deploy which apps which I desire
    - ./secrets.yaml
      - Cluster-specific native Kubernetes secrets, encrypted-at-rest by SOPS (Flux supports decrypting SOPS-encrypted files).
      - Can be used via variable substitution at deploy-time.
      - external-secrets is preferred over storing secrets in here to easily sync and/or consume namespace-specific secrets.
      - One hacky way to have cluster-specific yet namespace-specific native Kubernetes secrets (you can’t reference a secret stored in flux-system namespace from e.g. minecraft namespace) is to variable substitute the secret data into a kind: Secret manifest in kube/deploy/.
    - ./vars.yaml
      - Cluster-specific variables.
      - Can be used via variable substitution at deploy-time.
deploy/
- Apps, services and components to be deployed to clusters.
- Contains baseline common manifests that are written to work across all clusters.
- Cluster-specific configuration is achieved via variable substitution and optionally patches.
- core/
  - Backend(-related) components that are “core” to Kubernetes clusters for them to operate, such as CNI, storage solution, Ingress Controller, and more.
- apps/
  - User-facing apps and services.
  - app-name/
    - Replace app-name/ with app name, such as minecraft/, authentik/, or immich/.
    - ./ns.yaml
      - Namespace definition for app, or group of apps.
    - ./ks.yaml
      - All app-specific Fluxtomizations that deploy and control the app’s resources.
      - Points to either app/ and/or component-name/ folders which are explained below.
      - Use dependsOn to ensure that dependencies (such as storage solution and Ingress Controller) are deployed and configured before app’s resources are deployed and configured.
    - ./repo.yaml
      - Only needed if deployment method relies on Flux SourceRepository, such as HelmRepository or OCIRepository.
    - ./kustomization.yaml
      - Native Kubernetes Kustomization to control what resources are deployed, either by Fluxtomization or kubectl apply -k
      - Used here for opting-in all YAML files but not folders, since folders are controlled by app-specific Fluxtomization which itself is controlled by “master” Fluxtomization via this and cluster’s kustomization.yaml.
    - app/ OR component-name/
      - All manifests used to deploy Kubernetes resources needed for app are placed here.
      - Usually if both app/ and another folder such as config/ or certs/ are present, app/ is used to deploy components which are a dependency for the other folders’ resources to be deployed.
      - ./netpol.yaml
        
        CiliumNetworkPolicy or Kubernetes native Network Policy
        
        “Principle of Least Privilege” and “Default Deny” practices are followed, only allow necessary connections
        
        Commonly allowed include Ingress Controller, intra-namespace traffic, communication with other apps’ Kubernetes services as needed.
        
        Not all Kubernetes components need to be network accessible by every application, such as APIServer, storage solution pods/services, or Flux. Assign only if such traffic is necessary for app to function.
      - ./hr.yaml
        
        HelmRelease deployment method.

ZFS list rough cheatsheet

Sun, 05 Dec 2021 00:00:00 +0000

List & sort ascending only total used space by all snapshots of individual datasets:

1

zfs list -r -o name,usedsnap -s usedsnap

List & sort ascending actual disk usage = disk usage from specific dataset without children dataset’s storage

1

zfs list -r -o name,usedds -s usedds

List all space related columns available in `zfs list`:

1

zfs list -ro space

-s [column name] sorts zfs list output in ascending order by selected column’s data, -S [column name] sorts the same descending.