Kubernetes on JJGadgets

Tidbits: Ceph RGW - List All S3 Buckets Disk Usage

Sun, 11 May 2025 13:23:18 +0800

Quick one liner command to list all Ceph RGW buckets usage, using jq to extract only the bucket name and size KV pairs and converting from bytes to gigabytes (GB):

1

radosgw-admin bucket stats | jq 'def calc(val): if val!=null then val|tonumber/1000000000|tostring + "GB" else . end; .[] | {bucket} + (.usage."rgw.main" | {size: (calc(.size)), size_actual: (calc(.size_actual)), size_utilized: (calc(.size_utilized))})'

For me, I use Rook-Ceph on Kubernetes for my homelab, and like to view the output in Neovim, so the full command line for me looks like this:

1

kubectl exec -it -n rook-ceph deploy/rook-ceph-tools -- radosgw-admin bucket stats | jq 'def calc(val): if val!=null then val|tonumber/1000000000|tostring + "GB" else . end; .[] | {bucket} + (.usage."rgw.main" | {size: (calc(.size)), size_actual: (calc(.size_actual)), size_utilized: (calc(.size_utilized))})' | nvim

Or only the jq query if you prefer:

1
2
3
4
5
6
7


def calc(val): if val!=null then val|tonumber/1000000000|tostring + "GB" else . end;

.[] | {bucket} + (.usage."rgw.main" | {
 size: (calc(.size)),
 size_actual: (calc(.size_actual)),
 size_utilized: (calc(.size_utilized))
})

Here’s an example output:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84


{
 "bucket": "pg-authentik",
 "size": "1.688277968GB",
 "size_actual": "1.725648896GB",
 "size_utilized": "1.688277968GB"
}
{
 "bucket": "pg-gts-robo",
 "size": "0.003353182GB",
 "size_actual": "0.003444736GB",
 "size_utilized": "0.003353182GB"
}
{
 "bucket": "joplin",
 "size": null,
 "size_actual": null,
 "size_utilized": null
}
{
 "bucket": "pg-atuin",
 "size": "0.468781088GB",
 "size_actual": "0.48617472GB",
 "size_utilized": "0.468781088GB"
}
{
 "bucket": "gotosocial-media",
 "size": "7.746584026GB",
 "size_actual": "7.825215488GB",
 "size_utilized": "7.746584026GB"
}
{
 "bucket": "gts-robo-media",
 "size": null,
 "size_actual": null,
 "size_utilized": null
}
{
 "bucket": "zipline-data",
 "size": "0.00079149GB",
 "size_actual": "0.000794624GB",
 "size_utilized": "0.00079149GB"
}
{
 "bucket": "pg-home",
 "size": "2.763082224GB",
 "size_actual": "2.787741696GB",
 "size_utilized": "2.763082224GB"
}
{
 "bucket": "reactive-resume-media",
 "size": null,
 "size_actual": null,
 "size_utilized": null
}
{
 "bucket": "pg-gotosocial-valetudo",
 "size": null,
 "size_actual": null,
 "size_utilized": null
}
{
 "bucket": "volsync",
 "size": "373.431380366GB",
 "size_actual": "373.502193664GB",
 "size_utilized": "373.431380366GB"
}
{
 "bucket": "pg-default",
 "size": "2.899299768GB",
 "size_actual": "2.965950464GB",
 "size_utilized": "2.899299768GB"
}
{
 "bucket": "k8s-schemas-rgw",
 "size": "0.0391532GB",
 "size_actual": "0.040087552GB",
 "size_utilized": "0.0391532GB"
}
{
 "bucket": "pg-gotosocial",
 "size": "6.635240072GB",
 "size_actual": "6.69478912GB",
 "size_utilized": "6.635240072GB"
}

Flux Repo Structure

Wed, 03 May 2023 00:00:00 +0000

Introduction

In this post, I go over the repository structure that I use for my Kubernetes homelab, powered with GitOps by FluxCD.

Remember that there is no “right way” or “one size fits all” to repo structuring, define and be clear on your goals before you structure your repo.

NOTE: From here on out, all files will be in bold and is prefixed by ./ while all folders will be in bold and suffixed by / e.g. folder/ and ./file.yaml

Goals

Define desired configuration of apps and services deployed in Kubernetes cluster.
Version control all configuration changes with Git.
Automate all the things! (as much as possible)
Establish high security baseline of both Kubernetes cluster’s components, and GitOps components.
Allow for basic multi-cluster environments (production and non-production) while using monorepo
- Cluster-specific configurations using variable substitution in deploy manifests, and cluster-specific secrets.
Allow for opting-in which apps/services/components to deploy per cluster, to optimize resource allocation (CPU, memory, storage, network etc).
- I don’t really need a Minecraft Java server on both prod and staging if I don’t change any of its configuration, do I?
Minimize human errors or forgetfulness by keeping duplicated code to an absolute minimum.

Repo Structure (kube/)

bootstrap/
- flux/
  - ./kustomization.yaml
    - Installs Flux from the kustomization.yaml located in ./manifests/install of fluxcd/flux2 remote GitHub repo.
clusters/
- cluster-name/
  - Cluster specific folder
  - Any name is fine, e.g. prod/ and dev/
  - distro/
    - Cluster distribution’s configuration
    - e.g. Talos via talhelper (talos/)
  - config/
    - All Flux manifests for cluster-specific configuration state goes here
    - ./flux-install.yaml
      - Flux SourceRepository pointed to Flux’s manifests OCI repo, scoped to version branch
      - Fluxtomization to deploy and control Flux’s components (takes over control of Flux components from Flux bootstrap)
    - ./flux-repo.yaml
      - Flux SourceRepository pointed to user’s repo (e.g. JJGadgets/Biohazard, onedr0p/home-ops, 0dragosh/homelab etc), use SSH key to clone (and optionally push if configured)
      - “Master” Fluxtomization to deploy and control cluster-specific configuration (path points to cluster folder) and all other deployments (via cluster folder’s ./kustomization.yaml)
        
        Includes configuration and patches for variable substitution ( ${VARIABLE} ) and SOPS secret decryption of all other deployments
        
        Includes patches for DRYing configs
        
        For the truly lazy, patching a Fluxtomization with patches for other resources controlled by said Fluxtomization (e.g. HelmRelease) is possible
    - ./kustomization.yaml
      - Native Kubernetes Kustomization to control what resources are deployed, either by Fluxtomization or kubectl apply -k
      - Used here for opt-in deployment of all cluster-specific configuration manifests in cluster folder.
      - Used here for opt-in deployment of which apps folders to deploy to cluster (from kube/deploy/).
        
        References the apps folders to deploy like so: ../../../deploy/apps/jellyfin
        
        Each app folder will then have its own kustomization.yaml, which opts-in deployment of the namespaces needed as well as the app’s Fluxtomization (ks.yaml).
        
        Indirectly the “master” Fluxtomization also controls the namespaces deployed (since there’s no Fluxtomizations in between the chain of kustomization.yamls).
        
        This allows the app’s Fluxtomization to add “master” Fluxtomization as dependency (via dependsOn).
        
        Ensures that namespaces are always deployed before the resources are deployed within the namespaces.
        
        NOTE: (about namespacing)
        
        I choose to group apps in their own namespace, unless an app requires either multiple containers (e.g. server and web client, or microservices architecture), or 2 different apps must be in the same namespace to share Kubernetes resources or other reasons.
        
        My structure is a janky ghetto way to implement “multi-cluster” with a very specific purpose of having production and non-production (dev/test/staging/UAT, whichever is appropriate) clusters, where the differences in the clusters mostly come down to cluster-specific variables/secrets and control of which apps are deployed, allowing the non-prod cluster(s) to be scaled smaller than the prod cluster (e.g. I don’t really need Jellyfin on both prod and staging if I don’t change any of its configuration)
        
        Other folks like onedr0p, bjw-s, 0dragosh etc will have the “master” Fluxtomization deploy a separate “apps” Fluxtomization that points to kubernetes/apps which has kubernetes/apps/namespace/kustomization.yaml control both namespace and app Fluxtomizations (kubernetes/apps/namespace/app/ks.yaml) to deploy, which isn’t a wrong way to structure for their needs, however this means that there is no easy way to control which cluster should deploy which apps which I desire
    - ./secrets.yaml
      - Cluster-specific native Kubernetes secrets, encrypted-at-rest by SOPS (Flux supports decrypting SOPS-encrypted files).
      - Can be used via variable substitution at deploy-time.
      - external-secrets is preferred over storing secrets in here to easily sync and/or consume namespace-specific secrets.
      - One hacky way to have cluster-specific yet namespace-specific native Kubernetes secrets (you can’t reference a secret stored in flux-system namespace from e.g. minecraft namespace) is to variable substitute the secret data into a kind: Secret manifest in kube/deploy/.
    - ./vars.yaml
      - Cluster-specific variables.
      - Can be used via variable substitution at deploy-time.
deploy/
- Apps, services and components to be deployed to clusters.
- Contains baseline common manifests that are written to work across all clusters.
- Cluster-specific configuration is achieved via variable substitution and optionally patches.
- core/
  - Backend(-related) components that are “core” to Kubernetes clusters for them to operate, such as CNI, storage solution, Ingress Controller, and more.
- apps/
  - User-facing apps and services.
  - app-name/
    - Replace app-name/ with app name, such as minecraft/, authentik/, or immich/.
    - ./ns.yaml
      - Namespace definition for app, or group of apps.
    - ./ks.yaml
      - All app-specific Fluxtomizations that deploy and control the app’s resources.
      - Points to either app/ and/or component-name/ folders which are explained below.
      - Use dependsOn to ensure that dependencies (such as storage solution and Ingress Controller) are deployed and configured before app’s resources are deployed and configured.
    - ./repo.yaml
      - Only needed if deployment method relies on Flux SourceRepository, such as HelmRepository or OCIRepository.
    - ./kustomization.yaml
      - Native Kubernetes Kustomization to control what resources are deployed, either by Fluxtomization or kubectl apply -k
      - Used here for opting-in all YAML files but not folders, since folders are controlled by app-specific Fluxtomization which itself is controlled by “master” Fluxtomization via this and cluster’s kustomization.yaml.
    - app/ OR component-name/
      - All manifests used to deploy Kubernetes resources needed for app are placed here.
      - Usually if both app/ and another folder such as config/ or certs/ are present, app/ is used to deploy components which are a dependency for the other folders’ resources to be deployed.
      - ./netpol.yaml
        
        CiliumNetworkPolicy or Kubernetes native Network Policy
        
        “Principle of Least Privilege” and “Default Deny” practices are followed, only allow necessary connections
        
        Commonly allowed include Ingress Controller, intra-namespace traffic, communication with other apps’ Kubernetes services as needed.
        
        Not all Kubernetes components need to be network accessible by every application, such as APIServer, storage solution pods/services, or Flux. Assign only if such traffic is necessary for app to function.
      - ./hr.yaml
        
        HelmRelease deployment method.

Kubernetes on JJGadgets

Tidbits: Ceph RGW - List All S3 Buckets Disk Usage

Flux Repo Structure

Introduction

Goals

Repo Structure (kube/)

bootstrap/

clusters/

deploy/