Introduction

Welcome to the kueue-dev user guide! This comprehensive documentation will help you master the kueue-dev CLI tool for developing, testing, and debugging both the kueue-operator and upstream Kueue.

What is kueue-dev?

kueue-dev is a Rust-based CLI tool that provides a unified interface for:

• Cluster Management - Create and manage kind clusters with custom CNI configurations
• Operator Deployment - Deploy kueue-operator to kind, OpenShift, or via OLM bundles
• Upstream Deployment - Deploy upstream Kueue via kustomize or helm from source
• Image Building - Build and load container images for local development
• Testing - Run e2e tests with automatic retry and cleanup
• Debugging - Interactive menu for logs, metrics, and cluster inspection
• Developer Tools - Preflight checks, shell completions, and more

Features at a Glance

• Kind cluster lifecycle management with CNI support (Calico, default)
• Operator deployment to kind and OpenShift (OLM bundle or direct manifests)
• Upstream Kueue deployment via kustomize or helm
• Image building from source with automatic kind loading
• Automatic dependency installation (cert-manager, JobSet, LeaderWorkerSet, Prometheus)
• E2e test execution with Ginkgo and automatic retry/cleanup
• Interactive debugging menu for logs, metrics, and cluster inspection
• Configuration file support (TOML) and shell completions
• Preflight validation and multi-level logging

Quick Examples

Deploy Operator to Kind

# Create cluster and deploy operator
kueue-dev cluster create --name dev
kueue-dev deploy operator kind --name dev

# Run tests
kueue-dev test run --focus "webhook"

# Cleanup
kueue-dev cluster delete --name dev

Deploy Upstream Kueue from Source

# Create cluster
kueue-dev cluster create --name dev

# Deploy upstream kueue with locally built image
kueue-dev deploy upstream kustomize \
  --upstream-source /path/to/kueue/src \
  --build-image

# Or deploy with helm
kueue-dev deploy upstream helm \
  --upstream-source /path/to/kueue/src \
  --build-image

Interactive Debugging

# Launch interactive menu
kueue-dev interactive

# View operator logs, metrics, cluster state, etc.

Command Structure

kueue-dev
├── cluster          # Manage kind clusters (create, delete, list)
├── deploy
│   ├── operator     # Deploy kueue-operator
│   │   ├── kind     # Deploy to kind cluster
│   │   ├── olm      # Deploy via OLM bundle
│   │   └── openshift# Deploy to OpenShift
│   └── upstream     # Deploy upstream Kueue
│       ├── kustomize# Deploy using kustomize
│       └── helm     # Deploy using helm
├── test             # Run e2e tests
├── images           # Build and manage images
├── cleanup          # Clean up test resources
├── interactive      # Interactive debugging menu
├── check            # Verify prerequisites
├── completion       # Generate shell completions
└── version          # Show version information

Getting Help

• Run kueue-dev --help for command-line help
• Run kueue-dev <command> --help for command-specific help
• Use -vv flag for detailed debug output
• Check the FAQ for common questions
• See Troubleshooting for solutions

Next Steps

• New to kueue-dev? Start with Installation
• Ready to get started? Jump to Quick Start
• Deploy the operator? See Operator Deployment
• Deploy upstream Kueue? See Upstream Deployment
• Need help? See Troubleshooting

Installation

This chapter covers installing kueue-dev and its prerequisites.

Prerequisites

kueue-dev requires different tools depending on your use case:

Required (All Scenarios)

• kubectl - Kubernetes CLI (>= v1.28.0)

For Kind Clusters

• kind - Kubernetes in Docker (>= v0.20.0)
• go - Go programming language (>= 1.21)
• Docker or Podman - Container runtime

For OpenShift

• oc - OpenShift CLI tool

For OLM Deployment

• operator-sdk - Operator SDK for bundle deployment

Checking Prerequisites

Use the built-in check command to verify your setup:

# Check basic prerequisites
kueue-dev check

# Check kind-specific prerequisites
kueue-dev check --kind

# Check OpenShift-specific prerequisites
kueue-dev check --openshift

# Check OLM prerequisites
kueue-dev check --olm

# Check everything
kueue-dev check --kind --openshift --olm

Example output:

Checking prerequisites...
Container runtime: Docker
  ✓ kubectl found (v1.30.0)
  ✓ kind found (v0.20.0)
  ✓ go found (1.21.0)
✓ All prerequisites satisfied!

Building from Source

Clone the Repository

cd kueue-operator/kueue-dev

Build with Cargo

# Development build (faster)
cargo build

# Release build (optimized)
cargo build --release

The binary will be at:

• Development: target/debug/kueue-dev
• Release: target/release/kueue-dev

Build with Nix

If you have Nix with flakes enabled:

# Enter development shell
nix develop .

# Build
cargo build --release

Installing

Install via Cargo

cargo install --path .

This installs kueue-dev to ~/.cargo/bin/kueue-dev.

Manual Installation

Copy the binary to a directory in your PATH:

# After building
cp target/release/kueue-dev ~/.local/bin/

# Or system-wide (requires sudo)
sudo cp target/release/kueue-dev /usr/local/bin/

Verifying Installation

Check that kueue-dev is installed correctly:

# Check version
kueue-dev version

# Output:
# kueue-dev 0.1.0
# Development CLI tool for kueue-operator

# Verify help works
kueue-dev --help

Shell Completion

Set up command completion for your shell:

Bash

# Generate completion file
kueue-dev completion bash > /tmp/kueue-dev-completion.bash

# Install system-wide
sudo mv /tmp/kueue-dev-completion.bash /etc/bash_completion.d/kueue-dev

# Or per-user
mkdir -p ~/.local/share/bash-completion/completions
kueue-dev completion bash > ~/.local/share/bash-completion/completions/kueue-dev

# Reload shell
source ~/.bashrc

Zsh

# Generate completion file
kueue-dev completion zsh > _kueue-dev

# Install to fpath location
sudo mv _kueue-dev /usr/local/share/zsh/site-functions/

# Or per-user
mkdir -p ~/.zsh/completions
kueue-dev completion zsh > ~/.zsh/completions/_kueue-dev

# Add to .zshrc if not already present
echo 'fpath=(~/.zsh/completions $fpath)' >> ~/.zshrc
echo 'autoload -Uz compinit && compinit' >> ~/.zshrc

# Reload shell
source ~/.zshrc

Fish

# Generate and install
kueue-dev completion fish > ~/.config/fish/completions/kueue-dev.fish

# Reload
source ~/.config/fish/config.fish

PowerShell

# Generate completion script
kueue-dev completion powershell > kueue-dev.ps1

# Add to profile
Add-Content $PROFILE ". $(Get-Location)\kueue-dev.ps1"

Updating

To update kueue-dev to the latest version:

# Pull latest changes
cd kueue-operator/kueue-dev
git pull

# Rebuild and reinstall
cargo install --path . --force

Uninstalling

# If installed via cargo
cargo uninstall kueue-dev

# If manually installed
rm ~/.local/bin/kueue-dev
# or
sudo rm /usr/local/bin/kueue-dev

# Remove completion scripts
sudo rm /etc/bash_completion.d/kueue-dev
sudo rm /usr/local/share/zsh/site-functions/_kueue-dev
rm ~/.config/fish/completions/kueue-dev.fish

Next Steps

• Configure kueue-dev with a config file
• Quick Start to deploy your first cluster
• Command Reference for detailed command documentation

Configuration

kueue-dev supports configuration files to set defaults and customize behavior.

Configuration File

Create a configuration file to avoid repeating common flags:

Location

Configuration files are loaded in this order (first found wins):

1. .kueue-dev.toml - Project-specific (current directory)
2. ~/.config/kueue-dev/config.toml - Global user configuration

Format

Configuration uses TOML format with six main sections:

[defaults]
cluster_name = "kueue-test"
cni_provider = "calico"
images_file = "related_images.json"

[colors]
enabled = true
theme = "default"

[behavior]
confirm_destructive = true
parallel_operations = true
show_progress = true

[kueue]
namespace = "openshift-kueue-operator"
frameworks = ["BatchJob", "Pod", "Deployment", "StatefulSet", "JobSet", "LeaderWorkerSet"]

[versions]
cert_manager = "v1.18.0"
jobset = "v0.10.1"
leaderworkerset = "v0.7.0"
calico = "v3.28.2"
prometheus_operator = "v0.82.2"

[tests]
operator_skip_patterns = ["AppWrapper", "PyTorch", "Metrics", ...]
upstream_skip_patterns = ["AppWrapper", "PyTorch", "TrainJob", "Kueuectl", ...]

Configuration Sections

[defaults]

Set default values for command-line flags:

Example:

[defaults]
cluster_name = "my-dev-cluster"
cni_provider = "calico"
images_file = "my-images.json"

With this configuration:

# This command:
kueue-dev cluster create

# Is equivalent to:
kueue-dev cluster create --name my-dev-cluster --cni calico

[colors]

Control terminal color output:

Example:

[colors]
enabled = false  # Disable colors for CI/CD
theme = "default"

[behavior]

Configure tool behavior:

Example:

[behavior]
confirm_destructive = true   # Always ask before deletion
parallel_operations = true   # Use parallel operations
show_progress = true         # Show spinners and progress bars

Note: The confirm_destructive setting affects destructive operations like cluster deletion. When set to false, confirmations are skipped. You can also override this per-command using the --force flag:

# Skip confirmation for this deletion only
kueue-dev cluster delete --name test --force

[kueue]

Configure the Kueue Custom Resource (CR) that will be created during deployment:

Default frameworks:

• BatchJob - Kubernetes Batch Jobs
• Pod - Kubernetes Pods
• Deployment - Kubernetes Deployments
• StatefulSet - Kubernetes StatefulSets
• JobSet - JobSet framework
• LeaderWorkerSet - LeaderWorkerSet framework

Example:

[kueue]
# Enable only specific frameworks
frameworks = ["BatchJob", "Pod", "JobSet"]

Note: The namespace value should typically remain at its default ("openshift-kueue-operator") as this is the standard value expected by the kueue-operator. The Kueue CR name is always "cluster" and is not configurable.

Command-line override:

You can override the frameworks and namespace at deployment time, or skip CR creation entirely:

# Enable only specific frameworks for this deployment
kueue-dev deploy operator kind --kueue-frameworks BatchJob,Pod,JobSet

# Override the namespace
kueue-dev deploy operator kind --kueue-namespace my-kueue-namespace

# Skip Kueue CR creation (deploy operator only)
kueue-dev deploy operator kind --skip-kueue-cr

# Combine multiple overrides
kueue-dev deploy operator kind --kueue-frameworks BatchJob,Pod --kueue-namespace my-namespace

# Or when running tests
kueue-dev test operator --type kind --kueue-frameworks BatchJob,Pod --kueue-namespace my-namespace

# Skip CR creation during tests
kueue-dev test operator --type kind --skip-kueue-cr

The command-line options take precedence over the configuration file.

Use cases for --skip-kueue-cr:

• Testing operator deployment without deploying kueue components
• Manually creating/customizing the CR after deployment
• Advanced debugging scenarios where you want to deploy just the operator infrastructure first

[versions]

Configure versions of dependencies installed during deployment:

Example:

[versions]
cert_manager = "v1.17.0"
jobset = "v0.9.0"
prometheus_operator = "v0.81.0"

Command-line override:

You can override versions at deployment time:

# Override cert-manager version
kueue-dev deploy operator kind --cert-manager-version v1.17.0

# Override multiple versions
kueue-dev deploy operator kind --cert-manager-version v1.17.0 --jobset-version v0.9.0

# Override prometheus operator version
kueue-dev deploy operator kind --prometheus-version v0.81.0

The command-line options take precedence over the configuration file.

[tests]

Configure test skip patterns for both operator and upstream tests:

Default operator skip patterns:

• AppWrapper, PyTorch, JobSet, LeaderWorkerSet
• JAX, Kuberay, Metrics, Fair
• TopologyAwareScheduling, Kueue visibility server
• CPU-dependent tests: Failed Pod can be replaced in group, should allow to schedule a group of diverse pods, StatefulSet created with WorkloadPriorityClass

Default upstream skip patterns:

• Same as operator patterns, plus:
• TrainJob (instead of JobSet)
• Kueuectl (kueuectl not included in operator)

Example:

[tests]
# Custom skip patterns for operator tests
operator_skip_patterns = [
    "AppWrapper",
    "PyTorch",
    "Metrics"
]

# Custom skip patterns for upstream tests
upstream_skip_patterns = [
    "AppWrapper",
    "PyTorch",
    "Metrics",
    "Kueuectl"
]

Use cases:

• Customize which tests to skip based on your cluster capabilities
• Enable tests that are disabled by default once the cluster supports them
• Skip additional tests that are flaky or not relevant to your environment

Note: Test skip patterns use Ginkgo's skip pattern syntax and are combined into a regex like (pattern1|pattern2|pattern3).

Example Configurations

Minimal Configuration

[defaults]
cluster_name = "dev"

Development Configuration

# .kueue-dev.toml (in project root)
[defaults]
cluster_name = "kueue-dev"
cni_provider = "calico"
images_file = "dev-images.json"

[behavior]
confirm_destructive = false  # Skip confirmations for faster iteration
show_progress = true

CI/CD Configuration

# ~/.config/kueue-dev/config.toml (on CI runner)
[defaults]
cluster_name = "ci-test"

[colors]
enabled = false  # No colors in CI logs

[behavior]
confirm_destructive = false  # No interactive prompts
show_progress = false        # No progress bars in CI

Multi-Environment Setup

For developers working with multiple environments:

# ~/.config/kueue-dev/config.toml (global defaults)
[defaults]
cni_provider = "calico"

[colors]
enabled = true

[behavior]
confirm_destructive = true
show_progress = true

Then use project-specific configs:

# ~/projects/feature-a/.kueue-dev.toml
[defaults]
cluster_name = "feature-a"
images_file = "feature-a-images.json"

# ~/projects/bugfix-123/.kueue-dev.toml
[defaults]
cluster_name = "bugfix-123"
images_file = "bugfix-images.json"

Command-Line Override

Command-line flags always override configuration file values:

# Even with cluster_name = "dev" in config:
kueue-dev cluster create --name prod-test

# This creates a cluster named "prod-test"

Viewing Active Configuration

To see what configuration kueue-dev would use:

# Check if config exists
test -f .kueue-dev.toml && echo "Project config found"
test -f ~/.config/kueue-dev/config.toml && echo "Global config found"

# View config
cat .kueue-dev.toml
# or
cat ~/.config/kueue-dev/config.toml

Configuration Tips

Per-Feature Development

Create a config per feature branch:

# Switch to feature branch
git checkout feature/new-webhook

# Create branch-specific config
cat > .kueue-dev.toml <<EOF
[defaults]
cluster_name = "webhook-dev"
images_file = "webhook-test-images.json"
EOF

# Now all commands use these defaults
kueue-dev cluster create  # Creates "webhook-dev" cluster

Team Standards

Share a recommended config in the repository:

# .kueue-dev.toml.example
[defaults]
cni_provider = "calico"

[behavior]
show_progress = true

Team members copy to .kueue-dev.toml:

cp .kueue-dev.toml.example .kueue-dev.toml

Add .kueue-dev.toml to .gitignore for personal customization.

Troubleshooting

Config Not Loading

If your config isn't being used:

1. Check file location: Must be .kueue-dev.toml (note the dot)
2. Check file syntax: TOML syntax must be valid
3. Check file permissions: File must be readable

Validate TOML syntax:

# Install toml-cli if needed
cargo install toml-cli

# Validate config
toml get .kueue-dev.toml defaults

Syntax Errors

Common TOML mistakes:

# ❌ Wrong: quotes around boolean
confirm_destructive = "true"

# ✅ Correct: boolean without quotes
confirm_destructive = true

# ❌ Wrong: missing quotes around string
cluster_name = dev

# ✅ Correct: string with quotes
cluster_name = "dev"

Next Steps

• Quick Start to create your first cluster
• Common Workflows for real-world usage patterns
• Command Reference for detailed command documentation

Quick Start

This guide will walk you through your first kueue-dev workflow in under 5 minutes.

Prerequisites

Before starting, ensure you have:

• kueue-dev installed (Installation Guide)
• Docker or Podman running
• kubectl and kind installed

Verify prerequisites:

kueue-dev check --kind

Step 1: Create a Kind Cluster

Create a new Kubernetes cluster using kind with Calico CNI:

kueue-dev cluster create --name my-cluster --cni calico

You'll see output like:

Creating kind cluster 'my-cluster' with calico CNI...
Creating cluster "my-cluster" ...
 ✓ Ensuring node image (kindest/node:v1.30.0) 🖼
 ✓ Preparing nodes 📦 📦 📦 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
 ✓ Joining worker nodes 🚜
Set kubectl context to "kind-my-cluster"
Installing Calico CNI v3.28.2...
⠋ Downloading Calico manifest...
✓ Calico manifest applied
⠋ Waiting for Calico pods to be ready...
✓ Calico installed successfully
⠋ Labeling worker nodes...
✓ Worker nodes labeled

Cluster 'my-cluster' created successfully!
Kubeconfig: /home/user/.kube/config-my-cluster

This creates a 4-node cluster (1 control-plane + 3 workers) with Calico networking.

Step 2: Build Images (Optional)

If you're building from source, build and push the images:

kueue-dev images build --related-images dev-images.json

This will:

1. Build operator, operand, and must-gather images
2. Push them to your configured registry

Note: Skip this step if using pre-built images from a registry.

Step 3: Deploy the Operator

Deploy kueue-operator to your cluster:

kueue-dev deploy operator kind --name my-cluster --related-images related_images.json

Note: Replace related_images.json with your images file path.

The deployment process:

1. ✅ Sets kubeconfig context
2. ✅ Loads images to kind cluster
3. ✅ Installs cert-manager
4. ✅ Installs JobSet and LeaderWorkerSet
5. ✅ Installs operator CRDs
6. ✅ Deploys the operator
7. ✅ Waits for operator to be ready

Output:

Deploying kueue-operator to kind cluster 'my-cluster'...
Using images from: related_images.json

Images to be used:
  Operator:     quay.io/openshift/kueue-operator:latest
  Operand:      quay.io/openshift/kueue:latest
  Must-gather:  quay.io/openshift/kueue-must-gather:latest

⠋ Loading images to cluster...
✓ All images loaded

⠋ Installing cert-manager v1.13.3...
✓ cert-manager installed

⠋ Installing JobSet v0.10.1...
✓ JobSet installed

⠋ Installing LeaderWorkerSet v0.7.0...
✓ LeaderWorkerSet installed

⠋ Installing operator CRDs...
✓ CRDs installed

⠋ Deploying operator...
✓ Operator deployed

⠋ Waiting for operator to be ready...
✓ Operator ready

==========================================
Deployment completed successfully!
==========================================

To view operator logs:
  kubectl logs -n openshift-kueue-operator -l name=openshift-kueue-operator -f \
    --kubeconfig=/home/user/.kube/config-my-cluster

Customize Kueue CR (Optional):

You can customize which frameworks are enabled or skip CR creation entirely:

# Deploy with only specific frameworks
kueue-dev deploy operator kind --name my-cluster \
  --kueue-frameworks BatchJob,Pod,JobSet

# Use a custom namespace
kueue-dev deploy operator kind --name my-cluster \
  --kueue-namespace my-namespace

# Skip Kueue CR creation (deploy operator only, create CR manually later)
kueue-dev deploy operator kind --name my-cluster \
  --skip-kueue-cr

See Configuration for more options.

Step 4: Run Tests

Run the e2e test suite:

kueue-dev test run --kubeconfig ~/.kube/config-my-cluster

Or with focus on specific tests:

kueue-dev test run --focus "webhook" --kubeconfig ~/.kube/config-my-cluster

Or with a specific label filter:

kueue-dev test run --label-filter "network-policy" --kubeconfig ~/.kube/config-my-cluster

Note: Label filters use Ginkgo's label filtering syntax. The default is !disruptive which excludes disruptive tests. You can use expressions like "network-policy" to run only tests with that label, or "!slow" to exclude slow tests.

Output:

Running e2e tests...
⠋ Installing Ginkgo v2.1.4...
✓ Ginkgo installed

Running test suite...
Running Suite: E2E Test Suite
==============================
Random Seed: 1234

Will run 45 of 120 specs
⠋ Running tests...
••••••••••••••••••••••••••••••••••••••••••••••••

Ran 45 of 120 Specs in 180.234 seconds
SUCCESS! -- 45 Passed | 0 Failed | 0 Pending | 75 Skipped

All tests passed!

Step 3.5: Running Upstream Tests (OpenShift Only)

If you have an OpenShift cluster and want to run the upstream kueue tests, you can use the test upstream command:

kueue-dev test upstream --kubeconfig /path/to/openshift-kubeconfig

This command will:

1. Apply necessary git patches to the upstream kueue source
2. Label worker nodes for e2e testing
3. Configure OpenShift Security Context Constraints (SCC)
4. Run the upstream kueue test suite

Options:

# Run with focus pattern
kueue-dev test upstream --focus "webhook"

# Run with label filter
kueue-dev test upstream --label-filter "!disruptive"

# Run tests from specific target folder (default: singlecluster)
kueue-dev test upstream --target singlecluster

# Combine multiple options
kueue-dev test upstream --focus "webhook" --label-filter "!slow" --target singlecluster

Note: The test upstream command requires:

• An OpenShift cluster with oc CLI configured
• The upstream kueue source code in ../upstream/kueue/src
• Cluster admin permissions to configure SCC policies

Output:

Running upstream kueue tests...
⠋ Applying git patches...
✓ Applied patch: e2e.patch
✓ Applied patch: golang-1.24.patch

⠋ Labeling worker nodes...
✓ Labeled node-1 as on-demand
✓ Labeled node-2 as spot

⠋ Configuring OpenShift SCC...
✓ Added privileged SCC
✓ Added anyuid SCC

⠋ Running upstream e2e tests...
Running Suite: Kueue E2E Test Suite
===================================
Random Seed: 5678

Will run 250 of 350 specs
⠋ Running tests...

Ran 250 of 350 Specs in 450.123 seconds
SUCCESS! -- 250 Passed | 0 Failed | 0 Pending | 100 Skipped

==========================================
Upstream e2e tests passed successfully!
==========================================

Step 5: Interactive Debugging (Optional)

Launch the interactive debugging menu:

kueue-dev interactive --kubeconfig ~/.kube/config-my-cluster

Menu options:

==========================================
Interactive Menu
==========================================

Available actions:
  1) Port-forward to Prometheus UI (http://localhost:9090)
  2) View Prometheus Operator logs
  3) View Prometheus instance logs
  4) View Kueue Operator logs
  5) Show cluster information
  6) kubectl shell (interactive)
  7) Exit

Select an action [1-7]:

Select 4 to view operator logs in real-time.

Step 6: Cleanup

When you're done testing:

# Clean up test resources
kueue-dev cleanup --kubeconfig ~/.kube/config-my-cluster

# Delete the cluster
kueue-dev cluster delete --name my-cluster

Output:

Cleaning up test resources...
✓ Removed finalizers from resources
✓ Deleted test namespaces
✓ Deleted test PriorityClasses
✓ Cleanup completed

Deleting kind cluster 'my-cluster'...
Deleting cluster "my-cluster" ...
✓ Cluster deleted
✓ Kubeconfig removed

Quick Reference Card

Here's a cheat sheet for common operations:

# Create cluster
kueue-dev cluster create --name <name> [--cni calico|default]

# Deploy operator
kueue-dev deploy operator kind --name <name> --related-images <file>

# Run tests
kueue-dev test run [--focus <pattern>] [--label-filter <filter>]

# View logs
kueue-dev interactive

# Cleanup
kueue-dev cleanup
kueue-dev cluster delete --name <name>

# List clusters
kueue-dev cluster list

# Get help
kueue-dev --help
kueue-dev <command> --help

What's Next?

Now that you've completed the basic workflow, explore these topics:

Learn More

• Common Workflows - Real-world development patterns
• Command Reference - Detailed command documentation
• Advanced Features - Dry-run, verbosity, completions

Customize

• Configuration - Set defaults with config files
• Custom Images - Use your own operator images

Troubleshoot

• Troubleshooting - Solutions to common problems
• FAQ - Frequently asked questions

Tips for Success

Use Configuration Files

Save time with a .kueue-dev.toml:

[defaults]
cluster_name = "dev"
images_file = "my-images.json"

Then simply run:

kueue-dev cluster create  # Uses "dev" as name
kueue-dev deploy operator kind     # Uses "my-images.json"

Enable Verbose Output

For debugging, use -v flags:

kueue-dev -v deploy operator kind --name test    # Info level
kueue-dev -vv deploy operator kind --name test   # Debug level
kueue-dev -vvv deploy operator kind --name test  # Trace level

Use Dry-Run

Preview operations before executing:

kueue-dev --dry-run deploy operator kind --name test --related-images related_images.json

Tab Completion

Set up shell completion for faster command entry:

# Bash
kueue-dev completion bash > ~/.local/share/bash-completion/completions/kueue-dev

# Zsh
kueue-dev completion zsh > ~/.zsh/completions/_kueue-dev

# Fish
kueue-dev completion fish > ~/.config/fish/completions/kueue-dev.fish

Happy developing! 🚀

Common Workflows

This chapter covers real-world development workflows using kueue-dev.

The following sections detail specific workflow patterns:

• Local Development - Iterative development on your local machine
• CI/CD Integration - Automated testing in continuous integration
• OpenShift Deployment - Deploying to OpenShift clusters
• OLM Bundle Deployment - Using Operator Lifecycle Manager

General Workflow Pattern

Most kueue-dev workflows follow this pattern:

1. Create/Select Environment
   ↓
2. Deploy Operator
   ↓
3. Test/Debug
   ↓
4. Iterate (modify code, redeploy)
   ↓
5. Cleanup

Quick Workflow Examples

Minimal Test Cycle

Test a single feature quickly:

kueue-dev cluster create --name quick-test
kueue-dev deploy operator kind --name quick-test --related-images dev-images.json
kueue-dev test run --focus "MyFeature"
kueue-dev cleanup
kueue-dev cluster delete --name quick-test

Development with Iteration

Develop a feature with multiple test cycles:

# One-time setup
kueue-dev cluster create --name dev

# Iterate: code → deploy → test
while true; do
  # Make code changes...
  kueue-dev cleanup
  kueue-dev deploy operator kind --name dev --related-images dev-images.json
  kueue-dev test run --focus "MyFeature"
  read -p "Continue? (y/n) " -n 1 -r
  echo
  [[ ! $REPLY =~ ^[Yy]$ ]] && break
done

# Cleanup
kueue-dev cluster delete --name dev

Full Test Suite

Run complete test suite before merging:

kueue-dev cluster create --name full-test --cni calico
kueue-dev deploy operator kind --name full-test --related-images related_images.json
kueue-dev test run  # All tests
kueue-dev cleanup
kueue-dev cluster delete --name full-test

Continue to the specific workflow guides for detailed examples.

Local Development Workflow

The local development workflow is designed for iterative feature development and bug fixing on your local machine.

Overview

This workflow allows you to:

• Quickly iterate on code changes
• Test features in isolation
• Debug issues interactively

Step-by-Step Workflow

1. Create Development Cluster

kueue-dev cluster create --name dev --cni calico

2. Initial Deployment

kueue-dev deploy operator kind --name dev --related-images dev-images.json

3. Develop and Test Loop

# Make code changes
vim pkg/controllers/myfeature.go

# Build and push new images (use --parallel for speed)
kueue-dev images build --related-images dev-images.json --parallel

# Cleanup previous deployment
kueue-dev cleanup

# Redeploy with new images
kueue-dev deploy operator kind --name dev --related-images dev-images.json

# Test your changes
kueue-dev test run --focus "MyFeature"

4. Interactive Debugging

# Launch interactive menu
kueue-dev interactive

# Select option 4: View Kueue Operator logs
# Or option 5: Show cluster information

5. Cleanup When Done

kueue-dev cluster delete --name dev

Tips

• Use a configuration file to avoid repeating flags
• Keep the same cluster across sessions
• Use --skip-tests during rapid iteration
• Enable verbose logging with -v or -vv for debugging

See Configuration for setting up project-specific defaults.

CI/CD Integration

Integrate kueue-dev into your continuous integration pipeline.

GitHub Actions Example

name: E2E Tests

on: [push, pull_request]

jobs:
  e2e:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions-rs/toolchain@v1
        with:
          toolchain: stable
      - name: Install kueue-dev
        run: cargo install --path kueue-dev
      - name: Build and test
        run: |
          # Build images in parallel for faster CI builds
          kueue-dev images build --related-images ci-images.json --parallel

          # Create cluster and deploy
          kueue-dev cluster create --name ci-test
          kueue-dev deploy operator kind --name ci-test --related-images ci-images.json

          # Run tests
          kueue-dev test run

          # Cleanup
          kueue-dev cleanup
          kueue-dev cluster delete --name ci-test

Tips for CI

• Use --dry-run to validate commands first
• Disable colors and progress in CI: see Configuration
• Use deterministic cluster names
• Always cleanup, even on failure

See example CI configurations in the repository.

OpenShift Deployment Workflow

Deploy and test kueue-operator on OpenShift clusters.

Prerequisites

• OpenShift cluster access
• oc CLI tool installed
• Logged in with oc login

Workflow

1. Verify Connection

kueue-dev check --openshift

2. Deploy to OpenShift

kueue-dev deploy operator openshift --related-images related_images.json

3. Run Tests

kueue-dev test operator --type openshift --focus "webhook"

4. Debug

kueue-dev interactive

The interactive menu works with OpenShift clusters using the current oc context.

Notes

• Requires cluster-admin permissions for cert-manager installation
• Uses oc commands instead of kubectl
• Respects current OpenShift context

OLM Bundle Deployment

Deploy kueue-operator using Operator Lifecycle Manager bundles.

Prerequisites

• operator-sdk installed
• Kind cluster or OpenShift cluster
• Bundle image pushed to registry

Workflow

1. Create Cluster (if using kind)

kueue-dev cluster create --name olm-test

2. Deploy via OLM

kueue-dev deploy operator olm --bundle quay.io/my-org/kueue-bundle:v1.0.0 --name olm-test

This automatically:

• Installs OLM if not present
• Deploys the operator bundle
• Waits for operator to be ready

3. Verify Installation

kubectl get csv -n openshift-kueue-operator
kubectl get subscription -n openshift-kueue-operator

4. Run Tests

kueue-dev test run

Notes

• OLM is installed automatically if needed
• Works with both kind and OpenShift clusters
• Bundle image must be publicly accessible or credentials configured

Cluster Commands

Documentation for kueue-dev cluster commands.

Overview

The cluster commands manage kind (Kubernetes in Docker) clusters for local development and testing.

Commands

create

Create a new kind cluster for kueue-operator development.

kueue-dev cluster create [OPTIONS]

Options:

Examples:

# Create cluster with defaults (uses config file settings)
kueue-dev cluster create

# Create cluster with custom name
kueue-dev cluster create --name my-cluster

# Create cluster with specific CNI
kueue-dev cluster create --cni default

# Create cluster and save kubeconfig to specific location
kueue-dev cluster create --name dev --kubeconfig ./kubeconfig

# Override config file defaults
kueue-dev cluster create --name test --cni calico

CNI Provider:

The --cni flag is optional and defaults to the value in your configuration file (default: "calico"). Calico is recommended for most development scenarios as it provides better network policy support.

If you don't specify --cni and don't have a config file, it will use "calico".

delete

Delete an existing kind cluster.

kueue-dev cluster delete [OPTIONS]

Options:

Examples:

# Delete cluster (will prompt for confirmation)
kueue-dev cluster delete --name my-cluster

# Delete cluster without confirmation
kueue-dev cluster delete --name my-cluster --force

# Delete default cluster
kueue-dev cluster delete

list

List all kind clusters on the system.

kueue-dev cluster list

Examples:

# List all kind clusters
kueue-dev cluster list

Configuration

The cluster commands use configuration from .kueue-dev.toml:

[defaults]
cluster_name = "kueue-test"
cni_provider = "calico"

See Configuration for more details.

Related

• Quick Start
• Configuration
• Deploy Commands

Images Commands

Manage container images for kueue-operator.

Overview

The kueue-dev images command provides tools for building, listing, and loading container images used by kueue-operator.

Subcommands

build

Build and push container images for kueue-operator components.

kueue-dev images build [OPTIONS] [COMPONENTS]...

Arguments:

• [COMPONENTS]... - Components to build (operator, operand, must-gather, bundle)
  • If not specified, builds all components

Options:

• -i, --related-images <FILE> - Path to images configuration file
• -p, --parallel - Build components in parallel with animated spinners

Examples:

# Build all components
kueue-dev images build

# Build specific components
kueue-dev images build operator operand

# Build bundle only
kueue-dev images build bundle

# Build with custom images file
kueue-dev images build --related-images dev-images.json

# Build all components in parallel with live spinners
kueue-dev images build --parallel

Parallel Mode Output: When using --parallel, each component gets its own animated spinner with real-time status:

⠋ operator [3/4] Building image...
⠙ operand [2/4] Locating Dockerfile...
✓ must-gather Complete

The terminal title bar also updates to show progress:

• kueue-dev: Building 3 components (initial)
• kueue-dev: Building (2/3) - operand complete (during)
• kueue-dev: Build complete (finished)

See Build Commands for detailed documentation.

list

List images from configuration file.

kueue-dev images list [OPTIONS]

Options:

• -f, --file <FILE> - Path to related images JSON file (default: related_images.json)

Examples:

# List images from default file
kueue-dev images list

# List images from custom file
kueue-dev images list --file my-images.json

Output:

Images from: related_images.json

  operator: quay.io/openshift/kueue-operator:latest
  operand: quay.io/openshift/kueue:latest
  must-gather: quay.io/openshift/kueue-must-gather:latest

load

Load images from local container runtime to kind cluster.

kueue-dev images load [OPTIONS]

Options:

• -n, --name <NAME> - Cluster name (default: kueue-test)
• --related-images <FILE> - Path to related images JSON file (default: related_images.json)

Examples:

# Load images to default cluster
kueue-dev images load

# Load images to specific cluster
kueue-dev images load --name my-cluster

# Load images from custom file
kueue-dev images load --name dev --related-images dev-images.json

Process:

1. Reads image list from configuration file
2. Detects container runtime (podman or docker)
3. Loads each image into the kind cluster
4. Shows progress for each image

Images Configuration File

All image commands use a JSON configuration file to specify image tags:

[
  {
    "name": "operator",
    "image": "quay.io/myuser/kueue-operator:v0.1.0"
  },
  {
    "name": "operand",
    "image": "quay.io/myuser/kueue:v0.6.0"
  },
  {
    "name": "must-gather",
    "image": "quay.io/myuser/kueue-must-gather:v0.1.0"
  },
  {
    "name": "bundle",
    "image": "quay.io/myuser/kueue-bundle:v0.1.0"
  }
]

Default Location

Set a default images file in your .kueue-dev.toml:

[defaults]
images_file = "related_images.json"

Common Workflows

Build, Load, and Deploy

# Build images in parallel
kueue-dev images build --related-images dev-images.json --parallel

# Load to kind cluster (if testing locally without pushing)
kueue-dev images load --name dev --related-images dev-images.json

# Deploy
kueue-dev deploy operator kind --name dev --related-images dev-images.json

List Available Images

# Check what images will be used
kueue-dev images list --file related_images.json

Quick Development Cycle

# 1. Make code changes
vim pkg/controllers/myfeature.go

# 2. Build new images (use --parallel for speed)
kueue-dev images build --parallel

# 3. Cleanup and redeploy
kueue-dev cleanup
kueue-dev deploy operator kind --name dev

# 4. Test
kueue-dev test run --focus "MyFeature"

Container Runtime

All image commands automatically detect your container runtime:

• Tries podman first
• Falls back to docker if podman is not available

Troubleshooting

Images Not Found

If images cannot be loaded:

1. Verify images exist locally:

   podman images | grep kueue
   # or
   docker images | grep kueue
   

2. Check configuration file:

   kueue-dev images list
   

3. Pull images if missing:

   podman pull quay.io/openshift/kueue:latest
   # or
   docker pull quay.io/openshift/kueue:latest
   

Load Fails

If loading to kind fails:

1. Verify cluster exists:

   kind get clusters
   

2. Check kind CLI:

   kind --version
   

3. Verify runtime access:

   podman ps
   # or
   docker ps
   

Related

• Build Commands - Detailed build documentation
• Deployment - Deploy commands
• Configuration - Configure default images file

Build Commands

Build and push container images for kueue-operator components.

Overview

The kueue-dev images build command builds and pushes container images for the kueue-operator project. It supports building operator, operand (kueue), must-gather, and bundle components.

Usage

kueue-dev images build [OPTIONS] [COMPONENTS]...

Arguments

• [COMPONENTS]... - Components to build (operator, operand, must-gather, bundle)
  • If not specified, builds all components

Options

• -i, --related-images <FILE> - Path to images configuration file
  • If not specified, uses the config file setting
• -p, --parallel - Build components in parallel (faster for multiple components)
• -v, --verbose - Enable verbose output

Valid Components

• operator - The kueue-operator image
• operand - The kueue (upstream) image
• must-gather - The must-gather debugging image
• bundle - The OLM bundle image

Examples

Build All Components

Build all four components (operator, operand, must-gather, bundle):

kueue-dev images build

This uses the images file from your config (.kueue-dev.toml) or the default related_images.json.

Build Specific Components

Build only the operator:

kueue-dev images build operator

Build operator and operand:

kueue-dev images build operator,operand

Use Custom Images File

Build with a custom images configuration file:

kueue-dev images build --related-images dev-images.json

Build specific components with custom images file:

kueue-dev images build operator --related-images my-images.json

Build in Parallel

Build all components in parallel for faster builds:

kueue-dev images build --parallel

Build specific components in parallel:

kueue-dev images build operator,operand --parallel

Combine with custom images file:

kueue-dev images build --related-images dev-images.json --parallel

Images Configuration File

The images file is a JSON file that specifies the image tags to build:

[
  {
    "name": "operator",
    "image": "quay.io/myuser/kueue-operator:v0.1.0"
  },
  {
    "name": "operand",
    "image": "quay.io/myuser/kueue:v0.6.0"
  },
  {
    "name": "must-gather",
    "image": "quay.io/myuser/kueue-must-gather:v0.1.0"
  },
  {
    "name": "bundle",
    "image": "quay.io/myuser/kueue-bundle:v0.1.0"
  }
]

Container Runtime

The build command automatically detects your container runtime:

• Tries podman first
• Falls back to docker if podman is not available

Build Process

For each component, the command:

1. Validates the component name
2. Loads the image configuration
3. Detects the container runtime
4. Locates the appropriate Dockerfile:
   • operator: Dockerfile in project root
   • operand: Dockerfile.kueue in project root
   • must-gather: must-gather/Dockerfile
   • bundle: bundle.developer.Dockerfile in project root
5. Builds the image with the specified tag
6. Pushes the image to the registry

Sequential vs Parallel Builds

Sequential (default):

• Builds components one at a time
• Cleaner output with no interleaved logs
• Lower resource usage
• Simple status messages

Parallel (--parallel):

• Builds all components simultaneously
• Significantly faster for multiple components
• Higher CPU and memory usage
• Live animated spinners for each component
• Color-coded status updates

Build Output

The build command uses different output modes depending on whether parallel mode is enabled:

Sequential Mode:

Building and pushing container images...
==========================================
Building component: operator
==========================================
Image tag: quay.io/myuser/kueue-operator:latest
...
Successfully built and pushed: quay.io/myuser/kueue-operator:latest

Parallel Mode:

⠋ operator [3/4] Building image...
⠙ operand [2/4] Locating Dockerfile...
✓ must-gather Complete

Features:

• Animated spinners (⠋⠙⠹⠸⠼⠴⠦⠧⠇⠏)
• Color-coded components (blue/bold)
• Step indicators ([1/4], [2/4], etc.)
• Status symbols (✓ for success, ✗ for failure)
• Real-time updates
• Terminal title updates showing progress (e.g., "Building (2/3) - operand complete")

By default, stdout from Docker/Podman is suppressed. You'll only see:

• Spinner animations (in parallel mode)
• High-level status messages
• Error output (if builds fail)

To see full Docker/Podman output, enable debug logging:

# Show all build output
kueue-dev -vv images build

# Or set RUST_LOG
RUST_LOG=debug kueue-dev images build

Configuration

Set a default images file in your .kueue-dev.toml:

[defaults]
images_file = "related_images.json"

This allows you to run kueue-dev images build without specifying the images file.

Workflow Integration

Local Development

# Make code changes
vim pkg/controllers/myfeature.go

# Build and push new images
kueue-dev images build --related-images dev-images.json

# Deploy to cluster
kueue-dev deploy operator kind --name dev --related-images dev-images.json

CI/CD Pipeline

# Build all components in parallel (faster in CI)
kueue-dev images build --related-images ci-images.json --parallel

# Deploy and test
kueue-dev test operator --type kind --name ci --related-images ci-images.json

Troubleshooting

Build Fails

If the build fails, the error output from Docker/Podman will be displayed automatically. Common issues:

1. Dockerfile exists: Verify the Dockerfile is in the expected location
2. Build context: Ensure you're running from the project root
3. Container runtime: Verify podman or docker is installed and running
4. Image permissions: Ensure you're logged into the registry

For more detailed output during the build process, use debug logging:

kueue-dev -vv images build

Push Fails

If push fails:

1. Login to registry:

   podman login quay.io
   # or
   docker login quay.io
   

2. Check permissions: Ensure you have push access to the repository

3. Verify image tag: Check the image name in your images JSON file

Related

• Deployment - Deploy built images
• Images - Manage container images
• Configuration - Configure default images file

Deploy Commands

Documentation for kueue-dev deploy commands.

Overview

The kueue-dev deploy commands provide streamlined deployment workflows for both the kueue-operator and upstream Kueue.

kueue-dev deploy <SUBCOMMAND>

Deployment Options

Operator Deployment

Deploy the kueue-operator (OpenShift operator) which manages Kueue as an operand.

kueue-dev deploy operator <kind|olm|openshift>

Best for:

• OpenShift environments
• Production deployments
• Managed Kueue lifecycle

See Operator Deployment for full documentation.

Quick Examples:

# Deploy operator to kind cluster
kueue-dev deploy operator kind

# Deploy operator without OLM
kueue-dev deploy operator kind --no-bundle

# Deploy to OpenShift
kueue-dev deploy operator openshift

Upstream Deployment

Deploy upstream Kueue directly from source using kustomize or helm.

kueue-dev deploy upstream <kustomize|helm>

Best for:

• Testing upstream Kueue changes
• Vanilla Kubernetes environments
• Development without the operator

See Upstream Deployment for full documentation.

Quick Examples:

# Deploy with kustomize
kueue-dev deploy upstream kustomize --upstream-source /path/to/kueue/src

# Deploy with helm
kueue-dev deploy upstream helm --upstream-source /path/to/kueue/src

# Build image and deploy
kueue-dev deploy upstream kustomize --upstream-source /path/to/kueue/src --build-image

Command Structure

kueue-dev deploy
├── operator              # Deploy kueue-operator
│   ├── kind              # Deploy to kind cluster
│   ├── olm               # Deploy via OLM bundle
│   └── openshift         # Deploy to OpenShift
└── upstream              # Deploy upstream Kueue
    ├── kustomize         # Deploy using kustomize
    └── helm              # Deploy using helm

Comparison

Common Options

Both deployment types share these common options:

Related

• Operator Deployment
• Upstream Deployment
• Quick Start
• Common Workflows

Operator Deployment

Deploy the kueue-operator (OpenShift operator) to kind or OpenShift clusters.

Overview

The kueue-dev deploy operator commands deploy the kueue-operator, which manages Kueue as an operand. This is the recommended approach for OpenShift environments and production deployments.

Subcommands

deploy operator kind

Deploy kueue-operator to a kind cluster using OLM bundle (default) or direct manifests.

kueue-dev deploy operator kind [OPTIONS]

Options:

Examples:

# Deploy with OLM bundle (default)
kueue-dev deploy operator kind

# Deploy without bundle (direct manifests)
kueue-dev deploy operator kind --no-bundle

# Deploy to specific cluster with custom images
kueue-dev deploy operator kind --name dev --related-images dev-images.json

# Deploy without creating Kueue CR
kueue-dev deploy operator kind --skip-kueue-cr

# Deploy with specific frameworks enabled
kueue-dev deploy operator kind --kueue-frameworks BatchJob,Pod,JobSet

# Deploy with custom dependency versions
kueue-dev deploy operator kind --cert-manager-version v1.17.0 --jobset-version v0.9.0

Deployment Methods:

By default, deployment uses OLM bundle which:

• Installs OLM if not already present
• Deploys operator via operator-sdk run bundle
• Provides production-like deployment experience
• Requires operator-sdk binary

Use --no-bundle flag to deploy via direct manifests which:

• Installs cert-manager, JobSet, LeaderWorkerSet, and Prometheus Operator
• Applies CRDs and operator manifests directly
• Faster for development iteration
• Does not require operator-sdk

Dependencies Installed:

Both deployment methods install these dependencies in parallel:

• cert-manager (for webhook certificates)
• JobSet (for JobSet workloads)
• LeaderWorkerSet (for LeaderWorkerSet workloads)
• Prometheus Operator (for metrics collection)

deploy operator olm

Deploy via OLM bundle with explicit bundle image.

kueue-dev deploy operator olm [OPTIONS]

Options:

Examples:

# Deploy with specific bundle image
kueue-dev deploy operator olm --bundle quay.io/myuser/kueue-bundle:latest

# Deploy to specific cluster
kueue-dev deploy operator olm --bundle quay.io/myuser/kueue-bundle:v0.1.0 --name dev

deploy operator openshift

Deploy to OpenShift cluster.

kueue-dev deploy operator openshift [OPTIONS]

Options:

Examples:

# Deploy to current OpenShift context
kueue-dev deploy operator openshift

# Deploy with custom images
kueue-dev deploy operator openshift --related-images prod-images.json

Common Workflows

Quick Local Development

# Create cluster and deploy in one command
kueue-dev deploy operator kind --name dev

# Or deploy to existing cluster without OLM
kueue-dev deploy operator kind --name dev --no-bundle

Deploy with Custom Configuration

# Deploy with specific frameworks
kueue-dev deploy operator kind \
  --name dev \
  --kueue-frameworks BatchJob,Pod,Deployment,StatefulSet \
  --kueue-namespace kueue-system

Testing Bundle Before Production

# Build bundle
kueue-dev images build bundle

# Deploy with bundle to test
kueue-dev deploy operator kind --name bundle-test

Troubleshooting

OLM Installation Issues

If OLM installation fails, you can:

1. Use --no-bundle to skip OLM
2. Check if OLM is already installed: kubectl get ns olm
3. Verify operator-sdk is available: operator-sdk version

Missing operator-sdk

If you see an error about missing operator-sdk:

operator-sdk is required for bundle deployment but not found in PATH.
Install from: https://sdk.operatorframework.io/docs/installation/
Or use --no-bundle to deploy without OLM

Solution:

• Install operator-sdk from https://sdk.operatorframework.io/docs/installation/
• Or use --no-bundle flag to deploy via direct manifests

Related

• Upstream Deployment
• Quick Start
• OLM Workflow

Upstream Deployment

Deploy upstream Kueue (vanilla Kubernetes) using kustomize or helm from a source tree.

Overview

The kueue-dev deploy upstream commands deploy upstream Kueue directly from source, without the operator. This is useful for:

• Testing upstream Kueue changes
• Development against vanilla Kubernetes
• Environments where the operator is not needed

Prerequisites

• kustomize (for kustomize deployment): Install from https://kubectl.docs.kubernetes.io/installation/kustomize/
• helm (for helm deployment): Install from https://helm.sh/docs/intro/install/
• make (for building images): Required when using --build-image

Run kueue-dev check to verify prerequisites are installed.

Subcommands

deploy upstream kustomize

Deploy upstream Kueue using kustomize overlays.

kueue-dev deploy upstream kustomize [OPTIONS]

Options:

Examples:

# Deploy from source with default overlay
kueue-dev deploy upstream kustomize --upstream-source /path/to/kueue/src

# Deploy with dev overlay
kueue-dev deploy upstream kustomize --upstream-source /path/to/kueue/src --overlay dev

# Deploy with custom image
kueue-dev deploy upstream kustomize --upstream-source /path/to/kueue/src \
  --image gcr.io/my-project/kueue:dev

# Build image from source and deploy
kueue-dev deploy upstream kustomize --upstream-source /path/to/kueue/src --build-image

# Build with custom image tag
kueue-dev deploy upstream kustomize --upstream-source /path/to/kueue/src \
  --build-image --image-tag my-registry/kueue:test

# Skip dependency installation (if already installed)
kueue-dev deploy upstream kustomize --upstream-source /path/to/kueue/src --skip-deps

deploy upstream helm

Deploy upstream Kueue using the helm chart.

kueue-dev deploy upstream helm [OPTIONS]

Options:

Examples:

# Deploy with helm defaults
kueue-dev deploy upstream helm --upstream-source /path/to/kueue/src

# Deploy with custom values file
kueue-dev deploy upstream helm --upstream-source /path/to/kueue/src \
  -f my-values.yaml

# Deploy with inline value overrides
kueue-dev deploy upstream helm --upstream-source /path/to/kueue/src \
  --set controllerManager.replicas=2 \
  --set controllerManager.manager.image.pullPolicy=Always

# Build image from source and deploy
kueue-dev deploy upstream helm --upstream-source /path/to/kueue/src --build-image

# Deploy with custom release name
kueue-dev deploy upstream helm --upstream-source /path/to/kueue/src \
  --release-name kueue-dev

Source Path Resolution

The upstream source path is resolved in the following order:

1. CLI flag: --upstream-source /path/to/kueue/src
2. Environment variable: KUEUE_UPSTREAM_SOURCE=/path/to/kueue/src
3. Config file: Set defaults.upstream_source in your config
4. Current directory: If it contains config/default/kustomization.yaml or charts/kueue/Chart.yaml

Configuration Example

Add to ~/.config/kueue-dev/config.toml:

[defaults]
upstream_source = "/home/user/kueue/src"

Image Building

When --build-image is specified, the tool:

1. Runs make kind-image-build in the upstream source directory
2. The image is built and loaded into the local Docker daemon
3. Loads the image to the kind cluster
4. Configures the deployment to use the built image

Build Requirements

• make must be available in PATH
• Docker or Podman for building images
• Go toolchain (for compiling kueue)

Custom Image Tags

Use --image-tag to specify a custom image registry and tag. The upstream Makefile automatically appends /kueue to the registry, so you only need to specify the registry:

# Default: builds localhost/kueue:dev
kueue-dev deploy upstream kustomize \
  --upstream-source /path/to/kueue/src \
  --build-image

# Custom registry and tag: builds my-registry/kueue:feature-branch
kueue-dev deploy upstream kustomize \
  --upstream-source /path/to/kueue/src \
  --build-image \
  --image-tag my-registry:feature-branch

# You can also include /kueue explicitly (it will be handled correctly)
kueue-dev deploy upstream kustomize \
  --upstream-source /path/to/kueue/src \
  --build-image \
  --image-tag my-registry/kueue:v1.0

For helm deployments with --build-image, the tool automatically sets:

• controllerManager.manager.image.repository
• controllerManager.manager.image.tag
• controllerManager.manager.image.pullPolicy=Never

Dependencies

The following dependencies are installed automatically (unless --skip-deps is used):

Dependencies are installed in parallel for faster deployment.

Common Workflows

Testing Local Changes

# Make changes to upstream kueue
cd /path/to/kueue/src
# ... edit code ...

# Build and deploy
kueue-dev deploy upstream kustomize --upstream-source . --build-image

Switching Between Overlays

# Deploy default overlay
kueue-dev deploy upstream kustomize --upstream-source /path/to/kueue/src

# Switch to dev overlay
kueue-dev deploy upstream kustomize --upstream-source /path/to/kueue/src --overlay dev

# Switch to alpha-enabled overlay
kueue-dev deploy upstream kustomize --upstream-source /path/to/kueue/src --overlay alpha-enabled

Using Pre-built Images

# Use a released image
kueue-dev deploy upstream kustomize \
  --upstream-source /path/to/kueue/src \
  --image gcr.io/k8s-staging-kueue/kueue:main

# Use helm with specific image
kueue-dev deploy upstream helm \
  --upstream-source /path/to/kueue/src \
  --set controllerManager.manager.image.repository=gcr.io/k8s-staging-kueue/kueue \
  --set controllerManager.manager.image.tag=main

Troubleshooting

Missing kustomize or helm

kustomize is required but not found in PATH.
Install from: https://kubectl.docs.kubernetes.io/installation/kustomize/

Run kueue-dev check to see which tools are missing.

Source Path Not Found

No upstream kueue source specified.
Specify the path with --upstream-source or set defaults.upstream_source in config.

Ensure you provide --upstream-source or configure it in your config file.

Image Build Failures

If --build-image fails:

1. Ensure you have Docker or Podman running
2. Check that Go is installed and in PATH
3. Verify the source directory has a valid Makefile
4. Check make kind-image-build runs manually

Related

• Operator Deployment
• Cluster Management
• Configuration

Test Commands

Documentation for kueue-dev test commands.

Overview

The kueue-dev test command provides comprehensive testing capabilities for the kueue-operator. It supports running tests on existing clusters, deploying and testing on new clusters, and running upstream Kueue tests.

Subcommands

test run

Run e2e tests on an existing cluster with retry capability.

Usage:

kueue-dev test run [OPTIONS]

Options:

• -f, --focus <FOCUS> - Test focus pattern (regex)
• -l, --label-filter <LABEL_FILTER> - Label filter for tests (e.g., "!disruptive", "network-policy")
• -k, --kubeconfig <KUBECONFIG> - Path to kubeconfig file (or use KUBECONFIG env var)

Examples:

# Run all non-disruptive tests
kueue-dev test run

# Run tests with focus pattern
kueue-dev test run --focus "webhook"

# Run tests with label filter
kueue-dev test run --label-filter "network-policy"

# Run tests with custom kubeconfig
kueue-dev test run --kubeconfig /path/to/kubeconfig

test operator

Deploy the operator and run tests. Supports three cluster types: kubeconfig (default), kind, and openshift.

Usage:

kueue-dev test operator [OPTIONS]

Options:

• -t, --type <TYPE> - Type of cluster: kubeconfig, kind, or openshift (default: kubeconfig)
• -n, --name <NAME> - Cluster name (kind only, default: kueue-test)
• -f, --focus <FOCUS> - Test focus pattern (regex)
• -l, --label-filter <LABEL_FILTER> - Label filter for tests
• -k, --kubeconfig <KUBECONFIG> - Path to kubeconfig (kubeconfig type only)
• --related-images <IMAGES> - Path to related images JSON file (kind only, default: related_images.json)
• --skip-kueue-cr - Skip creating Kueue CR (only deploy operator)
• --kueue-frameworks <FRAMEWORKS> - Kueue frameworks to enable (comma-separated)
• --kueue-namespace <NAMESPACE> - Kueue CR namespace (default: openshift-kueue-operator)

Examples:

Using kubeconfig type (default)

# Run tests using KUBECONFIG environment variable
kueue-dev test operator

# Run tests with specific kubeconfig file
kueue-dev test operator --kubeconfig ~/.kube/my-cluster-config

# Run tests with focus pattern
kueue-dev test operator --focus "webhook"

# Run tests with label filter
kueue-dev test operator --label-filter "!disruptive"

# Explicitly specify kubeconfig type
kueue-dev test operator --type kubeconfig --kubeconfig /path/to/kubeconfig

Using kind type

# Deploy to kind cluster and run tests
kueue-dev test operator --type kind --name my-cluster

# Deploy with focus pattern
kueue-dev test operator --type kind --name my-cluster --focus "webhook"

# Deploy without creating Kueue CR
kueue-dev test operator --type kind --name my-cluster --skip-kueue-cr

# Deploy with specific frameworks
kueue-dev test operator --type kind --name my-cluster --kueue-frameworks BatchJob,Pod,JobSet

# Deploy with custom namespace
kueue-dev test operator --type kind --name my-cluster --kueue-namespace my-namespace

Using openshift type

# Run tests on OpenShift cluster (uses oc login context)
kueue-dev test operator --type openshift

# Run tests with focus
kueue-dev test operator --type openshift --focus "webhook"

# Run tests with label filter
kueue-dev test operator --type openshift --label-filter "!disruptive"

test upstream

Run upstream kueue tests (requires OpenShift cluster or Kind cluster).

Usage:

kueue-dev test upstream [OPTIONS]

Options:

• -f, --focus <FOCUS> - Test focus pattern (regex)
• -l, --label-filter <LABEL_FILTER> - Label filter for tests
• -k, --kubeconfig <KUBECONFIG> - Path to kubeconfig file
• --target <TARGET> - E2E target folder (default: singlecluster)

Examples:

# Run upstream tests
kueue-dev test upstream

# Run with focus pattern
kueue-dev test upstream --focus "webhook"

# Run with label filter
kueue-dev test upstream --label-filter "!disruptive"

# Run with custom target folder
kueue-dev test upstream --target multicluster

# Run with custom kubeconfig
kueue-dev test upstream --kubeconfig /path/to/kubeconfig

Kind Cluster Behavior:

When running upstream tests on a Kind cluster, the following actions are automatically performed:

1. Operator Scale Down - The openshift-kueue-operator deployment is scaled to 0 replicas and the command waits for all operator pods to terminate before proceeding
2. NetworkPolicy Removal - All NetworkPolicies are deleted from the cluster to avoid networking interference with upstream tests

This ensures that upstream tests run in a clean environment without conflicts from the operator or network policies.

Cluster Type Comparison

Test Filtering

Focus Patterns

The --focus flag accepts regex patterns to run specific tests:

# Run only webhook tests
kueue-dev test operator --focus "webhook"

# Run tests matching multiple patterns
kueue-dev test operator --focus "webhook|admission"

Label Filters

The --label-filter flag filters tests by labels:

# Exclude disruptive tests (default for test run)
kueue-dev test operator --label-filter "!disruptive"

# Run only network-policy tests
kueue-dev test operator --label-filter "network-policy"

# Combine filters
kueue-dev test operator --label-filter "!disruptive && network-policy"

Retry Behavior

The test run and test operator commands include automatic retry on failure:

1. Tests run to completion
2. If tests fail, you're prompted to debug
3. Press RETURN to re-run tests
4. Press Ctrl+C to exit

This is useful for iterative debugging of test failures.

Configuration

Test behavior can be configured in ~/.config/kueue-dev/config.toml:

[tests]
# Test patterns to skip for operator tests
operator_skip_patterns = [
    "AppWrapper",
    "PyTorch",
    "Metrics"
]

# Test patterns to skip for upstream tests
upstream_skip_patterns = [
    "AppWrapper",
    "PyTorch",
    "TrainJob",
    "Kueuectl"
]

Related

• Quick Start
• Local Development Workflow
• CI/CD Integration
• Troubleshooting Tests

Cleanup Commands

Documentation for kueue-dev cleanup commands.

Overview

[Brief description of cleanup commands]

Commands

See kueue-dev cleanup --help for detailed usage.

Examples

# Example commands
kueue-dev cleanup --help

Related

• Quick Start
• Common Workflows

Interactive Commands

Documentation for kueue-dev interactive commands.

Overview

The interactive command launches a menu-driven interface for debugging and monitoring the kueue-operator deployment.

Usage

kueue-dev interactive [OPTIONS]

Options:

• -k, --kubeconfig <FILE> - Path to kubeconfig file

Menu Options

The interactive menu provides:

1. Port-forward to Prometheus UI - Access Prometheus at http://localhost:9090
2. View Prometheus Operator logs - Stream logs from the prometheus-operator pod
3. View Prometheus instance logs - Stream logs from the Prometheus pod
4. View Kueue Operator logs - Stream logs from the kueue-operator pod
5. Show cluster information - Display cluster status and resources
6. Interactive kubectl shell - Drop into a kubectl session

Accessing Prometheus

Via Interactive Menu

kueue-dev interactive
# Select "Port-forward to Prometheus UI"
# Open http://localhost:9090 in your browser

Manual Port-Forward

# Port-forward to Prometheus service
kubectl port-forward -n default svc/prometheus-operated 9090:9090

# Open in browser
open http://localhost:9090

Useful Prometheus Queries

Once in the Prometheus UI, try these queries:

# Kueue workload queue depth
kueue_pending_workloads

# Admission attempts
kueue_admission_attempts_total

# Workload state
kueue_workload_state

# Resource quota usage
kueue_cluster_queue_resource_usage

Examples

# Launch interactive menu
kueue-dev interactive

# Launch with specific kubeconfig
kueue-dev interactive --kubeconfig /path/to/kubeconfig

Related

• Quick Start
• Common Workflows
• Deploy Commands

Utilities Commands

Documentation for kueue-dev utility commands.

Overview

Utility commands provide system checks, completions, and version information.

Commands

check

Check all prerequisites for kueue-dev development.

kueue-dev check

This command checks for all required and optional tools needed for kueue-operator development:

Checked tools:

• kubectl - Kubernetes CLI (required)
• kind - Kubernetes in Docker (for local development)
• go - Go programming language (for building)
• oc - OpenShift CLI (for OpenShift deployments)
• operator-sdk - Operator SDK (for OLM deployments)
• Container runtime (docker or podman)

Output:

The command provides detailed output showing:

• Which tools are found (with ✓ indicator)
• Which tools are missing (with ✗ indicator and installation hints)
• Container runtime status
• Summary of found vs. missing tools

Examples:

# Check all prerequisites
kueue-dev check

Sample output:

[INFO] Checking all prerequisites...
[INFO]
[INFO] ✓ Container runtime: docker
[INFO]
[INFO] Found tools:
[INFO]   ✓ kubectl
[INFO]   ✓ kind
[INFO]   ✓ go
[INFO]   ✓ oc
[INFO]
[ERROR] Missing tools:
[ERROR]   ✗ operator-sdk - Install from https://sdk.operatorframework.io/
[INFO]
[INFO] ==========================================
[INFO] Summary:
[INFO]   Found: 4
[INFO]   Missing: 1
[INFO]   Container runtime: OK
[INFO] ==========================================

Exit codes:

• 0 - All prerequisites satisfied
• 1 - One or more prerequisites missing

Note: This command no longer requires flags like --kind, --openshift, or --olm. It now checks all tools automatically and provides a comprehensive report.

completion

Generate shell completion scripts for kueue-dev.

kueue-dev completion <SHELL>

Supported shells:

• bash
• zsh
• fish
• powershell

Examples:

# Generate bash completion
kueue-dev completion bash > /etc/bash_completion.d/kueue-dev

# Generate zsh completion
kueue-dev completion zsh > "${fpath[1]}/_kueue-dev"

# Generate fish completion
kueue-dev completion fish > ~/.config/fish/completions/kueue-dev.fish

See Completions for detailed setup instructions.

Related

• Quick Start
• Completions
• Installation

Verbosity Levels

Control output detail with multiple -v flags.

Levels

• Default (no flag): Warnings and errors only
• -v: Info level - standard operational messages
• -vv: Debug level - detailed debugging information
• -vvv: Trace level - extremely verbose output

Examples

kueue-dev deploy operator kind --name test                # Minimal output
kueue-dev -v deploy operator kind --name test             # Info level
kueue-dev -vv deploy operator kind --name test            # Debug level
kueue-dev -vvv deploy operator kind --name test           # Trace level

Use Cases

• Default: Production use, CI/CD
• -v: Normal development
• -vv: Debugging issues
• -vvv: Troubleshooting deep problems

Shell Completions

Set up command completion for faster typing.

Supported Shells

• Bash
• Zsh
• Fish
• PowerShell
• Elvish

Installation

See Installation - Shell Completion for detailed instructions.

Usage

Once installed, press Tab to:

• Complete command names
• Complete subcommands
• Complete flag names
• Complete flag values (where applicable)

Example

kueue-dev clu<Tab>       # Completes to "cluster"
kueue-dev cluster cr<Tab> # Completes to "create"
kueue-dev cluster create --cn<Tab>  # Completes to "--cni"

Custom Kubeconfig

Use specific kubeconfig files with kueue-dev.

Methods

1. Flag

kueue-dev test run --kubeconfig /path/to/kubeconfig

2. Environment Variable

export KUBECONFIG=/path/to/kubeconfig
kueue-dev test run

3. Kind Cluster Auto-Config

When using kind clusters, kueue-dev automatically creates kubeconfig at:

~/.kube/config-<cluster-name>

Multiple Clusters

Switch between clusters easily:

# Cluster 1
kueue-dev deploy operator kind --name cluster1 --related-images images1.json

# Cluster 2  
kueue-dev deploy operator kind --name cluster2 --related-images images2.json

# Test on cluster 1
kueue-dev test run --kubeconfig ~/.kube/config-cluster1

# Test on cluster 2
kueue-dev test run --kubeconfig ~/.kube/config-cluster2

Common Issues

Solutions to frequently encountered problems.

Quick Diagnostic

Run verbose mode to see detailed information:

kueue-dev -vv <command>

Common Error Patterns

"Command not found"

Symptom: kueue-dev: command not found

Solution:

# Verify installation
which kueue-dev

# If not found, ensure it's in PATH
export PATH="$HOME/.cargo/bin:$PATH"

# Or reinstall
cargo install --path kueue-dev

"Permission denied"

Symptom: Various permission errors

Solution:

• Check Docker/Podman is running and you have access
• For OpenShift: Verify you have cluster-admin with oc auth can-i '*' '*' --all-namespaces
• Check file permissions on kubeconfig

"Image not found"

Symptom: Failed to load images to kind cluster

Solution:

# Verify image exists locally
docker images | grep kueue

# Pull image if needed
docker pull quay.io/openshift/kueue-operator:latest

# Check images file is correct
kueue-dev images list --file related_images.json

"Image 'X' not found in configuration"

Symptom: Error message like Image 'bundle' not found in configuration when building images

Cause: The configuration file (.kueue-dev.toml) is not being found, causing the tool to fall back to default settings.

Solution:

1. Verify your configuration file exists in the current directory or ~/.config/kueue-dev/config.toml:

   # Check for local config
   ls -la .kueue-dev.toml
   
   # Check for global config
   ls -la ~/.config/kueue-dev/config.toml
   

2. Check that the images file path is correct in your config:

   [defaults]
   images_file = "/path/to/related_images.json"
   

3. Verify the images file contains all required images:

   cat /path/to/related_images.json
   

   Make sure it includes entries for all components (operator, operand, must-gather, bundle):

   [
     {"name": "operator", "image": "quay.io/user/kueue-operator:latest"},
     {"name": "operand", "image": "quay.io/user/kueue:latest"},
     {"name": "must-gather", "image": "quay.io/user/kueue-must-gather:latest"},
     {"name": "bundle", "image": "quay.io/user/kueue-bundle:latest"}
   ]
   

4. Use an absolute path for the images file in your config to avoid path resolution issues.

See specific troubleshooting guides:

• Cluster Connection
• Deployment Failures
• Test Failures

Cluster Connection Issues

Troubleshoot cluster connectivity problems.

Cannot Connect to Cluster

Error: Cannot connect to cluster or connection refused

Diagnose

# Check cluster is running
kind get clusters

# Test kubectl access
kubectl cluster-info

# Check kubeconfig
echo $KUBECONFIG

Solutions

1. Verify cluster exists:

   kind get clusters
   kueue-dev cluster list
   

2. Use correct kubeconfig:

   export KUBECONFIG=~/.kube/config-<cluster-name>
   # or
   kueue-dev --kubeconfig ~/.kube/config-<cluster-name> test run
   

3. Recreate cluster:

   kueue-dev cluster delete --name <name>
   kueue-dev cluster create --name <name>
   

Context Issues

Error: The connection to the server was refused

Solution

# Switch to correct context
kubectl config use-context kind-<cluster-name>

# Or let kueue-dev handle it
kueue-dev deploy operator kind --name <cluster-name> --related-images <file>

Deployment Failures

Troubleshoot operator deployment issues.

Deployment Not Ready

Error: Deployment failed to become ready

Diagnose

# Check pod status
kubectl get pods -n openshift-kueue-operator

# View pod logs
kubectl logs -n openshift-kueue-operator -l name=openshift-kueue-operator

# Check events
kubectl get events -n openshift-kueue-operator --sort-by='.lastTimestamp'

Common Causes

1. Image pull failures

   • Verify images are loaded: docker images | grep kueue
   • Check images file: kueue-dev images list --file <file>

2. CRD conflicts

   • Clean existing resources: kueue-dev cleanup
   • Delete namespace: kubectl delete ns openshift-kueue-operator

3. Resource constraints

   • Check node resources: kubectl top nodes
   • Increase cluster size if needed

cert-manager Issues

Error: cert-manager installation fails

Solution

# Manual installation
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.3/cert-manager.yaml

# Verify
kubectl get pods -n cert-manager

Test Failures

Troubleshoot e2e test failures.

Tests Failing

Error: Tests fail or timeout

Diagnose

# Run with verbose output
kueue-dev -vv test run

# Focus on failing test
kueue-dev test run --focus "FailingTestName"

# Check operator logs
kueue-dev interactive
# Select option 4: View Kueue Operator logs

Solutions

1. Clean and retry:

   kueue-dev cleanup
   kueue-dev test run
   

2. Increase timeout: Tests have automatic timeout handling, but operator may need time to reconcile.

3. Check resource state:

   kubectl get all -n openshift-kueue-operator
   kubectl get clusterqueues
   kubectl get resourceflavors
   

Specific Test Issues

Webhook Tests Fail

Ensure cert-manager is running:

kubectl get pods -n cert-manager

Metrics Tests Fail

Verify Prometheus is configured (if using Prometheus).

Network Policy Tests Fail

Ensure using Calico CNI:

kueue-dev cluster create --name test --cni calico

Frequently Asked Questions

Common questions about kueue-dev.

General

Q: What is kueue-dev?

A: kueue-dev is a Rust CLI tool that consolidates the development workflow for kueue-operator. It replaces multiple shell scripts with a single, well-tested tool.

Q: Do I need Rust to use kueue-dev?

A: You need Rust to build kueue-dev from source, but once built, the binary runs standalone. Pre-built binaries may be available in releases.

Q: Can I use kueue-dev in CI/CD?

A: Yes! kueue-dev is designed for both interactive and automated use. See CI/CD Integration.

Usage

Q: How do I use custom operator images?

A: Create a JSON file with your image references:

{
  "operator": "quay.io/myuser/kueue-operator:my-tag",
  "operand": "quay.io/myuser/kueue:my-tag",
  "must-gather": "quay.io/myuser/kueue-must-gather:my-tag"
}

Then deploy: kueue-dev deploy operator kind --related-images my-images.json

Q: Can I run multiple clusters simultaneously?

A: Yes! Use different cluster names:

kueue-dev cluster create --name cluster1
kueue-dev cluster create --name cluster2

kueue-dev deploy operator kind --name cluster1 --related-images images1.json
kueue-dev deploy operator kind --name cluster2 --related-images images2.json

Q: How do I skip specific tests?

A: kueue-dev automatically skips disruptive tests. To run specific tests, use --focus:

kueue-dev test run --focus "webhook"

Q: What CNI should I use?

A: Calico is recommended for most scenarios as it provides NetworkPolicy support tested by the operator. Use default CNI only for basic testing.

Troubleshooting

Q: How do I update the operator after code changes?

A:

1. Build new images (outside kueue-dev)
2. Clean resources: kueue-dev cleanup
3. Redeploy: kueue-dev deploy operator kind --name <cluster> --related-images <file>

Q: Can I use kueue-dev with real Kubernetes clusters?

A: Yes! Use the --kubeconfig flag or KUBECONFIG environment variable. Be cautious as kueue-dev will make real changes.

Q: How do I debug operator startup issues?

A: Use the interactive menu:

kueue-dev interactive
# Select option 4: View operator logs
# Select option 5: See deployment status

Q: What files does kueue-dev modify?

A: kueue-dev only modifies Kubernetes cluster state. It creates:

• Kubeconfig files in ~/.kube/config-<cluster-name>
• Temporary directories (automatically cleaned up)

Local files are never modified.

Advanced

Q: How do I contribute to kueue-dev?

A: See Contributing for development guidelines.

Q: Can I add custom commands?

A: Currently, kueue-dev doesn't support plugins, but you can fork and modify the source code. Feature requests are welcome!

Q: How do I report bugs?

A: File an issue at: https://github.com/openshift/kueue-operator/issues

Include:

• kueue-dev version (kueue-dev version)
• Operating system
• Command that failed
• Output with -vv flag

Configuration

Q: Where should I put my config file?

A: Two options:

1. Project-specific: .kueue-dev.toml in project root
2. Global: ~/.config/kueue-dev/config.toml

Project config takes precedence.

Q: Can I disable colors?

A: Yes, in your config file:

[colors]
enabled = false

Q: How do I set default cluster name?

A:

[defaults]
cluster_name = "my-default"

Then kueue-dev cluster create uses "my-default" automatically.

More Questions?

• Check the User Guide
• See Troubleshooting
• Ask in GitHub Discussions
• File an issue for bugs

Contributing

Help improve kueue-dev!

Development Setup

Prerequisites

• Rust (>= 1.70)
• cargo
• All kueue-dev prerequisites for testing

Get Started

# Clone repository
git clone https://github.com/openshift/kueue-operator.git
cd kueue-operator/kueue-dev

# Enter Nix shell (if using Nix)
nix develop .

# Build
cargo build

# Run tests
cargo test

# Run clippy
cargo clippy

# Format code
cargo fmt

Project Structure

kueue-dev/
├── src/
│   ├── main.rs              # CLI entry point
│   ├── lib.rs               # Library root
│   ├── commands/            # Command implementations
│   ├── k8s/                 # Kubernetes operations
│   ├── install/             # Component installation
│   ├── config/              # Configuration
│   └── utils/               # Utilities
├── docs/                    # Documentation (mdBook)
├── Cargo.toml               # Dependencies
└── README.md                # Quick reference

Guidelines

Code Style

• Follow Rust conventions
• Run cargo fmt before committing
• Fix all cargo clippy warnings
• Add doc comments for public items

Error Handling

Use the enhanced error types:

#![allow(unused)]
fn main() {
use crate::utils::KueueDevError;

return Err(KueueDevError::cluster_not_found("my-cluster")
    .suggest("Create cluster first")
    .into());
}

Testing

• Add unit tests for new utilities
• Test commands manually on real clusters
• Update integration tests if needed

Documentation

• Update relevant mdBook chapters
• Update README.md for new features
• Add examples to command help text
• Document breaking changes

Making Changes

1. Create Branch

git checkout -b feature/my-feature

2. Make Changes

Follow the guidelines above.

3. Test

# Unit tests
cargo test

# Manual testing
cargo build
./target/debug/kueue-dev cluster create --name test
# ... test your changes ...

4. Commit

git add .
git commit -m "feat: add my feature"

Follow conventional commits format:

• feat: - New feature
• fix: - Bug fix
• docs: - Documentation only
• refactor: - Code refactoring
• test: - Test changes

5. Push and Create PR

git push origin feature/my-feature

Open a pull request on GitHub.

Areas for Contribution

High Priority

• Additional command documentation
• More workflow examples
• Bug fixes
• Performance improvements

Medium Priority

• Enhanced progress indicators
• Better error messages
• Additional preflight checks
• Unit test coverage

Ideas Welcome

• New features
• Documentation improvements
• User experience enhancements
• Platform support (Windows, macOS)

Questions?

• Open a GitHub Discussion
• Ask in pull request comments
• File an issue for bugs

Thank you for contributing! 🎉