Skip to main content

Introduction

This guide provides step-by-step instructions to deploy the W&B Platform in air-gapped, fully disconnected, or restricted network customer-managed environments. Air-gapped deployments are common in:
  • Secure government facilities
  • Financial institutions with strict network isolation
  • Healthcare organizations with compliance requirements
  • Industrial control systems (ICS) environments
  • Research facilities with classified networks
Use an internal container registry and Helm repository to host the required W&B images and charts. Run these commands in a shell console with proper access to the Kubernetes cluster. You can adapt these commands to work with any CI/CD tooling you use to deploy Kubernetes applications. For standard on-premises Kubernetes deployments with internet connectivity, see Deploy W&B with Kubernetes Operator.

Prerequisites

Before starting, ensure your air-gapped environment meets the following requirements.

Version requirements

SSL/TLS requirements

W&B requires a valid signed SSL/TLS certificate for secure communication between clients and the server. SSL/TLS termination must occur on the ingress/load balancer. The W&B Server application does not terminate SSL or TLS connections. Important: W&B does not support self-signed certificates and custom CAs. Using self-signed certificates will cause challenges for users and is not supported. If possible, using a service like Let’s Encrypt is a great way to provide trusted certificates to your load balancer. Services like Caddy and Cloudflare manage SSL for you. If your security policies require SSL communication within your trusted networks, consider using a tool like Istio and side car containers.

Hardware requirements

CPU Architecture: W&B runs on Intel (x86) CPU architecture only. ARM is not supported. Sizing: For CPU, memory, and disk sizing recommendations for Kubernetes nodes and MySQL, see the Sizing section in the reference architecture. Requirements vary based on whether you’re running Models, Weave, or both.

MySQL database

W&B requires an external MySQL database. For production, W&B strongly recommends using managed database services: Managed database services provide automated backups, monitoring, high availability, patching, and reduce operational overhead. See the reference architecture for complete MySQL requirements, including sizing recommendations and configuration parameters. For database creation SQL, see the bare-metal guide. For questions about your deployment’s database configuration, contact support or your AISE. For MySQL configuration parameters for self-managed instances, see the reference architecture MySQL configuration section.

Redis

W&B depends on a single-node Redis 7.x deployment used by W&B’s components for job queuing and data caching. For convenience during testing and development of proofs of concept, W&B Self-Managed includes a local Redis deployment that is not appropriate for production deployments. For production deployments, W&B can connect to a Redis instance in the following environments:

Object storage

W&B requires object storage with pre-signed URL and CORS support. Recommended storage providers:
MinIO Open Source is in maintenance mode with no active development or pre-compiled binaries. For production deployments, W&B recommends using managed object storage services or enterprise S3-compatible solutions such as MinIO Enterprise (AIStor).
For detailed bucket provisioning instructions including IAM policies, CORS configuration, and access setup, see the Bring Your Own Bucket (BYOB) guide. See the reference architecture object storage section for complete requirements. For detailed object storage provisioning guidance, see the Bring Your Own Bucket (BYOB) guide. In air-gapped environments, you’ll typically use on-premises S3-compatible storage such as MinIO Enterprise, NetApp StorageGRID, or Dell ECS.

Air-gapped specific requirements

In addition to the standard requirements above, air-gapped deployments require:
  • Internal container registry: Access to a private container registry (Harbor, JFrog Artifactory, Nexus, etc.) with all required W&B images
  • Internal Helm repository: Access to a private Helm chart repository with W&B Helm charts
  • Image transfer capability: A method to transfer container images from an internet-connected system to your air-gapped registry
  • License file: A valid W&B Enterprise license. To obtain a license (for example, from an internet-connected machine), see the License section on the Requirements page, or contact your W&B account team.
For complete infrastructure requirements, including networking and load balancer configuration, see the reference architecture.

Prepare your air-gapped environment

Step 1: Set up internal container registry

For a successful air-gapped deployment, all required container images must be available in your air-gapped container registry.
You are responsible for tracking the W&B Operator’s requirements and maintaining your container registry with updated images regularly. For the most current list of required container images and versions, refer to the Helm chart, or contact W&B Support or your assigned W&B support engineer.

Core W&B component containers

The following core images are required:

Dependency containers

The following third-party dependency images are required:

Get the complete image list

To extract the complete list of required images and versions from the Helm chart:
  1. On an internet-connected system, download the W&B Helm charts from the W&B Helm charts repository:
  2. Inspect the values.yaml files to identify all container images and their versions:
    Alternatively, use this command to extract just the repository names (without version tags):
    The list of repositories will look similar to the following:
    To get the specific version tags for each image, use the first command above (grep -E "repository:|tag:"), which will show both repository names and their corresponding version tags.

Transfer images to air-gapped registry

  1. On an internet-connected system, pull and save all required images.
    Replace version numbers in the examples below with the actual versions from your Helm chart inspection in step 2 above. The versions shown here are examples and will become outdated.
    Use shell variables to manage versions consistently:
  2. Transfer the .tar files to your air-gapped environment using your approved method (USB drive, secure file transfer, etc.).
  3. In your air-gapped environment, load and push images to your internal registry:

Step 2: Set up internal Helm chart repository

Along with the container images, ensure the following Helm charts are available in your internal Helm repository:
  1. On an internet-connected system, download the charts:
  2. Transfer the .tgz chart files to your air-gapped environment and upload them to your internal Helm repository according to your repository’s procedures. The operator chart deploys the W&B Kubernetes Operator (Controller Manager). The operator-wandb chart deploys the W&B Platform using the values configured in the Custom Resource (CR).

Step 3: Configure Helm repository access

  1. In your air-gapped environment, configure Helm to use your internal repository:
  2. Verify the charts are available:

Deploy W&B in air-gapped environment

Step 4: Install the Kubernetes Operator

The W&B Kubernetes Operator (controller manager) manages the W&B platform components. To install it in an air-gapped environment, configure it to use your internal container registry.
  1. Create a values.yaml file with the following content:
    Replace the repository and tag with the actual versions you transferred to your internal registry in Step 1. The version shown here (1.13.3) is an example and will become outdated.
  2. Install the operator and Custom Resource Definition (CRD):
  3. Verify the operator is running:
    You should see the operator pod in a Running state.
For full details about supported values, refer to the Kubernetes operator GitHub repository values file.

Step 5: Set up MySQL database

Before configuring the W&B Custom Resource, set up an external MySQL database. For production deployments, W&B strongly recommends using managed database services where available. However, if you are running your own MySQL instance, create the database and user: Create a database and a user with the following SQL commands. Replace SOME_PASSWORD with a secure password of your choice:
For MySQL configuration parameters, see the reference architecture MySQL configuration section.

Step 6: Configure W&B Custom Resource

After installing the W&B Kubernetes Operator, configure the Custom Resource (CR) to point to your internal Helm repository and container registry. This configuration ensures the Kubernetes operator uses your internal registry and repository when deploying the required components of the W&B platform.
The example configuration below includes image version tags that will become outdated. Replace all tag: values with the actual versions you transferred to your internal registry in Step 1.
Create a file named wandb.yaml with the following content:
Replace all placeholder values (hostnames, passwords, tags, etc.) with your actual configuration values. The example above shows the most commonly used components.
Depending on your deployment needs, you may also need to configure image repositories for additional components such as:
  • settingsMigrationJob
  • weave-trace
  • filestream
  • flat-runs-table
Refer to the W&B Helm repository values file for the complete list of configurable components.

Step 7: Deploy the W&B platform

  1. Apply the W&B Custom Resource to deploy the platform:
  2. Monitor the deployment progress:
    The deployment may take several minutes as the operator creates all necessary components.

OpenShift configuration

W&B fully supports deployment on air-gapped OpenShift Kubernetes clusters. OpenShift deployments require additional security context configurations due to OpenShift’s stricter security policies.

OpenShift security context constraints

OpenShift uses Security Context Constraints (SCCs) to control pod permissions. By default, OpenShift assigns the restricted SCC to pods, which prevents running as root and requires specific user IDs. Configure W&B components to run with the restricted SCC by setting appropriate security contexts in your Custom Resource:

Option 2: Create custom SCC (if required)

If your deployment requires capabilities not available in the restricted SCC, create a custom SCC:
  1. Apply the SCC:
  2. Bind the SCC to the W&B service accounts:

OpenShift routes

OpenShift uses Routes instead of standard Kubernetes Ingress. Configure W&B to use OpenShift Routes:

OpenShift image pull configuration

If your OpenShift cluster uses an internal image registry with authentication:
  1. Create an image pull secret:
  2. Reference the secret in your Custom Resource:

OpenShift complete example

Here’s a complete example CR for OpenShift air-gapped deployment:
Replace all tag: values in this example with the actual versions you transferred to your internal registry in Step 1. The versions shown are examples and will become outdated.
Contact W&B Support or your assigned W&B support engineer for comprehensive OpenShift configuration examples tailored to your security requirements.

Verify your installation

After deploying W&B, verify the installation is working correctly: To verify the installation, W&B recommends using the W&B CLI. The verify command executes several tests that verify all components and configurations.
This step assumes that the first admin user account is created with the browser.
Follow these steps to verify the installation:
  1. Install the W&B CLI:
  1. Log in to W&B:
For example:
  1. Verify the installation:
A successful installation and fully working W&B deployment shows the following output:
Contact W&B Support if you encounter errors.

Additional air-gapped verification

For air-gapped deployments, also verify:
  1. Image pull: Confirm all pods successfully pulled images from your internal registry:
    All images should point to your internal registry and all pods should be in Running state.
  2. External connectivity: Verify W&B is not attempting external connections (it shouldn’t in air-gapped mode):
  3. License validation: Access the W&B console and verify your license is active.

Troubleshooting

Image pull errors

If pods fail to pull images:
  1. Verify images exist in your internal registry
  2. Check image pull secret is correctly configured
  3. Verify network connectivity from Kubernetes nodes to registry
  4. Check registry authentication credentials

OpenShift SCC errors

If pods fail with permission errors on OpenShift:

Helm chart not found

If the operator cannot find the platform chart:
  1. Verify the chart repository URL in the Custom Resource
  2. Check that the operator pod can reach your internal Helm repository
  3. Verify the chart exists in your repository:

Frequently asked questions

Can I use a different ingress class?

Yes, configure your ingress class by modifying the ingress settings in your Custom Resource:

How do I handle certificate bundles with multiple certificates?

Split the certificates into multiple entries in the customCACerts section:

How do I prevent automatic updates?

Configure the operator to not automatically update W&B:
  1. Set airgapped: true in the operator installation (this disables automatic update checks)
  2. Control version updates by manually updating the spec.chart.version in your Custom Resource
  3. Optionally, disable automatic updates from the W&B System Console
See Disable automatic app version updates for more details.
W&B strongly recommends customers with Self-Managed instances update their deployments with the latest release at minimum once per quarter to maintain support and receive the latest features, performance improvements, and fixes. W&B supports a major release for 12 months from its initial release date. Refer to Release policies and processes.

Does the deployment work with no connection to public repositories?

Yes. When airgapped: true is set in the operator configuration, the Kubernetes operator uses only your internal resources and does not attempt to connect to public repositories.

How do I update W&B in an air-gapped environment?

To update W&B:
  1. Pull new container images on an internet-connected system
  2. Transfer images to your air-gapped registry
  3. Upload new Helm charts to your internal repository
  4. Update the spec.chart.version and image tags in your Custom Resource
  5. Apply the updated Custom Resource The operator will perform a rolling update of the W&B components.

Next steps

After successful deployment:
  1. Configure user authentication: Set up SSO or other authentication methods
  2. Set up monitoring: Configure monitoring for your W&B instance and infrastructure
  3. Plan for updates: Review the Server upgrade process and establish an update cadence
  4. Configure backups: Establish backup procedures for your MySQL database
  5. Document your process: Create runbooks for your specific air-gapped update procedures

Getting help

If you encounter issues during deployment:
  • Review the Reference Architecture for infrastructure guidance
  • Check the Operator guide for configuration details
  • Contact W&B Support or your assigned W&B support engineer
  • For OpenShift-specific issues, reference Red Hat OpenShift documentation