Deploying Sourcegraph executors natively on Kubernetes

This feature is in beta and is available in Sourcegraph 5.1.0 and later.

Kubernetes manifests are provided to deploy Sourcegraph Executors on a running Kubernetes cluster. If you are deploying Sourcegraph with helm, charts are available here.

Requirements

Feature flag

To instruct Sourcegraph to use Kubernetes-deployed executors, you will need to enable the native-ssbc-execution feature flag.

RBAC Roles

Executors interact with the Kubernetes API to handle jobs. The following are the RBAC Roles needed to run Executors on Kubernetes.

API Groups	Resources	Verbs	Reason
`batch`	`jobs`	`create`, `delete`	Executors create Job pods to run processes. Once Jobs are completed, they are cleaned up.
	`pods`, `pods/log`	`get`, `list`, `watch`	Executors need to look up and steam logs from the Job Pods.

See the example Role YAML for details.

Docker Image

The Executor Docker image is available on Docker Hub at sourcegraph/executor-kubernetes.

Environment Variables

The following are Environment Variables that are specific to the Kubernetes deployment method. These environment variables can be set on the Executor Deployment and will configure the Jobs that it spawns. Executor environment variables can also be defined on a Kubernetes ConfigMap or Secret that is mounted to the executor deployment.

Name	Default Value	Description
`EXECUTOR_KUBERNETES_CONFIG_PATH`	N/A	The path to the Kubernetes configuration file. If not specified, the in cluster config is used.
`EXECUTOR_KUBERNETES_NODE_NAME`	N/A	The name of the Kubernetes Node to create Jobs in. If not specified, the Pods are created in the first available node.
`EXECUTOR_KUBERNETES_NODE_SELECTOR`	N/A	A comma separated list of values to use as a node selector for Kubernetes Jobs. e.g. `foo=bar,app=my-app`
`EXECUTOR_KUBERNETES_NODE_REQUIRED_AFFINITY_MATCH_EXPRESSION`	N/A	The JSON encoded required affinity match expressions for Kubernetes Jobs. e.g. `[{"key": "foo", "operator": "In", "values": ["bar"]}]`
`EXECUTOR_KUBERNETES_NODE_REQUIRED_AFFINITY_MATCH_FIELDS`	N/A	The JSON encoded required affinity match fields for Kubernetes Jobs. e.g. `[{"key": "foo", "operator": "In", "values": ["bar"]}]`
`EXECUTOR_KUBERNETES_POD_AFFINITY`	N/A	The JSON encoded pod affinity for Kubernetes Jobs. e.g. `[{"labelSelector": {"matchExpressions": [{"key": "foo", "operator": "In", "values": ["bar"]}]}, "topologyKey": "kubernetes.io/hostname"}]`
`EXECUTOR_KUBERNETES_POD_ANTI_AFFINITY`	N/A	The JSON encoded pod anti-affinity for Kubernetes Jobs. e.g. `[{"labelSelector": {"matchExpressions": [{"key": "foo", "operator": "In", "values": ["bar"]}]}, "topologyKey": "kubernetes.io/hostname"}]`
`EXECUTOR_KUBERNETES_NODE_TOLERATIONS`	N/A	The JSON encoded tolerations for Kubernetes Jobs. e.g. `[{"key": "foo", "operator": "Equal", "value": "bar", "effect": "NoSchedule"}]`
`EXECUTOR_KUBERNETES_NAMESPACE`	`default`	The Kubernetes namespace in which the executor service will spawn executor jobs (K8s Jobs) in.
`EXECUTOR_KUBERNETES_PERSISTENCE_VOLUME_NAME`	`sg-executor-pvc`	The name of the Executor Persistence Volume. Must match the `PersistentVolumeClaim` configured for the instance.
`EXECUTOR_KUBERNETES_RESOURCE_LIMIT_CPU`	N/A	The maximum CPU resource for Kubernetes Jobs.
`EXECUTOR_KUBERNETES_RESOURCE_LIMIT_MEMORY`	`12Gi`	The maximum memory resource for Kubernetes Jobs.
`EXECUTOR_KUBERNETES_RESOURCE_REQUEST_CPU`	N/A	The minimum CPU resource for Kubernetes Jobs.
`EXECUTOR_KUBERNETES_RESOURCE_REQUEST_MEMORY`	`12Gi`	The minimum memory resource for Kubernetes Jobs.
`KUBERNETES_JOB_DEADLINE`	`1200`	The number of seconds after which a Kubernetes job will be terminated.
`KUBERNETES_RUN_AS_USER`	N/A	The user ID to run Kubernetes jobs as.
`KUBERNETES_RUN_AS_GROUP`	N/A	The group ID to run Kubernetes jobs as.
`KUBERNETES_FS_GROUP`	`1000`	The group ID to run all containers in the Kubernetes jobs as.
`KUBERNETES_KEEP_JOBS`	`false`	If true, Kubernetes jobs will not be deleted after they complete. Useful for debugging.
`KUBERNETES_JOB_ANNOTATIONS`	N/A	The JSON encoded annotations to add to the Kubernetes Jobs. e.g. `{"foo": "bar", "faz": "baz"}`
`KUBERNETES_JOB_POD_ANNOTATIONS`	N/A	The JSON encoded annotations to add to the Kubernetes Job Pods. e.g. `{"foo": "bar", "faz": "baz"}`
`KUBERNETES_IMAGE_PULL_SECRETS`	N/A	The names of Kubernetes image pull secrets to use for pulling images. e.g. my-secret,my-other-secret

Note: EXECUTOR_KUBERNETES_NAMESPACE should be set to either "default" or the specific namespace where your Executor is deployed.

Note: executor.frontendUrl must be set in the Site configuration for Executors to work correctly.

See other possible executor configuration environment variables here.

Job Scheduling

Note: Kubernetes has a max of 110 pods per node. If you run into this limit, you can lower the number of Job Pods running on a node by setting the environment variable EXECUTOR_MAXIMUM_NUM_JOBS.

Executors deployed on Kubernetes require Jobs to be scheduled on the same Node as the Executor. This is to ensure that Jobs are able to access the same Persistence Volume as the Executor.

To ensure that Jobs are scheduled on the same Node as the Executor, the following environment variables can be set,

EXECUTOR_KUBERNETES_NODE_NAME
EXECUTOR_KUBERNETES_NODE_SELECTOR
EXECUTOR_KUBERNETES_NODE_REQUIRED_AFFINITY_MATCH_EXPRESSIONS
EXECUTOR_KUBERNETES_NODE_REQUIRED_AFFINITY_MATCH_FIELDS

Node Name

Using the Downward API, the property spec.nodeName can be used to set the EXECUTOR_KUBERNETES_NODE_NAME environment variable.


    - name: EXECUTOR_KUBERNETES_NODE_NAME
      valueFrom:
        fieldRef:
          fieldPath: spec.nodeName

This ensures that the Job is scheduled on the same Node as the Executor.

However, if the node does not have enough resources to run the Job, the Job will not be scheduled.

Firewall Rules

The following are Firewall rules that are highly recommended when running Executors on Kubernetes in a Cloud Environment.

Disable access to internal resources.
Disable access to 169.254.169.254 (AWS / GCP Instance Metadata Service).

Batch Changes

To run Batch Changes on Kubernetes, Native Server-Side Batch Changes must be enabled.

Example Configuration YAML

See the local development YAMLs for an example of how to configure the Executor in Kubernetes.

Deployment

Executors on Kubernetes require specific RBAC rules to be configured in order to operate correctly. See RBAC Roles for more information.

Step-by-step Guide

Ensure you have the following tools installed:

kubectl
Helm if you're installing Sourcegraph with helm.

Deployment via Kustomize

Please refer to the Sourcegraph Kustomize docs for the latest instructions.

To include Native Kubernetes Executors, see configure Sourcegraph with Kustomize on how to specify the component (components/executors/k8s).

Deployment via Helm

Please refer to the Sourcegraph Helm docs for the latest instructions.

To specifically deploy Executors,

Create an overrides file, override.yaml, with any other customizations you may require.
1. See details on configurations.

Run the following command:


helm upgrade --install --values ./override.yaml --version <your Sourcegraph Version> sg-executor sourcegraph/sourcegraph-executor-k8s

Confirm executors are working by checking the Executors page under Site admin > Executors > Instances .

Note

Executors deployed on Kubernetes do not use Firecracker.

If you have security concerns, consider deploying via terraform or installing the binary directly.