Compare commits
20 Commits
better-tes
...
add-docs
| Author | SHA1 | Date |
|---|---|---|
 Charles Reid
|
9c909a2fb8 | 6 years ago |
|
Charles Reid
|
08a3250203 | 6 years ago |
|
Charles Reid
|
b381654c71 | 6 years ago |
|
Charles Reid
|
266a4d00ac | 6 years ago |
|
Charles Reid
|
95b79ef15f | 6 years ago |
|
Charles Reid
|
b2ab9ed8a2 | 6 years ago |
|
Charles Reid
|
7debeb24d6 | 6 years ago |
|
Charles Reid
|
c23f0656de | 6 years ago |
|
Charles Reid
|
064d98c6ea | 6 years ago |
|
Charles Reid
|
4cb9d64f25 | 6 years ago |
|
Charles Reid
|
38fa8a624e | 6 years ago |
|
Charles Reid
|
41e1a23009 | 6 years ago |
|
Charles Reid
|
58cfb0cb5a | 6 years ago |
|
Charles Reid
|
312621a9cd | 6 years ago |
|
Charles Reid
|
b8ba8ba860 | 6 years ago |
|
Charles Reid
|
1934a04bce | 6 years ago |
|
Charles Reid
|
249b9f05cd | 6 years ago |
|
Charles Reid
|
a2b2c0ddd4 | 6 years ago |
|
Charles Reid
|
ac9b351d4c | 6 years ago |
|
Charles Reid
|
2b8b7855d6 | 6 years ago |
20 changed files with 968 additions and 190 deletions
@ -0,0 +1,3 @@ |
|||||||
|
[submodule "mkdocs-material-dib"] |
||||||
|
path = mkdocs-material-dib |
||||||
|
url = https://github.com/dib-lab/mkdocs-material-dib.git |
||||||
@ -0,0 +1,11 @@ |
|||||||
|
.md-typeset h1 { font-weight: 600; } |
||||||
|
.md-typeset h2 { font-weight: 600; } |
||||||
|
.md-typeset h3 { font-weight: 600; } |
||||||
|
.md-typeset h4 { font-weight: 600; } |
||||||
|
|
||||||
|
body { |
||||||
|
background-color: #FAFAFA; |
||||||
|
} |
||||||
|
div.body { |
||||||
|
background-color: #FAFAFA; |
||||||
|
} |
||||||
@ -0,0 +1,125 @@ |
|||||||
|
# 2019-snakemake-byok8s |
||||||
|
|
||||||
|
[](https://travis-ci.org/charlesreid1/2019-snakemake-byok8s) |
||||||
|
[](https://github.com/charlesreid1/2019-snakemake-byok8s/blob/master/LICENSE) |
||||||
|
 |
||||||
|
 |
||||||
|
 |
||||||
|
 |
||||||
|
|
||||||
|
# Overview |
||||||
|
|
||||||
|
This is an example of a Snakemake workflow that: |
||||||
|
|
||||||
|
- is a **command line utility** called `byok8s` |
||||||
|
- is bundled as an installable **Python package** |
||||||
|
- is designed to run on a **Kubernetes (k8s) cluster** |
||||||
|
- can be **tested with Travis CI** (and/or locally) using [minikube](https://github.com/kubernetes/minikube) |
||||||
|
|
||||||
|
## What is byok8s? |
||||||
|
|
||||||
|
byok8s = Bring Your Own Kubernetes (cluster) |
||||||
|
|
||||||
|
k8s = kubernetes |
||||||
|
|
||||||
|
byok8s is a command line utility that launches |
||||||
|
a Snakemake workflow on an existing Kubernetes |
||||||
|
cluster. This allows you to do something |
||||||
|
like this (also see the [Installation](installing.md) |
||||||
|
and [Quickstart](quickstart.md) guides in the |
||||||
|
documentation): |
||||||
|
|
||||||
|
``` |
||||||
|
# Install byok8s |
||||||
|
python setup.py build install |
||||||
|
|
||||||
|
# Create virtual k8s cluster |
||||||
|
minikube start |
||||||
|
|
||||||
|
# Run the workflow on the k8s cluster |
||||||
|
cd /path/to/workflow/ |
||||||
|
byok8s my-workflowfile my-paramsfile --s3-bucket=my-bucket |
||||||
|
|
||||||
|
# Clean up the virtual k8s cluster |
||||||
|
minikube stop |
||||||
|
``` |
||||||
|
|
||||||
|
## Getting Up and Running |
||||||
|
|
||||||
|
See the [Quickstart Guide](quickstart.md) to get up and |
||||||
|
running with byok8s. |
||||||
|
|
||||||
|
## How does byok8s work? |
||||||
|
|
||||||
|
The command line utility requires the user to provide |
||||||
|
three input files: |
||||||
|
|
||||||
|
* A snakemake workflow, via a `Snakefile` |
||||||
|
* A workflow configuration file (JSON) |
||||||
|
* A workflow parameters file (JSON) |
||||||
|
|
||||||
|
Additionally, the user must create the following resources: |
||||||
|
|
||||||
|
* A kubernetes cluster up and running |
||||||
|
* An S3 bucket (and AWS credentials to read/write) |
||||||
|
|
||||||
|
A sample Snakefile, workflow config file, and workflow |
||||||
|
params file are provided in the `test/` directory. |
||||||
|
|
||||||
|
The workflow config file specifies which workflow targets |
||||||
|
and input files to use. |
||||||
|
|
||||||
|
The workflow parameters file specifies which parameters to |
||||||
|
use for the workflow steps. |
||||||
|
|
||||||
|
## Why S3 buckets? |
||||||
|
|
||||||
|
AWS credentials and an S3 bucket is required to run workflows because |
||||||
|
of restrictions on file I/O on nodes in a kubernes cluster. The Snakemake |
||||||
|
workflows use AWS S3 buckets as remote providers for the Kubernetes nodes, |
||||||
|
but this can be modified to any others that Snakemake supports. |
||||||
|
|
||||||
|
AWS credentials are set with the two environment variables: |
||||||
|
|
||||||
|
``` |
||||||
|
AWS_ACCESS_KEY_ID |
||||||
|
AWS_SECRET_ACCESS_KEY |
||||||
|
``` |
||||||
|
|
||||||
|
These are passed into the Kubernetes cluster by byok8s and Snakemake. |
||||||
|
|
||||||
|
## Kubernetes and Minikube |
||||||
|
|
||||||
|
[Kubernetes](https://kubernetes.io/) is a technology that utilizes Docker |
||||||
|
container to orchestrate a cluster of compute nodes. These compute nodes are |
||||||
|
usually real compute nodes requested and managed via a cloud provider, like AWS |
||||||
|
or Google Cloud. |
||||||
|
|
||||||
|
But the compute nodes can also be virtual, which is where |
||||||
|
[minikube](https://github.com/kubernetes/minikube) comes in. It creates a |
||||||
|
kubernetes cluster that is entirely local and virtual, which makes testing |
||||||
|
easy. See the [byok8s Minikube Guide](kubernetes_minikube.md) for details |
||||||
|
about how to use minikube with byok8s. |
||||||
|
|
||||||
|
The Travis CI tests also utilize minikube to run test workflows. See [byok8s |
||||||
|
Travis Tests](travis_tests.md) for more information. |
||||||
|
|
||||||
|
## Cloud Providers |
||||||
|
|
||||||
|
For real workflows, your options for |
||||||
|
kubernetes clusters are cloud providers. |
||||||
|
We have guides for the following: |
||||||
|
|
||||||
|
- AWS EKS (Elastic Container Service) |
||||||
|
- GCP GKE (Google Kuberntes Engine) |
||||||
|
- Digital Ocean Kubernetes service |
||||||
|
|
||||||
|
# Kubernetes + byok8s: In Practice |
||||||
|
|
||||||
|
| Cloud Provider | Kubernetes Service | Guide | State | |
||||||
|
|-----------------------------|---------------------------------|-------------------------------------------------|------------| |
||||||
|
| Minikube (on AWS EC2) | Minikube | [byok8s Minikube Guide](kubernetes_minikube.md) | Finished | |
||||||
|
| Google Cloud Platform (GCP) | Google Container Engine (GKE) | [byok8s GCP GKE Guide](kubernetes_gcp.md) | Finished | |
||||||
|
| Amazon Web Services (AWS) | Elastic Container Service (EKS) | [byok8s AWS EKS Guide](kubernetes_aws.md) | Unfinished | |
||||||
|
| Digital Ocean (DO) | DO Kubernetes (DOK) | [byok8s DO DOK Guide](kubernetes_dok.md) | Unfinished | |
||||||
|
|
||||||
@ -0,0 +1,114 @@ |
|||||||
|
# Installing byok8s |
||||||
|
|
||||||
|
byok8s requires two pieces of prerequisite software: |
||||||
|
|
||||||
|
- python (conda) |
||||||
|
- virtualenv (optional) |
||||||
|
|
||||||
|
It also requires an AWS S3 bucket to be specified |
||||||
|
(the bucket must exist and credentials to access it |
||||||
|
must be provided via environment variables, see the |
||||||
|
[Quickstart](quickstart.md)). |
||||||
|
|
||||||
|
Additionally, if you are planning to run byok8s on |
||||||
|
a local virtual kubernetes cluster, you must install: |
||||||
|
|
||||||
|
- minikube |
||||||
|
|
||||||
|
Otherwise, if you are planning on running byok8s on |
||||||
|
remote kubernetes clusters provided by cloud providers, |
||||||
|
you must install: |
||||||
|
|
||||||
|
- kubernetes, ***OR*** |
||||||
|
- a cloud provider command line tool (`gcloud`, `aws`) |
||||||
|
|
||||||
|
## Installing Python |
||||||
|
|
||||||
|
We recommend installing pyenv and using pyenv |
||||||
|
to install miniconda: |
||||||
|
|
||||||
|
```plain |
||||||
|
curl https://pyenv.run | bash |
||||||
|
``` |
||||||
|
|
||||||
|
Restart your shell and install miniconda: |
||||||
|
|
||||||
|
```plain |
||||||
|
pyenv update |
||||||
|
pyenv install miniconda3-4.3.30 |
||||||
|
pyenv global miniconda3-4.3.30 |
||||||
|
``` |
||||||
|
|
||||||
|
## Installing virtualenv |
||||||
|
|
||||||
|
You will need the virtualenv package to |
||||||
|
set up a virtual environment: |
||||||
|
|
||||||
|
```plain |
||||||
|
pip install virtualenv |
||||||
|
``` |
||||||
|
|
||||||
|
## Installing minikube |
||||||
|
|
||||||
|
This step is only required if you plan to run byok8s |
||||||
|
kubernetes workflows locally on a virtual kubernetes |
||||||
|
cluster (i.e., testing mode). |
||||||
|
|
||||||
|
Install the 64-bit Linux version of minikube, or visit the |
||||||
|
[installing minikube](https://kubernetes.io/docs/tasks/tools/install-minikube/) |
||||||
|
to find the right version: |
||||||
|
|
||||||
|
```plain |
||||||
|
curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 \ |
||||||
|
&& sudo install minikube-linux-amd64 /usr/local/bin/minikube |
||||||
|
``` |
||||||
|
|
||||||
|
(On a Mac you can do `brew install minikube`.) |
||||||
|
|
||||||
|
If you are planning on running on a bare metal |
||||||
|
machine, you will also need to install a hypervisor |
||||||
|
like VirtualBox or KVM, see [installing minikube](https://kubernetes.io/docs/tasks/tools/install-minikube/). |
||||||
|
|
||||||
|
If you are planning on running minikube on a compute |
||||||
|
node in the cloud, you cannot run a hypervisor, so you |
||||||
|
will need to run using the native driver; see |
||||||
|
[installing minikube](https://kubernetes.io/docs/tasks/tools/install-minikube/). |
||||||
|
|
||||||
|
Once you have installed minikube, you do not need to |
||||||
|
install kubernetes. |
||||||
|
|
||||||
|
## Installing byok8s |
||||||
|
|
||||||
|
Start by cloning the repo and installing byok8s: |
||||||
|
|
||||||
|
```plain |
||||||
|
cd |
||||||
|
git clone https://github.com/charlesreid1/2019-snakemake-byok8s.git |
||||||
|
cd ~/2019-snakemake-byok8s |
||||||
|
``` |
||||||
|
|
||||||
|
Next, you'll create a virtual environment: |
||||||
|
|
||||||
|
```plain |
||||||
|
virtualenv vp |
||||||
|
source vp/bin/activate |
||||||
|
|
||||||
|
pip install -r requirements.txt |
||||||
|
python setup.py build install |
||||||
|
``` |
||||||
|
|
||||||
|
Now you should be ready to rock: |
||||||
|
|
||||||
|
``` |
||||||
|
which byok8s |
||||||
|
``` |
||||||
|
|
||||||
|
This will only be present when you have activated |
||||||
|
your virtual environment. To activate/re-activate your |
||||||
|
virtual environment: |
||||||
|
|
||||||
|
``` |
||||||
|
cd ~/2019-snakemake-byok8s |
||||||
|
source vp/bin/activate |
||||||
|
``` |
||||||
|
|
||||||
@ -0,0 +1,4 @@ |
|||||||
|
# Kubernetes on AWS |
||||||
|
|
||||||
|
Check back soon for an EKS guide! |
||||||
|
|
||||||
@ -1,11 +1,9 @@ |
|||||||
# Kubernetes on Digital Ocean |
# Kubernetes on Digital Ocean |
||||||
|
|
||||||
## Digital Ocean Kubernetes |
Check back soon for a Digital Ocean kubernetes guide! |
||||||
|
|
||||||
(Use web interface to set up a Kubernetes cluster, |
(Use web interface to set up a Kubernetes cluster, |
||||||
then use `kubectl` to connect with Digital Ocean |
then use `kubectl` to connect with Digital Ocean |
||||||
via Digital Ocean credentials.) |
via Digital Ocean credentials.) |
||||||
|
|
||||||
## Quickstart |
|
||||||
|
|
||||||
[link](https://www.digitalocean.com/docs/kubernetes/how-to/connect-with-kubectl/) |
[link](https://www.digitalocean.com/docs/kubernetes/how-to/connect-with-kubectl/) |
||||||
@ -0,0 +1,263 @@ |
|||||||
|
# Kubernetes on Google Cloud Platform |
||||||
|
|
||||||
|
This document will walk you through how to start a kubernetes cluster using the |
||||||
|
Google Kubernetes Engine (GKE) on Google Cloud Platform (GCP), run the byok8s |
||||||
|
Snakemake workflow on the GKE kubernetes cluster, and tear down the cluster |
||||||
|
when the workflow is complete. |
||||||
|
|
||||||
|
## Setup |
||||||
|
|
||||||
|
Before you can create a kubernetes cluster on Google Cloud, |
||||||
|
you need a Google Cloud account and a Google Cloud project. |
||||||
|
You can sign up for a Google Cloud account [here](https://cloud.google.com/). |
||||||
|
You can create a new project from the [Google Cloud Console](https://console.cloud.google.com/). |
||||||
|
New accounts start with 300 free hours specifically to let you |
||||||
|
test drive features like GKE! Cool! |
||||||
|
|
||||||
|
Once you have your account and your project, you can install |
||||||
|
the `gcloud` Google Cloud SDK command line utility |
||||||
|
(see [Google Cloud SDK Quickstart Guide](https://cloud.google.com/sdk/docs/quickstarts)). |
||||||
|
|
||||||
|
Once you have installed the `gcloud` utility, you will need |
||||||
|
to log in with your Google acount using the `init` command: |
||||||
|
|
||||||
|
``` |
||||||
|
gcloud init |
||||||
|
``` |
||||||
|
|
||||||
|
This will give you a link to enter into your browser, where |
||||||
|
you will log in with your Google account and recieve a code to |
||||||
|
copy and paste into the terminal. |
||||||
|
|
||||||
|
The **Compute API** and **Kubernetes API** will both need to be |
||||||
|
enabled as well. These can be enabled via the |
||||||
|
[Google Cloud Console](https://console.cloud.google.com/) |
||||||
|
(or read on). |
||||||
|
|
||||||
|
If you aren't sure how to use the console to enable these APIs, just start |
||||||
|
running the commands below to create a kubernetes cluster, and the gcloud |
||||||
|
utility will let you know if it needs APIs enabled for actions. If it can't |
||||||
|
enable the API for you, it will give you a direct link to the relevant Google |
||||||
|
Cloud Console page. |
||||||
|
|
||||||
|
## Google Kubernetes Engine |
||||||
|
|
||||||
|
GKE uses Google Cloud compute nodes to run a kubernetes cluster |
||||||
|
on Google Cloud infrastructure. It automatically sets up the |
||||||
|
cluster for you, and allows you to use `kubectl` and `gcloud` to |
||||||
|
manage and interact with the remote cluster. |
||||||
|
|
||||||
|
Official Google link: <https://cloud.google.com/kubernetes-engine/> |
||||||
|
|
||||||
|
## Quickstart |
||||||
|
|
||||||
|
As mentioned, make sure your account credentials are initialized: |
||||||
|
|
||||||
|
``` |
||||||
|
gcloud init |
||||||
|
``` |
||||||
|
|
||||||
|
Create a new GKE cluster: |
||||||
|
|
||||||
|
``` |
||||||
|
gcloud container clusters create $CLUSTER_NAME --num-nodes=$NODES --region=us-west1 |
||||||
|
``` |
||||||
|
|
||||||
|
The `--scopes storage-rw` flag is required if you plan to use Google |
||||||
|
Cloud buckets instead of S3 buckets (not currently enabled in byok8s). |
||||||
|
|
||||||
|
Next get configuration details about the cluster so your local |
||||||
|
kubernetes controller can control the cluster: |
||||||
|
|
||||||
|
``` |
||||||
|
gcloud container clusters get-credentials $CLUSTER_NAME |
||||||
|
``` |
||||||
|
|
||||||
|
**This will take several minutes.** |
||||||
|
|
||||||
|
The cluster should now be up and running and ready to rock: |
||||||
|
|
||||||
|
``` |
||||||
|
$ kubectl get pods --namespace=kube-system |
||||||
|
NAME READY STATUS RESTARTS AGE |
||||||
|
event-exporter-v0.2.3-54f94754f4-5jczv 2/2 Running 0 4m |
||||||
|
fluentd-gcp-scaler-6d7bbc67c5-hkllz 1/1 Running 0 4m |
||||||
|
fluentd-gcp-v3.1.0-48pb2 2/2 Running 0 2m |
||||||
|
fluentd-gcp-v3.1.0-58dpx 2/2 Running 0 2m |
||||||
|
fluentd-gcp-v3.1.0-c4b49 2/2 Running 0 2m |
||||||
|
fluentd-gcp-v3.1.0-h24m5 2/2 Running 0 2m |
||||||
|
fluentd-gcp-v3.1.0-hbdj4 2/2 Running 0 2m |
||||||
|
fluentd-gcp-v3.1.0-rfnmt 2/2 Running 0 2m |
||||||
|
fluentd-gcp-v3.1.0-vwd8w 2/2 Running 0 2m |
||||||
|
fluentd-gcp-v3.1.0-wxt79 2/2 Running 0 2m |
||||||
|
fluentd-gcp-v3.1.0-xkt42 2/2 Running 0 2m |
||||||
|
heapster-v1.5.3-bc9f6bfd5-7jhqs 3/3 Running 0 3m |
||||||
|
kube-dns-788979dc8f-l7hch 4/4 Running 0 4m |
||||||
|
kube-dns-788979dc8f-pts99 4/4 Running 0 3m |
||||||
|
kube-dns-autoscaler-79b4b844b9-j48js 1/1 Running 0 4m |
||||||
|
kube-proxy-gke-mycluster-default-pool-9ad2912e-130p 1/1 Running 0 4m |
||||||
|
kube-proxy-gke-mycluster-default-pool-9ad2912e-lfpw 1/1 Running 0 4m |
||||||
|
kube-proxy-gke-mycluster-default-pool-9ad2912e-rt9m 1/1 Running 0 4m |
||||||
|
kube-proxy-gke-mycluster-default-pool-b44fa389-2ds8 1/1 Running 0 4m |
||||||
|
kube-proxy-gke-mycluster-default-pool-b44fa389-hc66 1/1 Running 0 4m |
||||||
|
kube-proxy-gke-mycluster-default-pool-b44fa389-vh3x 1/1 Running 0 4m |
||||||
|
kube-proxy-gke-mycluster-default-pool-d58ee1e7-2kkw 1/1 Running 0 4m |
||||||
|
kube-proxy-gke-mycluster-default-pool-d58ee1e7-3l6r 1/1 Running 0 4m |
||||||
|
kube-proxy-gke-mycluster-default-pool-d58ee1e7-4w18 1/1 Running 0 4m |
||||||
|
l7-default-backend-5d5b9874d5-ms75l 1/1 Running 0 4m |
||||||
|
metrics-server-v0.2.1-7486f5bd67-2n6cn 2/2 Running 0 3m |
||||||
|
``` |
||||||
|
|
||||||
|
Now assuming you have installed `byok8s` and it is located |
||||||
|
at `~/2019-snakemake-byok8s/`, you can run the test workflow |
||||||
|
on the kubernetes cluster: |
||||||
|
|
||||||
|
``` |
||||||
|
# Return to our virtual environment |
||||||
|
cd ~/2019-snakemake-byok8s/test/ |
||||||
|
source vp/bin/activate |
||||||
|
|
||||||
|
# Export AWS keys for Snakemake |
||||||
|
export AWS_ACCESS_KEY_ID="XXXXX" |
||||||
|
export AWS_SECRET_ACCESS_KEY="XXXXX" |
||||||
|
|
||||||
|
# Run byok8s |
||||||
|
byok8s workflow-alpha params-blue --s3-bucket=mah-bukkit |
||||||
|
``` |
||||||
|
|
||||||
|
Once the workflow has run successfully, the results will be written |
||||||
|
to S3 buckets and all the kubernetes containers created by snakemake |
||||||
|
will be gone. |
||||||
|
|
||||||
|
If all goes well, you should see output like this: |
||||||
|
|
||||||
|
``` |
||||||
|
$ byok8s --s3-bucket=mah-bukkit -f workflow-alpha params-blue |
||||||
|
-------- |
||||||
|
details! |
||||||
|
snakefile: /home/ubuntu/2019-snakemake-byok8s/test/Snakefile |
||||||
|
config: /home/ubuntu/2019-snakemake-byok8s/test/workflow-alpha.json |
||||||
|
params: /home/ubuntu/2019-snakemake-byok8s/test/params-blue.json |
||||||
|
target: target1 |
||||||
|
k8s namespace: default |
||||||
|
-------- |
||||||
|
Building DAG of jobs... |
||||||
|
Using shell: /bin/bash |
||||||
|
Provided cores: 1 |
||||||
|
Rules claiming more threads will be scaled down. |
||||||
|
Job counts: |
||||||
|
count jobs |
||||||
|
1 target1 |
||||||
|
1 |
||||||
|
Resources before job selection: {'_cores': 1, '_nodes': 9223372036854775807} |
||||||
|
Ready jobs (1): |
||||||
|
target1 |
||||||
|
Selected jobs (1): |
||||||
|
target1 |
||||||
|
Resources after job selection: {'_cores': 0, '_nodes': 9223372036854775806} |
||||||
|
|
||||||
|
[Mon Jan 28 23:49:51 2019] |
||||||
|
rule target1: |
||||||
|
output: cmr-0123/alpha.txt |
||||||
|
jobid: 0 |
||||||
|
|
||||||
|
echo alpha blue > cmr-0123/alpha.txt |
||||||
|
Get status with: |
||||||
|
kubectl describe pod snakejob-1ab52bdb-903b-5506-b712-ccc86772dc8d |
||||||
|
kubectl logs snakejob-1ab52bdb-903b-5506-b712-ccc86772dc8d |
||||||
|
Checking status for pod snakejob-1ab52bdb-903b-5506-b712-ccc86772dc8d |
||||||
|
Checking status for pod snakejob-1ab52bdb-903b-5506-b712-ccc86772dc8d |
||||||
|
Checking status for pod snakejob-1ab52bdb-903b-5506-b712-ccc86772dc8d |
||||||
|
Checking status for pod snakejob-1ab52bdb-903b-5506-b712-ccc86772dc8d |
||||||
|
Checking status for pod snakejob-1ab52bdb-903b-5506-b712-ccc86772dc8d |
||||||
|
[Mon Jan 28 23:50:41 2019] |
||||||
|
Finished job 0. |
||||||
|
1 of 1 steps (100%) done |
||||||
|
Complete log: /home/ubuntu/2019-snakemake-byok8s/test/.snakemake/log/2019-01-28T234950.253823.snakemake.log |
||||||
|
unlocking |
||||||
|
removing lock |
||||||
|
removing lock |
||||||
|
removed all locks |
||||||
|
``` |
||||||
|
|
||||||
|
Congratulations! You'e just run an executable Snakemake workflow |
||||||
|
on a Google Cloud kubernetes cluster! |
||||||
|
|
||||||
|
You can get more information about the containers running each step of |
||||||
|
the workflow using the `kubectl describe` commands printed in the output. |
||||||
|
Here is an example: |
||||||
|
|
||||||
|
``` |
||||||
|
$ kubectl describe pod snakejob-c91f804c-805a-56a2-b0ea-b3b74bc38001 |
||||||
|
Name: snakejob-c91f804c-805a-56a2-b0ea-b3b74bc38001 |
||||||
|
Namespace: default |
||||||
|
Node: gke-mycluster-default-pool-b44fa389-vh3x/10.138.0.7 |
||||||
|
Start Time: Mon, 28 Jan 2019 23:55:18 -0800 |
||||||
|
Labels: app=snakemake |
||||||
|
Annotations: <none> |
||||||
|
Status: Running |
||||||
|
IP: 10.0.6.4 |
||||||
|
Containers: |
||||||
|
snakejob-c91f804c-805a-56a2-b0ea-b3b74bc38001: |
||||||
|
Container ID: docker://2aaa04c34770c6088334b29c0332dc426aff2fbbd3a8af07b65bbbc2c5fe437d |
||||||
|
Image: quay.io/snakemake/snakemake:v5.4.0 |
||||||
|
Image ID: docker-pullable://quay.io/snakemake/snakemake@sha256:f5bb7bef99c4e45cb7dfd5b55535b8dc185b43ca610341476378a9566a8b52c5 |
||||||
|
Port: <none> |
||||||
|
Host Port: <none> |
||||||
|
Command: |
||||||
|
/bin/sh |
||||||
|
Args: |
||||||
|
-c |
||||||
|
cp -rf /source/. . && snakemake cmr-0123/.zetaB1 --snakefile Snakefile --force -j --keep-target-files --keep-remote --latency-wait 0 --attempt 1 --force-use-threads --wrapper-prefix None --config 'name='"'"'blue'"'"'' -p --nocolor --notemp --no-hooks --nolock --default-remote-provider S3 --default-remote-prefix cmr-0123 --allowed-rules target3sleepyB1 |
||||||
|
State: Running |
||||||
|
Started: Mon, 28 Jan 2019 23:56:15 -0800 |
||||||
|
Ready: True |
||||||
|
Restart Count: 0 |
||||||
|
Requests: |
||||||
|
cpu: 0 |
||||||
|
Environment: |
||||||
|
AWS_ACCESS_KEY_ID: <set to the key 'aws_access_key_id' in secret 'e077a45f-1274-4a98-a76c-d1a9718707db'> Optional: false |
||||||
|
AWS_SECRET_ACCESS_KEY: <set to the key 'aws_secret_access_key' in secret 'e077a45f-1274-4a98-a76c-d1a9718707db'> Optional: false |
||||||
|
Mounts: |
||||||
|
/source from source (rw) |
||||||
|
/var/run/secrets/kubernetes.io/serviceaccount from default-token-jmnv4 (ro) |
||||||
|
Conditions: |
||||||
|
Type Status |
||||||
|
Initialized True |
||||||
|
Ready True |
||||||
|
PodScheduled True |
||||||
|
Volumes: |
||||||
|
source: |
||||||
|
Type: Secret (a volume populated by a Secret) |
||||||
|
SecretName: e077a45f-1274-4a98-a76c-d1a9718707db |
||||||
|
Optional: false |
||||||
|
workdir: |
||||||
|
Type: EmptyDir (a temporary directory that shares a pod's lifetime) |
||||||
|
Medium: |
||||||
|
default-token-jmnv4: |
||||||
|
Type: Secret (a volume populated by a Secret) |
||||||
|
SecretName: default-token-jmnv4 |
||||||
|
Optional: false |
||||||
|
QoS Class: BestEffort |
||||||
|
Node-Selectors: <none> |
||||||
|
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s |
||||||
|
node.kubernetes.io/unreachable:NoExecute for 300s |
||||||
|
Events: |
||||||
|
Type Reason Age From Message |
||||||
|
---- ------ ---- ---- ------- |
||||||
|
Normal Scheduled 63s default-scheduler Successfully assigned snakejob-c91f804c-805a-56a2-b0ea-b3b74bc38001 to gke-mycluster-default-pool-b44fa389-vh3x |
||||||
|
Normal SuccessfulMountVolume 63s kubelet, gke-mycluster-default-pool-b44fa389-vh3x MountVolume.SetUp succeeded for volume "workdir" |
||||||
|
Normal SuccessfulMountVolume 63s kubelet, gke-mycluster-default-pool-b44fa389-vh3x MountVolume.SetUp succeeded for volume "default-token-jmnv4" |
||||||
|
Normal SuccessfulMountVolume 63s kubelet, gke-mycluster-default-pool-b44fa389-vh3x MountVolume.SetUp succeeded for volume "source" |
||||||
|
Normal Pulling 61s kubelet, gke-mycluster-default-pool-b44fa389-vh3x pulling image "quay.io/snakemake/snakemake:v5.4.0" |
||||||
|
Normal Pulled 10s kubelet, gke-mycluster-default-pool-b44fa389-vh3x Successfully pulled image "quay.io/snakemake/snakemake:v5.4.0" |
||||||
|
Normal Created 6s kubelet, gke-mycluster-default-pool-b44fa389-vh3x Created container |
||||||
|
Normal Started 6s kubelet, gke-mycluster-default-pool-b44fa389-vh3x Started container |
||||||
|
``` |
||||||
|
|
||||||
|
Delete the GKE cluster when you are done: |
||||||
|
|
||||||
|
``` |
||||||
|
gcloud container clusters delete $CLUSTER_NAME |
||||||
|
``` |
||||||
|
|
||||||
@ -0,0 +1,115 @@ |
|||||||
|
# Running byok8s with minikube |
||||||
|
|
||||||
|
## Installing |
||||||
|
|
||||||
|
See the [Installing](installing.md) page for details |
||||||
|
about installing byok8s and its prerequisites |
||||||
|
(including minikube). |
||||||
|
|
||||||
|
We cover two scenarios: |
||||||
|
|
||||||
|
- bare metal machine, i.e., a laptop or desktop machine |
||||||
|
that can run a hypervisor like VirtualBox |
||||||
|
|
||||||
|
- cloud machine, i.e., AWS EC2 node, which is itself a |
||||||
|
virtual machine and cannot run a hypervisor |
||||||
|
|
||||||
|
These quickstarts assume you have Python and minikube |
||||||
|
installed, and that you have cloned and installed byok8s |
||||||
|
at `~/2019-snakemake-byok8s/`. |
||||||
|
|
||||||
|
## Quickstart on Bare Metal Machine |
||||||
|
|
||||||
|
On a bare metal machine, the procedure is |
||||||
|
relatively uncomplicated: we create a cluster, |
||||||
|
we export some variables, we run the workflow, |
||||||
|
we tear down the cluster: |
||||||
|
|
||||||
|
```plain |
||||||
|
# Start a minikube cluster |
||||||
|
minikube start |
||||||
|
|
||||||
|
# Verify k8s is running |
||||||
|
minikube status |
||||||
|
|
||||||
|
# Export AWS credentials |
||||||
|
export AWS_ACCESS_KEY_ID="XXXXX" |
||||||
|
export AWS_SECRET_ACCESS_KEY="XXXXX" |
||||||
|
|
||||||
|
# Run the workflow |
||||||
|
byok8s workflow-alpha params-blue --s3-bucket=mah-bukkit |
||||||
|
|
||||||
|
# Stop the minikube cluster |
||||||
|
minikube stop |
||||||
|
``` |
||||||
|
|
||||||
|
## Quickstart on Cloud Machine |
||||||
|
|
||||||
|
As mentioned above, cloud compute nodes are virtual machines |
||||||
|
themselves and cannot run a hypervisor, so things are a bit |
||||||
|
more complicated. |
||||||
|
|
||||||
|
To tell minikube not to use a virtual machine driver, |
||||||
|
run the following command in a terminal to create |
||||||
|
a minikube config file: |
||||||
|
|
||||||
|
``` |
||||||
|
cat <<'EOF' > ~/.minikube/config/config.json |
||||||
|
{ |
||||||
|
"vm-driver": "none" |
||||||
|
} |
||||||
|
EOF |
||||||
|
``` |
||||||
|
|
||||||
|
Now you can start up a minikube cluster. |
||||||
|
|
||||||
|
There is an additional DNS problem that needs to be fixed |
||||||
|
in the containers before you proceed. You will know there |
||||||
|
is a problem if you run the `get pods` command with |
||||||
|
`kubectl` and see your CoreDNS containers in a |
||||||
|
`CrashLoopBackOff` state: |
||||||
|
|
||||||
|
```text |
||||||
|
$ kubectl get pods --namespace=kube-system |
||||||
|
NAME READY STATUS RESTARTS AGE |
||||||
|
coredns-86c58d9df4-lvq8b 0/1 CrashLoopBackOff 5 5m17s |
||||||
|
coredns-86c58d9df4-pr52t 0/1 CrashLoopBackOff 5 5m17s |
||||||
|
... ... ... ... ... |
||||||
|
``` |
||||||
|
|
||||||
|
To fix the problem with the DNS settings, we have to patch |
||||||
|
the CoreDNS image being used by `kube-system`. |
||||||
|
To do that, use the file |
||||||
|
[`test/fixcoredns.yml`](https://github.com/charlesreid1/2019-snakemake-byok8s/blob/master/test/fixcoredns.yml) |
||||||
|
in this repository with `kubectl apply`: |
||||||
|
|
||||||
|
```plain |
||||||
|
# Fix the DNS container |
||||||
|
kubectl apply -f fixcoredns.yml |
||||||
|
|
||||||
|
# Delete all kube-system containers |
||||||
|
kubectl delete --all pods --namespace kube-system |
||||||
|
``` |
||||||
|
|
||||||
|
The kube-system containers will be re-spawned by the cluster control system. |
||||||
|
It should happen in a few seconds, and then you'll be ready to run byok8s: |
||||||
|
|
||||||
|
``` |
||||||
|
# Return to our virtual environment |
||||||
|
cd ~/2019-snakemake-byok8s/test/ |
||||||
|
source vp/bin/activate |
||||||
|
|
||||||
|
# Verify k8s is running |
||||||
|
minikube status |
||||||
|
|
||||||
|
# Export AWS keys for Snakemake |
||||||
|
export AWS_ACCESS_KEY_ID="XXXXX" |
||||||
|
export AWS_SECRET_ACCESS_KEY="XXXXX" |
||||||
|
|
||||||
|
# Run byok8s |
||||||
|
byok8s workflow-alpha params-blue --s3-bucket=mah-bukkit |
||||||
|
``` |
||||||
|
|
||||||
|
Congratulations! You've just run an executable Snakemake workflow |
||||||
|
on a minikube kubernetes cluster. |
||||||
|
|
||||||
@ -0,0 +1,155 @@ |
|||||||
|
# Quickstart |
||||||
|
|
||||||
|
This runs through the installation and usage |
||||||
|
of `2019-snakemake-byok8s`. |
||||||
|
|
||||||
|
Step 1: Set up Kubernetes cluster with `minikube`. |
||||||
|
|
||||||
|
Step 2: Install `byok8s`. |
||||||
|
|
||||||
|
Step 3: Run the `byok8s` workflow using the Kubernetes cluster. |
||||||
|
|
||||||
|
Step 4: Tear down Kubernetes cluster with `minikube`. |
||||||
|
|
||||||
|
|
||||||
|
## Step 1: Set Up Virtual Kubernetes Cluster |
||||||
|
|
||||||
|
For the purposes of the quickstart, we will walk |
||||||
|
through how to set up a local, virtual Kubernetes |
||||||
|
cluster using `minikube`. |
||||||
|
|
||||||
|
Start by installing minikube: |
||||||
|
|
||||||
|
``` |
||||||
|
scripts/install_minikube.sh |
||||||
|
``` |
||||||
|
|
||||||
|
Once it is installed, you can start up a kubernetes cluster |
||||||
|
with minikube using the following commands: |
||||||
|
|
||||||
|
``` |
||||||
|
cd test |
||||||
|
minikube start |
||||||
|
``` |
||||||
|
|
||||||
|
NOTE: If you are running on AWS, run this command first |
||||||
|
|
||||||
|
``` |
||||||
|
minikube config set vm-driver none |
||||||
|
``` |
||||||
|
|
||||||
|
to set the the vm driver to none and use native Docker to run stuff. |
||||||
|
|
||||||
|
If you are running on AWS, the DNS in the minikube |
||||||
|
kubernetes cluster will not work, so run this command |
||||||
|
to fix the DNS settings (should be run from the |
||||||
|
`test/` directory): |
||||||
|
|
||||||
|
``` |
||||||
|
kubectl apply -f fixcoredns.yml |
||||||
|
kubectl delete --all pods --namespace kube-system |
||||||
|
``` |
||||||
|
|
||||||
|
|
||||||
|
## Step 2: Install byok8s |
||||||
|
|
||||||
|
Start by setting up a python virtual environment, |
||||||
|
and install the required packages into the |
||||||
|
virtual environment: |
||||||
|
|
||||||
|
``` |
||||||
|
pip install -r requirements.txt |
||||||
|
``` |
||||||
|
|
||||||
|
This installs snakemake and kubernetes Python |
||||||
|
modules. Now install the `byok8s` command line |
||||||
|
tool: |
||||||
|
|
||||||
|
``` |
||||||
|
python setup.py build install |
||||||
|
``` |
||||||
|
|
||||||
|
Now you can run: |
||||||
|
|
||||||
|
``` |
||||||
|
which byok8s |
||||||
|
``` |
||||||
|
|
||||||
|
and you should see `byok8s` in your virtual |
||||||
|
environment's `bin/` directory. |
||||||
|
|
||||||
|
This command line utility will expect a kubernetes |
||||||
|
cluster to be set up before it is run. |
||||||
|
|
||||||
|
Setting up a kubernetes cluster will create... |
||||||
|
(fill in more info here)... |
||||||
|
|
||||||
|
Snakemake will automatically create the pods |
||||||
|
in the cluster, so you just need to allocate |
||||||
|
a kubernetes cluster. |
||||||
|
|
||||||
|
|
||||||
|
## Step 3: Run byok8s |
||||||
|
|
||||||
|
Now you can run the workflow with the `byok8s` command. |
||||||
|
This submits the Snakemake workflow jobs to the Kubernetes |
||||||
|
cluster that minikube created. |
||||||
|
|
||||||
|
You should have your workflow in a `Snakefile` in the |
||||||
|
current directory. Use the `--snakefile` flag if it is |
||||||
|
named something other than `Snakefile`. |
||||||
|
|
||||||
|
You will also need to specify your AWS credentials |
||||||
|
via the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` |
||||||
|
environment variables. These are used to to access |
||||||
|
S3 buckets for file I/O. |
||||||
|
|
||||||
|
Finally, you will need to create an S3 bucket for |
||||||
|
Snakemake to use for file I/O. Pass the name of the |
||||||
|
bucket using the `--s3-bucket` flag. |
||||||
|
|
||||||
|
Start by exporting these two vars (careful to |
||||||
|
scrub them from bash history): |
||||||
|
|
||||||
|
``` |
||||||
|
export AWS_ACCESS_KEY_ID=XXXXX |
||||||
|
export AWS_SECRET_ACCESS_KEY=XXXXX |
||||||
|
``` |
||||||
|
|
||||||
|
Run the alpha workflow with blue params: |
||||||
|
|
||||||
|
``` |
||||||
|
byok8s --s3-bucket=mah-bukkit workflow-alpha params-blue |
||||||
|
``` |
||||||
|
|
||||||
|
Run the alpha workflow with red params: |
||||||
|
|
||||||
|
``` |
||||||
|
byok8s --s3-bucket=mah-bukkit workflow-alpha params-red |
||||||
|
``` |
||||||
|
|
||||||
|
Run the gamma workflow with red params, &c: |
||||||
|
|
||||||
|
``` |
||||||
|
byok8s --s3-bucket=mah-bukkit workflow-gamma params-red |
||||||
|
``` |
||||||
|
|
||||||
|
(NOTE: May want to let the user specify |
||||||
|
input and output directories with flags.) |
||||||
|
|
||||||
|
All input files are searched for relative to the working |
||||||
|
directory. |
||||||
|
|
||||||
|
|
||||||
|
## Step 4: Tear Down Kubernetes Cluster |
||||||
|
|
||||||
|
The last step once the workflow has been finished, |
||||||
|
is to tear down the kubernetes cluster. The virtual |
||||||
|
kubernetes cluster created by minikube can be torn |
||||||
|
down with the following command: |
||||||
|
|
||||||
|
``` |
||||||
|
minikube stop |
||||||
|
``` |
||||||
|
|
||||||
|
|
||||||
@ -0,0 +1,5 @@ |
|||||||
|
# Travis Tests with Minikube |
||||||
|
|
||||||
|
This page is in progress; see this post |
||||||
|
on the <https://charlesreid1.github.io> blog for info: |
||||||
|
[Building Snakemake Command Line Wrappers for Kubernetes Workflows](https://charlesreid1.github.io/building-snakemake-command-line-wrappers-for-kubernetes-workflows.html). |
||||||
@ -1,6 +0,0 @@ |
|||||||
# Kubernetes on AWS |
|
||||||
|
|
||||||
## Elastic Container Service |
|
||||||
|
|
||||||
## Quickstart |
|
||||||
|
|
||||||
@ -1,7 +0,0 @@ |
|||||||
# Kubernetes on Google Cloud Platform |
|
||||||
|
|
||||||
## Google Container Engine |
|
||||||
|
|
||||||
## Quickstart |
|
||||||
|
|
||||||
|
|
||||||
@ -1,6 +0,0 @@ |
|||||||
# Minikube on AWS EC2 Nodes |
|
||||||
|
|
||||||
## Quickstart |
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@ -0,0 +1,42 @@ |
|||||||
|
site_name: 2019-snakemake-byok8s |
||||||
|
site_url: https://charlesreid1.github.io/2019-snakemake-byok8s |
||||||
|
repo_name: 2019-snakemake-byok8s |
||||||
|
repo_url: https://github.com/charlesreid1/2019-snakemake-byok8s |
||||||
|
edit_uri: "" |
||||||
|
|
||||||
|
copyright: 'Copyright © 2018 <a href="https://charlesreid1.com">Charles Reid</a>, released under the <a href="https://opensource.org/licenses/MIT">MIT license</a> <br /><br /> |
||||||
|
<div>Icon made by Freepik, obtained from <a href="https://www.flaticon.com/" title="Flaticon">www.flaticon.com</a>, used under a <a href="http://creativecommons.org/licenses/by/3.0/" title="Creative Commons BY 3.0" target="_blank">CC 3.0 BY</a></div> license.' |
||||||
|
|
||||||
|
docs_dir: docs |
||||||
|
site_dir: site |
||||||
|
extra_css: |
||||||
|
- css/custom.css |
||||||
|
theme: |
||||||
|
name: null |
||||||
|
custom_dir: 'mkdocs-material-dib/material' |
||||||
|
palette: |
||||||
|
primary: 'blue' |
||||||
|
accent: 'blue' |
||||||
|
logo: 'images/ship.svg' |
||||||
|
font: |
||||||
|
text: 'Roboto' |
||||||
|
code: 'Roboto Mono' |
||||||
|
nav: |
||||||
|
- 'Index': 'index.md' |
||||||
|
- 'Installing': 'installing.md' |
||||||
|
- 'Quickstart': 'quickstart.md' |
||||||
|
- 'K8s with Minikube' : 'kubernetes_minikube.md' |
||||||
|
- 'K8s with GCP' : 'kubernetes_gcp.md' |
||||||
|
- 'K8s with AWS' : 'kubernetes_aws.md' |
||||||
|
- 'K8s with DigitalOcean' : 'kubernetes_dok.md' |
||||||
|
- 'Travis Tests': 'travis_tests.md' |
||||||
|
|
||||||
|
# Extensions |
||||||
|
markdown_extensions: |
||||||
|
- admonition |
||||||
|
- codehilite: |
||||||
|
guess_lang: false |
||||||
|
- toc: |
||||||
|
permalink: true |
||||||
|
|
||||||
|
strict: true |
||||||
Loading…
Reference in new issue