Compare commits
7 Commits
Author | SHA1 | Date | |
---|---|---|---|
ebbba235a1 | |||
949119a0dd | |||
e7cde484d5 | |||
924b8a83f5 | |||
06dfc9c901 | |||
0b860534e2 | |||
31c31544ba |
23
README.md
23
README.md
@@ -25,8 +25,8 @@ k8s = kubernetes
|
|||||||
byok8s is a command line utility that launches
|
byok8s is a command line utility that launches
|
||||||
a Snakemake workflow on an existing Kubernetes
|
a Snakemake workflow on an existing Kubernetes
|
||||||
cluster. This allows you to do something
|
cluster. This allows you to do something
|
||||||
like this (also see the [Installation](installing.md)
|
like this (also see the [Installation](docs/installing.md)
|
||||||
and [Quickstart](quickstart.md) guides in the
|
and [Quickstart](docs/quickstart.md) guides in the
|
||||||
documentation):
|
documentation):
|
||||||
|
|
||||||
```
|
```
|
||||||
@@ -45,12 +45,8 @@ minikube stop
|
|||||||
```
|
```
|
||||||
|
|
||||||
## Getting Up and Running
|
## Getting Up and Running
|
||||||
Snakemake workflows are provided via a Snakefile by
|
|
||||||
the user. Snakemake runs tasks on the Kubernetes (k8s)
|
|
||||||
cluster. The approach is for the user to provide
|
|
||||||
their own Kubernetes cluster (byok8s = Bring Your
|
|
||||||
|
|
||||||
See the [Quickstart Guide](quickstart.md) to get up and
|
See the [Quickstart Guide](docs/quickstart.md) to get up and
|
||||||
running with byok8s.
|
running with byok8s.
|
||||||
|
|
||||||
## How does byok8s work?
|
## How does byok8s work?
|
||||||
@@ -102,11 +98,11 @@ or Google Cloud.
|
|||||||
But the compute nodes can also be virtual, which is where
|
But the compute nodes can also be virtual, which is where
|
||||||
[minikube](https://github.com/kubernetes/minikube) comes in. It creates a
|
[minikube](https://github.com/kubernetes/minikube) comes in. It creates a
|
||||||
kubernetes cluster that is entirely local and virtual, which makes testing
|
kubernetes cluster that is entirely local and virtual, which makes testing
|
||||||
easy. See the [byok8s Minikube Guide](kubernetes_minikube.md) for details
|
easy. See the [byok8s Minikube Guide](docs/kubernetes_minikube.md) for details
|
||||||
about how to use minikube with byok8s.
|
about how to use minikube with byok8s.
|
||||||
|
|
||||||
The Travis CI tests also utilize minikube to run test workflows. See [byok8s
|
The Travis CI tests also utilize minikube to run test workflows. See [byok8s
|
||||||
Travis Tests](travis_tests.md) for more information.
|
Travis Tests](docs/travis_tests.md) for more information.
|
||||||
|
|
||||||
## Cloud Providers
|
## Cloud Providers
|
||||||
|
|
||||||
@@ -122,9 +118,8 @@ We have guides for the following:
|
|||||||
|
|
||||||
| Cloud Provider | Kubernetes Service | Guide | State |
|
| Cloud Provider | Kubernetes Service | Guide | State |
|
||||||
|-----------------------------|---------------------------------|-------------------------------------------------|------------|
|
|-----------------------------|---------------------------------|-------------------------------------------------|------------|
|
||||||
| Minikube (on AWS EC2) | Minikube | [byok8s Minikube Guide](kubernetes_minikube.md) | Finished |
|
| Minikube (on AWS EC2) | Minikube | [byok8s Minikube Guide](docs/kubernetes_minikube.md) | Finished |
|
||||||
| Google Cloud Platform (GCP) | Google Container Engine (GKE) | [byok8s GCP GKE Guide](kubernetes_gcp.md) | Finished |
|
| Google Cloud Platform (GCP) | Google Container Engine (GKE) | [byok8s GCP GKE Guide](docs/kubernetes_gcp.md) | Finished |
|
||||||
| Amazon Web Services (AWS) | Elastic Container Service (EKS) | [byok8s AWS EKS Guide](kubernetes_aws.md) | Unfinished |
|
| Amazon Web Services (AWS) | Elastic Container Service (EKS) | [byok8s AWS EKS Guide](docs/kubernetes_aws.md) | Unfinished |
|
||||||
| Digital Ocean (DO) | DO Kubernetes (DOK) | [byok8s DO DOK Guide](kubernetes_dok.md) | Unfinished |
|
| Digital Ocean (DO) | DO Kubernetes (DOK) | [byok8s DO DOK Guide](docs/kubernetes_dok.md) | Unfinished |
|
||||||
|
|
||||||
Own Kubernetes).
|
|
||||||
|
@@ -1,26 +1,21 @@
|
|||||||
# Installing byok8s
|
# Installing byok8s
|
||||||
|
|
||||||
byok8s requires two pieces of prerequisite software:
|
byok8s requires pieces of prerequisite software:
|
||||||
|
|
||||||
- python (conda)
|
- python (conda)
|
||||||
- virtualenv (optional)
|
- virtualenv (optional)
|
||||||
|
|
||||||
It also requires an AWS S3 bucket to be specified
|
|
||||||
(the bucket must exist and credentials to access it
|
|
||||||
must be provided via environment variables, see the
|
|
||||||
[Quickstart](quickstart.md)).
|
|
||||||
|
|
||||||
Additionally, if you are planning to run byok8s on
|
Additionally, if you are planning to run byok8s on
|
||||||
a local virtual kubernetes cluster, you must install:
|
a local virtual kubernetes cluster, you must install:
|
||||||
|
|
||||||
- minikube
|
- minikube
|
||||||
|
|
||||||
Otherwise, if you are planning on running byok8s on
|
Otherwise, if you are planning on running byok8s on remote
|
||||||
remote kubernetes clusters provided by cloud providers,
|
kubernetes clusters provided by cloud providers
|
||||||
you must install:
|
or etc., you must install:
|
||||||
|
|
||||||
- kubernetes, ***OR***
|
- kubernetes, ***OR***
|
||||||
- a cloud provider command line tool (`gcloud`, `aws`)
|
- a cloud provider command line tool (`gcloud`, `aws`)
|
||||||
|
|
||||||
## Installing Python
|
## Installing Python
|
||||||
|
|
||||||
|
@@ -40,9 +40,9 @@ utility will let you know if it needs APIs enabled for actions. If it can't
|
|||||||
enable the API for you, it will give you a direct link to the relevant Google
|
enable the API for you, it will give you a direct link to the relevant Google
|
||||||
Cloud Console page.
|
Cloud Console page.
|
||||||
|
|
||||||
## Google Kubernetes Engine
|
## Google Kubernetes Engine (GKE)
|
||||||
|
|
||||||
GKE uses Google Cloud compute nodes to run a kubernetes cluster
|
GKE uses GCP compute nodes to run a kubernetes cluster
|
||||||
on Google Cloud infrastructure. It automatically sets up the
|
on Google Cloud infrastructure. It automatically sets up the
|
||||||
cluster for you, and allows you to use `kubectl` and `gcloud` to
|
cluster for you, and allows you to use `kubectl` and `gcloud` to
|
||||||
manage and interact with the remote cluster.
|
manage and interact with the remote cluster.
|
||||||
@@ -255,6 +255,9 @@ Events:
|
|||||||
Normal Started 6s kubelet, gke-mycluster-default-pool-b44fa389-vh3x Started container
|
Normal Started 6s kubelet, gke-mycluster-default-pool-b44fa389-vh3x Started container
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Congratulations! You've successfully run an executable Snakemake workflow
|
||||||
|
on a Google Cloud kubernetes cluster!
|
||||||
|
|
||||||
Delete the GKE cluster when you are done:
|
Delete the GKE cluster when you are done:
|
||||||
|
|
||||||
```
|
```
|
||||||
|
@@ -81,13 +81,14 @@ To fix the problem with the DNS settings, we have to patch
|
|||||||
the CoreDNS image being used by `kube-system`.
|
the CoreDNS image being used by `kube-system`.
|
||||||
To do that, use the file
|
To do that, use the file
|
||||||
[`test/fixcoredns.yml`](https://github.com/charlesreid1/2019-snakemake-byok8s/blob/master/test/fixcoredns.yml)
|
[`test/fixcoredns.yml`](https://github.com/charlesreid1/2019-snakemake-byok8s/blob/master/test/fixcoredns.yml)
|
||||||
|
[`test/fixcoredns.yml`](https://github.com/charlesreid1/2019-snakemake-byok8s/blob/master/test/fixcoredns.yml)
|
||||||
in this repository with `kubectl apply`:
|
in this repository with `kubectl apply`:
|
||||||
|
|
||||||
```plain
|
```plain
|
||||||
# Fix the DNS container
|
# Fix the DNS container
|
||||||
kubectl apply -f fixcoredns.yml
|
kubectl apply -f fixcoredns.yml
|
||||||
|
|
||||||
# Delete all kube-system containers
|
# Delete all kube-system containers
|
||||||
kubectl delete --all pods --namespace kube-system
|
kubectl delete --all pods --namespace kube-system
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -110,6 +111,6 @@ export AWS_SECRET_ACCESS_KEY="XXXXX"
|
|||||||
byok8s workflow-alpha params-blue --s3-bucket=mah-bukkit
|
byok8s workflow-alpha params-blue --s3-bucket=mah-bukkit
|
||||||
```
|
```
|
||||||
|
|
||||||
Congratulations! You've just run an executable Snakemake workflow
|
Congratulations! You've just run an executable Snakemake workflow
|
||||||
on a minikube kubernetes cluster.
|
on a minikube kubernetes cluster.
|
||||||
|
|
||||||
|
Reference in New Issue
Block a user