update gitignore, add mkdocs toc, and move readme to index

.gitignore

@@ -0,0 +1,2 @@
+#mkdocs
+site/

.gitmodules

@@ -4,3 +4,6 @@
 [submodule "d-nginx-subdomains"]
 	path = d-nginx-subdomains
 	url = https://git.charlesreid1.com/docker/d-nginx-subdomains.git
+[submodule "mkdocs-material"]
+	path = mkdocs-material
+	url = https://git.charlesreid1.com/charlesreid1/mkdocs-material

README.md

@@ -9,38 +9,14 @@ This docker pod runs two services:
 
  These two services are in this repo as submodules.
 
-## How It Works
+## Links
 
-See [Running.md](/Running.md) for info about running this docker pod.
+See documentation page here: <https://pages.charlesreid1.com/pod-webhooks>
 
-* Running the Docker Pod from Comand Line
+Or view it locally in this repository here: [docs/index.md](/docs/index.md)
-* Workflow for Docker Pod Updates
 
-See [Services.md](/Services.md) for info about running startup services.
+Source code on git.charlesreid1.com: <https://git.charlesreid1.com/docker/pod-webhooks>
 
-* Running the Docker Pod as a Startup Service
+Source code on github.com: <https://github.com/charlesreid1-docker/pod-webhooks>
-* Running Docker-Host Canary Script
-
-## Volumes
-
-This docker container needs to share a few folders:
-
-This uses one shared volume between the two containers.
-
-Captain Hook updates the static content based on web hooks that it receives,
-while the nginx container serves up the static content.
-
-## Servers 
-
-This pod runs on blackbeard.
-
-The nginx service is reverse-proxied HTTP with krash,
-and accessible at ports 7777-7779 and up.
-
-The Captain Hook webhook server is also reverse-proxied HTTP.
-The krash nginx server will handle all traffic to 
-`https://hooks.charlesreid1.com` except URLs prefixed
-with `webhook`, which are forwarded on to Captain Hook
-on port 5000.
 
 

docker-compose.yml

@@ -7,7 +7,9 @@ services:
     command: /bin/bash -c "nginx -g 'daemon off;'"
     volumes:
       - "./d-nginx-subdomains/conf.d:/etc/nginx/conf.d"
-      - "stormy_webhooks_data:/www"
+      - "/www/pages.charlesreid1.com/htdocs:/www/pages.charlesreid1.com/htdocs:ro"
+      - "/www/hooks.charlesreid1.com/htdocs:/www/hooks.charlesreid1.com/htdocs:ro"
+      - "/www/bots.charlesreid1.com/htdocs:/www/bots.charlesreid1.com/htdocs:ro"
     ports:
       - "7777:7777"
       - "7778:7778"
@@ -18,7 +20,9 @@ services:
     ports:
       - "5000:5000"
     volumes:
-      - "stormy_webhooks_data:/www"
+      - "/www/pages.charlesreid1.com/htdocs:/www/pages.charlesreid1.com/htdocs"
+      - "/www/hooks.charlesreid1.com/htdocs:/www/hooks.charlesreid1.com/htdocs"
+      - "/www/bots.charlesreid1.com/htdocs:/www/bots.charlesreid1.com/htdocs"
+      - "/tmp/triggers:/tmp/triggers"
+      - "./b-captain-hook/hooks:/app/hooks"
 
-volumes:
-  stormy_webhooks_data:

docs/Adding.md

@@ -0,0 +1,34 @@
+## Adding Hooks
+
+To add a hook to Captain Hook:
+
+1. Create an executable script in [bots/b-captain-hook](https://git.charlesreid1.com/bots/b-captain-hook)
+   on git.charlesreid1.com with the name of the action, the name of the repo (not the owner),
+   and the name of the branch in the filename. For example, `push-my-dotfiles-master` would be 
+   matched every time I `push` changes to the `master` branch  of any repository named `my-dotfiles`.
+
+2. Add, commit, and push your hook to the master branch of Captain Hook
+
+3. Wait about 15 seconds for the canary script to run (it has to update
+   the Captain Hook git repo running on the remote server to the latest version
+   and restart the webhooks-subdomains docker pod.)
+
+4. Open the `my-dotfiles` repository on git.charlesreid1.com, go to the
+   Settings > Webhooks page, and add a Gitea webhook.
+
+5. Enter info:
+    a. Payload URL is the Captain Hook server, which is `https://hooks.charlesreid1.com/webhook`.
+    b. Content type is application/json
+    c. Secret is my little secret
+    d. Pick what you'd like, I usually go with "just the push event"
+
+6. Save the webhook, then click on the webhook again to open it back up.
+   Scroll down to the bottom right and click "Test Delivery". 
+
+You should see a green success sign. If you see a red warning sign:
+
+* ensure webhooks-subdomain pods are running
+* ensure port 5000 open in captain hook container and on blackbeard and on aws
+* ensure hook has been added to b-captain-hook repository's hooks folder 
+
+

docs/Canary.md

@@ -0,0 +1,138 @@
+# Captain Hook's Canary
+
+Captain Hook's Canary is a mechanism by which a docker pod
+running a webhook server can send a signal to the host to 
+restart the docker pod.
+
+This is done by bind-mounting a host directory at `/tmp/triggers/`
+inside the docker container, and when a webhook is received 
+from git.charlesreid1.com that indicates there was a change
+to Captain Hook, the docker pod creates a trigger file.
+
+The canary bash script, meanwhile, is a bash script that runs 
+forever and checks for a trigger file from the docker pod
+every 10 seconds.
+
+The pull host Captain Hook script is a script that updates the
+Captain Hook git repo on the host machine.
+
+On top of all of that, we also need a startup service that will
+actually run the captain hook canary script, and keep it running,
+and stop it when we ask it to stop.
+
+Sections below cover:
+* The canary bash script
+* The docker host pull script
+* The canary statup service
+
+## The Canary Bash Script
+
+Note: this needs an associated systemd service.
+See the services directory of the dotfiles repo.
+
+This is a canary script for connecting
+the Captain Hook container to the host 
+machine, and triggering tasks on the 
+host machine with webhooks.
+
+The Captain Hook container mounts the 
+following host directory inside the 
+container (same location for host/container):
+
+```
+/tmp/triggers/
+```
+
+When a webhook in Captain Hook wants to 
+trigger an event on the host (blackbeard),
+it puts a file in `/tmp/triggers/`.
+
+Meanwhile, on the host, this script checks
+every 10 seconds for trigger files.
+
+Each webhook can create its own trigger file,
+and this script processes each trigger differently.
+
+```bash
+#!/bin/bash
+
+while true
+do
+    # bootstrap-pull captain hook
+    if [ -f "/tmp/triggers/push-b-captain-hook-master" ]; then
+        echo "CAPTAIN HOOK'S CANARY:"
+        echo "Running trigger to update Captain Hook on the host machine (user charles)"
+        sudo -H -u charles python /home/charles/blackbeard_scripts/captain_hook_pull_host.py
+        echo "All done."
+        rm -f "/tmp/triggers/push-b-captain-hook-master"
+    fi
+
+    sleep 10;
+done
+```
+
+
+## The Pull Host Captain Hook Script
+
+Next we have a python script that actually updates the host's 
+version of Captain Hook:
+
+```python
+#!/usr/bin/env python3
+import subprocess
+import os
+import time
+
+"""
+Captain Hook: Pull Captain Hook on the Host 
+
+This script is called by the host machine 
+(blackbeard) running the Captain Hook container.
+
+This is triggered by push actions to the 
+master branch of b-captain-hook.
+
+The action is to update (git pull) the copy 
+of Captain Hook running on the host, and
+restart the container pod.
+"""
+
+work_dir = os.path.join('/home','charles','codes','bots','b-captain-hook')
+
+# Step 1:
+# Update Captain Hook
+pull_cmd = ['git','pull','origin','master']
+subprocess.call(pull_cmd, cwd=work_dir)
+
+time.sleep(5)
+
+# Step 2:
+# Restart Captain Hook pod
+pod_restart = ['docker-compose','restart']
+subprocess.call(pod_restart, cwd=work_dir)
+```
+
+## The Canary Startup Script
+
+Here is the startup file that runs the Captain Hook's Canary bash script.
+
+The stop directive uses pgrep to find the process id and stops any PIDs returned.
+
+```
+[Unit]
+Description=captain hook canary script
+Requires=dockerpod-captainhook.service
+After=dockerpod-captainhook.service
+
+[Service]
+Restart=always
+ExecStart=/home/charles/blackbeard_scripts/captain_hook_canary.sh
+ExecStop=/usr/bin/pgrep -f captain_hook_canary | /usr/bin/xargs /bin/kill 
+
+[Install]
+WantedBy=default.target
+```
+
+See [Services](Services.md) for more info on what to do with this file.
+
+

docs/Services.md

@@ -36,23 +36,51 @@ ExecStop=/usr/local/bin/docker-compose  -f /home/charles/codes/docker/pod-webhoo
 WantedBy=default.target
 ```
 
+**`captain-hook-canary.service:`**
+
+```
+[Unit]
+Description=captain hook canary script
+Requires=dockerpod-captainhook.service
+After=dockerpod-captainhook.service
+
+[Service]
+Restart=always
+ExecStart=/home/charles/blackbeard_scripts/captain_hook_canary.sh
+ExecStop=/usr/bin/pgrep -f captain_hook_canary | /usr/bin/xargs /bin/kill 
+
+[Install]
+WantedBy=default.target
+```
+
 Now install the service to `/etc/systemd/system/dockerpod-webhooks.servce`,
+and/or `/etc/systemd/system/captain-hook-canary.servce`,
 and activate it:
 
 ```
 sudo systemctl enable dockerpod-webhooks.service
+
+sudo systemctl enable captain-hook-canary.service
 ```
 
 Now you can start/stop the service with:
 
 ```
 sudo systemctl (start|stop) dockerpod-webhooks.service
+
+sudo systemctl (start|stop) captain-hook-canary.service
 ```
 
-As mentioned above, stop the serivce with 
+As mentioned above, these services should be stopped before
-`sudo systemctl stop dockerpod-webhooks.service` before
 doing a `docker-compose stop` or a `docker-compose up --build`
 to keep the pod from respawning in the middle of the task.
 
+Stop using:
+
+```
+sudo systemctl stop dockerpod-webhooks.service
+
+sudo systemctl stop captain-hook-canary.service
+```
 
 

docs/index.md

@@ -0,0 +1,132 @@
+# pod-webhooks
+
+This docker pod runs two services:
+
+ * Captain Hook webhook server (python + flask)
+    * [b-captain-hook](https://git.charlesreid1.com/bots/b-captain-hook)
+ * Static content server for subdomain pages (nginx)
+    * [d-nginx-subdomains](https://git.charlesreid1.com/docker/d-nginx-subdomains)
+
+ These two services are in this repo as submodules.
+
+## Adding Hooks
+
+Since this is probably the only thing you'll care about once everything
+is actually running... until it breaks.
+
+[How To Add A Hook](Adding.md)
+
+
+## How It Works
+
+See [Running.md](Running.md) for info about running this docker pod.
+
+* Running the Docker Pod from Comand Line
+* Workflow for Docker Pod Updates
+
+See [Services.md](Services.md) for info about running startup services.
+
+* Running the Docker Pod as a Startup Service
+* Running Captain Hook's Canary (Script)
+
+Enable/disable service (installs/uninstalls, but does not start):
+
+```
+sudo systemctl (enable|disable) dockerpod-webhooks.service
+sudo systemctl (enable|disable) captain-hook-canary.service
+```
+
+Start/stop:
+
+```
+sudo systemctl (start|stop) dockerpod-webhooks.service
+sudo systemctl (start|stop) captain-hook-canary.service
+```
+
+See [Captain Hook's Canary (Canary.md)](Canary.md) for details on the canary script that allows the
+webhooks docker pod to trigger itself to be re-loaded when there are new hooks
+added to captain hook.
+
+
+## Volumes and Files
+
+### Subdomains
+
+The static files hosted on charlesreid1.com subdomains are contained in
+subdirectories of `/www/*.charlesreid1.com/` and this is
+mounted by the subdomains docker container that has rules
+set up for which subdomains to serve.
+
+### Captain Hook
+
+Captain Hook mounts the `/www` folder, which is served by the subdomains
+nginx server, as well as the hooks folder in the Captain Hook repository
+(that's at `b-captain-hook/hooks`).
+
+When there is a change pushed to a particular branch on git.charlesreid1.com,
+the git.charlesreid1.com server will check if there is a corresponding hook that's
+been added to Captain Hook for that repo and branch. If so, git.charlesreid1.com
+runs that script. For pages.charlesreid1.com, that's usually just a git pull 
+on the contents of `/www/pages.charlesreid1.com/my-page`.
+
+### Captain Hook's Canary
+
+Captain Hook presents a bit of a paradox: the webhook docker pod needs to be 
+able to tell the host to restart the webhook docker pod when changes are pushed
+to Captain Hook itself.
+
+This is done by Captain Hook's Canary. This is a script that checks every 10 seconds
+for a trigger file in a directory mounted between the host and container. If the 
+trigger file is present, the host will update its copy of Captain Hook,
+then restart the webhooks-subdomains docker pod.
+
+As per the [`blackbeard_scripts`](https://git.charlesreid1.com/dotfiles/debian/src/branch/master/dotfiles/blackbeard_scripts)
+directory of [dotfiles/debian](https://git.charlesreid1.com/dotfiles/debian),
+the `captain_hook_canary.sh` canary will restart the webhooks docker pod if it 
+detects the file:
+
+```
+/tmp/triggers/push-b-captain-hook-master
+```
+
+(The canary script will clean up this file.)
+
+Now a hook can be added to Captain Hook that will be run when there is a push event
+on the master branch of [bots/b-captain-hook](https://git.charlesreid1.com/bots/b-captain-hook).
+By creating a hook named `push-b-captain-hook-master` in the
+`hooks/` directory of captain hook that runs a simple `touch` command,
+this webhook can trigger the script which triggers a restart of the
+docker pod.
+
+Code: <https://git.charlesreid1.com/bots/b-captain-hook/src/branch/master/hooks/push-b-captain-hook-master>
+
+
+## Network
+
+The `d-subodomains-nginx` container opens different ports for different
+subdomains, and reverse-proxies requests from charlesreid1.com.
+The port numbering starts at 7777 for pages.charlesreid1.com
+and goes up from there, one port per subdomain.
+
+Also see [pod-charlesreid1](https://git.charlesreid1.com/docker/pod-charlesreid1)
+on git.charlesreid1.com for the nginx reverse proxy configuration.
+
+Captain Hook runs a Flask server on port 5000 and listens for triggers
+from git.charlesreid1.com (gitea) web hooks. These web hooks must have 
+the correct secret or the trigger will be ignored.
+
+
+## Servers 
+
+This pod runs on blackbeard.
+
+The nginx service is reverse-proxied HTTP with krash,
+and accessible at ports 7777+ and up.
+
+The Captain Hook webhook server is also reverse-proxied HTTP.
+The krash nginx server will handle all traffic to 
+<https://hooks.charlesreid1.com> except URLs prefixed
+with `webhook`, which are forwarded on to Captain Hook
+on port 5000.
+
+

mkdocs.yml

@@ -0,0 +1,37 @@
+site_name: pod-webhooks
+site_url: https://pages.charlesreid1.com/pod-webhooks
+repo_name: pod-webhooks
+repo_url: https://git.charlesreid1.com/docker/pod-webhooks
+edit_uri: ""
+copyright: 'Copyright &copy; 2018 <a href="https://charlesreid1.com">Charles Reid</a>, released under the <a href="https://opensource.org/licenses/MIT">MIT license</a>'
+
+docs_dir: docs
+site_dir: site
+extra_css:
+  - css/custom.css
+theme: 
+  name: null
+  custom_dir: 'mkdocs-material/material'
+  palette:
+    primary: 'pink'
+    accent: 'pink'
+  logo:
+    icon: 'dns'
+  font:
+    text: 'Roboto'
+    code: 'Roboto Mono'
+pages:
+  - Home: 'index.md'
+  - Adding Webhooks: 'Adding.md'
+  - Running Captain Hook: 'Running.md'
+  - Captain Hook Startup Services: 'Services.md'
+  - Captain Hook's Canary: 'Canary.md'
+
+# Extensions
+markdown_extensions:
+  - admonition
+  - codehilite:
+      guess_lang: false
+  - toc:
+      permalink: true
+