Compare commits
81 Commits
28 changed files with 1276 additions and 177 deletions
@ -1,2 +1,12 @@
@@ -1,2 +1,12 @@
|
||||
#mkdocs |
||||
site/ |
||||
docker-compose.yml |
||||
|
||||
# ignore scripts generated from templates |
||||
scripts/pages_init_setup.py |
||||
scripts/subdomains_init_setup.py |
||||
scripts/captain-hook-canary.service |
||||
scripts/captain_hook_canary.sh |
||||
scripts/captain_hook_pull_host.py |
||||
scripts/pod-webhooks.service |
||||
scripts/output/ |
||||
|
||||
@ -1 +1 @@
@@ -1 +1 @@
|
||||
Subproject commit f084dce02c9a943a585e192d1e601196dfec5b14 |
||||
Subproject commit 361f59c21a733a484f48e9bd60bce2d94dbf7b1b |
||||
@ -1 +1 @@
@@ -1 +1 @@
|
||||
Subproject commit 3d82d7728b7e58e101138d8645a98dbb6cff266a |
||||
Subproject commit a87dcd5f3d997596c87deec03d396f0911b4b03f |
||||
@ -0,0 +1,42 @@
@@ -0,0 +1,42 @@
|
||||
version: "3.3" |
||||
services: |
||||
|
||||
stormy_nginx_subs: |
||||
restart: always |
||||
image: nginx |
||||
command: /bin/bash -c "nginx -g 'daemon off;'" |
||||
volumes: |
||||
- "./d-nginx-subdomains/conf.d:/etc/nginx/conf.d" |
||||
- "/www/pages.{{ server_name_default }}/htdocs:/www/pages.{{ server_name_default }}/htdocs:ro" |
||||
- "/www/hooks.{{ server_name_default }}/htdocs:/www/hooks.{{ server_name_default }}/htdocs:ro" |
||||
- "/www/bots.{{ server_name_default }}/htdocs:/www/bots.{{ server_name_default }}/htdocs:ro" |
||||
logging: |
||||
driver: "json-file" |
||||
options: |
||||
max-size: "1m" |
||||
max-file: "10" |
||||
ports: |
||||
- "7777:7777" |
||||
- "7778:7778" |
||||
- "7779:7779" |
||||
|
||||
stormy_captain_hook: |
||||
restart: always |
||||
build: b-captain-hook |
||||
volumes: |
||||
- "/www/pages.{{ server_name_default }}:/www/pages.{{ server_name_default }}" |
||||
- "/www/hooks.{{ server_name_default }}:/www/hooks.{{ server_name_default }}" |
||||
- "/www/bots.{{ server_name_default }}:/www/bots.{{ server_name_default }}" |
||||
- "/tmp/triggers:/tmp/triggers" |
||||
- "./b-captain-hook/config.json:/app/config.json" |
||||
- "./b-captain-hook/hooks:/app/hooks" |
||||
logging: |
||||
driver: "json-file" |
||||
options: |
||||
max-size: "1m" |
||||
max-file: "10" |
||||
ports: |
||||
- "5000:5000" |
||||
depends_on: |
||||
- stormy_nginx_subs |
||||
|
||||
@ -1,86 +0,0 @@
@@ -1,86 +0,0 @@
|
||||
## Running Hooks-Subdomains Docker Pod as Startup Service |
||||
|
||||
The webhooks-subdomains docker pod requires two startup |
||||
services - one keeps the docker pod running, the other |
||||
watches a folder shared between the host and container |
||||
for signals from the container, and uses that to trigger |
||||
updates to the subdomains web content. |
||||
|
||||
Also see the `services/` folder of the |
||||
[dotfiles/debian repository](https://git.charlesreid1.com/dotfiles/debian), |
||||
repository for the systemd services. |
||||
|
||||
|
||||
### Service 1: Webhooks Docker Pod |
||||
|
||||
This service keeps the webhooks docker pod service running |
||||
continuously. If the pod stops, this service will restart it. |
||||
|
||||
(This service should not be running if you are troubleshooting |
||||
the docker pod.) |
||||
|
||||
**`dockerpod-webhooks.service:`** |
||||
|
||||
``` |
||||
[Unit] |
||||
Description=webhooks and subdomains docker pod |
||||
Requires=docker.service |
||||
After=docker.service |
||||
|
||||
[Service] |
||||
Restart=always |
||||
ExecStart=/usr/local/bin/docker-compose -f /home/charles/codes/docker/pod-webhooks/docker-compose.yml up |
||||
ExecStop=/usr/local/bin/docker-compose -f /home/charles/codes/docker/pod-webhooks/docker-compose.yml down |
||||
|
||||
[Install] |
||||
WantedBy=default.target |
||||
``` |
||||
|
||||
**`captain-hook-canary.service:`** |
||||
|
||||
``` |
||||
[Unit] |
||||
Description=captain hook canary script |
||||
Requires=dockerpod-captainhook.service |
||||
After=dockerpod-captainhook.service |
||||
|
||||
[Service] |
||||
Restart=always |
||||
ExecStart=/home/charles/blackbeard_scripts/captain_hook_canary.sh |
||||
ExecStop=/usr/bin/pgrep -f captain_hook_canary | /usr/bin/xargs /bin/kill |
||||
|
||||
[Install] |
||||
WantedBy=default.target |
||||
``` |
||||
|
||||
Now install the service to `/etc/systemd/system/dockerpod-webhooks.servce`, |
||||
and/or `/etc/systemd/system/captain-hook-canary.servce`, |
||||
and activate it: |
||||
|
||||
``` |
||||
sudo systemctl enable dockerpod-webhooks.service |
||||
|
||||
sudo systemctl enable captain-hook-canary.service |
||||
``` |
||||
|
||||
Now you can start/stop the service with: |
||||
|
||||
``` |
||||
sudo systemctl (start|stop) dockerpod-webhooks.service |
||||
|
||||
sudo systemctl (start|stop) captain-hook-canary.service |
||||
``` |
||||
|
||||
As mentioned above, these services should be stopped before |
||||
doing a `docker-compose stop` or a `docker-compose up --build` |
||||
to keep the pod from respawning in the middle of the task. |
||||
|
||||
Stop using: |
||||
|
||||
``` |
||||
sudo systemctl stop dockerpod-webhooks.service |
||||
|
||||
sudo systemctl stop captain-hook-canary.service |
||||
``` |
||||
|
||||
|
||||
@ -0,0 +1,11 @@
@@ -0,0 +1,11 @@
|
||||
.md-typeset h1 { font-weight: 600; } |
||||
.md-typeset h2 { font-weight: 600; } |
||||
.md-typeset h3 { font-weight: 600; } |
||||
.md-typeset h4 { font-weight: 600; } |
||||
|
||||
body { |
||||
background-color: #FAFAFA; |
||||
} |
||||
div.body { |
||||
background-color: #FAFAFA; |
||||
} |
||||
@ -0,0 +1,93 @@
@@ -0,0 +1,93 @@
|
||||
## Running Hooks-Subdomains Docker Pod as Startup Service |
||||
|
||||
The webhooks-subdomains docker pod requires two startup |
||||
services: |
||||
|
||||
* Docker pod service - this startup service keeps the pod running, |
||||
and will restart it if it crashes |
||||
|
||||
* Captain Hook's canary - this watches a folder that is shared between |
||||
the docker pod host and the Captain Hook container, which allows the |
||||
pod to send triggers to the host. |
||||
|
||||
Also see the `scripts/` folder of this repo, |
||||
[pod-webhooks](https://git.charlesreid1.com/docker/pod-webhooks) |
||||
([Github mirror](https://github.com/charlesreid1-docker/pod-webhooks)). |
||||
|
||||
|
||||
### Docker Pod Startup Service |
||||
|
||||
This service keeps the webhooks docker pod service running |
||||
continuously. If the pod stops, this service will restart it. |
||||
|
||||
(This service should not be running if you are troubleshooting |
||||
the docker pod, otherwise every time you try and stop the pod |
||||
it will respawn.) |
||||
|
||||
**`pod-webhooks.service:`** |
||||
|
||||
``` |
||||
[Unit] |
||||
Description=webhooks and subdomains docker pod |
||||
Requires=docker.service |
||||
After=docker.service |
||||
|
||||
[Service] |
||||
Restart=always |
||||
ExecStart=/usr/local/bin/docker-compose -f /home/charles/codes/docker/pod-webhooks/docker-compose.yml up |
||||
ExecStop=/usr/local/bin/docker-compose -f /home/charles/codes/docker/pod-webhooks/docker-compose.yml down |
||||
|
||||
[Install] |
||||
WantedBy=default.target |
||||
``` |
||||
|
||||
|
||||
### Captain Hook's Canary Startup Service |
||||
|
||||
This service just watches a folder for a particular |
||||
watchfile, and runs a script if it sees the watchfile |
||||
appear. |
||||
|
||||
**`captain-hook-canary.service:`** |
||||
|
||||
``` |
||||
[Unit] |
||||
Description=captain hook canary script |
||||
Requires=pod-webhooks.service |
||||
After=pod-webhooks.service |
||||
|
||||
[Service] |
||||
Restart=always |
||||
ExecStart=/home/charles/pod-webhooks/scripts/captain_hook_canary.sh |
||||
ExecStop=/usr/bin/pgrep -f captain_hook_canary | /usr/bin/xargs /bin/kill |
||||
|
||||
[Install] |
||||
WantedBy=default.target |
||||
``` |
||||
|
||||
### Installing Services |
||||
|
||||
Install the services by copying the `*.service` files |
||||
to `/etc/systemd/system/dockerpod-webhooks.servce` |
||||
and `/etc/systemd/system/captain-hook-canary.servce`, |
||||
and activate the startup services: |
||||
|
||||
``` |
||||
sudo systemctl enable pod-webhooks.service |
||||
|
||||
sudo systemctl enable captain-hook-canary.service |
||||
``` |
||||
|
||||
Now you can start/stop the services with: |
||||
|
||||
``` |
||||
sudo systemctl (start|stop) pod-webhooks.service |
||||
|
||||
sudo systemctl (start|stop) captain-hook-canary.service |
||||
``` |
||||
|
||||
As mentioned above, these services should be stopped before |
||||
doing a `docker-compose stop` or a `docker-compose up --build` |
||||
to keep the pod from respawning in the middle of the task. |
||||
|
||||
|
||||
@ -1 +1 @@
@@ -1 +1 @@
|
||||
Subproject commit 6569122bb1e9d6bd73fd964639c33c9a3a4f46d3 |
||||
Subproject commit b0c6890853aa9138baf5f9749862b927518ab656 |
||||
@ -0,0 +1,24 @@
@@ -0,0 +1,24 @@
|
||||
# pod-webhooks scripts |
||||
|
||||
Copy this `pod-webhooks.service` service script to |
||||
`/etc/systemd/system/pod-webhooks.service`: |
||||
|
||||
``` |
||||
sudo cp pod-webhooks.service /etc/systemd/system/pod-webhooks.service |
||||
``` |
||||
|
||||
Enable/disable the service: |
||||
|
||||
``` |
||||
sudo systemctl enable pod-webhooks |
||||
sudo systemctl disable pod-webhooks |
||||
``` |
||||
|
||||
Start/restart/stop the service: |
||||
|
||||
``` |
||||
sudo systemctl start pod-webhooks |
||||
sudo systemctl restart pod-webhooks |
||||
sudo systemctl stop pod-webhooks |
||||
``` |
||||
|
||||
@ -0,0 +1,89 @@
@@ -0,0 +1,89 @@
|
||||
import os, re, sys |
||||
import glob |
||||
from jinja2 import Environment, FileSystemLoader, select_autoescape |
||||
|
||||
""" |
||||
Apply Default Values to |
||||
Captain Hook Jinja Templates |
||||
|
||||
|
||||
This script applies default values to |
||||
templates in this folder. |
||||
|
||||
The templates are used by Ansible, |
||||
but this script uses the same template |
||||
engine as Ansible to apply template |
||||
variable values to the template files |
||||
and make real files. |
||||
|
||||
variables are: |
||||
- `webhooks_install_dir` - location of pod-webhooks repo |
||||
|
||||
- `username` - user/group name to change ownership to |
||||
- `server_name_default` - name of server |
||||
(e.g., charlesreid1.com or charlesreid1.red) |
||||
""" |
||||
|
||||
|
||||
# Where templates live |
||||
TEMPLATEDIR = '.' |
||||
|
||||
# Where rendered templates will go |
||||
OUTDIR = 'output' |
||||
|
||||
# Should existing (destination) files |
||||
# be overwritten if they exist? |
||||
OVERWRITE = True |
||||
|
||||
# Template variables |
||||
TV = { |
||||
'server_name_default': 'charlesreid1.red', |
||||
'username': 'charles', |
||||
'webhooks_install_dir' : '/home/charles/pod-webhooks' |
||||
} |
||||
|
||||
|
||||
|
||||
def apply_templates(template_dir, output_dir, template_vars, overwrite=False): |
||||
"""Apply the template variables |
||||
to the template files. |
||||
""" |
||||
|
||||
if not os.path.exists(output_dir): |
||||
msg = "Error: output dir %s does not exist!"%(output_dir) |
||||
raise Exception(msg) |
||||
|
||||
if not os.path.exists(template_dir): |
||||
msg = "Error: template dir %s does not exist!"%(output_dir) |
||||
raise Exception(msg) |
||||
|
||||
# Jinja env |
||||
env = Environment(loader=FileSystemLoader('.')) |
||||
|
||||
# Render templates |
||||
template_files = glob.glob('captain*.j2') |
||||
render_files = [re.sub('\.j2','',s) for s in template_files] |
||||
|
||||
for rfile,tfile in zip(render_files,template_files): |
||||
|
||||
# Get rendered template content |
||||
content = env.get_template(tfile).render(**template_vars) |
||||
|
||||
# Write to file |
||||
dest = os.path.join(output_dir,rfile) |
||||
if os.path.exists(dest) and overwrite is False: |
||||
msg = "Error: template rendering destination %s already exists!"%(dest) |
||||
raise Exception(msg) |
||||
|
||||
with open(dest,'w') as f: |
||||
f.write(content) |
||||
|
||||
print("Rendered the following templates:%s\nOutput files:%s\n"%( |
||||
"".join(["\n- "+os.path.join(template_dir,j) for j in template_files]), |
||||
"".join(["\n- "+os.path.join(output_dir,j) for j in render_files]) |
||||
)) |
||||
|
||||
|
||||
if __name__=="__main__": |
||||
apply_templates(TEMPLATEDIR,OUTDIR,TV,OVERWRITE) |
||||
|
||||
@ -0,0 +1,84 @@
@@ -0,0 +1,84 @@
|
||||
import os, re, sys |
||||
import glob |
||||
from jinja2 import Environment, FileSystemLoader, select_autoescape |
||||
|
||||
""" |
||||
Apply Default Values to |
||||
pod-webhooks Jinja Templates |
||||
|
||||
|
||||
This script applies default values to |
||||
templates in this folder. |
||||
|
||||
The templates are used by Ansible, |
||||
but this script uses the same template |
||||
engine as Ansible to apply template |
||||
variable values to the template files |
||||
and make real files. |
||||
|
||||
variables are: |
||||
- `webhooks_install_dir` - location of pod-webhooks repo |
||||
(i.e., location of pod-webhooks docker-compose file) |
||||
""" |
||||
|
||||
|
||||
# Where templates live |
||||
TEMPLATEDIR = '.' |
||||
|
||||
# Where rendered templates will go |
||||
OUTDIR = 'output' |
||||
|
||||
# Should existing (destination) files |
||||
# be overwritten if they exist? |
||||
OVERWRITE = True |
||||
|
||||
# Template variables |
||||
TV = { |
||||
'webhooks_install_dir' : '/home/charles/pod-webhooks' |
||||
} |
||||
|
||||
|
||||
|
||||
def apply_templates(template_dir, output_dir, template_vars, overwrite=False): |
||||
"""Apply the template variables |
||||
to the template files. |
||||
""" |
||||
|
||||
if not os.path.exists(output_dir): |
||||
msg = "Error: output dir %s does not exist!"%(output_dir) |
||||
raise Exception(msg) |
||||
|
||||
if not os.path.exists(template_dir): |
||||
msg = "Error: template dir %s does not exist!"%(output_dir) |
||||
raise Exception(msg) |
||||
|
||||
# Jinja env |
||||
env = Environment(loader=FileSystemLoader('.')) |
||||
|
||||
# Render templates |
||||
template_files = glob.glob('*.service.j2') |
||||
render_files = [re.sub('\.j2','',s) for s in template_files] |
||||
|
||||
for rfile,tfile in zip(render_files,template_files): |
||||
|
||||
# Get rendered template content |
||||
content = env.get_template(tfile).render(**template_vars) |
||||
|
||||
# Write to file |
||||
dest = os.path.join(output_dir,rfile) |
||||
if os.path.exists(dest) and overwrite is False: |
||||
msg = "Error: template rendering destination %s already exists!"%(dest) |
||||
raise Exception(msg) |
||||
|
||||
with open(dest,'w') as f: |
||||
f.write(content) |
||||
|
||||
print("Rendered the following templates:%s\nOutput files:%s\n"%( |
||||
"".join(["\n- "+os.path.join(template_dir,j) for j in template_files]), |
||||
"".join(["\n- "+os.path.join(output_dir,j) for j in render_files]) |
||||
)) |
||||
|
||||
|
||||
if __name__=="__main__": |
||||
apply_templates(TEMPLATEDIR,OUTDIR,TV,OVERWRITE) |
||||
|
||||
@ -0,0 +1,90 @@
@@ -0,0 +1,90 @@
|
||||
import os, re, sys |
||||
import glob |
||||
import subprocess |
||||
from jinja2 import Environment, FileSystemLoader, select_autoescape |
||||
|
||||
""" |
||||
Apply Default Values to |
||||
SUBDOMAINS Jinja Templates |
||||
|
||||
|
||||
This script applies default values to |
||||
templates in this folder. |
||||
|
||||
The templates are used by Ansible, |
||||
but this script uses the same template |
||||
engine as Ansible to apply template |
||||
variable values to the template files |
||||
and make real files. |
||||
|
||||
only variables are: |
||||
- `username` - user/group name to change ownership to |
||||
- `server_name_default` - name of server |
||||
(e.g., charlesreid1.com or charlesreid1.red) |
||||
""" |
||||
|
||||
|
||||
# Where templates live |
||||
TEMPLATEDIR = '.' |
||||
|
||||
# Where rendered templates will go |
||||
OUTDIR = 'output' |
||||
|
||||
# Should existing (destination) files |
||||
# be overwritten if they exist? |
||||
OVERWRITE = True |
||||
|
||||
# Template variables |
||||
TV = { |
||||
'server_name_default': 'charlesreid1.red', |
||||
'username': 'charles' |
||||
} |
||||
|
||||
|
||||
|
||||
def apply_templates(template_dir, output_dir, template_vars, overwrite=False): |
||||
"""Apply the template variables |
||||
to the template files. |
||||
""" |
||||
|
||||
if not os.path.exists(output_dir): |
||||
msg = "Error: output dir %s does not exist!"%(output_dir) |
||||
raise Exception(msg) |
||||
|
||||
if not os.path.exists(template_dir): |
||||
msg = "Error: template dir %s does not exist!"%(output_dir) |
||||
raise Exception(msg) |
||||
|
||||
# Jinja env |
||||
env = Environment(loader=FileSystemLoader('.')) |
||||
|
||||
# Render templates |
||||
template_files = glob.glob('*_setup.py.j2') + glob.glob('*_pull.py.j2') |
||||
render_files = [re.sub('\.j2','',s) for s in template_files] |
||||
|
||||
for rfile,tfile in zip(render_files,template_files): |
||||
|
||||
# Get rendered template content |
||||
content = env.get_template(tfile).render(**template_vars) |
||||
|
||||
# Write to file |
||||
dest = os.path.join(output_dir,rfile) |
||||
if os.path.exists(dest) and overwrite is False: |
||||
msg = "Error: template rendering destination %s already exists!"%(dest) |
||||
raise Exception(msg) |
||||
|
||||
with open(dest,'w') as f: |
||||
f.write(content) |
||||
|
||||
x = 'executioner.py' |
||||
subprocess.call(['cp',x,os.path.join(output_dir,x)]) |
||||
|
||||
print("Rendered the following templates:%s\nOutput files:%s\n"%( |
||||
"".join(["\n- "+os.path.join(template_dir,j) for j in template_files]), |
||||
"".join(["\n- "+os.path.join(output_dir,j) for j in render_files]) |
||||
)) |
||||
|
||||
|
||||
if __name__=="__main__": |
||||
apply_templates(TEMPLATEDIR,OUTDIR,TV,OVERWRITE) |
||||
|
||||
@ -0,0 +1,12 @@
@@ -0,0 +1,12 @@
|
||||
[Unit] |
||||
Description=captain hook canary script |
||||
Requires=pod-webhooks.service |
||||
After=pod-webhooks.service |
||||
|
||||
[Service] |
||||
Restart=always |
||||
ExecStart={{ webhooks_install_dir }}/scripts/captain_hook_canary.sh |
||||
ExecStop=/usr/bin/pgrep -f captain_hook_canary | /usr/bin/xargs /bin/kill |
||||
|
||||
[Install] |
||||
WantedBy=default.target |
||||
@ -0,0 +1,47 @@
@@ -0,0 +1,47 @@
|
||||
#!/bin/bash |
||||
|
||||
: ' |
||||
Captain Hook Canary Script (Template) |
||||
|
||||
|
||||
Note: this needs an associated systemd service. |
||||
See the services directory of the dotfiles repo. |
||||
|
||||
This is a canary script for connecting |
||||
the Captain Hook container to the host |
||||
machine, and triggering tasks on the |
||||
host machine with webhooks. |
||||
|
||||
The Captain Hook container mounts the |
||||
following host directory inside the |
||||
container (same location for host/container): |
||||
|
||||
/tmp/triggers/ |
||||
|
||||
When a webhook in Captain Hook wants to |
||||
trigger an event on the host (blackbeard), |
||||
it puts a file in /tmp/triggers/. |
||||
|
||||
Meanwhile, on the host, this script checks |
||||
every 10 seconds for trigger files. |
||||
|
||||
Each webhook can create its own trigger file, |
||||
and this script processes each trigger differently. |
||||
' |
||||
|
||||
while true |
||||
do |
||||
# bootstrap-pull captain hook |
||||
if [ -f "/tmp/triggers/push-b-captain-hook-master" ]; then |
||||
echo "CAPTAIN HOOK'S CANARY:" |
||||
echo "Running trigger to update Captain Hook on the host machine (user charles)" |
||||
sudo -H -u charles python {{ webhooks_install_dir }}/scripts/captain_hook_pull_host.py |
||||
echo "All done." |
||||
rm -f "/tmp/triggers/push-b-captain-hook-master" |
||||
touch /tmp/canary-yup-host-works |
||||
fi |
||||
|
||||
sleep 10; |
||||
done |
||||
|
||||
|
||||
@ -0,0 +1,110 @@
@@ -0,0 +1,110 @@
|
||||
#!/usr/bin/env python3 |
||||
import subprocess |
||||
import os |
||||
import time |
||||
|
||||
""" |
||||
Captain Hook: Pull Captain Hook on the Host |
||||
|
||||
This script is called by the host machine |
||||
(blackbeard) running the Captain Hook container. |
||||
|
||||
This is triggered by push actions to the |
||||
master branch of b-captain-hook. |
||||
|
||||
The action is to update (git pull) the copy |
||||
of Captain Hook running on the host, and |
||||
restart the container pod. |
||||
""" |
||||
|
||||
pod_dir = '{{ webhooks_install_dir }}' |
||||
work_dir = os.path.join(pod_dir,'b-captain-hook') |
||||
|
||||
from datetime import datetime |
||||
d = datetime.now().strftime('%Y-m-%d') |
||||
with open('/tmp/captain_hook_pull_host_%s.log'%(d),'w') as f: |
||||
|
||||
# Step 1: |
||||
# Update Captain Hook |
||||
co_cmd = ['git','checkout','master'] |
||||
proc = subprocess.call(co_cmd, cwd=work_dir, stdout=subprocess.PIPE, stderr=subprocess.PIPE) |
||||
|
||||
stdout, stderr = proc.communicate() |
||||
o = stdout.decode('utf-8') |
||||
e = stderr.decode('utf-8') |
||||
f.write(" ".join(co_cmd)) |
||||
f.write("\n") |
||||
f.write("-"*40) |
||||
f.write("\n") |
||||
f.write(o) |
||||
f.write("\n") |
||||
f.write(e) |
||||
f.write("\n\n") |
||||
|
||||
f_cmd = ['git','fetch','--all'] |
||||
proc = subprocess.call(f_cmd, cwd=work_dir, stdout=subprocess.PIPE, stderr=subprocess.PIPE) |
||||
|
||||
stdout, stderr = proc.communicate() |
||||
o = stdout.decode('utf-8') |
||||
e = stderr.decode('utf-8') |
||||
f.write(" ".join(f_cmd)) |
||||
f.write("\n") |
||||
f.write("-"*40) |
||||
f.write("\n") |
||||
f.write(o) |
||||
f.write("\n") |
||||
f.write(e) |
||||
f.write("\n\n") |
||||
time.sleep(5) |
||||
|
||||
pull_cmd = ['git','pull','origin','master'] |
||||
proc = subprocess.call(pull_cmd, cwd=work_dir, stdout=subprocess.PIPE, stderr=subprocess.PIPE) |
||||
|
||||
stdout, stderr = proc.communicate() |
||||
o = stdout.decode('utf-8') |
||||
e = stderr.decode('utf-8') |
||||
f.write(" ".join(pull_cmd)) |
||||
f.write("\n") |
||||
f.write("-"*40) |
||||
f.write("\n") |
||||
f.write(o) |
||||
f.write("\n") |
||||
f.write(e) |
||||
f.write("\n\n") |
||||
time.sleep(10) |
||||
|
||||
submod_cmd = ['git','submodule','update','--remote'] |
||||
proc = subprocess.call(submod_cmd, cwd=work_dir, stdout=subprocess.PIPE, stderr=subprocess.PIPE) |
||||
|
||||
stdout, stderr = proc.communicate() |
||||
o = stdout.decode('utf-8') |
||||
e = stderr.decode('utf-8') |
||||
f.write(" ".join(pull_cmd)) |
||||
f.write("\n") |
||||
f.write("-"*40) |
||||
f.write("\n") |
||||
f.write(o) |
||||
f.write("\n") |
||||
f.write(e) |
||||
f.write("\n\n") |
||||
time.sleep(10) |
||||
|
||||
|
||||
# Step 2: |
||||
# Restart Captain Hook pod |
||||
pod_restart = ['systemctl','restart','pod-webhooks'] |
||||
#pod_restart = ['docker-compose','restart'] |
||||
proc = subprocess.call(pod_restart, cwd=pod_dir, stdout=subprocess.PIPE, stderr=subprocess.PIPE) |
||||
|
||||
stdout, stderr = proc.communicate() |
||||
o = stdout.decode('utf-8') |
||||
e = stderr.decode('utf-8') |
||||
f.write(" ".join(pod_restart)) |
||||
f.write("\n") |
||||
f.write("-"*40) |
||||
f.write("\n") |
||||
f.write(o) |
||||
f.write("\n") |
||||
f.write(e) |
||||
f.write("\n\n") |
||||
|
||||
@ -0,0 +1,21 @@
@@ -0,0 +1,21 @@
|
||||
#!/usr/bin/env python3 |
||||
import subprocess |
||||
import os |
||||
|
||||
def execute(cmd): |
||||
""" |
||||
A function to run a command and return the |
||||
lines of output as they are generated, |
||||
allowing the calling function to "stream" |
||||
the output of the command to print() or etc. |
||||
""" |
||||
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True) |
||||
for stdout_line in iter(p.stdout.readline, ""): |
||||
yield stdout_line |
||||
p.stdout.close() |
||||
err = "".join([j for j in iter(p.stderr.readline,"")]) |
||||
return_code = p.wait() |
||||
if return_code: |
||||
yield err |
||||
raise subprocess.CalledProcessError(return_code, cmd) |
||||
|
||||
@ -0,0 +1,125 @@
@@ -0,0 +1,125 @@
|
||||
#!/usr/bin/env python3 |
||||
import subprocess |
||||
import os |
||||
from executioner import execute |
||||
|
||||
""" |
||||
Pages Initial Setup |
||||
|
||||
This script sets up the initial pages.charlesreid1.XYZ |
||||
folder structure and checks out all pages that should |
||||
be present initially at pages.charlesreid1.XYZ. |
||||
|
||||
This clones the page for a given project to |
||||
|
||||
/www/pages.charlesreid1.XYZ/htdocs/project-name |
||||
|
||||
(where project = repo) |
||||
""" |
||||
|
||||
SERVER_NAME_DEFAULT = '{{ server_name_default }}' |
||||
USERNAME = '{{ username }}' |
||||
|
||||
|
||||
|
||||
# Repos must have a gh-pages branch |
||||
repo_names = '''bots/b-apollo |
||||
bots/b-captain-hook |
||||
bots/b-ginsberg |
||||
bots/b-milton |
||||
bots/uncle-archie |
||||
bots/boring-mind-machine |
||||
docker/d-gitea |
||||
docker/d-mediawiki |
||||
docker/d-mysql |
||||
docker/d-nginx-charlesreid1 |
||||
docker/d-nginx-subdomains |
||||
docker/d-phpmyadmin |
||||
docker/d-python-files |
||||
docker/d-python-helium |
||||
charlesreid1/dont-sudo-pip |
||||
bots/embarcadero-mind-machine |
||||
charlesreid1/git-commit-ectomy |
||||
charlesreid1/git-subway-maps |
||||
charlesreid1/github-heroku-attack-rabbits |
||||
charlesreid1/how-do-i-heroku |
||||
charlesreid1/how-do-i-pandoc |
||||
charlesreid1/how-do-i-pelican |
||||
charlesreid1/how-do-i-pyenv |
||||
charlesreid1/how-do-i-snakemake |
||||
docker/pod-bots |
||||
docker/pod-charlesreid1 |
||||
docker/pod-webhooks |
||||
bots/russian-rainbow-mind-machine |
||||
charlesreid1/scurvy-knave-theme |
||||
charlesreid1/translate-yer-docs |
||||
charlesreid1/wisko-manual'''.split('\n') |
||||
|
||||
repo_urls = ['https://git.charlesreid1.com/%s.git'%(j) for j in repo_names] |
||||
|
||||
root = '/www' |
||||
pages = 'pages.%s'%(SERVER_NAME_DEFAULT) |
||||
basedir = os.path.join(root,pages) |
||||
|
||||
if not os.path.exists(basedir): |
||||
raise Exception("Error: base directory %s does not exist!"%(basedir)) |
||||
|
||||
for name,url in zip(repo_names,repo_urls): |
||||
|
||||
# for a hypothetical repo "project": |
||||
# |
||||
# base dir: /www/pages.charlesreid1.com |
||||
# .git dir: /www/pages.charlesreid1.com/git.project |
||||
# htdocs dir: /www/pages.charlesreid1.com/htdocs/project |
||||
|
||||
# Step 1: clone |
||||
# Construct the clone command to set up |
||||
# live htdocs directory for this project |
||||
|
||||
repo_owner, repo_name = name.split("/") |
||||
|
||||
clonepth = os.path.join(basedir,"htdocs",repo_name) |
||||
dotgitpth = os.path.join(basedir,"git.%s"%(repo_name)) |
||||
|
||||
if os.path.exists(clonepth): |
||||
print("Git repository has already been cloned, %s already exists on disk!"%(clonepth)) |
||||
|
||||
else: |
||||
|
||||
clonecmd = ["git","clone", |
||||
"--separate-git-dir=%s"%(dotgitpth), |
||||
"-b","gh-pages", |
||||
url,clonepth] |
||||
|
||||
print("About to clone gh-pages branch for %s using command:\n"%(name)) |
||||
print(" $ " + " ".join(clonecmd)) |
||||
print("\n") |
||||
|
||||
try: |
||||
for loo in execute(clonecmd): |
||||
print(loo) |
||||
print("\n") |
||||
print("Done.") |
||||
|
||||
except subprocess.CalledProcessError: |
||||
print("Failed on clone command for project %s"%(name)) |
||||
print("Continuing...") |
||||
|
||||
|
||||
|
||||
# Step 2: chown everybody |
||||
# Construct chown command |
||||
chowncmd = ['chown','-R', |
||||
USERNAME+":"+USERNAME, |
||||
basedir] |
||||
|
||||
print("About to change owner of %s using command:\n"%(basedir)) |
||||
print(" $ " + " ".join(chowncmd)) |
||||
print("\n") |
||||
|
||||
for loo in execute(chowncmd): |
||||
print(loo) |
||||
print("\n") |
||||
|
||||
print("Done.") |
||||
|
||||
@ -0,0 +1,129 @@
@@ -0,0 +1,129 @@
|
||||
#!/usr/bin/env python3 |
||||
import subprocess |
||||
import os |
||||
from executioner import execute |
||||
|
||||
""" |
||||
Pages Pull Script |
||||
|
||||
This script pulls chnages to all repos in the |
||||
pages.charlesreid1.XYZ folder structure |
||||
|
||||
This clones the page for a given project to |
||||
|
||||
/www/pages.charlesreid1.XYZ/htdocs/project-name |
||||
|
||||
(where project = repo) |
||||
""" |
||||
|
||||
SERVER_NAME_DEFAULT = '{{ server_name_default }}' |
||||
USERNAME = '{{ username }}' |
||||
|
||||
|
||||
|
||||
# Repos must have a gh-pages branch |
||||
repo_names = '''bots/b-apollo |
||||
bots/b-captain-hook |
||||
bots/b-ginsberg |
||||
bots/b-milton |
||||
bots/uncle-archie |
||||
bots/boring-mind-machine |
||||
docker/d-gitea |
||||
docker/d-mediawiki |
||||
docker/d-mysql |
||||
docker/d-nginx-charlesreid1 |
||||
docker/d-nginx-subdomains |
||||
docker/d-phpmyadmin |
||||
docker/d-python-files |
||||
docker/d-python-helium |
||||
charlesreid1/dont-sudo-pip |
||||
bots/embarcadero-mind-machine |
||||
charlesreid1/git-commit-ectomy |
||||
charlesreid1/git-subway-maps |
||||
charlesreid1/github-heroku-attack-rabbits |
||||
charlesreid1/how-do-i-heroku |
||||
charlesreid1/how-do-i-pandoc |
||||
charlesreid1/how-do-i-pelican |
||||
charlesreid1/how-do-i-pyenv |
||||
charlesreid1/how-do-i-snakemake |
||||
docker/pod-bots |
||||
docker/pod-charlesreid1 |
||||
docker/pod-webhooks |
||||
bots/russian-rainbow-mind-machine |
||||
charlesreid1/scurvy-knave-theme |
||||
charlesreid1/translate-yer-docs |
||||
charlesreid1/wisko-manual'''.split('\n') |
||||
|
||||
repo_urls = ['https://git.charlesreid1.com/%s.git'%(j) for j in repo_names] |
||||
|
||||
root = '/www' |
||||
pages = 'pages.%s'%(SERVER_NAME_DEFAULT) |
||||
basedir = os.path.join(root,pages) |
||||
htdocsdir = os.path.join(basedir,"htdocs") |
||||
|
||||
if not os.path.exists(basedir): |
||||
raise Exception("Error: base directory %s does not exist!"%(basedir)) |
||||
|
||||
if not os.path.exists(htdocsdir): |
||||
raise Exception("Error: htdocs directory %s does not exist!"%(htdocsdir)) |
||||
|
||||
for name,url in zip(repo_names,repo_urls): |
||||
|
||||
# for a hypothetical repo "project": |
||||
# |
||||
# base dir: /www/pages.charlesreid1.com |
||||
# .git dir: /www/pages.charlesreid1.com/git.project |
||||
# htdocs dir: /www/pages.charlesreid1.com/htdocs/project |
||||
|
||||
# Step 1: pull |
||||
# Construct the git pull command to update |
||||
# the live htdocs directory for this project |
||||
|
||||
repo_owner, repo_name = name.split("/") |
||||
|
||||
workdir = os.path.join(basedir,"htdocs",repo_name) |
||||
dotgitdir = os.path.join(basedir,"git.%s"%(repo_name)) |
||||
|
||||
if not os.path.exists(workdir): |
||||
print("Git repository has not yet been cloned, %s does not exist on disk!"%(workdir)) |
||||
|
||||
else: |
||||
|
||||
pullcmd = ['git', |
||||
'-C', basedir, |
||||
'--git-dir=%s'%(dotgitdir), |
||||
'--work-tree=%s'%(workdir), |
||||
'pull','origin','gh-pages'] |
||||
|
||||
print("About to pull gh-pages branch for %s using command:\n"%(name)) |
||||
print(" $ " + " ".join(pullcmd)) |
||||
print("\n") |
||||
|
||||
try: |
||||
for loo in execute(pullcmd): |
||||
print(loo) |
||||
print("\n") |
||||
print("Done.") |
||||
|
||||
except subprocess.CalledProcessError: |
||||
print("Failed on pull command for project %s"%(name)) |
||||
print("Continuing...") |
||||
|
||||
|
||||
|
||||
# Step 2: chown everybody |
||||
# Construct chown command |
||||
chowncmd = ['chown','-R', |
||||
USERNAME+":"+USERNAME, |
||||
basedir] |
||||
|
||||
print("About to change owner of %s using command:\n"%(basedir)) |
||||
print(" $ " + " ".join(chowncmd)) |
||||
print("\n") |
||||
|
||||
for loo in execute(chowncmd): |
||||
print(loo) |
||||
print("\n") |
||||
|
||||
print("Done.") |
||||
|
||||
@ -0,0 +1,14 @@
@@ -0,0 +1,14 @@
|
||||
# Hopefully this comment works |
||||
|
||||
[Unit] |
||||
Description=webhooks and subdomains docker pod |
||||
Requires=docker.service |
||||
After=docker.service |
||||
|
||||
[Service] |
||||
Restart=always |
||||
ExecStart=/usr/local/bin/docker-compose -f {{ webhooks_install_dir }}/docker-compose.yml up |
||||
ExecStop=/usr/local/bin/docker-compose -f {{ webhooks_install_dir }}/docker-compose.yml down |
||||
|
||||
[Install] |
||||
WantedBy=default.target |
||||
@ -0,0 +1,89 @@
@@ -0,0 +1,89 @@
|
||||
#!/usr/bin/env python3 |
||||
import subprocess |
||||
import os |
||||
from executioner import execute |
||||
|
||||
""" |
||||
Subdomains Initial Setup |
||||
|
||||
This script sets up the initial folder structure |
||||
for the charlesreid1.com subdomains and their content. |
||||
""" |
||||
|
||||
SERVER_NAME_DEFAULT = '{{ server_name_default }}' |
||||
USERNAME = '{{ username }}' |
||||
|
||||
|
||||
|
||||
# Each subdomain page has an htdocs dir at |
||||
# |
||||
# /www/SUBDOMAIN.charlesreid1.XYZ/htdocs |
||||
# |
||||
# and a .git directory at |
||||
# |
||||
# /www/SUBDOMAIN.charlesreid1.XYZ/git.SUBDOMAIN.charlesreid1.XYZ |
||||
|
||||
# Name of all subdomains to set up |
||||
sub_names = ['bots','pages','hooks'] |
||||
subs = [n+'.'+SERVER_NAME_DEFAULT for n in sub_names] |
||||
|
||||
# Use git.charlesreid1.com |
||||
# (Note we have to hard-code the suffix |
||||
# charlesreid1.com b/c that is the name |
||||
# of the repo for ALL subdomains) |
||||
urls = ['https://git.charlesreid1.com/charlesreid1/%s.charlesreid1.com'%(n) for n in sub_names] |
||||
# |
||||
# Use github.com (alternative) |
||||
# (Note we have to hard-code the suffix charlesreid1.com) |
||||
#urls = ['https://github.com/charlesreid1-docker/%s.charlesreid1.com'%(n) for n in sub_names] |
||||
|
||||
# Base directory for all htdocs folders |
||||
pth = '/www' |
||||
|
||||
# Create the base directory |
||||
subprocess.call(['mkdir','-p',pth]) |
||||
|
||||
# Step 1: clone htdocs dir for each subdomain |
||||
for subname, sub, url in zip(sub_names,subs,urls): |
||||
|
||||
# Construct the clone command for the live |
||||
# htdocs directory for this subdomain |
||||
clonepth = os.path.join(pth,sub,'htdocs') |
||||
dotgitpth = os.path.join(pth,sub,'git.%s'%(subname)) |
||||
|
||||
if os.path.exists(clonepth): |
||||
print("Git repository has already been cloned, %s already exists on disk!"%(clonepth)) |
||||
|
||||
else: |
||||
|
||||
clonecmd = ['git','clone', |
||||
'--separate-git-dir=%s'%(dotgitpth), |
||||
'-b','gh-pages', |
||||
url,clonepth] |
||||
|
||||
print("About to clone htdocs dir for %s using command:\n"%(sub)) |
||||
print(" $ " + " ".join(clonecmd)) |
||||
print("\n") |
||||
|
||||
for loo in execute(clonecmd): |
||||
print(loo) |
||||
print("\n") |
||||
|
||||
print("Done.") |
||||
|
||||
|
||||
|
||||
# Step 2: chown |
||||
# Construct chown command to chown everybody |
||||
chowncmd = ['chown','-R',USERNAME+":"+USERNAME,pth] |
||||
|
||||
print("About to change owner of %s using command:\n"%(pth)) |
||||
print(" $ " + " ".join(chowncmd)) |
||||
print("\n") |
||||
|
||||
for loo in execute(chowncmd): |
||||
print(loo) |
||||
print("\n") |
||||
|
||||
print("Done.") |
||||
|
||||
@ -0,0 +1,95 @@
@@ -0,0 +1,95 @@
|
||||
#!/usr/bin/env python3 |
||||
import subprocess |
||||
import os |
||||
from executioner import execute |
||||
|
||||
""" |
||||
Subdomains Pull Script |
||||
|
||||
This script sets up the initial folder structure |
||||
for the charlesreid1.com subdomains and their content. |
||||
""" |
||||
|
||||
SERVER_NAME_DEFAULT = '{{ server_name_default }}' |
||||
USERNAME = '{{ username }}' |
||||
|
||||
|
||||
|
||||
# Each subdomain page has an htdocs dir at |
||||
# |
||||
# /www/SUBDOMAIN.charlesreid1.XYZ/htdocs |
||||
# |
||||
# and a .git directory at |
||||
# |
||||
# /www/SUBDOMAIN.charlesreid1.XYZ/git.SUBDOMAIN.charlesreid1.XYZ |
||||
|
||||
# Name of all subdomains to set up |
||||
sub_names = ['bots','pages','hooks'] |
||||
subs = [n+'.'+SERVER_NAME_DEFAULT for n in sub_names] |
||||
|
||||
# Use git.charlesreid1.com |
||||
# (Note we have to hard-code the suffix |
||||
# charlesreid1.com b/c that is the name |
||||
# of the repo for ALL subdomains) |
||||
urls = ['https://git.charlesreid1.com/charlesreid1/%s.charlesreid1.com'%(n) for n in sub_names] |
||||
# |
||||
# Use github.com (alternative) |
||||
# (Note we have to hard-code the suffix charlesreid1.com) |
||||
#urls = ['https://github.com/charlesreid1-docker/%s.charlesreid1.com'%(n) for n in sub_names] |
||||
|
||||
# Base directory for all htdocs folders |
||||
pth = '/www' |
||||
|
||||
# Create the base directory |
||||
subprocess.call(['mkdir','-p',pth]) |
||||
|
||||
# Step 1: clone htdocs dir for each subdomain |
||||
for subname, sub, url in zip(sub_names,subs,urls): |
||||
|
||||
# Step 1: pull |
||||
# Construct the git pull command to update |
||||
# the live htdocs directory for this project |
||||
|
||||
basedir = os.path.join(pth,sub) |
||||
htdocsdir = os.path.join(basedir,"htdocs") |
||||
dotgitdir = os.path.join(basedir,"git.%s"%(subname)) |
||||
|
||||
if not os.path.exists(htdocsdir): |
||||
print("Git repository has not been cloned, %s does not exist on disk!"%(htdocsdir)) |
||||
|
||||
else: |
||||
|
||||
# git pull, respecting the |
||||
# unusual .git/repo folder layout |
||||
pullcmd = ['git', |
||||
'-C', basedir, |
||||
'--git-dir=%s'%(dotgitdir), |
||||
'--work-tree=%s'%(htdocsdir), |
||||
'pull','origin','gh-pages'] |
||||
|
||||
|
||||
print("About to pull htdocs dir for %s using command:\n"%(sub)) |
||||
print(" $ " + " ".join(pullcmd)) |
||||
print("\n") |
||||
|
||||
for loo in execute(pullcmd): |
||||
print(loo) |
||||
print("\n") |
||||
print("Done.") |
||||
|
||||
|
||||
|
||||
# Step 2: chown |
||||
# Construct chown command to chown everybody |
||||
chowncmd = ['chown','-R',USERNAME+":"+USERNAME,basedir] |
||||
|
||||
print("About to change owner of %s using command:\n"%(basedir)) |
||||
print(" $ " + " ".join(chowncmd)) |
||||
print("\n") |
||||
|
||||
for loo in execute(chowncmd): |
||||
print(loo) |
||||
print("\n") |
||||
|
||||
print("Done.") |
||||
|
||||
Loading…
Reference in new issue