Compare commits
1 Commits
testing
...
releases/v
| Author | SHA1 | Date | |
|---|---|---|---|
| 2099c60c7a |
2
.gitignore
vendored
2
.gitignore
vendored
@@ -1,4 +1,4 @@
|
||||
feedback_database.json
|
||||
config_centillion.py
|
||||
config_flask.py
|
||||
vp
|
||||
credentials.json
|
||||
|
||||
@@ -12,13 +12,8 @@ one centillion is 3.03 log-times better than a googol.
|
||||
## What Is It
|
||||
|
||||
Centillion (https://github.com/dcppc/centillion) is a search engine that can index
|
||||
different kinds of document collections: Google Documents (.docx files), Google Drive files,
|
||||
Github issues, Github files, Github Markdown files, and Groups.io email threads.
|
||||
|
||||
|
||||
|
||||
|
||||
## What Is It
|
||||
three kinds of collections: Google Documents (.docx files), Github issues, and Markdown files in
|
||||
Github repos.
|
||||
|
||||
We define the types of documents the centillion should index,
|
||||
what info and how. The centillion then builds and
|
||||
|
||||
56
Todo.md
56
Todo.md
@@ -1,29 +1,47 @@
|
||||
# todo
|
||||
|
||||
ux improvements:
|
||||
- feedback tools
|
||||
- integrating master list into single list
|
||||
- providing advanced search interfce
|
||||
Main task:
|
||||
- hashing and caching
|
||||
- <s>first, working out the logic of how we group items into sets
|
||||
- needs to be deleted
|
||||
- needs to be updated
|
||||
- needs to be added
|
||||
- for docs, issues, and comments</s>
|
||||
- second, when we add or update an item, need to:
|
||||
- go through the motions, download file, extract text
|
||||
- check for existing indexed doc with that id
|
||||
- check if existing indexed doc has same hash
|
||||
- if so, skip
|
||||
- otherwise, delete and re-index
|
||||
|
||||
big picture improvements:
|
||||
- hypothesis API
|
||||
- folksonomy tagging with hypothesis
|
||||
- tags, expanded schema
|
||||
Other bugs:
|
||||
- Some github issues have no title (?)
|
||||
- <s>Need to combine issues with comments</s>
|
||||
- Not able to index markdown files _in a repo_
|
||||
- (Longer term) update main index vs update diff index
|
||||
|
||||
Needs:
|
||||
- <s>control panel</s>
|
||||
|
||||
Thursday product:
|
||||
- Everything re-indexed nightly
|
||||
- Search engine built on all documents in Google Drive, all issues, markdown files
|
||||
- Using pandoc to extract Google Drive document contents
|
||||
- BRIEF quickstart documentation
|
||||
|
||||
Future:
|
||||
- Future plans to improve - plugins, improving matching
|
||||
- Subdomain plans
|
||||
- Folksonomy tagging and integration plans
|
||||
|
||||
|
||||
|
||||
|
||||
feedback form: where we are at
|
||||
- feedback button
|
||||
- button triggers modal form
|
||||
- modal has emojis for feedback, text box, buttons
|
||||
- clicking emojis changes color, to select
|
||||
- clicking submit with filled out form submits to an endpoint
|
||||
- clicking submit also closes form, but only if submit successful
|
||||
|
||||
feedback form: what we need to do
|
||||
- fix alerts - thank you for your feedback doesn't show up until a refresh
|
||||
- probably an easy ajax fix
|
||||
config options for plugins
|
||||
conditional blocks with import github inside
|
||||
complicated tho - better to have components split off
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -3,7 +3,6 @@ import subprocess
|
||||
|
||||
import codecs
|
||||
import os, json
|
||||
from datetime import datetime
|
||||
|
||||
from werkzeug.contrib.fixers import ProxyFix
|
||||
from flask import Flask, request, redirect, url_for, render_template, flash, jsonify
|
||||
@@ -263,57 +262,13 @@ def list_docs(doctype):
|
||||
all_orgs = resp.json()
|
||||
for org in all_orgs:
|
||||
if org['login']=='dcppc':
|
||||
# Business as usual
|
||||
search = Search(app.config["INDEX_DIR"])
|
||||
return jsonify(search.get_list(doctype))
|
||||
|
||||
# nope
|
||||
return render_template('403.html')
|
||||
|
||||
|
||||
@app.route('/feedback', methods=['POST'])
|
||||
def parse_request():
|
||||
|
||||
if not github.authorized:
|
||||
return redirect(url_for("github.login"))
|
||||
username = github.get("/user").json()['login']
|
||||
resp = github.get("/user/orgs")
|
||||
if resp.ok:
|
||||
all_orgs = resp.json()
|
||||
for org in all_orgs:
|
||||
if org['login']=='dcppc':
|
||||
|
||||
|
||||
try:
|
||||
copper_team_id = '2700235'
|
||||
mresp = github.get('/teams/%s/members/%s'%(copper_team_id,username))
|
||||
if mresp.status_code==204:
|
||||
# Business as usual
|
||||
data = request.form.to_dict();
|
||||
data['github_login'] = username
|
||||
data['timestamp'] = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
|
||||
search = Search(app.config["INDEX_DIR"])
|
||||
return jsonify(search.get_list(doctype))
|
||||
|
||||
feedback_database = 'feedback_database.json'
|
||||
if not os.path.isfile(feedback_database):
|
||||
with open(feedback_database,'w') as f:
|
||||
json_data = [data]
|
||||
json.dump(json_data, f, indent=4)
|
||||
|
||||
else:
|
||||
json_data = []
|
||||
with open(feedback_database,'r') as f:
|
||||
json_data = json.load(f)
|
||||
|
||||
json_data.append(data)
|
||||
|
||||
with open(feedback_database,'w') as f:
|
||||
json.dump(json_data, f, indent=4)
|
||||
|
||||
## Should be done with Javascript
|
||||
#flash("Thank you for your feedback!")
|
||||
return jsonify({'status':'ok','message':'Thank you for your feedback!'})
|
||||
except:
|
||||
return jsonify({'status':'error','message':'An error was encountered while submitting your feedback. Try submitting an issue in the <a href="https://github.com/dcppc/centillion/issues/new">dcppc/centillion</a> repository.'})
|
||||
|
||||
|
||||
# nope
|
||||
return render_template('403.html')
|
||||
|
||||
@app.errorhandler(404)
|
||||
@@ -342,10 +297,5 @@ def store_search(query, fields):
|
||||
if __name__ == '__main__':
|
||||
# if running local instance, set to true
|
||||
os.environ['OAUTHLIB_INSECURE_TRANSPORT'] = 'true'
|
||||
port = os.environ.get('CENTILLION_PORT','')
|
||||
if port=='':
|
||||
port = 5000
|
||||
else:
|
||||
port = int(port)
|
||||
app.run(host="0.0.0.0",port=port)
|
||||
app.run(host="0.0.0.0",port=5000)
|
||||
|
||||
|
||||
@@ -1,28 +0,0 @@
|
||||
config = {
|
||||
"repositories" : [
|
||||
"dcppc/project-management",
|
||||
"dcppc/nih-demo-meetings",
|
||||
"dcppc/internal",
|
||||
"dcppc/organize",
|
||||
"dcppc/dcppc-bot",
|
||||
"dcppc/full-stacks",
|
||||
"dcppc/design-guidelines-discuss",
|
||||
"dcppc/dcppc-deliverables",
|
||||
"dcppc/dcppc-milestones",
|
||||
"dcppc/crosscut-metadata",
|
||||
"dcppc/lucky-penny",
|
||||
"dcppc/dcppc-workshops",
|
||||
"dcppc/metadata-matrix",
|
||||
"dcppc/data-stewards",
|
||||
"dcppc/dcppc-phase1-demos",
|
||||
"dcppc/apis",
|
||||
"dcppc/2018-june-workshop",
|
||||
"dcppc/2018-july-workshop",
|
||||
"dcppc/2018-august-workshop",
|
||||
"dcppc/2018-september-workshop",
|
||||
"dcppc/design-guidelines",
|
||||
"dcppc/2018-may-workshop",
|
||||
"dcppc/centillion"
|
||||
]
|
||||
}
|
||||
|
||||
@@ -1,181 +0,0 @@
|
||||
# Centillion Quality Engineering Plan
|
||||
|
||||
Table of Contents
|
||||
-------
|
||||
|
||||
* [Centillion Quality Engineering Plan](#centillion-quality-engineering-plan)
|
||||
* [Summary](#summary)
|
||||
* [Tracking Bugs and Issues](#tracking-bugs-and-issues)
|
||||
* [Branches, Versioning, and Git Workflow](#branches-versioning-and-git-workflow)
|
||||
* [Communication and Mailing Lists](#communication-and-mailing-lists)
|
||||
* [Checklists](#checklists)
|
||||
* [Documentation](#documentation)
|
||||
* [Configuration Management Tools](#configuration-management-tools)
|
||||
* [Tests](#tests)
|
||||
* [Code Reviews](#code-reviews)
|
||||
* [Formal Release Process](#formal-release-process)
|
||||
* [Continual Process Improvement](#continual-process-improvement)
|
||||
|
||||
Summary
|
||||
-------
|
||||
|
||||
This document contains a quality engineering plan for centillion, the
|
||||
Data Commons search engine.
|
||||
|
||||
Tracking Bugs and Issues
|
||||
------------------------
|
||||
|
||||
We utilize the [issues
|
||||
section](https://github.com/dcppc/centillion/issues) of the centillion
|
||||
repository to keep track of bugs and feature requests.
|
||||
|
||||
Branches, Versioning, and Git Workflow
|
||||
--------------------------------------
|
||||
|
||||
All code is kept under version control in the
|
||||
[dcppc/centillion](https://github.com/dcppc/centillion) Github
|
||||
repository.
|
||||
|
||||
**Primary Git Branches:**
|
||||
|
||||
We utillize a git branch pattern that has two primary branches: a
|
||||
development branch and a stable branch.
|
||||
|
||||
- The primary **development branch** is `dcppc` and is actively
|
||||
developed and deployed to <https://betasearch.nihdatacommons.us>.
|
||||
|
||||
- The primary **stable branch** is `releases/v1` and is stable and
|
||||
deployed to <https://search.nihdatacommons.us>.
|
||||
|
||||
All tagged versions of Centillion exist on the stable branch. Only
|
||||
tagged versions of centillion are run on
|
||||
<https://search.nihdatacommons.us>.
|
||||
|
||||
**Other Branches:**
|
||||
|
||||
Features are developed by creating a new branch from `dcppc`, working on
|
||||
the feature, and opening a pull request. When the pull request is
|
||||
approved, it can be merged into the `dcppc` branch.
|
||||
|
||||
When features have accumulated and a new version is ready, a new
|
||||
pre-release branch will be made to prepare for a new release. When the
|
||||
pre-release branch is ready, it is merged into the stable branch in a
|
||||
single merge commit and a new version of centillion is tagged. The new
|
||||
version is deployed on <https://search.nihdatacommons.us>.
|
||||
|
||||
Commits to fix bugs (hotfixes) may need to be applied to both the stable
|
||||
and development branches. In this case, a hotfix branch should be
|
||||
created from the head commit of the stable branch, and the appropriate
|
||||
changes should be made on the branch. A pull request should be opened to
|
||||
merge the hotfix into the release branch. A second pull request should
|
||||
be opened to merge the hotfix into the development branch. Once the
|
||||
hotfix is merged into the stable branch, a new version should be tagged.
|
||||
|
||||
Communication and Mailing Lists
|
||||
-------------------------------
|
||||
|
||||
- No mailing list currently exists for centillion.
|
||||
|
||||
- Github issues are the primary form of communication about
|
||||
development of centillion. This is the best method for communicating
|
||||
bug reports or detailed information.
|
||||
|
||||
- The Send Feedback button on the centillion page is the primary way
|
||||
of getting quick feedback from users about the search engine.
|
||||
|
||||
- The [\#centillion](https://nih-dcppc.slack.com/messages/CCD64QD6G)
|
||||
Slack channel in the DCPPC slack workspace is the best place for
|
||||
conversations about centillion (providing feedback, answering quick
|
||||
questions, etc.)
|
||||
|
||||
Checklists
|
||||
----------
|
||||
|
||||
We plan to utilize the Wiki feature of the Github repository to develop
|
||||
checlists:
|
||||
|
||||
- Checklist for releases
|
||||
- Checklist for deployment of https://search.nihdatacommons.us nginx
|
||||
etc.
|
||||
|
||||
Documentation
|
||||
-------------
|
||||
|
||||
The documentation is a pile of markdown documents, turned into a static
|
||||
site using mkdocs.
|
||||
|
||||
Configuration Management Tools
|
||||
------------------------------
|
||||
|
||||
We do not currently utilize any configuration management software,
|
||||
because centillion is not packaged as an importable Python module.
|
||||
|
||||
Packaging centillion is a future goal that is closely related to the
|
||||
need to improve and modularize the internal search schema/document type
|
||||
abstraction. These improvements would allow the types of collections
|
||||
being indexed to be separated from "core centillion", and core
|
||||
centillion would be packaged.
|
||||
|
||||
Tests
|
||||
-----
|
||||
|
||||
See (ref) for a full test plan with more detail.
|
||||
|
||||
Summary of test plan:
|
||||
|
||||
- Implement tests for the four major pages/components
|
||||
- Login/authentication
|
||||
- Search
|
||||
- Master List
|
||||
- Control Panel
|
||||
- Test authentication with two bot accounts (yammasnake and florence
|
||||
python)
|
||||
|
||||
- Separate frontend and backend tests
|
||||
|
||||
- Add a test flag in the flask config file to change the backend
|
||||
behavior of the server
|
||||
|
||||
Code Reviews
|
||||
------------
|
||||
|
||||
CI tests will be implemented for all pull requests.
|
||||
|
||||
Pull requests to the **stable branch** have the following checks in
|
||||
place:
|
||||
|
||||
- PRs to the stable branch require at least 1 PR review
|
||||
- PRs to the stable branch must pass CI tests
|
||||
|
||||
Pull requests to the **development branch** have the following checks in
|
||||
place:
|
||||
|
||||
- PRs to the development branch must pass CI tests
|
||||
|
||||
Formal Release Process
|
||||
----------------------
|
||||
|
||||
In order to ensure a stable, consistent product, we utilize the
|
||||
branching pattern described above to implement new features in the
|
||||
development branch and test them out on
|
||||
<https://betasearch.nihdatacommons.us>.
|
||||
|
||||
Once features and bug fixes have been tested and reviewed internally,
|
||||
they are ready to be deployed. A new pre-release branch is created from
|
||||
the development branch. The pre-release branch has a feature freeze in
|
||||
place. Changes are made to the pre-release branch to prepare it for the
|
||||
next major version release.
|
||||
|
||||
When the pre-release branch is finished, it is merged into the stable
|
||||
branch. The head commit of the stable version is tagged with the lastest
|
||||
release number.
|
||||
|
||||
Finally, the new version is deployed on
|
||||
<https://search.nihdatacommons.us>.
|
||||
|
||||
Continual Process Improvement
|
||||
-----------------------------
|
||||
|
||||
We will utilize the centillion wiki on Github to keep track of repeated
|
||||
processes and opportunities for improvement. Feedback and ideas for
|
||||
process improvement can also be submitted via Github issues.
|
||||
@@ -1,133 +0,0 @@
|
||||
// submitting form with modal:
|
||||
// https://stackoverflow.com/a/29068742
|
||||
//
|
||||
// closing a bootstrap modal with submit button:
|
||||
// https://stackoverflow.com/a/33478107
|
||||
//
|
||||
// flask post data as json:
|
||||
// https://stackoverflow.com/a/16664376
|
||||
|
||||
/* this function is called when the user submits
|
||||
* the feedback form. it submits a post request
|
||||
* to the flask server, which squirrels away the
|
||||
* feedback in a file.
|
||||
*/
|
||||
function submit_feedback() {
|
||||
// this function is called when submit button clicked
|
||||
// algorithm:
|
||||
// - check if text box has content
|
||||
// - check if happy/sad filled out
|
||||
|
||||
var smile_active = $('#modal-feedback-smile-div').hasClass('smile-active');
|
||||
var frown_active = $('#modal-feedback-frown-div').hasClass('frown-active');
|
||||
if( !( smile_active || frown_active ) ) {
|
||||
alert('Please pick the smile or the frown.')
|
||||
} else if( $('#modal-feedback-textarea').val()=='' ) {
|
||||
alert('Please provide us with some feedback.')
|
||||
} else {
|
||||
var user_sentiment = '';
|
||||
if(smile_active) {
|
||||
user_sentiment = 'smile';
|
||||
} else {
|
||||
user_sentiment = 'frown';
|
||||
}
|
||||
var escaped_text = $('#modal-feedback-textarea').val();
|
||||
|
||||
// prepare form data
|
||||
var data = {
|
||||
sentiment : user_sentiment,
|
||||
content : escaped_text
|
||||
};
|
||||
// post the form. the callback function resets the form
|
||||
$.post("/feedback",
|
||||
data,
|
||||
function(response) {
|
||||
$('#myModal').modal('hide');
|
||||
$('#myModalForm')[0].reset();
|
||||
add_alert(response);
|
||||
frown_unclick();
|
||||
smile_unclick();
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
function add_alert(response) {
|
||||
str = ""
|
||||
str += '<div id="feedback-messages-container" class="container">';
|
||||
|
||||
if (response['status']=='ok') {
|
||||
// if status is ok, use alert-success
|
||||
str += ' <div id="feedback-messages-alert" class="alert alert-success alert-dismissible fade in">';
|
||||
} else {
|
||||
// otherwise use alert-danger
|
||||
str += ' <div id="feedback-messages-alert" class="alert alert-danger alert-dismissible fade in">';
|
||||
}
|
||||
|
||||
str += ' <a href="#" class="close" data-dismiss="alert" aria-label="close">×</a>';
|
||||
str += ' <div id="feedback-messages-contianer" class="container-fluid">';
|
||||
str += ' <div id="feedback-messages-div" class="co-xs-12">';
|
||||
str += ' <p>'
|
||||
str += response['message'];
|
||||
str += ' </p>';
|
||||
str += ' </div>';
|
||||
str += ' </div>';
|
||||
str += '</div>';
|
||||
$('div#messages').append(str);
|
||||
}
|
||||
|
||||
|
||||
/* for those particularly wordy users... limit feedback to 1000 chars */
|
||||
function cool_it() {
|
||||
if($('#modal-feedback-textarea').val().length > 1100 ){
|
||||
$('#modal-too-long').show();
|
||||
} else {
|
||||
$('#modal-too-long').hide();
|
||||
}
|
||||
}
|
||||
|
||||
/* smiley functions */
|
||||
function smile_click() {
|
||||
$('#modal-feedback-smile-div').addClass('smile-active');
|
||||
$('#modal-feedback-smile-icon').addClass('smile-active');
|
||||
}
|
||||
function frown_click() {
|
||||
$('#modal-feedback-frown-div').addClass('frown-active');
|
||||
$('#modal-feedback-frown-icon').addClass('frown-active');
|
||||
}
|
||||
function smile_unclick() {
|
||||
$('#modal-feedback-smile-div').removeClass('smile-active');
|
||||
$('#modal-feedback-smile-icon').removeClass('smile-active');
|
||||
}
|
||||
function frown_unclick() {
|
||||
$('#modal-feedback-frown-div').removeClass('frown-active');
|
||||
$('#modal-feedback-frown-icon').removeClass('frown-active');
|
||||
}
|
||||
|
||||
function smile() {
|
||||
frown_unclick();
|
||||
smile_click();
|
||||
}
|
||||
function frown() {
|
||||
smile_unclick();
|
||||
frown_click();
|
||||
}
|
||||
|
||||
|
||||
/* for those particularly wordy users... limit feedback to 1100 chars */
|
||||
// how to check n characters in a textarea
|
||||
// https://stackoverflow.com/a/19934613
|
||||
/*
|
||||
$(document).ready(function() {
|
||||
|
||||
$('#modal-feedback-textarea').on('change',function(event) {
|
||||
if($('#modal-feedback-textarea').val().length > 1100 ){
|
||||
$('#modal-too-long').show();
|
||||
} else {
|
||||
$('#modal-too-long').hide();
|
||||
}
|
||||
});
|
||||
|
||||
}
|
||||
*/
|
||||
|
||||
@@ -1,61 +1,3 @@
|
||||
#modal-too-long {
|
||||
visibility: hidden;
|
||||
}
|
||||
|
||||
/* feedback smileys */
|
||||
#modal-feedback-smile-icon,
|
||||
#modal-feedback-frown-icon {
|
||||
padding-left: 100px;
|
||||
padding-right: 100px;
|
||||
padding-top: 20px;
|
||||
padding-bottom: 20px;
|
||||
}
|
||||
|
||||
div.smile-active {
|
||||
background-color: #2b2;
|
||||
}
|
||||
i.smile-active {
|
||||
color: #fff;
|
||||
}
|
||||
div.frown-active {
|
||||
background-color: #b22;
|
||||
}
|
||||
i.frown-active {
|
||||
color: #fff;
|
||||
}
|
||||
|
||||
/* feedback text area */
|
||||
#modal-feedback-textarea {
|
||||
width: 100%;
|
||||
}
|
||||
|
||||
/* feedback buttons */
|
||||
button.close {
|
||||
font-size: 35px;
|
||||
}
|
||||
button#submit-feedback-btn {
|
||||
width: 250px;
|
||||
}
|
||||
|
||||
button#feedback:hover {
|
||||
opacity: 1.0;
|
||||
filter: alpha(opacity=100); /* For IE8 and earlier */
|
||||
}
|
||||
|
||||
button#feedback {
|
||||
opacity: 0.5;
|
||||
filter: alpha(opacity=50); /* For IE8 and earlier */
|
||||
|
||||
width: 180px;
|
||||
height: 50px;
|
||||
position: fixed;
|
||||
z-index: 999;
|
||||
right: 120px;
|
||||
bottom: 10px;
|
||||
}
|
||||
|
||||
|
||||
/* search results table */
|
||||
td#search-results-score-col,
|
||||
td#search-results-type-col {
|
||||
width: 100px;
|
||||
|
||||
@@ -1,5 +1,4 @@
|
||||
{% extends "layout.html" %}
|
||||
{% set active_page = "403" %}
|
||||
{% block body %}
|
||||
|
||||
<div class="container">
|
||||
|
||||
@@ -1,5 +1,4 @@
|
||||
{% extends "layout.html" %}
|
||||
{% set active_page = "404" %}
|
||||
{% block body %}
|
||||
|
||||
<div class="container">
|
||||
|
||||
@@ -1,25 +0,0 @@
|
||||
<div class="container" id="banner-container">
|
||||
|
||||
{#
|
||||
banner image
|
||||
#}
|
||||
<div class="row" id="banner-row">
|
||||
<div class="col12sm" id="banner-col">
|
||||
<center>
|
||||
<a id="banner-a" href="{{ url_for('search')}}?query=&fields=">
|
||||
<img id="banner-img" src="{{ url_for('static', filename='centillion_white.png') }}">
|
||||
</a>
|
||||
</center>
|
||||
</div>
|
||||
</div>
|
||||
{% if config['TAGLINE'] %}
|
||||
<div class="row" id="tagline-row">
|
||||
<div class="col12sm" id="tagline-col">
|
||||
<center>
|
||||
<h2 id="tagline-tagline"> {{config['TAGLINE']}} </h2>
|
||||
</center>
|
||||
</div>
|
||||
</div>
|
||||
{% endif %}
|
||||
|
||||
</div>
|
||||
@@ -1,5 +1,4 @@
|
||||
{% extends "layout.html" %}
|
||||
{% set active_page = "control_panel" %}
|
||||
{% block body %}
|
||||
|
||||
<hr />
|
||||
|
||||
@@ -1,14 +0,0 @@
|
||||
<div id="messages">
|
||||
{% with messages = get_flashed_messages() %}
|
||||
{% if messages %}
|
||||
<div class="container" id="flashed-messages-container">
|
||||
<div class="alert alert-success alert-dismissible fade in">
|
||||
<a href="#" class="close" data-dismiss="alert" aria-label="close">×</a>
|
||||
{% for message in messages %}
|
||||
<p class="lead">{{ message }}</p>
|
||||
{% endfor %}
|
||||
</div>
|
||||
</div>
|
||||
{% endif %}
|
||||
{% endwith %}
|
||||
</div>
|
||||
@@ -1,5 +1,4 @@
|
||||
{% extends "layout.html" %}
|
||||
{% set active_page = "landing" %}
|
||||
{% block body %}
|
||||
<div class="container">
|
||||
<div class="row">
|
||||
|
||||
@@ -10,7 +10,6 @@
|
||||
<script src="{{ url_for('static', filename='bootstrap.min.js') }}"></script>
|
||||
<script src="{{ url_for('static', filename='master_list.js') }}"></script>
|
||||
<script src="{{ url_for('static', filename='search_list.js') }}"></script>
|
||||
<script src="{{ url_for('static', filename='feedback.js') }}"></script>
|
||||
|
||||
|
||||
{# ########## dataTables plugin ############ #}
|
||||
@@ -24,43 +23,52 @@
|
||||
|
||||
|
||||
|
||||
<div id="master-div">
|
||||
{# ########## github fork corner ############ #}
|
||||
|
||||
{#
|
||||
flashed messages
|
||||
#}
|
||||
{% include "flashed_messages.html" %}
|
||||
<div>
|
||||
|
||||
{#
|
||||
banner image
|
||||
#}
|
||||
{% include "banner.html" %}
|
||||
{% with messages = get_flashed_messages() %}
|
||||
{% if messages %}
|
||||
<div class="container">
|
||||
<div class="alert alert-success alert-dismissible">
|
||||
<a href="#" class="close" data-dismiss="alert" aria-label="close">×</a>
|
||||
<ul class=flashes>
|
||||
{% for message in messages %}
|
||||
<li>{{ message }}</li>
|
||||
{% endfor %}
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
{% endif %}
|
||||
{% endwith %}
|
||||
|
||||
{#
|
||||
feedback modal
|
||||
#}
|
||||
{% include "modal.html" %}
|
||||
|
||||
{% block body %}{% endblock %}
|
||||
|
||||
</div>
|
||||
|
||||
{% if active_page=="search" or active_page=="master_list" %}
|
||||
|
||||
{# feedback button #}
|
||||
<button id="feedback" type="button"
|
||||
data-toggle="modal"
|
||||
data-target="#myModal"
|
||||
class="btn btn-lg">Send Feedback</button>
|
||||
<div class="container">
|
||||
|
||||
{#
|
||||
banner image
|
||||
#}
|
||||
<div class="row">
|
||||
<div class="col12sm">
|
||||
<center>
|
||||
<a href="{{ url_for('search')}}?query=&fields=">
|
||||
<img src="{{ url_for('static', filename='centillion_white.png') }}">
|
||||
</a>
|
||||
{#
|
||||
need a tag line
|
||||
#}
|
||||
{% if config['TAGLINE'] %}
|
||||
<h2><a href="{{ url_for('search')}}?query=&fields=">
|
||||
{{config['TAGLINE']}}
|
||||
</a></h2>
|
||||
{% endif %}
|
||||
</center>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{# vertical spacing before the bottom, b/c of button #}
|
||||
<div id="footer-whitespace" class="container">
|
||||
<p> </p>
|
||||
<p> </p>
|
||||
<p> </p>
|
||||
</div>
|
||||
|
||||
{% endif %}
|
||||
{% block body %}{% endblock %}
|
||||
</div>
|
||||
|
||||
<a id="github-corner" href="https://github.com/dcppc/centillion" class="github-corner" aria-label="View source on Github"><svg width="80" height="80" viewBox="0 0 250 250" style="fill:#151513; color:#fff; position: absolute; top: 0; border: 0; right: 0;" aria-hidden="true"><path d="M0,0 L115,115 L130,115 L142,142 L250,250 L250,0 Z"></path><path d="M128.3,109.0 C113.8,99.7 119.0,89.6 119.0,89.6 C122.0,82.7 120.5,78.6 120.5,78.6 C119.2,72.0 123.4,76.3 123.4,76.3 C127.3,80.9 125.5,87.3 125.5,87.3 C122.9,97.6 130.6,101.9 134.4,103.2" fill="currentColor" style="transform-origin: 130px 106px;" class="octo-arm"></path><path d="M115.0,115.0 C114.9,115.1 118.7,116.5 119.8,115.4 L133.7,101.6 C136.9,99.2 139.9,98.4 142.2,98.6 C133.8,88.0 127.5,74.4 143.8,58.0 C148.5,53.4 154.0,51.2 159.7,51.0 C160.3,49.4 163.2,43.6 171.4,40.1 C171.4,40.1 176.1,42.5 178.8,56.2 C183.1,58.6 187.2,61.8 190.9,65.4 C194.5,69.0 197.7,73.2 200.1,77.6 C213.8,80.2 216.3,84.9 216.3,84.9 C212.7,93.1 206.9,96.0 205.4,96.6 C205.1,102.4 203.0,107.8 198.3,112.5 C181.9,128.9 168.3,122.5 157.7,114.1 C157.9,116.9 156.7,120.9 152.7,124.9 L141.0,136.5 C139.8,137.7 141.6,141.9 141.8,141.8 Z" fill="currentColor" class="octo-body"></path></svg></a><style>.github-corner:hover .octo-arm{animation:octocat-wave 560ms ease-in-out}@keyframes octocat-wave{0%,100%{transform:rotate(0)}20%,60%{transform:rotate(-25deg)}40%,80%{transform:rotate(10deg)}}@media (max-width:500px){.github-corner:hover .octo-arm{animation:none}.github-corner .octo-arm{animation:octocat-wave 560ms ease-in-out}}</style>
|
||||
<a href="https://github.com/dcppc/centillion" class="github-corner" aria-label="View source on Github"><svg width="80" height="80" viewBox="0 0 250 250" style="fill:#151513; color:#fff; position: absolute; top: 0; border: 0; right: 0;" aria-hidden="true"><path d="M0,0 L115,115 L130,115 L142,142 L250,250 L250,0 Z"></path><path d="M128.3,109.0 C113.8,99.7 119.0,89.6 119.0,89.6 C122.0,82.7 120.5,78.6 120.5,78.6 C119.2,72.0 123.4,76.3 123.4,76.3 C127.3,80.9 125.5,87.3 125.5,87.3 C122.9,97.6 130.6,101.9 134.4,103.2" fill="currentColor" style="transform-origin: 130px 106px;" class="octo-arm"></path><path d="M115.0,115.0 C114.9,115.1 118.7,116.5 119.8,115.4 L133.7,101.6 C136.9,99.2 139.9,98.4 142.2,98.6 C133.8,88.0 127.5,74.4 143.8,58.0 C148.5,53.4 154.0,51.2 159.7,51.0 C160.3,49.4 163.2,43.6 171.4,40.1 C171.4,40.1 176.1,42.5 178.8,56.2 C183.1,58.6 187.2,61.8 190.9,65.4 C194.5,69.0 197.7,73.2 200.1,77.6 C213.8,80.2 216.3,84.9 216.3,84.9 C212.7,93.1 206.9,96.0 205.4,96.6 C205.1,102.4 203.0,107.8 198.3,112.5 C181.9,128.9 168.3,122.5 157.7,114.1 C157.9,116.9 156.7,120.9 152.7,124.9 L141.0,136.5 C139.8,137.7 141.6,141.9 141.8,141.8 Z" fill="currentColor" class="octo-body"></path></svg></a><style>.github-corner:hover .octo-arm{animation:octocat-wave 560ms ease-in-out}@keyframes octocat-wave{0%,100%{transform:rotate(0)}20%,60%{transform:rotate(-25deg)}40%,80%{transform:rotate(10deg)}}@media (max-width:500px){.github-corner:hover .octo-arm{animation:none}.github-corner .octo-arm{animation:octocat-wave 560ms ease-in-out}}</style>
|
||||
|
||||
|
||||
@@ -1,5 +1,4 @@
|
||||
{% extends "layout.html" %}
|
||||
{% set active_page = "master_list" %}
|
||||
{% block body %}
|
||||
|
||||
<hr />
|
||||
|
||||
@@ -1,51 +0,0 @@
|
||||
<div class="modal fade" id="myModal" tabindex="-1" role="dialog" aria-labelledby="myModalLabel">
|
||||
|
||||
<form id="myModalForm" method="post">
|
||||
|
||||
<div class="modal-dialog" role="document">
|
||||
<div id="myModal-content" class="modal-content">
|
||||
<div id="myModal-header" class="modal-header">
|
||||
<button type="button" class="close" data-dismiss="modal" aria-label="Close">
|
||||
<span aria-hidden="true">×</span>
|
||||
</button>
|
||||
<h4 class="modal-title" id="myModalLabel">
|
||||
Send us feedback!
|
||||
</h4>
|
||||
</div>
|
||||
<div id="myModal-body" class="modal-body">
|
||||
<div id="modal-feedback-smile-frown-container" class="container-fluid">
|
||||
<div id="modal-feedback-smile-div" class="col-xs-6 text-center"
|
||||
onClick="smile()">
|
||||
<i id="modal-feedback-smile-icon" class="fa fa-smile-o fa-4x" aria-hidden="true"></i>
|
||||
</div>
|
||||
<div id="modal-feedback-frown-div" class="col-xs-6 text-center"
|
||||
onClick="frown()">
|
||||
<i id="modal-feedback-frown-icon" class="fa fa-frown-o fa-4x" aria-hidden="true"></i>
|
||||
</div>
|
||||
</div>
|
||||
<div class="container-fluid">
|
||||
<p> </p>
|
||||
</div>
|
||||
<div id="modal-feedback-textarea-container" class="container-fluid">
|
||||
<textarea id="modal-feedback-textarea" rows="6"></textarea>
|
||||
</div>
|
||||
<div id="modal-too-long" class="container-fluid" >
|
||||
<p id="modal-too-long-text" class="lead">Please limit the length of your feedback. Thank you in advance!</p>
|
||||
</div>
|
||||
</div>
|
||||
<div id="myModal-footer" class="modal-footer">
|
||||
<div class="text-center">
|
||||
<button id="submit-feedback-btn" type="button"
|
||||
onClick="submit_feedback()"
|
||||
class="btn btn-lg btn-primary">
|
||||
Send
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</form>
|
||||
|
||||
</div>
|
||||
|
||||
@@ -1,8 +1,7 @@
|
||||
{% extends "layout.html" %}
|
||||
{% set active_page = "search" %}
|
||||
{% block body %}
|
||||
|
||||
<div id="search-bar-container" class="container">
|
||||
<div class="container">
|
||||
<div class="row">
|
||||
|
||||
<div class="col-xs-12">
|
||||
@@ -13,11 +12,7 @@
|
||||
<p><button id="the-big-one" type="submit" style="font-size: 20px; padding: 10px; padding-left: 50px; padding-right: 50px;"
|
||||
value="search" class="btn btn-primary">Search</button>
|
||||
</p>
|
||||
|
||||
{% if parsed_query %}
|
||||
<p><a href="{{ url_for('search')}}?query=&fields=">[clear all results]</a>
|
||||
{% endif %}
|
||||
|
||||
<p><a href="{{ url_for('search')}}?query=&fields=">[clear all results]</a>
|
||||
</p>
|
||||
</form>
|
||||
</center>
|
||||
@@ -25,11 +20,18 @@
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div style="height: 20px;"><p> </p></div>
|
||||
|
||||
<div id="info-bars-container" class="container">
|
||||
<div class="container">
|
||||
<div class="row">
|
||||
|
||||
{% if directories %}
|
||||
<div class="col-xs-12 info directories-cloud">
|
||||
<b>File directories:</b>
|
||||
{% for d in directories %}
|
||||
<a href="{{url_for('search')}}?query={{d|trim}}&fields=filename">{{d|trim}}</a>
|
||||
{% endfor %}
|
||||
</div>
|
||||
{% endif %}
|
||||
|
||||
<ul class="list-group">
|
||||
|
||||
{% if config['SHOW_PARSED_QUERY'] and parsed_query %}
|
||||
@@ -44,9 +46,6 @@
|
||||
</li>
|
||||
{% endif %}
|
||||
|
||||
|
||||
{# use "if parsed_query" to check if this is
|
||||
a new search or search results #}
|
||||
{% if parsed_query %}
|
||||
<li class="list-group-item">
|
||||
<div class="container-fluid">
|
||||
@@ -60,7 +59,6 @@
|
||||
</li>
|
||||
{% endif %}
|
||||
|
||||
|
||||
<li class="list-group-item">
|
||||
<div class="container-fluid">
|
||||
<div class="row">
|
||||
@@ -101,7 +99,7 @@
|
||||
</div>
|
||||
|
||||
{% if parsed_query %}
|
||||
<div id="search-results-container" class="container">
|
||||
<div class="container">
|
||||
<div class="row">
|
||||
<table id="search-results" class="table">
|
||||
<thead id="search-results-header">
|
||||
@@ -128,21 +126,44 @@
|
||||
{% if e.kind=="gdoc" %}
|
||||
{% if e.mimetype=="document" %}
|
||||
<p><small>Drive Document</small</p>
|
||||
<!--
|
||||
<i class="fa fa-google fa-2x"></i>
|
||||
<i class="fa fa-file-text fa-2x"></i>
|
||||
-->
|
||||
{% else %}
|
||||
<p><small>Drive File</small</p>
|
||||
<!--
|
||||
<i class="fa fa-google fa-2x"></i>
|
||||
<i class="fa fa-file-o fa-2x"></i>
|
||||
-->
|
||||
{% endif %}
|
||||
|
||||
{% elif e.kind=="issue" %}
|
||||
<p><small>Issue</small</p>
|
||||
<!--
|
||||
<i class="fa fa-github fa-2x"></i>
|
||||
<i class="fa fa-question fa-2x"></i>
|
||||
-->
|
||||
|
||||
{% elif e.kind=="ghfile" %}
|
||||
<p><small>Github File</small</p>
|
||||
<!--
|
||||
<i class="fa fa-github fa-2x"></i>
|
||||
<i class="fa fa-file-o fa-2x"></i>
|
||||
-->
|
||||
|
||||
{% elif e.kind=="markdown" %}
|
||||
<p><small>Github Markdown</small</p>
|
||||
<!--
|
||||
<i class="fa fa-github fa-2x"></i>
|
||||
<i class="fa fa-file-text-o fa-2x"></i>
|
||||
-->
|
||||
|
||||
{% elif e.kind=="emailthread" %}
|
||||
<p><small>Email Thread</small</p>
|
||||
<!--
|
||||
<i class="fa fa-envelope-o fa-2x"></i>
|
||||
-->
|
||||
|
||||
{% else %}
|
||||
<p><small>Unknown</small</p>
|
||||
|
||||
196
tests/Readme.md
196
tests/Readme.md
@@ -1,196 +0,0 @@
|
||||
Centillion Tests
|
||||
================
|
||||
|
||||
Table of Contents
|
||||
------------------
|
||||
|
||||
* [Centillion Tests](#centillion-tests)
|
||||
* [Test Plan](#test-plan)
|
||||
* [Local Tests](#local-tests)
|
||||
* [Short Tests](#short-tests)
|
||||
* [Long Tests](#long-tests)
|
||||
* [Credentials](#credentials)
|
||||
* [Detailed Description of Tests](#detailed-description-of-tests)
|
||||
* [Authentication Layer Tests](#authentication-layer-tests)
|
||||
* [Search Function Tests](#search-function-tests)
|
||||
* [Master List Endpoint Tests](#master-list-endpoint-tests)
|
||||
* [Control Panel Endpoint Tests](#control-panel-endpoint-tests)
|
||||
* [Continuous Integration Plan](#continuous-integration-plan)
|
||||
* [Procedure/Checklist](#procedurechecklist)
|
||||
|
||||
|
||||
Test Plan
|
||||
---------
|
||||
|
||||
Related: <https://github.com/dcppc/centillion/issues/82>
|
||||
|
||||
The test suite for centillion needs to check each of the major
|
||||
components of centillion, as well as check the authentication mechanism
|
||||
using multiple login credentials.
|
||||
|
||||
We implement the following checks:
|
||||
|
||||
1. Check authentication mechanism(s) (yamasnake and florence python)
|
||||
|
||||
2. Check search function
|
||||
|
||||
3. Check master list endpoint
|
||||
|
||||
4. Check control panel endpoint
|
||||
|
||||
5. Check update search index endpoints
|
||||
|
||||
The tests are written such that the back end and front end are tested
|
||||
separately.
|
||||
|
||||
We need also need different tiers of tests, so we don't max out API
|
||||
calls by making lots of commits to multiple PRs.
|
||||
|
||||
We have three tiers of tests: \* Local tests - quick tests for CI, no
|
||||
API calls \* Short tests - tests using dummy API accounts \* Long tests
|
||||
- tests using DCPPC API accounts
|
||||
|
||||
### Local Tests
|
||||
|
||||
Local tests can be run locally without any interaction with APIs. These
|
||||
will still utilize centillion's search schema, but will load the search
|
||||
index with fake documents rather than fetching them from an API.
|
||||
|
||||
Uncle Archie, which runs CI tests, runs local tests only (unless you
|
||||
request it to run short test or long test.)
|
||||
|
||||
### Short Tests
|
||||
|
||||
Short tests utilize credentials for bot accounts that have intentionally
|
||||
been set up to have a "known" corpus of test documents. These would
|
||||
provide unit-style tests for centillion - are the mechanics of indexing
|
||||
a particular type of document from a particular API working?
|
||||
|
||||
### Long Tests
|
||||
|
||||
Long tests are indexing the real deal, utilizing the credentials used in
|
||||
the final production centillion. This test takes longer but is more
|
||||
likely to catch corner cases specific to the DCPPC documents.
|
||||
|
||||
Credentials
|
||||
-----------
|
||||
|
||||
Running tests on centillion requires multiple sets of credentials. Let's
|
||||
lay out what is needed:
|
||||
|
||||
- The Flask app requires a token/secret token API key pair to allow
|
||||
users to authenticate through Github and confirm they are members of
|
||||
the DCPPC organization. This OAuth application is owned by Charles
|
||||
Reid (@charlesreid1).
|
||||
|
||||
- The search index needs a Github access token so that it can
|
||||
interface with the Github API to index files and issues. This access
|
||||
token is specified (along with other secrets) in the Flask
|
||||
configuration file. The access key comes from Florence Python
|
||||
(@fp9695253).
|
||||
|
||||
- The search index also requires a Google Drive API access token. This
|
||||
must be an access token for a user who has authenticated with the
|
||||
Centillion Google Drive OAuth application. This access token comes
|
||||
from <mailroom@nihdatacommons.com>.
|
||||
|
||||
- The search index requires API credentials for any other APIs
|
||||
associated with other document collections (Groups.io, Hypothesis,
|
||||
Disqus).
|
||||
|
||||
- The backend test requires the credentials provided to Flask.
|
||||
|
||||
- The frontend test (Selenium) needs two Github username/passwords:
|
||||
one for Florence Python (@fp9695253) and one for Yamma Snake
|
||||
(@yammasnake). These are required to simulate the user
|
||||
authenticating with Github through the browser.
|
||||
- The frontend test credentials are a special case.
|
||||
- The frontend tests expect credentials to come from environment
|
||||
variables.
|
||||
- These environment variables get passed in at test time.
|
||||
- Tests are all run on [Uncle
|
||||
Archie](https://github.com/dcppc/uncle-archie).
|
||||
- Uncle Archie already has to protect a confidential config file
|
||||
containing Github credentials, so add additional credentials for
|
||||
frontend tests there.
|
||||
- Logical separation: these credentials are not needed to
|
||||
*operate* centillion, these credentials are needed to *test*
|
||||
centillion
|
||||
- Uncle Archie already requires github credentials, already
|
||||
protects sensitive info.
|
||||
- Google Drive requiring its own credentials file on disk is a
|
||||
pain.
|
||||
|
||||
In summary: tests use the `config_flask.py` and `config_centillion.py`
|
||||
files to provide it with the API keys it needs and to instruct it on
|
||||
what to index. The credentials and config files will control what the
|
||||
search index will actually index. The Uncle Archie CI tester config file
|
||||
contains the credentials needed to run frontend tests (check the
|
||||
login/authentication layer).
|
||||
|
||||
Detailed Description of Tests
|
||||
-----------------------------
|
||||
|
||||
### Authentication Layer Tests
|
||||
|
||||
Frontend tests run as Florence Python:
|
||||
|
||||
- Can we log in via github and reach centillion
|
||||
- Can we reach the control panel
|
||||
|
||||
Frontend tests run as Yamma Snake (DCPPC member):
|
||||
|
||||
- Can we log in via github and reach centillion
|
||||
- Can we reach the control panel
|
||||
|
||||
### Search Function Tests
|
||||
|
||||
Frontend tests:
|
||||
|
||||
- Can we enter something into search box and submit
|
||||
- Can we sort the results
|
||||
- Do the results look okay
|
||||
|
||||
Backend tests:
|
||||
|
||||
- Load the search index and run a query using whoosh API
|
||||
|
||||
### Master List Endpoint Tests
|
||||
|
||||
Frontend tests:
|
||||
|
||||
- Can we get to the master list page
|
||||
- Can we sort the results
|
||||
- Do the results look okay
|
||||
|
||||
Backend tests:
|
||||
|
||||
- Check the output of the `/list` API endpoint
|
||||
|
||||
### Control Panel Endpoint Tests
|
||||
|
||||
Frontend tests:
|
||||
|
||||
- Can we get to the control panel page
|
||||
- Can we click the button to trigger an indexing event
|
||||
|
||||
Backend tests:
|
||||
|
||||
- Trigger a re-index of the search index from the backend.
|
||||
|
||||
### Continuous Integration Plan
|
||||
|
||||
Tests are automatically run using Uncle Archie for continuous
|
||||
integration and deployment.
|
||||
|
||||
Procedure/Checklist
|
||||
-------------------
|
||||
|
||||
Pre-release procedure:
|
||||
|
||||
- prepare to run all test
|
||||
|
||||
- run short tests
|
||||
- deploy to beta
|
||||
- run long tests
|
||||
- test out
|
||||
Reference in New Issue
Block a user