update test plan with TOC + autoformatting changes from pandoc

This commit was the result of running a pandocs command
to clean up the formatting and generate a table of contents
for the markdown document.

.gitignore

@@ -1,4 +1,4 @@
-config_centillion.py
+feedback_database.json
 config_flask.py
 vp
 credentials.json

Readme.md

@@ -12,8 +12,13 @@ one centillion is 3.03 log-times better than a googol.
 ## What Is It
 
 Centillion (https://github.com/dcppc/centillion) is a search engine that can index 
-three kinds of collections: Google Documents (.docx files), Github issues, and Markdown files in 
-Github repos.
+different kinds of document collections: Google Documents (.docx files), Google Drive files,
+Github issues, Github files, Github Markdown files, and Groups.io email threads.
+
+
+
+
+## What Is It
 
 We define the types of documents the centillion should index,
 what info and how. The centillion then builds and

Todo.md

@@ -1,47 +1,29 @@
 # todo
 
-Main task:
-- hashing and caching
-    - <s>first, working out the logic of how we group items into sets
-        - needs to be deleted
-        - needs to be updated
-        - needs to be added
-        - for docs, issues, and comments</s>
-    - second, when we add or update an item, need to:
-        - go through the motions, download file, extract text
-        - check for existing indexed doc with that id
-        - check if existing indexed doc has same hash
-            - if so, skip
-            - otherwise, delete and re-index
+ux improvements:
+- feedback tools
+- integrating master list into single list
+- providing advanced search interfce
 
-Other bugs:
-- Some github issues have no title (?)
-- <s>Need to combine issues with comments</s>
-- Not able to index markdown files _in a repo_
-- (Longer term) update main index vs update diff index
-
-Needs:
-- <s>control panel</s>
-
-Thursday product:
-- Everything re-indexed nightly
-- Search engine built on all documents in Google Drive, all issues, markdown files
-- Using pandoc to extract Google Drive document contents
-- BRIEF quickstart documentation
-
-Future:
-- Future plans to improve - plugins, improving matching
-- Subdomain plans
-- Folksonomy tagging and integration plans
+big picture improvements:
+- hypothesis API
+- folksonomy tagging with hypothesis
+- tags, expanded schema
 
 
 
 
-config options for plugins
-conditional blocks with import github inside
-complicated tho - better to have components split off
-
-
+feedback form: where we are at
+- feedback button
+- button triggers modal form
+- modal has emojis for feedback, text box, buttons
+- clicking emojis changes color, to select
+- clicking submit with filled out form submits to an endpoint
+- clicking submit also closes form, but only if submit successful
+
+feedback form: what we need to do
+- fix alerts - thank you for your feedback doesn't show up until a refresh
+- probably an easy ajax fix
 
 
 

centillion.py

@@ -3,6 +3,7 @@ import subprocess
 
 import codecs
 import os, json
+from datetime import datetime
 
 from werkzeug.contrib.fixers import ProxyFix
 from flask import Flask, request, redirect, url_for, render_template, flash, jsonify
@@ -262,13 +263,57 @@ def list_docs(doctype):
         all_orgs = resp.json()
         for org in all_orgs:
             if org['login']=='dcppc':
-                copper_team_id = '2700235'
-                mresp = github.get('/teams/%s/members/%s'%(copper_team_id,username))
-                if mresp.status_code==204:
-                    # Business as usual
-                    search = Search(app.config["INDEX_DIR"])
-                    return jsonify(search.get_list(doctype))
+                # Business as usual
+                search = Search(app.config["INDEX_DIR"])
+                return jsonify(search.get_list(doctype))
 
+    # nope
+    return render_template('403.html')
+
+
+@app.route('/feedback', methods=['POST'])
+def parse_request():
+
+    if not github.authorized:
+        return redirect(url_for("github.login"))
+    username = github.get("/user").json()['login']
+    resp = github.get("/user/orgs")
+    if resp.ok:
+        all_orgs = resp.json()
+        for org in all_orgs:
+            if org['login']=='dcppc':
+
+
+                try:
+                    # Business as usual
+                    data = request.form.to_dict();
+                    data['github_login'] = username
+                    data['timestamp'] = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
+
+                    feedback_database = 'feedback_database.json'
+                    if not os.path.isfile(feedback_database):
+                        with open(feedback_database,'w') as f:
+                            json_data = [data]
+                            json.dump(json_data, f, indent=4)
+
+                    else:
+                        json_data = []
+                        with open(feedback_database,'r') as f:
+                            json_data = json.load(f)
+
+                        json_data.append(data)
+
+                        with open(feedback_database,'w') as f:
+                            json.dump(json_data, f, indent=4)
+
+                    ## Should be done with Javascript
+                    #flash("Thank you for your feedback!")
+                    return jsonify({'status':'ok','message':'Thank you for your feedback!'})
+                except:
+                    return jsonify({'status':'error','message':'An error was encountered while submitting your feedback. Try submitting an issue in the <a href="https://github.com/dcppc/centillion/issues/new">dcppc/centillion</a> repository.'})
+
+
+    # nope
     return render_template('403.html')
 
 @app.errorhandler(404)
@@ -297,5 +342,10 @@ def store_search(query, fields):
 if __name__ == '__main__':
     # if running local instance, set to true
     os.environ['OAUTHLIB_INSECURE_TRANSPORT'] = 'true'
-    app.run(host="0.0.0.0",port=5000)
+    port = os.environ.get('CENTILLION_PORT','')
+    if port=='':
+        port = 5000
+    else:
+        port = int(port)
+    app.run(host="0.0.0.0",port=port)
 

config_centillion.py

@@ -0,0 +1,28 @@
+config = {
+    "repositories" : [
+        "dcppc/project-management",
+        "dcppc/nih-demo-meetings",
+        "dcppc/internal",
+        "dcppc/organize",
+        "dcppc/dcppc-bot",
+        "dcppc/full-stacks",
+        "dcppc/design-guidelines-discuss",
+        "dcppc/dcppc-deliverables",
+        "dcppc/dcppc-milestones",
+        "dcppc/crosscut-metadata",
+        "dcppc/lucky-penny",
+        "dcppc/dcppc-workshops",
+        "dcppc/metadata-matrix",
+        "dcppc/data-stewards",
+        "dcppc/dcppc-phase1-demos",
+        "dcppc/apis",
+        "dcppc/2018-june-workshop",
+        "dcppc/2018-july-workshop",
+        "dcppc/2018-august-workshop",
+        "dcppc/2018-september-workshop",
+        "dcppc/design-guidelines",
+        "dcppc/2018-may-workshop",
+        "dcppc/centillion"
+    ]
+}
+

quality/Readme.md

@@ -0,0 +1,181 @@
+# Centillion Quality Engineering Plan
+
+Table of Contents
+-------
+
+*   [Centillion Quality Engineering Plan](#centillion-quality-engineering-plan)
+    *   [Summary](#summary)
+    *   [Tracking Bugs and Issues](#tracking-bugs-and-issues)
+    *   [Branches, Versioning, and Git Workflow](#branches-versioning-and-git-workflow)
+    *   [Communication and Mailing Lists](#communication-and-mailing-lists)
+    *   [Checklists](#checklists)
+    *   [Documentation](#documentation)
+    *   [Configuration Management Tools](#configuration-management-tools)
+    *   [Tests](#tests)
+    *   [Code Reviews](#code-reviews)
+    *   [Formal Release Process](#formal-release-process)
+    *   [Continual Process Improvement](#continual-process-improvement)
+
+Summary
+-------
+
+This document contains a quality engineering plan for centillion, the
+Data Commons search engine.
+
+Tracking Bugs and Issues
+------------------------
+
+We utilize the [issues
+section](https://github.com/dcppc/centillion/issues) of the centillion
+repository to keep track of bugs and feature requests.
+
+Branches, Versioning, and Git Workflow
+--------------------------------------
+
+All code is kept under version control in the
+[dcppc/centillion](https://github.com/dcppc/centillion) Github
+repository.
+
+**Primary Git Branches:**
+
+We utillize a git branch pattern that has two primary branches: a
+development branch and a stable branch.
+
+-   The primary **development branch** is `dcppc` and is actively
+    developed and deployed to <https://betasearch.nihdatacommons.us>.
+
+-   The primary **stable branch** is `releases/v1` and is stable and
+    deployed to <https://search.nihdatacommons.us>.
+
+All tagged versions of Centillion exist on the stable branch. Only
+tagged versions of centillion are run on
+<https://search.nihdatacommons.us>.
+
+**Other Branches:**
+
+Features are developed by creating a new branch from `dcppc`, working on
+the feature, and opening a pull request. When the pull request is
+approved, it can be merged into the `dcppc` branch.
+
+When features have accumulated and a new version is ready, a new
+pre-release branch will be made to prepare for a new release. When the
+pre-release branch is ready, it is merged into the stable branch in a
+single merge commit and a new version of centillion is tagged. The new
+version is deployed on <https://search.nihdatacommons.us>.
+
+Commits to fix bugs (hotfixes) may need to be applied to both the stable
+and development branches. In this case, a hotfix branch should be
+created from the head commit of the stable branch, and the appropriate
+changes should be made on the branch. A pull request should be opened to
+merge the hotfix into the release branch. A second pull request should
+be opened to merge the hotfix into the development branch. Once the
+hotfix is merged into the stable branch, a new version should be tagged.
+
+Communication and Mailing Lists
+-------------------------------
+
+-   No mailing list currently exists for centillion.
+
+-   Github issues are the primary form of communication about
+    development of centillion. This is the best method for communicating
+    bug reports or detailed information.
+
+-   The Send Feedback button on the centillion page is the primary way
+    of getting quick feedback from users about the search engine.
+
+-   The [\#centillion](https://nih-dcppc.slack.com/messages/CCD64QD6G)
+    Slack channel in the DCPPC slack workspace is the best place for
+    conversations about centillion (providing feedback, answering quick
+    questions, etc.)
+
+Checklists
+----------
+
+We plan to utilize the Wiki feature of the Github repository to develop
+checlists:
+
+-   Checklist for releases
+-   Checklist for deployment of https://search.nihdatacommons.us nginx
+    etc.
+
+Documentation
+-------------
+
+The documentation is a pile of markdown documents, turned into a static
+site using mkdocs.
+
+Configuration Management Tools
+------------------------------
+
+We do not currently utilize any configuration management software,
+because centillion is not packaged as an importable Python module.
+
+Packaging centillion is a future goal that is closely related to the
+need to improve and modularize the internal search schema/document type
+abstraction. These improvements would allow the types of collections
+being indexed to be separated from "core centillion", and core
+centillion would be packaged.
+
+Tests
+-----
+
+See (ref) for a full test plan with more detail.
+
+Summary of test plan:
+
+-   Implement tests for the four major pages/components
+    -   Login/authentication
+    -   Search
+    -   Master List
+    -   Control Panel
+-   Test authentication with two bot accounts (yammasnake and florence
+    python)
+
+-   Separate frontend and backend tests
+
+-   Add a test flag in the flask config file to change the backend
+    behavior of the server
+
+Code Reviews
+------------
+
+CI tests will be implemented for all pull requests.
+
+Pull requests to the **stable branch** have the following checks in
+place:
+
+-   PRs to the stable branch require at least 1 PR review
+-   PRs to the stable branch must pass CI tests
+
+Pull requests to the **development branch** have the following checks in
+place:
+
+-   PRs to the development branch must pass CI tests
+
+Formal Release Process
+----------------------
+
+In order to ensure a stable, consistent product, we utilize the
+branching pattern described above to implement new features in the
+development branch and test them out on
+<https://betasearch.nihdatacommons.us>.
+
+Once features and bug fixes have been tested and reviewed internally,
+they are ready to be deployed. A new pre-release branch is created from
+the development branch. The pre-release branch has a feature freeze in
+place. Changes are made to the pre-release branch to prepare it for the
+next major version release.
+
+When the pre-release branch is finished, it is merged into the stable
+branch. The head commit of the stable version is tagged with the lastest
+release number.
+
+Finally, the new version is deployed on
+<https://search.nihdatacommons.us>.
+
+Continual Process Improvement
+-----------------------------
+
+We will utilize the centillion wiki on Github to keep track of repeated
+processes and opportunities for improvement. Feedback and ideas for
+process improvement can also be submitted via Github issues.

static/feedback.js

@@ -0,0 +1,133 @@
+// submitting form with modal:
+// https://stackoverflow.com/a/29068742
+//
+// closing a bootstrap modal with submit button:
+// https://stackoverflow.com/a/33478107
+//
+// flask post data as json:
+// https://stackoverflow.com/a/16664376
+
+/* this function is called when the user submits
+ * the feedback form. it submits a post request
+ * to the flask server, which squirrels away the
+ * feedback in a file.
+ */
+function submit_feedback() {
+    // this function is called when submit button clicked
+    // algorithm:
+    // - check if text box has content
+    // - check if happy/sad filled out
+
+    var smile_active = $('#modal-feedback-smile-div').hasClass('smile-active');
+    var frown_active = $('#modal-feedback-frown-div').hasClass('frown-active');
+    if( !( smile_active || frown_active ) ) {
+        alert('Please pick the smile or the frown.')
+    } else if( $('#modal-feedback-textarea').val()=='' ) {
+        alert('Please provide us with some feedback.')
+    } else {
+        var user_sentiment = '';
+        if(smile_active) {
+            user_sentiment = 'smile';
+        } else {
+            user_sentiment = 'frown';
+        }
+        var escaped_text = $('#modal-feedback-textarea').val();
+
+        // prepare form data 
+        var data = {
+            sentiment : user_sentiment,
+            content : escaped_text
+        };
+        // post the form. the callback function resets the form
+        $.post("/feedback", 
+            data, 
+            function(response) {
+                $('#myModal').modal('hide');
+                $('#myModalForm')[0].reset();
+                add_alert(response);
+                frown_unclick();
+                smile_unclick();
+        });
+    }
+}
+
+
+function add_alert(response) {
+    str = ""
+    str += '<div id="feedback-messages-container" class="container">';
+
+    if (response['status']=='ok') {
+        // if status is ok, use alert-success
+        str += '    <div id="feedback-messages-alert" class="alert alert-success alert-dismissible fade in">';
+    } else {
+        // otherwise use alert-danger
+        str += '    <div id="feedback-messages-alert" class="alert alert-danger alert-dismissible fade in">';
+    }
+
+    str += '        <a href="#" class="close" data-dismiss="alert" aria-label="close">&times;</a>';
+    str += '        <div id="feedback-messages-contianer" class="container-fluid">';
+    str += '            <div id="feedback-messages-div" class="co-xs-12">';
+    str += '                <p>'
+    str += response['message'];
+    str += '                </p>';
+    str += '            </div>';
+    str += '    </div>';
+    str += '</div>';
+    $('div#messages').append(str);
+}
+
+
+/* for those particularly wordy users... limit feedback to 1000 chars */
+function cool_it() { 
+    if($('#modal-feedback-textarea').val().length > 1100 ){
+        $('#modal-too-long').show();
+    } else {
+        $('#modal-too-long').hide();
+    }
+}
+
+/* smiley functions */
+function smile_click() {
+    $('#modal-feedback-smile-div').addClass('smile-active');
+    $('#modal-feedback-smile-icon').addClass('smile-active');
+}
+function frown_click() {
+    $('#modal-feedback-frown-div').addClass('frown-active');
+    $('#modal-feedback-frown-icon').addClass('frown-active');
+}
+function smile_unclick() {
+    $('#modal-feedback-smile-div').removeClass('smile-active');
+    $('#modal-feedback-smile-icon').removeClass('smile-active');
+}
+function frown_unclick() {
+    $('#modal-feedback-frown-div').removeClass('frown-active');
+    $('#modal-feedback-frown-icon').removeClass('frown-active');
+}
+
+function smile() {
+    frown_unclick();
+    smile_click();
+}
+function frown() { 
+    smile_unclick();
+    frown_click();
+}
+
+
+/* for those particularly wordy users... limit feedback to 1100 chars */
+// how to check n characters in a textarea
+// https://stackoverflow.com/a/19934613
+/*
+$(document).ready(function() {
+
+    $('#modal-feedback-textarea').on('change',function(event) {
+        if($('#modal-feedback-textarea').val().length > 1100 ){
+            $('#modal-too-long').show();
+        } else {
+            $('#modal-too-long').hide();
+        }
+    });
+
+}
+*/
+

static/jquery.dataTables.min.js

@@ -141,7 +141,7 @@ sSearch:"",bRegex:!1,bSmart:!0};m.models.oRow={nTr:null,anCells:null,_aData:[],_
 sTitle:null,sType:null,sWidth:null,sWidthOrig:null};m.defaults={aaData:null,aaSorting:[[0,"asc"]],aaSortingFixed:[],ajax:null,aLengthMenu:[10,25,50,100],aoColumns:null,aoColumnDefs:null,aoSearchCols:[],asStripeClasses:null,bAutoWidth:!0,bDeferRender:!1,bDestroy:!1,bFilter:!0,bInfo:!0,bJQueryUI:!1,bLengthChange:!0,bPaginate:!0,bProcessing:!1,bRetrieve:!1,bScrollCollapse:!1,bServerSide:!1,bSort:!0,bSortMulti:!0,bSortCellsTop:!1,bSortClasses:!0,bStateSave:!1,fnCreatedRow:null,fnDrawCallback:null,fnFooterCallback:null,
 fnFormatNumber:function(a){return a.toString().replace(/\B(?=(\d{3})+(?!\d))/g,this.oLanguage.sThousands)},fnHeaderCallback:null,fnInfoCallback:null,fnInitComplete:null,fnPreDrawCallback:null,fnRowCallback:null,fnServerData:null,fnServerParams:null,fnStateLoadCallback:function(a){try{return JSON.parse((-1===a.iStateDuration?sessionStorage:localStorage).getItem("DataTables_"+a.sInstance+"_"+location.pathname))}catch(b){}},fnStateLoadParams:null,fnStateLoaded:null,fnStateSaveCallback:function(a,b){try{(-1===
 a.iStateDuration?sessionStorage:localStorage).setItem("DataTables_"+a.sInstance+"_"+location.pathname,JSON.stringify(b))}catch(c){}},fnStateSaveParams:null,iStateDuration:7200,iDeferLoading:null,iDisplayLength:10,iDisplayStart:0,iTabIndex:0,oClasses:{},oLanguage:{oAria:{sSortAscending:": activate to sort column ascending",sSortDescending:": activate to sort column descending"},oPaginate:{sFirst:"First",sLast:"Last",sNext:"Next",sPrevious:"Previous"},sEmptyTable:"No data available in table",sInfo:"Showing _START_ to _END_ of _TOTAL_ entries",
-sInfoEmpty:"Showing 0 to 0 of 0 entries",sInfoFiltered:"(filtered from _MAX_ total entries)",sInfoPostFix:"",sDecimal:"",sThousands:",",sLengthMenu:"Show _MENU_ entries",sLoadingRecords:"Loading...",sProcessing:"Processing...",sSearch:"Search:",sSearchPlaceholder:"",sUrl:"",sZeroRecords:"No matching records found"},oSearch:h.extend({},m.models.oSearch),sAjaxDataProp:"data",sAjaxSource:null,sDom:"lfrtip",searchDelay:null,sPaginationType:"simple_numbers",sScrollX:"",sScrollXInner:"",sScrollY:"",sServerMethod:"GET",
+sInfoEmpty:"Showing 0 to 0 of 0 entries",sInfoFiltered:"(filtered from _MAX_ total entries)",sInfoPostFix:"",sDecimal:"",sThousands:",",sLengthMenu:"Show _MENU_ entries",sLoadingRecords:"Loading...",sProcessing:"Processing...",sSearch:"Search Metadata:",sSearchPlaceholder:"",sUrl:"",sZeroRecords:"No matching records found"},oSearch:h.extend({},m.models.oSearch),sAjaxDataProp:"data",sAjaxSource:null,sDom:"lfrtip",searchDelay:null,sPaginationType:"simple_numbers",sScrollX:"",sScrollXInner:"",sScrollY:"",sServerMethod:"GET",
 renderer:null,rowId:"DT_RowId"};X(m.defaults);m.defaults.column={aDataSort:null,iDataSort:-1,asSorting:["asc","desc"],bSearchable:!0,bSortable:!0,bVisible:!0,fnCreatedCell:null,mData:null,mRender:null,sCellType:"td",sClass:"",sContentPadding:"",sDefaultContent:null,sName:"",sSortDataType:"std",sTitle:null,sType:null,sWidth:null};X(m.defaults.column);m.models.oSettings={oFeatures:{bAutoWidth:null,bDeferRender:null,bFilter:null,bInfo:null,bLengthChange:null,bPaginate:null,bProcessing:null,bServerSide:null,
 bSort:null,bSortMulti:null,bSortClasses:null,bStateSave:null},oScroll:{bCollapse:null,iBarWidth:0,sX:null,sXInner:null,sY:null},oLanguage:{fnInfoCallback:null},oBrowser:{bScrollOversize:!1,bScrollbarLeft:!1,bBounding:!1,barWidth:0},ajax:null,aanFeatures:[],aoData:[],aiDisplay:[],aiDisplayMaster:[],aIds:{},aoColumns:[],aoHeader:[],aoFooter:[],oPreviousSearch:{},aoPreSearchCols:[],aaSorting:null,aaSortingFixed:[],asStripeClasses:null,asDestroyStripes:[],sDestroyWidth:0,aoRowCallback:[],aoHeaderCallback:[],
 aoFooterCallback:[],aoDrawCallback:[],aoRowCreatedCallback:[],aoPreDrawCallback:[],aoInitComplete:[],aoStateSaveParams:[],aoStateLoadParams:[],aoStateLoaded:[],sTableId:"",nTable:null,nTHead:null,nTFoot:null,nTBody:null,nTableWrapper:null,bDeferLoading:!1,bInitialised:!1,aoOpenRows:[],sDom:null,searchDelay:null,sPaginationType:"two_button",iStateDuration:0,aoStateSave:[],aoStateLoad:[],oSavedState:null,oLoadedState:null,sAjaxSource:null,sAjaxDataProp:null,bAjaxDataGet:!0,jqXHR:null,json:k,oAjaxData:k,

static/master_list.js

@@ -123,15 +123,6 @@ function load_gdoc_table(){
                     lengthMenu: [50,100,250,500]
                 });
 
-                // Get the search filter section and search box
-                var searchsec = $(filtlabel).find('label');
-                var searchbox = searchsec.find('input');
-
-                // Replace search filter section text,
-                // then re-add the removed search box
-                searchsec.text('Search Metadata: ');
-                searchsec.append(searchbox);
-
                 initGdocTable = true
             });
             console.log('Finished loading Google Drive master list');
@@ -154,7 +145,7 @@ function load_issue_table(){
                 var r = new Array(), j = -1, size=result.length;
                 r[++j] = '<thead>'
                 r[++j] = '<tr class="header-row">';
-                r[++j] = '<th>Issue Name</th>';
+                r[++j] = '<th width="50%">Issue Name</th>';
                 r[++j] = '<th width="15%">Repository</th>';
                 r[++j] = '<th width="15%">Created</th>';
                 r[++j] = '<th width="15%">Modified</th>';
@@ -190,15 +181,6 @@ function load_issue_table(){
                     lengthMenu: [50,100,250,500]
                 });
 
-                // Get the search filter section and search box
-                var searchsec = $(filtlabel).find('label');
-                var searchbox = searchsec.find('input');
-
-                // Replace search filter section text,
-                // then re-add the removed search box
-                searchsec.text('Search Metadata: ');
-                searchsec.append(searchbox);
-
                 initIssuesTable = true;
             });
             console.log('Finished loading Github issues master list');
@@ -253,15 +235,6 @@ function load_ghfile_table(){
                     lengthMenu: [50,100,250,500]
                 });
 
-                // Get the search filter section and search box
-                var searchsec = $(filtlabel).find('label');
-                var searchbox = searchsec.find('input');
-
-                // Replace search filter section text,
-                // then re-add the removed search box
-                searchsec.text('Search Metadata: ');
-                searchsec.append(searchbox);
-
                 initGhfilesTable = true;
             });
             console.log('Finished loading Github file list');
@@ -314,15 +287,6 @@ function load_markdown_table(){
                     lengthMenu: [50,100,250,500]
                 });
 
-                // Get the search filter section and search box
-                var searchsec = $(filtlabel).find('label');
-                var searchbox = searchsec.find('input');
-
-                // Replace search filter section text,
-                // then re-add the removed search box
-                searchsec.text('Search Metadata: ');
-                searchsec.append(searchbox);
-
                 initMarkdownTable = true;
             });
             console.log('Finished loading Markdown list');
@@ -374,15 +338,6 @@ function load_emailthreads_table(){
                     lengthMenu: [50,100,250,500]
                 });
 
-                // Get the search filter section and search box
-                var searchsec = $(filtlabel).find('label');
-                var searchbox = searchsec.find('input');
-
-                // Replace search filter section text,
-                // then re-add the removed search box
-                searchsec.text('Search Metadata: ');
-                searchsec.append(searchbox);
-
                 initEmailthreadsTable = true;
             });
             console.log('Finished loading Groups.io email threads list');

static/style.css

@@ -1,3 +1,61 @@
+#modal-too-long {
+    visibility: hidden;
+}
+
+/* feedback smileys */
+#modal-feedback-smile-icon,
+#modal-feedback-frown-icon {
+    padding-left: 100px;
+    padding-right: 100px;
+    padding-top: 20px;
+    padding-bottom: 20px;
+}
+
+div.smile-active {
+    background-color: #2b2;
+}
+i.smile-active {
+    color: #fff;
+}
+div.frown-active {
+    background-color: #b22;
+}
+i.frown-active {
+    color: #fff;
+}
+
+/* feedback text area */
+#modal-feedback-textarea {
+    width: 100%;
+}
+
+/* feedback buttons */
+button.close {
+    font-size: 35px;
+}
+button#submit-feedback-btn {
+    width: 250px;
+}
+
+button#feedback:hover {
+    opacity: 1.0;
+    filter: alpha(opacity=100); /* For IE8 and earlier */
+}
+
+button#feedback {
+    opacity: 0.5;
+    filter: alpha(opacity=50); /* For IE8 and earlier */
+
+    width: 180px;
+    height: 50px;
+    position: fixed; 
+    z-index: 999;
+    right: 120px;
+    bottom: 10px;
+}
+
+
+/* search results table */
 td#search-results-score-col,
 td#search-results-type-col {
     width: 100px;

templates/403.html

@@ -1,4 +1,5 @@
 {% extends "layout.html" %}
+{% set active_page = "403" %}
 {% block body %}
 
 <div class="container">

templates/404.html

@@ -1,4 +1,5 @@
 {% extends "layout.html" %}
+{% set active_page = "404" %}
 {% block body %}
 
 <div class="container">

templates/banner.html

@@ -0,0 +1,25 @@
+<div class="container" id="banner-container">
+
+    {#
+    banner image
+    #}
+    <div class="row" id="banner-row">
+        <div class="col12sm" id="banner-col">
+            <center>
+                <a id="banner-a" href="{{ url_for('search')}}?query=&fields=">
+                    <img id="banner-img" src="{{ url_for('static', filename='centillion_white.png') }}">
+                </a>
+            </center>
+        </div>
+    </div>
+    {% if config['TAGLINE'] %}
+    <div class="row" id="tagline-row">
+        <div class="col12sm" id="tagline-col">
+            <center>
+                    <h2 id="tagline-tagline"> {{config['TAGLINE']}} </h2>
+            </center>
+        </div>
+    </div>
+    {% endif %}
+
+</div>

templates/controlpanel.html

@@ -1,4 +1,5 @@
 {% extends "layout.html" %}
+{% set active_page = "control_panel" %}
 {% block body %}
 
 <hr />

templates/flashed_messages.html

@@ -0,0 +1,14 @@
+<div id="messages">
+{% with messages = get_flashed_messages() %}
+    {% if messages %}
+        <div class="container" id="flashed-messages-container">
+            <div class="alert alert-success alert-dismissible fade in">
+                <a href="#" class="close" data-dismiss="alert" aria-label="close">&times;</a>
+                    {% for message in messages %}
+                        <p class="lead">{{ message }}</p>
+                    {% endfor %}
+            </div>
+        </div>
+    {% endif %}
+{% endwith %}
+</div>

templates/landing.html

@@ -1,4 +1,5 @@
 {% extends "layout.html" %}
+{% set active_page = "landing" %}
 {% block body %}
 <div class="container">
     <div class="row">

templates/layout.html

@@ -10,6 +10,7 @@
 <script src="{{ url_for('static', filename='bootstrap.min.js') }}"></script>
 <script src="{{ url_for('static', filename='master_list.js') }}"></script>
 <script src="{{ url_for('static', filename='search_list.js') }}"></script>
+<script src="{{ url_for('static', filename='feedback.js') }}"></script>
 
 
 {# ########## dataTables plugin ############ #}
@@ -23,52 +24,43 @@
 
 
 
-{# ########## github fork corner ############ #}
+<div id="master-div">
 
-<div>
+    {#
+    flashed messages
+    #}
+    {% include "flashed_messages.html" %}
 
-    {% with messages = get_flashed_messages() %}
-    {% if messages %}
-    <div class="container">
-        <div class="alert alert-success alert-dismissible">
-            <a href="#" class="close" data-dismiss="alert" aria-label="close">&times;</a>
-            <ul class=flashes>
-                {% for message in messages %}
-                <li>{{ message }}</li>
-                {% endfor %}
-            </ul>
-        </div>
-    </div>
-    {% endif %}
-    {% endwith %}
+    {#
+    banner image
+    #}
+    {% include "banner.html" %}
 
-    <div class="container">
-    
-        {#
-        banner image
-        #}
-        <div class="row">
-            <div class="col12sm">
-                <center>
-                    <a href="{{ url_for('search')}}?query=&fields=">
-                    <img src="{{ url_for('static', filename='centillion_white.png') }}">
-                    </a>
-                    {#
-                    need a tag line
-                    #}
-                    {% if config['TAGLINE'] %}
-                        <h2><a href="{{ url_for('search')}}?query=&fields=">
-                            {{config['TAGLINE']}}
-                        </a></h2>
-                    {% endif %}
-                </center>
-            </div>
-        </div>
-    
-    </div>
+    {#
+    feedback modal
+    #}
+    {% include "modal.html" %}
 
     {% block body %}{% endblock %}
+
 </div>
 
-<a href="https://github.com/dcppc/centillion" class="github-corner" aria-label="View source on Github"><svg width="80" height="80" viewBox="0 0 250 250" style="fill:#151513; color:#fff; position: absolute; top: 0; border: 0; right: 0;" aria-hidden="true"><path d="M0,0 L115,115 L130,115 L142,142 L250,250 L250,0 Z"></path><path d="M128.3,109.0 C113.8,99.7 119.0,89.6 119.0,89.6 C122.0,82.7 120.5,78.6 120.5,78.6 C119.2,72.0 123.4,76.3 123.4,76.3 C127.3,80.9 125.5,87.3 125.5,87.3 C122.9,97.6 130.6,101.9 134.4,103.2" fill="currentColor" style="transform-origin: 130px 106px;" class="octo-arm"></path><path d="M115.0,115.0 C114.9,115.1 118.7,116.5 119.8,115.4 L133.7,101.6 C136.9,99.2 139.9,98.4 142.2,98.6 C133.8,88.0 127.5,74.4 143.8,58.0 C148.5,53.4 154.0,51.2 159.7,51.0 C160.3,49.4 163.2,43.6 171.4,40.1 C171.4,40.1 176.1,42.5 178.8,56.2 C183.1,58.6 187.2,61.8 190.9,65.4 C194.5,69.0 197.7,73.2 200.1,77.6 C213.8,80.2 216.3,84.9 216.3,84.9 C212.7,93.1 206.9,96.0 205.4,96.6 C205.1,102.4 203.0,107.8 198.3,112.5 C181.9,128.9 168.3,122.5 157.7,114.1 C157.9,116.9 156.7,120.9 152.7,124.9 L141.0,136.5 C139.8,137.7 141.6,141.9 141.8,141.8 Z" fill="currentColor" class="octo-body"></path></svg></a><style>.github-corner:hover .octo-arm{animation:octocat-wave 560ms ease-in-out}@keyframes octocat-wave{0%,100%{transform:rotate(0)}20%,60%{transform:rotate(-25deg)}40%,80%{transform:rotate(10deg)}}@media (max-width:500px){.github-corner:hover .octo-arm{animation:none}.github-corner .octo-arm{animation:octocat-wave 560ms ease-in-out}}</style>
+{% if active_page=="search" or active_page=="master_list" %}
+
+    {# feedback button #}
+    <button id="feedback" type="button" 
+                          data-toggle="modal"
+                          data-target="#myModal"
+        class="btn btn-lg">Send Feedback</button>
+    
+    {# vertical spacing before the bottom, b/c of button #}
+    <div id="footer-whitespace" class="container">
+        <p>&nbsp;</p>
+        <p>&nbsp;</p>
+        <p>&nbsp;</p>
+    </div>
+
+{% endif %}
+
+<a id="github-corner" href="https://github.com/dcppc/centillion" class="github-corner" aria-label="View source on Github"><svg width="80" height="80" viewBox="0 0 250 250" style="fill:#151513; color:#fff; position: absolute; top: 0; border: 0; right: 0;" aria-hidden="true"><path d="M0,0 L115,115 L130,115 L142,142 L250,250 L250,0 Z"></path><path d="M128.3,109.0 C113.8,99.7 119.0,89.6 119.0,89.6 C122.0,82.7 120.5,78.6 120.5,78.6 C119.2,72.0 123.4,76.3 123.4,76.3 C127.3,80.9 125.5,87.3 125.5,87.3 C122.9,97.6 130.6,101.9 134.4,103.2" fill="currentColor" style="transform-origin: 130px 106px;" class="octo-arm"></path><path d="M115.0,115.0 C114.9,115.1 118.7,116.5 119.8,115.4 L133.7,101.6 C136.9,99.2 139.9,98.4 142.2,98.6 C133.8,88.0 127.5,74.4 143.8,58.0 C148.5,53.4 154.0,51.2 159.7,51.0 C160.3,49.4 163.2,43.6 171.4,40.1 C171.4,40.1 176.1,42.5 178.8,56.2 C183.1,58.6 187.2,61.8 190.9,65.4 C194.5,69.0 197.7,73.2 200.1,77.6 C213.8,80.2 216.3,84.9 216.3,84.9 C212.7,93.1 206.9,96.0 205.4,96.6 C205.1,102.4 203.0,107.8 198.3,112.5 C181.9,128.9 168.3,122.5 157.7,114.1 C157.9,116.9 156.7,120.9 152.7,124.9 L141.0,136.5 C139.8,137.7 141.6,141.9 141.8,141.8 Z" fill="currentColor" class="octo-body"></path></svg></a><style>.github-corner:hover .octo-arm{animation:octocat-wave 560ms ease-in-out}@keyframes octocat-wave{0%,100%{transform:rotate(0)}20%,60%{transform:rotate(-25deg)}40%,80%{transform:rotate(10deg)}}@media (max-width:500px){.github-corner:hover .octo-arm{animation:none}.github-corner .octo-arm{animation:octocat-wave 560ms ease-in-out}}</style>
 

templates/masterlist.html

@@ -1,4 +1,5 @@
 {% extends "layout.html" %}
+{% set active_page = "master_list" %}
 {% block body %}
 
 <hr />

templates/modal.html

@@ -0,0 +1,51 @@
+<div class="modal fade" id="myModal" tabindex="-1" role="dialog" aria-labelledby="myModalLabel">
+
+  <form id="myModalForm" method="post">
+
+    <div class="modal-dialog" role="document">
+      <div id="myModal-content" class="modal-content">
+        <div id="myModal-header" class="modal-header">
+          <button type="button" class="close" data-dismiss="modal" aria-label="Close">
+              <span aria-hidden="true">&times;</span>
+          </button>
+          <h4 class="modal-title" id="myModalLabel">
+          Send us feedback!
+          </h4>
+        </div>
+        <div id="myModal-body" class="modal-body">
+          <div id="modal-feedback-smile-frown-container" class="container-fluid">
+              <div id="modal-feedback-smile-div" class="col-xs-6 text-center" 
+                  onClick="smile()">
+                  <i id="modal-feedback-smile-icon" class="fa fa-smile-o fa-4x" aria-hidden="true"></i>
+              </div>
+              <div id="modal-feedback-frown-div" class="col-xs-6 text-center" 
+                  onClick="frown()">
+                  <i id="modal-feedback-frown-icon" class="fa fa-frown-o fa-4x" aria-hidden="true"></i>
+              </div>
+          </div>
+          <div class="container-fluid">
+              <p>&nbsp;</p>
+          </div>
+          <div id="modal-feedback-textarea-container" class="container-fluid">
+              <textarea id="modal-feedback-textarea" rows="6"></textarea>
+          </div>
+          <div id="modal-too-long" class="container-fluid" >
+              <p id="modal-too-long-text" class="lead">Please limit the length of your feedback. Thank you in advance!</p>
+          </div>
+        </div>
+        <div id="myModal-footer" class="modal-footer">
+            <div class="text-center">
+              <button id="submit-feedback-btn" type="button" 
+                  onClick="submit_feedback()"
+                  class="btn btn-lg btn-primary">
+                  Send
+              </button>
+            </div>
+        </div>
+      </div>
+    </div>
+
+  </form>
+
+</div>        
+

templates/search.html

@@ -1,7 +1,8 @@
 {% extends "layout.html" %}
+{% set active_page = "search" %}
 {% block body %}
 
-<div class="container">
+<div id="search-bar-container" class="container">
     <div class="row">
 
         <div class="col-xs-12">
@@ -12,7 +13,11 @@
                     <p><button id="the-big-one" type="submit" style="font-size: 20px; padding: 10px; padding-left: 50px; padding-right: 50px;" 
                         value="search" class="btn btn-primary">Search</button>
                     </p>
-                    <p><a href="{{ url_for('search')}}?query=&fields=">[clear all results]</a>
+
+                    {% if parsed_query %}
+                        <p><a href="{{ url_for('search')}}?query=&fields=">[clear all results]</a>
+                    {% endif %}
+
                     </p>
                 </form>
             </center>
@@ -20,17 +25,10 @@
     </div>
 </div>
 
-<div class="container">
-    <div class="row">
+<div style="height: 20px;"><p>&nbsp;</p></div>
 
-        {% if directories %}
-        <div class="col-xs-12 info directories-cloud">
-            <b>File directories:</b> 
-            {% for d in directories %}
-                <a href="{{url_for('search')}}?query={{d|trim}}&fields=filename">{{d|trim}}</a>
-            {% endfor %}
-        </div>
-        {% endif %}
+<div id="info-bars-container" class="container">
+    <div class="row">
 
         <ul class="list-group">
 
@@ -46,6 +44,9 @@
                 </li>
             {% endif %}
 
+
+            {# use "if parsed_query" to check if this is 
+               a new search or search results #}
             {% if parsed_query %}
                 <li  class="list-group-item">
                     <div class="container-fluid">
@@ -59,6 +60,7 @@
                 </li>
             {% endif %}
 
+
             <li  class="list-group-item">
                     <div class="container-fluid">
                         <div class="row">
@@ -99,7 +101,7 @@
 </div>
 
 {% if parsed_query %}
-<div class="container">
+<div id="search-results-container" class="container">
     <div class="row">
         <table id="search-results" class="table">
             <thead id="search-results-header">
@@ -126,44 +128,21 @@
                             {% if e.kind=="gdoc" %}
                                 {% if e.mimetype=="document" %}
                                     <p><small>Drive Document</small</p>
-                                    <!--
-                                    <i class="fa fa-google fa-2x"></i>
-                                    <i class="fa fa-file-text fa-2x"></i>
-                                    -->
                                 {% else %}
                                     <p><small>Drive File</small</p>
-                                    <!--
-                                    <i class="fa fa-google fa-2x"></i>
-                                    <i class="fa fa-file-o fa-2x"></i>
-                                    -->
                                 {% endif %}
 
                             {% elif e.kind=="issue" %}
                                 <p><small>Issue</small</p>
-                                <!--
-                                <i class="fa fa-github fa-2x"></i>
-                                <i class="fa fa-question fa-2x"></i>
-                                -->
 
                             {% elif e.kind=="ghfile" %}
                                 <p><small>Github File</small</p>
-                                <!--
-                                <i class="fa fa-github fa-2x"></i>
-                                <i class="fa fa-file-o fa-2x"></i>
-                                -->
 
                             {% elif e.kind=="markdown" %}
                                 <p><small>Github Markdown</small</p>
-                                <!--
-                                <i class="fa fa-github fa-2x"></i>
-                                <i class="fa fa-file-text-o fa-2x"></i>
-                                -->
 
                             {% elif e.kind=="emailthread" %}
                                 <p><small>Email Thread</small</p>
-                                <!--
-                                <i class="fa fa-envelope-o fa-2x"></i>
-                                -->
 
                             {% else %}
                                 <p><small>Unknown</small</p>

tests/Readme.md

@@ -0,0 +1,196 @@
+Centillion Tests
+================
+
+Table of Contents
+------------------
+
+*   [Centillion Tests](#centillion-tests)
+    *   [Test Plan](#test-plan)
+        *   [Local Tests](#local-tests)
+        *   [Short Tests](#short-tests)
+        *   [Long Tests](#long-tests)
+    *   [Credentials](#credentials)
+    *   [Detailed Description of Tests](#detailed-description-of-tests)
+        *   [Authentication Layer Tests](#authentication-layer-tests)
+        *   [Search Function Tests](#search-function-tests)
+        *   [Master List Endpoint Tests](#master-list-endpoint-tests)
+        *   [Control Panel Endpoint Tests](#control-panel-endpoint-tests)
+        *   [Continuous Integration Plan](#continuous-integration-plan)
+    *   [Procedure/Checklist](#procedurechecklist)
+
+
+Test Plan
+---------
+
+Related: <https://github.com/dcppc/centillion/issues/82>
+
+The test suite for centillion needs to check each of the major
+components of centillion, as well as check the authentication mechanism
+using multiple login credentials.
+
+We implement the following checks:
+
+1.  Check authentication mechanism(s) (yamasnake and florence python)
+
+2.  Check search function
+
+3.  Check master list endpoint
+
+4.  Check control panel endpoint
+
+5.  Check update search index endpoints
+
+The tests are written such that the back end and front end are tested
+separately.
+
+We need also need different tiers of tests, so we don't max out API
+calls by making lots of commits to multiple PRs.
+
+We have three tiers of tests: \* Local tests - quick tests for CI, no
+API calls \* Short tests - tests using dummy API accounts \* Long tests
+- tests using DCPPC API accounts
+
+### Local Tests
+
+Local tests can be run locally without any interaction with APIs. These
+will still utilize centillion's search schema, but will load the search
+index with fake documents rather than fetching them from an API.
+
+Uncle Archie, which runs CI tests, runs local tests only (unless you
+request it to run short test or long test.)
+
+### Short Tests
+
+Short tests utilize credentials for bot accounts that have intentionally
+been set up to have a "known" corpus of test documents. These would
+provide unit-style tests for centillion - are the mechanics of indexing
+a particular type of document from a particular API working?
+
+### Long Tests
+
+Long tests are indexing the real deal, utilizing the credentials used in
+the final production centillion. This test takes longer but is more
+likely to catch corner cases specific to the DCPPC documents.
+
+Credentials
+-----------
+
+Running tests on centillion requires multiple sets of credentials. Let's
+lay out what is needed:
+
+-   The Flask app requires a token/secret token API key pair to allow
+    users to authenticate through Github and confirm they are members of
+    the DCPPC organization. This OAuth application is owned by Charles
+    Reid (@charlesreid1).
+
+-   The search index needs a Github access token so that it can
+    interface with the Github API to index files and issues. This access
+    token is specified (along with other secrets) in the Flask
+    configuration file. The access key comes from Florence Python
+    (@fp9695253).
+
+-   The search index also requires a Google Drive API access token. This
+    must be an access token for a user who has authenticated with the
+    Centillion Google Drive OAuth application. This access token comes
+    from <mailroom@nihdatacommons.com>.
+
+-   The search index requires API credentials for any other APIs
+    associated with other document collections (Groups.io, Hypothesis,
+    Disqus).
+
+-   The backend test requires the credentials provided to Flask.
+
+-   The frontend test (Selenium) needs two Github username/passwords:
+    one for Florence Python (@fp9695253) and one for Yamma Snake
+    (@yammasnake). These are required to simulate the user
+    authenticating with Github through the browser.
+    -   The frontend test credentials are a special case.
+    -   The frontend tests expect credentials to come from environment
+        variables.
+    -   These environment variables get passed in at test time.
+    -   Tests are all run on [Uncle
+        Archie](https://github.com/dcppc/uncle-archie).
+    -   Uncle Archie already has to protect a confidential config file
+        containing Github credentials, so add additional credentials for
+        frontend tests there.
+    -   Logical separation: these credentials are not needed to
+        *operate* centillion, these credentials are needed to *test*
+        centillion
+    -   Uncle Archie already requires github credentials, already
+        protects sensitive info.
+    -   Google Drive requiring its own credentials file on disk is a
+        pain.
+
+In summary: tests use the `config_flask.py` and `config_centillion.py`
+files to provide it with the API keys it needs and to instruct it on
+what to index. The credentials and config files will control what the
+search index will actually index. The Uncle Archie CI tester config file
+contains the credentials needed to run frontend tests (check the
+login/authentication layer).
+
+Detailed Description of Tests
+-----------------------------
+
+### Authentication Layer Tests
+
+Frontend tests run as Florence Python:
+
+-   Can we log in via github and reach centillion
+-   Can we reach the control panel
+
+Frontend tests run as Yamma Snake (DCPPC member):
+
+-   Can we log in via github and reach centillion
+-   Can we reach the control panel
+
+### Search Function Tests
+
+Frontend tests:
+
+-   Can we enter something into search box and submit
+-   Can we sort the results
+-   Do the results look okay
+
+Backend tests:
+
+-   Load the search index and run a query using whoosh API
+
+### Master List Endpoint Tests
+
+Frontend tests:
+
+-   Can we get to the master list page
+-   Can we sort the results
+-   Do the results look okay
+
+Backend tests:
+
+-   Check the output of the `/list` API endpoint
+
+### Control Panel Endpoint Tests
+
+Frontend tests:
+
+-   Can we get to the control panel page
+-   Can we click the button to trigger an indexing event
+
+Backend tests:
+
+-   Trigger a re-index of the search index from the backend.
+
+### Continuous Integration Plan
+
+Tests are automatically run using Uncle Archie for continuous
+integration and deployment.
+
+Procedure/Checklist
+-------------------
+
+Pre-release procedure:
+
+-   prepare to run all test
+
+-   run short tests
+-   deploy to beta
+-   run long tests
+-   test out