Merge pull request #90 from dcppc/add-dates-and-subgroups-to-emails

Add dates and subgroups to emails

.gitignore

@@ -1,3 +1,4 @@
+feedback_database.json
 config_flask.py
 vp
 credentials.json

LICENSE

@@ -1,29 +1,133 @@
-BSD 3-Clause License
+# DEDICATED TO THE PUBLIC DOMAIN
-
+
-Copyright (c) 2018, Charles Reid
+The dcppc/organize repository has been dedicated to the public domain.
-All rights reserved.
+It is protected by the Creative Commons CC0 Universal Public Domain
-
+Dedication license. You can read the entire license below or at
-Redistribution and use in source and binary forms, with or without
+<http://creativecommons.org/publicdomain/zero/1.0/deed.en>.
-modification, are permitted provided that the following conditions are met:
+
-
+# CC0 UNIVERSAL PUBLIC DOMAIN DEDICATION LICENSE
-* Redistributions of source code must retain the above copyright notice, this
+
-  list of conditions and the following disclaimer.
+## Statement of Purpose
-
+
-* Redistributions in binary form must reproduce the above copyright notice,
+The laws of most jurisdictions throughout the world automatically confer
-  this list of conditions and the following disclaimer in the documentation
+exclusive Copyright and Related Rights (defined below) upon the creator
-  and/or other materials provided with the distribution.
+and subsequent owner(s) (each and all, an "owner") of an original work
-
+of authorship and/or a database (each, a "Work").
-* Neither the name of the copyright holder nor the names of its
+
-  contributors may be used to endorse or promote products derived from
+Certain owners wish to permanently relinquish those rights to a Work for
-  this software without specific prior written permission.
+the purpose of contributing to a commons of creative, cultural and
-
+scientific works ("Commons") that the public can reliably and without
-THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+fear of later claims of infringement build upon, modify, incorporate in
-AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+other works, reuse and redistribute as freely as possible in any form
-IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+whatsoever and for any purposes, including without limitation commercial
-DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+purposes. These owners may contribute to the Commons to promote the
-FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ideal of a free culture and the further production of creative, cultural
-DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+and scientific works, or to gain reputation or greater distribution for
-SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+their Work in part through the use and efforts of others.
-CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+
-OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+For these and/or other purposes and motivations, and without any
-OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+expectation of additional consideration or compensation, the person
+associating CC0 with a Work (the "Affirmer"), to the extent that he or
+she is an owner of Copyright and Related Rights in the Work, voluntarily
+elects to apply CC0 to the Work and publicly distribute the Work under
+its terms, with knowledge of his or her Copyright and Related Rights in
+the Work and the meaning and intended legal effect of CC0 on those
+rights.
+
+1.  **Copyright and Related Rights.** A Work made available under CC0
+    may be protected by copyright and related or neighboring rights
+    ("Copyright and Related Rights"). Copyright and Related Rights
+    include, but are not limited to, the following:
+    
+    i. the right to reproduce, adapt, distribute, perform, display,
+    communicate, and translate a Work;
+    
+    ii. moral rights retained by the original author(s) and/or
+    performer(s);
+    
+    iii. publicity and privacy rights pertaining to a person's image or
+    likeness depicted in a Work;
+    
+    iv. rights protecting against unfair competition in regards to a
+    Work, subject to the limitations in paragraph 4(a), below;
+    
+    v. rights protecting the extraction, dissemination, use and reuse of
+    data in a Work;
+    
+    vi. database rights (such as those arising under Directive 96/9/EC
+    of the European Parliament and of the Council of 11 March 1996 on
+    the legal protection of databases, and under any national
+    implementation thereof, including any amended or successor version
+    of such directive); and
+    
+    vii. other similar, equivalent or corresponding rights throughout
+    the world based on applicable law or treaty, and any national
+    implementations thereof.
+
+2.  **Waiver.** To the greatest extent permitted by, but not in
+    contravention of, applicable law, Affirmer hereby overtly, fully,
+    permanently, irrevocably and unconditionally waives, abandons, and
+    surrenders all of Affirmer's Copyright and Related Rights and
+    associated claims and causes of action, whether now known or unknown
+    (including existing as well as future claims and causes of action),
+    in the Work (i) in all territories worldwide, (ii) for the maximum
+    duration provided by applicable law or treaty (including future time
+    extensions), (iii) in any current or future medium and for any
+    number of copies, and (iv) for any purpose whatsoever, including
+    without limitation commercial, advertising or promotional purposes
+    (the "Waiver"). Affirmer makes the Waiver for the benefit of each
+    member of the public at large and to the detriment of Affirmer's
+    heirs and successors, fully intending that such Waiver shall not be
+    subject to revocation, rescission, cancellation, termination, or any
+    other legal or equitable action to disrupt the quiet enjoyment of
+    the Work by the public as contemplated by Affirmer's express
+    Statement of Purpose.
+
+3.  **Public License Fallback.** Should any part of the Waiver for any
+    reason be judged legally invalid or ineffective under applicable
+    law, then the Waiver shall be preserved to the maximum extent
+    permitted taking into account Affirmer's express Statement of
+    Purpose. In addition, to the extent the Waiver is so judged Affirmer
+    hereby grants to each affected person a royalty-free, non
+    transferable, non sublicensable, non exclusive, irrevocable and
+    unconditional license to exercise Affirmer's Copyright and Related
+    Rights in the Work (i) in all territories worldwide, (ii) for the
+    maximum duration provided by applicable law or treaty (including
+    future time extensions), (iii) in any current or future medium and
+    for any number of copies, and (iv) for any purpose whatsoever,
+    including without limitation commercial, advertising or promotional
+    purposes (the "License"). The License shall be deemed effective as
+    of the date CC0 was applied by Affirmer to the Work. Should any part
+    of the License for any reason be judged legally invalid or
+    ineffective under applicable law, such partial invalidity or
+    ineffectiveness shall not invalidate the remainder of the License,
+    and in such case Affirmer hereby affirms that he or she will not (i)
+    exercise any of his or her remaining Copyright and Related Rights in
+    the Work or (ii) assert any associated claims and causes of action
+    with respect to the Work, in either case contrary to Affirmer's
+    express Statement of Purpose.
+
+4.  **Limitations and Disclaimers.**
+    
+    a. No trademark or patent rights held by Affirmer are waived,
+    abandoned, surrendered, licensed or otherwise affected by this
+    document.
+    
+    b. Affirmer offers the Work as-is and makes no representations or
+    warranties of any kind concerning the Work, express, implied,
+    statutory or otherwise, including without limitation warranties of
+    title, merchantability, fitness for a particular purpose, non
+    infringement, or the absence of latent or other defects, accuracy,
+    or the present or absence of errors, whether or not discoverable,
+    all to the greatest extent permissible under applicable law.
+    
+    c. Affirmer disclaims responsibility for clearing rights of other
+    persons that may apply to the Work or any use thereof, including
+    without limitation any person's Copyright and Related Rights in the
+    Work. Further, Affirmer disclaims responsibility for obtaining any
+    necessary consents, permissions or other rights required for any use
+    of the Work.
+    
+    d. Affirmer understands and acknowledges that Creative Commons is
+    not a party to this document and has no duty or obligation with
+    respect to this CC0 or use of the Work.

Readme.md

@@ -6,14 +6,19 @@
 
 one centillion is 3.03 log-times better than a googol.
 
-![Screen shot of centillion](docs/images/ss.png)
+![Screen shot: centillion search](docs/images/search.png)
 
 
-## what is it
+## What Is It
 
 Centillion (https://github.com/dcppc/centillion) is a search engine that can index 
-three kinds of collections: Google Documents, Github issues, and Markdown files in 
+different kinds of document collections: Google Documents (.docx files), Google Drive files,
-Github repos.
+Github issues, Github files, Github Markdown files, and Groups.io email threads.
+
+
+
+
+## What Is It
 
 We define the types of documents the centillion should index,
 what info and how. The centillion then builds and
@@ -25,14 +30,43 @@ defined in `centillion.py`.
 
 The centillion keeps it simple.
 
-## authentication layer
+## Authentication Layer
 
 Centillion lives behind a Github authentication layer, implemented with 
 [flask-dance](https://github.com/singingwolfboy/flask-dance). When you first
 visit the site it will ask you to authenticate with Github so that it can 
 verify you have permission to access the site.
 
-## technologies
+![Screen shot: centillion authentication](docs/images/auth.png)
+
+## Master List
+
+There is a master list of all content indexed by centilion at the master list page,
+<https://search.nihdatacommons.us/master_list>.
+
+A master list for each type of document indexed by the search engine is displayed
+in a table:
+
+![Screen shot: centillion master list](docs/images/master_list.png)
+
+The metadata shown in these tables can be filtered and sorted:
+
+![Screen shot: centillion master list with sorting](docs/images/master_list2.png)
+
+## Control Panel
+
+There's also a control panel at <https://search.nihdatacommons.us/control_panel> 
+that allows you to rebuild the search index from scratch.  The search index
+stores versions/contents of files locally, so re-indexing involves going out and
+asking each API for new versions of a file/document/web page. When you re-index
+the main search index, it will ask every API for new versions of every document.
+You can also update only specific types of documents in the search index.
+
+![Screen shot: centillion control panel](docs/images/control_panel.png)
+
+
+
+## Technologies
 
 Centillion is a Python program built using whoosh (search engine library). It
 indexes the full text of docx files in Google Documents, just the filenames for
@@ -41,16 +75,61 @@ results are grouped by issue. Centillion requires Google Drive and Github OAuth
 apps. Once you provide credentials to Flask you're all set to go. 
 
 
-## control panel
+## Configuration
 
-There's also a control panel at <https://search.nihdatacommons.us/control_panel> 
+You will need to configure both the centillion search index and the flask app.
-that allows you to rebuild the search index from scratch (the Google Drive indexing 
+
-takes a while).
+The centillion search index is configured with `config_centillion.py`; this file
+sets the names of repositories to crawl when indxing issues and files.
+
+The flask app is configured with `config_flask.py`. This file contains sensitive
+information and is in the `.gitignore` file. This file contains API credentials 
+for Github and Groups.io.
+
+Exampls are provided in `config_centillion.example.py` and `config_flask.example.py`.
+
+
+## Authentication
+
+The search engine will need to connect to several APIs when it re-indexes the
+search index:
+
+* Github
+* Groups.io
+* Google Drive
+
+### Github
+
+Github API credentials (both an OAuth token for the centillion app's Github
+authentication mechanism, and a personal access token for accessing repositories
+during the re-indexing process) are provided in `config_flask.py`.
+
+### Groups.io
+
+The Groups.io API token is used to index email threads. This token is provided in
+`config_flask.py`.
+
+### Google Drive
+
+The Google Drive API credentials are provided in a file, `credentials.json`. This is
+the file that is generated when the OAuth process is complete.
+
+When you enable the Google Drive API in the Google Cloud Console, you will be provided
+with a file `client_secrets.json`. To authenticate centillion with Google Drive, you should
+download this file, and run the Google Drive utility directly:
+
+```
+python gdrive_util.py
+```
 
-![Screen shot of centillion control panel](docs/images/cp.png)
+This will initiate the authentication procedure. Sign in as a user that has access to
+the documents you want to index, and _only_ the documents you want to index (it is useful
+to set up a bot account for this purpose).
 
+Once you log in as that user, it will create `credentials.json`, and the Google Drive
+re-indexing procedure should not have any problems autheticating using that file.
 
-## quickstart (with Github auth)
+## Quickstart (With Github Auth)
 
 Start by creating a Github OAuth application.
 Get the public and private application key 
@@ -85,7 +164,7 @@ This will start a Flask server, and you can view the minimal search engine
 interface in your browser at `http://<ip>:5000`.
 
 
-## troubleshooting
+## Troubleshooting
 
 If you are having problems with your callback URL being treated
 as HTTP by Github, even though there is an HTTPS address, and

Todo.md

@@ -1,47 +1,29 @@
 # todo
 
-Main task:
+ux improvements:
-- hashing and caching
+- feedback tools
-    - <s>first, working out the logic of how we group items into sets
+- integrating master list into single list
-        - needs to be deleted
+- providing advanced search interfce
-        - needs to be updated
-        - needs to be added
-        - for docs, issues, and comments</s>
-    - second, when we add or update an item, need to:
-        - go through the motions, download file, extract text
-        - check for existing indexed doc with that id
-        - check if existing indexed doc has same hash
-            - if so, skip
-            - otherwise, delete and re-index
 
-Other bugs:
+big picture improvements:
-- Some github issues have no title (?)
+- hypothesis API
-- <s>Need to combine issues with comments</s>
+- folksonomy tagging with hypothesis
-- Not able to index markdown files _in a repo_
+- tags, expanded schema
-- (Longer term) update main index vs update diff index
 
-Needs:
-- <s>control panel</s>
 
-Thursday product:
-- Everything re-indexed nightly
-- Search engine built on all documents in Google Drive, all issues, markdown files
-- Using pandoc to extract Google Drive document contents
-- BRIEF quickstart documentation
 
-Future:
-- Future plans to improve - plugins, improving matching
-- Subdomain plans
-- Folksonomy tagging and integration plans
-
-
-
-
-config options for plugins
-conditional blocks with import github inside
-complicated tho - better to have components split off
 
+feedback form: where we are at
+- feedback button
+- button triggers modal form
+- modal has emojis for feedback, text box, buttons
+- clicking emojis changes color, to select
+- clicking submit with filled out form submits to an endpoint
+- clicking submit also closes form, but only if submit successful
 
+feedback form: what we need to do
+- fix alerts - thank you for your feedback doesn't show up until a refresh
+- probably an easy ajax fix
 
 
 

centillion.py

@@ -1,16 +1,18 @@
 import threading
-from subprocess import call
+import subprocess
 
 import codecs
 import os, json
+from datetime import datetime
 
 from werkzeug.contrib.fixers import ProxyFix
-from flask import Flask, request, redirect, url_for, render_template, flash
+from flask import Flask, request, redirect, url_for, render_template, flash, jsonify
 from flask_dance.contrib.github import make_github_blueprint, github
 
 # create our application
 from centillion_search import Search
 
+import config_centillion
 
 """
 The Centillion
@@ -27,8 +29,9 @@ You provide:
 
 
 class UpdateIndexTask(object):
-    def __init__(self, app_config, diff_index=False):
+    def __init__(self, app_config, diff_index=False,run_which='all'):
         self.diff_index = diff_index
+        self.run_which = run_which
         thread = threading.Thread(target=self.run, args=())
 
         self.gh_token = app_config['GITHUB_TOKEN']
@@ -46,14 +49,13 @@ class UpdateIndexTask(object):
         if(self.diff_index):
             raise Exception("diff index not implemented")
 
-        from get_centillion_config import get_centillion_config
+        #from get_centillion_config import get_centillion_config
-        config = get_centillion_config('config_centillion.json')
+        config = config_centillion.config
-
-        search.update_index_groupsioemails(self.groupsio_credentials,config)
-        ###search.update_index_ghfiles(self.gh_token,config)
-        ###search.update_index_issues(self.gh_token,config)
-        ###search.update_index_gdocs(config)
 
+        search.update_index(self.groupsio_credentials,
+                            self.gh_token,
+                            self.run_which,
+                            config)
 
 
 app = Flask(__name__)
@@ -70,159 +72,280 @@ github_bp = make_github_blueprint(
 
 app.register_blueprint(github_bp, url_prefix="/login")
 
-contents404 = "<html><body><h1>Status: Error 404 Page Not Found</h1></body></html>"
+last_searches_file = app.config["INDEX_DIR"] + "/last_searches.txt"
-contents403 = "<html><body><h1>Status: Error 403 Access Denied</h1></body></html>"
-contents200 = "<html><body><h1>Status: OK 200</h1></body></html>"
-
 
 ##############################
 # Flask routes
 
 @app.route('/')
 def index():
+    if not github.authorized:
+        return render_template("landing.html")
+    else:
+        username = github.get("/user").json()['login']
+        resp = github.get("/user/orgs")
+        if resp.ok:
 
+            # If they are in dcppc, redirect to search.
+            # Otherwise, hit em with a 403
+            all_orgs = resp.json()
+            for org in all_orgs:
+                if org['login']=='dcppc':
+                    # Business as usual
+                    return redirect(url_for("search", query="", fields=""))
+
+            # Not in dcppc
+            return render_template('403.html')
+
+        # Could not reach Github
+        return render_template('404.html')
+
+
+@app.route('/log_in')
+def log_in():
     if not github.authorized:
         return redirect(url_for("github.login"))
-
     else:
-
         username = github.get("/user").json()['login']
-
         resp = github.get("/user/orgs")
         if resp.ok:
 
-            # If they are in team copper, redirect to search.
+            # If they are in dcppc, redirect to search.
             # Otherwise, hit em with a 403
             all_orgs = resp.json()
             for org in all_orgs:
                 if org['login']=='dcppc':
-                    copper_team_id = '2700235'
+                    # Business as usual
-                    mresp = github.get('/teams/%s/members/%s'%(copper_team_id,username))
+                    return redirect(url_for("search", query="", fields=""))
-                    if mresp.status_code==204:
 
-                        # --------------------
+            # Not in dcppc
-                        # Business as usual
+            return render_template('403.html')
-                        return redirect(url_for("search", query="", fields=""))
 
-            return contents403
+        # Could not reach Github
+        return render_template('404.html')
 
-        return contents404
 
-### @app.route('/')
-### def index():
-###     return redirect(url_for("search", query="", fields=""))
 
 @app.route('/search')
 def search():
-
     if not github.authorized:
         return redirect(url_for("github.login"))
-
     username = github.get("/user").json()['login']
-
     resp = github.get("/user/orgs")
     if resp.ok:
 
+        # If they are in dcppc, show them search.html
+        # Otherwise, hit em with a 403
         all_orgs = resp.json()
         for org in all_orgs:
             if org['login']=='dcppc':
+                # Business as usual
+                query = request.args['query']
+                fields = request.args.get('fields')
+                if fields == 'None':
+                    fields = None
+
+                search = Search(app.config["INDEX_DIR"])
+                if not query:
+                    parsed_query = ""
+                    result = []
+
+                else:
+                    parsed_query, result = search.search(query.split(), fields=[fields])
+                    store_search(query,fields)
+
+                totals = search.get_document_total_count()
+
+                return render_template('search.html', 
+                                       entries=result, 
+                                       query=query, 
+                                       parsed_query=parsed_query, 
+                                       fields=fields, 
+                                       totals=totals)
+
+        # Not in dcppc 
+        return render_template('403.html')
+
+    # Could not reach Github
+    return render_template('404.html')
+
+
+@app.route('/update_index/<run_which>')
+def update_index(run_which):
+    """
+    TEAM COPPER ONLY!!!
+    """
+    if not github.authorized:
+        return redirect(url_for("github.login"))
+    username = github.get("/user").json()['login']
+    resp = github.get("/user/orgs")
+    if resp.ok:
 
+        # Only Team Copper members can update the index
+        all_orgs = resp.json()
+        for org in all_orgs:
+            if org['login']=='dcppc':
                 copper_team_id = '2700235'
-
                 mresp = github.get('/teams/%s/members/%s'%(copper_team_id,username))
                 if mresp.status_code==204:
-
-                    # --------------------
                     # Business as usual
-                    query = request.args['query']
+                    UpdateIndexTask(app.config,
-                    fields = request.args.get('fields')
+                                    diff_index=False,
-                    if fields == 'None':
+                                    run_which = run_which)
-                        fields = None
+                    flash("Rebuilding index, check console output")
+                    # This redirects user to /control_panel route
+                    # to prevent accidental re-indexing
+                    return redirect(url_for("control_panel"))
 
-                    search = Search(app.config["INDEX_DIR"])
+    return render_template('403.html')
-                    if not query:
-                        parsed_query = ""
-                        result = []
 
-                    else:
-                        parsed_query, result = search.search(query.split(), fields=[fields])
 
-                    totals = search.get_document_total_count()
+@app.route('/control_panel')
+def control_panel():
+    """
+    TEAM COPPER ONLY!!!
+    """
+    if not github.authorized:
+        return redirect(url_for("github.login"))
+    username = github.get("/user").json()['login']
+    resp = github.get("/user/orgs")
+    if resp.ok:
 
-                    return render_template('search.html', 
+        # Only Team Copper members can access the control panel
-                                           entries=result, 
+        all_orgs = resp.json()
-                                           query=query, 
+        for org in all_orgs:
-                                           parsed_query=parsed_query, 
+            if org['login']=='dcppc':
-                                           fields=fields, 
+                copper_team_id = '2700235'
-                                           totals=totals)
+                mresp = github.get('/teams/%s/members/%s'%(copper_team_id,username))
+                if mresp.status_code==204:
+                    # Business as usual
+                    return render_template("controlpanel.html")
+
+        # Not in dcppc 
+        return render_template('403.html')
 
-    return contents403
+    # Could not reach Github
+    return render_template('404.html')
 
 
-@app.route('/update_index')
-def update_index():
 
+@app.route('/master_list')
+def master_list():
     if not github.authorized:
         return redirect(url_for("github.login"))
-
     username = github.get("/user").json()['login']
-
     resp = github.get("/user/orgs")
     if resp.ok:
 
+        # If they are in dcppc, show them masterlist.html
+        # Otherwise, hit em with a 403
         all_orgs = resp.json()
         for org in all_orgs:
             if org['login']=='dcppc':
+                # Business as usual
+                return render_template("masterlist.html")
 
-                copper_team_id = '2700235'
+        # Not in dcppc 
+        return render_template('403.html')
 
-                mresp = github.get('/teams/%s/members/%s'%(copper_team_id,username))
+    # Could not reach Github
-                if mresp.status_code==204:
+    return render_template('404.html')
 
-                    # --------------------
-                    # Business as usual
-                    UpdateIndexTask(app.config,
-                                    diff_index=False)
-                    flash("Rebuilding index, check console output")
-                    return render_template("controlpanel.html", 
-                                           totals={})
 
-    return contents403
 
+@app.route('/list/<doctype>')
+def list_docs(doctype):
+    if not github.authorized:
+        return redirect(url_for("github.login"))
+    username = github.get("/user").json()['login']
+    resp = github.get("/user/orgs")
+    if resp.ok:
+        all_orgs = resp.json()
+        for org in all_orgs:
+            if org['login']=='dcppc':
+                # Business as usual
+                search = Search(app.config["INDEX_DIR"])
+                return jsonify(search.get_list(doctype))
+
+    # nope
+    return render_template('403.html')
 
 
-@app.route('/control_panel')
+@app.route('/feedback', methods=['POST'])
-def control_panel():
+def parse_request():
 
     if not github.authorized:
         return redirect(url_for("github.login"))
-
     username = github.get("/user").json()['login']
-
     resp = github.get("/user/orgs")
     if resp.ok:
-
         all_orgs = resp.json()
         for org in all_orgs:
             if org['login']=='dcppc':
 
-                copper_team_id = '2700235'
 
-                mresp = github.get('/teams/%s/members/%s'%(copper_team_id,username))
+                try:
-                if mresp.status_code==204:
+                    # Business as usual
+                    data = request.form.to_dict();
+                    data['github_login'] = username
+                    data['timestamp'] = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
+
+                    feedback_database = 'feedback_database.json'
+                    if not os.path.isfile(feedback_database):
+                        with open(feedback_database,'w') as f:
+                            json_data = [data]
+                            json.dump(json_data, f, indent=4)
 
-                    return render_template("controlpanel.html", 
+                    else:
-                                           totals={})
+                        json_data = []
+                        with open(feedback_database,'r') as f:
+                            json_data = json.load(f)
+
+                        json_data.append(data)
+
+                        with open(feedback_database,'w') as f:
+                            json.dump(json_data, f, indent=4)
 
-    return contents403
+                    ## Should be done with Javascript
+                    #flash("Thank you for your feedback!")
+                    return jsonify({'status':'ok','message':'Thank you for your feedback!'})
+                except:
+                    return jsonify({'status':'error','message':'An error was encountered while submitting your feedback. Try submitting an issue in the <a href="https://github.com/dcppc/centillion/issues/new">dcppc/centillion</a> repository.'})
 
 
+    # nope
+    return render_template('403.html')
+
 @app.errorhandler(404)
 def oops(e):
-    return contents404
+    return render_template('404.html')
+
+
+def store_search(query, fields):
+    """
+    Store searches in a text file
+    """
+    if os.path.exists(last_searches_file):
+        with codecs.open(last_searches_file, 'r', encoding='utf-8') as f:
+            contents = f.readlines()
+    else:
+        contents = []
+
+    search = "query=%s&fields=%s\n" % (query, fields)
+    if not search in contents:
+        contents.insert(0, search)
+
+    with codecs.open(last_searches_file, 'w', encoding='utf-8') as f:
+        f.writelines(contents)
+
 
 if __name__ == '__main__':
     # if running local instance, set to true
     os.environ['OAUTHLIB_INSECURE_TRANSPORT'] = 'true'
-    app.run(host="0.0.0.0",port=5000)
+    port = os.environ.get('CENTILLION_PORT','')
+    if port=='':
+        port = 5000
+    else:
+        port = int(port)
+    app.run(host="0.0.0.0",port=port)
 

centillion_search.py

@@ -5,7 +5,7 @@ from github import Github, GithubException
 import base64
 
 from gdrive_util import GDrive
-from groupsio_util import GroupsIOArchivesCrawler
+from groupsio_util import GroupsIOArchivesCrawler, GroupsIOException
 from apiclient.http import MediaIoBaseDownload
 
 import mistune
@@ -17,9 +17,12 @@ import pypandoc
 import os.path
 import codecs
 from datetime import datetime
+import dateutil.parser
 
 from whoosh.qparser import MultifieldParser, QueryParser
 from whoosh.analysis import StemmingAnalyzer
+from whoosh.qparser.dateparse import DateParserPlugin
+from whoosh import fields, index
 
 
 """
@@ -99,6 +102,58 @@ class Search:
         self.open_index(index_folder)
 
 
+    # ------------------------------
+    # Update the entire index
+
+    def update_index(self, groupsio_credentials, gh_token, run_which, config):
+        """
+        Update the entire search index
+        """
+        if run_which=='all' or run_which=='emailthreads':
+            try:
+                self.update_index_emailthreads(groupsio_credentials, config)
+            except GroupsIOException as e:
+                print("ERROR: While re-indexing: failed to update Groups.io email threads, hit API rate limit")
+                print("-"*40)
+                print(repr(e))
+                print("-"*40)
+                print("Continuing...")
+                pass
+
+        if run_which=='all' or run_which=='ghfiles':
+            try:
+                self.update_index_ghfiles(gh_token,config)
+            except Exception as e:
+                print("ERROR: While re-indexing: failed to update Github files")
+                print("-"*40)
+                print(repr(e))
+                print("-"*40)
+                print("Continuing...")
+                pass
+
+        if run_which=='all' or run_which=='issues':
+            try:
+                self.update_index_issues(gh_token,config)
+            except Exception as e:
+                print("ERROR: While re-indexing: failed to update Github issues")
+                print("-"*40)
+                print(repr(e))
+                print("-"*40)
+                print("Continuing...")
+                pass
+
+        if run_which=='all' or run_which=='gdocs':
+            try:
+                self.update_index_gdocs(config)
+            except Exception as e:
+                print("ERROR: While re-indexing: failed to update Google Drive files")
+                print("-"*40)
+                print(repr(e))
+                print("-"*40)
+                print("Continuing...")
+                pass
+
+
     # ------------------------------
     # Create a schema and open a search index
     # on disk.
@@ -127,30 +182,38 @@ class Search:
         # is defined.
 
         schema = Schema(
-                id = ID(stored=True, unique=True),
+                id = fields.ID(stored=True, unique=True),
-                kind = ID(stored=True),
+                kind = fields.ID(stored=True),
 
-                created_time = ID(stored=True),
+                created_time = fields.DATETIME(stored=True),
-                modified_time = ID(stored=True),
+                modified_time = fields.DATETIME(stored=True),
-                indexed_time = ID(stored=True),
+                indexed_time = fields.DATETIME(stored=True),
-                
-                title = TEXT(stored=True, field_boost=100.0),
-                url = ID(stored=True, unique=True),
                 
-                mimetype=ID(stored=True),
+                title = fields.TEXT(stored=True, field_boost=100.0),
-                owner_email=ID(stored=True),
+
-                owner_name=TEXT(stored=True),
+                url = fields.ID(stored=True),
                 
-                repo_name=TEXT(stored=True),
+                mimetype = fields.TEXT(stored=True),
-                repo_url=ID(stored=True),
 
-                github_user=TEXT(stored=True),
+                owner_email = fields.ID(stored=True),
+                owner_name = fields.TEXT(stored=True),
+
+                # mainly for email threads, groups.io, hypothesis
+                group = fields.ID(stored=True),
+
+                repo_name = fields.TEXT(stored=True),
+                repo_url = fields.ID(stored=True),
+                github_user = fields.TEXT(stored=True),
+
+                tags = fields.KEYWORD(commas=True,
+                                      stored=True,
+                                      lowercase=True),
 
                 # comments only
-                issue_title=TEXT(stored=True, field_boost=100.0),
+                issue_title = fields.TEXT(stored=True, field_boost=100.0),
-                issue_url=ID(stored=True),
+                issue_url = fields.ID(stored=True),
                 
-                content=TEXT(stored=True, analyzer=stemming_analyzer)
+                content = fields.TEXT(stored=True, analyzer=stemming_analyzer)
         )
 
 
@@ -190,24 +253,32 @@ class Search:
             writer.delete_by_term('id',item['id'])
 
             # Index a plain google drive file
-            writer.add_document(
+            created_time = dateutil.parser.parse(item['createdTime'])
-                    id = item['id'],
+            modified_time = dateutil.parser.parse(item['modifiedTime'])
-                    kind = 'gdoc',
+            indexed_time = datetime.now().replace(microsecond=0)
-                    created_time = item['createdTime'],
+            try:
-                    modified_time = item['modifiedTime'],
+                writer.add_document(
-                    indexed_time = datetime.now().replace(microsecond=0).isoformat(),
+                        id = item['id'],
-                    title = item['name'],
+                        kind = 'gdoc',
-                    url = item['webViewLink'],
+                        created_time = created_time,
-                    mimetype = mimetype,
+                        modified_time = modified_time,
-                    owner_email = item['owners'][0]['emailAddress'],
+                        indexed_time = indexed_time,
-                    owner_name = item['owners'][0]['displayName'],
+                        title = item['name'],
-                    repo_name='',
+                        url = item['webViewLink'],
-                    repo_url='',
+                        mimetype = mimetype,
-                    github_user='',
+                        owner_email = item['owners'][0]['emailAddress'],
-                    issue_title='',
+                        owner_name = item['owners'][0]['displayName'],
-                    issue_url='',
+                        group='',
-                    content = content
+                        repo_name='',
-            )
+                        repo_url='',
+                        github_user='',
+                        issue_title='',
+                        issue_url='',
+                        content = content
+                )
+            except ValueError as e:
+                print(repr(e))
+                print(" > XXXXXX Failed to index Google Drive file \"%s\""%(item['name']))
 
 
         else:
@@ -261,7 +332,7 @@ class Search:
                 )
                 assert output == ""
             except RuntimeError:
-                print(" > XXXXXX Failed to index document \"%s\""%(item['name']))
+                print(" > XXXXXX Failed to index Google Drive document \"%s\""%(item['name']))
 
 
             # If export was successful, read contents of markdown
@@ -289,24 +360,33 @@ class Search:
             else:
                 print(" > Creating a new record")
 
-            writer.add_document(
+            try:
-                    id = item['id'],
+                created_time = dateutil.parser.parse(item['createdTime'])
-                    kind = 'gdoc',
+                modified_time = dateutil.parser.parse(item['modifiedTime'])
-                    created_time = item['createdTime'],
+                indexed_time = datetime.now()
-                    modified_time = item['modifiedTime'],
+                writer.add_document(
-                    indexed_time = datetime.now().replace(microsecond=0).isoformat(),
+                        id = item['id'],
-                    title = item['name'],
+                        kind = 'gdoc',
-                    url = item['webViewLink'],
+                        created_time = created_time,
-                    mimetype = mimetype,
+                        modified_time = modified_time,
-                    owner_email = item['owners'][0]['emailAddress'],
+                        indexed_time = indexed_time,
-                    owner_name = item['owners'][0]['displayName'],
+                        title = item['name'],
-                    repo_name='',
+                        url = item['webViewLink'],
-                    repo_url='',
+                        mimetype = mimetype,
-                    github_user='',
+                        owner_email = item['owners'][0]['emailAddress'],
-                    issue_title='',
+                        owner_name = item['owners'][0]['displayName'],
-                    issue_url='',
+                        group='',
-                    content = content
+                        repo_name='',
-            )
+                        repo_url='',
+                        github_user='',
+                        issue_title='',
+                        issue_url='',
+                        content = content
+                )
+            except ValueError as e:
+                print(repr(e))
+                print(" > XXXXXX Failed to index Google Drive file \"%s\""%(item['name']))
+
 
 
 
@@ -340,34 +420,42 @@ class Search:
                 issue_comment_content += comment.body.rstrip()
                 issue_comment_content += "\n"
 
-        # Now create the actual search index record
+        # Now create the actual search index record.
-        created_time = clean_timestamp(issue.created_at)
-        modified_time = clean_timestamp(issue.updated_at)
-        indexed_time = clean_timestamp(datetime.now())
-
         # Add one document per issue thread,
         # containing entire text of thread.
-        writer.add_document(
+
-                id = issue.html_url,
+        created_time = issue.created_at
-                kind = 'issue',
+        modified_time = issue.updated_at
-                created_time = created_time,
+        indexed_time = datetime.now()
-                modified_time = modified_time,
+        try:
-                indexed_time = indexed_time,
+            writer.add_document(
-                title = issue.title,
+                    id = issue.html_url,
-                url = issue.html_url,
+                    kind = 'issue',
-                mimetype='',
+                    created_time = created_time,
-                owner_email='',
+                    modified_time = modified_time,
-                owner_name='',
+                    indexed_time = indexed_time,
-                repo_name = repo_name,
+                    title = issue.title,
-                repo_url = repo_url,
+                    url = issue.html_url,
-                github_user = issue.user.login,
+                    mimetype='',
-                issue_title = issue.title,
+                    owner_email='',
-                issue_url = issue.html_url,
+                    owner_name='',
-                content = issue_comment_content
+                    group='',
-        )
+                    repo_name = repo_name,
+                    repo_url = repo_url,
+                    github_user = issue.user.login,
+                    issue_title = issue.title,
+                    issue_url = issue.html_url,
+                    content = issue_comment_content
+            )
+        except ValueError as e:
+            print(repr(e))
+            print(" > XXXXXX Failed to index Github issue \"%s\""%(issue.title))
 
 
 
+    # ------------------------------
+    # Add a single github file 
+    # to a search index.
 
     def add_ghfile(self, writer, d, gh_token, config, update=True):
         """
@@ -391,7 +479,8 @@ class Search:
             print(" > XXXXXXXX Failed to find file info.")
             return
 
-        indexed_time = clean_timestamp(datetime.now())
+
+        indexed_time = datetime.now()
 
         if fext in MARKDOWN_EXTS:
             print("Indexing markdown doc %s from repo %s"%(fname,repo_name))
@@ -420,24 +509,31 @@ class Search:
             usable_url = "https://github.com/%s/blob/master/%s"%(repo_name, fpath)
 
             # Now create the actual search index record
-            writer.add_document(
+            try:
-                    id = fsha,
+                writer.add_document(
-                    kind = 'markdown',
+                        id = fsha,
-                    created_time = '',
+                        kind = 'markdown',
-                    modified_time = '',
+                        created_time = None,
-                    indexed_time = indexed_time,
+                        modified_time = None,
-                    title = fname,
+                        indexed_time = indexed_time,
-                    url = usable_url,
+                        title = fname,
-                    mimetype='',
+                        url = usable_url,
-                    owner_email='',
+                        mimetype='',
-                    owner_name='',
+                        owner_email='',
-                    repo_name = repo_name,
+                        owner_name='',
-                    repo_url = repo_url,
+                        group='',
-                    github_user = '',
+                        repo_name = repo_name,
-                    issue_title = '',
+                        repo_url = repo_url,
-                    issue_url = '',
+                        github_user = '',
-                    content = content
+                        issue_title = '',
-            )
+                        issue_url = '',
+                        content = content
+                )
+            except ValueError as e:
+                print(repr(e))
+                print(" > XXXXXX Failed to index Github markdown file \"%s\""%(fname))
+
+
 
         else:
             print("Indexing github file %s from repo %s"%(fname,repo_name))
@@ -445,24 +541,78 @@ class Search:
             key = fname+"_"+fsha
 
             # Now create the actual search index record
+            try:
+                writer.add_document(
+                        id = key,
+                        kind = 'ghfile',
+                        created_time = None,
+                        modified_time = None,
+                        indexed_time = indexed_time,
+                        title = fname,
+                        url = repo_url,
+                        mimetype='',
+                        owner_email='',
+                        owner_name='',
+                        group='',
+                        repo_name = repo_name,
+                        repo_url = repo_url,
+                        github_user = '',
+                        issue_title = '',
+                        issue_url = '',
+                        content = ''
+                )
+            except ValueError as e:
+                print(repr(e))
+                print(" > XXXXXX Failed to index Github file \"%s\""%(fname))
+
+
+
+
+    # ------------------------------
+    # Add a single github file 
+    # to a search index.
+
+    def add_emailthread(self, writer, d, config, update=True):
+        """
+        Use a Github file API record to add a filename
+        to the search index.
+        """
+        if 'created_time' in d.keys() and d['created_time'] is not None:
+            created_time = d['created_time']
+        else:
+            created_time = None
+
+        if 'modified_time' in d.keys() and d['modified_time'] is not None:
+            modified_time = d['modified_time']
+        else:
+            modified_time = None
+
+        indexed_time = datetime.now()
+
+        # Now create the actual search index record
+        try:
             writer.add_document(
-                    id = key,
+                    id = d['permalink'],
-                    kind = 'ghfile',
+                    kind = 'emailthread',
-                    created_time = '',
+                    created_time = created_time,
-                    modified_time = '',
+                    modified_time = modified_time,
                     indexed_time = indexed_time,
-                    title = fname,
+                    title = d['subject'],
-                    url = repo_url,
+                    url = d['permalink'],
                     mimetype='',
                     owner_email='',
-                    owner_name='',
+                    owner_name=d['original_sender'],
-                    repo_name = repo_name,
+                    group=d['subgroup'],
-                    repo_url = repo_url,
+                    repo_name = '',
+                    repo_url = '',
                     github_user = '',
                     issue_title = '',
                     issue_url = '',
-                    content = ''
+                    content = d['content']
             )
+        except ValueError as e:
+            print(repr(e))
+            print(" > XXXXXX Failed to index Groups.io thread \"%s\""%(d['subject']))
 
 
 
@@ -540,10 +690,10 @@ class Search:
                 full_items[f['id']] = f
             
             ## Shorter:
-            #break
+            break
-            # Longer:
+            ## Longer:
-            if nextPageToken is None:
+            #if nextPageToken is None:
-                break
+            #    break
 
 
         writer = self.ix.writer()
@@ -551,34 +701,41 @@ class Search:
         temp_dir = tempfile.mkdtemp(dir=os.getcwd())
         print("Temporary directory: %s"%(temp_dir))
 
+        try:
 
-
+            # Drop any id in indexed_ids
-        # Drop any id in indexed_ids
+            # not in remote_ids
-        # not in remote_ids
+            drop_ids = indexed_ids - remote_ids
-        drop_ids = indexed_ids - remote_ids
+            for drop_id in drop_ids:
-        for drop_id in drop_ids:
+                writer.delete_by_term('id',drop_id)
-            writer.delete_by_term('id',drop_id)
+
-
+
-
+            # Update any id in indexed_ids
-        # Update any id in indexed_ids
+            # and in remote_ids
-        # and in remote_ids
+            update_ids = indexed_ids & remote_ids
-        update_ids = indexed_ids & remote_ids
+            for update_id in update_ids:
-        for update_id in update_ids:
+                # cop out
-            # cop out
+                writer.delete_by_term('id',update_id)
-            writer.delete_by_term('id',update_id)
+                item = full_items[update_id]
-            item = full_items[update_id]
+                self.add_drive_file(writer, item, temp_dir, config, update=True)
-            self.add_drive_file(writer, item, temp_dir, config, update=True)
+                count += 1
-            count += 1
+
-
+
-
+            # Add any id not in indexed_ids
-        # Add any id not in indexed_ids
+            # and in remote_ids
-        # and in remote_ids
+            add_ids = remote_ids - indexed_ids
-        add_ids = remote_ids - indexed_ids
+            for add_id in add_ids:
-        for add_id in add_ids:
+                item = full_items[add_id]
-            item = full_items[add_id]
+                self.add_drive_file(writer, item, temp_dir, config, update=False)
-            self.add_drive_file(writer, item, temp_dir, config, update=False)
+                count += 1
-            count += 1
+
-
+        except Exception as e:
+            print("ERROR: While adding Google Drive files to search index")
+            print("-"*40)
+            print(repr(e))
+            print("-"*40)
+            print("Continuing...")
+            pass
 
         print("Cleaning temporary directory: %s"%(temp_dir))
         subprocess.call(['rm','-fr',temp_dir])
@@ -655,28 +812,13 @@ class Search:
         writer = self.ix.writer()
         count = 0
 
-        # Drop any issues in indexed_issues
+        # Drop issues in indexed_issues
-        # not in remote_issues
+        for drop_issue in indexed_issues:
-        drop_issues = indexed_issues - remote_issues
-        for drop_issue in drop_issues:
             writer.delete_by_term('id',drop_issue)
 
 
-        # Update any issue in indexed_issues
+        # Add any issue in remote_issues
-        # and in remote_issues
+        for add_issue in remote_issues:
-        update_issues = indexed_issues & remote_issues
-        for update_issue in update_issues:
-            # cop out
-            writer.delete_by_term('id',update_issue)
-            item = full_items[update_issue]
-            self.add_issue(writer, item, gh_token, config, update=True)
-            count += 1
-
-
-        # Add any issue not in indexed_issues
-        # and in remote_issues
-        add_issues = remote_issues - indexed_issues
-        for add_issue in add_issues:
             item = full_items[add_issue]
             self.add_issue(writer, item, gh_token, config, update=False)
             count += 1
@@ -706,7 +848,7 @@ class Search:
         # ------
         indexed_ids = set()
         p = QueryParser("kind", schema=self.ix.schema)
-        q = p.parse("ghfiles")
+        q = p.parse("ghfile")
         with self.ix.searcher() as s:
             results = s.search(q,limit=None)
             for result in results:
@@ -768,41 +910,35 @@ class Search:
                 fpath = d['path']
                 _, fname = os.path.split(fpath)
                 _, fext = os.path.splitext(fpath)
+                fpathpieces = fpath.split('/')
 
-                key = d['sha']
+                ignore_file = fname[0]=='.' or fname[0]=='_'
+                ignore_dir = False
+                for piece in fpathpieces:
+                    if piece[0]=='.' or piece[0]=='_':
+                        ignore_dir = True
 
-                d['org'] = this_org
+                if not ignore_file and not ignore_dir:
-                d['repo'] = this_repo
+                    key = d['sha']
-                value = d
 
-                remote_ids.add(key)
+                    d['org'] = this_org
-                full_items[key] = value
+                    d['repo'] = this_repo
+                    value = d
+
+                    remote_ids.add(key)
+                    full_items[key] = value
 
         writer = self.ix.writer()
         count = 0
 
         # Drop any id in indexed_ids
-        # not in remote_ids
+        for drop_id in indexed_ids:
-        drop_ids = indexed_ids - remote_ids
-        for drop_id in drop_ids:
             writer.delete_by_term('id',drop_id)
 
 
-        # Update any id in indexed_ids
+        # Add any issue in remote_ids
-        # and in remote_ids
-        update_ids = indexed_ids & remote_ids
-        for update_id in update_ids:
-            # cop out: just delete and re-add
-            writer.delete_by_term('id',update_id)
-            item = full_items[update_id]
-            self.add_ghfile(writer, item, gh_token, config, update=True)
-            count += 1
-
-
-        # Add any issue not in indexed_ids
         # and in remote_ids
-        add_ids = remote_ids - indexed_ids
+        for add_id in remote_ids:
-        for add_id in add_ids:
             item = full_items[add_id]
             self.add_ghfile(writer, item, gh_token, config, update=False)
             count += 1
@@ -817,27 +953,75 @@ class Search:
     # Groups.io Emails
 
 
-    def update_index_groupsioemails(self, groupsio_token, config):
+    def update_index_emailthreads(self, groupsio_token, config):
         """
         Update the search index using the email archives
-        of groups.io groups.
+        of groups.io groups. This method looks deceptively
+        simple, all the logic is hidden in the spider
+        (groupsio_util.py).
 
-        This requires the use of a spider.
         RELEASE THE SPIDER!!!
         """
+        # Algorithm:
+        # - get set of indexed ids
+        # - get set of remote ids
+        # - drop indexed ids not in remote ids
+        # - index all remote ids
+
+        # Get the set of indexed ids:
+        # ------
+        indexed_ids = set()
+        p = QueryParser("kind", schema=self.ix.schema)
+        q = p.parse("emailthread")
+        with self.ix.searcher() as s:
+            results = s.search(q,limit=None)
+            for result in results:
+                indexed_ids.add(result['id'])
+
+        # Get the set of remote ids:
+        # ------
         spider = GroupsIOArchivesCrawler(groupsio_token,'dcppc')
 
-        # - ask spider to crawl the archives
+        # ask spider to crawl the archives
         spider.crawl_group_archives()
 
-        # - ask spider for list of all email records
+        # now spider.archives is a list of dictionaries
-        #   - 1 email = 1 dictionary
+        # that each represent a thread:
-        #   - email records compiled by the spider
+        #   thread = {
+        #           'permalink' : permalink,
+        #           'subject' : subject,
+        #           'original_sender' : original_sender,
+        #           'content' : full_content
+        #   }
+        #
+        # It is hard to reliablly extract more information
+        # than that from the email thread.
+
+        writer = self.ix.writer()
+        count = 0
+
+        # archives is a dictionary
+        # keys are IDs (urls)
+        # values are dictionaries
         archives = spider.get_archives()
 
-        # - email object is sent off to add email method
+        # Start by collecting all the things
+        remote_ids = set()
+        for k in archives.keys():
+            remote_ids.add(k)
+
+        # drop indexed_ids
+        for drop_id in indexed_ids:
+            writer.delete_by_term('id',drop_id)
+
+        # add remote_ids
+        for add_id in remote_ids:
+            item = archives[add_id]
+            self.add_emailthread(writer, item, config, update=False)
+            count += 1
 
-        print("Finished indexing groups.io emails")
+        writer.commit()
+        print("Done, updated %d Groups.io email threads in the index" % count)
 
 
     # ---------------------------------
@@ -914,33 +1098,6 @@ class Search:
 
 
 
-    def search(self, query_list, fields=None):
-
-        with self.ix.searcher() as searcher:
-            query_string = " ".join(query_list)
-            query = None
-            if "\"" in query_string or ":" in query_string:
-                query = QueryParser("content", self.schema).parse(query_string)
-            elif len(fields) == 1 and fields[0] == "filename":
-                pass
-            elif len(fields) == 2:
-                pass
-            else:
-                # If the user does not specify a field,
-                # these are the fields that are actually searched
-                fields = ['title',
-                          'content']
-            if not query:
-                query = MultifieldParser(fields, schema=self.ix.schema).parse(query_string)
-            parsed_query = "%s" % query
-            print("query: %s" % parsed_query)
-            results = searcher.search(query, terms=False, scored=True, groupedby="kind")
-            search_result = self.create_search_result(results)
-
-        return parsed_query, search_result
-
-
-
     def cap(self, s, l):
         return s if len(s) <= l else s[0:l - 3] + '...'
 
@@ -952,6 +1109,7 @@ class Search:
                 "issue" : None,
                 "ghfile" : None,
                 "markdown" : None,
+                "emailthread" : None,
                 "total" : None
         }
         for key in counts.keys():
@@ -964,6 +1122,80 @@ class Search:
 
         return counts
 
+
+    def get_list(self,doctype):
+        """
+        Get a listing of all files, 
+        so we can construct the page that 
+        lists everyone and everything that
+        centillion indexes.
+        """
+        # Unfortunately, we have to treat
+        # each doctype separately, b/c of
+        # what is most relevant to display
+        # in the everything-list.
+        item_keys=''
+        if doctype=='gdoc':
+            item_keys = ['title','owner_name','url','mimetype','created_time','modified_time']
+        elif doctype=='issue':
+            item_keys = ['title','repo_name','repo_url','url','created_time','modified_time']
+        elif doctype=='emailthread':
+            item_keys = ['title','owner_name','url','created_time','modified_time']
+        elif doctype=='ghfile':
+            item_keys = ['title','repo_name','repo_url','url']
+        elif doctype=='markdown':
+            item_keys = ['title','repo_name','repo_url','url']
+        else:
+            raise Exception("Could not find document of type %s"%(doctype))
+
+        json_results = []
+
+        p = QueryParser("kind", schema=self.ix.schema)
+        q = p.parse(doctype)
+        with self.ix.searcher() as s:
+            results = s.search(q,limit=None)
+            for r in results:
+                d = {}
+                for k in item_keys:
+                    d[k] = r[k]
+                json_results.append(d)
+
+        return json_results
+
+
+
+    def search(self, query_list, fields=None):
+
+        with self.ix.searcher() as searcher:
+            query_string = " ".join(query_list)
+            query = None
+            if ":" in query_string:
+                query = QueryParser("content", self.schema)
+                query.add_plugin(DateParserPlugin(free=True))
+                query = query.parse(query_string)
+            elif len(fields) == 1 and fields[0] == "filename":
+                pass
+            elif len(fields) == 2:
+                pass
+            else:
+                # If the user does not specify a field,
+                # these are the fields that are actually searched
+                fields = ['title', 'content','owner_name','owner_email','url','created_date','modified_date']
+            if not query:
+                query = MultifieldParser(fields, schema=self.ix.schema)
+                query.add_plugin(DateParserPlugin(free=True))
+                query = query.parse(query_string)
+                #query = MultifieldParser(fields, schema=self.ix.schema).parse(query_string) 
+            parsed_query = "%s" % query
+            print("query: %s" % parsed_query)
+            results = searcher.search(query, terms=False, scored=True, groupedby="kind")
+            search_result = self.create_search_result(results)
+
+        return parsed_query, search_result
+
+
+
+
 if __name__ == "__main__":
 
     raise Exception("Error: main method not implemented (fix groupsio credentials first)")

config_centillion.example.py

@@ -1,4 +1,4 @@
-{
+config = {
     "repositories" : [
         "dcppc/project-management",
         "dcppc/nih-demo-meetings",
@@ -25,3 +25,4 @@
         "dcppc/centillion"
     ]
 }
+

config_centillion.py

@@ -0,0 +1,28 @@
+config = {
+    "repositories" : [
+        "dcppc/project-management",
+        "dcppc/nih-demo-meetings",
+        "dcppc/internal",
+        "dcppc/organize",
+        "dcppc/dcppc-bot",
+        "dcppc/full-stacks",
+        "dcppc/design-guidelines-discuss",
+        "dcppc/dcppc-deliverables",
+        "dcppc/dcppc-milestones",
+        "dcppc/crosscut-metadata",
+        "dcppc/lucky-penny",
+        "dcppc/dcppc-workshops",
+        "dcppc/metadata-matrix",
+        "dcppc/data-stewards",
+        "dcppc/dcppc-phase1-demos",
+        "dcppc/apis",
+        "dcppc/2018-june-workshop",
+        "dcppc/2018-july-workshop",
+        "dcppc/2018-august-workshop",
+        "dcppc/2018-september-workshop",
+        "dcppc/design-guidelines",
+        "dcppc/2018-may-workshop",
+        "dcppc/centillion"
+    ]
+}
+

gdrive_util.py

@@ -51,3 +51,6 @@ class GDrive(object):
         service = build('drive', 'v3', http=creds.authorize(Http()))
         return service
 
+if __name__=="__main__":
+    g = GDrive()
+    s = g.get_service()

get_centillion_config.py

@@ -1,12 +0,0 @@
-import json
-
-CONFIG_FILE = 'config_centillion.json'
-
-def get_centillion_config(filename=CONFIG_FILE):
-    """
-    Load the centillion configuration
-    """
-    with open(filename,'r') as f:
-        d = json.load(f)
-    return d
-

groupsio_util.py

@@ -1,5 +1,10 @@
 import requests, os, re
 from bs4 import BeautifulSoup
+import dateutil.parser
+import datetime
+
+class GroupsIOException(Exception):
+    pass
 
 class GroupsIOArchivesCrawler(object):
     """
@@ -59,6 +64,10 @@ class GroupsIOArchivesCrawler(object):
             v = re.sub(r'dcppc\+','',group['name'])
             subgroups[k] = v
 
+            ## Short circuit
+            ## for debugging purposes
+            break
+
         return subgroups
 
 
@@ -67,6 +76,8 @@ class GroupsIOArchivesCrawler(object):
         Spider will crawl the email archives of the entire group
         by crawling the email archives of each subgroup.
         """
+        self.archives = {}
+
         subgroups = self.get_subgroups_list()
 
         # ------------------------------
@@ -98,9 +109,9 @@ class GroupsIOArchivesCrawler(object):
         # Done. archives are now tucked away
         # in the variable self.archives
         # 
-        # self.archives is a list of dictionaries,
+        # self.archives is a dictionary of dictionaries,
-        # with each dictionary containing info about
+        # with each key a URL and each value a dictionary
-        # a topic/email thread in a subgroup.
+        # containing info about a thread.
         # ------------------------------
 
 
@@ -138,8 +149,6 @@ class GroupsIOArchivesCrawler(object):
         list of dictionaries.
 
         """
-        self.archives = []
-
         prefix = "https://{group}.groups.io".format(group=self.group_name)
 
         url = self.url.format(group=self.group_name, 
@@ -244,7 +253,7 @@ class GroupsIOArchivesCrawler(object):
             subject = soup.find('title').text
 
             # Extract information for the schema:
-            # - permalink for thread (done)
+            # - permalink for thread (done above)
             # - subject/title (done)
             # - original sender email/name (done)
             # - content (done)
@@ -259,11 +268,35 @@ class GroupsIOArchivesCrawler(object):
                     pass
                 else:
                     # found an email!
-                    # this is a maze, thanks groups.io
+                    # this is a maze, not amazing.
+                    # thanks groups.io!
                     td = tr.find('td')
-                    divrow = td.find('div',{'class':'row'}).find('div',{'class':'pull-left'})
+
+                    sender_divrow = td.find('div',{'class':'row'})
+                    sender_divrow = sender_divrow.find('div',{'class':'pull-left'})
                     if (i+1)==1:
-                        original_sender = divrow.text.strip()
+                        original_sender = sender_divrow.text.strip()
+
+                    date_divrow = td.find('div',{'class':'row'})
+                    date_divrow = date_divrow.find('div',{'class':'pull-right'})
+                    date_divrow = date_divrow.find('font',{'class':'text-muted'})
+                    date_divrow = date_divrow.find('script').text
+                    try:
+                        time_seconds = re.search(' [0-9]{1,} ',date_divrow).group(0)
+                        time_seconds = time_seconds.strip()
+                        # Thanks groups.io for the weird date formatting
+                        time_seconds = time_seconds[:10]
+                        mmicro_seconds = time_seconds[10:]
+                        if (i+1)==1:
+                            created_time  = datetime.datetime.utcfromtimestamp(int(time_seconds))
+                            modified_time = datetime.datetime.utcfromtimestamp(int(time_seconds))
+                        else:
+                            modified_time = datetime.datetime.utcfromtimestamp(int(time_seconds))
+
+                    except AttributeError:
+                        created_time = None
+                        modified_time = None
+
                     for div in td.find_all('div'):
                         if div.has_attr('id'):
 
@@ -292,19 +325,16 @@ class GroupsIOArchivesCrawler(object):
 
             thread = {
                     'permalink' : permalink,
+                    'created_time' : created_time,
+                    'modified_time' : modified_time,
                     'subject' : subject,
+                    'subgroup' : subgroup_name,
                     'original_sender' : original_sender,
                     'content' : full_content
             }
-            
+
-            print('*'*40)
+            print(" + Archiving thread: %s"%(thread['subject']))
-            for k in thread.keys():
+            self.archives[permalink] = thread
-                if k=='content':
-                    pass
-                else:
-                    print("%s : %s"%(k,thread[k]))
-            print('*'*40)
-            self.archives.append(thread)
 
 
     def extract_archive_page_items_(self, response):
@@ -319,15 +349,17 @@ class GroupsIOArchivesCrawler(object):
         soup = BeautifulSoup(response.content,"html.parser")
         rows = soup.find_all('tr',{'class':'test'})
         if 'rate limited' in soup.text:
-            raise Exception("Error: rate limit in place for Groups.io")
+            raise GroupsIOException("Error: rate limit in place for Groups.io")
 
         results = []
         for row in rows:
-            # We don't care about anything except title and ugly link
+            # This is where we extract
+            # a list of thread titles 
+            # and corresponding links.
             subject = row.find('span',{'class':'subject'})
             title = subject.get_text()
             link = row.find('a')['href']
-            print(title)
+
             results.append((title,link))
 
         return results
@@ -375,7 +407,7 @@ class GroupsIOArchivesCrawler(object):
         
         if csrf=='':
             err = "ERROR: Could not find csrf token on page."
-            raise Exception(err)
+            raise GroupsIOException(err)
     
         return csrf
 

requirements.txt

@@ -11,3 +11,4 @@ requests>=2.19
 pandoc>=1.0
 flask-dance>=1.0.0
 beautifulsoup4>=4.6
+python-dateutil>=2.6

static/dataTables.bootstrap.min.css

@@ -0,0 +1 @@
+table.dataTable{clear:both;margin-top:6px !important;margin-bottom:6px !important;max-width:none !important;border-collapse:separate !important}table.dataTable td,table.dataTable th{-webkit-box-sizing:content-box;box-sizing:content-box}table.dataTable td.dataTables_empty,table.dataTable th.dataTables_empty{text-align:center}table.dataTable.nowrap th,table.dataTable.nowrap td{white-space:nowrap}div.dataTables_wrapper div.dataTables_length label{font-weight:normal;text-align:left;white-space:nowrap}div.dataTables_wrapper div.dataTables_length select{width:75px;display:inline-block}div.dataTables_wrapper div.dataTables_filter{text-align:right}div.dataTables_wrapper div.dataTables_filter label{font-weight:normal;white-space:nowrap;text-align:left}div.dataTables_wrapper div.dataTables_filter input{margin-left:0.5em;display:inline-block;width:auto}div.dataTables_wrapper div.dataTables_info{padding-top:8px;white-space:nowrap}div.dataTables_wrapper div.dataTables_paginate{margin:0;white-space:nowrap;text-align:right}div.dataTables_wrapper div.dataTables_paginate ul.pagination{margin:2px 0;white-space:nowrap}div.dataTables_wrapper div.dataTables_processing{position:absolute;top:50%;left:50%;width:200px;margin-left:-100px;margin-top:-26px;text-align:center;padding:1em 0}table.dataTable thead>tr>th.sorting_asc,table.dataTable thead>tr>th.sorting_desc,table.dataTable thead>tr>th.sorting,table.dataTable thead>tr>td.sorting_asc,table.dataTable thead>tr>td.sorting_desc,table.dataTable thead>tr>td.sorting{padding-right:30px}table.dataTable thead>tr>th:active,table.dataTable thead>tr>td:active{outline:none}table.dataTable thead .sorting,table.dataTable thead .sorting_asc,table.dataTable thead .sorting_desc,table.dataTable thead .sorting_asc_disabled,table.dataTable thead .sorting_desc_disabled{cursor:pointer;position:relative}table.dataTable thead .sorting:after,table.dataTable thead .sorting_asc:after,table.dataTable thead .sorting_desc:after,table.dataTable thead .sorting_asc_disabled:after,table.dataTable thead .sorting_desc_disabled:after{position:absolute;bottom:8px;right:8px;display:block;font-family:'Glyphicons Halflings';opacity:0.5}table.dataTable thead .sorting:after{opacity:0.2;content:"---"}table.dataTable thead .sorting_asc:after{content:"/\\"}table.dataTable thead .sorting_desc:after{content:"\\/"}table.dataTable thead .sorting_asc_disabled:after,table.dataTable thead .sorting_desc_disabled:after{color:#eee}div.dataTables_scrollHead table.dataTable{margin-bottom:0 !important}div.dataTables_scrollBody>table{border-top:none;margin-top:0 !important;margin-bottom:0 !important}div.dataTables_scrollBody>table>thead .sorting:after,div.dataTables_scrollBody>table>thead .sorting_asc:after,div.dataTables_scrollBody>table>thead .sorting_desc:after{display:none}div.dataTables_scrollBody>table>tbody>tr:first-child>th,div.dataTables_scrollBody>table>tbody>tr:first-child>td{border-top:none}div.dataTables_scrollFoot>.dataTables_scrollFootInner{box-sizing:content-box}div.dataTables_scrollFoot>.dataTables_scrollFootInner>table{margin-top:0 !important;border-top:none}@media screen and (max-width: 767px){div.dataTables_wrapper div.dataTables_length,div.dataTables_wrapper div.dataTables_filter,div.dataTables_wrapper div.dataTables_info,div.dataTables_wrapper div.dataTables_paginate{text-align:center}}table.dataTable.table-condensed>thead>tr>th{padding-right:20px}table.dataTable.table-condensed .sorting:after,table.dataTable.table-condensed .sorting_asc:after,table.dataTable.table-condensed .sorting_desc:after{top:6px;right:6px}table.table-bordered.dataTable th,table.table-bordered.dataTable td{border-left-width:0}table.table-bordered.dataTable th:last-child,table.table-bordered.dataTable th:last-child,table.table-bordered.dataTable td:last-child,table.table-bordered.dataTable td:last-child{border-right-width:0}table.table-bordered.dataTable tbody th,table.table-bordered.dataTable tbody td{border-bottom-width:0}div.dataTables_scrollHead table.table-bordered{border-bottom-width:0}div.table-responsive>div.dataTables_wrapper>div.row{margin:0}div.table-responsive>div.dataTables_wrapper>div.row>div[class^="col-"]:first-child{padding-left:0}div.table-responsive>div.dataTables_wrapper>div.row>div[class^="col-"]:last-child{padding-right:0}

static/dataTables.bootstrap.min.js

@@ -0,0 +1,8 @@
+/*!
+ DataTables Bootstrap 3 integration
+ ©2011-2014 SpryMedia Ltd - datatables.net/license
+*/
+(function(){var f=function(c,b){c.extend(!0,b.defaults,{dom:"<'row'<'col-sm-6'l><'col-sm-6'f>><'row'<'col-sm-12'tr>><'row'<'col-sm-6'i><'col-sm-6'p>>",renderer:"bootstrap"});c.extend(b.ext.classes,{sWrapper:"dataTables_wrapper form-inline dt-bootstrap",sFilterInput:"form-control input-sm",sLengthSelect:"form-control input-sm"});b.ext.renderer.pageButton.bootstrap=function(g,f,p,k,h,l){var q=new b.Api(g),r=g.oClasses,i=g.oLanguage.oPaginate,d,e,o=function(b,f){var j,m,n,a,k=function(a){a.preventDefault();
+c(a.currentTarget).hasClass("disabled")||q.page(a.data.action).draw(!1)};j=0;for(m=f.length;j<m;j++)if(a=f[j],c.isArray(a))o(b,a);else{e=d="";switch(a){case "ellipsis":d="&hellip;";e="disabled";break;case "first":d=i.sFirst;e=a+(0<h?"":" disabled");break;case "previous":d=i.sPrevious;e=a+(0<h?"":" disabled");break;case "next":d=i.sNext;e=a+(h<l-1?"":" disabled");break;case "last":d=i.sLast;e=a+(h<l-1?"":" disabled");break;default:d=a+1,e=h===a?"active":""}d&&(n=c("<li>",{"class":r.sPageButton+" "+
+e,"aria-controls":g.sTableId,tabindex:g.iTabIndex,id:0===p&&"string"===typeof a?g.sTableId+"_"+a:null}).append(c("<a>",{href:"#"}).html(d)).appendTo(b),g.oApi._fnBindAction(n,{action:a},k))}};o(c(f).empty().html('<ul class="pagination"/>').children("ul"),k)};b.TableTools&&(c.extend(!0,b.TableTools.classes,{container:"DTTT btn-group",buttons:{normal:"btn btn-default",disabled:"disabled"},collection:{container:"DTTT_dropdown dropdown-menu",buttons:{normal:"",disabled:"disabled"}},print:{info:"DTTT_print_info"},
+select:{row:"active"}}),c.extend(!0,b.TableTools.DEFAULTS.oTags,{collection:{container:"ul",button:"li",liner:"a"}}))};"function"===typeof define&&define.amd?define(["jquery","datatables"],f):"object"===typeof exports?f(require("jquery"),require("datatables")):jQuery&&f(jQuery,jQuery.fn.dataTable)})(window,document);

static/dataTables.responsive.css

@@ -0,0 +1,106 @@
+table.dataTable.dtr-inline.collapsed > tbody > tr > td:first-child,
+table.dataTable.dtr-inline.collapsed > tbody > tr > th:first-child {
+  position: relative;
+  padding-left: 30px;
+  cursor: pointer;
+}
+table.dataTable.dtr-inline.collapsed > tbody > tr > td:first-child:before,
+table.dataTable.dtr-inline.collapsed > tbody > tr > th:first-child:before {
+  top: 8px;
+  left: 4px;
+  height: 16px;
+  width: 16px;
+  display: block;
+  position: absolute;
+  color: white;
+  border: 2px solid white;
+  border-radius: 16px;
+  text-align: center;
+  line-height: 14px;
+  box-shadow: 0 0 3px #444;
+  box-sizing: content-box;
+  content: '+';
+  background-color: #31b131;
+}
+table.dataTable.dtr-inline.collapsed > tbody > tr > td:first-child.dataTables_empty:before,
+table.dataTable.dtr-inline.collapsed > tbody > tr > th:first-child.dataTables_empty:before {
+  display: none;
+}
+table.dataTable.dtr-inline.collapsed > tbody > tr.parent > td:first-child:before,
+table.dataTable.dtr-inline.collapsed > tbody > tr.parent > th:first-child:before {
+  content: '-';
+  background-color: #d33333;
+}
+table.dataTable.dtr-inline.collapsed > tbody > tr.child td:before {
+  display: none;
+}
+table.dataTable.dtr-inline.collapsed.compact > tbody > tr > td:first-child,
+table.dataTable.dtr-inline.collapsed.compact > tbody > tr > th:first-child {
+  padding-left: 27px;
+}
+table.dataTable.dtr-inline.collapsed.compact > tbody > tr > td:first-child:before,
+table.dataTable.dtr-inline.collapsed.compact > tbody > tr > th:first-child:before {
+  top: 5px;
+  left: 4px;
+  height: 14px;
+  width: 14px;
+  border-radius: 14px;
+  line-height: 12px;
+}
+table.dataTable.dtr-column > tbody > tr > td.control,
+table.dataTable.dtr-column > tbody > tr > th.control {
+  position: relative;
+  cursor: pointer;
+}
+table.dataTable.dtr-column > tbody > tr > td.control:before,
+table.dataTable.dtr-column > tbody > tr > th.control:before {
+  top: 50%;
+  left: 50%;
+  height: 16px;
+  width: 16px;
+  margin-top: -10px;
+  margin-left: -10px;
+  display: block;
+  position: absolute;
+  color: white;
+  border: 2px solid white;
+  border-radius: 16px;
+  text-align: center;
+  line-height: 14px;
+  box-shadow: 0 0 3px #444;
+  box-sizing: content-box;
+  content: '+';
+  background-color: #31b131;
+}
+table.dataTable.dtr-column > tbody > tr.parent td.control:before,
+table.dataTable.dtr-column > tbody > tr.parent th.control:before {
+  content: '-';
+  background-color: #d33333;
+}
+table.dataTable > tbody > tr.child {
+  padding: 0.5em 1em;
+}
+table.dataTable > tbody > tr.child:hover {
+  background: transparent !important;
+}
+table.dataTable > tbody > tr.child ul {
+  display: inline-block;
+  list-style-type: none;
+  margin: 0;
+  padding: 0;
+}
+table.dataTable > tbody > tr.child ul li {
+  border-bottom: 1px solid #efefef;
+  padding: 0.5em 0;
+}
+table.dataTable > tbody > tr.child ul li:first-child {
+  padding-top: 0;
+}
+table.dataTable > tbody > tr.child ul li:last-child {
+  border-bottom: none;
+}
+table.dataTable > tbody > tr.child span.dtr-title {
+  display: inline-block;
+  min-width: 75px;
+  font-weight: bold;
+}

static/dataTables.responsive.js

@@ -0,0 +1,873 @@
+/*! Responsive 1.0.6
+ * 2014-2015 SpryMedia Ltd - datatables.net/license
+ */
+
+/**
+ * @summary     Responsive
+ * @description Responsive tables plug-in for DataTables
+ * @version     1.0.6
+ * @file        dataTables.responsive.js
+ * @author      SpryMedia Ltd (www.sprymedia.co.uk)
+ * @contact     www.sprymedia.co.uk/contact
+ * @copyright   Copyright 2014-2015 SpryMedia Ltd.
+ *
+ * This source file is free software, available under the following license:
+ *   MIT license - http://datatables.net/license/mit
+ *
+ * This source file is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE. See the license files for details.
+ *
+ * For details please refer to: http://www.datatables.net
+ */
+
+(function(window, document, undefined) {
+
+
+var factory = function( $, DataTable ) {
+"use strict";
+
+/**
+ * Responsive is a plug-in for the DataTables library that makes use of
+ * DataTables' ability to change the visibility of columns, changing the
+ * visibility of columns so the displayed columns fit into the table container.
+ * The end result is that complex tables will be dynamically adjusted to fit
+ * into the viewport, be it on a desktop, tablet or mobile browser.
+ *
+ * Responsive for DataTables has two modes of operation, which can used
+ * individually or combined:
+ *
+ * * Class name based control - columns assigned class names that match the
+ *   breakpoint logic can be shown / hidden as required for each breakpoint.
+ * * Automatic control - columns are automatically hidden when there is no
+ *   room left to display them. Columns removed from the right.
+ *
+ * In additional to column visibility control, Responsive also has built into
+ * options to use DataTables' child row display to show / hide the information
+ * from the table that has been hidden. There are also two modes of operation
+ * for this child row display:
+ *
+ * * Inline - when the control element that the user can use to show / hide
+ *   child rows is displayed inside the first column of the table.
+ * * Column - where a whole column is dedicated to be the show / hide control.
+ *
+ * Initialisation of Responsive is performed by:
+ *
+ * * Adding the class `responsive` or `dt-responsive` to the table. In this case
+ *   Responsive will automatically be initialised with the default configuration
+ *   options when the DataTable is created.
+ * * Using the `responsive` option in the DataTables configuration options. This
+ *   can also be used to specify the configuration options, or simply set to
+ *   `true` to use the defaults.
+ *
+ *  @class
+ *  @param {object} settings DataTables settings object for the host table
+ *  @param {object} [opts] Configuration options
+ *  @requires jQuery 1.7+
+ *  @requires DataTables 1.10.1+
+ *
+ *  @example
+ *      $('#example').DataTable( {
+ *        responsive: true
+ *      } );
+ *    } );
+ */
+var Responsive = function ( settings, opts ) {
+	// Sanity check that we are using DataTables 1.10 or newer
+	if ( ! DataTable.versionCheck || ! DataTable.versionCheck( '1.10.1' ) ) {
+		throw 'DataTables Responsive requires DataTables 1.10.1 or newer';
+	}
+
+	this.s = {
+		dt: new DataTable.Api( settings ),
+		columns: []
+	};
+
+	// Check if responsive has already been initialised on this table
+	if ( this.s.dt.settings()[0].responsive ) {
+		return;
+	}
+
+	// details is an object, but for simplicity the user can give it as a string
+	if ( opts && typeof opts.details === 'string' ) {
+		opts.details = { type: opts.details };
+	}
+
+	this.c = $.extend( true, {}, Responsive.defaults, DataTable.defaults.responsive, opts );
+	settings.responsive = this;
+	this._constructor();
+};
+
+Responsive.prototype = {
+	/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
+	 * Constructor
+	 */
+
+	/**
+	 * Initialise the Responsive instance
+	 *
+	 * @private
+	 */
+	_constructor: function ()
+	{
+		var that = this;
+		var dt = this.s.dt;
+
+		dt.settings()[0]._responsive = this;
+
+		// Use DataTables' private throttle function to avoid processor thrashing
+		$(window).on( 'resize.dtr orientationchange.dtr', dt.settings()[0].oApi._fnThrottle( function () {
+			that._resize();
+		} ) );
+
+		// Destroy event handler
+		dt.on( 'destroy.dtr', function () {
+			$(window).off( 'resize.dtr orientationchange.dtr draw.dtr' );
+		} );
+
+		// Reorder the breakpoints array here in case they have been added out
+		// of order
+		this.c.breakpoints.sort( function (a, b) {
+			return a.width < b.width ? 1 :
+				a.width > b.width ? -1 : 0;
+		} );
+
+		// Determine which columns are already hidden, and should therefore
+		// remain hidden. todo - should this be done? See thread 22677
+		//
+		// this.s.alwaysHidden = dt.columns(':hidden').indexes();
+
+		this._classLogic();
+		this._resizeAuto();
+
+		// Details handler
+		var details = this.c.details;
+		if ( details.type ) {
+			that._detailsInit();
+			this._detailsVis();
+
+			dt.on( 'column-visibility.dtr', function () {
+				that._detailsVis();
+			} );
+
+			// Redraw the details box on each draw. This is used until
+			// DataTables implements a native `updated` event for rows
+			dt.on( 'draw.dtr', function () {
+				dt.rows( {page: 'current'} ).iterator( 'row', function ( settings, idx ) {
+					var row = dt.row( idx );
+
+					if ( row.child.isShown() ) {
+						var info = that.c.details.renderer( dt, idx );
+						row.child( info, 'child' ).show();
+					}
+				} );
+			} );
+
+			$(dt.table().node()).addClass( 'dtr-'+details.type );
+		}
+
+		// First pass - draw the table for the current viewport size
+		this._resize();
+	},
+
+
+	/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
+	 * Private methods
+	 */
+
+	/**
+	 * Calculate the visibility for the columns in a table for a given
+	 * breakpoint. The result is pre-determined based on the class logic if
+	 * class names are used to control all columns, but the width of the table
+	 * is also used if there are columns which are to be automatically shown
+	 * and hidden.
+	 *
+	 * @param  {string} breakpoint Breakpoint name to use for the calculation
+	 * @return {array} Array of boolean values initiating the visibility of each
+	 *   column.
+	 *  @private
+	 */
+	_columnsVisiblity: function ( breakpoint )
+	{
+		var dt = this.s.dt;
+		var columns = this.s.columns;
+		var i, ien;
+
+		// Class logic - determine which columns are in this breakpoint based
+		// on the classes. If no class control (i.e. `auto`) then `-` is used
+		// to indicate this to the rest of the function
+		var display = $.map( columns, function ( col ) {
+			return col.auto && col.minWidth === null ?
+				false :
+				col.auto === true ?
+					'-' :
+					$.inArray( breakpoint, col.includeIn ) !== -1;
+		} );
+
+		// Auto column control - first pass: how much width is taken by the
+		// ones that must be included from the non-auto columns
+		var requiredWidth = 0;
+		for ( i=0, ien=display.length ; i<ien ; i++ ) {
+			if ( display[i] === true ) {
+				requiredWidth += columns[i].minWidth;
+			}
+		}
+
+		// Second pass, use up any remaining width for other columns. For
+		// scrolling tables we need to subtract the width of the scrollbar. It
+		// may not be requires which makes this sub-optimal, but it would
+		// require another full redraw to make complete use of those extra few
+		// pixels
+		var scrolling = dt.settings()[0].oScroll;
+		var bar = scrolling.sY || scrolling.sX ? scrolling.iBarWidth : 0;
+		var widthAvailable = dt.table().container().offsetWidth - bar;
+		var usedWidth = widthAvailable - requiredWidth;
+
+		// Control column needs to always be included. This makes it sub-
+		// optimal in terms of using the available with, but to stop layout
+		// thrashing or overflow. Also we need to account for the control column
+		// width first so we know how much width is available for the other
+		// columns, since the control column might not be the first one shown
+		for ( i=0, ien=display.length ; i<ien ; i++ ) {
+			if ( columns[i].control ) {
+				usedWidth -= columns[i].minWidth;
+			}
+		}
+
+		// Allow columns to be shown (counting from the left) until we run out
+		// of room
+		var empty = false;
+		for ( i=0, ien=display.length ; i<ien ; i++ ) {
+			if ( display[i] === '-' && ! columns[i].control ) {
+				// Once we've found a column that won't fit we don't let any
+				// others display either, or columns might disappear in the
+				// middle of the table
+				if ( empty || usedWidth - columns[i].minWidth < 0 ) {
+					empty = true;
+					display[i] = false;
+				}
+				else {
+					display[i] = true;
+				}
+
+				usedWidth -= columns[i].minWidth;
+			}
+		}
+
+		// Determine if the 'control' column should be shown (if there is one).
+		// This is the case when there is a hidden column (that is not the
+		// control column). The two loops look inefficient here, but they are
+		// trivial and will fly through. We need to know the outcome from the
+		// first , before the action in the second can be taken
+		var showControl = false;
+
+		for ( i=0, ien=columns.length ; i<ien ; i++ ) {
+			if ( ! columns[i].control && ! columns[i].never && ! display[i] ) {
+				showControl = true;
+				break;
+			}
+		}
+
+		for ( i=0, ien=columns.length ; i<ien ; i++ ) {
+			if ( columns[i].control ) {
+				display[i] = showControl;
+			}
+		}
+
+		// Finally we need to make sure that there is at least one column that
+		// is visible
+		if ( $.inArray( true, display ) === -1 ) {
+			display[0] = true;
+		}
+
+		return display;
+	},
+
+
+	/**
+	 * Create the internal `columns` array with information about the columns
+	 * for the table. This includes determining which breakpoints the column
+	 * will appear in, based upon class names in the column, which makes up the
+	 * vast majority of this method.
+	 *
+	 * @private
+	 */
+	_classLogic: function ()
+	{
+		var that = this;
+		var calc = {};
+		var breakpoints = this.c.breakpoints;
+		var columns = this.s.dt.columns().eq(0).map( function (i) {
+			var className = this.column(i).header().className;
+
+			return {
+				className: className,
+				includeIn: [],
+				auto:      false,
+				control:   false,
+				never:     className.match(/\bnever\b/) ? true : false
+			};
+		} );
+
+		// Simply add a breakpoint to `includeIn` array, ensuring that there are
+		// no duplicates
+		var add = function ( colIdx, name ) {
+			var includeIn = columns[ colIdx ].includeIn;
+
+			if ( $.inArray( name, includeIn ) === -1 ) {
+				includeIn.push( name );
+			}
+		};
+
+		var column = function ( colIdx, name, operator, matched ) {
+			var size, i, ien;
+
+			if ( ! operator ) {
+				columns[ colIdx ].includeIn.push( name );
+			}
+			else if ( operator === 'max-' ) {
+				// Add this breakpoint and all smaller
+				size = that._find( name ).width;
+
+				for ( i=0, ien=breakpoints.length ; i<ien ; i++ ) {
+					if ( breakpoints[i].width <= size ) {
+						add( colIdx, breakpoints[i].name );
+					}
+				}
+			}
+			else if ( operator === 'min-' ) {
+				// Add this breakpoint and all larger
+				size = that._find( name ).width;
+
+				for ( i=0, ien=breakpoints.length ; i<ien ; i++ ) {
+					if ( breakpoints[i].width >= size ) {
+						add( colIdx, breakpoints[i].name );
+					}
+				}
+			}
+			else if ( operator === 'not-' ) {
+				// Add all but this breakpoint (xxx need extra information)
+
+				for ( i=0, ien=breakpoints.length ; i<ien ; i++ ) {
+					if ( breakpoints[i].name.indexOf( matched ) === -1 ) {
+						add( colIdx, breakpoints[i].name );
+					}
+				}
+			}
+		};
+
+		// Loop over each column and determine if it has a responsive control
+		// class
+		columns.each( function ( col, i ) {
+			var classNames = col.className.split(' ');
+			var hasClass = false;
+
+			// Split the class name up so multiple rules can be applied if needed
+			for ( var k=0, ken=classNames.length ; k<ken ; k++ ) {
+				var className = $.trim( classNames[k] );
+
+				if ( className === 'all' ) {
+					// Include in all
+					hasClass = true;
+					col.includeIn = $.map( breakpoints, function (a) {
+						return a.name;
+					} );
+					return;
+				}
+				else if ( className === 'none' || className === 'never' ) {
+					// Include in none (default) and no auto
+					hasClass = true;
+					return;
+				}
+				else if ( className === 'control' ) {
+					// Special column that is only visible, when one of the other
+					// columns is hidden. This is used for the details control
+					hasClass = true;
+					col.control = true;
+					return;
+				}
+
+				$.each( breakpoints, function ( j, breakpoint ) {
+					// Does this column have a class that matches this breakpoint?
+					var brokenPoint = breakpoint.name.split('-');
+					var re = new RegExp( '(min\\-|max\\-|not\\-)?('+brokenPoint[0]+')(\\-[_a-zA-Z0-9])?' );
+					var match = className.match( re );
+
+					if ( match ) {
+						hasClass = true;
+
+						if ( match[2] === brokenPoint[0] && match[3] === '-'+brokenPoint[1] ) {
+							// Class name matches breakpoint name fully
+							column( i, breakpoint.name, match[1], match[2]+match[3] );
+						}
+						else if ( match[2] === brokenPoint[0] && ! match[3] ) {
+							// Class name matched primary breakpoint name with no qualifier
+							column( i, breakpoint.name, match[1], match[2] );
+						}
+					}
+				} );
+			}
+
+			// If there was no control class, then automatic sizing is used
+			if ( ! hasClass ) {
+				col.auto = true;
+			}
+		} );
+
+		this.s.columns = columns;
+	},
+
+
+	/**
+	 * Initialisation for the details handler
+	 *
+	 * @private
+	 */
+	_detailsInit: function ()
+	{
+		var that    = this;
+		var dt      = this.s.dt;
+		var details = this.c.details;
+
+		// The inline type always uses the first child as the target
+		if ( details.type === 'inline' ) {
+			details.target = 'td:first-child';
+		}
+
+		// type.target can be a string jQuery selector or a column index
+		var target   = details.target;
+		var selector = typeof target === 'string' ? target : 'td';
+
+		// Click handler to show / hide the details rows when they are available
+		$( dt.table().body() ).on( 'click', selector, function (e) {
+			// If the table is not collapsed (i.e. there is no hidden columns)
+			// then take no action
+			if ( ! $(dt.table().node()).hasClass('collapsed' ) ) {
+				return;
+			}
+
+			// Check that the row is actually a DataTable's controlled node
+			if ( ! dt.row( $(this).closest('tr') ).length ) {
+				return;
+			}
+
+			// For column index, we determine if we should act or not in the
+			// handler - otherwise it is already okay
+			if ( typeof target === 'number' ) {
+				var targetIdx = target < 0 ?
+					dt.columns().eq(0).length + target :
+					target;
+
+				if ( dt.cell( this ).index().column !== targetIdx ) {
+					return;
+				}
+			}
+
+			// $().closest() includes itself in its check
+			var row = dt.row( $(this).closest('tr') );
+
+			if ( row.child.isShown() ) {
+				row.child( false );
+				$( row.node() ).removeClass( 'parent' );
+			}
+			else {
+				var info = that.c.details.renderer( dt, row[0] );
+				row.child( info, 'child' ).show();
+				$( row.node() ).addClass( 'parent' );
+			}
+		} );
+	},
+
+
+	/**
+	 * Update the child rows in the table whenever the column visibility changes
+	 *
+	 * @private
+	 */
+	_detailsVis: function ()
+	{
+		var that = this;
+		var dt = this.s.dt;
+
+		// Find how many columns are hidden
+		var hiddenColumns = dt.columns().indexes().filter( function ( idx ) {
+			var col = dt.column( idx );
+
+			if ( col.visible() ) {
+				return null;
+			}
+
+			// Only counts as hidden if it doesn't have the `never` class
+			return $( col.header() ).hasClass( 'never' ) ? null : idx;
+		} );
+		var haveHidden = true;
+
+		if ( hiddenColumns.length === 0 || ( hiddenColumns.length === 1 && this.s.columns[ hiddenColumns[0] ].control ) ) {
+			haveHidden = false;
+		}
+
+		if ( haveHidden ) {
+			// Show all existing child rows
+			dt.rows( { page: 'current' } ).eq(0).each( function (idx) {
+				var row = dt.row( idx );
+
+				if ( row.child() ) {
+					var info = that.c.details.renderer( dt, row[0] );
+
+					// The renderer can return false to have no child row
+					if ( info === false ) {
+						row.child.hide();
+					}
+					else {
+						row.child( info, 'child' ).show();
+					}
+				}
+			} );
+		}
+		else {
+			// Hide all existing child rows
+			dt.rows( { page: 'current' } ).eq(0).each( function (idx) {
+				dt.row( idx ).child.hide();
+			} );
+		}
+	},
+
+
+	/**
+	 * Find a breakpoint object from a name
+	 * @param  {string} name Breakpoint name to find
+	 * @return {object}      Breakpoint description object
+	 */
+	_find: function ( name )
+	{
+		var breakpoints = this.c.breakpoints;
+
+		for ( var i=0, ien=breakpoints.length ; i<ien ; i++ ) {
+			if ( breakpoints[i].name === name ) {
+				return breakpoints[i];
+			}
+		}
+	},
+
+
+	/**
+	 * Alter the table display for a resized viewport. This involves first
+	 * determining what breakpoint the window currently is in, getting the
+	 * column visibilities to apply and then setting them.
+	 *
+	 * @private
+	 */
+	_resize: function ()
+	{
+		var dt = this.s.dt;
+		var width = $(window).width();
+		var breakpoints = this.c.breakpoints;
+		var breakpoint = breakpoints[0].name;
+		var columns = this.s.columns;
+		var i, ien;
+
+		// Determine what breakpoint we are currently at
+		for ( i=breakpoints.length-1 ; i>=0 ; i-- ) {
+			if ( width <= breakpoints[i].width ) {
+				breakpoint = breakpoints[i].name;
+				break;
+			}
+		}
+		
+		// Show the columns for that break point
+		var columnsVis = this._columnsVisiblity( breakpoint );
+
+		// Set the class before the column visibility is changed so event
+		// listeners know what the state is. Need to determine if there are
+		// any columns that are not visible but can be shown
+		var collapsedClass = false;
+		for ( i=0, ien=columns.length ; i<ien ; i++ ) {
+			if ( columnsVis[i] === false && ! columns[i].never ) {
+				collapsedClass = true;
+				break;
+			}
+		}
+
+		$( dt.table().node() ).toggleClass('collapsed', collapsedClass );
+
+		dt.columns().eq(0).each( function ( colIdx, i ) {
+			dt.column( colIdx ).visible( columnsVis[i] );
+		} );
+	},
+
+
+	/**
+	 * Determine the width of each column in the table so the auto column hiding
+	 * has that information to work with. This method is never going to be 100%
+	 * perfect since column widths can change slightly per page, but without
+	 * seriously compromising performance this is quite effective.
+	 *
+	 * @private
+	 */
+	_resizeAuto: function ()
+	{
+		var dt = this.s.dt;
+		var columns = this.s.columns;
+
+		// Are we allowed to do auto sizing?
+		if ( ! this.c.auto ) {
+			return;
+		}
+
+		// Are there any columns that actually need auto-sizing, or do they all
+		// have classes defined
+		if ( $.inArray( true, $.map( columns, function (c) { return c.auto; } ) ) === -1 ) {
+			return;
+		}
+
+		// Clone the table with the current data in it
+		var tableWidth   = dt.table().node().offsetWidth;
+		var columnWidths = dt.columns;
+		var clonedTable  = dt.table().node().cloneNode( false );
+		var clonedHeader = $( dt.table().header().cloneNode( false ) ).appendTo( clonedTable );
+		var clonedBody   = $( dt.table().body().cloneNode( false ) ).appendTo( clonedTable );
+
+		$( dt.table().footer() ).clone( false ).appendTo( clonedTable );
+
+		// This is a bit slow, but we need to get a clone of each row that
+		// includes all columns. As such, try to do this as little as possible.
+		dt.rows( { page: 'current' } ).indexes().flatten().each( function ( idx ) {
+			var clone = dt.row( idx ).node().cloneNode( true );
+			
+			if ( dt.columns( ':hidden' ).flatten().length ) {
+				$(clone).append( dt.cells( idx, ':hidden' ).nodes().to$().clone() );
+			}
+
+			$(clone).appendTo( clonedBody );
+		} );
+
+		var cells = dt.columns().header().to$().clone( false );
+		$('<tr/>')
+			.append( cells )
+			.appendTo( clonedHeader );
+
+		// In the inline case extra padding is applied to the first column to
+		// give space for the show / hide icon. We need to use this in the
+		// calculation
+		if ( this.c.details.type === 'inline' ) {
+			$(clonedTable).addClass( 'dtr-inline collapsed' );
+		}
+
+		var inserted = $('<div/>')
+			.css( {
+				width: 1,
+				height: 1,
+				overflow: 'hidden'
+			} )
+			.append( clonedTable );
+
+		// Remove columns which are not to be included
+		inserted.find('th.never, td.never').remove();
+
+		inserted.insertBefore( dt.table().node() );
+
+		// The cloned header now contains the smallest that each column can be
+		dt.columns().eq(0).each( function ( idx ) {
+			columns[idx].minWidth = cells[ idx ].offsetWidth || 0;
+		} );
+
+		inserted.remove();
+	}
+};
+
+
+/**
+ * List of default breakpoints. Each item in the array is an object with two
+ * properties:
+ *
+ * * `name` - the breakpoint name.
+ * * `width` - the breakpoint width
+ *
+ * @name Responsive.breakpoints
+ * @static
+ */
+Responsive.breakpoints = [
+	{ name: 'desktop',  width: Infinity },
+	{ name: 'tablet-l', width: 1024 },
+	{ name: 'tablet-p', width: 768 },
+	{ name: 'mobile-l', width: 480 },
+	{ name: 'mobile-p', width: 320 }
+];
+
+
+/**
+ * Responsive default settings for initialisation
+ *
+ * @namespace
+ * @name Responsive.defaults
+ * @static
+ */
+Responsive.defaults = {
+	/**
+	 * List of breakpoints for the instance. Note that this means that each
+	 * instance can have its own breakpoints. Additionally, the breakpoints
+	 * cannot be changed once an instance has been creased.
+	 *
+	 * @type {Array}
+	 * @default Takes the value of `Responsive.breakpoints`
+	 */
+	breakpoints: Responsive.breakpoints,
+
+	/**
+	 * Enable / disable auto hiding calculations. It can help to increase
+	 * performance slightly if you disable this option, but all columns would
+	 * need to have breakpoint classes assigned to them
+	 *
+	 * @type {Boolean}
+	 * @default  `true`
+	 */
+	auto: true,
+
+	/**
+	 * Details control. If given as a string value, the `type` property of the
+	 * default object is set to that value, and the defaults used for the rest
+	 * of the object - this is for ease of implementation.
+	 *
+	 * The object consists of the following properties:
+	 *
+	 * * `renderer` - function that is called for display of the child row data.
+	 *   The default function will show the data from the hidden columns
+	 * * `target` - Used as the selector for what objects to attach the child
+	 *   open / close to
+	 * * `type` - `false` to disable the details display, `inline` or `column`
+	 *   for the two control types
+	 *
+	 * @type {Object|string}
+	 */
+	details: {
+		renderer: function ( api, rowIdx ) {
+			var data = api.cells( rowIdx, ':hidden' ).eq(0).map( function ( cell ) {
+				var header = $( api.column( cell.column ).header() );
+				var idx = api.cell( cell ).index();
+
+				if ( header.hasClass( 'control' ) || header.hasClass( 'never' ) ) {
+					return '';
+				}
+
+				// Use a non-public DT API method to render the data for display
+				// This needs to be updated when DT adds a suitable method for
+				// this type of data retrieval
+				var dtPrivate = api.settings()[0];
+				var cellData = dtPrivate.oApi._fnGetCellData(
+					dtPrivate, idx.row, idx.column, 'display'
+				);
+				var title = header.text();
+				if ( title ) {
+					title = title + ':';
+				}
+
+				return '<li data-dtr-index="'+idx.column+'">'+
+						'<span class="dtr-title">'+
+							title+
+						'</span> '+
+						'<span class="dtr-data">'+
+							cellData+
+						'</span>'+
+					'</li>';
+			} ).toArray().join('');
+
+			return data ?
+				$('<ul data-dtr-index="'+rowIdx+'"/>').append( data ) :
+				false;
+		},
+
+		target: 0,
+
+		type: 'inline'
+	}
+};
+
+
+/*
+ * API
+ */
+var Api = $.fn.dataTable.Api;
+
+// Doesn't do anything - work around for a bug in DT... Not documented
+Api.register( 'responsive()', function () {
+	return this;
+} );
+
+Api.register( 'responsive.index()', function ( li ) {
+	li = $(li);
+
+	return {
+		column: li.data('dtr-index'),
+		row:    li.parent().data('dtr-index')
+	};
+} );
+
+Api.register( 'responsive.rebuild()', function () {
+	return this.iterator( 'table', function ( ctx ) {
+		if ( ctx._responsive ) {
+			ctx._responsive._classLogic();
+		}
+	} );
+} );
+
+Api.register( 'responsive.recalc()', function () {
+	return this.iterator( 'table', function ( ctx ) {
+		if ( ctx._responsive ) {
+			ctx._responsive._resizeAuto();
+			ctx._responsive._resize();
+		}
+	} );
+} );
+
+
+/**
+ * Version information
+ *
+ * @name Responsive.version
+ * @static
+ */
+Responsive.version = '1.0.6';
+
+
+$.fn.dataTable.Responsive = Responsive;
+$.fn.DataTable.Responsive = Responsive;
+
+// Attach a listener to the document which listens for DataTables initialisation
+// events so we can automatically initialise
+$(document).on( 'init.dt.dtr', function (e, settings, json) {
+	if ( e.namespace !== 'dt' ) {
+		return;
+	}
+
+	if ( $(settings.nTable).hasClass( 'responsive' ) ||
+		 $(settings.nTable).hasClass( 'dt-responsive' ) ||
+		 settings.oInit.responsive ||
+		 DataTable.defaults.responsive
+	) {
+		var init = settings.oInit.responsive;
+
+		if ( init !== false ) {
+			new Responsive( settings, $.isPlainObject( init ) ? init : {}  );
+		}
+	}
+} );
+
+return Responsive;
+}; // /factory
+
+
+// Define as an AMD module if possible
+if ( typeof define === 'function' && define.amd ) {
+	define( ['jquery', 'datatables'], factory );
+}
+else if ( typeof exports === 'object' ) {
+    // Node/CommonJS
+    factory( require('jquery'), require('datatables') );
+}
+else if ( jQuery && !jQuery.fn.dataTable.Responsive ) {
+	// Otherwise simply initialise as normal, stopping multiple evaluation
+	factory( jQuery, jQuery.fn.dataTable );
+}
+
+
+})(window, document);

static/feedback.js

@@ -0,0 +1,133 @@
+// submitting form with modal:
+// https://stackoverflow.com/a/29068742
+//
+// closing a bootstrap modal with submit button:
+// https://stackoverflow.com/a/33478107
+//
+// flask post data as json:
+// https://stackoverflow.com/a/16664376
+
+/* this function is called when the user submits
+ * the feedback form. it submits a post request
+ * to the flask server, which squirrels away the
+ * feedback in a file.
+ */
+function submit_feedback() {
+    // this function is called when submit button clicked
+    // algorithm:
+    // - check if text box has content
+    // - check if happy/sad filled out
+
+    var smile_active = $('#modal-feedback-smile-div').hasClass('smile-active');
+    var frown_active = $('#modal-feedback-frown-div').hasClass('frown-active');
+    if( !( smile_active || frown_active ) ) {
+        alert('Please pick the smile or the frown.')
+    } else if( $('#modal-feedback-textarea').val()=='' ) {
+        alert('Please provide us with some feedback.')
+    } else {
+        var user_sentiment = '';
+        if(smile_active) {
+            user_sentiment = 'smile';
+        } else {
+            user_sentiment = 'frown';
+        }
+        var escaped_text = $('#modal-feedback-textarea').val();
+
+        // prepare form data 
+        var data = {
+            sentiment : user_sentiment,
+            content : escaped_text
+        };
+        // post the form. the callback function resets the form
+        $.post("/feedback", 
+            data, 
+            function(response) {
+                $('#myModal').modal('hide');
+                $('#myModalForm')[0].reset();
+                add_alert(response);
+                frown_unclick();
+                smile_unclick();
+        });
+    }
+}
+
+
+function add_alert(response) {
+    str = ""
+    str += '<div id="feedback-messages-container" class="container">';
+
+    if (response['status']=='ok') {
+        // if status is ok, use alert-success
+        str += '    <div id="feedback-messages-alert" class="alert alert-success alert-dismissible fade in">';
+    } else {
+        // otherwise use alert-danger
+        str += '    <div id="feedback-messages-alert" class="alert alert-danger alert-dismissible fade in">';
+    }
+
+    str += '        <a href="#" class="close" data-dismiss="alert" aria-label="close">&times;</a>';
+    str += '        <div id="feedback-messages-contianer" class="container-fluid">';
+    str += '            <div id="feedback-messages-div" class="co-xs-12">';
+    str += '                <p>'
+    str += response['message'];
+    str += '                </p>';
+    str += '            </div>';
+    str += '    </div>';
+    str += '</div>';
+    $('div#messages').append(str);
+}
+
+
+/* for those particularly wordy users... limit feedback to 1000 chars */
+function cool_it() { 
+    if($('#modal-feedback-textarea').val().length > 1100 ){
+        $('#modal-too-long').show();
+    } else {
+        $('#modal-too-long').hide();
+    }
+}
+
+/* smiley functions */
+function smile_click() {
+    $('#modal-feedback-smile-div').addClass('smile-active');
+    $('#modal-feedback-smile-icon').addClass('smile-active');
+}
+function frown_click() {
+    $('#modal-feedback-frown-div').addClass('frown-active');
+    $('#modal-feedback-frown-icon').addClass('frown-active');
+}
+function smile_unclick() {
+    $('#modal-feedback-smile-div').removeClass('smile-active');
+    $('#modal-feedback-smile-icon').removeClass('smile-active');
+}
+function frown_unclick() {
+    $('#modal-feedback-frown-div').removeClass('frown-active');
+    $('#modal-feedback-frown-icon').removeClass('frown-active');
+}
+
+function smile() {
+    frown_unclick();
+    smile_click();
+}
+function frown() { 
+    smile_unclick();
+    frown_click();
+}
+
+
+/* for those particularly wordy users... limit feedback to 1100 chars */
+// how to check n characters in a textarea
+// https://stackoverflow.com/a/19934613
+/*
+$(document).ready(function() {
+
+    $('#modal-feedback-textarea').on('change',function(event) {
+        if($('#modal-feedback-textarea').val().length > 1100 ){
+            $('#modal-too-long').show();
+        } else {
+            $('#modal-too-long').hide();
+        }
+    });
+
+}
+*/
+

static/jquery.dataTables.min.js

@@ -0,0 +1,166 @@
+/*!
+ DataTables 1.10.12
+ ©2008-2015 SpryMedia Ltd - datatables.net/license
+*/
+(function(h){"function"===typeof define&&define.amd?define(["jquery"],function(D){return h(D,window,document)}):"object"===typeof exports?module.exports=function(D,I){D||(D=window);I||(I="undefined"!==typeof window?require("jquery"):require("jquery")(D));return h(I,D,D.document)}:h(jQuery,window,document)})(function(h,D,I,k){function X(a){var b,c,d={};h.each(a,function(e){if((b=e.match(/^([^A-Z]+?)([A-Z])/))&&-1!=="a aa ai ao as b fn i m o s ".indexOf(b[1]+" "))c=e.replace(b[0],b[2].toLowerCase()),
+d[c]=e,"o"===b[1]&&X(a[e])});a._hungarianMap=d}function K(a,b,c){a._hungarianMap||X(a);var d;h.each(b,function(e){d=a._hungarianMap[e];if(d!==k&&(c||b[d]===k))"o"===d.charAt(0)?(b[d]||(b[d]={}),h.extend(!0,b[d],b[e]),K(a[d],b[d],c)):b[d]=b[e]})}function Da(a){var b=m.defaults.oLanguage,c=a.sZeroRecords;!a.sEmptyTable&&(c&&"No data available in table"===b.sEmptyTable)&&E(a,a,"sZeroRecords","sEmptyTable");!a.sLoadingRecords&&(c&&"Loading..."===b.sLoadingRecords)&&E(a,a,"sZeroRecords","sLoadingRecords");
+a.sInfoThousands&&(a.sThousands=a.sInfoThousands);(a=a.sDecimal)&&db(a)}function eb(a){A(a,"ordering","bSort");A(a,"orderMulti","bSortMulti");A(a,"orderClasses","bSortClasses");A(a,"orderCellsTop","bSortCellsTop");A(a,"order","aaSorting");A(a,"orderFixed","aaSortingFixed");A(a,"paging","bPaginate");A(a,"pagingType","sPaginationType");A(a,"pageLength","iDisplayLength");A(a,"searching","bFilter");"boolean"===typeof a.sScrollX&&(a.sScrollX=a.sScrollX?"100%":"");"boolean"===typeof a.scrollX&&(a.scrollX=
+a.scrollX?"100%":"");if(a=a.aoSearchCols)for(var b=0,c=a.length;b<c;b++)a[b]&&K(m.models.oSearch,a[b])}function fb(a){A(a,"orderable","bSortable");A(a,"orderData","aDataSort");A(a,"orderSequence","asSorting");A(a,"orderDataType","sortDataType");var b=a.aDataSort;b&&!h.isArray(b)&&(a.aDataSort=[b])}function gb(a){if(!m.__browser){var b={};m.__browser=b;var c=h("<div/>").css({position:"fixed",top:0,left:0,height:1,width:1,overflow:"hidden"}).append(h("<div/>").css({position:"absolute",top:1,left:1,
+width:100,overflow:"scroll"}).append(h("<div/>").css({width:"100%",height:10}))).appendTo("body"),d=c.children(),e=d.children();b.barWidth=d[0].offsetWidth-d[0].clientWidth;b.bScrollOversize=100===e[0].offsetWidth&&100!==d[0].clientWidth;b.bScrollbarLeft=1!==Math.round(e.offset().left);b.bBounding=c[0].getBoundingClientRect().width?!0:!1;c.remove()}h.extend(a.oBrowser,m.__browser);a.oScroll.iBarWidth=m.__browser.barWidth}function hb(a,b,c,d,e,f){var g,j=!1;c!==k&&(g=c,j=!0);for(;d!==e;)a.hasOwnProperty(d)&&
+(g=j?b(g,a[d],d,a):a[d],j=!0,d+=f);return g}function Ea(a,b){var c=m.defaults.column,d=a.aoColumns.length,c=h.extend({},m.models.oColumn,c,{nTh:b?b:I.createElement("th"),sTitle:c.sTitle?c.sTitle:b?b.innerHTML:"",aDataSort:c.aDataSort?c.aDataSort:[d],mData:c.mData?c.mData:d,idx:d});a.aoColumns.push(c);c=a.aoPreSearchCols;c[d]=h.extend({},m.models.oSearch,c[d]);ja(a,d,h(b).data())}function ja(a,b,c){var b=a.aoColumns[b],d=a.oClasses,e=h(b.nTh);if(!b.sWidthOrig){b.sWidthOrig=e.attr("width")||null;var f=
+(e.attr("style")||"").match(/width:\s*(\d+[pxem%]+)/);f&&(b.sWidthOrig=f[1])}c!==k&&null!==c&&(fb(c),K(m.defaults.column,c),c.mDataProp!==k&&!c.mData&&(c.mData=c.mDataProp),c.sType&&(b._sManualType=c.sType),c.className&&!c.sClass&&(c.sClass=c.className),h.extend(b,c),E(b,c,"sWidth","sWidthOrig"),c.iDataSort!==k&&(b.aDataSort=[c.iDataSort]),E(b,c,"aDataSort"));var g=b.mData,j=Q(g),i=b.mRender?Q(b.mRender):null,c=function(a){return"string"===typeof a&&-1!==a.indexOf("@")};b._bAttrSrc=h.isPlainObject(g)&&
+(c(g.sort)||c(g.type)||c(g.filter));b._setter=null;b.fnGetData=function(a,b,c){var d=j(a,b,k,c);return i&&b?i(d,b,a,c):d};b.fnSetData=function(a,b,c){return R(g)(a,b,c)};"number"!==typeof g&&(a._rowReadObject=!0);a.oFeatures.bSort||(b.bSortable=!1,e.addClass(d.sSortableNone));a=-1!==h.inArray("asc",b.asSorting);c=-1!==h.inArray("desc",b.asSorting);!b.bSortable||!a&&!c?(b.sSortingClass=d.sSortableNone,b.sSortingClassJUI=""):a&&!c?(b.sSortingClass=d.sSortableAsc,b.sSortingClassJUI=d.sSortJUIAscAllowed):
+!a&&c?(b.sSortingClass=d.sSortableDesc,b.sSortingClassJUI=d.sSortJUIDescAllowed):(b.sSortingClass=d.sSortable,b.sSortingClassJUI=d.sSortJUI)}function Y(a){if(!1!==a.oFeatures.bAutoWidth){var b=a.aoColumns;Fa(a);for(var c=0,d=b.length;c<d;c++)b[c].nTh.style.width=b[c].sWidth}b=a.oScroll;(""!==b.sY||""!==b.sX)&&ka(a);u(a,null,"column-sizing",[a])}function Z(a,b){var c=la(a,"bVisible");return"number"===typeof c[b]?c[b]:null}function $(a,b){var c=la(a,"bVisible"),c=h.inArray(b,c);return-1!==c?c:null}
+function aa(a){var b=0;h.each(a.aoColumns,function(a,d){d.bVisible&&"none"!==h(d.nTh).css("display")&&b++});return b}function la(a,b){var c=[];h.map(a.aoColumns,function(a,e){a[b]&&c.push(e)});return c}function Ga(a){var b=a.aoColumns,c=a.aoData,d=m.ext.type.detect,e,f,g,j,i,h,l,q,t;e=0;for(f=b.length;e<f;e++)if(l=b[e],t=[],!l.sType&&l._sManualType)l.sType=l._sManualType;else if(!l.sType){g=0;for(j=d.length;g<j;g++){i=0;for(h=c.length;i<h;i++){t[i]===k&&(t[i]=B(a,i,e,"type"));q=d[g](t[i],a);if(!q&&
+g!==d.length-1)break;if("html"===q)break}if(q){l.sType=q;break}}l.sType||(l.sType="string")}}function ib(a,b,c,d){var e,f,g,j,i,n,l=a.aoColumns;if(b)for(e=b.length-1;0<=e;e--){n=b[e];var q=n.targets!==k?n.targets:n.aTargets;h.isArray(q)||(q=[q]);f=0;for(g=q.length;f<g;f++)if("number"===typeof q[f]&&0<=q[f]){for(;l.length<=q[f];)Ea(a);d(q[f],n)}else if("number"===typeof q[f]&&0>q[f])d(l.length+q[f],n);else if("string"===typeof q[f]){j=0;for(i=l.length;j<i;j++)("_all"==q[f]||h(l[j].nTh).hasClass(q[f]))&&
+d(j,n)}}if(c){e=0;for(a=c.length;e<a;e++)d(e,c[e])}}function N(a,b,c,d){var e=a.aoData.length,f=h.extend(!0,{},m.models.oRow,{src:c?"dom":"data",idx:e});f._aData=b;a.aoData.push(f);for(var g=a.aoColumns,j=0,i=g.length;j<i;j++)g[j].sType=null;a.aiDisplayMaster.push(e);b=a.rowIdFn(b);b!==k&&(a.aIds[b]=f);(c||!a.oFeatures.bDeferRender)&&Ha(a,e,c,d);return e}function ma(a,b){var c;b instanceof h||(b=h(b));return b.map(function(b,e){c=Ia(a,e);return N(a,c.data,e,c.cells)})}function B(a,b,c,d){var e=a.iDraw,
+f=a.aoColumns[c],g=a.aoData[b]._aData,j=f.sDefaultContent,i=f.fnGetData(g,d,{settings:a,row:b,col:c});if(i===k)return a.iDrawError!=e&&null===j&&(L(a,0,"Requested unknown parameter "+("function"==typeof f.mData?"{function}":"'"+f.mData+"'")+" for row "+b+", column "+c,4),a.iDrawError=e),j;if((i===g||null===i)&&null!==j&&d!==k)i=j;else if("function"===typeof i)return i.call(g);return null===i&&"display"==d?"":i}function jb(a,b,c,d){a.aoColumns[c].fnSetData(a.aoData[b]._aData,d,{settings:a,row:b,col:c})}
+function Ja(a){return h.map(a.match(/(\\.|[^\.])+/g)||[""],function(a){return a.replace(/\\./g,".")})}function Q(a){if(h.isPlainObject(a)){var b={};h.each(a,function(a,c){c&&(b[a]=Q(c))});return function(a,c,f,g){var j=b[c]||b._;return j!==k?j(a,c,f,g):a}}if(null===a)return function(a){return a};if("function"===typeof a)return function(b,c,f,g){return a(b,c,f,g)};if("string"===typeof a&&(-1!==a.indexOf(".")||-1!==a.indexOf("[")||-1!==a.indexOf("("))){var c=function(a,b,f){var g,j;if(""!==f){j=Ja(f);
+for(var i=0,n=j.length;i<n;i++){f=j[i].match(ba);g=j[i].match(U);if(f){j[i]=j[i].replace(ba,"");""!==j[i]&&(a=a[j[i]]);g=[];j.splice(0,i+1);j=j.join(".");if(h.isArray(a)){i=0;for(n=a.length;i<n;i++)g.push(c(a[i],b,j))}a=f[0].substring(1,f[0].length-1);a=""===a?g:g.join(a);break}else if(g){j[i]=j[i].replace(U,"");a=a[j[i]]();continue}if(null===a||a[j[i]]===k)return k;a=a[j[i]]}}return a};return function(b,e){return c(b,e,a)}}return function(b){return b[a]}}function R(a){if(h.isPlainObject(a))return R(a._);
+if(null===a)return function(){};if("function"===typeof a)return function(b,d,e){a(b,"set",d,e)};if("string"===typeof a&&(-1!==a.indexOf(".")||-1!==a.indexOf("[")||-1!==a.indexOf("("))){var b=function(a,d,e){var e=Ja(e),f;f=e[e.length-1];for(var g,j,i=0,n=e.length-1;i<n;i++){g=e[i].match(ba);j=e[i].match(U);if(g){e[i]=e[i].replace(ba,"");a[e[i]]=[];f=e.slice();f.splice(0,i+1);g=f.join(".");if(h.isArray(d)){j=0;for(n=d.length;j<n;j++)f={},b(f,d[j],g),a[e[i]].push(f)}else a[e[i]]=d;return}j&&(e[i]=e[i].replace(U,
+""),a=a[e[i]](d));if(null===a[e[i]]||a[e[i]]===k)a[e[i]]={};a=a[e[i]]}if(f.match(U))a[f.replace(U,"")](d);else a[f.replace(ba,"")]=d};return function(c,d){return b(c,d,a)}}return function(b,d){b[a]=d}}function Ka(a){return G(a.aoData,"_aData")}function na(a){a.aoData.length=0;a.aiDisplayMaster.length=0;a.aiDisplay.length=0;a.aIds={}}function oa(a,b,c){for(var d=-1,e=0,f=a.length;e<f;e++)a[e]==b?d=e:a[e]>b&&a[e]--; -1!=d&&c===k&&a.splice(d,1)}function ca(a,b,c,d){var e=a.aoData[b],f,g=function(c,d){for(;c.childNodes.length;)c.removeChild(c.firstChild);
+c.innerHTML=B(a,b,d,"display")};if("dom"===c||(!c||"auto"===c)&&"dom"===e.src)e._aData=Ia(a,e,d,d===k?k:e._aData).data;else{var j=e.anCells;if(j)if(d!==k)g(j[d],d);else{c=0;for(f=j.length;c<f;c++)g(j[c],c)}}e._aSortData=null;e._aFilterData=null;g=a.aoColumns;if(d!==k)g[d].sType=null;else{c=0;for(f=g.length;c<f;c++)g[c].sType=null;La(a,e)}}function Ia(a,b,c,d){var e=[],f=b.firstChild,g,j,i=0,n,l=a.aoColumns,q=a._rowReadObject,d=d!==k?d:q?{}:[],t=function(a,b){if("string"===typeof a){var c=a.indexOf("@");
+-1!==c&&(c=a.substring(c+1),R(a)(d,b.getAttribute(c)))}},S=function(a){if(c===k||c===i)j=l[i],n=h.trim(a.innerHTML),j&&j._bAttrSrc?(R(j.mData._)(d,n),t(j.mData.sort,a),t(j.mData.type,a),t(j.mData.filter,a)):q?(j._setter||(j._setter=R(j.mData)),j._setter(d,n)):d[i]=n;i++};if(f)for(;f;){g=f.nodeName.toUpperCase();if("TD"==g||"TH"==g)S(f),e.push(f);f=f.nextSibling}else{e=b.anCells;f=0;for(g=e.length;f<g;f++)S(e[f])}if(b=b.firstChild?b:b.nTr)(b=b.getAttribute("id"))&&R(a.rowId)(d,b);return{data:d,cells:e}}
+function Ha(a,b,c,d){var e=a.aoData[b],f=e._aData,g=[],j,i,n,l,q;if(null===e.nTr){j=c||I.createElement("tr");e.nTr=j;e.anCells=g;j._DT_RowIndex=b;La(a,e);l=0;for(q=a.aoColumns.length;l<q;l++){n=a.aoColumns[l];i=c?d[l]:I.createElement(n.sCellType);i._DT_CellIndex={row:b,column:l};g.push(i);if((!c||n.mRender||n.mData!==l)&&(!h.isPlainObject(n.mData)||n.mData._!==l+".display"))i.innerHTML=B(a,b,l,"display");n.sClass&&(i.className+=" "+n.sClass);n.bVisible&&!c?j.appendChild(i):!n.bVisible&&c&&i.parentNode.removeChild(i);
+n.fnCreatedCell&&n.fnCreatedCell.call(a.oInstance,i,B(a,b,l),f,b,l)}u(a,"aoRowCreatedCallback",null,[j,f,b])}e.nTr.setAttribute("role","row")}function La(a,b){var c=b.nTr,d=b._aData;if(c){var e=a.rowIdFn(d);e&&(c.id=e);d.DT_RowClass&&(e=d.DT_RowClass.split(" "),b.__rowc=b.__rowc?pa(b.__rowc.concat(e)):e,h(c).removeClass(b.__rowc.join(" ")).addClass(d.DT_RowClass));d.DT_RowAttr&&h(c).attr(d.DT_RowAttr);d.DT_RowData&&h(c).data(d.DT_RowData)}}function kb(a){var b,c,d,e,f,g=a.nTHead,j=a.nTFoot,i=0===
+h("th, td",g).length,n=a.oClasses,l=a.aoColumns;i&&(e=h("<tr/>").appendTo(g));b=0;for(c=l.length;b<c;b++)f=l[b],d=h(f.nTh).addClass(f.sClass),i&&d.appendTo(e),a.oFeatures.bSort&&(d.addClass(f.sSortingClass),!1!==f.bSortable&&(d.attr("tabindex",a.iTabIndex).attr("aria-controls",a.sTableId),Ma(a,f.nTh,b))),f.sTitle!=d[0].innerHTML&&d.html(f.sTitle),Na(a,"header")(a,d,f,n);i&&da(a.aoHeader,g);h(g).find(">tr").attr("role","row");h(g).find(">tr>th, >tr>td").addClass(n.sHeaderTH);h(j).find(">tr>th, >tr>td").addClass(n.sFooterTH);
+if(null!==j){a=a.aoFooter[0];b=0;for(c=a.length;b<c;b++)f=l[b],f.nTf=a[b].cell,f.sClass&&h(f.nTf).addClass(f.sClass)}}function ea(a,b,c){var d,e,f,g=[],j=[],i=a.aoColumns.length,n;if(b){c===k&&(c=!1);d=0;for(e=b.length;d<e;d++){g[d]=b[d].slice();g[d].nTr=b[d].nTr;for(f=i-1;0<=f;f--)!a.aoColumns[f].bVisible&&!c&&g[d].splice(f,1);j.push([])}d=0;for(e=g.length;d<e;d++){if(a=g[d].nTr)for(;f=a.firstChild;)a.removeChild(f);f=0;for(b=g[d].length;f<b;f++)if(n=i=1,j[d][f]===k){a.appendChild(g[d][f].cell);
+for(j[d][f]=1;g[d+i]!==k&&g[d][f].cell==g[d+i][f].cell;)j[d+i][f]=1,i++;for(;g[d][f+n]!==k&&g[d][f].cell==g[d][f+n].cell;){for(c=0;c<i;c++)j[d+c][f+n]=1;n++}h(g[d][f].cell).attr("rowspan",i).attr("colspan",n)}}}}function O(a){var b=u(a,"aoPreDrawCallback","preDraw",[a]);if(-1!==h.inArray(!1,b))C(a,!1);else{var b=[],c=0,d=a.asStripeClasses,e=d.length,f=a.oLanguage,g=a.iInitDisplayStart,j="ssp"==y(a),i=a.aiDisplay;a.bDrawing=!0;g!==k&&-1!==g&&(a._iDisplayStart=j?g:g>=a.fnRecordsDisplay()?0:g,a.iInitDisplayStart=
+-1);var g=a._iDisplayStart,n=a.fnDisplayEnd();if(a.bDeferLoading)a.bDeferLoading=!1,a.iDraw++,C(a,!1);else if(j){if(!a.bDestroying&&!lb(a))return}else a.iDraw++;if(0!==i.length){f=j?a.aoData.length:n;for(j=j?0:g;j<f;j++){var l=i[j],q=a.aoData[l];null===q.nTr&&Ha(a,l);l=q.nTr;if(0!==e){var t=d[c%e];q._sRowStripe!=t&&(h(l).removeClass(q._sRowStripe).addClass(t),q._sRowStripe=t)}u(a,"aoRowCallback",null,[l,q._aData,c,j]);b.push(l);c++}}else c=f.sZeroRecords,1==a.iDraw&&"ajax"==y(a)?c=f.sLoadingRecords:
+f.sEmptyTable&&0===a.fnRecordsTotal()&&(c=f.sEmptyTable),b[0]=h("<tr/>",{"class":e?d[0]:""}).append(h("<td />",{valign:"top",colSpan:aa(a),"class":a.oClasses.sRowEmpty}).html(c))[0];u(a,"aoHeaderCallback","header",[h(a.nTHead).children("tr")[0],Ka(a),g,n,i]);u(a,"aoFooterCallback","footer",[h(a.nTFoot).children("tr")[0],Ka(a),g,n,i]);d=h(a.nTBody);d.children().detach();d.append(h(b));u(a,"aoDrawCallback","draw",[a]);a.bSorted=!1;a.bFiltered=!1;a.bDrawing=!1}}function T(a,b){var c=a.oFeatures,d=c.bFilter;
+c.bSort&&mb(a);d?fa(a,a.oPreviousSearch):a.aiDisplay=a.aiDisplayMaster.slice();!0!==b&&(a._iDisplayStart=0);a._drawHold=b;O(a);a._drawHold=!1}function nb(a){var b=a.oClasses,c=h(a.nTable),c=h("<div/>").insertBefore(c),d=a.oFeatures,e=h("<div/>",{id:a.sTableId+"_wrapper","class":b.sWrapper+(a.nTFoot?"":" "+b.sNoFooter)});a.nHolding=c[0];a.nTableWrapper=e[0];a.nTableReinsertBefore=a.nTable.nextSibling;for(var f=a.sDom.split(""),g,j,i,n,l,q,t=0;t<f.length;t++){g=null;j=f[t];if("<"==j){i=h("<div/>")[0];
+n=f[t+1];if("'"==n||'"'==n){l="";for(q=2;f[t+q]!=n;)l+=f[t+q],q++;"H"==l?l=b.sJUIHeader:"F"==l&&(l=b.sJUIFooter);-1!=l.indexOf(".")?(n=l.split("."),i.id=n[0].substr(1,n[0].length-1),i.className=n[1]):"#"==l.charAt(0)?i.id=l.substr(1,l.length-1):i.className=l;t+=q}e.append(i);e=h(i)}else if(">"==j)e=e.parent();else if("l"==j&&d.bPaginate&&d.bLengthChange)g=ob(a);else if("f"==j&&d.bFilter)g=pb(a);else if("r"==j&&d.bProcessing)g=qb(a);else if("t"==j)g=rb(a);else if("i"==j&&d.bInfo)g=sb(a);else if("p"==
+j&&d.bPaginate)g=tb(a);else if(0!==m.ext.feature.length){i=m.ext.feature;q=0;for(n=i.length;q<n;q++)if(j==i[q].cFeature){g=i[q].fnInit(a);break}}g&&(i=a.aanFeatures,i[j]||(i[j]=[]),i[j].push(g),e.append(g))}c.replaceWith(e);a.nHolding=null}function da(a,b){var c=h(b).children("tr"),d,e,f,g,j,i,n,l,q,t;a.splice(0,a.length);f=0;for(i=c.length;f<i;f++)a.push([]);f=0;for(i=c.length;f<i;f++){d=c[f];for(e=d.firstChild;e;){if("TD"==e.nodeName.toUpperCase()||"TH"==e.nodeName.toUpperCase()){l=1*e.getAttribute("colspan");
+q=1*e.getAttribute("rowspan");l=!l||0===l||1===l?1:l;q=!q||0===q||1===q?1:q;g=0;for(j=a[f];j[g];)g++;n=g;t=1===l?!0:!1;for(j=0;j<l;j++)for(g=0;g<q;g++)a[f+g][n+j]={cell:e,unique:t},a[f+g].nTr=d}e=e.nextSibling}}}function qa(a,b,c){var d=[];c||(c=a.aoHeader,b&&(c=[],da(c,b)));for(var b=0,e=c.length;b<e;b++)for(var f=0,g=c[b].length;f<g;f++)if(c[b][f].unique&&(!d[f]||!a.bSortCellsTop))d[f]=c[b][f].cell;return d}function ra(a,b,c){u(a,"aoServerParams","serverParams",[b]);if(b&&h.isArray(b)){var d={},
+e=/(.*?)\[\]$/;h.each(b,function(a,b){var c=b.name.match(e);c?(c=c[0],d[c]||(d[c]=[]),d[c].push(b.value)):d[b.name]=b.value});b=d}var f,g=a.ajax,j=a.oInstance,i=function(b){u(a,null,"xhr",[a,b,a.jqXHR]);c(b)};if(h.isPlainObject(g)&&g.data){f=g.data;var n=h.isFunction(f)?f(b,a):f,b=h.isFunction(f)&&n?n:h.extend(!0,b,n);delete g.data}n={data:b,success:function(b){var c=b.error||b.sError;c&&L(a,0,c);a.json=b;i(b)},dataType:"json",cache:!1,type:a.sServerMethod,error:function(b,c){var d=u(a,null,"xhr",
+[a,null,a.jqXHR]);-1===h.inArray(!0,d)&&("parsererror"==c?L(a,0,"Invalid JSON response",1):4===b.readyState&&L(a,0,"Ajax error",7));C(a,!1)}};a.oAjaxData=b;u(a,null,"preXhr",[a,b]);a.fnServerData?a.fnServerData.call(j,a.sAjaxSource,h.map(b,function(a,b){return{name:b,value:a}}),i,a):a.sAjaxSource||"string"===typeof g?a.jqXHR=h.ajax(h.extend(n,{url:g||a.sAjaxSource})):h.isFunction(g)?a.jqXHR=g.call(j,b,i,a):(a.jqXHR=h.ajax(h.extend(n,g)),g.data=f)}function lb(a){return a.bAjaxDataGet?(a.iDraw++,C(a,
+!0),ra(a,ub(a),function(b){vb(a,b)}),!1):!0}function ub(a){var b=a.aoColumns,c=b.length,d=a.oFeatures,e=a.oPreviousSearch,f=a.aoPreSearchCols,g,j=[],i,n,l,q=V(a);g=a._iDisplayStart;i=!1!==d.bPaginate?a._iDisplayLength:-1;var k=function(a,b){j.push({name:a,value:b})};k("sEcho",a.iDraw);k("iColumns",c);k("sColumns",G(b,"sName").join(","));k("iDisplayStart",g);k("iDisplayLength",i);var S={draw:a.iDraw,columns:[],order:[],start:g,length:i,search:{value:e.sSearch,regex:e.bRegex}};for(g=0;g<c;g++)n=b[g],
+l=f[g],i="function"==typeof n.mData?"function":n.mData,S.columns.push({data:i,name:n.sName,searchable:n.bSearchable,orderable:n.bSortable,search:{value:l.sSearch,regex:l.bRegex}}),k("mDataProp_"+g,i),d.bFilter&&(k("sSearch_"+g,l.sSearch),k("bRegex_"+g,l.bRegex),k("bSearchable_"+g,n.bSearchable)),d.bSort&&k("bSortable_"+g,n.bSortable);d.bFilter&&(k("sSearch",e.sSearch),k("bRegex",e.bRegex));d.bSort&&(h.each(q,function(a,b){S.order.push({column:b.col,dir:b.dir});k("iSortCol_"+a,b.col);k("sSortDir_"+
+a,b.dir)}),k("iSortingCols",q.length));b=m.ext.legacy.ajax;return null===b?a.sAjaxSource?j:S:b?j:S}function vb(a,b){var c=sa(a,b),d=b.sEcho!==k?b.sEcho:b.draw,e=b.iTotalRecords!==k?b.iTotalRecords:b.recordsTotal,f=b.iTotalDisplayRecords!==k?b.iTotalDisplayRecords:b.recordsFiltered;if(d){if(1*d<a.iDraw)return;a.iDraw=1*d}na(a);a._iRecordsTotal=parseInt(e,10);a._iRecordsDisplay=parseInt(f,10);d=0;for(e=c.length;d<e;d++)N(a,c[d]);a.aiDisplay=a.aiDisplayMaster.slice();a.bAjaxDataGet=!1;O(a);a._bInitComplete||
+ta(a,b);a.bAjaxDataGet=!0;C(a,!1)}function sa(a,b){var c=h.isPlainObject(a.ajax)&&a.ajax.dataSrc!==k?a.ajax.dataSrc:a.sAjaxDataProp;return"data"===c?b.aaData||b[c]:""!==c?Q(c)(b):b}function pb(a){var b=a.oClasses,c=a.sTableId,d=a.oLanguage,e=a.oPreviousSearch,f=a.aanFeatures,g='<input type="search" class="'+b.sFilterInput+'"/>',j=d.sSearch,j=j.match(/_INPUT_/)?j.replace("_INPUT_",g):j+g,b=h("<div/>",{id:!f.f?c+"_filter":null,"class":b.sFilter}).append(h("<label/>").append(j)),f=function(){var b=!this.value?
+"":this.value;b!=e.sSearch&&(fa(a,{sSearch:b,bRegex:e.bRegex,bSmart:e.bSmart,bCaseInsensitive:e.bCaseInsensitive}),a._iDisplayStart=0,O(a))},g=null!==a.searchDelay?a.searchDelay:"ssp"===y(a)?400:0,i=h("input",b).val(e.sSearch).attr("placeholder",d.sSearchPlaceholder).bind("keyup.DT search.DT input.DT paste.DT cut.DT",g?Oa(f,g):f).bind("keypress.DT",function(a){if(13==a.keyCode)return!1}).attr("aria-controls",c);h(a.nTable).on("search.dt.DT",function(b,c){if(a===c)try{i[0]!==I.activeElement&&i.val(e.sSearch)}catch(d){}});
+return b[0]}function fa(a,b,c){var d=a.oPreviousSearch,e=a.aoPreSearchCols,f=function(a){d.sSearch=a.sSearch;d.bRegex=a.bRegex;d.bSmart=a.bSmart;d.bCaseInsensitive=a.bCaseInsensitive};Ga(a);if("ssp"!=y(a)){wb(a,b.sSearch,c,b.bEscapeRegex!==k?!b.bEscapeRegex:b.bRegex,b.bSmart,b.bCaseInsensitive);f(b);for(b=0;b<e.length;b++)xb(a,e[b].sSearch,b,e[b].bEscapeRegex!==k?!e[b].bEscapeRegex:e[b].bRegex,e[b].bSmart,e[b].bCaseInsensitive);yb(a)}else f(b);a.bFiltered=!0;u(a,null,"search",[a])}function yb(a){for(var b=
+m.ext.search,c=a.aiDisplay,d,e,f=0,g=b.length;f<g;f++){for(var j=[],i=0,n=c.length;i<n;i++)e=c[i],d=a.aoData[e],b[f](a,d._aFilterData,e,d._aData,i)&&j.push(e);c.length=0;h.merge(c,j)}}function xb(a,b,c,d,e,f){if(""!==b)for(var g=a.aiDisplay,d=Pa(b,d,e,f),e=g.length-1;0<=e;e--)b=a.aoData[g[e]]._aFilterData[c],d.test(b)||g.splice(e,1)}function wb(a,b,c,d,e,f){var d=Pa(b,d,e,f),e=a.oPreviousSearch.sSearch,f=a.aiDisplayMaster,g;0!==m.ext.search.length&&(c=!0);g=zb(a);if(0>=b.length)a.aiDisplay=f.slice();
+else{if(g||c||e.length>b.length||0!==b.indexOf(e)||a.bSorted)a.aiDisplay=f.slice();b=a.aiDisplay;for(c=b.length-1;0<=c;c--)d.test(a.aoData[b[c]]._sFilterRow)||b.splice(c,1)}}function Pa(a,b,c,d){a=b?a:Qa(a);c&&(a="^(?=.*?"+h.map(a.match(/"[^"]+"|[^ ]+/g)||[""],function(a){if('"'===a.charAt(0))var b=a.match(/^"(.*)"$/),a=b?b[1]:a;return a.replace('"',"")}).join(")(?=.*?")+").*$");return RegExp(a,d?"i":"")}function zb(a){var b=a.aoColumns,c,d,e,f,g,j,i,h,l=m.ext.type.search;c=!1;d=0;for(f=a.aoData.length;d<
+f;d++)if(h=a.aoData[d],!h._aFilterData){j=[];e=0;for(g=b.length;e<g;e++)c=b[e],c.bSearchable?(i=B(a,d,e,"filter"),l[c.sType]&&(i=l[c.sType](i)),null===i&&(i=""),"string"!==typeof i&&i.toString&&(i=i.toString())):i="",i.indexOf&&-1!==i.indexOf("&")&&(ua.innerHTML=i,i=Zb?ua.textContent:ua.innerText),i.replace&&(i=i.replace(/[\r\n]/g,"")),j.push(i);h._aFilterData=j;h._sFilterRow=j.join("  ");c=!0}return c}function Ab(a){return{search:a.sSearch,smart:a.bSmart,regex:a.bRegex,caseInsensitive:a.bCaseInsensitive}}
+function Bb(a){return{sSearch:a.search,bSmart:a.smart,bRegex:a.regex,bCaseInsensitive:a.caseInsensitive}}function sb(a){var b=a.sTableId,c=a.aanFeatures.i,d=h("<div/>",{"class":a.oClasses.sInfo,id:!c?b+"_info":null});c||(a.aoDrawCallback.push({fn:Cb,sName:"information"}),d.attr("role","status").attr("aria-live","polite"),h(a.nTable).attr("aria-describedby",b+"_info"));return d[0]}function Cb(a){var b=a.aanFeatures.i;if(0!==b.length){var c=a.oLanguage,d=a._iDisplayStart+1,e=a.fnDisplayEnd(),f=a.fnRecordsTotal(),
+g=a.fnRecordsDisplay(),j=g?c.sInfo:c.sInfoEmpty;g!==f&&(j+=" "+c.sInfoFiltered);j+=c.sInfoPostFix;j=Db(a,j);c=c.fnInfoCallback;null!==c&&(j=c.call(a.oInstance,a,d,e,f,g,j));h(b).html(j)}}function Db(a,b){var c=a.fnFormatNumber,d=a._iDisplayStart+1,e=a._iDisplayLength,f=a.fnRecordsDisplay(),g=-1===e;return b.replace(/_START_/g,c.call(a,d)).replace(/_END_/g,c.call(a,a.fnDisplayEnd())).replace(/_MAX_/g,c.call(a,a.fnRecordsTotal())).replace(/_TOTAL_/g,c.call(a,f)).replace(/_PAGE_/g,c.call(a,g?1:Math.ceil(d/
+e))).replace(/_PAGES_/g,c.call(a,g?1:Math.ceil(f/e)))}function ga(a){var b,c,d=a.iInitDisplayStart,e=a.aoColumns,f;c=a.oFeatures;var g=a.bDeferLoading;if(a.bInitialised){nb(a);kb(a);ea(a,a.aoHeader);ea(a,a.aoFooter);C(a,!0);c.bAutoWidth&&Fa(a);b=0;for(c=e.length;b<c;b++)f=e[b],f.sWidth&&(f.nTh.style.width=x(f.sWidth));u(a,null,"preInit",[a]);T(a);e=y(a);if("ssp"!=e||g)"ajax"==e?ra(a,[],function(c){var f=sa(a,c);for(b=0;b<f.length;b++)N(a,f[b]);a.iInitDisplayStart=d;T(a);C(a,!1);ta(a,c)},a):(C(a,!1),
+ta(a))}else setTimeout(function(){ga(a)},200)}function ta(a,b){a._bInitComplete=!0;(b||a.oInit.aaData)&&Y(a);u(a,null,"plugin-init",[a,b]);u(a,"aoInitComplete","init",[a,b])}function Ra(a,b){var c=parseInt(b,10);a._iDisplayLength=c;Sa(a);u(a,null,"length",[a,c])}function ob(a){for(var b=a.oClasses,c=a.sTableId,d=a.aLengthMenu,e=h.isArray(d[0]),f=e?d[0]:d,d=e?d[1]:d,e=h("<select/>",{name:c+"_length","aria-controls":c,"class":b.sLengthSelect}),g=0,j=f.length;g<j;g++)e[0][g]=new Option(d[g],f[g]);var i=
+h("<div><label/></div>").addClass(b.sLength);a.aanFeatures.l||(i[0].id=c+"_length");i.children().append(a.oLanguage.sLengthMenu.replace("_MENU_",e[0].outerHTML));h("select",i).val(a._iDisplayLength).bind("change.DT",function(){Ra(a,h(this).val());O(a)});h(a.nTable).bind("length.dt.DT",function(b,c,d){a===c&&h("select",i).val(d)});return i[0]}function tb(a){var b=a.sPaginationType,c=m.ext.pager[b],d="function"===typeof c,e=function(a){O(a)},b=h("<div/>").addClass(a.oClasses.sPaging+b)[0],f=a.aanFeatures;
+d||c.fnInit(a,b,e);f.p||(b.id=a.sTableId+"_paginate",a.aoDrawCallback.push({fn:function(a){if(d){var b=a._iDisplayStart,i=a._iDisplayLength,h=a.fnRecordsDisplay(),l=-1===i,b=l?0:Math.ceil(b/i),i=l?1:Math.ceil(h/i),h=c(b,i),k,l=0;for(k=f.p.length;l<k;l++)Na(a,"pageButton")(a,f.p[l],l,h,b,i)}else c.fnUpdate(a,e)},sName:"pagination"}));return b}function Ta(a,b,c){var d=a._iDisplayStart,e=a._iDisplayLength,f=a.fnRecordsDisplay();0===f||-1===e?d=0:"number"===typeof b?(d=b*e,d>f&&(d=0)):"first"==b?d=0:
+"previous"==b?(d=0<=e?d-e:0,0>d&&(d=0)):"next"==b?d+e<f&&(d+=e):"last"==b?d=Math.floor((f-1)/e)*e:L(a,0,"Unknown paging action: "+b,5);b=a._iDisplayStart!==d;a._iDisplayStart=d;b&&(u(a,null,"page",[a]),c&&O(a));return b}function qb(a){return h("<div/>",{id:!a.aanFeatures.r?a.sTableId+"_processing":null,"class":a.oClasses.sProcessing}).html(a.oLanguage.sProcessing).insertBefore(a.nTable)[0]}function C(a,b){a.oFeatures.bProcessing&&h(a.aanFeatures.r).css("display",b?"block":"none");u(a,null,"processing",
+[a,b])}function rb(a){var b=h(a.nTable);b.attr("role","grid");var c=a.oScroll;if(""===c.sX&&""===c.sY)return a.nTable;var d=c.sX,e=c.sY,f=a.oClasses,g=b.children("caption"),j=g.length?g[0]._captionSide:null,i=h(b[0].cloneNode(!1)),n=h(b[0].cloneNode(!1)),l=b.children("tfoot");l.length||(l=null);i=h("<div/>",{"class":f.sScrollWrapper}).append(h("<div/>",{"class":f.sScrollHead}).css({overflow:"hidden",position:"relative",border:0,width:d?!d?null:x(d):"100%"}).append(h("<div/>",{"class":f.sScrollHeadInner}).css({"box-sizing":"content-box",
+width:c.sXInner||"100%"}).append(i.removeAttr("id").css("margin-left",0).append("top"===j?g:null).append(b.children("thead"))))).append(h("<div/>",{"class":f.sScrollBody}).css({position:"relative",overflow:"auto",width:!d?null:x(d)}).append(b));l&&i.append(h("<div/>",{"class":f.sScrollFoot}).css({overflow:"hidden",border:0,width:d?!d?null:x(d):"100%"}).append(h("<div/>",{"class":f.sScrollFootInner}).append(n.removeAttr("id").css("margin-left",0).append("bottom"===j?g:null).append(b.children("tfoot")))));
+var b=i.children(),k=b[0],f=b[1],t=l?b[2]:null;if(d)h(f).on("scroll.DT",function(){var a=this.scrollLeft;k.scrollLeft=a;l&&(t.scrollLeft=a)});h(f).css(e&&c.bCollapse?"max-height":"height",e);a.nScrollHead=k;a.nScrollBody=f;a.nScrollFoot=t;a.aoDrawCallback.push({fn:ka,sName:"scrolling"});return i[0]}function ka(a){var b=a.oScroll,c=b.sX,d=b.sXInner,e=b.sY,b=b.iBarWidth,f=h(a.nScrollHead),g=f[0].style,j=f.children("div"),i=j[0].style,n=j.children("table"),j=a.nScrollBody,l=h(j),q=j.style,t=h(a.nScrollFoot).children("div"),
+m=t.children("table"),o=h(a.nTHead),F=h(a.nTable),p=F[0],r=p.style,u=a.nTFoot?h(a.nTFoot):null,Eb=a.oBrowser,Ua=Eb.bScrollOversize,s=G(a.aoColumns,"nTh"),P,v,w,y,z=[],A=[],B=[],C=[],D,E=function(a){a=a.style;a.paddingTop="0";a.paddingBottom="0";a.borderTopWidth="0";a.borderBottomWidth="0";a.height=0};v=j.scrollHeight>j.clientHeight;if(a.scrollBarVis!==v&&a.scrollBarVis!==k)a.scrollBarVis=v,Y(a);else{a.scrollBarVis=v;F.children("thead, tfoot").remove();u&&(w=u.clone().prependTo(F),P=u.find("tr"),w=
+w.find("tr"));y=o.clone().prependTo(F);o=o.find("tr");v=y.find("tr");y.find("th, td").removeAttr("tabindex");c||(q.width="100%",f[0].style.width="100%");h.each(qa(a,y),function(b,c){D=Z(a,b);c.style.width=a.aoColumns[D].sWidth});u&&J(function(a){a.style.width=""},w);f=F.outerWidth();if(""===c){r.width="100%";if(Ua&&(F.find("tbody").height()>j.offsetHeight||"scroll"==l.css("overflow-y")))r.width=x(F.outerWidth()-b);f=F.outerWidth()}else""!==d&&(r.width=x(d),f=F.outerWidth());J(E,v);J(function(a){B.push(a.innerHTML);
+z.push(x(h(a).css("width")))},v);J(function(a,b){if(h.inArray(a,s)!==-1)a.style.width=z[b]},o);h(v).height(0);u&&(J(E,w),J(function(a){C.push(a.innerHTML);A.push(x(h(a).css("width")))},w),J(function(a,b){a.style.width=A[b]},P),h(w).height(0));J(function(a,b){a.innerHTML='<div class="dataTables_sizing" style="height:0;overflow:hidden;">'+B[b]+"</div>";a.style.width=z[b]},v);u&&J(function(a,b){a.innerHTML='<div class="dataTables_sizing" style="height:0;overflow:hidden;">'+C[b]+"</div>";a.style.width=
+A[b]},w);if(F.outerWidth()<f){P=j.scrollHeight>j.offsetHeight||"scroll"==l.css("overflow-y")?f+b:f;if(Ua&&(j.scrollHeight>j.offsetHeight||"scroll"==l.css("overflow-y")))r.width=x(P-b);(""===c||""!==d)&&L(a,1,"Possible column misalignment",6)}else P="100%";q.width=x(P);g.width=x(P);u&&(a.nScrollFoot.style.width=x(P));!e&&Ua&&(q.height=x(p.offsetHeight+b));c=F.outerWidth();n[0].style.width=x(c);i.width=x(c);d=F.height()>j.clientHeight||"scroll"==l.css("overflow-y");e="padding"+(Eb.bScrollbarLeft?"Left":
+"Right");i[e]=d?b+"px":"0px";u&&(m[0].style.width=x(c),t[0].style.width=x(c),t[0].style[e]=d?b+"px":"0px");F.children("colgroup").insertBefore(F.children("thead"));l.scroll();if((a.bSorted||a.bFiltered)&&!a._drawHold)j.scrollTop=0}}function J(a,b,c){for(var d=0,e=0,f=b.length,g,j;e<f;){g=b[e].firstChild;for(j=c?c[e].firstChild:null;g;)1===g.nodeType&&(c?a(g,j,d):a(g,d),d++),g=g.nextSibling,j=c?j.nextSibling:null;e++}}function Fa(a){var b=a.nTable,c=a.aoColumns,d=a.oScroll,e=d.sY,f=d.sX,g=d.sXInner,
+j=c.length,i=la(a,"bVisible"),n=h("th",a.nTHead),l=b.getAttribute("width"),k=b.parentNode,t=!1,m,o,p=a.oBrowser,d=p.bScrollOversize;(m=b.style.width)&&-1!==m.indexOf("%")&&(l=m);for(m=0;m<i.length;m++)o=c[i[m]],null!==o.sWidth&&(o.sWidth=Fb(o.sWidthOrig,k),t=!0);if(d||!t&&!f&&!e&&j==aa(a)&&j==n.length)for(m=0;m<j;m++)i=Z(a,m),null!==i&&(c[i].sWidth=x(n.eq(m).width()));else{j=h(b).clone().css("visibility","hidden").removeAttr("id");j.find("tbody tr").remove();var r=h("<tr/>").appendTo(j.find("tbody"));
+j.find("thead, tfoot").remove();j.append(h(a.nTHead).clone()).append(h(a.nTFoot).clone());j.find("tfoot th, tfoot td").css("width","");n=qa(a,j.find("thead")[0]);for(m=0;m<i.length;m++)o=c[i[m]],n[m].style.width=null!==o.sWidthOrig&&""!==o.sWidthOrig?x(o.sWidthOrig):"",o.sWidthOrig&&f&&h(n[m]).append(h("<div/>").css({width:o.sWidthOrig,margin:0,padding:0,border:0,height:1}));if(a.aoData.length)for(m=0;m<i.length;m++)t=i[m],o=c[t],h(Gb(a,t)).clone(!1).append(o.sContentPadding).appendTo(r);h("[name]",
+j).removeAttr("name");o=h("<div/>").css(f||e?{position:"absolute",top:0,left:0,height:1,right:0,overflow:"hidden"}:{}).append(j).appendTo(k);f&&g?j.width(g):f?(j.css("width","auto"),j.removeAttr("width"),j.width()<k.clientWidth&&l&&j.width(k.clientWidth)):e?j.width(k.clientWidth):l&&j.width(l);for(m=e=0;m<i.length;m++)k=h(n[m]),g=k.outerWidth()-k.width(),k=p.bBounding?Math.ceil(n[m].getBoundingClientRect().width):k.outerWidth(),e+=k,c[i[m]].sWidth=x(k-g);b.style.width=x(e);o.remove()}l&&(b.style.width=
+x(l));if((l||f)&&!a._reszEvt)b=function(){h(D).bind("resize.DT-"+a.sInstance,Oa(function(){Y(a)}))},d?setTimeout(b,1E3):b(),a._reszEvt=!0}function Fb(a,b){if(!a)return 0;var c=h("<div/>").css("width",x(a)).appendTo(b||I.body),d=c[0].offsetWidth;c.remove();return d}function Gb(a,b){var c=Hb(a,b);if(0>c)return null;var d=a.aoData[c];return!d.nTr?h("<td/>").html(B(a,c,b,"display"))[0]:d.anCells[b]}function Hb(a,b){for(var c,d=-1,e=-1,f=0,g=a.aoData.length;f<g;f++)c=B(a,f,b,"display")+"",c=c.replace($b,
+""),c=c.replace(/&nbsp;/g," "),c.length>d&&(d=c.length,e=f);return e}function x(a){return null===a?"0px":"number"==typeof a?0>a?"0px":a+"px":a.match(/\d$/)?a+"px":a}function V(a){var b,c,d=[],e=a.aoColumns,f,g,j,i;b=a.aaSortingFixed;c=h.isPlainObject(b);var n=[];f=function(a){a.length&&!h.isArray(a[0])?n.push(a):h.merge(n,a)};h.isArray(b)&&f(b);c&&b.pre&&f(b.pre);f(a.aaSorting);c&&b.post&&f(b.post);for(a=0;a<n.length;a++){i=n[a][0];f=e[i].aDataSort;b=0;for(c=f.length;b<c;b++)g=f[b],j=e[g].sType||
+"string",n[a]._idx===k&&(n[a]._idx=h.inArray(n[a][1],e[g].asSorting)),d.push({src:i,col:g,dir:n[a][1],index:n[a]._idx,type:j,formatter:m.ext.type.order[j+"-pre"]})}return d}function mb(a){var b,c,d=[],e=m.ext.type.order,f=a.aoData,g=0,j,i=a.aiDisplayMaster,h;Ga(a);h=V(a);b=0;for(c=h.length;b<c;b++)j=h[b],j.formatter&&g++,Ib(a,j.col);if("ssp"!=y(a)&&0!==h.length){b=0;for(c=i.length;b<c;b++)d[i[b]]=b;g===h.length?i.sort(function(a,b){var c,e,g,j,i=h.length,k=f[a]._aSortData,m=f[b]._aSortData;for(g=
+0;g<i;g++)if(j=h[g],c=k[j.col],e=m[j.col],c=c<e?-1:c>e?1:0,0!==c)return"asc"===j.dir?c:-c;c=d[a];e=d[b];return c<e?-1:c>e?1:0}):i.sort(function(a,b){var c,g,j,i,k=h.length,m=f[a]._aSortData,p=f[b]._aSortData;for(j=0;j<k;j++)if(i=h[j],c=m[i.col],g=p[i.col],i=e[i.type+"-"+i.dir]||e["string-"+i.dir],c=i(c,g),0!==c)return c;c=d[a];g=d[b];return c<g?-1:c>g?1:0})}a.bSorted=!0}function Jb(a){for(var b,c,d=a.aoColumns,e=V(a),a=a.oLanguage.oAria,f=0,g=d.length;f<g;f++){c=d[f];var j=c.asSorting;b=c.sTitle.replace(/<.*?>/g,
+"");var i=c.nTh;i.removeAttribute("aria-sort");c.bSortable&&(0<e.length&&e[0].col==f?(i.setAttribute("aria-sort","asc"==e[0].dir?"ascending":"descending"),c=j[e[0].index+1]||j[0]):c=j[0],b+="asc"===c?a.sSortAscending:a.sSortDescending);i.setAttribute("aria-label",b)}}function Va(a,b,c,d){var e=a.aaSorting,f=a.aoColumns[b].asSorting,g=function(a,b){var c=a._idx;c===k&&(c=h.inArray(a[1],f));return c+1<f.length?c+1:b?null:0};"number"===typeof e[0]&&(e=a.aaSorting=[e]);c&&a.oFeatures.bSortMulti?(c=h.inArray(b,
+G(e,"0")),-1!==c?(b=g(e[c],!0),null===b&&1===e.length&&(b=0),null===b?e.splice(c,1):(e[c][1]=f[b],e[c]._idx=b)):(e.push([b,f[0],0]),e[e.length-1]._idx=0)):e.length&&e[0][0]==b?(b=g(e[0]),e.length=1,e[0][1]=f[b],e[0]._idx=b):(e.length=0,e.push([b,f[0]]),e[0]._idx=0);T(a);"function"==typeof d&&d(a)}function Ma(a,b,c,d){var e=a.aoColumns[c];Wa(b,{},function(b){!1!==e.bSortable&&(a.oFeatures.bProcessing?(C(a,!0),setTimeout(function(){Va(a,c,b.shiftKey,d);"ssp"!==y(a)&&C(a,!1)},0)):Va(a,c,b.shiftKey,d))})}
+function va(a){var b=a.aLastSort,c=a.oClasses.sSortColumn,d=V(a),e=a.oFeatures,f,g;if(e.bSort&&e.bSortClasses){e=0;for(f=b.length;e<f;e++)g=b[e].src,h(G(a.aoData,"anCells",g)).removeClass(c+(2>e?e+1:3));e=0;for(f=d.length;e<f;e++)g=d[e].src,h(G(a.aoData,"anCells",g)).addClass(c+(2>e?e+1:3))}a.aLastSort=d}function Ib(a,b){var c=a.aoColumns[b],d=m.ext.order[c.sSortDataType],e;d&&(e=d.call(a.oInstance,a,b,$(a,b)));for(var f,g=m.ext.type.order[c.sType+"-pre"],j=0,i=a.aoData.length;j<i;j++)if(c=a.aoData[j],
+c._aSortData||(c._aSortData=[]),!c._aSortData[b]||d)f=d?e[j]:B(a,j,b,"sort"),c._aSortData[b]=g?g(f):f}function wa(a){if(a.oFeatures.bStateSave&&!a.bDestroying){var b={time:+new Date,start:a._iDisplayStart,length:a._iDisplayLength,order:h.extend(!0,[],a.aaSorting),search:Ab(a.oPreviousSearch),columns:h.map(a.aoColumns,function(b,d){return{visible:b.bVisible,search:Ab(a.aoPreSearchCols[d])}})};u(a,"aoStateSaveParams","stateSaveParams",[a,b]);a.oSavedState=b;a.fnStateSaveCallback.call(a.oInstance,a,
+b)}}function Kb(a){var b,c,d=a.aoColumns;if(a.oFeatures.bStateSave){var e=a.fnStateLoadCallback.call(a.oInstance,a);if(e&&e.time&&(b=u(a,"aoStateLoadParams","stateLoadParams",[a,e]),-1===h.inArray(!1,b)&&(b=a.iStateDuration,!(0<b&&e.time<+new Date-1E3*b)&&d.length===e.columns.length))){a.oLoadedState=h.extend(!0,{},e);e.start!==k&&(a._iDisplayStart=e.start,a.iInitDisplayStart=e.start);e.length!==k&&(a._iDisplayLength=e.length);e.order!==k&&(a.aaSorting=[],h.each(e.order,function(b,c){a.aaSorting.push(c[0]>=
+d.length?[0,c[1]]:c)}));e.search!==k&&h.extend(a.oPreviousSearch,Bb(e.search));b=0;for(c=e.columns.length;b<c;b++){var f=e.columns[b];f.visible!==k&&(d[b].bVisible=f.visible);f.search!==k&&h.extend(a.aoPreSearchCols[b],Bb(f.search))}u(a,"aoStateLoaded","stateLoaded",[a,e])}}}function xa(a){var b=m.settings,a=h.inArray(a,G(b,"nTable"));return-1!==a?b[a]:null}function L(a,b,c,d){c="DataTables warning: "+(a?"table id="+a.sTableId+" - ":"")+c;d&&(c+=". For more information about this error, please see http://datatables.net/tn/"+
+d);if(b)D.console&&console.log&&console.log(c);else if(b=m.ext,b=b.sErrMode||b.errMode,a&&u(a,null,"error",[a,d,c]),"alert"==b)alert(c);else{if("throw"==b)throw Error(c);"function"==typeof b&&b(a,d,c)}}function E(a,b,c,d){h.isArray(c)?h.each(c,function(c,d){h.isArray(d)?E(a,b,d[0],d[1]):E(a,b,d)}):(d===k&&(d=c),b[c]!==k&&(a[d]=b[c]))}function Lb(a,b,c){var d,e;for(e in b)b.hasOwnProperty(e)&&(d=b[e],h.isPlainObject(d)?(h.isPlainObject(a[e])||(a[e]={}),h.extend(!0,a[e],d)):a[e]=c&&"data"!==e&&"aaData"!==
+e&&h.isArray(d)?d.slice():d);return a}function Wa(a,b,c){h(a).bind("click.DT",b,function(b){a.blur();c(b)}).bind("keypress.DT",b,function(a){13===a.which&&(a.preventDefault(),c(a))}).bind("selectstart.DT",function(){return!1})}function z(a,b,c,d){c&&a[b].push({fn:c,sName:d})}function u(a,b,c,d){var e=[];b&&(e=h.map(a[b].slice().reverse(),function(b){return b.fn.apply(a.oInstance,d)}));null!==c&&(b=h.Event(c+".dt"),h(a.nTable).trigger(b,d),e.push(b.result));return e}function Sa(a){var b=a._iDisplayStart,
+c=a.fnDisplayEnd(),d=a._iDisplayLength;b>=c&&(b=c-d);b-=b%d;if(-1===d||0>b)b=0;a._iDisplayStart=b}function Na(a,b){var c=a.renderer,d=m.ext.renderer[b];return h.isPlainObject(c)&&c[b]?d[c[b]]||d._:"string"===typeof c?d[c]||d._:d._}function y(a){return a.oFeatures.bServerSide?"ssp":a.ajax||a.sAjaxSource?"ajax":"dom"}function ya(a,b){var c=[],c=Mb.numbers_length,d=Math.floor(c/2);b<=c?c=W(0,b):a<=d?(c=W(0,c-2),c.push("ellipsis"),c.push(b-1)):(a>=b-1-d?c=W(b-(c-2),b):(c=W(a-d+2,a+d-1),c.push("ellipsis"),
+c.push(b-1)),c.splice(0,0,"ellipsis"),c.splice(0,0,0));c.DT_el="span";return c}function db(a){h.each({num:function(b){return za(b,a)},"num-fmt":function(b){return za(b,a,Xa)},"html-num":function(b){return za(b,a,Aa)},"html-num-fmt":function(b){return za(b,a,Aa,Xa)}},function(b,c){v.type.order[b+a+"-pre"]=c;b.match(/^html\-/)&&(v.type.search[b+a]=v.type.search.html)})}function Nb(a){return function(){var b=[xa(this[m.ext.iApiIndex])].concat(Array.prototype.slice.call(arguments));return m.ext.internal[a].apply(this,
+b)}}var m=function(a){this.$=function(a,b){return this.api(!0).$(a,b)};this._=function(a,b){return this.api(!0).rows(a,b).data()};this.api=function(a){return a?new r(xa(this[v.iApiIndex])):new r(this)};this.fnAddData=function(a,b){var c=this.api(!0),d=h.isArray(a)&&(h.isArray(a[0])||h.isPlainObject(a[0]))?c.rows.add(a):c.row.add(a);(b===k||b)&&c.draw();return d.flatten().toArray()};this.fnAdjustColumnSizing=function(a){var b=this.api(!0).columns.adjust(),c=b.settings()[0],d=c.oScroll;a===k||a?b.draw(!1):
+(""!==d.sX||""!==d.sY)&&ka(c)};this.fnClearTable=function(a){var b=this.api(!0).clear();(a===k||a)&&b.draw()};this.fnClose=function(a){this.api(!0).row(a).child.hide()};this.fnDeleteRow=function(a,b,c){var d=this.api(!0),a=d.rows(a),e=a.settings()[0],h=e.aoData[a[0][0]];a.remove();b&&b.call(this,e,h);(c===k||c)&&d.draw();return h};this.fnDestroy=function(a){this.api(!0).destroy(a)};this.fnDraw=function(a){this.api(!0).draw(a)};this.fnFilter=function(a,b,c,d,e,h){e=this.api(!0);null===b||b===k?e.search(a,
+c,d,h):e.column(b).search(a,c,d,h);e.draw()};this.fnGetData=function(a,b){var c=this.api(!0);if(a!==k){var d=a.nodeName?a.nodeName.toLowerCase():"";return b!==k||"td"==d||"th"==d?c.cell(a,b).data():c.row(a).data()||null}return c.data().toArray()};this.fnGetNodes=function(a){var b=this.api(!0);return a!==k?b.row(a).node():b.rows().nodes().flatten().toArray()};this.fnGetPosition=function(a){var b=this.api(!0),c=a.nodeName.toUpperCase();return"TR"==c?b.row(a).index():"TD"==c||"TH"==c?(a=b.cell(a).index(),
+[a.row,a.columnVisible,a.column]):null};this.fnIsOpen=function(a){return this.api(!0).row(a).child.isShown()};this.fnOpen=function(a,b,c){return this.api(!0).row(a).child(b,c).show().child()[0]};this.fnPageChange=function(a,b){var c=this.api(!0).page(a);(b===k||b)&&c.draw(!1)};this.fnSetColumnVis=function(a,b,c){a=this.api(!0).column(a).visible(b);(c===k||c)&&a.columns.adjust().draw()};this.fnSettings=function(){return xa(this[v.iApiIndex])};this.fnSort=function(a){this.api(!0).order(a).draw()};this.fnSortListener=
+function(a,b,c){this.api(!0).order.listener(a,b,c)};this.fnUpdate=function(a,b,c,d,e){var h=this.api(!0);c===k||null===c?h.row(b).data(a):h.cell(b,c).data(a);(e===k||e)&&h.columns.adjust();(d===k||d)&&h.draw();return 0};this.fnVersionCheck=v.fnVersionCheck;var b=this,c=a===k,d=this.length;c&&(a={});this.oApi=this.internal=v.internal;for(var e in m.ext.internal)e&&(this[e]=Nb(e));this.each(function(){var e={},e=1<d?Lb(e,a,!0):a,g=0,j,i=this.getAttribute("id"),n=!1,l=m.defaults,q=h(this);if("table"!=
+this.nodeName.toLowerCase())L(null,0,"Non-table node initialisation ("+this.nodeName+")",2);else{eb(l);fb(l.column);K(l,l,!0);K(l.column,l.column,!0);K(l,h.extend(e,q.data()));var t=m.settings,g=0;for(j=t.length;g<j;g++){var p=t[g];if(p.nTable==this||p.nTHead.parentNode==this||p.nTFoot&&p.nTFoot.parentNode==this){g=e.bRetrieve!==k?e.bRetrieve:l.bRetrieve;if(c||g)return p.oInstance;if(e.bDestroy!==k?e.bDestroy:l.bDestroy){p.oInstance.fnDestroy();break}else{L(p,0,"Cannot reinitialise DataTable",3);
+return}}if(p.sTableId==this.id){t.splice(g,1);break}}if(null===i||""===i)this.id=i="DataTables_Table_"+m.ext._unique++;var o=h.extend(!0,{},m.models.oSettings,{sDestroyWidth:q[0].style.width,sInstance:i,sTableId:i});o.nTable=this;o.oApi=b.internal;o.oInit=e;t.push(o);o.oInstance=1===b.length?b:q.dataTable();eb(e);e.oLanguage&&Da(e.oLanguage);e.aLengthMenu&&!e.iDisplayLength&&(e.iDisplayLength=h.isArray(e.aLengthMenu[0])?e.aLengthMenu[0][0]:e.aLengthMenu[0]);e=Lb(h.extend(!0,{},l),e);E(o.oFeatures,
+e,"bPaginate bLengthChange bFilter bSort bSortMulti bInfo bProcessing bAutoWidth bSortClasses bServerSide bDeferRender".split(" "));E(o,e,["asStripeClasses","ajax","fnServerData","fnFormatNumber","sServerMethod","aaSorting","aaSortingFixed","aLengthMenu","sPaginationType","sAjaxSource","sAjaxDataProp","iStateDuration","sDom","bSortCellsTop","iTabIndex","fnStateLoadCallback","fnStateSaveCallback","renderer","searchDelay","rowId",["iCookieDuration","iStateDuration"],["oSearch","oPreviousSearch"],["aoSearchCols",
+"aoPreSearchCols"],["iDisplayLength","_iDisplayLength"],["bJQueryUI","bJUI"]]);E(o.oScroll,e,[["sScrollX","sX"],["sScrollXInner","sXInner"],["sScrollY","sY"],["bScrollCollapse","bCollapse"]]);E(o.oLanguage,e,"fnInfoCallback");z(o,"aoDrawCallback",e.fnDrawCallback,"user");z(o,"aoServerParams",e.fnServerParams,"user");z(o,"aoStateSaveParams",e.fnStateSaveParams,"user");z(o,"aoStateLoadParams",e.fnStateLoadParams,"user");z(o,"aoStateLoaded",e.fnStateLoaded,"user");z(o,"aoRowCallback",e.fnRowCallback,
+"user");z(o,"aoRowCreatedCallback",e.fnCreatedRow,"user");z(o,"aoHeaderCallback",e.fnHeaderCallback,"user");z(o,"aoFooterCallback",e.fnFooterCallback,"user");z(o,"aoInitComplete",e.fnInitComplete,"user");z(o,"aoPreDrawCallback",e.fnPreDrawCallback,"user");o.rowIdFn=Q(e.rowId);gb(o);i=o.oClasses;e.bJQueryUI?(h.extend(i,m.ext.oJUIClasses,e.oClasses),e.sDom===l.sDom&&"lfrtip"===l.sDom&&(o.sDom='<"H"lfr>t<"F"ip>'),o.renderer)?h.isPlainObject(o.renderer)&&!o.renderer.header&&(o.renderer.header="jqueryui"):
+o.renderer="jqueryui":h.extend(i,m.ext.classes,e.oClasses);q.addClass(i.sTable);o.iInitDisplayStart===k&&(o.iInitDisplayStart=e.iDisplayStart,o._iDisplayStart=e.iDisplayStart);null!==e.iDeferLoading&&(o.bDeferLoading=!0,g=h.isArray(e.iDeferLoading),o._iRecordsDisplay=g?e.iDeferLoading[0]:e.iDeferLoading,o._iRecordsTotal=g?e.iDeferLoading[1]:e.iDeferLoading);var r=o.oLanguage;h.extend(!0,r,e.oLanguage);""!==r.sUrl&&(h.ajax({dataType:"json",url:r.sUrl,success:function(a){Da(a);K(l.oLanguage,a);h.extend(true,
+r,a);ga(o)},error:function(){ga(o)}}),n=!0);null===e.asStripeClasses&&(o.asStripeClasses=[i.sStripeOdd,i.sStripeEven]);var g=o.asStripeClasses,v=q.children("tbody").find("tr").eq(0);-1!==h.inArray(!0,h.map(g,function(a){return v.hasClass(a)}))&&(h("tbody tr",this).removeClass(g.join(" ")),o.asDestroyStripes=g.slice());t=[];g=this.getElementsByTagName("thead");0!==g.length&&(da(o.aoHeader,g[0]),t=qa(o));if(null===e.aoColumns){p=[];g=0;for(j=t.length;g<j;g++)p.push(null)}else p=e.aoColumns;g=0;for(j=
+p.length;g<j;g++)Ea(o,t?t[g]:null);ib(o,e.aoColumnDefs,p,function(a,b){ja(o,a,b)});if(v.length){var s=function(a,b){return a.getAttribute("data-"+b)!==null?b:null};h(v[0]).children("th, td").each(function(a,b){var c=o.aoColumns[a];if(c.mData===a){var d=s(b,"sort")||s(b,"order"),e=s(b,"filter")||s(b,"search");if(d!==null||e!==null){c.mData={_:a+".display",sort:d!==null?a+".@data-"+d:k,type:d!==null?a+".@data-"+d:k,filter:e!==null?a+".@data-"+e:k};ja(o,a)}}})}var w=o.oFeatures;e.bStateSave&&(w.bStateSave=
+!0,Kb(o,e),z(o,"aoDrawCallback",wa,"state_save"));if(e.aaSorting===k){t=o.aaSorting;g=0;for(j=t.length;g<j;g++)t[g][1]=o.aoColumns[g].asSorting[0]}va(o);w.bSort&&z(o,"aoDrawCallback",function(){if(o.bSorted){var a=V(o),b={};h.each(a,function(a,c){b[c.src]=c.dir});u(o,null,"order",[o,a,b]);Jb(o)}});z(o,"aoDrawCallback",function(){(o.bSorted||y(o)==="ssp"||w.bDeferRender)&&va(o)},"sc");g=q.children("caption").each(function(){this._captionSide=q.css("caption-side")});j=q.children("thead");0===j.length&&
+(j=h("<thead/>").appendTo(this));o.nTHead=j[0];j=q.children("tbody");0===j.length&&(j=h("<tbody/>").appendTo(this));o.nTBody=j[0];j=q.children("tfoot");if(0===j.length&&0<g.length&&(""!==o.oScroll.sX||""!==o.oScroll.sY))j=h("<tfoot/>").appendTo(this);0===j.length||0===j.children().length?q.addClass(i.sNoFooter):0<j.length&&(o.nTFoot=j[0],da(o.aoFooter,o.nTFoot));if(e.aaData)for(g=0;g<e.aaData.length;g++)N(o,e.aaData[g]);else(o.bDeferLoading||"dom"==y(o))&&ma(o,h(o.nTBody).children("tr"));o.aiDisplay=
+o.aiDisplayMaster.slice();o.bInitialised=!0;!1===n&&ga(o)}});b=null;return this},v,r,p,s,Ya={},Ob=/[\r\n]/g,Aa=/<.*?>/g,ac=/^[\w\+\-]/,bc=/[\w\+\-]$/,cc=RegExp("(\\/|\\.|\\*|\\+|\\?|\\||\\(|\\)|\\[|\\]|\\{|\\}|\\\\|\\$|\\^|\\-)","g"),Xa=/[',$£€¥%\u2009\u202F\u20BD\u20a9\u20BArfk]/gi,M=function(a){return!a||!0===a||"-"===a?!0:!1},Pb=function(a){var b=parseInt(a,10);return!isNaN(b)&&isFinite(a)?b:null},Qb=function(a,b){Ya[b]||(Ya[b]=RegExp(Qa(b),"g"));return"string"===typeof a&&"."!==b?a.replace(/\./g,
+"").replace(Ya[b],"."):a},Za=function(a,b,c){var d="string"===typeof a;if(M(a))return!0;b&&d&&(a=Qb(a,b));c&&d&&(a=a.replace(Xa,""));return!isNaN(parseFloat(a))&&isFinite(a)},Rb=function(a,b,c){return M(a)?!0:!(M(a)||"string"===typeof a)?null:Za(a.replace(Aa,""),b,c)?!0:null},G=function(a,b,c){var d=[],e=0,f=a.length;if(c!==k)for(;e<f;e++)a[e]&&a[e][b]&&d.push(a[e][b][c]);else for(;e<f;e++)a[e]&&d.push(a[e][b]);return d},ha=function(a,b,c,d){var e=[],f=0,g=b.length;if(d!==k)for(;f<g;f++)a[b[f]][c]&&
+e.push(a[b[f]][c][d]);else for(;f<g;f++)e.push(a[b[f]][c]);return e},W=function(a,b){var c=[],d;b===k?(b=0,d=a):(d=b,b=a);for(var e=b;e<d;e++)c.push(e);return c},Sb=function(a){for(var b=[],c=0,d=a.length;c<d;c++)a[c]&&b.push(a[c]);return b},pa=function(a){var b=[],c,d,e=a.length,f,g=0;d=0;a:for(;d<e;d++){c=a[d];for(f=0;f<g;f++)if(b[f]===c)continue a;b.push(c);g++}return b};m.util={throttle:function(a,b){var c=b!==k?b:200,d,e;return function(){var b=this,g=+new Date,h=arguments;d&&g<d+c?(clearTimeout(e),
+e=setTimeout(function(){d=k;a.apply(b,h)},c)):(d=g,a.apply(b,h))}},escapeRegex:function(a){return a.replace(cc,"\\$1")}};var A=function(a,b,c){a[b]!==k&&(a[c]=a[b])},ba=/\[.*?\]$/,U=/\(\)$/,Qa=m.util.escapeRegex,ua=h("<div>")[0],Zb=ua.textContent!==k,$b=/<.*?>/g,Oa=m.util.throttle,Tb=[],w=Array.prototype,dc=function(a){var b,c,d=m.settings,e=h.map(d,function(a){return a.nTable});if(a){if(a.nTable&&a.oApi)return[a];if(a.nodeName&&"table"===a.nodeName.toLowerCase())return b=h.inArray(a,e),-1!==b?[d[b]]:
+null;if(a&&"function"===typeof a.settings)return a.settings().toArray();"string"===typeof a?c=h(a):a instanceof h&&(c=a)}else return[];if(c)return c.map(function(){b=h.inArray(this,e);return-1!==b?d[b]:null}).toArray()};r=function(a,b){if(!(this instanceof r))return new r(a,b);var c=[],d=function(a){(a=dc(a))&&(c=c.concat(a))};if(h.isArray(a))for(var e=0,f=a.length;e<f;e++)d(a[e]);else d(a);this.context=pa(c);b&&h.merge(this,b);this.selector={rows:null,cols:null,opts:null};r.extend(this,this,Tb)};
+m.Api=r;h.extend(r.prototype,{any:function(){return 0!==this.count()},concat:w.concat,context:[],count:function(){return this.flatten().length},each:function(a){for(var b=0,c=this.length;b<c;b++)a.call(this,this[b],b,this);return this},eq:function(a){var b=this.context;return b.length>a?new r(b[a],this[a]):null},filter:function(a){var b=[];if(w.filter)b=w.filter.call(this,a,this);else for(var c=0,d=this.length;c<d;c++)a.call(this,this[c],c,this)&&b.push(this[c]);return new r(this.context,b)},flatten:function(){var a=
+[];return new r(this.context,a.concat.apply(a,this.toArray()))},join:w.join,indexOf:w.indexOf||function(a,b){for(var c=b||0,d=this.length;c<d;c++)if(this[c]===a)return c;return-1},iterator:function(a,b,c,d){var e=[],f,g,h,i,n,l=this.context,m,t,p=this.selector;"string"===typeof a&&(d=c,c=b,b=a,a=!1);g=0;for(h=l.length;g<h;g++){var o=new r(l[g]);if("table"===b)f=c.call(o,l[g],g),f!==k&&e.push(f);else if("columns"===b||"rows"===b)f=c.call(o,l[g],this[g],g),f!==k&&e.push(f);else if("column"===b||"column-rows"===
+b||"row"===b||"cell"===b){t=this[g];"column-rows"===b&&(m=Ba(l[g],p.opts));i=0;for(n=t.length;i<n;i++)f=t[i],f="cell"===b?c.call(o,l[g],f.row,f.column,g,i):c.call(o,l[g],f,g,i,m),f!==k&&e.push(f)}}return e.length||d?(a=new r(l,a?e.concat.apply([],e):e),b=a.selector,b.rows=p.rows,b.cols=p.cols,b.opts=p.opts,a):this},lastIndexOf:w.lastIndexOf||function(a,b){return this.indexOf.apply(this.toArray.reverse(),arguments)},length:0,map:function(a){var b=[];if(w.map)b=w.map.call(this,a,this);else for(var c=
+0,d=this.length;c<d;c++)b.push(a.call(this,this[c],c));return new r(this.context,b)},pluck:function(a){return this.map(function(b){return b[a]})},pop:w.pop,push:w.push,reduce:w.reduce||function(a,b){return hb(this,a,b,0,this.length,1)},reduceRight:w.reduceRight||function(a,b){return hb(this,a,b,this.length-1,-1,-1)},reverse:w.reverse,selector:null,shift:w.shift,sort:w.sort,splice:w.splice,toArray:function(){return w.slice.call(this)},to$:function(){return h(this)},toJQuery:function(){return h(this)},
+unique:function(){return new r(this.context,pa(this))},unshift:w.unshift});r.extend=function(a,b,c){if(c.length&&b&&(b instanceof r||b.__dt_wrapper)){var d,e,f,g=function(a,b,c){return function(){var d=b.apply(a,arguments);r.extend(d,d,c.methodExt);return d}};d=0;for(e=c.length;d<e;d++)f=c[d],b[f.name]="function"===typeof f.val?g(a,f.val,f):h.isPlainObject(f.val)?{}:f.val,b[f.name].__dt_wrapper=!0,r.extend(a,b[f.name],f.propExt)}};r.register=p=function(a,b){if(h.isArray(a))for(var c=0,d=a.length;c<
+d;c++)r.register(a[c],b);else for(var e=a.split("."),f=Tb,g,j,c=0,d=e.length;c<d;c++){g=(j=-1!==e[c].indexOf("()"))?e[c].replace("()",""):e[c];var i;a:{i=0;for(var n=f.length;i<n;i++)if(f[i].name===g){i=f[i];break a}i=null}i||(i={name:g,val:{},methodExt:[],propExt:[]},f.push(i));c===d-1?i.val=b:f=j?i.methodExt:i.propExt}};r.registerPlural=s=function(a,b,c){r.register(a,c);r.register(b,function(){var a=c.apply(this,arguments);return a===this?this:a instanceof r?a.length?h.isArray(a[0])?new r(a.context,
+a[0]):a[0]:k:a})};p("tables()",function(a){var b;if(a){b=r;var c=this.context;if("number"===typeof a)a=[c[a]];else var d=h.map(c,function(a){return a.nTable}),a=h(d).filter(a).map(function(){var a=h.inArray(this,d);return c[a]}).toArray();b=new b(a)}else b=this;return b});p("table()",function(a){var a=this.tables(a),b=a.context;return b.length?new r(b[0]):a});s("tables().nodes()","table().node()",function(){return this.iterator("table",function(a){return a.nTable},1)});s("tables().body()","table().body()",
+function(){return this.iterator("table",function(a){return a.nTBody},1)});s("tables().header()","table().header()",function(){return this.iterator("table",function(a){return a.nTHead},1)});s("tables().footer()","table().footer()",function(){return this.iterator("table",function(a){return a.nTFoot},1)});s("tables().containers()","table().container()",function(){return this.iterator("table",function(a){return a.nTableWrapper},1)});p("draw()",function(a){return this.iterator("table",function(b){"page"===
+a?O(b):("string"===typeof a&&(a="full-hold"===a?!1:!0),T(b,!1===a))})});p("page()",function(a){return a===k?this.page.info().page:this.iterator("table",function(b){Ta(b,a)})});p("page.info()",function(){if(0===this.context.length)return k;var a=this.context[0],b=a._iDisplayStart,c=a.oFeatures.bPaginate?a._iDisplayLength:-1,d=a.fnRecordsDisplay(),e=-1===c;return{page:e?0:Math.floor(b/c),pages:e?1:Math.ceil(d/c),start:b,end:a.fnDisplayEnd(),length:c,recordsTotal:a.fnRecordsTotal(),recordsDisplay:d,
+serverSide:"ssp"===y(a)}});p("page.len()",function(a){return a===k?0!==this.context.length?this.context[0]._iDisplayLength:k:this.iterator("table",function(b){Ra(b,a)})});var Ub=function(a,b,c){if(c){var d=new r(a);d.one("draw",function(){c(d.ajax.json())})}if("ssp"==y(a))T(a,b);else{C(a,!0);var e=a.jqXHR;e&&4!==e.readyState&&e.abort();ra(a,[],function(c){na(a);for(var c=sa(a,c),d=0,e=c.length;d<e;d++)N(a,c[d]);T(a,b);C(a,!1)})}};p("ajax.json()",function(){var a=this.context;if(0<a.length)return a[0].json});
+p("ajax.params()",function(){var a=this.context;if(0<a.length)return a[0].oAjaxData});p("ajax.reload()",function(a,b){return this.iterator("table",function(c){Ub(c,!1===b,a)})});p("ajax.url()",function(a){var b=this.context;if(a===k){if(0===b.length)return k;b=b[0];return b.ajax?h.isPlainObject(b.ajax)?b.ajax.url:b.ajax:b.sAjaxSource}return this.iterator("table",function(b){h.isPlainObject(b.ajax)?b.ajax.url=a:b.ajax=a})});p("ajax.url().load()",function(a,b){return this.iterator("table",function(c){Ub(c,
+!1===b,a)})});var $a=function(a,b,c,d,e){var f=[],g,j,i,n,l,m;i=typeof b;if(!b||"string"===i||"function"===i||b.length===k)b=[b];i=0;for(n=b.length;i<n;i++){j=b[i]&&b[i].split?b[i].split(","):[b[i]];l=0;for(m=j.length;l<m;l++)(g=c("string"===typeof j[l]?h.trim(j[l]):j[l]))&&g.length&&(f=f.concat(g))}a=v.selector[a];if(a.length){i=0;for(n=a.length;i<n;i++)f=a[i](d,e,f)}return pa(f)},ab=function(a){a||(a={});a.filter&&a.search===k&&(a.search=a.filter);return h.extend({search:"none",order:"current",
+page:"all"},a)},bb=function(a){for(var b=0,c=a.length;b<c;b++)if(0<a[b].length)return a[0]=a[b],a[0].length=1,a.length=1,a.context=[a.context[b]],a;a.length=0;return a},Ba=function(a,b){var c,d,e,f=[],g=a.aiDisplay;c=a.aiDisplayMaster;var j=b.search;d=b.order;e=b.page;if("ssp"==y(a))return"removed"===j?[]:W(0,c.length);if("current"==e){c=a._iDisplayStart;for(d=a.fnDisplayEnd();c<d;c++)f.push(g[c])}else if("current"==d||"applied"==d)f="none"==j?c.slice():"applied"==j?g.slice():h.map(c,function(a){return-1===
+h.inArray(a,g)?a:null});else if("index"==d||"original"==d){c=0;for(d=a.aoData.length;c<d;c++)"none"==j?f.push(c):(e=h.inArray(c,g),(-1===e&&"removed"==j||0<=e&&"applied"==j)&&f.push(c))}return f};p("rows()",function(a,b){a===k?a="":h.isPlainObject(a)&&(b=a,a="");var b=ab(b),c=this.iterator("table",function(c){var e=b;return $a("row",a,function(a){var b=Pb(a);if(b!==null&&!e)return[b];var j=Ba(c,e);if(b!==null&&h.inArray(b,j)!==-1)return[b];if(!a)return j;if(typeof a==="function")return h.map(j,function(b){var e=
+c.aoData[b];return a(b,e._aData,e.nTr)?b:null});b=Sb(ha(c.aoData,j,"nTr"));if(a.nodeName){if(a._DT_RowIndex!==k)return[a._DT_RowIndex];if(a._DT_CellIndex)return[a._DT_CellIndex.row];b=h(a).closest("*[data-dt-row]");return b.length?[b.data("dt-row")]:[]}if(typeof a==="string"&&a.charAt(0)==="#"){j=c.aIds[a.replace(/^#/,"")];if(j!==k)return[j.idx]}return h(b).filter(a).map(function(){return this._DT_RowIndex}).toArray()},c,e)},1);c.selector.rows=a;c.selector.opts=b;return c});p("rows().nodes()",function(){return this.iterator("row",
+function(a,b){return a.aoData[b].nTr||k},1)});p("rows().data()",function(){return this.iterator(!0,"rows",function(a,b){return ha(a.aoData,b,"_aData")},1)});s("rows().cache()","row().cache()",function(a){return this.iterator("row",function(b,c){var d=b.aoData[c];return"search"===a?d._aFilterData:d._aSortData},1)});s("rows().invalidate()","row().invalidate()",function(a){return this.iterator("row",function(b,c){ca(b,c,a)})});s("rows().indexes()","row().index()",function(){return this.iterator("row",
+function(a,b){return b},1)});s("rows().ids()","row().id()",function(a){for(var b=[],c=this.context,d=0,e=c.length;d<e;d++)for(var f=0,g=this[d].length;f<g;f++){var h=c[d].rowIdFn(c[d].aoData[this[d][f]]._aData);b.push((!0===a?"#":"")+h)}return new r(c,b)});s("rows().remove()","row().remove()",function(){var a=this;this.iterator("row",function(b,c,d){var e=b.aoData,f=e[c],g,h,i,n,l;e.splice(c,1);g=0;for(h=e.length;g<h;g++)if(i=e[g],l=i.anCells,null!==i.nTr&&(i.nTr._DT_RowIndex=g),null!==l){i=0;for(n=
+l.length;i<n;i++)l[i]._DT_CellIndex.row=g}oa(b.aiDisplayMaster,c);oa(b.aiDisplay,c);oa(a[d],c,!1);Sa(b);c=b.rowIdFn(f._aData);c!==k&&delete b.aIds[c]});this.iterator("table",function(a){for(var c=0,d=a.aoData.length;c<d;c++)a.aoData[c].idx=c});return this});p("rows.add()",function(a){var b=this.iterator("table",function(b){var c,f,g,h=[];f=0;for(g=a.length;f<g;f++)c=a[f],c.nodeName&&"TR"===c.nodeName.toUpperCase()?h.push(ma(b,c)[0]):h.push(N(b,c));return h},1),c=this.rows(-1);c.pop();h.merge(c,b);
+return c});p("row()",function(a,b){return bb(this.rows(a,b))});p("row().data()",function(a){var b=this.context;if(a===k)return b.length&&this.length?b[0].aoData[this[0]]._aData:k;b[0].aoData[this[0]]._aData=a;ca(b[0],this[0],"data");return this});p("row().node()",function(){var a=this.context;return a.length&&this.length?a[0].aoData[this[0]].nTr||null:null});p("row.add()",function(a){a instanceof h&&a.length&&(a=a[0]);var b=this.iterator("table",function(b){return a.nodeName&&"TR"===a.nodeName.toUpperCase()?
+ma(b,a)[0]:N(b,a)});return this.row(b[0])});var cb=function(a,b){var c=a.context;if(c.length&&(c=c[0].aoData[b!==k?b:a[0]])&&c._details)c._details.remove(),c._detailsShow=k,c._details=k},Vb=function(a,b){var c=a.context;if(c.length&&a.length){var d=c[0].aoData[a[0]];if(d._details){(d._detailsShow=b)?d._details.insertAfter(d.nTr):d._details.detach();var e=c[0],f=new r(e),g=e.aoData;f.off("draw.dt.DT_details column-visibility.dt.DT_details destroy.dt.DT_details");0<G(g,"_details").length&&(f.on("draw.dt.DT_details",
+function(a,b){e===b&&f.rows({page:"current"}).eq(0).each(function(a){a=g[a];a._detailsShow&&a._details.insertAfter(a.nTr)})}),f.on("column-visibility.dt.DT_details",function(a,b){if(e===b)for(var c,d=aa(b),f=0,h=g.length;f<h;f++)c=g[f],c._details&&c._details.children("td[colspan]").attr("colspan",d)}),f.on("destroy.dt.DT_details",function(a,b){if(e===b)for(var c=0,d=g.length;c<d;c++)g[c]._details&&cb(f,c)}))}}};p("row().child()",function(a,b){var c=this.context;if(a===k)return c.length&&this.length?
+c[0].aoData[this[0]]._details:k;if(!0===a)this.child.show();else if(!1===a)cb(this);else if(c.length&&this.length){var d=c[0],c=c[0].aoData[this[0]],e=[],f=function(a,b){if(h.isArray(a)||a instanceof h)for(var c=0,k=a.length;c<k;c++)f(a[c],b);else a.nodeName&&"tr"===a.nodeName.toLowerCase()?e.push(a):(c=h("<tr><td/></tr>").addClass(b),h("td",c).addClass(b).html(a)[0].colSpan=aa(d),e.push(c[0]))};f(a,b);c._details&&c._details.remove();c._details=h(e);c._detailsShow&&c._details.insertAfter(c.nTr)}return this});
+p(["row().child.show()","row().child().show()"],function(){Vb(this,!0);return this});p(["row().child.hide()","row().child().hide()"],function(){Vb(this,!1);return this});p(["row().child.remove()","row().child().remove()"],function(){cb(this);return this});p("row().child.isShown()",function(){var a=this.context;return a.length&&this.length?a[0].aoData[this[0]]._detailsShow||!1:!1});var ec=/^(.+):(name|visIdx|visible)$/,Wb=function(a,b,c,d,e){for(var c=[],d=0,f=e.length;d<f;d++)c.push(B(a,e[d],b));
+return c};p("columns()",function(a,b){a===k?a="":h.isPlainObject(a)&&(b=a,a="");var b=ab(b),c=this.iterator("table",function(c){var e=a,f=b,g=c.aoColumns,j=G(g,"sName"),i=G(g,"nTh");return $a("column",e,function(a){var b=Pb(a);if(a==="")return W(g.length);if(b!==null)return[b>=0?b:g.length+b];if(typeof a==="function"){var e=Ba(c,f);return h.map(g,function(b,f){return a(f,Wb(c,f,0,0,e),i[f])?f:null})}var k=typeof a==="string"?a.match(ec):"";if(k)switch(k[2]){case "visIdx":case "visible":b=parseInt(k[1],
+10);if(b<0){var m=h.map(g,function(a,b){return a.bVisible?b:null});return[m[m.length+b]]}return[Z(c,b)];case "name":return h.map(j,function(a,b){return a===k[1]?b:null});default:return[]}if(a.nodeName&&a._DT_CellIndex)return[a._DT_CellIndex.column];b=h(i).filter(a).map(function(){return h.inArray(this,i)}).toArray();if(b.length||!a.nodeName)return b;b=h(a).closest("*[data-dt-column]");return b.length?[b.data("dt-column")]:[]},c,f)},1);c.selector.cols=a;c.selector.opts=b;return c});s("columns().header()",
+"column().header()",function(){return this.iterator("column",function(a,b){return a.aoColumns[b].nTh},1)});s("columns().footer()","column().footer()",function(){return this.iterator("column",function(a,b){return a.aoColumns[b].nTf},1)});s("columns().data()","column().data()",function(){return this.iterator("column-rows",Wb,1)});s("columns().dataSrc()","column().dataSrc()",function(){return this.iterator("column",function(a,b){return a.aoColumns[b].mData},1)});s("columns().cache()","column().cache()",
+function(a){return this.iterator("column-rows",function(b,c,d,e,f){return ha(b.aoData,f,"search"===a?"_aFilterData":"_aSortData",c)},1)});s("columns().nodes()","column().nodes()",function(){return this.iterator("column-rows",function(a,b,c,d,e){return ha(a.aoData,e,"anCells",b)},1)});s("columns().visible()","column().visible()",function(a,b){var c=this.iterator("column",function(b,c){if(a===k)return b.aoColumns[c].bVisible;var f=b.aoColumns,g=f[c],j=b.aoData,i,n,l;if(a!==k&&g.bVisible!==a){if(a){var m=
+h.inArray(!0,G(f,"bVisible"),c+1);i=0;for(n=j.length;i<n;i++)l=j[i].nTr,f=j[i].anCells,l&&l.insertBefore(f[c],f[m]||null)}else h(G(b.aoData,"anCells",c)).detach();g.bVisible=a;ea(b,b.aoHeader);ea(b,b.aoFooter);wa(b)}});a!==k&&(this.iterator("column",function(c,e){u(c,null,"column-visibility",[c,e,a,b])}),(b===k||b)&&this.columns.adjust());return c});s("columns().indexes()","column().index()",function(a){return this.iterator("column",function(b,c){return"visible"===a?$(b,c):c},1)});p("columns.adjust()",
+function(){return this.iterator("table",function(a){Y(a)},1)});p("column.index()",function(a,b){if(0!==this.context.length){var c=this.context[0];if("fromVisible"===a||"toData"===a)return Z(c,b);if("fromData"===a||"toVisible"===a)return $(c,b)}});p("column()",function(a,b){return bb(this.columns(a,b))});p("cells()",function(a,b,c){h.isPlainObject(a)&&(a.row===k?(c=a,a=null):(c=b,b=null));h.isPlainObject(b)&&(c=b,b=null);if(null===b||b===k)return this.iterator("table",function(b){var d=a,e=ab(c),f=
+b.aoData,g=Ba(b,e),j=Sb(ha(f,g,"anCells")),i=h([].concat.apply([],j)),l,n=b.aoColumns.length,m,p,r,u,v,s;return $a("cell",d,function(a){var c=typeof a==="function";if(a===null||a===k||c){m=[];p=0;for(r=g.length;p<r;p++){l=g[p];for(u=0;u<n;u++){v={row:l,column:u};if(c){s=f[l];a(v,B(b,l,u),s.anCells?s.anCells[u]:null)&&m.push(v)}else m.push(v)}}return m}if(h.isPlainObject(a))return[a];c=i.filter(a).map(function(a,b){return{row:b._DT_CellIndex.row,column:b._DT_CellIndex.column}}).toArray();if(c.length||
+!a.nodeName)return c;s=h(a).closest("*[data-dt-row]");return s.length?[{row:s.data("dt-row"),column:s.data("dt-column")}]:[]},b,e)});var d=this.columns(b,c),e=this.rows(a,c),f,g,j,i,n,l=this.iterator("table",function(a,b){f=[];g=0;for(j=e[b].length;g<j;g++){i=0;for(n=d[b].length;i<n;i++)f.push({row:e[b][g],column:d[b][i]})}return f},1);h.extend(l.selector,{cols:b,rows:a,opts:c});return l});s("cells().nodes()","cell().node()",function(){return this.iterator("cell",function(a,b,c){return(a=a.aoData[b])&&
+a.anCells?a.anCells[c]:k},1)});p("cells().data()",function(){return this.iterator("cell",function(a,b,c){return B(a,b,c)},1)});s("cells().cache()","cell().cache()",function(a){a="search"===a?"_aFilterData":"_aSortData";return this.iterator("cell",function(b,c,d){return b.aoData[c][a][d]},1)});s("cells().render()","cell().render()",function(a){return this.iterator("cell",function(b,c,d){return B(b,c,d,a)},1)});s("cells().indexes()","cell().index()",function(){return this.iterator("cell",function(a,
+b,c){return{row:b,column:c,columnVisible:$(a,c)}},1)});s("cells().invalidate()","cell().invalidate()",function(a){return this.iterator("cell",function(b,c,d){ca(b,c,a,d)})});p("cell()",function(a,b,c){return bb(this.cells(a,b,c))});p("cell().data()",function(a){var b=this.context,c=this[0];if(a===k)return b.length&&c.length?B(b[0],c[0].row,c[0].column):k;jb(b[0],c[0].row,c[0].column,a);ca(b[0],c[0].row,"data",c[0].column);return this});p("order()",function(a,b){var c=this.context;if(a===k)return 0!==
+c.length?c[0].aaSorting:k;"number"===typeof a?a=[[a,b]]:a.length&&!h.isArray(a[0])&&(a=Array.prototype.slice.call(arguments));return this.iterator("table",function(b){b.aaSorting=a.slice()})});p("order.listener()",function(a,b,c){return this.iterator("table",function(d){Ma(d,a,b,c)})});p("order.fixed()",function(a){if(!a){var b=this.context,b=b.length?b[0].aaSortingFixed:k;return h.isArray(b)?{pre:b}:b}return this.iterator("table",function(b){b.aaSortingFixed=h.extend(!0,{},a)})});p(["columns().order()",
+"column().order()"],function(a){var b=this;return this.iterator("table",function(c,d){var e=[];h.each(b[d],function(b,c){e.push([c,a])});c.aaSorting=e})});p("search()",function(a,b,c,d){var e=this.context;return a===k?0!==e.length?e[0].oPreviousSearch.sSearch:k:this.iterator("table",function(e){e.oFeatures.bFilter&&fa(e,h.extend({},e.oPreviousSearch,{sSearch:a+"",bRegex:null===b?!1:b,bSmart:null===c?!0:c,bCaseInsensitive:null===d?!0:d}),1)})});s("columns().search()","column().search()",function(a,
+b,c,d){return this.iterator("column",function(e,f){var g=e.aoPreSearchCols;if(a===k)return g[f].sSearch;e.oFeatures.bFilter&&(h.extend(g[f],{sSearch:a+"",bRegex:null===b?!1:b,bSmart:null===c?!0:c,bCaseInsensitive:null===d?!0:d}),fa(e,e.oPreviousSearch,1))})});p("state()",function(){return this.context.length?this.context[0].oSavedState:null});p("state.clear()",function(){return this.iterator("table",function(a){a.fnStateSaveCallback.call(a.oInstance,a,{})})});p("state.loaded()",function(){return this.context.length?
+this.context[0].oLoadedState:null});p("state.save()",function(){return this.iterator("table",function(a){wa(a)})});m.versionCheck=m.fnVersionCheck=function(a){for(var b=m.version.split("."),a=a.split("."),c,d,e=0,f=a.length;e<f;e++)if(c=parseInt(b[e],10)||0,d=parseInt(a[e],10)||0,c!==d)return c>d;return!0};m.isDataTable=m.fnIsDataTable=function(a){var b=h(a).get(0),c=!1;h.each(m.settings,function(a,e){var f=e.nScrollHead?h("table",e.nScrollHead)[0]:null,g=e.nScrollFoot?h("table",e.nScrollFoot)[0]:
+null;if(e.nTable===b||f===b||g===b)c=!0});return c};m.tables=m.fnTables=function(a){var b=!1;h.isPlainObject(a)&&(b=a.api,a=a.visible);var c=h.map(m.settings,function(b){if(!a||a&&h(b.nTable).is(":visible"))return b.nTable});return b?new r(c):c};m.camelToHungarian=K;p("$()",function(a,b){var c=this.rows(b).nodes(),c=h(c);return h([].concat(c.filter(a).toArray(),c.find(a).toArray()))});h.each(["on","one","off"],function(a,b){p(b+"()",function(){var a=Array.prototype.slice.call(arguments);a[0].match(/\.dt\b/)||
+(a[0]+=".dt");var d=h(this.tables().nodes());d[b].apply(d,a);return this})});p("clear()",function(){return this.iterator("table",function(a){na(a)})});p("settings()",function(){return new r(this.context,this.context)});p("init()",function(){var a=this.context;return a.length?a[0].oInit:null});p("data()",function(){return this.iterator("table",function(a){return G(a.aoData,"_aData")}).flatten()});p("destroy()",function(a){a=a||!1;return this.iterator("table",function(b){var c=b.nTableWrapper.parentNode,
+d=b.oClasses,e=b.nTable,f=b.nTBody,g=b.nTHead,j=b.nTFoot,i=h(e),f=h(f),k=h(b.nTableWrapper),l=h.map(b.aoData,function(a){return a.nTr}),p;b.bDestroying=!0;u(b,"aoDestroyCallback","destroy",[b]);a||(new r(b)).columns().visible(!0);k.unbind(".DT").find(":not(tbody *)").unbind(".DT");h(D).unbind(".DT-"+b.sInstance);e!=g.parentNode&&(i.children("thead").detach(),i.append(g));j&&e!=j.parentNode&&(i.children("tfoot").detach(),i.append(j));b.aaSorting=[];b.aaSortingFixed=[];va(b);h(l).removeClass(b.asStripeClasses.join(" "));
+h("th, td",g).removeClass(d.sSortable+" "+d.sSortableAsc+" "+d.sSortableDesc+" "+d.sSortableNone);b.bJUI&&(h("th span."+d.sSortIcon+", td span."+d.sSortIcon,g).detach(),h("th, td",g).each(function(){var a=h("div."+d.sSortJUIWrapper,this);h(this).append(a.contents());a.detach()}));f.children().detach();f.append(l);g=a?"remove":"detach";i[g]();k[g]();!a&&c&&(c.insertBefore(e,b.nTableReinsertBefore),i.css("width",b.sDestroyWidth).removeClass(d.sTable),(p=b.asDestroyStripes.length)&&f.children().each(function(a){h(this).addClass(b.asDestroyStripes[a%
+p])}));c=h.inArray(b,m.settings);-1!==c&&m.settings.splice(c,1)})});h.each(["column","row","cell"],function(a,b){p(b+"s().every()",function(a){var d=this.selector.opts,e=this;return this.iterator(b,function(f,g,h,i,n){a.call(e[b](g,"cell"===b?h:d,"cell"===b?d:k),g,h,i,n)})})});p("i18n()",function(a,b,c){var d=this.context[0],a=Q(a)(d.oLanguage);a===k&&(a=b);c!==k&&h.isPlainObject(a)&&(a=a[c]!==k?a[c]:a._);return a.replace("%d",c)});m.version="1.10.12";m.settings=[];m.models={};m.models.oSearch={bCaseInsensitive:!0,
+sSearch:"",bRegex:!1,bSmart:!0};m.models.oRow={nTr:null,anCells:null,_aData:[],_aSortData:null,_aFilterData:null,_sFilterRow:null,_sRowStripe:"",src:null,idx:-1};m.models.oColumn={idx:null,aDataSort:null,asSorting:null,bSearchable:null,bSortable:null,bVisible:null,_sManualType:null,_bAttrSrc:!1,fnCreatedCell:null,fnGetData:null,fnSetData:null,mData:null,mRender:null,nTh:null,nTf:null,sClass:null,sContentPadding:null,sDefaultContent:null,sName:null,sSortDataType:"std",sSortingClass:null,sSortingClassJUI:null,
+sTitle:null,sType:null,sWidth:null,sWidthOrig:null};m.defaults={aaData:null,aaSorting:[[0,"asc"]],aaSortingFixed:[],ajax:null,aLengthMenu:[10,25,50,100],aoColumns:null,aoColumnDefs:null,aoSearchCols:[],asStripeClasses:null,bAutoWidth:!0,bDeferRender:!1,bDestroy:!1,bFilter:!0,bInfo:!0,bJQueryUI:!1,bLengthChange:!0,bPaginate:!0,bProcessing:!1,bRetrieve:!1,bScrollCollapse:!1,bServerSide:!1,bSort:!0,bSortMulti:!0,bSortCellsTop:!1,bSortClasses:!0,bStateSave:!1,fnCreatedRow:null,fnDrawCallback:null,fnFooterCallback:null,
+fnFormatNumber:function(a){return a.toString().replace(/\B(?=(\d{3})+(?!\d))/g,this.oLanguage.sThousands)},fnHeaderCallback:null,fnInfoCallback:null,fnInitComplete:null,fnPreDrawCallback:null,fnRowCallback:null,fnServerData:null,fnServerParams:null,fnStateLoadCallback:function(a){try{return JSON.parse((-1===a.iStateDuration?sessionStorage:localStorage).getItem("DataTables_"+a.sInstance+"_"+location.pathname))}catch(b){}},fnStateLoadParams:null,fnStateLoaded:null,fnStateSaveCallback:function(a,b){try{(-1===
+a.iStateDuration?sessionStorage:localStorage).setItem("DataTables_"+a.sInstance+"_"+location.pathname,JSON.stringify(b))}catch(c){}},fnStateSaveParams:null,iStateDuration:7200,iDeferLoading:null,iDisplayLength:10,iDisplayStart:0,iTabIndex:0,oClasses:{},oLanguage:{oAria:{sSortAscending:": activate to sort column ascending",sSortDescending:": activate to sort column descending"},oPaginate:{sFirst:"First",sLast:"Last",sNext:"Next",sPrevious:"Previous"},sEmptyTable:"No data available in table",sInfo:"Showing _START_ to _END_ of _TOTAL_ entries",
+sInfoEmpty:"Showing 0 to 0 of 0 entries",sInfoFiltered:"(filtered from _MAX_ total entries)",sInfoPostFix:"",sDecimal:"",sThousands:",",sLengthMenu:"Show _MENU_ entries",sLoadingRecords:"Loading...",sProcessing:"Processing...",sSearch:"Search Metadata:",sSearchPlaceholder:"",sUrl:"",sZeroRecords:"No matching records found"},oSearch:h.extend({},m.models.oSearch),sAjaxDataProp:"data",sAjaxSource:null,sDom:"lfrtip",searchDelay:null,sPaginationType:"simple_numbers",sScrollX:"",sScrollXInner:"",sScrollY:"",sServerMethod:"GET",
+renderer:null,rowId:"DT_RowId"};X(m.defaults);m.defaults.column={aDataSort:null,iDataSort:-1,asSorting:["asc","desc"],bSearchable:!0,bSortable:!0,bVisible:!0,fnCreatedCell:null,mData:null,mRender:null,sCellType:"td",sClass:"",sContentPadding:"",sDefaultContent:null,sName:"",sSortDataType:"std",sTitle:null,sType:null,sWidth:null};X(m.defaults.column);m.models.oSettings={oFeatures:{bAutoWidth:null,bDeferRender:null,bFilter:null,bInfo:null,bLengthChange:null,bPaginate:null,bProcessing:null,bServerSide:null,
+bSort:null,bSortMulti:null,bSortClasses:null,bStateSave:null},oScroll:{bCollapse:null,iBarWidth:0,sX:null,sXInner:null,sY:null},oLanguage:{fnInfoCallback:null},oBrowser:{bScrollOversize:!1,bScrollbarLeft:!1,bBounding:!1,barWidth:0},ajax:null,aanFeatures:[],aoData:[],aiDisplay:[],aiDisplayMaster:[],aIds:{},aoColumns:[],aoHeader:[],aoFooter:[],oPreviousSearch:{},aoPreSearchCols:[],aaSorting:null,aaSortingFixed:[],asStripeClasses:null,asDestroyStripes:[],sDestroyWidth:0,aoRowCallback:[],aoHeaderCallback:[],
+aoFooterCallback:[],aoDrawCallback:[],aoRowCreatedCallback:[],aoPreDrawCallback:[],aoInitComplete:[],aoStateSaveParams:[],aoStateLoadParams:[],aoStateLoaded:[],sTableId:"",nTable:null,nTHead:null,nTFoot:null,nTBody:null,nTableWrapper:null,bDeferLoading:!1,bInitialised:!1,aoOpenRows:[],sDom:null,searchDelay:null,sPaginationType:"two_button",iStateDuration:0,aoStateSave:[],aoStateLoad:[],oSavedState:null,oLoadedState:null,sAjaxSource:null,sAjaxDataProp:null,bAjaxDataGet:!0,jqXHR:null,json:k,oAjaxData:k,
+fnServerData:null,aoServerParams:[],sServerMethod:null,fnFormatNumber:null,aLengthMenu:null,iDraw:0,bDrawing:!1,iDrawError:-1,_iDisplayLength:10,_iDisplayStart:0,_iRecordsTotal:0,_iRecordsDisplay:0,bJUI:null,oClasses:{},bFiltered:!1,bSorted:!1,bSortCellsTop:null,oInit:null,aoDestroyCallback:[],fnRecordsTotal:function(){return"ssp"==y(this)?1*this._iRecordsTotal:this.aiDisplayMaster.length},fnRecordsDisplay:function(){return"ssp"==y(this)?1*this._iRecordsDisplay:this.aiDisplay.length},fnDisplayEnd:function(){var a=
+this._iDisplayLength,b=this._iDisplayStart,c=b+a,d=this.aiDisplay.length,e=this.oFeatures,f=e.bPaginate;return e.bServerSide?!1===f||-1===a?b+d:Math.min(b+a,this._iRecordsDisplay):!f||c>d||-1===a?d:c},oInstance:null,sInstance:null,iTabIndex:0,nScrollHead:null,nScrollFoot:null,aLastSort:[],oPlugins:{},rowIdFn:null,rowId:null};m.ext=v={buttons:{},classes:{},builder:"-source-",errMode:"alert",feature:[],search:[],selector:{cell:[],column:[],row:[]},internal:{},legacy:{ajax:null},pager:{},renderer:{pageButton:{},
+header:{}},order:{},type:{detect:[],search:{},order:{}},_unique:0,fnVersionCheck:m.fnVersionCheck,iApiIndex:0,oJUIClasses:{},sVersion:m.version};h.extend(v,{afnFiltering:v.search,aTypes:v.type.detect,ofnSearch:v.type.search,oSort:v.type.order,afnSortData:v.order,aoFeatures:v.feature,oApi:v.internal,oStdClasses:v.classes,oPagination:v.pager});h.extend(m.ext.classes,{sTable:"dataTable",sNoFooter:"no-footer",sPageButton:"paginate_button",sPageButtonActive:"current",sPageButtonDisabled:"disabled",sStripeOdd:"odd",
+sStripeEven:"even",sRowEmpty:"dataTables_empty",sWrapper:"dataTables_wrapper",sFilter:"dataTables_filter",sInfo:"dataTables_info",sPaging:"dataTables_paginate paging_",sLength:"dataTables_length",sProcessing:"dataTables_processing",sSortAsc:"sorting_asc",sSortDesc:"sorting_desc",sSortable:"sorting",sSortableAsc:"sorting_asc_disabled",sSortableDesc:"sorting_desc_disabled",sSortableNone:"sorting_disabled",sSortColumn:"sorting_",sFilterInput:"",sLengthSelect:"",sScrollWrapper:"dataTables_scroll",sScrollHead:"dataTables_scrollHead",
+sScrollHeadInner:"dataTables_scrollHeadInner",sScrollBody:"dataTables_scrollBody",sScrollFoot:"dataTables_scrollFoot",sScrollFootInner:"dataTables_scrollFootInner",sHeaderTH:"",sFooterTH:"",sSortJUIAsc:"",sSortJUIDesc:"",sSortJUI:"",sSortJUIAscAllowed:"",sSortJUIDescAllowed:"",sSortJUIWrapper:"",sSortIcon:"",sJUIHeader:"",sJUIFooter:""});var Ca="",Ca="",H=Ca+"ui-state-default",ia=Ca+"css_right ui-icon ui-icon-",Xb=Ca+"fg-toolbar ui-toolbar ui-widget-header ui-helper-clearfix";h.extend(m.ext.oJUIClasses,
+m.ext.classes,{sPageButton:"fg-button ui-button "+H,sPageButtonActive:"ui-state-disabled",sPageButtonDisabled:"ui-state-disabled",sPaging:"dataTables_paginate fg-buttonset ui-buttonset fg-buttonset-multi ui-buttonset-multi paging_",sSortAsc:H+" sorting_asc",sSortDesc:H+" sorting_desc",sSortable:H+" sorting",sSortableAsc:H+" sorting_asc_disabled",sSortableDesc:H+" sorting_desc_disabled",sSortableNone:H+" sorting_disabled",sSortJUIAsc:ia+"triangle-1-n",sSortJUIDesc:ia+"triangle-1-s",sSortJUI:ia+"carat-2-n-s",
+sSortJUIAscAllowed:ia+"carat-1-n",sSortJUIDescAllowed:ia+"carat-1-s",sSortJUIWrapper:"DataTables_sort_wrapper",sSortIcon:"DataTables_sort_icon",sScrollHead:"dataTables_scrollHead "+H,sScrollFoot:"dataTables_scrollFoot "+H,sHeaderTH:H,sFooterTH:H,sJUIHeader:Xb+" ui-corner-tl ui-corner-tr",sJUIFooter:Xb+" ui-corner-bl ui-corner-br"});var Mb=m.ext.pager;h.extend(Mb,{simple:function(){return["previous","next"]},full:function(){return["first","previous","next","last"]},numbers:function(a,b){return[ya(a,
+b)]},simple_numbers:function(a,b){return["previous",ya(a,b),"next"]},full_numbers:function(a,b){return["first","previous",ya(a,b),"next","last"]},_numbers:ya,numbers_length:7});h.extend(!0,m.ext.renderer,{pageButton:{_:function(a,b,c,d,e,f){var g=a.oClasses,j=a.oLanguage.oPaginate,i=a.oLanguage.oAria.paginate||{},k,l,m=0,p=function(b,d){var o,r,u,s,v=function(b){Ta(a,b.data.action,true)};o=0;for(r=d.length;o<r;o++){s=d[o];if(h.isArray(s)){u=h("<"+(s.DT_el||"div")+"/>").appendTo(b);p(u,s)}else{k=null;
+l="";switch(s){case "ellipsis":b.append('<span class="ellipsis">&#x2026;</span>');break;case "first":k=j.sFirst;l=s+(e>0?"":" "+g.sPageButtonDisabled);break;case "previous":k=j.sPrevious;l=s+(e>0?"":" "+g.sPageButtonDisabled);break;case "next":k=j.sNext;l=s+(e<f-1?"":" "+g.sPageButtonDisabled);break;case "last":k=j.sLast;l=s+(e<f-1?"":" "+g.sPageButtonDisabled);break;default:k=s+1;l=e===s?g.sPageButtonActive:""}if(k!==null){u=h("<a>",{"class":g.sPageButton+" "+l,"aria-controls":a.sTableId,"aria-label":i[s],
+"data-dt-idx":m,tabindex:a.iTabIndex,id:c===0&&typeof s==="string"?a.sTableId+"_"+s:null}).html(k).appendTo(b);Wa(u,{action:s},v);m++}}}},r;try{r=h(b).find(I.activeElement).data("dt-idx")}catch(o){}p(h(b).empty(),d);r&&h(b).find("[data-dt-idx="+r+"]").focus()}}});h.extend(m.ext.type.detect,[function(a,b){var c=b.oLanguage.sDecimal;return Za(a,c)?"num"+c:null},function(a){if(a&&!(a instanceof Date)&&(!ac.test(a)||!bc.test(a)))return null;var b=Date.parse(a);return null!==b&&!isNaN(b)||M(a)?"date":
+null},function(a,b){var c=b.oLanguage.sDecimal;return Za(a,c,!0)?"num-fmt"+c:null},function(a,b){var c=b.oLanguage.sDecimal;return Rb(a,c)?"html-num"+c:null},function(a,b){var c=b.oLanguage.sDecimal;return Rb(a,c,!0)?"html-num-fmt"+c:null},function(a){return M(a)||"string"===typeof a&&-1!==a.indexOf("<")?"html":null}]);h.extend(m.ext.type.search,{html:function(a){return M(a)?a:"string"===typeof a?a.replace(Ob," ").replace(Aa,""):""},string:function(a){return M(a)?a:"string"===typeof a?a.replace(Ob,
+" "):a}});var za=function(a,b,c,d){if(0!==a&&(!a||"-"===a))return-Infinity;b&&(a=Qb(a,b));a.replace&&(c&&(a=a.replace(c,"")),d&&(a=a.replace(d,"")));return 1*a};h.extend(v.type.order,{"date-pre":function(a){return Date.parse(a)||0},"html-pre":function(a){return M(a)?"":a.replace?a.replace(/<.*?>/g,"").toLowerCase():a+""},"string-pre":function(a){return M(a)?"":"string"===typeof a?a.toLowerCase():!a.toString?"":a.toString()},"string-asc":function(a,b){return a<b?-1:a>b?1:0},"string-desc":function(a,
+b){return a<b?1:a>b?-1:0}});db("");h.extend(!0,m.ext.renderer,{header:{_:function(a,b,c,d){h(a.nTable).on("order.dt.DT",function(e,f,g,h){if(a===f){e=c.idx;b.removeClass(c.sSortingClass+" "+d.sSortAsc+" "+d.sSortDesc).addClass(h[e]=="asc"?d.sSortAsc:h[e]=="desc"?d.sSortDesc:c.sSortingClass)}})},jqueryui:function(a,b,c,d){h("<div/>").addClass(d.sSortJUIWrapper).append(b.contents()).append(h("<span/>").addClass(d.sSortIcon+" "+c.sSortingClassJUI)).appendTo(b);h(a.nTable).on("order.dt.DT",function(e,
+f,g,h){if(a===f){e=c.idx;b.removeClass(d.sSortAsc+" "+d.sSortDesc).addClass(h[e]=="asc"?d.sSortAsc:h[e]=="desc"?d.sSortDesc:c.sSortingClass);b.find("span."+d.sSortIcon).removeClass(d.sSortJUIAsc+" "+d.sSortJUIDesc+" "+d.sSortJUI+" "+d.sSortJUIAscAllowed+" "+d.sSortJUIDescAllowed).addClass(h[e]=="asc"?d.sSortJUIAsc:h[e]=="desc"?d.sSortJUIDesc:c.sSortingClassJUI)}})}}});var Yb=function(a){return"string"===typeof a?a.replace(/</g,"&lt;").replace(/>/g,"&gt;").replace(/"/g,"&quot;"):a};m.render={number:function(a,
+b,c,d,e){return{display:function(f){if("number"!==typeof f&&"string"!==typeof f)return f;var g=0>f?"-":"",h=parseFloat(f);if(isNaN(h))return Yb(f);f=Math.abs(h);h=parseInt(f,10);f=c?b+(f-h).toFixed(c).substring(2):"";return g+(d||"")+h.toString().replace(/\B(?=(\d{3})+(?!\d))/g,a)+f+(e||"")}}},text:function(){return{display:Yb}}};h.extend(m.ext.internal,{_fnExternApiFunc:Nb,_fnBuildAjax:ra,_fnAjaxUpdate:lb,_fnAjaxParameters:ub,_fnAjaxUpdateDraw:vb,_fnAjaxDataSrc:sa,_fnAddColumn:Ea,_fnColumnOptions:ja,
+_fnAdjustColumnSizing:Y,_fnVisibleToColumnIndex:Z,_fnColumnIndexToVisible:$,_fnVisbleColumns:aa,_fnGetColumns:la,_fnColumnTypes:Ga,_fnApplyColumnDefs:ib,_fnHungarianMap:X,_fnCamelToHungarian:K,_fnLanguageCompat:Da,_fnBrowserDetect:gb,_fnAddData:N,_fnAddTr:ma,_fnNodeToDataIndex:function(a,b){return b._DT_RowIndex!==k?b._DT_RowIndex:null},_fnNodeToColumnIndex:function(a,b,c){return h.inArray(c,a.aoData[b].anCells)},_fnGetCellData:B,_fnSetCellData:jb,_fnSplitObjNotation:Ja,_fnGetObjectDataFn:Q,_fnSetObjectDataFn:R,
+_fnGetDataMaster:Ka,_fnClearTable:na,_fnDeleteIndex:oa,_fnInvalidate:ca,_fnGetRowElements:Ia,_fnCreateTr:Ha,_fnBuildHead:kb,_fnDrawHead:ea,_fnDraw:O,_fnReDraw:T,_fnAddOptionsHtml:nb,_fnDetectHeader:da,_fnGetUniqueThs:qa,_fnFeatureHtmlFilter:pb,_fnFilterComplete:fa,_fnFilterCustom:yb,_fnFilterColumn:xb,_fnFilter:wb,_fnFilterCreateSearch:Pa,_fnEscapeRegex:Qa,_fnFilterData:zb,_fnFeatureHtmlInfo:sb,_fnUpdateInfo:Cb,_fnInfoMacros:Db,_fnInitialise:ga,_fnInitComplete:ta,_fnLengthChange:Ra,_fnFeatureHtmlLength:ob,
+_fnFeatureHtmlPaginate:tb,_fnPageChange:Ta,_fnFeatureHtmlProcessing:qb,_fnProcessingDisplay:C,_fnFeatureHtmlTable:rb,_fnScrollDraw:ka,_fnApplyToChildren:J,_fnCalculateColumnWidths:Fa,_fnThrottle:Oa,_fnConvertToWidth:Fb,_fnGetWidestNode:Gb,_fnGetMaxLenString:Hb,_fnStringToCss:x,_fnSortFlatten:V,_fnSort:mb,_fnSortAria:Jb,_fnSortListener:Va,_fnSortAttachListener:Ma,_fnSortingClasses:va,_fnSortData:Ib,_fnSaveState:wa,_fnLoadState:Kb,_fnSettingsFromNode:xa,_fnLog:L,_fnMap:E,_fnBindAction:Wa,_fnCallbackReg:z,
+_fnCallbackFire:u,_fnLengthOverflow:Sa,_fnRenderer:Na,_fnDataSource:y,_fnRowAttributes:La,_fnCalculateEnd:function(){}});h.fn.dataTable=m;m.$=h;h.fn.dataTableSettings=m.settings;h.fn.dataTableExt=m.ext;h.fn.DataTable=function(a){return h(this).dataTable(a).api()};h.each(m,function(a,b){h.fn.DataTable[a]=b});return h.fn.dataTable});

static/master_list.js

@@ -0,0 +1,347 @@
+//////////////////////////////////
+// Centillion Master List
+// Javascript Functions 
+//
+// This file contains javascript functions used by
+// the master_list page of centillion. The master
+// list page uses the search engine as an API to
+// get a list of all documents by type.
+
+
+///////////////////////////////////
+// Process get parameters
+//
+// When the document is loaded, parse the GET params
+// from the URL (everything after the ?).
+//
+// If the "doctype" parameter is in the URL, use it to
+// determine which panel to open automatically.
+
+var initGdocTable = false;
+var initIssuesTable = false;
+var initGhfilesTable = false;
+var initMarkdownTable = false;
+var initEmailthreadsTable = false;
+
+$(document).ready(function() {
+    var url_string = document.location.toString();
+    var url = new URL(url_string);
+    var d = url.searchParams.get("doctype");
+
+    if (d==='gdoc') {
+        load_gdoc_table();
+        var divList = $('div#collapseDrive').addClass('in');
+
+    } else if (d==='emailthread') {
+        load_emailthreads_table();
+        var divList = $('div#collapseThreads').addClass('in');
+
+    } else if (d==='issue') {
+        load_issue_table();
+        var divList = $('div#collapseIssues').addClass('in');
+
+    } else if (d==='ghfile') {
+        load_ghfile_table();
+        var divList = $('div#collapseFiles').addClass('in');
+
+    } else if (d==='markdown') {
+        load_markdown_table();
+        var divList = $('div#collapseMarkdown').addClass('in');
+
+    }
+});
+
+
+
+//////////////////////////////////
+// API-to-Table Functions
+//
+// These functions ask centillion for a list of all documents
+// of a given type, and load the results into an HTML table.
+//
+// The dataTable bootstrap plugin is used to make the tables
+// sortable, searchable, and slick.
+//
+// Sections:
+// ----------
+// Google Drive files
+// Github issues
+// Github files
+// Github markdown
+// Groups.io email threads
+
+// ------------------------
+// Google Drive
+
+function load_gdoc_table(){
+    if(!initGdocTable) {
+        var divList = $('div#collapseDrive').attr('class');
+        if (divList.indexOf('in') !== -1) {
+            console.log('Closing Google Drive master list');
+        } else { 
+            console.log('Opening Google Drive master list');
+
+            $.getJSON("/list/gdoc", function(result){
+
+                var r = new Array(), j = -1, size=result.length;
+                r[++j] = '<thead>'
+                r[++j] = '<tr class="header-row">';
+                r[++j] = '<th width="40%">File Name</th>';
+                r[++j] = '<th width="15%">Owner</th>';
+                r[++j] = '<th width="15%">Type</th>';
+                r[++j] = '<th width="15%">Created</th>';
+                r[++j] = '<th width="15%">Modified</th>';
+                r[++j] = '</tr>';
+                r[++j] = '</thead>'
+                r[++j] = '<tbody>'
+                for (var i=0; i<size; i++){
+                    r[++j] = '<tr><td>';
+                    r[++j] = '<a href="' + result[i]['url'] + '" target="_blank">'
+                    r[++j] = result[i]['title'];
+                    r[++j] = '</a>'
+                    r[++j] = '</td><td>';
+                    r[++j] = result[i]['owner_name'];
+                    r[++j] = '</td><td>';
+                    r[++j] = result[i]['mimetype'];
+                    r[++j] = '</td><td>';
+                    r[++j] = result[i]['created_time'];
+                    r[++j] = '</td><td>';
+                    r[++j] = result[i]['modified_time'];
+                    r[++j] = '</td></tr>';
+                }
+                r[++j] = '</tbody>'
+
+                // Construct names of id tags
+                var doctype = 'gdocs';
+                var idlabel = '#' + doctype + '-master-list';
+                var filtlabel = idlabel + '_filter';
+
+                // Initialize the DataTable
+                $(idlabel).html(r.join(''));
+                $(idlabel).DataTable({
+                    responsive: true,
+                    lengthMenu: [50,100,250,500]
+                });
+
+                initGdocTable = true
+            });
+            console.log('Finished loading Google Drive master list');
+        }
+    }
+}
+
+// ------------------------
+// Github issues
+
+function load_issue_table(){
+    if(!initIssuesTable) {
+        var divList = $('div#collapseIssues').attr('class');
+        if (divList.indexOf('in') !== -1) {
+            console.log('Closing Github issues master list');
+        } else { 
+            console.log('Opening Github issues master list');
+
+            $.getJSON("/list/issue", function(result){
+                var r = new Array(), j = -1, size=result.length;
+                r[++j] = '<thead>'
+                r[++j] = '<tr class="header-row">';
+                r[++j] = '<th width="50%">Issue Name</th>';
+                r[++j] = '<th width="15%">Repository</th>';
+                r[++j] = '<th width="15%">Created</th>';
+                r[++j] = '<th width="15%">Modified</th>';
+                r[++j] = '</tr>';
+                r[++j] = '</thead>'
+                r[++j] = '<tbody>'
+                for (var i=0; i<size; i++){
+                    r[++j] ='<tr><td>';
+                    r[++j] = '<a href="' + result[i]['url'] + '" target="_blank">'
+                    r[++j] = result[i]['title'];
+                    r[++j] = '</a>'
+                    r[++j] = '</td><td>';
+                    r[++j] = '<a href="' + result[i]['repo_url'] + '" target="_blank">'
+                    r[++j] = result[i]['repo_name'];
+                    r[++j] = '</a>'
+                    r[++j] = '</td><td>';
+                    r[++j] = result[i]['created_time'];
+                    r[++j] = '</td><td>';
+                    r[++j] = result[i]['modified_time'];
+                    r[++j] = '</td></tr>';
+                }
+                r[++j] = '</tbody>'
+
+                // Construct names of id tags
+                var doctype = 'issues';
+                var idlabel = '#' + doctype + '-master-list';
+                var filtlabel = idlabel + '_filter';
+
+                // Initialize the DataTable
+                $(idlabel).html(r.join(''));
+                $(idlabel).DataTable({
+                    responsive: true,
+                    lengthMenu: [50,100,250,500]
+                });
+
+                initIssuesTable = true;
+            });
+            console.log('Finished loading Github issues master list');
+        }
+    }
+}
+
+// ------------------------
+// Github files
+
+function load_ghfile_table(){
+    if(!initGhfilesTable) {
+        var divList = $('div#collapseFiles').attr('class');
+        if (divList.indexOf('in') !== -1) {
+            console.log('Closing Github files master list');
+        } else { 
+            console.log('Opening Github files master list');
+
+            $.getJSON("/list/ghfile", function(result){
+                console.log("-----------");
+                console.log(result);
+                var r = new Array(), j = -1, size=result.length;
+                r[++j] = '<thead>'
+                r[++j] = '<tr class="header-row">';
+                r[++j] = '<th width="70%">File Name</th>';
+                r[++j] = '<th width="30%">Repository</th>';
+                r[++j] = '</tr>';
+                r[++j] = '</thead>'
+                r[++j] = '<tbody>'
+                for (var i=0; i<size; i++){
+                    r[++j] ='<tr><td>';
+                    r[++j] = '<a href="' + result[i]['url'] + '" target="_blank">'
+                    r[++j] = result[i]['title'];
+                    r[++j] = '</a>'
+                    r[++j] = '</td><td>';
+                    r[++j] = '<a href="' + result[i]['repo_url'] + '" target="_blank">'
+                    r[++j] = result[i]['repo_name'];
+                    r[++j] = '</a>'
+                    r[++j] = '</td></tr>';
+                }
+                r[++j] = '</tbody>'
+
+                // Construct names of id tags
+                var doctype = 'ghfiles';
+                var idlabel = '#' + doctype + '-master-list';
+                var filtlabel = idlabel + '_filter';
+
+                // Initialize the DataTable
+                $(idlabel).html(r.join(''));
+                $(idlabel).DataTable({
+                    responsive: true,
+                    lengthMenu: [50,100,250,500]
+                });
+
+                initGhfilesTable = true;
+            });
+            console.log('Finished loading Github file list');
+        }
+    }
+}
+
+// ------------------------
+// Github Markdown
+
+function load_markdown_table(){
+    if(!initMarkdownTable) { 
+        var divList = $('div#collapseMarkdown').attr('class');
+        if (divList.indexOf('in') !== -1) {
+            console.log('Closing Github markdown master list');
+        } else { 
+            console.log('Opening Github markdown master list');
+
+            $.getJSON("/list/markdown", function(result){
+                var r = new Array(), j = -1, size=result.length;
+                r[++j] = '<thead>'
+                r[++j] = '<tr class="header-row">';
+                r[++j] = '<th width="70%">Markdown File Name</th>';
+                r[++j] = '<th width="30%">Repository</th>';
+                r[++j] = '</tr>';
+                r[++j] = '</thead>'
+                r[++j] = '<tbody>'
+                for (var i=0; i<size; i++){
+                    r[++j] ='<tr><td>';
+                    r[++j] = '<a href="' + result[i]['url'] + '" target="_blank">'
+                    r[++j] = result[i]['title'];
+                    r[++j] = '</a>'
+                    r[++j] = '</td><td>';
+                    r[++j] = '<a href="' + result[i]['repo_url'] + '" target="_blank">'
+                    r[++j] = result[i]['repo_name'];
+                    r[++j] = '</a>'
+                    r[++j] = '</td></tr>';
+                }
+                r[++j] = '</tbody>'
+
+                // Construct names of id tags
+                var doctype = 'markdown';
+                var idlabel = '#' + doctype + '-master-list';
+                var filtlabel = idlabel + '_filter';
+
+                // Initialize the DataTable
+                $(idlabel).html(r.join(''));
+                $(idlabel).DataTable({
+                    responsive: true,
+                    lengthMenu: [50,100,250,500]
+                });
+
+                initMarkdownTable = true;
+            });
+            console.log('Finished loading Markdown list');
+        }
+    }
+}
+
+
+// ------------------------
+// Groups.io Email Threads
+
+function load_emailthreads_table(){
+    if(!initEmailthreadsTable) { 
+        var divList = $('div#collapseThreads').attr('class');
+        if (divList.indexOf('in') !== -1) {
+            console.log('Closing Groups.io email threads master list');
+        } else { 
+            console.log('Opening Groups.io email threads master list');
+    
+            $.getJSON("/list/emailthread", function(result){
+                var r = new Array(), j = -1, size=result.length;
+                r[++j] = '<thead>'
+                r[++j] = '<tr class="header-row">';
+                r[++j] = '<th width="70%">Topic</th>';
+                r[++j] = '<th width="30%">Started By</th>';
+                r[++j] = '</tr>';
+                r[++j] = '</thead>'
+                r[++j] = '<tbody>'
+                for (var i=0; i<size; i++){
+                    r[++j] ='<tr><td>';
+                    r[++j] = '<a href="' + result[i]['url'] + '" target="_blank">'
+                    r[++j] = result[i]['title'];
+                    r[++j] = '</a>'
+                    r[++j] = '</td><td>';
+                    r[++j] = result[i]['owner_name'];
+                    r[++j] = '</td></tr>';
+                }
+                r[++j] = '</tbody>'
+
+                // Construct names of id tags
+                var doctype = 'emailthreads';
+                var idlabel = '#' + doctype + '-master-list';
+                var filtlabel = idlabel + '_filter';
+
+                // Initialize the DataTable
+                $(idlabel).html(r.join(''));
+                $(idlabel).DataTable({
+                    responsive: true,
+                    lengthMenu: [50,100,250,500]
+                });
+
+                initEmailthreadsTable = true;
+            });
+            console.log('Finished loading Groups.io email threads list');
+        }
+    }
+}
+

static/search_list.js

@@ -0,0 +1,39 @@
+//////////////////////////////////
+// Centillion Search Results Listing
+// Javascript Functions 
+//
+// This file contains javascript functions used by
+// the search results page centillion.
+
+
+
+//////////////////////////////////
+// Results-to-DataTable Functions
+//
+// These functions post-process the table of search results
+// and make it into a dataTable.
+//
+// The dataTable bootstrap plugin is used to make the tables
+// sortable, searchable, and slick.
+
+
+$(document).ready(function() {
+    // Construct names of id tags
+    table_id = "#search-results";
+
+    // Initialize the DataTable
+    $(table_id).DataTable({
+        responsive: true,
+        searching: false,
+        order: [[0,'desc']],
+        aoColumnDefs: [
+            { bSortable: false,
+              aTargets : [2]
+            }
+        ],
+        lengthMenu: [50,100,250,500]
+    });
+
+    console.log('Finished loading search results list');
+});
+

static/style.css

@@ -1,3 +1,91 @@
+#modal-too-long {
+    visibility: hidden;
+}
+
+/* feedback smileys */
+#modal-feedback-smile-icon,
+#modal-feedback-frown-icon {
+    padding-left: 100px;
+    padding-right: 100px;
+    padding-top: 20px;
+    padding-bottom: 20px;
+}
+
+div.smile-active {
+    background-color: #2b2;
+}
+i.smile-active {
+    color: #fff;
+}
+div.frown-active {
+    background-color: #b22;
+}
+i.frown-active {
+    color: #fff;
+}
+
+/* feedback text area */
+#modal-feedback-textarea {
+    width: 100%;
+}
+
+/* feedback buttons */
+button.close {
+    font-size: 35px;
+}
+button#submit-feedback-btn {
+    width: 250px;
+}
+
+button#feedback:hover {
+    opacity: 1.0;
+    filter: alpha(opacity=100); /* For IE8 and earlier */
+}
+
+button#feedback {
+    opacity: 0.5;
+    filter: alpha(opacity=50); /* For IE8 and earlier */
+
+    width: 180px;
+    height: 50px;
+    position: fixed; 
+    z-index: 999;
+    right: 120px;
+    bottom: 10px;
+}
+
+
+/* search results table */
+td#search-results-score-col,
+td#search-results-type-col {
+    width: 100px;
+}
+
+div.container {
+    width: 90%;
+}
+
+/* control panel button width */
+.btn-reindex-type, .btn-reindex-all {
+    width: 350px;
+}
+
+/* landing page github button */
+#github-button {
+    display:inline-block;
+    font-size: 20px;
+    line-height: 40px;
+    padding: 10px 40px 10px 40px;
+    text-align: center;
+}
+
+/* search button */
+#the-big-one {
+    margin-top: 10px;
+    margin-bottom: 10px;
+}
+
+/* badges for number of docs indexed */
 span.badge {
     vertical-align: text-bottom;
 }
@@ -7,6 +95,10 @@ a.badgelinks, a.badgelinks:hover {
     text-decoration: none;
 }
 
+h2.masterlist-header a {
+    text-decoration: none;
+}
+
 div.list-group {
     border: 1px solid rgba(86,61,124,.2);
 }

templates/403.html

@@ -0,0 +1,26 @@
+{% extends "layout.html" %}
+{% set active_page = "403" %}
+{% block body %}
+
+<div class="container">
+    <div class="row">
+        <p>&nbsp;</p>
+    </div>
+    <div class="row">
+
+        <div class="col-xs-12">
+            <center>
+                <h2>403: Permission Denied
+                </h2>
+                <p>You don't seem to be a member of the proper Github team or organization.
+                Please contact <a href="mailto:dcppc-inbox@gmail.com">dcppc-inbox@gmail.com</a>
+                if you believe you are seeing this message in error.
+                </p>
+            </center>
+        </div>
+
+    </div>
+</div>
+
+{% endblock %}
+

templates/404.html

@@ -0,0 +1,22 @@
+{% extends "layout.html" %}
+{% set active_page = "404" %}
+{% block body %}
+
+<div class="container">
+    <div class="row">
+
+        <div class="col-xs-12">
+            <center>
+                <h2>404: Not Found
+                </h2>
+                <p>You have requested a page that does not exist.
+                Try clicking the centillion logo to go back home.
+                </p>
+            </center>
+        </div>
+
+    </div>
+</div>
+
+{% endblock %}
+

templates/banner.html

@@ -0,0 +1,25 @@
+<div class="container" id="banner-container">
+
+    {#
+    banner image
+    #}
+    <div class="row" id="banner-row">
+        <div class="col12sm" id="banner-col">
+            <center>
+                <a id="banner-a" href="{{ url_for('search')}}?query=&fields=">
+                    <img id="banner-img" src="{{ url_for('static', filename='centillion_white.png') }}">
+                </a>
+            </center>
+        </div>
+    </div>
+    {% if config['TAGLINE'] %}
+    <div class="row" id="tagline-row">
+        <div class="col12sm" id="tagline-col">
+            <center>
+                    <h2 id="tagline-tagline"> {{config['TAGLINE']}} </h2>
+            </center>
+        </div>
+    </div>
+    {% endif %}
+
+</div>

templates/controlpanel.html

@@ -1,52 +1,7 @@
 {% extends "layout.html" %}
+{% set active_page = "control_panel" %}
 {% block body %}
 
-{% with messages = get_flashed_messages() %}
-{% if messages %}
-<div class="container">
-    <div class="alert alert-success alert-dismissible">
-        <a href="#" class="close" data-dismiss="alert" aria-label="close">&times;</a>
-        <ul class=flashes>
-            {% for message in messages %}
-            <li>{{ message }}</li>
-            {% endfor %}
-        </ul>
-    </div>
-</div>
-{% endif %}
-{% endwith %}
-
-<div class="container">
-
-    <div class="row">
-        <div class="col-md-12">
-            <center>
-                <a href="{{ url_for('search')}}?query=&fields=">
-                <img src="{{ url_for('static', filename='centillion_white.png') }}">
-                </a>
-                {% if config['TAGLINE'] %}
-                    <h2><a href="{{ url_for('search')}}?query=&fields=">
-                        {{config['TAGLINE']}}
-                    </a></h2>
-                {% endif %}
-            </center>
-        </div>
-    </div>
-    
-    {% if config['zzzTAGLINE'] %}
-    <div class="row">
-        <div class="col12sm">
-            <center>
-                <h2><a href="{{ url_for('search')}}?query=&fields=">
-                    {{config['TAGLINE']}}
-                </a></h2>
-            </center>
-        </div>
-    </div>
-    {% endif %}
-
-</div>
-
 <hr />
 
 <div class="container">
@@ -66,9 +21,9 @@
                         <div class="col-md-12">
                             <p class="panel-text">Re-index <i>every</i> document in the
                             remote collection in the search index. <b>Warning: this operation may take a while.</b>
-                            <p/> <p>
+                            </p>
-                            <a href="{{ url_for('update_index') }}" class="btn btn-large btn-danger">Update Main Index</a>
+                            <p><a id="re-index-main" href="{{ url_for('update_index',run_which='all') }}" class="btn btn-large btn-danger btn-reindex-all">Update Main Index</a>
-                            <p/> 
+                            </p> 
                         </div>
                     </div>
                 </div>
@@ -76,22 +31,29 @@
             </div>
         </div>
 
-        {# update diff search index #}
+
+
+        {# update search index by type #}
         <div class="panel panel-danger">
             <div class="panel-heading">
                 <h3 class="panel-title">
-                    Update Diff Search Index
+                    Update Search Index by Type
                 </h3>
             </div>
             <div class="panel-body">
                 <div class="container-fluid">
                     <div class="row">
                         <div class="col-md-12">
-                            <p class="panel-text">Diff search index only re-indexes documents created after the last
+                            <p class="panel-text">Re-index individual document types in the search index.
-                            search index update. <b>Not currently implemented.</b>
+                            </p>
-                            <p/> <p>
+                            <p><a href="{{ url_for('update_index',run_which='gdocs') }}"        class="btn btn-large btn-danger btn-reindex-type">Update Google Drive Index</a>
-                            <a href="#" class="btn btn-large disabled btn-danger">Update Diff Index</a>
+                            </p>  
-                            <p/> 
+                            <p><a href="{{ url_for('update_index',run_which='ghfiles') }}"      class="btn btn-large btn-danger btn-reindex-type">Update Github Files Index</a>
+                            </p>  
+                            <p><a href="{{ url_for('update_index',run_which='issues') }}"       class="btn btn-large btn-danger btn-reindex-type">Update Github Issues Index</a>
+                            </p>  
+                            <p><a href="{{ url_for('update_index',run_which='emailthreads') }}" class="btn btn-large btn-danger btn-reindex-type">Update Groups.io Email Threads Index</a>
+                            </p> 
                         </div>
                     </div>
                 </div>

templates/flashed_messages.html

@@ -0,0 +1,14 @@
+<div id="messages">
+{% with messages = get_flashed_messages() %}
+    {% if messages %}
+        <div class="container" id="flashed-messages-container">
+            <div class="alert alert-success alert-dismissible fade in">
+                <a href="#" class="close" data-dismiss="alert" aria-label="close">&times;</a>
+                    {% for message in messages %}
+                        <p class="lead">{{ message }}</p>
+                    {% endfor %}
+            </div>
+        </div>
+    {% endif %}
+{% endwith %}
+</div>

templates/landing.html

@@ -0,0 +1,23 @@
+{% extends "layout.html" %}
+{% set active_page = "landing" %}
+{% block body %}
+<div class="container">
+    <div class="row">
+
+        <div class="col-xs-12">
+
+            <center>
+                <p class="lead">To begin searching the Data Commons, you must first sign in with Github.
+                </p>
+                <a href="/log_in" type="button" id="github-button" class="btn btn-default">
+                <i class="fa fa-github fa-2x"></i> Sign in with Github
+                </a>
+            </center>
+
+        </div>
+
+    </div>
+</div>
+{% endblock %}
+
+

templates/layout.html

@@ -1,12 +1,66 @@
 <!doctype html>
 <title>Centillion Search Engine</title>
+
 <link rel="stylesheet" type="text/css" href="{{ url_for('static', filename='style.css') }}">
 <link rel="stylesheet" type="text/css" href="{{ url_for('static', filename='github-markdown.css') }}">
 <link rel="stylesheet" type="text/css" href="{{ url_for('static', filename='bootstrap.min.css') }}">
+<link rel="stylesheet" type="text/css" href="{{ url_for('static', filename='font-awesome.min.css') }}">
 
 <script src="{{ url_for('static', filename='jquery.min.js') }}"></script>
 <script src="{{ url_for('static', filename='bootstrap.min.js') }}"></script>
+<script src="{{ url_for('static', filename='master_list.js') }}"></script>
+<script src="{{ url_for('static', filename='search_list.js') }}"></script>
+<script src="{{ url_for('static', filename='feedback.js') }}"></script>
+
+
+{# ########## dataTables plugin ############ #}
+
+<link rel="stylesheet" type="text/css" href="{{ url_for('static', filename='dataTables.bootstrap.min.css') }}">
+<link rel="stylesheet" type="text/css" href="{{ url_for('static', filename='dataTables.responsive.css') }}">
+
+<script src="{{ url_for('static', filename='jquery.dataTables.min.js') }}"></script>
+<script src="{{ url_for('static', filename='dataTables.bootstrap.min.js') }}"></script>
+<script src="{{ url_for('static', filename='dataTables.responsive.js') }}"></script>
+
+
+
+<div id="master-div">
+
+    {#
+    flashed messages
+    #}
+    {% include "flashed_messages.html" %}
+
+    {#
+    banner image
+    #}
+    {% include "banner.html" %}
+
+    {#
+    feedback modal
+    #}
+    {% include "modal.html" %}
 
-<div>
     {% block body %}{% endblock %}
+
 </div>
+
+{% if active_page=="search" or active_page=="master_list" %}
+
+    {# feedback button #}
+    <button id="feedback" type="button" 
+                          data-toggle="modal"
+                          data-target="#myModal"
+        class="btn btn-lg">Send Feedback</button>
+    
+    {# vertical spacing before the bottom, b/c of button #}
+    <div id="footer-whitespace" class="container">
+        <p>&nbsp;</p>
+        <p>&nbsp;</p>
+        <p>&nbsp;</p>
+    </div>
+
+{% endif %}
+
+<a id="github-corner" href="https://github.com/dcppc/centillion" class="github-corner" aria-label="View source on Github"><svg width="80" height="80" viewBox="0 0 250 250" style="fill:#151513; color:#fff; position: absolute; top: 0; border: 0; right: 0;" aria-hidden="true"><path d="M0,0 L115,115 L130,115 L142,142 L250,250 L250,0 Z"></path><path d="M128.3,109.0 C113.8,99.7 119.0,89.6 119.0,89.6 C122.0,82.7 120.5,78.6 120.5,78.6 C119.2,72.0 123.4,76.3 123.4,76.3 C127.3,80.9 125.5,87.3 125.5,87.3 C122.9,97.6 130.6,101.9 134.4,103.2" fill="currentColor" style="transform-origin: 130px 106px;" class="octo-arm"></path><path d="M115.0,115.0 C114.9,115.1 118.7,116.5 119.8,115.4 L133.7,101.6 C136.9,99.2 139.9,98.4 142.2,98.6 C133.8,88.0 127.5,74.4 143.8,58.0 C148.5,53.4 154.0,51.2 159.7,51.0 C160.3,49.4 163.2,43.6 171.4,40.1 C171.4,40.1 176.1,42.5 178.8,56.2 C183.1,58.6 187.2,61.8 190.9,65.4 C194.5,69.0 197.7,73.2 200.1,77.6 C213.8,80.2 216.3,84.9 216.3,84.9 C212.7,93.1 206.9,96.0 205.4,96.6 C205.1,102.4 203.0,107.8 198.3,112.5 C181.9,128.9 168.3,122.5 157.7,114.1 C157.9,116.9 156.7,120.9 152.7,124.9 L141.0,136.5 C139.8,137.7 141.6,141.9 141.8,141.8 Z" fill="currentColor" class="octo-body"></path></svg></a><style>.github-corner:hover .octo-arm{animation:octocat-wave 560ms ease-in-out}@keyframes octocat-wave{0%,100%{transform:rotate(0)}20%,60%{transform:rotate(-25deg)}40%,80%{transform:rotate(10deg)}}@media (max-width:500px){.github-corner:hover .octo-arm{animation:none}.github-corner .octo-arm{animation:octocat-wave 560ms ease-in-out}}</style>
+

templates/masterlist.html

@@ -0,0 +1,203 @@
+{% extends "layout.html" %}
+{% set active_page = "master_list" %}
+{% block body %}
+
+<hr />
+
+<div class="container">
+
+    <div class="row">
+
+    {#
+     # google drive files panel
+     #}
+    <div class="row">
+        <div class="panel">
+            <div class="panel-group" id="accordionDrive" role="tablist" aria-multiselectable="true">
+                <div class="panel panel-default">
+                    <div class="panel-heading" role="tab" id="googleDrive">
+
+                        <h2 class="masterlist-header">
+                            <a class="collapsed" 
+                                role="button"
+                                onClick="load_gdoc_table()"
+                                data-toggle="collapse" 
+                                data-parent="#accordionDrive"
+                                href="#collapseDrive" 
+                                aria-expanded="true"
+                                aria-controls="collapseDrive">
+                                Google Drive Files <small>indexed by centillion</small>
+                            </a>
+                        </h2>
+
+                    </div>
+                    <div id="collapseDrive" class="panel-collapse collapse" role="tabpanel" aria-labelledby="googleDrive">
+                        <div class="panel-body">
+                            <table class="table table-striped" id="gdocs-master-list">
+
+                            </table>
+                        </div>
+                    </div>
+                </div>
+            </div>
+        </div>
+    </div>
+
+
+
+    {#
+     # github issue panel
+     #}
+    <div class="row">
+        <div class="panel">
+            <div class="panel-group" id="accordionIssues" role="tablist" aria-multiselectable="true">
+                <div class="panel panel-default">
+                    <div class="panel-heading" role="tab" id="githubIssues">
+
+                        <h2 class="masterlist-header">
+                            <a class="collapsed" 
+                                role="button"
+                                onClick="load_issue_table()"
+                                data-toggle="collapse" 
+                                data-parent="#accordionIssues"
+                                href="#collapseIssues" 
+                                aria-expanded="true"
+                                aria-controls="collapseIssues">
+                                Github Issues <small>indexed by centillion</small>
+                            </a>
+                        </h2>
+
+                    </div>
+                    <div id="collapseIssues" class="panel-collapse collapse" role="tabpanel" 
+                        aria-labelledby="githubIssues">
+                        <div class="panel-body">
+                            <table class="table table-striped" id="issues-master-list">
+
+                            </table>
+                        </div>
+                    </div>
+                </div>
+            </div>
+        </div>
+    </div>
+
+
+
+
+    {#
+     # github file panel
+     #}
+    <div class="row">
+        <div class="panel">
+            <div class="panel-group" id="accordionFiles" role="tablist" aria-multiselectable="true">
+                <div class="panel panel-default">
+                    <div class="panel-heading" role="tab" id="githubFiles">
+
+                        <h2 class="masterlist-header">
+                            <a class="collapsed" 
+                                role="button"
+                                onClick="load_ghfile_table()"
+                                data-toggle="collapse" 
+                                data-parent="#accordionFiles"
+                                href="#collapseFiles" 
+                                aria-expanded="true"
+                                aria-controls="collapseFiles">
+                                Github Files <small>indexed by centillion</small>
+                            </a>
+                        </h2>
+
+                    </div>
+                    <div id="collapseFiles" class="panel-collapse collapse" role="tabpanel" 
+                        aria-labelledby="githubFiles">
+                        <div class="panel-body">
+                            <table class="table table-striped" id="ghfiles-master-list">
+
+                            </table>
+                        </div>
+                    </div>
+                </div>
+            </div>
+        </div>
+    </div>
+
+
+    {#
+     # gh markdown file panel
+     #}
+    <div class="row">
+        <div class="panel">
+            <div class="panel-group" id="accordionMarkdown" role="tablist" aria-multiselectable="true">
+                <div class="panel panel-default">
+                    <div class="panel-heading" role="tab" id="markdown">
+
+                        <h2 class="masterlist-header">
+                            <a class="collapsed" 
+                                role="button"
+                                onClick="load_markdown_table()"
+                                data-toggle="collapse" 
+                                data-parent="#accordionMarkdown"
+                                href="#collapseMarkdown" 
+                                aria-expanded="true"
+                                aria-controls="collapseMarkdown">
+                                Github Markdown Files <small>indexed by centillion</small>
+                            </a>
+                        </h2>
+
+                    </div>
+                    <div id="collapseMarkdown" class="panel-collapse collapse" role="tabpanel" 
+                        aria-labelledby="markdown">
+                        <div class="panel-body">
+                            <table class="table table-striped" id="markdown-master-list">
+
+                            </table>
+                        </div>
+                    </div>
+                </div>
+            </div>
+        </div>
+    </div>
+
+
+
+    {#
+     # groups.io
+     #}
+    <div class="row">
+        <div class="panel">
+            <div class="panel-group" id="accordionThreads" role="tablist" aria-multiselectable="true">
+                <div class="panel panel-default">
+                    <div class="panel-heading" role="tab" id="threads">
+
+                        <h2 class="masterlist-header">
+                            <a class="collapsed" 
+                                role="button"
+                                onClick="load_emailthreads_table()"
+                                data-toggle="collapse" 
+                                data-parent="#accordionThreads"
+                                href="#collapseThreads" 
+                                aria-expanded="true"
+                                aria-controls="collapseThreads">
+                                Groups.io Email Threads <small>indexed by centillion</small>
+                            </a>
+                        </h2>
+
+                    </div>
+                    <div id="collapseThreads" class="panel-collapse collapse" role="tabpanel" 
+                        aria-labelledby="threads">
+                        <div class="panel-body">
+                            <table class="table table-striped" id="emailthreads-master-list">
+
+                            </table>
+                        </div>
+                    </div>
+                </div>
+            </div>
+        </div>
+    </div>
+
+
+
+</div>
+
+{% endblock %}
+

templates/modal.html

@@ -0,0 +1,51 @@
+<div class="modal fade" id="myModal" tabindex="-1" role="dialog" aria-labelledby="myModalLabel">
+
+  <form id="myModalForm" method="post">
+
+    <div class="modal-dialog" role="document">
+      <div id="myModal-content" class="modal-content">
+        <div id="myModal-header" class="modal-header">
+          <button type="button" class="close" data-dismiss="modal" aria-label="Close">
+              <span aria-hidden="true">&times;</span>
+          </button>
+          <h4 class="modal-title" id="myModalLabel">
+          Send us feedback!
+          </h4>
+        </div>
+        <div id="myModal-body" class="modal-body">
+          <div id="modal-feedback-smile-frown-container" class="container-fluid">
+              <div id="modal-feedback-smile-div" class="col-xs-6 text-center" 
+                  onClick="smile()">
+                  <i id="modal-feedback-smile-icon" class="fa fa-smile-o fa-4x" aria-hidden="true"></i>
+              </div>
+              <div id="modal-feedback-frown-div" class="col-xs-6 text-center" 
+                  onClick="frown()">
+                  <i id="modal-feedback-frown-icon" class="fa fa-frown-o fa-4x" aria-hidden="true"></i>
+              </div>
+          </div>
+          <div class="container-fluid">
+              <p>&nbsp;</p>
+          </div>
+          <div id="modal-feedback-textarea-container" class="container-fluid">
+              <textarea id="modal-feedback-textarea" rows="6"></textarea>
+          </div>
+          <div id="modal-too-long" class="container-fluid" >
+              <p id="modal-too-long-text" class="lead">Please limit the length of your feedback. Thank you in advance!</p>
+          </div>
+        </div>
+        <div id="myModal-footer" class="modal-footer">
+            <div class="text-center">
+              <button id="submit-feedback-btn" type="button" 
+                  onClick="submit_feedback()"
+                  class="btn btn-lg btn-primary">
+                  Send
+              </button>
+            </div>
+        </div>
+      </div>
+    </div>
+
+  </form>
+
+</div>        
+

templates/oldsearch.html

@@ -0,0 +1,165 @@
+{% extends "layout.html" %}
+{% block body %}
+
+<div class="container">
+    <div class="row">
+
+        <div class="col-xs-12">
+            <center>
+                <form action="{{ url_for('search') }}" name="search">
+                    <p><input type="text" name="query" value="{{ query }}">
+                    </p>
+                    <p><button id="the-big-one" type="submit" style="font-size: 20px; padding: 10px; padding-left: 50px; padding-right: 50px;" 
+                        value="search" class="btn btn-primary">Search</button>
+                    </p>
+                    <p><a href="{{ url_for('search')}}?query=&fields=">[clear all results]</a>
+                    </p>
+                </form>
+            </center>
+        </div>
+    </div>
+</div>
+
+<div class="container">
+    <div class="row">
+
+        {% if directories %}
+        <div class="col-xs-12 info directories-cloud">
+            <b>File directories:</b> 
+            {% for d in directories %}
+                <a href="{{url_for('search')}}?query={{d|trim}}&fields=filename">{{d|trim}}</a>
+            {% endfor %}
+        </div>
+        {% endif %}
+
+        <ul class="list-group">
+
+            {% if config['SHOW_PARSED_QUERY'] and parsed_query %}
+                <li  class="list-group-item">
+                    <div class="container-fluid">
+                        <div class="row">
+                            <div class="col-xs-12 info">
+                                <b>Parsed query:</b> {{ parsed_query }}
+                            </div>
+                        </div>
+                    </div>
+                </li>
+            {% endif %}
+
+            {% if parsed_query %}
+                <li  class="list-group-item">
+                    <div class="container-fluid">
+                        <div class="row">
+                            <div class="col-xs-12 info">
+                                <b>Found:</b> <span class="badge">{{entries|length}}</span> results 
+                                out of <span class="badge">{{totals["total"]}}</span> total items indexed
+                            </div>
+                        </div>
+                    </div>
+                </li>
+            {% endif %}
+
+            <li  class="list-group-item">
+                    <div class="container-fluid">
+                        <div class="row">
+                            <div class="col-xs-12 info">
+                                <b>Indexing:</b>
+
+                                <span class="badge">{{totals["gdoc"]}}</span>
+                                <a href="/master_list?doctype=gdoc">
+                                Google Drive files
+                                </a>,
+
+                                <span class="badge">{{totals["issue"]}}</span>
+                                <a href="/master_list?doctype=issue">
+                                Github issues
+                                </a>,
+
+                                <span class="badge">{{totals["ghfile"]}}</span>
+                                <a href="/master_list?doctype=ghfile">
+                                Github files
+                                </a>,
+
+                                <span class="badge">{{totals["markdown"]}}</span>
+                                <a href="/master_list?doctype=markdown">
+                                Github Markdown files
+                                </a>,
+
+                                <span class="badge">{{totals["emailthread"]}}</span>
+                                <a href="/master_list?doctype=emailthread">
+                                Groups.io email threads
+                                </a>
+                            </div>
+                        </div>
+                </div>
+            </li>
+
+        </ul>
+    </div>
+</div>
+
+<div class="container">
+    <div class="row">
+        <ul class="list-group">
+
+            {% for e in entries %}
+                <li  class="search-group-item">
+
+                    <div class="url">
+                        {% if e.kind=="gdoc" %}
+                            {% if e.mimetype=="" %}
+                                <b>Google Document:</b>
+                                <a href='{{e.url}}'>{{e.title}}</a>
+                                (Owner: {{e.owner_name}}, {{e.owner_email}})<br />
+                                <b>Document Type</b>: {{e.mimetype}}
+                            {% else %}
+                                <b>Google Drive File:</b>
+                                <a href='{{e.url}}'>{{e.title}}</a><br />
+                                <b>Owner:</b> {{e.owner_name}}, {{e.owner_email}}<br />
+                            {% endif %}
+
+                        {% elif e.kind=="issue" %}
+                            <b>Github Issue:</b>
+                            <a href='{{e.url}}'>{{e.title}}</a>
+                            {% if e.github_user %}<br />
+                            <b>Opened by:</b> <a href='https://github.com/{{e.github_user}}'>@{{e.github_user}}</a>
+                            {% endif %}
+                            <br/>
+                            <b>Repository:</b> <a href='{{e.repo_url}}'>{{e.repo_name}}</a>
+
+                        {% elif e.kind=="markdown" %}
+                            <b>Github Markdown:</b>
+                            <a href='{{e.url}}'>{{e.title}}</a>
+                            <br/>
+                            <b>Repository:</b> <a href='{{e.repo_url}}'>{{e.repo_name}}</a>
+
+                        {% elif e.kind=="emailthread" %}
+                            <b>Groups.io Email Thread:</b>
+                            <a href='{{e.url}}'>{{e.title}}</a>
+                            <br/>
+                            <b>Started By:</b> {{e.owner_name}}
+
+                        {% else %}
+                            <b>Item:</b> (<a href='{{e.url}}'>link</a>)
+
+                        {% endif %}
+                        <br />
+                        Score: {{'%d'  % e.score}}
+                    </div>
+                    <div class="markdown-body">
+                        {% if e.content_highlight %}
+                            {{ e.content_highlight|safe}}
+                        {% else %}
+                        <p>(A preview of this document is not available.)</p>
+                        {% endif %}
+                    </div>
+
+                </li>
+            {% endfor %}
+        </ul>
+
+    </div>
+</div>
+
+{% endblock %}
+

templates/search.html

@@ -1,59 +1,34 @@
 {% extends "layout.html" %}
+{% set active_page = "search" %}
 {% block body %}
 
-
+<div id="search-bar-container" class="container">
-<div class="container">
-
-    {#
-    banner image
-    #}
-    <div class="row">
-        <div class="col12sm">
-            <center>
-                <a href="{{ url_for('search')}}?query=&fields=">
-                <img src="{{ url_for('static', filename='centillion_white.png') }}">
-                </a>
-                {#
-                need a tag line
-                #}
-                {% if config['TAGLINE'] %}
-                    <h2><a href="{{ url_for('search')}}?query=&fields=">
-                        {{config['TAGLINE']}}
-                    </a></h2>
-                {% endif %}
-            </center>
-        </div>
-    </div>
-
-</div>
-<div class="container">
     <div class="row">
 
         <div class="col-xs-12">
             <center>
                 <form action="{{ url_for('search') }}" name="search">
-                    <input type="text" name="query" value="{{ query }}"> <br />
+                    <p><input type="text" name="query" value="{{ query }}">
-                    <button type="submit" style="font-size: 20px; padding: 10px; padding-left: 50px; padding-right: 50px;" 
+                    </p>
+                    <p><button id="the-big-one" type="submit" style="font-size: 20px; padding: 10px; padding-left: 50px; padding-right: 50px;" 
                         value="search" class="btn btn-primary">Search</button>
-                    <br />
+                    </p>
-                    <a href="{{ url_for('search')}}?query=&fields=">[clear all results]</a>
+
+                    {% if parsed_query %}
+                        <p><a href="{{ url_for('search')}}?query=&fields=">[clear all results]</a>
+                    {% endif %}
+
+                    </p>
                 </form>
             </center>
         </div>
     </div>
 </div>
 
-<div class="container">
+<div style="height: 20px;"><p>&nbsp;</p></div>
-    <div class="row">
 
-        {% if directories %}
+<div id="info-bars-container" class="container">
-        <div class="col-xs-12 info directories-cloud">
+    <div class="row">
-            <b>File directories:</b> 
-            {% for d in directories %}
-                <a href="{{url_for('search')}}?query={{d|trim}}&fields=filename">{{d|trim}}</a>
-            {% endfor %}
-        </div>
-        {% endif %}
 
         <ul class="list-group">
 
@@ -69,6 +44,9 @@
                 </li>
             {% endif %}
 
+
+            {# use "if parsed_query" to check if this is 
+               a new search or search results #}
             {% if parsed_query %}
                 <li  class="list-group-item">
                     <div class="container-fluid">
@@ -82,15 +60,37 @@
                 </li>
             {% endif %}
 
+
             <li  class="list-group-item">
                     <div class="container-fluid">
                         <div class="row">
                             <div class="col-xs-12 info">
-                                <b>Indexing:</b> <span
+                                <b>Indexing:</b>
-                                    class="badge">{{totals["gdoc"]}}</span> Google Documents,
+
-                                <span class="badge">{{totals["issue"]}}</span> Github issues,
+                                <span class="badge">{{totals["gdoc"]}}</span>
-                                <span class="badge">{{totals["ghfile"]}}</span> Github files,
+                                <a href="/master_list?doctype=gdoc">
-                                <span class="badge">{{totals["markdown"]}}</span> Github markdown files.
+                                Google Drive files
+                                </a>,
+
+                                <span class="badge">{{totals["issue"]}}</span>
+                                <a href="/master_list?doctype=issue">
+                                Github issues
+                                </a>,
+
+                                <span class="badge">{{totals["ghfile"]}}</span>
+                                <a href="/master_list?doctype=ghfile">
+                                Github files
+                                </a>,
+
+                                <span class="badge">{{totals["markdown"]}}</span>
+                                <a href="/master_list?doctype=markdown">
+                                Github Markdown files
+                                </a>,
+
+                                <span class="badge">{{totals["emailthread"]}}</span>
+                                <a href="/master_list?doctype=emailthread">
+                                Groups.io email threads
+                                </a>
                             </div>
                         </div>
                 </div>
@@ -100,89 +100,123 @@
     </div>
 </div>
 
-<div class="container">
+{% if parsed_query %}
+<div id="search-results-container" class="container">
     <div class="row">
-        <ul class="list-group">
+        <table id="search-results" class="table">
-
+            <thead id="search-results-header">
-            {% for e in entries %}
+                <tr id="search-results-header-row">
-                <li  class="search-group-item">
+                    <td id="search-results-score-col">
+                        Score
+                    </td>
+                    <td id="search-results-type-col">
+                        Type
+                    </td>
+                    <td id="search-results-details-col">
+                        Result Details
+                    </td>
+                </tr>
+            </thead>
+            <tbody>
+
+                {% for e in entries %}
+                    <tr>
+                        <td>
+                            {{'%d'  % e.score}}
+                        </td>
+                        <td>
+                            {% if e.kind=="gdoc" %}
+                                {% if e.mimetype=="document" %}
+                                    <p><small>Drive Document</small</p>
+                                {% else %}
+                                    <p><small>Drive File</small</p>
+                                {% endif %}
+
+                            {% elif e.kind=="issue" %}
+                                <p><small>Issue</small</p>
+
+                            {% elif e.kind=="ghfile" %}
+                                <p><small>Github File</small</p>
+
+                            {% elif e.kind=="markdown" %}
+                                <p><small>Github Markdown</small</p>
+
+                            {% elif e.kind=="emailthread" %}
+                                <p><small>Email Thread</small</p>
 
-                    <div class="url">
-                        {% if e.kind=="gdoc" %}
-                            {% if e.mimetype=="" %}
-                                <b>Google Document:</b>
-                                <a href='{{e.url}}'>{{e.title}}</a>
-                                (Owner: {{e.owner_name}}, {{e.owner_email}})<br />
-                                <b>Document Type</b>: {{e.mimetype}}
                             {% else %}
-                                <b>Google Drive:</b>
+                                <p><small>Unknown</small</p>
-                                <a href='{{e.url}}'>{{e.title}}</a>
-                                (Owner: {{e.owner_name}}, {{e.owner_email}})
-                            {% endif %}
 
-                        {% elif e.kind=="issue" %}
-                            <b>Github Issue:</b>
-                            <a href='{{e.url}}'>{{e.title}}</a>
-                            {% if e.github_user %}
-                            opened by <a href='https://github.com/{{e.github_user}}'>@{{e.github_user}}</a>
                             {% endif %}
-                            <br/>
+                        </td>
-                            <b>Repository:</b> <a href='{{e.repo_url}}'>{{e.repo_name}}</a>
+                        <td>
-
+                            <ul class="list-group">
-                        {% elif e.kind=="markdown" %}
+                                <li  class="search-group-item">
-                            <b>Github Markdown:</b>
+                                    <div class="url">
-                            <a href='{{e.url}}'>{{e.title}}</a>
+                                        {% if e.kind=="gdoc" %}
-                            <br/>
+                                            {% if e.mimetype=="" %}
-                            <b>Repository:</b> <a href='{{e.repo_url}}'>{{e.repo_name}}</a>
+                                                <b>Google Document:</b>
-
+                                                <a href='{{e.url}}'>{{e.title}}</a>
-                        {% else %}
+                                                (Owner: {{e.owner_name}}, {{e.owner_email}})<br />
-                            <b>Item:</b> (<a href='{{e.url}}'>link</a>)
+                                                <b>Document Type</b>: {{e.mimetype}}
-
+                                            {% else %}
-                        {% endif %}
+                                                <b>Google Drive File:</b>
-                        <br />
+                                                <a href='{{e.url}}'>{{e.title}}</a><br />
-                        Score: {{'%d'  % e.score}}
+                                                <b>Owner:</b> {{e.owner_name}}, {{e.owner_email}}<br />
-                    </div>
+                                            {% endif %}
-                    <div class="markdown-body">
+
-                        {% if e.content_highlight %}
+                                        {% elif e.kind=="issue" %}
-                            {{ e.content_highlight|safe}}
+                                            <b>Github Issue:</b>
-                        {% else %}
+                                            <a href='{{e.url}}'>{{e.title}}</a>
-                        <p>(A preview of this document is not available.)</p>
+                                            {% if e.github_user %}<br />
-                        {% endif %}
+                                            <b>Opened by:</b> <a href='https://github.com/{{e.github_user}}'>@{{e.github_user}}</a>
-                    </div>
+                                            {% endif %}
-
+                                            <br/>
-                </li>
+                                            <b>Repository:</b> <a href='{{e.repo_url}}'>{{e.repo_name}}</a>
-            {% endfor %}
+
-        </ul>
+                                        {% elif e.kind=="ghfile" %}
-
+                                            <b>Github File:</b>
-    </div>
+                                            <a href='{{e.url}}'>{{e.title}}</a>
-</div>
+                                            <br/>
-
+                                            <b>Repository:</b> <a href='{{e.repo_url}}'>{{e.repo_name}}</a>
-<div class="container">
+
-    <div class="row">
+                                        {% elif e.kind=="markdown" %}
-        <ul class="list-group">
+                                            <b>Github Markdown:</b>
-
+                                            <a href='{{e.url}}'>{{e.title}}</a>
-            {% if config['FOOTER_REPO_NAME'] %}
+                                            <br/>
-                {% if config['FOOTER_REPO_ORG'] %}
+                                            <b>Repository:</b> <a href='{{e.repo_url}}'>{{e.repo_name}}</a>
-
+
-                    <li  class="list-group-item">
+                                        {% elif e.kind=="emailthread" %}
-                        <div class="container-fluid">
+                                            <b>Groups.io Email Thread:</b>
-                            <div class="row">
+                                            <a href='{{e.url}}'>{{e.title}}</a>
-                                <div class="col-xs-12 info">
+                                            <br/>
-                                    More information about {{config['FOOTER_REPO_NAME']}} can be found
+                                            <b>Started By:</b> {{e.owner_name}}
-                                    in the <a href="https://github.com/{{config['FOOTER_REPO_ORG']}}/{{config['FOOTER_REPO_NAME']}}">{{config['FOOTER_REPO_ORG']}}/{{config['FOOTER_REPO_NAME']}}</a>
+
-                                    repository on Github.
+                                        {% else %}
-                                </div>
+                                            <b>Item:</b> (<a href='{{e.url}}'>link</a>)
-                            </div>
+
-                        </div>
+                                        {% endif %}
-                    </li>
+                                    </div>
-
+                                    <div class="markdown-body">
-                {% endif %}
+                                        {% if e.content_highlight %}
-            {% endif %}
+                                            {{ e.content_highlight|safe}}
-
+                                        {% else %}
-        </ul>
+                                        <p>(A preview of this document is not available.)</p>
+                                        {% endif %}
+                                    </div>
+                                </li>
+                            </ul>
+                        </td>
+                    </tr>
+
+                {% endfor %}
+
+            </tbody>
+        </table>
 
     </div>
 </div>
+{% endif %}
 
 {% endblock %}