Add dates and mailing list names to search results

change "documents" to "issues" in reindexing message

.github/ISSUE_TEMPLATE.md

@@ -0,0 +1,17 @@
+Thanks for using Centillion. Your feedback is important to us. 
+
+### When reporting a bug, please be sure to include the following:
+- [ ] A descriptive title
+- [ ] The behavior you expect to see and the actual behavior observed
+- [ ] Steps to reproduce the behavior 
+- [ ] What browser you are using
+
+### When you open an issue for a feature request, please add as much detail as possible:
+- [ ] A descriptive title
+- [ ] A description of the problem you're trying to solve, including *why* you think this is a problem
+- [ ] An overview of the suggested solution
+- [ ] If the feature changes current behavior, please explain why your solution is better
+
+See read [our contributor guidelines](https://github.com/dcppc/centillion/blob/dcppc/CONTRIBUTING.md) 
+for more details about contributing to this project.
+

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,12 @@
+Thanks for contributing to centillion!
+
+Please place an x between the brackets to indicate a yes answer
+to the questions below.
+
+- [ ] Is this pull request mergeable?
+- [ ] Has this been tested locally?
+- [ ] Does this pull request pass the tests?
+- [ ] Have new tests been added to cover any new code?
+- [ ] Was a spellchecker run on the source code and documentation after
+  changes were made?
+

.gitignore

@@ -1,3 +1,4 @@
+feedback_database.json
 config_flask.py
 vp
 credentials.json

CODE_OF_CONDUCT.md

@@ -0,0 +1,43 @@
+# Code of Conduct
+
+## DCPPC Code of Conduct
+
+All members of the Commons are expected to agree with the following code
+of conduct. We will enforce this code as needed. We expect cooperation
+from all members to help ensuring a safe environment for everybody.
+
+## The Quick Version
+
+The Consortium is dedicated to providing a harassment-free experience
+for everyone, regardless of gender, gender identity and expression, age,
+sexual orientation, disability, physical appearance, body size, race, or
+religion (or lack thereof). We do not tolerate harassment of Consortium
+members in any form. Sexual language and imagery is generally not
+appropriate for any venue, including meetings, presentations, or
+discussions.
+
+## The Less Quick Version
+
+Harassment includes offensive verbal comments related to gender, gender
+identity and expression, age, sexual orientation, disability, physical
+appearance, body size, race, religion, sexual images in public spaces,
+deliberate intimidation, stalking, following, harassing photography or
+recording, sustained disruption of talks or other events, inappropriate
+physical contact, and unwelcome sexual attention.
+
+Members asked to stop any harassing behavior are expected to comply
+immediately.
+
+If you are being harassed, notice that someone else is being harassed,
+or have any other concerns, please contact [Titus
+Brown](mailto:ctbrown@ucdavis.edu) immediately. If Titus is the cause of
+your concern, please contact [Vivien
+Bonazzi](mailto:bonazziv@mail.nih.gov).
+
+We expect members to follow these guidelines at any Consortium event.
+
+Original source and credit: <http://2012.jsconf.us/#/about> & The Ada
+Initiative. Please help by translating or improving:
+<http://github.com/leftlogic/confcodeofconduct.com>. This work is
+licensed under a Creative Commons Attribution 3.0 Unported License
+

CONTRIBUTING.md

@@ -0,0 +1,21 @@
+# Contributing to the DCPPC Internal Repository
+
+Hello, and thank you for wanting to contribute to the DCPPC Internal
+Repository\!
+
+By contributing to this repository, you agree:
+
+1.  To obey the [Code of Conduct](./CODE_OF_CONDUCT.md)
+2.  To release all your contributions under the same terms as the
+    license itself: the [Creative Commons Zero](./LICENSE.md) (aka
+    Public Domain) license
+
+If you are OK with these two conditions, then we welcome both you and
+your contribution\!
+
+If you have any questions about contributing, please [open an
+issue](https://github.com/dcppc/internal/issues/new) and Team Copper
+will lend a hand ASAP.
+
+Thank you for being here and for being a part of the DCPPC project.
+

Hypothesis.md

@@ -0,0 +1,249 @@
+# Hypothesis API
+
+
+## Authenticating
+
+Example output call for authenticating with the API:
+
+```
+{
+    "links": {
+        "profile": {
+            "read": {
+                "url": "https://hypothes.is/api/profile",
+                "method": "GET",
+                "desc": "Fetch the user's profile"
+            },
+            "update": {
+                "url": "https://hypothes.is/api/profile",
+                "method": "PATCH",
+                "desc": "Update a user's preferences"
+            }
+        },
+        "search": {
+            "url": "https://hypothes.is/api/search",
+            "method": "GET",
+            "desc": "Search for annotations"
+        },
+        "group": {
+            "member": {
+                "add": {
+                    "url": "https://hypothes.is/api/groups/:pubid/members/:userid",
+                    "method": "POST",
+                    "desc": "Add the user in the request params to a group."
+                },
+                "delete": {
+                    "url": "https://hypothes.is/api/groups/:pubid/members/:userid",
+                    "method": "DELETE",
+                    "desc": "Remove the current user from a group."
+                }
+            }
+        },
+        "links": {
+            "url": "https://hypothes.is/api/links",
+            "method": "GET",
+            "desc": "URL templates for generating URLs for HTML pages"
+        },
+        "groups": {
+            "read": {
+                "url": "https://hypothes.is/api/groups",
+                "method": "GET",
+                "desc": "Fetch the user's groups"
+            }
+        },
+        "annotation": {
+            "hide": {
+                "url": "https://hypothes.is/api/annotations/:id/hide",
+                "method": "PUT",
+                "desc": "Hide an annotation as a group moderator."
+            },
+            "unhide": {
+                "url": "https://hypothes.is/api/annotations/:id/hide",
+                "method": "DELETE",
+                "desc": "Unhide an annotation as a group moderator."
+            },
+            "read": {
+                "url": "https://hypothes.is/api/annotations/:id",
+                "method": "GET",
+                "desc": "Fetch an annotation"
+            },
+            "create": {
+                "url": "https://hypothes.is/api/annotations",
+                "method": "POST",
+                "desc": "Create an annotation"
+            },
+            "update": {
+                "url": "https://hypothes.is/api/annotations/:id",
+                "method": "PATCH",
+                "desc": "Update an annotation"
+            },
+            "flag": {
+                "url": "https://hypothes.is/api/annotations/:id/flag",
+                "method": "PUT",
+                "desc": "Flag an annotation for review."
+            },
+            "delete": {
+                "url": "https://hypothes.is/api/annotations/:id",
+                "method": "DELETE",
+                "desc": "Delete an annotation"
+            }
+        }
+    }
+}
+```
+
+
+## Listing
+
+Here is the result of the API call to list an annotation
+given its annotation ID:
+
+```
+{
+    "updated": "2018-07-26T10:20:47.803636+00:00",
+    "group": "__world__",
+    "target": [
+        {
+            "source": "https://h.readthedocs.io/en/latest/api/authorization/",
+            "selector": [
+                {
+                    "conformsTo": "https://tools.ietf.org/html/rfc3236",
+                    "type": "FragmentSelector",
+                    "value": "access-tokens"
+                },
+                {
+                    "endContainer": "/div[1]/section[1]/div[1]/div[1]/div[2]/div[1]/div[1]/div[2]/p[2]",
+                    "startContainer": "/div[1]/section[1]/div[1]/div[1]/div[2]/div[1]/div[1]/div[2]/p[1]",
+                    "type": "RangeSelector",
+                    "startOffset": 14,
+                    "endOffset": 116
+                },
+                {
+                    "type": "TextPositionSelector",
+                    "end": 2234,
+                    "start": 1374
+                },
+                {
+                    "exact": "hich read or write data as a specific user need to be authorized\nwith an access token. Access tokens can be obtained in two ways:\n\nBy generating a personal API token on the Hypothesis developer\npage (you must be logged in to\nHypothesis to get to this page). This is the simplest method, however\nthese tokens are only suitable for enabling your application to make\nrequests as a single specific user.\n\nBy registering an \u201cOAuth client\u201d and\nimplementing the OAuth authentication flow\nin your application. This method allows any user to authorize your\napplication to read and write data via the API as that user.  The Hypothesis\nclient is an example of an application that uses OAuth.\nSee Using OAuth for details of how to implement this method.\n\n\nOnce an access token has been obtained, requests can be authorized by putting\nthe token in the Authorization header.",
+                    "prefix": "\n\n\nAccess tokens\u00b6\nAPI requests w",
+                    "type": "TextQuoteSelector",
+                    "suffix": "\nExample request:\nGET /api HTTP/"
+                }
+            ]
+        }
+    ],
+    "links": {
+        "json": "https://hypothes.is/api/annotations/kEaohJC9Eeiy_UOozkpkyA",
+        "html": "https://hypothes.is/a/kEaohJC9Eeiy_UOozkpkyA",
+        "incontext": "https://hyp.is/kEaohJC9Eeiy_UOozkpkyA/h.readthedocs.io/en/latest/api/authorization/"
+    },
+    "tags": [],
+    "text": "sdfsdf",
+    "created": "2018-07-26T10:20:47.803636+00:00",
+    "uri": "https://h.readthedocs.io/en/latest/api/authorization/",
+    "flagged": false,
+    "user_info": {
+        "display_name": null
+    },
+    "user": "acct:Aravindan@hypothes.is",
+    "hidden": false,
+    "document": {
+        "title": [
+            "Authorization \u2014 h 0.0.2 documentation"
+        ]
+    },
+    "id": "kEaohJC9Eeiy_UOozkpkyA",
+    "permissions": {
+        "read": [
+            "group:__world__"
+        ],
+        "admin": [
+            "acct:Aravindan@hypothes.is"
+        ],
+        "update": [
+            "acct:Aravindan@hypothes.is"
+        ],
+        "delete": [
+            "acct:Aravindan@hypothes.is"
+        ]
+    }
+}
+```
+
+## Searching
+
+Here is the output from a call to the endpoint to search annotations
+(we pass a specific URL to the search function):
+
+```
+{
+    "rows": [
+        {
+            "updated": "2018-08-10T02:21:46.898833+00:00",
+            "group": "__world__",
+            "target": [
+                {
+                    "source": "http://pilot.data-commons.us/organize/CopperInternalDeliveryWorkFlow/",
+                    "selector": [
+                        {
+                            "endContainer": "/div[1]/main[1]/div[1]/div[3]/article[1]/h2[1]",
+                            "startContainer": "/div[1]/main[1]/div[1]/div[3]/article[1]/h2[1]",
+                            "type": "RangeSelector",
+                            "startOffset": 0,
+                            "endOffset": 80
+                        },
+                        {
+                            "type": "TextPositionSelector",
+                            "end": 12328,
+                            "start": 12248
+                        },
+                        {
+                            "exact": "Deliverables are due internally on the first of each month, which here is Day 1,",
+                            "prefix": "               \n                ",
+                            "type": "TextQuoteSelector",
+                            "suffix": "\u00b6\nDay -30 through -10\nCopper PM "
+                        }
+                    ]
+                }
+            ],
+            "links": {
+                "json": "https://hypothes.is/api/annotations/IY2W_pxEEeiVuxfD3sehjQ",
+                "html": "https://hypothes.is/a/IY2W_pxEEeiVuxfD3sehjQ",
+                "incontext": "https://hyp.is/IY2W_pxEEeiVuxfD3sehjQ/pilot.data-commons.us/organize/CopperInternalDeliveryWorkFlow/"
+            },
+            "tags": [],
+            "text": "This is a sample annotation",
+            "created": "2018-08-10T02:21:46.898833+00:00",
+            "uri": "http://pilot.data-commons.us/organize/CopperInternalDeliveryWorkFlow/",
+            "flagged": false,
+            "user_info": {
+                "display_name": null
+            },
+            "user": "acct:charlesreid1dib@hypothes.is",
+            "hidden": false,
+            "document": {
+                "title": [
+                    "Copper Internal Delivery Workflow - Data Commons Internal Site"
+                ]
+            },
+            "id": "IY2W_pxEEeiVuxfD3sehjQ",
+            "permissions": {
+                "read": [
+                    "group:__world__"
+                ],
+                "admin": [
+                    "acct:charlesreid1dib@hypothes.is"
+                ],
+                "update": [
+                    "acct:charlesreid1dib@hypothes.is"
+                ],
+                "delete": [
+                    "acct:charlesreid1dib@hypothes.is"
+                ]
+            }
+        }
+    ],
+    "total": 1
+}
+```
+

LICENSE

@@ -1,29 +1,133 @@
-BSD 3-Clause License
+# DEDICATED TO THE PUBLIC DOMAIN
-
+
-Copyright (c) 2018, Charles Reid
+The dcppc/organize repository has been dedicated to the public domain.
-All rights reserved.
+It is protected by the Creative Commons CC0 Universal Public Domain
-
+Dedication license. You can read the entire license below or at
-Redistribution and use in source and binary forms, with or without
+<http://creativecommons.org/publicdomain/zero/1.0/deed.en>.
-modification, are permitted provided that the following conditions are met:
+
-
+# CC0 UNIVERSAL PUBLIC DOMAIN DEDICATION LICENSE
-* Redistributions of source code must retain the above copyright notice, this
+
-  list of conditions and the following disclaimer.
+## Statement of Purpose
-
+
-* Redistributions in binary form must reproduce the above copyright notice,
+The laws of most jurisdictions throughout the world automatically confer
-  this list of conditions and the following disclaimer in the documentation
+exclusive Copyright and Related Rights (defined below) upon the creator
-  and/or other materials provided with the distribution.
+and subsequent owner(s) (each and all, an "owner") of an original work
-
+of authorship and/or a database (each, a "Work").
-* Neither the name of the copyright holder nor the names of its
+
-  contributors may be used to endorse or promote products derived from
+Certain owners wish to permanently relinquish those rights to a Work for
-  this software without specific prior written permission.
+the purpose of contributing to a commons of creative, cultural and
-
+scientific works ("Commons") that the public can reliably and without
-THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+fear of later claims of infringement build upon, modify, incorporate in
-AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+other works, reuse and redistribute as freely as possible in any form
-IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+whatsoever and for any purposes, including without limitation commercial
-DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+purposes. These owners may contribute to the Commons to promote the
-FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ideal of a free culture and the further production of creative, cultural
-DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+and scientific works, or to gain reputation or greater distribution for
-SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+their Work in part through the use and efforts of others.
-CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+
-OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+For these and/or other purposes and motivations, and without any
-OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+expectation of additional consideration or compensation, the person
+associating CC0 with a Work (the "Affirmer"), to the extent that he or
+she is an owner of Copyright and Related Rights in the Work, voluntarily
+elects to apply CC0 to the Work and publicly distribute the Work under
+its terms, with knowledge of his or her Copyright and Related Rights in
+the Work and the meaning and intended legal effect of CC0 on those
+rights.
+
+1.  **Copyright and Related Rights.** A Work made available under CC0
+    may be protected by copyright and related or neighboring rights
+    ("Copyright and Related Rights"). Copyright and Related Rights
+    include, but are not limited to, the following:
+    
+    i. the right to reproduce, adapt, distribute, perform, display,
+    communicate, and translate a Work;
+    
+    ii. moral rights retained by the original author(s) and/or
+    performer(s);
+    
+    iii. publicity and privacy rights pertaining to a person's image or
+    likeness depicted in a Work;
+    
+    iv. rights protecting against unfair competition in regards to a
+    Work, subject to the limitations in paragraph 4(a), below;
+    
+    v. rights protecting the extraction, dissemination, use and reuse of
+    data in a Work;
+    
+    vi. database rights (such as those arising under Directive 96/9/EC
+    of the European Parliament and of the Council of 11 March 1996 on
+    the legal protection of databases, and under any national
+    implementation thereof, including any amended or successor version
+    of such directive); and
+    
+    vii. other similar, equivalent or corresponding rights throughout
+    the world based on applicable law or treaty, and any national
+    implementations thereof.
+
+2.  **Waiver.** To the greatest extent permitted by, but not in
+    contravention of, applicable law, Affirmer hereby overtly, fully,
+    permanently, irrevocably and unconditionally waives, abandons, and
+    surrenders all of Affirmer's Copyright and Related Rights and
+    associated claims and causes of action, whether now known or unknown
+    (including existing as well as future claims and causes of action),
+    in the Work (i) in all territories worldwide, (ii) for the maximum
+    duration provided by applicable law or treaty (including future time
+    extensions), (iii) in any current or future medium and for any
+    number of copies, and (iv) for any purpose whatsoever, including
+    without limitation commercial, advertising or promotional purposes
+    (the "Waiver"). Affirmer makes the Waiver for the benefit of each
+    member of the public at large and to the detriment of Affirmer's
+    heirs and successors, fully intending that such Waiver shall not be
+    subject to revocation, rescission, cancellation, termination, or any
+    other legal or equitable action to disrupt the quiet enjoyment of
+    the Work by the public as contemplated by Affirmer's express
+    Statement of Purpose.
+
+3.  **Public License Fallback.** Should any part of the Waiver for any
+    reason be judged legally invalid or ineffective under applicable
+    law, then the Waiver shall be preserved to the maximum extent
+    permitted taking into account Affirmer's express Statement of
+    Purpose. In addition, to the extent the Waiver is so judged Affirmer
+    hereby grants to each affected person a royalty-free, non
+    transferable, non sublicensable, non exclusive, irrevocable and
+    unconditional license to exercise Affirmer's Copyright and Related
+    Rights in the Work (i) in all territories worldwide, (ii) for the
+    maximum duration provided by applicable law or treaty (including
+    future time extensions), (iii) in any current or future medium and
+    for any number of copies, and (iv) for any purpose whatsoever,
+    including without limitation commercial, advertising or promotional
+    purposes (the "License"). The License shall be deemed effective as
+    of the date CC0 was applied by Affirmer to the Work. Should any part
+    of the License for any reason be judged legally invalid or
+    ineffective under applicable law, such partial invalidity or
+    ineffectiveness shall not invalidate the remainder of the License,
+    and in such case Affirmer hereby affirms that he or she will not (i)
+    exercise any of his or her remaining Copyright and Related Rights in
+    the Work or (ii) assert any associated claims and causes of action
+    with respect to the Work, in either case contrary to Affirmer's
+    express Statement of Purpose.
+
+4.  **Limitations and Disclaimers.**
+    
+    a. No trademark or patent rights held by Affirmer are waived,
+    abandoned, surrendered, licensed or otherwise affected by this
+    document.
+    
+    b. Affirmer offers the Work as-is and makes no representations or
+    warranties of any kind concerning the Work, express, implied,
+    statutory or otherwise, including without limitation warranties of
+    title, merchantability, fitness for a particular purpose, non
+    infringement, or the absence of latent or other defects, accuracy,
+    or the present or absence of errors, whether or not discoverable,
+    all to the greatest extent permissible under applicable law.
+    
+    c. Affirmer disclaims responsibility for clearing rights of other
+    persons that may apply to the Work or any use thereof, including
+    without limitation any person's Copyright and Related Rights in the
+    Work. Further, Affirmer disclaims responsibility for obtaining any
+    necessary consents, permissions or other rights required for any use
+    of the Work.
+    
+    d. Affirmer understands and acknowledges that Creative Commons is
+    not a party to this document and has no duty or obligation with
+    respect to this CC0 or use of the Work.

Readme.md

@@ -6,14 +6,19 @@
 
 one centillion is 3.03 log-times better than a googol.
 
-![Screen shot of centillion](docs/images/ss.png)
+![Screen shot: centillion search](docs/images/search.png)
 
 
-## what is it
+## What Is It
 
 Centillion (https://github.com/dcppc/centillion) is a search engine that can index 
-three kinds of collections: Google Documents, Github issues, and Markdown files in 
+different kinds of document collections: Google Documents (.docx files), Google Drive files,
-Github repos.
+Github issues, Github files, Github Markdown files, and Groups.io email threads.
+
+
+
+
+## What Is It
 
 We define the types of documents the centillion should index,
 what info and how. The centillion then builds and
@@ -25,14 +30,43 @@ defined in `centillion.py`.
 
 The centillion keeps it simple.
 
-## authentication layer
+## Authentication Layer
 
 Centillion lives behind a Github authentication layer, implemented with 
 [flask-dance](https://github.com/singingwolfboy/flask-dance). When you first
 visit the site it will ask you to authenticate with Github so that it can 
 verify you have permission to access the site.
 
-## technologies
+![Screen shot: centillion authentication](docs/images/auth.png)
+
+## Master List
+
+There is a master list of all content indexed by centilion at the master list page,
+<https://search.nihdatacommons.us/master_list>.
+
+A master list for each type of document indexed by the search engine is displayed
+in a table:
+
+![Screen shot: centillion master list](docs/images/master_list.png)
+
+The metadata shown in these tables can be filtered and sorted:
+
+![Screen shot: centillion master list with sorting](docs/images/master_list2.png)
+
+## Control Panel
+
+There's also a control panel at <https://search.nihdatacommons.us/control_panel> 
+that allows you to rebuild the search index from scratch.  The search index
+stores versions/contents of files locally, so re-indexing involves going out and
+asking each API for new versions of a file/document/web page. When you re-index
+the main search index, it will ask every API for new versions of every document.
+You can also update only specific types of documents in the search index.
+
+![Screen shot: centillion control panel](docs/images/control_panel.png)
+
+
+
+## Technologies
 
 Centillion is a Python program built using whoosh (search engine library). It
 indexes the full text of docx files in Google Documents, just the filenames for
@@ -41,16 +75,61 @@ results are grouped by issue. Centillion requires Google Drive and Github OAuth
 apps. Once you provide credentials to Flask you're all set to go. 
 
 
-## control panel
+## Configuration
 
-There's also a control panel at <https://search.nihdatacommons.us/control_panel> 
+You will need to configure both the centillion search index and the flask app.
-that allows you to rebuild the search index from scratch (the Google Drive indexing 
+
-takes a while).
+The centillion search index is configured with `config_centillion.py`; this file
+sets the names of repositories to crawl when indxing issues and files.
+
+The flask app is configured with `config_flask.py`. This file contains sensitive
+information and is in the `.gitignore` file. This file contains API credentials 
+for Github and Groups.io.
+
+Exampls are provided in `config_centillion.example.py` and `config_flask.example.py`.
+
+
+## Authentication
+
+The search engine will need to connect to several APIs when it re-indexes the
+search index:
+
+* Github
+* Groups.io
+* Google Drive
+
+### Github
+
+Github API credentials (both an OAuth token for the centillion app's Github
+authentication mechanism, and a personal access token for accessing repositories
+during the re-indexing process) are provided in `config_flask.py`.
+
+### Groups.io
+
+The Groups.io API token is used to index email threads. This token is provided in
+`config_flask.py`.
+
+### Google Drive
+
+The Google Drive API credentials are provided in a file, `credentials.json`. This is
+the file that is generated when the OAuth process is complete.
+
+When you enable the Google Drive API in the Google Cloud Console, you will be provided
+with a file `client_secrets.json`. To authenticate centillion with Google Drive, you should
+download this file, and run the Google Drive utility directly:
+
+```
+python gdrive_util.py
+```
 
-![Screen shot of centillion control panel](docs/images/cp.png)
+This will initiate the authentication procedure. Sign in as a user that has access to
+the documents you want to index, and _only_ the documents you want to index (it is useful
+to set up a bot account for this purpose).
 
+Once you log in as that user, it will create `credentials.json`, and the Google Drive
+re-indexing procedure should not have any problems autheticating using that file.
 
-## quickstart (with Github auth)
+## Quickstart (With Github Auth)
 
 Start by creating a Github OAuth application.
 Get the public and private application key 
@@ -85,7 +164,7 @@ This will start a Flask server, and you can view the minimal search engine
 interface in your browser at `http://<ip>:5000`.
 
 
-## troubleshooting
+## Troubleshooting
 
 If you are having problems with your callback URL being treated
 as HTTP by Github, even though there is an HTTPS address, and

Todo.md

@@ -1,47 +1,29 @@
 # todo
 
-Main task:
+ux improvements:
-- hashing and caching
+- feedback tools
-    - <s>first, working out the logic of how we group items into sets
+- integrating master list into single list
-        - needs to be deleted
+- providing advanced search interfce
-        - needs to be updated
-        - needs to be added
-        - for docs, issues, and comments</s>
-    - second, when we add or update an item, need to:
-        - go through the motions, download file, extract text
-        - check for existing indexed doc with that id
-        - check if existing indexed doc has same hash
-            - if so, skip
-            - otherwise, delete and re-index
 
-Other bugs:
+big picture improvements:
-- Some github issues have no title (?)
+- hypothesis API
-- <s>Need to combine issues with comments</s>
+- folksonomy tagging with hypothesis
-- Not able to index markdown files _in a repo_
+- tags, expanded schema
-- (Longer term) update main index vs update diff index
 
-Needs:
-- <s>control panel</s>
 
-Thursday product:
-- Everything re-indexed nightly
-- Search engine built on all documents in Google Drive, all issues, markdown files
-- Using pandoc to extract Google Drive document contents
-- BRIEF quickstart documentation
 
-Future:
-- Future plans to improve - plugins, improving matching
-- Subdomain plans
-- Folksonomy tagging and integration plans
-
-
-
-
-config options for plugins
-conditional blocks with import github inside
-complicated tho - better to have components split off
 
+feedback form: where we are at
+- feedback button
+- button triggers modal form
+- modal has emojis for feedback, text box, buttons
+- clicking emojis changes color, to select
+- clicking submit with filled out form submits to an endpoint
+- clicking submit also closes form, but only if submit successful
 
+feedback form: what we need to do
+- fix alerts - thank you for your feedback doesn't show up until a refresh
+- probably an easy ajax fix
 
 
 

centillion.py

@@ -1,16 +1,18 @@
 import threading
-from subprocess import call
+import subprocess
 
 import codecs
 import os, json
+from datetime import datetime
 
 from werkzeug.contrib.fixers import ProxyFix
-from flask import Flask, request, redirect, url_for, render_template, flash
+from flask import Flask, request, redirect, url_for, render_template, flash, jsonify
 from flask_dance.contrib.github import make_github_blueprint, github
 
 # create our application
 from centillion_search import Search
 
+import config_centillion
 
 """
 The Centillion
@@ -27,8 +29,9 @@ You provide:
 
 
 class UpdateIndexTask(object):
-    def __init__(self, app_config, diff_index=False):
+    def __init__(self, app_config, diff_index=False,run_which='all'):
         self.diff_index = diff_index
+        self.run_which = run_which
         thread = threading.Thread(target=self.run, args=())
 
         self.gh_token = app_config['GITHUB_TOKEN']
@@ -37,6 +40,7 @@ class UpdateIndexTask(object):
                 'groupsio_username' :  app_config['GROUPSIO_USERNAME'],
                 'groupsio_password' :  app_config['GROUPSIO_PASSWORD']
         }
+        self.disqus_token = app_config['DISQUS_TOKEN']
         thread.daemon = True
         thread.start()
 
@@ -46,14 +50,14 @@ class UpdateIndexTask(object):
         if(self.diff_index):
             raise Exception("diff index not implemented")
 
-        from get_centillion_config import get_centillion_config
+        #from get_centillion_config import get_centillion_config
-        config = get_centillion_config('config_centillion.json')
+        config = config_centillion.config
-
-        search.update_index_groupsioemails(self.groupsio_credentials,config)
-        ###search.update_index_ghfiles(self.gh_token,config)
-        ###search.update_index_issues(self.gh_token,config)
-        ###search.update_index_gdocs(config)
 
+        search.update_index(self.groupsio_credentials,
+                            self.gh_token,
+                            self.disqus_token,
+                            self.run_which,
+                            config)
 
 
 app = Flask(__name__)
@@ -70,69 +74,73 @@ github_bp = make_github_blueprint(
 
 app.register_blueprint(github_bp, url_prefix="/login")
 
-contents404 = "<html><body><h1>Status: Error 404 Page Not Found</h1></body></html>"
+last_searches_file = app.config["INDEX_DIR"] + "/last_searches.txt"
-contents403 = "<html><body><h1>Status: Error 403 Access Denied</h1></body></html>"
-contents200 = "<html><body><h1>Status: OK 200</h1></body></html>"
-
 
 ##############################
 # Flask routes
 
 @app.route('/')
 def index():
+    if not github.authorized:
+        return render_template("landing.html")
+    else:
+        username = github.get("/user").json()['login']
+        resp = github.get("/user/orgs")
+        if resp.ok:
+
+            # If they are in dcppc, redirect to search.
+            # Otherwise, hit em with a 403
+            all_orgs = resp.json()
+            for org in all_orgs:
+                if org['login']=='dcppc':
+                    # Business as usual
+                    return redirect(url_for("search", query="", fields=""))
+
+            # Not in dcppc
+            return render_template('403.html')
+
+        # Could not reach Github
+        return render_template('404.html')
+
 
+@app.route('/log_in')
+def log_in():
     if not github.authorized:
         return redirect(url_for("github.login"))
-
     else:
-
         username = github.get("/user").json()['login']
-
         resp = github.get("/user/orgs")
         if resp.ok:
 
-            # If they are in team copper, redirect to search.
+            # If they are in dcppc, redirect to search.
             # Otherwise, hit em with a 403
             all_orgs = resp.json()
             for org in all_orgs:
                 if org['login']=='dcppc':
-                    copper_team_id = '2700235'
-                    mresp = github.get('/teams/%s/members/%s'%(copper_team_id,username))
-                    if mresp.status_code==204:
-
-                        # --------------------
                     # Business as usual
                     return redirect(url_for("search", query="", fields=""))
 
-            return contents403
+            # Not in dcppc
+            return render_template('403.html')
+
+        # Could not reach Github
+        return render_template('404.html')
 
-        return contents404
 
-### @app.route('/')
-### def index():
-###     return redirect(url_for("search", query="", fields=""))
 
 @app.route('/search')
 def search():
-
     if not github.authorized:
         return redirect(url_for("github.login"))
-
     username = github.get("/user").json()['login']
-
     resp = github.get("/user/orgs")
     if resp.ok:
 
+        # If they are in dcppc, show them search.html
+        # Otherwise, hit em with a 403
         all_orgs = resp.json()
         for org in all_orgs:
             if org['login']=='dcppc':
-
-                copper_team_id = '2700235'
-
-                mresp = github.get('/teams/%s/members/%s'%(copper_team_id,username))
-                if mresp.status_code==204:
-
-                    # --------------------
                 # Business as usual
                 query = request.args['query']
                 fields = request.args.get('fields')
@@ -146,6 +154,7 @@ def search():
 
                 else:
                     parsed_query, result = search.search(query.split(), fields=[fields])
+                    store_search(query,fields)
 
                 totals = search.get_document_total_count()
 
@@ -156,73 +165,193 @@ def search():
                                        fields=fields, 
                                        totals=totals)
 
-    return contents403
+        # Not in dcppc 
+        return render_template('403.html')
 
+    # Could not reach Github
+    return render_template('404.html')
 
-@app.route('/update_index')
-def update_index():
 
+@app.route('/update_index/<run_which>')
+def update_index(run_which):
+    """
+    TEAM COPPER ONLY!!!
+    """
     if not github.authorized:
         return redirect(url_for("github.login"))
-
     username = github.get("/user").json()['login']
-
     resp = github.get("/user/orgs")
     if resp.ok:
 
+        # Only Team Copper members can update the index
         all_orgs = resp.json()
         for org in all_orgs:
             if org['login']=='dcppc':
-
                 copper_team_id = '2700235'
-
                 mresp = github.get('/teams/%s/members/%s'%(copper_team_id,username))
                 if mresp.status_code==204:
-
-                    # --------------------
                     # Business as usual
                     UpdateIndexTask(app.config,
-                                    diff_index=False)
+                                    diff_index=False,
+                                    run_which = run_which)
                     flash("Rebuilding index, check console output")
-                    return render_template("controlpanel.html", 
+                    # This redirects user to /control_panel route
-                                           totals={})
+                    # to prevent accidental re-indexing
-
+                    return redirect(url_for("control_panel"))
-    return contents403
 
+    return render_template('403.html')
 
 
 @app.route('/control_panel')
 def control_panel():
-
+    """
+    TEAM COPPER ONLY!!!
+    """
     if not github.authorized:
         return redirect(url_for("github.login"))
+    username = github.get("/user").json()['login']
+    resp = github.get("/user/orgs")
+    if resp.ok:
 
+        # Only Team Copper members can access the control panel
+        all_orgs = resp.json()
+        for org in all_orgs:
+            if org['login']=='dcppc':
+                copper_team_id = '2700235'
+                mresp = github.get('/teams/%s/members/%s'%(copper_team_id,username))
+                if mresp.status_code==204:
+                    # Business as usual
+                    return render_template("controlpanel.html")
+
+        # Not in dcppc 
+        return render_template('403.html')
+
+    # Could not reach Github
+    return render_template('404.html')
+
+
+
+@app.route('/master_list')
+def master_list():
+    if not github.authorized:
+        return redirect(url_for("github.login"))
     username = github.get("/user").json()['login']
+    resp = github.get("/user/orgs")
+    if resp.ok:
+
+        # If they are in dcppc, show them masterlist.html
+        # Otherwise, hit em with a 403
+        all_orgs = resp.json()
+        for org in all_orgs:
+            if org['login']=='dcppc':
+                # Business as usual
+                return render_template("masterlist.html")
+
+        # Not in dcppc 
+        return render_template('403.html')
+
+    # Could not reach Github
+    return render_template('404.html')
 
+
+
+@app.route('/list/<doctype>')
+def list_docs(doctype):
+    if not github.authorized:
+        return redirect(url_for("github.login"))
+    username = github.get("/user").json()['login']
     resp = github.get("/user/orgs")
     if resp.ok:
+        all_orgs = resp.json()
+        for org in all_orgs:
+            if org['login']=='dcppc':
+                # Business as usual
+                search = Search(app.config["INDEX_DIR"])
+                results_list = search.get_list(doctype)
+                for result in results_list:
+                    ct = result['created_time']
+                    result['created_time'] = datetime.strftime(ct,"%Y-%m-%d %I:%M %p")
+                return jsonify(results_list)
+
+    # nope
+    return render_template('403.html')
+
+
+@app.route('/feedback', methods=['POST'])
+def parse_request():
 
+    if not github.authorized:
+        return redirect(url_for("github.login"))
+    username = github.get("/user").json()['login']
+    resp = github.get("/user/orgs")
+    if resp.ok:
         all_orgs = resp.json()
         for org in all_orgs:
             if org['login']=='dcppc':
 
-                copper_team_id = '2700235'
 
-                mresp = github.get('/teams/%s/members/%s'%(copper_team_id,username))
+                try:
-                if mresp.status_code==204:
+                    # Business as usual
+                    data = request.form.to_dict();
+                    data['github_login'] = username
+                    data['timestamp'] = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
 
-                    return render_template("controlpanel.html", 
+                    feedback_database = 'feedback_database.json'
-                                           totals={})
+                    if not os.path.isfile(feedback_database):
+                        with open(feedback_database,'w') as f:
+                            json_data = [data]
+                            json.dump(json_data, f, indent=4)
 
-    return contents403
+                    else:
+                        json_data = []
+                        with open(feedback_database,'r') as f:
+                            json_data = json.load(f)
+
+                        json_data.append(data)
+
+                        with open(feedback_database,'w') as f:
+                            json.dump(json_data, f, indent=4)
 
+                    ## Should be done with Javascript
+                    #flash("Thank you for your feedback!")
+                    return jsonify({'status':'ok','message':'Thank you for your feedback!'})
+                except:
+                    return jsonify({'status':'error','message':'An error was encountered while submitting your feedback. Try submitting an issue in the <a href="https://github.com/dcppc/centillion/issues/new">dcppc/centillion</a> repository.'})
+
+
+    # nope
+    return render_template('403.html')
 
 @app.errorhandler(404)
 def oops(e):
-    return contents404
+    return render_template('404.html')
+
+
+def store_search(query, fields):
+    """
+    Store searches in a text file
+    """
+    if os.path.exists(last_searches_file):
+        with codecs.open(last_searches_file, 'r', encoding='utf-8') as f:
+            contents = f.readlines()
+    else:
+        contents = []
+
+    search = "query=%s&fields=%s\n" % (query, fields)
+    if not search in contents:
+        contents.insert(0, search)
+
+    with codecs.open(last_searches_file, 'w', encoding='utf-8') as f:
+        f.writelines(contents)
+
 
 if __name__ == '__main__':
     # if running local instance, set to true
     os.environ['OAUTHLIB_INSECURE_TRANSPORT'] = 'true'
-    app.run(host="0.0.0.0",port=5000)
+    port = os.environ.get('CENTILLION_PORT','')
+    if port=='':
+        port = 5000
+    else:
+        port = int(port)
+    app.run(host="0.0.0.0", port=port)
 

centillion_search.py

@@ -5,7 +5,9 @@ from github import Github, GithubException
 import base64
 
 from gdrive_util import GDrive
-from groupsio_util import GroupsIOArchivesCrawler
+from groupsio_util import GroupsIOArchivesCrawler, GroupsIOException
+from disqus_util import DisqusCrawler
+
 from apiclient.http import MediaIoBaseDownload
 
 import mistune
@@ -17,9 +19,13 @@ import pypandoc
 import os.path
 import codecs
 from datetime import datetime
+import dateutil.parser
 
+from whoosh import query
 from whoosh.qparser import MultifieldParser, QueryParser
-from whoosh.analysis import StemmingAnalyzer
+from whoosh.analysis import StemmingAnalyzer, LowercaseFilter, StopFilter
+from whoosh.qparser.dateparse import DateParserPlugin
+from whoosh import fields, index
 
 
 """
@@ -99,6 +105,69 @@ class Search:
         self.open_index(index_folder)
 
 
+    # ------------------------------
+    # Update the entire index
+
+    def update_index(self, groupsio_credentials, gh_token, disqus_token, run_which, config):
+        """
+        Update the entire search index
+        """
+        if run_which=='all' or run_which=='disqus':
+            try:
+                self.update_index_disqus(disqus_token, config)
+            except Exception as e:
+                print("ERROR: While re-indexing: failed to update Disqus comment threads")
+                print("-"*40)
+                print(repr(e))
+                print("-"*40)
+                print("Continuing...")
+                pass
+
+        if run_which=='all' or run_which=='emailthreads':
+            try:
+                self.update_index_emailthreads(groupsio_credentials, config)
+            except GroupsIOException as e:
+                print("ERROR: While re-indexing: failed to update Groups.io email threads, hit API rate limit")
+                print("-"*40)
+                print(repr(e))
+                print("-"*40)
+                print("Continuing...")
+                pass
+
+        if run_which=='all' or run_which=='ghfiles':
+            try:
+                self.update_index_ghfiles(gh_token,config)
+            except Exception as e:
+                print("ERROR: While re-indexing: failed to update Github files")
+                print("-"*40)
+                print(repr(e))
+                print("-"*40)
+                print("Continuing...")
+                pass
+
+        if run_which=='all' or run_which=='issues':
+            try:
+                self.update_index_issues(gh_token,config)
+            except Exception as e:
+                print("ERROR: While re-indexing: failed to update Github issues")
+                print("-"*40)
+                print(repr(e))
+                print("-"*40)
+                print("Continuing...")
+                pass
+
+        if run_which=='all' or run_which=='gdocs':
+            try:
+                self.update_index_gdocs(config)
+            except Exception as e:
+                print("ERROR: While re-indexing: failed to update Google Drive files")
+                print("-"*40)
+                print(repr(e))
+                print("-"*40)
+                print("Continuing...")
+                pass
+
+
     # ------------------------------
     # Create a schema and open a search index
     # on disk.
@@ -119,7 +188,8 @@ class Search:
             os.mkdir(index_folder)
 
         exists = index.exists_in(index_folder)
-        stemming_analyzer = StemmingAnalyzer()
+        #stemming_analyzer = StemmingAnalyzer()
+        stemming_analyzer = StemmingAnalyzer() | LowercaseFilter() | StopFilter()
 
         
         # ------------------------------
@@ -127,30 +197,38 @@ class Search:
         # is defined.
 
         schema = Schema(
-                id = ID(stored=True, unique=True),
+                id = fields.ID(stored=True, unique=True),
-                kind = ID(stored=True),
+                kind = fields.ID(stored=True),
+
+                created_time = fields.DATETIME(stored=True),
+                modified_time = fields.DATETIME(stored=True),
+                indexed_time = fields.DATETIME(stored=True),
+                
+                title = fields.TEXT(stored=True, field_boost=100.0),
+
+                url = fields.ID(stored=True),
                 
-                created_time = ID(stored=True),
+                mimetype = fields.TEXT(stored=True),
-                modified_time = ID(stored=True),
-                indexed_time = ID(stored=True),
 
-                title = TEXT(stored=True, field_boost=100.0),
+                owner_email = fields.ID(stored=True),
-                url = ID(stored=True, unique=True),
+                owner_name = fields.TEXT(stored=True),
 
-                mimetype=ID(stored=True),
+                # mainly for email threads, groups.io, hypothesis
-                owner_email=ID(stored=True),
+                group = fields.ID(stored=True),
-                owner_name=TEXT(stored=True),
 
-                repo_name=TEXT(stored=True),
+                repo_name = fields.TEXT(stored=True),
-                repo_url=ID(stored=True),
+                repo_url = fields.ID(stored=True),
+                github_user = fields.TEXT(stored=True),
 
-                github_user=TEXT(stored=True),
+                tags = fields.KEYWORD(commas=True,
+                                      stored=True,
+                                      lowercase=True),
 
                 # comments only
-                issue_title=TEXT(stored=True, field_boost=100.0),
+                issue_title = fields.TEXT(stored=True, field_boost=100.0),
-                issue_url=ID(stored=True),
+                issue_url = fields.ID(stored=True),
                 
-                content=TEXT(stored=True, analyzer=stemming_analyzer)
+                content = fields.TEXT(stored=True, analyzer=stemming_analyzer)
         )
 
 
@@ -190,17 +268,22 @@ class Search:
             writer.delete_by_term('id',item['id'])
 
             # Index a plain google drive file
+            created_time = dateutil.parser.parse(item['createdTime'])
+            modified_time = dateutil.parser.parse(item['modifiedTime'])
+            indexed_time = datetime.now().replace(microsecond=0)
+            try:
                 writer.add_document(
                         id = item['id'],
                         kind = 'gdoc',
-                    created_time = item['createdTime'],
+                        created_time = created_time,
-                    modified_time = item['modifiedTime'],
+                        modified_time = modified_time,
-                    indexed_time = datetime.now().replace(microsecond=0).isoformat(),
+                        indexed_time = indexed_time,
                         title = item['name'],
                         url = item['webViewLink'],
                         mimetype = mimetype,
                         owner_email = item['owners'][0]['emailAddress'],
                         owner_name = item['owners'][0]['displayName'],
+                        group='',
                         repo_name='',
                         repo_url='',
                         github_user='',
@@ -208,6 +291,9 @@ class Search:
                         issue_url='',
                         content = content
                 )
+            except ValueError as e:
+                print(repr(e))
+                print(" > XXXXXX Failed to index Google Drive file \"%s\""%(item['name']))
 
 
         else:
@@ -261,7 +347,7 @@ class Search:
                 )
                 assert output == ""
             except RuntimeError:
-                print(" > XXXXXX Failed to index document \"%s\""%(item['name']))
+                print(" > XXXXXX Failed to index Google Drive document \"%s\""%(item['name']))
 
 
             # If export was successful, read contents of markdown
@@ -289,17 +375,22 @@ class Search:
             else:
                 print(" > Creating a new record")
 
+            try:
+                created_time = dateutil.parser.parse(item['createdTime'])
+                modified_time = dateutil.parser.parse(item['modifiedTime'])
+                indexed_time = datetime.now()
                 writer.add_document(
                         id = item['id'],
                         kind = 'gdoc',
-                    created_time = item['createdTime'],
+                        created_time = created_time,
-                    modified_time = item['modifiedTime'],
+                        modified_time = modified_time,
-                    indexed_time = datetime.now().replace(microsecond=0).isoformat(),
+                        indexed_time = indexed_time,
                         title = item['name'],
                         url = item['webViewLink'],
                         mimetype = mimetype,
                         owner_email = item['owners'][0]['emailAddress'],
                         owner_name = item['owners'][0]['displayName'],
+                        group='',
                         repo_name='',
                         repo_url='',
                         github_user='',
@@ -307,6 +398,10 @@ class Search:
                         issue_url='',
                         content = content
                 )
+            except ValueError as e:
+                print(repr(e))
+                print(" > XXXXXX Failed to index Google Drive file \"%s\""%(item['name']))
+
 
 
 
@@ -340,13 +435,14 @@ class Search:
                 issue_comment_content += comment.body.rstrip()
                 issue_comment_content += "\n"
 
-        # Now create the actual search index record
+        # Now create the actual search index record.
-        created_time = clean_timestamp(issue.created_at)
-        modified_time = clean_timestamp(issue.updated_at)
-        indexed_time = clean_timestamp(datetime.now())
-
         # Add one document per issue thread,
         # containing entire text of thread.
+
+        created_time = issue.created_at
+        modified_time = issue.updated_at
+        indexed_time = datetime.now()
+        try:
             writer.add_document(
                     id = issue.html_url,
                     kind = 'issue',
@@ -358,6 +454,7 @@ class Search:
                     mimetype='',
                     owner_email='',
                     owner_name='',
+                    group='',
                     repo_name = repo_name,
                     repo_url = repo_url,
                     github_user = issue.user.login,
@@ -365,9 +462,15 @@ class Search:
                     issue_url = issue.html_url,
                     content = issue_comment_content
             )
+        except ValueError as e:
+            print(repr(e))
+            print(" > XXXXXX Failed to index Github issue \"%s\""%(issue.title))
 
 
 
+    # ------------------------------
+    # Add a single github file 
+    # to a search index.
 
     def add_ghfile(self, writer, d, gh_token, config, update=True):
         """
@@ -391,7 +494,8 @@ class Search:
             print(" > XXXXXXXX Failed to find file info.")
             return
 
-        indexed_time = clean_timestamp(datetime.now())
+
+        indexed_time = datetime.now()
 
         if fext in MARKDOWN_EXTS:
             print("Indexing markdown doc %s from repo %s"%(fname,repo_name))
@@ -420,17 +524,19 @@ class Search:
             usable_url = "https://github.com/%s/blob/master/%s"%(repo_name, fpath)
 
             # Now create the actual search index record
+            try:
                 writer.add_document(
                         id = fsha,
                         kind = 'markdown',
-                    created_time = '',
+                        created_time = None,
-                    modified_time = '',
+                        modified_time = None,
                         indexed_time = indexed_time,
                         title = fname,
                         url = usable_url,
                         mimetype='',
                         owner_email='',
                         owner_name='',
+                        group='',
                         repo_name = repo_name,
                         repo_url = repo_url,
                         github_user = '',
@@ -438,6 +544,11 @@ class Search:
                         issue_url = '',
                         content = content
                 )
+            except ValueError as e:
+                print(repr(e))
+                print(" > XXXXXX Failed to index Github markdown file \"%s\""%(fname))
+
+
 
         else:
             print("Indexing github file %s from repo %s"%(fname,repo_name))
@@ -445,17 +556,19 @@ class Search:
             key = fname+"_"+fsha
 
             # Now create the actual search index record
+            try:
                 writer.add_document(
                         id = key,
                         kind = 'ghfile',
-                    created_time = '',
+                        created_time = None,
-                    modified_time = '',
+                        modified_time = None,
                         indexed_time = indexed_time,
                         title = fname,
                         url = repo_url,
                         mimetype='',
                         owner_email='',
                         owner_name='',
+                        group='',
                         repo_name = repo_name,
                         repo_url = repo_url,
                         github_user = '',
@@ -463,6 +576,98 @@ class Search:
                         issue_url = '',
                         content = ''
                 )
+            except ValueError as e:
+                print(repr(e))
+                print(" > XXXXXX Failed to index Github file \"%s\""%(fname))
+
+
+
+
+    # ------------------------------
+    # Add a single github file 
+    # to a search index.
+
+    def add_emailthread(self, writer, d, config, update=True):
+        """
+        Use a Groups.io email thread record to add 
+        an email thread to the search index.
+        """
+        if 'created_time' in d.keys() and d['created_time'] is not None:
+            created_time = d['created_time']
+        else:
+            created_time = None
+
+        if 'modified_time' in d.keys() and d['modified_time'] is not None:
+            modified_time = d['modified_time']
+        else:
+            modified_time = None
+
+        indexed_time = datetime.now()
+
+        # Now create the actual search index record
+        try:
+            writer.add_document(
+                    id = d['permalink'],
+                    kind = 'emailthread',
+                    created_time = created_time,
+                    modified_time = modified_time,
+                    indexed_time = indexed_time,
+                    title = d['subject'],
+                    url = d['permalink'],
+                    mimetype='',
+                    owner_email='',
+                    owner_name=d['original_sender'],
+                    group=d['subgroup'],
+                    repo_name = '',
+                    repo_url = '',
+                    github_user = '',
+                    issue_title = '',
+                    issue_url = '',
+                    content = d['content']
+            )
+        except ValueError as e:
+            print(repr(e))
+            print(" > XXXXXX Failed to index Groups.io thread \"%s\""%(d['subject']))
+
+
+    # ------------------------------
+    # Add a single disqus comment thread
+    # to the search index.
+
+    def add_disqusthread(self, writer, d, config, update=True):
+        """
+        Use a disqus comment thread record
+        to add a disqus comment thread to the
+        search index.
+        """
+        indexed_time = datetime.now()
+
+        # created_time is already a timestamp
+
+        # Now create the actual search index record
+        try:
+            writer.add_document(
+                    id = d['id'],
+                    kind = 'disqus',
+                    created_time = d['created_time'],
+                    modified_time = None,
+                    indexed_time = indexed_time,
+                    title = d['title'],
+                    url = d['link'],
+                    mimetype='',
+                    owner_email='',
+                    owner_name='',
+                    repo_name = '',
+                    repo_url = '',
+                    github_user = '',
+                    issue_title = '',
+                    issue_url = '',
+                    content = d['content']
+            )
+        except ValueError as e:
+            print(repr(e))
+            print(" > XXXXXX Failed to index Disqus comment thread \"%s\""%(d['title']))
+
 
 
 
@@ -489,9 +694,8 @@ class Search:
         # Updated algorithm:
         # - get set of indexed ids
         # - get set of remote ids
-        # - drop indexed ids not in remote ids
+        # - drop all indexed ids
         # - index all remote ids
-        # - add hash check in add_
 
 
         # Get the set of indexed ids:
@@ -541,7 +745,7 @@ class Search:
             
             ## Shorter:
             #break
-            # Longer:
+            ## Longer:
             if nextPageToken is None:
                 break
 
@@ -551,7 +755,7 @@ class Search:
         temp_dir = tempfile.mkdtemp(dir=os.getcwd())
         print("Temporary directory: %s"%(temp_dir))
 
-
+        try:
 
             # Drop any id in indexed_ids
             # not in remote_ids
@@ -579,12 +783,19 @@ class Search:
                 self.add_drive_file(writer, item, temp_dir, config, update=False)
                 count += 1
 
+        except Exception as e:
+            print("ERROR: While adding Google Drive files to search index")
+            print("-"*40)
+            print(repr(e))
+            print("-"*40)
+            print("Continuing...")
+            pass
 
         print("Cleaning temporary directory: %s"%(temp_dir))
         subprocess.call(['rm','-fr',temp_dir])
 
         writer.commit()
-        print("Done, updated %d documents in the index" % count)
+        print("Done, updated %d Google Drive files in the index" % count)
 
 
     # ------------------------------
@@ -595,12 +806,6 @@ class Search:
         Update the search index using a collection of 
         Github repo issues and comments.
         """
-        # Updated algorithm:
-        # - get set of indexed ids
-        # - get set of remote ids
-        # - drop indexed ids not in remote ids
-        # - index all remote ids
-
         # Get the set of indexed ids:
         # ------
         indexed_issues = set()
@@ -655,35 +860,20 @@ class Search:
         writer = self.ix.writer()
         count = 0
 
-        # Drop any issues in indexed_issues
+        # Drop issues in indexed_issues
-        # not in remote_issues
+        for drop_issue in indexed_issues:
-        drop_issues = indexed_issues - remote_issues
-        for drop_issue in drop_issues:
             writer.delete_by_term('id',drop_issue)
 
 
-        # Update any issue in indexed_issues
+        # Add any issue in remote_issues
-        # and in remote_issues
+        for add_issue in remote_issues:
-        update_issues = indexed_issues & remote_issues
-        for update_issue in update_issues:
-            # cop out
-            writer.delete_by_term('id',update_issue)
-            item = full_items[update_issue]
-            self.add_issue(writer, item, gh_token, config, update=True)
-            count += 1
-
-
-        # Add any issue not in indexed_issues
-        # and in remote_issues
-        add_issues = remote_issues - indexed_issues
-        for add_issue in add_issues:
             item = full_items[add_issue]
             self.add_issue(writer, item, gh_token, config, update=False)
             count += 1
 
 
         writer.commit()
-        print("Done, updated %d documents in the index" % count)
+        print("Done, updated %d Github issues in the index" % count)
 
 
 
@@ -696,17 +886,11 @@ class Search:
         files (and, separately, Markdown files) from 
         a Github repo.
         """
-        # Updated algorithm:
-        # - get set of indexed ids
-        # - get set of remote ids
-        # - drop indexed ids not in remote ids
-        # - index all remote ids
-
         # Get the set of indexed ids:
         # ------
         indexed_ids = set()
         p = QueryParser("kind", schema=self.ix.schema)
-        q = p.parse("ghfiles")
+        q = p.parse("ghfile")
         with self.ix.searcher() as s:
             results = s.search(q,limit=None)
             for result in results:
@@ -768,7 +952,15 @@ class Search:
                 fpath = d['path']
                 _, fname = os.path.split(fpath)
                 _, fext = os.path.splitext(fpath)
+                fpathpieces = fpath.split('/')
+
+                ignore_file = fname[0]=='.' or fname[0]=='_'
+                ignore_dir = False
+                for piece in fpathpieces:
+                    if piece[0]=='.' or piece[0]=='_':
+                        ignore_dir = True
 
+                if not ignore_file and not ignore_dir:
                     key = d['sha']
 
                     d['org'] = this_org
@@ -782,27 +974,13 @@ class Search:
         count = 0
 
         # Drop any id in indexed_ids
-        # not in remote_ids
+        for drop_id in indexed_ids:
-        drop_ids = indexed_ids - remote_ids
-        for drop_id in drop_ids:
             writer.delete_by_term('id',drop_id)
 
 
-        # Update any id in indexed_ids
+        # Add any issue in remote_ids
         # and in remote_ids
-        update_ids = indexed_ids & remote_ids
+        for add_id in remote_ids:
-        for update_id in update_ids:
-            # cop out: just delete and re-add
-            writer.delete_by_term('id',update_id)
-            item = full_items[update_id]
-            self.add_ghfile(writer, item, gh_token, config, update=True)
-            count += 1
-
-
-        # Add any issue not in indexed_ids
-        # and in remote_ids
-        add_ids = remote_ids - indexed_ids
-        for add_id in add_ids:
             item = full_items[add_id]
             self.add_ghfile(writer, item, gh_token, config, update=False)
             count += 1
@@ -817,27 +995,139 @@ class Search:
     # Groups.io Emails
 
 
-    def update_index_groupsioemails(self, groupsio_token, config):
+    def update_index_emailthreads(self, groupsio_token, config):
         """
         Update the search index using the email archives
-        of groups.io groups.
+        of groups.io groups. This method looks deceptively
+        simple, all the logic is hidden in the spider
+        (groupsio_util.py).
 
-        This requires the use of a spider.
         RELEASE THE SPIDER!!!
         """
+        # Get the set of indexed ids:
+        # ------
+        indexed_ids = set()
+        p = QueryParser("kind", schema=self.ix.schema)
+        q = p.parse("emailthread")
+        with self.ix.searcher() as s:
+            results = s.search(q,limit=None)
+            for result in results:
+                indexed_ids.add(result['id'])
+
+        # Get the set of remote ids:
+        # ------
         spider = GroupsIOArchivesCrawler(groupsio_token,'dcppc')
 
-        # - ask spider to crawl the archives
+        # ask spider to crawl the archives
         spider.crawl_group_archives()
 
-        # - ask spider for list of all email records
+        # now spider.archives is a dictionary
-        #   - 1 email = 1 dictionary
+        # with one key per thread ID,
-        #   - email records compiled by the spider
+        # and a value set to the payload:
+        #   '<thread-id>'  : {
+        #                       'permalink' : permalink,
+        #                       'subject' : subject,
+        #                       'original_sender' : original_sender,
+        #                       'content' : full_content
+        #                    }
+        #
+        # It is hard to reliably extract more information
+        # than that from the email thread.
+
+        writer = self.ix.writer()
+        count = 0
+
+        # archives is a dictionary
+        # keys are IDs (urls)
+        # values are dictionaries
         archives = spider.get_archives()
 
-        # - email object is sent off to add email method
+        # Start by collecting all the things
+        remote_ids = set()
+        for k in archives.keys():
+            remote_ids.add(k)
+
+        # drop indexed_ids
+        for drop_id in indexed_ids:
+            writer.delete_by_term('id',drop_id)
+
+        # add remote_ids
+        for add_id in remote_ids:
+            item = archives[add_id]
+            self.add_emailthread(writer, item, config, update=False)
+            count += 1
+
+        writer.commit()
+        print("Done, updated %d Groups.io email threads in the index" % count)
+
+
+
+    # ------------------------------
+    # Disqus Comments
+
+
+    def update_index_disqus(self, disqus_token, config):
+        """
+        Update the search index using a collection of 
+        Disqus comment threads from the dcppc-internal 
+        forum.
+        """
+        # Updated algorithm:
+        # - get set of indexed ids
+        # - get set of remote ids
+        # - drop all indexed ids
+        # - index all remote ids
+
+        # Get the set of indexed ids:
+        # --------------------
+        indexed_ids = set()
+        p = QueryParser("kind", schema=self.ix.schema)
+        q = p.parse("disqus")
+        with self.ix.searcher() as s:
+            results = s.search(q,limit=None)
+            for result in results:
+                indexed_ids.add(result['id'])
+
+        # Get the set of remote ids:
+        # ------
+        spider = DisqusCrawler(disqus_token,'dcppc-internal')
+
+        # ask spider to crawl disqus comments
+        spider.crawl_threads()
+
+        # spider.comments will be a dictionary
+        # with keys as thread IDs and values as
+        # a dictionary item
+
+        writer = self.ix.writer()
+        count = 0
+
+        # archives is a dictionary
+        # keys are IDs (urls)
+        # values are dictionaries
+        threads = spider.get_threads()
+
+        # Start by collecting all the things
+        remote_ids = set()
+        for k in threads.keys():
+            remote_ids.add(k)
+
+        # drop indexed_ids
+        for drop_id in indexed_ids:
+            writer.delete_by_term('id',drop_id)
+
+        # add remote_ids
+        for add_id in remote_ids:
+            item = threads[add_id]
+            self.add_disqusthread(writer, item, config, update=False)
+            count += 1
+
+        writer.commit()
+        print("Done, updated %d Disqus comment threads in the index" % count)
+
+
+
 
-        print("Finished indexing groups.io emails")
 
 
     # ---------------------------------
@@ -875,9 +1165,9 @@ class Search:
             sr.id = r['id']
             sr.kind = r['kind']
 
-            sr.created_time = r['created_time']
+            sr.created_time =  datetime.strftime(r['created_time'],  "%Y-%m-%d %I:%M %p")
-            sr.modified_time = r['modified_time']
+            sr.modified_time = datetime.strftime(r['modified_time'], "%Y-%m-%d %I:%M %p")
-            sr.indexed_time = r['indexed_time']
+            sr.indexed_time =  datetime.strftime(r['indexed_time'],  "%Y-%m-%d %I:%M %p")
 
             sr.title = r['title']
             sr.url = r['url']
@@ -887,6 +1177,8 @@ class Search:
             sr.owner_email = r['owner_email']
             sr.owner_name = r['owner_name']
 
+            sr.group = r['group']
+
             sr.repo_name = r['repo_name']
             sr.repo_url = r['repo_url']
 
@@ -914,33 +1206,6 @@ class Search:
 
 
 
-    def search(self, query_list, fields=None):
-
-        with self.ix.searcher() as searcher:
-            query_string = " ".join(query_list)
-            query = None
-            if "\"" in query_string or ":" in query_string:
-                query = QueryParser("content", self.schema).parse(query_string)
-            elif len(fields) == 1 and fields[0] == "filename":
-                pass
-            elif len(fields) == 2:
-                pass
-            else:
-                # If the user does not specify a field,
-                # these are the fields that are actually searched
-                fields = ['title',
-                          'content']
-            if not query:
-                query = MultifieldParser(fields, schema=self.ix.schema).parse(query_string)
-            parsed_query = "%s" % query
-            print("query: %s" % parsed_query)
-            results = searcher.search(query, terms=False, scored=True, groupedby="kind")
-            search_result = self.create_search_result(results)
-
-        return parsed_query, search_result
-
-
-
     def cap(self, s, l):
         return s if len(s) <= l else s[0:l - 3] + '...'
 
@@ -952,6 +1217,8 @@ class Search:
                 "issue" : None,
                 "ghfile" : None,
                 "markdown" : None,
+                "emailthread" : None,
+                "disqus" : None,
                 "total" : None
         }
         for key in counts.keys():
@@ -964,6 +1231,89 @@ class Search:
 
         return counts
 
+
+    def get_list(self,doctype):
+        """
+        Get a listing of all files, 
+        so we can construct the page that 
+        lists everyone and everything that
+        centillion indexes.
+        """
+        # Unfortunately, we have to treat
+        # each doctype separately, b/c of
+        # what is most relevant to display
+        # in the everything-list.
+        item_keys=''
+        if doctype=='gdoc':
+            item_keys = ['title','owner_name','url','mimetype','created_time','modified_time']
+        elif doctype=='issue':
+            item_keys = ['title','repo_name','repo_url','url','created_time','modified_time']
+        elif doctype=='emailthread':
+            item_keys = ['title','owner_name','url','group','created_time','modified_time']
+        elif doctype=='disqus':
+            item_keys = ['title','created_time','url']
+        elif doctype=='ghfile':
+            item_keys = ['title','repo_name','repo_url','url']
+        elif doctype=='markdown':
+            item_keys = ['title','repo_name','repo_url','url']
+        else:
+            raise Exception("Could not find document of type %s"%(doctype))
+
+        json_results = []
+
+        p = QueryParser("kind", schema=self.ix.schema)
+        q = p.parse(doctype)
+        with self.ix.searcher() as s:
+            results = s.search(q,limit=None)
+            for r in results:
+                d = {}
+                for k in item_keys:
+                    d[k] = r[k]
+                json_results.append(d)
+
+        return json_results
+
+
+
+    def search(self, query_list, fields=None):
+
+        with self.ix.searcher() as searcher:
+            query_string = " ".join(query_list)
+            query = None
+            if ":" in query_string:
+
+                #query = QueryParser("content", 
+                #                    self.schema
+                #).parse(query_string)
+                query = QueryParser("content", 
+                                    self.schema,
+                                    termclass=query.Variations
+                )
+                query.add_plugin(DateParserPlugin(free=True))
+                query = query.parse(query_string)
+            elif len(fields) == 1 and fields[0] == "filename":
+                pass
+            elif len(fields) == 2:
+                pass
+            else:
+                # If the user does not specify a field,
+                # these are the fields that are actually searched
+                fields = ['title', 'content','owner_name','owner_email','url','created_date','modified_date']
+            if not query:
+                query = MultifieldParser(fields, schema=self.ix.schema)
+                query.add_plugin(DateParserPlugin(free=True))
+                query = query.parse(query_string)
+                #query = MultifieldParser(fields, schema=self.ix.schema).parse(query_string) 
+            parsed_query = "%s" % query
+            print("query: %s" % parsed_query)
+            results = searcher.search(query, terms=False, scored=True, groupedby="kind")
+            search_result = self.create_search_result(results)
+
+        return parsed_query, search_result
+
+
+
+
 if __name__ == "__main__":
 
     raise Exception("Error: main method not implemented (fix groupsio credentials first)")

config_centillion.example.py

@@ -1,4 +1,4 @@
-{
+config = {
     "repositories" : [
         "dcppc/project-management",
         "dcppc/nih-demo-meetings",
@@ -25,3 +25,4 @@
         "dcppc/centillion"
     ]
 }
+

config_centillion.py

@@ -0,0 +1,28 @@
+config = {
+    "repositories" : [
+        "dcppc/project-management",
+        "dcppc/nih-demo-meetings",
+        "dcppc/internal",
+        "dcppc/organize",
+        "dcppc/dcppc-bot",
+        "dcppc/full-stacks",
+        "dcppc/design-guidelines-discuss",
+        "dcppc/dcppc-deliverables",
+        "dcppc/dcppc-milestones",
+        "dcppc/crosscut-metadata",
+        "dcppc/lucky-penny",
+        "dcppc/dcppc-workshops",
+        "dcppc/metadata-matrix",
+        "dcppc/data-stewards",
+        "dcppc/dcppc-phase1-demos",
+        "dcppc/apis",
+        "dcppc/2018-june-workshop",
+        "dcppc/2018-july-workshop",
+        "dcppc/2018-august-workshop",
+        "dcppc/2018-september-workshop",
+        "dcppc/design-guidelines",
+        "dcppc/2018-may-workshop",
+        "dcppc/centillion"
+    ]
+}
+

config_flask.example.py

@@ -1,20 +1,38 @@
-# Location of index file
+######################################
-INDEX_DIR = "search_index"
+# github oauth
-
-# oauth client deets
 GITHUB_OAUTH_CLIENT_ID = "XXX"
 GITHUB_OAUTH_CLIENT_SECRET = "YYY"
-GITHUB_TOKEN = "ZZZ"
+
+######################################
+# github acces token
+GITHUB_TOKEN = "XXX"
+
+######################################
+# groups.io
+GROUPSIO_TOKEN = "XXXXX"
+GROUPSIO_USERNAME = "XXXXX"
+GROUPSIO_PASSWORD = "XXXXX"
+
+######################################
+# Disqus API public key
+DISQUS_TOKEN = "XXXXX"
+
+######################################
+# everything else
+
+# Location of index file
+INDEX_DIR = "search_index"
 
 # More information footer: Repository label
-FOOTER_REPO_ORG = "charlesreid1"
+FOOTER_REPO_ORG = "dcppc"
 FOOTER_REPO_NAME = "centillion"
 
 # Toggle to show Whoosh parsed query
 SHOW_PARSED_QUERY=True
 
-TAGLINE = "Search All The Things"
+TAGLINE = "Search the Data Commons"
 
 # Flask settings
 DEBUG = True
-SECRET_KEY = 'WWWWW'
+SECRET_KEY = 'XXXXX'
+

disqus_util.py

@@ -0,0 +1,154 @@
+import os, re
+import requests
+import json
+import dateutil.parser
+
+from pprint import pprint
+
+"""
+Convenience class wrapper for Disqus comments.
+
+This requires that the user provide either their
+API OAuth application credentials (in which case
+a user needs to authenticate with the application
+so it can access the comments that they can see)
+or user credentials from a previous login.
+"""
+
+class DisqusCrawler(object):
+
+    def __init__(self,
+                 credentials,
+                 group_name):
+
+        self.credentials = credentials
+        self.group_name = group_name
+        self.crawled_comments = False
+        self.threads = None
+
+
+    def get_threads(self):
+        """
+        Return a list of dictionaries containing
+        entries for each comment thread in the given 
+        disqus forum.
+        """
+        return self.threads
+
+
+    def crawl_threads(self):
+        """
+        This will use the API to get every thread,
+        and will iterate through every thread to 
+        get every comment thread. 
+        """
+        # The money shot
+        threads = {}
+
+        # list all threads
+        list_threads_url = 'https://disqus.com/api/3.0/threads/list.json'
+
+        # list all posts (comments)
+        list_posts_url = 'https://disqus.com/api/3.0/threads/listPosts.json'
+
+        base_params = dict(
+                api_key=self.credentials,
+                forum=self.group_name
+        )
+
+        # prepare url params
+        params = {}
+        for k in base_params.keys():
+            params[k] = base_params[k]
+
+        # make api call (first loop in fencepost)
+        results = requests.request('GET', list_threads_url, params=params).json()
+        cursor = results['cursor']
+        responses = results['response']
+
+        while True:
+
+            for response in responses:
+                if '127.0.0.1' not in response['link'] and 'localhost' not in response['link']:
+
+                    # Save thread info
+                    thread_id = response['id']
+                    thread_count = response['posts']
+
+                    print("Working on thread %s (%d posts)"%(thread_id,thread_count))
+                    if thread_count > 0:
+
+                        # prepare url params
+                        params_comments = {}
+                        for k in base_params.keys():
+                            params_comments[k] = base_params[k]
+
+                        params_comments['thread'] = thread_id
+
+                        # make api call
+                        results_comments = requests.request('GET', list_posts_url, params=params_comments).json()
+                        cursor_comments = results_comments['cursor']
+                        responses_comments = results_comments['response']
+
+                        # Save comments for this thread
+                        thread_comments = []
+
+                        while True:
+                            for comment in responses_comments:
+                                # Save comment info
+                                print("    + %s"%(comment['message']))
+                                thread_comments.append(comment['message'])
+                       
+                            if cursor_comments['hasNext']:
+                       
+                                # Prepare for the next URL call
+                                params_comments = {}
+                                for k in base_params.keys():
+                                    params_comments[k] = base_params[k]
+                                params_comments['thread'] = thread_id
+                                params_comments['cursor'] = cursor_comments['next']
+                       
+                                # Make the next URL call
+                                results_comments = requests.request('GET', list_posts_url, params=params_comments).json()
+                                cursor_comments = results_comments['cursor']
+                                responses_comments = results_comments['response']
+                       
+                            else:
+                               break
+
+                        link = response['link']
+                        clean_link = re.sub('data-commons.us','nihdatacommons.us',link)
+                        clean_link += "#disqus_comments"
+
+                        # Finished working on thread.
+
+                        # We need to make this value a dictionary
+                        thread_info = dict(
+                                id = response['id'],
+                                created_time = dateutil.parser.parse(response['createdAt']),
+                                title = response['title'],
+                                forum = response['forum'],
+                                link = clean_link,
+                                content = "\n\n-----".join(thread_comments)
+                        )
+                        threads[thread_id] = thread_info
+
+
+            if 'hasNext' in cursor.keys() and cursor['hasNext']:
+
+                # Prepare for next URL call
+                params = {}
+                for k in base_params.keys():
+                    params[k] = base_params[k]
+                params['cursor'] = cursor['next']
+
+                # Make the next URL call
+                results = requests.request('GET', list_threads_url, params=params).json()
+                cursor = results['cursor']
+                responses = results['response']
+
+            else:
+                break
+
+        self.threads = threads
+

gdrive_util.py

@@ -51,3 +51,6 @@ class GDrive(object):
         service = build('drive', 'v3', http=creds.authorize(Http()))
         return service
 
+if __name__=="__main__":
+    g = GDrive()
+    s = g.get_service()

get_centillion_config.py

@@ -1,12 +0,0 @@
-import json
-
-CONFIG_FILE = 'config_centillion.json'
-
-def get_centillion_config(filename=CONFIG_FILE):
-    """
-    Load the centillion configuration
-    """
-    with open(filename,'r') as f:
-        d = json.load(f)
-    return d
-

groupsio_util.py

@@ -1,5 +1,10 @@
 import requests, os, re
 from bs4 import BeautifulSoup
+import dateutil.parser
+import datetime
+
+class GroupsIOException(Exception):
+    pass
 
 class GroupsIOArchivesCrawler(object):
     """
@@ -59,6 +64,10 @@ class GroupsIOArchivesCrawler(object):
             v = re.sub(r'dcppc\+','',group['name'])
             subgroups[k] = v
 
+            ## Short circuit
+            ## for debugging purposes
+            #break
+
         return subgroups
 
 
@@ -67,6 +76,8 @@ class GroupsIOArchivesCrawler(object):
         Spider will crawl the email archives of the entire group
         by crawling the email archives of each subgroup.
         """
+        self.archives = {}
+
         subgroups = self.get_subgroups_list()
 
         # ------------------------------
@@ -98,9 +109,9 @@ class GroupsIOArchivesCrawler(object):
         # Done. archives are now tucked away
         # in the variable self.archives
         # 
-        # self.archives is a list of dictionaries,
+        # self.archives is a dictionary of dictionaries,
-        # with each dictionary containing info about
+        # with each key a URL and each value a dictionary
-        # a topic/email thread in a subgroup.
+        # containing info about a thread.
         # ------------------------------
 
 
@@ -138,8 +149,6 @@ class GroupsIOArchivesCrawler(object):
         list of dictionaries.
 
         """
-        self.archives = []
-
         prefix = "https://{group}.groups.io".format(group=self.group_name)
 
         url = self.url.format(group=self.group_name, 
@@ -244,7 +253,7 @@ class GroupsIOArchivesCrawler(object):
             subject = soup.find('title').text
 
             # Extract information for the schema:
-            # - permalink for thread (done)
+            # - permalink for thread (done above)
             # - subject/title (done)
             # - original sender email/name (done)
             # - content (done)
@@ -259,11 +268,35 @@ class GroupsIOArchivesCrawler(object):
                     pass
                 else:
                     # found an email!
-                    # this is a maze, thanks groups.io
+                    # this is a maze, not amazing.
+                    # thanks groups.io!
                     td = tr.find('td')
-                    divrow = td.find('div',{'class':'row'}).find('div',{'class':'pull-left'})
+
+                    sender_divrow = td.find('div',{'class':'row'})
+                    sender_divrow = sender_divrow.find('div',{'class':'pull-left'})
+                    if (i+1)==1:
+                        original_sender = sender_divrow.text.strip()
+
+                    date_divrow = td.find('div',{'class':'row'})
+                    date_divrow = date_divrow.find('div',{'class':'pull-right'})
+                    date_divrow = date_divrow.find('font',{'class':'text-muted'})
+                    date_divrow = date_divrow.find('script').text
+                    try:
+                        time_seconds = re.search(' [0-9]{1,} ',date_divrow).group(0)
+                        time_seconds = time_seconds.strip()
+                        # Thanks groups.io for the weird date formatting
+                        time_seconds = time_seconds[:10]
+                        mmicro_seconds = time_seconds[10:]
                         if (i+1)==1:
-                        original_sender = divrow.text.strip()
+                            created_time  = datetime.datetime.utcfromtimestamp(int(time_seconds))
+                            modified_time = datetime.datetime.utcfromtimestamp(int(time_seconds))
+                        else:
+                            modified_time = datetime.datetime.utcfromtimestamp(int(time_seconds))
+
+                    except AttributeError:
+                        created_time = None
+                        modified_time = None
+
                     for div in td.find_all('div'):
                         if div.has_attr('id'):
 
@@ -292,19 +325,16 @@ class GroupsIOArchivesCrawler(object):
 
             thread = {
                     'permalink' : permalink,
+                    'created_time' : created_time,
+                    'modified_time' : modified_time,
                     'subject' : subject,
+                    'subgroup' : subgroup_name,
                     'original_sender' : original_sender,
                     'content' : full_content
             }
 
-            print('*'*40)
+            print(" + Archiving thread: %s"%(thread['subject']))
-            for k in thread.keys():
+            self.archives[permalink] = thread
-                if k=='content':
-                    pass
-                else:
-                    print("%s : %s"%(k,thread[k]))
-            print('*'*40)
-            self.archives.append(thread)
 
 
     def extract_archive_page_items_(self, response):
@@ -319,15 +349,17 @@ class GroupsIOArchivesCrawler(object):
         soup = BeautifulSoup(response.content,"html.parser")
         rows = soup.find_all('tr',{'class':'test'})
         if 'rate limited' in soup.text:
-            raise Exception("Error: rate limit in place for Groups.io")
+            raise GroupsIOException("Error: rate limit in place for Groups.io")
 
         results = []
         for row in rows:
-            # We don't care about anything except title and ugly link
+            # This is where we extract
+            # a list of thread titles 
+            # and corresponding links.
             subject = row.find('span',{'class':'subject'})
             title = subject.get_text()
             link = row.find('a')['href']
-            print(title)
+
             results.append((title,link))
 
         return results
@@ -375,7 +407,7 @@ class GroupsIOArchivesCrawler(object):
         
         if csrf=='':
             err = "ERROR: Could not find csrf token on page."
-            raise Exception(err)
+            raise GroupsIOException(err)
     
         return csrf
 

hypothesis_util.py

@@ -0,0 +1,89 @@
+import requests
+import json
+import os
+
+def get_headers():
+
+    if 'HYPOTHESIS_TOKEN' in os.environ:
+        token = os.environ['HYPOTHESIS_TOKEN']
+    else:
+        raise Exception("Need to specify Hypothesis token with HYPOTHESIS_TOKEN env var")
+    
+    auth_header = 'Bearer %s'%(token)
+
+    return {'Authorization': auth_header}
+
+
+def basic_auth():
+
+    url = ' https://hypothes.is/api'
+
+    # Get the authorization header
+    headers = get_headers()
+
+    # Make the request
+    response = requests.get(url, headers=headers)
+
+    if response.status_code==200:
+
+        # Interpret results as JSON
+        dat = response.json()
+        print(json.dumps(dat, indent=4))
+
+    else:
+
+        print("Response status code was not OK: %d"%(response.status_code))
+
+
+def list_annotations():
+    # kEaohJC9Eeiy_UOozkpkyA
+
+    url = 'https://hypothes.is/api/annotations/kEaohJC9Eeiy_UOozkpkyA'
+
+    # Get the authorization header
+    headers = get_headers()
+
+    # Make the request
+    response = requests.get(url, headers=headers)
+
+    if response.status_code==200:
+
+        # Interpret results as JSON
+        dat = response.json()
+        print(json.dumps(dat, indent=4))
+
+    else:
+
+        print("Response status code was not OK: %d"%(response.status_code))
+
+
+def search_annotations():
+    url = ' https://hypothes.is/api/search'
+
+    # Get the authorization header
+    headers = get_headers()
+
+    # Set query params
+    params = dict(
+            url = '*pilot.nihdatacommons.us*',
+            limit = 200
+    )
+    #http://pilot.nihdatacommons.us/organize/CopperInternalDeliveryWorkFlow/',
+
+    # Make the request
+    response = requests.get(url, headers=headers, params=params)
+
+    if response.status_code==200:
+
+        # Interpret results as JSON
+        dat = response.json()
+        print(json.dumps(dat, indent=4))
+
+    else:
+
+        print("Response status code was not OK: %d"%(response.status_code))
+
+
+if __name__=="__main__":
+    search_annotations()
+

requirements.txt

@@ -11,3 +11,4 @@ requests>=2.19
 pandoc>=1.0
 flask-dance>=1.0.0
 beautifulsoup4>=4.6
+python-dateutil>=2.6

static/dataTables.bootstrap.min.css

@@ -0,0 +1 @@
+table.dataTable{clear:both;margin-top:6px !important;margin-bottom:6px !important;max-width:none !important;border-collapse:separate !important}table.dataTable td,table.dataTable th{-webkit-box-sizing:content-box;box-sizing:content-box}table.dataTable td.dataTables_empty,table.dataTable th.dataTables_empty{text-align:center}table.dataTable.nowrap th,table.dataTable.nowrap td{white-space:nowrap}div.dataTables_wrapper div.dataTables_length label{font-weight:normal;text-align:left;white-space:nowrap}div.dataTables_wrapper div.dataTables_length select{width:75px;display:inline-block}div.dataTables_wrapper div.dataTables_filter{text-align:right}div.dataTables_wrapper div.dataTables_filter label{font-weight:normal;white-space:nowrap;text-align:left}div.dataTables_wrapper div.dataTables_filter input{margin-left:0.5em;display:inline-block;width:auto}div.dataTables_wrapper div.dataTables_info{padding-top:8px;white-space:nowrap}div.dataTables_wrapper div.dataTables_paginate{margin:0;white-space:nowrap;text-align:right}div.dataTables_wrapper div.dataTables_paginate ul.pagination{margin:2px 0;white-space:nowrap}div.dataTables_wrapper div.dataTables_processing{position:absolute;top:50%;left:50%;width:200px;margin-left:-100px;margin-top:-26px;text-align:center;padding:1em 0}table.dataTable thead>tr>th.sorting_asc,table.dataTable thead>tr>th.sorting_desc,table.dataTable thead>tr>th.sorting,table.dataTable thead>tr>td.sorting_asc,table.dataTable thead>tr>td.sorting_desc,table.dataTable thead>tr>td.sorting{padding-right:30px}table.dataTable thead>tr>th:active,table.dataTable thead>tr>td:active{outline:none}table.dataTable thead .sorting,table.dataTable thead .sorting_asc,table.dataTable thead .sorting_desc,table.dataTable thead .sorting_asc_disabled,table.dataTable thead .sorting_desc_disabled{cursor:pointer;position:relative}table.dataTable thead .sorting:after,table.dataTable thead .sorting_asc:after,table.dataTable thead .sorting_desc:after,table.dataTable thead .sorting_asc_disabled:after,table.dataTable thead .sorting_desc_disabled:after{position:absolute;bottom:8px;right:8px;display:block;font-family:'Glyphicons Halflings';opacity:0.5}table.dataTable thead .sorting:after{opacity:0.2;content:"---"}table.dataTable thead .sorting_asc:after{content:"/\\"}table.dataTable thead .sorting_desc:after{content:"\\/"}table.dataTable thead .sorting_asc_disabled:after,table.dataTable thead .sorting_desc_disabled:after{color:#eee}div.dataTables_scrollHead table.dataTable{margin-bottom:0 !important}div.dataTables_scrollBody>table{border-top:none;margin-top:0 !important;margin-bottom:0 !important}div.dataTables_scrollBody>table>thead .sorting:after,div.dataTables_scrollBody>table>thead .sorting_asc:after,div.dataTables_scrollBody>table>thead .sorting_desc:after{display:none}div.dataTables_scrollBody>table>tbody>tr:first-child>th,div.dataTables_scrollBody>table>tbody>tr:first-child>td{border-top:none}div.dataTables_scrollFoot>.dataTables_scrollFootInner{box-sizing:content-box}div.dataTables_scrollFoot>.dataTables_scrollFootInner>table{margin-top:0 !important;border-top:none}@media screen and (max-width: 767px){div.dataTables_wrapper div.dataTables_length,div.dataTables_wrapper div.dataTables_filter,div.dataTables_wrapper div.dataTables_info,div.dataTables_wrapper div.dataTables_paginate{text-align:center}}table.dataTable.table-condensed>thead>tr>th{padding-right:20px}table.dataTable.table-condensed .sorting:after,table.dataTable.table-condensed .sorting_asc:after,table.dataTable.table-condensed .sorting_desc:after{top:6px;right:6px}table.table-bordered.dataTable th,table.table-bordered.dataTable td{border-left-width:0}table.table-bordered.dataTable th:last-child,table.table-bordered.dataTable th:last-child,table.table-bordered.dataTable td:last-child,table.table-bordered.dataTable td:last-child{border-right-width:0}table.table-bordered.dataTable tbody th,table.table-bordered.dataTable tbody td{border-bottom-width:0}div.dataTables_scrollHead table.table-bordered{border-bottom-width:0}div.table-responsive>div.dataTables_wrapper>div.row{margin:0}div.table-responsive>div.dataTables_wrapper>div.row>div[class^="col-"]:first-child{padding-left:0}div.table-responsive>div.dataTables_wrapper>div.row>div[class^="col-"]:last-child{padding-right:0}

static/dataTables.bootstrap.min.js

@@ -0,0 +1,8 @@
+/*!
+ DataTables Bootstrap 3 integration
+ ©2011-2014 SpryMedia Ltd - datatables.net/license
+*/
+(function(){var f=function(c,b){c.extend(!0,b.defaults,{dom:"<'row'<'col-sm-6'l><'col-sm-6'f>><'row'<'col-sm-12'tr>><'row'<'col-sm-6'i><'col-sm-6'p>>",renderer:"bootstrap"});c.extend(b.ext.classes,{sWrapper:"dataTables_wrapper form-inline dt-bootstrap",sFilterInput:"form-control input-sm",sLengthSelect:"form-control input-sm"});b.ext.renderer.pageButton.bootstrap=function(g,f,p,k,h,l){var q=new b.Api(g),r=g.oClasses,i=g.oLanguage.oPaginate,d,e,o=function(b,f){var j,m,n,a,k=function(a){a.preventDefault();
+c(a.currentTarget).hasClass("disabled")||q.page(a.data.action).draw(!1)};j=0;for(m=f.length;j<m;j++)if(a=f[j],c.isArray(a))o(b,a);else{e=d="";switch(a){case "ellipsis":d="&hellip;";e="disabled";break;case "first":d=i.sFirst;e=a+(0<h?"":" disabled");break;case "previous":d=i.sPrevious;e=a+(0<h?"":" disabled");break;case "next":d=i.sNext;e=a+(h<l-1?"":" disabled");break;case "last":d=i.sLast;e=a+(h<l-1?"":" disabled");break;default:d=a+1,e=h===a?"active":""}d&&(n=c("<li>",{"class":r.sPageButton+" "+
+e,"aria-controls":g.sTableId,tabindex:g.iTabIndex,id:0===p&&"string"===typeof a?g.sTableId+"_"+a:null}).append(c("<a>",{href:"#"}).html(d)).appendTo(b),g.oApi._fnBindAction(n,{action:a},k))}};o(c(f).empty().html('<ul class="pagination"/>').children("ul"),k)};b.TableTools&&(c.extend(!0,b.TableTools.classes,{container:"DTTT btn-group",buttons:{normal:"btn btn-default",disabled:"disabled"},collection:{container:"DTTT_dropdown dropdown-menu",buttons:{normal:"",disabled:"disabled"}},print:{info:"DTTT_print_info"},
+select:{row:"active"}}),c.extend(!0,b.TableTools.DEFAULTS.oTags,{collection:{container:"ul",button:"li",liner:"a"}}))};"function"===typeof define&&define.amd?define(["jquery","datatables"],f):"object"===typeof exports?f(require("jquery"),require("datatables")):jQuery&&f(jQuery,jQuery.fn.dataTable)})(window,document);

static/dataTables.responsive.css

@@ -0,0 +1,106 @@
+table.dataTable.dtr-inline.collapsed > tbody > tr > td:first-child,
+table.dataTable.dtr-inline.collapsed > tbody > tr > th:first-child {
+  position: relative;
+  padding-left: 30px;
+  cursor: pointer;
+}
+table.dataTable.dtr-inline.collapsed > tbody > tr > td:first-child:before,
+table.dataTable.dtr-inline.collapsed > tbody > tr > th:first-child:before {
+  top: 8px;
+  left: 4px;
+  height: 16px;
+  width: 16px;
+  display: block;
+  position: absolute;
+  color: white;
+  border: 2px solid white;
+  border-radius: 16px;
+  text-align: center;
+  line-height: 14px;
+  box-shadow: 0 0 3px #444;
+  box-sizing: content-box;
+  content: '+';
+  background-color: #31b131;
+}
+table.dataTable.dtr-inline.collapsed > tbody > tr > td:first-child.dataTables_empty:before,
+table.dataTable.dtr-inline.collapsed > tbody > tr > th:first-child.dataTables_empty:before {
+  display: none;
+}
+table.dataTable.dtr-inline.collapsed > tbody > tr.parent > td:first-child:before,
+table.dataTable.dtr-inline.collapsed > tbody > tr.parent > th:first-child:before {
+  content: '-';
+  background-color: #d33333;
+}
+table.dataTable.dtr-inline.collapsed > tbody > tr.child td:before {
+  display: none;
+}
+table.dataTable.dtr-inline.collapsed.compact > tbody > tr > td:first-child,
+table.dataTable.dtr-inline.collapsed.compact > tbody > tr > th:first-child {
+  padding-left: 27px;
+}
+table.dataTable.dtr-inline.collapsed.compact > tbody > tr > td:first-child:before,
+table.dataTable.dtr-inline.collapsed.compact > tbody > tr > th:first-child:before {
+  top: 5px;
+  left: 4px;
+  height: 14px;
+  width: 14px;
+  border-radius: 14px;
+  line-height: 12px;
+}
+table.dataTable.dtr-column > tbody > tr > td.control,
+table.dataTable.dtr-column > tbody > tr > th.control {
+  position: relative;
+  cursor: pointer;
+}
+table.dataTable.dtr-column > tbody > tr > td.control:before,
+table.dataTable.dtr-column > tbody > tr > th.control:before {
+  top: 50%;
+  left: 50%;
+  height: 16px;
+  width: 16px;
+  margin-top: -10px;
+  margin-left: -10px;
+  display: block;
+  position: absolute;
+  color: white;
+  border: 2px solid white;
+  border-radius: 16px;
+  text-align: center;
+  line-height: 14px;
+  box-shadow: 0 0 3px #444;
+  box-sizing: content-box;
+  content: '+';
+  background-color: #31b131;
+}
+table.dataTable.dtr-column > tbody > tr.parent td.control:before,
+table.dataTable.dtr-column > tbody > tr.parent th.control:before {
+  content: '-';
+  background-color: #d33333;
+}
+table.dataTable > tbody > tr.child {
+  padding: 0.5em 1em;
+}
+table.dataTable > tbody > tr.child:hover {
+  background: transparent !important;
+}
+table.dataTable > tbody > tr.child ul {
+  display: inline-block;
+  list-style-type: none;
+  margin: 0;
+  padding: 0;
+}
+table.dataTable > tbody > tr.child ul li {
+  border-bottom: 1px solid #efefef;
+  padding: 0.5em 0;
+}
+table.dataTable > tbody > tr.child ul li:first-child {
+  padding-top: 0;
+}
+table.dataTable > tbody > tr.child ul li:last-child {
+  border-bottom: none;
+}
+table.dataTable > tbody > tr.child span.dtr-title {
+  display: inline-block;
+  min-width: 75px;
+  font-weight: bold;
+}

static/dataTables.responsive.js

@@ -0,0 +1,873 @@
+/*! Responsive 1.0.6
+ * 2014-2015 SpryMedia Ltd - datatables.net/license
+ */
+
+/**
+ * @summary     Responsive
+ * @description Responsive tables plug-in for DataTables
+ * @version     1.0.6
+ * @file        dataTables.responsive.js
+ * @author      SpryMedia Ltd (www.sprymedia.co.uk)
+ * @contact     www.sprymedia.co.uk/contact
+ * @copyright   Copyright 2014-2015 SpryMedia Ltd.
+ *
+ * This source file is free software, available under the following license:
+ *   MIT license - http://datatables.net/license/mit
+ *
+ * This source file is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+ * or FITNESS FOR A PARTICULAR PURPOSE. See the license files for details.
+ *
+ * For details please refer to: http://www.datatables.net
+ */
+
+(function(window, document, undefined) {
+
+
+var factory = function( $, DataTable ) {
+"use strict";
+
+/**
+ * Responsive is a plug-in for the DataTables library that makes use of
+ * DataTables' ability to change the visibility of columns, changing the
+ * visibility of columns so the displayed columns fit into the table container.
+ * The end result is that complex tables will be dynamically adjusted to fit
+ * into the viewport, be it on a desktop, tablet or mobile browser.
+ *
+ * Responsive for DataTables has two modes of operation, which can used
+ * individually or combined:
+ *
+ * * Class name based control - columns assigned class names that match the
+ *   breakpoint logic can be shown / hidden as required for each breakpoint.
+ * * Automatic control - columns are automatically hidden when there is no
+ *   room left to display them. Columns removed from the right.
+ *
+ * In additional to column visibility control, Responsive also has built into
+ * options to use DataTables' child row display to show / hide the information
+ * from the table that has been hidden. There are also two modes of operation
+ * for this child row display:
+ *
+ * * Inline - when the control element that the user can use to show / hide
+ *   child rows is displayed inside the first column of the table.
+ * * Column - where a whole column is dedicated to be the show / hide control.
+ *
+ * Initialisation of Responsive is performed by:
+ *
+ * * Adding the class `responsive` or `dt-responsive` to the table. In this case
+ *   Responsive will automatically be initialised with the default configuration
+ *   options when the DataTable is created.
+ * * Using the `responsive` option in the DataTables configuration options. This
+ *   can also be used to specify the configuration options, or simply set to
+ *   `true` to use the defaults.
+ *
+ *  @class
+ *  @param {object} settings DataTables settings object for the host table
+ *  @param {object} [opts] Configuration options
+ *  @requires jQuery 1.7+
+ *  @requires DataTables 1.10.1+
+ *
+ *  @example
+ *      $('#example').DataTable( {
+ *        responsive: true
+ *      } );
+ *    } );
+ */
+var Responsive = function ( settings, opts ) {
+	// Sanity check that we are using DataTables 1.10 or newer
+	if ( ! DataTable.versionCheck || ! DataTable.versionCheck( '1.10.1' ) ) {
+		throw 'DataTables Responsive requires DataTables 1.10.1 or newer';
+	}
+
+	this.s = {
+		dt: new DataTable.Api( settings ),
+		columns: []
+	};
+
+	// Check if responsive has already been initialised on this table
+	if ( this.s.dt.settings()[0].responsive ) {
+		return;
+	}
+
+	// details is an object, but for simplicity the user can give it as a string
+	if ( opts && typeof opts.details === 'string' ) {
+		opts.details = { type: opts.details };
+	}
+
+	this.c = $.extend( true, {}, Responsive.defaults, DataTable.defaults.responsive, opts );
+	settings.responsive = this;
+	this._constructor();
+};
+
+Responsive.prototype = {
+	/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
+	 * Constructor
+	 */
+
+	/**
+	 * Initialise the Responsive instance
+	 *
+	 * @private
+	 */
+	_constructor: function ()
+	{
+		var that = this;
+		var dt = this.s.dt;
+
+		dt.settings()[0]._responsive = this;
+
+		// Use DataTables' private throttle function to avoid processor thrashing
+		$(window).on( 'resize.dtr orientationchange.dtr', dt.settings()[0].oApi._fnThrottle( function () {
+			that._resize();
+		} ) );
+
+		// Destroy event handler
+		dt.on( 'destroy.dtr', function () {
+			$(window).off( 'resize.dtr orientationchange.dtr draw.dtr' );
+		} );
+
+		// Reorder the breakpoints array here in case they have been added out
+		// of order
+		this.c.breakpoints.sort( function (a, b) {
+			return a.width < b.width ? 1 :
+				a.width > b.width ? -1 : 0;
+		} );
+
+		// Determine which columns are already hidden, and should therefore
+		// remain hidden. todo - should this be done? See thread 22677
+		//
+		// this.s.alwaysHidden = dt.columns(':hidden').indexes();
+
+		this._classLogic();
+		this._resizeAuto();
+
+		// Details handler
+		var details = this.c.details;
+		if ( details.type ) {
+			that._detailsInit();
+			this._detailsVis();
+
+			dt.on( 'column-visibility.dtr', function () {
+				that._detailsVis();
+			} );
+
+			// Redraw the details box on each draw. This is used until
+			// DataTables implements a native `updated` event for rows
+			dt.on( 'draw.dtr', function () {
+				dt.rows( {page: 'current'} ).iterator( 'row', function ( settings, idx ) {
+					var row = dt.row( idx );
+
+					if ( row.child.isShown() ) {
+						var info = that.c.details.renderer( dt, idx );
+						row.child( info, 'child' ).show();
+					}
+				} );
+			} );
+
+			$(dt.table().node()).addClass( 'dtr-'+details.type );
+		}
+
+		// First pass - draw the table for the current viewport size
+		this._resize();
+	},
+
+
+	/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
+	 * Private methods
+	 */
+
+	/**
+	 * Calculate the visibility for the columns in a table for a given
+	 * breakpoint. The result is pre-determined based on the class logic if
+	 * class names are used to control all columns, but the width of the table
+	 * is also used if there are columns which are to be automatically shown
+	 * and hidden.
+	 *
+	 * @param  {string} breakpoint Breakpoint name to use for the calculation
+	 * @return {array} Array of boolean values initiating the visibility of each
+	 *   column.
+	 *  @private
+	 */
+	_columnsVisiblity: function ( breakpoint )
+	{
+		var dt = this.s.dt;
+		var columns = this.s.columns;
+		var i, ien;
+
+		// Class logic - determine which columns are in this breakpoint based
+		// on the classes. If no class control (i.e. `auto`) then `-` is used
+		// to indicate this to the rest of the function
+		var display = $.map( columns, function ( col ) {
+			return col.auto && col.minWidth === null ?
+				false :
+				col.auto === true ?
+					'-' :
+					$.inArray( breakpoint, col.includeIn ) !== -1;
+		} );
+
+		// Auto column control - first pass: how much width is taken by the
+		// ones that must be included from the non-auto columns
+		var requiredWidth = 0;
+		for ( i=0, ien=display.length ; i<ien ; i++ ) {
+			if ( display[i] === true ) {
+				requiredWidth += columns[i].minWidth;
+			}
+		}
+
+		// Second pass, use up any remaining width for other columns. For
+		// scrolling tables we need to subtract the width of the scrollbar. It
+		// may not be requires which makes this sub-optimal, but it would
+		// require another full redraw to make complete use of those extra few
+		// pixels
+		var scrolling = dt.settings()[0].oScroll;
+		var bar = scrolling.sY || scrolling.sX ? scrolling.iBarWidth : 0;
+		var widthAvailable = dt.table().container().offsetWidth - bar;
+		var usedWidth = widthAvailable - requiredWidth;
+
+		// Control column needs to always be included. This makes it sub-
+		// optimal in terms of using the available with, but to stop layout
+		// thrashing or overflow. Also we need to account for the control column
+		// width first so we know how much width is available for the other
+		// columns, since the control column might not be the first one shown
+		for ( i=0, ien=display.length ; i<ien ; i++ ) {
+			if ( columns[i].control ) {
+				usedWidth -= columns[i].minWidth;
+			}
+		}
+
+		// Allow columns to be shown (counting from the left) until we run out
+		// of room
+		var empty = false;
+		for ( i=0, ien=display.length ; i<ien ; i++ ) {
+			if ( display[i] === '-' && ! columns[i].control ) {
+				// Once we've found a column that won't fit we don't let any
+				// others display either, or columns might disappear in the
+				// middle of the table
+				if ( empty || usedWidth - columns[i].minWidth < 0 ) {
+					empty = true;
+					display[i] = false;
+				}
+				else {
+					display[i] = true;
+				}
+
+				usedWidth -= columns[i].minWidth;
+			}
+		}
+
+		// Determine if the 'control' column should be shown (if there is one).
+		// This is the case when there is a hidden column (that is not the
+		// control column). The two loops look inefficient here, but they are
+		// trivial and will fly through. We need to know the outcome from the
+		// first , before the action in the second can be taken
+		var showControl = false;
+
+		for ( i=0, ien=columns.length ; i<ien ; i++ ) {
+			if ( ! columns[i].control && ! columns[i].never && ! display[i] ) {
+				showControl = true;
+				break;
+			}
+		}
+
+		for ( i=0, ien=columns.length ; i<ien ; i++ ) {
+			if ( columns[i].control ) {
+				display[i] = showControl;
+			}
+		}
+
+		// Finally we need to make sure that there is at least one column that
+		// is visible
+		if ( $.inArray( true, display ) === -1 ) {
+			display[0] = true;
+		}
+
+		return display;
+	},
+
+
+	/**
+	 * Create the internal `columns` array with information about the columns
+	 * for the table. This includes determining which breakpoints the column
+	 * will appear in, based upon class names in the column, which makes up the
+	 * vast majority of this method.
+	 *
+	 * @private
+	 */
+	_classLogic: function ()
+	{
+		var that = this;
+		var calc = {};
+		var breakpoints = this.c.breakpoints;
+		var columns = this.s.dt.columns().eq(0).map( function (i) {
+			var className = this.column(i).header().className;
+
+			return {
+				className: className,
+				includeIn: [],
+				auto:      false,
+				control:   false,
+				never:     className.match(/\bnever\b/) ? true : false
+			};
+		} );
+
+		// Simply add a breakpoint to `includeIn` array, ensuring that there are
+		// no duplicates
+		var add = function ( colIdx, name ) {
+			var includeIn = columns[ colIdx ].includeIn;
+
+			if ( $.inArray( name, includeIn ) === -1 ) {
+				includeIn.push( name );
+			}
+		};
+
+		var column = function ( colIdx, name, operator, matched ) {
+			var size, i, ien;
+
+			if ( ! operator ) {
+				columns[ colIdx ].includeIn.push( name );
+			}
+			else if ( operator === 'max-' ) {
+				// Add this breakpoint and all smaller
+				size = that._find( name ).width;
+
+				for ( i=0, ien=breakpoints.length ; i<ien ; i++ ) {
+					if ( breakpoints[i].width <= size ) {
+						add( colIdx, breakpoints[i].name );
+					}
+				}
+			}
+			else if ( operator === 'min-' ) {
+				// Add this breakpoint and all larger
+				size = that._find( name ).width;
+
+				for ( i=0, ien=breakpoints.length ; i<ien ; i++ ) {
+					if ( breakpoints[i].width >= size ) {
+						add( colIdx, breakpoints[i].name );
+					}
+				}
+			}
+			else if ( operator === 'not-' ) {
+				// Add all but this breakpoint (xxx need extra information)
+
+				for ( i=0, ien=breakpoints.length ; i<ien ; i++ ) {
+					if ( breakpoints[i].name.indexOf( matched ) === -1 ) {
+						add( colIdx, breakpoints[i].name );
+					}
+				}
+			}
+		};
+
+		// Loop over each column and determine if it has a responsive control
+		// class
+		columns.each( function ( col, i ) {
+			var classNames = col.className.split(' ');
+			var hasClass = false;
+
+			// Split the class name up so multiple rules can be applied if needed
+			for ( var k=0, ken=classNames.length ; k<ken ; k++ ) {
+				var className = $.trim( classNames[k] );
+
+				if ( className === 'all' ) {
+					// Include in all
+					hasClass = true;
+					col.includeIn = $.map( breakpoints, function (a) {
+						return a.name;
+					} );
+					return;
+				}
+				else if ( className === 'none' || className === 'never' ) {
+					// Include in none (default) and no auto
+					hasClass = true;
+					return;
+				}
+				else if ( className === 'control' ) {
+					// Special column that is only visible, when one of the other
+					// columns is hidden. This is used for the details control
+					hasClass = true;
+					col.control = true;
+					return;
+				}
+
+				$.each( breakpoints, function ( j, breakpoint ) {
+					// Does this column have a class that matches this breakpoint?
+					var brokenPoint = breakpoint.name.split('-');
+					var re = new RegExp( '(min\\-|max\\-|not\\-)?('+brokenPoint[0]+')(\\-[_a-zA-Z0-9])?' );
+					var match = className.match( re );
+
+					if ( match ) {
+						hasClass = true;
+
+						if ( match[2] === brokenPoint[0] && match[3] === '-'+brokenPoint[1] ) {
+							// Class name matches breakpoint name fully
+							column( i, breakpoint.name, match[1], match[2]+match[3] );
+						}
+						else if ( match[2] === brokenPoint[0] && ! match[3] ) {
+							// Class name matched primary breakpoint name with no qualifier
+							column( i, breakpoint.name, match[1], match[2] );
+						}
+					}
+				} );
+			}
+
+			// If there was no control class, then automatic sizing is used
+			if ( ! hasClass ) {
+				col.auto = true;
+			}
+		} );
+
+		this.s.columns = columns;
+	},
+
+
+	/**
+	 * Initialisation for the details handler
+	 *
+	 * @private
+	 */
+	_detailsInit: function ()
+	{
+		var that    = this;
+		var dt      = this.s.dt;
+		var details = this.c.details;
+
+		// The inline type always uses the first child as the target
+		if ( details.type === 'inline' ) {
+			details.target = 'td:first-child';
+		}
+
+		// type.target can be a string jQuery selector or a column index
+		var target   = details.target;
+		var selector = typeof target === 'string' ? target : 'td';
+
+		// Click handler to show / hide the details rows when they are available
+		$( dt.table().body() ).on( 'click', selector, function (e) {
+			// If the table is not collapsed (i.e. there is no hidden columns)
+			// then take no action
+			if ( ! $(dt.table().node()).hasClass('collapsed' ) ) {
+				return;
+			}
+
+			// Check that the row is actually a DataTable's controlled node
+			if ( ! dt.row( $(this).closest('tr') ).length ) {
+				return;
+			}
+
+			// For column index, we determine if we should act or not in the
+			// handler - otherwise it is already okay
+			if ( typeof target === 'number' ) {
+				var targetIdx = target < 0 ?
+					dt.columns().eq(0).length + target :
+					target;
+
+				if ( dt.cell( this ).index().column !== targetIdx ) {
+					return;
+				}
+			}
+
+			// $().closest() includes itself in its check
+			var row = dt.row( $(this).closest('tr') );
+
+			if ( row.child.isShown() ) {
+				row.child( false );
+				$( row.node() ).removeClass( 'parent' );
+			}
+			else {
+				var info = that.c.details.renderer( dt, row[0] );
+				row.child( info, 'child' ).show();
+				$( row.node() ).addClass( 'parent' );
+			}
+		} );
+	},
+
+
+	/**
+	 * Update the child rows in the table whenever the column visibility changes
+	 *
+	 * @private
+	 */
+	_detailsVis: function ()
+	{
+		var that = this;
+		var dt = this.s.dt;
+
+		// Find how many columns are hidden
+		var hiddenColumns = dt.columns().indexes().filter( function ( idx ) {
+			var col = dt.column( idx );
+
+			if ( col.visible() ) {
+				return null;
+			}
+
+			// Only counts as hidden if it doesn't have the `never` class
+			return $( col.header() ).hasClass( 'never' ) ? null : idx;
+		} );
+		var haveHidden = true;
+
+		if ( hiddenColumns.length === 0 || ( hiddenColumns.length === 1 && this.s.columns[ hiddenColumns[0] ].control ) ) {
+			haveHidden = false;
+		}
+
+		if ( haveHidden ) {
+			// Show all existing child rows
+			dt.rows( { page: 'current' } ).eq(0).each( function (idx) {
+				var row = dt.row( idx );
+
+				if ( row.child() ) {
+					var info = that.c.details.renderer( dt, row[0] );
+
+					// The renderer can return false to have no child row
+					if ( info === false ) {
+						row.child.hide();
+					}
+					else {
+						row.child( info, 'child' ).show();
+					}
+				}
+			} );
+		}
+		else {
+			// Hide all existing child rows
+			dt.rows( { page: 'current' } ).eq(0).each( function (idx) {
+				dt.row( idx ).child.hide();
+			} );
+		}
+	},
+
+
+	/**
+	 * Find a breakpoint object from a name
+	 * @param  {string} name Breakpoint name to find
+	 * @return {object}      Breakpoint description object
+	 */
+	_find: function ( name )
+	{
+		var breakpoints = this.c.breakpoints;
+
+		for ( var i=0, ien=breakpoints.length ; i<ien ; i++ ) {
+			if ( breakpoints[i].name === name ) {
+				return breakpoints[i];
+			}
+		}
+	},
+
+
+	/**
+	 * Alter the table display for a resized viewport. This involves first
+	 * determining what breakpoint the window currently is in, getting the
+	 * column visibilities to apply and then setting them.
+	 *
+	 * @private
+	 */
+	_resize: function ()
+	{
+		var dt = this.s.dt;
+		var width = $(window).width();
+		var breakpoints = this.c.breakpoints;
+		var breakpoint = breakpoints[0].name;
+		var columns = this.s.columns;
+		var i, ien;
+
+		// Determine what breakpoint we are currently at
+		for ( i=breakpoints.length-1 ; i>=0 ; i-- ) {
+			if ( width <= breakpoints[i].width ) {
+				breakpoint = breakpoints[i].name;
+				break;
+			}
+		}
+		
+		// Show the columns for that break point
+		var columnsVis = this._columnsVisiblity( breakpoint );
+
+		// Set the class before the column visibility is changed so event
+		// listeners know what the state is. Need to determine if there are
+		// any columns that are not visible but can be shown
+		var collapsedClass = false;
+		for ( i=0, ien=columns.length ; i<ien ; i++ ) {
+			if ( columnsVis[i] === false && ! columns[i].never ) {
+				collapsedClass = true;
+				break;
+			}
+		}
+
+		$( dt.table().node() ).toggleClass('collapsed', collapsedClass );
+
+		dt.columns().eq(0).each( function ( colIdx, i ) {
+			dt.column( colIdx ).visible( columnsVis[i] );
+		} );
+	},
+
+
+	/**
+	 * Determine the width of each column in the table so the auto column hiding
+	 * has that information to work with. This method is never going to be 100%
+	 * perfect since column widths can change slightly per page, but without
+	 * seriously compromising performance this is quite effective.
+	 *
+	 * @private
+	 */
+	_resizeAuto: function ()
+	{
+		var dt = this.s.dt;
+		var columns = this.s.columns;
+
+		// Are we allowed to do auto sizing?
+		if ( ! this.c.auto ) {
+			return;
+		}
+
+		// Are there any columns that actually need auto-sizing, or do they all
+		// have classes defined
+		if ( $.inArray( true, $.map( columns, function (c) { return c.auto; } ) ) === -1 ) {
+			return;
+		}
+
+		// Clone the table with the current data in it
+		var tableWidth   = dt.table().node().offsetWidth;
+		var columnWidths = dt.columns;
+		var clonedTable  = dt.table().node().cloneNode( false );
+		var clonedHeader = $( dt.table().header().cloneNode( false ) ).appendTo( clonedTable );
+		var clonedBody   = $( dt.table().body().cloneNode( false ) ).appendTo( clonedTable );
+
+		$( dt.table().footer() ).clone( false ).appendTo( clonedTable );
+
+		// This is a bit slow, but we need to get a clone of each row that
+		// includes all columns. As such, try to do this as little as possible.
+		dt.rows( { page: 'current' } ).indexes().flatten().each( function ( idx ) {
+			var clone = dt.row( idx ).node().cloneNode( true );
+			
+			if ( dt.columns( ':hidden' ).flatten().length ) {
+				$(clone).append( dt.cells( idx, ':hidden' ).nodes().to$().clone() );
+			}
+
+			$(clone).appendTo( clonedBody );
+		} );
+
+		var cells = dt.columns().header().to$().clone( false );
+		$('<tr/>')
+			.append( cells )
+			.appendTo( clonedHeader );
+
+		// In the inline case extra padding is applied to the first column to
+		// give space for the show / hide icon. We need to use this in the
+		// calculation
+		if ( this.c.details.type === 'inline' ) {
+			$(clonedTable).addClass( 'dtr-inline collapsed' );
+		}
+
+		var inserted = $('<div/>')
+			.css( {
+				width: 1,
+				height: 1,
+				overflow: 'hidden'
+			} )
+			.append( clonedTable );
+
+		// Remove columns which are not to be included
+		inserted.find('th.never, td.never').remove();
+
+		inserted.insertBefore( dt.table().node() );
+
+		// The cloned header now contains the smallest that each column can be
+		dt.columns().eq(0).each( function ( idx ) {
+			columns[idx].minWidth = cells[ idx ].offsetWidth || 0;
+		} );
+
+		inserted.remove();
+	}
+};
+
+
+/**
+ * List of default breakpoints. Each item in the array is an object with two
+ * properties:
+ *
+ * * `name` - the breakpoint name.
+ * * `width` - the breakpoint width
+ *
+ * @name Responsive.breakpoints
+ * @static
+ */
+Responsive.breakpoints = [
+	{ name: 'desktop',  width: Infinity },
+	{ name: 'tablet-l', width: 1024 },
+	{ name: 'tablet-p', width: 768 },
+	{ name: 'mobile-l', width: 480 },
+	{ name: 'mobile-p', width: 320 }
+];
+
+
+/**
+ * Responsive default settings for initialisation
+ *
+ * @namespace
+ * @name Responsive.defaults
+ * @static
+ */
+Responsive.defaults = {
+	/**
+	 * List of breakpoints for the instance. Note that this means that each
+	 * instance can have its own breakpoints. Additionally, the breakpoints
+	 * cannot be changed once an instance has been creased.
+	 *
+	 * @type {Array}
+	 * @default Takes the value of `Responsive.breakpoints`
+	 */
+	breakpoints: Responsive.breakpoints,
+
+	/**
+	 * Enable / disable auto hiding calculations. It can help to increase
+	 * performance slightly if you disable this option, but all columns would
+	 * need to have breakpoint classes assigned to them
+	 *
+	 * @type {Boolean}
+	 * @default  `true`
+	 */
+	auto: true,
+
+	/**
+	 * Details control. If given as a string value, the `type` property of the
+	 * default object is set to that value, and the defaults used for the rest
+	 * of the object - this is for ease of implementation.
+	 *
+	 * The object consists of the following properties:
+	 *
+	 * * `renderer` - function that is called for display of the child row data.
+	 *   The default function will show the data from the hidden columns
+	 * * `target` - Used as the selector for what objects to attach the child
+	 *   open / close to
+	 * * `type` - `false` to disable the details display, `inline` or `column`
+	 *   for the two control types
+	 *
+	 * @type {Object|string}
+	 */
+	details: {
+		renderer: function ( api, rowIdx ) {
+			var data = api.cells( rowIdx, ':hidden' ).eq(0).map( function ( cell ) {
+				var header = $( api.column( cell.column ).header() );
+				var idx = api.cell( cell ).index();
+
+				if ( header.hasClass( 'control' ) || header.hasClass( 'never' ) ) {
+					return '';
+				}
+
+				// Use a non-public DT API method to render the data for display
+				// This needs to be updated when DT adds a suitable method for
+				// this type of data retrieval
+				var dtPrivate = api.settings()[0];
+				var cellData = dtPrivate.oApi._fnGetCellData(
+					dtPrivate, idx.row, idx.column, 'display'
+				);
+				var title = header.text();
+				if ( title ) {
+					title = title + ':';
+				}
+
+				return '<li data-dtr-index="'+idx.column+'">'+
+						'<span class="dtr-title">'+
+							title+
+						'</span> '+
+						'<span class="dtr-data">'+
+							cellData+
+						'</span>'+
+					'</li>';
+			} ).toArray().join('');
+
+			return data ?
+				$('<ul data-dtr-index="'+rowIdx+'"/>').append( data ) :
+				false;
+		},
+
+		target: 0,
+
+		type: 'inline'
+	}
+};
+
+
+/*
+ * API
+ */
+var Api = $.fn.dataTable.Api;
+
+// Doesn't do anything - work around for a bug in DT... Not documented
+Api.register( 'responsive()', function () {
+	return this;
+} );
+
+Api.register( 'responsive.index()', function ( li ) {
+	li = $(li);
+
+	return {
+		column: li.data('dtr-index'),
+		row:    li.parent().data('dtr-index')
+	};
+} );
+
+Api.register( 'responsive.rebuild()', function () {
+	return this.iterator( 'table', function ( ctx ) {
+		if ( ctx._responsive ) {
+			ctx._responsive._classLogic();
+		}
+	} );
+} );
+
+Api.register( 'responsive.recalc()', function () {
+	return this.iterator( 'table', function ( ctx ) {
+		if ( ctx._responsive ) {
+			ctx._responsive._resizeAuto();
+			ctx._responsive._resize();
+		}
+	} );
+} );
+
+
+/**
+ * Version information
+ *
+ * @name Responsive.version
+ * @static
+ */
+Responsive.version = '1.0.6';
+
+
+$.fn.dataTable.Responsive = Responsive;
+$.fn.DataTable.Responsive = Responsive;
+
+// Attach a listener to the document which listens for DataTables initialisation
+// events so we can automatically initialise
+$(document).on( 'init.dt.dtr', function (e, settings, json) {
+	if ( e.namespace !== 'dt' ) {
+		return;
+	}
+
+	if ( $(settings.nTable).hasClass( 'responsive' ) ||
+		 $(settings.nTable).hasClass( 'dt-responsive' ) ||
+		 settings.oInit.responsive ||
+		 DataTable.defaults.responsive
+	) {
+		var init = settings.oInit.responsive;
+
+		if ( init !== false ) {
+			new Responsive( settings, $.isPlainObject( init ) ? init : {}  );
+		}
+	}
+} );
+
+return Responsive;
+}; // /factory
+
+
+// Define as an AMD module if possible
+if ( typeof define === 'function' && define.amd ) {
+	define( ['jquery', 'datatables'], factory );
+}
+else if ( typeof exports === 'object' ) {
+    // Node/CommonJS
+    factory( require('jquery'), require('datatables') );
+}
+else if ( jQuery && !jQuery.fn.dataTable.Responsive ) {
+	// Otherwise simply initialise as normal, stopping multiple evaluation
+	factory( jQuery, jQuery.fn.dataTable );
+}
+
+
+})(window, document);

static/feedback.js

@@ -0,0 +1,133 @@
+// submitting form with modal:
+// https://stackoverflow.com/a/29068742
+//
+// closing a bootstrap modal with submit button:
+// https://stackoverflow.com/a/33478107
+//
+// flask post data as json:
+// https://stackoverflow.com/a/16664376
+
+/* this function is called when the user submits
+ * the feedback form. it submits a post request
+ * to the flask server, which squirrels away the
+ * feedback in a file.
+ */
+function submit_feedback() {
+    // this function is called when submit button clicked
+    // algorithm:
+    // - check if text box has content
+    // - check if happy/sad filled out
+
+    var smile_active = $('#modal-feedback-smile-div').hasClass('smile-active');
+    var frown_active = $('#modal-feedback-frown-div').hasClass('frown-active');
+    if( !( smile_active || frown_active ) ) {
+        alert('Please pick the smile or the frown.')
+    } else if( $('#modal-feedback-textarea').val()=='' ) {
+        alert('Please provide us with some feedback.')
+    } else {
+        var user_sentiment = '';
+        if(smile_active) {
+            user_sentiment = 'smile';
+        } else {
+            user_sentiment = 'frown';
+        }
+        var escaped_text = $('#modal-feedback-textarea').val();
+
+        // prepare form data 
+        var data = {
+            sentiment : user_sentiment,
+            content : escaped_text
+        };
+        // post the form. the callback function resets the form
+        $.post("/feedback", 
+            data, 
+            function(response) {
+                $('#myModal').modal('hide');
+                $('#myModalForm')[0].reset();
+                add_alert(response);
+                frown_unclick();
+                smile_unclick();
+        });
+    }
+}
+
+
+function add_alert(response) {
+    str = ""
+    str += '<div id="feedback-messages-container" class="container">';
+
+    if (response['status']=='ok') {
+        // if status is ok, use alert-success
+        str += '    <div id="feedback-messages-alert" class="alert alert-success alert-dismissible fade in">';
+    } else {
+        // otherwise use alert-danger
+        str += '    <div id="feedback-messages-alert" class="alert alert-danger alert-dismissible fade in">';
+    }
+
+    str += '        <a href="#" class="close" data-dismiss="alert" aria-label="close">&times;</a>';
+    str += '        <div id="feedback-messages-contianer" class="container-fluid">';
+    str += '            <div id="feedback-messages-div" class="co-xs-12">';
+    str += '                <p>'
+    str += response['message'];
+    str += '                </p>';
+    str += '            </div>';
+    str += '    </div>';
+    str += '</div>';
+    $('div#messages').append(str);
+}
+
+
+/* for those particularly wordy users... limit feedback to 1000 chars */
+function cool_it() { 
+    if($('#modal-feedback-textarea').val().length > 1100 ){
+        $('#modal-too-long').show();
+    } else {
+        $('#modal-too-long').hide();
+    }
+}
+
+/* smiley functions */
+function smile_click() {
+    $('#modal-feedback-smile-div').addClass('smile-active');
+    $('#modal-feedback-smile-icon').addClass('smile-active');
+}
+function frown_click() {
+    $('#modal-feedback-frown-div').addClass('frown-active');
+    $('#modal-feedback-frown-icon').addClass('frown-active');
+}
+function smile_unclick() {
+    $('#modal-feedback-smile-div').removeClass('smile-active');
+    $('#modal-feedback-smile-icon').removeClass('smile-active');
+}
+function frown_unclick() {
+    $('#modal-feedback-frown-div').removeClass('frown-active');
+    $('#modal-feedback-frown-icon').removeClass('frown-active');
+}
+
+function smile() {
+    frown_unclick();
+    smile_click();
+}
+function frown() { 
+    smile_unclick();
+    frown_click();
+}
+
+
+/* for those particularly wordy users... limit feedback to 1100 chars */
+// how to check n characters in a textarea
+// https://stackoverflow.com/a/19934613
+/*
+$(document).ready(function() {
+
+    $('#modal-feedback-textarea').on('change',function(event) {
+        if($('#modal-feedback-textarea').val().length > 1100 ){
+            $('#modal-too-long').show();
+        } else {
+            $('#modal-too-long').hide();
+        }
+    });
+
+}
+*/
+

static/jquery.dataTables.min.js

@@ -0,0 +1,166 @@
+/*!
+ DataTables 1.10.12
+ ©2008-2015 SpryMedia Ltd - datatables.net/license
+*/
+(function(h){"function"===typeof define&&define.amd?define(["jquery"],function(D){return h(D,window,document)}):"object"===typeof exports?module.exports=function(D,I){D||(D=window);I||(I="undefined"!==typeof window?require("jquery"):require("jquery")(D));return h(I,D,D.document)}:h(jQuery,window,document)})(function(h,D,I,k){function X(a){var b,c,d={};h.each(a,function(e){if((b=e.match(/^([^A-Z]+?)([A-Z])/))&&-1!=="a aa ai ao as b fn i m o s ".indexOf(b[1]+" "))c=e.replace(b[0],b[2].toLowerCase()),
+d[c]=e,"o"===b[1]&&X(a[e])});a._hungarianMap=d}function K(a,b,c){a._hungarianMap||X(a);var d;h.each(b,function(e){d=a._hungarianMap[e];if(d!==k&&(c||b[d]===k))"o"===d.charAt(0)?(b[d]||(b[d]={}),h.extend(!0,b[d],b[e]),K(a[d],b[d],c)):b[d]=b[e]})}function Da(a){var b=m.defaults.oLanguage,c=a.sZeroRecords;!a.sEmptyTable&&(c&&"No data available in table"===b.sEmptyTable)&&E(a,a,"sZeroRecords","sEmptyTable");!a.sLoadingRecords&&(c&&"Loading..."===b.sLoadingRecords)&&E(a,a,"sZeroRecords","sLoadingRecords");
+a.sInfoThousands&&(a.sThousands=a.sInfoThousands);(a=a.sDecimal)&&db(a)}function eb(a){A(a,"ordering","bSort");A(a,"orderMulti","bSortMulti");A(a,"orderClasses","bSortClasses");A(a,"orderCellsTop","bSortCellsTop");A(a,"order","aaSorting");A(a,"orderFixed","aaSortingFixed");A(a,"paging","bPaginate");A(a,"pagingType","sPaginationType");A(a,"pageLength","iDisplayLength");A(a,"searching","bFilter");"boolean"===typeof a.sScrollX&&(a.sScrollX=a.sScrollX?"100%":"");"boolean"===typeof a.scrollX&&(a.scrollX=
+a.scrollX?"100%":"");if(a=a.aoSearchCols)for(var b=0,c=a.length;b<c;b++)a[b]&&K(m.models.oSearch,a[b])}function fb(a){A(a,"orderable","bSortable");A(a,"orderData","aDataSort");A(a,"orderSequence","asSorting");A(a,"orderDataType","sortDataType");var b=a.aDataSort;b&&!h.isArray(b)&&(a.aDataSort=[b])}function gb(a){if(!m.__browser){var b={};m.__browser=b;var c=h("<div/>").css({position:"fixed",top:0,left:0,height:1,width:1,overflow:"hidden"}).append(h("<div/>").css({position:"absolute",top:1,left:1,
+width:100,overflow:"scroll"}).append(h("<div/>").css({width:"100%",height:10}))).appendTo("body"),d=c.children(),e=d.children();b.barWidth=d[0].offsetWidth-d[0].clientWidth;b.bScrollOversize=100===e[0].offsetWidth&&100!==d[0].clientWidth;b.bScrollbarLeft=1!==Math.round(e.offset().left);b.bBounding=c[0].getBoundingClientRect().width?!0:!1;c.remove()}h.extend(a.oBrowser,m.__browser);a.oScroll.iBarWidth=m.__browser.barWidth}function hb(a,b,c,d,e,f){var g,j=!1;c!==k&&(g=c,j=!0);for(;d!==e;)a.hasOwnProperty(d)&&
+(g=j?b(g,a[d],d,a):a[d],j=!0,d+=f);return g}function Ea(a,b){var c=m.defaults.column,d=a.aoColumns.length,c=h.extend({},m.models.oColumn,c,{nTh:b?b:I.createElement("th"),sTitle:c.sTitle?c.sTitle:b?b.innerHTML:"",aDataSort:c.aDataSort?c.aDataSort:[d],mData:c.mData?c.mData:d,idx:d});a.aoColumns.push(c);c=a.aoPreSearchCols;c[d]=h.extend({},m.models.oSearch,c[d]);ja(a,d,h(b).data())}function ja(a,b,c){var b=a.aoColumns[b],d=a.oClasses,e=h(b.nTh);if(!b.sWidthOrig){b.sWidthOrig=e.attr("width")||null;var f=
+(e.attr("style")||"").match(/width:\s*(\d+[pxem%]+)/);f&&(b.sWidthOrig=f[1])}c!==k&&null!==c&&(fb(c),K(m.defaults.column,c),c.mDataProp!==k&&!c.mData&&(c.mData=c.mDataProp),c.sType&&(b._sManualType=c.sType),c.className&&!c.sClass&&(c.sClass=c.className),h.extend(b,c),E(b,c,"sWidth","sWidthOrig"),c.iDataSort!==k&&(b.aDataSort=[c.iDataSort]),E(b,c,"aDataSort"));var g=b.mData,j=Q(g),i=b.mRender?Q(b.mRender):null,c=function(a){return"string"===typeof a&&-1!==a.indexOf("@")};b._bAttrSrc=h.isPlainObject(g)&&
+(c(g.sort)||c(g.type)||c(g.filter));b._setter=null;b.fnGetData=function(a,b,c){var d=j(a,b,k,c);return i&&b?i(d,b,a,c):d};b.fnSetData=function(a,b,c){return R(g)(a,b,c)};"number"!==typeof g&&(a._rowReadObject=!0);a.oFeatures.bSort||(b.bSortable=!1,e.addClass(d.sSortableNone));a=-1!==h.inArray("asc",b.asSorting);c=-1!==h.inArray("desc",b.asSorting);!b.bSortable||!a&&!c?(b.sSortingClass=d.sSortableNone,b.sSortingClassJUI=""):a&&!c?(b.sSortingClass=d.sSortableAsc,b.sSortingClassJUI=d.sSortJUIAscAllowed):
+!a&&c?(b.sSortingClass=d.sSortableDesc,b.sSortingClassJUI=d.sSortJUIDescAllowed):(b.sSortingClass=d.sSortable,b.sSortingClassJUI=d.sSortJUI)}function Y(a){if(!1!==a.oFeatures.bAutoWidth){var b=a.aoColumns;Fa(a);for(var c=0,d=b.length;c<d;c++)b[c].nTh.style.width=b[c].sWidth}b=a.oScroll;(""!==b.sY||""!==b.sX)&&ka(a);u(a,null,"column-sizing",[a])}function Z(a,b){var c=la(a,"bVisible");return"number"===typeof c[b]?c[b]:null}function $(a,b){var c=la(a,"bVisible"),c=h.inArray(b,c);return-1!==c?c:null}
+function aa(a){var b=0;h.each(a.aoColumns,function(a,d){d.bVisible&&"none"!==h(d.nTh).css("display")&&b++});return b}function la(a,b){var c=[];h.map(a.aoColumns,function(a,e){a[b]&&c.push(e)});return c}function Ga(a){var b=a.aoColumns,c=a.aoData,d=m.ext.type.detect,e,f,g,j,i,h,l,q,t;e=0;for(f=b.length;e<f;e++)if(l=b[e],t=[],!l.sType&&l._sManualType)l.sType=l._sManualType;else if(!l.sType){g=0;for(j=d.length;g<j;g++){i=0;for(h=c.length;i<h;i++){t[i]===k&&(t[i]=B(a,i,e,"type"));q=d[g](t[i],a);if(!q&&
+g!==d.length-1)break;if("html"===q)break}if(q){l.sType=q;break}}l.sType||(l.sType="string")}}function ib(a,b,c,d){var e,f,g,j,i,n,l=a.aoColumns;if(b)for(e=b.length-1;0<=e;e--){n=b[e];var q=n.targets!==k?n.targets:n.aTargets;h.isArray(q)||(q=[q]);f=0;for(g=q.length;f<g;f++)if("number"===typeof q[f]&&0<=q[f]){for(;l.length<=q[f];)Ea(a);d(q[f],n)}else if("number"===typeof q[f]&&0>q[f])d(l.length+q[f],n);else if("string"===typeof q[f]){j=0;for(i=l.length;j<i;j++)("_all"==q[f]||h(l[j].nTh).hasClass(q[f]))&&
+d(j,n)}}if(c){e=0;for(a=c.length;e<a;e++)d(e,c[e])}}function N(a,b,c,d){var e=a.aoData.length,f=h.extend(!0,{},m.models.oRow,{src:c?"dom":"data",idx:e});f._aData=b;a.aoData.push(f);for(var g=a.aoColumns,j=0,i=g.length;j<i;j++)g[j].sType=null;a.aiDisplayMaster.push(e);b=a.rowIdFn(b);b!==k&&(a.aIds[b]=f);(c||!a.oFeatures.bDeferRender)&&Ha(a,e,c,d);return e}function ma(a,b){var c;b instanceof h||(b=h(b));return b.map(function(b,e){c=Ia(a,e);return N(a,c.data,e,c.cells)})}function B(a,b,c,d){var e=a.iDraw,
+f=a.aoColumns[c],g=a.aoData[b]._aData,j=f.sDefaultContent,i=f.fnGetData(g,d,{settings:a,row:b,col:c});if(i===k)return a.iDrawError!=e&&null===j&&(L(a,0,"Requested unknown parameter "+("function"==typeof f.mData?"{function}":"'"+f.mData+"'")+" for row "+b+", column "+c,4),a.iDrawError=e),j;if((i===g||null===i)&&null!==j&&d!==k)i=j;else if("function"===typeof i)return i.call(g);return null===i&&"display"==d?"":i}function jb(a,b,c,d){a.aoColumns[c].fnSetData(a.aoData[b]._aData,d,{settings:a,row:b,col:c})}
+function Ja(a){return h.map(a.match(/(\\.|[^\.])+/g)||[""],function(a){return a.replace(/\\./g,".")})}function Q(a){if(h.isPlainObject(a)){var b={};h.each(a,function(a,c){c&&(b[a]=Q(c))});return function(a,c,f,g){var j=b[c]||b._;return j!==k?j(a,c,f,g):a}}if(null===a)return function(a){return a};if("function"===typeof a)return function(b,c,f,g){return a(b,c,f,g)};if("string"===typeof a&&(-1!==a.indexOf(".")||-1!==a.indexOf("[")||-1!==a.indexOf("("))){var c=function(a,b,f){var g,j;if(""!==f){j=Ja(f);
+for(var i=0,n=j.length;i<n;i++){f=j[i].match(ba);g=j[i].match(U);if(f){j[i]=j[i].replace(ba,"");""!==j[i]&&(a=a[j[i]]);g=[];j.splice(0,i+1);j=j.join(".");if(h.isArray(a)){i=0;for(n=a.length;i<n;i++)g.push(c(a[i],b,j))}a=f[0].substring(1,f[0].length-1);a=""===a?g:g.join(a);break}else if(g){j[i]=j[i].replace(U,"");a=a[j[i]]();continue}if(null===a||a[j[i]]===k)return k;a=a[j[i]]}}return a};return function(b,e){return c(b,e,a)}}return function(b){return b[a]}}function R(a){if(h.isPlainObject(a))return R(a._);
+if(null===a)return function(){};if("function"===typeof a)return function(b,d,e){a(b,"set",d,e)};if("string"===typeof a&&(-1!==a.indexOf(".")||-1!==a.indexOf("[")||-1!==a.indexOf("("))){var b=function(a,d,e){var e=Ja(e),f;f=e[e.length-1];for(var g,j,i=0,n=e.length-1;i<n;i++){g=e[i].match(ba);j=e[i].match(U);if(g){e[i]=e[i].replace(ba,"");a[e[i]]=[];f=e.slice();f.splice(0,i+1);g=f.join(".");if(h.isArray(d)){j=0;for(n=d.length;j<n;j++)f={},b(f,d[j],g),a[e[i]].push(f)}else a[e[i]]=d;return}j&&(e[i]=e[i].replace(U,
+""),a=a[e[i]](d));if(null===a[e[i]]||a[e[i]]===k)a[e[i]]={};a=a[e[i]]}if(f.match(U))a[f.replace(U,"")](d);else a[f.replace(ba,"")]=d};return function(c,d){return b(c,d,a)}}return function(b,d){b[a]=d}}function Ka(a){return G(a.aoData,"_aData")}function na(a){a.aoData.length=0;a.aiDisplayMaster.length=0;a.aiDisplay.length=0;a.aIds={}}function oa(a,b,c){for(var d=-1,e=0,f=a.length;e<f;e++)a[e]==b?d=e:a[e]>b&&a[e]--; -1!=d&&c===k&&a.splice(d,1)}function ca(a,b,c,d){var e=a.aoData[b],f,g=function(c,d){for(;c.childNodes.length;)c.removeChild(c.firstChild);
+c.innerHTML=B(a,b,d,"display")};if("dom"===c||(!c||"auto"===c)&&"dom"===e.src)e._aData=Ia(a,e,d,d===k?k:e._aData).data;else{var j=e.anCells;if(j)if(d!==k)g(j[d],d);else{c=0;for(f=j.length;c<f;c++)g(j[c],c)}}e._aSortData=null;e._aFilterData=null;g=a.aoColumns;if(d!==k)g[d].sType=null;else{c=0;for(f=g.length;c<f;c++)g[c].sType=null;La(a,e)}}function Ia(a,b,c,d){var e=[],f=b.firstChild,g,j,i=0,n,l=a.aoColumns,q=a._rowReadObject,d=d!==k?d:q?{}:[],t=function(a,b){if("string"===typeof a){var c=a.indexOf("@");
+-1!==c&&(c=a.substring(c+1),R(a)(d,b.getAttribute(c)))}},S=function(a){if(c===k||c===i)j=l[i],n=h.trim(a.innerHTML),j&&j._bAttrSrc?(R(j.mData._)(d,n),t(j.mData.sort,a),t(j.mData.type,a),t(j.mData.filter,a)):q?(j._setter||(j._setter=R(j.mData)),j._setter(d,n)):d[i]=n;i++};if(f)for(;f;){g=f.nodeName.toUpperCase();if("TD"==g||"TH"==g)S(f),e.push(f);f=f.nextSibling}else{e=b.anCells;f=0;for(g=e.length;f<g;f++)S(e[f])}if(b=b.firstChild?b:b.nTr)(b=b.getAttribute("id"))&&R(a.rowId)(d,b);return{data:d,cells:e}}
+function Ha(a,b,c,d){var e=a.aoData[b],f=e._aData,g=[],j,i,n,l,q;if(null===e.nTr){j=c||I.createElement("tr");e.nTr=j;e.anCells=g;j._DT_RowIndex=b;La(a,e);l=0;for(q=a.aoColumns.length;l<q;l++){n=a.aoColumns[l];i=c?d[l]:I.createElement(n.sCellType);i._DT_CellIndex={row:b,column:l};g.push(i);if((!c||n.mRender||n.mData!==l)&&(!h.isPlainObject(n.mData)||n.mData._!==l+".display"))i.innerHTML=B(a,b,l,"display");n.sClass&&(i.className+=" "+n.sClass);n.bVisible&&!c?j.appendChild(i):!n.bVisible&&c&&i.parentNode.removeChild(i);
+n.fnCreatedCell&&n.fnCreatedCell.call(a.oInstance,i,B(a,b,l),f,b,l)}u(a,"aoRowCreatedCallback",null,[j,f,b])}e.nTr.setAttribute("role","row")}function La(a,b){var c=b.nTr,d=b._aData;if(c){var e=a.rowIdFn(d);e&&(c.id=e);d.DT_RowClass&&(e=d.DT_RowClass.split(" "),b.__rowc=b.__rowc?pa(b.__rowc.concat(e)):e,h(c).removeClass(b.__rowc.join(" ")).addClass(d.DT_RowClass));d.DT_RowAttr&&h(c).attr(d.DT_RowAttr);d.DT_RowData&&h(c).data(d.DT_RowData)}}function kb(a){var b,c,d,e,f,g=a.nTHead,j=a.nTFoot,i=0===
+h("th, td",g).length,n=a.oClasses,l=a.aoColumns;i&&(e=h("<tr/>").appendTo(g));b=0;for(c=l.length;b<c;b++)f=l[b],d=h(f.nTh).addClass(f.sClass),i&&d.appendTo(e),a.oFeatures.bSort&&(d.addClass(f.sSortingClass),!1!==f.bSortable&&(d.attr("tabindex",a.iTabIndex).attr("aria-controls",a.sTableId),Ma(a,f.nTh,b))),f.sTitle!=d[0].innerHTML&&d.html(f.sTitle),Na(a,"header")(a,d,f,n);i&&da(a.aoHeader,g);h(g).find(">tr").attr("role","row");h(g).find(">tr>th, >tr>td").addClass(n.sHeaderTH);h(j).find(">tr>th, >tr>td").addClass(n.sFooterTH);
+if(null!==j){a=a.aoFooter[0];b=0;for(c=a.length;b<c;b++)f=l[b],f.nTf=a[b].cell,f.sClass&&h(f.nTf).addClass(f.sClass)}}function ea(a,b,c){var d,e,f,g=[],j=[],i=a.aoColumns.length,n;if(b){c===k&&(c=!1);d=0;for(e=b.length;d<e;d++){g[d]=b[d].slice();g[d].nTr=b[d].nTr;for(f=i-1;0<=f;f--)!a.aoColumns[f].bVisible&&!c&&g[d].splice(f,1);j.push([])}d=0;for(e=g.length;d<e;d++){if(a=g[d].nTr)for(;f=a.firstChild;)a.removeChild(f);f=0;for(b=g[d].length;f<b;f++)if(n=i=1,j[d][f]===k){a.appendChild(g[d][f].cell);
+for(j[d][f]=1;g[d+i]!==k&&g[d][f].cell==g[d+i][f].cell;)j[d+i][f]=1,i++;for(;g[d][f+n]!==k&&g[d][f].cell==g[d][f+n].cell;){for(c=0;c<i;c++)j[d+c][f+n]=1;n++}h(g[d][f].cell).attr("rowspan",i).attr("colspan",n)}}}}function O(a){var b=u(a,"aoPreDrawCallback","preDraw",[a]);if(-1!==h.inArray(!1,b))C(a,!1);else{var b=[],c=0,d=a.asStripeClasses,e=d.length,f=a.oLanguage,g=a.iInitDisplayStart,j="ssp"==y(a),i=a.aiDisplay;a.bDrawing=!0;g!==k&&-1!==g&&(a._iDisplayStart=j?g:g>=a.fnRecordsDisplay()?0:g,a.iInitDisplayStart=
+-1);var g=a._iDisplayStart,n=a.fnDisplayEnd();if(a.bDeferLoading)a.bDeferLoading=!1,a.iDraw++,C(a,!1);else if(j){if(!a.bDestroying&&!lb(a))return}else a.iDraw++;if(0!==i.length){f=j?a.aoData.length:n;for(j=j?0:g;j<f;j++){var l=i[j],q=a.aoData[l];null===q.nTr&&Ha(a,l);l=q.nTr;if(0!==e){var t=d[c%e];q._sRowStripe!=t&&(h(l).removeClass(q._sRowStripe).addClass(t),q._sRowStripe=t)}u(a,"aoRowCallback",null,[l,q._aData,c,j]);b.push(l);c++}}else c=f.sZeroRecords,1==a.iDraw&&"ajax"==y(a)?c=f.sLoadingRecords:
+f.sEmptyTable&&0===a.fnRecordsTotal()&&(c=f.sEmptyTable),b[0]=h("<tr/>",{"class":e?d[0]:""}).append(h("<td />",{valign:"top",colSpan:aa(a),"class":a.oClasses.sRowEmpty}).html(c))[0];u(a,"aoHeaderCallback","header",[h(a.nTHead).children("tr")[0],Ka(a),g,n,i]);u(a,"aoFooterCallback","footer",[h(a.nTFoot).children("tr")[0],Ka(a),g,n,i]);d=h(a.nTBody);d.children().detach();d.append(h(b));u(a,"aoDrawCallback","draw",[a]);a.bSorted=!1;a.bFiltered=!1;a.bDrawing=!1}}function T(a,b){var c=a.oFeatures,d=c.bFilter;
+c.bSort&&mb(a);d?fa(a,a.oPreviousSearch):a.aiDisplay=a.aiDisplayMaster.slice();!0!==b&&(a._iDisplayStart=0);a._drawHold=b;O(a);a._drawHold=!1}function nb(a){var b=a.oClasses,c=h(a.nTable),c=h("<div/>").insertBefore(c),d=a.oFeatures,e=h("<div/>",{id:a.sTableId+"_wrapper","class":b.sWrapper+(a.nTFoot?"":" "+b.sNoFooter)});a.nHolding=c[0];a.nTableWrapper=e[0];a.nTableReinsertBefore=a.nTable.nextSibling;for(var f=a.sDom.split(""),g,j,i,n,l,q,t=0;t<f.length;t++){g=null;j=f[t];if("<"==j){i=h("<div/>")[0];
+n=f[t+1];if("'"==n||'"'==n){l="";for(q=2;f[t+q]!=n;)l+=f[t+q],q++;"H"==l?l=b.sJUIHeader:"F"==l&&(l=b.sJUIFooter);-1!=l.indexOf(".")?(n=l.split("."),i.id=n[0].substr(1,n[0].length-1),i.className=n[1]):"#"==l.charAt(0)?i.id=l.substr(1,l.length-1):i.className=l;t+=q}e.append(i);e=h(i)}else if(">"==j)e=e.parent();else if("l"==j&&d.bPaginate&&d.bLengthChange)g=ob(a);else if("f"==j&&d.bFilter)g=pb(a);else if("r"==j&&d.bProcessing)g=qb(a);else if("t"==j)g=rb(a);else if("i"==j&&d.bInfo)g=sb(a);else if("p"==
+j&&d.bPaginate)g=tb(a);else if(0!==m.ext.feature.length){i=m.ext.feature;q=0;for(n=i.length;q<n;q++)if(j==i[q].cFeature){g=i[q].fnInit(a);break}}g&&(i=a.aanFeatures,i[j]||(i[j]=[]),i[j].push(g),e.append(g))}c.replaceWith(e);a.nHolding=null}function da(a,b){var c=h(b).children("tr"),d,e,f,g,j,i,n,l,q,t;a.splice(0,a.length);f=0;for(i=c.length;f<i;f++)a.push([]);f=0;for(i=c.length;f<i;f++){d=c[f];for(e=d.firstChild;e;){if("TD"==e.nodeName.toUpperCase()||"TH"==e.nodeName.toUpperCase()){l=1*e.getAttribute("colspan");
+q=1*e.getAttribute("rowspan");l=!l||0===l||1===l?1:l;q=!q||0===q||1===q?1:q;g=0;for(j=a[f];j[g];)g++;n=g;t=1===l?!0:!1;for(j=0;j<l;j++)for(g=0;g<q;g++)a[f+g][n+j]={cell:e,unique:t},a[f+g].nTr=d}e=e.nextSibling}}}function qa(a,b,c){var d=[];c||(c=a.aoHeader,b&&(c=[],da(c,b)));for(var b=0,e=c.length;b<e;b++)for(var f=0,g=c[b].length;f<g;f++)if(c[b][f].unique&&(!d[f]||!a.bSortCellsTop))d[f]=c[b][f].cell;return d}function ra(a,b,c){u(a,"aoServerParams","serverParams",[b]);if(b&&h.isArray(b)){var d={},
+e=/(.*?)\[\]$/;h.each(b,function(a,b){var c=b.name.match(e);c?(c=c[0],d[c]||(d[c]=[]),d[c].push(b.value)):d[b.name]=b.value});b=d}var f,g=a.ajax,j=a.oInstance,i=function(b){u(a,null,"xhr",[a,b,a.jqXHR]);c(b)};if(h.isPlainObject(g)&&g.data){f=g.data;var n=h.isFunction(f)?f(b,a):f,b=h.isFunction(f)&&n?n:h.extend(!0,b,n);delete g.data}n={data:b,success:function(b){var c=b.error||b.sError;c&&L(a,0,c);a.json=b;i(b)},dataType:"json",cache:!1,type:a.sServerMethod,error:function(b,c){var d=u(a,null,"xhr",
+[a,null,a.jqXHR]);-1===h.inArray(!0,d)&&("parsererror"==c?L(a,0,"Invalid JSON response",1):4===b.readyState&&L(a,0,"Ajax error",7));C(a,!1)}};a.oAjaxData=b;u(a,null,"preXhr",[a,b]);a.fnServerData?a.fnServerData.call(j,a.sAjaxSource,h.map(b,function(a,b){return{name:b,value:a}}),i,a):a.sAjaxSource||"string"===typeof g?a.jqXHR=h.ajax(h.extend(n,{url:g||a.sAjaxSource})):h.isFunction(g)?a.jqXHR=g.call(j,b,i,a):(a.jqXHR=h.ajax(h.extend(n,g)),g.data=f)}function lb(a){return a.bAjaxDataGet?(a.iDraw++,C(a,
+!0),ra(a,ub(a),function(b){vb(a,b)}),!1):!0}function ub(a){var b=a.aoColumns,c=b.length,d=a.oFeatures,e=a.oPreviousSearch,f=a.aoPreSearchCols,g,j=[],i,n,l,q=V(a);g=a._iDisplayStart;i=!1!==d.bPaginate?a._iDisplayLength:-1;var k=function(a,b){j.push({name:a,value:b})};k("sEcho",a.iDraw);k("iColumns",c);k("sColumns",G(b,"sName").join(","));k("iDisplayStart",g);k("iDisplayLength",i);var S={draw:a.iDraw,columns:[],order:[],start:g,length:i,search:{value:e.sSearch,regex:e.bRegex}};for(g=0;g<c;g++)n=b[g],
+l=f[g],i="function"==typeof n.mData?"function":n.mData,S.columns.push({data:i,name:n.sName,searchable:n.bSearchable,orderable:n.bSortable,search:{value:l.sSearch,regex:l.bRegex}}),k("mDataProp_"+g,i),d.bFilter&&(k("sSearch_"+g,l.sSearch),k("bRegex_"+g,l.bRegex),k("bSearchable_"+g,n.bSearchable)),d.bSort&&k("bSortable_"+g,n.bSortable);d.bFilter&&(k("sSearch",e.sSearch),k("bRegex",e.bRegex));d.bSort&&(h.each(q,function(a,b){S.order.push({column:b.col,dir:b.dir});k("iSortCol_"+a,b.col);k("sSortDir_"+
+a,b.dir)}),k("iSortingCols",q.length));b=m.ext.legacy.ajax;return null===b?a.sAjaxSource?j:S:b?j:S}function vb(a,b){var c=sa(a,b),d=b.sEcho!==k?b.sEcho:b.draw,e=b.iTotalRecords!==k?b.iTotalRecords:b.recordsTotal,f=b.iTotalDisplayRecords!==k?b.iTotalDisplayRecords:b.recordsFiltered;if(d){if(1*d<a.iDraw)return;a.iDraw=1*d}na(a);a._iRecordsTotal=parseInt(e,10);a._iRecordsDisplay=parseInt(f,10);d=0;for(e=c.length;d<e;d++)N(a,c[d]);a.aiDisplay=a.aiDisplayMaster.slice();a.bAjaxDataGet=!1;O(a);a._bInitComplete||
+ta(a,b);a.bAjaxDataGet=!0;C(a,!1)}function sa(a,b){var c=h.isPlainObject(a.ajax)&&a.ajax.dataSrc!==k?a.ajax.dataSrc:a.sAjaxDataProp;return"data"===c?b.aaData||b[c]:""!==c?Q(c)(b):b}function pb(a){var b=a.oClasses,c=a.sTableId,d=a.oLanguage,e=a.oPreviousSearch,f=a.aanFeatures,g='<input type="search" class="'+b.sFilterInput+'"/>',j=d.sSearch,j=j.match(/_INPUT_/)?j.replace("_INPUT_",g):j+g,b=h("<div/>",{id:!f.f?c+"_filter":null,"class":b.sFilter}).append(h("<label/>").append(j)),f=function(){var b=!this.value?
+"":this.value;b!=e.sSearch&&(fa(a,{sSearch:b,bRegex:e.bRegex,bSmart:e.bSmart,bCaseInsensitive:e.bCaseInsensitive}),a._iDisplayStart=0,O(a))},g=null!==a.searchDelay?a.searchDelay:"ssp"===y(a)?400:0,i=h("input",b).val(e.sSearch).attr("placeholder",d.sSearchPlaceholder).bind("keyup.DT search.DT input.DT paste.DT cut.DT",g?Oa(f,g):f).bind("keypress.DT",function(a){if(13==a.keyCode)return!1}).attr("aria-controls",c);h(a.nTable).on("search.dt.DT",function(b,c){if(a===c)try{i[0]!==I.activeElement&&i.val(e.sSearch)}catch(d){}});
+return b[0]}function fa(a,b,c){var d=a.oPreviousSearch,e=a.aoPreSearchCols,f=function(a){d.sSearch=a.sSearch;d.bRegex=a.bRegex;d.bSmart=a.bSmart;d.bCaseInsensitive=a.bCaseInsensitive};Ga(a);if("ssp"!=y(a)){wb(a,b.sSearch,c,b.bEscapeRegex!==k?!b.bEscapeRegex:b.bRegex,b.bSmart,b.bCaseInsensitive);f(b);for(b=0;b<e.length;b++)xb(a,e[b].sSearch,b,e[b].bEscapeRegex!==k?!e[b].bEscapeRegex:e[b].bRegex,e[b].bSmart,e[b].bCaseInsensitive);yb(a)}else f(b);a.bFiltered=!0;u(a,null,"search",[a])}function yb(a){for(var b=
+m.ext.search,c=a.aiDisplay,d,e,f=0,g=b.length;f<g;f++){for(var j=[],i=0,n=c.length;i<n;i++)e=c[i],d=a.aoData[e],b[f](a,d._aFilterData,e,d._aData,i)&&j.push(e);c.length=0;h.merge(c,j)}}function xb(a,b,c,d,e,f){if(""!==b)for(var g=a.aiDisplay,d=Pa(b,d,e,f),e=g.length-1;0<=e;e--)b=a.aoData[g[e]]._aFilterData[c],d.test(b)||g.splice(e,1)}function wb(a,b,c,d,e,f){var d=Pa(b,d,e,f),e=a.oPreviousSearch.sSearch,f=a.aiDisplayMaster,g;0!==m.ext.search.length&&(c=!0);g=zb(a);if(0>=b.length)a.aiDisplay=f.slice();
+else{if(g||c||e.length>b.length||0!==b.indexOf(e)||a.bSorted)a.aiDisplay=f.slice();b=a.aiDisplay;for(c=b.length-1;0<=c;c--)d.test(a.aoData[b[c]]._sFilterRow)||b.splice(c,1)}}function Pa(a,b,c,d){a=b?a:Qa(a);c&&(a="^(?=.*?"+h.map(a.match(/"[^"]+"|[^ ]+/g)||[""],function(a){if('"'===a.charAt(0))var b=a.match(/^"(.*)"$/),a=b?b[1]:a;return a.replace('"',"")}).join(")(?=.*?")+").*$");return RegExp(a,d?"i":"")}function zb(a){var b=a.aoColumns,c,d,e,f,g,j,i,h,l=m.ext.type.search;c=!1;d=0;for(f=a.aoData.length;d<
+f;d++)if(h=a.aoData[d],!h._aFilterData){j=[];e=0;for(g=b.length;e<g;e++)c=b[e],c.bSearchable?(i=B(a,d,e,"filter"),l[c.sType]&&(i=l[c.sType](i)),null===i&&(i=""),"string"!==typeof i&&i.toString&&(i=i.toString())):i="",i.indexOf&&-1!==i.indexOf("&")&&(ua.innerHTML=i,i=Zb?ua.textContent:ua.innerText),i.replace&&(i=i.replace(/[\r\n]/g,"")),j.push(i);h._aFilterData=j;h._sFilterRow=j.join("  ");c=!0}return c}function Ab(a){return{search:a.sSearch,smart:a.bSmart,regex:a.bRegex,caseInsensitive:a.bCaseInsensitive}}
+function Bb(a){return{sSearch:a.search,bSmart:a.smart,bRegex:a.regex,bCaseInsensitive:a.caseInsensitive}}function sb(a){var b=a.sTableId,c=a.aanFeatures.i,d=h("<div/>",{"class":a.oClasses.sInfo,id:!c?b+"_info":null});c||(a.aoDrawCallback.push({fn:Cb,sName:"information"}),d.attr("role","status").attr("aria-live","polite"),h(a.nTable).attr("aria-describedby",b+"_info"));return d[0]}function Cb(a){var b=a.aanFeatures.i;if(0!==b.length){var c=a.oLanguage,d=a._iDisplayStart+1,e=a.fnDisplayEnd(),f=a.fnRecordsTotal(),
+g=a.fnRecordsDisplay(),j=g?c.sInfo:c.sInfoEmpty;g!==f&&(j+=" "+c.sInfoFiltered);j+=c.sInfoPostFix;j=Db(a,j);c=c.fnInfoCallback;null!==c&&(j=c.call(a.oInstance,a,d,e,f,g,j));h(b).html(j)}}function Db(a,b){var c=a.fnFormatNumber,d=a._iDisplayStart+1,e=a._iDisplayLength,f=a.fnRecordsDisplay(),g=-1===e;return b.replace(/_START_/g,c.call(a,d)).replace(/_END_/g,c.call(a,a.fnDisplayEnd())).replace(/_MAX_/g,c.call(a,a.fnRecordsTotal())).replace(/_TOTAL_/g,c.call(a,f)).replace(/_PAGE_/g,c.call(a,g?1:Math.ceil(d/
+e))).replace(/_PAGES_/g,c.call(a,g?1:Math.ceil(f/e)))}function ga(a){var b,c,d=a.iInitDisplayStart,e=a.aoColumns,f;c=a.oFeatures;var g=a.bDeferLoading;if(a.bInitialised){nb(a);kb(a);ea(a,a.aoHeader);ea(a,a.aoFooter);C(a,!0);c.bAutoWidth&&Fa(a);b=0;for(c=e.length;b<c;b++)f=e[b],f.sWidth&&(f.nTh.style.width=x(f.sWidth));u(a,null,"preInit",[a]);T(a);e=y(a);if("ssp"!=e||g)"ajax"==e?ra(a,[],function(c){var f=sa(a,c);for(b=0;b<f.length;b++)N(a,f[b]);a.iInitDisplayStart=d;T(a);C(a,!1);ta(a,c)},a):(C(a,!1),
+ta(a))}else setTimeout(function(){ga(a)},200)}function ta(a,b){a._bInitComplete=!0;(b||a.oInit.aaData)&&Y(a);u(a,null,"plugin-init",[a,b]);u(a,"aoInitComplete","init",[a,b])}function Ra(a,b){var c=parseInt(b,10);a._iDisplayLength=c;Sa(a);u(a,null,"length",[a,c])}function ob(a){for(var b=a.oClasses,c=a.sTableId,d=a.aLengthMenu,e=h.isArray(d[0]),f=e?d[0]:d,d=e?d[1]:d,e=h("<select/>",{name:c+"_length","aria-controls":c,"class":b.sLengthSelect}),g=0,j=f.length;g<j;g++)e[0][g]=new Option(d[g],f[g]);var i=
+h("<div><label/></div>").addClass(b.sLength);a.aanFeatures.l||(i[0].id=c+"_length");i.children().append(a.oLanguage.sLengthMenu.replace("_MENU_",e[0].outerHTML));h("select",i).val(a._iDisplayLength).bind("change.DT",function(){Ra(a,h(this).val());O(a)});h(a.nTable).bind("length.dt.DT",function(b,c,d){a===c&&h("select",i).val(d)});return i[0]}function tb(a){var b=a.sPaginationType,c=m.ext.pager[b],d="function"===typeof c,e=function(a){O(a)},b=h("<div/>").addClass(a.oClasses.sPaging+b)[0],f=a.aanFeatures;
+d||c.fnInit(a,b,e);f.p||(b.id=a.sTableId+"_paginate",a.aoDrawCallback.push({fn:function(a){if(d){var b=a._iDisplayStart,i=a._iDisplayLength,h=a.fnRecordsDisplay(),l=-1===i,b=l?0:Math.ceil(b/i),i=l?1:Math.ceil(h/i),h=c(b,i),k,l=0;for(k=f.p.length;l<k;l++)Na(a,"pageButton")(a,f.p[l],l,h,b,i)}else c.fnUpdate(a,e)},sName:"pagination"}));return b}function Ta(a,b,c){var d=a._iDisplayStart,e=a._iDisplayLength,f=a.fnRecordsDisplay();0===f||-1===e?d=0:"number"===typeof b?(d=b*e,d>f&&(d=0)):"first"==b?d=0:
+"previous"==b?(d=0<=e?d-e:0,0>d&&(d=0)):"next"==b?d+e<f&&(d+=e):"last"==b?d=Math.floor((f-1)/e)*e:L(a,0,"Unknown paging action: "+b,5);b=a._iDisplayStart!==d;a._iDisplayStart=d;b&&(u(a,null,"page",[a]),c&&O(a));return b}function qb(a){return h("<div/>",{id:!a.aanFeatures.r?a.sTableId+"_processing":null,"class":a.oClasses.sProcessing}).html(a.oLanguage.sProcessing).insertBefore(a.nTable)[0]}function C(a,b){a.oFeatures.bProcessing&&h(a.aanFeatures.r).css("display",b?"block":"none");u(a,null,"processing",
+[a,b])}function rb(a){var b=h(a.nTable);b.attr("role","grid");var c=a.oScroll;if(""===c.sX&&""===c.sY)return a.nTable;var d=c.sX,e=c.sY,f=a.oClasses,g=b.children("caption"),j=g.length?g[0]._captionSide:null,i=h(b[0].cloneNode(!1)),n=h(b[0].cloneNode(!1)),l=b.children("tfoot");l.length||(l=null);i=h("<div/>",{"class":f.sScrollWrapper}).append(h("<div/>",{"class":f.sScrollHead}).css({overflow:"hidden",position:"relative",border:0,width:d?!d?null:x(d):"100%"}).append(h("<div/>",{"class":f.sScrollHeadInner}).css({"box-sizing":"content-box",
+width:c.sXInner||"100%"}).append(i.removeAttr("id").css("margin-left",0).append("top"===j?g:null).append(b.children("thead"))))).append(h("<div/>",{"class":f.sScrollBody}).css({position:"relative",overflow:"auto",width:!d?null:x(d)}).append(b));l&&i.append(h("<div/>",{"class":f.sScrollFoot}).css({overflow:"hidden",border:0,width:d?!d?null:x(d):"100%"}).append(h("<div/>",{"class":f.sScrollFootInner}).append(n.removeAttr("id").css("margin-left",0).append("bottom"===j?g:null).append(b.children("tfoot")))));
+var b=i.children(),k=b[0],f=b[1],t=l?b[2]:null;if(d)h(f).on("scroll.DT",function(){var a=this.scrollLeft;k.scrollLeft=a;l&&(t.scrollLeft=a)});h(f).css(e&&c.bCollapse?"max-height":"height",e);a.nScrollHead=k;a.nScrollBody=f;a.nScrollFoot=t;a.aoDrawCallback.push({fn:ka,sName:"scrolling"});return i[0]}function ka(a){var b=a.oScroll,c=b.sX,d=b.sXInner,e=b.sY,b=b.iBarWidth,f=h(a.nScrollHead),g=f[0].style,j=f.children("div"),i=j[0].style,n=j.children("table"),j=a.nScrollBody,l=h(j),q=j.style,t=h(a.nScrollFoot).children("div"),
+m=t.children("table"),o=h(a.nTHead),F=h(a.nTable),p=F[0],r=p.style,u=a.nTFoot?h(a.nTFoot):null,Eb=a.oBrowser,Ua=Eb.bScrollOversize,s=G(a.aoColumns,"nTh"),P,v,w,y,z=[],A=[],B=[],C=[],D,E=function(a){a=a.style;a.paddingTop="0";a.paddingBottom="0";a.borderTopWidth="0";a.borderBottomWidth="0";a.height=0};v=j.scrollHeight>j.clientHeight;if(a.scrollBarVis!==v&&a.scrollBarVis!==k)a.scrollBarVis=v,Y(a);else{a.scrollBarVis=v;F.children("thead, tfoot").remove();u&&(w=u.clone().prependTo(F),P=u.find("tr"),w=
+w.find("tr"));y=o.clone().prependTo(F);o=o.find("tr");v=y.find("tr");y.find("th, td").removeAttr("tabindex");c||(q.width="100%",f[0].style.width="100%");h.each(qa(a,y),function(b,c){D=Z(a,b);c.style.width=a.aoColumns[D].sWidth});u&&J(function(a){a.style.width=""},w);f=F.outerWidth();if(""===c){r.width="100%";if(Ua&&(F.find("tbody").height()>j.offsetHeight||"scroll"==l.css("overflow-y")))r.width=x(F.outerWidth()-b);f=F.outerWidth()}else""!==d&&(r.width=x(d),f=F.outerWidth());J(E,v);J(function(a){B.push(a.innerHTML);
+z.push(x(h(a).css("width")))},v);J(function(a,b){if(h.inArray(a,s)!==-1)a.style.width=z[b]},o);h(v).height(0);u&&(J(E,w),J(function(a){C.push(a.innerHTML);A.push(x(h(a).css("width")))},w),J(function(a,b){a.style.width=A[b]},P),h(w).height(0));J(function(a,b){a.innerHTML='<div class="dataTables_sizing" style="height:0;overflow:hidden;">'+B[b]+"</div>";a.style.width=z[b]},v);u&&J(function(a,b){a.innerHTML='<div class="dataTables_sizing" style="height:0;overflow:hidden;">'+C[b]+"</div>";a.style.width=
+A[b]},w);if(F.outerWidth()<f){P=j.scrollHeight>j.offsetHeight||"scroll"==l.css("overflow-y")?f+b:f;if(Ua&&(j.scrollHeight>j.offsetHeight||"scroll"==l.css("overflow-y")))r.width=x(P-b);(""===c||""!==d)&&L(a,1,"Possible column misalignment",6)}else P="100%";q.width=x(P);g.width=x(P);u&&(a.nScrollFoot.style.width=x(P));!e&&Ua&&(q.height=x(p.offsetHeight+b));c=F.outerWidth();n[0].style.width=x(c);i.width=x(c);d=F.height()>j.clientHeight||"scroll"==l.css("overflow-y");e="padding"+(Eb.bScrollbarLeft?"Left":
+"Right");i[e]=d?b+"px":"0px";u&&(m[0].style.width=x(c),t[0].style.width=x(c),t[0].style[e]=d?b+"px":"0px");F.children("colgroup").insertBefore(F.children("thead"));l.scroll();if((a.bSorted||a.bFiltered)&&!a._drawHold)j.scrollTop=0}}function J(a,b,c){for(var d=0,e=0,f=b.length,g,j;e<f;){g=b[e].firstChild;for(j=c?c[e].firstChild:null;g;)1===g.nodeType&&(c?a(g,j,d):a(g,d),d++),g=g.nextSibling,j=c?j.nextSibling:null;e++}}function Fa(a){var b=a.nTable,c=a.aoColumns,d=a.oScroll,e=d.sY,f=d.sX,g=d.sXInner,
+j=c.length,i=la(a,"bVisible"),n=h("th",a.nTHead),l=b.getAttribute("width"),k=b.parentNode,t=!1,m,o,p=a.oBrowser,d=p.bScrollOversize;(m=b.style.width)&&-1!==m.indexOf("%")&&(l=m);for(m=0;m<i.length;m++)o=c[i[m]],null!==o.sWidth&&(o.sWidth=Fb(o.sWidthOrig,k),t=!0);if(d||!t&&!f&&!e&&j==aa(a)&&j==n.length)for(m=0;m<j;m++)i=Z(a,m),null!==i&&(c[i].sWidth=x(n.eq(m).width()));else{j=h(b).clone().css("visibility","hidden").removeAttr("id");j.find("tbody tr").remove();var r=h("<tr/>").appendTo(j.find("tbody"));
+j.find("thead, tfoot").remove();j.append(h(a.nTHead).clone()).append(h(a.nTFoot).clone());j.find("tfoot th, tfoot td").css("width","");n=qa(a,j.find("thead")[0]);for(m=0;m<i.length;m++)o=c[i[m]],n[m].style.width=null!==o.sWidthOrig&&""!==o.sWidthOrig?x(o.sWidthOrig):"",o.sWidthOrig&&f&&h(n[m]).append(h("<div/>").css({width:o.sWidthOrig,margin:0,padding:0,border:0,height:1}));if(a.aoData.length)for(m=0;m<i.length;m++)t=i[m],o=c[t],h(Gb(a,t)).clone(!1).append(o.sContentPadding).appendTo(r);h("[name]",
+j).removeAttr("name");o=h("<div/>").css(f||e?{position:"absolute",top:0,left:0,height:1,right:0,overflow:"hidden"}:{}).append(j).appendTo(k);f&&g?j.width(g):f?(j.css("width","auto"),j.removeAttr("width"),j.width()<k.clientWidth&&l&&j.width(k.clientWidth)):e?j.width(k.clientWidth):l&&j.width(l);for(m=e=0;m<i.length;m++)k=h(n[m]),g=k.outerWidth()-k.width(),k=p.bBounding?Math.ceil(n[m].getBoundingClientRect().width):k.outerWidth(),e+=k,c[i[m]].sWidth=x(k-g);b.style.width=x(e);o.remove()}l&&(b.style.width=
+x(l));if((l||f)&&!a._reszEvt)b=function(){h(D).bind("resize.DT-"+a.sInstance,Oa(function(){Y(a)}))},d?setTimeout(b,1E3):b(),a._reszEvt=!0}function Fb(a,b){if(!a)return 0;var c=h("<div/>").css("width",x(a)).appendTo(b||I.body),d=c[0].offsetWidth;c.remove();return d}function Gb(a,b){var c=Hb(a,b);if(0>c)return null;var d=a.aoData[c];return!d.nTr?h("<td/>").html(B(a,c,b,"display"))[0]:d.anCells[b]}function Hb(a,b){for(var c,d=-1,e=-1,f=0,g=a.aoData.length;f<g;f++)c=B(a,f,b,"display")+"",c=c.replace($b,
+""),c=c.replace(/&nbsp;/g," "),c.length>d&&(d=c.length,e=f);return e}function x(a){return null===a?"0px":"number"==typeof a?0>a?"0px":a+"px":a.match(/\d$/)?a+"px":a}function V(a){var b,c,d=[],e=a.aoColumns,f,g,j,i;b=a.aaSortingFixed;c=h.isPlainObject(b);var n=[];f=function(a){a.length&&!h.isArray(a[0])?n.push(a):h.merge(n,a)};h.isArray(b)&&f(b);c&&b.pre&&f(b.pre);f(a.aaSorting);c&&b.post&&f(b.post);for(a=0;a<n.length;a++){i=n[a][0];f=e[i].aDataSort;b=0;for(c=f.length;b<c;b++)g=f[b],j=e[g].sType||
+"string",n[a]._idx===k&&(n[a]._idx=h.inArray(n[a][1],e[g].asSorting)),d.push({src:i,col:g,dir:n[a][1],index:n[a]._idx,type:j,formatter:m.ext.type.order[j+"-pre"]})}return d}function mb(a){var b,c,d=[],e=m.ext.type.order,f=a.aoData,g=0,j,i=a.aiDisplayMaster,h;Ga(a);h=V(a);b=0;for(c=h.length;b<c;b++)j=h[b],j.formatter&&g++,Ib(a,j.col);if("ssp"!=y(a)&&0!==h.length){b=0;for(c=i.length;b<c;b++)d[i[b]]=b;g===h.length?i.sort(function(a,b){var c,e,g,j,i=h.length,k=f[a]._aSortData,m=f[b]._aSortData;for(g=
+0;g<i;g++)if(j=h[g],c=k[j.col],e=m[j.col],c=c<e?-1:c>e?1:0,0!==c)return"asc"===j.dir?c:-c;c=d[a];e=d[b];return c<e?-1:c>e?1:0}):i.sort(function(a,b){var c,g,j,i,k=h.length,m=f[a]._aSortData,p=f[b]._aSortData;for(j=0;j<k;j++)if(i=h[j],c=m[i.col],g=p[i.col],i=e[i.type+"-"+i.dir]||e["string-"+i.dir],c=i(c,g),0!==c)return c;c=d[a];g=d[b];return c<g?-1:c>g?1:0})}a.bSorted=!0}function Jb(a){for(var b,c,d=a.aoColumns,e=V(a),a=a.oLanguage.oAria,f=0,g=d.length;f<g;f++){c=d[f];var j=c.asSorting;b=c.sTitle.replace(/<.*?>/g,
+"");var i=c.nTh;i.removeAttribute("aria-sort");c.bSortable&&(0<e.length&&e[0].col==f?(i.setAttribute("aria-sort","asc"==e[0].dir?"ascending":"descending"),c=j[e[0].index+1]||j[0]):c=j[0],b+="asc"===c?a.sSortAscending:a.sSortDescending);i.setAttribute("aria-label",b)}}function Va(a,b,c,d){var e=a.aaSorting,f=a.aoColumns[b].asSorting,g=function(a,b){var c=a._idx;c===k&&(c=h.inArray(a[1],f));return c+1<f.length?c+1:b?null:0};"number"===typeof e[0]&&(e=a.aaSorting=[e]);c&&a.oFeatures.bSortMulti?(c=h.inArray(b,
+G(e,"0")),-1!==c?(b=g(e[c],!0),null===b&&1===e.length&&(b=0),null===b?e.splice(c,1):(e[c][1]=f[b],e[c]._idx=b)):(e.push([b,f[0],0]),e[e.length-1]._idx=0)):e.length&&e[0][0]==b?(b=g(e[0]),e.length=1,e[0][1]=f[b],e[0]._idx=b):(e.length=0,e.push([b,f[0]]),e[0]._idx=0);T(a);"function"==typeof d&&d(a)}function Ma(a,b,c,d){var e=a.aoColumns[c];Wa(b,{},function(b){!1!==e.bSortable&&(a.oFeatures.bProcessing?(C(a,!0),setTimeout(function(){Va(a,c,b.shiftKey,d);"ssp"!==y(a)&&C(a,!1)},0)):Va(a,c,b.shiftKey,d))})}
+function va(a){var b=a.aLastSort,c=a.oClasses.sSortColumn,d=V(a),e=a.oFeatures,f,g;if(e.bSort&&e.bSortClasses){e=0;for(f=b.length;e<f;e++)g=b[e].src,h(G(a.aoData,"anCells",g)).removeClass(c+(2>e?e+1:3));e=0;for(f=d.length;e<f;e++)g=d[e].src,h(G(a.aoData,"anCells",g)).addClass(c+(2>e?e+1:3))}a.aLastSort=d}function Ib(a,b){var c=a.aoColumns[b],d=m.ext.order[c.sSortDataType],e;d&&(e=d.call(a.oInstance,a,b,$(a,b)));for(var f,g=m.ext.type.order[c.sType+"-pre"],j=0,i=a.aoData.length;j<i;j++)if(c=a.aoData[j],
+c._aSortData||(c._aSortData=[]),!c._aSortData[b]||d)f=d?e[j]:B(a,j,b,"sort"),c._aSortData[b]=g?g(f):f}function wa(a){if(a.oFeatures.bStateSave&&!a.bDestroying){var b={time:+new Date,start:a._iDisplayStart,length:a._iDisplayLength,order:h.extend(!0,[],a.aaSorting),search:Ab(a.oPreviousSearch),columns:h.map(a.aoColumns,function(b,d){return{visible:b.bVisible,search:Ab(a.aoPreSearchCols[d])}})};u(a,"aoStateSaveParams","stateSaveParams",[a,b]);a.oSavedState=b;a.fnStateSaveCallback.call(a.oInstance,a,
+b)}}function Kb(a){var b,c,d=a.aoColumns;if(a.oFeatures.bStateSave){var e=a.fnStateLoadCallback.call(a.oInstance,a);if(e&&e.time&&(b=u(a,"aoStateLoadParams","stateLoadParams",[a,e]),-1===h.inArray(!1,b)&&(b=a.iStateDuration,!(0<b&&e.time<+new Date-1E3*b)&&d.length===e.columns.length))){a.oLoadedState=h.extend(!0,{},e);e.start!==k&&(a._iDisplayStart=e.start,a.iInitDisplayStart=e.start);e.length!==k&&(a._iDisplayLength=e.length);e.order!==k&&(a.aaSorting=[],h.each(e.order,function(b,c){a.aaSorting.push(c[0]>=
+d.length?[0,c[1]]:c)}));e.search!==k&&h.extend(a.oPreviousSearch,Bb(e.search));b=0;for(c=e.columns.length;b<c;b++){var f=e.columns[b];f.visible!==k&&(d[b].bVisible=f.visible);f.search!==k&&h.extend(a.aoPreSearchCols[b],Bb(f.search))}u(a,"aoStateLoaded","stateLoaded",[a,e])}}}function xa(a){var b=m.settings,a=h.inArray(a,G(b,"nTable"));return-1!==a?b[a]:null}function L(a,b,c,d){c="DataTables warning: "+(a?"table id="+a.sTableId+" - ":"")+c;d&&(c+=". For more information about this error, please see http://datatables.net/tn/"+
+d);if(b)D.console&&console.log&&console.log(c);else if(b=m.ext,b=b.sErrMode||b.errMode,a&&u(a,null,"error",[a,d,c]),"alert"==b)alert(c);else{if("throw"==b)throw Error(c);"function"==typeof b&&b(a,d,c)}}function E(a,b,c,d){h.isArray(c)?h.each(c,function(c,d){h.isArray(d)?E(a,b,d[0],d[1]):E(a,b,d)}):(d===k&&(d=c),b[c]!==k&&(a[d]=b[c]))}function Lb(a,b,c){var d,e;for(e in b)b.hasOwnProperty(e)&&(d=b[e],h.isPlainObject(d)?(h.isPlainObject(a[e])||(a[e]={}),h.extend(!0,a[e],d)):a[e]=c&&"data"!==e&&"aaData"!==
+e&&h.isArray(d)?d.slice():d);return a}function Wa(a,b,c){h(a).bind("click.DT",b,function(b){a.blur();c(b)}).bind("keypress.DT",b,function(a){13===a.which&&(a.preventDefault(),c(a))}).bind("selectstart.DT",function(){return!1})}function z(a,b,c,d){c&&a[b].push({fn:c,sName:d})}function u(a,b,c,d){var e=[];b&&(e=h.map(a[b].slice().reverse(),function(b){return b.fn.apply(a.oInstance,d)}));null!==c&&(b=h.Event(c+".dt"),h(a.nTable).trigger(b,d),e.push(b.result));return e}function Sa(a){var b=a._iDisplayStart,
+c=a.fnDisplayEnd(),d=a._iDisplayLength;b>=c&&(b=c-d);b-=b%d;if(-1===d||0>b)b=0;a._iDisplayStart=b}function Na(a,b){var c=a.renderer,d=m.ext.renderer[b];return h.isPlainObject(c)&&c[b]?d[c[b]]||d._:"string"===typeof c?d[c]||d._:d._}function y(a){return a.oFeatures.bServerSide?"ssp":a.ajax||a.sAjaxSource?"ajax":"dom"}function ya(a,b){var c=[],c=Mb.numbers_length,d=Math.floor(c/2);b<=c?c=W(0,b):a<=d?(c=W(0,c-2),c.push("ellipsis"),c.push(b-1)):(a>=b-1-d?c=W(b-(c-2),b):(c=W(a-d+2,a+d-1),c.push("ellipsis"),
+c.push(b-1)),c.splice(0,0,"ellipsis"),c.splice(0,0,0));c.DT_el="span";return c}function db(a){h.each({num:function(b){return za(b,a)},"num-fmt":function(b){return za(b,a,Xa)},"html-num":function(b){return za(b,a,Aa)},"html-num-fmt":function(b){return za(b,a,Aa,Xa)}},function(b,c){v.type.order[b+a+"-pre"]=c;b.match(/^html\-/)&&(v.type.search[b+a]=v.type.search.html)})}function Nb(a){return function(){var b=[xa(this[m.ext.iApiIndex])].concat(Array.prototype.slice.call(arguments));return m.ext.internal[a].apply(this,
+b)}}var m=function(a){this.$=function(a,b){return this.api(!0).$(a,b)};this._=function(a,b){return this.api(!0).rows(a,b).data()};this.api=function(a){return a?new r(xa(this[v.iApiIndex])):new r(this)};this.fnAddData=function(a,b){var c=this.api(!0),d=h.isArray(a)&&(h.isArray(a[0])||h.isPlainObject(a[0]))?c.rows.add(a):c.row.add(a);(b===k||b)&&c.draw();return d.flatten().toArray()};this.fnAdjustColumnSizing=function(a){var b=this.api(!0).columns.adjust(),c=b.settings()[0],d=c.oScroll;a===k||a?b.draw(!1):
+(""!==d.sX||""!==d.sY)&&ka(c)};this.fnClearTable=function(a){var b=this.api(!0).clear();(a===k||a)&&b.draw()};this.fnClose=function(a){this.api(!0).row(a).child.hide()};this.fnDeleteRow=function(a,b,c){var d=this.api(!0),a=d.rows(a),e=a.settings()[0],h=e.aoData[a[0][0]];a.remove();b&&b.call(this,e,h);(c===k||c)&&d.draw();return h};this.fnDestroy=function(a){this.api(!0).destroy(a)};this.fnDraw=function(a){this.api(!0).draw(a)};this.fnFilter=function(a,b,c,d,e,h){e=this.api(!0);null===b||b===k?e.search(a,
+c,d,h):e.column(b).search(a,c,d,h);e.draw()};this.fnGetData=function(a,b){var c=this.api(!0);if(a!==k){var d=a.nodeName?a.nodeName.toLowerCase():"";return b!==k||"td"==d||"th"==d?c.cell(a,b).data():c.row(a).data()||null}return c.data().toArray()};this.fnGetNodes=function(a){var b=this.api(!0);return a!==k?b.row(a).node():b.rows().nodes().flatten().toArray()};this.fnGetPosition=function(a){var b=this.api(!0),c=a.nodeName.toUpperCase();return"TR"==c?b.row(a).index():"TD"==c||"TH"==c?(a=b.cell(a).index(),
+[a.row,a.columnVisible,a.column]):null};this.fnIsOpen=function(a){return this.api(!0).row(a).child.isShown()};this.fnOpen=function(a,b,c){return this.api(!0).row(a).child(b,c).show().child()[0]};this.fnPageChange=function(a,b){var c=this.api(!0).page(a);(b===k||b)&&c.draw(!1)};this.fnSetColumnVis=function(a,b,c){a=this.api(!0).column(a).visible(b);(c===k||c)&&a.columns.adjust().draw()};this.fnSettings=function(){return xa(this[v.iApiIndex])};this.fnSort=function(a){this.api(!0).order(a).draw()};this.fnSortListener=
+function(a,b,c){this.api(!0).order.listener(a,b,c)};this.fnUpdate=function(a,b,c,d,e){var h=this.api(!0);c===k||null===c?h.row(b).data(a):h.cell(b,c).data(a);(e===k||e)&&h.columns.adjust();(d===k||d)&&h.draw();return 0};this.fnVersionCheck=v.fnVersionCheck;var b=this,c=a===k,d=this.length;c&&(a={});this.oApi=this.internal=v.internal;for(var e in m.ext.internal)e&&(this[e]=Nb(e));this.each(function(){var e={},e=1<d?Lb(e,a,!0):a,g=0,j,i=this.getAttribute("id"),n=!1,l=m.defaults,q=h(this);if("table"!=
+this.nodeName.toLowerCase())L(null,0,"Non-table node initialisation ("+this.nodeName+")",2);else{eb(l);fb(l.column);K(l,l,!0);K(l.column,l.column,!0);K(l,h.extend(e,q.data()));var t=m.settings,g=0;for(j=t.length;g<j;g++){var p=t[g];if(p.nTable==this||p.nTHead.parentNode==this||p.nTFoot&&p.nTFoot.parentNode==this){g=e.bRetrieve!==k?e.bRetrieve:l.bRetrieve;if(c||g)return p.oInstance;if(e.bDestroy!==k?e.bDestroy:l.bDestroy){p.oInstance.fnDestroy();break}else{L(p,0,"Cannot reinitialise DataTable",3);
+return}}if(p.sTableId==this.id){t.splice(g,1);break}}if(null===i||""===i)this.id=i="DataTables_Table_"+m.ext._unique++;var o=h.extend(!0,{},m.models.oSettings,{sDestroyWidth:q[0].style.width,sInstance:i,sTableId:i});o.nTable=this;o.oApi=b.internal;o.oInit=e;t.push(o);o.oInstance=1===b.length?b:q.dataTable();eb(e);e.oLanguage&&Da(e.oLanguage);e.aLengthMenu&&!e.iDisplayLength&&(e.iDisplayLength=h.isArray(e.aLengthMenu[0])?e.aLengthMenu[0][0]:e.aLengthMenu[0]);e=Lb(h.extend(!0,{},l),e);E(o.oFeatures,
+e,"bPaginate bLengthChange bFilter bSort bSortMulti bInfo bProcessing bAutoWidth bSortClasses bServerSide bDeferRender".split(" "));E(o,e,["asStripeClasses","ajax","fnServerData","fnFormatNumber","sServerMethod","aaSorting","aaSortingFixed","aLengthMenu","sPaginationType","sAjaxSource","sAjaxDataProp","iStateDuration","sDom","bSortCellsTop","iTabIndex","fnStateLoadCallback","fnStateSaveCallback","renderer","searchDelay","rowId",["iCookieDuration","iStateDuration"],["oSearch","oPreviousSearch"],["aoSearchCols",
+"aoPreSearchCols"],["iDisplayLength","_iDisplayLength"],["bJQueryUI","bJUI"]]);E(o.oScroll,e,[["sScrollX","sX"],["sScrollXInner","sXInner"],["sScrollY","sY"],["bScrollCollapse","bCollapse"]]);E(o.oLanguage,e,"fnInfoCallback");z(o,"aoDrawCallback",e.fnDrawCallback,"user");z(o,"aoServerParams",e.fnServerParams,"user");z(o,"aoStateSaveParams",e.fnStateSaveParams,"user");z(o,"aoStateLoadParams",e.fnStateLoadParams,"user");z(o,"aoStateLoaded",e.fnStateLoaded,"user");z(o,"aoRowCallback",e.fnRowCallback,
+"user");z(o,"aoRowCreatedCallback",e.fnCreatedRow,"user");z(o,"aoHeaderCallback",e.fnHeaderCallback,"user");z(o,"aoFooterCallback",e.fnFooterCallback,"user");z(o,"aoInitComplete",e.fnInitComplete,"user");z(o,"aoPreDrawCallback",e.fnPreDrawCallback,"user");o.rowIdFn=Q(e.rowId);gb(o);i=o.oClasses;e.bJQueryUI?(h.extend(i,m.ext.oJUIClasses,e.oClasses),e.sDom===l.sDom&&"lfrtip"===l.sDom&&(o.sDom='<"H"lfr>t<"F"ip>'),o.renderer)?h.isPlainObject(o.renderer)&&!o.renderer.header&&(o.renderer.header="jqueryui"):
+o.renderer="jqueryui":h.extend(i,m.ext.classes,e.oClasses);q.addClass(i.sTable);o.iInitDisplayStart===k&&(o.iInitDisplayStart=e.iDisplayStart,o._iDisplayStart=e.iDisplayStart);null!==e.iDeferLoading&&(o.bDeferLoading=!0,g=h.isArray(e.iDeferLoading),o._iRecordsDisplay=g?e.iDeferLoading[0]:e.iDeferLoading,o._iRecordsTotal=g?e.iDeferLoading[1]:e.iDeferLoading);var r=o.oLanguage;h.extend(!0,r,e.oLanguage);""!==r.sUrl&&(h.ajax({dataType:"json",url:r.sUrl,success:function(a){Da(a);K(l.oLanguage,a);h.extend(true,
+r,a);ga(o)},error:function(){ga(o)}}),n=!0);null===e.asStripeClasses&&(o.asStripeClasses=[i.sStripeOdd,i.sStripeEven]);var g=o.asStripeClasses,v=q.children("tbody").find("tr").eq(0);-1!==h.inArray(!0,h.map(g,function(a){return v.hasClass(a)}))&&(h("tbody tr",this).removeClass(g.join(" ")),o.asDestroyStripes=g.slice());t=[];g=this.getElementsByTagName("thead");0!==g.length&&(da(o.aoHeader,g[0]),t=qa(o));if(null===e.aoColumns){p=[];g=0;for(j=t.length;g<j;g++)p.push(null)}else p=e.aoColumns;g=0;for(j=
+p.length;g<j;g++)Ea(o,t?t[g]:null);ib(o,e.aoColumnDefs,p,function(a,b){ja(o,a,b)});if(v.length){var s=function(a,b){return a.getAttribute("data-"+b)!==null?b:null};h(v[0]).children("th, td").each(function(a,b){var c=o.aoColumns[a];if(c.mData===a){var d=s(b,"sort")||s(b,"order"),e=s(b,"filter")||s(b,"search");if(d!==null||e!==null){c.mData={_:a+".display",sort:d!==null?a+".@data-"+d:k,type:d!==null?a+".@data-"+d:k,filter:e!==null?a+".@data-"+e:k};ja(o,a)}}})}var w=o.oFeatures;e.bStateSave&&(w.bStateSave=
+!0,Kb(o,e),z(o,"aoDrawCallback",wa,"state_save"));if(e.aaSorting===k){t=o.aaSorting;g=0;for(j=t.length;g<j;g++)t[g][1]=o.aoColumns[g].asSorting[0]}va(o);w.bSort&&z(o,"aoDrawCallback",function(){if(o.bSorted){var a=V(o),b={};h.each(a,function(a,c){b[c.src]=c.dir});u(o,null,"order",[o,a,b]);Jb(o)}});z(o,"aoDrawCallback",function(){(o.bSorted||y(o)==="ssp"||w.bDeferRender)&&va(o)},"sc");g=q.children("caption").each(function(){this._captionSide=q.css("caption-side")});j=q.children("thead");0===j.length&&
+(j=h("<thead/>").appendTo(this));o.nTHead=j[0];j=q.children("tbody");0===j.length&&(j=h("<tbody/>").appendTo(this));o.nTBody=j[0];j=q.children("tfoot");if(0===j.length&&0<g.length&&(""!==o.oScroll.sX||""!==o.oScroll.sY))j=h("<tfoot/>").appendTo(this);0===j.length||0===j.children().length?q.addClass(i.sNoFooter):0<j.length&&(o.nTFoot=j[0],da(o.aoFooter,o.nTFoot));if(e.aaData)for(g=0;g<e.aaData.length;g++)N(o,e.aaData[g]);else(o.bDeferLoading||"dom"==y(o))&&ma(o,h(o.nTBody).children("tr"));o.aiDisplay=
+o.aiDisplayMaster.slice();o.bInitialised=!0;!1===n&&ga(o)}});b=null;return this},v,r,p,s,Ya={},Ob=/[\r\n]/g,Aa=/<.*?>/g,ac=/^[\w\+\-]/,bc=/[\w\+\-]$/,cc=RegExp("(\\/|\\.|\\*|\\+|\\?|\\||\\(|\\)|\\[|\\]|\\{|\\}|\\\\|\\$|\\^|\\-)","g"),Xa=/[',$£€¥%\u2009\u202F\u20BD\u20a9\u20BArfk]/gi,M=function(a){return!a||!0===a||"-"===a?!0:!1},Pb=function(a){var b=parseInt(a,10);return!isNaN(b)&&isFinite(a)?b:null},Qb=function(a,b){Ya[b]||(Ya[b]=RegExp(Qa(b),"g"));return"string"===typeof a&&"."!==b?a.replace(/\./g,
+"").replace(Ya[b],"."):a},Za=function(a,b,c){var d="string"===typeof a;if(M(a))return!0;b&&d&&(a=Qb(a,b));c&&d&&(a=a.replace(Xa,""));return!isNaN(parseFloat(a))&&isFinite(a)},Rb=function(a,b,c){return M(a)?!0:!(M(a)||"string"===typeof a)?null:Za(a.replace(Aa,""),b,c)?!0:null},G=function(a,b,c){var d=[],e=0,f=a.length;if(c!==k)for(;e<f;e++)a[e]&&a[e][b]&&d.push(a[e][b][c]);else for(;e<f;e++)a[e]&&d.push(a[e][b]);return d},ha=function(a,b,c,d){var e=[],f=0,g=b.length;if(d!==k)for(;f<g;f++)a[b[f]][c]&&
+e.push(a[b[f]][c][d]);else for(;f<g;f++)e.push(a[b[f]][c]);return e},W=function(a,b){var c=[],d;b===k?(b=0,d=a):(d=b,b=a);for(var e=b;e<d;e++)c.push(e);return c},Sb=function(a){for(var b=[],c=0,d=a.length;c<d;c++)a[c]&&b.push(a[c]);return b},pa=function(a){var b=[],c,d,e=a.length,f,g=0;d=0;a:for(;d<e;d++){c=a[d];for(f=0;f<g;f++)if(b[f]===c)continue a;b.push(c);g++}return b};m.util={throttle:function(a,b){var c=b!==k?b:200,d,e;return function(){var b=this,g=+new Date,h=arguments;d&&g<d+c?(clearTimeout(e),
+e=setTimeout(function(){d=k;a.apply(b,h)},c)):(d=g,a.apply(b,h))}},escapeRegex:function(a){return a.replace(cc,"\\$1")}};var A=function(a,b,c){a[b]!==k&&(a[c]=a[b])},ba=/\[.*?\]$/,U=/\(\)$/,Qa=m.util.escapeRegex,ua=h("<div>")[0],Zb=ua.textContent!==k,$b=/<.*?>/g,Oa=m.util.throttle,Tb=[],w=Array.prototype,dc=function(a){var b,c,d=m.settings,e=h.map(d,function(a){return a.nTable});if(a){if(a.nTable&&a.oApi)return[a];if(a.nodeName&&"table"===a.nodeName.toLowerCase())return b=h.inArray(a,e),-1!==b?[d[b]]:
+null;if(a&&"function"===typeof a.settings)return a.settings().toArray();"string"===typeof a?c=h(a):a instanceof h&&(c=a)}else return[];if(c)return c.map(function(){b=h.inArray(this,e);return-1!==b?d[b]:null}).toArray()};r=function(a,b){if(!(this instanceof r))return new r(a,b);var c=[],d=function(a){(a=dc(a))&&(c=c.concat(a))};if(h.isArray(a))for(var e=0,f=a.length;e<f;e++)d(a[e]);else d(a);this.context=pa(c);b&&h.merge(this,b);this.selector={rows:null,cols:null,opts:null};r.extend(this,this,Tb)};
+m.Api=r;h.extend(r.prototype,{any:function(){return 0!==this.count()},concat:w.concat,context:[],count:function(){return this.flatten().length},each:function(a){for(var b=0,c=this.length;b<c;b++)a.call(this,this[b],b,this);return this},eq:function(a){var b=this.context;return b.length>a?new r(b[a],this[a]):null},filter:function(a){var b=[];if(w.filter)b=w.filter.call(this,a,this);else for(var c=0,d=this.length;c<d;c++)a.call(this,this[c],c,this)&&b.push(this[c]);return new r(this.context,b)},flatten:function(){var a=
+[];return new r(this.context,a.concat.apply(a,this.toArray()))},join:w.join,indexOf:w.indexOf||function(a,b){for(var c=b||0,d=this.length;c<d;c++)if(this[c]===a)return c;return-1},iterator:function(a,b,c,d){var e=[],f,g,h,i,n,l=this.context,m,t,p=this.selector;"string"===typeof a&&(d=c,c=b,b=a,a=!1);g=0;for(h=l.length;g<h;g++){var o=new r(l[g]);if("table"===b)f=c.call(o,l[g],g),f!==k&&e.push(f);else if("columns"===b||"rows"===b)f=c.call(o,l[g],this[g],g),f!==k&&e.push(f);else if("column"===b||"column-rows"===
+b||"row"===b||"cell"===b){t=this[g];"column-rows"===b&&(m=Ba(l[g],p.opts));i=0;for(n=t.length;i<n;i++)f=t[i],f="cell"===b?c.call(o,l[g],f.row,f.column,g,i):c.call(o,l[g],f,g,i,m),f!==k&&e.push(f)}}return e.length||d?(a=new r(l,a?e.concat.apply([],e):e),b=a.selector,b.rows=p.rows,b.cols=p.cols,b.opts=p.opts,a):this},lastIndexOf:w.lastIndexOf||function(a,b){return this.indexOf.apply(this.toArray.reverse(),arguments)},length:0,map:function(a){var b=[];if(w.map)b=w.map.call(this,a,this);else for(var c=
+0,d=this.length;c<d;c++)b.push(a.call(this,this[c],c));return new r(this.context,b)},pluck:function(a){return this.map(function(b){return b[a]})},pop:w.pop,push:w.push,reduce:w.reduce||function(a,b){return hb(this,a,b,0,this.length,1)},reduceRight:w.reduceRight||function(a,b){return hb(this,a,b,this.length-1,-1,-1)},reverse:w.reverse,selector:null,shift:w.shift,sort:w.sort,splice:w.splice,toArray:function(){return w.slice.call(this)},to$:function(){return h(this)},toJQuery:function(){return h(this)},
+unique:function(){return new r(this.context,pa(this))},unshift:w.unshift});r.extend=function(a,b,c){if(c.length&&b&&(b instanceof r||b.__dt_wrapper)){var d,e,f,g=function(a,b,c){return function(){var d=b.apply(a,arguments);r.extend(d,d,c.methodExt);return d}};d=0;for(e=c.length;d<e;d++)f=c[d],b[f.name]="function"===typeof f.val?g(a,f.val,f):h.isPlainObject(f.val)?{}:f.val,b[f.name].__dt_wrapper=!0,r.extend(a,b[f.name],f.propExt)}};r.register=p=function(a,b){if(h.isArray(a))for(var c=0,d=a.length;c<
+d;c++)r.register(a[c],b);else for(var e=a.split("."),f=Tb,g,j,c=0,d=e.length;c<d;c++){g=(j=-1!==e[c].indexOf("()"))?e[c].replace("()",""):e[c];var i;a:{i=0;for(var n=f.length;i<n;i++)if(f[i].name===g){i=f[i];break a}i=null}i||(i={name:g,val:{},methodExt:[],propExt:[]},f.push(i));c===d-1?i.val=b:f=j?i.methodExt:i.propExt}};r.registerPlural=s=function(a,b,c){r.register(a,c);r.register(b,function(){var a=c.apply(this,arguments);return a===this?this:a instanceof r?a.length?h.isArray(a[0])?new r(a.context,
+a[0]):a[0]:k:a})};p("tables()",function(a){var b;if(a){b=r;var c=this.context;if("number"===typeof a)a=[c[a]];else var d=h.map(c,function(a){return a.nTable}),a=h(d).filter(a).map(function(){var a=h.inArray(this,d);return c[a]}).toArray();b=new b(a)}else b=this;return b});p("table()",function(a){var a=this.tables(a),b=a.context;return b.length?new r(b[0]):a});s("tables().nodes()","table().node()",function(){return this.iterator("table",function(a){return a.nTable},1)});s("tables().body()","table().body()",
+function(){return this.iterator("table",function(a){return a.nTBody},1)});s("tables().header()","table().header()",function(){return this.iterator("table",function(a){return a.nTHead},1)});s("tables().footer()","table().footer()",function(){return this.iterator("table",function(a){return a.nTFoot},1)});s("tables().containers()","table().container()",function(){return this.iterator("table",function(a){return a.nTableWrapper},1)});p("draw()",function(a){return this.iterator("table",function(b){"page"===
+a?O(b):("string"===typeof a&&(a="full-hold"===a?!1:!0),T(b,!1===a))})});p("page()",function(a){return a===k?this.page.info().page:this.iterator("table",function(b){Ta(b,a)})});p("page.info()",function(){if(0===this.context.length)return k;var a=this.context[0],b=a._iDisplayStart,c=a.oFeatures.bPaginate?a._iDisplayLength:-1,d=a.fnRecordsDisplay(),e=-1===c;return{page:e?0:Math.floor(b/c),pages:e?1:Math.ceil(d/c),start:b,end:a.fnDisplayEnd(),length:c,recordsTotal:a.fnRecordsTotal(),recordsDisplay:d,
+serverSide:"ssp"===y(a)}});p("page.len()",function(a){return a===k?0!==this.context.length?this.context[0]._iDisplayLength:k:this.iterator("table",function(b){Ra(b,a)})});var Ub=function(a,b,c){if(c){var d=new r(a);d.one("draw",function(){c(d.ajax.json())})}if("ssp"==y(a))T(a,b);else{C(a,!0);var e=a.jqXHR;e&&4!==e.readyState&&e.abort();ra(a,[],function(c){na(a);for(var c=sa(a,c),d=0,e=c.length;d<e;d++)N(a,c[d]);T(a,b);C(a,!1)})}};p("ajax.json()",function(){var a=this.context;if(0<a.length)return a[0].json});
+p("ajax.params()",function(){var a=this.context;if(0<a.length)return a[0].oAjaxData});p("ajax.reload()",function(a,b){return this.iterator("table",function(c){Ub(c,!1===b,a)})});p("ajax.url()",function(a){var b=this.context;if(a===k){if(0===b.length)return k;b=b[0];return b.ajax?h.isPlainObject(b.ajax)?b.ajax.url:b.ajax:b.sAjaxSource}return this.iterator("table",function(b){h.isPlainObject(b.ajax)?b.ajax.url=a:b.ajax=a})});p("ajax.url().load()",function(a,b){return this.iterator("table",function(c){Ub(c,
+!1===b,a)})});var $a=function(a,b,c,d,e){var f=[],g,j,i,n,l,m;i=typeof b;if(!b||"string"===i||"function"===i||b.length===k)b=[b];i=0;for(n=b.length;i<n;i++){j=b[i]&&b[i].split?b[i].split(","):[b[i]];l=0;for(m=j.length;l<m;l++)(g=c("string"===typeof j[l]?h.trim(j[l]):j[l]))&&g.length&&(f=f.concat(g))}a=v.selector[a];if(a.length){i=0;for(n=a.length;i<n;i++)f=a[i](d,e,f)}return pa(f)},ab=function(a){a||(a={});a.filter&&a.search===k&&(a.search=a.filter);return h.extend({search:"none",order:"current",
+page:"all"},a)},bb=function(a){for(var b=0,c=a.length;b<c;b++)if(0<a[b].length)return a[0]=a[b],a[0].length=1,a.length=1,a.context=[a.context[b]],a;a.length=0;return a},Ba=function(a,b){var c,d,e,f=[],g=a.aiDisplay;c=a.aiDisplayMaster;var j=b.search;d=b.order;e=b.page;if("ssp"==y(a))return"removed"===j?[]:W(0,c.length);if("current"==e){c=a._iDisplayStart;for(d=a.fnDisplayEnd();c<d;c++)f.push(g[c])}else if("current"==d||"applied"==d)f="none"==j?c.slice():"applied"==j?g.slice():h.map(c,function(a){return-1===
+h.inArray(a,g)?a:null});else if("index"==d||"original"==d){c=0;for(d=a.aoData.length;c<d;c++)"none"==j?f.push(c):(e=h.inArray(c,g),(-1===e&&"removed"==j||0<=e&&"applied"==j)&&f.push(c))}return f};p("rows()",function(a,b){a===k?a="":h.isPlainObject(a)&&(b=a,a="");var b=ab(b),c=this.iterator("table",function(c){var e=b;return $a("row",a,function(a){var b=Pb(a);if(b!==null&&!e)return[b];var j=Ba(c,e);if(b!==null&&h.inArray(b,j)!==-1)return[b];if(!a)return j;if(typeof a==="function")return h.map(j,function(b){var e=
+c.aoData[b];return a(b,e._aData,e.nTr)?b:null});b=Sb(ha(c.aoData,j,"nTr"));if(a.nodeName){if(a._DT_RowIndex!==k)return[a._DT_RowIndex];if(a._DT_CellIndex)return[a._DT_CellIndex.row];b=h(a).closest("*[data-dt-row]");return b.length?[b.data("dt-row")]:[]}if(typeof a==="string"&&a.charAt(0)==="#"){j=c.aIds[a.replace(/^#/,"")];if(j!==k)return[j.idx]}return h(b).filter(a).map(function(){return this._DT_RowIndex}).toArray()},c,e)},1);c.selector.rows=a;c.selector.opts=b;return c});p("rows().nodes()",function(){return this.iterator("row",
+function(a,b){return a.aoData[b].nTr||k},1)});p("rows().data()",function(){return this.iterator(!0,"rows",function(a,b){return ha(a.aoData,b,"_aData")},1)});s("rows().cache()","row().cache()",function(a){return this.iterator("row",function(b,c){var d=b.aoData[c];return"search"===a?d._aFilterData:d._aSortData},1)});s("rows().invalidate()","row().invalidate()",function(a){return this.iterator("row",function(b,c){ca(b,c,a)})});s("rows().indexes()","row().index()",function(){return this.iterator("row",
+function(a,b){return b},1)});s("rows().ids()","row().id()",function(a){for(var b=[],c=this.context,d=0,e=c.length;d<e;d++)for(var f=0,g=this[d].length;f<g;f++){var h=c[d].rowIdFn(c[d].aoData[this[d][f]]._aData);b.push((!0===a?"#":"")+h)}return new r(c,b)});s("rows().remove()","row().remove()",function(){var a=this;this.iterator("row",function(b,c,d){var e=b.aoData,f=e[c],g,h,i,n,l;e.splice(c,1);g=0;for(h=e.length;g<h;g++)if(i=e[g],l=i.anCells,null!==i.nTr&&(i.nTr._DT_RowIndex=g),null!==l){i=0;for(n=
+l.length;i<n;i++)l[i]._DT_CellIndex.row=g}oa(b.aiDisplayMaster,c);oa(b.aiDisplay,c);oa(a[d],c,!1);Sa(b);c=b.rowIdFn(f._aData);c!==k&&delete b.aIds[c]});this.iterator("table",function(a){for(var c=0,d=a.aoData.length;c<d;c++)a.aoData[c].idx=c});return this});p("rows.add()",function(a){var b=this.iterator("table",function(b){var c,f,g,h=[];f=0;for(g=a.length;f<g;f++)c=a[f],c.nodeName&&"TR"===c.nodeName.toUpperCase()?h.push(ma(b,c)[0]):h.push(N(b,c));return h},1),c=this.rows(-1);c.pop();h.merge(c,b);
+return c});p("row()",function(a,b){return bb(this.rows(a,b))});p("row().data()",function(a){var b=this.context;if(a===k)return b.length&&this.length?b[0].aoData[this[0]]._aData:k;b[0].aoData[this[0]]._aData=a;ca(b[0],this[0],"data");return this});p("row().node()",function(){var a=this.context;return a.length&&this.length?a[0].aoData[this[0]].nTr||null:null});p("row.add()",function(a){a instanceof h&&a.length&&(a=a[0]);var b=this.iterator("table",function(b){return a.nodeName&&"TR"===a.nodeName.toUpperCase()?
+ma(b,a)[0]:N(b,a)});return this.row(b[0])});var cb=function(a,b){var c=a.context;if(c.length&&(c=c[0].aoData[b!==k?b:a[0]])&&c._details)c._details.remove(),c._detailsShow=k,c._details=k},Vb=function(a,b){var c=a.context;if(c.length&&a.length){var d=c[0].aoData[a[0]];if(d._details){(d._detailsShow=b)?d._details.insertAfter(d.nTr):d._details.detach();var e=c[0],f=new r(e),g=e.aoData;f.off("draw.dt.DT_details column-visibility.dt.DT_details destroy.dt.DT_details");0<G(g,"_details").length&&(f.on("draw.dt.DT_details",
+function(a,b){e===b&&f.rows({page:"current"}).eq(0).each(function(a){a=g[a];a._detailsShow&&a._details.insertAfter(a.nTr)})}),f.on("column-visibility.dt.DT_details",function(a,b){if(e===b)for(var c,d=aa(b),f=0,h=g.length;f<h;f++)c=g[f],c._details&&c._details.children("td[colspan]").attr("colspan",d)}),f.on("destroy.dt.DT_details",function(a,b){if(e===b)for(var c=0,d=g.length;c<d;c++)g[c]._details&&cb(f,c)}))}}};p("row().child()",function(a,b){var c=this.context;if(a===k)return c.length&&this.length?
+c[0].aoData[this[0]]._details:k;if(!0===a)this.child.show();else if(!1===a)cb(this);else if(c.length&&this.length){var d=c[0],c=c[0].aoData[this[0]],e=[],f=function(a,b){if(h.isArray(a)||a instanceof h)for(var c=0,k=a.length;c<k;c++)f(a[c],b);else a.nodeName&&"tr"===a.nodeName.toLowerCase()?e.push(a):(c=h("<tr><td/></tr>").addClass(b),h("td",c).addClass(b).html(a)[0].colSpan=aa(d),e.push(c[0]))};f(a,b);c._details&&c._details.remove();c._details=h(e);c._detailsShow&&c._details.insertAfter(c.nTr)}return this});
+p(["row().child.show()","row().child().show()"],function(){Vb(this,!0);return this});p(["row().child.hide()","row().child().hide()"],function(){Vb(this,!1);return this});p(["row().child.remove()","row().child().remove()"],function(){cb(this);return this});p("row().child.isShown()",function(){var a=this.context;return a.length&&this.length?a[0].aoData[this[0]]._detailsShow||!1:!1});var ec=/^(.+):(name|visIdx|visible)$/,Wb=function(a,b,c,d,e){for(var c=[],d=0,f=e.length;d<f;d++)c.push(B(a,e[d],b));
+return c};p("columns()",function(a,b){a===k?a="":h.isPlainObject(a)&&(b=a,a="");var b=ab(b),c=this.iterator("table",function(c){var e=a,f=b,g=c.aoColumns,j=G(g,"sName"),i=G(g,"nTh");return $a("column",e,function(a){var b=Pb(a);if(a==="")return W(g.length);if(b!==null)return[b>=0?b:g.length+b];if(typeof a==="function"){var e=Ba(c,f);return h.map(g,function(b,f){return a(f,Wb(c,f,0,0,e),i[f])?f:null})}var k=typeof a==="string"?a.match(ec):"";if(k)switch(k[2]){case "visIdx":case "visible":b=parseInt(k[1],
+10);if(b<0){var m=h.map(g,function(a,b){return a.bVisible?b:null});return[m[m.length+b]]}return[Z(c,b)];case "name":return h.map(j,function(a,b){return a===k[1]?b:null});default:return[]}if(a.nodeName&&a._DT_CellIndex)return[a._DT_CellIndex.column];b=h(i).filter(a).map(function(){return h.inArray(this,i)}).toArray();if(b.length||!a.nodeName)return b;b=h(a).closest("*[data-dt-column]");return b.length?[b.data("dt-column")]:[]},c,f)},1);c.selector.cols=a;c.selector.opts=b;return c});s("columns().header()",
+"column().header()",function(){return this.iterator("column",function(a,b){return a.aoColumns[b].nTh},1)});s("columns().footer()","column().footer()",function(){return this.iterator("column",function(a,b){return a.aoColumns[b].nTf},1)});s("columns().data()","column().data()",function(){return this.iterator("column-rows",Wb,1)});s("columns().dataSrc()","column().dataSrc()",function(){return this.iterator("column",function(a,b){return a.aoColumns[b].mData},1)});s("columns().cache()","column().cache()",
+function(a){return this.iterator("column-rows",function(b,c,d,e,f){return ha(b.aoData,f,"search"===a?"_aFilterData":"_aSortData",c)},1)});s("columns().nodes()","column().nodes()",function(){return this.iterator("column-rows",function(a,b,c,d,e){return ha(a.aoData,e,"anCells",b)},1)});s("columns().visible()","column().visible()",function(a,b){var c=this.iterator("column",function(b,c){if(a===k)return b.aoColumns[c].bVisible;var f=b.aoColumns,g=f[c],j=b.aoData,i,n,l;if(a!==k&&g.bVisible!==a){if(a){var m=
+h.inArray(!0,G(f,"bVisible"),c+1);i=0;for(n=j.length;i<n;i++)l=j[i].nTr,f=j[i].anCells,l&&l.insertBefore(f[c],f[m]||null)}else h(G(b.aoData,"anCells",c)).detach();g.bVisible=a;ea(b,b.aoHeader);ea(b,b.aoFooter);wa(b)}});a!==k&&(this.iterator("column",function(c,e){u(c,null,"column-visibility",[c,e,a,b])}),(b===k||b)&&this.columns.adjust());return c});s("columns().indexes()","column().index()",function(a){return this.iterator("column",function(b,c){return"visible"===a?$(b,c):c},1)});p("columns.adjust()",
+function(){return this.iterator("table",function(a){Y(a)},1)});p("column.index()",function(a,b){if(0!==this.context.length){var c=this.context[0];if("fromVisible"===a||"toData"===a)return Z(c,b);if("fromData"===a||"toVisible"===a)return $(c,b)}});p("column()",function(a,b){return bb(this.columns(a,b))});p("cells()",function(a,b,c){h.isPlainObject(a)&&(a.row===k?(c=a,a=null):(c=b,b=null));h.isPlainObject(b)&&(c=b,b=null);if(null===b||b===k)return this.iterator("table",function(b){var d=a,e=ab(c),f=
+b.aoData,g=Ba(b,e),j=Sb(ha(f,g,"anCells")),i=h([].concat.apply([],j)),l,n=b.aoColumns.length,m,p,r,u,v,s;return $a("cell",d,function(a){var c=typeof a==="function";if(a===null||a===k||c){m=[];p=0;for(r=g.length;p<r;p++){l=g[p];for(u=0;u<n;u++){v={row:l,column:u};if(c){s=f[l];a(v,B(b,l,u),s.anCells?s.anCells[u]:null)&&m.push(v)}else m.push(v)}}return m}if(h.isPlainObject(a))return[a];c=i.filter(a).map(function(a,b){return{row:b._DT_CellIndex.row,column:b._DT_CellIndex.column}}).toArray();if(c.length||
+!a.nodeName)return c;s=h(a).closest("*[data-dt-row]");return s.length?[{row:s.data("dt-row"),column:s.data("dt-column")}]:[]},b,e)});var d=this.columns(b,c),e=this.rows(a,c),f,g,j,i,n,l=this.iterator("table",function(a,b){f=[];g=0;for(j=e[b].length;g<j;g++){i=0;for(n=d[b].length;i<n;i++)f.push({row:e[b][g],column:d[b][i]})}return f},1);h.extend(l.selector,{cols:b,rows:a,opts:c});return l});s("cells().nodes()","cell().node()",function(){return this.iterator("cell",function(a,b,c){return(a=a.aoData[b])&&
+a.anCells?a.anCells[c]:k},1)});p("cells().data()",function(){return this.iterator("cell",function(a,b,c){return B(a,b,c)},1)});s("cells().cache()","cell().cache()",function(a){a="search"===a?"_aFilterData":"_aSortData";return this.iterator("cell",function(b,c,d){return b.aoData[c][a][d]},1)});s("cells().render()","cell().render()",function(a){return this.iterator("cell",function(b,c,d){return B(b,c,d,a)},1)});s("cells().indexes()","cell().index()",function(){return this.iterator("cell",function(a,
+b,c){return{row:b,column:c,columnVisible:$(a,c)}},1)});s("cells().invalidate()","cell().invalidate()",function(a){return this.iterator("cell",function(b,c,d){ca(b,c,a,d)})});p("cell()",function(a,b,c){return bb(this.cells(a,b,c))});p("cell().data()",function(a){var b=this.context,c=this[0];if(a===k)return b.length&&c.length?B(b[0],c[0].row,c[0].column):k;jb(b[0],c[0].row,c[0].column,a);ca(b[0],c[0].row,"data",c[0].column);return this});p("order()",function(a,b){var c=this.context;if(a===k)return 0!==
+c.length?c[0].aaSorting:k;"number"===typeof a?a=[[a,b]]:a.length&&!h.isArray(a[0])&&(a=Array.prototype.slice.call(arguments));return this.iterator("table",function(b){b.aaSorting=a.slice()})});p("order.listener()",function(a,b,c){return this.iterator("table",function(d){Ma(d,a,b,c)})});p("order.fixed()",function(a){if(!a){var b=this.context,b=b.length?b[0].aaSortingFixed:k;return h.isArray(b)?{pre:b}:b}return this.iterator("table",function(b){b.aaSortingFixed=h.extend(!0,{},a)})});p(["columns().order()",
+"column().order()"],function(a){var b=this;return this.iterator("table",function(c,d){var e=[];h.each(b[d],function(b,c){e.push([c,a])});c.aaSorting=e})});p("search()",function(a,b,c,d){var e=this.context;return a===k?0!==e.length?e[0].oPreviousSearch.sSearch:k:this.iterator("table",function(e){e.oFeatures.bFilter&&fa(e,h.extend({},e.oPreviousSearch,{sSearch:a+"",bRegex:null===b?!1:b,bSmart:null===c?!0:c,bCaseInsensitive:null===d?!0:d}),1)})});s("columns().search()","column().search()",function(a,
+b,c,d){return this.iterator("column",function(e,f){var g=e.aoPreSearchCols;if(a===k)return g[f].sSearch;e.oFeatures.bFilter&&(h.extend(g[f],{sSearch:a+"",bRegex:null===b?!1:b,bSmart:null===c?!0:c,bCaseInsensitive:null===d?!0:d}),fa(e,e.oPreviousSearch,1))})});p("state()",function(){return this.context.length?this.context[0].oSavedState:null});p("state.clear()",function(){return this.iterator("table",function(a){a.fnStateSaveCallback.call(a.oInstance,a,{})})});p("state.loaded()",function(){return this.context.length?
+this.context[0].oLoadedState:null});p("state.save()",function(){return this.iterator("table",function(a){wa(a)})});m.versionCheck=m.fnVersionCheck=function(a){for(var b=m.version.split("."),a=a.split("."),c,d,e=0,f=a.length;e<f;e++)if(c=parseInt(b[e],10)||0,d=parseInt(a[e],10)||0,c!==d)return c>d;return!0};m.isDataTable=m.fnIsDataTable=function(a){var b=h(a).get(0),c=!1;h.each(m.settings,function(a,e){var f=e.nScrollHead?h("table",e.nScrollHead)[0]:null,g=e.nScrollFoot?h("table",e.nScrollFoot)[0]:
+null;if(e.nTable===b||f===b||g===b)c=!0});return c};m.tables=m.fnTables=function(a){var b=!1;h.isPlainObject(a)&&(b=a.api,a=a.visible);var c=h.map(m.settings,function(b){if(!a||a&&h(b.nTable).is(":visible"))return b.nTable});return b?new r(c):c};m.camelToHungarian=K;p("$()",function(a,b){var c=this.rows(b).nodes(),c=h(c);return h([].concat(c.filter(a).toArray(),c.find(a).toArray()))});h.each(["on","one","off"],function(a,b){p(b+"()",function(){var a=Array.prototype.slice.call(arguments);a[0].match(/\.dt\b/)||
+(a[0]+=".dt");var d=h(this.tables().nodes());d[b].apply(d,a);return this})});p("clear()",function(){return this.iterator("table",function(a){na(a)})});p("settings()",function(){return new r(this.context,this.context)});p("init()",function(){var a=this.context;return a.length?a[0].oInit:null});p("data()",function(){return this.iterator("table",function(a){return G(a.aoData,"_aData")}).flatten()});p("destroy()",function(a){a=a||!1;return this.iterator("table",function(b){var c=b.nTableWrapper.parentNode,
+d=b.oClasses,e=b.nTable,f=b.nTBody,g=b.nTHead,j=b.nTFoot,i=h(e),f=h(f),k=h(b.nTableWrapper),l=h.map(b.aoData,function(a){return a.nTr}),p;b.bDestroying=!0;u(b,"aoDestroyCallback","destroy",[b]);a||(new r(b)).columns().visible(!0);k.unbind(".DT").find(":not(tbody *)").unbind(".DT");h(D).unbind(".DT-"+b.sInstance);e!=g.parentNode&&(i.children("thead").detach(),i.append(g));j&&e!=j.parentNode&&(i.children("tfoot").detach(),i.append(j));b.aaSorting=[];b.aaSortingFixed=[];va(b);h(l).removeClass(b.asStripeClasses.join(" "));
+h("th, td",g).removeClass(d.sSortable+" "+d.sSortableAsc+" "+d.sSortableDesc+" "+d.sSortableNone);b.bJUI&&(h("th span."+d.sSortIcon+", td span."+d.sSortIcon,g).detach(),h("th, td",g).each(function(){var a=h("div."+d.sSortJUIWrapper,this);h(this).append(a.contents());a.detach()}));f.children().detach();f.append(l);g=a?"remove":"detach";i[g]();k[g]();!a&&c&&(c.insertBefore(e,b.nTableReinsertBefore),i.css("width",b.sDestroyWidth).removeClass(d.sTable),(p=b.asDestroyStripes.length)&&f.children().each(function(a){h(this).addClass(b.asDestroyStripes[a%
+p])}));c=h.inArray(b,m.settings);-1!==c&&m.settings.splice(c,1)})});h.each(["column","row","cell"],function(a,b){p(b+"s().every()",function(a){var d=this.selector.opts,e=this;return this.iterator(b,function(f,g,h,i,n){a.call(e[b](g,"cell"===b?h:d,"cell"===b?d:k),g,h,i,n)})})});p("i18n()",function(a,b,c){var d=this.context[0],a=Q(a)(d.oLanguage);a===k&&(a=b);c!==k&&h.isPlainObject(a)&&(a=a[c]!==k?a[c]:a._);return a.replace("%d",c)});m.version="1.10.12";m.settings=[];m.models={};m.models.oSearch={bCaseInsensitive:!0,
+sSearch:"",bRegex:!1,bSmart:!0};m.models.oRow={nTr:null,anCells:null,_aData:[],_aSortData:null,_aFilterData:null,_sFilterRow:null,_sRowStripe:"",src:null,idx:-1};m.models.oColumn={idx:null,aDataSort:null,asSorting:null,bSearchable:null,bSortable:null,bVisible:null,_sManualType:null,_bAttrSrc:!1,fnCreatedCell:null,fnGetData:null,fnSetData:null,mData:null,mRender:null,nTh:null,nTf:null,sClass:null,sContentPadding:null,sDefaultContent:null,sName:null,sSortDataType:"std",sSortingClass:null,sSortingClassJUI:null,
+sTitle:null,sType:null,sWidth:null,sWidthOrig:null};m.defaults={aaData:null,aaSorting:[[0,"asc"]],aaSortingFixed:[],ajax:null,aLengthMenu:[10,25,50,100],aoColumns:null,aoColumnDefs:null,aoSearchCols:[],asStripeClasses:null,bAutoWidth:!0,bDeferRender:!1,bDestroy:!1,bFilter:!0,bInfo:!0,bJQueryUI:!1,bLengthChange:!0,bPaginate:!0,bProcessing:!1,bRetrieve:!1,bScrollCollapse:!1,bServerSide:!1,bSort:!0,bSortMulti:!0,bSortCellsTop:!1,bSortClasses:!0,bStateSave:!1,fnCreatedRow:null,fnDrawCallback:null,fnFooterCallback:null,
+fnFormatNumber:function(a){return a.toString().replace(/\B(?=(\d{3})+(?!\d))/g,this.oLanguage.sThousands)},fnHeaderCallback:null,fnInfoCallback:null,fnInitComplete:null,fnPreDrawCallback:null,fnRowCallback:null,fnServerData:null,fnServerParams:null,fnStateLoadCallback:function(a){try{return JSON.parse((-1===a.iStateDuration?sessionStorage:localStorage).getItem("DataTables_"+a.sInstance+"_"+location.pathname))}catch(b){}},fnStateLoadParams:null,fnStateLoaded:null,fnStateSaveCallback:function(a,b){try{(-1===
+a.iStateDuration?sessionStorage:localStorage).setItem("DataTables_"+a.sInstance+"_"+location.pathname,JSON.stringify(b))}catch(c){}},fnStateSaveParams:null,iStateDuration:7200,iDeferLoading:null,iDisplayLength:10,iDisplayStart:0,iTabIndex:0,oClasses:{},oLanguage:{oAria:{sSortAscending:": activate to sort column ascending",sSortDescending:": activate to sort column descending"},oPaginate:{sFirst:"First",sLast:"Last",sNext:"Next",sPrevious:"Previous"},sEmptyTable:"No data available in table",sInfo:"Showing _START_ to _END_ of _TOTAL_ entries",
+sInfoEmpty:"Showing 0 to 0 of 0 entries",sInfoFiltered:"(filtered from _MAX_ total entries)",sInfoPostFix:"",sDecimal:"",sThousands:",",sLengthMenu:"Show _MENU_ entries",sLoadingRecords:"Loading...",sProcessing:"Processing...",sSearch:"Search Metadata:",sSearchPlaceholder:"",sUrl:"",sZeroRecords:"No matching records found"},oSearch:h.extend({},m.models.oSearch),sAjaxDataProp:"data",sAjaxSource:null,sDom:"lfrtip",searchDelay:null,sPaginationType:"simple_numbers",sScrollX:"",sScrollXInner:"",sScrollY:"",sServerMethod:"GET",
+renderer:null,rowId:"DT_RowId"};X(m.defaults);m.defaults.column={aDataSort:null,iDataSort:-1,asSorting:["asc","desc"],bSearchable:!0,bSortable:!0,bVisible:!0,fnCreatedCell:null,mData:null,mRender:null,sCellType:"td",sClass:"",sContentPadding:"",sDefaultContent:null,sName:"",sSortDataType:"std",sTitle:null,sType:null,sWidth:null};X(m.defaults.column);m.models.oSettings={oFeatures:{bAutoWidth:null,bDeferRender:null,bFilter:null,bInfo:null,bLengthChange:null,bPaginate:null,bProcessing:null,bServerSide:null,
+bSort:null,bSortMulti:null,bSortClasses:null,bStateSave:null},oScroll:{bCollapse:null,iBarWidth:0,sX:null,sXInner:null,sY:null},oLanguage:{fnInfoCallback:null},oBrowser:{bScrollOversize:!1,bScrollbarLeft:!1,bBounding:!1,barWidth:0},ajax:null,aanFeatures:[],aoData:[],aiDisplay:[],aiDisplayMaster:[],aIds:{},aoColumns:[],aoHeader:[],aoFooter:[],oPreviousSearch:{},aoPreSearchCols:[],aaSorting:null,aaSortingFixed:[],asStripeClasses:null,asDestroyStripes:[],sDestroyWidth:0,aoRowCallback:[],aoHeaderCallback:[],
+aoFooterCallback:[],aoDrawCallback:[],aoRowCreatedCallback:[],aoPreDrawCallback:[],aoInitComplete:[],aoStateSaveParams:[],aoStateLoadParams:[],aoStateLoaded:[],sTableId:"",nTable:null,nTHead:null,nTFoot:null,nTBody:null,nTableWrapper:null,bDeferLoading:!1,bInitialised:!1,aoOpenRows:[],sDom:null,searchDelay:null,sPaginationType:"two_button",iStateDuration:0,aoStateSave:[],aoStateLoad:[],oSavedState:null,oLoadedState:null,sAjaxSource:null,sAjaxDataProp:null,bAjaxDataGet:!0,jqXHR:null,json:k,oAjaxData:k,
+fnServerData:null,aoServerParams:[],sServerMethod:null,fnFormatNumber:null,aLengthMenu:null,iDraw:0,bDrawing:!1,iDrawError:-1,_iDisplayLength:10,_iDisplayStart:0,_iRecordsTotal:0,_iRecordsDisplay:0,bJUI:null,oClasses:{},bFiltered:!1,bSorted:!1,bSortCellsTop:null,oInit:null,aoDestroyCallback:[],fnRecordsTotal:function(){return"ssp"==y(this)?1*this._iRecordsTotal:this.aiDisplayMaster.length},fnRecordsDisplay:function(){return"ssp"==y(this)?1*this._iRecordsDisplay:this.aiDisplay.length},fnDisplayEnd:function(){var a=
+this._iDisplayLength,b=this._iDisplayStart,c=b+a,d=this.aiDisplay.length,e=this.oFeatures,f=e.bPaginate;return e.bServerSide?!1===f||-1===a?b+d:Math.min(b+a,this._iRecordsDisplay):!f||c>d||-1===a?d:c},oInstance:null,sInstance:null,iTabIndex:0,nScrollHead:null,nScrollFoot:null,aLastSort:[],oPlugins:{},rowIdFn:null,rowId:null};m.ext=v={buttons:{},classes:{},builder:"-source-",errMode:"alert",feature:[],search:[],selector:{cell:[],column:[],row:[]},internal:{},legacy:{ajax:null},pager:{},renderer:{pageButton:{},
+header:{}},order:{},type:{detect:[],search:{},order:{}},_unique:0,fnVersionCheck:m.fnVersionCheck,iApiIndex:0,oJUIClasses:{},sVersion:m.version};h.extend(v,{afnFiltering:v.search,aTypes:v.type.detect,ofnSearch:v.type.search,oSort:v.type.order,afnSortData:v.order,aoFeatures:v.feature,oApi:v.internal,oStdClasses:v.classes,oPagination:v.pager});h.extend(m.ext.classes,{sTable:"dataTable",sNoFooter:"no-footer",sPageButton:"paginate_button",sPageButtonActive:"current",sPageButtonDisabled:"disabled",sStripeOdd:"odd",
+sStripeEven:"even",sRowEmpty:"dataTables_empty",sWrapper:"dataTables_wrapper",sFilter:"dataTables_filter",sInfo:"dataTables_info",sPaging:"dataTables_paginate paging_",sLength:"dataTables_length",sProcessing:"dataTables_processing",sSortAsc:"sorting_asc",sSortDesc:"sorting_desc",sSortable:"sorting",sSortableAsc:"sorting_asc_disabled",sSortableDesc:"sorting_desc_disabled",sSortableNone:"sorting_disabled",sSortColumn:"sorting_",sFilterInput:"",sLengthSelect:"",sScrollWrapper:"dataTables_scroll",sScrollHead:"dataTables_scrollHead",
+sScrollHeadInner:"dataTables_scrollHeadInner",sScrollBody:"dataTables_scrollBody",sScrollFoot:"dataTables_scrollFoot",sScrollFootInner:"dataTables_scrollFootInner",sHeaderTH:"",sFooterTH:"",sSortJUIAsc:"",sSortJUIDesc:"",sSortJUI:"",sSortJUIAscAllowed:"",sSortJUIDescAllowed:"",sSortJUIWrapper:"",sSortIcon:"",sJUIHeader:"",sJUIFooter:""});var Ca="",Ca="",H=Ca+"ui-state-default",ia=Ca+"css_right ui-icon ui-icon-",Xb=Ca+"fg-toolbar ui-toolbar ui-widget-header ui-helper-clearfix";h.extend(m.ext.oJUIClasses,
+m.ext.classes,{sPageButton:"fg-button ui-button "+H,sPageButtonActive:"ui-state-disabled",sPageButtonDisabled:"ui-state-disabled",sPaging:"dataTables_paginate fg-buttonset ui-buttonset fg-buttonset-multi ui-buttonset-multi paging_",sSortAsc:H+" sorting_asc",sSortDesc:H+" sorting_desc",sSortable:H+" sorting",sSortableAsc:H+" sorting_asc_disabled",sSortableDesc:H+" sorting_desc_disabled",sSortableNone:H+" sorting_disabled",sSortJUIAsc:ia+"triangle-1-n",sSortJUIDesc:ia+"triangle-1-s",sSortJUI:ia+"carat-2-n-s",
+sSortJUIAscAllowed:ia+"carat-1-n",sSortJUIDescAllowed:ia+"carat-1-s",sSortJUIWrapper:"DataTables_sort_wrapper",sSortIcon:"DataTables_sort_icon",sScrollHead:"dataTables_scrollHead "+H,sScrollFoot:"dataTables_scrollFoot "+H,sHeaderTH:H,sFooterTH:H,sJUIHeader:Xb+" ui-corner-tl ui-corner-tr",sJUIFooter:Xb+" ui-corner-bl ui-corner-br"});var Mb=m.ext.pager;h.extend(Mb,{simple:function(){return["previous","next"]},full:function(){return["first","previous","next","last"]},numbers:function(a,b){return[ya(a,
+b)]},simple_numbers:function(a,b){return["previous",ya(a,b),"next"]},full_numbers:function(a,b){return["first","previous",ya(a,b),"next","last"]},_numbers:ya,numbers_length:7});h.extend(!0,m.ext.renderer,{pageButton:{_:function(a,b,c,d,e,f){var g=a.oClasses,j=a.oLanguage.oPaginate,i=a.oLanguage.oAria.paginate||{},k,l,m=0,p=function(b,d){var o,r,u,s,v=function(b){Ta(a,b.data.action,true)};o=0;for(r=d.length;o<r;o++){s=d[o];if(h.isArray(s)){u=h("<"+(s.DT_el||"div")+"/>").appendTo(b);p(u,s)}else{k=null;
+l="";switch(s){case "ellipsis":b.append('<span class="ellipsis">&#x2026;</span>');break;case "first":k=j.sFirst;l=s+(e>0?"":" "+g.sPageButtonDisabled);break;case "previous":k=j.sPrevious;l=s+(e>0?"":" "+g.sPageButtonDisabled);break;case "next":k=j.sNext;l=s+(e<f-1?"":" "+g.sPageButtonDisabled);break;case "last":k=j.sLast;l=s+(e<f-1?"":" "+g.sPageButtonDisabled);break;default:k=s+1;l=e===s?g.sPageButtonActive:""}if(k!==null){u=h("<a>",{"class":g.sPageButton+" "+l,"aria-controls":a.sTableId,"aria-label":i[s],
+"data-dt-idx":m,tabindex:a.iTabIndex,id:c===0&&typeof s==="string"?a.sTableId+"_"+s:null}).html(k).appendTo(b);Wa(u,{action:s},v);m++}}}},r;try{r=h(b).find(I.activeElement).data("dt-idx")}catch(o){}p(h(b).empty(),d);r&&h(b).find("[data-dt-idx="+r+"]").focus()}}});h.extend(m.ext.type.detect,[function(a,b){var c=b.oLanguage.sDecimal;return Za(a,c)?"num"+c:null},function(a){if(a&&!(a instanceof Date)&&(!ac.test(a)||!bc.test(a)))return null;var b=Date.parse(a);return null!==b&&!isNaN(b)||M(a)?"date":
+null},function(a,b){var c=b.oLanguage.sDecimal;return Za(a,c,!0)?"num-fmt"+c:null},function(a,b){var c=b.oLanguage.sDecimal;return Rb(a,c)?"html-num"+c:null},function(a,b){var c=b.oLanguage.sDecimal;return Rb(a,c,!0)?"html-num-fmt"+c:null},function(a){return M(a)||"string"===typeof a&&-1!==a.indexOf("<")?"html":null}]);h.extend(m.ext.type.search,{html:function(a){return M(a)?a:"string"===typeof a?a.replace(Ob," ").replace(Aa,""):""},string:function(a){return M(a)?a:"string"===typeof a?a.replace(Ob,
+" "):a}});var za=function(a,b,c,d){if(0!==a&&(!a||"-"===a))return-Infinity;b&&(a=Qb(a,b));a.replace&&(c&&(a=a.replace(c,"")),d&&(a=a.replace(d,"")));return 1*a};h.extend(v.type.order,{"date-pre":function(a){return Date.parse(a)||0},"html-pre":function(a){return M(a)?"":a.replace?a.replace(/<.*?>/g,"").toLowerCase():a+""},"string-pre":function(a){return M(a)?"":"string"===typeof a?a.toLowerCase():!a.toString?"":a.toString()},"string-asc":function(a,b){return a<b?-1:a>b?1:0},"string-desc":function(a,
+b){return a<b?1:a>b?-1:0}});db("");h.extend(!0,m.ext.renderer,{header:{_:function(a,b,c,d){h(a.nTable).on("order.dt.DT",function(e,f,g,h){if(a===f){e=c.idx;b.removeClass(c.sSortingClass+" "+d.sSortAsc+" "+d.sSortDesc).addClass(h[e]=="asc"?d.sSortAsc:h[e]=="desc"?d.sSortDesc:c.sSortingClass)}})},jqueryui:function(a,b,c,d){h("<div/>").addClass(d.sSortJUIWrapper).append(b.contents()).append(h("<span/>").addClass(d.sSortIcon+" "+c.sSortingClassJUI)).appendTo(b);h(a.nTable).on("order.dt.DT",function(e,
+f,g,h){if(a===f){e=c.idx;b.removeClass(d.sSortAsc+" "+d.sSortDesc).addClass(h[e]=="asc"?d.sSortAsc:h[e]=="desc"?d.sSortDesc:c.sSortingClass);b.find("span."+d.sSortIcon).removeClass(d.sSortJUIAsc+" "+d.sSortJUIDesc+" "+d.sSortJUI+" "+d.sSortJUIAscAllowed+" "+d.sSortJUIDescAllowed).addClass(h[e]=="asc"?d.sSortJUIAsc:h[e]=="desc"?d.sSortJUIDesc:c.sSortingClassJUI)}})}}});var Yb=function(a){return"string"===typeof a?a.replace(/</g,"&lt;").replace(/>/g,"&gt;").replace(/"/g,"&quot;"):a};m.render={number:function(a,
+b,c,d,e){return{display:function(f){if("number"!==typeof f&&"string"!==typeof f)return f;var g=0>f?"-":"",h=parseFloat(f);if(isNaN(h))return Yb(f);f=Math.abs(h);h=parseInt(f,10);f=c?b+(f-h).toFixed(c).substring(2):"";return g+(d||"")+h.toString().replace(/\B(?=(\d{3})+(?!\d))/g,a)+f+(e||"")}}},text:function(){return{display:Yb}}};h.extend(m.ext.internal,{_fnExternApiFunc:Nb,_fnBuildAjax:ra,_fnAjaxUpdate:lb,_fnAjaxParameters:ub,_fnAjaxUpdateDraw:vb,_fnAjaxDataSrc:sa,_fnAddColumn:Ea,_fnColumnOptions:ja,
+_fnAdjustColumnSizing:Y,_fnVisibleToColumnIndex:Z,_fnColumnIndexToVisible:$,_fnVisbleColumns:aa,_fnGetColumns:la,_fnColumnTypes:Ga,_fnApplyColumnDefs:ib,_fnHungarianMap:X,_fnCamelToHungarian:K,_fnLanguageCompat:Da,_fnBrowserDetect:gb,_fnAddData:N,_fnAddTr:ma,_fnNodeToDataIndex:function(a,b){return b._DT_RowIndex!==k?b._DT_RowIndex:null},_fnNodeToColumnIndex:function(a,b,c){return h.inArray(c,a.aoData[b].anCells)},_fnGetCellData:B,_fnSetCellData:jb,_fnSplitObjNotation:Ja,_fnGetObjectDataFn:Q,_fnSetObjectDataFn:R,
+_fnGetDataMaster:Ka,_fnClearTable:na,_fnDeleteIndex:oa,_fnInvalidate:ca,_fnGetRowElements:Ia,_fnCreateTr:Ha,_fnBuildHead:kb,_fnDrawHead:ea,_fnDraw:O,_fnReDraw:T,_fnAddOptionsHtml:nb,_fnDetectHeader:da,_fnGetUniqueThs:qa,_fnFeatureHtmlFilter:pb,_fnFilterComplete:fa,_fnFilterCustom:yb,_fnFilterColumn:xb,_fnFilter:wb,_fnFilterCreateSearch:Pa,_fnEscapeRegex:Qa,_fnFilterData:zb,_fnFeatureHtmlInfo:sb,_fnUpdateInfo:Cb,_fnInfoMacros:Db,_fnInitialise:ga,_fnInitComplete:ta,_fnLengthChange:Ra,_fnFeatureHtmlLength:ob,
+_fnFeatureHtmlPaginate:tb,_fnPageChange:Ta,_fnFeatureHtmlProcessing:qb,_fnProcessingDisplay:C,_fnFeatureHtmlTable:rb,_fnScrollDraw:ka,_fnApplyToChildren:J,_fnCalculateColumnWidths:Fa,_fnThrottle:Oa,_fnConvertToWidth:Fb,_fnGetWidestNode:Gb,_fnGetMaxLenString:Hb,_fnStringToCss:x,_fnSortFlatten:V,_fnSort:mb,_fnSortAria:Jb,_fnSortListener:Va,_fnSortAttachListener:Ma,_fnSortingClasses:va,_fnSortData:Ib,_fnSaveState:wa,_fnLoadState:Kb,_fnSettingsFromNode:xa,_fnLog:L,_fnMap:E,_fnBindAction:Wa,_fnCallbackReg:z,
+_fnCallbackFire:u,_fnLengthOverflow:Sa,_fnRenderer:Na,_fnDataSource:y,_fnRowAttributes:La,_fnCalculateEnd:function(){}});h.fn.dataTable=m;m.$=h;h.fn.dataTableSettings=m.settings;h.fn.dataTableExt=m.ext;h.fn.DataTable=function(a){return h(this).dataTable(a).api()};h.each(m,function(a,b){h.fn.DataTable[a]=b});return h.fn.dataTable});

static/master_list.js

@@ -0,0 +1,427 @@
+//////////////////////////////////
+// Centillion Master List
+// Javascript Functions 
+//
+// This file contains javascript functions used by
+// the master_list page of centillion. The master
+// list page uses the search engine as an API to
+// get a list of all documents by type.
+
+
+///////////////////////////////////
+// Process get parameters
+//
+// When the document is loaded, parse the GET params
+// from the URL (everything after the ?).
+//
+// If the "doctype" parameter is in the URL, use it to
+// determine which panel to open automatically.
+
+var initGdocTable = false;
+var initIssuesTable = false;
+var initGhfilesTable = false;
+var initMarkdownTable = false;
+var initEmailthreadsTable = false;
+var initDisqusTable = false;
+
+$(document).ready(function() {
+    var url_string = document.location.toString();
+    var url = new URL(url_string);
+    var d = url.searchParams.get("doctype");
+
+    if (d==='gdoc') {
+        load_gdoc_table();
+        var divList = $('div#collapseDrive').addClass('in');
+
+    } else if (d==='issue') {
+        load_issue_table();
+        var divList = $('div#collapseIssues').addClass('in');
+
+    } else if (d==='ghfile') {
+        load_ghfile_table();
+        var divList = $('div#collapseFiles').addClass('in');
+
+    } else if (d==='markdown') {
+        load_markdown_table();
+        var divList = $('div#collapseMarkdown').addClass('in');
+
+    } else if (d==='emailthread') {
+        load_emailthreads_table();
+        var divList = $('div#collapseThreads').addClass('in');
+
+    } else if (d==='disqus') {
+        load_disqusthreads_table();
+        var divList = $('div#collapseDisqus').addClass('in');
+
+    }
+});
+
+
+//////////////////////////////////
+// utility functions
+
+// https://stackoverflow.com/a/25275808
+function iso8601(date) {
+  var hours = date.getHours();
+  var minutes = date.getMinutes();
+  var ampm = hours >= 12 ? 'PM' : 'AM';
+  hours = hours % 12;
+  hours = hours ? hours : 12; // the hour '0' should be '12'
+  minutes = minutes < 10 ? '0'+minutes : minutes;
+  var strTime = hours + ':' + minutes + ' ' + ampm;
+  return date.getYear() + "-" + (date.getMonth()+1) + "-" + date.getDate() + "  " + strTime;
+}
+
+// https://stackoverflow.com/a/7390612
+var toType = function(obj) {
+  return ({}).toString.call(obj).match(/\s([a-zA-Z]+)/)[1].toLowerCase()
+}
+
+//////////////////////////////////
+// API-to-Table Functions
+//
+// These functions ask centillion for a list of all documents
+// of a given type, and load the results into an HTML table.
+//
+// The dataTable bootstrap plugin is used to make the tables
+// sortable, searchable, and slick.
+//
+// Sections:
+// ----------
+// Google Drive files
+// Github issues
+// Github files
+// Github markdown
+// Groups.io email threads
+
+// ------------------------
+// Google Drive
+
+function load_gdoc_table(){
+    if(!initGdocTable) {
+        var divList = $('div#collapseDrive').attr('class');
+        if (divList.indexOf('in') !== -1) {
+            //console.log('Closing Google Drive master list');
+        } else { 
+            //console.log('Opening Google Drive master list');
+
+            $.getJSON("/list/gdoc", function(result){
+
+                var r = new Array(), j = -1, size=result.length;
+                r[++j] = '<thead>'
+                r[++j] = '<tr class="header-row">';
+                r[++j] = '<th width="40%">File Name</th>';
+                r[++j] = '<th width="15%">Owner</th>';
+                r[++j] = '<th width="15%">Type</th>';
+                r[++j] = '<th width="15%">Created</th>';
+                r[++j] = '<th width="15%">Modified</th>';
+                r[++j] = '</tr>';
+                r[++j] = '</thead>'
+                r[++j] = '<tbody>'
+                for (var i=0; i<size; i++){
+                    r[++j] = '<tr><td>';
+                    r[++j] = '<a href="' + result[i]['url'] + '" target="_blank">'
+                    r[++j] = result[i]['title'];
+                    r[++j] = '</a>'
+                    r[++j] = '</td><td>';
+                    r[++j] = result[i]['owner_name'];
+                    r[++j] = '</td><td>';
+                    r[++j] = result[i]['mimetype'];
+                    r[++j] = '</td><td>';
+                    r[++j] = result[i]['created_time'];
+                    r[++j] = '</td><td>';
+                    r[++j] = result[i]['modified_time'];
+                    r[++j] = '</td></tr>';
+                }
+                r[++j] = '</tbody>'
+
+                // Construct names of id tags
+                var doctype = 'gdocs';
+                var idlabel = '#' + doctype + '-master-list';
+                var filtlabel = idlabel + '_filter';
+
+                // Initialize the DataTable
+                $(idlabel).html(r.join(''));
+                $(idlabel).DataTable({
+                    responsive: true,
+                    lengthMenu: [50,100,250,500]
+                });
+
+                initGdocTable = true
+            });
+            //console.log('Finished loading Google Drive master list');
+        }
+    }
+}
+
+// ------------------------
+// Github issues
+
+function load_issue_table(){
+    if(!initIssuesTable) {
+        var divList = $('div#collapseIssues').attr('class');
+        if (divList.indexOf('in') !== -1) {
+            //console.log('Closing Github issues master list');
+        } else { 
+            //console.log('Opening Github issues master list');
+
+            $.getJSON("/list/issue", function(result){
+                var r = new Array(), j = -1, size=result.length;
+                r[++j] = '<thead>'
+                r[++j] = '<tr class="header-row">';
+                r[++j] = '<th width="50%">Issue Name</th>';
+                r[++j] = '<th width="15%">Repository</th>';
+                r[++j] = '<th width="15%">Created</th>';
+                r[++j] = '<th width="15%">Modified</th>';
+                r[++j] = '</tr>';
+                r[++j] = '</thead>'
+                r[++j] = '<tbody>'
+                for (var i=0; i<size; i++){
+                    r[++j] ='<tr><td>';
+                    r[++j] = '<a href="' + result[i]['url'] + '" target="_blank">'
+                    r[++j] = result[i]['title'];
+                    r[++j] = '</a>'
+                    r[++j] = '</td><td>';
+                    r[++j] = '<a href="' + result[i]['repo_url'] + '" target="_blank">'
+                    r[++j] = result[i]['repo_name'];
+                    r[++j] = '</a>'
+                    r[++j] = '</td><td>';
+                    r[++j] = result[i]['created_time'];
+                    r[++j] = '</td><td>';
+                    r[++j] = result[i]['modified_time'];
+                    r[++j] = '</td></tr>';
+                }
+                r[++j] = '</tbody>'
+
+                // Construct names of id tags
+                var doctype = 'issues';
+                var idlabel = '#' + doctype + '-master-list';
+                var filtlabel = idlabel + '_filter';
+
+                // Initialize the DataTable
+                $(idlabel).html(r.join(''));
+                $(idlabel).DataTable({
+                    responsive: true,
+                    lengthMenu: [50,100,250,500]
+                });
+
+                initIssuesTable = true;
+            });
+            //console.log('Finished loading Github issues master list');
+        }
+    }
+}
+
+// ------------------------
+// Github files
+
+function load_ghfile_table(){
+    if(!initGhfilesTable) {
+        var divList = $('div#collapseFiles').attr('class');
+        if (divList.indexOf('in') !== -1) {
+            //console.log('Closing Github files master list');
+        } else { 
+            //console.log('Opening Github files master list');
+
+            $.getJSON("/list/ghfile", function(result){
+                //console.log("-----------");
+                //console.log(result);
+                var r = new Array(), j = -1, size=result.length;
+                r[++j] = '<thead>'
+                r[++j] = '<tr class="header-row">';
+                r[++j] = '<th width="70%">File Name</th>';
+                r[++j] = '<th width="30%">Repository</th>';
+                r[++j] = '</tr>';
+                r[++j] = '</thead>'
+                r[++j] = '<tbody>'
+                for (var i=0; i<size; i++){
+                    r[++j] ='<tr><td>';
+                    r[++j] = '<a href="' + result[i]['url'] + '" target="_blank">'
+                    r[++j] = result[i]['title'];
+                    r[++j] = '</a>'
+                    r[++j] = '</td><td>';
+                    r[++j] = '<a href="' + result[i]['repo_url'] + '" target="_blank">'
+                    r[++j] = result[i]['repo_name'];
+                    r[++j] = '</a>'
+                    r[++j] = '</td></tr>';
+                }
+                r[++j] = '</tbody>'
+
+                // Construct names of id tags
+                var doctype = 'ghfiles';
+                var idlabel = '#' + doctype + '-master-list';
+                var filtlabel = idlabel + '_filter';
+
+                // Initialize the DataTable
+                $(idlabel).html(r.join(''));
+                $(idlabel).DataTable({
+                    responsive: true,
+                    lengthMenu: [50,100,250,500]
+                });
+
+                initGhfilesTable = true;
+            });
+            //console.log('Finished loading Github file list');
+        }
+    }
+}
+
+// ------------------------
+// Github Markdown
+
+function load_markdown_table(){
+    if(!initMarkdownTable) { 
+        var divList = $('div#collapseMarkdown').attr('class');
+        if (divList.indexOf('in') !== -1) {
+            //console.log('Closing Github markdown master list');
+        } else { 
+            //console.log('Opening Github markdown master list');
+
+            $.getJSON("/list/markdown", function(result){
+                var r = new Array(), j = -1, size=result.length;
+                r[++j] = '<thead>'
+                r[++j] = '<tr class="header-row">';
+                r[++j] = '<th width="70%">Markdown File Name</th>';
+                r[++j] = '<th width="30%">Repository</th>';
+                r[++j] = '</tr>';
+                r[++j] = '</thead>'
+                r[++j] = '<tbody>'
+                for (var i=0; i<size; i++){
+                    r[++j] ='<tr><td>';
+                    r[++j] = '<a href="' + result[i]['url'] + '" target="_blank">'
+                    r[++j] = result[i]['title'];
+                    r[++j] = '</a>'
+                    r[++j] = '</td><td>';
+                    r[++j] = '<a href="' + result[i]['repo_url'] + '" target="_blank">'
+                    r[++j] = result[i]['repo_name'];
+                    r[++j] = '</a>'
+                    r[++j] = '</td></tr>';
+                }
+                r[++j] = '</tbody>'
+
+                // Construct names of id tags
+                var doctype = 'markdown';
+                var idlabel = '#' + doctype + '-master-list';
+                var filtlabel = idlabel + '_filter';
+
+                // Initialize the DataTable
+                $(idlabel).html(r.join(''));
+                $(idlabel).DataTable({
+                    responsive: true,
+                    lengthMenu: [50,100,250,500]
+                });
+
+                initMarkdownTable = true;
+            });
+            //console.log('Finished loading Markdown list');
+        }
+    }
+}
+
+
+// ------------------------
+// Groups.io Email Threads
+
+function load_emailthreads_table(){
+    if(!initEmailthreadsTable) { 
+        var divList = $('div#collapseThreads').attr('class');
+        if (divList.indexOf('in') !== -1) {
+            //console.log('Closing Groups.io email threads master list');
+        } else { 
+            //console.log('Opening Groups.io email threads master list');
+    
+            $.getJSON("/list/emailthread", function(result){
+                var r = new Array(), j = -1, size=result.length;
+                r[++j] = '<thead>'
+                r[++j] = '<tr class="header-row">';
+                r[++j] = '<th width="60%">Topic</th>';
+                r[++j] = '<th width="15%">Started By</th>';
+                r[++j] = '<th width="15%">Date</th>';
+                r[++j] = '<th width="10%">Mailing List</th>';
+                r[++j] = '</tr>';
+                r[++j] = '</thead>'
+                r[++j] = '<tbody>'
+                for (var i=0; i<size; i++){
+                    r[++j] ='<tr><td>';
+                    r[++j] = '<a href="' + result[i]['url'] + '" target="_blank">'
+                    r[++j] = result[i]['title'];
+                    r[++j] = '</a>'
+                    r[++j] = '</td><td>';
+                    r[++j] = result[i]['owner_name'];
+                    r[++j] = '</td><td>';
+                    r[++j] = result[i]['created_time'];
+                    r[++j] = '</td><td>';
+                    r[++j] = result[i]['group'];
+                    r[++j] = '</td></tr>';
+                }
+                r[++j] = '</tbody>'
+
+                // Construct names of id tags
+                var doctype = 'emailthreads';
+                var idlabel = '#' + doctype + '-master-list';
+                var filtlabel = idlabel + '_filter';
+
+                // Initialize the DataTable
+                $(idlabel).html(r.join(''));
+                $(idlabel).DataTable({
+                    responsive: true,
+                    lengthMenu: [50,100,250,500]
+                });
+
+                initEmailthreadsTable = true;
+            });
+            //console.log('Finished loading Groups.io email threads list');
+        }
+    }
+}
+
+// ------------------------
+// Disqus Comment Threads
+
+function load_disqusthreads_table(){
+    if(!initEmailthreadsTable) { 
+        var divList = $('div#collapseDisqus').attr('class');
+        if (divList.indexOf('in') !== -1) {
+            //console.log('Closing Disqus comment threads master list');
+        } else { 
+            //console.log('Opening Disqus comment threads master list');
+    
+            $.getJSON("/list/disqus", function(result){
+                var r = new Array(), j = -1, size=result.length;
+                r[++j] = '<thead>'
+                r[++j] = '<tr class="header-row">';
+                r[++j] = '<th width="70%">Page Title</th>';
+                r[++j] = '<th width="30%">Created</th>';
+                r[++j] = '</tr>';
+                r[++j] = '</thead>'
+                r[++j] = '<tbody>'
+                for (var i=0; i<size; i++){
+                    r[++j] ='<tr><td>';
+                    r[++j] = '<a href="' + result[i]['url'] + '" target="_blank">'
+                    r[++j] = result[i]['title'];
+                    r[++j] = '</a>'
+                    r[++j] = '</td><td>';
+                    r[++j] = result[i]['created_time'];
+                    r[++j] = '</td></tr>';
+                }
+                r[++j] = '</tbody>'
+
+                // Construct names of id tags
+                var doctype = 'disqus';
+                var idlabel = '#' + doctype + '-master-list';
+                var filtlabel = idlabel + '_filter';
+
+                // Initialize the DataTable
+                $(idlabel).html(r.join(''));
+                $(idlabel).DataTable({
+                    responsive: true,
+                    lengthMenu: [50,100,250,500]
+                });
+
+                initDisqusTable = true;
+            });
+            console.log('Finished loading Disqus comment threads list');
+        }
+    }
+}
+

static/search_list.js

@@ -0,0 +1,39 @@
+//////////////////////////////////
+// Centillion Search Results Listing
+// Javascript Functions 
+//
+// This file contains javascript functions used by
+// the search results page centillion.
+
+
+
+//////////////////////////////////
+// Results-to-DataTable Functions
+//
+// These functions post-process the table of search results
+// and make it into a dataTable.
+//
+// The dataTable bootstrap plugin is used to make the tables
+// sortable, searchable, and slick.
+
+
+$(document).ready(function() {
+    // Construct names of id tags
+    table_id = "#search-results";
+
+    // Initialize the DataTable
+    $(table_id).DataTable({
+        responsive: true,
+        searching: false,
+        order: [[0,'desc']],
+        aoColumnDefs: [
+            { bSortable: false,
+              aTargets : [2]
+            }
+        ],
+        lengthMenu: [10,20,50,100]
+    });
+
+    console.log('Finished loading search results list');
+});
+

static/style.css

@@ -1,3 +1,99 @@
+#modal-too-long {
+    visibility: hidden;
+}
+
+/* feedback smileys */
+#modal-feedback-smile-icon,
+#modal-feedback-frown-icon {
+    padding-left: 100px;
+    padding-right: 100px;
+    padding-top: 20px;
+    padding-bottom: 20px;
+}
+
+div.smile-active {
+    background-color: #2b2;
+}
+i.smile-active {
+    color: #fff;
+}
+div.frown-active {
+    background-color: #b22;
+}
+i.frown-active {
+    color: #fff;
+}
+
+/* feedback text area */
+#modal-feedback-textarea {
+    width: 100%;
+}
+
+/* feedback buttons */
+button.close {
+    font-size: 35px;
+}
+button#submit-feedback-btn {
+    width: 250px;
+}
+
+button#feedback:hover {
+    opacity: 1.0;
+    filter: alpha(opacity=100); /* For IE8 and earlier */
+}
+
+button#feedback {
+    opacity: 0.5;
+    filter: alpha(opacity=50); /* For IE8 and earlier */
+
+    width: 180px;
+    height: 50px;
+    position: fixed; 
+    z-index: 999;
+    right: 120px;
+    bottom: 10px;
+}
+
+
+/* search results table */
+td#search-results-score-col,
+td#search-results-type-col {
+    width: 90px;
+}
+
+div.container {
+    width: 90%;
+}
+
+/* control panel button width */
+.btn-reindex-type, .btn-reindex-all {
+    width: 350px;
+}
+
+/* landing page github button */
+#github-button {
+    display:inline-block;
+    font-size: 20px;
+    line-height: 40px;
+    padding: 10px 40px 10px 40px;
+    text-align: center;
+}
+
+/* search button */
+#the-big-one {
+    margin-top: 10px;
+    margin-bottom: 10px;
+}
+
+/* badges for number of docs indexed */
+span.results-count {
+    background-color: #555;
+}
+
+span.indexing-count {
+    background-color: #337ab7;
+}
+
 span.badge {
     vertical-align: text-bottom;
 }
@@ -7,6 +103,10 @@ a.badgelinks, a.badgelinks:hover {
     text-decoration: none;
 }
 
+h2.masterlist-header a {
+    text-decoration: none;
+}
+
 div.list-group {
     border: 1px solid rgba(86,61,124,.2);
 }
@@ -34,7 +134,7 @@ li.search-group-item {
 }
 
 div.url {
-    background-color: rgba(86,61,124,.15);
+    background-color: rgba(40,40,60,.15);
     padding: 8px;
 }
 
@@ -100,7 +200,7 @@ table {
 
 .info, .last-searches {
     color: gray;
-    font-size: 12px;
+    /*font-size: 12px;*/
     font-family: Arial, serif;
 }
 
@@ -110,27 +210,27 @@ table {
 
 div.tags a, td.tag-cloud a {
     color: #b56020;
-    font-size: 12px;
+    /*font-size: 12px;*/
 }
 
 td.tag-cloud, td.directories-cloud {
-    font-size: 12px;
+    /*font-size: 12px;*/
     color: #555555;
 }
 
 td.directories-cloud a {
-    font-size: 12px;
+    /*font-size: 12px;*/
     color: #377BA8;
 }
 
 div.path {
-    font-size: 12px;
+    /*font-size: 12px;*/
     color: #666666;
     margin-bottom: 3px;
 }
 
 div.path a {
-    font-size: 12px;
+    /*font-size: 12px;*/
     margin-right: 5px;
 }
 

templates/403.html

@@ -0,0 +1,26 @@
+{% extends "layout.html" %}
+{% set active_page = "403" %}
+{% block body %}
+
+<div class="container">
+    <div class="row">
+        <p>&nbsp;</p>
+    </div>
+    <div class="row">
+
+        <div class="col-xs-12">
+            <center>
+                <h2>403: Permission Denied
+                </h2>
+                <p>You don't seem to be a member of the proper Github team or organization.
+                Please contact <a href="mailto:dcppc-inbox@gmail.com">dcppc-inbox@gmail.com</a>
+                if you believe you are seeing this message in error.
+                </p>
+            </center>
+        </div>
+
+    </div>
+</div>
+
+{% endblock %}
+

templates/404.html

@@ -0,0 +1,22 @@
+{% extends "layout.html" %}
+{% set active_page = "404" %}
+{% block body %}
+
+<div class="container">
+    <div class="row">
+
+        <div class="col-xs-12">
+            <center>
+                <h2>404: Not Found
+                </h2>
+                <p>You have requested a page that does not exist.
+                Try clicking the centillion logo to go back home.
+                </p>
+            </center>
+        </div>
+
+    </div>
+</div>
+
+{% endblock %}
+

templates/banner.html

@@ -0,0 +1,32 @@
+<div class="container" id="banner-container">
+
+    {#
+    banner image
+    #}
+    <div class="row" id="banner-row">
+        <div class="col12sm" id="banner-col">
+            <center>
+                <a id="banner-a" href="{{ url_for('search')}}?query=&fields=">
+                    {% if 'betasearch' in request.url %}
+                        <img id="banner-img" src="{{ url_for('static', filename='centillion_white_beta.png') }}">
+                    {% elif 'localhost' in request.url %}
+                        <img id="banner-img" src="{{ url_for('static', filename='centillion_white_localhost.png') }}">
+                    {% else %}
+                        <img id="banner-img" src="{{ url_for('static', filename='centillion_white.png') }}">
+                    {% endif %}
+                </a>
+            </center>
+        </div>
+    </div>
+
+    {% if config['TAGLINE'] %}
+    <div class="row" id="tagline-row">
+        <div class="col12sm" id="tagline-col">
+            <center>
+                    <h2 id="tagline-tagline"> {{config['TAGLINE']}} </h2>
+            </center>
+        </div>
+    </div>
+    {% endif %}
+
+</div>

templates/controlpanel.html

@@ -1,52 +1,7 @@
 {% extends "layout.html" %}
+{% set active_page = "control_panel" %}
 {% block body %}
 
-{% with messages = get_flashed_messages() %}
-{% if messages %}
-<div class="container">
-    <div class="alert alert-success alert-dismissible">
-        <a href="#" class="close" data-dismiss="alert" aria-label="close">&times;</a>
-        <ul class=flashes>
-            {% for message in messages %}
-            <li>{{ message }}</li>
-            {% endfor %}
-        </ul>
-    </div>
-</div>
-{% endif %}
-{% endwith %}
-
-<div class="container">
-
-    <div class="row">
-        <div class="col-md-12">
-            <center>
-                <a href="{{ url_for('search')}}?query=&fields=">
-                <img src="{{ url_for('static', filename='centillion_white.png') }}">
-                </a>
-                {% if config['TAGLINE'] %}
-                    <h2><a href="{{ url_for('search')}}?query=&fields=">
-                        {{config['TAGLINE']}}
-                    </a></h2>
-                {% endif %}
-            </center>
-        </div>
-    </div>
-    
-    {% if config['zzzTAGLINE'] %}
-    <div class="row">
-        <div class="col12sm">
-            <center>
-                <h2><a href="{{ url_for('search')}}?query=&fields=">
-                    {{config['TAGLINE']}}
-                </a></h2>
-            </center>
-        </div>
-    </div>
-    {% endif %}
-
-</div>
-
 <hr />
 
 <div class="container">
@@ -66,9 +21,9 @@
                         <div class="col-md-12">
                             <p class="panel-text">Re-index <i>every</i> document in the
                             remote collection in the search index. <b>Warning: this operation may take a while.</b>
-                            <p/> <p>
+                            </p>
-                            <a href="{{ url_for('update_index') }}" class="btn btn-large btn-danger">Update Main Index</a>
+                            <p><a id="re-index-main" href="{{ url_for('update_index',run_which='all') }}" class="btn btn-large btn-danger btn-reindex-all">Update Main Index</a>
-                            <p/> 
+                            </p> 
                         </div>
                     </div>
                 </div>
@@ -76,22 +31,31 @@
             </div>
         </div>
 
-        {# update diff search index #}
+
+
+        {# update search index by type #}
         <div class="panel panel-danger">
             <div class="panel-heading">
                 <h3 class="panel-title">
-                    Update Diff Search Index
+                    Update Search Index by Type
                 </h3>
             </div>
             <div class="panel-body">
                 <div class="container-fluid">
                     <div class="row">
                         <div class="col-md-12">
-                            <p class="panel-text">Diff search index only re-indexes documents created after the last
+                            <p class="panel-text">Re-index individual document types in the search index.
-                            search index update. <b>Not currently implemented.</b>
+                            </p>
-                            <p/> <p>
+                            <p><a href="{{ url_for('update_index',run_which='gdocs') }}"        class="btn btn-large btn-danger btn-reindex-type">Update Google Drive Index</a>
-                            <a href="#" class="btn btn-large disabled btn-danger">Update Diff Index</a>
+                            </p>  
-                            <p/> 
+                            <p><a href="{{ url_for('update_index',run_which='ghfiles') }}"      class="btn btn-large btn-danger btn-reindex-type">Update Github Files Index</a>
+                            </p>  
+                            <p><a href="{{ url_for('update_index',run_which='issues') }}"       class="btn btn-large btn-danger btn-reindex-type">Update Github Issues Index</a>
+                            </p>  
+                            <p><a href="{{ url_for('update_index',run_which='emailthreads') }}" class="btn btn-large btn-danger btn-reindex-type">Update Groups.io Email Threads Index</a>
+                            </p> 
+                            <p><a href="{{ url_for('update_index',run_which='disqus') }}"       class="btn btn-large btn-danger btn-reindex-type">Update Disqus Comment Threads Index</a>
+                            </p> 
                         </div>
                     </div>
                 </div>

templates/flashed_messages.html

@@ -0,0 +1,14 @@
+<div id="messages">
+{% with messages = get_flashed_messages() %}
+    {% if messages %}
+        <div class="container" id="flashed-messages-container">
+            <div class="alert alert-success alert-dismissible fade in">
+                <a href="#" class="close" data-dismiss="alert" aria-label="close">&times;</a>
+                    {% for message in messages %}
+                        <p>{{ message }}</p>
+                    {% endfor %}
+            </div>
+        </div>
+    {% endif %}
+{% endwith %}
+</div>

templates/landing.html

@@ -0,0 +1,23 @@
+{% extends "layout.html" %}
+{% set active_page = "landing" %}
+{% block body %}
+<div class="container">
+    <div class="row">
+
+        <div class="col-xs-12">
+
+            <center>
+                <p class="lead">To begin searching the Data Commons, you must first sign in with Github.
+                </p>
+                <a href="/log_in" type="button" id="github-button" class="btn btn-default">
+                <i class="fa fa-github fa-2x"></i> Sign in with Github
+                </a>
+            </center>
+
+        </div>
+
+    </div>
+</div>
+{% endblock %}
+
+

templates/layout.html

@@ -1,12 +1,66 @@
 <!doctype html>
 <title>Centillion Search Engine</title>
+
 <link rel="stylesheet" type="text/css" href="{{ url_for('static', filename='style.css') }}">
 <link rel="stylesheet" type="text/css" href="{{ url_for('static', filename='github-markdown.css') }}">
 <link rel="stylesheet" type="text/css" href="{{ url_for('static', filename='bootstrap.min.css') }}">
+<link rel="stylesheet" type="text/css" href="{{ url_for('static', filename='font-awesome.min.css') }}">
 
 <script src="{{ url_for('static', filename='jquery.min.js') }}"></script>
 <script src="{{ url_for('static', filename='bootstrap.min.js') }}"></script>
+<script src="{{ url_for('static', filename='master_list.js') }}"></script>
+<script src="{{ url_for('static', filename='search_list.js') }}"></script>
+<script src="{{ url_for('static', filename='feedback.js') }}"></script>
+
+
+{# ########## dataTables plugin ############ #}
+
+<link rel="stylesheet" type="text/css" href="{{ url_for('static', filename='dataTables.bootstrap.min.css') }}">
+<link rel="stylesheet" type="text/css" href="{{ url_for('static', filename='dataTables.responsive.css') }}">
+
+<script src="{{ url_for('static', filename='jquery.dataTables.min.js') }}"></script>
+<script src="{{ url_for('static', filename='dataTables.bootstrap.min.js') }}"></script>
+<script src="{{ url_for('static', filename='dataTables.responsive.js') }}"></script>
+
+
+
+<div id="master-div">
+
+    {#
+    flashed messages
+    #}
+    {% include "flashed_messages.html" %}
+
+    {#
+    banner image
+    #}
+    {% include "banner.html" %}
+
+    {#
+    feedback modal
+    #}
+    {% include "modal.html" %}
 
-<div>
     {% block body %}{% endblock %}
+
 </div>
+
+{% if active_page=="search" or active_page=="master_list" %}
+
+    {# feedback button #}
+    <button id="feedback" type="button" 
+                          data-toggle="modal"
+                          data-target="#myModal"
+        class="btn btn-lg">Send Feedback</button>
+    
+    {# vertical spacing before the bottom, b/c of button #}
+    <div id="footer-whitespace" class="container">
+        <p>&nbsp;</p>
+        <p>&nbsp;</p>
+        <p>&nbsp;</p>
+    </div>
+
+{% endif %}
+
+<a id="github-corner" href="https://github.com/dcppc/centillion" class="github-corner" aria-label="View source on Github"><svg width="80" height="80" viewBox="0 0 250 250" style="fill:#151513; color:#fff; position: absolute; top: 0; border: 0; right: 0;" aria-hidden="true"><path d="M0,0 L115,115 L130,115 L142,142 L250,250 L250,0 Z"></path><path d="M128.3,109.0 C113.8,99.7 119.0,89.6 119.0,89.6 C122.0,82.7 120.5,78.6 120.5,78.6 C119.2,72.0 123.4,76.3 123.4,76.3 C127.3,80.9 125.5,87.3 125.5,87.3 C122.9,97.6 130.6,101.9 134.4,103.2" fill="currentColor" style="transform-origin: 130px 106px;" class="octo-arm"></path><path d="M115.0,115.0 C114.9,115.1 118.7,116.5 119.8,115.4 L133.7,101.6 C136.9,99.2 139.9,98.4 142.2,98.6 C133.8,88.0 127.5,74.4 143.8,58.0 C148.5,53.4 154.0,51.2 159.7,51.0 C160.3,49.4 163.2,43.6 171.4,40.1 C171.4,40.1 176.1,42.5 178.8,56.2 C183.1,58.6 187.2,61.8 190.9,65.4 C194.5,69.0 197.7,73.2 200.1,77.6 C213.8,80.2 216.3,84.9 216.3,84.9 C212.7,93.1 206.9,96.0 205.4,96.6 C205.1,102.4 203.0,107.8 198.3,112.5 C181.9,128.9 168.3,122.5 157.7,114.1 C157.9,116.9 156.7,120.9 152.7,124.9 L141.0,136.5 C139.8,137.7 141.6,141.9 141.8,141.8 Z" fill="currentColor" class="octo-body"></path></svg></a><style>.github-corner:hover .octo-arm{animation:octocat-wave 560ms ease-in-out}@keyframes octocat-wave{0%,100%{transform:rotate(0)}20%,60%{transform:rotate(-25deg)}40%,80%{transform:rotate(10deg)}}@media (max-width:500px){.github-corner:hover .octo-arm{animation:none}.github-corner .octo-arm{animation:octocat-wave 560ms ease-in-out}}</style>
+

templates/masterlist.html

@@ -0,0 +1,244 @@
+{% extends "layout.html" %}
+{% set active_page = "master_list" %}
+{% block body %}
+
+<hr />
+
+<div class="container">
+
+    <div class="row">
+
+    {#
+    # google drive files panel
+    #}
+    <a name="gdoc"></a>
+    <div class="row">
+        <div class="panel">
+            <div class="panel-group" id="accordionDrive" role="tablist" aria-multiselectable="true">
+                <div class="panel panel-default">
+                    <div class="panel-heading" role="tab" id="googleDrive">
+
+                        <h2 class="masterlist-header">
+                            <a class="collapsed" 
+                                role="button"
+                                onClick="load_gdoc_table()"
+                                data-toggle="collapse" 
+                                data-parent="#accordionDrive"
+                                href="#collapseDrive" 
+                                aria-expanded="true"
+                                aria-controls="collapseDrive">
+                                Google Drive Files <small>indexed by centillion</small>
+                            </a>
+                        </h2>
+
+                    </div>
+                    <div id="collapseDrive" class="panel-collapse collapse" role="tabpanel" aria-labelledby="googleDrive">
+                        <div class="panel-body">
+                            <table class="table table-striped" id="gdocs-master-list">
+
+                            </table>
+                        </div>
+                    </div>
+                </div>
+            </div>
+        </div>
+    </div>
+
+
+
+    {#
+    # github issue panel
+    #}
+    <a name="issue"></a>
+    <div class="row">
+        <div class="panel">
+            <div class="panel-group" id="accordionIssues" role="tablist" aria-multiselectable="true">
+                <div class="panel panel-default">
+                    <div class="panel-heading" role="tab" id="githubIssues">
+
+                        <h2 class="masterlist-header">
+                            <a class="collapsed" 
+                                role="button"
+                                onClick="load_issue_table()"
+                                data-toggle="collapse" 
+                                data-parent="#accordionIssues"
+                                href="#collapseIssues" 
+                                aria-expanded="true"
+                                aria-controls="collapseIssues">
+                                Github Issues <small>indexed by centillion</small>
+                            </a>
+                        </h2>
+
+                    </div>
+                    <div id="collapseIssues" class="panel-collapse collapse" role="tabpanel" 
+                        aria-labelledby="githubIssues">
+                        <div class="panel-body">
+                            <table class="table table-striped" id="issues-master-list">
+
+                            </table>
+                        </div>
+                    </div>
+                </div>
+            </div>
+        </div>
+    </div>
+
+
+
+
+    {#
+    # github file panel
+    #}
+    <a name="ghfile"></a>
+    <div class="row">
+        <div class="panel">
+            <div class="panel-group" id="accordionFiles" role="tablist" aria-multiselectable="true">
+                <div class="panel panel-default">
+                    <div class="panel-heading" role="tab" id="githubFiles">
+
+                        <h2 class="masterlist-header">
+                            <a class="collapsed" 
+                                role="button"
+                                onClick="load_ghfile_table()"
+                                data-toggle="collapse" 
+                                data-parent="#accordionFiles"
+                                href="#collapseFiles" 
+                                aria-expanded="true"
+                                aria-controls="collapseFiles">
+                                Github Files <small>indexed by centillion</small>
+                            </a>
+                        </h2>
+
+                    </div>
+                    <div id="collapseFiles" class="panel-collapse collapse" role="tabpanel" 
+                        aria-labelledby="githubFiles">
+                        <div class="panel-body">
+                            <table class="table table-striped" id="ghfiles-master-list">
+
+                            </table>
+                        </div>
+                    </div>
+                </div>
+            </div>
+        </div>
+    </div>
+
+
+    {#
+    # gh markdown file panel
+    #}
+    <a name="markdown"></a>
+    <div class="row">
+        <div class="panel">
+            <div class="panel-group" id="accordionMarkdown" role="tablist" aria-multiselectable="true">
+                <div class="panel panel-default">
+                    <div class="panel-heading" role="tab" id="markdown">
+
+                        <h2 class="masterlist-header">
+                            <a class="collapsed" 
+                                role="button"
+                                onClick="load_markdown_table()"
+                                data-toggle="collapse" 
+                                data-parent="#accordionMarkdown"
+                                href="#collapseMarkdown" 
+                                aria-expanded="true"
+                                aria-controls="collapseMarkdown">
+                                Github Markdown Files <small>indexed by centillion</small>
+                            </a>
+                        </h2>
+
+                    </div>
+                    <div id="collapseMarkdown" class="panel-collapse collapse" role="tabpanel" 
+                        aria-labelledby="markdown">
+                        <div class="panel-body">
+                            <table class="table table-striped" id="markdown-master-list">
+
+                            </table>
+                        </div>
+                    </div>
+                </div>
+            </div>
+        </div>
+    </div>
+
+
+
+    {#
+    # groups.io email threads
+    #}
+    <a name="emailthread"></a>
+    <div class="row">
+        <div class="panel">
+            <div class="panel-group" id="accordionThreads" role="tablist" aria-multiselectable="true">
+                <div class="panel panel-default">
+                    <div class="panel-heading" role="tab" id="threads">
+
+                        <h2 class="masterlist-header">
+                            <a class="collapsed" 
+                                role="button"
+                                onClick="load_emailthreads_table()"
+                                data-toggle="collapse" 
+                                data-parent="#accordionThreads"
+                                href="#collapseThreads" 
+                                aria-expanded="true"
+                                aria-controls="collapseThreads">
+                                Groups.io Email Threads <small>indexed by centillion</small>
+                            </a>
+                        </h2>
+
+                    </div>
+                    <div id="collapseThreads" class="panel-collapse collapse" role="tabpanel" 
+                        aria-labelledby="threads">
+                        <div class="panel-body">
+                            <table class="table table-striped" id="emailthreads-master-list">
+
+                            </table>
+                        </div>
+                    </div>
+                </div>
+            </div>
+        </div>
+    </div>
+
+    {#
+    # disqus comment threads
+    #}
+    <a name="disqus"></a>
+    <div class="row">
+        <div class="panel">
+            <div class="panel-group" id="accordionDisqus" role="tablist" aria-multiselectable="true">
+                <div class="panel panel-default">
+                    <div class="panel-heading" role="tab" id="disqus">
+
+                        <h2 class="masterlist-header">
+                            <a class="collapsed" 
+                                role="button"
+                                onClick="load_disqusthreads_table()"
+                                data-toggle="collapse" 
+                                data-parent="#accordionDisqus"
+                                href="#collapseDisqus" 
+                                aria-expanded="true"
+                                aria-controls="collapseDisqus">
+                                Disqus Comment Threads <small>indexed by centillion</small>
+                            </a>
+                        </h2>
+
+                    </div>
+                    <div id="collapseDisqus" class="panel-collapse collapse" role="tabpanel" 
+                        aria-labelledby="disqus">
+                        <div class="panel-body">
+                            <table class="table table-striped" id="disqus-master-list">
+
+                            </table>
+                        </div>
+                    </div>
+                </div>
+            </div>
+        </div>
+    </div>
+
+
+</div>
+
+{% endblock %}
+

templates/modal.html

@@ -0,0 +1,51 @@
+<div class="modal fade" id="myModal" tabindex="-1" role="dialog" aria-labelledby="myModalLabel">
+
+  <form id="myModalForm" method="post">
+
+    <div class="modal-dialog" role="document">
+      <div id="myModal-content" class="modal-content">
+        <div id="myModal-header" class="modal-header">
+          <button type="button" class="close" data-dismiss="modal" aria-label="Close">
+              <span aria-hidden="true">&times;</span>
+          </button>
+          <h4 class="modal-title" id="myModalLabel">
+          Send us feedback!
+          </h4>
+        </div>
+        <div id="myModal-body" class="modal-body">
+          <div id="modal-feedback-smile-frown-container" class="container-fluid">
+              <div id="modal-feedback-smile-div" class="col-xs-6 text-center" 
+                  onClick="smile()">
+                  <i id="modal-feedback-smile-icon" class="fa fa-smile-o fa-4x" aria-hidden="true"></i>
+              </div>
+              <div id="modal-feedback-frown-div" class="col-xs-6 text-center" 
+                  onClick="frown()">
+                  <i id="modal-feedback-frown-icon" class="fa fa-frown-o fa-4x" aria-hidden="true"></i>
+              </div>
+          </div>
+          <div class="container-fluid">
+              <p>&nbsp;</p>
+          </div>
+          <div id="modal-feedback-textarea-container" class="container-fluid">
+              <textarea id="modal-feedback-textarea" rows="6"></textarea>
+          </div>
+          <div id="modal-too-long" class="container-fluid" >
+              <p id="modal-too-long-text" class="lead">Please limit the length of your feedback. Thank you in advance!</p>
+          </div>
+        </div>
+        <div id="myModal-footer" class="modal-footer">
+            <div class="text-center">
+              <button id="submit-feedback-btn" type="button" 
+                  onClick="submit_feedback()"
+                  class="btn btn-lg btn-primary">
+                  Send
+              </button>
+            </div>
+        </div>
+      </div>
+    </div>
+
+  </form>
+
+</div>        
+

templates/oldsearch.html

@@ -0,0 +1,165 @@
+{% extends "layout.html" %}
+{% block body %}
+
+<div class="container">
+    <div class="row">
+
+        <div class="col-xs-12">
+            <center>
+                <form action="{{ url_for('search') }}" name="search">
+                    <p><input type="text" name="query" value="{{ query }}">
+                    </p>
+                    <p><button id="the-big-one" type="submit" style="font-size: 20px; padding: 10px; padding-left: 50px; padding-right: 50px;" 
+                        value="search" class="btn btn-primary">Search</button>
+                    </p>
+                    <p><a href="{{ url_for('search')}}?query=&fields=">[clear all results]</a>
+                    </p>
+                </form>
+            </center>
+        </div>
+    </div>
+</div>
+
+<div class="container">
+    <div class="row">
+
+        {% if directories %}
+        <div class="col-xs-12 info directories-cloud">
+            <b>File directories:</b> 
+            {% for d in directories %}
+                <a href="{{url_for('search')}}?query={{d|trim}}&fields=filename">{{d|trim}}</a>
+            {% endfor %}
+        </div>
+        {% endif %}
+
+        <ul class="list-group">
+
+            {% if config['SHOW_PARSED_QUERY'] and parsed_query %}
+                <li  class="list-group-item">
+                    <div class="container-fluid">
+                        <div class="row">
+                            <div class="col-xs-12 info">
+                                <b>Parsed query:</b> {{ parsed_query }}
+                            </div>
+                        </div>
+                    </div>
+                </li>
+            {% endif %}
+
+            {% if parsed_query %}
+                <li  class="list-group-item">
+                    <div class="container-fluid">
+                        <div class="row">
+                            <div class="col-xs-12 info">
+                                <b>Found:</b> <span class="badge">{{entries|length}}</span> results 
+                                out of <span class="badge">{{totals["total"]}}</span> total items indexed
+                            </div>
+                        </div>
+                    </div>
+                </li>
+            {% endif %}
+
+            <li  class="list-group-item">
+                    <div class="container-fluid">
+                        <div class="row">
+                            <div class="col-xs-12 info">
+                                <b>Indexing:</b>
+
+                                <span class="badge">{{totals["gdoc"]}}</span>
+                                <a href="/master_list?doctype=gdoc">
+                                Google Drive files
+                                </a>,
+
+                                <span class="badge">{{totals["issue"]}}</span>
+                                <a href="/master_list?doctype=issue">
+                                Github issues
+                                </a>,
+
+                                <span class="badge">{{totals["ghfile"]}}</span>
+                                <a href="/master_list?doctype=ghfile">
+                                Github files
+                                </a>,
+
+                                <span class="badge">{{totals["markdown"]}}</span>
+                                <a href="/master_list?doctype=markdown">
+                                Github Markdown files
+                                </a>,
+
+                                <span class="badge">{{totals["emailthread"]}}</span>
+                                <a href="/master_list?doctype=emailthread">
+                                Groups.io email threads
+                                </a>
+                            </div>
+                        </div>
+                </div>
+            </li>
+
+        </ul>
+    </div>
+</div>
+
+<div class="container">
+    <div class="row">
+        <ul class="list-group">
+
+            {% for e in entries %}
+                <li  class="search-group-item">
+
+                    <div class="url">
+                        {% if e.kind=="gdoc" %}
+                            {% if e.mimetype=="" %}
+                                <b>Google Document:</b>
+                                <a href='{{e.url}}'>{{e.title}}</a>
+                                (Owner: {{e.owner_name}}, {{e.owner_email}})<br />
+                                <b>Document Type</b>: {{e.mimetype}}
+                            {% else %}
+                                <b>Google Drive File:</b>
+                                <a href='{{e.url}}'>{{e.title}}</a><br />
+                                <b>Owner:</b> {{e.owner_name}}, {{e.owner_email}}<br />
+                            {% endif %}
+
+                        {% elif e.kind=="issue" %}
+                            <b>Github Issue:</b>
+                            <a href='{{e.url}}'>{{e.title}}</a>
+                            {% if e.github_user %}<br />
+                            <b>Opened by:</b> <a href='https://github.com/{{e.github_user}}'>@{{e.github_user}}</a>
+                            {% endif %}
+                            <br/>
+                            <b>Repository:</b> <a href='{{e.repo_url}}'>{{e.repo_name}}</a>
+
+                        {% elif e.kind=="markdown" %}
+                            <b>Github Markdown:</b>
+                            <a href='{{e.url}}'>{{e.title}}</a>
+                            <br/>
+                            <b>Repository:</b> <a href='{{e.repo_url}}'>{{e.repo_name}}</a>
+
+                        {% elif e.kind=="emailthread" %}
+                            <b>Groups.io Email Thread:</b>
+                            <a href='{{e.url}}'>{{e.title}}</a>
+                            <br/>
+                            <b>Started By:</b> {{e.owner_name}}
+
+                        {% else %}
+                            <b>Item:</b> (<a href='{{e.url}}'>link</a>)
+
+                        {% endif %}
+                        <br />
+                        Score: {{'%d'  % e.score}}
+                    </div>
+                    <div class="markdown-body">
+                        {% if e.content_highlight %}
+                            {{ e.content_highlight|safe}}
+                        {% else %}
+                        <p>(A preview of this document is not available.)</p>
+                        {% endif %}
+                    </div>
+
+                </li>
+            {% endfor %}
+        </ul>
+
+    </div>
+</div>
+
+{% endblock %}
+

templates/search.html

@@ -1,59 +1,34 @@
 {% extends "layout.html" %}
+{% set active_page = "search" %}
 {% block body %}
 
-
+<div id="search-bar-container" class="container">
-<div class="container">
-
-    {#
-    banner image
-    #}
-    <div class="row">
-        <div class="col12sm">
-            <center>
-                <a href="{{ url_for('search')}}?query=&fields=">
-                <img src="{{ url_for('static', filename='centillion_white.png') }}">
-                </a>
-                {#
-                need a tag line
-                #}
-                {% if config['TAGLINE'] %}
-                    <h2><a href="{{ url_for('search')}}?query=&fields=">
-                        {{config['TAGLINE']}}
-                    </a></h2>
-                {% endif %}
-            </center>
-        </div>
-    </div>
-
-</div>
-<div class="container">
     <div class="row">
 
         <div class="col-xs-12">
             <center>
                 <form action="{{ url_for('search') }}" name="search">
-                    <input type="text" name="query" value="{{ query }}"> <br />
+                    <p><input type="text" name="query" value="{{ query }}">
-                    <button type="submit" style="font-size: 20px; padding: 10px; padding-left: 50px; padding-right: 50px;" 
+                    </p>
+                    <p><button id="the-big-one" type="submit" style="font-size: 20px; padding: 10px; padding-left: 50px; padding-right: 50px;" 
                         value="search" class="btn btn-primary">Search</button>
-                    <br />
+                    </p>
-                    <a href="{{ url_for('search')}}?query=&fields=">[clear all results]</a>
+
+                    {% if parsed_query %}
+                        <p><a href="{{ url_for('search')}}?query=&fields=">[clear all results]</a>
+                    {% endif %}
+
+                    </p>
                 </form>
             </center>
         </div>
     </div>
 </div>
 
-<div class="container">
+<div style="height: 20px;"><p>&nbsp;</p></div>
-    <div class="row">
 
-        {% if directories %}
+<div id="info-bars-container" class="container">
-        <div class="col-xs-12 info directories-cloud">
+    <div class="row">
-            <b>File directories:</b> 
-            {% for d in directories %}
-                <a href="{{url_for('search')}}?query={{d|trim}}&fields=filename">{{d|trim}}</a>
-            {% endfor %}
-        </div>
-        {% endif %}
 
         <ul class="list-group">
 
@@ -69,44 +44,120 @@
                 </li>
             {% endif %}
 
+
+            {# use "if parsed_query" to check if this is 
+               a new search or search results #}
             {% if parsed_query %}
                 <li  class="list-group-item">
                     <div class="container-fluid">
                         <div class="row">
                             <div class="col-xs-12 info">
-                                <b>Found:</b> <span class="badge">{{entries|length}}</span> results 
+                                <b>Found:</b> <span class="badge results-count">{{entries|length}}</span> results 
-                                out of <span class="badge">{{totals["total"]}}</span> total items indexed
+                                out of <span class="badge results-count">{{totals["total"]}}</span> total items indexed
                             </div>
                         </div>
                     </div>
                 </li>
             {% endif %}
 
+
             <li  class="list-group-item">
                     <div class="container-fluid">
                         <div class="row">
                             <div class="col-xs-12 info">
-                                <b>Indexing:</b> <span
+                                <b>Indexing:</b>
-                                    class="badge">{{totals["gdoc"]}}</span> Google Documents,
+
-                                <span class="badge">{{totals["issue"]}}</span> Github issues,
+                                <span class="badge indexing-count">{{totals["gdoc"]}}</span>
-                                <span class="badge">{{totals["ghfile"]}}</span> Github files,
+                                <a href="/master_list?doctype=gdoc#gdoc">
-                                <span class="badge">{{totals["markdown"]}}</span> Github markdown files.
+                                Google Drive files
+                                </a>,
+
+                                <span class="badge indexing-count">{{totals["issue"]}}</span>
+                                <a href="/master_list?doctype=issue#issue">
+                                Github issues
+                                </a>,
+
+                                <span class="badge indexing-count">{{totals["ghfile"]}}</span>
+                                <a href="/master_list?doctype=ghfile#ghfile">
+                                Github files
+                                </a>,
+
+                                <span class="badge indexing-count">{{totals["markdown"]}}</span>
+                                <a href="/master_list?doctype=markdown#markdown">
+                                Github Markdown files
+                                </a>,
+
+                                <span class="badge indexing-count">{{totals["emailthread"]}}</span>
+                                <a href="/master_list?doctype=emailthread#emailthread">
+                                Groups.io email threads
+                                </a>,
+
+                                <span class="badge indexing-count">{{totals["disqus"]}}</span>
+                                <a href="/master_list?doctype=disqus#disqus">
+                                Disqus comment threads
+                                </a>
                             </div>
                         </div>
                 </div>
             </li>
 
+
         </ul>
     </div>
 </div>
 
-<div class="container">
+{% if parsed_query %}
+<div id="search-results-container" class="container">
     <div class="row">
-        <ul class="list-group">
+        <table id="search-results" class="table">
+            <thead id="search-results-header">
+                <tr id="search-results-header-row">
+                    <td id="search-results-score-col">
+                        Score
+                    </td>
+                    <td id="search-results-type-col">
+                        Type
+                    </td>
+                    <td id="search-results-details-col">
+                        Result Details
+                    </td>
+                </tr>
+            </thead>
+            <tbody>
 
                 {% for e in entries %}
-                <li  class="search-group-item">
+                    <tr>
+                        <td>
+                            {{'%d'  % e.score}}
+                        </td>
+                        <td>
+                            {% if e.kind=="gdoc" %}
+                                {% if e.mimetype=="document" %}
+                                    <p><small>Drive Document</small</p>
+                                {% else %}
+                                    <p><small>Drive File</small</p>
+                                {% endif %}
+
+                            {% elif e.kind=="issue" %}
+                                <p><small>Issue</small</p>
+
+                            {% elif e.kind=="ghfile" %}
+                                <p><small>Github File</small</p>
 
+                            {% elif e.kind=="markdown" %}
+                                <p><small>Github Markdown</small</p>
+
+                            {% elif e.kind=="emailthread" %}
+                                <p><small>Email Thread</small</p>
+
+                            {% else %}
+                                <p><small>Unknown</small</p>
+
+                            {% endif %}
+                        </td>
+                        <td>
+                            <ul class="list-group">
+                                <li  class="search-group-item">
                                     <div class="url">
                                         {% if e.kind=="gdoc" %}
                                             {% if e.mimetype=="" %}
@@ -114,20 +165,38 @@
                                                 <a href='{{e.url}}'>{{e.title}}</a>
                                                 (Owner: {{e.owner_name}}, {{e.owner_email}})<br />
                                                 <b>Document Type</b>: {{e.mimetype}}
+                                                {% if e.created_time %} 
+                                                    <br/>
+                                                    <b>Created:</b> {{e.created_time}}
+                                                {% endif %}
                                             {% else %}
-                                <b>Google Drive:</b>
+                                                <b>Google Drive File:</b>
-                                <a href='{{e.url}}'>{{e.title}}</a>
+                                                <a href='{{e.url}}'>{{e.title}}</a><br />
-                                (Owner: {{e.owner_name}}, {{e.owner_email}})
+                                                <b>Owner:</b> {{e.owner_name}}, {{e.owner_email}}<br />
+                                                {% if e.created_time %} 
+                                                    <br/>
+                                                    <b>Created:</b> {{e.created_time}}
+                                                {% endif %}
                                             {% endif %}
 
                                         {% elif e.kind=="issue" %}
                                             <b>Github Issue:</b>
                                             <a href='{{e.url}}'>{{e.title}}</a>
-                            {% if e.github_user %}
+                                            {% if e.github_user %}<br />
-                            opened by <a href='https://github.com/{{e.github_user}}'>@{{e.github_user}}</a>
+                                            <b>Opened by:</b> <a href='https://github.com/{{e.github_user}}'>@{{e.github_user}}</a>
                                             {% endif %}
                                             <br/>
                                             <b>Repository:</b> <a href='{{e.repo_url}}'>{{e.repo_name}}</a>
+                                            {% if e.created_time %} 
+                                                <br/>
+                                                <b>Date:</b> {{e.created_time}}
+                                            {% endif %}
+
+                                        {% elif e.kind=="ghfile" %}
+                                            <b>Github File:</b>
+                                            <a href='{{e.url}}'>{{e.title}}</a>
+                                            <br/>
+                                            <b>Repository:</b> <a href='{{e.repo_url}}'>{{e.repo_name}}</a>
 
                                         {% elif e.kind=="markdown" %}
                                             <b>Github Markdown:</b>
@@ -135,12 +204,31 @@
                                             <br/>
                                             <b>Repository:</b> <a href='{{e.repo_url}}'>{{e.repo_name}}</a>
 
+                                        {% elif e.kind=="emailthread" %}
+                                            <b>Groups.io Email Thread:</b>
+                                            <a href='{{e.url}}'>{{e.title}}</a>
+                                            <br/>
+                                            <b>Started By:</b> {{e.owner_name}}
+                                            <br/>
+                                            <b>Mailing List:</b> {{e.group}}
+                                            {% if e.created_time %} 
+                                                <br/>
+                                                <b>Date:</b> {{e.created_time}}
+                                            {% endif %}
+
+                                        {% elif e.kind=="disqus" %}
+                                            <b>Disqus Comment Thread:</b>
+                                            <a href='{{e.url}}'>{{e.title}}</a>
+                                            <br/>
+                                            {% if e.created_time %} 
+                                                <br/>
+                                                <b>Date:</b> {{e.created_time}}
+                                            {% endif %}
+
                                         {% else %}
                                             <b>Item:</b> (<a href='{{e.url}}'>link</a>)
 
                                         {% endif %}
-                        <br />
-                        Score: {{'%d'  % e.score}}
                                     </div>
                                     <div class="markdown-body">
                                         {% if e.content_highlight %}
@@ -149,40 +237,19 @@
                                         <p>(A preview of this document is not available.)</p>
                                         {% endif %}
                                     </div>
-
                                 </li>
-            {% endfor %}
                             </ul>
+                        </td>
+                    </tr>
 
-    </div>
+                {% endfor %}
-</div>
-
-<div class="container">
-    <div class="row">
-        <ul class="list-group">
-
-            {% if config['FOOTER_REPO_NAME'] %}
-                {% if config['FOOTER_REPO_ORG'] %}
-
-                    <li  class="list-group-item">
-                        <div class="container-fluid">
-                            <div class="row">
-                                <div class="col-xs-12 info">
-                                    More information about {{config['FOOTER_REPO_NAME']}} can be found
-                                    in the <a href="https://github.com/{{config['FOOTER_REPO_ORG']}}/{{config['FOOTER_REPO_NAME']}}">{{config['FOOTER_REPO_ORG']}}/{{config['FOOTER_REPO_NAME']}}</a>
-                                    repository on Github.
-                                </div>
-                            </div>
-                        </div>
-                    </li>
-
-                {% endif %}
-            {% endif %}
 
-        </ul>
+            </tbody>
+        </table>
 
     </div>
 </div>
+{% endif %}
 
 {% endblock %}