Merge pull request #9 from dcppc/ACharbonneau-patch-1

Update config_centillion.json

.gitignore

@@ -1,8 +1,8 @@
+config_flask.py
 vp
 credentials.json
 drive*.json
 *.pyc
-config.py
 out/
 search_index/
 venv/

.gitmodules

@@ -0,0 +1,3 @@
+[submodule "mkdocs-material-dib"]
+	path = mkdocs-material-dib
+	url = https://github.com/dib-lab/mkdocs-material-dib.git

Readme.md

@@ -1,67 +1,94 @@
-# centillion
+# The Centillion
 
-**the centillion**: a pan-github-markdown-issues-google-docs search engine.
+**centillion**: a pan-github-markdown-issues-google-docs search engine.
 
 **a centillion**: a very large number consisting of a 1 with 303 zeros after it.
 
-the centillion is 3.03 log-times better than the googol.
+one centillion is 3.03 log-times better than a googol.
+
+![Screen shot of centillion](docs/images/ss.png)
+
 
 ## what is it
 
-The centillion is a search engine built using [whoosh](#),
-a Python library for building search engines.
+Centillion (https://github.com/dcppc/centillion) is a search engine that can index 
+three kinds of collections: Google Documents, Github issues, and Markdown files in 
+Github repos.
 
 We define the types of documents the centillion should index,
-and how, using what fields. The centillion then builds and
-updates a search index.
+what info and how. The centillion then builds and
+updates a search index. That's all done in `centillion_search.py`.
 
 The centillion also provides a simple web frontend for running
-queries against the search index.
+queries against the search index. That's done using a Flask server
+defined in `centillion.py`.
 
 The centillion keeps it simple.
 
+## authentication layer
 
-## work that is done
+Centillion lives behind a Github authentication layer, implemented with 
+[flask-dance](https://github.com/singingwolfboy/flask-dance). When you first
+visit the site it will ask you to authenticate with Github so that it can 
+verify you have permission to access the site.
 
-See [Workdone.md](Workdone.md)
+## technologies
+
+Centillion is a Python program built using whoosh (search engine library). It
+indexes the full text of docx files in Google Documents, just the filenames for
+non-docx files. The full text of issues and their comments are indexed, and
+results are grouped by issue. Centillion requires Google Drive and Github OAuth
+apps. Once you provide credentials to Flask you're all set to go. 
 
 
-## work that is being done
+## control panel
 
-See [Workinprogress.md](Workinprogress.md) for details about
-route and function layout. Summary below.
+There's also a control panel at <https://search.nihdatacommons.us/control_panel> 
+that allows you to rebuild the search index from scratch (the Google Drive indexing 
+takes a while).
 
-### code organization
-
-centillion app routes:
-
-- home
-    - if not logged in, landing page
-    - if logged in, redirect to search
-- search
-- main_index_update
-    - update main index, all docs period
+![Screen shot of centillion control panel](docs/images/cp.png)
 
 
-centillion Search functions:
+## quickstart (with Github auth)
 
-- open_index creates the schema
+Start by creating a Github OAuth application.
+Get the public and private application key 
+(client token and client secret token)
+from the Github application's page.
+You will also need a Github access token
+(in addition to the app tokens).
 
-- add_issue, add_md, add_document have three diff method sigs and add diff types
-  of documents to the search index
+When you create the application, set the callback
+URL to `/login/github/authorized`, as in:
 
-- update_all_issues or update_all_md or update_all_documents iterates over items
-  and determines whether each item needs to be updated in the search index
+```
+https://<url>/login/github/authorized
+```
 
-- update_main_index - update the entire search index
-    - calls all three update_all methods
+Edit the Flask configuration `config_flask.py`
+and set the public and private application keys.
 
-- create_search_results - package things up for jinja
+Now run centillion:
 
-- search - run the query, pass results to the jinja-packager
+```
+python centillion.py
+```
+
+or if you used http instead of https:
+
+```
+OAUTHLIB_INSECURE_TRANSPORT="true" python centillion.py
+```
+
+This will start a Flask server, and you can view the minimal search engine
+interface in your browser at `http://<ip>:5000`.
 
 
-## work that is planned
+## troubleshooting
 
-See [Workplanned.md](Workplanned.md)
+If you are having problems with your callback URL being treated
+as HTTP by Github, even though there is an HTTPS address, and
+everything else seems fine, try deleting the Github OAuth app
+and creating a new one.
 

Todo.md

@@ -0,0 +1,47 @@
+# todo
+
+Main task:
+- hashing and caching
+    - <s>first, working out the logic of how we group items into sets
+        - needs to be deleted
+        - needs to be updated
+        - needs to be added
+        - for docs, issues, and comments</s>
+    - second, when we add or update an item, need to:
+        - go through the motions, download file, extract text
+        - check for existing indexed doc with that id
+        - check if existing indexed doc has same hash
+            - if so, skip
+            - otherwise, delete and re-index
+
+Other bugs:
+- Some github issues have no title (?)
+- <s>Need to combine issues with comments</s>
+- Not able to index markdown files _in a repo_
+- (Longer term) update main index vs update diff index
+
+Needs:
+- <s>control panel</s>
+
+Thursday product:
+- Everything re-indexed nightly
+- Search engine built on all documents in Google Drive, all issues, markdown files
+- Using pandoc to extract Google Drive document contents
+- BRIEF quickstart documentation
+
+Future:
+- Future plans to improve - plugins, improving matching
+- Subdomain plans
+- Folksonomy tagging and integration plans
+
+
+
+
+config options for plugins
+conditional blocks with import github inside
+complicated tho - better to have components split off
+
+
+
+
+

Workinprogress.md

@@ -1,106 +0,0 @@
-# Components
-
-The components of centillion are as follows:
-- Flask application, which creates a Search object and uses it to search index
-- Search object, which allows you to create/update/search an index
-
-## Routes layout
-
-Current application routes are as follows:
-
-- home -> search
-- search
-- update_index
-
-Ideal application routes (using github flask dance oauth):
-
-- home
-    - if not logged in, landing page
-    - if logged in, redirect to search
-- search
-- main_index_update
-    - update main index, all docs period
-- delta_index_update
-    - updates delta index, docs that have changed since last main index
-
-There should be one route to update the main index
-
-There should be another route to update the delta index
-
-These should go off and call the update index methods
-for each respective type of document/collection.
-For example, if I call `main_index_update` route it should
-
-- call `main_index_update` for all github issues
-- call `main_index_update` for folder of markdown docs
-- call `main_index_update` for google drive folder
-
-These are all members of the Search class
-
-## Functions layout
-
-Functions of the entire search app:
-- create a search index
-- load a search index
-- call the search() method on the index
-- update the search index
-
-The first and last, creating and updating the search index,
-are of greatest interest.
-
-The Schema affects everything so it is hard to separate
-functionality into a main Search class shared by many.
-(Avoid inheritance/classes if possible.)
-
-current Search:
-- open_index creates the schema
-- add_issue or add_document adds an item to the index
-- add_all_issues or add_all_documents iterates over items and adds them to index
-- update_index_incremental - update the search index
-- create_search_results - package things up for jinja
-- search - run the query, pass results to the jinja-packager
-
-
-centillion Search:
-
-- open_index creates the schema
-
-- add_issue, add_md, add_document have three diff method sigs and add diff types
-  of documents to the search index
-
-- update_all_issues or update_all_md or update_all_documents iterates over items
-  and determines whether each item needs to be updated in the search index
-
-- update_main_index - update the entire search index
-    - calls all three update_all methods
-
-- create_search_results - package things up for jinja
-
-- search - run the query, pass results to the jinja-packager
-
-
-Nice to have but focus on it later:
-
-- update_diff_issues or update_diff_md or update_diff_documents iterates over items
-  and indexes recently-added items
-
-- update_diff_index - update the diff search index (what's been added since last
-  time)
-    - calls all three update_diff methods
-
-
-
-
-
-
-## Files layout
-
-Schema definition:
-* include a "kind" or "class" to group objects
-* can provide different searches of different collections
-* eventually can provide user with checkboxes
-
-
-
-
-

centillion.py

@@ -2,8 +2,11 @@ import threading
 from subprocess import call
 
 import codecs
-import os
+import os, json
+
+from werkzeug.contrib.fixers import ProxyFix
 from flask import Flask, request, redirect, url_for, render_template, flash
+from flask_dance.contrib.github import make_github_blueprint, github
 
 # create our application
 from centillion_search import Search
@@ -22,10 +25,12 @@ You provide:
     - Google Drive API key via file
 """
 
+
 class UpdateIndexTask(object):
-    def __init__(self, diff_index=False):
+    def __init__(self, gh_access_token, diff_index=False):
         self.diff_index = diff_index
         thread = threading.Thread(target=self.run, args=())
+        self.gh_access_token = gh_access_token
         thread.daemon = True
         thread.start()
 
@@ -38,90 +43,181 @@ class UpdateIndexTask(object):
         from get_centillion_config import get_centillion_config
         config = get_centillion_config('config_centillion.json')
 
-        gh_token = os.environ['GITHUB_ACESS_TOKEN']
-        search.update_index_issues(gh_token, config)
+        search.update_index_issues(self.gh_access_token,config)
+        search.update_index_markdown(self.gh_access_token,config)
         search.update_index_gdocs(config)
 
 
 
 app = Flask(__name__)
+app.wsgi_app = ProxyFix(app.wsgi_app)
 
 # Load default config and override config from an environment variable
 app.config.from_pyfile("config_flask.py")
 
-last_searches_file = app.config["INDEX_DIR"] + "/last_searches.txt"
+#github_bp = make_github_blueprint()
+github_bp = make_github_blueprint(
+                        client_id = os.environ.get('GITHUB_OAUTH_CLIENT_ID'),
+                        client_secret = os.environ.get('GITHUB_OAUTH_CLIENT_SECRET'),
+                        scope='read:org')
+
+app.register_blueprint(github_bp, url_prefix="/login")
+
+contents404 = "<html><body><h1>Status: Error 404 Page Not Found</h1></body></html>"
+contents403 = "<html><body><h1>Status: Error 403 Access Denied</h1></body></html>"
+contents200 = "<html><body><h1>Status: OK 200</h1></body></html>"
 
 
 ##############################
 # Flask routes
 
-
 @app.route('/')
 def index():
-    return redirect(url_for("search", query="", fields=""))
+
+    if not github.authorized:
+        return redirect(url_for("github.login"))
+
+    else:
+
+        username = github.get("/user").json()['login']
+
+        resp = github.get("/user/orgs")
+        if resp.ok:
+
+            # If they are in team copper, redirect to search.
+            # Otherwise, hit em with a 403
+            all_orgs = resp.json()
+            for org in all_orgs:
+                if org['login']=='dcppc':
+                    copper_team_id = '2700235'
+                    mresp = github.get('/teams/%s/members/%s'%(copper_team_id,username))
+                    if mresp.status_code==204:
+
+                        # --------------------
+                        # Business as usual
+                        return redirect(url_for("search", query="", fields=""))
+
+            return contents403
+
+        return contents404
+
+### @app.route('/')
+### def index():
+###     return redirect(url_for("search", query="", fields=""))
 
 @app.route('/search')
 def search():
-    query = request.args['query']
-    fields = request.args.get('fields')
-    if fields == 'None':
-        fields = None
 
-    search = Search(app.config["INDEX_DIR"])
-    if not query:
-        parsed_query = ""
-        result = []
+    if not github.authorized:
+        return redirect(url_for("github.login"))
 
-    else:
-        parsed_query, result = search.search(query.split(), fields=[fields])
-        store_search(query, fields)
+    username = github.get("/user").json()['login']
 
-    total = search.get_document_total_count()
+    resp = github.get("/user/orgs")
+    if resp.ok:
 
-    return render_template('search.html', entries=result, query=query, parsed_query=parsed_query, fields=fields, last_searches=get_last_searches(), total=total)
+        all_orgs = resp.json()
+        for org in all_orgs:
+            if org['login']=='dcppc':
 
-@app.route('/open')
-def open_file():
-    path = request.args['path']
-    fields = request.args.get('fields')
-    query = request.args['query']
-    call([app.config["EDIT_COMMAND"], path])
+                copper_team_id = '2700235'
+
+                mresp = github.get('/teams/%s/members/%s'%(copper_team_id,username))
+                if mresp.status_code==204:
+
+                    # --------------------
+                    # Business as usual
+                    query = request.args['query']
+                    fields = request.args.get('fields')
+                    if fields == 'None':
+                        fields = None
+
+                    search = Search(app.config["INDEX_DIR"])
+                    if not query:
+                        parsed_query = ""
+                        result = []
+
+                    else:
+                        parsed_query, result = search.search(query.split(), fields=[fields])
+
+                    totals = search.get_document_total_count()
+
+                    return render_template('search.html', 
+                                           entries=result, 
+                                           query=query, 
+                                           parsed_query=parsed_query, 
+                                           fields=fields, 
+                                           totals=totals)
+
+    return contents403
 
-    return redirect(url_for("search", query=query, fields=fields))
 
 @app.route('/update_index')
 def update_index():
-    rebuild = request.args.get('rebuild')
-    UpdateIndexTask(diff_index=False)
-    flash("Rebuilding index, check console output")
-    return render_template("search.html", query="", fields="", last_searches=get_last_searches())
+
+    if not github.authorized:
+        return redirect(url_for("github.login"))
+
+    username = github.get("/user").json()['login']
+
+    resp = github.get("/user/orgs")
+    if resp.ok:
+
+        all_orgs = resp.json()
+        for org in all_orgs:
+            if org['login']=='dcppc':
+
+                copper_team_id = '2700235'
+
+                mresp = github.get('/teams/%s/members/%s'%(copper_team_id,username))
+                if mresp.status_code==204:
+
+                    #gh_oauth_token = github.token['access_token']
+                    gh_access_token = app.config['GITHUB_TOKEN']
+
+                    # --------------------
+                    # Business as usual
+                    UpdateIndexTask(gh_access_token, 
+                                    diff_index=False)
+                    flash("Rebuilding index, check console output")
+                    return render_template("controlpanel.html", 
+                                           totals={})
+
+    return contents403
 
 
-##############
-# Utility methods
 
-def get_last_searches():
-    if os.path.exists(last_searches_file):
-        with codecs.open(last_searches_file, 'r', encoding='utf-8') as f:
-            contents = f.readlines()
-    else:
-        contents = []
-    return contents
+@app.route('/control_panel')
+def control_panel():
 
-def store_search(query, fields):
-    if os.path.exists(last_searches_file):
-        with codecs.open(last_searches_file, 'r', encoding='utf-8') as f:
-            contents = f.readlines()
-    else:
-        contents = []
+    if not github.authorized:
+        return redirect(url_for("github.login"))
 
-    search = "query=%s&fields=%s\n" % (query, fields)
-    if not search in contents:
-        contents.insert(0, search)
+    username = github.get("/user").json()['login']
 
-    with codecs.open(last_searches_file, 'w', encoding='utf-8') as f:
-        f.writelines(contents[:30])
+    resp = github.get("/user/orgs")
+    if resp.ok:
+
+        all_orgs = resp.json()
+        for org in all_orgs:
+            if org['login']=='dcppc':
+
+                copper_team_id = '2700235'
+
+                mresp = github.get('/teams/%s/members/%s'%(copper_team_id,username))
+                if mresp.status_code==204:
+
+                    return render_template("controlpanel.html", 
+                                           totals={})
+
+    return contents403
+
+
+@app.errorhandler(404)
+def oops(e):
+    return contents404
 
 if __name__ == '__main__':
-    app.run()
+    os.environ['OAUTHLIB_INSECURE_TRANSPORT'] = 'true'
+    app.run(host="0.0.0.0",port=5000)
 

centillion_prepare.py

@@ -0,0 +1,5 @@
+from gdrive_util import GDrive
+
+gd = GDrive()
+service = gd.get_service()
+

centillion_search.py

@@ -1,7 +1,8 @@
 import shutil
 import html.parser
 
-from github import Github
+from github import Github, GithubException
+import base64
 
 from gdrive_util import GDrive
 from apiclient.http import MediaIoBaseDownload
@@ -14,6 +15,8 @@ import tempfile, subprocess
 import pypandoc
 import os.path
 import codecs
+from datetime import datetime
+
 from whoosh.qparser import MultifieldParser, QueryParser
 from whoosh.analysis import StemmingAnalyzer
 
@@ -40,6 +43,7 @@ Search object functions:
 Schema:
     - id
     - kind
+    - fingerprint
     - created_time
     - modified_time
     - indexed_time
@@ -57,6 +61,10 @@ Schema:
 """
 
 
+def clean_timestamp(dt):
+    return dt.replace(microsecond=0).isoformat()
+
+
 class SearchResult:
     score = 1.0
     path = None
@@ -89,6 +97,11 @@ class Search:
     def __init__(self, index_folder):
         self.open_index(index_folder)
 
+
+    # ------------------------------
+    # Create a schema and open a search index
+    # on disk.
+
     def open_index(self, index_folder, create_new=False):
         """
         Create a schema,
@@ -109,13 +122,13 @@ class Search:
 
         
         # ------------------------------
-        # IMPORTANT:
         # This is where the search index's document schema
         # is defined.
 
         schema = Schema(
                 id = ID(stored=True, unique=True),
-                kind = ID(),
+                kind = ID(stored=True),
+                #fingerprint = ID(stored=True),
 
                 created_time = ID(stored=True),
                 modified_time = ID(stored=True),
@@ -154,16 +167,13 @@ class Search:
     # Define how to add documents
 
 
-    def add_drive_file(self, writer, item, indexed_ids, temp_dir, config):
+    def add_drive_file(self, writer, item, temp_dir, config, update=False):
         """
         Add a Google Drive document/file to a search index.
         If it is a document, extract the contents.
         """
-        gd = GDrive()
-        service = gd.get_service()
 
-        # ------------------------
-        # Two kinds of documents:
+        # There are two kinds of documents:
         # - documents with text that can be extracted (docx)
         # - everything else
         
@@ -172,10 +182,34 @@ class Search:
                 'document' : 'docx',
         }
 
-        if(mimetype not in mimemap.keys()):
-            # Not a document - 
-            # Just a file
-            print("Indexing document %s of type %s"%(item['name'], mimetype))
+        content = ""
+        if mimetype not in mimemap.keys():
+
+            # Not a document - just a file
+            print("Indexing Google Drive file \"%s\" of type %s"%(item['name'], mimetype))
+            writer.delete_by_term('id',item['id'])
+
+            # Index a plain google drive file
+            writer.add_document(
+                    id = item['id'],
+                    kind = 'gdoc',
+                    created_time = item['createdTime'],
+                    modified_time = item['modifiedTime'],
+                    indexed_time = datetime.now().replace(microsecond=0).isoformat(),
+                    title = item['name'],
+                    url = item['webViewLink'],
+                    mimetype = mimetype,
+                    owner_email = item['owners'][0]['emailAddress'],
+                    owner_name = item['owners'][0]['displayName'],
+                    repo_name='',
+                    repo_url='',
+                    github_user='',
+                    issue_title='',
+                    issue_url='',
+                    content = content
+            )
+
+
         else:
             # Document with text
             # Perform content extraction
@@ -187,7 +221,8 @@ class Search:
             # This is a file type we know how to convert
             # Construct the URL and download it
 
-            print("Extracting content from %s of type %s"%(item['name'], mimetype))
+            print("Indexing Google Drive document \"%s\" of type %s"%(item['name'], mimetype))
+            print(" > Extracting content")
 
 
             # Create a URL and a destination filename
@@ -208,7 +243,7 @@ class Search:
                 outfile_name = name+'.'+out_ext
 
 
-            # assemble input/output file paths
+            # Assemble input/output file paths
             fullpath_input = os.path.join(temp_dir,infile_name)
             fullpath_output = os.path.join(temp_dir,outfile_name)
 
@@ -217,7 +252,6 @@ class Search:
             with open(fullpath_input, 'wb') as f:
                 f.write(r.content)
 
-
             # Try to convert docx file to plain text
             try:
                 output = pypandoc.convert_file(fullpath_input,
@@ -227,7 +261,7 @@ class Search:
                 )
                 assert output == ""
             except RuntimeError:
-                print("XXXXXX Failed to index document %s"%(item['name']))
+                print(" > XXXXXX Failed to index document \"%s\""%(item['name']))
 
 
             # If export was successful, read contents of markdown
@@ -240,7 +274,7 @@ class Search:
 
 
             # No matter what happens, clean up.
-            print("Cleaning up %s"%item['name'])
+            print(" > Cleaning up \"%s\""%item['name'])
 
             subprocess.call(['rm','-fr',fullpath_output])
             #print(" ".join(['rm','-fr',fullpath_output]))
@@ -248,80 +282,156 @@ class Search:
             subprocess.call(['rm','-fr',fullpath_input])
             #print(" ".join(['rm','-fr',fullpath_input]))
 
+            if update:
+                print(" > Removing old record")
+                writer.delete_by_term('id',item['id'])
+            else:
+                print(" > Creating a new record")
 
-        # ------------------------------
-        # IMPORTANT:
-        # This is where the search documents are actually created.
-
-        mimetype = re.split('[/\.]', item['mimeType'])[-1]
-        writer.add_document(
-                id = item['id'],
-                kind = 'gdoc',
-                created_time = item['createdTime'],
-                modified_time = item['modifiedTime'],
-                title = item['name'],
-                url = item['webViewLink'],
-                mimetype = mimetype,
-                owner_email = item['owners'][0]['emailAddress'],
-                owner_name = item['owners'][0]['displayName'],
-                repo_name=None,
-                repo_url=None,
-                github_user=None,
-                issue_title=None,
-                issue_url=None,
-                content = content
-        )
+            writer.add_document(
+                    id = item['id'],
+                    kind = 'gdoc',
+                    created_time = item['createdTime'],
+                    modified_time = item['modifiedTime'],
+                    indexed_time = datetime.now().replace(microsecond=0).isoformat(),
+                    title = item['name'],
+                    url = item['webViewLink'],
+                    mimetype = mimetype,
+                    owner_email = item['owners'][0]['emailAddress'],
+                    owner_name = item['owners'][0]['displayName'],
+                    repo_name='',
+                    repo_url='',
+                    github_user='',
+                    issue_title='',
+                    issue_url='',
+                    content = content
+            )
 
 
-    def add_issue(self, writer, issue, repo, config):
+
+
+    # ------------------------------
+    # Add a single github issue and its comments
+    # to a search index.
+
+
+    def add_issue(self, writer, issue, gh_access_token, config, update=True):
         """
         Add a Github issue/comment to a search index.
         """
-        repo_name = repo.name
+        repo = issue.repository
+        repo_name = repo.owner.login+"/"+repo.name
         repo_url = repo.html_url
 
-        count = 0
-
-
-        # Handle the issue content
         print("Indexing issue %s"%(issue.html_url))
-        writer.add_document(
-                id = issue.html_url,
-                kind = 'issue',
-                url = issue.html_url,
-                is_comment = False,
-                timestamp = issue.created_at,
-                repo_name = repo_name,
-                repo_url = repo_url,
-                issue_title = issue.title,
-                issue_url = issue.html_url,
-                user = issue.user.login,
-                content = issue.body.rstrip()
-        )
-        count += 1
 
+        # Combine comments with their respective issues.
+        # Otherwise just too noisy.
+        issue_comment_content = issue.body.rstrip()
+        issue_comment_content += "\n"
 
         # Handle the comments content
         if(issue.comments>0):
+
             comments = issue.get_comments()
             for comment in comments:
-                print(" > Indexing comment %s"%(comment.html_url))
-                writer.add_document(
-                        id = comment.html_url,
-                        kind = 'comment',
-                        url = comment.html_url,
-                        is_comment = True,
-                        timestamp = comment.created_at,
-                        repo_name = repo_name,
-                        repo_url = repo_url,
-                        issue_title = issue.title,
-                        issue_url = issue.html_url,
-                        user = comment.user.login,
-                        content = comment.body.strip()
-                )
 
-        count += 1
-        return count
+                issue_comment_content += comment.body.rstrip()
+                issue_comment_content += "\n"
+
+        # Now create the actual search index record
+        created_time = clean_timestamp(issue.created_at)
+        modified_time = clean_timestamp(issue.updated_at)
+        indexed_time = clean_timestamp(datetime.now())
+
+        # Add one document per issue thread,
+        # containing entire text of thread.
+        writer.add_document(
+                id = issue.html_url,
+                kind = 'issue',
+                created_time = created_time,
+                modified_time = modified_time,
+                indexed_time = indexed_time,
+                title = issue.title,
+                url = issue.html_url,
+                mimetype='',
+                owner_email='',
+                owner_name='',
+                repo_name = repo_name,
+                repo_url = repo_url,
+                github_user = issue.user.login,
+                issue_title = issue.title,
+                issue_url = issue.html_url,
+                content = issue_comment_content
+        )
+
+
+
+    def add_markdown(self, writer, d, gh_access_token, config, update=True):
+        """
+        Use a Github markdown document API record
+        to add a markdown document's contents to
+        the search index.
+        """
+        repo = d['repo']
+        org = d['org']
+        repo_name = org + "/" + repo
+        repo_url = "https://github.com/" + repo_name
+
+        fpath = d['path']
+        furl = d['url']
+        fsha = d['sha']
+        _, fname = os.path.split(fpath)
+        _, fext = os.path.splitext(fpath)
+
+        print("Indexing markdown doc %s from repo %s"%(fname,repo_name))
+
+        # Unpack the requests response and decode the content
+        # 
+        # don't forget the headers for private repos!
+        # useful: https://bit.ly/2LSAflS
+
+        headers = {'Authorization' : 'token %s'%(gh_access_token)}
+
+        response = requests.get(furl, headers=headers)
+        if response.status_code==200:
+            jresponse = response.json()
+            content = ""
+            try:
+                binary_content = re.sub('\n','',jresponse['content'])
+                content = base64.b64decode(binary_content).decode('utf-8')
+            except KeyError:
+                print(" > XXXXXXXX Failed to extract 'content' field. You probably hit the rate limit.")
+
+        else:
+            print(" > XXXXXXXX Failed to reach file URL. There may be a problem with authentication/headers.")
+            return 
+
+        # Now create the actual search index record
+        indexed_time = clean_timestamp(datetime.now())
+
+        usable_url = "https://github.com/%s/blob/master/%s"%(repo_name, fpath)
+
+        # Add one document per issue thread,
+        # containing entire text of thread.
+        writer.add_document(
+                id = fsha,
+                kind = 'markdown',
+                created_time = '',
+                modified_time = '',
+                indexed_time = indexed_time,
+                title = fname,
+                url = usable_url,
+                mimetype='',
+                owner_email='',
+                owner_name='',
+                repo_name = repo_name,
+                repo_url = repo_url,
+                github_user = '',
+                issue_title = '',
+                issue_url = '',
+                content = content
+        )
 
 
 
@@ -329,133 +439,360 @@ class Search:
     # Define how to update search index
     # using different kinds of collections
 
+
+    # ------------------------------
+    # Google Drive Files/Documents
+
     def update_index_gdocs(self, 
                            config):
         """
         Update the search index using a collection of 
         Google Drive documents and files.
+        
+        Uses the 'id' field to uniquely identify documents.
+
+        Also see:
+        https://developers.google.com/drive/api/v3/reference/files
         """
-        gd = GDrive()
-        service = gd.get_service()
 
-        # -----
-        # Get the set of all documents on Google Drive:
+        # Updated algorithm:
+        # - get set of indexed ids
+        # - get set of remote ids
+        # - drop indexed ids not in remote ids
+        # - index all remote ids
+        # - add hash check in add_
 
-        # ------------------------------
-        # IMPORTANT:
-        # This determines what information about the Google Drive files
-        # you'll get back, and that's all you're going to have to work with.
-        # If you need more information, modify the statement below.
-        # Also see:
-        # https://developers.google.com/drive/api/v3/reference/files
 
+        # Get the set of indexed ids:
+        # ------
+        indexed_ids = set()
+        p = QueryParser("kind", schema=self.ix.schema)
+        q = p.parse("gdoc")
+        with self.ix.searcher() as s:
+            results = s.search(q,limit=None)
+            for result in results:
+                indexed_ids.add(result['id'])
+
+
+        # Get the set of remote ids:
+        # ------
+        # Start with google drive api object
         gd = GDrive()
         service = gd.get_service()
         drive = service.files()
 
+        # Now index all the docs in the google drive folder
 
         # The trick is to set next page token to None 1st time thru (fencepost)
         nextPageToken = None
 
         # Use the pager to return all the things
-        items = []
+        remote_ids = set()
+        full_items = {}
         while True:
+            ps = 100
             results = drive.list(
-                    pageSize=100,
+                    pageSize=ps,
                     pageToken=nextPageToken,
-                    fields="files(id, kind, createdTime, modifiedTime, mimeType, name, owners, webViewLink)",
+                    fields = "nextPageToken, files(id, kind, createdTime, modifiedTime, mimeType, name, owners, webViewLink)",
                     spaces="drive"
             ).execute()
 
             nextPageToken = results.get("nextPageToken")
-            items += results.get("files", [])
+            files = results.get("files",[])
+            for f in files:
                 
+                # Add all remote docs to a set
+                remote_ids.add(f['id'])
+
+                # Also store the doc
+                full_items[f['id']] = f
+            
+            ## Shorter:
+            #break
+            # Longer:
             if nextPageToken is None:
                 break
 
-        indexed_ids = set()
-        for item in items:
-            indexed_ids.add(item['id'])
 
         writer = self.ix.writer()
-
+        count = 0
         temp_dir = tempfile.mkdtemp(dir=os.getcwd())
         print("Temporary directory: %s"%(temp_dir))
-        if not os.path.exists(temp_dir):
-            os.mkdir(temp_dir)
 
-        count = 0
-        for item in items:
-            self.add_item(writer, item, indexed_ids, temp_dir, config)
+
+
+        # Drop any id in indexed_ids
+        # not in remote_ids
+        drop_ids = indexed_ids - remote_ids
+        for drop_id in drop_ids:
+            writer.delete_by_term('id',drop_id)
+
+
+        # Update any id in indexed_ids
+        # and in remote_ids
+        update_ids = indexed_ids & remote_ids
+        for update_id in update_ids:
+            # cop out
+            writer.delete_by_term('id',update_id)
+            item = full_items[update_id]
+            self.add_drive_file(writer, item, temp_dir, config, update=True)
             count += 1
 
+
+        # Add any id not in indexed_ids
+        # and in remote_ids
+        add_ids = remote_ids - indexed_ids
+        for add_id in add_ids:
+            item = full_items[add_id]
+            self.add_drive_file(writer, item, temp_dir, config, update=False)
+            count += 1
+
+
+        print("Cleaning temporary directory: %s"%(temp_dir))
+        subprocess.call(['rm','-fr',temp_dir])
+
         writer.commit()
         print("Done, updated %d documents in the index" % count)
 
 
+    # ------------------------------
+    # Github Issues/Comments
 
-
-    def update_index_issues(self, 
-                            gh_access_token,
-                            config):
+    def update_index_issues(self, gh_access_token, config):
         """
         Update the search index using a collection of 
         Github repo issues and comments.
         """
-        # Strategy:
-        # To get the proof of concept up and running,
-        # we are just deleting and re-indexing every issue/comment.
+        # Updated algorithm:
+        # - get set of indexed ids
+        # - get set of remote ids
+        # - drop indexed ids not in remote ids
+        # - index all remote ids
 
+        # Get the set of indexed ids:
+        # ------
+        indexed_issues = set()
+        p = QueryParser("kind", schema=self.ix.schema)
+        q = p.parse("gdoc")
+        with self.ix.searcher() as s:
+            results = s.search(q,limit=None)
+            for result in results:
+                indexed_issues.add(result['id'])
+
+
+        # Get the set of remote ids:
+        # ------
+        # Start with api object
         g = Github(gh_access_token)
 
-        # Set of all URLs as existing on github
-        to_index = set()
+        # Now index all issue threads in the user-specified repos
 
-        writer = self.ix.writer()
+        # Start by collecting all the things
+        remote_issues = set()
+        full_items = {}
 
         # Iterate over each repo 
-        list_of_repos = config['repos']
+        list_of_repos = config['repositories']
         for r in list_of_repos:
 
             if '/' not in r:
                 err = "Error: specify org/reponame or user/reponame in list of repos"
                 raise Exception(err)
 
-            this_repo, this_org = re.split('/',r)
+            this_org, this_repo = re.split('/',r)
+            try:
+                org = g.get_organization(this_org)
+                repo = org.get_repo(this_repo)
+            except:
+                print("Error: could not gain access to repository %s"%(r))
+                continue
 
-            org = g.get_organization(this_org)
-            repo = org.get_repo(this_repo)
-
-            count = 0
-
-            # Iterate over each thread
+            # Iterate over each issue thread
             issues = repo.get_issues()
             for issue in issues:
 
-                # This approach is more work than is needed
-                # but PoC||GTFO
-
                 # For each issue/comment URL,
-                # remove the corresponding item
-                # and re-add it to the index
+                # grab the key and store the 
+                # corresponding issue object
+                key = issue.html_url
+                value = issue
 
-                to_index.add(issue.html_url)
-                writer.delete_by_term('url', issue.html_url)
-                comments = issue.get_comments()
+                remote_issues.add(key)
+                full_items[key] = value
 
-                for comment in comments:
-                    to_index.add(comment.html_url)
-                    writer.delete_by_term('url', comment.html_url)
+        writer = self.ix.writer()
+        count = 0
 
-                # Now re-add this issue to the index
-                # (this will also add the comments)
-                count += self.add_issue(writer, issue, repo, config)
+        # Drop any issues in indexed_issues
+        # not in remote_issues
+        drop_issues = indexed_issues - remote_issues
+        for drop_issue in drop_issues:
+            writer.delete_by_term('id',drop_issue)
+
+
+        # Update any issue in indexed_issues
+        # and in remote_issues
+        update_issues = indexed_issues & remote_issues
+        for update_issue in update_issues:
+            # cop out
+            writer.delete_by_term('id',update_issue)
+            item = full_items[update_issue]
+            self.add_issue(writer, item, gh_access_token, config, update=True)
+            count += 1
+
+
+        # Add any issue not in indexed_issues
+        # and in remote_issues
+        add_issues = remote_issues - indexed_issues
+        for add_issue in add_issues:
+            item = full_items[add_issue]
+            self.add_issue(writer, item, gh_access_token, config, update=False)
+            count += 1
 
 
         writer.commit()
         print("Done, updated %d documents in the index" % count)
 
 
+
+    # ------------------------------
+    # Github Markdown Files
+
+    def update_index_markdown(self, gh_access_token, config): 
+        """
+        Update the search index using a collection of 
+        Markdown files from a Github repo.
+        """
+        EXT = '.md'
+
+        # Updated algorithm:
+        # - get set of indexed ids
+        # - get set of remote ids
+        # - drop indexed ids not in remote ids
+        # - index all remote ids
+
+        # Get the set of indexed ids:
+        # ------
+        indexed_ids = set()
+        p = QueryParser("kind", schema=self.ix.schema)
+        q = p.parse("markdown")
+        with self.ix.searcher() as s:
+            results = s.search(q,limit=None)
+            for result in results:
+                indexed_ids.add(result['id'])
+
+        # Get the set of remote ids:
+        # ------
+        # Start with api object
+        g = Github(gh_access_token)
+
+        # Now index all markdown files
+        # in the user-specified repos
+
+        # Start by collecting all the things
+        remote_ids = set()
+        full_items = {}
+
+        # Iterate over each repo 
+        list_of_repos = config['repositories']
+        for r in list_of_repos:
+
+            if '/' not in r:
+                err = "Error: specify org/reponame or user/reponame in list of repos"
+                raise Exception(err)
+
+            this_org, this_repo = re.split('/',r)
+            try:
+                org = g.get_organization(this_org)
+                repo = org.get_repo(this_repo)
+            except:
+                print("Error: could not gain access to repository %s"%(r))
+                continue
+
+
+            # ---------
+            # begin markdown-specific code
+
+            # Get head commit
+            commits = repo.get_commits()
+            try:
+                last = commits[0]
+                sha = last.sha
+            except GithubException:
+                print("Error: could not get commits from repository %s"%(r))
+                continue
+
+            # Get all the docs
+            tree = repo.get_git_tree(sha=sha, recursive=True)
+            docs = tree.raw_data['tree']
+            print("Parsing doc ids from repository %s"%(r))
+
+            for d in docs:
+
+                # For each doc, get the file extension
+                # If it matches EXT, download the file
+                fpath = d['path']
+                _, fname = os.path.split(fpath)
+                _, fext = os.path.splitext(fpath)
+
+                if fext==EXT:
+
+                    key = d['sha']
+                    d['org'] = this_org
+                    d['repo'] = this_repo
+                    value = d
+
+                    # Stash the doc for later
+                    remote_ids.add(key)
+                    full_items[key] = value
+
+        writer = self.ix.writer()
+        count = 0
+
+
+        # Drop any id in indexed_ids
+        # not in remote_ids
+        drop_ids = indexed_ids - remote_ids
+        for drop_id in drop_ids:
+            writer.delete_by_term('id',drop_id)
+
+
+        # Update any id in indexed_ids
+        # and in remote_ids
+        update_ids = indexed_ids & remote_ids
+        for update_id in update_ids:
+            # cop out: just delete and re-add
+            writer.delete_by_term('id',update_id)
+            item = full_items[update_id]
+            self.add_markdown(writer, item, gh_access_token, config, update=True)
+            count += 1
+
+
+        # Add any issue not in indexed_ids
+        # and in remote_ids
+        add_ids = remote_ids - indexed_ids
+        for add_id in add_ids:
+            item = full_items[add_id]
+            self.add_markdown(writer, item, gh_access_token, config, update=False)
+            count += 1
+
+
+        writer.commit()
+        print("Done, updated %d markdown documents in the index" % count)
+
+
+
+    # ------------------------------
+    # Groups.io Emails
+
+
+    #def update_index_markdown(self, gh_access_token, config): 
+
+
+
+
+
     # ---------------------------------
     # Search results bundler
 
@@ -477,11 +814,6 @@ class Search:
             # contains a {% for e in entries %}
             # and then an {{e.score}}
 
-
-            # ------------------
-            # cheseburger
-            # create search results
-
             sr = SearchResult()
             sr.score = r.score
 
@@ -495,37 +827,29 @@ class Search:
 
             sr.id = r['id']
             sr.kind = r['kind']
-            sr.url = r['url']
+
+            sr.created_time = r['created_time']
+            sr.modified_time = r['modified_time']
+            sr.indexed_time = r['indexed_time']
+
             sr.title = r['title']
+            sr.url = r['url']
+
             sr.mimetype = r['mimetype']
 
             sr.owner_email = r['owner_email']
             sr.owner_name = r['owner_name']
 
-            sr.content = r['content']
-
-            
-            # -----------------
-            # github isuses
-            # create search results
-
-            sr = SearchResult()
-            sr.score = r.score
-            sr.url = r['url']
-            sr.title = r['issue_title']
-
             sr.repo_name = r['repo_name']
             sr.repo_url = r['repo_url']
 
             sr.issue_title = r['issue_title']
             sr.issue_url = r['issue_url']
 
-            sr.is_comment = r['is_comment']
+            sr.github_user = r['github_user']
 
             sr.content = r['content']
 
-            # ------------------
-
             highlights = r.highlights('content')
             if not highlights:
                 # just use the first 1,000 words of the document
@@ -533,21 +857,18 @@ class Search:
 
             highlights = self.html_parser.unescape(highlights)
             html = self.markdown(highlights)
+            html = re.sub(r'\n','<br />',html)
             sr.content_highlight = html
 
             search_results.append(sr)
 
         return search_results
 
-        # ------------------
-        # github issues
-        # create search results
-
-
 
 
 
     def search(self, query_list, fields=None):
+
         with self.ix.searcher() as searcher:
             query_string = " ".join(query_list)
             query = None
@@ -558,27 +879,15 @@ class Search:
             elif len(fields) == 2:
                 pass
             else:
-                fields = ['id',
-                          'kind',
-                          'created_time',
-                          'modified_time',
-                          'indexed_time',
-                          'title',
-                          'url',
-                          'mimetype',
-                          'owner_email',
-                          'owner_name',
-                          'repo_name',
-                          'repo_url',
-                          'issue_title',
-                          'issue_url',
-                          'github_user',
+                # If the user does not specify a field,
+                # these are the fields that are actually searched
+                fields = ['title',
                           'content']
             if not query:
                 query = MultifieldParser(fields, schema=self.ix.schema).parse(query_string)
             parsed_query = "%s" % query
             print("query: %s" % parsed_query)
-            results = searcher.search(query, terms=False, scored=True, groupedby="url")
+            results = searcher.search(query, terms=False, scored=True, groupedby="kind")
             search_result = self.create_search_result(results)
 
         return parsed_query, search_result
@@ -589,7 +898,31 @@ class Search:
         return s if len(s) <= l else s[0:l - 3] + '...'
 
     def get_document_total_count(self):
-        return self.ix.searcher().doc_count_all()
+        p = QueryParser("kind", schema=self.ix.schema)
+
+        kind_labels = {
+                "documents" : "gdoc",
+                "markdown" :  "markdown",
+                "issues" :    "issue",
+        }
+        counts = {
+                "documents" : None,
+                "markdown" : None,
+                "issues" : None,
+                "total" : None
+        }
+        for key in kind_labels:
+            kind = kind_labels[key]
+            q = p.parse(kind)
+            with self.ix.searcher() as s:
+                results = s.search(q,limit=None)
+                counts[key] = len(results)
+
+        ## These two should NOT be different, but they are...
+        #counts['total'] = self.ix.searcher().doc_count_all()
+        counts['total'] = counts['documents'] + counts['markdown'] + counts['issues']
+
+        return counts
 
 if __name__ == "__main__":
     search = Search("search_index")

config_centillion.json

@@ -1,7 +1,27 @@
 {
     "repositories" : [
+        "dcppc/project-management",
+        "dcppc/nih-demo-meetings",
+        "dcppc/internal",
+        "dcppc/organize",
+        "dcppc/dcppc-bot",
+        "dcppc/full-stacks",
+        "dcppc/markdown-issues",
+        "dcppc/design-guidelines-discuss",
+        "dcppc/dcppc-deliverables",
+        "dcppc/dcppc-milestones",
+        "dcppc/crosscut-metadata",
+        "dcppc/lucky-penny",
+        "dcppc/dcppc-workshops",
+        "dcppc/metadata-matrix",
+        "dcppc/data-stewards",
+        "dcppc/dcppc-phase1-demos",
+        "dcppc/apis",
         "dcppc/2018-june-workshop",
         "dcppc/2018-july-workshop",
-        "dcppc/data-stewards"
+        "dcppc/2018-august-workshop",
+        "dcppc/2018-september-workshop",
+        "dcppc/design-guidelines",
+        "dcppc/2018-may-workshop"
     ]
 }

config_flask.example.py

@@ -0,0 +1,20 @@
+# Location of index file
+INDEX_DIR = "search_index"
+
+# oauth client deets
+GITHUB_OAUTH_CLIENT_ID = "XXX"
+GITHUB_OAUTH_CLIENT_SECRET = "YYY"
+GITHUB_TOKEN = "ZZZ"
+
+# More information footer: Repository label
+FOOTER_REPO_ORG = "dcppc"
+FOOTER_REPO_NAME = "centillion"
+
+# Toggle to show Whoosh parsed query
+SHOW_PARSED_QUERY=True
+
+TAGLINE = "Search the Data Commons"
+
+# Flask settings
+DEBUG = True
+SECRET_KEY = '42c5a8eda356ca9d9c3ab2d149541e6b91d843fa'

config_flask.py

@@ -1,27 +0,0 @@
-# Path to markdown files
-MARKDOWN_FILES_DIR = "/Users/charles/codes/whoosh/markdown-search/fake-docs/"
-
-# Location of index file
-INDEX_DIR = "search_index"
-
-# Command to use when clicking on filepath in search results
-EDIT_COMMAND = "view"
-
-# Toggle to show Whoosh parsed query
-SHOW_PARSED_QUERY=True
-
-# Toogle to use tags
-USE_TAGS=True
-
-# Optional prefix in a markdown file, e.g. "tags: python search markdown tutorial"
-TAGS_PREFIX=""
-
-# List of tags that should be ignored
-TAGS_TO_IGNORE = "and are what how its not with the"
-
-# Regular expression to select tags, eg tag has to start with alphanumeric followed by at least two alphanumeric or "-" or "."
-TAGS_REGEX = r"\b([A-Za-z0-9][A-Za-z0-9-.]{2,})\b"
-
-# Flask settings
-DEBUG = True
-SECRET_KEY = '42c5a8eda356ca9d9c3ab2d149541e6b91d843fa'

docs/centillion_components.md

@@ -0,0 +1,22 @@
+# Centillion Components
+
+Centillion keeps it simple.
+
+There are two components:
+
+* The `Search` object, which uses whoosh and various
+  APIs (Github, Google Drive) to build and manage
+  the search index. The `Search` object also runs all
+  queries against the search index. (See the
+  [Centillion Whoosh](centillion_whoosh.md) page
+  or the `centillion_search`.py` file
+  for details.)
+
+* Flask app, which uses Jinja templates to present the
+  user with a minimal web frontend that allows them
+  to interact with the search engine. (See the
+  [Centillion Flask](centillion_flask.md) page
+  or the `centillion`.py` file
+  for details.)
+
+

docs/centillion_flask.md

@@ -0,0 +1,30 @@
+# Centillion Flask
+
+## What the flask server does
+
+Flask is a web server framework
+that allows developers to define
+behavior for specific endpoints,
+such as `/hello_world`, or
+<http://localhost:5000/hello_world>
+on a web server running locally.
+
+## Flask server routes
+
+- `/home`
+    - if not logged in, this redirects to a "log into github" landing page (not implemented yet)
+    - if logged in, this redirects to the search route
+
+- `/search`
+    - search template
+
+- `/main_index_update`
+    - update main index, all docs period
+
+- `/control_panel`
+    - this is the control panel, where you can trigger
+      the search index to be re-made
+
+
+
+

docs/centillion_whoosh.md

@@ -0,0 +1,34 @@
+# Centillion Whoosh
+
+The `centillion_search.py` file defines a
+`Search` class that serves as the backend
+for centillion.
+
+## What the Search class does
+
+The `Search` class has two roles:
+- create (and update) the search index
+    - this also requires the `Search` class
+      to define the schema for storing documents
+- run queries against the search index,
+  and package results up for Flask and Jinja
+
+## Search class functions
+
+The `Search` class defines several functions:
+
+- `open_index()` creates the schema
+
+- `add_issue()`, `add_md()`, `add_document()` have three diff method sigs and add diff types
+  of documents to the search index
+
+- `update_all_issues()` or `update_all_md()` or `update_all_documents()` iterates over items
+  and determines whether each item needs to be updated in the search index
+
+- `update_main_index()` - update the entire search index
+    - calls all three update_all methods
+
+- `create_search_results()` - package things up for jinja
+
+- `search()` - run the query, pass results to the jinja-packager
+

docs/index.md

@@ -0,0 +1,84 @@
+# Centillion
+
+**centillion**: a pan-github-markdown-issues-google-docs search engine.
+
+**a centillion**: a very large number consisting of a 1 with 303 zeros after it.
+
+centillion is 3.03 log-times better than the googol.
+
+## What is centillion
+
+Centillion is a search engine built using [whoosh](https://whoosh.readthedocs.io/en/latest/intro.html),
+a Python library for building search engines.
+
+
+We define the types of documents centillion should index,
+what info and how. Centillion then builds and
+updates a search index. That's all done in `centillion_search.py`.
+
+Centillion also provides a simple web frontend for running
+queries against the search index. That's done using a Flask server
+defined in `centillion.py`.
+
+Centillion keeps it simple.
+
+
+## Quickstart
+
+Run centillion with a github access token API key set via
+environment variable:
+
+```
+GITHUB_TOKEN="XXXXXXXX" python centillion.py
+```
+
+This will start a Flask server, and you can view the minimal search engine
+interface in your browser at <http://localhost:5000>.
+
+## Configuration
+
+### Centillion configuration
+
+`config_centillion.json` defines configuration variables
+for centillion - namely, what to index, and how, and where.
+
+### Flask configuration
+
+`config_flask.py` defines configuration variables
+used by flask, which controls the web frontend 
+for centillion.
+
+## Control Panel/Rebuilding Search Index
+
+To rebuild the search engine, visit the control panel route (`/control_panel`),
+for example at <http://localhost:5000/control_panel>.
+
+This allows you to rebuild the search engine index. The search index
+is stored in the `search_index/` directory, and that directory
+can be configured with centillion's configuration file.
+
+The diff search index is faster to build, as it only
+indexes documents that have been added since the last
+new document was added to the search index.
+
+The main search index is slower to build, as it will
+re-index everything.
+
+(Cron scripts? Threaded task that runs hourly?)
+
+## Details
+
+More on the details of how centillion works.
+
+Under the hood, centillion uses flask and whoosh.
+Flask builds and runs the web server.
+Whoosh handles search requests and management
+of the search index.
+
+[Centillion Components](centillion_components.md)
+
+[Centillion Flask](centillion_flask.md)
+
+[Centillion Whoosh](centillion_whoosh.md)
+
+

docs/plans.md

@@ -31,3 +31,4 @@ Stateless
 
 
 
+

docs/standalone.md

@@ -1,4 +1,4 @@
-## work that is done
+## work that is done: standalone
 
 **Stage 1: index folder of markdown files** (done)
 * See [markdown-search](https://git.charlesreid1.com/charlesreid1/markdown-search.git)
@@ -13,7 +13,7 @@
 
 Needs work:
 
-* More appropriate schema
+* <s>More appropriate schema</s>
 * Using more features (weights) plus pandoc filters for schema
 * Sqlalchemy (and hey waddya know safari books has it covered)
 
@@ -25,15 +25,16 @@ Needs work:
 * Main win here is uncovering metadata/linking/presentation issues
 
 Needs work:
-- treat comments and issues as separate objects, fill out separate schema fields
+- <s>treat comments and issues as separate objects, fill out separate schema fields
 - map out and organize how the schema is updated to make it more flexible
-- configuration needs to enable user to specify organization+repos
+- configuration needs to enable user to specify organization+repos</s>
 
 ```plain
 {
-    "to_index" : {
-        "google" : "google-api-python-client",
-        "microsoft" : ["TypeCode","api-guidelines"]
+    "to_index" : [
+        "google/google-api-python-client",
+        "microsoft/TypeCode",
+        "microsoft/api-guielines"
     }
 }
 ```
@@ -48,3 +49,4 @@ Needs work:
 * Use the google drive api (see simple-simon)
 * Main win is more uncovering of metadata issues, identifying
   big-picture issues for centillion
+

docs/workinprogress.md

@@ -0,0 +1,48 @@
+# Components
+
+The components of centillion are as follows:
+- Flask application, which creates a Search object and uses it to search index
+- Search object, which allows you to create/update/search an index
+
+## Routes layout
+
+Centillion flask app routes:
+
+- `/home`
+    - if not logged in, landing page
+    - if logged in, redirect to search
+- `/search`
+- `/main_index_update`
+    - update main index, all docs period
+
+## Functions layout
+
+Centillion Search class functions:
+
+- `open_index()` creates the schema
+
+- `add_issue()`, `add_md()`, `add_document()` have three diff method sigs and add diff types
+  of documents to the search index
+
+- `update_all_issues()` or `update_all_md()` or `update_all_documents()` iterates over items
+  and determines whether each item needs to be updated in the search index
+
+- `update_main_index()` - update the entire search index
+    - calls all three update_all methods
+
+- `create_search_results()` - package things up for jinja
+
+- `search()` - run the query, pass results to the jinja-packager
+
+Nice to have but focus on it later:
+- update diff search index (what's been added since last index time)
+    - max index time
+
+
+## Files layout
+
+Schema definition:
+* include a "kind" or "class" to group objects
+* can provide different searches of different collections
+* eventually can provide user with checkboxes
+

gdrive_util.py

@@ -29,8 +29,7 @@ class GDrive(object):
     ):
         """
         Set up the Google Drive API instance.
-        Factory method: create it and hand it over.
-        Then we're finished.
+        Factory method: create it here, hand it over in get_service().
         """
         self.credentials_file = credentials_file
         self.client_secret_file = client_secret_file
@@ -40,6 +39,9 @@ class GDrive(object):
         self.store = file.Storage(credentials_file)
 
     def get_service(self):
+        """
+        Return an instance of the Google Drive API service.
+        """
 
         creds = self.store.get()
         if not creds or creds.invalid:

install_pandoc.sh

@@ -0,0 +1,19 @@
+#!/bin/bash
+#
+# for ubuntu 
+
+if [ "$(id -u)" != "0" ]; then
+    echo ""
+    echo ""
+    echo "This script should be run as root."
+    echo ""
+    echo ""
+    exit 1;
+fi
+
+OFILE="/tmp/pandoc.deb"
+curl -L https://github.com/jgm/pandoc/releases/download/2.2.2.1/pandoc-2.2.2.1-1-amd64.deb -o ${OFILE}
+dpkg -i ${OFILE}
+rm -f ${OFILE}
+
+

requirements.txt

@@ -9,3 +9,4 @@ PyGithub>=1.39
 pypandoc>=1.4
 requests>=2.19
 pandoc>=1.0
+flask-dance>=1.0.0

static/style.css

@@ -1,3 +1,45 @@
+span.badge {
+    vertical-align: text-bottom;
+}
+
+a.badgelinks, a.badgelinks:hover {
+    color: #fff;
+    text-decoration: none;
+}
+
+div.list-group {
+    border: 1px solid rgba(86,61,124,.2);
+}
+
+li.list-group-item {
+    position: relative;
+    display: block;
+
+    /*padding: 20px 10px;*/
+    margin-bottom: -1px;
+
+    background-color: #f8f8f8;
+    border: 1px solid #ddd;
+}
+
+li.search-group-item {
+    position: relative;
+    display: block;
+
+    padding: 0px;
+    margin-bottom: -1px;
+
+    background-color: #fff;
+    border: 1px solid #ddd;
+}
+
+div.url {
+    background-color: rgba(86,61,124,.15);
+    padding: 8px;
+}
+
+/***************************/
+
 body {
     font-family: sans-serif;
 }
@@ -56,7 +98,7 @@ table {
     overflow: hidden;
 }
 
-td.info, .last-searches {
+.info, .last-searches {
     color: gray;
     font-size: 12px;
     font-family: Arial, serif;

templates/controlpanel.html

@@ -0,0 +1,108 @@
+{% extends "layout.html" %}
+{% block body %}
+
+{% with messages = get_flashed_messages() %}
+{% if messages %}
+<div class="container">
+    <div class="alert alert-success alert-dismissible">
+        <a href="#" class="close" data-dismiss="alert" aria-label="close">&times;</a>
+        <ul class=flashes>
+            {% for message in messages %}
+            <li>{{ message }}</li>
+            {% endfor %}
+        </ul>
+    </div>
+</div>
+{% endif %}
+{% endwith %}
+
+<div class="container">
+
+    <div class="row">
+        <div class="col-md-12">
+            <center>
+                <a href="{{ url_for('search')}}?query=&fields=">
+                <img src="{{ url_for('static', filename='centillion_white.png') }}">
+                </a>
+                {% if config['TAGLINE'] %}
+                    <h2><a href="{{ url_for('search')}}?query=&fields=">
+                        {{config['TAGLINE']}}
+                    </a></h2>
+                {% endif %}
+            </center>
+        </div>
+    </div>
+    
+    {% if config['zzzTAGLINE'] %}
+    <div class="row">
+        <div class="col12sm">
+            <center>
+                <h2><a href="{{ url_for('search')}}?query=&fields=">
+                    {{config['TAGLINE']}}
+                </a></h2>
+            </center>
+        </div>
+    </div>
+    {% endif %}
+
+</div>
+
+<hr />
+
+<div class="container">
+
+    <div class="row">
+
+        {# update main search index #}
+        <div class="panel panel-danger">
+            <div class="panel-heading">
+                <h3 class="panel-title">
+                    Update Main Search Index
+                </h3>
+            </div>
+            <div class="panel-body">
+                <div class="container-fluid">
+                    <div class="row">
+                        <div class="col-md-12">
+                            <p class="panel-text">Re-index <i>every</i> document in the
+                            remote collection in the search index. <b>Warning: this operation may take a while.</b>
+                            <p/> <p>
+                            <a href="{{ url_for('update_index') }}" class="btn btn-large btn-danger">Update Main Index</a>
+                            <p/> 
+                        </div>
+                    </div>
+                </div>
+
+            </div>
+        </div>
+
+        {# update diff search index #}
+        <div class="panel panel-danger">
+            <div class="panel-heading">
+                <h3 class="panel-title">
+                    Update Diff Search Index
+                </h3>
+            </div>
+            <div class="panel-body">
+                <div class="container-fluid">
+                    <div class="row">
+                        <div class="col-md-12">
+                            <p class="panel-text">Diff search index only re-indexes documents created after the last
+                            search index update. <b>Not currently implemented.</b>
+                            <p/> <p>
+                            <a href="#" class="btn btn-large disabled btn-danger">Update Diff Index</a>
+                            <p/> 
+                        </div>
+                    </div>
+                </div>
+
+            </div>
+        </div>
+
+
+    </div>
+
+</div>
+
+{% endblock %}
+

templates/layout.html

@@ -1,10 +1,12 @@
 <!doctype html>
-<title>Markdown Search</title>
-<link rel="stylesheet" type="text/css" href="{{ url_for('static', filename='github-markdown.css') }}">
+<title>Centillion Search Engine</title>
 <link rel="stylesheet" type="text/css" href="{{ url_for('static', filename='style.css') }}">
+<link rel="stylesheet" type="text/css" href="{{ url_for('static', filename='github-markdown.css') }}">
+<link rel="stylesheet" type="text/css" href="{{ url_for('static', filename='bootstrap.min.css') }}">
+
+<script src="{{ url_for('static', filename='jquery.min.js') }}"></script>
+<script src="{{ url_for('static', filename='bootstrap.min.js') }}"></script>
+
 <div>
-    {% for message in get_flashed_messages() %}
-        <div class="flash">{{ message }}</div>
-    {% endfor %}
     {% block body %}{% endblock %}
 </div>

templates/search.html

@@ -1,62 +1,185 @@
 {% extends "layout.html" %}
 {% block body %}
-<h1><a href="{{ url_for('search')}}?query=&fields=">Search directory: {{ config.MARKDOWN_FILES_DIR }}</a></h1>
-<a class="index" href="{{ url_for('update_index')}}">[update index]</a>
-<a class="index" href="{{ url_for('update_index')}}?rebuild=True">[rebuild index]</a>
-<form action="{{ url_for('search') }}" name="search">
-    <input type="text" name="query" value="{{ query }}">
-    <input type="submit" value="search">
-    <a href="{{ url_for('search')}}?query=&fields=">[clear]</a>
-</form>
-<table cellspacing="3">
-    {% if directories %}
-    <tr>
-        <td class="directories-cloud">File directories:&nbsp
+
+
+<div class="container">
+
+    {#
+    banner image
+    #}
+    <div class="row">
+        <div class="col12sm">
+            <center>
+                <a href="{{ url_for('search')}}?query=&fields=">
+                <img src="{{ url_for('static', filename='centillion_white.png') }}">
+                </a>
+                {#
+                need a tag line
+                #}
+                {% if config['TAGLINE'] %}
+                    <h2><a href="{{ url_for('search')}}?query=&fields=">
+                        {{config['TAGLINE']}}
+                    </a></h2>
+                {% endif %}
+            </center>
+        </div>
+    </div>
+
+</div>
+<div class="container">
+    <div class="row">
+
+        <div class="col-xs-12">
+            <center>
+                <form action="{{ url_for('search') }}" name="search">
+                    <input type="text" name="query" value="{{ query }}"> <br />
+                    <button type="submit" style="font-size: 20px; padding: 10px; padding-left: 50px; padding-right: 50px;" 
+                        value="search" class="btn btn-primary">Search</button>
+                    <br />
+                    <a href="{{ url_for('search')}}?query=&fields=">[clear all results]</a>
+                </form>
+            </center>
+        </div>
+    </div>
+</div>
+
+<div class="container">
+    <div class="row">
+
+        {% if directories %}
+        <div class="col-xs-12 info directories-cloud">
+            <b>File directories:</b> 
             {% for d in directories %}
                 <a href="{{url_for('search')}}?query={{d|trim}}&fields=filename">{{d|trim}}</a>
             {% endfor %}
-        </td>
-    </tr>
-    {% endif %}
-    {% if config['SHOW_PARSED_QUERY']%}
-    <tr>
-        <td class="info">Parsed query: {{ parsed_query }}</td>
-    </tr>
-    {% endif %}
-    <tr>
-        <td class="info">FOUND {{ entries | length }} results of {{total}} documents</td>
-    </tr>
+        </div>
+        {% endif %}
 
-    {% for e in entries %}
-    <tr>
-        <td class="search-result">
-            <!--
-                <div class="path"><a href='{{ url_for("open_file")}}?path={{e.path|urlencode}}&query={{query}}&fields={{fields}}'>{{e.path}}</a>score: {{'%d'  % e.score}}</div>
-            -->
-            <div class="url">
-                {% if e.is_comment %}
-                    <b>Comment</b> <a href='{{e.url}}'>(comment link)</a>
-                    on issue <a href='{{e.issue_url}}'>{{e.issue_title}}</a>
-                    in repo <a href='{{e.repo_url}}'>dcppc/{{e.repo_name}}</a>
-                    <br />
-                {% else %}
-                    <b>Issue</b> <a href='{{e.issue_url}}'>{{e.issue_title}}</a>
-                    in repo <a href='{{e.repo_url}}'>dcppc/{{e.repo_name}}</a>
-                    <br />
-                {% endif %}
-                score: {{'%d'  % e.score}}
-            </div>
-            <div class="markdown-body">{{ e.content_highlight|safe}}</div>
-        </td>
-    </tr>
-    {% endfor %}
-</table>
-<div class="last-searches">Last searches: <br/>
-    {% for s in last_searches %}
-        <span><a href="{{url_for('search')}}?{{s}}">{{s}}</a></span>
-    {% endfor %}
+        <ul class="list-group">
+
+            {% if config['SHOW_PARSED_QUERY'] and parsed_query %}
+                <li  class="list-group-item">
+                    <div class="container-fluid">
+                        <div class="row">
+                            <div class="col-xs-12 info">
+                                <b>Parsed query:</b> {{ parsed_query }}
+                            </div>
+                        </div>
+                    </div>
+                </li>
+            {% endif %}
+
+            {% if parsed_query %}
+                <li  class="list-group-item">
+                    <div class="container-fluid">
+                        <div class="row">
+                            <div class="col-xs-12 info">
+                                <b>Found:</b> <span class="badge">{{entries|length}}</span> results 
+                                out of <span class="badge">{{totals["total"]}}</span> total items indexed
+                            </div>
+                        </div>
+                    </div>
+                </li>
+            {% endif %}
+
+            <li  class="list-group-item">
+                    <div class="container-fluid">
+                        <div class="row">
+                            <div class="col-xs-12 info">
+                                <b>Indexing:</b> <span class="badge">{{totals["documents"]}}</span> Google Documents,
+                                <span class="badge">{{totals["issues"]}}</span> Github issues,
+                                <span class="badge">{{totals["markdown"]}}</span> markdown files.
+                            </div>
+                        </div>
+                </div>
+            </li>
+
+        </ul>
+    </div>
 </div>
-<p>
-    More info can be found in the <a href="https://github.com/BernhardWenzel/markdown-search">README.md file</a>
-</p>
+
+<div class="container">
+    <div class="row">
+        <ul class="list-group">
+
+            {% for e in entries %}
+                <li  class="search-group-item">
+
+                    <div class="url">
+                        {% if e.kind=="gdoc" %}
+                            {% if e.mimetype=="document" %}
+                                <b>Google Document:</b>
+                                <a href='{{e.url}}'>{{e.title}}</a>
+                                (Type: {{e.mimetype}}, Owner: {{e.owner_name}}, {{e.owner_email}})
+                            {% else %}
+                                <b>Google Drive:</b>
+                                <a href='{{e.url}}'>{{e.title}}</a>
+                                (Type: {{e.mimetype}}, Owner: {{e.owner_name}}, {{e.owner_email}})
+                            {% endif %}
+
+                        {% elif e.kind=="issue" %}
+                            <b>Github Issue:</b>
+                            <a href='{{e.url}}'>{{e.title}}</a>
+                            {% if e.github_user %}
+                            opened by <a href='https://github.com/{{e.github_user}}'>@{{e.github_user}}</a>
+                            {% endif %}
+                            <br/>
+                            <b>Repository:</b> <a href='{{e.repo_url}}'>{{e.repo_name}}</a>
+
+                        {% elif e.kind=="markdown" %}
+                            <b>Github Markdown:</b>
+                            <a href='{{e.url}}'>{{e.title}}</a>
+                            <br/>
+                            <b>Repository:</b> <a href='{{e.repo_url}}'>{{e.repo_name}}</a>
+
+                        {% else %}
+                            <b>Item:</b> (<a href='{{e.url}}'>link</a>)
+
+                        {% endif %}
+                        <br />
+                        Score: {{'%d'  % e.score}}
+                    </div>
+                    <div class="markdown-body">
+                        {% if e.content_highlight %}
+                            {{ e.content_highlight|safe}}
+                        {% else %}
+                        <p>(A preview of this document is not available.)</p>
+                        {% endif %}
+                    </div>
+
+                </li>
+            {% endfor %}
+        </ul>
+
+    </div>
+</div>
+
+<div class="container">
+    <div class="row">
+        <ul class="list-group">
+
+            {% if config['FOOTER_REPO_NAME'] %}
+                {% if config['FOOTER_REPO_ORG'] %}
+
+                    <li  class="list-group-item">
+                        <div class="container-fluid">
+                            <div class="row">
+                                <div class="col-xs-12 info">
+                                    More information about {{config['FOOTER_REPO_NAME']}} can be found
+                                    in the <a href="https://github.com/{{config['FOOTER_REPO_ORG']}}/{{config['FOOTER_REPO_NAME']}}">{{config['FOOTER_REPO_ORG']}}/{{config['FOOTER_REPO_NAME']}}</a>
+                                    repository on Github.
+                                </div>
+                            </div>
+                        </div>
+                    </li>
+
+                {% endif %}
+            {% endif %}
+
+        </ul>
+
+    </div>
+</div>
+
 {% endblock %}
+