rhodecode-enterprise-ce Files · rhodecode/lib/index/whoosh_fallback_schema.py

goto-switcher: optimized performance and query capabilities....

goto-switcher: optimized performance and query capabilities. - Previous implementation had on significant bug. The use of LIMIT 20 was limiting results BEFORE auth checks. In case of large ammount of similarly named repositories user didn't had access too, the result goto search was empty. This was becuase first 20 items fetched didn't pass permission checks and final auth list was empty. To fix this we now use proper filtering for auth using SQL. It means we first check user allowed repositories, and add this as a filter so end result from database is already to only the accessible repositories.

marcink - - Load All Authors

File last commit:

r1271:47a44c03 default


                r2038:2bdf9d4d

default

Download file

             whoosh_fallback_schema.py
        
                    75 lines
            
             | 2.8 KiB
            
                | text/x-python
            
             |
                PythonLexer
            
             / rhodecode / lib / index / whoosh_fallback_schema.py
          
                    History
                
                 |
                  Source
                 | Raw
                 |Copy content
                 |Copy permalink

        marcink
    
project: added all source files and assets

              r1
            
      # -*- coding: utf-8 -*-

        marcink
    
license: updated copyright year to 2017

              r1271
            
      # Copyright (C) 2012-2017 RhodeCode GmbH

        marcink
    
project: added all source files and assets

              r1
            
      #

      # This program is free software: you can redistribute it and/or modify

      # it under the terms of the GNU Affero General Public License, version 3

      # (only), as published by the Free Software Foundation.

      #

      # This program is distributed in the hope that it will be useful,

      # but WITHOUT ANY WARRANTY; without even the implied warranty of

      # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

      # GNU General Public License for more details.

      #

      # You should have received a copy of the GNU Affero General Public License

      # along with this program.  If not, see <http://www.gnu.org/licenses/>.

      #

      # This program is dual-licensed. If you wish to learn more about the

      # RhodeCode Enterprise Edition, including its added features, Support services,

      # and proprietary license terms, please see https://rhodecode.com/licenses/

      """

      Whoosh fallback schema for RhodeCode in case rhodecode_tools defined one is

      not available

      """

      from __future__ import absolute_import

      from whoosh.analysis import RegexTokenizer, LowercaseFilter

      from whoosh.formats import Characters

      from whoosh.fields import (

          TEXT, ID, STORED, NUMERIC, BOOLEAN, Schema, FieldType, DATETIME)

      # CUSTOM ANALYZER wordsplit + lowercase filter for case insensitive search

      ANALYZER = RegexTokenizer(expression=r"\w+") | LowercaseFilter()

      # FILE INDEX SCHEMA DEFINITION

      FILE_INDEX_NAME = 'FILE_INDEX'

      FILE_SCHEMA = Schema(

          fileid=ID(unique=True),  # Path

          repository=ID(stored=True),

          repository_id=NUMERIC(unique=True, stored=True),  # Numeric id of repo

          repo_name=TEXT(stored=True),

          owner=TEXT(),

          path=TEXT(stored=True),

          content=FieldType(format=Characters(), analyzer=ANALYZER,

                            scorable=True, stored=True),

          modtime=STORED(),

          md5=STORED(),

          extension=ID(stored=True),

          commit_id=TEXT(stored=True),

        marcink
    
whoosh: fallback schema uses now bigger ints, fixes #4035

              r413
            
          size=NUMERIC(int, 64, signed=False, stored=True),

        marcink
    
project: added all source files and assets

              r1
            
          mimetype=TEXT(stored=True),

        marcink
    
whoosh: fallback schema uses now bigger ints, fixes #4035

              r413
            
          lines=NUMERIC(int, 64, signed=False, stored=True),

        marcink
    
project: added all source files and assets

              r1
            
      )

      # COMMIT INDEX SCHEMA

      COMMIT_INDEX_NAME = 'COMMIT_INDEX'

      COMMIT_SCHEMA = Schema(

          commit_id=ID(unique=True, stored=True),

          repository=ID(unique=True, stored=True),

          repository_id=NUMERIC(unique=True, stored=True),

          commit_idx=NUMERIC(stored=True, sortable=True),

          commit_idx_sort=ID(),

        marcink
    
whoosh: fallback schema uses now bigger ints, fixes #4035

              r413
            
          date=NUMERIC(int, 64, signed=False, stored=True, sortable=True),

        marcink
    
project: added all source files and assets

              r1
            
          owner=TEXT(stored=True),

          author=TEXT(stored=True),

          message=FieldType(format=Characters(), analyzer=ANALYZER,

                            scorable=True, stored=True),

          parents=TEXT(stored=True),

          added=TEXT(stored=True),  # space separated names of added files

          removed=TEXT(stored=True),  # space separated names of removed files

          changed=TEXT(stored=True),  # space separated names of changed files

      )

	Site-wide shortcuts
/	Use quick search box
g h	Goto home page
g g	Goto my private gists page
g G	Goto my public gists page
g 0-9	Goto bookmarked items from 0-9
n r	New repository page
n g	New gist page

	Repositories
g s	Goto summary page
g c	Goto changelog page
g f	Goto files page
g F	Goto files page with file search activated
g p	Goto pull requests page
g o	Goto repository settings
g O	Goto repository access permissions settings
t s	Toggle sidebar on some pages

marcink project: added all source files and assets	r1	# -- coding: utf-8 --

marcink license: updated copyright year to 2017	r1271	# Copyright (C) 2012-2017 RhodeCode GmbH
marcink project: added all source files and assets	r1	#
		# This program is free software: you can redistribute it and/or modify
		# it under the terms of the GNU Affero General Public License, version 3
		# (only), as published by the Free Software Foundation.
		#
		# This program is distributed in the hope that it will be useful,
		# but WITHOUT ANY WARRANTY; without even the implied warranty of
		# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
		# GNU General Public License for more details.
		#
		# You should have received a copy of the GNU Affero General Public License
		# along with this program. If not, see <http://www.gnu.org/licenses/>.
		#
		# This program is dual-licensed. If you wish to learn more about the
		# RhodeCode Enterprise Edition, including its added features, Support services,
		# and proprietary license terms, please see https://rhodecode.com/licenses/

		"""
		Whoosh fallback schema for RhodeCode in case rhodecode_tools defined one is
		not available
		"""

		from __future__ import absolute_import

		from whoosh.analysis import RegexTokenizer, LowercaseFilter
		from whoosh.formats import Characters
		from whoosh.fields import (
		TEXT, ID, STORED, NUMERIC, BOOLEAN, Schema, FieldType, DATETIME)

		# CUSTOM ANALYZER wordsplit + lowercase filter for case insensitive search
		ANALYZER = RegexTokenizer(expression=r"\w+") \| LowercaseFilter()

		# FILE INDEX SCHEMA DEFINITION
		FILE_INDEX_NAME = 'FILE_INDEX'
		FILE_SCHEMA = Schema(
		fileid=ID(unique=True), # Path
		repository=ID(stored=True),
		repository_id=NUMERIC(unique=True, stored=True), # Numeric id of repo
		repo_name=TEXT(stored=True),
		owner=TEXT(),
		path=TEXT(stored=True),
		content=FieldType(format=Characters(), analyzer=ANALYZER,
		scorable=True, stored=True),
		modtime=STORED(),
		md5=STORED(),
		extension=ID(stored=True),
		commit_id=TEXT(stored=True),

marcink whoosh: fallback schema uses now bigger ints, fixes #4035	r413	size=NUMERIC(int, 64, signed=False, stored=True),
marcink project: added all source files and assets	r1	mimetype=TEXT(stored=True),
marcink whoosh: fallback schema uses now bigger ints, fixes #4035	r413	lines=NUMERIC(int, 64, signed=False, stored=True),
marcink project: added all source files and assets	r1	)


		# COMMIT INDEX SCHEMA
		COMMIT_INDEX_NAME = 'COMMIT_INDEX'
		COMMIT_SCHEMA = Schema(
		commit_id=ID(unique=True, stored=True),
		repository=ID(unique=True, stored=True),
		repository_id=NUMERIC(unique=True, stored=True),
		commit_idx=NUMERIC(stored=True, sortable=True),
		commit_idx_sort=ID(),
marcink whoosh: fallback schema uses now bigger ints, fixes #4035	r413	date=NUMERIC(int, 64, signed=False, stored=True, sortable=True),
marcink project: added all source files and assets	r1	owner=TEXT(stored=True),
		author=TEXT(stored=True),
		message=FieldType(format=Characters(), analyzer=ANALYZER,
		scorable=True, stored=True),
		parents=TEXT(stored=True),
		added=TEXT(stored=True), # space separated names of added files
		removed=TEXT(stored=True), # space separated names of removed files
		changed=TEXT(stored=True), # space separated names of changed files
		)