Adrian ec3d167175
Add new search infrastructure (#125)
* Enable CI on search branch

* LiveSync: Clean titles in verbose iterator

* LiveSync: Refactor initial export batching

And make the batch size configurable via commandline option

* LiveSync/Citadel: Support Citadel as the sync backend

Co-authored-by: penelope <penelope@fnal.gov>

* LiveSync/Citadel: Upgrade to python3

* LiveSync/Citadel: Update backend and refactor schemas

* LiveSync/Citadel: Add placeholders and filters/aggregations

* LiveSync/Citadel: Add range filters

* LiveSync/Citadel: Update the schemas and initial indexing

* LiveSync/Citadel: Refactor tests

* LiveSync/Citadel: Add status force list to upload retries

* LiveSync/Citadel: Update clean_old_entries filter

* LiveSync/Citadel: Update file headers

* LiveSync/Citadel: Add allowed methods in citadel retry

Commit more often in batches

* Citadel: Rename livesync_citadel to citadel

It's no longer just livesync

* Citadel: Move metadata to setup.cfg

* Citadel: Remove tika import

* Citadel: Mock things using the pytest way

* Citadel: Run pyupgrade

* Citadel: Run tests in CI

* Citadel: Use 'search' plugin category

* Citadel: Improve settings form

* Citadel: Fix model and alembic revision

* Citadel: Make livesync backend unique

* Citadel: Fix is_group error

* Citadel: Include schema attribute in the record context

* Citadel: Exclude content from Attachment schema

* Citadel: Use cached category path from context

* Citadel: Fix passing category tree cache to schemas

* Citadel: Remove unused imports

* Citadel: Be more strict with category cache

If the cache exists, we can expect ALL categories to be in there

* Citadel: Fix duplicate _get_identifiers call

* Citadel: Simplify record uploading

* ci: Allow `search` as valid upstream branch

* Add support for attachment file uploading

* Update file id mapping

* Update cli to a unique backend

* Merge file column migration with previous revision

* Refactor citadel and remove obsolete parts

* Citadel: Cache search_owner_role setting

* Citadel: Dump before parallelization

* Citadel: Do not retry on 500, use lower delays

* Remove change_type from upload_record

* Define a maximum number of pages

ES limits results up to 10,000 hits

* Add missing linebreak

* Citadel: Include category_id in search data

* Convert the aggregations to an object

* Raise request exceptions

* Move query formatting methods to util

* Remove search owner role setting

* Move the object type to filters

* Fix optional string based matches

Update tests

* Citadel: Include start/end dt for subcontribs

* Citadel: Add tests for schemas

* LiveSync: Pass force flag to backends

* Citadel: Skip already-exported entries unless forced

Like this resuming a failed initial export is as easy as just
running the command again.

* LiveSync: eager-load subcontrib timetable entry

* Citadel: Remove obsolete context

* Citadel: Do not upload empty files

* LiveSync: Fix changing root category protection mode

* Citadel: Fix access logic for empty ACLs

Also add unit tests for this

* Citadel: Add contrib/subcontrib duration to schemas

* LiveSync: Add verbose mode

* Remove record dumping if it's being deleted

* Simplify run_initial_export

* Citadel: Print change type in verbose mode

* Citadel: Ignore create-and-delete changes

* LiveSync: Fix tests

* LiveSync: Move citadel-specific code to citadel plugin

* Citadel: Make id mapping entries unique

* Citadel: Refactor http logic

* Citadel: Use 1:1 relationship for id mapping

* Citadel: Use more sensible argument order in id mapping

* LiveSync: Correctly handle (sub) contribution changes

- track contribution time changes
- cascade contribution changes to subcontributions

* Simplify livesync uploader entries

* LiveSync: Improve naming

* LiveSync: Fix attachment handling

Also fix cascading of contribution changes during simplification step

* Citadel: Fix enum display in error message

* LiveSync: Cascade creation to children

This is necessary e.g. when cloning events because in that case we only
have the event creation record but nothing for the elements inside that
event.

To avoid sending the same thing twice (e.g. a contribution) chunking now
only happens on the simplified change level, not on the record level, to
ensure all recorded changes are taken into account when simplifying them
to creation/updates/deletions.

* LiveSync: Allow deferring initial export done flag

In case of Citadel we want to require a file export as well before
starting to process the queue to avoid cases where we are still doing
an initial file upload but queue runs start happening and we would
end up sending old files as well.

* Citadel: Upload files after a queue run

* Refactor checks whether queue runs are possible

Also add an `indico citadel reset` command to throw away all local
citadel data

* Citadel: Re-upload attachments whose file changed

* LiveSync: Fix title in initial export progress

* Citadel: Make max file size configurable

* Check if a livesync plugins is properly configured

And use it to check whether citadel has url and token set

* LiveSync/Debug: Fix errors and use schemas

* Citadel: Refactor verbose printing

* LiveSync: Remove unnecessary default value

* Update the search API to support multiple types

* Move reset cli from citadel to livesync core

* Citadel: Rename model/table

* Citadel: Remove unnecessary timestamp column

* Citadel: Remove some overly verbose comments

* Citadel: Commit explicitly outside create()

* Citadel: Do not send None inside _data

* LiveSync: Fix handling notes

* Citadel: Integrate placeholders with Indico (#117)

* LiveSync: Fix query spam in user.is_system check

* Citadel: Use new result schemas

* Update search schemas

* Citadel: Various improvements/fixes

* Citadel: Integrate filters with Indico & update placeholders (#119)

* Move dump schemas from core to livesync

* Use type_format only in citadel code

* Citadel: Strip HTML from descriptions/notes

* Citadel: Show message if some fails failed to upload

* Add category id filter

* Citadel: Add some debug logging during file uploads

* Citadel: Allow filtering by category name

* Update title and type placeholders

* Citadel: Add -f as --force shortcut

* Fix quote dividers in a multi placeholder query

* Use all worker threads all the time

* Close responses from uploads

Hopefully this fixes running out of fds after ~240k uploads

* Add some more debug logging

* Fix incorrect output

* Rename type placeholder

* Supported quotes for exact matches

* LiveSync: Fix cascading issues

- do not cascade implicit changes to deleted attachments
- correctly cascade changes involving session-linked objects

* Improve error handling

* Citadel: Do not enable queue runs after max-size usage

* LiveSync: Add setting to disable queue runs

* LiveSync: Check category blacklist in initial export

* LiveSync: Add schema tests

* LiveSync: Handle and cascade location changes

* Move query formatting outside the f-string

* Refactor the search query parser

Ensures the keyword positioning isn't changed and correctly
sanitizes placeholder content.

* Escape whitelisted placeholders

Co-authored-by: Adrian <adrian@planetcoding.net>

* Strip the result query

* Remove placeholder operator

* Add default operator

* Update tests

* Add support for sort options (#123)

* Add support for sort options
* Rename SearchFilter to SearchOption
* Rename sort option keys

* Fix error with lazy strings in filter labels

* Revert "Enable CI on search branch"

This reverts commit 2ec3f650eed02cd6c72ea336fa92df6927b8da39.

Co-authored-by: Pedro Lourenço <pedro.lourenco@cern.ch>
Co-authored-by: Michal Kolodziejski <michal.kolodziejski@cern.ch>
Co-authored-by: Penelope Constanta <penelope@fnal.gov>
2021-05-07 14:47:59 +00:00

301 lines
8.1 KiB
Python

# This file is part of the Indico plugins.
# Copyright (C) 2002 - 2021 CERN
#
# The Indico plugins are free software; you can redistribute
# them and/or modify them under the terms of the MIT License;
# see the LICENSE file for more details.
from flask import g
from indico.core.db.sqlalchemy import PyIntEnum, UTCDateTime, db
from indico.modules.attachments.models.attachments import Attachment
from indico.modules.categories.models.categories import Category
from indico.util.date_time import now_utc
from indico.util.enum import IndicoEnum
from indico.util.string import format_repr
from indico_livesync.models.agents import LiveSyncAgent
from indico_livesync.util import obj_deref
class ChangeType(int, IndicoEnum):
created = 1
deleted = 2
moved = 3
data_changed = 4
protection_changed = 5
location_changed = 6
class EntryType(int, IndicoEnum):
category = 1
event = 2
contribution = 3
subcontribution = 4
session = 5
note = 6
attachment = 7
_column_for_types = {
EntryType.category: 'category_id',
EntryType.event: 'event_id',
EntryType.contribution: 'contribution_id',
EntryType.subcontribution: 'subcontribution_id',
EntryType.session: 'session_id',
EntryType.note: 'note_id',
EntryType.attachment: 'attachment_id',
}
def _make_checks():
available_columns = set(_column_for_types.values())
for link_type in EntryType:
required_col = _column_for_types[link_type]
forbidden_cols = available_columns - {required_col}
criteria = [f'{col} IS NULL' for col in sorted(forbidden_cols)]
criteria += [f'{required_col} IS NOT NULL']
condition = 'type != {} OR ({})'.format(link_type, ' AND '.join(criteria))
yield db.CheckConstraint(condition, f'valid_{link_type.name}_entry')
class LiveSyncQueueEntry(db.Model):
__tablename__ = 'queues'
__table_args__ = tuple(_make_checks()) + ({'schema': 'plugin_livesync'},)
#: Entry ID
id = db.Column(
db.Integer,
primary_key=True
)
#: ID of the agent this entry belongs to
agent_id = db.Column(
db.Integer,
db.ForeignKey('plugin_livesync.agents.id'),
nullable=False,
index=True
)
#: Timestamp of the change
timestamp = db.Column(
UTCDateTime,
nullable=False,
default=now_utc
)
#: if this record has already been processed
processed = db.Column(
db.Boolean,
nullable=False,
default=False
)
#: the change type, a :class:`ChangeType`
change = db.Column(
PyIntEnum(ChangeType),
nullable=False
)
#: The type of the changed object
type = db.Column(
PyIntEnum(EntryType),
nullable=False
)
#: The ID of the changed category
category_id = db.Column(
db.Integer,
db.ForeignKey('categories.categories.id'),
index=True,
nullable=True
)
#: ID of the changed event
event_id = db.Column(
db.Integer,
db.ForeignKey('events.events.id'),
index=True,
nullable=True
)
#: ID of the changed contribution
contrib_id = db.Column(
'contribution_id',
db.Integer,
db.ForeignKey('events.contributions.id'),
index=True,
nullable=True
)
#: ID of the changed session
session_id = db.Column(
'session_id',
db.Integer,
db.ForeignKey('events.sessions.id'),
index=True,
nullable=True
)
#: ID of the changed subcontribution
subcontrib_id = db.Column(
'subcontribution_id',
db.Integer,
db.ForeignKey('events.subcontributions.id'),
index=True,
nullable=True
)
#: ID of the changed note
note_id = db.Column(
'note_id',
db.Integer,
db.ForeignKey('events.notes.id'),
index=True,
nullable=True
)
#: ID of the changed attachment
attachment_id = db.Column(
'attachment_id',
db.Integer,
db.ForeignKey('attachments.attachments.id'),
index=True,
nullable=True
)
#: The associated :class:LiveSyncAgent
agent = db.relationship(
'LiveSyncAgent',
backref=db.backref('queue', cascade='all, delete-orphan', lazy='dynamic')
)
category = db.relationship(
'Category',
lazy=True,
backref=db.backref(
'livesync_queue_entries',
cascade='all, delete-orphan',
lazy=True
)
)
event = db.relationship(
'Event',
lazy=True,
backref=db.backref(
'livesync_queue_entries',
cascade='all, delete-orphan',
lazy=True
)
)
session = db.relationship(
'Session',
lazy=False,
backref=db.backref(
'livesync_queue_entries',
cascade='all, delete-orphan',
lazy='dynamic'
)
)
contribution = db.relationship(
'Contribution',
lazy=False,
backref=db.backref(
'livesync_queue_entries',
cascade='all, delete-orphan',
lazy='dynamic'
)
)
subcontribution = db.relationship(
'SubContribution',
lazy=False,
backref=db.backref(
'livesync_queue_entries',
cascade='all, delete-orphan',
lazy='dynamic'
)
)
note = db.relationship(
'EventNote',
lazy=False,
backref=db.backref(
'livesync_queue_entries',
cascade='all, delete-orphan',
lazy='dynamic'
)
)
attachment = db.relationship(
'Attachment',
lazy=False,
backref=db.backref(
'livesync_queue_entries',
cascade='all, delete-orphan',
lazy='dynamic'
)
)
@property
def object(self):
"""Return the changed object."""
if self.type == EntryType.category:
return self.category
elif self.type == EntryType.event:
return self.event
elif self.type == EntryType.session:
return self.session
elif self.type == EntryType.contribution:
return self.contribution
elif self.type == EntryType.subcontribution:
return self.subcontribution
elif self.type == EntryType.note:
return self.note
elif self.type == EntryType.attachment:
return self.attachment
def __repr__(self):
return format_repr(self, 'id', 'agent_id', 'change', 'type',
category_id=None, event_id=None, session_id=None, contrib_id=None, subcontrib_id=None,
note_id=None, attachment_id=None)
@classmethod
def create(cls, changes, ref, excluded_categories=set()):
"""Create a new change in all queues.
:param changes: the change types, an iterable containing
:class:`ChangeType`
:param ref: the object reference (returned by `obj_ref`)
of the changed object
:param excluded_categories: set of categories (IDs) whose items
will not be tracked
"""
ref = dict(ref)
obj = obj_deref(ref)
if isinstance(obj, Category):
if any(c.id in excluded_categories for c in obj.chain_query):
return
else:
event = obj.folder.event if isinstance(obj, Attachment) else obj.event
if event.category not in g.setdefault('livesync_excluded_categories_checked', {}):
g.livesync_excluded_categories_checked[event.category] = excluded_categories & set(event.category_chain)
if g.livesync_excluded_categories_checked[event.category]:
return
try:
agents = g.livesync_agents
except AttributeError:
agents = g.livesync_agents = LiveSyncAgent.query.all()
for change in changes:
for agent in agents:
entry = cls(agent=agent, change=change, **ref)
db.session.add(entry)
db.session.flush()