Triggering a change for the event is useless, but in case of scheduling
or unscheduling contributions (in conferences) we need to record this
change in order to update their time.
Those are not necessary, since any time a creation touches a parent
object (e.g. new timetable entry extending the event duration) we
already trigger an explicit change for that anyway; and in any other
case the parent change is not needed.
While simplify is still very spammy in case of large queues, at least
the actual data dumping for the export itself no longer spams queries.
This is especially crucial in the Citadel plugin where we have massive
parallelism which can easily exhaust the DB connection pool if we get
queries there.
For the same reason, the citadel ID mapping is now queried eagerly
so we don't need to try creating and then delete+update if we realize
there's already an existing citadel mapping, as we can simply check
before even sending something and then convert the creation to an
update.
This resulted in cascades to affect deleted attachments linked to an
event and thus failing runs e.g. after a protection change if it
cascaded to an already-deleted attachment.
Also added a check I forgot in #135
* Enable CI on search branch
* LiveSync: Clean titles in verbose iterator
* LiveSync: Refactor initial export batching
And make the batch size configurable via commandline option
* LiveSync/Citadel: Support Citadel as the sync backend
Co-authored-by: penelope <penelope@fnal.gov>
* LiveSync/Citadel: Upgrade to python3
* LiveSync/Citadel: Update backend and refactor schemas
* LiveSync/Citadel: Add placeholders and filters/aggregations
* LiveSync/Citadel: Add range filters
* LiveSync/Citadel: Update the schemas and initial indexing
* LiveSync/Citadel: Refactor tests
* LiveSync/Citadel: Add status force list to upload retries
* LiveSync/Citadel: Update clean_old_entries filter
* LiveSync/Citadel: Update file headers
* LiveSync/Citadel: Add allowed methods in citadel retry
Commit more often in batches
* Citadel: Rename livesync_citadel to citadel
It's no longer just livesync
* Citadel: Move metadata to setup.cfg
* Citadel: Remove tika import
* Citadel: Mock things using the pytest way
* Citadel: Run pyupgrade
* Citadel: Run tests in CI
* Citadel: Use 'search' plugin category
* Citadel: Improve settings form
* Citadel: Fix model and alembic revision
* Citadel: Make livesync backend unique
* Citadel: Fix is_group error
* Citadel: Include schema attribute in the record context
* Citadel: Exclude content from Attachment schema
* Citadel: Use cached category path from context
* Citadel: Fix passing category tree cache to schemas
* Citadel: Remove unused imports
* Citadel: Be more strict with category cache
If the cache exists, we can expect ALL categories to be in there
* Citadel: Fix duplicate _get_identifiers call
* Citadel: Simplify record uploading
* ci: Allow `search` as valid upstream branch
* Add support for attachment file uploading
* Update file id mapping
* Update cli to a unique backend
* Merge file column migration with previous revision
* Refactor citadel and remove obsolete parts
* Citadel: Cache search_owner_role setting
* Citadel: Dump before parallelization
* Citadel: Do not retry on 500, use lower delays
* Remove change_type from upload_record
* Define a maximum number of pages
ES limits results up to 10,000 hits
* Add missing linebreak
* Citadel: Include category_id in search data
* Convert the aggregations to an object
* Raise request exceptions
* Move query formatting methods to util
* Remove search owner role setting
* Move the object type to filters
* Fix optional string based matches
Update tests
* Citadel: Include start/end dt for subcontribs
* Citadel: Add tests for schemas
* LiveSync: Pass force flag to backends
* Citadel: Skip already-exported entries unless forced
Like this resuming a failed initial export is as easy as just
running the command again.
* LiveSync: eager-load subcontrib timetable entry
* Citadel: Remove obsolete context
* Citadel: Do not upload empty files
* LiveSync: Fix changing root category protection mode
* Citadel: Fix access logic for empty ACLs
Also add unit tests for this
* Citadel: Add contrib/subcontrib duration to schemas
* LiveSync: Add verbose mode
* Remove record dumping if it's being deleted
* Simplify run_initial_export
* Citadel: Print change type in verbose mode
* Citadel: Ignore create-and-delete changes
* LiveSync: Fix tests
* LiveSync: Move citadel-specific code to citadel plugin
* Citadel: Make id mapping entries unique
* Citadel: Refactor http logic
* Citadel: Use 1:1 relationship for id mapping
* Citadel: Use more sensible argument order in id mapping
* LiveSync: Correctly handle (sub) contribution changes
- track contribution time changes
- cascade contribution changes to subcontributions
* Simplify livesync uploader entries
* LiveSync: Improve naming
* LiveSync: Fix attachment handling
Also fix cascading of contribution changes during simplification step
* Citadel: Fix enum display in error message
* LiveSync: Cascade creation to children
This is necessary e.g. when cloning events because in that case we only
have the event creation record but nothing for the elements inside that
event.
To avoid sending the same thing twice (e.g. a contribution) chunking now
only happens on the simplified change level, not on the record level, to
ensure all recorded changes are taken into account when simplifying them
to creation/updates/deletions.
* LiveSync: Allow deferring initial export done flag
In case of Citadel we want to require a file export as well before
starting to process the queue to avoid cases where we are still doing
an initial file upload but queue runs start happening and we would
end up sending old files as well.
* Citadel: Upload files after a queue run
* Refactor checks whether queue runs are possible
Also add an `indico citadel reset` command to throw away all local
citadel data
* Citadel: Re-upload attachments whose file changed
* LiveSync: Fix title in initial export progress
* Citadel: Make max file size configurable
* Check if a livesync plugins is properly configured
And use it to check whether citadel has url and token set
* LiveSync/Debug: Fix errors and use schemas
* Citadel: Refactor verbose printing
* LiveSync: Remove unnecessary default value
* Update the search API to support multiple types
* Move reset cli from citadel to livesync core
* Citadel: Rename model/table
* Citadel: Remove unnecessary timestamp column
* Citadel: Remove some overly verbose comments
* Citadel: Commit explicitly outside create()
* Citadel: Do not send None inside _data
* LiveSync: Fix handling notes
* Citadel: Integrate placeholders with Indico (#117)
* LiveSync: Fix query spam in user.is_system check
* Citadel: Use new result schemas
* Update search schemas
* Citadel: Various improvements/fixes
* Citadel: Integrate filters with Indico & update placeholders (#119)
* Move dump schemas from core to livesync
* Use type_format only in citadel code
* Citadel: Strip HTML from descriptions/notes
* Citadel: Show message if some fails failed to upload
* Add category id filter
* Citadel: Add some debug logging during file uploads
* Citadel: Allow filtering by category name
* Update title and type placeholders
* Citadel: Add -f as --force shortcut
* Fix quote dividers in a multi placeholder query
* Use all worker threads all the time
* Close responses from uploads
Hopefully this fixes running out of fds after ~240k uploads
* Add some more debug logging
* Fix incorrect output
* Rename type placeholder
* Supported quotes for exact matches
* LiveSync: Fix cascading issues
- do not cascade implicit changes to deleted attachments
- correctly cascade changes involving session-linked objects
* Improve error handling
* Citadel: Do not enable queue runs after max-size usage
* LiveSync: Add setting to disable queue runs
* LiveSync: Check category blacklist in initial export
* LiveSync: Add schema tests
* LiveSync: Handle and cascade location changes
* Move query formatting outside the f-string
* Refactor the search query parser
Ensures the keyword positioning isn't changed and correctly
sanitizes placeholder content.
* Escape whitelisted placeholders
Co-authored-by: Adrian <adrian@planetcoding.net>
* Strip the result query
* Remove placeholder operator
* Add default operator
* Update tests
* Add support for sort options (#123)
* Add support for sort options
* Rename SearchFilter to SearchOption
* Rename sort option keys
* Fix error with lazy strings in filter labels
* Revert "Enable CI on search branch"
This reverts commit 2ec3f650eed02cd6c72ea336fa92df6927b8da39.
Co-authored-by: Pedro Lourenço <pedro.lourenco@cern.ch>
Co-authored-by: Michal Kolodziejski <michal.kolodziejski@cern.ch>
Co-authored-by: Penelope Constanta <penelope@fnal.gov>
