75 Commits

Author SHA1 Message Date
Adrian Moennich
0857d002c3 Fix flake8 errors 2020-11-12 14:47:29 +01:00
Adrian Moennich
60d02fe594 Fix modernize/pyupgrade screwup
modernize converted it to six.reraise() and somehow pyupgrade converted
it to `raise None.with_traceback()` afterwards...
2020-11-12 14:47:29 +01:00
Adrian Moennich
a2d2f1d723 Fix encode/decode usage 2020-11-12 14:47:29 +01:00
Indico Team
01406b3d2a Fix empty lines after file headers 2020-11-12 14:47:29 +01:00
Adrian Moennich
9c994fba61 Stop using return_ascii decorator 2020-11-12 14:47:29 +01:00
Indico Team
f2755b4be4 Run modernize, pyupgrade, autoflake and isort 2020-11-12 14:47:29 +01:00
Adrian Moennich
9d1c6268a0 Bump versions to 3.0-dev 2020-11-12 14:46:40 +01:00
Adrian Moennich
c3499f1813 Storage/S3: Use high-level API supporting chunks
Otherwise it's impossible to store files larger than 5GB
2020-10-30 13:51:17 +01:00
Adrian Moennich
d3a416679c Storage/S3: Do not require secret if readonly 2020-10-21 21:46:30 +02:00
Adrian Moennich
171cc04ce0 Bump plugin versions to 2.3 2020-09-14 17:01:19 +02:00
Adrian Moennich
2e53c7040b Storage/S3: Fix copying File files 2020-09-01 08:17:31 +02:00
Indico Team
2a73af5957 Update *.pot files 2020-08-19 20:41:59 +02:00
Adrian Moennich
8d64194619 Storage/S3: Fix handling non-ascii file names 2020-07-29 12:42:04 +02:00
Indico Team
43c703e1b4 Happy new year 2020 🎆 2020-01-16 11:32:21 +01:00
Adrian Moennich
5a8baf3dfe Storage/S3: Fix deleting objects 2019-11-08 10:42:33 +01:00
Adrian Moennich
f3f5fdfc1c Storage/S3: Bump version 2019-10-22 11:13:17 +02:00
Adrian Moennich
59bfc1e993 Storage/S3: Add readme 2019-10-22 11:13:17 +02:00
Adrian Moennich
7fc336f15f Storage/S3: Support 'on' for bool options 2019-10-22 11:13:17 +02:00
Adrian Moennich
4e42bce937 Storage/S3: Support nginx for proxying downloads 2019-10-22 11:13:17 +02:00
Indico Team
4b39714dbf Update license headers (and add missing) 2019-05-08 15:41:55 +02:00
Adrian Moennich
04219821e9 Update package licenses 2019-05-08 15:41:55 +02:00
Adrian Moennich
4a9d59340e Storage/S3: Bump version 2018-11-26 14:50:37 +01:00
Adrian Moennich
3a32e524f8 Storage/S3: Truncate bucket hash if necessary 2018-11-26 14:50:37 +01:00
Adrian Moennich
71a679267c Storage/S3: Cache and lazy-create S3 clients
Creating a client in new S3 sessions is somewhat expensive; don't do
that more often than needed
2018-11-26 14:34:41 +01:00
Adrian Moennich
b59949e100 Storage/S3: Fix source backend check
Also warn if non-fs backends are used with --rclone, as this only works
if they produce filesystem paths that are still valid after leaving the
context manager returning them.
2018-11-26 14:34:41 +01:00
Adrian Moennich
92f3676235 Storage/S3: Add filesystem-to-s3 migration tool 2018-11-26 14:34:41 +01:00
Adrian Moennich
e3fdc0b4dd Storage/S3: Change CLI command to indico s3 ... 2018-11-26 14:34:41 +01:00
Adrian Moennich
c87e24cee6 Storage/S3: Add addressing_style option 2018-11-12 12:02:20 +01:00
Adrian Moennich
13145470fd Storage/S3: Limit max bucket name length 2018-11-12 12:02:20 +01:00
Adrian Moennich
566503a0ac Storage/S3: Remove obsolete checks 2018-11-12 12:02:20 +01:00
Adrian Moennich
c6b629283c Storage/S3: Fix i18n name 2018-11-12 12:02:20 +01:00
Adrian Moennich
f75fd081ff Storage/S3: Add missing DISTINCT to query 2018-11-12 12:02:20 +01:00
Adrian Moennich
55d4de2efb Storage/S3: Fix grammar 2018-11-12 12:02:20 +01:00
Adrian Moennich
f222458a70 Storage/S3: Add option to download files through indico
By default we redirect to a presigned S3 download URL, with this setting
one can instead have Indico download the file and serve it to the client.
2018-11-12 12:02:20 +01:00
Adrian Moennich
d2a5aadd9b Storage/S3: Remove -dev from version 2018-11-12 12:02:20 +01:00
Adrian Moennich
1a49a6d7a6 Storage/S3: Add custom metadata exposed via API 2018-11-12 12:02:20 +01:00
Adrian Moennich
d19afe7157 Storage/S3: Add API to get list of buckets 2018-11-12 12:02:20 +01:00
Adrian Moennich
16069e4ac7 Storage/S3: Split plugin and storage 2018-11-12 12:02:20 +01:00
Adrian Moennich
60b692295a Storage/S3: Separate dynamic/static bucket backends
Like this the bucket name does not need to be part of the file_id when
using static bucket names, avoiding updating every single row
containing one when moving data from another storage backend to S3
2018-11-12 12:02:20 +01:00
Adrian Moennich
09d577ca88 Storage/S3: Expose session/client on the instance
This makes it much more convenient when using the backend in scripts,
e.g. to import data from another storage to S3.
2018-11-12 12:02:20 +01:00
Adrian Moennich
4c2767c1c6 Storage/S3: Add readonly version 2018-11-12 12:02:20 +01:00
Adrian Moennich
ef06b214b2 Storage/S3: Fix open() method 2018-11-12 12:02:20 +01:00
Adrian Moennich
9f9fb6a85d Storage/S3: Simplify invalid-placeholders logic 2018-11-12 12:02:20 +01:00
Adrian Moennich
2ccecb6b08 Storage/S3: Make dynamic bucket names non-predictable 2018-11-12 12:02:20 +01:00
Adrian Moennich
58b96018ed Storage/S3: Support bucket versioning 2018-11-12 12:02:20 +01:00
Adrian Moennich
14b7ce18d4 Storage/S3: Save content type during upload 2018-11-12 12:02:20 +01:00
Adrian Moennich
9d4f8aa154 Storage/S3: Don't assume unaccessible buckets don't exist
Now we fail loudly if a bucket is not accessible during the existence
check; that way we don't risk getting errors later when writes to the
bucket would fail (or worse, succeed, because it's someone else's
world-writable bucket)
2018-11-12 12:02:20 +01:00
Adrian Moennich
6e6183f880 Storage/S3: Use policy instead of ACL 2018-11-12 12:02:20 +01:00
Adrian Moennich
2e3a917535 Storage/S3: Make config more flexible
- make all the connection args optional (and thus support storing
  them in places like ~/.aws/credentials)
- add option to set profile (to use e.g. a readonly keypair for some
  buckets)
2018-11-12 12:02:20 +01:00
Adrian Moennich
90050ae8ae Storage/S3: Avoid unnecessary indentation 2018-11-12 12:02:20 +01:00