41 Commits

Author SHA1 Message Date
Adrian Moennich
490f86b498 Merge branch '2.3-maintenance' 2021-01-05 14:31:54 +01:00
Indico Team
81802aa297 Happy new year 2021 🎆 2021-01-05 12:30:39 +01:00
Indico Team
01406b3d2a Fix empty lines after file headers 2020-11-12 14:47:29 +01:00
Indico Team
f2755b4be4 Run modernize, pyupgrade, autoflake and isort 2020-11-12 14:47:29 +01:00
Indico Team
43c703e1b4 Happy new year 2020 🎆 2020-01-16 11:32:21 +01:00
Indico Team
4b39714dbf Update license headers (and add missing) 2019-05-08 15:41:55 +02:00
Adrian Moennich
92f3676235 Storage/S3: Add filesystem-to-s3 migration tool 2018-11-26 14:34:41 +01:00
Adrian Moennich
e3fdc0b4dd Storage/S3: Change CLI command to indico s3 ... 2018-11-26 14:34:41 +01:00
Adrian Moennich
55d4de2efb Storage/S3: Fix grammar 2018-11-12 12:02:20 +01:00
Adrian Moennich
d19afe7157 Storage/S3: Add API to get list of buckets 2018-11-12 12:02:20 +01:00
Adrian Moennich
16069e4ac7 Storage/S3: Split plugin and storage 2018-11-12 12:02:20 +01:00
Adrian Moennich
60b692295a Storage/S3: Separate dynamic/static bucket backends
Like this the bucket name does not need to be part of the file_id when
using static bucket names, avoiding updating every single row
containing one when moving data from another storage backend to S3
2018-11-12 12:02:20 +01:00
Adrian Moennich
09d577ca88 Storage/S3: Expose session/client on the instance
This makes it much more convenient when using the backend in scripts,
e.g. to import data from another storage to S3.
2018-11-12 12:02:20 +01:00
Adrian Moennich
4c2767c1c6 Storage/S3: Add readonly version 2018-11-12 12:02:20 +01:00
Adrian Moennich
ef06b214b2 Storage/S3: Fix open() method 2018-11-12 12:02:20 +01:00
Adrian Moennich
2ccecb6b08 Storage/S3: Make dynamic bucket names non-predictable 2018-11-12 12:02:20 +01:00
Adrian Moennich
58b96018ed Storage/S3: Support bucket versioning 2018-11-12 12:02:20 +01:00
Adrian Moennich
14b7ce18d4 Storage/S3: Save content type during upload 2018-11-12 12:02:20 +01:00
Adrian Moennich
9d4f8aa154 Storage/S3: Don't assume unaccessible buckets don't exist
Now we fail loudly if a bucket is not accessible during the existence
check; that way we don't risk getting errors later when writes to the
bucket would fail (or worse, succeed, because it's someone else's
world-writable bucket)
2018-11-12 12:02:20 +01:00
Adrian Moennich
6e6183f880 Storage/S3: Use policy instead of ACL 2018-11-12 12:02:20 +01:00
Adrian Moennich
2e3a917535 Storage/S3: Make config more flexible
- make all the connection args optional (and thus support storing
  them in places like ~/.aws/credentials)
- add option to set profile (to use e.g. a readonly keypair for some
  buckets)
2018-11-12 12:02:20 +01:00
Adrian Moennich
011893804a Storage/S3: Improve cli help 2018-11-12 12:02:20 +01:00
Natalia Juszka
fe0b7501e6 Storage/S3: Store bucket name 2018-11-12 12:02:20 +01:00
Natalia Juszka
e7790f8dde Storage/S3: Change tmp file name 2018-11-12 12:02:20 +01:00
Natalia Juszka
72f4ddec05 Storage/S3: Add create bucket command 2018-11-12 12:02:20 +01:00
Natalia Juszka
f1bf6fc7c7 Storage/S3: Improve open file method 2018-11-12 12:02:20 +01:00
Natalia Juszka
0346a10178 Storage/S3: Fix checksum 2018-11-12 12:02:20 +01:00
Natalia Juszka
f0d178731f Storage/S3: improve create bucket task 2018-11-12 12:02:20 +01:00
Natalia Juszka
6a1300e712 Storage/S3: Add tests 2018-11-12 12:02:20 +01:00
Natalia Juszka
dd82b05533 Storage/S3: Refactor code 2018-11-12 12:02:20 +01:00
Natalia Juszka
068ee646b8 Storage/S3: Escape filename in content-disp header 2018-11-12 12:02:20 +01:00
Natalia Juszka
24572e6f7e Storage/S3: Set new bucket acl from template 2018-11-12 12:02:20 +01:00
Natalia Juszka
9c6770992c Storage/S3: Add dynamic bucket task 2018-11-12 12:02:20 +01:00
Natalia Juszka
0b703f155a Storage/S3: add response headers to generate url 2018-11-12 12:02:20 +01:00
Natalia Juszka
6b3570ce2f Storage/S3: generate url to download file 2018-11-12 12:02:20 +01:00
Natalia Juszka
36f8ef06d2 Storage/S3: Add md5 checksum 2018-11-12 12:02:20 +01:00
Natalia Juszka
4699f85234 Storage/S3: Add error handling 2018-11-12 12:02:20 +01:00
Natalia Juszka
09bc16ecef Storage/S3: Override get_local_path method 2018-11-12 12:02:20 +01:00
Natalia Juszka
65e99933a7 Storage/S3: Add basic functionality 2018-11-12 12:02:20 +01:00
Natalia Juszka
15f9f4326e Storage/S3: Initialize storage 2018-11-12 12:02:20 +01:00
Natalia Juszka
eaf32f1153 Storage/S3: create plugin 2018-11-12 12:02:20 +01:00