Adrian Moennich
490f86b498
Merge branch '2.3-maintenance'
2021-01-05 14:31:54 +01:00
Indico Team
81802aa297
Happy new year 2021 🎆
2021-01-05 12:30:39 +01:00
Indico Team
01406b3d2a
Fix empty lines after file headers
2020-11-12 14:47:29 +01:00
Indico Team
f2755b4be4
Run modernize, pyupgrade, autoflake and isort
2020-11-12 14:47:29 +01:00
Indico Team
43c703e1b4
Happy new year 2020 🎆
2020-01-16 11:32:21 +01:00
Indico Team
4b39714dbf
Update license headers (and add missing)
2019-05-08 15:41:55 +02:00
Adrian Moennich
92f3676235
Storage/S3: Add filesystem-to-s3 migration tool
2018-11-26 14:34:41 +01:00
Adrian Moennich
e3fdc0b4dd
Storage/S3: Change CLI command to indico s3 ...
2018-11-26 14:34:41 +01:00
Adrian Moennich
55d4de2efb
Storage/S3: Fix grammar
2018-11-12 12:02:20 +01:00
Adrian Moennich
d19afe7157
Storage/S3: Add API to get list of buckets
2018-11-12 12:02:20 +01:00
Adrian Moennich
16069e4ac7
Storage/S3: Split plugin and storage
2018-11-12 12:02:20 +01:00
Adrian Moennich
60b692295a
Storage/S3: Separate dynamic/static bucket backends
...
Like this the bucket name does not need to be part of the file_id when
using static bucket names, avoiding updating every single row
containing one when moving data from another storage backend to S3
2018-11-12 12:02:20 +01:00
Adrian Moennich
09d577ca88
Storage/S3: Expose session/client on the instance
...
This makes it much more convenient when using the backend in scripts,
e.g. to import data from another storage to S3.
2018-11-12 12:02:20 +01:00
Adrian Moennich
4c2767c1c6
Storage/S3: Add readonly version
2018-11-12 12:02:20 +01:00
Adrian Moennich
ef06b214b2
Storage/S3: Fix open() method
2018-11-12 12:02:20 +01:00
Adrian Moennich
2ccecb6b08
Storage/S3: Make dynamic bucket names non-predictable
2018-11-12 12:02:20 +01:00
Adrian Moennich
58b96018ed
Storage/S3: Support bucket versioning
2018-11-12 12:02:20 +01:00
Adrian Moennich
14b7ce18d4
Storage/S3: Save content type during upload
2018-11-12 12:02:20 +01:00
Adrian Moennich
9d4f8aa154
Storage/S3: Don't assume unaccessible buckets don't exist
...
Now we fail loudly if a bucket is not accessible during the existence
check; that way we don't risk getting errors later when writes to the
bucket would fail (or worse, succeed, because it's someone else's
world-writable bucket)
2018-11-12 12:02:20 +01:00
Adrian Moennich
6e6183f880
Storage/S3: Use policy instead of ACL
2018-11-12 12:02:20 +01:00
Adrian Moennich
2e3a917535
Storage/S3: Make config more flexible
...
- make all the connection args optional (and thus support storing
them in places like ~/.aws/credentials)
- add option to set profile (to use e.g. a readonly keypair for some
buckets)
2018-11-12 12:02:20 +01:00
Adrian Moennich
011893804a
Storage/S3: Improve cli help
2018-11-12 12:02:20 +01:00
Natalia Juszka
fe0b7501e6
Storage/S3: Store bucket name
2018-11-12 12:02:20 +01:00
Natalia Juszka
e7790f8dde
Storage/S3: Change tmp file name
2018-11-12 12:02:20 +01:00
Natalia Juszka
72f4ddec05
Storage/S3: Add create bucket command
2018-11-12 12:02:20 +01:00
Natalia Juszka
f1bf6fc7c7
Storage/S3: Improve open file method
2018-11-12 12:02:20 +01:00
Natalia Juszka
0346a10178
Storage/S3: Fix checksum
2018-11-12 12:02:20 +01:00
Natalia Juszka
f0d178731f
Storage/S3: improve create bucket task
2018-11-12 12:02:20 +01:00
Natalia Juszka
6a1300e712
Storage/S3: Add tests
2018-11-12 12:02:20 +01:00
Natalia Juszka
dd82b05533
Storage/S3: Refactor code
2018-11-12 12:02:20 +01:00
Natalia Juszka
068ee646b8
Storage/S3: Escape filename in content-disp header
2018-11-12 12:02:20 +01:00
Natalia Juszka
24572e6f7e
Storage/S3: Set new bucket acl from template
2018-11-12 12:02:20 +01:00
Natalia Juszka
9c6770992c
Storage/S3: Add dynamic bucket task
2018-11-12 12:02:20 +01:00
Natalia Juszka
0b703f155a
Storage/S3: add response headers to generate url
2018-11-12 12:02:20 +01:00
Natalia Juszka
6b3570ce2f
Storage/S3: generate url to download file
2018-11-12 12:02:20 +01:00
Natalia Juszka
36f8ef06d2
Storage/S3: Add md5 checksum
2018-11-12 12:02:20 +01:00
Natalia Juszka
4699f85234
Storage/S3: Add error handling
2018-11-12 12:02:20 +01:00
Natalia Juszka
09bc16ecef
Storage/S3: Override get_local_path method
2018-11-12 12:02:20 +01:00
Natalia Juszka
65e99933a7
Storage/S3: Add basic functionality
2018-11-12 12:02:20 +01:00
Natalia Juszka
15f9f4326e
Storage/S3: Initialize storage
2018-11-12 12:02:20 +01:00
Natalia Juszka
eaf32f1153
Storage/S3: create plugin
2018-11-12 12:02:20 +01:00