Welcome to OStack Knowledge Sharing Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
237 views
in Technique[技术] by (71.8m points)

python - Download s3 file only if upload completed

At the moment I get a list of files from s3 by a prefix and download them. I want to make sure the file uploaded completely before I start the download.

How is that possible?

bucket='bucket-name'
prefix='files_prefix')

s3_client = boto3.client('s3')
objs = s3_client.list_objects_v2(Bucket=bucket, Prefix=prefix)['Contents']

for key in objs:
    # need to add an if here to check if file upload completed
    s3_client.download_file(bucket, key['Key'],'{}/{}'.format('./data/',key['Key']))

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Answer

0 votes
by (71.8m points)

There is no such concept as a partially uploaded object. Objects in Amazon S3 are immutable. They either exist in full, or do not exist at all.

Multi-part uploads consist of individual uploads (to which the above rule applies), but once the multi-part upload is completed, they convert into a single object with all required parts.

So... you don't need to do anything.


与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome to OStack Knowledge Sharing Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

2.1m questions

2.1m answers

60 comments

56.6k users

...