Open
Description
Environment details
- OS type and version: Linux, Ubuntu 20.04
- Python version: 3.9.21
- pip version: I used bazel.
- google-cloud-storage` version: 3.1.0
Steps to reproduce
- create a bucket.
- upload a large file (in 1k chunks) to the bucket with the Python client. It must be larger than "chunk_size".
- check that the value returned by the write() method returns the correct value.
- this works, until the client uploads the first chunk to the bucket. Then the write method returns a value that is too large.
Code example
import random
from absl import app
from google.cloud import storage
SMALL_CHUNK_SIZE = 256 * 1024 # smallest chunk size allowed.
BLOCK_SIZE = 1024 # 1kb
BLOCK_NUM = 40_000
HEADER = b'start'
FOOTER = b'end'
CREDS = ...
BUCKET_NAME = ...
BLOB_NAME = ...
def main(argv):
del argv # Unused.
storage_client = storage.Client.from_service_account_json(CREDS)
bucket = storage_client.bucket(BUCKET_NAME)
blob = bucket.blob(BLOB_NAME, chunk_size=SMALL_CHUNK_SIZE)
output_stream = blob.open('wb', ignore_flush=True)
output_stream.write(HEADER)
for _ in range(BLOCK_NUM):
rand_block = random.randbytes(BLOCK_SIZE)
written = output_stream.write(rand_block)
if written > len(rand_block):
raise ValueError(
'written is larger than expected: written = %d, len(rand_block)'
' = %d' % (written, len(rand_block))
)
output_stream.write(FOOTER)
output_stream.close()
if __name__ == '__main__':
app.run(main)Stack trace
raise ValueError(
ValueError: written is larger than expected: written = 263168, len(rand_block) = 1024