Copying from an email here for easier reference.

I added Client.query_and_wait in google-cloud-bigquery 3.14.0 late last year (https://github.com/googleapis/python-bigquery/blob/main/CHANGELOG.md#3140-2023-12-08). Since then there have been a few fixes and optimizations, but I think that should be safe as the minimum version for what ibis is doing. For small queries with small results (< 500 KB or so) that can save anywhere from a few hundred milliseconds to 3 seconds.

Looking at where .query() is currently, called in ibis

Lines 628 to 631 in 560ddf6

    
           query = self.client.query( 
        
               stmt, job_config=job_config, project=self.billing_project 
        
           ) 
        
           query.result()  # blocks until finished

it won't be quite trivial. For example, the pattern of waiting until .result() to set the page size

ibis/ibis/backends/bigquery/__init__.py

Line 810 in 560ddf6

query_result = query.result(page_size=chunk_size)

won't work. It needs to be set at query_and_wait time.

	query = self.client.query(
	stmt, job_config=job_config, project=self.billing_project
	)
	query.result() # blocks until finished

ci(bigquery): bigquery ci is very slow #8987

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

pFad - (p)hone/(F)rame/(a)nonymizer/(d)eclutterfier! Saves Data!