Should we guard other cases? (It normally won't happen, but could be a typo or so)

Agree, I am actually thinking about add one more generic Exception

But I feel this can result in the logic in the except block being duplicated. Not sure if there is a good way to implement it.

We don't really need elif if_error == "stop" here. we probably could try

else:
    raise e

I am facing a trade off here. I can define a function, just wondering if that is necessary, or there is a better way to keep code DRY.

Option 1

except (
    weaviate.exceptions.UnexpectedStatusCodeException,
    weaviate.exceptions.WeaviateDeleteManyError,
    Exception # capture generic exception, but the above two can also be captured by this, keeping for better readability
) as e:
    if if_error == "continue":
        self.log.error(e)
        failed_collection_list.append(collection_name)
    else:
        raise e

Option 2

except (
    weaviate.exceptions.UnexpectedStatusCodeException,
    weaviate.exceptions.WeaviateDeleteManyError
) as e:
    if if_error == "continue":
        self.log.error(e)
        failed_collection_list.append(collection_name)
    else:
        raise e
except Exception as e: # use another except block, but code don't follow DRY
   if if_error == "continue":
      self.log.error(e)
      failed_collection_list.append(collection_name)
    else:
        raise e

Is there any known exception other than those 2?

Actually I haven’t observed any failure while running the DAG testing. Probably due to the number of objects are small if the Filter is defined proper. For the two exceptions, I found those from the client API document, one is related to the connection and one is related to the delete_many operation. 🤔 I think I can define some invalid filters to see what issue may be raised. Like filter on a property that doesn’t exist.

Sounds good. We should avoid using Exception and AirflowException unless necessary

I run multiple tests with invalid filter, and the WeaviateDeleteManyError is captured. Basically, in the method, there are two main operations. The first one is to get the collection.

Scenarios

1. Collection not found: WeaviateDeleteManyError is captured, e.g. could not find class Collection_c in schema.
2. Connection issue, I think this should be captured by UnexpectedStatusCodeException
3. Invalid filter, WeaviateDeleteManyError is captured. e.g., no such prop with name 'label' found in class