wikitech/cloudweb hosts cannot access x1
Open, MediumPublicPRODUCTION ERROR
Actions

Assigned To

None

Authored By

	Dreamy_Jazz
	Jun 4 2024, 10:11 AM

Description

It seems that the extension1 database for labswiki is always overloaded. This means that files being uploaded to labswiki cannot be scanned by the MediaModeration scanning script as the reference to the image is never added to the mediamoderation_scan table stored on extension1.

While this could be deliberate to avoid overloading the DB, I am not convinced of this as other wikis (such as commonswiki) are not having this problem consistently. Connecting to the DB on mwmaint using sql labswiki --cluster=extension1 suggests that the DB is up.

Error

All events matching this are at https://logstash.wikimedia.org/goto/f710796edb768a691fa320f2e2217d02

message

[{reqId}] {exception_url}   Wikimedia\Rdbms\DBUnexpectedError: Database servers in extension1 are overloaded. In order to protect application servers, the circuit breaking to databases of this section have been activated. Please try again a few seconds.

trace

from /srv/mediawiki/php-1.43.0-wmf.7/includes/libs/rdbms/loadmonitor/LoadMonitor.php(124)
#0 /srv/mediawiki/php-1.43.0-wmf.7/includes/libs/rdbms/loadbalancer/LoadBalancer.php(455): Wikimedia\Rdbms\LoadMonitor->scaleLoads(array)
#1 /srv/mediawiki/php-1.43.0-wmf.7/includes/libs/rdbms/loadbalancer/LoadBalancer.php(781): Wikimedia\Rdbms\LoadBalancer->getReaderIndex(string)
#2 /srv/mediawiki/php-1.43.0-wmf.7/includes/libs/rdbms/database/DBConnRef.php(110): Wikimedia\Rdbms\LoadBalancer->getConnectionInternal(integer, array, string, integer)
#3 /srv/mediawiki/php-1.43.0-wmf.7/includes/libs/rdbms/database/DBConnRef.php(124): Wikimedia\Rdbms\DBConnRef->ensureConnection()
#4 /srv/mediawiki/php-1.43.0-wmf.7/includes/libs/rdbms/database/DBConnRef.php(341): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#5 /srv/mediawiki/php-1.43.0-wmf.7/includes/libs/rdbms/querybuilder/SelectQueryBuilder.php(748): Wikimedia\Rdbms\DBConnRef->selectField(array, string, array, string, array, array)
#6 /srv/mediawiki/php-1.43.0-wmf.7/extensions/MediaModeration/src/Services/MediaModerationDatabaseLookup.php(41): Wikimedia\Rdbms\SelectQueryBuilder->fetchField()
#7 /srv/mediawiki/php-1.43.0-wmf.7/extensions/MediaModeration/src/Hooks/Handlers/UploadCompleteHandler.php(69): MediaWiki\Extension\MediaModeration\Services\MediaModerationDatabaseLookup->fileExistsInScanTable(LocalFile)
#8 [internal function]: MediaWiki\Extension\MediaModeration\Hooks\Handlers\UploadCompleteHandler->MediaWiki\Extension\MediaModeration\Hooks\Handlers\{closure}()
#9 /srv/mediawiki/php-1.43.0-wmf.7/includes/deferred/MWCallableUpdate.php(42): call_user_func(Closure)
#10 /srv/mediawiki/php-1.43.0-wmf.7/includes/deferred/DeferredUpdates.php(486): MediaWiki\Deferred\MWCallableUpdate->doUpdate()
#11 /srv/mediawiki/php-1.43.0-wmf.7/includes/deferred/DeferredUpdates.php(198): MediaWiki\Deferred\DeferredUpdates::attemptUpdate(MediaWiki\Deferred\MWCallableUpdate)
#12 /srv/mediawiki/php-1.43.0-wmf.7/includes/deferred/DeferredUpdates.php(285): MediaWiki\Deferred\DeferredUpdates::run(MediaWiki\Deferred\MWCallableUpdate)
#13 /srv/mediawiki/php-1.43.0-wmf.7/includes/deferred/DeferredUpdatesScope.php(269): MediaWiki\Deferred\DeferredUpdates::MediaWiki\Deferred\{closure}(MediaWiki\Deferred\MWCallableUpdate, integer)
#14 /srv/mediawiki/php-1.43.0-wmf.7/includes/deferred/DeferredUpdatesScope.php(198): MediaWiki\Deferred\DeferredUpdatesScope->processStageQueue(integer, integer, Closure)
#15 /srv/mediawiki/php-1.43.0-wmf.7/includes/deferred/DeferredUpdates.php(304): MediaWiki\Deferred\DeferredUpdatesScope->processUpdates(integer, Closure)
#16 /srv/mediawiki/php-1.43.0-wmf.7/includes/MediaWikiEntryPoint.php(673): MediaWiki\Deferred\DeferredUpdates::doUpdates()
#17 /srv/mediawiki/php-1.43.0-wmf.7/includes/MediaWikiEntryPoint.php(496): MediaWiki\MediaWikiEntryPoint->restInPeace()
#18 /srv/mediawiki/php-1.43.0-wmf.7/includes/MediaWikiEntryPoint.php(454): MediaWiki\MediaWikiEntryPoint->doPostOutputShutdown()
#19 /srv/mediawiki/php-1.43.0-wmf.7/includes/MediaWikiEntryPoint.php(209): MediaWiki\MediaWikiEntryPoint->postOutputShutdown()
#20 /srv/mediawiki/php-1.43.0-wmf.7/index.php(58): MediaWiki\MediaWikiEntryPoint->run()
#21 /srv/mediawiki/w/index.php(3): require(string)
#22 {main}

Details

MediaWiki Version: 1.43-wmf.7

Related Objects

Mentioned In: T374210: Error connecting to db1237 as user wikiadmin2023: :real_connect(): (HY000/2002): Connection timed out on labswiki
T370304: Bursts of occasional severe contention on s4 (commonswiki) primary mariadb causing recurrent user-facing outages on all wikis

Event Timeline

Dreamy_Jazz created this task.Jun 4 2024, 10:11 AM

Restricted Application added a subscriber: • Aklapper. · View Herald TranscriptJun 4 2024, 10:11 AM

This is most likely the special hosts that run Wikitech not having the necessary firewall rules/grants on x1 hosts because they've not needed that in the psat.

I'm sure this is a bug. If extension1 was actually overloaded, all wikis would have had issues. Let me take a look. I have a feeling this relates to some special hard-coding of some config for wikitech. Give me a bit.

Thanks both.

Can we wait until wikitech host is moved to production network? That'd automatically solve this issue.

That should be okay as long as this is planned within the next few months. Once the backlog of images to scan on commonswiki is complete, we will probably want to do wikis such as wikitech.

I hope we get there before that. The thing is that every time we add a firewall exception or a new grant to the host in public network, it makes me even more uneasy about the situation. @joanna_borun is taking the lead on making this happen soon.

ABran-WMF triaged this task as Medium priority.Jun 5 2024, 7:12 AM

ABran-WMF moved this task from Triage to Blocked on the DBA board.

taavi renamed this task from extension1 database for wikitech is always overloaded to wikitech/cloudweb hosts cannot access x1.Jun 5 2024, 9:28 AM

• Aklapper mentioned this in T370304: Bursts of occasional severe contention on s4 (commonswiki) primary mariadb causing recurrent user-facing outages on all wikis.Jul 17 2024, 4:19 PM

XtexChooser subscribed.Jul 19 2024, 7:53 AM

Krinkle moved this task from Untriaged to Jun 2024 on the Wikimedia-production-error board.Jul 25 2024, 4:04 PM

mdaniels5757 subscribed.Aug 13 2024, 6:56 PM

Dreamy_Jazz mentioned this in T374210: Error connecting to db1237 as user wikiadmin2023: :real_connect(): (HY000/2002): Connection timed out on labswiki.Fri, Sep 6, 11:06 AM

RhinosF1 subscribed.Fri, Sep 6, 11:09 AM

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.

wikitech/cloudweb hosts cannot access x1Open, MediumPublicPRODUCTION ERRORActions