Help:Job queue
|
The job queue performs long-running tasks asynchronously. It is designed to hold many short tasks using batch processing.
Job examples
Updating links tables when a template changes
When a template changes, MediaWiki adds a job to the job queue for each article transcluding that template. Each job is a command to read an article, expand any templates, and update the link table accordingly. Previously, the host articles would remain outdated until either their parser cache expires or until a user edits the article.
HTML cache invalidation
A wider class of operations can result in invalidation of the HTML cache for a large number of pages:
- Changing an image (all the thumbnails have to be re-rendered, and their sizes recalculated)
- Deleting a page (all the links to it from other pages need to change from blue to red)
- Creating or undeleting a page (like above, but from red to blue)
- Changing a template (all the pages that transclude the template need updating)
Except for template changes, these operations do not invalidate the links tables, but they do invalidate the HTML cache of all pages linking to that page, or using that image. Invalidating the cache of a page is a short operation; it only requires updating a single database field and sending a multicast packet to clear the caches. But if there are more than about 1000 to do, it takes a long time. By default, one job is added per 300 operations (see $wgUpdateRowsPerJob)
Note, however, that even if purging the cache of a page is a short operation, reparsing a complex page that is not in the cache may be expensive, especially if a highly used template is edited and causes lots of pages to be purged in a short period of time and your wiki has lots of concurrent visitors loading a wide spread of pages. This can be mitigated by reducing the number of pages purged in a short period of time, by reducing $wgUpdateRowsPerJob to a small number (20, for example) and also set $wgJobBackoffThrottling for htmlCacheUpdate to a low number (5, for example).
Audio and video transcoding
When using TimedMediaHandler to process local uploads of audio and video files, the job queue is used to run the potentially very slow creation of derivative transcodes at various resolutions/formats.
These are not suitable for running on web requests -- you will need a background runner.
It's recommended to set up separate runners for the webVideoTranscode and webVideoTranscodePrioritized job types if possible. These two queues process different subsets of files -- the first for high resolution HD videos, and the second for lower-resolution videos and audio files which process more quickly.