The life of a schedule job (Community)

View the original community article here

Last tested: Jan 24, 2020
 

When people talk about schedules, they are generally referring to one of two things: schedule plans and schedule jobs.

A schedule plan refers to the configuration of a report: it's format, destination, and the conditions -- time interval or datagroup -- that determine when it should be sent. When you are are 'editing' a schedule in the Looker UI, you are editing the schedule plan.

A schedule job refers to a single occurrence of a report (specified by the schedule plan) being generated and sent out.

A schedule job will go through up to 4 stages in its lifecycle:

1. enqueue_for_execute
2. execute
3. enqueue_for_delivery
4. delivery

enqueue_for_executE

The schedule job is "waiting in line" for an open execution thread.

ExecutE

Execute is the stage where querying and rendering (if needed) happens. By default, there are 3 schedule execution threads on an instance. The number of scheduler threads can be changed in the startup parameters.

enqueue_for_delivery

The schedule job is "waiting in line" for an open delivery thread

Delivery

The generated report is packaged up and sent in an email, slack message, sftp upload, or whichever destination is specified by the schedule plan. There are 3 delivery threads, and this number cannot be modified.

For non-unlimited jobs

1. A process called the dispatcher will regularly look for any jobs that are ready to be run. Once it finds one, the job will be put in the execute queue.
2. Once an execute thread finishes its last job and is ready for the next one, it will take the next job from the queue and start executing. For render jobs, this means starting a render process, and for non-render jobs (CSVs, Excel, JSON, etc) simply running the query.
3. Once the query finishes, the job will be put in the delivery queue to wait for a delivery thread to open up.
4. Once a delivery thread is ready for a new job, it will take the query results, transform them in whatever way necessary (for example, creating an email with a subject for email jobs), and send the report on its way.

For unlimited jobs

For unlimited jobs the process is mostly the same, but the execution and delivery happens all in one step; since chunks of data are sent a few thousand rows at a time, the connection between the destination and database both have to be open simultaneously.

Note: there are some exceptions to all of this, but these are the general rules and a good way to think about schedule jobs in general, especially when thinking about where they can go wrong.