In the UI, the date time is always shown in UTC. This is stored internally and in the database, which means the date parameters mentioned above are all described in UTC. (Note: Airflow 2.2 replaced the term 'execution date' with 'logical date'.)ĭealing with different time zones around the world always requires a lot of care, especially with time changes such as Daylight Saving Time (DST).īy default, Airflow stores datetime information in UTC (coordinated universal time, a global standard by which time zones are measured). Now we can explain why the first DAG run could only start after the start date + the scheduled interval: the data isn’t gathered completely until this time point. The start point of the range is called the execution date. We are not able to run the DAG until the range ends. We run the DAG to process the data that we have already obtained during a certain time range - this is called the schedule interval. (The data gathered during the range is called a data interval.) It’s only after the end of a time range that we can start processing the data that has been inside it. Every execution date indicates the beginning of a certain range. ![]() The reason we set the schedule_interval to the 10th minute of every hour is that we want to deal with the data within this time range (yyyy-mm-dd hh:10:00 ~ yyyy-mm-dd hh+1:10:00). And the start time is one hour later than the execution date. Also, the item run_id contains the execution date. In this example it’s 15:10:00.Īs the screenshot below shows, the value of the item Run is the execution date. This IS NOT the date at which the DAG gets triggered (as you might expect it to be) but is instead date of the beginning of the data interval that you want to process. Now that the DAG is triggered, there’s another important - yet confusing - concept worth explaining: the "execution date". Don't confuse the start_date with the date that the first DAG run starts. So, it was at 16:10:00, rather than 15:10:00 that this DAG was initially triggered. Here, the first DAG run will be triggered after the start date + the scheduled interval. In part two we’ll discuss when and how a DAG will be triggered. ![]() In the first part of this two-part series, we explored the basic concepts of cron expressions and DAG run parameters in the context of Airflow’s scheduling mechanism.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |