Saturday, May 29, 2010

how to solve the daylight savings time problem (cron)

As it stands now, daylight savings time (DST) is a royal pain in the ass. Spring forward, fall back. This reeks havoc on computer systems that expect time to always move in a forward direction. If your computer is on UTC/GMT time, then this isn't an issue because DST isn't even a concept... time keeps on ticking. However, if you are like me, living in PST/PDT, two times every year, my clock goes bonkers.

There are two sides to this equation. Spring and Fall. This year, for spring forward, the time between March 14th, 2010 @ 2:00:00am till 2:59:59am just does not exist. You go from 1:59:59 to suddenly being 3:00:00. Now, if you have a scheduled job (via a cron type system) that runs every day at 2:00am, then on March 14th, your job just won't run. It will pick up again on March 15th. Of course, this is a big problem for systems where a job needs to run every 24 hours. Generally, the solution to this is to fix the computers time in UTC/GMT so that DST rules just don't apply. However, at my work, we need to run our jobs in our client local timezone.

As I mentioned before, there is two sides to this equation. Now we have to consider the fall back problem. In that case, November, 7 2010 @ 1:00am happens two times. You go till 1:59:59 and then suddenly the clock jumps back to 1:00am. So, if you have a job that runs every day at 1:30am, then on November 7th, it will run two times. If it is a long running job that takes up a lot of system resources, that is not a good thing.

One obvious solution to this whole problem is to not run any cron jobs between 11pm and 4am. This covers most of the DST changes across the entire planet. However, if this ever changes and it changes a lot, then suddenly you might be running into this issue in some weird timezone you have never heard of before. It is also important to consider that those are prime times for execution of background jobs because generally the systems are more at rest in the off hours. I'd hate to give up that many hours in a day.

After much discussion with my coworkers, we finally came up with a fairly brilliant solution to both sides of this problem which I have not seen written up elsewhere. For both solutions, the trick is to look at the UTC offset. For PST->PDT spring forward, the offset grows from -8 to -7. If your cron is scheduled to run every day at 2am, then just move the cronjob and the next scheduled time to an hour later (4am is when things will run) when you notice the change in offset. Your job will run an hour late, but that is better than not at all.

For fall, things are a tiny bit more complicated because you probably want to run the second instance of 1am instead of the first. The reason is that you will have that extra hour worth of data to process. So, when the first 1am rolls around, always check to see if the next scheduled cron execution is a different (smaller, -7 to -8) UTC offset. If so, then don't do anything. This way, the execution will run correctly the next time.

I hope that clarifies things. Now you are as much of an expert in DST as I am.


Julius Davies said...

I like it! Governments sometimes change when the DST change-over occurs, but there is no way to know about this in advance. Your technique appears to accommodate this problem very nicely.

Tyler said...

From the cron(1) man page:

Special considerations exist when the clock is changed by less than 3 hours, for example at the beginning and end of daylight savings time. If the time has moved forwards, those jobs which would have run in the time that was skipped will be run soon after the change. Conversely, if the time has moved backwards by less than 3 hours, those jobs that fall into the repeated time will not be re-run.

Jon Scott Stevens said...

Yup. However, not all cron implementations follow those rules.

Also, that doesn't take into account the issue of wanting to run the second instance instead of the first as I described in my posting.

Additionally, we manage our own 'nextRuntime' setting externally from our cron daemon so that we can manage when our cron jobs run. For example, if we sometimes want to move the nextRuntime to somewhere in the future if we don't want jobs to run right now.