Illuminated Computing

TQuel Paper: Implementing Temporal Operators in Postgres

2026-07-21T00:00:00Z

With UPDATE/DELETE FOR PORTION OF looking like it will land in Postgres 19, I’ve been thinking about next steps. I read a very helpful paper on temporal relational algebra last May by Richard Snodgrass. Here are some notes on it.

The paper was “An Overview of TQuel”. It’s Chapter 6 in Temporal Databases: Theory, Design, and Implementation from 1993.

TQuel was an extension to Quel, the query language for Ingres. Ingres, of course, was the predecessor to Postgres!

My main motivation reading this paper was to learn more about the algebraic identities of temporal relational operators. A query planner depends on such identities to transform your query into a more efficient shape. For instance, if you can filter rows before joining tables instead of after, you’ll get a much faster execution. The useful identities for regular relational operators are well-known, but what about temporal operators? If we ever want to support temporal joins and setops in Postgres, we have to figure that out.

I implemented some temporal operators in SQL in my temporal_ops extension. Since they are just SQL, I don’t have to worry about optimizer correctness. But it would be better to have dedicated executor nodes. That should get us closer to an optimal implementation. I want to teach that extension to inject CustomScans . . . somehow . . . maybe with a post-parser hook? Once I have that, I can experiment with planner transformations.

But as a first step, I’m trying to see what is in the research already. One surprise in Snodgrass’s paper was that TQuel valid-times are not intervals (like Postgres rangetypes or SQL:2011 PERIODs). Instead they are sets of “chronons”: all the times that the tuple is true, whether contiguous or not. I can see how that might be a more “pure” representation. There is something artificial to forcing the valid-time to be only a single contiguous stretch of time. Fortunately a set of chronons is exactly a multirange, so it is still something you could represent in Postgres.

A bigger surprise was that TQuel tracks valid-time for each attribute, instead of for the overall tuple. That is a bit more complicated! Actually it reminds me of how Date/Darwen/Lorentzos and Johnston both propose a separate-table-per-attribute structure (which Date calls “Sixth Normal Form”). You can see my temporal databases bibliography for more about that. There are hints in both of their books that they suspect this is asking too much from ordinary database users. And Snodgrass’s choice here made me worried that his results would be hard to apply to a practical RDBMS like Postgres.

But then I realized that any SQL:2011 valid-time tuple can be trivially represented in TQuel’s format: just copy the tuple valid-time onto each attribute. In fact Snodgrass has a term for that: a “homogeneous relation”. (Such TQuel relations still have multirange valid-time though.) That makes me hopeful that identities that are true in TQuel might still be true in SQL:2011. They might not be though, if TQuel operators on homogeneous relations don’t always have homogeneous relation results: if for those operators homogeneous relations aren’t “closed”.

Also TQuel has no nulls and no duplicates, but even that may not matter much. A valid-time never has null endpoints. Well with Postgres rangetypes we do use null to represent “unbounded”, but that has different behavior than a normal null. It is no different than using, say, 3000-01-01, except it works for any base type (unlike 'Infinity'). And I don’t think valid-time adds any further differences between duplicate and non-duplicate behavior. So the TQuel algebra should be as applicable as any non-temporal no-NULL no-duplicate algebra. At least it doesn’t add any new issues.

One result in the paper is that for homogeneous relations, TQuel operators are “snapshot reducible”. For such relations, “the valid-time operators ∪̂, −̂, ⨯̂, σ̂, and π̂ reduce to their snapshot counterparts.” In other words, if you want a final result at a single time t (a snapshot), you can evaluate temporal operators and then take the snapshot (which is slow) or take snapshots first, from all your base tables, and then evaluate non-temporal operators. You can “push down the qual” that valid_at @> now() (or whatever time you care about). It is a homomorphism:

σ(R ⨯̂ S) = σ(R) × σ(S)

I think this is the most common query shape against temporal tables, especially for complicated reports. You don’t need the whole history; you just want the result at a given moment. It means that if you are converting an existing app to use application-time tables, you don’t need temporal operators yet. You don’t even need to rewrite the queries. Just declare views that filter your history tables by now() (or by some variable), and run the queries on them. I talked about this in my PG DATA 2026 talk Migrating to a Temporal Schema.

The real reward of the paper though was Theorem 13. I found this a couple years ago from skimming, and I had wanted to read the whole thing ever since. Snodgrass says that most traditional algebraic identities are preserved by his temporal operators, but not all. A big exception is that Cartesian product does not distribute over difference:

Q ⨯̂ (R −̂ S) ≢ (Q ⨯̂ R) −̂ (Q ⨯̂ S)

In SQL the name for “difference” is EXCEPT. Since joins are all defined in terms of Cartesian product, this is a big one.

Why doesn’t this work?

To explore that question, I vibe-coded a “relational calculator”, relsim. It is a Racket library that lets you evaluate tuples, relations, and operators on them. It supports all the traditional operators (even division!), and also temporal versions. There are temporal operators for SQL:2011 tuples (a range for the whole tuple), multirange tuples (the same but a multirange), and TQuel tuples (a multirange for each attribute). To make it more SQL-like, I allow nulls and duplicates.

You don’t need to know Racket to use it. Actually I haven’t written Lisp since college myself. But I chose a Lisp so I could get a free parser and REPL, with syntax as math-like as possible. It is pretty easy to use, I think.

I used relsim to find a counterexample to the Cartesian-product-over-difference identity, and then “step through it” to see where it went wrong. Claude and I wrote up the investigation. Using TQuel-style tuples, suppose we have these relations:

     Q
-------------
'a', {[0,20)}

     R
-------------
'b', {[0,20)}

     S
-------------
'b', {[5,10)}

Each relation has one attribute: a string with an attached multirange valid-time. And each relation has a single row.

Then for the left-hand side of the identity, we have:

Q ⨯̂ (R −̂ S)
{('a', {[0,20)})} ⨯̂ ({('b', {[0,20)})} −̂ {('b', {[5,10)})})
{('a', {[0,20)})} ⨯̂ {('b', {[0,5), [10,20)})}
{('a', {[0,20)}, 'b', {[0,5), [10,20)})}

Here I’m using {...} for a relation (since it’s a set of tuples), and (...) for a tuple. For the valid-time I’m using Postgres multirange notation.

For the right-hand side, we have:

(Q ⨯̂ R) −̂ (Q ⨯̂ S)
({('a', {[0,20)})} ⨯̂ {('b', {[0,20)})}) −̂ ({('a', {[0,20)})} ⨯̂ {('b', {[5,10)})})
{('a', {[0,20)}, 'b', {[0,20)})} −̂ {('a', {[0,20)}, 'b', {[5,10)})}
{('a', {}, 'b', {[0,5), [10,20)})}

So b is the same, but we lost all of a! I’m not sure I have an intuitive explanation for “why” that happens.

What about non-TQuel tuples? I tried those too, and sadly got the same result. For example with rangetypes (where two tuples might be needed for a difference result), the LHS:

Q ⨯̂ (R −̂ S)
{('a', [0,20))} ⨯̂ ({('b', [0,20))} −̂ {('b', [5,10))})
{('a', [0,20))} ⨯̂ {('b', [0,5)),
                   ('b', [10,20))}
{('a', [0,20), 'b', [0,5), [0,5)),
 ('a', [0,20), 'b', [10,20), [10,20))}

Note that here for Cartesian product I’m keeping the inputs’ valid-times, as if they were regular attributes. Then I add a result valid-time that is the inputs’ intersection.

Versus the RHS:

(Q ⨯̂ R) −̂ (Q ⨯̂ S)
({('a', [0,20))} ⨯̂ {('b', [0,20))}) −̂ ({('a', [0,20))} ⨯̂ {('b', [5,10))})
{('a', [0,20), 'b', [0,20), [0,20))} −̂ {('a', [0,20), 'b', [5,10), [5,10))}
{('a', [0,20), 'b', [0,20), [0,20))}

Again, different! Here the problem is that the RHS didn’t subtract anything. That’s because the old valid-times didn’t match, and we compare those like regular attributes.

But if we drop input valid-times, then the algebra works out! The LHS:

Q ⨯̂ (R −̂ S)
{('a', [0,20))} ⨯̂ ({('b', [0,20))} −̂ {('b', [5,10))})
{('a', [0,20))} ⨯̂ {('b', [0,5)),
                   ('b', [10,20))}
{('a', 'b', [0,5)),
 ('a', 'b', [10,20))}

and the RHS:

(Q ⨯̂ R) −̂ (Q ⨯̂ S)
({('a', [0,20))} ⨯̂ {('b', [0,20))}) −̂ ({('a', [0,20))} ⨯̂ {('b', [5,10))})
{('a', 'b', [0,20))} −̂ {('a', 'b', [5,10))}
{('a', 'b', [0,5)),
 ('a', 'b', [10,20))}

We got the same answer! The same thing happens for multirange valid-times. Keeping the original valid-times spoils the algebra.

I find several layers of irony here.

One of the main features of Anton Dignös’s work is precisely preserving the input valid-times (for example the “extends” operator in his “Temporal Alignment” paper), and in temporal_ops I tried to follow that. That can be useful for scaling aggregates, also for filtering (e.g. find all hotel reservations of a certain length) and joining (apply a discount depending on reservation length). But maybe it should not be the default. If the user wants them, they can SELECT them explicitly. And indeed for these examples, the input valid-times are really used internally by the temporal operator (by the aggregation function or the join’s theta condition), and they don’t need to be in the result.

Another irony is that Snodgrass’s TSQL2 and SQL:2011 PERIODs have been criticized for not treating valid-time as an attribute within relational theory. And so for PERIODs you can’t SELECT them, put them in a VIEW or subquery, pass them to a function or return them, GROUP BY them, etc. I’ve repeated that criticism myself. The lack of composability is annoying. But semantically, Snodgrass was right: valid-time is not an attribute, but a qualifier of the other attributes.

To be fair to the critics, they don’t treat valid-times like regular attributes either. Since Date/Darwen/Lorentzos define temporal operators in terms of PACK and UNPACK, their inputs’ valid-times “disappear”, and the original values aren’t carried through to the result.

It has been a big help to me to think that you can replace any valid-time tuple with a bunch of ordinary tuples, one for each moment in the valid-time, and that is how temporal features should behave. So valid-time really is something “magic” and different. I still don’t love how PERIODs are not values in SQL, but somehow we do need to treat valid-time differently.

I’m very interested in seeing some proofs of temporal algebraic identities. Snodgrass cites a paper he wrote with E. McKenzie, “Supporting valid time in an historical relational algebra: Proofs and extensions.” But I haven’t been able to find a copy. It is “Technical Report TR-91-15, Department of Computer Science, University of Arizona, Tucson, AZ, August 1991.” That sounds like it might not have been widely distributed. Some other citations which I now have, but haven’t yet read, are Jeffrey D. Ullman, Database and Knowledge-Base Systems Vol I & II and David Maier, The Theory of Relational Databases.

It looks like the 80s and 90s were really a golden age for temporal research. There are a lot of different temporal models and algebraic systems. Already in 1990, this paper by Alexander Tuzhilin and James Clifford was complaining about how to compare them, and they give proofs to show that one is at least as powerful as another. But I’m doing my best to come to grips with what is out there.

Thanks to Jeff Davis, Noah Misch, Boris Novikov, and Kent Bulza for helpful conversations!

Benchmarking Temporal Foreign Keys

2024-09-15T00:00:00Z

Way back in Februrary Peter Eisentraut asked me if I’d tested the performance of my patch to add temporal foreign keys to Postgres.

Have you checked that the generated queries can use indexes and have suitable performance? Do you have example execution plans maybe?

Here is a report on the tests I made. I gave a talk about this last month at pdxpug, but this blog post will be easier to access, and I’ll focus just on the foreign key results.

Method

As far as I know there are no published benchmark schemas or workflows for temporal data. Since the tables require start/end columns, you can’t use an existing benchmark like TCP-H. The tables built in to pgbench are no use either. I’m not even sure where to find a public dataset. The closest is something called “Incumben”, mentioned in the “Temporal Alignment” paper. They authors say it has 85,857 entries for job assignments across 49,195 employees at the University of Arizona—but I can’t find any trace of it online. (I’ll update here if I hear back from them about it.)

So I built a temporal benchmark of my own using CMU’s Benchbase framework. (Thanks to Mark Wong and Grant Holly for that recommendation!) It also uses employees and positions, both temporal tables with a valid_at column (a daterange). Each position has a reference to an employee, checked by a temporal foreign key. Primary and foreign keys have GiST indexes combining the integer part and the range part. Here is the DDL:

CREATE TABLE employees (
    id          int GENERATED BY DEFAULT AS IDENTITY NOT NULL,
    valid_at    daterange NOT NULL,
    name        text NOT NULL,
    salary      int NOT NULL,
    PRIMARY KEY (id, valid_at WITHOUT OVERLAPS)
);

CREATE TABLE positions (
    id          int GENERATED BY DEFAULT AS IDENTITY NOT NULL,
    valid_at    daterange NOT NULL,
    name        text NOT NULL,
    employee_id int NOT NULL,
    PRIMARY KEY (id, valid_at WITHOUT OVERLAPS),
    FOREIGN KEY (employee_id, PERIOD valid_at) REFERENCES employees (id, PERIOD valid_at)
);
CREATE INDEX idx_positions_employee_id ON positions USING gist (employee_id, valid_at);

Naturally you can’t run that unless you’ve compiled Postgres with the temporal patches above.

The benchmark has procedures that exercise foreign keys (update/delete employee, insert/update position). There are other procedures too: selecting one row, selecting many rows, inner join, outer join, semijoin, antijoin. I plan to add aggregates and set operations (union/except/intersect), as well as better queries for sequenced vs non-sequenced semantics. But right now the foreign key procedures are better developed than anything else. I also plan to change the SQL from rangetypes to standard SQL:2011 PERIODs, at least for non-Postgres RDBMSes. I’ll write more about all that later; this post is about foreign keys.

`range_agg` Implementation

Temporal foreign keys in Postgres are implemented like this:

SELECT 1
FROM    (
  SELECT pkperiodatt AS r
  FROM   [ONLY] pktable x
  WHERE  pkatt1 = $1 [AND ...]
  AND    pkperiodatt && $n
FOR KEY SHARE OF x
) x1
HAVING $n <@ range_agg(x1.r)

This is very similar to non-temporal checks. The main difference is we use range_agg to aggregate referenced records, since it may require their combination to satisfy the reference. For example if the employee got a raise in the middle of the position, neither employee record alone covers the position’s valid time:

In our query, the HAVING checks that the “sum” of the employee times covers the position time.

A subquery is not logically required, but Postgres currently doesn’t allow FOR KEY SHARE in a query with aggregations.

I like this query because it works not just for rangetypes, but multiranges too. In fact we could easily support arbitrary types, as long as the user provides an opclass with an appropriate support function (similar to the stratnum support function introduced for temporal primary keys). We would call that function in place of range_agg. But how does it perform?

`EXISTS implementation`

I compared this query with two others. The original implementation for temporal foreign keys appears on pages 128–129 of Developing Time-Oriented Database Applications in SQL by Richard Snodgrass. I call this the “EXISTS implementation”. Here is the SQL I used:

SELECT 1
-- There was a PK when the FK started:
WHERE EXISTS
  SELECT  1
  FROM    [ONLY] <pktable>
  WHERE   pkatt1 = $1 [AND ...]
  AND     COALESCE(lower(pkperiodatt), '-Infinity')
       <= COALESCE(lower($n), '-Infinity')
  AND     COALESCE(lower($n), '-Infinity')
       <  COALESCE(upper(pkperiodatt), 'Infinity')
)
-- There was a PK when the FK ended:
AND EXISTS (
  SELECT  1
  FROM    [ONLY] <pktable>
  WHERE   pkatt1 = $1 [AND ...]
  AND     COALESCE(lower(pkperiodatt), '-Infinity')
       <  COALESCE(upper($n), 'Infinity')
  AND     COALESCE(upper($n), 'Infinity')
       <= COALESCE(upper(pkperiodatt), 'Infinity')
)
-- There are no gaps in the PK:
-- (i.e. there is no PK that ends early,
-- unless a matching PK record starts right away)
AND NOT EXISTS (
  SELECT  1
  FROM    [ONLY] <pktable> AS pk1
  WHERE   pkatt1 = $1 [AND ...]
  AND     COALESCE(lower($n), '-Infinity')
       <  COALESCE(upper(pkperiodatt), 'Infinity')
  AND     COALESCE(upper(pkperiodatt), 'Infinity')
       <  COALESCE(upper($n), 'Infinity')
  AND     NOT EXISTS (
    SELECT  1
    FROM    [ONLY] <pktable> AS pk2
    WHERE   pk1.pkatt1 = pk2.pkatt1 [AND ...]
            -- but skip pk1.pkperiodatt && pk2.pkperiodatt
    AND     COALESCE(lower(pk2.pkperiodatt), '-Infinity')
         <= COALESCE(upper(pk1.pkperiodatt), 'Infinity')
            COALESCE(upper(pk1.pkperiodatt), 'Infinity')
         <  COALESCE(upper(pk2.pkperiodatt), 'Infinity')
  )
);

The main idea here is that we check three things: (1) the referencing row is covered in the beginning, (2) it is covered in the end, (3) in between, the referenced row(s) have no gaps.

I made a few changes to the original:

It can’t be a CHECK constraint, since it references other rows.
There is less nesting. The original is wrapped in a big NOT EXISTS and looks for bad rows. Essentially it says “there are no invalid records.” In Postgres we check one referencing row at a time, and we give a result if it is valid. You could say we look for good rows. This also requires inverting the middle-layer EXISTS and NOT EXISTS predicates, and changing ORs to ANDs. I’ve often run into trouble with OR, so this is probably fortunate.
We have to “unwrap” the start/end times since they are stored in a rangetype. I could have used rangetype operators here, but I wanted to keep the adaptation as straightforward as possible, and the previous changes felt like a lot already. Unwrapping requires dealing with unbounded ranges, so I’m using plus/minus Infinity as a sentinel. This is not perfectly accurate, since in ranges a null bound is “further out” than a plus/minus Infinity. (Try select '{(,)}'::datemultirange - '{(-Infinity,Infinity)}'::datemultirange.) But again, solving that was taking me too far from the original, and it’s fine for a benchmark.
We need to lock the rows with FOR KEY SHARE in the same way as above. We need to do this in each branch, since they may use different rows.

Given the complexity, I didn’t expect this query to perform very well.

`lag` implementation

Finally there is an implementation in Vik Fearing’s periods extension. This is a lot like the EXISTS implementation, except to check for gaps it uses the lag window function. Here is the SQL I tested:

SELECT  1
FROM    (
  SELECT  uk.uk_start_value,
          uk.uk_end_value,
          NULLIF(LAG(uk.uk_end_value) OVER
            (ORDER BY uk.uk_start_value), uk.uk_start_value) AS x
  FROM   (
    SELECT  coalesce(lower(x.pkperiodatt), '-Infinity') AS uk_start_value,
            coalesce(upper(x.pkperiodatt), 'Infinity') AS uk_end_value
    FROM    pktable AS x
    WHERE   pkatt1 = $1 [AND ...]
    AND     uk.pkperiodatt && $n
    FOR KEY SHARE OF x
  ) AS uk
) AS uk
WHERE   uk.uk_start_value < upper($n)
AND     uk.uk_end_value >= lower($n)
HAVING  MIN(uk.uk_start_value) <= lower($n)
AND     MAX(uk.uk_end_value) >= upper($n)
AND     array_agg(uk.x) FILTER (WHERE uk.x IS NOT NULL) IS NULL

Again I had to make some adaptations to the original

There is less nesting, for similar reasons as before.
We unwrap the ranges, much like the EXISTS version. Again there is an Infinity-vs-null discrepancy, but it is harder to deal with since the query uses null entries in the lag result to indicate gaps.
I couldn’t resist using && instead of <= and >= in the most-nested part to find relevant rows. The change was sufficiently obvious, and if it makes a difference it should speed things up, so it makes the comparison a bit more fair.

I made a new branch rooted in my valid-time branch, and added an extra commit to switch between each implementation with a compile-tag flag. By default we still use range_agg, but instead you can say ‑DRI_TEMPORAL_IMPL_LAG or ‑DRI_TEMPORAL_IMPL_EXISTS. I installed each implementation in a separate cluster, listening on port 5460, 5461, and 5462 respectively.

I also included procedures in Benchbase to simply run the above queries as SELECTs. Since we are doing quite focused microbenchmarking here, I thought that would be less noisy than doing the same DML for each implementation. It also means we can run a mix of all three implementations together: they use the same cluster, and if there is any noise on the machine it affects them all. If you look at my temporal benchmark code, you’ll see the same SQL but adapted for the employees/positions tables.

Hypothesis

Here is the query plan for the range_agg implementation:

Aggregate
  Filter: ('[2020-10-10,2020-12-12)'::daterange <@ range_agg(x1.r))
  ->  Subquery Scan on x1
    ->  LockRows
      ->  Index Scan using employees_pkey on employees x
        Index Cond: ((id = 500) AND (valid_at && '[2020-10-10,2020-12-12)'::daterange))

It uses the index, and it all seems like what we’d want. It is not an Index Only Scan, but that’s because we lock the rows. Non-temporal foreign keys are the same way. This should perform pretty well.

Here is the query plan for the EXISTS implementation:

Result
  One-Time Filter: ((InitPlan 1).col1 AND (InitPlan 2).col1 AND (NOT (InitPlan 4).col1))
  InitPlan 1
  ->  LockRows
    ->  Index Scan using employees_pkey on employees x
      Index Cond: ((id = 500) AND (valid_at && '[2020-10-10,2020-12-12)'::daterange))
      Filter: ((COALESCE(lower(valid_at), '-infinity'::date) <= '2020-10-10'::date) AND ('2020-10-10'::date < COALESCE(upper(valid_at), 'infinity'::date)))
  InitPlan 2
  ->  LockRows
    ->  Index Scan using employees_pkey on employees x_1
      Index Cond: ((id = 500) AND (valid_at && '[2020-10-10,2020-12-12)'::daterange))
      Filter: ((COALESCE(lower(valid_at), '-infinity'::date) < '2020-12-12'::date) AND ('2020-12-12'::date <= COALESCE(upper(valid_at), 'infinity'::date)))
  InitPlan 4
  ->  LockRows
    ->  Index Scan using employees_pkey on employees pk1
      Index Cond: ((id = 500) AND (valid_at && '[2020-10-10,2020-12-12)'::daterange))
      Filter: (('2020-10-10'::date < COALESCE(upper(valid_at), 'infinity'::date)) AND (COALESCE(upper(valid_at), 'infinity'::date) < '2020-12-12'::date) AND (NOT EXISTS(SubPlan 3)))
      SubPlan 3
      ->  LockRows
        ->  Index Scan using employees_pkey on employees pk2
          Index Cond: (id = pk1.id)
          Filter: ((COALESCE(lower(valid_at), '-infinity'::date) <= COALESCE(upper(pk1.valid_at), 'infinity'::date)) AND (COALESCE(upper(pk1.valid_at), 'infinity'::date) < COALESCE(upper(valid_at), 'infinity'::date)))

That looks like a lot of work!

And here is the plan for the lag implementation:

Aggregate
  Filter: ((array_agg(uk.x) FILTER (WHERE (uk.x IS NOT NULL)) IS NULL) AND (min(uk.uk_start_value) <= '2020-10-10'::date) AND (max(uk.uk_end_value) >= '2020-12-12'::date))
  ->  Subquery Scan on uk
    Filter: ((uk.uk_start_value < '2020-12-12'::date) AND (uk.uk_end_value >= '2020-10-10'::date))
    ->  WindowAgg
      ->  Sort
        Sort Key: uk_1.uk_start_value
        ->  Subquery Scan on uk_1
          ->  LockRows
            ->  Index Scan using employees_pkey on employees x
              Index Cond: ((id = 500) AND (valid_at && '[2020-10-10,2020-12-12)'::daterange))

This looks a lot like the range_agg version. We still use our index. There is an extra Sort step, but internally the range_agg function must do much the same thing (if not something worse). Maybe the biggest difference (though a slight one) is aggregating twice.

So I expect range_agg to perform the best, with lag a close second, and EXISTS far behind.

One exception may be a single referencing row that spans many referenced rows. If range_agg is O(n²), it should fall behind as the referenced rows increase.

Results

I started by running a quick test on my laptop, an M2 Macbook Air with 16 GB of RAM. I tested the DML commands on each cluster, one after another. Then I checked the benchbase summary file for the throughput. The results were what I expected:

Similarly, range_agg had the best latency at the 25th, 50th, 75th, 90th, and 99th percentiles:

But the difference throughout is pretty small, and at the time my Benchbase procedures used a lot of synchronized blocks to ensure there were few foreign key failures, and that kind of locking seemed like it might throw off the results. I needed to do more than this casual check.

I ran all the future benchmarks on my personal desktop, running Ubuntu 22.04.

It was hard to make things reproducible, but I wrote various scripts as I went, and I tried to capture results. The repo for all that is here. My pdxpug talk above contains some reflections about improving my benchmark methodology.

I also removed the synchronized blocks and dealt with foreign key failures a better way (by categorizing them as errors but not raising an exception).

The first more careful tests used the direct SELECT statements.

Again, the 95th percentile latency was what I expected:

But the winner for mean latency was EXISTS!:

A clue was in the Benchbase output showing successful transactions vs errors. (The Noop procedure is so can make the proportions 33/33/33/1 instead of 33/33/34.):

Completed Transactions:
com.oltpbenchmark.benchmarks.temporal.procedures.CheckForeignKeyRangeAgg/01      [72064] ********************************************************************************
com.oltpbenchmark.benchmarks.temporal.procedures.CheckForeignKeyLag/02           [71479] *******************************************************************************
com.oltpbenchmark.benchmarks.temporal.procedures.CheckForeignKeyExists/03        [71529] *******************************************************************************
com.oltpbenchmark.benchmarks.temporal.procedures.Noop/04                         [ 4585] *****
Aborted Transactions:
<EMPTY>

Rejected Transactions (Server Retry):
<EMPTY>

Rejected Transactions (Retry Different):
<EMPTY>

Unexpected SQL Errors:
com.oltpbenchmark.benchmarks.temporal.procedures.CheckForeignKeyRangeAgg/01      [80861] ********************************************************************************
com.oltpbenchmark.benchmarks.temporal.procedures.CheckForeignKeyLag/02           [80764] *******************************************************************************
com.oltpbenchmark.benchmarks.temporal.procedures.CheckForeignKeyExists/03        [80478] *******************************************************************************

More than half of the transactions were an invalid reference.

And if we put one of those into EXPLAIN ANALYZE, we see that most of the plan was never executed:

Result (actual time=0.034..0.035 rows=0 loops=1)
  One-Time Filter: ((InitPlan 1).col1 AND (InitPlan 2).col1 AND (NOT (InitPlan 4).col1))
  InitPlan 1
  ->  LockRows (actual time=0.033..0.033 rows=0 loops=1)
    ->  Index Scan using employees_pkey on employees x (actual time=0.033..0.033 rows=0 loops=1)
      Index Cond: ((id = 5999) AND (valid_at && '[2020-10-10,2020-12-12)'::daterange))
      Filter: ((COALESCE(lower(valid_at), '-infinity'::date) <= '2020-10-10'::date) AND ('2020-10-10'::date < COALESCE(upper(valid_at), 'infinity'::date)))
  InitPlan 2
  ->  LockRows (never executed)
    ->  Index Scan using employees_pkey on employees x_1 (never executed)
      Index Cond: ((id = 5999) AND (valid_at && '[2020-10-10,2020-12-12)'::daterange))
      Filter: ((COALESCE(lower(valid_at), '-infinity'::date) < '2020-12-12'::date) AND ('2020-12-12'::date <= COALESCE(upper(valid_at), 'infinity'::date)))
  InitPlan 4
  ->  LockRows (never executed)
    ->  Index Scan using employees_pkey on employees pk1 (never executed)
      Index Cond: ((id = 5999) AND (valid_at && '[2020-10-10,2020-12-12)'::daterange))
      Filter: (('2020-10-10'::date < COALESCE(upper(valid_at), 'infinity'::date)) AND (COALESCE(upper(valid_at), 'infinity'::date) < '2020-12-12'::date) AND (NOT EXISTS(SubPlan 3)))
      SubPlan 3
      ->  LockRows (never executed)
        ->  Index Scan using employees_pkey on employees pk2 (never executed)
          Index Cond: (id = pk1.id)
          Filter: ((COALESCE(lower(valid_at), '-infinity'::date) <= COALESCE(upper(pk1.valid_at), 'infinity'::date)) AND (COALESCE(upper(pk1.valid_at), 'infinity'::date) < COALESCE(upper(valid_at), 'infinity'::date)))

In this example, the beginning of the referencing range wasn’t covered, so Postgres never had to check the rest. Essentially the query is a AND b AND c, so Postgres can short-circuit the evaluation as soon as it finds a to be false. Using range_agg or lag doesn’t allow this, because an aggregate/window function has to run to completion to get a result.

As confirmation (a bit gratuitous to be honest), I ran the EXISTS benchmark with this bpftrace script:

// Count how many exec nodes per query were required,
// and print a histogram of how often each count happens.
// Run this for each FK implementation separately.
// My hypothesis is that the EXISTS implementation calls ExecProcNode far fewer times,
// but only if the FK is invalid.

u:/home/paul/local/bench-*/bin/postgres:standard_ExecutorStart {
  @nodes[tid] = 0
}
u:/home/paul/local/bench-*/bin/postgres:ExecProcNode {
  @nodes[tid] += 1
}
u:/home/paul/local/bench-*/bin/postgres:standard_ExecutorEnd {
  @calls = hist(@nodes[tid]);
  delete(@nodes[tid]);
}

For EXISTS I got this histogram when there were no invalid references:

@calls:
[0]                    6 |                                                    |
[1]                    0 |                                                    |
[2, 4)                 0 |                                                    |
[4, 8)            228851 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[8, 16)                1 |                                                    |
[16, 32)               1 |                                                    |
[32, 64)               2 |                                                    |
[64, 128)              2 |                                                    |
[128, 256)             2 |                                                    |
[256, 512)             5 |                                                    |

But with 50%+ errors I got this:

@calls:
[0]                    6 |                                                    |
[1]                    0 |                                                    |
[2, 4)            218294 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[4, 8)            183438 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@         |
[8, 16)              231 |                                                    |
[16, 32)               1 |                                                    |
[32, 64)               2 |                                                    |
[64, 128)              2 |                                                    |
[128, 256)             2 |                                                    |
[256, 512)             5 |                                                    |

So more than half the time, Postgres ran the query with half the steps (maybe one-fourth).

After tuning the random numbers to bring errors closer to 1%, I got results more like the original ones. Mean latency:

Median latency:

95th percentile latency:

Conclusions

All foreign key implementations have expected query plans. We use indexes where we should, etc.

When most foreign key references are valid, range_agg outperforms the other two implementations by a small but consistent amount. But with a large number of invalid references, EXISTS is a lot faster.

In most applications I’ve seen, foreign keys are used as guardrails, and we expect 99% of checks to pass (or more really). When using ON DELETE CASCADE the situation is different, but these benchmarks are for NO ACTION or RESTRICT, and I don’t think CASCADE affords the EXISTS implementation the same shortcuts. So it seems right to optimize for the mostly-valid case, not the more-than-half-invalid case.

These results are good news, because range_agg is also more general: it supports multiranges and custom types.

Further Work

There are more things I’d like to benchmark (and if I do I’ll update this post):

Replace separate start/end comparisons with range operators in the EXISTS and lag implementations. I just need to make sure they still pass all the tests when I do that.
Correct the Infinity-vs-null discrepancy.
Monitor the CPU and disk activity under each implementation and compare the results. I don’t think I’ll see any difference in disk, but CPU might be interesting.
Compare different scale factors (i.e. starting number of employees/positions).
Compare implementations when an employee is chopped into many small records, and a single position spans all of them. If range_agg is O(n²) that should be worse than the sorting in the other options.
Compare temporal foreign keys to non-temporal foreign keys (based on B-tree indexes, not GiST). I’m not sure yet how to do this in a meaningful way. Of course b-trees are faster in general, but how do I use them to achieve the same primary key and foreign key constraints? Maybe the best way is to create the tables without constraints, give them only b-tree indexes, and run the direct SELECT statements, not the DML.

PDXPUG Talk: Benchbase and Temporal Foreign Keys

2024-08-26T00:00:00Z

Last Thursday I gave a talk at PDXPUG about using Benchbase to compare the performance of temporal foreign keys. It was a lot of fun, and a really good turnout. There were even folks from Seattle and Bend. After listening for an hour, people stuck around and talked about databases and benchmarks for another two, then the last few holdouts went out for drinks for another hour and a half. At least half the audience were way more qualified to give the talk than me. To my surprise Mark Callaghan was there, who has published database benchmarks non-stop for years.

I had two major goals: to document how to use Benchbase and to report on comparing three implementations of temporal foreign keys. A couple minor goals were to share the start of a broader general-purpose benchmark for temporal databases and to talk about a benchmarking methodology, especially mistakes I made and how I tried to improve.

Temporal Ops

2024-07-17T00:00:00Z

One silver lining of temporal primary & foreign keys getting reverted is I got to meet Hettie Dombrovskaya and Boris Novikov.

I’ve been working with them to write SQL for various temporal operations not covered by the SQL:2011 standard. There is no support there for outer joins, semijoins, antijoins, aggregates, or set operations (UNION, INTERSECT, EXCEPT). As far as I know no one has ever shown how to implement those operations in SQL. I have queries so far for outer join, semijoin, and antijoin, and I’m planning to include aggregates based on this article by Boris. The set operations look pretty easy to me, so hopefully I’ll have those soon too.

If you’re interested, the repo is on Github.

Temporal Reverted

2024-07-05T00:00:00Z

My work adding temporal primary keys and foreign keys to Postgres was reverted from v17. The problem is empty ranges (and multiranges). An empty range doesn’t overlap anything, including another empty range. So 'empty' && 'empty' is false. But temporal PKs are essentially an exclusion constraint using (id WITH =, valid_at WITH &&). Therefore you can insert duplicates, as long as the range is empty:

INSERT INTO t (id, valid_at, name) VALUES (5, 'empty', 'foo');
INSERT INTO t (id, valid_at, name) VALUES (5, 'empty', 'bar');

That might be okay for some users, but it surely breaks expectations for others. And it’s a questionable thing to do that we should probably just forbid. The SQL standard forbids empty PERIODs, so we should make sure that using plain ranges does the same. Adding a record with an empty application time doesn’t really have a meaning in the temporal model.

I think this is a pretty small bump in the road. At the Postgres developers conference we found a good solution to excluding empty ranges. My original attempt used CHECK constraints, but that had a lot of complications. Forbidding them in the executor is a lot simpler. I’ve already sent in a new set of patches for v18 that implement that change.

Temporal PKs Merged!

2024-01-24T00:00:00Z

UPDATE: My temporal patches were reverted from v17. Hopefully they will be accepted for v18 instead.

Today first thing in the morning I saw that the first part of my temporal tables work for Postgres got merged. It was two patches actually: a little one to add a new GiST support function and then the main patch adding support for temporal primary keys and unique constraints based on range types. The support for SQL:2011 PERIODs comes later; for now you must use ranges—although in my opinion that is better anyway. Also this patch allows multiranges or, keeping with Postgres’s long history of extensibility, any type with an overlaps operator. So unless some big problem appears, PKs and UNIQUE constraints are on track to be released in Postgres 17.

Probably I can get (basic) foreign keys into v17 too. Temporal update/delete, foreign keys with CASCADE, and PERIODs will more likely take ’til 18.

If you are interested in temporal features, early testing is always appreciated! :-)

Getting this into Postgres has been a ten-year journey, and the rest of this post is going to be a self-indulgent history of that work. You’ve been warned. :-)

It started in 2013 when I kept noticing my clients needed a better way to track the history of things that change over time, and I discovered Richard Snodgrass’s book Developing Time-Oriented Database Applications in SQL. He offered a rigorous, systematic approach, with working SQL solutions for everything. This was exactly what I needed. His approach was vastly better than the ad hoc history-tracking I’d seen so far. But no one had implemented any of it!

My first Postgres patch in 2015 was motivated by temporal databases: I added UUID support to the btree_gist extension. A temporal primary key is basically an exclusion constraint on (id WITH =, valid_at WITH &&), and I had a project with UUID ids. But that exclusion constraint requires a GiST index that knows how to perform equal comparisons against the id column and overlap comparisons against the valid_at column. Out-the-box GiST indexes can’t do that (unless your ids are something weird like range types). If your ids are integers, you can install btree_gist to create a GiST opclass that knows what integer = means, but at the time UUIDs were not supported. So I started there. I liked that temporal databases had a manageable feature set and a manageable body of literature, so that even a working programmer like me could break new ground (not like Machine Learning or even Time Series databases). Nonetheless that patch took a year and a half to get committed, and it was really other people like Chris Bandy who finished it.

I kept reading about temporal databases, and in 2017 I wrote a proof-of-concept for temporal foreign keys, mostly at AWS Re:Invent. I happened to be given a free registration & hotel room, but it was too late to register for any of the good talks. But all that time with nothing to do was fantastically productive, and I remember by the flight home I was adding tons of tests, trying to cover every feature permutation—ha, as if. A few days after I returned I also published my annotated bibliography, which I’ve updated many times since.

In Snodgrass a temporal foreign key is a page-and-a-half of SQL, mostly because a referencing row may need more than one referenced row to completely cover its time span. But I realized we could make the check much simpler if we used an aggregate function to combine all the relevant rows in the referenced table first. So I wrote range_agg, first as an extension, then as a core patch. Jeff Davis (who laid the foundation for temporal support with range types and exclusion constraints) said my function was too narrow and pushed me to implement multiranges, a huge improvement. Again it took a year and a half, and I had trouble making consistent progress. There was a lot of work at the end by Alvaro Herrera and Alexander Korotkov (and I’m sure others) to get it committed. That was a few days before Christmas 2020.

Although the Postgres review process can take a long time, I cherish how it pushes me to do better. As a consultant/freelancer I encounter codebases of, hmm, varying quality, and Postgres gives me an example of what high standards look like.

One thing I still remember from reading Programmers at Work many years ago was how many inteviewees said they tried to build things at a higher level of abstraction than they thought they’d need. I’ve seen enough over-engineered tangles and inner-platform effects that my own bias is much closer to YAGNI and keeping things concrete, but the advice in those interviews still prods me to discover good abstractions. The Postgres codebase is full of things like that, and really it’s such a huge project that strong organizing ideas are essential. Multiranges was a great example of how to take a concrete need and convert it into something more general-purpose. And I thought I was doing that already with range_agg! I think one thing that makes an abstraction good is a kind of definiteness, something opinionated. So it is not purely general, but really adds something new. It always requires an act of creation.

The coolest thing I’ve heard of someone doing with multiranges was using them in astronomy to search for neutrinos, gravitational waves, and gamma-ray bursts. By using multiranges, they were able to compare observations with maps of the night sky “orders of magnitude faster” than with other implementations. (Hopefully I’ve got that right: I read a pre-print of the paper but it was not all easy for me to understand!)

My first patch for an actual temporal feature was primary keys back in 2018. Then foreign keys followed in 2019, just a couple weeks before I gave a talk at PgCon about temporal databases. By the end of the year I had FOR PORTION OF as well. At first FOR PORTION OF was implemented in the Executor Phase, but when I gave a progress report for PgCon 2020 I was already working on a trigger-based reimplementation, though it wasn’t submitted until June 2021. I also pulled in work by Vik Fearing from 2018 to support ADD/DROP PERIOD.

Soon after that progress got harder: my wife and I had our sixth baby in August, and somehow he seemed to be more work than the others. I took over daily math lessons (we homeschool), and I had to let go my biggest client, who needed more hours than I could give. (I’m proud to have given them an orderly transition over several months though.) In January 2022 Peter Eisentraut gave me a thorough review, but I went silent. Still, I had a lot of encouragement from the community, especially Corey Huinker, and eventually doing Postgres got easier again. I had a talk accepted for PgCon 2023, and I worked hard to submit new patches, which I did only weeks before the conference.

The best part of PgCon was getting everyone who cared about my work together in the hallway to agree on the overall approach. I had worried for years about using ranges as well as PERIODs, since the standard doesn’t know anything about ranges. The second-best part was when someone told me I should stop calling myself a Postgres newbie.

At PgCon Peter asked me to re-organize the patches, essentially implementing PERIODs as GENERATED range columns. It made the code much nicer. I also went back to an Executor Phase approach for FOR PORTION OF. Using triggers had some problems around updateable views and READ COMMITTED transaction isolation.

Since May I’ve felt more consistent than during my other Postgres work. I’ve been kept busy by excellent feedback by a meticulous reviewer, Jian He, who has caught many bugs. Often as soon as I get caught up, before I’ve written the email with the new patch files, he finds more things!

Another thing that’s helped is going out once a week (for nearly a year now) to get early dinner then work on Postgres at a local bar. Somehow it’s much easier to do Postgres from somewhere besides my home office, where I do all my normal work. Getting dinner lets me read something related (lately Designing Data-Intensive Applications by Martin Klepmann and PostgreSQL 14 Internals by Egor Rogov), and it’s fun. Doing just a little every week helps me keep momentum, so that fitting in further progress here and there seems easy. I’m lucky to have a wife who has supported it so often, despite leaving her with the kids and dishes.

I think I have years more work of temporal features to add, first finishing SQL:2011 then going beyond (e.g. temporal outer joins, temporal aggregates, temporal upsert). It’s been a great pleasure!

Survey of SQL:2011 Temporal Features

2019-09-04T00:00:00Z

Introduction

This blog post is a survey of SQL:2011 Temporal features from MariaDB, IBM DB2, Oracle, and MS SQL Server. I’m working on adding temporal features to Postgres, so I wanted to see how other systems interpret the standard.

If you’re new to temporal databases, you also might enjoy this talk I gave at PGCon 2019.

In this post I cover both application-time (aka valid-time) and system-time, but I focus more on valid-time. Valid-time tracks the history of the thing “out there”, e.g. when a house was remodeled, when an employee got a raise, etc. System-time tracks the history of when you changed the database. In general system-time is more widely available, both as native SQL:2011 features and as extensions/plugins/etc., but is less interesting. It is great for compliance/auditing, but you’re unlikely to build application-level features on it. Also since it’s generated automatically you don’t need special DML commands for it, and it is less important to protect yourself with temporal primary and foreign keys.

At this point all the major systems I survey have some temporal support, although none of them support it completely. On top of that the standard itself is quite modest, although in some ways it can be interpreted more or less expansively.

The Standard

I’ll start by giving a quick overview of the standard. Here I’m working from the draft documents (downloaded from here), and my interpretation may not be correct. If you have any corrections please let me know! Also you can find a more complete description of the standard at this article by Kulkarni and Michels (pdf).

In SQL:2011 the gateway to temporal features is a PERIOD, which is something you declare on your table. It is a range-like structure derived from two existing date columns. (Actually the standard also supports timestamp and timestamp with time zone, but I’ll use date as a synecdoche throughout this post.)

Periods

You can declare a valid-time PERIOD when you create the table or afterwards:

CREATE TABLE t (
  id          INTEGER,
  valid_from  DATE,
  valid_til   DATE,
  PERIOD FOR valid_at (valid_from, valid_til)
);

You can call the PERIOD whatever you like except SYSTEM_TIME, which is magical and enables system-time features. Both of the PERIOD‘s source columns must be NOT NULL, and if not they are automatically converted to it. (Most databases do the same thing with a PRIMARY KEY.) Note that the NOT NULL requirement means to represent “forever” or “until further notice” you must use a sentinel value like 3000-01-01.

Naturally a PERIOD adds an implicit constraint that valid_from must be less than valid_til.

You can also define a SYSTEM_TIME period and ask the database to track changes for you:

CREATE TABLE t (
  id        INTEGER,
  sys_from  TIMESTAMP GENERATED ALWAYS AS ROW START,
  sys_til   TIMESTAMP GENERATED ALWAYS AS ROW END,
  PERIOD FOR system_time (valid_from, valid_til)
) WITH SYSTEM VERSIONING;

Technically the standard lets you use DATE columns for system-time periods, but it’s hard to imagine how that would work in practice. Really anything short of the RDBMS’s finest granularity could “squeeze out” some history.

Primary Keys

If you have a valid-time PERIOD then you can declare a temporal primary key when you create the table:

CREATE TABLE t (
  id          INTEGER,
  valid_from  DATE,
  valid_til   DATE,
  PERIOD FOR valid_at (valid_from, valid_til),
  CONSTRAINT tpk_t PRIMARY KEY (id, valid_at WITHOUT OVERLAPS)
);

A temporal primary key is a lot like a normal primary key, except the scalar part (here just id) does not have to be unique, as long as rows with the same key don’t overlap in time. In other words you can give product 5 one price today and another tomorrow, and there’s no contradiction. But if you have two rows with the same scalar key covering the same date, that’s a violation of temporal entity integrity.

Foreign Keys

Temporal referential integrity is like ordinary referential integrity, except the non-unique nature of temporal primary keys makes it trickier. In a temporal foreign key, the child row’s lifespan must be completely “covered” by one (or more!) rows in the parent table. In other words some parent record must exist for every moment the child record exists. You can declare a temporal foreign key between two tables that both have PERIODs, e.g.:

CREATE TABLE ch (
  id          INTEGER,
  valid_from  DATE,
  valid_til   DATE,
  t_id        INTEGER,
  PERIOD FOR valid_at (valid_from, valid_til),
  CONSTRAINT tpk_ch PRIMARY KEY (id, valid_at WITHOUT OVERLAPS),
  CONSTRAINT tfk_ch_to_t FOREIGN KEY (id, PERIOD valid_at)
    REFERENCES t (id, PERIOD valid_at)
);

Projecting

A PERIOD is not included in the projection when you SELECT * FROM t. It is questionable whether you can project it explicitly with SELECT *, valid_at FROM t, but since it’s not a full-fledged data type I’d say probably not.

Filtering

Also you can’t reference a PERIOD in most other contexts, e.g. as a function input, or a GROUP BY criterion, or when ORDERing, or joining. You can use it in a “period predicate”, which lets you test these period relationships:

overlaps
equals
contains
precedes
succeeds
immediately precedes
immediately succeeds

Either side of the relationship can use a previously-named PERIOD or an anonymous dynamically-constructed one, e.g.

x.valid_at OVERLAPS PERIOD(y.valid_from, y.valid_til)

It’s not clear to me where you can use a period predicate, although the standard groups it with other kinds of predicate under the <predicate> object, so maybe anywhere you like? This browsable BNF grammer makes it easy to see that a <predicate> can go anywhere that accepts a boolean expression, which can be used in a <search condition>, which is what you put into your WHERE clause, or a join’s ON, or a CASE WHEN, or lots of other places. If you have a firmer read of the standard here, let me know!

Also there is a special syntax for querying based on system-time. The standard doesn’t mention using it for valid-time, although you could imagine doing it:

SELECT * FROM t FOR SYSTEM_TIME AS OF t1
SELECT * FROM t FOR SYSTEM_TIME BETWEEN t1 AND t2
SELECT * FROM t FOR SYSTEM_TIME BETWEEN SYMMETRIC t1 AND t2
SELECT * FROM t FOR SYSTEM_TIME FROM t1 TO t2

If you ask for a limited time range, the stard/end columns do not get truncated to match your request. In other words, if you query FOR SYSTEM_TIME BETWEEN '2000-01-01' AND '2020-01-01', your result records’ sys_til attributes are still 3000-01-01 (or whatever your sentinel is).

DML

In UPDATE and DELETE commands you can restrict the timespan you want changed:

UPDATE  t
FOR PORTION OF valid_at FROM t1 TO t2
SET     ...
...

and

DELETE FROM t
FOR PORTION OF valid_at FROM t1 TO t2
...

These commands may require special transformations if they “hit” only part of an existing record. For example if you delete the middle of a longer timespan, then you need to replace the old big record with your new version plus two short records (one on each end). An update is the same: after changing the targeted portion, you’d have to insert new records to preserve each end of the original. The standard gives careful instructions here: the RDBMS should include these extra inserts within the “primary effect” of the operation.

There is no need for any special syntax for INSERT, nor for special transformations.

The standard doesn’t have anything to say about a MERGE statement (in Postgres ON CONFLICT DO UPDATE), except in the case of system-time tables, where there is no new syntax and it does what you’d expect.

Questions

Since a PERIOD is attached to a table and isn’t part of the relational model, it isn’t part of a result set. It gets lost when you query a table. That makes it hard to query non-table temporal data, like views, subqueries, CTEs, and set-returning functions. (This was a major criticism of the original TSQL2 proposal from the 90s.) Nonetheless I can imagine how SQL:2011 leaves open some workarounds, e.g. by letting you use anonymous PERIODs inside period predicates, and letting you use period predicates as widely as possible. Also you could argue that projecting a PERIOD is unnecessary since you already have the start and end columns. So if an RDBMS gave you deep support for period predicates, composing temporal results would still be possible—albeit awkward. In practice though, no one does this, as we will see.

SQL:2011 also has no support for joining temporal results. You can effect an inner join with the OVERLAPS operator, but not the other kinds.

Snodgrass suggested that temporal databases should “coalesce” results before presenting them or at least before saving them to a table. Coalescing means that when two rows have adjacent or overlapping timespans and all other attributs are identical, they get merged to become just one row. Duplicates are removed. This seems like good behavior, both for clarity and to avoid cutting up your data more and more finely as time goes on, but SQL:2011 doesn’t mention it.

There is also no explicit mention of how triggers combine with the new temporal DML operations.

MariaDB

MySQL doesn’t support any temporal features, but recent versions of MariaDB have started to add support. Version 10.3.4 (released Jan 2018) included system-time support; Version 10.4.3 (Feb 2019), valid-time.

System Time

MariaDB supports the normal syntax for declaring system-time tables, but you can also use this abbreviated syntax if you like:

CREATE TABLE t (
  id INT
) WITH SYSTEM VERSIONING;

That will automatically add pseudo-columns named ROW_START and ROW_END (which also don’t appear in SELECT * FROM t.

Or the standard syntax works too:

CREATE TABLE t (
  id        INTEGER,
  sys_from  TIMESTAMP(6) GENERATED ALWAYS AS ROW START,
  sys_til   TIMESTAMP(6) GENERATED ALWAYS AS ROW END,
  PERIOD FOR SYSTEM_TIME (valid_from, valid_til)
) WITH SYSTEM VERSIONING;

Either way, for a timestamp(6) column (which is what the docs use) it looks like the max future date is 2038:

MariaDB [temporal]> insert into t (id) values (2);
Query OK, 1 row affected (0.008 sec)

MariaDB [temporal]> select * from t2;
+------+----------------------------+----------------------------+
| id   | valid_from                 | valid_til                  |
+------+----------------------------+----------------------------+
|    2 | 2019-07-27 17:07:51.849190 | 2038-01-18 19:14:07.999999 |
+------+----------------------------+----------------------------+
1 row in set (0.004 sec)

That seems awfully soon to me.

You can use these three ways of asking for system-time filters:

SELECT * FROM t FOR SYSTEM_TIME AS OF '2020-01-01';
SELECT * FROM t FOR SYSTEM_TIME FROM '2020-01-01' TO '2030-01-01';
SELECT * FROM t FOR SYSTEM_TIME BETWEEN '2020-01-01' AND '2030-01-01';

MariaDB doesn’t know about BETWEEN SYMMETRIC.

You can also say FOR SYSTEM_TIME ALL, which is useful because the default (with no FOR SYSTEM_TIME at all) is to filter AS OF NOW().

MariaDB partially addresses the composability problem by letting you say FOR SYSTEM_TIME against a view, which “pushes down” the filter to the underlying tables. This even works if the view queries some non-system-time tables. Since every system-time PERIOD is named the same thing, the database can sensibly interpret FOR SYSTEM_TIME against your view.

System-Time Partitions

To prevent tables getting too large, you can automatically partition a table by its SYSTEM_TIME:

CREATE TABLE t (
  id INT
) WITH SYSTEM VERSIONING
  PARTITION BY SYSTEM_TIME (
    PARTITION p_hist HISTORY,
    PARTITION p_curr CURRENT
  );

That will keep current records in one partition and historical records in another. You can also have multiple historical partitions and ask the system to switch to the next one every n rows. You can also drop older partitions to keep your data growth under control.

Excluded Columns

To further economize on disk, you can qualify specific columns as WITHOUT SYSTEM VERSIONING to exclude them from history.

Application Time

Declaring an application-time PERIOD works, but you can’t include a temporal PRIMARY KEY:

CREATE TABLE t (
  id          INTEGER,
  valid_from  DATE,
  valid_til   DATE,
  PERIOD FOR valid_at (valid_from, valid_til),
  -- This next line breaks!:
  CONSTRAINT tpk PRIMARY KEY (id, valid_at WITHOUT OVERLAPS)
);

Naturally you can’t create temporal foreign keys either.

If you omit the NOT NULL for the PERIOD source columns (as above), they become NOT NULL automatically.

DML

In UPDATE and DELETE statements you can use FOR PORTION OF valid_at, per the standard. You can’t use an anonymous period:

UPDATE  t
FOR PORTION OF PERIOD (valid_from, valid_til)
SET     ...

(It’s hard to imagine why you’d want to though.)

Projecting

You can’t SELECT a period, named or anonymous:

SELECT * FROM t;
SELECT *, valid_at FROM t;
SELECT *, PERIOD (valid_from, valid_til) FROM t;

Filtering

In a SELECT you can’t use FOR valid_at to filter things. That’s a little sad but perhaps understandable since arguably the standard only requires FOR SYSTEM_TIME. But period predicates don’t work either. These were all errors for me:

SELECT * FROM t WHERE valid_at CONTAINS '2020-01-01';
SELECT * FROM t WHERE valid_at OVERLAPS PERIOD('2020-01-01', '2030-01-01');

So if you want to ask questions about your valid-time history, you need to query against the scalar date columns.

Triggers

You can declare triggers on valid-time tables, and the triggers do fire for the extra inserts. Here is what I did to test things:

CREATE TABLE thist (
  id INTEGER,
  old_valid_from DATE,
  old_valid_til DATE,
  new_valid_from DATE,
  new_valid_til DATE, op CHAR(1)
);

CREATE TRIGGER tins AFTER INSERT ON t
FOR EACH ROW
INSERT INTO thist VALUES 
(NEW.id, NULL, NULL, NEW.valid_from, NEW.valid_til, 'i');

CREATE TRIGGER tupd AFTER UPDATE ON t
FOR EACH ROW
INSERT INTO thist VALUES
(NEW.id, OLD.valid_from, OLD.valid_til, NEW.valid_from, NEW.valid_til, 'u');

CREATE TRIGGER tdel AFTER DELETE ON t
FOR EACH ROW
INSERT INTO thist VALUES
(OLD.id, OLD.valid_from, OLD.valid_til, NULL, NULL, 'd');

If you UPDATE in the middle of a larger record, you get two INSERTs for the unaltered ends followed by an UPDATE of the middle. (The INSERTs come first.) The NEW.valid_from and NEW.valid_til mark the part that is being inserted/updated, as you’d expect.

If you DELETE in the middle of a larger record, you also get two INSERTS followed by a DELETE of the part you touched. In the delete trigger the OLD.valid_{from,til} columns have their actual old values, not the slice you’re deleting. (This probably makes sense, but it feels a little too mechanical/literal. It means your DELETE trigger doesn’t know what slice of history you’re actually removing.)

Bitemporal

You can also define bitemporal tables!

IBM DB2

DB2 has the fullest temporal support of all the databases I examined. My tests used version 11.5.0.0 on Linux.

System Time

System-time works with a few syntax differences:

CREATE TABLE t (
  id        INTEGER NOT NULL PRIMARY KEY,
  sys_from  TIMESTAMP(12) NOT NULL GENERATED ALWAYS AS ROW BEGIN,
  sys_til   TIMESTAMP(12) NOT NULL GENERATED ALWAYS AS ROW END,
  PERIOD SYSTEM_TIME (sys_from, sys_til)
);

You have to omit WITH SYSTEM VERSIONING, and you have to explicitly make the period source columns NOT NULL. Also you say GENERATED ALWAYS AS ROW BEGIN not GENERATED ALWAYS AS ROW START. Finally it is PERIOD SYSTEM_TIME not PERIOD FOR SYSTEM_TIME.

The sentinel for “forever” is 9999-12-30-00.00.00.000000000000.

Application Time

DB2 supports many valid-time features—but only if you name the period BUSINESS_TIME. At IBM, it’s always business time! (I am shamelessly stealing this joke from my audience at PGCon 2019.)

Valid-time periods have the same syntax quirks as system-time.

You can define temporal primary keys!

According to the docs you can define temporal foreign keys, but I couldn’t make it work:

db2 => create table t2 (id integer not null, valid_from date not null, valid_til date not null, \
db2 (cont.) => t_id integer, period business_time (valid_from, valid_til), \
db2 (cont.) => constraint t2pk primary key (id, business_time without overlaps), \
db2 (cont.) => constraint tfk foreign key (t_id, period business_time) \
db2 (cont.) => references t (id, period business_time));
DB21034E  The command was processed as an SQL statement because it was not a 
valid Command Line Processor command.  During SQL processing it returned:
SQL0104N  An unexpected token "business_time" was found following "gn key 
(t_id, period".  Expected tokens may include:  "<space>".  SQLSTATE=42601

Someome else can’t make it work either, according to this forum thread.

ALTER TABLE failed for me too:

db2 => create table t2 (id integer not null, valid_from date not null, valid_til date not null, \
db2 (cont.) => t_id integer, period business_time (valid_from, valid_til), \
db2 (cont.) => constraint t2pk primary key (id, business_time without overlaps));
DB20000I  The SQL command completed successfully.
db2 => alter table t2 add constraint tfk foreign key (t_id, period business_time) \
db2 (cont.) => references t (id, period business_time);
DB21034E  The command was processed as an SQL statement because it was not a 
valid Command Line Processor command.  During SQL processing it returned:
SQL0104N  An unexpected token "business_time" was found following "gn key 
(t_id, period".  Expected tokens may include:  "<space>".  SQLSTATE=42601

If I learn a way to make it work, I’ll update this article.

Projecting

As usual SELECT * FROM t does not give you the period, and SELECT *, valid_at FROM t is an error. Periods are not first-class types.

Filtering

DB2 nicely interprets the standard generously and lets you use the system-time SELECT syntax for application-time too:

SELECT * FROM t FOR business_time FROM t1 TO t2
SELECT * FROM t FOR business_time AS OF t1

but not:

SELECT * FROM t FOR business_time BETWEEN t1 AND t2

I couldn’t get any of the period predicates to work, e.g.:

db2 => select * from t where business_time contains '2015-01-01';
SQL0104N  An unexpected token "contains" was found following "where 
business_time".  Expected tokens may include:  "CONCAT".  SQLSTATE=42601

I also couldn’t do anything creative with anonymous periods, e.g.:

db2 => select * from t for period(valid_from, valid_til) as of '2015-01-01';
SQL0104N  An unexpected token "period" was found following "select * from t 
for".  Expected tokens may include:  "<space>".  SQLSTATE=42601

IBM doesn’t even care if you call it business_time:

db2 => select * from t for period business_time(valid_from, valid_til) as of '2015-01-01';
SQL0104N  An unexpected token "period business_time" was found following 
"select * from t for".  Expected tokens may include:  "<space>".  
SQLSTATE=42601

That means temporal features are going to break down when used with views, subqueries, CTEs, and set-returning functions. A period is tied to a table, but not a result set.

DML

IBM DML is pretty standard. You can UPDATE or DELETE FOR PORTION OF BUSINESS_TIME FROM '2010-06-01' TO '2010-06-15'. The extra INSERTs happen as expected.

Triggers

Like MariaDB, DB2 does call triggers for the derived INSERTs. Here is some setup to add a row to thist whenever a trigger gets called:

create table thist (id integer, old_valid_from date, old_valid_til date, new_valid_from date, new_valid_til date, op char(1));

create trigger tins after insert on t referencing new as new \
for each row insert into thist values \
(NEW.id, null, null, NEW.valid_from, NEW.valid_til, 'i');

create trigger tupd after update on t referencing old as old new as new \
for each row insert into thist values \
(NEW.id, OLD.valid_from, OLD.valid_til, NEW.valid_from, NEW.valid_til, 'u');

create trigger tdel after delete on t referencing old as old \
for each row insert into thist values \
(OLD.id, OLD.valid_from, OLD.valid_til, null, null, 'd');

If we UPDATE FOR PORTION OF in the middle of a larger record, our INSERT trigger is called twice:

db2 => update t \
db2 (cont.) => for portion of business_time \
db2 (cont.) => from '2015-01-01' to '2016-01-01' \
db2 (cont.) => set foo = 'bar';
DB20000I  The SQL command completed successfully.
db2 => select * from t;

ID          VALID_FROM VALID_TIL  FOO

          1 01/01/2015 01/01/2016 bar       
          1 01/01/2020 01/01/2030 -         
          1 01/01/2010 01/01/2015 -         
          1 01/01/2016 01/01/2020 -         

  4 record(s) selected.

db2 => select * from thist;

ID          OLD_VALID_FROM OLD_VALID_TIL NEW_VALID_FROM NEW_VALID_TIL OP

          1 -              -             01/01/2010     01/01/2015    i 
          1 -              -             01/01/2016     01/01/2020    i 
          1 01/01/2010     01/01/2020    01/01/2015     01/01/2016    u 

  3 record(s) selected.

Bitemporal

Bitemporal works too!

Oracle

For my tests I used Oracle 19c (version 19.3) for Linux and ran it on CentOS 7.

System time

Oracle has its own way of tracking table history, so it doesn’t bother with SQL:2011 system-time.

Application time

Oracle lets you declare a PERIOD, but like MariaDb you can’t define a temporal primary key:

CREATE TABLE t (
  id          INTEGER,
  valid_from  DATE,
  valid_til   DATE,
  PERIOD FOR valid_at (valid_from, valid_til),
  -- This next line breaks!:
  CONSTRAINT tpk PRIMARY KEY (id, valid_at WITHOUT OVERLAPS)
);

Of course that means no foreign keys either.

One interesting thing is that a PERIOD doesn’t force your columns to NOT NULL:

SQL> desc t;        
 Name                                      Null?    Type

 ID                                                 NUMBER(38)
 VALID_FROM                                         DATE
 VALID_TIL                                          DATE

And that’s because nulls are allowed in PERIOD-source columns:

SQL> insert into t values (6, null, null);

1 row created.

SQL> select * from t where id = 6;

        ID VALID_FRO VALID_TIL

Projecting

When you say SELECT * FROM t you don’t get the period. You also can’t say this either, but in Oracle’s case it’s a parser error:

SELECT *, valid_at FROM t;

This doesn’t work either:

SELECT *, 1+1 FROM t;

But if you avoid the * you can select it!:

SQL> SELECT id, valid_from, valid_til, valid_at FROM t;

        ID VALID_FRO VALID_TIL   VALID_AT

         1 01-JAN-00 01-JAN-30      33426
         2 01-JAN-10 01-JAN-30      33426
         3 01-JAN-20 01-JAN-30      33426
         4 01-JAN-00 01-JAN-10      33426

The result doesn’t mean much to me though. Anyone have any ideas?

Filtering

Like in DB2 you are able to filter by a valid-time period, although the syntax is a little non-standard (and wordy):

SQL> SELECT * FROM t
  2  AS OF PERIOD FOR valid_at DATE '2005-01-01';

        ID VALID_FRO VALID_TIL

         1 01-JAN-00 01-JAN-30
         4 01-JAN-00 01-JAN-10
         6

Incidentally, you can see here that NULL in a period means “unbounded”. You can also make just one of the bounds NULL, and AS OF queries give the expected results. This is just like Postgres ranges! If Oracle does this for PERIODs, perhaps Postgres should too?

You can use BETWEEN too, but its syntax is similarly garbled:

SQL> SELECT * FROM t
  2  VERSIONS PERIOD FOR valid_at
  3  BETWEEN DATE '2025-01-01' AND DATE '2035-01-01';

        ID VALID_FRO VALID_TIL

         2 01-JAN-10 01-JAN-30
         1 01-JAN-00 01-JAN-30
         3 01-JAN-20 01-JAN-30
         6

Anonymous periods don’t seem to work though:

SQL> SELECT * FROM t
  2  AS OF PERIOD FOR (valid_from, valid_til) DATE '2005-01-01';
AS OF PERIOD FOR (valid_from, valid_til) DATE '2005-01-01'
                 *
ERROR at line 2:
ORA-00904: : invalid identifier

You also can’t use standard period predicates:

SQL> SELECT * FROM t WHERE valid_at CONTAINS DATE '2015-01-01';
SELECT * FROM t WHERE valid_at CONTAINS DATE '2015-01-01'
                               *
ERROR at line 1:
ORA-00920: invalid relational operator


SQL> SELECT * FROM t WHERE valid_at OVERLAPS PERIOD('2015-01-01', '2020-01-01');
SELECT * FROM t WHERE valid_at OVERLAPS PERIOD('2015-01-01', '2020-01-01')
                               *
ERROR at line 1:
ORA-00920: invalid relational operator

DML

Oracle doesn’t understand FOR PORTION OF:

SQL> UPDATE t FOR PORTION OF valid_at
  2  FROM DATE '2005-01-01' TO DATE '2006-01-01'
  3  SET id = 8 WHERE id = 1;
UPDATE t FOR PORTION OF valid_at
             *
ERROR at line 1:
ORA-00905: missing keyword


SQL> DELETE FROM t FOR PORTION OF valid_at
  2  FROM DATE '2005-01-01' TO DATE '2006-01-01'
  3  WHERE id = 1;
DELETE FROM t FOR PORTION OF valid_at
              *
ERROR at line 1:
ORA-00933: SQL command not properly ended

Triggers

In Oracle you can define triggers on tables with a valid-time period, but without temporal DML there are no interesting questions about how they should behave. Nonetheless here are the same triggers as above but in Oracle syntax (in case I ever want to test this in the future):

CREATE TABLE thist (
  id INTEGER,
  old_valid_from DATE,
  old_valid_til DATE,
  new_valid_from DATE,
  new_valid_til DATE, op CHAR(1)
);

CREATE TRIGGER tins AFTER INSERT ON t
FOR EACH ROW
BEGIN
INSERT INTO thist VALUES 
(:NEW.id, NULL, NULL, :NEW.valid_from, :NEW.valid_til, 'i');
END;
/

CREATE TRIGGER tupd AFTER UPDATE ON t
REFERENCING OLD AS OLD NEW AS NEW
FOR EACH ROW
BEGIN
INSERT INTO thist VALUES
(:NEW.id, :OLD.valid_from, :OLD.valid_til, :NEW.valid_from, :NEW.valid_til, 'u');
END;
/

CREATE TRIGGER tdel AFTER DELETE ON t
REFERENCING OLD AS OLD
FOR EACH ROW
BEGIN
INSERT INTO thist VALUES
(:OLD.id, :OLD.valid_from, :OLD.valid_til, NULL, NULL, 'd');
END;
/

MS SQL Server

I tested an evaluation copy of MS SQL Server 2017 (version 14.0.1000.169, RTM).

SQL Server doesn’t support application-time periods at all, just system-time.

System Time

The syntax for system-time tables is just a little non-standard:

CREATE TABLE dbo.t (
  id INTEGER PRIMARY KEY,
  valid_from datetime2 GENERATED ALWAYS AS ROW START,
  valid_til datetime2 GENERATED ALWAYS AS ROW END,
  PERIOD FOR SYSTEM_TIME (valid_from, valid_til)
) WITH (
  SYSTEM_VERSIONING = ON
);

Note the parens, the underscore, and the = ON.

The history is stored in a separate invisible table with a generated name. But you can query that table like any other, so if you want to give it a nicer name you can:

WITH (
  SYSTEM_VERSIONING = ON (HISTORY_TABLE = dbo.thist)
);

The valid_til sentinel will be 9999-12-31 23:59:59.9999999.

To ask about a certain time you can say any of these:

SELECT * FROM t FOR SYSTEM_TIME AS OF '2020-01-01';
SELECT * FROM t FOR SYSTEM_TIME BETWEEN '2020-01-01' AND '2030-01-01';
SELECT * FROM t FOR SYSTEM_TIME FROM '2020-01-01' TO '2030-01-01';

but not BETWEEN SYMMETRIC.

Conclusion

So basically everyone has at least one kind of PERIOD.

Everyone but Oracle has system-time (and they have another older approach).

The only database with temporal primary keys is DB2. They claim to have temporal foreign keys too, but I couldn’t make it work.

I was pleased that two databases let you select with FOR and a valid-time period. No one lets you build anonymous periods (in FOR, FOR PORTION OF, or elsewhere), and no one supports period predicates.

With temporal DML, the extra inserts seem to be consistent (between MariaDB and DB2), and both databases fire triggers on them the same way.

I hope this helps the Postgres community work out their own temporal behavior with respect to the standard. I think it was an interesting study in its own right, too. One thing I learned is that “every other RDBMS supports SQL:2011” is only sort of true, at least as of today. :-)

Temporal Databases Annotated Bibliography

2017-12-05T00:00:00Z

I’ve been reading about temporal databases for a few years now, so I think it’s time I share my bibliography and notes. This is presented in “narrative order”, so that you can get a sense of how the research has developed. This article somewhat overlaps a mini literature review I wrote on the Postgres hackers mailing list, but this article is more complete and in a place where I can keep it updated.

Temporal databases let you track the history of things over time: both the history of changes to the database (e.g. for auditing) and the history of the thing itself. They are not the same thing as time-series databases: whereas a time-series database has time-stamped events, a temporal database stores the history of things, typically by adding a start/end time to each row (so two timestamps, not one). With time-series the challenge is typically scale; with temporal the challenge is with complexity and correctness.

Research

Abdullah Tansel, James Clifford, Shashi Gadia, Sushil Jajodia, Arie Segev, and Richard T. Snodgrass (editors). Temporal Databases: Theory, Design, and Implementation. 1993. I only became aware of this in 2023. I own a copy now but haven’t read it cover to cover. I’m curious to see the variety of ideas back then, before things got standardized. It’s amazing how many old ideas are just forgotten. Codd’s “Extending the data base relational model to capture more meaning” from 1979 is famous for introducing NULL and outer joins, but no one seems to remember the second half, with graph databases and queries that dynamically build themselves by querying the system catalog. So I bet this book has creative ideas that could inspire modern developments.

Already this book answered one question I had. Can a planner reorder/transform temporal operations the same way it can non-temporal ones? In other words, do they have the same algebraic properties? On page 175 in the chapter on TQuel I found this:

Theorem 12 The following equivalences hold for the valid-time algebra.

Q ∪̂ R ≡ R ∪̂ Q

Q ⨯̂ R ≡ R ⨯̂ Q

σ̂_F₁(σ̂_F₂(Q)) ≡ σ̂_F₂(σ̂_F₁(Q))

Q ∪̂ (R ∪̂ S) ≡ (Q ∪̂ R) ∪̂ S

Q ⨯̂ (R ⨯̂ S) ≡ (Q ⨯̂ R) ⨯̂ S

Q ⨯̂ (R ∪̂ S) ≡ (Q ⨯̂ R) ∪̂ (Q ⨯̂ S)

σ̂_F(Q ∪̂ R) ≡ σ̂_F(Q) ∪̂ σ̂_F(R)

σ̂_F(Q −̂ R) ≡ σ̂_F(Q) −̂ σ̂_F(R)

π̂_X(Q ∪̂ R) ≡ π̂_X(Q) ∪̂ π̂_X(R)

Theorem 13 The distributive property of Cartesian product over difference, or Q ⨯̂ (R −̂ S) ≡ (Q ⨯̂ R) −̂ (Q ⨯̂ S), does not hold for the valid-time algebra.

I discuss this in more depth in “TQuel Paper: Implementing Temporal Operators in Postgres”.

Snodgrass, Richard T. Developing Time-Oriented Database Applications in SQL. 1999. The seminal work on temporal databases and still the most useful introduction I know. Covers the “combinatorial explosion” of non-temporal/state-temporal/system-temporal/bi-temporal tables, current/sequenced/non-sequenced queries, SELECT/INSERT/UPDATE/DELETE, different RDBMS vendors, etc. Very similar to the proposed TSQL2 standard that was ultimately not accepted but still influenced Teradata’s temporal support. Available as a free PDF from his website.

Hugh Darwen and C. J. Date. “An Overview and Analysis of Proposals Based on the TSQL2 Approach.” Latest draft 2005, but originally written earlier. Criticizes the TSQL2 proposal’s use of “statement modifiers”, especially their problems with composability when a view/subquery/CTE/function returns a temporal result. Available as a PDF.

Ralph Kimball and Margy Ross. The Data Warehouse Toolkit. 3rd edition, 2013. (2nd edition 2002, 1st 1996.) (My notes are based on reading the 2nd edition, but I don’t think there are major relevant changes.) This book is not about temporal databases per se, but in Chapter 4 (and scattered around elsewhere) he talks about dealing with data that changes over time (“Slowly Changing Dimensions”). His first suggestion (Type 1) is to ignore the problem and overwrite old data with new. His Type 2 approach (make a new row) is better but loses the continuity between the old row and the new. Type 3 fixes that but supports only one change, not several. This writing is evidence for the need to handle temporal data, and the contortions that result from not having a systematic approach. (These criticisms and the realization that Kimball is trying to solve the same problems as temporal databases come from Johnston’s second book below; I’m glad he made the connection between Kimball and temporal databases!) (pdf)

C. J. Date, Hugh Darwen, Nikos Lorentzos. Time and Relational Theory, Second Edition: Temporal Databases in the Relational Model and SQL. 2nd edition, 2014. (First edition published in 2002.) I read this in March 2018. If you’ve read Date & Darwen’s other works you know they are quite rigorous, often critical of SQL’s compromises vs the pure relational model (e.g. NULL and non-distinct results), and not always very practical. Nonetheless this book is full of great ideas, and I hope anyone thinking about implementing temporal features in an RDBMS product will read it. They mostly deal with state-time, not transaction-time, which is great since that’s where the field most needs attention, although they also show how to apply their ideas to transaction-time tables too.

The core idea is that a temporal table should really have a row for each second (or millisecond or day or whatever is your finest granularity of time), saying what was true at that moment. Then the primary key is just the second plus the regular primary key. This idea isn’t new, but they use it to define everything else in the book. This makes thinking about temporal queries a lot easier, so that if you ever have any hesitation about something, you can think about it in terms of one-row-per-second and it starts to be easy.

Of course no database could adopt that in practice, so they offer operators to transform rows with start/end ranges to one-row-per-second and back again. In almost every case you don’t have to use those operators, but they are a solid theoretical basis for the next step: defining interval-aware primary keys, foreign keys, and all the relational operators. Finally they show how you implement things without “materializing” the expanded version of the data.

Another idea implicit in that is that temporal concerns belong in the SQL operators, not as “statement modifiers” like in TSQL2. Really that might be the most important idea in the whole book, so I’m glad to see it being applied to Postgres in the papers by Dignös et al (below).

They also argue (vs SQL:2011) that a single interval column type is better than separate start/end columns, which I agree with. Strangely they go against all the existing research by using closed/closed intervals (e.g. [Jan2017,Dec2017]) instead of closed/open ([Jan2017,Jan2018)), without really giving much justification. They also avoid using NULL for an unbounded side (e.g. [Jan2017,NULL)), preferring a magic “end of time” date ([Jan2017,Jan3000)). I wasn’t surprised by the second decision, given the authors’ history, but the first was less explicable. In both cases their approach sadly marginalizes their work and imposes barriers to adopting it in real SQL products.

I really appreciate how these authors have insisted that valid time (and the same with transaction time) should be a regular column, not something different. That was a big part of their complaint against TSQL2. To be honest when I first read Snodgrass the idea of using pseudo-columns seemed so suboptimal it was hard to take seriously (“Of course they don’t mean it.”), so I’m glad these authors have insisted on pointing out that approach’s shortcomings. Unfortunately in SQL:2011 valid-time is still a pseudo-column made up of two regular date/time columns. The problem is in composability: using a temporal query as a subquery, view, or function result: it all works cleanly if the interval is just another input to your interval-aware operators, but not if you need some kind of extra pseudo-column metadata. I hope implementers will take their advice seriously and not build temporal features on such a distorting idea.

Something I disagreed with was their suggestion to use tables in sixth-normal form—basically every column gets its own table—since attributes can have different lifespans. I can see how that is purer, but it seems like just too much complexity. They probably suspected the same because they always show how to do things with either that approach or tables in a more traditional third-normal form (or BCNF if you prefer). Even that is slightly distorted in order to avoid NULLs, but you can easily look past that.

Finally, I appreciated that on page 282 they mention DDL on temporal databases. Like everyone they say it’s beyond the scope of their current discussion, but it’s a penetrating insight to say, “the database catalog might itself need to be treated as a temporal database.”

SQL:2011 Draft standard. (pdf) Personally I find the standard pretty disappointing. It uses separate start/end columns instead of built-in range types, although range types offer benefits like exclusion constraints and convenient operators for things like “overlaps” that are verbose to code correctly by hand. It only mentions inner joins, not the various outer joins, semi-joins (EXISTS), anti-joins (NOT EXISTS), or aggregates. Many of its features apply only to system-time, not application-time, even though applicaion-time is the more interesting and less-available feature. (There are lots of auditing add-ons, but almost nothing for tracking the history of things.) The syntax seems too specific, lacking appropriate generality. A lot of these drawbacks seem motivated by a goal that goes back to TSQL2: to let people add temporal support to old tables without breaking any existing queries. That has always seemed to me like an unlikely possibility, and an unfortunate source of distortions. I don’t expect something for free, and I don’t mind doing work to migrate a table to a temporal format, as long as the result is good. Instead we get an (ostensible) one-time benefit for a prolonged compromise in functionality and ease-of-use.

Krishna Kulkarni and Jan-Eike Michels. “Temporal Features in SQL:2011”. SIGMOD Record, September 2012. Nice overview of the temporal features included in the SQL:2011 standard. Here is a PDF of the paper. See also these slides by Kulkani.

Peter Vanroose. “Temporal Data & Time Travel in PostgreSQL,” FOSDEM 2015. (Slides as a pdf) Lots of good detail here about SQL:2011. I’d love to see a recording of this talk if it’s available, but I haven’t found it yet.

Tom Johnston and Randall Weis. Managing Time in Relational Databases: How to Design, Update and Query Temporal Data. 2010. I finished this in May/June 2019. Although it concentrates on the authors’ proprietary middleware add-on to MS SQL Server called Asserted Versioning, it nonetheless has many ideas for building practical temporal features on any RDBMS. I liked their creativity and willingness to improve on SQL:2011 when they could. (Technically their book predates the final form of the standard.) Their middleware approach did sometimes lead them to behavior that would be strange in a more SQL-oriented solution. For example their temporal updates and deletes can only target a single record by ID (but they also mention they have a fix for this), and if that ID is not found you get an error. In ordinary SQL you would not get an error if an UPDATE or DELETE didn’t change any rows, and on the other hand you could hit more rows than one by not using the primary key. They mention that at the time of writing they were trying to replace their middleware approach with an implemention using views and instead-of triggers (something that has been done elsewhere too), and I expect that would wind up behaving more like ordinary SQL. They also introduce an interesting idea of future-dating records’ system-time (“assertion-time” in their lingo) so that the database will not make them “effective” until you’re ready. This has some nice practical use-cases, but there are pretty serious drawbacks. In their system a future-dated record is essentially immutable, so you had better not change your mind! Despite this book’s many contributions, I think everything here was also covered in Johnston’s second book, which is probably what I’d read if I had to choose one.

Tom Johnston. Bitemporal Data: Theory and Practice. 2014. I felt like I found a kindred soul when I read how he connects database design and ontology, as I’ve always thought of programming as “applied philosophy.” Databases as Aristotelian propositional logic is inseparable from the mathematical set-based theory. Johnston gives helpful distinctions between the physical rows in the table, the assertions they represent, and the things themselves. Eventually this leads to a grand vision of connecting every assertion’s bitemporal (or tritemporal) history to its speaker, somewhat like some ideas in the Semantic Web, although this doesn’t sound very practical. Like Date he seems to be landing on something like sixth-normal form, with a view-like presentation layer to bring all the attributes back together again. (Also like Date he seems to acknowledge 6NF may not be practical.) He points out how unsatisfactory Kimball’s suggestions are. He also criticizes the limitations of SQL:2011 and offers some amendments to make it more useful. Describes a (patented) idea of “episodes” to optimize certain temporal queries.

Anton Dignös, Michael H. Böhlen, and Johann Gamper. “Temporal Alignment”, SIGMOD ’12. Amazing! Shows how to define temporal versions of every relational operator by a combination of the traditional operators and just two simple transforms, which they call “align” and “split”. Gives a very readable exposition of the new theory and then describes how they patched Postgres 9.0 and benchmarked the performance. I think this solves the composability problems Date objected to in TSQL2, and unlike SQL:2011 it is general and comprehensive. The focus is on state-time, and I’m not sure how it will map onto bi-temporal, but even just having good state-time functionality would be tremendous. And the paper is only 12 easy-to-read pages! (pdf)

Anton Dignös, Michael Hanspeter Böhlen, Johann Gamper, and Christian S. Jensen. “Extending the kernal of a relational DBMS with comprehensive support for sequenced temporal queries,” ACM Transactions on Database Systems, 41(4):1-46. Continues the previous paper but adds support for scaling the inputs to aggregate groups according to how much of their time period goes into each group. Gives more benchmarks against a patched Postgres 9.5. (pdf) These researchers are now trying to contribute their work to the Postgres core project, of which I am very much in favor. :-)

Tools

Finally here are some tools for temporal support in Postgres. The sad theme is that pretty much everything gives audit support but not history support:

Postgres

The most complete implementation of bi-temporal tables is Vik Fearing’s periods extension. Since Postgres uses a bison grammar, an extension can’t add new syntax, but this one does a great job working around that limitation to provide almost everything from the SQL:2011 standard.

Henrietta (Hettie) Dombrovskaya has a pg_temporal repository with talks, slides, and code. I haven’t tried building it but I believe her company is using this in production. She is an awesome speaker, so her talks are great intros to this topic too.

The pgaudit extension looks pretty useful but I haven’t tried it yet. According to the AWS docs you can even use this on RDS.

Vlad Arkhipov’s temporal tables extension only supports system-time (auditing). Also on Github and a nice writeup by Clark Dave.

Magnus Hagander presented an approach to capturing system-time history in a separate schema at PGConf US 2015 and PGDay’15 Russia. Here are slides and video. Quite elegant if you want to ask questions like “what did we think as of time t?” If I recall correctly this is similar to one of the ideas proposed at the end of Snodgrass, although I haven’t compared them carefully. Hagander points out that DDL changes against temporal databases are challenging and hopefully infrequent. This is a topic that is almost completely absent from the literature, except for a brief mention in Johnston 2014.

I’ve heard about a Julia project for temporal data in Postgres. I’m not sure how it works, but it looks interesting!

Ruby

Chronomodel extends the ActiveRecord ORM to record system-time history. The pg_audit_log gem works fine but like many audit solutions is rather write-only. I wouldn’t want to build any functionality that has to query its tables to reconstruct history. You could also try paper_trail or audited (formerly acts_as_audited). Of these projects only Chronomodel seems to be aware of temporal database research.

Further Research

Temporal databases are exciting because there is still so much to do. For example:

What should the UI look like? Even one dimension adds a lot of complexity, let alone bi-temporal. How do you present this to users? As usual an audit history is easier, and it’s possible to find existing examples, whereas a state-time history is more challenging but probably more valuable. How should we let people view and edit the history of something? How does it work if there is a “Save” button vs save-as-you-type?
What does “full stack” temporal support look like? Do we extend REST? What would be a nice ORM interface? Should we use triggers to hide the temporal behavior behind regular-looking SQL? Or maybe extend SQL so you can more explicitly say what you want to do?
SELECT support for “as of” semantics or “over time” semantics.
Temporal foreign keys. I have a POC Postgres extension and am making progress adding it to core.
DDL changes. For example if you want to add a NOT NULL column, what do you do with the old data? Could there be built-in support to apply constraints only to a given time span?
Future time. There are some cool ideas about this in Johnson’s books, and I’ve heard of fintech folks doing something here.
Temporal upsert. This would be an INSERT that falls back to FOR PORTION OF for spans that already exist. Quite often this is what an application really wants. Johnson talks about how the standard should have included something here.
Outer joins and aggregates. Dignös et al show how to do this, but it’s not in the standard. I’d love to get their old patch working again and try to get something similar into Postgres, once the basics are finished. I’ve written SQL implementations of temporal outer join, semi-join, anti-join, UNION, EXCEPT, and INTERSECT, but those won’t perform as well as built-in RDBMS support. If Postgres gets inlining PL/pgSQL functions it will be faster. Even better would be a C implementation using CustomScan.

Javascript Daylight Savings Time: One Weird Trick Your Application Hates

2017-11-08T00:00:00Z

I’ve talked in the past about how to handle timezones in Rails, so here is a tip for handling timezones in Javascript, in particular around Daylight Savings Time.

Suppose you have a time: April 3, 2017, at midnight Pacific Time. You want to express it as UTC in ISO 8601 format, for instance to send it over the wire as JSON. The result is "2017-04-03T07:00:00.000Z". Note the 07:00. Pacific Time is -8 hours from UTC during Standard Time, and -7 hours during Daylight Savings Time. April 3 falls in Daylight Savings Time.

Now suppose we change the year: April 3, 1969, still at midnight Pacific Time. DST started later that year, so now the answer is "1969-04-03T08:00:00.000Z". But if we run new Date(1969, 3, 3).toISOString() your browser gives us: . That might look correct, or you might see a 07:00 again.

Believe it or not, the original Javascript specification said that browsers should use the current year’s Daylight Savings Time transition dates when building dates from any year. If you just re-read that sentence in disbelief and still think it is too crazy to be real, here is a conversation with links to the old and new spec. I think it’s crazy too!

Right now, some browsers do the right thing (ignore the old spec), some do the wrong thing (follow the old spec), and it also depends on what version you’re running. It even seems to depend on what year you’re asking about. For instance modern Chrome seems to give me the right answers back to 1970, but then is wrong before that. Also, even if your browser does the wrong thing, you might still get lucky based on the current year and the date you’re trying to build. I wrote a jsbin page you can load in multiple browsers to see if they agree.

I think the only safe answer is to use moment-timezone to build your dates. For instance if you know the timezone:

moment.tz([y, m, d], tz)

or if you don’t:

moment.tz([y, m, d], moment.tz.guess())

(And don’t forget the m is off by one.)

If you need to force that back into a regular Date object, you could do:

new Date(moment.tz([y, m, d], tz).toJSON())

Just make sure that you’re using moment-timezone-with-data.js, not plain moment-timezone.js, or you’ll still be relying on the browser’s own idiosyncratic behavior.

I hope this is helpful to someone. If your users enter birthdays with some kind of date picker, you probably suffer from this bug!

Dates and Time Zones in Rails

2014-04-18T00:00:00Z

I work on a video education app that needs to report how many lessons were assigned each day. I have a timetracking app that needs to invoice based on when one month starts and another ends. And I’ve got another app for watching cronjobs, that counts how many failures happened in a given day. All these apps share a problem: if they get time zones wrong, they will give incorrect results. For instance, if a video lesson was assigned on Tuesday after 5pm PDT, its UTC time will be Wednesday. Even though all three apps only show dates, not times, time zones still matter. So here are some lessons I’ve learned dealing with Rails time zones, both programming myself and leading a team.

But first, here are two articles that lay some groundwork:

Both those posts describe some nice methods provided by ActiveSupport/Rails, as well as why you’d want to use them instead of built-in Ruby methods. For instance, saying Time.now will get you the time in your OS’s default time zone, whereas Time.zone.now will get you the time in whatever timezone you put in config/application.rb. It’s a good idea to use the Rails timezone, since you have more control over that and it’s part of your source control.

If you want to explore, take a look at ActiveSupport::TimeZone. You can get a specific time zone like this:

tz = ActiveSupport::TimeZone['Pacific Time (US & Canada)']

That tz will be the start of just about everything you do with time. An ActiveSupport::TimeZone is also what you get from Time.zone.

Those other articles are great, but this post talks about time zones from a higher level, aspiring to share best practices rather than helpful methods. In my examples I’ll assume a Postgres database, although there’s nothing here that won’t work on MySQL as well.

Scope Your Time Zones

The first thing to remember is that all times are relative to somebody. Probably that’s a user, but maybe it’s something else. For my video education app, sometimes it’s a teacher and sometimes it’s a student. For my cronjob app, it’s a job (and sometimes a user). But there is always a reference point. In that sense, the articles above that recommend Time.zone.now are wrong. Time.zone.now is no better than Time.now if you need many time zones—and who doesn’t? So it should really be user.time_zone.now. I’d recommend adding this method to your User class right from the beginning, so that even if you haven’t yet implemented user-specific time zones, you are still writing the rest of your app correctly:

def time_zone
  ActiveSupport::TimeZone['Pacific Time (US & Canada)']
end

Ignore Time Zones as Much as Possible

Now that you’ve got time zones everwhere, the next step is to get rid of them. Thinking about time zones is hard, and finding time zone bugs is hard. You want to avoid them as much as possible!

It’s been said that in C, if you start thinking about big endian vs little endian, you’re probably doing things wrong. Oftentimes when programmers half-understand something, they do more than they should, and that’s very true with time zones. Your code will be a lot easier to understand if you tackle time zones with a few well-placed strokes, rather than lots of fiddling all over the place.

One thing to remember is that 07:05:00 PDT and 14:05:00 UTC are the same instant. If you converted both to seconds since the epoch, you’d get the same number. So in that sense, changing the time zone doesn’t change anything: it’s just a bit of extra metadata hanging onto the time representing someone’s perspective. Knowing that x PDT and y UTC are the same instant is really helpful when you feel the urge to fiddle with timezones. Is your fiddling just a noop?

By default, Rails comes configured with its default time zone as UTC. Leave it that way! Since every time needs a reference point anyway, your code shouldn’t care about the Rails-wide setting. UTC is a good neutral choice. For one thing, it doesn’t have daylight savings time. And if you see it, you know you’re dealing with a time-zone-less value.

You should also leave your OS time zone as UTC, if possible. Keeping it consistent with Rails will remove one chance for abiguiuty. And again, it’s a good neutral.

You also want UTC in your database. If you use a migration to create columns with t.timestamps or t.datetime :foo, Rails will make a TIMESTAMP WITHOUT TIME ZONE column. You can think of this as a time in UTC if you like. Really it’s an int of (micro)seconds since the epoch. Whenever you give a time to ActiveRecord, it will convert it to UTC before it hits Postgres. Or more correctly, it will strip off the timezone part and give Postgres the int. Remember, it’s the same instant! But it’s nice to imagine the column as UTC. If you’re in psql and type SELECT created_at FROM lessons, that’s what you’re seeing.

When everything in your stack is UTC, it’s like looking through nice clear glass. You don’t have to think about conversions at each layer.

You should also strive to make your app code as time-zone-less as possible. The big principle here is to handle time zones once, hopefully at the beginning of the HTTP request, when you decide the correct reference point for all times. Usually that’s current_user.time_zone.

The second article above suggests you add this to your controller:

around_filter :user_time_zone, if: :current_user

def user_time_zone(&block)
  Time.use_zone(current_user.time_zone, &block)
end

That makes me pretty uncomfortable. The idea is that you can say Time.zone everywhere else in your app and always get the current user’s time zone. But I’d rather be explicit about where the time zone is coming from. (Also see below for why use_time doesn’t help at all for parsing times.) Instead, write code to take a tz argument if necessary (emphasis on the if necessary). This will make your code less surprising, less coupled, and easier to test.

Even better is to write your code to take just a time. Almost always that’s really what you want. Remember that no matter the time zone, it’s all the same instant. Usually a single time is sufficient, because it can be a reference point for creating other times, using t + 1.day or t + 2.weeks or whatever. Use a (user|cronjob|foo)-scoped time zone to get your first time, and then forget about time zones for the rest of the stack. If you implemented the User#time_zone method above, your time zone code is already well-encapsulated, so there’s no need for an around_filter to further abbreviate things.

Here is another approach I don’t like. This blog post suggests you deal with time zones in Postgres like so:

SELECT created_at AT TIME ZONE 'UTC' AT TIME ZONE 'US/Pacific'

Does that look strange to you? What’s happening is that you start with a TIMESTAMP WITHOUT TIME ZONE, so first you tack on a time zone (just metadata), then you convert it to Pacific time. In other words, you’re doing this:

time without zone -> assumed to be UTC -> converted to Pacific Time

It’s good to know that this is how to get a Postgres timestamp converted to whatever time zone you want, but I wouldn’t recommend it in a Rails app (which is the article’s context). For one thing, Rails and Postgres don’t use the same names for all timezones, so you have to maintain your own mapping between the two. This is one of those chores that will keep nagging you for the life of your app, so I’d rather just avoid it. But more important, this approach means you have to pass your timezone all the way down to the database layer. I’d rather deal with time zones early, get to a TimeWithZone (or even an int), and then forget about time zones for the rest of the code.

Remember Daylight Savings Time

This is a small tip, but be careful about DST. You should avoid ever writing a fixed offset or a string like “PDT”. This is wrong:

Time.new(2014, 4, 8, 3, 15, 0, '-07:00')

If you were after PDT, your code is broken during PST. If you were after MST, your code is broken during MDT. Similarly in SQL if you say AT TIME ZONE 'PDT', you’ve broken PST.

If you stick with the ActiveSupport::TimeZone instances, you can forget about DST, and things will just work. That’s why there is no such thing as this:

ActiveSupport::TimeZone['PDT']

only this:

ActiveSupport::TimeZone['Pacific Time (US & Canada)']

Even Dates are Times

When you think you’re just dealing with dates, time zones probably still matter. Was the lesson assigned on Tuesday or Wednesday? It depends. From the teacher’s perspective? The student’s? The principal’s? Unless you’re really sure, I’d recommend always storing a full date+time in your database. Also, in Ruby avoid converting things to Date. Of course if you read the articles above you know you don’t want this:

Date.today

But this isn’t any better:

Time.zone.today

Or even this:

current_user.time_zone.today

Here is a query I’ve seen:

Lesson.where("date(created_at) >= ?", tz.today - 3.days)

But that’s wrong, because you’re stripping off all the time zone information. It’s going to report Tuesday’s lessons in Wednesday. (It also means you need a database index on the expression date(created_at), which is probably less often useful than a normal index on just the column.)

To improve this query, I’d write it like this:

Lesson.where("created_at >= ?", tz.now.midnight - 3.days)

If today is Wednesday (for you), that will give all the lessons created since the beginning of Sunday. Because we’re carrying a full time all the way down to the database query, we don’t have problems with the definition of “today.”

If you are ever tempted to use today or to_date, you probably want these methods instead:

tz.now.midnight     # the start of today, 00:00:00
tz.now.end_of_day   # 23:59:59

Parsing times

Now that you’ve got a per-user time zone, you want to use that to parse date/time inputs from that user. If you user types “3:15”, you want to interpret that as 3:15 in the user’s time zone.

There is a parse method on ActiveSupport::TimeZone, but unfortunately no strptime. If you say this, you get the wrong result:

Time.strptime("2014-04-08 03:15", "%Y-%m-%d %H:%M").in_time_zone(tz)

That’s because strptime assumes the time is in the default time zone (hopefully UTC), and in_time_zone does not change the “instant” the time represents, only the perspective used to view it. If you’re lucky to have a standardish format, this will work:

tz.parse("2014-04-08 03:15")

If your format is something weirder, or you just don’t trust parse with its heuristic-based approach, then you’re probably out of luck. A previous version of this article recommended Time.use_zone(tz) { Time.strptime(...) }, but that doesn’t work, because use_time only changes Time.zone. It doesn’t change Time.strptime.

Another approach with strptime that doesn’t work is to concatenate the time zone to your string and use %Z (and friends) in your format, like this:

Time.strptime('2014-04-08 03:15' + ' ' + tz.formatted_offset, "%Y-%m-%d %H:%M %:z")

The problem here is that tz.formatted_offset really depends on the time of year because of Daylight Savings Time. But you don’t know that until you parse the string. You could pass the timezone abbreviation instead, like PDT, but that has the same problem. And %Z doesn’t understand long names like Pacific Time (US & Canada).

The only approach I know that works is to parse the time with your default time zone, then feed the bits into tz.local, like so:

t = Time.strptime("2014-04-08 03:15", "%Y-%m-%d %H:%M")
tz.local(t.year, t.month, t.mday, t.hour, t.min)

Sorry, that’s the best I can do!

UPDATE: I added a strptime method to the TimeZone class (my first Rails code contribution), so hopefully you’ll start seeing it in future versions!

Displaying times

So much for times as inputs. Times as outputs is a lot easier. Your rule should be to ignore time zones until the last minute, when you actually format the value for rendering. So it should go right into your view:

= lesson.created_at.in_time_zone(current_user.time_zone).strftime("%A, %B %-d, %Y")

You might want a helper for this though, something like:

def format_time(time, tz, format)
  time.in_time_zone(tz).strftime(format)
end

Then your Haml can be:

= format_time(lesson.created_at, current_user.time_zone, "%A, %B %-d, %Y")

Or de-parameterize that as much as you like:

def format_time(time)
  time.in_time_zone(current_user.time_zone).strftime("%A, %B %-d, %Y")
end

and:

= format_time(lesson.created_at)

Conclusion

So in general, my principles for handling times in Rails are:

Even dates are times.
Set “global” time zones to UTC everywhere you can.
Don’t ever use global time zones; scope it to a user or whatever is appropriate.
To most of your code, a time is just an instant.
Push time zone inputs to the very beginning of your request, to the top of your stack.
Push time zone outputs to the very end of your request, when rendering the view.

Good luck!

Illuminated Computing

TQuel Paper: Implementing Temporal Operators in Postgres

Benchmarking Temporal Foreign Keys

Method

range_agg Implementation

EXISTS implementation

lag implementation

Hypothesis

Results

Conclusions

Further Work

PDXPUG Talk: Benchbase and Temporal Foreign Keys

Temporal Ops

Temporal Reverted

Temporal PKs Merged!

Survey of SQL:2011 Temporal Features

Introduction

The Standard

Periods

Primary Keys

Foreign Keys

Projecting

Filtering

DML

Questions

MariaDB

System Time

System-Time Partitions

Excluded Columns

Application Time

DML

Projecting

Filtering

Triggers

Bitemporal

IBM DB2

System Time

Application Time

Projecting

Filtering

DML

Triggers

Bitemporal

Oracle

System time

Application time

Projecting

Filtering

DML

Triggers

MS SQL Server

System Time

Conclusion

Temporal Databases Annotated Bibliography

Research

Tools

Postgres

Ruby

Further Research

Javascript Daylight Savings Time: One Weird Trick Your Application Hates

Dates and Time Zones in Rails

Scope Your Time Zones

Ignore Time Zones as Much as Possible

Remember Daylight Savings Time

Even Dates are Times

Parsing times

Displaying times

Conclusion

`range_agg` Implementation

`EXISTS implementation`

`lag` implementation