Illuminated Computing

Rails ActionMailer Internals

2023-10-16T00:00:00Z

When it comes to sending email in Rails, I’ve wondered for years about the gap between this:

class UserMailer < ApplicationMailer
  def welcome(user)
    @user = user
    mail(to: user.email)
  end
end

and this:

class User
  def send_welcome_notification
    UserMailer.welcome(self).deliver_later
  end
end

We are defining an instance method, but we are calling a class method. What’s going on there? I finally decided to take a closer look.

Well naturally this is implemented by method_missing. When you call UserMailer.welcome, the class will call your instance method—sort of! Actually method_missing just returns a MessageDelivery object, which provides lazy evaluation. It’s like a promise (but not asynchronous). Your method doesn’t get called until you resolve the “promise,” which normally would happen when you say deliver_now. You can also call #message which must resolve the promise (and returns whatever your method returned—sort of!).

What if you say deliver_later? That still doesn’t call your method. Instead it queues up a job, and later that will say deliver_now to finally call your method.

But if you’re using Sidekiq (with config.active_job.queue_adapter = :sidekiq), you might wonder how that #welcome method works, since we’re passing a User class and Sidekiq can only serialize primitive types. But it does work! The trick is that Rails’ queue adapter for Sidekiq does its own serialization before handing off the job to Sidekiq, and it tells Sidekiq to run its own Worker subclass that will deserialize things correctly.

All this assumes that your mailer method returns a Mail::Message instance. That’s what #mail is giving you. But what if you don’t? What if you call mail but not as the last line of your method? What if you call it more than once?

Well actually #mail (linking to the source code this time) remembers the message it generated, so even if you don’t return that from your own method, Rails will still send it properly. In fact it doesn’t matter what your own method returns!

And if you call #mail multiple times, then Rails will return early and do nothing for the second and third calls—sort of! If you pass any arguments or a block, then Rails will evaluate it again. But it still only knows how to store one Message. So when you finally call deliver_now, only one email will go out (ask me how I know).

Btw it turns out this is pretty much all documented on the ActionMailer::Base class, but it’s not really covered in the Rails Guide, so I never came across it. I only found those docs when I decided to read the code. I don’t know if other Rails devs spend much time reading Rails’ own code, but I’ve found it helpful again and again. It’s not hard and totally worth it!

Another trick I’ve used for years is bundle show actionmailer (or in the old days cd $(bundle show actionmailer), before they broke that with a deprecation notice), and then you can add pp or binding.pry wherever you like. It’s a great way to test your understanding of what’s happening or discover the internals of something.

Rails dirty methods

2023-06-29T00:00:00Z

Rails has lots of methods to see what attributes have changed on your model. Some tell you the changes you haven’t yet saved; some, the changes you just saved. But the behavior and names of these attributes have changed over time.

I thought I had a handle on this until I saw saved_change_to_attribute? and wondered how it differs from attribute_previously_changed?. Turns out they are identical!

Well sort of. The spelling I’m used to, attribute_previously_changed?, comes from ActiveModel::Dirty (and is a bit older), whereas saved_change_to_attribute? is defined in ActiveRecord::AttributeMethods::Dirty. Not all ActiveModels are ActiveRecords. But in your ActiveRecord classes, they do the same thing.

I’ve linked to Rails 6.1 here. They were nearly identical before that, but for a while one took extra options and the other didn’t. You have to go back to Rails 5.0 to get a more substantial difference, when we had attribute_previously_changed? but not saved_change_to_attribute?. They are still identical today in Rails 7. I’m surprised they don’t deprecate the ActiveRecord methods and just use ActiveModel.

Just to give a quick catalog, here is the full set of methods. Anywhere you see attribute you can replace it with the name of the attribute you care about (which just calls the generic method with its name as parameter).

before you save:

changes
changed_attributes              # can't replace "attribute"
attribute_change
attribute_changed?
attribute_was

changes_to_save
has_changes_to_save?
attributes_in_database          # can't replace "attribute"
attribute_in_database
changed_attribute_names_to_save # can't replace "attribute"
attribute_change_to_be_saved
will_save_change_to_attribute?

after you save:

previous_changes
attribute_previous_change
attribute_previously_changed?
attribute_previously_was

saved_changes
saved_changes?
saved_change_to_attribute
saved_change_to_attribute?
attribute_before_last_save

I’ve grouped the methods from each file, and you can see there are many synonyms.

By the way if you are making heavy use of ActiveRecord callbacks and using these methods to trigger them (e.g. after_commit :send_shipped_notification if: :shipped_at_previously_changed?), watch out! The conditions on these get evaluated one-by-one, so if some earlier callback saves further changes to the model, your old previous_changes are lost! The callback you expect to get called just doesn’t. I’ve had to debug that failure way too many times.

Testing Your ActionMailer Configuration

2018-08-04T00:00:00Z

Sometimes with Rails you just want to test basic email sending from the console. Here is how you can do it, without needing a Mailer or views or anything:

ActionMailer::Base.mail(
  to:      'dest@example.com',
  from:    'src@example.com', 
  subject: 'test',
  body:    'testing',
).deliver_now

Or in older versions use deliver instead of deliver_now. This will use your ActionMailer settings but doesn’t require anything else, so you can test just the SMTP config (or whatever).

Adding an if method to ActiveRecord::Relation

2018-06-30T00:00:00Z

I often find myself chaining ActiveRecord scopes, like this:

q = Article
  .not_deleted
  .published_within(start_date, end_date)
q = q.with_tags(tags) if tags.any?

I wish there were a nicer way to put in the conditional scopes, without assigning to a temporary variable. What I really want is to write this:

Article
  .not_deleted
  .published_within(start_date, end_date)
  .if(tags.any?) { with_tags(tags) }

Wouldn’t that be nicer?

Many years ago I brought this up on the pdxruby mailing list, and no one seemed very interested, but I’ve always wanted it in my projects.

Here is a naïve implementation, which just for simplicity I’ll add to Object instead of ActiveRecord::Relation (where it really belongs):

class Object

  def if(condition, &block)
    condition ? instance_eval(&block) : self
  end

end

5.if(true)  { self + 7 }    # equals 12
5.if(false) { self + 7 }    # equals 5

That almost works! The problem is that inside the block, things don’t really act like a closure. If we used this implementation, our example above would give a NameError about not finding a tags method or local variable. That’s because everything inside the block is evaluated with self set to the ActiveRecord::Relation instance.

Fortunately there is a way to fix it! Before calling the block, we can save the outside self, and then we can use method_missing to delegate any failures to there.

There is a nice writeup of this “cloaking” trick if you want more details. But if you read that article, perhaps you will notice it is not thread-safe, because it temporarily adds a method to the class, and two threads could stomp on each other if they did that at the same time.

This approach was in Rails ActiveSupport for a while as Proc#bind. They even fixed the multi-threading problem (more or less . . .) by generating a different method name every time. Unfortunately that created a new problem: since define_method takes a symbol, this creates more and more symbols, which in Ruby are never garbage collected! Effectively this is a memory leak. Eventually the Rails team deprecated it.

But we can still add something similar that doesn’t leak memory and is thread-safe. We just have to protect the brief moment when we define the method and then remove it, which is simple with a Mutex. In theory taking a lock adds some overhead, and possibly contention, but we don’t expect this to be a “hot spot”, so in practice the contention should be zero and the overhead trivial.

And here is our implementation (not on Object any more):

module ActiveRecord
  class Relation

    CLOAKER_MUTEX = Mutex.new

    def if(condition, &block)
      if condition
        meth = self.class.class_eval do
          CLOAKER_MUTEX.synchronize do
            define_method :cloaker_, &block
            meth = instance_method :cloaker_
            remove_method :cloaker_
            meth
          end
        end
        with_previous_context(block.binding) { meth.bind(self).call }
      else
        self
      end
    end

    def with_previous_context(binding, &block)
      @previous_context = binding.eval('self')
      result = block.call
      @previous_context = nil
      result
    end

    def method_missing(method, *args, &block)
      super
    rescue NameError => e
      if @previous_context
        @previous_context.send(method, *args, &block)
      else
        raise e
      end
    end
  end
end

Put that in config/initializers and try it out!

UPDATE: I’ve been asked for a license for this code, so I’m releasing it under the MIT license:

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Temporal Databases Annotated Bibliography

2017-12-05T00:00:00Z

I’ve been reading about temporal databases for a few years now, so I think it’s time I share my bibliography and notes. This is presented in “narrative order”, so that you can get a sense of how the research has developed. This article somewhat overlaps a mini literature review I wrote on the Postgres hackers mailing list, but this article is more complete and in a place where I can keep it updated.

Temporal databases let you track the history of things over time: both the history of changes to the database (e.g. for auditing) and the history of the thing itself. They are not the same thing as time-series databases: whereas a time-series database has time-stamped events, a temporal database stores the history of things, typically by adding a start/end time to each row (so two timestamps, not one). With time-series the challenge is typically scale; with temporal the challenge is with complexity and correctness.

Research

Abdullah Tansel, James Clifford, Shashi Gadia, Sushil Jajodia, Arie Segev, and Richard T. Snodgrass (editors). Temporal Databases: Theory, Design, and Implementation. 1993. I only became aware of this in 2023. I own a copy now but haven’t read it cover to cover. I’m curious to see the variety of ideas back then, before things got standardized. It’s amazing how many old ideas are just forgotten. Codd’s “Extending the data base relational model to capture more meaning” from 1979 is famous for introducing NULL and outer joins, but no one seems to remember the second half, with graph databases and queries that dynamically build themselves by querying the system catalog. So I bet this book has creative ideas that could inspire modern developments.

Already this book answered one question I had. Can a planner reorder/transform temporal operations the same way it can non-temporal ones? In other words, do they have the same algebraic properties? On page 175 in the chapter on TQuel I found this:

Theorem 12 The following equivalences hold for the valid-time algebra.

Q ∪̂ R ≡ R ∪̂ Q

Q ⨯̂ R ≡ R ⨯̂ Q

σ̂_F₁(σ̂_F₂(Q)) ≡ σ̂_F₂(σ̂_F₁(Q))

Q ∪̂ (R ∪̂ S) ≡ (Q ∪̂ R) ∪̂ S

Q ⨯̂ (R ⨯̂ S) ≡ (Q ⨯̂ R) ⨯̂ S

Q ⨯̂ (R ∪̂ S) ≡ (Q ⨯̂ R) ∪̂ (Q ⨯̂ S)

σ̂_F(Q ∪̂ R) ≡ σ̂_F(Q) ∪̂ σ̂_F(R)

σ̂_F(Q −̂ R) ≡ σ̂_F(Q) −̂ σ̂_F(R)

π̂_X(Q ∪̂ R) ≡ π̂_X(Q) ∪̂ π̂_X(R)

Theorem 13 The distributive property of Cartesian product over difference, or Q ⨯̂ (R −̂ S) ≡ (Q ⨯̂ R) −̂ (Q ⨯̂ S), does not hold for the valid-time algebra.

I discuss this in more depth in “TQuel Paper: Implementing Temporal Operators in Postgres”.

Snodgrass, Richard T. Developing Time-Oriented Database Applications in SQL. 1999. The seminal work on temporal databases and still the most useful introduction I know. Covers the “combinatorial explosion” of non-temporal/state-temporal/system-temporal/bi-temporal tables, current/sequenced/non-sequenced queries, SELECT/INSERT/UPDATE/DELETE, different RDBMS vendors, etc. Very similar to the proposed TSQL2 standard that was ultimately not accepted but still influenced Teradata’s temporal support. Available as a free PDF from his website.

Hugh Darwen and C. J. Date. “An Overview and Analysis of Proposals Based on the TSQL2 Approach.” Latest draft 2005, but originally written earlier. Criticizes the TSQL2 proposal’s use of “statement modifiers”, especially their problems with composability when a view/subquery/CTE/function returns a temporal result. Available as a PDF.

Ralph Kimball and Margy Ross. The Data Warehouse Toolkit. 3rd edition, 2013. (2nd edition 2002, 1st 1996.) (My notes are based on reading the 2nd edition, but I don’t think there are major relevant changes.) This book is not about temporal databases per se, but in Chapter 4 (and scattered around elsewhere) he talks about dealing with data that changes over time (“Slowly Changing Dimensions”). His first suggestion (Type 1) is to ignore the problem and overwrite old data with new. His Type 2 approach (make a new row) is better but loses the continuity between the old row and the new. Type 3 fixes that but supports only one change, not several. This writing is evidence for the need to handle temporal data, and the contortions that result from not having a systematic approach. (These criticisms and the realization that Kimball is trying to solve the same problems as temporal databases come from Johnston’s second book below; I’m glad he made the connection between Kimball and temporal databases!) (pdf)

C. J. Date, Hugh Darwen, Nikos Lorentzos. Time and Relational Theory, Second Edition: Temporal Databases in the Relational Model and SQL. 2nd edition, 2014. (First edition published in 2002.) I read this in March 2018. If you’ve read Date & Darwen’s other works you know they are quite rigorous, often critical of SQL’s compromises vs the pure relational model (e.g. NULL and non-distinct results), and not always very practical. Nonetheless this book is full of great ideas, and I hope anyone thinking about implementing temporal features in an RDBMS product will read it. They mostly deal with state-time, not transaction-time, which is great since that’s where the field most needs attention, although they also show how to apply their ideas to transaction-time tables too.

The core idea is that a temporal table should really have a row for each second (or millisecond or day or whatever is your finest granularity of time), saying what was true at that moment. Then the primary key is just the second plus the regular primary key. This idea isn’t new, but they use it to define everything else in the book. This makes thinking about temporal queries a lot easier, so that if you ever have any hesitation about something, you can think about it in terms of one-row-per-second and it starts to be easy.

Of course no database could adopt that in practice, so they offer operators to transform rows with start/end ranges to one-row-per-second and back again. In almost every case you don’t have to use those operators, but they are a solid theoretical basis for the next step: defining interval-aware primary keys, foreign keys, and all the relational operators. Finally they show how you implement things without “materializing” the expanded version of the data.

Another idea implicit in that is that temporal concerns belong in the SQL operators, not as “statement modifiers” like in TSQL2. Really that might be the most important idea in the whole book, so I’m glad to see it being applied to Postgres in the papers by Dignös et al (below).

They also argue (vs SQL:2011) that a single interval column type is better than separate start/end columns, which I agree with. Strangely they go against all the existing research by using closed/closed intervals (e.g. [Jan2017,Dec2017]) instead of closed/open ([Jan2017,Jan2018)), without really giving much justification. They also avoid using NULL for an unbounded side (e.g. [Jan2017,NULL)), preferring a magic “end of time” date ([Jan2017,Jan3000)). I wasn’t surprised by the second decision, given the authors’ history, but the first was less explicable. In both cases their approach sadly marginalizes their work and imposes barriers to adopting it in real SQL products.

I really appreciate how these authors have insisted that valid time (and the same with transaction time) should be a regular column, not something different. That was a big part of their complaint against TSQL2. To be honest when I first read Snodgrass the idea of using pseudo-columns seemed so suboptimal it was hard to take seriously (“Of course they don’t mean it.”), so I’m glad these authors have insisted on pointing out that approach’s shortcomings. Unfortunately in SQL:2011 valid-time is still a pseudo-column made up of two regular date/time columns. The problem is in composability: using a temporal query as a subquery, view, or function result: it all works cleanly if the interval is just another input to your interval-aware operators, but not if you need some kind of extra pseudo-column metadata. I hope implementers will take their advice seriously and not build temporal features on such a distorting idea.

Something I disagreed with was their suggestion to use tables in sixth-normal form—basically every column gets its own table—since attributes can have different lifespans. I can see how that is purer, but it seems like just too much complexity. They probably suspected the same because they always show how to do things with either that approach or tables in a more traditional third-normal form (or BCNF if you prefer). Even that is slightly distorted in order to avoid NULLs, but you can easily look past that.

Finally, I appreciated that on page 282 they mention DDL on temporal databases. Like everyone they say it’s beyond the scope of their current discussion, but it’s a penetrating insight to say, “the database catalog might itself need to be treated as a temporal database.”

SQL:2011 Draft standard. (pdf) Personally I find the standard pretty disappointing. It uses separate start/end columns instead of built-in range types, although range types offer benefits like exclusion constraints and convenient operators for things like “overlaps” that are verbose to code correctly by hand. It only mentions inner joins, not the various outer joins, semi-joins (EXISTS), anti-joins (NOT EXISTS), or aggregates. Many of its features apply only to system-time, not application-time, even though applicaion-time is the more interesting and less-available feature. (There are lots of auditing add-ons, but almost nothing for tracking the history of things.) The syntax seems too specific, lacking appropriate generality. A lot of these drawbacks seem motivated by a goal that goes back to TSQL2: to let people add temporal support to old tables without breaking any existing queries. That has always seemed to me like an unlikely possibility, and an unfortunate source of distortions. I don’t expect something for free, and I don’t mind doing work to migrate a table to a temporal format, as long as the result is good. Instead we get an (ostensible) one-time benefit for a prolonged compromise in functionality and ease-of-use.

Krishna Kulkarni and Jan-Eike Michels. “Temporal Features in SQL:2011”. SIGMOD Record, September 2012. Nice overview of the temporal features included in the SQL:2011 standard. Here is a PDF of the paper. See also these slides by Kulkani.

Peter Vanroose. “Temporal Data & Time Travel in PostgreSQL,” FOSDEM 2015. (Slides as a pdf) Lots of good detail here about SQL:2011. I’d love to see a recording of this talk if it’s available, but I haven’t found it yet.

Tom Johnston and Randall Weis. Managing Time in Relational Databases: How to Design, Update and Query Temporal Data. 2010. I finished this in May/June 2019. Although it concentrates on the authors’ proprietary middleware add-on to MS SQL Server called Asserted Versioning, it nonetheless has many ideas for building practical temporal features on any RDBMS. I liked their creativity and willingness to improve on SQL:2011 when they could. (Technically their book predates the final form of the standard.) Their middleware approach did sometimes lead them to behavior that would be strange in a more SQL-oriented solution. For example their temporal updates and deletes can only target a single record by ID (but they also mention they have a fix for this), and if that ID is not found you get an error. In ordinary SQL you would not get an error if an UPDATE or DELETE didn’t change any rows, and on the other hand you could hit more rows than one by not using the primary key. They mention that at the time of writing they were trying to replace their middleware approach with an implemention using views and instead-of triggers (something that has been done elsewhere too), and I expect that would wind up behaving more like ordinary SQL. They also introduce an interesting idea of future-dating records’ system-time (“assertion-time” in their lingo) so that the database will not make them “effective” until you’re ready. This has some nice practical use-cases, but there are pretty serious drawbacks. In their system a future-dated record is essentially immutable, so you had better not change your mind! Despite this book’s many contributions, I think everything here was also covered in Johnston’s second book, which is probably what I’d read if I had to choose one.

Tom Johnston. Bitemporal Data: Theory and Practice. 2014. I felt like I found a kindred soul when I read how he connects database design and ontology, as I’ve always thought of programming as “applied philosophy.” Databases as Aristotelian propositional logic is inseparable from the mathematical set-based theory. Johnston gives helpful distinctions between the physical rows in the table, the assertions they represent, and the things themselves. Eventually this leads to a grand vision of connecting every assertion’s bitemporal (or tritemporal) history to its speaker, somewhat like some ideas in the Semantic Web, although this doesn’t sound very practical. Like Date he seems to be landing on something like sixth-normal form, with a view-like presentation layer to bring all the attributes back together again. (Also like Date he seems to acknowledge 6NF may not be practical.) He points out how unsatisfactory Kimball’s suggestions are. He also criticizes the limitations of SQL:2011 and offers some amendments to make it more useful. Describes a (patented) idea of “episodes” to optimize certain temporal queries.

Anton Dignös, Michael H. Böhlen, and Johann Gamper. “Temporal Alignment”, SIGMOD ’12. Amazing! Shows how to define temporal versions of every relational operator by a combination of the traditional operators and just two simple transforms, which they call “align” and “split”. Gives a very readable exposition of the new theory and then describes how they patched Postgres 9.0 and benchmarked the performance. I think this solves the composability problems Date objected to in TSQL2, and unlike SQL:2011 it is general and comprehensive. The focus is on state-time, and I’m not sure how it will map onto bi-temporal, but even just having good state-time functionality would be tremendous. And the paper is only 12 easy-to-read pages! (pdf)

Anton Dignös, Michael Hanspeter Böhlen, Johann Gamper, and Christian S. Jensen. “Extending the kernal of a relational DBMS with comprehensive support for sequenced temporal queries,” ACM Transactions on Database Systems, 41(4):1-46. Continues the previous paper but adds support for scaling the inputs to aggregate groups according to how much of their time period goes into each group. Gives more benchmarks against a patched Postgres 9.5. (pdf) These researchers are now trying to contribute their work to the Postgres core project, of which I am very much in favor. :-)

Tools

Finally here are some tools for temporal support in Postgres. The sad theme is that pretty much everything gives audit support but not history support:

Postgres

The most complete implementation of bi-temporal tables is Vik Fearing’s periods extension. Since Postgres uses a bison grammar, an extension can’t add new syntax, but this one does a great job working around that limitation to provide almost everything from the SQL:2011 standard.

Henrietta (Hettie) Dombrovskaya has a pg_temporal repository with talks, slides, and code. I haven’t tried building it but I believe her company is using this in production. She is an awesome speaker, so her talks are great intros to this topic too.

The pgaudit extension looks pretty useful but I haven’t tried it yet. According to the AWS docs you can even use this on RDS.

Vlad Arkhipov’s temporal tables extension only supports system-time (auditing). Also on Github and a nice writeup by Clark Dave.

Magnus Hagander presented an approach to capturing system-time history in a separate schema at PGConf US 2015 and PGDay’15 Russia. Here are slides and video. Quite elegant if you want to ask questions like “what did we think as of time t?” If I recall correctly this is similar to one of the ideas proposed at the end of Snodgrass, although I haven’t compared them carefully. Hagander points out that DDL changes against temporal databases are challenging and hopefully infrequent. This is a topic that is almost completely absent from the literature, except for a brief mention in Johnston 2014.

I’ve heard about a Julia project for temporal data in Postgres. I’m not sure how it works, but it looks interesting!

Ruby

Chronomodel extends the ActiveRecord ORM to record system-time history. The pg_audit_log gem works fine but like many audit solutions is rather write-only. I wouldn’t want to build any functionality that has to query its tables to reconstruct history. You could also try paper_trail or audited (formerly acts_as_audited). Of these projects only Chronomodel seems to be aware of temporal database research.

Further Research

Temporal databases are exciting because there is still so much to do. For example:

What should the UI look like? Even one dimension adds a lot of complexity, let alone bi-temporal. How do you present this to users? As usual an audit history is easier, and it’s possible to find existing examples, whereas a state-time history is more challenging but probably more valuable. How should we let people view and edit the history of something? How does it work if there is a “Save” button vs save-as-you-type?
What does “full stack” temporal support look like? Do we extend REST? What would be a nice ORM interface? Should we use triggers to hide the temporal behavior behind regular-looking SQL? Or maybe extend SQL so you can more explicitly say what you want to do?
SELECT support for “as of” semantics or “over time” semantics.
Temporal foreign keys. I have a POC Postgres extension and am making progress adding it to core.
DDL changes. For example if you want to add a NOT NULL column, what do you do with the old data? Could there be built-in support to apply constraints only to a given time span?
Future time. There are some cool ideas about this in Johnson’s books, and I’ve heard of fintech folks doing something here.
Temporal upsert. This would be an INSERT that falls back to FOR PORTION OF for spans that already exist. Quite often this is what an application really wants. Johnson talks about how the standard should have included something here.
Outer joins and aggregates. Dignös et al show how to do this, but it’s not in the standard. I’d love to get their old patch working again and try to get something similar into Postgres, once the basics are finished. I’ve written SQL implementations of temporal outer join, semi-join, anti-join, UNION, EXCEPT, and INTERSECT, but those won’t perform as well as built-in RDBMS support. If Postgres gets inlining PL/pgSQL functions it will be faster. Even better would be a C implementation using CustomScan.

Scaling Sidekiq

2017-04-12T00:00:00Z

Sidekiq is a great option for handling background jobs in Ruby projects. Here I’ll show you how to get the best utilization out of a box dedicated to running Sidekiq jobs. Whether you have one machine or ten, the goal is to work off as many jobs as possible from each machine.

CPU Utilization

To do that, we want to keep every core busy. You can monitor your CPU activity with a tool like vmstat(8). If you say vmstat 10, you’ll get a new row every ten seconds, like so:

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 9  0  92056 553620 174128 6268208    0    0     1    25 1323 1443 23 77  0  0  0
 8  0  92056 552628 174128 6268492    0    0    14    38 1307 1447 22 78  0  0  0
 8  0  92056 551760 174132 6268672    0    0     3    49 1317 1461 23 77  0  0  0
 8  0  92056 550544 174140 6268832    0    0     1    28 1353 1490 23 77  0  0  0
 8  0  92056 550296 174148 6268940    0    0     1    35 1240 1360 22 78  0  0  0

The last few columns show percentage of CPU time doing user work (us), system work (sy)—which more or less means executing kernel system calls—, idle (id), and blocked on I/O (wa, “waiting”). Add the first two together to get how much your CPU is working.

There is also st, which means “stolen time”! If you are running a VM, this means the time your hypervisor gives to someone else. You will probably see all zeros in this column, so I’ll ignore it going forward. We want to get our idle and wait time as close to zero as possible, without drowning the machine in too much work. Above we have a machine that is doing a good job keeping busy. You can see that its user and system time are at 100, and the other columns are zero.

So how do we do this? Usually it means running lots of jobs at the same time, not one-after-another. Even a one-core machine can juggle many jobs. That’s because jobs typically have to block on I/O, for example when they talk to the network or save things to disk. While that job is waiting for an answer, your CPU can work on another one.

(This is not really relevant here, but disk I/O is a little different than other I/O. Technically, reading/writing with a regular file can’t “block” but only “sleep”. That completely messes up non-blocking I/O for regular files, and even the newer aio functions have many limitations and in fact are implemented by threads in userspace—but fortunately we are not talking about single-threaded non-blocking I/O; we are talking about multiple processes/threads. Whether you call it blocking or sleeping, the CPU will still schedule different work if something is stuck on a regular file read. And if this paragraph doesn’t make sense, feel free to dismiss it as a pedantic footnote. :-)

Threads

Sidekiq is great here because it supports multi-threading. Multiple threads let you do concurrent work in one Ruby process (at least as long as you are not still stuck on Ruby 1.8). Without threads you’d need a separate process for each concurrent job, and that can use up memory quickly, especially with something like Rails. It is always sad to have more CPU available that you can’t use because you’re out of RAM.

In practice, threads mostly help if you are using a concurrent Ruby implementation like JRuby. MRI Ruby has a Global Interpreter Lock (GIL), which prevents two threads from executing at once. Still, even in MRI you will still see some benefit, because when one thread blocks on I/O, MRI can make progress on another. So despite the GIL, MRI can still make sure at least some thread is running.

With Sidekiq, you can say how many threads to run with the concurrency setting. Normally you’d set this in your sidekiq.yml file. Note that each thread needs its own database connection! That means if your concurrency is 10, then in your database.yml you must have a pool of 10 also (or more). Otherwise the threads will halt each other waiting to check out a database connection, and what is the point of that? They might even get timeout errors.

Processes

But wait, there’s more! We can push the concurrency up and up, and still see something like this from vmstat:

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0  92048 585840 174148 6269120    0    0     0    16  956  990  7 18 75  0  0
 1  0  92048 585212 174160 6269256    0    0     1    23 1012 1037  8 17 75  0  0
 1  0  92048 580456 174164 6269312    0    0     1    33  896  884  8 17 75  0  0

Why is the idle CPU time stuck at 75?

It turns out this is a four-core machine, and even with multiple threads, a single MRI process can only use one core, because of the GIL. That’s not what we want at all!

So the answer is to run more processes—at least one per core. You can see how many cores you’ve got by saying cat /proc/cpuinfo. If you are using something like god, it is easy to put several processes into a group and control them together, like so:

app_root = '/var/www/myapp/current'
4.times do |i|
  God.watch do |w|
    w.name     = "myapp-sidekiq-#{i}"
    w.group    = "myapp-sidekiq"
    w.log      = File.join(app_root, 'log', "#{w.name}.log")
    w.pid_file = File.join(app_root, 'current', 'tmp', 'pids', "#{w.name}.pid")
    w.start    = <<-EOS.gsub("\n", " ")
      cd '#{app_root}' &&
        bundle exec ./bin/sidekiq --environment production
                                  --pidfile '#{w.pid_file}'
                                  --logfile '#{w.log}'
                                  --daemon
    EOS
    # ...
  end
end

If you aren’t using god and don’t know how to do this with your own process manager, then I think Sidekiq Enterprise has a similar feature called Swarms.

Scheduling cores

That gets us almost there, but there is still a problem. You might still see idle time stuck somewhere, like (on a 4-core system) 25. The problem is that while the kernel does its best to use your whole CPU, it can still wind up putting two Sidekiq processes on the same core, and since they are such long-lived processes, they are stuck that way, competing for their shared core while another one sits idle. The kernel doesn’t know ahead of time that they are going to run for days and keep so busy.

Fortunately we can still force each process onto its own core. For that we use the taskset(1) command. When you say taskset -c 2 date, you are telling Linux to run date on core 2. (Core numbers start from zero, as you can see in /proc/cpuinfo.) So our god config would become:

w.start    = <<-EOS.gsub("\n", " ")
  cd '#{app_root}' &&
    bundle exec taskset -c #{i} ./bin/sidekiq --environment production
                                              --pidfile '#{w.pid_file}'
                                              --logfile '#{w.log}'
                                              --daemon
EOS

After that, we’ll have one process on each core. At this point, you should start experimenting with your concurrency setting, to make each core fully utilized. The right setting there will depend on how much blocking a job does, but I have seen useful numbers up to 20. Just try some things out, and watch vmstat.

Note that you don’t have to increase the connection pool size in database.yml as you add cores (just concurrency). That’s because each process is a separate thing, each with its own pool. But you do have to increase the max connections on your database server. For instance with Postgres you’d want to set max_connections in postgresql.conf. Here you need to allow enough connections for concurrency times cores (times servers), plus some more for your actual Rails app serving web requests, plus some more for anything else you’ve got going on. That can be a lot of connections! Don’t be surprised if improving your job throughput exposes a bottleneck elsewhere in your system.

Conclusion

Tuning Sidekiq can be complicated, because of the double layers of threads plus processes. You need a confident understanding of how Ruby and your operating system handle concurrency, and it helps to use tools like vmstat to measure what’s going on and verify your understanding. If you have a box dedicated to just Sidekiq jobs, my recommendation is to run one process per core, using taskset to keep them separate, and then tune concurrency from there. Hopefully this will help with your own projects!

Paperclip expiring_url and "Request has expired"

2015-04-28T00:00:00Z

I hit a weird error today. My project stores user uploads on S3 with private permissions, and we use the Paperclip expiring_url method to let people download them. This started failing with a error like this:

<Error>
  <Code>AccessDenied</Code>
  <Message>Request has expired</Message>
  <Expires>2015-04-28T18:30:51Z</Expires>
  <ServerTime>2015-04-28T18:32:52Z</ServerTime>
  <RequestId>BLAHBLAHBLAHBLAH</RequestId>
  <HostId>
    MOREBLAHMOREBLAHMOREBLAHMOREBLAHMOREBLAHMOREBLAHMOREBLAH
  </HostId>
</Error>

The problem was that this error occurred immediately, even though we were generating the URLs with a 10 minute expiration time. Well, it turns out S3 was living in the future by 7 minutes (according to ServerTime above), and our own app was living in the past by another 5. So 5 + 7 = 12 and our URLs were expired before we even generated them. 10 minutes is the example time in the Paperclip documentation linked above, but I guess it’s too short. Watch out for clock skew out there folks!

Rails acts_as_list with Soft Delete

2014-12-23T00:00:00Z

Two really useful gems for a Rails project are acts_as_list and permanent_records (or the somewhat more common acts_as_paranoid). The first one, acts_as_list, adds a position column to your model to provide natural ordering. Usually you don’t want global ordering, but only within an owning model, e.g. the Page objects inside a Book have position 1, 2, 3, and the Page objects inside a different Book also go 1, 2, 3. You can use the scope option to achieve that:

class Page
  belongs_to :book
  acts_as_list scope: :book
end

The problem is that if you are soft-deleting pages using a deleted_at column, acts_as_list doesn’t know to exclude deleted pages from the list. This gets annoying if you want to move pages up and down: Imagine you have page 1, then 5 deleted pages, and now you add a new page p to the end. You want to move your new page to the beginning. You have a button that calls p.move_higher, but the page won’t appear to move until the user presses the button five times. That is pretty confusing!

Fortunately acts_as_list lets you pass a string to scope that it will evaluate when necessary, and you can even embed Ruby code in it. Look at this, and notice the single quotes!:

class Page
  belongs_to :book
  acts_as_list scope: 'book_id = #{book_id} AND deleted_at IS NULL'
end

With that code, acts_as_list will only consider non-deleted pages, so moving a page works as it should.

Flash movie (.swf) won't load in Rails 4

2014-07-24T00:00:00Z

I work on a Rails project that serves Flash .swf files out of Rails controllers, like this:

send_file swf_filename, disposition: 'inline'

I know that’s a crazy approach, but the reason is that these movies should only be seen by logged-in users. (Also I didn’t set it up like this. :-)

Recently some customers complained that Flash movies wouldn’t load. I was pulling my hair out figuring out why. I couldn’t reproduce it. The Rails and nginx logs showed no errors. Their browsers were retrieving the file, but then they would just display a blank white page, sometimes with a thin border where the movie should be.

Finally I was able to reproduce it on a different machine, and from there I eventually figured out it only happened for people running on the latest version of Flash. I installed the Flash debugger, I turned on tracing, but that didn’t detect any errors.

Another clue was that the problem wasn’t happening on another site that shares this codebase. By comparing browser requests between the sites, I was able to figure out the cause was an X-Content-Type-Options HTTP header with a nosniff value.

It turns out that Rails 4 adds this header by default, and although the working site was still on Rails 3, my predecessor had upgraded the problem site to 4. So one fix is to turn off that header like this:

response.headers.except! 'X-Content-Type-Options'

But a better fix would be to add a header that indicates the correct Content Type. We were serving the file as application/octet-stream, so with nosniff the browser couldn’t figure out how to handle the file. We needed to serve the .swf file as application/x-shockwave-flash instead:

response.headers['Content-Type'] = 'application/x-shockwave-flash'

Hopefully this will help someone else out there!

Dates and Time Zones in Rails

2014-04-18T00:00:00Z

I work on a video education app that needs to report how many lessons were assigned each day. I have a timetracking app that needs to invoice based on when one month starts and another ends. And I’ve got another app for watching cronjobs, that counts how many failures happened in a given day. All these apps share a problem: if they get time zones wrong, they will give incorrect results. For instance, if a video lesson was assigned on Tuesday after 5pm PDT, its UTC time will be Wednesday. Even though all three apps only show dates, not times, time zones still matter. So here are some lessons I’ve learned dealing with Rails time zones, both programming myself and leading a team.

But first, here are two articles that lay some groundwork:

Both those posts describe some nice methods provided by ActiveSupport/Rails, as well as why you’d want to use them instead of built-in Ruby methods. For instance, saying Time.now will get you the time in your OS’s default time zone, whereas Time.zone.now will get you the time in whatever timezone you put in config/application.rb. It’s a good idea to use the Rails timezone, since you have more control over that and it’s part of your source control.

If you want to explore, take a look at ActiveSupport::TimeZone. You can get a specific time zone like this:

tz = ActiveSupport::TimeZone['Pacific Time (US & Canada)']

That tz will be the start of just about everything you do with time. An ActiveSupport::TimeZone is also what you get from Time.zone.

Those other articles are great, but this post talks about time zones from a higher level, aspiring to share best practices rather than helpful methods. In my examples I’ll assume a Postgres database, although there’s nothing here that won’t work on MySQL as well.

Scope Your Time Zones

The first thing to remember is that all times are relative to somebody. Probably that’s a user, but maybe it’s something else. For my video education app, sometimes it’s a teacher and sometimes it’s a student. For my cronjob app, it’s a job (and sometimes a user). But there is always a reference point. In that sense, the articles above that recommend Time.zone.now are wrong. Time.zone.now is no better than Time.now if you need many time zones—and who doesn’t? So it should really be user.time_zone.now. I’d recommend adding this method to your User class right from the beginning, so that even if you haven’t yet implemented user-specific time zones, you are still writing the rest of your app correctly:

def time_zone
  ActiveSupport::TimeZone['Pacific Time (US & Canada)']
end

Ignore Time Zones as Much as Possible

Now that you’ve got time zones everwhere, the next step is to get rid of them. Thinking about time zones is hard, and finding time zone bugs is hard. You want to avoid them as much as possible!

It’s been said that in C, if you start thinking about big endian vs little endian, you’re probably doing things wrong. Oftentimes when programmers half-understand something, they do more than they should, and that’s very true with time zones. Your code will be a lot easier to understand if you tackle time zones with a few well-placed strokes, rather than lots of fiddling all over the place.

One thing to remember is that 07:05:00 PDT and 14:05:00 UTC are the same instant. If you converted both to seconds since the epoch, you’d get the same number. So in that sense, changing the time zone doesn’t change anything: it’s just a bit of extra metadata hanging onto the time representing someone’s perspective. Knowing that x PDT and y UTC are the same instant is really helpful when you feel the urge to fiddle with timezones. Is your fiddling just a noop?

By default, Rails comes configured with its default time zone as UTC. Leave it that way! Since every time needs a reference point anyway, your code shouldn’t care about the Rails-wide setting. UTC is a good neutral choice. For one thing, it doesn’t have daylight savings time. And if you see it, you know you’re dealing with a time-zone-less value.

You should also leave your OS time zone as UTC, if possible. Keeping it consistent with Rails will remove one chance for abiguiuty. And again, it’s a good neutral.

You also want UTC in your database. If you use a migration to create columns with t.timestamps or t.datetime :foo, Rails will make a TIMESTAMP WITHOUT TIME ZONE column. You can think of this as a time in UTC if you like. Really it’s an int of (micro)seconds since the epoch. Whenever you give a time to ActiveRecord, it will convert it to UTC before it hits Postgres. Or more correctly, it will strip off the timezone part and give Postgres the int. Remember, it’s the same instant! But it’s nice to imagine the column as UTC. If you’re in psql and type SELECT created_at FROM lessons, that’s what you’re seeing.

When everything in your stack is UTC, it’s like looking through nice clear glass. You don’t have to think about conversions at each layer.

You should also strive to make your app code as time-zone-less as possible. The big principle here is to handle time zones once, hopefully at the beginning of the HTTP request, when you decide the correct reference point for all times. Usually that’s current_user.time_zone.

The second article above suggests you add this to your controller:

around_filter :user_time_zone, if: :current_user

def user_time_zone(&block)
  Time.use_zone(current_user.time_zone, &block)
end

That makes me pretty uncomfortable. The idea is that you can say Time.zone everywhere else in your app and always get the current user’s time zone. But I’d rather be explicit about where the time zone is coming from. (Also see below for why use_time doesn’t help at all for parsing times.) Instead, write code to take a tz argument if necessary (emphasis on the if necessary). This will make your code less surprising, less coupled, and easier to test.

Even better is to write your code to take just a time. Almost always that’s really what you want. Remember that no matter the time zone, it’s all the same instant. Usually a single time is sufficient, because it can be a reference point for creating other times, using t + 1.day or t + 2.weeks or whatever. Use a (user|cronjob|foo)-scoped time zone to get your first time, and then forget about time zones for the rest of the stack. If you implemented the User#time_zone method above, your time zone code is already well-encapsulated, so there’s no need for an around_filter to further abbreviate things.

Here is another approach I don’t like. This blog post suggests you deal with time zones in Postgres like so:

SELECT created_at AT TIME ZONE 'UTC' AT TIME ZONE 'US/Pacific'

Does that look strange to you? What’s happening is that you start with a TIMESTAMP WITHOUT TIME ZONE, so first you tack on a time zone (just metadata), then you convert it to Pacific time. In other words, you’re doing this:

time without zone -> assumed to be UTC -> converted to Pacific Time

It’s good to know that this is how to get a Postgres timestamp converted to whatever time zone you want, but I wouldn’t recommend it in a Rails app (which is the article’s context). For one thing, Rails and Postgres don’t use the same names for all timezones, so you have to maintain your own mapping between the two. This is one of those chores that will keep nagging you for the life of your app, so I’d rather just avoid it. But more important, this approach means you have to pass your timezone all the way down to the database layer. I’d rather deal with time zones early, get to a TimeWithZone (or even an int), and then forget about time zones for the rest of the code.

Remember Daylight Savings Time

This is a small tip, but be careful about DST. You should avoid ever writing a fixed offset or a string like “PDT”. This is wrong:

Time.new(2014, 4, 8, 3, 15, 0, '-07:00')

If you were after PDT, your code is broken during PST. If you were after MST, your code is broken during MDT. Similarly in SQL if you say AT TIME ZONE 'PDT', you’ve broken PST.

If you stick with the ActiveSupport::TimeZone instances, you can forget about DST, and things will just work. That’s why there is no such thing as this:

ActiveSupport::TimeZone['PDT']

only this:

ActiveSupport::TimeZone['Pacific Time (US & Canada)']

Even Dates are Times

When you think you’re just dealing with dates, time zones probably still matter. Was the lesson assigned on Tuesday or Wednesday? It depends. From the teacher’s perspective? The student’s? The principal’s? Unless you’re really sure, I’d recommend always storing a full date+time in your database. Also, in Ruby avoid converting things to Date. Of course if you read the articles above you know you don’t want this:

Date.today

But this isn’t any better:

Time.zone.today

Or even this:

current_user.time_zone.today

Here is a query I’ve seen:

Lesson.where("date(created_at) >= ?", tz.today - 3.days)

But that’s wrong, because you’re stripping off all the time zone information. It’s going to report Tuesday’s lessons in Wednesday. (It also means you need a database index on the expression date(created_at), which is probably less often useful than a normal index on just the column.)

To improve this query, I’d write it like this:

Lesson.where("created_at >= ?", tz.now.midnight - 3.days)

If today is Wednesday (for you), that will give all the lessons created since the beginning of Sunday. Because we’re carrying a full time all the way down to the database query, we don’t have problems with the definition of “today.”

If you are ever tempted to use today or to_date, you probably want these methods instead:

tz.now.midnight     # the start of today, 00:00:00
tz.now.end_of_day   # 23:59:59

Parsing times

Now that you’ve got a per-user time zone, you want to use that to parse date/time inputs from that user. If you user types “3:15”, you want to interpret that as 3:15 in the user’s time zone.

There is a parse method on ActiveSupport::TimeZone, but unfortunately no strptime. If you say this, you get the wrong result:

Time.strptime("2014-04-08 03:15", "%Y-%m-%d %H:%M").in_time_zone(tz)

That’s because strptime assumes the time is in the default time zone (hopefully UTC), and in_time_zone does not change the “instant” the time represents, only the perspective used to view it. If you’re lucky to have a standardish format, this will work:

tz.parse("2014-04-08 03:15")

If your format is something weirder, or you just don’t trust parse with its heuristic-based approach, then you’re probably out of luck. A previous version of this article recommended Time.use_zone(tz) { Time.strptime(...) }, but that doesn’t work, because use_time only changes Time.zone. It doesn’t change Time.strptime.

Another approach with strptime that doesn’t work is to concatenate the time zone to your string and use %Z (and friends) in your format, like this:

Time.strptime('2014-04-08 03:15' + ' ' + tz.formatted_offset, "%Y-%m-%d %H:%M %:z")

The problem here is that tz.formatted_offset really depends on the time of year because of Daylight Savings Time. But you don’t know that until you parse the string. You could pass the timezone abbreviation instead, like PDT, but that has the same problem. And %Z doesn’t understand long names like Pacific Time (US & Canada).

The only approach I know that works is to parse the time with your default time zone, then feed the bits into tz.local, like so:

t = Time.strptime("2014-04-08 03:15", "%Y-%m-%d %H:%M")
tz.local(t.year, t.month, t.mday, t.hour, t.min)

Sorry, that’s the best I can do!

UPDATE: I added a strptime method to the TimeZone class (my first Rails code contribution), so hopefully you’ll start seeing it in future versions!

Displaying times

So much for times as inputs. Times as outputs is a lot easier. Your rule should be to ignore time zones until the last minute, when you actually format the value for rendering. So it should go right into your view:

= lesson.created_at.in_time_zone(current_user.time_zone).strftime("%A, %B %-d, %Y")

You might want a helper for this though, something like:

def format_time(time, tz, format)
  time.in_time_zone(tz).strftime(format)
end

Then your Haml can be:

= format_time(lesson.created_at, current_user.time_zone, "%A, %B %-d, %Y")

Or de-parameterize that as much as you like:

def format_time(time)
  time.in_time_zone(current_user.time_zone).strftime("%A, %B %-d, %Y")
end

and:

= format_time(lesson.created_at)

Conclusion

So in general, my principles for handling times in Rails are:

Even dates are times.
Set “global” time zones to UTC everywhere you can.
Don’t ever use global time zones; scope it to a user or whatever is appropriate.
To most of your code, a time is just an instant.
Push time zone inputs to the very beginning of your request, to the top of your stack.
Push time zone outputs to the very end of your request, when rendering the view.

Good luck!