<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://bridgetownrb.com/" version="2.2.2">Bridgetown</generator><link href="https://peterboling.com/feed/resque_unique_at_runtime.xml" rel="self" type="application/atom+xml" /><link href="https://peterboling.com/" rel="alternate" type="text/html" /><updated>2026-11-15T23:03:36+00:00</updated><id>https://peterboling.com/feed/resque_unique_at_runtime.xml</id><title type="html">Peter Boling | Resque_unique_at_runtime</title><subtitle>Yet Another Paradigm by Peter Boling, featuring technical blog posts about Ruby, JavaScript, TypeScript, Go, Python, DevOps, and various open source projects. Ramblings, mumblings, cv/resume, technical blog, and documentation for many popular, and unpopular, RubyGems and other FLOSS projects, including oauth2, omniauth-identity, sanitize_email, debug_logging, flag_shih_tzu, gem_bench, etc. Semi-lifely state-of-the-petris updates.</subtitle><author><name>Peter Boling</name><email>floss@galtzo.com</email></author><entry><title type="html">Introducing resque-unique_at_runtime</title><link href="https://peterboling.com/resque_unique_at_runtime/introducing-resque-unique_at_runtime/" rel="alternate" type="text/html" title="Introducing resque-unique_at_runtime" /><published>2017-10-01T19:30:16+00:00</published><updated>2017-10-01T19:30:16+00:00</updated><id>https://peterboling.com/resque_unique_at_runtime/introducing-resque-unique_at_runtime/</id><content type="html" xml:base="https://peterboling.com/resque_unique_at_runtime/introducing-resque-unique_at_runtime/">&lt;h2 id=&quot;ensures-that-only-one-job-for-a-given-queue-will-be-running-concurrently&quot;&gt;Ensures that only one job for a given queue will be running concurrently.&lt;/h2&gt;

&lt;p&gt;&lt;a href=&quot;https://github.com/pboling/resque-unique_at_runtime&quot;&gt;&lt;img src=&quot;https://img.shields.io/gem/rt/resque-unique_at_runtime.svg&quot; alt=&quot;Total Downloads&quot; /&gt;&lt;/a&gt;
&lt;a href=&quot;https://github.com/pboling/resque-unique_at_runtime&quot;&gt;&lt;img src=&quot;https://img.shields.io/gem/rd/resque-unique_at_runtime.svg&quot; alt=&quot;Downloads Today&quot; /&gt;&lt;/a&gt;
&lt;a href=&quot;https://codeclimate.com/github/pboling/resque-unique_at_runtime/test_coverage&quot;&gt;&lt;img src=&quot;https://api.codeclimate.com/v1/badges/bd4c69c621a3d7890405/test_coverage&quot; alt=&quot;Test Coverage&quot; /&gt;&lt;/a&gt;
&lt;a href=&quot;https://codeclimate.com/github/pboling/resque-unique_at_runtime/maintainability&quot;&gt;&lt;img src=&quot;https://api.codeclimate.com/v1/badges/bd4c69c621a3d7890405/maintainability&quot; alt=&quot;Maintainability&quot; /&gt;&lt;/a&gt;
&lt;a href=&quot;https://github.com/pboling/resque-unique_at_runtime/network&quot;&gt;&lt;img src=&quot;https://img.shields.io/github/forks/pboling/resque-unique_at_runtime.svg?style=social&quot; alt=&quot;Network&quot; /&gt;&lt;/a&gt;
&lt;a href=&quot;https://github.com/pboling/resque-unique_at_runtime/stargazers&quot;&gt;&lt;img src=&quot;https://img.shields.io/github/stars/pboling/resque-unique_at_runtime.svg?style=social&quot; alt=&quot;Stars&quot; /&gt;&lt;/a&gt;
&lt;a href=&quot;https://rubygems.org/gems/resque-unique_at_runtime&quot;&gt;&lt;img src=&quot;https://img.shields.io/gem/v/resque-unique_at_runtime.svg&quot; alt=&quot;Version&quot; /&gt;&lt;/a&gt;
&lt;a href=&quot;https://travis-ci.org/pboling/resque-unique_at_runtime&quot;&gt;&lt;img src=&quot;https://travis-ci.org/pboling/resque-unique_at_runtime.svg?branch=master&quot; alt=&quot;Build Status&quot; /&gt;&lt;/a&gt;
&lt;a href=&quot;http://inch-ci.org/github/pboling/resque-unique_at_runtime&quot;&gt;&lt;img src=&quot;http://inch-ci.org/github/pboling/resque-unique_at_runtime.svg&quot; alt=&quot;Documentation&quot; /&gt;&lt;/a&gt;
&lt;a href=&quot;https://depfu.com/github/pboling/resque-unique_at_runtime?project_id=4349&quot;&gt;&lt;img src=&quot;https://badges.depfu.com/badges/99a79f37d1bc55353d0480f5138a0ce0/count.svg&quot; alt=&quot;Depfu&quot; /&gt;&lt;/a&gt;
&lt;a href=&quot;https://gitter.im/pboling/resque-unique_at_runtime&quot;&gt;&lt;img src=&quot;https://img.shields.io/gitter/room/pboling/resque-unique_at_runtime.svg&quot; alt=&quot;Chat&quot; /&gt;&lt;/a&gt;
&lt;a href=&quot;https://opensource.org/licenses/MIT&quot;&gt;&lt;img src=&quot;https://img.shields.io/badge/License-MIT-green.svg&quot; alt=&quot;License: MIT&quot; /&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;A &lt;a href=&quot;http://semver.org/&quot;&gt;semanticaly versioned&lt;/a&gt;
&lt;a href=&quot;https://github.com/resque/resque&quot;&gt;Resque&lt;/a&gt; plugin which ensures for a given
queue, that only one worker is working on a job at any given time.&lt;/p&gt;

&lt;p&gt;Resque&lt;mark&gt;Plugins&lt;/mark&gt;UniqueAtRuntime differs from &lt;a href=&quot;https://github.com/wallace/resque-lonely_job&quot;&gt;resque-lonely_job&lt;/a&gt; in that it is compatible with, and can be used at the same time as, &lt;a href=&quot;https://github.com/neighborland/resque_solo&quot;&gt;resque-solo&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Resque&lt;mark&gt;Plugins&lt;/mark&gt;UniqueAtRuntime differs from &lt;a href=&quot;https://github.com/neighborland/resque_solo&quot;&gt;resque_solo&lt;/a&gt; in that &lt;code class=&quot;highlighter-rouge&quot;&gt;resque-solo&lt;/code&gt; offers &lt;strong&gt;queue-time&lt;/strong&gt; uniqueness, while &lt;code class=&quot;highlighter-rouge&quot;&gt;resque-unique_at_runtime&lt;/code&gt; offers &lt;strong&gt;runtime&lt;/strong&gt; uniqueness.  The same difference applies to other queue-time uniqueness gems: &lt;a href=&quot;https://github.com/mashion/resque-queue-lock&quot;&gt;resque-queue-lock&lt;/a&gt;, &lt;a href=&quot;https://github.com/defunkt/resque-lock&quot;&gt;resque-lock&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Runtime uniqueness without queue-time uniqueness means the same job may be queued multiple times but you’re guaranteed that first job queued will run to completion before subsequent jobs are run.&lt;/p&gt;

&lt;p&gt;However, you can use both runtime and queue-time uniqueness together in the same project.&lt;/p&gt;

&lt;p&gt;To use &lt;code class=&quot;highlighter-rouge&quot;&gt;resque-solo&lt;/code&gt; and &lt;code class=&quot;highlighter-rouge&quot;&gt;resque-unique_at_runtime&lt;/code&gt; together, with fine control of per job configuration of uniqueness at runtime and queue-time, it is recommended to use &lt;a href=&quot;https://github.com/pboling/resque-unique_by_arity&quot;&gt;resque-unique_by_arity&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;NOTE: There is a &lt;em&gt;strong&lt;/em&gt; possibility that subsequent jobs are re-ordered due to
the implementation of
&lt;a href=&quot;https://github.com/pboling/resque-unique_at_runtime/blob/master/lib/resque-unique_at_runtime.rb#L35&quot;&gt;reenqueue&lt;/a&gt;.
(See Example #2 for an alternative approach that attempts to preserve job
ordering but introduces the possibility of starvation.)&lt;/p&gt;

&lt;p&gt;Therefore it is recommended that the payload for jobs be stored in a separate
redis list distinct from the Resque queue (see Example #3).&lt;/p&gt;

&lt;h2 id=&quot;installation&quot;&gt;Installation&lt;/h2&gt;

&lt;p&gt;Add this line to your application’s Gemfile:&lt;/p&gt;

&lt;div class=&quot;highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;gem &apos;resque-unique_at_runtime&apos;, &apos;~&amp;gt; 1.0.0&apos;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;And then execute:&lt;/p&gt;

&lt;div class=&quot;highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;$ bundle
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Or install it yourself as:&lt;/p&gt;

&lt;div class=&quot;highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;$ gem install resque-unique_at_runtime
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;h2 id=&quot;usage&quot;&gt;Usage&lt;/h2&gt;

&lt;h4 id=&quot;example-1--one-job-running-per-queue&quot;&gt;Example #1 – One job running per queue&lt;/h4&gt;

&lt;div class=&quot;highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;require &apos;resque-unique_at_runtime&apos;

class StrictlySerialJob
  extend Resque::Plugins::UniqueAtRuntime

  @queue = :serial_work

  def self.perform
    # only one at a time in this block, no parallelism allowed for this
    # particular queue
  end
end
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;h4 id=&quot;example-2--one-job-running-per-user-defined-attribute&quot;&gt;Example #2 – One job running per user-defined attribute&lt;/h4&gt;

&lt;p&gt;Let’s say you want the serial constraint to apply at a more granular
level.  Instead of applying at the queue level, you can overwrite the .redis_key
method.&lt;/p&gt;

&lt;div class=&quot;highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;require &apos;resque-unique_at_runtime&apos;

class StrictlySerialJob
  extend Resque::Plugins::UniqueAtRuntime

  @queue = :serial_work

  # Returns a string that will be used as the redis key
  # NOTE: it is recommended to prefix your string with the &apos;unique_at_runtime:&apos; to
  # namespace your key!
  def self.unique_at_runtime_redis_key(account_id, *args)
    &quot;unique_at_runtime:strictly_serial_job:#{account_id}&quot;
  end

  # Overwrite reenqueue to lpush instead of default rpush.  This attempts to
  # preserve job ordering but job order is *NOT* guaranteed and also not
  # likely. See the comment on SHA: e9912fb2 for why.
  def self.reenqueue(*args)
    Resque.redis.lpush(&quot;queue:#{Resque.queue_from_class(self)}&quot;, Resque.encode(class: self, args: args))
  end

  def self.perform(account_id, *args)
    # only one at a time in this block, no parallelism allowed for this
    # particular unique_at_runtime_redis_key
  end
end
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;&lt;em&gt;NOTE&lt;/em&gt;: Without careful consideration of your problem domain, worker starvation
and/or unfairness is possible for jobs in this example.  Imagine a scenario
where you have three jobs in the queue with two resque workers:&lt;/p&gt;

&lt;div class=&quot;highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;+---------------------------------------------------+
| :serial_work                                      |
|---------------------------------------------------|
|             |             |             |         |
| unique_at_runtime_redis_key:  | unique_at_runtime_redis_key:  | unique_at_runtime_redis_key:  | ...     |
|    A        |    A        |    B        |         |
|             |             |             |         |
| job 1       | job 2       | job 3       |         |
+---------------------------------------------------+
                                  ^
                                  |
  Possible starvation +-----------+
  for this job and
  subsequent ones
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;When the first worker grabs job 1, it’ll acquire the mutex for processing
  redis_key A.  The second worker tries to grab the next job off the queue but
  is unable to acquire the mutex for redis_key A so it places job 2 back at the
  head of the :serial_work queue.  Until worker 1 completes job 1 and releases
  the mutex for redis_key A, no work will be done in this queue.&lt;/p&gt;

&lt;p&gt;This issue may be avoided by employing dynamic queues,
  http://blog.kabisa.nl/2010/03/16/dynamic-queue-assignment-for-resque-jobs/,
  where the queue is a one to one mapping to the redis_key.&lt;/p&gt;

&lt;h4 id=&quot;example-3--one-job-running-per-user-defined-attribute-with-job-ordering-preserved&quot;&gt;Example #3 – One job running per user-defined attribute with job ordering preserved&lt;/h4&gt;

&lt;p&gt;The secret to preserving job order semantics is to remove critical data from the
resque job and store data in a separate redis list. Part of a running job’s
responsibility will be to grab data off of the separate redis list needed for it
to complete its job.&lt;/p&gt;

&lt;div class=&quot;highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;+---------------------------------------------------+
| :serial_work for jobs associated with key A       |
|---------------------------------------------------|
|   data x    |   data y    |   data z    | ...     |
+---------------------------------------------------+

+---------------------------------------------------+
| :serial_work for jobs associated with key B       |
|---------------------------------------------------|
|   data m    |   data n    |   data o    | ...     |
+---------------------------------------------------+

+---------------------------------------------------+
| :serial_work                                      |
|---------------------------------------------------|
|             |             |             |         |
| unique_at_runtime_redis_key:  | unique_at_runtime_redis_key:  | unique_at_runtime_redis_key:  | ...     |
|    A        |    A        |    B        |         |
|             |             |             |         |
| job 1       | job 2       | job 3       |         |
+---------------------------------------------------+
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;It now doesn’t matter whether job 1 and job 2 are re-ordered as whichever goes
first will perform an atomic pop on the redis list that contains the data needed
for its job (data x, data y, data z).&lt;/p&gt;

&lt;h4 id=&quot;example-4--requeue-interval&quot;&gt;Example #4 – Requeue interval&lt;/h4&gt;

&lt;p&gt;The behavior when multiple jobs exist in a queue protected by resque-unique_at_runtime
is for one job to be worked, while the other is continuously dequeued and
requeued until the first job is finished.  This can result in that worker
process pegging a CPU/core on a worker server.  To guard against this, the
default behavior is to sleep for 1 second before the requeue, which will allow
the cpu to perform other work.&lt;/p&gt;

&lt;p&gt;This can be customized using a &lt;code class=&quot;highlighter-rouge&quot;&gt;@runtime_requeue_interval&lt;/code&gt; class instance variable
in your job like so:&lt;/p&gt;

&lt;div class=&quot;highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;require &apos;resque-unique_at_runtime&apos;

class StrictlySerialJob
  extend Resque::Plugins::UniqueAtRuntime

  @queue = :serial_work
  @runtime_requeue_interval = 5         # sleep for 5 seconds before requeueing

  def self.perform
    # some implementation
  end
end
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;</content><category term="development" /><category term="ruby" /><category term="rails" /><category term="rubygems" /><category term="resque" /><category term="resque-unique_at_runtime" /></entry></feed>