Introducing Firebase Queue

May 15, 2015

Many of our developers are using Firebase with a server to perform tasks like background processing, integrating with third party APIs, or handling advanced authentication requirements. Today, we’re introducing Firebase Queue, a fault-tolerant multi-worker job pipeline built on Firebase.

If you’re writing your server code in Node, Firebase Queue makes it easy to handle background jobs with a worker queue. We’re already using it in our private backups feature, where we reliably handle hundreds of jobs per day across multiple machines.

Creating a Queue

Once you’ve installed firebase and firebase-queue, you can get started by creating a new Queue and passing it your Firebase reference and a processing function. This processing function specifies the data transformation that the worker performs and when called, will be passed the following four parameters:

data - A plain JavaScript object containing the claimed task’s data
progress() - A callback function that takes a number from 0 to 100 to report the progress state of the item
resolve() - A callback function that marks the task as complete and allows the worker to process another task
reject() - A callback function that reports any errors from processing the job and allows the worker to move on to processing new tasks

Below is a simple example of what an instance of the queue looks like:

var Queue = require('firebase-queue'),
    Firebase = require('firebase');

var queueRef = new Firebase('https://<YOUR-FIREBASE-APP>.firebaseio.com/queue');
var queue = new Queue(queueRef, function(data, progress, resolve, reject) {
  // Read and process task data
  console.log(data);

  // Update the progress state of the task
  setTimeout(function() {
    progress(50);
  }, 500);

  // Finish the job asynchronously
  setTimeout(function() {
    resolve();
  }, 1000);
});

When your Queue runs, the data will be stored in Firebase under a /queue/tasks path and removed when complete. For an item in your queue called example1, the data in Firebase would look like the following:

{
  "queue": {
    "tasks": {
      "example1": {
        "_owner": "worker_1", // The worker process who owns this task
        "_progress": 88, // A number from 0 to 100 indicating the item’s status in the queue
        "_state": "in_progress", // Indicates whether the job has errored, is in progress or finished
        "_state_changed": 1430937731946, // Timestamp when the state was last updated
        "your": "data" // Original data in your task
      }
    }
  }
}

Adding Tasks to the Queue

You can push an object with some data to the /queue/tasks location in your Firebase using any Firebase client or the REST API. Workers listening at that location will automatically pick up and process the job:

Firebase ref = new Firebase("https://<YOUR-FIREBASE-APP>.firebaseio.com/queue/tasks");
Map task = new HashMap();
task.put("your", "data");
ref.push().setValue(task);

Firebase *ref = [[Firebase alloc] initWithUrl:@"https://<YOUR-FIREBASE-APP>.firebaseio.com/queue/tasks"];
[[ref childByAutoId] setValue:@{@"your": @"data"}];

var ref = Firebase(url: "https://<YOUR-FIREBASE-APP>.firebaseio.com/queue/tasks")
ref.childByAutoId().setValue(["your": "data"])

var ref = new Firebase("https://<YOUR-FIREBASE-APP>.firebaseio.com/queue/tasks");
ref.push({"your": "data"});

curl -X POST -d '{"your": "data"}' https://<YOUR-FIREBASE-APP>.firebaseio.com/queue/tasks.json"

If you don’t want to use the default spec, you can define your own. You can also chain several specs together to create custom jobs. More details on customizing and chaining jobs can be found in the Firebase Queue README.

Queue Examples

We have a few examples to help you get started with Firebase Queue. This gist creates a new Queue with 10 workers and a custom task task_1. In this example, the queue sanitizes chat messages to make sure they don’t contain any banned words.

I’d like your feedback!

When you try out firebase-queue in your app I’d like to hear what you think. Send a pull request, file an issue, or reach out to me on Twitter @drTriumph.

The Firebase Blog