The Firebase Blog

Client-side fan-out for data consistency

October 7, 2015

Fan-out is a great strategy when it comes to Firebase. Fan-out itself is the process duplicating data in the database. When data is duplicated it eliminates slow joins and increases read performance.

Not all is perfect with fan-out though. The difficult part of fan-out is keeping each duplicated piece up to date.

The latest release of the Firebase SDKs included multi-path updates. This introduced a technique called client-side fan-out.

With client-side fan-out, fanned-out databases can be both performant and consistent.

Client-side fan-out

Multi-path updates allow the client to update several locations with one object. We call this client-side fan-out, because data is “fanned” across several locations. This proven fan-out technique is used by the giants of social messaging.

The fan-out object is the single object that “fans” out the data. To create a fan-out object, specify the deep path as the key of the object.

var updatedUser = { name: 'Shannon', username: 'shannonrules' };
var ref = new Firebase("https://<YOUR-FIREBASE-APP>.firebaseio.com");
var fanoutObject = {};

fanoutObject['/users/1'] = updatedUser;
fanoutObject['/usersWhoAreCool/1'] = updatedUser;
fanoutObject['/usersToGiveFreeStuffTo/1'] = updatedUser;

ref.update(fanoutObject); // atomic updating goodness

let updatedUser = ["name": "Shannon", "username": "shannonrules"]
let ref = Firebase(url: "https://<YOUR-FIREBASE-APP>.firebaseio.com")

let fanoutObject = ["/users/1": updatedUser, "/usersWhoAreCool/1": updatedUser, "/usersToGiveFreeStuffTo/1", updatedUser]

ref.updateChildValues(updatedUser) // atomic updating goodness

Map updatedUser = new HashMap();
newPost.put("name", "Shannon");
newPost.put("username": "shannonrules");

Firebase ref = new Firebase("https://<YOUR-FIREBASE-APP>.firebaseio.com/");

Map fanoutObject = new HashMap();
fanoutObject.put("/users/1", updatedUser);
fanoutObject.put("/usersWhoAreCool/1", updatedUser);
fanoutObject.put("/usersToGiveFreeStuffTo/1", updatedUser);

ref.updateChildren(fanoutObject); // atomic updating goodness

Then save the object to a top-level location and the updates will be applied to every specific child. In this example we save the fanout object to the root.

The key concept is that this is an atomic operation. Since the update involves only one object the operation will either succeed or fail. This means there’s no incomplete states to deal with.

Client-side fan-out can help keep fanned-out data structures consistent.

Consistency in fanned-out structures

Let’s say you’re a big time social network. A fanned-out data structure is a great use case due to the large amount of users, posts, followers, and timelines. A user has a list of followers. Whenever a user makes a post it shows up in their follower’s timelines. In a large data scenario, a query of all posts would be too slow to get a single user’s timeline. The faster option is duplication.

A flat structure represents each user’s timeline as a data structure.

{
  "timeline": {
    "user2": {
      "-K-zOrtjiCGe7tgRk8DG": {
        "text": "I love emojis!",
        "uid": "user1"
      }
    },
    "user3": {
      "-K-zOrtjiCGe7tgRk8DG": {
        "text": "I love emojis!",
        "uid": "user1"
      }
    }
  },
  "followers": {
    "user1": {
      "user2": true,
      "user3": true
    }
  }
}

This structure duplicates the post to each timeline. Both user2 and user3 are following user1. When user1 sends a post, it’s written to each follower’s timeline. With this structure any given user’s timeline is returned with a single read.

var postsRef = new Firebase('https://<YOUR-FIREBASE-APP>.firebaseio.com/posts/user1');
postsRef.on('value', (snap) => console.log(snap.val()));

let postsRef = Firebase(url: 'https://<YOUR-FIREBASE-APP>.firebaseio.com/posts/user1')
postsRef.observeEventType(.Value) { print($0) }

Firebase postsRef = new Firebase("https://<YOUR-FIREBASE-APP>.firebaseio.com/posts/user1");
postsRef.addValueEventListener(new ValueEventListener() {
    @Override
    public void onDataChange(DataSnapshot snapshot) {
        System.out.println(snapshot.getValue());
    }
    @Override
    public void onCancelled(FirebaseError firebaseError) {
        System.out.println("The read failed: " + firebaseError.getMessage());
    }
});

You’ve finally decided to include the highly-requested edit feature for posts. Updating a single post is not a simple process with a fanned-out structure. Each follower must receive the update. Before client-side fan-out the code was problematic. Below is what we had to do before client-side fan-out. For your safety, please don’t copy and paste this code.

var postsRef = new Firebase('https://<YOUR-FIREBASE-APP>.firebaseio.com/posts');
var queryRef = postsRef.orderByChild('uid').equalTo('user1');
queryRef.on('value', (snap) => snap.forEach((subSnap) => subSnap.ref().update({ title: 'New title' })));

let postsRef = Firebase(url: 'https://<YOUR-FIREBASE-APP>.firebaseio.com/posts')
let queryRef = postsRef.queryOrderedByChild("uid").queryEqualToValue("user1")
queryRef.observeEventType(.Value) { snapshot in
    for child in snapshot.children {
      let childSnap: FDataSnapshot = child as! FDataSnapshot
      childSnap.ref.updateChildValues([ "title": "New title"])
    }
})

Firebase postsRef = new Firebase("https://<YOUR-FIREBASE-APP>.firebaseio.com/posts");
Query queryRef = ref.orderByChild("height").equalTo("user1");
queryRef.addValueEventListener(new ValueEventListener() {
    @Override
    public void onDataChange(DataSnapshot snapshot) {
        System.out.println(snapshot.getValue());
        for (DataSnapshot childSnap : snapshot.getChildren()) {
            Map updatedPost = new HashMap();
            updatedPost.put("title", "New title");
            childSnap.getRef().updateChildren();
        }
    }

    @Override
    public void onCancelled(FirebaseError firebaseError) {
        System.out.println("The read failed: " + firebaseError.getMessage());
    }
});

The problem with this approach is that this duplication process can fail if there’s a lost connection. This failure leaves the data in an incomplete state that can only be fixed by a complicated rollback.

This is the type of problem solved by client-side fan-out.

Fanning out the data

Rather than execute multiple writes to the Firebase database, let’s create one fan-out object to send.

To write to every timeline we need know the user’s uid and their followers. In this example, we pass the user’s uid, an array of the user’s followers, and the post as parameters.

function fanoutPost({ uid, followersSnaphot, post }) {
  // Turn the hash of followers to an array of each id as the string
  var followers = Object.keys(followersSnaphot.val());
  var fanoutObj = {};
  // write to each follower's timeline
  followers.forEach((key) => fanoutObj['/timeline/' + key + '/' + postId] = post);
  return fanoutObj;  
}

func fanoutPost(uid: String, followersSnaphot: FDataSnapshot, post: [String, AnyObject]) -> [String, AnyObject] {
  let followersData = followersSnaphot as! [String: AnyObject]
  let followers = followersData.keys.array
  var fanoutObj = [String, AnyObject]
  // write to each follower's timeline
  followers.forEach { fanoutObject["/timeline/(key)"] = post }
  return fanoutObj
}


public Map fanoutPost(String uid, DataSnapshot followersSnaphot, Map post) {
  Map fanoutObject = new HashMap<>();
  for (DataSnapshot follower: followersSnaphot.getChildren()) {
      fanoutObject.put( "/timeline/" + follower.getKey(), post);
  }
  return fanoutObject;
}

To get the followers, create a listener at the /followers/$uid location. Whether you’re dealing with an Activity, UIViewController, or web page you need to synchronize the followers on load.

(function() {
  var followersRef = new Firebase('https://<YOUR-FIREBASE-APP>.firebaseio.com/followers');
  var followers = {};
  followersRef.on('value', (snap) => followers = snap.val());
}());

class MyViewController : UIViewController {

  var followersRef: Firebase!
  var followersSnap: FDataSnapshot!

  override func viewWillAppear(animated: Bool) {
    super.viewWillAppear(animated);
    followersRef = Firebase(url: "https://<YOUR-FIREBASE-APP>.firebaseio.com/followers")
    followersRef.observeEventType(.Value) { self.followersSnap = $0 }
  }

}


public class MainActivity extends AppCompatActivity {

  Firebase mFollowersRef;
  Map mFollowers;

  @Override
  public void onStart() {
    super.onStart();
    mFollowersRef = new Firebase("https://<YOUR-FIREBASE-APP>.firebaseio.com/followers");
    mFollowersRef.addValueEventListener(new ValueEventListener() {
      @Override
      public void onDataChange(DataSnapshot dataSnapshot) {
        mFollowers = (Map) dataSnapshot.getValue();
      }

      @Override
      public void onCancelled(FirebaseError firebaseError) {

      }
    })
  }
}

When the user sends their post, loop through the follower’s keys and create the fan-out object. The key is the timeline location with the follower’s key as the child.

(function() {
  var followersRef = new Firebase('https://<YOUR-FIREBASE-APP>.firebaseio.com/followers');
  var followers = {};
  followersRef.on('value', (snap) => followers = snap.val());
  var btnAddPost = document.getElementById('btnAddPost');
  var txtPostTitle = document.getElementById('txtPostTitle');
  btnAddPost.addEventListener(() => {
    // make post
    var post = { title: txtPostTitle.value };
    // make fanout-object
    var fanoutObj = fanoutPost({ uid: followersRef.getAuth().uid, followers: followers, post: post });
    // Send the object to the Firebase db for fan-out
    rootRef.update(fanoutObj);
  });

  // ... fanoutPost method would be below
}());

class MyViewController : UIViewController {

  @IBOutlet var titleTextField: UITextField;

  var followersRef: Firebase!
  var followersSnap: FDataSnapshot!

  override func viewWillAppear(animated: Bool) {
    super.viewWillAppear(animated);
    followersRef = Firebase(url: "https://<YOUR-FIREBASE-APP>.firebaseio.com/followers")
    followersRef.observeEventType(.Value) { self.followersSnap = $0 }
  }

  @IBAction func addPostDidTouch(sender: UIButton) {
    // Make a post
    var post = ["title", titleTextField.text]
    // Make fan-out object
    fanoutPost(followersList.uid, followers: followersSnap, post: post)
  }

  // ... fanoutPost method would be below

}

public class MainActivity extends AppCompatActivity {

  Firebase mFollowersRef;
  Map mFollowers;

  @Override
  public void onStart() {
    super.onStart();
    mFollowersRef = new Firebase("https://<YOUR-FIREBASE-APP>.firebaseio.com/followers");
    mFollowersRef.addValueEventListener(new ValueEventListener() {
      @Override
      public void onDataChange(DataSnapshot dataSnapshot) {
        mFollowers = (Map) dataSnapshot.getValue();
      }

      @Override
      public void onCancelled(FirebaseError firebaseError) {

      }
    })

    // Add click handler
    Button mPostButton = (Button) findViewById(R.id.postButton);
    TextView mPostTitle = (TextView) findViewById(R.id.postTitle);

    mPostButton.setOnClickListener(new View.OnClickListener() {
       public void onClick(View v) {
         Map post = new HashMap();
         post.put("title", mPostTitle.getText());
         fanoutPost(mFollowersRef.getAuth(), mFollowers, post);
       }
    });
  }
  // ... fanoutPost method would be below
}

And just like that we have a performant, atomic, and consistent data structure.

Takeaways

If you remember anything from this article, remember these three things:

  • Fan-out is for big data solutions (tens of thousands to millions of records).
  • Duplicate your data to lead to fast single reads.
  • Keep the duplicate data consistent with a client-side fan-out.

How do you structure your data? Let us know if you’ve included the new mutli-path update feature.