Today, in partnership with Google Cloud, we’re unveiling a significant evolution for Firestore that gives you a completely reimagined query engine designed for maximum expressiveness and power. Along with Firestore’s new Pipeline operations, this advanced query engine features over a hundred new query capabilities driven by the latest advancements within the Firestore Enterprise edition. Now you can build even more complex and efficient applications while still getting the real-time sync, offline caching, and AI-powered coding help you love.

With a new query engine comes new querying capabilities

To introduce Pipeline operations, we overhauled Firestore Enterprise’s query engine to give you more queryability and flexibility, something you’ve long requested. This new query engine brings us to feature parity with other major NoSQL databases and also allows us to add more rich query features.

Starting today, you can chain together multiple stages within a query for things like arbitrary aggregations, map operations, and string matching using regular expressions. You can run queries with indexes being optional, giving you full control over when you want to create indexes to improve query performance. To give you an idea of what’s now possible, here’s an example:

Note: This example assumes you’re familiar with Firestore’s data model and query methods.

Suppose you have an app for sharing food recipes, and you allow user-defined tags on recipes. You might represent a recipe as a document with some fields for essential data like the recipe text itself. Since a tag can be represented with just a string, you could add tags directly to the recipe document as an array of strings:

{
  title: "My recipe",
  instructions: "Cook the ingredients",
  authorId: "SomeAuthorID",
  tags: ["easy", "high protein", "low carb"],
  ...
}

If you’re familiar with Firestore, you know how to write a query to search for all recipes that contain a given tag. But what if you want to present a list of most-popular suggested search tags for the user? Previously, there was no way to extract the array data from this document and aggregate it within a query. You would have to maintain tag metadata elsewhere in your database and keep it up-to-date.

With Pipeline operations in Firestore Enterprise edition, we can unnest the array and pick out the most popular tags to suggest to the user. Here’s how to do just that in JavaScript:

// Fetch 10 tags sorted by most popular.
const snapshot = await db.pipeline()

  // Starting with the collection of recipe documents:
  .collection("recipes")

  // Unnest each tag within the `tags` array to its own document.
  .unnest(field("tags").as("tagName"))

  // Count the number of instances of each tag across recipes and
  // consolidate documents sharing a tagName into a single document
  // per tagName.
  .aggregate({
    accumulators: [countAll().as("tagCount")],
    groups: ["tagName"]
  })

  // Sort the resulting tags by their count.
  .sort(field("tagCount").descending())

  // Limit query results to just the top ten tags.
  .limit(10)

  .execute()

Firestore now embodies the best of both worlds, letting you choose between the simple performance guarantee of Standard edition and the new, more powerful Enterprise edition.

Pipeline operations

Pipeline operations are a set of APIs built into the Firestore SDKs that allow developers to describe transformations within Firestore that are composed of stages, expressions, and functions. Like in a programming language, stages and expressions can be combined in nearly any way, provided the data types are correct. This vastly improves on the capabilities of Standard edition queries. You can try out Pipeline operations immediately by updating your client SDKs to the latest version and connecting your client to an Firestore Enterprise edition database (you’ll need to create a new database). After upgrading, you can convert existing queries to Pipeline operations:

// Existing query
const query = db.collection("recipes").where("authorId", "==", user.id);

// Convert the query into a pipeline
const pipeline = db.pipeline.createFrom(query);

And then you can immediately begin working with the new Pipeline operations.

// From the last snippet
const pipeline = db.pipeline.createFrom(query);

const snapshot = pipeline
  .where(field("rating").greaterThan(4))
  .execute();

We’ve added over a hundred new query features in this update and we’re working on even more, so stay tuned for more updates. You can see the full list of new features in the docs. Pipeline operations are available now on Android, iOS, web, and the admin SDKs and will be coming to Flutter, Unity, and C++ soon.

Indexing and Write Performance

Firestore Enterprise edition doesn’t index collections by default, nor does it require a collection be indexed in order for it to be queried. For veteran Firestore developers, this means Enterprise edition will generally be more performant when writing data and incur lower storage costs. Querying large unindexed collections will be slow, so Firestore provides query analysis tools like Query Explain and Query Insights to help you identify performance bottlenecks in your queries so you can build only the indexes your app really needs.

Enterprise edition also supports sparse, non-sparse and unique indexes, useful for enforcing collection behavior as well as improving query performance.

Should I use the Enterprise edition?

Despite its name, Firestore Enterprise edition is for everyone. You can create an Enterprise edition database via Firebase console and get started with a generous free tier just like Standard edition. When choosing between the two, there are a couple of significant differences:

• Firestore Enterprise edition combines writes and deletes, as simply write operations. It also includes a free tier for write operations. This means you’ll often get more mileage out of the free tier when prototyping on Enterprise edition, especially compared to Standard edition, unless you’re doing a perfectly equal number of deletes and non-delete writes.
• Enterprise edition bills by data chunks for reads and write operations, so if you have small documents you’ll generally be billed for less. See the pricing guide for more details.
• Enterprise edition leaves you in charge of your database indexes, meaning it won’t automatically create single field indexes for you like Standard edition does. This impacts both performance and pricing—unindexed collections may be more expensive and slower to query.
• And, of course, Enterprise edition supports Pipeline operations.

How do I adopt this in my existing Firestore app?

You can use Firestore’s import and export service to migrate your data from an existing Standard edition database into a new Enterprise edition database. Keep in mind you’ll need to recreate your indexes and security rules in the new database, and any existing versions of client apps that point to the old database will still read from and write to the old database.

Once you’ve created your Enterprise edition database, all you need to do is update your SDK to the latest Firestore SDK and initialize it with the name of the database you created earlier.

const db = new Firestore({
  projectId: "your-project-id",
  databaseId: "your-new-enterprise-database"
});

Can I stay on Standard edition forever?

Yes! Standard edition isn’t going anywhere. The SDKs’ query methods you’re familiar with will also continue to remain available, so you do not need to migrate your code in the future. However, pipeline operations will result in a server error if run against a Standard edition database.

What’s next?

Regardless of what stage of the app development lifecycle you’re in, Firestore Enterprise edition is ready in public preview for all of your database needs. Take a look at the new docs to get started, and if you have any feedback, you can contact us here or on the issue trackers of the GitHub repositories for the Firestore SDKs.