In November last year, we announced pipeline operations on Firestore enterprise edition in public preview. This launch brought Firestore’s query configurability and capabilities closer to the capabilities of relational databases like PostgreSQL, while maximizing Firestore’s NoSQL data model, differentiated serverless architecture, built-in real-time sync and embedded offline querying. For example, this launch gave developers granular control over when to create an index to improve performance, as well as the ability to create complex multi-stage queries that weren’t possible in Firestore previously. At Next, pipeline operations graduated to general availability, and with it we’re adding several major heavily-requested queries: full text search, geospatial queries, JOIN capabilities via subqueries, and data manipulation.

Full text search and geospatial queries (in public preview)

Firestore now provides built-in support for full-text search, which allows you to use familiar search query syntax like that of google.com to retrieve relevant Firestore documents. Unlike traditional queries, search queries use special text indexes that tokenize fields in your document. These indexes are then utilized to perform a relevancy search using Google’s best-in-class models, and a search relevance score is appended to the documents returned, allowing you to sort documents by relevance. As an example, suppose you’re the CTO of a company and you want an in-house database of vetted agent skills to provide to employees for security reasons. Your data model might look like this:

{
  "skill": {
    "name": "generate-docs",
    "description": "Generate reference documentation from supported Android, NodeJS, and iOS projects.",
    "skillMD": "The full skill.md text",
    "repo": "https://interal-git-service/skills/generate-docs"
  },
  "platforms": ["macOS", "linux"],
  "agents": ["gemini-cli"],
  "tags": ["codegen", "documentation"]
}

You want your agent skills to be discoverable both via a human-browsable marketplace landing page as well as via another agent skill for finding and installing skills. You can combine Firestore’s search with other pipeline filters to produce a narrow list of results ranked by relevance:

const result = await execute(db.pipeline().collection('skills')

  // Run the full text search. This must occur before other pipeline stages.
  .search({
    // Example query: "\"iOS\" docs generation"
    query: query,
    // Add the relevance score to the document for ranking later.
    addFields: [
      score().as('score'),
    ]
  })

  // Narrow results down to just what the user can use.
  .where(field('agents').arrayContains('gemini-cli')
  .where(field('platforms').arrayContains(targetPlatform)

  // Order by relevance score.
  .sort(field('score').descending)
);

Firestore’s native full-text search provides a number of benefits over third-party search providers or even PostgreSQL’s native search solution. Furthermore, it allows you to run search queries in conjunction with other pipeline stages, automatically handling synonyms, spelling errors, and even difficult-to-handle search languages like traditional Chinese naturally without any plugins. It also uses a familiar search syntax that feels similar to google.com’s search syntax, meaning you don’t need to transform user input much before passing it to Firestore. In this case, combining the Firestore search stage with the two other where filters would allow you to both order by relevance based on the skills.md content and filter only skills compatible with your user’s local environment.

Geospatial queries

We’ve also added geospatial queries so that you can create location-aware applications. For example, you can query geographical coordinates by user proximity, sorting them by their distance to a specified location:

// Find restaurants with "waffles" on the menu, but order the results
// only by distance to a query point.
firestore.pipeline().collection('restaurants')
  .search({
    query: field('menu').matches('waffles'),
    sort: field('location')
      .geoDistance(new GeoPoint(38.989177, -107.065076))
      .ascending()
  });

Since these are new major features, they’re available in preview while the rest of pipelines are GA.

Subqueries: Joins, but better

With Next’s GA release we’ve added a feature called subqueries, allowing you to write queries that combine two arbitrary pipelines into a single pipeline. This means that, for example, if you have a collection of restaurants and a subcollection of reviews:

// Restaurant
{
  "id": "some_id",
  "name": "Pastry Location",
  // ...

  // Review (subcollection)
  {
    "value": 5,
    "body": "A must-visit pastry location."
    // ...
  }
}

You could write a query to aggregate each restaurant’s average rating into a single field per restaurant:

db.pipeline().collection('restaurants')
  .addFields(
    subcollection('reviews')
      .aggregate(average(field('rating')).as('result'))
	// Convert the subquery to a scalar value in the parent query.
      .toScalarExpression()
      .as('average_rating')
  );

You can also use this feature to join documents from separate collections together within a pipeline by subquerying the collection you’d like to join and then passing it as a map into the document of the parent query’s collection, for example if you wanted to display a preview of a restaurant’s reviews under a restaurant thumbnail:

db.pipeline().collection('restaurants')
  .addFields(subcollection('reviews')
  .aggregate(
    average('rating').as('avg_rating'),
    count().as('reviews'))
  .toScalarExpression()
  .as('review_summary')

The resulting document would look like this:

{
  "id": "some_id",
  "name": "Pastry Location",
  "review_summary": {
    "rating": 4.7,
    "reviews": 298
  }
}

Data manipulation language (DML)

Have you ever written a Firestore pipeline query and decided you wanted to pipe the output of that query into some kind of update operation? Maybe you’re building a restaurant finder application and you want to add food categories to your app, requiring you to backfill a new field across every document that doesn’t already have it defined. Now you can do this directly within Firestore.

const snapshot = await db.pipeline()
  .collection('restaurants')
  .where(field('category').exists().not())
  .addFields(constant('uncategorized').as('category'))
  .update()
  .execute();

DML is a set of pipeline stages that allow you to perform updates and deletes, so now you can make granular adjustments to your database fully within Firestore using the expressive pipeline operations to describe exactly which documents should be affected. Calling the update() function will merge the added field back into the document, persisting the value added by addFields(). Paired with the exists() check, this code conditionally adds fields only if they don’t exist. As this example shows, combining DML with other pipeline operations is an effective way to update entire collections of documents within Firestore without having to spin up a Cloud Function or other external service.

DML launched last week with updates and deletes, with inserts and transactions coming soon.

What’s next for Firestore?

After many years of waiting and many words on updates to the internals of Firestore, the queries you have been requesting from launch are now finally here for you to use. With these changes, we have achieved a major milestone in bringing Firestore to query feature parity with other major relational databases like PostgreSQL. Now, when you’re choosing a database for your new project, you no longer have to worry that Firestore’s queries won’t support an app feature you want to build.

Thanks for your patience, and we hope you continue to enjoy Firestore. As always, you can read more about search, DML, and subqueries as well as our getting started guide in the Firebase documentation.