---
title: "Extracting structured data from PDFs using Gemini 2.0 and Genkit"
description: "Extract structured data from PDFs using Gemini 2.0's vision and Genkit. Step-by-step guide with code examples for Node.js developers."
published: "2025-02-20"
last_updated: "2025-02-20"
authors:
  - "Peter Friese (Developer Advocate)"
categories:
  - "Genkit"
  - "AI"
  - "Gemini"
  - "Node.js"
  - "TypeScript"
canonical_url: "https://peterfriese.dev/blog/2025/gemini-genkit-pdf-structured-data/"
---

# Extracting structured data from PDFs using Gemini 2.0 and Genkit

_Leverage Gemini's vision capabilities_

*Published February 20, 2025 by Peter Friese (Developer Advocate)*

---
Last week, Google [released Gemini 2.0](https://developers.googleblog.com/en/gemini-2-family-expands/), and after reading this blog post about [parsing PDFs using Gemini 2.0](https://www.philschmid.de/gemini-pdf-to-data), I decided to rebuild the sample using [Genkit](https://genkit.dev/docs/get-started/).

For those who are unfamiliar, Genkit is an AI integration framework that makes implementing AI features for your apps easier. 

In this blog post, I am going to focus on Genkit for Node.js, but the team is also working on building support for Go ([available in alpha](https://genkit.dev/go/docs//get-started-go)) and Python ([under development](https://github.com/firebase/genkit/tree/main/py)). This means you can use Genkit on any runtime that supports one of those three languages. The sample app I'm discussing in this blog post is a simple command-line utility that you can run on macOS or any other platform that supports Node.js, but you can also use Genkit on the backend, for example in [Cloud Functions for Firebase](https://firebase.google.com/docs/functions), [Cloud Run](https://cloud.google.com/run), or any other environment that supports Node.js.

### Setup

Before you can use Gemini 2.0 Flash in Genkit, you need to install and add Genkit and the Google AI SDK plugin for Genkit. 

You can then instantiate Genkit and define a default model. This model will then be used in all subsequent calls you make via the `ai` instance.

**`analyseInvoice.ts`**

```typescript
import { gemini20Flash, googleAI } from '@genkit-ai/googleai';
import { genkit, z } from 'genkit/beta';

// instantiate Genkit
const ai = genkit({
  plugins: [googleAI()],
  model: gemini20Flash,
});
```

For a detailed walkthrough of the setup process, check out this video:

[Video: Getting started with Genkit 1.0 for Node.js](https://www.youtube.com/watch?v=3p1P5grjXIQ)

### Extracting structured data

In this first example, let's look at how you can use Gemini 2.0 Flash and Genkit to extract structured data from PDFs. Let's imagine you've received a PDF with an invoice. To further process the invoice, you need to extract the invoice number, date, all the line items (including description, amount, and their individual price), as well as the gross price of the invoice.

![Invoice](https://firebasestorage.googleapis.com/v0/b/first-class-blog.appspot.com/o/posts%2Fgemini-genkit-pdf-structured-data-invoice_1600x800.webp?alt=media&token=fcaa8b38-35b6-4fce-868b-1ab8317d8cd2)

Thanks to Gemini's [vision capabilities](https://ai.google.dev/gemini-api/docs/vision?lang=node), we can let it handle the OCR for us. The following code snippet shows how to call *any LLM* via Genkit's unified generation API, and pass a multimodal prompt (including a URL to the PDF file we want to analyse as well as an instruction).

**`analyseInvoice.ts`**

```typescript
export async function analyseInvoice(url: string) {
  const { output } = await ai.generate({
    prompt: [
      { media: { url } },
      { text: "Extract the structured data from the following PDF file" }
    ]
  });
  return output;
}
```

If you'd call this method now for example by passing [this URL](https://storage.googleapis.com/generativeai-downloads/data/pdf_structured_outputs/invoice.pdf) (which points to an invoice), you'd get the following result:

**`result.json`**

```json
[
  {
    'Invoice no': '27301261',
    'Date of issue': '10/09/2012',
    Seller: {
      Name: 'Williams LLC',
      Address: '72074 Taylor Plains Suite 342\nWest Alexandria, AR 97978',
      Tax Id': '922-88-2832',
      IBAN: 'GB70FTNR64199348221780'
    },
    Client: {
      Name: 'Hernandez-Anderson',
      Address: '084 Carter Lane Apt. 846\nSouth Ronaldbury, AZ 91030',
      'Tax Id': '959-74-5868'
    },
    Items: [
      [Object], [Object],
      [Object], [Object],
      [Object], [Object],
      [Object]
    ],
    Summary: {
      'VAT [%]': '10%',
      'Net worth': '494,96',
      VAT: '49,50',
      'Gross worth': '544,46',
      Total: '$ 494,96',
      'Total VAT': '$ 49,50',
      'Total Gross worth': '$ 544,46'
    }
  }
]
```

Which is pretty impressive - after all, the model was able to correctly extract all information. However, this might not be in the format you need this in.

This is where [Genkit's support for structured output](https://g.dev/ai/genkit-structured-output-models) comes in handy. You can define a schema for your data using [Zod](https://zod.dev/), like this:

**`analyseInvoice.ts`**

```typescript {11-22}
export async function analyseInvoice(url: string) {
  const { output } = await ai.generate({
    prompt: [
      { media: { url } },
      { text: "Extract the structured data from the following PDF file" }
    ]
  });
  return output;
}

const LineItemSchema = z.object({
  description: z.string(),
  quantity: z.number(),
  grossWorth: z.number(),
});

const InvoiceSchema = z.object({
  invoiceNumber: z.string(),
  date: z.string().describe("The invoice date"),
  items: z.array(LineItemSchema),
  totalGrossWorth: z.number()
});
```

Then, pass the schema to Genkit's `generate` call to enforce this schema.

**`analyseInvoice.ts`**

```typescript {7-9}
export async function analyseInvoice(url: string) {
  const { output } = await ai.generate({
    prompt: [
      { media: { url } },
      { text: "Extract the structured data from the following PDF file" }
    ],
    output: {
      schema: InvoiceSchema
    }
  });
  return output;
}
```

And now, calling the function with the same PDF file will yield the following result:

**`result.json`**

```json
{
  date: '10/09/2012',
  invoiceNumber: '27301261',
  items: [
    {
      description: 'Lilly Pulitzer dress Size 2',
      grossWorth: 247.5,
      quantity: 5
    },
    {
      description: 'New ERIN Erin Fertherston Straight Dress White Sequence Lining Sleeveless SZ 10',
      grossWorth: 65.99,
      quantity: 1
    },
    {
      description: 'Sequence dress Size Small',
      grossWorth: 115.5,
      quantity: 3
    },
    {
      description: 'fire los angeles dress Medium',
      grossWorth: 21.45,
      quantity: 3
    },
    {
      description: "Eileen Fisher Women's Long Sleeve Fleece Lined Front Pockets Dress XS Gray",
      grossWorth: 52.77,
      quantity: 3
    },
    {
      description: 'Lularoe Nicole Dress Size Small Light Solid Grey/ White Ringer Tee Trim',
      grossWorth: 8.25,
      quantity: 2
    },
    {
      description: 'J.Crew Collection Black & White sweater Dress sz S',
      grossWorth: 33,
      quantity: 1
    }
  ],
  totalGrossWorth: 544.46
}
```

Since this adheres to the schema we provided, it will be much easier to process.

Gemini 2.0 Flash supports structured output, but even if you're using an LLM that doesn't support natively structured output, Genkit can help by augmenting the prompt and coercing the returned data for you. What's really amazing about this: you won't have to change your code, thanks to Genkit's unified generation API (check out [this thread on Bluesky](https://bsky.app/profile/peterfriese.dev/post/3li5dz6evvc2n) in which I demonstrate this).

### Parsing hand-written form input

I mentioned Gemini's OCR capabilities, so let's try extracting the hand-written entries in the following PDF:

![Form](https://firebasestorage.googleapis.com/v0/b/first-class-blog.appspot.com/o/posts%2Fgemini-genkit-pdf-structured-data-form_1600x800.webp?alt=media&token=df2891c7-0a9a-4b3a-b1b3-6f724b0f2cb7)

One way to achieve this would be to define a new schema and then call Genkit's `generate` method using this new schema. But there is a more flexible way to do this. It turns out that the actual code for calling Gemini to extract data from a PDF is always the same - only the output format (and potentially the instructions) change for each type of PDF we want to analyse. 

So why not make this a bit more generic?

#### Managing prompts

Genkit provides a way to extract prompts into external files, using the [Dotprompt](https://genkit.dev/docs/dotprompt) library. This allows you to keep your code free from any prompt, making it easier to change the prompt independently of the code.

Let's first define a prompt file:

**`form.prompt`**

```handlebars
---
model: googleai/gemini-2.0-flash
input:
  schema:
    pdfUrl: string
  default:
    pdfUrl: "https://storage.googleapis.com/generativeai-downloads/data/pdf_structured_outputs/handwriting_form.pdf"
output:
  schema:
    formNumber: string, Handwritten Form Number
    startDate: string, Effective Date of the plan
    beginningOfYear: integer, The plan liabilities beginning of the year
    endOfYear: integer, The plan liabilities end of the year
    name: string, Name of the plan
    address: string, Mailing address
    phoneNumber: string, Employer's phone number
    ein: string, Employer Identification Number
---
Extract the hand-written form data.

{{media url=pdfUrl}}
```

This prompt file defines the model Genkit should use (Gemini 2.0 Flash), as well as the prompt itself. But it also specifies the input *and* output schemas for the prompt. For example, under `input/schema`, we define an input parameter named `pdfUrl` of type `string`, which can then be used in the prompt to pass the URL. Likewise, under `output/schema`, we define all the keys we want to extract from the PDF file, including their types. 

Note that the descriptions of the keys do not just serve the purpose of documentation - Genkit will use them in the prompt to specifically tell the model how to find the data for the respective keys.

#### Prompting the model

To load a prompt from a file, use `ai.prompt(filename)`. This has a beneficial side effect: we can now create `.prompt` files for all the different types of data we want to extract from PDF files, and just pass the filename to the `extract` function:

**`extract.ts`**

```typescript
export async function extract(promptName: string, pdfUrl: string) {
  const prompt = ai.prompt(promptName);
  const { output } = await prompt({ pdfUrl });
  return output;
}
```

Instead of using `ai.generate()` to call the LLM, you can execute a prompt by caling the prompt itself, and passing the required parameters.

Calling this method using `extract("form", "https://storage.googleapis.com/generativeai-downloads/data/pdf_structured_outputs/handwriting_form.pdf")` will yield the following result:

**`result.json`**

```json
{
  address: '235, Park Street Avenue, FL',
  beginningOfYear: 40000,
  ein: '13626819',
  endOfYear: 55000,
  formNumber: 'CA530082',
  name: 'Annual Return Plan',
  phoneNumber: '0.11 536289',
  startDate: '02/05/.2022'
}
```

Which - if you ask me - is mind-boggling.

### Where to go from here

To learn more about Genkit, check out the [documentation](https://genkit.dev/docs/get-started/). Also, feel free to drop by my [weekly livestreams](https://www.youtube.com/playlist?list=PLsnLd2esiGRTmfGZcZMnEy6hkBHXBH_en) - I'm currently working on implementing a Second Brain app, and I'm using Genkit for some of the AI-powered features.

If you'd like to try out the sample app, the [source code is available on GitHub](https://github.com/peterfriese/genkit-samples).