# Create a media set batch pipeline with Pipeline Builder In this tutorial, you will use Pipeline Builder to create a standard batch pipeline with media sets to extract text from PDF. This example uses PDFs of publicly available documents published by Palantir. At the end of this tutorial, you will have a pipeline that looks like the following: ![A complete Pipeline Builder pipeline.](../../../assets/cc4c0818725d86815b693d8ca1944e9b0c9c62bee5dabb626599cef4c6a38f05.webp) The pipeline will produce a new Object output of the extracted PDF text, which can be used for further exploration. ## Part 1: Initial setup First, create a new pipeline. 1. When logged into Foundry, access **Pipeline Builder** from the left navigation bar. If Pipeline Builder is not in the list of applications, select **View all** and find **Pipeline Builder** under the **Build & Monitor Pipelines** section. ![The Pipeline Builder application, found in the application search.](../../../assets/1f911c21d656a17d579794258b511ae867ed9c41826f1cde664ce1e279683c75.webp) 2. Next, on the top right of the Pipeline Builder landing page, create a new pipeline by selecting **New pipeline**, then choose **Batch pipeline**. Under **Select batch compute**, choose **Standard** or **Faster**. For this example, choose **Standard**. ![Choose to create a standard batch pipeline.](../../../assets/e818454cfb6eaceb3e14120a188477cc2f10cf7430aadd616c8e4cde94bc7393.webp) :::callout{theme="neutral"} The ability to create a streaming pipeline is not available on all Foundry environments. Contact Palantir Support for more information if your use case requires it. ::: 3. Select a location to save your pipeline. You cannot save pipelines in personal folders. ![The choose pipeline location popover.](../../../assets/b1c16350d031a6a4110a437896420a97c0d3753309a0f1b67e93f62d1787d5dd.webp) 4. Choose **Create pipeline**. ## Part 2: Add media sets Now, you can add datasets to your pipeline workflow. For this tutorial, you will use PDFs of publicly available documents from Palantir. 1. From the Pipeline Builder page, select **Add Foundry data** on the home page.

![Add media sets to your pipeline.](../../../assets/54ea8b42a12970938d5d7d0ecaddd5eb1687d583bc7765084207e47c0ec0412f.webp)

You can also select the **Add data** action on the top panel.

![The Add data option in the pipeline graph.](../../../assets/bf42f43ca1fec51bf89506bec03dafadcb35f6737258206321d00e205a87ddb3.webp)

Alternatively, you can drag and drop a file from your computer to use as your media set. 2. If you selected **Add data** or **Add Foundry data**, you will have the option to select your desired media sets.

![Screenshot of Add media set from location popover](../../../assets/ff1f659be0c0c5da555851be4cb9e302c54e8e635cfc128b3bdd868d6a1338ac.webp)

3. When all media sets are selected, choose **Add data**. 4. When you have imported your media set you will be able to see the media set with thumbnail preview.

![Screenshot of imported media set](../../../assets/cafc541f5c8f2ca18d5d747e5e0b4cd485ac49acc83c8b29771a89325e9f3a5a.webp)

## Part 3: Media set transformations After adding raw media sets, you can perform some basic transformations. For this workflow, you will extract the text from these PDF files. ### Extract text from PDF You can directly transform media or extract information from media using [media references](../../data-integration/media-sets/index.md). In this example, you will extract text from the `Media set of Annual Letters` media set. 1. Choose the `Media set of Annual Letters` node in your graph. 2. Select **Transform**.

![Screenshot of Media set of Annual Letters node](../../../assets/437f6937d1813a17e7558e0fcd7d6163e95d3624c6a2641749c80c7558c0cd2b.webp)

3. Search for and select the **Extract text from PDF** transform from the dropdown to open the board.

![Extract text transform options](../../../assets/541d7e637d4e325c15e937b31f1101d4ce30573c9b2df7b93d79aa0f5ce79a91.webp)

4. Select the extract method according to your needs and fill out the rest of the parameters. * `Raw text`: Computer-generated PDFs * `OCR`: Photocopies * `Layout aware`: Text and bounding boxes

![Text Extraction options](../../../assets/7ab52b42914404ef1110f3494c135d867fef4c70702d0d1e312520b87e1b5452.webp)

5. Choose **Apply** to add the transform to your pipeline. 6. Your output should look like this when you hover over the extracted text:

![Text extraction output with Hover](../../../assets/e25931f3dcb36916e8d9f812a0575134eda7c429793c6fd1e145b6f9c8917e9b.webp)

You can now run available string transformations on the extracted text column. 7. Select **Close** at the top right to return to your pipeline graph.

![Screenshot of the transform](../../../assets/7f620f127ca27094ed0e2efd1ce6eb1fa66dcd8a943f27552b38920d3f1347ba.webp)

#### (Optional) Semantic search workflow If desired, you can continue with a [semantic search workflow](../../ontology/overview-semantic-search/index.md) with your extracted text. ## Part 4: Add an output Now that you have finished extracting text from your PDFs and potentially running extra string transformations, you can add an output. For this tutorial, you will add an object output. 1. In the `Transforms` node where you have completed your transformations, select **Add output**.

![Add output from media set transformation](../../../assets/6f92d09605acf16c471ffff5aae1f86cca59a0be8afa3cdcb68616553cfa83d8.webp)

2. Select **New object type**.

![Add new object type](../../../assets/845565f4c551a64a1ee2f5e200073683ded05d8e86f01965ad56c368478ca685.webp)

3. Name your object type and set the Ontology by choosing **Please select an ontology**.

![Rename and set ontology output](../../../assets/8270e74c777af206ec83479f14aeeffa4b3228871f4ae9a4cae77705f01318e3.webp)

4. Select **Edit** and edit any column mapping. Ensure that you choose a valid column for the primary key.

![Edit column mapping](../../../assets/739c3045bcf343748b40efc614f2f543acadfb19b8e709e1acb8c0cc1cb7e332.webp)

## Part 5: Build the pipeline 1. To build your pipeline, make sure to select **Save**, then **Deploy > Deploy pipeline**.

![Screenshot of scheme-filled dataset output pane](../../../assets/6974eed8b7086b2a85191c5687a3055c645b60e48be86375a56430301f3dd439.webp)

2. You should see `Initializing deployment` under the `Deploy Pipeline` sidebar option.

![Initializing deployment](../../../assets/c9963389bffdbb0306f74ba4d9590761d8fb200ed8ced589fbde762a0096dab1.webp)

3. Select **View deployment history** to track the progress of your deployment. You should be led to the `History` tab in your pipeline where you can view the statuses and history of your deployments:

![Deployment in progress](../../../assets/5e10b81c23949197ebd38512cd6bbccb6e2051ba462a1c1f33b7eed50f440f7d.webp)

![Deployment complete](../../../assets/f060f7b3dfab610f9c1658b584145824cb9e738b27e576f62777a075e3173957.webp)

## (Optional) Part 6: North of the Ontology Once deployment has completed and your object is initialized, you should be able to directly action on your object output. Select **Create Workshop module** to generate a Workshop module with your pipeline output. ![Create Workshop module](../../../assets/94016c113900a1e89bb57d2000c8718d727a97568746e608c3d834f36868f04a.webp) With this last step, you have generated your pipeline output and a Workshop module. ## FAQ ### Can I transform media before extracting data from it? Yes, you can transform media before extracting data from it, and this works with many extraction operations including extracting text and using large language models (LLMs). As an example, consider a media set containing landscape-oriented images that need to be rotated clockwise 90 degrees before performing OCR-based text extraction. You can add an expression to rotate images and subsequently add an expression to extract text with OCR. ![Pipeline diagram showing a media set being transformed and then processed by a tabular extraction expression.](../../../assets/6ac0be5206fec4fbaed141d3c08092b8e909fffa84f0200b47a55fd5b3d76eca.webp) :::callout{theme="warning"} To safely write the pipeline results to a dataset output, you must remove the media reference column from the output schema as transformed media references are not valid column outputs. To do this, remove the column using the output dialog. :::