# Create a streaming pipeline with Pipeline Builder In this tutorial, we will use Foundry Streaming and Pipeline Builder to create a simple pipeline with an output of a single dataset with information on sensor temperatures. You will learn how to create a stream in Foundry, push records into that stream, and transform them in Pipeline Builder. ## Part 1. Initial setup First, we need to create a new stream. 1. When logged into Foundry, navigate to a [Project](../../compass/move-and-share-resources/index.md) in Foundry, select **+ New** in the top right corner, then select **Stream**. ![Screenshot of stream create dropdown](../../../assets/ed2f0e01233eac3355b136b3aec28bd5a6ef8ec1f34b1f89a0308f7b3bb1342b.webp) 2. Next, you will need to define your stream. For this guide, we will create a simple one partition stream and manually push records to it. On the **Define** page, select **Normal** for the throughput and define a basic schema as: **sensor\_id:** `String`, **temperature:** `Double`. ![Screenshot of stream definition page](../../../assets/9d719fee8fe21a0baebbad73c79a2f61199925e713a101b161d3d9e531a0e109.webp) 3. Select **Create stream**. This will take you to the **Connect** page where you can specify how to connect to the streaming data. ## Part 2. Push records into the stream We are now ready to connect our stream. At this point, we could set up a streaming data ingestion task with a [source](../../data-integration/source-type-overview/index.md). For this tutorial, we will instead manually push records to the stream with **Curl.** 1. First, select **Curl (Bash)** under the **Connect via API** section to set up authentication for your stream. We will use a personal token to submit records. ![Screenshot of stream connection page](../../../assets/5cf9b65a35639b050f2f6da2f5c8f39b3e3f23b3e8fa7ffbd2c495fb7c98958f.webp) 2. Select **Test with a personal token** and follow the on-screen prompts for generating a short-lived personal token. Personal tokens should not be used for production pipelines. Production pipelines should use an [OAuth token workflow](../../platform-security-third-party/writing-oauth2-clients/index.md#oauth2-api-reference). ![Screenshot of stream connection auth page](../../../assets/fee55c4e5c9da48c4d4c6ae4fc5d76725f5e222186c41ee84828e553d727b208.webp) 3. Paste your generated token into the text box, then click **Next Step**. 4. Copy the **Curl** command. Open a terminal on your computer that can execute Bash and paste the command. Run the command in your terminal. ![Screenshot of stream push with curl page](../../../assets/15141f0579ff4b366bc45523eec4a9cf96e25bcbd2e17ede9923ed60cc958512.webp) Within seconds, you will see a record appear in the stream viewer on the page: ![Screenshot of stream view records tab](../../../assets/29315750f4a5f04f39ca6b25348bfc1b2146948b19fb4dce062ea74eef29e181.webp) You have now ingested streaming data in real time. Next, transform that data. ## Part 3. Transform a stream 1. Select the **Start pipelining** button to begin writing a basic streaming transform in Pipeline Builder. ![Screenshot of stream view records tab](../../../assets/7664521b7bdb4f6f0af68ea420ff7b15d0a8e4703b8aa406b450dd86e359629e.webp) 2. In the **Create new pipeline** modal, select the **Streaming pipeline** type, and click **Create Pipeline**. ![Screenshot of create builder stream pipeline](../../../assets/02890a269a780de74f02fa7229f475da1a4b5297f87e47e77114253fb0d24a0c.webp) This will create a pipeline for the input stream, displayed on a graph. Selecting the input stream node will display a preview of the data. The preview runs on a cold storage view of the stream; records from the stream will be delayed before they appear. ![Screenshot of builder graph with input stream](../../../assets/bb8241f43dcb53befd27b29800c6c9341e2bfef9010eccfd0818dc41a8a5a214.webp) 3. Click on the input stream node on the graph and select the **Transform** action (the blue **T** icon next to the input node). This will open a list of all transforms currently supported for streams based on the input types of the columns in the stream. For this tutorial, we will convert all `sensor_ids` to uppercase, remove any whitespace on them, and filter by temperatures exceeding three degrees. ![Screenshot of builder stream transform dropdown](../../../assets/ce37ef81172cc556c32144f871fe400e3c885aa7acae7022c9f70f9572e99f67.webp) 4. Select the **Uppercase** transform, choose the `sensor_id` column, and click **Apply**. ![Screenshot of builder stream uppercase transform](../../../assets/225b80db758988bb42b229fdf3d8f1de79b7bcb2679fd1616ae75a45f43dddb9.webp) 5. Then, search for the **Trim whitespace** transform and select it. Choose the `sensor_id` column again, and click **Apply**. ![Screenshot of builder trim whitespace transform](../../../assets/c06fe0c3f737f963556360f753ba52a03d7e7f22b5d519969eac618b3970ec97.webp) 6. For the final transform, first search for the **Filter** transform and choose **Keep rows**. Then, select the `temperature` column, set the filter to **greater than** `3`, and select **Apply**. ![Screenshot of builder filter transform](../../../assets/ce7a93ba322ad711a61d03860a2f43dedd0b8d1a4c6d422769a063baa3906a7b.webp) 7. Click **Apply all changes** to the top right of your screen. Then, select **Back to graph** to return to your pipeline. ![Screenshot of builder graph with transform](../../../assets/092f3b868146a63c0891329a6fbff88f47fc196c1b9e5b338528ac77d1c23a3a.webp) 8. Select the **Transform path** node we just created, then click **New dataset**. ![Screenshot of builder graph creating new output](../../../assets/4661a3f518d7dbf1c6e35a2db45d3851fc2b5f6f0240d99eb2608d45d7e8dab6.webp) 9. In the top right corner of the application, first click **Save** to apply all new changes to your pipeline. Then, click **Deploy** and **Deploy pipeline**. :::callout{theme="neutral" title="Deployment validation"} Before deployment begins, Foundry validates that the pipeline is deployable. This validation checks whether the pipeline configuration and required resources are ready for deployment. If validation fails, review the messages in the deployment dialog, resolve each issue, save your changes, and deploy again.

If you save your changes without deploying them, your pipeline logic will **not** update to the latest changes. You must deploy the pipeline to capture changes to transform logic. ::: ![Screenshot of builder graph deploy dropdown](../../../assets/da9dfb1e848b5ffb71fdc4b9099b96a40c7778202ef62bdd3ab8948ba1f0a82a.webp) 10. Select the output stream node you just created, then click on the stream name above the **Data preview** section at the bottom of your graph. ![Screenshot of builder graph with deployed output](../../../assets/e85a347fb911399b0913a73b03e86b3a7d97662975876bdab84d2958898591e2.webp) This will take you to the stream preview page with the output stream from your transform. The streaming cluster takes about one minute to start, so you may not see records immediately. Once running, however, the cluster will process all new records in real time. ![Screenshot of output stream](../../../assets/db17f9454d83c784407d72a321a8befa4c9b0b174e81d5812f355acf0d1b963f.webp) ## Next steps Now that you know how to create a simple streaming pipeline, learn more about managing streams by exploring how to [debug a failing stream](../../optimizing-pipelines/debug-stream/index.md). For more advanced transform functionality, learn more about [Pipeline Builder](../../pipeline-builder/overview/index.md).