November 6, 2025
Isaac Warren
Fine-tuning Large Language Models (LLMs) is a powerful way to adapt them to specific tasks, but it often exposes a major bottleneck in AI workflows: the data pipeline. Typically, data engineering and model training are two separate, disconnected worlds. Data teams use tools like Spark to query and preprocess data from a data lake or warehouse (like Iceberg or Delta Lake), save the results to intermediate files (e.g., JSON on S3), and then the ML team spins up a separate cluster to load those files and begin training.
This two-step process is inefficient and not built for the modern AI world where these components have to live together side-by-side, with speedy iteration. It creates data silos, introduces I/O latency, and complicates the entire workflow, requiring two different systems to be managed and scaled. The intermediate files also lose the benefits of data warehousing such as strong schemas and version control which can complicate the data loading process.
Using Bodo, we can close this gap. With Bodo, the entire pipeline, from raw data in your warehouse to a fine-tuned model, exists in a single, unified application. Bodo DataFrames provides a distributed, high-performance engine for data loading and preprocessing, while Bodo AI Toolkit seamlessly hands that data off to a distributed PyTorch training job, all using familiar Python APIs and scaling to large clusters efficiently.
In this post, we’ll show how to use Bodo DataFrames to load and preprocess data directly from an Apache Iceberg table and feed it seamlessly into Bodo AI Toolkit to fine-tune a Llama 3.1 8B model using LoRa. We'll be training a chatbot on its own "liked" feedback to improve its responses.
Bodo is an open-source, high-performance DataFrame library for Python that is a drop-in replacement for Pandas. Bodo simplifies accelerating and scaling Python workloads from laptops to clusters without code rewrites. Under the hood, Bodo relies on MPI-based high-performance computing (HPC) technology and an innovative auto-parallelizing just-in-time (JIT) compiler. This makes it both easier to use and often orders of magnitude faster than tools like Spark or Dask.
First, let’s start by installing the packages we need:
conda install bodo-ai torch transformers peft
Next, open a new notebook or file and import the required packages and set up our configurations.
Our goal is to create a seamless flow from data preprocessing to model training, eliminating the need to save intermediate files.
First, we define our load_data function. This function uses Bodo DataFrames to read directly from our Iceberg tables stored in S3.
Note: We use a “directory catalog” that just consists of metadata files in S3 in this example to ease deployment of a public, read-only Iceberg table but directory catalogs should not be used in a production environment.
This code looks like standard Pandas, but Bodo executes it as a high-performance, distributed query. Instead of a separate Spark job, our Python application itself is handling the large-scale ETL. The resulting final_df is a BodoDataFrame, a distributed object ready for the next step.
Next, we define our PyTorch Dataset. This is a standard class, but it’s designed to work directly with the Bodo DataFrame that load_data will provide.
Now, we set up our main training function. We load the Llama 3.1 model and apply a LoRa configuration using PEFT. The key line is model = bodo.ai.prepare_model(model), which prepares our model for distributed training.
Notice what's happening:
train_df (our Bodo DataFrame) is passed directly into train_main.bodo.ai.prepare_model takes the model and automatically distributes it across all available GPUs in the node(s) in the cluster.bodo.ai.prepare_dataset rebalances the DataFrame onto workers assigned to GPUs and then loads it into LlamaDataset to create a high-performance, distributed-aware data loader.The data flows directly from the Iceberg query (ETL) to the DataFrame (preprocessing) to the GPU workers (training) without ever being written to disk as an intermediate file.
Finally, we tie it all together with:
The train_df created by load_data is passed directly to bodo.ai.torch_train, which orchestrates the entire distributed training run defined in train_main. The full example is available on our Github.
This example demonstrates a unified pipeline for AI. We went from raw data in an Iceberg warehouse to a fine-tuned LoRa adapter for Llama 3.1, all within a single Python application.
By leveraging Bodo, we eliminate the traditional barrier between data engineering and machine learning. There is no separate Spark ETL job, no intermediate JSON files, and no complex data hand-off. Bodo’s ability to use the Pandas API for distributed data processing and feed it directly into a distributed PyTorch job brings HPC-grade performance and scalability to the entire workflow, not just one piece of it.
To get started using Bodo yourself:
