With Pachyderm, we can easily create a data pipeline with a single worker that executes our python script, performing the join, and outputs the joined data: However, we can also instruct Pachyderm to increase the parallelism of our data pipeline (which is controlled in a “pipeline specification”).