LogicBlox, Inc.

Workflow showing how to partition a data.csv file into multiple files given a map.csv categorization file.

For example, the data file can contain sales and returns with columns SKU|SALES|RETURNS, while the map file categorizes skus with SKU|CATEGORY. The result of the test is a bunch of folders in dest_dir, one for each CATEGORY (partition) in the map, each containing a data.csv file with the skus that belong to that partition.

main {
  lb.PartitionCsv(
    src_file = "partition/files/data.csv",
    partition_file = "partition/files/map.csv",
    dest_dir = "partition/files/%PARTITION_ID%",
    delim = "|"
  )
}