Video by Hugging Face via YouTube
Files keep landing in your Hugging Face Storage Bucket. Here is how to process each one automatically, without downloading anything.
In this live-coding video we set up a webhook on a bucket that triggers a Hugging Face Job whenever a new file arrives. The Job runs the `optimize-parquet.py` uv script, converts the new CSV to optimized Parquet, and writes it to a second bucket. We upload a 10,000-row slice of the NYC taxi dataset, and the triggered Job finishes in about 8 seconds. For data engineers already using Buckets who want to automate what happens next.
Topics covered:
– Hugging Face Storage Buckets with the hf CLI
– Running a uv script as a Hugging Face Job
– Webhooks that trigger Jobs on bucket changes
– Converting CSV to Parquet automatically
– Checking Job status, logs and output
Resources:
– Webhooks guide (bucket to Jobs): https://huggingface.co/docs/hub/webhooks-guide-bucket-jobs
– optimize-parquet.py script: https://huggingface.co/datasets/uv-scripts/data-processing/raw/main/optimize-parquet.py
– My notes and demo code: https://github.com/prpatel/simple-buckets-demo
– Hands-on with Buckets (article): https://huggingface.co/blog/prpatel/hands-on-with-buckets
Chapters:
0:00 Intro
0:16 Why automate bucket processing
0:39 Input and output buckets
1:39 Define the Job
2:20 Create a webhook on the bucket
3:21 Upload a new CSV
4:01 Check the Job and its output
5:04 Wrap up