Anyone have a solid reference architecture for exporting PostGIS to GeoParquet via Arrow Flight, landing in S3, then serving ad‑hoc reads through DuckDB/HTTP? I’m pushing a nightly batch of about 15M features with GDAL 3.8 in a K8s CronJob and I/O is the choke point; pointers to tuning flags, sample repos, or minimal stacks that worked in production would help?
For the “I/O is the choke point”, the one tweak that moved the needle for me was pre-sorting features by a spatial key (we used H3) before writing and targeting about 128MB Parquet row groups with ZSTD, which cut DuckDB/HTTP range requests by about 3x; can you sort upstream in PostGIS and test on a shard?