How about list of BlobMetadata objects? That way this doesn't use Blob for the binary payload and for the metadata.

Is this needed or can we put some of the small payloads in a map stored in table metadata?

Yes, we can do that. That would obviously make the spec more complicated. Do you think it is worth it?

I meant to make this document simpler by not including this. We can have a map of summary stats in the table spec changes.

i wanted the writer to sort blobs by type, so that NDVs end up within single read.
Of course, this still requires two reads to get this information (1 for the footer, and 1 for the NDV).
Is it a problem? IDK. We do lots of reads when planning a scan over an Iceberg table.

if you feel like this is a problem, or a suboptimality that we shouldn't have, I can remove it from here and add it to table spec instead.

I think this approach adds a lot of complexity. If a blob can be distilled into a few scalar values, like the theta sketch and a single NDV, then I would simply put the values into blob metadata. No need for additional blobs and encodings.

I think this approach adds a lot of complexity.

I think it's actually simpler, because it doesn't require any new type of entity like "blob metadata without the blob itself".

Note that NDV information can exist without a Theta sketch, so we can't just assume that NDV is just an attached info to the sketch.

For example, current Trino version already allows computing NDV and some engine & SPI changes are required to support Theta.

Uncompressed length would be nice.

I'd rather let the compression handled that. Both ZSTD and LZ4 can write uncompressed size in the frame header.

IIRC some compressions libraries (in Go) don't make the information easy to obtain from the frame header. I don't recall which compression formats.

I don't see why this is needed, since it isn't a sketch. I think it is valuable to have either in the metadata that tracks this file, or in the footer. But a blob just to hold it seems overly complex to me.

What about introducing a summary object to BlobMetadata that can contain high-level information like this? Then we can specify that for the apache-datasketches-theta-v1 blob, the summary metadata will have a "ndv-count" field with the number of distinct values as a JSON long.

I don't see why this is needed, since it isn't a sketch.

This file format is not about "sketches". It's a container format for statistics and indexes, and some of them may be "sketches". This is why the spec uses a more vague term "blob".

NDV number is defined as a blob type, to avoid special casing this particular information.
Otherwise we should add this information in the Table spec directly, not in this file format at all.

This is a summary of a blob. We don't need a specific blob type for it and I think that having one makes this overly complex. What is the argument for making this a blob vs keeping it in the sketch metadata as a summary?

removed