Expose table statistics in Table API #4741

findepi · 2022-05-10T12:55:42Z

Follows #4945 and #5021

rdblue · 2022-05-10T15:40:10Z

@findepi, I think this is too big to go in on its own. Can you split it into separate PRs for different parts?

findepi · 2022-05-10T20:17:01Z

@rdblue i have this split. #4537 has already been reviewed.

core/src/main/java/org/apache/iceberg/SnapshotParser.java

core/src/main/java/org/apache/iceberg/GenericStatisticsFile.java

core/src/main/java/org/apache/iceberg/SnapshotParser.java

alexjo2144 · 2022-05-18T19:02:22Z

We need to remember to keep remove_orphan_files from deleting Statistics files

findepi · 2022-05-19T07:19:42Z

We need to remember to keep remove_orphan_files from deleting Statistics files

Thanks for reminding about this here. It's on my todo list, see eg trinodb/trino#12317 (comment)

findepi · 2022-05-20T09:15:13Z

We need to remember to keep remove_orphan_files from deleting Statistics files

Fixed & test added.

findepi · 2022-08-29T15:03:23Z

Updated to the current state of #5021

findepi · 2022-08-30T17:38:46Z

Rebased after #5021 has been merged to make Conflicts disappear.

findepi · 2022-09-06T13:33:54Z

@rdblue please let me know if you have any comments.

rdblue · 2022-09-08T16:56:38Z

api/src/main/java/org/apache/iceberg/Table.java

+   *
+   * @return the current statistics files for the table
+   */
+  List<StatisticsFile> statisticsFiles();


Do we want to expose these directly? What about instead adding an API for finding statistics? That way consumers don't need to decide which one to use themselves.

Either way, this is for consuming stats and we should add the implementation of the update stats API in its own PR.

Do we want to expose these directly?

Yes, these should be exposed, just like we expose other information about table (files, their min/maxes, etc.) and we provide convenience functionality on top of them (like planFiles).

What about instead adding an API for finding statistics?

Good idea. What would you like to see here?

api/src/main/java/org/apache/iceberg/UpdateStatistics.java

rdblue · 2022-09-08T16:58:28Z

core/src/main/java/org/apache/iceberg/GenericBlobMetadata.java

-public class GenericBlobMetadata implements BlobMetadata {
+public class GenericBlobMetadata implements BlobMetadata, Serializable {
+
+  public static BlobMetadata from(org.apache.iceberg.puffin.BlobMetadata puffinMetadata) {


Why don't we make it so that the Puffin BlobMetadata implements BlobMetadata? Then this would be unnecessary.

Puffin BlobMetadata contains all the blob metadata information from the Puffin footer.
In particular, it contains more fields, like offset and length, which are not carried over to the table metadata.

We can make Puffin BlobMetadata implement the BlobMetadata, but

we don't want to serialize (as in Serializable) the Puffin BlobMetadata, as we're not interested in sending over those additional fields (I know these are small today, and maybe are never big in the future)

there is a challenge with equality semantics. GenericBlobMetadata implements equals, but with more complicated class hierarchy it wouldn't be useful anymore.

Let me know what you think

we don't want to serialize (as in Serializable) the Puffin BlobMetadata

i see #4741 (comment) now

core/src/main/java/org/apache/iceberg/GenericStatisticsFile.java

core/src/main/java/org/apache/iceberg/ReachableFileUtil.java

core/src/main/java/org/apache/iceberg/GenericBlobMetadata.java

findepi · 2022-09-19T12:29:14Z

Per #4741 (comment), I've shrunk the scope of this PR
and extracted #5794 and also #5795 .

rdblue · 2022-09-19T18:10:24Z

Thanks, @findepi! I'll give this another look.

findepi · 2022-09-19T18:52:43Z

Thanks @rdblue . #5799 might be the easiest to start from.

This adds support in `Table` for the table statistics.

findepi · 2022-09-27T11:46:03Z

@rdblue this is ready to review.

rdblue · 2022-09-29T21:57:32Z

Thanks, @findepi!

findepi · 2022-09-30T09:46:24Z

Thank you for the merge!

This was referenced May 10, 2022

Add reader and writer for Puffin, indexes and stats file format #4537

Merged

Analyze Iceberg tables trinodb/trino#12317

Closed

github-actions bot added API build core labels May 10, 2022

This was referenced May 10, 2022

Add statistics information in table snapshot apache/iceberg-docs#77

Closed

Add stats format specification apache/iceberg-docs#69

Closed

findepi force-pushed the findepi/stats-in-table branch from e258b4f to e35cf57 Compare May 10, 2022 14:18

findepi requested review from rdblue, danielcweeks and RussellSpitzer May 10, 2022 14:19

findepi mentioned this pull request May 12, 2022

Remove redundant assignment #4755

Merged

zinking reviewed May 13, 2022

View reviewed changes

core/src/main/java/org/apache/iceberg/SnapshotParser.java Outdated Show resolved Hide resolved

zinking reviewed May 13, 2022

View reviewed changes

core/src/main/java/org/apache/iceberg/GenericStatisticsFile.java Outdated Show resolved Hide resolved

zinking reviewed May 13, 2022

View reviewed changes

core/src/main/java/org/apache/iceberg/SnapshotParser.java Outdated Show resolved Hide resolved

findepi force-pushed the findepi/stats-in-table branch 2 times, most recently from 2a9585d to 290152b Compare May 18, 2022 14:21

findepi force-pushed the findepi/stats-in-table branch 2 times, most recently from 24b1a32 to 3e94f20 Compare May 20, 2022 09:14

github-actions bot added the spark label May 20, 2022

findepi force-pushed the findepi/stats-in-table branch 5 times, most recently from 486c731 to ac219c0 Compare May 25, 2022 10:42

findepi force-pushed the findepi/stats-in-table branch from 8a5b261 to 765de2e Compare August 29, 2022 15:01

findepi requested a review from rdblue August 30, 2022 08:10

findepi force-pushed the findepi/stats-in-table branch from 765de2e to 9179106 Compare August 30, 2022 17:38

rdblue reviewed Sep 8, 2022

View reviewed changes

api/src/main/java/org/apache/iceberg/UpdateStatistics.java Outdated Show resolved Hide resolved

rdblue reviewed Sep 8, 2022

View reviewed changes

core/src/main/java/org/apache/iceberg/GenericStatisticsFile.java Outdated Show resolved Hide resolved

rdblue reviewed Sep 8, 2022

View reviewed changes

core/src/main/java/org/apache/iceberg/ReachableFileUtil.java Outdated Show resolved Hide resolved

rdblue reviewed Sep 8, 2022

View reviewed changes

core/src/main/java/org/apache/iceberg/GenericBlobMetadata.java Outdated Show resolved Hide resolved

findepi force-pushed the findepi/stats-in-table branch from b8da141 to 0ce95d9 Compare September 19, 2022 12:12

This was referenced Sep 19, 2022

Support setting table statistics #5794

Merged

Retain table statistics during orphan files removal #5795

Merged

findepi mentioned this pull request Sep 19, 2022

Core: Serialize statistics files in TableMetadata #5799

Merged

findepi force-pushed the findepi/stats-in-table branch from 0ce95d9 to 2444114 Compare September 19, 2022 18:45

findepi changed the title ~~Add implementation for statistics information in table snapshot~~ Expose table statistics in Table API Sep 21, 2022

Expose table statistics in Table API

11fab99

This adds support in `Table` for the table statistics.

findepi force-pushed the findepi/stats-in-table branch from 2444114 to 11fab99 Compare September 21, 2022 10:34

findepi mentioned this pull request Sep 26, 2022

Calculate Iceberg NDV with a Theta sketch trinodb/trino#14290

Merged

rdblue approved these changes Sep 29, 2022

View reviewed changes

rdblue merged commit ae8e669 into apache:master Sep 29, 2022

findepi deleted the findepi/stats-in-table branch September 30, 2022 09:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Expose table statistics in Table API #4741

Expose table statistics in Table API #4741

findepi commented May 10, 2022 •

edited

rdblue commented May 10, 2022

findepi commented May 10, 2022

alexjo2144 commented May 18, 2022

findepi commented May 19, 2022

findepi commented May 20, 2022

findepi commented Aug 29, 2022

findepi commented Aug 30, 2022

findepi commented Sep 6, 2022

rdblue Sep 8, 2022 •

edited

findepi Sep 9, 2022

rdblue Sep 8, 2022

findepi Sep 9, 2022

findepi Sep 9, 2022

findepi commented Sep 19, 2022

rdblue commented Sep 19, 2022

findepi commented Sep 19, 2022

findepi commented Sep 27, 2022

rdblue commented Sep 29, 2022

findepi commented Sep 30, 2022

Expose table statistics in Table API #4741

Expose table statistics in Table API #4741

Conversation

findepi commented May 10, 2022 • edited

rdblue commented May 10, 2022

findepi commented May 10, 2022

alexjo2144 commented May 18, 2022

findepi commented May 19, 2022

findepi commented May 20, 2022

findepi commented Aug 29, 2022

findepi commented Aug 30, 2022

findepi commented Sep 6, 2022

rdblue Sep 8, 2022 • edited

Choose a reason for hiding this comment

findepi Sep 9, 2022

Choose a reason for hiding this comment

rdblue Sep 8, 2022

Choose a reason for hiding this comment

findepi Sep 9, 2022

Choose a reason for hiding this comment

findepi Sep 9, 2022

Choose a reason for hiding this comment

findepi commented Sep 19, 2022

rdblue commented Sep 19, 2022

findepi commented Sep 19, 2022

findepi commented Sep 27, 2022

rdblue commented Sep 29, 2022

findepi commented Sep 30, 2022

findepi commented May 10, 2022 •

edited

rdblue Sep 8, 2022 •

edited