By far, the biggest hurdle for breaking the 64K glyph limit is in OTL tables: to support 32-bit GIDs in GPOS, GSUB and GDEF would entail a very major change affecting many dozens of structures. That would require a large engineering investment for ipmlementers.

Not really. Everything is relative of course...

By far the biggest hurdle for breaking the 64K glyph limit is the broken process for contributing to OFF as monopolized by you, Peter while at Microsoft and continuing to do so. Until you apologize for your 2016 intimidating email (https://gist.github.com/behdad/d08f958f8a5e2cb6badf6e32598427df) and commit to addressing the problems you have created in this process, I don't see how we can make anything happen.

I wouldn't want to comment on other aspects/usages of 64k+ glyphs, but as far as CJK is concerned, while it is true that there are more than 64k Han ideographs, the typical educated person only use about 3k of that. e.g. newspapers can operate quite happily with just a specific set of ~4K glyphs. Many of them are rarely used historical forms etc. Few people complain about the simsun / simsun extB split, because few actually need or use the latter.

Perhaps some ideas involving split tables / sub - tables of up to 64k glyphs, perhaps by unicode ranges, would be useful and less disruptive?

I currently have four reasons to want to break the 64k limit, enumerated @ https://rsheeter.github.io/more_gids.

Also, on subsetting, I have always held the view that the Japanese people did it right - they sorted JIS encoding based on frequency of usage, and essentially have a contiguous 3k code blocks of "frequently used characters" and another contiguous 3k block of rarely used character.

The older Chinese encodings were essentially dictionary order (or by character "complexity"). It is like a kitchen sink. That have extremely rarely used characters in the middle, and that got inherited by the unicode encoding.

@HinTak Glyph IDs are a different issue than content encodings (Unicode or otherwise). Fonts provide a map between these two separate things and one of the jobs of a shaper is to lookup what glyph(s) to use for given codepoint(s) in the input. How the input is encoded is completely out of scope for this issue, and how Glyph IDs are organized is completely up to a given font and hence also irrelevant to this issue. The only point here is the theoretical cap on how many glyphs can be stuffed in a single font file.

@alerque encoding (and unfortunately kitchensink-style encoding...) matters, because a large part of argument for large glyph IDs is CJK/ Han usage. As I mentioned at the beginning, that's the part I am commenting on.

I think I remembered that the unicode CJK submission was based on, or often citing the Kangxi dictionary, an 18th-century encyclopedia of Chinese characters compiled by imperial china at its height. While it is a piece of indisputable scholarship, it is also a kitchen sink, in a way, like any encyclopedia, full of rare and outdated items.

encoding (and unfortunately kitchensink-style encoding...) matters, because a large part of argument for large glyph IDs is CJK/ Han usage.

@HinTak Encoding certainly is important... though it's not clear to me what bearing it has on the number of available glyph IDs. The ordering of code point positions for characters and the ordering of glyphs in a font are independent. So, for this topic,, it doesn't matter how CJK characters were assigned to code positions in Unicode or any other encoding. I agree with Caleb in this regard. What does matter, in terms of CJK characters is how many of them someone wants to support in a font.

@rsheeter

I currently have four reasons...

I see three scenarios. Your fourth point gives a mitigation (both relevant and valuable) to concerns about really large fonts and the Web or, potentially, devices with limited storage but connectivity.

Pan-Unicode fonts were of interest in the 1990s, but then fell out of interest. In part, that may have been because of file size (Arial Unicode MS is 22MB), but I think bigger issues were:

• Coercing all scripts in Unicode into a single design doesn't make good design sense.
• There's actually not much need in practice for pan-Unicode fonts.

Now, while those points are valid, "not much need" isn't the same as "no need". Font fallback that covers all Unicode characters is a valid and important need. A pan-Unicode font can certainly make sense for that purpose. And for UI/fallback purposes, uniformity in design makes sense.*

(* In a universe with unlimited resources, different fallback fonts with different designs to match different designs according to the primary UI language might be desirable in some situations.)

But fallback doesn't require a single, pan-Unicode font. You mentioned 150+ Noto fallback fonts. The current 64K glyph limit isn't causing there to be 150+ fonts. Rather, it's a design choice to have one script per font. The count could be reduced to a handful even with a 64K limit.

So, I think pan-Unicode fonts can be considered another potential use for 32-bit GIDs. But I'm not convinced they strongly add to any business case for 32-bit GIDs.

Encoding matters, when a font or a font vendor claims to be pan-unicode etc. Imagine a world where the CJK code blocks are called CJK level 1, 2, sup 1, sup 2, sup 3, sup 4 sup 5, (totalling to 21k) more like the Japanese JIS model of 3 k each, instead of a single block of 21k CJK unified ideograph, and another extension B taking up 43k.

If you only claim to support CJK level 1/2, the normal daily usage, there is a lot more room for other glyphs. The 15k extra in CJK unified ideographs over JIS level 1 and 2 is in a way, dead weight as they don't have shaping requirements for GPOS/GSUB and purely taking up code space.

I see what you're saying: If CJK characters were organized in "levels" based on frequencies or contexts of usage, that would allow having a font with smaller character (hence glyph) repertoires that would meet the needs for particular scenarios.

Ordering of characters in the encoding could make it easier to define sets of characters or to assign CJK characters into levels, but it doesn't enable or prevent it.

@PeterConstable yes, that's my idea. Unfortunately the CJK inclusion to unicode was less usage-driven than politics/marketing driven, and instead of having 4-6k glyphs sufficient for daily usage, plus extensions, we have a system where one have to carry 15k glyphs which nobody uses, to be able to claim to support basic CJK.

What are the technical requirements for having 32-bit glyph ids? I suppose many of the tables need to have new versions with expanded fields, and there needs to be software prototypes that support the expanded fields.

Perhaps asking a different question: how about finding a way of letting font vendors claiming some kind of "CJK daily usage" subset compliance? If one loses the dead weight of trying to cover 21k, but instead say, 6k, one gets a lot of id space back for other kind of maneuvers like shaping and color fonts. (I know this requires perhaps government co-operation, so probably not going to happen...)

ISO/IEC 10646 defines character collections including (among others)

• collection 370, IICORE: 9,810 characters
• collection 375, Japanese Core Kanji: 2136 characters

WRT to CJK we looked at frequency of character use over millions of pages when looking at shipping CJK on Google Fonts. The reality is you get pages with mostly high usage characters plus a few rarer ones. If you ship just the high frequency ones a lot of pages will have a few characters that break out of the font. This is a broken solution.

Pan-Unicode fonts were of interest in the 1990s, but then fell out of interest

I would dispute this; Noto is a pan-unicode font. It ships on Android, ChromeOS, and even Apple platforms. My team owns this for Google and actively wants to break the 16 bit gid limit for it.

So, I think pan-Unicode fonts can be considered another potential use for 32-bit GIDs. But I'm not convinced they strongly add to any business case for 32-bit GIDs.

To be clear, my team has a need to break this limit for the reasons I outlined. This isn't hypothetical, it's a real issue for real operating systems and use cases. Can we collapse to fewer files today? - Yes. Can we collapse as far as we'd like? - no. Are there workarounds? - Yes. Are they suboptimal? - also yes.

I find it confusing this would be dismissed as not adding to a business case.

ISO/IEC 10646 is fine and good if everybody subscribes to it... I am talking about the existence of GB18030, and a country which has a tendency of "doing its own thing" for years :-P .

To be clear, my team has a need to break this limit for the reasons I outlined. This isn't hypothetical, it's a real issue for real operating systems and use cases. Can we collapse to fewer files today? - Yes. Can we collapse as far as we'd like? - no. Are there workarounds? - Yes. Are they suboptimal? - also yes.

I find it confusing this would be dismissed as not adding to a business case.

I didn't dismiss it or say pan-unicode fonts don't add to a business case. I just said I'm not sure they add strongly to the business case because pan-unicode fonts are of much interest beyond fallback fonts and there are workarounds for fallback scenarios.

But it certainly adds something to the business case and deserves consideration.

I've added pan-unicode/fallback into my opening comment.

Btw, my opening comment for the issue was intended only to seed discussion, not as a draft business case statement.