> Quite long devlog coming up, apologies—I got a little carried away with this one!
mlugg, please don't apologize for creating something I actually want to read. I'm drowning in low effort garbage, the in depth technical explanation is a refreshing breath of fresh air.
Might as well apologize for creating a language without a garbage collector, sure most people are unwilling to think, but some of us like nice things and are actually willing to apply effort.
Why I've moved more to a couple of language/software dev discords and away from Hacker News. Way too much uninteresting AI nonsense on here for a while now.
oh, I think it's mostly frustration over how eager everyone is to delegate their thinking to literally anything else, accelerated by [gestures at reality]. Is frustration with apathy really pretentious?
This change + the existing packed struct logic will be great for working with bit packed binary headers w/o having to manually twiddle so much about the bit handling along the way.
It's so interesting to read comments like this and contrast them with the "don't read the code" type of vibes out right now. It feels like half of the developer world is optimizing low-level struct packing and the other half is YOLO'ing 300 KLOC Electron apps. Very confusing.
yeah, remember those newfangled fancy Node.js guys who would just copy/paste from Stackoverflow without any understanding?
Or the Java guys who wrote bloated apps that wasted CPU cycles on garbage collection instead of writing in C++, like God intended?
Or the Fortran / Cobol guys who wrote in those God-damned, wasteful, useless high-level languages, instead of using assembly, like a proper programmer should?
I think it makes sense, if one sees that LLMs exposed various pre-existing splits in the developer world.
Those who viewed code as a means to build something else, are happy to switch to LLMs if they can build that something faster/cheaper.
Whereas, those who liked coding for its own sake, don't want to use LLMs, and fear for their jobs and their happiness.
Unfortunately for the latter group, we're moving to a world where most development is done by LLMs, and only cutting-edge or hobbyist work is done manually. E.g., Japanese artisanal wood-working and joinery is beautiful and elegant... but modern carpentry doesn't build that way.
i think it's perfect: AI allows you to go incredibly deep (you have unlimited access to context to make incredibly impactful surgical changes), or you can go incredibly broad (you have unlimited access to context to tie a mind numbing amount of components together). what shakes out is the middle layer: "infra" between "algorithms" and "product".
though, to be fair, the middle layer itself is composed of this same work. so it's fractal, or turtles all the way down.
Zig is already great for this with ‘packed struct’ and arbitrary size ints. Allows for very clean protocol creation between systems with known properties. This is another great step in that direction.
you need different packed structs for little- and big-endian data. and casting with little-endian data is a nightmare - you need to reverse-cascade your struct fields to be in accordance with the little-endian bit-pattern. (or have a comptime function that does it for you, of course. but then you lose all declarations for the struct). what should be a simple writing down of a protocol is now a pedantic and error-prone ordeal.
Generally those edge cases are always the same endianness. You don't need big and little endianness versions of the structures. What's important is that everyone agrees on the same thing.
if someone chooses to do that they own the problems.
> network byte order isn't a thing
if the network serializes/deserializes for you (kernel primitives) then you don't care what it does. if it doesn't and for some reason you choose to use big endian, again, you own the problem.
Network byte order has nothing to do with the kernel and you have to care about it
It’s a standard because neither side of the connection knows the endianness of the other side so there must be a standard. That standard is big endian regardless of your architecture or kernel or anything else
So any serialization intended go over the network should be big endian
You may have never done socket programming, or do you use wrapper libs in Zig? Because you have to send the kernel big endian port numbers for example.
What do you do if you program a kernel in Zig, or just generally do low level networking?
My point is to refute the statement that everyone has agreed to little endian, and so there aren't use cases to want to do conversion. Programs do not exist in a vacuum, most programs do not.
Well you would, of course, have a mapping layer between wire types and domain types, like in any good codebase. You do the endianness conversion at that boundary, and then you can just send it out.
then you either use an existing C library (the most likely approach) or if you are determined to re-implement it you have to be careful parsing their bytes.
When I first found out about bit fields in C, I was left wondering what the order of bits was in a byte, eventually I convinced myself it doesn't matter, since the byte is the smallest I/O unit, and lived with the fact that casting between bitfields and bytes was UB (or unspecified, I can't remember), and as such, was another thing I wasn't suppossed to do when writing C.
All this to say that Zig just keeps cleaning up and giving well-defined semantics to warts I learned to live with in C.
Writing linkers must be incredibly rewarding - go has its own, there's mold, there's LLD, there's the OG GNU bfd LD and now Zig has one too! I am sure there's a Rust one too - Wild!
Every one of them is faster than the others too lol! Mold for one tries really hard to be GNU ld and to be useful as an independent linker most have to - I guess Zig/Go ones are purpose built so at least those don't duplicate GNU ld compatibility.
Sure, but one might imagine that linkers are generic and reusable, so you can just pick one off the shelf instead of making a new one 1-1 for each language. Empirically this line of reasoning seems to be incorrect.
Different programming languages are very obviously not the same thing - different cp command implementations are similar conceptually to having different linker implementations that all do the same thing. But you knew that so not sure if there was a point you were trying to make there.
> Different programming languages are very obviously not the same thing
It isn't obvious to me at all. The difference between Java, C# and Go is about as important as the difference between Makita and Bosch for power tools. Yes, some people swear by one or the other, but by the end of the day it really doesn't matter
Interesting read, even as someone who isn't using Zig.
I wonder, these arbitrary-width integers... Is it actually even really worth it? My intuition is to prefer manually packing/unpacking things instead (in any language, even C that has bit width for struct fields), because it gives me a better mental picture of the code that is actually generated. Particularly for something like an signed odd-bit integer - what kind of code gets generated for sign-extension, a presumably common operation?
Does anybody have other experiences with them, one way or the other?
IIRC, for "normal" bit widths the codegen basically uses the next larger machine type and preserves zero bits on the high end. An i3 is an i8 with five MSB zeroes (with more custom behavior for "packed" i3 values). It's UB to fill those with non-zero values. For larger bit widths, like u729, you concatenate many large machine types, the compiler generates instructions in an unrolled loop, and the LLVM optimization pass usually doesn't clean that up (though, now that integers are apparently not using the LLVM u729 implementation, perhaps there are some more optimization opportunities).
They're situationally useful, especially when performance isn't an enormous concern. That u729 example above came from a variant sudoku solver I wrote to aid developing new puzzles (easy to check the rough magnitude of the solution space for whatever idea I was mulling over and examine how restricted the board actually was -- just an intermediate step in puzzle design). It's not optimal (hard on the icache, can be hard on registers, other issues abound), but it's dead simple to use, and the assembly isn't terrible, beating all the normal solvers I saw floating around. It's a nice point on the laziness/correctness/good-enough-perf pareto curve.
Another comment mentioned this, but they're great in packed structs for representing weird numeric entities (I think I have a logarithmic number system floating around which does that).
One thing the language does quite a lot is use them to guard against certain classes of human error at compile time. It doesn't perfectly make impossible actions unrepresentable, but shoving a full u32 into a shift argument usually doesn't make sense, so the types are constrained to be smaller.
I can't imagine any situation where I'd use a u729 instead of a StaticBitSet. For size 729, it would end up backed by a bit_set.Array, not a bit_set.Integer.
I don't program zig, so it's not clear to me if you can use zig's bitsets arithmetically.
Sometimes it's just more clear to work with integers than other representations. Most situations with a state space of N bits have meaningful integer representations, where arithmetic functions on those representations are also meaningful.
For example, CRCs can be written as the remainder from long division of the message by the polynomial. Defining nontrivial cyclic permutations is also much more straightforward as functions on integers than on bitsets.
I was talking about GP's u729, which is 9*9*9, the state space of a sudoku board. Can you come up with a situation where dividing that number by anything is meaningful?
If I had to steel-man the idea, I'm pretty sure the integer-based solution has better codegen with many kinds of sparse, comptime-known masks. I think you're right though, StaticBitSet looks better.
For your specific case, even a simple `[9][9]u16` might perform better (where you make use of nine bits in each u16). For each entry, the nine mask bits would be in the same bit positions, so the compiler won't have to do a bunch of shifts to extract/align the bits. CPUs love consistency. I doubt it's worth the additional codegen complexity to save 70 bytes in your data model.
My initial implementation used [9][9]u9 (which desugars to [9][9]u16 with some zero bits) and was a fair bit slower. If I had to guess, it's because the shift/extract/align you're describing isn't actually a part of the core solving algorithm, and when you have box constraints, knights-move constraints, etc, you're usually not doing anything which fits in a single u16.
It's pretty great in my toy emulator project (https://github.com/floooh/chipz) as 'system bus' where each bit is a 'wire' which is then mapped to chip input/output pins.
The bus-width is a generic parameter and can be below or above 64 bits (depending on the emulated system). With arbitrary-width integers the high level code remains the same no matter what the bus-width is, and from looking at the compiler output, as long as bit operations don't straddle the underlying 64-bit integer boundary, those bit operations are just as efficient as working on a simple 64-bit int.
Also AFAIK LLVM supports arbitrary-width integers since pretty much forever, Zig just 'exposed' them in the language (as later did Clang via _ExtInt(N), which is now deprecated in favour of C23's _BitInt(N)).
The other nice usage (also in emulators) is for chip registers and counters, those often have odd widths (like 5 bits), and writing those as u5 instead of u8 in the code is just nicer since it matches the chip documentation, and when reading the code it's immediately clear that this u5 is a 5-bit counter or register.
I love it. Easily one of my favorite things about the language.
Example: shifting more than the width of the shifted integer is illegal behavior in Zig: therefore, the, what, shiftand? let's go with that, the shiftand for a u64 must be a u6 or smaller.
As an fpga engineer dealing with bitwidths that are non-byte multiples is very normal and when I end up writing software for various reasons, I often miss it. Usually when trying to slice and parse or construct messages.
Obviously there are ways around pretty much everything, but it’s nice to have first class language support for bit slices.
except it isn't bit slice, it isn't indexing within a range - it's just integer type that only allows values up to 2^width, with same alignment rounding up as with the rest
It's a bit slice if you put it in a packed struct.
I like them, they're nicer than C's bitfields: The order isn't implementation-defined, and the types remember their range rather than being converted to a power-of-two size upon read. (Maybe that's possible with C23 _BitInt(n), I haven't tried if those work in bitfields)
IMO they're fantastic. You can write out a bit layout from a CPU's manual fro example and you can just use whatever bit width the manual specifies, and the compiler takes care of figuring out all the underlying manipulation for you. Which results in much more readable code because you don't have to worry about packing/unpacking it because the compiler will do that for you.
OT: I'm always surprised at how popular Zig discussions get here, or Youtube and other medias.
Don't get me wrong, I love Zig and I think it's a great C replacement, but I'm very confused on why C3 or Odin rarely get any attention at all, despite being in the same C-replacement crowd.
But still surprised at what Zig does better than these other projects? Is Andrew much better at marketing/promoting the language? He's very hard to dislike.
I can only answer for me, and while I do think it's more significant a metric for me, I equally assume it probably has some influence on others as well.
C3 uses :: for namespaces, that makes it a competitor with C++ more than C. Equally Odin's syntax is more at home among python, not systems programming.
The appeal of Zig is it feels like C. To many people, this is a downside. C is very very scary to them. But for people who feel at home in C, it's not a downside.
Additionally, the selling point for both are "c replacement" where the selling point of Zig is "good systems programming language" C is only mentioned by it's users as a heuristic.
If 2 groups are trying to replace a language that people are running away from, and that's their best selling point... I'd assume they're less likely to be as successful as a different language just trying to be as good as it can be.
I've even stopped comparing Zig to C, IMO, it does a disservice to both. And I say that as someone who likes C.
Full disclosure, I need to spend a bit more time with both odin and c3 to know exactly how this compares. But the reason I keep writing Zig, and still love it, is how simple it is. Zig is aggressively insistant on simplicity at the expense of functionally or comfort. The only other high level language I know of that is as aggressive about it's design simplicity is infact C. While I assume it's an accident when C does it, it's definitely not an accident in Zig.
I think Andrew is a big part of it, and the people he surrounded himself with are the other part.
What kind of pre-1.0 language hosts conventions? Crazy that they manage to do that.
Andrew's vision has always been clear and inspiring to me. I think this got Zig its initial following, and they have capitalized extremely well on it to grow as a community.
Andrew doesn't strike me as someone who does any marketing at all. He just wants to make the language he wants to use, and does it well.
Sometimes its just right time, right place. But also, Zig has received attention via projects like Ghostty, TigerBeetle, and Bun (prior to rewrite of course)
I believe I read a post by Andrew detailing how he intentionally did marketting in a way to attract users, the right contributers, and donations - he was quite intentional about making his full-time role sustainable (and now more roles).
They have definitely done a lot of marketing through social media and forums like HN. There have been large numbers of posts here by Zig's developers for years, and a few releases of LLVM even mentioned Zig prominently in their release notes.
There is a mountain of code written in C that you can simply include in Zig without a wrapper dependency and without having to create the wrapper yourself.
> Don't get me wrong, I love Zig and I think it's a great C replacement, but I'm very confused on why C3 or Odin rarely get any attention at all, despite being in the same C-replacement crowd.
Doesn't matter as neither will see significant adoption.
Zig has a really great backwards compatibility story with C, and it also is a better C compiler even if you don't write a single line of Zig. It's not hard to see why that is popular.
I’ve followed Zig fairly closely and this is the first I’ve heard of Andrew pushing “social issues”. I don’t believe for a second that it’s a factor at all.
Yeah, nah. Not so sure about that. I love zig, and I appreciate the rigour, care and thought that goes into the language and it's libs. What Andrew Kelley and the team are doing is excellent work, creating a useful, simple language with which to write efficient, correct programs.
His politics don't matter to me. Hell, if the politics of technologists dictated whether I used their products, I'd have to go live in the wilds, without any tech. :-)
How confident are you? I ask because I'm a zig zealot, and am constantly shilling for it. But I disagree with a number of ark's positions, and think of him as a bit of a shitter... So I don't think "cult of personality" accounts for it, despite how easy it would be for someone to be able disregard zig if was just a personality cult.
I have no issues w/ the zig language and I'm not saying that's the only reason why people talk about it.
There is however a subset of very vocal people who will go out of there way to bring stuff up and push something if they do see that is a part of it. Not that it's the only reason why either, just additional motivation for people to go out and push it that otherwise you might not get. All you need is a couple people who view that as a kind of campaign and they can radically increase the visibility of something on the internet, and turning programming that has some broader social or moral thing related to it even just through the creator is a very easy way of doing that. Rust has a similar thing.
I don't view the instinct that leads to language zealotry or zealotry related to social issues(or say religion) being that distinct and it's probably a similar personality trait that encourages both, and it's generally one I find unpleasant regardless of the particular content. FP can also lean in that direction. If you get some narrative you can say this language fixes stuff in a fundamental way + also can appeal to the social thing it just riles people up who will go around talking about it online non stop.
> Consider, for instance, bitcasting a [2]u8 to a u16. Under the old semantics, the result of this operation depends on the target endian: on big-endian targets, the first array element became the 8 most significant bits, whereas on little-endian targets, the first array element became the 8 least significant bits. Under the new semantics, because we only care about logical bit representation (which is endian-agnostic), the operation behaves identically on every target:
This is a huge mistake. You would never expect something like bitCast to do this.
I don't understand this approach. Why change something so simple and low level to be complicated and high level?
Just don't allow casting to u24, as it makes no sense unless you define u24 to be u32 sized as I think c standard does.
I think this approach as an idea is bad but at least just add another built-in that implements this higher level idea to not break a simple expectation and current behavior?
> Just don't allow casting to u24, as it makes no sense unless you define u24 to be u32 sized as I think c standard does.
The reason u32->u24 casting must be well defined is because some hardware (e.g. many GPUs, microcontrollers) only have floating point multipliers. A 24 bit unsigned integer (stored in a 32 bit register) can be losslessly converted to a 32 bit float by the hardware, multiplied, then converted back.
This is much faster than doing 32 bit multiplication in software, however, you still need to tell the compiler about this constraint.
I am criticizing the part where they allowed [3]u8 to u24 bitCast in the first place. It doesn't make sense logically as u24 is likely not 24 bits in any targets let alone portably on every target.
Interpreting u24 like it is actually 24 bits sounds like programming in crazy land since it is not 24 bits in any relevant architecture afaik.
They didn't allow []u24 with a similar rationale as far as I can remember. I agree with this as someone programming at this level should be able to understand there is no real u24 layout and they should use []u32. Going with the same magical rational they went with here, compiler should generate unaligned u24 loading code when you use []u24 since it is "logically 24 bits"
The ease of dealing with arbitrary bit-width integers and packed structs is actually one of the 'killer features' for me in zig.
Zig natively supports arbitrary bit-width integers, the ABI is defined and you could simply think it as a slice of the next larger backing integer.
The[3]u8 to u24 bitCast will simply be backed by a 32bit int, using the same ABI. As you have u1 - u65535, sometimes it can be multiple words.
The 24 Bits (3 Bytes) [3]u8 to u24 example is exactly related to utf-8 that covers all the languages but excludes the emojis.
There are very valid use cases when you want to limit utf-8 to U+0000-U+FFFF, and it is valuable if your language allows you to make those decisions.
Remember, in zig packed structs are just integers and integers are just a group of logically consecutive bits.
Arrays like []u24 do not have the same ABI, arrays are not bit/byte packed, are not universally LSB across archs etc..
The compiler isn't producing unaligned code, don't confuse the abstraction with the concrete implementation. And yes [8]u1 and [8]u8 are exactly the same size and shape, even though they are arrays.
My current project is parsing ELF/Macho files, I can easily have zero allocations in my hot path with zig, the same is far more challenging in C, so I am biased, especially with zig allowing methods on structs.
And yes, I do use that crazy casting to 0xdeadbeef and other ascii metadata that is in those files.
To be clear here, I am not trying to prove you wrong, this is one of the places zig is very different and (IMHO) useful. Especially with streaming data or where you have network ordering etc... It is so nice to only cast what you need to but it does take a little while to wrap your head around how this interacts with buffers which are not your native endianness. At least for me, once I figured out to separate the shape of those data streams from their values it was super useful.
> The 24 Bits (3 Bytes) [3]u8 to u24 example is exactly related to utf-8 that covers all the languages but excludes the emojis.
I'm not familiar with Zig, so maybe it's doing something weird here, but that doesn't really make sense with Unicode in general.
First, the largest Unicode codepoint that will ever be allocated is U+10FFFF [0], which is less than 2^21, so all Unicode characters will fit in a 24-bit integer. Perhaps you're thinking of UCS-2 or UTF-16 without surrogates, which are both 16 bits wide and are limited to the BMP [1] [2] (and therefore don't include most emojis).
Second, while the characters needed for most languages lie within the BMP, not all of them do [3], so it isn't really possible to support all languages while excluding emoji, aside from using the Unicode character database to exclude certain categories [4] [5].
Note the utf-8[0] in my response, the answers are on the pages you linked, but not in the sections you linked,
utf-8 encodes code points in one to four bytes, it is byte oriented vs utf-16 etc.
In zig u8 is a byte, and is also (by convention) a char, although there isn't an explicit char type in zig. Technically there are chars in languages that need all 4 bytes in utf-8, but almost all of them are historical or emoji's in utf-8.
24bits (3 bytes) in utf-8 gets you Chinese, Japanese, Korean. 16 bits (2 bytes) gets you Latin letters with diacritics, Greek, and Arabic scripts. With 8 bits (1 byte) getting you Standard ASCII etc...
There is a point you could make that it may have been better to use utf-16 etc... and that we should have dropped ascii/latin-1 support, but once again go up to the 'Basic Multilingual Plane' in your [3] and notice that is covered by 24bits (3 bytes) in utf-8 encoding.
> ... but almost all of them are historical or emoji's in utf-8.
I just posted a comment, five minutes after you wrote that, which I won't repeat here since it was quite long. But one of the languages whose alphabet is found in the higher multilingual plane is Fulani, spoken natively by 37 million people (plus another two and a half million who have learned it as a second language). While it can be written in other alphabets (both Latin and Arabic have been used to write it in the past, for example), other alphabets don't usually represent all the sounds of the language properly, making it awkward. There's a reason why the Adlam script was invented to write Fulani with; and that invention was recent enough that it was assigned the U+1E900 to U+1E95F block, since the basic multilingual plane was full by then.
So although it's easy to think that the astral planes are only used for emoji and historical languages, that's not actually true. There are languages spoken by millions of people in those astral planes as well (yes, languages plural; Fulani isn't the only one, it's just the largest).
To be clear, I was talking about a use case, not all use cases.
There are very real times where you have to support all 4 bytes, there are others where other drivers require you to restrict the domain of discorse.
It doesn't change the value/cost of bit casting in a language with arbitrary bit width languages, especially when combined with the fact that int overflows are detectable illegal behaviour and you have saturating and wrapping operators.
This is in addition to the ease of using packed structs I mentioned above.
A list of some advantages:
* Zig's arbitrary-sized integers have a fully defined ABI for padding
* Allows for strict domain modeling using them as platform independent refinement types
* Precise memory packing, allowing more utilization of register space etc...
* OOB compile time checks
* Bit masking optimization, where sequential changes to packed values are often merged into a small number of and/or masks
To move to a more information theory example:
DNA nucleotides (A, C, G, T) represents quaternary state pairs.
If you wanted to store an array of 1,000 DNA nucleotides, each symbol is one of 4 bases, requiring exactly 2 bits of information. The Shannon Information would be: 1000 * 2bits = 2000 bits.
With uint8_t this would take 8k bits, vs 2k bits of u2. That is 300% more for uint8_t.
It is still horses for courses, but as an example consider 12-bit sensor reading in a standard u16, the data type allows invalid states. To ensure safety, requires manual defensive logic throughout your program in the traditional C/Rust/...
That traditional model in zig:
fn processSensor(value: u16) !void {
if (value > 4095) return error.InvalidSensorData; // Extra logic branch
// ... logic ...
And the lower overall Kolmogorov complexity (cherry picked) form:
fn processSensor(value: u12) void {
// Zero validation boilerplate code required here
C23 does have _BitInt types for structs which can help if bit packing is your primary need, IMHO it doesn't offer the same advantages.
As an example, and I may be wrong, but I think you cant easily perform checked arithmetic or use standard overflow operations on individual C bit-fields without copying them out into standard standard types (like int), modifying them, masking them, and copying them back.
With Zig the invariant is maintained implicitly at the type layer, removing runtime validation branches, error paths, and testing code
Does it solve all problems, no. Is @bitCast, a zero runtime overhead, compile-time checked bit reinterpretation and [3]u8 \to u24 useless and silly, no.
Yes, there are certainly use cases where you know the data you're parsing will only come from a narrow range of Unicode, such as U+0000 to U+007F — or from just the letters GCAT, as you mentioned. The overhead of converting 8-bit input to 7-bit might not be worth the cost, but the benefit of storing your input in just 2 bits per "letter" is definitely worth it.
I mostly wanted to make sure people know that the upper multilingual planes are a very real use case, and you need to test them. This is more important for languages such as C# where UTF-16 is the norm: many programmers don't know that they're handling surrogate pairs wrong until someone tries to backspace over an emoji character and it turns into something weird. It's probably less relevant to Zig, which didn't make the mistake that C# and Java did by starting out with UCS-2 (to be fair to them, they were designed in the era where people still thought that 65,536 codepoints would be enough for every language and Unicode would never need more than 16 bits). But the upper planes are important, and need to be tested no matter what language your code is written in.
> utf-8 encodes code points in one to four bytes, it is byte oriented vs utf-16 etc. In zig u8 is a byte, and is also (by convention) a char, although there isn't an explicit char type in zig. […]
> 24bits (3 bytes) in utf-8 gets you Chinese, Japanese, Korean. 16 bits (2 bytes) gets you Latin letters with diacritics, Greek, and Arabic scripts. With 8 bits (1 byte) getting you Standard ASCII etc...
Ah ok, so if I understand you correctly, you're taking a variable-length encoding (UTF-8), and limiting and/or padding it to 3 octets (24 bits)? In that case, what you said in your original post makes sense, but I'm not really sure why you'd ever want to encode something this way: you have to deal with the complexities of a variable-length encoding to parse each u24, you have the poor space usage of a fixed-length encoding, and you're using 24 bits to encode only 0xFFFF characters (even though you can fit all of Unicode in only 21 bits).
> Technically there are chars in languages that need all 4 bytes in utf-8, but almost all of them are historical or emoji's in utf-8.
Yes, the majority of the characters in the non-BMP planes are for archaic languages, but that's not really the right way to look at it, since most languages only need <100 characters, and there are more dead languages than living ones. Instead, I'd look at it from the reverse lens of how many living languages need non-BMP characters. This sibling comment [0] gives one example, but there are lots more [1] [2] [3] [4] [5] [6].
Now, it's fine to not support these characters, but the argument in that case should be that you've decided that the characters aren't important enough to outweigh the technical challenges, not that nobody needs the characters.
> 24bits (3 bytes) in utf-8 gets you Chinese, Japanese, Korean.
It gets you a subset of CJK that's probably sufficient for many purposes, but there are nearly 75k CJK characters outside of the BMP.
> There is a point you could make that it may have been better to use utf-16 etc... and that we should have dropped ascii/latin-1 support, but once again go up to the 'Basic Multilingual Plane' in your [3] and notice that is covered by 24bits (3 bytes) in utf-8 encoding.
If you are willing and able to use a 24-bit encoding, then I'd argue that you should just use UCS-3/UTF-24, since those allow you to encode every Unicode character. The only downside is that these encodings aren't formally-defined so other programs won't understand them, but if that's an issue you can use UCS-4/UTF-32.
> ... utf-8 that covers all the languages but excludes the emojis ...
Ah, but the U+0000 to U+FFFF plane does not cover all the languages. You might think that only historical and archaic languages are found in Unicode's astral planes (e.g., U+20000 to U+2A6DF is used for historical Chinese characters no longer used today), but in fact there are modern languages found in the U+10000 plane.
You might not care about Osage (the language of the Osage Nation of northern Oklahoma) since its last native speaker passed away in 2005, but there is a revival program trying to teach Osage to people. Osage's script was developed quite recently as part of the revival program, so it couldn't fit into the U+0000 to U+FFFF block and it was assigned U+104B0 to U+104FF.
The Toto language of Bengal, on the other hand, is still active: over 1000 speakers, all living in the village of Totopara. It also never had an alphabet until recently, so its Unicode block is U+1E290 to U+1E2BF.
Then there's Wancho, spoken by about 60,000 people in India. Its alphabet was created between 2001 and 2012, and added to Unicode in 2019. It was assigned the U+1E2C0 to U+1E2FF block (immmediately after the Toto language, you might notice).
Then there's the Ho language spoken by over a million people in India. Wikipedia cites a 2001 census as having 2.2 million speakers, and a 2011 census as having 1.4 million speakers. I very much doubt that both of those are accurate (you don't lose half a million people from an ethnic group in just ten years without some kind of war or genocide, and the Wikipedia article would have at least mentioned that if such a thing had happened), but to be safe, let's go with the lower estimate and say that at least one and a half million people speak Ho. It can be written with the Latin alphabet, but its own alphabet is Warang Chiti (sometimes spelled Warang Citi), which was added to Unicode in 2014 and assigned the U+118A0 to U+118FF block.
And then there's the Adlam script for writing Fulani, the language of the Fufulde people of western Africa. Fulani is spoken natively by 37 million people, and as a second language by another 2.7 million. Adlam's Unicode block is U+1E900 to 1+1E95F.
So if you restrict your program to only working with the basic multilingual plane, it's not just emoji you'll be leaving out. It's also modern languages, spoken by anywhere from 1000 people to 37 million. How many speakers of a language are enough to draw the line and say "No, I won't ever translate my software into your language"?
Now, if your software is only targeting one language and you never intend to translate it, then yes, you'll only lose out on emoji if you stick to the U+0000 to U+FFFF range of the basic multilingual plane.
But realize that the higher planes are not just for dead languages. Living languages have ended up there too, and there are likely to be more in the future. It's quite possible that right now, someone somewhere is saying "Hey, why doesn't my language have its own alphabet instead of using Latin characters to write it? The Latin characters don't express the sounds of my language very well." And when they do get that alphabet worked out and manage to get it accepted into Unicode, it'll certainly land in one of the higher planes. Most likely the U+10000 to U+1FFFF plane which isn't at all full yet, but who knows. If you want to be able to handle every language spoken (and written) in the world today, you must be able to accept the full range of Unicode, not just the 16-bit range.
> You don't lose half a million people from an ethnic group in just ten years without some kind of war or genocide.
Nothing happened to the people, they are growing year on year. But languages can die very easily if governments don't put efforts on teaching it to children. That is exactly what happened to the Ho language. There is no advantage on learning these small regional languages so children put their effort on more popular languages like Hindi, Odia and English.
I'm familiar with the phenomenon, as my wife is a linguist who did her master's thesis on the phonology of a small language spoken by about 7000 people: many of the kids don't want to learn it, and just want to learn the majority language of the country since that's what they have to use in school. But I didn't think that could be the explanation for a 25% decline in ten years: new people may not be learning the language, but the only way people stop speaking their mother tongue is if they immigrate to a new country and fully adapt to it (happens to a few people, usually who immigrated as children) or if they die (by far the most common reason for language-use decline: the old people are dying and the young people aren't learning it). If the decline was a couple hundred thousand that would be the outside limit of probability, as far as I know.
More likely, in my opinion, is that both are happening: yes, the language is declining, but either the earlier census overcounted speakers (e.g. counting children as speaking it when they weren't actually learning it) or else the later census undercounted speakers; either way the language decline would look larger than it actually is. Given that Ethnologue (https://www.ethnologue.com/language/hoc/) rates the language vitality as "Stable" — "The language is not being sustained by formal institutions, but it is still the norm in the home and community that all children learn and use the language" — and they usually know what they're talking about, I suspect the language decline isn't that fast and a census counting mistake is a more likely explanation for the discrepancy over ten years.
>,Multiply two unsigned 24-bit integer inputs and store the result as an unsigned 32-bit integer into a vector
register.
D0.u32 = 32'U(S0.u24) * 32'U(S1.u24)
> Notes
> This opcode is expected to be as efficient as basic single-precision opcodes since it utilizes the single-precision
floating point multiplier. See also V_MUL_HI_U32_U24.
Nvidia GPUs used to do the same thing and theres a umul24 intrinsic if you care to use it.
This is super-super-niche since it basically only applies to 32-bit integer multiplication.
You likely won't run into it unless you're doing high performance embedded systems or GPU programming on non-NVDIA cards, and for some unknowable reason, your workload does a 32-bit integer multiplication in the hot path.
That's literally only for 32bx24b (I don't remember why we did that specifically for CDNA - I'll ask someone) but as you see from V_MUL_HI_I32, V_MUL_LO_U32 there is very much vector arithmetic hardware (nevermind that we're not talking about VALU but conventional scalar ALU).
I think he has a point, but I am still not 100% convinced by the arguments relating to casting.
There is a difference between a u24 data type inside u32 and a u24 datatype inside u24 and that is what's so frustrating here. u24 is an alignment nightmare so it will basically never exist as "u24 in u24" and only ever as "u24 in u32".
For casting to make sense, the alignment must be compatible and it's not clear how you can simultaneously make arbitrary bit data types simultaneously useful for the scenario of describing bit fields in packets, where padding is inherently undesirable and performing integer arithmetic with an FPU, where padding is an acceptable cost for alignment. These appear to be mutually exclusive use cases.
While the GP might be technically wrong in a narrow sense, GPUs are built for FP, and that's what you want to be doing if you're using them as accelerators.
You don't know what you're talking about: an enormous amount of TOPs now runs through quantized (read: integer) kernels. Many GPUs don't have even FP64 or even FP32 support.
> The quantized integer kernels aren't running true integer multiplication, the quantization is it's own thing, they're basically enums not integers
ELI-a-GPU-compiler-engineer-working-at-a-major-vendor (because I am). Ie I can pull up the design docs for our ALUs and literally see that you're wrong.
I understand the reaction, but I don't agree. I suggest reading the associated proposal[0] along with the devlog, and having a real think about what's going on here. I'm responding to you saying that you "don't understand" the approach: reasonable, and resembles my initial reaction.
I was inclined to agree with you, but what decided it for me is that Zig has another mechanism for "reinterpret bytes". It's exposed on the stdlib as std.mem.asBytes, but this is literally a wrapper for the following:
@ptrCast(@alignCast(ptr));
So nothing is lost here: if you need, for whatever reason (and those do exist), to get a raw array of underlying bytes, you absolutely may. Std.mem also has bytesToValue(T, bytes) T, which makes a copy. All the ingredients are there, and this family of mem functions are thin wrappers over builtins, which boil down to pointer casting, dereferencing, and comptime magic.
Also worth noting: packed structs in Zig are already defined as logically little-endian: the first field is of low significance, the second is above that, and so on. So this makes `@bitCast` consistent with an existing convention of treating integers as logically little-ended, without regard to how they're actually arrayed in memory.
Plus it stands to make low-level bit-twiddling, using oddly-sized integers, optimize better. I like that, especially when what we trade for that is: nothing. Nothing at all, this is a pure win.
I'd even guess it's that rare language update which silently fixes buggy code, where someone figured "well, basically everything is little-endian already" (or just didn't think about it), and now that code works properly on big-endian machines.
GCC has had __int24 for the AVR backend for some time. Useful for larger integers than int16_t while saving 25% over a 32-bit value. C23 does not mandate padding for _BitInt types. It is wrong to assume that will happen or is the optimal implementation for portable code.
Thanks for the context, but what I am criticising is this part:
> it became allowed to use @bitCast to reinterpret a [3]u8 as a u24
This cant't make sense unless u24 is defined to be 24bits in the first place. It is just silly to allow something like this. It would make so much more sense to me if they started disallowing this or just even print a deprecation notice for it for one release version.
> Useful for larger integers than int16_t while saving 25% over a 32-bit value
You can't even do []u24 in zig as far as I can remember and understand anyway so this is only happening in a packed struct context.
C doesn't mandate padding but C compilers allow having pointers and arrays of irregular _BitInt types as far as I can understand.
In this [1] document, in Abi considerations section, it writes that it is defined to have next-power-of-two layout size.
Also here (for RISCV) [2] it seems like it is defined with next-power-of-two layout.
Also the document here (for x86_64) defines it similarly [3]
> This cant't make sense unless u24 is defined to be 24bits in the first place
It's worth remembering that zig is a ~hll that should be platform agnostic. suppose someone built a byte-chip with a 24 bit word. the "new" zig way of doing things will be more portable and slot right in, and support 32- and 16-bit datatypes just fine.
I sort of agree ... bit casting from an N width integer into an array of ... woah ... that's too far. It's bitcast not byte-cast which has an implied reinterpretation on a same or smaller word size in the cpu.
Once you see that the fact somebody has a u24 in their code is between them and the compiler alone.
As others probably noted byte casting (keeping the same endianess) is what unions are for.
To me it makes sense. If you don't know what endianness is, it doesn't make sense that a program you write in one programming language works for one target but doesn't work for the other.
I think endianness is the footgun that Zig is solving, rather than Zig being the one introducing a footgun when you deal with endianness.
It is not feasible for someone to write endian portable code in a language like Zig without understanding what endianness is imo. Regardless of how they change @bitCast there will be other cases that break this like doing @ptrCast + @memcpy.
Also this breaks currently written code that is endian portable and uses @byteSwap like it is done in most other programming languages that do these things.
> This is a huge mistake. You would never expect something like bitCast to do this.
Is there at least some sort of @transmute or something ? If Zig wants to say "bitCast" means this odd operation, but provides the thing most people actually want under some plausible name that's just an extra thing to learn which seems OK.
So, since I don't write Zig I had to go look this up, to save anyone else the bother this is what Rust would call an 'as' cast or C programmers might think of as a value cast, it's going to try to make a value which has a similar meaning but of another type, which may be arbitrarily expensive. What people often want here is a transmute, Rust's core::mem::transmute which changes nothing about the bits except what those bits mean, since the bits didn't change and the machine only has bits anyway this is "free".
For @ptrCast I also now need to care what the language's pointer provenance model is, and AFAICT the answer is Zig didn't get to that yet. If there's some kind of @transmute then we don't need to explain why the pointer cast worked because we never wrote one so that ducks the question, which is simpler.
If I understand it correctly, it basically boils down to copying bits from the source to the destination, in order from the least significant bit to the most significant bit. It's not equivalent to C++'s reinterpret_cast.
I'm no Zig expert, but if you want endian-dependent semantics I'd assume either @ptrCast or a packed union would do the job.
zig does not allow arrays in packed structs/unions specifically for endianness reasons (there may be other reasons as well but endianness is what i know of)
Ah, that is useful to know. Is that documented somewhere? From what I can quickly find in the obvious place [0], the only requirement is that "all fields in a packed union must have the same @bitSizeOf" and [2]u8 does satisfy that requirement.
no, but the documentation for packed structs gives the list of allowed field types. it's also not documented that packed union fields must be valid packed struct fields but people may be able to assume that
My understanding is that the "logical bits" view breaks down for unions, because the nth logical bit could be at different offsets depending on the union variant that's considered active.
I may be damaged from working on IC hardware design and various weird architectures, but I truly can’t comprehend why you’d think this doesn’t make sense.
Yeah, if your architecture doesn’t support 24-bit int it maps to 32-bits. But it also declares that the numbers you’re storing should never be larger than 2^24. It’s about type safety, and also run time checks in safe mode I believe. Bitcasting three bytes to a 24-bit type makes just as much sanse as casting 4 bytes to 32-bit. Theres zero reasons to introduce arbitrary artificial constraints on what you can do based on details of (most of) the underlying architectures, which doesn’t even matter for the operation you’re performing.
If the architecture supports 3 byte types that means it needs to support 3 byte alignments and their powers 9, 27, 81, etc. The easiest way to support this is to always map every 3-byte read operation to two 2-byte reads and then use multiplexers to recombine it into a 24 bit data type.
Of course you could also go crazy and store data in 24 bit blocks in your SRAM. That kind of ruins the 8 bit and 16 bit reads though.
You could use it to define a function that implements bitCast. Which defeats the purpose of having any @bitCast intrinsic instead of using @mempcy for everything
Take the address and deref afterwards, and it's exactly the same. Or to say another way: if you want bits to be reinterpreted raw as if they're in memory, then... put them in memory, then reinterpret them.
> You could use it to define a function that implements bitCast. Which defeats the purpose of having any @bitCast intrinsic
Yes, and this is one reason @bitCast was changed to have different semantics that are not trivially achieved with @ptrCast.
> Take the address and deref afterwards, and it's exactly the same.
It is significantly worse to take address and deref afterwards.
You have to do something like:
@as(const u32, @ptrCast(&x)).
instead of just
@bitCast(x)
> Yes, and this is one reason @bitCast was changed to have different semantics that are not trivially achieved with @ptrCast.
This makes sense except breaking existing code that properly handled endianness by doing a conditional @byteSwap. And what you end up with is a more complicated intrinsic compared to something that reinterprets values with same layout size
Your example is incorrect. @ptrCast has the same (similar, if you want to be pedantic to the exclusion of good faith) result rules. If you need @as to @ptrcast, you'd need it to @bitCast as well.
> It is significantly worse to take address and deref afterwards.
How are you measuring worse? Because my understanding from the article is that's exactly the behavior @bitCast used to have. So, instead of worse, it'd be exactly the same?
If you mean it's simply more things that you have to type... You're describing a core language feature as "worse". For all the builtins, some of them can help the compiler emit better code, but can for some doesn't mean will for all. As an example
Could zig auto convert between these types? Yes, absolutely. But it doesn't as a design decision. On some arch, converting between float and int can be very expensive. A competent engineer will ensure they're type converting in a reasonable order. Zig requires this painfully verbose syntax it order to make it painful. Are there times where it's is actually the only reasonable option? yes, but even if there wasn't it'd still need to exist because I'm not rewriting my whole program to avoid a single float conversion. But because it's a bit painful, I will rewrite this one function to make it less painful.
And, yes having already made that exact mistake... I now write better code from the start because there's no way I'm gonna ruin all my beautiful code with a bunch of ugly, annoying, hard to read, casts.
I used to complain about unused variable errors, unhandled enum branch, var unmodified (hint: use const) errors, hell even result ignored or error ignored when I'm trying to test some unrelated single line of code. But now that I'm used to them, I emit better code without thinking. It's made me a better programmer. Is it annoying? abso-fucking-lutely but I'm better now than I used to be, so: worth it; and: thankyou sir can I have another. :D
> As a general rule, the new semantics tend to match the behavior of the old semantics on little-endian targets.
They've basically said that bit casting is going to be little endian. This simplifies things for the 100% of people that are on little endian machines, while making the code still work for the 0% of people (rounded to the nearest 0.0000001%) that are using big endian machines.
FTA: “Under the new semantics, because we only care about logical bit representation (which is endian-agnostic), the operation behaves identically on every target: the first array element becomes the 8 least significant bits”
I wouldn’t call that endian-agnostic. It’s explicitly picking little-endian.
It also makes things look weird for beginners. I know how it works, but in the
test "bitcast [2]u3 to @Vector(3, u2)"
example, turning two 3-bit values [abc def] into three 2-bit values [bc fa de] is way less intuitive than turning it into [ab cd ef].
`u3` would be base 8, i.e. octal---I think you meant to use `[400]u6`?
Aside from that: I'm not familiar with how standard base64 deals with endianness, so I'm not sure if it would match that, but this `@bitCast` would certainly give you a base64 encoding. But it would probably emit pretty terrible code to do that---our lowering of `@bitCast` isn't really optimized for moving around huge amounts of data in one operation! (But maybe LLVM would surprise me.)
Macintoshes have had mnemonic keyboard shortcuts for inserting en- and em-dashes since forever: option-hyphen and option-shift-hyphen. They've been in my digital repertoire since I first switched to a Mac around 2004.
Uh, no? My writing style just happens to include a lot of em-dashes, as is very common. And it's not like I'm pasting a weird Unicode codepoint all over the place, that's just (rightly) how my Markdown gets rendered...
> Quite long devlog coming up, apologies—I got a little carried away with this one!
mlugg, please don't apologize for creating something I actually want to read. I'm drowning in low effort garbage, the in depth technical explanation is a refreshing breath of fresh air.
Might as well apologize for creating a language without a garbage collector, sure most people are unwilling to think, but some of us like nice things and are actually willing to apply effort.
I appreciate the kind words :)
BAH! and I forgot to say the most important part.
Much more important, thanks for not just the devlog, and explaining the changes. But also; thanks for fixing/improving this!
I appreciate all the work you've put in, I really enjoy watching the the language I like constantly improve.
Why I've moved more to a couple of language/software dev discords and away from Hacker News. Way too much uninteresting AI nonsense on here for a while now.
Would like to join as well if you're willing to share
Usually I bounce between the Zig and Odin discords, as well as the Handmade Network discord.
It wasn't even long! It seemed much shorter than the typical LLM-expanded drivel that crosses the HN front page daily.
[flagged]
??
Think theyre just implying that the quoted text comes off as a bit pretentious..
oh, I think it's mostly frustration over how eager everyone is to delegate their thinking to literally anything else, accelerated by [gestures at reality]. Is frustration with apathy really pretentious?
This change + the existing packed struct logic will be great for working with bit packed binary headers w/o having to manually twiddle so much about the bit handling along the way.
It's so interesting to read comments like this and contrast them with the "don't read the code" type of vibes out right now. It feels like half of the developer world is optimizing low-level struct packing and the other half is YOLO'ing 300 KLOC Electron apps. Very confusing.
Same as it ever was.
yeah, remember those newfangled fancy Node.js guys who would just copy/paste from Stackoverflow without any understanding?
Or the Java guys who wrote bloated apps that wasted CPU cycles on garbage collection instead of writing in C++, like God intended?
Or the Fortran / Cobol guys who wrote in those God-damned, wasteful, useless high-level languages, instead of using assembly, like a proper programmer should?
Which ones made the money?
So vibe coding = copy/paste from SO = interpreter with GC = compiler = punch cards. Got it.
Here's a description of a "real programmer" from all the way back in 1983:
https://users.cs.utah.edu/~elb/folklore/mel.html
I think it makes sense, if one sees that LLMs exposed various pre-existing splits in the developer world.
Those who viewed code as a means to build something else, are happy to switch to LLMs if they can build that something faster/cheaper.
Whereas, those who liked coding for its own sake, don't want to use LLMs, and fear for their jobs and their happiness.
Unfortunately for the latter group, we're moving to a world where most development is done by LLMs, and only cutting-edge or hobbyist work is done manually. E.g., Japanese artisanal wood-working and joinery is beautiful and elegant... but modern carpentry doesn't build that way.
I used to be paid to go in and fix messes of code, created by juniors who were forced to build things they didn't understand.
Now I get to fix things created by managers who enjoyed building things they still don't understand.
I know which kind I want to be.
i think it's perfect: AI allows you to go incredibly deep (you have unlimited access to context to make incredibly impactful surgical changes), or you can go incredibly broad (you have unlimited access to context to tie a mind numbing amount of components together). what shakes out is the middle layer: "infra" between "algorithms" and "product".
though, to be fair, the middle layer itself is composed of this same work. so it's fractal, or turtles all the way down.
And why do you mention AI here at all? These statements are ridiculous, world didn't start last year.
because i was responding to a comment about it.
Zig is already great for this with ‘packed struct’ and arbitrary size ints. Allows for very clean protocol creation between systems with known properties. This is another great step in that direction.
you need different packed structs for little- and big-endian data. and casting with little-endian data is a nightmare - you need to reverse-cascade your struct fields to be in accordance with the little-endian bit-pattern. (or have a comptime function that does it for you, of course. but then you lose all declarations for the struct). what should be a simple writing down of a protocol is now a pedantic and error-prone ordeal.
This has been largely solved by everyone agreeing to use little endian. There aren't really use cases for wanting to convert between them.
Does that mean there are no file formats thatbuse big endian? And network byte order isn't a thing?
Generally those edge cases are always the same endianness. You don't need big and little endianness versions of the structures. What's important is that everyone agrees on the same thing.
There are some cursed data formats where something is little endian in some places, big endian in other places
Generally speaking though the types you handle in business logic (what your application actually do) shouldn't have any endianness
Network byte order has nothing to do with the kernel and you have to care about it
It’s a standard because neither side of the connection knows the endianness of the other side so there must be a standard. That standard is big endian regardless of your architecture or kernel or anything else
So any serialization intended go over the network should be big endian
right, so a zig app will just do little endian. in the very unlikely event you have it running on a big endian machine you have to do extra work.
You may have never done socket programming, or do you use wrapper libs in Zig? Because you have to send the kernel big endian port numbers for example.
What do you do if you program a kernel in Zig, or just generally do low level networking?
My point is to refute the statement that everyone has agreed to little endian, and so there aren't use cases to want to do conversion. Programs do not exist in a vacuum, most programs do not.
Well you would, of course, have a mapping layer between wire types and domain types, like in any good codebase. You do the endianness conversion at that boundary, and then you can just send it out.
> You do the endianness conversion
Exactly my point.
And what happens if your zig app happens to be a network driver running on a microcontroller?
If someone chooses to load a TIFF or a PSD or an AIFF or…
then you either use an existing C library (the most likely approach) or if you are determined to re-implement it you have to be careful parsing their bytes.
Or you just go ahead and forget that big endian ever existed. It's not coming back.
it’s little-endian protocols that require that you juggle your struct fields.
plus, there are still big-endian protocols that will stay for a long time. for example, MIDI clip files in MIDI 2.0 are big-endian.
When I first found out about bit fields in C, I was left wondering what the order of bits was in a byte, eventually I convinced myself it doesn't matter, since the byte is the smallest I/O unit, and lived with the fact that casting between bitfields and bytes was UB (or unspecified, I can't remember), and as such, was another thing I wasn't suppossed to do when writing C.
All this to say that Zig just keeps cleaning up and giving well-defined semantics to warts I learned to live with in C.
Writing linkers must be incredibly rewarding - go has its own, there's mold, there's LLD, there's the OG GNU bfd LD and now Zig has one too! I am sure there's a Rust one too - Wild!
Every one of them is faster than the others too lol! Mold for one tries really hard to be GNU ld and to be useful as an independent linker most have to - I guess Zig/Go ones are purpose built so at least those don't duplicate GNU ld compatibility.
Wait till you hear how many programming languages there are
Sure, but one might imagine that linkers are generic and reusable, so you can just pick one off the shelf instead of making a new one 1-1 for each language. Empirically this line of reasoning seems to be incorrect.
Different programming languages are very obviously not the same thing - different cp command implementations are similar conceptually to having different linker implementations that all do the same thing. But you knew that so not sure if there was a point you were trying to make there.
> Different programming languages are very obviously not the same thing
It isn't obvious to me at all. The difference between Java, C# and Go is about as important as the difference between Makita and Bosch for power tools. Yes, some people swear by one or the other, but by the end of the day it really doesn't matter
Interesting read, even as someone who isn't using Zig.
I wonder, these arbitrary-width integers... Is it actually even really worth it? My intuition is to prefer manually packing/unpacking things instead (in any language, even C that has bit width for struct fields), because it gives me a better mental picture of the code that is actually generated. Particularly for something like an signed odd-bit integer - what kind of code gets generated for sign-extension, a presumably common operation?
Does anybody have other experiences with them, one way or the other?
IIRC, for "normal" bit widths the codegen basically uses the next larger machine type and preserves zero bits on the high end. An i3 is an i8 with five MSB zeroes (with more custom behavior for "packed" i3 values). It's UB to fill those with non-zero values. For larger bit widths, like u729, you concatenate many large machine types, the compiler generates instructions in an unrolled loop, and the LLVM optimization pass usually doesn't clean that up (though, now that integers are apparently not using the LLVM u729 implementation, perhaps there are some more optimization opportunities).
They're situationally useful, especially when performance isn't an enormous concern. That u729 example above came from a variant sudoku solver I wrote to aid developing new puzzles (easy to check the rough magnitude of the solution space for whatever idea I was mulling over and examine how restricted the board actually was -- just an intermediate step in puzzle design). It's not optimal (hard on the icache, can be hard on registers, other issues abound), but it's dead simple to use, and the assembly isn't terrible, beating all the normal solvers I saw floating around. It's a nice point on the laziness/correctness/good-enough-perf pareto curve.
Another comment mentioned this, but they're great in packed structs for representing weird numeric entities (I think I have a logarithmic number system floating around which does that).
One thing the language does quite a lot is use them to guard against certain classes of human error at compile time. It doesn't perfectly make impossible actions unrepresentable, but shoving a full u32 into a shift argument usually doesn't make sense, so the types are constrained to be smaller.
I can't imagine any situation where I'd use a u729 instead of a StaticBitSet. For size 729, it would end up backed by a bit_set.Array, not a bit_set.Integer.
https://ziglang.org/documentation/master/std/#std.bit_set.St...
I don't program zig, so it's not clear to me if you can use zig's bitsets arithmetically.
Sometimes it's just more clear to work with integers than other representations. Most situations with a state space of N bits have meaningful integer representations, where arithmetic functions on those representations are also meaningful.
For example, CRCs can be written as the remainder from long division of the message by the polynomial. Defining nontrivial cyclic permutations is also much more straightforward as functions on integers than on bitsets.
For other situations like a CRC on an arbitrarily-sized message, a big int would be better, surely? You can do long division on those. https://ziglang.org/documentation/0.16.0/std/#std.math.big.i...
I was talking about GP's u729, which is 9*9*9, the state space of a sudoku board. Can you come up with a situation where dividing that number by anything is meaningful?
Old habits :)
If I had to steel-man the idea, I'm pretty sure the integer-based solution has better codegen with many kinds of sparse, comptime-known masks. I think you're right though, StaticBitSet looks better.
For your specific case, even a simple `[9][9]u16` might perform better (where you make use of nine bits in each u16). For each entry, the nine mask bits would be in the same bit positions, so the compiler won't have to do a bunch of shifts to extract/align the bits. CPUs love consistency. I doubt it's worth the additional codegen complexity to save 70 bytes in your data model.
My initial implementation used [9][9]u9 (which desugars to [9][9]u16 with some zero bits) and was a fair bit slower. If I had to guess, it's because the shift/extract/align you're describing isn't actually a part of the core solving algorithm, and when you have box constraints, knights-move constraints, etc, you're usually not doing anything which fits in a single u16.
It's pretty great in my toy emulator project (https://github.com/floooh/chipz) as 'system bus' where each bit is a 'wire' which is then mapped to chip input/output pins.
The bus-width is a generic parameter and can be below or above 64 bits (depending on the emulated system). With arbitrary-width integers the high level code remains the same no matter what the bus-width is, and from looking at the compiler output, as long as bit operations don't straddle the underlying 64-bit integer boundary, those bit operations are just as efficient as working on a simple 64-bit int.
Also AFAIK LLVM supports arbitrary-width integers since pretty much forever, Zig just 'exposed' them in the language (as later did Clang via _ExtInt(N), which is now deprecated in favour of C23's _BitInt(N)).
The other nice usage (also in emulators) is for chip registers and counters, those often have odd widths (like 5 bits), and writing those as u5 instead of u8 in the code is just nicer since it matches the chip documentation, and when reading the code it's immediately clear that this u5 is a 5-bit counter or register.
I love it. Easily one of my favorite things about the language.
Example: shifting more than the width of the shifted integer is illegal behavior in Zig: therefore, the, what, shiftand? let's go with that, the shiftand for a u64 must be a u6 or smaller.
Sounds annoying? No, it's great! Check this out:
It's really nice!As an fpga engineer dealing with bitwidths that are non-byte multiples is very normal and when I end up writing software for various reasons, I often miss it. Usually when trying to slice and parse or construct messages.
Obviously there are ways around pretty much everything, but it’s nice to have first class language support for bit slices.
except it isn't bit slice, it isn't indexing within a range - it's just integer type that only allows values up to 2^width, with same alignment rounding up as with the rest
It's a bit slice if you put it in a packed struct.
I like them, they're nicer than C's bitfields: The order isn't implementation-defined, and the types remember their range rather than being converted to a power-of-two size upon read. (Maybe that's possible with C23 _BitInt(n), I haven't tried if those work in bitfields)
It's great for defining fancy floats used in machine learning
e.g. https://github.com/zml/zml/blob/33ced8fa078b3c7c8c709bd526ae...
IMO they're fantastic. You can write out a bit layout from a CPU's manual fro example and you can just use whatever bit width the manual specifies, and the compiler takes care of figuring out all the underlying manipulation for you. Which results in much more readable code because you don't have to worry about packing/unpacking it because the compiler will do that for you.
OT: I'm always surprised at how popular Zig discussions get here, or Youtube and other medias.
Don't get me wrong, I love Zig and I think it's a great C replacement, but I'm very confused on why C3 or Odin rarely get any attention at all, despite being in the same C-replacement crowd.
But still surprised at what Zig does better than these other projects? Is Andrew much better at marketing/promoting the language? He's very hard to dislike.
I can only answer for me, and while I do think it's more significant a metric for me, I equally assume it probably has some influence on others as well.
C3 uses :: for namespaces, that makes it a competitor with C++ more than C. Equally Odin's syntax is more at home among python, not systems programming.
The appeal of Zig is it feels like C. To many people, this is a downside. C is very very scary to them. But for people who feel at home in C, it's not a downside.
Additionally, the selling point for both are "c replacement" where the selling point of Zig is "good systems programming language" C is only mentioned by it's users as a heuristic.
If 2 groups are trying to replace a language that people are running away from, and that's their best selling point... I'd assume they're less likely to be as successful as a different language just trying to be as good as it can be.
I've even stopped comparing Zig to C, IMO, it does a disservice to both. And I say that as someone who likes C.
Full disclosure, I need to spend a bit more time with both odin and c3 to know exactly how this compares. But the reason I keep writing Zig, and still love it, is how simple it is. Zig is aggressively insistant on simplicity at the expense of functionally or comfort. The only other high level language I know of that is as aggressive about it's design simplicity is infact C. While I assume it's an accident when C does it, it's definitely not an accident in Zig.
C3 is a contender to C++ because of its namespace operator?
I think Andrew is a big part of it, and the people he surrounded himself with are the other part. What kind of pre-1.0 language hosts conventions? Crazy that they manage to do that. Andrew's vision has always been clear and inspiring to me. I think this got Zig its initial following, and they have capitalized extremely well on it to grow as a community.
Andrew doesn't strike me as someone who does any marketing at all. He just wants to make the language he wants to use, and does it well.
Sometimes its just right time, right place. But also, Zig has received attention via projects like Ghostty, TigerBeetle, and Bun (prior to rewrite of course)
I believe I read a post by Andrew detailing how he intentionally did marketting in a way to attract users, the right contributers, and donations - he was quite intentional about making his full-time role sustainable (and now more roles).
They have definitely done a lot of marketing through social media and forums like HN. There have been large numbers of posts here by Zig's developers for years, and a few releases of LLVM even mentioned Zig prominently in their release notes.
Maybe people just like the language
Successful marketing is like successful propaganda - it cannot look like it.
With Zig, I can just import SDL.h and use it without writing a binding.
Can I do that in C3 or Odin?
And then you can get AI do a nicer port of SDL.zig and you get way better decls.
Proper enums, proper tagged unions, and often reading the docs can allow the AI to distinguish T * to one of
1. [*]T
2. [:0]T
3. ?T
4. *T
And these are just the most common ones. If you know it’s a read only pointer/array then you can add the const modifier
Odin has SDL built into the language (shipped as a vendored library).
That's not what I mean...
There is a mountain of code written in C that you can simply include in Zig without a wrapper dependency and without having to create the wrapper yourself.
> Don't get me wrong, I love Zig and I think it's a great C replacement, but I'm very confused on why C3 or Odin rarely get any attention at all, despite being in the same C-replacement crowd.
Doesn't matter as neither will see significant adoption.
Zig has a really great backwards compatibility story with C, and it also is a better C compiler even if you don't write a single line of Zig. It's not hard to see why that is popular.
Yes, Andrew did a lot of internet cult marketing over the years, and then you have exponential free cult marketing.
Andrew pushes lots of "social issues" so he has that crowd and they push zig as a way of pushing their social views.
I’ve followed Zig fairly closely and this is the first I’ve heard of Andrew pushing “social issues”. I don’t believe for a second that it’s a factor at all.
Yeah, nah. Not so sure about that. I love zig, and I appreciate the rigour, care and thought that goes into the language and it's libs. What Andrew Kelley and the team are doing is excellent work, creating a useful, simple language with which to write efficient, correct programs.
His politics don't matter to me. Hell, if the politics of technologists dictated whether I used their products, I'd have to go live in the wilds, without any tech. :-)
What "social issues" would those be?
How confident are you? I ask because I'm a zig zealot, and am constantly shilling for it. But I disagree with a number of ark's positions, and think of him as a bit of a shitter... So I don't think "cult of personality" accounts for it, despite how easy it would be for someone to be able disregard zig if was just a personality cult.
I have no issues w/ the zig language and I'm not saying that's the only reason why people talk about it. There is however a subset of very vocal people who will go out of there way to bring stuff up and push something if they do see that is a part of it. Not that it's the only reason why either, just additional motivation for people to go out and push it that otherwise you might not get. All you need is a couple people who view that as a kind of campaign and they can radically increase the visibility of something on the internet, and turning programming that has some broader social or moral thing related to it even just through the creator is a very easy way of doing that. Rust has a similar thing. I don't view the instinct that leads to language zealotry or zealotry related to social issues(or say religion) being that distinct and it's probably a similar personality trait that encourages both, and it's generally one I find unpleasant regardless of the particular content. FP can also lean in that direction. If you get some narrative you can say this language fixes stuff in a fundamental way + also can appeal to the social thing it just riles people up who will go around talking about it online non stop.
> Consider, for instance, bitcasting a [2]u8 to a u16. Under the old semantics, the result of this operation depends on the target endian: on big-endian targets, the first array element became the 8 most significant bits, whereas on little-endian targets, the first array element became the 8 least significant bits. Under the new semantics, because we only care about logical bit representation (which is endian-agnostic), the operation behaves identically on every target:
This is a huge mistake. You would never expect something like bitCast to do this.
I don't understand this approach. Why change something so simple and low level to be complicated and high level?
Just don't allow casting to u24, as it makes no sense unless you define u24 to be u32 sized as I think c standard does.
I think this approach as an idea is bad but at least just add another built-in that implements this higher level idea to not break a simple expectation and current behavior?
> Just don't allow casting to u24, as it makes no sense unless you define u24 to be u32 sized as I think c standard does.
The reason u32->u24 casting must be well defined is because some hardware (e.g. many GPUs, microcontrollers) only have floating point multipliers. A 24 bit unsigned integer (stored in a 32 bit register) can be losslessly converted to a 32 bit float by the hardware, multiplied, then converted back.
This is much faster than doing 32 bit multiplication in software, however, you still need to tell the compiler about this constraint.
I am criticizing the part where they allowed [3]u8 to u24 bitCast in the first place. It doesn't make sense logically as u24 is likely not 24 bits in any targets let alone portably on every target.
Interpreting u24 like it is actually 24 bits sounds like programming in crazy land since it is not 24 bits in any relevant architecture afaik.
They didn't allow []u24 with a similar rationale as far as I can remember. I agree with this as someone programming at this level should be able to understand there is no real u24 layout and they should use []u32. Going with the same magical rational they went with here, compiler should generate unaligned u24 loading code when you use []u24 since it is "logically 24 bits"
The ease of dealing with arbitrary bit-width integers and packed structs is actually one of the 'killer features' for me in zig.
Zig natively supports arbitrary bit-width integers, the ABI is defined and you could simply think it as a slice of the next larger backing integer.
The[3]u8 to u24 bitCast will simply be backed by a 32bit int, using the same ABI. As you have u1 - u65535, sometimes it can be multiple words.
The 24 Bits (3 Bytes) [3]u8 to u24 example is exactly related to utf-8 that covers all the languages but excludes the emojis.
There are very valid use cases when you want to limit utf-8 to U+0000-U+FFFF, and it is valuable if your language allows you to make those decisions.
Remember, in zig packed structs are just integers and integers are just a group of logically consecutive bits.
Arrays like []u24 do not have the same ABI, arrays are not bit/byte packed, are not universally LSB across archs etc..
The compiler isn't producing unaligned code, don't confuse the abstraction with the concrete implementation. And yes [8]u1 and [8]u8 are exactly the same size and shape, even though they are arrays.
My current project is parsing ELF/Macho files, I can easily have zero allocations in my hot path with zig, the same is far more challenging in C, so I am biased, especially with zig allowing methods on structs.
And yes, I do use that crazy casting to 0xdeadbeef and other ascii metadata that is in those files.
To be clear here, I am not trying to prove you wrong, this is one of the places zig is very different and (IMHO) useful. Especially with streaming data or where you have network ordering etc... It is so nice to only cast what you need to but it does take a little while to wrap your head around how this interacts with buffers which are not your native endianness. At least for me, once I figured out to separate the shape of those data streams from their values it was super useful.
> The 24 Bits (3 Bytes) [3]u8 to u24 example is exactly related to utf-8 that covers all the languages but excludes the emojis.
I'm not familiar with Zig, so maybe it's doing something weird here, but that doesn't really make sense with Unicode in general.
First, the largest Unicode codepoint that will ever be allocated is U+10FFFF [0], which is less than 2^21, so all Unicode characters will fit in a 24-bit integer. Perhaps you're thinking of UCS-2 or UTF-16 without surrogates, which are both 16 bits wide and are limited to the BMP [1] [2] (and therefore don't include most emojis).
Second, while the characters needed for most languages lie within the BMP, not all of them do [3], so it isn't really possible to support all languages while excluding emoji, aside from using the Unicode character database to exclude certain categories [4] [5].
[0]: https://www.unicode.org/faq/utf_bom.html#gen0
[1]: https://www.unicode.org/faq/utf_bom.html#utf16-11
[2]: https://en.wikipedia.org/wiki/Universal_Coded_Character_Set
[3]: https://en.wikipedia.org/wiki/Plane_(Unicode)#Supplementary_...
[4]: https://www.unicode.org/reports/tr44/tr44-34.html#General_Ca...
[5]: https://en.wikipedia.org/wiki/Unicode_character_property#Gen...
Note the utf-8[0] in my response, the answers are on the pages you linked, but not in the sections you linked,
utf-8 encodes code points in one to four bytes, it is byte oriented vs utf-16 etc. In zig u8 is a byte, and is also (by convention) a char, although there isn't an explicit char type in zig. Technically there are chars in languages that need all 4 bytes in utf-8, but almost all of them are historical or emoji's in utf-8.
24bits (3 bytes) in utf-8 gets you Chinese, Japanese, Korean. 16 bits (2 bytes) gets you Latin letters with diacritics, Greek, and Arabic scripts. With 8 bits (1 byte) getting you Standard ASCII etc...
There is a point you could make that it may have been better to use utf-16 etc... and that we should have dropped ascii/latin-1 support, but once again go up to the 'Basic Multilingual Plane' in your [3] and notice that is covered by 24bits (3 bytes) in utf-8 encoding.
[0] https://en.wikipedia.org/wiki/UTF-8
> ... but almost all of them are historical or emoji's in utf-8.
I just posted a comment, five minutes after you wrote that, which I won't repeat here since it was quite long. But one of the languages whose alphabet is found in the higher multilingual plane is Fulani, spoken natively by 37 million people (plus another two and a half million who have learned it as a second language). While it can be written in other alphabets (both Latin and Arabic have been used to write it in the past, for example), other alphabets don't usually represent all the sounds of the language properly, making it awkward. There's a reason why the Adlam script was invented to write Fulani with; and that invention was recent enough that it was assigned the U+1E900 to U+1E95F block, since the basic multilingual plane was full by then.
So although it's easy to think that the astral planes are only used for emoji and historical languages, that's not actually true. There are languages spoken by millions of people in those astral planes as well (yes, languages plural; Fulani isn't the only one, it's just the largest).
To be clear, I was talking about a use case, not all use cases.
There are very real times where you have to support all 4 bytes, there are others where other drivers require you to restrict the domain of discorse.
It doesn't change the value/cost of bit casting in a language with arbitrary bit width languages, especially when combined with the fact that int overflows are detectable illegal behaviour and you have saturating and wrapping operators.
This is in addition to the ease of using packed structs I mentioned above.
A list of some advantages:
* Zig's arbitrary-sized integers have a fully defined ABI for padding
* Allows for strict domain modeling using them as platform independent refinement types
* Precise memory packing, allowing more utilization of register space etc...
* OOB compile time checks
* Bit masking optimization, where sequential changes to packed values are often merged into a small number of and/or masks
To move to a more information theory example:
DNA nucleotides (A, C, G, T) represents quaternary state pairs.
If you wanted to store an array of 1,000 DNA nucleotides, each symbol is one of 4 bases, requiring exactly 2 bits of information. The Shannon Information would be: 1000 * 2bits = 2000 bits.
With uint8_t this would take 8k bits, vs 2k bits of u2. That is 300% more for uint8_t.
It is still horses for courses, but as an example consider 12-bit sensor reading in a standard u16, the data type allows invalid states. To ensure safety, requires manual defensive logic throughout your program in the traditional C/Rust/...
That traditional model in zig:
And the lower overall Kolmogorov complexity (cherry picked) form: C23 does have _BitInt types for structs which can help if bit packing is your primary need, IMHO it doesn't offer the same advantages.As an example, and I may be wrong, but I think you cant easily perform checked arithmetic or use standard overflow operations on individual C bit-fields without copying them out into standard standard types (like int), modifying them, masking them, and copying them back.
With Zig the invariant is maintained implicitly at the type layer, removing runtime validation branches, error paths, and testing code
Does it solve all problems, no. Is @bitCast, a zero runtime overhead, compile-time checked bit reinterpretation and [3]u8 \to u24 useless and silly, no.
Yes, there are certainly use cases where you know the data you're parsing will only come from a narrow range of Unicode, such as U+0000 to U+007F — or from just the letters GCAT, as you mentioned. The overhead of converting 8-bit input to 7-bit might not be worth the cost, but the benefit of storing your input in just 2 bits per "letter" is definitely worth it.
I mostly wanted to make sure people know that the upper multilingual planes are a very real use case, and you need to test them. This is more important for languages such as C# where UTF-16 is the norm: many programmers don't know that they're handling surrogate pairs wrong until someone tries to backspace over an emoji character and it turns into something weird. It's probably less relevant to Zig, which didn't make the mistake that C# and Java did by starting out with UCS-2 (to be fair to them, they were designed in the era where people still thought that 65,536 codepoints would be enough for every language and Unicode would never need more than 16 bits). But the upper planes are important, and need to be tested no matter what language your code is written in.
> utf-8 encodes code points in one to four bytes, it is byte oriented vs utf-16 etc. In zig u8 is a byte, and is also (by convention) a char, although there isn't an explicit char type in zig. […]
> 24bits (3 bytes) in utf-8 gets you Chinese, Japanese, Korean. 16 bits (2 bytes) gets you Latin letters with diacritics, Greek, and Arabic scripts. With 8 bits (1 byte) getting you Standard ASCII etc...
Ah ok, so if I understand you correctly, you're taking a variable-length encoding (UTF-8), and limiting and/or padding it to 3 octets (24 bits)? In that case, what you said in your original post makes sense, but I'm not really sure why you'd ever want to encode something this way: you have to deal with the complexities of a variable-length encoding to parse each u24, you have the poor space usage of a fixed-length encoding, and you're using 24 bits to encode only 0xFFFF characters (even though you can fit all of Unicode in only 21 bits).
> Technically there are chars in languages that need all 4 bytes in utf-8, but almost all of them are historical or emoji's in utf-8.
Yes, the majority of the characters in the non-BMP planes are for archaic languages, but that's not really the right way to look at it, since most languages only need <100 characters, and there are more dead languages than living ones. Instead, I'd look at it from the reverse lens of how many living languages need non-BMP characters. This sibling comment [0] gives one example, but there are lots more [1] [2] [3] [4] [5] [6].
Now, it's fine to not support these characters, but the argument in that case should be that you've decided that the characters aren't important enough to outweigh the technical challenges, not that nobody needs the characters.
> 24bits (3 bytes) in utf-8 gets you Chinese, Japanese, Korean.
It gets you a subset of CJK that's probably sufficient for many purposes, but there are nearly 75k CJK characters outside of the BMP.
> There is a point you could make that it may have been better to use utf-16 etc... and that we should have dropped ascii/latin-1 support, but once again go up to the 'Basic Multilingual Plane' in your [3] and notice that is covered by 24bits (3 bytes) in utf-8 encoding.
If you are willing and able to use a 24-bit encoding, then I'd argue that you should just use UCS-3/UTF-24, since those allow you to encode every Unicode character. The only downside is that these encodings aren't formally-defined so other programs won't understand them, but if that's an issue you can use UCS-4/UTF-32.
[0]: https://news.ycombinator.com/item?id=48682043
[1]: https://en.wikipedia.org/wiki/Unified_Canadian_Aboriginal_Sy...
[2]: https://en.wikipedia.org/wiki/Chakma_(Unicode_block)
[3]: https://en.wikipedia.org/wiki/Mro_(Unicode_block)
[4]: https://en.wikipedia.org/wiki/Kirat_Rai_(Unicode_block)
[5]: https://en.wikipedia.org/wiki/Nag_Mundari_(Unicode_block)
[6]: https://en.wikipedia.org/wiki/Ethiopic_Extended-B
> ... utf-8 that covers all the languages but excludes the emojis ...
Ah, but the U+0000 to U+FFFF plane does not cover all the languages. You might think that only historical and archaic languages are found in Unicode's astral planes (e.g., U+20000 to U+2A6DF is used for historical Chinese characters no longer used today), but in fact there are modern languages found in the U+10000 plane.
You might not care about Osage (the language of the Osage Nation of northern Oklahoma) since its last native speaker passed away in 2005, but there is a revival program trying to teach Osage to people. Osage's script was developed quite recently as part of the revival program, so it couldn't fit into the U+0000 to U+FFFF block and it was assigned U+104B0 to U+104FF.
The Toto language of Bengal, on the other hand, is still active: over 1000 speakers, all living in the village of Totopara. It also never had an alphabet until recently, so its Unicode block is U+1E290 to U+1E2BF.
Then there's Wancho, spoken by about 60,000 people in India. Its alphabet was created between 2001 and 2012, and added to Unicode in 2019. It was assigned the U+1E2C0 to U+1E2FF block (immmediately after the Toto language, you might notice).
Then there's the Ho language spoken by over a million people in India. Wikipedia cites a 2001 census as having 2.2 million speakers, and a 2011 census as having 1.4 million speakers. I very much doubt that both of those are accurate (you don't lose half a million people from an ethnic group in just ten years without some kind of war or genocide, and the Wikipedia article would have at least mentioned that if such a thing had happened), but to be safe, let's go with the lower estimate and say that at least one and a half million people speak Ho. It can be written with the Latin alphabet, but its own alphabet is Warang Chiti (sometimes spelled Warang Citi), which was added to Unicode in 2014 and assigned the U+118A0 to U+118FF block.
And then there's the Adlam script for writing Fulani, the language of the Fufulde people of western Africa. Fulani is spoken natively by 37 million people, and as a second language by another 2.7 million. Adlam's Unicode block is U+1E900 to 1+1E95F.
So if you restrict your program to only working with the basic multilingual plane, it's not just emoji you'll be leaving out. It's also modern languages, spoken by anywhere from 1000 people to 37 million. How many speakers of a language are enough to draw the line and say "No, I won't ever translate my software into your language"?
Now, if your software is only targeting one language and you never intend to translate it, then yes, you'll only lose out on emoji if you stick to the U+0000 to U+FFFF range of the basic multilingual plane.
But realize that the higher planes are not just for dead languages. Living languages have ended up there too, and there are likely to be more in the future. It's quite possible that right now, someone somewhere is saying "Hey, why doesn't my language have its own alphabet instead of using Latin characters to write it? The Latin characters don't express the sounds of my language very well." And when they do get that alphabet worked out and manage to get it accepted into Unicode, it'll certainly land in one of the higher planes. Most likely the U+10000 to U+1FFFF plane which isn't at all full yet, but who knows. If you want to be able to handle every language spoken (and written) in the world today, you must be able to accept the full range of Unicode, not just the 16-bit range.
> You don't lose half a million people from an ethnic group in just ten years without some kind of war or genocide.
Nothing happened to the people, they are growing year on year. But languages can die very easily if governments don't put efforts on teaching it to children. That is exactly what happened to the Ho language. There is no advantage on learning these small regional languages so children put their effort on more popular languages like Hindi, Odia and English.
Here is a good article on this topic:
https://www.vogue.in/content/when-languages-in-india-disappe...
I'm familiar with the phenomenon, as my wife is a linguist who did her master's thesis on the phonology of a small language spoken by about 7000 people: many of the kids don't want to learn it, and just want to learn the majority language of the country since that's what they have to use in school. But I didn't think that could be the explanation for a 25% decline in ten years: new people may not be learning the language, but the only way people stop speaking their mother tongue is if they immigrate to a new country and fully adapt to it (happens to a few people, usually who immigrated as children) or if they die (by far the most common reason for language-use decline: the old people are dying and the young people aren't learning it). If the decline was a couple hundred thousand that would be the outside limit of probability, as far as I know.
More likely, in my opinion, is that both are happening: yes, the language is declining, but either the earlier census overcounted speakers (e.g. counting children as speaking it when they weren't actually learning it) or else the later census undercounted speakers; either way the language decline would look larger than it actually is. Given that Ethnologue (https://www.ethnologue.com/language/hoc/) rates the language vitality as "Stable" — "The language is not being sustained by formal institutions, but it is still the norm in the home and community that all children learn and use the language" — and they usually know what they're talking about, I suspect the language decline isn't that fast and a census counting mistake is a more likely explanation for the discrepancy over ten years.
> many GPUs
Citation please - every single GPU in the literal world supports integer arithmetic for operating on tid, gid, etc.
From page 175 of the AMD CDNA4 ISA:
https://www.amd.com/content/dam/amd/en/documents/instinct-te...
> V_MUL_U32_U24
>,Multiply two unsigned 24-bit integer inputs and store the result as an unsigned 32-bit integer into a vector register. D0.u32 = 32'U(S0.u24) * 32'U(S1.u24)
> Notes
> This opcode is expected to be as efficient as basic single-precision opcodes since it utilizes the single-precision floating point multiplier. See also V_MUL_HI_U32_U24.
Nvidia GPUs used to do the same thing and theres a umul24 intrinsic if you care to use it.
https://stackoverflow.com/questions/5544355/cuda-umul24-func...
This is super-super-niche since it basically only applies to 32-bit integer multiplication.
You likely won't run into it unless you're doing high performance embedded systems or GPU programming on non-NVDIA cards, and for some unknowable reason, your workload does a 32-bit integer multiplication in the hot path.
That's literally only for 32bx24b (I don't remember why we did that specifically for CDNA - I'll ask someone) but as you see from V_MUL_HI_I32, V_MUL_LO_U32 there is very much vector arithmetic hardware (nevermind that we're not talking about VALU but conventional scalar ALU).
I think he has a point, but I am still not 100% convinced by the arguments relating to casting.
There is a difference between a u24 data type inside u32 and a u24 datatype inside u24 and that is what's so frustrating here. u24 is an alignment nightmare so it will basically never exist as "u24 in u24" and only ever as "u24 in u32".
For casting to make sense, the alignment must be compatible and it's not clear how you can simultaneously make arbitrary bit data types simultaneously useful for the scenario of describing bit fields in packets, where padding is inherently undesirable and performing integer arithmetic with an FPU, where padding is an acceptable cost for alignment. These appear to be mutually exclusive use cases.
While the GP might be technically wrong in a narrow sense, GPUs are built for FP, and that's what you want to be doing if you're using them as accelerators.
You don't know what you're talking about: an enormous amount of TOPs now runs through quantized (read: integer) kernels. Many GPUs don't have even FP64 or even FP32 support.
EDIT: I was completely wrong, I have mostly worked with GGUF and related quantizations that are LUTs, thank you for correcting me.
> The quantized integer kernels aren't running true integer multiplication, the quantization is it's own thing, they're basically enums not integers
ELI-a-GPU-compiler-engineer-working-at-a-major-vendor (because I am). Ie I can pull up the design docs for our ALUs and literally see that you're wrong.
I understand the reaction, but I don't agree. I suggest reading the associated proposal[0] along with the devlog, and having a real think about what's going on here. I'm responding to you saying that you "don't understand" the approach: reasonable, and resembles my initial reaction.
I was inclined to agree with you, but what decided it for me is that Zig has another mechanism for "reinterpret bytes". It's exposed on the stdlib as std.mem.asBytes, but this is literally a wrapper for the following:
So nothing is lost here: if you need, for whatever reason (and those do exist), to get a raw array of underlying bytes, you absolutely may. Std.mem also has bytesToValue(T, bytes) T, which makes a copy. All the ingredients are there, and this family of mem functions are thin wrappers over builtins, which boil down to pointer casting, dereferencing, and comptime magic.Also worth noting: packed structs in Zig are already defined as logically little-endian: the first field is of low significance, the second is above that, and so on. So this makes `@bitCast` consistent with an existing convention of treating integers as logically little-ended, without regard to how they're actually arrayed in memory.
Plus it stands to make low-level bit-twiddling, using oddly-sized integers, optimize better. I like that, especially when what we trade for that is: nothing. Nothing at all, this is a pure win.
I'd even guess it's that rare language update which silently fixes buggy code, where someone figured "well, basically everything is little-endian already" (or just didn't think about it), and now that code works properly on big-endian machines.
[0]: https://github.com/ziglang/zig/issues/19755
GCC has had __int24 for the AVR backend for some time. Useful for larger integers than int16_t while saving 25% over a 32-bit value. C23 does not mandate padding for _BitInt types. It is wrong to assume that will happen or is the optimal implementation for portable code.
Thanks for the context, but what I am criticising is this part:
> it became allowed to use @bitCast to reinterpret a [3]u8 as a u24
This cant't make sense unless u24 is defined to be 24bits in the first place. It is just silly to allow something like this. It would make so much more sense to me if they started disallowing this or just even print a deprecation notice for it for one release version.
> Useful for larger integers than int16_t while saving 25% over a 32-bit value
You can't even do []u24 in zig as far as I can remember and understand anyway so this is only happening in a packed struct context.
C doesn't mandate padding but C compilers allow having pointers and arrays of irregular _BitInt types as far as I can understand.
In this [1] document, in Abi considerations section, it writes that it is defined to have next-power-of-two layout size.
Also here (for RISCV) [2] it seems like it is defined with next-power-of-two layout.
Also the document here (for x86_64) defines it similarly [3]
[1] https://www.open-std.org/jtc1/sc22/wg14/www/docs/n2709.pdf
[2] https://github.com/riscv-non-isa/riscv-elf-psabi-doc/issues/...
[3] https://gitlab.com/x86-psABIs/x86-64-ABI/-/tree/master?ref_t...
> This cant't make sense unless u24 is defined to be 24bits in the first place
It's worth remembering that zig is a ~hll that should be platform agnostic. suppose someone built a byte-chip with a 24 bit word. the "new" zig way of doing things will be more portable and slot right in, and support 32- and 16-bit datatypes just fine.
I sort of agree ... bit casting from an N width integer into an array of ... woah ... that's too far. It's bitcast not byte-cast which has an implied reinterpretation on a same or smaller word size in the cpu.
Once you see that the fact somebody has a u24 in their code is between them and the compiler alone.
As others probably noted byte casting (keeping the same endianess) is what unions are for.
To me it makes sense. If you don't know what endianness is, it doesn't make sense that a program you write in one programming language works for one target but doesn't work for the other.
I think endianness is the footgun that Zig is solving, rather than Zig being the one introducing a footgun when you deal with endianness.
> If you don't know what endianness is
It is not feasible for someone to write endian portable code in a language like Zig without understanding what endianness is imo. Regardless of how they change @bitCast there will be other cases that break this like doing @ptrCast + @memcpy.
Also this breaks currently written code that is endian portable and uses @byteSwap like it is done in most other programming languages that do these things.
> This is a huge mistake. You would never expect something like bitCast to do this.
Is there at least some sort of @transmute or something ? If Zig wants to say "bitCast" means this odd operation, but provides the thing most people actually want under some plausible name that's just an extra thing to learn which seems OK.
@intCast
So, since I don't write Zig I had to go look this up, to save anyone else the bother this is what Rust would call an 'as' cast or C programmers might think of as a value cast, it's going to try to make a value which has a similar meaning but of another type, which may be arbitrarily expensive. What people often want here is a transmute, Rust's core::mem::transmute which changes nothing about the bits except what those bits mean, since the bits didn't change and the machine only has bits anyway this is "free".
There's `@ptrCast` as mentioned in the post, but I agree that it's not optimal. I would really like a `@transmute` builtin.
Alternatively I suppose `extern` unions do that, but again, not quite the same syntactically.
For @ptrCast I also now need to care what the language's pointer provenance model is, and AFAICT the answer is Zig didn't get to that yet. If there's some kind of @transmute then we don't need to explain why the pointer cast worked because we never wrote one so that ducks the question, which is simpler.
If I understand it correctly, it basically boils down to copying bits from the source to the destination, in order from the least significant bit to the most significant bit. It's not equivalent to C++'s reinterpret_cast.
I'm no Zig expert, but if you want endian-dependent semantics I'd assume either @ptrCast or a packed union would do the job.
But doesn't that show why this is a bad idea? If I understand correctly, this code:
...will now succeed or fail depending on the endianness of the target. That looks like the type of footgun that will bring decades of joy.zig does not allow arrays in packed structs/unions specifically for endianness reasons (there may be other reasons as well but endianness is what i know of)
Ah, that is useful to know. Is that documented somewhere? From what I can quickly find in the obvious place [0], the only requirement is that "all fields in a packed union must have the same @bitSizeOf" and [2]u8 does satisfy that requirement.
[0] https://ziglang.org/documentation/0.16.0/#packed-union
no, but the documentation for packed structs gives the list of allowed field types. it's also not documented that packed union fields must be valid packed struct fields but people may be able to assume that
edit: also, this is a relevant issue: https://github.com/ziglang/zig/issues/12547
I wonder if packed union also got/will get the same "logical bits" treatment?
My understanding is that the "logical bits" view breaks down for unions, because the nth logical bit could be at different offsets depending on the union variant that's considered active.
I may be damaged from working on IC hardware design and various weird architectures, but I truly can’t comprehend why you’d think this doesn’t make sense.
Yeah, if your architecture doesn’t support 24-bit int it maps to 32-bits. But it also declares that the numbers you’re storing should never be larger than 2^24. It’s about type safety, and also run time checks in safe mode I believe. Bitcasting three bytes to a 24-bit type makes just as much sanse as casting 4 bytes to 32-bit. Theres zero reasons to introduce arbitrary artificial constraints on what you can do based on details of (most of) the underlying architectures, which doesn’t even matter for the operation you’re performing.
If the architecture supports 3 byte types that means it needs to support 3 byte alignments and their powers 9, 27, 81, etc. The easiest way to support this is to always map every 3-byte read operation to two 2-byte reads and then use multiplexers to recombine it into a 24 bit data type.
Of course you could also go crazy and store data in 24 bit blocks in your SRAM. That kind of ruins the 8 bit and 16 bit reads though.
You don't need to use @bitCast for the behavior you're talking about. @ptrCast still exists.
@ptrCast,
> Converts a pointer of one type to a pointer of another type. [1]
[1] https://ziglang.org/documentation/master/#toc-ptrCast
So it is not the same.
You could use it to define a function that implements bitCast. Which defeats the purpose of having any @bitCast intrinsic instead of using @mempcy for everything
Take the address and deref afterwards, and it's exactly the same. Or to say another way: if you want bits to be reinterpreted raw as if they're in memory, then... put them in memory, then reinterpret them.
> You could use it to define a function that implements bitCast. Which defeats the purpose of having any @bitCast intrinsic
Yes, and this is one reason @bitCast was changed to have different semantics that are not trivially achieved with @ptrCast.
> Take the address and deref afterwards, and it's exactly the same.
It is significantly worse to take address and deref afterwards.
You have to do something like:
@as(const u32, @ptrCast(&x)).
instead of just
@bitCast(x)
> Yes, and this is one reason @bitCast was changed to have different semantics that are not trivially achieved with @ptrCast.
This makes sense except breaking existing code that properly handled endianness by doing a conditional @byteSwap. And what you end up with is a more complicated intrinsic compared to something that reinterprets values with same layout size
Your example is incorrect. @ptrCast has the same (similar, if you want to be pedantic to the exclusion of good faith) result rules. If you need @as to @ptrcast, you'd need it to @bitCast as well.
> It is significantly worse to take address and deref afterwards.
How are you measuring worse? Because my understanding from the article is that's exactly the behavior @bitCast used to have. So, instead of worse, it'd be exactly the same?
If you mean it's simply more things that you have to type... You're describing a core language feature as "worse". For all the builtins, some of them can help the compiler emit better code, but can for some doesn't mean will for all. As an example
Could zig auto convert between these types? Yes, absolutely. But it doesn't as a design decision. On some arch, converting between float and int can be very expensive. A competent engineer will ensure they're type converting in a reasonable order. Zig requires this painfully verbose syntax it order to make it painful. Are there times where it's is actually the only reasonable option? yes, but even if there wasn't it'd still need to exist because I'm not rewriting my whole program to avoid a single float conversion. But because it's a bit painful, I will rewrite this one function to make it less painful.And, yes having already made that exact mistake... I now write better code from the start because there's no way I'm gonna ruin all my beautiful code with a bunch of ugly, annoying, hard to read, casts.
I used to complain about unused variable errors, unhandled enum branch, var unmodified (hint: use const) errors, hell even result ignored or error ignored when I'm trying to test some unrelated single line of code. But now that I'm used to them, I emit better code without thinking. It's made me a better programmer. Is it annoying? abso-fucking-lutely but I'm better now than I used to be, so: worth it; and: thankyou sir can I have another. :D
> This makes sense except breaking existing code
Before Zig hits 1.0, users should expect language changes. Has anyone claimed otherwise?
If you need the old thing often enough, you can write a wrapper for it. It's a trivial one-liner, as you've shown.
I completely disagree.
> As a general rule, the new semantics tend to match the behavior of the old semantics on little-endian targets.
They've basically said that bit casting is going to be little endian. This simplifies things for the 100% of people that are on little endian machines, while making the code still work for the 0% of people (rounded to the nearest 0.0000001%) that are using big endian machines.
FTA: “Under the new semantics, because we only care about logical bit representation (which is endian-agnostic), the operation behaves identically on every target: the first array element becomes the 8 least significant bits”
I wouldn’t call that endian-agnostic. It’s explicitly picking little-endian.
It also makes things look weird for beginners. I know how it works, but in the
example, turning two 3-bit values [abc def] into three 2-bit values [bc fa de] is way less intuitive than turning it into [ab cd ef].The behavior is agnostic of the endianness of the target platform.
That's only because we write numbers in big endian.
[dead]
Can I convert a 300-byte message to Base64 with a single instruction? Like:
`u3` would be base 8, i.e. octal---I think you meant to use `[400]u6`?
Aside from that: I'm not familiar with how standard base64 deals with endianness, so I'm not sure if it would match that, but this `@bitCast` would certainly give you a base64 encoding. But it would probably emit pretty terrible code to do that---our lowering of `@bitCast` isn't really optimized for moving around huge amounts of data in one operation! (But maybe LLVM would surprise me.)
> I think you meant to use [400]u6
Of course! I guess it was too early to do the maths correctly... :)
These posts make you want not only to use Zig, but also to marry it.
No jokes aside, these posts are the best advertisements of the language.
And I truly like their AI stance.
[flagged]
Is pasting em-dashes everywhere some kind of inside joke?
Macintoshes have had mnemonic keyboard shortcuts for inserting en- and em-dashes since forever: option-hyphen and option-shift-hyphen. They've been in my digital repertoire since I first switched to a Mac around 2004.
You too could have it easily accessible on your keyboard by using EurKEY: https://eurkey.steffen.bruentjen.eu/
I pity the fools who don't have compose keys for all their em—dash and “smart quote” needs.
Uh, no? My writing style just happens to include a lot of em-dashes, as is very common. And it's not like I'm pasting a weird Unicode codepoint all over the place, that's just (rightly) how my Markdown gets rendered...