With the still only partial return to offices, I expect lots of people are running into an unpleseant side of modern computing. Frequent software updates, logins and operating system patches will interrupt your work or at best, restart your computer between work sessions, potentially leaving you some wreckage with your morning coffee.
[Read More]
Little 'Big Ideas' in Programming Language Design
What follows are a few sort of random observations on topics I’ve pondered while evaluating new languages and thinking about building my own language projects. They aren’t radical design choices or anything groundbreaking but they lend a language its feel for better or worse.
[Read More]
The Software Peter Principle
“In a Hierarchy Every Employee Tends to Rise to His Level of Incompetence”
[Read More]
Save Arrow Record Batches Fast to Parquet With Custom Metadata During Incremental Writes
Adding custom metadata is easy and documented when saving an entire table, but adding to batched output is different.
Saving custom metadata – “schema metadata” or “file metadata” – to Parquet could be really useful. You can put versions of an application’s data format, release notes or many other things right into the data files. The documentation is somewhat lacking on how to accomplish it with PyArrow – but you totally can. Last time I reviewed the docs for Polars and DuckDB they didn’t allow for adding your own metadata to Parquet output at all.
[Read More]
Notes on simplifying complex Parquet data
Not all tools can read nested logical Map or List type data (often made by Spark.) Here are some tips to make the data more accessible by more tools.
The Parquet columnar data format typically has columns of simple types: int32, int64, string and a few others. However, columns can have logical types of “List”, “Map” as well, and their members may be more “List” or “Map” structures or primitive types.
[Read More]
Investigating Mojo 🔥
I spent the afternoon learning about Mojo. Here are my notes.
Mojo aims to be a super-set of Python by supporting the Python syntax and adding in new keywords for more performant and safe code. Mojo is a compiler that produces extremely high-performance executable binary files. It offers interop with existing Python libraries and a limited set of Python types
[Read More]
Effectively Avoid Problems When Consuming Legacy Character Encodings in Rust
There’s still a lot of old “extended” ASCII out there and you may need to deal with it. One source can be the old-fashioned “fixed-width” data formats, but it may be found in any old files like spreadsheets in Windows. Whatever the case you can’t just wish it away, sadly.
[Read More]
RustConf 2023 Notes
Notes on RustConf 2023 Talks I Attended
We saw a wide range of talks, from Rust success stories to language improvement projects, to team dynamics / Rust org growing pains. I didn’t (couldn’t) attend all talks so I may have missed some really interesting stuff. This is just what I took away from what I went to.
[Read More]
Better Code Organization by Nesting Functions
The other day I found myself writing a really long Python script full of small groups of “helper” functions. Each group only “helped” a single caller. Something felt off. What a mess. Hidden under all the clutter, the script had a fairly simple structure. There’s only one path through the code. Breaking it into separate files would only obscure the logic. So how could I make that more clear?
[Read More]
Add Key-Value Metadata to Parquet Files in C++
File-level arbitrary metadata on a parquet file could be extremely useful but adding it in C++ isn't well documented. Here's how to do it.
Although the Parquet format allows extra metadata and the C++ libraries provide a means to read and write extra metadata the capability isn’t well documented. I’ll show some example code to clarify how to read and write key-value Parquet metadata. This advice is specific to directly using the C++ libraries in the Arrow project.
[Read More]