The blog post is the author's impressions of Gleam after it released version 1.4.0. Gleam is an upcoming language that is getting a lot of highly-ranked articles.

It runs on the Erlang virtual machine (BEAM), making it great for distributed programs and a competitor to Elixir and Erlang (the language). It also compiles to JavaScript, making it a competitor to TypeScript.

But unlike Elixir, Erlang, and TypeScript, it's strongly typed (not just gradually typed). It has "functional" concepts like algebraic data types, immutable values, and first-class functions. The syntax is modeled after Rust and its tutorial is modeled after Go's. Lastly, it has a very large community.

Book Preface:

Welcome to Apache Iceberg: The Definitive Guide! We’re delighted you have embarked on this learning journey with us. In this preface, we provide an overview of this book, why we wrote it, and how you can make the most of it.

About This Book

In these pages, you’ll learn what Apache Iceberg is, why it exists, how it works, and how to harness its power. Designed for data engineers, architects, scientists, and analysts working with large datasets across various use cases from BI dashboards to AI/ML, this book explores the core concepts, inner workings, and practical applications of Apache Iceberg. By the time you reach the end, you will have grasped the essentials and possess the practical knowledge to implement Apache Iceberg effectively in your data projects. Whether you are a newcomer or an experienced practitioner, Apache Iceberg: The Definitive Guide will be your trusted companion on this enlightening journey into Apache Iceberg.

Why We Wrote This Book

As we observed the rapid growth and adoption of the Apache Iceberg ecosystem, it became evident that a growing knowledge gap needed to be addressed. Initially, we began by sharing insights through a series of blog posts on the Dremio platform to provide valuable information to the burgeoning Iceberg community. However, it soon became clear that a comprehensive and centralized resource was essential to meet the increasing demand for a definitive Iceberg reference. This realization was the driving force behind the creation of Apache Iceberg: The Definitive Guide. Our goal is to provide readers with a single authoritative source that bridges the knowledge gap and empowers individuals and organizations to make the most of Apache Iceberg’s capabilities in their data-related endeavors.

What You Will Find Inside

In the following chapters, you will learn what Apache Iceberg is and how it works, how you can take advantage of the format with a variety of tools, and best practices to manage the quality and governance of the data in Apache Iceberg tables. Here is a summary of each chapter’s content:

• Chapter 1, “Introduction to Apache Iceberg”
  Exploration of the historical context of data lakehouses and the essential concepts underlying Apache Iceberg.
• Chapter 2, “The Architecture of Apache Iceberg”
  Deep dive into the intricate design of Apache Iceberg, examining how its various components function together.
• Chapter 3, “Lifecycle of Write and Read Queries”
  Examination of the step-by-step process involved in Apache Iceberg transactions, highlighting updates, reads, and time-travel queries.
• Chapter 4, “Optimizing the Performance of Iceberg Tables”
  Discussions on maintaining optimized performance in Apache Iceberg tables through techniques such as compaction and sorting.
• Chapter 5, “Iceberg Catalogs”
  In-depth explanation of the role of Apache Iceberg catalogs, exploring the different catalog options available.
• Chapter 6, “Apache Spark”
  Practical sessions using Apache Spark to manage and interact with Apache Iceberg tables.
• Chapter 7, “Dremio’s SQL Query Engine”
  Exploration of the Dremio lakehouse platform, focusing on DDL, DML, and table optimization for Apache Iceberg tables.
• Chapter 8, “AWS Glue”
  Demonstration of the use of AWS Glue Catalog and AWS Glue Studio for working with Apache Iceberg tables.
• Chapter 9, “Apache Flink”
  Practical exercises in using Apache Flink for streaming data processing with Apache Iceberg tables.
• Chapter 10, “Apache Iceberg in Production”
  Insights into managing data quality in production, using metadata tables for table health monitoring and employing table and catalog versioning for various operational needs.
• Chapter 11, “Streaming with Apache Iceberg”
  Use of tools such as Apache Spark, Flink, and AWS Glue for streaming data processing into Iceberg tables.
• Chapter 12, “Governance and Security”
  Exploration of the application of governance and security at various levels in Apache Iceberg tables, such as storage, semantic layers, and catalogs.
• Chapter 13, “Migrating to Apache Iceberg”
  Guidelines on transforming existing datasets from different file types and databases into Apache Iceberg tables.
• Chapter 14, “Real-World Use Cases of Apache Iceberg”
  A look at real-world applications of Apache Iceberg, including business intelligence dashboards and implementing change data capture

Direct link to PDF

Dremio bills itself as a "Unified Analytics Platform for a Self-Service Lakehouse". The authors of the book work for Dremio and may have ownership interest in Dremio.

July 18, 2024 Narek Galstyan writes:

We were naturally curious when we saw Pinecone's blog post comparing Postgres and Pinecone.

In their post on Postgres, Pinecone recognizes that Postgres is easy to start with as a vector database, since most developers are familiar with it. However, they argue that Postgres falls short in terms of quality. They describe issues with index size predictability, index creation resource intensity, metadata filtering performance, and cost.

This is a response to Pinecone's blog post, where we show that Postgres outperforms Pinecone in the same benchmarks with a few additional tweaks. We show that with just 20 lines of additional code, Postgres with the pgvector or lantern extension outperforms Pinecone by reaching 90% recall (compared to Pinecone's 60%) with under 200ms p95 latency.

Read Postgres vs. Pinecone

7/3/2024

Steven Wang writes:

Many in the data space are now aware of Iceberg and its powerful features that bring database-like functionality to files stored in the likes of S3 or GCS. But Iceberg is just one piece of the puzzle when it comes to transforming files in a data lake into a Lakehouse capable of analytical and ML workloads. Along with Iceberg, which is primarily a table format, a query engine is also required to run queries against the tables and schemas managed by Iceberg. In this post we explore some of the query engines available to those looking to build a data stack around Iceberg: Snowflake, Spark, Trino, and DuckDB.

...

DuckDB + Iceberg Example

We will be loading 12 months of NYC yellow cab trip data (April 2023 - April 2024) into Iceberg tables and demonstrating how DuckDB can query these tables.

Read Comparing Iceberg Query Engines

Isaac Harris-Holt writes:

Modern software engineering can be a bit of a nightmare. You've got all these different third-party services to keep track of, and often many first-party microservices too! As it turns out, things can be made more simple using Gleam. Let me show you how.

Louis Pilfold, the creator of the Gleam Programming Language, has been maintaining a Youtube Playlist of Gleam videos.

Mar 8, 2024 | Hakampreet Singh Pandher writes:

Yelp relies heavily on streaming to synchronize enormous volumes of data in real time. This is facilitated by Yelp’s underlying data pipeline infrastructure, which manages the real-time flow of millions of messages originating from a plethora of services. This blog post covers how we leverage Yelp’s extensive streaming infrastructure to build robust data abstractions for our offline and streaming data consumers. We will use Yelp’s Business Properties ecosystem (explained in the upcoming sections) as an example.

Read Building data abstractions with streaming at Yelp

cross-posted from: https://programming.dev/post/9979951

Monday, February 12, 2024
Ribbon and Ron Williams write:

As Redox functionality becomes more complete, we have been working hard to get a wide variety of software working.

This post will cover our porting strategy for Linux/BSD programs.

We have ported the following games and emulators:

2048
ClassiCube
DevilutionX
DOSBox
eduke32
FreeCiv
Gigalomania
Hematite
Mednafen
Neverball
OpenJK
OpenTTD
PrBoom (Doom engine)
ScummVM
Space Cadet Pinball
and others.

Porting is a major part of the Redox development effort. We are using porting as a way to prioritize and validate Redox functionality.

Currently dozens of programs and many more libraries work. Our initial focus has been on porting Rust programs, but we also recognize the importance of supporting programs written in other languages.

In last year Ribbon began the porting of more than 1000 programs and libraries to Redox! They are still work-in-progress and many require customized cross-compilation scripts or improved library support. You can see them here.

With our recent change to a Linux-compatible path format, we have removed a major hurdle to supporting Linux applications. In the future we plan to expand our POSIX support, port more Rust crates and continue to improve Relibc.

Some thought is being given to virtual machines and Wine as possible mechanisms for running proprietary binaries and possibly even proprietary drivers. However, there are no specific plans for that capability at this time.

Read Porting Strategy - Redox - Your Next(Gen) OS

December 28 2023 Pankaj Singh writes:

In big data processing and analytics, choosing the right tool is paramount for efficiently extracting meaningful insights from vast datasets. Two popular frameworks that have gained significant traction in the industry are Apache Spark and Presto. Both are designed to handle large-scale data processing efficiently, yet they have distinct features and use cases. As organizations grapple with the complexities of handling massive volumes of data, a comprehensive understanding of Spark and Presto’s nuances and distinctive features becomes essential. In this article, we will compare Spark vs Presto, exploring their performance and scalability, data processing capabilities, ecosystem, integration, and use cases and applications.

Read Spark vs Presto: A Comprehensive Comparison

Since there's only one mod here and they are also an Admin. I'd like to volunteer to moderate this community.

Karl W. Broman & Kara H. Woo write:

Spreadsheets are widely used software tools for data entry, storage, analysis, and visualization. Focusing on the data entry and storage aspects, this article offers practical recommendations for organizing spreadsheet data to reduce errors and ease later analyses. The basic principles are: be consistent, write dates like YYYY-MM-DD, do not leave any cells empty, put just one thing in a cell, organize the data as a single rectangle (with subjects as rows and variables as columns, and with a single header row), create a data dictionary, do not include calculations in the raw data files, do not use font color or highlighting as data, choose good names for things, make backups, use data validation to avoid data entry errors, and save the data in plain text files.

Read Data Organization in Spreadsheets

This article is weird in that it appears to be written for an audience that would find its contents irrelevant, but it has great information for people that are trying to reduce or eliminate their use of spreadsheets.

cross-posted from: https://programming.dev/post/8098358

KDE set up their own Lemmy instance and has an active KDE community [relative link]

Paolo Amoroso (@amoroso@fosstodon.org) writes:

This paper on the early standardization of Common Lisp (full text) is very interesting despite having been published in 1982, even prior to CLtL1.

By tracing the history of Lisp the paper highlights how the two major dialects at the time, MacLisp and Interlisp, influenced the design of Common Lisp. And it explains why the former was the major influence and little of the latter ended up in the new standard.