Last Update 6:44 AM May 26, 2024 (UTC)

Identity Blog Catcher

Brought to you by Identity Woman and Infominer.
Support this collaboration on Patreon!!!

Sunday, 26. May 2024

John Philpin : Lifestream

📸 🌊 🚧 Definitely a choice today … this one, because it is a

📸 🌊 🚧 Definitely a choice today … this one, because it is a little different the the others to date.

📸 🌊 🚧 Definitely a choice today … this one, because it is a little different the the others to date.

Saturday, 25. May 2024

John Philpin : Lifestream

🔗 😂 Impeach Nixon! - a lesson from history.

🔗 😂 Impeach Nixon! - a lesson from history.

🔗 😂 Impeach Nixon! - a lesson from history.


🎈145/366 The ‘Tiny Pod’ 🚧 is indeed a thing to 🔗 behold -

🎈145/366 The ‘Tiny Pod’ 🚧 is indeed a thing to 🔗 behold - for those of you that have a watch but don’t want to wear it on your wrist?

🎈145/366 The ‘Tiny Pod’ 🚧

is indeed a thing to 🔗 behold - for those of you that have a watch but don’t want to wear it on your wrist?


Ben Werdmüller

Nostr Journalism Accelerator

Nos is running a "journalism accelerator", which onboards independent journalists and publications onto Nostr with guaranteed promotion and 1:1 help. Nostr is a different kind of open network, in the sense that it's decentralized rather than federated. Famously, Jack Dorsey defected there from Bluesky, in part because Bluesky started offering service-level features like com

Nos is running a "journalism accelerator", which onboards independent journalists and publications onto Nostr with guaranteed promotion and 1:1 help.

Nostr is a different kind of open network, in the sense that it's decentralized rather than federated. Famously, Jack Dorsey defected there from Bluesky, in part because Bluesky started offering service-level features like community moderation rather than just focusing on the protocol. It's also much more closely tied to crypto communities than either the fediverse or Bluesky.

I'm curious about the kinds of journalists who might sign up for this. I spotted The Conversation there while I was nosing around, but I haven't found any other publishers I recognized; the network really is very open to build on, so I wonder if more might follow - and if they skew in any particular direction. #Technology

[Link]


Why publishers are preparing to federate their sites

"At least two digital media companies are exploring the fediverse as a way to take more control over their referral traffic and onsite audience engagement." The Verge and 404 Media will both support ActivityPub (a protocol, not a "plugin", as the article calls it) and plug into the fediverse. This dovetails with what I've been talking about for some time: "Instead of sp

"At least two digital media companies are exploring the fediverse as a way to take more control over their referral traffic and onsite audience engagement."

The Verge and 404 Media will both support ActivityPub (a protocol, not a "plugin", as the article calls it) and plug into the fediverse.

This dovetails with what I've been talking about for some time: "Instead of spending time building a presence on other platforms for their benefit, a publisher can do that on their own sites — while giving readers the ability to see those posts on other federated platforms." And while the fediverse is still in an early, growing stage, it's worth taking a bet on.

As Flipboard's Mike McCue says further into the piece, "What The Verge is doing is definitely pioneering the future for media." I'm hopeful that more publishers follow suit - with this and other experiments that have the potential to help them build more direct first-party relationships with their audiences. #Fediverse

[Link]


Simon Willison

Why Google’s AI might recommend you mix glue into your pizza

Why Google’s AI might recommend you mix glue into your pizza I got “distrust and verify” as advice on using LLMs into this Washington Post piece by Shira Ovide.

Why Google’s AI might recommend you mix glue into your pizza

I got “distrust and verify” as advice on using LLMs into this Washington Post piece by Shira Ovide.

John Philpin : Lifestream

🖇️📸 The series - I guess it is now - is building nicely

🖇️📸 The series - I guess it is now - is building nicely

I know it’s 🔗 here - but I have to look it up every time. Of

I know it’s 🔗 here - but I have to look it up every time. Of course ‘every’ shouldn’t be too many - but sadly - not the case.

I know it’s 🔗 here - but I have to look it up every time. Of course ‘every’ shouldn’t be too many - but sadly - not the case.


📸 🌊 🚧 Just Another Day

📸 🌊 🚧 Just Another Day

📸 🌊 🚧 Just Another Day

Friday, 24. May 2024

John Philpin : Lifestream

🎈144/366 Today Is My Anniversary 🚧

It’s 🔗 Towel Day ( thank you @odd ) - but today also happens to be my 25th Anniversary Yup - 25 years. Truth be told it had been going on a lot longer - but it was 25 years ago that we actually tied the knot .. and I have had my ‘personal space’ on the internet ever since May 25th, 1999 (Creation Date: 1999-05-25T20:14:36Z). Now of course, we had known each other a lot longer than that - long

It’s 🔗 Towel Day ( thank you @odd ) - but today also happens to be my 25th Anniversary

Yup - 25 years. Truth be told it had been going on a lot longer - but it was 25 years ago that we actually tied the knot .. and I have had my ‘personal space’ on the internet ever since May 25th, 1999 (Creation Date: 1999-05-25T20:14:36Z).

Now of course, we had known each other a lot longer than that - long before the web in fact, but you know when you first connect you don’t really note the dates do you? It’s like a permanent honeymoon. Hours merge into days into weeks into months … who really knows when ‘it’ really all started?

I can tell you that I first got ‘involved’ with the internet in 1985. I mean it wasn’t what we know today. My own internet experience in those days was email around our international offices .. but it was the internet (ok, ok - intranet - there’s a word you don’t hear much these days.)

Although Google Trends suggests I am wrong …

.. again!

Anyway, I do remember being annoyed that my messages wouldn’t go out to the people I wanted to talk to - you know customers, partners, vendors - all the people a business should be talking to. .. we could only talk amongst ourselves. I mean the Cerne account team could connect to their customer .. why did I have to use Compusurf (sic)? … but that’s a different story.

( .. and no I didn’t do AOL, Prodigy, Compuserve .. they were never for me - maybe even back then I had the innate desire to avoid being locked into a silo.)

Anyway, that’s it … 25 Years with my 🔗 own domain

Onwards.

How will I celebrate? Why - taking my towel down to the beach of course.


No need to bring you up to speed about that google short cod

No need to bring you up to speed about that google short code to bypass Google’s world of enshitification - except to say 🔗 Ernie at Tedium built a web site to make it ‘easy for all’

No need to bring you up to speed about that google short code to bypass Google’s world of enshitification - except to say 🔗 Ernie at Tedium built a web site to make it ‘easy for all’


I stumbled across this today as I was collecting 🖇️ these. I

I stumbled across this today as I was collecting 🖇️ these. I nearly moved through because ‘Kelly’ - but I am a sucker for Jeff, so … Holy Crap …🔗 🎵 📼 Jeff Daniels .. I was under the impression he had only recently written it - but this from 13 years ago - suggests otherwise. 🔗 🎵 📼 Jeff Daniels. If you are reading this post on my site, the inline video gives an error. Click on the link

I stumbled across this today as I was collecting 🖇️ these. I nearly moved through because ‘Kelly’ - but I am a sucker for Jeff, so …

Holy Crap …🔗 🎵 📼 Jeff Daniels

.. I was under the impression he had only recently written it - but this from 13 years ago - suggests otherwise.

🔗 🎵 📼 Jeff Daniels.

If you are reading this post on my site, the inline video gives an error. Click on the link above. The 'error' is only because the creator wants you to got to YouTube to watch it.

 

… another two ‘broken links’ .. has something changed?


In no particular order (except for perhaps the first two) …

In no particular order (except for perhaps the first two) … 🔗 🎵 📼 Nina Simone 🔗 🎵 📼 The Animals 🔗 🎵 📼 Jac Ross 🔗 🎵 📼 Cat Stevens 🔗 🎵 📼 Elvis Costello 🔗 🎵 📼 Joe Cocker 🔗 🎵 📼 Cyndi Lauper 🔗 🎵 📼 Santa Esmeralda If you are reading this post on my site, the inline video gives an error. Click on the link above. The 'error' is only because the creator wants you to got

In no particular order (except for perhaps the first two) …

🔗 🎵 📼 Nina Simone

🔗 🎵 📼 The Animals

🔗 🎵 📼 Jac Ross

🔗 🎵 📼 Cat Stevens

🔗 🎵 📼 Elvis Costello

🔗 🎵 📼 Joe Cocker

🔗 🎵 📼 Cyndi Lauper

🔗 🎵 📼 Santa Esmeralda

If you are reading this post on my site, the inline video gives an error. Click on the link above. The 'error' is only because the creator wants you to got to YouTube to watch it.

 

We all get that broken link message from time to time don’t we? What is particularly interesting on this post is to find EIGHT blocks - I wonder if it is the writers insisting on the block - I don’t think there I any other constant.


Ben Werdmüller

The Fatal Flaw in Publishers' OpenAI Deals

"It’s simply too early to get into bed with the companies that trained their models on professional content without permission and have no compelling case for how they will help build the news business." This piece ends on the most important point: nobody is coming to save the news industry, and certainly not the AI vendors. Software companies don't care about news. They do

"It’s simply too early to get into bed with the companies that trained their models on professional content without permission and have no compelling case for how they will help build the news business."

This piece ends on the most important point: nobody is coming to save the news industry, and certainly not the AI vendors. Software companies don't care about news. They don't think your content is more valuable because it's fact-checked and edited. They don't have a vested interest in ensuring you survive. They just want the training data - all of it, in order to build what they consider to be the best product possible. Everything else is irrelevant. #AI

[Link]


The AI data goldmine

If I was a nefarious software company, here’s how I might be thinking: AI functionality tends to require that data is sent to a centralized service for processing. This is often data that is not being shared online in any other way that is easily available for analysis: existential work questions, internal documents, and so on. This makes it very valuable data to sell to brokers or to use in

If I was a nefarious software company, here’s how I might be thinking:

AI functionality tends to require that data is sent to a centralized service for processing.

This is often data that is not being shared online in any other way that is easily available for analysis: existential work questions, internal documents, and so on.

This makes it very valuable data to sell to brokers or to use in targeting advertising.

So, let’s add lots of AI functionality to our services to encourage people to share that data with us.

We’ll provide AI services.

We’ll mine the data that is provided to us when people use those AI services.

And then we’ll sell it.

The AI revolution is also the private data sharing revolution. It’s worth asking: does this AI feature I’m interested in using puncture a privacy hole in the service it is a part of? Who am I sharing this information with? What will they do with it?


Unoffice Hours

I’m enamored with Matt Webb’s unoffice hours: a way to chat with him about anything, without needing to email him first, for 30 minutes. As Matt says: I loved those open conversations over coffee in the Before Times. There’s an ostensible reason to connect, so you talk about work, or compare notes about an idea, or whatever. But then the unexpected emerges. (Sometimes you have to hunt for i

I’m enamored with Matt Webb’s unoffice hours: a way to chat with him about anything, without needing to email him first, for 30 minutes.

As Matt says:

I loved those open conversations over coffee in the Before Times. There’s an ostensible reason to connect, so you talk about work, or compare notes about an idea, or whatever. But then the unexpected emerges. (Sometimes you have to hunt for it.) There are things in your head that you only know are there when you say them. And there are encounters with new ideas and new perspectives.

Exactly. So let’s do it.

Introducing my own Unoffice Hours: I’ve set aside a little time on Fridays to connect about anything.

Here are some topics that might be interesting to chat about:

Feedback on a project you’re working on (startups, software, a writing project) Following up on something I’ve written in this space Product and technology strategy in the public interest (news, education, libraries, other mission-driven organizations) The indie web Fostering a collaborative organizational culture Saying hello

Matt calls the effect “manufactured serendipity”; I call it “intentional serendipity,” but the intent is the same. It’s good to chat and meet people, and you never know where it will lead.

To book a 30-minute chat, click here.


Simon Willison

Golden Gate Claude

Golden Gate Claude This is absurdly fun and weird. Anthropic's recent LLM interpretability research gave them the ability to locate features within the opaque blob of their Sonnet model and boost the weight of those features during inference. For a limited time only they're serving a "Golden Gate Claude" model which has the feature for the Golden Gate Bridge boosted. No matter what question you

Golden Gate Claude

This is absurdly fun and weird. Anthropic's recent LLM interpretability research gave them the ability to locate features within the opaque blob of their Sonnet model and boost the weight of those features during inference.

For a limited time only they're serving a "Golden Gate Claude" model which has the feature for the Golden Gate Bridge boosted. No matter what question you ask it the Golden Gate Bridge is likely to be involved in the answer in some way. Click the little bridge icon in the Claude UI to give it a go.

I asked for names for a pet pelican and the first one it offered was this:

Golden Gate - This iconic bridge name would be a fitting moniker for the pelican with its striking orange color and beautiful suspension cables.

And from a recipe for chocolate covered pretzels:

Gently wipe any fog away and pour the warm chocolate mixture over the bridge/brick combination. Allow to air dry, and the bridge will remain accessible for pedestrians to walk along it.

UPDATE: I think the experimental model is no longer available, approximately 24 hours after release. We'll miss you, Golden Gate Claude.


Nilay Patel reports a hallucinated ChatGPT summary of his own article

Nilay Patel reports a hallucinated ChatGPT summary of his own article Here's a ChatGPT bug that's a new twist on the old issue where it would hallucinate the contents of a web page based on the URL. The Verge editor Nilay Patel asked for a summary of one of his own articles, pasting in the URL. ChatGPT 4o replied with an entirely invented summary full of hallucinated details. It turns out T

Nilay Patel reports a hallucinated ChatGPT summary of his own article

Here's a ChatGPT bug that's a new twist on the old issue where it would hallucinate the contents of a web page based on the URL.

The Verge editor Nilay Patel asked for a summary of one of his own articles, pasting in the URL.

ChatGPT 4o replied with an entirely invented summary full of hallucinated details.

It turns out The Verge blocks ChatGPT's browse mode from accessing their site in their robots.txt:

User-agent: ChatGPT-User Disallow: /

Clearly ChatGPT should reply that it is unable to access the provided URL, rather than inventing a response that guesses at the contents!

Via Gemini is the new Google+


Quoting Scott Jenson

I just left Google last month. The "AI Projects" I was working on were poorly motivated and driven by this panic that as long as it had "AI" in it, it would be great. This myopia is NOT something driven by a user need. It is a stone cold panic that they are getting left behind. The vision is that there will be a Tony Stark like Jarvis assistant in your phone that locks you into their ecosystem s

I just left Google last month. The "AI Projects" I was working on were poorly motivated and driven by this panic that as long as it had "AI" in it, it would be great. This myopia is NOT something driven by a user need. It is a stone cold panic that they are getting left behind.

The vision is that there will be a Tony Stark like Jarvis assistant in your phone that locks you into their ecosystem so hard that you'll never leave. That vision is pure catnip. The fear is that they can't afford to let someone else get there first.

Scott Jenson


Quoting Nivia Henry

The leader of a team - especially a senior one - is rarely ever the smartest, the most expert or even the most experienced. Often it’s the person who can best understand individuals’ motivations and galvanize them towards an outcome, all while helping them stay cohesive. — Nivia Henry

The leader of a team - especially a senior one - is rarely ever the smartest, the most expert or even the most experienced.

Often it’s the person who can best understand individuals’ motivations and galvanize them towards an outcome, all while helping them stay cohesive.

Nivia Henry


Some goofy results from ‘AI Overviews’ in Google Search

Some goofy results from ‘AI Overviews’ in Google Search John Gruber collects two of the best examples of Google’s new AI overviews going horribly wrong. Gullibility is a fundamental trait of all LLMs, and Google’s new feature apparently doesn’t know not to parrot ideas it picked up from articles in the Onion, or jokes from Reddit. I’ve heard that LLM providers internally talk about “screens

Some goofy results from ‘AI Overviews’ in Google Search

John Gruber collects two of the best examples of Google’s new AI overviews going horribly wrong.

Gullibility is a fundamental trait of all LLMs, and Google’s new feature apparently doesn’t know not to parrot ideas it picked up from articles in the Onion, or jokes from Reddit.

I’ve heard that LLM providers internally talk about “screenshot attacks”—bugs where the biggest risk is that someone will take an embarrassing screenshot.

In Google search’s case this class of bug feels like a significant reputational threat.


John Philpin : Lifestream

About content. 🔗 📼 😂 Stewart Lee on Social Media (Content

About content. 🔗 📼 😂 Stewart Lee on Social Media (Content Provider) Don’t worry, it’s not my usual rant. Just a very funny piece of standup that needs to be watched more than it has.

About content.

🔗 📼 😂 Stewart Lee on Social Media (Content Provider)

Don’t worry, it’s not my usual rant. Just a very funny piece of standup that needs to be watched more than it has.


Back Of The Boat

Back Of The Boat Your browser does not support the video tag.
Back Of The Boat

Your browser does not support the video tag.

💬

💬

💬


📸 🌊 🚧 No ‘drama’ this morning, so I just let the sun move up

📸 🌊 🚧 No ‘drama’ this morning, so I just let the sun move up for a while …

📸 🌊 🚧 No ‘drama’ this morning, so I just let the sun move up for a while …


Simon Willison

Quoting Molly White

But increasingly, I’m worried that attempts to crack down on the cryptocurrency industry — scummy though it may be — may result in overall weakening of financial privacy, and may hurt vulnerable people the most. As they say, “hard cases make bad law”. — Molly White

But increasingly, I’m worried that attempts to crack down on the cryptocurrency industry — scummy though it may be — may result in overall weakening of financial privacy, and may hurt vulnerable people the most. As they say, “hard cases make bad law”.

Molly White


A Grand Unified Theory of the AI Hype Cycle

A Grand Unified Theory of the AI Hype Cycle Glyph outlines the pattern of every AI hype cycle since the 1960s: a new, novel mechanism is discovered and named. People get excited, and non-practitioners start hyping it as the path to true “AI”. It eventually becomes apparent that this is not the case, even while practitioners quietly incorporate this new technology into useful applications while do

A Grand Unified Theory of the AI Hype Cycle

Glyph outlines the pattern of every AI hype cycle since the 1960s: a new, novel mechanism is discovered and named. People get excited, and non-practitioners start hyping it as the path to true “AI”. It eventually becomes apparent that this is not the case, even while practitioners quietly incorporate this new technology into useful applications while downplaying the “AI” branding. A new mechanism is discovered and the cycle repeats.

Thursday, 23. May 2024

John Philpin : Lifestream

Saw 🔗 this .. (sorry ..Facebook) .. thought of @Ron

Saw 🔗 this .. (sorry ..Facebook) .. thought of @Ron

Saw 🔗 this ..

(sorry ..Facebook) .. thought of @Ron


The Pragmatic Engineer

The Pulse #94: OpenAI’s ethics crisis

Claims of predatory stock clawback clause and unethical use of an actor’s voice plague leading AI startup. Also: Microsoft’s urgent focus on security.

The Pulse is a series covering insights, patterns, and trends within Big Tech and startups. Notice an interesting event or trend? Send me a message.

Today, we cover:

Industry pulse. Replit pivots from “hobbyist coders” to enterprise, Microsoft takes on Apple with ARM64 laptop, Vercel’s late-stage funding round, GitHub Copilot extensions, and more.

OpenAI’s ethics crisis. Déjà vu: six months after OpenAI’s board booted out Sam Altman as CEO for “not being consistently candid in his communications,” Altman (since restored to his old role) faces new but familiar allegations. This time, he insists he knew nothing of predatory vested stock clawback terms, or that the demo voice of ChatGPT-4o sounds almost identical to Scarlett Johansson in the movie, ‘Her.’

Microsoft’s urgent security focus. After a string of embarrassing, high-profile security incidents, Microsoft is making security a top priority and putting its money where its mouth is, by linking executives’ compensation to security milestones. Tech companies might be wise to consider taking security as seriously as Microsoft.

1. Industry pulse Replit pivots from consumer to enterprise focus

Replit is a software development and deployment platform offering an online integrated development experience (IDE.) Founded in 2016, the company raised $222M in funding and was valued just above $1B in early 2023. In 2021, Replit’s vision was to bring the “next billion” software creators online. During the 2023 fundraising, the stated focus was to expand cloud services and AI development.

There is a lot to like about Replit in the boldness of its vision, the generous free plan for developers, and a friendly pricing model. They also treat employees well; in November 2023 the company offered a total of $20M in liquidity to current and former employees.

Unfortunately, last week the company announced letting go 20% of staff (30 people) and that it is switching focus to enterprise customers: not the “next billion software creators,” but “knowledge workers” and the “Fortune 500.”

To me, this makes sense. The company has built neat AI capabilities – a tool called Replit AI – but AI is expensive to build and run. In order to have any shot at turning a profit, Replit needs professionals to pay more. There is little point trying to reach a billion or more developers if the company’s future is uncertain. Should Replit succeed in building a viable business from enterprise customers, I’m sure they will consider re-focusing on the “software creator” market.

Anecdotally, I’ve heard about the usage of Replit by startups and scaleups. A scaleup CEO told me they pay for Replit so they can hack together things; sometimes, when the engineering team says that building a prototype will take weeks to complete, the CEO spins up a Replit service, and builds a prototype of the project in a few hours which gets the point across. This CEO tells me they can’t roll out Replit to their company because it’s a “hobbyist product” without sufficient enterprise features. Well, this seems to be exactly what Replit is changing!

I expect more consumer-focused AI startups to pivot into enterprise. A takeaway from Replit is that AI-powered products are a hard sell for consumers, who are unlikely to pay more for them. For the economics to work, enterprise customers are needed, otherwise big trouble awaits down the line. Replit is making this pivot when it’s in a strong financial position. Good luck to the team in executing this shift!

Microsoft takes on Apple with ARM64 Surface Laptop

Read more


Simon Willison

Quoting Will Larson

The most effective mechanism I’ve found for rolling out No Wrong Door is initiating three-way conversations when asked questions. If someone direct messages me a question, then I will start a thread with the question asker, myself, and the person I believe is the correct recipient for the question. This is particularly effective because it’s a viral approach: rolling out No Wrong Door just requir

The most effective mechanism I’ve found for rolling out No Wrong Door is initiating three-way conversations when asked questions. If someone direct messages me a question, then I will start a thread with the question asker, myself, and the person I believe is the correct recipient for the question. This is particularly effective because it’s a viral approach: rolling out No Wrong Door just requires any one of the three participants to adopt the approach.

Will Larson


Werdmüller on Medium

Dispatches from the media apocalypse

In saving news, culture is key. Continue reading on Medium »

In saving news, culture is key.

Continue reading on Medium »


Ben Werdmüller

Dispatches from the media apocalypse

Without serious intervention, newsrooms are going to disappear. Changes to social media and the advent of generative AI threaten their businesses and the impact of their work. They need to own their online presences outright and build direct relationships with their communities—and they need to do it now. Social media audiences are plummeting. Less than 35% of internet searches lead users to

Without serious intervention, newsrooms are going to disappear. Changes to social media and the advent of generative AI threaten their businesses and the impact of their work. They need to own their online presences outright and build direct relationships with their communities—and they need to do it now.

Social media audiences are plummeting. Less than 35% of internet searches lead users to click on a website. The views and engagement that newsrooms depend upon to survive are disappearing.

It’s happening quickly. Semafor’s Max Tani reported recently:

Washington Post CEO Will Lewis is introing the paper’s new “Build It” plan today. In a meeting with staff, he noted that the paper lost $77 million over the past year, and saw a 50% drop off in audience since 2020: “To be direct, we are in a hole, and we have been for some time."

Addressing this challenge will require radical changes to how newsrooms invest in and build technology.

In this post, I’ll attempt to describe the challenges in more detail and then discuss how they can be more adequately addressed.

Some context: my move into news

I’ve recently gained a new perspective on these challenges. For over a decade, I’ve worked adjacent to news and journalism. I’ve seen the industry as an engineer, startup founder, product lead, investor, and advisor. More recently, I decided I could be more useful in directly leading technology efforts inside newsrooms. It’s been eye-opening, rewarding work.

My experience alongside news was diverse. I built product for newsrooms, founded a startup used by public media, invested in early stage media startups, and have taught human-centered product design to teams at organizations like the New York Times and the Associated Press, as well as at institutions like the Newmark School of Journalism and the Harvard Kennedy School of Government. I’ve built software, founded, grown, and supported startups, and taught product design to some of the biggest names in journalism.

My immersion inside newsrooms has been much more recent. ProPublica investigates abuses of the public trust by government, businesses, and other institutions. I’ve worked on technology strategy for the last year, first as a contractor, and now as its Senior Director of Technology. Before that, I was the first CTO at The 19th, which reports on the intersection of gender, politics, and power.

I made this career shift at a pivotal moment for journalism—though it seems every moment for journalism over the last fifteen years has felt pivotal. The industry has struggled to weather the seismic shifts brought about by the internet, which have impacted its business, the state of our politics, and public discourse. It’s been a struggle for decades.

The audience threat

It’s getting harder and harder for newsrooms to reach their audiences — and for them to sustain themselves.

I’ve often remarked that journalism treats the internet as something that happened to it rather than something it can actively shape and build, but it at least had some time to adjust to its new normal. The internet landscape has been largely static for well over a decade — roughly from the introduction of the iPhone 3G to Twitter’s acquisition by Elon Musk. People used more or less the same services; they accessed the internet more or less the same way. Publications and online services came and went, but the laws of physics of the web were essentially constants.

Over the last year in particular, that’s all changed. Shifts in the social media landscape and the growing popularity and prevalence of generative AI have meant that the rules that newsrooms began to rely on no longer hold.

At their heart, online newsrooms have a reasonably simple funnel. They publish journalism, which finds an audience, some of which either decide to pay for it or view ads that theoretically cover the cost of the work. Hopefully, they will make enough money to publish more journalism.

This description is a little reductive: there are lots of different revenue models in play, for one thing. I’m particularly enamored with patronage models that allow those with the means to support open-access journalism for anyone to read freely. Still, some are entirely ad-supported, some are sponsored, and others are protected behind a paywall (or some combination of the above). For another, journalism isn’t always the sole driver of subscriptions. The New York Times receives tens of millions of subscribers from its games like Wordle and Connections, as well as its Cooking app.

Still, there are two pivotal facts for every newsroom: their work must reach an audience, and someone must pay for it. The first is a prerequisite of the second: if nobody discovers the journalism, nobody will pay for it. So, reaching and growing an audience is crucial.

For the last decade and a half, newsrooms have used social media and search engines as the primary way to reach people. People share stories across social media—particularly Facebook and Twitter—and search for topics they’re interested in. It’s generally worked.

Over the last year, social media has radically fragmented. Twitter transformed into X under its new management; users began to flee the platform in the face of more toxic discourse, and active use plummeted. Facebook is slowly declining and referrals to news sites have fallen by 50% over the last year. Instagram is not in decline. Still, it’s harder to post links to external sites there, which means that while newsrooms can reach users, they have more difficulty converting them to subscribers.

On top of these changes, we’ve also seen the rise of Threads, Mastodon, and Bluesky, as well as a long tail of other social apps, platforms, and forums on which to reach people. Audiences on social media used to be found in a very small number of places and are now spread out across very different platforms. The fediverse and AT Protocol also yield different problems: which instance should a newsroom choose to make its home? How can it measure engagement in what it posts in a decentralized system so that it knows what’s working and where it should continue to invest its meager resources?

Much has been written about newsrooms’ inability to move away from X even as it has become a hotbed of white supremacy and far-right rhetoric. The honest truth is that it still drives significant traffic to their websites, and in an environment where traffic referrals are dropping overall, intentionally further deepening the traffic shortfall is understandably not a career risk newsroom leaders are willing to make.

Social media isn’t the only way newsrooms are finding it harder to find an audience. Even search engines, long the stalwarts of the web, are moving away from referring traffic.

As search engines move to make AI-driven answers more prominent than links to external websites, they threaten to reduce newsroom audiences, too. More than 65% of Google searches already ended without a click to an external site. Now, it’s planning to roll out AI-driven answers to over a billion people. It’s not that other links are going away entirely. Still, because AI answers are the most prominent information on the page, clickthroughs to the external websites where the answers were found initially will be significantly reduced.

A similar dynamic is at play with the rise of AI services like ChatGPT, emerging as stiff competition for search engines like Google. These services answer questions definitively (although not always correctly), usually with no external links on the page. ChatGPT could learn from a newsroom’s articles and display information gleaned from an expensive investigative story while never revealing its source or allowing readers to support the journalism.

Generative AI models seem like magic: they answer questions succinctly, in natural language, based on prompts that look a lot like talking to a real human being. They work by training a neural network on a vast corpus of information, often obtained by crawling the web. Based on these enormous piles of data, AI engines answer questions by predicting which word should come next: a magic trick of statistics empowered by something close to the sum total of human knowledge.

That’s not hyperbole. It’s not a stretch to say that OpenAI’s ChatGPT and Google’s Gemini were trained on most of the internet, including websites, published books, videos, articles, art, science, and research. They couldn’t function without this data — but, ironically, they rarely credit their sources for any of it. Users see the benefit of fast answers; the sources of that information are starved of oxygen.

We’re at the foothills of both changes: social media is likely to fragment further, and generative AI will become even more prevalent as it becomes more powerful. Newsrooms can no longer rely on their old tactics to reach their audiences, and they will need to build new tactics that take these trends into account if they hope to survive.

Some models are more resilient than others

The 19th’s Alexandra Smith recently wrote about the state of play in Columbia Journalism Review:

In our current reality, journalism exists in various formats splintered across platforms and products. People are just as likely to get their news on Instagram as from a news website. It no longer makes sense to rely primarily on measuring readership by traditional website metrics.

This is a depressing fact if you rely on paywalled subscriptions or ad impressions. Nobody’s looking at your ads if they’re consuming your journalism off-platform, and how can you possibly get someone to subscribe if they never touch your app or website? Instagram and TikTok don’t have built-in subscriptions.

Over the years, many people have suggested micropayments — tiny payments you make every time you read a news article anywhere — but this depends on everyone on the web having some kind of micropayment account that is on and funded by default and the platforms all participating. It’s a reasonable idea if the conditions are right, but the conditions will never be right — and, like subscription models, it shuts out people who can’t pay, who are often the people most in need of public service journalism to begin with.

For newsrooms like The 19th, the picture is much rosier: like most non-profit newsrooms, it depends on donors who support it based on its journalistic impact. (The same is true of ProPublica, my employer.) That impact could occur anywhere, on any platform; the trick is to measure it so donors can be informed. Alexandra developed a new metric, Total Journalism Reach, that captures precisely this:

Right now, it includes website views; views of our stories that are republished on other news sites and aggregation apps, like Apple News; views of our newsletters based on how many emails we send and their average open rates, reduced for inflation since Apple implemented a new privacy feature; event attendees; video views; podcast listens; and Instagram post views.

This is clearly valuable work that will help newsrooms like The 19th prove their impact to current and potential donors. The quote above doubles as a useful example of the places The 19th is reaching its audience.

It’s worth considering how these might change over time. Some of the media Alexandra describes are inside The 19th’s control, and some are less so.

Supplier power

In his classic piece ‌How Competitive Forces Shape Strategy, Michael Porter described five forces that shape competitive strategy. One of them is supplier power: the ability of providers of essential inputs to a business to exert influence over the organization. If suppliers to the industry have too much power — because there are few alternatives, for example — they can effectively force the company’s strategy by raising costs or enforcing adverse policies.

Newsrooms’ platforms for reaching their audiences, such as social media and Apple News, currently have outsized supplier power over the journalism industry. As a result, the industry is disproportionately vulnerable to the effects of business decisions made by the owners of those platforms.

In April, Instagram introduced a new automatic filter, switched on by default, to remove political content, which affected many newsrooms, and illustrates the kind of changes service providers can make on a whim.

Newsrooms on Apple News tend to see a multiple of the number of reads they see on their websites, but Apple could pull the product tomorrow. Even today, the number of views you get highly depends on which stories the Apple News team chooses to highlight. Ads in publications on Apple News need to use Apple’s ad network. It’s a closed shop. Apple News is only successful because it comes installed by default on Apple devices; hundreds of similar news aggregators have all failed to survive in their own right. It’s a precarious place to hang your hat.

We’ve already discussed the impact of search engine design decisions like prioritizing AI over click-through rates. Only one search engine is prominent enough to have disproportionate supplier power: a position Google has bought by spending over $21 billion a year to be the default search engine in every web browser.

However, not all conduits to readers have this outsized supplier power as a feature. Social media platforms, search engines, and news aggregators are all run by wealthy individual companies like X, Meta, Google, and Apple, who have the potential to exert their power. If you choose to leave them for any reason, you’re also leaving behind the relationships you’ve built up with your audience there: there’s no audience portability.

In contrast, email, podcasts (real podcasts, not the single-platform kind where you ink an exclusive deal with Spotify or Audible), and the web are well-used methods to reach audiences that aren’t owned by any platform. There are certainly market leaders for each communication type. Still, each is based on an open protocol that no single company controls — which means, for those methods, no supplier can exert adverse supplier power. If one service provider misbehaves, you can simply switch to another without losing functionality. You can bring your audience with you. They’re safer methods, as long as enough readers want to be reached in those ways.

That’s why so many publications have focused their strategies on their email newsletters. Everyone already has an email address, and (barring technical difficulties) if a publisher sends a subscriber a message, they’re guaranteed to receive it. Moreover, people engaged enough to hit the “subscribe” button are far more likely to convert to donors or upgrade to a paid subscription.

Newsletters, unfortunately, are also in decline. Open rates have fallen over the last decade; Gmail’s dominant position and aggressive filtering have made it harder for newsletters to be noticed; there’s more competition for attention. There aren’t any great ways for new readers to discover newsletters — those subscription pages are subject to the same internet traffic dynamics as articles. It’s getting harder and harder to direct new visitors to subscribe, which is why we see more overt “please subscribe” popup overlays on news sites. The focus has needfully shifted to converting more existing subscribers into donors or customers rather than widening the funnel and finding more newcomers.

Newsrooms need alternative media that allow them to make direct connections with their audiences. These media must be free from undue supplier power and have a large base of existing users that can be tapped into.

So what else is out there?

The answer is not much. Yet.

The innovation squeeze

Most non-profit newsrooms have tiny technology teams. The 19th, when I was CTO, had two engineers in addition to me; ProPublica has four. (Other interactive developers work on standalone stories but don’t address platform needs.) In contrast, I led a team of twenty-two engineers at the last startup I worked at, and we had over a hundred at Medium.

To bridge that gap, there is a small community of digital agencies that make supporting newsroom platform needs a core part of their business. Probably the most famous are Alley and Upstatement, but there are around a dozen more that are actively used by newsrooms.

They do beautiful work and are an excellent way for a newsroom to start strong with a modern brand and a well-functioning web platform. I strongly recommend that a new newsroom consults with them.

There is an emerging dynamic, though, where the technology vision for a newsroom is outsourced to the agencies. As we’ve discussed, a newsroom’s success and impact depend highly on core internet technologies like the web and email. Newsrooms quite reasonably spec and build a platform based on what will work well today. However, because the vision and expertise for harnessing the internet lie with the agencies, they don’t have any meaningful technology capability for innovating around what will work well tomorrow.

Newsrooms absolutely need to focus on today. That’s an obvious prerequisite: they must meet their audiences, subscribers, and donors where they’re at right now. However, they also need to be aware of what is coming down the road and prepared to experiment with, engage with, and potentially help shape new technologies that could impact their businesses in the future. If the internet changes, they need to be ready for it. To reference an overused Wayne Gretzky quote: you need to skate to where the puck will be, not where it is right now.

Nobody knows for certain where the puck will be. That means newsrooms need to make bets about the future of technology — which, in turn, means they must have the capacity to make bets about the future of technology.

Most newsrooms already have technical staff who maintain their websites, fix broken platform stacks, and build tools for the newsroom. These staff must also highlight future business risks and allow them to experiment with new platform opportunities. In a world where newsrooms rely on the internet as a publishing mechanism, technology expertise must be integral to their strategy discussions. And because technology changes so quickly and unpredictably, maintaining the time, space, and intellectual curiosity for experimentation is critical.

Nothing will work, but anything might

Experimentation doesn’t need to be resource-intensive or time-consuming. Alongside in-house expertise, the most important prerequisite is the willingness of a newsroom to test: to say “yes” to trying something out, but being clear about the parameters for success, and always rooting success or failure in a concrete understanding of their communities.

I’ve written before about how, if the fediverse is successful, it will be a powerful asset to media organizations that combines the direct relationship properties of email with the conversational and viral properties of social media. At the same time, there’s no doubt that the network is relatively small today, that the experience of using Mastodon falls short of corporate social networks like the Twitter everyone remembers, and that features like blocking referrer data makes life much harder for audience teams. There are lots of good reasons for a resource-strapped management team to say “no” to joining it.

At the same time, because it has the potential to be interesting, some newsrooms (including my employer) are experimenting with a presence. The ones who make the leap are often pleasantly surprised: engagement per capita is dramatically higher, particularly around social justice topics. Anecdotally, I discovered that posting a fundraising call to action to the network yielded more donations than from every other social network — combined.

It’s worth looking at Rest of World’s “More Ways to Read” page — a massive spread of platforms that runs the gamut from every social network to news apps, messaging platforms, audio, newsletters, and RSS feeds. The clear intention, taken seriously, is to meet audiences where they’re at, even if some of those networks have not yet emerged as a clear winner. All this from a tiny team.

However, experimenting isn’t just about social media. It’s worth experimenting with anything and everything, from push notifications to website redesigns that humanize journalists to new ways for communities to support the newsroom itself.

On the last point, I’m particularly enamored with how The 19th allows members to donate their time instead of money. Understanding that not everyone who cares about their mission has discretionary spending ability, they’re harnessing their community to create street teams of people who can help promote, develop, and share the work in other ways. It’s brilliant — and very clearly something that was arrived at through an experimental process.

I learned a formal process for human-centered experimentation as a founder at Matter, the accelerator for early-stage media startups, which changed the way I think about building products forever. A similarly powerful program is now taught as Columbia Journalism School’s Sulzberger Fellowship. If you can join a program like this, it’s well worth it, but consultants like Tiny Collaborative’s Tran Ha and Matter’s Corey Ford are also available to engage in other ways. And again, the most important prerequisites are in-house expertise and the willingness to say “yes”.

To achieve this, they must shift their cultures. The principles of experimentation, curiosity, and empathy that are the hallmarks of great journalism must also be applied to the platforms that power their publishing and fundraising activities. They must foster great ideas, wherever they come from, and be willing to try stuff. That inherently also implies building a culture of transparency and open communication in organizations that have, on average, underinvested in these areas. As Bo Hee Kim, then a Director of Newsroom Strategy at the New York Times, wrote back in 2020:

Companies will need to address broader issues with communication, access, and equity within the workplace. Leaders will need to believe that newsroom culture has a bigger impact on the journalism than they understood in previous years — that a strong team dynamic is as important as their sharp and shiny stars. Managers are key to this transition and will need to reset with a new definition of success, followed by support and training to change.

Gary P. Pisano in Harvard Business Review:

Too many leaders think that by breaking the organization into smaller units or creating autonomous “skunk works” they can emulate an innovative start-up culture. This approach rarely works. It confuses scale with culture. Simply breaking a big bureaucratic organization into smaller units does not magically endow them with entrepreneurial spirit. Without strong management efforts to shape values, norms, and behaviors, these offspring units tend to inherit the culture of the parent organization that spawned them.

Creating an innovative culture is complex, intentional work. But it is work that must be done if news organizations are to innovate and, therefore, survive.

Conclusion

The internet is changing more rapidly than it has in years, creating headwinds for newsrooms and jeopardizing independent journalism’s viability. We need those organizations to exist: they reduce corruption, inform the voting public, and allow us to connect with and understand our communities in vital ways.

These organizations must own their digital presence outright to shield themselves from risks created by third parties that wield outsized supplier power over their business models. They must build direct relationships with their communities, prioritizing open protocols over proprietary systems. They need to invest in technology expertise that can help them weather these changes and make that expertise a first-class part of their senior leadership teams.

To get there, they must build an open culture of experimentation, where transparency and openness are core values cemented through excellent, intentional communication. They must be empathetic, un-hierarchical workplaces where a great idea can be fostered from anywhere. They must build a mutual culture of respect and collaboration between editorial and non-editorial staff and ensure that the expertise to advise on and predict technology challenges is present and well-supported in-house.

Experimentation and innovation are key. Newsrooms can discover practical ways to navigate these challenges by testing new strategies, technologies and mindsets. The road ahead is challenging, but with strategic investments and a forward-looking approach, newsrooms can continue to fulfill their vital role in a well-functioning democratic society. The best time for action was ten years ago; the second best time is now.


John Philpin : Lifestream

SlashPages are here. I’d already playing with them with

SlashPages are here. I’d already playing with them without knowing the name. I was nearly ‘done’ and then Pratik shared this. Menu item added. Hat Tips to @rknightuk @bjhess @sivers @alastairjohnston @manuelmoreale @goblingumbo @pratik @simonwoods - you have each inspired me on this one.

SlashPages are here.

I’d already playing with them without knowing the name. I was nearly ‘done’ and then Pratik shared this. Menu item added.

Hat Tips to @rknightuk @bjhess @sivers @alastairjohnston @manuelmoreale @goblingumbo @pratik @simonwoods - you have each inspired me on this one.


I tried the Spark introductory Premium offer last year - and

I tried the Spark introductory Premium offer last year - and while some things are interesting and cool - at a renewal of 60 bucks … yeah - not that interesting and cool … and then I just spotted that they seem to be in the SetApp store.

I tried the Spark introductory Premium offer last year - and while some things are interesting and cool - at a renewal of 60 bucks … yeah - not that interesting and cool … and then I just spotted that they seem to be in the SetApp store.


If I know the answer - can I win? Or, do I really have to gu

If I know the answer - can I win? Or, do I really have to guess?

If I know the answer - can I win? Or, do I really have to guess?

Wednesday, 22. May 2024

John Philpin : Lifestream

From The Daily Beast The Bulwark reported that the staff

From The Daily Beast The Bulwark reported that the staffer was Natalie Harp, a Trump confidante who follows him around with a portable printer so he can read a constant feed of news and other information on paper. … my bold

From The Daily Beast

The Bulwark reported that the staffer was Natalie Harp, a Trump confidante who follows him around with a portable printer so he can read a constant feed of news and other information on paper.

… my bold


A gamble? He just wants out.

A gamble? He just wants out.

A gamble? He just wants out.


📸 Trick bicyclists this way.

📸 Trick bicyclists this way.

📸 Trick bicyclists this way.


🖇️ 🎙️ Vinyl Sneak Peek Conan’s short promo had me laughing

🖇️ 🎙️ Vinyl Sneak Peek Conan’s short promo had me laughing out loud 😂 as I wandered down to the ferry this morning.

🖇️ 🎙️ Vinyl Sneak Peek

Conan’s short promo had me laughing out loud 😂 as I wandered down to the ferry this morning.


Just a Theory

Mini Summit Six

A rough transcript of the sixth and final Extension Ecosystem Mini-Summit, in which we review potential topics for the in-person summit at PGConf.dev and discuss how to organize it.

Last week, a few members of the community got together for for the sixth and final Postgres Extension Ecosystem Mini-Summit. Follow these links for the video and slides:

Video Keynote PDF Slides

Or suffer through my interpolation of YouTube’s auto-generated transcript, interspersed with chat activity, if you are so inclined.

Introduction I opened the meeting, welcomed everyone, and introduced myself as host. I explained that today I’d give a brief presentation on the list of issues I I’ve dreamed up and jotted down over the last couple mini-summits as possible potential topics to take on at the Summit in Vancouver on May 28th. Presentation

These are things that I’ve written down as I’ve been thinking through the whole architecture myself, but also that come up in these Summits. I’m thinking that we could get some sense of the topics that we want to actually cover at the summit. There is room for about 45 people, and I assume we’ll break up “unconference style” into four or five working groups. People an move to corners, hallways, or outdoors to discuss specific topics.

Recall the first mini-summit I showed a list of things that of potential topics that might come up as we think through what’s issues in the ecosystem. I left off with the prompt “What’s important to you?” We hope to surface the most important issues to address at the summit and create a hierarchy. To that end, I’ve created this Canva board1 following Open Space Technology2 to set things up, with the rules and an explanation for how it workjs.

I expect one of us (organizers) to give a brief introduction at the start of the summit to outline the principles of Open Space Technology, which are similar to unconferences.

Open Space Technology principles are:

Whoever comes are the right people Whatever happens is the only thing that could happen Whenever it starts at the right time (but we start at 2 p.m. and we have only three hours so we’ll try to make the best of it) When it’s over it’s over And whatever happens is the right place

There is also a “Law of Mobility”. If you start out interested in one topic and attending a session or discussion about one topic, and you decide you want to do something else, you can wander over to another session . Open Space Technology calls these people “bumblebees” who cross-pollinate between topics. “Butterflies” are the people who hover around a particular topic to make it happen.

And “Come to be Surprised” about what will come up.

I’ve split potential topics into topics in Post-its. we might have four or five spaces. Pick a space, pick a session; we have two two-hour-long sessions. I assume we’ll have 15-30 minutes to open the Summit, do intros, and split up the sessions; then have people do an hour on one topic and an hour on a second topic. At the end, we’ll do the readout in which we talk about decisions we came to.

If you’re interested in facilitating any of these topics, simply drag it in and stick your name on it.

First I thought I’d briefly go over the list of topics as I’ve imagined them. I posted the list on Slack a couple weeks ago and added to it as things have come up in the discussions. But I want to give a high level view of what these brief descriptions mean.

This is ad-hoc; I don’t have anything super planned. Please feel free to jump in at any time! I think I’ve turned on “talking permitted” for everybody, or stick your hand up and we’ll be glad to figure out other stuff, especially if you’re thinking of other topics or related things, or if you think things should be merged.

Any questions or thoughts or comments?

I put the topics in broad categories. There’s some crossover, but the the first one I think of is metadata. I’ve thought about metadata a fair bit, and drafted an RFC for the kinds of things to put in an updated metadata standard, like:

How do you specify third-party dependencies? For example, PostGIS depends on additional libraries; how can those be specified in an ideally platform neutral way within the metadata?

How to specify the different types of extensions there are? Stephen wrote a blog post last year about this: you have CREATE EXTENSION extensions, LOAD command extensions, background workers, applications, and more. You have things that need shared_preload_libraries and things that don’t. How do we describe those things about an extension within a distribution package?

Taxonomies have come up a few times. PGXN currently allows extension authors to put an arbitrary number of tags into their META.json file. Maybe in part because of the precedent of the stuff that that I released early on, people mostly put stuff in there to describe it, like “fdw”, or “function” or “JSON”. Some of the newer uh binary distribution packaging systems, in particular Trunk, have a curated list of categories that they assign. so there might be different ways we want to classify stuff.

Another approach is crates.io, which has a canonical list of categories (or “slugs”), that authors can assign. These are handy they group things together in a more useful way, like “these are related to data analytics” or “these are related to Vector search” — as opposed to the descriptive tags PGXN has now. So, what ought that to look like? What kind of controls should we have? And who might want to use it?

How would we specify system requirements. For example “this package requires only a subset of OSes”, or the version of an OS, or the version of postgres, or CPU features. Steven’s mentioned vector-based ones a few times, but there’s also things like encryption instructions provided by most chips. Or the CPU architecture, like “this supports aarch64 but not amd64.” How should we specify that?

I covered categorization under taxonomies

Versioning. I blogged about this a couple months ago. I’m reasonably sure we should just stick to SemVer, but it’s worth bringing up.

Thoughts on metadata, or stuff I’ve left out? This is in addition to the stuff that’s in the META.json spec. It leaves room for overlap with core stuff. How do we create one sort of metadata for everything, that might subsume the control file as well as the metadata spec or trunk.toml?

Jeremy S in chat: So far this is seeming like a good recap of ground that’s been covered, questions & topics that have been raised. Great to see how broad it’s been

The next category is the source registry. This is thinking through how we should evolve the PGXN root registry for distributing extension source code. There are questions like identity, namespacing, and uniqueness.

These are broad categories but identity is how do you identify yourself to the system and claim ownership over something.

What sort of namespacing should we use? Most systems, including PGXN, just use an arbitrary string and you own a string from [first release]. But other registries, like Go, allow you to use domain-based namespacing for packages. This is really nice because it allows a lot more flexibility, such as the ability to switch between different versions or forks.

Then there’s the level of uniqueness of the namespacing. This is kind of an open question. Another another approach I thought of is that, rather than string that names your extension distribution being unique, it could be your username and the string. That makes it easier when somebody abandoned something and somebody else forks it and has a new username. Then maybe people can switch more easily. To be able to account for and handle that sort of evolution in a way that single string uniqueness makes trickier.

Distributed versus centralized publishing. I’ve written about this a couple times. I am quite attracted to the Go model where packages are not centrally distributed but are in three or four supported Version Control Systems, and as long as they use SemVers and appropriate tags, anybody can use them. The centralized index just indexes a package release the first time it’s pulled. This is where host names come into play as part of the namespacing. It allows the system to be much more distributed. Now Go caches all of them in a number of different regions, so when you download stuff it goes through the Go stuff. When you say “give me the XYZ package,” it’ll generally give you the cached version, but will fall back on the repositories as well. So there’s still the centralized stuff.

I think there’s a a lot to that and it goes along with the namespacing issue. But there are other ideas at play as well. For example, almost all the other source code distribution systems just use a centralized system: crates.io, CPAN, npm, and all the rest.

And maybe there are other questions to consider, like is there some sort of protocol we should adopt as an abstraction, such as Docker, where Docker is not a centralized repository other than hub.docker.com. Anyone can create a new Docker repository, give it a host name, and then it becomes something that anybody can pull from. It’s much more distributed. So there are a number of ideas to think through.

Binary packaging and distribution patterns. I have a separate slide that goes into more detail, but there are implications for source code distribution, particularly with the metadata but perhaps other things. We also might want to think through how it might vary from source distribution.

Federated distribution gets at the Docker idea, or the OCI idea that Alvaro proposed a few weeks ago. Stuff like that.

What services and tools to improve or build. This goes to the fundamental question of why we’ve had all these new packaging systems pop up in the last year or so. People were saying “there are problems that aren’t solved by PGXN.” How do we as a community collectively decide what are the important bits and what we should build and provide. Features include developer tools, command line clients, search & browse, and discovery.

Stats, reports, and badging. This is another fundamental problem that some of the emerging registries have tried to to address: How do you find something? How do you know if it’s any good? How do you know who’s responsible for it? How do you know whether there’s some consensus across the community to use it? The topic, then, is what sort of additional metadata could we provide at the registry level to include some hint about these issues. For example, a system to regularly fetch stars and statistical analysis of a GitHub or a Bitbucket project. Or people wanted review sites or the ability to comment on on systems.

There’s also badging, in particular for build and test matrices for extensions that will not only encourage people to better support broad arrays of versions of Postgres and platforms. There could be badges for that. so you can see how well an extension supports various platforms. And any other sort of badging, like quality badging. The idea is a brainstorming of what sorts of things might be useful there, and what what might be best to build first, might be the the low hanging fruit.

Any questions, comments,m thoughts, additional suggestions on the root registry?

Interlude

Steven Miller: So the idea is there are topics on the left and then they get lined up into the schedule? So there are five five different rooms, so horizontally aligned it4ms are at the same time?

David Wheeler (he/him): Correct. These are session one and these are session two.

Jeremy S: I was kind of waiting to jump to that. It seemed like you were just doing a review of all the topics we’ve covered, but I was waiting until till you got through everything to bring that up.

Steven Miller: Oh yeah, good call, good call.

Jeremy S: I have the same kind of question/concern. This is a great list of topics, now what do we want to do with the time in Vancouver? David, do you think we need to go through everything on the list? How do you want to spend the time today?

David Wheeler (he/him): I was trying to do a quick review just so people knew what these words mean. If you all feel like you have a good idea, or you want to add topics of your own, please do!

Jeremy S: Like I commented in the chat, it’s amazing to see how much ground we’ve covered, and it’s good to have a a quick recap. It’s 9:22 right now Pacific time — 22 after the hour wherever you are — I just want to make sure we don’t run out of time going through everything.

David Wheeler (he/him): I agree, I’ll make it work. I can speed up a little. I know I can be verbose about some of this stuff.

David G. Johnson: Unless the ones from India, in which case they have half hour time zone.

David Wheeler (he/him): I was gonna say! [Laughs]

Presentation Continues

Binary packaging. This is the problem that PGXMan and trunk have tried to solve with varying degrees of success. I think it’d be worthwhile for us to think through as a community what, ideally, should a community-provided binary packaging system look like?

And what’s the format? Do we want to do tarballs, do OCI like Alvaro proposed? Do we want something like RPM or Apt or Python wheels? That’s a that’s actually something I’m super interested to get into. There was a question that came up two weeks ago in Yurri’s presentation. I think Daniele suggested that the Python wheel package format allows you to put dynamic libs into the wheel. That’s pretty interesting and worth looking into as well.

How we go about building a community-based binary packaging registry? How do we do the build farming, what platforms and architectures and OSes would it support, and what sort of security, trust, and verification? And the centralization: who runs it, who’s responsible for it, how should it work at a high level?

Philippe Noël in chat: Phil from ParadeDB here (pg_search, pg_analytics, pg_lakehouse) — First minisummit I can attend, glad to be here

Thank for coming, Philippe! Again, interrupt me anytime.

The next topic is developer tooling. Developer tooling today is kind of all over the place. There a PGXN client, there’s the PGXN utils client (which doesn’t compile anymore, as far as I can tell), there’s pgrx stuff, and maybe a few other things. What sorts of tools would be useful for developers who actually develop and build extensions?

CLIs and APIs can do metadata management, or scaffolding your source code and adding new features through some sort of templating system.

The packaging and Publishing system based on how we uh ultimately elect to distribute source code, and how we ultimately elect to distribute binary code. How does that get packaged up with the namespacing and all the decisions we made, to be as easy as possible for the developer?

What build pipelines do we support? today PGXS and pgrx are perhaps the most common, but I’ve seen GNU autoconf configure stuff and stuff that uses Rust or Go or Python-based builds. Do we want to support those? And how do we integrate them with our binary packaging format and where Postgres expects to put stuff?

I think this is an important topic. One of the things I’ve been dealing with as I’ve talked to the people behind Apache Age and a couple other projects is how they put put stuff in /usr/local by default. I suggest that it’d be better if it went where pg_config wants to put it. How do we want to go about integrating those things?

Tooling for CI/CD workflows to make it as easy as possible to test across a variety of platforms, Postgres versions, and those pipelines.

Kind of a broad Community topic here. This gets to where things are hosted. There’s a Postgres identity service that does Oauth 2; is that something we want to plug into? Is there a desire for the community to provide the infrastructure for the systems or for at least the core systems of PGXN v2? Who would support it? The people who work on the development ideally would also handle the devops, but should work closely with whoever provides the infrastructure to make sure it’s all copacetic. And that there’s a a plan for when something happens. People exit the community for whatever reason; how will systems continue to be maintained? I don’t think there’s a plan today for PGXN.

Another topic is documentation. How do we help engineers figure out how to build extensions; tutorials and references for all the things and all the various details. Do we end up writing a book, or do we just have very specifically-focused tutorials like, “So you want to build a foreign data wrapper; here’s a guide for that.” Or you just need to write a background worker, here’s an example repository to clone. What should those things look like?

CREATE EXTENSION Hooks Background workers CLI apps/services Web apps Native apps

This also kind of covers the variety of different kinds of extensions we might want to package and distribute.

Then there’s all the stuff that I filed under “core,” because I think it impacts the core Postgres project and how it may need to evolve or we might want it to evolve over time. One is the second extension directory; there’s a patch pending now, but it’ll probably be deferred until until Postgres 17 ships; it’s on hold while we’re in the freeze. This is a patch that Christoph Berg wrote for the Debian distribution; it allows you to have a second destination directory for your extensions where Postgres knows to find stuff, including shared object libraries. This would make it easier for projects like Postgres.app and for immutable Docker containers to mount a new directory and have all the stuff be there.

I would love to see some sort of more coherent idea of what an extension pack package looks like, where like if I install pgTAP, all of its files are in a single subdirectory that Postgres can access. Right now it’s in package config, and the sharedir and the libder and the docdir — it’s kind spread all over.

Should there be a documentation standard, in the way you have JavaDoc and rustdoc and Godoc, where docs are integrated into the source code, so it’s easy to use, and there’s tooling to build effective documentation. Today, people mostly just write short READMEs and leave it at that, which is not really sufficient for a lot of projects.

There’s the longstanding idea of inline extensions that Dimitri proposed back as far as 2013, something they called “units”. Oracle calls them “packages” or “modules”. Trusted Language Extensions start making a stab at the problem, trying to make something like inline extensions with the tooling we have today. How should that evolve? What sorts of ideas do we want to adapt to make it so that you don’t have to have physical access to the file system to manage your extensions? Where you could do it all over SQL or libpq.

How can we minimize restarts? A lot of extensions require loading DSOs in the shared_preload_libraries config, which requires a cluster restart. How can we minimize that need? There are ways to minimize restarts; it’s just a broad category I threw in.

What Namespacing is there? I touched on this topic when I wrote about Go Namespacing a while ago. My current assumption is, if we decided to support user/extension_string or hostname/user/extension_string namespacing for package and source distribution, Postgres itself still has to stick to a single string. How would we like to see that evolve in the future?

What kind of sandboxing, code signing, security and trust could be built into the system? Part of the reason they’ve resisted having a second extension directory up to now is to have one place where everything was, where the DBA knows where things are, and it’s a lot it’s easier to manage there. But it’s also because otherwise people will put junk in there. Are there ideas we can borrow from other projects or technologies where anything in some directory is sandboxed, And how is it sandboxed? Is it just for a single database or a single user? Do we have some sort of code signing we can build into the system so that Postgres verifies an extension when you install it? What other kinds of security and trust could implement?

This is a high level, future-looking topic that occurred to me, but it comes up especially when I talk to the big cloud vendors.

An idea I had is dynamic module loading. It came up during Jonathan’s talk, where there was a question about how one could use Rust crates in PL/Rust, a trusted language. Well, a DBA has to approve a pre-installed list of crates that’s on the file system where PL/Rust can load them. But what if there was a hook where, in PL/Perl for example, you use Thing and a hook in the Perl use command knows to look in a table that the DBA manages and can load it from there. Just a funky idea I had, a way to get away from the file system and more easily let people, through permissions, be able to load modules in a safe way.

A topic that came up during Yurri’s talk was binary compatibility of minor releases, or some sort of ABI stability. I’d be curious what to bring up with hackers on formalizing something there. Although it has seemed mostly pretty stable over time to me, that doesn’t mean it’s been fully stable. I’d be curious to hear about the exceptions.

So! That’s my quick review. I did the remainder of them in 11 minutes!

Discussion

Jeremy S: Well done.

David Wheeler (he/him): What I’d like to do is send an email to all the people who are registered to come to The Summit in two weeks, as well as all of you, to be able to access this board and put stars or icons or something — stickers which you can access —

Jeremy S: I do feel like there’s something missing from the board. I don’t know that it’s something we would have wanted to put on sooner, but I kind of feel like one of the next steps is just getting down into the trenches and looking at actual extensions, and seeing how a lot of these topics are going to apply once we start looking like at the list. I was looking around a bit.

It’s funny; I see a mailing list thread from a year or two ago where, right after Joel made his big list of 1,000 extensions, he jumped on the hackers list and said, “hey could we stick this somewhere like on the wiki?” And it looks like nobody quite got around to doing anything like tha. But that’s where I was thinking about poking around, maybe maybe starting to work on something like that.

But I think once we start to look at some of the actual extensions, it’ll help us with a lot of these topics, kind of figure out what we’re talking about. Like when you’re when you’re trying to figure out dependencies, once you start to figure out some of the actual extensions where this is a problem and other ones where it’s not, it might help us to have be a lot more specific about the problem that we’re trying to solve. Or whether it’s versioning, which platform something is going to build on, all that kind of stuff. That’s where I was thinking a topic — or maybe a next step or a topic that’s missing, or you were talking about how many extensions even build today. If you go through the extensions on PGXN right now, how many of them even work, at all. So starting to work down that list.

David Wheeler (he/him): So, two thoughts on that. One is: create a sticky with the topic you want and stick it in a place that’s appropriate, or create another category if you think that’s relevant.

Jeremy S: It’s kind of weird, because what I would envision is what I want to do on the wiki — I’ll see if I can start this off today, I have rights to make a Postgres Wiki page — is I want to make a list of extensions, like a table, where down the left is the extensions and across the top is where that extension is distributed today. So just extensions that are already distributed like in multiple places. I’m not talking about the stuff that’s on core, because that’s a given that it’s everywhere. But something like pg_cron or PGAudit, anybody who has extensions probably has them. That gives some sense of the extensions that everybody already packages. Those are obviously really important extensions, because everybody is including them.

And then the next thing I wanted to do was the same thing with the list of those extensions on the left but a column for each of the categories you have here. For, say, PGAudit, for stuff across the top — metadata, registry packaging, developer stuff — for PGAudit are their packaging concerns? For PGAudit, go down the list of registry topics like identity, where’s the where is the source for PGAudit, is the definitive upstream GitLab, isit GitHub, is it git.postgresql.org? I could go right down the list of each of these topics for PGAudit. and then go down the list of all of your topics for pg_hint_plan. That’s another big one; pg_hint_plan is all over the place. Each of your topics I could take and apply to each of the top 10 extensions and there might be different things that rise to the surface for pg_hint_plan than there are for, like, pgvector.

David Wheeler (he/him): That sounds like a worthwhile project to me, and it could be a useful reference for any of these topics. Also a lot of work!

Jeremy S: Well, in another way to like think about Vancouver might be, instead of like splitting people up by these topics — I’m spitballing here, this this might be a terrible idea — but you could take a list of like 20 or 30 important extensions split people up into groups and say, “here’s five extensions for you, now cover all these topics for your five extensions.” You might have one group that’s looking at like pg_hint_plan and pgvector and PGAudit, and then a different group that has pg_cron and whatever else we come up with. That’s just another way you could slice it up.

David Wheeler (he/him): Yeah, I think that you’re thinking about it the inverse the way I’ve been thinking of it. I guess mine is perhaps a little more centralized and top down, and that comes from having worked on PGXN in the past and thinking about what we’d like to build in the future. But there’s no reason it couldn’t be bottom up from those things. I will say, when I was working on the metadata RFC, I did work through an example of some actually really fussy extension — I don’t remember which one it was — or no, I think it was the ML extension.3 I think that could be a really useful exercise.

But the idea the Open Technology Space is that you can create a sticky, make a pitch for it, and have people vote by putting a star or something on them. I’m hoping that, a. we can try to figure out which ones we feel are the most important, but ultimately anybody can grab one of these and say “I want to own this, I’m putting it in session session one, and put your put your name on it. They ca be anything, for the most part.

Jeremy S: Sure. I think I don’t totally grok the Canva board and how that all maps out, but at the end of the day whatever you say we’re doing in Vancouver I’m behind it 100%.

David Wheeler (he/him): I’m trying to make it as open as possible. If there’s something you want to talk about, make a sticky.

Jeremy S: I’ll add a little box. I’m not sure how this maps to what you want to do with the time in Vancouver.

David Wheeler (he/him): Hopefully this will answer the question. First we’ll do an intro and welcome and talk about the topics, give people time to look at them, I want to send it in advance so people can have a sense of it in advance. I know the way they do the the Postgres unconference that’s been the last day of PGCon for years, they have people come and put a sticky or star or some sort of sticker on the topics they like, and then they pick the ones that have the most and and those are the ones they line up in here [the agenda]. But the idea of the Open Technology stuff is a person can decide on whatever topic they want, they can create their sticky, they can put it in the set slot they want and whatever space they want, and —

Jeremy S: Ooooh, I think I get it now. Okay, I didn’t realize that’s what you were doing with the Canva board. Now I get it.

David Wheeler (he/him): Yeah, I was trying to more or less do an unconference thing, but because we only have three hours try to have a solid idea of the topics we want to address are before we get there.

Jeremy S: I don’t know though. Are you hoping a whole bunch of people are going to come in here and like put it — Okay, it took me five or ten minutes to to even realize what you were doing, and I don’t have high hopes that we’ll get 20 people to come in and vote on the Post-it notes in the next seven days.

David Wheeler (he/him): Yeah, maybe we need to… These instructions here are meant to help people understand that and if that needs to be tweaked, let’s do it.

David G. Johnston in chat: How many people are going to in this summit in Vancouver? David G. Johnston in chat: Is the output of a session just discussions or are action items desired?

Steven Miller: I have another question. Are people invited to present at the Summit if they’re not physically present at the Summit? And then same question for viewership

David Wheeler (he/him): I don’t think they are providing remote stuff at the Summit

Steven Miller: okay

David Wheeler (he/him): David, last I heard there were 42 people registered. I think we have space for 45. We can maybe get up to 50 with some standing room, and there’s a surprisingly large number of people (laughs).

David G. Johnston in chat: So average of 10 in each space?

Jeremy S: Have you gone down the list of names and started to figure out who all these people? Cuz that’s another thing. There might be people who have very little background and just thought “this sounds like an interesting topic.” How those people would contribute and participate would be very different from someone who’s been working with extensions for a long time.

David Wheeler (he/him): David, yeah, and we can add more spaces or whatever if it makes sense, or people can just arbitrarily go to a corner. Because it’s an unconference they can elect to do whatever interests them. I’m just hoping to have like the top six things we think are most important to get to ahead of time.

Jeremy, Melanie sent me the list of participants, and I recognized perhaps a quarter of the names were people who’re pretty involved in the community, and the rest I don’t know at all. so I think it’s going to be all over the map.

Steven Miller: So would it work if somebody wanted to do a presentation, they can. They grab stickies from the left and then you could also duplicate stickies because maybe there’d be some overlap, and then you put them in a session. But there’s basically supposed to be only one name per field, and that’s who’s presenting.

David Wheeler (he/him): You can put however many names on it as you want. Open technology usually says there’s one person who’s facilitating and another person should take notes.

Steven Miller: Okay.

David Wheeler (he/him): But whatever works! The way I’m imagining it is, people say, “Okay I want to talk to other people about make some decisions about, I don’t know, documentation standards.” So they go off to a corner and they talk about it for an hour. There are some notes. And the final half hour we’ll have readouts from those, from whatever was talked about there.

Steven Miller: These are small working sessions really,it’s not like a conference presentation. Okay, got it

David Wheeler (he/him): Yeah. I mean, somebody might come prepared with a brief presentation if they want to set the context. [Laughs] Which is what I was trying to do for the overall thing here. But the idea is these are working sessions, like “here’s the thing we want to look at” and we want to have some recommend commendations, or figure out the parameters, or you have a plan — maybe — at the end of it. My ideal, personally, is that at the at the end of this we have a good idea of what are the most important topics to address earlier on in the process of building out the ecosystem of the future, so we can start planning for how to execute on that from those proposals and decisions. That’s how I’m thinking about it

Steven Miller: Okay, yeah I see.

Jeremy S: This sounds a lot like the CoffeeOps meetups that I’ve been to. They have a similar process where you use physical Post-it notes and vote on topics and then everybody drops off into groups based on what they’re interested in.

David Wheeler (he/him): Yeah it’s probably the same thing, the Open Technology stuff.

Steven Miller: Maybe we should do one field so we kind of get an idea?

David Wheeler (he/him): Sure. Let’s say somebody comes along and there are a bunch of stickers on this one [drops stickers on the sticky labeled “Identity, namespacing, and uniqueness”]. So so we know that it’s something people really want to talk about. So if somebody will take ownership of it, they can control click, select “add your name”, find a slot that makes sense (and we may not use all of these) and drag it there. So “I’m going to take the first session to talk about this.” Then people can put the stickies on it over here [pasties stickers onto the topic sticky in the agenda], so you have some sense of how many people are interested in attending and talking about that topic. But there are no hard and fast rules.

Whether or not they do that, say, “David wants to talk about identity name spacing uniqueness in the core registry,” we’re going to do that in the first session. We’ll be in the northeast corner of the room — I’m going to try to get access to the room earlier in the day so I can have some idea of how it breaks up, and I’ll tweak the the Canva to to add stuff as appropriate.

David G. Johnston in chat: Same thing multiple times so people don’t miss out on joining their #2 option? David G. Johnston in chat: How about #1, #2, #3 as labels instead of just one per person?

Jeremy S: Are you wanting us to put Post-it notes on the agenda now, before we know what’s been voted for?

David Wheeler (he/him): Yep! Especially if there’s some idea you had Jeremy. If there’s stuff you feel is missing or would be a different approach, stick it in here. It may well be not that many people interested in what I’ve come up with but they want to talk about those five extensions.

David Wheeler (he/him): (Reading comment from David Johnson): “One two and three as labels instead of just one per person?” David I’m sorry I don’t follow.

David G. Johnston: So basically like rank choice. If you’re gonna do I core one time and binary packaging one time, and they’re running at the same time, well I want to do both. I want to do core — that’s my first choice — I want to do binary packaging — that’s my second choice. If I had to choose, I’d go to number one. But if you have enough people saying I want to see this, that’s my number two option, you run binary packaging twice, not conflicting with core so you can get more people.

David Wheeler (he/him): I see, have people stick numbers on the topics that most interest in them. Let’s see here… [pokes around the Canva UX, finds stickers with numbers.] There we go. I’ll stick those somewhere that’s reasonable so people can rank them if they want, their top choices.

This is all going to be super arbitrary and unscientific. The way I’ve seen it happen before is people just drop stars on stuff and say, okay this one has four and this one has eight so we definitely want to talk about that one, who’s going to own it, that sort of thing. I think what makes sense is to send this email to all the participants in advance; hopefully people will take a look, have some sense of it, and maybe put a few things on. Then, those of us who are organizing it and will be facilitating on the day, we should meet like a day or two before, go over it, and make some decisions about what we definitely think should be covered, what things are open, and get a little more sense of how we want to run things. Does that make sense?

Jeremy S: Yeah, I think chatting ahead of time would be a good idea. It’ll be interesting to see how the Canva thing goes and what happens with it.

David Wheeler (he/him): It might be a mess! Whatever! But the answer is that whatever happens this is the right place. Whenever it starts is the right time. Whatever happens could only happen here. It’s super arbitrary and free, and we can adapt as much as we want as it goes.

David Wheeler (he/him): I think that’s it. Do you all feels like you have some sense of what we want to do?

Jeremy S: Well not really, but that’s okay! [Laughs]

Steven Miller: Okay, so here’s what we are supposed to do. Are we supposed to go find people who might be interested to present — they will already be in the list of people who are going to Vancouver. Then we talk to them about these Post-its and we say, “would you like to have a small discussion about one of these things. If you are, then put a sticky note on it.” And then we put the sticky notes in the fields, we have a list of names associated with the sticky notes. Like, maybe Yurri is interested in binary distribution, and then maybe David is also interested in that. So there’s like three or four people in each section, and we’re trying to make sure that if you’re interested multiple sections you get to go to everything?

David Wheeler (he/him): Yeah you can float and try to organize things. I put sessions in here assuming people would want to spend an hour, but maybe a topic only takes 15 minutes.

David G. Johnston: Staying on my earlier thought on what people want to see, people who are willing to present and can present on multiple things, if we have a gold star for who’s willing to actually present on this topic. So here’s a topic, I got eight people who want to see it but only one possible presenter. Or I got five possible presenters and three possible viewers. But you have that dynamic of ranked choice for both “I’ll present stuff” or “I’m only a viewer.

David Wheeler (he/him): I think that typically these things are self-organizing. Somebody says, “I want to do this, I will facilitate, and I need a note taker.” But they negotiate amongst themselves about how they want to go about doing it. I don’t think it necessarily has to be formal presentation, and usually these things are not. Usually it’s like somebody saying, “here’s what this means, this is the topic, we’re going to try to cover, these are the decisions we want to make, Go!”

Jeremy S: You’re describing the the the unconference component of PGCon that has been down in the past.

David Wheeler (he/him): More or less, yes

Jeremy S: So should we just come out and say this is a unconference? Then everybody knows what you’re talking about really fast, right?

David Wheeler (he/him): Sure, sure, yeah. I mean —

Jeremy S: We’re just we’re doing the same thing as – yeah.

David Wheeler (he/him): Yeah, I try to capture that here but we can use the word “unconference” for sure. [Edits the Canva to add “an unconference session” to the title.] There we go.

Steven Miller: I imagine there are people who might be interested to present but they just aren’t in this meeting right now. So maybe we need to go out and advertise this to people.

David Wheeler (he/him): Yeah, I want to draft an email to send to all the attendees. Melanie told me we can send an email to everybody who’s registered.

Jeremy S: And to be clear it’s full, right? Nobody new can register at this point?

David Wheeler (he/him): As far as I know, but I’m not sure how hard and fast the rules are. I don’t think any more people can register, but it doesn’t mean other people won’t wander in. People might have registered and then not not come because they’rein the patch the patch session or something.

So I volunteer to draft that email today or by tomorrow and share it with the Slack channel for feedback. Especially if you’re giving me notes to clarify what things mean, because it seems like there are more questions and confusions about how it works than I anticipated — in part because it’s kind of unorganized by design [chuckles].

Jeremy S in chat: https://wiki.postgresql.org/wiki/PgConUnconferenceFAQ

David Wheeler (he/him): Oh that’s a good thing to include Jeremy. that’s a good call. But to also try to maximize participation of the people who’re planning to be there. It may be that they say, “Oh this sounds interesting,” or whatever, so and I’ll add some different stickers to this for some different meanings, like “I’m interested” or “I want to take ownership of this” or “this is my first, second, third, or fourth choice”. Sound good?

Steven Miller: Yes, it sounds good to me!

David Wheeler (he/him): Thanks Steven.

Jeremy S: Sounds good, yeah.

David Wheeler (he/him): All right, great! Thanks everybody for coming!

Hit the #extensions channel on the Postgres Slack for the link! ↩︎

In the meeting I kept saying “open technology” but meant Open Space Technology 🤦🏻‍♂️. ↩︎

But now I can look it up. It was pgml, for which I mocked up a META.json↩︎

More about… Postgres PGXN Extensions PGConf Summit

John Philpin : Lifestream

Public Service Announcement 🔗 Daring Fireball: How to Make

Public Service Announcement 🔗 Daring Fireball: How to Make Google’s ‘Web’ View Your Search Default Not tried this yet … but I will be . and I know it will be of interest to most people in my world.

Public Service Announcement

🔗 Daring Fireball: How to Make Google’s ‘Web’ View Your Search Default

Not tried this yet … but I will be . and I know it will be of interest to most people in my world.


Advice from a pro If you are wondering if the croissant th

Advice from a pro If you are wondering if the croissant that is heating in the oven. Is done .. don’t use your finger tip to test. 🤦‍♂️

Advice from a pro

If you are wondering if the croissant that is heating in the oven. Is done .. don’t use your finger tip to test. 🤦‍♂️


📸 🌊 🚧A particularly stunning one this morning …. .. in fa

📸 🌊 🚧A particularly stunning one this morning …. .. in fact, it was worth stepping back from today. .. and then 10 minutes later, the sun peaks through and we can see the rays pouring through as the day opens up.

📸 🌊 🚧A particularly stunning one this morning ….

.. in fact, it was worth stepping back from today.

.. and then 10 minutes later, the sun peaks through and we can see the rays pouring through as the day opens up.


Wrench in the Gears

Blockchain Brain Boxing – Why People Continue To Disregard Electronic Health Records And Pay for Success Wellness Data

This video elaborates on a comment exchange that was posted this morning to a video I made last night unpacking Bobby’s Lunarpunk “Ape” signal. Links to maps referenced in the video for those interested in further inquiry.   This is the video I did with Shai on the RAND Noopolitik report.

This video elaborates on a comment exchange that was posted this morning to a video I made last night unpacking Bobby’s Lunarpunk “Ape” signal.

Links to maps referenced in the video for those interested in further inquiry.

https://embed.kumu.io/f6b7ddc6451089c299ff8f3c09ac7613#untitled-map?s=bm9kZS1DOXhZUnRJbw%3D%3D https://www.rand.org/content/dam/rand/pubs/perspectives/PEA200/PEA237-1/RAND_PEA237-1.pdf https://www.rand.org/content/dam/rand/pubs/perspectives/PEA200/PEA237-1/RAND_PEA237-1.pdf https://web.archive.org/web/20210122170720/https://littlesis.org/oligrapher/6310-industrial-agriculture https://web.archive.org/web/20201207235929/https://littlesis.org/oligrapher/5719-produce-rx-food-pay-for-success-medicaid https://web.archive.org/web/20200925091209/https://littlesis.org/oligrapher/4954-blockchain-healthcare-challenge-2016 https://web.archive.org/web/20201208182811/https://littlesis.org/oligrapher/4968-bloomberg-mypass-austin-blockchain-id https://web.archive.org/web/20210110131607/https://littlesis.org/oligrapher/5814-e-government-impact-investing-digital-israel https://web.archive.org/web/20201204194140/https://littlesis.org/oligrapher/5757-dassault-digital-twin-simulations-blockchain-id https://docs.google.com/presentation/d/1beurviKJ8YkZBla-HLtXy8OxsyIWkJAnGBXwf28Tbw8/edit?usp=sharing https://docs.google.com/presentation/d/1beurviKJ8YkZBla-HLtXy8OxsyIWkJAnGBXwf28Tbw8/edit?usp=sharing https://docs.google.com/presentation/d/1beurviKJ8YkZBla-HLtXy8OxsyIWkJAnGBXwf28Tbw8/edit?usp=sharing https://docs.google.com/presentation/d/1beurviKJ8YkZBla-HLtXy8OxsyIWkJAnGBXwf28Tbw8/edit?usp=sharing

 

This is the video I did with Shai on the RAND Noopolitik report.


Heres Tom with the Weather

Feditesting!

It’s cool to see the progress of the FediTest project. On March 7, there was a show-and-tell online meeting and at the end of April, a FediTest implementation update included a Quickstart to try out some examples. I was pleasantly surprised by the experience (including the specification annotations referencing each test) and the organization of the testing framework even at an early stage. I

It’s cool to see the progress of the FediTest project. On March 7, there was a show-and-tell online meeting and at the end of April, a FediTest implementation update included a Quickstart to try out some examples.

I was pleasantly surprised by the experience (including the specification annotations referencing each test) and the organization of the testing framework even at an early stage. I was able to get all the tests for the sass-imp-webfinger-server testplan passing last night for Irwin. For each failing test, I created an issue and referenced the test (e.g. Well-known webfinger should respond with access-control-allow-origin header #15 ).

Here’s the output of this example testplan:

TAP version 14 # test plan: Unnamed # started: 2024-05-22 06:33:53.423983+00:00 # ended: 2024-05-22 06:34:42.924770+00:00 # platform: Linux-6.5.0-28-generic-x86_64-with-glibc2.35 # username: tbbrown # hostname: agency # session: Unnamed # constellation: Unnamed # roles: # - name: client # driver: imp.ImpInProcessNodeDriver # - name: server # driver: saas.SaasFediverseNodeDriver ok 1 - webfinger.server.4_1__2_parameter_ordering_not_significant::parameter_ordering ok 2 - webfinger.server.4_2__14_must_only_redirect_to_https::must_only_redirect_to_https ok 3 - webfinger.server.4_2__3_requires_resource_uri::requires_resource_uri ok 4 - webfinger.server.4_2__4_do_not_accept_malformed_resource_parameters::double_equals ok 5 - webfinger.server.4_2__4_do_not_accept_malformed_resource_parameters::not_percent_encoded ok 6 - webfinger.server.4_2__5_status_404_for_nonexisting_resources::status_404_for_nonexisting_resources ok 7 - webfinger.server.4_5__1_any_uri_scheme_for_resource_identifiers::any_uri_scheme_for_resource_identifiers ok 8 - webfinger.server.4__1_accepts_all_link_rels_in_query::accepts_combined_link_rels_in_query ok 9 - webfinger.server.4__1_accepts_all_link_rels_in_query::accepts_known_link_rels_in_query ok 10 - webfinger.server.4__1_accepts_all_link_rels_in_query::accepts_unknown_link_rels_in_query ok 11 - webfinger.server.4__3_only_returns_jrd_in_response_to_https_requests::only_returns_jrd_in_response_to_https ok 12 - webfinger.server.5_1_cors_header_required::cors_header_required 1..12 # test run summary: # total: 12 # passed: 12 # failed: 0 # skipped: 0 # errors: 0

Getting these tests passing certainly improved the code and my understanding of the webfinger spec. Thanks to Johannes Ernst and the Fediverse Developer Network community for this.


Simon Willison

What is prompt optimization?

What is prompt optimization? Delightfully clear explanation of a simple automated prompt optimization strategy from Jason Liu. Gather a selection of examples and build an evaluation function to return a numeric score (the hard bit). Then try different shuffled subsets of those examples in your prompt and look for the example collection that provides the highest averaged score. Via @jxnlco

What is prompt optimization?

Delightfully clear explanation of a simple automated prompt optimization strategy from Jason Liu. Gather a selection of examples and build an evaluation function to return a numeric score (the hard bit). Then try different shuffled subsets of those examples in your prompt and look for the example collection that provides the highest averaged score.

Via @jxnlco


Phil Windleys Technometria

Can LLMs Provide References for Their Work? An Experiment

Recently my friend Scott Mace asked this on X in response to a post about the evolution of software and how LLMs will likely follow that same trajectory: Yes, but how trustworthy will it be? Can I at least see the links it used to create its response?

Recently my friend Scott Mace asked this on X in response to a post about the evolution of software and how LLMs will likely follow that same trajectory:

Yes, but how trustworthy will it be? Can I at least see the links it used to create its response?

From X
Referenced 2024-05-21T18:09:06-0400

The short answer is "no", but the real answer is a bit more nuanced because of how LLMs like ChatGPT work. First the short answer: LLMs don't understand what they're saying so they can't reference the material that it used to create a response. They don't work like that. They don't go out, do a bunch of research and then create a response. Instead lots (and I mean LOTS) of material is fed into the program that creates the model, a multi-dimensional matrix of word orders. Then when you ask a question ("give it prompt" in LLM-speak), it processes the question and generates text based on the probabilities of words in its model1. There no way if can satisfy Scott directly cause the model doesn't—can't, as far as I know—include the references that it used to create the text cause it's generating it on the fly from material that has been thoroughly blended together.

But, in fairness, that's not what people do most of the time either. If you asked me over lunch or at the end of a tech presentation to describe zero trust and its pros and cons, I'd start talking. I wouldn't start looking up things on the web and somehow weave them into text complete with references. I'd go look for references if you asked me to back up something I said. So, let's try that with ChatGPT 4o. I gave it the following prompt:

You are a writer for a website focusing on technology. Your specialty is security. Write a 150 word explanation of zero trust with three pros and three cons.

ChaptGPT answered with this text:

Zero Trust is a security model based on the principle of "never trust, always verify." Unlike traditional security models that assume everything inside the network is safe, Zero Trust assumes that threats can exist both inside and outside the network. It requires strict identity verification for every user and device attempting to access resources, regardless of their location.

Pros:

Enhanced Security: By continuously verifying users and devices, Zero Trust reduces the risk of breaches and unauthorized access.

Minimized Attack Surface: It limits access to only what is necessary, reducing potential entry points for attackers.

Improved Compliance: Helps meet regulatory requirements by ensuring strict access controls and logging all access attempts.

Cons:

Complex Implementation: Deploying Zero Trust can be complex and time-consuming, requiring a significant overhaul of existing systems and processes.

Increased Costs: The initial investment in technology and training can be high.

User Experience Impact: Continuous verification can lead to friction and inconvenience for users if not implemented smoothly.

Zero Trust is essential for modern cybersecurity, but it requires careful planning and resources to implement effectively.

That's pretty good. But what if we need some references to other material to back up the pros and cons? I asked ChatGPT to supply them:

Now repeat that task, but this time include references as hyperlinks to back up each pro and con.

ChaptGPT answered with this text:

Zero Trust is a security model based on the principle of "never trust, always verify." Unlike traditional security models that assume everything inside the network is safe, Zero Trust assumes that threats can exist both inside and outside the network. It requires strict identity verification for every user and device attempting to access resources, regardless of their location.

Pros:

Enhanced Security: By continuously verifying users and devices, Zero Trust reduces the risk of breaches and unauthorized access (CSO Online).

Minimized Attack Surface: It limits access to only what is necessary, reducing potential entry points for attackers (NIST).

Improved Compliance: Helps meet regulatory requirements by ensuring strict access controls and logging all access attempts (Gartner).

Cons:

Complex Implementation: Deploying Zero Trust can be complex and time-consuming, requiring a significant overhaul of existing systems and processes (Forrester).

Increased Costs: The initial investment in technology and training can be high (ZDNet).

User Experience Impact: Continuous verification can lead to friction and inconvenience for users if not implemented smoothly (TechRepublic).

Zero Trust is essential for modern cybersecurity, but it requires careful planning and resources to implement effectively.

That looks great! Nice references for each bullet. There's just one problem. All the links, except the NIST reference, are dead. I don't know if they're wrong, made up, or stale. I tried finding content on a few of the sites by searching and came up dry. As I heard Johannes Ernst say once, ChatGPT is a "large language model" not a "large fact model" and this bears that out.

Still, asking for references or evidence for statements is something people are going to want. So, I tried a different tack. This time I asked this:

Why do you say complex implementation is a problem for zero trust? Can you give me a few references on that I can read myself?

ChatGPT responded with this:

Implementing a Zero Trust security model can be challenging for several reasons:

Overall Complexity: Zero Trust requires a comprehensive understanding of your network's intricacies, including users, devices, applications, and data. This can involve significant changes to existing infrastructure, which can be disruptive and costly. Organizations need to be prepared for the extensive planning and resources required for successful implementation​ (Help Net Security)​​ (Enterprise Networking Planet​).

Legacy Systems: Many organizations have legacy systems that are not designed to operate within a Zero Trust framework. These systems might require different security measures or even complete replacement, adding to the complexity and cost (Help Net Security)​​ (Enterprise Networking Planet​)

Cultural Shift: Zero Trust is not just a technological shift but also a cultural one. It requires continuous verification and strict access controls, which can lead to user frustration and resistance if not managed properly. Ensuring user buy-in and minimizing disruption to workflows through training and communication is essential (Deloitte United States)​​ (Guidehouse).

Resource Intensive: Implementing and maintaining Zero Trust can put a strain on IT resources due to the need for continuous monitoring and management. The need for advanced security tools and ongoing updates further adds to the resource demands (Help Net Security)​​ (Enterprise Networking Planet​).

Stakeholder Alignment: Achieving buy-in from all stakeholders and aligning them with the Zero Trust principles is critical. This requires a coordinated effort to educate stakeholders, prioritize gaps, and ensure consistent understanding and application of Zero Trust principles across the organization (Deloitte United States)​​ (Guidehouse).

These challenges highlight the importance of a phased and well-planned approach to Zero Trust implementation, considering both technological and human factors. For more detailed insights, you can refer to articles from Help Net Security and Guidehouse.

Much better. The links in the bullets all work, but the ones at the end don't. The articles it references aren’t very detailed, so I wouldn’t say they’re stellar references, but at least there’s a pointer to outside material that discusses similar challenges. So, partial success.

I've found that when you're using ChatGPT to do research, treating it like a librarian and asking questions that follow up are the best way to gain confidence that the material is correct and find the supporting material, just like I did above. I'm sure you could wrap additional helper functions around an LLM to make getting references easier. But for now interrogation through chat is the best way to get comfortable with the results.

Notes

Yeah, this is really simplified and thus wrong, but it's directionally correct. For a better explanation, see What Is ChatGPT Doing … and Why Does It Work?.

Photo Credit: Older Man Talking to Robot Librarian from DALL-E (public domain) Prompt: draw me a rectanglular picture of an older man talking with a robot librarian. Use earth tones.


Patrick Breyer

„Lass dich überwachen“: EU-Rat will sich auf Chatkontrolle mit Zustimmung der Nutzer einigen [ergänzt]

Aufgrund eines neuen Vorschlags der belgischen Innenministerin könnten die EU-Regierungen die vielkritisierte Chatkontrolle doch bereits im Juni beschließen. Das ergibt sich aus internen Informationen, die dem Europaabgeordneten der Piratenpartei und Schattenberichterstatter …

Aufgrund eines neuen Vorschlags der belgischen Innenministerin könnten die EU-Regierungen die vielkritisierte Chatkontrolle doch bereits im Juni beschließen. Das ergibt sich aus internen Informationen, die dem Europaabgeordneten der Piratenpartei und Schattenberichterstatter Dr. Patrick Breyer zugespielt wurden.

Konkret sollen nach dem neuesten Vorschlag Nutzer von Apps mit Kommunikationsfunktion per AGB oder Popup-Nachricht zustimmen müssen, dass alle verschickten Bilder und Videos automatisiert und verdachtslos gescannt und gegebenenfalls der EU und der Polizei gemeldet werden. Dazu sollen wohl auch Überwachungsfunktionen in bisher sicher Ende-zu-Ende verschlüsselte Messenger eingebaut werden, auch wenn die Belgier behaupten, die von ihnen vorgeschlagene „Uploadmoderation“ unterscheide sich vom „Client-Side Scanning“. Stimmt ein Nutzer der Chatkontrolle nicht zu, soll er den Chat weiter nutzen nutzen können, aber keine Bilder und Videos mehr versenden können. An EU und Polizei ausleiten sollen die Scanner-Algorithmen bekanntes Material und unbekannte Bilder und Videos, die von „künstlicher Intelligenz“ für potenziell verdächtig gehalten werden. Auf die Durchsuchung von Textnachrichten nach Hinweisen auf „Grooming“, die auch bisher kaum zum Einsatz kommt, soll genauso verzichtet werden wie auf das noch nie eingesetzte Scannen von Audiokommunikation.

Bei Vorstellung des Vorschlags am 8. Mai haben sich mehrere bisher kritische Regierungen wohlwollend und aufgeschlossen geäußert. Schon übermorgen (24. Mai) soll erneut über den „neuen Vorschlag“ beraten werden. Direkt nach der Europawahl treffen sich die EU-Innenminister und könnten die Chatkontrolle beschließen.

„Der geleakte Vorschlag zeigt, dass der extreme Ausgangsentwurf der EU-Kommission zur in der freien Welt einzigartigen Chatkontrolle im Kern unverändert beibehalten werden soll“, warnt der Europaabgeordnete und profilierteste Gegner der Chatkontrolle Dr. Patrick Breyer (Piratenpartei).„Der neueste Vorstoß zur Chatkontrolle erinnert an die Fernsehshow ‚Lass dich überwachen‘. Messengerdienste rein textbasiert zu nutzen, ist im 21. Jahrhundert keine ernsthafte Option. Und Auswüchse der Chatkontrolle zu streichen, die ohnehin in der Praxis keine Rolle spielen, ist eine Mogelpackung.

Es bleibt dabei: Millionen privater Chats und Privatfotos unbescholtener Bürger sollen mit unzuverlässiger Technik durchsucht und ausgeleitet werden, ohne dass die Betroffenen auch nur entfernt mit Kindesmissbrauch zu tun haben – das zerstört unser digitales Briefgeheimnis. So landen unsere Nacktfotos und Familienbilder bei Fremden, in deren Hände sie nicht gehören und bei denen sie nicht sicher sind. Trotz Lippenbekenntnissen zu Verschlüsselung soll mit Client-Side-Scanning bisher sichere Ende-zu-Ende-Verschlüsselung allgemein ausgehebelt werden, um unsere Smartphones zu Spionen umzufunktionieren – das zerstört sichere Verschlüsselung.

Mich alarmiert, dass bisher kritische EU-Regierungen die umverpackten Pläne loben und dadurch die bisherige Sperrminorität zu kippen droht. Die Bundesregierung schweigt und ist wegen der SPD-Innenministerin Faeser seit Monaten ‚nicht sprechfähig‘ – dementsprechend bildet sie auch keine Allianzen mit anderen kritischen Staaten. Nicht einmal ein schriftliches Gutachten des Rechtsdienstes des Rates zu diesem offensichtlichen Grundrechtsverstoß ist bisher angefordert worden. Wenn die EU-Regierungen tatsächlich mit dieser radikalen Position zur Chatkontrolle in die Trilogverhandlungen gehen sollten, droht das Parlament hinter verschlossenen Türen erfahrungsgemäß seine Ausgangsposition schrittweise aufzugeben und sich auf schlechte und gefährliche Kompromisse einzulassen, die unsere Sicherheit im Netz auf Spiel setzen.

Jetzt ist die Zeit für Privatsphäre und sichere Verschlüsselung auf die Barrikaden zu gehen!“

Breyers Infoportal und Dokumentenarchiv zur Chatkontrolle

Ergänzung vom 23.05.2024: netzpolitik.org hat das Protokoll, auf das sich diese Analyse stützt, inzwischen auch im Volltext veröffentlicht.


Simon Willison

Quoting D. Richard Hipp, 18 years ago

The default prefix used to be "sqlite_". But then Mcafee started using SQLite in their anti-virus product and it started putting files with the "sqlite" name in the c:/temp folder. This annoyed many windows users. Those users would then do a Google search for "sqlite", find the telephone numbers of the developers and call to wake them up at night and complain. For this reason, the default name

The default prefix used to be "sqlite_". But then Mcafee started using SQLite in their anti-virus product and it started putting files with the "sqlite" name in the c:/temp folder. This annoyed many windows users. Those users would then do a Google search for "sqlite", find the telephone numbers of the developers and call to wake them up at night and complain. For this reason, the default name prefix is changed to be "sqlite" spelled backwards.

D. Richard Hipp, 18 years ago


Wrench in the Gears

Bobby Goes Ape Signaling Austin’s Lunar Punks

There aren’t many out there who could begin to decipher RFK Jr.’s strange message to the Apes; but, since I’m one of them, here’s forty minutes of off-the-cuff analysis in which I attempt to weave together crypto gaming, esotericism, microbiome management, fin-tech behavioral economics, gut instinct, and cyclical harmonization of the commons – based on [...]

There aren’t many out there who could begin to decipher RFK Jr.’s strange message to the Apes; but, since I’m one of them, here’s forty minutes of off-the-cuff analysis in which I attempt to weave together crypto gaming, esotericism, microbiome management, fin-tech behavioral economics, gut instinct, and cyclical harmonization of the commons – based on a foundation of ubiquitous sensing starting with carbon-nano permaculture protocols and personalized food prescriptions for AI-managed “wellness.” Remember, our biology is the post-Moore’s law technology. The future microbial network will run on orchestrated morphic fields coordinated by dynamic (Kundalini?) community currencies – always signaling the “wisdom of the crowd.”

See if you can follow along with me…

https://x.com/RobertKennedyJr/status/1792970117204287992

 

Here’s my map in progress if you care to explore.

Interactive map: https://embed.kumu.io/a2ceb5922c02e84c6977d4665c2e5a9c#untitled-map?s=bm9kZS15NDVjc29aVw%3D%3D

 

Here are some links to things mentioned in my chat. You can click on the image or caption to go to the source.

https://fintel.io/so/us/amc https://consensus2024.coindesk.com/speakers/ https://medium.com/@hasanmahira/crypto-punk-cultures-cypherpunk-solarpunk-and-lunarpunk-55775b8676db https://signalsfromtheedge.org/lunarpunk-solarpunk

 

https://www.osti.gov/opennet/manhattan-project-history/publications/ORNLClassifiedActivities.pdf https://youtu.be/jfCCDB9LOcY?t=172 https://www.youtube.com/watch?v=jB5gChpa9OY https://cycles.org/news/harmonics-theory-part-17-origins/

 

https://drrichardsmith.com/ https://cycles.org/letters/a-letter-from-our-new-chairman-dr-richard-smith/ https://cycles.org/letters/a-letter-from-our-new-chairman-dr-richard-smith/ https://youtu.be/Zy5Uq_pTUns?t=34 https://www.frontiersin.org/journals/microbiology/articles/10.3389/fmicb.2023.1214870/full https://wrenchinthegears.com/wp-content/uploads/2024/05/biosensors-12-00377.pdf https://blog.chain.link/reversing-climate-change-how-hybrid-smart-contracts-incentivize-regenerative-agriculture/ https://www.youtube.com/watch?v=rV481ArX8UA&list=PLUcrhDVaUj97YUUo8RfmmO4OJav6g5tS4&index=2

 

https://youtu.be/M0KvPkCCJf0?t=129 https://embed.kumu.io/e4740d5fc69c4eb9522fcb75c14dd6ef#untitled-map?s=bm9kZS16ejJFaFBXVg%3D%3D https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4304725/

 

https://www.conducttr.com/ https://plato.stanford.edu/Entries/qualia/ https://pubmed.ncbi.nlm.nih.gov/37003434/

 


Simon Willison

Mastering LLMs: A Conference For Developers & Data Scientists

Mastering LLMs: A Conference For Developers & Data Scientists I’m speaking at this 5-week (maybe soon 6-week) long online conference about LLMs, presenting about “LLMs on the command line”. Other speakers include Jeremy Howard, Sophia Yang from Mistral, Wing Lian of Axolotl, Jason Liu of Instructor, Paige Bailey from Google, my former co-worker John Berryman and a growing number of fascina

Mastering LLMs: A Conference For Developers & Data Scientists

I’m speaking at this 5-week (maybe soon 6-week) long online conference about LLMs, presenting about “LLMs on the command line”.

Other speakers include Jeremy Howard, Sophia Yang from Mistral, Wing Lian of Axolotl, Jason Liu of Instructor, Paige Bailey from Google, my former co-worker John Berryman and a growing number of fascinating LLM practitioners.

It’s been fun watching this grow from a short course on fine-tuning LLMs to a full-blown multi-week conference over the past few days!

Via @hugobowne

Tuesday, 21. May 2024

Ben Werdmüller

ProPublica’s new “50 states” commitment builds on a decade-plus of local news partnerships

"It’s a good time to be ProPublica. And it’s a good thing that we have ProPublica." Hey, that's where I work! The article continues: "Spreading its journalistic wealth has long been core to its mission. The latest iteration of that is the 50 State Initiative, announced last month." The 50 State Initiative is a commitment to publishing accountability journalism in

"It’s a good time to be ProPublica. And it’s a good thing that we have ProPublica."

Hey, that's where I work!

The article continues:

"Spreading its journalistic wealth has long been core to its mission. The latest iteration of that is the 50 State Initiative, announced last month."

The 50 State Initiative is a commitment to publishing accountability journalism in every US state over the next five years. This is an expansion of the Local Reporting Network, which was already doing great work in partnership with local newsrooms. As this piece points out, there are actually only two states where ProPublica hasn't run some kind of an investigative story - but, of course, the 50 States Initiative goes much deeper than that. It's an exciting time to be working here. #Media

[Link]


Decentralized Systems Will Be Necessary To Stop Google From Putting The Web Into Managed Decline

"The various decentralized social media systems that have been growing over the past few years offer a very different potential approach: one in which you get to build the experience you want, rather than the one a giant company wants." There's a chicken and egg problem here: while decentralized systems are absolutely going to be part of the solution, or at least hold most

"The various decentralized social media systems that have been growing over the past few years offer a very different potential approach: one in which you get to build the experience you want, rather than the one a giant company wants."

There's a chicken and egg problem here: while decentralized systems are absolutely going to be part of the solution, or at least hold most of the properties that make for a good solution, they also need to have a critical mass of people who use them.

A lot of people are looking towards Threads to provide this critical mass, but just as I'd invite newsrooms to consider how to gain more traffic without Apple News, I'd invite the federated social web community to consider what a growth looks like without Meta. It's not that Threads won't help - it's that you don't want to be dependent on a megacorp to provide the assistance you need. You never know when they'll change their policies and look elsewhere.

Still, the point stands: decentralization is a key part of the answer. There's a lot to be gained from investing in projects that provide strong user experiences, solve concrete real human problems alongside the ideological ones and the existential threats, and onboard a new generation of internet users to a better way to share and browse.

That's a tall order, but, as always, I'm hopeful. #Technology

[Link]


Simon Willison

New Phi-3 models: small, medium and vision

New Phi-3 models: small, medium and vision I couldn't find a good official announcement post to link to about these three newly released models, but this post on LocalLLaMA on Reddit has them in one place: Phi-3 small (7B), Phi-3 medium (14B) and Phi-3 vision (4.2B) (the previously released model was Phi-3 mini - 3.8B). You can try out the vision model directly here, no login required. It didn'

New Phi-3 models: small, medium and vision

I couldn't find a good official announcement post to link to about these three newly released models, but this post on LocalLLaMA on Reddit has them in one place: Phi-3 small (7B), Phi-3 medium (14B) and Phi-3 vision (4.2B) (the previously released model was Phi-3 mini - 3.8B).

You can try out the vision model directly here, no login required. It didn't do a great job with my first test image though, hallucinating the text.

As with Mini these are all released under an MIT license.

UPDATE: Here's a page from the newly published Phi-3 Cookbook describing the models in the family.


Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet Big advances in the field of LLM interpretability from Anthropic, who managed to extract millions of understandable features from their production Claude 3 Sonnet model (the mid-point between the inexpensive Haiku and the GPT-4-class Opus). Some delightful snippets in here such as this one: We also find a variet

Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

Big advances in the field of LLM interpretability from Anthropic, who managed to extract millions of understandable features from their production Claude 3 Sonnet model (the mid-point between the inexpensive Haiku and the GPT-4-class Opus).

Some delightful snippets in here such as this one:

We also find a variety of features related to sycophancy, such as an empathy / “yeah, me too” feature 34M/19922975, a sycophantic praise feature 1M/847723, and a sarcastic praise feature 34M/19415708.

Via Hacker News


The Pragmatic Engineer

Startups on hard mode: Oxide. Part 1: Hardware

What is tougher than building a software-only or hardware-only startup? Building a combined hardware and software startup. This is what Oxide is doing, as they build a “cloud computer.” A deepdive.

👋 Hi, this is Gergely with a subscriber-only issue of the Pragmatic Engineer Newsletter. In every issue, I cover challenges at Big Tech and startups through the lens of engineering managers and senior engineers. To get articles like this in your inbox, every week, subscribe:

Subscribe now

What does an early-stage startup look like? Usually, there’s an office with devs working on laptops, a whiteboard with ideas, and lots of writing of code. Sometimes, there’s no physical office because team members work remotely, sharing ideas on virtual whiteboards.

But what about companies which aren’t “pure” software startups, and which focus on hardware? I recently got a glimpse of one in San Francisco, at the offices of Oxide Computer Company, which is a startup building a new type of server. I was blown away by the working environment and the energy radiating from it. This is their office:

Hive of activity: Oxide co-founder and CTO, Bryan Cantrill at Oxide’s office, with under-construction servers, hardware measurement tools, laptops, and cables

Some things are hard to describe without experiencing them, and this includes being at a hardware+software startup right as the first product is being finished, with the team already iterating on it. In today’s issue, we cover hardware at Oxide:

Why build a new type of cloud computer?

Building a networking switch from scratch

Using “proto boards” to build faster

A remote-first hardware company

Custom hardware manufacturing process

The importance of electrical engineers (EE)

Working closely with hardware vendors

1. Why build a new type of cloud computer?

If you want to build an application or service for millions of users, there are two main options for the infrastructure:

Cloud provider. AWS, GCP, Azure, Oracle, etc. Elsewhere, Uber is making the move from on-prem to GCP and Oracle, as previously covered.

On-premises (prem.) Operate your own servers, or more commonly rent rack space at a data center for them. This approach is sometimes called a “private cloud.” We’ve covered how and why booking platform Agoda is staying on-prem and how its private cloud is built. Social media site Bluesky also uses its own data centers since leaving AWS.

In data centers, the unit of measurement is “one rack.” A rack is a storage unit that can hold a few dozen servers; often referred to as “pizza box servers” because of their shape. Thicker types are called “rack servers.” Side note: an alternative to pizza box servers is blade servers, inserted into blade enclosures as building blocks in data centers. 

Here’s a “pizza box server” that online travel booking platform Agoda utilized heavily:

A server Agoda used in its private cloud. See Inside Agoda’s Private Cloud: Part 1

And here’s some commodity servers in Oxide’s office:

A typical server rack at Oxide’s office at 50% capacity, running Gigabyte and Tyan servers.

The rack in the image above was operating during my visit. It was loud and generated a lot of heat as expected, and there were lots of cables. It’s messy to look at and also to operate: the proprietary PC-era firmware causes security, reliability and performance issues. “PC-era” refers to the 1980s – early-2000s period, before x86 64-bit machines became the servers of choice.

Elsewhere, Big Tech companies have manufactured their own highly optimized racks and servers, but these aren’t for sale. The likes of Meta, Google, and Amazon no longer use traditional racks, and have “hyper-scaled” their servers to be highly energy efficient, easier to maintain, and with few cables.

Google’s custom-built servers. Source: Google

Joe Kava, VP of Google's Data Center Operations, described these racks back in 2017:

“Many of our racks don’t look like traditional server racks. They are custom designed and built for Google, so we can optimize the servers for hyper-efficiency and high-performance computing.”

Back to Oxide, whose vision is to build a cloud computer that incorporates the technological advances of Big Tech’s cloud racks, but makes them available to all. What if smaller tech companies could purchase energy-efficient servers like those that Meta, Amazon, Google and Microsoft have designed for themselves, and which customers of the big cloud providers like AWS, GCP, and Azure use – but without being locked in?

This is what the Oxide Computer offers, and I’ve seen one of its first racks. It appears similar in size to a traditional rack, but the company says it actually occupies 33% less space than a traditional rack, while offering the same performance. It’s much quieter than an everyday commodity server; in comparison the Gigabyte and Tyan servers are ear-splitting, and there are hardly any cables compared to a typical server.

Side and rear views of a fully-loaded Oxide rack. Fewer cables and SO MUCH quieter

The benefits of the Oxide computer compared to traditional racks:

Faster installation: installing a traditional rack typically takes weeks or months, says Bryan, because the servers need to be put in, wired up, and then tested. The Oxide rack comes fully assembled; it just needs to be slotted in at a data center.

Space and power efficiency. Uses less power and occupies less space. Less noise indicates superior power efficiency, with fans not needing to work as hard because the heat is channeled better.

Comes with integrated software to manage elastic infrastructure. With traditional rackmounted servers, it’s necessary to select software to manage virtual machines, like VMware, Metal as a Service, Proxmox Virtual Environment, and OpenStack Ironic. The Oxide cloud computer includes built-in virtualization for storage (an equivalent of AWS’s Elastic Block Store) and also networking (an alternative to virtual private clouds.) 

Oxide’s target customer is anyone running large-scale infrastructure on-prem for regulatory, security, latency, or economic reasons. The Oxide rack comes with 2,048 CPU cores (64 cores per “sled,” where one sled is Oxide’s version of a “rackmount server”,) 16-32TB of memory (512GB or 1TB of memory per sled) and 1PB (petabyte) of storage (32TB storage per sled). See full specification.

This kind of setup makes sense for companies that already operate thousands of CPU cores. For example, we previously covered how Agoda operated 300,000 CPU cores in its data centers in 2023; at such scale investing in racks like Oxide’s could make sense. Companies in the business of selling virtual machines as a service might also find this rack an interesting investment to save money on operations, compared to traditional racks.

An interesting type of customer are companies running thousands of CPU cores in the public cloud, but which are frustrated by network latencies. There’s a growing sense that multi-tenancy in public clouds; where one networking switch serves several racks and customers, causes worse latency which cannot be debugged or improved. In contrast, an Oxide rack offers dedicated rack space in data centers. Using these servers can also considerably reduce network latencies because the customer can choose the data center they use, based on their own regional needs. Customers also get full control over their networking and hardware stack – something not possible to do when using a cloud provider.

Oxide doesn’t target smaller startups that only need a few hundred CPU cores. For these businesses, using cloud providers, or buying/renting and operating smaller bare metal servers is the sensible solution.

2. Building a networking switch from scratch

In server manufacturing, where does innovation come from? I asked Bryan:

“Companies like Google, Meta and similar companies producing their custom hardware and software to build better servers, could bring competition to the market. However, it’s highly unlikely that these companies would release their servers as a commercial product. It’s not their business model. 

So, no, the next big server innovation will not come from Google or a similar company. It will come from a startup. And we want to be that startup.”

Oxide had to design two pieces of hardware from scratch: the switch and the server.

Why build a switch instead of integrating a third-party switch?

Oxide’s mission is to build their own cloud computer. Building a custom server usually means taking off-the-shelf components for a system and integrating it all together, including the server chassis, a reference design system board, and a separately-developed network switch. A “reference design” is a blueprint for a system containing comprehensive guidance on where to place its elements, that’s been certified to work as intended: it should not overheat, or cause unexpected interference.

However, Oxide also needed to build their own networking switch, as well as build a custom server – which is quite an undertaking! This need came from the constraint that Oxide wanted to control the entire hardware stack, end-to-end. A networking switch is a “mini-computer” in itself. So in practice, they designed and built two computers, not just one.

Producing a separate switch meant integrating a switching application-specific integrated circuit (ASIC), management CPU, power supplies, and physical network ports.

Oxide’s goals for this switch were:

Highly available operation. Each Oxide rack has two networking switches which operate simultaneously, as per high availability. If links to one switch have issues, or a switch needs to be serviced, then the servers can still access networks via the other switch, ensuring more reliable operation than with a single-switch setup. 

Integrated with control plane software. The single most important factor in Oxide’s decision was the desire to deliver a quality end-to-end experience for multi-tenant elastic infrastructure. The team knew from their experience of deploying public cloud infrastructure that the switch is often a nexus of reliability and performance issues.

Use the same “core” hardware as the server. The switch must use the same regulators and power controller as the Oxide servers.

Building the custom networking switch took around 2 years, from designing it in March 2020, to the first unit being assembled in January 2022.

The initial build of Oxide’s network switch, codenamed “Sidecar” The front view of the initial engineering build of Sidecar

Building custom hardware almost always comes with unexpected challenges. In the case of the networking switch, the team had to work around an incorrect voltage regulator on the board, marked with yellow tape in the image above.

3. Using proto boards to build hardware faster

Proto boards is short for “prototype printed circuit boards,” which help the company test small components to ensure they work independently. Once validated, those components can be used as building blocks.

“When we set out to build a server from scratch, we didn’t want to go straight to building the server motherboard. However, when we started we had zero full-time electrical engineers!

There’s a process I learned from robotics people in a previous job, called a ‘roadkill build.’ You get all the parts that will end up in the thing you eventually build, but instead of being integrated, they are all spread out across a bench with nasty cables between them, so that you can probe them, poke them, and replace them. We thought it would be a good idea to do this for the servers.”

First prototype board

For the initial prototype printed circuit board, the team started with the service processor. This was a well understood, very critical part of the server and the switch. The team decided to build it from two off-the shelf microcontrollers, and built a prototype board around this:

Starting out, the company took inspiration from robotics. Founding engineer, Cliff L. Biffle, shares:

The service processor’s prototype board, which facilitates all communication between components of the server

The team found that It’s possible to bite off more than you can chew, even with prototype circuit boards, The first board was a success in that it worked: upon start up, the hardware and software “came up” (in electrical engineering, “coming up” refers to the successful powering up and initializing of an electronic system board.) But the time it took to develop was longer than the team wanted.

It turns out this prototype board was too highly integrated, with too many moving parts and too many I/O pins. There was simply too much on the board for it to be productive. The team learned that progress would be faster with multiple, simpler boards as pluggable hardware modules, instead of one complicated board with lots of functionality and many fixed functions. As engineer Rick Altherr – who worked on the board – noted:

“We put too many things on. The concept we ran into was that an x86 server, with all of its management, is way too complicated. Let’s slim it down to just the management subsystems. This board is intended to be the service processor. But it turns out that even that’s too much. 

By having so many things be on one board, instead of pluggable modules, it meant that we committed to a lot of design choices. For example, the two ethernet jacks were never actually used because we changed our philosophy on how that was going to work, before we got the boards back from manufacturing.”

A more modular approach

Before building the service processor board, the team separated out the “root of trust” (RoT) functionality onto a separate board. The RoT hardware is the foundational base upon which the security and trust of the entire system are built. A RoT has “first instruction integrity,” guaranteeing exactly which instructions run upon startup. The RoT sets up the secure boot and permanently locks the device to ensure ongoing secure operation. Below is the prototype of Oxide’s RoT module:

The root of trust module; a separate, pluggable prototype

Other modules the Oxide team built included a power module, a multiplexer (a device with multiple input signals, which selects which signal to send to the output):

The Oxide team built many modules for prototyping and testing

Over time, the team found the right balance of how much functionality the prototype board needed. Below is an evolved prototype board version of the service processor:

Evolved version of the prototype service processor. Redundant ports like Ethernet are absent

The Oxide team calls this board a “workhorse” because they can plug in so many modules, and do so much testing and hardware and software development with it. Here’s an example:

The hardware version of modular development: a prototype service processor board with modules plugged in

A prototype board is unit testing for hardware. In software development, unit tests ensure that components continue to work correctly as the system is modified. Oxide found that prototype boards come pretty close to this approach, and allowed Oxide to iterate much faster on hardware design, than by manufacturing and validating test devices.

Using smart workarounds to iterate faster

Read more


Mike Jones: self-issued

Proposed Implementer’s Draft of OpenID Federation

The OpenID Connect working group has started working group last call (WGLC) for a proposed Implementer’s Draft of the OpenID Federation specification. As described in the WGLC message: OpenID Federation -35 has been published at https://openid.net/specs/openid-federation-1_0-35.html and https://openid.net/specs/openid-federation-1_0.html. This draft is being proposed as the fourth (and hopefully fi

The OpenID Connect working group has started working group last call (WGLC) for a proposed Implementer’s Draft of the OpenID Federation specification. As described in the WGLC message:

OpenID Federation -35 has been published at https://openid.net/specs/openid-federation-1_0-35.html and https://openid.net/specs/openid-federation-1_0.html. This draft is being proposed as the fourth (and hopefully final) Implementer’s Draft of the specification.

An Implementer’s Draft is a stable version of a specification providing intellectual property protections to implementers of the specification. The two-week working group last call ends on Friday, May 31, 2024. Unless reasons are identified during the last call to substantially revise the specification, the 45-day OpenID Foundation-wide review of the specification for approval as an OpenID Implementer’s Draft will shortly follow.

Special thanks to all the implementers and deployers who provided feedback to get us to this point!

Monday, 20. May 2024

Simon Willison

Quoting Scarlett Johansson

Last September, I received an offer from Sam Altman, who wanted to hire me to voice the current ChatGPT 4.0 system. He told me that he felt that by my voicing the system, I could bridge the gap between tech companies and creatives and help consumers to feel comfortable with the seismic shift concerning humans and AI. He said he felt that my voice would be comforting to people. After much consider

Last September, I received an offer from Sam Altman, who wanted to hire me to voice the current ChatGPT 4.0 system. He told me that he felt that by my voicing the system, I could bridge the gap between tech companies and creatives and help consumers to feel comfortable with the seismic shift concerning humans and AI. He said he felt that my voice would be comforting to people. After much consideration and for personal reasons, I declined the offer.

Scarlett Johansson


CRDT: Text Buffer

CRDT: Text Buffer Delightfully short and clear explanation of the CRDT approach to collaborative text editing by Evan Wallace (of Figma and esbuild fame), including a neat interactive demonstration of how the algorithm works even when the network connection between peers is temporarily paused. Via Lobste.rs

CRDT: Text Buffer

Delightfully short and clear explanation of the CRDT approach to collaborative text editing by Evan Wallace (of Figma and esbuild fame), including a neat interactive demonstration of how the algorithm works even when the network connection between peers is temporarily paused.

Via Lobste.rs


Ben Werdmüller

As clicks dry up for news sites, could Apple News be a lifeline?

"The free version of Apple News is one of the biggest news platforms in the world. It’s the most widely used news application in the United States, the U.K., Canada, and Australia, and boasted over 125 million monthly users in 2020." And publications are becoming dependent on it. I agree strongly with the journalist's view at the bottom of this piece: "It incentivize

"The free version of Apple News is one of the biggest news platforms in the world. It’s the most widely used news application in the United States, the U.K., Canada, and Australia, and boasted over 125 million monthly users in 2020."

And publications are becoming dependent on it.

I agree strongly with the journalist's view at the bottom of this piece:

"It incentivizes users to subscribe to Apple News+ rather than to publications directly, likely cannibalizing some potential revenue. It’s driving editorial decisions, meaning publishers are once again changing their content strategy to placate a platform. And of course the company could wake up one day and decide, like Facebook, that it no longer really wants to be in the news business, leaving news publishers stranded."

Newsrooms - say it with me - need to establish direct, first-party connections with their audiences. Anything else gives a third party too much supplier power over their businesses and presents an existential risk. Apple News is useful right now, but at its heart the dynamics that drive it are no different to Facebook or Twitter. There's nothing to say it's here for good, and there's nothing smart about letting Apple own your relationship with your readers. #Media

[Link]


The IndieWeb’s next stage?

"I want the IndieWeb to be a viable alternative to social media, gradually widening the audience beyond tech-savvy folks by making the tools easier to use and more reliable." This is what we were trying for with Known: something that felt social but was fully under the user's control. We had installers at third-party hosts; we had our own managed service; we had the open s

"I want the IndieWeb to be a viable alternative to social media, gradually widening the audience beyond tech-savvy folks by making the tools easier to use and more reliable."

This is what we were trying for with Known: something that felt social but was fully under the user's control. We had installers at third-party hosts; we had our own managed service; we had the open source code for people who wanted to use that directly.

The fediverse adds a missing piece here: Known suffered immensely from a blank page and no reader view when you logged in for the first time. Now we can build platforms that immediately connect people to a much wider social network that is outside of monolithic corporate control but also makes it (relatively) easy to find the people you care about.

A combination between the fediverse and indieweb is almost inevitable. This is what Ghost appears to be building today, for example, with its new integrated fediverse reader tool. WordPress may also be headed in that direction. And there will be many others.

A huge +1, also, to the idea that we can "manifest momentum by speaking aloud your dreams and letting others share them with you". This is how community-building works.

And, for the record, I'm all-in. #Technology

[Link]


Patrick Breyer

Piraten-Europaabgeordneter zu Assange-Urteil: Whistleblower ermutigen statt sie mundtot zu machen!

Zu der heutigen Entscheidung des britischen High Court, die Berufung von Wikileaks-Gründer Julian Assange gegen seine Auslieferung an die USA zuzulassen, erklärt der Europaabgeordnete Dr. Patrick Breyer: „Die heutige Gerichtsentscheidung gibt …

Zu der heutigen Entscheidung des britischen High Court, die Berufung von Wikileaks-Gründer Julian Assange gegen seine Auslieferung an die USA zuzulassen, erklärt der Europaabgeordnete Dr. Patrick Breyer:

„Die heutige Gerichtsentscheidung gibt Hoffnung, dass europäische Menschenrechte und Gerichte die Aufdeckung von Unrecht, Machtmissbrauch, Korruption und Kriegsverbrechen schützen, ohne Freund und Feind mit zweierlei Maßstäben zu messen. Ich wünsche mir, dass aus der von den USA gewollten Abschreckung von Whistleblowern und Journalisten weltweit ihre Ermutigung durch ein Grundsatzurteil wird.

Dass Zensursula von der Leyen zu dem Fall schweigt, ist beschämend, aber bezeichnend für die Mutter der Pläne zur Chatkontrolle, die ihre eigenen dienstlichen SMS mit Pharmakonzernen geheim halten will. Für mich als Pirat ist Transparenz von Regierungshandeln die unverzichtbare Grundlage jeder Demokratie. Transparenz darf kein Verbrechen sein!“

Breyer veranstaltet heute um 15 Uhr ein Webinar mit Rechtsanwalt Sören Schomburg aus Assanges Verteidigungsteam und Marie Wasilewski als Sprecherin des Aktivistenbündnisses FreeAssange.EU, um über die Gerichtsentscheidung und die nächsten Schritte zu sprechen.

Freitag hatte 31 Europaabgeordnete einen von Breyer initiierten Brief an den britischen Innenminister gesandt, in dem die Freilassung von Assange gefordert wird. Zu den Unterzeichnern gehören die deutschen Europaabgeordneten Martin Sonneborn (die PARTEI) und Erik Marquardt (Bündnis 90/Grüne).


Ben Werdmüller

ShareOpenly is now on Tedium

I adore the way ShareOpenly has been added to Tedium: You can see it for yourself on all its posts, including this great one about the decline of the ball mouse. Its founder, Ernie Smith, told me: “figured I had to have fun with it”.

I adore the way ShareOpenly has been added to Tedium:

You can see it for yourself on all its posts, including this great one about the decline of the ball mouse. Its founder, Ernie Smith, told me: “figured I had to have fun with it”.

Sunday, 19. May 2024

Simon Willison

Spam, junk … slop? The latest wave of AI behind the ‘zombie internet’

Spam, junk … slop? The latest wave of AI behind the ‘zombie internet’ I'm quoted in this piece in the Guardian about slop: I think having a name for this is really important, because it gives people a concise way to talk about the problem. Before the term ‘spam’ entered general use it wasn’t necessarily clear to everyone that unwanted marketing messages were a bad way to behave. I’m hoping

Spam, junk … slop? The latest wave of AI behind the ‘zombie internet’

I'm quoted in this piece in the Guardian about slop:

I think having a name for this is really important, because it gives people a concise way to talk about the problem.

Before the term ‘spam’ entered general use it wasn’t necessarily clear to everyone that unwanted marketing messages were a bad way to behave. I’m hoping ‘slop’ has the same impact – it can make it clear to people that generating and publishing unreviewed AI-generated content is bad behaviour.


NumFOCUS DISCOVER Cookbook: Minimal Measures

NumFOCUS DISCOVER Cookbook: Minimal Measures NumFOCUS publish a guide "for organizers of conferences and events to support and encourage diversity and inclusion at those events." It includes this useful collection of the easiest and most impactful measures that events can put in place, covering topics such as accessibility, speaker selection, catering and provision of gender-neutral restrooms.

NumFOCUS DISCOVER Cookbook: Minimal Measures

NumFOCUS publish a guide "for organizers of conferences and events to support and encourage diversity and inclusion at those events."

It includes this useful collection of the easiest and most impactful measures that events can put in place, covering topics such as accessibility, speaker selection, catering and provision of gender-neutral restrooms.


Fast groq-hosted LLMs vs browser jank

Fast groq-hosted LLMs vs browser jank Groq is now serving LLMs such as Llama 3 so quickly that JavaScript which attempts to render Markdown strings on every new token can cause performance issues in browsers. Taras Glek's solution was to move the rendering to a requestAnimationFrame() callback, effectively buffering the rendering to the fastest rate the browser can support. Via lobste.rs

Fast groq-hosted LLMs vs browser jank

Groq is now serving LLMs such as Llama 3 so quickly that JavaScript which attempts to render Markdown strings on every new token can cause performance issues in browsers.

Taras Glek's solution was to move the rendering to a requestAnimationFrame() callback, effectively buffering the rendering to the fastest rate the browser can support.

Via lobste.rs


A Plea for Sober AI

A Plea for Sober AI Great piece by Drew Breunig: “Imagine having products THIS GOOD and still over-selling them.”

A Plea for Sober AI

Great piece by Drew Breunig: “Imagine having products THIS GOOD and still over-selling them.”

Saturday, 18. May 2024

Michael Ruminer

Trust Over IP in Decentralized Information and the Role of Delegates

As I read the Issuer Requirements Guide from Trust Over IP Foundation it only reinforces my long-standing belief that well-known trusted proxy/witness/delegate issuers will serve an important role. Every organization or even many organizations will not want to maintain the governance needs to be a trusted issuer and instead will outsource that activity to a delegate that can ensure integrity and g

As I read the Issuer Requirements Guide from Trust Over IP Foundation it only reinforces my long-standing belief that well-known trusted proxy/witness/delegate issuers will serve an important role. Every organization or even many organizations will not want to maintain the governance needs to be a trusted issuer and instead will outsource that activity to a delegate that can ensure integrity and governance over their issuing activities.

For example, a gig platform such as Uber will not issue proof of work and proof of income credentials but will instead have a delegate do this on their behalf. The overhead for them in proper governance would be too steep. Instead the proxy/witness/delegate will collect the information from the source and issue a credential on their behalf. The delegate would be the trusted party in Trust Over IP (TOIP).

We see this in today’s world with service bureaus for different activities. The future will not be quite as decentralized as the potential for it to be because orgs will not wish to maintain the needed structures to ensure it. There is a place for well-known delegates.


Simon Willison

AI counter app from my PyCon US keynote

AI counter app from my PyCon US keynote In my keynote at PyCon US this morning I ran a counter at the top of my screen that automatically incremented every time I said the words "AI" or "artificial intelligence", using vosk, pyaudio and Tkinter. I wrote it in a few minutes with the help of GPT-4o - here's the code I ran as a GitHub repository. I'll publish full detailed notes from my talk once

AI counter app from my PyCon US keynote

In my keynote at PyCon US this morning I ran a counter at the top of my screen that automatically incremented every time I said the words "AI" or "artificial intelligence", using vosk, pyaudio and Tkinter. I wrote it in a few minutes with the help of GPT-4o - here's the code I ran as a GitHub repository.

I'll publish full detailed notes from my talk once the video is available on YouTube.


Quoting Patrick Reynolds

I rewrote it [the Oracle of Bacon] in Rust in January 2023 when I switched over to TMDB as a data source. The new data source was a deep change, and I didn’t want the headache of building it in the original 1990s-era C codebase. — Patrick Reynolds

I rewrote it [the Oracle of Bacon] in Rust in January 2023 when I switched over to TMDB as a data source. The new data source was a deep change, and I didn’t want the headache of building it in the original 1990s-era C codebase.

Patrick Reynolds

Friday, 17. May 2024

Simon Willison

Understand errors and warnings better with Gemini

Understand errors and warnings better with Gemini As part of Google's Gemini-in-everything strategy, Chrome DevTools now includes an opt-in feature for passing error messages in the JavaScript console to Gemini for an explanation, via a lightbulb icon. Amusingly, this documentation page includes a warning about prompt injection: Many of LLM applications are susceptible to a form of abuse kn

Understand errors and warnings better with Gemini

As part of Google's Gemini-in-everything strategy, Chrome DevTools now includes an opt-in feature for passing error messages in the JavaScript console to Gemini for an explanation, via a lightbulb icon.

Amusingly, this documentation page includes a warning about prompt injection:

Many of LLM applications are susceptible to a form of abuse known as prompt injection. This feature is no different. It is possible to trick the LLM into accepting instructions that are not intended by the developers.

They include a screenshot of a harmless example, but I'd be interested in hearing if anyone has a theoretical attack that could actually cause real damage here.

Via Hacker News


Commit: Add a shared credentials relationship from twitter.com to x.com

Commit: Add a shared credentials relationship from twitter.com to x.com A commit to shared-credentials.json in Apple's password-manager-resources repository. Commit message: "Pour one out." Via @rmondello@hachyderm.io

Commit: Add a shared credentials relationship from twitter.com to x.com

A commit to shared-credentials.json in Apple's password-manager-resources repository. Commit message: "Pour one out."

Via @rmondello@hachyderm.io


Quoting Kelsey Piper

I have seen the extremely restrictive off-boarding agreement that contains nondisclosure and non-disparagement provisions former OpenAI employees are subject to. It forbids them, for the rest of their lives, from criticizing their former employer. Even acknowledging that the NDA exists is a violation of it. If a departing employee declines to sign the document, or if they violate it, they can lo

I have seen the extremely restrictive off-boarding agreement that contains nondisclosure and non-disparagement provisions former OpenAI employees are subject to. It forbids them, for the rest of their lives, from criticizing their former employer. Even acknowledging that the NDA exists is a violation of it.

If a departing employee declines to sign the document, or if they violate it, they can lose all vested equity they earned during their time at the company, which is likely worth millions of dollars.

Kelsey Piper


Ben Werdmüller

Share Openly: A simple icon for a new social sharing service

A lovely blog post by Jon Hicks on his process for creating the ShareOpenly icon. Characteristically, lots of care and attention went into this. I'm really glad you get to see the open hand icons, which we eventually decided against, but feel really warm and human. Jon's amazing, lovely to work with, and has a really impressive body of work. I'm grateful he was able to c

A lovely blog post by Jon Hicks on his process for creating the ShareOpenly icon. Characteristically, lots of care and attention went into this.

I'm really glad you get to see the open hand icons, which we eventually decided against, but feel really warm and human.

Jon's amazing, lovely to work with, and has a really impressive body of work. I'm grateful he was able to contribute such an important part of this personal project. #Technology

[Link]


A new Ani DiFranco album is something to celebrate

I’ve been following Ani DiFranco for decades. I’ve seen her play live around twenty times: she always brings a kind of joyful, progressive energy that leaves me motivated and buzzing. She has a new album out, and it feels like a return to visceral, honest form. It’s not quite the acoustic punk from the late nineties / early aughts — seriously, go check out Living in Clip, Not a Pretty Girl or D

I’ve been following Ani DiFranco for decades. I’ve seen her play live around twenty times: she always brings a kind of joyful, progressive energy that leaves me motivated and buzzing.

She has a new album out, and it feels like a return to visceral, honest form. It’s not quite the acoustic punk from the late nineties / early aughts — seriously, go check out Living in Clip, Not a Pretty Girl or Dilate — and it goes to some really experimental places, but I’m into it. This time, rather than making it on her own, she’s worked with producer BJ Burton, who’s also worked with Bon Iver and Taylor Swift.

We need progressive, momentum-bringing, energetic music more than ever. Ani delivers. And even the name of the album itself — Unprecedented Sh!t — feels very apt for the era.

From the liner notes:

The title Unprecedented Sh!t is not only representative of how much of a sonic departure the 11-track album is from Ani’s other work, but also a political and social commentary on the current state of the world. “We find ourselves in unprecedented times in many ways, faced with unprecedented challenges. So, our responses to them and our discourse around them, need to rise to that level.”

Amen.


Apple, SpaceX, Microsoft return-to-office mandates drove senior talent away

"Taken together, our findings imply that return to office mandates can imply significant human capital costs in terms of output, productivity, innovation, and competitiveness for the companies that implement them." There's no doubt that there's a lot of value in being in the same physical room together; I'm writing this on the day after a work summit that brought my team to

"Taken together, our findings imply that return to office mandates can imply significant human capital costs in terms of output, productivity, innovation, and competitiveness for the companies that implement them."

There's no doubt that there's a lot of value in being in the same physical room together; I'm writing this on the day after a work summit that brought my team together from across the country, and I'm still buzzing from the energy. But I think anyone in tech that proposes a full-time return to office policy needs to rethink.

It comes down to this: "it's easier to manage a team that's happy". People want their lives and contexts to be respected; everyone's relationship with their employers has been reset over the last few years. This goes hand in hand with the resurgence of unions, too: the contract between workers and employers is being renegotiated, and particularly for parents and carers, but really for everyone, working from home yields a kind of freedom that's hard to replace. And asking people to come back reads as a lack of trust and autonomy that erodes relationships and decimates morale. #Business

[Link]


Simon Willison

PSF announces a new five year commitment from Fastly

PSF announces a new five year commitment from Fastly Fastly have been donating CDN resources to Python—most notably to the PyPI package index—for ten years now. The PSF just announced at PyCon US that Fastly have agreed to a new five year commitment. This is a really big deal, because it addresses the strategic risk of having a key sponsor like this who might change their support policy based

PSF announces a new five year commitment from Fastly

Fastly have been donating CDN resources to Python—most notably to the PyPI package index—for ten years now.

The PSF just announced at PyCon US that Fastly have agreed to a new five year commitment. This is a really big deal, because it addresses the strategic risk of having a key sponsor like this who might change their support policy based on unexpected future conditions.

Thanks, Fastly. Very much appreciated!


Talking Identity

Talking Ethics in Identity at Identiverse 2024

Identiverse being the best identity conference around means that it’s always a challenge coming up with a talk proposal, as Andrew Hindle and team raise the bar each year. The process acts as a forcing function for me to think about the topics I want to bring to the community for discussion, topics that I […]

Identiverse being the best identity conference around means that it’s always a challenge coming up with a talk proposal, as Andrew Hindle and team raise the bar each year. The process acts as a forcing function for me to think about the topics I want to bring to the community for discussion, topics that I encounter all the time in #DayJob, but seldom have time to dig into.

This year I wanted to build on my talk from last year, but realized I couldn’t tackle it alone. Thankfully, I managed to con(vince) two absolute rock stars in Michelle Dennedy and Eve Maler to join me on stage for a fireside chat about the ethics imperative facing our industry.

Screenshot

As the abstract says, “What happens when you throw a privacy wonk, a standards guru, and a product architect that have been working in identity far too long on to a conference stage together?” Join us in Vegas and find out.


Simon Willison

Programming mantras are proverbs

Programming mantras are proverbs I like this idea from Luke Plant that the best way to think about mantras like "Don’t Repeat Yourself" is to think of them as proverbs that can be accompanied by an equal and opposite proverb. DRY, "Don't Repeat Yourself" matches with WET, "Write Everything Twice". Proverbs as tools for thinking, not laws to be followed. Via lobste.rs

Programming mantras are proverbs

I like this idea from Luke Plant that the best way to think about mantras like "Don’t Repeat Yourself" is to think of them as proverbs that can be accompanied by an equal and opposite proverb.

DRY, "Don't Repeat Yourself" matches with WET, "Write Everything Twice".

Proverbs as tools for thinking, not laws to be followed.

Via lobste.rs


Patrick Breyer

EU-Abgeordnete fordern die britische Regierung auf, Julian Assanges Auslieferung zu stoppen

Heute haben sich auf Initiative des EU-Abgeordneten Dr. Patrick Breyer (Piratenpartei) 31 Europaabgeordnete verschiedener Fraktionen in einem offenen Brief an den britischen Innenminister James Cleverly gewandt und …

Heute haben sich auf Initiative des EU-Abgeordneten Dr. Patrick Breyer (Piratenpartei) 31 Europaabgeordnete verschiedener Fraktionen in einem offenen Brief an den britischen Innenminister James Cleverly gewandt und dringlich appelliert, die Auslieferung von Wikileaks-Gründer Julian Assange an die Vereinigten Staaten zu stoppen. Dies geschieht im Vorfeld der bevorstehenden Gerichtsentscheidung am 20. Mai 2024, die voraussichtlich final über Assanges Schicksal entscheiden wird.

In dem Brief wird die britische Regierung aufgefordert, ihre Verantwortung im Hinblick auf Menschenrechte und Pressefreiheit wahrzunehmen. „Genau wie die EU-Kommission verbreitet die britische Regierung die Mär, über die Auslieferung hätten ausschließlich die Gerichte zu entscheiden. § 70 (2) des britischen Auslieferungsgesetzes gibt dem Innenminister die Befugnis, die Auslieferung zu verweigern, wenn sie gegen das Recht auf Leben oder das Verbot der Folter und unmenschlicher oder erniedrigender Behandlung der Europäischen Menschenrechtskonvention verstoßen würde“, erklärt Breyer. „Die psychische Gesundheit von Julian Assange, die potenziellen Haftbedingungen in den USA und das reale Risiko eines Suizids bei Auslieferung führen dazu, dass die Auslieferung eine solche unmenschliche oder erniedrigende Behandlung darstellen würde.“

Die Abgeordneten betonen in ihrem Schreiben außerdem, dass die Verfolung von Julian Assange politisch motiviert sei. Die Bestimmungen des britisch-amerikanischen Auslieferungsvertrags verbieten zu Recht die Auslieferung wegen politischer Straftaten. Die eindeutig politische Natur dieses Falls wird durch zahlreiche und hochgradig voreingenommene Aussagen führender Persönlichkeiten der US-Politik deutlich, die seit mindestens 2011 die extralegale Bestrafung oder Ermordung von Herrn Assange fordern.

Ein weiterer Kritikpunkt ist das Fehlen einer Garantie der US-Regierung, dass Assange vor Gericht dieselben Rechte wie ein US-Bürger erhalten würde. „Ein Verfahren gegen jemanden, der Dokumente veröffentlicht hat, in einem Land, das möglicherweise grundlegende Rechte auf Meinungs- und Pressefreiheit nicht anerkennt oder anwendet, ist inakzeptabel“, so Breyer.

Breyer und die anderen unterzeichnenden EU-Abgeordneten fordern die britische Regierung auf, die Pressefreiheit und das Recht auf freie Meinungsäußerung zu schützen und die Auslieferung von Julian Assange zu stoppen.

Die Unterzeichner des Briefes schließen sich den Forderungen von großen Organisationen wie Amnesty International und Reporter ohne Grenzen an, die die sofortige Freilassung von Julian Assange verlangen. Sie betonen, dass seine anhaltende Inhaftierung das Recht auf freie Meinungsäußerung gefährdet.


Doc Searls Weblog

Blog + Newsletter

Newsletters are all the rage now. In recognition of that, I blogged here two years ago about the idea of writing a solo newsletter. Since then I’ve been co-producing this one with Katherine Druckman at Reality 2.o. It’s a Substack one, so I know how that game works on the production as well as the […]

Newsletters are all the rage now. In recognition of that, I blogged here two years ago about the idea of writing a solo newsletter. Since then I’ve been co-producing this one with Katherine Druckman at Reality 2.o. It’s a Substack one, so I know how that game works on the production as well as the consumption ends.

Recently I also learned that WordPress makes it easy to turn blogs into newsletters, which is why you see “Get New Posts by Email” in the right column here. So, wanting an image to go with the news that this blog can now be a newsletter if you like, I said to ChatGPT, “Draw me Doc Searls blogging on the left side of the image, and producing a newsletter on the right side.” It gave me this, including the caption:

Here’s the illustration depicting Doc Searls engaging in two different activities. On the left, he is blogging, and on the right, he is producing a newsletter. This split scene captures his multitasking abilities in digital content creation.

Except for the slouch, I never looked like that. But what the hell. AI is wacky shit, so there ya go.

Feel free to subscribe.

Thursday, 16. May 2024

The Pragmatic Engineer

The Pulse #93: OpenAI makes Google dance

Google’s core search business has never been under as much threat as it is today – and the threat comes from OpenAI. Also: Google Cloud deletes a customer’s infra, and Tesla in trouble.

The Pulse is a series covering insights, patterns, and trends within Big Tech and startups. Notice an interesting event or trend? Send me a message.

Today, we cover:

Industry pulse. NetBSD and Gentoo ban AI-generated commits, StackOverflow to sell data to OpenAI, AWS losing market share, jail time for for Tornado Cash developer, and more. 

OpenAI makes Google dance. OpenAI is setting the pace for Google with AI phone assistant capabilities, and is probably the reason that Google started shipping AI answers on top of search results.

Google Cloud deletes Australian trading fund’s infra. A $124B fund in Australia would have lost all data stored with Google Cloud, had they not relied on a third-party backup. A rare blunder from GCP, where regional replication did not stop the deletion – and a just as rare statement from Google Cloud’s CEO taking the blame.

Tesla trouble accompanies poorly executed layoffs. The market leader for electric vehicles in the US is seeing demand for cars drop, and is responding with mass layoffs. This included firing the 500-person Supercharger team: only to start to rehire them a week later.

1. Industry pulse Open source projects ban AI-generated code commits

Read more


Ben Werdmüller

Google’s broken link to the web

"A quarter-century into its existence, a company that once proudly served as an entry point to a web that it nourished with traffic and advertising revenue has begun to abstract that all away into an input for its large language models." This has the potential to be a disaster for the web and everyone who depends on it: for journalism, for bloggers, for communities, for every

"A quarter-century into its existence, a company that once proudly served as an entry point to a web that it nourished with traffic and advertising revenue has begun to abstract that all away into an input for its large language models."

This has the potential to be a disaster for the web and everyone who depends on it: for journalism, for bloggers, for communities, for every voice that couldn't be heard without an open, egalitarian platform.

The answer for all of those stakeholders has to be depending on forging real, direct relationships with real people. It doesn't scale; it doesn't fit well with a unidirectional broadcast model for publishing; it's now how most people who make content think about what they do. But it's how all of them are going to survive and continue to find each other.

I've been urging publishers to stop using the word "audience" and to replace it with "community", and to think about what verb might replace "publish" in a multi-directional web that is more about relationships than it is reaching mass eyeballs.

Of course, it might go in a direction we haven't predicted. We'll find out very soon; the only real certainty is that things are changing, and the bedrock that many people have depended on for two decades is shifting. #Technology

[Link]


Mozilla Foundation Welcomes Nabiha Syed as Executive Director

This is great news for Mozilla, for everyone who uses the internet, and for everyone who cares about ethics, privacy, and human rights. We need a well-functioning Mozilla more than ever - and that much-needed presence has been absent for years. The spirit in the following quote gives me a lot of hope - I think this is how all technology should be built, and how all techn

This is great news for Mozilla, for everyone who uses the internet, and for everyone who cares about ethics, privacy, and human rights.

We need a well-functioning Mozilla more than ever - and that much-needed presence has been absent for years.

The spirit in the following quote gives me a lot of hope - I think this is how all technology should be built, and how all technologists should approach their work, but it's rarely true:

“After all, the technology we have now was once just someone’s imagination. We can dream, build, and demand technology that serves all of us, not just the powerful few.”

I hope - and believe - that she can make it happen. #Technology

[Link]


Patrick Breyer

Piraten zur Verurteilung eines Tornado Cash-Programmierers: Privatsphäre ist kein Verbrechen!

Ein niederländisches Gericht hat gestern den Programmierer Alexey Pertsev zu fünf Jahren Haft verurteilt. Er sei deshalb der Geldwäsche schuldig, weil die von ihm entwickelte Software „Tornado Cash“ auch Straftätern vollständig …

Ein niederländisches Gericht hat gestern den Programmierer Alexey Pertsev zu fünf Jahren Haft verurteilt. Er sei deshalb der Geldwäsche schuldig, weil die von ihm entwickelte Software „Tornado Cash“ auch Straftätern vollständig anonyme und nicht rückverfolgbare Krypto-Transaktionen ermögliche (sog. „crypto mixer“). „Die Wahrung der Anonymität des Nutzers und die Verschleierung des Transaktionsverlaufs standen dabei im Mittelpunkt,“ so der Vorwurf des Gerichts. Aufgrund ihrer Funktionsweise sei die Software „speziell für Straftäter bestimmt“. In 36 Fällen seien gestohlene Kryptowährungen mithilfe der Software anonym in Umlauf gebracht worden, wofür der Programmierer verantwortlich sei. Das Gericht warf dem Angeklagten in seinem Urteil eine „Ideologie maximaler Privatsphäre“ vor.

Der Europaabgeordnete der Piratenpartei und Jurist Dr. Patrick Breyer warnt vor den Konsequenzen der Verurteilung: „Dieses Urteil kriminalisiert legitime Anonymität und alle, die sie als Programmierer ermöglichen. Die bei Bargeld selbstverständliche Anonymität, die unsere finanzielle Freiheit schützt, darf bei Digitalwährungen nicht kriminalisiert werden.

Die Konsequenzen dieser Denkweise könnten weit über Kryptowährungen hinaus auch Programmierer von Messengersoftware oder Anonymisierungsnetzwerke treffen. Aus diesem Geist heraus hat die EU zuletzt anonyme Barzahlungen eng begrenzt und mit der Chatkontrolle die Zerstörung des digitalen Briefgeheimnisses vorgeschlagen.

In jeder Freiheit nur den Missbrauch durch Einzelne zu sehen, schafft Unfreiheit und ist eine autoritäre Denkweise. Wir Piraten sind digitale Freiheitskämpfer, weil Freiheit Grundlage unserer Gesellschaft ist und ihr weit mehr nützt als der befürchtete Schaden durch ihren kriminellen Missbrauch.“

Die Spitzenkandidatin der Piratenpartei zur Europawahl und Informatikerin Anja Hirschel erklärt: „Wir Piraten sind klar pro Anonymisierung. Die Programmierung, der Betrieb und die Verwendung von Privacy-Tools muss diskriminierungsfrei möglich sein. Aus ihrer Verwendung reflexartig strafbare Handlungen unterstellen zu wollen ist grundlegend falsch.”


Simon Willison

Quoting Richard Schneeman

[...] by default Heroku will spin up multiple dynos in different availability zones. It also has multiple routers in different zones so if one zone should go completely offline, having a second dyno will mean that your app can still serve traffic. — Richard Schneeman

[...] by default Heroku will spin up multiple dynos in different availability zones. It also has multiple routers in different zones so if one zone should go completely offline, having a second dyno will mean that your app can still serve traffic.

Richard Schneeman


Webistemology - John Wunderlich

Bill 194, Strengthening Cyber Security and Building Trust in the Public Sector Act, 2024

While Bill 194 introduces some welcome upgrades to Ontario's cybersecurity and privacy legislation, it falls short of delivering in several key areas, particularly protecting employees' privacy.

This post intends to contribute to the public debate on what could have been a significant legislation - Bill 194. This post is not a summary of Bill 194. I am not a lawyer, and this is not a legal analysis. The post below draws on my experience as a privacy and data protection expert and understanding of current standards and legislation. I will provide an overview of the bill's scope, goals, and provisions and assess its potential to enhance Ontario’s cybersecurity and respect the privacy of Ontarians. While Bill 194 introduces some welcome upgrades to Ontario's cybersecurity and privacy legislation, it falls short of delivering in several key areas, particularly protecting employees' privacy.

Overview

Bill 194, Strengthening Cyber Security and Building Trust in the Public Sector Act, 2024 (the Bill), was introduced in the Ontario Legislature for first reading and passed on May 13, 2024. It has been ordered for its Second Reading. Bill 194 has been introduced in the current context of the ongoing evolution of cybersecurity and privacy threats and the explosive growth of artificial intelligence. The Bill is, therefore, not surprising in what it is intended to address:

The Act addresses cyber security and artificial intelligence systems at public sector entities. Public sector entities are institutions within the meaning of the Freedom of Information and Protection of Privacy Act and the Municipal Freedom of Information and Protection of Privacy Act, children’s aid societies and school boards. (See explanatory note.)

It is worth noting that the Bill does not make amendments to the Municipal Freedom of Information and Protection of Privacy - MFIPPA (the sister act to the Freedom of Information and Protection of Privacy Act - FIPPA). Hopefully, this can be addressed as the Bill goes through the legislative process.

It must be said that if one of the government's objectives in the Bill were to improve cyber security and privacy protections for Ontarians, this would have been a golden opportunity to introduce private sector legislation to provide a made-in-Ontario solution that could supplement and extend the protections offered by Federal legislation and ensure that Ontarians have robust and equivalent protection in both the public and private sectors. In particular, the government of Ontario's failure to protect employees' privacy is a long-standing issue highlighted by the gaps in this legislation. I note that the current Federal private-sector privacy law is due to be superseded by the contents of Bill C-27, but that is not part of this post.

Employees in Ontario do not have legislation that protects their privacy in either the public or the public sector. Public sector privacy protections were removed in 1995, making Ontario unique among Canadian provinces in that it does not protect the privacy of civil servants at work. It is also the case that, due to employment being in provincial jurisdiction, Federal private-sector privacy legislation does not protect employees in Ontario.

Ontario-based employees in the federal public sector or employed under the federal labour code (entities like banks, for example) have privacy protection under federal legislation. Still, those are estimated to be less than 500,000 of Ontario's nearly 8 million employees or slightly more than 6%. In the private sector, employees under collective agreements, based on arbitral jurisprudence and the specifics of their contract, will have privacy protection, but that accounts for less than 14% of private sector works. I derived these numbers mainly from searching for available Statistics Canada and other online sources.

TL;DR — employees in Ontario are the least likely to have privacy protection at work compared to other provinces or territories.

The Bill

The Bill has two significant elements. Schedule 1, “Enhancing Digital Security and Trust Act,” addresses cyber security issues, the use of artificial intelligence systems, the impact of digital technology on children, and some general provisions, all of which will be addressed below. Schedule 2, “Freedom of Information and Protection of Privacy Act,” amends the Freedom of Information and Protection of Privacy Act, RSO 1990, c F.31. Bill 194 is 51 pages long. From a content perspective, that is about 17 pages in English, with a matching section in French. If you think, "This seems a bit perfunctory, given the complicated nature of cyber security, digital protection of children, and privacy," you would be right. It seems to me that the entire bill could be summarized by saying that the government recognizes the importance of issues and will, therefore, write and implement regulations sometime in the future to deal with them. "Just trust us and pass the bill." When you compare this to the 4 years of discussion that went into creating the 458-page EU Artificial Intelligence Act, it comes up short, literally and figuratively. Closer to home, Bill C-27, which includes the Artificial Intelligence Data Act, is 148 pages (or 74 pages in English) but is accompanied by more than 100 civil society, industry, and expert submissions on the provisions and issues of the bill.

Schedule 1, Enhancing Digital Security and Trust Act

The following describes some of the more significant elements of this part of the Act. This includes Definitions (s. 1), Cyber Security (s. 2 - 4), Use of Artificial Intelligence Systems (s. 5 - 8), and Digital Technology Affecting Individuals Under Age 18 (s. 9 - 11), and some concluding general sections.

Definitions

The Bill adds a definition of artificial intelligence that appears to be derived, at least in part, from the definition of an AI system in Article 3 of the EU Artificial Intelligence Act. (An easier-to-use reference than the official text can be found in the AI Act Explorer prepared by The Future of Life Institute). It may be summarized as any system that infers from input to generate outputs to accomplish explicit or implicit objectives. Using an AI chatbot is an example that is fulfilled by this definition. A sample of definitions that are included in the AI Act but not this act include:

reasonably foreseeable misuse safety component training data input data

It is good that the Bill includes procured services and systems as a "use" of artificial intelligence systems. Still, much of the success of this approach will be determined by the nature of the due diligence in Ontario Public Service (OPS) procurement requirements for AI and machine learning systems. Another positive inclusion is that digital information includes collection, use, retention or disclosure by a third party. This will help ensure that accountability remains with the originating government institution.

Cyber Security

This part of Bill 194 boils down to a requirement for the government to make regulations governing cyber security, including s. 2 (1):

requiring public sector entities to develop and implement programs for ensuring cyber security; governing programs mentioned in clause (1), which may include prescribing elements to be included in the programs; requiring public sector entities to submit reports to the Minister or a specified individual in respect of incidents relating to cyber security, which may include different requirements in respect of different types of incidents; prescribing the form and frequency of reports.

In the absence of a public consultation on the content and purpose of the governing regulations, there is no assurance that the regulations that will be promulgated will meet diverse stakeholder needs nor that they will be effective in proving the desired effect of protecting security. While section 3 allows the government to make regulations setting technical standards, the devil will be in the details here. Noting that there are boatloads of security standards to choose from. There needs to be governance to ensure that the standards chosen are enforced. For example, I have been a consultant on several projects inside various Ministries, and it sometimes surprises information architects and project managers that there are Government of Ontario Information and Technology Standards (GO-ITS) to which their projects should abide. There is nothing to suggest in the Bill that even if good standards are adopted, they will be enforced with any rigour.

Use of Artificial Intelligence Systems

This part of Bill 194, similar to the prior section, mainly sets out the authority for the government to make regulations to govern the use of AI systems without creating content that could be publicly reviewed or publicly debated. I will note two particular gaps I feel should be addressed.

Developing an accountability framework

Section 5. (3) of the Bill states that each entity using artificial intelligence systems will develop and implement an accountability framework following the yet-to-be-published regulations. I will highlight what I believe to be two flaws with this approach.

There are no assurances in the Bill that marginalized or disadvantaged communities will provide input or be engaged in developing an Accountability Framework for an artificial intelligence system that may significantly impact their lives. Secondly, it appears that the approach in this Bill could lead to a proliferation of entity-specific Accountability Frameworks. This burdens both citizens whose data may be processed in multiple artificial intelligence systems with different frameworks and entities without the appropriate accountability expertise being asked to develop and implement their frameworks.

Rather than a proliferation of frameworks, creating a single Accountability Framework based on transparent, inclusive, and robust stakeholder engagement would be better.

Creating a risk framework

All that Bill 194 says on managing the risk of using artificial intelligence systems is, "A public sector entity to which this section applies shall take such steps as may be prescribed to manage risks associated with the use of the artificial intelligence system." This is woefully inadequate. The high-level risks and harms that can be created using artificial intelligence need to be articulated so that systems that may create high risks to individuals or Ontario as a whole can be identified, and those risks and harms can be identified and either avoided or mitigated. There is no identification of what might be termed unacceptable uses of AI systems or a way to identify whether a high-risk AI system - such as a system that collects biometric information about Ontarians and uses that as a basis for determining access to systems - is acceptable. (In my mind, such a system is inherently unacceptable.)

Digital Technology Affecting Individuals Under Age 18

This section replicates the section above; it essentially boils down to allowing the government to make regulations that

set out how children's information may be collected, used, or disclose require reports about how children's information may be collected, used, or disclosed may prohibit some processing of children's information

I have two broad comments here. The first is that I am somewhat relieved that the government is not trying to introduce broad systems of digital control or censorship in the name of protecting children. Such legislation is usually both overly broad and ineffective in its intended purpose. That isn't to say that there aren't real risks to students that could have been articulated, not least of which is using easily available tools to enable students to create deep fake photos and videos of other students - creating real trauma and having real-world consequences.

My second comment is that many digital risks to students are also digital risks for their parents, including misinformation and other social harms. This legislation would have been a great opportunity, for example, to create a requirement for school boards to develop and provide curricula and training to support students in identifying misinformation through critical digital media training.

General

The last section of Bill 194 includes section 12, which states that nothing in the Act establishes a private law duty of care owed to any person. I'm not a lawyer, but when I looked up the phrase, it said, "A duty recognized by law to take reasonable care to avoid conduct that poses an unreasonable risk of harm to others." My only comment here is to note that despite the title of the bill, the writers of the bill have taken care to ensure that the heads of government institutions do not have a duty to ensure that they take reasonable care to avoid the risk of harm (aside from the requirement of privacy safeguards addition Schedule 2, which doesn't appear to me to be the same thing). It seems that where an individual's information, especially sensitive information, is collected under a legislative authority, the institution or head should have a duty of care for that individual's information. It may be that this is standard language in this kind of legislation, but it still leaves me a little perplexed. 🤷‍♂️

Schedule 2, Freedom of Information and Protection of Privacy Act

This schedule is, in some ways, simpler in that it provides amendments to an existing Act (FIPPA) and doesn't endlessly defer to yet-to-be-determined regulations. Schedule 2 defines "information practices" to FIPPA, which will help those responsible for building systems comply with FIPPA. Some worthwhile elements for reporting have been added. I will take particular note of two significant changes: requirements for privacy impact assessments (PIAs) as well as breach reporting and notification requirement

Privacy Impact Assessments

This is a welcome addition to FIPPA. PIAs are a standard tool for identifying the risks to privacy in a system and recommending steps for their remediation. By standardizing the information required in a PIA, this legislation goes some distance to raising the floor for privacy protection and providing the ability to develop consistent expertise across all of government. I look forward to any prescribed requirements. This is followed by a section on risk mitigation that directs government institutions to implement the recommendations of the PIA

I would be remiss if I didn't point out the obvious gap between this and Schedule 1. There is no directive in Schedule 1 concerning impact assessments for AI systems nor is there a direction to heads to mitigate identified risks.

A copy of PIAs is required to be provided to the Information and Privacy Commissioner if asked. This could be improved by changing this to a mandatory filing with the Commissioner. This doesn’t require the IPC to approve the PIA but does make it available to the Commissioner promptly in case of a complaint or breach related to a system with a PIA.

Breach Reporting and Notice

Schedule 2 adds a Privacy Safeguards section to FIPPA. Specifically, the requirement is that "The head of an institution shall take steps that are reasonable in the circumstances to ensure that personal information in the custody or under the control of the institution is protected against theft, loss and unauthorized use or disclosure and to ensure that the records containing the personal information are protected against unauthorized copying, modification or disposal." This begs the question of why this requirement for privacy safeguards is only being added now, but suffice to applaud it.

The requirement for privacy safeguards provides the underpinning for defining a breach as "any theft, loss or unauthorized use or disclosure of personal information in the custody or under the control of the institution if it is reasonable in the circumstances to believe that there is a real risk that a significant harm to an individual would result...". Such breaches will be reported to the Commissioner, whose budget will hopefully reflect this new obligation. The factors identified as determining whether there is a real risk of significant harm include:

the sensitivity of the personal information; the probability of misuse; the availability of steps that a person could take to reduce the risk of harm mitigate the risk of harm directions or guidance from the Commissioner

With safeguards, breaches, and risks of harm defined, the last piece is the addition of a requirement to notify individuals if there has been a breach of their information. This is welcome but has consequences. In some circumstances, such a notification can be traumatic or require expenditures by the individual to compensate. Where is the requirement to compensate the individual or help them mitigate the impact?

Order Making Power

It is worth noting that the amended FIPPA will provide the Commissioner concerning privacy breaches a new power for the Commissioner and, I suspect, a welcome one to bring the Commissioner's powers for privacy in FIPPA in alignment with her order-making powers for Freedom of Information issues.

Wrapping Up

This post was created within a day or two of Bill 194's First Reading. I look forward to other and deeper contributions to the debate in the days to come. In the meantime, I have these takeaways:

It is past time for Ontario to stop being a laggard in the protection of employee privacy and the government should, at the very least, amend Bill 194 to give public sector employees the privacy protection and respect they deserve. A private sector privacy bill could address employment privacy issues, putting it under the authority of the Commissioner with private sector order-making powers. Alternatively, elements of privacy protection for employees could also be addressed by adding to Ontario's Employment Standards Act. The government should use Bill 194's second reading and committee review to ensure that there is a clear legislative articulation of: What are the acceptable and unacceptable uses of artificial intelligence How to identify, categorize, and mitigate individual and social risks associated with the use of artificial intelligence If the government wants to ensure that digital technology doesn't harm children, it should start with digital media training and take steps to prevent children from using technology to bully other children. Consider recognizing that the government has a duty of care when it processes sensitive personal information under a legislative authority that deprives individuals of the ability to refuse that processing. Adding PIA requirements with breach notifications will raise the bar for institutions processing Ontarians' personal information. This may lead to some interesting changes or headlines in the short term, but the longer-term consequences should be good.

At the end of the day, the government appears to want to be able to take steps to address cybersecurity, children's data processing, and artificial intelligence through regulations. It will be interesting to see how, or if, the consultation process will significantly alter this approach. The public consultation is open until June 11th and can be found at https://www.ontariocanada.com/registry/view.do?postingId=47433&language=en

Wednesday, 15. May 2024

Simon Willison

Quoting Casey Newton

But where the company once limited itself to gathering low-hanging fruit along the lines of “what time is the super bowl,” on Tuesday executives showcased generative AI tools that will someday plan an entire anniversary dinner, or cross-country-move, or trip abroad. A quarter-century into its existence, a company that once proudly served as an entry point to a web that it nourished with traffic a

But where the company once limited itself to gathering low-hanging fruit along the lines of “what time is the super bowl,” on Tuesday executives showcased generative AI tools that will someday plan an entire anniversary dinner, or cross-country-move, or trip abroad. A quarter-century into its existence, a company that once proudly served as an entry point to a web that it nourished with traffic and advertising revenue has begun to abstract that all away into an input for its large language models.

Casey Newton


PaliGemma model README

PaliGemma model README One of the more over-looked announcements from Google I/O yesterday was PaliGemma, an openly licensed VLM (Vision Language Model) in the Gemma family of models. The model accepts an image and a text prompt. It outputs text, but that text can include special tokens representing regions on the image. This means it can return both bounding boxes and fuzzier segment outlines

PaliGemma model README

One of the more over-looked announcements from Google I/O yesterday was PaliGemma, an openly licensed VLM (Vision Language Model) in the Gemma family of models.

The model accepts an image and a text prompt. It outputs text, but that text can include special tokens representing regions on the image. This means it can return both bounding boxes and fuzzier segment outlines of detected objects, behavior that can be triggered using a prompt such as "segment puffins".

You can try it out on Hugging Face.

It's a 3B model, making it feasible to run on consumer hardware.

Via Roboflow: PaliGemma: Open Source Multimodal Model by Google


Ben Werdmüller

Former Far-Right Hard-Liner Says Billionaires Are Targeting Texas Public Education

"When Courtney Gore ran for a seat on her local school board in 2021, she warned about a movement to indoctrinate children with “leftist” ideology. After 2 1/2 years on the board, Gore said she believes a much different scheme is unfolding: an effort by wealthy conservative donors to undermine public education in Texas and install a voucher system in which public money flows t

"When Courtney Gore ran for a seat on her local school board in 2021, she warned about a movement to indoctrinate children with “leftist” ideology. After 2 1/2 years on the board, Gore said she believes a much different scheme is unfolding: an effort by wealthy conservative donors to undermine public education in Texas and install a voucher system in which public money flows to private and religious schools."

An interesting ProPublica story about the motivation behind some of the money that's funded these bizarre right-wing school board elections. It's not so much about the ideology as it is about undermining trust in public education itself, so that it can be replaced with a voucher system that would benefit the underwriters.

This quote says it all:

“It’s all about destroying the trust with the citizens to the point where they would tolerate something like doing away with public schools.” #Education

[Link]


@_Nat Zone

プロ責法あらため情プラ法が成立しました(5月10日)

わたしも構成員であった総務省プラットフォームに関する研究会での検討を受けて、この3月に国会に提出された改正「プロバイダ責任制限法」改め「情プラ法」が、木村花さんの4回目の命日(5月23日)を前に、5月10日、国会で成立しました。法律名が変更されたのは「これまでの投稿の発信者情報の…

わたしも構成員であった総務省プラットフォームに関する研究会での検討を受けて、この3月に国会に提出された改正「プロバイダ責任制限法」1改め「情プラ法」2が、木村花さんの4回目の命日(5月23日)を前に、5月10日、国会で成立しました。法律名が変更されたのは「これまでの投稿の発信者情報の開示等にとどまらない内容となったため」3です。

法案、その他の関係資料は以下のとおりです4

令和6年3月1日特定電気通信役務提供者の損害賠償責任の制限及び発信者情報の開示に関する法律の一部を改正する法律案概要【325 KB】
要綱【97 KB】
法律案・理由【160 KB】
新旧対照条文【254 KB】
参照条文【310 KB】
 (所管課室名)
総合通信基盤局電気通信事業部利用環境課(出所)総務省 <https://www.soumu.go.jp/menu_hourei/k_houan.html> (2024-05-15取得)

この改正は、近年のSNS等での権利侵害情報の流通による被害の増加を受け、プラットフォーム事業者の責任を強化し、被害者の保護を図ることを目的としています。大規模プラットフォーム事業者には削除等の義務が課され、違反には罰則が科されることになりました。

主な改正点は以下の通りです:

法律の題名と通称名が変更された。新しい題名は上記の通りで、通称名は「情報流通プラットフォーム対処法(情プラ法)」となった。 第5章として「大規模特定電気通信役務提供者の義務」が新設された。これにより、一定規模以上のプラットフォーム事業者に対し、権利侵害情報の流通防止措置等の義務が課されることになった。 第6章として「罰則」が新設された。義務に違反した場合の罰則規定が設けられた。 条文数が従来の19条から38条へと倍増した。 総務省プラットフォームに関する研究会について

総務省プラットフォームに関する研究会は平成30年(2018年)10月18日から足掛け6年、全52回にわたって開催されてきた研究会です。令和2年2月に最終報告書を出し、一旦は終了したはずだったのですが、木村花さんの誹謗中傷と自殺事件を受け、緊急再招集され、以後、言論の自由を含め様々座な角度から検討を進めてきました。その経緯は、プラットフォームに関する研究会の概要ページにある議事概要や以下のような文書から見ていただけると思います。

インターネット上の誹謗中傷への対応の在り方についての意見募集(令和2年7月3日) インターネット上の誹謗中傷への対応の在り方に関する緊急提言及び意見募集の結果の公表 中間とりまとめ(案)についての意見募集に関する報道発表(令和3年7月16日) 中間とりまとめ(案)についての意見募集の結果及び中間とりまとめの公表に関する報道発表(令和3年9月15日) 第二次とりまとめ(案)についての意見募集に関する報道発表(令和4年7月4日) 第二次とりまとめ(案)についての意見募集の結果及び第二次とりまとめの公表に関する報道発表(令和4年8月25日) 第三次とりまとめ(案)についての意見募集に関する報道発表(令和5年12月13日) 第三次とりまとめ(案)についての意見募集の結果及び第三次とりまとめの公表に関する報道発表(令和6年2月2日)

そして、本年、第52回(令和6年1月31日)をもって最終回となり、第三次とりまとめを発表し、クローズいたしました。今次法改正は、この第三次とりまとめがベースとなっています。

この間、総務省および事務局の方々には困難な議論を取りまとめていただき、法改正にまでつなげていただけたことに本当に感謝しております。また、微力ながらわたくしも本件に貢献できたことをとても嬉しく思っております。

Tuesday, 14. May 2024

The Pragmatic Engineer

Applied AI Software Engineering: RAG

Retrieval-Augmented Generation (RAG) is a common building block of AI software engineering. A deep dive into what it is, its limitations, and some alternative use cases. By Ross McNairn.

👋 Hi, this is Gergely with a subscriber-only issue of the Pragmatic Engineer Newsletter. In every issue, I cover challenges at Big Tech and startups through the lens of engineering managers and senior engineers. To get articles like this in your inbox, every week, subscribe:

Subscribe now

I recently spoke with Karthik Hariharan, who heads up engineering at VC firm Goodwater Capital, and he highlighted a trend he’d spotted:

“There’s an engineering project I’m seeing almost every startup building a Large Language Model (LLM) put in place: building their own Retrieval Augmentation Generation (RAG) pipelines.

RAGs are a common pattern for anyone building an LLM application. This is because it provides a layer of ‘clean prompts’ and fine-tuning. There are some existing open-source solutions, but almost everyone just builds their own, anyway.”

I asked a few Artificial Intelligence (AI) startups about this, and sure enough, all do build their own RAG. So, I reached out to a startup I know is doing the same: Wordsmith AI. It’s an AI startup for in-house legal teams that’s making heavy use of RAG, and was co-founded by Ross McNairn. He and I worked for years together at Skyscanner and he offered to share Wordsmith AI’s approach for building RAG pipelines, and some learnings. Declaration of interest: I’m an investor in Wordsmith, and the company has recently launched out of stealth.

Today, we cover:

Providing an LLM with additional context

The simplest RAGs

What is a RAG pipeline?

Preparing the RAG pipeline data store

Bringing it all together

RAG limitations

Real-world learnings building RAG pipelines

Today’s article includes a “code-along,” so you can build your own RAG. View the code used in this article at this GitHub repository: hello-wordsmith. To keep up with Ross, subscribe to his blog or follow him on LinkedIn.

With that, it’s over to Ross:

Introduction

Hi there! This post is designed to help you get familiar with one of the most fundamental patterns of AI software engineering: RAG, aka Retrieval Augmented Generation.

I co-founded a legal tech startup called Wordsmith, where we are building a platform for running a modern in-house legal team. Our founding team previously worked at Meta, Skyscanner, Travelperk and KPMG.

We are working in a targeted domain – legal texts – and building AI agents to give in-house legal teams a suite of AI tools to remove bottlenecks and improve how they work with the rest of the business. Performance and accuracy are key characteristics for us, so we’ve invested a lot of time and effort in how to best enrich and “turbo charge” these agents with custom data and objectives.

We ended up building our RAG pipeline, and I will now walk you through how we did it and why. We’ll go into our learnings, and how we benchmark our solution. I hope that the lessons we learned are useful for all budding AI engineers.

1. Providing an LLM with additional context

Have you ever asked ChatGPT a question it does not know how to answer, or its answer is too high level? We’ve all been there, and all too often, interacting with a GPT feels like talking to someone who speaks really well, but doesn’t know the facts. Even worse, they can make up the information in their responses!

Here is one example. On 1 February 2024, during an earnings call, Mark Zuckerberg laid out the strategic benefits of Meta’s AI strategy. But when we ask ChatGPT a question about this topic, this model will make up an answer that is high-level, but is not really what we want:

ChatGPT 3.5’s answer to a question about Meta’s AI strategy. The answer is generalized, and misses a critical source which answers the question

This makes sense, as the model’s training cutoff date was before Mark Zuckerberg made the comments. If the model had access to that information, it would have likely been able to summarize the facts of that meeting, which are:

“So I thought it might be useful to lay out the strategic benefits [of Meta’s open source strategy) here. (...)

The short version is that open sourcing improves our models. (...)

First, open-source software is typically safer and more secure as well as more compute-efficient to operate due to all the ongoing feedback, scrutiny and development from the community. (...)

Second, open-source software often becomes an industry standard. (...)

Third, open source is hugely popular with developers and researchers. (...)

The next part of our playbook is just taking a long-term approach towards the development.”

LLMs’ understanding of the world is limited to the data they’re trained on. If you’ve been using ChatGPT for some time, you might remember this constraint in the earlier version of ChatGPT, when the bot responded: “I have no knowledge after April 2021,” in several cases. 

Providing an LLM with additional information

There is a bunch of additional information you want an LLM to use. In the above example, I might have the transcripts of all of Meta’s shareholders meetings that I want the LLM to use. But how can we provide this additional information to an existing model?

Option 1: input via a prompt

The most obvious solution is to input the additional information via a prompt; for example, by prompting “Using the following information: [input a bunch of data] please answer the question of [ask your question].”

This is a pretty good approach. The biggest problem is that this may not scale because of these reasons:

The input tokens limit. Every model has an input prompt token limit. At the time of publication this is 4.069 tokens for GPT-3, 16,385 for GPT-3.5, 8,192 for GPT-4, 128,000 for GPT-4 Turbo, 200.000 for Anthropic models. Google’s Gemini model allows for an impressive one million token limit. While a million-token limit greatly increases the possibilities, it might still be too low for use cases with a lot of additional text to input.

Performance. The performance of LLMs substantially decreases with longer input prompts; in particular, you get degradation of context in the middle of your prompt. Even when creating long input prompts is a possibility, the performance tradeoff might make it impractical.

Option 2: fine-tune the model

We know LLMs are based on a massive weights matrix. Read more on how ChatGPT works in this Pragmatic Engineer issue. All LLMs use the same principles.

An option is to update these weight matrices based on additional information we’d like our model to know. This can be a good option, but it is a much higher upfront cost in terms of time, money, and computing resources. Also, it can only be done with access to the model’s weightings, which is not the case when you use models like ChatGPT, Anthropic, and other “closed source” models.

Option 3: RAG

The term ‘RAG’ originated in a 2020 paper led by Patrick Lewis. One thing many people notice is that “Retrieval Augmented Generation” sounds a bit ungrammatical. Patrick agrees, and has said this:

“We always planned to have a nicer-sounding name, but when it came time to write the paper, no one had a better idea.”

RAG is a collection of techniques which help to modify a LLM, so it can fill in the gaps and speak with authority, and some RAG implementations even let you cite sources. The biggest benefits of the RAG approach:

Give a LLM domain-specific knowledge You can pick what data you want your LLM to draw from, and even turn it into a specialist on any topic there is data about. 

This flexibility means you can also extend your LLMs’ awareness far beyond the model’s training cutoff dates, and even expose it to near-real time data, if available.

Optimal cost and speed. For all but a handful of companies, it's impractical to even consider training their own foundational model as a way to personalize the output of an LLM, due to the very high cost and skill thresholds. 

In contrast, deploying a RAG pipeline will get you up-and-running relatively quickly for minimal cost. The tooling available means a single developer can have something very basic functional in a few hours.

Reduce hallucinations. “Hallucination” is the term for when LLMs “make up” responses. A well-designed RAG pipeline that presents relevant data will all but eliminate this frustrating side effect, and your LLM will speak with much greater authority and relevance on the domain about which you have provided data.

For example, in the legal sector it’s often necessary to ensure an LLM draws its insight from a specific jurisdiction. Take the example of asking a model a seemingly simple question, like:

How do I hire someone?

Your LLM will offer context based on the training data. However, you do not want the model to extract hiring practices from a US state like California, and combine this with British visa requirements! 

With RAG, you control the underlying data source, meaning you can scope the LLM to only have access to a single jurisdiction’s data, which ensures responses are consistent.

Better transparency and observability. Tracing inputs and answers through LLMs is very hard. The LLM can often feel like a “black box,” where you have no idea where some answers come from. With RAG, you see the additional source information injected, and debug your responses.

2. The simplest RAGs

The best way to understand new technology is often just to play with it. Getting a basic implementation up and running is relatively simple, and can be done with just a few lines of code. To help, Wordsmith has created a wrapper around the LlamaIndex open source project to help abstract away some complexity. You can get up and running, easily. It has a README file in place that will get you set up with a local RAG pipeline on your machine, and which chunks and embeds a copy of the US Constitution, and lets you search away with your command line.

This is as simple as RAGs get; you can “swap out” the additional context provided in this example by simply changing the source text documents!

This article is designed as a code-along, so I'm going to link you to sections of this repo, so you can see where specific concepts manifest in code.

To follow along with the example, the following is needed:

An active OpenAI subscription with API usage. Set one up here if needed. Note: running a query will cost in the realm of $0.25-$0.50 per run.

Follow the instructions to set up a virtual Python environment, configure your OpenAI key, and start the virtual assistant.

This example will load the text of the US constitution from this text file, as a RAG input. However, the application can be extended to load your own data from a text file, and to “chat” with this data.

Here’s an example of how the application works when set up, and when the OpenAI API key is configured:

The example RAG pipeline application answering questions using the US Constitution supplied as additional context

If you’ve followed along and have run this application: congratulations! You have just executed a RAG pipeline. Now, let’s get into explaining how it works.

3. What is a RAG pipeline?

A RAG pipeline is a collection of technologies needed to enable the capability of answering using provided context. In our example, this context is the US Constitution and our LLM model is enriched with additional data extracted from the US Constitution document. 

Here are the steps to building a RAG pipeline:

 
Step 1: Take an inbound query and deconstruct it into relevant concepts
Step 2: Collect similar concepts from your data store
Step 3: Recombine these concepts with your original query to build a more relevant, authoritative answer.

Weaving this together:

A RAG pipeline at work. It extends the context an LLM has access to, by fetching similar concepts from the data store to answer a question

While this process appears simple, there is quite a bit of nuance in how to approach each step. A number of decisions are required to tailor to your use case, starting with how to prepare the data for use in your pipeline.

4. Preparing the RAG pipeline data store

Read more

Monday, 13. May 2024

Just a Theory

Extension Summit Topic Review

Final Postgres Extension Mini-Summit! On Wednesday May 15 at noon Eastern / 16:00 UTC, we’ll review some of the topics that have come up in previous Mini-Summits with the goal to winnow down and select issues to address at PGConf.dev.

Boy howdy that went fast.

This Wednesday, May 15, the final Postgres extension ecosystem mini-summit will review topics covered in previous Mini-Summits, various Planet PostgreSQL posts, the #extensions channel on the Postgres Slack and the Postgres Discord. Following a brief description of each, we’ll determine how to reduce the list to the most important topics to take on at the Extension Ecosystem Summit at PGConf.dev in Vancouver on May 28. I’ll post a summary later this week along with details for how to participate in the selection process.

In the meantime, here’s the list as of today:

Metadata: Third-party dependencies Types of extensions Taxonomies System requirements (OS, version, CPU, etc.) Categorization Versioning Registry: Identity, namespacing, and uniqueness Distributed vs. centralized publishing Binary packaging and distribution patterns Federated distribution Services and tools to improve or build Stats, Reports, Badging: (stars, reviews, comments, build & test matrices, etc.) Packaging: Formats (e.g., tarball, OCI, RPM, wheel, etc.) Include dynamic libs in binary packaging format? (precedent: Python wheel) Build farming Platforms, architectures, and OSes Security, trust, and verification Developer: Extension developer tools Improving the release process Build pipelines: Supporting PGXS, prgx, Rust, Go, Python, Ruby, Perl, and more Community: Community integration: identity, infrastructure, and support How-Tos, tutorials, documentation for creating, maintaining, and distributing extensions Docs/references for different types of extensions: CREATE EXTENSION, hooks, background workers, CLI apps/services, web apps, native apps, etc. Core: Second extension directory (a.k.a. variable installation location, search path) Keeping all files in a single directory Documentation standard Inline extensions: UNITs, PACKAGEs, TLEs, etc. Minimizing restarts Namespacing Sandboxing, code signing, security, trust Dynamic module loading (e.g., use Thing in PL/Perl could try to load Thing.pm from a table of acceptable libraries maintained by the DBA) Binary compatibility of minor releases and/or /ABI stability

Is your favorite topic missing? Join us at the mini-summit or drop suggestions into the #extensions channel on the Postgres Slack.

More about… Postgres Yum PGConf Summit

Ben Werdmüller

Facebook news referrals: no sign of the slow-down stopping

"Aggregate Facebook traffic to a group of 792 news and media sites that have been tracked by Chartbeat since 2018 shows that referrals to the sites have plunged by 58%." I'll bang this drum forever: establish direct relationships with your audience. Do not trust social media companies to be your distribution. That means through your website. That means through email.

"Aggregate Facebook traffic to a group of 792 news and media sites that have been tracked by Chartbeat since 2018 shows that referrals to the sites have plunged by 58%."

I'll bang this drum forever: establish direct relationships with your audience. Do not trust social media companies to be your distribution.

That means through your website.

That means through email.

That means through direct social like the fediverse.

It's long past time that media learned this and internalized it forever. #Media

[Link]


The SF Bay Area Has Become The Undisputed Leader In AI Tech And Funding Dollars

"Last year, more than 50% of all global venture funding for AI-related startups went to companies headquartered in the Bay Area, Crunchbase data shows, as a cluster of talent congregates in the region." In other news, water is wet. There was a moment during the pandemic when it looked like everyone was going to work remotely and there was an opportunity for startups to

"Last year, more than 50% of all global venture funding for AI-related startups went to companies headquartered in the Bay Area, Crunchbase data shows, as a cluster of talent congregates in the region."

In other news, water is wet.

There was a moment during the pandemic when it looked like everyone was going to work remotely and there was an opportunity for startups to be founded anywhere. I think that time has gone: the San Francisco Bay Area is once again the place to found any kind of technology startup.

Yes, there are always exceptions, but the confluence of community density, living conditions, universities, and mindset make for a perfect storm. NYC and London - and maybe Boston / Cambridge - are pretty good too, for what it's worth, but the sheer volume of startup activity in the area gives San Francisco the edge.

This is something I fought earlier in my career: my first startup was proudly founded in Scotland and largely run from England. I wish we'd just moved to San Francisco.

This isn't to completely sing the praises of the city: the cost of living is now astronomical, and there's a contingent of right-wing activists that seem to want to paint it as some doom spiraling hellhole, as if its progressive past isn't something to be proud of. But there is still beauty, there is still that can-do sense of adventure, and if I was founding something new, that's probably where I'd be. #Technology

[Link]


The Philadelphia Inquirer is here to fight

SEPTA - the South Eastern Pennsylvania Transportation Authority — trains are covered with these ads for the Philadelphia Inquirer: I’m curious to know if they actually work. They feel very negative to me: a pot-shot at the New York Times rather than an argument for why the Inquirer is great in its own right. There’s an underlying assumption here that newspaper subscriptions are zero-sum: tha

SEPTA - the South Eastern Pennsylvania Transportation Authority — trains are covered with these ads for the Philadelphia Inquirer:

I’m curious to know if they actually work. They feel very negative to me: a pot-shot at the New York Times rather than an argument for why the Inquirer is great in its own right.

There’s an underlying assumption here that newspaper subscriptions are zero-sum: that each household will only receive one. Of course, most households aren’t even that: it’s increasingly rare for anyone to subscribe to a paper newspaper. But for digital subscriptions, I’d have assumed that it would be additional: households might subscribe to both the Inquirer and the Times (as well as a few other publications; maybe the New Yorker and Philadelphia Magazine).

Is their assumption right, or is mine? I don’t know. What I do know is that the ad feels combative and what I’m left with is the conflict rather than anything about the Inquirer’s own coverage. While there is definitely some anti-New York feeling among multi-generational Philadelphians, it feels like an odd choice.


Damien Bod

Using SonarCloud with ASP.NET Core, Angular and github actions

This article demonstrates how to implement code analysis and Static Application Security Testing (SAST) using SonarCloud and GitHub Actions. The solution involves building a secure web application with ASP.NET Core for the backend and an Angular UI for the frontend, following a backend-for-frontend security architecture. Both the ASP.NET Core (C#) codebase and the Angular (TypeScript […]

This article demonstrates how to implement code analysis and Static Application Security Testing (SAST) using SonarCloud and GitHub Actions. The solution involves building a secure web application with ASP.NET Core for the backend and an Angular UI for the frontend, following a backend-for-frontend security architecture. Both the ASP.NET Core (C#) codebase and the Angular (TypeScript and JavaScript) files undergo analysis during the quality build process with SonarCloud.

Code: https://github.com/damienbod/EndToEndSecurity

Solution Setup

The application uses an Angular UI implemented with NX tools and services. During deployment builds, the UI is integrated into the ASP.NET Core backend, while in development, it operates as a standalone component. Microsoft YARP facilitates the connection between Angular development and local ASP.NET Core APIs. The Angular application is an integral part of the backend system when deployed. Both technical stacks require code analysis and Static Application Security Testing (SAST). Additionally, the solution is secured as a single OpenID Connect confidential client, utilizing the code flow with Proof Key for Code Exchange (PKCE).

SonarCloud setup

I set up SonarCloud testing and integrated it with GitHub Actions using the repository from Marc Rufer.

https://github.com/rufer7/github-sonarcloud-integration

This references the docs from SonarCloud and all the steps required for setting up a build and analysis of the different technical stacks are documented.

ASP.NET Core project setup

To enable SonarCloud to analyze both the ASP.NET Core project and the Angular projects, you’ll need to make adjustments in the .NET Core csproj file settings. Specifically, the Angular components should be added as hidden elements so that SonarCloud can properly detect and analyze them.

<ItemGroup> <!-- This is required to include ts and js files in SonarCloud analysis --> <!-- Add to the sonar cloud build: EndToEndSecurity == github repo --> <!-- /d:sonar.projectBaseDir="D:\a\EndToEndSecurity\EndToEndSecurity" /d:sonar.exclusions=**/node_modules/** --> <!-- See https://docs.sonarsource.com/sonarqube/9.8/analyzing-source-code/scanners/sonarscanner-for-dotnet/#advanced-topics --> <Content Include="..\ui\**\*.ts" Visible="false"> <CopyToOutputDirectory>Never</CopyToOutputDirectory> </Content> <Content Include="..\ui\**\*.js" Visible="false"> <CopyToOutputDirectory>Never</CopyToOutputDirectory> </Content> </ItemGroup> Quality build

The SonarCloud github action YAML file implements the quality build. Normally this would be integrated with the default build, PRs and feature branches would run this. The dotnet testing tools are added but not active. The build uses a windows-latest image and java. When testing the code of the two technical stacks, you should ignore folders like node_modules and so on. This can be excluded in the YAML file. For this to work, the SonarCloud project must match the YAML file definitions. This is well documented in the Sonar documentation.

name: SonarCloud on: push: branches: - develop - main pull_request: types: [opened, synchronize, reopened] jobs: build: name: Analyze dotnet and Augular projects runs-on: windows-latest steps: - name: Set up JDK 17 uses: actions/setup-java@v4 with: java-version: 17 distribution: 'zulu' # Alternative distribution options are available. - uses: actions/checkout@v4 with: fetch-depth: 0 # Shallow clones should be disabled for a better relevancy of analysis - name: Cache SonarCloud packages uses: actions/cache@v4 with: path: ~\sonar\cache key: ${{ runner.os }}-sonar restore-keys: ${{ runner.os }}-sonar - name: Cache SonarCloud scanner id: cache-sonar-scanner uses: actions/cache@v4 with: path: .\.sonar\scanner key: ${{ runner.os }}-sonar-scanner restore-keys: ${{ runner.os }}-sonar-scanner - name: Install SonarCloud scanner if: steps.cache-sonar-scanner.outputs.cache-hit != 'true' shell: powershell run: | New-Item -Path .\.sonar\scanner -ItemType Directory dotnet tool update dotnet-sonarscanner --tool-path .\.sonar\scanner - name: Install dotnet-coverage # not used as not tests exist in backend shell: powershell run: | dotnet tool install --global dotnet-coverage - name: Build and analyze env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} # Needed to get PR information, if any SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }} shell: powershell run: | .\.sonar\scanner\dotnet-sonarscanner begin /k:"damienbod_EndToEndSecurity" /o:"damienbod" /d:sonar.token="${{ secrets.SONAR_TOKEN }}" /d:sonar.host.url="https://sonarcloud.io" /d:sonar.projectBaseDir="D:\a\EndToEndSecurity\EndToEndSecurity" /d:sonar.exclusions=**/node_modules/** dotnet build .\Bff.sln --configuration Release # dotnet-coverage collect 'dotnet test .\src\--testproj--.Tests\--testproj--.Tests.csproj' -f xml -o 'coverage.xml' .\.sonar\scanner\dotnet-sonarscanner end /d:sonar.token="${{ secrets.SONAR_TOKEN }}" Badges

Badges from SonarCloud can be added directly in the readme file of the github repository. The badges are created in SonarCloud and I switched them to the overall results and not just the last delta. By clicking the badges in the readme, you are redirected to the SonarCloud test results.

SonarCloud is a great service for code quality analysis and has a good SAST implementation with very good access into the github UI and tools. Security alerts can be directly viewed in github.

Links

https://docs.sonarsource.com/sonarcloud/getting-started/github

https://github.com/rufer7/github-sonarcloud-integration

[HOWTO] Integrate SonarCloud analysis in an Azure DevOps YAML pipeline

https://community.sonarsource.com/t/code-coverage-report-for-net-not-working-on-linux-agent/62087

https://docs.sonarsource.com/sonarcloud/advanced-setup/ci-based-analysis/sonarscanner-for-net/#analyzing-languages-other-than-c-and-vb

https://andreiepure.ro/2023/08/20/analyze-web-files-with-s4net.html

Sunday, 12. May 2024

Wrench in the Gears

Blessings

It’s mother’s day. Some get to celebrate with loved ones, while others navigate the bittersweet reality of bereavement or estrangement. My heart is with you all. If there’s one lesson I’ve learned these past few years it is the importance of love, even across distance, even when it may appear to be unreciprocated. The system [...]

It’s mother’s day. Some get to celebrate with loved ones, while others navigate the bittersweet reality of bereavement or estrangement. My heart is with you all. If there’s one lesson I’ve learned these past few years it is the importance of love, even across distance, even when it may appear to be unreciprocated. The system seeks to harness the power of mothers’ hearts through token engineering, but our hearts are wild and unruly. Our torus fields will not be tamed. 

Today I enjoyed many blessings. I am working to move forward from a place of gratitude, so I want to acknowledge them in this short post with a few pictures from Shrine Mont, Virginia. It is a resort maintained by the Anglican Church that includes several mineral springs that were sacred to the original people’s of the Shenandoah. At some point in the past year, I looked up who founded MIT. It turns out it was a Virginia geologist William Barton Rogers. Having studied the role of water as a solvent on minerals and overseen the state geological survey in 1835, Rogers would have been familiar with these springs. His survey was made three years after Andrew Jackson put the hot springs of Arkansas under federal ownership. In addition to the springs, the area also features a subtle energy vortex that you can read about here.

This is my second visit. I made a video of my first trip in the summer of 2022. 

Things for which I am grateful today: 

Friends of like mind who offer good conversation, a soft bed, hot shower, and laundry services to a wayward traveler.

Unconditional love from an affectionate dog.

A swift recovery from a debilitating migraine.

A sky of clear blue, full of natural, puffy clouds. 

Stands of purple and white woodland phlox.

A burbling brook with a nicely-spaced stone crossing. 

A labyrinth to walk. 

An abundance of tulip poplar buds.

Varied shades of pink in myrtle flowers and lady slipper orchids.

The sweet fragrance of locust tree blossoms.

A quiet moment to enjoy the view of white wooden buildings with dark bottle green trim beyond a pond ringed by last season’s cattails and trilling red-winged blackbirds.

 


Heres Tom with the Weather


Ben Werdmüller

British newspaper groups warn Apple over ad-blocking plans, FT reports

"British newspaper groups have warned Apple that any move to impose a so-called "web eraser" tool to block advertisements would put the financial sustainability of journalism at risk, the Financial Times reported on Sunday." Counterpoint: block the ads. The web is designed to be a flexible platform that can be mixed and remixed however you need. One of the points of CSS

"British newspaper groups have warned Apple that any move to impose a so-called "web eraser" tool to block advertisements would put the financial sustainability of journalism at risk, the Financial Times reported on Sunday."

Counterpoint: block the ads.

The web is designed to be a flexible platform that can be mixed and remixed however you need. One of the points of CSS was that you could have your own styles for a site and they would supersede the interface that came out of the box.

Relying on ads is a race to the bottom. There are plenty of other ways to make money and build deeper relationships with your audience - many of which don't require paywalls or any invasive technology at all.

Ad technology profiles and tracks users; slows down websites; wastes energy; obliterates the user experience; and isn't even all that profitable. It's hard to square an organization that claims to be acting in the public interest advocating for them. #Media

[Link]


Abortion bans drive away young talent: New CNBC/Generation Lab survey

"The youngest generation of American workers is prepared to move away from states that pass abortion bans and to turn down job offers in states where bans are already in place, a new survey from CNBC/Generation Lab finds." This stands to reason: why would you move to a place where government wants to control what you do with your body? Whether you have a uterus or not, cari

"The youngest generation of American workers is prepared to move away from states that pass abortion bans and to turn down job offers in states where bans are already in place, a new survey from CNBC/Generation Lab finds."

This stands to reason: why would you move to a place where government wants to control what you do with your body? Whether you have a uterus or not, caring for the well-being of people who do is obvious. And all the societal overreach and Handmaid's Tale overtones affect everybody.

I'm interested to see how this affects those locations over time. Of course, there are other implications of this legislation, too: it's likely to be one of the major drivers for voters in November. #Business

[Link]


Aaron Parecki

FedCM for IndieAuth

IndieWebCamp Düsseldorf took place this weekend, and I was inspired to work on a quick hack for demo day to show off a new feature I've been working on for IndieAuth.

IndieWebCamp Düsseldorf took place this weekend, and I was inspired to work on a quick hack for demo day to show off a new feature I've been working on for IndieAuth.

Since I do actually use my website to log in to different websites on a regular basis, I am often presented with the login screen asking for my domain name, which is admittedly an annoying part of the process. I don't even like having to enter my email address when I log in to a site, and entering my domain isn't any better.

So instead, I'd like to get rid of this prompt, and let the browser handle it for you! Here's a quick video of logging in to a website using my domain with the new browser API:

So how does this work?

For the last couple of years, there has been an ongoing effort at the Federated Identity Community Group at the W3C to build a new API in browsers that can sit in the middle of login flows. It's primarily being driven by Google for their use case of letting websites show a Google login popup dialog without needing 3rd party cookies and doing so in a privacy-preserving way. There's a lot to unpack here, more than I want to go into in this blog post. You can check out Tim Cappalli's slides from the OAuth Security Workshop for a good explainer on the background and how it works.

However, there are a few experimental features that are being considered for the API to accommodate use cases beyond the "Sign in with Google" case. The one that's particularly interesting to the IndieAuth use case is the IdP Registration API. This API allows any website to register itself as an identity provider that can appear in the account chooser popup, so that a relying party website doesn't have to list out all the IdPs it supports, it can just say it supports "any" IdP. This maps to how IndieAuth is already used today, where a website can accept any user's IndieAuth server without any prior relationship with the user. For more background, check out my previous blog post "OAuth for the Open Web".

So now, with the IdP Registration API in FedCM, your website can tell your browser that it is an IdP, then when a website wants to log you in, it asks your browser to prompt you. You choose your account from the list, the negotiation happens behind the scenes, and you're logged in!

One of the nice things about combining FedCM with IndieAuth is it lends itself nicely to running the FedCM IdP as a separate service from your actual website. I could run an IndieAuth IdP service that you could sign up for and link your website to. Since your identity is your website, your website would be the thing ultimately sent to the relying party that you're signing in to, even though it was brokered through the IdP service. Ultimately this means much faster adoption is possible, since all it takes to turn your website into a FedCM-supported site is adding a single <link> tag to your home page.

So if this sounds interesting to you, leave a comment below! The IdP registration API is currently an early experiment, and Google needs to see actual interest in it in order to keep it around! In particular, they are looking for Relying Parties who would be interested in actually using this to log users in. I am planning on launching this on webmention.io as an experiment. If you have a website where users can sign in with IndieAuth, feel free to get in touch and I'd be happy to help you set up FedCM support as well!


Kent Bull

KERI Internals Part 1: Concurrency and Async Programming with the HIO Async Framework and I/O Library

Welcome to the inner workings of the Python implementation of KERI! HIO stands for Hierarchical IO. Disclaimer: this post is for a technical audience who have a need to read and understand the WebOfTrust Python implementation of the KERI, ACDC, and CESR Trust over IP (ToIP) specifications. Have you ever […]

Welcome to the inner workings of the Python implementation of KERI! HIO stands for Hierarchical IO.

Disclaimer: this post is for a technical audience who have a need to read and understand the WebOfTrust Python implementation of the KERI, ACDC, and CESR Trust over IP (ToIP) specifications.

Have you ever wanted to contribute to the KERI ecosystem and been curious about the way the Python implementations are written? Or have you wanted to build on top of or modify the Python KERI codebase?

Not to worry, this technical series on KERI internals has your back. You will be ready to read through, understand, and build on top of the Python KERI code base once you understand the HIO async runtime, the focus of this article.

You are wanted as a contributor to the KERI ecosystem! The goal of this article is to assist you in becoming either a contributor to the Python implementation of KERI & ACDC or an advanced user of said implementation.

HIO Introduction

HIO is an asynchronous runtime and input/output (IO) framework written by Dr. Samuel Smith that supports cooperative multitasking. It is used throughout the Python implementation of the KERI suite of protocols.

This article serves as an introduction to the three primary classes composing the basis for HIO’s asynchronous runtime and as the lifecycle context functions for the main task class, the Doer. Additionally, you will have an idea of how these concepts relate to similar concepts in Python’s AsyncIO runtime. The three HIO classes include:

the Doist, the root scheduler, the DoDoer, the heirarchical container of Doer and DoDoer instances Doer, the core task concept in HIO.

Due to its nature as the asynchronous runtime engine, HIO is found at the heart of the core Python libraries in the WebOfTrust ecosystem including the core library KERIpy, the agent server KERIA, and the SignifyPy client companion to KERIA.

In order to understand the purpose of three classes mentioned above and how they compare to Python’s AsyncIO it is important to clarify terminology around concurrent and asynchronous programming in Python. As Python’s async/await is much more common and familiar than HIO this article starts there to introduce the concepts.

Why is HIO used in KERIpy, KERIA, and SignifyPy?

Performance, control, and features, at a high level, are the reason why HIO was used for KERIpy. HIO’s use of what are called “classic coroutines” and asynchronous buffers for I/O provide a level of control and performance that is difficult to achieve with Python’s AsyncIO implementation. An API into the timing system used for the event loop and scheduler provide tight, deterministic control over scheduling order of tasks.

A future article will go deeper than this short overview into the argument for using HIO and what specifically sets it apart from other async frameworks like AsyncIO, Curio, and Trio.

Async Framework Short Comparison

An asynchronous framework typically consists of a number of major abstractions including an event loop, task or coroutine, scheduler, queues for communicating between tasks, futures, callbacks, non-blocking I/O, synchronization primitives (locks, semaphores), timeouts and cancellation, and some notion of lifecycle for tasks. This article focuses specifically on the event loop, scheduler, and task abstractions in HIO and Python’s AsyncIO.

Cooperative Multitiasking

Both HIO and AsyncIO allow you to accomplish what is called “cooperative multitasking” which is where each coroutine yields control to a central scheduler so that other coroutines can be activated for their next execution. In AsyncIO the scheduler is the asyncio event loop and a coroutine is any function declared with the async def syntax. In HIO the scheduler is the Doist class and the coroutine is the Doer class.

Concurrency and parallelism in Python

When discussing concurrency or asynchronous programming it is important to distinguish between what is typically meant by concurrency and parallelism.

Concurrency is about dealing with lots of things at once.

Parallelism is about doing lots of things at once.

Not the same, but related.
One is about structure, one is about execution.

Concurrency provides a way to structure a solution to solve a problem that may (but not necessarily) be parallelizable.

— Rob Pike, co-inventor of the Go language

Parallelism is a special case of concurrency. In Python threading, multiprocessing, and asyncio are the core packages for concurrent programming. In this post we only address the asyncio package, which supports what are called native coroutines.

Python’s AsyncIO package Native coroutines – async/await

A native coroutine is any function that, as mentioned earlier, uses the async def syntax to define a function, introduced with PEP-492 in Python 3.5 (2015). Calling an async def function does not automatically execute the code in the function. To execute the code the await keyword must be used when calling the function. This instructs the asyncio event loop to schedule execution of the function.

import asyncio # Native coroutine - uses the "async def" syntax to define a function async def print_after(delay, what): await asyncio.sleep(delay) print(what) # An asyncio-compatible main function async def main(): print(f"started at {time.strftime('%X')}") await print_after(1, 'hello') await print_after(2, 'world') print(f"finished at {time.strftime('%X')}") # The asyncio task scheduler - uses the default asyncio event loop asyncio.run(main())

In Python the asyncio package provides the run function where you can run the default event loop and task scheduler with asyncio.run(my_main_coroutine()).

The image below illustrates how coroutines, the core task concept in asyncio, are run in the AsyncIO framework.

You have your program, the Python process, that sends tasks to the asyncio event loop with either an explicit call to asyncio.create_task() or use the await keyword to schedule a task in the asyncio event loop and wait for its completion within the body of the function that used the await keyword.

AsyncIO can be difficult to use correctly though it is usually easy to recognize due to most library authors targeting asyncio mark their async functions with async def. There is also the @types.coroutine annotation to make an existing generator function compatible with native coroutines. In order to use AsyncIO and get the performance benefits of using asyncio your whole program has to embrace the AsyncIO paradigm, meaning you use async def native coroutines for anything that does I/O or long-running tasks and you use await to schedule all coroutines.

Summary of AsyncIO

AsyncIO has a few main concepts for the async runtime, the asyncio event loop and an async def function as a coroutine. These basic concepts map nicely onto the HIO concepts of a Doist, the root scheduler in HIO, and the Doer, the coroutine or task concept in HIO. The main point where AsyncIO and HIO differ are that HIO has an explicit construct for creating hierarchies of tasks, the DoDoer. While there is no explicit construct in AsyncIO for a DoDoer any async/await coroutine could be considered to be a container for other coroutines.

Combining AsyncIO and HIO

Though asyncio native coroutines are not used at all in HIO the two approaches are compatible and composable. You can schedule AsyncIO tasks from a HIO task (a Doer) and you can also schedule a HIO task from an asyncio coroutine.

Yet first we must describe what HIO is. The subject of combining AsyncIO and HIO will be covered in a future article. This article is a short introduction to the three main classes of HIO’s async runtime implementation.

What is HIO?

HIO stands for Hierarchical IO. The README [1] describes it as weightless, hierarchical asynchronous coroutines and I/O in Python. This means that the task abstractions in HIO allow for nesting subtasks within tasks. HIO has three primary classes that make up its asynchronous runtime:
1. the Doist, or the root scheduler,
2. the DoDoer, a container holding either other DoDoer instances or Doer instances allowing you to create task hierarchies,
3. the Doer, the basic task or coroutine construct in HIO.

HIO makes heavy use of what are now known as “classic coroutines“[2] where the scheduler uses the my_coro.send(data) function to send data into a Python generator function. This generator function is the classic coroutine.

A few keywords distinguish classical coroutines including:

yield: used to pause execution of a coroutine (generator function), send a value out of a coroutine, and receive a value into a coroutine. yield from: used when nesting generators (inner generators) to pause execution of the outer generator and pass, or delegate, conrol to a sub-generator. Once the sub-generator completes then control is passed back to the outer generator. The yield from keyword is very similar to the await keyword from AsyncIO. Both drive sub-generators and both allow consumption of values returned by sub-generators. await does not completely replace yield from because await must be used inside a native coroutine and must be used with an awaitable object. yield from can be used in any function and with any iterable.

The yield keyword used in the body of a Python generator function allows it to receive values from the my_coro.send() function, similar to how Erlang/Elixir use the OTP to pass messages between processes with send and receive. The Python my_coro.send(data) is the “send” and the myvar = yield from invocation is the “receive.” And the yield from keyword used in the body of a classic coroutine allows delegating to, or transferring execution to, a nested or sub-generator.

This classic coroutine approach HIO uses is grounded in structured concurrency where there are clear entry and exit points to tasks, errors in concurrently executing tasks propagate up the task chain, and clear expression of control flow within the structure of source code despite the presence of concurrency. The context methods of a HIO Doer task provide the clear entry and exit points as well as a clear exception handling mechanism.

Overview

The root scheduler, the Doist, processes an array of Doer and DoDoer tasks. The DoDoer is the hierarchical task concept, and the Doer is the core task concept as shown below in the diagram.

Your program, the Python process, runs the Doist and the Doist runs the list of tasks until they finish or the program is terminated.

# from github.com/WebOfTrust/keripy/src/keri/app/cli/directing.py # module: keri.app.cli.directing # receives a list of tasks for the scheduler to run def runController(doers, expire=0.0): """ Utility Function to create doist to run doers """ tock = 0.03125 # creates the Doist, the root scheduler doist = doing.Doist(limit=expire, tock=tock, real=True) # adds tasks to the Doist to run. Calling "do" runs the Doist doist.do(doers=doers)

Here is a code example of creating an array of doers to pass to the root scheduler, the Doist, from KERIpy. This runWitness function shows the set of tasks that must be created in order to run a KERIpy witness.

# from github.com/WebOfTrust/keripy/src/keri/app/cli/commands/witness/start.py # module: keri.app.cli.commands.witness # Function used by the CLI to run a single basic witness def runWitness(name="witness", base="", alias="witness", bran="", tcp=5631, http=5632, expire=0.0): """ Setup and run one witness """ ks = keeping.Keeper(name=name, base=base, temp=False, reopen=True) aeid = ks.gbls.get('aeid') if aeid is None: hby = habbing.Habery(name=name, base=base, bran=bran) else: hby = existing.setupHby(name=name, base=base, bran=bran) hbyDoer = habbing.HaberyDoer(habery=hby) # setup doer doers = [hbyDoer] # list of tasks # extends the task list with the tasks from indirecting.setupWitness doers.extend(indirecting.setupWitness(alias=alias, hby=hby, tcpPort=tcp, httpPort=http)) # calls the Doist root scheduler with a list of tasks directing.runController(doers=doers, expire=expire)

This function creates a few tasks to be run and hands them off to the Doist scheduler with directing.runController. The scheduler then runs the tasks to completion, or infinitely, depending on the contents of the recur function shown below in the Doer.

HIO Task – a Doer

The core task concept in HIO is expressed as the Doer class shown in the UML diagram below. The HIO scheduler, a Doist, runs the Doer task until the .done attribute becomes True. There are six context functions five of which are executed over the lifecycle of the task including enter, recur, clean, close, and exit. The abort function is only called when a task is cancelled or an exception is raised.

HIO Scheduler – the Doist

At the top of the execution hierarchy in the HIO library you find the Doist class, the root scheduler of all task instances, or Doer instances. The generator returned from invoking a Doer is called a “deed” and is handed over to the Doist function. The Doist shown below has a list of deeds that are these generator functions, classic coroutines, that it runs when the Doist is executed.

To run a Doist you invoke the .do function on the Doist as shown below in a test adapted from HIO.

def test_doist_doers(): """ Test doist.do with .close of deeds """ tock = 0.03125 doist = doing.Doist(tock=tock) # creates a Doer, an example doer doer0 = doing.ExDoer(tock=tock, tymth=doist.tymen()) # creates a Doer, an example doer doer2 = doing.ExDoer(tock=tock, tymth=doist.tymen()) doers = [doer0, doer1] doist.do(doers=doers) # run the Doist assert doer0.done == True Context Functions

The six context functions in the Doer are run by the enter and exit functions of the Doist as well as the do function of the Doer. Each of these functions serve as a lifecycle hook for a different time in the execution of the Doer. The .do function reproduced below shows where each context function is executed after calling Doer.do. Take special notice of the while loop inside of the try/except block. This is the loop that continues to run the body of the Doer, the function or generator that does the work of the Doer.

# from github.com/ioflo/hio/src/hio/base.doing.py class Doer(tyming.Tymee): ... def do(self, tymth, *, tock=0.0, **opts): """ Generator method to run this doer. Calling this method returns generator. Interface matches generator function for compatibility. To customize create subclasses and override the lifecycle methods: .enter, .recur, .exit, .close, .abort Parameters: tymth is injected function wrapper closure returned by .tymen() of Tymist instance. Calling tymth() returns associated Tymist .tyme. tock is injected initial tock value args is dict of injected optional additional parameters """ try: # enter context self.wind(tymth) # update tymist dependencies self.tock = tock # set tock to parameter self.done = False # allows enter to override completion state self.enter() # (1) first context function, enter #recur context if isgeneratorfunction(self.recur): # .recur is generator method self.done = yield from self.recur() # (2) recur context delegated, second context function else: # .recur is standard method so iterate in while loop while (not self.done): # recur context tyme = (yield (self.tock)) # yields .tock then waits for next send self.done = self.recur(tyme=tyme) # (2) second context function, recur except GeneratorExit: # close context, forced exit due to .close self.close() # (3) third context function, close except Exception as ex: # abort context, forced exit due to uncaught exception self.abort(ex=ex) # (4) fourth context function, abort raise else: # clean context self.clean() # (5) fifth context function, clean finally: # exit context, exit, unforced if normal exit of try, forced otherwise self.exit() # (6) sixth context function, exit # return value of yield from or StopIteration.value indicates completion return self.done # Only returns done state if normal return not close or abort raise

In the normal execution of a Doer the .do() function calls, in this order, enter, recur, clean, and then exit. The close context function is only executed when it is explicitly called by some higher level construct such as a DoDoer or the Doist scheduler itself.

In an error case, or abnormal execution of a Doer, the abort context function is called. This can also be called as a part of normal execution of a program to catch a shutdown signal to instruct a DoDoer or a Doist to perform a graceful shutdown.

HIO DoDoer – where task hierarchies are defined

This post touches lightly on DoDoers to say that the DoDoer provides hierarchical task management which means you can nest tasks for a clear hierarchy of task execution for groups of tasks. A future article will detail the definition and usage of the DoDoer.

AsyncIO vs HIO – How do they compare?

Classic coroutines are very powerful constructs that provide a richer control flow construct as compared to AsyncIO’s async def coroutine construc. This is because you can use any number of yield or yield from statements in the body of a classic coroutine, which provides you with the ability to custom-fit the execution of a generator-based coroutine to your specific use case. The async/await syntax does a similar thing for you, yet with a standard syntax that you cannot customize.

With HIO you can also repeatedly accept information into a classic coroutine instance through the yield from syntax. The fact that classic coroutines are just generator functions means you have full control over iteration of that generator, and all of it’s contained state including any state it has closed over, from an async context with all the power of Python iterators.

For example, you could run a classic coroutine any arbitrary number of times within a custom scheduler depending on special rules and have fine-grained access to what is sent into the coroutine with the .send() function.

Yet with this additional power comes the potential to have complicated and hard to understand control flow. It is understandable why there would be so much support in the Python community for a simpler, less powerful syntax, which is what async/await is. The linked article[2] from Luciano Ramalho goes in depth on the features of both classic coroutines and Python’s AsyncIO.

Wrap up and Next Steps

This article focused on the “what” of the async framework side of HIO, specifically the three primary classes at the core of the async runtime in HIO, the Doist scheduler, DoDoer hierarchical task container, and the Doer task class. The raw power of classic coroutines significantly influenced the decision to use them in HIO as well as in KERIpy, KERIA, and SignifyPy. Yet, this is not an either-or, all-or-nothing situation. You can use HIO and AsyncIO together.

Major topics not covered in this article that are important to understand HIO include details of the DoDoer and the network and file I/O capabilities of the HIO package.

Future articles will delve deeper into the “why” of HIO, the rationale behind HIO, how and when to use it properly, as well as how to use HIO and AsyncIO together. To gain a deeper understanding of HIO one of your next steps would be to read some of the tests in the HIO source code repository, specifically the test_doist_once and test_nested_doers tests.

References

[1] S. Smith, “hio/README.md at main · ioflo/hio,” GitHub, Aug. 21, 2021. https://github.com/ioflo/hio/blob/main/README.md (accessed May 09, 2024).

[2] L. Ramalho, “Classic Coroutines,” Fluent Python, the lizard book, Apr. 2022. https://www.fluentpython.com/extra/classic-coroutines (accessed May 11, 2024).

[3] Real Python, “Async IO in python: A complete walkthrough,” Real Python, https://realpython.com/async-io-python (accessed May 9, 2024).

Saturday, 11. May 2024

Ben Werdmüller

The itch

I’m really itching to build something new again. Not a new widget or open source project, but a new service. Something that makes peoples’ lives better. I love startups. And the ideas are brewing.

I’m really itching to build something new again.

Not a new widget or open source project, but a new service. Something that makes peoples’ lives better.

I love startups. And the ideas are brewing.


Werdmüller on Medium

Think twice before exercising your stock options

Startup options and when they’re worth it Continue reading on Medium »

Startup options and when they’re worth it

Continue reading on Medium »


Ben Werdmüller

Think twice before exercising your stock options

I recently wrote a short aside about stock options: But in general, for regular employees, I think options are rarely worth it. They typically require an up-front investment that many employees simply can’t make, so it’s a bit of a fake benefit to begin with, and their future value is little more certain than a lottery ticket. Hunter Walk kindly reshared it on a few networks with some

I recently wrote a short aside about stock options:

But in general, for regular employees, I think options are rarely worth it. They typically require an up-front investment that many employees simply can’t make, so it’s a bit of a fake benefit to begin with, and their future value is little more certain than a lottery ticket.

Hunter Walk kindly reshared it on a few networks with some of his own thoughts; a conversation with Tony Stubblebine arose in the comments that Hunter wrote up as its own post. In particular, he says it helped him articulate the ups and downs of private stock to the average person:

For much of a startup’s life new FUNDING VALUATIONS are LEADING indications of POTENTIAL. They are what someone is willing to pay for shares today based on what they believe the company CAN DO in the FUTURE.

DOWN ROUNDS and RECAPS are LAGGING indications of PERFORMANCE. They are what someone is willing to pay for shares today based upon what the company HAS DONE in the PAST.

It’s a great post, and the comments from Tony were thoughtful. Which led me to feeling a bit bad about how flippant and imprecise my original post had been.

So, on that note, I’d love to define options, make some corrections, and dive a little deeper into my core argument.

The ins and outs of options

First, let’s define options and explain why they’re so common as a factor of startup compensation.

An option is the right to buy a specified number of shares in a company at a specific price. That price is typically defined by an external auditor. It’s good practice for this to happen once a year, but it’ll also be triggered when the company raises a round of equity funding (i.e., when it sells shares to outside investors in order to raise significant capital).

If a startup were simply to grant stock directly to employees, it would be taxable as compensation. Options are almost always non-taxable at the point where they are issued, so they’re a favorite way to give employees the ability to see some of the potential upside in a venture.

Typically in a startup you’ll receive an option grant as part of your compensation package. So, for example, you might receive the right to buy (“exercise”) 40,000 shares at 50 cents a share (the “strike price”). This is almost always on what’s called a vesting schedule: you won’t be able to buy any shares in the first year, but then when you cross that threshold (the “cliff”), you’ll be able to buy 25% of your allocation (the first 10,000 shares in my example). Over the next three years, the amount of your allocation that you can exercise will increase proportionally, until you can buy them all at the end of four years.

If you leave the company, you usually only have 90 days to exercise whichever options have vested. Some particularly progressive companies extend that exercise window — sometimes to a couple of years. But for 80-90% of startups, it’s 90 days.

If the startup is excited about keeping you, you may find that they’ll grant you more options periodically, each with their own vesting schedules. This, they hope, will keep you at the company.

In my example above, you might have done the math to realize: 40,000 shares at 50 cents a share is $20,000. You would need to lay out that amount of money to acquire the shares — and you need to hope that the company’s shares increase in value in order to see any upside.

If the company’s share price has increased in the time between the options were granted and when the employee exercises them, the difference is taxable. In the above example, recall that my options are for 40,000 shares at 50 cents a share. Let’s say I choose to exercise them all at the end of my four year vesting period: as we’ve discussed, I pay $20,000. But let’s say that the real fair market value has risen to 75 cents a share. The difference between 40,000 shares at 50 cents and 40,000 shares at the market value of 75 cents is $10,000 is usually taxed as income. So I’m actually paying $20K + income tax on another $10K. (This isn’t by any means the full extent of potential tax implications; I’m not going to touch ISOs and AMT in this post, for example.)

Early employees, who join before most funding rounds have taken place, will receive options with a very low exercise price. Later employees will usually receive options with a higher price, because more growth and fundraising has taken place in the interim. (Down rounds and recaps are certainly possible, though: many startups go through tough times where their valuation decreases. Not every graph always goes up and to the right.)

In both cases, any stock they buy is largely illiquid. Because the startup is likely a private company rather than a publicly traded one, their shares are not liquid. They will need to wait for the company to go public or hope that management will allow them to trade their stock on the secondary market.

Some corrections

So the first thing to say is: no, options are not really like a lottery ticket. They are a sort of gamble, but it’s one where (depending on your position, seniority, and what size the company was when you joined) you have a say in the outcome.

The second, which I’ve already corrected in the original post is: as Hunter pointed out in his post, a recap is not the thing that actually lowers the stock price. It’s a trailing signal of what the company has already done. A change in stock price is an effect of what has already happened.

And a clarification: options don’t require an up-front investment at the time that they’re granted. You invest at the time when you exercise them, which may still be as a lump sum.

Why I think exercising options isn’t worth it for many employees

If you’re on a rocket ship startup, exercising your options is almost certainly worth it (depending on the strike price of your particular options grant). The problem is: how do you know you’re on a rocket ship? Or, given that most startup employees won’t be part of a startup with hockey-stick growth, how can you be reasonably sure that your company will grow in such a way that exercising your options is worth it?

90% of startups fail. That doesn’t mean that every startup has an equal 1 in 10 chance of success: a lot depends on a range of factors that include internal culture, management expertise, execution quality, and market conditions. Still, there is not a small amount of luck involved. Most startups won’t make it.

You should never make an investment that you can’t afford to lose. As Hunter says in his post:

Don’t behave as if they’re worth anything until they actually are

Don’t over-extend yourself to exercise [options] in scenarios which put your financial well-being at risk.

If you’re obviously, unquestionably on a rocket ship: by all means, buy the options. (Yes, sometimes it really is obvious.)

If it’s not clear that you’re on a rocket ship, but you’re feeling good about the startup, and you can definitely afford to spend the money it would take to exercise your options: knock yourself out. Honestly, I don’t really care what people with wealth do in this scenario. My worries do not relate to you.

If it’s not clear that you’re on a rocket ship and spending the money to exercise your options would be a stretch: I would suggest you think twice before doing so. I also would warn you to never take out debt (which many startup employees do!) in order to exercise your options.

And that’s really the crux of my argument.

Startup employees without significant independent spending power who work for a venture with an uncertain future and who did not join their ventures at a very early stage — which I would argue describes most startup employees — should think long and hard before exercising their options.

It’s more than a little bit unfair that the people who can most easily realize upside from the startups they work for are people who already have wealth. Granting the ability for employees to buy shares directly at their fair market value is limited, too: this would make them investors, who the SEC says mostly need to be accredited. The definition of accreditation is either being a licensed investor, earning over $200,000 a year for the last two years, or having a net worth of over a million dollars excluding the value of their home. So the door is effectively closed to people from regular backgrounds.

I wish more equitable systems were commonly in use. Some different tactics are in use, which include:

Restricted Stock Units. Here, stock is granted directly as part of an employee’s compensation. Upside: the employee has the shares. Downside: they’re taxed on them as soon as they vest, and selling them is restricted. So the employee effectively receives an additional tax bill with no way of recouping the lost funds until much later (if they’re lucky). RSUs are common in later-stage companies but very uncommon in riskier, early-stage companies for this reason. Phantom stock. Really this is a bonus plan tied to stock performance, income tax and all. Profit sharing. Which is only useful if the startup makes a profit (most don’t).

While some have value in their own right in particular contexts, I see them as compensation strategies that might sit alongside stock options, rather than replacing them.

I would love it to be less risky for the employees who are actually doing the work of making a startup valuable to see more of the upside of that work. But, at least for now, my advice remains to take those inflated Silicon Valley salaries and bank them in more traditional investments.


Timothy Ruff

KERI 101: Witnesses, Watchers, & Detectability

The “oil lights” of cryptographic key management. TL;DR KERI is an open, decentralized protocol for discovery, recovery, and management of cryptographic keys. KERI enables anyone to digitally sign digital things and to verify what others have signed, without the need for shared networks, blockchains, registries, platforms, or identity providers. KERI solves seven hard problems of key managemen
The “oil lights” of cryptographic key management. TL;DR KERI is an open, decentralized protocol for discovery, recovery, and management of cryptographic keys. KERI enables anyone to digitally sign digital things and to verify what others have signed, without the need for shared networks, blockchains, registries, platforms, or identity providers. KERI solves seven hard problems of key management: Rotation, Recovery, Detectability, Discovery, Delegability, Revocability, and Multi-Signature. KERI has three Detectability functions — Witnesses, Watchers, and Anchored Issuances — that are like the oil light of a car: they detect key compromise and malicious behaviors of Signers and service providers.

With KERI Detectability functions…

Signers can detect:

Their own compromised keys when used by hackers for signing; Their own compromised keys when used by hackers to rotate, delegate, or revoke their signing keys.

And Verifiers can detect:

Compromised Signers; Malicious Signers; Their own malicious or compromised verification service provider.

Without KERI Anchored Issuances in particular, if a Signer suspects key compromise and performs a rotation, it cannot determine which issuances were made fraudulently, and must revoke and reissue all previously issued credentials. (!)

With KERI Anchored Issuances, fraudulently issued credentials will not successfully verify before or after key rotations, whereas authentically issued credentials will continue to successfully verify, even after an unlimited number of key rotations.

What is KERI?

KERI — Key Event Receipt Infrastructure — is a new (2019), open, decentralized protocol for discovering, recovering, and managing cryptographic keys. KERI enables people, organizations, and things to sign digital things and verify what others have signed, without the need for shared networks, blockchains, registries, platforms, or identity providers.

With KERI, Signers and Verifiers worldwide maintain their own infrastructure without anything in common, but they do share one game-changing thing: a protocol for discovering the Signer’s current key state. This enables Verifiers to securely confirm that a piece of data has been signed by a particular Signer, and that it hasn’t been tampered with, revoked, or expired.

KERI is “end verifiable”; the means through which data travels needn’t be secure for it to still be verifiably authentic when it arrives at its destination. KERI is zero trust, enabling objective verifiability without subjective trust, and it creates no centralized troves of valuable data. KERI is quantum-resistant and uses existing cryptography and cloud infrastructure.

In short, KERI brings the ability to distinguish fake from real in the digital realm, across networks, borders, and boundaries of all kinds.

Seven Hard Problems of Key Management

KERI address seven hard problems of key management, listed below. Most were previously not solvable in a decentralized manner, and some not in a centralized one.

Seven hard problems (with the KERI solution):

Rotation (change current keys for new, without re-issuing past issuances) Recovery (from private key loss or compromise, without re-issuing past issuances) Detectability (of key compromise or malicious Signer behavior) Discovery (just-in-time discovery of key state) Delegability (directly from one Signer/key controller to another) Revocability (decentralized, instant, with privacy) Multi-signature (weighted m-of-n, verifiable) Detectability: Witnesses, Watchers, & Anchored Issuances Detectability Is the ‘Oil Light’ of Key Management

Together, KERI Witnesses, Watchers, and Anchored Issuances, all described below, enable Detectability. Detectability is like the oil light of a car: ignore it, disconnect it, or omit it at your peril, because it is the most critical indicator on your dashboard.

Removing Witnesses, Watchers, and Anchored Issuances from a system removes Detectability.

Without Detectability, Signers cannot detect:

Their own compromised keys when used by hackers for signing; Their own compromised keys when used by hackers to rotate, delegate, or revoke the Signer’s keys.

And Verifiers cannot detect:

A Signer’s compromised keys; Malicious Signers; Their own malicious or compromised verification service provider. What Are Witnesses?

KERI Witnesses are secure, distributed replicas of a Signer’s key event log. They are established (built or bought) and controlled by a Signer (also a “key controller” or “issuer”). A Signer can be anyone or anything, anywhere.

Witnesses make separate, additional replicated points of verification available to Verifiers, enabling Verifiers to detect both compromised and malicious behaviors by Signers. Witnesses can be web servers, distributed databases, blockchains, or any other addressable data source.

To compromise a KERI identifier, each of its Witnesses must also be compromised.

There are no limits or requirements as to whether a Signer uses Witnesses at all or how many; the KERI protocol simply enables any number of Witnesses to be established, deployed however the Signer chooses, and for Verifiers to verify that all Witnesses are acting in unanimity about the Signer’s key state.

What Are Watchers?

KERI Watchers are services established (built or bought) by a Verifier — who can be anyone or anything anywhere — that enable the Verifier to confidently rely on what their own service providers are telling them about a Signer’s key state.

Watchers help Verifiers avoid the single-oracle problem: if you have only one source of truth, you cannot know if that source has been compromised or is acting maliciously (deliberately lying to you). When multiple distinct Watchers all report the same key state of a Signer, it can be more safely relied upon. Having Watchers removes a tempting single point of vulnerability for hackers.

As with Witnesses, there are no limits or requirements as to whether Watchers are used at all or how many; the KERI protocol simply enables any number of Watchers to be established and deployed however desired.

Anchored Issuances: A Breakthrough in Detecting and Preventing the Use of Stolen Keys

Another critical element of Detectability, separate from Witnesses and Watchers, is “Anchored Issuances” (referred to as “seals” in the KERI white paper).

When issued credentials are properly anchored in one’s Key Event Log (KEL) and properly witnessed and watched, it becomes impossible for a hacker of your private keys to use them without your detection; the unauthorized issuances will not successfully verify. With un-anchored / “unbound” issuances, hackers can use stolen private keys as they please with zero detectability and, unfortunately, every issuance they make will successfully verify.

Akin to a check register, Anchored Issuances provide a secure record of issued credentials that can be instantly and cryptographically confirmed by Verifiers. No other system we are aware of provides detectability or prevention of the use of stolen private keys.

No Detectability? Rotate, Revoke, and Re-Issue Everything. (Really)

Some alternative key management systems borrow KERI’s primary functions — pre-rotation, SCIDS, etc. — but forgo its Detectability functions. For them and for all non-KERI key management systems, a big future headache awaits: if a Signer suspects key compromise and performs a rotation, there is no way to distinguish valid issuances from invalid ones — they will all continue to verify successfully — so a Signer must revoke and reissue all previously issued credentials. (!)

That is, if a Signer wishes to maintain a positive reputation for its credentials. Revoking and reissuing everything would likely be a user experience nightmare, which alone defeats the purpose of having pre-rotation. Pre-rotation is arguably KERI’s greatest innovation, but does not achieve its full potential without Detectability.

Can’t We Just Rotate More Often?

Yes, but it won’t help much. KERI pre-rotation can be done repeatedly without downstream costs or consequences for KERI identifiers, a breathtaking breakthrough in key management. But… while a Signer’s more-frequent rotation might attempt to repair one vulnerability — compromise of the Signer’s keys — it does not address other vulnerabilities listed in the “KERI Detectability” section, and it still leaves the ugly problem of revoking and re-issuing everything if a compromise is suspected. In other words, removing Detectability leaves significant holes in the security model no matter how often a Signer rotates their keys.

Such a tradeoff may be acceptable for less critical use cases or at smaller scale, but is likely unacceptable for most organizations.

In Conclusion

KERI Witnesses, Watchers, and Anchored Issuances are the ‘oil lights’ of cryptographic key management, enabling robust detectability of malicious or compromised digital signing behaviors for both Signers and Verifiers. KERI Anchored Issuances goes beyond detection to full-on prevention, actively blocking the use of stolen private keys for credential signing.

KERI Detectability brings critical advances in key management specifically and cybersecurity generally, and should be seriously considered for inclusion within any comprehensive cybersecurity, identity, or key management system.

Friday, 10. May 2024

Just a Theory

PGXN Language Poll Result

Results of the Postgres community poll for building PGXN v2 in Go, Rust, or both.

Back on March 28, I asked the Postgres community whether new services for PGXN v2 should be written in Go, Rust, or “some of each”. I went so far as to create a poll, which ran through April 12. A month later you might reasonably be wondering what became of it. Has David been refusing to face reality and accept the results?

The answer is “no”. Or at least I don’t think so. Who among us really knows ourselves. Since it closed, the poll has provided the results since it closed, but I suspect few have looked. So here they are:

Candidate Votes % All Votes 🦀 Rust 102 60.4% 🐿️ Go 53 31.4% 🐿️ + 🦀 Some of each 13 7.7%

🦀 Rust is the clear winner.

I don’t know whether some Rust brigade descended upon the poll, but the truth is that the outcome was blindingly apparent within a day of posting the poll. So much so that I decided to get ahead of things and try writing a pgrx extension. I released jsonschema on PGXN on April 30. Turned out to be kind of fun, and the pgrx developers kindly answered all my questions and even made a new release to simplify integration testing, now included in the pgrx-build-test utility in the pgxn-tools Docker image.

But I digress. As a result of this poll and chatting with various holders of stakes at work and haunting the #extensions Slack channel, I plan to use Rust for all new PGXN projects — unless there is an overwhelmingly compelling reason to use something else for a specific use case.

Want to help? Rustaceans welcome! Check out the project plan plan or join us in the #extensions channel on the Postgres Slack.

More about… Postgres PGXN Go Rust Perl

Doc Searls Weblog

Personal vs. Personalized AI

There is a war going on. Humanity and nature are on one side and Big Tech is on the other. The two sides are not opposed. They are orthogonal. The human side is horizontal and the Big Tech side is vertical.* The human side is personal, social, self-governed, heterarchical, open, and grounded in the physical […]

There is a war going on. Humanity and nature are on one side and Big Tech is on the other. The two sides are not opposed. They are orthogonal. The human side is horizontal and the Big Tech side is vertical.*

The human side is personal, social, self-governed, heterarchical, open, and grounded in the physical world. Its model is nature, and the cooperative contexts in which competition, creation, and destruction happen in the natural world.

The Big Tech side is corporate, industrial, hierarchical, competitive, mechanistic, extractive, and closed, even though it produces many positive-sum products and services that are good for people and good for nature. It is also, being competitive and rewarding toward winner-take-most outcomes, dominated by giants.

This war has been fought over many other things in the past, especially in tech. But AI is the big one right now—and perhaps the biggest one of all time.

Over the long run, both sides will win, because we need the best of what both bring to the world’s big round table. In the past, this has happened in countless markets, countries, polities, societies, and other contexts. In tech it happened with the conflict between PCs and mainframes, between the open and public Internet and closed private networks, and between open operating systems and closed ones. None of those conflicts have final outcomes, but they have arrived at stable and cooperative states. They arrived at those states because everyone recognized that people can do more with their tools than those big entities can do on their own, or by using people as extracted resources.

This conflict is perhaps best represented by the ironies that have been at work within OpenAI and Sam Altman, its prior and current CEO. We can also see it with sparkling clarity in something amazing that Sam said near the close of his keynote at the first OpenAI Developer Conference, in November of last year:

Lets unpack that.

Individual empowerment means exactly what it says. Agency is the ability to take action or to choose what action to take (Cambridge Dictionary). Scale means growth toward ubiquity.

You have individual empowerment and agency when you ride your bike, drive your car, mow your lawn, or button your shirt. You also have it when you work on your laptop, phone, and other smart rectangles. Scale means your power and agency will grow to the max while the same will happen for nearly everyone else. Scale for you alone won’t cut it. When most of us have it, that’s scale.

Technologies extend us. They enlarge our capacities in the world. Intelligence is one of those capacities. So is memory. Your rectangles help a lot with both. That’s why those have already scaled to ubiquity, or close enough.

AI will do the same, but only if it’s personal. Not if it’s  just “personalized.”

As Jamie Smith made clear here a year ago, “your personal AI isn’t really personal.”

ChatGPT4 doesn’t know that yet. I can tell because I just asked:

Me:
What is “personal AI”?

ChatGPT4:
“Personal AI” refers to artificial intelligence systems that are designed to serve individual users through personalized interactions and services. These AI systems learn from the specific preferences, behaviors, and needs of their users to offer more tailored assistance, recommendations, or information. The goal of personal AI is to enhance user experience by adapting to unique personal profiles.

There are various applications of personal AI, including:

Virtual Assistants: Like Siri, Google Assistant, and others, these AIs help with daily tasks such as scheduling, reminders, searching information, and controlling smart home devices, all customized to the user’s habits and preferences. Recommendation Systems: Used in streaming services like Netflix or shopping platforms like Amazon, these AI systems analyze your past behavior to recommend movies, shows, or products you might like. Healthcare: Personal AI can track individual health metrics and provide personalized advice, monitor conditions, or even alert medical professionals if needed. Learning and Development: In education, personal AI can adapt learning materials to the pace and learning style of the student, offering a customized educational experience. Gaming: AI in gaming can adapt to provide a challenge tailored to the player’s skill level or alter the game narrative based on choices made by the player.

The development of personal AI focuses on enhancing user engagement by creating more intuitive and responsive interactions that feel uniquely suited to each individual.

For a second opinion, I asked Google’s Gemini the same question.

Me:
What is “personal AI”?

Gemini:
Personal AI refers to artificial intelligence designed to assist and interact with a specific individual. It’s essentially your own AI companion that learns about you and tailors its services to your needs. Here are some key aspects of personal AI:

Personalization: Unlike general-purpose AI assistants, personal AI focuses on you. It learns from your interactions, preferences, and habits to become more helpful over time. Focus on Memory and Knowledge: Some personal AI systems aim to become an extension of your mind. They can store your memories, knowledge, and experiences, making them readily accessible and even usable to help with tasks like writing or communication. Varied Applications: Personal AI can have a wide range of applications. Imagine an AI that helps you: Organize your life: Manage schedules, appointments, and reminders. Boost productivity: Assist with tasks, research, and communication. Enhance creativity: Help you brainstorm ideas or generate creative content. Improve learning: Personalize learning experiences and provide targeted information.

Privacy Concerns: Storing personal data on an AI raises concerns about privacy and security.

Overall, personal AI holds promise for a future where technology seamlessly integrates with our lives, offering personalized assistance and support.

To both companies, personal AI is a personalized service—from them. It’s not something you own and control. It’s not about individual empowerment and agency. It doesn’t extend you. It’s hired prosthesis: an extension of them rather than of you.

But hats off. OpenAI and Gemini do an amazing job. So do lots of other AI services by wannabe giants. I use AI every day to improve what I write, to generate and fix images, to answer questions that old-fashioned search engines can’t answer or suck at. I even pay extra for some of it (such as ChatGPT4 and Adobe’s Creative Suite). And it seems they all get better, all the time, at everything. Won’t personalization be good enough, eventually?

No. Because they aren’t you. They also aren’t yours, so they can’t extend you. They can seem to. But they don’t. They also don’t have access to most of your private life. Nor should they.

But your private life could use some AI help. For example—

Schedules, including on your calendars, past and future Health data, including all your medical reports, prescriptions, appointments, insurance information, past and present providers, plus what your watch, phone, and other devices record about you Financial records, including bills, receipts, taxes, and anything called an account that involves money Travel, including all the movements your phone (and phone company), watch, and car record about where you go and where you’ve been Work—past and present, including whatever HR systems know or knew about you Contacts—all the people, businesses, and other entities you know Business relationships, with brokers, retailers, service providers, whatever Subscriptions, including all those “just $1 for the first four weeks” offers you’ve accepted, plus other forms of screwage that are stock-in-trade for companies selling subscription systems to businesses. Property, including all the stuff on your shelves, floors, closets, garages, and storage spaces—plus your stocks and real estate.

It’s not easy to visualize what a personal AI might do for those, but let’s try. Here’s how Microsoft’s Copilot (or whatever it’s called this week) did it for me before I got rid of all its misspellings and added my own hunks of text:

All that stuff is data. But most of it is scattered between apps and clouds belonging to Apple, Google, Microsoft, Amazon, Meta, phone companies, cable companies, car makers, health care systems, insurance companies, banks, credit card companies, retailers, and other systems that are not yours. And most of them also think that data is theirs and not yours.

To collect and manage all that stuff, you need tools that don’t yet exist: tools that are yours and not theirs. We could hardly begin to imagine those tools before AI came along. Now we can.

For example, you should be able to take a picture of the books on your shelves and have a complete record of what those books are and where you got them. You’ll know where you got them because you have a complete history of what you bought, where and from whom. You should be able to point your camera in your closets, at the rugs on your floors, at your furniture, at the VIN number of your car that’s visible under your windshield, at your appliances and plumbing fixtures, and have your AI tell you what those are, or at least make far more educated guesses than you can make on your own.

Yes, your AI should be able to tap into external databases and AI systems for help, but without divulging identity information or other private data. Those services should be dependent variables, not independent ones. For full individual empowerment and agency, you need to be independent. So does everyone else with personal AI.

Now imagine having a scanner that you can feed every bill, every receipt, every subscription renewal notice, and have AI software that tells you what’s what with each of them, and sorts records into the places they belong.

Ever notice that the Amazon line items on your credit card bill not only aren’t itemized, but don’t match Amazon’s online record of what you ordered? Your personal AI can sort that out. It can help say which are business and personal expenses, which are suspicious in some way, what doesn’t add up, and much more.

Your personal AI should be able to answer questions like, How many times have I had lunch at this place? Who was I with? When was it we drove to see so-and-so in Wisconsin? What route did we take? What was that one car we rented that we actually liked?

Way back in 1995, when our family first got on the Internet over dial-up, using the first graphical browsers on our PC, and e-commerce began to take off with Amazon, eBay, and other online retailers, my wife asked an essential question: Why can’t I have my own shopping cart that I take from site to site?

Twenty-nine years later, we still don’t have the answer, because every retailer wants you to use its own. And we’re stuck in that system. It’s the same system that has us separately consenting to what sites ironically call “your privacy choices.” And aren’t.

There are countless nice things we can’t have in the digital world today because we aren’t people there. We are accounts. And we are reduced to accounts by every entity that requires a login and password.

This system is a legacy of client-server, a euphemism for slave-master. We might also call it calf-cow, because that’s how we relate to businesses with which we have accounts. And that model is leveraged on the Web like this:

We go to sites for the milk of content and free cookies, whether we want them or not. We are also just “users.”

In the client-server world, servers get scale. Clients have no more scale than what each account—each cow—separately allows. Sure, users get lots of benefits, but scale across many cows is not one of them. And no, “login with Google” and “login with Facebook” are just passes that let calves of ruling cows wander into vassal pastures.

For individual empowerment and scale to happen, we need to be self-sovereign and independent. Personal AI can give that to us. It can do that by solving problems such as the ones listed above, and by working as agents that represent us as human beings—rather than mere users—when we engage with Big Tech’s cows.

This will be a fight at first, because the cows think they run all of nature and not just their own farms. And $trillions are being invested in the same old cattle industry, with AI painted all over the new barns. Comparatively speaking, close to nothing is going toward giving independent and self-sovereign individuals the kind of power and scale Sam Altman says he wants to give us but can’t because he’s on the big cow side of this thing.

So where do we start?

First, with open source code and open standards. We have some already. Llama 3, from Meta AI, is “your own intelligent assistant,” and positions Meta as a more open and user-friendly cow than OpenAI. Meta is still on the top-down Big Tech side of the war we’re in. But hell, we can use what they’ve got. So let’s play with it.

Here on the ground there are all these (with quotage lifted from their sites or reviews such as this one)—

MindsDB: “an open-source AutoML framework” Alt.ai: “It’s an A.I. which aims to digitize users’ intentions and place it on the cloud to let our clones deal with all digital operations.” Keras: “a multi-backend deep learning framework, with support for JAX, TensorFlow, and PyTorch” PyTorch: “Python package that provides two high-level features: Tensor computation (like NumPy) with strong GPU acceleration, and Deep neural networks built on a tape-based autograd system Tensor Flow: “open-source framework for machine learning” CoreNet: a deep neural network toolkit for small and large-scale models, from Apple Haystack: an “open source Python framework by deepset for building custom apps with large language models (LLMs).” Image Super-Resolution (ISR): “(an) open source tool employs a machine learning model that you can train to guess at the details in a low-resolution image: Blender: “A rich interface and numerous plugins make it possible to create complex motion graphics or cinematic vistas” DeepFaceLab: “open source deepfake technology that runs on Python” tflearn: “an advanced deep learning library” PYTensor: “a Python library that allows you to define, optimize/rewrite, and evaluate mathematical expressions involving multi-dimensional arrays efficiently.” (Was Theano) LM Studio: “Discover, download, and run local LLMs” HuggingFace Transformers: “a popular open-source library for Natural Language Processing (NLP) tasks” Fast.ai: “a library for working with deep learning tasks” OpenCV: “a popular Computer Vision and Image Processing library developed by Intel” Detectron2: “a next-generation library that provides advanced detection and segmentation algorithm” and “a PyTorch-based modular object detection library” Ivy.ai: “an open-source deep learning library in Python focusing on research and development” OpenAssistant: “a project aimed at giving everyone access to a great chat-based large language model” PaddleNLP: “a popular open source NLP library that you can use to glean search sentiment and flag important entities” Delphi.AI: “Clone yourself. Build the digital version of you to scale your expertise and availability, infinitely.” Fauxpilot: “This is an attempt to build a locally hosted alternative to GitHub Copilot. It uses the SalesForce CodeGen models inside NVIDIA’s Triton Inference Server with the FasterTransformer backend.” Ray: “An open source framework to build and scale your ML and Python applications easily” Solid: “Solid is a specification that lets individuals and groups store their data securely in decentralized data stores called Pods. Pods are like secure web servers for data. When data is stored in a Pod, its owners control which people and applications can access it.” Sagen.ai: “Your very own AI Personal Assistant to manage your digital life.” YOLOv7: “is one of the fastest and most accurate open source object detection tools. Just provide the tool with a collection of images full of objects and see what happens next.”

—and lots of others that readers can tell me about. Do that and I will add links later. This is a work in progress.

Below all of those we still need something Linux-like that will become the open base on which lots of other stuff runs. The closest I’ve seen so far is pAI-OS, by Kwaai.ai, a nonprofit I now serve as Chief Intention Officer. I got recruited by Reza Rassool, Kwaai’s founder and chair, because he believes personal AI is required to make The Intention Economy finally happen. So that was a hard offer to refuse. Kwaai also has a large, growing, and active community, which I believe is necessary, cool, and very encouraging.

As with most (maybe all) of the projects listed above, Kwaai is a grass-roots effort by human beings on the natural, human, and horizontal side of a battle with giants who would rather give us personalized AI than have us meet them in a middle to which we will bring personal AI powers of our own. In the long run, we will meet in that middle, because personal AI will be better for everyone than personalized AI alone.

Watch us prove it. Better yet, join the effort.

*I am indebted to Lavonne Reimer for introducing and co-thinking the horizontal vs. vertical frame, and look forward eagerly to her own writings and lecturings on the topic.


Wrench in the Gears

Unwinding – Smart City to Country Roads

I recognize going on and on about my personal situation may be rather tedious. I really don’t mean to make it all about “me,” because I feel like each of our journeys are part of this unfolding multi-dimensional tapestry. Since this is my blog, it seems like an appropriate place to tell the story from [...]

I recognize going on and on about my personal situation may be rather tedious. I really don’t mean to make it all about “me,” because I feel like each of our journeys are part of this unfolding multi-dimensional tapestry.

Since this is my blog, it seems like an appropriate place to tell the story from my particular point of view. For now, I’m giving myself leeway to share these loose, stream of consciousness “notes” – observations, pictures, mullings over – a work in progress. If this upsets or bores you, feel free to take a break from my blog and check back later in the summer or fall. Hopefully by then, I’ll have found my new footing.

On the way down to the Shenandoah yesterday, I was listening again to the biography of Elizebeth Friedman, a skilled codebreaker, who with her husband William laid the groundwork for cryptanalysis in the twentieth century. They were initially based at Riverbank Labs on the Fox River outside of Chicago, a stone’s throw from the future site of Fermi Lab.

https://www.cabinetmagazine.org/issues/40/sherman.php

The secretive Colonel Fabyan founded one of the first Defense Department think tanks in the 1910s (pre-Alfred Loomis), cultivating ties with the University of Chicago. He used his fortune to underwrite private research into plant and fruit fly genetics , bone structure, x-rays, and acoustics, the latter in partnership with Wallace Sabine. You can still purchase precision tuning forks from Riverbank Labs.

http://riverbanklabs.com/about/

What took up much of the Colonel’s attention was research into the Baconian cipher, supposedly encrypted messages about Elizabethan England hidden in Shakespeare’s first folio. It was work on this cipher that eventually led to Elizebeth and William’s future careers in codebreaking.

https://www.goodreads.com/en/book/show/32025298

https://www.cabinetmagazine.org/issues/40/sherman.php

https://sirbacon.org/bacon-forum/index.php?/topic/107-friedman-cipher-study/

Consider William’s involvement in plant genetics and codebreaking later in this post in relation to the Matter CBD sticker and the push to blockchain cannabis genetics…

https://www.cabinetmagazine.org/issues/40/sherman.php

https://medicinalgenomics.com/blockchained-cannabis-dna/#:~:text=Cannabis%20breeders%20and%20cultivators%20can,file%20for%20their%20own%20patent.

William went on to found the NSA. As a team, he and Elizebeth solved complex puzzles – uniting as an extended complementary mind. That, I think, is what is intended with AR LARPy civic gaming. Maybe we can get out ahead of Niantic, Nvidia, the CIA, the Vatican…and untangle some interdimensional knots on our own?

I invite you to to consider my oversharing in that context. It’s an invitation into a collective puzzle-solving game. The one trick is we don’t have the box top image, and we don’t know what we are supposed to be making with all of these seemingly random loose parts.

Anyway, I’m going to make these notes for myself at the very least. I know looking back months or years later with more context, makes all the difference.

https://en.m.wikipedia.org/wiki/Qualia

Qualia – it’s a topic Stephers has been digging into – personalization, biophysics, digital ledgers, umwelt, creating a big picture – the BIGGEST picture. I hope to talk more about that after I get settled.

https://www.conducttr.com

Settlement was yesterday. I guess shouldn’t have been surprised that the office where we were to sign the documents to finalize the sale of the family home was in the Penn Mutual Life Insurance Building, founded in 1847. Think of the probability and stochastic modeling upon which the industry was built, as well as its ties to web3 through parametric deals that run on ubiquitous sensor networks.

The massive complex takes up an entire block and sits on the south side of Walnut (tree, choice theory) opposite Independence Hall (sovereign agents). It was in a Berkshire Hathaway Real Estate office, evidently they are under The Trident Group. As the notary quipped, “Warren Buffett owns us all.”

A close friend of my husband’s grandfather, Phil Carret, was a role model for Buffett. Carret, then in his 90s, came to our wedding. He helped create the world of mutual funds and was a solar eclipse chaser all his life. I continue to think automated financial markets, linked to collectivity and mutualism, are part of the superorganism emergence trajectory. Gates, Buffett, and their “Giving Pledge” is, in my opinion less about philanthropy, and more about using open source software, sensors, web3, and social impact data, to catalyze new forms of informational “life.”

https://youtu.be/KOqr17GNTJ8?feature=shared

A section of the Penn Mutual complex incorporates the Egyptian Revival facade of another nineteenth-century insurance company. Remember insurance was the brain child of inventor, publisher, electricity explorer, diplomat Ben Franklin, a supreme consciousness engineer.

The facade of the Pennsylvania Fire Insurance Company features palm columns and lintels of winged suns, both linked to longevity and the soul.

Out in front is a random sign for “The Bagel Factory,” but there was no retail store anywhere on the block that I could see. It reminded me of the symbolic everything bagel in the Academy Award winning film “Everything Everywhere All At Once.”

https://www.sportskeeda.com/comics/everything-everywhere-once-what-the-bagel-actually-represent#:~:text=The%20Everything%20Bagel%20becomes%20more,be%20felt%20throughout%20the%20universe.

Nope, we can’t control everything.

After it was all over, my not-quite ex husband, and I parted ways surrounded by the bronze reliefs that comprised a massive narrative portal. An auspicious place, I suppose, to step forward into what comes next.

I couldn’t not cross the street to take one last look at the iconic symbol of freeDOM with its clock and its bell giving form to how we perceive “reality.” After taking the photo below, I turned to walk back to the car when I spied a piece of Bitcoin sticker art stuck to a pseudo-colonial lamp post, above it (upside down) was another sticker for Matter CBD, an organic cannabis outfit based southern Oregon where the growers, Cleve Backster-style, sing and talk to their plants.

https://legacy.iftf.org/fileadmin/user_upload/downloads/blockchain/IFTF_BlockchainFutures_Map.pdf

https://legacy.iftf.org/fileadmin/user_upload/downloads/blockchain/IFTF_BlockchainFutures_Map.pdf

Upside down matter in an era of prescribed plant medicine for altered consciousness where tokens masquerade as money, but are perhaps actually cells of some as yet indescribable life form jumping off bibliometry (Eugene Garfield) and maybe geoengineering? How to account for this sticker – a slightly blurry Bitcoin stencil on a scan of an old library check-out card (also upside down with multiple dates from the height of altered state / human potential 60s and 70s) with a jet plane flying below it all? Folks who know more about numerology, do the featured dates have any significance to you?

I kept walking, on past the American Philosophical Society, an off-shoot of the Royal Society laying out what is proper/acceptable science, while keeping esoteric objects like Kelpius’s Horologium of Achaz away from prying eyes. There you have good old Ben in the alcove, Philadelphia’s business, civic, and intellectual patriarch.

https://www.jstor.org/stable/pdf/983156.pdf?refreqid=fastly-default%3Acb0b6221286adcbe5ea9d638716a01e8&ab_segments=&origin=&initiator=&acceptTC=1

And in the next block, flanking Independence Hall to the east is the Bourse, once a commodities exchange, now an upscale tourist food court. The xoxo sculpture out front reminds me of Galia’s digital heart tokens – part of “the city that loves you back” PR campaign I suppose. Affection as tradable commodity. Boy, I got shorted this round for sure. As if to further sour my entanglement with the city, I came back to a parking ticket on my windshield. I’ve maybe gotten one other in my thirty plus years here, and I still had a half hour paid up. Stupid “smart” parking tech. I guess their sham will be the final dues I pay for my “freeDOM.”

After packing the last of my things into the car, and giving my honey locust street tree one more big hug, I headed to I-95 south. There’s part of me that hopes the children who will grow up in its lacy shade will decorate the sidewalk with chalk art as my child once did. For a time, 24th Street was a wonderful place to be.

I ended up pulling off at the University City exit to say good-bye to Bartram’s Garden where I worked for seventeen years until the lockdowns started stealing things from my life. I wanted to get a jar of local honey to sweeten the parting and see my favorite trees. The tide was in, so no mudlarking this time around. I gathered a bit of compost from inside the ancient yellow wood and rubbed it into my hands, I found a walnut with a hole that looks like a twinned portal and put it in my backpack as a keepsake. When I found my favorite, a majestic London Plane, there was the trickster energy, again. It was hosting a piece of outdoor art featuring a fox with a quizzical expression. Well played Philadelphia, well played.

For the past two nights I have been camping at Bear Den off the Appalachian Trail outside Blumont, VA. The site was the former summer estate of a wealthy DC OB-GYN and his opera singer wife. It sits on veins of milky white quartz. A few miles south is Weather Mountain, a Cold War bunker for the federal government. According to a local history write up in the lodge, Mahlon Loomis, a rather visionary nineteenth century Boston dentist conducted experiments transmitting wireless communications between two hills nearby, decades before Marconi, using gilded kites!

It’s been cold for May, and rainy. But my tent stayed dry and between the showers I explored the grounds and went out to the rocky overlook and gathered up plants to make a heart. I spied a fire pit with lots of charcoal. So, I decided to make one with that in honor of my friend Sean and his family who use biochar to feed the garden beds on their small farm in Washington State.

The lodge’s volunteer caretaker found it and was excitedly telling me about it. She said she took photos to send to her boyfriend, and was pointing it out to the incoming hikers, including a couple who said they’d recently relocated to Harpers Ferry from Washington State, which closes the circle. This heart has rose petals, wood sorrel, bramble blossoms, sassafras, puffballs, hemlock cones, fern, and quartz.

I need to pack up my tent and head a bit farther down the road to Bayse. I hear conversations of families across the clearing hauling dripping blankets and debating the merits of camping after a wet, cold night. I remember those days. I’m not sure we ever had a camping trip with good weather. Now I am a solo traveler with my stuffed bear and my house plants. I will see what it feels like to be on my own reconnaissance, finding my own grounding and my own views.


Ben Werdmüller

Monetizing ShareOpenly

I was asked if I’m planning to monetize ShareOpenly. Short answer: I have no plans to do so. This is a personal project. If it’s wildly successful and the infrastructure costs skyrocket, I may look for donations or sponsorship of some kind in order to cover those costs. I’m not looking for it to be profitable or for it to be my job. It’s intentionally very very lightweight, so I don’t expect

I was asked if I’m planning to monetize ShareOpenly.

Short answer: I have no plans to do so. This is a personal project.

If it’s wildly successful and the infrastructure costs skyrocket, I may look for donations or sponsorship of some kind in order to cover those costs. I’m not looking for it to be profitable or for it to be my job.

It’s intentionally very very lightweight, so I don’t expect that to happen for a long time to come.

Wednesday, 08. May 2024

Heres Tom with the Weather

60+ Journalism Profs Demand Investigation into Controversial NYT Article

As the “paper of record” this NYT article was extremely influential in shaping public opinion. 60+ Journalism Profs Demand Investigation into Controversial NYT Article Alleging Mass Rape on Oct. 7

As the “paper of record” this NYT article was extremely influential in shaping public opinion.

60+ Journalism Profs Demand Investigation into Controversial NYT Article Alleging Mass Rape on Oct. 7


Wrench in the Gears

Creative Placemaking As Cryptographic Cultural Computation

I’ll be leaving Philadelphia tomorrow after the house closing paperwork is signed. The cleaners are getting it ready for its new family. A chapter closes, a new one opens. I’ll slowly make my way down to Little Rock, camping along the way. After three back-to-back, three-day, 1,300-mile u-haul drives in as many weeks, I’m sick [...]

I’ll be leaving Philadelphia tomorrow after the house closing paperwork is signed. The cleaners are getting it ready for its new family. A chapter closes, a new one opens. I’ll slowly make my way down to Little Rock, camping along the way. After three back-to-back, three-day, 1,300-mile u-haul drives in as many weeks, I’m sick of bland hotels with oddly coded decor and ready to sleep under the stars without worrying if someone broke into the truck while I was sleeping.

I built in a few extra days to poke around Oak Ridge outside Knoxville and see the colossal Athena in Nashville’s 1931 Parthenon replica with the snakey Erichthonius (born of Hephasteus’s semen falling to the soul). Interesting allusions to biogeochemistry and the origins of life in a city growing by leaps and bound on economies of biotech and blockchain health systems.

Yesterday I hit REI and got some new Tevas a year after my 20-year old pair gave up the ghost. I need shower shoes for the campgrounds, and Arkansas lakes are clear with sharp rocky bottoms. I also grabbed a tiny USB-rechargeable lantern. It’s been raining a lot. If I get stuck in the tent, I have “Godel, Escher, Bach” to keep me busy – all strange loops, and recursion, self-referencing refrains that call to mind for me reincarnation or many worlds depending on your ideas about spacetime. Using digital light to process print information seems par for the course.

I’d hoped to be able to tour the Barnes Foundation today, but it was closed to the general public. In reading how Godel assigned symbols to numbers to generate a coded language that generated insights into the limitations of mathematical proofs, I started to think more about Steganography, secretly encoding objects with alternate meanings. It is a form of information exchange that can exist out in the open, but only able to be translated by those who know the coding. Stephers brought it to my attention a few years back.

The Barnes Foundation has an unusual, and highly politicized history. Albert Barnes, a Philadelphian, got his MD at the University of Pennsylvania, then trained as a corporate chemist in Heidelberg, Germany, and later made a fortune selling, Argyrol, silver nitrate drops used in infants eyes after delivery.

He used his fortune to acquire a vast, eclectic collection of Impressionist, Post-Impressionist, and Modern art. These pieces are displayed in methodical arrangements, or ensembles, that incorporate pieces of decorative arts and furniture as well as ritual objects from a range of world cultures. Barnes created sets, and the interplay of the objects in each set generates additional meaning, context, through juxtaposition.

Barnes focused on the application of scientific principles to art, and established education programs around the collection. His wife Laura managed an arboretum on the property, also focused on continuing adult education. They considered the estate in Lower Merion to be a laboratory.

Albert, who had grown up poor, was often at odds with Philadelphia’s social elite. When he died, management of the collection and its programs was entrusted to a board run by Lincoln University, a historically Black college in Chester County. For decades there was rancor with the Barnes neighbors in Merion, and the elite wanted to break the trust, push out the Lincoln trustees and bring Barnes’s art lab, in a rather more neutered format to the Ben Franklin Parkway, which is where I’m now sitting killing time under the fancy blooming buckeyes until my car is ready.

The “new” Barnes on the Ben Franklin Parkway

Now I am factoring in artifacts and Godel’s numbering system that to me has digital ID overtones, as well as agents, civic gaming, tokens, signals, and computation within layers of meaning where some (maybe many) interconnected information fields are obscured or subject to the Pitt/Ober Athenian “knowledge management” protocols.

What is AR crypto “play to earn,” really? If I listen to my gut, I would say it is about massive parallel post-Moore’s law information processing through networked biophysics and social physics and dynamic game mechanics linked to the sensors and actuators of spatial computing.

Are we meant to be Godel Numbers, dancing, harmonizing, particles in a cosmic conversation only our subconscious can access?

Maybe notes in a musical performance? Orchestral music? Jazz? Sacred? Profane? Remember, Tik Tok started out as ByteDance, and a Conductrr crisis simulation modelling system that promotes the use of xAPI to build fluid reality has been running in the background gor a decade or more. I was intrigued to read that Ada Lovelace imagined Babbage’s computing machine might be used to realize complex scientific forms of music. Was it music of the spheres, cymatics, manifestation?

If Galia of Bancor seeks to harness the power of mothers’ hearts through tokens, what does it mean that Joseph Lubin, founder of Consensys and a lead builder of smart contract worlds got his start in Princeton in robotics and automated music?

Jamie Wheal’s book “Stealing Fire” has a whole chapter on the tech behind the electronic dance music scene as a catalyst for ekstasis. Is the point to use token engineering combined with nano and frequency to network human embodied intelligence into a global composition so that we collectively unlock access to subconscious fields of information and serve as some kind of vast liquid crystal manifesting system that uses encoded cultural artifacts as a kind of creative language? These are all crazy possibilities I am mulling over.

I just can’t help but wonder about Barnes and Heidelberg and baby eyes and arranged artifacts and encoded (perhaps interdimensional?) communication and botany and arborization and many worlds and the Montecarlo method and what jokes this gritty, esoteric city is playing on me…

Two imposing sculptures flank the median of the Parkway, The Soldiers and Sailors Civil War Monument, erected by the city during World War I. I can’t help but think of the intergenerational polarity generated by the Civil War, still reverberating through our shared consciousness as the culture wars, now with memes, hashtags, social graphs, and digital stigmergy. When I read the statements about freeing slaves and freedom under a unified Constitution, one destiny, I picture a future with Nicole Shanahan’s equitable smart contract laws, logic melded culture that has been codified, standardized, and made accessible to a global outside-in robot that has no soul.

“In Giving Freedom To The Slave, We Assure Freedom To The Free”

I think of sovereign agents in a never-ending series of loosely programmed scenarios, degrees of freedom set in advance, where AI life coaches log choices made on permanent ledgers, and assess our relative worth in the current fitness landscape. Not slaves, but “free” agents responding to a digital constitution reimagined as a dynamic, responsive gameboard, cells in the collective Athenian smart-city superorganism.

“One Country, One Constitution, One Destiny”

As I mull all of this over – my city speaking to me in steganography – you are an agent who is failing to advance the noetic biohybrid computer. Therefore you are a cancer to be shunned, excluded, expelled. That’s just how the program works. I see it, and my bags are packed. Rafael says the fluids are all topped off and my old dumb Subaru is ready to head South into unknown territory.

I made some site visits over the last few days, left a few hearts behind – shocker, right? The first was under an ancient oak behind Jeffrey Yass’s Susquehanna International Group headquarters in Bala Cynwyd on the Main Line. I think the fact that it’s labelled “revolutionary” is significant.

I used pollen and fallen oak leaves with pine cones and dandelions – so cheerful and humble. In the center I placed a rose nicked from the parking lot of my Jungian analyst whom I’m sad to leave. Even though he didn’t follow a lot of what I said, he valued my authenticity and my journey, as odd as it is turning out to be. I told him the heart would be a tribute to entelechy, as represented by the acorn – the tiny spark of powerful potential that is our soul’s quest. I had to look really hard to find three tiny acorns, but I did and placed them on some mugwort (good for dreams).

I hugged the massive girth of the oak, a symbol of arborization, anima mundi, branching choices, montecarlo, and asked for guidance in what comes next – to be of service with the time I have left.

The sculpture below is right out in front of the building. It is white metal and consists of three overlapping circles topped by linear waves. To me this connected to Hofstadter’s discussion of recursion, cannon, and fugues. Remember SIG was the esrly lead investor in Tik Tok, pendulum, time, entrainment.

Yesterday I visited the Kelpius cave, and to my surprise it seems like some Pythagoreans had gotten there first. When I arrived there were sizeable sticks, along with rocks and rubble, arranged in a large triangle.

Inside was a base and from it extended several smaller sticks that created a simple “tree” with three branches. There was a symbol with paired triangles, like the Star of David etched in the damp earth near the doorway. Scattered around seemed to be broken remnants of a popsicle stick project, which to me symbolized configurations of crystal lattice.

Upon that I placed my heart of purple Pawlonia blossoms, dandelions, fern, may apple, pine cones, and tulip poplar petals. Scattered around the edge were Star of Bethlehem flowers.

When I finished I chatted with a white-haired gentleman, a birder named Paul (which factors into my cathedral visit) who’d been watching migrating warblers around the cave. He told me he had a friend who he used to walk with who had been sort of an informal caretaker of the cave. I pointed out the fading Nephele graffiti and showed him the heart and branched sticks, and then went into avianmagnetoreception as a study case for quantum biology and how that tied into US Navy funded anthropology research into Micronesian open water navigation. He told me a bit about local garnets, and I shared some mica flakes I found. It was a gentle way to close my time in the Wissahickon.

So then on my drive to take my car to the mechanic, I stopped at a light on Broad Street at MARCONI Plaza and in the median was a woman in a sun hat digging dandelions!

Then heading home I walked through Love Park, noticing it was now sponsored by Bank of America. There were some guys with big cameras set up to take pictures of people taking pictures posing in front of the sculpture. I wondered if they were working on a documentary, so I tried to chat them up about gap junctions, group mind, the energy of the heart torus field, and Lev “Heart” being the name of Galia Benarzi’s first digital community currency – structured around an Israeli mothers’ babysitting co-op. They were not very interested.

The next person to not be interested was the gal staffing the open budget pop-up cargo container civic education junket sponsored in part by Mural Arts. I tried to explain web3 token voting linked to AI twins and social impact betting. Then a guy showed up with flyers about advice on home ownership and career services, and I had to point out that when Philadelphia’s City Council had anti-poverty hearings five years ago, the region’s job growth report said the average wage was going to be $15 an hour. You can’t buy a house in Philadelphia with that income. I pointed to the office buildings around us and said – those will be redone as affordable rental housing, only it will be like a dorm and your bedroom will have no windows, and you will pay your rent with UBI and data tied to sustainability and wellness behaviors. If you want sunlight the common area by the windows will be full of AI vision cameras and sensors like at WeWork (now Flow) and if you play the game well you can earn a small bit of equity in the ant computer. This will be set up by strange bedfellows partnerships between private equity (Blackstone/Ancestry.com DNA), community banks, high net individuals, and religious groups.

They barely registered what I was saying. In the end I told her to look up Cesar Hidslgo’s TED talk on radical participatory voting. They are training voters for the old game, just as they are about to be taken for a ride by the new web3 tokenomics game.

And why is Mural Arts behind this? Think cultural artifacts, symbols, civic gaming. This is a useful paper on social impact finance linked to public art. The city remade as a digital museum for collective manipulation and “creative placemaking.”

https://www.sciencedirect.com/science/article/abs/pii/S0304422X18303747

My final stop was at the Catholic Cathedral of Saints Peter and Paul overlooking Logan Circle. I hadn’t actually been inside before, but yesterday was the day. I entered the foyer with plaques commemorating masses given by Pope Francis and Pope John Paul II, then I went inside. What struck me was all the different ways sacred light was communicated, the many references to paired keys (encryption), and the book and the sword (Paul) that evoked for me Philip Pullman’s subtle knife that can cut through dimensions. or perhaps information fields as represented by the good book.

There was a huge painting of the Magi under a beaming star of Bethlehem. I’d used Star of Bethlehem blossoms in my Kelpius heart.

The final synchronicity was the strange appearance of a new white ball cap with a Northeastern University Huskies logo on it. I’d been going back and forth the day before, texting with a friend about the oversize role it seemed the Boston university was playing in the AI, biotech, ed-tech transformation of Maine under the guidance of former Jackson Labs spokesperson David Roux. And now here was a hat for that very school, the first thing I saw upon entering the sanctuary.

So my last day in the city of brotherly love is winding down. I read this passage on the subway today – how to step outside the system.

I don’t think it’s truly possible to exit, but for now I’m unplugging from Philly, a city that taught me much and broke my heart. With time I hope it will heal, and I can become a wise old crone.


Patrick Breyer

Gutachten zum Computerspiele-Sterben: Piraten wollen klare EU-Regeln gegen willkürliches Lahmlegen von Spielen

Vor dem Hintergrund der Spielerinitiative „Stop killing games“, die sich gegen das Lahmlegen von Computerspielen durch ihre Hersteller wendet, hat der Wissenschaftliche Dienst des Europäischen Parlaments …

Vor dem Hintergrund der Spielerinitiative „Stop killing games“, die sich gegen das Lahmlegen von Computerspielen durch ihre Hersteller wendet, hat der Wissenschaftliche Dienst des Europäischen Parlaments im Auftrag der Europaabgeordneten der Piratenpartei Dr. Patrick Breyer die einschlägigen EU-Gesetze untersucht und sieben potenziell relevante Regelungen aufgeführt. Breyer sieht Lücken im Rechtsrahmen:

„Es fehlen klare EU-Verbraucherschutzregelungen gegen die branchenübliche Praxis, dass Spielehersteller beliebte Spiele gewinnbringend verkaufen, sich dabei aber die willkürliche, jederzeitige Unbrauchbarmachung vorbehalten und nicht einmal eine Rückerstattung des Kaufpreises anbieten. Es gibt zwar eine allgemeine EU-Richtlinie gegen ‚missbräuchliche Klauseln in Verbraucherverträgen‘, aber deren Auslegung muss langwierig von Gerichten entschieden werden. Es gibt außerdem eine EU-Richtlinie, derzufolge digitale Inhalte während der Vertragsdauer bereit gestellt werden müssen, aber gegen die kurzfristige Kündigung und Einstellung eines gerade erst gekauften Spiels schützt sie nicht. Die EU-Regeln werden der kulturellen Bedeutung von Games nicht gerecht, die gewachsene Gemeinschaften von Millionen von Spielern zusammen bringen können. Wir Piraten fordern, dass die EU Games als Kulturgut schützt, ein Lahmlegen nach Gutdünken verbietet und, wenn der Hersteller Games aufgibt, ein Weiterführen durch die Community ermöglicht.“

Tuesday, 07. May 2024

Just a Theory

Mini Summit Five

Links, notes, and commentary from Yurri Rashkovskii’s presentation, “Universally Buildable Extensions: Dev to Prod,” at the fifth Postgres Extension Ecosystem Mini-Summit on May 1, 2024.

The video for Yurri Rashkovskii’s presentation at the fifth Postgres Extension Ecosystem Mini-Summit last week is up. Links:

Video PDF Slides

Here’s my interpolation of YouTube’s auto-generated transcript, interspersed with chat activity.

Introduction I opened the meeting and introduced Omnigres’s Yurri Rashkovskii. Presentation

Yurri: Today I’m going to be talking about universally buildable extensions. This is going to be a shorter presentation, but the point of it is to create some ideas, perhaps some takeaways, and actually provoke a conversation during the call. It would be really amazing to explore what others think, so without further ado…

I’m with Omnigres, where we’re building a lot of extensions. Often they push the envelope of what extensions are supposed to do. For example, one of our first extensions is an HTTP server that embeds a web server inside of Postgres. We had to do a lot of unconventional things. We have other extensions uniquely positioned to work both on developer machines and production machines — because we serve the the developers and devops market.

The point of Omnigres is turning Postgres into an application runtime — or an application server — so we really care how extensions get adopted. When we think about application developers, they need to be able to use extensions while they’re developing, not just in production or on some remote server. They need extensions to work on their machine.

The thing is, not everybody is using Linux Other people use macOS and Windows and we have to account for that. There are many interesting problems associated with things like dependencies.

So there’s a very common approach used by those who who try to orchestrate such setups and by some package managers: operating out of container. The idea is that with a can create a stable environment where you bring all the dependencies that your extension would need, and you don’t have to deal with the physical reality of the host machine. Whether it’s a developer machine, CI machine, production machine, you always have the same environment. That’s definitely a very nice property.

However, there are some interesting concerns that we have to be aware when we operate out of a container. One is specifically mapping resources. When you have a container you have to map how many cores are going there, memory, how do we map our volumes (especially on Docker Desktop), how we connect networking, how we pass environment variables.

That means whenever you’re running your application — especially locally, especially in development — you’re always interacting with that environment and you have to set it up. This is particularly problematic with Docker Desktop on macOS and Windows because these are not the same machines. You’re operating out of a virtual machine machine instead of your host machine, and obviously containers are Linux-specific, so it’s always Linux.

What we found is that often times it really makes a lot of sense to test extensions, especially those written in C, on multiple platforms. Because in certain cases bugs, especially critical memory-related bugs, don’t show up on one platform but show up on another. That’s a good way to catch pretty severe bugs.

There are also other interesting, more rare concerns. For example, you cannot access the host GPU through Docker Desktop on macOS or through Colima. If you’re building something that could have use the host GPU that would work on that machine it’s just not accessible. If you’re working something ML-related, that can be an impediment

This also makes me wonder: what are other reasons why we’re using containers. One reason that struck out very prominently was that Postgres always has paths embedded during compile time. That makes it very difficult to ship extensions universally across different installations, different distributions. I wonder if that is one of the bigger reasons why we want to ship Postgres as a Docker container: so that we always have the same path regardless of where where it’s running.

Any questions so far about Docker containers? Also if there’s anybody who is operating a Docker container setup — especially in their development environment — if you have any thoughts, anything to share: what are the primary reasons for you to use a Docker container in your development environment?

Jeremy S in chat: When you say it’s important to test on multiple platforms, do you mean in containers on multiple platforms, or directly on them?

Jeremy S in chat: That is - I’m curious if you’ve found issues, for example, with a container on Mac/windows that you wouldn’t have found with just container on linux

Daniele: Probably similarity with the production deployment environments. That’s one. Being free from whatever is installed on your laptop, because maybe I don’t feel like upgrading the system Python version and potentially breaking the entire Ubuntu, whereas in a Docker container you can have whatever version of Python, whatever version of NodeJS or whatever other invasive type of service. I guess these are these are good reasons. These were the motivation that brought me to start developing directly in Docker instead of using the desktop.

Yurri: Especially when you go all the way to to production, do you find container isolation useful to you?

Daniele: Yeah I would say so; I think the problem is more to break isolation when you’re are developing. So just use your editor on your desktop, reload the code, and have a direct feedback in the container. So I guess you have to break one barrier or two to get there. At least from the privilege points of having a Linux on desktop there is a smoother path, because it’s not so radically different being in the container. Maybe for Windows and macOS developers it would be a different experience

Yurri: Yeah, I actually wanted to drill down a little bit on this In my experience, I build a lot on macOS where you have to break through the isolation layers with the container itself and obviously the VM. I’ve found there are often subtle problems that make the experience way less straightforward.

One example I found it that, in certain cases, you’re trying to map a certain port into the container and you already have something running [on that port] on your host machine. Depending on how you map the port — whether you specify or don’t specify the address to bind on — you might not get Docker to complain that this port is actually overridden.

So it can be very frustrating to find the port, I’m trying to connect to it but it’s not connecting to to the right port. There’s just very small intricate details like this, and sometimes I’ve experienced problems like files not perfectly synchronizing into the VM — although that has gotten a little better in the past 2–3 years — but there there were definitely some issues. That’s particularly important for the workflows that we’re doing at Omnigres, where you’re running this entire system — not just the database but your back end. To be able to connect to what’s running inside of the container is paramount to the experience.

Daniele: Can I ask a question about the setup you describe? When you go towards production, are those containers designed to be orchestrated by Kubernetes? Or is there a different environments where you have your Docker containers in a local network, I assume, so different Dockers microservices talking to each other. Are you agnostic from what you run in it, or do you run it on Kubernetes or on Docker Compose or some other form of glue that you you set up yourself, or your company has set up?

Steven Miller in chat: … container on Mac/windows [versus linux] Steven Miller in chat: Have seen with chip specific optimizations like avx512

Yurri: Some of our users are using Docker Compose to run everything together. However, I personally don’t use Docker containers. This is part of the reason why the topic of this presentation is about universally buildable extensions. I try to make sure that all the extensions are easily compilable and easily distributable on any given supported platform. But users do use Docker Compose, it’s quite common.

Does anyone else here have a preference for how to move Docker containers into production or a CI environment?

Nobody? I’ll move on then.

Steven Miller in chat: Since in docker will run under emulation, but on linux will run with real hardware, so the environment has different instruction set support even though the docker —platform config is the same

Jeremy S in chat: That makes sense

Yurri: I wanted to show just a little bit of a proof of concept tool that we’ve been working on, on and off for the last year—

David Wheeler (he/him): Yurri, there are a couple comments and questions in chat, I don’t know if saw that

Yurri: I didn’t see that sorry.

Jeremy is saying, “when you say it’s important to test on multiple platforms do you mean in containers on multiple platforms or directly on them?” In that particular instance I meant on multiple platforms, directly.

The other message from Jeremy was, “I’m curious if you found issues for example with a container on Mac or Windows that you wouldn’t have found with just container on Linux?” Yeah I did see some issues depending on the type of memory-related bug. Depending on the system allocator, I was either hitting a problem or not. I was not hitting it on Linux, I believe and it was hidden macOS. I don’t remember the details right now, unfortunately, but that difference was indicative of a bug.

Steven wrote, trying to connect this… “Have * seen chip-specific optimizations for containers?” And, “Docker will run under emulation but on Linux will run with real Hardware.” Yeah that’s an interesting one about ax512. I suppose this relates to the commentary about about GPU support, but this is obviously the other part of supporting specific hardware, chip-specific optimizations That’s an interesting thing to learn; I was not aware of that! Thank you Steven.

Let’s move on. postgres.pm is a pro of concept that I was working on for some time. The idea behind it was both ambitious but also kind of simple: Can we try describing Postgres extensions in such a way that they will be almost magically built on any supported platform?

The idea was to build an expert system of how to build things from a higher level definition. Here’s an example for pgvector:

:- package(vector(Version), imports([git_tagged_revision_package(Version)])). git_repo("https://github.com/pgvector/pgvector"). :- end_package.

It’s really tiny! There are only two important things there: the Git tagged revision package and Git repo. There’s nothing else to describe the package.

The way this works is by inferring as much information as possible from what’s available. Because it’s specified as a Git-tagged revision package, it knows that it can download the list of version-shaped revisions — the versions — and it can checkout the code and do further inferences. It infers metadata from META.json if it’s available, so it will know the name of the package, the description, authors, license, and everything else included there.

David G. Johnston in chat: PG itself has install-check to verify that an installed instance is functioning. What are the conventions/methods that extension authors are using so that a deployed container can be tested at a low level of operation for the installed extensions prior to releasing the image to production?

It automatically infers the build system. For example for C extensions, if it sees that there’s a Makefile and C files, it infers that you need make and a C compiler and it tries to find those on the system: it will try to find cc, gcc, Clang — basically all kinds of things.

*David Wheeler (he/him)() in chat: Feel free to raise hands with questions

Here’s a slightly more involved example for pg_curl. Ah, there was a question from David Johnson. David says, “PG has install-check to verify that installed instance is functioning. What are the conventions methods that extension authors are using so the deployed container can be tested at a low level of operation for the installed extension prior to releasing the image to production?”

I guess the question is about general conventions for how extension authors ensure that the extensions work, but I suppose maybe part of this question is whether that’s also testable in a production environment. David, are you talking about the development environment alone or both?

David G. Johnston: Basically, the pre-release to production. You go in there in development and you cut up an extension and source and then you build your image where you compile it — you compile PG, you compile it, or you deploy packages. But now you have an image, but you’ve never actually tested that image. I can run installcheck on an installed instance of Postgres and know that it’s functioning, but it won’t test my extension. So if I install PostGIS, how do I test that it has been properly installed into my database prior to releasing that image into production?

Tobias Bussmann in chat: shouldn’t have the extension a make installcheck as well?

Yurri: To my knowledge there’s no absolutely universal method. Of course the PGXS methods are the most standard ones — like installcheck — to to run the tests. In our [Omnigres’s] case, we replaced pg_regress with pg_yregress, another tool that we’ve developed. It allows for more structural tests and tests that test certain things that pg_regress cannot test because of the way it operates.

David Wheeler (he/him) in chat: https://docs.omnigres.org/pg_yregress/intro/

I can share more about this later if that’s of interest to anybody. So we basically always run pg_yregress on our extensions; it creates a new instance of Postgres — unless told to use a pre-existing instance — and it runs all the tests there as a client. It basically deploys the the extension and runs the set of tests on it.

David G. Johnston: Okay.

Yurri: I guess you know it depends on how you ship it. For example, if you look at the pgrx camp, they have their own tooling for that, as well. I’ve also seen open-source extensions where they could be written in, say, Rust, but still using pg_regress tests to test their behavior. That would often depend on how their build system is integrated in those tests. I guess the really short answer is there’s probably no absolutely Universal method.

David thank you for pasting the link to pg_yregress. If there are ny questions about it, feel free to ask me. Any other thoughts or questions before I finish this slide? Alright will carry on then.

:- package(pg_curl(Version), imports(git_explicit_revision_package(Version))). :- inherit(requires/1). git_repo("https://github.com/RekGRpth/pg_curl"). git_revisions([ '502217c': '2.1.1', % ... older versions omitted for now ... ]). requires(when(D := external_dependency(libcurl), version::match(D, '^7'))). :- end_package.

The difference between this example and the previous one is that here it specifies that there will be an explicit revision map because that project does not happen to have version tags, so they have to be done manually. You can see that in the Git revision specification. But what’s more interesting about this is that it specifies what kind of dependency it needs. In this particular instance it’s libcurl, and the version has to match version 7 — any version 7.

These kinds of requirements, as well as compiler dependencies, make dependencies, and others are always solved by pluggable satisfiers. They look at what’s available depending on the platform — Linux, a particular flavor of Linux, macOS, etc — and picks the right tools to see what’s available. In the future there’s a plan to add features like building these dependencies automatically, but right now it depends on the host system, but in a multi-platform way.

David Wheeler (he/him) in chat: How does it detect that libcurl is required?

The general idea behind this proof of concept is that we want to specify high level requirements and not how exactly to satisfy them. If you compare this to a Docker file, the Docker file generally tells you exactly what to do step by step: let’s install this package and that package, let copy files, etc. so it becomes a very specific set of instructions.

Jeremy S in chat: And how does it handle something with different names in different places?

There was a question: “how does it detect that libcurl is required?” There there is this line at the bottom says “requires external dependency libcurl, so that was the definition.”

The other question was “how does it handle something with different names in different places?” I’m not sure I understand this question.

Jeremy S: I can be more spe specific. A dependency like libc is called libc on Debian platforms and it’s called glibc on Enterprise Linux. You talked about available satisfiers like Homebrew, Apt and package config, but what if it has a different name in Homebrew than in Apt or something like? Does it handle that or is that just something you haven’t tackled yet?

Yurri: It doesn’t tackle this right now, but it’s part of the division where it should go. For certain known libraries there’s an easy way to add a mapping that will kick in for a distribution, and otherwise it will be a satisfier for another one. They’re completely pluggable, small satisfiers looking at all the predicates that describe the system underneath.

David G. Johnston in chat: How is the upcoming move to meson in core influencing or impacting this?

Just for point of reference, this is built on top of Prolog, so it’s like a knowledge base and rules for how to apply on this knowledge to particular requirements.

Tobias Bussmann in chat: Prolog 👍

Shaun Thomas in chat: What if there are no satisfiers for the install? If something isn’t in your distro’s repo, how do you know where to find the dependency? And how is precedence handled? If two satisfiers will fulfill a requirement, will the highest version win?

Jeremy S: I remember Devrim talking about, if you read through the [RPM] spec files, what find is all this spaghetti code with #ifdefs and logic branches and in his case is just dealing with differences between Redhat and SUSE. If this is something that we manually put in, we kind of end up in a similar position where it’s on us to create those mappings, it’s on us to maintain those mappings over time — we kind of own it — versus being able to automate some kind of automatic resolution. I don’t know if there is a good automatic way to do it. David had found something that he posted, which I looked at a little bit, but Devrim talked about how much of maintenance overhead it becomes in the long run to constantly have to maintain this which seemed less than ideal.

Yurri: It is less than ideal. For now, I do think that would have to be manual, which is less than ideal. But it could be addressed at least on on a case-by-case basis. Because we don’t really have thousands of extensions yet — in the ecosystem maybe a thousand total — I think David Wheeler would would know best from his observations, and I think he mentioned some numbers in his presentation couple of weeks ago. But basically handling this on on a case-by-case basis where we need this dependency and apparently it’s a different one on a different platform, so let’s address that. But if there can be a method that can at least get us to a certain level of unambiguous resolution automatically or semi-automatically, that would be really great.

Samay Sharma in chat: +1 on the meson question.

Jeremy S: I think there’s a few more questions in the chat.

Yurri: I’m just looking at them now. “how is the upcoming move to meson and core influencing or impacting this?” I don’t think it’s influencing this particular part in any way that I can think of right now. David, do you have thoughts how it can? I would love to learn.

David G. Johnston: No, I literally just started up a new machine yesterday and decided to build it from meson instead of make and the syntax of the meson file seems similar to this. I just curious if there are any influences there or if it’s just happenstance.

Yurri: Well from from what I can think right now, there’s just general reliance on either implicitly found PG config or explicitly specified PG config. That’s just how you discover Postgres itself. There’s no relation to how Postgres itself was built. The packaging system does not handle say building Postgres itself or providing it so it’s external to this proof of concept.

David G. Johnston: That’s a good separation of concerns, but there’s also the idea that, if core is doing something, we’re going to build extensions against PostgresSQL, if we’re doing things similar to how core is doing them, there’s less of a learning curve and less of everyone doing their own thing and you have 500 different ways of doing testing.

Yurri: That’s a good point. That’s something definitely to reflect on.

I’ll move on to the next question from Sean. “What if there are no satisfiers for the install? If something isn’t in your distro how do you know where to find the dependency?” And “if two satisfiers will fulfill a requirement, will the highest version win?” If there are no satisfiers right now it will just say it’s not solvable. So we fail to do anything. You would have to go and figure that out. It is a proof of concept, it’s not meant to be absolutely feature complete but rather an exploration of how we can describe the the packages and their requirements.

David Wheeler (he/him): I assume the idea is that, as you come upon these you would add more satisfiers.

Yurri: Right, you basically just learn. We learn about this particular need in a particular extension and develop a satisfier for it. The same applies to precedence: it’s a question of further evolution. Right now it just finds whatever is available within the specified range.

If there are no more pressing questions I’ll move to the next slide. I was just mentioning the problem of highly specific recipes versus high-level requirements. Now I want to shift attention to another topic that has been coming up in different conversations: whether to build and ship your extension against minor versions of Postgres.

Different people have different stances in this, and even package managers take different stands on it. Some say, just build against the latest major version of Postgres and others say build extensions against every single minor version. I wanted to research and see what the real answer should be: should we build against minor versions or not?

I’ve done a little bit of experimentation and my answer is “perhaps”, and maybe even “test against different minor versions.” In my exploration of version 16 (and also 15 bu Id didn’t include it) there there are multiple changes between minor versions that can potentially be dangerous. One great example is when you have a new field inserted in the middle of a structure that is available through a header file. That definitely changes the layout of the structure.

typedef struct BTScanOpaqueData { - /* these fields are set by _bt_preprocess_keys(): */ + /* all fields (except arrayStarted) are set by _bt_preprocess_keys(): */ bool qual_ok; /* false if qual can never be satisfied */ + bool arrayStarted; /* Started array keys, but have yet to "reach + * past the end" of all arrays? */ int numberOfKeys /* number of preprocessed scan keys */ }

In this particular case, for example, will not get number of keys if you’re intending to. I think that change was from 16.0 to 16.1. If you build against 16.0 and then try to run on 16.1, it might not be great.

The other concern that I found is there are new apis appearing in header files between different versions. Some of them are implemented in header files, either as macros or static and line functions. When you’re building against that particular version, you’ll get the particular implementation embedded.

Others are exports of symbols, like in this case, try index open and contain mutable functions after planning, if you’re using any of this. But this means that these symbols are not available on some minor versions and they’re available later on, or vice versa: they may theoretically disappear.

There are also changes in inline behavior. There was a change between 16.0 and 16.1 or 16.2 where an algorithm was changed. Instead of just > 0 there’s now >= 0, and that means that particular behavior will be completely different between these implementations. This is important because it’s coming from a header file, not a source file, so you’re embedding this into your extension.

David Wheeler (he/him) in chat: That looks like a bug fix

Yeah it is a bug fix. But what I’m saying is, if you build your extension against say 16.0m which did not have this bug fix, and then you deploy it on 16.1, then you still have the bug because it’s coming from the header file.

*David Wheeler (he/him): Presumably they suggest that you build from the latest minor release and that’s Backward compatible to the earlier releases.

Yurri: Right and that’s a good middle ground for this particular case. But but of course sometimes when you do a minor upgrade you have to remember that you have to rebuild your extensions against that minor version so you can just easily transfer them yeah.

Jeremy S in chat: The struct change in a minor is very interesting

*David Wheeler (he/him)Jeremy points out that struct change is pretty interesting.

Yurri: Yeah, it’s interesting because it’s super dangerous! Like if somebody is expecting a different versioned structure, then it can be pretty nasty.

Shaun Thomas in chat: Yeah. It’s a huge no-no to insert components into the middle of a struct.

Jeremy S: Is that common? I’m really surprised to see that in a minor version. On the other hand, I don’t know that Postgres makes promises about — some of this seems to come down to, when you’re coding in C and you’re coding directly against structures in Postgres, that’s really interesting. That’s — I’m surprised to see that still.

Steven Miller in chat: In the case of trunk, we would have built against minor versions in the past then upgrade the minor version of postgres without reinstalling the binary source of the extension, so this is an issue

David G. Johnston in chat: Yeah, either that isn’t a public structure and someone is violating visibility (in which case yes, you should be tracking minor builds)

Shaun Thomas in chat: I’m extremely shocked that showed up in 16.2.

Yurri: Yeah, I didn’t expect that either, because that’s just a great way to have absolutely undefined behavior. Like if somebody forgot to rebuild their extension against a new minor, then this can be pretty terrible.

But my general answer to all of this unless you’re going really deep into the guts of Postgres, unless you’re doing something very deep in terms query planning, query execution, you’re probably okay? But who knows.

Jason Petersen in chat: yeah it feels like there’s no stated ABI guarantee across minor updates

Jason Petersen in chat: other than “maybe we assume people know not to do this"

David Christensen in chat: yeah ABI break in minor versions seems nasty

Jeremy S: But it’s not just remembering to rebuild your extension. Let’s let’s suppose somebody is just downloading their extensions from the PGDG repo, because there’s a bunch of them there. They’re not compiling anything! They’re they’re downloading an RPM and the extension might be in a different RPM from Postgres and the extension RPMs — I don’t know that there have been any cases with any of the extensions in PGDG, so far, where a particular extension RPM had to have compatibility information at the level of minors.

Shaun Thomas in chat: There was actually a huge uproar about this a couple year ago because they broke the replication ABI by doing this.

David G. Johnston in chat: I see many discussions about ABI stability on -hackers so it is a goal.

Steven Miller in chat: PGDG is the same binaries for each minor version because the postgres package is only major version, right?

Yurri: Yeah, that’s definitely a concern, especially when it comes to the scenario when you rebuild your extensions but just get pre-built packages. It’s starting to leak out of the scope of this presentation, but I thought it was a very interesting topic to bring to everybody’s attention.

Jason Petersen in chat: “it’s discussed on hackers” isn’t quite the same as “there’s a COMPATIBILITY file in the repo that states a guarantee”

Jason Petersen in chat: (sorry)

My last item. Going back to how we ship extensions and why do we need complex build systems and packaging. Oftentimes you want your extensions to depend on some library, say OpenSSL or SQLite or whatever, and the default is to bring the shared dependency that would come from different packages on different systems.

What we have found at Omnigres is that it is increasingly simpler to either statically link with your dependencies — and pay the price of larger libraries — but then you have no questions about where it comes from — what what package, which version – you know exactly what which version it is and how it’s getting built. But of course you also have a problem where, if you want to change the version of the dependency it’s harder because it’s statically linked. The question is whether you should be doing that or not, depending on the authors of the extension and their promises for compatibility with particular versions of their dependencies. This one is kind of naive and simple, as in just use static. Sometimes it’s not possible or very difficult to do so, some some libraries don’t have build systems amenable to static library production.

What we found that works pretty nicely is using rpath in your dynamic libraries. You can use special variables — $ORIGIN or @loader_path on Linux or macOS, respectively, to specify that your dependency is literally in the same folder or directory where your extension is. So you can ship your extension with the dependencies alongside, and it will not try to load them immediately from your system but from the same directory. We find this pretty pretty useful.

That’s pretty much it. Just to recap I talked about the multi-platform experience, the pros and cons of containers, inferencing how you build and how you can build extensions with dependencies, static and rpath dependencies, and the problems with PG minor version differences. If anybody has thoughts, questions, or comments I think that would be a great. Thank you.

Discussion

David Wheeler (he/him): Thank you, Yurri, already some good discussion. What else do you all have?

David G. Johnston: PG doesn’t use semantic versioning. They we have a major version and a minor version. The minor versions are new releases, they do change behaviors. There are goals from the hackers to not break things to the extent possible. But they don’t guarantee that this will not change between dot-three and dot-four. When you’re releasing once a year that’s not practical if things are broken, you can’t wait nine months to fix something. Some things you need to fix them in the next update and back-patch.

Steven Miller in chat: Thank you, this is very useful info

Jeremy S in chat: Dependency management is hard 🙂 it’s been a topic here for awhile

David G. Johnston: So we don’t have a compatibility file, but we do have goals and if they get broken there’s either a reason for it or someone just missed it. From an extension standpoint, if you want to be absolutely safe but absolutely cost intensive, you want to update every minor release: compile, test, etc. Depending on what your extension is, you can trade off some of that risk for cost savings. That’s just going to be a personal call. The systems that we build should make it easy enough to do releases every “dot” and back-patching. Then the real cost is do you spend the time testing and coding against it to make sure that the stuff works. So our tool should assume releasing extensions on every minor release, not every major release, because that’s the ideal.

Shaun Thomas in chat: It’s good we’re doing all of this though. It would suck to do so much work and just become another pip spaghetti.

Yurri: That’s exactly what I wanted to bring to everybody’s attention, because there’s still a lot of conversations about this and there was not enough clarity. So that helps a lot.

Jeremy S: Did you say release or did you say build with every Miner? I think I would use the word “build”.

David G. Johnston: Every minor release, the ones that go out to the public. I mean every commit you could update your extension if you wanted. but really the ones that matter are the ones that go public. So, 16.3 or 16.4 comes out, automation would ideally would build your extension against it run your test and see if anything broke. And then deploy the new [???] of your extension against version 16.3. Plus that would be your your release.

Jeremy S: I think there are two things there: There’s rebuilding it — because you can rebuild the same version of the extension and that would pick up if they they added a field in the middle of a struct which is what happened between 16.0 and 16.1, rebuild the same version. Versus: the extension author … what would they be doing? If they they could tag a new version but they’re not actually changing any code I don’t think it is a new release of the extension, because you’re not even changing anything in the extension, you’re just running a new build. It’s just a rebuild.

David Wheeler (he/him) in chat: It’d be a new binary release of the same version. In RPM it goes from v1.0.1-1 to v1.0.1-2

It reminds me of what Alvaro did in his his OCI blog post, where he said you really have to … Many of us don’t understand how tightly coupled the extensions need to be to the database. And these C extensions that we’re we’re building have risks when we separate them don’t just build everything together.

David G. Johnston: The change there would be metadata. Version four of my extension, I know it works on 16.0 to 16.1. 16.2 broke it, so that’s where it ends and my version 4.1 is known to work on 16.2.

Jeremy S: But there is no difference between version 4 and version 4.1. There’s a difference in the build artifact that your build farm spit out, but there’s no difference in the extension, right?

Keith Fiske in chat: Still confusing if you don’t bump the release version even with only a library change

Keith Fiske in chat: How are people supposed to know what library version is running?

David G. Johnston: Right. If the extension still works, then` your metadata would just say, “not only do I work through version 16.2, I now work through 16.3.

Jeremy S: But it goes back to the question: is the version referring to a build artifact, or is the version referring to a version of the code? I typically think of versions as a user of something: a version is the thing. It would be the code of the extension. Now we’re getting all meta; I guess there are arguments to be made both ways on that.

Jason Petersen in chat: (it’s system-specific)

Jason Petersen in chat: no one talks in full version numbers, look at an actual debian apt-cache output

David Wheeler (he/him): Other questions? Anybody familiar with the rpath stuff? That seems pretty interesting to me as a potential solution for bundling all the parts of an extension in a single directory — as opposed to what we have now, where it’s scattered around four different directories.

Jason Petersen: I’ve played around with this. I think I was trying to do fault injection, but it was some dynamically loaded library at a different point on the rpath. I’m kind of familiar with the mechanics of it.

I just wanted to ask: In a bigger picture, this talks about building extensions that sort of work everywhere. But the problems being solved are just the duplication across the spec files, the Debian files, etc. You still have to build a different artifact for even the same extension on the same version of Postgres on two different versions of Ubuntu, Right? Am I missing something? It is not an extension that runs everywhere.

Yurri: No, you still have to build against the set of attributes that constitute your target, whether that’s architecture, operating system, flavor. It’s not yet something you can build and just have one binary. I would love to have that, actually! I’ve been pondering a lot about this. There’s an interesting project, not really related to plugins, but if you’ve seen A.P.E. and Cosmopolitan libc, they do portable executables. It’s a very interesting hack that allows you to run binaries on any operating system.

Jason Petersen: I expected that to be kind of “pie in the sky.”

Yurri: It’s more of a work of art.

Jason Petersen: Do you know of other prior art for the rpath? Someone on Mastodon the other day was talking about Ruby — I can’t remember the library, maybe it was ssh — and they were asking, “Do I still have to install this dynamic library?” And they said, “No, we vendor that now; whenever you install this it gets installed within the Ruby structure.” I’m not sure what they’re doing; maybe it’s just a static linking. But I was curious if you were aware of any prior art or other packaging systems where system manages its own dynamic libraries, and use rpath to override the loading of them so we don’t use the system ones and don’t have to conflict with them. Because I think that’s a really good idea! I just was wondering if there’s any sort of prior art.

Daniele: There is an example: Python Wheels binaries us rpath. A wheel is a ZIP file with the C extension and all the depending libraries the with the path modified so that they can refer to each other in the the environment where they’re bundled. There is a tool chain to obtain this packaging — this vendoring — of the system libraries. There are three, actually: one for Unix, one for macOS, one for Windows. But they all more or less achieve the same goal of having libraries where they can find each other in the same directory or in a known directory. So you could take a look at the wheel specification for Python and the implementation. That could be a guideline.

Jason Petersen: Cool.

Yurri: That’s an excellent reference, thank you.

David Wheeler (he/him): More questions?

Jeremy S: Yeah, I have one more. Yurri, the build inferencing was really interesting. A couple things stood out to me. One that you mentioned was that you look for The META.json file. That’s kind of neat, just that it’s acknowledged a useful thing; and a lot of extensions have it and we want to make use of it. I think everybody knows part of the background of this whole series of meetings is — one of the things we’re asking is, how can we improve what’s the next generation of META.json to make all of this better? Maybe I missed this, but what was your high-level takeaway from that whole experience of trying to infer the stuff that wasn’t there, or infer enough information to build something if there isn’t a META.json at all? Do you feel like it worked, that it was successful? That it was an interesting experiment but not really viable long term? How many different extensions did you try and did it work for? Once you put it together, were you ever able to point it at a brand new extension you’d never seen before and actually have it work? Or was it still where you’d try a new extension and have to add a little bit of extra logic to handle that new extension? What’s your takeaway from that experience?

Yurri: The building part is largely unrelated to META.json, that was just primarily the metadata itself. I haven’t used in a lot of extensions because I was looking for different cases — extensions that exhibit slightly different patterns — not a whole ton of them yet. I would say that, so far, this is more of a case-by-case scenario to see for a particular type of or shape of extension what we need to do. But generally, what I found so far that it works pretty nicely for C extensions: it just picks up where all the stuff is, downloads all the necessary versions, allows to discover the new versions — for example you don’t need to update the specification for a package if you have a new release, it will just automatically pick that up rom the list of tags. These these were the current findings. I think overall the direction is promising, just need to continue adjusting the results and see how much further it can be taken and how much more benefit it can bring.

Jeremy S: Thank you.

Yurri: Any other comments or thoughts?

David Wheeler (he/him): Any more questions for Yurri?

David Wheeler (he/him): I think this is a an interesting space for some research between Devrim’s presentation talking about how much effort it is to manually maintain all the extensions in the Yum repository. I’ve been doing some experiments trying to build everything from PGXN, and the success rate is much lower than I’d like. I think there are some interesting challenges to automatically figuring out how things work versus convincing authors to specify in advance.

Jeremy S: Yep. Or taking on that maintenance. Kind of like what a spec file maintainer or a Debian package maintainer is doing.

Yurri: Yeah, precisely.

Wrap Up

David Wheeler (he/him): Thanks, Yurri, for that. I wanted to remind everyone that we have our final Mini-Summit before PGConf on May 15th. That’s two weeks from today at noon Eastern or 4 pm UTC. We’re going to talk about organizing the topics for the Summit itself. I posted a long list of stuff that I’ve extracted from my own brain and lots more topics that I’ve learned in these presentations in the Slack. Please join the community Slack to participate.

The idea is to winnow down the list to a reasonable size. We already are full with about 45 attendees, and we we can maybe have a few more with standing room and some hallway track stuff. We’ll figure that out, but it’s a pretty good size, so I think we’ll be able to take on a good six or maybe eight topics. I’m going to go over them all and we’ll talk about them and try to make some decisions in advance, so when we get there we don’t have to spend the first hour figuring out what we want to, we can just dive in.

And that’s it. Thank you everybody for coming, I really appreciate. We’ll see you next time

Tobias Bussmann in chat: Thanks for the insights and discussion!

Jeremy S: Thank you!

More about… Postgres Yurri Rashkovskii PGXN Extensions PGConf Summit

The Pragmatic Engineer

Getting an Engineering Executive Job

An overview of successful, tried-and-true routes into CTO, VPE, and Head of Engineering jobs, from the new book, ‘The Engineering Executive’s Primer’ by Will Larson.

So, just how do you win that first, coveted CTO, VP of Engineering, Head of Engineering, or Director of Engineering role? I know several professionals who have achieved such promotions internally, and a couple who were successful external candidates for their first CTO role.

These jobs are usually classed as executive roles, and author and CTO, Will Larson, has just published a handy book about these upper echelons. Before this latest title, he wrote a book on engineering management (An Elegant Puzzle,) and one on staff+ engineering (Staff Engineer). Side note; we almost overlapped at Uber, with him departing the San Francisco office just as I joined the company in Amsterdam. We met in-person last year when I was in SF.

I picked up this new book, The Engineering Executive’s Primer, and can report that I’m thoroughly impressed. It’s honest in how it covers topics like navigating tricky power dynamics with CEOs, surviving “peer panic,” dealing with competition within leadership teams, and in its overall breadth and depth. There’s the usual topics: onboarding, hiring, engineering processes, performance processes, compensation, etc, and the book also goes deep into how to navigate mergers and acquisitions, cultural surveys, and how to onboard peer executives.

My copy of The Engineering Executive’s Primer

I reached out to Will to ask if he’d be open to publishing a chapter from the new book in this newsletter, and he generously agreed. In today’s issue, we cover two topics:

The book’s and Will’s backgrounds. How did the idea of the book come about, how long did it take to write, and what is Will’s advice for software engineers aiming to be engineering executives.

Chapter 1. The opening chapter is titled “Getting the Job.” It’s a thorough summary of how to get that first engineering executive position; more so than I’ve read elsewhere. An excerpt from the book.

As usual with all my recommendations, I am not paid to recommend this book, and none of the links are affiliate ones. See my ethics statement for more detail.

1. The book’s and Will’s backgrounds

Before we jump into the chapter, Will answered a few questions:

How did you get your first engineering executive job?

‘In late 2019, I was getting close to four years at Stripe, and understood the problems I was working on a bit too well. I started thinking about how to get my hands on a new set of problems. In particular, I was interested in finding ones to bring me back into product engineering, rather than continuing along the infrastructure engineering path, which I got on to somewhat accidentally at Uber.

‘Most inbound interest I received was for roles similar to the one I was in. So I asked some recently-hired executives how they’d found their jobs. Most mentioned working with executive recruiters, and I asked for referrals. This led me to Sam, the executive recruiter who helped find my CTO role at Calm, which I accepted following deep discussions with 3-4 other companies.’

What made you decide to write this book on executive roles, and can you discuss your choice to be so candid about the behind-the-scenes reality?

‘This is my third book, and I learned a lot from writing ‘An Elegant Puzzle’ and ‘Staff Engineer.’ There’s a curse in authorship, where I see so many things I could have done better in writing those books! Still, writing two intentionally different books helped me understand what I want to accomplish with my writing.

‘My goal when writing is to create something both useful and unique. Being useful is the only good reason to write a professional book, while being unique is the only way to create something durable that resonates with readers over time, and doesn’t fade away after a year or two.

‘When I first became an executive, I learned a lot by making mistakes. I wanted to scrutinize those errors, figure out frameworks that would’ve helped me avoid them, and collate these insights in a book to help others entering that role. Helping others avoid my mistakes is my best tool for advancing the technology industry, which is the professional goal I’m trying to channel my energy toward.’

How did you write the book, and how long did it take? I guess you started taking notes a long time ago, with so many observations across 24 chapters and more than 300 pages.

‘I have two concurrent writing processes:

I write down things I learn, as I learn them. This is the closest I have to a separate note-taking process. For example, I wrote the blog posts Hard to work with and Reading a Profit & Loss statement in 2022 as standalone pieces, and updated versions of each show up in The Engineering Executive’s Primer. 

I write based upon an outline of topics for a book. The chapter, Measuring an engineering organization, is a good example, and was the first piece I wrote explicitly with the new book in mind.

‘Writing this book took about 14 months. I started working on the outline in late 2022, and iterated on it three or four times, before it stabilized roughly into these chapters. I wrote about two-thirds of chapters before I started talking with the publisher, O’Reilly, about whether they’d be interested in it. 

‘After signing the contract, I reworked the topics a bit with my editor, Virginia Wilson, completed the remaining chapters, and revised individual chapters several times based on feedback from her and reviewers. Writing books really does get easier the more you do it, and this book was relatively peaceful compared to my first two.’

You’ve been a software engineer, and are now a CTO. What are the biggest differences between these roles? 

‘A friend recently emailed me asking if it’s possible to have friends at work as an executive, because he was finding that harder and harder, the more senior he’d gotten. In general, I’ve found it’s complex to have work friends as an executive, and that it only works when both individuals take the long view on outcomes. I miss my earlier roles where these things were less fraught.

‘Conversely, I’m really motivated by being able to engage with problems blocking forward progress, and being in an effective executive team is a job characterized by exclusively dealing with that kind of problem. For me, engaging with problems is the most energizing part of being an executive. At the same time, in my earlier work as an engineer I often found myself stymied by organizational constraints. To be fair, I was also a lot less effective at navigating organizational constraints back then.’

What’s your advice for software engineers ambitious to be a Head of Engineering, VP of Engineering, or CTO?

‘Spend more time understanding why people’s perspectives are “right,” rather than why they’re “wrong.” I think of this as extracting the core of what people say. 

‘Being able to learn from those who don’t communicate clearly is a super power. Many who are theoretically good communicators – maybe the executives you work with – are too busy to always communicate clearly, and getting good at understanding them despite messy formatting is invaluable.

‘Spend less time on pursuits you don’t find energizing. For example, I’ve seen so many people try to become “content creators” to further their career, despite having no interest in creating content. The vast majority of successful executives don’t write stuff online and don’t speak at conferences. Do so if it’s energizing for you, but if it isn’t, find something that is!’

Thanks Will for sharing your thoughts! With that, we dive into Chapter 1 of the book.

2. Getting an Engineering Executive Job

The below excerpt is from The Engineering Executive's Primer, by Will Larson. Copyright © 2024 Will Larson. Published by O'Reilly Media, Inc. Used with permission.

At Digg, I ended up running Engineering, but I certainly wasn’t hired to do so. It wasn’t until a decade later, when I joined Calm, that a company deliberately hired me into my first executive role. If you start researching executive career paths, you’ll find folks who nominally became Engineering executives at 21 when they found a company, and others who were more than 30 years into their career before taking an Engineering executive role.

As these anecdotes suggest, there is no “one way” to get an Engineering executive job. However, the more stories you hear about folks assuming executive roles, the more they start to sound pretty similar. I’ve condensed the many stories I’ve heard, along with my own experiences, into a repeatable process that prospective candidates typically follow.

This chapter will cover:

deciding whether to pursue an executive role

why each executive job search is unique, and how that will shape your process

finding executive roles externally and internally

navigating the often chaotic executive interview process after you’ve gotten comfortable interviewing in well-designed middle management interview processes

negotiating an executive contract, particularly the terms that rarely come up in the non-executive contracts you may have negotiated prior

deciding whether to accept an executive offer once you have it

If you’re kicking off the search for your first executive role, reading through this chapter will provide a clear roadmap through the process.

Why Pursue an Executive Role?

If you’re spinning up your first executive role search, you should have a clear answer to a question you’ll get a number of times, “Why are you looking for an executive role?” It’s important to answer this for yourself, as it will be a valuable guide throughout your search. If you’re not sure what the answer is, spend time thinking this through until you have a clear answer (maybe in the context of a career checkup).

There’s no right answer, but here are some examples from real people:

“I’m heavily motivated by learning. I’ve directly reported into an Engineering executive for my past two roles, and I’m looking to step into the role myself.”

“I’ve enjoyed working in a fast-growing company, but I also miss the direct ownership and pace of working at a small company. I’m excited to combine my previous startup experience with my recent experience at scale as an Engineering executive.”

The rationale doesn’t need to be particularly compelling, just something positive that expresses your excitement and qualification for the role. Don’t get discouraged if your statement isn’t profound—there are very few profound ways to say that it’s the next logical step in your career. Once you’ve written your rationale down, review it with a few peers or mentors who have already been in executive roles. Incorporate their feedback, and you’re done. (If you don’t have peers or mentors in executive roles, do some cold outreach to executives at companies you’ve worked at with your rationale and see if they’ll weigh in.)

The other side of this is that interviewers are also very curious about your reason for pursuing an executive role, but not necessarily for the reason you’d expect. Rather than looking for your unique story (although, yes, they’ll certainly love a memorable, unique story), they’re trying to filter out candidates with red flags: ego, jealousy, excessive status-orientation, and ambivalence.

One of One

Limited-release luxury items like fancy cars sometimes label each item with their specific production number, along with the size of the overall run. For example, you might get the fifth car in a run of 20 cars overall. The most exclusive possible production run is “one of one.” That item is truly bespoke, custom, and one of a kind.

All executive roles and processes are “one of one.”

For non-executive roles, good interviewing processes are systematized, consistent, and structured. Sometimes the interview processes for executive roles are well-structured, but more often they aren’t. If you approach these bespoke processes like your previous experiences interviewing, your instincts may mislead you through the process.

The most important thing to remember when searching for an executive role is that while there are guidelines, stories, and even statistics, there are no rules when it comes to finding executive jobs. There is a selection bias in executive hiring for confidence, which makes it relatively easy to find an executive who will tell you with complete confidence how things work but be a bit wary.

It’s not just the hiring process that is not standardized; the Engineering executive roles themselves vary greatly as well. Sometimes they’ll include managing Product Management, and sometimes they’ll exclude managing some parts of Engineering. Working with technology-oriented founders, you may provide more organizational support than technical guidance, whereas working in an older business may mean there are few other executives with a technology background. “One of one” means that anything is possible, in both the best and worst possible sense.

Finding Internal Executive Roles

Relatively few folks find their first executive job through an internal promotion. These are rare for a couple reasons. The first is that each company only has one Engineering executive, and that role is usually already filled. The second is that companies seeking a new Engineering executive generally need someone with a significantly different set of skills than the team they already have in place.

Even in cases where folks do take on an executive role at their current company, they often struggle to succeed. Their challenges mirror those of taking on tech lead manager roles, where they are stuck learning how to do their new job in addition to performing their previous role. They are often also dealing with other internal candidates who were previously their peers and who may feel slighted by not getting the role themselves. This makes their new job even more challenging, and can lead to departures that hollow out the organization’s key leaders at a particularly challenging time.

That’s not to say that you should avoid or decline an internal promotion into an executive engineering role; just that you should go into it with your eyes open. In many ways, it’s harder to transition internally than externally. Because of that, even if an internal transition into an executive role goes poorly for you, don’t assume that means you wouldn’t do well as a newly hired executive at another company.

Finding External Executive Roles

Most executive roles are never posted on the company’s jobs page. So before discussing how you should approach your executive job search, let’s dig into how companies usually find candidates for their executive roles. Let’s imagine that my defunct company Monocle Studios had been a wild success and we wanted to hire our first CTO.

How would we find candidates? Something along the lines of:

Consider any internal candidates for the role.

Reach out to the best folks in my existing network, seeing if any are interested in interviewing for the role.

Ask our internal executive recruiter to source candidates. (I’d skip this step if we didn’t have any executive recruiters internally, as generally there’s a different network and approach to running an executive search than a non-executive search; executive candidates also tend to ask different questions than non-executive candidates, which makes hiring them with non-executive recruiters even messier.)

Reach out to our existing investors for their help, relying on both their networks and their firms’ recruiting teams.

Hire an executive recruiting firm to take over the search.

Certainly not every company does every job search this way, but it does seem to be the consistent norm. This structure exposes why it’s difficult to answer the question, “How do I find my first executive role?” The quick answer is to connect with an executive recruiter—ideally one that peers have worked with before—but that approach comes with some implications on the sort of roles you’ll get exposed to. Typically, these will be roles that have been challenging to fill for some reason.

It’s important to note that the most desirable roles, and roles being hired by a well-networked and well-respected CEO, will never reach an executive recruiting firm. If you try to enter your search without an established network and rely solely on executive recruiters to find roles, you are almost certain to be selecting from second-tier opportunities.

This is, by the way, absolutely not a recommendation against using executive recruiters. Executive recruiting firms can be fantastic. A good executive recruiter will coach you through the process much more diligently than the typical company or investor’s in-house recruiter. I found my first executive role through an executive recruiter, as did the majority of my peers. (Note that the executive recruiters of tomorrow are your internal recruiting colleagues of today, so learning to partner effectively with Recruiting will pay dividends in both your current hiring and your long-term career options.) Similarly, it’s not true that all founder-led searches are for desirable jobs—almost all executive roles start as founder-led searches before working their way through the pipeline.

​​Looking at the pipeline, there are many ways to increase your odds of getting executive opportunities at each step. The basics still matter: Maintain an updated LinkedIn profile, and respond politely to recruiters who do reach out. Both have a surprising way of creating job search serendipity, and ensuring your network is aware that you’re looking. If you don’t personally know many recruiters at investors or executive recruiters, your network can be particularly helpful for making those introductions.

There are also a small number of companies that do post executive roles publicly, and there’s certainly no harm in looking through those as well. The one challenge is that you’ll have to figure out whether it’s posted publicly because the company is very principled about searching for talent outside their personal networks (often a good sign), or if the role has already passed unsuccessfully through the entire funnel described above (often not a good sign). Most companies with strong principles like to talk about them a lot, and you should be able to find public evidence to support their posting coming from a principled belief. If you can’t, then it’s likely desperation.

Finally, if you’re laying the groundwork for an executive search a few years down the road, there’s quite a bit you can do to prepare. You can join a large or high-growth company to expand your network (more on this in Chapter 12), work in a role where you get exposure to the company’s investors, create more visibility of your work (more on this in Chapter 12 and Chapter 15) to make it more likely for founders to reach out to you, or get more relevant experience growing and operating an Engineering organization.

Interview Process

The interview process for executive roles is always a bit chaotic. The most surprising thing for most candidates is that the process often feels less focused or effective than their other recent interviews. This is because your hiring manager as a director of Engineering is usually an experienced engineering leader, but your hiring manager as an Engineering executive is usually someone with no engineering experience at all. In the first case, you’re being interviewed by someone who understands your job quite well, and in the second, the interviewer usually has never worked in the role.

There are, inevitably, exceptions! Sometimes your interviewer was an Engineering executive at an earlier point in their career, but that usually isn’t the case. A relatively common scenario in startups is when a technical founder interviews you for the role, potentially with them staying as the CTO and you taking on the VPE title. But, even then, it’s worth noting that the title is a bit of a smokescreen, and they likely have limited experience as an Engineering executive.

Consequently, Engineering executive interviews depend more heavily on perceived fit, prestige, the size of the teams you’ve previously managed, being personable, and navigating the specific, concrete concerns of would-be direct reports and peers. This makes the “little things” particularly important in executive interviews: send quick and polite follow-ups, use something like the STAR method to keep your answers concise and organized, prepare questions that show you’re strengthening your mental model of how the company works, and generally show energy and excitement.

The general interview process that I’ve seen for executive roles is as follows:

Call with a recruiter to validate you meet the minimum requirements, are a decent communicator, and won’t embarrass them if you talk to the CEO. Recruiters are heavily scrutinized on the quality of candidates they bring forward and will go out of their way to help you show up well. This is also a good opportunity for you to understand whether there are obvious issues that might make this a bad role for you, such as wrong job location, wrong travel expectations, and so forth.

Call with the CEO or another executive to assess interest in the role, and very high-level potential fit for the role. You’ll be evaluated primarily on your background, your preparation for the discussion, the quality of your communication, and perceived excitement for the company.

Series of discussions with the CEO or founder, where you dig into the business and their priorities for the role. This will be a mix of you learning from them, them learning about you, and getting a mutual sense of whether you’ll work well together. The exact structure will vary depending on the CEO or founder, and it will give you an understanding of what kind of person they are to work with.

One-on-one discussions with a wide smattering of peer executives and members of the team that you would manage. These vary widely across companies, and it is surprisingly common for the interviews to be poorly coordinated—for example, the same topics may come up multiple times across different interviewers. This is somewhat frustrating. Generally, it means the company is missing someone with the right position, experience, and energy to invest into designing the loop. I’ve had these interviews turn into general chats, programming screens, architecture interviews, and anything else you can imagine. All I can say is: Roll with it to the best of your ability.

Presentation interview to the executive team, your directors, or a mix of both. Usually, you’ll be asked to run a 60-minute presentation describing your background, a point of view on what’s important for the business moving forward, your understanding of what you would focus on in the new role if hired, and your plan for your first 90 days.

Here are a few tips that I’ve found effective for these interviews:

Ask an interviewer for feedback on your presentation before the session.

Ask what other candidates have done that was particularly well received.

Make sure to follow the prompt directly.

Prioritize where you want to spend time in the presentation (on the highest-impact topics).

Make sure to leave time for questions (while also having enough bonus content to fill the time if there aren’t many).

If this sounds surprisingly vague and a bit random, then you’ve read it correctly. Gone are the days of cramming in all the right answers. Now, it’s a matter of reading each individual effectively and pairing the right response to their perspective. If that feels arbitrary, keep in mind that navigating various perspectives will be a significant part of your role as an executive going forward!

Negotiating the Contract

Once a company decides to make you an offer, you enter into the negotiation phase. While the general rules of negotiation still apply—particularly, don’t start negotiating until the company knows it wants to hire you—this is a moment when it’s important to remember that these are one of one jobs. Compensation consultants and investors will have recommended pay ranges, but each company only hires one Engineering executive at a time, and every company is unique.

Fair pay will vary greatly depending on the company, the size of its business, your location, and your own background. Your best bet will be reaching out to peers in similar roles to understand their compensation. I’ve found folks to be surprisingly willing to share compensation details. It’s also helpful to read DEF 14A filings for public companies, which explain their top executives’ base, bonus, and equity compensation (for example, here is Splunk’s DEF 14A from 2022).

There are a few aspects of this negotiation that are sufficiently different from earlier compensation negotiations:

Equity

Equity is issued in many formats: stock options, Restricted Stock Units, and so on. Equity is also issued with many conditions: vesting periods (often 4 years), vesting cliffs before vesting accrues (often 1 year), and the duration of the period after you depart when you’re able to exercise options before they expire (often 90 days).

Most of these terms are negotiable in an executive offer, but it all comes down to the particular company you’re speaking with. You may be able to negotiate away your vesting cliff, and immediately start vesting monthly rather than waiting a year; or negotiate an extended post-departure exercise window, even if that isn’t an option more widely; or have the company issue you a loan to cover your exercise costs, which combined with early exercise might allow you to exercise for “free” except for the very real tax consequences.

To determine your negotiation strategy, I highly recommend consulting with a tax advisor, as the “best” option will depend on your particular circumstances.

Equity acceleration

Equity acceleration is another negotiation point around equity. This is worth calling out as it’s common in executive offers, and extremely uncommon in other cases. Acceleration allows you to vest equity immediately if certain conditions are met. Many consider this a standard condition for a startup contract, although there are many executives who don’t have an acceleration clause.

One topic that gets perhaps undue attention is the distinction between single and double trigger acceleration. “Single trigger” acceleration has only one condition to be met (for example, your company is acquired), whereas “double trigger” acceleration will specify two conditions (for example, your company is acquired and you lose your job). My sense is that people like to talk about single and double triggers because it makes them sound knowledgeable about the topic rather than it being a particularly nuanced aspect of the discussion.

Severance packages

Severance packages can be negotiated, guaranteeing compensation after you exit the role. There is little consistency on this topic. Agreements range from executives at very small companies that have pre-negotiated a few months’ salary as a severance package, to executives leaving highly compensated roles that require their new company to make them whole on the compensation they’re leaving behind. There are also many executive contracts that don’t pre-negotiate severance at all, leaving the negotiation until the departure (when you admittedly have limited leverage).

Bonus

Bonus size and calculation can be negotiated. On average, bonus tends to be a larger component of roles outside of engineering, such as a sales executive, but like everything, this is company- and size-specific. A CTO at a public company might have their bonus be equal in size to their salary. A CTO at a Series C company might have a 20% bonus. A CTO at a 50-person company might have no bonus at all.

In addition to the size of your bonus, you may be able to negotiate the conditions for earning it. This won’t matter with companies that rely on a shared bonus goal for all executives (sometimes excluding sales), but may matter a great deal with other companies that employ bespoke, per-executive goals instead.

Parental leave

Parental leave can be negotiated. For example, some companies might only offer paid parental leave after a year of service, but you can absolutely negotiate to include that after a shorter amount of service. (It’s worth noting that this is often negotiable in less senior roles, as well.)

Start date

Start date is generally quite easy to negotiate in less senior roles but can be unexpectedly messy for executive roles. The reason it gets messy is that the hiring company often has an urgent need for the role to be filled, while also wanting to see a great deal of excitement from the candidate about joining.

The quiet part is that many recruiters and companies have seen executive candidates accept but later not join due to an opposing offer being sweetened, which makes them uncomfortable delaying, particularly for candidates who have been negotiating with other companies, including their current one.

Support

Support to perform your role successfully is another point that can be negotiated. The typical example of vain requests for support are guaranteed business- or first-class seats on business travel, but there are other dimensions of support that will genuinely impact your ability to perform your role. For example, negotiating for an executive assistant can free up hours every week for focus work, and negotiating a sufficient budget to staff your team can easily be the difference between a great and terrible first year.

The negotiation phase is the right time to ask for whatever you’ll need to succeed in the role. You’ll never have an easier time to ensure you and your organization can succeed.

Negotiate knowing that anything is possible but remember that you have to work with the people you’re negotiating with after the negotiation ends. If you push too many times, you won’t be the first candidate to have their offer pulled because the offering company has lost confidence that you really want to be there.

Deciding to Take the Job

Once you get an offer for an executive position, it can be remarkably hard to say no. The recruiters you’re working with will push you to accept. The company you’re speaking with will push you to accept. You’ll have invested a great deal of work into the process, and that will bias you toward wanting to accept as well.

It’s also challenging to evaluate an executive offer, because ultimately you’re doing two very difficult things. First, you’re trying to predict the company’s future trajectory, which is hard even for venture capitalists who do it a lot (and they’re solving for an easier problem as they get to make many concurrent investments, and you can only have one job at a time). Second, you’re trying to make a decision that balances all of your needs, which a surprising number of folks get wrong (including taking prestigious or high-paying jobs that they know they’re going to hate, but just can’t say no to).

I can’t really tell you whether to accept your offer, but there are a few steps that I would push you to take before finalizing your decision:

Spend enough time with the CEO to be sure you’ll enjoy working with them, and that you’ll trust them to lead the company. While it changes a bit as companies scale, and particularly as they go public, the CEO is the person who will be deciding company direction, determining the members of the executive team, and taking responsibility to resolve the trickiest decisions.

Speak to at least one member of their board. Admittedly, board members won’t directly tell you anything too spicy, but their willingness to meet with you is an important signal, and it’s the best opportunity to start building your relationship with the board.

Make sure you’ve spoken with every member of the executive team that you’d work with regularly. Sometimes you’ll miss someone in your interview process due to scheduling issues, and it’s important to chat with everyone and make sure they’re folks you can build an effective working relationship with.

Make sure they’ve actually answered your questions. I once interviewed to be a company’s head of Engineering, and they refused to share their current valuation with me! I pushed a few times, but ultimately they told me it was unreasonable to ask, and I decided I couldn’t move forward with a company that wouldn’t even share their valuation with an executive candidate.

Don’t assume they’ll disclose this information after you join the company if they won’t tell you when trying to convince you to accept their offer. You will never have more leverage to get questions answered than during the hiring process: If it’s important and they won’t answer, be willing to walk away.

If the company has recently had executives depart, see if you can get an understanding for why. This could be learned through mutual friends with the departed executive, or even chatting with them directly. Sometimes you’ll even have executives who interviewed you depart before, or shortly after, you join. You should absolutely reach out to them and understand the reasons for their departure.

As you work through these steps, ask yourself: Are you still excited? Have you explained your thinking about the role to at least two friends (who didn’t raise any concerns)? If the answer to these questions is yes, then take the job!

Not Getting the Job

You can’t talk about running an Engineering executive search without talking about not getting the job. Who doesn’t have a story of getting contacted by a recruiter who then ghosts them after an initial screen? A public company recently invited a friend of mine to interview in their CTO search. They got my friend very excited, and then notified them the next week that they had already tentatively filled the role. I’ve had first discussions with CEOs where we both immediately knew we wouldn’t be a good fit to work together. I’ve discussed roles where both I and the CEO wanted to move forward, but where I lacked a specific skill they felt was required to succeed (for example, deep experience in machine learning).

Although rejection isn’t fun, the perspective that I find helpful is: The goal of your search is not to find an executive job, but rather to find an executive job where you will thrive. It’s much better to realize a job isn’t the right fit for you before taking it, and each opportunity that doesn’t move forward is for the best.

Gergely again. I hope you have enjoyed this in-depth look into how to get a coveted engineering executive role, via The Engineering Executive’s Primer.

Check out the full book

Many things Will discusses above are open secrets among engineering leaders; like that each executive role and hiring process is unique, and CTO roles at different companies often have vastly different processes, expectations, and compensation ranges!

Nearer the heights, compensation also becomes more negotiable; not only equity and salary, but also equity acceleration. That’s why, when negotiating a compensation package in a new country, it’s sensible to invest in tax advice about the equity component from a local accountant or equity expert. I recently spoke with an engineering director in San Francisco who was offered a CTO role with generous equity in Germany; but equity taxation is quite different there and they were researching it, first.

The rest of this book keeps up the quality, with lots of behind-the-scenes insights. If you’ve gotten something from this excerpt, you’ll likely get even more from the rest. So, if you’re an engineering executive, or want to become one, I reckon The Engineering Executive’s Primer is a very useful volume, as the author hoped for. 

This week, there won’t be a new issue of The Pulse on Thursday, as I’m on spring break. As of next Tuesday onward, things are back to the usual schedule. Thanks for your support!


Doc Searls Weblog

Burning to Write. And Vice Versa.

Among all artists, writers alone suffer the illusion that the world needs to hear what they have to say. I thought that line, or something like it, came from Rollo May, probably in The Courage to Create. But a search within that book says no. ChatGPT and Gemini both tell me May didn’t say it […]
I shot this photo in July 2007, while sitting with friends in that same booth at the Eagle and Child in Oxford.

Among all artists, writers alone suffer the illusion that the world needs to hear what they have to say.

I thought that line, or something like it, came from Rollo May, probably in The Courage to Create. But a search within that book says no. ChatGPT and Gemini both tell me May didn’t say it anywhere.

Still, I think it’s true that writers write because they have to. They can’t not write. It’s what they do, how they live. And I’m one of them.

The need to write is for me a vivid fact this morning, a few days past a catheter ablation on my heart. There is so much I need to say, because I suffer the illusion that the world needs to hear it. Does it?

I am sure C.S. Lewis and J.R.R. Tolkein had the same affliction. I am also sure that the world is better for having read both of them, even if the world could have lived without their writing.

As for time, I have had twelve more years to write than Lewis got, so far, and five less than Tolkein.

Time to say what?

I want to say that personal AI will do far more for all of us than what we will ever get from AI as a corporate service. And to say it better than I just did.

I want to say that we will do better with rulers who care about people than with rulers who merely rule. And to say that better than I just did.

I want to complete the work of John McPhee by reporting as best I can what has happened to the great characters that anchored every one of his essays and books. But that project is not on the back burner. It’s in the fridge, where I’ve kept it for decades (while continuing to read the entire McPhee oeuvre, much of it repeatedly).

Speaking of burning, I am impelled by Dylan Thomas, who wrote “Do not go gentle into That Good Night,” before dying at just thirty-nine. The poem was for his father:

Old age should burn and rave at close of day,
Rage, rage against the dying of the light.

And so I do.


Michael Ruminer

Let Me Save You Some Time on Azure OpenAI And Copilot

I continue to develop my interest and some proficiency in AI. I am specifically focusing on Microsoft’s Copilot and Azure offerings. How did I end on that choice? I used to be a Microsoft MVP and also have a personal Visual Studio license that comes with $150 a month in Azure credits. So I have a long history in the Microsoft stack and tools plus some credits to play with each month. Those credits

I continue to develop my interest and some proficiency in AI. I am specifically focusing on Microsoft’s Copilot and Azure offerings. How did I end on that choice? I used to be a Microsoft MVP and also have a personal Visual Studio license that comes with $150 a month in Azure credits. So I have a long history in the Microsoft stack and tools plus some credits to play with each month. Those credits don’t go far when dealing with Azure AI pricing but I appreciate them. This last weekend I was going through an exercise I found on connecting up Azure AI search with Azure OpenAI layered on top, and then letting that be the foundation for a Copilot. You can find the excellent video by Lisa Crosbie here. This is a post on what I encountered that took me a while to overcome so that if you have a similar set of conditions you may spend less time to get running.

The crux of my arrangement and some heartache is the personal Visual Studio Azure subscription id; an emphasis on “personal”. This is where I have the monthly credits, but you need a business Microsoft account to use the Copilot Studio.

Here is the short end of it. If you can call this short. It’s complicated, what I did to get there, but doesn’t have to be too complicated for you. Here are the steps I went through so you don’t have to:

I needed to apply for a Microsoft OpenAI Studio preview access. I needed to use a non-personal email. No outlook.com, gmail etc. Once approved the following day when trying to kick off a Copilot creation from the deployed OpenAi instance it asked me to log in and would only accept a business or school Microsoft account — my personal account wouldn’t work. I created a new Microsoft business account by subscribing to Office 365 basic business I tried to kick off the Copilot creation from the OpenAI deployment on my personal account and when it asked me to log in for the Copilot I put in my new business Microsoft account. It complained that it couldn’t connect with the Azure OpenAi deployment. Which made total sense- it was under a different account than the Copilot I was trying to create, but I had to try it. So, I subscribed to Azure using that newly minted business account. It came with $200 credit the first month. I tried to apply for OpenAI using the “business” email address the Azure subscription gave me when subscribing to Azure- a ruminio944.onmicrosoft.com domain. It immediately denied me saying that the email address was a personal email account. I wasn’t expecting that. I had a dormant domain sitting in GoDaddy (who doesn’t) and added it to my Azure account as a domain and set up a shared email for that domain. I now had factraft@factraft.com email address. I am sure I could have set up an email box directly on GoDaddy for the domain but this was more fun and a learning experience. I now had a MS business login with an Azure subscription and $200 credit to play with and what should be considered a business email. I applied for the OpenAI Studio Preview again this time using my new Azure subscription id and new factraft email. I expected it to be denied as all the other information such as name, address, website etc. was the same as I had already submitted and been approved for under my personal Azure subscription id. Surprisingly, the next day I received a welcome message for OpenAi Studio Preview. I went through the video exercise in my new Azure account and most things worked as in the video. The Lesson

To make it work you need the OpenAI Studio Preview which requires a business email address, a website, and an approved use case to request admission to the preview, and that is no guarantee of approval. You’ll need a business Microsoft account to log into in order to kick off the Copilot studio. Personal emails and a personal Microsoft account won’t cut it. I created a business Microsoft account by subscribing to Office 365 business basic for about $8. Then added an Azure instance to this with $200 credits for the first month. Then I was off to the races- mostly. I was able to make it all work for a mere mortal for the cost of about $8, the one month Office 365 subscription, and a partial days effort . All in all, not bad. I’ll make another write up on what I discovered in the creation itself. If just the right person(s) finds this trying to play with Azure OpenAI and Copilot studio then it might save them a fair amount of time.

Monday, 06. May 2024

Patrick Breyer

Computerspiele-Sterben: PIRATEN fordern Kulturschutz für Games

In Zusammenarbeit mit der Spielerinitiative „Stop killing games“, die sich gegen das Lahmlegen von Computerspielen durch ihre Hersteller wendet, haben der Europaabgeordnete der Piratenpartei Dr. Patrick …

In Zusammenarbeit mit der Spielerinitiative „Stop killing games“, die sich gegen das Lahmlegen von Computerspielen durch ihre Hersteller wendet, haben der Europaabgeordnete der Piratenpartei Dr. Patrick Breyer und ein Mitglied des Kulturausschusses eine Dringlichkeitsanfrage an die EU-Kommission gerichtet. Hintergrund des Gamer-Protests ist die Entscheidung des französischen Computerspieleherstellers Ubisoft, das bis Dezember 2023 verkaufte Rennspiel „The Crew 1“ ab April 2024 unbenutzbar zu machen. Breyer erklärt:

„Wir wollen eine Stellungnahme der EU-Kommission dazu, ob Spielehersteller beliebte Spiele gewinnbringend verkaufen, sich dabei aber die willkürliche, jederzeitige Unbrauchbarmachung vorbehalten dürfen. Diese Praxis der Branche dürfte als ‚missbräuchliche Klausel in Verbraucherverträgen‘ illegal sein. Sie wird vor allem der kulturellen Bedeutung von Games nicht gerecht, die gewachsene Gemeinschaften von Millionen von Spielern zusammen bringen können. Wir Piraten fordern, dass die EU Games als Kulturgut schützt, ein Lahmlegen nach Gutdünken verbietet und, wenn der Hersteller Games aufgibt, ein Weiterführen durch die Community ermöglicht.“

Die Digitalexpertin und Spitzenkandidatin der Piratenpartei zur Europawahl Anja Hirschel ergänzt: „Aus Sicht der Spieler ist es geradezu unverschämt, ein Spiel anzubieten und dieses dann plötzlich vollständig ‚abzuwürgen‘. Eine gewisse Nutzungsdauer ab Kaufdatum muss garantiert werden. Daher ist dies auch eine Frage des Verbraucherschutzes.“

Breyer hatte bereits vor einem Monat die EU-Kommission zur Einstellung von Computerspielen befragt, wobei die Antwort aussteht. Die jetzt eingereichte Dringlichkeitsanfrage muss innerhalb von drei Wochen, also bis 16. Mai, beantwortet werden.

Friday, 03. May 2024

Wrench in the Gears

Could Bobby’s Tokenized Environmental Markets Really Be About Using Web3 to Game Quantum Mechanics?

This will be my last post for a few weeks. During my trips down to Arkansas, I’ve been reading Gary Zukav’s 1979, “The Dancing Wu Li Masters,” a lay person’s guide to the “new” physics that emerged out of meetings he had at Esalen, home base for the human potential movement. I recognize there continue [...]

This will be my last post for a few weeks. During my trips down to Arkansas, I’ve been reading Gary Zukav’s 1979, “The Dancing Wu Li Masters,” a lay person’s guide to the “new” physics that emerged out of meetings he had at Esalen, home base for the human potential movement. I recognize there continue to be many questions about quantum mechanics. At this point, I am leaning towards Hugh Everett’s “Many World’s Theory” and John Wheeler’s  “Participatory Universe.” I don’t claim to have any of this pinned down, and yet I wanted to share some thoughts I’ve been having on relativity, perception, spacetime, lattices, and construction of personalized “realities” using consciousness and information theory.

When Leo first started his work on Web3 tokenization of natural capital, he looked into the origins of the International Geophysical Year, use of radio isotopes to track energy transfer in ecosystems, and UNESCO’s “Man and the Biosphere” (Noosphere?) programme. I am starting to think that the focus on One Health, climate, and the UN Sustainable Development Goals could actually be about relativity – employing digital identity and Web3 to situate individual representations of consciousness within webs of relationship over time. That data could then be used not only to discern patterns but to incorporate game mechanics towards some form of networked collective manifestation.

Individual experiences of the ant computer would be shaped by each participant’s environmental context. Through gamification and spatial computing we, as agents, would emit signals that could be used to coordinate a vast sociotechnical computational system of domesticated consciousness. But first, the system would need to know where each of us is at a given time, what information flows we have consumed, what our perception of reality is, and how best to use the archetypal fields that surround us to nudge us towards pre-programmed goal-oriented behaviors, how to optimize the agents with the least amount of friction. Go with the flow….

Thinking of carbon-credit behaviors in this way – as gamified agent signaling within fractal, nested complex systems – has some legs I think. In any event, it is a much more interesting scenario than Klaus wants to turn you into a puppet, take all your things, and make you eat bugs, right?

So here is the video I made last night.

It will be interesting to watch as the RFK Jr. campaign tries to weave together the split personalities of Market-Based Environmental Solutions Bobby and Health FreeDOM Bobby. Liam Sturgess’s initial attempt was rather weak.

https://www.depauw.edu/news-media/latest-news/details/19648/

I expect a lot more information flows will need to pour forth in order to create a narrative compelling enough to get people to sign on to big data, AI for good green progress. Stay tuned.

This is the excerpt I read. Below it are screenshots of the featured maps if you’d like to explore further. It’s quite fascinating to see how the various threads spread out over many maps have come together around the Camelot candidate.

Interactive Map: https://embed.kumu.io/10176e997f243e67277e7d615ba4bbb1#untitled-map?s=bm9kZS1HYVJhVVpTNA%3D%3D

Interactive Map: https://embed.kumu.io/45f342eb350f7b4c707484423f71d6ff#untitled-map?s=bm9kZS1XWmNzcFV0Mg%3D%3D

Interactive Map: https://embed.kumu.io/c9645fc5fbabfed02dd66e0a0aea9905#untitled-map?s=bm9kZS05N3prSzlrVg%3D%3D

Interactive map: https://embed.kumu.io/ddfc18d01ccaffb4db6b8ac0711ec30a

Interactive Map: https://embed.kumu.io/a8d369aeea2eb819d4961542e0e3f91c

Interactive Map: https://embed.kumu.io/2f5210a439fe8343ae208a20c70d83fc

Interactive Map: https://embed.kumu.io/d52b8a5f6b97bfde04624f29a903ef4c

Interactive Map: https://web.archive.org/web/20210327185247/https://littlesis.org/oligrapher/6635-metronome

Interactive Map: https://embed.kumu.io/0712f3e054d2055302b00b258e24e1c0#untitled-map?s=bm9kZS1TRkFROG0xRw%3D%3D

Thursday, 02. May 2024

Aaron Parecki

OAuth for Browser-Based Apps Working Group Last Call!

The draft specification OAuth for Browser-Based Applications has just entered Working Group Last Call!

The draft specification OAuth for Browser-Based Applications has just entered Working Group Last Call!

https://datatracker.ietf.org/doc/html/draft-ietf-oauth-browser-based-apps

This begins a two-week period to collect final comments on the draft. Please review the draft and reply on the OAuth mailing list if you have any comments or concerns. And if you've reviewed the document and are happy with the current state, it is also extremely helpful if you can reply on the list to just say "looks good to me"!

If joining the mailing list is too much work, you're also welcome to comment on the Last Call issue on GitHub.

In case you were wondering, yes your comments matter! Even just a small indication of support goes a long way in these discussions!

I am extremely happy with how this draft has turned out, and would like to again give a huge thanks to Philippe De Ryck for the massive amount of work he's put in to the latest few versions to help get this over the finish line!


Wrench in the Gears

Austin’s Learn Capital Takes A Field Trip to the Vatican To Talk Entrepreneurial Social Impact

Late last April, Lynn Davenport and I recorded a conversation unpacking the visit Evan Baehr, managing partner and social impact investor with Learn Capital, made to the Vatican in the fall of 2022 for the Humanity 2.0 conference hosted in partnership with Harvard University. I think it is useful to consider Humanity 2.0 as an [...]

Late last April, Lynn Davenport and I recorded a conversation unpacking the visit Evan Baehr, managing partner and social impact investor with Learn Capital, made to the Vatican in the fall of 2022 for the Humanity 2.0 conference hosted in partnership with Harvard University. I think it is useful to consider Humanity 2.0 as an extension of the eugenics (optimization) movement that became aligned with the Human Potential Movement in the 1970s and is now merging with quantified-self “wellness management” and alt-health biohacking.

The social science division of the Catholic Church appears to be totally onboard with the idea of aligning big data impact investing with preventative care (cue Bobby’s “chronic health” epidemic plans) using their far-reaching health system infrastructure. You can explore overlaps between Social Finance’s Ronald Cohen and Rome here, here, here, and here. In particular, note this post on the role of Providence Health, a Catholic Health System, in the early days of Covid.

Lumedic = light. Note the lighting in the feature image.

Interactive Map: https://embed.kumu.io/052a2ea893a8f845488a6f5b14515d4d#untitled-map?s=bm9kZS1iMzZHYXkwaw%3D%3D

We need to understand that networked biology and social gaming will be at the core of light-based, post-Moore’s law computing. Such a system will combine non-traditional interfaces, cyber-physical systems (including nanotechnology and frequency transmission), and spatial computing. In such a system our participation, through the perpetual signaling of our “choices,” will dynamically reshape the “game,” though we may not ever truly understand what the ultimate goal of the game is due to the deployment of sophisticated “knowledge management” techniques.

In my opinion, digital education linked to biometric monitoring is an essential part of a plan that aims to harness embodied group consciousness for collective problem-solving and/or pattern identification, possibly within the context of an abstracted mathematical / metaphysical spacetime continuum the general public cannot readily comprehend.

Networked consciousness, De Chardin’s noosphere, is the goal. Widespread deployment of a “free” “open” global ed-tech protocol layer will hasten the planned evolutionary trajectory using Google’s Lamarckian “selfish ledgers.” Social Impact will be used to justify ongoing monitoring and management of key performance indicators, ubiquitous transmission parsed by machine learning within an ever evolving computational “game of life” where Earth’s beings are remade as agents in an infinitely complex cellular automata program. That’s something surely Nicole Shanahan, an alumna of Santa Clara University (lead in the social impact deployment space) and Sergey Brin’s ex would know about, right?

Shortly after we recorded it, life got complicated, and the clips got put on the back burner. This week I pulled everything off the shelf, blew off the dust, did some editing and found the content surprisingly current with respect to the manufactured school choice wars, funded by the likes of free market options traders like Jeff Yass (see maps below) as well as the Catholic undertones of Bobby’s presidential campaign. It’s the fourth installment in a series on the roll out of Open Education Resources (aka digital playlist education with competency badges for global gig labor).

Here are links to the other installments if you want context for this one: 

OER in Texas: Building Digital Twins for a Global Gig Economy: Here

VR Apprenticeships, Income Sharing Agreements and the Omega Point: Here

Learn Capital Comes to Austin: Here

 

 

OER Slide Deck – Presentation Starts At Slide 98.

Interactive Map: https://embed.kumu.io/91af355393887ce4efd9c139cf0d69d4#untitled-map?s=bm9kZS1LeHFNU2Zkcg%3D%3D

 

Interactive Map: https://embed.kumu.io/cbaaef670490292b6d28fd89a578a55c

Learn Capital, launched by Greg Mauro on Powder Mountain outside Salt Lake City, opened their Austin campus last year and has been an influential player in promoting adoption of digital education and blockchain / web3 technology in the Lone Star state. Lynn and I have a lot more to talk about with respect to Texas, classical antiquity, and archetypal morphic fields. Check out this years Garden’s of Greece Dallas Arboretum benefit! A tribute to Athena, goddess of wisdom and the smart city? 

See this post on Mondragon and the Catholic Church’s Focolare movement for a model of the freeDOM cell / intentional community collective that I suspect will be the model for the ant computer system. 

Interactive Map: https://embed.kumu.io/76cd68ced810f82da69e19f2dca42b47

 

 

 


The Pragmatic Engineer

The Pulse #92: GitHub’s AI-assisted developer workflow vision

Also: Google laying off engineers as their business is booming; a good period for startup fundraising and IPOs; and how WorkOS acquired devtools startup Warrant

Programming note: next week, I’ll be on spring break, spending time in Florida, visiting my in-laws. This will mean no The Pulse on Thursday: but there will still be a Tuesday article, as usual. Thank you for your understanding and support!

The Pulse is a series covering insights, patterns, and trends within Big Tech and startups. Notice an interesting event or trend? Send me a message.

Today, we cover:

Industry pulse. Cloudflare handles an extra 12M req/sec with no issues, how does being a PM at Google or Meta compare, and are nearly 1 in 10 of all developers worldwide using GitHub Copilot? And more.

GitHub’s AI-assisted developer workflow vision: Copilot Workspace. Several startups are aiming to build an AI coding tool that goes beyond GitHub Copilot. With exceptionally fast execution, GitHub seems to have beaten them to it. It’s called GitHub Copilot Workspace, and doesn’t aim to replace devs. A review of this developer-driven, AI-assisted workflow, including insider details from the GitHub team.

Google lays off devs while business booms. Google made layoffs in its Python and Flutter platform teams, right as the company’s stock price hit an all-time high. 

Good time for tech scaleups fundraising and IPOs. It’s been a while since we’ve had so much positive news on tech IPOs, and late and early-stage fundraising. Hopefully, this trend continues!

Platform scaleup buys feature startup: WorkOS acquires Warrant. Warrant is an innovative fine-grained access control seed-stage devtools startup, built on the principles of Google’s authorization system. In what looks like a win-win outcome, WorkOS has acquired it, intending to keep intact the team, product, and all existing customers. Details on this acquisition from the founder and CEO of WorkOS.

1. Industry pulse Cloudflare doesn’t notice 12M extra req/sec

Cloudflare engineering director Benjamin Yule shared an interesting non-event:

I have so many questions as 12.5 million requests per second is a huge load! A few thousand requests per second is usually considered high, and handling a load in the millions surely requires thousands of servers (if not more!) to be spun up to handle it.

How much load a single server can handle depends on how long a request takes to handle, how much memory a request uses, and then calculating whether a request is limited in CPU or memory. Whichever resource (CPU capacity or memory) runs out first determines a server’s maximum load. You can push things pretty far with clever workarounds; it’s how the WhatsApp team served 2 million requests per second on a server with 24 cores and 100GB RAM, back in 2012!

In this case, Cloudflare soaked up the load by using Cloudflare Workers, a serverless application platform, which spun up 40,000 CPU cores (an average of 320 requests/second per CPU core.) The tweet says the Cloudflare Workers team didn’t even notice the increase, which suggests they could be handling 50-100x more load on the service! If we assume a 100x load, that could mean operating 4 million CPU cores or more simultaneously, just for Workers. By comparison, we previously covered travel booking service Agoda operating a total of 300,000 physical cores and 600,000 virtual ones. It’s likely Cloudflare is operating more than 10x this number.

Do nearly 1 in 10 developers use GitHub Copilot?

Read more


Patrick Breyer

Der Pirat-o-mat: Onlineanwendung zum Vergleich der politischen Meinung mit echten Abstimmungen im EU-Parlament startet

Mit dem Pirat-o-mat (piratomat.de) ist ab sofort ein innovatives Tool verfügbar, das Nutzern dabei hilft, ihre Standpunkte zu Themen wie Netzpolitik, digitale Rechte und Demokratie …

Mit dem Pirat-o-mat (piratomat.de) ist ab sofort ein innovatives Tool verfügbar, das Nutzern dabei hilft, ihre Standpunkte zu Themen wie Netzpolitik, digitale Rechte und Demokratie zu entwickeln und zu vergleichen. Die Besonderheit: Im Quiz sind 12 Fragen zu beantworten, die im EU-Parlament seit 2019 tatsächlich zur Abstimmung standen. In der Auswertung erfährt der Nutzer seine Übereinstimmung mit dem Abstimmungsverhalten des Europaabgeordneten der Piratenpartei und der anderen deutschen Europaabgeordneten.

Doch der Piratomat bietet mehr als nur einen einfachen Vergleich: Geordnet nach Parteien und Themengebieten ist potenziell problematisches Abstimmungsverhalten zusammengestellt. Der Schwerpunkt der ausgewählten Abstimmungen liegt auf Netzpolitik, aber auch die Themen Gesundheit, Demokratie, Umweltschutz und Transparenz sind vertreten.

Der EU-Abgeordnete der Piratenpartei Dr. Patrick Breyer, der die Website bereit stellt, erläutert:

“Als Normalbürger bekommt man in aller Regel nichts davon mit, wer in Brüssel wie abstimmt und ob das Abstimmungsverhalten den Versprechen einer Partei gerecht wird. Weil das EU-Parlament seine Abstimmungsaufzeichnungen maximal unverständlich macht, sorgen unsere Abstimmungsgrafiken für Transparenz. Der Pirat-o-mat wird viele Nutzer überraschen, manche verärgern und hoffentlich auch zu Nachfragen und Gesprächen führen. Unsere Demokratie lebt von Transparenz, Beteiligung, Glaubwürdigkeit und Rechenschaft – der Pirat-o-mat ist unser Beitrag dazu.”

Zum Pirat-o-maten

Wednesday, 01. May 2024

Phil Windleys Technometria

Internet Identity Workshop XXXVIII Report

We recently completed the 38th edition of the Internet Identity Workshop. We had 330 people from around the world who called 169 sessions. As usual there was lots of energy and thousands of side conversations. IIW is a place to get things done and it showed in the excitement in the rooms and the comments people made to me about how much they enjoyed it.

We recently completed the 38th edition of the Internet Identity Workshop. We had 330 people from around the world who called 169 sessions. As usual there was lots of energy and thousands of side conversations. IIW is a place to get things done and it showed in the excitement in the rooms and the comments people made to me about how much they enjoyed it.

Opening on Tuesday

As you can see by the pins in the map at the top of this post, there were attendees from all over the world. Not surprisingly, most of the attendees were from the US (241), followed by Canada (11). Germany, India, and Switzerland rounded out the top five with 9, 8, and 7 attendees respectively. Attendees from India (5), Thailand (3), and Korea (3) showed IIW's diversity with attendees from APAC. And there were 4 attendees from South America this time. Sadly, there were no attendees from Africa again. Please remember we offer scholarships for people from underrepresented areas, so if you'd like to come to IIW39, please let us know. If you're working on identity, we want you there.

Session on Personal AI

For states and provinces, California was first with 122. Washington (16), Utah (10), Texas (10) and New York (10) rounded out the top five. San Francisco (14) Oakland (13), San Jose (12), Seattle (11), and New York (9) were the top cities.

Demo Hour on Wednesday

In addition to sessions, we have a demo hour on Wednesday that is a little like speed dating. There were 20 different projects highlighted.

Drummond Reed hosts a session

There's always more than one session that I want to attend in any given time slot and choosing is hard. That's a common refrain. Luckily we have sessions notes that we publish in a Book of Proceedings. You can find additional photos from Doc Searls: Day 1 and Day 2.

IIW is where you will meet people to help you solve problems and move your ideas forward. Please come! IIW 39 will be held October 29-31, 2024 at the Computer History Museum. We'll have tickets available soon.

Tuesday, 30. April 2024

Patrick Breyer

Piraten zur Vorratsdatenspeicherung: Wo ein Trog ist, sammeln sich die Schweine

Der Europäische Gerichtshof hat heute seine Rechtsprechung zur Vorratsdatenspeicherung aufgeweicht und die Nutzung auf Vorrat gespeicherter Internet-Verbindungsdaten schon zur Verfolgung von Filesharing zugelassen. Der Europaabgeordnete der Piratenpartei und langjährige Gegner …

Der Europäische Gerichtshof hat heute seine Rechtsprechung zur Vorratsdatenspeicherung aufgeweicht und die Nutzung auf Vorrat gespeicherter Internet-Verbindungsdaten schon zur Verfolgung von Filesharing zugelassen. Der Europaabgeordnete der Piratenpartei und langjährige Gegner der Vorratsdatenspeicherung Dr. Patrick Breyer fordert:

„Zuerst hat der EuGH zur Verfolgung von Kindesmissbrauchsdarstellungen und anderer schwerer Straftaten eine flächendeckende Vorratsdatenspeicherung über die Internetverbindungen der gesamten Bevölkerung erlaubt. Ab heute soll unsere Internetnutzung aber schon zur Verfolgung von Filesharing und anderer Bagatelldelikte offengelegt werden. Wo ein Trog ist, sammeln sich die Schweine. Deswegen müssen wir alles daran setzen, die grenzenlose Datengier der Konservativen und Sozialdemokraten zu stoppen. Obwohl Deutschland Internetdelikte auch ohne IP-Vorratsdatenspeicherung sehr erfolgreich aufklärt, arbeitet die EU im Verborgenen bereits daran uns nach der Europawahl wieder eine Internet-Vorratsdatenspeicherung aufzuzwingen. Wir Piraten werden für unser Recht auf anonyme Internetnutzung kämpfen!

IP-Vorratsdatenspeicherung ist, wie wenn jede:r Bürger:in ein sichtbares Kennzeichen um den Hals gehängt bekäme und dieses auf Schritt und Tritt notiert würde. Niemand würde sich eine solche Totalerfassung des täglichen Lebens gefallen lassen. IP-Vorratsdatenspeicherung würde jeden Internetnutzer unter Generalverdacht stellen und die Internetnutzung der gesamten Bevölkerung, die unsere intimsten Vorlieben und Schwächen abbildet, nachvollziehbar machen. Eine so totale Erfassung würde Kriminalitätsvorbeugung durch anonyme Beratung und Seelsorge, Opferhilfe durch anonyme Selbsthilfeforen und auch die freie Presse gefährden, die auf anonyme Informanten angewiesen ist. 99,99 Prozent dieser Daten wären völlig nutzlos, da sie Bürger:innen betreffen, die nie auch nur in den Verdacht einer Straftat kommen.“

Dr. Patrick Breyer

Hintergrund: Der Vorsitzende von CDU/CSU im EU-Parlament Weber forderte am Samstag eine EU-weite IP-Vorratsdatenspeicherung. Unterdessen arbeitet die sog. #EUGoingDark-Arbeitsgruppe im Auftrag der EU-Kommission und EU-Regierung entsprechende Pläne aus.


The Pragmatic Engineer

What is Security Engineering? Part 2.

A broad overview of the ever-developing security engineering field; a domain that can feel intimidating to some software engineers. With Nielet D'Mello, security engineer at Datadog. Part 2.

👋 Hi, this is Gergely with a subscriber-only issue of the Pragmatic Engineer Newsletter. In every issue, I cover challenges at Big Tech and startups through the lens of engineering managers and senior engineers. To get articles like this in your inbox, every week, subscribe:

Subscribe now

Q: “As a software engineer, I’d like to learn more about security engineering. What’s a good way to understand this vast field?”

This is the second and final part of exploring this important – and, yet, often intimidating! – topic of security engineering. Giving us an overview of this field is Nielet D'Mello: a security engineer at Datadog (previously at Intel and McAfee).

In Part 1 we already covered:

Myths and misconceptions about security engineering

History of security engineering

The present

A mental model: seven core dimensions to think about application security

Towards a secure software development lifecycle (SDLC).

In today’s issue, Nielet takes us through:

Defining the criticality of a system. Security dimensions to consider as we talk about a service or systems’s criticality.

Scoring a system’s criticality. The “napkin math” approach for scoring a system’s security criticality, and a case study to bring all it to life.

Threat modeling. A criteria for threat modeling, and pre-work for this exercise.

Security paved roads. For platform teams, building pre-approved security solutions and configurations is pragmatic.

“Defense in depth,” “least privilege,” and “zero trust.” A strategy, a principle, and a security model. Use in combination to build more layered, secure systems.

The bottom of this article could be cut off in some email clients. Read the full article uninterrupted, online.

Read the full article online

With that, it’s over to Nielet.

Common security engineering terms

As a brief refresher, we use three terms frequently in this article, so let’s start by defining them:

Vulnerability: An exploitable flaw or weakness in a system’s design, implementation or deployment

Threat: The potential for a threat actor to exploit the vulnerability

Risk: Loss or damage that could occur when a threat actualizes

1. Defining the criticality of a service or system

Do all services and systems need to invest in a security design review? Not necessarily, as the need for a review depends on a service’s or system’s business risk profile. Vulnerabilities will surface as you identify security concerns in a system’s design and architecture. Code reviews and dynamic testing also surface security issues.

For critical systems, it’s worth investing in processes like security design reviews. However, how do you decide just how critical  a service or system is? Use the dimensions below for a better sense of this:

Business purpose

Public access

Custom access controls

Users of the system

Deployment environments

Data classification

Business purpose

What are the primary objectives and functions of the service or system within the context of the organization's business operations? Identify how the service contributes to achieving business goals, generating revenue, or providing stakeholder value. To figure out the risks, it’s essential to know:

The nature of business

The industry the business operates in

Regulatory requirements

Sensitivity of data involved. For example, is it restricted, or subject to PII?

Public access

Is the service accessible to external users outside of the organization's network, or the general public? Public access systems offer expanded attack surfaces.

For these, you need to assess the potential exposure to security threats and risks associated with providing services over the internet, or other public networks, as these systems are at a much higher risk of automated bot attacks, for example.

Custom access controls

All systems need custom access controls for their data, apps and resources to determine who has access to it and in what circumstances. Role-based access control (RBAC,) or attribute-based access control (ABAC,) are two examples of custom access controls. These have specific access permissions defined for users and identities, and restrictions tailored to the service’s requirements and security needs to ensure confidentiality. 

The decision to build custom access controls is usually made with the following factors in mind:

Granularity

Dynamic decisions based on real-time information and conditions

Implementation efforts

Simplicity

Flexibility

Users of the system

What different types of users interact with the service? This is key information for defining:

User roles

Authentication mechanisms

Access requirements

User activity auditing

Threat detections associated with anomalous user behavior patterns 

Adherence to regulatory compliance

The last one is especially important. Several regulatory frameworks and industry standards mandate the protection of sensitive data through user identification and access controls. Examples of such frameworks include the General Data Protection Regulation (GDPR) and California Consumer Privacy Act (CCPA). In these cases, putting these in place is not “just” about making the system secure; it’s about what ensures the system is compliant with privacy and security regulation.

Users include:

Internal users: employees, administrators

External users: customers, partners, third-party vendors. 

Deployment environments

Development, testing, staging, and production environments in which the service operates. Each environment may have different security requirements and configurations. These varying requirements depend on:

Level of risk tolerance

Need for data protection

Data availability requirements

Compliance with industry standards, regulations, and legal requirements. 

For example, a staging environment may have broader internal employee access, meaning it can be accessed by most (if not all) employees. However, the production environment tends to have much stricter access control: only specific employees or groups can access it, and even fewer will have the rights to deploy to it. And while the staging environment is unlikely to have data that is considered confidential customer data: the production environment will! So the production environment will have much more strict data security and monitoring measures deployed on its  infrastructure.

It’s pretty common for an environment to be a shared infrastructure for various services. When this is the case, robust security controls (like stricter isolation for applications and databases) are even more important! Multi-tenant architectures are a good example for such “shared infrastructure”  where stricter security controls are necessary.

Data classification

This refers to labeling data based on sensitivity, confidentiality, and regulatory requirements. Understanding the classification of data helps determine:

Appropriate security controls

Suitable encryption methods

Access restrictions for safeguarding sensitive information and preventing unauthorized disclosure or misuse.

2. Scoring a system’s “criticality”

It’s helpful to calculate a criticality score for services. For this, I like to assign weights to the security dimensions. Below is a sample of how these scores could look. It’s just an example; simpler than I usually use, and it doesn’t encompass all factors relevant for every system. Just treat it as inspiration:

Now we’ve established the basic factors for understanding risk and criticality, we can do some napkin math with criticality scores, based on characteristics:

Calculating criticality, simplified: define dimensions by important security factors

A simple way to think about a total risk “score” is to add together the weights for each dimension. So, in this case: Total Risk Score = BP + PA + CAC + US + DE + DC.

Scoring criticality

Let’s take the example of building a payment system for an e-commerce site. It needs to process a high volume of transactions via credit cards, debit cards, and other payment methods. It also needs to have payment gateway integration, account for fraud prevention, and is subject to PCI DSS compliance.

Let’s do the napkin math for this system’s criticality:

Scoring the criticality of this example system

We get the total risk score by adding up the dimensions. In this case, it comes to 15 out of a maximum of 18 points (3 + 1 + 2 + 3 + 3 + 3.) This score indicates we are talking about a critical system from a security standpoint.

All companies have unique risk-scoring and risk-tracking processes. As a software engineer, you need to figure out what a “high” service risk score means, and at what point you should reach out to the security team at your organization, if there is one.

3. Threat modeling

Read more


Just a Theory

Mini Summit: Universally Buildable Extensions

Tomorrow, May 1 2024, Yurii Rashkovskii of Omnigres will give the Postgres Extension Mini-Summit talk, “Universally buildable extensions: dev to prod.” at noon Eastern / 16:00 UTC, followed by community discussion.

Well that snuck up on me. Tomorrow, May 1 2024, Yurii Rashkovskii of Omnigres will be giving a presentation at the fifth installment of the Postgres extension ecosystem mini-summit. The tal, “Universally buildable extensions: dev to prod”, should provoke some interesting asymmetrical thinking and discussion. I hope to see you there.

Note! If you reserved a spot at a prior mini-summit, you will need to do so again for each subsequent event or get no reminders from Eventbrite. If it’s sold out just email david@ this domain, ping me on Mastodon or via the #extensions channel on the Postgres Slack or the Postgres Discord for a link or, if you’d like a calendar invitation.

More about… Postgres Extensions PGConf PGXN Trunk Summit

Wrench in the Gears

AI Wellness + Tokenized Democracy = Game B: Will The Health FreeDOM Community Buy Into Bobby’s BS?

In between packing and getting my third and final UHaul round ready this weekend, I managed to put together a selection of clips touching on RFK Jr.’s blockchain / token habit and automated Stanford law specialist Nicole Shanahan’s plans to data mine precision health systems, an effort that will underpin quantified self social impact bonds [...]

In between packing and getting my third and final UHaul round ready this weekend, I managed to put together a selection of clips touching on RFK Jr.’s blockchain / token habit and automated Stanford law specialist Nicole Shanahan’s plans to data mine precision health systems, an effort that will underpin quantified self social impact bonds and global ant computer operations. Last night after stopping for the night in Staunton “Game B – Eugenics – Woodrow Wilson” Virginia, I did an overview of the themes covered in the clip compilation, to make the concepts clearer. After speaking out against blockchain digital ID for four years, it makes me ill to see the manufactured “hero” that is Camelot Bobby pushing blockchain government, crypto, and AI / machine learning for social good. The health freeDOM – biohacking community IS Game B. So many are already totally invested in playing the game, even as they imagine themselves to be the rebels. Once you see it, you can’t unsee it, but it’s lonely looking at the world from this vantage point. 

PS: If you want to better understand my concerns around Bobby’s planned blockchain budget, check out some of my old blog posts. The first goes into Michael Bloomberg’s plans for “what works” data-driven government along with a public presentation I did at Wooden Shoe Books in Philadelphia in the summer of 2019. The second touches on token engineering and participatory “democracy.”

 

 

We have to start to wrap our minds around the idea that the AI automated smart contract law that Shanhan specializes in is actually about running cyberphysical / sociotechnical systems. Sergey Brin’s father Michael, was a specialist on complex dynamical systems. We as agents will knowingly or unknowingly participate in managing the game board for emergent behavior via governance tokens. THESE are the ideas more people need to start educating themselves about and discussing. 

 

Cyberphysical Systems and Choice Link

 

What Works Government Link

 

 


Tokenized Participatory Governance Link

 

 

Link to Moneyball Government Slide Deck – May 2019

 

 

 

Monday, 29. April 2024

Kent Bull

IIW 38 KERI & ACDC Session List

It’s been two weeks since the 38th bi-annual Internet Identity Workshop. The KERI community had a big showing with 19 session across three days. This included a variety of sessions on the vLEI ecosystem, the Trust Over IP (ToIP) Foundation, the Trust Spanning Protocol, KERI basics, CESR 2.0, new Witness […]

It’s been two weeks since the 38th bi-annual Internet Identity Workshop. The KERI community had a big showing with 19 session across three days.

This included a variety of sessions on the vLEI ecosystem, the Trust Over IP (ToIP) Foundation, the Trust Spanning Protocol, KERI basics, CESR 2.0, new Witness infrastructure, and business-focused sessions. A developing tradition of the “Bearer Tokens are bad, don’t use them” session was held this year as well by Sam Smith titled “I don’t sign my credentials and neither should you!”

See the list below for a complete reference of KERI sessions and their meeting notes.

IIW 38 – 19 KERI and ACDC Sessions Day 1 – Apr 16 The Hitchhiker’s Guide To KERI  / Nuttawut Kongsuwan Trust Over IP (ToIP) Foundation / Judith Fleenor Organizational Identity & Verifiable Authority / Timothy Ruff EBA Pilot Using the vLEI / Karla McKenna & Lance Byrd Simple SSI – How to make the SSI Codebase attractive for devs to use? / Jonathan Rayback The Business of SSI & Authentic Data / Timothy Ruff TSP draft Part I / Wenjing Chu Day 2 – Apr 17 KERI for Dummies / Timothy Ruff. Phil Feairheller The 5 Rights of Secure Health Data = a Proof of Concept – KERI-ACDC / Jared J and Phil Feairheller TSP Part II / Wenjing Chu (notes missing/blank) CESR 2.0 Performance features – Deep dive annotation – Comparison to JSON CBOR and more / Sam Smith Day 3 – Apr 18 Deploying Multi Tenant Secure Witnesses for KERI / Phil Feairheller TSP Draft Part III Implementation / Wenjing Chu Running Lean w/SSI The business model to go from Idea to Company / Jared J & Timothy Ruff Five Failed Blockchains – Why Trade Needs Protocols, Not Platforms / Timothy Ruff Cardano Transactions and KERI / Can a cardano Tx be affected based on AID’s current keys? / Ed Eykholt vLEI (verified Legal Entity Identifier) Demystified / Nuttawut Kongsuwan (notes missing/blank) I don’t sign my credentials and neither should you! Why unbound signatures  (tokens) are insecure and how to use KERI instead. KERI for Enterprise / Sam Smith

For more sessions come to the next Internet Identity Workshop in Fall of 2024 from OCTOBER 29, 2024 – OCTOBER 31, 2024.


Patrick Breyer

Leak: EU-Regierungen wollen datenschutzfreundliche und verschlüsselte Messaging-Dienste mit Chatkontrolle bestrafen

Ein neues Dokument zur geplanten Chatkontrolle, das zuerst von netzpolitik.org veröffentlicht und dann auch dem Nachrichtenportal Contexte zugespielt wurde, enthüllt weitere Details über die Pläne der EU-Ratspräsidentschaft: Wenn Dienste …

Ein neues Dokument zur geplanten Chatkontrolle, das zuerst von netzpolitik.org veröffentlicht und dann auch dem Nachrichtenportal Contexte zugespielt wurde, enthüllt weitere Details über die Pläne der EU-Ratspräsidentschaft: Wenn Dienste unter Pseudonymen oder ohne Nutzeraccount, über VPNs oder mit Verschlüsselung genutzt werden, schneiden sie auf einer Risikoskala schlechter ab und sollen eher zur Chatkontrolle verpflichtet werden. Die Akzeptanz von Kryptowährungen zählt ebenso als Risikofaktor. Wenn ein Dienst den „direkten Austausch von Inhalten ohne Verwendung zentraler Server“ (P2P) ermöglicht, wird dies als Risiko eingestuft. Wenn eine datenschutzfreundliche Plattform keine Daten zur Überwachung des Nutzerverhaltens sammeln kann oder will, wird sie schlechter bewertet. Dienste, bei denen die Nutzer „überwiegend öffentlich kommunizieren“ (d.h. nicht über private Chats), sollen besser abschneiden und somit weniger wahrscheinlich zur Chatkontrolle verpflichtet werden.

Der Piratenabgeordnete und digitale Freiheitskämpfer Dr. Patrick Breyer, im EU-Parlament Schattenberichterstatter des Gesetzentwurfs zur Chatkontrolle, kommentiert: „Grundlegende Dienste wie Signal, TOR, verschlüsselte E-Mail-Dienste wie ProtonMail oder Torrenting-Plattformen sollen verteufelt und per Verpflichtung zur Chatkontrolle letztlich zerstört werden. Bie bisher datenschutzfreundlichesten Kommunikationsdienste sollen künftig zu den am stärksten überwachten Diensten werden. Das durchgesickerte Papier offenbart Massenüberwachung und Unterwanderung sicherer Verschlüsselung von Diensten als wahres Ziel der meisten EU-Regierungen. Im Gegensatz dazu will das Europäische Parlament nur Chats von Personen mit Verbindung zu sexuellem Kindesmissbrauch abhören lassen, stattdessen aber viel mehr Sicherheitsmaßnahmen zum Schutz vor der Anbahnung sexueller Kontakte zu Kindern vorschreiben, die der EU-Rat in seinem Papier nur erwähnt. Dieses Leak ist ein Augenöffner. Wir Piraten werden nicht aufhören, für unser Grundrecht auf digitales Briefgeheimnis und sichere Verschlüsselung zu kämpfen!“

Im Mai sind zwei Beratungsrunden über den Vorschlag zur Chatkontrolle geplant, bevor die EU-Innenminister im Juni entscheiden sollen.

Breyers Informationspostal zur Chatkontrolle


Mike Jones: self-issued

Securing Verifiable Credentials using JOSE and COSE is now a W3C Candidate Recommendation

The Securing Verifiable Credentials using JOSE and COSE specification (a.k.a. VC-JOSE-COSE) has reached W3C Candidate Recommendation status. The Candidate Recommendation milestone is described in the W3C Process document. Please review the Candidate Recommendation of VC-JOSE-COSE. Thanks especially to Gabe Cohen, Orie Steele, and Brent Zundel for doing the hard work of getting us to this […]

The Securing Verifiable Credentials using JOSE and COSE specification (a.k.a. VC-JOSE-COSE) has reached W3C Candidate Recommendation status. The Candidate Recommendation milestone is described in the W3C Process document. Please review the Candidate Recommendation of VC-JOSE-COSE. Thanks especially to Gabe Cohen, Orie Steele, and Brent Zundel for doing the hard work of getting us to this point!

Since I last wrote about this work, the W3C Verifiable Credentials Data Model (VCDM), which is also at Candidate Recommendation stage, has been narrowed to only use JSON-LD to represent credentials. VC-JOSE-COSE secures VCDM payloads with JOSE, SD-JWT, or COSE signatures. While I’m admittedly not a fan of JSON-LD, to the extent that Verifiable Credentials using the VCDM are in use, I’m committed to finishing a solid VC-JOSE-COSE specification so there is a simple, secure, standards-based way to sign these credentials.

Of course, there are lots of Verifiable Credential formats to choose from, and more on the way. Choices already existing include ISO mDoc, IETF SD-JWT, IETF JSON Web Proof (JWP), and W3C VCDM. The IETF is also planning to create a CBOR-based selective disclosure representation in the newly formed SPICE working group. It will be interesting to see how these all shake out in the marketplace!

Saturday, 27. April 2024

Werdmüller on Medium

Where I’m coming from

Why I fight for a more inclusive, international perspective. Continue reading on Medium »

Why I fight for a more inclusive, international perspective.

Continue reading on Medium »

Friday, 26. April 2024

Patrick Breyer

Hessens GroKo will flächendeckende Internet-Vorratsdatenspeicherung durch den Bundesrat einführen

Der hessische Ministerpräsident Boris Rhein (CDU) und Justizminister Christian Heinz (CDU) haben einen „Entwurf eines Gesetzes zur Einführung einer Mindestspeicherung von IP-Adressen für die Bekämpfung schwerer Kriminalität“ (PDF) vorgelegt. Am …

Der hessische Ministerpräsident Boris Rhein (CDU) und Justizminister Christian Heinz (CDU) haben einen „Entwurf eines Gesetzes zur Einführung einer Mindestspeicherung von IP-Adressen für die Bekämpfung schwerer Kriminalität“ (PDF) vorgelegt. Am Freitag, 26. April 2024 soll das geplante Gesetz im Bundesrat behandelt werden.

Der Europaabgeordnete Dr. Patrick Breyer (Piratenpartei / Grüne/EFA) kommentiert:

„Dieser Vorstoß ist ein Rückschritt für die Kinderschutz-Debatte. Echter Kinderschutz wird seit Jahrzehnten vernachlässigt – statt für echte Lösungen zu sorgen wird das Problem als Vorwand für eine Internet-Massenüberwachung genutzt, die pauschal alle Bürgerinnen und Bürger unter Generalverdacht stellen würde. Der vorgelegte Gesetzentwurf ignoriert nicht nur dringend notwendige Maßnahmen für echten Kinderschutz, sondern auch den Fakt, dass es keinen messbaren Nutzen einer solchen Massenüberwachung gibt. Anstatt echte Lösungen auszuarbeiten, wollen Ministerpräsident Boris Rhein (CDU) und Justizminister Christian Heinz (CDU) offenbar das Rad zurückdrehen:

Vor gerade einmal zwei Wochen hat die Ampel anstelle anlassloser IP-Speicherung den Weg frei gemacht für das anlassbezogene und zielgerichtete Quick Freeze-Verfahren, wie es beispielsweise auch in Österreich eingesetzt wird. Der hessische Vorschlag hat das Potenzial die Einführung eines verhältnismäßigen Quick Freeze-Instruments aufzuhalten.

Rückschrittlich ist der Entwurf auch, weil das Bundesverwaltungsgericht erst 2023 die von CDU und SPD beschlossene Vorratsdatenspeicherung in vollem Umfang für unionsrechtswidrig und damit für nicht anwendbar erklärt hat – inklusive der IP-Vorratsdatenspeicherung.

IP-Vorratsdatenspeicherung ist, wie wenn jede:r Bürger:in ein sichtbares Kennzeichen um den Hals gehängt bekäme und dieses auf Schritt und Tritt notiert würde. Niemand würde sich eine solche Totalerfassung des täglichen Lebens gefallen lassen. IP-Vorratsdatenspeicherung würde jeden Internetnutzer unter Generalverdacht stellen und die Internetnutzung der gesamten Bevölkerung, die unsere intimsten Vorlieben und Schwächen abbildet, nachvollziehbar machen. Eine so totale Erfassung würde Kriminalitätsvorbeugung durch anonyme Beratung und Seelsorge, Opferhilfe durch anonyme Selbsthilfeforen und auch die freie Presse gefährden, die auf anonyme Informanten angewiesen ist. 99,99 Prozent dieser Daten wären völlig nutzlos, da sie Bürger:innen betreffen, die nie auch nur in den Verdacht einer Straftat kommen.“


Just a Theory

Mini Summit Four

Links, notes, and commentary from Jonathan Katz’s presentation at the fourth Postgres Extension Ecosystem Mini-Summit, “Trusted Language Extensions for PostgreSQL”.

My thanks to Jonathan Katz for his presentation, “Trusted Language Extensions for PostgreSQL”, at last week’s Postgres Extension Ecosystem Mini-Summit. As usual I’ve collected the transcript here interspersed with comments from the chat window. First, links!

Video PDF Slides [TBD]

And now, rivers of text!

Introduction I opened the meeting and introduced Jonathan Katz. Presentation

Thank you for having me. I’m very excited to talk about this, and extensions in general. I see a lot of folks here I know and recognize and some new folks or folks I’ve not met in person yet.

Borrowed from the original presentation on TLEs from November of 2022, to level set on why we built it. I know this is a slightly more advanced audience, so some stuff that might seem relatively introductory to some of you, though there is some material on the internals of extensions.

The premise is why we decided to build TLEs, what were the problems we’re trying to solve. Part of it is just understanding Postgres extensions. In general this group is very familiar with Extensions but there are two points I want to hit on.

One of the things that excites me most about Postgres is that, when you look back at Postgres as the Berkeley database project researching how to create an object relational database, an accidental or intentional features is not just that Postgres is an object-relational database, but that Postgres is an extensible database, built from the get-go to be able to add functionality without necessarily having to fork it.

Early on you’d have to Fork the database to add additional functionality, but the beauty of the Postgres design was the ability to keep adding functionality without forking.

It did require making changes to Postgres to further enhance that capability, which we’ll discuss in a bit, but that’s a really powerful concept.

The second point is that there is the large landscape of both open source and Commercial extensions. I think a lot of folks in this group are very familiar with the available open source extensions, but there are entire businesses built on, effectively, commercial extensions on top of Postgres. Again, that’s a really powerful notion!

It’s kind of like the Postgres economy: we created something that’s so special that it’s able to spawn all these different businesses whether it’s building things on top of Postgres or using Postgres as the heart of their business. Extensions have a very large role to to to play in that.

Which gets us to the history of extensions. The Syntax for Postgres extensions has been around for over a decade, since Postgres 9.1. I know there’s folks here well familiar with building extensions prior to that syntax! But we’re very grateful for the extension syntax because, as a recovering application developer, I would say it made things a lot easier.

Instead of having to futz around the file system to figure out where to install everything (wink wink nudge nudge on the topic today), you had a very simple syntax. Once an extension is installed, I can CREATE EXTENSION postgis (or whatever extension) and there you go! I have PostGIS installed.

Again, that’s really cool! Because anything we can do to make it simpler to install and use extensions further drives their adoption, and ultimately makes it even easier to develop and build applications with Postgres and continues to drive that forward.

So what can you build with Postgres, what extensions are available? It’s a whole range of things. For starters, there are extensions that I call “the ones that you take for granted”. If you’re using any monitoring tool you’re likely running and you may not even realize it. pg_stat_statements helps to aggregate statistics as queries execute and bubbles it up to whatever monitoring tool you use. It’s a great tool for performance tuning.

The example I like to give for my personal life was that, back when I was an application developer trying to manage my own databases, I had some query that was blocking my entire logical decoding system, so we weren’t keeping up with transactions. Looking in pg_stat_statements I see a recursive query where I should have had a UNION with SELECT DISTINCT instead of a gigantic query that was amassing so many rows. Fixed it: I had a 100x speed up in the query. Thank you pg_stat_statements!

Side note: I got to say, “hey, I sped up this query by 100x,” even though it was my fault it was slow to begin with.

There are utility functions to help with data types. UID OSSP is very widely used. Newer versions of Postgres have a random UUID function, but previously, anytime you needed a UUI you would CREATE EXTENSION "uuid-ossp".

The other fun thing about this extension is that developers learned about SQL identifiers that require double quotes to install the extension.

I think I saw Keith on here today. pg_partman! What’s really cool about pg_partman too is that a lot of it is PL/pgSQL. This PL/pgSQL code provides a way to manage partitions across all your tables in your entire database. Again, that’s really powerful because Postgres has added declarative partitioning in version 10, but pg_partman is still incredibly useful because there are all sorts of aspects to partition management not supported in Postgres today. This is another example where Postgres provides the core functionality and you can use the extension to package additional functionality that makes it easier for use.

Foreign data wrappers.Postgres has a whole interface to interface with other databases. It could be other Postgres databases, other relational databases, non-relational databases, file systems, etc. Postgres has a C-level interface that allows you to design the calls to optimally use all these different databases. Again, packaged up as an extension, being able to add things as we go on.

I’ll wait till the end to answer questions this will be a relatively short presentation, so we should have some time for discussion

Last but not least, a little bit on PostGIS. I think this is one of the most powerful aspects of Postgres. PostGIS itself is incredibly powerful because you have a geospatial database that happens to be Postgres underneath. A lot of heavy PostGIS users don’t even realize they’re using Postgres! They think they’re using PostGIS. That is really the power of Postgres extensibility in a nutshell: It looks like you have like a brand new, domain-specific database – and yet underneath it it’s just boring old Postgres doing all the things you expect a database to do. That is also a very powerful notion/

Tobias Bussmann in chat: Many PostGIS users don’t know they are using PostgreSQL 🤣

nils in chat: 🎉

To add a coda to it, you have pgRouting, an extension built on top of PostGIS, which is built on top of Postgres. So you have a cascading effect of extensions building on top of extensions building on top of Postgres.

So we’re supposed to talk about trusted language extensions. To really to TLEs it’s important to understand the anatomy of an extension. There are certain things that you need in order to have an extension: You need a control file, which is kind of like your packaging manifest. it tells you what’s in the extension. It goes into a directory.

You have SQL files, which effectively map out the objects that you’re going to have in your database. If you have functions that need to map to a C function or if you need to create a table access method in order to build your new your new storage layer, the SQL files are the building block.

If you have C- specific code, it goes in a library file or a shared object file that gets stored in a library directory.

It’s a very simple layout. What’s cool is if you go to create an extension, there’s a particular pattern that it forms: You need to know that when you have an extension, the information in the control file goes into the Postgres catalog. Then, if there are any functions or objects or whatever in that extension, we need to install the object itself, but we also need to make sure that there’s a dependency on the extension. That way, if we need to remove the extension or upgrade it, we know all the objects that we’ve collected.

So why this “extension building 101”? This gets at the heart of why we built TLes. Because the first thing to notice is that we install extensions directly on the file system. There are a lot of domains where that’s restricted — certainly managed service providers.

I worked a lot with containers previously, and a container is effectively an immutable file system: once you have things installed on it it’s installed. You typically don’t want to give your app developers access to your production systems, because your app developers are going to install everything under the sun on them, myself included. You certainly want to be able to restrict certain domains.

But we also don’t want to inhibit what developers want to build. We want to make it as easy as possible for them to manage their code and be able to install in different environments. That gets to another point beyond restricting the file system. Some extensions may not be universally available, depending on where you’re running them. You might be running an extension on an on-premise environment that might not work in a managed service provider. Or different managed service providers have different extensions available.

The final bit — and something that I’ve definitely personally experienced — is that, between major versions of Postgres, the API or the ABI will change. These are the interface points at the C layer. When they change it can break extensions. Ask any of the package managers how much they need to nag people to upgrade their extensions: they always want to make sure that they’re keeping it up-to-date, and ensuring that it’s compatible.

But this can also lead to other issues, because as the end user, this makes it challenging to perform major version upgrades — particularly if I’m dependent on an extension that hasn’t been updated to work with the latest version of Postgres. A subtle line of code change in Postgres could end up breaking an extension.

Quick story: that actually happened to me while I was managing pg_tle. I think it was a change in Postgres 15.1 actually broke something in the pg_tle extension. I had to to fix it. I think that’s part of the point: if you’re able to use a language that’s on top of C, and you have the C layer abstracted away, in theory it could make it easier to perform major version upgrades.

That leads into TLE.

I think there were two Notions behind trusted language extensions, or TLEs, when they were initially designed. The first is giving power to app developers to be able to build extensions. It’s actually one thing I noticed as I started making the journey from app developer to quasi-DBA to, ultimately, product manager not writing code. Part of that Journey was the power of putting some of my business logic in the database.

There’s always this tension between how much business logic to put in application code versus the database. But there are certain things that were just clear wins for me when they wer in the database. The first was a lot of search functions I wrote where filtering data down to a very small set in the database and returning to the application would save on network time, processing time on the app side, etc. There were some very clear wins by encapsulating them in functions.

But also solving things that were just much easier to solve in the database. Having specific data types that solve a particular problem — geospatial extensions keep coming to mind, pgvector dealing with Vector data, and being able to store it in a database without having delegate it out into an application certainly is a is a clear win.

The other thing was installing extensions. I think the notion of portability is very powerful. If I have a way to manage my extensions from a SQL interface, it makes it much easier to move it between different systems.

Now, we do need to be careful as soon as we start saying “SQL interface”. I don’t want to suggest that we should have a SQL interface to ship C code everywhere We know there are some challenges with C code. C is highly performant, you can effectively build anything under the sun using C, but it’s not memory-safe, and it’s very easy if you’re not familiar with what you’re doing — and even if you are familiar with what you’re doing! — you can easily make mistakes that could either lead to crashes or or possibly worse.

As we were thinking about all this with TLE, there’s three things. First, we need an interface to be able to install and manage extension code effectively regardless of environment. We need a SQL interface to do that. We also need to make sure there’s an appropriate trust boundary. Now, Postgres provides a trust boundary with the notion of a trusted language. But there are other things we need to build with trust, as well.

For example, you might not want everyone in your environment to be be able to install the a TLE, so we need to make sure there’s appropriate access controls there.

Finally, we need a way to package it up — which I think is what we’re going to talk about at the Extension Summit.

If there are any takeaways from why we built TLE (I think this is the the slide that encapsulates it), it’s that, by using by using Postgres’s built-in trusted language interface it allows you to write extension code in languages that we know are going to respect Postgres security boundaries.

Postgres has this definition of a trusted language which, if you look at for it, you have to effectively dance around the documentation to find it.

But effectively I’d summarize as, if you allow an unprivileged user to write code in a trusted language, they can’t do anything to escalate their privileges, access the file system directly, or do anything that would violate Postgres’s security boundary.

It’s a pretty good definition. Arguably, the easiest way to violate that definition is that you as the database administrator mark an untrusted language as trusted in the catalog. But I strongly advise to not do that!

What trusted languages are available. There’s a great wiki page called the “PL Matrix” on the Postgres Wiki that shows the status of all all the known PLs in Postgres and whether they’re trusted or not. I suggest looking at that.

David Wheeler (he/him) in chat: https://wiki.postgresql.org/wiki/PL_Matrix

Jeremy S in chat: Linux had kernel modules forever, but many people were rightfully hesitant because a kernel module could easily crash your entire system. One of the reasons eBPF is exploding today is because it’s verified and safe and enables code (like entire networking apps) to run directly in the linux kernel.

I see TLE similarly

A big thing regarding a trusted language is performance. There are a variety of trusted languages, and they all have different properties you know around them. The ones I’m showing today are the ones available in RDS Postgres. But the reason I want to show them is that, part of the idea of trusted language extensions is allowing app developers who may be less familiar with C to write extension code and access some of the same internals as a C extension, but from one of these languages.

Here are some of the known trusted languages today that work with TLE. If you’re using the TlE open source project, you can use any available trusted language — or you can use untrusted languages an just use the TLE packaging mechanism. In that case you lose the trusted part, but gain the extension installation aspect of TLE.

There are a few things included in TLE to make sure that TLE can be installed safely. It is an opt-in feature. We do have a shared preload library for pg_tle called “pg underscore TLE”, and you do need to have your database super user install pg_le initially. This ensures that we’re respecting your security boundary, If you’re going to use trusted language extensions, you do have an explicit opt-in to using it.

David Wheeler (he/him) in chat: https://github.com/aws/pg_tle

After that, an app developer can create their own trusted language extension.

Here’s a simple example from the TlE announcement with two functions and packaged into an extension you can install. You can give it a name like any Postgres extension; this one is called “tle_test”. The code looks like the SQL file in any extension. And it’s effectively packaged up like an extension using the pgtle.install_extension command. If you go to the pg_le GitHub project you can see the the different interface points.

Once it’s installed you can use CREATE EXTENSION like any other extension: it follows all the usual Postgres semantics: extension installation, uninstallation, software life cycle management. pg_tle has its own interface for that functionality, bu once you’ve installed it, managing the extension is just like managing any other Postgres extension, and follows those conventions.

Effectively TLE is offering, loosely, a packaging mechanism (I think packaging has a lot more connotations): it’s a grouping mechanism for your code. One of the parts that I always found most useful in pg_tle was this part, effectively versioning my store procedures.

When I talked about the example where I was putting business logic into the database, one part I would usually mess up is: what version of my stored procedures was running in a given database. Particularly if you have hundreds or thousands of databases that you’re managing, that can be a challenge.

Now I had far fewer databases I was managing, I was more dealing with our developer environments: staging and production. But I was managing the store procedures within our migration scripts — which is totally fine, because if I know what version of the migration that I ran then I would know what version of the stored procedures are on that database. Kind of. Sort of. Maybe. You know: unless someone manually modified it — in which case shame on me for giving access to the database. But there basically could be some gaps in knowing what version of a stored procedure was on a particular server.

With pg_le we can significantly reduce the risk of that problem because we have a way to version our store procedures, and be able to know exactly what we’re running at any given time, and create a consistent packaging mechanism wherever we’re running our code. And it goes beyond stored procedures because there’s far more that you can can build with your code.

What else does TLE add? We discussed was the packaging mechanism, but a lot of the power of Postgres extensions is the ability to use the underlying Postgres internals. One of these types of internals is called a “hook”.

Hooks are the Postgres feature that you’ve never heard of, that are not well documented, and yet are the foundational part of many extensions. Hooks are almost everywhere in Postgres. You particularly see a lot of them during the query execution process. For example the process utility hook which allows you to modify any utility command, anything that’s not a direct SQL statement. There are all sorts of hooks: there are password check hooks, client authentication hooks, hooks called around shared memory allocation, hooks called at each step of the the execution phase.

Florents Tselai in chat: Hooks are almost undocumented indeed

The best resources I’ve found:

https://github.com/taminomara/psql-hooks and the standard: https://wiki.postgresql.org/images/e/e3/Hooks_in_postgresql.pdf

Hooks are very powerful; particularly enabling a lot of extensions adding different semantic behavior to Postgres. We could probably do a whole series of talks just on all the different ways you can extend Postgres. I mean, that’s why David has organized the summit! But hooks are very simply a powerful mechanism to define behavior and Postgres.

Because they’re so powerful, for the hooks that we expose in tle we make sure that there is a super user opt-in. Remember, an unprivileged user can define this behavior but you do need someone with privilege to be able to enable something like a hook.

For example, a password check hook probably means that you have the ability to evaluate a plain text password that’s coming through. Now on that topic we can have a very long debate, but let’s save that for Vancouver. But with this hook, you do have the ability to do password checks, so you want to make sure that, when you enable a function that calling a password check hook that there’s a certain level of privilege to that function. Or you you know you want to make sure you do your appropriate evaluation to make sure that you trust that function.

In addition to that check, there’s an additional check from the pg_tle admin role that requires someone with administrative privileges over your TLE to register that hook. The concept of “TLE features” are the way to map hooks into the TLE. We’ve been building it up since we launched TLE by adding a few hooks. There’s both the check password hook and the client authentication hook.

There’s also the ability to register custom data types — which is pretty cool, because data types are what attracted me to Postgres when I was an app developer: “Oh! There are all these data types! I can do all these rich comparisons against an index? Cool! Oh wait, you can even add custom data types? That’s even cooler!”

TLE allows you to create the base data type, so you can really expand the data types that you’re able to add. This is what TLE features does: it enables that safe mapping between trusted language code and the Postgres C internals.

In order to create a hook, you need to match the hook function definition. The TLE documentation documents how to create it appropriately, but it doesn’t need all the parameters that you would find in the hook function.

In this check password hook — I call this the “delay check password test”, meaning you’re probably trying to avoid someone trying to guess your password repeatedly, and if they keep failing so what, because they’re not going to brute force it anyway. There are actually more practical examples of check password hooks. But what’s cool is that you can define everything around the your hook behavior from within the hook function and then it acts as if you wrote a C-based hook! You just happen to write it in a in a trusted language.

Hooks do execute with elevated privileges, particularly around authentication you want to be very careful. So there are some safeguards built into TLE to make sure that you only enable hooks when you want to.

Last but not least: choosing a trusted language. I know this group is more focused on extension building, but I do want to talk about what an app developer goes through when choosing a trusted language.

Because everything has its trade-offs to consider. The Golden Rule (I actually took this from Jim Mlodgensky) is: when in doubt use PL/pgSQL, because it does have a lot of access to context that’s already available in Postgres. What’s interesting about this is that what we see today is based on PL/SQL. PL/pgSQL was developed to try to make it simpler to migrate from Oracle, but at the same time to provide a lot of rich functionality around Postgres.

As someone much more familiar with Ruby and Python, I can tell you that PL/pgSQL can be a little bit quirky. But it is very well documented, and it can solve all the problems that you need to in Postgres. And it already has a lot of very simple ways to directly access your data from Postgres. Certainly an easy choice to go with.

But wait, there’s more!

like PL/v8, writing JavaScript in your database, this is really cool! I remember when it came out and how mind-blowing it was, in particular for JSON processing. PL/v8 is awesome. PL/v8 came out right around the same time as the document database! So you kind of had perfect storm of being able to process JSON and write it in JavaScript — both within your Postgres database and it could be quite powerful.

Another really cool feature of PL/v8 is the ability to directly call another function or another PL/v8 function from within PL/v8, and not have to go through Postgres function processing, which adds a lot of additional overhead.

And now the one that’s all abuzz right now: PL/Rust. Being able to write and execute Rust code within Postgres. This is pretty cool, because Rust is a compiled language! There’s a trusted way to run PL/Rust within Postgres. There are a few techniques to do it. First, whenever you’re running Rust on your server, to make sure that you’re guarding against breakouts.

There is a library, I believe it called postgres FTD, that effectively compiles out some of the less dressed parts of Rust, such as unsafe function calls. But you can still get everything that you want in PL/Rust today: you get the Rust standard Library, the ability to run crates — and you do want to evaluate crates to make sure that you’re comfortable running them in your environment. But then you get this compiled language that is CPU efficient, memory efficient, and memory safe. (Well, a lot of Rust is memory safe) It’s pretty cool!

Steven Miller in chat: In PL/Rust, does it run the compilation when the function is created? Then if there is a compiler issue it just shows up right there?

I wrote a blog post last year that compared some different function calls between PL/pgSQL, PL/v8, and PL/Rust. First I was doing some array processing, and you could see that the Pl/Rust calls were very comparable to the C calls. And then there’s some additional Vector processing, given that I’ve been obsessing on vectors for the past 14 months. Seeing rust actually win against PL/pgSQL and PL/v8 (I don’t remember the numbers off the top of my head I can look up that blog as soon as I switch windows). Pretty cool!

This brings us in some ways to the best of all worlds, because I can take an extension that normally I would write in C, particularly because I’m focused on performance, I can write it in PL/Rust, package it as a trusted language extension, and run it anywhere that TLE and PL/Rust are supported. Again, that is very powerful, because suddenly I have what I hope is the best of all worlds: I have this portability, I don’t have to worry as much about major version upgrades because pg_le is acting as that abstraction layer between the Postgres C code and the application code that I’m writing.

Jeremy S in chat: Versioning of stored procedures is a very interesting use case

Darren Baldwin in chat: Agreed! Moving stuff to the database layer seems to be something very foreign and “scary” to most app devs I’ve talked to

Anup Sharma in chat: Is TLE a requirement for any PostgreSQL extension, or is it dependent?

Steven Miller in chat: So during a major version upgrade, the function declaration stays the same, so that’s why your application doesn’t need to change with respect to the extensions during a major version upgrade. And at some point during the migration, you create the function again, which recompiles. So it all works the same! That’s great

Last slide, then I’m certainly looking forward to discussion. pg_tle is open source, and it’s open source for a lot of reasons. A lot of it is because we want to make sure that trusted language extension are as portable as possible. But in some ways the ideas behind TLE are not original. If you look at other databases there is this notion of, let’s call it inline extensions, or inline SQL, ou call them modules, you call them packages. But the idea is that I can take reusable chunks of code, package them together, and have them run anywhere. It doesn’t matter where the database is located or hosted.

This is something that I personally want to work with folks on figuring out how we can make this possible in Postgres. Because even in Postgres this is not an original idea. Dimitri Fontaine was talking about this as far back as 2012 in terms of his vision of where of the extension framework was going.

What I’m looking forward to about this Extension Summit — and hopefully and hopefully I’m not in conflicting meetings while it’s going on — is talking about how we can allow app developers to leverage all the great parts of Postgres around function writing, function building, and ultimately packaging these functions, and making it simple simpler for them to be able to move it wherever their applications are running.

So it is open source, open to feedback, under active development, continue to add more features to support Postgres. Iltimately we want to hear what’ll make it easier for extension writers to be able to use TLE, both as a packaging mechanism and as a as a development mechanism.

So with that uh I that is the end of my slides and happy to uh get into a discussion about this.

Discussion

David Wheeler (he/him): Awesome, thank you Jonathan. there was one question about PL/Rust in the comments. Stephen asks whether it compiles when you create the function, so if there are compiler issues they they show up there.

Jonathan Katz: Correct It compiles when you create the function and that’s where you’ll get compile errors. I have definitely received my fair share of those [chuckles]. There is a Discord. PL/Rust is developed principally by the folks uh responsible for the pgrx project, the folks at ZomboDB, and they were super helpful and debugging all of my really poor Rust code.

David Wheeler (he/him): While while people are thinking about the questions I’ll just jump in here. You mentioned using crates with PL/Rust. How does that work with pg_le since they have to be loaded from somewhere?

Jonathan Katz: That’s a good question. I kind of call it shifting the problem. TLE solves one problem in that you don’t need to necessarily have everything installed on your on your local file system outside of pg_tle itself. If you’re using PL/Rust and you need crates, you do need those crates available either within your file system or within whatever package management tools you’re using. So it shifts the problem. I think it’s going to be a good discussion, about what we can do to help ensure that there is a trusted way of loading those.

David Wheeler (he/him): Yeah I wonder if they could be vendored and then just included in the upload through the function call.

Anup Sharma asked asked if pg_tle s a requirement any extension or is it dependent.

Jonathan Katz: It’s not requirement. This is a project that is making it possible to write Postgres extensions in trusted languages. There ar plenty of extension authors on this call who have written very, very, very good extensions in C that do not use TLE.

David Wheeler (he/him): You can use trusted languages to write extensions without TLE as well. It’s just a way of getting it into the database without access to the file system, right?

Jonathan Katz: Correct. I think I saw Keith here. pg_partman is PL/pgSQL.

Anup Sharma in chat: Understood. Thanks

Tobias Bussmann in chat: I think it is important not to confuse Trusted Language Extensions TLE with “trusted extensions” which is a feature of Postgres 13

Keith Fiske in chat: Pretty much all of it is. Just the background worker isn’t

Jonathan Katz in chat: hat’s what I thought but didn’t want to misspeak 🙂

David Wheeler (he/him): Right Any other questions or comments or any implications that you’re thinking about through for extension distribution, extension packaging, extension development?

Steven Miller in chat: Is background worker the main thing that a TLE could not do in comparison to traditional extensions?

Jason Petersen: The crates thing kind of raised my interest. I don’t know if Python has this ability to bring in libraries, or if JavaScript has those dependencies as well. But has there been any thought within pg_tle for first classing the idea of having a local subdirectory or a local file system layout for the “native” dependencies? I’m using “native” in quotes here because it could be JavaScript, it could be Python, whatever of those languages, so they could be installed in a way that’s not operating system independent.

I know this is kind of a complex setup, but what I’m getting at is that a lot of times you’ll see someone say “you need to install this package which is called this and Red Hat or this on Mac or this on Debian — and then you can install my extension. Has there been any push towards solving that problem by having your TLE extensions load things from like a a sort of Walled Garden that you set up or something? So it’s specific to the database instead of the OS?

Jonathan Katz: That’s a good question. There has been thought around this. I think this is going to be probably something that requires a thorough discussion in Vancouver. Because if you look at the trusted languages that exist in Postgres today, the definition of trusted language is: thou shall not access the file system. But if you look at all these different languages, they all have external dependencies in some in some way shape or form. Through Perl there’s everything in CPAN; through JavaScript there’s everything in npm. Let’s say installed the appropriate CPAN libs and npm libs within uh your database for everything I recall from playing with trusted PL/v8 and PL/Perl is that you still can’t access those libraries. You can’t make the include or the require call to get them.

Where PL/Rust is unique is that first off we just said, “yes, you can use your Cargo crates here.” But I think that also requires some more thinking in terms of like how we make that available, if it’s OS specific, vendor specific, or if there’s something universal that we can build that helps to make that more of a trusted piece. Because I think at the end of the day, we still want to give the administrative discretion in terms of what they ultimately install.

With the trusted language extensions themselves, we’re able to say, “here’s the post security boundary, we’re operating within that security boundary.” As soon as we start introducing additional dependencies, effectively that becomes a judgment call: are those dependencies going to operate within that security boundary or not. We need to be make sure that administrators still have the ability to to make that choice.

I think there are some very good discussion topics around this, not just for something like PL/Rust but extension distribution in general I think that is you know one of the I think that’ll be one of the key discussions at the Extension Summit.

David Wheeler (he/him) in chat: What if the required modules/packages/whatever were in a table. e.g. in Perl I do use Foo::Bar and it has a hook to load a record with the ID Foo::Bar from a table

David G. Johnson: Has there been any thought to having the default version of an extension tied to the version of PostgreSQL? Instead of it just being 1.3 and, whether I’m on version 12 or 15, because 1.3 might not even work on version 12 but it would work on version 15. The versioning of the an extension and the versioning of PostgreSQL seem like they’re almost too independent.

Jonathan Katz: So David, I think what you need to do is chastise the extension developers to let them know they should be versioning appropriately to to the the version of Postgres that they’re using. [Chuckles]

There is a good point in there, though. There is a lot of freedom in terms of how folks can build extensions. For example, just top of mind, pgvector supports all the supported versions of Postgres. Version 0.7.0 is going to be coming out soon so it’s able to say, “pgvector 0.7.0 works with these versions.” Dumb. PG plan meanwhile maintains several back releases; I think 1.6.0 is the latest release and it only supports Postgres 16. I don’t believe it supports the earlier versions (I have to double check), but there’s effectively things of that nature.

And then there aer all sorts of different things out there, like PostGIS has its own life cycles. So there’s something good in that and maybe the answer is that becomes part of the control file, saying what versions ov Postgres an extension is compatible with. That way we’re not necessarily doing something to break some environment. I’m just brainstorming on on live TV.

David G. Johnson: The other day I open a but report on this. but PostgreSQL dump and restore will dump it without the version that’s in the source database, and when yoq restore it, it’s going to restore to whatever the current version for the control file is even if you’re upgrading to a different database. versus restoring it to whatever the original version was. That dynamic just seemed problematic.

David Wheeler (he/him): I think it’s less problematic for trusted language extensions or extensions that have no C code in them, because pg_dump does dump the extension, so you should be able to load it up. I assume base backup and the others do the same thing.

David G. Johnson: I haven’t checked into that. It dumps CREATE EXTENSION and then it dump any user tables that are marked by the extension. So these code tables are marked as being user tables for TLE?

David Wheeler (he/him): What do you mean by code tables?

Regina Obe: That’s a good point. For example my Tiger geocoder is all PL/pgSQL, but it’s only the CREATE EXTENSION thing that’s named. So for your TLE table, it would try to reload it from the original source, wouldn’t it? In which case it would be the wrong version.

Jonathan Katz: We had to add some things into TLE to make sure it worked appropriately with pg_dump. Like I know for a fact that if you dump and load the extension it works it works fine. Of it doesn’t then there’s a bug and we need to fix it.

David G. Johnson: Okay yeah I haven’t played with this. Literally this is new to me for the most part. I found the whole fact that the control file is not updated when you do ALTER EXTENSION to be, at least in my mind, buggy.

Jonathan Katz: In the case of TLE, because it’s in theory major version-agnostic. When I say “in theory,” it’s because we need to make sure the TLE code in library itself is able to work with every major version. But once that’s abstracted away the TLEs themselves can just be dumped and reloaded into different versions of Postgres. I think we I we have a TAP test for that, I have to double check. But major version upgrades was something we 100% tested for

David Wheeler (he/him): I assume it’d be easier with pg_tle since there’s no need to make sure the extension is is installed on the file system of the new server.

Jonathan Katz: Yep. if you look at the internals for pg_tle, effectively the TLEs themselves are in a table. When you do a CREATE EXTENSION it gets loaded from that particular table.

David G. Johnson: Right, and when you do a pg_dump you make suer that table was dumped to the dump file.

Jonathan Katz: Yes. But this is a key thing that we we had to make sure would does work: When loading in a pg_dump, a lot of the CREATE EXTENSIONS get called before the table. So we need to make sure that we created the appropriate dependency so that we load the TLE data before the CREATE EXTENSION. Or the CREATE EXTENSION for the TLE itself.

Jeremy S in chat, replying to “Is background worker the main…”: doing a background worker today, I think requires working in C, and I don’t think core PG exposes this yet. Maybe it could be possible to create a way to register with a hook to a rust procedure or something, but maybe a better way in many cases is using pg_cron

Jonathan Katz in chat: We can add support for BGWs via the TLE API; it’s just not present currently.

nils in chat: Creative thinking, if a background worker doesn’t work in TLE, how about create your UDF in tle and schedule with pg_cron 🤡

David Wheeler (he/him): You mentioned in the comments that you think that background workers could be added. How would that work?

Jonathan Katz: It would be similar to the the other things that we’ve added, the data types and the hooks. It’s effectively creating the interface between the C API and what we’d expose as part of the TLE API. It’s similar to things like pgrx, where it’s binding to Postgres C API but it’s exposing it through a Rust API. We do something similar with the TLE API.

Steven Miller in chat: Thank you Jeremy. I like this idea to use TLE, then depend on cron for a recurring function call

Steven Miller in chat: Ah yes Nils same idea 😄

Jason Petersen in chat: Thumbs up to nils about pgcron. If you need a recurring BGW just write it in plpgsql and schedule it

nils in chat: Great hackers think alike

Jason Petersen in chat: (I know I do this)

David Wheeler (he/him): That that makes sense. I just thought the background workers were literally applications that are started when the postmaster starts up shut down when the postmaster shuts down.

Jonathan Katz: But there’s dynamic background workers.

David Wheeler (he/him): Oh, okay.

Jonathan Katz: That’s how a parallel query works.

Jeremy S in chat: Threading? 😰

David Wheeler (he/him): Gotcha, okay. Sorry my information’s out of date. [chuckles]

Jonathan Katz: Well maybe one day we’ll have you know some some form of threading, too. I don’t think like we’ll get a wholesale replacement with threads, but I think there are certain areas where threads would help and certain areas workers are the way to go/

David Wheeler (he/him): Yeah, yeah that makes sense.

Jonathan Katz: Hot take!

David Wheeler (he/him): What other questions do you have for about TLEs or extensions more broadly and packaging in relation to TLEs?

David G. Johnson: Just a random thought: Have you thought about incorporating foreign servers and pointing the TLE, instead of a local database, point it to a master, company-wide foreign table?

David Wheeler (he/him): Like a TLE registry?

David G. Johnson: Right, yeah something global would be nice. like okay we hosted on PGXN at there’s a TLE registry. But because for a company who wants maintain code internally between projects, and they want a shared library, they can publish it on one server, send up a link to it over foreign server, and then just point at that.

Jonathan Katz: Could be!

David Wheeler (he/him): I mean you could just use foreign foreign tables for that for the tables that TLE uses for its its registry, right?

David G. Johnson: That’s I’m thinking.

David Wheeler (he/him): Yeah that’s a cute idea.

Jonathan Katz: I think that just to to go back a few more minutes. I think you I was asked to talk about the vision. One one way to view extensions is trying things out before they’re in core, or before they’re in Postgres. The aspect that I would ultimately like to see in core someday is the ability to do that’s called “inline modules.” There is a SQL standard syntax, CREATE MODULE, that for this purpose. Some folks were trying to see see if we could get it into, I believe, Postgres 15. There was some push back on the design and it died on the vine for the time being.

But I do think it’s something to consider because when I talk to folks, whether it’s random Postgres users RDS customers, etc., and I go through TLE, one of the things that really stands out is one of the things that we had discussed here and I saw in the chat, which is this aspect: being able to version your stored procedures. This is in part what modules aims to solve. One is just having a SQL interface to load all these things and group it together. But then once you have that grouping you have the ability to version it. This is the part that’s very powerful. As soon as I saw this I was like, “man I could have used that that would have saved me like hours of debugging code in production.” Mot saying that I was ever sloppy and you know in random store procedures in my production database!

David Wheeler (he/him) in chat: I see CREATE MODULE in the db2 docs.

Jonathan Katz: But that’s kind of the vision. The fact that Postgres is extensible has led to this very widely adopted database. But I think there are things that we can also learn in our extensions and bring back upstream. There are certainly reasons why they we developing things in extensions! Like pgvector is an example of that, where we talked about it at PGCon last year. And part of the thought of not trying to add a vector data type to Postgres was, first, to make sure we could settle on what the the binary format would be; and once that’s solidified, then we could add it.

But I had an aside with Tom [Lane] where we talked about the fact that this is something we need to move fast on, the vector space is moving very quickly, extensions are a way to be able to move quickly when something like Postgres moves more deliberately.

This is in some ways where TLE is, our way to be able to see what kind of interface makes sense for being able to do inline extension loading and ultimately how we want that to look in core.

David Wheeler (he/him): Can you create data types with a binary representation in TLE?

Jonathan Katz: Yes as of (I want to say) the the 1.3 release. I have to double check the version. The way we’re able to do it safely is that it actually leverages the BYTEA type. When you create that representation it stores it as a BYTEA. What you get for free today is that, if you create your equality/inequality operators, you can use a b-tree look up on these data types.

So there’s a “dot dot dot” there. If we wanted to be able to use like GIST in GIN and build data types for our other index interfaces, there’s more work to be done. That would require a TLE interface. I spent a lot of time playing with GIST and GIN, and the interface calls themselves involve pointers. So that will require some thought yeah.

David Wheeler (he/him): I assume it’s a similar issue for Rust data types that are basically just serde-serialized.

Jonathan Katz: Yeah we can at least like store things in BYTEA, and that’s half the battle. It allows us to do a safe representation on disk as opposed just “here’s some random binary; good luck and don’t crash the database!”

Jason Petersen in chat: I also wondered about the function interface for things like storage features (table access methods).

I assume they’re similarly hairy

David Wheeler (he/him): Any other last minute questions?

Jonathan Katz: Table access methods. Yes table access methods are very hairy as are index access methods. I spent a lot of time the past 14 months looking at the index access method interface, which has a lot of brilliance in it, and certainly some more areas to develop. But it’s amazing! The fact that we can implement vector indexes and get all the best parts of Postgres is a phenomenal advantage.

Jeremy S: One last question. We’re leading up to Vancouver and we’re going to be starting to think about some of the topics that we want to make sure to talk about at the Summit. I think you mentioned one earlier (I should have written it down), but any final thoughts about topics that we should make sure to discuss?

Jonathan Katz: Just in general or TLE specific?

Jeremy S: Both. I mean for sure TLE-specific, but also just generally related to extensions

Jonathan Katz: My TLE-specific one dovetails into the general one. The first one is: is there ultimately a path forward to having some kind of inline extension management mechanism in core Postgres. That’s the top, part one, I spent the past five minutes talking about that.

But I think the big thing, and why we’re all here today, is how do we make it easier for developers to install extensions, manage extensions, etc. I think the notion of package management thanks to the work of Andres finding the backdoor to xz also shines a new light, because there’s a huge security component to this. I remember, David, some of our earlier chats around this. I think you know —- again, being ap-developer sympathetic — I definitely want to see ways to make it easier to be able to load extensions.

Having spend spent a lot of time on the other side, the first thing that comes to mind is security. How do we create a protocol for managing the extension ecosystem that also allows folks to opt into it and apply their own security or operational or whatever the requirements are on top of it. That’s the thing that’s most top of mind. I don’t expect to have like a full resolution from the Extension Summit on it, but at least the start of it. What is ultimately that universal packaging distribution protocol for Postgres extensions that we can all agree on?

David Wheeler (he/him): Thank you so much! Before we go I just wanted to tee up that in two weeks Yuri Rashkovskii is going to talk about his idea for universally buildable extensions: dev to prod. That’ll be on May 1st at noon Eastern and 4pm UTC. Thank you everybody for coming.

More about… Postgres Jonathan Katz TLEs PGXN Extensions PGConf Summit

Thursday, 25. April 2024

The Pragmatic Engineer

The Pulse #91: The end of US non-competes within sight?

Also: the Humane AI pin flop and how it relates to the “AI goldrush,” and a look at whether developers will move from Redis to Valkey after a license change, or stay.

The Pulse is a series covering insights, patterns, and trends within Big Tech and startups. Notice an interesting event or trend? Send me a message.

Today, we cover:

Industry pulse. IBM buys HashiCorp; Google’s new operating reality; Section 174 still not repealed; Meta’s unexpected AI play, and more.

End of non-competes within sight in the US? The US Federal Trade Commission issued a ruling that would ban almost all non-compete agreements nationwide. If this passes, NDAs could become a lot more important for tech companies. However, the rule passing is far from a done deal: whether or not the rule lives will be decided in the courtroom. A closer look at the proposed changes.

The Humane AI pin flop, and “AI goldrush.” After years of developing hardware for the “new iPhone,” startup Humane suddenly pivoted to an AI use case. This bet was made soon after ChatGPT was released. Did Humane expect ChatGPT to improve much faster than it has?

Redis or Valkey? It’s only been a month since Valkey – a permissively licensed Redis ”continuation” announced its formation. I asked developers if they are planning to switch: and a quarter said they do.

1. Industry pulse IBM buys HashiCorp

Read more

Wednesday, 24. April 2024

Just a Theory

Test and Release pgrx Extensions with pgxn-tools

The v1.6.0 release of the pgxn-tools Docker image adds a new command to efficiently build and test pgrx extensions on a wide variety of Postgres versions.

Yesterday I released v1.6.0 of the pgxn/pgxn-tools Docker image with a new command: pgrx-build-test works much like the existing pg-build-test utility for PGXS extensions, but for pgrx extensions. Here’s an example from pg-jsonschema-boon, a pgrx extension I’ve been working on:

name: 🧪 Test on: push: jobs: test: runs-on: ubuntu-latest container: pgxn/pgxn-tools strategy: matrix: pg: [11, 12, 13, 14, 15, 16] name: 🐘 Postgres ${{ matrix.pg }} steps: - name: Checkout uses: actions/checkout@v4 - name: Start PostgreSQL ${{ matrix.pg }} run: pg-start ${{ matrix.pg }} - name: Setup Rust Cache uses: Swatinem/rust-cache@v2 - name: Test on PostgreSQL ${{ matrix.pg }} run: pgrx-build-test

The format is the same as for pg-build-test, starting with installing a specific version of Postgres from the Postgres Apt repository (supporting versions 8.2 – 17). It then adds the Swatinem/rust-cache action to speed up Rust builds by caching dependencies, and then simply calls pgrx-build-test instead of pg-build-test. Here’s what it does:

Extracts the pgrx version from the Cargo.toml file and installs it (requires v0.11.4 or higher) Initializes pgrx to use the Postgres installed by pg-start Builds the extension with cargo pgrx package Tests the extension with cargo pgrx test Installs the extension with cargo pgrx install Checks for a Makefile with installcheck configured and, if it exists, runs make installcheck

This last step allows one to include PGXS-style pg_regress tests in addition to Rust/pgrx tests, as pg-jsonschema-boon does. Here’s a successful run.

Special thanks to Eric Ridge and @Jubilee for all the help and improvements in pgrx v0.11.4 that enable this to work transparently.

pgrx Release Pattern

The pattern for releasing a prgx extension on PGXN is the same as before, although you may want to generate the META.json file from a template. For example, the pg-jsonschema-boon Makefile creates META.json from META.json.in by reading the version from Cargo.toml and replacing @CARGO_VERSION@, like so:

DISTVERSION = $(shell perl -nE '/^version\s*=\s*"([^"]+)/ && do { say $$1; exit }' Cargo.toml) META.json: META.json.in Cargo.toml @sed "s/@CARGO_VERSION@/$(DISTVERSION)/g" $< > $@

The release workflow uses it like so:

name: 🚀 Release on PGXN on: push: # Release on semantic version tag. tags: ['v[0-9]+.[0-9]+.[0-9]+'] jobs: release: name: 🚀 Release on PGXN runs-on: ubuntu-latest container: pgxn/pgxn-tools env: PGXN_USERNAME: ${{ secrets.PGXN_USERNAME }} PGXN_PASSWORD: ${{ secrets.PGXN_PASSWORD }} steps: - name: Check out the repo uses: actions/checkout@v4 - name: Bundle the Release env: { GIT_BUNDLE_OPTS: --add-file META.json } run: make META.json && pgxn-bundle - name: Release on PGXN run: pgxn-release

Note the “Bundle the Release” step, which first calls make META.json to generate the dated file, and tells pgxn-bundle to add the META.json via the GIT_BUNDLE_OPTS environment variable. The project also excludes the META.json.in file from the bundle in its .gitattributes file, and excludes META.json from the project repository in its .gigignore file.

Looking forward to seeing all your pgrx projects on PGXN!

More about… Postgres PGXN pgrx pgxn-tools

Jon Udell

The future of SQL: conversational hands-on problem solving

Here’s the latest installment in the series on working with LLMS: https://thenewstack.io/the-future-of-sql-conversational-hands-on-problem-solving/ I keep returning to the theme of choral explanations (#4 on my list of best practices), and it’s especially relevant in the SQL domain where there are just so many ways to write a query. Exploring the range of possibilities used to be … Continue reading

Here’s the latest installment in the series on working with LLMS: https://thenewstack.io/the-future-of-sql-conversational-hands-on-problem-solving/

I keep returning to the theme of choral explanations (#4 on my list of best practices), and it’s especially relevant in the SQL domain where there are just so many ways to write a query.

Exploring the range of possibilities used to be arduous, time-consuming and hard to justify. Now it’s becoming hard to justify not doing that; optimizations (sometimes major ones) can and do emerge.

The rest of the series:

1 When the rubber duck talks back

2 Radical just-in-time learning

3 Why LLM-assisted table transformation is a big deal

4 Using LLM-Assisted Coding to Write a Custom Template Function

5 Elevating the Conversation with LLM Assistants

6 How Large Language Models Assisted a Website Makeover

7 Should LLMs Write Marketing Copy?

8 Test-Driven Development with LLMs: Never Trust, Always Verify

9 Learning While Coding: How LLMs Teach You Implicitly

10 How LLMs Helped Me Build an ODBC Plugin for Steampipe

11 How to Use LLMs for Dynamic Documentation

12 Let’s talk: conversational software development

13 Using LLMs to Improve SQL Queries

14 Puzzling over the Postgres Query Planner with LLMs

15 7 Guiding Principles for Working with LLMs

16 Learn by Doing: How LLMs Should Reshape Education

17 How to Learn Unfamiliar Software Tools with ChatGPT

18 Using AI to Improve Bad Business Writing

19 Code in Context: How AI Can Help Improve Our Documentation

Tuesday, 23. April 2024

Just a Theory

PGXN V2 Architecture

I’ve written and published a document outlining the vision and strategy for the next iteration of PGXN. It includes diagrams. Everybody loves diagrams.
PGXN Future Architecture

High-level diagram of the six logical services making up the proposed future extension distribution architecture. The Root Registry sits at the center, providing APIs for the other services to consume for their own use cases. Trusted instances of those services submit additional data about extensions via the Interactions service to enhance and enrich the service to better inform and delight users.

Over on the Postgres Wiki I’ve published a new document for the PGXN v2 project: PGXN v2 Architecture. It has diagrams, such as the one above! From the introduction:

This document outlines the project to build extension distribution, discovery, and packaging tools and services to power the growth, accessability, and utility of the Postgres extension ecosystem. Taking the overall Postgres community as its audience, it defines the services to be provided and the architecture to run them, as well as the strategic vision to guide project planning and decision-making.

With the goal to think strategically and plan pragmatically, this document describes the former to enable the latter. As such, it is necessarily high-level; details, scoping, and planning will be surfaced in more project-focused documents.

Bear in mind that this document outlines an ambitious, long-term strategy. If you’re thinking that there’s too much here, that we’er over-thinking and over-designing the system, rest assured that project execution will be fundamentally incremental and pragmatic. This document is the guiding light for the project, and subject to change as development proceeds and new wrinkles arise.

For those of you interested in the future of Postgres extension distribution, please give it a read! I expect it to guide the planning and implementation of the the new services and tools in the coming year. Please do consider it a living document, however; it’s likely to need updates as new issues and patterns emerge. Log in and hit the “watch” tab to stay in the loop for those changes or the “discussion” tab to leave feedback.

I’ve also moved the previously-mentioned document Extension Ecosystem: Jobs and Tools to the wiki, and created a top-level PGXN v2 and PGXN category for all PGXN-related content. It also includes another new document, Service Disposition, which describes itself as:

A summary of the ambitiously-envisioned future PGXN services and architecture, followed by an examination of existing services and how they will gradually be refactored or replaced for the updated platform.

Check it out for how I expect existing services to evolve into or be replaced by the updated platform.

More about… Postgres PGXN Architecture

The Pragmatic Engineer

Building Bluesky: a Distributed Social Network (Real-World Engineering Challenges)

Bluesky is built by around 10 engineers, and has amassed 5 million users since publicly launching in February this year. A deep dive into novel design decisions, moving off AWS, and more.

Before we start: AI tooling for software development feels like it has hit "peak hype" across mainstream media. We would like to do a "reality check" and find out how engineers and teams are using these tools (and which tools/use cases are genuinely efficient). Please help us by filling out this survey.

Fill out the survey on AI tools

We will share the full report with all of you who share detailed insights. Thanks for your help!

‘Real-world engineering challenges’ is a series in which we interpret interesting software engineering or engineering management case studies from tech companies.

Bluesky is known as a Twitter-alternative. It launched two years ago, with an invite-only beta launch last year. It’s already grown to an impressive 5.5 million registered users. Interestingly for software engineers, Bluesky is also a fascinating engineering project unlike any other mainstream social network. Martin Kleppman, author of the Designing Data Intensive Applications book, is involved as a technical advisor, and has published a paper outlining the novel approaches Bluesky has taken.

The biggest differences between Bluesky and other large social networks:

Decentralized. Bluesky is a “decentralized social network,” meaning anyone can run their own servers. If Bluesky’s core team turned off all services today, the network would keep functioning. As such, Bluesky offers a way for users to truly own their data and services.

Open source. Nearly everything about Bluesky builds is open source, and hosted on GitHub.

Rapid growth. The product went from zero to 5 million users in around 12 months after announcing an invite-only beta.

Small team. Bluesky was built with a small team of 3 engineers during the first year, and with 12 software engineers at the time of publication.

Other social networks have achieved some of these things; such as Mastodon allowing users to own their data and identity, and Meta achieving eye-catching growth by getting 100 million users in just a week. Still, only Bluesky has pulled off them all.

Today, we dive into how Bluesky is built, sitting down with its two founding engineers: Daniel Holmgren and Paul Frazee. They take us through:

Development timeline. How Bluesky went from a vague idea with few specific details, to a decentralized social network with millions of users.

Experimentation phase. A team of 2-3 engineers prototyped for 9 months, established the development principles, and laid the groundwork for the protocol and app.

v1 architecture. An overview of Bluesky’s architecture at the launch of its public beta offering. This was a Postgres database built on top of AWS, and used Pulumi.

v2 architecture. Extending Bluesky to support “federation,” allowing users to run their own Bluesky instances.

Scaling the database layer. PostgreSQL didn’t scale with the site’s growth, so it was time to migrate. The team chose ScyllaDB and SQLite.

Infra stack: from AWS to on-prem. AWS was becoming too costly, so Bluesky moved over to dedicated data centers and bare-metal machines.

Reality of building a social network. Typical firefighting issues, Elon Musk, and outages not being “life-or-death” crises.

1. Development timeline

Bluesky has been in development for just over 2 years, and has been publicly available for around 12 months. Here’s the timeline:

Bluesky’s development timeline and major milestones

Adding in the three phases we’ll discuss below:

Bluesky’s development, split into three phases Phase 1: Experimentation

The first 10 months of the project between January and October 2022 were all about exploration, and the team started to work fully in the open after 4 months. The first project the team open sourced was Authenticated Data Experiment (ADX), an experimental personal data server and a command-line client, accompanied by a network architecture overview.

In April 2022, heavy Twitter user, Elon Musk, raised the prospect of potentially acquiring the site, which created interest in alternatives to the bird app, as any major change in a market-leading social network does.

The first commit for the Bluesky mobile app was made in June 2022, and Paul Frazee worked on it. It started as a proof-of-concept to validate that the protocol worked correctly, and to aid protocol development via real-world use. Conventional wisdom says that prototypes are thrown away after serving their purpose. 

However, in this case this mobile app that a single person had built, became the production app, following the unforeseen spike of interest in it caused by takeover news at Twitter. This is a good reminder that real world events can push conventional wisdom out of the window!

In October 2022, the team announced the Authenticated Transfer Protocol (AT Protocol) and the app’s waitlist, just a few days after news that Elon Musk was to acquire Twitter. This led many tweeters to seek alternative social networks, and drove a major signup spike for Bluesky’s private beta. This development put pressure on the Bluesky team to seize the unexpected opportunity by getting the protocol and app ready for beta users. See details on the AT Protocol.

Phase 2: invite-only launch and the first 1M users

In October 2022, Bluesky consisted solely of Jay Graber CEO, and two software engineers; Daniel and Paul. Engineer #3, Devin, joined the same month. Announcing the AT Protocol and waitlist generated some media buzz and Bluesky attracted more interest during this period.

In March 2023, the company was confident that the protocol and mobile app were stable enough to invite more users by sending invites.

“Blocking” was implemented in a single night. After the app opened up to more users, there was an influx of offensive posts and of users verbally harassing other accounts. This made it clear that implementing blocks to restrict individual accounts from viewing and commenting on a user’s posts, was urgently-needed functionality.

The three earliest developers – Paul, Devin and Daniel – jumped on a call, then got to work. In the community, developers saw the pull requests (PRs) on this feature appear on GitHub, and started to point out bugs, and cheer on the rapid implementation. They wrapped it up and launched the feature by the end of the same day. To date, this is the most rapidly-built feature, and is still used across the protocol and the app!

In June 2023, Bluesky passed the 100,000-users milestone when the team numbered 6 developers, who’d shipped features like custom feeds, blocking and muting, moderation controls, and custom domains. A web application built on React Native was also in production.

In September 2023, Bluesky passed 1 million users – a 900,000 increase in just 3 months!

Phase 3: Preparing for public launch

In the 6 months following the 1 million-user milestone, the focus was on preparing to open up Bluesky to the public with no waitlist or throttling of invites.

Federation (internal.) To prepare for “proper” federation, the team made architecture changes to enable internal federation of Bluesky servers. 

Federation is a key concept in distributed networks. It means a group of nodes can send messages to one another. For Bluesky, it meant that – eventually – users should be able to run their own PDS instances that host their own user information (and user information of users on that server.) And the Bluesky network operates seamlessly with this distributed backend.

A new logo and a reference to Twitter. The team prepared a new logo for launch, and announced it in December 2023:

The old and new logo

The butterfly logo is intended as a symbol of freedom and change. Existing centralized social media platforms – like X (formerly Twitter,) Instagram, TikTok, and Youtube – are platforms that want to lock users into their website and apps. Bluesky, on the other hand, offers its protocol, but doesn’t dictate which apps or websites people use. It doesn’t even want to dictate the hosting of content:

The final slide of Jay Graber’s presentation for Twitter about the vision for Bluesky. It was Twitter that granted Bluesky its initial $13M funding in 2021 – partially based on this vision. The image visualizes the blue bird freed from a closed platform into Bluesky’s open ecosystem. Source: Bluesky

Let’s dive into each phase of the building process.

2. Experimentation phase

During Bluesky’s first 9 months (January-September 2022) two software engineers built the protocol and apps – Daniel Holmgren and Paul Frazee – and Jay the CEO signed off design decisions. The first couple of months were about experimenting and tech “spiking,” which means timeboxing the time and effort spent building and trying out ideas. Here’s Paul:

“We would greenfield for a period, then attack what we had just created to see if it holds up. We gave the existing technologies a really close look; if we didn’t see meaningful improvements from the existing protocols, then we decided we’d use what was already out there.”

When the direction wasn’t clear, the team kept trying out new approaches, says Daniel:

“We set out to use as many existing specs as we could. We spent a lot of time early on investigating things like Activity Pub and seriously trying to figure out how we could make it work, and realizing that it didn't really work for our use case.”

Development principles

The still-small team set up principles to ensure continuous progress:

No backward steps. Ease of use, scale, and feature developer experience, can not be worse than existing social networks’.

Unify app development with protocol development. Never make tech decisions in isolation from practical use cases.

Don’t be precious! If an idea or design doesn’t work, just throw it out!

Approach to building a new, novel decentralized protocol

The team prioritized flexible design choices in order to not lock themselves into a technology, until they knew exactly what they were building. Not coupling the data layer too closely with Postgres is an example of this. See below.

Building for flexibility, not scalability, was deliberate. The idea was to swap this approach to prioritize scale once everyone knew exactly what to build. The knowledge that decisions are hard to undo made the team’s own decision-making more thorough, Daniel reflects:

“The most difficult part of building Bluesky has been the constant awareness that small decisions you make may be locked in for years and have ripple effects. In a decentralized environment, these can be difficult to unwind. It puts a lot of weight on every decision, and we have to double and triple check choices that we make so that we hopefully don’t regret them.” 

Inventing new approaches was never a goal. The original idea was to take a protocol or technology off the shelf, and push it as far as possible to reveal a requirement that didn’t quite fit. For example, Lexicon – the schema used to define remote procedure call (RPC) methods and record types – started out as JSON schemas. The team tried hard to keep it lightweight, and stuck to JSON schemas. But they ended up bending over backwards to make it work. In the end, the team decided to fork off from JSON schemas and added features to it, which is how Lexicon was born.

Bluesky gets criticism for inventing new approaches which are non-standard across decentralized networks. Paul explains it like this:

“We never set out to live the ‘not invented here’ (NIH) syndrome. I don’t think anyone building something new has this goal. In the end, it just naturally evolved in this direction.

No one had done a high-scale decentralized social network before this! If someone had, we probably wouldn’t have needed to invent as many things.”

Bluesky takes inspiration from existing web technologies. As Daniel puts it:

“The AT Protocol is a pretty typical JSON API collection over HTTP. The architecture of Bluesky looks very similar to a traditional social media data center turned inside out. The firehose API looks a lot like Kafka – and we’re probably going to shard it in a similar way.”

3. v1 architecture: not really scalable and not federated – yet Infrastructure choices

PostgreSQL was the team’s database of choice when starting development. Postgres is often called the “Swiss Army knife of databases” because it’s speedy for development, great for prototyping, with a vast number of extensions. One drawback is that Postgres is a single bottleneck in the system, which can cause issues when scaling to handle massive loads that never materialize for most projects.

For the team, using Postgres worked really well while they were unsure exactly what they were building, or how they would query things. Paul’s summary of the choice to use Postgres:

“You start with a giant Postgres database and see how far that can take you, so that you can move quickly early on.”

AWS infrastructure was what the team started with because it’s quick to set up and easy to use, says Daniel:

“We were running everything out of AWS, and that is great because you can just spin up new VMs very easily, and spin up new stacks and services easily.”

The first infra hire at Bluesky, Jake Gold, iterated on the AWS setup:

“The basic idea we have right now is we’re using AWS, we have auto-scaling groups, and those auto-scaling groups are just EC2 instances running Docker Community Edition (CE) for the runtime and for containers. And then we have a load balancer in front and a Postgres multi-availability zone instance in the back on Relational Database Service (RDS). It’s a really simple setup.”

To facilitate deployments on AWS, the team used infrastructure-as-code service, Pulumi.

Modularizing the architecture for an open network was an effort the team kicked off early. The goal of modularization was to spin out parts of the network which users could host themselves. Daniel says:

“Our early insight was that we should give developers building on top of Bluesky the ability to focus on the parts of the network that they want to focus on. This is the microservices part.

An external developer building a feed should not need to index every “like” in the network. Someone self-hosting their own account should not need to consume thousands of posts to create a timeline. You can split the network into specific roles and have them work in concert.”

Personal Data Server

At first, the architecture of Bluesky consisted of one centralized server, the PDS (Personal Data Server.)

The Bluesky backend in March 2023

The strategy was to split this centralized service into smaller parts and allow for federation, eventually.

Bluesky being a federated network means individual users can run their own “Bluesky instance” and curate their own network.

The feed generator The backend in May 2023, after the feed generator was moved into its own component

In May 2023, the Bluesky team moved the feed generator to its own role. This service allows any developer to create a custom algorithm, and choose one to use. Developers can spin up a new Feed Generator service and make it discoverable to the Bluesky network, to add a new algorithm. Bluesky also allows users to choose from several predefined algorithms.

The Feed Generator interface was the first case of Bluesky as a decentralized network. From then, the Bluesky network was not solely the services which the Bluesky team operated, it was also third-party services like Feed Generator instances that plugged into the Bluesky network.

Dedicated “Appview” service

For the next step, the view logic was moved from the PDS, to an “Appview” service. This is a pretty standard approach for backend systems, to move everything view-related to its own service, and not to trouble other systems with presenting data to web and mobile applications.

Splitting out application view logic into its own service Relays to crawl the network

In the future, there could be hundreds or thousands of PDSs in the Bluesky network. So, how will all the data be synchronized with them? The answer is that a “crawler” will go through all these PDSs. In preparation for this crawl the team introduced a Relay service:

Preparing for federation and multiple PDSs by adding a Relay service to “crawl” later 4. v2 architecture: scaleable and federated

The v1 architecture needed to evolve in order to support full federation, and the team always planned to move on from it. But they expected v1 to last longer than only 6 months.

Federation

Federation sandbox. Before shipping a first version of federation, the team built a Federation Sandbox to test the architecture, as a safe space to try new features like modulation and curation tooling.

Internal federation. To prepare for federation proper, the next refactoring was to add support for multiple Personal Data Servers. As a first step, the Bluesky team did this internally. Users noticed nothing of this transition, which was intentional, and Bluesky was then federated! Proving that federation worked was a large milestone.

As a reminder, federation was critical to Bluesky because it made the network truly distributed. With federation, any user can run their own Bluesky server.

Prior to federation, Bluesky created 10 PDS services, wrapped into an Entryway interface

The “internally federated” PDS servers worked exactly like a self-hosted PDS. Bluesky made one addition, to wrap the internal PDS servers into a new service called “Entryway,” which provides the “bsky.social” identity to the PDSes. Entryway will become the “official” Bluesky OAuth authorization server for users who choose bsky.social servers, and one operated as a self-hosted server.

Later, Bluesky increased the number of internal PDS servers from 10 to 20 for capacity reasons, and to test that adding PDS servers worked as expected.

External federation. With everything ready to support self-hosted Personal Data Servers, Bluesky flipped to switch, and started to “crawl” those servers in February 2024:

Adding support for “proper” federation. Anyone can self-host a “Bluesky instance” in PDS form

To date, Bluesky has more than 300 self-hosted PDSs. This change has made the network properly distributed, anyone wanting to own their data on Bluesky can self-host an instance. Over time, we could also see services launch which self-host instances and allow for full data ownership in exchange for a fee.

Appview: further refactoring

Recently, Bluesky further refactored its Appview service, and pulled out the moderation functionality into its own service, called Ozone:

Moving moderation and post labeling functionality from Appview to Ozone

Users can run their own Ozone service – meaning to be a moderator in the Bluesky system. Here are details on how to self-host this service, and more about Ozone.

An architectural overview, with Martin Kleppman

Martin is the author of the popular software engineering book, Designing Data Intensive Applications, and he also advises the Bluesky team in weekly calls.

Martin and the Bluesky team published a paper describing the Bluesky system, Bluesky and the AT Protocol: Usable decentralized social media. In it, they offer a detailed overview of the architecture:

The architecture of Bluesky. Image source: Bluesky and the AT Protocol

The diagram above shows how data flows occur in the application:

Personal data server (PDS): these can be Bluesky-hosted (around 20 today) or self-hosted (around 300)

Relays: these collect events from the PDSs. Bluesky has its “official” relay hosted in its own infrastructure, but other developers can set up alternative relays that listen to all PDSs.

Firehose: the output of the relays.

Labelers and feed generators: these digest firehose events. They can be Bluesky-hosted, or be hosted independently of Bluesky.

App View: The Bluesky-hosted “official” app view, or alternate app views

Data flowing back to PDSs: feed generators hosted by Bluesky or externally, feed events data back to the PDSs.

5. Scaling the database layer Scaling issues with Postgres

Scaling issues emerged 2-3 months after the public beta launch in mid-2023.

Connection pool issues and lock contention. The Postgres connection pool backup and Node’s event loop got into a bad feedback loop. The team observed Postgres lock contention issues. This refers to multiple processes trying to access the same data simultaneously, but the data is locked to all except one process. For example, when multiple processes attempt to update the same row.

Small Postgres outages. Postgres doesn’t give the developer much control over which query plan it will take. Bluesky had a few smaller outages due to a query plan randomly flipping to something that ran about 1,000x times slower.

The need for horizontal scaling. Horizontal scaling is adding more machines to a service, so that the throughput of this system improves linearly with each new machine. But Postgres does not support horizontal scaling because it runs as a single database with transactional guarantees, meaning it becomes a bottleneck – if a necessary one – for the entire network.

As a reminder, the team was still tiny when all these scaling challenges emerged. There were only 6 developers (Daniel, Devin, Bryan and Jake on the backend, and Paul and Ansh on the frontend). Then in summer 2023, Daniel had a dream:

“After one stressful day, I dreamt that me, Jay [Bluesky’s CEO], and Devin were in my backyard. There were snakes everywhere you looked. We were going to wrangle and round up the snakes in a panic. But that that point, Devin stops and says to all of us: ‘wait, wait, guys, I think there’s a Postgres extension for this!’”

ScyllaDB replacing Postgres

The team knew they needed a horizontally scalable data storage solution, with fine-grained control of how data is indexed and queried.

ScyllaDB was an obvious choice because it supports horizontal scalability due to being a wide-column database (a NoSQL type.) Wide-column databases store data in flexible columns that can be spread across multiple servers or database rows. They can also support two rows having different columns, which gives a lot more flexibility for data storage!

Wide-column databases store data in columns so it’s highly scalable and flexible. Two rows in one table can have different types or numbers of columns. Source: AWS

The biggest tradeoffs:

Data must be denormalized, meaning it isn’t stored as efficiently as in a relational database. Basically, you’ll store more data and require more storage space.

Data needs to be indexed on write. Writing to a wide column database is more expensive than to a relational database. For each row and column changed, the relevant indexes need to be updated, which typically makes these databases more write-intensive than relational ones.

The team was satisfied with their early choice of Postgres, says Daniel:

“Postgres was great early on because we didn’t quite know exactly what questions we’d be asking of the data. It let us toss data into the database and figure it out from there. Now we understand the data and the types of queries we need to run, it frees us up to index it in Scylla in exactly the manner we need and provide APIs for the exact queries we’ll be asking.”

SQLite

ScyllaDB is used for the Appview, which is Bluesky’s most read-heavy service. However, the Personal Data Servers use something else entirely: SQLite. This is a database written in the C language which stores the whole database in a single file on the host machine. SQLite is considered “zero configuration,” unlike most other databases that require service management – like startup scripts – or access control management. SQLite requires none of this and can be started up from a single process with no system administrative privileges. It “just works.”

Daniel explains why SQLite was ideal for the PDSs:

“We took a somewhat novel approach of giving every user their own SQLite database. By removing the Postgres dependency, we made it possible to run a ‘PDS in a box’ without having to worry about managing a database. We didn’t have to worry about things like replicas or failover. For those thinking this is irresponsible: don’t worry, we are backing up all the data on our PDSs!”

SQLite worked really well because the PDS – in its ideal form – is a single-tenant system. We owned up to that by having these single tenant SQLite databases.

We also leaned into the fact that we’re building a federated network. We federated our data hosting in the exact same manner that it works for non-Bluesky PDSs.” 

Migrating the PDSs from Postgre to SQLite created fantastic improvement in operations, Daniel adds:

“PDSs have been a dream to run since this refactor. They are cheap to operate (no Postgres service!) and require virtually no operational overhead!”

6. Infra stack: from AWS to on-prem

Bluesky’s infrastructure was initially hosted on Amazon Web Services (AWS) and the team used infrastructure-as-a-code service, Pulumi. This approach let them move quickly early on, and also to scale their infra as the network grew. Of course, as the network grew so did the infrastructure bill. 

Move to on-prem

Cost and performance were the main drivers in moving on-prem. The team got hardware that was more than 10x as powerful as before, for a fraction of the price. How was this decision made? A key hire played a big role.

Bluesky’s first hire with large-scale experience was Jake Gold, who joined in January 2023, and began a cost analysis of AWS versus on-prem. He eventually convinced the team to make this big change.

But how did the team forecast future load, and calculate the hardware footprint they’d need? Daniel recalls:

“We looked at the trends and tried to make a safe bet. We were thinking: ‘okay, today we're over-provisioned. We want to stay over-provisioned, so we have room to grow without upgrading the hardware, but also just so we have stability if something happens in the world, and everyone decides to post about it.’

We built our architecture to be horizontally scalable so that we can add more capacity just by throwing more machines at it. There is some lead time to buying new machines, but we have space in the rack. We have room in the network connections. The switches are good for it.

If we need to scale, it’s really just about ‘get some more servers and hook them up!’ We can get to twice the capacity after doubling the machines we’re running in our data center. This is sweet!”

Becoming cloud-agnostic was the first step in moving off AWS. By June 2023, six months after Jake joined, Bluesky’s infrastructure was cloud agnostic. 

Bluesky always has the option of using AWS to scale if needed, and is designed in a way that it would not be overly difficult to stand up additional virtual machines on AWS, if the existing infrastructure has capacity or scaling issues.

Today, the Personal Data Servers are bare-metal servers hosted by cloud infrastructure vendor, Vultr. Bluesky currently operates 20 and shards them so that each PDS supports about 300,000 users.

Bluesky’s load by the numbers

Currently, Bluesky’s system sees this sort of load:

60-100 events/second received by the firehose service, which is the “main” service that emits messages sent on the network in real time. During the public launch of Bluesky in February, the peak was 400 events/second.

400 timeline loads/second. A timeline load is when a user (or client) makes a request to fetch their current timeline.

3,500 requests/second across the network.

7. Reality of building a social network

To close, we (Gergely and Elin) asked the teams some questions on what it’s like to build a high-growth social network.

What is a typical firefighting issue you often encounter?

“Every influx of users brought new problems, and we found ourselves doing quite a bit of firefighting. One day, after a particularly notable incident, growth showed no signs of stopping, and we had to temporarily disable signups in order to keep the service running.” – Daniel

What were the events referred to as “Elon Musk?”

“We never quite knew when a user bump was going to come, and invites were out in the wild waiting to be used. Then something would happen, and thousands of users suddenly joined. We started referring to these days as EMEs (Elon Musk Events) because they were normally precipitated by some change on Twitter.” – Daniel

“It was a bit like throwing a party and everybody showing up 2 hours early, while you’re still setting up the chairs and telling people to get drinks from the fridge. And then about ten times more people show up than expected.” – Paul

How are outages different for a social network?

“Disabling signups or pausing the service is never fun to do, but it actually created a bunch of excitement and a strange sense of pride in the user base.” – Daniel

“Outages are not fun, but they’re not life and death, generally. And if you look at the traffic, usually what happens is after an outage, traffic tends to go up. And a lot of people who joined, they’re just talking about the fun outage that they missed because they weren’t on the network.” – Jake

The whole developer team is on Bluesky, and actively responding to user feedback. How do you do this, and why?

“People just pinging us in the app and explaining their problem, is so good. We can just respond, "Hey, can you give me a screenshot? What platform are you on?" It's such a fast support turnaround. The big benefit of building a social app is that your customers are right there, and will tell you if something's not working.

Real time user feedback was how mute words got prioritized, recently. In terms of a signal about how important something is, when you start getting PRs to add the feature, and you get a ton of people plus-oneing the issue – not to mention people asking for it in the app – that tells you a lot.” – Paul

Takeaways

Gergely here. Many thanks to Daniel and Paul for part one of this deep dive into how Bluesky works! You can try out Bluesky for yourself, learn more about Bluesky’s AT Protocol, or about its architecture. And I’m also on Bluesky.

Decentralized architectures require a different way of thinking. I’ll be honest, I’m so used to building and designing “centralized” architecture, that the thought of servers being operated outside of the company is very alien. My immediate thoughts were:

Is it secure enough? Malicious actors could run anything on those servers and attempt to overload the network or exploit vulnerabilities in the system. The Bluesky team also stressed how the security model is something you thoroughly need to consider as you design APIs for such a system.

What about external nodes that don’t ever update the version of the software? How do they get bug fixes? And what about versioning? How to ensure “outdated clients” are cut off from the network?

Finally, I thought; “wow, this kind of reminds me of the confusion I initially felt about Skype’s peer-to-peer network

I’m delighted we did a deep dive about Bluesky because it has forced me to think more broadly. A server drawing on a diagram no longer just means “a group of our servers,” it can also mean “plus, a group of external servers.” Once this is understood, it’s easy. And this skill of designing distributed and federated systems may be useful in the future, as I expect the concept of distributed architecture to become more popular.

It’s impressive what a tiny team of experienced engineers can build. I had to triple-check that Bluesky’s core team was only two engineers for almost nine months, during which time they built the basics of the protocol, and made progress with the iOS and Android apps. Even now, Bluesky is a very lean team of around 12 engineers for the complexity they build with and the company’s growth. 

In the next part of this deep dive into Bluesky, we cover more on how the team works. 

Owning your own infrastructure instead of using the cloud seems a rational choice. Bluesky found large savings by moving off AWS once they could forecast the type of load they needed. Jake Gold, the engineer driving this transition, has been vocal about how cloud providers have become more expensive than many people realize. Speaking on the podcast, Last Week in AWS, he said:

“With the original vision of AWS I first started using in 2006, or whenever launched, they said they would lower your bill every so often, as Moore’s law makes their bill lower. And that kind of happened a little bit here and there, but it hasn’t happened to the same degree as I think we all hoped it would.”

Don’t forget, it’s not only Bluesky which rejects cloud providers for efficiency. We previously did a deep dive into travel booking platform Agoda, and why it isn’t on the cloud.

I’m slowly changing my mind about decentralized and federated social networks. I also tried out Mastodon, which is another federated social network, when it launched. At the time, Mastodon felt a lot more clunky in onboarding than Bluesky. You had to choose a server to use, but different servers have different rules, whereas Bluesky was much smoother. Still, as a user, I was blissfully unaware of how different these social networks are from the dominant platforms.

It was only by learning about Bluesky’s architecture that I appreciated the design goals of a decentralized social network. Currently, mainstream social networks are operated exclusively by the company that owns them. But a decentralized network allows servers to be operated by other teams/organizations/individuals. This might not seem like a big deal, but it means a social network is no longer dependent on the moderation policies of a parent company.

Decentralized social networks also allows users to use custom algorithms, websites and mobile apps, which creates opportunities for developers to build innovative experiences. In contrast, you cannot build a custom third-party client for X, Threads, or LinkedIn.

I’m still unsure how much mainstream appeal decentralized social networks hold for non-technical people, but I’m rooting for Bluesky, Mastodon, and the other decentralized social apps. Perhaps they can challenge Big Tech’s dominance of social media, or at least change people’s understanding of what a social network can be.

In a follow-up issue, we’ll look deeper into the engineering culture at Bluesky: the company culture, a deeper look at the tech stack, and how they are building seemingly so much with a surprisingly small team and company. I suspect we can all learn a lot in how a dozen engineers help a startup scale to more than 5 million users.

Enjoyed this issue? Subscribe to get this newsletter every week 👇

Subscribe now


@_Nat Zone

NIST SP 800-63Bへの補遺が出ました〜パスキーの追加です

近年、多要素認証(MFA)の普及が進む中、ユーザーの利便性とセキュリティのバランスを取ることが課題となっています。MFAとは、パスワードに加え、生体認証やワンタイムパスワードなど複数の認証方式を組み合わせることで、アカウントへの不正アクセスを防ぐセキュリティ手法です。しかし、MF…

近年、多要素認証(MFA)の普及が進む中、ユーザーの利便性とセキュリティのバランスを取ることが課題となっています。MFAとは、パスワードに加え、生体認証やワンタイムパスワードなど複数の認証方式を組み合わせることで、アカウントへの不正アクセスを防ぐセキュリティ手法です。しかし、MFAの導入はユーザーにとって面倒な手順が増えることを意味します。この課題を解決するために登場したのが、Syncable Authenticator(同期可能認証器)、別名Passkey(パスキー)と呼ばれる新しい認証技術です。

同期可能認証器は、認証に用いる秘密鍵を複数のデバイス間で同期できるようにすることで、ユーザーはどのデバイスでも同じ認証情報を使えるようになります。これにより利便性が大幅に向上します。一方で、秘密鍵を複数の端末で共有することはセキュリティ上のリスクを伴います。そこで、米国国立標準技術研究所(NIST)は、シンカブル認証器の安全な利用に関するガイドライン(SP 800-63Bの補遺)を公開しました。

NIST: Incorporating Syncable Authenticators Into NIST SP 800-63B

この文書は、NIST Special Publication 800-63Bへの補足であり、認証キーを複製しデバイス間で同期できる同期可能認証器(パスキーとしても知られている)の使用に関するガイダンスを提供するものです。主なポイントは以下の通りです。

適切に構成された同期可能認証器は、中間者攻撃、検証者のなりすまし、リプレイ攻撃などの脅威を軽減し、認証の意図を提供することで、認証保証レベル2(AAL2)を達成できます。AAL2は、フィッシング耐性など高度なセキュリティを求める一方で、ある程度の使いやすさも確保するレベルです。 本文書では、キーの生成、保存、アクセス制御に関する一定の要件を満たすことを条件に、同期可能認証器の認証キーの複製を許可するようSP 800-63Bを更新しています。これによって、政府機関でもパスキーを使うことができるようになります。 WebAuthn仕様のフラグを使用して、認証器がAAL2の要件を満たしているかどうかを判断するなど、実装上の考慮事項について説明しています。エンタープライズでのユースケースでは、Attestationを利用してオーセンティケータの機能を検証することができます。 同期可能認証器の潜在的な脅威と課題を概説し、不正なキーの使用、同期ファブリックの侵害、失効の難しさなどを示し、それらを軽減するための方法を提案しています。 本文書は、一部の実装におけるユーザー間のキー共有のリスクを認識し、エンタープライズおよび公開向けのユースケースについてガイダンスを提供しています。

全体として、この補足資料は、適切に導入されれば、便利でフィッシング耐性のある認証を提供できる同期可能認証器の統合について、機関がリスクに基づいた informed な意思決定を行うことを目的としています。

本補遺は、AAL2への同期可能認証器の適合性を検証する上で重要な基準を提供しています。また、鍵の不正使用やクラウドストレージの侵害といった新たな脅威についても言及し、対策を提言しています。同期可能認証器の潜在的なリスクを認識しつつ、適切に導入することで、利便性を損なわずにセキュリティを向上できることを示しています。

本補遺は、セキュリティ担当者だけでなく、同期可能認証器の導入を検討する全ての関係者にとって有益な情報を提供しています。NISTによる公的なガイドラインの存在は、同期可能認証器の安全性に対する信頼を高め、普及を後押しするでしょう。ユーザー数の増加は、さらなる技術革新を促します。本補遺は、セキュリティと利便性を両立する新認証技術の健全な発展に寄与する重要な一里塚になると考えられます。

なお、背景等については @phr_eidentity さんのこの記事1が詳しいです。


アイデンティティをやっているなら読むべき本〜『メタバース進化論――仮想現実の荒野に芽吹く「解放」と「創造」の新世界』

まずはじめにアイデンティティ業界の人へ とりあえず、アイデンティティに関心のある人は、本書『メタバース進化論』を読んだほうが良い。メタバース・バ美肉として現実の物理的制限から存在を解き放ち、抽象空間で取り扱うことができるようになることによって、アイデンティティとプライバシー管理の…
まずはじめにアイデンティティ業界の人へ

とりあえず、アイデンティティに関心のある人は、本書『メタバース進化論』を読んだほうが良い。メタバース・バ美肉として現実の物理的制限から存在を解き放ち、抽象空間で取り扱うことができるようになることによって、アイデンティティとプライバシー管理の課題や要件がこんなにも明確になるのか!曰く

そもそも「アイデンティティ(自己同一性)とは、私たちが私たち自身をどのようなものと捉えるかという「認識」であり、他者や社会からそれが認められているという「感覚」のことです。 物理現実では、基本的に生まれたままの名前・姿・声を受け入れるしかありませんでした。つまり、アイデンティティとは「与えられる」ものでした。 基本的には与えられた固定のものを「受け入れる」しかなかった物理現実時代のそれとは違い、メタバース時代のアイデンティティは自由に「デザインする」ものになり、「なりたい自分」としての人生を送ることが可能になる 人間を分割可能な「分人(Dividual)」として捉える 複数のアイデンティティを「切り替える」ことで人生を自由にデザインできる プラトンの「イデア論」〜見ている現実世界は、実体ではなく、イデアの影にすぎないのだ アイデンティティを自在にデザインして「なりたい自分」になれるメタバース

(出所)『メタバース進化論』

わたしの本1やブログ、講演などに接しているかたは、「はいはいアレね」と思うだろう。次の図は、2011年6月に書いた「非技術者のためのデジタル・アイデンティティ入門」の頃から使いまわしている図だが、ほとんど同様のことを言っている。分人 v.s. Identity (Partial Identity) など使っている言葉は違えど殆どマッピングできる。

ただ、多分、本書のほうが100倍わかりやすい。これは、この図では自己像と言って抽象的に扱っているものにアバターという実体を与えて手触り感を持たせているのと同時に、何でもかんでも「属性」という分類でくくってしまわずに、「言霊世界」「視覚世界」「音響世界」のようにカテゴライズして説明しているからだろう。もちろん単純な筆力の違いもあるが。

【図表1】自己像(アイデンティティ)と幸福追求

とりわけ重要なのが、魂の自由性とか、アイデンティティは他者から与えられるものではなく自分で確立するものだというところだ。このあたり、「アイデンティティの専門家」でも忘れている人達が多い。神だとか政府だとかに与えてもらいそれを受容するものだという中世的な考え方の人たちが多いのだ。

中世的アイデンティティ v.s. 近代的アイデンティティ

近代的な個人観と中世的な個人観の主な違いは以下のようにまとめらるだろう。

中世的個人観

個人は共同体の一部として存在し、共同体からの役割や地位によって定義されていた。2 キリスト教の影響で、個人は神との関係性において「世俗外個人」として捉えられていた。3 4 個人の内面性よりも、共同体への所属と役割が重視されていた。5

近代的個人観

個人は自律的な存在として捉えられ、個人の内面性や自我意識が重視されるようになった。67 啓蒙主義の影響で、個人は中世的な共同体からの拘束から解放され、「内面的孤独」を経験するようになった。8 個人の権利や自由が強調され、個人主義的価値観が台頭した。910 プライバシーの概念が生まれ、私的領域が神聖視されるようになった。11

つまり、中世では個人は共同体の一部として捉えられていたのに対し、近代では個人の自律性と内面性が強調され、個人主義的価値観が確立されたということができる。魂の自由性と言っても良い。でも残念ながら、この個人主義的価値観・近代的個人観に行き着いていない人が結構多い。だから、アバターは現実世界の自分に似ていなければならないだとか、アイデンティティウォレットには政府発行の個人属性証明(PID)を入れて、それを中心に回していくとかの考えになってくる。

これは別にウォレットに限ったことではなくて、Web 2.0的なSNSにおける本人確認でも似たようなことが起きてくる。先ほど経験したことだと、LinkedInで本人確認済みにするには、氏名はパスポート表記と同一でなければならないという。パスポート表記などというのは、国境を超えるときの識別子として政府に登録されたものであって、自分を表出するものではない。わたしは英語では Nat Sakimura として活動しているが、それが「本人確認」をすると、Natsuhiko Sakimuraであることを強制される。なんだそれは。そんなことしたら過去の業績とのリンクができなくなるではないか。いやそんな些細なことではない。名乗りを奪うということは、人格を否定することに等しい。ちょうど千と千尋の神隠しの湯婆婆が千尋の名前を奪うように。

まさに、本書がいうように、名前は「言霊世界のアイデンティティ」なのだ。この「名乗りの自由性」はとても大切だ。なぜならば、本書が言うように『周りの方々からこの「言霊」で毎日繰り返し呼びかけられ続けることで、わたしは◯◯としての自己認識を獲得することができる』からだ。

また、言うまでもなく「アバター:視覚世界のアイデンティティ」(本書)の自由性も重要だ。『自分自身の「アイデンティティ表現」としてみた時は現実のような「人間」のかたちに縛られたくない』(本書)からだ。それともあれだろうか。四肢が不自由だったら、メタバースでも四肢が不自由でなければいけないとでも言うのか。

その自由性の象徴がバ美肉である。日本では、人形浄瑠璃や歌舞伎の女形などの文化的背景がありだいぶ受容が進んできたが、西洋ではヘタをしたら56されかねない。バ美肉を受容できるかどうかは、潜在的な差別意識のあるなしのリトマス試験紙になる。

本書ではMetaが規制を入れてくる可能性に触れているが、AppleもVision Pro でやってきても驚かない。場合によっては、VRのアバタを簡単に作らさせるためにUXとしてそういうことにするといってくるかもしれない。だがこうした動きには徹底抗戦しなければなるまい。日本では、自己イメージコントロール権説や、「身だしなみなどに関する自己決定権」から、「なりたい自分になれる権利」は認められ、間接適用説からmetaなど事業者もこの権利の侵害は許されないはずだ12

こうしたアイデンティティに関する話は第4章にまとめられているので、時間がなかったらまずそこだけでも読んでほしいところだ。

プライバシーに関しても参考になる

また、アイデンティティと違って一つの章になっているわけではないが、プライバシーに関しても参考になる記述がいろいろとある。曰く、

同じく日本の伝統芸能「人形浄瑠璃」でも、優美に踊る美しい女性の人形の背後には黒子のおじさんが見えてしまっています。しかし、それを指摘するのは野暮

これは、「データが見えてしまっても、意図された目的以外では使ってはいけない」というデータ保護の基本につながる。わたしが昔から「大人のプライバシー」と呼んでいるものだ。もちろん「なりたい自分になれる」というのが、「自己イメージコントロール権」13と表裏一体であるのは言うまでもなく。

ISOでは現在、メタバースのプライバシーについての検討が始まっているが、関係者にぜひ本書を読ませたいところだ。英語版がほしい。

もちろんメタバースの解説も

さて、自分がアイデンティティ&プライバシー業界の人間だから、いきなりそのあたりの各論に入ってしまったが、そもそも本書はメタバースについての本だ。「メタバースの定義:実現に必要な七要件」であるとか、4大ソーシャルVRの紹介であるとか、それらを支える技術であるとか、メタバースに住む住民たちの特性、なぜ人々はバ美肉になるのか、などなど、データの裏付けをとりながらきっちりと解説している。しかも、マーカーで線を引きたくなるようなキャッチーなフレーズが満載で。たとえばこんな感じ:

声:音響世界のアイデンティティ 自分の精神に作用するプロテウス効果 バ美肉は枯山水。見立ての文化の延長線。 (現在は)「永遠に新しいことを学び続けていく」ことが当然(の社会)。すべての人が永遠に「未熟」であることを受け入れざるを得ない社会において、「大人としての自尊心」に変わる新たな価値観こそが、お互いの「未熟さ」を愛おしいと思う価値観「かわいい」

(出所)『メタバース進化論』

わたしは、Kindleで大量に線を引きながら読んでしまった。上記はそのほんの一部だ。ぜひ本書を買って読んいただきたい。

動画による解説も一緒にみるとなお良し

なお、本書は文章による紹介としては大変優れているが、やはりメタバースのイメージを得るには映像のほうが適している。そういう意味で、筆者である「バーチャル美少女ねむ」さんが、国連のIGF 2023で行ったプレゼン「バーチャル美少女だけど国連登壇してみた」も並行して見ることをおすすめする。非常にわかりやすいプレゼンだ。英語ですが…。

これは、国連が主催するインターネット・ガバナンス・フォーラム(IGF14 )2023 京都のDay 0

「Event No.134 Talk with Metaverse residents – a new identity and diversity | Internet Governance Forum」(2023年10月8日 18:00〜19:00)

の模様を録画したものだ。わたしもこの日だけは、自分のセッション15があったのでIGFの会場にいたのに、生参加できなくて残念だった。多分この時間はOECDの人とTrusted Data Intermeidaries (TDI: 信頼できるデータ仲介者) について話していたと思う。

というわけで

本書、本当におすすめだ。サンフランシスコからの機内で読んでいて、第4章を読んでいて涙が自然に流れ出てきたくらい。ぜひ買って読みましょう!

【変更履歴】

2024-04-23 初版 2024-04-24 中世的アイデンティティ v.s. 近代的アイデンティティを追加、敬体を常態に変更。

Monday, 22. April 2024

Just a Theory

🎙️ Hacking Postgres s02e03

I quite enjoyed the chat with Ry Walker on the “Hacking Postgres” podcast, posted last week.

Hacking Postgres S2E3: David Wheeler, Principal Architect, Tembo

Last week I appeared on s02e03 of the Hacking Postgres podcast.

The experience I had after my independent Consulting gig for 10 years working in companies was, like, bringing up other people and being supportive of other people and hearing from a diversity of voices and perspectives makes everything better.

That’s part of why I want to get so much input on and feedback on the stuff that I’m hoping do with PGXN v2 — or whatever we ultimately call it. But people matter, more than the technology, more than any of the rest of it.

I quite enjoyed this wide-ranging discussion. We covered my history with the Postgres community, a bunch of the projects I’ve worked on over the years, plans and hopes for the PGXN v2 project, perspectives on people and technology, and exciting new and anticipated features of Postgres. Find it wherever fine podcasts are streamed, including:

YouTube Apple Podcasts Overcast Twitter More about… Postgres Podcast Hacking Postgres Sqitch pgTAP PGXN

Phil Windleys Technometria

Using X.509 Certs for DID Provenance

Summary: The abundance of X.509 certificate authorities who already perform identity proofing for businesses provides a rich resouce that can be leveraged to boot the verifiable data ecosystem. Here’s how it could work. When you used a verifiable credential to prove something about yourself, the verifier can know cryptographically: (1) the identifiers for the issuer, (2) the credential hasn't been

Summary: The abundance of X.509 certificate authorities who already perform identity proofing for businesses provides a rich resouce that can be leveraged to boot the verifiable data ecosystem. Here’s how it could work.

When you used a verifiable credential to prove something about yourself, the verifier can know cryptographically: (1) the identifiers for the issuer, (2) the credential hasn't been tampered with, (3) the credential was issued to you, and (4) the credential hasn't been revoked. These four checks are important because their establish the fidelity of the data being transferred. They don't, however, tell them whether they can trust the issuer. For that, they need to take the issuer's decentralized identifier (DID) that they got from credential presentation and determine who it belongs to.

At Internet Identity Workshop last week, Drummond Reed gave a session on how X.509 certificates could help with this. The first step, like always, is to resolve the DID and retrieve the DIDDoc that associates keys and endpoints with the DID. The endpoint can be an HTTP server and, of course, should have an X.509 certificate providing TLS security. That certificate, at the very least, has a a domain name to bind that to the certificate's public key. It can, if you pay for the feature, also include information about the entity that applied for the certificate. The certificate authority proofs that information and is vouching for it when they sign the certificate.

The key to making the X.509 certificate useful for checking the provenance of a DID lies in one key change. X.509 certificates can contain and extended field called a Subject Alternative Name. This following figure shows how it can help.

Using X.509 Certificates to establish the owner of a DID

The issuer (Attestor) creates the DID they will use to issue the certificate along with its associated DIDDoc, including an HTTP endpoint for DID verification.

Attestor applies for a X.509 certificate for that endpoint, including in the application the DID they created in (1).

The certificate authority does it's usual proofing of the application and issues a certificate that includes the DID in the Subject Alternative Name field.

The issuer creates a credential definition in the usual way that includes their DID and writes it to whatever Verifiable Data Registry their DID method dictates.

Attestor issues a credential to a holder (Alice) using that credential definition.

At some later time, Alice presents the credential to the verifier (Certiphi).

Certiphi resolves the DID to get the DIDDoc and retrieves the verfication endpoint from the DIDDoc

Certiphi retrieves the certificate for that endpoint1.

Certiphi verifies the certificate by checking it's signature and ensures that the DID in the DIDDoc for the credential matches the one in certificate.2

The issuer's DID has now been tied in a verifiable way to whatever information is in the certificate. Provided the certificate includes information about the entity beyond the domain name, the verifier can use that information to determine whether or not the credential is authentic (i.e., issued by who the credential definition purports issued it). That might be all the evidence they need to determine whether to trust the entity. Certificate authorities could also issue verifiable credentials to the customer attesting the same verified claims—after all, it's one more product they can offer.

The benefit of doing issuer validation using X.509 certificates is that there are already many trusted X.509 certificate authorities in business who already do proofing of attributes about businesses. That's a huge chunk of the verifiable data ecosystem that doesn't need to be built because it can be leveraged. To make this work, digital certificate authorities would need to start offering to validate DIDs and include them in a certificate as a Subject Alternative Name. I don't discount that this will take some bureaucratic maneuvering. Certificate authorities will need to see a business opportunity. I'd love to see Digitcert or someone do a pilot on this.

Notes

Note that this step might be combined with the previous step if the Verifiable Data Registry is the same server as the endpoint, but that's not necessarily going to be the case for a number of reasons.

Note that this does not create a call back wherein Attestor can determine which credential was used, preserving the privacy of the presentation. Attestor does know one of its credentials has been presented to Certiphi.

Subscribe now

Share


Doc Searls Weblog

Archives as Commons

The Santa Barbara News-Press was born in 1868 and died in 2023 at age 155. Its glory years ran from 1932 until 2000, when the New York Times sold it to Wendy McCaw, who rode it to hell. That ride began with the Santa Barbara News Press Controversy in 2006 and ended when Ampersand, the company McCaw created […]
The Santa Barbara library, viewed from the county courthouse. Is this where the dead local paper’s archives will go? How about future archives of all the local news organs?

The Santa Barbara News-Press was born in 1868 and died in 2023 at age 155. Its glory years ran from 1932 until 2000, when the New York Times sold it to Wendy McCaw, who rode it to hell.

That ride began with the Santa Barbara News Press Controversy in 2006 and ended when Ampersand, the company McCaw created to hold the paper’s bag of assets (which did not include its landmark building downtown, which McCaw kept), filed for Chapter 7 bankruptcy in late July of last year. Here are stories about the death of the paper in three local news journals that have done a great job of taking up the slack left when the News-Press began to collapse, plus one in the LA Times:

‘Santa Barbara News-Press’ Files for Bankruptcy: Publisher Ampersand Claims Few Assets and Many Creditors, by Jean Yamamura in the Santa Barbara Independent (July 23, 2023) Santa Barbara News-Press Declares Bankruptcy, Staff Told All Jobs ‘Eliminated’, by Joshua Molina, in Noozhawk (July 23, 2023) Santa Barbara News-Press Files for Bankruptcy, Staff Fired, by Edhat Staff (July 24, 2023) Santa Barbara News-Press bankruptcy brings uneasy end to an owner’s bitter tenure, by James Rainey, in the Los Angeles Times. (July 29, 2023)

I followed those with this in We Need Deep News:

From what I’ve read so far (and I’d love to be wrong) none of those news reports touch on the subject of the News-Press‘ archives, which conceivably reach back across the century and a half it was published. There can’t be a better first draft of history for Santa Barbara than that one. If it’s gone, the loss is incalculable. (August 18 2023)

Last month brought bad news about that:

‘Santa Barbara News-Press’ Online Assets to Be Sold: Bankruptcy Sale for $250,000 Subject to Bids in April, by Jean Yamamura in the Santa Barbara Independent March 8, 2024 A local paper went bankrupt. Now a faraway buyer wants its assets: The Santa Barbara News-Press’s digital assets are up for sale. Locals worry they could become a farm for AI-generated SEO bait. By Ernie Smith in Fast Company. (March 5, 2024) Santa Barbara’s Collective Memory, Sold for Kindling: Will ‘NewsPress.Com’ Become a Zombie Website? by William Belfiore in the Independent. (March 29, 2024)

But then, thanks to William Belfiore’s appeal in that last piece, we learned this:

‘Santa Barbara News-Press’ Website Goes to ‘Local Kids’ Group Fronted by Ben Romo Makes Winning Auction Bid of $285,000, by Jean Yamamura in the Independent (Apr 09, 2024)

The only mention of archives was in the closing sentences of that piece:

The purchase of the website included the Santa Barbara News-Press trademark, which would be important to the groups looking at the physical archive of back issues, photographs, and clippings by topic. Romo, who was once a paper boy for the daily, acknowledged that his group was supportive of the archive remaining local, too.

I don’t know what that means, and I haven’t checked. But I am sure that the archives ought to be managed by the community as a common pool resource.

As it happens, my wife and I are visiting scholars at the Ostrom Workshop at Indiana University, which is concerned with this kind of thing, because its namesake, Elinor Ostrom, won a Nobel Prize in Economics for her work on how commons are self-governed. In her landmark book, Governing the Commons: The Evolution of Institutions for Collective Action, she lists eight principles for managing a commons, which are summarized here:

Define clear group boundaries. Match rules governing use of common goods to local needs and conditions. Ensure that those affected by the rules can participate in modifying the rules. Make sure the rule-making rights of community members are respected by outside authorities. Develop a system, carried out by community members, for monitoring members’ behavior. Use graduated sanctions for rule violators. Provide accessible, low-cost means for dispute resolution. Build responsibility for governing the common resource in nested tiers from the lowest level up to the entire interconnected system.

Journalists, especially those who report news, are not herding animals. They tend to be competitive and territorial by both nature and purpose. So the collection of news entities I wrote about in We Need Wide News and We Need Whole News will almost certainly not cohere into a commons such as Lin (her nickname) Ostrom addresses in that list.

But they should cohere around archives—not only because that’s the right thing to do, but because they need those archives. We all do.

So I hope Santa Barbara’s many journals, journalists, friends, supporters, and interested local institutions get together around this challenge. Build a commons around those archives, whatever and wherever they happen to be.

Meanwhile here in Bloomington, my other hometown, we are pushing forward with The Online Local Chronicle that Dave Askins wrote about in the previous installment in this series. We might call that a commons interest here.

 

 


Wrench in the Gears

On The Road Again – Vermont Trip Wrap Up

I’m Arkansas bound in the morning, so I don’t have time for a long post. All I have time for is to share a video with musings about the second part of my trip north and the photos that go along with it. Keep me in your prayers these next few weeks. I’m going to [...]

I’m Arkansas bound in the morning, so I don’t have time for a long post. All I have time for is to share a video with musings about the second part of my trip north and the photos that go along with it. Keep me in your prayers these next few weeks. I’m going to be putting a lot of miles behind me.

 

 

 

 

 

 

 

 

 

Sunday, 21. April 2024

Doc Searls Weblog

This Thing is Bigger Than Journalism

Journalism as we knew it is washing away. But the story is bigger than journalism alone, and bigger than a story alone can tell. (Image borrowed from the brilliant Despair.com.) We who care about journalism are asked to join the Save Journalism Project, and its fight against Big Tech. Their pitch begins, and adds, On the first point, we […]
Journalism as we knew it is washing away. But the story is bigger than journalism alone, and bigger than a story alone can tell. (Image borrowed from the brilliant Despair.com.)

We who care about journalism are asked to join the Save Journalism Project, and its fight against Big Tech. Their pitch begins,

and adds,

On the first point, we should note that journalists have been working for magazines, broadcasters, newsletters and themselves for many dozens of years. So journalism isn’t just about newspapers. Also, because so many journalists have long made livings in those other media, the loss of work is far greater than the 2,400 gone from newspapers. It’s truly massive. I don’t know any field where the loss of paying jobs is larger on a percentage basis. Not taxi driving, not hospitality, not retail, not manufacturing… not anything I can think of. (Well, maybe nuns. I don’t see many of those these days.)

We should also respect the simple fact that now there is more journalism than ever: in blogs, social media, podcasting, and other places. Most of those kinds of journalism don’t pay, but that doesn’t disqualify the work from the label. Hell, I’m committing journalism here and this doesn’t pay.

“The story of big tech’s threat to journalism” (what the Project wants us all to tell) is also something of a red herring because it distracts our attention from causes much bigger than Big Tech.

Every new technology “works us over completely,” Marshall McLuhan says (in The Medium is the Massage). And no new medium, no new technologies, have ever worked us more than the digital kind. The change began with digital tech and integrated circuits and then went absolute with the Internet. Together, digital technologies and the Internet have radiacally changed our species, our civilization, and our planet.

Not long ago, in a conversation about this with Joi Ito, I asked him how big he thought the digital transformation was. Bigger than broadcast? Print? Writing? Speech? Stone tools?

“No,” he replied. “It’s the biggest thing since oxygenation.” In case you don’t remember, that happened between about two and a half billion years ago. (Joi also writes about it here.)

So, while journalism matters enormously, it’s just one casualty of digitalization. And, let’s face it, a beneficiary as well. Either way, we need to understand the whole picture, which is about a lot more than what journalism sees happening in the mirror.

Here’s one outfit working on that bigger picture. I‘m involved with it.

I also don’t expect most journalists to take much interest in the subject, because it’s too big, and it doesn’t make full sense as a story, which is journalism’s stock in trade. (I explain a bit about journalism’s “story problem” in this TEDx talk.)

Still, some journalists are on the case, including me. Love to have others join in. But please don’t bother if you think Big Tech is alone to blame. Because the story is bigger than that, and far more than a story.

I just copied and pasted this post from here in Medium, where I posted it in July 2019. It expands on a post now archived here. It’s kinda sad that not much has changed over all that time.

Thursday, 18. April 2024

Mike Jones: self-issued

OpenID Federation Session at April 2024 IIW

John Bradley and I convened a session on Trust Establishment with OpenID Federation at the Internet Identity Workshop (IIW) on Thursday, April 18, 2024. The material used to drive the discussion was: Trust Establishment with OpenID Federation (PowerPoint) (PDF) The session was well attended and the discussion lively. Numerous people with trust establishment problems to […]

John Bradley and I convened a session on Trust Establishment with OpenID Federation at the Internet Identity Workshop (IIW) on Thursday, April 18, 2024. The material used to drive the discussion was:

Trust Establishment with OpenID Federation (PowerPoint) (PDF)

The session was well attended and the discussion lively. Numerous people with trust establishment problems to solve contributed, including experts from the SAML federation world, people involved in digital wallet projects, and several people already using or considering using OpenID Federation. Thanks to all who participated!


The Pragmatic Engineer

The Pulse #90: Devin reversing ambitious claims

The “world’s first AI developer” tones down expectations and has been outperformed by an open source tool. Also: hiring upticks at Big Tech; a very realistic AI video generator by Microsoft, and more.

The Pulse is a series covering insights, patterns, and trends within Big Tech and startups. Notice an interesting event or trend? Send me a message.

Today, we cover:

Industry pulse. Fintech valuations rising again; pre-earnings layoffs at Tesla and Google; Google fires staff trying to interfere with business; Rippling offering a secondary to its employees, and more.

Devin: Reversing ambitious claims. A month ago, Devin launched with fanfare as “the world’s first AI developer,” claiming that it “even completed real jobs at Upwork.” Upon closer inspection, this claim did not hold up. The company behind Devin had since toned down expectations. Also: open source solution AutoCodeRover is offering even better performance than Devin’s closed-source and not-yet-publicly available tool. This space is commoditizing rapidly.

Microsoft’s disturbingly realistic AI video generator. Microsoft Research showcased a tool that generated very realistic videos from a single image. The #1 use case will surely be fraudulent deepfakes generation. This development could well speed up AI regulation in several countries.

Hiring upticks at Meta, Netflix and Amazon? Data from interview preparation website interviewing.io suggests hiring is back at full speed at Meta, Netflix and – possibly – Amazon.

1. Industry pulse Fintech valuations rising again

Read more

Tuesday, 16. April 2024

Mike Jones: self-issued

OpenID Presentations at April 2024 OpenID Workshop and IIW

As has become traditional, I gave the following presentation at the Monday, April 15, 2024 OpenID Workshop at Google: OpenID Connect Working Group Update (PowerPoint) (PDF) I also gave this invited “101” session presentation at the Internet Identity Workshop (IIW) on Tuesday, April 16, 2024: Introduction to OpenID Connect (PowerPoint) (PDF)

As has become traditional, I gave the following presentation at the Monday, April 15, 2024 OpenID Workshop at Google:

OpenID Connect Working Group Update (PowerPoint) (PDF)

I also gave this invited “101” session presentation at the Internet Identity Workshop (IIW) on Tuesday, April 16, 2024:

Introduction to OpenID Connect (PowerPoint) (PDF)

The Pragmatic Engineer

What is Security Engineering? Part 1.

A deep dive into the ever-changing field of security engineering; a domain that can feel intimidating to some software engineers. With Nielet D'Mello, security engineer at Datadog.

👋 Hi, this is Gergely with a subscriber-only issue of the Pragmatic Engineer Newsletter. In every issue, I cover challenges at Big Tech and startups through the lens of engineering managers and senior engineers. To get articles like this in your inbox, every week, subscribe:

Subscribe now

Q: “As a software engineer, I’d like to learn more about security engineering. What’s a good way to understand this vast field?”

Security is so important in our industry. There’s frequently news stories about security incidents, like the authentication provider Okta which was breached, then responded poorly and got schooled on “Security 101” practices by its own customer, Cloudflare. The criticism that followed for Okta was inevitable and also deserved, as it essentially sells security. But what about engineers who want to build things securely, where do they start?

I figured there’s no better place to find out than by asking a security engineer, so I reached out to Nielet D'Mello. She’s a security engineer at Datadog, whose job is incorporating security into products from the very start of the development process. Nielet has been working in the security domain for nearly a decade, and before that she was at Intel, where she worked closely with the security team, as well as at McAfee, in consumer and enterprise security products. Nielet’s also speaks at security conferences – here’s her 2023 talk on security design and guidance at scale.

In today’s issue, Nielet takes us through:

Myths and misconceptions about security engineering. Common misconceptions, like that security is only security engineers' responsibility, or that security through obscurity is sufficient, and other myths.

History of security engineering. Security engineering’s evolution since the 1990s; especially network and perimeter defense up to today.

The present. A transformation to a proactive approach, and a shift to “decentralized security.”

A mental model. Seven core dimensions for thinking about application security, with a close look at each one.

Towards a Secure SDLC. An approach to make all steps of the software development lifecycle (SDLC,) “security-first.”

In Part 2, we cover:

Defining the criticality of a system.

Scoring a system’s criticality.

Threat modeling.

Security paved roads.

“Defense in depth,” “least privilege,” and “zero trust.”

As a note, throughout this article we cover application security engineering (aka, “AppSec.”) This is the most common type of security engineering at tech companies building software products. Other specializations within security engineering include cloud security (focusing on cloud infrastructure security,) infrastructure security (securing hardware, operating systems, middleware,) and even physical security (physical access controls and surveillance.) These topics are out of scope for this series.

With that, it’s over to Nielet.

Common security engineering terms

Hi! We use three terms frequently in this article, so let’s start by defining them:

Vulnerability: An exploitable flaw or weakness in a system’s design, implementation or deployment

Threat: The potential for a threat actor to exploit a vulnerability

Risk: Loss or damage that could occur when a threat actualizes

Security engineering vs software engineering

How intertwined are security engineering and software engineering?

When it comes to software engineering, there’s nothing too special about security. Yet, its extensive depth, breadth, and nuance, mean the security domain has long felt intimidating to engineers. But it has existed for as long as software engineering; so, why does security engineering still feel like an “emerging” field? 

It’s due to software engineering’s ever-increasing complexity: distributed systems, microservices, cloud computing, Artificial Intelligence (AI,) and more. Security engineering aims to stay ahead in this dynamic, ever-evolving threat landscape, and businesses are starting to prioritize it more.

Some statistics reveal why investing in security is increasingly important:

$4.45M: global average cost of a single data breach in 2023, a 15% rise over 3 years

16% more application security attack surfaces. In 2023 alone, this meant 29,000 new vulnerabilities were identified, which organizations need to defend against.

A security engineering organization is usually tasked with:

Risk prevention and detection: Aim to defend an organization's assets: its data, applications, code, infrastructure, etc.

Response and recovery: react to threats and remediate attacks.

1. Myths and misconceptions about security engineering

I’ve observed several common misconceptions, and this article seems like a good place to debunk them.

Myth #1: Security is only the responsibility of security engineers

This is surprisingly common, but not exactly true. Security engineers are stewards of the organization's overall security posture, but realistically, they can never keep up with all developments in the product and platform space – just within their organizations! 

Security teams also tend to be lean, meaning there aren’t many engineers. If they focus too much on the weeds; like constantly triaging incidents or security findings, this will take away from high-value work that brings company-wide impact. Examples of high-value work include:

Security design reviews done product-wide

Building and running programs and services for a secure software development lifecycle

Relying solely on a security team to make all security design decisions is a common anti-pattern. Amazon Web Services, in its “AWS Well-Architected” guide, recommends against this practice, and instead suggests:

“Build a program or mechanism that empowers builder teams to make security decisions about the software that they create. Your security team still needs to validate these decisions during a review, but embedding security ownership in builder teams allows for faster, more secure workloads to be built. This mechanism also promotes a culture of ownership that positively impacts the operation of the systems you build. (...)

Common anti-patterns:

Leaving all security design decisions to the security team

Not addressing security requirements early enough in the development process.

Not obtaining feedback from builders and security people on the operation of the program.”

Myth #2: Security through obscurity is sufficient

Security through obscurity is the assumption that safeguarding certain details or functions of a system's operations can guarantee security. The principle is, “if only we know how this thing works, then it will be secure enough because others won’t be able to figure it out.”

This approach leads to a false sense of security! It can also lead to exploits. For example:

You have a web application with an admin panel, and this panel has features like managing users, managing content, and configuring the system. The admin panel has URL endpoints like /admin/user-management, /admin/content-management, /admin/system-configuration. How do you make these endpoints secure? The obvious way is to add authentication. However, this is a lot of effort. A simpler idea is to use obfuscation, remapping URLs to something hard to guess:

Security through obscurity. The newly mapped URLs at the bottom are obscurer, but no more secure than those at the top

In this case, the developer relies on the obscurity of the URLs to prevent unauthorized access. However, all it takes is for the URL endpoints information to leak, or an attacker to brute-force the URLs, and the website can be exploited.

Myth #3: More security measures makes software more secure

It’s tempting to believe, right? Unfortunately, in my experience, it’s simply untrue.

Implementing multiple security measures can enhance the overall security posture of software, but it’s essential to strike a balance between security and usability. For each security measure, carefully consider these things:

Effectiveness

Complexity

Performance impact

Management overhead

Your goal should be that collectively, the security measures provide meaningful protection against threats to the product or platform.

Myth #4: Once secure, always secure

So, your system passed all its security reviews and penetration tests, and you have evidence it is secure. Can you now step away, and assume it will continue to be secure? No!

The threat landscape is constantly changing. Over the past year, there’s been a surge in attacks aimed at businesses and organizations around the world. These attacks intended to damage brands’ reputations, steal sensitive data, seek financial gain, and more. They are often done by ransomware groups, such as BlackCat’s attack on Change Healthcare and Reddit, or mass account hacking through credential-stuffing.

New vulnerabilities and attack vectors emerge regularly. For example, applications built on top of large language models (LLMs) are now susceptible to prompt injection, which is a class of attack against applications built on top of LLMs. They work by concatenating untrusted user input with a trusted prompt constructed by the application’s developer. So, security mechanisms built against existing injection attacks must factor this in, as security measures that used to be effective become obsolete or insufficient against new and advanced threats, rendering software vulnerable.

Things that can introduce vulnerabilities and weaken your overall security posture:

Accumulation of technical debt

Using deprecated components and libraries

Outdated dependencies

Security vulnerabilities in a dependency, framework, library, or service

Zero-day exploits are disclosed vulnerabilities for which no patch is available. These are a special kind of vulnerability, unknown to all consumers of the software. Finding such exploits is very challenging, but organizations with large security teams can do it. For example, Google discovered 93 zero-days in 2023.

Regulatory requirements and industry standards often mandate regular security assessments, audits and updates, to ensure compliance with data protection laws and regulations. Adhering to these requirements may necessitate ongoing security improvements, regardless of the software's initial security status.

Myth #5: Penetration testing by itself ensures security

Penetration testing, aka pen testing, involves simulating real-world security attacks against a system, network, application, or organization's infrastructure. The main goal is to identify and exploit vulnerabilities in a system's security defenses, mimicking the tactics, techniques, and procedures of attacks. Pen testing allows organizations to understand their security posture and to prioritize remediation efforts accordingly.

Downsides of pen testing:

It’s a snapshot of the security posture at a single, specific moment

Costly and labor-intensive

A system deemed secure by a penetration test one day, may become vulnerable the next day to new exploits or changes in the environment. Plus, scoping plays a huge role in the impact of penetration test results. Scoping refers to applications, users, networks, devices, accounts, and other assets that should be tested to achieve the organization's objectives.

When pen tests are incorrectly scoped, broader security issues or systemic weaknesses may be missed, which attackers can exploit. Scoping pen tests correctly means providing enough information for the pen testing team upfront, so they can be productive. Here’s a summary from Jessica La Bouve, Solutions Architect at penetration testing vendor, BishopFox, on the importance of scoping:

“If a criminal has decided to target you, they have infinite time to find your weaknesses. (...) The assessment team has a finite amount of time to identify critical weaknesses. What they’re able to accomplish in that time depends on the amount of information you give them during scoping. (...)

Keeping your pen tester in the dark only makes it harder for them to find weaknesses that you need to fix. Even if an attacker starts from zero, they have plenty of time to conduct reconnaissance and learn a lot about your organization, giving your pen tester a head start means they can get right down to the business of finding the real threats to your systems. Attackers also don’t have any limitations on what they can try. They don’t usually worry about knocking your systems offline, but a pen tester would. To maximize a pen tester’s limited time and balance out the technical limitations placed on them, provide as much information as you can.”

2. History of security engineering

Security engineering teams tend to be lean by design and also by constraints, like the specialized skill sets needed, and budget limitations. This lean approach applies at whatever the scale of a company. 

Security teams are much smaller than product/platform engineering teams, and tend to be “two-pizza teams” of between 5-10 application security engineers. As the security org is small, it focuses on projects and initiatives offering high return on investment in value, risk reduction, and impact terms.

If we look at the evolution of security engineering, there’s been significant shifts over the decades due to technological advancement, changes in threat landscapes, and systems’ increasing interconnectedness. Below are some examples.

The 1990s 

The widespread adoption of the internet led to the development of various secure protocols (SSL, HTTPS,) and measures like firewalls and antivirus software to protect networks and data. The primary focus of security activities was network and perimeter defense, largely due to the dominance of client server architectures. 

The 2000s

Web applications gained popularity and security engineering shifted focus towards securing web applications and the network. As web vulnerabilities like SQL injection, cross-site scripting and buffer overflows became common, so did awareness of and focus on secure coding practices. 

Around the same time, compliance and regulatory frameworks like SOX, HIPAA, and PCI DSS came into effect, and led organizations to boost efforts to comply with security requirements and guidelines.

Early 2010s

Cloud computing created new security challenges, like data privacy, data encryption, secure authentication, access control, and secure infrastructure configurations. The vulnerability landscape evolved in tandem with rapid technological shifts, and security shifted to efforts to automate security testing and remediation.

Mid-2010s, early 2020s

The rise of containerization and microservices architecture, the emerging field of AI and machine learning, and a shift to zero-trust architectures. This means security engineering must deal with increased complexity and more attack vectors.

3. Security engineering today

Read more


Phil Windleys Technometria

Relationships are Entangled

People aren't digitally embodied. As a result their online relationships are anemic. As we think about the kinds of digital identity systems that will help people live authentic online lives, we must build identity system that give people tools to operationalize rich digital relationships.

People aren't digitally embodied. As a result their online relationships are anemic. As we think about the kinds of digital identity systems that will help people live authentic online lives, we must build identity system that give people tools to operationalize rich digital relationships.

Identity is the ability to recognize, remember, and react to people, organizations, systems, and things. In the current web, companies employ many ponderous technological systems to perform those functions. In these systems, we are like ghosts in the machines. We have "accounts" in companies' systems, but no good way to recognize, remember, and react to them or anyone else. We are not digital embodied.

One of great benefits of embodiment is the ability to form and operationalize rich digital relationships. I've written a lot about the nature of digital relationships.

Relationships and Identity

Authentic Digital Relationships

Ephemeral Relationships

Operationalizing Digital Relationships

Relationships in the Self-Sovereign Internet of Things

The Architecture of Identity Systems

Are Transactional Relationships Enough?

Fluid Multi-Pseudonymity

One of the discussions at VRM Day caused me to think about a feature of digital relationships I hadn't considered before. Someone said that if you think about a graph with people (or things, organizations, and so on) as the nodes, the relationships are the edges, like so:

Bi-directional relationship

In this figure Alice and Bob have a bi-directional relationship. This is how I've normally thought about it and how I'd have drawn it. But in today's discussion, someone said that the relationship is shared and that Alice and Bob both control it. But I realized that viewpoint is too simple. Specifically, Alice and Bob each have a different perspective of that relationship and will use it separately.

For example, imagine that Alice is the cashier at a grocery store and Bob is a customer. Alice gives great service, so Bob seeks her out when he shops. Alice on the other hand has no particular recollection of Bob from encounter to encounter. For Alice, the relationship is ephemeral, but for Bob, it's longer term. The nature of each relationship is different. So, we might look at it like this[1]:

Two, uni-directional relationships

But after discussing it some more, I realized that these relationships aren't independent. They're entangled like this:

Entangled relationships

In the example I gave above, as Bob seeks out Alice more and more, Alice might come to recognize him and call him by name, changing the nature of her relationship with Bob. And that may influence the nature of Bob's relationship with Alice. Over time, these interactions influence both relationships. So, while Alice and Bob both have control over their relationship with the other, actions by one influence the other.

I frequently say that we don't build identity systems to manage identities, but rather to manage relationships. The problem with contemporary identity systems is that they are all one sided, controlled by one party—almost always a company. As I've said before, people are not digital embodied and thus have no good way to manage their online relationships. As we strive to build better digital identity systems, I think it's paramount that we build systems that provide people with tools that embody them and provide them with the ability to operationalize their online relationships. These are more than decentralized; they are self-sovereign.

Notes

Peer decentralized identifiers (DIDs) are a great technology for creating bi-directional relationships.

Share

Subscribe now

Monday, 15. April 2024

Just a Theory

Mini Summit: Jonathan Katz on TLEs

The topic of the Postgres Extension Mini-Summit this week is Trusted Language Extensions, a.k.a. TLEs. Jonathan Katz of AWS will be giving a brief presentation on the “TLE Vision and Specifics” at noon Eastern / 16:00 UTC on Wednesday April 17, followed by community discussion.

Coming up this week: the fourth installment of the Postgres extension ecosystem mini-summit. The topic: Trusted Language Extensions, a.k.a. TLEs. Jonathan Katz of AWS will give a brief presentation on “TLE Vision and Specifics” followed by community discussion of the issues TLEs aim to address, what works and what doesn’t, and the role of TLEs in the future of the extension ecosystem.

Join us! Note! that if you reserved a spot at a prior mini-summit, sadly you will need to do so again for each subsequent summit or miss out on reminders from Eventbrite. And if Eventbrite says the event is sold out, rest assured we have plenty more virtual seats! Just send at david@ this domain, ping me on Mastodon or via the #extensions channel on the Postgres Slack or the Postgres Discord.

More about… Postgres TLE Extensions PGConf Summit

Damien Bod

Implement a secure Blazor Web application using OpenID Connect and security headers

This article shows how to implement a secure .NET 8 Blazor Web application using OpenID Connect and security headers with CSP nonces. The NetEscapades.AspNetCore.SecurityHeaders nuget package is used to implement the security headers and OpenIddict is used to implement the OIDC server. Code: https://github.com/damienbod/BlazorWebOidc OpenIddict is used as the identity provider and an OpenID connect

This article shows how to implement a secure .NET 8 Blazor Web application using OpenID Connect and security headers with CSP nonces. The NetEscapades.AspNetCore.SecurityHeaders nuget package is used to implement the security headers and OpenIddict is used to implement the OIDC server.

Code: https://github.com/damienbod/BlazorWebOidc

OpenIddict is used as the identity provider and an OpenID connect client is setup to allow an OpenID Connect confidential code flow PKCE client. The Web application is a server rendered application using Blazor server components implemented using Blazor Web, ASP.NET Core and .NET 8.

Step 1: Init solution from the .NET Blazor samples

The solution was created using the Blazor samples from Microsoft. The .NET 8 BlazorWebAppOidc project was used to setup the solution.

https://github.com/dotnet/blazor-samples/tree/main/8.0/BlazorWebAppOidc

The code sample implements the client profile parts and the CSRF protection. Login and Logout plumbing is also implemented.

Step 2: Switch the OpenID Connect server

OpenIddict is used as the identity provider and so the OIDC client set up needs to be changed. The program file was updated and the OpenID Connect Microsoft Entra ID client was replaced with the OpenIddict client. The client on the server is setup directly in the worker class in the Openiddict server. Both of the setups must match. The client uses an OpenID Connect confidential client with code flow and PKCE.

builder.Services.AddAuthentication(OIDC_SCHEME) .AddOpenIdConnect(OIDC_SCHEME, options => { // From appsettings.json, keyvault, user-secrets // "OpenIDConnectSettings": { // "Authority": "https://localhost:44318", // "ClientId": "oidc-pkce-confidential", // "ClientSecret": "--secret-in-key-vault-user-secrets--" // }, builder.Configuration.GetSection("OpenIDConnectSettings").Bind(options); options.SignInScheme = CookieAuthenticationDefaults.AuthenticationScheme; options.ResponseType = OpenIdConnectResponseType.Code; options.SaveTokens = true; options.GetClaimsFromUserInfoEndpoint = true; options.MapInboundClaims = false; // Remove Microsoft mappings options.TokenValidationParameters = new TokenValidationParameters { NameClaimType = "name" }; }) .AddCookie();

Note: You could also use the OpenIddict client packages to implement the client. I like to use the defaults.

Step 3: Disable WASM mode

Any web application should protect the session, not just implement authentication using an OIDC server. One of the most important browser protection is the CSP header and a good CSP uses a nonce. Blazor Web using WASM does not support this and so this must be disabled. Remove the WASM part from the middleware.

In the program.cs, update Blazor Web to:

builder.Services.AddRazorComponents() .AddInteractiveServerComponents();

and

app.MapRazorComponents<App>() .AddInteractiveServerRenderMode() .AddAdditionalAssemblies( typeof(BlazorWebAppOidc.Client._Imports).Assembly);

Remove the WASM usage in the UI components. Switch to InteractiveServer mode.

<HeadOutlet @rendermode="InteractiveServer" /> </head> <body> <Routes @rendermode="InteractiveServer" /> Step 4: Add CSP nonce middleware

The CSP nonce can be used in Blazor (Server) components with some extra effort because the Blazor components cannot read the HTTP headers from the responses. The CircuitHandler class can be used for this. A BlazorNonceService class can be created to add the nonce. This class inherits the CircuitHandler implementation.

using Microsoft.AspNetCore.Components; using Microsoft.AspNetCore.Components.Server.Circuits; namespace BlazorWebAppOidc.CspServices; /// <summary> /// Original src: https://github.com/javiercn/BlazorWebNonceService /// </summary> public class BlazorNonceService : CircuitHandler, IDisposable { private readonly PersistentComponentState _state; private readonly PersistingComponentStateSubscription _subscription; public BlazorNonceService(PersistentComponentState state) { if (state.TryTakeFromJson("nonce", out string? nonce)) { if (nonce is not null) { Nonce = nonce; } else { throw new InvalidOperationException( "Nonce can't be null when provided"); } } else { _subscription = state.RegisterOnPersisting(PersistNonce); } _state = state; } public string? Nonce { get; set; } private Task PersistNonce() { _state.PersistAsJson("nonce", Nonce); return Task.CompletedTask; } public void SetNonce(string nonce) { ArgumentException.ThrowIfNullOrWhiteSpace(nonce); if (Nonce != null) { throw new InvalidOperationException("Nonce already defined"); } Nonce = nonce; } public void Dispose() => ((IDisposable)_subscription)?.Dispose(); }

A NonceMiddleware ASP.NET Core middleware service can now be used to read the nonce from the headers and set this in the BlazorNonceService CircuitHandler implementation. NetEscapades.AspNetCore.SecurityHeaders is used to implement the security headers and if a CSP nonce is created, the NETESCAPADES_NONCE http header is set.

namespace BlazorWebAppOidc.CspServices; public class NonceMiddleware { private readonly RequestDelegate _next; public NonceMiddleware(RequestDelegate next) { _next = next; } public async Task Invoke(HttpContext context, BlazorNonceService blazorNonceService) { var success = context.Items .TryGetValue("NETESCAPADES_NONCE", out var nonce); if (success && nonce != null) { blazorNonceService.SetNonce(nonce.ToString()!); } await _next.Invoke(context); } }

The middleware for the nonce is added to the ASP.NET Core services.

builder.Services.TryAddEnumerable( ServiceDescriptor.Scoped<CircuitHandler, BlazorNonceService>(sp => sp.GetRequiredService<BlazorNonceService>())); builder.Services.AddScoped<BlazorNonceService>();

Use the middleware is in the ASP.NET Core pipelines.

app.UseMiddleware<NonceMiddleware>(); Step 5: Add HTTP browser security headers

The NetEscapades.AspNetCore.SecurityHeaders nuget package is used to implement the security headers as best possible for this type of application. The SecurityHeadersDefinitions class implements this. CSP nonces are configuration as well as other security headers.

namespace BlazorWebAppOidc; public static class SecurityHeadersDefinitions { public static HeaderPolicyCollection GetHeaderPolicyCollection( bool isDev, string? idpHost) { ArgumentNullException.ThrowIfNull(idpHost); var policy = new HeaderPolicyCollection() .AddFrameOptionsDeny() .AddContentTypeOptionsNoSniff() .AddReferrerPolicyStrictOriginWhenCrossOrigin() .AddCrossOriginOpenerPolicy(builder => builder.SameOrigin()) .AddCrossOriginResourcePolicy(builder => builder.SameOrigin()) .AddCrossOriginEmbedderPolicy(builder => builder.RequireCorp()) .AddContentSecurityPolicy(builder => { builder.AddObjectSrc().None(); builder.AddBlockAllMixedContent(); builder.AddImgSrc().Self().From("data:"); builder.AddFormAction().Self().From(idpHost); builder.AddFontSrc().Self(); builder.AddBaseUri().Self(); builder.AddFrameAncestors().None(); builder.AddStyleSrc() .UnsafeInline() .Self(); // due to Blazor builder.AddScriptSrc() .WithNonce() .UnsafeEval() // due to Blazor WASM .StrictDynamic() .OverHttps() .UnsafeInline(); // only a fallback for older browsers }) .RemoveServerHeader() .AddPermissionsPolicy(builder => { builder.AddAccelerometer().None(); builder.AddAutoplay().None(); builder.AddCamera().None(); builder.AddEncryptedMedia().None(); builder.AddFullscreen().All(); builder.AddGeolocation().None(); builder.AddGyroscope().None(); builder.AddMagnetometer().None(); builder.AddMicrophone().None(); builder.AddMidi().None(); builder.AddPayment().None(); builder.AddPictureInPicture().None(); builder.AddSyncXHR().None(); builder.AddUsb().None(); }); if (!isDev) { // maxage = one year in seconds policy.AddStrictTransportSecurityMaxAgeIncludeSubDomains(); } policy.ApplyDocumentHeadersToAllResponses(); return policy; } }

The security headers are added using middleware as early as possible in the pipeline. I add the headers for all requests.

app.UseSecurityHeaders( SecurityHeadersDefinitions.GetHeaderPolicyCollection( app.Environment.IsDevelopment(), app.Configuration["OpenIDConnectSettings:Authority"]));

The CSP nonce can now be used in the Blazor components and scripts can only be read using the nonce. Unsecure scripts or unsecure inline scripts should never be read anywhere in a browser application.

<pre class="wp-block-syntaxhighlighter-code"> <a href="http://_framework/blazor.web.js">http://_framework/blazor.web.js</a> </body> </html> @code { /// <summary> /// Original src: https://github.com/javiercn/BlazorWebNonceService /// </summary> [CascadingParameter] HttpContext Context { get; set; } = default!; protected override void OnInitialized() { var nonce = GetNonce(); if (nonce != null) { BlazorNonceService.SetNonce(nonce); } } public string? GetNonce() { if (Context.Items.TryGetValue("nonce", out var item) && item is string nonce and not null) { return nonce; } return null; } }</pre>

The applications can be started and the authentication and the session protection can be validated. Using the WASM mode in Blazor Web requires a weaker security setup and you need to disable the CSP nonces. This is not a good idea.

Links

https://github.com/dotnet/blazor-samples/tree/main/8.0/BlazorWebAppOidc

Using a CSP nonce in Blazor Web

https://github.com/damienbod/BlazorServerOidc


@_Nat Zone

NHK「最深日本研究~外国人博士の目~」からメタバースとアイデンティティとプライバシーについて考えてみる

4/13(土) 午前8:10-午前8:40にNHKで「最深日本研究~外国人博士の目~」という番組が放映されました。4/20(土) 午前8:39 までNHK+で視聴可能ですでは触れられているようです。また、日本における「理想」の実体化としての「カワイイ」の考察などもされています。昨…

4/13(土) 午前8:10-午前8:40にNHKで「最深日本研究~外国人博士の目~」という番組が放映されました。4/20(土) 午前8:39 までNHK+で視聴可能です2では触れられているようです。また、日本における「理想」の実体化としての「カワイイ」の考察などもされています。昨今、アイデンティティとプライバシーの国際標準化の分野でもメタバースが関心の対象になってきているところなので、とても参考になりました。

以下に番組概要のメモと、その後にアイデンティティやプライバシーの観点での考察を記載します。

番組概要

番組の概要はこんな感じです。なぐり書きメモなんで整っていませんが、まぁなんとなくは分かるでしょう。NHK+での番組公開終了までは、ぜひ番組をご覧になることをおすすめします。

(出所)NHK+ 「最深日本研究~外国人博士の目~」https://www.nhk.jp/p/ts/RJ5G2XZ4N3/episode/te/J1K96JLJ9K/

導入部 2000年代以降ネット上に広がったもう一つの世界、メタバース。人々はアバターと呼ばれる自らの分身を使い見知らぬ人との出会いを楽しむ。 その中で、日本人コミュニティではバ美肉が目立つ。 スイスの文化人類学者リュドミラ・ブレディキナ氏(31)3はこのことに興味を持つ。 なぜ日本人男性は美少女キャラクターになりたがるのか、その答えを求めて研究 アニメ・和食・J-POP日本文化に注がれる世界からの熱い視線。そして、日本を愛しわれわれには無い目線でより深く日本を知ろうとする外国人研究者 彼らに日本はどう写っているか。バーチャル人類学者がわれわれの知らない日本の真の姿を解き明かす〜最深日本研究 第1章:最新メタバース体験施設を調査 スイスにはこのような施設はない メタバースを体験するためのすべてが揃っている 初心者には案内人もついている。やはり美少女キャラクター 一般視聴者に向けたメタバースの紹介 アバターの選択〜好きなアバターになる〜アイデンティティの表出 バーチャル空間では一瞬で何十万もある世界を移動できる そこで景色を楽しんだり、人々とお酒を飲んで交流したりなど、現実世界と同じように楽しめる メタバースの利用者は世界中でおよそ4億人 花火をましたから見上げるなどここでしかできない体験 第2章:日本のバーチャル世界を研究したい ミラさんがバーチャル世界の研究を始めたのは4年前 研究調査で驚きを持って知った言葉「バ美肉」(バーチャル美少女受肉)男性が美少女キャラを持つときに使われることば 「受肉」には西洋では宗教的な意味があるが、バ美肉たちは宗教に全く関心がない。そこにミラさんは興味を惹かれた。 調査をすると驚くべきことがわかった。男性のおよそ8割が美少女キャラクターを使っている 女性 78.3% 男性 15.2% その他 6.5% その見た目には大きな特徴:アニメに出るような可愛い女の子。これに対して西洋では、クールでセクシー。カワイイとは違って強い女性という印象を受ける。

(出所)NHK「最深日本研究~外国人博士の目~」https://www.nhk.jp/p/ts/RJ5G2XZ4N3/episode/te/J1K96JLJ9K/

ミラさん:わたしは西洋で育ち、女性として美しくエレガントでありたいと思っていた。なぜ日本の男性はカワイイと言われたいのか。西洋人としてこの異なる視点をもっと理解したいと思います。 文化人類学の紹介 20世紀初頭に一つの学問領域として確立。 クロード・レヴィ・ストロース ルース・ベネディクト 梅棹忠夫 机に向かうだけではなく、現地に赴き直接人と触れあるフィールドワークという手法で人間とは何かを探求する学問 人類学者は世界各地の民族を訪ね、彼らとともに暮らし、彼らの生活をつぶさに観察、自分の属する社会と比較することで、その共通点や違い、多様性を明らかにしてきた バーチャル世界という新たな地平 最初SNSを使いバ美肉たちに話しかけたが、部外者であるわたしは信用されなかった。わたしが西洋人であり女性だったから。そこで、ミラさんは人類学の手法であるフィールドワークを使ってバ美肉たちに接近。自分のアバターを作ることは重要だった。バーチャル世界で彼らの信用を得ることができたから。 アバターを纏うことでバ美肉たちと同じ目線に立ち直接ふれあい始めたリュドミラ ときには一緒にライブ配信を行い、オリジナル曲の英訳を買って出た そして1年ほどかけて、徐々にコミュニティの中に入っていった 第3章 声までカワイイってどういうこと? 両声類4の紹介 声は男性のままか音声変換器を使うのが一般的 しかし、あまちじょんこ(Youtuber @johnko_amachi X: @johnko20100711) などは自らの超絶技巧で声を変える。 中の人は、じょんアニキ。 めざすもの:天真爛漫で無邪気で、こういう小さい子っているよねという女の子をしたい。 高い声にするだけだと、男の人の高い声になってしまう。声は楽器と一緒なので楽器の形を変えてあげる。そうやって音質を変えて、その上で音の高さを変える。両方が合わさってじょんこの声になる 2年かけ、理想の可愛い声を獲得した 「別の自分になれるってめちゃくちゃ楽しくて。で、なんかそれで、ただ楽しいということを突き詰めていく。天才じゃないから頑張るしかない。」 カワイイ動きのトレーニング。いざやっていむと難しい。ちいさい子の気持ちになって動かないといけない。たとえば、歩くときに右の足を出すと右に重心がよってしまう。これを繰り返すとふらふら見える。このふらふらした動きが、見ている人からしたら守ってあげたいというので可愛く見える。 ミラさん 歩く時にかわいいを意識したことがなかったのでとてもおもしろいと思った 前は日本の男性といえばサラリーマンかオタクという表層的なステレオタイプなイメージしかなかった。しかし、研究をしてみると、日本人男性の多様性や日本文化の寛容性を知りとても面白いということがわかってきた ミラさんの生い立ちの説明 ロシア生れ。12歳でスイスに移住。フランス語の勉強に日本の漫画。「天使禁猟区」5。今までこのような美しい絵を見たことがなかったので新しい世界が開けた。 27際のときに日本文化を学ぶためにジュネーブ大学の修士課程に。 論文を発表「Babiniku」6 日本の伝統芸能の一つである歌舞伎の女形と同様に男性が美少女キャラクターを演じる類似点 人形浄瑠璃とアバターをあやつるバ美肉の共通性など日本の伝統文化を通してバ美肉を考察した論文は大学から学術賞を受けた 日本の文化はとても興味深く、趣味と実益を兼ねた研究をしたいと考えた ブレンダ・マーフィー教授 「彼女の研究の魅力は西洋で主流ではない”日本文化”に着目している点です」 バーチャル人類学はとても新しい分野であり、彼女はグローバルな文化への挑戦者といえる リュドミラが来日するのはこれで6度目。街に行けば目につくのは美少女キャラクター。看板や飲食店など町のいたるところで見かける。美少女キャラクターは秋葉原だけでなく日本文化で不可欠な存在 アニメや漫画の専門店に立ち寄り、次々に新しいものが生まれる日本のサブカルチャーもミラさんにとって研究の一環 第4章 美少女キャラに会いたい アイドルVTuber のらきゃっと (@NoraCat0415 ch. 登録者86000人以上)。ファンネーム「ねずみさん」 これを生み出したのが「のらねこP」(@PIMR, X: @VR_Produce_Nora)インタビュー のらきゃっとにガチ恋だと言っているファンとの関係についておしえて のらちゃんがちょっとドジなところもあってでも強くて可愛くてというのはファンがそう思っているから作られたという部分が多くて、ネズミさんたちがのらきゃっとという存在にそうあってほしいという願いのもとに作られたのでファンからの影響はとても強い のらきゃっとがファンから可愛いと言われるとどう感じるか 娘が公園で仲の良い友達と遊んでいて幸せそうだと親が感じるのに似ている のらきゃっとが生まれて人生は変わったか? 今までの仕事に行って帰ってきて家で寝るだけという生活でそんなに友達もいないというような人生から、本当に楽しく話せる友達が10倍以上に増えた。誰かと話したいときに話せるので寂しいと思う瞬間がなくなった かわいい美少女になれるって最高ですね「みんななれる〜」 ファンから求められることで美少女キャラはよりかわいくくなり、その美少女を生み出した彼の生活も豊かにしている。今日いちばん重要な発見だった 第5章 美少女になる理由を知りたい バーチャル美少女とファンとの交流はさらに進化。現実世界のファンが集まる店。友人のバ美肉が出演すると聞き訪ねた。「新宿ねむバー」 バーチャル美少女ねむさん(Youtube @nemchan_nel, X @nemchan_nel )=『メタバース進化論』(技術評論社)の著者で、ミラさんの研究の頼もしい協力者 ミラさん:「直接あって美少女やメタバースとか私達が情熱を注いでいることについてもっと話そう」 ねむさん:「楽しそう。たくさん話すのを楽しみにしてるよ」 ミラさん:「ではリアル世界で」 ねむはプライベート時間の殆どをメタバースで過ごし、専門書7を出版するほどこの世界について考え続けてきた人物。 ミラさん:直接美少女キャラクターになる理由を訊いてみたい 匿名で活動したいというねむさんの希望により映像と音声を加工しています→モザイクの大きさはかなり調整したらしい SNSを通じ知り合ったのは4年前。ねむさんとはメタバースユーザの生活実態を一緒に調査したことも。 ミラさんによるねむさんへのインタビュー 初めてねむになったときどう思ったの? すごく恥ずかしかった。 罪悪感〜違う自分になるというのが悪いことをしているような感覚だった でもだんだんいつもの自分では出てこないような言葉が口から出てきた 美少女になると自己表現も変わるということか? 「現実の自分だとカッコつけちゃうのかな」 「そこまで心揺れ動かない結構冷静な人間だと自分のことを思っていたけど、ねむちゃんのときはそういうのから開放されてもっと自由に表現できる」 あなたにとっての美少女ってどういう存在なのか? “カワイイ”という概念を具現化した存在が美少女キャラクターだと思っている 肩書とか現実世界のしがらみを剥ぎ取ると人間の本質ってすごくカワイイ。魂って実はカワイイと思っていて、カワイイはその人の本質を褒めている気がする。 自信になるというか、ありのままでちゃんと意味があるんだと思える気がする なぜ日本人男性が美少女になりたいと思うのか。その答えを求めてフィールドワークを続けてきたミラさんの今考える結論 日本人男性は美少女になりカワイイを獲得することでストレスから開放される。 なぜならカワイイは失敗しても許されるし責任を負わなくていいからです。 私は日本文化から生まれたカワイイに可能性を感じました しかし私は人類学者としてコミュニティを一般化するのではなく、これからも多くの人たちの声に耳を傾けたいと思っています

エンド:若き人類学者が日本のバーチャル世界に分け入っていく

考察 アイデンティティとバ美肉

アイデンティティという言葉は番組の中に出て来ませんが、多くの発言は「アイデンティティ」関連だとわたしの中では整理されました。たとえば、じょんこさんの「別の自分になれるってめちゃくちゃ楽しくて。で、なんかそれで、ただ楽しいということを突き詰めていく。」という発言は、現実社会の仕事だとか交友関係という既存のコンテキストを超えた新たなコンテキストでの自我の確立とその表出(自観するアイデンティティの表出)およびそれがコミュニティ内で他の人に観測されるもののフィードバック(他観)という関係性の享受から幸福感を得ているように見えます。(自観と他観のあたりは、「非技術者のためのデジタルアイデンティティ入門」をご参照ください)

類似のことは、ねむさんの「(最初ねむになったとき)すごく恥ずかしかった。罪悪感〜違う自分になるというのが悪いことをしているような感覚だった。でもだんだんいつもの自分では出てこないような言葉が口から出てきた。」「ねむちゃんのときはそういうのから開放されてもっと自由に表現できる。」という言葉にも出てきます。

一方、のらねこPの「のらちゃんがちょっとドジなところもあってでも強くて可愛くてというのはファンがそう思っているから作られたという部分が多くて、ネズミさんたちがのらきゃっとという存在にそうあってほしいという願いのもとに作られたのでファンからの影響はとても強い。」という発言は、自観と他観のズレの最小化を、表出する属性の調整によって他観の方を変えて行うという「通常」のやり方とは逆で、他観の方に自観をあわせてしまうというアプローチに見えます。根源的自我に他観を合わせに行くのではなく、自我の方を調整してしまうわけですね。ただ、これが本当に自分の自我だとかなり心理的負荷が大きくなってくるのではないかと思われます。心理的負荷を減らすには、自分の中からその人格を外だししてしまって客観化してしまうことが必要であるようにも思えます。

このあたりが、のらねこPの「娘が公園で仲の良い友達と遊んでいて幸せそうだと親が感じるのに似ている」という発言からも表出されているように感じられました。

ここの部分で、ミラさんは「ファンから求められることで美少女キャラはよりかわいくくなり、その美少女を生み出した彼の生活も豊かにしている。今日いちばん重要な発見だった」と述べています。

免罪符としては機能しない美少女キャラクターとプライバシー

次に検討してみたいのが、ミラさんの「日本人男性は美少女になりカワイイを獲得することでストレスから開放される。なぜならカワイイは失敗しても許されるし責任を負わなくていいから。」という発言です。わたしの解釈違いかもしれませんが、カワイイが免罪符的に機能するので、日本人男性はカワイイを獲得したいと言っているように聞こえます。「日本人男性は」としているので、この「カワイイ」の概念とその通用する範囲が日本にまず限定されているのだろうということは想像できます。そのうえで「カワイイ」が免罪符的に機能するのかと言うと、これはちょっと難しいところだと思っています。

「カワイイ」を体現するものとしては、いわゆるアイドルVTuberなどがいると思います。ところが、その免罪符を持っているはずの彼女たちは、時として誹謗中傷されたりコミュニティから叩かれ、「卒業」を余儀なくされたりしているのは、このあたりをちょっと追っている人ならよくご存知のことと思います。もし「カワイイ」が免罪符として機能するならばこういうことは起きないと思うのです。

おそらく理想としての「カワイイ」には人々の期待がつまっていて、その期待を裏切ると叩かれるという構図があるのではないかとも思えます。つまり「カワイイ」が免罪符として機能するわけではない。ある意味、「”カワイイ”という概念を具現化した存在としての美少女キャラクター」は現代の巫女であり、古代と同じく、期待を裏切ったときにはスケープゴートにされる存在だと言えるかもしれません。で、わたしはスケープゴートって作ってはいけないと思うんですよね。

現代の巫女がスケープゴート化してしまう原因の多くは、コンテキスト外の情報の当該コンテキストへの混入です。たとえば、誰かからのチャットが画面に映り込んでしまったとか、他の人の声が混入してしまったとか、自分の顔が反射で映り込んでしまったとかですね。今後は他のワールドでの行動が当該ワールドにレポーティングされてしまうというような形の情報混入も起きてくることでしょう。場合によっては悪意の人が監視用のキャラクターを各所に配置して意図的にこれをやってくることも考えられます。中の人と、その人が運用する異なるアバターの名寄せとかもこの中に入ってきます。

番組の中でねむさんは「匿名で活動したい」としてモザイクをかけて声も変えて登場していますが、これも望まない名寄せであるとか情報の混入を防ぐ取り組みの一環と整理することもできます。

こうしたことをどうやって防ぐかとか取り締まっていくかなどが、メタバースにおけるプライバシーの尊重という意味では重要性を増してくると思われます。メタバースプライバシーの国際標準作成作業が始まっている背景でもあります。

しがらみを削ぎ落とした存在としての「ありのままの自分=美少女キャラクター」

さて、ではなぜ日本人男性はバ美肉化するのか?上記のことを勘案すると、ミラさんのいう「日本人男性は美少女になりカワイイを獲得することでストレスから開放される。なぜならカワイイは失敗しても許されるし責任を負わなくていいから。」というのはちょっと違いそうです。

番組内でねむさんは「”カワイイ”という概念を具現化した存在が美少女キャラクターだと思っている。肩書とか現実世界のしがらみを剥ぎ取ると人間の本質ってすごくカワイイ。魂って実はカワイイと思っていて、カワイイはその人の本質を褒めている気がする。自信になるというか、ありのままでちゃんと意味があるんだと思える気がする。」とおっしゃっておられます。現実世界のしがらみを剥ぎ取った人の本質は「カワイイ」であり、そのありのままの自分の自我「自観」を表出すると、日本文化においては美少女になるということでしょうか。そうかもしれません。なにせ、8割近くのひとがそうあろうとするのですから。

番組では、各被取材者ごとに数時間のインタビューがあったのがそれぞれ5分くらいにまとめられてしまっているようです。削ぎ落とされたものの中に、このあたりに光を当ててくれる発言も多くあったことでしょう。その意味で、アフタートークも準備されているようなので、それに期待したいと思います。

【ガチ勢各位】
実際にはみんな2 ~3時間喋った内容をそれぞれ一般人向けに5分くらいにまとめられてるので、ガチ勢のみんなとしたはぶっちゃけ物足りない面もあると思う。という訳で、アフタートーク企画中です!!! #最深日本研究 https://t.co/db9ToxqQpl

— バーチャル美少女ねむ/Nem4/13 NHK「最深日本研究」出演! (@nemchan_nel) April 14, 2024

(追記)アフタートークの日時が決まったようです。4/20(土)22:00〜。わたしはサンフランシスコからの飛行機の時間と微妙なのですが…。アーカイブは公開されるのだろうか…。

【緊急ライブ告知】4/20 (土) 22:00~
ゴメン全然喋り足りない!!! #最深日本研究 出演者4名が、番組には収まりきらなかった”バ美肉”や活動への想いなど、時間無制限で喋り倒します!

NHK「最深日本研究」非公式アフタートーク【ねむ✕ミラ✕のらねこP✕じょんこ】https://t.co/2zG4EbuVGO pic.twitter.com/0G2R0zHGEo

— バーチャル美少女ねむ/Nem4/13 NHK「最深日本研究」出演! (@nemchan_nel) April 15, 2024

<変更履歴>

ミラさんのことを博士と書いていましたが、まだ博士課程在学中のようですので「ミラさん」の表記に変えました。 アフタートークの日時を追加しました。 サムネがFacebookだとはねられてしまうので差し替えました。 「ブレディキナ」を「プレディキナ」と表記してしまっていたのを修正

Sunday, 14. April 2024

Kent Bull

KERI Update: Major release and Branch Strategy Change

The WebOfTrust community recently released a major update to the Key Event Receipt Infrastructure (KERI) and Authentic Chained Data Containers (ACDC) implementation as a coordinated release across the suite of WebOfTrust repositories. Coordinated Release This resulted in the following release versions: Branch Strategy Change Recent Blog Posts See Nuttawut Kongsuwan’s […]

The WebOfTrust community recently released a major update to the Key Event Receipt Infrastructure (KERI) and Authentic Chained Data Containers (ACDC) implementation as a coordinated release across the suite of WebOfTrust repositories.

Coordinated Release

This resulted in the following release versions:

Repository: KERIpy (used for witnesses, command line-managed decentralized identities, and as the core library for the Python KERY space)
Version: 1.1.12 (PyPi) – Apr 9, 2024 Repository: KERIA (agent server used for individual decentralized identity agents – your data sits here)
Version: 0.2.0.dev0 Pre-Release (PyPi) – Apr 11, 2024 Repository: SignifyPy (edge-signing client implementation – your keys sit here)
Version: 0.1.0 (PyPi) – Feb 13, 2024 Branch Strategy Change “development” branch merged to “main” as the old GitFlow style branching strategy was dropped in favor of trunk-based development (single main branch strategy). See the keripy#726 GitHub discussion for the rationale. This occurred for the following repositories: Python KERIpy KERIA SignifyPy Typescript SignifyTS

Recent Blog Posts

See Nuttawut Kongsuwan’s explanation of how to use KERI in his “The Hitchhiker’s Guide to KERI. Part 3: How do you use KERI?

Saturday, 13. April 2024

Michael Ruminer

Refocusing a Career

Now for something entirely different. I write only occasionally on a topic of interest to me, verifiable credentials. Today it is something entirely different, my career. Most specifically, on how it may be time for a refocus. A movement from software development life-cycle process back toward a new more technical focus, AI. I have, in some ways, refocused my career a couple of times. It was

Now for something entirely different. I write only occasionally on a topic of interest to me, verifiable credentials. Today it is something entirely different, my career. Most specifically, on how it may be time for a refocus. A movement from software development life-cycle process back toward a new more technical focus, AI.

I have, in some ways, refocused my career a couple of times. It wasn’t a full restart or anything; just moving into new directions within software development. Some were more a natural progressions than a refocus. This time it would certainly be a refocus.

What I do and what I have done?

For the last several years I have been a software technical release manager focused on not only getting software out the door but on process improvement. That is not where I started my career. It all started as a system admin, then a programmer, then getting involved in migrations and software develpment for mitigating Y2K, on to a software team lead. Then came the first refocus, I got deeply invested in a new Microsoft product Team Foundation Server. This took me down a fun rabbit hole where I spent some years consulting on implementations and use of the product. I spent a few years of my consulting as a Micrsoft MVP on the product, which was very rewarding. Ultimately, with a focus on version control, automated builds and deploys, and release processes with the product and in general, I made the next refocus. I moved into release management. Decidely less technical, but something I have now enjoyed for a number of years. I find the cross-functional nature rewarding and plenty of opportunity to still practice process improvement.

Why the consideration of a refocus?

I see it as a dying domain. I have found that the role is late to join organizations and more easily released from those same organizations. In my experience, companies find unexpected value in the role and can really enjoy having the “glue” bring higher quality and improved flow to production and beyond, but “enjoy” is too much of an operative word here. When belts are tightened it is one of the first roles to go — seen as a luxury not a requirement. I also see diminishing new opportunities in the marketplace. All this places downward pressure on compensation. Additionally, upward progression from the role is often difficult. So it may be time for a full refocus, again.

Too technical for the role?

It wasn’t too long ago I was told as a final feedback for a failed pursuit of an open release management position:

You are too technical.

Hmmm… I knew what they meant, but I had never had my quals work against me- not that I know of. Most find it to be a boon. It made me think. With all the considerations of why there might need to be a refocus, is it time to shift my momentum?

Where to next?

I really enjoy what I do and would hate to give it up but the writing might be on the wall. So, as I survey the landscape, I say to myself “what might be next”. Where is the best opportunity to delve into new things and make an impact for organizations? Is it the AI domain? Can I go from zero to hero? What does that look like? How long would it take and what path is there? I see a lot of potential in Microsoft’s Azure AI offerings and the ones that will inevtiably come down the line. Plus, I have a long history in the Microsoft stack. Let’s see where the next few years take me. Regardless of anything, I’ll certainly be digging deeper into the practical applications and implementations of the technology all why continuing my enjoyable current role in software life cycle.

Back to a common topic

Now, if I can come up with an AI realm play that ties in verifiable, portable, credentials. What will be bigger in the future than identity and AI? Nothing.

Friday, 12. April 2024

Mike Jones: self-issued

Fully-Specified Algorithms Presentation at 2024 OAuth Security Workshop

I gave a presentation on Fully-Specified Algorithms for JOSE and COSE at the 2024 OAuth Security Workshop in Rome. The slides used to update participants on the progress of the work are available as PowerPoint and PDF. Thanks to the organizers for another great OAuth Security Workshop! And special thanks to the colleagues from Fondazione […]

I gave a presentation on Fully-Specified Algorithms for JOSE and COSE at the 2024 OAuth Security Workshop in Rome. The slides used to update participants on the progress of the work are available as PowerPoint and PDF.

Thanks to the organizers for another great OAuth Security Workshop! And special thanks to the colleagues from Fondazione Bruno Kessler who did a great job with local arrangements in Rome!


COSE “typ” (type) Header Parameter Specification in RFC Editor Queue

I’m pleased to report that the COSE “typ” (type) Header Parameter Specification has been approved by the IESG and is now in the RFC Editor queue. The version approved by the IESG and sent to the RFC Editor is: https://www.ietf.org/archive/id/draft-ietf-cose-typ-header-parameter-05.html It joins CBOR Web Token (CWT) Claims in COSE Headers in the RFC Editor queue. […]

I’m pleased to report that the COSE “typ” (type) Header Parameter Specification has been approved by the IESG and is now in the RFC Editor queue.

The version approved by the IESG and sent to the RFC Editor is:

https://www.ietf.org/archive/id/draft-ietf-cose-typ-header-parameter-05.html

It joins CBOR Web Token (CWT) Claims in COSE Headers in the RFC Editor queue. Because of the reference to this spec by CWT Claims in Headers, they form a cluster, and therefore will become RFCs at the same time.

Thursday, 11. April 2024

The Pragmatic Engineer

The Pulse #89: The end of Hopin

In only 5 years, Hopin went from zero to a $7.7B valuation, and back to zero again. Also: Bending Spoons’ startup acquisition model, hiring upticks at Meta, Netflix and Amazon, and more

The Pulse is a series covering insights, patterns, and trends within Big Tech and startups. Notice an interesting event or trend? Send me a message.

Today, we cover:

Industry pulse. The first mass layoffs at Apple since 1997 (or not?); amateurish URL rewrite at X (formerly Twitter); never-ending job interviews for engineering executives, and more.

The end of Hopin. It took Hopin just two years to become the fastest-ever growing European startup by valuation. Four years later, the company is no more. The final valuable parts of Hopin are being sold, and all staff are expected to be let go. Exclusive details on the StreamYard sale.

Weekend maintenance kicks an Italian bank offline for days. It is now day five that Italian bank Sella has its apps and internetbank down, after a weekend systems update went south. The problem seems to be database-related: “something, something Oracle.”

Adyen, the only major Fintech with zero mass layoffs? All major Fintech startups have let go of some staff over the past two years, except Adyen. Meanwhile, the business has quietly become one of Stripe’s biggest competitors. A close look at this curious phenomenon.

1. Industry pulse First layoffs at Apple since 1997, or not…?

Read more

Wednesday, 10. April 2024

David Kelts on ID

“Ripping IDs” Pt 2: The Unwinnable Cost Problem of Identity Verification

In a story on “Ripping IDs Should Go the Way of CDs”, I wrote about the questionable accuracy, usability, and privacy of “ripping” analog IDs into the digital space — a/k/a Mobile Know Your Customer (mKYC). Now I‘ll analyze the “ $1 cost problem” and you’ll see why important steps in mKYC are being skipped. Analog to Digital. Cost, accuracy, usability, and privacy problems of Mobile ID Verifi
In a story on “Ripping IDs Should Go the Way of CDs”, I wrote about the questionable accuracy, usability, and privacy of “ripping” analog IDs into the digital space — a/k/a Mobile Know Your Customer (mKYC). Now I‘ll analyze the “ $1 cost problem” and you’ll see why important steps in mKYC are being skipped. Analog to Digital. Cost, accuracy, usability, and privacy problems of Mobile ID Verification (image cidaas.com)

Services that require identity verification have become increasingly dependent on “scan the front, scan the back, take a selfie, match” for everything from renting a sidewalk scooter to opening bank accounts to provisioning your mobile driver’s license. The cost of inaccuracy in each situation varies greatly. The practical amount of money to spend within these use cases doesn’t — from 0 to 25 cents. Yet most services cost $1. Startups eat the difference hoping the future cost will go down (sorry, it won’t). Many businesses eliminate steps to keep costs down but still roll out a high-demand feature. Avoid offering mobile services that depend on ID and you fall behind. Do it, and accept the risk of fraudulent inputs.

There are both financial risk mitigation use cases and government identity use cases that are currently utilizing mKYC. I’d like to separate these concepts (read to the end to see why).

Note: I do not use the overloaded term document verification. I specifically split document verification into two distinct steps: document authentication (is it from a trusted issuer) and document validation (is the data on it accurate and the account behind it still valid). (See also definitions NIST 800–63). Both of these would be resolved with digital government-signed identity documents.
Cost Analysis of mKYC
Note: I invite those with more insider knowledge of the costs of IDV to connect with me to correct my educated summary of costs below.

There are at least 10 mKYC issues that negatively impact user experience, accuracy, and privacy. mKYC is a technology where we haven’t yet been able to achieve Privacy by Design Principle #4 of full functionality without zero-sum tradeoffs. It’s still cost vs. accuracy vs. user experience vs. privacy (spoiler alert: cost wins). Doing analog-to-digital, this won’t ever change.

The Document Authentication Cost Problem
Is the physical document real or fake? How much does it cost to accurately know?

The cost problem here is simply explained. In physical documents, the goal of winning the card security feature race against fraudsters was to make it expensive, in dollars and knowledge, to produce fake IDs. So cards have proprietary visual, tool-based (8x magnify, multi-spectral lighting), and forensic (destroy the card to detect) security features built into their structure, printing, and lamination. Digitally created for analog use... At great expense, including admirable IP, layered over time… but the fraudsters still caught up, able to replicate lower-cost visual features (and therefore make them acceptable by every bar bouncer on earth).

This gave rise to the supremacy of tool-based authentication for high-risk use cases — use an 8x loupe magnifier; combine white, UV, and IR lights (multi-spectral) to detect secrets built or printed into the card; purposeful design flaws, raised surfaces… amazing technology! Until you try to scan and authenticate it with a white-light phone camera held by a human hand.

Now We Expect to Go Back from Analog to Digital?

Try to ingest a secure physical card back into the digital realm and you must try to detect all three levels of physical card security features with a hand-held camera designed for action photos and portraits taken at focal lengths much further away than human arm length. There’s no UV or IR light, no optical magnification, very few steady cams, and no foolproof UX coaching for a person who isn’t aware of what’s being detected.

Do you see where this is taking us? Massive cost, diminishing return, user friction, bad experience… dropoff. What do mKYC purchasers do? Turn features off to avoid paying for document authentication and user friction.

The Document Validation Cost Problem

Once you’ve scanned an ID document, extracted the data from the front with OCR, decoded the barcode from the back, and skipped sending it for document authentication (lol), you’re still left with a few questions…

Is the document still valid? Is the identity data accurate? Are the attributes fresh or have they changed? Can I ask the authoritative source to validate them? How much does it cost me to perform that document data validation?

If you want to establish the provenance, accuracy, and freshness of the data, you need to connect to the source that mitigates your risk level.

At the first level of reliability are “credible sources”. For example, data aggregators charge dimes and quarters for the service of validating data you collected against what they algorithmically believe to be accurate. It’s due diligence on your part. Whether you consider that credible depends on your level of risk versus the cost of service.

At the higher level of reliability are “authoritative sources” — the official issuers of identity documents. Who can connect? Mostly government and law enforcement. You can pay a quarter or two per record to validate data in privacy-preserving manners — yes/no on the accuracy of what you have versus looking up someone’s identity data using an identifier.

See the problem? It costs more just to validate than businesses should logically spend on the whole identity problem. And most don’t have access.

Then there’s the “use at your own risk” level of reliability of unofficial sources (not NIST-defined, surprisingly). The problem with using unofficial sources is that all our data was breached. Assume that to be true — zero trust. Check your mailbox for legal breach notices — confirmed. It’s cheaper to buy that data from the dark web than to use either a credible or authoritative source (if available to you). Advantage fraudsters… and the reason fraudsters can pass Knowledge-Based Authentication more easily than our memories. Skip KBA as an alternative (please).

What’s the point? Eh, given the cost, you’ll skip the document validation step. And you skipped document authentication. So you’re left with having snapped an off-angle photo of that ID document so that you can crop the low-resolution face out of it (resplendent with holograms, security lines, watermarking, and reflective lamination) for biometric matching. Better set that False Accept Rate high enough to get a decent Match Score!

Or you can just produce digital documents in the first place. (See #3 below)
Cost Consolidation

From what I see, there are some alternatives for us to start to win-win the zero-sum problem we are facing.

1. Do this Once at High Cost/Accuracy and Reuse it at Lower Cost

If one analog-to-digital event can be run at a high level of identity assurance(IAL2 minimum) by an authoritative entity, and the event can be re-used, then we start to lower the repeated costs to all future entities. Perhaps then the cost-benefit ratio will make sense. Note that I said an “authoritative” entity and not a credible or trusted entity. I like banks. Banks act in trustworthy ways, perform due diligence when required, and have been willing to step to the forefront in many regions. But if you require a “government-issued ID” for your use case, bank KYC isn’t that. Bank KYC is an Identity Verification not a government signed ID.

A reusable identity solution should be a government-issued, privacy-protecting credential with user authentication and selective disclosure. You might look at mDL as this, however, there’s no standardized mechanism for user authentication yet in ISO/IEC 18013–5 or -7. Even without built-in user authentication, ingesting a digital mDL will short-circuit dozens of my list of 10 problems (pun intended), increase accuracy, and lower cost while allowing fewer fraudulent documents through. That’s why NIST wants to solve this first through the NCCoE.

2. Separate Financial Risk Mitigation from Official Identity

Why this hasn’t happened yet confuses me. Yet we still call it “identity fraud” when criminals are using freely available identity data to perform “financial fraud”.

But seriously, scooter rentals don’t need my ID document, they need to know I can or will pay. Keep going and we will realize the ACLU’s dystopian over-identification state (link will download PDF). None of us want that.

The solution Financial Services needs to look at is reusable user authentication against the KYC they or their partners already performed… without sharing identity attributes that won’t mitigate financial risk.

3. Issue Digitally-Signed Originals

Better yet, just skip the analog to digital steps. Issue and accept digital birth certificates. Issue and accept Mobile or digital ID documents. If you’re not yet accepting digital mDLs, start.

People in the (self-labeled) Self-Sovereign or Decentralized Identity spaces have created the nearly ideal architecture for issuing authoritatively signed, lightweight, digital birth certificates. So far, to my knowledge, only Singapore has jumped on this technology and is realizing the cost savings and accuracy. There’s still one privacy hole to close… whoever controls the distributed ledger can potentially see all authentication and validation requests against the ledger if they control the API layer. This is a solvable technical problem with the architectural concepts of blinding, brokerage, or anonymizing API access (rotating relying party identifiers). This is solvable with policy (link will open video) if not law.

4. Put the User/Holder in Control of their Credentials and Data

When you provide the user (aka citizen, resident, digital subject, holder) with their document in digital form, they gain control over how they share it. EU Digital Identity Wallets are making progress. More importantly, the ease of sharing means that the recipient (relying party) doesn’t need to store identity attributes to mitigate their risk. Ask, verify, dispose. This is a concept of Interactive Information Privacy(TM) that I’ll write about more in the future and that I presented at Cloud Identity Summit in 2017.

Don’t forget to put your reusable identity-proofing credential in that wallet.
Call to Action

We want to stop “ripping IDs” just like we stopped ripping CDs when digital content services became ubiquitous. Digital Identity Services are the next frontier, but everybody is trying to solve everything everywhere all at once. I laid out 4 pathways above, each with existing possible solutions and a legitimate business model to make them self-sustaining. Let’s create actionable, cost-effective solutions to our “identity fraud” problems. The architects have done decent jobs… now the product, market, and business sides should capitalize in better ways than we are currently employing.


Just a Theory

Mini Summit Three

A rough transcript of my Devrim Gündüz’s talk, “Overview of {yum,zypp}.postgresql.org, and the challenges around RPMifying extensions”, along with interspersed chat comments.

Terrific presentation and discussion at last week’s Postgres Extension Ecosystem Mini-Summit. In fact, I later learned that some people couldn’t make it because the Eventbrite event page showed it as sold out!

This is a limitation of the free level at Eventbrite, which caps tickets at 25. But this is a fully-remote event and we can host more people than that. We had over 30 last week! So if you’d like to attend one of the remaining three events and Eventbrite shows it as sold out, have a look at the bookmarks for the #extensions channel on the Postgres Slack, or email the username david at this domain and I will send you the link!

Okay, back to business. Last week, Devrim Gündüz kindly gave a super thorough presentation in his capacity as the maintainer of the Postgres community Yum and ZYpp repositories. This topic sparked lots of interesting discussion and commentary, which I detail below. But why read when you can look?

Video PDF Slides

Still reading? Okay then, here we go!

Introduction I opened the meeting and introduced Devrim Gündüz. Presentation

Thank you for the introduction. Going to talk about Yum and ZYpp dot postgresql.org, these are the challenge for us, and the challenge of making RPMs from extensions.

Work for EDB as Postgres expert, Postgres Major contributor responsible for the Postgres Yum repo. If you’re using RPMs, I’m building them. I also contribute to Rocky, SUSE, and Fedora Linux, proud Postgres community member, live in London, and I’m also doing some DJing, so if I get fired I have an extra job to do.

Agenda: Last year at PGConf.eu in Prague, I had 60 slides for 5 minutes, so expect huge number of slides for today. I want to leave some time for discussion discussion and questions.

I’m going to start with how it started back in the 2000s, talk about the contents of the repos, which distributions we t because that’s another challenge um how do we do the extension packaging how to RPM how to build RPMs of an extension and how to update an extension RPM.

Then the horror story um when what happens when there’s a new Postgres beta is out, which is due in the next next month or so for Postgres 17.

Then we are going to speak about the extension maintainers, problem for us, and then the problems about relying on the external repos.

So if you have any questions please ask as soon as you can. I may not be able to follow the chat during the presentation, but I’ll try as much as I can.

So let’s talk about how it started. When I started using Linux in 1996 um and Postgres in 1998, we always had Tom Lane and we had we had Lamar for for who build built RPMs for RedHta Upstream. So they were just building the distribution packages, nothing community. It was only specific to RedHat — not RedHat Enterprise Linux but RedHat 3-4-5 — not the non-enterprise version of RedHat back then, but it was called it Fedora core back then, the first version was released in November 2003, which was another challenge for packaging

One of the problems with the distribution packaging was that it was slightly behind the minor Postgres releases, sometimes major post releases

So that was one single Postgres version available for a given distro, say Postgres 6.4 or something, 7.0, and multiple versions were not available back then, and the minor version was slightly behind.

I started building packages for my laptop because I wanted to use Postgres and not all versions wer available back then. So I started building packages for my laptop and my server. They were based on the packaging of Tom Lane and Lamar.

Then I uploaded them to my personal server and emailed the PG mailing lists lists and said, “I’m running on own packages, use at your own risk. This is just a rebuild of the upstream packages on the RedHat version that I’m using or the Fedora version that I’m using. Up to you! This is a personal project, nothing serious.”

So then then people started downloading them, and using them. There was no signature, nothing official back then. Then Lamar said he didn’t have enough time for the RPMs. He sent an email to the mailing lists and I said Devrim is stepping up to the plate, and I did it. So that was I think in 2004, about which is about 20 years ago.

So 19 years ago we had the first domain, postgresql.rpm.org, and then we had more packages. In 2007 we had the first repository RPM and then we had yum.postgresql.org. This means that, starting 2007, this began to be the official RPM repo of the community, which which was a good thing because we could control everything under the Community.

I was living in Canada back then. We had the first RPM build farm — instead of using my laptop and my VMs — we had the actual machine back then.

In 2010 we had what was then called multiple postmaster support, which means the parallel installation of the Postgres RPMs. That was a that was a revolution because even still, the current Fedora or RedHat RPMs cannot be installed in parallel. So if you want to install post 13, 14, 15, 16, and 17 or whatever, it is available in the community repo. This is a great feature because you may want to test or do something else. This is how we how we started, 14 years ago we had this feature in the community repo.

Then next year we moved the repo to the community servers and unified the spec files. Our builds are extremely simple — like you can start your own builds in less than 10 minutes: five minutes for pulling the git repo and then a few minutes for for a package. Extremely simple builds, and now we are in 2024

Let’s talk about the contents, because we have four different contents for the repo. The first one is the “common” repo. We call it “common” in the repository config file. It has the RPMs that work across all Postgres major releases. Let’s say pg_badger, or the ODBC driver, JDBC driver, or GDAL for PostGIS. There’s lots of stuff: Python packages, which are not extensions but we like Patroni, which actually works for all Postgres releases.

This was an effort to get rid of duplicates in the Postgres repo. I think we shaved lots of gigabytes back then, and we still save a lot

Then, to address one of the topics of today’s talks, we have the “non-common” RPMs. (These are the name of the directories, by the way.) They are the RPMs which are Postgres version-specific, basically they are extensions. Say plpgsql_check 15 or whatever. Lots of extensions are there.

Then we have extras. They are not actually Postgres packages, they shouldn’t be included in our repo by default, but many people actually look for these packages because they they want to use Patroni and they don’t have the supporting RPMs or supporting RTM RPMs, or they’re not up-to-date.

I’m building a console, console-template, ETCD, HAProxy keepalived and vip-manager. They are all open source, some of them are Go packages — which, actually, I don’t build them, I just distribute the precompiled binaries via the repo. So that makes easier for people to deploy the packages.

And then we have the “non-free” repo. These are the packages that depend on closed-source software like Oracle libraries, or that have license restrictions. For example, ora2pg depends on Perl DBD::Oracle, oracle_fdw depends on Oracle, pg_storm depends on Cuda Nvidia stuff, timescaledb-tsl actually is Timescale DB with the TSL license, informix_fdw and db2_ftw.

So we have some non-free packages which actually depend on non-free stuff as well. All of them are well-maintained: I’m trying to keep everything up to date — like real up-to-date! That brings some problems but we will get there.

We support RedHat Enterprise Linux and of course Rocky Linux. This year we started supporting Alma Linux as well. Of course they are more or less identical, but we test them, install, and support to verify the packages in these three distributions.

We have x86_64 aarchm64, ppc64le, and RedHat 9, 8, and 7. We have also RedHat 6 for Postgres 12, but it’s going to be killed by the end of this year. We have Fedora, but only two major releases, which matches the Fedora lifecycle. And SUSE — my pain in the neck — um I’ll get there.

Since you all are here for extension packaging, let’s get there: what happens for extension packaging.

First of all, we have the first extension, which is the in-core extensions. They are delivered with the contrib sub-package, which matches the directory name in The Postgres tarball. There are separate packages for each major version, so postgres15-contrib, postgres13-contrib, etc. These are the installation directories for each extension. We are going to use those directories for the other [non-cre] extensions as well.

When we add a new extension to the repo, it’s going to use these directories if they have a binary or if they an extension config file, if the library or the mem files — all are going are all installed under these directories. This magic is done by PGXS, which has been there forever. We just provide initial stuff and then the rest is done by the the PGXS magic. This is the base for a lot of the core extensions.

So what happens when we do non-common and non-free package? First of all, they are distributed separately for each Postgres major version. Let’s go back to the one of the extensions, say plpgsql_check. We have a separate package for Postgres 14, a separate package for Postgres 15, 16, 13, and 12. If they build against all the supported Postgres versions, we have separate packages for each.

Of course from time to time — as far as I remember Timescale DB does this —- Timescale DB only supports Postgres 15 and 16 nowadays. So we drop support for the older versions in the Git repo. But they are separate packages; they are all installed in these directories along with the main contrib package.

This is the naming convention that I use: $extensionName_PGMajorVersion. Four or six years ago, some of the packages didn’t have an underscore before the PG major version. It was a problem, so someone complained in the hackers mailing list, and then I made the changes.

Currently all the previous and all the new packages have this naming convention except a few packages like PostGIS — because in PostGIS we have multiple versions of the same extension. let’s say we have PostGIS 3.3, 3.2, 3.4, 3.1, 3.0. We have combinations of each — I think we have PostGIS 3.0 in some of the distributions but mostly PostGIS 3.1, 3.2, 3.3, and 3.4, and then all the Postgres versions they support — A lot of builds! So there are some exceptions where we have the extension name and extension major version before the Postgres major version.

Jeremy S in chat: question: curious if you have thoughts about relocatable rpm packages. ever tried? just best to avoid?

I have a question from Jeremy. This is a very basic question to answer. This is actually forbidden by the packaging rules. The RPM packaging rules forbid you to distribute or create relocatable RPM packages. We we stick to the packaging guidelines, so this this cannot be done.

Jeremy S in chat: Thanks! (Didn’t realize packaging guidelines forbid this)

Let’s talk about how we build extensions. Often our develop package is enough: many of our packages just just rely on on Postgres itself. But of course packages like PostGIS may depend on some other packages, like GDAL, which requires lots of lots of extra dependencies as well. The most problematic one is the GIS Stack: they need EPEL on RHEL and RHEL and its derivatives.

There there has been a discussion in the past about whether should require EPEL by default. The answer is “no,” because not all not all of our users are installing, for example, the GIS stack or other packages. Most of our users — not the majority of our users —- rely on the um rely on just our repo.

On the other hand, in order to provide completeness for our users, I added lots of python packages in the past to support Patroni — because the upstream packages (I’m sorry not maybe upstream packages) were not enough. The version wasn’t enough or maybe too low. So From some time to time I add non Postgres-related packages to the repo just to support the Postgres package. In the past it was PGAdmin, but now it’s not in our repo so it’s not a problem: their upstream is building their own RPMs, which is a good thing for us. We are building extra packages mostly for Patroni.

However, this is a potential problem for some enterprise users because large companies don’t even want to use the EPEL repo because they feel like it’s like it’s not a community repo, but a community repo controlled by Fedora and RedHat. That’s why from time to time I try to add some of the packages to our repo.

If it’s a problem for enterprise users, does it mean we should we maintain tons of extra packages in the EPEL repo for the GIS stack? The answer is “no, definitely no”. Not just because of the human power that we need to maintain those those packages — I mean rebuilding them is easy: I just get the source RPM, commit the spec files into our repo, and rebuild them. But maintaining them is something else.

We will have a similar problem when we release our own ICU package in the next few years. Because, now that we have the in core collation — but just for C Locale —- and people are trying to get rid of glibc, maybe we should have an ICU package. But maintaining an ICU Library across a single Postgres major version is a real big challenge that I don’t know how to solve for now, at least.

And then SLES — my pain in the neck — the GIS stack requires lots of repos on SLES 15. They are they are well documented on on our website.

Fedora is safe because Fedora is Fedora, everything is there, it’s easy to get a package there.

Jeremy S in chat: “Building them is easy. Maintaining them is something else.”

Yeah that’s that’s the problem, Jeremy.

So how do you RPMify an extension?

The first thing is to know that the extension exists. This is one of the big problems between developers and users. The developer creates a useful extension and then they don’t create a tarball, they don’t release anything. They expect people to install Git on their production databases and git pull, install make, gcc, all the development libraries and build a binary, blah, blah blah.

I’m sorry that’ss not going to work. It also doesn’t work for pip — pip is not a package manager, it just destroys your server. It downloads things to random places and then everything’s gone. That’s why I added lots of Python packages to support Patroni, because most of the users use the packaging package manager to install Postgres and other packages to their servers. It’s the same for Debian, Ubuntu, for RPMs, for Windows, for macOS.

So first of all we know have to know that the extension exists and we have to have a tallbal. If I see an extension that seems good enough I’ll get there. PGXN is a good place, because when I go to pgxn.org a few times per day and see if there is a new version of an extension or if there’s new extension, it’s a good piece. But there’s a problem: we have hundreds of extensions — maybe thousands — but not all of them are on PGXN. They should be!

David Wheeler (he/him) in chat: You should follow @pgxn@botsin.space for new releases :-)

nils in chat: pg_xz incoming

I don’t know how to solve this problem, but we should expect every extension to announce their extensions on PGXN. I’m not just talking about installing everything through PGXN, but at least have an entry that there’s a new extension, this is repo, the website, the readme and the is a tarball. It doesn’t have to be on PGXN, as long as we have something.

And then I check the version. If there is an extension that will kill your database and the version is 0.001, that’s not going to be added to the repo, because we don’t want to distribute an experimental feature.

David Wheeler (he/him) in chat: LOL, all my extensions start at 0.0.1

David G. Johnston in chat: If it isn’t on PGXN it doesn’t exist.

Another problem is that lots of people write extensions but some of them are just garbage. I’m sorry but that’s the truth. I mean they just release a version and then do nothing.

David Wheeler (he/him) in chat: LOLOLOL

From the chat, “pgxn_xz is coming”: that’s right! We have blackhole_fdw, which was written by Andrew Dunstan. When you create blackhole_fdw, it throws all of your data into black hole, so everything is gone.

Yeah, “if it’s not on PGXN it doesn’t exist,” that’s what I hope we achieve achieve in the next year or so.

Jimmy Angelakos in chat, replying to “If it isn’t on PGXN …”: I would correct that to “If it isn’t on PGXN it isn’t maintained.” Sometimes even ON PGXN…

Yeah Jimmy, that’s one of the big problems that we have: maintenance.

We create the spec file, just copy one of the existing ones and start editing. It’s easy but sometimes we have to add patches. We used to carry patches for each Postgres major version to change the Makefiles for the specific Postgres major version. But I realized that it was [not a great pattern]. Now we just export the path, which fixes the problem.

David G. Johnston in chat: As a policy though, someone who cares and wants responsibility needs to apply to be part of the RPM community.

Then I initiate a scratch build for any missing requirements. If there are any missing build requirements it fails to build. I only do it on Fedora latest, not for every package because it doesn’t always help because some distros may not have the missing dependency

Alvaro Hernandez in chat: Hi everybody!

David G. Johnston in chat: Delegated to PGXN for that directory.

Let’s say we rely on some really good feature that comes with a latest version of something, but that latest version may not appear in RedHat 7 or 8. So some dist dros may have it, but the version may be lower than required. Or some distros may have the dependency under different name. Now in the spec file we have “if SUSE then this” and “if RedHat then this” “if Fedora then”, “if RedHat nine then this”, etc. That’s okay, it’s expected. As long as we have the package, I don’t care.

Then I push it to the Git repo, which I use not just for the spec files and patches, but also for carrying the spec files and pitches to the build instances.

Jorge in chat: How to handle extension versioning properly? I mean, in the control file the version could be anything, like for ex. citus default_version = ‘12.2-1’ where the “published version” is v12.1.2, then the “default version” could remain there forever.

Also have seen in the wild extensions that the control file have a version 0.1 (forever) and the “released” version is 1.5

If something fails I go back to the drawing board. GCC may fail (gcc 14 has been released on Fedora 40 and is causing lots of issues for for packaging nowadays), it could be cmake — too recent or too old. It could be LLVM — LLVM18 is a problem for Postgres nowadays. I either try to fix it ping upstream. I often ping upstream because the issue must be fixed anyway

If everything is okay, just push the packages to the repo.

Ruohang Feng (Vonng) in chat: question: how about adding some good extensions written in Rust/pgrx to the repo? like pgml, pg_bm25, pg_analytics, pg_graphql….

One issues is that there is no proper announcement. Maybe I have an awesome extension available in the Postgres repo that people crave and, we build the extensions, it took a lot of time (thank you Jimmy, he helped me a lot) and then I didn’t actually announce it that much. On the other hand, people just can use PG stat base [?] to install and start using it in a few seconds. This is something that we should improve.

Steven Miller in chat: How to handle system dependencies like libc versions being updated on the target system? Do extensions need to be complied against exactly a specific libc version?

From Nevzat in chat: how can we make sure bugfix or extension is safe before installing it

vrmiguel in chat: Interesting approach to identify build/runtime requirements

Over at Tembo we have a project called trunk-packager which attempts to automatically create .deb packages given the extension’s shared object.

We try to identify the dynamic libs the extension requires by parsing its ELF and then trying to map the required .so to the Debian package that provides it, saving this info in the .deb’s control file

From the chat: How to handle extension versions properly? That’s a good thing but, extension version and the release version don’t have to match. Thr extension version isn’t the same thing as the release version. It’s the version of the SQL file or the functions or the tables, the views, sort procedures, or whatever. If it’s 0.1 it means it’s 0.1 it means nothing nothing has changed in this specific regarding the control file. They they may bump up the package version because they may add new features, but if they don’t add new features to the SQL file, then they don’t update the extensions. I hope that answers your question George

I have another question from Ruohang. Yaaaaay! I was afraid that someone would ask that one. We have no extensions written in Rust in repo so far. It’s not like Go; there is a ban against Go because we don’t want to download the world, all the internet just to build an extension. If I recall correctly they’re rewriting pg_anonymizer in Rust. They will let me know when they release it or they’re ready to release it, and then I’ll build it. It’s not something I don’t like, it just hasn’t happened.

Keith Fiske in chat: I still update the control file with my extensions even if it’s only a library change. Makes it easier to know what version is installed from within the database, not just looking at the package version (which may not be accessible)

Ruohang Feng (Vonng) inchat: question: How to handle RPM and extension name conflicts, e.g., Hydra’s columnar and Citus’s columnar.

*David Wheeler (he/him) in chat, replying to “I still update the c…” Yeah I’ve been shifting to this pattern, it’s too confusing otherwise

If you think there are good extensions like these, just create a ticket on redmine.postgresql.org. I’m happy to take a look as long as I know them. That’s one of the problems: I have never heard about pg_analytics or pgml, because they’re not on PGXN. Or maybe they are. This is something that we should improve in the next few months.

Jimmy Angelakos in chat: Go is a pretty terrible ecosystem. It has its own package manager in the language, so it’s kinda incompatible with distro packages

Jason Petersen in chat: (but that means a build is safe within a single release, it doesn’t mean you can move a built extension from one Fedora version to another, right?)

David Wheeler (he/him) in chat, replying to “How to handle system…”: Libc is stable in each major version of the OS, and there are separate RPMs for each.

Another question from Steven: how to handle system dependencies like libc version updates. The answer is no. It’s mostly because they don’t update the libc major version across the across across the lifetime of the of the release. So we don’t need to rebuild the extension against libc.

Steven Miller in chat, replying to “How to handle system…”: Ok I see, TY

Jason Petersen in chat, replying to “How to handle system…”: Is that how we deploy in trunk, though?

David Wheeler (he/him) in chat, replying to “Go is a pretty terri…”: Huh? You can build binaries in a sandbox and then you just need the binary in your package.

[Addressing Nevzat’s question]: That’s a great question. It’s up to you! It’s no different than installing Postges or any other thing. I just build RPMs. If you’re reading the hackers mailing list nowadays, people rely on me an Christoph and others, so that we don’t inject any code into the RPMs or Debian packages. You just need to trust us not to add extra code to the packages. But if there’s a feature problem or any bug then you should complain upstream, not to us. so you should just test.

Jimmy Angelakos in chat, replying to “Go is a pretty terri…”: Find me one person that does this.

Steven Miller in chat, replying to “How to handle system…”: We don’t have OS version as one of the dimensions of version packaging but should

[Addressing vrmiguel’s comment]: Yeah, that could be done but like I don’t like complex things, that’s why I’m an RPM packager.

Jason Petersen in chat, replying to “Go is a pretty terri…”: (doesn’t go statically link binaries, or did they drop that philosophy?)

vrmiguel in chat: I think citus has changed it to citus_columnar

David Wheeler (he/him) in chat, replying to “Go is a pretty terri…”: Hugo: https://github.com/hugomods/docker/tree/main/docker/hugo

David Wheeler (he/him) in chat, replying to “Go is a pretty terri…”: Jason: Static except for libc, yes

Another question from Ruohang: uh how to handle RPM and extension name conflicts. I think Citus came first, so you should complain to Hydra and ask them to change the name. They shouldn’t be identical. We have something similar with Pgpool: they they are conflicting with the PCP Library ,which has been in the Linux for the last 25 years. I think Pgpool has to change their name.

Jeremy S in chat, replying to “I still update the c…”: So you think people will run the “alter extension upgrade” eh?

[Addressing Keith Fiske’s comment]: I’m not saying I don’t agree with you, but it means every time I have to update my extension version in my running database — it’s some extra work but that’s okay. It’s the user problem, not my problem.

Question from Jason [on moving an extension from one Fedora to another]: Right, it may not be safe because the GCC version may be different and other stuff may be different. One distro to another is not safe, Jason; sorry about that.

[Back to Steven’s question]: Yes, David’s answer is right.

[Addressing vrmiguel’s comment about citus_columnar]: You are right.

Jimmy I’m not going to read your comment about go because I don’t think think you can swear enough here.

vrmiguel in chat, replying to “Go is a pretty terri…”: Are there known Postgres extensions written in Go? Not sure how Go is relevant here

Jason Petersen in chat: you said “gcc” and not “libc” there, are you implying that things like compiler versions and flags need to be identical between postgres and extensions

Keith Fiske in chat, replying to “I still update the c…”: I think they should …

David Wheeler (he/him) in chat, replying to “Go is a pretty terri…”: Were some experiments a few years ago. https://pkg.go.dev/github.com/microo8/plgo

Let me continue now. First you have to know the extension exists, and then the you also need to know that the extension has an update. Unfortunately the same problem: the extension exists or has an update and they just don’t let us know.

Jimmy Angelakos in chat, replying to “Go is a pretty terri…”: @vrmiguel now you know why :D

This is a big challenge Fedora has in house solution.When you add a new package to Fedora, I think they crawl their repo once a day and if there’s new release they create a ticket in their bug tracker automatically, so that the maintainer knows there’s a new version. This can be done, but would need a volunteer to do it. Orr maybe the easiest thing is just add everything to the to PGXN,

When we update an extension we, have to make sure it doesn’t break anything. It requires some testing. As I said earlier, building is one thing, maintaining the extension is a bigger thing. If you want to raise a baby, you are responsible until until the end of your life. Consider this like your baby: either just let us know if you can’t maintain an extension anymore or please respond to the tickets that I open.

Steven Miller in chat: One other detail about compatibility dimensions. We have noticed some extensions can be complied with chip-specific instructions like AVX512, for example vector does this which optimizes performance in some cases

Alvaro Hernandez in chat, replying to “you said “gcc” and n…”: I’d reverse the question: do we have strong guarantees that there are no risks if versions and/or flags may differ?

I believe extensions are already risky in several ways, and we should diminish any other risks, like packaging ones.

So I’d say absolutely yes, compile extensions and Postgres in exactly the same way, versions and environments.

Sometimes a new minor version of an extension breaks a previous Postgres release. For example, an extension drops support for Postgres 12 even though Postgres 12 is still supported. Or they didn’t do the upgrade path work. I have to make sure everything is safe