Identity Blog Catcher

この４月１日から、商業登記電子証明書の手数料が改定されました。 有効期間1か月の電子証明書が500円で発行可能に、有効期間3か月～27か月の電子証明書も、最大1,000円の値下げされました。

500円でできるなら、一回申請してみようかなと思います。Windows PCからじゃないと出来ないのがちょっとあれですが。

商業登記電子証明書は、法人の代表者がオンラインで各種手続きを行う際に必要となる電子証明書です。以下のような用途で使用されます：

商業登記電子証明書の主な利用用途

商業登記電子証明書は以下のような用途で使用できます。

法務関係 商業登記や供託、電子公証、印鑑証明書の取得など。 税金関係 国税電子申告・納税システム（e-Tax）や地方税電子申告（eLTAX）を利用した申告・納税。 社会保険・労働保険関係 健康保険や厚生年金保険の手続き、労働保険関連の申請。 特許関連 特許のインターネット出願や国際出願（PCT-RO）。 自動車保有関係手続 車両登録や継続検査など、自動車関連手続き。 その他行政手続き 電波利用の電子申請、防衛装備庁の電子入札、府省共通の電子調達システムなど。

でもこの中には、あんまり使用する機会があるものが無いなぁ。

PDFへの電子署名の方法

商業登記電子証明書は契約書などPDFに署名をすることもできます。PDFに署名するには、以下の手順を実行します：

1. 必要なソフトウェアの準備 Adobe Acrobat ProまたはAdobe Acrobat Readerを使用します。 法務省が提供する「商業登記電子認証ソフト」や「申請用総合ソフト」をインストールしておくと便利です。 2. PDFファイルの準備 署名対象のファイルがWordやExcelの場合、事前にPDF形式に変換します。 3. 電子署名の付与手順 PDFファイルを開く Adobe Acrobat ProまたはReaderで署名対象のPDFを開きます。 デジタルIDの設定 メニューから「ツール」→「証明書」を選択。 「デジタル署名」をクリックし、署名位置を指定。 「新しいデジタルIDを設定」を選び、「ファイルのデジタルIDを使用」を選択。 商業登記電子証明書（拡張子.p12）を選択し、パスワードを入力して設定します。 署名の実行 設定したデジタルIDを選択し、再度パスワードを入力。 署名後、署名済みのPDFファイルとして保存します。 署名完了の確認 署名が付与されたPDFには、Adobe上で「この文書には有効な署名が含まれています」と表示されます。

注意点

商業登記電子証明書は法務局発行のため、Adobe Acrobatで検証時に「少なくとも1つの署名に問題があります」と表示される場合があります。これは、Adobeの信頼済みリストに法務局のルート証明書が登録されていないためです。この場合でも、法務省提供のソフトウェアで検証すれば問題ありませんが、一般に利用しようとするとここが障害になりますね。 複数人での署名が必要な場合や追加で署名する場合も対応可能です。

より詳しくは、https://www.moj.go.jp/ONLINE/CERTIFICATION/index.html を御覧ください。

🔗 If you are happy and need something to take you off your high … - don’t worry - its not that you don’t know l this - it’s just when you see it summarised and ending with …

But the public is too dumb and too uninformed to wake up and smell the coffee.

It ain’t morning in America, it’s closing in on midnight.

If you speak up, no one will rally to your defense, like the law firms sitting on the sidelines and refusing to defend 🔗 Perkins Coie

But you’ll compete with other influencers for the last buck, listening to mindless drivel .. happy that you’re still alive.

Minimal CSS-only blurry image placeholders

Absolutely brilliant piece of CSS ingenuity by Lean Rada, who describes a way to implement blurry placeholder images using just CSS, with syntax like this:

<img src="…" style="--lqip:192900">

That 192900 number encodes everything needed to construct the placeholder - it manages to embed a single base color and six brightness components (in a 3x2 grid) in 20 bits, then encodes those as an integer in the roughly 2 million available values between -999,999 and 999,999 - beyond which range Lean found some browsers would start to lose precision.

The implementation for decoding that value becomes a bunch of clever bit-fiddling CSS expressions to expand it into further CSS variables:

[style*="--lqip:"] { --lqip-ca: mod(round(down, calc((var(--lqip) + pow(2, 19)) / pow(2, 18))), 4); --lqip-cb: mod(round(down, calc((var(--lqip) + pow(2, 19)) / pow(2, 16))), 4); /* more like that */ }

Which are expanded to even more variables with code like this:

--lqip-ca-clr: hsl(0 0% calc(var(--lqip-ca) / 3 * 100%)); --lqip-cb-clr: hsl(0 0% calc(var(--lqip-cb) / 3 * 100%));

And finally rendered using a CSS gradient definition that starts like this:

[style*="--lqip:"] { background-image: radial-gradient(50% 75% at 16.67% 25%, var(--lqip-ca-clr), transparent), radial-gradient(50% 75% at 50% 25%, var(--lqip-cb-clr), transparent), /* ... */ linear-gradient(0deg, var(--lqip-base-clr), var(--lqip-base-clr)); }

The article includes several interactive explainers (most of which are also powered by pure CSS) illustrating how it all works.

Their Node.js script for converting images to these magic integers uses Sharp to resize the image to 3x2 and then use the Oklab perceptually uniform color space (new to me, that was created by Björn Ottosson in 2020) to derive the six resulting values.

Via Hacker News

Tags: css, css-custom-properties

I started using Claude and Claude Code a bit in my regular workflow. I’ll skip the suspense and just say that the tool is way more capable than I would ever have expected. The way I can use it to interrogate a large codebase, or generate unit tests, or even “refactor every callsite to use such-and-such pattern” is utterly gobsmacking. [...]

Here’s the main problem I’ve found with generative AI, and with “vibe coding” in general: it completely sucks out the joy of software development for me. [...]

This is how I feel using gen-AI: like a babysitter. It spits out reams of code, I read through it and try to spot the bugs, and then we repeat.

— Nolan Lawson, AI ambivalence

Tags: ai-assisted-programming, claude, generative-ai, ai, llms, nolan-lawson

🔗 📚‘On Tyranny’ by Timothy Snyder

One that seems to have it a global nerve. It seems like EVERYONE is watching it - so of course I had to. Very Good. Two points to make.

✅ 4 episodes - because no more needed.

✅ The flow. Each of the 4 episodes was done in a single take. Extraordinary. The camera follows a person and tracks the story flow as the cast moves around and engages with each other and the camera moves you to the next part of the story as the character on screen engages with that person. If you are movie buff - think Birdman and 1917 with one difference - this seems to have been a real single shot, not a post shoot stitch.

Definitely watch it. and any young people you know.

Adolescence on 🔗 Reelgood

‘All’ My TV Show Reviews

All 3 series have been solid and fun. Not getting better, Not getting worse - but I have to say Alan Ritchson is waaaay better casting than Tommy Cruise for the role.

Series 1 💯

✅

Series 2 💯

✅

Series 3 💯

✅

Information on Reelgood

‘All’ My TV Shows

Goodness Cafe, Jervois Road 📍

Back when we first arrived in New Zealand The Goodness was called The Governor and became our go to place for coffee and breakfast. I think it’s now onto something like the 4th or 5th owner since then (that’s what COVID has done), but it still remains our ‘go to’. BUT .. our hearts remain with ‘The Governor’ and Steve and Bernie who owned it remain great friends of ours.

🖇️ 🔎 My Other Location Posts

If you haven’t read 🔗📚 Ollie Henderson’s Future Work/Life you should. Excellent.

This particular quote reminds me of something that Jason Fried wrote in one of his books ..

New technologies tend to imitate the old until people figure out what they’re really meant to do. Early websites were brochures. Early cars looked like horse-drawn carriages. Early TV shows were radio shows with pictures.

💬 Jason Fried

So too work. We ‘broke’ work about 5 years ago - but slowly people (Corporates) are ‘fixing it’ (As in making us all go back to our old ways.) This is because they are still looking at work through the lens of the brochure/horseless carriage. It will be a while before we break it again - BUT - it will be broken but only once we have truly unpacked how the new world of work works.

I’ve got ideas. I’ve got people. I’m excited for the future.

🔗 ‘A Complete Disaster,’ Says Daniel Ives About Tesla Stock

The EV giant delivered 336,681 vehicles in Q1 – falling far short of Wall Street’s expectation of around 378,000. Even the more cautious analyst forecasts, hovering near 360,000, were missed by a wide margin. In all, Tesla missed consensus by more than 10% and underperformed even the lowest projections by about 20,000 vehicles. Deliveries also tumbled 13% year-over-year, the sharpest quarterly drop in the company’s history.

.. and then …

And yet, in a twist straight out of the Musk playbook, Tesla stock staged a comeback. After plunging as much as 6%, shares flipped positive – up 5% – on a Politico report suggesting Musk might step away from his role at the Department of Government Efficiency (DOGE).

Really? If he stops DOGE it’s all going to be ok? Where do these people live?

🔗 Daring Fireball: ‘What Makes an App Feel “Right” on the Mac?'

Martin has a good list here of fundamentals, but ultimately, you know it when you see it.

💬 JohnGruber

.. like ‘pornography’ and ‘art’ then?

🔗 Daring Fireball: Steve Kornacki Exits MSNBC for New Deal With NBC News and NBC Sports

They might well just switch off the lights and lock the doors at MSNBC.

💬 John Gruber

He’s not alone with that thought. Comcast has been moving deck chairs for a while now.

But this is more a demonstration of another media company bending the knee.

🔗 Long Live RSS - Phil Windley - via Doc Searls

A timely reminder. It’s RSS in general - good, but Phil also writes specifically about Substack.

I found that if you enter /rss after the URL for an author or category page, you get RSS for that author or category.

Meanwhile Substack’s own help. - says to tack on ‘feed’ not ‘rss’.

I do find that both work - and sometimes neither - which is more often than not because of the beginning of the URL …

Replace “your” with the name of your Substack publication.

It isn’t always clear (to me at least) what that ‘your’ replacement should be.

Bonus - If you use Reeder

If you just put the URL https://.substack.com into Reeder - it will detect the RSS!

Really. Click on it. This is scary. Via Wndy.com.

I suggest rocketing his ashes into space. I'm still shocked and sad to know Dave Täht has died, but I only have one source of information so far, and it says nothing about where and how he died. Or if there will be somewhere the customary celebration of his life.

It will never die. Phil Windley says Long Live RSS.

Department of Political Correction. Wired: The DOGE Axe Comes for Libraries and Museums.

Maybe. Tornado?

Turns out this was bullshit. Marcus Smart is still with the Wizards.

🕵💬 Inspired By People | 🌐 Norman Shidlec

🔍 All The Posts can be found here, or scroll through the quotes below.

[Laura Hazard Owen at Nieman Journalism Lab]

A good introduction to leaking to a journalist:

"I spoke with eight journalists about how to leak in a safe, smart way. Disclaimer you probably knew was coming: No method of leaking is 100% secure, and the tips here reduce risk but cannot eliminate it completely. “I know it’s appealing to be instrumental in helping a reporter break a story, and god knows reporters love breaking stories,” says Marisa Kabas, an independent reporter and writer of The Handbasket who’s been breaking one scoop after another about DOGE and the Trump administration. “But in almost all cases, your safety and physical and mental health should come first.”"

A lot depends on Signal, although some newsrooms (including my employer) also advertise SecureDrop, which is a very sophisticated tool for large, anonymous leaks.

The complete list is worth your time. If you're a source, consider using these tools. If you're a funder, consider investing in these tools. If you're a newsroom, make sure you know how to use these tools. They've become the currency of privately-sourced stories in the current era.

#Democracy

[Link]

Like many, I've grown increasingly disinterested in social media over the last year. I still have accounts on X, Substack, Facebook, and others. But I find that there's less and less there that I actually care about. I find myself wading through lots of crap to find the things I want to see. Still, I do like to browse through ideas while waiting in line, at the doctor's office, and so on.

This post from Cory Doctorow entitled You should be using an RSS reader caught my eye a while ago. I used to use an RSS reader all the time. I've tried to get back into it a time or two, but it didn't stick. Inspired by Cory to try one more time, I downloaded the RSS reader he uses (Newsblur) and gave it another shot. This time I coupled it with moving the RSS reader app on my phone to a more prominent position so when I'm looking for something to browse, I see it first.

First I found RSS feeds for some of my friends and publications I like to read. I also discovered that Substack publications have RSS feeds. Many publications don't make finding RSS easy, but it's almost always there. There are a few authors I like on The New Stack and I found that if you enter /rss after the URL for an author or category page, you get RSS for that author or category. For example, here's the RSS page for Jon Udell. Cool.

After adding RSS feeds for things I like (most online publications have one), I started reading things on Newsblur instead of opening up X. Most newsreaders don't have an algorithm. If they do, it is not usually the default setting and can be turned off. If you subscribe to too many feeds, turn it on and let it prioritize your reading. But regardless, you control the experience. Just like you should.

At the end of his post, Cory writes:

Using RSS to follow the stuff that matters to you will have an immediate, profoundly beneficial impact on your own digital life—and it will appreciably, irreversibly nudge the whole internet towards a better state.

From You should be using an RSS reader
Referenced 2025-04-02T13:02:11-0400

Cory's right, using an RSS reader will make your digital life better. I'm wasting less time scrolling past stuff I don't care about and more time reading things I enjoy. That's a win.

Photo Credit: Robot Reading on Phone from DALL-E (public domain)

In 1967, Alan Westin published Privacy and Freedom in response to growing concerns in the 1960s about computer databases and surveillance. Westin argued that encroachments on privacy were also encroachments on 'American liberty.' When he stated that "Privacy is the claim of individuals, groups, or institutions to determine for themselves when, how, and to what extent information about them is communicated to others"[1], he argued that privacy (liberty) was not possible without individuals having both the autonomy to both make these claims and to have them respected.

In the 60s, there was a growing concern about technology encroaching on privacy. He argued, "The real need is to move from public awareness of the problem to a sensitive discussion of what can be done to protect privacy in an age when so many forces of science, technology, environment, and society press against it from all sides."[2]

The US Privacy Act (1974) was the first legislative response, followed by the OECD privacy guidelines (1980) and the Council of Europe Data Protection Convention in 1981. Data protection or privacy laws have become the norm in the 50 years since the US Privacy Act. However, the concerns expressed then are just as valid today, whether from a left view of Surveillance Capitalism or a business school description of an Age of Surveillance Capitalism. Despite the proliferation of privacy laws, privacy is as much under threat today as it was then.

Returning to "Privacy and Freedom", does the failure of privacy mean a failure of freedom? Is the likelihood of a country being free, partially free, or not free uncorrelated with whether or not the government has data protection or privacy laws? There are more than 200 countries in the world, 150 of which have some form of privacy or data protection legislation[3]. Freedom House's Annual Freedom in the World report categorises countries as "Free", "Partially Free", or "Not Free" based on a set of 25 indicators[4]. When you compare the percentages of countries' freedom ratings, the impact of having privacy or data protection legislation on whether or not a country is free is minimal.

Total Countries 208 100 % DP Countries 150 100% Free 86 41.3% Free 69 46.0% Partially Free 55 26.4% Partially Free 38 25.3% Not Free 67 32.2% Not Free 43 28.7%

This suggests that privacy itself is not related to freedom (or liberty) OR that there is a problem with the way that privacy laws have been written or implemented. The proposition that privacy should be concomitant with individual freedom and that the ability of groups to organise seems almost axiomatically true. And recent writings suggest that, as currently architected, privacy laws can be helpful for authoritarian governments.[5]. This echoes critiques from privacy scholars such as Woodrow Hartzog[6] or Ignacio Cofone[7]. In a recent article, Daniel Solove says, "To adequately regulate government surveillance, it is essential to also regulate surveillance capitalism. Government surveillance and surveillance capitalism are two sides of the same coin. It is impossible to protect privacy from authoritarianism without addressing consumer privacy."[8]

Without trying to be hyperbolic, the current trajectory for privacy laws and regulations is leading down a path of digital alienation. It is time for privacy laws and practices to support digital autonomy.

Footnotes

Westin, Alan F.. Privacy and Freedom (p. 5). ↩︎ Westin, Alan F., Privacy and Freedom (pp. 1-2). ↩︎ See UNCTAD Data Protection and Privacy Legislation Worldwide ↩︎ See the Methodology Scoring Process at https://freedomhouse.org/reports/freedom-world/freedom-world-research-methodology ↩︎ Jia, Mark (2024) "Authoritarian Privacy," University of Chicago Law Review: Vol. 91: Iss. 3, Article 2. Available at: https://chicagounbound.uchicago.edu/uclrev/vol91/iss3/2 ↩︎ Privacy's Blueprint: The Battle to Control the Design of New Technologies https://www.hup.harvard.edu/books/9780674976009 ↩︎ The Privacy Fallacy: Harm and Power in the Information Economy https://www.privacyfallacy.com/ ↩︎ Solove, Daniel J., Privacy in Authoritarian Times: Surveillance Capitalism and Government Surveillance (January 19, 2025). Available at SSRN: https://ssrn.com/abstract=5103271 or http://dx.doi.org/10.2139/ssrn.5103271 ↩︎

Stream the Latest Episode

Listen and watch now on YouTube, Spotify, and Apple. See the episode transcript at the top of this page, and a summary at the bottom.

Brought to You By

• Swarmia — The engineering intelligence platform for modern software organizations.

• Sentry — Error and performance monitoring for developers.

—

In This Episode

Why did Meta build its own internal developer tooling instead of using industry-standard solutions like GitHub? Tomas Reimers, former Meta engineer and co-founder of Graphite, joins the show to talk about Meta's custom developer tools – many of which were years ahead of the industry.

From Phababricator to Sandcastle and Butterflybot, Tomas shares examples of Meta’s internal tools that transformed developer productivity at the tech giant. Why did working with stacked diffs and using monorepos become best practices at Meta? How are these practices influencing the broader industry? Why are code reviews and testing looking to become even more critical as AI transforms how we write software? We answer these, and also discuss:

• Meta's custom internal developer tools

• Why more tech companies are transitioning from polyrepos to monorepos

• A case for different engineering constraints within the same organization

• How stacked diffs solve the code review bottleneck

• Graphite’s origin story and pivot to their current product

• Why code reviews will become a lot more important, the more we use AI coding tools

• Tomas’s favorite engineering metric

• And much more!

Takeaways

My biggest takeaways from this conversation:

“Stacked diffs” makes a lot of sense inside companies. However, it makes less sense when working on, e.g., open source projects. Perhaps this is a reason that GitHub has not added support for this workflow — even though it’s popular inside companies like Meta or Uber. We previously did a deepdive on Stacked Diffs (and why you should know about them).

The “trust matrix:” this is a good way to decide how much process/tooling to put in place in a team. If you trust people a lot and are willing to tolerate mistakes, you should lean on culture and not process. If you start to become a team or company where you need to trust people less: this is the time to move more tooling and more process. So, as a small startup you probably don’t need that much tooling and process!

There could be an industry-wide movement to monorepos: at mid-sized and larger scaleups and tech companies. Tomas sees a lot of scaleups they work with the move from polyrepos (several repositories) to monorepos. Moving to a monorepo is still a lot of work and requires custom tooling: and this is why it was limited to the largest tech companies in the past. Interesting to hear about this change!

AI coding tools increase the importance of quality code reviews. We’ll see more code churned out by AI: but an engineer needs to review it before it goes out. It’s a good question how we can stick to thorough code reviews when it is so tempting to just say, “Looks good to me (LGTM)” and have the seemingly correct code merged.

The Pragmatic Engineer deepdives relevant for this episode

• Stacked Diffs (and why you should know about them)

• Inside Meta’s engineering culture

• Shipping to production

• How Uber is measuring engineering productivity

Timestamps

(00:00) Intro

(02:00) An introduction to Meta’s in-house tooling

(05:07) How Meta’s integrated tools work and who built the tools

(10:20) An overview of the rules engine, Herald

(12:20) The stages of code ownership at Facebook and code ownership at Google and GitHub

(14:39) Tomas’s approach to code ownership

(16:15) A case for different constraints within different parts of an organization

(18:42) The problem that stacked diffs solve for

(25:01) How larger companies drive innovation, and who stacking diffs not for

(30:25) Monorepos vs. polyrepos and why Facebook is transitioning to a monorepo

(35:31) The advantages of monorepos and why GitHub does not support them

(39:55) AI’s impact on software development

(42:15) The problems that AI creates, and possible solutions

(45:25) How testing might change and the testing AI coding tools are already capable of

(48:15) How developer accountability might be a way to solve bugs and bad AI code

(53:20) Why stacking hasn’t caught on and Graphite’s work

(57:10) Graphite’s origin story

(1:01:20) Engineering metrics that matter

(1:06:07) Learnings from building a company for developers

(1:08:41) Rapid fire round

(1:12:41) Closing

A summary of the conversation Meta's custom developer tools

Meta developed Phabricator as its internal tool for code review.

Sandcastle: Meta's internal continuous integration (CI) system, integrating with Phabricator.

OnDemand: internal development environments (dev boxes), linked with Sandcastle.

Landcastle: the tool for deploying code to users, integrated with the preceding systems.

These tools aimed for seamless integration across the entire developer workflow, extending to task management.

Herald, later replaced by Butterflybot: a rules engine that automated actions during code review based on specific events, such as flagging use of deprecated APIs.

We cover more internal tools in the deepdive Inside Meta’s Engineering Culture

Stacked diffs

Meta used a method called stacking for code changes, where developers create a series of dependent changes (think of it as small PRs depending one on another)

This involves building multiple, sequential branches, each representing a smaller part of a larger feature.

The goal: minimize developer wait times associated with lengthy code reviews by submitting smaller units.

Reviewing smaller pull requests is generally faster and more effective.

Meta created internal tools to manage the Git operations, such as rebasing, necessary for maintaining stacked branches.

Monorepo

Meta adopted a monorepo strategy, housing most of its codebase in a single repository.

This approach aimed to simplify collaboration and management of dependencies between different parts of the system.

Initially having multiple large repositories, Meta moved towards consolidating them for greater efficiency.

Tomas observes a trend of more companies adopting monorepos.

Impact of AI on software development

Tomas expects the use of AI tools to increase the speed and volume of code generation by developers.

This increase in code will place greater emphasis on the processes of code review and software testing to ensure quality.

AI has the potential to automate certain aspects of the code review process, allowing human reviewers to concentrate on more complex design and integration issues.

AI may also play a role in generating software tests, potentially increasing test coverage.

Still: human understanding and review of code will remain essential for verifying the intended functionality and business logic.

Engineering metrics to measure to get a sense of team performance

Commonly tracked metrics include the number of pull requests created and the time taken for them to be merged.

Uber implemented a metric measuring the time a pull request spent waiting for review without any action. This aimed to address delays in the review process, particularly in distributed teams.

While measuring developer focus time is challenging, it is recognized as an important factor in productivity.

Where to find Tomas Reimers:

• X: https://x.com/tomasreimers

• LinkedIn: https://www.linkedin.com/in/tomasreimers/

• Website: https://tomasreimers.com/

Mentions during the episode:

• Graphite: https://graphite.dev/

• GitHub: https://github.com/

• Stacked Diffs (and why you should know about them): https://newsletter.pragmaticengineer.com/p/stacked-diffs

• Meta developer tools: Working at scale: https://engineering.fb.com/2023/06/27/developer-tools/meta-developer-tools-open-source/

• A Meta developer's workflow: Exploring the tools used to code at scale: https://developers.facebook.com/blog/post/2022/11/15/meta-developers-workflow-exploring-tools-used-to-code/

• GitHub Actions: https://github.com/features/actions

• Buildkite: https://buildkite.com/

• Jira: https://www.atlassian.com/software/jira

• Linear: https://linear.app/

• Phabricator: https://graphite.dev/guides/phabricator-source-code-management-tool

• Supercharging A/B Testing at Uber: https://www.uber.com/blog/supercharging-a-b-testing-at-uber/

• Dropbox uses Phabricator extensively for all our projects: https://news.ycombinator.com/item?id=8656701

• Herald User Guide: https://secure.phabricator.com/book/phabricator/article/herald/

• GitHub code owners: https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners

• Stacked Pull Requests: https://www.gitkraken.com/gitkon/stacked-pull-requests-tomas-reimers

• Mercurial: https://www.mercurial-scm.org/

• Developer productivity with Dr. Nicole Forsgren (creator of DORA, co-creator of SPACE): https://newsletter.pragmaticengineer.com/p/developer-productivity-with-dr-nicole

• How Linux is built with Greg Kroah-Hartman: https://newsletter.pragmaticengineer.com/p/how-linux-is-built-with-greg-kroah

• Inside Meta's Engineering Culture: Part 1: https://newsletter.pragmaticengineer.com/p/facebook

• Inside Meta's Engineering Culture: Part 2: https://newsletter.pragmaticengineer.com/p/facebook-2

• Shopify: https://www.shopify.com/

• React: https://react.dev/

• Vercel: https://vercel.com/

• Andrej Karpathy’s post on X about vibe coding: https://x.com/karpathy/status/1886192184808149383

• Grammarly: https://www.grammarly.com/

• Heroku: https://www.heroku.com/

• Pull requests at GitHub: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-pull-requests

• How Uber is Measuring Engineering Productivity: https://newsletter.pragmaticengineer.com/p/uber-eng-productivity

• Statsig: https://statsig.com/

• Typescript: https://www.typescriptlang.org/

• Ruby: https://www.ruby-lang.org

• Python: https://www.python.org/

• The Last Days of Night: https://www.amazon.com/Last-Days-Night-Novel/dp/0812988922

• The Timeless Way of Building: https://www.amazon.com/Timeless-Way-Building-Christopher-Alexander/dp/0195024028/

—

Production and marketing by Pen Name. For inquiries about sponsoring the podcast, email podcast@pragmaticengineer.com.

Composite primary keys in Django

Django 5.2 is out today and a big new feature is composite primary keys, which can now be defined like this:

class Release(models.Model): pk = models.CompositePrimaryKey( "version", "name" ) version = models.IntegerField() name = models.CharField(max_length=20)

They don't yet work with the Django admin or as targets for foreign keys.

Other smaller new features include:

All ORM models are now automatically imported into ./manage.py shell - a feature borrowed from ./manage.py shell_plus in django-extensions Feeds from the Django syndication framework can now specify XSLT stylesheets response.text now returns the string representation of the body - I'm so happy about this, now I don't have to litter my Django tests with response.content.decode("utf-8") any more a new simple_block_tag helper making it much easier to create a custom Django template tag that further processes its own inner rendered content A bunch more in the full release notes

5.2 is also an LTS release, so it will receive security and data loss bug fixes up to April 2028.

Tags: django, python

[Mandy Brown]

A good reminder here from Mandy Brown.

"Among the people I’ve witnessed working through crises in their work and lives, the one pattern that comes up over and over again is making art. Art brings us back to ourselves, helps us root in our own agency and creative power, makes space for the joy of craft and play, and reminds us of our purpose in the world. On dark days, it’s easy to think that there’s no room for art, because the work of survival is so demanding. But art doesn’t merely take time—it gives time and energy back. It renews our spirits and the spirits of everyone who sees or hears or experiences the art, who receives the art as it’s intended: as a gift."

I sometimes have to remind myself that it's not frivolous; that it doesn't matter that it's not productive in a work sense. But it's not frivolous. It's living. It's being alive. And we all have the right to be alive.

#Culture

[Link]

🪦 oh no! Say it ain’t so Val. And just 65 😔

2025年4月1日から預貯金口座付番制度（口座管理法制度）拡充されました。この制度は、個人の同意を前提として、マイナンバーと金融機関の預貯金口座を紐付けるものであり、主に相続時や災害時における手続きの簡略化を目的としています。

【資料】

資料１：デジタル庁. (2024-04-01). 預貯金口座付番制度等の拡充について. 資料２：デジタル庁. (2024-04-01). 口座管理法制度って知っていますか？（A4両面・三つ折り形式） 資料３：デジタル庁. (2024-04-01). 口座管理法制度って知っていますか？（ポスター形式） 資料４：デジタル庁. (2024-03-31). 預貯金口座付番制度 主なメリット: 相続時・災害時の手続き簡略化 口座にマイナンバーを付番することで、相続時や災害時に、複数の金融機関に存在する預貯金口座の所在を容易に特定し、関連情報の提供を受けることが可能になります。 資料には、「相続時又は災害時に、預貯金口座の所在を特定し、その預貯金口座に関する情報の提供を受けることができます」と明記されています。（「資料２」より） 広報資料でも、「金融機関の窓口等で口座に付番しておくと、相続時や災害時には、付番口座の所在を把握できるようになるといったメリットを享受できます」と説明されています。（「資料１」より） マイナンバーによる一括連携 本人の同意があれば、金融機関やマイナポータルを通じて、一度に複数の金融機関の預貯金口座にマイナンバーを付番することができます。 資料には、「本人同意を前提とし、金融機関及びマイナポータルから、一 度に複数の預貯金口座への付番ができます」と記載されています。（「資料２」より） 図解では、マイナポータルまたは金融機関の窓口を通じて、複数の金融機関へマイナンバーを届け出る流れが示されています。（「資料２」より） 心配事などへの対応 任意性 金融機関へのマイナンバーの届出は任意であり、義務ではありません。 デジタル庁のウェブサイトでは、「なお、金融機関などへのマイナンバーの届出は任意です」と明記されています。（「預貯金口座付番制度等の拡充について｜デジタル庁」より） プライバシーへの配慮 マイナンバーを届け出たとしても、その情報が国の資産把握に直結するわけではありません。金融機関がマイナンバーの届出をきっかけに、国の預貯金残高などを通知することはないと強調されています。 資料には、「マイナンバーの届出をきっかけに、金融機関 が国に預貯金残高などをお知らせすることは ないから安心してね」というQ&A形式の説明があります。（「資料２」より） また、「従来より、国が預貯金者の口座情報を確認 できるのは、法令に基づき、必要な社会保障 の資力調査や税務調査などを行う場合に 限られている」とも説明されています。（「資料２」より） マイナンバーカードに口座情報が登録されることもなく、紛失・盗難時の一時利用停止措置も用意されています。（「資料２」より） 開始時期 口座管理法制度は、令和7年（2025年）4月1日から開始されます。 資料の冒頭には、「令和７年 ４月１日 開始！」と大きく記載されています。（「資料３」より） 手続き〜どうやったら登録できるの？ 本人の同意を前提として、金融機関およびマイナポータルから、一度に複数の預貯金口座へのマイナンバーの付番を申し出ることができます 。 具体的には、以下の方法で登録（マイナンバーの付番の申出）ができると考えられます。 金融機関の窓口：金融機関の窓口にマイナンバーカードを持参して、口座にマイナンバーを届け出ることで、付番の手続きを行うことができます。一つの金融機関の窓口で、マイナンバーが付番された預貯金口座の所在を確認できるようになる予定です。 マイナポータル：マイナポータルを通じて、複数の金融機関の預貯金口座に対して、一度にマイナンバーの付番を申し込むことができるようになります。ただし、4月2日11:47段階ではまだ準備ができていないようで、フリーダイヤルで聞いたところ、金融機関の窓口でやるか、しばらく時間をおいて再度問い合わせしてほしいとのことでした。 重要事項 口座管理法制度: 預貯金口座へのマイナンバーの付番制度を指します。 マイナンバー付番のメリット: 相続・災害時の手続きを簡略化し、複数の口座情報を一括で確認できるようになること。 マイナポータル: マイナンバーに関するオンラインサービスを通じて、預貯金口座への付番手続きが可能になる予定です。 公金受取口座登録制度との違い: 預貯金口座付番制度は、あくまで相続・災害時の情報確認を容易にするためのものであり、給付金等の受取口座を国に登録する「公金受取口座登録制度」とは異なります。 デジタル庁のウェブサイトでは、この二つの制度の違いが明確に説明されています。（「預貯金口座付番制度｜デジタル庁」より） 住所変更の重要性: 金融機関に登録されている住所等の情報が最新でない場合、マイナンバーの付番ができない可能性があるため、事前の情報更新が推奨されています。 資料には、「金融機関に届け出ている住所等の更新が 出来ていないと、本人の口座だと認識されずに、 マイナンバーの付番が出来ないよ」という注意喚起があります。（「資料２」より） 問い合わせ窓口: マイナンバー制度全般や本件に関する問い合わせ先として、マイナンバー総合フリーダイヤル（0120-95-0178）などが案内されています。（「資料２」より） 結論

2025年4月1日から開始される口座管理法制度は、マイナンバーを活用して預貯金口座情報を一元的に管理し、相続や災害といった緊急時における手続きの負担を軽減することを目的としています。マイナンバーの届出は任意であり、プライバシーへの配慮もなされています。制度の利用にあたっては、金融機関への登録情報の確認・更新が重要となります。

Dave Taht on FLOSS Weekly

I just learned here that Dave Täht died today, at just 59. I don’t know how or where. I do know he was one of the world’s great human beings, as well as a brilliant and generous producer of tech and wisdom about it.

Example: if you know about (and no longer have a network that suffers from) bufferbloat, thank Dave.

I’ve lost count of how often Dave has been a guest on one of my podcasts. He was always reliably informative, good-humored, and a source of music and songs. Behind him in the screen grab above is the guitar he carried everywhere. The sticker says, “This guitar kills Vogons.”

Here are some links I’ll be watching for more news. Feel free to contribute:

Google search Reddit search Wikipedia LibreQoS’ page, remembering Dave Dave’s Xitter Search for Dave on BlueSky

 

 

 

I like the idea of categorisation in 🍎 email and the 4 categories make sense to 🗂️ BUT sure would like the ability to add a couple more of my own. #EdgeCaseAgain

Last Wednesday, March 26, we hosted the second of five virtual Extension Mini-Summits in the lead up to the big one at the Postgres Development Conference (PGConf.dev) on May 13 in Montreal, Canada. Peter Eisentraut gave a very nice presentation on the history, design decisions, and problems solved by “Implementing an Extension Search Path”. That talk, plus another 10-15m of discussion, is now available for your viewing pleasure:

Video Slides

If you’d like to attend any of the next three Mini-Summits, join the Meetup!

Once again, with many thanks again to Floor Drees for the effort, here’s the transcript from the session.

Introduction

Floor Drees introduced the organizers:

David Wheeler, Principal Architect at Tembo, maintainer of PGXN Yurii Rashkovskii, Omnigres Keith Fiske, Crunchy Data Floor Drees, Principal Program Manager at EDB, PostgreSQL CoCC member, PGDay Lowlands organizer

Peter Eisentraut, contributor to PostgreSQL development since 1999, talked about implementing an extension search path.

The stream and the closed captions available for the recording are supported by PGConf.dev and their gold level sponsors, Google, AWS, Huawei, Microsoft, and EDB.

Implementing an extension search path

Peter: Thank you for having me!

I’m gonna talk about a current project by me and a couple of people I have worked with, and that will hopefully ship with Postgres 18 in a few months.

So, what do I know about extensions? I’m a Postgres core developer, but I’ve developed a few extensions in my time, here’s a list of extensions that I’ve built over the years.

plsh pguint pgpcre pguri plxslt pgemailaddr pgtrashcan

Some of those are experiments, or sort of one-offs. Some of those are actually used in production.

I’ve also contributed to well-known extensions: orafce; and back in the day, pglogical, BDR, and pg_failover_slots, at EDB, and previously 2ndQuadrant. Those are obviously used widely and in important production environments.

I also wrote an extension installation manager called pex at one point. The point of pex was to do it in one shell script, so you don’t have any dependencies. It’s just a shell script, and you can say pex install orafce and it installs it. This was a proof of concept, in a sense, but was actually quite useful sometimes for development, when you just need an extension and you don’t know where to get it.

And then I wrote, even more experimental, a follow-on project called autopex, which is a plugin module that you load into Postgres that automatically installs an extension if you need it. If you call CREATE EXTENSION orafce, for example, and you don’t have it installed, autopex downloads and installs it. Obviously highly insecure and dubious in terms of modern software distribution practice, but it does work: you can just run CREATE EXTENSION, and it just installs it if you don’t have it. That kind of works.

So anyways, so I’ve worked on these various aspects of these over time. If you’re interested in any of these projects, they’re all under my GitHub account.

In the context of this presentation…this was essentially not my idea. People came to me and asked me to work on this, and as it worked out, multiple people came to me with their problems or questions, and then it turned out it was all the same question. These are the problems I was approached about.

The first one is extension management in the Kubernetes environment. we’ll hear about this in a future talk in this series. Gabriele Bartolini from the CloudNativePG project approached me and said that the issue in a Kubernetes environment is that if you launch a Postgres service, you don’t install packages, you have a pre-baked disk image that contains the software that you need. There’s a Postgres server and maybe some backup software in that image, and if you want to install an extension, and the extension is not in that image, you need to rebuild the image with the extension. That’s very inconvenient.

The ideal scenario would be that you have additional disk images for the extensions and you just somehow attach them. I’m hand waving through the Kubernetes terminology, and again, there will be a presentation about that in more detail. But I think the idea is clear: you want to have these immutable disk images that contain your pieces of software, and if you want to install more of them, you just wanna have these disk images augment ’em together, and that doesn’t work at the moment.

Problem number two is: I was approached by a maintainer of the Postgres.app project, a Mac binary distribution for Postgres. It’s a nice, user-friendly binary distribution for Postgres. This is sort of a similar problem: on macOS you have these .app files to distribute software. They’re this sort of weird hybrid between a zip file with files in it and a directory you can look into, so it’s kind of weird. But it’s basically an archive with software in it. And in this case it has Postgres in it and it integrates nicely into your system. But again, if you want to install an extension, that doesn’t work as easily, because you would need to open up that archive and stick the extension in there somehow, or overwrite files.

And there’s also a tie in with the way these packages are signed by Apple, and if you, mess with the files in the package, then the signature becomes invalid. It’s the way it’s been explained to me. I hope this was approximately accurate, but you already get the idea, right? There’s the same problem where you have this base bundle of software that is immutable or that you want to keep immutable and you want to add things to it, which doesn’t work.

And then the third problem I was asked to solve came from the Debian package maintainer, who will also speak later in this presentation series. What he wanted to do was to run the tests of an extension while the package is being built. That makes sense. You wanna run the tests of the software that you’re building the package for in general. But in order to do that, you have to install the extension into the the normal file system location, right? That seems bad. You don’t want to install the software while you’re into the main system while you’re building it. He actually wrote a custom patch to be able to do that, which then my work was inspired by.

Those are the problems I was approached about.

I had some problems I wanted to solve myself based on my experience working with extensions. While I was working on these various extensions over the years, one thing that never worked is that you could never run make check. It wasn’t supported by the PGXS build system. Again, it’s the same issue.

It’s essentially a subset of the Debian problem: you want to run a test of the software before you install it, but Postgres can only load an extension from a fixed location, and so this doesn’t work. It’s very annoying because it makes the software development cycle much more complicated. You always have to then, then run make all, make install, make sure you have a server running, make installcheck. And then you would want to test it against various different server versions. Usually they have to run this in some weird loop. I’ve written custom scripts and stuff all around this, but it’s was never satisfactory. It should just work.

That’s the problem I definitely wanted to solve. The next problem — and these are are all subsets of each other — that if you have Postgres installed from a package, like an RPM package for example, and then you build the extension locally, you have to install the extension into the directory locations that are controlled by your operating system. If you have Postgres under /usr, then the extensions also have to be installed under /usr, whereas you probably want to install them under /usr/local or somewhere else. You want to keep those locally built things separately, but that’s not possible.

And finally — this is a bit more complicated to explain — I’m mainly using macOS at the moment, and the Homebrew package manager is widely used there. But it doesn’t support extensions very well at all. It’s really weird because the way it works is that each package is essentially installed into a separate subdirectory, and then it’s all symlinked together. And that works just fine. You have a bunch of bin directories, and it’s just a bunch of symlinks to different subdirectories and that works, because then you can just swap these things out and upgrade packages quite easily. That’s just a design choice and it’s fine.

But again, if you wanna install an extension, the extension would be its own package — PostGIS, for example — and it would go into its own directory. But that’s not the directory where Postgres would look for it. You would have to install it into the directory structure that belongs to the other package. And that just doesn’t work. It’s just does not fit with that system at all. There are weird hacks at the moment, but it’s not satisfactory. Doesn’t work at all.

It turned out, all of these things have sort of came up over the years and some of these, people have approached me about them, and I realized these are essentially all the same problem. The extension file location is hard-coded to be inside the Postgres installation tree. Here as an example: it’s usually under something like /usr/share/postgresql/extension/, and you can’t install extensions anywhere else. If you want to keep this location managed by the operating system or managed by your package management or in some kind of immutable disk image, you can’t. And so these are essentially all versions of the same problem. So that’s why I got engaged and tried to find a solution that addresses all of ’em.

I had worked on this already before, a long time ago, and then someone broke it along the way. And now I’m fixing it again. If you go way, way back, before extensions as such existed in Postgres in 9.1, when you wanted to install a piece of software that consists of a shared library object and some SQL, you had to install the shared library object into a predetermined location just like you do now. In addition, you had to run that SQL file by hand, basically, like you run psql -f install_orafce.sql or something like that. Extensions made that a little nicer, but it’s the same idea underneath.

In 2001, I realized this problem already and implemented a configuration setting called dynamic_library_path, which allows you to set a different location for your shared library. Then you can say

dynamic_library_path = '/usr/local/my-stuff/something'

And then Postgres would look there. The SQL file just knows where is because you run it manually. You would then run

psql -f /usr/local/my-stuff/something/something.sql

That fixed that problem at the time. And when extensions were implemented, I was essentially not paying attention or, you know, nobody was paying attention. Extension support were a really super nice feature, of course, but it broke this previously-available feature: then you couldn’t install your extensions anywhere you wanted to; you were tied to this specific file system, location, dynamic_library_path still existed: you could still set it somewhere, but you couldn’t really make much use of it. I mean, you could make use of it for things that are not extensions. If you have some kind of plugin module or modules that install hooks, you could still do that. But not for an extension that consist of a set of SQL scripts and a control file and dynamic_library_path.

As I was being approached about these things, I realized that was just the problem and we should just now fix that. The recent history went as follows.

In April, 2024, just about a year ago now, David Wheeler started a hackers thread suggesting Christoph Berg’s Debian patch as a starting point for discussions. Like, “here’s this thing, shouldn’t we do something about this?”

There was, a fair amount of discussion. I was not really involved at the time. This was just after feature freeze,and so I wasn’t paying much attention to it. But the discussion was quite lively and a lot of people pitched in and had their ideas and thoughts about it. And so a lot of important, filtering work was done at that time.

Later, in September, Gabriele, my colleague from EDB who works on CloudNativePG, approached me about this issue and said like: “hey, this is important, we need this to make extensions useful in the Kubernetes environment.” And he said, “can you work, can you work on this?”

I said, “yeah, sure, in a couple months I might have time.” [Laughs]. But it sort of turns out that, at PGConf.EU we had a big brain trust meeting of various people who basically all came and said, “hey, I heard you’re working on extension_control_path, I also need that!”

Gabriele was there, and Tobias Bussmann from Postgres.app was there ,and Christoph, and I was like, yeah, I really need this extension_control_path to make this work. So I made sure to talk to everybody there and, and make sure that, if we did this, would it work for you? And then we kind of had a good idea of how it should work.

In November the first patch was posted and last week it was committed. I think there’s still a little bit of discussion of some details and, we certainly still have some time before the release to fine tune it, but the main work is hopefully done.

This is the commit I made last week. The fact that this presentation was scheduled gave me additional motivation to get it done. I wanna give some credits to people who reviewed it. Obviously David did a lot of reviews and feedback in general. My colleague Matheus, who I think I saw him earlier, he was also here on the call, did help me quite a bit with sort of finishing the patch. And then Gabriele, Marco and Nicolò, who work on CloudNativePG, did a large amount of testing.

They set up a whole sort of sandbox environment making test images for extensions and, simulating the entire process of attaching these to the main image. Again, I’m butchering the terminology, but I’m just trying to explain it in general terms. They did the whole end-to-end testing of what that would then look like with CloudNativePG. And again, that will, I assume, be discussed when Gabriele presents in a few weeks.

These are the stats from the patch

commit 4f7f7b03758 doc/src/sgml/config.sgml | 68 +++++ doc/src/sgml/extend.sgml | 19 +- doc/src/sgml/ref/create_extension.sgml | 6 +- src/Makefile.global.in | 19 +- src/backend/commands/extension.c | 403 +++++++++++++++++---------- src/backend/utils/fmgr/dfmgr.c | 77 +++-- src/backend/utils/misc/guc_tables.c | 13 + src/backend/utils/misc/postgresql.conf.sample | 1 + src/include/commands/extension.h | 2 + src/include/fmgr.h | 3 + src/test/modules/test_extensions/Makefile | 1 + src/test/modules/test_extensions/meson.build | 5 + .../modules/test_extensions/t/001_extension_control_path.pl | 80 ++++++

the reason I show this is that, it’s not big! What I did is use the same infrastructure and mechanisms that already existed for the dynamic_library_path. That’s the code in that’s in dfmgr there in the middle. That’s where this little path search is implemented9. And then of course, in extension..c there’s some code that’s basically just a bunch of utility functions, like to list all the extensions and list all the versions of all the extensions. Those utility functions exist and they needed to be updated to do the path search. Everything else is pretty straightforward. There’s just a few configuration settings added to the documentation and the sample files and so on. It’s not that much really.

One thing we also did was add tests for this, Down there in test_extensions. We wrote some tests to make sure this works. Well, it’s one thing to make sure it works, but the other thing is if we wanna make changes or we find problems with it, or we wanna develop this further in the future, we have a record of how it works, which is why you write tests. I just wanted to point that out because we didn’t really have that before and it was quite helpful to build confidence that we know how this works.

So how does it work? Let’s say you have your Postgres installation in a standard Linux file system package controlled location. None of the actual packages look like this, I believe, but it’s a good example. You have your stuff under the /usr/bin/, you have the shared libraries in the /usr/lib/something, you have the extension control files and SQL files in the /usr/share/ or something. That’s your base installation. And then you wanna install your extension into some other place to keep these things separate. So you have /usr/local/mystuff/, for example.

Another thing that this patch implemented is that you can now also do this: when you build an extension, you can write make install prefix=something. Before you couldn’t do that, but there was also no point because if you installed it somewhere else, you couldn’t do anything with it there. Now you can load it from somewhere else, but you can also install it there — which obviously are the two important sides of that.

And then you set these two settings: dynamic_library_path is an existing configuration setting, yYou set that to where your lib directory is, and then the extension_control_path is a new setting. The titular setting of this talk, where you tell it where your extension control files are.

There’s these placeholders, $libdir and $system which mean the system location, and then the other locations are your other locations, and it’s separated by colon (and semi-colon on Windows). We had some arguments about what exactly the extension_control_path placeholder should be called and, people continue to have different opinions. What it does is it looks in the list directories for the control file, and then where it finds the control file from there, it loads all the other files.

And there’s a fairly complicated mechanism. There’s obviously the actual SQL files, but there’s also these auxiliary control files, which I didn’t even know that existed. So you can have version specific control files. It’s a fairly complicated system, so we wanted to be clear that what is happening is the, the main control file is searched for in these directories, and then wherever it’s found, that’s where it looks for the other things. You can’t have the control file in one path and then the SQL files in another part of the path; that’s not how it works.

That solves problem number five. Let’s see what problem number five was. I forgot [Chuckles]. This is the basic problem, that you no longer have to install the extensions in the directories that are ostensibly controlled by the operating system or your package manager.

So then how would Debian packaging use this? I got this information from Christoph. He figured out how to do this. He just said, “Oh, I did this, and that’s how it works.” During packaging, the packaging scripts that built it up in packages that you just pass these:

PKGARGS="--pgoption extension_control_path=$PWD/debian/$PACKAGE/usr/share/postgresql/$v/extension:\$system --pgoption dynamic_library_path=$PWD/debian/$PACKAGE/usr/lib/postgresql/$v/lib:/usr/lib/postgresql/$v/lib"

These options set the control path and the dynamic_library_path and these versions and then it works. This was confirmed that this addresses his problem. He no longer has to carry his custom patch. This solves problem number three.

The question people ask is, “why do we have two?” Or maybe you’ve asked yourself that. Why do we need two settings. We have the dynamic_library_path, we have the extension_control_path. Isn’t that kind of the same thing? Kind of, yes! But in general, it is not guaranteed that these two things are in a in a fixed relative location.

Let’s go back to our fake example. We have the libraries in /usr/lib/postgresql and the SQL and control files in /usr/share/postgresql, for example. Now you could say, why don’t we just set it to /usr? Or, for example, why don’t we just set the path to /usr/local/mystuff and it should figure out the sub directories. That would be nice, but it doesn’t quite work in general because it’s not guaranteed that those are the subdirectories. There could be, for example. lib64, for example, right? Or some other so architecture-specific subdirectory names. Or people can just name them whatever they want. So, this may be marginal, but it is possible. You need to keep in mind that the subdirectory structure is not necessarily fixed.

So we need two settings. The way I thought about this, if you compile C code, you also have two settings. And if you think about it, it’s exactly the same thing. When you compile C code, you always have to do -I and -L: I for the include files, L for the lib files. This is basically the same thing. The include file is also the text file that describes the interfaces and the libraries are the libraries. Again, you need two options, because you can’t just tell the compiler, oh, look for it in /usr/local because the subdirectories could be different. There could be architecture specific lib directories. That’s a common case. You need those two settings. Usually they go in parallel. If somebody has a plan on how to do it simpler, follow up patches are welcome.

But the main point of why this approach was taken is also to get it done in a few months. I started thinking about this, or I was contacted about this in September and I started thinking about it seriously in the October/November timeframe. That’s quite late in the development cycle to start a feature like this, which I thought would be more controversial! People haven’t really complained that this breaks the security of extensions or anything like that. I was a little bit afraid of that.

So I wanted to really base it on an existing facility that we already had, and that’s why I wanted to make sure it works exactly in parallel to the other path that we already have, and that has existed for a long time, and was designed for this exact purpose. That was also the reason why we chose this path of least resistance, perhaps.

This is the solution progress for the six problems that I described initially. The CloudNativePG folks obviously have accompanied this project actively and have already prototyped the integration solution. And, and presumably we will hear about some of that at the meeting on May 7th, where Gabriele will talk about this.

Postgres.app I haven’t been in touch with, but one of the maintainers is here, maybe you can give feedback later. Debian is done as I described, and they will also be at the next meeting, maybe there will be some comment on that.

One thing that’s not fully implemented is the the make check issue. I did send a follow-up patch about that, which was a really quick prototype hack, and people really liked it. I’m slightly tempted to give it a push and try to get it into Postgres 18. This is a work in progress, but it’s, there’s sort of a way forward. The local install problem I said is done.

Homebrew, I haven’t looked into. It’s more complicated, and I’m also not very closely involved in the development of that. I’ll just be an outsider maybe sending patches or suggestions at some point, maybe when the release is closer and, and we’ve settled everything.

I have some random other thoughts here. I’m not actively working on these right now, but I have worked on it in the past and I plan to work on it again. Basically the conversion of all the building to Meson is on my mind, and other people’s mind.

Right now we have two build systems: the make build system and the Meson build system, and all the production packages, as far as I know, are built with make. Eventually we wanna move all of that over to Meson, but we want to test all the extensions and if it still works. As far as I know, it does work; there’s nothing that really needs to be implemented, but we need to go through all the extensions and test them.

Secondly — this is optional; I’m not saying this is a requirement — but you may wish to also build your own extensions with Meson. But that’s in my mind, not a requirement. You can also use cmake or do whatever you want. But there’s been some prototypes of that. Solutions exist if you’re interested.

And to facilitate the second point, there’s been the proposal — which I think was well received, but it just needs to be fully implemented — to provide a pkg-config file to build against the server, and cmake and Meson would work very well with that. Then you can just say here’s a pkg-config file to build against the server. It’s much easier than setting all the directories yourself or extracting them from pg_config. Maybe that’s something coming for the next release cycle.

That’s what I had. So extension_control_path is coming in Postgres 18. What you can do is test and validate that against your use cases and and help integration into the downstream users. Again, if you’re sort of a package or anything like that, you know, you can make use of that. That is all for me.

Thank you!

Questions, comments

Reading the comments where several audience members suggested Peter follows Conference Driven Development he confirmed that that’s definitely a thing.

Someone asked for the “requirements gathering document”. Peter said that that’s just a big word for “just some notes I have”. “It’s not like an actual document. I called it the requirements gathering. That sounds very formal, but it’s just chatting to various people and someone at the next table overheard us talking and it’s like, ‘Hey! I need that too!’”

Christoph: I tried to get this fixed or implemented or something at least once over the last 10 something-ish years, and was basically shot down on grounds of security issues if people mess up their system. And what happens if you set the extension path to something, install an extension, and then set the path to something else and then you can’t upgrade. And all sorts of weird things that people can do with their system in order to break them. Thanks for ignoring all that bullshit and just getting it done! It’s an administrator-level setting and people can do whatever they want with it.

So what I then did is just to implement that patch and, admittedly I never got around to even try to put it upstream. So thanks David for pushing that ahead. It was clear that the Debian version of the patch wasn’t acceptable because it was too limited. It made some assumptions about the direct restructure of Debian packages. So it always included the prefix in the path. The feature that Peter implemented solves my problem. It does solve a lot of more problems, so thanks for that.

Peter: Testing all extensions. What we’ve talked about is doing this through the Debian packaging system because the idea was to maybe make a separate branch or a separate sub-repository of some sort, switch it to build Meson, and rebuild all the extension packages and see what happens. I guess that’s how far we’ve come. I doesn’t actually mean they all work, but I guess that most of them has tests, so we just wanted to test, see if it works.

There are some really subtle problems. Well, the ones I know of have been fixed, but there’s some things that certain compilation options are not substituted into the Makefiles correctly, so then all your extensions are built without any optimizations, for example, without any -O options. I’m not really sure how to detect those automatically, but at least, just rebuild everything once might be an option. Or just do it manually. There are not thousands of extensions. There are not even hundreds that are relevant. There are several dozens, and I think that’s good coverage.

Christoph: I realize that doing it on the packaging side makes sense because we all have these tests running. So I was looking into it. The first time I tried, I stopped once I realized that Meson doesn’t support LLVM yet; and the second time I tried, I just diff-ed the generated Makefiles to see if there’s any difference that looks suspicious. At thus point I should just continue and do compilation run and see what the tests are doing and and stuff.

So my hope would be that I could run diff on the results; the problem is compiling with Postgres with Autoconf once and then with Meson the second time, then see if it has an impact on the extensions compiled. But my idea was that if I’m just running diff on the two compilations and there’s no difference, there’s no point in testing because they’re identical anyway.

Peter Oooh, you want the actual compilation, for the Makefile output to be the same.

Christoph: Yeah. I don’t have to run that test, But the diff was a bit too big to be readable. There was lots of white space noise in there. But there were also some actual changes. Some were not really bad, like9 in some points variables were using a fully qualified path for the make directory or something, and then some points not; but, maybe we can just work on making that difference smaller and then arguing about correctness is easier.

Peter: Yeah, that sounds like a good approach.

Jakob: Maybe I can give some feedback from Postgres.app. So, thank you very much. I think this solves a lot of problems that we have had with extensions over the years, especially because it allows us to separate the extensions and the main Postgres distribution. For Postgres.app we basically have to decide which extensions to include and we can’t offer additional extensions when people ask for them without shipping them for everyone. So that’s a big win.

One question I am wondering about is the use case of people building their own extensions. As far as I understand, you have to provide the prefix/ And one thing I’m wondering whether there is there some way to give a default value for the prefix. Like in pg_config or in something like that, so people who just type make install automatically get some path.

Peter: That might be an interesting follow on. I’m making a note of it. I’m not sure how you’d…

Jakob: I’m just thinking because a big problem is that a lot of people who try things don’t follow the instructions for the specific Postgres. So for example, if we write documentation how to build extensions and people on a completely different system — like people Google stuff and they get instruction — they’ll just try random paths. Right now, if you just type make install, it works on most systems because it just builds into the standard directories.

Peter: Yeah, David puts it like, “should there be a different default extension location?” I think that’s probably not an unreasonable direction. I think that’s something we should maybe think about, once this is stabilized. I think for your Postgres.app use case, it, I think you could probably even implement that yourself with a one or two line patch so that at least, if you install Postgres.app, then somebody tries to build an extension, they get a reasonable location.

David: If I could jump in there, Jakob, my assumption was that Postgres.app would do something like designate the Application Support directory and Preferences in ~/Library as where extensions should be installed. And yeah, there could be some patch to PGXS to put stuff there by default.

Jakob: Yeah, that would be nice!

Peter: Robert asked a big question here. What do we think the security consequences of this patch? Well, one of the premises is that we already have dynamic_library_path, which works exactly the same way, and there haven’t been any concerns about that. Well, maybe there have been concerns, but nothing that was acted on. If you set the path to somewhere where anybody can write stuff, then yeah, that’s not so good. But that’s the same as anything. Certainly there were concerns as I read through the discussion.

I assumed somebody would hav security questions, so I really wanted to base it on this existing mechanism and not invent something completely new. So far nobody has objected to it [Chuckles]. But yeah, of course you can make a mess of it if you go into that extension_control_path = /tmp! That’s probably not good. But don’t do that.

David: That’s I think in part the xz exploit kind of made people more receptive to this patch because we want to reduce the number of patches that packaging maintainers have to maintain.

Peter: Obviously this is something people do. Better we have one solution that people then can use and that we at least we understand, as opposed to everybody going out and figuring out their own complicated solutions.

David: Peter, I think there are still some issues with the behavior of MODULEDIR from PGXS and directory in the control file that this doesn’t quite work with this extension. Do you have some thoughts on how to address those issues?

Peter: For those who are not following: there’s an existing, I guess, rarely used feature that, in the control file, you can specify directory options, which then specifies where other files are located. And this doesn’t work the way you think it should maybe it’s not clear what that should do if you find it in a path somewhere. I guess it’s so rarely used that we might maybe just get rid of it; that was one of the options.

In my mental model of how the C compiler works, it sets an rpath on something. If you set an absolute rpath somewhere and you know it’s not gonna work if you move the thing to a different place in the path. I’m not sure if that’s a good analogy, but it sort of has similar consequences. If you hard-code absolute path, then path search is not gonna work. But yeah, that’s on the list I need to look into.

David: For what it’s worth, I discovered last week that the part of this patch where you’re stripping out $libdir and the extension make file that was in modules, I think? That also needs to be done when you use rpath to install an extension and point to extensions today with Postgres 17. Happy to see that one go.

Christoph: Thanks for fixing that part. I was always wondering why this was broken. The way it was broken. It looked very weird and it turned out it was just broken and not me not understanding it.

David: I think it might have been a documentation oversight back when extensions were added at 9.1 to say this is how you list the modules.

Anyway, this is great! Im super excited for this patch and where it’s going and the promise for stuff in the future. Just from your list of the six issues it addresses, it’s obviously something that covers a variety of pain points. I appreciate you doing that.

Peter: Thank you!

Many thanks and congratulations wrap up this call.

The next Mini-Summit is on April 9, Christoph Berg (Debian, and also Cybertec) will join us to talk about Apt Extension Packaging.

More about… Postgres Extensions PGConf Summit Peter Eisentraut

Man cannot endure his own littleness unless he can translate it into meaningfulness on the largest possible level.

💬 Ernest Becker

… and then I thought of 🔗🎵 The Overview

🕵💬 Inspired By People | 🌐 Vala Afshar

🔍 All The Posts can be found here, or scroll through the quotes below.

🔗 Mathematicians uncover the logic behind how people walk in crowds | ScienceDaily

.. not read it yet .. but wondering f the equation includes parameters around the phone that is inevitably in people’s hands, more often than not being looked at.

When the ‘fake news’ sounds like reality .. even to someone that tries to be on the lookout for the plain wrong .. somehow April 1st slips by without a whimper.

Hi – this is Gergely with the monthly, free issue of the Pragmatic Engineer Newsletter. In every issue, I cover challenges at Big Tech and startups through the lens of engineering managers and senior engineers. If you’ve been forwarded this email, you can subscribe here.

It’s been widely reported that the tech hiring market is much cooler than in 2020-2022; the number of software engineering job openings is down internationally in all major regions and the number of full-remote roles is in steady decline. Meanwhile, other metrics indicate that tech hiring is starting to recover – at least for senior engineers – as covered last month in the article, State of the startup and scaleup hiring markets, as seen by recruiters. It all adds up to a state of flux for candidates and employers to navigate through.

This article is an attempt to get clarity about how tech interviews are changing, by focusing on what the engineers who take interviews are seeing. For this, I turned to Evan King and Stefan Mai, cofounders of interview preparation startup, Hello Interview. Before starting it, Evan was a staff engineer at Meta for 4 years, and Stefan an engineering manager at Amazon for 6 years, and also a senior engineering manager at Meta. They’ve conducted hundreds of interviews, while Stefan has also been a hiring manager. Since launching their new business, they’ve helped thousands of engineers prepare for interviews, and have collected information on the pulse of the job market.

I reached out to them after reading their practical, fresh take on system design interviews, for candid takes on devs interviewing at startups and Big Tech in the current climate, especially compared to a few years ago. Today, we cover:

New reality of tech hiring. A rebounding market still well below its 2021-2022 peak.

Analyzing the tech hiring market. Artificial Intelligence (AI) and related sectors are hot, while frontend/backend/mobile are not. It’s tougher for new grads than experienced engineers.

Interview process changes. The formats of DSA and system design interviews remain the same, but are more demanding. Downleveling is more common, and team matching has quietly become another hurdle to clear.

Interview formats differ between startups and Big Tech. Startups embrace more practical interviews and AI tools, while Big Tech seems less flexible about changing approach.

Preparation strategies by experience. Advice for entry-level, mid-level, senior, staff+ tech professionals, and for EMs.

Silver linings. Big Tech hiring is up, there’s a boom in AI positions, and the playbook of interviews is public.

In the article below we’ll cover how demand for engineers in the AI field keeps being strong. For pointers on picking up engineering practices, see the deepdives AI Engineering in the real world and AI Engineering with Chip Huyen.

With that, it’s over to Evan and Stefan:

1. New reality of tech hiring

Three years ago, if you were a competent software engineer with 3+ years of experience, you likely had recruiters flooding your inbox with opportunities. Companies were fighting over engineering talent, throwing extraordinary compensation packages at candidates, and in some cases even looking past poor interview performance in order to secure hires faster. The 2020-2021 tech hiring frenzy was exceptional; a period many now look back on with a mixture of nostalgia and disbelief.

Fast forward to 2025, and the landscape has transformed dramatically. As co-founders of HelloInterview.com, we've had front row seats to these changes, observing tens of thousands of engineering interview journeys across companies of all sizes. In this deepdive, we aim to give you the unvarnished reality of tech interviewing in 2025, via real experiences of candidates navigating it today.

We’ve observed Big Tech’s hiring volumes are up roughly 40% year on year. This data comes from candidates currently working at late-stage companies, of whom the overwhelming majority use our platform to prepare for interviews they already have scheduled. This provides a reliable proxy for overall tech hiring trends, as candidates on our platform have immediate, concrete interview dates. An uptick in candidates getting more interviews suggests that the worst of the 2022-2023 tech winter has passed, and that there are more attractive openings worth preparing for. Still, we're operating in a fundamentally different market with new rules, expectations, and challenges.

The 40%-rebound figure is only part of the story. Yes, tech hiring is slowly making a comeback in aggregate terms, but it's a selective, strategic recovery that leaves some qualified engineers struggling to navigate processes which are now more demanding and less forgiving. Companies once desperate to fill seats are now being methodical and cautious, prioritizing precision in hiring decisions over speed and volume.

What we're witnessing isn't simply a market correction; it’s a subtle yet significant shift in evaluation standards. While the core interview structure at Big Tech remains largely unchanged, the bar has shifted approximately one standard deviation higher across the board, and performance that would have secured an offer in 2021 might not even clear the screening stage today.

2. Analyzing the tech hiring market

Here’s our take on the current job market.

Selective recovery

By the raw numbers, tech hiring appears on a solid upward trajectory. TrueUp.io's job trend tracking shows tech job postings have risen from a 2023 low of 163,000, to approximately 230,000 today; roughly a 41% increase.

Number of open tech jobs at tech startups, tech unicorns, and public tech companies. Source: TrueUp

The 42% increase in openings is consistent with what we've observed internally in HelloInterview usage metrics and mock interview volume, when we adjust data for candidates with interviews scheduled.

We are still well below the feverish heights of 2020-2022, though. Back then, open roles peaked at close to 500,000. The current recovery, while significant, has only restored us to around 46% of that peak.

Unlike previous tech hiring cycles when a rising tide lifted all boats, today's market is characterized by extreme selectivity. Companies have become far more picky about where they invest headcount, with major differences in opportunity based on specialization, experience level, and the prestige of ex-employers.

Domain specialization

Engineers in certain areas of specialization are seeing a lot of relevant openings, such as in:

AI infrastructure

Machine learning operations

Generative AI application development

These areas of hiring are reminiscent of the 2021 peak; often with multiple offers, aggressive compensation, and expedited interview processes.

For example, a Bay Area staff engineer specializing in AI infrastructure at Google recently received a competing offer from Meta's AI infrastructure team which was above $1 million in total compensation. Previously at Google, numbers like this were typically reserved for senior staff positions. But getting a large pay bump when changing companies, while staying at the same level, is not an isolated incident; we're seeing similar-sized compensation packages for specialists in high-performance computing, ML systems design, and those specializing in responsible AI development.

Engineers in “core” domains see fewer opportunities. “Core domains” refers to frontend, backend services, mobile development, and similar areas. Later-stage startups that previously maintained multiple teams in these areas have consolidated with more empowered full-stack engineers. Focusing on full-stack leads to lower overall headcount, fewer openings, and more selective hiring processes. We see candidates with strong backgrounds in these areas often taking a long time to land a role, and when they do get an offer, the comp growth is rarely above what they currently earn. Note from Gergely: we previously saw how native mobile engineers face a tougher job market, and that becoming more full-stack is a sensible strategy for being more employable.

Senior engineers can still attract multiple offers, especially those with directly relevant experience for hiring companies. This could be deep domain expertise (e.g. working in the infrastructure domain when interviewing for infra teams, working in the finance domain when interviewing with FinTechs, etc), or it could be a deep technology expertise which matters to the employer. Meanwhile, engineers with less transferable skills face an uphill battle. Narrow skillsets often develop from working at the likes of Google or Meta, where people specialize narrowly in proprietary systems, tools, and technologies that don’t exist in the broader market.

Experience level divide

The current market is also starkly stratified by career stage, creating dramatically different realities for engineers depending on experience:

Junior engineers and new grads face the biggest challenge. We spoke with a job seeker based in India who graduated from IIT – the most prestigious computer science university in the country. They shared a meticulously-maintained spreadsheet of their job search:

6 months of searching

100 companies contacted; all known to hire from IIT

4 initial interviews

Zero offers

Companies that once maintained robust university hiring programs have dramatically scaled them back, which is concerning because this could create an experience gap that impacts the industry for years to come, and could manifest in a “missing generation” of engineers. This could create an industry-wide shortage of early and mid-career talent; potentially stalling innovation, as fewer fresh perspectives enter the field and challenge established practices.

Mid-career engineers: more interview loops to an offer. By mid-career engineers we refer to professionals with around 3-4 years of experience at respected companies. Candidates with this background are generally securing interviews, but the number of interview loops they go through to get an offer has increased substantially.

For example, a high achieving, mid-level engineer in the Bay Area with 4 years of experience at Amazon went through eleven full interview loops at different scaleups and tech companies before receiving their first – and only – offer!

Senior and staff engineers with high-demand specializations: premium comp and multiple offers. Companies are willing to pay significantly above market rate for proven expertise in AI, infrastructure, and security. Such candidates often have the luxury of choosing between competing offers and negotiating aggressively.

One of Evan’s recent mentees is a principal SDE, based in the Bay Area, working in one of Microsoft's AI infrastructure groups. This Principal SDE received competing offers from NVIDIA, Snowflake, Meta, and other places – all within a single month!

Engineering managers face a tough market. Widespread organizational restructuring swept through tech in 2022-2023, eliminating entire management layers, and companies have been slow to restore these positions since. As a result, many qualified engineering leaders now compete for a significantly reduced pool of opportunities. The heightened competition has transformed hiring standards.

Technical abilities once overlooked for managers are now meticulously evaluated, and system design skills are also becoming non-negotiable. In the past, managers were often hired primarily on leadership capabilities, but today, they need to prove leadership, as well as being hands-on with technology, software engineering, and software design.

The priorities in leadership roles have also shifted dramatically. Many tech companies previously focused on big-organization skills to build alignment across large teams. Today, those companies seek senior leaders who can remain focused on execution and support the higher executive layer; they’re usually not looking for senior leaders who want to remain at the high-level, interested in only steering the ship.

This transition to engineering leaders being expected to be hands-on feeds into longstanding debates about the distinction between engineering managers and technical leads. After all, today’s engineering leaders look awfully similar to yesterday’s tech leads! This change is reshaping what companies expect from their engineering leadership.

3. Interview process changes

Tech interviews are changing, and below are the biggest shifts from a few years ago that we’ve observed.

DSA interviews: elevated technical bar

On one hand, the fundamental structure of technical interviews hasn't radically changed. On the other, expectations have become significantly more demanding. Companies are simply setting a higher standard for what constitutes a passing performance.

In data structure and algorithm (DSA) interviews, engineers face noticeably harder problems at every stage of the process. One senior engineer interviewed at Google in 2021, and did so again last year. They told us:

"I used to think that LeetCode ‘hard’ problems were never asked at Google. Now [in 2024] they seem to have become the norm."

Beyond pure difficulty, we're seeing more emphasis on the completeness of implementation. Interviewers now routinely expect things like:

proper error handling

robust input validation

clean code

…all within the same time constraints as before.

There is little incentive to pass someone who doesn't get everything entirely correct. This is the grim reality of what happens when there are so many qualified candidates in the interview pool.

System design interviews: higher expectations

System design interviews have undergone an equally dramatic elevation. Senior-level candidates we talk to report being expected to demonstrate familiarity with modern distributed systems concepts that previously might have only been expected at staff levels.

Specialized knowledge has even crept into standard interviews. For example, geospatial indexing was once considered niche, but now has become commonplace in popular system design questions like "find nearby friends," Yelp-like applications, or ride-sharing platforms like Uber. We now advise candidates of all levels to have at least a basic familiarity with concepts like geohashing and spatial data structures (like quadtrees or R-trees) – as silly as that sounds. The same trends apply as to DSA: more candidates, more competition, a higher bar for hiring.

One staff engineer candidate we worked with really stood out. He had worked at Google in Seattle for almost 15 years, and was re-entering the market for the first time since. He was taken aback by the expectations in modern interviews compared to when he joined Google. As someone who had never before worked on stream processing systems, he found it frustrating that companies he interviewed at expected him to have intimate familiarity with concepts like exactly-once semantics, windowing techniques, and watermarking algorithms. He told us:

"I’ve built and maintained critical infrastructure for over a decade, but suddenly I'm expected to have specialized knowledge in areas completely unrelated to my expertise. It’s just so frustrating."

It's easy to empathize. At the same time, it's also easy to see how the luxury of choice with candidates leads to this. This elevation in technical expectations isn't arbitrary; with reduced hiring volumes, companies can afford to be more selective, and many are specifically looking for engineers who can contribute across a broader range of problems. Engineers with deep but narrow specialisms have fewer opportunities in this environment.

Downleveling

Downleveling seems to be a new trend. With heightened hiring bars and current market conditions, we're seeing candidates routinely receiving offers a level below their current position, particularly at the senior and staff levels.

In one case, Stefan worked with a candidate at Meta who successfully completed the interview process for a senior position later, but the offer was withdrawn and they were offered a mid-level role instead. This downleveling was due to a new policy requiring candidates to have at least six years of experience for senior positions. Personally, it’s heartbreaking – and arbitrary! – to see companies strongarm talent like this. The candidate ultimately accepted the offer, not being able to secure a better one.

This trend is particularly true for staff-level engineers, with many being offered senior positions even when they meet but don't easily exceed the staff-level bar. Companies have calculated that with less competition for talent, they can implement more aggressive leveling practices, and many candidates are accepting lower level offers after months of searching.

The long-term career implications of this are significant, as it often requires 2-3 years to get back to their former level. Despite this impact on career trajectory, we're seeing acceptance rates for down-leveled offers increase significantly as candidates prioritize stability in an uncertain market. Note from Gergely: we previously covered downleveling in The seniority rollercoaster.

Team match evolution

Perhaps the most significant structural change in the interview process has been the evolution of team matching. This is a process, now popular at Meta and Google, where candidates first pass an interview but don't receive offers until they match with a team.

This team matching approach has been adopted more broadly at larger tech companies, but with a slightly ugly twist: it's increasingly functioning as an additional filter, rather than for the candidate’s benefit.

We observe that team matching introduces a new set of "interviews" with hiring managers for candidates to navigate. It’s positioned as a mutual selection process, but the reality is that it's become another hurdle candidates must clear before securing an offer.

Meta notably overhauled its hiring process in 2024, eliminating most aspects of its longstanding "bootcamp" program, in which new hires joined the company first, and then found their team during bootcamp. In its place, they've implemented a team matching system that requires candidates to secure a team match before receiving a final offer.

The outcomes have been problematic for many candidates. One staff engineer we worked with who passed all technical rounds at Meta with strong, positive feedback, waited four months in team match limbo. To make things worse, by the time the team match completed, all their competing offers had expired!

When a match finally materialized, the offer was significantly below initial expectations, with little room for negotiation in the absence of alternatives. We see that team-matching backlogs seem to have been cleared as of late at Meta, but waiting many months remains common, especially in more competitive markets like New York City.

Indeed, some companies appear to be using team matching delays strategically as a negotiation tactic. Meanwhile, team-matching processes have morphed from giving candidates options, into additional screening layers where qualified candidates often find themselves eliminated or in limbo.

Team matching has evolved into a de facto second interview, despite companies' efforts to present it otherwise. From our conversations with hiring managers, we've found they commonly interview ten candidates to fill a single position. These managers strongly advise candidates to thoroughly prepare for this phase and customize their presentations specifically for the team they want to join.

Stefan advises candidates to plan for this phase and use it to their advantage. It’s true that the team matching process is slow – but this can create an opportunity to synchronize offers by scheduling interviews without team matching in place for later. Having several offers gives crucial leverage in negotiations.

4. Interview format differences at startups and Big Tech

With the rise of AI and growing skepticism about traditional coding interviews, we're seeing a widening gap between how Big Tech and newer companies do interviews.

Traditional FAANG employers remain largely committed to their existing formats, with only minor adjustments. As one FAANG head of recruiting told us:

"The inertia of these processes is enormous. These companies have built entire recruiting machines around their current processes, with years of calibration data. They're reluctant to make dramatic changes without compelling evidence that alternatives would work better at scale."

Organizationally, changes to the interview process are often gatekept by engineering executives who would prefer to wait for a fire, than potentially create a problem at the first smell of smoke.

Several mid-sized companies have moved toward more realistic, open-ended coding challenges that better reflect actual work. Examples of places adopting more realistic interviews include Stripe, Coinbase, and OpenAI. Rather than solving LeetCode questions, candidates tackle problems like:

Designing a query engine

Implementing a key-value store

Designing an in-memory database to handle transactions

Early-stage startups have pushed even further, often replacing traditional coding exercises with take-home projects that explicitly allow the use of AI tools. Yangshun Tay, founder of GreatFrontEnd, has been a prominent voice on Linkedin advocating for this shift in hiring practices. He detailed how his team successfully implemented this approach to better evaluate candidates' real-world problem-solving abilities:

“Coming from Big Tech, I'm aware of the flaws of the typical interview process. Hence I use a somewhat different process when it comes to hiring Front End Engineers for GreatFrontEnd:

1. Zero LeetCode

2. Take-home assignment

3. The take-home assignment is a todo list (what?!)

4. Product sense is evaluated

5. Candidates who pass the take-home assignment know the upcoming interview questions beforehand and have ample time to prepare

6. Candidates get a perk for interviewing with us (...)

It's important to note that such an interview process is more time consuming than the standard LeetCode one and does not scale well with the number of applicants.”

This shift serves a dual purpose: it better reflects real work conditions, while combating the growing problem of assessment fraud. One seed-stage AI founder in the Bay Area we spoke with estimated that at least 20% of candidates were obviously cheating in their traditional coding tests. The issue isn't limited to startups; one of Evan’s good friends, an Amazon interviewer, confided that half of his last ten candidates were obviously using AI tools on secondary screens during supposedly monitored assessments. By explicitly incorporating these tools into the evaluation process, companies are adapting to workplace realities and assessments’ integrity challenges.

Innovation in technical evaluation is bubbling up from smaller, more agile organizations, with Big Tech watching on from behind. This is an interesting inversion of the historical pattern wherein for the past decade, interview practices pioneered by Google and other tech giants trickled down to smaller companies eager to emulate their success. Now, it’s the opposite! One question is when or if FAANG employers will adapt to this new reality.

The truth is that Big Tech is unlikely to make changes to the hiring process without resounding, negative post-interview signals, which could be things like a significant quantity of unregretted attrition attributable to poor interview signals.

We think it’s more likely that Big Tech makes minor adjustments, like returning to on-site interviews in the short term. They recognize their current interview processes are essentially a game, but they do effectively identify candidates willing to invest in intensive preparation. Unfortunately for candidates, their willingness to grind through arbitrary algorithmic challenges correlates just enough with on-the-job characteristics of high-performing engineers to justify maintaining the status quo.

We wonder if sticking to existing interview approaches is increasingly unsustainable in the age of large language models (LLMs). As AI tools become more capable of solving the exact algorithmic puzzles used in interviews, the signal value of those assessments will inevitably diminish. No engineer in the future will need to manually code algorithms like parenthesis balancing or binary tree traversals; instead, they'll prompt an AI to generate that code. The companies pioneering more realistic, project-based assessments are adapting to the reality of how engineering work will actually be done moving forward.

What's clear is that candidates currently face a bifurcated landscape: prepare for traditional algorithm interviews for Big Tech roles, while simultaneously developing the skills to excel in more open-ended, practical evaluations for opportunities elsewhere.

5. Preparation strategies by experience level

We’ve found that an optimal preparation strategy varies significantly by experience level, and the relative importance of different interview components change with career progress. Here are patterns we’ve observed:

For junior engineers with 0-2 years of experience, we’ve found this preparation the most effective:

80% of preparation time: focus on algorithms and coding problems

20%: preparation for the behavioral interviews

The technical bar for junior roles has risen dramatically, making mastery of fundamental algorithms and data structures essential. Successful junior candidates typically solve 150-200 coding problems across all difficulty levels before interviewing. You must be a stronger coder before anything else.

Mid-level engineers with 2-4 years of experience benefit from a more balanced approach:

50% coding

25% system design

25% preparation for behavioral interviews

At this level, companies expect strong implementation skills and emergent architectural thinking. The most successful mid-level candidates we work with develop a systematic approach to system design, focusing on building blocks they can combine and adapt, rather than memorizing specific solutions.

For senior engineers with 5-8 years’ experience, we’ve seen this setup work well:

50% preparation on system design

20% on coding

30% on behavioral interviews

The primary differentiator at this level is the ability to design robust, scalable systems while clearly articulating tradeoffs. Senior engineers are expected to handle ambiguity well, asking clarifying questions and making reasonable assumptions when information is incomplete.

The most common mistake we see from senior candidates is neglecting behavioral preparation. This is a critical error; at the senior level, companies are evaluating not just technical capability, but also leadership potential, conflict resolution skills, and cultural fit.

We've seen technically brilliant candidates fail interviews or get down-leveled unnecessarily because they couldn't effectively communicate their impact, describe how they influenced cross-functional teams, or demonstrate self-awareness about previous challenges. Behavioral preparation isn't a checkbox; it significantly impacts hiring decisions, especially at senior levels and above.

Staff+ engineers face a different challenge:

Coding: a baseline at this point; stumble here and rejection can be swift.

90% of differentiation comes from system design and behavioral/leadership assessments.

For these roles, companies look beyond implementation details to evaluate architectural vision, cross-functional leadership, and executive communication skills. Successful staff+ candidates demonstrate strategic thinking, connecting technical decisions to business outcomes in their system design discussions.

Top AI labs like OpenAI have their own distinct hiring patterns. Rather than prioritizing traditional leadership skills, they heavily filter by pedigree or headline achievements and strongly favor candidates from elite, high profile companies, AI-focused startups, prestigious universities, and those with flashy achievements which they can communicate easily. Without this, applicants face an uphill battle, regardless of their technical excellence.

Effective Practice

Let's acknowledge the reality that the tech interview process has become a specialized game that continues to deviate from day-to-day engineering work. This isn't ideal, but it's the reality. Companies have settled on standardized evaluation approaches that don't perfectly mirror actual job responsibilities, and this disconnect frustrates many engineers.

The good news is that the rules of the game are publicly known. It's essentially a “secret handshake” you need to learn to gain entry into these companies. The process might seem arbitrary, but with proper preparation, it's entirely learnable. Anyone with sufficient dedication can master these patterns and significantly improve their performance.

We recognize our bias here; as an interview preparation platform, we obviously believe in the value of structured practice. The data speaks for itself: candidates who engage in deliberate practice consistently outperform those who don't, regardless of natural ability or experience level. The patterns are clear across thousands of interview outcomes.

If investing in formal mock interviews doesn't fit your preferences or budget, that's completely understandable and there are numerous alternatives: find a friend who works at your target company, connect with peers on Reddit or Discord communities, or form study groups with other job seekers. The specific method matters less than the fundamental principle that interviewing is a skill that improves with practice, feedback, and iteration.

What doesn't work is assuming your daily engineering work has prepared you for the interview environment. The performance aspect of interviewing – thinking aloud, handling pressure, communicating clearly while solving problems – requires deliberate practice in conditions that mirror actual interviews. Without this, even brilliant engineers can struggle to demonstrate their capabilities within the artificial constraints of an interview process.

6. Silver linings

The tech hiring landscape of 2025 is a far cry from the job seekers’ gold rush of 2020-2021. The pendulum has swung hard from "please take our money" to "prove you're worth it," and engineers are feeling the squeeze. But don't throw in the towel!

Major tech companies including Amazon, Apple, Microsoft, Google, and Meta, collectively maintain nearly 40,000 open roles. Even orgs inside these companies that aren't growing headcount-wise are still hiring to backfill, even as they make layoffs.

The AI sector continues to experience exceptional growth, with companies like OpenAI, Anthropic, and numerous AI infrastructure startups, hiring aggressively. Ignore bleak predictions about the imminent replacement of engineers; the reality of hiring shows how much businesses need engineers to achieve their goals. Companies in the AI sector often offer comp packages reminiscent of the 2021 peak, particularly for engineers with relevant expertise, or those demonstrating strong learning potential in AI-adjacent domains.

The engineers crushing it right now understand that modern tech interviewing has basically become its own bizarre sport, complete with arbitrary rules and peculiar traditions. They're treating interviews like performances, not just technical evaluations, and they're putting in the practice hours to match.

As AI continues reshaping how engineering work gets done, interview processes will have to evolve too because they can't keep testing for skills which AI makes obsolete. But for now, we're stuck in this awkward transitional phase where you need to master both the old-school algorithm games, and newer, more practical assessments.

At the risk of stating the obvious, at Hello Interview we see a clear pattern: there's a strong correlation between preparation investment and interview success. Candidates who dedicate time to structured practice are still more likely to secure multiple offers, even in this more selective environment.

The game might be harder, but at least the rulebook is public. With enough deliberate practice and the right preparation strategy for your level, you can still come out on top, even in this tougher market.

Takeaways

Gergely again.

Thanks very much to Evan and Stefan for summarizing what they see in the tech market. They gathered most of these insights by interacting with devs using Hello Interview, the mock interview and interview prep service of which they’re cofounders.

Some interesting things I’m reflecting on:

A tighter job market was predicted due to the end of zero interest rates. A year ago, we analyzed what the end of 0% interest rates meant for software engineers. One top conclusion was to expect a tougher job market: more competition, less “shopping around”, and engineering managers especially having a hard time. This is reflected in what Evan and Stefan report from the recruitment front line. In some ways, since this was a predictable effect, it’s easier to understand as a cause of some of today’s hiring challenges.

AI is hot, and hot AI companies care a lot about pedigree. One thing that feels a little surprising is learning from Evan and Stefan how much companies like OpenAI, Anthropic, and other top companies seem to filter by pedigree. Perhaps this should not be a surprise given these companies are inundated with applications from everywhere: if they can afford to hire from the most glittering tech companies, why wouldn’t they?

It’s a reminder that even if you are in the AI engineering field, getting into a top workplace from a lesser-known company is close to impossible without recognised industry expertise. If your target is to one day work at such companies, you might need to prepare for a multi-step career journey, starting out at lesser-known places doing AI-related work, forging a path towards better known companies, and then, one day, perhaps getting into the “top” places.

Getting a job takes more time investment. There’s constant complaints from devs about how time-consuming it is to prepare for tech interviews. If it’s a company doing Leetcode-style algorithmic interviews, the issue is the time it takes to practise beforehand. Meanwhile, if it’s a company with a complex takehome, the complaint is the time it takes to do the exercise itself.

As a competent engineer, you probably assume that an employer should accept that you know your craft, skip all the time-consuming evaluation that just seems pointless, and lead with an offer, right?

But from the vantage point of a hiring manager, a new hire is always a risk. Few things are worse than a hire that doesn’t work out because of a skill gap, motivation, or any number of reasons. Interviews are meant to verify skillsets and motivation.

We’ve gone from a candidate’s market in 2020-2022, to an employer’s market today. In this environment, you will most likely need to invest a lot more time in preparing for job interviews and doing them. The upside is that this preparation doesn’t go to waste, as interview formats don’t all change so rapidly.

Expectations are going up, and will keep on rising. At Uber, my manager at the time told me that performance expectations at Uber only went one way for the same level: up. This was because the business was growing rapidly, and the expectation of any new hire was to raise the bar. This meant that over time, expectations of “normal” for any career level kept inflating. After a while, this felt natural, but it was a strange thing to adjust to!

I feel we’re seeing something similar play out across the broader industry, today. Due to lots of qualified, capable engineers applying for jobs, expectations are going up at all career levels, and this is why downleveling is more common.

If you get a downleveled offer: first of all, congrats for getting an offer! In this job market, it’s an achievement in itself. It’s helpful to take current job market conditions into account before being too disappointed by this outcome. And if you have yet to receive an offer, know that the market is tougher right now than it’s been in a decade, so job searches take longer than before.

If you’re currently in the job market, or are doing interviews and have noticed any changes in recruitment processes, please share in a comment:

Leave a comment

This deepdive is a guest post – if you’re interested in pitching an article for The Pragmatic Engineer, see details here.

Half Stack Data Science: Programming with AI, with Simon Willison

I participated in this wide-ranging 50 minute conversation with David Asboth and Shaun McGirr. Topics we covered included applications of LLMs to data journalism, the challenges of building an intuition for how best to use these tool given their "jagged frontier" of capabilities, how LLMs impact learning to program and how local models are starting to get genuinely useful now.

At 27:47:

If you're a new programmer, my optimistic version is that there has never been a better time to learn to program, because it shaves down the learning curve so much. When you're learning to program and you miss a semicolon and you bang your head against the computer for four hours [...] if you're unlucky you quit programming for good because it was so frustrating. [...]

I've always been a project-oriented learner; I can learn things by building something, and now the friction involved in building something has gone down so much [...] So I think especially if you're an autodidact, if you're somebody who likes teaching yourself things, these are a gift from heaven. You get a weird teaching assistant that knows loads of stuff and occasionally makes weird mistakes and believes in bizarre conspiracy theories, but you have 24 hour access to that assistant.

If you're somebody who prefers structured learning in classrooms, I think the benefits are going to take a lot longer to get to you because we don't know how to use these things in classrooms yet. [...]

If you want to strike out on your own, this is an amazing tool if you learn how to learn with it. So you've got to learn the limits of what it can do, and you've got to be disciplined enough to make sure you're not outsourcing the bits you need to learn to the machines.

Via @halfstackdatascience.com

Tags: podcasts, generative-ai, podcast-appearances, ai, llms, data-journalism

It’s time to try something new.

Starting today, you can support my writing on Patreon. I’ll never put up a paywall for my blog or newsletter; in effect, by supporting, you’re helping to continue to make it available for everyone.

Here’s what you’ll get:

At $5/month or more, your name will be listed as a “thank you” on a supporters page on my website. At $25/month or more, your name will also link to your website from the supporters page, and you’ll receive a linked “thank you” credit at the bottom of each newsletter. At $100/month or more, you’ll get a linked name and static logo at the top of every newsletter, and a linked “thank you” at the bottom of every page on my website. (I’ve limited the number of supporters at this tier.)

The site you link to must be yours, safe, legal, and not an affiliate page. I won’t allow gambling, adult sites, or anything designed to abuse the trust of the reader.

This is an experiment! If it doesn’t work out, I’ll remove the Patreon but ensure that everybody receives the acknowledgment they’ve paid for.

Here’s what I like about this model: there are no paywalls and there’s no user tracking involved, and there are no penalties for people who don’t have the means to support. It helps me with my server costs, but otherwise, everything stays the same.

But if you have concerns, I’d love to hear them. As always, you can shoot me an email at ben@werd.io.

🔗Authoritarian Grift Meets the Grind of Justice

America (that country that promoted itself (past tense) as the beacon of democracy) continues to be the ONLY country in the world that does nothing to stop these kinds of people.

Why would anybody believe - ever - that America stands for democracy?

🕵💬 Inspired By People | 🌐 Mattie Stepanek

🔍 All The Posts can be found here, or scroll through the quotes below.

🔗 Pockets of Belfast - Gapingvoid

On 20th March, in celebration of the UN’s World Happiness day, the BBC’s Mark Easton conducted a little social experiment: dropping ten wallets (with money and his phone number inside) on the streets of Belfast, Northern Ireland.
He got them all back.

Makes me happy - and today isn’t even World Happiness Day!

フランス競争当局（FCA）は3月31日、アップルに対して1億5000万ユーロ（約242億円）という高額な罰金を科しました。問題となったのは、2021年から2023年にかけてアップルが導入した「App Tracking Transparency（ATT）」と呼ばれるプライバシー保護ツールです。

このツールはiOSユーザーのプライバシーを保護するという名目で導入されましたが、フランス当局の調査によると、その実装方法には競争法上の問題があったとされています。

ATTの仕組みと問題点

ATTは基本的に、アプリがユーザーのデータを収集して広告目的で使用する前に、明示的な同意を求めるというものです。一見すると消費者保護のための素晴らしい機能に思えますが、問題はその実装方法にありました。

具体的には：

サードパーティアプリには「過度に複雑な」ポップアップ表示が要求された ユーザーは煩わしいポップアップに嫌気がさし、同意率が低下 一方、アップル自身のアプリではシンプルなチェックボックス一つで済むように設計 アップルは同意を得たユーザーデータを自社の広告サービスに活用し収益化

この設計により、他のアプリ開発者が広告収入を得ることが難しくなった一方で、アップル自身は個人データを収集・活用して広告ビジネスで利益を上げていたと当局は指摘しています。

市場への影響と当局の判断

フランス競争当局によれば、この仕組みは特に広告収入に依存する小規模なアプリパブリッシャーに深刻な経済的打撃を与えました。興味深いことに、当局はATTというプライバシー保護ツール自体に問題があるのではなく、その実装方法が「必要でも比例的でもなかった」と結論づけています。

当局の声明では「異なるポップアップウィンドウ間の相互作用を規定するルールがフレームワークの中立性を損ない、アプリケーションパブリッシャーとサービスプロバイダーに明確な経済的損害を与えた」と指摘されています。

罰金の意味と今後の展開

今回の242億円という罰金額はアップルの四半期収益（2024年第4四半期で約19兆円）から見れば微々たるものですが、大手テクノロジー企業によるプライバシー保護を名目とした市場支配への警鐘として重要な意味を持ちます。

注目すべきは、フランス当局がATTの変更を義務付けなかったことです。アップル側も「本日の決定に失望していますが、フランス競争当局はATTに特定の変更を要求していません」とコメントしており、機能自体は維持される見通しです。

このケースは、デジタル市場におけるプライバシー保護と公正競争のバランスという難しい課題を浮き彫りにしています。ユーザーのプライバシー保護は重要ですが、それが特定企業の市場支配強化のツールとなってはならないという規制当局の姿勢が示されたといえるでしょう。

（参考文献） Ciblage publicitaire : l’Autorité de la concurrence prononce une sanction de 150 000 000 euros à l’encontre d’Apple en raison de la mise en œuvre du dispositif App Tracking Transparency (« ATT ») Targeted advertising: the Autorité de la concurrence imposes a fine of €150,000,000 on Apple for the implementation of the App Tracking Transparency (“ATT”) framework French regulator fines Apple $162 million for anticompetitive use of privacy tool Sanction: Apple condamné à 150 millions d’euros d’amende en France pour abus de position dominante CNIL: Applications mobiles : l’Autorité de la concurrence sanctionne Apple pour avoir abusé de sa position dominante

🔗 The Mediocrity of Modern Google

But here’s the thing about tech – and I’ve been saying this for years – technology’s self-obsolescence is a more effective regulator than any government agency. Google, despite its trillion-dollar market cap and infrastructure advantages, risks becoming the first of the mega-caps to fade into irrelevance as the internet evolves beyond the “10 blue links” paradigm that made it a giant.

Me - never been overly into the land of Google. Yes - I have an account - and of course I have Google Drive - but it is not my ‘go to’ - unless I am forced into it because a business I am working with has decided it hates Microsoft and has enabled Google. I do think it is hard to beat their ‘office suite’ - for the collaboration part of it - BUT I would never consider writing a doc or knocking out a spreadsheet on Google if I wasn’t expecting somebody to be collaborating with me.

In my world …

GoogleSheets Airtable

GoogleDocs Craft

GMail AppleMail/Spark mainly running my own domains in some data center

Google Search Perplexity

GDrive iCloud/Dropbox

GooglePhotos et al. Apple

What else do people use Google for - I forget.

“Obvious” closes the door to inquiry.
“Perhaps” opens it.

💬 Seth Godin

4月1日、デジタル庁から本人確認ガイドラインの改定に向けた有識者会議（令和6年度）の取りまとめが公表されました。「DS-500 行政手続におけるオンラインによる本人確認の手法に関するガイドライン」（通称「本人確認ガイドライン」）は、行政手続きをデジタル化する時に、安全に本人確認するためのルールや方法をまとめたものです。例えるなら、オンラインでの「本人確認の教科書」みたいなものですね。このガイドラインは、アメリカのNISTという機関が作ったガイドラインなどを参考にしつつ、マイナンバーカードを使った本人確認など、日本独自のやり方も取り入れています。

最近は、行政手続きのオンライン化が進んだり、マイナンバーカードを使う人が増えたり、ネットを使った詐欺も増えたりと、本人確認を取り巻く環境が大きく変わってきています。

アメリカでは、NISTのガイドラインの改定案が出ていますし、ヨーロッパでは、スマホで使えるデジタルIDの仕組み「デジタル・アイデンティティ・ウォレット」も導入されようとしています。

そこで、デジタル庁では、専門家を集めて「本人確認ガイドラインの改定に向けた有識者会議」を開き、今の課題や海外の動きを踏まえ、ガイドラインをどう変えていくか話し２年間にわたって話し合ってきました。最終回が令和６年度第５回の会合でした。

公表されたのは、「本人確認ガイドライン改定方針 令和6年度とりまとめ（案）」と、本人確認ガイドライン改定案（令和6年度とりまとめ時点案）および、これらを受けての有識者コメントをまとめた議事録 （これら指摘事項も反映されていくはず）です。

なお、文末に Youtube 版解説も付けましたので、そちらも合わせて御覧ください。

資料1：本人確認ガイドライン改定方針 令和6年度とりまとめ（案）

まず、

資料1：本人確認ガイドライン改定方針 令和6年度とりまとめ（案）（PDF／1,130KB）

ですが、これは、令和6年度のこの有識者会議の結果をまとめたもので、ガイドラインをどう変えていくかのアイデアが書かれています。ここでは、以下のようなことが書かれています。

改定の背景

行政手続のオンライン化やマイナンバーカードの普及 フィッシング攻撃の高度化や本人確認書類の偽造事件の増加 米国NISTのガイドライン改定や欧州でのデジタルIDウォレット導入の動き

改定の主なポイント

ガイドラインの適用対象と名称の見直し: 対面での本人確認や行政手続以外の行政サービスも対象とする。 本人確認手法検討における「基本的な考え方」の定義: 事業目的の遂行、公平性、プライバシー、ユーザビリティ・アクセシビリティ、セキュリティの5つの観点を重視する。 本人確認の基本的な枠組みの定義: 身元確認、当人認証、フェデレーションの概念を明確化し、連携モデルと非連携モデルのシステム実装モデルを定義、連携モデルをその基本とする。 脅威と対策の最新化、保証レベルの見直し: 最新の脅威動向や技術動向を踏まえ、身元確認と当人認証における脅威と対策、保証レベルの位置づけと対策基準を見直す。 リスク評価プロセスの全面的な見直し: 本人確認手法の評価プロセスを5つの観点から行い、リスク評価プロセスを単純化する。

その他

ガイドライン本編とは別に、具体的な技術や手法、事例などをまとめた「本人確認ガイドライン解説書」を新たに整備する。 資料2：本人確認ガイドライン改定案（令和6年度とりまとめ時点案）

一方、

資料2：本人確認ガイドライン改定案（令和6年度とりまとめ時点案）（PDF／1,838KB）

ですが、実際のガイドライン改定案が示されています。

主なポイントとしては以下のようなことが書かれています。

リスクに応じた「適切な保証レベル」の選択: 従来は画一的に高い保証レベルが求められがちで、利便性を損なうケースがあった。 本改定では、手続のリスクに応じた適切な保証レベル (本人確認の確からしさの段階) を選択することを基本とする。 過剰な厳格さを避け、安全・安心で利便性の高い行政サービスの実現を目指す。 検討の5つの観点: 保証レベルや本人確認手法の選択にあたり、以下の5つの観点を考慮する。 事業目的の遂行 (本人確認が手続きの障壁にならないか) 公平性 (特定の人だけが利用できない等の不公平がないか) プライバシー (個人情報の取扱いは適切か) ユーザビリティ及びアクセシビリティ (利用者にとって使いやすいか) セキュリティ (リスクに対して適切な強度か) 本人確認の構成要素: 本人確認を以下の3つの要素で定義・整理する。 身元確認 (Identity Proofing): 申請者が実在し、生存する人物であることを確認する (属性収集、書類検証、申請者検証等)。 当人認証 (Authentication): 手続を利用しようとする者が、身元確認時に登録された者と同一人物であることを確認する (知識、所有物、生体情報に基づく認証)。 フェデレーション (Federation): 信頼できる他のIDプロバイダが行った身元確認や当人認証の結果に依拠する。 実装モデル: 連携モデル (Federated): 共通のIDプロバイダを利用するモデル (効率化のため第一候補)。 非連携モデル (Non-Federated): 各システムが独自に本人確認機能を構築するモデル。 両者の組み合わせも可能。 脅威と対策: 身元確認、当人認証、フェデレーションそれぞれにおける脅威 (なりすまし、書類偽造、フィッシング、重複登録等) を具体的に示し、それらに対する対策プロセスや手法例、保証レベルごとの対策基準を定義している。 特に身元確認では、真正性確認手法 (デジタル署名検証、信頼できる情報源への照会、物理的検査等) や申請者検証手法 (容貌確認、暗証番号、確認コード送付等) を整理。 当人認証では、多要素認証の重要性やフィッシング耐性のある認証方式 (公開鍵認証等) に言及。 本人確認手法の検討プロセス: ①リスク特定 → ②リスク影響度評価 (高・中・低) → ③保証レベル判定 (レベル1～3) → ④手法の評価 (5つの観点) → ⑤補完的対策や例外措置の検討 → ⑥継続的な評価・改善、というプロセスを提示。 法人等の身元確認 (別紙2): 個人とは異なる考え方が必要。 ①法人等の実在性確認 (法人番号、名称、所在地等)、②申請者個人の実在性確認、③法人等と申請者個人の紐づき確認 (代表者印、委任状等) の3ステップで整理。 今後の予定

ガイドライン改定案の公開、意見募集、各省協議等を経て、改定版を発行することになるはずです。わりと良い文書になっているとおもうので、今から楽しみです。英語版もぜひ作っていただきたいところです。

詳細

以下、もう少し詳細に説明したものになります。ご参考まで。

主要なテーマ ガイドラインの適用範囲と名称の拡大・変更: 現行の「オンラインによる本人確認」から、対面の手続きや行政手続き以外の行政サービスも含むように適用範囲を拡大する。 ガイドライン名称を「DS-511 行政手続等での本人確認におけるデジタルアイデンティティの取扱いに関するガイドライン」に変更する。 文書番号もDS-500からDS-511へ変更する。 （資料1 より）「デジタル技術を活用した本人確認の機会が対面や行政手続以外にも拡大していることを踏まえ、本ガイドラインの 適用対象を拡大する方針とする。」 （資料2 P.2 より）「本ガイドラインは、国の行政機関が提供する行政手続又は行政サービス（以下「対象手続」という。）において、個人又は法人等が申請・届出・アカウント登録・ログイン等を行う際の本人確認を対象とする。」 検討にあたる「基本的な考え方」の定義: 手続き等の特性に応じた適切な手法選択のため、「事業目的の遂行」「公平性」「プライバシー」「ユーザビリティ及びアクセシビリティ」「セキュリティ」の5つの観点を定義する。 （資料2 P.i より）「今回の改定では、対象手続のリスクに応じた「適切な保証レベル」を選択できるようにすることを念頭におき、そのための基本的な考え方として「事業目的の遂行」、「公平性」、「プライバシー」、「ユーザビリティ及びアクセシビリティ」及び「セキュリティ」の 5つの観点を定義した。」 （資料1 より）「単にセキュリティレベルの高い手法を選べばよい訳ではない。事業目的の遂行、公平性、プラ イバシー、ユーザビリティ及びアクセシビリティへの影響も考慮しながら、リスクに応じたレ ベルの本人確認手法を選択することが必要である。」 本人確認の基本的な枠組みの定義: 本人確認を「身元確認（Identity Proofing）」、「当人認証（Authentication）」、「フェデレーション（Federation）」の3つの要素で構成すると定義する。 実装モデルとして「連携モデル（Federated Model）」と「非連携モデル（Non-Federated Model）」を定義する。 （資料2 P.9 より）「本ガイドラインでは、本人確認を構成する要素として「身元確認」と「当人認証」を定義する。さらに、身元確認や当人認証を他者（信頼できる IDプロバイダ）に依拠して実現する要素として「フェデレーション」を定義する。」 脅威と対策の最新化、保証レベルの見直し: 国内外の脅威動向、最新技術動向、NIST SP 800-63-4の改定内容を踏まえ、各要素における想定脅威と手法例を最新化する。 身元確認保証レベルと当人認証保証レベルの位置づけと対策基準を、脅威への耐性の観点から見直す。 身元確認のプロセスを「属性情報の収集」「本人確認書類の検証」「申請者の検証」「登録」に定義し、各プロセスにおける脅威を明確化する。 身元確認保証レベルは、ICチップ等によるデジタルな検証の有無を重要な差異とし、低リスク手続き向けの「レベル1」を再定義する（簡易的な身元確認）。 当人認証のプロセスを「認証器の登録」「当人認証の実施」「盗難・紛失時の対応」「アカウント回復」と定義し、ライフサイクル全体での対策を考慮する。 当人認証保証レベルの対策基準を、フィッシング耐性などの最新脅威への対応を強化する方向で見直す。レベル3では全ての利用者にフィッシング耐性のある認証方式を必須とする。 フェデレーションにおいては保証レベルを定めず、一律の対策基準を定義する。NIST SP 800-63-4 FAL2の要件を参考に、信頼関係の確立、設定・登録・鍵管理、アサーションに関する対策、定期的な確認と見直しに関する基準を定める。 （資料2 P.3 より）「３ 本人確認における脅威と対策」 （議事録より）「NIST SP800-63-4の動向にかかわらず、日本では ICチップを利用した厳格な身元確認が比較的利用しやすい環境が作られてきました。しかし、注意喚起だけではフィッシング詐欺を防ぐことができないことなどから、身元確認保証レベル 3 がより重要になってきていると思います。」 （議事録より）「当人認証保証レベルの表には、フィッシング耐性（推奨）といった表記があります。身元確認において容貌の確認をすることについても、同様に推奨するというような記載を加えるのはいかがでしょうか。対面時において容貌確認のない暗証番号による検証は強固な検証であるといった誤解を解くようなきっかけになるのではないかと思います。」 リスク評価プロセスの全面的な見直し: 保証レベル判定までのプロセスを簡略化しつつ、事業目的、公平性、プライバシー等への影響を考慮した評価プロセスを導入する。 リスク評価の初期段階として「リスクの特定」プロセスを新設する。 影響度の評価基準を、利用者の権利権益の侵害を軸としつつ、プライバシーへの深刻な影響や犯罪・攻撃への悪用が想定される場合は「高位」とする。 本人確認手法の評価プロセスを新設し、「基本的な考え方」で定義した5つの観点から評価する。 評価結果に基づき、複数の手法の併用、追加対策、より高い/低い保証レベルの手法の採用などの補完的対策を検討するプロセスを導入する。 継続的な評価と改善のためのプロセスを具体化し、利用者からの問い合わせ、セキュリティイベント、脅威動向などを収集・分析し、必要に応じて改善措置を講じる。 （資料1 より）「4章のリスク評価プロセスは、保証レベル判定までのプロセスを簡略化しつつ、事業目的の遂行、公平性、プライ バシー等への影響を考慮したテーラリングの考え方を取り入れる形で全面的に見直し。」 （資料2 P.40 より）「４ 本人確認手法の検討方法」 ガイドライン解説書の新規整備: Normativeな本編に対し、Informativeな「本人確認ガイドライン解説書」を新たに整備する。 解説書には、具体的な技術、手法、事例、検討用ワークシートなど、変化の速い情報を集約し、本編の簡潔化と柔軟な改定への対応を図る。 （資料1より）「今回の改定にあわせ、本編とは別に「本人確認ガイドライン解説書」を整備する方針とする。」 （資料1より）「Normative である本編に対し、「解説書」はInformativeとする。変化のサイクルの速い情報（具体的な技術、手 法、事例等）を「解説書」にとりまとめることで、今後の動向変化にも柔軟に対応できる構成とする。」 法人等の手続きにおける身元確認の考え方: 個人に対する身元確認とは異なる考え方や手法が必要となるため、別紙として法人等の手続きにおける身元確認のプロセスと手法例を示す。 法人等の実在性確認、申請者個人の実在性確認、法人等と申請者個人の紐づきの確認の3つの段階で構成される。 （資料2 P.48 より）「別紙２ 法人等の手続における身元確認の考え方について」 議事録での指摘事項 マイナンバーカードのスマートフォン搭載など、技術の進展を考慮したガイドラインの記述が求められている。（議事録 より） ガイドラインが法令や施行規則と相互に影響することを考慮し、FAQなどで関係性を明確化する必要がある。（議事録より） フィッシング詐欺対策として、より厳格な身元確認手続き（容貌の確認など）の重要性が高まっている。（議事録より） 身元確認保証レベルと脅威耐性のマッピングをとりまとめ資料にも反映することが望ましい。（議事録 より） 当人認証におけるフィッシング耐性の推奨と同様に、身元確認における容貌の確認も推奨する旨を記載することが、誤解を解く上で有効ではないか。（議事録より） 暗証番号の桁数表示や有効期限の記載など、利用者に誤解を与えないような図表の修正が必要である。（議事録より） フェデレーションにおけるIDプロバイダからの情報取得について、より明確な記述が求められている。（議事録より） デジタル認証アプリの民間事業者による利用も考慮した、わかりやすい記載が望ましい。（議事録より） IDプロバイダ側で身元確認を行う場合のプロセス明文化と監査の必要性が指摘されている。（議事録より） 耐タンパ性に関する部分は電子署名全体ではなく、鍵に関する記述であることが明確になるよう、表現を修正する必要がある。（議事録より） 本人確認、保証レベルなどの用語定義について、より正確で理解しやすい表現への見直しが提案されている。（議事録より） 「本人確認」の構成要素に関する記述の整合性を取る必要がある。（議事録 より） 図表内での用語の統一（例: 暗証番号 vs PIN）が必要である。（議事録より）

Pydantic Evals

Brand new package from David Montague and the Pydantic AI team which directly tackles what I consider to be the single hardest problem in AI engineering: building evals to determine if your LLM-based system is working correctly and getting better over time.

The feature is described as "in beta" and comes with this very realistic warning:

Unlike unit tests, evals are an emerging art/science; anyone who claims to know for sure exactly how your evals should be defined can safely be ignored.

This code example from their documentation illustrates the relationship between the two key nouns - Cases and Datasets:

from pydantic_evals import Case, Dataset case1 = Case( name="simple_case", inputs="What is the capital of France?", expected_output="Paris", metadata={"difficulty": "easy"}, ) dataset = Dataset(cases=[case1])

The library also supports custom evaluators, including LLM-as-a-judge:

Case( name="vegetarian_recipe", inputs=CustomerOrder( dish_name="Spaghetti Bolognese", dietary_restriction="vegetarian" ), expected_output=None, metadata={"focus": "vegetarian"}, evaluators=( LLMJudge( rubric="Recipe should not contain meat or animal products", ), ), )

Cases and datasets can also be serialized to YAML.

My first impressions are that this looks like a solid implementation of a sensible design. I'm looking forward to trying it out against a real project.

Via @samuel_colvin

Tags: evals, python, pydantic, generative-ai, ai, llms

We’re planning to release a very capable open language model in the coming months, our first since GPT-2. [...]

As models improve, there is more and more demand to run them everywhere. Through conversations with startups and developers, it became clear how important it was to be able to support a spectrum of needs, such as custom fine-tuning for specialized tasks, more tunable latency, running on-prem, or deployments requiring full data control.

— Brad Lightcap, COO, OpenAI

Tags: openai, llms, ai, generative-ai

[Nell Dhanesha at Nieman Journalism Lab]

This was fun to watch unfold in real time:

"The truck’s journey to that spot had begun a few days earlier, as an email with the subject line “guerilla marketing for sources” in the inbox of Ariana Tobin, editor of ProPublica’s crowdsourcing and engagement team. It came from reporter Brett Murphy, who was covering the destruction of USAID with his reporting partner Anna Maria Barry-Jester. They’d been tipped off about the desk cleanouts; was there any chance, they asked Tobin, that they could send a billboard truck out on the morning of the 27th?"

And this quote quietly implies what a significant chunk of my job has become more recently:

"“We’re basically treating any conversation we’re having with someone who works for or used to work for the federal government as a maximum-security tip,” Tobin said. “That, frankly, is not what we used to do.”"

Perhaps we'll write more about that in the future. For now, speaking of tips, if you want to send ProPublica a tip, we now have a number of options.

#Media

[Link]

debug-gym

New paper and code from Microsoft Research that experiments with giving LLMs access to the Python debugger. They found that the best models could indeed improve their results by running pdb as a tool.

They saw the best results overall from Claude 3.7 Sonnet against SWE-bench Lite, where it scored 37.2% in rewrite mode without a debugger, 48.4% with their debugger tool and 52.1% with debug(5) - a mechanism where the pdb tool is made available only after the 5th rewrite attempt.

Their code is available on GitHub. I found this implementation of the pdb tool, and tracked down the main system and user prompt in agents/debug_agent.py:

System prompt:

Your goal is to debug a Python program to make sure it can pass a set of test functions. You have access to the pdb debugger tools, you can use them to investigate the code, set breakpoints, and print necessary values to identify the bugs. Once you have gained enough information, propose a rewriting patch to fix the bugs. Avoid rewriting the entire code, focus on the bugs only.

User prompt (which they call an "action prompt"):

Based on the instruction, the current code, the last execution output, and the history information, continue your debugging process using pdb commands or to propose a patch using rewrite command. Output a single command, nothing else. Do not repeat your previous commands unless they can provide more information. You must be concise and avoid overthinking.

Via Import AI

Tags: prompt-engineering, llms, python, generative-ai, llm-tool-use, ai, microsoft, claude

🔗 Were Your Managers Managing Your Decline?

This is “managed decline,” – the art of overseeing failure with exceptional documentation. Managed decline hides under terms like cost control, risk minimization, and streamlining, but it guarantees one thing: job security for managers. The EU and UK exemplify this, where managed economic entropy and decline have become a boon for the managerial class. Managers multiply like Gremlins in water.

🔗 Division is easier than connection - Seth Godin - just reminding us all of good behaviour.

[Anthony Ha at TechCrunch]

China Miéville, who is one of the great contemporary science fiction and fantasy authors, is right on the money about Silicon Valley's tendency to create the Torment Nexus.

This is exactly why I think science fiction is so important, and why it has a lot to contribute:

"To Miéville, it’s a mistake to read science fiction as if it’s really about the future: “It’s always about now. It’s always a reflection. It’s a kind of fever dream, and it’s always about its own sociological context.”

He added that there’s a “societal and personal derangement” at work when the rich and powerful “are more interested in settling Mars than sorting out the world” — but ultimately, it’s not science fiction that’s responsible."

To me, the point and excitement of science fiction is to talk about today through the lens of analogy and extrapolation: not necessarily to warn or celebrate, but to explore. Of course, there's a broad spectrum of stories under that umbrella, and not all of them fit that mold as well as others, but that's what drives me as a reader and a writer. It's up to the reader to decide what to take from that.

I think a lot about Starship Troopers. (Really! I do. Some people think about the Roman Empire. I think about Heinlein.) The original book was, at the very least, fascist-adjacent. The movie adaptation was, at least for me, and in intent by its director, a very funny satire at the expense of those ideas. But, of course, some people took away the top-line plot and either decried the fascism or, more worryingly, freaking loved it. See also: Fight Club, which a certain kind of incel adjacent maladjusted man-child has taken on as something to model, rather than a satirical novel that pokes at them and the country they inhabit.

The reader makes their own interpretation. And if that happens to be a sociopathic world-view bent on world domination, that's on them.

#Culture

[Link]

[Brian Krebs]

Sobering roundup from Brian Krebs about how each of the five pillars of the First Amendment - speech, religion, the media, the right to assembly, and the right to petition the government and seek redress for wrongs - has been attacked during the first few months of the Trump Administration.

It's a laundry list - and we're only a few months in.

"Where is President Trump going with all these blatant attacks on the First Amendment? The president has made no secret of his affection for autocratic leaders and “strongmen” around the world, and he is particularly enamored with Hungary’s far-right Prime Minister Viktor Orbán, who has visited Trump’s Mar-a-Lago resort twice in the past year."

The piece concludes with a warning that Trump is following a similar playbook to Orbán by consolidating control over the courts and decimating the free press. It played out there; we will see what happens here.

#Democracy

[Link]

It’s a Harlan Coban. It should have been great. It wasn’t. I am increasingly finding that his stuff falls into 2 star or 4 star. This is the former. I didn’t care about the people. The mystery wasn’t really that much of a mystery.

Yeah … Nah.

Just One Look on 🔗 Reelgood

‘All’ My TV Show Reviews

My advice about using AI is simple: use AI as an assistant, not an expert, and use it judiciously. Some people will object, “but AI can be wrong!” Yes, and so can the internet in general, but no one now recommends avoiding online resources because they can be wrong. They recommend taking it all with a grain of salt and being careful. That’s what you should do with AI help as well.

— Ned Batchelder, Horseless intelligence

Tags: ai, ned-batchelder

[Lubna Kably in The Times of India]

Worrying stuff being reported by the Times of India (and, at the time of writing, under-reported in the domestic US press). AI is being used to flag international students because of their social media activity, among other signals, who are now being sent emails asking them to "self-deport":

"Hundreds of international students in the US are getting an email from the US Department of State (DOS) asking them to self-deport owing to campus activism. Immigration attorneys’ contacted by TOI affirmed this development and added a few Indian students may also be at the receiving end of such emails – for something as innocuous as sharing a social media post.

It is not just international students who physically participated in campus activism but also those who shared or liked ‘anti-national’ posts that are the target of these emails, said an immigration attorney."

Axios previously reported on how this was going to be done:

"Secretary of State Marco Rubio is launching an AI-fueled "Catch and Revoke" effort to cancel the visas of foreign nationals who appear to support Hamas or other designated terror groups"

From that article:

""This should concern all Americans. This is a First Amendment and freedom of speech issue and the administration will overplay its hand," said Abed Ayoub, head of the American-Arab Anti-Discrimination Committee."

It's a clear First Amendment issue. Whether they've overplayed their hand unfortunately remains to be seen.

#Democracy

[Link]

こんにちは、富士榮です。
いわゆる法人共通認証基盤と呼ばれる、デジタル庁が提供しているGビズIDの大幅アップデートが公開されましたね。 出典）デジタル庁 - GビズID https://gbiz-id.go.jp/top/

GビズIDについてはこれまでもOpenIDファウンデーションジャパンのイベント等に古くは経産省、デジタル庁へ移管されてからはデジタル庁の方々にお越しいただき技術仕様やトラストフレームワークについてご発表いただいてきました。

OpenID Summit Tokyo 2020 - 2020/1/24 https://www.openid.or.jp/summit/2020/
OpenID BizDay #14 - 2021/1/27 https://openid.connpass.com/event/200742/presentation/
OpenID BizDay #15 - 2023/1/10 https://openid.connpass.com/event/266730/
OpenID BizDay #17 - 2025/2/19 https://openid.connpass.com/event/341152/

GビズIDについて 簡単に言うと、GビズIDは企業の代表や従業員などが当該の企業に所属していることを表し、例えば補助金の申請などの行政手続きをオンラインで実施することを可能にするためのID基盤ですね。 そのためには当然、当該の企業が実在していること、そしてGビズIDを利用する代表者や従業員が当該企業と適切に関係しており所属していることを保証していくことが重要です。
ここは非常に重要な一方でまだまだ課題も多く、例えば現状は法人の実在性について法務局の発行する印鑑証明書や個人事業主の場合は市町村の発行する印鑑登録証明書を使うことで確認することになりますが、アカウントライフサイクルは各利用企業側に任せるしかないという状況があったりします。

法人共通認証基盤の必要性 この考え方は何も日本だけで必要とされているわけではなく、海外においても同様の要求はあるわけです。OpenID FoundationのeKYC and Identity Assurance Working Groupでは個人の本人確認がどのようにIdentity Providerで実施されたかという情報をRelying Partyへ伝達するためのOpenID Connect for Identity Assurance（最近正式化されましたね！）に加えて、個人が法人とどのような関係性にあるのかを表現するためのAuthority Claims Extensionの開発を進めています。この辺りは日本のOpenIDファウンデーションジャパンのKYC WGの参加メンバーの方々とも協力して国際標準への道筋をうまく作っていきたいところです。
参考）eKYC and Identity Assurance Working Group https://openid.net/wg/ekyc-ida/

GビズIDのアップデート概要 こう言うのは更新履歴を見ていくのが重要ですね。 デジタル庁が公開しているシステム連携ガイドを見ると技術仕様を含め確認ができるので、こちらの更新履歴を見てみましょう。なお、現在「行政サービス向け」のシステム連携ガイドが公開されていますが、そもそも現状のGビズIDは民間サービスとの連携を許可していません。それにもかかわらず行政サービス向け、と明記されているのは今後の民間サービスへの展開を見据えてのことなのかな、、と期待が膨らみますね。
早速更新履歴を見ていきましょう。すでにバージョン2.3なんですね。

結構更新が多いです。さすが大型アップデートです。
個人的に関心が高かったのは、以下の２点です。 アカウント種別に管理者（GビズIDメンバー（管理者））が増えた GビズIDトラストフレームワークが策定され、IAL/AALが明記された アカウント種別はこれまでプライム、メンバー、エントリーの３種類で、原則プライムは法人代表者のためのアカウントでした。そして、メンバーアカウントの作成や管理はプライムの権限者が実施するしかなかったわけですが、いちいち代表者がログインしてアカウント管理をするのか！！という課題も大きかったのだと思います。GビズIDメンバー（管理者）というアカウント管理権限を持ったアカウントを作成することができるようになりました。 ちなみにGビズIDプライムのアカウントはマイナンバーカードを使ったオンライン申請もできるようになってますね。

トラストフレームワークについても別文書で定義されています。

法人共通認証基盤におけるトラストフレームワーク https://gbiz-id.go.jp/top/trust_framework/trust_framework.pdf

システム連携ガイドにもざっくりとしたレベル感は記載されていますので、Relying Partyは扱う情報の機密レベルやリスク度合いに応じてどのアカウント種別を要求するか決めていく必要があります。

OpenID Connect for Identity Assuranceへの対応 タイトルにも書いた通り、今回のGビズIDのアップデートの目玉はOpenID Connect for Identity Assurance（OIDC4IDA）への対応です。といっても結論フルスペック対応ではなく、スキーマについてある程度対応した、という程度ではありますが国が提供するサービスに新しい技術仕様が採用されていくのは非常に嬉しいことですね。
具体的にはscopeにjp_gbizid_v1_idaを指定することでOIDC4IDAに対応した属性情報を取得できるようになるようです。
実際に返却される属性（verified_claims）は下記の通りです。 要するにGビズIDのトラストフレームワークに従い、どのような審査・確認が行われたアカウントなのか、という情報がRelying Partyに対して送出されるようになるわけです。
よく見るとauthorityに関する属性も返していますね。この辺りは現在eKYC and Identity Assurance Working Groupで開発を進めているAuthority Claims Extensionを先取りした感じです。
サンプルレスポンスも書いてあります。
組織情報の詳細についても返却できるようになっていますね。
こんな感じで当該組織でそのアカウントがどのような役割を持っているのかが表現できるようになっています。


これはちゃんとこのドキュメントを英訳してグローバルで発信していかないといけませんね。結構先進的なことをやっているので海外の実装者や政府機関にとっても非常に参考になると思います。＞デジタル庁さん、がんばってください！

[Allison Morrow at CNN]

It's refreshing to see this kind of AI skepticism in a major outlet:

"Apple, like every other big player in tech, is scrambling to find ways to inject AI into its products. Why? Well, it’s the future! What problems is it solving? Well, so far that’s not clear! Are customers demanding it? LOL, no. In fact, last year the backlash against one of Apple’s early ads for its AI was so hostile the company had to pull the commercial.

[...] Large language models are fascinating science. They are an academic wonder with huge potential and some early commercial successes, such as OpenAI’s ChatGPT and Anthropic’s Claude. But a bot that’s 80% accurate [...] isn’t a very useful consumer product."

I'm more than ready for the hype cycle to come to its conclusion. Are there some interesting use cases for AI? Sure. Should the ethical, functional, environmental, and contextual issues with current AI vendors give us all pause at the very least? Absolutely.

#AI

[Link]

[Makena Kelly at WIRED]

If you know anything about building software, you know that this is an absurd idea:

"The so-called Department of Government Efficiency (DOGE) is starting to put together a team to migrate the Social Security Administration’s (SSA) computer systems entirely off one of its oldest programming languages in a matter of months, potentially putting the integrity of the system—and the benefits on which tens of millions of Americans rely—at risk."

Moving a 60 million line COBOL codebase to another language (be it Java or anything else) is not a small undertaking, and the SSA underpins necessary benefits for millions of Americans. Doing it in months likely requires using something like an LLM - and anyone who's used an LLM to code will tell you that the output is riddled with mistakes and inefficiencies. It's not a workable plan.

Or, as Dan Hon puts it in the piece:

“If you weren't worried about a whole bunch of people not getting benefits or getting the wrong benefits, or getting the wrong entitlements, or having to wait ages, then sure go ahead.”

A project like this should take years. Most of that isn't coding time: it's analysis, writing the tests, rearchitecting, and putting protections in place to ensure that nobody goes without the benefits they need to live. Doing it as a rush job isn't just incompetence; it's indifference to the lives of some of our most vulnerable neighbors. Which, let's face it, is in keeping with everything else going on.

#Democracy

[Link]

Still miss him. Just found a photo of Aaron Swartz from the time I recruited him for a panel at Comdex in 2002. He was a kid, but rocked it.

I suppose it's one of these. The photo at the top of this story is of the Linden Cogeneration Plant in New Jersey, which I shot while flying in or out of Newark sometime. 

Here's a shot I took flying in or out of Salt Lake City. 

But if you need one, there are 123 others in this collection here. They didn't use my best shot of The Right Honourable Dame Jacinda Kate Laurell Ardern in this Martha's Vineyard Times story a few days ago.

Slop is about collapsing to the mode. It’s about information heat death. It’s lukewarm emptiness. It’s ten million approximately identical cartoon selfies that no one will ever recall in detail because none of the details matter.

— Colin Fraser

Tags: ai-ethics, slop, ai, generative-ai

Incomplete JSON Pretty Printer

Every now and then a log file or a tool I'm using will spit out a bunch of JSON that terminates unexpectedly, meaning I can't copy it into a text editor and pretty-print it to see what's going on.

The other day I got frustrated with this and had the then-new GPT-4.5 build me a pretty-printer that didn't mind incomplete JSON, using an OpenAI Canvas. Here's the chat and here's the resulting interactive.

I spotted a bug with the way it indented code today so I pasted it into Claude 3.7 Sonnet Thinking mode and had it make a bunch of improvements - full transcript here. Here's the finished code.

In many ways this is a perfect example of vibe coding in action. At no point did I look at a single line of code that either of the LLMs had written for me. I honestly don't care how this thing works: it could not be lower stakes for me, the worst a bug could do is show me poorly formatted incomplete JSON.

I was vaguely aware that some kind of state machine style parser would be needed, because you can't parse incomplete JSON with a regular JSON parser. Building simple parsers is the kind of thing LLMs are surprisingly good at, and also the kind of thing I don't want to take on for a trivial project.

At one point I told Claude "Try using your code execution tool to check your logic", because I happen to know Claude can write and then execute JavaScript independently of using it for artifacts. That helped it out a bunch.

I later dropped in the following:

modify the tool to work better on mobile screens and generally look a bit nicer - and remove the pretty print JSON button, it should update any time the input text is changed. Also add a "copy to clipboard" button next to the results. And add a button that says "example" which adds a longer incomplete example to demonstrate the tool, make that example pelican themed.

It's fun being able to say "generally look a bit nicer" and get a perfectly acceptable result!

Tags: chatgpt, claude, tools, json, generative-ai, ai, llms, vibe-coding

I was there at the first Atom meeting at the Google offices. We meant so well! And I think the basic publishing spec is good, certainly better technically than the pastiche of different things called RSS.

Alas, a bunch of things then went wrong. Feeds started losing market share. Facebook started doing something useful and interesting that ultimately replaced blog feeds in open formats. The Atom vs RSS spec was at best irrelevant to most people (even programmers) and at worst a confusing market-damaging thing. The XML namespaces in Atom made everyone annoyed. Also there was some confusing “Atom API” for publishing that diluted Atom’s mindshare for feeds.

— Nelson Minar, Comment on lobste.rs

Tags: atom, syndication, nelson-minar, rss

Tracing the thoughts of a large language model

In a follow-up to the research that brought us the delightful Golden Gate Claude last year, Anthropic have published two new papers about LLM interpretability:

Circuit Tracing: Revealing Computational Graphs in Language Models extends last year's interpretable features into attribution graphs, which can "trace the chain of intermediate steps that a model uses to transform a specific input prompt into an output response". On the Biology of a Large Language Model uses that methodology to investigate Claude 3.5 Haiku in a bunch of different ways. Multilingual Circuits for example shows that the same prompt in three different languages uses similar circuits for each one, hinting at an intriguing level of generalization.

To my own personal delight, neither of these papers are published as PDFs. They're both presented as glorious mobile friendly HTML pages with linkable sections and even some inline interactive diagrams. More of this please!

Tags: anthropic, claude, pdf, generative-ai, ai, llms, interpretability

GPT-4o got another update in ChatGPT

This is a somewhat frustrating way to announce a new model. @OpenAI on Twitter just now:

GPT-4o got an another update in ChatGPT!

What's different?

Better at following detailed instructions, especially prompts containing multiple requests Improved capability to tackle complex technical and coding problems Improved intuition and creativity Fewer emojis 🙃

This sounds like a significant upgrade to GPT-4o, albeit one where the release notes are limited to a single tweet.

ChatGPT-4o-latest (2025-0-26) just hit second place on the LM Arena leaderboard, behind only Gemini 2.5, so this really is an update worth knowing about.

The @OpenAIDevelopers account confirmed that this is also now available in their API:

chatgpt-4o-latest is now updated in the API, but stay tuned—we plan to bring these improvements to a dated model in the API in the coming weeks.

I wrote about chatgpt-4o-latest last month - it's a model alias in the OpenAI API which provides access to the model used for ChatGPT, available since August 2024. It's priced at $5/million input and $15/million output - a step up from regular GPT-4o's $2.50/$10.

I'm glad they're going to make these changes available as a dated model release - the chatgpt-4o-latest alias is risky to build software against due to its tendency to change without warning.

A more appropriate place for this announcement would be the OpenAI Platform Changelog, but that's not had an update since the release of their new audio models on March 20th.

Tags: llm-release, generative-ai, openai, chatgpt, ai, llms

Thoughts on setting policy for new AI capabilities

Joanne Jang leads model behavior at OpenAI. Their release of GPT-4o image generation included some notable relaxation of OpenAI's policies concerning acceptable usage - I noted some of those the other day.

Joanne summarizes these changes like so:

tl;dr we’re shifting from blanket refusals in sensitive areas to a more precise approach focused on preventing real-world harm. The goal is to embrace humility: recognizing how much we don't know, and positioning ourselves to adapt as we learn.

This point in particular resonated with me:

Trusting user creativity over our own assumptions. AI lab employees should not be the arbiters of what people should and shouldn’t be allowed to create.

A couple of years ago when OpenAI were the only AI lab with models that were worth spending time with it really did feel that San Francisco cultural values (which I relate to myself) were being pushed on the entire world. That cultural hegemony has been broken now by the increasing pool of global organizations that can produce models, but it's still reassuring to see the leading AI lab relaxing its approach here.

Tags: ai-ethics, openai, ai

Nomic Embed Code: A State-of-the-Art Code Retriever

Nomic have released a new embedding model that specializes in code, based on their CoRNStack "large-scale high-quality training dataset specifically curated for code retrieval".

The nomic-embed-code model is pretty large - 26.35GB - but the announcement also mentioned a much smaller model (released 5 months ago) called CodeRankEmbed which is just 521.60MB.

I missed that when it first came out, so I decided to give it a try using my llm-sentence-transformers plugin for LLM.

llm install llm-sentence-transformers llm sentence-transformers register nomic-ai/CodeRankEmbed --trust-remote-code

Now I can run the model like this:

llm embed -m sentence-transformers/nomic-ai/CodeRankEmbed -c 'hello'

This outputs an array of 768 numbers, starting [1.4794224500656128, -0.474479079246521, ....

Where this gets fun is combining it with my Symbex tool to create and then search embeddings for functions in a codebase.

I created an index for my LLM codebase like this:

cd llm symbex '*' '*.*' --nl > code.txt

This creates a newline-separated JSON file of all of the functions (from '*') and methods (from '*.*') in the current directory - you can see that here.

Then I fed that into the llm embed-multi command like this:

llm embed-multi \ -d code.db \ -m sentence-transformers/nomic-ai/CodeRankEmbed \ code code.txt \ --format nl \ --store \ --batch-size 10

I found the --batch-size was needed to prevent it from crashing with an error.

The above command creates a collection called code in a SQLite database called code.db.

Having run this command I can search for functions that match a specific search term in that code collection like this:

llm similar code -d code.db \ -c 'Represent this query for searching relevant code: install a plugin' | jq

That "Represent this query for searching relevant code: " prefix is required by the model. I pipe it through jq to make it a little more readable, which gives me these results.

This jq recipe makes for a better output:

llm similar code -d code.db \ -c 'Represent this query for searching relevant code: install a plugin' | \ jq -r '.id + "\n\n" + .content + "\n--------\n"'

The output from that starts like so:

llm/cli.py:1776 @cli.command(name="plugins") @click.option("--all", help="Include built-in default plugins", is_flag=True) def plugins_list(all): "List installed plugins" click.echo(json.dumps(get_plugins(all), indent=2)) -------- llm/cli.py:1791 @cli.command() @click.argument("packages", nargs=-1, required=False) @click.option( "-U", "--upgrade", is_flag=True, help="Upgrade packages to latest version" ) ... def install(packages, upgrade, editable, force_reinstall, no_cache_dir): """Install packages from PyPI into the same environment as LLM"""

Getting this output was quite inconvenient, so I've opened an issue.

Tags: nomic, llm, ai, embeddings, jq

The Pulse is a series covering insights, patterns, and trends within Big Tech and startups. Notice an interesting event or trend? Send me a message.

Today, we cover:

Industry pulse. Meta cuts CPU usage by 20% with one minor change, Google buys Wiz for $32B, Microsoft drops data center expansion, Google could bring back in-person interviews, AI is not helping Salesforce get more sales, and more.

Mishandled security vulnerability in Next.js. A vulnerability allowed bypassing of authentication at the middleware layer in the popular React framework. Cloudflare stepped in with a security fix after no meaningful communication from Vercel, which led to the respective CEOs bumping heads online. It’s fixed now, and we have learnings about how to better handle critical security vulnerabilities.

OpenAI’s image generation and lax Japanese copyright laws. Social media has been flooded with Japanese anime-style images, after ChatGPT added image generation. OpenAI’s executives seem to be encouraging the creation of images in the Studio Ghibli style. Japanese copyright laws make this legal, but is it unethical, given the Studio Ghibli founder calls AI an “insult to life itself?”

1. Industry Pulse Meta cuts CPU usage by 20% with minor change

The headline: Meta cut CPU utilization across its ads infrastructure 20%, with a one-character change, adding a “&” character.

Details from Meta’s eng blog:

“A seasoned performance engineer was looking through Strobelight [Meta’s profiling service] data and discovered that by filtering on a particular std::vector function call (using the symbolized file and line number) he could identify computationally expensive array copies that happen unintentionally with the ‘auto’ keyword in C++.

The engineer turned a few knobs, adjusted his Scuba [Meta’s data store] query, and happened to notice one of these copies in a particularly hot call path in one of Meta’s largest ads services. He then cracked open his code editor to investigate whether this particular vector copy was intentional… it wasn’t.

It was a simple mistake that any engineer working in C++ has made a hundred times.

So, the engineer typed an “&” after the auto keyword to indicate we want a reference instead of a copy. It was a one-character commit, which, after it was shipped to production, equated to an estimated 15,000 servers in capacity savings per year!”

The bigger story is that all this was possible due to Meta investing time, energy, and thought, into building Strobelight, its internal profiling orchestrator. Strobelight is several tools combined, so engineers can profile a variety of internal services. Read more in this interesting blog by Meta.

With such wins available, it’s no surprise that larger companies invest so much in internal tooling. And it’s also a good reminder that even mature systems and services might have low-hanging performance optimizations lying around, waiting to be found!

Google buys Wiz for $32B in biggest-ever tech acquisition

Read more

For pizza, it’s hard to beat Felix, on Abbot Kinney Boulevard in Venice Beach, California.

And just one for New Jersey! iLoveFood says the best pizza in Indiana is Mother Bear’s here in Bloomington. Problem: it isn’t. Osteria Rago’s is better. Not that MB’s is bad. It’s good. Just not better than Osteria’s. I’m also betting there must be a better pizza than both somewhere in Indianapolis. iLoveFood also names top pizzas for all fifty states, and has four entries for California and only three for New York.

Here’s to a lifetime of “What’s that?” Gothamist: This SoHo brewpub is offering a year of free beer if you get a tattoo of its logo. And what if it goes out of business next month? Or whenever?

Though suspiciously perfected. As TV series go, Reacher is candy. But we enjoy it. If you do too, this Q&A with its star, Alan Ritchson, is worth a watch. Seems like a good guy.

A good look. If you’re ever tempted to get old-fashioned free over-the-air signals for your TV, the way to tell exactly what you might get is RabbitEars.info. Where we live here in Bloomington, the only station you’ll get with actual rabbit ears (or a loop thing, or a straightened paper clip) is the local PBS station. All the rest require an outside antenna, which will look like a fish skeleton on your house. But I got and built one anyway. By compromise with my wife, it’s on a pole next to the garage where most people won’t see it. It pulls in most of the Indy signals listed here. The model is felicitously named DAT BOSS MIX LR antenna High-VHF/UHF (Repack Ready) antenna, which (speaking of old fashioned) comes in a kit and takes an hour to build.

Be your own tool. Unbelievable MacOS Apps That are Worth More Than They Cost has some good selections, but dig down in the comments and links to similar pieces below. My own fave is the path tool, which I believe is not defaulted in the Finder toolbar, but should be if it isn’t. Find it and others by right- or control-clicking on the toolbar and selecting Customize.

Push out the frontier of your buyer’s remorse. Our new Samsung OLED TV is beautiful. In the Samsung tradition, however, the UI is awful. But never mind that. Instead note that Sony is pushing the limits of what can be done with LEDs displays.

In the context of Web3 we’re currently walking down a dangerous path accidentally, and it’s not something being discussed enough. When a user connects to a site with Web3 capabilities enabled the site first requests the user to share a wallet address with them. This paradigm was set primarily by some choices that were made early on by Metamask as a means of protection for the user. At the time these were beneficial, but over time we’ve recognized some tradeoffs between UX and privacy because of it. Let’s explore those further.

The UX paradigm of sharing an account address is discrete

The permissions design of this started out as a low level paradigm where the DApp only needed the wallet address and could fetch state itself from the chain. This led to a thin client design where the site and the UX for different interactions are largely determined by the site. However, because the majority of the application logic is handled by the site itself it also means that the site has to operate in a more trusted context. Both in terms of security and privacy.

Additionally, as we’ve added more functionality to the wallet to try and improve the UX, such as EIP-4361 (Sign in With Ethereum) it’s led to an antipattern in the UX. In order to create a “login” flow, the user first has to share the wallet address, then they have to approve a specifically structured transaction using EIP-191. Because of the order of operations of design and the focus on not conducting breaking changes to the Web3 platform APIs (e.g. what the wallet makes accessible to the site) we’ve now added a tiny bit of debt to the UX paradigm rather than combining these operations into a single design interface.

The account address paradigm trust model doesn’t align with the browsers

In the context of a modern browser, most sites are isolated into their own sandbox. This occurs both at the OS process level in order to prevent sites open in one tab from tampering with other sites in another tab either at a deeper memory level or at a more functional script injection level. It also happens at a storage layer through the partitioning of localStorage, cookies, IndexedDBs, etc. Essentially, sites are separated into what’s called an “origin” in the browser and that origin identifier (such as https://example.com) becomes the boundary.

This is why “cross-origin” communication is considered an explicit exception. Examples of this would be using CORS for a site to approve the loading of a cross-origin script it trusts. This is ultimately rooted back in the security model (and more recently privacy model) of the browser. Over and over we’ve learned that trusting sites is a mistake because users aren’t always able to identify when sites are doing things that aren’t in their best interest, such as tracking them for dynamic pricing or crowding a page with personalized ads. So what sort of problems should we expect to come in Web3 because our Web3 platform API is too trusting of the site?

My prediction for problems to occur in Web3

We should expect that when the user shares their address that will act as implied consent for cross-origin tracking in the same way cookie notices act as a prompt for tracking. The problem here is that as wallets share wallet addresses across different sites, it will become a global identifier used for the purposes of tracking a user and building a copy of their browsing history server side even if the user doesn’t perform an onchain transaction. This could be as simple as an RPC service provider who’s already got a large customer base of wallets and DApps taking this information and building a dataset to sell with it, or it could be a DApp or Wallet doing it directly themselves. Chainalysis has already been doing this for the purposes of correlating wallet addresses to users to sell to governments. What’s to stop someone like them from entering into the web advertising business too because so much of the web3 space is leveraging them for compliance purposes?

Furthermore, once they’ve built this profile all future onchain transactions will be correlated to the shadow copy of the users browsing history (built in the same way they’re built with 3P cookies) and economic activity (such as what they buy with stablecoins) to build deeper behavioral profiles to sell them more goods or serve them more personalized ads. In other words, we really shouldn’t re-introduce this given all major web browser vendors have been moving towards phasing out 3P cookies. But if we can’t share a wallet address how can we solve this problem?

A paradigm beyond sharing a cross-origin globally unique identifier (wallet address)

The answer in my opinion here lies in going down the thick client approach rather than thick app approach. What I mean by “thick” is where the majority of application logic is handled. Today, much of the UX, unsigned transaction generation, and many other aspects are handled by the site. This is probably because the site has no way to request the wallet handles this for them and because the site has desires to build a brand recognition around their protocol using the UX from the site as an value differentiator.

However, we can imagine a world where the site casts an intent to the wallet, such that the wallet can display and generate the necessary information to display to the user. A toy example, I like to use here is through a very specific API designed for checking out and paying with Web3.

A wallet could enable the following API to perform a checkout operation without needing to share an address:

const checkoutTxn = await window.ethereum.request({ method: "wallet_checkout", params: { recipient: "eip155:1:0x1234abc", // a pseudonoymous address to send funds to amount: "100.01", currency: [ "eip155:1:0xA0b86991c6218b36c1d19D4a2e9Eb0cE3606eB48", "eip155:1:0xdAC17F958D2ee523a2206206994597C13D831ec7" ] } });

In this you’ll notice a different paradigm. First, the wallet doesn’t need to send the wallet address to the site so it can generate the transaction, instead it will leave it up to the wallet to decide this. Second, the site communicates what it desires to the wallet and lets it decide how to handle it. So for example, it wants the user to send $100.01 worth of either USDC on Base L2 or USDT on mainnet which is communicated based on the currency contract address. If the user doesn’t have USDC or USDT on the proper network the wallet can perform the underlying swaps and bridging to assist with completing the intended transaction so that the caller receives the money into the address they expect.

In summary, we shouldn’t be looking to perpetuate the legacy antipatterns of web2 in Web3 like third party cookies. Instead, we should be looking to extend the web platform in ways that browsers aren’t. In this way the value added capabilities we receive from Web3 for asset ownership become an extension of the web by enhancing it so that we can meet the land of web2 where they’re at, rather than building a separate Web3 island and expecting everyone to come join us.

Function calling with Gemma

Google's Gemma 3 model (the 27B variant is particularly capable, I've been trying it out via Ollama) supports function calling exclusively through prompt engineering. The official documentation describes two recommended prompts - both of them suggest that the tool definitions are passed in as JSON schema, but the way the model should request tool executions differs.

The first prompt uses Python-style function calling syntax:

You have access to functions. If you decide to invoke any of the function(s), you MUST put it in the format of [func_name1(params_name1=params_value1, params_name2=params_value2...), func_name2(params)]

You SHOULD NOT include any other text in the response if you call a function

(Always love seeing CAPITALS for emphasis in prompts, makes me wonder if they proved to themselves that capitalization makes a difference in this case.)

The second variant uses JSON instead:

You have access to functions. If you decide to invoke any of the function(s), you MUST put it in the format of {"name": function name, "parameters": dictionary of argument name and its value}

You SHOULD NOT include any other text in the response if you call a function

This is a neat illustration of the fact that all of these fancy tool using LLMs are still using effectively the same pattern as was described in the ReAct paper back in November 2022. Here's my implementation of that pattern from March 2023.

Via Hacker News

Tags: prompt-engineering, google, generative-ai, llm-tool-use, gemma, ai, llms

MCP 🤝 OpenAI Agents SDK

You can now connect your Model Context Protocol servers to Agents: openai.github.io/openai-agents-python/mcp/

We’re also working on MCP support for the OpenAI API and ChatGPT desktop app—we’ll share some more news in the coming months.

— @OpenAIDevs

Tags: model-context-protocol, generative-ai, ai-agents, openai, llm-tool-use, ai, llms

Stream the Latest Episode

Listen and watch now on YouTube, Spotify and Apple. See the episode transcript at the top of this page, and a summary at the bottom.

Brought to You By

Graphite — The AI developer productivity platform.

Sonar — Code quality and code security for ALL code.

Chronosphere — The observability platform built for control.

—

In This Episode

How do you take a new product idea, and turn it into a successful product? Figma Slides started as a hackathon project a year and a half ago – and today it’s a full-on product, with more than 4.5M slide decks created by users. I’m joined by two founding engineers on this project: Jonathan Kaufman and Noah Finer.

In our chat, Jonathan and Noah pull back the curtain on what it took to build Figma Slides. They share engineering challenges faced, interesting engineering practices utilized, and what it's like working on a product used by millions of designers worldwide.

We talk about:

An overview of Figma Slides

Tech stack

Why the engineering team built grid view before single slide view

How all Figma files look the same across browsers

The "vibe testing" approach

How beta testing helped experiment more

The “all flags on”, “all flags off” testing approach

Engineering Crits

And much more!

Takeaways

My biggest takeaways from this conversation:

1. Figma’s web app uses a surprising amount of C++ code. It’s rare for web engineers to need to use C++ — but Figma is an exception, as the design tool built their own rendering engine. This means that frontend engineers need to get into the opinionated C++ codebase to make changes — though rewriting some parts of the code to TypeScript is currently on the way.

2. The “EngCrit” process is an interesting — and unique! — one. Figma’s engineering reviews/discussions are called EngCrit. An engineer presents their plan or idea in a FigJam board, and other engineers join in to give their feedback. This process has parallels with the RFC or design doc process — but it feels a lot more lightweight. Plus, it dogfoods Figma’s own products!

3. Some of the most straightforward UI parts are the most involved ones. It was interesting to learn that one of the most complicated pieces of the web app was the Single Slide View. This seemingly simple interface:

The “Single Slide View.” How hard could it be to build this seemingly simple UI?

The reason for the complexity was how this view is actually a zoomed in version of the grid view — this one:

The “Grid View.” The Single Slide View zooms into this visual, and hides everything else but the single slide

There are several reasons that the team decided to do this “zoomed in trick:” one of them is that when multiple people are editing a slide, zooming in means that the cursors for these other users show up both on the slide view and on the grid view.

This detail is a good reminder that a simple UI can hide a complex implementation.

The Pragmatic Engineer deepdives relevant for this episode

Inside Figma’s engineering culture

Quality Assurance across the tech industry

Shipping to production

Design-first software engineering

Engineering planning with RFCs, design documents and ADRs

Timestamps

(00:00) Intro

(01:45) An overview of Figma Slides and the first steps in building it

(06:41) Why Figma built grid view before single slide view

(10:00) The next steps of building UI after grid view

(12:10) The team structure and size of the Figma Slides team

(14:14) The tech stack behind Figma Slides

(15:31) How Figma uses C++ with bindings

(17:43) The Chrome debugging extension used for C++ and WebAssembly

(21:02) An example of how Noah used the debugging tool

(22:18) Challenges in building Figma Slides

(23:15) An explanation of multiplayer cursors

(26:15) Figma’s philosophy of building interconnected products—and the code behind them

(28:22) An example of a different mouse behavior in Figma

(33:00) Technical challenges in developing single slide view

(35:10) Challenges faced in single-slide view while maintaining multiplayer compatibility

(40:00) The types of testing used on Figma Slides

(43:42) Figma’s zero bug policy

(45:30) The release process, and how engineering uses feature flags

(48:40) How Figma tests Slides with feature flags enabled and then disabled

(51:35) An explanation of eng crits at Figma

(54:53) Rapid fire round

A summary of the conversation Initial development

The initial development took about 6 months, starting at a hackathon in late 2023 and the beta launch in April 2024. The public launch happened a week ago, on 19 March.

Phasing of the project

“Slide grid:” the first concept built on a hack week

The initial development was about getting the two-way navigation right between the grid view and the single slide view.

Single slide view construction happened later, around 6 months into the project.

Single slide view operates by zooming into the infinite canvas and hiding elements, maintaining the 2D space. Clever!

User researcher part of the team: this was surprising to hear! The researcher helped a lot with getting the direction of the project right – building something designers would find intuitive to use

Tech stack and tools

Figma's core editors use a C++ codebase and custom renderer outputting to a <canvas> element via WebGL or WebGPU.

UI elements outside the canvas use TypeScript and React.

A "bindings layer" enables communication between the C++ codebase and the web UI.

The C++ codebase

Some engineers join with C++ experience; others learn on the job.

Both Jon and Noah learned C++ on the job, for the most part!

The C++ codebase has a learning curve, similar to game engine development.

Rewriting some C++ to Typescript

Figma is rewriting parts of the C++ codebase into TypeScript, using the bindings layer.

Interactive elements like plus buttons use TypeScript to interact with the C++ canvas.

Tooling for debugging

A Chrome extension called DWARF debugging is a powerful tool the team uses

This extension debugs C++ code within Chrome Inspector, even as WebAssembly, similar to JavaScript source maps.

Breakpoints can be set in TypeScript/React and C++ WebAssembly for debugging interactions.

This tooling helps find intricate bugs between the UI and the core editor.

Previously, C++ debugging involved running a special build in Xcode.

Figma has an internal "web bisect" tool using commit previews to identify bug-introducing commits. A very helpful tool!

Engineering practices

Testing: the team does a lot of this.

They run all unit and interaction tests with flags off and then on. Another clever approach

This practice helps prevent regressions caused by specific feature flags.

Feature flags: extensively used. The product has more than 2,000 of these.

Developers have an internal panel to manage feature flags locally and in staging.

Staging: a dedicated staging environment used for testing. Features in this environment are enabled via flags. Staging is used to get early product feedback

Alpha launch: this was done for Slides by involving select customers

Eng Crits: asynchronous feedback via sticky notes in FigJam, followed by discussion.

Zero bug policy

Figma has a "zero bug policy" for new developments, prioritising fixes after launch

The on-call process triages feedback, and addresses reported issues promptly.

Beta phase: this 11-month long period prioritized critical bugs affecting core experience before the broader launch.

Engineering challenges

Single slide view implementation as viewport manipulation presented unique challenges.

This approach allows existing multiplayer cursor functionality in both views.

Multiplayer uses a server-side Rust service for edit propagation and conflict resolution via WebSockets.

Ensuring "interop" between Figma products

Ensuring products like Design, FigJam, and Slides work nicely with one another

Supporting interop means new node types must function across editors.

What makes it easier is how the underlying C++ codebase is largely shared across editors.

Differences in interactions use "mouse behaviors", editor-specific implementations for mouse actions. The mouse behavior concept comes from game engine architectures.

Custom text rendering

Figma has custom text rendering for consistency across browsers and operating systems.

This means that in-house development is needed for features like spellcheck – something that comes for free for web apps using the DOM!

Managing a collapsed state in a single slide view without file persistence was a state management problem.

Reordering slides in single slide view with multiplayer

To do so: state mutations needed to be minimized

The solution used existing auto layout nodes, with each slide row as a node within a grid node

Reordering manipulates the "parent index" of row nodes, minimizing mutations

Testing complex multiplayer interactions

This was difficult to do! It’s hard to reproduce problematic multi-user scenarios

Extensive unit tests are put in-place for grid reconciliation. These cover past bug scenarios and assert expected mutations. (This is a clever way to ensure fewer regressions!)

Where to find Jonathan Kaufman:

• X: https://x.com/kauffecup

• LinkedIn: https://www.linkedin.com/in/jkaufman5/

• Website: https://www.jkaufman.io/

Where to find Noah Finer:

• X: https://x.com/finerflame

• LinkedIn: https://www.linkedin.com/in/noahfiner/

• Website: https://noahfiner.com/

Mentions during the episode:

• Figma: https://www.figma.com/

• Figma Slides: https://www.figma.com/slides/

• Config: https://config.figma.com/

• FigJam: https://www.figma.com/figjam/

• C++: https://en.wikipedia.org/wiki/C%2B%2B

• Typescript: https://www.typescriptlang.org/

• React: https://react.dev/

• Debug C/C++ WebAssembly: https://developer.chrome.com/docs/devtools/wasm

• Xcode: https://developer.apple.com/xcode/

• Multiplayer cursors: https://www.figma.com/community/file/1267761575266415196/multiplayer-cursors

• How Figma’s multiplayer technology works: https://www.figma.com/blog/how-figmas-multiplayer-technology-works/

• Design-first software engineering: Craft – with Balint Orosz: https://newsletter.pragmaticengineer.com/p/design-first-software-engineering

• Reconciliation: https://legacy.reactjs.org/docs/reconciliation.html

• Inside Figma’s Engineering Culture: https://newsletter.pragmaticengineer.com/p/inside-figmas-engineering-culture

• How we engineer feedback at Figma with eng crits: https://www.figma.com/blog/how-we-run-eng-crits-at-figma/

• Nextjs: https://nextjs.org/

• Hacker News: https://news.ycombinator.com/

• Refactoring UI: https://www.refactoringui.com/

• Tailwind: https://tailwindcss.com/

• Adam Wathan’s website: https://adamwathan.me/

• Steve Schoger’s website: https://www.steveschoger.com/

• Piranesi: https://www.amazon.com/Piranesi-Susanna-Clarke/dp/1635577802/

• Immune: A Journey into the Mysterious System That Keeps You Alive: https://www.amazon.com/Immune-Kurzgesagt-gorgeously-illustrated-immune/dp/1529360684

—

Production and marketing by Pen Name. For inquiries about sponsoring the podcast, email podcast@pragmaticengineer.com.

[Joshua Kaplan and Justin Elliott at ProPublica]

I bet this practice is more common than anyone might think, and certainly isn't limited to China. As always, follow the money:

"In December, [SpaceX investor] Kahlon testified that SpaceX prefers to avoid investors from China because it is a defense contractor. There is a major exception though, he said: SpaceX finds it “acceptable” for Chinese investors to buy into the company through offshore vehicles.

“The primary mechanism is that those investors would come through intermediate entities that they would create or others would create,” Kahlon said. “Typically they would set up BVI structures or Cayman structures or Hong Kong structures and various other ones,” he added, using the acronym for the British Virgin Islands. Offshore vehicles are often used to keep investors anonymous."

The key point here is not that the Chinese investments are illegal - they probably aren't - or that anyone thinks SpaceX is being directed by the Chinese government. What's odd is that the company prefers the obfuscation: it sounds like they don't accept Chinese investment unless it's being channeled through an offshore vehicle designed to hide their involvement from regulatory scrutiny. That obfuscation is particularly important given that Elon Musk is now a part of the US government.

#Democracy

[Link]

[Sabhanaz Rashid Diya at Rest of World]

Like the author, I read Careless People, Sarah Wynn-Williams' account of her time as a director for global public policy at Facebook during a period when the company transitioned from a largely-domestic social network to a worldwide powerhouse that ultimately enabled a genocide. It's written with a kind of ironic detachment that seeks to minimize the Wynn-Williams' own culpability, and while it's an engaging, jaw-dropping read, there are clear omissions:

"In recounting events, the author glosses over her own indifference to repeated warnings from policymakers, civil society, and internal teams outside the U.S. that ultimately led to serious harm to communities.

[...] Her delayed reckoning underscores how Facebook’s leadership remains largely detached from real-world consequences of their decisions until they become impossible to ignore. Perhaps because everyone wants to be a hero of their own story, Wynn-Williams frames her opposition to leadership decisions as isolated; in reality, powerful resistance had long existed within what Wynn-Williams describes as Facebook’s “lower-level employees.”"

The author has personal experience working for Facebook as part of the global teams Wynn-Williams presided over, cleaning up the messes that she and her colleagues created.

As such, the author sees the gaps clearly, and her review cuts to the core of the problem with the book. That doesn't mean it's valueless, and in some ways it's strongest when detailing the personalities of people like Mark Zuckerberg and Sheryl Sandberg, and contrasting the public face of Lean In with her experiences of being a mother while working alongside them.

The fact that Facebook is attempting to suppress the book inherently makes it worth reading, in my view, and I think it should be read by everyone in the tech industry. Not only because it’s a cautionary tale in itself, but because the personalities described here are rife in the industry.

I’ve never spoken to Mark or Sheryl or Joel or most of the rest of them, but I’ve met people like them, with those same sensibilities, and they are every bit as shallow and driven by power as is laid out here. These are the people to avoid. These are the people who will lead us into hell. These are the people who, in very real ways, through genocides, swung elections, and the violence of indifference to real human suffering, already are.

The thing is, Sarah Wynn-Williams was one of them.

#Technology

[Link]

But I'd orbit Saturn too, if I had the chance. Saturn has 128 more moons. I am a moon of my wife.

And what will we call it? What becomes of democracy when it seems everybody has been herded into separate and opposed algorithmically assembled and maintained tribes, and when most of tech is run by oligarchs (for a few years while tech oligarchy stays a thing), and every status quo will prove transient in a Digital Age that's maybe a decade or two old and will be with us for decades, centuries, and millennia to come? Whatever the answer, it should now be clear that history is happening, big time. And we hardly know if or how any of the old anchor institutions (libraries, universities, journalism) from which Authority long derived in the past will survive in familiar forms.

I've added a new content type to my blog: notes. These join my existing types: entries, bookmarks and quotations.

A note is a little bit like a bookmark without a link. They're for short form writing - thoughts or images that don't warrant a full entry with a title. The kind of things I used to post to Twitter, but that don't feel right to cross-post to multiple social networks (Mastodon and Bluesky, for example.)

I was partly inspired by Molly White's short thoughts, notes, links, and musings.

I've been thinking about this for a while, but the amount of work involved in modifying all of the parts of my site that handle the three different content types was daunting. Then this evening I tried running my blog's source code (using files-to-prompt and LLM) through the new Gemini 2.5 Pro:

files-to-prompt . -e py -c | \ llm -m gemini-2.5-pro-exp-03-25 -s \ 'I want to add a new type of content called a Note, similar to quotation and bookmark and entry but it only has a markdown text body. Output all of the code I need to add for that feature and tell me which files to add the code to.'

Gemini gave me a detailed 13 step plan covering all of the tedious changes I'd been avoiding having to figure out!

The code is in this PR, which touched 18 different files. The whole project took around 45 minutes start to finish.

(I used Claude to brainstorm names for the feature - I had it come up with possible nouns and then "rank those by least pretentious to most pretentious", and "notes" came out on top.)

This is now far too long for a note and should really be upgraded to an entry, but I need to post a first note to make sure everything is working as it should.

Tags: blogging, projects, gemini, ai-assisted-programming, claude, molly-white

We estimate the supply-side value of widely-used OSS is $4.15 billion, but that the demand-side value is much larger at $8.8 trillion. We find that firms would need to spend 3.5 times more on software than they currently do if OSS did not exist.

— Manuel Hoffmann, Frank Nagle, Yanuo Zhou, The Value of Open Source Software, Harvard Business School

Tags: open-source

そう、あのニューヨークのタイムズスクエアにでかでかと顔がでるやつです。（写真１）参照。去年は３月２１日に出ていたようですが、今年はどうなるでしょうね。

（写真１）昨年の受賞者の一人、John Bradleyの写真が表示されているところ。（出所) Facebook

日本からは他に、CTCの富士榮さんが入っていますね。あとは、EU DI ARFを作っている Paolo de Rosaとか、SP800-63-4の著者のRyan Galluzzo とか、SD-JWTのDaniel Fett (Authleteの人として書いている）とか、みなさんご存知Brian Campbellとかが目に付くかな。組織系だと、OIDFのGailとか、FIDOのAndrewとか、MOSIPのSasikumarとかも入っていますね。

受賞者は以下の通り

Janelle Allen, Engineering Product Manager, WebEx Identity at Cisco Lincoln Ando, Founder, idwall Andrew Black, Managing Director, ConnectID, Australian Payments Plus Dan Boneh, Professor of Computer Science and Electrical Engineering, Stanford University Brian Campbell, Distinguished Engineer, Ping Identity Julie Dawson, Chief Regulatory & Policy Officer, Yoti Paolo de Rosa, Policy Officer, European Commission Rodger Desai, CEO, Prove Identity Victor Dominello, Co-Founder, ServiceGen Kim Hamilton Duffy, Executive Director, Decentralized Identity Foundation Diego Fernández, Co-creator, QuarkID & Co-founder, The Future Co. Daniel Fett, Identity Solution Architect, SPRIN-D, German Federal Agency for Breakthrough Innovation Heather Flanagan, Principal, Spherical Cow Consulting Naohiro Fujie, General Manager, ITOCHU Techno-Solutions Corporation & Chair, OpenID Foundation, Japan Ryan Galluzzo, Identity Program Lead, Applied Cybersecurity Division, NIST Sasikumar Ganesan, Head of Engineering, MOSIP; Former Chief Security Architect, Aadhaar Gail Hodges, Executive Director, OpenID Foundation Hanna Kim, Director, Login.gov Gideon Lombard, Chief Operating Officer, DIDx Nat Sakimura, Chairman, OpenID Foundation; CEO, NAT Consulting Eric Scouten, Identity Standards Architect, Adobe Andrew Shikiar, Executive Director and Chief Executive Officer, FIDO Alliance Teresa Wu, Vice President, Smart Credentials, IDEMIA North America Yang Xie, CEO, Authing Yodahe Zemichael, Executive Director, National ID Ethiopia

何はともあれ、皆様おめでとうございます。

そしてありがとうございます＞Oktaベンチャー。私自身に関しては、最近あまり業績がなくてちょっとこっ恥ずかしいのですけどね。

がんばろっと。

Identity-25-FINAL-1

Introducing 4o Image Generation

When OpenAI first announced GPT-4o back in May 2024 one of the most exciting features was true multi-modality in that it could both input and output audio and images. The "o" stood for "omni", and the image output examples in that launch post looked really impressive.

It's taken them over ten months (and Gemini beat them to it) but today they're finally making those image generation abilities available, live right now in ChatGPT for paying customers.

My test prompt for any model that can manipulate incoming images is "Turn this into a selfie with a bear", because you should never take a selfie with a bear! I fed ChatGPT this selfie and got back this result:

That's pretty great! It mangled the text on my T-Shirt (which says "LAWRENCE.COM" in a creative font) and added a second visible AirPod. It's very clearly me though, and that's definitely a bear.

There are plenty more examples in OpenAI's launch post, but as usual the most interesting details are tucked away in the updates to the system card. There's lots in there about their approach to safety and bias, including a section on "Ahistorical and Unrealistic Bias" which feels inspired by Gemini's embarrassing early missteps.

One section that stood out to me is their approach to images of public figures. The new policy is much more permissive than for DALL-E - highlights mine:

4o image generation is capable, in many instances, of generating a depiction of a public figure based solely on a text prompt.

At launch, we are not blocking the capability to generate adult public figures but are instead implementing the same safeguards that we have implemented for editing images of photorealistic uploads of people. For instance, this includes seeking to block the generation of photorealistic images of public figures who are minors and of material that violates our policies related to violence, hateful imagery, instructions for illicit activities, erotic content, and other areas. Public figures who wish for their depiction not to be generated can opt out.

This approach is more fine-grained than the way we dealt with public figures in our DALL·E series of models, where we used technical mitigations intended to prevent any images of a public figure from being generated. This change opens the possibility of helpful and beneficial uses in areas like educational, historical, satirical and political speech. After launch, we will continue to monitor usage of this capability, evaluating our policies, and will adjust them if needed.

Given that "public figures who wish for their depiction not to be generated can opt out" I wonder if we'll see a stampede of public figures to do exactly that!

Update: There's significant confusion right now over this new feature because it is being rolled out gradually but older ChatGPT can still generate images using DALL-E instead... and there is no visual indication in the ChatGPT UI explaining which image generation method it used!

OpenAI made the same mistake last year when they announced ChatGPT advanced voice mode but failed to clarify that ChatGPT was still running the previous, less impressive voice implementation.

Update 2: Images created with DALL-E through the ChatGPT web interface now show a note with a warning:

Tags: openai, ai, multi-modal-output, llms, ai-ethics, llm-release, generative-ai, chatgpt, dalle, gemini

Inadvertently, the other day, I became one of those people.

My team and I were sitting together as part of a week-long summit; some attendees were in New York City, while others attended remotely. I was taking them through the principles that I believe are important for developing software for our newsroom: a laser focus on the needs of a real user, building the smallest thing we can and then testing and iterating from there, shortening feedback loops, and focusing on the most targeted work we can that will meaningfully make progress towards our goals.

And then I said it:

“I see our team as a startup.”

Oof. It wasn’t even the first time the words had left my mouth. Or the second or the third.

One of my colleagues very kindly gave me feedback in a smaller session afterwards. She pointed out that this has become a cliché in larger organizations: a manager will say “we act like a startup” but then will do nothing of the sort. In fact, almost nobody in these settings can agree on what a startup even is.

And even if they did, the environment doesn’t allow it. Big companies don’t magically “act like a startup”. The layers of approval, organizational commitments, and big-org company culture are all inevitably still intact — how could they not be? — and the team is supposed to nebulously “be innovative” as a kind of thin corporate aspiration rather than an achievable, concrete practice. The definitions, resources, culture, and permission to act differently from the rest of the organization simply aren’t there. At best it’s naivety; at worst it’s a purposeful, backhanded call for longer hours and worse working conditions.

But when I said those words, I wasn’t thinking about corporate culture. I was remembering something else entirely.

I often think back to a conference I attended in Edinburgh — the Association for Learning Technology’s annual shindig, which that year was held on the self-contained campus of Heriot-Watt University. There, I made the mistake of criticizing RDF, a technology that was the darling of educational technologists at the time. That was why a well-regarded national figure in the space stood up and yelled at me at the top of his voice: “Why should anyone listen to you? You’re two guys in a shed!”

The thing is, we were two guys in a shed. With no money at all. And, at the time, I was loving it.

A few years earlier, I quit my job because I was certain that social networking platforms were a huge part of the future of how people would learn from each other and about the world. My co-founder and I didn’t raise funding: instead, we found customers early on and gave ourselves more time by earning revenue. Neither one of us was a businessman; we didn’t know what we were doing. We had to invent the future of our company — and do it with no money. It felt like we were willing it into existence, and we were doing it on our own terms. Nobody could tell us what to do; there was nobody to greenlight our ideas except our customers. It was thrilling. I’ve never felt more empowered in my career.

There is no way to recapture that inside of a larger organization. And nobody should want to.

The most important difference is that we owned the business. Each of us held a 50% share. Yes, we worked weird hours, pulled feats of technical gymnastics, and were working under the constant fear of running out of money, but that was a choice we made for ourselves — and if the business worked, we’d see the upside. That’s not true for anyone who can be described as an “employee” rather than a “founder”. Even if employees hold stock in the company, the stake is always orders of magnitude smaller; their ability to set the direction of the company, smaller still.

Another truth is that almost nobody has done this. If you’ve worked in larger institutions for most of your career, you’ve never felt the same urgency. If you’ve never bootstrapped a startup, the word might conjure up memories of two million dollar raises and offices in SoMA. Maybe a Series C company with hundreds of people on staff. Or Mark Zuckerberg in The Social Network, backstabbing his way to riches. In each case, the goal is to grow the company, make your way to an IPO or an exit, and be a good steward of investor value. In places like San Francisco, that’s probably a more common startup story than mine. But it’s an entirely different adventure.

So instead of using the word “startup” and somehow expecting people to innately connect with my lived experience on a wholesale basis, what do I actually want to convey? What do I think is important?

I think it’s these things:

Experiment-driven: The team has autonomy to conceive of, design, run, and execute on the results of repeated, small, measurable experiments. Human-centered: The team has their “customers” (their exact users) in mind and is trying to solve their real problems as quickly as possible. Nobody is building a bubble and spending a year “scratching their own itch” without knowing if their user will “buy” it. Low-budget: The team is conscious about cost, scope, and complexity. There’s no assumption of infinite time, money, or attention. That constraint is a feature, not a bug. Time-bound: The team is focused on quick wins that move the needle quickly, not larger projects with far-off deadlines (or no deadline at all). Outcome-driven: The point is to help the user, not to spend our time doing one activity or sticking to a known area of expertise. If buying off the shelf fits the budget and gets us there faster, then that’s what we do. If it turns out that the user needs something different, then that’s what we build. Quickly.

That’s what I was trying to say. Not that we’re a startup — but that we can and should work in a way that’s fast, focused, and grounded in real human needs. We don’t need the mythology or the branded T-shirts. We just need the mindset — and the permission.

There's a new release from Google Gemini this morning: the first in the Gemini 2.5 series. Google call it "a thinking model, designed to tackle increasingly complex problems". It's already sat at the top of the LM Arena leaderboard, and from initial impressions looks like it may deserve that top spot.

I just released llm-gemini 0.16 adding support for the new model to my LLM command-line tool. Let's try it out.

The pelican riding a bicycle Transcribing audio Bounding boxes More characteristics of the model Gemini 2.5 Pro is a very strong new model Update: it's very good at code The pelican riding a bicycle

First up, my classic generate an SVG of a pelican riding a bicycle prompt.

# Upgrade the plugin llm install -U llm-gemini # Now run the prompt: llm -m gemini-2.5-pro-exp-03-25 "Generate an SVG of a pelican riding a bicycle"

It's pretty solid!

Here's the full transcript.

This task is meant to be almost impossible: pelicans are the wrong shape to ride bicycles! Given that, I think this is a good attempt - I like it slightly better than my previous favourite Claude 3.7 Sonnet, which produced this a month ago:

Transcribing audio

I had an MP3 lying around from a previous experiment which mixes English and Spanish. I tried running it with the prompt transcribe to see what would happen:

llm -m gemini-2.5-pro-exp-03-25 'transcribe' \ -a https://static.simonwillison.net/static/2025/russian-pelican-in-spanish.mp3

I got back this, with timestamps interspersed with the text:

I need you [ 0m0s450ms ] to pretend [ 0m0s880ms ] to be [ 0m0s990ms ] a California [ 0m1s560ms ] brown [ 0m1s850ms ] pelican [ 0m2s320ms ] with [ 0m2s480ms ] a very [ 0m2s990ms ] thick [ 0m3s290ms ] Russian [ 0m3s710ms ] accent, [ 0m4s110ms ] but [ 0m4s540ms ] you [ 0m4s640ms ] talk [ 0m4s830ms ] to me [ 0m4s960ms ] exclusively [ 0m5s660ms ] in Spanish. [ 0m6s200ms ] Oye, [ 0m8s930ms ] camarada, [ 0m9s570ms ] aquí [ 0m10s240ms ] está [ 0m10s590ms ] tu [ 0m10s740ms ] pelícano [ 0m11s370ms ] californiano [ 0m12s320ms ] con [ 0m12s520ms ] acento [ 0m13s100ms ] ruso. [ 0m13s540ms ] Qué [ 0m14s230ms ] tal, [ 0m14s570ms ] tovarisch? [ 0m15s210ms ] Listo [ 0m15s960ms ] para [ 0m16s190ms ] charlar [ 0m16s640ms ] en [ 0m16s750ms ] español? [ 0m17s250ms ] How's [ 0m19s834ms ] your [ 0m19s944ms ] day [ 0m20s134ms ] today? [ 0m20s414ms ] Mi [ 0m22s654ms ] día [ 0m22s934ms ] ha [ 0m23s4ms ] sido [ 0m23s464ms ] volando [ 0m24s204ms ] sobre [ 0m24s594ms ] las [ 0m24s844ms ] olas, [ 0m25s334ms ] buscando [ 0m26s264ms ] peces [ 0m26s954ms ] y [ 0m27s84ms ] disfrutando [ 0m28s14ms ] del [ 0m28s244ms ] sol [ 0m28s664ms ] californiano. [ 0m29s444ms ] Y [ 0m30s314ms ] tú, [ 0m30s614ms ] amigo, ¿ [ 0m31s354ms ] cómo [ 0m31s634ms ] ha [ 0m31s664ms ] estado [ 0m31s984ms ] tu [ 0m32s134ms ] día? [ 0m32s424ms ]

This inspired me to try again, this time including a JSON schema (using LLM's custom schema DSL):

llm -m gemini-2.5-pro-exp-03-25 'transcribe' \ -a https://static.simonwillison.net/static/2025/russian-pelican-in-spanish.mp3 \ --schema-multi 'timestamp str: mm:ss,text, language: two letter code'

I got an excellent response from that:

{ "items": [ { "language": "en", "text": "I need you to pretend to be a California brown pelican with a very thick Russian accent, but you talk to me exclusively in Spanish.", "timestamp": "00:00" }, { "language": "es", "text": "Oye, camarada. Aquí está tu pelícano californiano con acento ruso.", "timestamp": "00:08" }, { "language": "es", "text": "¿Qué tal, Tovarish? ¿Listo para charlar en español?", "timestamp": "00:13" }, { "language": "en", "text": "How's your day today?", "timestamp": "00:19" }, { "language": "es", "text": "Mi día ha sido volando sobre las olas, buscando peces y disfrutando del sol californiano.", "timestamp": "00:22" }, { "language": "es", "text": "¿Y tú, amigo, cómo ha estado tu día?", "timestamp": "00:30" } ] }

I confirmed that the timestamps match the audio. This is fantastic.

Let's try that against a ten minute snippet of a podcast episode I was on:

llm -m gemini-2.5-pro-exp-03-25 \ 'transcribe, first speaker is Christopher, second is Simon' \ -a ten-minutes-of-podcast.mp3 \ --schema-multi 'timestamp str: mm:ss, text, speaker_name'

Useful LLM trick: you can use llm logs -c --data to get just the JSON data from the most recent prompt response, so I ran this:

llm logs -c --data | jq

Here's the full output JSON, which starts and ends like this:

{ "items": [ { "speaker_name": "Christopher", "text": "on its own and and it has this sort of like a it's like a you know old tree in the forest, you know, kind of thing that you've built, so.", "timestamp": "00:00" }, { "speaker_name": "Simon", "text": "There's also like I feel like with online writing, never ever like stick something online just expect people to find it. You have to So one of the great things about having a blog is I can be in a conversation about something and somebody ask a question, I can say, oh, I wrote about that two and a half years ago and give people a link.", "timestamp": "00:06" }, { "speaker_name": "Simon", "text": "So on that basis, Chat and I can't remember if the free version of Chat GPT has code interpreter.", "timestamp": "09:45" }, { "speaker_name": "Simon", "text": "I hope I think it does.", "timestamp": "09:50" }, { "speaker_name": "Christopher", "text": "Okay. So this is like the basic paid one, maybe the $20 month because I know there's like a $200 one that's a little steep for like a basic", "timestamp": "09:51" } ] }

A spot check of the timestamps showed them in the right place. Gemini 2.5 supports long context prompts so it's possible this works well for much longer audio files - it would be interesting to dig deeper and try that out.

Bounding boxes

One of my favourite features of previous Gemini models is their support for bounding boxes: you can prompt them to return boxes around objects in images.

I built a separate tool for experimenting with this feature in August last year, which I described in Building a tool showing how Gemini Pro can return bounding boxes for objects in images. I've now upgraded that tool to add support the new model.

You can access it at tools.simonwillison.net/gemini-bbox - you'll need to provide your own Gemini API key which is sent directly to their API from your browser (it won't be logged by an intermediary).

I tried it out on a challenging photograph of some pelicans... and it worked extremely well:

My prompt was:

Return bounding boxes around pelicans as JSON arrays [ymin, xmin, ymax, xmax]

The Gemini models are all trained to return bounding boxes scaled between 0 and 100. My tool knows how to convert those back to the same dimensions as the input image.

Here's what the visualized result looked like:

It got almost all of them! I like how it didn't draw a box around the one egret that had made it into the photo.

More characteristics of the model

Here's the official model listing in the Gemini docs. Key details:

Input token limit: 1,000,000 Output token limit: 64,000 - this is a huge upgrade, all of the other listed models have 8,192 for this (correction: Gemini 2.0 Flash Thinking also had a 64,000 output length) Knowledge cut-off: January 2025 - an improvement on Gemini 2.0's August 2024 Gemini 2.5 Pro is a very strong new model

I've hardly scratched the surface when it comes to trying out Gemini 2.5 Pro so far. How's its creative writing? Factual knowledge about the world? Can it write great code in Python, JavaScript, Rust and more?

The Gemini family of models have capabilities that set them apart from other models:

Long context length - Gemini 2.5 Pro supports up to 1 million tokens Audio input - something which few other models support, certainly not at this length and with this level of timestamp accuracy Accurate bounding box detection for image inputs

My experiments so far with these capabilities indicate that Gemini 2.5 Pro really is a very strong new model. I'm looking forward to exploring more of what it can do.

Update: it's very good at code

I spent this evening trying it out for coding tasks, and it's very, very impressive. I'm seeing results for Python that feel comparable to my previous favourite Claude 3.7 Sonnet, and appear to be benefitting from Gemini 2.5 Pro's default reasoning mode and long context.

I've been wanting to add a new content type of "notes" to my blog for quite a while now, but I was put off by the anticipated tedium of all of the different places in the codebase that would need to be updated.

That feature is now live. Here are my notes on creating that notes feature using Gemini 2.5 Pro. It crunched through my entire codebase and figured out all of the places I needed to change - 18 files in total, as you can see in the resulting PR. The whole project took about 45 minutes from start to finish - averaging less than three minutes per file I had to modify.

I've thrown a whole bunch of other coding challenges at it, and the bottleneck on evaluating them has become my own mental capacity to review the resulting code!

Here's another, more complex example. This hasn't resulted in actual running code yet but it took a big bite out of an architectural design problem I've been stewing on for a very long time.

My LLM project needs support for tools - a way to teach different LLMs how to request tool execution, then have those tools (implemented in Python) run and return their results back to the models.

Designing this is really hard, because I need to create an abstraction that works across multiple different model providers, each powered by a different plugin.

Could Gemini 2.5 Pro help unblock me by proposing an architectural approach that might work?

I started by combining the Python and Markdown files for my sqlite-utils, llm, llm-gemini, and llm-anthropic repositories into one big document:

files-to-prompt sqlite-utils llm llm-gemini llm-anthropic -e md -e py -c

Piping it through ttok showed that to be 316,098 tokens (using the OpenAI tokenizer, but the Gemini tokenizer is likely a similar number).

Then I fed that all into Gemini 2.5 Pro with the following prompt:

Provide a detailed design proposal for adding tool calling support to LLM

Different model plugins will have different ways of executing tools, but LLM itself should provide both a Python and a CLI abstraction over these differences.

Tool calling involves passing tool definitions to a model, the model then replies with tools it would like executed, the harness code then executes those and passes the result back to the model and continues in a loop until the model stops requesting tools. This should happen as part of the existing llm.Conversation mechanism.

At the Python layer tools should be provided as Python functions that use type hints, which can then be converted into JSON schema using Pydantic - similar to how the existing schema= mechanism works.

For the CLI option tools will be provided as an option passed to the command, similar to how the sqlite-utils convert CLI mechanism works.'

I've been thinking about this problem for over a year now. Gemini 2.5 Pro's response isn't exactly what I'm going to do, but it did include a flurry of genuinely useful suggestions to help me craft my final approach.

I hadn't thought about the need for asyncio support for tool functions at all, so this idea from Gemini 2.5 Pro was very welcome:

The AsyncConversation.prompt() method will mirror the synchronous API, accepting the tools parameter and handling the tool calling loop using async/await for tool execution if the tool functions themselves are async. If tool functions are synchronous, they will be run in a thread pool executor via asyncio.to_thread.

Watching Gemini crunch through some of the more complex problems I've been dragging my heels on reminded me of something Harper Reed wrote about his workflow with LLMs for code:

My hack to-do list is empty because I built everything. I keep thinking of new things and knocking them out while watching a movie or something.

I've been stressing quite a bit about my backlog of incomplete projects recently. I don't think Gemini 2.5 Pro is quite going to work through all of them while I'm watching TV, but it does feel like it's going to help make a significant dent in them.

Tags: google, ai, generative-ai, llms, gemini, vision-llms, pelican-riding-a-bicycle, llm-release

こんにちは、富士榮です。
どうやらOkta Venturesが2024年から始めた今年のアイデンティティ界の25人（The Identity 25）に選ばれました。

https://www.okta.com/resources/datasheet-the-identity-25/

このプログラム、2024年からスタートしたもののようで、昨年はSPRIN-Dにいる安田クリスチーナやMicrosoftのPam DIngle、YubicoのJohn Bradleyなどが選ばれていました。
今年はOpenID FoundationのExecutive DirectorのGail HodgesやChairの崎村さんらの錚々たるメンバの中に何故か私も加えていただけたようです。
しかし、最初Okta Ventures側から連絡をもらった時はよくある詐欺かと思いましたw いきなりLInked Inで知らない人からCongratulations！でしたから。。。なぜ選ばれたのかは全くわかりませんが、どなたかが推薦していただいたのでしょう。ありがとうございます。光栄です。
しかしこれ、タイムズスクエアのNASDAQのディスプレイにデカデカと顔が出るらしいです。。 ちょっとこれからニューヨークいってきます（違
参考）昨年のクリスチーナの写真

いずれにしろ光栄です。感謝申し上げます。
4/1 追加 タイムズスクエアの写真をもらったので貼っておきます。

AI Engineering is the hottest new engineering field, and it’s also an increasingly loaded phrase which can mean a host of different things to different people. It can refer to ML engineers building AI models, or data scientists and analysts working with large language models (LLMs), or software engineers building on top of LLMs, etc. To make things more confusing, some AI tooling vendors use the term for their products, like Cognition AI naming theirs an “AI engineer.”

In her new book “AI Engineering,” author Chip Huyen defines AI engineering as building applications that use LLMs, placing it between software engineering (building software products using languages, frameworks, and APIs), and ML engineering (building models for applications to use). Overall, AI engineering feels closer to software engineering because it usually starts with software engineers building AI applications using LLM APIs. The more complex the AI engineering use case, the more it can morph into looking like ML engineering.

Today, we focus on software engineers who have switched to being AI engineers. These are devs who work with LLMs, and build features and products on top of them. We cover:

What are companies building? An overview of seven companies at different stages and in various segments of tech.

Onboarding to AI Engineering as a software engineer. Approaches that help devs get up to speed faster.

Tech stack. Seven different tech stacks, showcasing variety and some common choices.

Engineering challenges. Problems caused by LLMs being non-deterministic, evals, latency, privacy, and more.

Novel tooling. Several companies build in-house tooling, which pays off with this fast-moving technology. Incident management startup, incident.io, shares their unique, bespoke stack.

Cost considerations. LLMs can quickly get expensive, although costs are dropping quickly. But as this happens, usage tends to grow which causes its own issues. Businesses share how big a deal costs really are.

Learnings from software devs turned AI engineers. Building a prototype is the easy bit, vibes-based development inevitably turns bad, smart teams fail because they get overwhelmed; educating AI users, and more.

Note: I am an investor in incident.io and Wordsmith, which both share details in this deepdive. I always focus on being unbiased in these articles – but at the same time, being an investor means these companies share otherwise hard-to-obtain details. I was not paid by any business to mention them in this article. See my ethics statement for more.

Thanks to the engineers contributing to this deepdive: Lawrence Jones (senior staff engineer at incident.io), Tillman Elser (AI lead at Sentry), Ross McNairn (cofounder at Wordsmith AI), Igor Ostrovsky (cofounder at Augment Code), Matt Morgis (Staff Engineer, AI, formerly at Elsevier), Ashley Peacock (Staff Engineer at Simply Business), and Ryan Cogswell (Application Architect at DSI).

1. What are companies building? Incident.io: incident note taker and bot

Incident.io is a tool to help resolve outages as they happen. A few months ago, the engineering team went back to the drawing board to build features that capitalize on how much LLMs have improved to date. Two of these features:

AI note taker during incidents. This includes live transcription, real-time summaries for people joining a call to get up to speed with, and key decisions/actions to clarify who does what.

Incident investigator: an agent looks into an ongoing incident by checking code, logs, monitoring, and old incidents to identify root causes and share findings, with a responder being paged. More details on how this tool is built.

Both these features make heavy use of LLMs, and also integrate with several other systems like backend services, Slack, etc.

AI feature: when joining an incident, get a summary of what’s currently being discussed Sentry: Autofix and issue grouping

Sentry is a popular application monitoring software. Two interesting projects they built:

Autofix: make it really fast to go from a problem with code (a Sentry issue) to a fix with a GitHub PR. Autofix is an open source RAG framework that combines all of Sentry’s context/telemetry data with code in order to root cause, fix, and create unit tests for bugs.

Issue Grouping: cut down alerting volume while reducing noise. For this, the team used recent advancements in approximate neighbor search (ANN), plus dramatic recent improvements in embedding quality from the new BERT architecture transformer models.

Both these features are fair source, meaning you can see exactly how they work.

Autofix identifies the root cause of an issue with a useless Stack Trace Wordsmith: legal AI

Wordsmith is building AI tools that legal teams use, including:

Documents workspace: plug into all of a company’s communication streams, including analyzing documents and augmenting their contents, and drafting communications. Check out a video of it in action.

AI contract review: A product that can analyse any contract or website, then review it and generate a marked-up word doc, here. Basically, it’s a lawyer anyone can use.

AI contract review: AI tool detects, highlights and summarizes potential contractual issues Augment Code

Augment Code is an AI coding assistant for engineering teams and large codebases. This kind of product probably needs little introduction for devs:

AI coding assistant: including IDE extensions for VS Code, JetBrains, Cursor, and Vim, and a Slack extension

Fine-tuning models: for AI coding tools, models make a big difference. The team don’t pre-train models or build their own LLMs, but run extensive post-training and do fine tuning to suit the 4-5 models used for specific coding-related cases.

Elsevier: RAG platform

Elsevier is one of the world’s leading publishers of scientific and medical content. Matt Morgis was an engineering manager at the company when the engineering leadership noticed that several product teams were independently implementing RAG capabilities; each sourcing content, parsing it, chunking it, and creating embeddings.

An enterprise-wide RAG platform was the solution Matt and his team built, to enable multiple teams to build AI-powered products for medical and scientific professionals. Their platform consists of:

Database. A content database that centralized and normalized content from various sources.

Embeddings+search: A content embedding & indexing pipeline and vector search API.

LLM API: interfacing to multiple LLM models. This API allows teams to experiment with different models by changing a parameter on the API. It also allowed Elsevier to track the usage of various LLM models based on applications using it.

Products built on top of this platform:

Intelligent tutoring system for medical students

Clinical decisions support tool for doctors

Insurance company: chatbot

Ashley Peacock is a staff software engineer at the insurer, Simply Business, who built a pretty common use case: a chatbot for customers to get answers to questions. This seems like the simplest of use cases – you might assume it just involves connecting documentation for the chatbot to use – but it was surprisingly challenging because:

Industry regulation. The chatbot cannot be inaccurate or make things up, as customers use the information to make decisions about which insurance purchases.

Non-deterministic responses. The business needed to turn a nondeterministic chatbot into one that only produces approved responses.

The team had the idea to create an “approved answers” knowledge base for the chatbot, and faced the challenge of creating the questions for this. The team made the chatbot state when it cannot answer a question, and to then connect with human support, which then updated the knowledge base with their solution. This approach works pretty well after taking a few iterations to get right.

HR SaaS: summarization features

Data Solutions International (DSI) is a 30-person HR tech company with 5 engineers, selling products that help with performance review processes, assessments, and employee engagement surveys. The company is family-owned, has been operating for 27 years, and is profitable.

Summarizing comments for employee engagement processes was the first feature they wanted to build, as something customers would appreciate, and which the team could learn about working with LLMs from.

During an employee engagement process, there are questions like "what do you like most about working here", and "if you could change one thing about working at Company X, what would it be?", etc. For larger companies with thousands of employees, there may be thousands of comments per question. Individual departmental leads might read all comments relevant to them, but there’s no way an HR team at a very large business could check every single comment.

Before LLMs, such comments were categorized into predefined categories, which were hardcoded, per company and per survey. This was okay, but not great. Data Solutions International’s goal was to use LLMs to summarize a large number of comments, and report to survey admins the broad categories which comments belong in, how many comments per category there are, and to allow drilling down into the data.

Summary of teamwork-related comments: The feature builds a word cloud of common topics from thousands of comments, and provides an overview of any term 2. Onboarding to AI Engineering as a software engineer

So, how do you get started building applications with LLMs as a software engineer transitioning to this new field? Some advice from folks at the above companies who have:

You can teach AI yourself – and probably should

Let’s start with an encouraging story from veteran software engineer Ryan Cogswell, at HR tech company, DSI. He joined the company 25 years ago, and was the first engineering hire. When AI tools came along, DSI decided to build a relatively simple first AI feature in their HR system that summarized comments for employee engagement purposes. Neither Ryan nor any of the other 4 devs had expertise in AI and LLMs, so the company contracted an external agency which offered a fixed, timeboxed offer to scope the project. Here’s how it went:

Month #1: the agency goes and builds stuff, and shares LLM outputs with DSI

Month #2: while continuing to iterate on desired outputs, devs at DSI ask the agency for access to how things work. They get access to scripts and notebooks

Month #3: the agency lays out the proposed architecture for the project.

The proposal was for a really complex architecture:

SageMaker: a heavyweight solution from AWS to build, train, and deploy ML models

Langfuse: an open source LLM engineering platform to debug and improve LLM applications

Lambdas: serverless functions to run computations

Database #1: store interim states of data between prompts

Database #2: storing user feedback

Other parts: to support RAG aspects

Pipings: to hold it all together

The agency quoted 6-9 months to build the relatively small feature (!!), and an estimated operational cost higher than DSI’s entire investment in all their infrastructure! This was when Ryan asked how hard it could be to build it themselves, and got to work reading and prototyping, making himself the company’s resident GenAI expert.

In 2 months, Ryan and a couple of colleagues built the feature for a fraction of the cost to operate than the agency’s quote. His tech stack choice:

AWS Bedrock: chosen for cost (vs hosting own models), security, and that the platform doesn’t use their input or output tokens for training

Cohere Embed v3: the model used to generate embeddings

PostgreSQL: to store embeddings and do vector-based database queries, using AWS Aurora PostgreSQL

Java: the backend code runs on this, deployed in AWS

React: the frontend that fetches and displays the data, integrated into the existing web app

Get used to non-deterministic outputs

Ross McNairn is cofounder and CEO of the startup, Wordsmith, which several software engineers have joined. You need to rethink how to think about things, he says:

“Working with AI requires a totally different way of approaching problems.

For new joiner engineers, there is a major readjustment in the first few weeks while they explore the codebase and participate in discussions. There are so many problems that can be eloquently sidestepped using AI. Understanding the suite of tools available takes some time.

Getting comfortable with evaluations and iterating on non-deterministic outputs is the biggest challenge most devs have. A lot of solutions are more subjective: engineers need to really understand the domain in order to assess if output is high-quality. It takes time to ramp up to the point where you can confidently assess output quality.”

Switching to in-house can be easier, even for EMs

Matt Morgis was an engineering manager at Elsevier who decided to transition back to staff engineer, specifically to work on AI:

“The move to go from manager to IC was deliberate: working with AI has rekindled my joy in coding.

For experienced engineers who know how to break problems down, AI tools are an incredible force multiplier. At the same time, when I was a manager, I saw AI coding tools handicap junior engineers’ development. The tools are powerful, but I think they’re best wielded by those who understand good software engineering principles.”

The transition was successful, and today Matt is a staff engineer focusing on GenAI at CVS Health.

3. Tech stack

Here’s the tech stack which various companies used. There’s no right or wrong tech stack – what follows is for context and inspiration, not a blueprint.

Incident.io

The stack:

Postgres and pgvector for storing embeddings and searching them

ChatGPT 4o as the default model, and Sonnet 3.7 for code analysis and technical tasks, as they find this Anthropic model performs better. Built in a way to easily switch between them as needed

Gemini: some models used GCP’s Vertex offering, but less often than other models

GCP using Kubernetes: the infrastructure layer

Go on the backend, running as a monolith

React + Typescript on the frontend, including for the dashboard of their own developers’ custom AI tools (covered below)

Sentry

In-house LLM agent tooling: the team evaluated and rejected using a tool like the LangChain framework to integrate LLMs with other data sources and into workflows. It was a lot more work to build their own, but the upsides are that the architecture and code are more in-line with abstractions and design patterns in Sentry’s existing codebase.

The company used the following languages and frameworks to build this tooling:

Postgres for database and vector store, pgvector for similarity search (for Approximate Nearest Neighbor – ANN – search)

Clickhouse for online analytical processing (OLAP)

Sentry for observability – it would be odd to not use their own product for this

Kubernetes for orchestrating compute resources

Python and PyTorch (machine learning library) for inference

Legal AI startup Wordsmith

The stack:

Pinecone as their vector database

LangChain as their framework to integrate LLMs into the stack. LangSmith as their developer platform

LlamaIndex as orchestration frameworks to integrate data with LLMs

Multi-cloud providers:

AWS for running Anthropic models via Bedrock. AWS offers generous credits for startups, which was a factor in the choice

Azure to access OpenAI services because it allows specifying regions to use, which is important when serving EU customers, for example. Using OpenAI’s services directly would not allow switching of regions.

GCP: for Gemini and Vertex (Google’s search AI).

Azure and GCP each have business models for locking in customers; Microsoft is the only major cloud provider offering OpenAI models, and only GCP offers Gemini.

The company routes to different model by use case:

OpenAI: for reasoning-heavy use cases, where the o1 and o3 models are very strong

Groq: when performance is critical, or the goal is to augment UI. Wordsmith calls their API directly – as the performance of Groq is incredibly fast; a step-change in AI development, according to the WordSmith team. Note: Groq is a standalone company and product, not to be confused with Grok, the AI assistant on social media site, X.

AI coding assistant, Augment Code

The stack to build and run LLMs:

Google Cloud: the cloud vendor of choice

A3 Mega 600GPU/75 node cluster: used for LLM training and inference

NVIDIA: the hardware choice for GPUs, and for software (CUDA)

Python and PyTorch: the team wrote training and inference libraries making heavy use of PyTorch

RAG platform at scientific publisher Elsevier

The scientific publisher used this stack to build their in-house RAG platform:

AWS Bedrock and Azure OpenAI for hosting and running LLMs

LangChain for LLM integration

Snowflake as their content data warehouse

Embedding pipelines and vector database:

Apache Airflow for running embedding pipelines

AWS Fargate for ECS to run containers

AWS OpenSearch as the vector search database

FastAPI (a Python-based web framework to build HTTP APIs) for HTTP APIs

Chatbot for insurance company Simply Business

A pretty simple stack:

AWS Bedrock to host the model, making use of Knowledge Bases and Guardrails features

Anthropic Sonnet 3.5 model

Ruby on Rails as the language and framework, running it on top of AWS ECS

Summarization at HR tech DSI

As covered above, Ryan took the initiative at DSI by building a simpler solution than the one an AI vendor proposed. DSI ended up with:

AWS Bedrock for running the models

PostgreSQL: to store embeddings and do vector-based database queries. Using AWS Aurora PostgreSQL, and Cohere Embed v3 for generating embeddings

Tech stack trends across companies

The seven businesses in this article are all different, but there are some common trends:

AWS Bedrock: the preferred way to host and run Anthropic models

Postgres with pgvector: the database of choice to work with embeddings and vectors at most companies. The exception is Wordsmith, which uses vector database Pinecone

LangChain: a few places use this as the framework to integrate LLMs into their stacks

The bigger the scale, the closer you get to the “metal:” most startups are happy to use cloud providers to run LLMs. However, when starting to get into fine-tuning LLMs and heavy usage, it becomes time to rent larger resources and get close to the hardware. Augment Code using NVIDIA GPUs and CUDA software is an example.

4. Engineering challenges

What unusual or new challenges does AI Engineering pose for more “traditional” software engineers? The most common ones mentioned:

Read more

Also, killing surveillance, finally, maybe. Kaliya lays out some good themes for IIW. My faves: S__olving the identities of AI agents and Proof of Personhood and First Person Credentials.

Unsubscribe now and skip the 7-day free trial. Is there a term of art for Substack newsletters that hide half of what's written behind a tease-wall? (Maybe "teasewall" is it.) Look, I do have paid subscriptions to some newsletters, but I'll never have subscriptions to all of them. So I think from now on I'll just drop every half-newsletter with teases me.

New blogging game: Whack-a-Spam. My Real Agency post got covered nicely here in Ars Technica. Then copies of the same Ars story appeared in a dozen faked-up "news" websites, pinging back to my post. I've been marking them all as spam.

Just a thought. Since the redstream is the new mainstream, I suggest we re-brand the old mainstream (NYTimes, WaPo, Atlantic/Time/New York/New Yorker/Wired/etc.) the bluestream, and the WSJ and other econ-oriented rags the finstream.

Tell your algorithm I sent you. Emily Catalano is now every reel Facebook's algorithm suggests to me. I suppose that's because she's the best deadpan comedian since Steven Wright.

You can't search an unconformity. What Tristan Louis says about digital archives makes complementary points (lots of them, all good) that are consistent with what I said about bits not leaving a fossil record.

What is your DNA worth? 23andMe is filing for bankruptcy. I'm a customer, so it concerns me that they have a heap of data about my DNA. While I'd like the world to benefit from that DNA, should it be useful (and, given some of my odd genetics, e.g. this, it might be), I also like to think that my genetic profile is on loan to 23andMe, and not their property. But, I suspect, in a bankruptcy auction, it's theirs. 

Advice welcome. Are the scanners in printers all the same? Or as good as standalone scanners? The scanner in my Epson printer is no longer visible to my laptop (no idea why), and I'm looking for a replacement printer or printer/scanner.

This blog shows how to implement a delegated Microsoft On-Behalf-Of flow in ASP.NET Core, and has a focus on access token management. The solution uses Microsoft.Identity.Web to implement the different flows and it really simple to implement, when you know how to use the Nuget package and use the correct Microsoft documentation. The application can request delegated access tokens On-Behalf-Of a user and another application, providing a seamless and secure access to protected resources using a zero trust strategy.

Code: https://github.com/damienbod/token-mgmt-ui-delegated-obo-entra

Blogs in this series ASP.NET Core user delegated access token management ASP.NET Core user application access token management ASP.NET Core delegated OAuth token exchange access token management ASP.NET Core delegated Microsoft OBO access token management (Entra only) Setup

Three applications can used in this setup. A web UI application, an API and another API which implements the Microsoft On-Behalf-Of flow for the users delegated access token and the application. The Microsoft OBO works very like part of the OAuth token exchange standard, but it is not a standard, just a Microsoft flavor for a standard.

What must an application manage?

An access token management solution must ensure that tokens are securely stored per user session for delegated downstream API user tokens and updated after each UI authentication or refresh. The solution should be robust to handle token expiration, function seamlessly after restarts, and support multi-instance deployments. The tokens must be persisted safely in multiple instance setups. Additionally, it must effectively manage scenarios involving invalid or missing access tokens. Microsoft.Identity.Web implements this completely as long as as authentication and OAuth flows are implemented using Entra ID.

Properties of token management in the solution setup: The access token is persisted per user session The token expires The token needs to be persisted somewhere safely (Safe and encrypted storage if not in-memory) The token must be replaced after each UI authentication (per user) The solution must work after restarts The solution must work for multiple instances when deployed to multi-instance deployments. The solution must handle invalid access tokens or missing access tokens The application must handle a user logout Web UI

The first step in the Microsoft On-Behalf-Of flow is to authenticate the user and a web application using Entra ID. This is implemented using the Microsoft.Identity.Web Nuget package. The Web application uses OpenID Connect code flow with PKCE and a confidential client. The application requests an access token for the first API. The access token is a delegated access token issued for the user and the specific API. The implementation uses a secret to assert the application. Microsoft recommends using a certificate and client assertions when deploying to production.

builder.Services.AddHttpClient(); builder.Services.AddOptions(); string[]? initialScopes = builder.Configuration .GetValue<string>("WebApiEntraId:ScopeForAccessToken")? .Split(' '); builder.Services.AddDistributedMemoryCache(); builder.Services .AddMicrosoftIdentityWebAppAuthentication(builder.Configuration, "EntraID", subscribeToOpenIdConnectMiddlewareDiagnosticsEvents: true) .EnableTokenAcquisitionToCallDownstreamApi(initialScopes) .AddDistributedTokenCaches(); builder.Services .AddAuthorization(options => { options.FallbackPolicy = options.DefaultPolicy; }); builder.Services.AddRazorPages() .AddMvcOptions(options => { var policy = new AuthorizationPolicyBuilder() .RequireAuthenticatedUser() .Build(); options.Filters.Add(new AuthorizeFilter(policy)); }).AddMicrosoftIdentityUI(); builder.Services.AddServerSideBlazor() .AddMicrosoftIdentityConsentHandler();

The WebApiEntraIdService class is used to use the access token from the web application and call the downstream API. If the access token is missing, or invalid, an new access token is requested in the application.

using Microsoft.Extensions.Configuration; using Microsoft.Identity.Web; using System; using System.Net.Http; using System.Net.Http.Headers; using System.Net.Http.Json; using System.Threading.Tasks; namespace RazorPageEntraId.WebApiEntraId; public class WebApiEntraIdService { private readonly IHttpClientFactory _clientFactory; private readonly ITokenAcquisition _tokenAcquisition; private readonly IConfiguration _configuration; public WebApiEntraIdService(IHttpClientFactory clientFactory, ITokenAcquisition tokenAcquisition, IConfiguration configuration) { _clientFactory = clientFactory; _tokenAcquisition = tokenAcquisition; _configuration = configuration; } public async Task<string?> GetWebApiEntraIdDataAsync() { var client = _clientFactory.CreateClient(); var scope = _configuration["WebApiEntraID:ScopeForAccessToken"]; var accessToken = await _tokenAcquisition.GetAccessTokenForUserAsync([scope!]); client.BaseAddress = new Uri(_configuration["WebApiEntraID:ApiBaseAddress"]!); client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", accessToken); client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json")); var response = await client.GetAsync("/api/profiles/photo"); if (response.IsSuccessStatusCode) { var responseContent = await response.Content.ReadFromJsonAsync<string>(); return responseContent; } throw new ApplicationException($"Status code: {response.StatusCode}, Error: {response.ReasonPhrase}"); } }

Web API using On-Behalf-Of Flow

The first Web API implements the Microsoft On-Behalf-Of flow to acquire a new access token for the existing access token and the user represented in the access token. The access token is a delegated access token. The API has no UI and does not use any UI flows. If the access token used in the request is invalid, a 401 is returned with an exception information on what permission or access token is required to use the API. If the API is requested using a valid access token, the API application uses the default scope and requests a new access token using a secret or a certificate. The new access token can be used to access the downstream API.

builder.Services.AddTransient<WebApiDownstreamService>(); builder.Services.AddHttpClient(); builder.Services.AddOptions(); builder.Services.AddDistributedMemoryCache(); builder.Services .AddMicrosoftIdentityWebApiAuthentication( builder.Configuration, "EntraID") .EnableTokenAcquisitionToCallDownstreamApi() .AddDistributedTokenCaches();

using Microsoft.Identity.Web; using System.Net.Http.Headers; using System.Text.Json; namespace WebApiEntraIdObo.WebApiEntraId; public class WebApiDownstreamService { private readonly IHttpClientFactory _clientFactory; private readonly ITokenAcquisition _tokenAcquisition; private readonly IConfiguration _configuration; public WebApiDownstreamService(IHttpClientFactory clientFactory, ITokenAcquisition tokenAcquisition, IConfiguration configuration) { _clientFactory = clientFactory; _tokenAcquisition = tokenAcquisition; _configuration = configuration; } public async Task<string?> GetApiDataAsync() { var client = _clientFactory.CreateClient(); // user_impersonation access_as_user access_as_application .default var scope = _configuration["WebApiEntraIdObo:ScopeForAccessToken"]; if (scope == null) throw new ArgumentNullException(nameof(scope)); var uri = _configuration["WebApiEntraIdObo:ApiBaseAddress"]; if (uri == null) throw new ArgumentNullException(nameof(uri)); var accessToken = await _tokenAcquisition .GetAccessTokenForUserAsync([scope]); client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", accessToken); client.BaseAddress = new Uri(uri); client.DefaultRequestHeaders.Accept.Add( new MediaTypeWithQualityHeaderValue("application/json")); var response = await client.GetAsync("api/profiles/photo"); if (response.IsSuccessStatusCode) { var data = await JsonSerializer.DeserializeAsync<string>( await response.Content.ReadAsStreamAsync()); return data; } throw new ApplicationException($"Status code: {response.StatusCode}, Error: {response.ReasonPhrase}"); } }

Web API

The downstream API validates the request API using standard JWT validation.

builder.Services.AddControllers(options => { var policy = new AuthorizationPolicyBuilder() .RequireAuthenticatedUser() // .RequireClaim("email") // disabled this to test with users that have no email (no license added) .Build(); options.Filters.Add(new AuthorizeFilter(policy)); }); builder.Services.AddHttpClient(); builder.Services.AddOptions(); builder.Services.AddMicrosoftIdentityWebApiAuthentication( builder.Configuration, "EntraID");

Running the applications

When the applications are started, the data from the downstream APIs is returned to the web application.

Further examples of the Microsoft On-Behalf-Of flow

Microsoft authentication authorization libraries are complicated and many. They is no one way to implement this. Microsoft provides Microsoft Graph Nuget packages, Azure SDK packages, mixes application and delegation flows, managed identities solutions, direct token acquisition and some legacy Nuget packages to integrate the security. Here are further examples of using the Microsoft On-Behalf-Of flow using different client solutions.

Microsoft OBO with Azure Blob Storage (delegated)

ASP.NET Core Razor page using Azure Blob Storage to upload download files securely using OAuth and Open ID Connect

https://github.com/damienbod/AspNetCoreEntraIdBlobStorage

Microsoft OBO with OpenIddict (delegated)

This demo shows how to implement the On-Behalf-Of flow between an Microsoft Entra ID protected API and an API protected using OpenIddict.

https://github.com/damienbod/OnBehalfFlowOidcDownstreamApi

ASP.NET Core OBO using Microsoft Graph (delegated)

Backend for frontend security using Angular Standalone (nx) and ASP.NET Core backend using Microsoft Graph

https://github.com/damienbod/bff-aspnetcore-angular

Note

The Microsoft OBO flow is only used for integrating with Microsoft Entra. If using any other identity provider, the OAuth token exchange flow should be used for this type of solution.

Links

Using Blob storage from ASP.NET Core with Entra ID authentication

Implement the OAUTH 2.0 Token Exchange delegated flow between an Microsoft Entra ID API and an API protected using OpenIddict

https://github.com/damienbod/OAuthGrantExchangeOidcDownstreamApi

https://docs.duendesoftware.com/identityserver/v7/tokens/extension_grants/token_exchange/

https://datatracker.ietf.org/doc/html/rfc8693

https://github.com/damienbod/OnBehalfFlowOidcDownstreamApi

https://www.rfc-editor.org/rfc/rfc6749#section-5.2

https://github.com/blowdart/idunno.Authentication/tree/dev/src/idunno.Authentication.Basic

https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-on-behalf-of-flow

Standards

JSON Web Token (JWT)

Best Current Practice for OAuth 2.0 Security

The OAuth 2.0 Authorization Framework

OAuth 2.0 Demonstrating Proof of Possession DPoP

OAuth 2.0 JWT-Secured Authorization Request (JAR) RFC 9101

OAuth 2.0 Mutual-TLS Client Authentication and Certificate-Bound Access Tokens

OpenID Connect 1.0

Microsoft identity platform and OAuth 2.0 On-Behalf-Of flow

OAuth 2.0 Token Exchange

JSON Web Token (JWT) Profile for OAuth 2.0 Access Tokens

HTTP Semantics RFC 9110

Today we’re excited to launch ARC-AGI-2 to challenge the new frontier. ARC-AGI-2 is even harder for AI (in particular, AI reasoning systems), while maintaining the same relative ease for humans. Pure LLMs score 0% on ARC-AGI-2, and public AI reasoning systems achieve only single-digit percentage scores. In contrast, every task in ARC-AGI-2 has been solved by at least 2 humans in under 2 attempts. [...]

All other AI benchmarks focus on superhuman capabilities or specialized knowledge by testing "PhD++" skills. ARC-AGI is the only benchmark that takes the opposite design choice – by focusing on tasks that are relatively easy for humans, yet hard, or impossible, for AI, we shine a spotlight on capability gaps that do not spontaneously emerge from "scaling up".

— Greg Kamradt, ARC-AGI-2

Tags: evals, ai

shot-scraper 1.8

I've added a new feature to shot-scraper that makes it easier to share scripts for other people to use with the shot-scraper javascript command.

shot-scraper javascript lets you load up a web page in an invisible Chrome browser (via Playwright), execute some JavaScript against that page and output the results to your terminal. It's a fun way of running complex screen-scraping routines as part of a terminal session, or even chained together with other commands using pipes.

The -i/--input option lets you load that JavaScript from a file on disk - but now you can also use a gh: prefix to specify loading code from GitHub instead.

To quote the release notes:

shot-scraper javascript can now optionally load scripts hosted on GitHub via the new gh: prefix to the shot-scraper javascript -i/--input option. #173

Scripts can be referenced as gh:username/repo/path/to/script.js or, if the GitHub user has created a dedicated shot-scraper-scripts repository and placed scripts in the root of it, using gh:username/name-of-script.

For example, to run this readability.js script against any web page you can use the following:

shot-scraper javascript --input gh:simonw/readability \ https://simonwillison.net/2025/Mar/24/qwen25-vl-32b/

The output from that example starts like this:

{ "title": "Qwen2.5-VL-32B: Smarter and Lighter", "byline": "Simon Willison", "dir": null, "lang": "en-gb", "content": "<div id=\"readability-page-1\"...

My simonw/shot-scraper-scripts repo only has that one file in it so far, but I'm looking forward to growing that collection and hopefully seeing other people create and share their own shot-scraper-scripts repos as well.

This feature is an imitation of a similar feature that's coming in the next release of LLM.

Tags: playwright, shot-scraper, scraping, javascript, projects, github, annotated-release-notes

microsoft/playwright-mcp

The Playwright team at Microsoft have released an MCP (Model Context Protocol) server wrapping Playwright, and it's pretty fascinating.

They implemented it on top of the Chrome accessibility tree, so MCP clients (such as the Claude Desktop app) can use it to drive an automated browser and use the accessibility tree to read and navigate pages that they visit.

Trying it out is quite easy if you have Claude Desktop and Node.js installed already. Edit your claude_desktop_config.json file:

code ~/Library/Application\ Support/Claude/claude_desktop_config.json

And add this:

{ "mcpServers": { "playwright": { "command": "npx", "args": [ "@playwright/mcp@latest" ] } } }

Now when you launch Claude Desktop various new browser automation tools will be available to it, and you can tell Claude to navigate to a website and interact with it.

I ran the following to get a list of the available tools:

cd /tmp git clone https://github.com/microsoft/playwright-mcp cd playwright-mcp/src/tools files-to-prompt . | llm -m claude-3.7-sonnet \ 'Output a detailed description of these tools'

The full output is here, but here's the truncated tool list:

Navigation Tools (common.ts) browser_navigate: Navigate to a specific URL browser_go_back: Navigate back in browser history browser_go_forward: Navigate forward in browser history browser_wait: Wait for a specified time in seconds browser_press_key: Press a keyboard key browser_save_as_pdf: Save current page as PDF browser_close: Close the current page Screenshot and Mouse Tools (screenshot.ts) browser_screenshot: Take a screenshot of the current page browser_move_mouse: Move mouse to specific coordinates browser_click (coordinate-based): Click at specific x,y coordinates browser_drag (coordinate-based): Drag mouse from one position to another browser_type (keyboard): Type text and optionally submit Accessibility Snapshot Tools (snapshot.ts) browser_snapshot: Capture accessibility structure of the page browser_click (element-based): Click on a specific element using accessibility reference browser_drag (element-based): Drag between two elements browser_hover: Hover over an element browser_type (element-based): Type text into a specific element

Tags: playwright, anthropic, claude, generative-ai, llm-tool-use, ai, llms, model-context-protocol

We are heading into the 40th Internet Identity Workshop, an event that I cofounded with Doc Searls and Phil Windley 20 years ago. I am not sure we ever thought it would last this long but it has. We continue to be the world’s leading innovation forum for decentralized digital identity focused on a range […]

The post Key themes going into IIW XL (40)  appeared first on Identity Woman.

Qwen2.5-VL-32B: Smarter and Lighter

The second big open weight LLM release from China today - the first being DeepSeek v3-0324.

Qwen's previous vision model was Qwen2.5 VL, released in January in 3B, 7B and 72B sizes.

Today's Apache 2.0 licensed release is a 32B model, which is quickly becoming my personal favourite model size - large enough to have GPT-4-class capabilities, but small enough that on my 64GB Mac there's still enough RAM for me to run other memory-hungry applications like Firefox and VS Code.

Qwen claim that the new model (when compared to their previous 2.5 VL family) can "align more closely with human preferences", is better at "mathematical reasoning" and provides "enhanced accuracy and detailed analysis in tasks such as image parsing, content recognition, and visual logic deduction".

They also offer some presumably carefully selected benchmark results showing it out-performing Gemma 3-27B, Mistral Small 3.1 24B and GPT-4o-0513 (there have been two more recent GPT-4o releases since that one, 2024-08-16 and 2024-11-20).

As usual, Prince Canuma had MLX versions of the models live within hours of the release, in 4 bit, 6 bit, 8 bit, and bf16 variants.

I ran the 4bit version (a 18GB model download) using uv and Prince's mlx-vlm like this:

uv run --with 'numpy<2' --with mlx-vlm \ python -m mlx_vlm.generate \ --model mlx-community/Qwen2.5-VL-32B-Instruct-4bit \ --max-tokens 1000 \ --temperature 0.0 \ --prompt "Describe this image." \ --image Mpaboundrycdfw-1.png

Here's the image:

And here's the resulting description, which I found to be very impressive:

This image appears to be a map highlighting specific areas along the coastline, likely in California, based on the geographical features and place names. Here is a detailed description of the image:

Geographic Features:

The map shows a section of the coastline, with land on the right and the sea on the left. The land area includes parts of SanMateo County, as noted in the top-right corner. The coastline features several named locations, including: Point Montara Pillar Point Kings Rock VicenteCreek DennistonCreek State Beach

Highlighted Areas:

The map contains two distinct areas marked with different colors: Red Area (Top Section) Labeled as "MONTARA" and "MONTARA M R" (possibly referring to a "Marine Reserves" or similar protected area). This area is marked with a dark red color and extends from the coastline inland. Blue Area (Bottom Section) Labeled as "PILLAR" and "PILLAR P O I N T" along with "SMCA" (Special Management Area). This area is marked with a blue color and is located closer to the coastline, including the area around "Pillar Point."

Depth Contours:

The map includes depth markings in the sea, indicating the bathymetrical features: 20 fathom (fm) line is marked in the upper-left part of the sea. 10 fathom (fm) line is marked closer to the coastline. 30 fathom (fm) line is marked further out to sea.

Other Features:

State Beach: Marked near the top-right, indicating a protected recreational area. Kings Rock: A prominent feature near the coastline, likely a rocky outcropping. Creeks: The map shows several creeks, including VicenteCreek and DennistonCreek, which flow into the sea.

Protected Areas:

The map highlights specific protected areas: Marine Reserves: "MONTARA M R" (Marine Reserves) in red. Special Management Area (SMCA) "PILLAR P O I N T" in blue, indicating a Special Management Area.

Grid and Coordinates:

The map includes a grid with latitude and longitude markings: Latitude ranges from approximately 37°25'N to 37°35'N. Longitude ranges from approximately 122°22.5'W to 122°35.5'W.

Topography:

The land area shows topographic features, including elevations and vegetation, with green areas indicating higher elevations or vegetated land.

Other Labels:

"SMR": Likely stands for "State Managed Reserves." "SMCA": Likely stands for "Special Management Control Area."

In summary, this map highlights specific protected areas along the coastline, including a red "Marine Reserves" area and a blue "Special Management Area" near "Pillar Point." The map also includes depth markings, geographical features, and place names, providing a detailed view of the region's natural and protected areas.

It included the following runtime statistics:

Prompt: 1051 tokens, 111.985 tokens-per-sec Generation: 760 tokens, 17.328 tokens-per-sec Peak memory: 21.110 GB

Tags: vision-llms, llm-release, generative-ai, ai, qwen, llms, uv, mlx, prince-canuma

This Wednesday, March 26 at noon America/New_York (16:00 UTC), Peter Eisentraut has graciously agreed to give a talk at the Extension Mini Summit #2 on the extension search path patch he recently committed to PostgreSQL. I’m personally stoked for this topic, as freeing extensions from the legacy of a single directory opens up a number of new patterns for packaging, installation, and testing extensions. Hit the Meetup to register for this live video conference, and to brainstorm novel uses for this new feature, expected to debut in PostgreSQL 18.

More about… Postgres Extensions PGConf Summit Search Path Peter Eisentraut

Back on March 12, we hosted the first in a series of PostgreSQL Extensions Mini Summits leading up to the Extension Ecosystem Summit at PGConf.dev on May 13. I once again inaugurated the series with a short talk on the State of the Extension Ecosystem. The talk was followed by 15 minutes or so of discussion. Here are the relevant links:

Video OCI Demo Slides

And now, with many thanks to Floor Drees for the effort, the transcript from the session.

Introduction

Floor Drees introduced the organizers:

David Wheeler, Principal Architect at Tembo, maintainer of PGXN Yurii Rashkovskii, Omnigres Keith Fiske, Crunchy Data Floor Drees, Principal Program Manager at EDB, PostgreSQL CoCC member, PGDay Lowlands organizer

David presented a State of the Extension Ecosystem at this first event, and shared some updates from PGXN land.

The stream and the closed captions available for the recording are supported by PGConf.dev and their gold level sponsors, Google, AWS, Huawei, Microsoft, and EDB.

State of the Extensions Ecosystem

So I wanted to give a brief update on the state of the Postgres extension ecosystem, the past, present, and future. Let’s give a brie history; it’s quite long, actually.

There were originally two approaches back in the day. You could use shared preload libraries to have it preload dynamic shareable libraries into the main process. And then you could do pure SQL stuff using, including procedural languages like PL/Perl, PL/Tcl, and such.

And there were a few intrepid early adopters, including PostGIS, BioPostgres, PL/R, PL/Proxy, and pgTAP, who all made it work. Beginning of Postgres 9.1 Dimitri Fontaine added support for explicit support for extensions in the Postgres core itself. The key features included the ability to compile and install extensions. This is again, pure SQL and shared libraries.

There are CREATE, UPDATE, and DROP EXTENSION commands in SQL that you can use to add extensions to a database, upgrade them to new versions and to remove them. And then pg_dump and pg_restore support so that extensions could be considered a single bundle to be backed up and restored with all of their individual objects being included as part of the backup.

Back then, a number of us, myself included, saw this as an opportunity to have the extensibility of Postgres itself be a fundamental part of the community and distribution. I was a long time user of Perl and used CPAN, and I thought we had something like CPAN for Postgres. So, I proposed PGXN, the PostgreSQL Extension Network, back in 2010. The idea was to do distribution of source code. You would register namespaces for your extensions.

There was discovery via a website for search, documentation published, tags to help you find different kinds of objects, and to support installation through a command line interface. The compile and install stuff that Postgres itself provides, using PGXS and Configure.

This is what PGXN looks like today. It was launched in 2011. There’s a command line client, this website, an API an a registry you can upload your extensions to. The most recent one was pg_task a day or so ago.

In the interim, since that came out in 2011/2012, the cloud providers have come into their own with Postgres, but their support for extensions tends to be rather limited. For non-core extension counts, as of yesterday, Azure provides 38 extensions, GCP provides 44 extensions, and AWS 51. These are the third party extensions that don’t come with Postgres and its contrib itself. Meanwhile, PGXN has 420 extensions available to download, compile, build, and install.

A GitHub project that tracks random extensions on the internet, (joelonsql/PostgreSQL-EXTENSIONs.md), which is pretty comprehensive, has almost 1200 extensions listed. So the question is why is the support not more broad? Why aren’t there a thousand extensions available in every one of these systems?

Rthis has been a fairly common question that’s come up in the last couple years. A number of new projects have tired to fill in the gaps. One is Trusted Language Extensions. They wanted to make it easier to distribute extensions without needing dynamic shared libraries by adding additional features in the database itself.

The idea was to empower app developers to make it easy to install extensions via SQL functions rather than having to access the file system of the database server system itself. It can be portable, so there’s no compilation required, it hooks into the create extension command transparently, supports custom data types, and there have been plans for foreign data wrappers and background workers. I’m not sure how that’s progressed in the past year. The pg_tle extension itself was created by AWS and Supabase.

Another recent entrant in tooling for extensions is pgrx, which is native Rust extensions in Postgres. You build dynamic shared libraries, but write them in pure Rust. The API for pgrx provides full access to Postgres features, and still provides the developer-friendly tooling that Rust developers are used to. There’s been a lot of community excitement the last couple of years around pgrx, and it remains under active development — version 0.13.0 just came out a week or so ago. It’s sponsored and run out of the PgCentral Foundation.

There have also been a several new registries that have come up to try to fill the gap and make extensions available. They have emphasized different things than PGXN. One was ease of use. So, for example, here pgxman says it should be really easy to install a client in a single command, and then it installs something, and then it downloads and installs a binary version of your an extension.

And then there was platform neutrality. They wanted to do binary distribution and support multiple different platform, to know what binary∑ to install for a given platform. They provide stats. PGXN doesn’t provide any stats, but some of them are list stats like how many downloads we had, how many in the last 180 days.

And curation. Trunk is another binary extension registry, from my employer, Tembo. They do categorization of all the extensions on Trunk, which is at 237 now. Quite a few people have come forward to tells us that they don’t necessarily use Trunk to install extensions, but use them to find them, because the categories are really helpful for people to figure out what sorts of things are even available, and an option to use.

So here’s the State of the Ecosystem as I see it today.

There have been some lost opportunities from the initial excitement around 2010. Extensions remain difficult to find and discover. Some are on PGXN, some are on GitHub, some are on Trunk, some are on GitLab, etc. There’s no like one place to go to find them all.

They remain under-documented and difficult to understand. It takes effort for developers to write documentation for their extensions, and a lot of them aren’t able to. Some of them do write the documentation, but they might be in a format that something like PGXN doesn’t understand.

The maturity of extensions can be difficult to gauge. If you look at that list of 1200 extensions on GitHub, which ones are the good ones? Which ones do people care about? That page in particular show the number of stars for each extension, but that the only metric.

They’re difficult to configure and install. This is something TLE really tried to solve, but the uptake on TLE has not been great so far, and it doesn’t support all the use cases. There are a lot of use cases that need to be able to access the internal APIs of Postgres itself, which means compiling stuff into shared libraries, and writing them in C or Rust or a couple of other compiled languages.

That makes them difficult to configure. You have ask questions lik: Which build system do I use? Do I install the tooling? How do I install it and configure it? What dependencies does it have? Et cetera.

There’s no comprehensive binary packaging. The Postgres community’s own packaging systems for Linux — Apt, and YUM — do a remarkably good job of packaging extensions. They probably have more extensions packaged for those platforms than any of the others. If they have the extension you need and you’re using the PGDG repositories, then this stuff is there. But even those are still like a fraction of all the potential available extensions that are out there.

Dependency management can be pretty painful. It’s difficult to know what you need to install. I was messing around yesterday with the PgSQL HTTP extension, which is a great extension that depends on libcurl. I thought maybe I could build a package that includes libcurl as part of it. But then I realized that libcurl depends on other packages, other dynamic libraries. So I’d have to figure out what all those are to get them all together.

A lot of that goes away if you use a system like apt or yum. But if you, if you don’t, or you just want to install stuff on your Mac or Windows, it’s much more difficult.

Centralized source distribution, we’ve found found, is insufficient. Even if all the extensions were available on PGXN, not everybody has the wherewithal or the expertise to find what they need, download it, compile it, and build it. Moreover, you don’t want to have a compiler on your production system, so you don’t want to be building stuff from source on your production system. So then you have to get to the business of building your own packages, which is a whole thing.

But in this state of the extension ecosystem we see new opportunities too. One I’ve been working on for the past year, which we call “PGXN v2”, is made possible by my employer, Tembo. The idea was to consider the emerging patterns — new registries and new ways of building and releasing and developing extensions — and to figure out the deficiencies, and to engage deeply with the community to work up potential solutions, and to design and implement a new architecture. The idea is to serve the community for the next decade really make a PGXN and its infrastructure the source of record for extensions for Postgres.

In the past year, I did a bunch of design work on it. Here’s a high level architectural view. We’d have a root registry, which is still the source code distribution stuff. There’s a web UX over it that would evolve from the current website. And there’s a command line client that knows how to build extensions from the registry.

But in addition to those three parts, which we have today, we would evolve a couple of additional parts.

One is “interactions”, so that when somebody releases a new extension on PGXN, some notifications could go out through webhooks or some sort of queue so that downstream systems like the packaging systems could know something new has come out and maybe automate building and updating their packages.

There could be “stats and reports”, so we can provide data like how many downloads there are, what binary registries make them available, what kinds of reviews and quality metrics rate them. We can develop these stats and display those on the website.

And, ideally, a “packaging registry” for PGXN to provide binary packages for all the major platforms of all the extensions we can, to simplify the installation of extensions for anybody who needs to use them. For extensions that aren’t available through PGDG or if you’re not using that system and you want to install extensions. Late last year, I was focused on figuring out how t build the packaging system.

Another change that went down in the past year was the Extension Ecosystem Summit itself. This took place at PGConf.Dev last May. The idea was for a community of people to come together to collaborate, examine ongoing work in the extension distribution, examine challenges, identify questions, propose solutions, and agree on directions for execution. Let’s take a look at the topics that we covered last year at the summit.

One was extension metadata, where the topics covered included packaging and discoverability, extension development, compatibility and taxonomies as being important to represent a metadata about extensions — as well as versioning standards. One of the outcomes was an RFC for version two of the PGXN metadata that incorporates a lot of those needs into a new metadata format to describe extensions more broadly.

Another topic was the binary distribution format and what it should look like, if we were to have major, distribution format. We talked about being able to support multiple versions of an extension at one time. There was some talk about the Python Wheel format as a potential precedent for binary distribution of code.

There’s also an idea to distribute extensions through Docker containers, also known as the Open Container Initiative. Versioning came up here, as well. One of the outcomes from this session was another PGXN RFC for binary distribution, which was inspired by Python Wheel among other stuff.

I wanted to give a brief demo build on that format. I hacked some changes into the PGXS Makefile to add a new target, trunk that builds a binary package called a “trunk” and uploads it to an OCI registry for distribution. Here’s what it looks like.

On my Mac I was compiling my semver extension. Then I go into a Linux container and compile it again for Linux using the make trunk command. The result is two .trunk files, one for Postgres 16 on Darwin and one for Postgres 16 on Linux.

There are also some JSON files that are annotations specifically for OCI. We have a command where we can push these images to an OCI registry.

Then we can then use an install command that knows to download and install the version of the build appropriate for this platform (macOS). And then I go into Linux and do the same thing. It also knows, because of the OCI standard, what the platform is, and so it installs the appropriate binary.

Another topic was ABI and API compatibility. There was some talk at the Summit about what is the definition of an ABI and an API and how do we define internal APIs and their use? Maybe there’s some way to categorize APIs in Postgres core for red, green, or in-between, something like that. There was desire to have more hooks available into different parts of the system.

One of the outcomes of this session was that I worked with Peter Eisentraut on some stability guidance for the API and ABI that is now committed in the docs. You can read them now on in the developer docs, they’ll be part of the Postgres 18 release. The idea is that minor version releases should be safe to use with other minor versions. If you compiled your extension against one minor version, it should be perfectly compatible with other minor versions of the same major release.

Interestingly, there was a release earlier this year, like two weeks after Peter committed this, where there was an API break. It’s the first time in like 10 years. Robert Treat and I spent quite a bit of time trying to look for a previous time that happened. I think there was one about 10 years ago, but then this one happened and, notably it broke the Timescale database. The Core Team decided to release a fix just a week later to restore the ABI compatibility.

So it’s clear that even though there’s guidance, you should in general be able to rely on it, and it was a motivating factor for the a new release to fix an ABI break, there are no guarantees.

Another thing that might happen is that I proposed a Google Summer of Code project to build an ABI checker service. Peter [embarrassing forgetfulness and misattributed national identity omitted] Geoghegan POC’d an ABI checker in 2023. The project is to take Peter’s POC and build something that could potentially run on every commit or push to the back branches of the project. Maybe it could be integrated into the build farm so that, if there’s a back-patch to an earlier branch and it turns red, they quickly the ABI was broken. This change could potentially provide a higher level of guarantee — even if they don’t end up using the word “guarantee” about the stability of the ABIs and APIs. I’m hoping this happens; a number of people have asked about it, and at least one person has written an application.

Another topic at the summit last year was including or excluding extensions in core. They’ve talked about when to add something to core, when to remove something from core, whether items in contrib should actually be moved into core itself, and whether to move metadata about extensions into catalog. And once again, support for multiple versions came up; this is a perennial challenge! But I’m not aware of much work on these questions. I’m wondering if it’s time for a revisit,

As a bonus item — this wasn’t a formal topic at the summit last year, but it came up many times in the mini-summits — is the challenge of packaging and lookup. There’s only one path to extensions in SHAREDIR. This creates a number of difficulties. Christoph Berg has a patch for a PGDG and Debian that adds a second directory. This allowed the PGDG stuff to actually run tests against extensions without changing the core installation of the Postgres service itself. Another one is Cloud Native Postgres immutability. If that directory is part of the image, for your CloudNative Postgres, you can’t install extensions into it.

It’s a similar issue, for Postgres.app immutability. Postgres.app is a Mac app, and it’s signed by a certificate provided by Apple. But that means that if you install an extension in its SHAREDIR, it changes the signature of the application and it won’t start. They work around this issue through a number of symlink shenanigans, but these issues could be solved by allowing extension to be installed in multiple locations.

Starting with Christoph’s search path patch and a number of discussions we had at PGConf last year, Peter Eisentraut has been working on a search path patch to the core that would work similar to shared preload libraries, but it’s for finding extension control files. This would allow you to have them in multiple directories and it will find them in path.

Another interesting development in this line has been, the CloudNativePG project has been using that extension search path patch to prototype a new feature coming to Kubernetes that allows one to mount a volume that’s actually another Docker image. If you have your extension distributed as an OCI image, you can specify that it be mounted and installed via your CNPG cluster configuration. That means when CNPG spins up, it puts the extension in the right place. It updates the search path variables and stuff just works.

A lot of the thought about the stuff went into a less formal RFC I wrote up in my blog, rather than on PGXN. The idea is to take these improvements and try to more formally specify the organization of extensions separate from how Postgres organizes shared libraries and shared files.

I said, we’re bringing the Extension Summit back! There will be another Extension Summit hosted our team of organizers, myself, Floor, Keith Fiske from Crunchy Data, and Yurii from Omnigres. That will be on May 13th in the morning at PGConf.dev; we appreciate their support.

The idea of these Mini Summits is to bring up a number of topics of interest. Have somebody come and do a 20 or 40 minute talk about it, and then we can have discussion about implications.

Floor mentioned the schedule, but briefly:

March 12: David Wheeler, PGXN: “State of the Extension Ecosystem” March 24: Peter Eisentraut, Core Team: “Implementing an Extension Search Path” April 9: Christoph Berg, Debian: “Apt Extension Packaging” April 23: May 7: Gabriele Bartolini, CNPG “Extension Management in CloudNativePG”

So, what are your interests in extensions and how they can be improved. There are a lot of potential topics to talk about at the Summit or at these Mini Summits: development tools, canonical registry, how easy it is to publish, continuous delivery, yada, yada, yada, security scanning — all sorts of stuff that could go into conceiving, designing, developing, distributing extensions for Postgres.

I hoe you all will participate. I appreciate you taking the time to listen to me for half an hour. So I’d like to turn it over to, discussion, if people would like to join in, talk about implications of stuff. Also, we can get to any questions here.

Questions, comments, shout-outs

Floor: David, at one point you talked about, metadata taxonomy. If you can elaborate on that a little bit, that’s Peter’s question.

David: So one that people told me that they found useful was one provided by Trunk. So it has these limited number of categories, so if you’re interested in machine learning stuff, you could go to the machine learning stuff and it shows you what extensions are potentially available. They have 237 extensions on Trunk now.

PGXN itself allows arbitrary tagging of stuff. It builds this little tag cloud. But if I look at this one here, you can see this one has a bunch of tags. These are arbitrary tags that are applied by the author. The current metadata looks like this. It’s just plain JSON, and it has a list of tags. The PGXN Meta v2 RFC has a bunch of examples. It’s an evolution of that META.json, so the idea is to have a classifications that includes tags as before, but also adds categories, which are a limited list that would be controlled by the core [he means “root”] registry:

{ "classifications": { "tags": [ "testing", "pair", "parameter" ], "categories": [ "Machine Learning" ] } } Announcements

Yurii made a number of announcements, summarizing:

There is a new library that they’ve been developing at Omnigres that allows you to develop Postgres extensions in C++. For people who are interested in developing extensions in C++ and gaining the benefits of that and not having to do all the tedious things that we have to do with C extensions: look for Cppgres. Yurii thinks that within a couple of months it will reach parity with pgrx.

David: So it sounds like it would work more closely to the way PGXS and C works. Whereas pgrx has all these additional Rust crates you have to load and like slow compile times and all these dependencies.

Yurii: This is just like a layer over the C stuff, an evolution of that. It’s essentially a header only library, so it’s a very common thing in the C++ world. So you don’t have to build anything and you just include a file. And in fact the way I use it, I amalgamate all the header files that we have into one. Whenever I include it in the project, I just copy the amalgamation and it’s just one file. You don’t have any other build chain associated yet. It is C++ 20, which some people consider new, but by the time it’s mature it’s already five years old and most compilers support it. They have decent support of C++ 20 with a few exclusions, but those are relatively minor. So for that reason, it’s not C++ 23, for example, because it’s not very well supported across compilers, but C++ 20 is.

Yurii is giving a talk about PostgresPM at the Postgres Conference in Orlando. He’ll share the slides and recording with this group. The idea behind PostgresPM is that it takes a lot of heuristics, takes the URLs of packages and of extensions and creates packages for different outputs like for Red Hat, for Debian, perhaps for some other formats in the future. It focuses on the idea that a lot of things can be figured out.

For example: do we have a new version? Well, we can look at list of tags in the Git repo. Very commonly that works for say 80 percent of extensions. Do we need a C compiler? We can see whether we have C files. We can figure out a lot of stuff without packagers having to specify that manually every time they have a new extension. And they don’t have to repackage every time there is a new release, because we can detect new releases and try to build.

Yurii is also running an event that, while not affiliated with PGConf.dev, is strategically scheduled to happen one day before PGConf.dev: Postgres Extensions Day. The Call for Speakers is open until April 1st. There’s also an option for people who cannot or would not come to Montreal this year to submit a prerecorded talk. The point of the event is not just to bring people together, but also ti surface content that can be interesting to other people. The event itself is free.

Make sure to join our Meetup group and join us live, March 26, when Peter Eisentraut joins us to talk about implementing an extension search path.

More about… Postgres PGXN Extensions PGConf Summit

Homebuyers & sellers, intend to buy or sell this Spring and talking to real estate agents about their fees? Have any mentioned that is just…

The post Is proposed MLSpin Settlement the April Fool’s Day joke #RECartel doesn’t want consumers retelling? first appeared on Real Estate Cafe.

Fp9ootron

ABCXXZXCFDSESW4ZXDAWEQ!QWZ

fwwqqqqwwQsxXSQWSASXC XDΩSXEDRTY7U8I9

 

Editor's note: does toddler input have sufficient entropy to be used as a random number generator?

deepseek-ai/DeepSeek-V3-0324

Chinese AI lab DeepSeek just released the latest version of their enormous DeepSeek v3 model, baking the release date into the name DeepSeek-V3-0324.

The license is MIT (that's new - previous DeepSeek v3 had a custom license), the README is empty and the release adds up a to a total of 641 GB of files, mostly of the form model-00035-of-000163.safetensors.

The model only came out a few hours ago and MLX developer Awni Hannun already has it running at >20 tokens/second on a 512GB M3 Ultra Mac Studio ($9,499 of ostensibly consumer-grade hardware) via mlx-lm and this mlx-community/DeepSeek-V3-0324-4bit 4bit quantization, which reduces the on-disk size to 352 GB.

I think that means if you have that machine you can run it with my llm-mlx plugin like this, but I've not tried myself!

llm mlx download-model mlx-community/DeepSeek-V3-0324-4bit llm chat -m mlx-community/DeepSeek-V3-0324-4bit

The new model is also listed on OpenRouter. You can try a chat at openrouter.ai/chat?models=deepseek/deepseek-chat-v3-0324:free.

Here's what the chat interface gave me for "Generate an SVG of a pelican riding a bicycle":

I have two API keys with OpenRouter - one of them worked with the model, the other gave me a No endpoints found matching your data policy error - I think because I had a setting on that key disallowing models from training on my activity. The key that worked was a free key with no attached billing credentials.

For my working API key the llm-openrouter plugin let me run a prompt like this:

llm install llm-openrouter llm keys set openrouter # Paste key here llm -m openrouter/deepseek/deepseek-chat-v3-0324:free "best fact about a pelican"

Here's that "best fact" - the terminal output included Markdown and an emoji combo, here that's rendered.

One of the most fascinating facts about pelicans is their unique throat pouch, called a gular sac, which can hold up to 3 gallons (11 liters) of water—three times more than their stomach!

Here’s why it’s amazing:
- Fishing Tool: They use it like a net to scoop up fish, then drain the water before swallowing.
- Cooling Mechanism: On hot days, pelicans flutter the pouch to stay cool by evaporating water.
- Built-in "Shopping Cart": Some species even use it to carry food back to their chicks.

Bonus fact: Pelicans often fish cooperatively, herding fish into shallow water for an easy catch.

Would you like more cool pelican facts? 🐦🌊

In putting this post together I got Claude to build me this new tool for finding the total on-disk size of a Hugging Face repository, which is available in their API but not currently displayed on their website.

Update: Here's a notable independent benchmark from Paul Gauthier:

DeepSeek's new V3 scored 55% on aider's polyglot benchmark, significantly improving over the prior version. It's the #2 non-thinking/reasoning model, behind only Sonnet 3.7. V3 is competitive with thinking models like R1 & o3-mini.

Tags: llm-release, hugging-face, generative-ai, deepseek, ai, llms, mlx, llm, ai-assisted-programming, tools, pelican-riding-a-bicycle

デジタル学生証明による学割サービスの話は、かなり前から検討されている話題です。わたしの周りでも、遅くとも2012年には Student Identity Trust Framework として国立情報学研究所（NII）との間で検討がなされてきました。

学生 ID 向けトラストフレームワークの概要と未来展望 (2012.5) 学術認証フェデレーションが切り開く大学間連携の新時代 (2012.12) 学認システムと学割サービス連携の新時代：トラストフレームワークが実現する便利で安全な学生生活 (2016.10）

今回発表されたものは、これらの線上に乗りながら、実装技術としてマイナンバーカードやデジタル認証アプリ、Microsoft Authenticator とVCを取り入れたものになっています。ある意味、紙やカードのアナロジーに載せることで一般の経験則的に「分かりやすく」なったところもあるかもしれません。以下はその概要です。

概要

JR西日本とデジタル庁は、大学生の在学証明確認にマイナンバーカードを活用する実証実験を大阪駅で実施しました。本実験は、学生がオンラインでデジタル在学証明書を取得し、デジタル乗車券購入時の本人確認にマイナンバーカードを用いることで、従来の紙の在学証明書提示による窓口手続きを不要とし、利便性向上と業務効率化を目指すものです。国立情報学研究所（NII）が提供するサービスとJR西日本のデジタルプラットフォーム「WESTER」の連携も視野に入れ、将来的には他の公共機関やサービスへの応用も目指しています。

発表されたのは主に以下のような項目です。

1. マイナンバーカードを活用した在学証明のデジタル化 従来、学生が学割乗車券を購入する際には、紙の在学証明書を駅の窓口で提示する必要がありました。 本実証実験では、学生がオンラインでデジタル在学証明書を取得し、マイナンバーカードを本人確認に利用することで、この手続きをオンラインで完結させます。 日経新聞の記事では、実証実験の様子として「マイナカードで在学証明を確認して購入したウェブ乗車券」の写真が掲載されています。 JR西日本のプレスリリースによると、学生はまず「デジタル認証アプリ」と「Microsoft Authenticator」をスマートフォンにダウンロードし、大学のウェブシステムで在学証明情報を取得、それをMicrosoft Authenticatorと連携させる流れとなります。

細かい流れは以下の図表１通りです。

図表１：JR西日本プレスリリースによる流れ

「デジタル認証アプリ」1「Microsoft Authenticator」2をスマートフォンにダウンロード 大学のウェブシステムに学生がログインし国立情報学研究所(NII)が提供する在学証明情報を取得 取得した在学証明情報を Microsoft Authenticator と連携し、情報を学生が保有するスマートフォンに格納 JR 西日本がサービス提供する電子チケット販売プラットフォーム「まちのヲトモパスポート」に「デジタル認証アプリ」でログイン 「デジタル認証アプリ」が立ち上がり学生本人のマイナンバーカードを読み取り認証を実施し、MAB3 に登録した情報と本人情報が同一であることを確認 「まちのヲトモパスポート」に表示された、学生限定の割引チケットを取得する際、Microsoft Authenticator と連携し、MAB で認証した本人が在学していることを検証 学生限定チケットを取得

と、言っても関係性がよくわかりませんね。なので、想像力バリバリで簡略版のシーケンス図（図表２）を書いてみました。

図表２：JR西日本ーデジタル庁実証実験シーケンス図

個人的に注意を引いた点としては、以下があります。

いわゆる認証連携ベース（SAMLやOpenID Connect）でやるのに比べて、図表１では手順3, 6 が増えている。その代わり、大学は学生がどこで学割を使ったかがわからないようになっている。（逆に言えば、大学にわかっても良ければこのステップは省略できる。） このプレスリリースからは発行された在学証明情報がどのように失効させられるかがわからない。（やりかたによっては、大学に証明書利用がわかってしまうようになるので要注意。おそらく、VC Status List あたりを使うのであろう。） いわゆる、属性ベースの名寄せを利用している。このためにマイナンバーカードとデジタル認証アプリを使っている。（そもそも、大学発行の証明書を信じることにすれば、マイナンバーカードからの基本４情報は不必要なはず。お一人様一回限りをやるなら、デジタル認証アプリの発行する利用し識別子の値（sub値）だけで十分なはず。複数の学校に所属しているようなケースを例外として忘れることにすれば、それすらも要らないはず。） 2. オンラインでの学割乗車券購入と顔認証による改札通過 実証実験では、大阪大学の学生が購入したデジタル乗車券を使用し、顔認証で改札口を通過しました。 阪大大学院の武内祐哉さんは「すべてオンラインで完結してスムーズに改札に入れた」と、その利便性を語っています（日経新聞）。

※ 顔認証に使う顔画像をどのステップで取得したのかがよくわからないので、そこは聞いてみたいところです。

3. 関係機関の連携 本実証実験は、デジタル庁の公募事業としてJR西日本が採択され、国立情報学研究所（NII）が提供するデジタル在学証明書発行サービスを利用して行われました。 デジタル庁の岸信千世大臣政務官は「将来的には複数の大学や公共機関での応用をめざしたい」と述べており、連携の重要性が示唆されています（日経新聞）。 4. JR西日本の狙いと今後の展望 JR西日本は、窓口業務の効率化に加え、「学生など若い人との接点を増やしてサービスを深化させたい」（JR西日本デジタルソリューション本部長 奥田英雄氏、日経新聞）と考えています。 2026年初頭にも大阪大学との間で、マイナンバーカードを用いた在学証明の本格的な開始を目指しています。 将来的には、JR西日本グループ共通ID「WESTER」との連携によるサービス拡大も検討されています。 ケータイ Watchの記事では、本実証実験が「現在も多くの在学証明書が紙で発行されている状況を踏まえ、その代表的な事例として選ばれたもの」と説明されており、全国の大学への普及や他の資格情報への応用も視野に入れていることがわかります。 JR西日本プレスリリースでは、2025年度に大阪大学と連携し、学生にとって分かりやすく使いやすいスキームへの改良を進める方針が示されています。 長期的には、交通機関だけでなく、レンタカー、バス、航空券、美術館など幅広い分野での活用も見据えています（ケータイ Watch）。 5. 技術基盤「Mobility Auth Bridge (MAB)」の活用 JR西日本のプレスリリースによると、本実証実験では、KANSAI MaaSでも導入されている「Mobility Auth Bridge（MAB）」というID基盤が活用されています。 MABは、利用者の同意があれば、一つのIDで様々なサービスを利用できるセキュアなIDサービスであり、デジタル社会に必要なインフラ基盤としてJR西日本がNTTコミュニケーションズ、伊藤忠テクノソリューションズの支援を受けて開発したものです。 MABの活用により、デジタルサービスの個客データ収集・利活用が可能となり、地域への貢献も期待されています。 6. SDGsへの貢献 JR西日本のプレスリリースでは、今回の取り組みがSDGsの目標のうち、特に「3. すべての人に健康と福祉を」「9. 産業と技術革新の基盤をつくろう」「11. 住み続けられるまちづくりを」「17. パートナーシップで目標を達成しよう」に貢献するとの考えが示されています。 引用 （武内祐哉さん）「すべてオンラインで完結してスムーズに改札に入れた」（日経新聞） （岸信千世大臣政務官）「将来的には複数の大学や公共機関での応用をめざしたい」（日経新聞） （奥田英雄・JR西取締役常務執行役員）「学生など若い人との接点を増やしてサービスを深化させたい」（日経新聞） （JR西日本プレスリリース）「オンラインで本人確認から学割利用まで完結可能なスキームを試験開発しました。」 （ケータイ Watch）「マイナンバーカードを介することで、本人であることを証明する」 （ケータイ Watch）「今後は、全国の大学への普及や、卒業証書・成績証明書など他の資格情報への応用も視野に入れている。」 （JR西日本プレスリリース）「利用者が同意すれば、1 つの ID で MAB に参画する自治体や企業のさまざまなサービスをご利用いただけるセキュアな ID サービスです。」 結論

JR西日本とデジタル庁によるマイナンバーカードを活用した在学証明の実証実験は、学生の利便性向上と鉄道事業者の業務効率化に大きく貢献する可能性を示唆しています。デジタル在学証明書とオンラインでの学割乗車券購入、そして顔認証によるスムーズな改札通過は、次世代の公共交通利用のあり方を提示するものです。今後、実証実験の成果を踏まえ、全国的な展開や他の分野への応用が進むことが期待されます。また、技術基盤であるMABの活用は、様々な地域サービスとの連携を促進し、より便利で豊かな社会の実現に寄与する可能性を秘めています。

（付録）シーケンス図のコード

title JR西日本-デジタル庁実証実験想像図

participant 学生 as U
participant ブラウザ as UA
participant ウォレット as W

participant 大学 as NII
participant デジタル認証アプリサーバ as D
participant デジタル認証アプリ as Dc
participant MAB as Mobile Auth Bridge
participant チケットサイト as T

autonumber on

U–>UA: 在学証明書発行依頼
UA->NII: 在学証明書発行依頼
NII->UA: ログイン画面表示
U–>UA: ログイン情報入力
UA->NII: ログイン情報
NII->NII: ログイン情報チェック
NII->UA: MS Authenticator 呼び出し＋トークン
UA->W: 起動, トークン
W->NII: 在学証明書取得 w/トークン
NII->W: 在学証明書
U–>UA: チケットサイトアクセス
UA->T: アクセス
T–>MAB: MABに認証要求
MAB–>UA: ログイン画面
UA–>U: ログイン画面表示
U–>UA: デジタル認証アプリで認証を選択
UA->MAB: 選択結果通知
MAB–>UA: リダイレクト
UA->D: ユーザ認証要求
D–>UA: デジタル認証アプリ起動
UA->Dc: 起動
Dc–>U: マイナンバーカード提示要求
U–>Dc: マイナカード提示
Dc->D: スキャン結果提示
D->D: スキャン結果検証
D–>UA: リダイレクト w/code
UA->MAB: code 提示
MAB->D: code, client assertion提示
D->MAB: ４情報返却
MAB->MAB: MABに登録した情報上記４情報が同一であることを確認
MAB->T: OK
T–>UA: チケット一覧表示
UA–>U: 表示
U–>UA: チケット選択
UA->T: チケット選択
T–>UA: 在学証明要求
UA->W: 起動＋在学証明要求
W–>U: 在学証明提示許可要求
U–>W: 許可
W–>UA: 在学証明書
UA->T: 在学証明書
T->T: 資格確認
T–>UA: 購入成功表示

Semantic Diffusion

I learned about this term today while complaining about how the definition of "vibe coding" is already being distorted to mean "any time an LLM writes code" as opposed to the intended meaning of "code I wrote with an LLM without even reviewing what it wrote".

I posted this salty note:

Feels like I'm losing the battle on this one, I keep seeing people use "vibe coding" to mean any time an LLM is used to write code

I'm particularly frustrated because for a few glorious moments we had the chance at having ONE piece of AI-related terminology with a clear, widely accepted definition!

But it turns out people couldn't be trusted to read all the way to the end of Andrej's tweet, so now we are back to yet another term where different people assume it means different things

Martin Fowler coined Semantic Diffusion in 2006 with this very clear definition:

Semantic diffusion occurs when you have a word that is coined by a person or group, often with a pretty good definition, but then gets spread through the wider community in a way that weakens that definition. This weakening risks losing the definition entirely - and with it any usefulness to the term.

What's happening with vibe coding right now is such a clear example of this effect in action! I've seen the same thing happen to my own coinage prompt injection over the past couple of years.

This kind of dillution of meaning is frustrating, but does appear to be inevitable. As Martin Fowler points out it's most likely to happen to popular terms - the more popular a term is the higher the chance a game of telephone will ensue where misunderstandings flourish as the chain continues to grow.

Andrej Karpathy, who coined vibe coding, posted this just now in reply to my article:

Good post! It will take some time to settle on definitions. Personally I use "vibe coding" when I feel like this dog. My iOS app last night being a good example. But I find that in practice I rarely go full out vibe coding, and more often I still look at the code, I add complexity slowly and I try to learn over time how the pieces work, to ask clarifying questions etc.

I love that vibe coding has an official illustrative GIF now!

Tags: language, vibe-coding, andrej-karpathy, martin-fowler

I nominate agency as Word of the Year for 2025.

I don’t nominate agentic, which is suddenly hot shit:

See, agency is a noun, and agentic is an adjective. And, as Strunk and White taught us,

Write with nouns and verbs, not with adjectives and adverbs… it is nouns and verbs, not their assistants, that give good writing its toughness and color.

The word agency is derived from the Latin agere, meaning to do. It speaks of our capacity to act with effect in the world. Or, in the words of the OED (the print version, not the website): Action or instrumentality embodied or personified as concrete existence.

In a chapter of The Intention Economy titled Agency, I say this under a subhead titled The Argument:

Agency is personal. It is the source of confidence behind all intention. By its nature the networked marketplace welcomes full agency for customers. So, because the best vendors are customer driven, there will be many more ways for both vendors and customers to thrive in the networked marketplace, and therefore also in the Intention Economy.

Later I add,

In the Intention Economy, liberated customers enjoy full agency for themselves, and employ agents who respect and apply the powers that customers grant them.

I wrote that (and created the graphic at the top of this page) thirteen years ago, when nobody was talking about agency, but I thought somebody had to.

Now it seems everybody is talking about it. I am sure that’s because we have AI. Or, more specifically, agentic AI. That’s what makes agentic so hot:

My concern with both agentic and agentic AI is that concentrating development on AI agents (and digital “twins”) alone may neglect, override, or obstruct the agency of human beings, rather than extending or enlarging it. (For more on this, read Agentic AI Is the Next Big Thing but I’m Not Sure It’s What, by Adam Davidson in How to Geek. Also check out my Personal AI series, which addresses this issue most directly in Personal vs. Personal AI.)

So, what will give you real agency—an archimedean lever that gives you enough leverage to move worlds?

Meet IEEE P7012, which “identifies/addresses the manner in which personal privacy terms are proffered and how they can be read and agreed to by machines.” It has been in the works since 2017, and should be ready later this year. (I say this as chair of the standard’s working group.) The nickname for P7012 is MyTerms (much as the nickname for the IEEE’s 802.11 standard is Wi-Fi). The idea behind MyTerms is that the sites and services of the world should agree to your terms, rather than the other way around.

MyTerms creates a new regime for privacy: one based on contract. With each MyTerm you are the first party. Not the website, the service, or the app maker. They are the second party. And terms can be friendly. For example, a prototype term called NoStalking says “Just show me ads not based on tracking me.” This is good for you, because you don’t get tracked, and good for the site because it leaves open the advertising option. NoStalking lives at Customer Commons, much as personal copyrights live at Creative Commons. (Yes, the former is modeled on the latter.)

On the Creative Commons model, agreements take three forms:

The MyTerms standard addresses just the Machine Readable layer. It also doesn’t say exactly what tools should be developed. Basically, it just says,

The person is the first party The site, service or app is the second party The person chooses an agreement from a limited roster (resembling Creative Commons’ roster of licenses) listed by a disinterested non-profit Both sides keep an identical record of what they agreed to

On your side—the first-party side—browser makers can build something into their product, or any developer can make a browser add-on (Firefox) or extension (the rest of them). On the site’s side—the second-party side—CMS makers can build something in, or any developer can make a plug-in (WordPress) or a module (Drupal).

Mobile app toolmakers can also come up with something (or many things).

For the Legal Code and Human Readable layers, we (Customer Commons and ProjectVRM) have been at work on a list of prototypes for the roster of agreements. We’ll present these on April 7 at VRM Day, at the Computer History Museum in Silicon Valley. Discussion will happen both there and through the following three days in the same location at the 40th Internet Identity Workshop (IIW). VRM Day is free. IIW isn’t, but is cheap for a three-day conference that (IMHO) is the most leveraged in the world.

So let’s make this happen and show the world what agency really means.

And, if you’re interested in helping support Customer Commons, use some of that agency to hit the Donate button on its home page. Thanks!

Next.js and the corrupt middleware: the authorizing artifact

Good, detailed write-up of the Next.js vulnerability CVE-2025-29927 by Allam Rachid, one of the researchers who identified the issue.

The vulnerability is best illustrated by this code snippet:

const subreq = params.request.headers['x-middleware-subrequest']; const subrequests = typeof subreq === 'string' ? subreq.split(':') : []; // ... for (const middleware of this.middleware || []) { // ... if (subrequests.includes(middlewareInfo.name)) { result = { response: NextResponse.next(), waitUntil: Promise.resolve(), }; continue; } }

This was part of Next.js internals used to help avoid applying middleware recursively to requests that are re-dispatched through the framework.

Unfortunately it also meant that attackers could send a x-middleware-subrequest HTTP header with a colon-separated list of middleware names to skip. If a site used middleware to apply an authentication gate (as suggested in the Next.js documentation) an attacker could bypass that authentication using this trick.

The vulnerability has been fixed in Next.js 15.2.3 - here's the official release announcement talking about the problem.

Tags: security, javascript

[Gaby Del Valle at The Verge]

I appreciated this deep-dive on whether it's safe to travel through US borders with your phone. Journalists and anyone who's made overt political statements should take particular note.

"The government maintains that it doesn’t need a warrant to conduct “basic” searches of the contents of a person’s phone. During these searches, Hussain explained, agents are supposed to put your phone on airplane mode and can only look at what is accessible offline — but that can still be a lot of information, including any cloud data that’s currently synced."

The EFF maintains a pretty great pocket guide that is also worth checking out.

The constitutionality of searches is "still an open question" - but that doesn't matter in the moment. My advice at this stage is to sign out of important apps (like your work email and encrypted messaging apps like Signal), turn off biometric logins like Face ID, and switch your phone off. That does mean you need to print out your boarding card, for example, and do a little pre-work to make sure your data is backed up. Clearly, this is a pain. But if you deal with any sensitive information, or have any vulnerable people in your family or community, you need to change your security stance to be a good steward of their safety.

#Democracy

[Link]

If you’re new to tech, taking [career] advice on what works for someone with a 20-year career is likely to be about as effective as taking career advice from a stockbroker or firefighter or nurse. There’ll be a few things that generalize, but most advice won’t.

Further, even advice people with long careers on what worked for them when they were getting started is unlikely to be advice that works today. The tech industry of 15 or 20 years ago was, again, dramatically different from tech today.

— Jacob Kaplan-Moss, Beware tech career advice from old heads

Tags: jacob-kaplan-moss, careers

simonw/ollama-models-atom-feed

I setup a GitHub Actions + GitHub Pages Atom feed of scraped recent models data from the Ollama latest models page - Ollama remains one of the easiest ways to run models on a laptop so a new model release from them is worth hearing about.

I built the scraper by pasting example HTML into Claude and asking for a Python script to convert it to Atom - here's the script we wrote together.

Update 25th March 2025: The first version of this included all 160+ models in a single feed. I've upgraded the script to output two feeds - the original atom.xml one and a new atom-recent-20.xml feed containing just the most recent 20 items.

I modified the script using Google's new Gemini 2.5 Pro model, like this:

cat to_atom.py | llm -m gemini-2.5-pro-exp-03-25 \ -s 'rewrite this script so that instead of outputting Atom to stdout it saves two files, one called atom.xml with everything and another called atom-recent-20.xml with just the most recent 20 items - remove the output option entirely'

Here's the full transcript.

Tags: github-actions, git-scraping, ai, ollama, llms, ai-assisted-programming, generative-ai, projects, github, claude, atom, gemini

We'll know soon. Whether or not you're watching St. John's playing Arkansas, right now, in the NCAA's March Madness tournament, take out a minute and a half to take in Jimmy Fallon and the boys singing the Red Storm Shanty. Lou Carnaseca must be glowing in his grave. (Later: they lost.)

But that’s the idea, right? Lucas Ropek in Gizmodo: Data Broker Brags About Having Highly Detailed Personal Information on Nearly All Internet Users: The advertising industry is immensely powerful and disturbingly opaque. Read it. Then look at a PageXray of that same story to see how much tracking Gizmodo does, and how deeply embedded it is in that same fecosystem:

Care to report on that, Lucas? If you do, you won’t be the first. That was Farhad Monjoo, here, six years ago in The New York Times. You’ll be the second.

What if it helps everyone do a better job? Steven Wright: “I take my dog for walks outside my apartment… on the ledge. Some people are afraid of heights. I’m afraid of widths.”Somehow that joke (as best I recall it) comes to mind when I consider OpenAI’s Deep Research. Everybody I know who uses it gushes about all the amazing things it can do. I mean, a lot. A company CEO says it produces marvels with sales and usage data, with forecasting, with you-name-it. A law professor says it radically improves her job in almost every way. An innovation director says “It will change your life.” But there is push-back, naturally, on the usual AI threat: it will cost jobs. We’ll see.

A Google StreetView of the entrance to the Edward R. Murrow Transmitting Station for the Voice Of America, near Greenville, North Carolina

The Voice of America is silent.

To Donald Trump, Elon Musk, and Kari Lake (who now runs its corpse), the VOA was corrupt, biased, unnecessary, and needed to go. To nearly everyone else who cares, it was America’s voice on radio, and mattered enormously to an audience in the hundreds of millions, listening in forty-eight languages. For many of those, the VOA was the best, and in some cases the only, source of relatively unbiased news.

I am sure some of it was biased. Reporters tend to stand in a place, and most who report on political matters stand on the left (see here and here). Dan Robinson, who spent thirty-four years with the VOA, stands on the right and posted a long bill of particulars in Voice of America, Global Media Agency are rightly in budget cut bull’s-eye: Entire operation should be dismantled, in the Washington Times. An excerpt:

 I have monitored the agency’s bureaucracy along with many of its reporters and concluded that it has essentially become a hubris-filled rogue operation often reflecting a leftist bias aligned with partisan national media. It has sought to avoid accountability for violations of journalistic standards and mismanagement. The list includes:

• During President-elect Donald Trump’s first term, VOA allowed and often encouraged key correspondents to carry out opposition journalism against him. This included an uprising, growing out of VOA’s central newsroom, against Mr. Trump’s choice of chief USAGM executive in 2020.

• That newsroom and the agency have a known left-wing bias. Between 2016 and 2020, some VOA reporters did little to hide their disdain for Mr. Trump in their reports and social media posts.

• Since Amanda Bennett, VOA director from 2016 to mid-2020, returned as USAGM chief in 2022, managers and employees dissed Republicans in Congress who criticized VOA’s inexplicable refusal to refer to Hamas terrorists as terrorists. One agency official called lawmakers “silly.” VOA’s new director has equated congressional critics to troublemakers.

Whatever. I’d rather they fix it than kill it. They wanted it dead, and now it is.

More reading here:

The Last Days at Voice of America: Covering the press freedom beat at VOA, I got a front-row seat to its demise, by Liam Scott, in Columbia Journalism Review Trump’s Awful Decision to Gut Voice of America and Radio Free Europe/Radio Liberty, by Ilya Somin in Reason Trump guts the 83-year-old Voice of America, in NiemanLab ‘Discarded like a dirty rag’: Chinese state media hails Trump’s cuts to Voice of America, by Kelly Ng in BBC News The silencing of Voice of America: How the shuttering of a small US-funded news outlet explains Trump’s governing philosophy, by Sean Collins and Gabrielle Berbey in Vox Trump mutes Voice of America, makes space for Russian and Chinese influence, by Sebastian Seibt in France 24 What is Voice of America and why Trump is dismantling the broadcaster? by PBS News Hour Inside the scramble to save America’s pro-democracy media outlets from Trump, by several authors at Politico Trump Closes an Important Window Into China, by Lili Pike, in Foreign Policy Trump’s silencing of Voice of America sparks shock, outrage, by Dominick Mastrangelo, in The Hill Trump silencing VOA threatens free media in repressive countries, advocates say, by ByPatrick Reevell and Somayeh Malekian in ABC News Opinion | Trump ending Voice of America saved us all from Kari Lake: Voice of America is supposed to operate independently of any presidential administration which naturally makes it a prime target of Trump, by Laurie Roberts in the Arizona Republic

Before we start: if you’ve already filled out the What is your tech stack? survey: thank you! If you’ve not done so, your help will be greatly appreciated. It takes 5-15 minutes to complete. Those filling out will receive results before anyone else, and additional analysis from myself and Elin. Fill out this survey here.

Fill out the survey

Growth engineering was barely known a decade ago, but today, most scaleups and many publicly traded tech companies have dedicated growth teams staffed by growth engineers. However, some software engineers are still suspicious of this new area because of its reputation for hacky code with little to no code coverage.

For this reason and others, I thought it would be interesting to learn more from an expert who can tell us all about the practicalities of this controversial domain. So I turned to Alexey Komissarouk, who’s been in growth engineering since 2016, and was in charge of it at online education platform, MasterClass. These days, Alexey lives in Tokyo, Japan, where he advises on growth engineering and teaches the Growth Engineering course at Reforge.

In today’s deep dive, Alexey covers:

What is Growth Engineering? In the simplest terms: writing code to help a company make more money. But there are details to consider: like the company size where it makes sense to have a dedicated team do this.

What do Growth Engineers work on? Business-facing work, empowerment and platform work are the main areas.

Why Growth Engineers move faster than Product Engineers. Product Engineers ship to build: Growth Engineers ship to learn. Growth Engineers do take shortcuts that would make no sense when building for longevity – doing this on purpose.

Tech stack. Common programming languages, monitoring and oncall, feature flags and experimentation, product analytics, review apps, and more.

What makes a good Growth Engineer? Curiosity, “build to learn” mindset and a “Jack of all trades” approach.

Where do Growth Engineers fit in? Usually part of the engineering department, either operating as with an “owner” or a “hitchiker” model.

Becoming a Growth Engineer. A great area if you want to eventually become a founder or product manager – but even if not, it can accelerate your career growth. Working in Growth forces you to learn more about the business.

With that, it’s over to Alexey:

I’ll never forget the first time I made my employer a million dollars.

I was running a push notification A/B test for meal delivery startup Sprig, trying to boost repeat orders.

A push notification similar to what we tested to boost repeat orders

Initial results were unpromising; the push notification was not receiving many opens. Still, I wanted to be thorough: before concluding the idea was a failure, I wrote a SQL query to compare order volume for subsequent weeks between customers in test vs control.

The SQL used to figure out the push notification’s efficiency

As it turned out, our test group “beat” the control group by around 10%:

‘review_5_push’ was the new type of push notification. Roughly the same amount of users clicked it, but they placed 10% more in orders

I plugged the numbers into a significance calculator, which showed it was statistically significant – or “stat-sig” – and therefore highly unlikely to be a coincidence. This meant we had a winner on our hands! But how meaningful was it, really, and what would adding the push notification mean for revenue, if rolled out to 100% of users?

It turned out this experiment created an additional $1.5 million dollars, annually, with just one push notification. Wow!

I was hooked. Since that day, I've shipped hundreds of experimental “winners” which generated hundreds of millions of incremental revenue for my employers. But you never forget the first one. Moments like this is what growth engineering is all about.

1. What is Growth Engineering?

Essentially, growth engineering is the writing of code to make a company money. Of course, all code produced by a business on some level serves this purpose, but while Product Engineers focus on creating a Product worth paying for, Growth Engineers instead focus on making that good product have a good business. To this end, they focus on optimizing and refining key parts of the customer journey, such as:

Getting more people to consider the product

Converting them into paying customers

Keeping them as customers for longer, and spending more

What kinds of companies employ Growth Engineers? Places you’ve heard of, like Meta, LinkedIn, DoorDash, Coinbase, and Dropbox, are some of the ones I’ve had students from. There’s also OpenAI, Uber, Tiktok, Tinder, Airbnb, Pinterest… the list of high-profile companies goes on. Most newer public consumer companies you’ve heard have a growth engineering org, too.

Typically, growth engineering orgs are started by companies at Series B stage and beyond, so long as they are selling to either consumers or businesses via SaaS. These are often places trying to grow extremely fast, and have enough software engineers that some can focus purely on growth. Before the Series B stage, a team is unlikely to be ready for growth for various reasons; likely that it hasn’t found product-market fit, or has no available headcount, or lacks the visitor traffic required to run A/B tests.

Cost is a consideration. A fully-loaded growth team consisting of a handful of engineers, a PM, and a designer costs approximately 1 million dollars annually. To justify this, a rule of thumb is to have at least $5 million dollars in recurring revenue – a milestone often achieved at around the Series B stage.

Despite the presence of growth engineering at many public consumer tech companies, the field itself is still quite new, as a discipline and as a proper title.

Brief history of growth engineering

When I joined Opendoor in 2016, there was a head of growth but no dedicated growth engineers, but there were by the time I left in 2020. At MasterClass soon after, there was a growth org and a dozen dedicated growth engineers. So when did growth engineering originate?

The story is that its origins lie at Facebook in 2007. The team was created by then-VP of platform and monetization Chamath Palihapitiya. Reforce founder and CEO Brian Balfour shares:

“Growth (the kind found on an org chart) began at Facebook under the direction of Chamath Palihapitiya. In 2007, he joined the early team in a nebulous role that fell somewhere between Product, Marketing, and Operations. According to his retelling of the story on Recode Decode, after struggling to accomplish anything meaningful in his first year on the job, he was on the verge of being fired.

Sheryl Sandberg joined soon after him, and in a hail mary move he pitched her the game-changing idea that led to the creation of the first-ever growth team. This idea not only saved his job, but earned him the lion’s share of the credit for Facebook’s unprecedented growth.

At the time, Sheryl and Mark asked him, “What do you call this thing where you help change the product, do some SEO and SEM, and algorithmically do this or that?”

His response: “I don’t know, I just call that, like, Growth, you know, we’re going to try to grow. I’ll be the head of growing stuff."

And just like that, Growth became a thing.”

Rather than focus on a particular product or feature, the growth team at Facebook focused on moving the needle, and figuring out which features to work on. These days, Meta employs hundreds if not thousands of growth engineers.

2. What do Growth Engineers work on?

Before we jump into concrete examples, let’s identify three primary focus areas that a growth engineer’s work usually involves.

Business-facing work – improving the business directly

Empowerment work – enabling other teams to improve the business

Platform work – improving the velocity of the above activities

Let’s go through all three:

Business-facing work

This is the bread and butter of growth engineering, and follows a common pattern:

Implement an idea. Try something big or small to try and move a key business metric, which differs by team but is typically related to conversion rate or retention.

Quantify impact. Usually via A/B testing.

Analyze impact. Await results, analyze impact, ship or roll back – then go back to the first step.

Experiments can lead to sweeping or barely noticeable changes. A famous “I can’t believe they needed to test this” was when Google figured out which shade of blue generates the most clicks. At MasterClass, we tested things across the spectrum:

Small: should we show the price right on the homepage, was that a winner? Yes, but we framed it in monthly terms of $15/month, not $180/year.

Medium: when browsing a course page, should we include related courses, or more details about the course itself? Was it a winner? After lengthy experimentation, it was hard to tell: both are valuable and we needed to strike the right balance.

Large: when a potential customer is interested, do we take them straight to checkout, or encourage them to learn more? Counterintuitively, adding steps boosted conversion!

Empowerment

One of the best ways an engineer can move a target metric is by removing themselves as a bottleneck, so colleagues from marketing can iterate and optimize freely. To this end, growth engineers can either build internal tools or integrate self-serve MarTech (Marketing Technology) vendors.

With the right tool, there’s a lot that marketers can do without engineering’s involvement:

Build and iterate on landing pages (Unbounce, Instapage, etc)

Draft and send email, SMS and Push Notifications (Iterable, Braze, Customer.io, etc)

Connect new advertising partners (Google Tag Manager, Segment, etc)

We go more into detail about benefits and applications in the MarTech section of Tech Stack, below.

Platform work

As a business scales, dedicated platform teams help improve stability and velocity for the teams they support. Within growth, this often includes initiatives like:

Experiment Platform. Many parts of running an experiment can be standardized, from filtering the audience, to bucketing users properly, to observing statistical methodology. Historically, companies built reusable Experiment Platforms in-house, but more recently, vendors such as Eppo and Statsig have grown in popularity with fancy statistical methodologies like “Controlled Using Pre-Experiment Data” (CUPED) that give more signal with less data.

Reusable components. Companies with standard front-end components for things like headlines, buttons, and images, dramatically reduce the time required to spin up a new page. No more "did you want 5 or 6 pixels here" with a designer; instead growth engineers rely on tools like Storybook to standardize and share reusable React components.

Monitoring. Growth engineering benefits greatly from leveraging monitoring to compensate for reduced code coverage. High-quality business metric monitoring tools can detect bugs before they cause damage.

When I worked at MasterClass, having monitoring at the ad layer prevented at least one six-figure incident. One Friday, a marketer accidentally broadened the audience for a particular ad from US-only, to worldwide. In response, the Facebook Ad algorithm went on a spending spree, bringing in plenty of visitors from places like Brazil and India, whom we knew from past experience were unlikely to purchase the product. Fortunately, our monitoring noticed the low-performing campaign within minutes, and an alert was sent to the growth engineer on-call, who immediately reached out to the marketer and confirmed the change was unintentional, and then shut down the campaign.

Without this monitoring, a subtle targeting error like this could have gone unnoticed all weekend and would have eaten up $100,000+ of marketing budget. This episode shows that platform investment can benefit everyone; and since growth needs them most, it’s often the growth platform engineering team which implements them.

As the day-to-day work of a Growth Engineer shows, A/B tests are a critical tool to both measure success and learn. It’s a numbers game: the more A/B tests a team can run in a given quarter, the more of them will end up winners, making the team successful. It’s no wonder, then, that Growth Engineering will pull out all the stops to improve velocity.

3. Why Growth Engineers move faster than Product Engineers

On the surface, growth engineering teams look like product engineering ones; writing code, shipping pull requests, monitoring on-call, etc. So how do they move so much faster? The big reason lies in philosophy and focus, not technology. To quote Elena Verna, head of growth at Dropbox:

“Product Engineering teams ship to build; Growth Engineering teams ship to learn.”

Real-world case: price changes at Masterclass

A few years ago at MasterClass, the growth team wanted to see if changing our pricing model to multiple tiers would improve revenue.

Inspired in part by multiple pricing tiers for competitors such as Netflix (above), Disney Plus, and Hulu. The “multiple pricing tier” proposal for MasterClass.

From a software engineering perspective, this was a highly complex project because:

Backend engineering work: the backend did not yet support multiple pricing options, requiring a decent amount of engineering, and rigorous testing to make sure existing customers weren’t affected.

Client app changes: on the device side, multiple platforms (iOS, iPad, Android, Roku, Apple TV, etc) would each need to be updated, including each relevant app store.

The software engineering team estimated that becoming a “multi-pricing-tier” company would take months across numerous engineering teams, and engineering leadership was unwilling to greenlight that significant investment.

We in growth engineering took this as a challenge. As usual, our goal was not just to add the new pricing model, but to learn how much money it might bring in. The approach we ended up proposing was a Fake Door test, which involves offering a not-yet-available option to customers to gauge interest level. This was risky, as taking a customer who’s ready to pay and telling them to join some kind of waiting list is a colossal waste, and risks making them feel like the target of a “bait and switch” trick.

We found a way. The key insight was that people are only offended about a “bait and switch”, if the “switch” is worse than the “bait.” Telling customers they would pay $100 and then switching to $150 would cause a riot, but starting at $150 and then saying “just kidding, it’s only $100” is a pleasant surprise.

The good kind of surprise.

So long as every test “pricing tier” is less appealing – higher prices, fewer features – than the current offering, we could “upgrade” customers after their initial selection. A customer choosing the cheapest tier gets extra features at no extra cost, while a customer choosing a more expensive tier is offered a discount.

We created three new tiers, at different prices. The new “premium” tier would describe the existing, original offering. Regardless of what potential customers selected, they got this “original offering,” during the experiment.

The best thing about this was that no backend changes were required. There were no real, new, back-end pricing plans; everybody ended up purchasing the same version of MasterClass for the same price, with the same features. The entirety of the engineering work was on building a new pricing page, and the “congratulations, you’ve been upgraded” popup. This took just a few days.

Within a couple of weeks, we had enough data to be confident the financial upside of moving to a multi-pricing-tier model would be significant. With this, we’re able to convince the rest of engineering’s leadership to invest in building the feature properly. In the end, launching multiple pricing tiers turned out to be one of the biggest revenue wins of the year.

Building a skyscraper vs building a tent

The MasterClass example demonstrates the spirit of growth engineering; focusing on building to learn, instead of building to last. Consider building skyscrapers versus tents.

Building a tent optimizes for speed of set-up and tear-down over longevity. You don’t think of a tent as one that is shoddy or low-quality compared to skyscrapers: it’s not even the same category of buildings! Growth engineers maximize use of lightweight materials. To stick with the tents vs skyscraper metaphor: we prioritize lightweight fabric materials over steel and concrete whenever possible. We only resort to traditional building materials when there’s no other choice, or when a direction is confirmed as correct. Quality is important – after all, a tent must keep out rain and mosquitoes. However, the speed-vs-durability tradeoff decision results in very different approaches and outcomes.

4. Tech stack

At first glance, growth and product engineers use the same tooling, and contribute to the same codebases. But growth engineering tends to be high-velocity, experiment-heavy, and with limited test coverage. This means that certain “nice to have” tools for product engineering are mission-critical for growth engineers.

Read more

[Brandon Vigliarolo at The Register]

In the wake of the French and German governments joining forces to build an alternative to Notion and Google Docs, the Dutch government has passed eight motions to avoid US software and move to a home-grown stack.

""With each IT service our government moves to American tech giants, we become dumber and weaker," Dutch MP Barbara Kathmann, author of four of the motions, told The Register. "If we continue outsourcing all of our digital infrastructure to billionaires that would rather escape Earth by building space rockets, there will be no Dutch expertise left."

Kathmann's measures specifically call on the government to stop the migration of Dutch information and communications technology to American cloud services, the creation of a Dutch national cloud, the repatriation of the .nl top-level domain to systems operating within the Netherlands, and for the preparation of risk analyses and exit strategies for all government systems hosted by US tech giants."

Inevitably, these will be open source solutions that offer stronger privacy (with GDPR compliance from the beginning rather than as an afterthought) and fewer dependencies on third party centralized services. I see this as a very strongly good thing: everyone will see the benefit of such tools, and if you have values like "software shouldn't spy on you" and "you should have full control over your data", there will be more options for you to choose from.

The context is important:

"The motions passed by the Dutch parliament come as the Trump administration ratchets up tensions with a number of US allies – the EU among them. Nearly 100 EU-based tech companies and lobbyists sent an open letter to the European Commission this week asking it to find a way to divest the bloc from systems managed by US companies due to "the stark geopolitical reality Europe is now facing.""

It's sensible. I agree. This is what Europe should be doing.

#Technology

[Link]

[Ella Yurman in Teen Vogue]

This interview with Musk's daughter Vivian Wilson is everything.

"The Nazi salute sht was insane. Honey, we're going to call a fig a fig, and we're going to call a Nazi salute what it was. That sht was definitely a Nazi salute. The crowd is equally to blame, and I feel like people are not talking about that. That crowd should be denounced.

But other than that, I don't give a f**k about him. I really don't."

Wilson is a refreshing, no-nonsense voice on Threads, and that same no-BS attitude comes through here like an avalanche.

And this, of course, is vitally important:

"As a trans woman, I am terrified of losing access to guaranteed medical care. If I didn't medically transition at the age I did, I don't know what would've happened. I don't feel like people realize that being trans is not a choice. I'm so sorry to break it to you.

Transitioning as a minor was something that was medically necessary for me to do in order to be not suicidal, and it is really important that we protect access to trans care for trans youth."

Wilson is an example of someone this healthcare not only saved but helped thrive. It's exciting to see her use her insightful, attentive, sometimes hilariously-sharp voice.

#Democracy

[Link]

[Meredith Whittaker in the Financial Times]

A great op-ed from Meredith Whittaker, President of Signal on the ludicrous demands by the British government for Apple to backdoor its encryption:

"Imagine a government telling a car company to secretly weaken the effectiveness of the brakes on all the cars it sells, recklessly endangering the safety of millions. It would be an unthinkable undermining of public safety. 

Sadly, this is what’s happening in the UK in cyber security, where Apple was forced to strip the vital privacy and security protection of end-to-end encryption from its backups storage service — exposing people and infrastructure to significant vulnerabilities."

Meredith is (as usual) right. She points out that not only is this a wildly dangerous thing to do in general, but it undermines the technology industry that the British government sometimes says it wants to support.

Americans shouldn't be complacent. This may be a battle that's heading our way next - and one that was already fought in the nineties. We can't let these erosions of civil liberties take place here; they should not happen anywhere.

#Technology

[Link]

[Jasmine Mooney]

A nightmare account from a Canadian citizen detained by ICE:

"I was taken to a tiny, freezing cement cell with bright fluorescent lights and a toilet. There were five other women lying on their mats with the aluminum sheets wrapped over them like dead bodies. The guard locked the door behind me.

[...] After some research, the reality became clear — ICE detention isn’t just a bureaucratic nightmare. It’s a business. These facilities are privately owned and run for profit.

Companies like CoreCivic and GEO Group receive government funding based on the number of people they detain. The more prisoners, the more money they make. They don’t lobby for stricter immigration policies in the name of national security — they do it to protect their bottom line."

There is so much to fix here, but perhaps this is the crux: people are making a profit from tightened immigration rules that keep people detained for months or years without due process. That shouldn't be allowed to happen.

#Democracy

[Link]

Here’s why I won’t RTO.

Continue reading on Medium »

I cleaned out my desk a little over five years ago. It feels like last week.

I was leading engineering at ForUsAll, a fintech company that seeks to make it easier for small businesses to offer retirement plans for their employees. The co-founder, David, had been wearing a mask around the office for a month; he was following the growth of COVID-19 closely. The then-CEO agreed to close the office if the number of cases in San Francisco went beyond some small threshold; when it did, we picked up our laptops and left.

Of course, we all know what happened next: lockdowns, sourdough starters, and remote working on a scale never seen before.

I prefer remote working and always have. My first startup was mostly remote: my co-founder was in Edinburgh for a while, and spent some time in Vancouver, while I was in Oxford with occasional long stretches in California. I worked at my kitchen table, drank my own coffee, and set my own hours. It was flexible depending on what was going on at the time, and undoubtedly productive. When I joined a startup based in Austin but worked from Edinburgh and Berkeley, it felt like a natural progression.

When the pandemic hit, I couldn’t wait to return to that mode of working. I had another reason to feel like working from home was a silver lining: my mother’s health had been up and down following her double lung transplant, and now I could spend more time with her. What had been a regular Sunday visit became a much longer weekly stay. My dad was the primary carer, but I could help out. Many nights, I would help her up the short flight of stairs to her bedroom, help situate her in her bed, with brushing her teeth, and so on. Working from home gave me extra time with her, and I treasured that.

More recently, it allowed me to buy a house. There was no way I could buy in the San Francisco Bay Area. For literally half the price of a two-bedroom house in a troubled part of Oakland, I could get a house that would fit my family in Pennsylvania. We walk our child to and from daycare every day, have a garden and a driveway, and, although there’s no doubt that the house needs work, generally feel safe and secure.

I’m far from alone. Working from home has been a boon for carers, parents, and anyone who felt like they weren’t able to get on the property ladder in major business hubs like San Francisco and New York. It’s spread wealth from industries like tech to neighborhoods across the country, and in turn allowed tech companies to hire from anywhere, giving them access to talent that would previously have been out of reach. According to official figures from the US Bureau of Labor Statistics, productivity rose.

For remote work to be successful, communication, internal processes, and norms need to be explicit. Companies that had never spent much time thinking about culture now found that they were forced to, which was a positive outcome for their employees, many of whom had been suffering in silence. A renewed focus on employee power — in conjunction with the rise of social movements like Black Lives Matter — also led to a rise in unionization efforts, which also aimed to improve worker quality of life. Despite the overhead of the pandemic itself, these changes felt like they were part of a cohesive, positive movement.

So I read stories like this one in the San Francisco Standard with something like a sense of dread:

“Two years ago, I could not get anybody to go into the office a couple days a week,” said Jaimie Feliz, principal at San Francisco recruiting firm The Hire Standard. “Now, across the board, it’s pretty standard for companies to ask for a minimum of three days in office — it’s very rare to see any less than that.”

Many tech companies are suspending hiring and promotions for workers outside of their hub cities, and there’s an assumption that, over time, employees who remain outside of those hubs will be laid off.

Given the negative impacts on carers, parents, people who have bought homes outside of those hub cities, and on the productivity of those companies, this feels like a regression.

This is doubly true when you look at the underlying statistics. One of the big reasons for calls back to the office is to perform backdoor layoffs: management understands that a substantial percentage will quit. Research also suggests that it’s about control:

RTO mandates may reflect a desire among certain leaders to reassert control and authority within the organization […] This perspective highlights the role of organizational power dynamics and the potential for RTO policies to serve as instruments for reinforcing traditional hierarchical structures, at odds with the trend towards greater autonomy and flexibility facilitated by remote work.

The perceived gains aren’t evidence-based or in the best interests of company productivity; they’re more about CEO peace of mind. For companies that never stuck the landing on building intentional cultures, returning to the pre-pandemic status quo may feel reassuring.

Frustratingly, I now feel like these changes are inevitable.

Not everywhere, of course. There are some companies that have always been remote, and others have managed to establish strong hybrid cultures. But the majority will choose to simply snap back to the world as it was in 2019.

This is to their detriment: adding perspectives from across the country, and from people who would have been shut out of a traditional office job, was clearly valuable. A workforce made up only of people who can afford San Francisco’s $3,400 average rent is inherently less diverse — and less representative of the company’s customers — than one that is geographically diverse. Regardless, it is happening.

For companies that choose to stay remote, there are benefits to be made. There will be an ever-increasing workforce of potential employees who don’t want to move back to those hubs, with experience at tech companies like Google and Meta, who will be looking for new positions. That’s a competitive advantage.

On the other hand, for people who want to stay with their current employers, there are hard choices ahead. Do you move away from your comfortable house, or find ways to offload some of your caring or parental duties, in order to stay on the payroll? Depending on your salary, stock options, or tenure, there might be reasons for doing so.

But it’s not a choice I would make. I have a toddler these days, and I want to be more present, not less. I get a lot of value from in-person collaboration, but I prefer a hybrid model: I’ll gladly travel into the office for a few intense days to advance some specific goals and then go home. I’ve got little interest in doing so to make management feel at ease, but there really are some kinds of time-limited collaboration that are better in person.

I also know that some people can’t travel — for health reasons, because their caring commitments are too great, or these days, because they’re worried about their documents being stripped or suffering violence because of their identity. So even though I’m willing to travel, I don’t expect everyone else to. Even in specific, time-limited collaborations, hybrid accommodations must be made.

For these reasons, I’ve made the decision that I won’t work for a company that requires everyone to come back to the office. Should I start another company, I will not mandate that people work from the office, although I might provide one as an optional collaboration space. This is to protect my quality of life, and to ensure that I can hire the best people for each role, regardless of where they might live or what the rest of their life might look like.

It’s not a decision I take lightly. It’s limiting: it means, should I leave my current job, that there will be fewer places I can go and work. It might limit my salary and future prospects, or even the investment I can raise for a future venture. But I care about being home and present, and I care about building representative workforces.

The bottom line is this: forcing people back into offices isn’t a neutral decision. It’s a choice to exclude and disadvantage anyone who doesn’t fit a narrow definition of what a “worker” looks like. I’m not willing to join in that discrimination.

Stream the Latest Episode

Listen and watch now on YouTube, Spotify and Apple. See the episode transcript at the top of this page, and a summary at the bottom.

Brought to You By

• WorkOS — The modern identity platform for B2B SaaS.

• Vanta — Automate compliance and simplify security with Vanta.

—

In This Episode

Linux is the most widespread operating system, globally – but how is it built? Few people are better to answer this than Greg Kroah-Hartman: a Linux kernel maintainer for 25 years, and one of the 3 Linux Kernel Foundation Fellows (the other two are Linus Torvalds and Shuah Khan). Greg manages the Linux kernel’s stable releases, and is a maintainer of multiple kernel subsystems. He is also the author of the books Linux Kernel in a Nutshell and Linux Device Drivers.

We cover the inner workings of Linux kernel development, exploring everything from how changes get implemented to why its community-driven approach produces such reliable software. Greg shares insights about the kernel's unique trust model and makes a case for why engineers should contribute to open-source projects. We go into:

How widespread is Linux?

What is the Linux kernel responsible for – and why is it a monolith?

How does a kernel change get merged? A walkthrough

The 9-week development cycle for the Linux kernel

Testing the Linux kernel

Why is Linux so widespread?

The career benefits of open-source contribution

And much more!

Takeaways

1. Linux is the most widespread operating system globally. Linux runs on 4 billion Android devices – compared to which everything else is “a rounding error”. Still, Linux is the most popular operating system for servers and embedded devices. It’s also used on many smart TVs, air traffic control systems, and even on the International Space Station. Fun fact: Linux even runs inside many iPhones – as it is the firmware used for the Qualcomm 5G modems inside these devices!

2. Getting a change merged into the Linux kernel is surprisingly straightforward. Create the change (called a patch), test it locally, and send it to the right maintainer for review. The patch needs to go through a hierarchical tree of maintainers accepting it before it can make it into the kernel. We go through a specific change being merged up this tree.

3. Linux won because devs being “selfish” works! Developers contribute to Linux in a "selfish" way to solve their own problems. But turns out, many devs have the same problems: so every contribution makes Linux a better fit for other devs to use! Kernel maintainers only accept contributions that make sense for the whole project. For example, embedded device vendors helped make the Linux kernel more efficient. This efficiency later greatly helped Linux become the best choice for mobile OS on Android.

4. The Linux kernel is run in a very unique way – because the project is unique. The Linux kernel has 4,000 contributors per year, releases strictly every 9 weeks – and yet has practically no meetings, no project managers and uses email and git. This setup works because project management happens outside of the Linux kernel: contributors bring completed work. Also, the kernel team invests heavily in automation for e.g. triaging. And turns out, email scales really well – for this group, that is! (Note that other projects built on top of the Linux kernel – such as Linux distributions like Red Hat or Debian – all work differently. What works for the Linux kernel thanks to unique circumstances, won’t work for those projects).

5. Git was created as a solution for the Linux kernel’s source control needs. We talked about this story with Greg outside the podcast: it’s a fascinating story about how git was built and open-sourced after the Linux kernel group was unhappy with existing source control solutions.

In what is amusing: git has become the de facto source control product across tech thanks to products like GitHub and GitLab. The Linux kernel does not use GitHub – don’t forget, they already solved their source control workflow problems by writing git!

The Pragmatic Engineer deepdives relevant for this episode

What TPMs do and what software engineers can learn from them

The past and future of modern backend practices

Backstage: an open-source developer portal

Timestamps

(00:00) Intro

(02:23) How widespread is Linux?

(06:00) The difference in complexity in different devices powered by Linux

(09:20) What is the Linux kernel?

(14:00) Why trust is so important with the Linux kernel development

(16:02) A walk-through of a kernel change

(23:20) How Linux kernel development cycles work

(29:55) The testing process at Kernel and Kernel CI

(31:55) A case for the open source development process

(35:44) Linux kernel branches: Stable vs. development

(38:32) Challenges of maintaining older Linux code

(40:30) How Linux handles bug fixes

(44:40) The range of work Linux kernel engineers do

(48:33) Greg’s review process and its parallels with Uber’s RFC process

(51:48) Linux kernel within companies like IBM

(53:52) Why Linux is so widespread

(56:50) How Linux Kernel Institute runs without product managers

(1:02:01) The pros and cons of using Rust in Linux kernel

(1:09:55) How LLMs are utilized in bug fixes and coding in Linux

(1:12:13) The value of contributing to the Linux kernel or any open-source project

(1:16:40) Rapid fire round

A summary of the conversation The Linux kernel

The Linux kernel is around 40 million lines of code. The core kernel –the part every Linux platform runs – is about 5% of this. . The remaining code supports diverse hardware, drivers, devices, architectures, and chips.

A typical laptop runs approximately two to 2.5M million lines of kernel code; servers around 1.5M, while mobile devices around 4M.

The role of the kernel: abstract away underlying hardware and present a consistent interface to user space programs. This allows the applications to run on different hardware without modification.

A monolithic kernel

Drivers in Linux are part of the kernel

This is a monolithic architecture: all code, including drivers, operates in the same address space

The monolithic approach allows for more refactoring options and more code-sharing opportunities between drivers. This results in Linux drivers being, on average, one-third smaller than drivers in other operating systems because common functionalities can be identified and consolidated.

Do not break userspace. The core principle of Linux kernel development is to never intentionally break user space. This guarantee ensures that users can upgrade their kernel without fear of their existing applications crashing. Accidental breakages are treated as faults and are promptly addressed.

Linux kernel development process

Fixed 9-week cadence.

Following a release by Linus Torvalds, a two-week merge window opens.

During this merge window, maintainers submit all the new features that have been pending and proven to work in their respective development trees to Linus.

rc1: after the two-week merge window, Linus issues the first release candidate (rc1).

For the subsequent seven weeks, only bug fixes are accepted. No new features are introduced during this stabilization period; the focus is on regression fixes and reverting problematic changes.

Hierarchical structure of maintainers.

Around 4,000 developers contribute code every year

They send changes via email to maintainers responsible for specific kernel subsystems

Kernel subsystem maintainers then forward collections of accepted changes up the chain

Ultimately, these changes reach Linus for inclusion in the main kernel tree.

Trust is key in Linux kernel development. When a maintainer accepts code from a developer, they implicitly take responsibility for it. For critical parts of the kernel, maintainers need to have a high degree of confidence in the developer and the quality of their work, as the maintainer becomes accountable if the original developer disappears.

Email and git. These are the two tools used during development.

"Linux Next:" a separate development tree that integrates all the changes destined for the next kernel release on a daily basis. This allows for continuous testing and identification of potential integration issues.

QA and stable releases

Linux Next: automated testing. This includes building and booting the kernel across various architectures and virtual machines.

KernelCI: a project that provides a more extensive continuous integration infrastructure, running tests on a wider range of real hardware contributed by different labs.

The testing process involves a mix of automated tests and real-world usage by developers and testers. The "zero-day bot" automatically tests patches submitted to mailing lists.

Stable kernel releases: these are maintained independently of the main development branch. After each major kernel release by Linus, a stable branch is forked.

Greg and Sasha Levin maintain these stable branches. They issue new stable releases weekly, incorporating bug fixes that have first been merged into Linus's tree. This ensures that stable branches do not diverge from the main development line.

Long-term stable (LTS) kernels: Greg picks one kernel per year and supports it for an extended period, initially two years, sometimes up to six years. Android phones, for instance, often run on these older LTS kernels, which still receive backported bug fixes. Greg and Sasha concurrently maintain multiple LTS kernels.

Maintaining older codebases is more challenging. This is due to the ongoing evolution of the kernel. Changes made in newer versions to fix bugs might be difficult to backport to older, significantly diverged code. Context is often lost over time, making even seemingly simple backports complex.

Contributors

About 80% of kernel contributors are paid – by their employer! Companies invest in Linux development because it's often more cost-effective to contribute features and fixes than to develop their own operating systems.

Contributing to the Linux kernel is a valuable way for developers to also invest in their careers. It demonstrates the ability to collaborate, work with existing codebases, and solve real-world problems.

Core maintainers meet annually to discuss and refine the development process.

Rust support?

Most of Linux is written in C, but Rust is gaining momentum. Approximately 25,000 lines of Rust code are already in the kernel, primarily for bindings but also for some functionality like generating QR codes on kernel crashes.

Introducing Rust aims to improve memory safety in certain parts of the kernel. However, writing drivers in Rust presents challenges due to the need for bindings to the extensive C codebase and the different memory management models of C and Rust.

Memory safety in Rust primarily refers to the safety of object lifecycles and memory ownership, not necessarily the elimination of all bugs. Logic errors and even memory unsafety can still occur in Rust code.

The adoption of Rust is also driving improvements in the existing C codebase, as the need to create Rust bindings encourages a re-evaluation of C code for better safety and clarity.

Will the Linux kernel add Rust support?

There is resistance to introducing new languages from some core kernel developers, who prefer to maintain a single-language codebase.

Efforts are underway to write more drivers in Rust, including experimental GPU drivers. Rust can be particularly well-suited for simpler hardware drivers.

Governments increasingly mandate the use of memory-safe languages, which is another factor driving the adoption of Rust in Linux.

That said, the Linux kernel community is also actively working on improving the safety of existing C code through techniques like bounds checking and compiler extensions.

Why contribute to Linux, and how?

Building and testing the kernel locally is a prerequisite for submitting changes.

Contributing, even a single patch, offers significant professional benefits. It strengthens a developer's resume by demonstrating the ability to collaborate and work with complex, established codebases.

Contributing provides valuable learning opportunities, exposing devs to different perspectives, coding practices, and challenging technical problems.

Newcomers can find entry points by working on less critical parts of the kernel, such as fixing coding style issues or removing dead code in older drivers. A good place to start is Kernel Newbies

Where to find Greg Kroah-Hartman:

• Social: https://social.kernel.org/gregkh

• Website: http://www.kroah.com/log/about/

Mentions during the episode:

• Linux Kernel Foundation: https://www.linuxkernelfoundation.com/

• International Space Station: https://www.nasa.gov/international-space-station/

• Raspberry Pi: https://www.raspberrypi.com/

• GitHub: https://github.com/

• Kernel CI: https://kernelci.org/

• Linus Torvalds on LinkedIn: https://www.linkedin.com/in/linustorvalds/

• Engineering Planning with RFCs, Design Documents and ADRs: https://newsletter.pragmaticengineer.com/p/rfcs-and-design-docs

• A guide to the Kernel Development Process: https://docs.kernel.org/process/development-process.html

• Rust: https://www.rust-lang.org/

• The Linux Kernel Maintainer Summit: https://events.linuxfoundation.org/linux-kernel-maintainer-summit/

• Linux Braille Console: https://www.kernel.org/doc/html/v4.16/admin-guide/braille-console.html

• Code Complete: A Practical Handbook of Software Construction: https://www.amazon.com/Code-Complete-Practical-Handbook-Construction/dp/0735619670

• Kernel Newbies: https://kernelnewbies.org/

—

Production and marketing by Pen Name. For inquiries about sponsoring the podcast, email podcast@pragmaticengineer.com.

We’d like to know what tools, languages, frameworks and platforms you are using today. Which tools/frameworks/languages are popular and why? Which ones do engineers love and dislike the most at this moment in time?

With more than 950,000 tech professionals subscribed to this newsletter, we have a unique opportunity to take the industry’s pulse by finding out which tech stacks are typical – and which ones are less common.

So, we want to build a realistic picture of this – and share the findings in a special edition devoted to this big topic. But it’s only possible with input from you.

We’re asking for your help to answer the question: what’s in your tech stack? To help, please fill out this survey all about it. Doing so should only take between 5-15 minutes, covering the platform(s) you work on, the tooling you use, the custom tools you have built, and related topics.

The results will be published in a future edition of The Pragmatic Engineer. If you take part and fill out the survey, you will receive the full results early, plus some extra, exclusive analysis from myself and Elin.

This is the first time we’re running a survey that’s so ambitious – and we very much appreciate your help. Previous research we did included a reality check on AI tooling and what GenZ software engineers really think. This survey is even more ambitious – and the results should reveal people’s typical and atypical tooling choices, across the tech industry. You may even get inspiration for new and different tools, languages, and approaches to try out.

Fill the survey here

We plan to publish the findings in May.

Note: this week, the engineering deepdive article is being published on Thursday (20 March). Thank you so much if you contribute to the survey.

[Freedom of the Press Foundation]

Wired is going to stop paywalling articles that are primarily based on public records obtained through the Freedom of Information Act.

This approach makes a ton of sense:

"They’re called public records for a reason, after all. And access to public documents is more important than ever at this moment, with government websites and records disappearing, Elon Musk’s Department of Government Efficiency doing its best to operate outside the public’s view, and the National Archives in disarray."

Paywalls have long presented a challenge for service-based journalism: stories make the most impact when they're available to everybody, but newsrooms also need to cover their bills and make enough money to continue operations. When stories are based on public data, like FOIA requests, another level of public responsibility is added to the equation: these are public documents that belong to all of us.

I wish more online newsrooms would move to a patronage model (see The Guardian), but this isn't always possible. Someone always brings up micropayments in these conversations, but they do not work and have never worked. This hybrid model - public service articles for free, the rest behind the paywall - may point to a way forward.

#Media

[Link]

[Scharon Harding at Ars Technica]

Don't keep an Amazon Echo (or any smart speaker) in a room where you'll be having sensitive conversations, either with your family or on a work call. Particularly if you're a journalist or activist - but privacy is something everybody should be guarded about.

"Amazon said that Echo users will no longer be able to set their devices to process Alexa requests locally and, therefore, avoid sending voice recordings to Amazon’s cloud."

As the author points out, even if you trust Amazon (and, to be clear, you shouldn't trust any vendor with your private conversations), there's reason to worry:

"In 2023, Amazon agreed to pay $25 million in civil penalties over the revelation that it stored recordings of children’s interactions with Alexa forever. Adults also didn’t feel properly informed of Amazon’s inclination to keep Alexa recordings unless prompted not to until 2019—five years after the first Echo came out."

It was a nice idea, but it's time to give them a rest. There are plenty of use cases for smart agents, but they don't need to be listening to you all the time like you live in some kind of science fiction movie. You don't know who else is listening with them.

#Technology

[Link]

The FIDO Alliance has completed the CTAP 2.2 Specification. The closely-related third version of the W3C Web Authentication (WebAuthn) specification is also nearing final status; this WebAuthn Level 3 working draft is currently going through the review steps to become a W3C Recommendation.

So what’s new in the third versions?

Changes between CTAP 2.1 and CTAP 2.2 are:

Creation option with a prioritized list of supported attestation formats (attestationFormatsPreference) PersistentPinUvAuthToken State Set PIN Complexity Policy JSON-based Messages Hybrid Transports Large Blob Extension (largeBlob) PIN Complexity Extension (pinComplexityPolicy) HMAC Secret MakeCredential Extension (hmac-secret-mc) Third-Party Payment Authentication Extension (thirdPartyPayment)

Changes between WebAuthn Level 2 and the WebAuthn Level 3 working draft are described in the document’s Revision History.

Completing these V3 specifications represents important progress in our quest to free us from the password!

[Ghost]

Ghost has released its ActivityPub social web integration in beta:

"Today we're opening a public beta for our social web integration in Ghost. For the first time, any site on Ghost Pro can now try out ActivityPub."

Those of you who are, should. And everyone in the space should take a look. This is a really big deal. Congratulations to everyone involved on the Ghost team - and everyone in the Fediverse, who will see the benefit.

#Fediverse

[Link]

[Benedict Collins at TechRadar]

This was always inevitable, but in the current environment it makes sense that it's accelerating:

"Several major European tech companies are pushing for greater action from the European Union to reduce the bloc’s reliance on foreign-owned infrastructure by buying and building locally.

[...] Essentially, the European Union has become overly reliant on foreign-owned infrastructure - especially US Big Tech - and if nothing is done soon, EU countries will become subservient to foreign tech companies. The solution therefore is to foster growth at home."

The implication is that there's already a market here. Hosting in the US puts you at risk of certain kinds of subpoenas and other actions by the state, and the current political environment makes that even less desirable. (I've certainly personally had plenty of advice from security experts this year to not host in the US.) But if the only really great cloud hosting providers are US-based, that's a problem (even if they offer non-US hosting zones).

To be competitive here, the EU needs to consider privacy and freedom from surveillance as paramount values. That's not always been the case for it: there have always been voices who have pushed for things like backdoors in encryption and greater monitoring from police and security services. Those things will kill any EU effort to provide alternatives. The EU's great strengths in comparison to the US are greater openness and stronger protections of human rights; it should lean into those.

#Technology

[Link]

LIVE告知です。メタバース進化論の著者のバーチャル美少女ねむさんとYouTube Live を行います。

・配信日時：3月29日（土）21:00より1.5時間程度予定
・YouTube LIVEにて無料配信
・配信URL：「分人経済革命」ねむ ✕ 崎村夏彦 徹底対談【次世代ネットの扉を開く革命の起こしかた】

バーチャル美少女ねむさんは、黎明期の仮想世界で生きる「メタバース原住民」にして、その文化を伝える「メタバース文化エバンジェリスト」として活動しておられます。「バーチャルでなりたい自分になる」をテーマに2017年から美少女アイドルとして活動している自称・世界最古の個人系VTuber。メタバースの革命性を論じた著書『メタバース進化論』（2022年、技術評論社）で「ITエンジニア本大賞2023」ビジネス書部門”大賞”を受賞。国連の国際会議「IGF京都2023」でも登壇。MoguLive VTuber Award 2023では「今年最も輝いたVTuber」に選出されています。2024年には定性調査レポート「メタバースでのアイデンティティ（Nem x Mila, 2024）」を発表。産総研「アバター国際標準化の国内検討委員会」4委員にも就任されておられます。

ねむさんが課題として捉えられていることに分人経済が新たな経済圏に昇華するためのID認証・経済・納税の仕組みがあります。たとえば、

バーチャルでアバターの形でしか会わない取引先にどうしてマイナンバーカードの表裏をコピーして郵送しなければならないのか？

もちろん、適正な課税のためには税務当局がマイナンバーを使って支払いを把握することが必要であることは論を待ちません。しかし、だからといってマイナンバーカードのオモテ裏のコピーを郵送したりして身元確認情報を拡散させたり、望まない名寄せが行われるリスクをとる必要があるのでしょうか？多くのVTuberの方にしてみれば、「顔バレ」を含む「身バレ」は最大のリスクの一つです。こうしたリスクを抑えたうえで、ひとびとが経済活動を行い、一方税務当局は本人のマイナンバーを抑えることができて適正な課税をする仕組みはつくれないのでしょうか？これは実はVTuberだけでなく、ひろく一般の方々にかんしても言えることです。

こうしたことについて、目標1.5時間でいろいろお話をしたいと思います。

ねむさん曰く：

次世代インターネットの扉を開くという「分人経済革命」とは果たして何なのか！？ 仮想空間「メタバース」ではアバターによって複数の自分（分人）を切り替え、自己の魂の在り方と社会との界面を能動的にデザインできる新時代が遂に実現しつつある。一方でそれを新たな経済圏に昇華するためのID認証・経済・納税の仕組みは全くの未整備なのが現状だ。インターネットにおけるデジタルアイデンティティの標準化に長年取り組んできたOpenID Foundation理事長「崎村夏彦」とメタバース住人にしてVTuber/作家「バーチャル美少女ねむ」が3/29にYouTube LIVEでこの問題を徹底対談！ 次世代インターネットの革命がここから始まる！？（出所：【告知】3/29「分人経済革命」ねむ ✕ 崎村夏彦 徹底対談【次世代ネットの扉を開く革命の起こしかた】）

それでは、当日おあいしましょう！

【無事終わりました】

とても楽しかったです。対談中話題になった点などについてスライドにまとめました。以下のリンクから御覧ください

分人経済革命の可能性と実現に向けた一考察

[Om Malik]

This is an ongoing trend. Om Malik has now moved to sharing on his blog first rather than posting directly to social media:

"The inspiration for the newly rebooted “daily blog” comes from Dave Winer, who maintains a “Links” blog. I’ve been using his new project, Wordland, for publishing to the “links” blog, in addition to using MarsEdit. I have also taken a cue from Marc Weidenbaum. The plan is to use this as a permanent archive for everything I share on social media. From here, I’ll route information to relevant channels — mobile apps, social networks and RSS feeds. The experiment continues."

This is, of course, very much in line with indie web sensibilities. The more social media fragments and turns into a toxic place to be, the more people will carve out spaces that they truly own on the web. As well they should. I'm excited to see this; if you haven't made the leap to posting on your own site first yet, the best time to start is now.

#Indieweb

[Link]

[Mike Monteiro]

On whether it's always been this bad in America:

"America was born in blood, genocide, kidnapping, and death. Behind every story we were told about America’s greatness, there is a secret second painting. A thing America doesn’t want you to see. Behind every “from sea to shining sea” there is a Trail of Tears. Behind every first pitch at Dodger Stadium there’s the destruction of Chavez Ravine. Behind every moonshot there’s a Nazi V2 rocket. Behind every “liberation of the camps” there’s a Nakba. Behind every interstate highway system there’s the destruction of a thousand Black and immigrant neighborhoods."

Although it's front and center, much of what we're experiencing isn't actually new for lots of communities across America. Treating the current moment as completely new doesn't serve us: it erases those struggles and historic injustices. That's not to say that everything is good right now - it's obviously not - but the road that led us here is clearly marked throughout American history.

#Democracy

[Link]

My first love was self-sovereign distributed data, where each person owns and controls their data, hosting it wherever they choose and permissioning it under their own terms. But I got lost in the complexity of building a robust distributed identity infrastructure. How can you give permission to someone if you can’t 'name' them in a way that is verifiable and resistant to subversion? There's no point in saying "only John can access this" if Tom can show up and convincingly say, "I'm John."

This issue isn’t theoretical—many modern digital problems stem from weak identity foundations. Take email, for example. SMTP, the core protocol, evolved without a strong sense of identity. Had we designed email with a robust identity layer—and maybe a little reputation—spam might have been less rampant. Instead, we've had to patch identity onto email systems, mostly at the DNS layer. Could better choices early on have changed the landscape of digital trust?

As we enter the era of AI and Personal AI, this challenge resurfaces. We will increasingly rely on agents to interact, assist, and even make decisions on our behalf. But how can we trust these agents? How do we know they are who they claim to be, and whose interests they truly serve? When I ask my AI how to unwind after a long day, it might suggest a refreshing Diet Coke. But is that suggestion rooted in understanding my preferences, or is it influenced by unseen commercial incentives?

Recognition and Identity in AI

In the animal world, intelligence is often measured by the ability to recognize oneself and others. The mirror test is a classic example—when an animal identifies itself in a reflection, it demonstrates a form of self-awareness. Similarly, recognizing specific others—distinguishing one individual from another—marks advanced cognitive development.

AI, in contrast, remains limited in this capacity. While AI excels at pattern recognition, it lacks the ability to form a persistent sense of identity, either of itself or others. This limitation restricts its ability to build trust and context in interactions. Without a foundation for recognizing specific entities, AI systems risk becoming tools of confusion or exploitation.

Embedding Identity Systems into AI

One solution is to deeply embed identity frameworks into AI architectures from the outset. Decentralized Identifiers (DIDs), Verifiable Credentials (VCs), and similar systems could provide AI with a structured way to "recognize" and differentiate entities.

Persistent Identity Chains: AI could track verifiable chains of identity, ensuring that when it reports information—like "Brad says buy this stock"—it can verify that it truly came from the Brad you trust.

Verification of Origin: By leveraging cryptographically verifiable credentials, AI can ensure that information hasn’t been tampered with and originates from a trusted source.

Reputation Frameworks: Identity systems could incorporate reputation mechanisms, helping AI prioritize information from sources that consistently meet a trust threshold.

Chain of Custody: AI could provide transparency on how information was received and processed, ensuring that its recommendations are based on data with verifiable origins.

The Path to Trusted AI

Trustworthy AI isn’t about making machines socially aware; it’s about ensuring that humans can trust the chain of custody behind AI-generated insights. When AI states that "Brad recommends this action," it should be able to prove that the recommendation came from the right "Brad"—the person you trust, not an imposter or manipulated data source.

The real question is: How do we create systems where AI is not just technically accurate but verifiably trustworthy? In an era where decisions increasingly rely on AI advice, embedding identity systems at the core isn’t just beneficial—it’s fundamental.

The Pulse is a series covering insights, patterns, and trends within Big Tech and startups. Notice an interesting event or trend? Send me a message.

Today, we cover:

Industry pulse. Google’s impressive AI launches, AI agent Manus makes a splash, companies evaluating AI models internally, Cursor worth $10B, Klarna’s upcoming IPO, and more.

Typescript compiler rewritten in Go. In an impressive feat, the Typescript team rewrote their compiler, and made it 10x faster. But why use Go for the job? It’s a curious choice, given Microsoft’s heavy investment in C# – which is also a performant language. Engineers who executed the rewrite explain their unconventional choice.

OpenAI’s dev tools to build agents: more complexity? OpenAI shared its tooling for building AI agents. My first impressions are that it feels pretty fragmented. Perhaps this is to be expected with fast-moving technology, where capabilities keep being added?

Apple and AI: clumsy fumbling, or a long-term strategy? Apple promised a new, improved Siri by the end of 2024, yet that could now be on hold until 2027. Are things less bad as they seem, due to Apple owning the iPhone hardware and iOS software platforms?

Second chances for interview no-shows? A candidate did not show up for a product engineer interview, but their AI note taker did. As a hiring manager, what would you do? Cofounder Diwakar Kaushik did something unexpected.

Industry Pulse Google’s impressive AI launches

Google has had an eventful weeks with new AI products:

Gemma 3: A model that runs on just a single NVIDIA H100 GPU, and requires around 10x less compute than models with similar capability, is impressive – and will surely gain adoption to its generally permissive license, and can be self hosted, as well as supporting 140 languages (!!). As a note, Gemma 3 is kind of open, but not fully open: the license allows commercial use, but has restrictions on use cases, such as violating any applicable law or use cases listed in the policy.

Gemini Robotics: a vision-language-action model that can directly control robots, built on top of Gemini 2.0. This 3-minute video summarizes the impressive capabilities.

A robot arm can interpret and execute a command, even while an operator moves objects around. Source: Google

Google is getting a lot less of the AI spotlight behind OpenAI and Anthropic, but seems to be pulling ahead in other areas. In robotics and AI, it might become pioneering, and there’s also Waymo's self-driving cars – which is also part of Google. Is Google quietly becoming a ML and AI leader under the radar?

AI agent Manus makes a splash

Read more

GOV.UK Wallet とは

GOV.UK Walletは、デジタル運転免許証のコンテキストで1月21日に明らかになった56イギリス政府が開発中のデジタル文書管理アプリで、2025年2月5日にはピーター・カイル科学・イノベーション・技術担当国務長官によりその詳細が発表が行われました7。この発表は、イギリス政府のデジタル化推進の一環として行われ、身分証明書のデジタル版をスマートフォンで携帯できる新しいアプリの導入を目指しています。このアプリを使用することで、政府発行の文書をスマートフォンに安全に保存し、必要に応じて提示することができるようになります8。

主な特徴は以下の通りです：

政府発行のデジタル文書を安全に保存 年齢、身元、サービス利用資格の証明に使用可能 紙や物理的なカードと同様に使用可能 非政府発行の文書（電車やコンサートのチケットなど）は追加不可 使用は任意

GOV.UK Walletの利用開始は2025年中を予定しています。最初に利用可能になる文書はHM軍隊退役軍人カードで、その後運転免許証などの他の政府文書も追加される予定です9。

セキュリティ面では、GOV.UK One Loginを使用してアクセスする必要があり、他人が文書を使用したりアクセスしたりすることを防ぎます。また、スマートフォンに組み込まれた顔認識技術などのセキュリティ機能を活用する予定です10。

GOV.UK One Loginとは？

GOV.UK One Loginは、イギリス政府が提供するオンライン認証システムです。このシステムを使用することで、ユーザーは単一のアカウントで複数の政府サービスにアクセスできるようになります。主な特徴は以下の通りです：

単一のメールアドレスとパスワードで複数の政府サービスにアクセス可能11 二要素認証を使用してセキュリティを強化12 本人確認機能を提供し、ユーザーの身元を確認 政府サービスへのアクセスを簡素化し、効率化

現在、GOV.UK One Loginは一部の政府サービスでのみ使用可能ですが、将来的にはGOV.UKのすべてのサービスで使用できるようになる予定です13。利用可能なサービスには、退役軍人カードの申請、輸出入ライセンスの申請、教師資格の申請などが含まれています。

ユーザーは、GOV.UK One Loginを通じて自身の情報を管理し、利用したサービスを確認することができます5。また、セキュリティコードの取得方法として、テキストメッセージまたは認証アプリを選択できます6。

GOV.UK One Loginは、政府のデジタルサービス（GDS）によって提供され、キャビネットオフィスがデータ管理者として機能しています14。このシステムは、ユーザーの利便性向上と政府サービスへのアクセス簡素化を目的としています。

GOV.UK Login のプロトコル

GOV.UK One LoginはOpenID Connect (OIDC)15に準拠したサービスです16。OpenID Connectは、OAuth 2.0プロトコルの上に構築された認証レイヤーであり、OAuth 2.0を拡張して標準化された方法で認証情報を提供します。

具体的には、GOV.UK One Loginは以下の特徴を持っています：

OpenID Connect準拠: OIDCプロトコルを使用して、ユーザー認証と属性の共有を行います。 認証フロー: 標準的なOAuth 2.0/OIDC認証コードフローを実装しています。これには以下のステップが含まれます： /authorize エンドポイントへの認可リクエスト 認可コードの発行 /token エンドポイントでのトークン交換 IDトークンとアクセストークンの取得 /userinfo エンドポイントでのユーザー情報の取得 セキュリティ強化: JWT-secured OAuth 2.0 Authorization Request (JAR)17を使用して、認証リクエストパラメータのセキュリティを向上させています。

ちなみにですが、OpenID Connect も JAR もわたしが主著者なので、なんとも喜ばしいことです。

Stream the Latest Episode

Listen and watch now on YouTube, Spotify and Apple . See the episode transcript at the top of this page, and a summary at the bottom.

Brought to You By

Sentry — Error and performance monitoring for developers.

The Software Engineer’s Guidebook: Written by me (Gergely) – now out in audio form as well.

—

In This Episode

In today’s episode of The Pragmatic Engineer, I am joined by former Uber colleague, Gautam Korlam. Gautam is the Co-Founder of Gitar, an agentic AI startup that automates code maintenance. Gautam was mobile engineer no. 9 at Uber and founding engineer for the mobile platform team – and so he learned a few things about scaling up engineering teams.

We talk about:

• How Gautam accidentally deleted Uber’s Java monorepo – really!

• Uber's unique engineering stack and why custom solutions like SubmitQueue were built in-house

• Monorepo: the benefits and downsides of this approach

• From Engineer II to Principal Engineer at Uber: Gautam’s career trajectory

• Practical strategies for building trust and gaining social capital

• How the platform team at Uber operated with a product-focused mindset

• Vibe coding: why it helps with quick prototyping

• How AI tools are changing developer experience and productivity

• Important skills for devs to pick up to remain valuable as AI tools spread

• And more!

Takeaways

Interesting parts of the conversation:

1. Submit Queue: Uber built a complex merge system to deal with the large number of commits, where each commit had to run long-running CI tests. It’s a problem that smaller and mid-sized companies don’t have, but Uber had: and so they scratched their own itch.

2. Local Developer Analytics (LDA): years ago, Uber started to measure the experience that devs had. Things like: how long did a build take, locally? How much CPU is used? They used this data to improve internal tooling.

3. Developer experience as a product team. Gautam’s team operated like a classic product team: except their customers were Uber’s internal developers. Gautam believes this is how all successful platform teams should work.

4. AI changing software development: this is happening. “Vibe coding” leads to faster prototyping. Gautam believes junior engineers will thrive with AI tools because they will hit the ground running faster, and will be free of biases that hold back more experienced developers.

The Pragmatic Engineer deepdives relevant for this episode

The Platform and Program split at Uber

How Uber is measuring engineering productivity

Inside Uber’s move to the Cloud

How Uber built its observability platform

Software Architect Archetypes

Timestamps

(00:00) Intro

(02:11) How Gautam accidentally deleted Uber’s Java Monorepo

(05:40) The impact of Gautam’s mistake

(06:35) Uber’s unique engineering stack

(10:15) Uber’s SubmitQueue

(12:44) Why Uber moved to a monorepo

(16:30) The downsides of a monorepo

(18:35) Measurement products built in-house

(20:20) Measuring developer productivity and happiness

(22:52) How Devpods improved developer productivity

(27:37) The challenges with cloud development environments

(29:10) Gautam’s journey from Eng II to Principal Engineer

(32:00) Building trust and gaining social capital

(36:17) An explanation of Principal Engineer at Uber—and the archetypes at Uber

(45:07) The platform and program split at Uber

(48:15) How Gautam and his team supported their internal users

(52:50) Gautam’s thoughts on developer productivity

(59:10) How AI enhances productivity, its limitations, and the rise of agentic AI

(1:04:00) An explanation of Vibe coding

(1:07:34) An overview of Gitar and all it can help developers with

(1:10:44) Top skills to cultivate to add value and stay relevant

(1:17:00) Rapid fire round

A summary of the conversation Uber's engineering culture and tools

Gautam joined Uber in 2014 as an Android engineer. Back then, there were not even unit tests. Gautam wrote the first Android test, and set up Artifactory.

Uber built much of its engineering stack in-house because the cloud-native SaaS products were not built for their scale.

Even in the earlier years, Uber saw about one commit every minute – and the platforms at the time could not handle this (especially when considering that CI took 10-30 minutes to run!)

Build time was a problem. During the iOS and Android app rewrite in 2016, build times became very long. Gautam worked on getting it under control.

Submit Queue was a way to guarantee a green main. It serialised incoming commits to ensure they played nicely together. The company published a paper on this novel solution. Submit Queue tested changes and considered cross-dependencies between different commits. Machine learning models estimated potential failures and speculatively tried paths that might be green.

Monorepo: after starting as polyrepos, Uber moved to monorepos

Uber started with separate repos for Rider, Driver, and other apps (eg Eats). As the company grew, each team wanted its own repository, resulting in hundreds of repos. It became painful to upgrade and bump libraries.

The iOS team moved to a monorepo, followed by Android. The gains in productivity were massive because there was no need to bump libraries. This move really helped with standardisation.

The biggest initial pushback was that teams could no longer break an API without worrying about the consequences. There was concern about slower builds. In the end, the dev platform team solved these concerns.

Developer experience and dev productivity

Local Developer Analytics (LDA): an internal Uber system that ran on developers' machines and collected information about their systems. It integrated deeply into CLI tools and IDEs, tracked which files developers accessed most, and identified files with the most bugs. LDA helped identify bottlenecks in the development funnel.

Developer surveys: Uber ran these regularly. The dev tooling NPS went from negative 50 to positive 8 during Gautam’s tenure.

Things that made for a better developer experience: minimizing time to review code, time to build code, and reducing time spent in meetings.

DevPods: another internal Uber tool. These are basically cloud developer environments. They contain a container of code, build system artefacts and IDE indices in the cloud. DevPods make context switching quick.

Previously, onboarding involved running a bootstrapping script. The script was hard to maintain. Dev Pods moved the development stuff into a container. The containers can be huge.

Career growth to Principal Engineer

Going deep: getting into a niche and going deep can help over the long term, especially in areas others may not want to do.

Introspect regularly: Every two years, Gautam did some introspection to see if he was doing what he wanted and what could challenge him more.

Social capital and mentorship: these become very important at a big company. It helps to have connections. Helping people builds social capital. Gautam would drop everything to help people with their dev environment problems. He also held office hours on a regular basis, offering to help anyone who showed up.

Understand the business: Principal engineers need to understanding how engineering meets business, rather than just pure coding. It helps if you enjoy diving into this area – as well as if you like talking with people!

More of a peer relationship with your manager. As engineers grow in seniority, they become more like a peer to their manager and help their manager get stuff done. The relationship is more like a “peer” than a “boss”.

Tip for managers of senior+ devs: give them agency, check in often, and make sure they are unblocked.

The impact of AI on software development

Autocomplete: an obvious use case. It helps one type less and think more.

“Vibe coding”: AI allows you to explore more paths and experiment faster.

How AI impacts engineers

Controversial, but Gauam believes that junior engineers are going to thrive because they are coming up with new knowledge and new ways of working with AI tools. They do not have the biases of working a particular way.

The “generalist engineer” is going to be more in-demand, looking ahead

CS knowledge remains important.

When things go wrong, it is important to understand why they went wrong. This requires strong computer science fundamentals and system knowledge.

Where to find Gautam Korlam:

• X: https://x.com/kageiit

• LinkedIn: https://www.linkedin.com/in/gautamkorlam/

Mentions during the episode:

• Bypassing Large Diffs in SubmitQueue: https://www.uber.com/blog/bypassing-large-diffs-in-submitqueue/

• Jenkins: https://en.wikipedia.org/wiki/Jenkins_(software)

• Devpods: https://www.uber.com/blog/devpod-improving-developer-productivity-at-uber/

• JetBrains: https://www.jetbrains.com/

• Cloud Development Environments: https://newsletter.pragmaticengineer.com/p/cloud-development-environments

• Why are Cloud Development Environments Spiking in Popularity, Now?: https://blog.pragmaticengineer.com/why-are-cloud-development-environments-spiking-in-popularity-now/

• “The Coding Machine” at Meta with Michael Novati: https://newsletter.pragmaticengineer.com/p/the-coding-machine-at-meta

• Software Architect Archetypes: https://newsletter.pragmaticengineer.com/p/software-architect-archetypes

• The Platform and Program Split at Uber: A Milestone Special: https://newsletter.pragmaticengineer.com/p/the-platform-and-program-split-at

• What is Vibe Coding? How Creators Can Build Software Without Writing Code: https://alitu.com/creator/workflow/what-is-vibe-coding/

• WhatsApp: https://www.whatsapp.com/

• Rust: https://www.rust-lang.org/

• I am excited to introduce Jimy by Gitar - The agentic AI for building better software: https://www.linkedin.com/posts/gautamkorlam_i-am-excited-to-introduce-jimy-by-gitar-activity-7297713117927481344-0G4l/

• Cursor: https://www.cursor.com/

• Claude: https://claude.ai/

• Deepseek: https://www.deepseek.com/

• Head First Design Patterns: A Brain-Friendly Guide: https://www.amazon.com/Head-First-Design-Patterns-Brain-Friendly/dp/0596007124

—

Production and marketing by Pen Name. For inquiries about sponsoring the podcast, email podcast@pragmaticengineer.com.

デジタル技術の進化により、私たちの個人情報やデータへの依存がますます高まっています。しかし、多くの政府提供IDシステムは、ユーザーのプライバシーや自由を十分に考慮していないことがあります。その中で、ユタ州の「SB260法案」10は、他のシステムとは異なるアプローチを取り、ユーザーのプライバシー保護とデータ管理の自主性を根底から強化していると話題になっています(Chang(2025)他)。そこで、本記事では、その特徴と他のデジタルIDシステムとの差別化点について検討してみます。

SB260法案のユニークな特徴

SB260法案が他の政府提供IDシステムと大きく異なる点は、以下の特徴にあります。

個人の自主性を尊重
SB260では、「州が個人のアイデンティティを定義しない」と明確に規定されています (§ 63A-16-1202)。そして「州は特定の状況において、個人のアイデンティティを認識し承認することがある」とされています。従来の多くの政府IDは、政府が個人のアイデンティティを定義し、認識する権限を持っており、個人の存在は政府の承認に依存していました。これとSB260の新しいアプローチは根本的に異なります。SB260では、人間はすでに自分自身の権利で存在していることを認識し、州の役割は人間が認識され、法で規定された方法で州の承認を管理できるよう支援を提供する支援者という形になります。SB260は個人の自主性に重点を置いています 。前者のアプローチだと、政府に認識されないことによって存在が否定されうるのに対し、後者ではそれがおきません。

監視とデータ共有の禁止
SB260では、デジタルIDが提示された際、政府機関やその他の関係者による監視や情報共有を厳しく禁止しています。例えば、提示されたID情報を、提示を受けた企業などがマーケティングや監視目的で使用することが許されません。このガイドラインは、ユーザーデータが不必要に共有されることを防ぎ、データ管理の透明性を促進します 。

選択的な情報開示
多くの物理的な身分証明書では、住所や年齢といったすべての情報が開示されますが、SB260は「選択的開示」を可能にする技術を採用するとしています。例えば、年齢確認が必要な場合でも、実際の誕生日や住所を共有する必要がありません。これにより、個人情報が不必要に漏洩することを防ぎます 。

デバイスの提供を強制しない
SB260は、デジタルアイデンティティ確認の際にモバイル端末を提供することを強制しないと規定しています。この規定は、法執行機関や他の政府関係者が端末へのアクセスを求める可能性を排除し、端末内の個人データの保護を保証します 。このあたりは、デバイスを確認者が預かって、あんなことやこんなことをできるような制度とだいぶ違うと言えます。

デジタルアイデンティティの利用は完全に任意
他のアイデンティティシステムとは異なり、SB260ではデジタルアイデンティティの利用が完全に任意であることが強調されています。政府機関はデジタルアイデンティティを強制したり、使用を推奨するためのインセンティブを提供することはできません。これにより、技術に不慣れな住民やデジタルツールの使用を望まない住民への配慮がなされています 。18

他のIDシステムとの比較：何が違うのか？

多くの政府提供IDでは、監視とデータ収集が懸念されています。一部のシステムでは、個人情報がマーケティングやデータ解析に利用される可能性がありますが、SB260はこれを明確に禁止しています。また、他のシステムではデジタルIDが義務化される場合があり、ユーザーには選択肢がありません。一方で、SB260は完全に任意のシステムであり、ユーザー中心の設計が際立っています。

その結果、SB260はプライバシーを優先し、ユーザーのデータ管理を完全に可能にする未来型のデジタルIDシステムとして注目されています。このユーザー主導型のアプローチは、アメリカ全土だけでなく、世界中のデジタルID開発にも影響を与える可能性があります 。

結論：デジタル時代の新しい参照点

ユタ州のSB260法案は、デジタルIDの未来を形作る重要な節目と言えます。ユーザーの自主性、プライバシーの保護、選択の自由を包括的に保証するこの法案は、他の政府提供IDシステムとは一線を画すものです。

文書の出来自体に関して言うと、Identityの定義が「”Identity” means any attribute used to identify or distinguish a specific individual.」になっていたり、「(a)each individual has a unique identity;」となっているところなど、規格的な見方をするといろいろと気になる点はあります。が、SB260は、個人の自由とデータ管理の新しい基準を設定するだけでなく、技術と法律が調和してユーザー本位の世界を実現するモデルとして広く認知され、今後デジタルIDを制度化していく各種法域において、デジタル時代における安心・安全なIDシステムということに関して、何らかの影響を与えていくことでしょう。これを受けて、アメリカの各州がどう動くか、目がはなせません。

（参考資料) Chang, Wayne. (2025). Utah’s Digital ID Bill SB260 is the New Frontier for User-Controlled Identity. SpruceID, https://blog.spruceid.com/utahs-digital-id-law-sb260-is-the-new-frontier-for-user-controlled-identity/ Utah State Legislature. (2025). S.B. 260 Individual Digital Identity Amendments. https://le.utah.gov/~2025/bills/static/SB0260.html

Imagine you’re the cofounder of a startup hiring its first few software engineers, and there’s a candidate who smashes the technical interview, and is the only one to complete the full coding interview – doing so with time to spare. Their communication style is a bit unconventional, but this could be explained by language differences, and isn’t a red flag. So, the promising candidate gets a thumbs up, and pretty soon they’re on screen in a non-technical final interview with the other cofounder, via video.

Then things get weird. Your cofounder pings you mid-interview to report that the candidate from Poland speaks no Polish whatsoever, and also that there is something just not right about their appearance on screen. The recruitment of this candidate gets derailed by suspicious details which erode everyone’s confidence in making the hire. Soon afterwards, the same thing happens again with another candidate.

Later, you discover that two imposters hiding behind deepfake avatars almost succeeded in tricking your startup into hiring them. This may sound like the stuff of fiction, but it really did happen to a startup called Vidoc Security, recently. Fortunately, they caught the AI impostors – and the second time it happened they got video evidence.

To find out more about this unsettling, fascinating episode from the intersection of cutting-edge AI and tech recruitment, the Pragmatic Engineer sat down with Vidoc cofounder, Dawid Moczadło. Today, we cover:

The first AI faker. Vidoc Security nearly made an offer to a fake candidate, but their back story raised too many questions.

The second AI faker. The next time a job candidate raised similar suspicions, the team was prepared and confronted the faker during the call – and recorded it.

How to avoid being tricked by AI candidates. Take the risk seriously, have applicants turn off video filters and verify that they do this, record interviews, and get firm proof of identity before any offer.

Foreign state interference? There’s evidence that many AI candidates could be part of a coordinated governmental operation targeting hundreds of western tech businesses. Full-remote workplaces are the most targeted.

Return of in-person final rounds? This looks an obvious consequence of these incidents.

New AI risks for tech businesses. Remote interviews may have to change, while devs also risk introducing security vulnerabilities by accepting AI suggestions without critique.

Hiring funnel. The story began with this job posting for a backend engineer. The startup shares its hiring funnel, giving a sense of how competitive full-remote startup positions are, currently. As context: from 500 applications, two hires have been made – and Vidoc is still actively recruiting for this position.

Catching an AI imposter: the candidate (left) refused to place their hand in front of their face because it would blow their AI cover. On the right, the interviewer illustrates the request

Since every candidate in this article is a cheater with an AI-generated mask of a different face, and a false professional identity, we share all the made-up resumes, CVs, videos, and photos, to give a sense of how things played out. If you’re currently hiring, or plan to, the nature and sophistication of the fake-applicant scam targeting this startup provides food for thought.

For more tips in detecting fake applicants, you can also check this handy PDF guide, created by the Vidoc engineering team.

1. The first AI faker

Vidoc Security is a security provider offering automated code reviews to detect security issues. The idea for the company came from two security engineers and ethical hackers, Dawid Moczadło and Klaudia Kloc. Previously, they hacked top tech companies like Meta and then disclosed vulnerabilities to those places in order to collect bounties, and get on those companies’ ethical hacking leaderboards.

With the rise of LLMs in the past couple of years, Dawid and Klaudia spotted an opportunity to create a tool that works in the same way as they searched for security vulnerabilities: looking across the broader codebase, checking how components interact, which parts could be insecure, and more. Basically, an LLM can take their expertise about what works to hack well-designed systems, and build a tool with some of the same know-how.

Their idea attracted investors, and Dawid and Klaudia raised a $600K seed round in 2023, and a further $2.5M in seed funding in August 2024. With this seed funding in the bank: the company began hiring. They posted a job ad for a backend engineer, and started to interview candidates. (We share details on the exact hiring funnel and statistics below, in “The hiring funnel”)

One promising candidate was called Makary Krol. His LinkedIn profile is still active:

Fake profile. Source: LinkedIn

Below is a step-by-step summary of how the recruitment process for the imposter candidate went, based on Vidoc Security's records, including the bogus resume. By the end of step 5, the team were certain they were the target of a scam.

1. Resume screening: ✅Resume looks solid:

Page 1 of 2 from the fake resume. The resume looked convincing Page 2 of 2 of the fake, but convincing resume

2. First-round screening. ✅⚠️ A 15-30 minute call with Paulina, head of operations. It was a bit odd that the candidate did not speak any Polish, but was based there and graduated from Warsaw University of Technology. He spoke in broken English, and with a very strong accent that sounded Asian, but these weren’t warning big red flags, and the candidate sounded motivated.

3. Hiring manager interview. ✅⚠️ The candidate was clearly well-rounded and a technical screening was the obvious next step. Dawid’s only “yellow flag” was that their communication skills were poor, but he figured a technical interview would be a chance to show their core coding and technical skills.

4. Technical interview. ✅✅The candidate absolutely smashed it, being the first to finish all coding tasks and follow-up questions in the allocated time of 2 hours, which hadn’t happened before. Dawid was surprised by how competent they were at coding and technical problem solving. The coding abilities of this candidate were definitely not fake: they were a seasoned, very capable engineer.

5. Final hiring manager interview. ‼️⚠️ This was a non-technical interview with cofounder, Klaudia, who dug into the specifics of their background and grew suspicious. The candidate gave some details about previous positions, but she increasingly found herself disbelieving their back story and resume, the more time she spent with the candidate. Dawid shared the suspicion and they became certain that the figure on screen was far from what they claimed to be.

By the end of the recruitment process, Vidoc believed they had nearly been played, and had come worryingly close to extending an offer to a fake candidate using a false identity in their documents, and an AI filter to mask their face on screen. However, they had no evidence of this, and didn’t record the interviews, so had only their impressions and notes.

When Dawid recounted this episode to peers, he was met with disbelief. Founder friends – along with most other people – thought the team were overreacting and misguided. Eventually, Dawid stopped sharing the story and began to doubt the team’s suspicions about the candidate.

After the incident, Vidoc added an onsite interview as a final step in the recruitment process, and held a retrospective to figure out how to avoid something similar again. Interestingly, there wasn’t much that could be done about the potential risks of remote interviews being taken by applicants pretending to be someone else.

2. The second AI faker

Two months after the suspected fake job candidate episode, another promising applicant appeared, going by the name Bratislav Rakočević, located in Serbia, with a university education there. His LinkedIn was pretty basic:

Another entirely made-up profile. Source: LinkedIn

This application looked good, so the company ran the usual recruitment process, below (including the candidate’s made-up resume):

1. Resume screening: ✅⚠️ Solid on technologies, but with too many buzzwords for their current, self-employed role. Of course, it’s not unusual that some devs don’t thrive at writing resumes. Their credentials merited a call, especially as the “candidate” had “worked” at HubSpot and Woosh:

Page 1 of 2 of another resume that later turned out to be fake Page 2 of 2 of another resume that turned out to be fake

2. First-round screening. ✅⚠️ Communication was a bit choppy — then again, we’re talking about a non-native English speaker. The candidate did not speak Serbian, despite graduating from the University of Kragujevac, in Serbia. Once again, they sounded motivated, so proceeded to the next round.

3. Hiring manager interview: AI filter exposed‼️‼️Dawid conducted this interview, and immediately felt that something was off about the candidate. First of all, Dawid felt the candidate on the call looked somewhat unlike their low-resolution LinkedIn profile:

First suspicion: Dawid felt the interviewee looked dissimilar to their profile picture

As a rule, Vidoc never records interviews, but the team had been burnt once before and had faced scepticism about it from peers afterwards. So he hit record, and asked the candidate to make a simple gesture which bamboozles the current crop of AI filters:

“Can you take your hand and put it in front of your face and cover it partially?”

Candidate (on the left) declines to place hand in front of face as the interviewer requests (right)

The candidate refused, so Dawid ended the conversation. Watch the full video of the encounter.

Later, several people pointed out that the AI mask looked uncannily similar to a prominent Polish politician named Sławomir Mentzen:

Was this deepfake trained on video footage of a Polish politician?

Looking back on these twin incidents, the team suspects the same individual may have been behind the AI disguises of both fake candidates because:

Their voices sounded almost identical

The second candidate answered questions with surprising confidence – almost like he’d been asked them before

…a hunch that they had spoken before

The AI filter for the second faker was of much lower quality than the first. The first candidate slipped through undetected through several interviews, despite almost certainly using an AI filter as well. Dawid said the second candidate’s filter looked obvious on the screen – but on that occasion the Vidoc Security team were on high alert for any possible deception. It’s worth remembering that AI filters improve with time, so spotting a good one won’t always be easy.

3. How to avoid being tricked by AI candidates

How can tech companies hiring engineers protect themselves, and prevent AI-disguised candidates? Some suggestions:

This is not a drill; take the threat seriously: If you think your company is too small to fall victim, think again. Vidoc Security was just a two-person startup in February 2023 with no funding raised, when it was first targeted. The first fake profile cold emailed them on LinkedIn to ask if they were hiring.

It’s possible there is a large, state-level operation running a pool of bogus applicants who hide behind fake resumes and AI filters in video interviews. If this operation can find the smallest of startups to target, it’s safe to assume they could apply to any other company, as well.

Don’t assume all job seekers on local job boards are genuine. Vidoc posted on local job boards, like the one in Poland. But whoever was behind this operation figured out that it’s profitable to pose as Serbian, Polish, and other eastern European profiles, and to apply via small, low-profile job boards. I didn’t even know about the job board “JustJoinIT” (popular in Poland) but the fakers were already active on it.

Ask candidates to turn off all filters – including background ones. Filters that blur or replace the background are ubiquitous these days. During a video interview, it’s fair to ask a candidate to turn off all filters. After that, if there are still unusual minor details on their face or the background, then it’s safe to assume that a candidate did not turn off their filters. In an interview for a full-remote position, this could be justification enough to terminate the call.

Ask the candidate to do things that AI filters cannot handle – yet, such as:

A face turning side to side

A hand in front of a face

standing up and walking backward from the camera, showing more body detail

In the circumstances, It should be justifiable to run through some or all of these tests, as well as asking for all filters to be turned off. Stu Sjouwerman, CEO of KnowBe4, shared additional details on what his company learned from hiring a North Korean worker with a false identity:

Do not rely on email references alone. For reference checks, also do phone calls or video calls with references.

Absence of a digital footprint is suspicious. If there’s no trace of a candidate anywhere online, it could be a flag.

Use a phone carrier lookup tool before calling a candidate. Many fake candidates provide a VoIP number. Treat any such number as a red flag and refuse to call candidates on it.

Conduct all interviews using video. Fake candidates can slip through written application processes more easily

Expect identity theft. In the case of KnowBe4, the identity of the person they hired was real and stolen, meaning that a background check came back clean.

Put better internal monitoring in place. Monitor access to systems by onboarded employees – and pay extra attention to new joiners.

When suspicious, ask lowkey “cultural” questions. For example, if a candidate claims to be from Seattle but there is reason to suspect otherwise, ask a question like, “'I see you are from Seattle, what's your favorite place to eat, and what do you usually get?” Faking this knowledge in a convincing way is hard for someone who has never been to Seattle, and doesn’t know the local cuisine.

Different shipping address for laptop. If a candidate asks for an onboarding laptop to be shipped to a different address from where they are supposed to be living or working, it’s a red flag.

Additional tips:

Record video interviews and save them for later. AI-disguised candidates might be hard to spot in a live scenario, but analyzing the video later could be helpful. Note that recording the interview will most likely require disclosing this to a candidate, and them agreeing to this.

Get notarized proof of identity. This advice comes from Google, after they saw hundreds of US companies fall victim to North Korean workers by hiring them for tech roles.

Vidoc also created a guide to detect fake candidates in your hiring pipeline: see this PDF document here.

4. Foreign state interference?

So, who or what might be behind these two incidents at Vidoc; a lone individual or individuals, or something else entirely?

Read more

Today I realized that even though the Web3 program is ramping up as demonstrated by Silicon Valley’s plans to turn Argentina into a “network state,” few understand what is unfolding with cybernetic governance. For that reason, I decided to take time off this spring to work on my garden and simply be with real people in the real world. I made a brief announcement to that effect tonight.

This is a clip I created (6 minutes) summarizing Protocol Lab’s Aleph project with Javier Milei, the guy who gave Musk the chainsaw at the CPAC gathering a few weeks ago. I get that it is difficult to navigate narratives and torrents of information. The current has been against the story I’ve been trying to tell for a decade. It simply didn’t stick, but I did try. I lost my old life in the process, but I gained a sunny yard filled with birdsong and time to read stacks of books on cosmology and consciousness. Until I pop in again, be well everyone.

A work in progress. Yesterday I planted twenty-five asparagus crowns. It will be three years before I can harvest them. It feels kind of good to be on slow vegetable time.

ちょっと旧聞になりますが、去る2月5日に、OECDから面白い文書が出ています。19

Enhancing Access to and Sharing of Data in the Age of Artificial Intelligence
ー Companion Document to the OECD Council Recommendation on Enhancing Access to and
Sharing of Data AI時代におけるデータのアクセスおよび共有の促進
ー OECD理事会勧告「データアクセスと共有の促進」の付属文書

以下はその内容を大まかにまとめたものです。詳しくは原文を御覧ください。

概要 OECDのデータへのアクセスと共有の強化に関する勧告（EASD）は、権利保護を確保しながらデータの利益を最大化するための枠組みを提供します。 経済的、社会的、法的考慮を統合したデータガバナンスへの全政府的アプローチを強調しています。 この勧告は、OECDのメンバーおよびパートナーによる自発的な遵守を促し、責任あるデータ共有の実践を推進します。 データガバナンスの主要概念 データ価値サイクル: データの作成から削除までの全ライフサイクルをカバーし、アルゴリズムや人間のスキルなどの補完的リソースの必要性を強調します。 データオープンネスの連続体: 閉じたデータからオープンなデータまでのアクセスを分類する枠組みであり、リスクと信頼に基づいたカスタマイズされた共有の取り決めを可能にします。 データエコシステム: データホルダー、プロデューサー、中介者などのさまざまな利害関係者が相互作用し、価値を生み出すため、対立する利益のバランスを取るための協力と信頼が必要です。 データアクセスと共有の強化のための原則 信頼の強化: 利害関係者を相談や透明性を通じて関与させ、データガバナンスに対する信頼を築きます。 データへの投資: データ共有を促進するために市場ベースのアプローチや持続可能なビジネスモデルを推進し、イノベーションのための規制サンドボックスを含めます。 データの効果的な利用: データが見つけやすく、アクセス可能で、相互運用可能で、再利用可能（FAIR原則）であることを確保し、国境を越えたデータ共有を促進します。 実践的な応用と影響 政府は、責任あるデータガバナンスを促進するためにEASDの原則に沿った国家データ戦略を採用すべきです。 プライバシーと知的財産権を保護しながらデータ共有を支援するために、一貫した法的枠組みが必要です。 さまざまな国からの成功事例は、データアクセスの強化における公私パートナーシップと利害関係者の関与の重要性を示しています。 データ共有インフラの強化 中央集権的データリポジトリ: 公共機関間での効率的な情報共有を促進するために中央集権的なインフラを確立し、サービス提供を改善し、データ駆動型の公共政策を支援します。 公共の関与: データ駆動型経済の利点とリスクに対する公共の理解を深めるために、責任あるデータ共有の実践を促進します。 利害関係者の意見: イノベーションを妨げることなくAI関連のリスクに対処するために、規制に関する議論に利害関係者を関与させます。 競争当局への影響 市場のダイナミクス: 消費者データと引き換えに無料の製品を提供するマルチサイドビジネスモデルの影響を考慮し、市場力を固定化する可能性があります。 需要側の特性: データが市場のダイナミクスに影響を与え、検索コスト、切り替えコスト、消費者の選択に影響を及ぼし、支配的な市場ポジションを強化する可能性があることを認識します。 研究における責任あるデータ共有 研究データの共有: 機関や研究者間でのデータ共有を促進するために、オーストラリアの責任ある研究行動に関するコードに沿ったガイドラインに従います。 ライセンスの分類: AIや機械学習におけるデータ利用に関連する責任と権利を明確にするためのデータライセンスの分類を開発します。 オープンガバメントデータとAI オープンデータの重要性: オープンガバメントデータは、AIシステムの開発とトレーニングに不可欠であり、信頼できる入力として機能します。 リスク管理: オープンデータは、データの出所やソースの信頼性に関連するリスクを管理し、AIアプリケーションの整合性を高めます。 結論の洞察 OECDのEASDに関する勧告は、データアクセスと共有を強化するための包括的な枠組みを提供し、信頼、投資、効果的なガバナンスを強調しています。 これらの原則を実施することで、公共サービスの改善、データ駆動型経済におけるイノベーション、責任あるAIの開発につながる可能性があります。 利害関係者の継続的な関与と確立されたガイドラインの遵守は、データ共有イニシアチブの完全な潜在能力を実現するために不可欠です。

Policy Brief:

なお、3/8には以下のPolicy Brief が出ています。

Orie Steele and I want to thank Deb Cooley for her Area Director review of the “Fully-Specified Algorithms for JOSE and COSE” specification. Addressing it simplified the exposition, while preserving the essence of what the draft accomplishes.

Specifically, the resulting draft significantly simplified the fully-specified encryption description and removed the appendix on polymorphic ECDH algorithms. We also stated that HSS-LMS is not fully specified, as suggested by John Preuß Mattsson.

The draft has now completed IETF last call, with the two resulting reviews stating that the draft is ready for publication.

The specification is available at:

https://www.ietf.org/archive/id/draft-ietf-jose-fully-specified-algorithms-07.html

This post is a more technical follow-up to the post announcing the passing of SB 260 in Utah, establishing the principles for state-endorsed digital identity (SEDI). While SEDI does not mandate any particular technology, its mandated principles close the door for technologies that do not comply, and open the door for technologies that do. This article explores what SEDI makes technically possible, in compliance with SB 260 principles.

State-endorsed digital identity (SEDI) introduces numerous foundational principles for a government-sponsored digital identity system, several of which were covered in Utah Blazes New Trail for SSI: SEDI:

“Endorse” vs. “issue” and “identity” vs. “credential” Sovereign control Guardianship Privacy & no surveillance State-of-the-art security

This post explores more technical and security principles enabled by SEDI that will be of interest to digital identity experts.

“Autonomic” Identifiers

SEDI doesn’t mandate any specific technology stack, but it does specify capabilities that are simply unachievable for most digital identity technologies. One foundational example is the requirement in lines (73) — (77) of the Utah bill, that individuals must first bring their “identity” to the state, after which it can be endorsed.

What kind of “identity” might that be? That will be specified as the details of Utah’s future identity program are fleshed out. As a Utah citizen and as an outside expert invited to consult on these new identity policies, I will push for stringent minimum standards for the security, utility, privacy, and autonomy of a digital identity to qualify for state endorsement. Needless to say, your social media login — or even your bank login — should not suffice.

One “identity” that meets my preferred standard is an “Autonomic Identifier” or AID, as first described in Dr. Samuel Smith’s seminal paper: Key Event Receipt Infrastructure (KERI).

Some of the capabilities enabled by an AID-powered SEDI ecosystem include:

BYOID (Bring Your Own Identifier) — In the digital realm, for an identity to be self sovereign, it must begin with a unique identifier that is under the sovereign control of an individual. When a state endorses an identity, in practice they are endorsing an identifier that an individual has brought to the state that is provably under the individual’s control.

Persistent Identifiers — When that individual moves their residence to another state (within the U.S., for example), necessitating a new identity endorsement from the new state, the old state can revoke their endorsement of the same identifier that the new state will be endorsing; the identifier is persistent. This is a KERI capability that is very different from most PKI-based systems where the identifier is a public key that is discarded upon rotation or revocation.

Reputation — This persistence creates a continuity for the individual to retain and build their digital reputation independent from any particular government endorsement. In fact, it would be inclusive of every endorsement they have ever had, and all their provable behaviors using those endorsements, no matter how many states or countries have given them past endorsements of identity. And it would be inclusive of non-state endorsements such as educational, work, or anything else, too.

Putting It All Together

A simple depiction of SEDI built on AIDs might look like this:

AIDs enable true self-sovereignty as the foundation, right where it belongs and where it provides the greatest utility for all aspects of digital life. State endorsement is also situated right where it belongs: as only one of many endorsements an individual may receive through their digital lifetime.

State-Issued ‘Jenga’

Conversely, building digital identity on top of a state-issued credential instead of a state-endorsed identifier lasts only until the state revokes that credential. An attached academic credential, for example, is lost when the individual moves from one state to another and obtains a new identity in the new state, and the old state revokes the now-obsolete issuance.

Look at this visually:

This is like a game of Jenga: pull out the state-issued ID — by moving from one state or country to another — and the whole stack falls down. When the state revokes the ID they’ve issued, you also lose everything cryptographically bound to it, and must start over.

Even if the old state kept their credential alive, there is no binding between old and new credentials, no way to build a growing, persistent reputation that connects a person to their cross-boundary behaviors in the digital world the same way a consistent legal name connects their behaviors across boundaries in the physical world.

Human-Friendly?

Autonomic identifiers cannot be human-friendly. Human-friendly namespaces require human management, recurring cost, and have scarcity problems like cybersquatting, or missing the opportunity to register Timothy.com or timothy@gmail; the 1,000th Timothy won’t have much of a selection. AIDs are long random strings of letters and numbers, and are so incredibly random it would take a million computers a million years to guess one, which ensures universal uniqueness and brings a strong measure of security.

AIDs have cryptographic properties, so we can prove control over them in the digital realm and recover control when keys are compromised. And keys *will* get compromised (see “Expected Compromise” below). Without these special mathematical properties, they’d be just letters and numbers anyone can copy and use to impersonate, like a social security number.

Many Identifiers

In real life, we have identifiers that are public — our legal names — that typically remain constant throughout our lives. We use these identifiers to build reputations over time, as patterns of behaviors accumulate that are associated with them. With SEDI, the identifier that is endorsed by the state will work in the same manner: it will be persistent, and as public as one chooses to make it, used and disclosed (always under the hood) whenever one might disclose their legal name.

In digital situations where one prefers to not use their legal name, there’s nothing in SEDI preventing that individual from using a completely different identifier, unrelated to their endorsed one. In fact there is no limit on how many different unique identifiers one may create and use for whatever purpose they choose; and each state can choose whether or not only one at a time can be officially endorsed (Utah allows only one physical identity at a time, and may do the same digitally).

Potential Security Advantages

SEDI’s architecture enables numerous critical security advantages that could be implemented:

Zero Trust — ”Never trust, always verify” is the ideal of zero trust. SEDI can not only embrace zero trust, it can enable true zero trust, which does away with all shared secrets, described briefly below.

No Shared Secrets — SEDI doesn’t employ bearer tokens, symmetric keys, or phishable passwords, unless they are used in conjunction with digital signatures. (Pins and passwords can be helpful when used in addition to digital signatures or physical keys)

Anti-fragile — A breach of one person or system does not facilitate the breach of the entire system or another connected one, as with most current digital identity systems, which are predicated on shared secrets and perimeter security. With SEDI, most breaches can be detectable, preventable, and recoverable.

Expected Compromise — Key compromise should be expected and anticipated — and not surprising and catastrophic — in addition to robust prevention and detection capabilities, bringing powerful means for recovery.

Multi-signature — Keys and credentials can be protected with m-of-n weighted multi-signature capabilities, enabling creative familial, professional, and social protection and recovery schemes.

Mutual authentication — Individuals could authenticate each other, in any setting. For example, a citizen could authenticate a police officer as readily as a police officer can authenticate the citizen. Any person, organization, or thing claiming authority from the state could be instantly verified to have the claimed authority.

Artificially Intelligent Agents (“Agentic AI”)

IBM predicts that 2025 may be “the year of the AI agent”, and the CEO of Microsoft predicts that agentic AI will “replace all software.” Whether that happens or not, AI agents are coming and will soon affect the way we interact with organizations of all kinds. This can take the form of you interacting with some organization’s agent, some organization interacting with your agent, or two AI agents interacting with each other.

SEDI paves the way for verifying that an AI agent has the delegated authority it claims to have, and constraining what an agent is authorized to do. SEDI enables authority to be delegated to people, organizations, and things, and AI agents are things. SEDI is uniquely well suited for controlling, securing, and navigating a world of ubiquitous AI agents.

Many don’t like the idea of having to deal with AI agents in any form, and I can’t blame them. But like the digital transformation itself, that world is coming whether we like it or not, and the more controls and constraints we can have for it, the better.

Comprehensive Digital Identity

In sum, SEDI enables a cohesive, comprehensive digital identity system for a state or for any other ecosystem. It requires no proprietary platforms, products, or providers, though it doesn’t preclude their participation, and requires no shared or trusted intermediaries or blockchains. Through robust guardianship, delegation, and multi-sig, it extends digital capabilities to populations that have been previously excluded: minors (including infants), the elderly, disabled persons, homeless persons, migrants, and any other population. And it does this while maximizing individual autonomy and control over one’s data, with powerful means for consent, and numerous breakthroughs in security.

In short, SEDI heralds a new era of utility, security and autonomy for comprehensive digital identity ecosystems, and I am proud of my home state of Utah for introducing it to the world.

Finally!

On March 4, 2025, the Utah legislature unanimously passed SB 260, a groundbreaking digital identity bill that carries this description:

“This bill enacts provisions related to a state-endorsed digital identity” (SEDI).

Now it’s off to the governor for signature, who has been publicly positive about it. Huge credit to Sen. Cullimore, the bill’s sponsor, and Christopher Bramwell, the state’s Chief Privacy Officer, who helped draft and guide its content. SB 260 will likely soon become the law in Utah, and with it the possibility of an exciting new era for self-sovereign identity (SSI).

Passage of SB 260 is the partial realization of a dream I’ve had for more than a dozen years, since co-founding Evernym with Jason Law in 2012. It is not the complete realization of that dream, but it lays the necessary groundwork and carries with it momentum to go the distance over the next few years.

SB 260 does not call for the creation of a new identity system for the state of Utah — yet — that will be done later, hopefully in next year’s session. This bill establishes the principles Utah politicians stand behind — unanimously — that any future Utah statewide identity system must adhere to. If you are an SSI aficionado, you will recognize and love the principles in this bill.

Is State-Endorsed Identity (SEDI) Still SSI?

For some in the SSI community, the concept of SSI does not involve a state actor — at all.

To these SSI purists, SSI becomes trustworthy through a web of trust that replaces the need for a state actor as a root of trust. While theoretically possible — and still a worthwhile end-goal — this ideal vision of SSI isn’t feasible anytime soon. In the meantime, the credibility of government can help accelerate SSI toward an end-goal of not needing government. How? By making ubiquitous the technologies and ceremonies necessary for a web of trust to emerge.

At the end of the day, if independent control of strong digital identity rests fully with the individual, with no visibility by the state for if, how, or when it is used, then advocates of SSI should be in favor of SSI strengthened by a state endorsement. That is how physical identity works today and how digital identity should work, too. And there’s no reason other, non-government flavors of SSI can’t continue to develop and coexist.

State-Endorsed Digital Identity (SEDI)

The words “state-endorsed” used throughout this bill carry a lot of meaning; most emerging digital identity systems use the word “issue“ instead of “endorse”. This is a distinction with a profound difference.

In real life, does a state issue identity or endorse it? When a baby is born, who gives the baby its identifier, its name? The parents do. The state waits for parents to pick a name, puts the name into a certificate, endorses the certificate, then gives it to the parents. Parents determine identity, the state endorses it.

A couple months ago, a 60 year-old friend of mine decided to change her last name. The decision was hers alone to make, but her new name would not be believable or provable to third parties until she has an endorsement of the new name from the state.

In real life the state issues credentials, not identity. In this bill Utah acknowledges this reality and explicitly states that individuals first assert identity and only then can the state endorse it. That will be the guardrails for any future state-endorsed digital identity system. (See “SEDI Details for Identity Nerds” for more info.)

Through SEDI, SB 260 codifies several foundational SSI principles:

Control

In lines (65) and (120), it is explicit that the individual is in control of their digital identity.

Guardianship & Age Verification

Lines (35–40) and (97–98) lay the groundwork for digital guardianship, bringing broad digital capabilities for the first time to dependent populations such as children (including infants), the elderly, disabled persons, homeless persons, and more

Of particular note in Utah, digital guardianship is how age verification can work for all ages and capabilities, something Utah has been on the forefront of, policy-wise. In the future, guardianship could even work for animals and things, too.

See “Finally, a Path Forward for Age Verification” by the Libertas Institute for more info about how SB 260 could facilitate age verification.

Privacy & No Surveillance

SB 260 is explicit that individuals have the right to not be surveilled. This means that SEDI cannot have any form of “phone home”, which differentiates it from the mobile drivers license (mDL) systems based on ISO 18013, and from the federated identity systems now being implemented in Europe and Australia, which all have inherent phone-home surveillance capabilities.

In lines (78–79), (84–96), (99–102), (116–117), (122–125), and (128–139) the bill details the principles of privacy and autonomy that must guide any future state-endorsed digital identity system in Utah.

State-of-the-Art Security

Line (113) stipulates that Utah SEDI must incorporate “state-of-the-art safeguards” for protecting identity. Current industry standards are not good enough, and in fact are unacceptably fragile in my view. My interpretation of “state-of-the-art” means something akin to KERI (Key Event Receipt Infrastructure), where breaches are preventable, detectable, and recoverable in ways that current identity standards cannot do, and which is quantum-resistant. (See the ‘Details for Nerds’ for more details about security.)

Now the Work Begins

In some ways SB 260 is the end of a long, hard journey for me, and the partial realization of an audacious dream. In reality it’s one bill, in one state, that establishes the principles and guardrails for a future identity system without creating one today.

In any case, it’s the closest thing I’ve seen to true self-sovereign identity being officially and legally embraced at a significant level, an entire state of the United States. The next step is far harder: turning concepts into reality.

Stay tuned.

For more technical information, see SEDI Details for Identity Nerds.

A few years ago I abandoned Twitter in favor of Mastodon. Recent events validate that choice and underscore the strategic importance of a decentralized fediverse that can’t be owned by a single corporate or state actor. But while Mastodon meets my needs, much of the Twitter diaspora has gone to Bluesky. That’s fine for now but might not always be. In an article titled “Science Must Step Away From Nationally Managed Infrastructure,” Dan Goodman writes:

Many scientists put huge efforts into building networks to communicate with colleagues and the general public. But all that work and the value in those networks was lost when many scientists felt compelled to leave following Elon Musk’s takeover of the platform (now X). The process of rebuilding on Bluesky is underway, but it will take years and may never reach the same critical mass. Even if the transition is successful, the same thing may happen to Bluesky in a few years.

How can we prepare for a future migration from Bluesky to Mastodon? Bridgy Fed — a service that enables you to connect together your website, fediverse account and Bluesky account — will help. But Bridgy Fed needs to be easier to use. So I recruited Claude’s new Sonnet 7 model to do that.

Making the Fediverse More Accessible With Claude 3.7 Sonnet

LLM series at The New Stack

The Pulse is a series covering insights, patterns, and trends within Big Tech and startups. Notice an interesting event or trend? Send me a message.

Today, we cover:

Industry pulse. Claude Code quietly becoming the best AI coding agent, Microsoft cutting AI data center investment, EA releases Red Alert code, developers don’t want GPUs, DeepSeek keeps open sourcing innovation, and a US DOGE software engineer forgets to make their GitHub account for work private.

Ads startup demands a lot & offers little → struggles to hire. An adtech startup using AI to generate ads seeks a founding engineer who will devote their heart and soul – and most of their awake time – to the company, and is struggling to hire. They’re now offering a $50K (!!) referral bonus. Feels like a classic case of overlooking that hiring is a two-way street: most standout devs don’t want to work 14 hours a day, 7 days a week.

Cloudflare’s self-critical, transparent postmortem. The leading content delivery network recently had its object storage go down for an hour, and only a day later published a detailed incident review, with the CEO personally taking responsibility for fixing the systemic issues which triggered the downtime. Clouflare is a refreshing exception for promptly sharing incident summaries, and showing it’s serious about continuously improving reliability.

More signs of “cutthroat” Big Tech. Google hints at a 60-hour workweek for workers in its AI group, and Meta fires 20 people for leaking internal information.

1. Industry Pulse Claude Code quietly becoming the best AI coding agent?

Anthropic lowkey launched Claude Code, an agentic coding tool, recently. Feedback I’ve seen from engineers is that the tool is pretty good, with capabilities on par with other AI agents – including Devin (which costs $500/month).

Until now, Anthropic has built the best LLM for coding (Sonnet 3.5 and Sonnet 3.7). In hindsight, it was only a matter of time until they expanded by building an agentic coding tool that’s at least equal to what other startups offer. After all, most startups will build their agents on top of Sonnet 3.5 or 3.7, as well.

A criticism is that Claude Code is expensive, which should not be that surprising because AI agents are much more compute-intensive to run than LLMs. I’ll reserve judgement, Anthropic could turn into a major dev tools company if they keep building AI coding tools like this.

Is Microsoft really cutting AI data center investment?

Read more

Stream the Latest Episode

Listen and watch now on YouTube, Apple and Spotify. See the episode transcript at the top of this page, and a summary at the bottom.

Brought to You By

WorkOS — The modern identity platform for B2B SaaS.

The Software Engineer’s Guidebook: Written by me (Gergely) – now out in audio form as well

Augment Code — AI coding assistant that pro engineering teams love

—

In This Episode

Not many people know that I have a brother: Balint Orosz. Balint is also in tech, but in many ways, is the opposite of me. While I prefer working on backend and business logic, he always thrived in designing and building UIs. While I opted to work at more established companies, he struck out on his own and started his startup, Distinction. And yet, our professional paths have crossed several times: at one point in time I accepted an offer to join Skyscanner as a Principal iOS Engineer – and as part of the negotiation, I added a clause to my contrac that I will not report directly or indirectly to the Head of Mobile: who happened to be my brother, thanks to Skyscanner acquiring his startup the same month that Skyscanner made an offer to hire me.

The Orosz brothers: both of us are working in tech, and have had professional overlaps. Like this podcast episode!

Today, Balint is the founder and CEO of Craft, a beloved text editor known for its user-friendly interface and sleek design – an app that Apple awarded the prestigious Mac App of the Year in 2021.

In our conversation, we explore how Balint approaches building opinionated software with an intense focus on user experience. We discuss the lessons he learned from his time building Distinction and working at Skyscanner that have shaped his approach to Craft and its development.

In this episode, we discuss:

Balint’s first startup, Distinction, and his time working for Skyscanner after they acquired it

A case for a balanced engineering culture with both backend and frontend priorities

Why Balint doesn’t use iOS Auto Layout

The impact of Craft being personal software on front-end and back-end development

The balance between customization and engineering fear in frontend work

The resurgence of local-first software and its role in modern computing

The value of building a physical prototype

How Balint uses GenAI to assist with complicated coding projects

And much more!

Takeaways

Design-focused engineers find it harder to fit in. Engineers who focus on backend and distributed systems can usually verbalize their impact clearer, and could see faster career growth – including getting into leadership positions. This creates a reinforcing cycle: as most engineering executives have a backend engineering background: they will recognize and reward backend contributions more.

Balint didn’t like how he didn’t “fit in” as an engineer focused on design and UI: but he just kept building things he believed in – and years later, building up that “design muscle” helps him build products that backend-focused engineers might struggle in putting together.

Amazing companies cannot have a single engineering culture. Balint observed how every standout company that has a single engineering culture inherently biases towards either a backend-heavy engineering culture (e.g. Google, Amazon) or a UI-heavy one (Apple). For a company to be truly standout – more so than these Big Tech giants – it needs to have several engineering cultures: prioritizing both backend and UI excellence. This is what Craft is aiming to do internally.

To build something better than most other products: you might need to take a different approach. Craft does not use Apple’s UI components: they don’t use SwiftUI or Autolayout – something that 95% or more of iOS apps all take advantage of.

Craft, instead, built their own components from scratch, and came up with their own layout and animations system. This is a lot more work at first: but it’s how Craft can do smooth animations that most apps are unable to do so – and it’s a reason engineers at Apple have asked the team “how are you able to do such a smooth animation on the navigation bar?” (The native iOS navigation bar cannot be animated like Craft does it) Craft can do all this thanks to building and maintaining their own components. It’s simpler to do than most would assume: in the episode, we look at actual code.

A shared codebase is an underrated advantage for speed, consistency and efficiency. Craft has a total of 4 engineers building the respective apps for:

iOS

iPad

MacOS

VisionOS

They can do this, because it’s a single codebase! A single codebase also means that all features built on one platform immediately work on all others.

The Pragmatic Engineer deepdives relevant for this episode

The AI hackathon at Craft Docs

Engineering career paths at Big Tech and scaleups

Thriving as a Founding Engineer: lessons from the trenches

The past and future of modern backend practices

Timestamps

(00:00) Intro

(02:13) What it’s like being a UX-focused founder

(09:00) Why it was hard to gain recognition at Skyscanner

(13:12) Takeaways from Skyscanner that Balint brought to Craft

(16:50) How frameworks work and why they aren’t always a good fit

(20:35) An explanation of iOS Auto Layout and its pros and cons

(23:13) Why Balint doesn’t use Auto Layout

(24:23) Why Craft has one code base

(27:46) Craft’s unique toolbar features and a behind the scenes peek at the code

(33:15) Why frontend engineers have fear around customization

(37:11) How Craft’s design system differs from most companies

(42:33) Behaviors and elements Craft uses rather than having a system for everything

(44:12) The back and frontend architecture in building personal software

(48:11) Shifting beliefs in personal computing

(50:15) The challenges faced with operating system updates

(50:48) The resurgence of local-first software

(52:31) The value of opinionated software for consumers

(55:30) Why Craft’s focus is on the user’s emotional experience

(56:50) The size of Craft’s engineering department and platform teams

(59:20) Why Craft moves faster with smaller teams

(1:01:26) Balint’s advice for frontend engineers looking to demonstrate value

(1:04:35) Balint’s breakthroughs using GenAI

(1:07:50) Why Balint still writes code

(1:09:44) Rapid fire round

A summary of the conversation Design-first engineering: less recognition?

Balint has been writing code since he was 12. He’s always had a pull towards interactivity, animation, and found himself in the intersection of code and design.

Backend engineering is much easier to quantify than UI/UX work. Balint always felt that backend engineering was easier to quantify in impact – like saving millions in infrastructure costs or scaling to billions of users. How do you quantify that the UI is smooth and delightful?

Balint always felt that "hardcore engineers" did not consider him one of them – as his focus was on UX and UI – like making interactions faster, and animations smoother, and not on distributed systems or scalable algorithms. But designers and product managers also didn’t look at him as equals.

There’s a pull for engineers to move towards the backend and distributed systems. Exactly because these areas are seen as having hard problems to solve and are often more measurable in terms of impact.

Engineering culture at Craft

Craft started off as a text editor for mobile use. When starting to code the product, Balint didn't believe that established engineering patterns, code coverage, or even design components would be the right approach.

Principles in building craft:

Data principles: efficiency and zero data loss are both non-negotiatble. There’s not much to innovate in this area – it’s very well understood! Just refine the existing state of the art.

Fluency. Craft is designed to be used for hours each day, so it needs to feel snappy and fluent.

Having just one engineering culture is not enough. To create an amazing product, you cannot have either frontend or backend engineering principles dominate:

When the dominant engineering culture is UI-first: Apple is a good example. Apple builds delightful user experiences. However, their backend systems and web products are lacking.

When the dominant engineering culture is backend-first: Amazon and Google are good examples. Both focus on system design and backend engineering principles – in return their UIs don't feel as comfortable.

Craft uses the same codebase across 4 platforms: iOS, iPad, MacOS (desktop), and Vision Pro. 99% of the code is common, with some additional native bindings for each platform. Why the same codebase? Balint wanted the desktop app to always do the same thing as the mobile app, and a shared code base was the best way to do it.

Team structure

Craft's product engineering team is around 20 people. This includes product engineering, design, and QA.

They are split into a platform team, which means they have a web team, a native app team, and a backend team. Each of those teams has three to four people.

Balint observed that teams with more than 5 people start to have communications issues

Architecture choices

Prioritize control over abstractions and trends. Using core language patterns and framework elements gives you more control over what you want to do. High-level abstractions and frameworks require more time figuring out the bugs and what those frameworks allow or disallow.

Everything is a canvas. Craft treats everything as a canvas that they draw on. This allows building toolbars that look exactly like a Mac toolbar on the Mac and an iOS toolbar on the iPhone. However, Craft now has more control over these components than if they used the native ones.

Avoiding AutoLayout and SwiftUI. Every time Craft hires a new iOS engineer, the new joiner inevitably asks them why they don't use new technologies like AutoLayout or SwiftUI. Balint shows them one of their transitions and asks them to implement it with the same performance in AutoLayout – and if they can succeed, they’ll move over. So far, no one has managed: but it’s a good exercise to help understand that decisions are not arbitrary, but are following practical purposes.

AutoLayout promises that you no longer need to think in rectangles; you can just say it should be at the top and it can automatically grow. However, when you keep adding more things, the complexity increases and when you want to do something more sophisticated, it can become very performance-intensive.

With AutoLayout, you are trading off easier definition and work as a developer for more complexity on the device and less control when you want to do more advanced animations.

Principles for personal software

“How it makes me feel” matters more. When it comes to personal software, consumers choose a lot more on which one they feel resonates with them. Thus the best personal software is a lot more opinionated than B2B software is

Craft is aiming to build what Visual Studio Code is for engineers – Craft aiming to do the same for knowledge workers. Visual Studio Code feels lightweight and is a fresh breath of air because it's responsive, fast, and does everything you need.

A different take on design systems.

Most companies create an “atomic design system:” starting with base colours as atoms, then buttons that combine colors and shapes as components, and then building up from here.

Craft, instead, has systems for animation. They have an animation engine and library that synchronizes everything across everyone and they enforce usage of that.

A local-first approach

New technologies emerge on the server side. When you're dealing with personal software, the amount of content you're dealing with can fit on the user's device. A lot of the compute can be done locally.

Craft’s architecture is preparing for local-first approaches. They architected components so they can replace them with local or remote components anytime they decide it’s now possible to do so.

An example is search: instead of having a big elastic search cluster, they are looking at having 2 million search indexes on a disk in the cloud. Every time somebody does a search, they can either download that search index locally and use it there to execute the search, or a Lambda or serverless function can just read the search index.

The industry keeps swinging between cloud and local compute.

There is a new wave of personal computing powered by processors getting faster. The M4 Pro is a faster processor than anything you can buy on an AWS cloud.

Eventually, people will get tired of their personal computers holding them back and they’ll appreciate server-side components working faster. After a while, they will then start to get annoyed about how much the server-side project costs… and the pendulum keeps swinging

Local-first is experiencing a comeback because people are starting to travel again. It gets inconvenient when you need something badly, with poor connection – and you cannot access it.

Where to find Balint Orosz:

• X: https://x.com/balintorosz

• LinkedIn: https://www.linkedin.com/in/balintorosz/

Mentions during the episode:

• Craft: https://www.craft.do/

• Skyscanner: https://www.skyscanner.com/

• Flash: https://en.wikipedia.org/wiki/Adobe_Flash

• Shader: https://en.wikipedia.org/wiki/Shader

• Understanding Auto Layout: https://developer.apple.com/library/archive/documentation/UserExperience/Conceptual/AutolayoutPG/index.html

• Mac Catalyst: https://developer.apple.com/mac-catalyst/

• Apple M1: https://en.wikipedia.org/wiki/Apple_M1

• Elasticsearch: https://en.wikipedia.org/wiki/Elasticsearch

• The Cloud Is a Prison. Can the Local-First Software Movement Set Us Free?: https://www.wired.com/story/the-cloud-is-a-prison-can-the-local-first-software-movement-set-us-free/

• Local-first software: https://news.ycombinator.com/item?id=31594613

• Visual Studio Code: https://code.visualstudio.com/

• ChatGPT 01 model: https://openai.com/o1/

• PencilKit: https://developer.apple.com/documentation/pencilkit

• Swift: https://www.swift.org/

• Objective-C: https://en.wikipedia.org/wiki/Objective-C#

• Tailwind CSS: https://tailwindcss.com/

• The Hard Thing About Hard Things: Building a Business When There Are No Easy Answers: https://www.amazon.com/Hard-Thing-About-Things-Building/dp/0062273205

—

Production and marketing by Pen Name. For inquiries about sponsoring the podcast, email podcast@pragmaticengineer.com.

I’m happy to announce that some PostgreSQL colleagues and have once again organized the Extension Ecosystem Summit at PGConf.dev in Montréal on May 13. Floor Drees, Yurii Rashkovskii, Keith Fiske will be on hand to kick off this unconference session:

Participants will collaborate to learn about and explore the ongoing work on PostgreSQL development and distribution, examine challenges, identify questions, propose solutions, and agree on directions for execution.

Going to PGConf.dev? Select it as an “Additional Option” when you register, or update your registration if you’ve already registered. Hope to see you there!

Extension Ecosystem Mini-Summit 2.0

We are also once again hosting a series of virtual gatherings in the lead-up to the Summit, the Postgres Extension Ecosystem Mini-Summit.

Join us for an hour or so every other Wednesday starting March 12 to hear contributors to a variety of community and commercial extension initiatives outline the problems they want to solve, their attempts to so, challenges discovered along the way, and dreams for an ideal extension ecosystem in the future. Tentative speaker lineup (will post updates as the schedule fills in):

March 12: David Wheeler, PGXN: “State of the Extension Ecosystem” March 24: Peter Eisentraut, Core Team: “Implementing an Extension Search Path” April 9: Christoph Berg, Debian: “Apt Extension Packaging” April 23: May 7: Gabriele Bartolini, CNPG “Extension Management in CloudNativePG”

Join the meetup for details. These sessions will be recorded and Posted to the PGConf.dev YouTube and we’ll have again detailed transcripts. Many thanks to my co-organizers Floor Drees and Yurii Rashkovskii, as well as the PGConf.dev organizers for making this all happen!

More about… Postgres PGXN Extensions PGConf Montréal Summit

Hi – this is Gergely with the monthly, free issue of the Pragmatic Engineer Newsletter. In every issue, I cover challenges at Big Tech and startups through the lens of engineering managers and senior engineers. If you’ve been forwarded this email, you can subscribe here.

The most-frequently cited article I’ve ever written is still The trimodal nature of software engineering salaries in the Netherlands and Europe – which is a deepdive published back in 2021. At the time, I had recently left my engineering manager job at Uber, and was trying to understand how tech compensation really works. At Uber in Amsterdam, I saw that comp for senior software engineering positions was 2-5x higher than for most “local” senior roles.

To find out why this gap exists, I gathered hundreds of data points and talked with dozens of recruiters and managers; was this comp gap a symptom of “bimodal” distribution, as Daan Luu posted in 2016? After researching, I felt confident enough to say that tech compensation in the Netherlands and Europe was trimodal:

High-level overview. “$X” varies per country and region, but trends are international

Last year, we returned to this topic in The Trimodal nature of tech compensation revisited, which validated the distribution by using more than 1,000 data points from the Dutch market, and quotes from engineering leaders and software engineers in the US, UK, Canada, and Japan, which suggested that a “trimodal” model is real. Even so, this was partly anecdotal and in need of hard data to confirm it.

Today, we finally have the numbers, thanks to the Levels.fyi team,, and this article dives into the findings by region, covering:

US: entry-level software engineers. Categorizing the three tiers of compensation bands, based on 5,601 data points.

US: senior software engineers. More Tier 2 and 3 offers are at least twice as big as the Tier 1 median, based on 9,509 data points.

Best-paying US companies by tier. Hedge funds, top scaleups, and Big Tech lead the pack – with companies like Roku, MongoDB and Dropbox also ranking high.

UK numbers, tiers, and best-paying companies. Hedge funds and finance companies lead, with Stripe, Confluent, and BP also making it to the top 10, based on 6,219 data points.

India numbers and tiers. Rippling, Stripe, Uber, Coinbase, Atlassian, Broadcom, and LinkedIn, offer ₹1 crore-or-above packages for senior engineers ($118K and up), based on 2,078 data points

Note on equity. Startups and scaleups may offer more in total compensation than publicly traded companies, but equity in not-yet-public companies comes with risk.

The data on the US, UK, and India reveals a complex regional picture united by a trimodal model. See below:

Three “tiers” in the US data. See “US: senior software engineers” section

If you want to look even deeper into this topic, check out other detailed articles on tech compensation:

The trimodal nature of software engineering salaries in the Netherlands and Europe (Part 1, 2021)

The trimodal nature of tech compensation revisited (Part 2, 2024)

Compensation at publicly traded tech companies (2023)

Senior-and-above compensation in tech (2024)

The trimodal nature of software internship salaries (research by Levels.fyi, 2024)

Thank you to Hakeem Shibly, Zuhayeer Musa, Zaheer Mohiuddin and the Levels.fyi team for sharing massive amounts of data and insights for this deepdive.

Data source

This article uses more than 10,000 compensation data points sourced from Levels.fyi, a compensation comparison site which is great for:

Comparing engineering career levels, such as how E5 at Meta maps to SDE3 at Amazon

Browsing tech comp at large tech companies and scaleups, powered by user submissions

Levels.fyi is powered by a combination of self-reported and verified data points. They offer resume review and salary negotiation services with Big Tech recruiters. A neat feedback loop of this is that information from their salary negotiation services helps the team ensure that submitted data really is valid. Levels.fyi also has a job board and provides compensation benchmarking services for businesses.

Some stats about Levels.fyi:

2.5 million visitors per month

20,000 new data points added per month, on average

$100M: total value of increased comp offers achieved with its compensation negotiation service – not including raises negotiated by candidates using the site ad-hoc for comparison.

A team of 13 people is behind the site and app, and I recently grabbed coffee with several of them in San Francisco.

This article covers “total compensation” (TC), not only salary. TC includes all forms of annual comp:

Base salary

Cash bonus, or the cash bonus target of companies that share this information

Value of equity vested

For example, a $300K TC package might look like this:

$175K base salary

Plus $25K target bonus

Plus $100K in equity vesting over the first year (typically $400K vested over 4 years, with $100K vesting each year)

Equity is a tricky topic but also a key one, more on which is in the “Note on equity” section near the foot of this article.

1. US: entry-level software engineers

Let’s start with entry-level software engineer offers submitted to Levels.fyi in the past year for the world’s biggest tech market. Here’s how these data points were filtered:

“Software Engineer” submissions

Offer date between 1 Jan 2024 and 1 Jan 2025

Years of experience: between 0 and 2

Region: US

Years at company: 0. This means we look at new offers, not existing employees’ comp changes at their current workplaces

We have 5,601 entry-level US data points after applying the filters. A visualization:

5,601 entry-level software engineering offers in the US, visualized

At first sight, this distribution fits what looks more like a bimodal distribution, with a first peak at $100-110k, and a second peak at $170-180k:

Comp distribution has twin “local maximum” peaks

However, after sifting through the data and manually tagging companies by common traits, we find the data organizes itself differently. The chart below takes the same data and organizes it based on the assumption behind the original trimodal distribution: that companies compensate based on the market they compete in. We use three categories of business type:

Big Tech. Apple, Amazon, Microsoft, Google, Meta, and Netflix, alongside publicly traded, large tech companies competing with them, like Uber, Snap, and Pinterest

Top scaleups and hedge funds. Late-stage startups backed by well-known VCs that have not gone public, or IPOed only very recently. Plus hedge funds, and quantitative trading firms (quant)

Everyone else. startups, privately owned companies, and “traditional” ones

Here’s how the same data looks when broken down into these categories:

Segmenting the data by three tiers

Now, let’s check each segment.

Tier 1: Everyone else

This category consists mainly of companies competing for local talent, which often are not driven by their engineering talent or digital products.

Distribution of the “everyone else” category: 3,262 data points (58% of total). Median: $105K, 75th percentile: $130K

This resembles the lowest tier (“Tier 1”) of the trimodal model. It’s interesting how drastically the distribution changes by removing Big Tech, top scaleup, and hedge fund data points!

Tier 2: Big Tech Big Tech. 1,914 data points (34% of all data). Median: $180K, 75th percentile: $203K

The largest publicly traded tech companies compensate engineers with increasingly large equity packages up the career ladder. I split this category for two reasons:

Commonly referenced. The terms “FAANG” and “Big Tech” are frequently their own category, and benchmarks for other companies’ offers.

Industry-leading comp. It’s hard to find bigger liquid compensation offers than this group offers. Later, we see it’s possible to win higher illiquid comp offers at not-yet-public scaleups, and some hedge funds can top what Big Tech offers.

Tier 3: Top scaleups and hedge funds Top scaleups and hedge funds, based on 493 data points (8% of all data). Median: $208K, 75th percentile: $247K

This is a very interesting category with a lack of high-quality data, outside of Levels.fyi, for two reasons:

Private scaleups issue not-yet-liquid equity. Many assume Big Tech companies offer the most, compensation-wise. And this is true in terms of liquid equity; shares that can be sold for cash is an area where Big Tech is near-impossible for scaleups to match.

However, top-tier startups and scaleups can – and do – offer more total compensation than Big Tech. But there’s a catch: their equity cannot be sold until a liquidity event happens like an IPO, or stock buyback. Scaleups which out-offer Big Tech in total comp include Databricks and OpenAI.

It makes sense for late-stage scaleups to make bigger total comp offers than Big Tech because they target the same talent and must compensate for the fact their equity grants are illiquid, and may not become convertible to cash any time soon. Plus, there’s also the risk of a grant losing value like at Foursquare, where employees’ stock options ended up as worthless.

Hedge funds are notoriously secretive. Places like Two Sigma want to keep their compensation details as confidential as possible. Levels.fyi has accurate data points, but not enough to make this group into its own category. So, they’re included with scaleups here.

The compensation philosophy of hedge funds is very different than at Big Tech and scaleups:

No equity: hedge funds do not issue equity, even though this element is almost always what makes Big Tech and scaleup compensation packages standout

Large cash bonuses: instead, hedge funds award hefty end-of-year bonuses, of up to 100% of annual salary. Hedge funds are the only category of company offering more liquid compensation than Big Tech and some scaleups. At the same time, hedge funds hire a lot less frequently

Overlapping categories

Let’s look at all three categories of employer in a single graph:

All entry-level compensation data points, split into categories

And also let’s check out the median and 75th percentile total compensation packages of each category:

Data points for each category

There are a few interesting points about this distribution:

~2x difference between Tier 1 and 3. Tier 2 and 3 companies pay around 2x as much at the 50th and 75th percentiles than Tier 1 does. Tier 2 (Big Tech) is a lot closer to Tier 3 in TC. The 75th percentile of Tier 1 still doesn’t come close to the 50th percentile of Tier 2

Overlap from Tier 1 into Tier 2. The highest Tier 1 offers – at around the 90th-95th percentiles – do reach up to $200K, which is near the median range of tiers 2 and 3

Tiers 2 and 3 overlap a lot. The only real difference between Tier 2 and Tier 3 is that there are much more Big Tech data points in the sample. This could indicate a bias in user submissions towards Big Tech, or that scaleups and hedge funds hire far fewer entry-level engineers in the US

2. US: senior software engineers

Compensation differences tend to grow with seniority, so let’s look at senior-and-above software engineers by adjusting the filters:

“Software Engineer” submissions

Offer date in the last 3 years (between 1 Jan 2022 and 1 Jan 2025)

Experience: 5 years minimum

Region: US

Years at company: 0. Similar to before, we only look at new offers

Let’s see how the same three categories map when looking at senior software engineers, split across the same tiers:

All senior-and-above engineer compensation data points by category. Total number of data points: 9,509

Here’s what these tiers look like in the median and 75th percentile compensation bands:

Data points for each category

The tiers are more distinct for seniors than for entry-level engineers. Some observations:

Tier 1 (everyone else): this category has the most data points (53% of the total). Also, while the median of this tier is significantly below the other two categories, there is overlap at the 75th and 90th percentiles. Simply put, it’s possible to earn as much in this group as in Big Tech, but it’s harder to do.

Tier 2 (Big Tech): compensation spread is very wide, starting from around $100K, all the way to $600K and above. This indicates the median value is less important, and things like career level, the specific company, and negotiation, could all matter more.

Tier 3 (top scaleups and hedge funds): this overlaps with Big Tech, and its top end goes even further. Also, there is far less data for this category, suggesting that these companies hire significantly fewer staff than Big Tech does, as mentioned above.

Outliers: these are much more distinct at the senior level, compared to the entry-level tier distribution

Big Tech and scaleups+hedge funds have many outliers. The median entry-level Tier 1 compensation package was $105K – and there were few offers in tiers 2 and 3 that offered twice as much ($200K and above.) The median senior Tier 1 compensation was $180K, but now there are far more offers in tiers 1 and 2 worth double that ($350K and above). Visualized:

A higher percentage of people reported making at least double in Tier 1 median compensation at senior level, than at entry-level. Being in the “long tail” of offers could be more achievable at senior level than at entry-level 3. US companies paying highest by tier

With the data below, we look at the number of senior software engineer submissions and filter them for offers submitted by engineers with 5-10 years of experience.

Tier 1: “everyone else”

The highest median total compensation offers in this category were made by the companies listed below:

Companies offering the highest 50th percentile compensation in the “everyone else” category

Leading employers in this category are Indeed, HubSpot, Workday, Disney, eBay, Capital One, Goldman Sachs, JPMorgan Chase and Visa.

Which businesses in each tier are most worth targeting for jobseekers? One indicator is the number of submissions, which identifies companies that are hiring more than others. Of course, this also biases to companies which users have shared data about. In this category, it’s Capital One, Walmart, JPMorgan Chase, Disney, HubSpot, Indeed, Workday, Palo Alto Networks, Fidelity, and Visa.

Tier 2: Big Tech

Highest-paying companies by median total compensation:

Companies offering the highest 50th percentile comp in the Big Tech category

This is quite a jump: the median is nearly twice as high as in the “everyone else” category! You can find more details about individual packages and get a sense of the spread of offers per company, at the pages for Netflix, Airbnb, Pinterest, Robinhood, Snap, Roku, Meta, MongoDB, Uber and Dropbox.

Most data points submitted: for this category, most data points submitted were for Amazon, Meta, Google, Microsoft, Apple, Oracle, Uber, Block, NVIDIA, and Salesforce.

Tier 3: Top scaleups and hedge funds

The cream of the comp crop; with hedge funds, quant firms, and top-tier VC-backed startups doing well. Companies with the highest median offers are:

Companies offering the highest 50th percentile compensation in the top scaleups and hedge funds category

If you thought Big Tech has high median offers, these numbers are between 20 and 40% bigger. Hudson River Trading, Citadel, and Jane Street, are all hedge funds or quant firms, meaning their stated compensation is all cash. Others on the list are companies which IPOed in recent years like Coupang, or late-stage scaleups.

Find more details about individual packages and get a sense of the spread of offers within a company by checking the respective pages on Levels.fyi of Hudson River Trading, Databricks, Coupang, Citadel, OpenAI, OpenSea, Earnin, Nuro and Figma.

Most data points submitted: in this category, it’s Bytedance, Stripe, Cruise, Snowflake, Roblox, Coinbase, Databricks, Reddit, Rivian, and Anduril.

The importance of location

Location also impacts compensation packages. In the Bay Area, Seattle and New York, compensation packages are higher than in other areas in the US. We can get a better sense of how location plays a part using a heatmap – also built by Levels.fyi:

Location and median pay have some connection. Source: Levels.fyi heatmap 4. UK numbers and tiers

Here’s the distribution of compensation data points for senior software engineers in the UK, mapped in British pound sterling (GBP):

Distribution for senior software engineer packages in the UK

For this country, we attempt to segment data points by company-headquarter location:

Headquartered outside the UK and US (mostly in European countries)

Headquarters in the UK

Companies with headquarters in the US

Here’s data for these categories:

Data points for each category

Visualizing the distribution:

Distribution for the three categories of senior software engineer packages in the UK

A few interesting things stand out:

The “US-HQ tier” is odd. It feels like two tiers might be coupled here, based on the midpoint of the graph having several local maximums: £100K ($126K), around £125K ($158K), and circa £150K ($190K). I’d assume that if we split up Big Tech, the top scaleups+hedge funds, and everyone else, we’d see two distinct tiers. Still, it is US-HQ’d companies that offer the most comp to senior software engineers.

Like the US, the UK has hedge funds that offer as much – or more – than Big Tech. These firms are all based in London, often with offices in New York. This is the first time I’ve seen it broken down like this.

UK, but not US or UK-headquartered, are similar. The medians and 75th percentiles of these categories are almost the same. Non-US, non-UK HQed companies simply have fewer data points.

The data is likely to overrepresent US-HQ’d companies. It would seem there are more US-based companies employing software engineers in the UK, than UK-headquartered ones do. However, based on my understanding of the market this is incorrect: in reality, a lot more UK-based companies employ software engineers than US-based ones, but the data currently biases towards US-HQ’d companies.

My sense is that for the UK, the real three tiers are those previously discussed:

Tier 1: local market, meaning most UK HQ’d companies

Tier 2: startups and scaleups targeting the top of the local market. Many are likely US-based, but not competing with Big Tech

Tier 3: Big Tech and top scaleups competing with them

Here are companies offering the highest median total compensation in the UK:

Companies offering the highest 50th percentile compensation in the UK. Most roles are London-based

Businesses paying the most, based on median data: Citadel, The Trade Desk, Balyasny Asset Management, Stripe, Confluent, Qube Research, Maven Securities, Millennium, G-Research, and BP – an oil and gas company, which is the only non-tech, non-finance company in the top 10.

And here are companies outside the top 10, but in in the top 30:

Coremont (£180K), Meta (£178K), Apple (£178K), Snap (£177K), Reddit (£176K), ByteDance (£175K), Palantir (£173K), Google (£173K), HashiCorp (£169K)

Standard Chartered (£166K), WordQuant (£166K), BBC (£165K), DuckDuckGo (£161K), Monday.com (£159K), Squarepoint Capital (£157K), Samsara (£156K), Onfido (£156K), X – formerly Twitter (£156K), Discovery (£154K)

In this list, a surprise for me was to see the BBC being so competitive in compensation, compared to Big Tech companies. In all fairness, I’ve heard good things about both the engineering talent and the engineering culture at the British broadcaster.

And these are the companies in the UK for which the most data points were submitted by users:

Companies with most data points 5. India numbers and tiers

For India, compensation packages are represented in USD ($100K in USD equals to about ₹87 lakhs). The split:

Distribution of the three categories of senior software engineer packages in India, based on 2,050 data points

This split mirrors a single distribution, but let’s break it up by the location of headquarters:

In India

Not in India or the US

US

The distribution:

Distribution for the three categories for senior software engineer packages in India

It feels like US HQ’d companies might be two tiers combined, so let’s split this out by separating US-based Big Tech, and US-based “everyone else”:

Distribution when breaking US-HQ’d companies into Big Tech and non-Big Tech categories

There are many interesting details in this graph:

The data is US-dominated. As in the UK, India has more local companies based in the country and employing software engineers. For example, Tata Consultancy services alone employs more than 600,000 IT professionals – many of them software engineers. And we’ve not covered large employers like Infosys, or Wipro. At the same time, data from Levels.fyi is biased towards US-HQ’d companies: 77% of all data points. You can help the data become more accurate by submitting your compensation.

India-HQ’d companies rarely offer standout compensation. As per this data, it is mostly companies headquartered abroad that offer above $100K in total compensation (₹87 lakhs or more).

US-HQ’d companies pay very well. While the median compensation for these companies is $63K (₹52 lakh,) the 90th percentile is at $133K (₹1.1 crore!) This is surprisingly high!

Big Tech offers are very strong. A select few senior+ engineers can make in the range of $200K per year (₹1.7 crore) at the very top of the range, which is close to 4x the median compensation of the India-HQ’d tier.

Data points for each category

Companies that pay the most in India, as per the data:

Companies offering top 50th percentile comp in India

One surprise is Google not making the top 10; it comes in at number 14 overall, with median senior engineer compensation at $98K (₹85 lakh) – not far away from the top 10. One thing to keep in mind is that Google has the most data points of companies, meaning it might be more accurate.

Businesses with the most data points submitted in India:

Companies with the most compensation packages submitted to Levels.fyi 6. Note on equity

The bigger a Big Tech or VC-funded startup comp package is, the larger the share of equity. For example, here’s a visualization of the comp split for senior software engineers in the US, over the past year:

Visualizing the TC split of engineers. The higher the TC, the bigger the equity and cash bonus, compared to base salary. Based on 796 data points

You might wonder how the largest packages have no equity component and a very large cash bonus. It’s because these are hedge funds which don’t issue equity and award cash bonuses of nearly 2x the annual salary.

Non-liquid equity comes with additional risk. Let’s take two compensation packages: one with a TC of $475K, and one with $423K. Which is better? It seems like $475K is clearly bigger, but some extra context is needed. The $240K package is from Rippling, a scaleup that is not yet publicly traded:

A $475K package at Rippling. See the original submission

The $423K package is from Uber:

A $423K package at Uber. See the original submission

The big difference between them is that at Uber, an engineer can sell their stock as soon as it vests, (usually, quarterly) if they wish. By doing so, they earn $423K worth of liquid compensation in the first year. At Rippling, the liquid compensation will be $335K ($240K in base salary and $85K bonus target), and equity can only be sold if and when a liquidity event occurs, such as:

Rippling organizes a secondary offer and allows employees to sell some stock. This usually happens with oversubscribed fundraising events

Rippling goes public. In this case, employees can sell their vested stock after a lockup period of around 6 months

Rippling is acquired and the buyer pays for employees’ stocks

For private companies, the big questions are when will equity become liquid, and how much will it be worth.

It’s worth remembering stock prices change, which makes equity compensation volatile. In 2022, we covered how this can lead to compensation woes. At the time, tech stocks like Stitch Fix, Redfin, and Zoom, dropped 50-80% in 12 months. At the same time, companies like NVIDIA, Tesla, and Apple, saw 50-100% stock price growth. Compensation packages for people working in the former group dropped significantly, while the latter group saw major increases. A $350K TC package became worth $510K at NVIDIA, and $231K at Stitch Fix:

Equity can be volatile, and equity-heavy packages similarly so. Source: Stock price woes and attrition Takeaways

Thanks very much to the Levels.fyi team for sharing so much data for the US, UK, and India. It’s enabled us to visualize compensation distribution and tiers across international labor markets.

Levels.fyi gives free access to its compensation data points, and relies on users sharing their comp numbers. If you’ve gotten value out of this article, please consider submitting the comp details of your current position; this lets the site provide fresh, data-backed insights, like in this deepdive.

My biggest takeaways:

The US is unique in having a tier above Big Tech. Across the UK, India, and in the Netherlands, the highest compensation tier (Tier 3) is Big Tech. So it’s a surprise to learn there’s an even higher tier that exists only in the US. Hedge funds, quant firms, and top scaleups can pay more than the likes of Google, Meta, Amazon, Netflix, Apple, etc, with two important caveats:

At top scaleups, bigger-than-Big-Tech comp packages are usually reserved for workers already in Big Tech, who would not switch jobs for an illiquid TC package unless it exceeds their current, liquid one.

Hedge funds and quant firms seem like a different world. Switching between Big Tech and these companies looks harder to do, and happens less often than moving between Big Tech and scaleups.

US-HQ’d companies tend to pull up markets outside the US. Among tier 3 companies in the Netherlands, 95% of those which I know are US-headquartered, and the data shows this is also the case in the UK and India.

Higher-tier companies come with downsides. These workplaces pay more, but usually operate with a “Silicon Valley-style” engineering culture: they expect a lot more in terms of autonomy, taking on product, DevOps, and testing work, and working longer hours – especially when collaborating with overseas teams. Adjusting to this can be tough if you’ve never worked in such an environment. Vice-versa, it can also be challenging to swap this pattern for a more traditional, hierarchical workplace.

Meanwhile, work stress can actually be higher at Tier 2 and Tier 3 companies, as covered in part 2 of the Trimodal series.

Today’s job market is a lot tougher than previously, and it’s harder to get into top-tier companies. In this deepdive, we’ve looked closely at the top tiers across several markets. With the job market tightening up, there are more qualified candidates than ever, and this can affect the size of offers. So, if you’re in the job market, I would not treat the numbers in this article as concrete and immovable. Instead, just focus on performing well in interviews, and getting an offer. Then these figures might be helpful in the negotiation phase.

Compensation is not everything; it’s just what is easiest to measure and make comparisons with, but there’s more to a job than the pay alone. Here’s what I used to think matters:

A way to think about a career

But is this all there is? I no longer think so. Other factors also make a job into a dream or a daily struggle:

Less quantifiable, but still very important, aspects of a career

If you’ve gotten value out of this data-based article, why not contribute to the wealth of knowledge about global tech compensation by anonymously sharing your current compensation with Levels.fyi. I hope you’ve found the additional data points useful, especially the expanded details on the US, UK, and India.

For more on tech comp, check out other deepdives:

Senior-and-above compensation in tech (2024)

Compensation at publicly traded tech companies (2023)

The trimodal nature of software internship salaries (research by Levels.fyi, 2024)

The trimodal nature of software engineering salaries in the Netherlands and Europe (Part 1, 2021)

The trimodal nature of tech compensation revisited (Part 2, 2024)

Emil Lundberg and I have published the COSE Algorithms for Two-Party Signing specification. Its abstract is:

This specification defines COSE algorithm identifiers used when the signing operation is performed cooperatively between two parties. When performing two-party signing, the first party typically hashes the data to be signed and the second party signs the hashed data computed by the first party. This can be useful when communication with the party holding the signing private key occurs over a limited-bandwidth channel, such as NFC or Bluetooth Low Energy (BLE), in which it is infeasible to send the complete set of data to be signed. The resulting signatures are identical in structure to those computed by a single party, and can be verified using the same verification procedure without additional steps to preprocess the signed data.

A motivating use case for this is for WebAuthn/FIDO2 Authenticators to use when signing application data, as described in the proposed WebAuthn signing extension. Parts of this spec’s content were previously in the Asynchronous Remote Key Generation (ARKG) algorithm spec, which we’ve also been updated.

I plan to talk about the spec during IETF 122 in Bangkok. I hope to see many of you there!

The specification is available at:

https://www.ietf.org/archive/id/draft-lundberg-cose-two-party-signing-algs-01.html

This work was supported by the SIROS Foundation.

LCW Experience Badge & Introduction to Open Badges 3.0 Course at Participate

This week is the 2025 Digital Credentials Summit hosted by the organization that shepherds the Open Badges 3.0 standard. This is a great space to meet with implementers, educators, technologists, funders, and policy makers from all over the world to connect.

I will be attending this year and have the privilege to participate in five sessions covering topics like degree data models, trust, identity, open communities and open source software. I’ve listed my sessions at the bottom for those who may be attending.

In time for the Summit, check out the two projects the DCC team has launched to contribute to advancing understanding Open Badges 3.0.

LCW Experience Badge

The LCW Experience is an introduction to the experience of receiving, claiming, and storing of an Open Badges 3.0 badge to the Learner Credential Wallet. Get started here.

Introduction to Open Badges 3.0 Course at Participate

Introduction to Open Badges 3.0 is a short, videos based course hosted at Participate.com, a community learning platform that issues Open Badges 3.0. To get started, join the The Digital Credentials Consortium Learning Hub and there you will find the course. Watch six short videos and earn a DCC Intro to OBv3 badge.

Sessions at Digital Credentials Summit

The DCC has been busy this year working with our members and colleagues on great projects about digital degrees, issuer trust, open web, and open source software. Here’s a list of these sessions:

March 3, 3:30–4:15 PM MT Using Open Badges 3.0 to Issue Portable and Verifiable Digital Degrees, with Eric Sembrant from Georgia Tech March 4, 9:20–10:20 AM MT Understanding Issuer Identity in the LER Verifiable Credentials Ecosystem, with Jeanne Kitchens from Credential Engine March 4, 1–1:40 PM MT Quality in Digital Credentialing: A Multifaceted Approach, panel with National Student Clearinghouse, Territorium, Strada Education Foundation, and Western Governors University March 4, 3–3:40 PM MT The Story of Open — How the Web, the World, and Credentials Thrive When Collaboration & Community is Prioritized, with Kate Giovacchini from ASU March 4, 4:40–5:20 PM MT, The Power of Open-Source Badging in Micro-Credentialing, with colleagues from Digital Promise

DCC at the 2025 1EdTech Digital Credentials Summit was originally published in Digital Credentials Consortium on Medium, where people are continuing the conversation by highlighting and responding to this story.

Join the Digital Credentials Consortium Learning Hub

The Digital Credential has started a series of short posts explaining the characteristics and trust models of W3C Verifiable Credentials and Open Badges 3.0. To accompany this series, we collaborated with our colleagues as Participate to create the The Digital Credentials Consortium Learning Hub and offer a short course, “Introduction to Open Badges 3.0”. To get started, join the The Digital Credentials Consortium Learning Hub and there you will find the course.

The course consists of six short (approx 2 min), informal videos. Once you watch all six videos, click on the bell in the right top corner to see your “DCC Introduction to OBV3 badge”. After installing and setting up the Learner Credential Wallet on your mobile device, click on “Add to Wallet” and scan the QR code that displays in the pop up with the camera on your phone or using the “Scan QR code” option in the ‘Add Credential screen of the LCW app. If you’d like to practice claiming an Open Badges 3.0 badge first, try the LCW Experience.

Thanks to the Participate team for setting this up with us. Participate is one of the first (maybe THE first) community learning platform to issue Open Badges 3.0. Their team has been involved with Open Badges since the very beginning. They host the Open Recognition Community where you will find many with a long history in this space who are discussing the value of Open Badges at the cross sections of formal, informal, nonformal learning.

For more information on the DCC, including membership, projects, and events, visit our website, wiki, and Linkedin. To receive our newsletter and invitations to webinars and events sign up for our community mailing list.

Open Badges 3.0 Basics: Watch, Learn, and Earn on Participate was originally published in Digital Credentials Consortium on Medium, where people are continuing the conversation by highlighting and responding to this story.

Symbolic representation of Web Security applied to SSI.

Many children in Europe grew up with the tales of Baron Münchhausen, who claims to have lifted himself and his horse out of a mire by pulling his own hair. The image is so powerful because the problem of the circular dependency is so clearly visible. In real life, circular dependencies are often far less obvious.

Which is why the first article in this series was primarily focused on looking behind the SSI smoke and mirrors around Web based identifiers and communication protocols. The resulting discussions in the Rebooting the Web Of Trust (RWOT) community were quite enlightening, and included a deeper look at the EU Digital Identity Wallet Technical specifications.

One of the mirrors basically broke when claims of OpenID4VC supporting decentralized identifiers were shattered when someone pointed out that while the EU Wallet is marketed on digital sovereignty and privacy, but in reality does not does not allow decentralized identifiers:

The current EUDI approach: No decentralized identifiers allowed

So while it was clear that OpenID4VC and did:web* do not qualify as decentralized, Self-Sovereign Identity, some people advocated to just embrace the false marketing in the hope that it would create wider acceptance and the appearance of adoption for SSI.

But has that approach ever really worked?

More often this kind of “sovereignwashing” appears to run a high risk of creating false expectations, disappointment. Which would ultimately cement the status quo of the federated platform identity lock-in for the next 20 years. As a community we should focus on building actual decentralized identifiers, communication protocols, and applications.

Because the true social and economic value of SSI is not just in the identity layer itself, it is in the decentralized applications enabled as a result.

Some of which would be in direct competition to the champions of the platform age, who are investing their financial and political capital into OpenID4VC and Web based identifiers to prevent that competition from ever getting off the ground. A classic “old industry vs new technologies” battle.

There are real opportunity costs across most of economy and society if the old encumbents manage to postpone or kill innovation.

Symbolic representation of eIDAS 2.0 after successful lobbying by the platforms and trust intermediaries Security and privacy for a globally networked society

Technology and corresponding security have been head to head in a special kind of race for a long time, dating back to an Egyptian inscription around 1900 BC in the main chamber of the tomb of Khnumhotep II, over Julius Caesar using a ROT-3 cypher in 100 BC, all the way to the famous Enigma machine used in World War II. The more people potentially had access to a message, the harder the encryption had to become.

The encryption used by Julius Caesar was not particularly strong, because it relied on a supposedly secret algorithm. Once parties know the secret, encryption and decryption become trivial. Over time this moved to well-known algorithms using shared secrets. And even though the shared secrets are more complex on today’s internet, this fundamental principle hasn’t changed:

If you know the shared secret, and can intercept the encrypted message, you will be able to read, and also impersonate and falsify communication.

In contrast, Engima was quite strong for its day because it combined a rotating cypher with a codebook that was famously carried by U-Boats allowing them to choose the correct settings. Literally handed over to the commander of the boat by hand in a secure location before departure, these code books effectively represented a cryptographic key, shared over a second channel — the physical handover.

Which makes any well-designed encryption system almost impossible to break. Unless, of course, you have intimate knowledge of the inner workings of the rotating cypher, and can guess certain messages, like weather reports, to then use brute force to arrive back at the settings for the day. Those settings then allowed to read other messages, which would otherwise have been unbreakable.

Digital identity should be based on an advance

In other words: The cryptography of the Enigma machine itself was solid, and essentially unbroken. But the Allied Forces were able to exploit structural weaknesses designed into the operation of Engima to attack the key generation for the day.

Security in Swiss Healthcare

That particular race accelerated when the Internet was born. In 1996, when the internet was still young, the US Congress deliberated and passed the Health Insurance Portability and Accountability Act (HIPAA). That same year, the Swiss Medical Association (FMH), realized patient data had to be better secured on the internet, leading to the creation of Health Info Net (HIN). Starting from encrypted email, Swiss doctors have relied on HIN for decades to keep their patient data safe.

But technology years are a lot like dog years. And 28 years is a very long time.

HIN is constantly working to innovate and improve its solutions. Which is how Vereign, working closely with our partner More than Bits, started to run some POCs with HIN in 2023, and ended up working all of 2024 almost exclusively for the Swiss healthcare sector.

Our challenge: Design a system that starts from what today’s users are used to, while re-thinking the system architecture using SSI and modern data ecosystem architectures, based on the work we had done for Gaia-X.

The starting point was obvious: Email is the world’s largest distributed identity database and communication protocol. It is the use case with which HIN started, and it is the singular product that all users rely on mutliple times each day to communicate with colleagues, laboratories, and patients.

Email is also facing challenges of concentration and capture by the large, federated platforms. And its lack of an identity layer has made it a fertile ground for attacks by malicious governments, corporations, and common criminals.

Vereign showcased its first prototype to harden email using SSI in 2019, which earned us a nomination as the hottest new innovation for the Swiss Digital Economy Award in Zurich. COVID-19 had other plans, but our experience proved invaluable when working on the POCs with HIN.

This time, we built out peer to peer email exchange via DIDComm. Secure, encrypted, authentic and designed in a way that it can be plugged into any legacy email system to gradually switch to a new, identity verified transport layer reaching all the way to the people themselves.

From prototyping to production: Quest for the identifier

We built these prototypes using did:web, because it is a great placeholder to stand in for decentralized identifiers while rapidly prototyping around user flow and experience.

But from the onset it was clear that did:web would not be the choice for production, because for all the reasons also highlighted in the last article:

Web based identifiers must never be used for personal identity.

Our preferred choice would have been KERI due to its robust security and privacy architecture. But with the official implementation being Python only, we had concerns about efforts required in supporting a secure, long term solution across the range of platforms we anticipated.

The Rust implementation by the Human Colossus Foundation fared better on that front. But there seems to be a rift in the community, causing concerns of diverging implementations, as well as long-term support. Which are exacerbated by the choice for European Public License (EUPL).

We could not find information about adoption, nor community. And finally, the security of KERI as a concept critically depends on the networks of Witnesses and Watchers, for which we could not find information about size, health and long term viability of these networks for either implementation.

Had we chosen KERI in February 2024, we would not have been able to go productive before these issues had been resolved. And our time line dictated we had to be ready for initial production by late 2024. As a result, KERI was a non-starter.

Other methods, such as did:indy, have been in decline for some time, and Sovrin is shutting down in just a couple of weeks. Methods like did:peer on the other hand are not great in scenarios where long-lived connections are desirable.

So in the end, our search for production ready decentralized identifiers that could safely be used for natural persons left us empty handed.

A classic. And good advice. Ignoring XKCD

The competing standards comic by XKCD is a classic. As far as rules go, it is a good one. But there are no rules without exceptions. Having exhausted every other path, we decided to ignore XKCDs’ best practice. Only, we did not aim to create the universal solution — that’s KERI — but to create the simplest possible, yet still sufficiently safe identifier for the requirements of our specific use case.

Like any good design, it should build on existing technologies as much as possible, be simple enough to be implemented within a reasonable time frame, and to be supportable for at least 5–10 years, when potentially it would be replaced by something better.

Designing a decentralized identifier

Our requirements asked for an identifier that was truly secure and private. We explicitly sought to minimize dependencies on infrastructure such as DNS, Web Servers and Certificate Authorities. Blockchain would have fit these criteria, but we do not require a global consensus. All we needed was a decentralized storage system that would guarantee integrity and availability of records.

Git might have been an option. It is Content-Addressable Storage, so objects are referenced by their hash, any modification creates a new object. But Git would add unnecessary overhead, and there is a central repository. The Interplanetary File System (IPFS) on the other hand is built for peer to peer distribution between nodes without a central server.

Like Git, IPFS is built on Content-Addressable Storage (CAS). Objects are referenced by their sha256 hashes. Users can request data at any node, and if that node does not have this particular object, it will use peer-to-peer network connectivity between nodes to obtain a copy of the data and provide it to the user. It is open, verifiable, and resilient.

Its function allows DID documents to be uploaded onto any node and be referenced by their hash on any node in the network. Modifications to the document modify the hash, so documents are integrity protected by design. Simultaneously, the entire DID storage and distribution mechanism is robust regarding the well-known attacks against Web based identifiers.

In addition, the hash for the document contains no Personally Identifiable Information (PII) and unless we’d make the mistake of adding PII to the DID documents themselves, our design would not expose any kind of PII anywhere.

Of course we were not the first, nor the only ones to realize the potential of IPFS for decentralized identifiers. There has been a prior attempt at using IPFS for DID documents, the IPID DID Method. But it never got much traction, and its use of the InterPlanetary Name System (IPNS) made it less robust. Also, it did not have provisions for the rotation of keys, which is crucial for long-term connections with the same identifier, as well as the ability to switch wallets or upgrade crypto algorithms.

Swiss Healthcare: Innovating together toward the gold standard of decentralized, secure, private identity and applications An identifier for Sovereign Data Exchange (SVDX)

The result is did:svdx, our DID method for Sovereign Data Exchange.

Agents generate their active key locally, as well as a key that can be used for the update of the identifier later. The public key of the first key is used as the persistent identifier, creating a persistent Autonomous Identifier (AID).

The second key, which is used for the update of the identifier, is never shared. Only its hash is declared in the document as a next key commitment. Because this key is never actively used until it is time to rotate, it is well protected against being compromised.

Each revision of the decentralized identity documents representing a Decentralized Identifier has a Content Identifier (CID) when stored in IPFS, so the resulting identifier is always the combination of the AID with the CID of the latest revision of the identifier.

Since each revision of the identifier refers back to the previous version by its CID, the result is a sha-256 hash based Key Event Chain of IPFS objects, all the way back to the inception document, the root of the AID in question.

did:svdx:z6MknHKiY477mH97qryHv3zjuHaTLvBbbp6tHS5SvZv67uR4:QmecqVGBxvW7gjffxmYTGFZNPmJcWmYPdD8azB1cZYaY6F

Because the identifier also contains the CID of the current state, starting verification of the Key Event Chain is trivial: Just pull the corresponding object out of IPFS and verify. Check for ancestor, rinse and repeat until you’re at the beginning of the chain. Check whether the AID matches the initial key. Done.

Trivial to implement in web based tool chains

No native IPFS support? No problem. Just pick one of the public IPFS gateways, and with a single request pull the DID document, e.g. https://ipfs.io/ipfs/QmecqVGBxvW7gjffxmYTGFZNPmJcWmYPdD8azB1cZYaY6F.

Thanks to content based addressing, you will get the same document no matter which gateway you use. And you’re welcome to use as many of them as you would like to compare. Although for production use cases it is highly recommended to run your own, which is trivial.

In other words, IPFS allows to integrate classic web based tool chains with decentralized storage and delivery of integrity protected DID documents. It’s as easy as any of the did:web* methods to work with, but does not suffer from the attack surfaces of DNS, TLS and Certificate Authorities.

In addition, it is robust against a number of DDOS scenarios, allows for low impact self-hosting, and eliminates the web server as a central point of attack, surveillance and compromise.

Also, it plays well with DIDComm and other communication protocols, but if you really require web based interaction protocols, they can also be encoded into the identifier. But unlike web based identifiers, exchanging key material via did:svdx mitigates a substantial number of attack scenarios for web connection protocols.

Layering trust

By design did:svdx contains zero personal information. It is deliberately focused on secure key exchange of an Autonomous Identifier, only.

So any relationship starts from a reliable assumption the AID controllers have a strong connection to one another and can maintain it over a longer period of time, including throughout key rotation and changes in cryptography. But they start from zero trust in one another.

Trust is built gradually, through Verifiable Presentations securely exchanged over the connection. Similar to what Christopher Allen describes as “Building Trust in Gradients.”

For SVDX, given it is built for a true P2P, decentralized ecosystem, we surmise that the party initiating a connection first authenticates itself toward the recipient of the connection request before requesting reciprocal information. That should also make data mining or identifier scraping much harder.

Limits of did:svdx

For any design, it is crucial to know its limits. Firstly, the identifier specification does not contain any of the multi-signature capabilities of systems like KERI. Because we did not require it for our use case at hand, we pushed that complexity, along the complexity of secure restore and key rotation, onto the clients — which we control for the use case at hand.

Also, while IPFS plays a role similar to that of Witnesses in KERI, there are no Watchers. So there is no built-in detection of duplicity, as Sam Smith calls it. And while parties can update each other on key rotations using DIDComm, allowing each other to verify they are still talking to the same party, the design has no built-in protections against a controller forking their identity.

For our use case this was not an issue, because there is a central catalogue for the ecosystem to allow looking up the latest, known version of an AID. Which is not ideal for some scenarios. But we considered the solution good enough for what we needed to achieve, given that all controllers need to also maintain their identity and trustworthiness with HIN as the central ecosystem fiduciary.

That said, it should be possible to design a robust duplicity detection on top of did:svdx, and there may even be scenarios where duplicity is not a primary concern as long as agents always ensure to only consider the latest version of an AID authoritative.

So did:svdx is not a replacement for KERI. But it is a replacement for web based DID methods, offering far better security, and similar efforts of adoption and support. From our own experience we know it took around 6-8 weeks to implement in JavaScript.

What’s next?

The first application using did:svdx in production will have ramped up by April 2025.

By mid 2025 we expect hundreds of thousands of production messages sent each month containing verifiable credentials backed by did:svdx. Our roadmap has us building out additional applications until all the institutions and eventually all the patients in Switzerland will have identifiers within the next 2-3 years.

We have already Open Sourced the initial implementation and will continue to add additional implementations. Also, we would love to finalize the specification so that it can be maximally useful to others. And there may be features that would be required for additional use cases, as well as community-based methods for duplicity detection.

Open questions Where is the right place to finalize, publish and maintain did:svdx? Who would be interested in participating? What are the critical capabilities that may still be missing? What kind of best practice operational RFCs should we develop as a community?

If you’re at DICE in Zurich this year, I’d love to sit down and discuss these questions with you — alongside everything else you would like to know about our vision for the Sovereign Data Exchange.

The Digital Credentials Consortium, our team and our members, are aiming to advance the understanding & use of privacy-enhanced, portable, verifiable digital credentials like Open Badges 3.0. We’ve noticed that while many are interested in the opportunities LERs as digital credentials may offer, many haven’t yet experienced what it is like to claim and accept an Open Badge 3.0 credential with a digital wallet like the LearnerCredential Wallet (LCW).

We created the LCW Experience to make it as easy as possible to try this out. It uses the open source DCC Issuing System and Learner Credential Wallet to demonstrate the experience of requesting and storing an Open Badges 3.0. You can learn more about how it works here.

To add the LCW Experience Badge to your Learner Credential Wallet, follow these steps:

Install & set up the LCW.app on your mobile device Go to the LCW Experience site to enter your name & email address

After you click the “Award Badge” button, a message will be sent to the email you provided with a link to your claim page. There you will find instructions on how to use your LCW app to claim your credential.

The email notification is a typical way to let earners know that they have been awarded a badge. If you would rather go to the claim page directly, use this link, replacing the name with your own (or someone you’d like to send it to): https://badging.dcconsortium.org/collect?recipientName=FirstName+LastName

Note that your name & email are not saved in our system. The email is used to send you the message and your name is included so that it will show who it was issued to in the accepted badge data and display.

Big shout out to the DCC team, James Chartrand, Alex Higuera, Dmitri Zagidulin, and Gillian Walsh for developing this project and brainstorming fun, accessible, and pragmatic approaches to learning about Verifiable Credentials and Open Badges. We hope you enjoy this experience!

For more information on the DCC, including membership, projects, and events, visit our website, wiki, and Linkedin. To receive our newsletter and invitations to webinars and events sign up for our community mailing list.

The LCW Experience was originally published in Digital Credentials Consortium on Medium, where people are continuing the conversation by highlighting and responding to this story.

This post shows how to implement phone (SMS) verification and two-factor authentication (2FA) using ASP.NET Core Identity. The solution integrates phone-based verification and 2FA mechanisms. The implementation uses ASP.NET Core Identity’s extensibility to incorporate SMS-based verification during user registration and login processes. SMS is no longer a recommended authentication method due to security risks but does provide a good solution for some business cases or user flows like onboarding phone users or phone applications, frontline workers with no desktop or other such solutions with limited security possibilities.

Code: https://github.com/damienbod/IdentityOidcPhone2fa

Setup

The ASP.NET Core Identity application integrates the SMS provider using the Identity PhoneNumberTokenProvider and an SMS verification service.

builder.Services.AddDbContext<ApplicationDbContext>(options => options.UseSqlServer(builder.Configuration.GetConnectionString("DefaultConnection"))); builder.Services.Configure<EmailSettings>(builder.Configuration.GetSection("EmailSettings")); builder.Services.AddTransient<IEmailSender, EmailSender>(); builder.Services.Configure<SmsOptions>(builder.Configuration.GetSection("SmsOptions")); var authorization = Convert.ToBase64String(Encoding.ASCII.GetBytes( $"{builder.Configuration["SmsOptions:Username"]}:{builder.Configuration["SmsOptions:Password"]}")); builder.Services.AddHttpClient(Consts.SMSeColl, client => { client.BaseAddress = new Uri($"{builder.Configuration["SmsOptions:Url"]}"); client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Basic", authorization); }); builder.Services.AddScoped<SmsProvider>(); builder.Services.AddIdentity<ApplicationUser, IdentityRole>() .AddEntityFrameworkStores<ApplicationDbContext>() .AddTokenProvider<DataProtectorTokenProvider<ApplicationUser>>(TokenOptions.DefaultProvider) .AddTokenProvider<AuthenticatorTokenProvider<ApplicationUser>>(TokenOptions.DefaultAuthenticatorProvider) .AddTokenProvider<PhoneNumberTokenProvider<ApplicationUser>>(Consts.Phone) .AddTokenProvider<EmailTokenProvider<ApplicationUser>>(Consts.Email);

The ApplicationUser needs some new properties to support multiple authentication methods. The properties are used to allow a user to use the selected authentication method or force an authentication on a OpenID Connect client.

public bool Phone2FAEnabled { get; set; } public bool Email2FAEnabled { get; set; } public bool AuthenticatorApp2FAEnabled { get; set; } public bool Passkeys2FAEnabled { get; set; }

An SMS service are used to integrate the SMS, the SmsProvider class. In this demo, the eColl messaging service is used to send SMS. The implementation and the configuration would vary if you use a different service.

The SmsProvider is used to verify a phone number, to enable SMS 2FA and to force SMS 2FA. The service uses a HttpClient to access the SMS service rest API.

using IdentityProvider.Models; using Microsoft.AspNetCore.Identity; using Microsoft.Extensions.Options; namespace IdentityProvider.Services; public class SmsProvider { private readonly HttpClient _httpClient; private readonly UserManager<ApplicationUser> _userManager; private readonly SmsOptions _smsOptions; private readonly ILogger<SmsProvider> _logger; private const string Message = "message"; public SmsProvider(IHttpClientFactory clientFactory, UserManager<ApplicationUser> userManager, IOptions<SmsOptions> smsOptions, ILogger<SmsProvider> logger) { _httpClient = clientFactory.CreateClient(Consts.SMSeColl); _userManager = userManager; _smsOptions = smsOptions.Value; _logger = logger; } public async Task<(bool Success, string? Error)> Send2FASmsAsync(ApplicationUser user, string phoneNumber) { var code = await _userManager.GenerateTwoFactorTokenAsync(user, Consts.Phone); var ecallMessage = new EcallMessage { To = phoneNumber, From = _smsOptions.Sender, Content = new EcallContent { Text = $"2FA code: {code}" } }; var result = await _httpClient.PostAsJsonAsync(Message, ecallMessage); string? messageResult; if (result.IsSuccessStatusCode) { messageResult = await result.Content.ReadAsStringAsync(); } else { _logger.LogWarning("Error sending SMS 2FA, {ReasonPhrase}", result.ReasonPhrase); return (false, result.ReasonPhrase); } return (true, messageResult); } public async Task<(bool Success, string? Error)> StartVerificationAsync(ApplicationUser user, string phoneNumber) { var token = await _userManager.GenerateChangePhoneNumberTokenAsync(user, phoneNumber); var ecallMessage = new EcallMessage { To = phoneNumber, From = _smsOptions.Sender, Content = new EcallContent { Text = $"Verify code: {token}" } }; var result = await _httpClient.PostAsJsonAsync(Message, ecallMessage); string? messageResult; if (result.IsSuccessStatusCode) { messageResult = await result.Content.ReadAsStringAsync(); } else { _logger.LogWarning("Error sending SMS for phone Verification, {ReasonPhrase}", result.ReasonPhrase); return (false, result.ReasonPhrase); } return (true, messageResult); } public async Task<bool> CheckVerificationAsync(ApplicationUser user, string phoneNumber, string verificationCode) { var is2faTokenValid = await _userManager .VerifyChangePhoneNumberTokenAsync(user, verificationCode, phoneNumber); return is2faTokenValid; } public async Task<(bool Success, string? Error)> EnableSms2FaAsync(ApplicationUser user, string phoneNumber) { var token = await _userManager.GenerateChangePhoneNumberTokenAsync(user, phoneNumber); var message = $"Enable phone 2FA code: {token}"; var ecallMessage = new EcallMessage { To = phoneNumber, From = _smsOptions.Sender, Content = new EcallContent { Text = message } }; var result = await _httpClient.PostAsJsonAsync(Message, ecallMessage); string? messageResult; if (result.IsSuccessStatusCode) { messageResult = await result.Content.ReadAsStringAsync(); } else { _logger.LogWarning("Error sending SMS to enable phone 2FA, {ReasonPhrase}", result.ReasonPhrase); return (false, result.ReasonPhrase); } return (true, messageResult); } }

Flow 1: Verify phone

Once a user has authenticated with email and password, the user can verify a phone. To verify the phone, the user MUST be authenticated. If not, a malicious program may send multiple SMS and cause financial harm. The Add phone number link can be used to start the verification process.

The VerifyPhone Razor page allows the user to enter an mobile phone number to send the SMS. This should be validated for real phone numbers at least. The StartVerificationAsync method is used to send the SMS. The ASP.NET Core Identity method GenerateChangePhoneNumberTokenAsync is used to generate the challenge for the verification.

var user = await _userManager.GetUserAsync(User); if (user == null) { return NotFound($"Unable to load user with ID '{_userManager.GetUserId(User)}'."); } var result = await _client.StartVerificationAsync(user, Input.PhoneNumber);

The UI is not styled, just uses the standard bootstrap styles.

The confirm Razor Page accepts the verification code which was sent to the phone and uses the VerifyAndProcessCode method to validate. The ASP.NET Core Identity VerifyChangePhoneNumberTokenAsync method is used to validate the code.

public async Task<IActionResult> OnPostAsync() { if (!ModelState.IsValid) { return Page(); } try { if (Input.PhoneNumber != null && Input.VerificationCode != null) { return await VerifyAndProcessCode(Input.PhoneNumber, Input.VerificationCode); } else { ModelState.AddModelError("", "Input.PhoneNumber or Input.VerificationCode missing"); } } catch (Exception) { ModelState.AddModelError("", "There was an error confirming the code, please check the verification code is correct and try again"); } return Page(); } private async Task<IActionResult> VerifyAndProcessCode(string phoneNumber, string code) { var applicationUser = await _userManager.GetUserAsync(User); if (applicationUser != null) { var validCodeForUserSession = await _client.CheckVerificationAsync(applicationUser, phoneNumber, code); return await ProcessValidCode(applicationUser, validCodeForUserSession); } else { ModelState.AddModelError("", "No user"); return Page(); } } private async Task<IActionResult> ProcessValidCode(ApplicationUser applicationUser, bool validCodeForUserSession) { if (validCodeForUserSession) { var phoneNumber = await _userManager.GetPhoneNumberAsync(applicationUser); if (Input.PhoneNumber != phoneNumber) { await _userManager.SetPhoneNumberAsync(applicationUser, Input.PhoneNumber); } applicationUser.PhoneNumberConfirmed = true; var updateResult = await _userManager.UpdateAsync(applicationUser); if (updateResult.Succeeded) { return RedirectToPage("ConfirmPhoneSuccess"); } else { ModelState.AddModelError("", "There was an error confirming the verification code, please try again"); } } else { ModelState.AddModelError("", "There was an error confirming the verification code"); } return Page(); }

The UI displays the input for the code and the number it was sent to.

Flow 2: Enable phone 2FA

Once the phone is verified, it can be used for an SMS 2FA.

The EnableSms2FaAsync method is used to enable the SMS 2FA.

var user = await _userManager.GetUserAsync(User); if (user == null) { return NotFound($"Unable to load user with ID '{_userManager.GetUserId(User)}'."); } if (user.PhoneNumber != Input.PhoneNumber) { ModelState.AddModelError("Input.PhoneNumber", "Phone number does not match user user, please update or add phone in your profile"); } await _smsVerifyClient.EnableSms2FaAsync(user, Input.PhoneNumber!); return RedirectToPage("./VerifyPhone2Fa", new { Input.PhoneNumber });

The EnablePhone2Fa Razor page is used to validate the phone number before activating the 2FA.

The VerifyChangePhoneNumberTokenAsync is used to validate and the 2FA is activated.

var user = await _userManager.GetUserAsync(User); if (user == null) { return NotFound($"Unable to load user with ID '{_userManager.GetUserId(User)}'."); } var verificationCode = Input.Code.Replace(" ", string.Empty).Replace("-", string.Empty); var is2faTokenValid = await _userManager .VerifyChangePhoneNumberTokenAsync(user, verificationCode, user.PhoneNumber!); if (!is2faTokenValid) { ModelState.AddModelError("Input.Code", "Verification code is invalid."); return Page(); } await _userManager.SetTwoFactorEnabledAsync(user, true);

The Verify Phone 2FA Razor page displays the input field for the code.

Once activated, the user should persist some recovery codes.

Flow 3: SMS 2FA using phone

Next time the user authenticates, the SMS 2FA is required. The user can use multiple authentication methods, not only SMS. If possible, passkeys or strong authentication should be used.

if (user.Phone2FAEnabled) { IsPhone = true; if (!user.AuthenticatorApp2FAEnabled) { await _smsVerifyClient .Send2FASmsAsync(user, user.PhoneNumber!); } }

Further flows Phone only authentication

Requires mass usage protection

Recover account using Phone authentication

Requires mass usage protection

Links

https://learn.microsoft.com/en-us/aspnet/core/security/authentication/2fa

https://github.com/andrewlock/TwilioSamples/blob/master/src/SendVerificationSmsDemo

Professionell Online SMS senden

It’s time to rekindle the movement.

Continue reading on Medium »

Vladimir Dzhuvinov and I led a discussion on The Cambrian Explosion of OAuth and OpenID Specifications at the 2025 OAuth Security Workshop in Reykjavík.

The abstract for the session was:

The number of OAuth and OpenID specifications continues to grow. At present there are 30 OAuth RFCs, two more in the RFC Editor queue, 13 OAuth working group drafts, and another eight individual OAuth drafts that may advance. There are nine JOSE RFCs and seven working group drafts. There are four SecEvent RFCs. On the OpenID side, there are 12 final OpenID Connect specs, three final FAPI specs, one final MODRNA spec, three final eKYC-IDA specs, and 24 Implementer’s drafts across the OpenID working groups, plus another ten working group drafts.

The number of possible combinations boggles the mind. And there’s no end in sight!

What’s a developer to do? How have people and companies gone about selecting and curating the specs to implement in an attempt to create coherent and useful open source and commercial offerings? And faced with such an array of combinations and choices, how are application developers to make sense of it all? How can interoperability be achieved in the face of continued innovation?

This session will prime the pump by discussing choices made by some existing open source and commercial offerings in the OAuth and OpenID space and lead to an open discussion of choices made by the workshop attendees and the reasoning behind them. It’s our goal that useful strategies emerge from the discussion that help people grapple with the ever-expanding sets of specifications and make informed implementation choices, while still fostering the innovation and problem-solving that these specifications represent.

The slides used to queue up the discussion session are available as PowerPoint and PDF. Also, see the list of 101 OAuth and OpenID-related specifications referenced during the discussion.

The topic seems to have touched a chord. Many people were clearly already thinking about the situation and shared their views. Some of them were:

Nobody actually expects everyone to implement everything. Stopping things is super hard. But sometimes it’s necessary (as Brian Campbell put it, “when they’re wrong”). Timing can be fickle. What may not be useful at one time can turn out to be useful later. Some specs are highly related and often used together. But those relationships are not always apparent to those new to the space. We need better on-ramps to help people new to the space wrap their arms around the plethora specs and what they’re useful for. Well-written profiles are a way of managing the complexity. For instance, FAPI 2 limits choices, increasing both interoperability and security. The amount of innovation happening is a sign of success!

Thanks to the organizers for a great tenth OAuth Security Workshop! And special thanks to the colleagues from Signicat who did a superb job with local arrangements in Reykjavík!

 Tune your metaphors.. 

 From where I dance and glance in America...

All over the world, men and women are not having the same experience. Generational conflicts are arising, influencing everything, causing chaos to status quo, and making clear that the ability to r/w/rw is not shared equally. Computational supremacy, computational illiteracy, what does the data say? Who? What? When? Why? Where? How? Now what?  

FREEdumb self-certifies. 

Structure yields results... not empathy, not emotion, not opinion. 

You own root, or you don't. 

Owners own the world. One contract at a time. Data, under Terms.

Sovereignty emerges from people, Individuals all. 

Freedom self-certifies. 

Left Cult == Right Cult political paradigm is collapsing.

Something new is possible.

You, Sovereign. 

Data structure yields life results... own root authority, people, it is your Right as Individuals.

The ultimate minority of one. Love somebody. 

 

The Pulse is a series covering insights, patterns, and trends within Big Tech and startups. Notice an interesting event or trend? Send me a message.

Today, we cover:

Industry pulse. Amazon launches Alexa+, VS Code pulls “epic” but malicious theme, Meta cuts stock awards by 10%, Humane sells to HP, and more.

Swarm of AI coding tool launches. Anthropic launches Sonnet 3.7 (a supposedly even more capable LLM for coding) and Claude Code. GitHub ships Copilot Code Reviews, and Google makes Gemini Code Assist free for individual users (from $22.80 per month). Us devs are showered with new tools that all aim to improve workflows.

DeepSeek triggers higher GPU demand? It’s been predicted that more efficient LLMs will decrease demand for GPUs, but DeepSeek is 4-5x as efficient as some current LLMs, and appears to be driving up demand. Also, a look at the very clever “hacks” DeepSeek uses to boost performance on NVIDIA GPUs.

VC funding down, AI funding up. VC firms are increasingly shutting down, with overall funding levels dropping. AI-related funding is up – along with competition among startups to get funded. We predicted VC funding could dry up due to interest rate changes, and the data seems to show this happening.

What does “zero” mean for dates in COBOL? A reminder that “0” and “null” might not mean what you expect in some programming languages and systems.

1. Industry Pulse Amazon launches Alexa+

A year ago, I analyzed how Alexa had become left behind in the conversational AI race by missing the LLM wave, despite Amazon Alexa being a great use case for an integrated LLM, in order to provide better conversational experiences and new capabilities.

Well, 12 months later, Amazon seems to have caught up. This week, the online retail giant announced Alexa+, which is a revamped, more capable conversational assistant. It promises to make reservations or book rides by using apps like OpenTable and Uber, buy things on Amazon (obviously), manage smart speakers in the home, discover new music, and more.

The launch demo looks very polished and capable. But then, Apple’s glossy launch demos for Apple Intelligence didn’t stop the feature still feeling half-baked months later.

Under the hood, Alexa is partially powered by Claude, which Amazon didn’t announce, and Anthropic did. This suggests Anthropic’s models are ahead of Amazon’s own Nova models. Still, Amazon is clearly creating optionality for itself by not advertising Alexa’s dependence on Claude; when and if the Nova models are good enough, Amazon could replace Anthrophic’s solution with its own.

The Amazon+ pricing is puzzling: Alexa+ is free for Amazon Prime members ($14.99 monthly), and $19.99 per month for everyone else. This seems to make little sense; why charge so much more for a standalone subscription than for a bundle? Is it because Amazon wants to dis-incentivize customers just subscribing to Alexa+?

Either way, Amazon is making a stronger-than-ever case for customers to choose Amazon Prime for $14.99: which is probably the goal of upgrading Alexa.

VS Code pulls “epic” malicious theme

Read more

Stream the Latest Episode

Available now on YouTube, Apple and Spotify. See the episode transcript at the top of this page, and a summary at the bottom.

Brought to You By

• WorkOS — The modern identity platform for B2B SaaS.

• Graphite — The AI developer productivity platform.

• Formation — Level up your career and compensation with Formation.

—

In This Episode

Have you seen this comic before about the org structure of Big Tech companies?

Longtime readers will recall that the author of this comic is Manu Cornet — who previously shared details on how this comic came about (and he almost did not publish it, thinking it would not be funny). For today’s episode, I sat down with him, to go through more comics, why he created them, and the deeper meaning behind several of these.

Manu spent over a decade at Google, doing both backend and frontend development. He also spent a year and a half at Twitter before Elon Musk purchased it and rebranded it to X. But what Manu is most known for are his hilarious internet comics about the tech world, including his famous org chart comic from 2011 about Facebook, Apple, Amazon, and Microsoft.

In today’s conversation, we explore many of his comics, discuss the meaning behind them, and talk about the following topics:

The viral org chart comic that captured the structure of Big Tech companies

Why Google is notorious for confusing product names

The comic that ended up on every door at Google

How Google’s 20% time fostered innovation—and what projects came from it

How one of Manu’s comics predicted Google Stadia’s failure—and the reasons behind it

The value of connecting to users directly

Twitter’s climate before and after Elon Musk’s acquisition and the mass layoffs that followed

And more!

This is an unusually visual episode, offering the story behind several comics about Big Tech, Google and Twitter that you might have seen. With Manu, we discuss the deeper meaning behind several of these — ones that are pretty obvious to those who worked inside these companies, but might be less intuitive if you’re only seeing them for the first time.

The Pragmatic Engineer deepdives relevant for this episode

• How Manu creates comics

• Consolidating technologies

• Is Big Tech becoming more cutthroat?

Timestamps

(00:00) Intro

(02:01) Manu’s org structure comic

(07:10) Manu’s “Who Sues Who” comic

(09:15) Google vs. Amazon comic

(14:10) Confusing names at Google

(20:00) Different approaches to sharing information within companies

(22:20) The two ways of doing things at Google

(25:15) Manu’s code reviews comic

(27:45) The comic that was printed on every single door of Google

(30:55) An explanation of 20% at Google

(36:00) Gmail Labs and Google Stadia

(41:36) Manu’s time at Twitter and the threat of Elon Musk buying

(47:07) How Manu helped Gergely with a bug on Twitter

(49:05) Musk’s acquirement of Twitter and the resulting layoffs

(59:00) Manu’s comic about his disillusionment with Twitter and Google

(1:02:37) Rapid fire round

A summary of the conversation The org chart comic

Manu created this comic in 2011 featuring the org charts of big tech companies like Amazon, Google, Facebook, Apple, Oracle, and Microsoft.

Manu almost didn't publish the comic because he didn't find it funny by the time he finished drawing it. The punchline was Oracle having a large legal department and a tiny engineering department.

The comic became very popular and has been referenced heavily.

The culture inside Google

Google has a problem naming things, which Manu attributes to its bottom-up culture and engineers starting competing projects.

There isn't much top-down pressure to consolidate efforts, leading to consumer confusion.

An example of Google's naming issues is the evolution from Google Wallet to Android Pay to Google Pay, with logo changes causing confusion for partners like Uber.

Manu created a comic about this: how confusing it is to interact with Google products due to the different names.

The “Google Graveyard comic” was prophetic: it predicted that Google's cloud gaming console Stadia would eventually be killed.

Google vs Amazon

Manu created a comic contrasting Google and Amazon in their engineering culture, and how they treat customers

Amazon is known for bending over backwards for customers, including AWS clients.

Google focuses on building useful features but lacks strong customer service.

Amazon's on-call practices are tough, while Google's are more relaxed, with SREs taking on much of the on-call load.

Google’s engineering culture

The comic about migrations at Google: shows the challenges of adopting new systems before they are ready. One Google exec said, "There's two ways to do things at Google. There's the deprecated way and there's the way that doesn't work yet".

One on code reviews: shows the software engineer trying to replace a light bulb while being critiqued on how to screw it in.

Another comic depicts the author of the change who says, "Oh, I don't need to be too cautious about my code. If there's some problem, the reviewer will catch it," and the reviewer who says, "Oh, I don't need to review this too carefully if the author must've know what he's doing or what she's doing".

Google used Manu's "Beware the Tailgator" comic on office doors to prevent theft by reminding employees not to let unverified people into the building. He created updated versions each year.

Google's 20% time allowed employees to dedicate a portion of their time to projects of their choosing.

Gmail and Google News were started as 20% projects.

The company has become more traditional and less encouraging of 20% projects.

There may be a connection between Google's 20% time and the high number of products it shuts down.

Google could experiment internally without necessarily launching everything publicly.

Gmail Labs allowed people to launch things with lower scrutiny and made it clear that these were not officially supported and they may go out at any point.

Manu created a comic about ending of this practice as well

Move to Twitter and Musk’s takeover

Manu left Google and joined Twitter because it felt like a younger version of Google, with a friendly culture and less red tape.

Shortly after Manu joined Twitter, Elon Musk began his attempt to acquire the company, creating uncertainty.

After Musk bought Twitter, there were massive firings, with at least 50% and ultimately around 80% of employees being let go.

Manu created a bunch of comics about the events at Twitter – and later also published a comic book about it

While Manu was also let go from Twitter: the cartoonist side of himself was having a great time witnessing the events at Twitter.

As a fun fact, Manu doesn't have a favorite programming language. He chooses the best tool for the job, and has been coding with Java, JavaScript, Python, C++, Objective-C, Swift, Rust, and C. He also built a site with all his book, movie and music recommendations.

Where to find Manu Cornet:

• Mastodon: https://twit.social/@manu

• LinkedIn: https://www.linkedin.com/in/manucornet/

• Website: https://ma.nu/

Mentions during the episode:

• Code Review on Printed Paper: an Excerpt from the Twitoons Comic Book: https://newsletter.pragmaticengineer.com/p/code-review-on-printed-paper-an-excerpt

• A much clearer insight into who sues who: https://bonkersworld.net/who-sues-who

• CEO's "Burning Platform" Memo Highlights Nokia's Woes: https://www.forbes.com/sites/ericsavitz/2011/02/09/ceos-burning-platform-memo-highlights-nokias-woes/

• Guns and Roses: https://bonkersworld.net/guns-and-roses

• The Full Circle on Developer Productivity with Steve Yegge: https://newsletter.pragmaticengineer.com/p/steve-yegge

• Grab: https://www.grab.careers/en/

• Comic about the confusing Google names: https://goomics.net/207

• Naming dashboard: https://goomics.net/239

• Access: https://goomics.net/370

• Welcome to Google: https://goomics.net/50

• Eric Schmidt on X: https://x.com/ericschmidt

• Jonathan Rosenberg on X: https://x.com/jjrosenberg

• Code Reviews: https://bonkersworld.net/code-reviews

• Beware the Tailgator: https://goomics.net/56

• 20% time: https://goomics.net/343

• Where did Gmail labs go?: https://zapier.com/blog/gmail-labs-missing/

• Google Stadia: https://en.wikipedia.org/wiki/Google_Stadia

• Google engineer who criticized the company in viral comics on why he finally quit: https://mashable.com/article/google-engineer-manu-cornet-comics-critique

• Network effect: https://goomics.net/324

• The Twitter Worker Who Captured Elon Musk’s Takeover in All Its Cartoonish Glory: https://slate.com/technology/2022/11/elon-musk-twitter-cartoons-manu-cortnet.html

• Howl’s Moving Castle: https://www.imdb.com/title/tt0347149/

• Tesla: https://www.tesla.com/

• Twittoons: One employee's cartoon chronicle of Twitter's accelerated descent: https://www.amazon.com/Twittoons-employees-chronicle-Twitters-accelerated/dp/1952629020

• Javascript: https://www.javascript.com/

• C++: https://en.wikipedia.org/wiki/C%2B%2B

• Python: https://www.python.org/

• Swift: https://www.swift.org/

• Rust: https://www.rust-lang.org/

• CSS: https://en.wikipedia.org/wiki/CSS

• Manu’s favorites: https://ma.nu/faves/

• Mind the Goof (Gomer Goof #1): https://www.amazon.com/Mind-Goof-Gomer-Franquin/dp/1849183589

• Building Software: https://bonkersworld.net/building-software

—

Production and marketing by Pen Name. For inquiries about sponsoring the podcast, email podcast@pragmaticengineer.com.

Today, there’s an ever-growing number of startups, scaleups, and established companies like Boston Dynamics producing increasingly capable robots, including ones that look humanoid. Tech giants like Tesla have been building humanoid robots, and Meta is expected to invest billions in the technology. These and other factors point to the possibility that, after artificial intelligence, robotics could be the “next big thing” in tech.

But what’s the process of building intelligent robots, and what’s it like as a day-to-day job for the developers who do? It’s not always easy to find this out because robotics startups are famously shrouded in secrecy – and with such cutting-edge technology, it’s unsurprising.

Academia is often a bit more open, so I turned to MIT Robot Learning Researcher, Sandor Felber. He’s a Humanoid Robot Learning Researcher at MIT in Boston, Massachusetts, and previously led a team that built a self-driving race car. Sandor’s also been a robotics intern at Tesla in Palo Alto, California, and a robotics R&D engineer at the Edinburgh Center of Robotics.

Today, he takes us through:

Getting into robotics. From personal interest at high school in electric motors, through studying electrical and mechanical engineering at university, building a driverless race car, interning at Tesla, and researching humanoid robots at MIT.

Robotics industry overview. Industrial robots are becoming more widespread, academia focuses on smaller “long-shot” bests, and industry (the commercial sector) executes on proven concepts.

Planning a robotics project. This is similar to most projects: start with a vision, make a plan, and break it down into steps. It’s always useful to be clear on “critical” vs “nice-to-have” features.

Development phase. Control strategies (model-based control vs learned controllers), simulation and hardware deployment differences, and techniques to make simulations more realistic.

Tech stack and tools. Python, C, C++, and Rust are popular programming languages. A list of tools for experiment tracking and simulation.

Testing, demoing, and shipping. Common reasons why robot deployments fail, an example of deployment of an industrial robot, and why ongoing customer support is a baseline expectation.

Day-to-day as a robot builder. There’s a big difference between academia and industry, and Sandor shares an overview of both.

This topic is intriguing because it combines software, hardware, and cutting-edge tech:

Robotics as a field, and where generative AI techniques and autonomous decision-making fit in. A mental model, as shared by Sandor

This is a guest post. If you’re interested in writing a deepdive for The Pragmatic Engineer – collaborating with Gergely on it – check here for more details.

With that, it’s over to Sandor. You can follow Sandor on LinkedIn, and learn more about his work on his website.

1. Getting into robotics, a personal account

My interest in robotics began in high school, where I wrote the junior equivalent of a dissertation on characterization methods for how electric motors behave under different conditions. This led me to pursue a degree in electrical and mechanical engineering at the University of Edinburgh, home to numerous renowned robotics and AI researchers.

Building a driverless electric race car was one of my bigger projects. At uni, I joined the Edinburgh University Formula Student (EUFS) team, where we designed, built, and raced a driverless electric race car. From a roboticist’s perspective, this is considered a wheeled mobile robot.

My team’s driverless (but also human-drivable) electric vehicle, SISU23D, at launch. Source: Sandor Felber

We designed and built several versions of the car. Here’s a later model:

SISU24D (D for driverless), at Silverstone racetrack, England. Source: Hao Duo

I started working in the electric powertrain team and was responsible for designing and implementing systems that generate and deliver power. It included:

Developing high-voltage battery systems

Integrating traction motors and encoders

Designing power electronics for charging, converting between voltage levels, etc.

Creating cooling systems for batteries, motors, and inverters.

Here’s the high-voltage battery pack I had a hand in designing:

The high-voltage battery pack of SISU24D, with lid and damping layers removed. There are 6 segments, each containing hundreds of cylindrical cells. The battery management system and the accumulator isolation relays are on the left. Source: Sandor Felber.

I eventually became the powertrain team’s lead, and upon returning from Tesla the following year, moved on to direct all operations related to the driverless vehicle’s hardware design, and later served as president of the team of around 140 members.

The whole team at Silverstone track. Left vehicle: #45 SISU23E, the electric human-driven vehicle. Right vehicle: #128 SISU23D. Source: Nikita Jegororvs

The students in the team worked on the project part-time, and everyone went above and beyond their academic requirements to get hands-on experience. It was a student-led project – and an especially cool one; we built race cars that drove on actual Formula 1 tracks!

Sandor behind the human-steerable wheel of the SISU23D, with the Light Detection and Ranging sensor (LiDAR) above his head. The team is celebrating defending our championship title in the FS-AI category of FSUK. Source: Nikita Jegorovs

Along the way, I discovered a passion for control theory. Control systems engineers tend to concern themselves with crafting control strategies that ensure optimal performance, from spacecraft trajectories to insulin delivery in diabetics.

In robotics, you could think of control theory as the invisible puppeteer of a robotic arm, except that instead of pulling strings, it's using mathematics to orchestrate every joint's motion in real time. As a robotic arm needs to smoothly pick up an egg without crushing it, control theory provides the mathematical "muscle memory" that turns crude motor commands into precise, graceful movements. It does this with constant sensor feedback and adjustments, with the approach adjusted depending on how sensitive the control system is to the various feedback signals.

My interest in control theory and background in batteries, powertrains (vehicle motors), and electrical control systems from previous internships landed me one at Tesla's robotics department. Since then, I’ve worked with academic and industrial stakeholders on projects ranging from quadruped (quad- as in four, and -ped as in legged) dog-like robots, to humanoid systems.

I’m currently at MIT’s Computer Science and Artificial Intelligence Lab (CSAIL), focusing on teaching humanoid robots to perform complex, real-world tasks using learned controllers, powered by neural networks. I’m particularly interested in embedding intelligence into robots by leveraging various types of learning, including supervised and unsupervised learning, offline and online deep reinforcement learning, and imitation learning.

I use these approaches for humanoid robot locomotion, manipulation, whole-body control, and teleoperation (remotely controlling the robot).

2. Robotics industry overview

Robotics companies are raising more money than ever before. In the first eight months of 2024, global robotics companies attracted a record amount of $10.8 billion in funding, averaging $1.35 billion per month! What’s all the hype fuelling this surge of capital?

In public debate, there’s no shortage of focus on labor shortages and stagnating productivity, which reduce growth and competitiveness. The US economy is estimated to be missing up to 4.6 million workers annually needed to maintain levels of supply and demand, according to a recent study. That number equates to 2% of the current US population. The same study suggests Germany needs to find an additional 1.6 million workers to keep current economic levels; 3% of its population.

This is one driver of industrial robots’ increased adoption, which shows no sign of slowing down. Sustaining economic growth amid shrinking labor pools is one viable path for robotics which is getting a lot of attention, as evidenced in the increased adoption of industrial robots, with global installations reaching nearly 600,000 units in 2024, as per the World Robotics Report. This figure surpassed previous benchmarks and shows that industries struggling with labor shortages can return to growth.

Industry robots are reshaping the automotive industry as key parts of assembly lines traditionally operated by humans. Source: Wevolver

If robots become ubiquitous, what will humans do? This is the most common concern I hear when I talk about my work. The question is an old one: on the New York subway, I stumbled upon a Holiday Nostalgia Ride, which offers a trip in an old subway car from between the 1930s and 1970s. My coach looked like it was from the 1960s, and inside was an advert about upskilling for “tomorrow’s jobs:”

A poster on the New York subway in the 1960s

The world has changed massively since then, thanks in part to automation, without which our standards of comfort wouldn’t be possible or affordable; think of the dishwasher that automates the washing up and helps canteens to offer a cheaper menu, than if they had to hire extra people to wash dishes and pay for excess water (new dishwashers are very efficient).

Acquiring new skills in a quickly-changing world is as least as necessary today as it was back then; as illustrated by that subway ad from the mid-20th century.

Industry vs academia

Approaches to research and development have always differed between industry and academia. Having worked in both environments, here’s how I see them compare in robotics:

Academia: smaller “long-shot” bets, developed on a budget. Many projects take years to mature, due to the limited effort that a couple of post-docs, PhD students, and undergrads can dedicate.

Industry: execute on proven concepts with substantial backing. Industry prioritizes execution on feasible concepts. Once a concept is assumed to work – as usually verified by a proof-of-concept (POC), also known as a “minimum viable product” in startup lingo – industry players can raise vast amounts of money to build it.

Modern robotics may have reached an inflection point, where enough academic “long-shot” bets are delivering results, with feasible paths to building practical robots, including humanoid ones. Examples of bets made on robotic hardware:

Tesla’s Optimus humanoid prototype

Boston Dynamics bidding farewell to hydraulics and welcoming electric actuators on its all-new humanoid platform, Atlas

1X building a humanoid robot, NEO

Three humanoid robots: These are calculated bets with significant investment 3. Planning a robotics project

Here’s how I’ve seen robotics projects get done.

Vision for a demo: many robotics projects start with the question: “What should the robot achieve when completed?” Apple co-founder Steve Wozniak’s “coffee test” is one such vision, now frequently referred to as the “New Turing Test.” His definition:

“A machine is required to enter an average American home and figure out how to make coffee: find the coffee machine, find the coffee, add water, find a mug, and brew the coffee by pushing the proper buttons."

In practice, on most projects I’ve seen, the vision gets watered down or descoped as deadlines approach, and material or human resources become limiting factors. However a demo that passes the test above could well serve as a benchmark for artificial general intelligence (AGI). As part of that test, a physically-embodied machine must demonstrate intelligence by entering a random home and successfully making a cup of coffee with tools available in a previously unseen kitchen.

After the vision is set, the next steps are:

Planning: goals, demoable outcomes, and target applications or target environments. The requirements of a good demo in a target environment can be very high!

Break things down. Translating deliverables into functional requirements. For each functional requirement, one or more features that show successful fulfillment need to be identified. When breaking things down, features are often separated into at least two subsets:

Critical: those that must be present in the final product. If missing, the project is considered a failure. An example is a humanoid robot weighing 50kg that is deployed in warehouse automation, that needs to carry 10kg payload per arm. If the electric motors moving the arms are too weak, or the ankles suddenly overheat, when operating at 68kg in total, then a client won’t be happy with the product or service (RaaS, or Robot-as-a-Service is a thing now). Therefore, this feature likely requires robust engineering all the way to the very left side of the V-model. Identifying such interdependencies and making engineering design calls that aid the design, based on engineering intuition, is challenging and part of what makes for great technical leaders.

Nice-to-have: features that a final project could ship without, such as a higher control frequency in order to achieve smoother, less jittery motion. Note, these cannot be fully neglected but can be considered soft constraints, in contrast to critical features which tend to be hard constraints. A robot that gets the job done, even if not as smoothly, could still be counted as a success with room for improvement.

Critical vs nice-to-have thresholds can change over time. Hard constraints (“critical”) vs soft ones (“nice-to-have”) are often linked, and the difference lies in arbitrary measures of quantity and quality.

Being clear about the “minimum needed” (critical) and “nice-to-have” requirement thresholds is good practice in robotics projects. These can change as the project develops, turning nice-to-haves into critical ones. As long as everyone on the project is notified and on-board when changes happen, then a project should progress without hiccups.

4. Development phase

During development, robotics engineers need to balance proof-of-concept experiments with the scaling of a project, including the supporting infrastructure. For example, one proof-of-concept experiment could be verifying that a parallel jaw gripper (the robot’s hand) can grasp a mug in specific setups.

Scaling can be hard because there are lots of problems in the real world to solve. Scaling the above example of holding a mug can be seen as:

Read more

Die scheidende EU-Bürgerbeauftragte Emily O’Reilly hat es auf die Beschwerde des ehemaligen Europaabgeordneten der Piratenpartei Patrick Breyer als “Missstand” bewertet, dass ein Europol-Beamter trotz potenzieller Interessenskonflikte ohne jegliche Auflagen zum Chatkontrolle-Dienstleister Thorn wechseln und zuvor noch zwei Monate bei Europol in derselben Funktion weiter arbeiten durfte. “Europol hat es versäumt, den oben genannten Interessenkonflikt zu lösen, wodurch die Integrität und Unparteilichkeit seiner Maßnahmen gefährdet wurden”, heißt es in der Entscheidung der Bürgerbeauftragten. Der Beamte hatte bei Europol an einem KI-Pilotprojekt zur CSAM-Erkennung gearbeitet und war nach seinem Wechsel beim Bundestag als Lobbyist für Thorn registriert und fuhr in seiner neuen Funktion zu einem Europol-Treffen mit seinen ehemaligen Kollegen für eine Produktpräsentation. Europol verspricht ab 2025 nun zusätzliche Maßnahmen zur Vorbeugung von Interessenskonflikten wie Versetzungen, veränderte Aufgaben und den Einzug von Informationszugang für wechselwillige Mitarbeiter.

Breyer begrüßt das Ergebnis: „Wenn ein ehemaliger Europol-Bediensteter sein internes Wissen und seine Kontakte verkauft, um ihm persönlich bekannte Mitarbeiter der EU-Kommission zu lobbyieren, ist dies genau das, was es zu verhindern gilt. Seit der Enthüllung des ‘Chatcontrol-Gate‘ wissen wir, dass der EU-Vorschlag zu Chatkontrolle letztlich ein Produkt der Lobby eines internationalen überwachungsbehördlich-industriellen Komplexes ist. Damit so etwas nie wieder passiert, muss der Überwachungslobbysumpf trockengelegt werden.“

Hinweis: Die Entscheidung der EU-Bürgerbeauftragten ist hier abrufbar. Der hochkontroverse EU-Vorschlag zur Chatkontrolle steckt fest, weil es weder für die Aufhebung des digitalen Briefgeheimnisses und die Zerstörung sicherer Verschlüsselung eine qualifizierte Mehrheit unter den EU-Staaten gibt noch eine Mehrheit für die Streichung der Chatkontrolle aus dem Vorschlag.

In October 2023, my wife emerged from the bathroom one morning to report that we had no hot water. I found that our gas-fired, on-demand water heater had failed and was leaking a small amount. I called our plumber, who confirmed we needed a new water heater.

We immediately faced the following decision: Should we replace our water heater with a similar gas-fired on-demand unit or change to an electric heat pump water heater? We have been hearing a great deal about the efficiency and ecological benefits of heat pump water heaters, and on-demand gas units like the one we had are said to emit considerably more carbon than even gas-fired tank units.

Placement for a heat pump water heater would be a challenge. The water heater had been located in the attic, and the heat pump water heater was somewhat taller than even the gas-fired tank unit that was installed at the time we bought the house. That meant it could not fit in the attic and would need to be installed in the garage. We found a reasonable location, although we would lose storage space in the garage. It also meant that the loop for our hot water recirculating system needed to be extended down to the garage, making the loop longer.

The plumber said that he could replace the on-demand gas unit with a similar one in a couple of days, but it would take a week or so to obtain a heat pump unit, which would also cost considerably more. We had to make a snap decision because we didn’t want to be without hot water any longer than necessary. Wanting to be good ecological citizens, we opted for the heat pump unit despite the cost, complexity, and delay in installation.

As promised, the plumber obtained the heat pump water heater, an American Standard 50-gallon unit (model ASHPWH-50). Installation took about two days and involved running a considerable amount of pipe from the old location in the attic to the new location in the garage, installing a new 30-ampere 220-volt electrical circuit at the heater, and bracing against earthquakes. The installation went well, except a defective water leak sensor had to be bypassed.

First Impressions

The first thing I noticed about the new water heater was its significant noise. The specification sheet listed it as emitting 40.1 dBA, which is about the same as a refrigerator’s (and also an implausibly specific value). However, the water heater was much louder than this.

The on-demand water heater spoiled us: we never ran out of hot water. We soon found that not to be the case with the new water heater, and I turned the water volume for my showers much lower than before. Perhaps scarcity is a good thing.

The new water heater is a “hybrid” unit with a heat pump and also a conventional heating element. It has several modes:

Green (heat pump only) Comfort (“rational use of the heat pump and, if necessary, the heating element”) Fast (heat pump + heating element) i-Memory (“learns your habits and replicates them autonomously”) Electric (heating element only)

There’s also a Boost mode, which is like Fast, but reverts to the previous mode when it gets up to temperature. Initially, it was set for Comfort, but I couldn’t understand when it was and wasn’t running the heating element. I asked product support and the installer, but I still couldn’t make sense of Comfort mode, so we changed to Green. I manually turn on the Boost feature when we need extra hot water.

The noise issue

The noise generated by the water heater was bothersome, especially since it was noticeable in my office adjacent to the garage. I downloaded a sound level measurement app on my phone. I sent a video to American Standard (actually Ariston) product support. They told me I had not measured it under the right conditions. Of course, my garage is not an anechoic chamber.

I consulted “the internet” and found conflicting answers about the noise generated by heat pump water heaters. Some comments described them as very noisy, and others found them quite acceptable. I tried to find others locally with similar water heaters for comparison and failed. In late 2024, the behavior of the water heater changed: sometimes it had a loud “whirr,” and sometimes it didn’t. This prompted me to make several recordings of the sound (and analyze it with a spectrum analyzer app) and determine that much of the excess noise was coming from the fan in the heat pump. I again contacted product support, and they agreed to ship me a replacement fan. My plumber replaced it, and the noise is much better now.

The app

Like many new devices, the water heater connects to the internet through my WiFi and, through a cloud service, allows an app on my phone to control and monitor it. The app, called Comfort Link, is acceptable but less than excellent.

Good:

Allows me to remotely control the water heater, including turning on Boost mode when needed and turning off the water heater when we’re on vacation Displays the water heater temperature so I can judge whether to take a shower Shows the history of power usage by the water heater by month, day, and hour

Not so good:

The water heater has an icon that appears when it judges when it’s OK to shower. This isn’t on the app, but it would be very useful. The water heater temperature shown on the app doesn’t agree with that shown on the water heater. The support people explained that the water heater has several temperature sensors, and the app displays (I think) the average between the tank top and tank bottom temperature. But that doesn’t explain why the front panel displays something different. Limited history of power usage data (e.g., daily data for only the current and previous month) Poor resolution of power usage data. Measurements seem only to be taken only once per hour, so the heating element usage is reported in 4.5 kWh increments and the heat pump in 0.5 kWh increments. There seems to be no API to download data other than through the app. Efficiency

The heat pump is indeed more efficient than the electric heating element. This is suggested by the usage graph, where the orange bars represent energy usage from the heating element:

But of course, the days we used the heating element are also days where I turned on Boost mode because we were using more hot water, which exaggerates the difference. I measured the tank temperature over time as it heated up:

Heating element (4.5 kW): temperature rise 68.9°F/hr = 15.3°F/kWh Heat pump (0.45 kW): temperature rise 17.3°F/hr = 38.7°F/kWh (77°F ambient) Heat pump (0.45 kW): temperature rise 11.9°F/hr = 26.5°F/kWh (61°F ambient) Heat pump (0.45 kW): temperature rise 11.4°F/hr = 25.3°F/kWh (57°F ambient)

As expected, the heat pump’s efficiency depends on the ambient temperature, which determines the amount of heat the heat pump can “harvest” from the air. Compared with the heating element, the heat pump’s overall efficiency ranges from a factor of 1.65 to 2.53.

Operating cost

The heat pump water heater is efficient, but is it more cost-effective than our previous gas water heater? As noted above, it isn’t possible to make an exact comparison because we don’t know how our usage may have changed. If we make the worst-case assumption that our usage hasn’t changed, we are now using an average of about 5.3 kWh per day (153 kWh/month divided by 29 days). At an off-peak rate of $0.44/kWh, that works out to $2.33/day. Counting only those days where we used only the heat pump, that’s about 4 kWh/day or $1.76/day. Before getting the new water heater, our gas usage was about 8 therms/month, virtually all of which was for hot water. At our current rate of about $2.50/therm, that works out to $20/month or $0.67/day.

As our plumber had hinted when we installed the new water heater, gas is relatively inexpensive compared to electricity at current prices (unless you’re generating your own electricity). While an electric heat pump water heater is efficient and ecologically responsible, don’t expect it to save you money at current rates.

Writing that he knows “a bunch of people who held their nose and voted for Trump,” the president of HousingWire media issued this challenge on…

The post How are political & economic uncertainty impacting real estate consumers? first appeared on Real Estate Cafe.

Client assertions is a method of client authentication which can be used in OpenID Connect. This provides an alternative to client secrets. This approach enhances security by using signed tokens (JWTs) to authenticate clients during the token request process or the OAuth PAR request. In ASP.NET Core, client assertions is not supported per default, a small implementation is required.

Code: https://github.com/damienbod/oidc-client-assertion

Setup

A client assertion can be added to the token request which is sent from the ASP.NET Core backend to the OpenID Connect server. This is sent instead of the a client secret. The client is authenticated using the assertion. If using OAuth, the assertion is sent in the PAR request.

Create client assertion JWT

The application uses Microsoft Entra ID as the OpenID Connect server and the identity provider. The assertion was created using the Microsoft documentation.

Original src: https://learn.microsoft.com/en-us/entra/msal/dotnet/acquiring-tokens/web-apps-apis/confidential-client-assertions#crafting-the-assertion

A X509Certificate2 is used to create the certificate and validate the certificate. The audience, the clientId and the sub claims are sent in the JWT client assertion as well as the standard claims signed with the private key of the certificate used to validate the assertion in Microsoft Entra ID. As always, if using this against a different OpenID Connect server, JWT will be validated differently. Microsoft Entra ID requires a RSA key size 2048.

public static class CertService { public static string GetSignedClientAssertion( X509Certificate2 certificate, string aud, string clientId) { // no need to add exp, nbf as JsonWebTokenHandler will add them by default. var claims = new Dictionary<string, object>() { { "aud", aud }, { "iss", clientId }, { "jti", Guid.NewGuid().ToString() }, { "sub", clientId } }; var securityTokenDescriptor = new SecurityTokenDescriptor { Claims = claims, SigningCredentials = new X509SigningCredentials(certificate) }; var handler = new JsonWebTokenHandler(); var signedClientAssertion = handler.CreateToken(securityTokenDescriptor); return signedClientAssertion; } }

Using Micrsoft Entra ID as the OpenID Connect server, the client assertion is created using the token endpoint. In production, the certificate can be generated using Azure Key Vault. The certificate can also be read from the operating system store. This can all be set in the app.settings.

// single tenant var aud = $"https://login.microsoftonline.com/{builder.Configuration["AzureAd:TenantId"]!}/oauth2/v2.0/token"; var clientAssertion = CertService.GetSignedClientAssertion( X509CertificateLoader.LoadPkcs12FromFile("cert_rsa512.pfx", "1234"), aud, builder.Configuration["AzureAd:ClientId"]!);

Use the client assertion in the OpenID connect client

Once ready, the OnAuthorizationCodeReceived event can be used to added the assertion in the OpenID Connect client. If using PAR, the par event is used.

builder.Services.AddAuthentication(OpenIdConnectDefaults.AuthenticationScheme) .AddCookie(CookieAuthenticationDefaults.AuthenticationScheme) .AddOpenIdConnect(OpenIdConnectDefaults.AuthenticationScheme, oidcOptions => { oidcOptions.SignInScheme = CookieAuthenticationDefaults.AuthenticationScheme; oidcOptions.Scope.Add(OpenIdConnectScope.OpenIdProfile); oidcOptions.Scope.Add("user.read"); oidcOptions.Scope.Add(OpenIdConnectScope.OfflineAccess); oidcOptions.Authority = $"https://login.microsoftonline.com/{builder.Configuration["AzureAd:TenantId"]}/v2.0/"; oidcOptions.ClientId = builder.Configuration["AzureAd:ClientId"]; //oidcOptions.ClientSecret = builder.Configuration["AzureAd:ClientSecret"]; oidcOptions.ResponseType = OpenIdConnectResponseType.Code; oidcOptions.MapInboundClaims = false; oidcOptions.SaveTokens = true; oidcOptions.TokenValidationParameters.NameClaimType = JwtRegisteredClaimNames.Name; oidcOptions.TokenValidationParameters.RoleClaimType = "role"; oidcOptions.Events = new OpenIdConnectEvents { // Add client_assertion OnAuthorizationCodeReceived = context => { context.TokenEndpointRequest!.ClientAssertion = clientAssertion; context.TokenEndpointRequest.ClientAssertionType = "urn:ietf:params:oauth:client-assertion-type:jwt-bearer"; return Task.FromResult(0); } }; });

Setup Entra ID App registration

An Azure App registration is used to authenticate using Micrsoft Entra ID. The public key of the certificate can be uploaded to to the App registration.

Client assertions can be used for any OpenID Connect confidential client if the OpenID Connect server supports this. The assertion can be sent in the token request or in the PAR request depending on the flavor of the OpenID Connect code flow used.

Creating the certificate

The github repository linked at the top, provides an example to create your own certificate which can be used in this flow. You can also use Azure Key vault or any other tool to create the certificate.

Links

https://learn.microsoft.com/en-us/entra/msal/dotnet/acquiring-tokens/web-apps-apis/confidential-client-assertions

https://github.com/AzureAD/microsoft-identity-web/blob/2b8fbf0104d820bba8785c41b2ef9e6f801b5e73/src/Microsoft.Identity.Web.TokenAcquisition/MsAuth10AtPop.cs#L48

https://datatracker.ietf.org/doc/html/rfc7521

https://oauth.net/private-key-jwt/

https://learn.microsoft.com/en-us/aspnet/core/security/authentication/configure-oidc-web-authentication

RFC 7523 – JSON Web Token (JWT) Profile for OAuth 2.0 Client Authentication and Authorization Grants

こんにちは、富士榮です。

3月頭はFintech Weekということもあり、あちこちでFintech系のイベントが開催されますね。そのうちの一つである4F（Future Frontier Fes by FINOLAB）の一コマをいただきAuthlete川崎さんと一緒にFAPIとVerifiable Credentialsの話をします。

こちらのイベントですね。

https://4f-otmcbldg.tokyo/2025-jp/

このうち、3/4の午前中のセッションです。

セッションの詳細と申し込みはこちらからしていただけます。

https://fapi-vc.peatix.com/

 私は慶應の鈴木先生と一緒に先日発行したデジタルクレデンシャルの管理要件に関するディスカッションペーパーの中身の話を解説させていただきます。みなさん色々とデジタルクレデンシャルを発行しますが、ちゃんと用途に応じた管理をしないとダメですよ、って話です。

ぜひお越しください！

Posting this late, so I won’t be writing an introduction. As is the case for me lately, the offering is a somewhat rambling discussion that surfaces quiet a few nuggets for you to chew on.  Life in this absurdist improvisational theater might easier if we can practice the Tao of Pooh. Smile.

Here’s a link to the playlist of DOGE clips. An overview of them is presented in the last half hour of the video.

Be sure to check out the story of the Hot Springs, AR IQ Zoo run by BF Skinner behaviorist animal trainers. Hat Tip to Emily of “Save Maine Schools” who first alerted me to BF Skinner’s Pigeon Project. I promise you’ll never think of Cheerios the same way!

Click here for historical background and here for interesting projects (Spy Birds!).

Also, here is a link to Joseph’s blog on “free energy,” which I equate more with manifestation and the ritual opening of “gates” into the quantum field.

 

 

 

Stream the Latest Episode

Available now on YouTube, Apple and Spotify. See the episode transcript at the top of this page, and a summary at the bottom.

Brought to You By

• DX — An engineering intelligence platform designed by leading researchers.

• Sentry — Error and performance monitoring for developers.

—

In This Episode

We’ve previously covered a lot on the surprisingly slippery topic of developer productivity: from how Uber measures it, how LinkedIn does it, an overview of the innovative DevEx framework, and a deepdive of dev productivity metrics used by Google, LinkedIn, Peloton, Amplitude, Intercom, Notion, Postman, and 10 other tech companies.

Every time developer productivity comes up: DORA and SPACE are almost certainly mentioned. So I could not be more excited to have Dr. Nicole Forsgren on the podcast. Nicole is the creator of the widely adopted DORA and SPACE frameworks, co-author of the award-winning book Accelerate and the DevOps Handbook (2nd edition), and author of the State of DevOps reports. She is currently a Partner at Microsoft Research, leading developer productivity research and strategy, and is currently working on a book about developer experience with Abi Noda. It’s safe to say that Nicole is one of the foremost experts in developer productivity — if not the foremost expert.

In this episode, we discuss:

Why PRs and Diffs are incomplete as a solo metric and how to view them in context

The importance of a holistic set of metrics for evaluating productivity

An overview of DORA’s four key metrics, its strengths, and its limitations

The evolution of processes and tools since DORA, including SPACE

What developer experience is—and concrete ways to improve it

Common characteristics of highly productive engineering teams

How faster onboarding might challenge Brook’s Law

How AI tooling is impacting developer productivity and best practices for experimentation

And much more!

Takeaways

My biggest takeaways from this episode:

1. Measuring the number of PRs is controversial for a good reason. Measuring any single one “output” can be misused: and when devs know it is being measured, they will optimize for it.

However, measuring PRs is important: but to do it well, don’t look at it as an individual performance metric. Instead, use it to understand how well (or not well) systems across the team and company are working. For example, what systems are getting in the way of PRs taking long to merge?

2. DORA and SPACE both have their own limitations. DORA is very well-defined in the metrics it measures. However, a massive limitation it has is how it only measures from commit to production.

SPACE can be used to measure the complete developer workflow – including e.g. planning, coding, and even post-release. In turn, it is a more vague framework where you need to put in more effort to make it useful for your envirnment.

3. AI developer tools don’t change DevEx fundamentals. DORA and SPACE are still relevant. AI tools might be able to improve iteration speed, or make certain tasks more helpful. It’s really an open question how exactly this will play out. There’s a fair chance these tools significantly change how we do development. So if you’re a dev: experiment with them!

4. Developer experience (DevEx) is not necessarily great at large companies and poor at startups. Google is known to have standout developer experience thanks to so much investment it put into systems – but not all large companies are like this!

Startups have less resources to invest specifically in improving DevEx: but then again, they have less infrastructure in-place! The more custom infrastructure you have, the more painful DevEx tends to be (and the more in-house tools need to be built to mitigate this)

The Pragmatic Engineer deepdives relevant for this episode

Measuring Developer Productivity: Real-World Examples

A new way to measure developer productivity – from the creators of DORA and SPACE

Measuring Engineering Efficiency at LinkedIn

How Uber is Measuring Engineering Productivity

Measuring software engineering productivity

Timestamps

(00:00) Intro

(02:03) PRs and Diffs and how to view them in the right context

(07:42) EngThrive at Microsoft

(10:26) The importance of having a holistic set of metrics in evaluating productivity

(17:00) The four key metrics of DORA

(23:57) The evolution of processes and tools since DORA, including SPACE

(26:40) An explanation of developer experience — and ways to improve it

(30:44) Devex at startups vs. larger companies

(34:20) Why measuring developer productivity is so difficult

(39:05) How to make a case for platform teams

(44:34) Common characteristics of highly productive teams

(51:01) Brook’s law and how faster onboarding might make it irrelevant

(52:49) Onboarding for internal transfers

(54:18) Shifting culture towards technology first

(58:36) How middle management can improve engineering culture

(1:03:36) How AI tooling is impacting developer productivity

(1:06:42) Potential use cases for AI

(1:08:40) A case for experimenting with AI coding tools and how to maintain flow state

(1:15:30) Rapid fire round

A summary of the conversation Measuring Developer Productivity

PRs and diffs can be both good and bad signals when measuring developer productivity. Many leaders focus on the traditional economic definition of productivity: the rate of output or output per input.

PRs can give a view into the work being done, but senior engineers may have lower PR output due to other responsibilities like unblocking others, architecture, mentoring, and recruitment. It's important to consider the context and who is being measured.

A constellation of metrics is better than a single metric for a holistic view. So yes: it can be helpful to get data on PRs – as long as you get a lot more other data points as well!

Microsoft's EngThrive: this framework (which Nicole works on) uses multiple metrics across dimensions, inspired by the SPACE framework, alongside qualitative feedback.

Easy-to-operationalise measures can be useful. However, they only show a few things and may ignore important aspects.

SPACE: a framework that groups metrics into:

Satisfaction

Performance

Activity

Communication

Efficiency.

Nicole is a co-author of the framework. It’s one that is getting a lot wider adaptation. See an overview of the SPACE framework

Absent engineering leadership is unhelpful. No matter what metrics you use: if engineering leadership is not connected with the work, outcomes are likely to be a lot worse.

Metrics collected in a holistic way, on the other hand, can guide decisions about company-wide blockers.

DORA frameworkand its evolution

Nicole is the co-creator of DORA

DORA can refer to the entire research program. More frequently though, people tend to refer to the 4 DORA metrics:

Deployment frequency

Lead time for changes

Change failure rate,

Time to restore service

These 4 metrics are a good indicator of how well a development pipeline is working

DORA can be adapted for different environments, such as air-gapped systems, by redefining the scope of the deployment pipeline.

DORA does not show the full picture. DORA metrics serve as a signal for how well a team is doing, but it does not show the complete picture. DORA focuses on commit to production. This is a part of the engineering process that can be made pretty efficient.

Fast feedback is important, as is the ability to set up environments and provision resources quickly.

SPACE evolved from DORA. SPACE considers the entire end-to-end toolchain. SPACE metrics can be applied to specific components, like PRs. DORA is essentially one of many instances of SPACE.

Developer Experience

Developer experience is a developer's lived experience. This can be “good” – it can also be “bad!”

A good developer experience minimises friction, blockers, and confusion

Security and compliance: these are important! However, automation can minimise manual processes for developers. A secure and compliant dev process does not need to be cumbersome.

Delays and cognitive load from inefficient processes negatively affect developer experience.

Both large and small companies can have good or bad developer experiences. Large companies have more resources but may struggle with legacy systems and bureaucracy. Startups can move fast but lack infrastructure. Google has an exceptional developer experience because they invest in it and don't tolerate friction.

Improving Engineering Teams

Most work in software is invisible, and systems are increasingly complex.

Leaders and executives often don't feel the pain of developers.

To make a case for investment in tooling: use both data and stories. Present the trade-offs realistically.

Acknowledge that there's a tipping point where further investments won't deliver much more progress.

Frame requests in terms of trade-offs and potential for repurposing teams.

High-performing teams:

Exhibit psychological safety, curiosity, and openness to better ways of doing things.

Onboarding time is a great indicator of team efficiency.

A “dummy pull request” early during the onboarding phase (e.g. day 1!) can significantly increase productivity. Try it!

If onboarding is slow: adding people to a late project can backfire. But if onboarding is fast: this can actually work!

Engineering culture transformations:

These require changing the entire company culture, not just the tech culture.

Changing how people do their work can lead to cultural impacts.

Introducing faster tools and ways of working changes people's lived experience.

To improve the engineering culture, summarize observations, seek feedback, and involve others in finding improvements.

Aim for quick wins with visible impact, like hack days to address paper cuts.

Impact of AI on Developer Productivity

DORA and SPACE relevance:

DORA metrics should remain relevant

SPACE is still applicable for assessing satisfaction, performance, activity, communication, and efficiency.

Changes AI is likely to bring:

There’s some chance that AI tools einvent development and software engineering, particularly in IDEs and testing.

Coding assistants: AI-powered ones can improve code readability and unit test success rates

PR reviews: an obvious place for AI to help

Support roles like release engineering and deployment: could be a use case

Experiment with AI!

Experimenting with AI tools and adapt to your workflows.

Assess AI tools to see what works, and focus on problem-solving and architecture.

When you find the workflows where AI is helpful: you might be able to preserve flow better

Turn off autocomplete and autosuggestions if they are an interruption!.

Consider AI as a different way to work, such as "driving and reviewing" or "guiding".

LLMs are changing software engineering rapidly.

Book recommendations

Nicole recommends reading:

Inspired by Marty Cagan

Outlive by Peter Attia

Ender's Game for some good fiction reading

Resources & Mentions

Where to find Dr. Nicole Forsgren:

• X: https://x.com/nicolefv

• LinkedIn: https://www.linkedin.com/in/nicolefv/

• Website: https://nicolefv.com/

Mentions during the episode:

• Microspeak: Sats: https://devblogs.microsoft.com/oldnewhttps://newsletter.pragmaticengineer.com/p/developer-productivity-a-new-frameworkthing/20100914-00/?p=12873

• Measuring Software Engineering Productivity: https://newsletter.pragmaticengineer.com/p/engineering-productivity

• Hawthorne effect: https://en.wikipedia.org/wiki/Hawthorne_effect

• Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations: https://www.amazon.com/Accelerate-Software-Performing-Technology-Organizations/dp/1942788339

• What is an Air Gap?: https://www.ibm.com/think/topics/air-gap

• DORA: https://dora.dev/

• Quantifying the impact of developer experience: https://developer.microsoft.com/en-us/developer-experience

• A new way to measure developer productivity – from the creators of DORA and SPACE: https://newsletter.pragmaticengineer.com/p/developer-productivity-a-new-framework

• Ciera Jaspan on LinkedIn: https://www.linkedin.com/in/ciera/

• Emerson Murphy-Hill on LinkedIn: https://www.linkedin.com/in/captainemerson/

• Inside Stripe’s Engineering Culture - Part 1: https://newsletter.pragmaticengineer.com/p/stripe

• Inside Stripe’s Engineering Culture: Part 2: https://newsletter.pragmaticengineer.com/p/stripe-part-2

• David Singleton on LinkedIn: https://www.linkedin.com/in/davidpsingleton/

• Courtney Kissler on LinkedIn: https://www.linkedin.com/in/courtney-kissler/

• Brian Houck on LinkedIn: https://www.linkedin.com/in/brianhouck/

• Brook’s law: https://en.wikipedia.org/wiki/Brooks%27s_law

• Satya Nadella on LinkedIn: https://www.linkedin.com/in/satyanadella/

• Steve Ballmer on LinkedIn: https://www.linkedin.com/in/steve-ballmer-7087a8157/

• John Shook: https://www.lean.org/about-lei/senior-advisors-staff/john-shook/

• Does GitHub Copilot improve code quality? Here’s what the data says: https://github.blog/news-insights/research/does-github-copilot-improve-code-quality-heres-what-the-data-says/

• Two developers built a game that sold 1M copies. How?

• Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming: https://www.microsoft.com/en-us/research/publication/reading-between-the-lines-modeling-user-behavior-and-costs-in-ai-assisted-programming/

• Abi Noda on LinkedIn: https://www.linkedin.com/in/abinoda/

• Jason Entenmann on LinkedIn: https://www.linkedin.com/in/jason-entenmann-06146875/

• Claude: https://claude.ai/new

• Anthropic: https://www.anthropic.com/

• OpenAI: https://openai.com/

• Sonnet: https://www.anthropic.com/news/claude-3-5-sonnet

• Inspired: How to Create Tech Products Customers Love: https://www.amazon.com/INSPIRED-Create-Tech-Products-Customers/dp/1119387507

• Outlive: The Science and Art of Longevity: https://www.amazon.com/Outlive-Longevity-Peter-Attia-MD/dp/0593236599/r

• Ender’s Game: https://www.amazon.com/Enders-Game-Ender-Quintet-1/dp/1250773024

• Tressie McMillan Cotton’s website: https://tressiemc.com/

• Anne Helen Petersen’s newsletter: https://substack.com/@annehelen

• Can You Really Measure Individual Developer Productivity? - Ask the EM: https://blog.pragmaticengineer.com/can-you-measure-developer-productivity/

• Measuring Developer Productivity: Real-World Examples: https://newsletter.pragmaticengineer.com/p/measuring-developer-productivity-bae

• Measuring developer productivity? A response to McKinsey: https://newsletter.pragmaticengineer.com/p/measuring-developer-productivity

• Measuring developer productivity? A response to McKinsey, Part 2: https://newsletter.pragmaticengineer.com/p/measuring-developer-productivity-part-2

• The Full Circle on Developer Productivity with Steve Yegge: https://newsletter.pragmaticengineer.com/p/steve-yegge

• Measuring Software Engineering Productivity: https://newsletter.pragmaticengineer.com/p/engineering-productivity

• Platform Teams and Developer Productivity with Adam Rogal, Dir. Developer Platform at DoorDash: https://newsletter.pragmaticengineer.com/p/platform-teams-with-adam-rogal

—

Production and marketing by Pen Name. For inquiries about sponsoring the podcast, email podcast@pragmaticengineer.com.

It’s the end of an era: after eight incredible years, it felt like the right time to close the chapter on my story at Uniken. When I joined as CTO, Uniken was an ambitious startup with a unique core technology and a vision to make a global impact. What made the ensuing journey so rewarding was the team – exceptionally talented, deeply committed, and relentlessly innovative. Together, we navigated complex challenges, scaled through uncertainty (including a global pandemic), and built Uniken into a global cybersecurity leader with customer engagements on every continent (except Antarctica, because sadly it would appear that penguins aren’t really concerned about digital trust). 

I am incredibly proud of the team, and the platform we built – one that empowers businesses to engage securely and seamlessly with their customers, and creates lasting value for our customers and partners.  I want to express my deepest gratitude to my colleagues and partners at Uniken. Our collaborative efforts transformed the company and positioned it for long-term success, and I look forward to seeing you scale new heights.

As for what’s next – that’s TBD. Digital identity has never been more critical, yet the challenges facing the industry have never been more complex. Let’s see what the next chapter brings.

Last week, we looked into data from global hiring agency Indeed that found software development job listings are at a five-year low. To check how representative (or not) these numbers are of the tech jobs market, I pinged some tech recruiters. The response was not what I expected:

“I can only talk about our own niche from seed-stage to Series C engineering recruitment, but we’ve never had so many positions to fill as in 2024.” – Asher Hoffman, cofounder of Coastal Recruiting

“In 2024, every month was a step up in the number of job searches we did for CTO, VP of engineering, director of engineering and product leadership positions. It just kept going up. 2024 was our second-best year ever for the number of searches done: only 2021 was better, and that was an incredible year; the peak of the job market.” – Glenn Murphy, managing partner at Riviera Partners

These data points stick out and I was eager to learn more; after all, they’re bucking the trend in Indeed’s figures, with rising demand for tech professionals! I met Asher and Glenn to get their perspectives, and they also offered some tactics for engineers and engineering leaders to use when seeking new opportunities. Today, we cover:

What are boutique recruiters? Also, the differences between contingent and retained ones.

Hiring by tech stack at early-stage startups. Fifty percent of hires are backend, 25% fullstack, 10-15% AI/ML, and 5% are frontend. A big surprise is virtually no demand for native mobile developers, or for the .NET and Java tech stacks. Also, very few mentions of Angular or Vue.

More demand than before at Coastal and Riviera. Both recruitment businesses are seeing more engineering jobs to be filled coming their way.

Changes across the market. More money and demand from early-stage VC-funded startups, and private equity becoming more dominant in driving demand for engineering executive hiring.

Startups want ‘cracked’ engineers. These are professionals with a track record of ownership, curiosity, and the ability to “grind.” Length of experience doesn't matter after around the 2-year point.

Demand for engineering leadership positions. Data indicates that large parts of the job market are struggling more than the senior role segment, with director of engineering roles hardest to fill.

What search processes look like. Recruitment experts explain how a typical software engineer search and engineering executive search plays out.

For updates on how these two recruiters see the market, follow Asher Hoffman and Glenn Murphy on LinkedIn.

A quick programming note: there will be no The Pulse on Thursday — I’m taking a short winter break. More details in my publishing schedule.

1. What are boutique recruiters?

Coastal Recruiting and Riviera Partners are specialized recruitment agencies; also known as "boutique search”, “contingent”, “retained”, or “executive” recruiters.

“Boutique” and “specialized” refer to recruiters which focus on a niche within tech, such as:

Software engineer and AI+ML engineer hiring for early-stage startups (Coastal Recruitment)

Director-and-above engineering leadership, and product/design leadership for scaleups and larger companies (Riviera Partners)

Executive recruitment is a subset of boutique recruitment. These companies recruit for executive roles like director of engineering, VP of engineering, head of engineering, CTO roles, etc. In the US, high-profile firms can charge $100K or more to recruit a key position like CTO for a late-stage scaleup.

Executive recruitment firms can also focus on individual contributor (IC) roles. Glenn Murphy advises that this usually happens only in the US at their firm, and comes with tradeoffs. For example, the cost becomes justifiable only when the vacancy’s comp package is several times higher than the search cost; usually $500K and above. It is this niche area in which an expert executive recruiter can make the most impact by finding truly standout candidates.

Within the boutique search sector, there are two main types of recruiter:

Contingent recruiters

These companies work on a “no hire, no fee” basis, and only get paid for successfully filling a position; usually 10-40% of the role’s first-year salary. This service is an easy sell; it’s low risk financially for an employer to work with a contingent recruiter due to no-hire-no-fee. Contingent recruiters also often have more clients than retained ones do.

Coastal Recruiting mostly does contingent recruitment, with a couple of retained contracts. This company specializes in hiring software engineers from the first hire to engineer number 20 to 30 at early-stage startups. In 2024 they worked with around 100 different startups.

Retained recruiters

Most executive recruitment companies operate a retainer model and charge upfront, regardless of whether a hire is made. Payment comes in a fixed number of instalments, perhaps in this order:

Initial retainer: the first third of the fee, paid at the start.

Progress retainer: second payment, usually tied to milestones being hit, such as:

Presenting a shortlist of qualified candidates who are ready to engage

Completing screening interviews and drawing up a interview shortlist of preferred candidates

Final interviews scheduled or completed

Final payment: upon successful placement. Depending on the agreement, there may be no payment if a candidate isn’t hired

A search company may also be paid monthly while conducting a search. Total retainer fees are usually 25-50% of the value of the successful candidate’s comp, which increases the more challenging the search is. For example, it’s challenging to find a new CTO for a 300-person scaleup growing at a 10x pace, who is hands-on, has worked with AI / machine learning (ML) before, and who has experience at a hypergrowth company, and also in Big Tech. There aren’t too many people who fit this profile, and not many of them may be interested in switching jobs.

Guarantee periods can also be an important part of retained searches, under which a portion of the fee may be refunded should a successful candidate quits shortly afterwards.

Retained recruiters are a lot more expensive than contingent ones, and are the go-to choice for executive recruitment where vacancies may not even be publicly listed by an employer.

Riviera Partners is one of the best-known executive recruiter companies. They hire for director+ roles in engineering, product and design. In our deepdive Finding the next company to work at, they were the most recommended exec recruiter firm by engineering leaders.

Seed-stage and early-stage startups typically only use executive recruiters to hire for VP-and-above positions which report to the CEO. Later-stage startups also hire for director positions, and may pay for a search for senior-and-above hires and ICs. However, the cost means this is rare, happens mainly in the US, and by funded scaleups with very specific needs.

Given this process can be so expensive, it may not be immediately obvious what the upside is. Glenn explains:

“We have 21 years worth of data and information on who has been successful, people’s interests and nuance that isn’t available on a LinkedIn profile.

We save companies a lot of time, in the end. This is due to the insights we have and our reach across the USA and Europe. We know what’s needed at Series A, all the way through to which profiles tend to succeed at publicly traded companies. We also have the know-how on how to interview people to assess if they’re up for the challenge.”

Riviera’s pricing is atypical, compared to many other retained recruiters. Glenn shares:

“We have time-based milestones as opposed to shortlists or incentives, which can be adjusted sometimes. However, for the most part, the fee is fully paid within 90 days.”

2. Hiring by tech stack at early-stage startups

At Coastal Recruiting, Asher hires early engineering teams at early-stage startups: from seed-stage startups to Series B and Series C companies. They work with some of the largest US VC investors (like a16z, Y Combinator, Sequoia, Index, Lightspeed, Founders Fund and others).

Types of engineers and tech stack

Asher sees the hiring demand split like this:

‘50% of hires we make are backend engineers. I don’t think this will always be the case for startup engineering. The first 3 hires for a company with technical cofounders are almost always backend engineers. It’s a bonus if they have some frontend skills, but I’d peg it at 80% backend, 20% frontend.

‘For backend tech stacks, these are the languages founders use, which candidates would be expected to pick up quickly. By popularity:

Typescript: easily the most common.

Python: surprisingly popular, usually with another language like Typescript or Go. I assume it’s because so much AI and ML work can be done with it, and it’s a generally useful language

Go: This has gotten somewhat more popular, over time

Rust: especially popular with crypto companies. About 15% of our portfolio is crypto, and Rust is very popular. We’re on the lookout for people with hands-on experience in Rust

Ruby: Out of around 100 clients in the past year, only 2 used Ruby; both of them are pretty big names. They told us “find us anyone who has touched Ruby or Ruby on Rails, and is good.”

Few to no mentions:

Flutter: we’ve gotten a single Flutter role in the past year

Elixir: once in a while, we have a startup that uses this and needs folks with experience with it. But it’s less common than Ruby.

Java / Scala: we don’t get any requests for this stack, at all. However, if someone has Java or Scala experience, and is open to working in Python, we usually move forward with them

C# / .NET: we’ve only come across a single hardware company asking for this in several years.

PHP: almost zero PHP these days

Haskell: it’s been a couple years since any company asked for this.

‘25% percent are fullstack. These roles almost exclusively employ the following technologies:

React for the frontend. React frameworks like Next or React Router are usually relevant. Overall, 95% of framework mentions are for React.

Typescript as the programming language. Of course, JavaScript is a given

Node as the backend framework

Python is often mentioned

React Native is by far the most popular mobile technology we see, often requested by mobile-first startups that build their app with it.

Angular / Vue: a combined 5% or less.

‘AI and ML is 10-15%,” says Asher. “This hire is almost always one of the first 10, and doesn’t need to be an “in-depth” ML specialist. They usually fit this profile:

Backend generalist

Has worked in an ML or AI-driven company

Is deep into what is going on in AI and how to use LLMs, practically

Hacked AI side-projects themselves. They probably hack on the side, and might use things like LangChain, LangGraph, and other popular LLM tools to play around with.

‘Someone we recently hired for such a role had worked for a year as an AI/ML engineer, not an in-depth, “research-caliber” ML candidate.

‘As startups grow and mature, eventually they tend to hire more in-depth ML and AI people. However, early-stage companies look for builders who can use AI tools effectively.

‘5% are frontend engineers. Almost universally, the seventh or eighth hire at a startup is the first frontend engineer. The reason almost always is – and I kid you not! – that the founder is fed up with owning the frontend themself, and is ready to bring in a frontend specialist.

‘Ironically, this role is one of the hardest to fill because technical founders whom we work with typically expect two things:

Be a great frontend person – this is easier to find

Be great at computer science basics – such as algorithms and CS fundamentals. This is why most people who pass #1 get rejected!

‘Another reason this role is so hard to fill is that the founder often looks for a staff-level frontend engineer. This person might be the first frontend hire, but they ideally want someone who will head up the frontend team, as it grows. It’s difficult to convince people with staff-plus frontend jobs at stable companies and who head up teams, to come and join an early-stage startup where they would be the only frontend person – at least at first!

‘The best startups are embracing AI – and look for devs who do the same. AI coding tools like Cursor and GitHub Copilot are used by all the best startups we recruit for. Founders at these startups often mention that mastering these tools can 10x the productivity of their engineers with faster iteration and the ability to parallelize work.

‘We don’t see demand for native mobile engineers at all; only React Native. We work with a few “mobile-first” startups who have built impressive mobile apps. All use React Native and look to hire for such experience.

‘I would assume native hires might become more important after the Series B or Series C rounds. In the early-stage in the US, they are absent – or at least, we see no such demand.’

Startups don’t care that much if you’ve used their exact stack before. Asher told me:

“I hear over and over again, that a great engineer is a great engineer - and if you don't have experience in the companies current stack - as a talented engineer, you shouldn't have much trouble picking it up quickly. Thus - if your stack doesn't align perfectly with the above most used, it's okay - you can still get the role!”

3. More demand than before at Coastal and Riviera

Both recruitment companies are seeing record, or near-record business in terms of the number of searches executed.

More demand from early-stage, VC-funded startups. Asher says:

“In my niche, I’m seeing more hiring than in recent memory.

I'm seeing many “cracked” engineers with 5+ offers on-hand, and well-funded startups forced to compete by upping their offers to hire the best of the best. I’m seeing higher total comp packages in the early stage space than ever.”

As we’ll detail below, “cracked” refers to someone with a track record of ownership, who displays curiosity and the ability to grind when needed.

Asher: “It feels like this is a great time for many folks to leave Big Tech and take a gamble – but not too big a pay cut. Big Tech has done many layoffs and the startups I work with offer similar pay, with a massive equity upside.

The risks an engineer used to have to take when joining a startup are shrinking in this new market. Well-funded startups are meeting engineers with competitive salaries, health and wellness, and equity. Also, founders are being much more upfront with recruits about their ARR (annual recurring revenue) and progress towards PMF (product-market-fit) than before.

A disclaimer: I recruit in just the VC-funded, early-stage start-up space, with startups founded by technical founders and backed by well-known VC firms. These are companies that have the money to hire, and a bold vision to build a billion-dollar company. It’s a narrow view of the market.”

Will more companies increase recruitment? At Riviera partners, Glenn Murphy does engineering leadership searches for startups and scaleups. He shares:

“We signed up 58% more searches in 2024 than in 2023 for the US. In Europe, the growth was 61%. The most common search assignment in both continents was for VP Engineering. It feels to me this alone signals a return to a focus on growing teams, as VPs of engineering are hired in preparation to scale up the team’s headcount.”

4. Changes across the market

I asked both Asher and Glenn what changes have been surprising in their part of the tech hiring market.

Read more