Last Update 6:49 AM October 10, 2024 (UTC)

Identity Blog Catcher

Brought to you by Identity Woman and Infominer.
Support this collaboration on Patreon!!!

Thursday, 10. October 2024

Simon Willison

Announcing Deno 2

Announcing Deno 2 The big focus of Deno 2 is compatibility with the existing Node.js and npm ecosystem: Deno 2 takes all of the features developers love about Deno 1.x — zero-config, all-in-one toolchain for JavaScript and TypeScript development, web standard API support, secure by default — and makes it fully backwards compatible with Node and npm (in ESM). Another new feature that caugh

Announcing Deno 2

The big focus of Deno 2 is compatibility with the existing Node.js and npm ecosystem:

Deno 2 takes all of the features developers love about Deno 1.x — zero-config, all-in-one toolchain for JavaScript and TypeScript development, web standard API support, secure by default — and makes it fully backwards compatible with Node and npm (in ESM).

Another new feature that caught my eye was this:

deno jupyter now supports outputting images, graphs, and HTML

Deno has apparently shipped with a Jupyter notebook kernel for a while, and it's had a major upgrade in this release.

Here's Ryan Dahl's demo of the new notebook support in his Deno 2 release video.

I tried this out myself, and it's really neat. First you need to install the kernel:

deno juptyer --install

I was curious to find out what this actually did, so I dug around in the code and then further in the Rust runtimed dependency. It turns out installing Jupyter kernels, at least on macOS, involves creating a directory in ~/Library/Jupyter/kernels/deno and writing a kernel.json file containing the following:

{ "argv": [ "/opt/homebrew/bin/deno", "jupyter", "--kernel", "--conn", "{connection_file}" ], "display_name": "Deno", "language": "typescript" }

That file is picked up by any Jupyter servers running on your machine, and tells them to run deno jupyter --kernel ... to start a kernel.

I started Jupyter like this:

jupyter-notebook /tmp

Then started a new notebook, selected the Deno kernel and it worked as advertised:

import * as Plot from "npm:@observablehq/plot"; import { document, penguins } from "jsr:@ry/jupyter-helper"; let p = await penguins(); Plot.plot({ marks: [ Plot.dot(p.toRecords(), { x: "culmen_depth_mm", y: "culmen_length_mm", fill: "species", }), ], document, });

Tags: deno, javascript, jupyter, observable, typescript, nodejs, npm


Ben Werdmüller

Waffle House Index labels Hurricane Milton red, closes stores

[Ben Kesslen at Quartz] "Waffle House, the iconic American restaurant chain with over 1,600 locations known for cooking up Southern breakfast food, has developed an advanced storm center FEMA consults with." Stores in the path of Milton were closed in advance of the storm, which is rare for Waffle House, which is often the last store standing. It's been sophisticated about

[Ben Kesslen at Quartz]

"Waffle House, the iconic American restaurant chain with over 1,600 locations known for cooking up Southern breakfast food, has developed an advanced storm center FEMA consults with."

Stores in the path of Milton were closed in advance of the storm, which is rare for Waffle House, which is often the last store standing.

It's been sophisticated about storm predictions and response since Katrina:

"The chain also developed the Waffle House Storm Index, which was started after former FEMA Administrator Craig Fugate said, “If you get there and the Waffle House is closed? That’s really bad. That’s where you go to work.”"

As Pat Warner, a member of "the Waffle House crisis management team" said in the article, it's not about the extra sales Waffle House gets when it does re-open, often using generators and other emergency equipment. It's more to do with how this integrates the stores with their communities. They wouldn't do it if there wasn't a positive uplift for the business, but it comes across as a genuine desire to help.

#Society

[Link]


Forums Are Still Alive, Active, And A Treasure Trove Of Information

[Chris Person at Aftermath] "Over the years, forums did not really get smaller, so much as the rest of the internet just got bigger. Reddit, Discord and Facebook groups have filled a lot of that space, but there is just certain information that requires the dedication of adults who have specifically signed up to be in one kind of community. This blog is a salute to those foru

[Chris Person at Aftermath]

"Over the years, forums did not really get smaller, so much as the rest of the internet just got bigger. Reddit, Discord and Facebook groups have filled a lot of that space, but there is just certain information that requires the dedication of adults who have specifically signed up to be in one kind of community. This blog is a salute to those forums that are either worth participating in or at least looking at in bewilderment."

What an amazing index of indie forums still going strong on the web.

I'd love to do a survey of what they're powered by, and in turn, I'd love to read interviews of the product / engineering leads for each of these platforms. Are they individual developers, keeping the lights on out of love? Are they thriving companies? Something else? I'm fascinated that there's these corners of the web that haven't changed all that much in decades, but are full of life, supported by platforms that surely must have to evolve to deal with threats and abuse at the very least.

I love all of it. This kind of thing is what makes the web great.

#Culture

[Link]

Wednesday, 09. October 2024

IdM Laboratory

Windowsのパスキー対応の今後

こんにちは、富士榮です。 いよいよ来週はAuthenticate 2024ですね。残念ながら参加できませんが。 ということで、Authenticateに向けて各社パスキー周りの話題が進んできていそうです。 MicrosoftからもWindowsのパスキー対応について記事を公開しています。 Passkeys on Windows: Authenticate seamlessly with passkey providers https://blogs.windows.com/windowsdeveloper/2024/10/08/passkeys-on-windows-authenticate-seamlessly-with-passkey-providers/ こちらの機能がWindows Insiderチャネルで配信されるようです。久しぶりにWind

こんにちは、富士榮です。

いよいよ来週はAuthenticate 2024ですね。残念ながら参加できませんが。


ということで、Authenticateに向けて各社パスキー周りの話題が進んできていそうです。


MicrosoftからもWindowsのパスキー対応について記事を公開しています。

Passkeys on Windows: Authenticate seamlessly with passkey providers

https://blogs.windows.com/windowsdeveloper/2024/10/08/passkeys-on-windows-authenticate-seamlessly-with-passkey-providers/


こちらの機能がWindows Insiderチャネルで配信されるようです。久しぶりにWindows PCでも触ろうかな・・・

A plug-in model for third-party passkey providers Enhanced native UX for passkeys A Microsoft synced passkey provider


サードパーティプロバイダとの連携では1Passwordなどとの連携ができるようになるようです。3点目のMicrosoftが提供する同期ファブリックと連携できたりすると面白そうです。Credential Exchange Specificationが実装されてくると面白いと思います。

いずれにしても来週のAuthenticateで詳しく言及されるのかと思います。楽しみですね。


John Philpin : Lifestream

I thought the new micro blog release would fix my problem. I

I thought the new micro blog release would fix my problem. It didn’t. On phone - a 09st longer than 600 you just can’t get the cursor to the bottom to write or edit. Anyone else?

I thought the new micro blog release would fix my problem. It didn’t.

On phone - a 09st longer than 600 you just can’t get the cursor to the bottom to write or edit.

Anyone else?


Simon Willison

Forums are still alive, active, and a treasure trove of information

Forums are still alive, active, and a treasure trove of information Chris Person: When I want information, like the real stuff, I go to forums. Over the years, forums did not really get smaller, so much as the rest of the internet just got bigger. Reddit, Discord and Facebook groups have filled a lot of that space, but there is just certain information that requires the dedication of adults w

Forums are still alive, active, and a treasure trove of information

Chris Person:

When I want information, like the real stuff, I go to forums. Over the years, forums did not really get smaller, so much as the rest of the internet just got bigger. Reddit, Discord and Facebook groups have filled a lot of that space, but there is just certain information that requires the dedication of adults who have specifically signed up to be in one kind of community.

This is a very comprehensive directory of active forums.

Via Andy Baio

Tags: forums


Free Threaded Python With Asyncio

Free Threaded Python With Asyncio Jamie Chang expanded my free-threaded Python experiment from a few months ago to explore the interaction between Python's asyncio and the new GIL-free build of Python 3.13. The results look really promising. Jamie says: Generally when it comes to Asyncio, the discussion around it is always about the performance or lack there of. Whilst peroformance is certa

Free Threaded Python With Asyncio

Jamie Chang expanded my free-threaded Python experiment from a few months ago to explore the interaction between Python's asyncio and the new GIL-free build of Python 3.13.

The results look really promising. Jamie says:

Generally when it comes to Asyncio, the discussion around it is always about the performance or lack there of. Whilst peroformance is certain important, the ability to reason about concurrency is the biggest benefit. [...]

Depending on your familiarity with AsyncIO, it might actually be the simplest way to start a thread.

This code for running a Python function in a thread really is very pleasant to look at:

result = await asyncio.to_thread(some_function, *args, **kwargs)

Jamie also demonstrates asyncio.TaskGroup, which makes it easy to execute a whole bunch of threads and wait for them all to finish:

async with TaskGroup() as tg: for _ in range(args.tasks): tg.create_task(to_thread(cpu_bound_task, args.size))

Tags: async, gil, python


Justin Richer

GNAP: A Conversation of Authorization

After five years of standardization work, GNAP is now officially RFC9635! This long and intense process actually started a few years prior to that, when I was talking with a lot of folks in the security industry about some of the shortcomings of OAuth 2.0, and what we could do about them as an industry. These conversations led to the XYZ proposal (and implementations) which eventually led to the f

After five years of standardization work, GNAP is now officially RFC9635! This long and intense process actually started a few years prior to that, when I was talking with a lot of folks in the security industry about some of the shortcomings of OAuth 2.0, and what we could do about them as an industry. These conversations led to the XYZ proposal (and implementations) which eventually led to the formation of the GNAP working group along with a bunch of others. In particular, the work that Fabien Imbault, Yaron Sheffer, Leif Johannsen, and Aaron Parecki put into the documents and conversations in the working group over these years.

I’m really proud of what we’ve built in GNAP. One of the core tenets of GNAP was to look at the world of OAuth and surrounding technologies and figure out how we could do a lot of that better. It’s been great to see GNAP getting applied in a bunch of places over the web, from payments to key management, and especially in places where OAuth doesn’t reach as well. While OAuth remains deeply entrenched over the world, and likely will be for some time, the community has learned many things from GNAP. Alot of things that started in GNAP have been making their way back to the OAuth ecosystem in some form.

The most obvious of this is RFC9396: OAuth Rich Authorization Requests. This replacement of OAuth’s scope parameter was a direct and intentional backport of what became GNAP’s resource access rights, which also acronyms to RAR. In the OAuth world, we don’t get some of the clean features of GNAP, like being able to substitute strings for objects as a shorthand, but a lot of the core enhancements are there.

We’re also seeing yet another intent registration addition to OAuth 2 (on top of the pushed authorization request, device grant type, and CIBA extensions), and this one mimics a lot of the flexibility of GNAP’s interaction system. It’s a more narrow use case in the OAuth specification, but it’s clear that the pattern that GNAP was built on is here to stay.

And then there’s RFC9421: HTTP Message Signatures. This is work that started independently from GNAP but grew up around the same time, and GNAP utilizes HTTP Message Signatures as a core security function. I don’t think we’d have gotten the signing spec to be as robust as it is without some of the GNAP key proofing use cases driving the discussion.

And finally, the GNAP Resource Servers document has just passed IESG review and is on its own way to becoming an RFC as well. This document represents key abstractions in how and RS and AS relate to each other, and I hope we can continue to build this out and pull the best ideas out into the world.

The GNAP working group is shutting down now that its core work is done, but GNAP is far from over. I look forward to seeing it grow into its spaces, and serve as a beacon of how a delegation protocol can be engineered and built.


Heres Tom with the Weather

We fix the fucking networks

“A lot of us remember what it was like to live and work on an Internet that was deeply flawed but not systematically designed to burn our emotions and time and safety for fuel.”

“A lot of us remember what it was like to live and work on an Internet that was deeply flawed but not systematically designed to burn our emotions and time and safety for fuel.”


Simon Willison

The Fair Source Definition

The Fair Source Definition Fail Source (fair.io) is the new-ish initiative from Chad Whitacre and Sentry aimed at providing an alternative licensing philosophy that provides additional protection for the business models of companies that release their code. I like that they're establishing a new brand for this and making it clear that it's a separate concept from Open Source. Here's their defin

The Fair Source Definition

Fail Source (fair.io) is the new-ish initiative from Chad Whitacre and Sentry aimed at providing an alternative licensing philosophy that provides additional protection for the business models of companies that release their code.

I like that they're establishing a new brand for this and making it clear that it's a separate concept from Open Source. Here's their definition:

Fair Source is an alternative to closed source, allowing you to safely share access to your core products. Fair Source Software (FSS):

is publicly available to read; allows use, modification, and redistribution with minimal restrictions to protect the producer’s business model; and undergoes delayed Open Source publication (DOSP).

They link to the Delayed Open Source Publication research paper published by OSI in January. (I was frustrated that this is only available as a PDF, so I converted it to Markdown using Gemini 1.5 Pro so I could read it on my phone.)

The most interesting background I could find on Fair Source was this GitHub issues thread, started in May, where Chad and other contributors fleshed out the initial launch plan over the course of several months.

Via Hacker News

Tags: licenses, sentry, pdf, open-source, chad-whitacre


Just a Theory

PGXN v2 Update

A lot has happened in he five months since the last PGXN v2 update. The time has been used for continuing community discussions, planning, designs, and the start of implementation. Read on for a full accounting.

Speaking of PGXN news, I neglected to link to this post I wrote for the Tembo Blog last month, a fairly detailed accounting of what’s been happening on the PGXN v2 project:

Forgive me Postgres community, for it has been five months since my last PGXN v2 Update. In my defense, it has been super busy! The time went into ongoing community discussions, planning, designs, and the start of implementation. Join me below for the lowdown.

A few highlights:

PGXN RFCs Repository and rfcs.pgxn.org Binary Distributution POC and OCI POC Extension Ecosystem Summit API and ABI guidance pgxn_meta v0.1.0 PGXN Meta JSON Schemas project plan

There’s been quite a bit of activity since then, including the aforementioned PGXN RFC–5 — Release Certification. More soon!

More about… PGXN

PGXN Certifications RFC

A request for comments on a new PGXN RFC for signing releases, plus a link to an initial implementation.

A couple weeks ago, I drafted PGXN RFC–5 — Release Certification, which proposes to replace the simple inclusion of a SHA-1 hash digests in PGXN release META.json files with a JWS-signed release payload. From the introduction:

This RFC therefore proposes to extend v2 distribution metadata with a single additional property, certs, that contains one or more certifications that attest to the authenticity or other characteristics of a release on PGXN.

The certs value is an object that contains at least one property, pgxn, which itself contains a PGXN-generated RFC 7515 JSON Web Signature in the JWS JSON Serialization format. The pgxn property will allow clients not only to assemble the release URL and verify the downloaded file against checksums, but also validate it against a public key provided by PGXN.

The design allows multiple signatures, certifications, or other attestations, which in the future MAY allow authors or other entities to sign releases with their own keys. The new format appends a structure such as this to the distribution META.json file:

{ "certs": { "pgxn": { "payload": "eyJ1c2VyIjoidGhlb3J5IiwiZGF0ZSI6IjIwMjQtMDktMTNUMTc6MzI6NTVaIiwidXJpIjoiZGlzdC9wYWlyLzAuMS43L3BhaXItMC4xLjcuemlwIiwiZGlnZXN0cyI6eyJzaGE1MTIiOiJiMzUzYjVhODJiM2I1NGU5NWY0YTI4NTllN2EyYmQwNjQ4YWJjYjM1YTdjMzYxMmIxMjZjMmM3NTQzOGZjMmY4ZThlZTFmMTllNjFmMzBmYTU0ZDdiYjY0YmNmMjE3ZWQxMjY0NzIyYjQ5N2JjYjYxM2Y4MmQ3ODc1MTUxNWI2NyJ9fQ", "signature": "cC4hiUPoj9Eetdgtv3hF80EGrhuB__dzERat0XF9g2VtQgr9PJbu3XOiZj5RZmh7AAuHIm4Bh-rLIARNPvkSjtQBMHlb1L07Qe7K0GarZRmB_eSN9383LcOLn6_dO--xi12jzDwusC-eOkHWEsqtFZESc6BfI7noOPqvhJ1phCnvWh6IeYI2w9QOYEUipUTI8np6LbgGY9Fs98rqVt5AXLIhWkWywlVmtVrBp0igcN_IoypGlUPQGe77Rw" } } }

Review and feedback would be very much appreciated, especially on the list of unresolved questions toward the end.

Thanks to David Christensen and Steven Miller for the early reviews!

Meanwhile, I’ve released pgxn_meta v0.4.0, which adds support for this format, as well as code to rewrite PGXN v1 release fields to the new format. It doesn’t actually do signature verification, yet, as the server back end hasn’t been updated with the pattern and PKI. But I expect to modify it in response to feedback and get it implemented in early 2025.

More about… Postgres PGXN RFC JWS

John Philpin : Lifestream

There is a play running in London called The Mousetrap. Barr

There is a play running in London called The Mousetrap. Barring ‘COVID TIMES’ it has been running for 72 years. The musical equivalent has to be The Stones. Hats off and all that, but in both cases, longevity is surely the driver of success rather than the quality of material?

There is a play running in London called The Mousetrap. Barring ‘COVID TIMES’ it has been running for 72 years.

The musical equivalent has to be The Stones. Hats off and all that, but in both cases, longevity is surely the driver of success rather than the quality of material?


To be real clear - I am NOT an Airline Analyst. I haven’t

To be real clear - I am NOT an Airline Analyst. I haven’t even played one on TV. But I do tend to look around and question claims that seem spurious. (And in my time down here, I do hear a large number of claims made that do not seem to add up). []🔗 My take on Chris Luxon. If you don’t know the name, that’s because he’s no Jacinda Arden .. though he holds her office in Aotearoa](https://subs

To be real clear - I am NOT an Airline Analyst. I haven’t even played one on TV. But I do tend to look around and question claims that seem spurious. (And in my time down here, I do hear a large number of claims made that do not seem to add up).

[]🔗 My take on Chris Luxon. If you don’t know the name, that’s because he’s no Jacinda Arden .. though he holds her office in Aotearoa](https://substack.philpin.com/p/was-he-any-good)

Written in response to a piece suggesting that Luxon’s not doing a good job. Why? My point is that despite the press, I’m not sure he ever did a good job.


The Pragmatic Engineer

Efficient scaleups in 2024 vs 2021: Sourcegraph (with CEO & Co-founder Quinn Slack)

On today’s episode of The Pragmatic Engineer, I’m joined by Quinn Slack, CEO and co-founder of Sourcegraph, a leading code search and intelligence platform.

The Pragmatic Engineer Podcast had a very warm reception with the first episode — thank you! Two weeks later, here is episode #2, with a CEO who still codes every day: Quinn Slack at Sourcegraph.

Listen now on Apple, Spotify, and YouTube.

Brought to you by:

Paragon: ​​Build native, customer-facing SaaS integrations 7x faster.

WorkOS: For B2B leaders building enterprise SaaS

On today’s episode of The Pragmatic Engineer, I’m joined by Quinn Slack, CEO and co-founder of Sourcegraph, a leading code search and intelligence platform. Quinn holds a degree in Computer Science from Stanford and is deeply passionate about coding: to the point that he still codes every day! He also serves on the board of Hack Club, a national nonprofit dedicated to bringing coding clubs to high schools nationwide. In this insightful conversation, we discuss:            

How Sourcegraph's operations have evolved since 2021

Why more software engineers should focus on delivering business value

Why Quinn continues to code every day, even as a CEO

Practical AI and LLM use cases and a phased approach to their adoption

The story behind Job Fairs at Sourcegraph and why it’s no longer in use

Quinn’s leadership style and his focus on customers and product excellence

The shift from location-independent pay to zone-based pay at Sourcegraph

And much more!

Takeaways

Some of my main takeaways from this conversation are these.

1. As software engineers, it’s increasingly important to understand what value you add to the business. A big difference between 2021 and 2024 is how companies are much more focused on efficiency: meaning they are hiring more conservatively, and less likely to fund teams with headcount that don’t contribute to the core focus of the company.

As a developer or manager, try to figure out how much your team contributes in revenue, or savings, or other key goals to the company. Are you working in what the company would consider as a profit center, or what is more as a cost center? We did a deepdive on this topic in The Pragmatic Engineer: check out the article, linked in the show notes.

2. AI tools are great to eliminate the toil that we, developers face, day-to-day. There are AI tools that position themselves as their goal being “replacing developers.” I found it sympathetic that Quinn did not think this is the sensible path. His approach is to start by using AI tools with some of the “dumbest things” like generating the changelog for a software release – I mean, assuming you generate a changelog. And then you take tedious tasks where these tools could help, and see if you can automate some more.

Do this one step at a time: and it will actually help devs and teams, and it’s a lot more achievable than saying “let’s replace this whole complicated workflow with AI.”

3. The reality of location independent pay is that it stops being sensible above a certain company size. Sourcegraph was one of the few companies that offered the same base salary regardless of where people worked at. They did this until they grew to about 200 people, and switched this model to a location-indexed model.

Quinn was honest about why they did it: because keeping this would have not made sense for the company, from the business point of view. Basically, location-independent pay means the company can hire very easily in low-cost regions, but it’s hard or impossible to do this in high-cost regions. It also creates the incentive for employees to move to a low cost region where they can save more. In the end, I don’t know of any company with more than 200 people that pays location-independent: all large companies have some kind of indexing on location, and the best companies just pay the top of the local market. We cover more about compensation in the deepdive on The trimodal nature of software engineering salaries.

If you enjoy the podcast, I’d very much appreciate if you subscribe on your favorite podcast player, and leave a review. It helps the podcast be discovered by more people. Thank you!

Show notes

Where to find Quinn Slack:

• X: https://x.com/sqs

• LinkedIn: https://www.linkedin.com/in/quinnslack/

• Website: https://slack.org/

In this episode, we cover:

(01:35) How Sourcegraph started and how it has evolved over the past 11 years

(04:14) How scale-ups have changed 

(08:27) Learnings from 2021 and how Sourcegraph’s operations have streamlined

(15:22) Why Quinn is for gradual increases in automation and other thoughts on AI

(18:10) The importance of changelogs

(19:14) Keeping AI accountable and possible future use cases 

(22:29) Current limitations of AI

(25:08) Why early adopters of AI coding tools have an advantage 

(27:38) Why AI is not yet capable of understanding existing codebases 

(31:53) Changes at Sourcegraph since the deep dive on The Pragmatic Engineer blog

(40:14) The importance of transparency and understanding the different forms of compensation

(40:22) Why Sourcegraph shifted to zone-based pay

(47:15) The journey from engineer to CEO

(53:28) A comparison of a typical week 11 years ago vs. now

(59:20) Rapid fire round

The Pragmatic Engineer deepdives relevant for this episode:

• Inside Sourcegraph’s engineering culture: Part 1 https://newsletter.pragmaticengineer.com/p/inside-sourcegraphs-engineering-culture
• Inside Sourcegraph’s engineering culture: Part 2 https://newsletter.pragmaticengineer.com/p/inside-sourcegraphs-engineering-culture-part-2

Referenced:

• Sourcegraph: https://sourcegraph.com/

• SharePoint: https://www.microsoft.com/en-us/microsoft-365/sharepoint/collaboration

• Google Code Search: https://developers.google.com/code-search

• Steve Yegge on LinkedIn: https://www.linkedin.com/in/steveyegge

• Shopify: https://www.shopify.com/

• Tobi Lutke still writes code for Shopify: https://news.ycombinator.com/item?id=25003574

• Gitlab: https://about.gitlab.com/

• Scott Kirby on LinkedIn: https://www.linkedin.com/in/jscott-kirby/

• Beyang Liu on LinkedIn: https://www.linkedin.com/in/beyang-liu/

• Perl: https://www.perl.org/

• Booking.com: https://www.booking.com/

• Loom: https://www.loom.com/

• CleanShot: https://cleanshot.com/

• Kazam: https://launchpad.net/kazam

• Ollama: https://ollama.com/

• StarCoder: https://huggingface.co/blog/starcoder

• Llama: https://www.llama.com/

• The Years of Lyndon Johnson: The Path to Power; Means of Ascent; Master of the Senate; The Passage of Power: https://www.amazon.com/Robert-Caros-Years-Lyndon-Johnson/dp/038535147X

• The Life Of Mahatma Gandhi: https://www.amazon.com/Life-Mahatma-Gandhi-Louis-Fischer/dp/1784700401

• Nimitz at War: Command Leadership from Pearl Harbor to Tokyo Bay: https://www.amazon.com/Nimitz-War-Command-Leadership-Harbor/dp/0190062363/r

Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email podcast@pragmaticengineer.com.


Just a Theory

⛰️ Postgres Ecosystem Summit EU

The sequel to the successful PGConf.dev event, the Extension Ecosystem Summit EU will showcases some exemplary extension use cases.

Given the success of the Extension Ecosystem Summit at PGConf.dev back in May, my colleague Floor Drees has organized a sequel, the Extension Ecosystem Summit EU on Tuesday, October 22, at the Divani Caravel Hotel in Athens. That’s “Day 0” at the same hotel as PGConf.eu. Tembo, Percona, Xata, and Timescale co-sponsor.

While the May event took the form of an open-space technology (OST)-style unconference aimed at extension developers, the EU event aims to inform an audience of Postgres users about the history and some exemplary use cases for extensions. From the invite:

Join us for a gathering to explore the current state and future of Postgres extension development, packaging, and distribution. Bring your skills and your devices and start contributing to tooling underpinning many large Postgres installations.

Jimmy Angelakos - pg_statviz: pg_statviz is a minimalist extension and utility pair for time series analysis and visualization of PostgreSQL internal statistics. Adam Hendel (Tembo) - pgmq: pgmq is a lightweight message queue. Like AWS SQS and RSMQ but on Postgres. Adam is pgmq’s maintainer since 2023, and will present a journey from pure Rust → pgrx → pl/pgsql. Alastair Turner (Percona) - pg_tde: pg_tde offers transparent encryption of table contents at rest, through a Table Access Method extension. Percona has developed pg_tde to deliver the benefits of encryption at rest without requiring intrusive changes to the Postgres core. Gülçin Yıldırım Jelínek (Xata) - pgzx: pgzx is a library for developing PostgreSQL extensions written in Zig. Mats Kindahl (Timescale) - TimescaleDB (C), [pgvectorscale] (Rust) and pgai (Python): maintaining extensions written in different languages.

I will also deliver the opening remarks, including a brief history of Postgres extensibility. Please join us if you’re in the area or planning to attend PGConf.eu. See you there!

More about… Postgres PGXN Extensions PGConf Athens Summit

Simon Willison

otterwiki

otterwiki It's been a while since I've seen a new-ish Wiki implementation, and this one by Ralph Thesen is really nice. It's written in Python (Flask + SQLAlchemy + mistune for Markdown + GitPython) and keeps all of the actual wiki content as Markdown files in a local Git repository. The installation instructions are a little in-depth as they assume a production installation with Docker or sys

otterwiki

It's been a while since I've seen a new-ish Wiki implementation, and this one by Ralph Thesen is really nice. It's written in Python (Flask + SQLAlchemy + mistune for Markdown + GitPython) and keeps all of the actual wiki content as Markdown files in a local Git repository.

The installation instructions are a little in-depth as they assume a production installation with Docker or systemd - I figured out this recipe for trying it locally using uv:

git clone https://github.com/redimp/otterwiki.git cd otterwiki mkdir -p app-data/repository git init app-data/repository echo "REPOSITORY='${PWD}/app-data/repository'" >> settings.cfg echo "SQLALCHEMY_DATABASE_URI='sqlite:///${PWD}/app-data/db.sqlite'" >> settings.cfg echo "SECRET_KEY='$(echo $RANDOM | md5sum | head -c 16)'" >> settings.cfg export OTTERWIKI_SETTINGS=$PWD/settings.cfg uv run --with gunicorn gunicorn --bind 127.0.0.1:8080 otterwiki.server:app

Via Hacker News

Tags: python, wikis, uv, markdown, git, flask, sqlalchemy, sqlite


openai/openai-realtime-console

openai/openai-realtime-console I got this OpenAI demo repository working today - it's an extremely easy way to get started playing around with the new Realtime voice API they announced at DevDay last week: cd /tmp git clone https://github.com/openai/openai-realtime-console cd openai-realtime-console npm i npm start That starts a localhost:3000 server running the demo React application. It as

openai/openai-realtime-console

I got this OpenAI demo repository working today - it's an extremely easy way to get started playing around with the new Realtime voice API they announced at DevDay last week:

cd /tmp git clone https://github.com/openai/openai-realtime-console cd openai-realtime-console npm i npm start

That starts a localhost:3000 server running the demo React application. It asks for an API key, you paste one in and you can start talking to the web page.

The demo handles voice input, voice output and basic tool support - it has a tool that can show you the weather anywhere in the world, including panning a map to that location. I tried adding a show_map() tool so I could pan to a location just by saying "Show me a map of the capital of Morocco" - all it took was editing the src/pages/ConsolePage.tsx file and hitting save, then refreshing the page in my browser to pick up the new function.

Be warned, it can be quite expensive to play around with. I was testing the application intermittently for only about 15 minutes and racked up $3.87 in API charges.

Tags: nodejs, javascript, openai, websockets, generative-ai, ai, llms, react

Tuesday, 08. October 2024

IdM Laboratory

SIDI Hub - ベルリンレポートを読む(5)

こんにちは、富士榮です。 引き続きSIDI Hubベルリンレポートを読んできましょう。 今回はユースケースをベースにしたGap分析です。Deboraがレポートしてくれています。 相互運用性を担保する上で大きな障壁になりそうな課題として以下を挙げています。 We focused on three topics: Relying Party Registration: it is tackled in the scope of EUDIW and covered by Aadhaar, NIMC, and others on a country-by-country basis. But how does this interoperate across borders on a global scale? Issuing Authority Discover

こんにちは、富士榮です。

引き続きSIDI Hubベルリンレポートを読んできましょう。


今回はユースケースをベースにしたGap分析です。Deboraがレポートしてくれています。

相互運用性を担保する上で大きな障壁になりそうな課題として以下を挙げています。


We focused on three topics:

Relying Party Registration: it is tackled in the scope of EUDIW and covered by Aadhaar, NIMC, and others on a country-by-country basis. But how does this interoperate across borders on a global scale? Issuing Authority Discovery: ICAO centralized this for passports after many years. But how will this work for public and private sector issuers? Legal Entity Identifiers: the LEI (GLEIF) and DNS (ICANN) are two current examples. What is the best way to achieve legal entity linking?

We then facilitated a discussion, and the following summarizes the key points addressed in the room.

我々は3つのトピックに焦点を当てた:

リライングパーティの登録:EUDIWの範囲内で取り組まれており、AadhaarやNIMCなどが国ごとにカバーしている。しかし、世界規模で国境を越えてどのように相互運用するのか。 発行機関の発見: ICAOは何年も経ってから、パスポートのためにこれを一元化した。しかし、公的機関や民間企業の発行者にとってはどのように機能するのだろうか。 法的実体識別子:LEI(GLEIF)とDNS(ICANN)が現在の2つの例である。取引主体の連結を実現する最善の方法は何か?

その後、ファシリテーターによるディスカッションが行われ、その中で取り上げられたポイントを以下に要約する。

どれも頭の痛い問題ですね。特に2点目、3点目は答えが出そうにない課題ですねぇ。。いつまで経ってもIssuerのディスカバリは難しい問題です。ここで言っているのは単純に公開鍵を取得するためのURLのディスカバリだけじゃないですからね・・・どうやってIssuerが正当な機関であることを信じられるか、みたいな話です。また、識別子も非常に難しい問題です。DNSは比較的成功したモデルではありますが、それでも過去に使っていたドメインを別の機関が取得するという問題などもありますので、長期的に運用する上では非常に難しいかと思います。

それぞれ深掘りしていきます。

Relying Party (RP) Registration:

The group discussed the nature of Registration, its requirements, and how Trust establishment could work globally.

Are we focused only on foundational identity, or do we include functional identity systems? Example from Nigeria: the agency responsible for ID management is NIMC. In the case of foundational identity, one of the first things they do is a process of due diligence called Verification. RPs are registered mainly for the foundational part. It is specific to the country.

依拠当事者(RP)登録:

このグループでは、登録の性質、要件、および信頼性確立がグローバルにどのように機能するかについて議論した。

私たちは基盤的 ID のみに焦点を当てているのか、機能的 ID システムも含めるのか。

ナイジェリアの例:ID 管理を担当する機関は NIMC である。基盤的 ID の場合、最初に行うことの 1 つは、検証(Verification)と呼ばれるデュー・ディリジェンス・プロセスである。RP は主に基礎部分のために登録される。これはその国特有のものである。 Why are RPs registering? What are the requirements? What are the types of problems we are trying to solve?

Example of the mDL standard: the Trust ecosystem is only for Issuers. If I share my mDL with you, why should I trust you? This concern is especially relevant for a commercial vendor, e.g., Aadhaar they have to register all RP fingerprint devices with governments to know they are trusted 

なぜRPは登録するのか?要件は何か?どんな問題を解きたいのか? 

mDL標準の例:トラスト・エコシステムは発行者のためだけのもの。mDLを共有した場合、なぜ信用しなければならないのか?この懸念は特に商業ベンダーに関連する。例えば Aadhaar の場合、信頼できることを知るためにすべての RP 指紋デバイスを政府に登録しなければならない。
Should the solution be based on use cases? Should it be a risk-based approach?
It depends on the type of credentials, e.g., education with entity categories. A commercial entity doesn’t need your entire transcripts. ソリューションはユースケースに基づくべきか。リスク・ベースのアプローチにすべきか?
クレデンシャルのタイプ(例えば、エンティティ・カテゴリーを持つ教育)によって異なる。営利団体は成績証明書全体を必要としない。  Should it be public or private-led, or a combination of both? 
Example, more public-led: ICAO 
Example, more private-led: ICANN 公共主導か民間主導か、あるいは両者の組み合わせか? 
より公共主導の例:ICAO 
より民間主導の例:CANN Governance relates to funding the operating costs: would it be self-funded like ICAO? Should it be external funding? What are good reference models? 
ICAOのように自己資金で運営するのか?外部資金とすべきか?良い参考モデルは何か? 
Should it be global or regional? 
AAMVA is in North America and only about driver's licenses 
 グローバルかリージョナルか 
 AAMVAは北米にあり、運転免許証に関するものだけである。 
How would we approach the following: 
Lifecycle management? 
Type of data? 
Legitimacy & KYB? 
Policy enforcement? 
以下について、どのようにアプローチしますか? 
ライフサイクル管理? 
データのタイプ? 
正当性およびKYB? 
ポリシーの施行? Should we pursue an academic analysis of the options? オプションについて学術的な分析を行うべきでしょうか? 
Who are the decision-makers, and why? 意思決定者は誰で、その理由は? 
Is there a hierarchy or a pre-existing way to navigate views? 階層やビューをナビゲートする既存の方法はあるのでしょうか? 
What is the appropriate role for: 
Governments? 
NGOs like the UN? Is the UN sufficiently independent? 
Standards Organizations? 
以下について適切な役割とはどのようなものか: 
政府? 
国連のようなNGO?国連は十分に独立しているか? 
標準化団体? What is required to achieve consensus? コンセンサスを得るために必要なことは何でしょうか? 

We discussed that the Champion Use Cases will indicate the breadth of the issues we have to face if we go for the widest possible interoperability

チャンピオンユースケースは、最大限の相互運用性を実現しようとする場合に直面する問題の広がりを示すことになるだろう、という点について話し合いました。

当然ですが、相互運用を考えるとかなり幅広い議論が必要となりますね。

もう少しスコープを絞って議論をシャープにしていかないとまとまらない気もします・・・(少なくとも一気に全体ミーティングでまとまる量じゃない)

リライングパーティだけで上記ボリュームだったので、他にもガバナンスなどもあるので、この辺りは明日以降に。


 


Simon Willison

If we had $1,000,000…

If we had $1,000,000… Jacob Kaplan-Moss gave my favorite talk at DjangoCon this year, imagining what the Django Software Foundation could do if it quadrupled its annual income to $1 million and laying out a realistic path for getting there. Jacob suggests leaning more into large donors than increasing our small donor base: It’s far easier for me to picture convincing eight or ten or fifteen l

If we had $1,000,000…

Jacob Kaplan-Moss gave my favorite talk at DjangoCon this year, imagining what the Django Software Foundation could do if it quadrupled its annual income to $1 million and laying out a realistic path for getting there. Jacob suggests leaning more into large donors than increasing our small donor base:

It’s far easier for me to picture convincing eight or ten or fifteen large companies to make large donations than it is to picture increasing our small donor base tenfold. So I think a major donor strategy is probably the most realistic one for us.

So when I talk about major donors, who am I talking about? I’m talking about four major categories: large corporations, high net worth individuals (very wealthy people), grants from governments (e.g. the Sovereign Tech Fund run out of Germany), and private foundations (e.g. the Chan Zuckerberg Initiative, who’s given grants to the PSF in the past).

Also included: a TIL on Turning a conference talk into an annotated presentation. Jacob used my annotated presentation tool to OCR text from images of keynote slides, extracted a Whisper transcript from the YouTube livestream audio and then cleaned that up a little with LLM and Claude 3.5 Sonnet ("Split the content of this transcript up into paragraphs with logical breaks. Add newlines between each paragraph.") before editing and re-writing it all into the final post.

Tags: jacob-kaplan-moss, django, dsf, claude-3-5-sonnet, llm, whisper


Anthropic: Message Batches (beta)

Anthropic: Message Batches (beta) Anthropic now have a batch mode, allowing you to send prompts to Claude in batches which will be processed within 24 hours (though probably much faster than that) and come at a 50% price discount. This matches the batch models offered by OpenAI and by Google Gemini, both of which also provide a 50% discount. Via @alexalbert__ Tags: gemini, anthro

Anthropic: Message Batches (beta)

Anthropic now have a batch mode, allowing you to send prompts to Claude in batches which will be processed within 24 hours (though probably much faster than that) and come at a 50% price discount.

This matches the batch models offered by OpenAI and by Google Gemini, both of which also provide a 50% discount.

Via @alexalbert__

Tags: gemini, anthropic, claude, generative-ai, openai, ai, llms


The Pragmatic Engineer

What is Reliability Engineering?

A history of SRE practice and where it stands today, plus advice on working with reliability engineers, as a software engineer. A guest post by SRE expert and former Googler, Dave O’Connor

Hi, this is Gergely with a subscriber-only issue of the Pragmatic Engineer Newsletter. In every issue, I cover challenges at Big Tech and startups through the lens of engineering managers and senior engineers. To get articles like this in your inbox, every week, subscribe:

Subscribe now

For software engineers, the job involves more than just building software systems; these systems must also be reliable. This is easy enough for a website with a small number of visitors; but the larger the system gets, the trickier reliability is to achieve. There’s a huge amount of complexity involved in making an app or website with tens, or hundreds, of millions of daily users work reliably for (almost) all of them.

Google pioneered the concept of Site Reliability Engineering (SRE), and it has become a pretty mainstream discipline with many mid size-and-above tech companies having dedicated SRE or reliability teams. To find out more about SRE and reliability engineering in general, I reached out to SRE veteran, Dave O’Connor. He was at Google in 2004 – working with the team, from where the SRE discipline emerged just a year before, in 2003.

Today, Dave covers:

Common terms. Why it’s called ‘reliability engineering’ instead of SRE, DevOps teams, etc.

History. From conception in 2003 at Google, to industry convergence in the 2010s.

Reliability engineering today. This is a discipline in growth mode. But near-perfect reliability increases cost exponentially, and most companies should not copy Google’s approach to the SRE role.

Four promises. A well-run reliability team sets and delivers on four promises: SLA/SLO/SLI, service substrate management, tech design involvement, and tactical heroism.

Working with reliability engineers. If there are reliability engineers in your workplace, find out their approach to the four promises, involve them in tech design discussions, and seek their help with reliability challenges.

Future of reliability engineering. Tooling is evolving, and a trend of companies considering quitting the cloud and being more opinionated about reliability practice.

Further reading. Article and book recommendations.

For related reading, see also these The Pragmatic Engineer deepdives:

Shipping to production

Healthy oncall practices

What is Security Engineering?

What is Data Engineering?

What is ML Engineering?

With this, it’s over to Dave.

Hi, I’m Dave. I’ve been a site reliability engineer (SRE) for 20 years, before many folks outside the Google ecosystem called it that. I joined the company in 2004, on a team tasked with (re)installing and (re)configuring machines in the fleet. 

We quickly realized that due to sheer numbers, it was not a job that could be done by humans at the then-current scale, let alone at the scale expected. At the time, the common capability to run and manage more than a few hundred machines simply didn’t exist. Then began the chartering of what became known as ‘site reliability engineering’ at Google. The rest, as they say, is history.

Several years later, that history started to be told in Site Reliability Engineering (I contributed chapter 29), and various publications thereafter. 

Since leaving Google in 2021 after 17 years as an SRE, I’ve led SRE and production groups at Elastic and Twilio, and I’m currently freelancing as a leadership practitioner for busy teams (SREs included), and as a coach for engineering leaders, focusing on reliability engineers. Check out my consulting services, and coaching practice.

1. Common terms

I use the term ‘reliability engineering’ in this article. This is because as a set of practices, it stands on its own and can be implemented via specialized ‘SREs’, the ‘DevOps’ model, or individually as software is built. Many publications go to great lengths to make this distinction, and the question of whether reliability should be done by specialized SREs, or everyone, is a resounding ‘it depends’. See my article 6 Reasons You Don’t Need an SRE Team for why I believe many companies don’t need a dedicated function.

As with any engineering specialization, anybody can do reliability engineering up to a point. The decision on hiring or building a dedicated SRE group is similar to the choice of whether to hire a dedicated QA, or an information security group. Does the business care enough about the outcomes to dedicate people and organizational headspace to it? Often, the answer is initially “no”. However, almost every enterprise encounters novel and domain-specific engineering challenges at some point, and in these situations dedicated SREs provide better outcomes, tailored to individual needs.

“Site reliability” as a name was coined by Google. The “site” in question was google.com. However, the role has grown since; for many years at Google I led the SRE group in charge of all the storage and databases. These were not ‘sites’ per se, but the name had stuck by that point. 

As with most nascent engineering functions, folks who do reliability engineering go by many titles:

SREs

Production engineers

DevOps

Platform engineering

DevSecOps

Platform trust

 Infrastructure engineering 

… and other verbal gymnastics! 

Reliability engineering goes by a lot of titles, the best-known being SRE

These titles all encompass pretty common practices. It’s also argued – correctly, if you ask me – that reliability engineering predates what Google did. Many of its common-sense or even specialized practices are taken from other disciplines. For example, the concept of the incident commander role for large-scale outages derives from the US Federal Emergency Management Agency (FEMA), founded in 1979. 

2. History

As computing has evolved and gone increasingly online, our needs have scaled beyond the capability of humans to perform tasks, or even understand the scale at which we find ourselves doing things. When there are 100,000+ machines, it’s impossible to eyeball a graph of all data points like servers or machines. We can not take a well-understood administrative action on a machine if we have to replicate it 100,000 times.

In the early 2000s, many enterprises went from a manageable number of physical computers to large numbers of servers. These servers performed one function and were clustered, or they could take more generic workloads. 

As Gavin McCance, Compute Services lead at CERN, put it, we started thinking of individual machines as cattle, not pets:

Transition from ‘cats’ to ‘cattle’. Source: CERN Data Centre Evolution by Gavin McCance

In the early days, the frame of reference for how many “a lot” of machines was, shifted almost monthly. At Google in around 2006, I went on a site visit to another large tech employer in Ireland, during which our group peered through a small window into “the largest datacenter in Ireland”, numbering thousands of machines. Cue some raised eyebrows and polite ‘ahems’ because we knew we had a room with twice as many machines, located half an hour’s drive away. 

Google’s scaling ability lay in being able to assemble and power machines, but also in being able to pour concrete and purchase supporting equipment like generators, when supply chains simply weren’t set up for our scale. It represented an enormous uptick in the amount of real, difficult, and unsolved engineering problems in delivering services. 

For 99% of enterprises, this kind of massive scaling up in servers isn’t (nor should be) a core competency. Hence the explosion in cloud computing because amassing the human expertise to understand and run all layers in the stack is far beyond the purview of most businesses. 

The early SRE role

At places like Google, it made sense to build the expertise and technology to cover all layers of the stack, from the frontend serving infrastructure and network links, back to the physical machines and power infrastructure on the ground. This was for reasons that can be linked back to one thing: scale.

When I joined the tech giant the number of physical machines was in the process of sailing comfortably into six digits, crossing 100,000 and growing fast. By comparison, at most enterprises, a couple of hundred machines was considered a pretty large footprint.

This had two major forcing functions, both related to scale:

Tooling at the time was simply not set up to handle the scale at which Google needed to operate. There were no datacenter-scale tools (where ‘datacenter’ now means many thousands of machines) that could install, configure, or administer machines at our scale. At the control plane level, there were no job or workload management systems suited to the task. Machine virtualization was already in use in many folks’ production setups, but Containerisation, Kubenetes, Terraform, Prometheus, and many other familiar components didn’t exist yet. In fact, Kubernetes and Prometheus are based, in part, on later internal Google tools. 

Also basically non-existent were SaaS options such as PagerDuty, and observability tools like Datadog and Honeycomb. 

It would take entire buildings and campuses full of people to manually operate machines. We were capable of hiring a lot of people, but we knew that without a giant shift in the technology and tooling, the tools and ways to operate machines weren’t going to work. For places like Google, this meant we’d quickly run out of people to be able to deal with the complexity and workload. We needed toolchains and practices beyond what the industry could provide. 

An additional forcing function for us was Google’s leadership’s almost fanatical desire for reliability and speed. Larry, in particular, cared a lot. Just as Gmail was launching and offering users an previously-unthinkable entire gigabyte of email storage, we were aiming for levels of precision and speed in serving content that were unheard of in most industries.

The fervent belief of Google’s founders was that speed and reliability mattered more than features. This belief was coupled with the understanding that we couldn’t achieve it traditionally, which made it an existential issue. The level of investment in building out all layers of the serving stack was a case of “because we can”, but also “because we have to, as nowhere else does what we need”.

There was never a question of whether traditional ‘ops’ would work at Google. We needed a specialized role, staffed by folks familiar with the problem space and engineering methods required to make it work. 

In 2003, the SRE role was born. Ben Treynor Sloss had been tasked with building Google’s “production team” and in his own words, he built “what happens when you ask a software engineer to design an operations team.” This turned into the birth of the SRE function at Google. From the outset, SRE was staffed in varying measures by systems/operations experts and software engineers. A large part of the remit of the team was to build the tools and practices required to operate Google’s fleet.

I joined as one of the first non-US SREs, based in Ireland. My background is in systems administration, and my previous employer’s machine fleet numbered somewhere in the high double-digits. I was one of the newer, specialized breed of “sysadmins who code”. We didn’t have a snappy name, but did have the drive to embody the three virtues of ‘laziness, impatience and hubris.’ 

When I joined, my first gig was ‘babysitting’ Gmail’s machine fleet. Basically, the job was to ensure there were enough machines to serve storage and serving needs, and to juggle decisions on waiting for tools and processes to catch up, or building them. In particular, many practices for working in distributed teams containing up to nine time zones, came from the early experience of collaborating with our SRE and product development counterparts in Mountain View and other offices.

Industry Convergence

Eventually, other companies caught onto the scaling issues, especially the hyperscalers. Each had their own approach, but over time, the notion grew industry-wide that making things reliable was a real-life engineering discipline, not simply ‘ops’. 

This step saw a number of terms coined to describe this engineering, including ‘DevOps’. At its core, this was the notion that the disciplines and practices of reliability engineering should be ingrained into the overall engineering organization. At places other than Google, this mostly took the form of combined developer/operations roles (i.e. “you build it, you run it”), which differed from Google’s implementation, but the practices were similar.

Around this time, Google started opening up about SRE, eventually publishing the first SRE book, and follow ups. Conferences such as USENIX SRECon, Devops Days, and other movements have solidified reliability engineering as a discipline that scales well beyond Google. Indeed, the company has become a consumer of many state-of-the-art developments.

3. Reliability Engineering Today

Reliability engineering is still in its growth and adoption phase. Unreliable software and systems which are slow or function incorrectly, are no longer tolerated by businesses and direct consumers. Fast, reliable internet access is becoming ubiquitous, and the services people use must be the same.

But aiming for near-perfect reliability scales costs exponentially. It’s estimated, based on experiences at AWS, that every “nine” of additional guaranteed availability (the difference between 99%, 99.9% and 99.99% uptime) scales overall costs by roughly ten times. This includes staffing, development and testing costs, and may only partially account for the opportunity costs of a necessarily slower release cycle. But slower release cycles aren’t for everyone! If you’re in a product space which can’t tolerate spending too much time and energy on testing and resiliency, the right answer may well be to aim lower.

It’s sensible practice for any organization to explicitly state how much they care about reliability. Know that it is not a race to the top: be realistic about balancing the price you’re willing to pay, with the reliability the business needs! 

In the end, business outcomes win. I have been in several versions of the same meeting where a product owner demands a certain number of ‘nines’ of availability, but when probed on what the business outcome of falling below this target is, they don’t really have an answer. This especially applies to non-serving systems and data pipelines, which can be behind in processing by several hours with no ill effects. However, it’s often seen as easy or necessary to demand to-the-minute SLAs from the outset, without reference to the ‘North Star’ of business needs.

As in disciplines such as security, there is a tradeoff. The downside when things go wrong is bad, but we don’t have carte blanche to do absolutely everything for extra reliability. It may be possible to neglect these areas entirely and risk the enormous downside of a security incident or outage, or to pursue reliability goals at the expense of the core mission.

Try to avoid implementing what Google does for its SRE practice. One of the authors of the original SRE Book, Niall Murphy, famously tore up a copy of it during a keynote at SRECon in 2022. But far from disavowing the book’s content, he was sounding a note of caution about copying what Google does, wholesale.

Absorbing Google’s approach effectively is not about copying as much of it as possible, it’s about discovering which elements make sense for you and what you’re building.

Since departing Google, I’ve worked with more than one company with its own “SRE Book Club”. This is great because there’s a lot of knowledge contained therein. However, I never saw rooms full of database admins poring page-by-page over database design publications and figuring out which features to include wholesale in their own setup, and this definitely applies to a modern SRE practice. The Google model of building everything yourself is just one way. It worked in the 2000s, but likely wouldn’t work today. 

The availability of technology and products that didn’t exist when Google was building SRE makes the tradeoffs a lot more understandable and explicit, in hindsight. I go into more detail on this topic in my article, “A Short History”.

4. Four promises

At its core, reliability engineering is this:

Read more


Ben Werdmüller

Earth’s ‘vital signs’ show humanity’s future in balance, say climate experts

[Damian Carrington at The Guardian] Meanwhile, while we're all paying attention elsewhere: "More and more scientists are now looking into the possibility of societal collapse, said the report, which assessed 35 vital signs in 2023 and found that 25 were worse than ever recorded, including carbon dioxide levels and human population. This indicates a “critical and unpredictabl

[Damian Carrington at The Guardian]

Meanwhile, while we're all paying attention elsewhere:

"More and more scientists are now looking into the possibility of societal collapse, said the report, which assessed 35 vital signs in 2023 and found that 25 were worse than ever recorded, including carbon dioxide levels and human population. This indicates a “critical and unpredictable new phase of the climate crisis”, they said."

And:

"“Climate change has already displaced millions of people, with the potential to displace hundreds of millions or even billions,” he said. “That would likely lead to greater geopolitical instability, possibly even partial societal collapse.”"

And:

"The assessment concludes: “Only through decisive action can we safeguard the natural world, avert profound human suffering, and ensure that future generations inherit the livable world they deserve. The future of humanity hangs in the balance.”"

In a world where everything seems amplified and like it's from some kind of comic book version of reality, making it hard to engage with it as actual truth, it's imperative that we don't gloss over this. We all have to change, and we all have to demand change.

#Climate

[Link]


John Philpin : Lifestream

They don’t move too fast in Oneroa 😵‍💫

They don’t move too fast in Oneroa 😵‍💫

They don’t move too fast in Oneroa 😵‍💫


Michael Ruminer

Sometimes It’s All About The Chunking

As I continue my study and experimentation with coding up AI solutions, and especially, at the moment, with Retrieval-Augmented Generation (RAG), I decided to work with a post from the Metadocs blog titled “Simple Agentic RAG for Multi Vector stores with LangChain and LangGraph”. It seemed it would cover two areas of interest, agentic operations and RAG. Little did I expect to learn a valuable les

As I continue my study and experimentation with coding up AI solutions, and especially, at the moment, with Retrieval-Augmented Generation (RAG), I decided to work with a post from the Metadocs blog titled “Simple Agentic RAG for Multi Vector stores with LangChain and LangGraph”. It seemed it would cover two areas of interest, agentic operations and RAG. Little did I expect to learn a valuable lesson in chunking. In this post I’ll pass along the obvious but well demonstrated lesson I gathered in the process.

It started with the prior mentioned post that referred to a prior post that it built upon. Following a link to that pre-requisite post, “Create a Langchain app with multiple vector store the easy way”, it in turn referenced an even earlier post as a pre-requisite. So down that rabbit hole I went. The earlier post was “Deploy a RAG app with Langchain in minutes”. I read the earliest of the three posts and it was a very simple RAG application. I coded it up ensuring I understood each line along the way. Most notable was that it was splitting the text into chunks on “\n\n”. I hadn’t looked at the source document they were providing as a sample. It turns out it was a text file of a US State of the Union address with a twist. Each sentence was followed by “\n\n” (two line feed carriage returns); an interesting if unrealistic formatting. I have my own example PDF that I have been using for testing out various RAG implementations and how it handles a specific prompt, so I copied two paragraphs from the document that contains the context I was after and formatted it with a “\n\n” after each sentence. Normally, I have been extracting the text from the PDF as part of the code and then chunking usually with recursive character text splitting, but I didn’t want to change this program since I was going to build on it. When done the results actually returned what I was after, a list of the 10 principles of SSI.

For no apparent reason, I decided to edit the text file and format with a single “\n” after each paragraph. This should return the same results if I edited the text split to represent this. It didn’t. I was, and still am, perplexed by this. It makes no sense that a double line feed split should return any different results than a single line feed results under the circumstances. I plan to revisit this as I believe I must be wrong despite trying multiple times. What was most important in the process was that with the right chunking, as simple as it was, I got the desired results when prompted whereas in all my past experiences it had failed. There was a difference in that I was reading a text file versus using a PDF text extractor and that I was using only two paragraphs focused on the context I wanted versus 70 pages of an academic paper that is probably very hard to extract even if the proper area of context was two clean paragraphs of text within that PDF. The real lesson for me is showing how chunking is so important. I suspect the major differential in the success was in the chunk divisions. I won’t rule out the simpler PDF as a contributor.

Next, I plan to try a few things in this naive RAG implementation before I move on to the multi-vector store — a PDF creation of just the two paragraphs that contain the needed context and split on paragraphs (\n) to see how that comes out. I’ll try the two paragraphs in a PDF with text extracted and chunked using RecursiveCharacterTextSplitter and separators=[“\n\n”, “\n”, “. “, “ “, “”] and a chunk size of 1000 and two different settings for overlap (0, and 200) as well as with SentenceTransformerEmbeddingFunction and then the default OpenAi embedding function. Let’s see how all those combinations work.

To recap, though I can’t explain why I got wildly different results depending on the parsing character used with the tex file format changed, I can suspect the simple chunking by sentence made a lot of difference. The other likely impacting result was clean simple text versus PDF-extracted text. I plan to experiment more and will report back the results. A take away for me, even if it was not entirely or even primarily the impactful element, was how important the chunking is for good results.


Simon Willison

Django Commons

Django Commons Django Commons is a really promising initiative started by Tim Schilling, aimed at the problem of keeping key Django community projects responsibly maintained on a long-term basis. Django Commons is an organization dedicated to supporting the community's efforts to maintain packages. It seeks to improve the maintenance experience for all contributors; reducing the barrier to en

Django Commons

Django Commons is a really promising initiative started by Tim Schilling, aimed at the problem of keeping key Django community projects responsibly maintained on a long-term basis.

Django Commons is an organization dedicated to supporting the community's efforts to maintain packages. It seeks to improve the maintenance experience for all contributors; reducing the barrier to entry for new contributors and reducing overhead for existing maintainers.

I’ve stated recently that I’d love to see the Django Software Foundation take on this role - adopting projects and ensuring they are maintained long-term. Django Commons looks like it solves that exact problem, assuring the future of key projects beyond their initial creators.

So far the Commons has taken on responsibility for django-fsm-2, django-tasks-scheduler and, as-of this week, diango-typer.

Here’s Tim introducing the project back in May. Thoughtful governance has been baked in from the start:

Having multiple administrators makes the role more sustainable, lessens the impact of a person stepping away, and shortens response time for administrator requests. It’s important to me that the organization starts with multiple administrators so that collaboration and documentation are at the forefront of all decisions.

Tags: open-source, django


@_Nat Zone

NIST SP800-63B-4 2pdのパスワード基準は2017年から踏襲されている〜新しいのはフィッシング耐性の要求

なんか、「定期的に変更するな」 NISTがパスワードポリシーのガイドラインについて第2版公開草案を発表 みたいな記事だのツイートだのが出回って「パスワード定期変更が禁止」されたのが新しいことみたいな言説が出回っているので言っておくが、2017年6月発行のNIST SP800-63…

なんか、「定期的に変更するな」 NISTがパスワードポリシーのガイドラインについて第2版公開草案を発表 みたいな記事だのツイートだのが出回って「パスワード定期変更が禁止」されたのが新しいことみたいな言説が出回っているので言っておくが、2017年6月発行のNIST SP800-63 第3版ですでにそうなってるからね。新しいことではない。パスワードの文字の構成に対する規制も同様。

NIST SP 800-63B-4 2pd のガイドラインによると、パスワードの要件は以下のものを含みます:

長さ: パスワードは少なくとも8文字以上で、推奨される最低長は15文字です。パスワードの最大長は少なくとも64文字にする必要があります。 文字の受付: ASCII印刷文字、スペース文字、Unicode文字を受け入れるべきです。Unicodeのコードポイントはパスワードの長さを評価する際に1文字としてカウントします。 構成ルール: 特定の文字種を混ぜるなどの構成ルールを課すべきではありません。 禁止リスト: 一般的に使用される、予想される、または漏洩したパスワードとの比較を行う必要があります。比較はパスワード全体で行い、部分文字列では行いません。 保存: ソルトとハッシュを使った適切なパスワードハッシング方式で保存する必要があります。 変更と複雑さ: パスワード定期変更を要求してはいけません。しかし、危殆化が検出されたときには変更を要求しなければなりません。 利便性: 強力なパスワード選択のためのガイダンスを提供し、パスワードが拒否された場合には理由を通知すべきです7

これらのガイドラインは、セキュリティとユーザーの利便性のバランスを考慮しています。

また、中程度以上のリスクのシステムにおいては、パスワードのみというのはダメで、AAL2として以下をはじめとする要求事項があります。

認証装置の管理: 請求者が、加入者アカウントに紐づけられた1つ以上の認証装置を管理しているという高い信頼性が求められます。 2つの異なる認証要素の所有と管理の証明が、安全な認証プロトコルを通じて必要となります。

暗号化:承認済みの暗号化技術を使用しなければなりません。AAL2で使用される認証装置は、承認済みの暗号化認証装置でなければなりません。

リプレイ攻撃からの保護:AAL2で使用される認証装置は、少なくとも1つはリプレイ攻撃に耐えるものでなければなりません。
認証済みチャネル:請求者と検証者の間の通信は、1つ以上の認証済み保護チャネルを介して行われなければなりません。

生体認証の使用:生体認証要素が使用される場合、特定の性能要件を満たさなければなりません.

フィッシング耐性: 検証者は、少なくとも1つのフィッシング対策認証オプションを提供しなければなりません。連邦機関は、職員、請負業者、およびパートナーに対し、連邦情報システムへのアクセスにフィッシング対策認証を使用することを義務付けなければなりません。

新しいとして注目すべきはむしろこの「フィッシング耐性」。事実上、パスワードオンリーはもちろん、パスワード+OTPも禁じたと見るのが正しいでしょう。

Monday, 07. October 2024

@_Nat Zone

今週の世界のデジタルID最新動向 (10月8日)

概要 詳細 SOURCE: https://www.thinkdigitalpartners.com/news/2024/10/07/digital-identity-global-roundup-186/
概要 アメリカ合衆国:
ノースカロライナ州の控訴裁判所が、州立大学の学生と職員が投票時に大学発行のデジタルIDを使用することを禁止しました。これは新しい写真付きID義務化に関連する決定です。 イギリス:
IT管理サービスプロバイダーのXalientがアイデンティティセキュリティ企業SailPointと提携しました。この協力により、グローバル企業のネットワークセキュリティ強化、重要資産の保護、規制基準の遵守を支援します。 ナイジェリア:
政府が外国人居住者に対し、国民識別番号(NIN)の取得を義務付ける法改正を提案しました。これは expatriates を正式な課税システムに組み込むことが目的の一つです。 ヨーロッパ:
Signicatが「オープンバンキングハブ」を立ち上げました。これは銀行口座を通じて個人情報を安全に確認する方法を提供します。 ドイツ:
政府がスマートフォン用の国家デジタルウォレットの開発計画を発表しました。これはEU全域でのデジタル身分証明を可能にするプロジェクトの一環です。 インド:
Aadhaar(国民ID)データ漏洩の疑いを受け、政府が関連ウェブサイトの閉鎖に動きました。 イギリス:
ロイズ・バンキング・グループが生体認証ID検証企業Yotiに対する1000万ポンドの融資を株式に転換しました。 ブータン:
世界初の国家デジタルIDプログラムとしてGlobal Acceptance Network(GAN)に参加しました。 リトアニア:
RegTech企業iDenfyが電子マネー機関FinCauseと提携し、KYCプロセスの自動化を目指します。 マレーシア:
ブロックチェーンプラットフォームZetrixが、中国国民の公式IDを海外でデジタル認証できるアプリケーションを発表しました。 ボスニア・ヘルツェゴビナ:
2025年半ばまでに市民向けデジタルIDウォレットの開発を計画しています。 パプアニューギニア:
新しいデジタルID、ウォレット、オンライン政府プラットフォームの試験運用を開始しました。 ヨルダン:
アメリカとカナダに住むヨルダン人向けにデジタルID有効化サービスを開始しました。 詳細 アメリカ合衆国 (ノースカロライナ州):
ノースカロライナ州控訴裁判所が、州立大学ノースカロライナ大学チャペルヒル校の学生と職員による大学発行のデジタルIDの投票時使用を禁止しました。これは新しい写真付きID義務化法に関連する決定です。州選挙管理委員会(民主党多数)が大学のモバイルIDを承認したのに対し、共和党全国委員会と州共和党が訴訟を起こし、法律では物理的なIDカードのみが認められると主張しました。 イギリス (Xalientの提携):
IT管理サービスプロバイダーのXalientがアイデンティティセキュリティ企業SailPointと提携しました。この協力により、XalientのアイデンティティコンサルティングサービスとSailPointの統合アイデンティティセキュリティプラットフォームを組み合わせ、グローバル企業のネットワークセキュリティ強化、重要資産の保護、規制基準の遵守を支援します。特にゼロトラストサービスを提供し、SailPointのIdentity Security Cloudを活用して、統合されたカスタマイズ可能なアイデンティティセキュリティソリューションを提供します。 ナイジェリア:
連邦執行評議会(FEC)が、国内に居住する外国人に対し国民識別番号(NIN)の取得を義務付ける法改正を提案しました。現在、外国人居住者はこの生体認証デジタルIDを取得できません。この提案の主な目的の一つは、expatriates を正式な課税システムに組み込むことです。また、ナイジェリアで働き収入を得ている外国人を特定し課税することを目指す経済安定化法案も同時に提案されています。 ヨーロッパ (Signicatのサービス):
Signicatが「オープンバンキングハブ」を立ち上げました。これは銀行口座を通じて個人情報を安全かつ同意ベースで確認する方法を提供します。消費者により広範な本人確認オプションを提供すると同時に、企業に対しては銀行口座の所有権、支払能力、口座情報の確認においてより高いセキュリティを提供します。このプロセスでは、個人が銀行口座を所有していることとその正当性を確認し、口座保有者の名前や銀行口座などのデータの完全性を提供します。 ドイツ:
ドイツ政府がスマートフォン用の国家デジタルウォレット開発計画を発表しました。これはEU全域でのデジタル身分証明を可能にする欧州デジタルアイデンティティウォレット(EUDI Wallet)プロジェクトの一環です。このウォレットにより、求人応募、オンラインバンキング、行政手続きなど様々な活動において、スマートフォンを通じて安全かつ容易にデジタル本人確認が可能になります。また、身分証明データや公的文書をデジタルで保存し、電子署名を行うこともできるようになります。 インド:
Aadhaar(国民ID)データの疑わしい漏洩を受けて、インド政府が関連ウェブサイトの閉鎖に乗り出しました。Aadhaar法では、個人情報を公開したり、どのようなドメインでも共有したりすることが禁止されています。インド固有識別番号庁(UIDAI)が問題のサイトについて警察に通報する責任を負っています。IT省は、センシティブな個人情報(AadhaarカードやPAN詳細を含む)を公開しているウェブサイトがあることを認識し、この問題を深刻に受け止めていると述べました。政府はサイバーセキュリティ実践の安全性に最高の優先順位を置いていると強調しています。 イギリス(Lloyd’s Banking Groupの投資):
Lloyd’s Banking Groupが生体認証ID検証企業Yotiに対する1000万ポンド(約1900万ドル)の融資を株式に転換しました。Yotiの最高経営責任者Robin Tombsは、顧客数で英国最大の銀行グループを株主として獲得したことで、英国市場における再利用可能なデジタルIDの採用を加速させるのに役立つと確信していると述べています。Yotiは生体認証ID検証、年齢確認、年齢推定サービスを提供しています。 ブータン:
ブータン王国が、国家デジタルIDプログラムとしてGlobal Acceptance Network (GAN)に参加しました1。国家IDプログラムとしてGANに参加するのはブータンが初めての事例になります。
GANは検証可能なデータを利用可能、信頼可能、相互運用可能にすることを支援する分野横断的なイニシアチブです。2023年9月17日に30以上の創設メンバー組織とともにソフトローンチされました。ブータンはGANにエコシステムメンバーとして参加することで、自国のデジタルIDを他のデジタルIDシステムや信頼エコシステムと統合する位置に立ちました。これにより、ブータン国民が世界中で自分のデジタル資格情報を使用できるようになる可能性があります。 リトアニア:
リトアニアを拠点とするRegTech企業iDenfyが、リトアニア銀行の規制を受ける新しく立ち上げられた電子マネー機関(EMI)FinCauseと提携しました。この協力を通じて、iDenfyはFinCauseのKnowYourCustomer(KYC)プロセスの自動化を目指しています。欧州経済地域(EEA)のB2B顧客やEEAで事業を展開するアジア企業向けに、シンプルな4ステップの認証プロセスを通じてより多くのコンバージョンを確保することを目的としています。 マレーシア:
レイヤー1のパブリックブロックチェーンプラットフォームZetrixが、中国国民の公式IDを海外でデジタル認証・検証できるアプリケーションを発表しました2。ZCertサービスを通じて、Zetrixは中国国外の企業が中国国民から提示されたデジタルIDに含まれる情報の信憑性を確認する電子Know Your Customer(eKYC)プロセスを簡素化・促進します。このサービスは、Zetrixと中国の National public blockchain Xinghuo BIFとの統合によって可能になりました。中国国民は自分のデジタルIDをXinghuo BIFに公開することを選択でき、その後、海外の検証機関がZetrixのZCertサービスを通じてそのデータにアクセスできるようになります。 ボスニア・ヘルツェゴビナ:
ボスニア・ヘルツェゴビナの市民向けデジタルIDウォレットが開発中で、2025年半ばまでの導入が予定されています。このプロジェクトはクロアチアのテクノロジースタートアップIdentity Consortiumが主導し、’EU4DigitalSME‘イニシアチブの一環として行われています。
目的は、安全な電子識別、認証、デジタル文書署名を通じて識別プロセスを近代化することです。EU4DigitalSMEは、ボスニア・ヘルツェゴビナの中小企業(SME)のデジタル変革を支援することに焦点を当てています。欧州連合の資金提供とドイツ政府の支援を受けて、このプログラムはデジタルインフラとサービスを近代化することで競争力を高め、より広範な欧州デジタル経済への統合を目指しています。
デジタルIDウォレットはIdentyumブランドの下でAdverta Proと協力して開発され、バックエンドシステムとモバイルアプリケーションの両方を提供します。このプラットフォームにより、ボスニアの市民はデジタルIDを使用して公共および民間のサービスにアクセスできるようになり、欧州の規制に準拠しつつ、ボスニアのEUデジタルエコシステムへの統合を促進します。 パプアニューギニア:
パプアニューギニアが新しいデジタルID、ウォレット、オンライン政府プラットフォームを立ち上げ、今後数週間で10,000人のユーザーを対象にパイロット運用を開始する予定です3
SevisPassデジタルID、SevisPortal、SevisWalletの試験運用が今週、限定的に開始されました。SevisPassは国民IDプログラムと既存の物理的IDから派生し、パプアニューギニアの住民はこれを使用してSevisPortalを通じて政府サービスにアクセスできます。SevisWalletアプリはSevisPassを保存します。
すでに7つの政府サービスがSevisPortalを通じて利用可能となっており、今後12〜18ヶ月の間に、国家経済を後押しするための「公共および民間サービスのデジタルエコシステム」が構築される予定です。国の情報通信技術省(DICT)の公共サービスICT運営委員会のSteven Matainaho委員長がこのように述べています。 ヨルダン:
ヨルダンが、アメリカとカナダに住むヨルダン人向けにデジタルID有効化サービスを開始しました。
このサービスはSanadアプリを通じて利用可能で、IrisGuardとVFS Globalとのパートナーシップにより確立されました。海外に住むヨルダン人の政府サービスへのアクセスを向上させることを目的としています。
外務・在外居住者省が土曜日に発表したプレスリリースによると、この新サービスにより、海外に居住するヨルダン国民がVFS Globalのオフィスを通じてデジタルIDを有効化できるようになります。
このサービスは、米国とカナダに居住する約20万人のヨルダン人に恩恵をもたらすと期待されています。

SOURCE: https://www.thinkdigitalpartners.com/news/2024/10/07/digital-identity-global-roundup-186/


John Philpin : Lifestream

Dear Twitter … NO. Just NO.

Dear Twitter … NO. Just NO.

Dear Twitter … NO. Just NO.


Simon Willison

Thoughts on the Treasurer Role at Tech NonProfits

Thoughts on the Treasurer Role at Tech NonProfits Will Vincent, Django Software Foundation treasurer from 2020-2022, explains what’s involved in the non-profit role with the highest level of responsibility and trust. Tags: dsf, django

Thoughts on the Treasurer Role at Tech NonProfits

Will Vincent, Django Software Foundation treasurer from 2020-2022, explains what’s involved in the non-profit role with the highest level of responsibility and trust.

Tags: dsf, django


IdM Laboratory

SIDI Hub - ベルリンレポートを読む(4)

こんにちは、富士榮です。 引き続きSIDI Hubベルリンイベントのレポートを見ていきます。 今回はユースケース分析です。相互運用性を目指しましょう、といっても具体的なユースケースを見つけてボトムアップで考えていかないと進まないので、このワークストリームでは有用なユースケースを各地域ごとに発見して分析して行きます。 Champion Use Cases: Process and Progress to Date - Elizabeth The Champion Use Case workstream is in the process of identifying champion use cases and then prioritizing using an agreed framework. In Berlin, the Champio
こんにちは、富士榮です。
引き続きSIDI Hubベルリンイベントのレポートを見ていきます。

今回はユースケース分析です。相互運用性を目指しましょう、といっても具体的なユースケースを見つけてボトムアップで考えていかないと進まないので、このワークストリームでは有用なユースケースを各地域ごとに発見して分析して行きます。
Champion Use Cases: Process and Progress to Date - Elizabeth

The Champion Use Case workstream is in the process of identifying champion use cases and then prioritizing using an agreed framework. In Berlin, the Champion Use Cases workstream sought to do three things throughout the day: Ground Minimum Requirements conversations in salient use cases Add more use cases and more texture to the data already gathered Gain input on prioritization criteria

チャンピオン・ユースケース・ワークストリームは、チャンピオンのユースケースを特定し、合意されたフレームワークを用いて優先順位を決定しているところである。ベルリンでは、チャンピオン・ユースケース・ワークストリームは、一日を通して3つのことを行おうとした:

重要なユースケースにおける最低要件の会話の基礎固め より多くのユースケースを追加し、すでに収集されているデータにさらに質感を加える。 優先順位付けの基準について意見を得る 

これまでパリ、ケープタウン、ベルリン、ワシントンDC、そして東京の企画を通して見てきていますが、やはりユースケースに関する関心度、優先順位の置き方は地域によってかなり異なるイメージです。その意味で各地域を回りながら状況をヒアリングして回る、というSIDI Hubのアプローチは理にかなっていると思います。(どうしてもUSとEUだけで話が決まっていく傾向がある世界ですし)

つまり、せっかくなのでアジアからも意見を出していかないとダメですよ、ってことです。


Early in the day, we reviewed the inputs from other sources and past SIDI Hub events:

Paris Summit and write specific user stories W3C credentials working group EU Wallet use cases EU + US TTP bilateral analysis SIDI Hub Cape Town New input from SIDI Berlin

Wishing to spend the morning on technical requirements, SIDI Hub Berlin grounded further discussions in two use cases: Refugees and Opening a Bank Account. In this early session, presenters reviewed the outputs from the deep dive conducted at SIDI Hub Cape Town.

その日の早い段階で、他の情報源や過去のSIDI Hubイベントからのインプットを見直した:

パリサミットと具体的なユーザーストーリーの作成 W3Cクレデンシャル・ワーキンググループ EU ウォレットのユースケース EUと米国のTTP二国間分析 SIDIハブ・ケープタウン SIDI ベルリンからの新しいインプット

SIDIハブ・ベルリンは、午前中を技術的な要件に費やすことを希望し、2つのユースケースでさらなる議論を行った。「難民」と「銀行口座開設」である。この早いセッションで、発表者はSIDI Hub Cape Townで行われたディープダイブからのアウトプットをレビューした。

ベルリンでは先に書いたようにこれまでの取り組みについて確認し、その後、さらなるテーマの深掘りをしていっています。日本にいると難民のユースケースはリアリティがない人も多いと思いますが、これまで移民を受け入れてきたドイツや欧州諸国においては重要なキーワードなんだと思います。


こちら、難民のユースケースですね。

特徴として、自国の法的な身元証明が受けられない状況にあるので、UNHCRが発行する証明書を利用できるか?というのが大きなポイントになります。これはワシントンDCでも話があり、先日のクィックレビューでも書きましたが、どうしても出生からの流れを含め身元を証明することが困難であり、Identity Verificationをする際の照合先がないところから身分を付与していくことになります。その付与プロセス自体がどこまでの保証レベル(IAL/Identity Assurance Level)を持つのか?テロリストが混入している可能性や身元ロンダリングに悪用されていないかを踏まえて、どこまでVerifierが受け入れることができるのか?人権や人道支援の文脈を含めてどのように判断していくのかは非常に難しい話です。ただ、世界的に助けを求めている人々の数がますます増えている昨今、目を背けるべき案件ではありませんね。


こちらは銀行口座の開設のユースケースです。

こちらも移民のケースにも少々関わってきますが、国境を超えて別の国で銀行口座を開設するのは非常に難しい状態です。KYCが難しいのはもちろん、CDDについても元となる実績情報などが取得しにくい状態にあるので、どうしてもリスクベースで考えるとリジェクトもしくはネガティブな判断が下されがちです。こちらもAML/CFTの観点も踏まえて良い落とし所を作っていく必要がありそうです。


今回はここまでです。

この後、ギャップ分析が続きます。





FACILELOGIN

The lessons learned in building a secure, scalable, and reliable identity platform @ DevRev

Image credits: https://www.networkrailmediacentre.co.uk/resources/ap-0409 At DevRev, we are building an API-first dev-centric platform that gives you one unified view of your customers and products by connecting your data, teams, and users to our AgentOS platform and app experiences. This blog post shares some insights into how we built (and keep evolving) a secure, scalable, and reliable identity
Image credits: https://www.networkrailmediacentre.co.uk/resources/ap-0409

At DevRev, we are building an API-first dev-centric platform that gives you one unified view of your customers and products by connecting your data, teams, and users to our AgentOS platform and app experiences. This blog post shares some insights into how we built (and keep evolving) a secure, scalable, and reliable identity platform at DevRev. The decisions we made and the lessons we learned throughout this journey, we believe, would be equally applicable to most SaaS companies.

I would like to acknowledge and thank Akanksha Deswal, and the DevRev identity team, who were part of this journey, and instrumental in figuring out what works best for us.

An identity platform serves as a secure bridge between users and the services they access — whether apps, APIs, or microservices — delivering a seamless and trustworthy way to manage, verify, and protect user identities in increasingly interconnected environments. While the expectations for an identity platform can vary depending on specific use cases, at DevRev, we primarily focused on the following core areas.

The expectations for an identity platform can vary depending on specific use cases; and at DevRev, we primarily focused on the above core areas.

The decision to buy or build components of an identity platform depends on several factors, such as business requirements (both functional and non-functional), cost at scale, available resources, in-house expertise, and more. We wanted to maintain complete control over our object model (organizations, users, groups, etc.) while delegating more generic identity capabilities to an external identity service (identity provider). Having control over the foundational object model is essential for innovation, and allowing a third party to dictate its structure or limitations can stifle growth and introduce significant risks.

A few years ago, I enjoyed reading Ask Your Developer: How to Harness the Power of Software Developers and Win in the 21st Century by Jeff Lawson. In the book, Jeff reframes the classic ‘build vs. buy’ debate into a more urgent choice: ‘build vs. die.’ As every company evolves into a software company, the true competitive advantage lies in the software they build. When software becomes the critical interface between our services and our customers, not building it to meet the exact needs means risking irrelevance — or worse, failure.

Building doesn’t always mean starting from scratch. We shouldn’t reinvent the wheel if existing solutions meet our needs. Instead, focus on developing what’s core to our business — the components that give us a competitive edge. Everything else, the foundational components that enable us to build our vision, can be sourced from the digital supply chain. I touched on this concept in the blog post, The Next TCP/IP Moment in Identity.

We had no interest in building identity federation standards like OpenID Connect or SAML, nor did we want to develop authentication protocols in-house. These are commodity features supported by almost all identity providers. However, we wanted to ensure the external identity service handling federation and authentication remained a ‘thin’ layer, separate from our business logic. This approach minimizes vendor lock-in.

Vendor lock-in isn’t just about making it easy to switch providers — though that’s certainly part of it. More importantly, it’s about retaining the freedom to innovate independently, which was our primary focus. Within the DevRev platform, Janus serves as the identity service, managing the identity object model and interacting with the external identity provider during organization and user onboarding flows. For the rest of the DevRev platform components, the external identity provider remains a black box, only interacting with Janus.

Choosing the right identity provider requires a thorough and thoughtful evaluation. Our objective wasn’t to find an identity provider that perfectly matched the DevRev object model but rather to focus on the core constructs, their relationships, and the integration effort with the DevRev platform.

In the DevRev platform, each organization can choose its preferred authentication method. For example, one organization might opt for Google login, while another may use their identity provider via SAML. The external identity provider’s object model must support this flexible relationship between organizations and their connections. For instance, the connection to Google (social) is made using OpenID Connect, and ideally, we should be able to reuse that Google connection (or any other social connection) across multiple organizations.

This means the identity provider’s object model should allow connections to be defined independently of a specific organization and provide the ability to associate them with any org. During our evaluation, we encountered one identity provider that could only define a connection within the scope of an individual organization (or tenant), which led us to reject it.

Apart from key functional requirements for authentication and identity federation, we also worried about the scalability, availability, reliability, testability, compliance, and interoperability of the identity provider, and of course the cost at scale. We picked Auth0, mostly because of the lightweight organization model it had and it could meet our requirement to scale the number of organizations to 1 million. Most of the prominent identity providers do support almost all the key identity federation and authentication needs. Only in non-functional requirements and object relationships, you would find the differentiators.

In our hunt for an external identity provider, we mostly looked for a SaaS product. Going for a SaaS product gave us the freedom to focus on our core business problem, and delegate anything related to the external identity provider to the SaaS vendor. A key reason one would pick an on-prem product over a SaaS product is the freedom to do heavy customizations. Building heavy customization in the identity provider is something we wanted to avoid from day 1.

Then again, I am not completely ruling out identity provider on-prem deployments. It would depend on the industry you are in and your business-specific needs. For example, highly regulated industries would pick an on-prem identity provider to keep all their customer data within their boundaries. I led the development of the open-source WSO2 Identity Server for almost 14 years, before joining DevRev, and saw many large-scale successful on-prem deployments.

When picking a SaaS identity provider, non-functional requirements play a key role. We have little control over the non-functional requirements, and even from the SaaS vendor’s point of view, they will find it hard to accommodate change requests related to non-functional requirements; as it could require core changes in their product. For example, Auth0 offers an availability of four 9’s. When you create a tenant in Auth0, it’s bound to one region, and internally they create a replica of that tenant in another availability zone. However, when the primary Auth0 region is down, we do not have the control to switch to the corresponding availability zone. It’s the Auth0 team that decides when to switch based on the severity of the incident. Auth0 also offers a private cloud deployment, but, its a costly option.

Another limitation we found in Auth0 (and of course in most of the other prominent SaaS identity providers as well) replication of data between regions. For example, when you pin your tenant to the Auth0 US-1 region, all the API calls from any region globally must be routed to the US-1 region. Due to this limitation, we’ve witnessed increased latency levels in login API calls (OpenID Connect) for the users connecting from Asia. This hurts the app load time and at the moment we are in the process of building a solution in-house to minimize the impact.

The DevRev platform is built with 250+ microservices running on Kubernetes. Janus is one of them. Each service has its CI/CD pipeline, which deploys each commit to main, to Prod after running end-to-end tests in Dev and QA. We’ve integrated Auth0 into the Janus integration tests. We have a dedicated Auth0 tenant for that, and in the CI/CD pipeline, we connect to Auth0 via the API to create organizations and users; and then associate users with organizations. Once the tests are complete, we run a clean-up task to remove the resources created in Auth0, as it has certain resource restrictions.

There was a time when the clean-up task failed to run and unfortunately went unnoticed. This filled up Auth0 resources and soon reached its limits. Auth0 has a limit on the size of the app metadata that can be associated with a user. We used the app metadata to associate the DevRev-generated organization identifier with an Auth0 user. Once we hit the limit, the Auth0 system became unstable, and none of the data in the test tenant could be accessed either via the UI or the API. This became a blocker for the prod promotion of the Janus service and all other services that relied on Janus for testing. As a quick workaround (while the Auth0 team was working on recovering the tenant), we had to spin up a new Auth0 tenant and use that in our tests. Usually, these limits enforced by Auth0 are way beyond our production needs, and we encountered this issue due to the failed clean-up task from our end.

Auth0 has a rate limit on its management API. During our tests, we used this management API to create organizations and users. Due to the nature of the tests, we started to hit the rate limits (even with proper retries) which resulted in some flaky tests. As a solution, to this and the resource limit issue we discussed before, we ended up mocking Auth0 in our integration tests, rather than directly hitting the Auth0 test tenant.

We follow GitOps at DevRev to automate and manage infrastructure and application deployments. Git is the single source of truth. This helps us to take any change via a review process and audit all changes. We followed the same model with Auth0, where appropriate. For example, we use Auth0 actions to inject the DevRev-generated org/user identifiers into the tokens it produces. The actions are Javascript code and we keep it in a Git repo and deploy to the Auth0 Prod tenant via Dev and QA. This helped us avoid any manual edits from the Auth0 management console, and keep all three environments consistent.

Currently, not all operations on Auth0 are fully managed through GitOps. For example, we still update certain settings, such as token lifetimes for applications, directly through the Auth0 management console. Although these are infrequent tasks, it’s essential to have a complete audit trail in production for every operation and change made. While Auth0 provides logs, they are only retained for a month. To ensure long-term visibility, we push Auth0 logs to Datadog and set up Slack alerts for specific cases. Additionally, we’re exploring the use of the Auth0 Terraform provider to automate and manage all Auth0 operations via GitOps.

Passwords are painful, not just from the user’s point of view but also from the product side. If an app has to support password-based login, it must also build the supporting infrastructure to recover passwords, rotate them periodically, implement strong second-factor authentication, and so on. From day one, we made a design decision to avoid passwords.

We use Auth0 as an identity bridge. The contract between the DevRev app and Auth0 is defined by OpenID Connect. Auth0 helps us connect multiple other identity providers via SAML and OpenID Connect to authenticate users. These identity providers are either the social identity providers or the enterprise identity providers owned by DevRev customers. Following the identity bridge pattern helped avoid storing user passwords, and also introducing a new identity provider or an authentication method requires no changes to the DevRev app. It can be done transparently.

The identity bridge pattern effectively addresses the common challenges posed by the spaghetti-identity and identity-silos antipatterns. Spaghetti identity occurs when a SaaS application tries to establish direct, point-to-point integrations with multiple identity providers, leading to complex, tangled connections. On the other hand, identity silos arise when an application is limited to a single federation or authentication protocol, making it difficult and costly to introduce new protocols in the future. By adopting the identity bridge pattern, organizations can simplify integration and enable flexibility in supporting multiple protocols without the downsides of these antipatterns.

Auth0 provides the latency numbers it introduces during the login operations; however, that does not reflect the latency the users experience at the edge. To address this concern, and also to have better visibility into the OpenID Connect login flow, we started routing all login calls to Auth0 via our CDN provider, which is Fastly. On the Fastly Compute@Edge platform, we run an edge gateway, which pushes all the API stats to Google BigQuery for further analysis. There we would know the latency at the edge corresponding to all the login operations. These latency numbers are very much closer to what the users experience. The following graph shows the p90 latency numbers in milliseconds for the last 31 days against a set of selected regions, for the Auth0 token endpoint.

p90 latency numbers in milliseconds for the last 31 days against a set of selected regions, for the Auth0 token endpoint

All DevRev applications utilize the OpenID Connect authorization code flow. The token endpoint, highlighted in the graph above, is called once during the initial app load and periodically thereafter for token refreshes. While the periodic refreshes happen asynchronously and do not affect the user experience, the latency of the initial token request during page load is more critical.

Two main factors contribute to this initial latency:

Geographic latency — All users, regardless of location, connect to a centralized Auth0 instance in the US-East region. Web-worker-based login flow — The DevRev web app, a single-page application (SPA), leverages this flow, which adds complexity and impacts performance during authentication.

We opted for the web-worker-based login flow to enhance security by storing user tokens in browser memory more securely. However, web worker memory is scoped to a single browser tab, which means that even if a user has an active Auth0 session, opening a new tab requires calling both the authorize and token endpoints during the initial page load. Upon analyzing the stats, we found that 91.1% of the calls to the Auth0 authorize endpoint were from users who were already authenticated. To address this, we developed an edge-based solution targeting these 91.1% of users, aiming to reduce the latency at the edge to almost zero. As of this writing, this solution is being tested in our development and QA environments and will soon be available in production.

91.1% of the calls to the Auth0 authorize endpoint were from users who were already authenticated (page reloads). The users once logged in to the app, keep the login session. This graph shows the average number of active vs silent logins by weekday for the last 90 days. The edge-based solution we are working on will reduce the latency at edge for the login calls for already authenticated users by almost 100%.

In addition to publishing stats related to Auth0 API performance from the edge to Google BigQuery, we also send logs to Datadog. This allows us to trigger alerts in case of Auth0 failures and detect downtimes early. Moreover, this helps us identify which users and organizations are impacted during Auth0 outages — events that have occurred a few times in the past couple of years.

Most of the support queries we receive from our customers related to login are due to some kind of an issue with the OTP delivery. Even though the number of such queries is quite low (compared to the success cases), when it happens, it’s painful to the users and hurts the business. Most of the DevRev customers at the stage of evaluation use email OTP-based login and then later switch to login with their identity provider, with SAML or OpenID Connect. Apart from our direct customers, we also offer a support portal for our customers’ customers (who we call Rev users), which also offers login with email OTP.

Most email OTP-related issues arise from one of the following causes:

The user’s email is added to the email provider’s global suppression list. The OTP email is marked as spam. The OTP email is rejected by the user’s mailbox due to internal domain restrictions, or other similar factors.

Email service providers (ESPs) maintain global suppression lists to prevent sending emails to addresses that have previously bounced (due to invalid addresses) or opted out of receiving emails from certain senders. If a user’s email address is on such a list, the OTP email won’t be delivered. This can happen if the user’s email previously encountered delivery issues, such as soft or hard bounces.

Initially, we used SendGrid as our ESP for sending OTP emails as well as marketing emails. Although we used two separate SendGrid accounts for these purposes, SendGrid maintains a global suppression list shared across accounts when emails originate from the same domain — devrev.ai in our case. This meant that if a user unsubscribed from marketing emails, they were automatically added to the global suppression list, preventing them from receiving OTPs. In some cases, users didn’t even unsubscribe themselves; certain plugins handled it automatically.

This behavior was undesirable, as it significantly impacted OTP delivery. To address the issue, we switched to AWS SES for sending OTP emails, which reduced global suppression list-related problems by nearly 90%.

We once encountered a particularly interesting issue related to email OTPs, and while it wasn’t a delivery problem, it’s worth mentioning. The user received the OTP email, but every time they tried entering the OTP, it failed. What made this case even more intriguing was that we discovered the OTP had already been attempted from a different IP address just before the user tried it. This explained why the OTP kept failing — the user was attempting to use an OTP that had already been used.

After further troubleshooting, we discovered the root cause. Along with the OTP in the email, we had also embedded an OTP link that allowed users to log in directly by clicking it. Although this link was commented out in the HTML template and not visible to the user, the organization this user belonged to had strict security protocols that automatically scanned and clicked on links embedded in incoming emails. This security bot didn’t distinguish between commented-out links and visible ones, so it clicked the hidden link, sending a request to the Auth0 server and invalidating the OTP before the user could use it. The issue was resolved by completely removing the commented-out OTP link from the email template.

Many email providers have spam filters that automatically classify certain emails as spam based on various criteria, such as the content of the email, sending frequency, or the sender’s reputation. If the OTP email is flagged as spam, it ends up in the user’s spam or junk folder, leading to the user not receiving it in their primary inbox. This could occur if the email provider deems the sender domain as suspicious or if the email contains certain keywords that trigger spam filters. After moving from SendGrid to AWS SES and then carefully designing the email template, we could get the number of spam emails almost down to 0.

Every organization and user in the DevRev platform is assigned a unique, immutable, and persistent pseudonym known as a DON (DevRev Object Name). Every resource in the platform has its own DON. The format of a DON is: don:<service>:<partition>:(<type>/<id>)+, inspired by Amazon Resource Names (ARN).

For example, don:identity:dvrv-us-1:devo/xyz is the DON of an organization. Here, identity refers to the service name, and dvrv-us-1 is the partition identifier, which indicates the organization’s location. For example, an organization in the us-east-1 region has dvrv-us-1 as its partition ID, while one in ap-south-1 has dvrv-in-1.

The object type is defined by the type field. For example, devo represents a Dev organization, while devu represents a Dev user. In the DevRev platform, an organization created is called a Dev org, and its users are Dev users. Customers of a Dev organization are known as Rev users and belong to a Rev organization (or workspace). Both Rev organizations and Rev users exist within the scope of a Dev organization. For instance, the DON of a Rev organization would be: don:identity:dvrv-us-1:devo/xyz:revo/pqr.

All personally identifiable information (PII) of DevRev users is stored exclusively in Janus, making it the single source of truth for PII. Other services in the platform reference users through their DONs. For example, when a client calls the works.get API to retrieve a work item, the request goes through the DevRev gateway, which makes a gRPC call to the corresponding service. The service returns the work item data, including the DONs of the users who created and are assigned to the item.

The DevRev platform is built with 250+ microservices running on Kubernetes. Fastly Compute@Edge serves as the entry point at the edge for DevRev services. At the origin, an API gateway intercepts all incoming traffic. Once the gateway verifies the JWT accompanying the API request, it dispatches the request to the appropriate service.

Since raw user DONs are not user-friendly, the gateway resolves them into user details, such as display name, email, and full name, by querying Janus (the API composition pattern). These user summaries are cached in Redis to avoid repetitive calls to Janus (from the gateway), ensuring faster response times for future requests.

A DON attached to a user or an organization is an immutable identifier. It’s not just immutable, but also resolvable by its own — or embeds information in it. For example, we use the partition ID embedded into the organization DON to route requests to the corresponding region of the organization, in our multi-regional routing implementation built at the edge. Designing a system based on immutable identifiers needs to be done with care because the cost of a bad design decision is high. For example, we embed the Dev organization ID into the user DON, which means we cannot move this user to another Dev organization or a user cannot be part of multiple Dev organizations. This is a conscious decision we made to scope all the resources in the DevRev platform under a Dev organization.

However, we made a mistake in our Rev user implementation. The Rev user DON embeds both the Dev organization ID and the corresponding Rev organization ID. This means, a Rev user cannot switch Rev organizations, or cannot be part of multiple Rev organizations. This was not a requirement initially, but as we kept learning from our customers, we figured that it was a too restrictive constraint to limit a Rev user to one Rev organization. We made the Rev user scoped at the Dev organization level and removed the Rev organization ID from the DON. This was a painful change.

The Dev organization ID (don:identity:dvrv-us-1:devo/xyz), which is embedded into a DON is not incremental. An incremental ID can expose sensitive information, such as the total number of organizations created over time. Malicious actors could exploit this to infer patterns about system growth or attempt ID enumeration attacks to access or manipulate unauthorized resources. Non-incremental IDs make it harder to guess or predict other organization IDs, improving security and reducing the risk of ID enumeration.

The randomly generated IDs provide global uniqueness, meaning IDs can be safely generated across distributed systems without the risk of collision. This allows for a more flexible, decentralized architecture where different systems or regions can generate IDs without coordinating with a central system.

We initially used a 48-bit random value as the Dev organization ID. Once base62 encoded, it becomes a string with either 8 or 9 characters. When we moved to the multi-regional deployment, we wanted to ensure the Dev organization ID is unique across all regions by adding a regional prefix, for example, “1” for us-east1. However, since the previously generated Dev organization IDs were already 8 or 9 characters long, this introduced a challenge. A new region could potentially generate an ID with the prefix that results in 9 characters, which could match an already generated ID from a different region, before the introduction of the regional prefix. To address this, we first ensured the base62-encoded 48-bit value was always 9 characters by adding padding, and then introduced a regional prefix on top of that, effectively making the Dev organization ID a fixed-length random string of 10 characters.

What does deleting a Dev organization mean? A Dev organization has many other resources linked to it within and outside the identity space (or Janus). Dev users, Rev users, workspaces, accounts, service accounts, system users, and preferences are all identity-related objects, while conversations, articles, tickets, and issues, for example, are objects outside the identity space.

This is a common problem in a distributed system. What we need is a cascade delete. Implementing a cascade delete in a distributed system requires careful consideration of consistency, availability, fault tolerance, and scalability. Unlike a traditional monolithic system, where database constraints and relationships can directly enforce cascade deletes, distributed systems involve multiple services, databases, and possibly different regions, making the process more complex.

When a Dev organization is deleted, we mark it as soft-deleted. A soft delete marks a Dev organization as deleted without actually removing it from the database. This is, in general, useful to avoid cascading failures in case other services depend on the deleted entity. When a Dev organization is marked as deleted, it triggers a CDC (change data capture) event, and we have a dedicated service (garbage collection service), listening to these events and taking actions based on the type of the object being deleted and the dependent object types.

Once the garbage collection service picks the event to soft delete an organization, it spins up a temporal workflow, which will trigger the cascade delete of the dependent objects after a retention period. As you have figured out already this is a recursive process, and we only need to specify the first level of dependencies.

In general, there are two common saga patterns for cascade delete implementation, orchestration-based saga and choreography-based saga. In the orchestration-based saga pattern, a central coordinator service orchestrates the cascade delete by calling each service that manages dependent objects in a defined sequence. It ensures the cascade completes across all services or triggers compensating actions (e.g., undo the deletion) if a step fails. In the choreography-based saga pattern, each service listens for events and performs its part of the cascade delete, triggering further events to continue the process. We picked the orchestration-based saga pattern over the choreography-based saga.

At DevRev, we leverage a range of databases — including MongoDB, PostgreSQL, and DynamoDB — each selected to address specific use cases. For all identity-related data managed by Janus and the STS (which we’ll discuss later in this blog), we rely on MongoDB. MongoDB is ideal for transactional (OLTP) workloads, excelling in scenarios that demand high write throughput and the ability to manage large volumes of concurrent transactions. Its built-in replication via replica sets provides automatic failover and data redundancy across multiple nodes, enhancing both reliability and fault tolerance. To ensure data consistency, we utilize a majority write concern and read from the primary node. Currently, we are focusing on improving read and write performance in this setup.

MongoDB also offers horizontal scaling through sharding, where each shard operates as a replica set. Data is distributed across shards using a shard key, which is a field (or combination of fields) that determines how MongoDB partitions and routes data. Although we currently operate with a replica set without sharding, we’ve designed each collection in the identity database with a Dev organization ID field that can serve as the shard key, allowing us to seamlessly enable sharding when needed.

Unlike traditional relational databases, which require predefined schema structures, MongoDB’s document-oriented nature supports schema-less design. This flexibility has allowed us to rapidly develop and iterate on identity objects without the need to define a schema or handle migrations upfront.

Along with MongoDB we also use Google BigQuery, which is a fully managed, serverless, and highly scalable data warehouse designed for fast SQL-based queries and real-time analytics on large datasets. We are using the MongoDB Kafka connector to capture change data capture (CDC) events from MongoDB and push them to BigQuery. This also acts as an audit for all changes at the data level.

SCIM support had always been planned but was deprioritized due to other tasks. We’re excited that it is now an integral part of the DevRev identity platform. SCIM offers a standardized protocol that streamlines the exchange of identity information between identity providers and external applications, significantly reducing manual processes and minimizing errors. It ensures that user data is synchronized and consistent across all systems. Furthermore, SCIM enhances security by automating the revocation of user access when employees leave or change roles, reducing the risk of unauthorized access and ensuring proper access management.

DevRev provides a SCIM endpoint for provisioning users and groups from an organization’s identity provider. Initial testing was done using Okta as the identity provider, and the DevRev SCIM integration will soon be available on the Okta Integration Network. Following that, we plan to expand availability to Azure and Google platforms. As a DevRev platform customer, the DevRev organization has already implemented SCIM for integration with Okta, streamlining identity management within the organization.

The identity bridge pattern (which we discussed before) facilitates trust mediation between the identity provider trusted by the DevRev platform (Auth0) and external identity providers that authenticate users from DevRev organizations. This pattern is primarily focused on human identities. To bridge the human identity into the services, we’ve introduced a security token service (STS). This adds another level of indirection in trust, and all DevRev services only need to trust the STS. You can read more about STS from this blog: Securing north/south and east/west traffic @ DevRev.

STS is a key part of the DevRev identity architecture, which helps to make the identity provider a thin layer, which we discussed earlier in this blog. An Auth0 token tells us who the user is and does not embed any PII data other than the email address. However, when exchanging an Auth0 token for an STS-issued token, we have the flexibility to enrich the token with the data available in the DevRev platform.

The STS is capable of issuing tokens of different types. This graph shows the number of tokens issued by the STS for the last 30 days.

The STS is capable of issuing tokens of different types. For example Personal Access Tokens (PAT) for Dev users, Application Access Tokens (AAT) for service accounts, Rev session tokens for Rev users, and so on. Each of these token types is bound to a set of processing rules. For example, while issuing a Rev session token, the STS talks to Janus, and just-in-time provisions the Rev user, if they do not exist in the system. Another example would be how we handle delegation workflows. When a snap-in wants to act on behalf of a Rev user, it authenticates with its AAT (which represents the snap-in service account) and requests a Rev session token. Also, the STS-issued tokens are used in the multi-regional setup to facilitate certain region-region interactions.

In this blog post, we highlighted key aspects of the DevRev identity platform and the lessons we’ve learned throughout its development. This is an ongoing effort, and as we continue to gain insights from our customers, we constantly discover new ways to enhance the platform and tackle exciting challenges in identity and security.

The lessons learned in building a secure, scalable, and reliable identity platform @ DevRev was originally published in FACILELOGIN on Medium, where people are continuing the conversation by highlighting and responding to this story.


Kerri Lemole

Explaining Verifiable Credentials and Open Badges 3.0

Part 1: The Trust Model of Open Badges When the Open Badges v 0.5 specification was first launched by the Mozilla Foundation in 2012, the intention was to recognize learning that happens at any time, anywhere, and any how. The trust of the badges was based on 1) the persistent hosting of the credential data and 2) the content of the credential in the context it was being evaluated (including
Part 1: The Trust Model of Open Badges

When the Open Badges v 0.5 specification was first launched by the Mozilla Foundation in 2012, the intention was to recognize learning that happens at any time, anywhere, and any how. The trust of the badges was based on 1) the persistent hosting of the credential data and 2) the content of the credential in the context it was being evaluated (including evidence which was considered much more significantly in early open badges). While digital signatures were included in subsequent versions, the trust shifted to the badging platforms who would presumably check the identity of the issuers before they allowed them to use their platform.

The challenges with this have been:

1) assuming that the badges will always be available for the earner. Trust is based on the presumption that badging platforms persist and will continue a relationship with the issuer, ensuring that even if an earner’s email address has changed that they will still have access to their badges.

2) ensuring that once a badge has been awarded to an earner that its contents do not change. There is no assured permanence for web platforms. Databases can be altered purposely or accidentally. They can be hacked and some badging platforms allow their clients to change badge descriptions after they’ve been awarded.

With the most recent update of Open Badges, version 3.0, a new trust model has been adopted that makes it possible for earners to

have access to their digital credentials for as long as they think they will need them, decide which online identity they would like associated with their credentials, know that their badges will be perceived as being trustworthy because they have been digitally signed by an identified issuer and can prove that their badge data has not changed since they claimed it.

1EdTech Open Badges 3.0 (OBv3) is based on a standard developed and maintained by the World Wide Web Consortium (W3C) called Verifiable Credentials (VCs). The W3C was founded by the creator of the web, Tim Berners Lee, and is responsible for providing the standards that make the web work. VCs can be used to prove any claim online. OBv3 builds upon the VC standard with the purpose of proving claims made about educational achievements. These credentials have historically included badges and micro-credentials but can now also be used for degrees, certifications, and licenses. This means that earners could prove all of their achievements the same way across the web with convenience and security.

This blog post is Part 1 of a series of content explaining W3C Verifiable Credentials and Open Badges 3.0. Stay tuned for Part 2 — “How do Verifiable Credentials and Open Badges 3.0 Work?”

For more information on the DCC, including membership, projects, and events, visit our website, wiki, and Linkedin. To receive our newsletter and invitations to webinars and events sign up for our community mailing list.

Explaining Verifiable Credentials and Open Badges 3.0 was originally published in Digital Credentials Consortium on Medium, where people are continuing the conversation by highlighting and responding to this story.


John Philpin : Lifestream

We know who he’s talking about … don’t we?

We know who he’s talking about … don’t we?

We know who he’s talking about … don’t we?


Simon Willison

What's New In Python 3.13

What's New In Python 3.13 It's Python 3.13 release day today. The big signature features are a better REPL with improved error messages, an option to run Python without the GIL and the beginnings of the new JIT. Here are some of the smaller highlights I spotted while perusing the release notes. iOS and Android are both now Tier 3 supported platforms, thanks to the efforts of Russell Keith-Mage

What's New In Python 3.13

It's Python 3.13 release day today. The big signature features are a better REPL with improved error messages, an option to run Python without the GIL and the beginnings of the new JIT. Here are some of the smaller highlights I spotted while perusing the release notes.

iOS and Android are both now Tier 3 supported platforms, thanks to the efforts of Russell Keith-Magee and the Beeware project. Tier 3 means "must have a reliable buildbot" but "failures on these platforms do not block a release". This is still a really big deal for Python as a mobile development platform.

There's a whole bunch of smaller stuff relevant to SQLite.

Python's dbm module has long provided a disk-backed key-value store against multiple different backends. 3.13 introduces a new backend based on SQLite, and makes it the default.

>>> import dbm >>> db = dbm.open("/tmp/hi", "c") >>> db["hi"] = 1

The "c" option means "Open database for reading and writing, creating it if it doesn’t exist".

After running the above, /tmp/hi was a SQLite database containing the following data:

sqlite3 /tmp/hi .dump PRAGMA foreign_keys=OFF; BEGIN TRANSACTION; CREATE TABLE Dict ( key BLOB UNIQUE NOT NULL, value BLOB NOT NULL ); INSERT INTO Dict VALUES(X'6869',X'31'); COMMIT;

The dbm.open() function can detect which type of storage is being referenced. I found the implementation for that in the whichdb(filename) function.

I was hopeful that this change would mean Python 3.13 deployments would be guaranteed to ship with a more recent SQLite... but it turns out 3.15.2 is from November 2016 so still quite old:

SQLite 3.15.2 or newer is required to build the sqlite3 extension module. (Contributed by Erlend Aasland in gh-105875.)

The conn.iterdump() SQLite method now accepts an optional filter= keyword argument taking a LIKE pattern for the tables that you want to dump. I found the implementation for that here.

And one last change which caught my eye because I could imagine having code that might need to be updated to reflect the new behaviour:

pathlib.Path.glob() and rglob() now return both files and directories if a pattern that ends with "**" is given, rather than directories only. Add a trailing slash to keep the previous behavior and only match directories.

With the release of Python 3.13, Python 3.8 is officially end-of-life. Łukasz Langa:

If you're still a user of Python 3.8, I don't blame you, it's a lovely version. But it's time to move on to newer, greater things. Whether it's typing generics in built-in collections, pattern matching, except*, low-impact monitoring, or a new pink REPL, I'm sure you'll find your favorite new feature in one of the versions we still support. So upgrade today!

Tags: beeware, sqlite, python, mobile, ios, android, russell-keith-magee, lukasz-langa


What's New in Ruby on Rails 8

What's New in Ruby on Rails 8 Rails 8 takes SQLite from a lightweight development tool to a reliable choice for production use, thanks to extensive work on the SQLite adapter and Ruby driver. With the introduction of the solid adapters discussed above, SQLite now has the capability to power Action Cable, Rails.cache, and Active Job effectively, expanding its role beyond just prototyping or t

What's New in Ruby on Rails 8

Rails 8 takes SQLite from a lightweight development tool to a reliable choice for production use, thanks to extensive work on the SQLite adapter and Ruby driver.

With the introduction of the solid adapters discussed above, SQLite now has the capability to power Action Cable, Rails.cache, and Active Job effectively, expanding its role beyond just prototyping or testing environments. [...]

Transactions default to IMMEDIATE mode to improve concurrency.

Also included in Rails 8: Kamal, a new automated deployment system by 37signals for self-hosting web applications on hardware or virtual servers:

Kamal basically is Capistrano for Containers, without the need to carefully prepare servers in advance. No need to ensure that the servers have just the right version of Ruby or other dependencies you need. That all lives in the Docker image now. You can boot a brand new Ubuntu (or whatever) server, add it to the list of servers in Kamal, and it’ll be auto-provisioned with Docker, and run right away.

More from the official blog post about the release:

At 37signals, we're building a growing suite of apps that use SQLite in production with ONCE. There are now thousands of installations of both Campfire and Writebook running in the wild that all run SQLite. This has meant a lot of real-world pressure on ensuring that Rails (and Ruby) is working that wonderful file-based database as well as it can be. Through proper defaults like WAL and IMMEDIATE mode. Special thanks to Stephen Margheim for a slew of such improvements and Mike Dalessio for solving a last-minute SQLite file corruption issue in the Ruby driver.

Via Hacker News

Tags: docker, ruby, sqlite, 37-signals, rails


Datasette 0.65

Datasette 0.65 Python 3.13 was released today, which broke compatibility with the Datasette 0.x series due to an issue with an underlying dependency. I've fixed that problem by vendoring and fixing the dependency and the new 0.65 release works on Python 3.13 (but drops support for Python 3.8, which is EOL this month). Datasette 1.0a16 added support for Python 3.13 last month. Tags: project

Datasette 0.65

Python 3.13 was released today, which broke compatibility with the Datasette 0.x series due to an issue with an underlying dependency. I've fixed that problem by vendoring and fixing the dependency and the new 0.65 release works on Python 3.13 (but drops support for Python 3.8, which is EOL this month). Datasette 1.0a16 added support for Python 3.13 last month.

Tags: projects, datasette, python


@_Nat Zone

マイナ保険証利用時の手続き簡素化:10月7日から開始

厚生労働省は、マイナ保険証を使用する際の手続きを簡素化し、より使いやすくする改善を10月7日から順次開始します。主な変更点は以下の2つです: 限度額適用認定証情報の提供同意プロセスの簡略化 変更前: 変更後: 医療情報提供の同意方法の一本化 変更前: 変更後: これらの変更により…

厚生労働省は、マイナ保険証を使用する際の手続きを簡素化し、より使いやすくする改善を10月7日から順次開始します。主な変更点は以下の2つです:

限度額適用認定証情報の提供同意プロセスの簡略化

変更前:

「高額療養費制度を利用する方はこちら」をクリック 別画面に移動 「限度額情報を提供しますか」という質問に回答

変更後:

画面遷移と同意プロセスを省略 自動的に限度額情報が提供される (出所)厚生労働省「マイナ保険証の利用促進等について」P.13 医療情報提供の同意方法の一本化

変更前:

手術歴、診療・薬剤情報、健診結果の3つの画面で個別に同意

変更後:

1つの画面ですべての情報提供に同意可能 必要に応じて個別の同意画面にアクセスすることも可能

これらの変更により、マイナ保険証使用時の手続きがよりスムーズになり、利用者の利便性が向上することが期待されます。

(出所)厚生労働省「マイナ保険証の利用促進等について」P.13

Ben Werdmüller

Silicon Valley, the New Lobbying Monster

[Charles Duhigg at the New Yorker] "As the tech industry has become the planet’s dominant economic force, a coterie of specialists—led, in part, by the political operative who introduced the idea of “a vast right-wing conspiracy” decades ago—have taught Silicon Valley how to play the game of politics. Their aim is to help tech leaders become as powerful in Washington, D.C., a

[Charles Duhigg at the New Yorker]

"As the tech industry has become the planet’s dominant economic force, a coterie of specialists—led, in part, by the political operative who introduced the idea of “a vast right-wing conspiracy” decades ago—have taught Silicon Valley how to play the game of politics. Their aim is to help tech leaders become as powerful in Washington, D.C., and in state legislatures as they are on Wall Street."

This is a major change - it wasn't so long ago that journalists were remarking that tech was hopeless at influencing Washington.

That's not always a bad thing, but it sometimes very much is - for example when Silicon Valley lobbies politicians against crypto protections, or against privacy legislation, or prevent rideshare drivers from receiving standard workplace benefits.

What is certainly true, which this article takes pains to point out, is that tech is now one of the most powerful cohorts in politics. Future Presidencies - perhaps including this next one - will be chosen in large part based on tech's agenda. That's a new normal we need to get used to, and tech workers who care about equity need to deeply understand.

#Democracy

[Link]


The blogosphere is in full bloom. The rest of the internet has wilted

[ John Naughton ] "If you log into Dave Winer’s blog, Scripting News, you’ll find a constantly updated note telling you how many years, months, days, hours, minutes and seconds the blog has been running. Sometime tomorrow morning the year field will switch to 30." Running a blog for 30 years is no small feat. Dave Winer's Scripting News is a big deal that has an enduring co

[ John Naughton ]

"If you log into Dave Winer’s blog, Scripting News, you’ll find a constantly updated note telling you how many years, months, days, hours, minutes and seconds the blog has been running. Sometime tomorrow morning the year field will switch to 30."

Running a blog for 30 years is no small feat. Dave Winer's Scripting News is a big deal that has an enduring community which he's built from scratch over that time.

This also resonates:

"In my experience, most journalists failed to understand the significance of the blogosphere. This was partly due to the fact that, like Dr Johnson, they thought that “No man but a blockhead ever wrote except for money”, and so bloggers must be weird."

My position: everyone should blog. Every new voice adds something new to the conversation. And long-term bloggers like Dave have shown the way.

#Media

[Link]


If Harris Wins, Whether She Keeps Lina Khan Will Be Extremely Telling

[Karl Bode at TechDirt] "The Harris campaign has remained largely silent on whether Khan will be allowed to stick around. And it remains entirely unclear whether Harris will continue Biden’s support of something that, for once, at least vaguely resembles antitrust reform and a crackdown of concentrated corporate power." Many tech leaders - the article calls out Reid Hoffman

[Karl Bode at TechDirt]

"The Harris campaign has remained largely silent on whether Khan will be allowed to stick around. And it remains entirely unclear whether Harris will continue Biden’s support of something that, for once, at least vaguely resembles antitrust reform and a crackdown of concentrated corporate power."

Many tech leaders - the article calls out Reid Hoffman - have put open pressure on Harris to let go of Khan. FTC leaders often change between administrations, but I agree the premise that Lina Khan has actually done a pretty good job - and certainly better at anti-trust than we've seen in decades.

That's important because tech hasn't been a sideline industry for a long time. It's integrated into every aspect of how we live our lives and learn about the world. We should care about how much power an individual tech company (and its backers) can get, both to protect a competitive market and to ensure no one company has outsized influence on our democracy.

And as Karl Bode points out, it will say a lot about Harris's Presidency:

"Right now, Harris is remaining ambiguous about whether Khan will be allowed to stay at her post; allowing voters to fill in the blanks using vibes and their imagination. Whether Khan is kept in office, or replaced with yet another cookie cutter careerist, should prove pretty immediately telling in the new year."

We may find out soon.

#Technology

[Link]


John Philpin : Lifestream

40 minutes waiting for the call center peep to pick up - onl

40 minutes waiting for the call center peep to pick up - only to be told that for what I need - I need to call a different number. I wouldn’t mind (ok - I do) - but it is to the other side of the world so stayed up late to do this - not got the energy to start all over again. 😬

40 minutes waiting for the call center peep to pick up - only to be told that for what I need - I need to call a different number. I wouldn’t mind (ok - I do) - but it is to the other side of the world so stayed up late to do this - not got the energy to start all over again.

😬


Chief of Station, 2024 - ★★★

Not bad to begin with.. but kind of fell off into cliches .. and isn’t Aaron a bit old to be brawling and fighting in the field?

Not bad to begin with.. but kind of fell off into cliches .. and isn’t Aaron a bit old to be brawling and fighting in the field?


Damien Bod

Microsoft Entra ID App-to-App security architecture

This article looks at the different setups when using App-to-App security with Microsoft Entra ID (OAuth client credentials). Microsoft Entra App registrations are used to configure the OAuth clients and resources. For each tenant, an Enterprise application is created for the client App registration when the consent is granted. The claims in the access token […]

This article looks at the different setups when using App-to-App security with Microsoft Entra ID (OAuth client credentials). Microsoft Entra App registrations are used to configure the OAuth clients and resources. For each tenant, an Enterprise application is created for the client App registration when the consent is granted. The claims in the access token are validated as much as possible to prevent unauthorized clients from using the resource.

Code: https://github.com/damienbod/GrpcAzureAppServiceAppAuth

Use a single ME-ID App registration for client and resource

A single Microsoft Entra ID App registration can be used for both the client and the resource specification. The aud and the azp claims in the access token will have the same value in this setup. The roles claims are of no benefit. This could be considered a bad architecture because the ME-ID App registration is used for two separate intentions, the client requesting the access token and the resource providing the service. Deployment is simplified as only one App registration is required. The setup cannot be used for multiple clients.

The access token created for this setup will have the same value for the aud claim and the azp claim as the client and the resource are the same. Both of the claims should be validated to prevent other clients accessing the resource.

{ "aud": "19893e32-3f4d-4c5a-b5ca-27891cf75666", "iss": "https://login.microsoftonline.com/7ff95b15-dc21-4ba6-bc92-824856578fc1/v2.0", "iat": 1727688828, "nbf": 1727688828, "exp": 1727692728, "aio": "k2BgYDD6x3jLSW9uSapm1it3vtdnC66c40v83qzsVLhQga//6EsA", "azp": "19893e32-3f4d-4c5a-b5ca-27891cf75666", "azpacr": "1", "oid": "a517247c-86ba-43d9-8e02-7cf7918c15f4", "rh": "0.AR8AFVv5fyHcpku8koJIVlePwTI-iRlNP1pMtconiRz3VmaFAAA.", "roles": [ "access_as_app" ], "sub": "a517247c-86ba-43d9-8e02-7cf7918c15f4", "tid": "7ff95b15-dc21-4ba6-bc92-824856578fc1", "uti": "DBleEOFcqk-rcld809IQAA", "ver": "2.0" }

In the Azure portal, the Microsoft Entra ID App registration is configured to add an application App Role. This role is returned in the access token as shown above.. It is not required in this setup.

The access token can be validated using Microsoft.Identity.Web in a Web API.

builder.Services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme) .AddMicrosoftIdentityWebApi(builder.Configuration.GetSection("AzureAd")); builder.Services.AddAuthorization(options => { options.AddPolicy("ValidateAccessTokenPolicy", validateAccessTokenPolicy => { // Validate id of application for which the token was created // In this case the CC client application validateAccessTokenPolicy.RequireClaim("azp", "19893e32-3f4d-4c5a-b5ca-27891cf75666"); // only allow tokens which used "Private key JWT Client authentication" // // https://docs.microsoft.com/en-us/azure/active-directory/develop/access-tokens // Indicates how the client was authenticated. For a public client, the value is "0". // If client ID and client secret are used, the value is "1". // If a client certificate was used for authentication, the value is "2". validateAccessTokenPolicy.RequireClaim("azpacr", "1"); }); });

Use separate ME-ID App registrations for client and resource

This architecture allows for separation of clients and is required if more than one client can use the resource. The different clients can be identified in the application using the azp claim or the oid claim. The aud claim has the resource value and must be validated. The separate clients allows you to distribute different client certificates or secrets for each client. Audit logs can also be per client. The application can also implement separate business logic or authorization based on the client_id, object ID. (azp or oid)

The aud claim returns the resource App registration client_id and the azp returns the client App registration client_id. As two App registrations are used, the claims have different values. The roles claim is also returned in the access token. The roles permission is required to setup the client and resource relationship in the portal, but not required for authorization, if the other claims are validated correctly. It must be validated that only the allowed client acquired the access token for the resource and not any application access token from the same tenant.

{ "aud": "1a03257f-18a2-4cfa-81c1-d3cfaba6b09e", "iss": "https://login.microsoftonline.com/7ff95b15-dc21-4ba6-bc92-824856578fc1/v2.0", "iat": 1727685006, "nbf": 1727685006, "exp": 1727688906, "aio": "k2BgYFCeXa1fbNX34Odf08MOyzuu8k5eIO5xfYJi0rL41gkxx9QB", "azp": "912584f0-442c-41e3-87b3-3232edd82940", "azpacr": "1", "oid": "eb11b352-b9cc-489b-9e50-5eae9ee92e09", "rh": "0.AR8AFVv5fyHcpku8koJIVlePwX8lAxqiGPpMgcHTz6umsJ6FAAA.", "roles": [ "SharedAppRoleForAppToAppClients" ], "sub": "eb11b352-b9cc-489b-9e50-5eae9ee92e09", "tid": "7ff95b15-dc21-4ba6-bc92-824856578fc1", "uti": "Poa-FXigHkWg2mrtySMPAA", "ver": "2.0" }

Claim values returned in the access token:

aud

This MUST be validated and has the client_id from the resource ME-ID App registration.

roles: SharedAppRoleForAppToAppClients

This can be used to validate the access token, if multiple clients can be used to access the resource. This is not required, if the aud is validated. When using OAuth client credentials and an App-to-App client (no delegated user), the default scope is used to access the resource and all roles from the App registration are included in the access token.

azp

The azp can be used to lock down the resource to intended clients. This is different for each client. If you validate this, only the allowed clients can use the resource.

tid

The tid claim can be used to validate the tenant which requested the resource. It is important to validate this when using multi-tenant App registrations, if the resource is not for “All” Microsoft tenants. Only allow the tenants intended for the resource.

oid

The oid claim is the Microsoft Entra ID Enterprise application object ID created for the client App registration. If you are using only a single client and validate the OID, then you can prevent other Enterprise applications using the resource.

Application implementation in ASP.NET Core

The resource can be implemented using Microsoft.Identity.Web, if it is an ASP.NET Core Web API application.

// Add services to the container. builder.Services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme) .AddMicrosoftIdentityWebApi(builder.Configuration.GetSection("AzureAd")); builder.Services.AddAuthorization(options => { options.AddPolicy("ValidateAccessTokenPolicy", validateAccessTokenPolicy => { // Validate id of application for which the token was created // In this case the CC client application // Works with multi-tenant App registrations validateAccessTokenPolicy.RequireClaim("azp", builder.Configuration["AzureAd:ClientId"]!); // Value of Azure App registration where role is defined (resource) validateAccessTokenPolicy.RequireClaim("aud", builder.Configuration["AzureAd:Audience"]!); // Single tenant Enterprise application object ID // Only validate if locking down to a single Enterprise application. validateAccessTokenPolicy.RequireClaim("oid", builder.Configuration["AzureAd:Oid"]!); // only allow tokens which used "Private key JWT Client authentication" // https://docs.microsoft.com/en-us/azure/active-directory/develop/access-tokens // Indicates how the client was authenticated. For a public client, the value is "0". // If client ID and client secret are used, the value is "1". // If a client certificate was used for authentication, the value is "2". validateAccessTokenPolicy.RequireClaim("azpacr", "1"); }); });

The application configuration would look like this with your tenant ids.

"AzureAd": { "Instance": "https://login.microsoftonline.com/", "Domain": "damienbodhotmail.onmicrosoft.com", "TenantId": "7ff95b15-dc21-4ba6-bc92-824856578fc1", "ClientId": "912584f0-442c-41e3-87b3-3232edd82940", // aud claim in the access token, Azure App registration client ID "Audience": "1a03257f-18a2-4cfa-81c1-d3cfaba6b09e", "Oid": "eb11b352-b9cc-489b-9e50-5eae9ee92e09" },

Is the App Role required?

In no setup, is the App role required in the application implementation as the aud claim can be used to validate the resource. As the default scope is used, all roles in the resource App registration are included in the access token.

The role is required to add a permission in the client ME-ID App registration for a separate App registration resource specification. The role is required in the Microsoft Entra ID portal, if you have different clients App registrations and resources App registrations.

Multi client, Multiple resources

In more complex architectures, the solution might have multiple services. A single client can used multiple resources.

The Azure App roles are required to assign the resources to the clients. The application requesting the access token for the resource must specify the default scope of the Microsoft Entra App registration resource. This means that in Microsoft Entra ID, a separate access token is used for each resource. A client can request an access token for any resource where the role has been assigned. In the resource implementation, i.e. the API access token validation, the App roles can be used to authorize the request. The aud claim can also be used and the roles are not required.

Notes

If you are using multiple clients, separate the client and the resource specifications. If you only have a single client for an API, then you can just deploy the single Azure App registration for both client and resource. If you are a purist and require that the App registrations match the implementations and are architectural correct, use only one App registration for each application, one the client and one for the resource. If requirements are changed, you can always split this later without complication. Less is more.

Links

https://github.com/AzureAD/microsoft-identity-web/wiki

https://learn.microsoft.com/en-us/entra/identity-platform/


Simon Willison

fav.farm

fav.farm Neat little site by Wes Bos: it serves SVG (or PNG for Safari) favicons of every Emoji, which can be added to any site like this: <link rel="icon" href="https://fav.farm/🔥" /> The source code is on GitHub. It runs on Deno and Deno Deploy, and recently added per-Emoji hit counters powered by the Deno KV store, implemented in db.ts using this pattern: export function increment

fav.farm

Neat little site by Wes Bos: it serves SVG (or PNG for Safari) favicons of every Emoji, which can be added to any site like this:

<link rel="icon" href="https://fav.farm/🔥" />

The source code is on GitHub. It runs on Deno and Deno Deploy, and recently added per-Emoji hit counters powered by the Deno KV store, implemented in db.ts using this pattern:

export function incrementCount(emoji: string) { const VIEW_KEY = [`favicon`, `${emoji}`]; return db.atomic().sum( VIEW_KEY, 1n ).commit(); // Increment KV by 1 }

Via Wes Bos on TikTok

Tags: deno, favicon, javascript, svg


VTracer

VTracer VTracer is an open source library written in Rust for converting raster images (JPEG, PNG etc) to vector SVG. This VTracer web app provides access to a WebAssembly compiled version of the library, with a UI that lets you open images, tweak the various options and download the resulting SVG. I heard about this today on Twitter in a reply to my tweet demonstrating a much, much simple

VTracer

VTracer is an open source library written in Rust for converting raster images (JPEG, PNG etc) to vector SVG.

This VTracer web app provides access to a WebAssembly compiled version of the library, with a UI that lets you open images, tweak the various options and download the resulting SVG.

I heard about this today on Twitter in a reply to my tweet demonstrating a much, much simpler Image to SVG tool I built with the help of Claude and the handy imagetracerjs library by András Jankovics.

Via @jpohhhh

Tags: svg, rust, webassembly


John Philpin : Lifestream

The worlds is a small place.

The worlds is a small place.

The worlds is a small place.


This popped up in my feed today. Opened up Readwise - a

This popped up in my feed today. Opened up Readwise - and this quote resurfaced. It seems to be connected. No?

This popped up in my feed today.

Opened up Readwise - and this quote resurfaced.

It seems to be connected. No?

Sunday, 06. October 2024

IdM Laboratory

Entra IDを使ったパスワードレスでのオンボーディングシナリオ

こんにちは、富士榮です。 Entra IDもVerified IDやFIDOなど色々な要素が組み合わさってきているので、それらの機能をどうやって組み合わせて使うのが良いのか?という疑問が湧いてきます。 そんな時にパスワードレスでオンボーディングをするというシナリオに基づくデザイン〜実装ガイドがMicrosoftから発行されていますので、見てみようかと思います。 Phishing-resistant passwordless authentication deployment in Microsoft Entra ID こちらのドキュメントです。 https://learn.microsoft.com/en-us/entra/identity/authentication/how-to-plan-prerequisites-phishing-resistant-p
こんにちは、富士榮です。
Entra IDもVerified IDやFIDOなど色々な要素が組み合わさってきているので、それらの機能をどうやって組み合わせて使うのが良いのか?という疑問が湧いてきます。
そんな時にパスワードレスでオンボーディングをするというシナリオに基づくデザイン〜実装ガイドがMicrosoftから発行されていますので、見てみようかと思います。
Phishing-resistant passwordless authentication deployment in Microsoft Entra ID こちらのドキュメントです。 https://learn.microsoft.com/en-us/entra/identity/authentication/how-to-plan-prerequisites-phishing-resistant-passwordless-authentication
全体像はこんな感じですね。

Onboarding step 1: Identity verification 最初のステップではEntra Verified ID(+3rdパーティソリューション)を使って政府発行のIDなどで本人確認するところからスタートします。その後、PCのBootstrapではTAP(Temporary Access Pass)を使ってドメイン参加〜認証器のエンロールをする、という流れですね。(もしくは、最近PreviewになったGraph APIで事前にFIDO認証器をプロビジョニングしておく、という方法もありますね)
関連資料) Entra Verified ID https://learn.microsoft.com/en-us/entra/verified-id/remote-onboarding-new-employees-id-verification Temporary Access Pass https://learn.microsoft.com/en-us/entra/identity/authentication/howto-authentication-temporary-access-pass#enable-the-temporary-access-pass-policy Graph APIでもFIDOクレデンシャルのプロビジョニング https://learn.microsoft.com/en-us/entra/identity/authentication/how-to-enable-passkey-fido2#provision-fido2-security-keys-using-microsoft-graph-api-preview Onboarding step 2: Bootstrap a portable credential 前のフェーズでTAPでBootstrapし、最初のクレデンシャルのエンロールをするタイミングです。ここで重要なのはデバイスにバインドされたクレデンシャルではなくポータブルなクレデンシャルをエンロールすべきである、という点です。当然働き方・デバイスの使い方によって事情は異なりますが、最初のクレデンシャルがデバイスにバインドされてしまうと後々困ることになるからですね。
Onboarding step 3: Bootstrap local credentials on computing devices ポータブルなクレデンシャルがエンロールされれば、あとは個別のデバイスのセットアップを自由にできるわけです。この段階でデバイスごとのローカルクレデンシャルをエンロールしていきます。典型的にはWindows HelloのPINの生成ですね。要するにローカルの鍵ストアをオープンするための手段を作っていくところです。

まぁ、非常に典型的な話ではありますが、ドキュメントではもっと細かくパターン分けされたデザインが出てきますので、みなさんの仕事の仕方、デバイスの種類を考えて適切なデザインをしていってください。

Simon Willison

SVG to JPG/PNG

SVG to JPG/PNG The latest in my ongoing series of interactive HTML and JavaScript tools written almost entirely by LLMs. This one lets you paste in (or open-from-file, or drag-onto-page) some SVG and then use that to render a JPEG or PNG image of your desired width. I built this using Claude 3.5 Sonnet, initially as an Artifact and later in a code editor since some of the features (loading a

SVG to JPG/PNG

The latest in my ongoing series of interactive HTML and JavaScript tools written almost entirely by LLMs. This one lets you paste in (or open-from-file, or drag-onto-page) some SVG and then use that to render a JPEG or PNG image of your desired width.

I built this using Claude 3.5 Sonnet, initially as an Artifact and later in a code editor since some of the features (loading an example image and downloading the result) cannot run in the sandboxed iframe Artifact environment.

Here's the full transcript of the Claude conversation I used to build the tool, plus a few commits I later made by hand to further customize it.

The code itself is mostly quite simple. The most interesting part is how it renders the SVG to an image, which (simplified) looks like this:

// First extract the viewbox to get width/height const svgElement = new DOMParser().parseFromString( svgInput, 'image/svg+xml' ).documentElement; let viewBox = svgElement.getAttribute('viewBox'); [, , width, height] = viewBox.split(' ').map(Number); // Figure out the width/height of the output image const newWidth = parseInt(widthInput.value) || 800; const aspectRatio = width / height; const newHeight = Math.round(newWidth / aspectRatio); // Create off-screen canvas const canvas = document.createElement('canvas'); canvas.width = newWidth; canvas.height = newHeight; // Draw SVG on canvas const svgBlob = new Blob([svgInput], {type: 'image/svg+xml;charset=utf-8'}); const svgUrl = URL.createObjectURL(svgBlob); const img = new Image(); const ctx = canvas.getContext('2d'); img.onload = function() { ctx.drawImage(img, 0, 0, newWidth, newHeight); URL.revokeObjectURL(svgUrl); // Convert that to a JPEG const imageDataUrl = canvas.toDataURL("image/jpeg"); const convertedImg = document.createElement('img'); convertedImg.src = imageDataUrl; imageContainer.appendChild(convertedImg); }; img.src = svgUrl;

Here's the MDN explanation of that revokeObjectURL() method, which I hadn't seen before.

Call this method when you've finished using an object URL to let the browser know not to keep the reference to the file any longer.

Tags: claude-3-5-sonnet, images, claude, ai, llms, svg, claude-artifacts, javascript, ai-assisted-programming, generative-ai


Quoting Ethan Mollick

Students who use AI as a crutch don’t learn anything. It prevents them from thinking. Instead, using AI as co-intelligence is important because it increases your capabilities and also keeps you in the loop. […] AI does so many things that we need to set guardrails on what we don’t want to give up. It’s a very weird, general-purpose technology, which means it will affect all kinds of things, and

Students who use AI as a crutch don’t learn anything. It prevents them from thinking. Instead, using AI as co-intelligence is important because it increases your capabilities and also keeps you in the loop. […]

AI does so many things that we need to set guardrails on what we don’t want to give up. It’s a very weird, general-purpose technology, which means it will affect all kinds of things, and we’ll have to adjust socially.

Ethan Mollick

Tags: ethan-mollick, ai


Ben Werdmüller

Rules for Resters

[Ben Werdmuller on Medium] I find myself returning to this piece I wrote on Medium about building downtime into your work and lifestyle. It's important: Eight years into working in America, I’m still getting used to the macho culture around vacations. I had previously lived in a country where 28 days per year is the minimum that employers can legally provide; taking time off

[Ben Werdmuller on Medium]

I find myself returning to this piece I wrote on Medium about building downtime into your work and lifestyle. It's important:

Eight years into working in America, I’m still getting used to the macho culture around vacations. I had previously lived in a country where 28 days per year is the minimum that employers can legally provide; taking time off is just considered a part of life. The US is one of the only countries in the world that doesn’t guarantee any vacation at all (the others are Tonga, Palau, Nauru, Micronesia, Kiribati, and the Marshall Islands). It’s telling that American workers often respond to this simple fact with disbelief. How does anything get done?! Well, it turns out that a lot gets done when people aren’t burned out or chained to their desks.

When was the last time you took a real lunch hour? I don't think I've had one in at least five years. That's not a good sign.

#Business

[Link]


You Can’t Own The Social Web

[Bix Frankonis] Bix Frankonis does not agree with my analysis of the Fediverse and the Social Web Foundation. For him, much of the issue relates to appropriation of the "social web" name: "Like many trade groups, this one is named and self-described in a manner deliberately meant to capture and colonize an entire area. To become, in effect, synonymous with what its name name

[Bix Frankonis]

Bix Frankonis does not agree with my analysis of the Fediverse and the Social Web Foundation. For him, much of the issue relates to appropriation of the "social web" name:

"Like many trade groups, this one is named and self-described in a manner deliberately meant to capture and colonize an entire area. To become, in effect, synonymous with what its name names. It shits on twenty-five years of the web."

He's obviously entitled to his opinion, but I personally think it's a stretch to say that it shits on 25 years of the web. Of course there was a social web before the Fediverse - I'm a long-term indieweb participant and an even more long-term blogger. But I don't think that precludes this name, which is more of a bet on one embodiment of the future of the social web.

But here's what I really love: this conversation is playing out across platforms, across blogs, and across sites. In many ways, it's an illustration in itself of what the web is, and why blogging remains wonderful.

#Technology

[Link]

Saturday, 05. October 2024

Simon Willison

UV with GitHub Actions to run an RSS to README project

UV with GitHub Actions to run an RSS to README project Jeff Triplett demonstrates a very neat pattern for using uv to run Python scripts with their dependencies inside of GitHub Actions. First, add uv to the workflow using the setup-uv action: - uses: astral-sh/setup-uv@v3 with: enable-cache: true cache-dependency-glob: "*.py" This enables the caching feature, which stores uv's own

UV with GitHub Actions to run an RSS to README project

Jeff Triplett demonstrates a very neat pattern for using uv to run Python scripts with their dependencies inside of GitHub Actions. First, add uv to the workflow using the setup-uv action:

- uses: astral-sh/setup-uv@v3 with: enable-cache: true cache-dependency-glob: "*.py"

This enables the caching feature, which stores uv's own cache of downloads from PyPI between runs. The cache-dependency-glob key ensures that this cache will be invalidated if any .py file in the repository is updated.

Now you can run Python scripts using steps that look like this:

- run: uv run fetch-rss.py

If that Python script begins with some dependency definitions (PEP 723) they will be automatically installed by uv run on the first run and reused from the cache in the future. From the start of fetch-rss.py:

# /// script # requires-python = ">=3.11" # dependencies = [ # "feedparser", # "typer", # ] # ///

uv will download the required Python version and cache that as well.

Tags: uv, jeff-triplett, github-actions, python


IdM Laboratory

SIDI Hub - ベルリンレポートを読む(3)

こんにちは、富士榮です。 粛々とSIDI Hub Tokyoの準備は進んでいるわけですが、始まるまでにちゃんとベルリンのレポートを読み終わっておきましょう。 前回まででPart Oneの概要が終わったので今回からはPart Twoのセッションごとの詳細レポートを見ていきます。 SIDI Summit Introduction - Mark Haine まずはイントロです。Markがレポートしてくれています。  Debora Comparin (SIA), one of the founders and organizers of SIDI Hub, opened the day and welcomed participants to the third convening of SIDI Hub before EIC in Berlin. Afforded b

こんにちは、富士榮です。

粛々とSIDI Hub Tokyoの準備は進んでいるわけですが、始まるまでにちゃんとベルリンのレポートを読み終わっておきましょう。


前回まででPart Oneの概要が終わったので今回からはPart Twoのセッションごとの詳細レポートを見ていきます。

SIDI Summit Introduction - Mark Haine

まずはイントロです。Markがレポートしてくれています。 

Debora Comparin (SIA), one of the founders and organizers of SIDI Hub, opened the day and welcomed participants to the third convening of SIDI Hub before EIC in Berlin. Afforded by its association with EIC, SIDI Hub is pleased to have expertise in the room that spans deep technical knowledge of transnational governance. Further attesting to this, representatives from Germany’s Federal Ministry of the Interior & Community and SPRIN-D, Germany’s Federal Agency for Disruptive Innovation, spoke to the importance of SIDI Hub’s focus on cross-border interoperability and open standards.

SIDIハブの創設者の一人であり、主催者でもあるデボラ・コンパリン(SIA)は、ベルリンのEICの前に開催されたSIDIハブの第3回会合への参加者を歓迎し、開会を宣言した。EICとの提携により、SIDIハブはトランスナショナル・ガバナンスに関する深い技術的知識を持つ専門家を会場に迎えることができた。このことをさらに証明するように、ドイツ連邦内務・地域省およびドイツ連邦破壊的イノベーション機関SPRIN-Dの代表者は、国境を越えた相互運用性とオープンスタンダードに焦点を当てたSIDI Hubの重要性を語った。

私も現地で参加しましたが、ドイツ政府の100%出資の機関であるSPRIN-Dのオフィスでイベントは開催されました。そのため、最初のWelcome keynoteはSPRIN-Dの方が担当しました。

Nick Mothershaw (OIX) reviewed the agenda, which emphasized the following:

1. Identifying Champion Use Cases

2. Identifying Major Barriers to Interoperability

3. Minimum Technical Requirements for Interoperability

4. Deepening our Trust Framework Analysis

5. Critical Research Questions

Nick Mothershaw(OIX)は、以下の点を強調したアジェンダをレビューした。

1 チャンピオンのユースケースの特定

2. 相互運用性に対する主な障壁の特定

3. 相互運用性のための最低技術要件

4. 信頼フレームワーク分析の深化

5. 重要な研究課題

そのあとはNickによるアジェンダの紹介がありました。


SIDI Strategy and Structure - Mark

続いてGailによるSIDI Hubのストラテジーとストラクチャの話です。
Gail Hodges provided an overview of the origins of SIDI Hub, which derived from an ID4Africa presentation and the “Human-Centric Digital Identity” paper. In particular, the problem of cross-border interoperability in the context of national Digital Identity strategies encompassing a wide range of technical architectures and governance models. Gail Hodges は、ID4Africa のプレゼンテーションと「人間中心のデジタル ID」論文から派生した SIDI ハブの起源について概要を説明した。特に、広範な技術アーキテクチャーとガバナンス・モデルを包含する各国のデジタル ID 戦略の文脈における国境を越えた相互運用性の問題について述べた。


この辺りはいつものGailのセッションなのですが、各国のデジタルID戦略をPublic Governance-Private Governance、Centralized-Decentralizedの2軸で4象限に分類し、現状のばらつきを表現しつつ、この環境のもとで国境を超えた相互運用性を達成することの必要性について話しているわけです。

Despite these challenges - and the ongoing need for domestic sovereignty - can one’s Digital Identity be as easy to present as an email, a phone number, or a passport? SIDI Hub seeks to build a blueprint for how we build Digital Identity ecosystems within and across ecosystems. The goal is for implementers to build interoperable Digital Identity credentials by default. But this, of course, requires measurement and metrics, policies, open standards, open source code (in many jurisdictions), and scientific analysis for best practice security.

このような課題があるにもかかわらず、そして国内主権の継続的な必要性があるにもかか わらず、デジタル ID は電子メール、電話番号、パスポートのように簡単に提示することができるのだろうか?SIDI ハブは、エコシステム内およびエコシステム間でデジタル ID エコシステムを構築する方法の青写真を構築することを目指す。目標は、実装者がデフォルトで相互運用可能なデジタル ID クレデンシャルを構築することである。しかし、これにはもちろん、測定と測定基準、ポリシー、オープン・スタンダード、オープン・ ソース・コード(多くの法域で)、およびベスト・プラクティスのセキュリティのための科学 的分析が必要である。


こんなバラバラな状態の中でもデジタルIDをメールや電話やパスポートのように国境を超えて世界中で相互運用できる状態にするにはやることがたくさんありますね。まさにこれがSIDI Hubがやろうとしていること、というわけです。



SIDI Hub is self-organized into five workstreams:

Champion Use Cases Trust Framework Mapping Minimum Requirements for Interoperability Metrics of Success Governance

As referenced above, SIDI Hub has no governance authority of its own. We therefore discussed where decisions are made, which remain unchanged as a result of SIDI Hub, and how SIDI aims to support them.

SIDI Hubは、以下の5つのワークストリームから構成される。

チャンピオンのユースケース トラストフレームワークマッピン 相互運用のための最低要件 成功の指標 ガバナンス

上記で言及したように、SIDI Hub はそれ自体のガバナンス権限を持たない。そのため、SIDI Hubの結果として変わることのない意思決定がどこで行われるのか、また、SIDIがどのようにそれをサポートすることを目指しているのかについて議論した。


こちらはいつものSIDI Hubとは何なのか、という話と構成するワークストリームの話ですね。 非常に難しい部分なのですがコミュニティなのでコンセンサスを取りながら意思決定をしていくというのが特徴でもあります。この辺りは今後変わっていくかもしれません。


今回はこのくらいです。ユースケース分析のセッションについて次回解説します。


Simon Willison

marimo v0.9.0 with mo.ui.chat

marimo v0.9.0 with mo.ui.chat The latest release of the Marimo Python reactive notebook project includes a neat new feature: you can now easily embed a custom chat interface directly inside of your notebook. Marimo co-founder Myles Scolnick posted this intriguing demo on Twitter, demonstrating a chat interface to my LLM library “in only 3 lines of code”: import marimo as mo import llm model

marimo v0.9.0 with mo.ui.chat

The latest release of the Marimo Python reactive notebook project includes a neat new feature: you can now easily embed a custom chat interface directly inside of your notebook.

Marimo co-founder Myles Scolnick posted this intriguing demo on Twitter, demonstrating a chat interface to my LLM library “in only 3 lines of code”:

import marimo as mo import llm model = llm.get_model() conversation = model.conversation() mo.ui.chat(lambda messages: conversation.prompt(messages[-1].content))

I tried that out today - here’s the result:

marimo.ui.chat() takes a function which is passed a list of Marimo chat messages (representing the current state of that widget) and returns a string - or other type of renderable object - to add as the next message in the chat. This makes it trivial to hook in any custom chat mechanism you like.

Marimo also ship their own built-in chat handlers for OpenAI, Anthropic and Google Gemini which you can use like this:

mo.ui.chat( mo.ai.llm.anthropic( "claude-3-5-sonnet-20240620", system_message="You are a helpful assistant.", api_key="sk-ant-...", ), show_configuration_controls=True )

Tags: llm, marimo, python, llms, ai, generative-ai


Werdmüller on Medium

The two Fediverses

For some, it’s about growth. For others, it’s a movement. They must work together. Continue reading on Medium »

For some, it’s about growth. For others, it’s a movement. They must work together.

Continue reading on Medium »


Ben Werdmüller

The two Fediverses

I was tagged in a fairly critical SocialHub post about the Social Web Foundation launch announcement. I wasn’t in a position to add to the conversation then, but I’ve been thinking about it all week. Before I dive further, a reminder: I am not an employee or founder of the Social Web Foundation. I am in touch with the founders and have been an unpaid advisor, but I can’t and don’t speak

I was tagged in a fairly critical SocialHub post about the Social Web Foundation launch announcement. I wasn’t in a position to add to the conversation then, but I’ve been thinking about it all week.

Before I dive further, a reminder: I am not an employee or founder of the Social Web Foundation. I am in touch with the founders and have been an unpaid advisor, but I can’t and don’t speak for it. This post is mine alone, and doesn’t necessarily reflect anyone else’s opinions or ideas. I also haven’t vetted or previewed it with anyone.

There are three main criticisms I’ve seen of the Social Web Foundation:

Meta is a partner It’s called The Social Web Foundation but is focused on ActivityPub, ignoring AT Protocol, Nostr, and other decentralized social web protocols that are emerging elsewhere It’s focused on substantially growing the Fediverse, which is not something everyone wants

I believe they’re interrelated, and that these differences can be overcome.

Meta enters the chat

Perhaps the biggest red flag to critics is Meta’s presence as one of the SWF’s thirteen launch partners. Many consider it to be an extremely negative force on the web. Its presence is certainly divisive. I’ve been a critic of its Facebook product in particular since its inception: a company that imposes its centralized view of the world on the communications of its billions of users, and in the process has caused real harms.

Those harms include potential mental health and social media addiction effects in teenagers, failing to protect LGBTQ users, and more — up to and including enabling a genocide.

The last claim might seem outlandish, but it’s real. As Harvard Law School’s Systemic Justice Project pointed out:

Scholars, reporters, and United Nations investigators agree that the social media giant played a role in an explosion of ethnic conflict in 2017 that led to the death and displacement of hundreds of thousands Rohingya Muslims in Northern Myanmar.

Given this, the argument goes, why would anyone — particularly an organization trying to build the future of the social web — even consider working with Meta? Doesn’t its presence as a partner taint the work of the Foundation?

As the writer, researcher, and community lead Erin Kissane has pointed out:

I think it’s unwise to assume that an organization that has demonstrably and continuously made antisocial and sometimes deadly choices on behalf of billions of human beings and allowed its products to be weaponized by covert state-level operations behind multiple genocides and hundreds (thousands? tens of thousands?) of smaller persecutions, all while ducking meaningful oversight, lying about what they do and know, and treating their core extraction machines as fait-accompli inevitabilities that mustn’t be governed except in patently ineffective ways will be a good citizen after adopting a new, interoperable technical structure.

These profoundly negative impacts are possible because it is one of the most prominent — potentially the most prominent — platform owner on the internet. Around four billion users use one of Meta’s products every month; that’s half all the humans on earth, or around 75% of all the people in the world aged 15 or older. Arguably no platform should ever be allowed to become this big or influential (can any government claim to have this level of reach or insight into this many people?). Still, at least for now, here it is.

For many people, Meta is the internet. This clearly doesn’t absolve aiding a genocide, throwing an election, or thwarting academic research, but it also makes Meta a platform owner that’s hard to ignore.

Meta sits in a position of influence over the social web. Threads, its fairly recent Twitter-like platform, is rolling out support for the ActivityPub standard that underlies the Fediverse, so it is poised to also be influential there. Once Threads supports the Fediverse bidirectionally, it will easily be the largest social platform on the network. It will consequently have an enormous amount of influence on how the network evolves, regardless of its participation in the Social Web Foundation.

What is a successful Fediverse?

Meta’s involvement and potential dominance inevitably raises the question: What kind of future do we want for the Fediverse? Whether we focus on technical interoperability or grassroots social activism, the answer to this question will shape how we approach growth, inclusivity, and the role of large corporations in the decentralized web.

If you see the Fediverse as a way to interoperate between social networks, such that a user on one platform can communicate with a user on another, you might welcome a large tech company supporting the standard (a bit like one might have welcomed a company to standards-based HTML a generation ago). If, on the other hand, you see the Fediverse as an antidote to technology corporations or a movement that is more about a collaborative grassroots movement than pure technical interoperability — a sort of work of activism — you might be quite alarmed.

These mindsets are analogous to Evan Prodromou’s Big Fedi / Small Fedi dichotomy, but I’d like to apply a slightly different lens.

If your model of the Fediverse is an interoperable standard that underpins all social networks: All parties should focus on a single technical standard in order ensure everyone can interoperate and the network can grow. The focus should be on onboarding, education, and developer experience. Growth is paramount. The goal is to bring the whole world in. Having the creator of the biggest social network join is an opportunity. The end state is likely a handful of very large social networks, followed by a significant long tail of small ones.

For ease of reference, let’s call this the growth Fediverse.

If your model of the Fediverse is a social movement intentionally set apart from corporate social media: A plurality of underlying protocols is allowable and maybe even desirable: the important thing is the support of grassroots communities outside the usual bounds of the tech industry. The focus should be on equity, community dynamics, relationships, and movement-building in service of community. Preserving the values of the existing community is paramount. The rest of the world can stay away; there’s no need for growth. The presence of the largest corporate social media vendor is inherently a threat. The end state is likely a collection of small, interoperable communities united by their desire for an alternative to “big tech”.

Let’s call this one the movement Fediverse.

Both models of the Fediverse clearly exist. I’m hardly the first to have discussed them, but the Social Web Foundation announcement has re-ignited the conversation.

Very clearly, the Foundation is closer to the first model than the second. As such, people who don’t care for that model have accused it of being an agent of oligarchy; of doing harm by partnering with Meta; of using the term “social web” while focusing solely on ActivityPub.

A false binary

The thing is, the lines between these two paths are blurry. It’s not necessarily an either-or. The priority for the first is growth of the network and a large, interoperable social web; the priority of the second is small, pro-social communities that exist outside of usual tech industry dynamics. Someone might well feel that the way to get to small, pro-social communities is as a by-product of interoperability, just as not everything on the web itself is corporate even though partners to the W3C body that defines web standards include Google and Amazon.

Some of the things that the movement Fediverse wants are intrinsically important to the growth Fediverse. You can’t grow a giant social network without caring about community safety, for example; over the two years since he acquired Twitter, Elon Musk has ably demonstrated that most users don’t want to stick around on a platform where they don’t feel safe. Community standards are therefore very important to any network that seeks to grow and retain users. Usability and accessibility are similarly vital: what use is a movement that is exclusionary to less-technical people, or, say, the visually-impaired? Any healthy network needs to support diverse voices and ensure that those authors are welcome. The list of shared values goes on.

But there are also undeniable differences. Hanging the needs of an anti-corporate social movement on a technology is a big ask. I’m not critical of the values of the people who do — I largely share them — but I don’t think you can reasonably expect everybody involved in a technology to have the same ideals.

Like any community, the movement Fediverse also has areas where it, too, could benefit from introspection and growth in order to live up to its own values. Some parts of the community have struggled with inclusivity, particularly when onboarding marginalized users who wished to discuss systemic injustice openly. As Marcia X recounted in Logic(s):

What took me aback regarding the fediverse is that my networks were mostly “leftists” and self-proclaimed radical thinkers regarding race, ableism, gender, patriarchy, sexuality, et cetera, and yet what I was being exposed to was a lot of naiveté or hostility for questioning whiteness as a basis for many people’s takes or approaches to these subject matters. And if I were to question or push back on their whiteness, I was often accused of being biased myself.

While many people in the movement are already working hard to address these issues, more can be done to ensure that all users feel safe, heard, and respected. In some cases, the movement Fediverse has fallen short when it comes to fully supporting the lived experiences of new users, especially those from marginalized groups. However, there is clear potential — and growing momentum — to improve this. By continuing to evolve and actively listen to new voices, the movement Fediverse can better embody the values of inclusivity and social justice that it stands for. But there is work to do.

In other words, it’s important to recognize that both groups have challenges to address. Each needs to continue working to ensure decisions are made inclusively, with an eye on the safety of users and the accessibility of communities. By recognizing these shared goals, there’s a real opportunity for mutual learning and growth.

Each has much to gain from each other. One doesn’t need to be a subscriber to the growth Fediverse to enjoy gains from user experience research, technology onboarding, and outreach conducted there. Similarly, one doesn’t need to subscribe to the ideals of the movement Fediverse to feel the benefit of their community dynamics and social goals. In fact, there may be a productive tension between the two that keeps each of their worst impulses in check. One might consider the movement Fediverse to be akin to a labor movement: a way for users to organize and advocate for stronger, safer, and more progressive community design. In turn, the growth Fediverse could be a check against becoming too insular and leaving the rest of the world out in the cold.

While the movement and growth Fediverse may have differing approaches, both share a commitment to user safety, inclusivity, and decentralization. The question is not whether these goals are shared, but how best to achieve them.

Moving forward

Just as unions create productive tensions in businesses that create better working conditions and higher productivity, I think the discussion between the movement Fediverse and the growth Fediverse has the potential to push the open social web further than might otherwise have been possible.

The checks and balances produced by an open debate between the two approaches are particularly useful when considering partners like Meta. The productive tension between these two visions could ensure that while larger platforms like Meta are held accountable, the values of grassroots communities — safety, inclusivity, and equity, for example — are not sacrificed in the pursuit of growth.

It’s not a foregone conclusion that Meta will dominate how the Social Web Foundation is run, but it’s also not a foregone conclusion that it won’t. The Social Web Foundation clearly states in its mission statement (emphasis mine):

A Fediverse that is controlled only by one company isn’t really a Fediverse at all. We think a productive, creative and healthy Fediverse needs multiple providers, none of whom dominate the space.

The goal is a multipolar federated social web. I think a large part of the solution is not to say this, but to show it: conduct meetings and make decisions with as much transparency as possible, so as to prove that Meta (and any other partner) is not dominant. By structurally providing as much sunlight as possible, allowing feedback and comment, and repeatedly demonstrating that this feedback is being considered and acted on where appropriate, both the potential harms and concerns from the movement Fediverse community can be reduced. Just as source code that is open to scrutiny is auditable and verifiable, decision-making process that are open to sunlight can be held accountable. Public meeting notes, decision documents, and so on, all help to support accountability.

In any event, the Social Web Foundation doesn’t need to be the foundation to cover all views of what the Fediverse should be. It’s a foundation that is going to try and do great work to expand the Fediverse. From its mission statement:

We believe that increased use of the Fediverse has the potential to make all of our online social experiences better, as well as to create lots of new opportunities for creation and self-expression. So we’re committed to growing the number of people using the Fediverse.

As Evan Prodromou said in that SocialHub thread abut people who don’t feel the Foundation represents them:

We want a united social web, using a single protocol for internetwork communication. I’d compare email, where proprietary LAN email protocols like Microsoft Exchange are gatewayed into the formal standard protocol SMTP. […] The SWF is not mandatory. People who want to do other things for the Fediverse should definitely do so. But I do want to extend the invitation for people who are interested to reach out.

This doesn’t have to be one size fits all. It’s worth considering what organizing more concretely for the movement Fediverse looks like, and how it might intersect and act as a check on the growth Fediverse.

It’s understandable that some in the movement Fediverse feel uncomfortable with large corporate platforms, particularly those with a history of past harms, joining the network. However, engaging with these platforms — rather than dismissing their involvement outright — may offer a unique opportunity to influence their practices and ensure they align with the values of the community. Constructive engagement with Meta and other large platforms could offer a unique opportunity for the movement Fediverse to influence how these entities engage with the broader social web, ensuring they uphold the values of safety, inclusivity, and equity.

Likewise, ignoring the concerns of the movement Fediverse is not wise: these are valid ideas rooted in real experiences. The tech industry carries real systemic inequalities that go all the way back to its origins in military funding. Addressing those inequities is a prerequisite to the web reaching its potential as a way for everyone in the world to connect and learn from each other. Companies like Meta, as I’ve explained at length above, have committed real harms as a byproduct of their priorities, business models, and funding partners. Grassroots communities that practice intentionality, activism, mutual aid, and radical equity have a lot to offer, and in many ways are models for how the world should be.

The movement Fediverse’s emphasis on mutual aid, radical equity, and intentionality offers invaluable lessons for how the larger Fediverse — and even corporate actors — could operate. Practices like community-driven moderation, transparent governance, and prioritizing marginalized voices could help ensure that the Fediverse grows without losing its soul.

Each group is approaching the problem in good faith. In the end, it’s up to all of us to ensure that the future of the web remains decentralized, inclusive, and safe. We must continue to engage, advocate, and, most importantly, listen to one another as we navigate and build this space together. The Fediverse is made of pluralities: of implementations, communities, vendors, and visions of the future. That’s at the heart of its beauty and its opportunity. The software interoperates; so should we.


Jon Udell

Geothermal power in the North Bay

I was aware of The Geysers, a geothermal field about 35 miles north of my home in Santa Rosa, but I never gave it much thought until my first bike ride through the area. Then I learned a number of interesting things. It’s the world’s largest geothermal field, producing more than 700 megawatts. It accounts … Continue reading Geothermal power in the North Bay

I was aware of The Geysers, a geothermal field about 35 miles north of my home in Santa Rosa, but I never gave it much thought until my first bike ride through the area. Then I learned a number of interesting things.

It’s the world’s largest geothermal field, producing more than 700 megawatts.

It accounts for 20% of California’s renewable energy.

The naturally-occurring steam was used up almost 30 years ago, and steam is now recharged by pumping in 11 million gallons of sewage effluent daily, through a 42-mile pipeline, from the Santa Rosa plain.

That daily recharge is implicated in the region’s frequent small earthquakes. (But nobody seems too worried about that, and maybe it’s a good thing? Many small better than one big?)

An article in today’s paper reports that AB-1359, signed last week by governor Gavin Newsom, paves the way for new geothermal development in the region that could add 600 megawatts of geothermal production.

How much electric power is that? I like to use WolframAlpha for quick and rough comparisons.

So, 2/3 of a nuke plant. 4/5 of a coal-fired power plant. These kinds of comparisons help me contextualize so many quantitative aspects of our lives. They’re the primary reason I visit WolframAlpha. I wish journalists would use it for that purpose.


Simon Willison

Wikidata is a Giant Crosswalk File

Wikidata is a Giant Crosswalk File Drew Breunig shows how to take the 140GB Wikidata JSON export, use sed 's/,$//' to convert it to newline-delimited JSON, then use DuckDB to run queries and extract external identifiers, including a query that pulls out 500MB of latitude and longitude points. Tags: wikipedia, drew-breunig, duckdb, json

Wikidata is a Giant Crosswalk File

Drew Breunig shows how to take the 140GB Wikidata JSON export, use sed 's/,$//' to convert it to newline-delimited JSON, then use DuckDB to run queries and extract external identifiers, including a query that pulls out 500MB of latitude and longitude points.

Tags: wikipedia, drew-breunig, duckdb, json


John Philpin : Lifestream

No link. Listened to New York Radio Hour - specifically Davi

No link. Listened to New York Radio Hour - specifically David Remnick talking to Newt Gingrich. Never have been a fan of Newt - but I kinda like to check in on my assumptions from time to time. Hah - still the same, now proudly sporting an armband of hypocrisy.

No link. Listened to New York Radio Hour - specifically David Remnick talking to Newt Gingrich.

Never have been a fan of Newt - but I kinda like to check in on my assumptions from time to time.

Hah - still the same, now proudly sporting an armband of hypocrisy.


The Union, 2024 - ★★★

I dunno .. Marky, Halle AND J.K. .. great cast .. so you expect a bit more than average? Don’t you? That said the choreography of chasing the case down the side of a building was well done. But if that’s the peak .. you wonder why I gave it 3 stars. Don’t you?

I dunno .. Marky, Halle AND J.K. .. great cast .. so you expect a bit more than average? Don’t you? That said the choreography of chasing the case down the side of a building was well done. But if that’s the peak .. you wonder why I gave it 3 stars. Don’t you?


Rebel Ridge, 2024 - ★★★★

Three and a half stars … rounding up to 4 .. just for the audacity to remake Rambo and not give Sky any credit. Seriously .. not bad at all.

Three and a half stars … rounding up to 4 .. just for the audacity to remake Rambo and not give Sky any credit. Seriously .. not bad at all.


Rebel Ridge, 2024

Watched on Saturday October 5, 2024.

Watched on Saturday October 5, 2024.


IdM Laboratory

Kim Cameron Awardの受賞者によるIdentiverseへの参加レポート

こんにちは、富士榮です。 先日お知らせしたVittorio Bertocciアワードと並行してDIAF(Digital Identity Advancement Foundation)が提供する個人向けの世界2大アイデンティティ・アワードであるKim Cameronアワードの受賞者であるMatthew SpenceがIdentiverse 2024への参加レポートを書いています。 https://digitalidadvancement.org/news/2024-kim-cameron-awardee-reflections-matthew-spence/ DIAFではアワード受賞者にIdentiverseやIIW、EICなどのカンファレンスへの参加をサポートしており、有能でやる気があっても費用面で課題がある若手などへのスポンサーをしています。 ダイバーシティを確保のために

こんにちは、富士榮です。

先日お知らせしたVittorio Bertocciアワードと並行してDIAF(Digital Identity Advancement Foundation)が提供する個人向けの世界2大アイデンティティ・アワードであるKim Cameronアワードの受賞者であるMatthew SpenceがIdentiverse 2024への参加レポートを書いています。


https://digitalidadvancement.org/news/2024-kim-cameron-awardee-reflections-matthew-spence/

DIAFではアワード受賞者にIdentiverseやIIW、EICなどのカンファレンスへの参加をサポートしており、有能でやる気があっても費用面で課題がある若手などへのスポンサーをしています。

ダイバーシティを確保のためにはこのような取り組みは非常に有用ですね。日本でも何かやれないかなぁ、、、と思いますが、まずは日本からもDIAFのアワードにApplyしてみる方が出てくることに期待です。






Friday, 04. October 2024

Simon Willison

Database Remote-Copy Tool For SQLite (draft)

Database Remote-Copy Tool For SQLite (draft) Neat new SQLite utilities often show up in branches of the SQLite repository. Here's a new one from last month: sqlite3-rsync, providing tools for efficiently creating and updating copies of WAL-mode SQLite databases on either the same machine or across remote machines via SSH. The way it works is neat, inspired by rsync (hence the tool's name):

Database Remote-Copy Tool For SQLite (draft)

Neat new SQLite utilities often show up in branches of the SQLite repository. Here's a new one from last month: sqlite3-rsync, providing tools for efficiently creating and updating copies of WAL-mode SQLite databases on either the same machine or across remote machines via SSH.

The way it works is neat, inspired by rsync (hence the tool's name):

The protocol is for the replica to send a cryptographic hash of each of its pages over to the origin side, then the origin sends back the complete content of any page for which the hash does not match.

SQLite's default page size is 4096 bytes and a hash is 20 bytes, so if nothing has changed then the client will transmit 0.5% of the database size in hashes and get nothing back in return.

The tool takes full advantage of SQLite's WAL mode - when you run it you'll get an exact snapshot of the database state as it existed at the moment the copy was initiated, even if the source database continues to apply changes.

I wrote up a TIL on how to compile it - short version:

cd /tmp git clone https://github.com/sqlite/sqlite.git cd sqlite git checkout sqlite3-rsync ./configure make sqlite3.c cd tool gcc -o sqlite3-rsync sqlite3-rsync.c ../sqlite3.c -DSQLITE_ENABLE_DBPAGE_VTAB ./sqlite3-rsync --help

Update: It turns out you can now just run ./configure && make sqlite-rsync in the root checkout.

Something I’ve worried about in the past is that if I want to make a snapshot backup of a SQLite database I need enough additional free disk space to entirely duplicate the current database first (using the backup mechanism or VACUUM INTO). This tool fixes that - I don’t need any extra disk space at all, since the pages that have been updated will be transmitted directly over the wire in 4096 byte chunks.

I tried feeding the 1800 lines of C through OpenAI’s o1-preview with the prompt “Explain the protocol over SSH part of this” and got a pretty great high level explanation - markdown copy here.

Via lobste.rs

Tags: sqlite, c, o1


Hybrid full-text search and vector search with SQLite

Hybrid full-text search and vector search with SQLite As part of Alex’s work on his sqlite-vec SQLite extension - adding fast vector lookups to SQLite - he’s been investigating hybrid search, where search results from both vector similarity and traditional full-text search are combined together. The most promising approach looks to be Reciprocal Rank Fusion, which combines the top ranked items

Hybrid full-text search and vector search with SQLite

As part of Alex’s work on his sqlite-vec SQLite extension - adding fast vector lookups to SQLite - he’s been investigating hybrid search, where search results from both vector similarity and traditional full-text search are combined together.

The most promising approach looks to be Reciprocal Rank Fusion, which combines the top ranked items from both approaches. Here’s Alex’s SQL query:

-- the sqlite-vec KNN vector search results with vec_matches as ( select article_id, row_number() over (order by distance) as rank_number, distance from vec_articles where headline_embedding match lembed(:query) and k = :k ), -- the FTS5 search results fts_matches as ( select rowid, row_number() over (order by rank) as rank_number, rank as score from fts_articles where headline match :query limit :k ), -- combine FTS5 + vector search results with RRF final as ( select articles.id, articles.headline, vec_matches.rank_number as vec_rank, fts_matches.rank_number as fts_rank, -- RRF algorithm ( coalesce(1.0 / (:rrf_k + fts_matches.rank_number), 0.0) * :weight_fts + coalesce(1.0 / (:rrf_k + vec_matches.rank_number), 0.0) * :weight_vec ) as combined_rank, vec_matches.distance as vec_distance, fts_matches.score as fts_score from fts_matches full outer join vec_matches on vec_matches.article_id = fts_matches.rowid join articles on articles.rowid = coalesce(fts_matches.rowid, vec_matches.article_id) order by combined_rank desc ) select * from final;

I’ve been puzzled in the past over how to best do that because the distance scores from vector similarity and the relevance scores from FTS are meaningless in comparison to each other. RRF doesn’t even attempt to compare them - it uses them purely for row_number() ranking within each set and combines the results based on that.

Tags: embeddings, sql, vector-search, sqlite, search, alex-garcia, full-text-search, rag


Ben Werdmüller

Getting my daily news from a dot matrix printer

[Andrew Schmelyun] Following my piece about reading the news on paper, I came across this post from Anrew Schmelyun: "I recently purchased a dot matrix printer from eBay, and thought it would be a great excuse to have a custom "front page" printed out and ready for me each day. So, that's what I built!" What a neat idea: he's called a few APIs (the New York Times, Reddit,

[Andrew Schmelyun]

Following my piece about reading the news on paper, I came across this post from Anrew Schmelyun:

"I recently purchased a dot matrix printer from eBay, and thought it would be a great excuse to have a custom "front page" printed out and ready for me each day. So, that's what I built!"

What a neat idea: he's called a few APIs (the New York Times, Reddit, Open-Meteo, and so on), installed it to run on a Raspberry Pi, and connected it to an old-school dot matrix printer to create a kind of Telex newspaper each morning,

I'd thought about doing this with an e-ink display, but honestly, why not just print it out?

I think I would want to pick some different news sources (the NYT is no longer my go-to) and leave out Reddit in favor of links that my contacts had shared on, say, Mastodon, but this is really fun. I might try and put together something similar, albeit with my existing laser printer rather than a dot matrix setup.

#Media

[Link]


John Philpin : Lifestream

More on Kris I ’never’ do this - so take this as the excep

More on Kris I ’never’ do this - so take this as the exception that proves the rule. We all know that 🖇️ Kris K recently passed - on Maui My friend Randall wrote to me (these extracts reproduced with his permission) Such a sad loss of a hero whom I actually knew …. Sarah Teed and I had the privilege of a hanging w/ him and John Prine when we played the Castle theater many years ago and I ope

More on Kris

I ’never’ do this - so take this as the exception that proves the rule. We all know that 🖇️ Kris K recently passed - on Maui

My friend Randall wrote to me (these extracts reproduced with his permission)

Such a sad loss of a hero whom I actually knew …. Sarah Teed and I had the privilege of a hanging w/ him and John Prine when we played the Castle theater many years ago and I opened for him at the MACC many times sharing moments with him and his wife Lisa.

Randall went on and shared some Kris wisdom …

”He believed that songwriting is a spiritual communion of mind, body, and soul, and he believed that William Blake was correct in asserting that anyone divinely ordered for spiritual communion but buries his talent will be pursued by sorrow and desperation through life and by shame and confusion for eternity.

” (Blake) is telling you that you’ll be miserable if you don’t do what you’re supposed to do," Kristofferson said in the Ken Burns' documentary ‘Country Music’.”


Argylle, 2024 - ★★½

Cute. A good watch on a lazy sunny afternoon.

Cute. A good watch on a lazy sunny afternoon.


Anyone successfully — or not — following Threads or BlueSky

Anyone successfully — or not — following Threads or BlueSky in Microblog?

Anyone successfully — or not — following Threads or BlueSky in Microblog?

Thursday, 03. October 2024

Ben Werdmüller

Is There Still a Place for Print in the Future of Media?

I think there’s more work to be done to explore print as a modern product to support great writing and journalism. Lots has been said about its death — but comparatively little about its potential to live on in new forms. I think print has a lot of life left in it: particularly if we overcome the idea of preserving the exact form it’s taken in the past and consider what a more modern, reconsi

I think there’s more work to be done to explore print as a modern product to support great writing and journalism. Lots has been said about its death — but comparatively little about its potential to live on in new forms.

I think print has a lot of life left in it: particularly if we overcome the idea of preserving the exact form it’s taken in the past and consider what a more modern, reconsidered print product might look like.

There’s a lot to be said for reading on paper. One of my more recent indulgences has been a daily subscription to The Financial Times, which on weekdays is a sober paper that reports the news fairly objectively. On weekends it’s a different beast: in particular it includes a magazine pull-out called How to Spend It that is apparently aimed at the worst people on earth and is generally indistinguishable from satire.

The Financial Times has been publishing since 1888, but some endeavors are much newer. Speaking of indistinguishable from satire, I subscribed to The Onion’s print edition, now it has been bought from its private equity owner. It’s been fun seeing it adopt similar membership strategies to other, more “serious” publications. Most exciting among those is its resumed print edition, which is an old idea given a new spin:

“I think for the same reason that 18-year-old kids are buying Taylor Swift on vinyl,” Jordan LaFlure, The Onion’s executive editor also told the Times, “we can introduce those same kids to the notion that a print publication is a much richer way to consume media.”

It’s not obvious to me that a similar strategy couldn’t work for other publications — or even as a digest of independent publications that work together. Would I buy a subscription to a paper edition of independent journalism across various topics? Absolutely I would, and I don’t think I’m alone. Think of it as a lo-fi RSS reader or a retro Apple News: articles I care about from around the web in a form factor that looks more like The New Yorker (or The Onion).

This product could take several forms. It could combine an algorithmic component — here are the writers I care about — with a more human-driven curatorial component from editors who want to highlight interesting journalism from sources the reader might not have encountered yet. Or it could be a purely editorial product with no algorithmic component: one size fits all, for every reader. Or you could subscribe to personalized editions with different human editors who get a cut of subscriptions for putting it all together. (A monthly tech periodical organized by Casey Newton or Molly White? Take my money.)

Publications like ProPublica (my current employer) and The 19th (which I’ve worked for previously) produce content that is more long-form journalism than breaking news, which is highly suitable for reading in a collected periodical. They also make their content freely available via a Creative Commons license, meaning that, technically, anyone could put this together. But it would clearly be better in partnership with newsrooms, with revenue and subscriber information flowing back to them in exchange for letting their journalism be included.

This isn’t a traditional startup: it’s hard for me to see how this product would enjoy the rapid growth or high valuations which justify venture investment. But it’s potentially a really interesting small business. If the numbers work out, it could also potentially be a fascinating add-on product for a service like Medium. There’s user and market research to be done here, but it’s possible that the decline of legacy print products does not necessarily mean that new print products won’t be successful.

The act of reading on paper feels different to sitting in front of a screen. Maybe I’m getting old, but I like sitting at the dining room table, leafing through print. It is an old school product that is a little like vinyl, but it also feels like I’m using my brain a bit differently. I’d love to do more of it. In a world where everything is digital, maybe a thoughtfully curated print product could be exactly what we need to slow down and engage more deeply. Or maybe not, but I think it would be cool.

I’d love to hear what you think. Am I alone in preferring an offline, analogue, tactile reading experience? Is there something here, or is the future of media entirely, irrevocably digital?


IdM Laboratory

OpenID Connect for Identity Assuranceの仕様が承認されました

こんにちは、富士榮です。 Great newsです。 先日より投票が開始されていたOpenID Connect for Identity Assuranceの仕様が最終化、承認されました。 投票のお知らせ) https://idmlab.eidentity.jp/2024/09/openid-connect-for-identity-assurance.html 最終化に関する公式アナウンス) https://openid.net/final-openid-connect-for-identity-assurance-specifications-approved/ 今回承認された仕様は以下のとおりです。 OpenID Identity Assurance Schema Definition 1.0 - https://openid.net/s
こんにちは、富士榮です。
Great newsです。 先日より投票が開始されていたOpenID Connect for Identity Assuranceの仕様が最終化、承認されました。 投票のお知らせ) https://idmlab.eidentity.jp/2024/09/openid-connect-for-identity-assurance.html
最終化に関する公式アナウンス) https://openid.net/final-openid-connect-for-identity-assurance-specifications-approved/

今回承認された仕様は以下のとおりです。 OpenID Identity Assurance Schema Definition 1.0 - https://openid.net/specs/openid-ida-verified-claims-1_0-final.html OpenID Connect for Identity Assurance Claims Registration 1.0 - https://openid.net/specs/openid-connect-4-ida-claims-1_0-final.html OpenID Connect for Identity Assurance 1.0 - https://openid.net/specs/openid-connect-4-identity-assurance-1_0-final.html
皆さん、使っていきましょう。

Simon Willison

Gemini 1.5 Flash-8B is now production ready

Gemini 1.5 Flash-8B is now production ready Gemini 1.5 Flash-8B is "a smaller and faster variant of 1.5 Flash" - and is now released to production, at half the price of the 1.5 Flash model. It's really, really cheap: $0.0375 per 1 million input tokens on prompts <128K $0.15 per 1 million output tokens on prompts <128K $0.01 per 1 million input tokens on cached prompts <128K

Gemini 1.5 Flash-8B is now production ready

Gemini 1.5 Flash-8B is "a smaller and faster variant of 1.5 Flash" - and is now released to production, at half the price of the 1.5 Flash model.

It's really, really cheap:

$0.0375 per 1 million input tokens on prompts <128K $0.15 per 1 million output tokens on prompts <128K $0.01 per 1 million input tokens on cached prompts <128K

Prices are doubled for prompts longer than 128K.

I believe images are still charged at a flat rate of 258 tokens, which I think means a single non-cached image with Flash should cost 0.00097 cents - a number so tiny I'm doubting if I got the calculation right.

OpenAI's cheapest model remains GPT-4o mini, at $0.15/1M input - though that drops to half of that for reused prompt prefixes thanks to their new prompt caching feature (or by half if you use batches, though those can’t be combined with OpenAI prompt caching. Gemini also offer half-off for batched requests).

Anthropic's cheapest model is still Claude 3 Haiku at $0.25/M, though that drops to $0.03/M for cached tokens (if you configure them correctly).

I've released llm-gemini 0.2 with support for the new model:

llm install -U llm-gemini llm keys set gemini # Paste API key here llm -m gemini-1.5-flash-8b-latest "say hi"

Via @OfficialLoganK

Tags: vision-llms, gemini, anthropic, openai, ai, llms, google, generative-ai, llm


Quoting Cal Newport

At first, I struggled to understand why anyone would want to write this way. My dialogue with ChatGPT was frustratingly meandering, as though I were excavating an essay instead of crafting one. But, when I thought about the psychological experience of writing, I began to see the value of the tool. ChatGPT was not generating professional prose all at once, but it was providing starting points: int

At first, I struggled to understand why anyone would want to write this way. My dialogue with ChatGPT was frustratingly meandering, as though I were excavating an essay instead of crafting one. But, when I thought about the psychological experience of writing, I began to see the value of the tool. ChatGPT was not generating professional prose all at once, but it was providing starting points: interesting research ideas to explore; mediocre paragraphs that might, with sufficient editing, become usable. For all its inefficiencies, this indirect approach did feel easier than staring at a blank page; “talking” to the chatbot about the article was more fun than toiling in quiet isolation. In the long run, I wasn’t saving time: I still needed to look up facts and write sentences in my own voice. But my exchanges seemed to reduce the maximum mental effort demanded of me.

Cal Newport

Tags: writing, generative-ai, chatgpt, ai, llms


Doc Searls Weblog

Think Globally, Eat Here

Fifteenth in the News Commons series. This semester’s Beyond the Web salon series for the Ostrom Workshop and Hamilton Lugar School at Indiana University is themed Think Globally, Eat Here—Small Solutions for Big Tech Problems. I will give the opening talk, about the News Commons (subject of fourteen prior posts here) at noon (Eastern) next […]

Fifteenth in the News Commons series.

This semester’s Beyond the Web salon series for the Ostrom Workshop and Hamilton Lugar School at Indiana University is themed Think Globally, Eat Here—Small Solutions for Big Tech Problems. I will give the opening talk, about the News Commons (subject of fourteen prior posts here) at noon (Eastern) next Tuesday, October 10. If you’re in town, please attend in person. If not, join us by Zoom. Do that here.

Our plan is to prototype and prove locally what can apply globally for local news, starting with what Columbia Journalism Review called “news deserts” back in 2017—a label that has since caught on. There are many efforts toward seeding and watering these deserts, most prominently Press Forward, which is devoting $500 million to that challenge.

Bloomington is advantaged by not being one of those deserts, and instead having a talented pool of local journals, journalists, and organizations—including its legacy newspaper—all doing good work that could still be improved by putting to use some of the innovations I’ll be talking about, and by working together as a commons.

So join the conversation. I look forward to seeing you in the room or on the wall (because one whole wall is our Zoom screen).

 


Simon Willison

Announcing FLUX1.1 [pro] and the BFL API

Announcing FLUX1.1 [pro] and the BFL API FLUX is the image generation model family from Black Forest Labs, a startup founded by members of the team that previously created Stable Diffusion. Released today, FLUX1.1 [pro] continues the general trend of AI models getting both better and more efficient: FLUX1.1 [pro] provides six times faster generation than its predecessor FLUX.1 [pro] while a

Announcing FLUX1.1 [pro] and the BFL API

FLUX is the image generation model family from Black Forest Labs, a startup founded by members of the team that previously created Stable Diffusion.

Released today, FLUX1.1 [pro] continues the general trend of AI models getting both better and more efficient:

FLUX1.1 [pro] provides six times faster generation than its predecessor FLUX.1 [pro] while also improving image quality, prompt adherence, and diversity.

Black Forest Labs appear to have settled on a potentially workable business model: their smallest, fastest model FLUX.1 [schnell] is Apache 2 licensed. The next step up is FLUX.1 [dev] which is open weights for non-commercial use only. The [pro] models are closed weights, made available exclusively through their API or partnerships with other API providers.

I tried the new 1.1 model out using black-forest-labs/flux-1.1-pro on Replicate just now. Here's my prompt:

Photograph of a Faberge egg representing the California coast. It should be decorated with ornate pelicans and sea lions and a humpback whale.

The FLUX models have a reputation for being really good at following complex prompts. In this case I wanted the sea lions to appear in the egg design rather than looking at the egg from the beach, but I imagine I could get better results if I continued to iterate on my prompt.

The FLUX models are also better at applying text than any other image models I've tried myself.

Via Hacker News

Tags: stable-diffusion, ai, generative-ai, replicate


The Pragmatic Engineer

The Pulse #109: Open source business model struggles at Wordpress

Also: OpenAI’s biggest-ever fundraise even as key people keep quitting; why executive recruiters ignore tech professionals, and more

Today, we cover:

Industry pulse. Microsoft won’t follow Amazon in 5-day RTO, Cloudflare auto-mitigates world-record DDoS attack, California nearly regulates AI companies, Revolut starts selling performance management framework as a product, and more.

Open source business model struggles: Wordpress. Automattic, creator of Wordpress, is being sued by one of the largest WordPress hosting providers. The conflict fits into a trend of billion-dollar companies struggling to effectively monetize open source, and are changing tactics to limit their competition and increase their revenue.

OpenAI: biggest-ever fundraise, as key people keep quitting. OpenAI raised a $6.6B in funding on a $157B valuation, making it the largest fundraiser ever. And yet, cofounders and executives continue to depart. This time it’s the CTO, Chief Research Officer, the VP of Research. Also, OpenAI will most likely finally become a for-profit.

Why executive recruiters ignore tech professionals. Connecting with executive recruiters is a helpful strategy for engineering leaders. Senior product director shares key reasons these outreaches fail, and what you can do to improve chances.

1. Industry pulse Microsoft won’t follow Amazon in 5-day RTO

In Seattle, the two largest tech employers are Microsoft and Amazon. Both companies are willing and able to match compensation of each other, so when Amazon announced two weeks ago that it’s mandating a strict 5-days-from-office policy from 1 Jan 2025, the question was whether Microsoft would follow.

The company has internally confirmed it will not, as per Business Insider, Microsoft EVP Scott Guthrie told employees that Microsoft has no plans to tighten remote work policies. This means Microsoft workers can expect to spend 2-3 days per week in the office as a hybrid arrangement. I expect experienced Amazon software engineers who are unhappy about their workplace’s 5-days-per-week policy, and who live within commuting distance of a Microsoft office, to start applying for open positions at Microsoft.

Cloudflare auto-mitigates world-record DDoS attack

Read more


Simon Willison

Ask HN: What happens to ".io" TLD after UK gives back the Chagos Islands?

Ask HN: What happens to ".io" TLD after UK gives back the Chagos Islands? This morning on the BBC: UK will give sovereignty of Chagos Islands to Mauritius. The Chagos Islands include the area that the UK calls the British Indian Ocean Territory. The .io ccTLD uses the ISO-3166 two-letter country code for that designation. As the owner of datasette.io the question of what happens to that ccTLD

Ask HN: What happens to ".io" TLD after UK gives back the Chagos Islands?

This morning on the BBC: UK will give sovereignty of Chagos Islands to Mauritius. The Chagos Islands include the area that the UK calls the British Indian Ocean Territory. The .io ccTLD uses the ISO-3166 two-letter country code for that designation.

As the owner of datasette.io the question of what happens to that ccTLD is suddenly very relevant to me.

This Hacker News conversation has some useful information. It sounds like there's a very real possibility that .io could be deleted after a few years notice - it's happened before, for ccTLDs such as .zr for Zaire (which renamed to Democratic Republic of the Congo in 1997, with .zr withdrawn in 2001) and .cs for Czechoslovakia, withdrawn in 1995.

Could .io change status to the same kind of TLD as .museum, unaffiliated with any particular geography? The convention is for two letter TLDs to exactly match ISO country codes, so that may not be an option.

Tags: dns, domains, hacker-news


John Philpin : Lifestream

📸 A couple o' friends hanging at the bus stop.

📸 A couple o' friends hanging at the bus stop.

📸 A couple o' friends hanging at the bus stop.


Just listened to Gilbert’s 2024 release - he’s 78 for goodne

Just listened to Gilbert’s 2024 release - he’s 78 for goodness sake. Not bad - but the classics are definitely not improved .. like the original 🔗 🎵 Nothing Rhymed .. always a place in my heart for this one. (I always liked his singles - but it was my sister that introduced me to the albums (my natural musical habitat.)

Just listened to Gilbert’s 2024 release - he’s 78 for goodness sake. Not bad - but the classics are definitely not improved .. like the original 🔗 🎵 Nothing Rhymed .. always a place in my heart for this one. (I always liked his singles - but it was my sister that introduced me to the albums (my natural musical habitat.)

Wednesday, 02. October 2024

John Philpin : Lifestream

We Don't Need No Edukashun - Actually - We Do.

A Posterity Post I had cause to add a comment on a post that featured this image. It is funny - but I think it is funny for all the reasons that the commenters were not seeing …. A university is to educate .. not train. If you are wondering about the difference, ask yourself if you prefer your children to attend sex education classes or sex training classes. Sadly most institutions ha

A Posterity Post

I had cause to add a comment on a post that featured this image.

It is funny - but I think it is funny for all the reasons that the commenters were not seeing ….

A university is to educate .. not train. If you are wondering about the difference, ask yourself if you prefer your children to attend sex education classes or sex training classes.

Sadly most institutions have moved to thinking of themselves as trainers .. so they are ‘workplace ready’. Witness a presentation I sat in just last week where someone from an educational facility was telling us how they are currently revamping their syllabus so that their students could learn how best to work with Ai (sic) … see Stephen Fry’s latest missive.

I stood up and asked her which University she thought the people making Ai had taught them what they know.

(Apparently .. that’s different).

As for philosophy .. oh and let’s throw in ethics, humanities, liberal arts, critical thinking, hell, even english .. are (should be) all on the rise in education, because it is very clear that we need them …badly.

I get the picture .. funny.

But the humor doesn’t lie in the kid having it wrong and dreaming what might be .. it’s in the dad having it wrong and thinking the future is anything like the past.


Just for kicks I signed up for a month of 🥐 🔗 .. A butte

Just for kicks I signed up for a month of 🥐 🔗 .. A buttery smooth app for cross posting to Bluesky, Mastodon, and Threads. Big benefit for me is that I can add multiple accounts for the same service - useful as we start to push out The Future Found AND When I send I can choose which accounts it sends to. Next job (request?) - as I publish to MicroBlog or LinkedIn or Facebook or …. ha

Just for kicks I signed up for a month of 🥐 🔗 ..

A buttery smooth app for cross posting to Bluesky, Mastodon, and Threads.

Big benefit for me is that I can add multiple accounts for the same service - useful as we start to push out The Future Found

AND

When I send I can choose which accounts it sends to.

Next job (request?) - as I publish to MicroBlog or LinkedIn or Facebook or …. have that post appear in Croissant ready for me to decide where I want to cross post. #Maybe.


Simon Willison

OpenAI DevDay: Let’s build developer tools, not digital God

I had a fun time live blogging OpenAI DevDay yesterday - I’ve now shared notes about the live blogging system I threw other in a hurry on the day (with assistance from Claude and GPT-4o). Now that the smoke has settled a little, here are my impressions from the event. Compared to last year Prompt caching, aka the big price drop GPT-4o audio via the new WebSocket Realtime API Mod

I had a fun time live blogging OpenAI DevDay yesterday - I’ve now shared notes about the live blogging system I threw other in a hurry on the day (with assistance from Claude and GPT-4o). Now that the smoke has settled a little, here are my impressions from the event.

Compared to last year Prompt caching, aka the big price drop GPT-4o audio via the new WebSocket Realtime API Model distillation is fine-tuning made much easier Let’s build developer tools, not digital God Compared to last year

Comparison with the first DevDay in November 2023 are unavoidable. That event was much more keynote-driven: just in the keynote OpenAI released GPT-4 vision, and Assistants, and GPTs, and GPT-4 Turbo (with a massive price drop), and their text-to-speech API. It felt more like a launch-focused product event than something explicitly for developers.

This year was different. Media weren’t invited, there was no livestream, Sam Altman didn’t present the opening keynote (he was interviewed at the end of the day instead) and the new features, while impressive, were not as abundant.

Several features were released in the last few months that could have been saved for DevDay: GPT-4o mini and the o1 model family are two examples. I’m personally happy that OpenAI are shipping features like that as they become ready rather than holding them back for an event.

I’m a bit surprised they didn’t talk about Whisper Turbo at the conference though, released just the day before - especially since that’s one of the few pieces of technology they release under an open source (MIT) license.

This was clearly intended as an event by developers, for developers. If you don’t build software on top of OpenAI’s platform there wasn’t much to catch your attention here.

As someone who does build software on top of OpenAI, there was a ton of valuable and interesting stuff.

Prompt caching, aka the big price drop

I was hoping we might see a price drop, seeing as there’s an ongoing pricing war between Gemini, Anthropic and OpenAI. We got one in an interesting shape: a 50% discount on input tokens for prompts with a shared prefix.

This isn’t a new idea: both Google Gemini and Claude offer a form of prompt caching discount, if you configure them correctly and make smart decisions about when and how the cache should come into effect.

The difference here is that OpenAI apply the discount automatically:

API calls to supported models will automatically benefit from Prompt Caching on prompts longer than 1,024 tokens. The API caches the longest prefix of a prompt that has been previously computed, starting at 1,024 tokens and increasing in 128-token increments. If you reuse prompts with common prefixes, we will automatically apply the Prompt Caching discount without requiring you to make any changes to your API integration.

50% off repeated long prompts is a pretty significant price reduction!

Anthropic's Claude implementation saves more money: 90% off rather than 50% - but is significantly more work to put into play.

Gemini’s caching requires you to pay per hour to keep your cache warm which makes it extremely difficult to effectively build against in comparison to the other two.

It's worth noting that OpenAI are not the first company to offer automated caching discounts: DeepSeek have offered that through their API for a few months.

GPT-4o audio via the new WebSocket Realtime API

Absolutely the biggest announcement of the conference: the new Realtime API is effectively the API version of ChatGPT advanced voice mode, a user-facing feature that finally rolled out to everyone just a week ago.

This means we can finally tap directly into GPT-4o’s multimodal audio support: we can send audio directly into the model (without first transcribing it to text via something like Whisper), and we can have it directly return speech without needing to run a separate text-to-speech model.

The way they chose to expose this is interesting: it’s not (yet) part of their existing chat completions API, instead using an entirely new API pattern built around WebSockets.

They designed it like that because they wanted it to be as realtime as possible: the API lets you constantly stream audio and text in both directions, and even supports allowing users to speak over and interrupt the model!

So far the Realtime API supports text, audio and function call / tool usage - but doesn't (yet) support image input (I've been assured that's coming soon). The combination of audio and function calling is super exciting alone though - several of the demos at DevDay used these to build fun voice-driven interactive web applications.

I like this WebSocket-focused API design a lot. My only hesitation is that, since an API key is needed to open a WebSocket connection, actually running this in production involves spinning up an authenticating WebSocket proxy. I hope OpenAI can provide a less code-intensive way of solving this in the future.

Code they showed during the event demonstrated using the native browser WebSocket class directly, but I can't find those code examples online now. I hope they publish it soon. For the moment the best things to look at are the openai-realtime-api-beta and openai-realtime-console repositories.

The new playground/realtime debugging tool - the OpenAI playground for the Realtime API - is a lot of fun to try out too.

Model distillation is fine-tuning made much easier

The other big developer-facing announcements were around model distillation, which to be honest is more of a usability enhancement and minor rebranding of their existing fine-tuning features.

OpenAI have offered fine-tuning for a few years now, most recently against their GPT-4o and GPT-4o mini models. They’ve practically been begging people to try it out, offering generous free tiers in previous months:

Today [August 20th 2024] we’re launching fine-tuning for GPT-4o, one of the most requested features from developers. We are also offering 1M training tokens per day for free for every organization through September 23.

That free offer has now been extended. A footnote on the pricing page today:

Fine-tuning for GPT-4o and GPT-4o mini is free up to a daily token limit through October 31, 2024. For GPT-4o, each qualifying org gets up to 1M complimentary training tokens daily and any overage will be charged at the normal rate of $25.00/1M tokens. For GPT-4o mini, each qualifying org gets up to 2M complimentary training tokens daily and any overage will be charged at the normal rate of $3.00/1M tokens

The problem with fine-tuning is that it’s really hard to do effectively. I tried it a couple of years ago myself against GPT-3 - just to apply tags to my blog content - and got disappointing results which deterred me from spending more money iterating on the process.

To fine-tune a model effectively you need to gather a high quality set of examples and you need to construct a robust set of automated evaluations. These are some of the most challenging (and least well understood) problems in the whole nascent field of prompt engineering.

OpenAI’s solution is a bit of a rebrand. “Model distillation” is a form of fine-tuning where you effectively teach a smaller model how to do a task based on examples generated by a larger model. It’s a very effective technique. Meta recently boasted about how their impressive Llama 3.2 1B and 3B models were “taught” by their larger models:

[...] powerful teacher models can be leveraged to create smaller models that have improved performance. We used two methods—pruning and distillation—on the 1B and 3B models, making them the first highly capable lightweight Llama models that can fit on devices efficiently.

Yesterday OpenAI released two new features to help developers implement this pattern.

The first is stored completions. You can now pass a "store": true parameter to have OpenAI permanently store your prompt and its response in their backend, optionally with your own additional tags to help you filter the captured data later.

You can view your stored completions at platform.openai.com/chat-completions.

I’ve been doing effectively the same thing with my LLM command-line tool logging to a SQLite database for over a year now. It's a really productive pattern.

OpenAI pitch stored completions as a great way to collect a set of training data from their large models that you can later use to fine-tune (aka distill into) a smaller model.

The second, even more impactful feature, is evals. You can now define and run comprehensive prompt evaluations directly inside the OpenAI platform.

OpenAI’s new eval tool competes directly with a bunch of existing startups - I’m quite glad I didn’t invest much effort in this space myself!

The combination of evals and stored completions certainly seems like it should make the challenge of fine-tuning a custom model far more tractable.

The other fine-tuning announcement, greeted by applause in the room, was fine-tuning for images. This has always felt like one of the most obviously beneficial fine-tuning use-cases for me, since it’s much harder to get great image recognition results from sophisticated prompting alone.

From a strategic point of view this makes sense as well: it has become increasingly clear over the last year that many prompts are inherently transferable between models - it’s very easy to take an application with prompts designed for GPT-4o and switch it to Claude or Gemini or Llama with few if any changes required.

A fine-tuned model on the OpenAI platform is likely to be far more sticky.

Let’s build developer tools, not digital God

In the last session of the day I furiously live blogged the Fireside Chat between Sam Altman and Kevin Weil, trying to capture as much of what they were saying as possible.

A bunch of the questions were about AGI. I’m personally quite uninterested in AGI: it’s always felt a bit too much like science fiction for me. I want useful AI-driven tools that help me solve the problems I want to solve.

One point of frustration: Sam referenced OpenAI’s five-level framework a few times. I found several news stories (many paywalled - here's one that isn't) about it but I can’t find a definitive URL on an OpenAI site that explains what it is! This is why you should always Give people something to link to so they can talk about your features and ideas.

Both Sam and Kevin seemed to be leaning away from AGI as a term. From my live blog notes (which paraphrase what was said unless I use quotation marks):

Sam says they're trying to avoid the term now because it has become so over-loaded. Instead they think about their new five steps framework.

"I feel a little bit less certain on that" with respect to the idea that an AGI will make a new scientific discovery.

Kevin: "There used to be this idea of AGI as a binary thing [...] I don't think that's how think about it any more".

Sam: Most people looking back in history won't agree when AGI happened. The turing test wooshed past and nobody cared.

I for one found this very reassuring. The thing I want from OpenAI is more of what we got yesterday: I want platform tools that I can build unique software on top of which I colud not have built previously.

If the ongoing, well-documented internal turmoil at OpenAI from the last year is a result of the organization reprioritizing towards shipping useful, reliable tools for developers (and consumers) over attempting to build a digital God, then I’m all for it.

And yet… OpenAI just this morning finalized a raise of another $6.5 billion dollars at a staggering $157 billion post-money valuation. That feels more like a digital God valuation to me than a platform for developers in an increasingly competitive space.

Tags: websockets, ai, openai, generative-ai, llms


IdM Laboratory

SIDI Hub東京、前々夜祭を開きます

こんにちは、富士榮です。 いよいよ今月末はSIDI Hub東京イベントです。 が、小規模でのディスカッション中心、かつ英語イベントということもあり招待者に限りご参加いただくという形となります。 そのため、日本のアイデンティティ関係者の皆さんにも概要を知っていただく場として前々夜祭としてイベントを開くことにしました。 https://openid.connpass.com/event/332975/ こちらは日本語で、かつ実際に活動をしている方からも話をしてもらえるようにしたいと思いますので、ぜひご参加ください。
こんにちは、富士榮です。
いよいよ今月末はSIDI Hub東京イベントです。 が、小規模でのディスカッション中心、かつ英語イベントということもあり招待者に限りご参加いただくという形となります。
そのため、日本のアイデンティティ関係者の皆さんにも概要を知っていただく場として前々夜祭としてイベントを開くことにしました。
https://openid.connpass.com/event/332975/
こちらは日本語で、かつ実際に活動をしている方からも話をしてもらえるようにしたいと思いますので、ぜひご参加ください。

Ben Werdmüller

Solving the Maker-Taker problem

[Dries Buytaert] "Addressing the Maker-Taker challenge is essential for the long-term sustainability of open source projects. Drupal's approach may provide a constructive solution not just for WordPress, but for other communities facing similar issues." Dries lays out a constructive approach to crediting open source contributors. There's no stick here: just a series of what

[Dries Buytaert]

"Addressing the Maker-Taker challenge is essential for the long-term sustainability of open source projects. Drupal's approach may provide a constructive solution not just for WordPress, but for other communities facing similar issues."

Dries lays out a constructive approach to crediting open source contributors. There's no stick here: just a series of what amount to promotion and status levels in return for making contributions like "code, documentation, mentorship, marketing, event organization" and so on.

I've certainly had to deal with the maker-taker problem too, although not at the magnitude that either Drupal or WordPress need to consider it. When I worked on Elgg, the open source ecosystem was relatively underdeveloped, and I don't remember it being much of a problem. In contrast, Known plugged into a significantly more advanced ecosystem. The solution Dries lays out makes a ton of sense to me, and I wish we'd done more along these lines in both cases.

#Technology

[Link]


John Philpin : Lifestream

It’s a pretty low bar …

It’s a pretty low bar …

It’s a pretty low bar …


Simon Willison

Ethical Applications of AI to Public Sector Problems

Ethical Applications of AI to Public Sector Problems Jacob Kaplan-Moss developed this model a few years ago (before the generative AI rush) while working with public-sector startups and is publishing it now. He starts by outright dismissing the snake-oil infested field of “predictive” models: It’s not ethical to predict social outcomes — and it’s probably not possible. Nearly everyone claimin

Ethical Applications of AI to Public Sector Problems

Jacob Kaplan-Moss developed this model a few years ago (before the generative AI rush) while working with public-sector startups and is publishing it now. He starts by outright dismissing the snake-oil infested field of “predictive” models:

It’s not ethical to predict social outcomes — and it’s probably not possible. Nearly everyone claiming to be able to do this is lying: their algorithms do not, in fact, make predictions that are any better than guesswork. […] Organizations acting in the public good should avoid this area like the plague, and call bullshit on anyone making claims of an ability to predict social behavior.

Jacob then differentiates assistive AI and automated AI. Assistive AI helps human operators process and consume information, while leaving the human to take action on it. Automated AI acts upon that information without human oversight.

His conclusion: yes to assistive AI, and no to automated AI:

All too often, AI algorithms encode human bias. And in the public sector, failure carries real life or death consequences. In the private sector, companies can decide that a certain failure rate is OK and let the algorithm do its thing. But when citizens interact with their governments, they have an expectation of fairness, which, because AI judgement will always be available, it cannot offer.

On Mastodon I said to Jacob:

I’m heavily opposed to anything where decisions with consequences are outsourced to AI, which I think fits your model very well

(somewhat ironic that I wrote this message from the passenger seat of my first ever Waymo trip, and this weird car is making extremely consequential decisions dozens of times a second!)

Which sparked an interesting conversation about why life-or-death decisions made by self-driving cars feel different from decisions about social services. My take on that:

I think it’s about judgement: the decisions I care about are far more deep and non-deterministic than “should I drive forward or stop”.

Jacob:

Where there’s moral ambiguity, I want a human to own the decision both so there’s a chance for empathy, and also for someone to own the accountability for the choice.

That idea of ownership and accountability for decision making feels critical to me. A giant black box of matrix multiplication cannot take accountability for “decisions” that it makes.

Tags: jacob-kaplan-moss, ai, ethics


Building an automatically updating live blog in Django

Building an automatically updating live blog in Django Here's an extended write-up of how I implemented the live blog feature I used for my coverage of OpenAI DevDay yesterday. I built the first version using Claude while waiting for the keynote to start, then upgraded it during the lunch break with the help of GPT-4o to add sort options and incremental fetching of new updates. Tags: claud

Building an automatically updating live blog in Django

Here's an extended write-up of how I implemented the live blog feature I used for my coverage of OpenAI DevDay yesterday. I built the first version using Claude while waiting for the keynote to start, then upgraded it during the lunch break with the help of GPT-4o to add sort options and incremental fetching of new updates.

Tags: claude, ai, django, llms, javascript, ai-assisted-programming, generative-ai, chatgpt


John Philpin : Lifestream

So having a hard time connecting Threads to Micro.Blog.. mea

So having a hard time connecting Threads to Micro.Blog.. meanwhile BlueSky connected - no problem .. but I can’t find the account to follow on Micro.Blog … So far Mighty Masto is the only one doing both .. connect AND follow .. Anyone else with similar challenges!

So having a hard time connecting Threads to Micro.Blog.. meanwhile BlueSky connected - no problem .. but I can’t find the account to follow on Micro.Blog …

So far Mighty Masto is the only one doing both .. connect AND follow ..

Anyone else with similar challenges!


And once more from a different place.

And once more from a different place.

And once more from a different place.


🪦 Farewell Kris - taking your own last ride - but nothing li

🪦 Farewell Kris - taking your own last ride - but nothing like Casey’s.

🪦 Farewell Kris - taking your own last ride - but nothing like Casey’s.


Ben Werdmüller

How to share your access to media with family and simultaneously sweep the annual nerdy nephew of the year awards

[Matt Haughey] "A couple months ago I was hanging out with my aunt, and she mentioned her cable+internet bill was around $250 per month. I thought that was insane and that I should do something about it. She's a 75 year old retiree that watches baseball and the hallmark channel, and she shouldn't have to pay as much as a car payment every month to do it." What follows is a

[Matt Haughey]

"A couple months ago I was hanging out with my aunt, and she mentioned her cable+internet bill was around $250 per month. I thought that was insane and that I should do something about it. She's a 75 year old retiree that watches baseball and the hallmark channel, and she shouldn't have to pay as much as a car payment every month to do it."

What follows is a very smart way to share media profiles with a family member who doesn't live in your house, using Tailscale as a way to make them seamlessly appear like they're a part of your household.

Tailscale is easy-to-use and is virtually magic. I use it across my devices, and recommend it to others. This is a use case that makes a lot of sense.

#Technology

[Link]

Tuesday, 01. October 2024

IdM Laboratory

Death and the Digital Estate(DADE)CGが発足

こんにちは、富士榮です。 DADE CG(Death and the Digital Estate Community Group)の発足がアナウンスされています。死後のデジタルアイデンティティや遺産について扱うコミュニティグループです。 https://openid.net/announcing-death-and-the-digital-estate-cg/ 4月のIIWの前日のOpenID Foundation Workshopで触れられていたコミュニティですね。 https://idmlab.eidentity.jp/2024/04/openid-foundation-workshopgoogle.html 当時AWSにいたDean Saxe(右から二人目。今はBeyond Identityに移籍)がChairを務めるようです。 メーリングリス
こんにちは、富士榮です。
DADE CG(Death and the Digital Estate Community Group)の発足がアナウンスされています。死後のデジタルアイデンティティや遺産について扱うコミュニティグループです。 https://openid.net/announcing-death-and-the-digital-estate-cg/
4月のIIWの前日のOpenID Foundation Workshopで触れられていたコミュニティですね。 https://idmlab.eidentity.jp/2024/04/openid-foundation-workshopgoogle.html

当時AWSにいたDean Saxe(右から二人目。今はBeyond Identityに移籍)がChairを務めるようです。
メーリングリストへの参加なども受け付けていますので参加してみてはいかがでしょうか?

Doc Searls Weblog

2024_10_01 Postings

A radio item Over on my blog about infrastructure, I put up a brief post about WART, volunteer-powered community radio station with studios in a railroad caboose, that was lost in the flood that just devastated Marshall, North Carolina. Write once, publish everywhere Dave turned me on to Croissant today. Looks good. I’d even be […]
Croissants (the edible kind) on display at Peets in Santa Barbara.

A radio item

Over on my blog about infrastructure, I put up a brief post about WART, volunteer-powered community radio station with studios in a railroad caboose, that was lost in the flood that just devastated Marshall, North Carolina.

Write once, publish everywhere

Dave turned me on to Croissant today. Looks good. I’d even be willing to pay the monthly fee to post once across Bluesky, Mastodon, Threads, and Xitter. But it appears to be only for iOS mobile devices. I have some of those (including a new iPhone 16 Pro), but I mostly write on a computer. So I’ll hold out for real textcasting, like Tim Carmody talks up here. Because why should you have to post separately at all those places? Why should you have to go to a place at all, when you’ve got your own devices to write on and distribute from?

A heading convention

I started curating my photos (e.g. these) in the last millennium using this date-based naming convention: YYYY_MM_DD_topical-text_NNN.jpg (where the NNNs are just sequential numbers and the file type suffix could be .jpg, .arw, .cr2, .png or whatever. Same for folder titles.) So, because I don’t want a new title for every day I do this, I’m adopting the same convention, at least for now.

Not fast enough

In The End of Advertising, Michael Mignano says (in the subhead, and beyond), The business model that funded the internet is going away, and the open web will never be the same. He says AI is already killing it, by giving us answers to everything, and performing other handy tasks, without advertising to distract or annoy us. He also says AI services will attempt to invent ads, but that’s a losing proposition, mostly because it won’t work and we’ll hate it, but also because “content wants to be free.” (I submit that no art forms, ever, wanted to be called “content.”) I agree. I also agree that “Premium content will become even more premium.” He concludes, “the relationship between us and publishers will become much more transactional and direct. And we will feel it. Over time, it’ll be a new internet, and the open web will be a memory. Great content will still find a way to reach us, just like it always has. But we’ll look back on the first few decades of the internet as the golden age of content, when everything felt free.” Well, you’re reading some premium content right now, and it’s free. Thanks to what I do here, I can make money in other ways. We call those because effects.

 

 

 


Simon Willison

OpenAI DevDay 2024 live blog

I'm at OpenAI DevDay in San Francisco, and I'm trying something new: a live blog, where this entry will be updated with new notes during the event. See OpenAI DevDay: Let’s build developer tools, not digital God for my notes written after the event, and Building an automatically updating live blog in Django for details about how this live blogging system worked under the hood. Ta

I'm at OpenAI DevDay in San Francisco, and I'm trying something new: a live blog, where this entry will be updated with new notes during the event.

See OpenAI DevDay: Let’s build developer tools, not digital God for my notes written after the event, and Building an automatically updating live blog in Django for details about how this live blogging system worked under the hood.

Tags: blogging, ai, openai, generative-ai, llms


The Pragmatic Engineer

Bug management that works (Part 1)

Finding and triaging bugs, fixing bugs on the spot instead of ‘managing’ them, and how to make time for bug fixing

Hi, this is Gergely with a subscriber-only issue of the Pragmatic Engineer Newsletter. In every issue, I cover challenges at Big Tech and startups through the lens of engineering managers and senior engineers. To get articles like this in your inbox, every week, subscribe:

Subscribe now

Before we start: The Pragmatic Engineer Podcast started off last week, with episode #1: AI tools for software engineers, but without the hype. New episodes come every other Wednesday. Thank you to everyone who listened to this first one. If you enjoy podcasts, please do add it on Apple, Spotify, YouTube or your in your favorite player – and you will have episodes automatically show up, while also helping the show.

How do you deal with bugs in software products you build? This topic seems very under-discussed, but it affects pretty much every software team. To find out what professionals think, with we reached out to two dozen engineering leaders and software engineers, who kindly shared approaches which work for their team and organization.

This topic is evergreen, and one that has outsized (potentially decisive!) impact on product quality and user experience. 

In this issue, we cover:

Finding bugs. Dogfood products where possible, invest in test automation, stay close to users, and more.

Reporting bugs. Making it easy for users to report bugs, having an accessible engineering team, and scaling the reporting process by team or company size.

Bug triaging. Figure out which bugs are critical by using bug priority rubrics, consider priority definitions, and get inspired by Spotify’s bug prioritization framework.

Don’t manage bugs: fix them! See bug fixing as like gardening, not maintenance. Consider approaches like deleting all amassed bugs, and regularly pruning the backlog.

Zero bugs policy. An approach where all inbound bugs are either fixed on the spot: or the bug is deemed invalid. A subset of this approach is to fix all bugs for recently released features – which is what Figma does.

As a refresher, we have a few previous deepdives that are related to this topic:

Shipping to production

Incident review and postmortem best practices

Healthy oncall practices

How Big Tech does Quality Assurance (QA)

QA approaches across the industry

Thank you to everyone who contributed insights to this article: Ahmed Saher (engineering manager), Anaïs van Asselt (senior QA engineer), Andrea Sipos (product leader), Bernd Kampl (Software Engineering Team Lead), Jason Diller (VP of Engineering), John Cutler (product leader), Magnus L. Udbjørg (CTO), Michał Borek (Principal Engineer), Rebecca Frost (QA leader), Rebecca Holm Ring (engineering leader), Ruben Weijers (engineering manager), Ryan Hanni (Director of Engineering), Serdar Biyik (engineering manager), Walter de Bruijn (Head of Engineering Productivity)

1. Finding bugs

How can we be confident that the software we release has no known issues? We need to validate that it works correctly and there are common approaches for this.

Dogfood products. The term “dogfooding” is the name of the common practice of devs and employees using a product while they are building it, pre-release. For example, when I worked at Uber, the company issued free credits for staff to use the internal beta app for rides and food deliveries. At Skype, we used running internal beta versions of Skype for all internal chat and video calling. The business gave Skype credits to employees, so we could dogfood paid features like landline calls. Spotify does the same, as Rebecca Holm Ring, a former engineering manager there, shares:

“Each employee at Spotify is expected to be on the master or internal Spotify release, and report any issues they experience. A problem here though is that most Spotify employees will be iOS users, and so the Android app doesn’t get nearly enough testing before rolling out.”

Invest in test automation. Anaïs van Asselt – senior QA at Choco – shares their approach:

“Proactively finding and fixing bugs before they reach production is crucial. We invest in various test automation techniques to act as quality gates. Additionally, we practice bug hunting and gradually roll out new features to a limited user base, allowing us to catch and fix bugs early, reducing their overall cost.”

At smaller companies, be close to users. These places tend to be closer to users and can use this to build a relationship with users who get invested in the product and the reporting of bugs. Bernd Kampl – Software Engineering Team Lead at Anyline, a smaller cartech AI company – shares:

“As we are still smaller, we usually find bugs when our customers report them. If that's not the case, we identify them mostly during day-to-day development work.”

Magnus Udbjørg is CTO of Testaviva, a 50-person startup in Denmark. His take is that it’s optimal to build trust with users so they report issues:

“To find, identify, and prevent bugs, we foster a culture of dialogue and mutual understanding. We believe our developers need to understand our users deeply, and sometimes, our users need to understand the technical constraints we face. Open communication is key to building trust.”

A fair question is why not do lots of testing, themselves? The smaller the company and the fewer the customers, the more it feels too expensive to invest a lot in testing, early on. Of course, there are always countercases, like how Figma spent nearly 3 years iterating on the first release, in order to get the performance of their collaborative, web-based editor right, to give users a fun “wow moment.” Worth noting that Figma is a product the dev team used continuously while developing it, getting lots of testing during the building phase.  We cover Figma’s engineering culture in a deep dive.

Consider alpha and beta testing at larger companies. Alpha and beta testing is about giving customers access to unfinished, less stable versions of a product. “Alpha” usually refers to a latest build that has had little to no QA testing. “Beta” versions have had some testing, but not as much as a full release.

Rebecca Holm Ring shares how this worked at Spotify:

“There’s an Alpha testing, and Beta testing program, where external users are on the master branch and release branch respectively, and are expected to report any issues they encounter. With these versions of Spotify, it is also possible for a user to report a bug directly from the app, and logs will be attached as the bug report will be automatically added in JIRA.”

Automation: testing and code analysis. Unit tests, integration tests, end-to-end-tests, and other automated tests, are great ways to catch regressions, which is a software bug introduced into a feature after the feature was working correctly; the feature has ‘regressed’ into a faulty state.

This is true for static code analysis and other tools that automate quality assurance. We cover more on these methods in Shipping to production and QA approaches across the industry.

Code reviews. These serve multiple purposes, offering a second pair of eyes to double check code, spread knowledge, and follow not-yet-automated conventions, and more. Catching bugs before they make it into the codebase is an occasional side effect.

Even so, bugs can easily slip through code reviews, which are nowhere near a perfect way to defend against shipping bugs and regressions.

Define what a bug is. Users often report “bugs” when they mean missing features, so it can be helpful for teams to agree what a bug is and how to categorize them. In general, a bug is a flaw that results in a software product behaving incorrectly. Categorizations can be granular, like splitting bugs into concurrency bugs, syntax ones, arithmetic, logical errors, human errors and so on. 

The simplest categorization is to split bugs into functional ones, when the behavior of the software is clearly wrong, and non-functional ones, when a bug is revealed in things like a system slowing down, increased latency, and other harder-to-spot issues.

It might be helpful to devise your own categorizations, based on the type of bugs you observe, and in a way that’s helpful for your product and organization.

2. Users reporting bugs

Gathering bugs can be a great source of data, providing a sense of product quality for feedback to teams, the organization, or company. However, data quality depends on how good the bug reporting process is – and how likely people are to report bugs!

Great reports and data come from simple, suitable processes. Features of useful bug reports:

Useful metadata (e.g. version, device, system metrics)

Relevant context (e.g. on mobile while connected to bluetooth speaker and poor connectivity, on a server in this region during lunch hour, on a debug build with these feature flags active, etc)

Straightforward to reproduce, or have reproduction steps

Reported by users who trust a reported bug will be fixed

Bad reports can create extra work and poor bug reporting processes can cause people to not commit to recording issues in the first place, and a spiral is created of deteriorating product quality, with the engineering team clueless of how bad things are.

To avoid an outcome like that, here are some processes tech companies use to support good bug reporting processes.

Make it easy to create quality bug reports. Walter de Bruijn, Head of Engineering Productivity at Miro suggests this is critical:

“The easier the reporting is, the more you will learn. For internally discovered bugs an internal #bugs Slack channel can go a long way. 

One of my best recommendations is that you start logging bugs properly: you can’t manage what you can’t see. You need a good, well-documented, and known process for reporting bugs and a follow-up.”

QA leader Rebecca Frost on why quality bug reports count:

“Capture meaningful metadata on your bugs now to help with insights later. 

Make sure your funnel for incoming bugs is helping not hurting you. If bugs are coming in from all over and the quality of the info on the bugs is a mess, consider routing bug reports through one team or group.”

Make the reporting process accessible. If creating a bug report is too complicated, it discourages reporting. There are ways to make it accessible:

Don’t make less technical users use unfamiliar tech jargon

Make it easy to find version numbers and other data required for bug reports. Even better, collect it automatically

Consider having engineers help support teams to know which questions to ask to get the right data for reports

A example of effective engineering support is at SF-based scaleup Ontra, as shared by director of engineering, Ryan Hanni:

“We have an Engineering Support team that sits between Product Support/Customer Success and R&D, which helps bridge the customer with R&D teams potentially further away from that context.”

Scaling bug reporting processes

There’s no one process that works best everywhere. Here are some common approaches by company size:

Smaller companies and startups: bug reports are usually simple, and the reporting process is lean because time is precious and knowledge is dense. Such workplaces are small enough that most tech folks can keep tabs on what’s happening, and people can submit bug reports pretty easily. 

There’s rarely a need for formal processes. Here are some efficient, less formal ones:

Set up a #bug-reports channel on Slack/Teams or other chat product

Use a #bugs tag in the ticket system of choice to keep track of issues

Prefix bug fix pull requests or commits with [Bugfix]

… and similar, simple but effective conventions that anyone can start using

Mid-sized companies and scaleups: process matters more, and these places are big enough for it to be wasteful for everyone to keep tabs on reported bugs. There are also more bug reports, and it’s a time waster to get repeated information and metadata in bug reports. 

Bug report templates and processes also matter. Good onboarding and documentation for processes and standards for bugs can have a big impact on efficiency.

Large companies: investing in automated processes is worthwhile due to the size and nature of the business:

Noisy bug reporting means lots of duplicate reports

“JIRA ping pong”: tickets created in JIRA are continually reassigned between teams and end up ‘returned to sender’

Time wasted discussing bugs: lots of overhead on debating whether a bug is a feature request, who’s best placed to fix it, etc

Here’s a good example of what JIRA ping pong looks like. Engineering manager Rebecca Holm Ring shares how it plays out a larger company:

“Bugs showing up in one user-facing feature could be caused by a number of different teams that own the deeper layers of the tech stack. Still, reports would always first land on the user facing team!

There would often be a game of ping pong, where bugs got sent back-and-forth between different teams, and sometimes found the right owners. In turn, these owners didn’t feel it was a high enough issue for them to fix. However, for the user facing team, it would often be a higher priority to get it resolved. As many different teams were involved in building the Spotify app, conflicting priorities often made bugs linger longer than they should have, and occasionally required intervention from the higher-ups to get these bugs prioritized by the owning team.” 

At larger companies, some things can help deal with an ever-growing pile of bug reports, and improve processes and tooling:

Automation across the bug reporting process

Automated data collection for gathering context

Bug reporting templates for internal use

Automatic code/team attribution for inbound reports

Ryan Hanni, director of engineering at Ontra, shares examples of manual and mostly-automated processes he’s implemented at different stages of an org’s lifecycle:

Manual process: 

Use a bug template in Trello, JIRA, or similar 

Notify Eng Support and copy the bug description to their slack channel 

→ Eng support aids in determining severity, and the team takes it from there

→ Eng Director uses a Google Spreadsheet to track all bugs, and includes details like report date and links

→ Eng Director does monthly follow ups on bugs, and adds details to the spreadsheet, like the timestamp of a fix. This spreadsheet is used for calculating stats like time-to-resolution for DORA metrics.

Partially Automated: 

Use a bug template in Trello, JIRA, or similar. 

With a press of a button, automatically post to Slack

→ Eng Support 'triages' the bug with the dev team in a thread

→ For high or medium severity bugs, support copies the details to an Airtable form and submits it 

→ The form submitted uses automation to:

Post to Slack

Send an email to internal google groups for stakeholders and departments

Automatically adds bug information into a Airtable columns

→ Partially automated reporting:

Metrics for ‘Reported at’ date and severity are automatically filled in

Metrics are filed on a monthly basis to generate stats like DORA metrics and cycle time to fix (Time to Respond, Time to Fix, Total Time In Prod, etc). 

→ When a bug is fixed, teams reply to the automated email detailing when it will be released to customers

Mostly Automated: 

Use a bug template in Trello, JIRA, or similar. 

→Use automation to send a message to Eng Support slack. 

→ Triage happens with the Dev Team and Eng Support

→ Work moves through Jira columns (Kanban style), into a Done column 

→ A release to production happens and the Jira card gets an automatic “released at” date

→ After a release to Prod, Jira sends a slack/email message with issue details to close the loop for a resolved bug of a given severity.

→ Reporting occurs by generating reports in JIRA. Run reports with all relevant metrics to the org to get information like cycle time, time to fix, response time, etc

→ Visuals for share outs on the bug fixing process is manually generated in Google Sheets

3. Bug triage

You now have a process for the bug reports to flow in, so the next step is to figure out which ones are critical, which are duplicates, and which ones not to bother with. Here are some common approaches:

Read more

Tuesday, 01. October 2024

Mike Jones: self-issued

OpenID Connect specifications published as ISO standards

I’m thrilled to report that the OpenID Connect specifications have now been published as ISO/IEC standards. They are: ISO/IEC 26131:2024 — Information technology — OpenID connect — OpenID connect core 1.0 incorporating errata set 2 ISO/IEC 26132:2024 — Information technology — OpenID connect — OpenID connect discovery 1.0 incorporating errata set 2 ISO/IEC 26133:2024 — […]

I’m thrilled to report that the OpenID Connect specifications have now been published as ISO/IEC standards. They are:

ISO/IEC 26131:2024 — Information technology — OpenID connect — OpenID connect core 1.0 incorporating errata set 2 ISO/IEC 26132:2024 — Information technology — OpenID connect — OpenID connect discovery 1.0 incorporating errata set 2 ISO/IEC 26133:2024 — Information technology — OpenID connect — OpenID connect dynamic client registration 1.0 incorporating errata set 2 ISO/IEC 26134:2024 — Information technology — OpenID connect — OpenID connect RP-initiated logout 1.0 ISO/IEC 26135:2024 — Information technology — OpenID connect — OpenID connect session management 1.0 ISO/IEC 26136:2024 — Information technology — OpenID connect — OpenID connect front-channel logout 1.0 ISO/IEC 26137:2024 — Information technology — OpenID connect — OpenID connect back-channel logout 1.0 incorporating errata set 1 ISO/IEC 26138:2024 — Information technology — OpenID connect — OAuth 2.0 multiple response type encoding practices ISO/IEC 26139:2024 — Information technology — OpenID connect — OAuth 2.0 form post response mode

I submitted the OpenID Connect specifications for publication by ISO as Publicly Available Specifications (PAS) for the OpenID Foundation in December 2023. Following the ISO approval vote, they are now published. This should foster even broader adoption of OpenID Connect by enabling deployments in jurisdictions around the world that have legal requirements to use specifications from standards bodies recognized by international treaties, of which ISO is one.

Before submitting the specifications, the OpenID Connect working group diligently worked through the process of applying errata corrections to the specifications, so that the ISO versions would have all known corrections incorporated.

Having successfully gone through the ISO PAS submission process once, the OpenID Foundation now plans to submit additional families of final specifications for publication by ISO. These include the FAPI 1.0 specifications, and once they’re final, the eKYC-IDA specifications and FAPI 2.0 specifications.

Thanks to all who helped us achieve this significant accomplishment!

Tuesday, 01. October 2024

Doc Searls Weblog

Podcasts, Wallcasts, and Paycasts

Would a blog be a blog if it went behind a paywall, or if you needed a subscription to read it? Of course not. Blogs are on the open Web, and tend to stay there so long as they don’t move away from their original location. Same should go for podcasts. “Wherever you get your […]
The Ezra Klein Show, as it appeared on my podcast app this morning. It is now a wallcast.

Would a blog be a blog if it went behind a paywall, or if you needed a subscription to read it?

Of course not. Blogs are on the open Web, and tend to stay there so long as they don’t move away from their original location.

Same should go for podcasts. “Wherever you get your podcasts” certifies the open nature of podcasting.

But now the New York Times is putting Ezra Klein’s podcast archives behind a paywall.

Never mind how icky this is on several grounds. Our challenge now is classification. We need a new noun for restricted ‘casts such as Ezra’s. I suggest wallcasts.

For subscription-only ‘casts, such as some on SiriusXM*, I suggest paycasts.

Bottom line: It can’t be a podcast if you have to pay for any of it, including archives.

By the way, it won’t matter if a Times subscription opens wallcast archives, as it does for print. By putting their podcast archives behind a paywall, the Times is changing the DNA of those casts. A wallcast is not a podcast. Full stop.

Spread the words.

*SiriusXM’s paycasts include “SmartLess,” “Freakonomics Radio,” “The Joel Osteen Podcast,” “Last Podcast on the Left,” and “Andy Cohen’s Daddy Diaries.” They require a subscription to SiriusXM or its Podcasts+ service​. Some, such as “Marvel’s Wastelanders” and “Marvel/Method also require a subscription​. I’m not sure what kind. (FWIW, I’ve been a SiriusXM subscriber since 2005, but only listen to live subscription streams. I’ve never listened to any of its podcasts.) SiriusXM does have some shows in podcast form, however. Examples are “The Megyn Kelly Show,” “Best Friends with Nicole Byer and Sasheer Zamata,” and “Chasing Life with Dr. Sanjay Gupta.” I believe it also has some wallcasts. For example, “SmartLess” episodes are on the open Web, but early access and bonus episodes are behind a paywall​. Or so it seems to me in the here and now. I invite corrections.


Michael Ruminer

Google’s NotebookLM, RAG And Then Some

I played around just a little bit with Google’s NotebookLM. It’s a Retrieval-Augmented Generation (RAG) tool. One thing it does is convert your content into a podcast format and it provides an impressive-sounding result. It's a little superficial on the content synthesis, but it is a podcast after all. Its other features are your typical RAG chat interface. All I can say is give it a try, or check

I played around just a little bit with Google’s NotebookLM. It’s a Retrieval-Augmented Generation (RAG) tool. One thing it does is convert your content into a podcast format and it provides an impressive-sounding result. It's a little superficial on the content synthesis, but it is a podcast after all. Its other features are your typical RAG chat interface. All I can say is give it a try, or check out my example effort.

Here is a link to a single source document I provided it- Self Sovereign Identity A Systematic Review Mapping and Taxonomy and the link to the SSI Paper Podcast results.

Monday, 30. September 2024

IdM Laboratory

SIDI Hub - ベルリンレポートを読む(2)

こんにちは、富士榮です。 しばらく別のネタばかりでSIDI Hubについてかけていませんでしたが、10月に入ったので東京開催秒読みということでベルリンレポートの続きを読んでいきます。 前回からしばらく開きましたが、今回は続きです。 Users of a Trust Framework Analysis Tool A major output of the SIDI Hub 2024 strategy, led by the Open Identity Exchange (OIX), will be a Trust Framework Comparison Tool. This will be bolstered by further analysis and normalization of Trust Frameworks supported by SIDI Hub

こんにちは、富士榮です。


しばらく別のネタばかりでSIDI Hubについてかけていませんでしたが、10月に入ったので東京開催秒読みということでベルリンレポートの続きを読んでいきます。


前回からしばらく開きましたが、今回は続きです。

Users of a Trust Framework Analysis Tool

A major output of the SIDI Hub 2024 strategy, led by the Open Identity Exchange (OIX), will be a Trust Framework Comparison Tool. This will be bolstered by further analysis and normalization of Trust Frameworks supported by SIDI Hub. At the SIDI Berlin session, breakout groups shaped the value proposition and requirements for such a comparison tool, which will directly influence the final deliverable. Further information is found in the Rapporteur’s notes (next section).

信頼フレームワーク分析ツールのユーザー 

Open Identity Exchange (OIX) が主導する SIDI Hub 2024 戦略の主な成果のひとつは、信頼フレームワーク比較ツールです。これは、SIDI Hub がサポートする信頼フレームワークのさらなる分析と標準化によって強化されます。SIDI Berlin セッションでは、分科会がこのような比較ツールの価値提案と要件を策定し、最終成果物に直接影響を与えることになります。詳細は、ラポータのメモ(次項)をご覧ください。 

トラストフレームワークのマッピングに関して書かれていますね。

現在、各国で制定が進んでいるトラストフレームワークの相互運用が可能な状態にならないと国の間で相互運用性の担保ができなくなるのでここでいうマッピングは非常に重要です。OpenIDファウンデーションジャパンではOIXに協力する形で日本のトラストフレームワークのマッピングを支援しています。

BOLTS: Business, Operational, Legal, Technical, and Social

Given the above take-aways, which span Business, Operational, Legal, Technical, and Social forces that impact the global interoperability effort, the group will use a “BOLTS” framework as a core part of its Champion Use Case analysis.

BOLTS:ビジネス、運用、法律、技術、社会

グローバルな相互運用性への取り組みに影響を与えるビジネス、運用、法律、技術、社会の各分野における上記の要点を踏まえ、当グループは「BOLTS」フレームワークをチャンピオンユースケース分析の中核として使用します。

相互運用性を考える上では技術だけを考えていたは不十分です。ここにあるようにビジネス、運用、法律、社会を含めて考える必要がある、ということです。

Government Participation

A final point of reflection relates to the audience for SIDI Hub events. Given the light attendance from government officials in Berlin, the agenda skewed towards a technical audience that discussed technical possibilities. This is not ideal.

政府の参加

最後に、SIDI Hubのイベントの聴衆について考察したいと思います。ベルリンでの政府関係者の出席が少なかったため、技術的な可能性について議論する技術的な聴衆に偏ったアジェンダとなりました。これは理想的ではありません。

先に記載した通り、法律や社会についても検討が必要です。ベルリンでは政府機関の設備で開催したにもかかわらず確かにあまり多くの政府関係者が参加したわけではありませんでした。この辺りは日本開催をする際のバランスに関する考慮点となるでしょう。

SIDI Hub was founded to unite global audiences to define the users, benefits, and overall business case for globally interoperable digital identity to normalize approaches and define minimum requirements. It was, therefore, somewhat premature to attempt a solution-oriented agenda. With that said, the lessons were valuable, and SIDI Hub has had valuable contributions from European stakeholders through other avenues, e.g., the SIDI Paris Summit, eIDAS 2.0 documentation, etc. Regardless, the SIDI organizers have determined that baseline government participation will be a critical go/no-go criterion for the events planned in Washington, D.C., Tokyo, and Brazil.

SIDI Hubは、世界中のオーディエンスをまとめ、世界規模で相互運用可能なデジタルIDのユーザー、利点、全体的なビジネスケースを定義し、アプローチを標準化し、最低限の要件を定義するために設立されました。そのため、ソリューション志向のアジェンダを試みるには時期尚早でした。とはいえ、そこから得られた教訓は貴重であり、SIDIハブは、SIDIパリサミットやeIDAS 2.0文書など、他の手段を通じて欧州の利害関係者から貴重な貢献を得ることができました。それでも、SIDIの主催者は、ワシントンD.C.、東京、ブラジルで計画されているイベントについては、政府の基本的な参加が実施の可否を決定する重要な基準となると判断しました。

ベルリンでもユースケースの取りまとめ要件整備を行いました。次のワシントンDCや東京・ブラジルでの開催に向けて議論をしていく必要がありそうです。なお、ここに記載がある通りソリューションとして自立させるためのきっかけには早すぎるイメージはありました。しかし読者の皆さんは気にせずにアプライしてくださいね。

 

 

 

 

 









Doc Searls Weblog

When Radio Delivers

Helene was Western North Carolina‘s Katrina—especially for the counties surrounding Asheville: Buncombe, Mitchell, Henderson, McDowell, Rutherford, Haywood, Yancey, Burke, and some adjacent ones in North Carolina and Tennessee. As with Katrina, the issue wasn’t wind. It was flooding, especially along creeks and rivers. Most notably destructive was the French Broad River, which runs through Ashevill
For live reports on recovery from recent Hurricane Helene flooding, your best sources are Blue Ridge Public Radio (WCQS/88.1) and iHeart (WWNC/570 and others above, all carrying the same feed). Three FM signals come from the towers on High Top Mountain, which overlooks Asheville from the west side:  1) WCQS, 2) a translator on 102.1 for WNCW/88.7, and 3) a translator on 97.7 for WKSF/99.9’s HD-2 stream. At this writing, WCQS (of Blue Ridge Public Radio) and the iHeart stations (including WKSF, called Kiss Country) are running almost continuous public service coverage toward rescue and recovery. Hats off to them.

Helene was Western North Carolina‘s Katrina—especially for the counties surrounding Asheville: Buncombe, Mitchell, Henderson, McDowell, Rutherford, Haywood, Yancey, Burke, and some adjacent ones in North Carolina and Tennessee. As with Katrina, the issue wasn’t wind. It was flooding, especially along creeks and rivers. Most notably destructive was the French Broad River, which runs through Asheville. Hundreds of people are among the missing. Countless roads, including interstate and other major highways, are out. Towns and communities—Spruce Pine, Swananoa, Chimney Rock, Mitchell, Lake Lure, and many others—have been wiped away, or are in ruins. Roads across the region are gone, or closed. Electric, water, gas, sewer, and other utilities are expected to be down in many places for weeks.

One public utility that is working enough for people to keep up with news is radio. Many (perhaps most) stations are off the air, but some survive, and are providing constant service to residents and people out of the area who want to stay informed. The two I recommend most are Blue Ridge Public Radio (WCQS/88.1) and any of the local iHeart stations . All of the iHeart stations listed in the image above are carrying the same continuous live coverage, which is excellent.  (I’m listening right now to the WWNC/570 stream.)

Of course, there’s lots of information on social media (e.g. BlueSky, Xitter, Threads), but if you want live coverage, radio still does what only it can do. Yes, you need special non-phone equipment to get it when the cell system doesn’t work, but a lot of us still have those things. Enjoy the medium while we still have it.

Item: WWNC just reported that WART/95.5 FM in Marshall, with its studios in a train caboose by the river, is gone (perhaps along with much of the town). [Later…For more on that, go here.]

More sources:

WISE/1310 stream WTMT/105.9 stream

This is cross-posted on Trunkli, my blog on infrastructure.


Ben Werdmüller

Bop Spotter

[Bop Spotter] "I installed a box high up on a pole somewhere in the Mission of San Francisco. Inside is a crappy Android phone, set to Shazam constantly, 24 hours a day, 7 days a week. It's solar powered, and the mic is pointed down at the street below." As surveillance goes, I'm into it. I appreciate the commentary: "Heard of Shot Spotter? Microphones are installed acros

[Bop Spotter]

"I installed a box high up on a pole somewhere in the Mission of San Francisco. Inside is a crappy Android phone, set to Shazam constantly, 24 hours a day, 7 days a week. It's solar powered, and the mic is pointed down at the street below."

As surveillance goes, I'm into it. I appreciate the commentary:

"Heard of Shot Spotter? Microphones are installed across cities across the United States by police to detect gunshots, purported to not be very accurate. This is that, but for music."

I don't give it much time before someone figures out where it is and tries to mess with it, though.

#Culture

[Link]


Jon Udell

Making a Vote Forward checklist

In How and why to write letters to voters I discussed Vote Forward, my favorite way for those of us who aren’t in swing states to reach out to voters in swing states. The site works really well for adopting batches of voters, and downloading packets of form letters. As I close in on 1000 … Continue reading Making a Vote Forward checklist

In How and why to write letters to voters I discussed Vote Forward, my favorite way for those of us who aren’t in swing states to reach out to voters in swing states. The site works really well for adopting batches of voters, and downloading packets of form letters. As I close in on 1000 letters, though, I’m finding it isn’t great for tracking progress at scale. Here’s how my dashboard page looks.

With 50 bundles in play, many of which are farmed out to friends and neighbors who are helping with the project, it’s become cumbersome to keep track of which bundles are prepped (ready to mail) or not. Here is the checklist I needed to see.

VoteForward Dashboard Report mmorg: 1-UNPREPPED r23Pp: 2-UNPREPPED v9Kbo: 3-UNPREPPED wLMPw: 4-UNPREPPED 24L4o: 5-PREPPED 4nNnj: 6-PREPPED 5rQmV: 7-PREPPED ... YV4dL: 48-PREPPED zKjne: 49-PREPPED ZrKJz: 50-PREPPED

If you’re in the same boat, here’s a piece of code you can use to make your own checklist. It’s gnarly, if you aren’t a programmer I advise you not even to look at it, just copy it, and then paste it into your browser to have it open a new window with your report.

Vote Forward checklist maker (expand to copy) javascript:(function(){ // First part: Adjust height of divs with inline styles document.querySelectorAll('div[style]').forEach(div => { let inlineStyle = div.getAttribute('style'); if (inlineStyle.includes('position: relative')) { div.style.height = '20000px'; // Set the height to 20000px } }); // Introduce a delay before processing the list of items setTimeout(() => { const items = document.querySelectorAll('li.bundle-list-item.individual'); let dataList = []; // Iterate over the items to capture data-testid and ID items.forEach(item => { let dataTestId = item.getAttribute('data-testid'); // Use the id attribute of the input element to extract the ID const toggleInput = item.querySelector('input.slide-out-toggle'); const toggleId = toggleInput ? toggleInput.getAttribute('id') : ''; // Extract the ID part from the toggleId pattern "toggle-24L4o-PREPPED" const id = toggleId ? toggleId.split('-')[1] : 'ID not found'; // Remove "bundle-" and the number part from dataTestId, keeping only "PREPPED" or "UNPREPPED" dataTestId = dataTestId.split('-').pop(); // Extract only the "PREPPED" or "UNPREPPED" part // Push the data into the array dataList.push({ dataTestId, id }); }); // Sort first by whether it's PREPPED or UNPREPPED (descending for UNPREPPED first), // then by the ID within each group dataList.sort((a, b) => { if (a.dataTestId.includes("PREPPED") && b.dataTestId.includes("UNPREPPED")) { return 1; // UNPREPPED comes before PREPPED } else if (a.dataTestId.includes("UNPREPPED") && b.dataTestId.includes("PREPPED")) { return -1; } // Sort by ID if they belong to the same category return a.id.localeCompare(b.id); }); // Prepare the output string let output = ''; dataList.forEach((item, index) => { output += `${item.id}: ${index + 1}-${item.dataTestId}\n`; }); // Open a new window with the output in a text area for easy copying let newWindow = window.open('', '', 'width=500,height=500'); newWindow.document.write('<html><body><h2>VoteForward Dashboard Report</h2><pre>' + output + '</pre></body></html>'); newWindow.document.close(); }, 2000); // Adjust delay as needed })();

Here are instructions for Chrome/Edge, Safari, and Firefox. You might need to tell your browser to allow the popup window in which it writes the report.

Chrome/Edge: Open the VoteForward dashboard in your browser. Open the developer console: Windows/Linux: Press Ctrl + Shift + J. Mac: Press Cmd + Option + J. Paste the code into the console. Press Enter to run the code. Firefox: Open the VoteForward dashboard in your browser. Open the developer console: Windows/Linux: Press Ctrl + Shift + K. Mac: Press Cmd + Option + K. Paste the code into the console. Press Enter to run the code. Safari: Open the VoteForward dashboard in your browser. Enable the developer console (if it’s not already enabled): Go to Safari > Preferences. Click the Advanced tab. Check “Show Develop menu in menu bar” at the bottom. Open the developer console: Press Cmd + Option + C. Paste the code into the console. Press Enter to run the code.

It would be nice to have this as a built-in feature of the site but, as we come down to the wire, this may be a helpful workaround.

Thanks, again, to the Vote Forward team for all you do! It’s a great way to encourage voter turnout.


Ben Werdmüller

How the UK became the first G7 country to phase out coal power

[Molly Lempriere and Simon Evans in CarbonBrief] "Remarkably, the UK’s coal power phaseout – as well as the closure of some of the country’s few remaining blast furnaces at Port Talbot in Wales and Scunthorpe in Lincolnshire – will help push overall coal demand in 2024 to its lowest level since the 1600s." The UK aims to fully decarbonize its power supplies by 2030. That in

[Molly Lempriere and Simon Evans in CarbonBrief]

"Remarkably, the UK’s coal power phaseout – as well as the closure of some of the country’s few remaining blast furnaces at Port Talbot in Wales and Scunthorpe in Lincolnshire – will help push overall coal demand in 2024 to its lowest level since the 1600s."

The UK aims to fully decarbonize its power supplies by 2030. That involves phasing out gas power in under six years: a big milestone and an ambitious goal, and one it hopes will be a case study for other nations.

Meanwhile, the US continues to limp along, generating around 60% of its electricity from fossil fuels. In light of accelerating climate change, that's a figure we should be truly embarrassed about.

#Climate

[Link]


@_Nat Zone

マイナ保険証のスマートフォン搭載:2025年春に実現へ

厚生労働省は、2025年春をめどにマイナ保険証のスマートフォン搭載を開始する計画を発表しました。この新システムは、AndroidとiPhoneの両方に対応する予定です。 主なポイント: 利用方法: この計画は、9月30日の16時00分から18時30分に開催された第183回社会保障…

厚生労働省は、2025年春をめどにマイナ保険証のスマートフォン搭載を開始する計画を発表しました。この新システムは、AndroidとiPhoneの両方に対応する予定です。

主なポイント:

導入時期:2025年春を予定 対応機種:Android(グーグル)とiPhone(アップル) 医療機関側の準備:新たな読み取り機器の設置が必要

利用方法:

患者は医療機関の顔認証付きカードリーダーでスマホ利用を選択 専用リーダーにスマホをかざす 画面上で情報提供に同意して手続き完了 (出所)厚生労働省「マイナ保険証の利用促進等について」P.15 (2024-09-30)

この計画は、9月30日の16時00分から18時30分に開催された第183回社会保障審議会医療保険部会で公表されました。厚生労働省は、物理的なマイナンバーカードがなくても医療機関を受診できるようになるため、マイナ保険証の利用が促進されると期待しています。

今後、実証事業を通じて普及を目指していく方針です。


Ben Werdmüller

Digital Divinity

[Rest of World] "Technology has transformed how we spend, study, live, eat — even how we sleep. And for the 6.75 billion people around the world who consider themselves religious, technology is also changing their faith. How people worship, pray, and commune with the divine is transforming from Seoul to Lagos." These are amazing stories that sometimes sound like provocative

[Rest of World]

"Technology has transformed how we spend, study, live, eat — even how we sleep. And for the 6.75 billion people around the world who consider themselves religious, technology is also changing their faith. How people worship, pray, and commune with the divine is transforming from Seoul to Lagos."

These are amazing stories that sometimes sound like provocative satire: PETA is building robot elephants for Hindu temples, for example. Or take this app, which will narrate the Bible in your own voice, perhaps so that you can make it more accessible for your children.

Many of the examples feel a lot like startups spotting new markets without consideration for whether they should. Some are more authentic. All are continuing examples of how the internet is changing religious life all over the world.

#Technology

[Link]


The secret power of a blog

[Tracy Durnell] "Blogs coax out deeper thinking in smaller blocks. A blog gives you the space to explore and nurture ideas over time, perhaps growing so slowly you hardly notice the extent of the evolution of your thoughts till you read something you wrote a few years ago." Everyone should blog. It's been the single most transformative tool in my career - and a huge part of

[Tracy Durnell]

"Blogs coax out deeper thinking in smaller blocks. A blog gives you the space to explore and nurture ideas over time, perhaps growing so slowly you hardly notice the extent of the evolution of your thoughts till you read something you wrote a few years ago."

Everyone should blog. It's been the single most transformative tool in my career - and a huge part of my life.

Given the latter part, I needed to hear this:

"We know, when we’re reading a blog, that we’re getting a glimpse into the writer’s active psyche, a tour of their studio as it were — not hearing their thesis presentation or reading their pre-print publication; hearing from other people being people is part of the appeal of blogs."

Over the last few years I've downgraded the amount of personal writing in this space in favor of more thoughts about technology. I never quite know where the balance is, but I think there's a lot to be said for turning the dial closer to the personal.

If you haven't started yet: try it and let me know about it. I'd love to read your thoughts.

And if you know you want to start but don't know where, Get Blogging! has your back.

#Technology

[Link]


Doc Searls Weblog

Post flow

A watershed* is land that drains through a river to the sea or into an inland body of water. That’s what came to mind for me when I read this from Dave Winer: If you want to help the open web, when you write something you’re proud of on a social web site like Bluesky […]
These are eight (among many other) failed attempts to get ChatGPT and Copilot to create an image of posts in X (née Twitter), Linkedin, Facebook, Threads, BlueSky, Mastodon, and Instagram to flow or rain down from their clouds into a river of blogs.

A watershed* is land that drains through a river to the sea or into an inland body of water. That’s what came to mind for me when I read this from Dave Winer:

If you want to help the open web, when you write something you’re proud of on a social web site like Bluesky or Mastodon, also post it to your blog. Not a huge deal but every little bit helps.#

I love the idea of using one’s blog (as Dave does) as the personal place to collect what one posts on various social media. So the flow, which we might call a postshed (would post shed be better because it’s easier to read?) is from social media clouds into one’s own river of blog posts. (Maybe postflow would be better. I invite better nouns and/or verbs.)

So I’ll try doing some of that flow today. Here goes:

On X I pointed to Death as a Feature, which was my response six years ago to Elon Musk’s Martian ambitions. This was in response Doge Designer tweeting “Elon Musk is projected to become world’s first trillionaire. He said ‘My plan is to use the money to get humanity to Mars & preserve the light of consciousness,” to which Elon’s reply was, “That’s the goal.”

On Threads (and perhaps on other federated media, if that works): Besides Blue Ridge Public Radio bpr.org (which is great) what else should we tune in to hear what’s happening in the flooded parts of Western North Carolina? The best answer is any iHeart station in the region, over the air or on the iHeart app. I’m listening right now to WWNC/570.

On Facebook: Due to popular request (by one person, but not the first), I’ve put a pile of headshots up here: https://www.flickr.com/photos/docsearls/albums/72177720312529167/

Also on Facebook, While I was never a fan of his teams (I swang with the Mets), I loved watching Pete Rose play baseball. He was truly great. RIP, Charlie Hustle. https://www.nytimes.com/2024/09/30/sports/baseball/pete-rose-baseball-star-who-earned-glory-and-shame-dies-at-83.html

*Wikipedia calls watersheds “drainage basins.” Not appetizing.

Sunday, 29. September 2024

IdM Laboratory

Ruby-SAMLの脆弱性(XML署名ラッピング攻撃)

こんにちは、富士榮です。 XML署名ラッピングによるRuby-SAMLの脆弱性が報告されていますね。 https://ssoready.com/blog/engineering/ruby-saml-pwned-by-xml-signature-wrapping-attacks/ CVE-2024-45409としても登録されています。 https://nvd.nist.gov/vuln/detail/CVE-2024-45409 内容としては割とオーソドックスなXML署名の実装の問題っぽいですね。 XML署名の特徴としてJWSとは違いドキュメント全体ではなく要素を指定して署名をおこなうことができる点、署名された値そのものも当該XMLの内部に埋め込まれることが挙げられます。 今回は部分的な署名を行うことができるという点について悪用された感じですね。上記の
こんにちは、富士榮です。
XML署名ラッピングによるRuby-SAMLの脆弱性が報告されていますね。
https://ssoready.com/blog/engineering/ruby-saml-pwned-by-xml-signature-wrapping-attacks/
CVE-2024-45409としても登録されています。 https://nvd.nist.gov/vuln/detail/CVE-2024-45409
内容としては割とオーソドックスなXML署名の実装の問題っぽいですね。
XML署名の特徴としてJWSとは違いドキュメント全体ではなく要素を指定して署名をおこなうことができる点、署名された値そのものも当該XMLの内部に埋め込まれることが挙げられます。 今回は部分的な署名を行うことができるという点について悪用された感じですね。上記の例で言うと、真ん中にあるalice@customer.comの値を含むid=dead[....]beefの部分に対して署名値が生成される一方で、攻撃者が偽の値をXML内に埋め込んだ同じid=dead[...]beefの部分は署名されない、ということが起きてしまうわけです。
この攻撃は2012年にJPCERTが以下のペーパーを出していたり、と過去もしばしば起きている話なので、実装する際は先人の知恵に頼りながらやっていけるといいでしょう。 参考)JPCERTの資料 https://www.jpcert.or.jp/securecoding/2012/No.10_Apache_Axis.pdf



dockがmDLのWebinarをやるようです

こんにちは、富士榮です。 パスポートや免許証のApple Wallet/Google Walletへの格納の話も多く、世の中はすっかりmDoc祭りですね。 そんな中、各社も色々イベントやセミナーを仕掛けてきているわけですが、VCやWalletの界隈ではそろそろ老舗?になりつつあるdockもmDLに関するWebinarをやるようです。 https://www.dock.io/live-event-mdls-are-coming 13 US states have already rolled out mobile digital driver's licenses (mDLs), and many more are testing the waters. Why the buzz? These government-issued digital ID
こんにちは、富士榮です。
パスポートや免許証のApple Wallet/Google Walletへの格納の話も多く、世の中はすっかりmDoc祭りですね。
そんな中、各社も色々イベントやセミナーを仕掛けてきているわけですが、VCやWalletの界隈ではそろそろ老舗?になりつつあるdockもmDLに関するWebinarをやるようです。

https://www.dock.io/live-event-mdls-are-coming
13 US states have already rolled out mobile digital driver's licenses (mDLs), and many more are testing the waters. Why the buzz? These government-issued digital IDs promise game-changing benefits: enhanced privacy, smoother online transactions, and a streamlined process for everything from opening a bank account to securing a loan. So, here's the real question: how will mDLs transform remote ID verification?

米国ではすでに13の州でモバイル・デジタル運転免許証(mDL)が導入され、さらに多くの州で試験運用が行われている。なぜ話題になっているのか?これらの政府発行のデジタルIDは、プライバシーの強化、よりスムーズなオンライン取引、銀行口座の開設からローンの確保までの合理化されたプロセスなど、ゲームチェンジャー的なメリットを約束している。

mDLは遠隔地でのID認証にどのような変革をもたらすのだろうか?

なかなか興味深いですね。

例によって日本時間だと10月3日(木)AM1:00-という酷い時間ですが、興味のある方は参加してみると米国の様子などわかるかもしれませんね。



Saturday, 28. September 2024

Ben Werdmüller

IRL taking priority

We’ve been dealing with some intense family health events since Wednesday night, so I’m running on very little sleep and not updating much over here. I’ll be popping in from time to time, but probably not running on all cylinders for a little while. There’s a lot to say — about WordPress, about the independent web, about media, about some of the conversations coming out of ONA — but they will n

We’ve been dealing with some intense family health events since Wednesday night, so I’m running on very little sleep and not updating much over here. I’ll be popping in from time to time, but probably not running on all cylinders for a little while.

There’s a lot to say — about WordPress, about the independent web, about media, about some of the conversations coming out of ONA — but they will need to wait. See you soon!

Friday, 27. September 2024

IdM Laboratory

AuthZENのAuthorization APIとは(5)

こんにちは、富士榮です。 AuthZEN WGが策定しているAuthorization API 1.0 draftを引き続き見ていきます。 https://openid.net/specs/authorization-api-1_0-01.html 最後はTransportです。 7. Transport This specification defines an HTTPS binding which MUST be implemented by a compliant PDP. Additional transport bindings (e.g. gRPC) MAY be defined in the future in the form of profiles, and MAY be implemented by a PDP. 7.トランスポート
こんにちは、富士榮です。
AuthZEN WGが策定しているAuthorization API 1.0 draftを引き続き見ていきます。 https://openid.net/specs/authorization-api-1_0-01.html
最後はTransportです。
7. Transport This specification defines an HTTPS binding which MUST be implemented by a compliant PDP. Additional transport bindings (e.g. gRPC) MAY be defined in the future in the form of profiles, and MAY be implemented by a PDP. 7.トランスポート この仕様は、準拠した PDP によって実装されなければならない HTTPS バインディングを定義します 追加のトランスポートバインディング (例: gRPC) は、将来プロファイルの形式で定義され、PDP によって実装される可能性があります

設計思想としてTransport Agnosticにしているのはいいことだと思います。今回は手始めにHTTPSバインディングからスタートしますが、将来的にgRPCなどへもバインドされる可能性はありますね。むしろIoT文脈なども考えるならOver BLEとかも出てくるかもしれません。

7.1. HTTPS Binding
7.1.1. HTTPS Access Evaluation Request

The Access Evaluation Request is an HTTPS request with content-type of application/json. Its body is a JSON object that contains the Access Evaluation Request, as defined in Section 6.1.

The following is a non-normative example of the HTTPS binding of the Access Evaluation Request:

7.1. HTTPSバインディング
7.1.1. HTTPSアクセス評価リクエスト

content-type、のHTTPS リクエストです。その本体は、セクション 6.1application/jsonで定義されているアクセス評価リクエストを含む JSON オブジェクトです。

以下は、アクセス評価リクエストの HTTPS バインディングの非規範的な例です: 

POST /access/v1/evaluation HTTP/1.1 Host: pdp.mycompany.com Authorization: Bearer <myoauthtoken> X-Request-ID: bfe9eb29-ab87-4ca3-be83-a1d5d8305716 { "subject": { "type": "user", "id": "alice@acmecorp.com" }, "resource": { "type": "todo", "id": "1", }, "action": { "name": "can_read" }, "context": { "time": "1985-10-26T01:22-07:00" } } Figure 14Example of an HTTPS Access Evaluation Request

まずはリクエストをOver HTTPSで実装する例です。

まぁ、POSTでしょうね。アクセストークンでの保護も重要なポイントです。別途Security Considerationでも今後議論されると思いますが、ここではPEPからのリクエストに対する攻撃(盗聴、置き換えや改ざんなど)が一番の考慮事項になると思います。単純にSubjectのロールやResourceなどを置き換えてしまうことでアクセス制御をバイパスできてしまうとまずいわけです。(現状のSecurity Considerationはまだ薄っぺらいのでもうちょっと詰めないとダメな気がしています。一応Sender Constraintについては触れられてはいますが)

7.1.2. Access Evaluation HTTPS Response

The success response to an Access Evaluation Request is an Access Evaluation Response. It is an HTTPS response with a status code of 200, and content-type of application/json. Its body is a JSON object that contains the Access Evaluation Response, as defined in Section 6.2.

Following is a non-normative example of an HTTPS Access Evaluation Response:

7.1.2.アクセス評価 HTTPS レスポンス

アクセス評価リクエストに対する成功応答は、アクセス評価応答です。statusが200でcontent-typeはapplication/jsonのHTTPS応答です。ボディはセクション 6.2で定義されているアクセス評価応答を含む JSON オブジェクトです。

以下は、HTTPS アクセス評価応答の非規範的な例です: 

HTTP/1.1 OK Content-type: application/json X-Request-ID: bfe9eb29-ab87-4ca3-be83-a1d5d8305716 { "decision": true } Figure 15Example of an HTTPS Access Evaluation Response

ここは前回書いた通りtrue/falseだけでいいのか問題は置いておいて、普通かと。

7.1.3. Error Responses

The following error responses are common to all methods of the Authorization API. The error response is indicated by an HTTPS status code (Section 15 of [RFC9110]) that indicates error.

The following errors are indicated by the status codes defined below:

7.1.3.エラー応答

以下のエラー応答は、Authorization API のすべてのメソッドに共通です。エラー応答は、RFC9110セクション15に定義されているHTTPS ステータス コード によって識別されます。

以下のエラーは、以下に定義されているステータス コードによって示されます

Table 1HTTPS Error status codesCodeDescriptionHTTPS Body Content400Bad RequestAn error message string401UnauthorizedAn error message string403ForbiddenAn error message string500Internal errorAn error message string

Note: HTTPS errors are returned by the PDP to indicate an error condition relating to the request or its processing, and are unrelated to the outcome of an authorization decision, which is always returned with a 200 status code and a response payload.

To make this concrete: * a 401 HTTPS status code indicates that the caller (policy enforcement point) did not properly authenticate to the PDP - for example, by omitting a required Authorization header, or using an invalid access token. * the PDP indicates to the caller that the authorization request is denied by sending a response with a 200 HTTPS status code, along with a payload of { "decision": false }.

注: HTTPS エラーは、リクエストまたはその処理に関連するエラー状態を示すために PDP によって返され、200ステータス コードと応答ペイロードとともに常に返される承認決定の結果とは無関係です

具体的には、次のようになります。 * 401HTTPS ステータス コードは、呼び出し元 (ポリシー適用ポイント) が PDP に対して適切に認証しなかったことを示します (必要なヘッダーを省略した、無効なアクセス トークンを使用したなど)。 * PDP は、 HTTPS ステータス コード200とペイロード{ "decision": false }を含むAuthorization応答を送信することで、呼び出し元に認可要求が拒否されたことを示します

注釈にもある通り、あくまでリクエストに対するHTTPSのステータスを示すレスポンスなので、認可決定に関するtrue/falseとは関係ないことに注意です。

7.1.4. Request Identification

All requests to the API MAY have request identifiers to uniquely identify them. The API client (PEP) is responsible for generating the request identifier. If present, the request identifier SHALL be provided using the HTTPS Header X-Request-ID. The value of this header is an arbitrary string. The following non-normative example describes this header:

7.1.4.リクエストの識別

API へのすべてのリクエストには、リクエストを一意に識別するためのリクエスト識別子が含まれる場合があります。API クライアント (PEP) は、リクエスト識別子を生成する責任があります。存在する場合、リクエスト識別子は HTTPS ヘッダーを使用して提供される必要がありますX-Request-ID。このヘッダーの値は任意の文字列です。次の非規範的な例は、このヘッダーについて説明しています: 

POST /access/v1/evaluation HTTP/1.1 Authorization: Bearer mF_9.B5f-4.1JqM X-Request-ID: bfe9eb29-ab87-4ca3-be83-a1d5d8305716 Figure 16Example HTTPS request with a Request Id Header

実装する際は、ここに記載されているリクエストの識別が結構重要なんじゃないかと思います。先に書いた通り、アクセス許可する・しないの非常に重要な決定を含む話なので、しっかりとステータスを管理しておく責任がPEP/PDPの両方にあると思います。

7.1.5. Request Identification in a Response

A PDP responding to an Authorization API request that contains an X-Request-ID header MUST include a request identifier in the response. The request identifier is specified in the HTTPS Response header: X-Request-ID. If the PEP specified a request identifier in the request, the PDP MUST include the same identifier in the response to that request.

The following is a non-normative example of an HTTPS Response with this header:

7.1.5.レスポンスにおけるリクエストの識別

ヘッダーを含む Authorization API リクエストに応答する PDP はX-Request-ID、レスポンスにリクエスト識別子を含める必要があります。リクエスト識別子は、HTTPS レスポンス ヘッダー(X-Request-ID)で指定されます。PEP がリクエストにリクエスト識別子を指定した場合、PDP はそのリクエストへのレスポンスに同じ識別子を含める必要があります

以下は、このヘッダーを含む HTTPS レスポンスの非標準的な例です: 

HTTP/1.1 OK Content-type: application/json X-Request-ID: bfe9eb29-ab87-4ca3-be83-a1d5d8305716 Figure 17Example HTTPS response with a Request Id Header

先に書いた通り、ちゃんとリクエストとレスポンスが紐づいていることを管理することは重要ですね。


ということで現時点のdraftはこんなところです。

非常にシンプルな仕様になっているので実装も簡単だと思いますが、今後のポイントは以下にしてActionやResourceを標準化していくのか、そして実装をどこまで増やすか、というところにかかってくると思います。メガSaaSな人たちが実装してくれるといいんですけどね。

まぁ、引き続き様子は気にしていきたいと思います。



Ben Werdmüller

Hire HTML and CSS people

[Robin Rendle] "Every problem at every company I’ve ever worked at eventually boils down to “please dear god can we just hire people who know how to write HTML and CSS.”" Yes. Co-signed. Speaking of which ... "ProPublica, the nation’s leading nonprofit investigative newsroom, is in search of a full-stack senior product engineer to lead work on our publishing systems and

[Robin Rendle]

"Every problem at every company I’ve ever worked at eventually boils down to “please dear god can we just hire people who know how to write HTML and CSS.”"

Yes. Co-signed.

Speaking of which ...

"ProPublica, the nation’s leading nonprofit investigative newsroom, is in search of a full-stack senior product engineer to lead work on our publishing systems and core website."

I'm looking for an exceptional engineer who cares about the open web to join my team. If that's you - or you know someone who fits this description - there are more details at this link. I'm here to answer any questions!

#Technology

[Link]


Bill Wendels Real Estate Cafe

Talk to Compensation Coach before signing showing agreement to maximize savings

#LegalHackers: Know about the new role emerging at the start of real estate transactions because homebuyers are now being asked to sign an agreement before… The post Talk to Compensation Coach before signing showing agreement to maximize savings first appeared on Real Estate Cafe.

#LegalHackers: Know about the new role emerging at the start of real estate transactions because homebuyers are now being asked to sign an agreement before…

The post Talk to Compensation Coach before signing showing agreement to maximize savings first appeared on Real Estate Cafe.

Thursday, 26. September 2024

IdM Laboratory

AuthZENのAuthorization APIとは(4)

こんにちは、富士榮です。 引き続きAuthZEN WGのAuthorization API 1.0のdraftを見ていきます。 https://openid.net/specs/authorization-api-1_0-01.html 今回はいよいよPEP(ポリシー実施ポイント)とPDP(ポリシー決定ポイント)の間のAPI仕様の部分です。 6. Access Evaluation API The Access Evaluation API defines the message exchange pattern between a client (PEP) and an authorization service (PDP) for executing a single access evaluation. 6.アクセス評価API アクセス評価 API
こんにちは、富士榮です。
引き続きAuthZEN WGのAuthorization API 1.0のdraftを見ていきます。 https://openid.net/specs/authorization-api-1_0-01.html
今回はいよいよPEP(ポリシー実施ポイント)とPDP(ポリシー決定ポイント)の間のAPI仕様の部分です。
6. Access Evaluation API The Access Evaluation API defines the message exchange pattern between a client (PEP) and an authorization service (PDP) for executing a single access evaluation.
6.アクセス評価API アクセス評価 API は、単一のアクセス評価を実行するためのクライアント (PEP) と認可サービス (PDP) 間のメッセージ交換パターンを定義します

 まずはリクエストからです。

6.1. The Access Evaluation API Request

The Access Evaluation request is a 4-tuple constructed of the four previously defined entities:

subject: REQUIRED. The subject (or principal) of type Subject action: REQUIRED. The action (or verb) of type Action. resource: REQUIRED. The resource of type Resource. context: OPTIONAL. The context (or environment) of type Context.

6.1.アクセス評価APIリクエスト

アクセス評価リクエストは、以前に定義された4つのエンティティから構成される4つのタプルです: 

subject: 必須。Subject 型のサブジェクト(またはプリンシパル) action: 必須。アクション型のアクション(または動詞) resource: 必須。リソースタイプのリソース context: オプション。Context 型のコンテキスト (または環境) 

{ "subject": { "type": "user", "id": "alice@acmecorp.com" }, "resource": { "type": "account", "id": "123" }, "action": { "name": "can_read", "properties": { "method": "GET" } }, "context": { "time": "1985-10-26T01:22-07:00" } } Figure 9Example Request

データセットは前回見てきた通りですね。

誰(subject)が何(resource)に対して何を(action)ができるかを問い合わせるわけですね。上記の例だとalice@acmecorp.comという主体が123というアカウントの情報を読み取ることができるか?ということを問い合わせています。もちろんコンテキストの情報も判断基準になります。

次はレスポンスです。

6.2. The Access Evaluation API Response

The simplest form of a response is simply a boolean representing a Decision, indicated by a "decision" field.

decision: REQUIRED. A boolean value that specifies whether the Decision is to allow or deny the operation.

In this specification, assuming the evaluation was successful, there are only 2 possible responses:

true: The access request is permitted to go forward. false: The access request is denied and MUST NOT be permitted to go forward.

The response object MUST contain this boolean-valued Decision key.

6.2.アクセス評価APIレスポンス

応答の最も単純な形式は、decisionフィールドによって示される決定を表すブール値です

decision: 必須。Decision が操作を許可するか拒否するかを指定するブール値

この仕様では、評価が成功したと仮定すると、可能な応答は2つだけです: 

true: アクセス要求は続行が許可されます false: アクセス要求は拒否され、続行を許可してはなりません

レスポンスオブジェクトには、このブール値の Decision キーが含まれている必要があります

{ "decision": true } Figure 10Example Decision

決定された結果がdecisionとしてtrue/falseで返却されるわけですね。

前回のデータセットのactionの部分でcan_readなどはリクエストに使ってレスポンスは単純にbooleanで返すわけです。まぁシンプルではありますが、リクエストのactionのパターンをどこまで標準として定義できるかどうか?にかかってきそうです。

6.2.2. Additional Context in a Response

In addition to a "decision", a response may contain a "context" field which can be any JSON object. This context can convey additional information that can be used by the PEP as part of the decision evaluation process. Examples include:

XACML's notion of "advice" and "obligations" Hints for rendering UI state Instructions for step-up authentication

6.2.2.レスポンス内の追加コンテキスト "decision"に加えて、レスポンスには任意の JSON オブジェクトの"context"フィールドが含まれる場合があります。このコンテキストは、PEP が意思決定評価プロセスの一部として使用できる追加情報を伝えることができます。例: 

XACML の「アドバイス」と「義務」の概念 UI 状態をレンダリングするためのヒント ステップアップ認証の手順

 responseには追加の情報を含めることができるんですね。これによりPEP側でとるべきアクションをある程度標準として定義しておくことができそうです。

6.2.3. Example Context

An implementation MAY follow a structured approach to "context", in which it presents the reasons that an authorization request failed.

A list of identifiers representing the items (policies, graph nodes, tuples) that were used in the decision-making process. A list of reasons as to why access is permitted or denied.

6.2.3.コンテキストの例

実装は、認可リクエストが失敗した理由を提示する"context"に対する構造化されたアプローチに従っても良い 

意思決定プロセスで使用された項目 (ポリシー、グラフ ノード、タプル) を表す識別子のリスト  アクセスが許可または拒否される理由のリスト

要するに単にtrue/falseの結果だけではなく理由などの情報をレスポンスに含めることができるということですね。

6.2.3.1. Reasons

Reasons MAY be provided by the PDP.

6.2.3.1.1. Reason Field

A Reason Field is a JSON object that has keys and values of type string. The following are non-normative examples of Reason Field objects:

6.2.3.1.理由

理由は PDP によって提供される場合があります

6.2.3.1.1.理由フィールド

Reason フィールドは、String型のキーと値を持つ JSON オブジェクトです。以下は、Reason フィールド オブジェクトの非規範的な例です: 

{ "en": "location restriction violation" } 

Figure 11Example Reason 

主に失敗の場合の理由の提供方法ですね。

6.2.3.1.2. Reason Object

A Reason Object specifies a particular reason. It is a JSON object that has the following fields:

id: REQUIRED. A string value that specifies the reason within the scope of a particular response. reason_admin: OPTIONAL. The reason, which MUST NOT be shared with the user, but useful for administrative purposes that indicates why the access was denied. The value of this field is a Reason Field object (Section 6.2.3.1.1). reason_user: OPTIONAL. The reason, which MAY be shared with the user that indicates why the access was denied. The value of this field is a Reason Field object (Section 6.2.3.1.1).

The following is a non-normative example of a Reason Object:

6.2.3.1.2.理由オブジェクト

Reason オブジェクトは特定の理由を指定します。これは次のフィールドを持つ JSON オブジェクトです: 

id: 必須。特定の応答の範囲内で理由を指定する文字列値 reason_admin: オプション。ユーザーと共有してはならない理由ですが、アクセスが拒否された理由を示す管理目的に役立ちます。このフィールドの値は、Reason Field オブジェクト (セクション 6.2.3.1.1 )です reason_user: オプション。アクセスが拒否された理由をユーザーと共有する場合があります。このフィールドの値は、Reason Field オブジェクト (セクション 6.2.3.1.1 )です

以下は、Reason オブジェクトの非規範的な例です: 

{ "id": "0", "reason_admin": { "en": "Request failed policy C076E82F" }, "reason_user": { "en-403": "Insufficient privileges. Contact your administrator", "es-403": "Privilegios insuficientes. Póngase en contacto con su administrador" } } Figure 12Example of a Reason Object
{ "decision": true, "context": { "id": "0", "reason_admin": { "en": "Request failed policy C076E82F" }, "reason_user": { "en-403": "Insufficient privileges. Contact your administrator", "es-403": "Privilegios insuficientes. Póngase en contacto con su administrador" } } } Figure 13Example Response with Context

理由は割と詳しく書くことができるようです。

ただ現状はここまでしか定義がないので、実際に中身の書き方をどうするか、や言語セットのデフォルトの扱いをどうするのか、、などはこれから決めていかないといけないと思います。


ということでAPI仕様はここまでです。

次回はTransportです。


Wrench in the Gears

What If The Totalitarian Game Is Actually A Participatory Play? Musings on CogNarr, Eusociality, and Synanon

Yesterday I put together a 40-minute video in which I talk about Karl Friston’s “Free Energy Principle,” a theory developed in the 1990s that centers the unique narrative world models we hold internally and how those models in consciousness are shaped by sensory input, the incoming information hurtling towards us through the feed every day. [...]

Yesterday I put together a 40-minute video in which I talk about Karl Friston’s “Free Energy Principle,” a theory developed in the 1990s that centers the unique narrative world models we hold internally and how those models in consciousness are shaped by sensory input, the incoming information hurtling towards us through the feed every day. I relate Friston’s work to a new effort by John Boik, a Houston-based bioengineer with a background in Oriental medicine who is now pursuing open source software in AI assisted direct democracy and community currency programs. If you’ve been following my work around Michael Levin’s research you can probably see the connection between Boik’s former career in cancer research and cooperative digital money systems – morphogenetic fields linked to expanding cognitive light cones.

In a recent presentation for the Active Inference Institute, based at UC Davis in Daniel Ari Friedman’s Entomology Lab (Deborah Gordon, Stanford ant behavior expert is an advisor), Boik laid out his plans for CogNarr, a system that would structure human “stories” in ways that would be intelligible to computers and facilitate collaboration at a scale of a half dozen people to several million. This parallels Thomas Seeley’s work on distributed decision-making described in his book “Honeybee Democracy.”

I am settling in here in Hot Springs and I met a lovely elder who placed in my hands a memoir written by her son who is around my age. It describes his educational journey as a divergent learner and his family’s decade-long involvement with the “utopian” community Synanon based in Marin County, CA with ties to influential group psychologists and likely an experiment tied to the Human Ecology / MK Ultra Project. The main campus was on the site of an old lodge built by Marconi for wireless telegraphy.

While founded as an addiction treatment program that used attack therapy, “The Game,” the community later attracted intellectuals who were interested in a collective “utopian” lifestyle. I believe Synanon was a testbed for exploring the opportunities and limits of a hive / colony-oriented communal behavior. In other words Synanon was an ant computer lab. I get the feeling that while a lot of media attention was paid to increasingly unhinged adults with weapon stockpiles, including a rattlesnake placed in a lawyer’s mailbox, the primary area of investigation may have been the children.

It’s funny, with the cooler weather I’ve been enjoying my sunporch, but I’ve found that I have to keep an eye on the sugar ants. There are regularly 3-4 foragers exploring my Ikea Kallax furniture. One afternoon I got preoccupied with sewing and left a plate of sliced apples unattended for a few hours. Yowza – pheromonic stigmergy in action! It was shocking, but also quite fascinating to see how the superorganism works up close and personal. This video that is part of the Santa Fe Institute podcast offers helpful insights into this dynamic: “The Physics of Collectives.” I keep pushing the idea of the ant computer in front of people, because we are trained into linguistic and conceptual boxes that fuel polarity – libertarianism vs. socialism, etc. etc. when most of the roads are all using custom stories to take us to the same place – Conway’s Game of Life, counting down the iterations until we get the gliders coming on.

Abraham Maslow was involved with Synanon as well as being affiliated with Julius Stulman, Fritz Kunz, and Ervin Laszlo via the Foundation for Integrated Education. Maslow contributed a chapter to Stulman and Laszlo’s book “Emergent Man.” Synanon shut its doors in the 1990s, but spun out other programs including an abusive pioneer in the “troubled teen” space CEDU. Paris Hilton is an alumna of that program. Rory Kennedy, Bobby’s youngest sister, produced a documentary on Synanon that was released this spring by HBO. I speculate that this documentary as well as last year’s Netflix expose on San Patrignano are pre-bunking efforts to neutralize resistance to the institutional “wellness,” addiction and mental health treatment programs Bobby has planned for rural areas across the United States. My gut tells me these centers, like refugee camps, will be testbeds for tokenized eusocial behavioral experiments leading to advancements in collective decision making using emerging technologies. Nothing new under the sun, only this time we have nano-photonics in the mix to make social signaling more seamless and intuitive.

Here’s my video:

There were some good questions and comments on this video. Given how those tend to disappear, I chose to transcribe one exchange that provides some additional insight into how I’ve been thinking about things lately. I know my theory may sound far-fetched, but remember, consciousness is a coherence engine. I’ve been pulling in A LOT of disparate information for years. Trying to weave it into a meaningful narrative is tricky. Certainly the plot line that seems to be unfolding is NOTHING I would have ever expected. I’m not saying I’m correct, but I think there is enough information of concern that my thoughts do merit public consideration with respect to ethics, informed consent, and guard rails. The past four years have given me a crash course in the subtle practice of knowledge management refined by well-trained and well-funded social scientists and psychologists. All that I’m asking is that we actually engage in information streams outside of the ones being poured down our throats by mainstream media, academic gatekeepers, and social media feeds. But at the end of the day, I am proposing a pretty far out story. It makes sense that it’s going to be near to impossible to get much traction. Anyway, if you want to immerse yourself in the information fields that brought me to this place, I have linked a ton of my Kumu.io maps. Click away. I’m eager to hear what you come up with. 

Commenter One:

“What I see is the constant reinforcement of being in mind, abdication of physical being, being in your body and developing a relationship with our humanness is the most important connection. I wonder if we stay truly connected to our human-manifestation would that stop people going automatic and getting lost with narrative.”

Commenter Two:

“This right here. There aro a whole slew of us that have been guided towards esoteric info to the point that we’ve started commenting “who’s here in 24?” To 10 plus year old videos. They think the algorithm has discovered that people really like this….I just think this is a test group of heavy users of you tube. A particular…group of people who would be prone to action in the physical realm. It’s a test for how well it neutralizes them. Doesn’t matter to them if it’s real or empowering or helpful or healing….just does it prevent action by a physical being and bend them to spread denying the physical. Many in that group have kids…many may be teaching their children to place the physical 2nd behind mental. And if that’s the case – why not climb into the box and never leave?”

Commenter Three:

“I would be curious to know more about this. So, to clarify, YouTube is pushing videos on a group of people to see if they will remain docile? I’ve often wondered about certain spiritual viewpoints, and if they serve to keep people from sticking up for themselves. Any specifics you have on this subject would be fascinating to me, thanks.”

Alison’s Comment:

“Robert Bosnak’s take is that apocalypse narratives are quite generative. I do see the ascension storyline as running parallel with the Christian imminent apocalypse storyline. Based on experiences I’ve had over the past several years as well as natural cycles, I sense that we don’t have a grasp of what spacetime really is and that it can loop back and feature recursion. If that is the case, the linear narratives of some transformative end point seem likely socially constructed for some purpose I can’t conjecture about. I’m skeptical of both. If you try on the theory I’ve been developing around heart-soul-based nested and networked human massive parallel processing with access to the field through quantum biology and companion microbiome management, then the system would need two phases – a foraging out-of-body ego-death phase where individuals or groups go into the field to retrieve information which is then processed in-vivo with support from bacterial colonies that live in and on us. I sense that both are required, but training and practice will be needed so people can do it as second nature.

Again, in both apocalypse narratives, the storyline and teaming aspect is super important. Consider the industries that have been built up around both from New Age TV and webinars and books, etc. to Christian megachurches and of course the Vatican’s deep dive into social impact and biotech and Oral Roberts setting up Bible verse magic AI mirrors in student dorms. It never occurred to me that “social impact” was meant to be a hub for ongoing remote cognitive-biological monitoring (now for “wellness”) that will enable society to become a noetic superorganism available for spiritual on-demand information processing. Just like I believe the Georgia Guidestones were an engineered memetic trope to focus negative energy and fear. I also think the apocalypse stories are meant to structure consciousness so that when the intended program rolls out it can be embedded in a dramatic arc each particular groomed group is already comfortable with. The question though…is it a cookbook?”

Commenter One Follow Up:

“So with the embodied imagination, which sounds a bit like shamanic journeying, it is the mind which is the ‘driver’ to facilitate this third state which is embodied. In the mirror’d virtual world it is presumably still the original mind inhabiting this new embodiment? I wonder also what happens to the energy field of the actual heart & body in the new embodiment. Is this measured by the galvanic response also? Narrative means each group (being coalesced) can happily volunteer to inhabit this state and the body can then be harnessed and utilised.”

Alison’s Reply:

“I just wanted to state I am working through this theory and don’t have everything. buttoned up – but as I picture it, it would be the physical body and etheric body working in tandem to facilitate the information processing – two cycles. So picture a bee scouting a site for a new hive. Bees do this “waggle dance” to communicate what they’ve found and make a case to their peers about why their location is good. Then, by consensus (think tokenized democracy) the bees come to a decision. So maybe we are using our etheric self to forage for information in the field, and we bring it back to “the collective,” but that collective isn’t a hive of bees, but rather a cosmos of microbiota and mitochondria who communicate not through dance vibrations, but perhaps through photonics in the fascia? Now picture all of us as agents linked in a parallel processing system via frequency facilitated by electroceuticals and liposomes, etc. in our “wellness” supplements. I’m kind of spitballing here and pardon me if I’m misinterpreting your framing, but I don’t think it’s – hey send your consciousness out into the ether and let the system use your physical body while you are gone. There is the issue of embodiment, narrative coherence, umwelt, and qualia. My logical self says that individuals need to maintain their integrity as agents over time in order to be productive in the collective decision making process. I think they want to be able to nest individual experiences and then aggregate them at a scale relevant to the particular “cognitive light cone” task. Think about ants – they are agents working within a social structure that facilitates complex emergence even if the individual ants have no clue what is going on at levels about their pay grade. As I have been researching Julian Gresser’s “intelligent heart” it seems that the torus field is useful for processing 4-D information. So that may be another reason, beyond the role of the microbiome, to integrate knowledge acquired through astral projection – to integrate the information in a space outside regular 3-D existence.

Also, Bosnak is very hip on group dreaming in VR “Asclepian” healing caves. My intuition tells me that tokens and the protocol layer could be an interdimensional language of manifestation. This paper shared with me today by Stephers has some interesting thoughts on consciousness and projecting geometry (sacred?). https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2020.01680/full

Interactive Map: https://embed.kumu.io/745bb8ce3aad470199efea9e9528807d#untitled-map?s=bm9kZS1aUERrT3Vxbg%3D%3D

I mention psychodrama and Gurdjieff in the video. Here is a map I’ve been working on related to this:

Interactive map JG Bennet and Gurdjieff’s “The Work”: https://embed.kumu.io/0bf33678ed9f4126e528c09beb40370e#untitled-map?s=bm9kZS1RdE5jbkJ0Tg%3D%3D

This map links Maslow’s theories of self-actualization with Ocean Protocol (Trent McConaghy and Jim Rutt’s Game B), token engineering, game mechanics, sustainability, and AGI:

Interactive Map – Maslow Self-Actualization: https://embed.kumu.io/4011b2b1901acf8b24cbb1119bbb9e1c#untitled-map?s=bm9kZS1ObjBud3VEdg%3D%3D Source: https://embed.kumu.io/1ff16f690770c1d381e267f258358a32#untitled-map?s=bm9kZS1mVjhCN2ZOUw%3D%3D Source: https://archive.org/details/emergentmanhisch00stul/mode/2up Source: https://archive.org/details/emergentmanhisch00stul/mode/2up Source: https://archive.org/details/emergentmanhisch00stul/mode/2up

Source: https://embed.kumu.io/51fe5ee0c16765f2c748936cc86c28ee#untitled-map?s=bm9kZS1JbG83a1dOZQ%3D%3D Source: https://web.archive.org/web/20210531074023/https://littlesis.org/oligrapher/6930-arizona-state-games-and-education Source: https://embed.kumu.io/035ea6860f513bab36fa1a9815d93a74#untitled-map?s=bm9kZS1ENDF4aWZZZA%3D%3D Source: https://embed.kumu.io/d393e288b03d6a5ef05fc9bc6e51a6a6#untitled-map?s=bm9kZS05SndwcVNqQg%3D%3D Source: https://embed.kumu.io/f77a772b1d3ed24cf4140d0a3d270348#untitled-map?s=bm9kZS1HR0M1SWphVg%3D%3D Source: https://principledsocietiesproject.org/about_psp

Interactive map: https://embed.kumu.io/f662cd5969e9debc347b10601a5e3282#untitled-map?s=bm9kZS1Tc2dLUmd3Uw%3D%3D Source: https://embed.kumu.io/097cefb708139ffa65e906e397da3824#untitled-map?s=bm9kZS1hVzVyR1FaeA%3D%3D Source: https://embed.kumu.io/89566f995693843319e1b4970dac9dd1#untitled-map?s=bm9kZS1VSk9pY0JYTA%3D%3D

Lee Eyre, Boik’s collaborator on these papers, worked for Orange (ambient smart environments) for ten years. The video below is how the world looks when everything is run on a permission-based smart contract. This scenario presumes all agents have a digital identifier, which brings in Web3, blockchain, and spatial computing. Watch this with an eye towards interactions feeding into a personal story – qualia, umwelt.

LEDDA (Leda) is an interesting name / acronym choice here. 

Source: https://www.youtube.com/watch?v=z9ZCjd2rqGY Source: https://www.researchgate.net/lab/Active-Inference-Institute-Daniel-Ari-Friedman Screenshot

 

 

Source: https://embed.kumu.io/f03f97508c7993eef8aefa20cd265e98#untitled-map?s=bm9kZS1SUU1ZTkdvdA%3D%3D

 

Source: https://lodgeatmarconi.com/the-lodge/ Source: https://web.archive.org/web/20210110011432/https://littlesis.org/oligrapher/4196-adl-iot-education Source: https://web.archive.org/web/20201025032840/https://littlesis.org/oligrapher/5647-blockchain-and-transmedia-learning Source: https://web.archive.org/web/20210104142455/https://littlesis.org/oligrapher/6018-kneomedia-gamified-edu-tainment Source: https://web.archive.org/web/20210531074023/https://littlesis.org/oligrapher/6930-arizona-state-games-and-education Source: https://littlesis.org/oligrapher/5084-vr-gaming-surveillance-human-capital Source: https://web.archive.org/web/20201204194140/https://littlesis.org/oligrapher/5757-dassault-digital-twin-simulations-blockchain-id Source: https://web.archive.org/web/20210329114848/https://littlesis.org/oligrapher/6637-california-dyslexia-gamification Source: https://web.archive.org/web/20210407065329/https://littlesis.org/oligrapher/6678-oakland-dyslexia-screening-pilot Source: https://web.archive.org/web/20210129211602/https://littlesis.org/oligrapher/6317-biotech-microbiome-blockchain Source: https://web.archive.org/web/20201207235929/https://littlesis.org/oligrapher/5719-produce-rx-food-pay-for-success-medicaid Source: https://web.archive.org/web/20201224191109/https://littlesis.org/oligrapher/6016-wearable-health-tech-impact-investing

 

 

 

 

 


The Pragmatic Engineer

The Pulse #108: Elasticsearch unexpectedly goes open source again

Also: Amazon cuts the number of managers, engineers critiquing YouTube’s biggest critic, and key people keep quitting OpenAI.

The Pulse is a series covering insights, patterns, and trends within Big Tech and startups. Notice an interesting event or trend? Send me a message.

Today, we cover:

Industry pulse. Meta previews next-gen AI glasses, X probably needs to pay severance to engineers it laid off, on-prem datacenter threat for AWS, tech jobs drying up, Arc’s embarrassing secur…

Read more

Wednesday, 25. September 2024

IdM Laboratory

AuthZENのAuthorization APIとは(3)

こんにちは、富士榮です。 引き続きAuthZEN WGのAuthorization API 1.0のdraftを見ていきます。 https://openid.net/specs/authorization-api-1_0-01.html 今回はAPIが扱うデータモデルについてです。 5. Information Model The information model for requests and responses include the following entities: Subject, Action, Resource, Context, and Decision. These are all defined below. 5.情報モデル リクエストとレスポンスの情報モデルには、サブジェクト、アクション、リソース、コンテキスト、決定というエンティティ

こんにちは、富士榮です。

引き続きAuthZEN WGのAuthorization API 1.0のdraftを見ていきます。

https://openid.net/specs/authorization-api-1_0-01.html


今回はAPIが扱うデータモデルについてです。

5. Information Model

The information model for requests and responses include the following entities: Subject, Action, Resource, Context, and Decision. These are all defined below.

5.情報モデル

リクエストとレスポンスの情報モデルには、サブジェクト、アクション、リソース、コンテキスト、決定というエンティティが含まれます。これらはすべて以下で定義されています

前回の概要で述べた通り、Authorization APIはPEP(ポリシー実施ポイント)からの問い合わせを受けて認可に関する判定結果を返却するPDP(ポリシー決定ポイント)として機能します。そのため、対象となるサブジェクト(主体)、認可されるアクション(操作)、認可対象となるリソース、コンテキスト、決定結果を含むことになります。


5.1. Subject

A Subject is the user or robotic principal about whom the Authorization API is being invoked. The Subject may be requesting access at the time the Authorization API is invoked.

A Subject is a JSON ([RFC8259]) object that contains two REQUIRED keys, type and id, which have a value typed string, and an OPTIONAL key, properties, with a value of a JSON object.

type:

REQUIRED. A string value that specifies the type of the Subject.

id:

REQUIRED. A string value containing the unique identifier of the Subject, scoped to the type.

properties:

OPTIONAL. A JSON object containing any number of key-value pairs, which can be used to express additional properties of a Subject.

The following is a non-normative example of a Subject:

5.1.主体

サブジェクトとは、Authorization API が呼び出されるユーザーまたはロボット プリンシパルです。サブジェクトは、Authorization API が呼び出された時点でアクセスを要求している可能性があります

サブジェクトは、文字列の値を持つ2つの必須キー(typeとid)と、オプションとしてJSONオブジェクトの値を持つpropertiesを含むJSONオブジェクトです

type:

必須。Subjectstringのタイプを指定する値

id:

必須。stringにスコープ設定された、Subject の一意の識別子を含むtype。¶

properties:

オプション。任意の数のキーと値のペアを含む JSON オブジェクト。これを使用して、Subject の追加プロパティを表現することができます 

以下は、Subject の非規範的な例です: 

{ "type": "user", "id": "alice@acmecorp.com" } Figure 1Example Subject


まぁ、単純に対象となる主体を表す部分ですね。ポイントはtype属性があり、この例のようにユーザを示す以外にも汎用的な用途とすることが想定されています。

5.1.1. Subject Properties

Many authorization systems are stateless, and expect the client (PEP) to pass in any properties or attributes that are expected to be used in the evaluation of the authorization policy. To satisfy this requirement, Subjects MAY include zero or more additional attributes as key-value pairs, under the properties object.

An attribute can be single-valued or multi-valued. It can be a primitive type (string, boolean, number) or a complex type such as a JSON object or JSON array.

The following is a non-normative example of a Subject which adds a string-valued department property:

5.1.1.主題のプロパティ

多くの認可システムはステートレスであり、認可ポリシーの評価で使用されることが予想されるプロパティまたは属性をクライアント (PEP) が渡すことを期待しています。この要件を満たすために、サブジェクトはpropertiesオブジェクトの下に 0 個以上の追加属性をキーと値のペアとして含めることができます

属性は単一値または複数値にすることができます。プリミティブ型 (文字列、ブール値、数値) または JSON オブジェクトや JSON 配列などの複合型にすることができます

以下は、文字列値のdepartmentプロパティを追加する Subject の非規範的な例です

{ "type": "user", "id": "alice@acmecorp.com", "properties": { "department": "Sales" } } Figure 2Example Subject with Additional Property


PEPからのリクエストに応じて認可決定をするわけなので、そのために必要となる主体に関する情報が欲しいわけです。この例では部署属性をもらっているわけですね。まぁ、本当に難しいのはPEPとPDPでこのpropertiesの中身の値について標準化・同意をとっておくこと、そしてマスターデータが最新状態として同期されていることなのですが。

その意味で相互運用性を保つために幾つかのpropertiesはあらかじめ定義をしておこうとしています。

5.1.1.1. IP Address

The IP Address of the Subject, identified by an ip_address field, whose value is a textual representation of an IP Address, as defined in Textual Conventions for Internet Network Addresses [RFC4001].

The following is a non-normative example of a subject which adds the ip_address property:

5.1.1.1. IPアドレス

サブジェクトのIPアドレス。フィールドによって識別され、その値は[ RFC4001 ]で定義されているIPアドレステキスト表現です。

以下は、ip_addressプロパティを追加するサブジェクトの非規範的な例です: 

{ "type": "user", "id": "alice@acmecorp.com", "properties": { "department": "Sales", "ip_address": "172.217.22.14" } } Figure 3Example Subject with IP Address

最初の例がIPアドレスになっているのはよくわかりませんが、ゼロトラストの文脈におけるコンテキストを表現するためにはどのネットワークからの認可要求なのか、は想定されるべき要件だと思います。

5.1.1.2. Device ID

The Device Identifier of the Subject, identified by a device_id field, whose value is a string representation of the device identifier.

The following is a non-normative example of a subject which adds the device_id property:

5.1.1.2.デバイスID

サブジェクトのデバイス識別子。device_idフィールドによって識別され、その値はデバイス識別子の文字列表現です

以下は、device_idプロパティを追加するサブジェクトの非規範的な例です: 

{ "type": "user", "id": "alice@acmecorp.com", "properties": { "department": "Sales", "ip_address": "172.217.22.14", "device_id": "8:65:ee:17:7e:0b" } } Figure 4Example Subject with Device ID

やっぱりゼロトラストをかなり意識した作りになっている感じがしますね。

次はリソースの話です。

5.2. Resource

A Resource is the target of an access request. It is a JSON ([RFC8259]) object that is constructed similar to a Subject entity. It has the follow keys:

type:

REQUIRED. A string value that specifies the type of the Resource.

id:

REQUIRED. A string value containing the unique identifier of the Resource, scoped to the type.

properties:

OPTIONAL. A JSON object containing any number of key-value pairs, which can be used to express additional properties of a Resource.

5.2.リソース

リソースはアクセス要求の対象です。これは、Subject エンティティと同様に構築された JSON オブジェクトです。次のキーがあります: 

type:

必須。stringリソースのタイプを指定する値

id:

必須。stringリソースの一意の識別子を含む値type。

properties:

オプション。任意の数のキーと値のペアを含む JSON オブジェクト。これを使用して、リソースの追加プロパティを表現することができます

こちらもサブジェクトと同じような構造ですね。

5.2.1. Examples (non-normative)

The following is a non-normative example of a Resource with a type and a simple id:

5.2.1.例(非規範的)

type以下は、と単純なを持つリソースの非規範的な例ですid: 

{ "type": "book", "id": "123" } Figure 5Example Resource

5.1のサブジェクトのところとExampleの章立て構造が異なるので若干読みづらさを感じますが、この辺りは今後のUpdateに期待というところです。

The following is a non-normative example of a Resource containing a library_record property, that is itself a JSON object:

library_record以下は、それ自体が JSON オブジェクトであるプロパティを含むリソースの非規範的な例です

{ "type": "book", "id": "123", "properties": { "library_record":{ "title": "AuthZEN in Action", "isbn": "978-0593383322" } } } Figure 6Example Resource with Additional Property

認可対象となるリソースの表現についてもpropertiesで細かく制御ができるようになっています。ただ、ここでいきなり本のサンプル??って思うところもあるのでゼロトラスト文脈でサンプルは統一した方がわかりやすいんじゃないかな?とは思いました。

5.3. Action

An Action is the type of access that the requester intends to perform.

Action is a JSON ([RFC8259]) object that contains a REQUIRED name key with a string value, and an OPTIONAL properties key with a JSON object value.

name:

REQUIRED. The name of the Action.

properties:

OPTIONAL. A JSON object containing any number of key-value pairs, which can be used to express additional properties of an Action.

The following is a non-normative example of an action:

5.3.アクション

アクションは、リクエスト者が実行しようとしているアクセスのタイプです

アクションは、値を持つ必須のキーと、JSON オブジェクト値を持つオプションのキーを含むJSON オブジェクトです

name:

必須。アクションの名前

properties:

オプション。任意の数のキーと値のペアを含む JSON オブジェクト。アクションの追加プロパティを表現するために使用できます

以下はアクションの非規範的な例です:  

{ "name": "can_read" } Figure 7Example Action


まぁ、単純にどんなアクションが許可されるのか?という表現ですね。

ここも値の標準化が肝要なところですので、ある程度決めうちで標準化しようとしています。

5.3.1. Common Action Values

Since many services follow a Create-Read-Update-Delete convention, a set of common Actions are defined. That said, an Action may be specific to the application being accessed or shared across applications but not listed in the common Actions below.

The following common Actions are defined:

can_access: A generic Action that could mean any type of access. This is useful if the policy or application is not interested in different decisions for different types of Actions. can_create: The Action to create a new entity, which MAY be defined by the resource field in the request. can_read: The Action to read the content. Based on the Resource being accessed, this could mean a list functionality or reading an individual Resource's contents. can_update: The Action to update the content of an existing Resource. This represents a partial update or an entire replacement of an entity that MAY be identified by the Resource in the request. can_delete: The Action to delete a Resource. The specific entity MAY be identified by the Resource in the request.

PDP Policies MAY incorporate common Action names to provide different decisions based on the Action.

5.3.1.共通アクション値

多くのサービスは作成、読み取り、更新、削除の規則に従うため、一連の共通アクションが定義されています。ただし、アクションはアクセスされるアプリケーションに固有のものである場合や、アプリケーション間で共有される場合もありますが、以下の共通アクションにはリストされていません

以下の共通アクションが定義されています: 

can_access: あらゆるタイプのアクセスを意味する可能性のある汎用アクション。これは、ポリシーまたはアプリケーションが、異なるタイプのアクションに対して異なる決定を行う必要がない場合に便利です can_create: 新しいエンティティを作成するアクション。resourceリクエスト内のフィールドによって定義される場合があります can_read: コンテンツを読み取るアクション。アクセスされるリソースに基づいて、これはリスト機能または個々のリソースのコンテンツの読み取りを意味する場合があります can_update: 既存のリソースのコンテンツを更新するアクション。これは、リクエスト内のリソースによって識別される可能性のあるエンティティの部分的な更新または全体の置き換えを表します can_delete: リソースを削除するアクション。特定のエンティティは、リクエスト内のリソースによって識別される場合があります

PDP ポリシーには、アクションに基づいて異なる決定を提供するために共通のアクション名が組み込まれる場合があります 

共通の値として手始めにCRUD(Create/Read/Update/Delete)を中心に設計をしているようですが、PEPからのリクエスト方法にも依存しそうです。単にアクセス権を教えて、と言われたらcan_readという回答になる可能性はありますが、更新可能?っと聞かれたらtrue/falseで回答した方が特にエッジの機器を考えると軽量で済む気もします。

この辺りはフィードバックしてもいいのかもしれません。

5.4. Context

The Context object is a set of attributes that represent environmental or contextual data about the request such as time of day. It is a JSON ([RFC8259]) object.

The following is a non-normative example of a Context:

 5.4.コンテキスト

Context オブジェクトは、時刻などのリクエストに関する環境またはコンテキストデータを表す属性のセットです。これは JSON オブジェクトです

以下はコンテキストの非規範的な例です: 

{ "time": "1985-10-26T01:22-07:00" } Figure 8Example Context

うーん、ABACを考えるとコンテキストは非常に重要なんですが、5.1でサブジェクトのpropertiesとして設定する部分とコンテキストに指定する部分の切り分けが難しくなってしまっているような気がします。IPアドレスやデバイスIDなどは上記の例の時間と合わせて動的に変化しうる環境属性と言える場合もあるので、コンテキストに入れた方が良いのではないかと個人的には思います。

この辺もフィードバックかもしれません。


ということで、今回はデータモデルの部分について読んでいきました。この後はAPIそのものについて見ていきます。





 


The Pragmatic Engineer

AI tools for software engineers, but without the hype – with Simon Willison (co-creator of Django)

Simon is one of the best-known software engineers experimenting with LLMs to boost his own productivity: he’s been doing this for more than three years, blogging about it in the open.

The first episode of The Pragmatic Engineer Podcast is out. Expect similar episodes every other Wednesday. You can add the podcast in your favorite podcast player, and have future episodes downloaded automatically.

Listen now on Apple, Spotify, and YouTube.

Brought to you by:

Codeium: ​​Join the 700K+ developers using the IT-approved AI-powered code assistant.

TLDR: Keep up with tech in 5 minutes

On the first episode of the Pragmatic Engineer Podcast, I am joined by Simon Willison.

Simon is one of the best-known software engineers experimenting with LLMs to boost his own productivity: he’s been doing this for more than three years, blogging about it in the open.

Simon is the creator of Datasette, an open-source tool for exploring and publishing data. He works full-time developing open-source tools for data journalism, centered on Datasette and SQLite. Previously, he was an engineering director at Eventbrite, joining through the acquisition of Lanyrd, a Y Combinator startup he co-founded in 2010. Simon is also a co-creator of the Django Web Framework. He has been blogging about web development since the early 2000s.

In today’s conversation, we dive deep into the realm of Gen AI and talk about the following: 

Simon’s initial experiments with LLMs and coding tools

Why fine-tuning is generally a waste of time—and when it’s not

RAG: an overview

Interacting with GPTs voice mode

Simon’s day-to-day LLM stack

Common misconceptions about LLMs and ethical gray areas 

How Simon’s productivity has increased and his generally optimistic view on these tools

Tips, tricks, and hacks for interacting with GenAI tools

And more!

I hope you enjoy this epsiode.

In this episode, we cover:

(02:15) Welcome

(05:28) Simon’s ‘scary’ experience with ChatGPT

(10:58) Simon’s initial experiments with LLMs and coding tools

(12:21) The languages that LLMs excel at

(14:50) To start LLMs by understanding the theory, or by playing around?

(16:35) Fine-tuning: what it is, and why it’s mostly a waste of time

(18:03) Where fine-tuning works

(18:31) RAG: an explanation

(21:34) The expense of running testing on AI

(23:15) Simon’s current AI stack 

(29:55) Common misconceptions about using LLM tools

(30:09) Simon’s stack – continued 

(32:51) Learnings from running local models

(33:56) The impact of Firebug and the introduction of open-source 

(39:42) How Simon’s productivity has increased using LLM tools

(41:55) Why most people should limit themselves to 3-4 programming languages

(45:18) Addressing ethical issues and resistance to using generative AI

(49:11) Are LLMs are plateauing? Is AGI overhyped?

(55:45) Coding vs. professional coding, looking ahead

(57:27) The importance of systems thinking for software engineers 

(1:01:00) Simon’s advice for experienced engineers

(1:06:29) Rapid-fire questions

Some takeaways:

If you are not using LLMs for your software engineering workflow, you are falling behind. So use them! Simon outlined a bunch a of reasons that hold back many devs from using these tools – like ethical concerns, or energy concerns. But LLM tools are here to stay, and those who use them get more productive.

It takes a ton of effort to learn how to use these tools efficiently. As Simon puts it: “You have to put in so much effort to learn, to explore and experiment and learn how to use it. And there's no guidance.” Also, in related research we did in The Pragmatic Engineer about AI tools, with about 200 software engineers responding, we saw some similar evidence. Those who have not used AI tools for 6 months, were more likely to be negative in their perception of these. In fact, a very common feedback from engineers not using these tools was “I used it a few times, but it didn’t live up to my expectations, and so I’m not using it any more”

Use local models to learn more about LLMs. Running local models has two bigger benefits:

Tou figure out how to do these! It’s less complicated than one would think, thanks to tools like HuggingFace. Go and play around with them, and try out a smaller local model.

You learn a LOT more about how LLMs work, thanks to local models being less capable. So it feels less “magic”. As Simon said, “ I think it's really useful to have a model hallucinate at you early because it helps you get that better mental model of, of, of what it can do. And the local models hallucinate wildly.”

Where to find Simon Willison:

• X: https://x.com/simonw

• LinkedIn: https://www.linkedin.com/in/simonwillison/

• Website: https://simonwillison.net/

• Mastodon: https://fedi.simonwillison.net/@simon

Referenced:

• Simon’s LLM project: https://github.com/simonw/llm

• Jeremy Howard’s Fast Ai: https://www.fast.ai/

• jq programming language: https://en.wikipedia.org/wiki/Jq_(programming_language)

• Datasette: https://datasette.io/

• GPT Code Interpreter: https://platform.openai.com/docs/assistants/tools/code-interpreter

• Open Ai Playground: https://platform.openai.com/playground/chat

• Advent of Code: https://adventofcode.com/

• Rust programming language: https://www.rust-lang.org/

• Applied AI Software Engineering: RAG: https://newsletter.pragmaticengineer.com/p/rag

• Claude: https://claude.ai/

• Claude 3.5 sonnet: https://www.anthropic.com/news/claude-3-5-sonnet

• ChatGPT can now see, hear, and speak: https://openai.com/index/chatgpt-can-now-see-hear-and-speak/

• GitHub Copilot: https://github.com/features/copilot

• What are Artifacts and how do I use them?: https://support.anthropic.com/en/articles/9487310-what-are-artifacts-and-how-do-i-use-them

• Large Language Models on the command line: https://simonwillison.net/2024/Jun/17/cli-language-models/

• Llama: https://www.llama.com/

• MLC chat on the app store: https://apps.apple.com/us/app/mlc-chat/id6448482937

• Firebug: https://en.wikipedia.org/wiki/Firebug_(software)#

• NPM: https://www.npmjs.com/

• Django: https://www.djangoproject.com/

• Sourceforge: https://sourceforge.net/

• CPAN: https://www.cpan.org/

• OOP: https://en.wikipedia.org/wiki/Object-oriented_programming

• Prolog: https://en.wikipedia.org/wiki/Prolog

• SML: https://en.wikipedia.org/wiki/Standard_ML

• Stabile Diffusion: https://stability.ai/

• Chain of thought prompting: https://www.promptingguide.ai/techniques/cot

• Cognition AI: https://www.cognition.ai/

• In the Race to Artificial General Intelligence, Where’s the Finish Line?: https://www.scientificamerican.com/article/what-does-artificial-general-intelligence-actually-mean/

• Black swan theory: https://en.wikipedia.org/wiki/Black_swan_theory

• Copilot workspace: https://githubnext.com/projects/copilot-workspace

• Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems: https://www.amazon.com/Designing-Data-Intensive-Applications-Reliable-Maintainable/dp/1449373321

• Bluesky Global: https://www.blueskyglobal.org/

• The Atrocity Archives (Laundry Files #1): https://www.amazon.com/Atrocity-Archives-Laundry-Files/dp/0441013651

• Rivers of London: https://www.amazon.com/Rivers-London-Ben-Aaronovitch/dp/1625676158/

• Vanilla JavaScript: http://vanilla-js.com/

• jQuery: https://jquery.com/

• Fly.io: https://fly.io/

Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email podcast@pragmaticengineer.com.


Ben Werdmüller

Google Discover is sending U.S. news publishers much more traffic. (Social? Still falling.)

[Laura Hazard Owen at NiemanLab] There are some interesting referral statistics embedded in this piece. Facebook referral traffic has fallen more than 40% over the last year; referrals from Reddit have increased by 88%. But the focus is this: "Search traffic, still dominated by Google search, has remained relatively steady during the period, Brad Streicher, sales director a

[Laura Hazard Owen at NiemanLab]

There are some interesting referral statistics embedded in this piece. Facebook referral traffic has fallen more than 40% over the last year; referrals from Reddit have increased by 88%.

But the focus is this:

"Search traffic, still dominated by Google search, has remained relatively steady during the period, Brad Streicher, sales director at Chartbeat, said in a panel at the Online News Association’s annual conference in Atlanta last week. Google Discover — the Google product offering personalized content recommendations via Google’s mobile apps — is increasingly becoming a top referrer, up 13% across Chartbeat clients since January 2023."

I think what's particularly notable here is the shift between kind of product. Google Search, despite the black box nature of its ever-changing algorithm, always felt like it was a part of the open web.

Discover, on the other hand, is an algorithmic recommendation product that tries to proactively give users more of what they want to read. It's much more akin to a Facebook newsfeed than it is a search index. There are likely editors behind the scenes, and a human touch to what gets surfaced. Publishers are even more in the dark about how to show up there than they were about how to rise through search engine rankings.

I'm curious about what this means for the web. Is this just an advertising / walled garden play from a company that wants to maximize advertising revenue and time on platform? Or is it a reflection of the web getting too big and too messy for many users, creating the need for a firmer hand to show them where the good content is? Is it a function of increased skittishness about an open web that might publish content and ideas that aren't brand safe? Or is it just changing user behavior in light of other apps?

Perhaps some elements of all of the above?

#Media

[Link]


Back to Basics

[Paul Bradley Carr] "I’ve worked at (and founded!) my fair share of billionaire-funded publications and I’ve always had a firm rule: You have to be more critical of the people writing the checks (and their cronies) than you are of anyone else. It’s the only way to offset the inherent bias of taking their money." Paul Carr discusses quitting his column at the SF Standard bec

[Paul Bradley Carr]

"I’ve worked at (and founded!) my fair share of billionaire-funded publications and I’ve always had a firm rule: You have to be more critical of the people writing the checks (and their cronies) than you are of anyone else. It’s the only way to offset the inherent bias of taking their money."

Paul Carr discusses quitting his column at the SF Standard because of its newfound apparent shyness when it comes to criticizing tech moguls - which is a serious journalistic flaw when you consider how important said moguls are to the culture and politics of San Francisco.

This is in the wake of fallout from its coverage of Ben Horowitz's conversion to MAGA, to which the subjects publicly objected. The SF Standard's backer, Michael Moritz, is another wealthy tech backer, who has actually been collaborating with Horowitz's partner Marc Andreessen to build a sort of city of the future on repurposed agricultural land in the North Bay.

As Paul points out, there must be a separation of church and state between editorial and business operations in a newsroom in order to maintain journalistic integrity. That doesn't seem to be something every newcomer understands.

#Media

[Link]


Arc was supposed to be a key to The Washington Post’s future. It became a problem instead.

[Dan Kennedy at Media Nation] Dan Kennedy picks up on a detail in Brian Stelter's Atlantic article about troubles at the Washington Post: "The Post’s content-management system, Arc, which was supposed to be a money-maker, had instead turned out to be a drag on the bottom line." He goes on to sing Arc's praises, but notes that 25% of its staff were just laid off, and wonder

[Dan Kennedy at Media Nation]

Dan Kennedy picks up on a detail in Brian Stelter's Atlantic article about troubles at the Washington Post:

"The Post’s content-management system, Arc, which was supposed to be a money-maker, had instead turned out to be a drag on the bottom line."

He goes on to sing Arc's praises, but notes that 25% of its staff were just laid off, and wonders what went wrong there.

Here's what I think happened. There were two parallel forces at play:

Newsrooms are not natural software companies (except for their own ends). Content management systems are a commodity technology.

It's notable that almost every newsroom that has built its own CMS has eventually left it in favor of a platform built by someone else - most commonly WordPress. Sinking resources into building your own means spending money to solve problems that someone else has already solved, and often solved well.

Particularly in tough times for the industry, newsrooms need to be spending money on the things that differentiate them, not by reinventing perfectly good wheels. WordPress isn't zero cost - most newsrooms partner with an agency and a managed hosting provider like WordPress VIP - but it's a lot cheaper than building all those features yourself would be. And the outcome by picking an open source platform is likely higher quality.

The exception is if the way you both think about and present content is radically different to anyone else. If you're truly a beautiful and unique snowflake, then, yes, building your own CMS is a good idea. But there isn't a single newsroom out there that is unique.

Likewise, if I'm a potential customer (and, as it turns out, I am!), I don't know why I'd pick a proprietary platform that's subject to the changing business strategies of its troubled owner over an open source platform which gives me direct ownership over the code and powers a significant percentage of the web. The upside would have to be stratospherically good. Based on sales emails I get that choose to focus on Arc's AI readiness, that case isn't being made.

The outcome is a bit sad. We need newsrooms; we need journalism; we need an informed voting population. Honestly, the Arc bet was worth trying: I can see how a platform play would have been a decent investment. But that doesn't seem to be how it's panned out, to the detriment of its parent.

#Media

[Link]

Tuesday, 24. September 2024

IdM Laboratory

AuthZENのAuthorization APIとは(2)

こんにちは、富士榮です。 では、Authorization APIのImplementer's draftを見ていきましょう。 https://openid.net/specs/authorization-api-1_0-01.html とりあえず前段部分を見ていきます。 Abstract The Authorization API enables Policy Decision Points (PDPs) and Policy Enforcement Points (PEPs) to communicate authorization requests and decisions to each other without requiring knowledge of each other's inner workings. The Authorization
こんにちは、富士榮です。
では、Authorization APIのImplementer's draftを見ていきましょう。 https://openid.net/specs/authorization-api-1_0-01.html

とりあえず前段部分を見ていきます。
Abstract The Authorization API enables Policy Decision Points (PDPs) and Policy Enforcement Points (PEPs) to communicate authorization requests and decisions to each other without requiring knowledge of each other's inner workings. The Authorization API is served by the PDP and is called by the PEP. The Authorization API includes an Evaluation endpoint, which provides specific access decisions. Other endpoints may be added in the future for other scenarios, including searching for subjects or resources.
概要

Authorization API により、ポリシー決定ポイント (PDP) とポリシー適用ポイント (PEP) は、互いの内部動作を知らなくても、認​​可要求と決定を相互に通信できます。Authorization API は PDP によって提供され、PEP によって呼び出されます。Authorization API には、特定のアクセス決定を提供する評価エンドポイントが含まれています。将来的には、サブジェクトやリソースの検索など、他のシナリオ用に他のエンドポイントが追加される可能性があります。

前回書いた通り、Authorization APIはPDP(ポリシー決定ポイント)によって提供されるAPIです。要するにPEP(ポリシー適用ポイント)からの問い合わせに応じて、認可ポリシーを返却する、というAPIですね。本格的に繊細な認可制御をしようとするとリソースやサブジェクトの情報も必要になるので、将来的にはその辺りまで見ていくことになるのかもしれませんね。


1. Introduction

Computational services often implement access control within their components by separating Policy Decision Points (PDPs) from Policy Enforcement Points (PEPs). PDPs and PEPs are defined in XACML and NIST's ABAC SP 800-162. Communication between PDPs and PEPs follows similar patterns across different software and services that require or provide authorization information. The Authorization API described in this document enables different providers to offer PDP and PEP capabilities without having to bind themselves to one particular implementation of a PDP or PEP.

1.はじめに

計算サービスでは、ポリシー決定ポイント (PDP) とポリシー適用ポイント (PEP) を分離することで、コンポーネント内にアクセス制御を実装することがよくあります。PDP と PEP は、XACMLと NIST の ABAC SP 800-162定義されています。PDP と PEP 間の通信は、認可情報を必要とする、または認可情報を提供するさまざまなソフトウェアやサービスで同様のパターンに従います。このドキュメントで説明するAuthorization API を使用すると、さまざまなプロバイダーが、特定の PDP または PEP の実装に縛られることなく、PDP および PEP 機能を提供できます

基本的にはPDP APIの標準化をしていきますよ、ってことですね。


2. Model

The Authorization API is a transport-agnostic API published by the PDP, to which the PEP acts as a client. Possible bindings of this specification, such as HTTPS or gRPC, are described in Transport.

Authorization for the Authorization API itself is out of scope for this document, since authorization for APIs is well-documented elsewhere. For example, the Authorization API's HTTPS binding MAY support authorization using an Authorization header, using a basic or bearer token. Support for OAuth 2.0 ([RFC6749]) is RECOMMENDED.

2.モデル

Authorization API は PDP によって公開されたトランスポートに依存しない API であり、PEP はクライアントとして機能します。HTTPS や gRPC など、この仕様の可能なバインディングについては、トランスポートで説明されています

Authorization API 自体の認可は、API の認可については他の場所で十分に文書化されているため、このドキュメントの範囲外です。たとえば、Authorization API の HTTPS バインディングは、ヘッダーAuthorization、basicまたはトークンを使用した認可をサポートする場合があります。OAuth 2.0 ( [ RFC6749 ] )bearerのサポートが推奨されます

Authorization API自体はトランスポートアグノスティックな設計になっているようです。この辺りは他のAPIと同じく汎用的な思想です。またAuthorization API自体の認可という一瞬混乱する話が出てきますが、PDPサービスを提供するAPIとしてのAuthorization APIと、そのAPIへの認可(OAuth的な意味での認可)をするのはOAuthを使いましょうね、という話です。やっぱりOAuthの「認可」という言葉はここでも混乱を招きますね。クライアントであるPEPへのアクセス権限の付与(委譲)、っていう文脈での「認可」ですね。 

3. Features

The core feature of the Authorization API is the Access Evaluation API, which enables a PEP to find out if a specific request can be permitted to access a specific resource. The following are non-normative examples:

Can Alice view document #123? Can Alice view document #123 at 16:30 on Tuesday, June 11, 2024? Can a manager print?

3.特徴

Authorization API のコア機能はアクセス評価 API であり、これにより PEP は特定のリクエストが特定のリソースへのアクセスを許可されるかどうかを確認できます。以下は非規範的な例です: 

アリスは文書 #123 を閲覧できますか?  アリスは2024年6月11日火曜日の16:30に文書#123を閲覧できますか?  マネージャーは印刷できますか? 

このAPIの特徴はPEPからの問い合わせに対してアクセス評価を行い結果を返却するというところにありそうですね。

4. API Version

This document describes the API version 1. Any updates to this API through subsequent revisions of this document or other documents MAY augment this API, but MUST NOT modify the API described here. Augmentation MAY include additional API methods or additional parameters to existing API methods, additional authorization mechanisms, or additional optional headers in API requests. All API methods for version 1 MUST be immediately preceded by the relative URL path /v1/.

4. APIバージョン

このドキュメントでは、API バージョン 1 について説明します。このドキュメントまたは他のドキュメントの今後の改訂によるこの API の更新では、この API を拡張できますが、ここで説明する API を変更してはなりません。拡張には、追加の API メソッド、既存の API メソッドへの追加のパラメーター、追加の認証メカニズム、または API リクエスト内の追加のオプション ヘッダーが含まれる場合があります。バージョン 1 のすべての API メソッドの直前には、相対 URL パスがなければなりませ/v1/。

まぁ、ここは互換性のためのバージョニングの話なので、このAPIに限った話じゃありませんので省略。


次からデータモデルの話に入っていくのでまずはここまでです。



Doc Searls Weblog

Open-Source Journalism

Fourteenth in the News Commons series. The main work of journalism is producing stories. Questions following that statement might begin with prepositions: on what, of what, about what. But the preposition that matters most is with what. Ideally, that would be with facts. Of course, facts aren’t always available in the pure form that researchers call data. […]

Fourteenth in the News Commons series.

Craig Burton’s view of the open source ecosystem.

The main work of journalism is producing stories.

Questions following that statement might begin with prepositions: on what, of what, about what. But the preposition that matters most is with what.

Ideally, that would be with facts. Of course, facts aren’t always available in the pure form that researchers call data. Instead, we typically have reports, accounts, sightings, observations, memories, and other fuzzy coatings or substitutes for facts.

Craig Burton used to say that he discounted first-hand reports by 50% and withheld trust in second- and third-hand reports completely. In some cases, he didn’t even trust his memory, because he knew (and loved) everyone’s fallibility, including his own.

But still, we need facts, in whatever form. And those come from what we call sources. Those can be anybody or anything.

Let’s look at anything, and into the subset in archives that are not going away.

How much of that is produced by news organizations? And how much of what’s archived is just what is published?

I ask for two reasons.

One is because in this series I have made a strong case, over and over, for archiving everything possible that might be relevant in future news reporting and scholarship.

The other is because I have piles of unpublished source material that informed my writing in Linux Journal. This material is in the following forms:

Text files on this laptop and on various connected and disconnected drives Sound recordings on— Cassette tapes Microcassette tapes Sony MiniDisc disks .mp3, ogg, and other digital files, mostly on external drives Video recordings on— Hi-8 tapes Mini-DV tapes

And I’m not sure what to do with them. Yet.

Open-sourcing them will take a lot of time and work. But they cover the 24 years I wrote for Linux Journal, and matter to the history of Linux and the open source movement (or movements, since there were many, including the original free software movement).

Suggestions welcome.


Ben Werdmüller

More coverage of the Social Web Foundation

More coverage of the Social Web Foundation has been rolling in today. (See my coverage of the announcement over here.) The New Stack: The fediverse has been a critical development in the open web over the past several years, since most of the social media landscape is dominated by centralized platforms — including Meta. If we want the open web to not just survive, but perhaps thrive ag

More coverage of the Social Web Foundation has been rolling in today. (See my coverage of the announcement over here.)

The New Stack:

The fediverse has been a critical development in the open web over the past several years, since most of the social media landscape is dominated by centralized platforms — including Meta. If we want the open web to not just survive, but perhaps thrive again one day, we should all (hopefully including the father of the web) get behind the fediverse and support the Social Web Foundation.

WeDistribute:

“I wish I would’ve started it five years ago,” Evan explains in a call, “We’re seeing growth of ActivityPub in the commercial sector, we want to help guide that work, especially for devs that don’t know how to engage with the Fediverse, or the work that happens in private spaces. As we’re seeing a lot of growth, it’s important to help push that growth forward, we’re really filling in the crack no other organization is doing.”

TechCrunch:

Part of the group’s efforts will be focused on making the fediverse more user-friendly. Though Mastodon offers a service that functions much like Twitter/X, its decentralized nature — meaning there are multiple servers to choose from — makes getting started confusing and difficult for less technical users. Then, much like X, there’s the cold start problem of finding interesting people to follow.

The W3C:

We are happy to share that today the Social Web Foundation launched with a mission to help the fediverse to grow healthy, multi-polar, and financially viable. We are looking forward to continuing to support the work that [Evan Prodromou, Tom Coates, and Mallory Knodel] are planning in the new non-profit foundation for expanding and improving ActivityPub and the fediverse. We are delighted that to the Foundation will be becoming a W3C Member.

Vivaldi:

The Fediverse reminds us of the early days of the Web. We are competing against silos and corporate interests, using a W3C-based open standard and a distributed solution. It’s great that social networking companies are supporting the Fediverse, and Vivaldi is pleased to support Social Web Foundation so that we can once again have a town square free of algorithms and corporate control.

Independent Federated Trust & Safety:

ActivityPub has enabled thousands of platforms to communicate seamlessly across the Fediverse. This framework encourages a healthier online experience by supporting diversity of thought and content while redistributing governance back to the communities that can best serve their members. In an era where centralised networks dominate, the SWF’s commitment to open standards represents a renewed opportunity for a democratic and inclusive web.

And then Evan Prodromou wrote his own post on the launch:

Many people have ideas about what the Fediverse needs to be bigger, safer, and easier to use. But the solutions they propose fall between the cracks of any one implementer or service. We want the SWF to be the entity that takes on those jobs.

Not everyone agrees that the Fediverse needs to be available to more people. That’s OK. And not everyone is going to be comfortable with the mix of commercial and Open Source implementers plus civil society groups that form the support for the SWF. That’s OK too. Hopefully, our work will still benefit you.

Exciting times for the web.


Doc Searls Weblog

On Intelligence

Now that AI is a huge thing, it’s worth visiting what intelligence is, and how we mismeasure it—for example, by trying to measure it at all. I’ve been on this case for a while now, mostly by answering questions ab0ut IQ on Quora. My answer with the most upvotes is this one, to the question […]
My father, Allen H. Searls, scored 159 on an Army IQ test when he re-enlisted to fight in WWII. But that didn’t make him a great fisherman, even though he loved to do it (and scored big with that striped bass, one of the few he ever caught.) Nor did it make him a good speller. (He was awful.) Or a good student. At fifteen he dropped out of high school and went to work as a longshoreman in New York City, commuting to work from New Jersey on a ferry, and then later working high steel construction on the bridge that obsolesced that ferry. But he was a great card player (almost always winning at poker), a math whiz, brilliant at making tools in his shop, and outstanding in many other ways (friend, husband, father) that can’t and shouldn’t be measured. Except maybe his looks. The guy was a 10.

Now that AI is a huge thing, it’s worth visiting what intelligence is, and how we mismeasure it—for example, by trying to measure it at all.

I’ve been on this case for a while now, mostly by answering questions ab0ut IQ on Quora. My answer with the most upvotes is this one, to the question “What is considered a good IQ?” Here is the full text:

What makes an IQ score “good” is the advantage it brings. That’s it.

When I read the IQ questions here in Quora — “How high an IQ do I need to have to become a good hacker?” “Is 128 a good IQ for a nine year old?” — my heart sinks.

IQ tests insult the intelligence of everybody who takes them, by reducing one of the most personal, varied and human qualities to a single number. Worse, they do this most typically with children, often with terrible results.

It is essential to remember that nobody has “an IQ.” Intelligence cannot be measured as if by a ruler, a thermometer or a dipstick. It is not a “quotient.” IQ test scores are nothing more than a number derived from correct answers to puzzle questions on a given day and setting. That’s it.

Yet our culture puts great store in IQ testing, and many actually believe that one’s “IQ” is as easily measured and unchanging as a fingerprint. This is highly misleading and terribly wrong. I speak from ample experience at living with the results of it.

I grew up in the 1950s and ’60s, going a public school system that sorted kids in each grade by a combination of IQ test scores, achievement test scores and teacher judgement. (My mother taught in the same system, so she knew a lot about how it worked, plus the IQ scores of my sister and myself.) After testing well in kindergarten, I was put in the smart kids class, where I stayed through 6th grade, even though my IQ and achievement test scores fell along with my grades, which were worse every year.

In 6th grade the teacher insisted that I was too dumb for his class and should be sent to another one. My parents had me IQ-tested by an independent center that said I was still smart, so I stayed. By 8th grade, however, my IQ score, grades and achievement test scores were so low that the school re-classified me from the “academic” to the “general” track, and shunted me toward the region’s “vocational-technical” high school to learn a “trade” such as carpentry or auto mechanics. I was no longer, as they put it, “college material.”

So my parents decided to take me out of the public school system and send me to a private school. All the ones we visited used IQ tests in their admissions process. I did so poorly at the school I most wanted to attend (because a girl I had a crush on was already headed there) that the school told my parents I was downright stupid, and that it was cruel of them to have high expectations of me. At another school they forgot to time the test, which gave me a chance to carefully answer the questions. I got all of them right. Impressed by my score, the admissions director told my parents they were lucky to have a kid like me. But the school was itself failing, so my parents kept looking.

The school that ended up taking me was short on students, so my IQ score there (which I never learned) wasn’t a factor. I got bad grades and test scores there too, including the SAT. Luckily, I ended up going to a good small private college that took me because it needed out-of-state students and I was willing to commit to an early decision. I did poorly there until my junior year, when I finally developed skilled ways of working with the system.

Since college I’ve done well in a variety of occupations, and in all of them I’ve been grateful to have been judged by my work rather than by standardized tests.

Looking back on this saga, I was lucky to have parents who respected my intelligence without regard for what schools and test scores told them. Other kids weren’t so lucky, getting categorized in ways that shut off paths to happy futures, violating their nature as unique individuals whose true essence cannot be measured. To the degree IQ tests are still used, the violation continues, especially for kids not advantaged by scoring at the right end of the bell curve.

John Taylor Gatto says a teacher’s main purpose is not to add information to a kid’s empty head (the base assumption behind most formal schooling, ) but to subtract everything that “prevents a child’s inherent genius from gathering itself.”

All of us have inherent genius. My advice is to respect that, and quit thinking IQ testing is anything but a way of sorting people into groups for the convenience of a system that manufactures outputs for its own purposes, often at great human cost.

Here is Walt Whitman on inherent genius:

It is time to explain myself. Let us stand up.

I am an acme of things accomplished,
and I an encloser of things to be.
Rise after rise bow the phantoms behind me.
Afar down I see the huge first Nothing,
the vapor from the nostrils of death.
I know I was even there.
I waited unseen and always.
And slept while God carried me
through the lethargic mist.
And took my time.

Long I was hugged close. Long and long.
Infinite have been the preparations for me.
Faithful and friendly the arms that have helped me.

Cycles ferried my cradle, rowing and rowing
like cheerful boatmen;
For room to me stars kept aside in their own rings.
They sent influences to look after what was to hold me.

Before I was born out of my mother
generations guided me.
My embryo has never been torpid.
Nothing could overlay it.
For it the nebula cohered to an orb.
The long slow strata piled to rest it on.
Vast vegetables gave it substance.
Monstrous saurids transported it in their mouths
and deposited it with care.

All forces have been steadily employed
to complete and delight me.
Now I stand on this spot with my soul.

I know that I have the best of time and space.
And that I was never measured, and never will be measured.

Back to Gatto. Here is the full context of that pull-quote on genius. It’s from Dumbing Us Down: The Hidden Curriculum of Compulsory Schooling:

Over the past thirty years, I’ve used my classes as a laboratory where I could learn a broader range of what human possibility is — the whole catalogue of hopes and fears — and also as a place where I could study what releases and what inhibits human power.

During that time, I’ve come to believe that genius is an exceedingly common human quality, probably natural to most of us. I didn’t want to accept that notion — far from it: my own training in two elite universities taught me that intelligence and talent distributed themselves economically over a bell curve and that human destiny, because of those mathematical, seemingly irrefutable scientific facts, was as rigorously determined as John Calvin contended.

The trouble was that the unlikeliest kids kept demonstrating to me at random moments so many of the hallmarks of human excellence — insight, wisdom, justice, resourcefulness, courage, originality — that I became confused. They didn’t do this often enough to make my teaching easy, but they did it often enough that I began to wonder, reluctantly, whether it was possible that being in school itself was what was dumbing them down. Was it possible I had been hired not to enlarge children’s power, but to diminish it? That seemed crazy on the face of it, but slowly I began to realize that the bells and the confinement, the crazy sequences, the age-segregation, the lack of privacy, the constant surveillance, and all the rest of the national curriculum of schooling were designed exactly as if someone had set out to prevent children from learning how to think and act, to coax them into addiction and dependent behavior.

Bit by bit I began to devise guerrilla exercises to allow as many of the kids I taught as possible the raw material people have always used to educate themselves: privacy, choice, freedom from surveillance, and as broad a range of situations and human associations as my limited power and resources could manage. In simpler terms, I tried to maneuver them into positions where they would have a chance to be their own teachers and to make themselves the major text of their own education.

In theoretical, metaphorical terms, the idea I began to explore was this one: that teaching is nothing like the art of painting, where, by the addition of material to a surface, an image is synthetically produced, but more like the art of sculpture, where, by the subtraction of material, an image already locked in the stone is enabled to emerge. It is a crucial distinction.

In other words, I dropped the idea that I was an expert whose job it was to fill the little heads with my expertise, and began to explore how I could remove those obstacles that prevented the inherent genius of children from gathering itself. I no longer felt comfortable defining my work as bestowing wisdom on a struggling classroom audience. Although I continue to this day in those futile assays because of the nature of institutional teaching, wherever possible I have broken with teaching tradition and sent kids down their separate paths to their own private truths.

The italics are mine.

Knowing that we have industrialized education should help us understand how un-human AI “training,” “learning,” and “knowledge” actually are. (Side note: I love AI and use it every day. I also don’t think it’s going to kill us. But this post isn’t about that.)

Start with the simple fact that institutional teaching and its traditions don’t work for lots of kids. Today’s system is better in some ways than the one Gatto bested. then quit, but it’s still a system. If I were a child in the system we have today, I would surely be classified as an ADHD and ALD case,* given drugs, and put in a special class for the otherwise unteachable.

What worked for me as a student was one kind statement from one teacher: Pastor Ronald Schmidt, who taught English in my junior year. One day he said to me, “You’re a good writer.” It was as if the heavens opened. That was the first compliment I had ever received from any teacher, ever, through twelve years of schooling. I wish he were still alive, so I could thank him.

Fortunately, I can thank my high school roommate, Paul Marshall, who was (and still is) a brilliant writer, musician, preacher—and exceptionally funny. He was voted Class Wit (among other distinctions, which he declined, preferring the Wit one), and as a senior he substitute-taught biology to sophomores when their teacher was out sick. (These days he is the retired Episcopal Bishop of Bethlehem Pennsylvania. Before that, he was a professor at Yale Divinity School. There’s more at both those links.)

I remember a day when a bunch of us were hanging in our dorm room, talking about SAT scores. Mine was the lowest of the bunch. (If you must know, the total was 1001: a 482 in verbal and a 519 in math. Those numbers will remain burned in my brain until I die.) Others, including Paul, had scores that verged on perfection—or so I recall. (Whatever, they were all better than mine.). But Paul defended me from potential accusations of relative stupidity by saying this: “But David has insight.” (I wasn’t Doc yet.) Then he gave examples, which I’ve forgotten. By saying I had insight, Paul kindly and forever removed another obstacle from my path forward in life. From that moment on, insight became my stock in trade. Is it measurable? Thankfully, no.

Okay, back to AI.

As Don Norman told us in his salon here at Indiana University,

First, these machines are not intelligent. Second, remember the A in AI. A means artificial. They don’t work the way we do. And it’s a mistake to think they do. So let’s take a look at what they are. They are pattern-matchers.

I could let Don go on (which you can, at that last link), but there are a zillion explanations of what AI is and does, which you’ll find everywhere on the Web, and in answers from questions you can ask ChatGPT, CoPilot, Anthropic, Perplexity, Claude, and the rest of them. And all of them will be full of metaphorical misdirection. (Which Don avoids, being the linguist that he is.)

We may say an AI is “trained,” that it “learns,” “knows” stuff, and is “smart” because it can beat the most skilled players of chess and go. But none of those metaphors are correct, even though they make sense to us. Still, we can’t help using those metaphors, because we understand everything metaphorically. (To digress into why, go here. Or dig into George Lakoff‘s work, starting here. A summary statement might be, all metaphors are wrong, and that’s why they work. )

To be human is to be different from every other human, by design. We all look and sound different so we can tell each other apart. We also differ from how we were ten minutes ago, because we learn constantly.

So, to be human is to diverge in many ways from norms. Yet, being pattern recognizers and given to organizing our collective selves into institutional systems, we tend to isolate and stigmatize those who are, as we now say, divergent. Constantly recognizing patterns and profiling everything we see, hear, smell, taste, and touch is not just one of the many ways we are all human, but also how we build functioning societies, prejudices included. (As a side note, I am sure the human diaspora was caused both by our species’ natural wanderlust and by othering those who were not like us. We would fight those others, or just migrate away from them until we filled the world. Welcome to now.)

To sum this all up, just remember that when we talk about intelligence, we are talking about a human quality, not a quantity of anything. That machines test out better at pattern recognition than we do does not make them intelligent in a human sense. It just makes them more useful in ways that appear human but are not.

So have all the fun you want with AI. Just remember its first name.

*In my forties and at my wife’s urging (because my ability to listen well and follow directions was sub-optimal), I spent whole days being tested for all kinds of what we now call neurodivergent conditions. The labels I came away with were highly qualified variants of ADHD and APD. Specifics:

I was easily distracted and had trouble listening to and sorting out instructions for anything. (I still have trouble listening to the end of a long joke.) On puzzle-solving questions, I was very good. My smarts with spacial and sequence puzzles were tops, as was my ability to see and draw patterns, even when asked to rotate them 90° or 180°. My memory was good. I had “synchronization issues,” such as an inability to sing and play drums at the same time. This also involved deficiencies around “cognitive overload,” “context switching,” multitasking, coping with interruptions, and “bottlenecks” in response selection. They also said I had become skilled at masking all those problems, to myself and others. I could easily grasp math concepts but made many mistakes with ordinary four-function calculations. I did much better at hearing and reading long words than short ones, and I did better reading wide columns of text than narrow ones. When made to read out loud a simple story comprised of short and widely spaced words in a narrow column, I stumbled through it all and remembered little of the story afterward. They told me that if I had been given this test alone, they would have said I had trouble reading at a first-grade level and I would have been called (as they said in those days) mentally retarded. My performance on many tests suggested dyslexia, but my spelling was perfect and I wasn’t fooled by misplaced or switched letters in words. They also said that I had mostly self-corrected for some of my innate deficiencies, such as dyslexia. (I remember working very hard to become a good speller in the fourth grade, just as a challenge to myself.) They said I did lots of “gestalt substitution,” when reading out loud, for example replacing “feature” with “function,” assuming I had read the latter when in fact I’d read the former. Unlike other ADHD cases, I was also not more impulsive, poorly socialized, or easily addicted to stuff than normal people. Like some ADHD types, I could hyperfocus at times. My ability to self-regulate wasn’t great, it also wasn’t bad. Just a bit below average. (So perhaps today they’d call me ADHD-PI, a label I just found in Wikipedia). The APD (auditory processing disorder) diagnosis came mostly from hearing tests. But, as with ADHD, I only hit some of the checkboxes. (Specifically, about half of the ten symptoms listed here.) My ability to understand what people say in noisy settings was in the bottom 2%. And that was when my hearing was still good.

I also apologize for the length of this post. If I had more time, I would have made it shorter.

Which, being a blog post, I will. Meanwhile, thanks for staying (or jumping) to the end.


The Pragmatic Engineer

Software engineers training software engineers

What is it like to teach software engineers, full time? Reuven M. Lerner has done this for 15 years, and shares his hands-on learnings – including how to teach efficiently

Did you ever consider becoming a teacher of software engineers? I’m assuming many of us have not – simply because it’s an uncommon career path, and teaching rarely feels likely to be lucrative, compared to hands-on building (we previously covered Common engineering career paths as Big Tech and scaleups). But teaching software engineers is an interesting challenge for a few reasons:

Many engineers are good at learning by themselves, so may initially assume there’s little value in being taught by others

But, great teachers make a real difference in getting up to speed, including for software engineers

There’s demand at tech companies for innovative teaching approaches and new technologies for devs

To discover what being a full-time trainer of tech professionals is really like, I turned to software developer turned full-time trainer, Reuven M. Lerner.

Reuven worked as a developer for 15 years, and for the past decade and a half he's been a full-time instructor. He teaches Python, Pandas, and Git for a range of companies, including Apple, IBM, and Cisco. He does both corporate training, as well as online Python courses for individuals.

Today, Reuven takes us behind the scenes of technical training, covering:

Coding vs teaching it. You optimize software when coding, whereas with training you optimize how to best teach the writing of software.

Is training helpful? Many software engineers learn by themselves, and it can be hard to get dedicated time at work for training. But group courses boost communication across tech teams.

What makes effective teaching? Reuven’s thoughts, including on using interactive notebooks over slides, exercises above theory, and lots of pair programming.

Day to day. Teaching learners is just one part of being a trainer: sales, marketing, customer support, and continuous learning are also key.

Business of teaching. To work as a career, teaching must be a solid business. Reuven shares how he runs his operation, from closing new clients and educating decision makers, to collecting payment.

Advice for future trainers. Get comfortable with public speaking, go deep into a technology, don’t shy away from niches, and more.

With that, it’s over to Reuven:

When I got my computer science degree 30 years ago, I knew what my career would look like: I would develop software, eventually move up to manage other people, or maybe start my own business. Indeed, after writing software for Hewlett Packard and Time Warner’s “Pathfinder” website, I opened my own consulting shop, writing Web applications and running Linux-based servers.

Fast forward to today, and my career looks very different. I’m still self employed, but instead of developing software, I’m a full-time trainer in Python and Pandas. I teach at big companies like Apple, Arm, Cisco, and Western Digital, and at startups and financial institutions. I offer more than 30 courses, ranging from “Python for non-programmers,” and “Data analysis with Pandas,” to advanced practice workshops. Between these, I have a growing business of online courses and newsletters for people without access to company training programs.

I feel like I have the best of all worlds: I help people improve their careers, learn new technologies, and interact with smart people all over the world. Plus, I set my own schedule far in advance, have only a handful of meetings a month, spend time with my family, and get paid well — better, in fact, than many developers. I’ve never earned more, and I’ve never enjoyed my work more.

In this post, I introduce the world of tech training. I reveal how it operates, what I’ve found does (and doesn’t) work for training, how I run my business, and how you can explore the world of training.

How I became a full-time trainer

When I started consulting in 1995, I positioned myself as a coder and Linux expert. But some companies asked me not to develop software for them, but to teach their people how to do it. That was my first taste of training and I rather liked it, but saw it as just one part of my consultancy work. Indeed, I rarely spent more than 20 percent of my time on training.

In 2003, I started a PhD program, continuing to consult part-time in order to support my family. While working on my dissertation, a colleague suggested I concentrate on training, and offered to connect me with a company. I said yes – a decision which changed my career.

This training company marketed my Python courses, and filled up my calendar with training sessions. Soon, my schedule was full several months in advance. As convenient as it was to work with them, I also knew that they were keeping half the income.

When I finished my PhD in 2014 (after 11 years!) I left the training company and rebranded myself as a trainer. I’ve now been teaching Python, Pandas, and Git full time for around 15 years and absolutely love it.

My focus on Python turned out to be fortunate because it is used just about everywhere. Even hardware companies that mainly work in C, like Apple, Arm, and Western Digital, use Python on all sorts of internal testing and analysis projects. Financial institutions are moving to Python instead of Excel, and want help in making the switch. Companies doing numerical analysis with Matlab are tiring of the high per-seat licensing cost, and are moving to Python – and need help easing employees into a new environment.

I mostly teach people who are highly schooled and very smart, many of whom have engineering degrees and at least some experience of coding. In theory, their employer could buy them books or video courses, and ask them to learn Python solo. In practice, we all know this doesn’t work; we’re often too busy to use such materials. A timeboxed course, delivered in person and with everyone in the same place is the fastest option with the best results, and it helps establish best practices, instead of just learning the syntax.

1. Coding vs teaching it

How is my life and work different as a trainer, than as a coder? Some of the biggest differences:

As a trainer, my goals are fundamentally different from a full-time software engineer’s. A coder’s goal is to get new or improved functionality out the door. In contrast, my job is to help someone do their job better and faster by writing more idiomatic, maintainable, and efficient code quicker.

I spend much of my time thinking about code. However, I do not do this in the same way I did when working on software projects. I’m not trying to optimize software; I’m trying to optimize learning about writing software. I always seek to simplify and improve my explanations, and find stories, metaphors, and examples that improve my teaching. I’m constantly trying to understand how certain packages and techniques work, so I can explain and illustrate them better to students.

In many ways, I’m like a stand-up comedian. I teach so often, so I see which examples, explanations and exercises work, and which don’t. Just as a comedian changes their jokes from show to show and iterates repeatedly until they find what works, I’m constantly experimenting with what and how I teach, trying to find the optimal way to get information across.

I particularly enjoy using stories in my teaching. Good stories reinforce the ideas being taught, and also enliven classes on potentially dry, abstract topics.

Often, these stories come from personal experience. One recent example: Meta banned me from advertising my courses and newsletters on their platforms, apparently because they believe I was illegally trading in exotic animals (pythons and pandas – the irony!) This event was widely discussed on programming forums like Hacker News.

Python (left) vs a python (right.) Facebook doesn’t allow adverts for Python courses because they assume you’re selling serpents! Read more about this incident.

This was as bizarre and frustrating as it was amusing, but you can be sure I’ll tell this story every time I teach a course on machine learning, and the need to test models before deploying them to production.

When I was doing software projects, it was hard to set my schedule in advance. Typically, someone needs a software project done now, or they don’t want it at all. Talking to someone about a project six months hence is generally a non-starter.

By contrast, there’s almost never a training emergency. As such, training can be scheduled two, four, or even six months in advance. At the time of writing, I already have courses in my schedule for January 2025, and I’m talking to clients about scheduling beyond that.

This ability to plan ahead has improved my personal life and my business. I can now schedule vacations knowing when I will have training gigs. I also have a much better sense of how much I’ll earn in a given month; a much better situation than the feast-or-famine roller coaster of my first years of freelancing.

Shock news: training can pay far better than coding! On the topic of money, here’s a lesser-known detail about training I’ve experienced: It pays better, often far better, than coding because:

If you help 20 developers to become 10 percent more effective, that’s worth a lot of money. So it’s absolutely worthwhile for a company to invest in good, effective training.

The budget doesn’t come from R&D. Rather, it comes from HR, or from a special training budget. Whereas a company might balk at paying thousands of dollars per day for a developer, this is considered to be a normal rate for training services!

Training is usually done through companies with overheads like offices and employees in sales/marketing. A freelancer doesn’t have these costs. Companies will pay roughly the same for training regardless of the training vendor’s size and overheads. I’m a one-person company based in a home office, so I can basically pocket what other companies spend on their costs!

Hardly any meetings. This is another major difference between doing coding and providing training. I’ll typically speak with a new client two or three times before the first class takes place, and maybe once after the first session to wrap things up. But if they ask me to teach again, we just exchange some email, mainly about dates. If I have 4-5 meetings a month, that’s a lot – which means I can spend more time teaching and developing new course materials.

I do miss software projects. I’ve experienced first-hand that there’s nothing like pushing technological boundaries and launching a product, knowing that people around the world are using and enjoying it. And there’s a definite limit to the size and scope of things I can do on my own, rather than in a larger team.

That said, most projects I worked on weren’t pushing boundaries. And while many were exciting, completing them didn’t give me the same sense of purpose and fulfillment I get from teaching. Besides, now I get to write whatever code I want – and there is definitely code to write, whether as part of my courses or running the backend of my online store and newsletters.

My online store’s tech stack combines:

Podia: a SaaS where my video courses live

WooCommerce: an e-commerce SaaS handling payment and subscriptions

Drip: email marketing SaaS, used for two of my newsletters and marketing blasts. I use a fair amount of custom programming (“workflows”) here

Ghost: a CRM and email service used for Bamboo Weekly

GitHub: I create a new repo for each course I teach

Flask: a Python framework I run on a standalone server for one-time coupon codes

Discord: used for discussion among my members.

Zapier: an integrations platform I use to connect these systems. For example, someone subscribing to my Python+Data product is enrolled in all my courses, added to my Better Developers list, and is added to the appropriate GitHub repos.

Custom Python scripts: These help me set up and tear down environments when I give corporate training. Each class gets a new GitHub repo, as well as its own set of Jupyter notebooks. This, along with the “gitautopush” package, lets me work on my own computer and share the course contents with participants in a given course in near-real time.

Do I plan to consolidate these into a smaller number of services? Yes, absolutely. But one person can only do so much in a day. Between teaching, writing three weekly newsletters, responding to learners and researching new topics, I don’t have much time for major technological shifts. But I do have a roadmap; for example, I’ll soon move discussions from Podia to Discord, which seems to foster a greater sense of community.

2. Is training helpful?

I once met someone with a background in engineering and education. I told him what I did and he replied:

“Oh, so you’re an entertainer? Because we both know that you’re not giving any real educational value.” 

This comment hurt. Still, I’m sure many developers who attend my classes also believe they could learn the same material as quickly and as well by themselves, and that my courses are a nice vacation from “real” work. I understand this, but here’s what I’ve learned from years of teaching.

Most people benefit from having someone explain things, including developers who could learn on their own! After I gave a talk at PyCon US this year, a developer told me my presentation answered questions they didn’t even know they wanted to ask. 

I spend a lot of time thinking about the questions people might have beyond simple use cases and syntax, and I integrate them into my teaching. People could get these insights themselves, but it would take longer and not necessarily be contextualized appropriately.

Pressure at work stops many developers learning new things by themselves. One client of mine decided to save money and bought my video courses for their staff. When I came in to do a live Q&A based on the videos, the only person who had really watched them had red eyes, because he had finished at 2:30 a.m. In the end, we returned to in-person lectures.

Learning the nuances of a language is faster with an instructor. Python is a good example; I’m often told this language has such simple syntax that a course isn’t really needed, and it is true the language is pretty simple, with just a few core data structures. So how long can it really take for an engineer to figure it alone? 

This argument is similar to saying chemistry is simple because there are only 118 chemical elements in the universe. Learning the nuances, use cases, limitations, and conventions takes time. This is as true for Python as for chemistry. Going beyond basic syntax is usually faster and more memorable with an instructor.

For example, when I teach Python I dive into the details of the += operator. I explain that even though it does what you expect, one should be careful when using it to concatenate strings. If preserving memory is important, then you should always use a combination of a list and the str.join method to conserve memory. I talk about the different ways to iterate over a dictionary, and why using the dict.keys method is almost always a bad idea. We discuss the difference between the “__str__” and “__repr__” methods, and when to use each (and why I think it’s OK to only define “__repr__”).

Having everyone take a course can improve workplace communication. If people learn solo they’ll understand different things, and choose their own styles/conventions. Giving the same training across a company ensures everyone has the same (or similar) skill levels and understanding, making communication easier within and across teams.

Hands-on exercises are the most efficient way I know how to teach. I’ve fine-tuned coding exercises over years to illuminate certain techniques, syntax, and conventions. I call these exercises “controlled frustration.” The goal is to solve a problem without a manager or deadlines adding to stress levels. 

Learning from other people's mistakes is a great way to learn and in a group setting, this is much easier. As important as it is for students to do exercises, it’s also important to review the exercises together and learn from each other’s buggy code. Also, when I demonstrate how to solve a problem, I’m modeling a process they can apply to their jobs.

Companies rarely give people time to pick up new techniques and technologies. It is true there are plenty of developers who can learn on their own. The trouble is finding dedicated time to focus on learning. I’ve found people often enjoy being in advanced classes – especially exercise-only classes – where they can solve interesting problems they might not have the opportunity to do at work.

As a manager, when does it make sense to consider bringing in a trainer? If your team is adopting a new technology, or if you’re all a little shaky with using it, or you observe devs always going to ChatGPT (or StackOverflow – if you still use it!) to solve problems, then you might want to consider bringing in an instructor. A good instructor with plenty of experience can anticipate which mental models help engineers, and has exercises to take their understanding to the next level.

Also, training empowers members of staff; improving their communication skills and distributing knowledge across organizations. Six months after I taught a Git course at one company, an engineer told me he was now the Git expert in his group, and no longer had to guess what to do when they got in trouble. Not only did he feel great about himself and this new knowledge, but his group benefited from having a local expert.

3. What makes effective teaching?

Read more


Ben Werdmüller

Unlocking the Fediverse: The Social Web Foundation is Shaping the Next Era of the Web

I’m extraordinarily excited about the launch of the Social Web Foundation, which has been created to promote and support the growth of the Fediverse: the interoperable social network powered by the ActivityPub protocol. Users of services on the Fediverse can follow, share, and interact with each other, regardless of which service each one is using. The most famous Fediverse platform is M

I’m extraordinarily excited about the launch of the Social Web Foundation, which has been created to promote and support the growth of the Fediverse: the interoperable social network powered by the ActivityPub protocol.

Users of services on the Fediverse can follow, share, and interact with each other, regardless of which service each one is using. The most famous Fediverse platform is Mastodon, but there are many more participants, including Threads, Flipboard, and Ghost.

From the announcement:

[…] Advocates of this increased platform choice say it will bring more individual control, more innovation, and a healthier social media experience. But there is work to do: journalism, activism, and the public square remain in a state of uncertain dissonance and privacy, safety and agency remain important concerns for anyone participating in a social network.

The Foundation’s founding members are Mallory Knodel, the former CTO of the Center for Democracy and Technology; Evan Prodromou, one of the creators of ActivityPub and its current editor (who just published the canonical book on the topic); and Tom Coates, a product designer and founder who was one of the earliest bloggers and has been involved in many things that have been good on the web. They become the Executive Director, Research Director, and Product Director respectively.

Excitingly, the Foundation’s partners are a who’s who of companies doing great work on the web today. Those include Automattic, Ghost, Flipboard, Fastly, Medium, and Mastodon itself. Meta is also a backer, in an indication of its continued investment in the Fediverse, moving away from the walled garden strategy that it used with Facebook and Instagram for decades.

In a conversation with Richard MacManus over on The New Stack, Evan explained the Foundation’s relationship with existing standards organizations like the W3C:

“W3C as a standards organization mostly does coordinating the work of a number of different groups to make protocols […] So we’ll still be participating in the W3C — we’re going to become a member organization of the W3C.”

Prodromou added that the SWF will take on the role of advocacy and user education, which is typically outside of the W3C’s purview for standards work.

My opinion: this is the future of the social web. Every new service and platform that contains social features — which is most of them — will support the ActivityPub protocol within the next few years. Service owners can use it to easily avoid the “cold start” problem when creating new networks, and to plug their existing platforms into a ready-made network of hundreds of millions of people. Publishers will use it to reach their audiences more easily. And it’s where the global conversation will be held.

When I was building social platforms in the 2000s, this is what we dreamed of. Elgg, the open source social networking platform which launched my career, was intended to be the center of a federated social web. Although we made some crucial steps towards open data protocols and embracing open standards, we didn’t get there. I’m beyond thrilled that the Fediverse and ActivityPub exist, and that there are so many robust platforms that support it. The Social Web Foundation is another great step towards building the social web that we all deserve.

As Casey Newton published just yesterday about the future of his publication, Platformer:

One way I hope it will evolve is to become part of the fediverse: the network of federated sites and apps that are built with interoperability in mind. The fediverse is built on top of protocols, not platforms, which offers us a chance to decentralize power on the internet and built a more stable foundation for media and social apps.

The Social Web Foundation’s existence as an advocacy, research, and development organization is another key step towards making that happen. But to be clear, its role is in support: each one of its partner organizations has already taken concrete steps towards supporting ActivityPub, and the movement is well underway.

Check out the Social Web Foundation and its projects at its website.

Updated: Read more coverage of the launch.


What I learned in year four of Platformer

[Casey Newton at Platformer] This fantastic round-up post focuses on Platformer's decision in January to leave Substack in protest of its content policies that permitted full-throated Nazis to earn money on the platform. With a long-term view, it's been a good strategic move: "We’re much less vulnerable to platform shifts than we were before. I had long worried that Substac

[Casey Newton at Platformer]

This fantastic round-up post focuses on Platformer's decision in January to leave Substack in protest of its content policies that permitted full-throated Nazis to earn money on the platform.

With a long-term view, it's been a good strategic move:

"We’re much less vulnerable to platform shifts than we were before. I had long worried that Substack’s unprofitable business would eventually lead it to make decisions that were not in the best interest of our readers or our business. (Besides not removing literal 1930s Nazi content, I mean.)"

This is the reason publishers should publish from a website they control. Sure, you can syndicate out to meet readers where they're at, but owning your own space makes you much less subject to the whims of someone else's platform.

And even that syndication to social platforms is becoming more controllable. One hope for the future that Casey notes:

"One way I hope [Platformer] will evolve is to become part of the fediverse: the network of federated sites and apps that are built with interoperability in mind. The fediverse is built on top of protocols, not platforms, which offers us a chance to decentralize power on the internet and built a more stable foundation for media and social apps."

Ghost, the open source platform that now powers Platformer, is building fediverse support directly into its platform at a rapid pace, so this almost feels like an inevitability. The benefit will be that Platformer can reach its readers on platforms like Threads, Flipboard, and Mastodon and maintain full control over its relationships with them. That's a game-changer for publishers.

#Media

[Link]

Monday, 23. September 2024

IdM Laboratory

Apple Walletで選択的情報開示

こんにちは、富士榮です。 最近、Google WalletやApple Walletに免許証やパスポートが次々と搭載されてきているわけですが、選択的情報開示のUIがようやく見えてきました。 参考)これまでの記事 Google Walletと選択的情報開示 Google Walletへ搭載できる証明書 カリフォルニア州のモバイル運転免許証のハッカソンが開催されます 選択的情報開示とウォレットと本人確認書類 今回、カリフォルニア州のモバイル運転免許証がApple Walletに搭載されるというニュースを見ると選択的情報開示の動画イメージが掲載されています。 アナウンス https://www.dmv.ca.gov/portal/california-mdl/apple-wallet/ 年齢証明を求めるシナリオです。 Age Over 21という形で2

こんにちは、富士榮です。

最近、Google WalletやApple Walletに免許証やパスポートが次々と搭載されてきているわけですが、選択的情報開示のUIがようやく見えてきました。

参考)これまでの記事

Google Walletと選択的情報開示 Google Walletへ搭載できる証明書 カリフォルニア州のモバイル運転免許証のハッカソンが開催されます 選択的情報開示とウォレットと本人確認書類


今回、カリフォルニア州のモバイル運転免許証がApple Walletに搭載されるというニュースを見ると選択的情報開示の動画イメージが掲載されています。

アナウンス
https://www.dmv.ca.gov/portal/california-mdl/apple-wallet/

年齢証明を求めるシナリオです。


Age Over 21という形で21歳以上であることが提示される、という感じになります。



Verifierからの要求に対して全体として提示する・しないの実装となってしまうのは仕方ないんでしょうが、こんな感じで選択的情報開示のUIが実装されてくることが見えてきました。

Age Verificationは良いシナリオである一方で国によって成人年齢が異なるということもあるので、うまく国によって実装が使い分けられるようになっていると良いですね。この辺りがどうなっているのかもう少し調べてみようかと思いました。(確かできた気がするので)

ちなみに国別の成人年齢の一覧をMicrosoftがリストにしてくれているので参考までに。

https://learn.microsoft.com/ja-jp/azure/active-directory-b2c/manage-user-access


Damien Bod

Implement a Geo-distance search using .NET Aspire, Elasticsearch and ASP.NET Core

This article shows how to implement a geo location search in an ASP.NET Core application using a LeafletJs map. The selected location can be used to find the nearest location with an Elasticsearch Geo-distance query. The Elasticsearch container and the ASP.NET Core UI application are setup for development using .NET Aspire. Code: https://github.com/damienbod/WebGeoElasticsearch Setup For […]

This article shows how to implement a geo location search in an ASP.NET Core application using a LeafletJs map. The selected location can be used to find the nearest location with an Elasticsearch Geo-distance query. The Elasticsearch container and the ASP.NET Core UI application are setup for development using .NET Aspire.

Code: https://github.com/damienbod/WebGeoElasticsearch

Setup

For local development, .NET Aspire is used to setup the two services and the HTTPS connections between the services. The services are configured in the Aspire AppHost project .

The Elasticsearch client is setup as a singleton and requires the connection configuration. This can be changed, if for example an API key is used instead. The connection URL is read from the configuration as well as the secrets.

using Elastic.Clients.Elasticsearch; using Elastic.Transport; namespace WebGeoElasticsearch.ElasticsearchApi; public class ElasticClientProvider { private readonly ElasticsearchClient? _client = null; public ElasticClientProvider(IConfiguration configuration) { if (_client == null) { var settings = new ElasticsearchClientSettings(new Uri(configuration["ElasticsearchUrl"]!)) .Authentication(new BasicAuthentication(configuration["ElasticsearchUserName"]!, configuration["ElasticsearchPassword"]!)); _client = new ElasticsearchClient(settings); } } public ElasticsearchClient GetClient() { if (_client != null) { return _client; } throw new Exception("Elasticsearch client not initialized"); } }

Create Index with mapping

The index cannot be created by adding a document because the mapping is created incorrectly using the default settings. The mapping can be created for the defined index using the Mappings extension from the Elastic.Clients.Elasticsearch Nuget package. This was added to the client project in the Aspire.Elastic.Clients.Elasticsearch package. The mapping is really simple and probably not complete for a production index, some keyword optimizations are required. The detailsCoordinates field is defined as a GeoPointProperty.

var mapping = await _client.Indices.CreateAsync<MapDetail>(IndexName, c => c .Mappings(map => map .Properties( new Properties<MapDetail>() { { "details", new TextProperty() }, { "detailsCoordinates", new GeoPointProperty() }, { "detailsType", new TextProperty() }, { "id", new TextProperty() }, { "information", new TextProperty() }, { "name", new TextProperty() } } ) ) );

The created mapping can be validated using the “IndexName”/_mapping GET request. This returns the definitions as a Json response.

https://localhost:9200/mapdetails/_mapping

Documents can be added to the Elasticsearch index using the IndexAsync method.

response = await _client.IndexAsync(dotNetGroup, IndexName, "1");

Search Query

A Geo-distance query is used to find the distance from the selected location to the different Geo points in the index. This using latitude and longitude coordinates.

public async Task<List<MapDetail>> SearchForClosestAsync( uint maxDistanceInMeter, double centerLatitude, double centerLongitude) { // Bern Lat 46.94792, Long 7.44461 if (maxDistanceInMeter == 0) { maxDistanceInMeter = 1000000; } var searchRequest = new SearchRequest(IndexName) { Query = new GeoDistanceQuery { DistanceType = GeoDistanceType.Plane, Field = "detailsCoordinates", Distance = $"{maxDistanceInMeter}m", Location = GeoLocation.LatitudeLongitude( new LatLonGeoLocation { Lat = centerLatitude, Lon = centerLongitude }) }, Sort = BuildGeoDistanceSort(centerLatitude, centerLongitude) }; searchRequest.ErrorTrace = true; _logger.LogInformation("SearchForClosestAsync: {SearchBody}", searchRequest); var searchResponse = await _client .SearchAsync<MapDetail>(searchRequest); return searchResponse.Documents.ToList(); }

The found results are returned sorted using the Geo-distance sort. This puts the location with the smallest distance first. This is used for the map display.

private static List<SortOptions> BuildGeoDistanceSort( double centerLatitude, double centerLongitude) { var sorts = new List<SortOptions>(); var sort = SortOptions.GeoDistance( new GeoDistanceSort { Field = new Field("detailsCoordinates"), Location = new List<GeoLocation> { GeoLocation.LatitudeLongitude( new LatLonGeoLocation { Lat = centerLatitude, Lon = centerLongitude }) }, Order = SortOrder.Asc, Unit = DistanceUnit.Meters } ); sorts.Add(sort); return sorts; }

Display using Leaflet.js

The ASP.NET Core displays the locations and the results of the search in a Leafletjs map component. The location closest to the center location is displayed differently. You can click around the map and test the different searches. The data used for this display is powered using the Geo-distance query.

Testing

The applications can be started using the .NET Aspire host project. One is run as a container, the other is a project. The docker container requires a Desktop docker installation on the host operating system. When the applications started, the containers need to boot up first. An optimization would remove this boot up.

Notes

Using Elasticsearch, it is very simple to create fairly complex search requests for your web applications. With a bit of experience complex reports, queries can be implemented as well. You can also use Elasticsearch aggregations to group and organize results for data analysis tools, reports. .NET Aspire makes it easy to develop locally and use HTTPS everywhere.

Links

https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-point.html

https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-geo-distance-query.html

https://leafletjs.com/

https://www.elastic.co/guide/en/elasticsearch/reference/current/explicit-mapping.html

Using Elasticsearch with .NET Aspire

Sunday, 22. September 2024

IdM Laboratory

AuthZENのAuthorization APIとは(1)

こんにちは、富士榮です。 昨日、AuthZEN WGがAuthorization API 1.0のImplementer's draftを提案している、という話をしましたが、そもそもこの認可APIはどういうものなのか見ていきたいと思います。 いきなり仕様を読んでもいいのですが、AuthZEN WGのCo-Chairで仕様のEditorでもあるAsertoのCEOのOmri Gazittが良い記事を書いているのでまずはこちらを読んでおきましょう。 https://www.aserto.com/blog/openid-authzen-implementers-draft-why-it-matters Authentication is "solved" The authentication world has mature specifications that
こんにちは、富士榮です。
昨日、AuthZEN WGがAuthorization API 1.0のImplementer's draftを提案している、という話をしましたが、そもそもこの認可APIはどういうものなのか見ていきたいと思います。
いきなり仕様を読んでもいいのですが、AuthZEN WGのCo-Chairで仕様のEditorでもあるAsertoのCEOのOmri Gazittが良い記事を書いているのでまずはこちらを読んでおきましょう。
https://www.aserto.com/blog/openid-authzen-implementers-draft-why-it-matters
Authentication is "solved" The authentication world has mature specifications that are universally adopted, such as OAuth2 and OpenID Connect. This has helped the industry solve "single sign-on for the web".
認証は「解決済み」

認証の世界には、OAuth2 や OpenID Connect など、広く採用されている成熟した仕様があります。これにより、業界は「Web のシングル サインオン」の問題を解決できました。 


OAuth 2.0は置いておいて、認証の世界はOpenID ConnectやSAMLなどのID連携のための仕組みによりシングルサインオンの実現など、複雑性を解決してきました。

Authorization is next

The authorization world has lagged behind. Today, each application has its own way of assigning permissions to users, what we call an "N * M problem".

次は認可です

認可の世界は遅れています。現在、各アプリケーションはユーザーに権限を割り当てる独自の方法を持っており、これを「N * M 問題」と呼びます。


著者がn*m問題として記載している通り、アプリケーションが個々にユーザの権限を管理せざるを得ない、という状況が確かに存在しています。実際、認可を集中管理しにくかった理由としては、アプリケーションごとに保持しているリソースや権限の粒度はバラバラかつ変化が激しく、集中管理するには複雑すぎる、ということがしばしば挙げられます。まさにn*m問題です。

But help is one the way! OpenID AuthZEN aims to become the "OpenID Connect of authorization", and has just entered the review phase for the first Implementer's Draft of the Authorization API version 1.0.

Having served as co-chair of the WG and co-editor of the spec, this means a lot to us at Aserto. Why?

しかし、助けは必ずあります! OpenID AuthZEN は「認可の OpenID Connect」になることを目指しており、Authorization API バージョン 1.0の最初の実装者ドラフトのレビュー段階に入ったところです。

WG の共同議長および仕様の共同編集者を務めた Aserto にとって、これは大きな意味を持ちます。なぜでしょうか?

わくわくしますね!この問題が解けると確かに大きなインパクトです。


Why standardize authorization?

Standards efforts are multi-year affairs, so we don't take them lightly. Standardizing a developer technology needs three things to succeed:

It addresses a significant pain point. Competing technology providers find areas where standardization doesn't erode differentiation, but accelerates adoption. It provides significant benefits to consumers and end-users of the technology, which drives adoption.

認可を標準化する理由は何ですか?

標準化の取り組みは数年にわたる作業であるため、私たちはそれを軽視しません。開発者テクノロジーの標準化を成功させるには、次の 3 つのことが必要です。

それは重大な問題点に対処します。 競合するテクノロジー プロバイダーは、標準化によって差別化が損なわれることなく、導入が加速される領域を見つけます。 これは、テクノロジーの利用者とエンドユーザーに大きなメリットをもたらし、採用を促進します。

これはAuthorization APIに限らず、すべての標準仕様について言えることだと思いますが、実装するプロバイダの競争領域と協調領域の特定から始める必要があるわけです。当然のことながら競争領域では各ベンダが差別化をしていくポイントなので、その領域に関して情報開示はしないわけです。ただ協調した方が全体にとってメリットがある部分は必ず存在するので、そこを特定して標準化をしていくわけです。また、これらのテクノロジーはコンシューマ(実装する人たち)やエンドユーザにとってメリットがなければ設計しても使われることはありません。


ODBC

Early in my career, around 1993, I witnessed this first hand with Open Database Connectivity, or ODBC. Consumers wanted data-centric applications (Excel, Access, Visual Basic, Powerbuilder, and countless more) to be able to talk to a bunch of data sources (Oracle, Sybase, SQL Server, DB2, Informix, and many others).

This is what we call an "N * M" problem: N applications need to build connectors to M data sources. Wasteful and expensive.

ODBC addressed this challenge by defining a universal data access API, which database vendors could implement, and applications could consume, transforming it into an "N + M" problem. All of the sudden, any data application could immediately talk to a whole bunch of data sources simply by being a consumer of ODBC.

My startup, NEON Systems, bet on this standard and rode its success by integrating data-centric applications with a wide variety of enterprise data sources. NEON was so successful it went public in 1999.

ODBC

キャリアの初期、1993 年頃に、私は Open Database Connectivity (ODBC) でこれを直接目にしました。消費者は、データ中心のアプリケーション (Excel、Access、Visual Basic、Powerbuilder など数え切れないほど) が、多数のデータ ソース (Oracle、Sybase、SQL Server、DB2、Informix など) と通信できることを望んでいました。

これは「N * M」問題 と呼ばれるもので、 N 個のアプリケーションがM 個のデータ ソースへのコネクタを構築する必要があります。これは無駄が多く、コストもかかります。

ODBC は、データベース ベンダーが実装し、アプリケーションが使用できるユニバーサル データ アクセス API を定義することでこの課題に対処し、これを「N + M」問題に変換しました。突然、ODBC のコンシューマーになるだけで、あらゆるデータ アプリケーションが大量のデータ ソースとすぐに通信できるようになりました。

私のスタートアップ企業である NEON Systems は、この標準に賭け、データ中心のアプリケーションをさまざまなエンタープライズ データ ソースと統合することで成功を収めました。NEON は大成功を収め、1999 年に株式を公開しました。 

ODBC!今となっては懐かしいですね。。私もめちゃくちゃ使ってました。まさにデータベース管理システム(OracleとSQL Serverなど)が乱立している時代(今もか)にユニバーサルなAPIは必須アイテムでした。まぁ、もちろん固有の機能を使うには各社のクライアントを使う必要があったりしたわけですが。

Open ID Connect

Two decades after ODBC, OpenID Connect (OIDC) became a standard that solved a similar problem. N SaaS applications needed to integrate with M corporate identity providers. By adopting the OIDC protocol, each SaaS app allows its users to sign-in with their corporate identity provider (Okta, Azure AD, Google Workspace, Ping ID, etc).

Admins no longer have to worry about corporate users creating their own logins on each SaaS application - they can use a single corporate login across all these applications. Onboarding and offboarding become a breeze!

At Microsoft, our vision for this started with SAML, WS-Security, and WS-Federation in the early 2000's, but it wasn't until 2013 that OIDC reached its tipping point. The journey took a while, but the result has been nothing short of transformational.

OpenID Connect

ODBC の 20 年後、OpenID Connect (OIDC) が同様の問題を解決する標準になりました。N個のSaaS アプリケーションをM個の企業 ID プロバイダーと統合する必要がありました。OIDC プロトコルを採用することで、各 SaaS アプリのユーザーは企業 ID プロバイダー (Okta、Azure AD、Google Workspace、Ping ID など) を使用してサインインできるようになります。

管理者は、企業ユーザーが各 SaaS アプリケーションで独自のログインを作成することを心配する必要がなくなりました。管理者は、これらすべてのアプリケーションで単一の企業ログインを使用できます。オンボーディングとオフボーディングが簡単になります。

Microsoft では、このビジョンは 2000 年代初頭に SAML、WS-Security、WS-Federation から始まりましたが、OIDC が転換点に達したのは 2013 年になってからでした。この道のりには時間がかかりましたが、その結果はまさに変革をもたらすものでした。

まぁ、この辺りまでは歴史の話ですな。
Open ID AuthZEN Fast forward a decade to 2024: the same "N * M" problem exists in the authorization space. Every corporate application has its own way of assigning permissions to users. And the solution is similar to how applications have externalized authentication to an OIDC-compliant IDP: externalizing authorization to an AuthZEN-compliant Policy Decision Point (PDP). Applications that do this not only save a bunch of time and effort rolling out their own bespoke authorization. They also allow IT administrators to enforce common policies across applications, ensure compliance across applications, and answer questions like "which users have access to which resources" across applications. While authorization vendors want to differentiate on expressing policies, ensuring compliance, facilitating forensics, and managing authorization at scale, none of us really care about what the authorization API actually looks like. We all have similar APIs that are arbitrarily different, and they are not a real source of differentiation. Standardizing the way a policy enforcement point (PEP) such as an application or API gateway calls an authorization platform (PDP) greatly reduces the friction of integrating the PEP with a wide variety of authorization solutions. It helps everyone: Applications and API gateways can integrate with a bunch of externalized authorization systems using a single API, instead of creating bespoke integrations with each. Authorization platforms become relevant to a broader set of applications and other policy enforcement points. IT Administrators have a single "Authorization control plane" to manage policies and entitlements, and answer questions such as "what resources does this user have access to". 
OpenID AuthZEN

10 年後の 2024 年、同じ「N * M」問題が認可の分野で存在します。すべての企業アプリケーションには、ユーザーに権限を割り当てる独自の方法があります。そして、その解決策は、アプリケーションが認証を OIDC 準拠の IDP に外部化する方法と似ています。つまり、認可を AuthZEN 準拠のポリシー決定ポイント (PDP) に外部化します。

これを実行するアプリケーションは、独自のカスタム認証を展開する時間と労力を大幅に節約するだけではありません。IT 管理者は、アプリケーション間で共通のポリシーを適用し、アプリケーション間でコンプライアンスを確保し、アプリケーション間で「どのユーザーがどのリソースにアクセスできるか」などの質問に答えることもできます。

認可ベンダーは、ポリシーの表現、コンプライアンスの確保、フォレンジックの促進、大規模な認可の管理で差別化を図りたいと考えていますが、認可 API が実際にどのようなものであるかについては、誰も気にしていません。ベンダーは皆、任意に異なる類似の API を持っており、それらは差別化の本当の源ではありません。

アプリケーションや API ゲートウェイなどのポリシー適用ポイント (PEP) が認可プラットフォーム (PDP) を呼び出す方法を標準化すると、PEP をさまざまな認可ソリューションと統合する際の摩擦が大幅に軽減されます。これにより、すべての人が次のメリットを享受できます。

アプリケーションと API ゲートウェイは、それぞれにカスタマイズされた統合を作成する代わりに、単一の API を使用して多数の外部認証システムと統合できます。 認可プラットフォームは、より広範なアプリケーションやその他のポリシー適用ポイントに関連するようになります。 IT 管理者は、ポリシーと権限を管理し、「このユーザーはどのリソースにアクセスできるか」などの質問に答えるための単一の「承認コントロール プレーン」を持ちます。 



まぁ、この手の仕組みとして非常にオーソドックスなアプローチですね。PDPを一箇所に集めて、PEPはそのAPIを通じて呼び出すという仕掛けですね。

What does this milestone mean?

So standardizing authorization is important. Why celebrate this milestone?

We started the OpenID AuthZEN WG in late October 2023. In one short year, we've been able to define a number of iterations of our first spec, the PEP-PDP API. We now have 13 interoperable implementations of a preview version of this spec.

We've learned quite a bit from the interop events we conducted at Identiverse 2024 and EIC 2024, and now have a candidate Implementer's Draft.

This milestone means that vendors can safely incorporate AuthZEN into their products, without worrying that the spec will "move under them". This also means that Policy Enforcement Points, such as API Gateways, SaaS applications, and identity providers can start calling out to AuthZEN PDPs to make authorization decisions.

このマイルストーンは何を意味するのでしょうか?

したがって、認可の標準化は重要です。なぜこのマイルストーンを祝うのでしょうか?

私たちは、2023 年 10 月下旬に OpenID AuthZEN WG を開始しました。わずか 1 年で、最初の仕様であるPEP-PDP APIの反復を何度も定義することができました。現在、この仕様のプレビュー バージョンの相互運用可能な実装が 13 個あります。

私たちは、Identiverse 2024 と EIC 2024 で実施した相互運用イベントから多くのことを学び、現在は実装者ドラフトの候補が揃っています。

このマイルストーンは、ベンダーが仕様が「自分たちの下に移る」ことを心配することなく、AuthZEN を自社の製品に安全に組み込むことができることを意味します。これはまた、API ゲートウェイ、SaaS アプリケーション、ID プロバイダーなどのポリシー適用ポイントが、AuthZEN PDP を呼び出して承認の決定を下せるようになることも意味します。

そう、このAuthZENワーキンググループはまだ1年経ってないんですよね。1年未満でImplementer's draftが出てくるのは異常なスピード感ですね。また、書いてある通り13の相互運用性が確認できる実装があるのもすごいことです。

ちなみに相互運用性検証の結果はこちらで見れます。

https://authzen-interop.net/


What's next?

The review and voting period extends through November 9, 2024. At that point, we will have a formal Implementer's Draft.

Aserto is committed to incorporating AuthZEN in a first-class way into our authorization engine, Topaz, as well as our commercial products.

In addition, we're looking forward to working with the community to create developer SDKs for a wide variety of languages, and working with API gateway vendors to make it trivial to call an AuthZEN-compliant PDP from a request filter.

The next AuthZEN interop event is at Authenticate 2024 on October 15, 2024. We plan on testing the next iteration of the spec, which defines how to send multiple evaluations in a single request, facilitating scenarios such as turning on and off features in a web or native UI based on a user's permissions.

次は何ですか?

レビューと投票期間は 2024 年 11 月 9 日までです。その時点で、正式なImplementer's draftが作成されます。

Aserto は、AuthZEN を当社の認証エンジンTopazおよび商用製品に最高レベルの方法で組み込むことに尽力しています。

さらに、私たちはコミュニティと協力してさまざまな言語向けの開発者 SDK を作成し、API ゲートウェイ ベンダーと協力してリクエスト フィルターから AuthZEN 準拠の PDP を簡単に呼び出せるようにすることを楽しみにしています。

次の AuthZEN 相互運用イベントは、2024 年 10 月 15 日の Authenticate 2024 です。私たちは、単一のリクエストで複数の評価を送信する方法を定義し、ユーザーの権限に基づいて Web またはネイティブ UI で機能をオン/オフにするなどのシナリオを容易にする、仕様の次のイテレーションをテストする予定です。

先にも書きましたが相互運用性テストが継続的に行われているのはとても良いことですね。DCP WGのVerifiable Credentials関連の仕様や、SSFでも相互運用性テストが積極的に実施されているのと同様に仕様を作りつつ実装で試験をしていくという流れは標準化にとって非常に重要な意味を持ちます。

Future work

We have lofty goals for AuthZEN. We will define a search API which standardizes answering questions like "which resources does this user have access to", and "which users can access this resource".

We also plan on defining ways in which upstream data sources can send data updates to policy decision points and policy information points, so that PDPs can have the latest user, group, and relationship information when evaluating access decisions.

今後の仕事

AuthZEN には高い目標があります。 「このユーザーはどのリソースにアクセスできるか」や「どのユーザーがこのリソースにアクセスできるのか」といった質問への回答を標準化する検索 API を定義します。

また、上流のデータ ソースがポリシー決定ポイントとポリシー情報ポイントにデータ更新を送信する方法も定義する予定です。これにより、PDP はアクセス決定を評価する際に最新のユーザー、グループ、関係情報を取得できるようになります。 

今後の動きに期待です!楽しみですね。


今回はイントロということでしたので、次から実際の仕様を見ていこうと思います。



Kent Bull

KERI Series: Understanding Self-Addressing Identifiers (SAID)

What is a self addressing identifier, a SAID? What does this mean and how is a SAID created and verified? This post answers these questions. We show a generalized process for calculating SAIDs and delve into the encoding format for CESR-compliant self addressing identifiers. Examples with three popular algorithms, SHA2-256, […]

What is a self addressing identifier, a SAID? What does this mean and how is a SAID created and verified? This post answers these questions. We show a generalized process for calculating SAIDs and delve into the encoding format for CESR-compliant self addressing identifiers. Examples with three popular algorithms, SHA2-256, SHA3-256, and Blake3-256, show specifics of applying the general process. This general process can be used for calculating SAIDs with other cryptographic algorithms.

For those who want to skim there are pictures below including bit diagrams that illustrate exactly what is happening.

What is a SAID?

Fundamentally, a SAID is a cryptographic digest of a given set of data and is embedded within the data it is a digest of. A CESR-style SAID pads the digest to 33 bytes and adds a type code into the padded digest to replace resulting Base64 pad characters. It looks like this:

HPJbVi6fZvGNCASDiwABn2wpQ0lI-2cR0yaoRErkD-j6

This is a SHA3-256 digest encoded in the CESR format.

What is the CESR format? It is the Base64 URL Safe encoding of the raw digest along with some front-padding of zero bits and a type code, as shown in detail below. From the above SAID, the ‘H’ character is the type code. The rest of the string is composed of Base64 URL Safe characters.

Why Base64? More Space

Why was Base64 encoding used rather than something like hex encoding? Because Base64 encoding allows maximally compact text encoding of data using a well-known encoding protocol of alphanumeric characters (0-9, a-z, A-Z, -_). As compared to hexadecimal (“hex”) encoding Base64 encodes 6 bits of data per Base64 character whereas hex encoding encodes 4 bits of data per Base64 character, so Base64 can store 50% more data in the same space compared to hex. This helps reduce bandwidth and power costs, optimizing performance overall.

Note on Hash or Digest Terminology

A note on terminology, sometimes digests are called hashes or hash values. The technical definition of the term hash refers to a hash function. Hash functions transform data into a fixed-size string. This fixed-size string is the digest, the output of a hash function.

Back to SAIDs, the fact that a SAID can be embedded in the data it is a digest of is why it is called “self addressing.” The digest is essentially a unique identifier of the data it is embedded in.

A SAID (Self-Addressing Identifier) is a special type of content-addressable identifier based on an encoded cryptographic digest that is self-referential.

Composable Event Streaming Representation ToIP Specification – Section 12.6 – Dr. Samuel M. Smith

What is a content addressable identifier? A content addressable identifier is an identifier derived from the content being stored which makes a useful lookup key in content addressable storage, such as IPFS or a key-value store database like LevelDB, LMDB, Redis, DynamoDB, Couchbase, Memcached, or Cassandra.

Embedding a digest changes the source data and hash, right?

How can the SAID digest could be accurate given that placing the SAID in the data it identifies changes the data, thus producing a different hash? The way SAIDs accomplish this is with a two step generation and embedding process.

Two step SAID generation and embedding process During SAID calculation the destination field of the SAID is filled with pound sign filler characters (“#”) up to the same length of the SAID. The digest is then calculated, encoded, and placed in the destination field.

The reverse occurs for verification of a SAID.

The SAID is replaced with filler ‘#’ characters up to the same length of the SAID. The digest is calculated, encoded and compared with the SAID

How does the generation step work? This question kicks off a larger discussion about CESR-style encoding of cryptographic digests using pre-padding and type codes. First, let’s start with some code examples that cut right to the chase. You can come back to these examples after reading the post if they don’t make sense to you at first.

Code examples with multiple algorithms

Let’s start with some code examples showing how to create a correct SAID including the appropriate pre-padding characters. For additional understanding come back and review these examples after you have read the sections on 24 bit boundaries, pad characters, and pad bytes.

For now, say you want to use other cryptographic digest algorithms to create your SAIDs. How would you go about doing that?

It is as easy as changing your hashing function and then using the corresponding type code from the CESR Master Code Table corresponding to your desired digest algorithm.

The following code examples in Python illustrate the process for each of the following algorithms, Blake2b-256, Blake3-256, and SHA2-256. The SHA3-256 algorithm is shown above in the example in the main body of the article.

Filler ‘#’ characters in digest ‘d’ field

The following examples all use the raw value that includes the filler ‘#’ pound sign characters for the digest field ‘d’ which will both be explained later. The “d” digest field is supposed to contain the same number of filler characters as the eventual SAID that will replace the filler characters.

Creating a Blake2b-256 SAID – Step By Step

For a Blake2b-256 SAID with Python you just change the hash function and specify a digest size.

import hashlib from base64 import urlsafe_b64encode raw_value = b'{"d":"############################################","first":"john","last":"doe"}' digest = hashlib.blake2b(raw_value, digest_size=32).digest() # <-- See the different algorithm blake2b padded_digest = b'\x00' + digest encoded = urlsafe_b64encode(padded_digest) b64_str_list = list(encoded.decode()) # convert bytes to string of chars for easy replacement of 'A' b64_str_list[0] = 'F' # replace first 'A' character with 'F' type code b64_str = ''.join(b64_str_list) # convert string of chars to string with .join() assert b64_str == 'FFfZ4GYhyBRBEP3oTgim3AAfJS0nPcqEGNOGAiAZgW4Q' assert len(b64_str) == 44 # length should still be 44 characters, 264 base64 bits, a multiple of 24 bits Creating a Blake3-256 SAID – Step By Step

Blake3-256 is even easier, though it requires the blake library

import blake3 from base64 import urlsafe_b64encode raw_value = b'{"d":"############################################","first":"john","last":"doe"}' digest = blake3.blake3(raw_value).digest() # <-- See the different algorithm blake3.blake3 padded_digest = b'\x00' + digest encoded = urlsafe_b64encode(padded_digest) b64_str_list = list(encoded.decode()) # convert bytes to string of chars for easy replacement of 'A' b64_str_list[0] = 'E' # replace first 'A' character with 'E' type code b64_str = ''.join(b64_str_list) # convert string of chars to string with .join() assert b64_str == 'EKITsBR9udlRGaSGKq87k8bgDozGWElqEOFiXFjHJi8Y' assert len(b64_str) = 44 # length should still be 44 characters, 264 base64 bits, a multiple of 24 bits Creating a SHA2-256 SAID – Step By Step

And finally SHA2-256 is also easy, just changing the hash function used:

import hashlib from base64 import urlsafe_b64encode raw_value = b'{"d":"############################################","first":"john","last":"doe"}' digest = hashlib.sha256(raw_value).digest() # <-- See the different algorithm sha3_256 padded_digest = b'\x00' + digest encoded = urlsafe_b64encode(padded_digest) b64_str_list = list(encoded.decode()) # convert bytes to string of chars for easy replacement of 'A' b64_str_list[0] = 'I' # replace first 'A' character with 'I' type code b64_str = ''.join(b64_str_list) # convert string of chars to string with .join() assert b64_str == 'IDuyELkLPw5raKP32c7XPA7JCp0OOg8kvfXUewhZG3fd' assert len(b64_str) == 44 # length should still be 44 characters, 264 base64 bits, a multiple of 24 bits

Now on to a visual introduction.

Visual Introduction to SAID

Here is a SAID using the SHA3-256 algorithm on the sample JSON object used in this post.

HPJbVi6fZvGNCASDiwABn2wpQ0lI-2cR0yaoRErkD-j6

Adding this SAID to a document looks like taking the following JSON,

computing the SAID, encoding it, and placing it in the SAID field, or digest field, which is the “d” field in this example:

The ‘H’ character is highlighted here to draw attention to the fact that is a special character. This special character is the type code in the CESR Master Code Table. This indicates the type of cryptographic algorithm being used, SHA3-256 in this case.

I see a problem…

Those new to calculating and encoding SAIDs often encounter a problem here. If you take the raw Base64 encoded value of the JSON value {"d":"","first":"john","last":"doe"} then you end up with the string value eyJkIjoiIiwiZmlyc3QiOiJqb2huIiwibGFzdCI6ImRvZSJ9, which is nowhere close to the value shown in the picture of HPJbVi6fZvGNCASDiwABn2wpQ0lI-2cR0yaoRErkD-j6. Why are they different?

Doing a plain Base64 encoding of the JSON bytes misses an important step, the encoding step referred to above. The rest of the post dives deep into this encoding as it shows and explains how to construct a correct, CESR-encoded, SAID digest and explains the rationale behind why CESR encoding is designed the way it is.

Five parts of a SAID (SAID)?

As mentioned earlier, a SAID is a cryptographic digest. Specifically, it is a kind of digest usable as a content addressable identifier, and it is embedded in the content it identifies. SAIDs were invented by Dr. Samuel Smith as a part of his work on key event receipt infrastructure (KERI), authentic chained data containers (ACDC), and composable event streaming representation (CESR).

To understand how SAIDs work you must learn the interplay of five different concepts including:

Bit boundaries – aligning on 24 bit boundaries using pre-padded bytes on the left/front of raw bytes Hash values – hashing input bytes with hashing functions to produce output hash values (digests) Encoding with the URL-safe variant of Base64 encoding, Using type codes to indicate type of hashing function and size of digest, The two-pass SAID calculation and embedding process.

This article specifically covers SAIDs that are encoded in the CESR format. These CESR-style SAIDs

use pre-padding of pad bytes for bit padding to align on 24 bit boundaries, are compatible with a variety of common hashing functions, are encoded in the URL-safe variant of Base64 encoding (a.k.a. Base64URL), substitute type codes from the CESR Master code table (section 12.4.2) for ‘A’ front zero characters and are calculated from and embedded in the data they identify. How does it work? How are SAIDs calculated?

The easiest way to understand a self addressing identifier is to create one. Starting with the JSON from above we walk through each of the five major concepts required to create a CESR encoded SAID.

7 Steps to Calculate and Embed a SAID

Briefly, the process is listed here. A detailed explanation and example follows this set of steps.

Get an object to calculate a SAID for with a digest field that will hold the SAID. In this case we use the JSON object below and the “d” field will hold the SAID. The field does not have to be empty though it can be. Prior to digest calculation it will be cleared and filled with the correct number of filler characters. Calculate the quantity of Base64 characters the final encoded bytes will take up and fill the digest field with that many ‘#’ characters. This value may be looked up from a parse table like the CESR Master Code Table based on the type of hashing function used. Replace the contents of the digest field, “d” in our case, with pound sign (“#”) characters up to the number of filler characters calculated in step 2. The calculated size and pad values used for this step are reused in step 4. Calculate a digest of the object with the filler ‘#’ characters added using the hash function selected. This will result in a quantity of digest bytes, specifically 32 bytes for the SHA3-256 algorithm. Calculate the quantity of pad bytes that when added to the digest bytes will give you a value length that is multiple of 24 bits. This math is shown below. For us this is 1 pad character giving us 33 bytes. This value may be looked up from a parse table like the CESR Master Code Table. Perform pre-padding by prepending the pad byte to the digest bytes to get padded raw bytes. Encode the padded raw bytes with the Base64 URL Safe alphabet. Pre-padding causes some characters at the start of the digest to be encoded as “A” characters which represent zero in the Base64 URL Safe alphabet. Substitute the type code for the correct number of “A” zero character(s) in the Base64 encoded string according to the CESR encoding rules from the CESR Master Code Table. Use the type code corresponding to the cryptographic hash algorithm used. In our case this is “H” because we are using the SHA3-256 algorithm. This is your SAID! Place the Base64 encoded, type code substituted string (your SAID!) into the digest field in your object. This makes your object self-addressing. 3 Steps to Verify a SAID Start with a SAID from an object you already have. Calculate the SAID for the object using the process shown above Compare the SAID you pulled out of the object with the SAID you calculated. If they match then the SAID verifies. Otherwise the SAID does not verify.

An illustration will make clear why and how this process is done. Let’s walk through an example with a small JSON object. The concept applies to any size JSON object and objects of any serialization format such as CBOR, MessagePack, arbitrary text, or otherwise.

Example walkthrough with JSON and SHA3-256 Create Step 1: Get an object with some data and a digest field

Starting with the JSON below we have a “d” field, or digest field, in which the SAID will eventually be placed. In our case it is empty though it could start with the SAID in the “d” field and the process would still work.

JSON being SAIDified: { "d": "", "first": "john", "last": "doe" } Create Step 2: Calculate the quantity of filler ‘#’ characters

The expected final size of the SAID must be known in advance in order to create a JSON object with a stable size. Calculating this quantity requires that you understand a major concept in CESR:

How to calculate pad sizes (quantity of pad bytes) and full sizes of values.

Understanding this calculation will get you most of the way towards understanding another major CESR concept called “fully qualified Base64 representation” of a cryptographic primitive. A digest is a kind of cryptographic primitive.

Knowing the size in advance, and having it be stable, is critical for CESR’s type, length, value (TLV) encoding scheme. This stable size is achieved by filling the digest field with the same number of pound sign ‘#’ characters as the size of the SAID, which looks like this:

Correct number of filler characters added to digest field { "d": "############################################", "first": "john", "last": "doe" }

This enables the JSON to have the same size during and after the SAID calculation process, giving a stable size. In order to know the number of filler characters then you must calculate how many Base64 characters will be in the final SAID. Calculating how many Base64 characters are needed involves summing raw bytes and pad bytes needed to align on what is called a 24 bit boundary.

Final output has same size since Base64 characters count equals filler length

Aligning on this 24 bit boundary allows the final result with the SAID to have the same length as the version with the filler characters, 44 characters in our case:

{ "d": "HPJbVi6fZvGNCASDiwABn2wpQ0lI-2cR0yaoRErkD-j6", "first": "john", "last": "doe" }

Remember when the “encoding” step was mentioned from above? That’s where this filler character and size calculation knowledge comes in. In this encoding step you learn about the CESR-style encoding using pre-padding, pre-conversion. Knowing how many filler characters to use requires understanding the concept of aligning on a 24 bit boundary. Aligning on a 24 bit boundary is where the pre-padding of CESR comes in. This calculation of pad bytes required to align on a 24 bit boundary is the primary difference between raw, or “naive”, Base64 encoding and CESR encoding.

First let’s delve into what a 24 bit boundary is, why it matters to Base64 encoded values, and then look at some diagrams that make Base64 post-padding and CESR pre-padding clear. In doing this we jump ahead a bit and show byte diagrams of the actual encoded digest since that will help introduce later steps.

24 bit boundary – from Base64

The 24 bit boundary comes from the Base64 encoding format standard, RFC4648, specifically section 4. The reason a 24 bit boundary matters is because you can only use whole Base64 characters; there is no such thing as a fractional Base64 character. A Base64 character represents 6 bits of your raw bytes. A single byte is 8 bits. How do you reconcile the 6 bit Base64 character encoding to the 8 bits of your raw bytes? This is where a little math comes in, specifically the least common multiple.

Section 4 of the Base64 RFC 4648 describes the 24-bit groups that are the origin of the 24-bit boundary:

The encoding process represents 24-bit groups of input bits as output strings of 4 encoded characters. Proceeding from left to right, a 24-bit input group is formed by concatenating 3 8-bit input groups. These 24 bits are then treated as 4 concatenated 6-bit groups, each
of which is translated into a single character in the base 64 alphabet.

RFC 4648 The Base16, Base32, and Base64 Data Encodings – Section 4

Using these 24-bit groups ensures the value coming out of a Base64 decoder is the same value you put in. Separating raw bits into these 24 bit groups is where the phrase “aligning on 24 bit boundaries” comes from.

Splitting the 8-bit groups up into 6-bit groups requires a little math because 8 does not split evenly into 6. The math equation to do this is the least common multiple (LCM). LCM is used to determine the lowest number that both 8 and 6 divide into evenly, which is 24, thus the need for 24-bit groups, or 24-bit boundaries. Any value that is encoded into Base64 characters must be padded to reach a multiple of 24 bits. These 24-bit groupings allows you to cleanly convert all of your 8-bit bytes in to 6-bit Base64 characters and back to bytes without missing any bits.

Yet, if we have a stream that does not align on a 24 bit boundary then how do we create that alignment?

Pad characters on the END of a string are the answer to this in Base64.

By adding the correct number of pad characters on the end of a Base64 stream then you always end up with a value aligned on a 24 bit boundary. The ‘=’ equals sign pad characters in a plain Base64 encoding indicate the quantity of pad bits that were used in the final Base64 character adjacent to the ‘=’ pad characters.

Pad bytes at the START of the raw bytes are the answer to this in CESR.

By prepending the correct number of pad bytes on the start of a set of raw digest bytes then you always end up with a value aligned on a 24 bit boundary. Since the pad bytes are all zero bits then the resulting encoded value will start with one or more ‘A’ characters since they correspond to all zero bits in the Base64 alphabet.

Pad characters Calculation

In a plain Base64 encoding when encoding an array of bytes into Base64 that does not align on a 24 bit boundary the correct number of Base64 pad characters ‘=’ must be included. Why? Because in order to avoid data corruption in the decoded value you must know the precise original value, which means knowing how many pad characters to strip off and how many pad bits to strip out of the Base64 character adjacent to the padding. The decoder of your Base64 character needs to know how many bits of the last character used were just padding and how many were a part of your raw value.

You must signal the end of your raw bytes somehow. If, instead, you ignore, drop, or omit pad characters then you will confuse a Base64 decoder into thinking that pad bits were a part of your raw bytes, which you want to avoid because that will give you a different output value than what your input value was, meaning you would experience data corruption.

Pad characters must be included with a plain or “naïve” Base64 encoded value so that a Base64 decoder can strip the correct number of pad bits from the output giving you your original input bytes when decoding from Base64 characters to raw bytes. This is the purpose that Base64 pad characters serve. The pad characters indicate how many pad byes were used to encode a value in Base64.

CESR uses pad bytes and characters in a similar way, yet on the front, and with pre-conversion padding, so the rules for identifying and stripping pad bits are slightly different.

Yet, let’s stick with Base64 padding for now and come back to CESR padding later. If you are starting to get confused or lost then skip ahead to the diagrams below and come back to this explanation.

ASIDE – Calculating the quantity of Base64 pad characters based on input byte quantity

For a SHA3-256 digest this count is 44 characters. See the math below for an explanation. This number may also be found in the CESR Master Code Table for the type of algorithm used. Since we measure every raw value in terms of bytes (8 bits) then there are three possible scenarios, detailed here in the Base64 RFC, for the number of pad bytes required and thus pad characters.

A value ending with a single byte (8 bits) beyond a 24 bit boundary requires two bytes (16 bits) to meet a 24 bit boundary. This will have two ‘=’ pad characters. This means that your 8 raw bits + the 16 padding bits (two bytes) will equal 24 bits, aligning your raw value on a 24 bit boundary. A value ending with two bytes (16 bits) beyond a 24 bit boundary requires one byte (8 bits) to align on a 24 bit boundary. This will have one ‘=’ pad character. Take the 16 bits + one pad byte (8 bits) to get to 24 bits to align on the 24 bit boundary. A value ending with three bytes is already aligned on a 24 bit boundary (3 * 8 = 24)

You can use the modulus operator ‘%’ to determine the number of ending bits you have. For 256 bits (32 bytes * 8 bits per byte) you end up with 16 bits, or two bytes, rule number two above. So we need the equivalent of one pad byte.

How Base64 handles pad bits

The way that Base64 handles the need for pad bytes is to split the last byte into two characters, add zero bits to the last Base64 character, and then add the correct number of pad ‘=’ equals sign characters to the final output to end up with groups of 4 Base64 characters, which aligns on a 24 bit boundary because 4 * 6 bits per Base64 character = 24 bits.

What this means for a SAID – Calculating Pre-pad Bytes for CESR

In CESR padding is handled a bit differently because it repurposes the pad characters for type codes in its TLV encoding scheme. This means that what would have been zero bits representing ‘A’ characters in the Base64 encoded CESR value gets replaced with the type code, also called derivation code, in the final CESR value. To accomplish this CESR does pre-padding prior to conversion to Base64 characters. What this means for SAIDs is that all digest bytes must be padded at the front of the digest bytes to reach a multiple of 24 bits. Compare this to Base64 padding which occurs at the end of the digest bytes. Both scenarios are pictured below, Base64 padding and CESR padding.

Since the SHA3-256 digest we start with is 32 bytes, or 256 bits (not a multiple of 24), then all we need to add is one byte to get to 264 bits, which is a multiple of 24, or 33 bytes.

Now once you know the quantity of bytes that align on a 24 bit boundary you can do a simple calculation to get to the number of pad characters for your digest. Since 6 bits of every byte are put into a Base64 character (6 bit groups) then you can divide your total number of bits (264) by 6 to get the number of Base64 characters of your final digest.

264 (bits) / 6 (bits per Base64 char) = 44 (Base64 chars)

This means the total length of the resulting SAID will be 44 Base64 characters. So, you need 44 filler ‘#’ pound sign characters in your digest field of your JSON object prior to calculating the SAID.

Fixed width output – why is it needed?

Consistent sizing of the resulting JSON object for stable size of the overall output is the primary reason for pad characters. In order to create the same size output both before and after the SAID is added into the JSON there must be an equivalently sized number of pound signs (44 in this case) placed into the same field where the SAID will go. This is used in CESR encoding because CESR data types are encoded with to a type, length, and value scheme (TLV scheme) that simplifies parsing. Size of the overall output is the length, or “L,” in TLV and it only works if you have a known width data.

{ "d": "############################################", "first": "john", "last": "doe" }

Now that you know the rules for calculating the number of pad characters then we are ready to illustrate the calculation process with diagrams.

Diagram for plain “naïve” Base64 encoding of SHA3-256 digest

Base64 uses post-padding, post-conversion of pad characters, as shown in the diagram below. You start with the raw digest. All the boxes in this diagram represent the raw bytes of the digest. There is no padding yet because the value is raw and is not yet converted to Base64 characters.

Binary bits of 32 byte SHA3-256 digest of above JSON with ‘#’ filler

For those following along in code the raw bytes of the 32 byte SHA3-256 digest of the JSON above (with the ‘#’ filler characters) are represented in binary as follows:

1111001001011011010101100010111010011111011001101111000110001101000010000000010010000011100010110000000000000001100111110110110000101001010000110100100101001000111110110110011100010001110100110010011010101000010001000100101011100100000011111110100011111010

Take a look at the last two bytes are 11101000 and 11111010. This factors in to the last two characters adjacent to the pad character as you see below.

Encode this 32 byte digest to Base64 URL Safe and you get get:

What happened here is that four bits (1010) of the last byte ( 11111010) were encoded into the last character lowercase ‘o’ adjacent to the pad character. If you look at the value for lowercase o in the Base64 alphabet you will see that lowercase ‘o’ has the bit pattern 101000. Yet it only pulled four bits from the last byte of 11111010 so where did the last two bits (00) come from? They were added in by the Base64 encoder. These two pad bits are why the corresponding final value has a single equals sign ‘=’ pad character. That instructs the Base64 encoder to strip two bits from the last character during the decoding process:

IMPORTANT: Base64 does not add the padding to the raw bytes prior to conversion. Instead it adds the padding while converting the 6 bit groups of the raw bytes into Base64 characters.

Due to the fact that 32 bytes, 256 bits, does not evenly align on a 24 bit boundary, is not a multiple of 24, the Base64 encoder splits the last byte into two different Base64 characters since 8 bits does not evenly fit in one 6 bit group and must be spread across two 6-bit groups. Each of these 6 bit groups each get their own Base64 character. In this case, the last two bytes 11101000 and 11111010 get spread across the last two characters ‘P’ (001111) and ‘o’ (101000).

Because of how the math works when splitting the 8-bit byte groups into 6-bit Base64 character groups the ‘o’ character got four bits from the very end of the digest. Yet four bits is not enough for a Base64 character so the Base64 encoder adds two zero bits on the end, signified with white boxes containing zeroes. Before the pad character is added then we are at 43 Base64 characters (6 bit groups, 258 bits), which is not a multiple of 24 bits. When the pad character ‘=’ is added then we get to a 44 characters (264), which is a multiple of 24 bits, meaning the encoding completed successfully.

Base64 Encoded SHA3-256 Digest

With the fully padded value you end up with a valid, encoded, Base64 value that looks like the following bit diagram:

The C2 character at the end shares some bits with the raw bytes of the digest and also contains some padding zero bits. The last character, C1, is an equals sign ‘=’ pad character. The fact that there is one pad character indicates to the Base64 decoder that there are two zeroed pad bits to remove from the last character, ‘C2’, during decoding in order to get back to the original digest bytes.

‘=’ is wasted space?

You could consider the pad characters ‘=’ as wasted space that could be useful if repurposed. All of the pad bits used for the equals sign could represent something. This is exactly what CESR does except it moves the padding to the front of the bytes so that it can have a uniform TLV encoding format. TLV encoding formats require the type character to be at the front of the value, so using post-padding like Base64 does would not work.

Along these same lines, SAIDs do not use Base64-style padding because it does not enable separability of individual concatenated values due to the fact that there is no easy way to cleanly and reliably separate individual values out of a Base64 encoded stream of bytes. The CESR specification introduction mentions this:

This Composability property enables the round-trip conversion en-masse of concatenated Primitives between the text domain and binary domain while maintaining the separability of individual Primitives.

Composable Event Streaming Representation ToIP specification – Dr. Sam Smith

Now that you understand how the plain or “naïve” Base64 encoding works then we turn our attention to CESR style pre-padding.

CESR Byte Padding: Pre-padding, Pre-conversion

In CESR the padding of values occurs with the raw bytes prior to encoding to Base64 as shown below in the white box containing ‘B33.’

What this means is that the raw value, prior to conversion, already aligns on a 24 bit boundary. Due to this alignment pre-conversion then there will never be any Base64 pad characters ‘=’ in the output.

How many bytes to prepend?

How do you know how many bytes to prepend? With a similar calculation we did above to find the number of filler characters

Since the SHA3-256 digest we start with is 32 bytes, or 256 bits (not a multiple of 24), then all we need to add is one byte to get to 264 bits, which is a multiple of 24, or 33 bytes.

Again, once you know the quantity of bytes that align on a 24 bit boundary you can do a simple calculation to get to the number of pad characters for your digest. Since 6 bits of every byte are put into a Base64 character (6 bit groups) then you can divide your total number of bits (264) by 6 to get the number of Base64 characters of your final digest.

264 (bits) / 6 (bits per Base64 character) = 44 (Base64 Characters)

So 44 will be the quantity of filler characters to put into the JSON object in order to calculate a SAID.

What happens when prepending bytes for CESR style encodings?

When encoding a value that requires padding with CESR-style padding (up front), instead of ‘=’ at the end like Base64-style padding would produce you end up with ‘A’ characters on the front of your encoded value. You also end up with the one character adjacent to the ‘A’ character(s) including some pad bits and some raw bits, as shown below in the bit diagram.

The intermediate encoded value looks like the below value that is not yet a SAID. This is not yet a SAID because the ‘A’ character has not yet been replaced with a type code from the TLV scheme indicating this is a SHA3-256 digest.

This ‘A’ character represents all zero bits (000000) in the Base64 alphabet.

In binary the full, pre-padded digest value (all 33 bytes) looks like the following. Notice the zero bits at the front.

000000001111001001011011010101100010111010011111011001101111000110001101000010000000010010000011100010110000000000000001100111110110110000101001010000110100100101001000111110110110011100010001110100110010011010101000010001000100101011100100000011111110100011111010

The first two bytes are 00000000 and 11110010 which get encoded into Base64 as shown below. Six of the zero pad bits get encoded as an ‘A’ character and two of the zero pad bits get included in the capital ‘P’ character which also has four bits from the next raw byte of data from the digest.

Bit diagram of Base64 encoded, CESR pre-padded raw value.

This diagram illustrates how CESR does pre-padding with pad bytes of zero bits prior to performing a Base64 encoding on the fully padded raw value. The next diagram of characters shows the space a fully padded, encoded, CESR-style value would look like.

As you can see, the padding is at the front of the encoded value rather than the back like Base64 does. And the character with shared pad and raw bits is adjacent to the pad character at the front of the Base64 encoded value.

To get to the final SAID then you replace the ‘A’ character with the appropriate type code, or derivation code, yet we are getting ahead of ourselves a bit too much. Let’s now get into the calculation of the digest.

This step showed you how to calculate the appropriate number of filler ‘#’ pound sign characters to put in to the digest field in your JSON object. The next step shows you how to calculate a digest of that JSON object.

Creation Step 3: Calculate a digest of the data

When calculating a digest then you take the data with the correct number of filler characters added to the digest field and you simply take a digest of it. So in our case we would take a digest of the following

{ "d": "############################################", "first": "john", "last": "doe" }

In Python taking a digest of this data would be as simple as the following:

import hashlib raw_value = b'{"d":"############################################","first":"john","last":"doe"}' digest = hashlib.sha3_256(raw_value).digest() # hash function ↑↑↑↑

This is a simple step and is very similar for any other algorithm such as SHA2-256, Blake3-256 or otherwise. You use the desired type of hash function.

The only other thing to be aware of here is that if you create a digest that is sized differently than 32 bytes, such as a SHA3-512 digest (64 bytes) then you need to also change the number of pad bytes, which gets into the next step.

Creation Step 4: Calculate the quantity of pad bytes

The calculation for the quantity of pad bytes is very similar to the calculation for the quantity of filler ‘#’ characters needed in Step 2. In fact, it is a subset of that calculation. The goal with pad characters is to make sure that the final value aligns on a 24 bit boundary as mentioned above.

For example, since the SHA3-256 digest we start with is 32 bytes, or 256 bits (not a multiple of 24), then all we need to add is one byte to get to 264 bits, which is a multiple of 24, or 33 bytes.

Deeper into Modulus Math for Pad Bytes

To get a bit deeper into the math, one way to do this calculation with the modulus operator is to find out how many characters are necessary to completely fill a 3 byte group. Since 3-byte groups are multiples of 24 then you can use a modulus calculation to see how far away you are from filling a three byte group by doing a modulus 3 operation in two steps:

Step 1: take bytes mod 3

32 bytes mod 3 = 2 (bytes)

meaning there are two bytes already in the last group of three (24 bit boundary).

Step 2: subtract bytes in group from group size

So to see how many bytes you must add to get to the 24 bit boundary (3 byte group) you subtract the quantity of bytes you have from the group size:

3 (group size) – 2 (bytes in group) = 1 (pad bytes needed to fill group)

Due to how modulus arithmetic works you will only ever have three possible values from this equation:

3 – (bytes mod 3) = 0 (pad bytes) 3 – (bytes mod 3) = 1 (pad bytes) 3 – (bytes mod 3) = 2 (pad bytes)

You never have to worry about three pad bytes because that would be an even multiple of 24 which means your raw value would already align on a 24 bit boundary and thus not need any pad bytes.

So, to review, for us the calculation of (3 - (32 mod 3)) = 1 pad byte gives us a single pad byte to be prepended to our raw value, as shown below in the ‘B33’ box.

As mentioned before, CESR does pre-padding, pre-conversion which means that the pad byte we found we need is added to the front of the array of raw bytes for the SHA3-256 digest. The next step covers encoding this padded raw value.

Creation Step 5: Base64 URL Safe Encode the padded raw bytes

Now that the raw value from Step 4 is properly padded then you encode it with Base64 URL Safe encoding. CESR uses Base64 URL Safe encoding rather than plain Base64 encoding so that CESR values can safely be used in URLs and filenames.

import hashlib from base64 import urlsafe_b64encode raw_value = b'{"d": "############################################", "first": "john", "last": "doe"}' digest = hashlib.sha3_256(raw_value).digest() padded_digest = b'\x00' + digest encoded = urlsafe_b64encode(padded_digest) # encode to base64 ↑↑↑↑ assert encoded == b'APJbVi6fZvGNCASDiwABn2wpQ0lI-2cR0yaoRErkD-j6' assert len(encoded) == 44

Now that you have the Base64 URL Safe encoded value then you are ready to finish off this SAID creation by replacing the ‘A’ pad character at the front of the encoded value with the appropriate value from the CESR Master Code Table.

Creation Step 6: Substitute Type Code for the front ‘A’ character(s)

When CESR pre-padded the raw value to get to a 24 bit boundary the purpose of that was to be able to repurpose the wasted space of the pad character for a type code in CESR’s TLV encoding scheme. The ‘A’ character at the front of the value in this scheme is considered to be a pad character. This pad ‘A’ character will be replaced with the appropriate type code, or derivation code in CESR parlance, from the CESR Master Code Table.

For a SHA3-256 digest that type code is ‘H’ as seen in the following subset of the CESR Master Code Table.

The substitution gives us a final value of HPJbVi6fZvGNCASDiwABn2wpQ0lI-2cR0yaoRErkD-j6 as seen in the following substitution diagram.

The substitution of the ‘A’ character with the ‘H’ character is the final part of what is called CESR encoding a raw digest value into a CESR-style self addressing identifier. This SAID is a front-padded, Base64 encoded, and type-code substituted, string of Base64 characters.

The final value can be created by the code as follows:

import hashlib from base64 import urlsafe_b64encode raw_value = b'{"d":"############################################","first":"john","last":"doe"}' digest = hashlib.sha3_256(raw_value).digest() padded_digest = b'\x00' + digest encoded = urlsafe_b64encode(padded_digest) b64_str_list = list(encoded.decode()) # convert bytes to string of chars for easy replacement of 'A' b64_str_list[0] = 'H' # replace first 'A' character with 'H' type code b64_str = ''.join(b64_str_list) # convert string of chars to string with .join() assert b64_str == 'HPJbVi6fZvGNCASDiwABn2wpQ0lI-2cR0yaoRErkD-j6' assert len(b64_str) == 44 # length should still be 44 characters, 264 base64 bits, a multiple of 24 bits Creation Step 7: Place the Front-Padded, Base64 encoded, Type-code Substituted string in the digest field

Now we can take this correctly padded, CESR encoded value and place it into the digest field in our JSON object, replacing the filler ‘#’ characters with the final, valid SAID:

{ "d": "HPJbVi6fZvGNCASDiwABn2wpQ0lI-2cR0yaoRErkD-j6", "first": "john", "last": "doe" }

This takes us back to where we started off, with a valid SAID and a SAIDified JSON object.

What about verification?

What is nice about verification is that it is as simple as calculating the SAID again of a JSON object and comparing that to a SAID you are handed.

Verification Step 1: Start with a SAID from the object you already have

Say you are starting with the below object that has already had a SAID calculated and embedded in the digest field, the “d” field here.

{ "d": "HPJbVi6fZvGNCASDiwABn2wpQ0lI-2cR0yaoRErkD-j6", "first": "john", "last": "doe" }

To get the SAID from this object you extract the value of the “d” field, giving you HPJbVi6fZvGNCASDiwABn2wpQ0lI-2cR0yaoRErkD-j6

Verification Step 2: Calculate the SAID of the object using the SAID creation steps

Verification is easy because all you need to do is take steps 1 through 6 above and re-calculate the SAID on the JSON object provided. Once you have recalculated the SAID, which will be HPJbVi6fZvGNCASDiwABn2wpQ0lI-2cR0yaoRErkD-j6 again, you can perform the comparison in step 3.

Verification Step 3: Compare the SAID from the object to the calculated SAID

If the SAID the object started with matches the SAID you calculated from the object then you know the object has not been changed and that the SAID is valid. Otherwise either your SAID is invalid or the object has changed.

Review Calculating a SAID

Now you understand how we SAIDify a JSON object by doing the following seven step process:

Start with a JSON object we want to add a SAID to that has a digest field. Calculate the quantity of Base64 characters the final, pre-padded, encoded raw digest bytes (SAID) will take up and fill the digest field with that many ‘#’ characters. Calculate a digest of the bytes of the JSON object after the ‘#’ filler characters are added. Calculate the quantity of pad bytes needed to align on a 24 bit boundary and prepend that to the raw bytes for a digest. Encode the padded raw bytes with the Base64URLSafe alphabet. Substitute the appropriate type code in place of the ‘A’ character(s) at the front of the encoded string. This final value is your SAID Place the final SAID value into the digest field of your JSON object.

Pre-padding and type code substitution prior to Base64 encoding is the essence of CESR-style self addressing identifiers. The steps above may seem overwhelming at first, though once you mentally anchor in that CESR pads at the start and that padding gives you ‘A’ characters you can reuse for type codes then you have mastered the fundamentals of what makes CESR style SAIDs work.

Verifying a SAID

Verification of a SAID is easy because you just calculate it again from the original JSON object, or other data object you are using. If the SAIDs match then it verifies; if they don’t then the data changed.

Extra Learning Alert – fully qualified Base64 primitive

And, as a nice side note, you happen to now know what the phrase “fully qualified base64 primitives” in KERIpy means. All that means is that your encoded value has been pre-padded, pre-conversion, and has had its type code added to the front, as we did here with substitution, with the exception that some CESR primitives

Give me a library please! I don’t want to manage these details

In case this article has convinced you that you do not ever again want to worry about the vagaries of aligning on 24 bit boundaries for Base64 or CESR values then you are in luck. There are multiple implementations of the SAID process that can meet your needs in a variety of different languages.

The Python reference implementation in Web Of Trust’s KERIpy’s Saider.saidify. The Human Colossus Foundation’s Rust implementation with WASM bindings for their JavaScript package. See their cool SAID generator and verifier demo here where you can try a whole list of different algorithms. SAIDify, my own Typescript implementation of the SAID creation process. Implementations Web Of Trust KERIpy Python

The Python example below from KERIpy shows a unit test showing the usage of the KERIpy Saider.saidify library code to calculate a SAID. The SAID is stored in the .qb64 property of Saider. The term qb64 stands for “qualified base64” which means a left-padded, Base64 encoded, type code substituted value as described above.

import json from keri.core.coring import MtrDex, Saider def test_saidify_john_doe(): code = MtrDex.SHA3_256 ser0 = b'{"d": "", "first": "john", "last": "doe"}' sad0 = json.loads(ser0) saider, sad = Saider.saidify(sad=sad0, code=code) assert saider.qb64 == 'HPJbVi6fZvGNCASDiwABn2wpQ0lI-2cR0yaoRErkD-j6' Human Colossus Foundation Rust SAID demo and test code

Start with their cool demo site of generating and verifying SAIDs:

If you want to dive into their code the linked test basic_derive_test shows the Rust code for the cool SAD macro #[derive(SAD, Serialize)]that can turn any Rust struct along with the #[said] field attribute for the SAID digest field into a self-verifying data structure.

use said::derivation::HashFunctionCode; use said::sad::SAD; use said::version::format::SerializationFormats; use said::SelfAddressingIdentifier; use serde::Serialize; #[test] pub fn basic_derive_test() { #[derive(SAD, Serialize)] struct Something { pub text: String, #[said] pub d: Option<SelfAddressingIdentifier>, } let mut something = Something { text: "Hello world".to_string(), d: None, }; let code = HashFunctionCode::Blake3_256; let format = SerializationFormats::JSON; something.compute_digest(&code, &format); let computed_digest = something.d.as_ref(); let derivation_data = something.derivation_data(&code, &format); assert_eq!( format!( r#"{{"text":"Hello world","d":"{}"}}"#, "############################################" ), String::from_utf8(derivation_data.clone()).unwrap() ); assert_eq!( computed_digest, Some( &"EF-7wdNGXqgO4aoVxRpdWELCx_MkMMjx7aKg9sqzjKwI" .parse() .unwrap() ) ); assert!(something .d .as_ref() .unwrap() .verify_binding(&something.derivation_data(&code, &format))); } SAIDify

If you want to use a Typescript library that is about 530 lines of code you can go with my SAIDify library. The below example shows how to use the library with Typescript.

Start with an NPM install

npm install saidify

And then you can use the saidify(data, label) function to SAIDify any JavaScript object you have as long as you indicate which field is the digest field, the “label” field, which defaults to the “d” field.

import { saidify, verify } from 'saidify' // create data to become self-addressing const myData = { a: 1, b: 2, d: '', } const label = 'd' const [said, sad] = saidify(myData, label) // said is self-addressing identifier // sad is self-addressing data console.log(said) // ...Vitest test assertion expect(said).toEqual('ELLbizIr2FJLHexNkiLZpsTWfhwUmZUicuhmoZ9049Hz') // verify self addressing identifier const computedSAID = 'ELLbizIr2FJLHexNkiLZpsTWfhwUmZUicuhmoZ9049Hz' const doesVerify = verify(sad, computedSAID, label) // can verify with original myData or sad // ...Vitest test assertion expect(doesVerify).toEqual(true) Conclusion

The key takeaways from calculating SAIDs are:

Use pre-padded bytes to align on a 24 bit boundary prior to encoding as Base64 characters. Substitute type codes in for the leading ‘A’ character(s) of a SAID. It is easy to chose different algorithms for the SAID calculation process. Just make sure you use a code on the CESR Master Code Table if you want to be CESR compliant. There are multiple implementations of the SAID algorithm you can use.

Now go make some SAIDs!

References: HCF oca-spec #58 RFC 4648: The Base16, Base32, and Base64 Data Encodings, specifically section 5 Composable Event Streaming Representation (CESR) ToIP Specification, specifically section 12.6 Self Addressing Identifier IETF draft specification SADs, SAIDs, and ACDCs video presentation by Daniel Hardman

Michael Ruminer

Anthropic’s Contextual Retrieval

One of the problems I have encountered in trying to do Retrieval Augmented Generation (RAG) where a complete single document was not uploaded for queries on that document alone is that chunks created from documents for embedding often lose context. I have often considered what would be the outcome if one were to put some limited but useful context along with each chunk. Now I know. Introducing Ant

One of the problems I have encountered in trying to do Retrieval Augmented Generation (RAG) where a complete single document was not uploaded for queries on that document alone is that chunks created from documents for embedding often lose context. I have often considered what would be the outcome if one were to put some limited but useful context along with each chunk. Now I know. Introducing Anthropic's contextual retrieval concept.

One of my concerns with placing some additional context for each chunk is that you’d probably need to pass the entire document being chunked as context along with each individual chunk. That would be very expensive and slow. Now, with Anthropic’s Claude prompt caching, the cost and latency is reduced significantly and it seems actually doable for chunk contexting (is contexting a word?). An initial prompt including the full document could be done with that prompt cached for future prompt reference.

I plan to try this out.

Check out the Anthropic “Introducing Contextual Retrieval” post for greater details.

Saturday, 21. September 2024

IdM Laboratory

AuthZEN WGがAuthorization APIのImplementer's draftがPublic Review期間に入ります

こんにちは、富士榮です。 XACMLの苦い思い出からでしょうか?認可とかロール管理は鬼門と言われて久しいわけですが、昨年末〜今年の頭にOpenID Foundatinoのワーキンググループとして立ち上がったばかりのAuthZEN WGが早くもAuthorization API 1.0のImplementer's draftを出してきました。 https://openid.net/review-period-proposed-authzen-authorization-api-1-0-implementers-draft/ 今後のスケジュールはこんな感じです。 Implementer’s Draft public review period: Wednesday, September 17, 2024 to Friday, November 1, 20
こんにちは、富士榮です。
XACMLの苦い思い出からでしょうか?認可とかロール管理は鬼門と言われて久しいわけですが、昨年末〜今年の頭にOpenID Foundatinoのワーキンググループとして立ち上がったばかりのAuthZEN WGが早くもAuthorization API 1.0のImplementer's draftを出してきました。
https://openid.net/review-period-proposed-authzen-authorization-api-1-0-implementers-draft/


今後のスケジュールはこんな感じです。 Implementer’s Draft public review period: Wednesday, September 17, 2024 to Friday, November 1, 2024 (45 days) Implementer’s Draft vote announcement: Saturday, October 19, 2024 Implementer's Draft early voting opens: Saturday, October 26, 2024 * Implementer’s Draft voting period: Saturday, November 2, 2024 to Saturday, November 9, 2024 (7 days) 対象となるAuthorization APIの仕様はこちらにあります。 https://openid.net/specs/authorization-api-1_0-01.html
今年は読むものが多すぎる・・・、まぁ徐々に読んでいきましょう。

Jon Udell

deo absente deum culpa

On a recent trip I saw this pair of Latin phrases tattooed on the back of a flight attendant’s arms: Left: Deo absente. Right: Deum culpa. I took Latin in middle school, and could guess what the combination might mean. It’s not a common construction, and a search seems to confirm my guess. Both Google … Continue reading deo absente deum culpa

On a recent trip I saw this pair of Latin phrases tattooed on the back of a flight attendant’s arms:

Left: Deo absente. Right: Deum culpa.

I took Latin in middle school, and could guess what the combination might mean. It’s not a common construction, and a search seems to confirm my guess. Both Google and Bing take you to a couple of Reddit posts in r/Latin.

Would this be the correct translation?

A song I like, Deus in absentia by Ghost, has that line in it intending to mean “In the absence of God”, so I was looking into alternate translations/syntax of the phrase intending to mean “In the absence of God; Blame/Fault God”. Would this make sense: “Deum in absente; Culpa Deus” or “Deus Culpa”?

Does the phrase “Deus In Absentia, Deus Culpa” make sense?

I’m using this for a tattoo and want to be absolutely sure it works in the sense of ‘In the absence of God, blame God’. All help appreciated!

Is that the same person I saw? If so, the responses in r/Latin seem to have guided them to the final text inked on their arms. And if so, the message is essentially what I had guessed. The intent of the message, though, is open to interpretation. I’m not quite sure how to take it. What do you think it means? Would it have been rude to ask?

Friday, 20. September 2024

IdM Laboratory

LINEログインの拡張に向けたロードマップ

こんにちは、富士榮です。 LINEログインやLIFFのロードマップが更新されていますね。 https://developers.line.biz/ja/docs/line-login/roadmap/ ログイン回りをピックアップしておきましょう。 リリース時期機能概要対象プロダクト2024年後半LINEログインのダークモード表示の対象拡大LINEログインにおいて、ダークモード表示に対応するページの範囲を拡大します。LINEログイン時期未定LINEログイン v1、v2.0へのアクセス遮断既に廃止されているLINEログイン v1、および非推奨となっているLINEログイン v2.0へのアクセスを完全に遮断します。LINEログイン継続実施アクセシビリティ対応LINEログインを、より多くのエンドユーザーにとって使いやすいプロダクトにするため、アクセシビリティ対応を継続して推進してい

こんにちは、富士榮です。

LINEログインやLIFFのロードマップが更新されていますね。

https://developers.line.biz/ja/docs/line-login/roadmap/



ログイン回りをピックアップしておきましょう。

リリース時期機能概要対象プロダクト2024年後半LINEログインのダークモード表示の対象拡大LINEログインにおいて、ダークモード表示に対応するページの範囲を拡大します。LINEログイン時期未定LINEログイン v1、v2.0へのアクセス遮断既に廃止されているLINEログイン v1、および非推奨となっているLINEログイン v2.0へのアクセスを完全に遮断します。LINEログイン継続実施アクセシビリティ対応LINEログインを、より多くのエンドユーザーにとって使いやすいプロダクトにするため、アクセシビリティ対応を継続して推進していきます。LINEログイン継続実施同意画面をより使いやすいものに改善LINEログインを、より多くのエンドユーザーにとって使いやすいプロダクトにするため、A/Bテストなどの分析に基づき、機能やUIの改善を継続して推進していきます。LINEログイン


まぁ、見た目の話が多いですが、旧バージョンのAPIが遮断される時期は引き続きみておかないといけませんね。(少なくとも1.0はもう使われていませんが、2.0からは結構使われているので)


Jon Udell

SQL translation from Postgres to SQLite and DuckDB

Powerpipe dashboards can now connect not only to Steampipe but also to SQLite and DuckDB. This creates a combinatorial explosion of possibilities, including dashboards that use SQL to visualize large datasets read from Parquet files by DuckDB. SQL Translation From Postgres to SQLite and DuckDB Part of the LLM series at The New Stack.

Powerpipe dashboards can now connect not only to Steampipe but also to SQLite and DuckDB. This creates a combinatorial explosion of possibilities, including dashboards that use SQL to visualize large datasets read from Parquet files by DuckDB.

SQL Translation From Postgres to SQLite and DuckDB

Part of the LLM series at The New Stack.

Thursday, 19. September 2024

IdM Laboratory

MVPアワードでもらったOpenBadgeを覗いてみる

こんにちは、富士榮です。 MicrosoftからMVP(Most Valuable Professional)というアワードをいただき初めて今年で15年目、という話を前に書きましたが、最近この手のアワードやIT系の資格試験などではOpenBadge形式で資格証明がもらえることがあります。 これまで何度かイベントではOpenBadgeの中身について話をしてきましたが、要するにPNGやSVGファイルの中身(PNGならiTXt領域、SVGならopenbadge属性)へJSON-LDで記述されたクレデンシャルの情報を埋め込んでいます(OpenBadge 2.0、Hosted形式の場合)。 この埋め込み作業のことをBake(ベイク)と言い、こちらに技術仕様が公開されています。 https://www.imsglobal.org/sites/default/files/Badges/OBv2p

こんにちは、富士榮です。

MicrosoftからMVP(Most Valuable Professional)というアワードをいただき初めて今年で15年目、という話を前に書きましたが、最近この手のアワードやIT系の資格試験などではOpenBadge形式で資格証明がもらえることがあります。

これまで何度かイベントではOpenBadgeの中身について話をしてきましたが、要するにPNGやSVGファイルの中身(PNGならiTXt領域、SVGならopenbadge属性)へJSON-LDで記述されたクレデンシャルの情報を埋め込んでいます(OpenBadge 2.0、Hosted形式の場合)。

この埋め込み作業のことをBake(ベイク)と言い、こちらに技術仕様が公開されています。

https://www.imsglobal.org/sites/default/files/Badges/OBv2p0Final/baking/index.html


なお、最近はOpenBadge 3.0がW3C Verifiable Credentials Data Model 2.0を使って定義されようとしています。(Finalizeと言いつつVCDM2.0がFinalにならないのでどうするつもりなんだ、という話もありますが)

OpenBadgeにはHosted型とSigned型の2種類が定義されており、これまでは検証時に発行者のURLへ問い合わせを行うHosted型が中心でしたが、OpenBadge 3.0からはVerifiable Credentialsを利用することでSigned型(バッジ単体で検証ができる)が中心になってくると思われます。


ただ、現状はMVPバッジはCredly社(IT系の資格の受験をするときにお世話になった方も多いであろうピアソン社を数年前に買収した会社で民間の資格証明の最大手ですね)が発行するOpenBadge 2.0、Hosted型のバッジです。

こんな感じでCredlyのバッジポータルでバッジの確認やダウンロードができます。



ちなみに実際のOpenBadgeイメージはこちらから取得できますので、こうやってブログやWebサイトで他の方へ提示(配布)することもできるわけです。




さて、では中身を覗いてみましょう。

先ほど書いた通りiTXt領域にJSON-LD形式で埋め込まれているわけですが、IMS Globalがこちらで検証サイトを公開しているので、こちらを使いましょう。

https://openbadgesvalidator.imsglobal.org/

このサイトを使って取り出したOpenBadgeがこちらです。

{ "input": { "value": "https://www.credly.com/api/v1/obi/v2/badge_assertions/8793fdbf-80a5-4240-977e-12ac45574df3", "input_type": "url" }, "graph": [ { "@context": "https://w3id.org/openbadges/v2", "type": "Assertion", "issuedOn": "2024-09-17T00:00:00.000Z", "recipient": { "type": "email", "identity": "sha256$071281e466032326ffe4c3238545d31970b2b61d75fae181a283ac0aace09264", "hashed": true }, "badge": "https://www.credly.com/api/v1/obi/v2/issuers/214390fb-07bc-4575-80f2-f2c325f71c49/badge_classes/2b797f06-fdd4-4ec7-b13c-e8f79915da0c", "verification": { "type": "HostedBadge" }, "evidence": [], "id": "https://www.credly.com/api/v1/obi/v2/badge_assertions/8793fdbf-80a5-4240-977e-12ac45574df3" }, { "@context": "https://w3id.org/openbadges/v2", "tags": [ "Community", "Leadership", "Technology" ], "name": "2024 Microsoft Most Valuable Professional (MVP)", "image": "https://images.credly.com/images/9e9359a4-fe7e-4e02-8eb0-6c2b7947345a/image.png", "alignment": [], "criteria": { "narrative": "Award: Individuals must have deep knowledge and expertise aligned to a Microsoft product or service or related open-source technologies. Through community work, qualifying applicants should be able to demonstrate their technical expertise. https://mvp.microsoft.com/", "id": "https://www.credly.com/org/microsoft-student-programs/badge/2024-microsoft-most-valuable-professional-mvp" }, "id": "https://www.credly.com/api/v1/obi/v2/issuers/214390fb-07bc-4575-80f2-f2c325f71c49/badge_classes/2b797f06-fdd4-4ec7-b13c-e8f79915da0c", "type": "BadgeClass", "description": "The Microsoft MVP Program recognizes outstanding members of technical communities for their community participation and willingness to help others. Above all else, it is a people-powered program, made up of individuals whose passionate commitment to innovation has made its dynamic growth possible.", "issuer": "https://www.credly.com/api/v1/obi/v2/issuers/214390fb-07bc-4575-80f2-f2c325f71c49" }, { "@context": "https://w3id.org/openbadges/v2", "type": "Issuer", "id": "https://www.credly.com/api/v1/obi/v2/issuers/214390fb-07bc-4575-80f2-f2c325f71c49", "name": "Microsoft MVP and Student Ambassadors Communities", "image": "https://images.credly.com/images/8f11d73e-9c33-4788-a602-88c761957c90/blob.png", "description": "The Microsoft MVP and Student Ambassadors Programs are communities where student experts, technology professionals or industry leaders transform their passions and skills into globally recognized community leadership.", "email": "info@credly.com", "url": "https://mvp.microsoft.com/" } ], "report": { "valid": true, "messages": [], "warningCount": 0, "validationSubject": "https://www.credly.com/api/v1/obi/v2/badge_assertions/8793fdbf-80a5-4240-977e-12ac45574df3", "openBadgesVersion": "2.0", "errorCount": 0 } }

このとおり、IssuerはCredly社となっており、マイクロソフトが直接発行しているわけではないことがわかります。 また、OpenBadgeの特徴としてあくまで資格情報を表現するものとなっていることが挙げられます。何を言っているかというとバッジを提示する人とバッジが指し示す人が一致していることは表現していないということです。(私が他人のバッジを保持して提示することも可能。まぁ単なる画像ファイルですから当然ですね) そのため、Verifyをする際にバッジを提示する主体とバッジが指し示す主体が一致していることを検証サイト側で行うことが必要となります。
具体的な方法としては、recipientのidentity要素に入っている値(バッジが指し示す主体)が提示してくる主体と同じことを確認することになりますので、提示する際に利用者にメールアドレス(type: emailの場合)を入力させ、 到達性確認を行うことで提示者が当該メールアドレスに対してアクセスが可能な状態であることを確認する 到達性確認ができたメールアドレスとバッジを検証URL(発行元。今回のケースだとCredly)へ投げ込む 検証URL側は自身が管理するsalt値を使って送られてきたメールアドレスの値をSHA256(もしくはMD5)でハッシュ化する バッジの中にはいっているidentityの値と生成したハッシュ値が同一であることを確認する(提示者はバッジの中を見てもメールアドレスの値がハッシュ化されているため、誰のバッジかわからない。持ち主だけがバッジ発行時のRecipient情報として設定したメールアドレスを知っているので発行対象の主体とバッジの指し示す主体が一致しているであろうことを推測する) この部分) "recipient": { "type": "email", "identity": "sha256$071281e466032326ffe4c3238545d31970b2b61d75fae181a283ac0aace09264", "hashed": true },

非常に簡易的な仕組みですが、身分証明に使うわけではありませんしこのくらいのゆるさでもOKってことでしょう。
しかしVCにもVC2.0があったり、SD-JWTがあったりと複雑ですが、こういう形で教育業界でもOpenBadgeのバージョンやアーキテクチャ(Hosted/Signedなど)の混在による混乱もありそうですね。
引き続き見ていきたいと思います。


The Pragmatic Engineer

The Pulse #107: What does Amazon’s 5-day RTO mean for tech?

Amazon is the first Big Tech to mandate a strict 5-day return to office. What are the real reasons, will Amazon see a ‘brain drain’ as a result, and could other Big Tech companies follow?

The Pulse is a series covering insights, patterns, and trends within Big Tech and startups. Notice an interesting event or trend? Send me a message.

Today, we cover:

Industry pulse. OpenAI’s new model, LinkledIn opts everyone into GenAI training (save for those in the EU), Unity reverses runtime fees, what a startup “fire sale” looks like for employees, and more.

What does Amazon’s 5-day RTO mean for tech? The online retailer is the first Big Tech giant to mandate a strict, 5 days in the office policy. It says it wants things back exactly like they were before the pandemic. One reason is likely something less discussed: tax incentives from cities granted for staff being onsite, daily

Tech debt: further learnings. Two weeks ago, Lou Franco shared hard-earned lessons on keeping tech debt at bay. Lou shares two additional learnings: using tech debt payments to get into the flow, and how big rewrites need heavyweight support.

1. Industry pulse OpenAI’s new model: o1

Last week, OpenAI revealed its latest model called OpenAI o1. Until now, LLM models got bigger in how much data they were trained on, but o1 is a change of direction. It spends more time “thinking” before it responds: basically, it chains several LLM operations one after the other. As software engineer Simon Willison puts it:

“One way to think about these new models is as a specialized extension of the chain of thought prompting pattern — the “think step by step” trick that we’ve been exploring as a a community for a couple of years now, first introduced in the paper Large Language Models are Zero-Shot Reasoners in May 2022.”

The new model is far more capable in its responses – even though these responses take more time to process – and brings a new dimension in how to use language models. Because the model spends more compute time on answers, it has become more expensive to use: o1 costs 100x as much as GPT-4o mini, and 3-6x as much as GPT-4.

This approach feels like the biggest recent development in LLMs this year. It’s worth withholding judgment until we use it more, and learn more about its capabilities and tradeoffs. AI keeps moving fast, and OpenAI is at the forefront.

LinkedIn opts everyone into GenAI training, except in EU

Read more


Michael Ruminer

A Great AI RAG Resource

I came across a great AI Retrieval Augmented Generation resource. It is a Github repo: Advanced RAG Techniques: Elevating Your Retrieval-Augmented Generation Systems.I’ll just copy and paste their introduction here. “Welcome to one of the most comprehensive and dynamic collections of Retrieval-Augmented Generation (RAG) tutorials available today. This repository serves as a hub for cutting-edge t

I came across a great AI Retrieval Augmented Generation resource.
It is a Github repo: Advanced RAG Techniques: Elevating Your Retrieval-Augmented Generation Systems.I’ll just copy and paste their introduction here.

“Welcome to one of the most comprehensive and dynamic collections of Retrieval-Augmented Generation (RAG) tutorials available today. This repository serves as a hub for cutting-edge techniques aimed at enhancing the accuracy, efficiency, and contextual richness of RAG systems.”

All I can say is, wow. It really covers a lot of ground. I plan to dig into it and will report back.

Wednesday, 18. September 2024

IdM Laboratory

SAMLerのためのOpenID Connect入門

こんにちは、富士榮です。 SAMLaiです。AIではありません。愛です。サムライです。 ということで、以前アナウンスさせていただいた大学ICT推進協議会(AXIES)の認証基盤部会主催の勉強会でSAMLを知っている人向けのOpenID Connect講座をやってきましたので資料をこちらにおいておきます。 もちろん一般企業などにも使える話だと思うので、ご覧ください。

こんにちは、富士榮です。

SAMLaiです。AIではありません。愛です。サムライです。


ということで、以前アナウンスさせていただいた大学ICT推進協議会(AXIES)の認証基盤部会主催の勉強会でSAMLを知っている人向けのOpenID Connect講座をやってきましたので資料をこちらにおいておきます。


もちろん一般企業などにも使える話だと思うので、ご覧ください。

Tuesday, 17. September 2024

IdM Laboratory

次回のOAuth Security Workshopはアイスランド

こんにちは、富士榮です。 次回のOAuth Security Workshop(OSW)の開催がアナウンスされています。 https://oauth.secworkshop.events/osw2025 2025年2月26日〜28日、アイスランドのレイキャビクで開催とのことです。 また一番寒い時期に寒いところへ・・・ 最高気温で3℃くらい、最低気温で-3℃くらいってところですね。 気象庁HPより いずれにしても11月24日、1月12日の2回に分けてセッションの公募の締め切りが設定されていますので、寒いところとOAuthが好きな人はぜひ、ってところですね。 (ちょっと行ってみたい)
こんにちは、富士榮です。
次回のOAuth Security Workshop(OSW)の開催がアナウンスされています。 https://oauth.secworkshop.events/osw2025

2025年2月26日〜28日、アイスランドのレイキャビクで開催とのことです。 また一番寒い時期に寒いところへ・・・
最高気温で3℃くらい、最低気温で-3℃くらいってところですね。
気象庁HPより


いずれにしても11月24日、1月12日の2回に分けてセッションの公募の締め切りが設定されていますので、寒いところとOAuthが好きな人はぜひ、ってところですね。 (ちょっと行ってみたい)

The Pragmatic Engineer

The Pragmatic Engineer: Three Years

How the newsletter evolved, popular topics the last year, and The Pragmatic Engineer Podcast launching

 Hi, this is Gergely with a free issue of the Pragmatic Engineer Newsletter. In every issue, I cover topics related to Big Tech and startups through the lens of software engineers and engineering managers. To get weekly articles in your inbox, subscribe:

Subscribe now

This publication has turned three years old, which feels like the right time to pause and reflect on the past 12 months for The Pragmatic Engineer.

At time of publication, 759,402 readers subscribe to this newsletter. This is 300,000 readers up on a year ago, when it was just under 461,000. This figure far exceeds the size of the audience I assumed would be interested in software engineering deep dives into some fascinating and challenging topics, when I launched this publication. Thank you for your trust!

I’d like to extend a very special thanks to all paying subscribers: the publication runs on your support, and it enables us to deliver well-researched deep dives. Many paying subscribers expense this newsletter from their company learning and development budget. If you have such a budget, here’s an email to send to your manager. There’s also reduced prices for people in countries with lower average income than the US and Western Europe, and student discounts. If you enjoy the newsletter, you can also gift a subscription to others.

Speaking of deep dives: over the past year full subscribers received two articles almost every week: a deepdive on Tuesdays, and tech news in The Pulse on Thursdays. The last 12 months of issues adds up to around 5-7 books’ worth of reading (about 550,000 words). Free subscribers also got well over a nonfiction book’s worth of reading in their inbox: the first parts of the Tuesday articles, and a full article monthly.

Today, we look back, and peer in to the future:

How The Pragmatic Engineer has evolved

Popular, interesting topics

What I learned about readers!

Notable events

Looking ahead, and The Pragmatic Engineer Podcast

Before we jump in: if you’re someone who enjoys podcasts, I have a treat for you. The Pragmatic Engineer Podcast is launching with the first episode next week. To get the first episode in your podcast player, subscribe on:

YouTube — you can also watch the podcast announcement here

Spotify

Apple Podcasts

… or your favorite player!

1. How The Pragmatic Engineer evolved in the last year

Software engineering is an ever-changing field where the best teams continuously experiment with new approaches, and change how they work. I’m also an engineer at heart, so it’s only natural to do the same with the newsletter. 

Here’s what’s changed in recent months; some of which you might have noticed, and other things that happened behind the scenes.

Unshared details from interesting tech companies

Some companies attract media attention due to how well they execute, and what this publication tries to do is learn how exactly they achieve it. I aim to bring details straight from the source, talking exclusively with engineering teams at companies in the spotlight. The last 12 months has seen exclusive articles on:

OpenAI: How they scaled ChatGPT, and Inside OpenAI: how does ChatGPT ship so quickly?

Anthropic: How the company built Artifacts (and details on how it operates)

Stripe: Inside Stripe’s engineering culture

Meta: How the company built Threads with a small team in 6 months

Bluesky: How a tiny team built a Twitter/X competitor and Inside Bluesky’s engineering culture

Oxide: A hardware/software startup on “hard mode”

In terms of deep dives, it’s been the most prolific year yet for this publication. Doing lengthy articles which lift the lid on how leading teams get things done, can be challenging. Understandably, some companies do not normally share the details you’ve been able to read in this newsletter. At the heart of delivering articles like those above, is trust. Engineering leaders and software engineers ultimately feel they have scope to be candid with me, and that means a lot. Their trust in myself and The Pragmatic Engineer team is a very rewarding part of writing this newsletter.

More articles mixing deep research with pragmatic observations

Since February, it’s not just me doing the extensive research that goes into each article: Elin Nilsson is this publication’s first Tech Industry Researcher. Previously, Elin spent 7 years at Spotify, working in various mobile and platform engineering teams. Before, she interned at both Spotify as an Android developer and Google as a web/UX prototyper. She’s brought a whole lot of things onboard since she joined in February: like her infectious enthusiasm for nerding out about tech topics, diving deep into technologies and trends, and bringing order to the chaos of having too much information to process on any given topic.

Since she’s come onboard, there’s been more deeply researched pieces published than ever, and each one of them went into more depth. Some of the articles that started with Elin’s insightful research:

GenZ software engineers according to older colleagues, and what GenZ devs really think

Bluesky: How it was built, and Inside Bluesky’s engineering culture

How do AI software engineering agents really work?

AI tooling for software engineers: Reality check, Reality check (part 2) and Rolling LLM tools out company-wide (Part 3)

Thanks to Elin, we can take on ambitious projects that were simply not possible when the Pragmatic Engineer launched because the research can take months, We’re on a roll with Elin, so expect these to keep coming! You can always suggest interesting topics for us to potentially look into

Goodbye “Scoop,” hello “Pulse”

During the first two years of this newsletter, there was usually an article every Thursday called “The Scoop,” which rounded up interesting things in the industry, and occasionally broke big tech stories. But as of a year ago, I am no longer doing this and the name change to “The Pulse” was part of this shift. I shared details about the change:

‘The Pulse’ just better represents the mission of these articles, which is to help you keep an “ear to the ground” on what’s happening across Big Tech and at startups – sectors which regularly affect the whole tech industry.

I receive and validate plenty of interesting information from insiders at these companies, but my main focus is analyzing what’s going on in our industry; the implications of that and the opportunities for software engineers and tech businesses.

Names matter, and I feel “The Scoop” suggests a more gossipy, tabloid newspaper-style approach than what these articles actually deliver. Since The Scoop’s first issue I’ve focused on relevant industry changes and what they mean. I hope “The Pulse” makes the intent of this series clear: to provide an ear to the ground, and the latest analysis of the state of our industry.

This change has been one I’m very happy with; it’s also partly why we’ve been able to publish more company deep dives straight from tech companies themselves.

When writing “The Scoop”, tech companies and devs whom I met were never quite certain if I had my “journalist” hat on (and would leak what they shared as news), or my “analytical” software engineer’s hat. 

This made people reluctant to share information that could potentially identify their company, or affect their career. This was taxing for me personally and I decided I don’t want to be in the “gossip” business, but in the software engineering one. For that reason, the change from The Scoop was made.

But while writing it, I learned a lot about journalism and met a lot of reporters. It’s a fascinating world, but one I don’t feel is for me. Also, the time I save on no longer verifying information for  news articles, means there’s more time to dive deep into relevant, interesting software engineering topics.

Industry Pulse

In the spirit of experimentation, I tried out a new section inside the Thursday “The Pulse” article. Industry Pulse is a roundup of tech events relevant for devs and EMs, with commentary.

It has delivered early insights on topics like The impact of Section 174 on the software industry, predicting pressure on commercial open source to make more money, the unfortunate “startup purge event,” analyzing why Amazon sunset several of its products in one fell swoop, and much more.

Readers feedback was very positive from the start. Also, I enjoy writing it and look forward to it every day: so it has stayed!

2. Popular, interesting topics

In case you’re looking for articles to read or re-read, here’s a selection of some standout ones from the past 12 months, in my opinion.

Popular topics

The end of 0% interest rates, and what this means for software engineers was the single most-read article of the past year. This article was in a highly-detailed 4-part series about what feels like the biggest industry change in years: the end of low interest rates.

It is rare for an economic change to have the kind of impact on an industry that higher interest rates are having. We’re finding out that tech startups and interest rates are more connected than many of us would like to think.

Measuring developer productivity? A response to McKinsey. Consultancy giant McKinsey is known for many things, but software engineering expertise is not one of them. Yet the globally known consultancy started selling advisory services about this: including the claim that they had developed a way to measure devs’ productivity.

Dev productivity is a notoriously tricky thing to accurately measure, so I teamed up with industry veteran for a grounded take on why measuring productivity is difficult, and sharing our suggestions on how to go about this kind of measurement.

Inside OpenAI: How does ChatGPT Ship So Quickly? Up to early 2024, OpenAI was dominating headlines by releasing new products and features with an incredible pace. The company was out-executing the likes of Google, Meta and every other AI startup. But how did they do it?

I sat down with Evan Morikawa, who headed up the Applied engineering team (the team also building ChatGPT). We learned how OpenAI operates ChatGPT like a “startup inside a startup,” engineering has an unusually tight integration with Research, has a high talent density in their San Francisco offices, and has a lot of smaller day-to-day habits that add up. While I advise to never blindly copy engineering practices: it’s helpful to understand what makes a startup like OpenAI execute so quickly.

Surprise uptick in software engineering recruitment. Recruitment activity tends to be quiet in the summer months. This is why I was surprised to hear from several developers that they are seeing a spike in LinkedIn reachouts in June and July – mostly from startups.

Talking with dozens of developers – and some hiring managers – we uncovered that startups seem to have increased their hiring pace for a variety of reasons. What was common is how they were all searching for experienced engineers, and were becoming more “bold” in their reachouts: pinging senior+ engineers working at Big Tech and well-funded startups, who are profiles that usually are more reluctant to move.

Other interesting topics

Here are articles that I especially enjoyed for one reason or the other. If you’ve not yet read them, they could be worth your time:

The past and future of modern backend practices. From the birth of the internet, through SOA and virtualization, to microservices, modular monoliths and beyond.

How Big Tech does Qualty Assurance (QA). Most Big Tech companies have no dedicated SDET, QA, or tester roles. How do they produce quality software? A look into how Microsoft, Google, Meta, Apple, Amazon, Uber and Netflix do it.

Stacked diffs (and why you should know about them). Meta and Google have been using stacking for closer to a decade: a coding workflow that is very efficient for small PRs. So what is stacking, and how come it’s not more widespread in the industry?

What is Secruity Engineering? A deep dive into the ever-changing field of security engineering; a domain that can feel intimidating to some software engineers.

Applied AI Software Engineering: RAG. Retrieval-Augmented Generation (RAG) is a common building block of AI software engineering. A deep dive into what it is, its limitations, and some alternative use cases.

Thriving as a Founding Engineer: Lessons from the Trenches. Being a founding engineer at an early-stage startup is a vastly different, broader role than many people think.

Adobe can’t buy Figma: the impact on the industry. Months after we covered Figma’s engineering culture, regulators blocked Adobe’s acquisition of Figma. This event was likely what prompted Big Tech to stop acquiring companies, and to do “talent raids” instead with AI companies they seek to purchase.

There’s more, of course! For the full list of published articles, browse past deepdives, or The Pulse issues.

3. What I learned about readers — about you!

This year was the first time I asked details about who you are: in the form of a survey. 

Disciplines

I write the newsletter for software engineers and engineering managers, and was curious to know if most readers are, indeed, in this field. Turns out this is broadly the case:

Readers, based on profession. Based on 2,395 responses

The majority of readers are in the software engineering field – ranging from entry-level to senior/lead/staff and above roles – and a good portion is in engineering leadership/management (ranging from engineering managers to VP and C-level engineering leadership positions.)

I’m pleased to hear so many folks in product management following along – extrapolating 5% to the whole reader base would translate to closer to 37,500 such readers! I hope that topics on engineers’ mind like paying down tech debt or adopting engineering practices are helpful to understand even at the product level.

The “other” bucket still accounts for a lot of people – closer to 75,000 folks, looking at the number of readers! Commonly mentioned titles included data scientist/analyst, security engineer, and recruitment. More sparse mentions included marketing, sales – and retired folks!

Types of companies

The publication covers Big Tech, startups and scaleups. Naturally, I was interested to see where readers work:

About 65% of readers indeed work at these types of companies. At the same time, there’s a healthy representation of people working at:

More traditional, but tech-heavy companies

Consultancies and developer agencies

More traditional, non-tech heavy companies

Bootstrapped companies (4.2%)

Academia and research labs (1.8%)

Public sector (1.5%)

Nonprofits (1.1%)

Other (~3%): self employed, founding a startup and in-between jobs were the most common mentions

Interests

I asked people for feedback on the publication, and gathered these observations: 

Variation of topics makes the newsletter enjoyable. Many of you shared that you like it when topics are switched up. At the same time, the few times I did a multi-part series on a topic: I’ve received complaints from some of you how you’re ready to learn about other areas.

The “practical research” pieces are a definite hit. Lots of professionals shared how they appreciate getting what feels like a closer to real-time feedback on what is happening in the market, and the industry.

Disruptive technologies and companies: a common request. Many of you are asking to cover more about disruptions as we see them happen. Timely pieces like the ones on stacked diffs, modern developer productivity measurements, or companies like OpenAI are consistently popular, validating this ask.

4. Notable events

A few other things this happened in year three, that are worth a mention:

The Software Engineer’s Guidebook released

I started to write The Software Engineer’s Guidebook before starting the newsletter. In what is amusing irony: writing the newsletter delayed finishing the book – there’s only so much writing one can do, after all! Still, After four years of work, the book was finally released in print in November 2023, as an e-book in February 2024. The book became a #1 best seller across tech books on Amazon on launch. In the first 10 months, the book has seen more than 30,000 sales. Thank you to everyone who picked up a copy!

By the end of this year, I am aiming to release an audiobook version. Translations to 8 different languages are in progress (German, Japanese, Korean, Russian, Traditional Chinese, Simplified Chinese, Hungarian, Mongolian). The translations should be out between this fall and the spring of 2025 in respective countries.

I self published the book – as tech book publishers either passed on the title, or requested too many changes I did not agree with – and the self publishing process has been full of learnings. I’ll share more about this process, as self publishing, audiobook producing, and selling foreign rights has been full of learnings.

Leap Day

This year, February had 29 days. This small change caused a series of hiccups within software systems: from airlines to payment terminals. We shared a roundup of these. The next leap day will happen in 2028 – hopefully we’ll see more robust software running these critical systems by then.

The largest software-inflicted outage

In July of this year, we collectively witnessed the largest-ever software inflicted global outage where a routine Crowdstrike configuration update took airlines, banks and retailers offline across the globe.

We can now conclude that Crowdstrike did not do canarying or a staged rollout of this change, because they likely assumed that this change carries no risk. It’s always cheaper to learn from others’ mistakes: and this incident is a reminder that you should avoid doing “YOLO releases” when operating a critical line of business – even if that release is assumed to be safe to roll out.

5. Looking ahead

Some of the best-received articles come from guest writers: software engineers with deep experience in a topic. They then share their stories, learnings, mental models in a way that is helpful for more of us. Guest posts are always paid, and I closely work with guest authors to create a polished and enjoyable final piece. I’m always on the lookout for new guest writers:

Express interest in writing a guest post

One last thing… The Pragmatic Engineer Podcast. In the first two years of The Pragmatic Engineer, I ran several interviews with engineering leaders. A few examples:

Platform teams and the platform team mindset with Ganesh Srinivasan

Platform teams and developer productivity with Adam Rogal, Director of Developer Platform at DoorDash

The full circle on developer productivity with Steve Yegge, formerly Head of Engineering at Sourcegraph 

The last 12 months, there have been no similar interviews – even though they are interesting and useful. At the same time, I kept talking with interesting people in the software industry, and frequently wished I could share these conversations wider. This is how the idea of the podcast was born.

The Pragmatic Engineer Podcast is an addition to the newsletter. Every second week, on Wednesday, a new episode will be out. 

This show covers software engineering and Big Tech and startups, from the inside. I do deepdives with experienced engineers and tech professionals who share their hard-earned lessons, interesting stories and advice they have on building software. 

After each episode, you’ll walk away with pragmatic approaches you can use to build stuff – whether you are a software engineer, or a manager of engineers. Expect a similarly polished experience as what the newsletter already delivers. Please subscribe using your favorite podcast player, or sign up the podcast’s YouTube channel or on Spotify or on Apple Podcasts. The episodes will also be shared in the newsletter, under the “Podcast” category.

I hope you’ll enjoy this addition!

For the next year, expect more deeply researched and practical deepdives both for technologies and interesting tech companies. If you’d like to suggest a topic or a company to research, you can do it here:

Suggest a topic

Thank you – and see you in the next issue!


The Pragmatic Engineer Podcast: Trailer

Listen now (1 min) | A podcast covering software engineering at Big Tech and startups, from the inside

Welcome to The Pragmatic Engineer Podcast, hosted by Gergely Orosz, the author of The Pragmatic Engineer newsletter. In each episode, we dive deep into the world of software engineering, offering practical insights on scaling teams, engineering leadership, and navigating the evolving tech landscape. With industry veterans and successful engineers as guests, this podcast is perfect for anyone looking to level up their engineering career with real-world advice.

Subscribe to the podcast on YouTube, on Spotify, or Apple.

You can also subscribe to the newsletter here.

Monday, 16. September 2024

Heather Vescent

Ask A [cybersecurity] Futurist

Welcome to the inaugural “Ask a [cybersecurity] futurist column. This month’s question comes from Anna Borg. She asks: How can we change the “rules in the game” and make it less interesting to use the internet for attacks on organisations and societies including democracy. We will never “win the game” if we are forced to play defence play 24/7, 365 days per year. How do we change the incent

Welcome to the inaugural “Ask a [cybersecurity] futurist column. This month’s question comes from Anna Borg.

She asks:
How can we change the “rules in the game” and make it less interesting to use the internet for attacks on organisations and societies including democracy. We will never “win the game” if we are forced to play defence play 24/7, 365 days per year. How do we change the incentives for attackers?

I won’t sugar coat it, this is a complex and interconnected problem. Our increased cyber attacks are a symptom of a much larger problem. (Which means the solutions will not just address the cybersecurity problem at the core of this question, but bigger problems that society deals with.)

The Complexity of Cybersecurity Threats

For defense teams, the current state of cybersecurity feels like an endless battle, with organizations and societies constantly reacting to the latest threats. For hackers, it’s a landscape with abundant vulnerabilities if one is willing to put in the work to find (or create) the magic key — like a recent Advanced Persistent Teenager.

Cybersecurity is a long imbalanced game of whack-a-mole. Defenders have to guard everything, while attackers have the luxury of focusing. So how can you move out of a reactive mode to stay one step ahead of the next attack? And further, how can we make it even pointless for attackers to target us in the first place?

Understanding Hacker Motivation

To change the rules, we need to understand why cyber attacks are happening. What drives someone to become a black hat hacker? Or to start a black hat collective? Or join an elite nation state hacking team? I’ve spoken with security professionals, hackers and analyzed research data, and the motivations come down to: money, espionage, and the thrill of the con.

Viktor and Giselle from Better Call Saul

In a survey I conducted last year with identity-security experts, financial gain and espionage were the top motivations, followed by a desire for notoriety or the thrill of hacking. Most hackers today aren’t lone wolves; they’re part of organized collectives that pool their skills and resources to hit bigger, complex targets. They might even employ the Silicon Valley approved “AAS/as a service” business model.

There’s another factor that’s often overlooked: socioeconomic conditions. During my research for the paper Shifting Paradigms: Using Strategic Foresight to Plan for Security Evolution, I was genuinely surprised to hear about the overproduction of well-educated young people unable to find good jobs after their security education. There are very few well paid entry level security jobs — even in the US and developed nations.

Changing the Incentives

So how do we change the incentives to make cyber-attacks less appealing and less profitable for would-be attackers?

I’m going to skip over the obvious answer of creating/using better security technology. Everyone is racing to implement better tech solutions, but this is just a band aid — admittedly a really big band aid. I’m going to talk about non-tech solutions.

Economic Solutions

If it’s true that we are overproducing highly educated security people, could we solve some security problems by having full employment for all educated security experts?

One scenario in Shifting Paradigms envisioned this idea.

5.10 Scenario 9: Money, Cash, Hoes
In this scenario, whose name is inspired by the Jay Z song, security defensive work has become so attractive and well-paid that black-hat work is no longer attractive, and people with security-relevant technical skills are almost universally employed in defensive jobs. Governments can still hire offensive hackers, but criminal syndicates have a very hard time competing with white-hat employers.
Changes from Today: Employers pay great salaries, offer significant flexibility and benefits, and recruit extremely aggressively, especially in poor countries with good technical talent and in ungoverned spaces. There are many good entry level security jobs.

Could one of the most effective ways to reduce cybercrime to increase legitimate employment opportunities for those with technical skills? If well-paying jobs in cybersecurity or other tech fields were more readily available, especially in economically disadvantaged regions, would the incentive to engage in illegal activities diminish?

I think this scenario is illustrated in Wired’s recent article about Priscila Barbosa, Queen of the Rideshare Mafia, which describes the elaborate identity theft and fraud Priscila engaged in. Barbosa took IT classes back in her native Brazil and was a successful entrepreneur until the recession caused business to disappear. She came to find her fortune in the US — which she did. But because she overstayed her visa, she could not work legitimate jobs. I’d like to imagine, that if there was a legitimate way to work in the US, she would have never participated in criminal activities. And maybe, if there had been good tech jobs in Brazil, she might not have even needed to seek her fortune in the US.

In my view, Barbosa is a victim of economic lack of opportunity.

Disrupting [Criminal] Business Models

What are the business models that make cybercrime profitable? I remember Kim Cameron, Identity Architect, saying (not sure if this is his quote or he was quoting someone), that “the most profitable business model on the internet is crime. Back in the 90s and again now.”

Even entrepreneurial criminals have embraced the successful “as a Service” software development model. But to me, this echoes back to the economic themes. If there were legitimate economic paths half as successful, would there be less interest in criminal activity?

Nation-State Attacks

Nation-state hackers are a whole different beast. Their motivations are tied to geopolitical loyalty and belief. To reduce these types of attacks, we need to focus on improving global cooperation and reducing international tensions. This might sound idealistic, but fostering peace and global coexistence could significantly decrease the incentive for nations to engage in cyber warfare.

Reducing the Thrill

Then there’s the issue of hacking for fun, cultural subversion, or “the thrill of the con.” This is harder to combat, but one approach is to create more avenues for ethical hacking while embracing the dark side of hackers. Bug bounty programs, for example, allow hackers to flex their skills in a legal, constructive way. These programs are subversive in their own right but channel that energy into something positive.

Finding Solutions

Is the solution simply a transparent and interdependent global economy with legitimate well paid jobs so we can all live our best lives without stepping on someone to do so?

I don’t know about you, but that feels like a big and impossible solution — perhaps a bit too utopian. So, I want to leave Anna with some reasonable actions she or her company might take.

Entry level security roles: Increase well paid jobs & support development programs to transition people into well paid jobs. As part of this, eliminate the rockstar mentality, and hire someone who is good enough for the job and then treat them well. Embrace the Lulz factor: Embrace the shadow of cybersecurity by engaging and encouraging the exact hacker mindset but to improve security — and again, offer an economic incentive. Bug bounties are both subversive and helpful.

Both of these ideas could have a corporate initiative with a budget, KPI and program manager. They could be funded out of HR or professional development, IT or risk management line items. Companies could partner with universities, hacker collectives and government agencies, maybe sponsoring a hackathon or participating in a local cyber wargame activity.

Got Questions

If you could see the future of cybersecurity, what would you ask? Submit your question for a chance to be featured in the next “Ask a Cybersecurity Futurist” column.

If you found this insightful please consider booking me for a keynote talk or a lunch and learn session. Or sign up for the Future of Cybersecurity newsletter.


IdM Laboratory

Google Walletと選択的情報開示

こんにちは、富士榮です。 先日、「選択的情報開示とウォレットと本人確認書類」というタイトルで投稿しました。 https://idmlab.eidentity.jp/2024/08/blog-post_28.html 内容としては、本人確認書類をデジタル化するならば選択的情報開示を含めデジタル化をすることによるメリットがちゃんと見えるようにならないといけないよね〜という話だったわけですが、昨日も触れたGoogle Walletの記事には今後のロードマップが明示されています。 How we're working to make digital identity a reality for everyone https://blog.google/products/google-pay/google-wallet-digital-id-privacy-securit
こんにちは、富士榮です。
先日、「選択的情報開示とウォレットと本人確認書類」というタイトルで投稿しました。 https://idmlab.eidentity.jp/2024/08/blog-post_28.html

内容としては、本人確認書類をデジタル化するならば選択的情報開示を含めデジタル化をすることによるメリットがちゃんと見えるようにならないといけないよね〜という話だったわけですが、昨日も触れたGoogle Walletの記事には今後のロードマップが明示されています。
How we're working to make digital identity a reality for everyone https://blog.google/products/google-pay/google-wallet-digital-id-privacy-security/


要するに安心してGoogle Walletなどのアイデンティティソリューションを利用するためにGoogleが取り組んでいることについて書かれたポストですが、その中に選択的情報開示に関して記載があります。
Selective disclosure and user control: With digital identity, the relying party (a party requesting information, for example a car rental company or a merchant) is able to request only the relevant parts of a person’s ID. Today, if you’re presenting your physical ID (for example to confirm your age or your address) you have to share everything — your name, address, your physical description and more. However, with a digital ID, you can share only the required data. Additionally, you must authenticate the device with a fingerprint, PIN or passcode in order for any of your ID information to be shared with the requester.

選択的開示とユーザー制御:デジタル ID では、依拠当事者(情報を要求する当事者、たとえばレンタカー会社や商業者)は、個人の ID の関連部分のみを要求することができる。現在、物理的なIDを提示する場合(たとえば年齢や住所を確認する場合)、氏名、住所、身体的特徴など、すべてを共有しなければならない。しかし、デジタルIDでは、必要なデータのみを共有することができる。さらに、あなたのID情報を要求者と共有するためには、指紋、PIN、パスコードでデバイスを認証する必要があります。 


選択的情報開示のUXがどうなるのか気になりますが、個人的な意見としてはリライングパーティが全体ではなく最低限の要求ができるようになるので、ウォレットとしてはそのリクエストに対応できるようにするよ、という話だけでは全然足りない気がしています。(実装する立場としては理解できますが)

あくまでユーザの意思によって開示する情報を選択できるという体験が重要だと思うので、リライングパーティがどんな属性を要求してきているかに関わらず、自身で開示する属性を選べる状態にはなっていないといけないと思います。(結果的にリライングパーティの要求を満たさなかったとしても)

また、オフラインでの利用についても考慮をしていってもらえるといいなぁ、、と思います。たとえば、画面を見せる際に検証者の目線では「これは正式な書類である」ということが視認できる状態が重要なので、表面は正式な証明書であることが視認できるだけ、タップして裏面を見せるとユーザがあらかじめ設定した開示したい最低限の情報だけが記載されている、という状態が作れるといいのではないかと思います。


いずれにしても3rdパーティウォレットを含むエコシステムが正常に出来上がるような規制などは政府が中心に整備してもらえるといいですね。ユーザーの声を正しく吸い上げるためにもAppleとGoogleだけに任せるのではなく、エコシステム全体として進化できていくことが重要な気がします。


Michael Ruminer

RAG Disappointment and Azure Open AI Success With RAG

Recently, I have been working on programming small modules for Retrieval Augmented Generation (RAG) using Open AI. Also recently, I did a Coursera class on advanced RAG and have taken in several videos and posts on it. I have used samples from those sources and tested using various LLMs to generate simple Python to perform RAG with Open AI. In general , I have been disappointed with the outco

Recently, I have been working on programming small modules for Retrieval Augmented Generation (RAG) using Open AI. Also recently, I did a Coursera class on advanced RAG and have taken in several videos and posts on it. I have used samples from those sources and tested using various LLMs to generate simple Python to perform RAG with Open AI. In general , I have been disappointed with the outcomes until I tried Azure Open AI with Azure AI semantic search.

Disappointment

My general disappointments have come from the self-coded RAG attempts on a single PDF. The basic approach has been:

take a single PDF (not a small PDF — about 43,000 token when uploaded to Open AI API in whole and extract the text using PyPDF chunk the text using at best recursive text and/or sentence transformer; sometimes just naive simple character count split embed the chunks trying Chroma or text-embedding-3-small query the collection using Chroma or FAISS and in one instance also with a simple augmented prompt call the LLM with the initial prompt and context from the embeddings query the prompt was for a list of 10 principles in the document that were all outlined in a single paragraph

It’s nothing sophisticated by any stretch but they corresponded with the examples I had available.

The results — abysmal. I’m not really surprised. I’m not sure how such an approach could do very well with simple chunking, embeddings, and basically running off keyword proximity. But since these were supposed examples of the use of RAG I expected better results. There were only two times I received good results. One of those times I’ll outline below; the other was when I didn’t parse or embed and just passed the entire document as context. Of course, the latter worked well but that was not the exercise I was after.

Success

My successful attempt came when I didn’t create the code directly but used Microsoft Azure Open AI playground. I have no doubt that coding it up would have worked just as well since it relies on the Azure AI infrastructure and the code would be little more than passing a prompt to the Azure LLM instance and getting the results. Here is what it consisted of:

setup of an Azure Open AI instance using gpt-4o-mini model and, I think, the text-embedding-3-small embedding model setup an Azure AI Search instance with semantic search and indexing pointed to a blob storage container with a single PDF in it

I think that was it. I then went to the Azure Open AI Studio playground grounded the model with the search instance, provided my simple prompt, and got back the desired results. Was it the semantic search that made it work well? I suspect it helped a lot. I need to try it without the semantic search and see what happens. Sorry, I forgot to try that scenario.

Recap

All in all, I was very disappointed with the RAG results, as I coded it, especially as they were based on examples or AI generated single document RAG code. But, I was very pleased with the Azure test and I think the semantic search made all the difference.


Doc Searls Weblog

Remembering Iris Harrelson

In the late ’70s, I worked for a while at the Psychical Research Foundation, which occupied a couple of houses on Duke University property and did scientific research into the possibility of life after death. My time there was a lever that has lifted my life on Earth ever since, including many deep and enduring […]

In the late ’70s, I worked for a while at the Psychical Research Foundation, which occupied a couple of houses on Duke University property and did scientific research into the possibility of life after death. My time there was a lever that has lifted my life on Earth ever since, including many deep and enduring friendships.

Bill Roll ran the place then, and many fascinating characters passed through the PRF orbit. Eda and Lawrence Lashan, for example. Mac McDonnell. John Fetzer. One of the most memorable was Iris, a brilliant woman from Savannah with a strong personality, dyed red hair, and lots of talents. Her surname was pronounced “Mock,” but spelled with an “a” or two. (I think it was Maack.) She also mentioned occasionally that her brother was Ken “Hawk” Harrelson, the baseball player I remember best from his peak years with the Boston Red Sox. For many years since then, however, he has been the play-by-play announcer for the Chicago White Sox. Now 83, Hawk is still calling games for what will almost certainly be the losing-ist team in major league history.

So this morning, after I read “Hawk Harrelson spent 3 decades calling the White Sox. Now he can’t stand to watch” in the NY Times (sorry, paywall), I looked up “Hawk Harrelson” plus “sister” and “Iris” and landed on this 2018 page, which has this passage from Hawk’s autobiography:

“My sister was a heavy smoker who had died of lung cancer when she was only 40. I was so proud of what Iris had done with her life after such a tough start, having to get married at the age of 14. She wrote a few books and became a gourmet cook. She also learned to speak fluent German, Russian, and Spanish. She lectured at Duke University and at the University of Toronto. She became an accomplished pianist. She also dabbled in acting and landed a few holes on stage in New York. And she was an interior designer, having turned my penthouse pad into a beautiful home. “But we had a falling out several years before she died when I turned down her request to borrow $250,000. She had wanted to open a nightclub in downtown Savannah and I didn’t have that much cash at the time. Plus, I didn’t think her business idea was a good one. When I refused her request, she walked out the door and I never saw her again. “Apparently, she never quit smoking.”
While I’m not surprised to learn that Iris is gone (she was older than me, and I’m seventy-seven), it’s a shock to hear that she died so young, so long ago, and so full of talent and promise. I’m also not surprised that there is almost zero information about her on the Web, given that it was born after she died, and is a whiteboard as well. So maybe this post (titled with her birth name) will at least help make her more remarkable than I’ve made her so far.

Damien Bod

Using Elasticsearch with .NET Aspire

This post shows how to use Elasticsearch in .NET Aspire. Elasticsearch is setup to use HTTPS with the dotnet developer certificates and and simple client can be implemented to query the data. Code: https://github.com/damienbod/keycloak-backchannel Setup Two services are setup to run in .NET Aspire. The first service is the official Elasticsearch docker container and deployed […]

This post shows how to use Elasticsearch in .NET Aspire. Elasticsearch is setup to use HTTPS with the dotnet developer certificates and and simple client can be implemented to query the data.

Code: https://github.com/damienbod/keycloak-backchannel

Setup

Two services are setup to run in .NET Aspire. The first service is the official Elasticsearch docker container and deployed using dotnet developer certificates. The second service is an ASP.NET Core application using the Elastic.Clients.Elasticsearch Nuget package. The App.Host project is used to set this up and to link the services together.

Elasticsearch development server

The Elasticsearch container is configured in the program class of the App.Host project. The container is run using HTTPS and takes the Aspire parameters for configuration of the default account.

var elasticsearch = builder.AddElasticsearch("elasticsearch", password: passwordElastic) .WithDataVolume() .RunElasticWithHttpsDevCertificate(port: 9200);

The developer certificates needs to be created and copied to the specific folder inside the Elasticsearch docker container. This is implemented using a shared folder and the Elasticsearch xpack.security.http.ssl properties are set to match. The following three properties are used:

xpack.security.http.ssl.enabled xpack.security.http.ssl.certificate xpack.security.http.ssl.key using System.Diagnostics; using System.IO.Hashing; using System.Text; namespace Aspire.Hosting; // original src: https://github.com/dotnet/aspire-samples/tree/damianedwards/keycloak-sample/samples/Keycloak public static class HostingElasticExtensions { public static IResourceBuilder<ElasticsearchResource> RunElasticWithHttpsDevCertificate(this IResourceBuilder<ElasticsearchResource> builder, int port = 9200, int targetPort = 9200) { if (builder.ApplicationBuilder.ExecutionContext.IsRunMode) { builder .RunElasticWithHttpsDevCertificate() .WithHttpsEndpoint(port: port, targetPort: targetPort) .WithEnvironment("QUARKUS_HTTP_HTTP2", "false"); } return builder; } public static IResourceBuilder<TResource> RunElasticWithHttpsDevCertificate<TResource>(this IResourceBuilder<TResource> builder) where TResource : IResourceWithEnvironment { const string DEV_CERT_DIR = "/usr/share/elasticsearch/config/certificates"; if (builder.ApplicationBuilder.ExecutionContext.IsRunMode) { // Export the ASP.NET Core HTTPS development certificate & private key to PEM files, bind mount them into the container // and configure it to use them via the specified environment variables. var (certPath, _) = ExportElasticDevCertificate(builder.ApplicationBuilder); var bindSource = Path.GetDirectoryName(certPath) ?? throw new UnreachableException(); if (builder.Resource is ContainerResource containerResource) { builder.ApplicationBuilder.CreateResourceBuilder(containerResource) .WithBindMount(bindSource, DEV_CERT_DIR, isReadOnly: false); } builder .WithEnvironment("xpack.security.http.ssl.enabled", "true") .WithEnvironment("xpack.security.http.ssl.certificate", $"{DEV_CERT_DIR}/dev-cert.pem") .WithEnvironment("xpack.security.http.ssl.key", $"{DEV_CERT_DIR}/dev-cert.key"); } return builder; } private static (string, string) ExportElasticDevCertificate(IDistributedApplicationBuilder builder) { var appNameHashBytes = XxHash64.Hash(Encoding.Unicode.GetBytes(builder.Environment.ApplicationName).AsSpan()); var appNameHash = BitConverter.ToString(appNameHashBytes).Replace("-", "").ToLowerInvariant(); var tempDir = Path.Combine(Path.GetTempPath(), $"aspire.{appNameHash}"); var certExportPath = Path.Combine(tempDir, "dev-cert.pem"); var certKeyExportPath = Path.Combine(tempDir, "dev-cert.key"); if (File.Exists(certExportPath) && File.Exists(certKeyExportPath)) { // Certificate already exported, return the path. return (certExportPath, certKeyExportPath); } else if (Directory.Exists(tempDir)) { Directory.Delete(tempDir, recursive: true); } Directory.CreateDirectory(tempDir); var exportProcess = Process.Start("dotnet", $"dev-certs https --export-path \"{certExportPath}\" --format Pem --no-password"); var exited = exportProcess.WaitForExit(TimeSpan.FromSeconds(5)); if (exited && File.Exists(certExportPath) && File.Exists(certKeyExportPath)) { return (certExportPath, certKeyExportPath); } else if (exportProcess.HasExited && exportProcess.ExitCode != 0) { throw new InvalidOperationException($"HTTPS dev certificate export failed with exit code {exportProcess.ExitCode}"); } else if (!exportProcess.HasExited) { exportProcess.Kill(true); throw new InvalidOperationException("HTTPS dev certificate export timed out"); } throw new InvalidOperationException("HTTPS dev certificate export failed for an unknown reason"); } }

When the App.Host project is started, the Elasticsearch containers boot up and the server can be tested using the “_cat” HTTP Get requests or the default base URL will give a server information about Elasticsearch.

https://localhost:9200/_cat

Elasticsearch client

The Elasticsearch client was implemented using the Elastic.Clients.Elasticsearch Nuget package. The client project in .NET Aspire needs to reference the Elasticsearch server using the WithReference method.

builder.AddProject<Projects.ElasticsearchAuditTrail>( "elasticsearchaudittrail") .WithExternalHttpEndpoints() .WithReference(elasticsearch);

Elasticsearch can be queried used a simple query search.

public async Task<IEnumerable<T>> QueryAuditLogs(string filter = "*", AuditTrailPaging auditTrailPaging = null) { var from = 0; var size = 10; EnsureElasticClient(_indexName, _options.Value); await EnsureAlias(); if (auditTrailPaging != null) { from = auditTrailPaging.Skip; size = auditTrailPaging.Size; if (size > 1000) { // max limit 1000 items size = 1000; } } var searchRequest = new SearchRequest<T>(Indices.Parse(_alias)) { Size = size, From = from, Query = new SimpleQueryStringQuery { Query = filter }, Sort = BuildSort() }; var searchResponse = await _elasticsearchClient .SearchAsync<T>(searchRequest); return searchResponse.Documents; }

See the source code: https://github.com/damienbod/keycloak-backchannel/blob/main/AuditTrail/AuditTrailProvider.cs

Notes

With this setup, it is easy to develop using Elasticsearch as a container and no service needs to be implemented on the developer host PC. Setting up HTTPS is a little bit complicated and it would be nice to see this supported better. The development environment should be as close as possible to the deployed versions. HTTPS should be used in development.

Links

https://learn.microsoft.com/en-us/dotnet/aspire/search/elasticsearch-integration

https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html

https://www.elastic.co/products/elasticsearch

https://github.com/elastic/elasticsearch-net

https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-simple-query-string-query.html


IdM Laboratory

Google Walletへ搭載できる証明書

こんにちは、富士榮です。 Gogole Walletへの米国パスポートの搭載が先日発表されましたね。 崎村さんがGoogleのアナウンスについてポストされているのでこちらを見ると良いと思います。 https://www.sakimura.org/2024/09/6359/ 簡単にいうと、米国のパスポートをGoogle Walletへ格納することができるいう話で、現在はまだ紙のパスポートと併用、かつTSAチェックでしか使えないが、将来的にはもっと使える場所の拡大をしていこう、としているという話です。 日本でも早く使えるようになるといいですねぇ。 まだ、日本では決済以外だとイベントのチケットや航空券やポイントカードなどが搭載できるくらいですし。 日本で追加できるのは 支払いカード ポイントカード ギフトカード 写真 の4カテゴリ。
こんにちは、富士榮です。
Gogole Walletへの米国パスポートの搭載が先日発表されましたね。 崎村さんがGoogleのアナウンスについてポストされているのでこちらを見ると良いと思います。 https://www.sakimura.org/2024/09/6359/
簡単にいうと、米国のパスポートをGoogle Walletへ格納することができるいう話で、現在はまだ紙のパスポートと併用、かつTSAチェックでしか使えないが、将来的にはもっと使える場所の拡大をしていこう、としているという話です。
日本でも早く使えるようになるといいですねぇ。 まだ、日本では決済以外だとイベントのチケットや航空券やポイントカードなどが搭載できるくらいですし。
日本で追加できるのは 支払いカード ポイントカード ギフトカード 写真 の4カテゴリ。

ポイントカードは色々と使えるものが増えていますね。


ところで、米国を含む海外ではGoogle Walletは何に使えるのかみていきましょう。
Google Walletのヘルプページを見ると色々なものが搭載できるようになっているようです。 https://support.google.com/wallet/answer/12059409?hl=ja&ref_topic=11925097&sjid=8720886013754835920-AP
日本語ページなのでちょっと直訳感がありますが、右側のナビゲーションを見ると、関連トピックスとしてこちらが記載されています。 お支払い方法 搭乗券やイベント チケットを保存して使用する ポイントカードとギフトカード Google ウォレットを公共交通機関で使用する Google ウォレットにヘルスパスを追加する 自動車用デジタルキー Google ウォレットに学生 ID を保存する 米国の運転免許証または州発行の身分証明書を追加する Google ウォレットに社員バッジを保存する Use transit loyalty cards in Google Wallet (UK only) スマートウォッチの Google ウォレットでパスを使用する Google ウォレットのパスの分類 Google ウォレットのリンクされたパスについて Google ウォレットのウェブサイトの利用を開始する ホテルキー
ヘルスパスはワクチン接種証明で日本もやっていましたね。 https://idmlab.eidentity.jp/2021/12/verifiable-credentials.html
その他ポイントカードなど日本も使えるもの以外を見ていくと、 自動車用デジタルキー 学生ID 米国の運転免許証または週発行の身分証明書(今回のパスポートの件はこちらですね) 社員バッジ Transit loyalty card ホテルキー などが面白そうです。 自動車の鍵だと古くからCCC(Car Connectivity Consortium)が取り組んできた活動はありますが、電子運転免許証(mDL)と連携していく動きは活発化していきそうですね。
学生IDも面白いトピックです。 マサチューセッツ工科大学(MIT)が2021年にアナウンスしたデジタル学生証や、卒業証明書のデジタル化の動きはエポックメイクングでしたが、Googleのヘルプを見ると「米国、カナダ、オーストラリアの加盟大学」がこの機能を使えるようになっているようです。
社員バッジも学生IDと同様に普及していくと面白いですね。

Credential APIも本格化してきましたし、引き続きこの分野は要ウォッチです。





Saturday, 14. September 2024

Mike Jones: self-issued

OAuth 2.0 Protected Resource Metadata draft addressing reviews since IETF Last Call

Aaron Parecki and I published a new version the “OAuth 2.0 Protected Resource Metadata” specification that addresses the review comments received since the IETF Last Call. Per the history entries, the changes were: Added metadata values declaring support for DPoP and mutual-TLS client certificate-bound access tokens. Added missing word caught during IANA review. Addressed ART, […]

Aaron Parecki and I published a new version the “OAuth 2.0 Protected Resource Metadata” specification that addresses the review comments received since the IETF Last Call. Per the history entries, the changes were:

Added metadata values declaring support for DPoP and mutual-TLS client certificate-bound access tokens. Added missing word caught during IANA review. Addressed ART, SecDir, and OpsDir review comments by Arnt Gulbrandsen, David Mandelberg, and Bo Wu, resulting in the following changes: Added step numbers to sequence diagram. Defined meaning of omitting bearer_methods_supported metadata parameter. Added internationalization of human-readable metadata values using the mechanism from [RFC7591]. Added resource_name metadata parameter, paralleling client_name in [RFC7591]. Added Security Considerations section on metadata caching. Used and referenced Resource Identifier definition. Added motivating example of an email client to intro.

The specification is available at:

https://www.ietf.org/archive/id/draft-ietf-oauth-resource-metadata-09.html

Friday, 13. September 2024

Georg C. F. Greve

Coming full circle: How Self-Sovereign Identity Could Lose Its Way

“A picture showing the knight of Self Sovereign Identity losing his way, going into the valley of doom where certain death awaits.” Self-Sovereign Identity (SSI) offers a robust and scale-able solution to the challenges of digital identity that is secure while also respecting privacy, freedom, and human rights. However, a mix of political and industry pressures, along with a push for quick ado
“A picture showing the knight of Self Sovereign Identity losing his way, going into the valley of doom where certain death awaits.”

Self-Sovereign Identity (SSI) offers a robust and scale-able solution to the challenges of digital identity that is secure while also respecting privacy, freedom, and human rights. However, a mix of political and industry pressures, along with a push for quick adoption, risks turning SSI into just another marketing buzzword.

Nowhere is this trend more visible than in the mobile Drivers License (mDL) and the ongoing conversations around the second version of European Unions “electronic IDentification, Authentication and trust Services” (eIDAS).

Old Interests, New Technology

The first version of eIDAS was an adoption failure. While it was not providing privacy by design, technologically it was sufficiently correct to function. It did so by giving a central role to Trust Service Providers (TSPs) and Certificate Authorities (CAs).

These intermediaries sought to exploit their government issued monopoly and consequently choked off adoption for the first version of eIDAS.

In doing so, they repeated the same mistake made on the wider web, where excessive pricing led to stagnation in the adoption of encrypted communications. In 2014, only 10 short years ago, the vast majority of all connections was not using any kind of transport layer security (TLS).

HTTPS adoption became mainstream only between 2015 and 2019 — via Matthew Greene, article linked below
It was the Snowden Revelations that changed everything. But not in the way the intermediaries had hoped.

While I am sure many were rubbing their hands, and expecting the pipe they were sitting on to finally start flowing, everyone in the industry knew that predatory pricing was to blame for the low adoption. So in November 2014, Let’s Encrypt was founded in order to provide for free what Certificate Authorities are selling at premium prices.

Today, Let’s Encrypt provides over 55% of all certificates used on the web. Which provides for two conclusions:

Let’s Encrypt has become the single point of failure for security on the internet. And the addressable market for TSPs has shrunk dramatically, and keeps shrinking.

Simultaneously, the costs for TSPs are growing. Not only in terms of technology cost. But also in terms of compliance cost. Because there is a whole industry of consultants and auditors that service TSPs, extracting six figure payments each year for initial certification and re-certification. Growing cost and shrinking markets are not a great prospect for any industry.

So when the new version of eIDAS came along, promising to switch to Self Sovereign Identity for privacy by design, security, and elimination of intermediaries, the TSPs deployed their lobbyists to Brussels to stop this threat to their remaining business.

The result was a bureaucratic monster, for which some people at DICE 2024 in Zurich voiced doubt whether it could even be implemented in practice.

Intermediaries are once again involved in every step. And all the technological choices are based on legacy Web2.0 technology shaped by the TSPs, without privacy by design, and with well known and regularly exploited security issues.

So TSP lobbying successfully defended their government mandated monopoly over the digital domain, and the fundamental issue that stopped adoption of the first version of eIDAS remains unresolved. Only the future will show whether this second attempt of the TSPs at monetizing all digital interactions will fare any better.

But that isn’t even the biggest problem.

The Web is a bad idea for personal identity
The web is a fundamentally unsound technology choice for personal identity, for a variety of reasons, starting with security.

The early web wasn’t really designed for privacy or security. Its original design was inspired by the need for institutional information sharing. Invented in 1989, security was added in 1995 when Netscape came up with Secure Sockets Layer (SSL), which led the way for today’s Transport Layer Security (TLS). This allowed encryption of communication, but all security was dependent on key management.

Because it was the only practical way at the time, intermediaries became the root of all trust for virtually all users. On top of those issues, secure communication requires two parties who may know nothing about one another to negotiate a secure key to use for encryption with one another. That negotiation must happen over the same channel that is being used for encrypted communication.

Which is a bit like yelling the combination of the key pad to your house through the open window over to your neighbour so he can come in and water the plants. If there are no eavesdroppers at that time: Great. If there are, they now also have the key. You’ll find out whether you were safe in a few weeks when you get back from vacation.

Diffie-Hellman key exchange was invented to solve that particular problem. But it is only secure if used correctly and keys are being rotated and refreshed often. Re-use of static keys or weak parameters can weaken security considerably.

On top of that, cryptography is an ever evolving field, and the need for web servers to support older browsers means that sometimes, outdated or known vulnerable settings are still being allowed. Managing all that is complex. It requires organisational level security teams.

And much like charging your modern Electric Vehicle using a coal powered steam engine to drive a generator, trying to fast track adoption for Self Sovereign Identity by using Web based Decentralized Identifier (DID) methods effectively negates all the conceptual and architectural progress that has been made with SSI.

All the underlying security issues our industry has struggled with for the past 30+ years, all the conceptual weaknesses of the Web, all the privacy issues and compromises also apply to SSI when using did:web and its descendants.

In practical terms this translates into most people once again finding themselves forced to entrust their personal identity to intermediaries in the form of custodial wallets and platforms. Just as they are doing today with Google, Facebook, Amazon, Apple, Microsoft.

Most people will rent usage of their own identities, but they will have no ownership or agency. And those intermediaries will once again have full visibility and control over everything people are doing online.

That is a steep price to pay. Is it worth it at least in terms of security?

Even these large organisations get it wrong often enough for data leaks and other issues to occur frequently. When using the Web for personal identity, these compromises will have consequences beyond anything we have seen so far. What happens when the compromise has enabled the attacker to provide legally valid signatures in your name to sign contracts, or consent to sharing your personal pictures with the wrong parties?

And that’s only the start of the valley of doom.

The Web is based on top of the Domain Name System (DNS). Which is a system in which you rent usage of certain human readable domains on a yearly basis. Without it, the Web falls apart.

So let’s assume enlightened users rent their own domains to become independent from the intermediaries. Now they can move between them in case one of them has issues, goes out of business, or gets acquired by a billionaire with a questionable political agenda. Just move your domain and you’re good, right?

Not so fast.

Not only is it possible you may lose your domain, either due to changes in DNS policy, legal aspects, such as trademarks, or because you’ve missed a yearly payment.

If a personal identity can be lost that easily, was it ever really yours?

But there is an even bigger issue.

The W3C Decentralized Identifier (DID) standard was designed to provide for unique, non-correlatable identifiers for each individual interaction. Which sounds great in theory. But how non-correlatable are did:tdw:johnsmith.eu:223499182 and did:tdw:johnsmith.eu:673403151?

Would you be able to tell they might both belong to the same person, whose name might be John Smith?

Because the domain is always an intrinsic part of each Web based DID method, none of them allow for personal ownership without complete loss of pseudonymity, and non-correlatability.

So whatever pseudonymity and privacy can be achieved leads back full circle to large intermediaries and custodial wallets — just like the federated identity infrastructure of today.

“Anakin Skywalker at a crossroads. Go down the Web based SSI method path, and become Darth Vader, or choose a world of privacy, freedom and security?” TL;DR: Don’t do it, Anakin!
Web technologies are a fundamentally poor choice for personal identity.

They can work in domains where privacy and non-correlatability are not required, for instance organisational identity. But for personal identity, they compromise almost everything that Self Sovereign Identity (SSI) was intended and designed for.

You cannot retrofit privacy and security onto an architecture that wasn’t built for them. Just look at Email. It has the same problem. Decades of adding layers of complexity and auxiliary systems have kept moving the problem somewhere else in the stack, to another component. Lots of snake oil has been sold. Some people got very rich this way. But the fundamental issues remain unsolved.

We should not repeat that same mistake for Self Sovereign Identity.

No system that is intended for personal identity should ever use did:web, did:tdw, or any of the other web based DID specifications. The concepts of SSI can solve all the issues above. They can provide people with real agency and control over their own identity. But not when using the Web.

The solution is simple.

Stop using Web based DID methods for personal identity!
Humans are not organizations. Stop using Web based DID methods for personal identity. did:web and its descendants are very useful for some organizational use cases, and for prototyping new ideas. They have NO place when it comes to personal identity, or use cases that require strong privacy or security.

There are (at least) three superior alternatives that come to mind, all of which can also provide a secondary channel for key negotiation.

Stay tuned.


@_Nat Zone

グーグルがパスポートのウォレット搭載を発表〜Google Walletで実現する新時代のデジタルID

米国時間2024年9月12日(日本時間9月13日)、グーグルが米国パスポートのグーグルウォレットへの搭載を発表しました。 近年、日常生活のあらゆる面でデジタル化が進んでいますが、その中でも特に注目を集めているのがデジタルIDです。Googleは昨年から一部の州発行のデジタルIDを…

米国時間2024年9月12日(日本時間9月13日)、グーグルが米国パスポートのグーグルウォレットへの搭載を発表しました。

近年、日常生活のあらゆる面でデジタル化が進んでいますが、その中でも特に注目を集めているのがデジタルID1です。Googleは昨年から一部の州発行のデジタルIDをGoogle Walletに保存できる機能の展開を開始しましたが、さらなる進化を遂げようとしています。

まもなく、Google Walletに新しいタイプのデジタルIDを導入するベータテストが始まります。この新機能により、より多くの人々が、より多くの場所でデジタルIDを作成し保存できるようになります。注目すべきは、今回から米国パスポートを使用してデジタルIDを作成できるようになることです。

このデジタルIDパスは、一部のTSA(運輸保安局)チェックポイントで使用可能となり、国内旅行時の空港での時間と手間を省くことができます。

デジタルIDパスの作成プロセスは非常に簡単です:

Google Walletアプリで「米国パスポートでIDパスを作成」というプロンプトを選択 パスポートの裏面にあるセキュリティチップをスキャンする指示に従う 本人確認のためにセルフィービデオを撮影 数分以内にIDパスの準備が整ったという通知を受け取る https://storage.googleapis.com/gweb-uniblog-publish-prod/original_videos/PassportinGoogleWallet_Revised_R3.mp4 Google Wallte へのパスポート登録の様子 (出所)Google

現在、このIDパスは一部のTSAチェックポイントでのみ受け付けられていますが、Googleはパートナー企業と協力して、将来的にはアカウント復旧、本人確認、さらにはレンタカーの利用など、より多くの場面でデジタルIDを使用できるよう取り組んでいます。

ただし、この技術はまだ初期段階にあるため、Google WalletのデジタルIDは現時点では物理的なIDの代替とはなりません。旅行時には引き続き物理的なIDの携帯が必要です。

セキュリティ面では、Google Walletは「プライベート・バイ・デザイン」原則によって個人情報を保護するように設計されたAndroidの高度なセキュリティを使用してデジタルIDを安全に保管します。IDパスは暗号化されて保存され、利用には指紋認証やPIN、パスコードでの認証が必要となります。また、本人確認時には共有される情報を事前に確認できるため、ユーザーが情報を管理できます。

さらに、カリフォルニア州のAndroidユーザーは既に州発行のID2や運転免許証をGoogle Walletアプリに保存できるようになっており、今後数か月以内にアイオワ州、ニューメキシコ州、オハイオ州でも同様のサービスが展開される予定です。

コロラド州では、MyColoradoアプリ内に新しいリーダーを導入し、州内の企業がデジタルIDを安全かつ簡単に受け入れられるようにする取り組みも始まっています。

このように、GoogleはデジタルIDの利便性と適用範囲を拡大し続けており、私たちの日常生活をよりスマートで効率的なものに変えようとしています。デジタルIDの未来に、今後も注目が集まりそうです。

(参考資料)

New Google Wallet features for travelers and commuters (2024-09-12) How we’re working to make digital identity a reality for everyone (2024-09-12)

Thursday, 12. September 2024

Bill Wendels Real Estate Cafe

Leverage $25K downpayment assistance to protect homebuyers & revitalize buyer agency movement

Three decades ago was invited to participate in a collaborative effort to develop President Bill Clinton’s National Homeownership Strategy. Those 100 recommendations raised homeownership from… The post Leverage $25K downpayment assistance to protect homebuyers & revitalize buyer agency movement first appeared on Real Estate Cafe.

Three decades ago was invited to participate in a collaborative effort to develop President Bill Clinton’s National Homeownership Strategy. Those 100 recommendations raised homeownership from…

The post Leverage $25K downpayment assistance to protect homebuyers & revitalize buyer agency movement first appeared on Real Estate Cafe.

The Pragmatic Engineer

The Pulse #106: Why does the EU tech sector lag behind the US?

Also: non-AI companies like Klarna want to look like AI ones, unusual details about Google, Amazon, and Microsoft levels, and more

The Pulse is a series covering insights, patterns, and trends within Big Tech and startups. Notice an interesting event or trend? Send me a message.

Today, we cover:

Industry pulse. Apple announces AI features you can’t use yet, Stripe is a hit with popular AI companies, Confluent buys an up-and-coming competitor before it grows into a threat, game studio…

Read more


@_Nat Zone

「マイナ免許証」25年3月導入へ 住所変更ワンストップで。更新時講習もオンラインに。

運転免許証とマイナンバーカードの一体化が、まもなく現実のものとなります。 この画期的な変更は、私たちの日常生活に大きな影響を与える可能性を秘めています。 警察庁が9月12日に発表した新しい方針によると、2025年3月から「マイナ免許証」が導入される予定です。 これは、現在の運転免…

運転免許証とマイナンバーカードの一体化が、まもなく現実のものとなります。 この画期的な変更は、私たちの日常生活に大きな影響を与える可能性を秘めています。

警察庁が9月12日に発表した新しい方針によると、2025年3月から「マイナ免許証」が導入される予定です。 これは、現在の運転免許証の機能をマイナンバーカードに統合するという画期的な取り組みです。

マイナ免許証制度の概要

マイナンバーカードは、 2016年に導入された個人番号制度の一環として発行されている身分証明書です。 このカードには、すでに様々な個人情報が ICチップに記録されていますが、 新たに運転免許証の情報も追加されることになります。

具体的には、免許保有者の写真、免許の有効期間、取得している免許の種類、 そして眼鏡着用などの条件といった情報がマイナンバーカードのICチップに記録されます。 これにより、1枚のカードで身分証明と運転資格の証明が可能になります。

マイナンバーカードと運転免許証を一体化手続きは、運転免許センターや免許更新を扱う警察署で受け付ける方向で調整が進められています。これは、運転免許証の暗証番号を覚えている人は、スマホでできるようにして欲しいところではあります。

また、今の法律では免許取消や停止処分の対象者について、所在が分からず処分ができていない場合に警察官が対象者を見つけた場合、免許証を預かる制度があります。2025年3月24日以降はこの制度がなくなり、「出頭命令書」を出す制度に変わります。

マイナ免許証のメリット

この一体化には多くのメリットがあります。 特に注目すべきは、住所変更手続きの簡素化です。 現在、引っ越しの際には自治体と警察の両方に別々に届け出る必要がありますが、 マイナ免許証では自治体での手続きだけで完結します。

さらに、免許更新時の利便性も向上します。 優良運転者と一般運転者は、マイナポータルという個人向けサイトで講習動画を視聴するだけで、 更新時の講習を受けたとみなされます。 これにより、更新手続きにかかる時間と手間が大幅に削減されます。 ただし、すべての手続きがオンラインで完結するわけではありません。 写真撮影や視力検査などは、従来通り免許センターなどで実施されます。 これは、安全運転に必要な基本的な身体能力を確認する重要な過程だからです。

マイナ免許証の導入は任意であり、既存の運転免許証も併用して使用することができます。 しかし、マイナ免許証のみを保有する場合、 新規取得時や更新時の手数料が比較的安くなるという経済的なメリットもあります。

免許取得免許更新現行の免許証2350円2850円マイナ免許証1550円2100円現行+マイナ2450円2950円2025年3月24日以降の手数料 懸念点の解消

この変更は、デジタル化が進む現代社会における行政サービスの効率化の一環と言えるでしょう。 すでに「マイナ保険証」として健康保険証の機能を統合する取り組みも進められており、 2024年7月末時点でマイナンバーカード保有者の80%が登録を完了しています。

誤まった紐付けについて

しかし、このようなデジタル化の急速な進展には懸念の声も上がっています。 特に、個人情報の管理や情報セキュリティに関する不安は根強く存在します。 政府は、これらの不安に対して丁寧な説明と十分な対策を講じる必要があるでしょう。例えば、以前、マイナンバーカードと健康保険証の連携でミスが発生しました。マイナ免許証ではこのことの繰り返しを防ぐための対策が施されていることを訴えかけていく必要があるでしょう。現在検討されているのは、申請時に本人がマイナンバーカードと運転免許証の両方を持参し、係員が申請者の目の前で両方の顔写真が一致することを確認するということです。これでちゃんと脅威に対策できているのかは要検討ですが。

落とした・無くした時の再発行に時間がかかる点

また、マイナ免許証に一本化した場合、これを落としたりした場合の再発行にかかる時間の問題も心配の一つです。車の運転には免許証の携帯が必要で、これに対応するために現行の免許証は落としたりした時に即日再発行が可能ですが、マイナンバーカードは1ヶ月半ほどかかることもあります。この間運転できなくなったら困ります。

これは、実は、現行の運転免許証を再発行してもらうことで回避できます。結局2枚に戻ってしまいますが。

今後の予定

この新しいシステム(マイナ免許証)の運用開始に向けて、警察庁は道路交通法の改正を2025年3月24日に施行する予定です。さらに、2024年9月13日から30日間、この新しい運用方法について一般の人々から意見を募集します。

マイナ免許証の導入は、日本社会のデジタル化における重要なマイルストーンとなります。 この変革が私たちの生活にどのような影響を与えるのか、今後の展開に注目が集まります。

参考記事

» 【マイナ免許証】来年3月24日開始へ マイナカードと運転免許証一体化 現行免許証の選択・併用も可能だが手数料はマイナ免許証より割高に?(FNNプライムオンライン(フジテレビ系)) – Yahoo!ニュース https://t.co/5lCDeGcip8

— Nat Sakimura/崎村夏彦 (@_nat) September 12, 2024

「マイナ免許証」25年3月導入へ 更新時講習オンラインに:日本経済新聞https://t.co/vImZKSbCp9

— Nat Sakimura/崎村夏彦 (@_nat) September 12, 2024

Wednesday, 11. September 2024

Patrick Breyer

Neuester EU-Anlauf zur Chatkontrolle: Werden Messenger-Dienste in Europa gesperrt?

Ungarn hat den EU-Innenministern diese Woche einen neuen Vorschlag zur Einführung der Chatkontrolle vorgelegt, der von POLITICO geleakt wurde. Der Einsatz „künstlicher Intelligenz“ zur Suche nach verdächtigen Bildern und Chats …

Ungarn hat den EU-Innenministern diese Woche einen neuen Vorschlag zur Einführung der Chatkontrolle vorgelegt, der von POLITICO geleakt wurde. Der Einsatz „künstlicher Intelligenz“ zur Suche nach verdächtigen Bildern und Chats soll Messenger- und Chatanbietern danach freigestellt bleiben. Zur Suche nach bekanntem illegalem Material bleibt es aber bei dem weltweit einzigartigen Vorhaben, die massenhafte, verdachtslose und nach BKA-Angaben höchst fehleranfällige Kontrolle aller privaten Chats für sämtliche Diensteanbieter verpflichtend zu machen, selbst wenn dazu die bisher sichere Ende-zu-Ende-Verschlüsselung von Messengerdiensten ausgehebelt werden muss. Die EU-Regierungen sollen sich bis zum 23. September zu dem Vorstoß positionieren, die EU-Innenminister sollen ihn am 10. Oktober beschließen. Die Messengeranbieter Signal und Threema haben bereits angekündigt, solche Überwachungsfunktionen keinesfalls in ihre Apps einzubauen und ihre Dienste in der EU gegebenenfalls komplett einzustellen.

“Statt Kindern durch sicherere Apps zu helfen sich im Netz vor Erpressung und Ausbeutung zu schützen, werden Missbrauchsopfer durch einen realitätsfremden und vor Gericht zum Scheitern verurteilten Gesetzentwurf im Regen stehen gelassen,” kritisiert Dr. Patrick Breyer, ehemaliger Europaabgeordneter der Piratenpartei und Mitverhandler der kritischen Position des Europäischen Parlaments. “Unsere Polizei mit großteils irrelevanten Hinweisen auf längst bekanntes Material zu überfluten, stoppt keinen Missbrauch, sondern erschwert seine Aufdeckung zusätzlich. Falls diese Chatkontrolle kommt, werden wir Bürger gängige sichere Messenger nicht mehr nutzen können – das bedeutet wir verlieren den Kontakt zu unseren Freunden und Kollegen in der ganzen Welt. Wollen wir wirklich, dass die EU weltweit führend beim Abhören unserer Smartphones und der flächendeckenden Überwachung der Chats von Millionen gesetzestreuer Bürger wird?”

Breyer ruft dazu auf, Druck auf Bundesinnenministerin Faeser und Bundesjustizminister Buschmann zu machen: “Nur dagegen zu sein und dann überstimmt zu werden, reicht nicht zur Verteidigung unseres digitalen Briefgeheimnisses und sicherer Verschlüsselung. Die Bundesregierung muss jetzt dringend Verbündete suchen, um eine Sperrminorität und eine grundlegende Überarbeitung zu erreichen. Die Sicherheit unserer Kommunikation steht auf dem Spiel!”

Info-Website: https://www.chatkontrolle.de

Hilf jetzt mit die Chatkontrolle zu stoppen

Doc Searls Weblog

On Journalism and Principles

Thirteenth in the News Commons series. I grabbed the spottedhawk.org domain after hearing Garrison Keilor read this passage from Walt Whitman’s Song of Myself over Leo Kottke improvising on guitar: The spotted hawk swoops by and accuses me. He complains of my gab and my loitering. I too am not a bit tamed. I too […]

Thirteenth in the News Commons series.

I grabbed the spottedhawk.org domain after hearing Garrison Keilor read this passage from Walt Whitman’s Song of Myself over Leo Kottke improvising on guitar:

The spotted hawk swoops by and accuses me.
He complains of my gab and my loitering.

I too am not a bit tamed. I too am untranslatable.
I sound my barbaric yawp over the roofs of the world.

Most of what I do is in deficient obedience to Whitman’s spotted hawk. Including this blog.

Early in this millennium, when it was as easy to blog as it later was to tweet, I blogged constantly. The stretch from 1999 to 2007 was blogging’s golden age, though we didn’t know it at the time. (My blog from that time is archived at weblog.searls.com.) My blog then was a daily journal, and in a literal way that made me even more of the journalist I had always been.

On that career side, I was also employed for all that time by Linux Journal. My name was on its masthead for twenty-four years, from 1996 to 2019. When LJ was sold at the end of that stretch, I left as editor-in-chief. After that, I was the host of FLOSS Weekly on the TWiT network. Both were paid gigs, and when the FLOSS Weekly gig ended last December, so did my long career in journalism.

And maybe that happened just in time, because journalism has since then acquired a taint. In this past weekend’s  Cornerstone of Democracy newsletter, Dan Gillmor sources Josh Marshall of Talking Points Memo on the topic:

I guess I would say that as journalists our core mission is fundamental honesty with readers. That means always telling readers the truth, an accurate story as nearly as we are able to uncover it and understand it, as well as being transparent with the values, commitments and core beliefs we bring to the work we do. We believe in always being fair to everyone and everything we write about. Fairness is really only another permutation of accuracy. Balance is a construct applied after the fact that is often as not at odds with accuracy. A belief in democratic republicanism or civic democracy has always been at the core of what we do. It’s central to what stories we choose to focus on, it’s a value structure that permeates our organizational approach to what we do. I can’t speak for everyone at TPM. But as founder and still a guiding light, I think our understanding of what journalism is or should be is inextricably connected with democratic republicanism/civic democracy. I don’t think I would say we’re activists for democracy. But to me being on the side of civic democracy is inextricably connected to what we do and who we are. We’re on the side of civic democracy as much as we’re on the side of journalism.

I don’t want to label other journalists. But to the extent many other journalists don’t operate in this way, or understand their job this way, it’s because they work for publications whose business models simply aren’t compatible with this approach to journalism. What we now commonly call “both-sidesism” is rooted in the business structure of most contemporary journalism, specifically the need to have purchase across of wide ideological spectrum of the population.This is especially so in large corporate journalism because the need for scale and advertiser buy-in really requires that access to all parts or most parts of the political spectrum. In a hyperpolarized political culture when one side veers off the democratic rail that inevitably pulls a lot of that kind of journalism with it. I see that as fundamentally a structural reality. This goes back many decades, long before the current moment. It was more workable in a different political climate. But in this one that’s how it works. Journalists think they’re following core journalistic principles. But it’s driven really by business models.

That’s why now I just call myself a writer. Or a photographer.† Unless it’s to write a book (which typically pays little or nothing), to be a paid journalist today requires frequent or constant production of “content.” But I’d rather yawp, like I am now, here, or over at ProjectVRM, TrunkLine (a blog on infrastructure), or Customer Commons (which will in the next year become more important than everything else I’m doing).

All three of those involve work, but don’t pay. Same goes for serving as Chief Intention Officer of Kwaai.ai, a commuity of 600+ volunteers working on personal AI. (Check it out. Good stuff going on there.)

But before I give up on the journalist label, let’s ask Perplexity.ai* to answer What is the purpose of journalism? Here’s the answer I just got:

Based on the search results, the primary purpose of journalism can be summarized as follows:

To provide citizens with accurate and reliable information they need to function in a free society. This is considered the central purpose of journalism. To inform the public about current events, issues, and developments happening in their communities and around the world. To serve as an independent monitor of power and act as a watchdog over those in positions of authority. To provide a forum for public criticism, debate, and compromise on important issues. To convey context and help citizens understand the significance of events. To set the tone for public discourse and help define community goals and values. To empower citizens to make informed decisions about their lives and society. To facilitate democratic processes by educating voters and inspiring civic engagement. To support commerce by creating awareness about products and services. To drive cultural transformation by connecting people to communities and ideas beyond their immediate surroundings.

Journalism is considered essential because it helps citizens navigate their lives, make important decisions, and participate effectively in democratic societies. It serves as a crucial source of verified information in an increasingly complex world, helping to separate fact from fiction and provide context to current events.

Is that what journalists are doing today? Was it ever? Really? All of it?

I don’t know. Mostly journalists write stories. If you’ve read Where Journalism Fails, or Stories vs. Facts, you know I think the story-telling imperative has its own problems.

Back in the early ’90s, when I was doing PR for a hot Silicon Valley startup, I asked a top reporter at one of the tech magazines why he didn’t want to see my client’s brilliant and charismatic marketing director. The reporter said, “He abuses the principle of instrumentality.” When I pressed the reporter to expand on that, he explained that everyone involved knows that reporters are used as instruments by whoever spins them. The “principle of instrumentality” is about knowing, and trying to ignore, the simple fact that journalism is instrumented in many ways. While Josh Marshall talks above about the instrumenting of journalism by business models, in this reporter’s case, it was by the persuasive charisma of a strong personality who wanted positive coverage.

I realized then that I wasn’t being hired at the same magazine (or at any publication before Linux Journal, and I pitched many) because I didn’t want to be an instrument. More specifically, they all wanted me to write sports stories about competing companies. “Apple vs. Microsoft,” for example. “Motorola vs. Intel.” I wanted to write about interesting stuff without favor to anybody or anything other than what seemed right, important, fun, or just interesting. Sports coverage of tech didn’t do it for me. Nor was any of the other usual stuff. Linux was a cause, however, so I worked to make my Linux Journal writing as non-evangelical as possible, though I did get credited with helping put both Linux and open source on the map.

Was I a journalist while working as an editor there? I suppose so, given that my work hit at least some of the ten items above. At least I thought of myself that way.

A difference today is that we are all both digital and physical beings. Here in the digital world (where I am now), anybody can publish anything, on many different platforms, including their own if they’re geeky enough to make that work. According to the Podcast Index, there are 4,262,711 podcasts right now. Instagram has over two billion users. Says here there are over three billion blog posts published every year, and over six hundred million active bloggers. (I suppose I am three of them.) The same piece says “Over 90% of blog posts receive zero traffic.” Many of those blogs are faked-up, of course, but it’s still clear that the world of online publishing is a red ocean, while Mastodon, Threads, Bluesky, Nostr, and the like are more like small rivers or bays than one blue ocean. (Links in that last sentence go to my tiny presence in each. I’m also still on Xitter and Linkedin, for what those are worth.)

So now I’m thinking about what principles, old and new, work in the digital media environment and not so much in the old analog one.

Here’s one: We’re past the era of “What’s on.”** Unless it’s a live sport or some other kind of scheduled must-see or must-hear event, you can catch it later, on your own time.

Here’s another: We don’t have to fill time and space with a sum of “content.” We don’t. Yes, it helps to have a schedule and be predictable. But it’s not necessary, especially if you’re being paid little or nothing.

Here’s another: The challenge now is building and maintaining an archive of facts, and not just of stories. I’ve written about this elsewhere in this series. Go look it up.

Another:  Try to grab as many of those facts as you can before and after they turn into stories or don’t. This is what calendars are for. Even if nothing comes out of a meeting or an event that appears on a calendar, it’s good to know that something happened. And to archive that as well.

I also believe both of those principles are easiest to apply in local contexts, simply because there is a relatively finite sum of facts to work with locally, and facts still matter there. (Scott Adams tells us they don’t in the wider world. And he has a case.)

This is one reason I’m embedded in Bloomington, Indiana. We’re working on all that stuff here.

† My photos here and here on Flickr have about twenty millon views so far. The last peak was five thousand on Sunday. The top attraction that day was this shot of Chicago I got on a cross-country flight between Phoenix and Boston in 2011. That one photo has logged 26,159 views so far. All my photographs are CC-licensed and free to use, which is why over 4,000 of them are in Wikimedia Commons, a library of images used in Wikipedia. So thousands of those (it’s hard to tell exactly how many) end up in Wikipedia. Many more accompany news stories, such as this one from Lawrence, Massthis one from a power plantthis one from a lithium mine, and all these from Chicago. And I put none of them in either Wikimedia Commons or Wikipedia. Other people do that. I just put the photos out there. Meanwhile, this blog maxed at a little over 300 views one day last week, but usually gets a dozen or so. My old blog ran at about five thousand a day, and sometimes ten times that many. To bring this back to the theme of this post, while I do a lot of photography, I don’t think of myself as a photographer. I take pictures. And I write. And I talk some. All are for the same purpose: to be useful and to make stuff happen. Good stuff, hopefully.

By the way, the bird at the top is a juvenile red-tailed hawk. There is no one species called a spotted hawk, so this one will have to do. I shot this bird a couple of months ago, perched on a 480-volt line in the alley behind our house in Bloomington, Indiana. I was looking to hear a barbaric yawp, but he, or she, failed me on that one.

*I first asked ChatGPT 4o and got a cliché’d answer with no sources. Perplexity gave me a longer answer, just as cliché’d, but with eight sources. For fun, go ask both and see what you get. Try Claude and Gemini too. No two will be the same. Some will be better, some worse.

**I’ve written about this, but haven’t published it yet. Stay tuned.

 

Tuesday, 10. September 2024

The Pragmatic Engineer

AI Tooling for Software Engineers: Rolling Out Company-Wide (Part 3)

Why are some companies not using GenAI yet? Also: AI guidelines and practical approaches for embracing LLM tools for software development on the individual developer, and organizational level

Hi, this is Gergely with a subscriber-only issue of the Pragmatic Engineer Newsletter. In every issue, I cover challenges at Big Tech and startups through the lens of engineering managers and senior engineers. To get articles like this in your inbox, every week, subscribe:

Subscribe now

Before we start: you can now see use “table of contents” quick navigation on the right side of each article, when reading the newsletter on the web. Just click on the sidebar, and you can navigate this article — and all other The Pragmatic Engineer articles. See it in action on the web. Happy browsing!

There’s no shortage of big claims about what LLM tools will be able to do, or should be able to do in the software engineering field. But what do they actually do, right now? We asked software engineers who regularly use these tools, and engineering leaders who oversee these tools in their organizations.

This article is based on a survey of 216 professionals and is the third and final part of a mini-series on GenAI tooling. It covers how these tools are being used ‘day-to-day’ in tech workplaces, and what engineers think about them. Today, we cover:

AI usage guidelines. A quarter of respondents follow company-wide usage guidelines. Interestingly, a minority of companies have banned GenAI tools over security and copyright worries.

Internal LLMs at Meta, Microsoft, Netflix, Pinterest, Stripe. Large, fast-moving companies not only embrace GenAI tools, but build their own internal versions for their engineers. Vendors are starting to offer similar boxed products.

Reservations and concerns. Most common reservations, and how to overcome them – and why devs tend to start using LLMs regularly.

Advice for devs to get started with AI tools. Start small, verify outputs, don’t “outsource” coding and other advice.

Advice for engineering leaders to roll out AI tooling, org-wide. A roundup of how companies adopted these tools successfully: guidelines, tooling, training, and how these impact junior engineers.

Measuring the impact of GenAI tools. Most engineering leaders say these tools have no visible or measurable impact – at least not yet. We suspect this is because the impact is hard to measure: and not due to the lack of impact.

AI strategy. Why do companies incorporate GenAI into their software engineering workflow? Experimentation and hoping for increased productivity are two big reasons.

In Part 1 of this series, we covered:

Survey overview

Popular software engineering AI tools

AI-assisted software engineering workflows

The good

The bad

What’s changed since last year?

Part 2 was about:

What are AI tools similar to?

State of AI tooling in 2024: opinions

Critiques of AI tools

Changing views about AI tooling over time

Which tasks can AI already replace?

Time saved – and what it’s used for

Now, let’s dive into this final part of this mini-series.

1. AI usage guidelines across companies

We asked survey participants “how is AI tooling used for development at your company?” The responses reveal different approaches:

Common ways LLM tooling is used at tech companies

The most referenced approaches:

No formal guidelines. Around 25% of respondents (53 out of 216) say their company has no recommendations about AI tooling. People use it as they see fit.

Some guidelines. 25% (50 respondents) say their workplaces have rules and guidelines for AI tooling.

AI tools banned. Circa 12.5% (25 responses) say their businesses ban usage of AI tools, mostly due to concerns about code security, and potential copyright infringement. We previously covered how several open source projects have banned AI-generated code commits for this reason. 

Working on guidelines. 7% of respondents (15 people) share that their company is trialing AI tooling, or is in the process of adopting guidelines.

Strongly encourage AI tool usage. 6% of respondents (12 people) work at places which encourage using these tools wherever possible.

Common features of guidelines across workplaces, based on survey responses:

Specifying which tools and LLM models may be used

No inputting of sensitive information into AI tools like ChatGTP

No entering of internal (closed-sourced) code into AI chat tools

It’s pretty clear some guidelines are responses to fears that LLMs may retain the data employees input and use it for training. This is also a reason why a handful of respondents shared that their companies go through the added complexity of running LLMs on their own infrastructure. It’s a reminder that LLM solutions which don’t store company data have a strong selling point for tech companies.

2. Internal LLMs at Meta, Netflix, Pinterest, Stripe

Only a fraction of respondents say their companies strongly encourage the use of LLM tools, but some of these are cutting-edge market leaders in tech. Let’s take a look at how a well-built internal LLM can help a business.

Meta

The social media giant has been investing heavily in ML and AI since before ChatGPT was released. Back in 2022, we covered how Meta was already preparing for AI/ML ‘wartime’ by investing heavily both in AI hardware, and hiring large numbers of AI and ML engineers. This investment has not slowed down since, and it’s little surprise that Meta seems to have built one of the leading in-house AI tools.

Meta’s internal tool is called Metamate. Director of Engineering Esther Crawford describes it:

“It’s an AI for employees that’s trained on an enormous corpus of internal company docs. I use it all the time for efficiency gains.

Any sizable company operating without an internal AI tool is already behind the curve.”

Esther explains what Metamate does:

“It has a ton of capabilities from summarizing to coding. Simple use cases: 

Summarizing docs 

Recapping work 

Finding info across wikis 

Exploring metrics and visualizing data

PSC feedback editor

Ask about diffs, SEVs, Deltoids, Tasks

Do the math for things like calculating capacity 

Getting project status updates from GSD

Create queries, and so much more.”

Here’s a practical example on how useful Meta’s tool is, from Shana Britt E, director of strategic initiatives:

“Recent use case: Performance reviews. Writing self-review, cleaning up peer reviews. For self-review, it can capture your diffs landed, status updates about your work from documents you published, etc. and puts it in a nice summary that you can then review and edit.”

Microsoft

The company offers Microsoft Copilot for Microsoft 365 for enterprises, and is dogfooding this system inside the company. I talked with software engineers who confirmed that the internal Microsoft Copilot is integrated with internal documents, and can thus provide more relevant context. It is also used in places like pull request reviews – although for this use case, I heard it’s more of a hit-and-miss in the quality of feedback.

Stripe

The fintech company has a similar system to Metamate. Miles Matthias, product manager, shares:

“We have something similar [to Metamate] at Stripe and I spend a bunch of my time talking to it. I can imagine a world where I’m basically having a voice conversation with it all day every day as ‘work’ - especially when agents boom.”

Netflix

The company has a place to access Netflix-provided versions of LLMs. A senior software engineer told us:

“There are AI guidelines, and corporate-provided versions of GPT, Claude and other models in a unified interface. People can share prompts that they find useful to colleagues. 

My org is also exploring AI for our specific use cases, but thus far have not found any AI tools to be where we need. There is an opportunity to automate some manual business processes and we thought GenAI could help, but it seems traditional engineered solutions are still much better than GenAI."

Pinterest

The company builds internal LLM tools. One clever utility is called Text-to-SQL: a feature where internal users can use plain text to ask for a type of query, and the tool generates the right SQL to be used with the company’s internal data store called Querybook. The engineering team improved the first version with RAG, to help identify the right table names to use (we previously did a deepdive on applied RAG). The results are promising. As the company shares:

“We find a 35% improvement in task completion speed for writing SQL queries using AI assistance.”

Vendors offering similar capabilities

There are plenty of vendors offering a “Metamate-like” experience out of the box. Glean seems to be the leader in this area. Other options include Village Labs, Microsoft Copilot for M365, Coveo and Akooda. This category is relatively new and there are plenty of up-and-coming startups. Search for terms like “AI company knowledge management tools” to find them.

The productivity perception of these systems rarely matches reality. Despite being a leader in the AI field, Meta is just figuring out how these tools can help it operate more efficiently. Metamate sounds impressive – and it’s ahead of what most companies have – but it doesn’t work optimally just yet, as we hear. I got this detail from talking with current engineers working at Meta.

The reason companies like Meta are investing so much into this area was articulated by CEO Mark Zuckerberg two months ago, on the company’s earnings call. He talked about how AI investments will take years to pay off, and Meta wants to be early. He said:

“You can also get a pretty good sense of when things are going to work years in advance. And I think that the people who bet on those early indicators tend to do pretty well, which is why I wanted to share in my comments the early indicator that we had on Meta AI, which is [...] early.”

3. Reservations and concerns

When starting to use AI tooling, companies and developers often need to overcome reservations, or find workarounds. Let’s start by summarizing these reservations.

Reasons for not using AI tooling

Reasons for disallowing – or heavily limiting – AI tools include security and privacy worries; especially about internal, confidential information, and proprietary code being leaked. A few respondents also mention customer data.

Several larger companies have worked around these concerns by using in-house, self-hosted, LLMs, and their security and compliance teams add filtering to the inputs and outputs of these tools. This approach is clever:

Security and compliance teams can tweak filters to catch confidential or customer information that shouldn’t be shared

If confidential information is fed into a self-hosted model, this data does not leave the company to an external vendor

The obvious downside is that it’s not trivial to build and maintain. However, given that leading tech companies already have internal models and are heavy users, it’s likely other businesses will follow by either building in house, or using a vendor offering hosted LLMs with capability for internal security teams to tweak filters.

Developers’ reservations

But it’s not just companies dragging their feet; developers are also hesitant about LLMs in the survey:

Commonly cited ethical and environmental concerns:

“The model used to power most AIs represents a large theft of labor from the commons, all to deliver increasingly terrible results.” – Senior software engineer, 19 YOE

“I have ethical concerns about code theft, and environmental concerns about energy consumption.” – Project lead, 9 YOE

“I feel their massive energy use goes against my personal code of ethics” – Software engineer, 8 YOE

“I am uncomfortable with its resource and energy usage, biases and AI hype, as ways to concentrate even more money and power at big tech companies and their culty leaders, which feels too adjacent to the Blockchain hype and grifts from a few years back for comfort.” – Software engineer, 40 YOE

These criticisms are valid. Large language models are known to be trained on copyrighted code, as well as on copyleft-licensed code, where the license is not complied with. And the surge in energy usage is also real, as covered in Is GenAI creating more carbon pollution by cloud providers?:

“It appears that the latest technological wave of GenAI may be getting in the way of corporate climate goals. Large language models (LLM) are very hardware and energy-intensive, and Azure, Google Cloud and AWS have all rapidly expanded their data center capacity and power usage, in order to meet demand. With energy usage surging, so are carbon emissions; which is the opposite direction from what they need to be going in, if these companies are to hit Net Zero in 2030 or any time after.

Google: carbon emissions up 48% in 2023, compared to 2019

Microsoft: carbon emissions up 30% in 2023, compared to 2020.”

There are clear benefits to GenAI, but also technological downsides. The ethical concerns seem to have no easy answers, while the history of computing has been about making computers more energy efficient, so we should expect the same here. At the same time, it’s concerning that GenAI is used to justify creating data centers which consume massive amounts of energy, or considering nuclear-powered data centers to keep up with computing demand.

Not enough utility, yet: We previously summarized negative sentiments in “Unimpressed” critiques in Part 2 of this survey. Common complaints about AI from engineers include:

Useful for simple stuff only, poor performance in more complex tasks

Little use outside of repetitive changes and boilerplate generation

Unreliable due to generating buggy code

Seen as a “fancy autocomplete”

More a hindrance than a help

Tedious to work with 

Here are two more comments from engineers who stopped using AI tools:

“Seems useful for certain tasks, particularly writing related. For specific coding I could see it being used to generate more boilerplate, but personally my team tends to invest more in common libraries that reduce boilerplate anyway (while ensuring best practices are followed)” – Senior software engineer, 5 YOE

“ChatGPT is a novel tool with some potential to speed up boilerplate work and learning/investigation. It is not a high value for expert software engineers yet, but I’m optimistic that it will improve in a few years.” – Principal software engineer 20 YOE

These reservations are valid, but survey responses show that using LLM tools for 6+ months changes the views of many developers: mostly to a more positive, or more grounded, viewpoint. If you have an underwhelming first impression of these tools, it might be worth trying them daily for a bit before making up your mind.

Why do devs start using LLMs?

We asked tech professionals why they started using these tools. The most common responses listed by frequency:

Company pushes LLM usage. Several large businesses set targets for departments of numbers of developers using LLM tools. Companies buying GitHub Copilot licenses also pushed workers to onboard. We’ve heard about company mandates, LLM training, and management expecting devs to use these tools for productivity.

To be more efficient / lazy to google it. Developers may adopt these tools to become more efficient, or because they can’t be bothered doing a web search. All found the tools help them get unblocked faster.

Peer pressure. Friends and peers in tech, and cofounders at startups, recommended them

Hype. Non-stop news about AI played a role in influencing software engineers to check out how the new technology works. 

Pressure to keep up. Not wanting to fall behind in the industry, while seeing others use AI tools.

Curiosity. Discovering how the tech can help with their work, or how much better (or worse) it works compared to their existing workflow (especially versus Googling when solving problems, or using Stack Overflow)

An interesting detail for us is that company mandates and pushes are the single most-cited reasons for starting to use AI tools. It seems these do work – at least for that initial “push” to give the tools a go.

4. Advice for devs to get started with AI tools

Read more


@_Nat Zone

戸籍のフリガナの届出は2026年5月26日まで。届け出がない場合、市区町村長により職権でフリガナ記載

2024年9月10日、戸籍にフリガナを記載する改正戸籍法の施行日が2025年5月26日と閣議決定されました。これにより、戸籍へのフリガナ記載に向けた準備が本格化します。 改正の背景と目的 この法改正の主な目的は、 の3点です。 現在の戸籍では氏名の読み方が不明確な場合があり、行政…

2024年9月10日、戸籍にフリガナを記載する改正戸籍法1の施行日が2025年5月26日と閣議決定されました。これにより、戸籍へのフリガナ記載に向けた準備が本格化します。

改正の背景と目的

この法改正の主な目的は、

行政のデジタル化基盤整備の促進 本人確認情報としての利用 各種規制の潜脱行為の防止

の3点です。

現在の戸籍では氏名の読み方が不明確な場合があり、行政手続きや本人確認の場面で不都合が生じていました。

新制度の概要

2025年5月26日以降、戸籍にカタカナでフリガナが記載されるようになります。主な変更点は以下の通りです:

新生児の場合、出生届に記載されたフリガナが戸籍に反映されます。 すでに戸籍に氏名が記載されている人も、フリガナの届け出が必要になります。 フリガナには一定の基準が設けられます。 届け出の方法と期限 制度開始から1年以内に「振り仮名の届出」をする必要があります。 戸籍の筆頭者は氏名、それ以外の人は名前のフリガナを届け出られます。 届け出は書面またはマイナポータル(予定)を通じて行うことができます。 注意点 届け出がない場合、市区町村長により職権でフリガナが記載される可能性があります。現在使っている読み方と異なる読み方が登録されてしまう可能性があります。 フリガナは一般に認められる読み方でなければなりません。つまり、キラキラネームはNGです。逆に、一般の読み方以外の読み方を現在利用しておられる方は、パスポートや預金通帳など証拠を提示することによってそれを登録することができるようになります。 経過措置として、一度のみ家庭裁判所の許可なしでフリガナの変更が可能です。 自治体の対応

施行に向けて、自治体は以下の準備を進めています:

正確かつ迅速な通知発送:本籍地の市町村長が戸籍に氏名の振り仮名を記載する前提として、戸籍に記載される予定の氏名の振り仮名等を認識する機会を確保することとしています。具体的には、住民票において市町村が事務処理の用に供するため便宜上保有する情報等を参考に、本籍地の市町村長から皆様に、氏名の振り仮名に関する情報を通知することとしています。この通知は、改正法の施行日(令和7年5月26日)から遅滞なく送付することとしています。 オンラインとアナログの併用:氏名の振り仮名の届出については、当該届出をする者の本籍地又は所在地の市町村に行うこととなりますが、窓口への出頭及び郵送による方法が可能です。また、マイナポータルを利用する方法についても現在検討中です。

この制度変更により、戸籍におけるフリガナの取り扱いが明確になり、行政手続きの効率化や本人確認の精度向上が期待されています。市民の皆様は、制度開始後1年以内にフリガナの届け出を行うことを忘れずに、新しい制度に対応していく必要があります。

詳しい情報は、法務省のサイトをご覧ください。

【9月11日追記】

総務省から住民記録システム標準仕様書【第5.1版】等の公表がありました

令和6年9月11日

住民記録システム標準仕様書【第5.1版】等の公表

「自治体システム等標準化検討会(住民記録システム等標準化検討会)」(座長:庄司昌彦 武蔵大学社会学部教授)において、住民記録システム標準仕様書【第5.1版】及び戸籍附票システム標準仕様書【第3.1版】が取りまとめられましたので、公表します。

公表資料

・ 住民記録システム標準仕様書【第5.1版】等の改正概要PDF
・ 住民記録システム標準仕様書【第5.1版】
・ 戸籍附票システム標準仕様書【第3.1版】

※ 住民記録システム標準仕様書【第5.1版】及び戸籍附票システム標準仕様書【第3.1版】については、以下の総務省ホームページにおいても公開します。

総務省トップ > 組織案内 > 研究会等 > 自治体システム等標準化検討会
https://www.soumu.go.jp/main_sosiki/kenkyu/jichitaishisutemu_hyojunka/index.html

Monday, 09. September 2024

Michael Ruminer

My Surprising Llama 3.1 8B Post Stats

Back in July, I made a post here in Medium on running Llama 3.1 8B locally not long after the model dropped. It is my most viewed and most read post by between 1 and 2 orders of magnitude based on my poor general reach at this writing. It was surely the timeliness and high interest in all things Llama 3.1 at that moment. I looked at the stats and they were enlightening on the distribution of refer

Back in July, I made a post here in Medium on running Llama 3.1 8B locally not long after the model dropped. It is my most viewed and most read post by between 1 and 2 orders of magnitude based on my poor general reach at this writing. It was surely the timeliness and high interest in all things Llama 3.1 at that moment. I looked at the stats and they were enlightening on the distribution of referrals. Google is still king by a wide margin and my LinkedIn effort produced almost nothing.

Here are the stats:

The whole set was surprising to me. I made a post about this article only to LinkedIn so I was surprised at only one linkedin.com referral. I thought I had clicked on it more times than that. :-) It kind of makes me sad. Wow, Google. Not sure what I did to show up in some search results but I wish I did that more often. 146 in email, IM and direct. YCombinator… really. I’m honored. Gemini.google.com — interesting. Twitter — 2; very surprising as I didn’t post there and haven’t used Xitter in about a year for what would be obvious reasons for some people. Getliner.com — I don’t even know what that is. Sorry getliner.com. Identosphere.net — mildly surprised. I get picked up in the blog aggregator as I have in the past posted more reliably on digital identity and verifiable credentials.


Damien Bod

Implement OpenID Connect Back-Channel Logout using ASP.NET Core, Keycloak and .NET Aspire

This post shows how to implement an OpenID Connect back-channel logout using Keycloak, ASP.NET Core and .NET Aspire. The Keycloak and the Redis cache are run as containers using .NET Aspire. Two ASP.NET Core UI applications are used to demonstrate the server logout. Code: https://github.com/damienbod/keycloak-backchannel Setup The applications are run and tested using .NET Aspire. […]

This post shows how to implement an OpenID Connect back-channel logout using Keycloak, ASP.NET Core and .NET Aspire. The Keycloak and the Redis cache are run as containers using .NET Aspire. Two ASP.NET Core UI applications are used to demonstrate the server logout.

Code: https://github.com/damienbod/keycloak-backchannel

Setup

The applications are run and tested using .NET Aspire. The UI applications are setup using the OpenID Connect code flow with PKCE and OAuth PAR. Two docker containers are used, one for the Redis cache and one for the Keycloak server.

.NET Aspire Setup

The AppHost project in .NET Aspire is used to configure the different services. The Keycloak container is run using HTTPS with develop certificates. The ASP.NET Core applications are run using development certificates as well. For the Keycloak server to access the docker host, it must register the same developer certificates from the host, or disable the certificate trust manger inside the container. This works good, but should only be setup like this in development. The different project must reference each other as required. To use Redis and Keycloak, the Aspire Nuget packages for these containers need to be installed.

var keycloak = builder.AddKeycloakContainer("keycloak", userName: userName, password: password, port: 8080) .WithArgs("--features=preview") // for more details regarding disable-trust-manager see https://www.keycloak.org/server/outgoinghttp#_client_configuration_command // IMPORTANT: use this command ONLY in local development environment! .WithArgs("--spi-connections-http-client-default-disable-trust-manager=true") .WithDataVolume() .RunWithHttpsDevCertificate(port: 8081); var cache = builder.AddRedis("cache", 6379) .WithDataVolume(); var mvcpar = builder.AddProject<Projects.MvcPar>("mvcpar") .WithExternalHttpEndpoints() .WithReference(keycloak) .WithReference(cache); var mvcbackchanneltwo = builder.AddProject<Projects.MvcBackChannelTwo>("mvcbackchanneltwo") .WithExternalHttpEndpoints() .WithReference(keycloak) .WithReference(cache);

Keycloak OpenID Connect client configuration

The Keycloak client should have the backchannel logout activated. The container uses the localhost applications from the docker host and so the host.docker.internal domain is used. The logout endpoint is implemented in the ASP.NET Core application.

ASP.NET Core Logout

Each ASP.NET Core application that supports the back-channel logout must have a server implementation and provide a web hook for the identity provider (Keycloak) logout event. If multiple instances are running, one can send a logout event to Keycloak. This ends the session on the identity provider and sends a logout post request to all server backends hosting the UI application. The logout event is handled and persisted to a distributed cache. For all other instances which request data from the server for the same user and session, the session is ended and the user must authentication again.

Logout request Webhook for logout event from Keycloak server for all instances Persist event to cache if missing and logout Logout UI on next HTTP request for user sessions

services.AddTransient<CookieEventHandler>(); services.AddSingleton<LogoutSessionManager>(); services.AddHttpClient(); services.Configure<AuthConfiguration>(configuration.GetSection("AuthConfiguration")); var authConfiguration = configuration.GetSection("AuthConfiguration"); builder.AddRedisDistributedCache("cache"); services.AddAuthentication(options => { options.DefaultScheme = CookieAuthenticationDefaults.AuthenticationScheme; options.DefaultChallengeScheme = OpenIdConnectDefaults.AuthenticationScheme; }) .AddCookie(options => { options.ExpireTimeSpan = TimeSpan.FromMinutes(60); options.Cookie.Name = "MvcPar"; options.EventsType = typeof(CookieEventHandler); }) .AddOpenIdConnect(OpenIdConnectDefaults.AuthenticationScheme, options => { options.Authority = authConfiguration["StsServerIdentityUrl"]; options.ClientSecret = authConfiguration["ClientSecret"]; options.ClientId = authConfiguration["Audience"]; options.ResponseType = OpenIdConnectResponseType.Code; options.Scope.Clear(); options.Scope.Add("openid"); options.Scope.Add("profile"); options.Scope.Add("email"); options.Scope.Add("offline_access"); options.ClaimActions.Remove("amr"); options.ClaimActions.MapJsonKey("website", "website"); options.GetClaimsFromUserInfoEndpoint = true; options.SaveTokens = true; options.PushedAuthorizationBehavior = PushedAuthorizationBehavior.Require; options.TokenValidationParameters = new TokenValidationParameters { NameClaimType = JwtClaimTypes.Name, RoleClaimType = JwtClaimTypes.Role, }; });

Note: The sample code in the repository was created using the IdentityServer4 Samples.

Redis cache

Redis Insight can be used to view the Redis cache data. Each time the application handles a new user and session logout event, it persists the event to the cache. If any further application instances are authenticated using this session and user, the application will sign-out as well on the next server event. The expiry time of the cache should be long enough so that an offline session cannot be opened after it expires.

Limitations

The back-channel logout only works on a per browser session because Keycloak creates new sessions for each browser. When the logout is received it is filtered and handled using the Keycloak session. If Keycloak can use a single session for all browsers of each user, then the logout can work for all active UI apps of the same user.

Links

https://www.keycloak.org/

https://openid.net/specs/openid-connect-backchannel-1_0.html

https://ldapwiki.com/wiki/OpenID%20Connect%20Back-Channel%20Logout

https://datatracker.ietf.org/meeting/97/materials/slides-97-secevent-oidc-logout-01

https://docs.microsoft.com/en-us/aspnet/core/fundamentals/app-state

https://docs.microsoft.com/en-us/azure/azure-cache-for-redis/cache-dotnet-core-quickstart

https://github.com/RedisInsight/RedisInsight

Sunday, 08. September 2024

Werdmüller on Medium

A conversation with myself about immigration

What is your ideal place? What are its values? Continue reading on Medium »

What is your ideal place? What are its values?

Continue reading on Medium »

Friday, 06. September 2024

Michael Ruminer

Having AI Tools Generate Code For AI Tools

I recently posted on my experience with using a few Python examples of retrieval augmented generation (RAG) with an LLM (OpenAI API). They were underwhelming, though they did provide a lot of insight for me. This post is about my next experiment: let’s get AI tools to generate examples for me. The results were mixed but not for reasons you might think. Here’s what I found using ChatGPT, Microsoft

I recently posted on my experience with using a few Python examples of retrieval augmented generation (RAG) with an LLM (OpenAI API). They were underwhelming, though they did provide a lot of insight for me. This post is about my next experiment: let’s get AI tools to generate examples for me. The results were mixed but not for reasons you might think. Here’s what I found using ChatGPT, Microsoft Copilot and Cursor.

Note: you can find the generated code in my LLM_Generated_RAG Github repo. I will likely add to this repo as I try new prompts, LLMs, and approaches.

ChatGPT 4o

Let’s start with the most well-known, ChatGPT. I used a simple prompt to see what it would do. I didn’t provide prompts for different parts of the code. It should not be very extensive code so I didn’t feel the need to divide it into separate prompts. If you want to see my very elementary prompt pull up the ChatGPT related file in the repo.

The first thing I noticed is that it was using PyPDF2 which was deprecated in Decemeber of 2022. Not surprising as, almost certainly, a lot of what the model was trained on used PyPDF2. It actually created well-structured code with functions for the different discrete operations. What I realized later, due to the output of the Cursor tool, is that it created a more granular set of code than it had to. This is exactly what I was looking for but didn’t specify in the prompt. What do I mean? It did the following:

read the PDF and extracted the text split the document into chunks created embeddings for the chunks using the OpenAI text-embedding-ada-002 created an embedding for the query searched the chunks using faiss generated a response using the query and the chunk(s) as context

This is what I wanted. Turns out there was a very different way to interpret the prompt.

The downside. The code didn’t run out the box. Also, I believe it only used one chunk for the context to send along with the query. I have to investigate the code more closely to be sure on that single chunk thing. The embedding search chunk function I need to investigate more to understand it. I didn’t try to make the code run for now as that was not part of the experiment. I expect I will try to modify it soon to function. I’ll report back the results.

This granular set of actions was very different than what I got from Cursor.

Cursor

The Cursor prompt was not exactly the same as what I used for ChatGPT, which was bad of me to do. I got a little lazier, but the crux was the same. You can see that prompt here.

The results were very different. It did not provide the granular steps that ChatGPT did. It met the requirement more succinctly.

extract the text from the PDF, also using PyPDF2 pass the entire text to ChatGPT as context along with the prompt

When I first posted this I said it rain out of the gate, but that was wrong. It suffered from the same issue all three code generations did. It tried to use an old call of openai chat completion. With that fixed I believe it would, hypothetically, return the expected results. It would work similarly to how one does it in the ChatGPT interface. Not what I wanted, but I hadn’t specified to break it down so that I was creating embeddings etc. Fair game.

The downside, the context was too large (43,000+ tokens) for the tokens per minute limit of my OpenAPI account level and gpt-4o model (see attached note for an explanation). So I didn’t get to see the output, but have no doubt it would have produced similar results as if I had done it through the ChatGPT user interface.

Microsoft Copilot

What can I say? I don’t know what I would have gotten here because Copilot blocked the output. This is apparently because “GitHub Copilot checks code completion suggestions with their surrounding code of about 150 characters against public code on GitHub. If there is a match, or a near match, the suggestion is not shown to you.”

There is a flag you can set in your Github settings to turn this on or off. I checked and my flag and it is set to “Allowed” but you see the results.

I’ll continue to try to troubleshoot this issue and see if I can get something out of Copilot and update this post if I do.

In recap

In recap. One provided the detail I wanted even though I hadn’t specified it, a happy coincidence, but it did not run out of the gate. The second took a very different approach and would have provided the desired results in LLM response if my OpenAPI account level had supported a large enough tokens per minute for 43,000+ tokens in the context. But it wasn’t the code I was hoping for. Copliot just didn’t work for me for reasons I don’t yet understand. More experimentation to come.


Justin Richer

Making Bubbles: Re-connecting

If a set of accounts live in isolation forever, what happens to those accounts only matters within that isolated system. But when we make a bubble in our federated network, we aren’t just making a stand-alone system that can go live in a fully disconnected state. Over time, we expect things to re-connect, and when that happens, data needs to be harmonized across the boundaries. So many bubbles, so

If a set of accounts live in isolation forever, what happens to those accounts only matters within that isolated system. But when we make a bubble in our federated network, we aren’t just making a stand-alone system that can go live in a fully disconnected state. Over time, we expect things to re-connect, and when that happens, data needs to be harmonized across the boundaries.

So many bubbles, so many connections Data Synchronization

When multiple independent systems live together in the world and share data, inevitably that data is going to get out of sync. In a purely heirarchical system, we’re mostly concerned with building a consensus around the correct state of the shared data. We see this approach in distributed ledger systems, where nodes eventually come to a common view of what the shared state of the world ought to be.

But we don’t have that in a bubble-based architecture, because we don’t expect everyone to have the same view. Instead, we expect many different independent views to contribute to each other in a distributed fashion. This effectively means that for each bubble, it can send updates in and out to other systems. In most cases, there is a directionality to the data flow: one side is going to be treated as more authoritative than the other for a given context. What a bubble does when it’s on either end of that gap changes how we view the synchronization.

Account Updates From Above

When changes are pushed to us from an authoritative source, the simplest thing is to overwrite everything. After all, if it’s authoritative, why wouldn’t we just take that statement as truth? But the reality is that we’ve likely augmented our record of that user with additional details, overrides, and other localized changes that we don’t want to lose.

Local data (blue) shadows updates from the source (orange), in some cases

In these cases, we can shadow the data. In other words, we keep a copy of the source’s data separate from our own local view. When we get an update from that source, we can update our copy of the source’s data with wild abandon. We can then decide, by local policy, whether we want to adjust our overrides based on the update. Importantly, this decision is separate from accepting and processing the updated data from the source. This setup allows us to keep local information in the bubble at the same time that we sync from elsewhere.

Account Updates From Below

An authoritative bubble is sometimes going to want to pull updated information from the bubbles that live downstream. These are systems that we’ve sent accounts out to, and those systems might have something to tell us about our users. Maybe there’s an additional accreditation that’s been earned, or something to represent additional access details outside of our bubble, or even just an update to one of the core fields we sent down.

In any case, the downstream bubble is sending us additional data about a user, and we now have the chance to do something about it. If nothing else, we can store it and note it. If we want to, we can update the user’s record that we hold locally, and even go so far as to propagate that downward again to other bubbles.

Changing Trust

It’s not just user data that we can pass around, though that’s the most common piece we’d expect to see. The bubbles can also pass about other bubbles to each other, and incorporate what they learn into their own systems.

For many situations that fit the bubble architecture patterns, we expect to meet new peers in the field and make new decisions based on local context and requirements. These newly-discovered bubbles can then be propagated through the network, along with potentially interesting information about what the presenting bubble trusts it for.

Audit Logs

Now that we can identify users and the systems they come from, we can start to do one of the most interesting and perhaps complex jobs of a reconnected bubble: audit logs. When a bubble gets provisioned, that provisioning authority is likely to want to know what happens in that bubble during the disconnected spells. The bubble can package up the relevant audit log history and pass it along to the authorities that need it.

But auditing can go further than that: for any user that enters our bubble from somewhere else, we probably want to report back to the system that sent them to us. Since we know where we originally learned of them, and we know how to identify that system as a whole, we can filter and target the information we’re sending over. And we can do this while processing the updates they’re sending us about the account.

Conclusion

The bubble pattern sits between the always-connected and always-separate models, but the moment of reconnection gives us a profound opportunity to process user and trust data in ways that we may have never considered.

Thursday, 05. September 2024

The Pragmatic Engineer

The Pulse #105: More trad tech companies to move off the public cloud?

Also: CockroachDB joins the trend of going from open source to proprietary license, a software glitch nearly floods Amsterdam, and more.

The Pulse is a series covering insights, patterns, and trends within Big Tech and startups. Notice an interesting event or trend? Send me a message.

Today, we cover:

Industry pulse. Microsoft Recall to be unremovable, another “AI talent raid” by Amazon, Ticketmaster struggles to handle common load challenges – sometimes, rise and fall of OpenSea, and more.

More trad tech companies to move off public cloud? In a recent report, 83% of chief information officers say they want to move more workload to private clouds. Cloud computing has matured and is becoming more expensive, while private cloud technology is increasingly accessible.

CockroachDB: open source product goes proprietary. Another VC-funded open source product closes itself off, in hope of generating more revenue and profits. Many users need to open their wallets and buy annual licenses, or fork and maintain an older version of CockroachDB.

Software glitch almost floods Amsterdam. Dams near the Dutch capital that were supposed to be closed, mysteriously opened at night during a violent storm. Disaster was averted by a curious employee – and the workaround to avoid similar incidents includes 24-hour human monitoring of dams and water levels.

1. Industry pulse Microsoft Recall to be unremovable

Microsoft’s struggle with its controversial Recall feature continues. Recall is a “photographic memory” feature planned for Windows 11 which takes constant screenshots of a device, and makes it searchable. Its launch in June was a disaster, as Microsoft skipped basic security and privacy appro