Identity Blog Catcher

https://tools.simonwillison.net/extract-urls

Extract URLs

I found myself wanting to extract all of the underlying URLs that were linked to from a chunk of text on a web page. I realized the fastest way to do that would be to spin up an artifact that could accept rich-text HTML pastes and use an HTML parser to extract those links.

https://tools.simonwillison.net/clipboard-viewer

Clipboard viewer

Messing around with a tool that lets you paste in rich text reminded me that the browser clipboard API is a fascinating thing. I decided to build a quick debugging tool that would let me copy and paste different types of content (plain text, rich text, files, images etc) and see what information was available to me in the browser.

https://claude.site/artifacts/a3f85567-0afc-4854-b3d3-3746dd1a37f2

Pyodide REPL

I didn't put a lot of effort into this one. While poking around with Claude Artifacts in the browser DevTools I spotted this CSP header:

content-security-policy: default-src https://www.claudeusercontent.com; script-src 'unsafe-eval' 'unsafe-inline' https://www.claudeusercontent.com https://cdnjs.cloudflare.com https://cdn.jsdelivr.net/pyodide/; connect-src https://cdn.jsdelivr.net/pyodide/; worker-src https://www.claudeusercontent.com blob:; style-src 'unsafe-inline' https://www.claudeusercontent.com https://cdnjs.cloudflare.com https://fonts.googleapis.com; img-src blob: data: https://www.claudeusercontent.com; font-src data: https://www.claudeusercontent.com; object-src 'none'; base-uri https://www.claudeusercontent.com; form-action https://www.claudeusercontent.com; frame-ancestors https://www.claudeusercontent.com https://claude.ai https://preview.claude.ai https://claude.site https://feedback.anthropic.com; upgrade-insecure-requests; block-all-mixed-content

The https://cdn.jsdelivr.net/pyodide/ in there caught my eye, because it suggested that the Anthropic development team had deliberately set it up so Pyodide - Python compiled to WebAssembly - could be loaded in an artifact.

I got Claude to spin up a very quick demo to prove that this worked:

I've not bothered to extract this one to my own tools.simonwillison.net site yet because it's purely a proof of concept that Pyodide can load correctly in that environment.

Photo Camera Settings Simulator

I was out on a photo walk and got curious about whether or not JavaScript could provide a simulation of camera settings. I didn't get very far with this one (prompting on my phone while walking along the beach) - the result was buggy and unimpressive and I quickly lost interest. It did expose me to the Fabric.js library for manipulating canvas elements though.

https://claude.site/artifacts/e645c231-8c13-4374-bb7d-271c8dd73825

LLM pricing calculator

This one I did finish. I built this pricing calculator as part of my experiments with Video scraping using Google Gemini, because I didn't trust my own calculations for how inexpensive Gemini was! Here are detailed notes on how I built that.

https://tools.simonwillison.net/llm-prices

YAML to JSON converter

I wanted to remind myself how certain aspects of YAML syntax worked, so I span up a quick YAML to JSON converter tool that shows the equivalent JSON live as you type YAML.

https://claude.site/artifacts/ffeb439c-fc95-428a-9224-434f5f968d51

Claude transcript

OpenAI Audio

This is my most interesting artifact of the week. I was exploring OpenAI's new Audio APIs and decided to see if I could get Claude to build we a web page that could request access to my microphone, record a snippet of audio, then base64 encoded that and send it to the OpenAI API.

Here are the full details on how I built this tool.

https://tools.simonwillison.net/openai-audio

Claude Artifacts can't make API requests to external hosts directly, but it can still spin up enough of a working version that it's easy to take that, move it to different hosting and finish getting it working.

I wrote more about this API pattern in Building a tool showing how Gemini Pro can return bounding boxes for objects in images.

QR Code Decoder

I was in a meeting earlier this week where one of the participants shared a slide with a QR code (for joining a live survey tool). I didn't have my phone with me, so I needed a way to turn that QR code into a regular URL.

https://tools.simonwillison.net/qr

Knocking up this QR decoder in Claude Artifacts took just a few seconds:

Build an artifact (no react) that lets me paste in a QR code and displays the decoded information, with a hyperlink if necessary

[ ... ]

have a file open box that also lets you drag and drop and add a onpaste handler to the page that catches pasted images as well

Full conversation here.

Image Converter and Page Downloader

Another very quick prototype. On Hacker News someone demonstrated a neat idea for a tool that let you drop photos onto a page and it would bake them into the page as base64 URLs such that you could "save as HTML" and get a self-contained page with a gallery.

I suggested they could add a feature that generated a "Download link" with the new page baked in - useful on mobile phones that don't let you "Save as HTML" - and got Claude to knock up a quick prototype:

In this case I shared the code in a Gist and then used the new-to-me https://gistpreview.github.io/?GIST_ID_GOES_HERE trick to render the result:

https://gistpreview.github.io/?14a2c3ef508839f26377707dbf5dd329

gistpreview turns out to be a really quick way to turn a LLM-generated demo into a page people can view.

https://claude.site/artifacts/46897436-e06e-4ccc-b8f4-3df90c47f9bc

HTML Entity Escaper

Another example of on-demand software: I needed to escape the HTML entities in a chunk of text on my phone, so I got Claude to build me a tool for that:

Here's the prompt I used:

Build an artifact (no react) where I can paste text into a textarea and it will return that text with all HTML entities - single and double quotes and less than greater than ampersand - correctly escaped. The output should be in a textarea accompanied by a "Copy to clipboard" button which changes text to "Copied!" for 1.5s after you click it. Make it mobile friendly

text-wrap-balance-nav

Inspired by Terence Eden I decided to do a quick experiment with the text-wrap: balance CSS property. I got Claude to build me an example nav bar with a slider and a checkbox. I wrote about that here.

https://tools.simonwillison.net/text-wrap-balance-nav

ARES Phonetic Alphabet Converter

I was volunteering as a HAM radio communications operator for the Half Moon Bay Pumpkin Run and got nervous that I'd mess up using the phonetic alphabet - so I had Claude build me this tool:

https://claude.site/artifacts/aaadab20-968a-4291-8ce9-6435f6d53f4c

Claude transcript here. Amusingly it built it in Python first, then switched to JavaScript after I reminded it that I wanted "an interactive web app".

This is so useful, and so much fun!

As you can see, I'm a heavy user of this feature - I just described 14 projects produced in a single week. I've been using artifacts since they were released on 20th June (alongside the excellent Claude 3.5 Sonnet, still my daily-driver LLM) and I'm now at a point where I fire up a new interactive artifact several times a day.

I'm using artifacts for idle curiosity, rapid prototyping, library research and to spin up tools that solve immediate problems.

Most of these tools took less than five minutes to build. A few of the more involved ones took longer than that, but even the OpenAI Audio one took 11:55am to 12:07pm for the first version and 12:18pm to 12:27pm for the second iteration - so 21 minutes total.

Take a look at my claude-artifacts tag for even more examples, including SVG to JPG/PNG, Markdown and Math Live Renderer and Image resize and quality comparison.

I also have a dashboard of every post that links to my tools.simonwillison.net site, and the underlying simonw/tools GitHub repo includes more unlisted tools, most of which link to their Claude conversation transcripts in their commit history.

I'm beginning to get a little frustrated at their limitations - in particular the way artifacts are unable to make API calls, submit forms or even link out to other pages. I'll probably end up spinning up my own tiny artifacts alternative based on everything I've learned about them so far.

If you're not using artifacts, I hope I've given you a sense of why they're one of my current favourite LLM-based tools.

Tags: javascript, projects, tools, ai, generative-ai, llms, ai-assisted-programming, anthropic, claude, claude-artifacts, claude-3-5-sonnet

14:32 • a day ago

Ben Werdmüller

Online Safety and the “Great Decentralization” – The Perils and Promises of Federated Social Media

[Samantha Lai and Yoel Roth at Tech Policy Press] "Decentralized social media platforms offer the promise of alternative governance structures that empower consumers and rebuild social media on a foundation of trust. However, over two years after Elon Musk’s acquisition of Twitter sparked an exodus of users seeking new homes on the social web, federated platforms remain ill-e

[Samantha Lai and Yoel Roth at Tech Policy Press]

"Decentralized social media platforms offer the promise of alternative governance structures that empower consumers and rebuild social media on a foundation of trust. However, over two years after Elon Musk’s acquisition of Twitter sparked an exodus of users seeking new homes on the social web, federated platforms remain ill-equipped to meet the threats of abuse, harassment, coordinated manipulation, and spam that have plagued social media for years. Given the porous nature of decentralized services, these limitations will not just affect individual servers, but reverberate through the social web."

Most major decentralized and federated platforms don't have the necessary tooling "for scalable management of harmful content and conduct — or even the enforcement of their own rules."

For some, of course, this is by design: the same version of "free speech" which animates Elon Musk and in effect prevents speech from anyone except for in-groups and the loud and powerful. To have truly free speech - where people from vulnerable communities can have a voice and real debate can be held without threat of violence - there must be trust and safety and moderation.

The piece rightly calls out IFTAS for the great work it's doing in this area. More must be done - which in part means convincing federated communities that these ideas are important.

Unfortunately a common attitude is that "we don't have these problems" - a common refrain when your bias makes you blind to your lack of inclusion. As many Black users found when they joined Mastodon and were asked to hide the details of their lived experiences under content warnings, or when people told them that these were American-only experiences (which, of course, they aren't), a predominantly white and male Fediverse that seeks to maintain the status quo rather than learning and growing can be quite a conservative place.

This is an important piece, and an important finding, which everyone working on decentralized tech should pay attention to.

#Fediverse

[Link]

13:19 • a day ago

Michael Ruminer

Using LM Studio and CrewAI with Llama

I was recently experimenting with CrewAI, but in one experiment I wanted to use a local LLM hosted by LM Studio. Why LMStudio? Well… because that was the experiment. I wanted to see how I’d use LMStudio to host my model versus Ollama and then use it from CrewAI. Below is a very simple setup for doing this. It’s good to point out that CrewAI uses LiteLLM under the hood as a sort of proxy. Thi

It’s good to point out that CrewAI uses LiteLLM under the hood as a sort of proxy. This provides CrewAI with the ability to talk to a range of LLMs without needing to really do much in terms of handling it.

What Doesn’t Work and Why

The first thing I found is what doesn’t work. The LLM class in CrewAI allows for the instantiation of an LLM that can then be used by agents. Spinning up an LLM instance on a local Ollama hosted model can look like below.

ollama_31_8b = LLM(
model="ollama/llama3.1",
base_url="http://localhost:11434"
)

This works just fine if hosting the LLM inside Ollama, but you get response exceptions if, instead, you try to host inside LMStudio at the same server port.

ERROR: LiteLLM call failed: litellm.APIConnectionError: 'response'

First, you notice that the base_url doesn’t have a “/v1” at the end which LMStudio uses in their server setup. If you fix that, thinking it might work, you’ll find that you likely get the same error.

Secondly, you may realize that the model property in your LLM instantiation uses a [PROVIDER]/[MODEL] format. I tried removing the provider portion to see what would happen. The results were:

llm.py-llm:161 - ERROR: LiteLLM call failed: litellm.BadRequestError: LLM Provider NOT provided.

That’s a reasonable outcome.

What Does Work

Lastly, I remembered that LM Studio is using OpenAI endpoints.

A quick look at the LiteLLM docs provided the answer I needed; set the provider as “openai”. This results in a final outcome of:

ollama_31_8b = LLM(model="openai/llama3.1", base_url="http://localhost:11434/v1")

Now, if you try running it with the agent using the LLM instantiated it will work. Below is example code of the LLM and agent creation where I had Llama 3.1:8b model hosted in LM Studio on port 11434.

@CrewBase
class MyCrew():

llama_31_8b = LLM(
model="openai/llama3.1",
base_url="http://localhost:11434/v1"
)

@agent
def joke_generator(self) -> Agent:
return Agent(
config=self.agents_config['joke_generator'],
verbose=True,
llm=self.llama_31_8b
) Note

Note that on LMStudio I had my server port set to 11434 versus the default of 1234. It made it easier as I switched back and forth between Ollama and LM Studio; I didn’t need to modify the port. 11434 is the default Ollama port.

When Might I Use This

When might I use this? If I am programming, probably rarely. I could instead host the model in Ollama. I’d use LM Studio if I want to host a model and chat with it. In that scenario, I’d probably be more likely to use Ollama with AnythingLLM which would also provide me with some Retrieval-Augmented Generation (RAG) capabilities. Nonetheless, it was an experiment and I proved, for myself, it could easily be done.

12:34 • 2 days ago

John Philpin : Lifestream

2 of N I like to think of every organization being a magic

2 of N I like to think of every organization being a magic box. 3 inputs [I] : people, capital and tangible assets 2 outputs [0] : products and services The rest is just context. #nofN | 🔗 More

2 of N

I like to think of every organization being a magic box.

3 inputs [I] : people, capital and tangible assets
2 outputs [0] : products and services

The rest is just context.

#nofN | 🔗 More

10:30 • 2 days ago

I get it. I really do. But how do you KNOW?

10:16 • 2 days ago

Damien Bod

Implement security headers for an ASP.NET Core API supporting OpenAPI Swagger UI

This article shows how to implement security headers for an application supporting an API and a swagger UI created from a open API in .NET 9. The security headers are implemented using the NetEscapades.AspNetCore.SecurityHeaders Nuget packages from Andrew Lock. Code: https://github.com/damienbod/WebApiOpenApi Deploying a web application which supports both an API and a UI have different […]

Code: https://github.com/damienbod/WebApiOpenApi

Deploying a web application which supports both an API and a UI have different levels of security requirements. The Swagger UI is created from an Open API specification and uses inline Javascript with no hashes or nonces which requires weak security definitions. The API has no UI and can use the maximum security header definitions. It can be locked down as much as possible against the typical web UI attacks.

The API endpoints can be secured using a definition with strict security headers and a lot of browser features locked down.

public static class SecurityHeadersDefinitionsAPI { private static HeaderPolicyCollection? policy; public static HeaderPolicyCollection GetHeaderPolicyCollection(bool isDev) { // Avoid building a new HeaderPolicyCollection on every request for performance reasons. // Where possible, cache and reuse HeaderPolicyCollection instances. if (policy != null) return policy; policy = new HeaderPolicyCollection() .AddFrameOptionsDeny() .AddContentTypeOptionsNoSniff() .AddReferrerPolicyStrictOriginWhenCrossOrigin() .AddCrossOriginOpenerPolicy(builder => builder.SameOrigin()) .AddCrossOriginEmbedderPolicy(builder => builder.RequireCorp()) .AddCrossOriginResourcePolicy(builder => builder.SameOrigin()) .RemoveServerHeader() .AddPermissionsPolicyWithDefaultSecureDirectives(); policy.AddContentSecurityPolicy(builder => { builder.AddObjectSrc().None(); builder.AddBlockAllMixedContent(); builder.AddImgSrc().None(); builder.AddFormAction().None(); builder.AddFontSrc().None(); builder.AddStyleSrc().None(); builder.AddScriptSrc().None(); builder.AddBaseUri().Self(); builder.AddFrameAncestors().None(); builder.AddCustomDirective("require-trusted-types-for", "'script'"); }); if (!isDev) { // maxage = one year in seconds policy.AddStrictTransportSecurityMaxAgeIncludeSubDomains(maxAgeInSeconds: 60 * 60 * 24 * 365); } return policy; } }

The API calls would return the following headers in the HTTP response. The CSP blocks Javascript.

The Swagger definition allows unsafe Javascript. This allows for XSS attacks and is a weak level of security. This is required due to the way the Swagger UI is created.

public static class SecurityHeadersDefinitionsSwagger { private static HeaderPolicyCollection? policy; public static HeaderPolicyCollection GetHeaderPolicyCollection(bool isDev) { // Avoid building a new HeaderPolicyCollection on every request for performance reasons. // Where possible, cache and reuse HeaderPolicyCollection instances. if (policy != null) return policy; policy = new HeaderPolicyCollection() .AddFrameOptionsDeny() .AddContentTypeOptionsNoSniff() .AddReferrerPolicyStrictOriginWhenCrossOrigin() .AddCrossOriginOpenerPolicy(builder => builder.SameOrigin()) .AddCrossOriginEmbedderPolicy(builder => builder.RequireCorp()) .AddCrossOriginResourcePolicy(builder => builder.SameOrigin()) .RemoveServerHeader() .AddPermissionsPolicyWithDefaultSecureDirectives(); policy.AddContentSecurityPolicy(builder => { builder.AddObjectSrc().None(); builder.AddBlockAllMixedContent(); builder.AddImgSrc().Self().From("data:"); builder.AddFormAction().Self(); builder.AddFontSrc().Self(); builder.AddStyleSrc().Self().UnsafeInline(); builder.AddScriptSrc().Self().UnsafeInline(); //.WithNonce(); builder.AddBaseUri().Self(); builder.AddFrameAncestors().None(); }); if (!isDev) { // maxage = one year in seconds policy.AddStrictTransportSecurityMaxAgeIncludeSubDomains(maxAgeInSeconds: 60 * 60 * 24 * 365); } return policy; } }

The HTTP response would look something like this:

The security headers are setup to always use the API swagger definitions except for the swagger routes. This is used in development. I still don’t deploy this to production, unless the Swagger UI is absolutely required. In production, all responses use the stricter security header definitions.

// Open up security restrictions to allow this to work // Not recommended in production var deploySwaggerUI = builder.Configuration.GetValue<bool>("DeploySwaggerUI"); var isDev = builder.Environment.IsDevelopment(); builder.Services.AddSecurityHeaderPolicies() .SetPolicySelector((PolicySelectorContext ctx) => { // sum is weak security headers due to Swagger UI deployment // should only use in development if (deploySwaggerUI) { // Weakened security headers for Swagger UI if (ctx.HttpContext.Request.Path.StartsWithSegments("/swagger")) { return SecurityHeadersDefinitionsSwagger.GetHeaderPolicyCollection(isDev); } // Strict security headers return SecurityHeadersDefinitionsAPI.GetHeaderPolicyCollection(isDev); } // Strict security headers for production else { return SecurityHeadersDefinitionsAPI.GetHeaderPolicyCollection(isDev); } });

The security headers are added as middleware using the UseSecurityHeaders methods.

app.UseSecurityHeaders();

Notes

This setup works good and the correct headers for the API are used in both development with the Swagger UI or without the Swagger UI. No weaken headers are deployed to production.

Links

Improving application security in an ASP.NET Core API using HTTP headers – Part 3

https://securityheaders.com/

https://csp-evaluator.withgoogle.com/

Security by Default Chrome developers

A Simple Guide to COOP, COEP, CORP, and CORS

https://github.com/andrewlock/NetEscapades.AspNetCore.SecurityHeaders

https://scotthelme.co.uk/coop-and-coep/

https://github.com/OWASP/ASVS

07:26 • 2 days ago

John Philpin : Lifestream

Here’s something I just did in another post in the Micro Blo

Here’s something I just did in another post in the Micro Blog client and 🤯 Type a number like 7 and then type a maths symbol (say *) and then type another number (say 13) and finish of with ‘=’ 7 * 13 =91 Here’s the thing - ‘91’ just appeared automatically when I typed ‘=' I know that Apple were describing all this earlier this year - I just was amazed when it happened since I hadn’t reall

Here’s something I just did in another post in the Micro Blog client and 🤯

Type a number like 7 and then type a maths symbol (say *) and then type another number (say 13) and finish of with ‘=’

7 * 13 =91

Here’s the thing - ‘91’ just appeared automatically when I typed ‘='

I know that Apple were describing all this earlier this year - I just was amazed when it happened since I hadn’t really thought about it.

06:36 • 2 days ago

1 of N Occasionally a post on Daring Fireball reads someth

1 of N Occasionally a post on Daring Fireball reads something like this: After being sold out for months, the upcoming sponsorship schedule at DF is unusually open at the moment—especially this week! I don’t sell sponsorship, I sell me, but at the highest level we both trade in what we might call perishable inventory; A hotel room not sold on Monday night can’t be sold Tuesday. Jo

1 of N

Occasionally a post on Daring Fireball reads something like this:

After being sold out for months, the upcoming sponsorship schedule at DF is unusually open at the moment—especially this week!

I don’t sell sponsorship, I sell me, but at the highest level we both trade in what we might call perishable inventory;

A hotel room not sold on Monday night can’t be sold Tuesday.
John can’t sell this week’s sponsorship next week.
If I don’t bill today I can’t catch up tomorrow.

I’ll be back with some more thoughts, but for now, I’m just letting this lie.

#nofN | 🔗 More

05:58 • 2 days ago

Simon Willison

Dashboard: Tools

Dashboard: Tools I used Django SQL Dashboard to spin up a dashboard that shows all of the URLs to my tools.simonwillison.net site that I've shared on my blog so far. It uses this (Claude assisted) regular expression in a PostgreSQL SQL query: select distinct on (tool_url) unnest(regexp_matches( body, '(https://tools\.simonwillison\.net/[^<"\s)]+)', 'g' )) as

Dashboard: Tools

I used Django SQL Dashboard to spin up a dashboard that shows all of the URLs to my tools.simonwillison.net site that I've shared on my blog so far. It uses this (Claude assisted) regular expression in a PostgreSQL SQL query:

select distinct on (tool_url) unnest(regexp_matches( body, '(https://tools\.simonwillison\.net/[^<"\s)]+)', 'g' )) as tool_url, 'https://simonwillison.net/' || left(type, 1) || '/' || id as blog_url, title, date(created) as created from content

I've been really enjoying having a static hosting platform (it's GitHub Pages serving my simonw/tools repo) that I can use to quickly deploy little HTML+JavaScript interactive tools and demos.

Tags: django-sql-dashboard, ai-assisted-programming, tools, projects, postgresql, sql, javascript

03:33 • 2 days ago

Doc Searls Weblog

Comet, second try

Comet Tsuchinshan–ATLAS, flying toward its tail, away from the Sun.

I shot the comet this time with a real camera: my Sony a7iv with a FE 70-200 mm F2.8 GM OSS II lens set at f3.5 at 135mm for 10 seconds on a shitty tripod I got at a thrift shop for $5. (I have good ones elsewhere.) This was at 8:40pm, just as the moon was rising behind my back and before dusk had turned to night with plenty of unwanted light from street lamps and the other usual challenges.

I couldn’t see it, but my iPhone 16 Pro Max did:

Using other shots I maneuvered myself to a place where the camera was in shadow from street lights and I could put the camera and tripod on the sidewalk while locating the comet to the right of that evergreen tree, so it was easy to aim in the general direction and get the shot.

If it’s clear again tomorrow night, I’ll go find a darker place at a later time and see what I can do.

03:02 • 2 days ago

Sunday, 20. October 2024

John Philpin : Lifestream

🔗 You are the center of concentric circles, Mark Dykeman I

🔗 You are the center of concentric circles, Mark Dykeman It’s another one of those Dunbar posts, the concentric circles are so random that I was reduced to asking ChatGPT .. “What is the difference between a friend, good friends, best friends and close friends? A less than satisfactory answer, though it did seem to agree with me (and the author) that a ‘Best Friend’ is closer than a ‘Clo

🔗 You are the center of concentric circles, Mark Dykeman

It’s another one of those Dunbar posts, the concentric circles are so random that I was reduced to asking ChatGPT ..

“What is the difference between a friend, good friends, best friends and close friends?

A less than satisfactory answer, though it did seem to agree with me (and the author) that a ‘Best Friend’ is closer than a ‘Close Friend’. Other than that? Yeah - no.

And the numbers

5 close/best friends
10 more best / close friends
35 more good friends

Why did he bother writing the piece? Why have I bothered commenting on it?

23:40 • 2 days ago

📚 A Story

Not mine, but it was so lovely I wanted to drop it in here, so that as ‘link rot’ continues - I will at least have it here. Who knows, you might already know it, it seems to have been circulating the socials - so of course - I did a little bit of triangulation. It looks legit. 🔗 More Here The Story At 40, Franz Kafka (1883-1924), who never married and had no children, was walking through a p

🔗 More Here

The Story

At 40, Franz Kafka (1883-1924), who never married and had no children, was walking through a park one day in Berlin when he met a girl who was crying because she had lost her favourite doll. She and Kafka searched for the doll unsuccessfully.

Kafka told her to meet him there the next day and they would come back to look for her.

The next day, when they had not yet found the doll, Kafka gave the girl a letter “written” by the doll saying

“please don’t cry. I took a trip to see the world. I will write to you about my adventures.”

Thus began a story which continued until the end of Kafka’s life.

During their meetings, Kafka read the letters of the doll carefully written with adventures and conversations that the girl found adorable.

Finally, Kafka brought back the doll (he bought one) that had returned to Berlin.

“It doesn’t look like my doll at all,” said the girl.

Kafka handed her another letter in which the doll wrote:

“my travels have changed me.”

The little girl hugged the new doll and brought the doll with her to her happy home.

A year later Kafka died.

Many years later, the now-adult girl found a letter inside the doll. In the tiny letter signed by Kafka it was written:

“Everything you love will probably be lost, but in the end, love will return in another way.”

Embrace change. It’s inevitable for growth. Together we can shift pain into wonder and love, but it is up to us to consciously and intentionally create that connection.

23:26 • 2 days ago

Simon Willison

Knowledge Worker

Knowledge Worker Forrest Brazeal: Last month, I performed a 30-minute show called "Knowledge Worker" for the incredible audience at Gene Kim's ETLS in Las Vegas. The show included 7 songs about the past, present, and future of "knowledge work" - or, more specifically, how it's affecting us, the humans between keyboard and chair. I poured everything I've been thinking and feeling about AI fo

Knowledge Worker

Forrest Brazeal:

Last month, I performed a 30-minute show called "Knowledge Worker" for the incredible audience at Gene Kim's ETLS in Las Vegas.

The show included 7 songs about the past, present, and future of "knowledge work" - or, more specifically, how it's affecting us, the humans between keyboard and chair. I poured everything I've been thinking and feeling about AI for the last 2+ years into this show, and I feel a great sense of peace at having said what I meant to say.

Videos of all seven songs are included in the post, with accompanying liner notes. AGI (Artificial God Incarnate) is a banger, and What’s Left for Me? (The AI Existential Crisis Song) captures something I've been trying to think through for a while.

Via Matt Campbell

Tags: forrest-brazeal, ai, music

23:16 • 2 days ago

John Philpin : Lifestream

🎵🖇️ 18 - 1975 - Pete Wingfield. 🎵🔗 19 - 1996 - Another Gre

🎵🖇️ 18 - 1975 - Pete Wingfield. 🎵🔗 19 - 1996 - Another Great (though very different) Track - this one from Paul Hardcastle. Sometimes time goes slow.

🎵🖇️ 18 - 1975 - Pete Wingfield.

🎵🔗 19 - 1996 - Another Great (though very different) Track - this one from Paul Hardcastle.

Sometimes time goes slow.

23:10 • 2 days ago

Ben Werdmüller

Helping to build the open social web

As regular readers know, I care a lot about growing the open social web: the rapidly-growing decentralized network of interoperable social platforms that includes Mastodon, Threads, Ghost, Flipboard, and many other platforms, both emerging and established. This is for a few reasons, including but not limited to: Support for strong communities Support for niche interests and diversity: Small

Support for strong communities

Support for niche interests and diversity: Smaller, independent communities can flourish without the pressure to appeal to mass audiences, leading to richer, more diverse conversations and interactions. But these aren’t silos: any member from one community can easily follow someone from any other. Community-driven moderation: Instead of top-down moderation, communities set their own rules and guidelines, which can lead to healthier and more relevant interactions. Community health isn’t subject to a single corporation’s policies and motivations.

Better developer experience

An easier way to build social apps: Shared libraries, tools and protocols let developers get started faster. And developers no longer have to worry about their social products feeling empty: every new product can plug into communities of millions of people. Developer stability: Developers don’t need to ask anyone for permission to build on open social web protocols. Nobody will suddenly turn off the open social web and charge developers to access it: just like the web itself, it’s open and permissionless, forever. The result is a less risky playing field for new entrants.

Respect for users

Decentralized governance: Users have more control over their data, identity, and interactions, without reliance on a single corporation or platform. Freedom from corporate algorithms: No algorithm-driven feeds prioritize ads or engagement-maximizing content, allowing for more authentic and community-driven interaction (and significantly less election interference, for example). Data ownership and portability: Users have greater control over their data and are not at the mercy of corporate interests. The open social web has the potential to connect every social platform, allowing anyone to be in conversation. And users can move from provider to provider at any time without losing their communities. Reduced surveillance: Federated systems are often less focused on advertising and surveillance-based business models, reducing targeted ads and invasive data collection. A more ethical ecosystem: It’s far easier for developers to build ethical apps that don’t hold user data hostage.

I’d love to be more involved in helping it grow. Here are some ways I’ve thought about doing that. As always, I’d love to hear what you think.

Acting as an advocate between publishers and vendors.

Status: I’m already doing this informally.

Open social web vendors like Mastodon seem to want to understand the needs of news publishers; there are already lots of advantages for news publishers who join the open social web. There’s some need for a go-between to help both groups understand each other.

Publishers need to prove that there’s return on investment on getting involved in any social platform. Mastodon in particular has some analytics-hostile features, including preventing linked websites from knowing where traffic is coming from, and stripping the utm tags that audience teams use to analyze traffic. There’s also no great analytics dashboard and little integration with professional social media tools.

Meanwhile, the open social web already has a highly engaged, intelligent, action-oriented community of early adopters who care about the world around them and are willing to back news publishers they think are doing good work. I’ve done work to prove this, and have found that publishers can easily get more meaningful engagement (subscriptions, donations) on the open social web than on all closed social networks combined. That’s a huge advantage.

But both groups need to collaborate — and in the case of publishers, need to want to collaborate. There’s certainly work to do here.

Providing tertiary services.

Status: I built ShareOpenly, but there’s much more work to do.

There are a lot of ways a service provider could add value to the open social web.

Automattic, the commercial company behind WordPress, got its start by providing anti-spam services through a tool called Akismet. Automattic itself is unfortunately not a wonderful example to point to at this moment in time, but the model stands: take an open source product and make it more useful through add-ons.

There’s absolutely the need for anti-spam and moderation services on the open social web (which are already provided by Independent Federated Trust And Safety, which is a group that deserves to be better-funded).

My tiny contribution so far is ShareOpenly, a site that provides “share to …” buttons for websites that are inclusive of Mastodon and other Fediverse platforms. A few sites, like my own blog and Tedium, include ShareOpenly links on posts, and it’s been used to share to hundreds of Mastodon instances. (I don’t track links shared at all, so don’t have stats about that.) But, of course, it could be a lot bigger.

I think there’s potential in anti-spam services in particular: unlike trust and safety, they can largely be automated, and there’s a proven model with Akismet.

Rebuilding Known to support the Fediverse — or contributing to an existing Fediverse platform.

Status: I just need more time.

My publishing platform Known could be rewritten to have a new, faster, cleaner architecture that is Fediverse-first.

It’s not clear to me what the sustainability model is here: how can I make sure I continue to have the time and resources to work on it? But I do think there’s a lot of potential for it to be useful — particularly for individual bloggers and smaller publishers — once it was built.

And of course, there are many other open source Fediverse platforms (like Mastodon) that always need extra hands. The question remains: how can I find the time and resources to be able to make those contributions?

(I’ve already tried: funding as a startup, consultancy services, donations, and a paid hosting service. If you’ve got other ideas, I’d love to hear them!)

An API engine for the Fediverse

Status: idea only, but validated with both experts and potential customers. Would need to be funded.

ActivityPub, the underlying protocol underneath the Fediverse, can sometimes be hard to implement. Unlike many web apps, you often need to set up asynchronous queues and process data in potentially expensive ways when both publishing and reading data from other instances.

So why not abstract all of that away? Here smaller communities and experimental developers can rely on shared infrastructure that handles inboxes and queues automatically behind a simple RESTful API with SDKs in every modern language. Rather than have to build out all that infrastructure to begin with, developers can start with the Fediverse API, saving them a bunch of time and allowing them to focus on their unique idea.

It would start out with a free tier, allowing experimentation, and then scale up to affordable, use-based billing.

Add-on services could provide the aforementioned anti-spam, and there could be plugins from services like IFTAS in order to provide real human moderation for communities that need it.

Suddenly, developers can build a fully Fediverse-compatible app in an afternoon instead of in weeks or months, and know that they don’t need to be responsible for maintaining its underlying ActivityPub infrastructure.

A professional open social network (Fediverse VIP)

Status: idea only, but validated with domain experts.

A first-class social network with top-tier UX and UI design, particularly around onboarding and discovery, built explicitly to be part of the Fediverse. The aim is to be the destination for anyone who wants to join the Fediverse for professional purposes — or if they simply don’t know what other instance to join.

There is full active moderation and trust and safety for all users. Videos are supported out of the box. Images all receive automatic alt text generation by default (or you can specify your own). There is a first-class app across all mobile platforms, and live search for events, TV shows, sports, and so on. Posts can easily be embedded on third-party sites.

You can break out long-form posts from shorter posts, allowing you to read stories from Ghost and other platforms that publish long-form text to the Fediverse.

If publishers and brands join Fediverse VIP, profiles of their employees can be fully branded and be associated with their domains. A paid tier offers full analytics (in contrast in particular to Mastodon, which offers almost none) and scheduled posts, as well as advanced trust and safety features for journalists and other users from sensitive organizations. Publishers can opt to syndicate full-content feeds into the Fediverse. This becomes the best, safest, most feature-supported and brand-safe way for publishers to share with the hundreds of millions of Fediverse users.

Finally, an enterprise concierge tier allows Fediverse VIP to be deeply customized and integrated with any website or tool, for example to run Fediverse-aware experiments on their own sites, do data research (free for accredited academic institutions and non-profit newsrooms), build new tools that work with Fediverse VIP, or use live feeds of content on TV or at other events.

What do you think?

Those are some ideas I have. But I’m curious: what do you think would be most effective? Is this even an important goal?

I’d love to hear what you think.

23:01 • 2 days ago

John Philpin : Lifestream

🔗 🎵 A Trip Down Memory Lane With Pete Wingfield.

22:54 • 2 days ago

Last time I did this (next post) in Songwhip I had all kinds

Last time I did this (next post) in Songwhip I had all kinds of problems - but I didn’t know why 🎵 🔗 TIL Why

22:48 • 2 days ago

🎵 For me - a collection that totally outshines everything he

🎵 For me - a collection that totally outshines everything he did around and for the Pistols.

22:42 • 2 days ago

IdM Laboratory

Auth0 Labの生成AI向けの認証・認可のサンプルを試す

こんにちは、富士榮です。イベント続きでほぼ毎日プレゼンしている気がしますが、ストレスが溜まるので現実逃避です。 Auth0が生成AI向けの認証・認可に関するサイトをオープンしました。 https://www.auth0.ai/ まぁ、もともとOpenAIの認証はAuth0だったこともあり、この分野は以前から取り組んできていたんだと思います。生成AIの認証・認可といっても単純にチャットボットへのログインだけでは面白くないわけで、ユーザの代わりにAPIを読んだり、RAGの認証をしたり、ユーザの確認を非同期で行ったり、とやることはたくさんあります。この辺りをAuth0 Labでパッケージングしたサンプルを公開している、ということですね。 Auth0 Labのアカウントで先ほどのサイトのデモを試すと、ChatGPTもどきのアプリケーションが動きます。

こんにちは、富士榮です。

イベント続きでほぼ毎日プレゼンしている気がしますが、ストレスが溜まるので現実逃避です。

Auth0が生成AI向けの認証・認可に関するサイトをオープンしました。

https://www.auth0.ai/

まぁ、もともとOpenAIの認証はAuth0だったこともあり、この分野は以前から取り組んできていたんだと思います。

生成AIの認証・認可といっても単純にチャットボットへのログインだけでは面白くないわけで、ユーザの代わりにAPIを読んだり、RAGの認証をしたり、ユーザの確認を非同期で行ったり、とやることはたくさんあります。

この辺りをAuth0 Labでパッケージングしたサンプルを公開している、ということですね。

Auth0 Labのアカウントで先ほどのサイトのデモを試すと、ChatGPTもどきのアプリケーションが動きます。

なお、このソースコードはこちらのgithubレポジトリで公開されているので、自分のローカル環境でも試すことができます。

https://github.com/auth0-lab/market0

こういうフロントエンドとAPI管理周りは生成AIのエンジンとは独立したレイヤですが、自前で作るのは面倒な領域なのでこういうものがあると便利ですね。

22:30 • 2 days ago

Simon Willison

Quoting John Gruber

I really dislike the practice of replacing passwords with email “magic links”. Autofilling a password from my keychain happens instantly; getting a magic link from email can take minutes sometimes, and even in the fastest case, it’s nowhere near instantaneous. Replacing something very fast — password autofill — with something slower is just a terrible idea. — John Gruber Tags: passwords

I really dislike the practice of replacing passwords with email “magic links”. Autofilling a password from my keychain happens instantly; getting a magic link from email can take minutes sometimes, and even in the fastest case, it’s nowhere near instantaneous. Replacing something very fast — password autofill — with something slower is just a terrible idea.

— John Gruber

Tags: passwords, security, john-gruber

22:17 • 2 days ago

The 3 AI Use Cases: Gods, Interns, and Cogs

The 3 AI Use Cases: Gods, Interns, and Cogs Drew Breunig introduces an interesting new framework for categorizing use cases of modern AI: Gods refers to the autonomous, human replacement applications - I see that as AGI stuff that's still effectively science fiction. Interns are supervised copilots. This is how I get most of the value out of LLMs at the moment, delegating tasks to them that

The 3 AI Use Cases: Gods, Interns, and Cogs

Drew Breunig introduces an interesting new framework for categorizing use cases of modern AI:

Gods refers to the autonomous, human replacement applications - I see that as AGI stuff that's still effectively science fiction. Interns are supervised copilots. This is how I get most of the value out of LLMs at the moment, delegating tasks to them that I can then review, such as AI-assisted programming. Cogs are the smaller, more reliable components that you can build pipelines and automations on top of without needing to review everything they do - think Whisper for transcriptions or maybe some limited LLM subtasks such as structured data extraction.

Drew also considers Toys as a subcategory of Interns: things like image generators, “defined by their usage by non-experts. Toys have a high tolerance for errors because they’re not being relied on for much beyond entertainment.”

Tags: drew-breunig, ai-assisted-programming, generative-ai, ai, llms

22:12 • 2 days ago

Quoting Jens Ohlig

Who called it “intellectual property problems around the acquisition of training data for Large Language Models” and not Grand Theft Autocomplete? — Jens Ohlig, on March 8th 2024 Tags: training-data, llms, ai, generative-ai

Who called it “intellectual property problems around the acquisition of training data for Large Language Models” and not Grand Theft Autocomplete?

— Jens Ohlig, on March 8th 2024

Tags: training-data, llms, ai, generative-ai

21:02 • 2 days ago

John Philpin : Lifestream

Always fascinated by the never ending models of customer eng

Always fascinated by the never ending models of customer engagement that the internet keeps spitting out. And everytime I see one, I hear George whispering into my ear All models are wrong .. some are useful. Which is actually right when you think about it. I won’t name this ‘6 step model’, but my assumption is that it was created by someone who got half way through the ‘12 steps’ .. I mea

Always fascinated by the never ending models of customer engagement that the internet keeps spitting out. And everytime I see one, I hear George whispering into my ear

All models are wrong .. some are useful.

Which is actually right when you think about it. I won’t name this ‘6 step model’, but my assumption is that it was created by someone who got half way through the ‘12 steps’ .. I mean, when one of your steps is ‘Build Rapport’, you do wonder who exactly these sales people are that need this kind of guidance.

18:46 • 3 days ago

Simon Willison

Quoting Jacob Kaplan-Moss

It feels like we’re at a bit of an inflection point for the Django community. [...] One of the places someone could have the most impact is by serving on the DSF Board. Like the community at large, the DSF is at a transition point: we’re outgrowing the “small nonprofit” status, and have the opportunity to really expand our ambition and reach. In all likelihood, the decisions the Board makes over

It feels like we’re at a bit of an inflection point for the Django community. [...] One of the places someone could have the most impact is by serving on the DSF Board. Like the community at large, the DSF is at a transition point: we’re outgrowing the “small nonprofit” status, and have the opportunity to really expand our ambition and reach. In all likelihood, the decisions the Board makes over the next year or two will define our direction and strategy for the next decade.

— Jacob Kaplan-Moss

Tags: jacob-kaplan-moss, django, dsf

16:34 • 3 days ago

You can use text-wrap: balance; on icons

You can use text-wrap: balance; on icons Neat CSS experiment from Terence Eden: the new text-wrap: balance CSS property is intended to help make text like headlines display without ugly wrapped single orphan words, but Terence points out it can be used for icons too: This inspired me to investigate if the same technique could work for text based navigation elements. I used Claude to build th

You can use text-wrap: balance; on icons

Neat CSS experiment from Terence Eden: the new text-wrap: balance CSS property is intended to help make text like headlines display without ugly wrapped single orphan words, but Terence points out it can be used for icons too:

This inspired me to investigate if the same technique could work for text based navigation elements. I used Claude to build this interactive prototype of a navigation bar that uses text-wrap: balance against a list of display: inline menu list items. It seems to work well!

My first attempt used display: inline-block which worked in Safari but failed in Firefox.

Notable limitation from that MDN article:

Because counting characters and balancing them across multiple lines is computationally expensive, this value is only supported for blocks of text spanning a limited number of lines (six or less for Chromium and ten or less for Firefox)

So it's fine for these navigation concepts but isn't something you can use for body text.

Tags: css, claude-artifacts, anthropic, ai-assisted-programming, claude, terence-eden

13:23 • 3 days ago

Ben Werdmüller

US startup charging couples to ‘screen embryos for IQ’

[Hannah Devlin, Tom Burgis, David Pegg and Jason Wilson at The Guardian] Quite a disturbing new startup coming to light in The Guardian: “The footage appears to show experimental genetic selection techniques being advertised to prospective parents. A Heliospect employee, who has been helping the company recruit clients, outlined how couples could rank up to 100 embryos based

[Hannah Devlin, Tom Burgis, David Pegg and Jason Wilson at The Guardian]

Quite a disturbing new startup coming to light in The Guardian:

“The footage appears to show experimental genetic selection techniques being advertised to prospective parents. A Heliospect employee, who has been helping the company recruit clients, outlined how couples could rank up to 100 embryos based on “IQ and the other naughty traits that everybody wants”, including sex, height, risk of obesity and risk of mental illness.”

Eugenics is a discredited, troubling idea, and the startup’s claims are akin to junk science, even if the underlying data was drawn from UK Biobank, which seems like a great resource when used for good. Still, the startup is clearly out there offering its services, while using a regulatory arbitrage strategy (operating between jurisdictions to exploit legal differences and finding ways to exploit loopholes in the law) that isn’t a million miles away from techniques used by startups like Uber, and throwing up all kinds of ethical questions in the process.

A major figure in the startup is Jonathan Anomaly (his real name), who has been advocating for “liberal eugenics” for some time:

“Anomaly is a well-known figure in a growing transatlantic movement that promotes development of genetic selection and enhancement tools, which he says should not be confused with coercive state-sponsored eugenics. “All we mean by [liberal eugenics] is that parents should be free and maybe even encouraged to use technology to improve their children’s prospects once it’s available,” he told the podcast The Dissenter.”

Of course, eugenics isn’t controversial or unethical solely when it’s forcibly done by the government. As the article notes:

“Katie Hasson, associate director of the Center for Genetics and Society, in California, said: “One of the biggest problems is that it normalises this idea of ‘superior’ and ‘inferior’ genetics.” The rollout of such technologies, she said, “reinforces the belief that inequality comes from biology rather than social causes”.”

Enough ink has been spilled on science fiction stories that describe the effects of exactly this startup’s mission that the founders should have understood they were building a biotech torment nexus: something that was described in fiction as a technology that must never be built for the good of humanity, lest we fall victim to both intended and unintended consequences. Regardless, if someone can build it, they eventually will, and here we are.

There’s a related ethical question raised here, which related to who, exactly, should have access to biological research data. It turns out that UK Biobank also gave access to its database to a race science group. Should it have? Or should there be ethical safeguards on these databases? I’m more inclined to say that knowledge bases should be as open access as possible, but the implications for use by eugenicists and racist groups are pretty dire.

https://datatracker.ietf.org/doc/draft-ietf-oauth-selective-disclosure-jwt/

[Link]

12:18 • 3 days ago

John Philpin : Lifestream

“Stories are developed – not wordsmithed.” 💬 Mike Mapl

“Stories are developed – not wordsmithed.” 💬 Mike Maples

“Stories are developed – not wordsmithed.”

💬 Mike Maples

03:44 • 3 days ago

I got to the end of the internet. Or at least this little co

I got to the end of the internet. Or at least this little corner of it …

03:44 • 3 days ago

It’s A Trap.

03:05 • 3 days ago

Saturday, 19. October 2024

IdM Laboratory

SD-JWT draft 13がリリース

こんにちは、富士榮です。 SD-JWTのdraft 13が10/18に発行されています。 https://datatracker.ietf.org/doc/draft-ietf-oauth-selective-disclosure-jwt/ draft12→13での変更点はこの辺り。 -13 * WGLC (part 1) updates * Rewrote introduction * Added note on algorithm for Holder's verification of the SD-JWT これまでは用途は問わない、としていましたが今回からJWTの選択的情報開示がプライマリユースケースだと明記されています This specification defines a mechanism for the

こんにちは、富士榮です。

SD-JWTのdraft 13が10/18に発行されています。

draft12→13での変更点はこの辺り。

-13 * WGLC (part 1) updates * Rewrote introduction * Added note on algorithm for Holder's verification of the SD-JWT

これまでは用途は問わない、としていましたが今回からJWTの選択的情報開示がプライマリユースケースだと明記されています

This specification defines a mechanism for the selective disclosure of individual elements of a JSON-encoded data structure used as the payload of a JSON Web Signature (JWS). The primary use case is the selective disclosure of JSON Web Token (JWT) claims.

イントロダクションが全面的に書き換えられています明示的な型付けの部分でtypに加えてペイロードのコンテンツタイプ（cty）に関する記述が追加された

Use of the cty content type header parameter to indicate the content type of the SD-JWT payload can also be used to distinguish different types of JSON objects, or different kinds of JWT Claim Sets.

このくらいかな、と。

22:30 • 3 days ago

John Philpin : Lifestream

“First they ignore you. Then they ridicule you. And then t

“First they ignore you. Then they ridicule you. And then they attack you and want to burn you. And then they build monuments to you.” 💬 Nicolas Klein (🔗 not Gandhi)

“First they ignore you. Then they ridicule you. And then they attack you and want to burn you. And then they build monuments to you.”

💬 Nicolas Klein

(🔗 not Gandhi)

22:26 • 3 days ago

“Happiness is when what you think, what you say, and what

“Happiness is when what you think, what you say, and what you do are in harmony.” 💬 Gandhi

“Happiness is when what you think, what you say, and what you do are in harmony.”

💬 Gandhi

22:23 • 3 days ago

As a Brit living in New Zealand .. am I conflicted? Not at a

As a Brit living in New Zealand .. am I conflicted? Not at all. To be honest I barely knew that it all started up again. Not to take away from Aotearoa .. not just a win, but a very substantial win … for the third time … in a row. Massive Congratulations.

As a Brit living in New Zealand .. am I conflicted? Not at all. To be honest I barely knew that it all started up again.

Not to take away from Aotearoa .. not just a win, but a very substantial win … for the third time … in a row.

Massive Congratulations.

19:10 • 3 days ago

Simon Willison

Running Llama 3.2 Vision and Phi-3.5 Vision on a Mac with mistral.rs

mistral.rs is an LLM inference library written in Rust by Eric Buehler. Today I figured out how to use it to run the Llama 3.2 Vision and Phi-3.5 Vision models on my Mac. Despite the name, it's not just for the Mistral family of models - like how llama.cpp has grown beyond Llama, mistral.rs has grown beyond Mistral. I already have a Rust installation, so I checked out and compiled the library

mistral.rs is an LLM inference library written in Rust by Eric Buehler. Today I figured out how to use it to run the Llama 3.2 Vision and Phi-3.5 Vision models on my Mac.

Despite the name, it's not just for the Mistral family of models - like how llama.cpp has grown beyond Llama, mistral.rs has grown beyond Mistral.

I already have a Rust installation, so I checked out and compiled the library like this:

git clone https://github.com/EricLBuehler/mistral.rs.git
cd mistral.rs
cargo run --features metal --release -- -i --isq Q4K vision-plain -m lamm-mit/Cephalo-Llama-3.2-11B-Vision-Instruct-128k -a vllama

This downloaded around 20GB of model files to ~/.cache/huggingface/hub/models--lamm-mit--Cephalo-Llama-3.2-11B-Vision-Instruct-128k, loaded the model into memory and started a terminal-based chat session.

I ran this prompt:

\image https://niche-museums.imgix.net/pioneer-history.jpeg?w=1600&h=800&fit=crop&auto=compress <|image|> describe this image including any text

The prompt uses this image from my Niche Museums write-up of the Pioneer Memorial Museum:

I got back this response:

The image shows the exterior of the Daughters of Utah Pioneers (DUP) Museum, which is also known as the Pioneer Memorial Museum. The building is located in Salt Lake City, Utah, USA.

Visible in the image are:

The DUP Museum Building: The main structure of the museum is a prominent feature in the image. It is a two-story building with a rectangular shape and a flat roof. The exterior walls are made of brick or stone, giving it a classic and historic appearance.

Signage: Above the entrance, there is a sign that reads "Daughters of Utah Pioneers Museum" in bold letters. Below this sign, another sign indicates that the building is also known as the "Pioneer Memorial Museum."

Entrance: The main entrance to the museum is marked by a doorway with a decorative frame and a pair of large wooden doors.

Landscaping: The surrounding area around the museum appears well-maintained with neatly trimmed grass and some trees nearby.

Statue or Monument: In front of the museum, there is a small statue or monument that appears to be dedicated to pioneers or early settlers.

The Daughters of Utah Pioneers (DUP) Museum showcases historical artifacts and exhibits related to Utah's pioneer history, focusing on the experiences and contributions of early settlers in the region.

Note that the text it generates correctly describes the museum's content and identifies the museum as being located in Salt Lake City, neither of which are details that were displayed in the image itself.

Activity Monitor confirmed that this Metal build was using my GPU, and about 10GB of memory, while the prompt was running.

mistral.rs also provides a Python binding, which can be installed from PyPI but requires a Rust toolchain (I filed an issue requesting PyPI wheels):

pip install mistralrs-metal

I got an error running the Python Llama vision demo but the Phi 3.5 Vision example worked as advertised. I modified that to enable streaming output and ran it against a photograph from Mendenhall's Museum of Gasoline Pumps & Petroliana:

from mistralrs import Runner, Which, ChatCompletionRequest, VisionArchitecture res = runner.send_chat_completion_request( ChatCompletionRequest( model="phi3v", messages=[ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://niche-museums.imgix.net/mendenhalls-16.jpeg?w=1200&auto=compress" }, }, { "type": "text", "text": "<|image_1|>\nWhat is shown in this image? Write a detailed response analyzing the scene.", }, ], } ], max_tokens=1024, presence_penalty=1.0, top_p=0.1, temperature=0.1, stream=True, ) ) for item in res: print(item.choices[0].delta.content, end='')

Here's what It gave me:

The image captures a nostalgic scene of a vintage gas station, reminiscent of a bygone era. Dominating the left side of the frame is a red and white gas pump, standing tall and proud, its metallic surface gleaming under the soft light filtering through the wooden ceiling. Adjacent to it, a white and black gas canister stands ready for service, its presence adding to the authenticity of the setting.

On the right, a red and white gasoline sign hangs from the ceiling, its bold colors contrasting with the muted tones of the surroundings. Above it, a yellow and white gasoline sign is suspended, its vibrant hues drawing the eye upwards.

The floor beneath these relics of the past is a checkerboard pattern, a common design choice for gas stations of yesteryears. It provides a stark contrast to the smooth, unblemished surfaces of the gas pumps and canisters.

In the background, a variety of other signs and advertisements add to the eclectic mix of objects. They are a testament to the diverse range of products and services that were once available at this location.

Despite the passage of time, the gas station retains a certain charm, its vintage gas pumps and signs serving as a tangible link to a different era. The image is a snapshot of history, frozen in time, waiting to be discovered and appreciated by those who take the time to look closer.

This description looks fantastic at first glance, but if you review it carefully and compare it to the image you'll see that it's full of inaccuracies. The vibes of the description match the image but the actual details are definitely incorrect.

This model downloaded 7.7GB to ~/.cache/huggingface/hub/models--microsoft--Phi-3.5-vision-instruct - significantly smaller than Llama 3.2's 20GB. I wonder if that size difference helps explain the greater hallucination rate in Phi-3.5 Vision.

If you're running Python 3.10 on Apple Silicon you may be able to skip the Rust compiler by installing the wheel I built here:

pip install https://static.simonwillison.net/static/2024/mistralrs_metal-0.3.1-cp310-cp310-macosx_11_0_arm64.whl

Tags: microsoft, python, ai, rust, generative-ai, llama, llms, mistral, phi, vision-llms, meta

16:14 • 4 days ago

Ben Werdmüller

Republicans, young adults trust news on social media about as much as news from national outlets

[Kirsten Eddy at Pew Research Center] The lede is a little buried here behind some pretty shocking top-line stats: "Today, 37% of Republicans and Republican-leaning independents say they have a lot of or some trust in the information that comes from social media sites. This is nearly on par with the 40% of Republicans who express this level of trust in national news organiza

[Kirsten Eddy at Pew Research Center]

The lede is a little buried here behind some pretty shocking top-line stats:

"Today, 37% of Republicans and Republican-leaning independents say they have a lot of or some trust in the information that comes from social media sites. This is nearly on par with the 40% of Republicans who express this level of trust in national news organizations."
"[...] Adults under 30 are now nearly as likely to have a lot of or some trust in the information that comes from social media sites (52%) as from national news organizations (56%)."

Okay, but what's fascinating is that both groups trust local news outlets a great deal more. These have been systemically underfunded and are often run on a shoestring, but there's something about the local voice that really matters.

My suspicion - which is really just a hunch, so take it with a pinch of salt - is that it's because local news outlets don't tend to deal as much with abstract partisan politics. They're not going to comment on what Trump said now, or perceived shortcomings in the Harris campaign.

But, of course, local politics really matters. So it's interesting to think about what might happen if there's more investment in the space - something that initiatives like Tiny News Collective, the American Journalism Project and The Lenfest Institute are already thinking hard about. We need diverse, mission-driven outlets like Open Vallejo and Mission Local to spring up across the country.

My question as a technologist is how platforms, and more pointedly, open protocols can support these newsrooms. How can technology help great local journalists find the reach and make the impact they need, on their terms? And how can journalists, technologists, and product thinkers work together to shine a light on local politics and improve life in communities across the country?

Cory Doctorow discusses how he reads writers like Molly White:

[Link]

13:41 • 4 days ago

You should be using an RSS reader

[Cory Doctorow] Cory Doctorow discusses how he reads writers like Molly White: "This conduit is anti-lock-in, it works for nearly the whole internet. It is surveillance-resistant, far more accessible than the web or any mobile app interface. It is my secret super-power." I agree. I start every day in my RSS reader (I maintain a very simple live list of my subscriptions ove

[Cory Doctorow]

"This conduit is anti-lock-in, it works for nearly the whole internet. It is surveillance-resistant, far more accessible than the web or any mobile app interface. It is my secret super-power."

I agree. I start every day in my RSS reader (I maintain a very simple live list of my subscriptions over here) and it's one of the best tools I use. I rarely miss a news story from a publisher I care about - whether that's a newsroom, an individual, or an organization. And nobody's getting in the way to try and predict what I should be interested in.

RSS is free, open, well-established, and easy to use. More people should be using it. Even you.

[Abrahm Lustgarten at ProPublica]

[Link]

13:12 • 4 days ago

Far-Right Extremists Embrace Environmentalism to Justify Violent Anti-Immigrant Beliefs

[Abrahm Lustgarten at ProPublica] "For a generation, conservatives — not just the far right, which Crusius appeared to identify with — had propelled the notion that climate change was a hoax fabricated so the government could impose new restrictions on the economy and society. Yet Crusius hadn’t denied climate change at all. Instead, he seemed to claim its impacts were themse

"For a generation, conservatives — not just the far right, which Crusius appeared to identify with — had propelled the notion that climate change was a hoax fabricated so the government could impose new restrictions on the economy and society. Yet Crusius hadn’t denied climate change at all. Instead, he seemed to claim its impacts were themselves arguments justifying his violence."

Abrahm Lustgarten has written a sobering piece about the far right's embrace of climate change as a root for anti-immigrant and eugenicist sentiments. We can see this playing out among conservative groups across the country: in a world where resources are becoming more scarce, preserving "white European ideals and beliefs" becomes a part of "preserving the purity of [America's] ecology".

Ecofascism has been with us for a long time, and unfortunately has long been a subset of climate movements. But as the planet heats up and climate discussions become less hypothetical and more immediate, these conversations are becoming louder, too.

#Climate

[Link]

12:35 • 4 days ago

John Philpin : Lifestream

I’m thinking my next iPhone might be the new iPad mini.

02:30 • 4 days ago

Here’s a very noticeable benefit of Arc over Safari - IF you

Here’s a very noticeable benefit of Arc over Safari - IF you use 1Password - OMG - it is slick, fast, seamless and just a plain joy to work with. I had always assumed that ‘this;’ is just the way it is. Apparently not - it’s a Safari issue.

Apparently not - it’s a Safari issue.

01:58 • 4 days ago

🔗 An attempt at humo(u)r over on Substack … by yours truly.

🔗 An attempt at humo(u)r over on Substack … by yours truly. I didn’t despatch it to the subscriber list - just experimenting with various engagement ideas over there.

01:28 • 4 days ago

Friday, 18. October 2024

Ben Werdmüller

My solar-powered and self-hosted website

[Dries Buytaert] "I'm excited to share an experiment I've been working on: a solar-powered, self-hosted website running on a Raspberry Pi." Lovely! The key seems to be a Voltaic 50-watt panel and 18 amp-hour battery, which run to around $300 in total. That's not a lot of money for something that can theoretically run in perpetuity. I've been wanting to make my own website

[Dries Buytaert]

"I'm excited to share an experiment I've been working on: a solar-powered, self-hosted website running on a Raspberry Pi."

Lovely!

The key seems to be a Voltaic 50-watt panel and 18 amp-hour battery, which run to around $300 in total. That's not a lot of money for something that can theoretically run in perpetuity.

I've been wanting to make my own website run on fully green energy for a long time, and it's hard to find a web host that does this directly rather than through trading carbon credits, which I'm deeply suspicious of. (The exception is Iceland, where geothermal energy is common.)

I wonder what it would take to productize something like this and make it an all-in-one home server solution? Or to put your wifi router and modem on solar? (Assuming your whole house isn't on solar, that is, which mine sadly isn't.)

This also seems fair:

"It may seem unconventional, but I believe it's worth considering: many websites, mine included, aren't mission-critical. The world won't end if they occasionally go offline. That is why I like the idea of hosting my 10,000 photos on a solar-powered Raspberry Pi."

I feel the same way.

https://openid.net/announcing-ipsie-working-group/

[Link]

23:40 • 4 days ago

IdM Laboratory

IPSIE WGが爆誕（企業向けのIAMプロファイル策定に向けたWG）

こんにちは、富士榮です。ちょっと前に某MLで話題になっていたIPSIE（イプシー） WG（Interoperability Profiling for Secure Identity in the Enterprise Working Group）がOpenID Foundationに爆誕しています。 https://openid.net/announcing-ipsie-working-group/ Identity and Access Management (IAM) within the enterprise is a multifaceted endeavor, as indicated by the growing Body of Knowledge maintained by IDPro. There is a broad range of specif

こんにちは、富士榮です。

ちょっと前に某MLで話題になっていたIPSIE（イプシー） WG（Interoperability Profiling for Secure Identity in the Enterprise Working Group）がOpenID Foundationに爆誕しています。

Identity and Access Management (IAM) within the enterprise is a multifaceted endeavor, as indicated by the growing Body of Knowledge maintained by IDPro. There is a broad range of specifications that are relevant to securing the many IAM functions that underpin operations. Some of these are OIDF standards - like OpenID Connect, FAPI, and Shared Signals - while others are maintained in different standards bodies. For example, IPSIE has already identified the IETF's OAuth 2.0 and System for Cross-Domain Identity Management (SCIM) as relevant to their initial scope (below). But these specifications are written to support many contexts and use cases; they contain optionality that reduces the likelihood that independent implementations will interoperate.
The IPSIE Working Group will develop secure-by-design profiles of these existing specifications with a primary goal of achieving interoperability across enterprise implementations.

企業内のアイデンティティとアクセス管理（IAM）は多面的な取り組みである。運用を支える多くの IAM 機能のセキュリティ確保に関連する幅広い仕様がある。OpenID Connect、FAPI、Shared Signalsなどの OIDF 標準もあれば、別の標準化団体で維持されているものもある。例えば、IPSIEはすでにIETFのOAuth 2.0と System for Cross-Domain Identity Management (SCIM)を初期スコープに関連するものとして特定している（下記）。しかし、これらの仕様は多くのコンテキストやユースケースをサポートするように書かれており、独立した実装が相互運用できる可能性を低くするオプション性を含んでいる。
IPSIE ワーキンググループは、企業実装間の相互運用性を達成することを主な目的として、これら既存の仕様のセキュアバイデザインプロファイルを開発する。

名前の通り、エンタープライズシナリオにおける各種仕様のIDPro/BoKのベストプラクティスをプロファイルとしてまとめていきましょう！というプロファイルですね。（プロファイル＝プロトコルや仕様の組み合わせ。ここでいうとOpenID Connect、FAPI、OAuthやSCIMなど）

これはいよいよCIDProが流行る時代が来るのかもしれません。

そういえば最近メンテナンスできていませんが、IDProのBody of Knowledgeの日本語化プロジェクトもありますので、ご興味のある方はお声がけください。（一応、公認日本語化プロジェクトです）

https://idpro.jp/

最新化したいんですが、マンパワー不足です。

22:30 • 4 days ago

Ben Werdmüller

Underrepresented journalists most impacted by layoffs, says new report

[James Salanga at The Objective] "Layoffs in journalism since 2022 have disproportionately impacted people of marginalized genders and people of color, according to a new report from the Institute of Independent Journalists (IIJ). It collects data from a survey with 176 journalist respondents who had undergone a layoff or buyout since 2022." This mirrors the impact of layof

[James Salanga at The Objective]

"Layoffs in journalism since 2022 have disproportionately impacted people of marginalized genders and people of color, according to a new report from the Institute of Independent Journalists (IIJ). It collects data from a survey with 176 journalist respondents who had undergone a layoff or buyout since 2022."

This mirrors the impact of layoffs in tech, and likely other industries. In 2023, Prism reported that:

"Recent surveys have found that women, people of color, disabled workers, and other marginalized groups are disproportionately affected by mass layoffs in tech despite being underrepresented in the industry. According to Harvard Business Review, companies rely heavily on position and tenure when deciding on cuts, which translates to wiping out “most or all of the gains they’ve made in diversity.”"

This is damning in itself, but also suggests that many diversity gains were in positions closer to entry level than management level.

The irony for journalism is that it's the diverse members of newsrooms who can help them find broader audiences by ensuring that diverse perspectives are represented both in coverage and in management decisions. For a declining industry, it's a self-sabotaging thing to do. But, again, it says a lot about the demographics of the people who make the decisions.

tools.simonwillison.net/openai-audio

[Link]

22:10 • 4 days ago

Simon Willison

Experimenting with audio input and output for the OpenAI Chat Completion API

OpenAI promised this at DevDay a few weeks ago and now it's here: their Chat Completion API can now accept audio as input and return it as output. OpenAI still recommend their WebSocket-based Realtime API for audio tasks, but the Chat Completion API is a whole lot easier to write code against. Generating audio Audio input via a Bash script A web app for recording and prompting agains

Generating audio Audio input via a Bash script A web app for recording and prompting against audio The problem is the price Generating audio

For the moment you need to use the new gpt-4o-audio-preview model. OpenAI tweeted this example:

curl https://api.openai.com/v1/chat/completions \ -H "Authorization: Bearer $OPENAI_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "gpt-4o-audio-preview", "modalities": ["text", "audio"], "audio": { "voice": "alloy", "format": "wav" }, "messages": [ { "role": "user", "content": "Recite a haiku about zeros and ones." } ] }' | jq > response.json

I tried running that and got back JSON with a HUGE base64 encoded block in it:

{ "id": "chatcmpl-AJaIpDBFpLleTUwQJefzs1JJE5p5g", "object": "chat.completion", "created": 1729231143, "model": "gpt-4o-audio-preview-2024-10-01", "choices": [ { "index": 0, "message": { "role": "assistant", "content": null, "refusal": null, "audio": { "id": "audio_6711f92b13a081908e8f3b61bf18b3f3", "data": "UklGRsZr...AA==", "expires_at": 1729234747, "transcript": "Digits intertwine, \nIn dance of noughts and unity, \nCode's whispers breathe life." } }, "finish_reason": "stop", "internal_metrics": [] } ], "usage": { "prompt_tokens": 17, "completion_tokens": 181, "total_tokens": 198, "prompt_tokens_details": { "cached_tokens": 0, "cached_tokens_internal": 0, "text_tokens": 17, "image_tokens": 0, "audio_tokens": 0 }, "completion_tokens_details": { "reasoning_tokens": 0, "text_tokens": 33, "audio_tokens": 148 } }, "system_fingerprint": "fp_6e2d124157" }

The full response is here - I've truncated that data field since the whole thing is 463KB long!

Next I used jq and base64 to save the decoded audio to a file:

cat response.json | jq -r '.choices[0].message.audio.data' \ | base64 -D > decoded.wav

That gave me a 7 second, 347K WAV file. I converted that to MP3 with the help of llm cmd and ffmpeg:

llm cmd ffmpeg convert decoded.wav to code-whispers.mp3 > ffmpeg -i decoded.wav -acodec libmp3lame -b:a 128k code-whispers.mp3

That gave me a 117K MP3 file.

Your browser does not support the audio element.

The "usage" field above shows that the output used 148 audio tokens. OpenAI's pricing page says audio output tokens are $200/million, so I plugged that into my LLM pricing calculator and got back a cost of 2.96 cents.

Audio input via a Bash script

Next I decided to try the audio input feature. You can now embed base64 encoded WAV files in the list of messages you send to the model, similar to how image inputs work.

I started by pasting a curl example of audio input into Claude and getting it to write me a Bash script wrapper. Here's the full audio-prompt.sh script. The part that does the work (after some argument parsing) looks like this:

# Base64 encode the audio file AUDIO_BASE64=$(base64 < "$AUDIO_FILE" | tr -d '\n') # Construct the JSON payload JSON_PAYLOAD=$(jq -n \ --arg model "gpt-4o-audio-preview" \ --arg text "$TEXT_PROMPT" \ --arg audio "$AUDIO_BASE64" \ '{ model: $model, modalities: ["text"], messages: [ { role: "user", content: [ {type: "text", text: $text}, { type: "input_audio", input_audio: { data: $audio, format: "wav" } } ] } ] }') # Make the API call curl -s "https://api.openai.com/v1/chat/completions" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $OPENAI_API_KEY" \ -d "$JSON_PAYLOAD" | jq .

From the documentation it looks like you can send an "input_audio"."format" of either "wav" or "mp3".

You can run it like this:

./audio-prompt.sh 'describe this audio' decoded.wav

This dumps the raw JSON response to the console. Here's what I got for that sound clip I generated above, which gets a little creative:

The audio features a spoken phrase that is poetic in nature. It discusses the intertwining of "digits" in a coordinated and harmonious manner, as if engaging in a dance of unity. It mentions "codes" in a way that suggests they have an almost life-like quality. The tone seems abstract and imaginative, possibly metaphorical, evoking imagery related to technology or numbers.

A web app for recording and prompting against audio

I decided to turn this into a tiny web application. I started by asking Claude to create a prototype with a "record" button, just to make sure that was possible:

Build an artifact - no React - that lets me click a button to start recording, shows a counter running up, then lets me click again to stop. I can then play back the recording in an audio element. The recording should be a WAV

Then I pasted in one of my curl experiments from earlier and told it:

Now add a textarea input called "prompt" and a button which, when clicked, submits the prompt and the base64 encoded audio file using fetch() to this URL

The JSON that comes back should be displayed on the page, pretty-printed

The API key should come from localStorage - if localStorage does not have it ask the user for it with prompt()

I iterated through a few error messages and got to a working application! I then did one more round with Claude to add a basic pricing calculator showing how much the prompt had cost to run.

You can try the finished application here:

Here's the finished code. It uses all sorts of APIs I've never used before: AudioContext().createMediaStreamSource(...) and a DataView() to build the WAV file from scratch, plus a trick with FileReader() .. readAsDataURL() for in-browser base64 encoding.

Audio inputs are charged at $100/million tokens, and processing 5 seconds of audio her cost 0.6 cents.

The problem is the price

Audio tokens are currently charged at $100/million for input and $200/million for output. Tokens are hard to reason about, but a note on the pricing page clarifies that:

Audio input costs approximately 6¢ per minute; Audio output costs approximately 24¢ per minute

Translated to price-per-hour, that's $3.60 per hour of input and $14.40 per hour of output. I think the Realtime API pricing is about the same. These are not cheap APIs.

Meanwhile, Google's Gemini models price audio at 25 tokens per second (for input only, they don't yet handle audio output). That means that for their three models:

Gemini 1.5 Pro is $1.25/million input tokens, so $0.11 per hour Gemini 1.5 Flash is $0.075/milllion, so $0.00675 per hour (that's less than a cent) Gemini 1.5 Flash 8B is $0.0375/million, so $0.003375 per hour (a third of a cent!)

This means even Google's most expensive Pro model is still 32 times less costly than OpenAI's gpt-4o-audio-preview model when it comes to audio input, and Flash 8B is 1,066 times cheaper.

(I really hope I got those numbers right. I had ChatGPT double-check them. I keep find myself pricing out Gemini and not believing the results.)

I'm going to cross my fingers and hope for an OpenAI price drop in the near future, because it's hard to justify building anything significant on top of these APIs at the current price point, especially given the competition.

Tags: audio, projects, ai, openai, generative-ai, gpt-4, llms, ai-assisted-programming, claude

15:17 • 5 days ago

Ben Werdmüller

I joined Dot Social for a conversation about the future of media

I was lucky enough to sit down with Mike McCue, CEO at Flipboard, and 404 Media co-founder (and former Motherboard Editor-in-Chief) Jason Koebler to talk about the future of media and its intersection with the future of the social web. Savvy journalists at forward-thinking newsrooms are not letting this happen to them. Instead, they’re doing the work that arguably has been most critical

Savvy journalists at forward-thinking newsrooms are not letting this happen to them. Instead, they’re doing the work that arguably has been most critical all along: building direct connections with their audiences. It’s common to do this through email lists and subscription models, but the open social web offers a new, more equitable ecosystem for quality journalism to thrive.
Two people on the frontlines of this movement are Jason Koebler, a journalist and co-founder at 404 Media, and Ben Werdmuller, the senior director of technology at ProPublica. In this episode of Dot Social, the two talk about their fediverse experiences so far and why they’re hopeful for publishing in the future.

I loved being a part of this conversation. You can watch / listen over here.

11:53 • 5 days ago

Doc Searls Weblog

A Comet Hunt

Tonight was the first completely clear sky in a while, almost perfect for hunting Comet Tsuchinshan-ATLAS, which for a few more nights will be gracing our evening sky. With a full moon high in the eastern sky, and plenty of light pollution from the town around me, the comet was hard to see. Fortunately, the […]

Comet Tsuchinshan-ATLAS, more visible to the camera than the eye.

Tonight was the first completely clear sky in a while, almost perfect for hunting Comet Tsuchinshan-ATLAS, which for a few more nights will be gracing our evening sky.

With a full moon high in the eastern sky, and plenty of light pollution from the town around me, the comet was hard to see. Fortunately, the camera in my new iPhone 16 Pro Max sees better than I do, so I was able first to find the comet in a photo of the western sky, and then go back outside (from the restaurant where we were eating—the Half Bottle Bistro) and aim the phone’s 5x lens at it. That’s what got me the shot above. Step through this album of shots to see how I got the phone to zero in on the comet while my eyes could hardly see it.

I found this guide from the Stamford Advocate very helpful for finding it as well.

02:25 • 5 days ago

Simon Willison

Quoting D. Richard Hipp

I'm of the opinion that you should never use mmap, because if you get an I/O error of some kind, the OS raises a signal, which SQLite is unable to catch, and so the process dies. When you are not using mmap, SQLite gets back an error code from an I/O error and is able to take remedial action, or at least compose an error message. — D. Richard Hipp Tags: d-richard-hipp, sqlite

I'm of the opinion that you should never use mmap, because if you get an I/O error of some kind, the OS raises a signal, which SQLite is unable to catch, and so the process dies. When you are not using mmap, SQLite gets back an error code from an I/O error and is able to take remedial action, or at least compose an error message.

— D. Richard Hipp

Tags: d-richard-hipp, sqlite

01:21 • 5 days ago

Thursday, 17. October 2024

Simon Willison

Using static websites for tiny archives

Using static websites for tiny archives Alex Chan: Over the last year or so, I’ve been creating static websites to browse my local archives. I’ve done this for a variety of collections, including: paperwork I’ve scanned documents I’ve created screenshots I’ve taken web pages I’ve bookmarked video and audio files I’ve saved This is such a neat idea. These tiny little personal

Using static websites for tiny archives

Alex Chan:

Over the last year or so, I’ve been creating static websites to browse my local archives. I’ve done this for a variety of collections, including:
paperwork I’ve scanned documents I’ve created screenshots I’ve taken web pages I’ve bookmarked video and audio files I’ve saved

This is such a neat idea. These tiny little personal archive websites aren't even served through a localhost web server - they exist as folders on disk, and Alex browses them by opening up the index.html file directly in a browser.

Via @alex

Tags: html, archives

23:02 • 5 days ago

IdM Laboratory

G7メンバー国のデジタルアイデンティティガイドラインのマッピングが発表されています

こんにちは、富士榮です。 G7メンバー国でやっているIdentityガイドラインのマッピングエクセサイズのレポートが出ています。 https://www.oecd.org/en/publications/g7-mapping-exercise-of-digital-identity-approaches_56fd4e94-en.html This report presents a mapping exercise to identify commonalities in digital identity approaches among G7 members that can support future interoperability efforts. These commonalities include shared concepts and defi

This report presents a mapping exercise to identify commonalities in digital identity approaches among G7 members that can support future interoperability efforts. These commonalities include shared concepts and definitions, the use of international technical standards and approaches to levels of assurance. The report was prepared at the request of the 2024 Italian G7 Presidency and G7 members, to inform discussions within the G7 Digital and Technology Working Group. It was launched during the G7 Digital and Technology Ministerial Meeting in Como, Italy, on 15 October 2024.

本報告書は、将来の相互運用性の取り組みを支援することができる、G7 メンバー間のデジタル ID アプローチの共通点を特定するためのマッピング作業を提示する。これらの共通点には、共有される概念および定義、国際技術標準の使用、保証レベルへのアプローチなどが含まれる。この報告書は、2024 年イタリア G7 議長国および G7 メンバーの要請により、G7 デジタル・テクノロジー作業部会での議論に情報を提供するために作成された。2024年10月15日にイタリアのコモで開催されたG7デジタル・テクノロジー閣僚会合で発表された。

中身は順次見ていきたいと思いますが、カナダ、欧州、日本、英国、米国のそれぞれのガイドライン（例えば日本ならDS-500、米国ならNIST SP800-63-3）の比較・マッピングをしています。

これはSIDI Hubのワークストリームとも協調していくべき動きで、今後国境を跨いだコミュニケーションの中でデジタルアイデンティティがシームレスに利用できる世の中の実現に向けて非常に重要なステップですね。

22:30 • 5 days ago

Mike Jones: self-issued

OAuth 2.0 Protected Resource Metadata Specification in RFC Editor Queue

I’m pleased to report that the “OAuth 2.0 Protected Resource Metadata” specification has been approved by the IESG and is now in the RFC Editor queue. The version approved by the IESG and sent to the RFC Editor is: https://www.ietf.org/archive/id/draft-ietf-oauth-resource-metadata-13.html It joins OAuth 2.0 Security Best Current Practice and JWT Response for OAuth Token Introspection, […]

I’m pleased to report that the “OAuth 2.0 Protected Resource Metadata” specification has been approved by the IESG and is now in the RFC Editor queue.

The version approved by the IESG and sent to the RFC Editor is:

https://www.ietf.org/archive/id/draft-ietf-oauth-resource-metadata-13.html

It joins OAuth 2.0 Security Best Current Practice and JWT Response for OAuth Token Introspection, which are also both currently there.

Thanks to the IETF directorate reviewers and IESG members for their feedback that resulted in improvements to the specification!

19:05 • 5 days ago

Simon Willison

New in NotebookLM: Customizing your Audio Overviews

New in NotebookLM: Customizing your Audio Overviews The most requested feature for Google's NotebookLM "audio overviews" (aka automatically generated podcast conversations) has been the ability to provide direction to those artificial podcast hosts - setting their expertise level or asking them to focus on specific topics. Today's update adds exactly that: Now you can provide instructions b

New in NotebookLM: Customizing your Audio Overviews

The most requested feature for Google's NotebookLM "audio overviews" (aka automatically generated podcast conversations) has been the ability to provide direction to those artificial podcast hosts - setting their expertise level or asking them to focus on specific topics.

Today's update adds exactly that:

Now you can provide instructions before you generate a "Deep Dive" Audio Overview. For example, you can focus on specific topics or adjust the expertise level to suit your audience. Think of it like slipping the AI hosts a quick note right before they go on the air, which will change how they cover your material.

I pasted in a link to my post about video scraping and prompted it like this:

You are both pelicans who work as data journalist at a pelican news service. Discuss this from the perspective of pelican data journalists, being sure to inject as many pelican related anecdotes as possible

Here's the resulting 7m40s MP3, and the transcript.

Your browser does not support the audio element.

It starts off strong!

You ever find yourself wading through mountains of data trying to pluck out the juicy bits? It's like hunting for a single shrimp in a whole kelp forest, am I right?

Then later:

Think of those facial recognition systems they have for humans. We could have something similar for our finned friends. Although, gotta say, the ethical implications of that kind of tech are a whole other kettle of fish. We pelicans gotta use these tools responsibly and be transparent about it.

And when brainstorming some potential use-cases:

Imagine a pelican citizen journalist being able to analyze footage of a local council meeting, you know, really hold those pelicans in power accountable, or a pelican historian using video scraping to analyze old film reels, uncovering lost details about our pelican ancestors.

Plus this delightful conclusion:

The future of data journalism is looking brighter than a school of silversides reflecting the morning sun. Until next time, keep those wings spread, those eyes sharp, and those minds open. There's a whole ocean of data out there just waiting to be explored.

And yes, people on Reddit have got them to swear.

Tags: notebooklm, data-journalism, google, llms, ai, generative-ai, gemini

17:27 • 6 days ago

The Pragmatic Engineer

The Pulse #111: Did Automattic commit open source theft?

The maker of WordPress took 2M customers from its biggest rival: has a red line been crossed? Also: OpenAI’s impossible business projections, top AI researchers making more than engineers, and more.

The Pulse is a series covering insights, patterns, and trends within Big Tech and startups. Notice an interesting event or trend? Send me a message.

Today, we cover:

Industry pulse. Sudden layoffs at Meta, Spotify confirms work-from-anywhere, US mandates “click-to-cancel,” a historic SpaceX booster catch – and more.

Did Automattic commit open source theft? The maker of WordPress took a plugin with 2M installs, owned by its biggest competitor (WP Engine), and commandeered it. The goal was clear: hit WP Engine where it hurts the most, throwing decades of open source ethics in the bin. It sets a dangerous precedent for the industry. Read the un-paywalled version of this section here.

OpenAI’s impossible business projections. According to internal documents, OpenAI expects to generate $100B in revenue in 5 years, which is 25x more than it currently makes. This would mean OpenAI brings in more money than NVIDIA or Tesla!

Top AI research scientists earn more than engineers. Few companies can compete with the compensation which AI startups pay ML engineers. Still, research scientists at these startups can make roughly double of the highest-paid ML engineers. It’s an interesting new dynamic.

1. Industry pulse Sudden layoffs at Meta

Meta is executing layoffs in the US inside WhatsApp, Instagram, and Reality Labs groups, as reported by The Verge. A prolific person let go in this round is security engineer Jane Manchun Wong, who became known in tech circles by uncovering new features shipped in Big Tech mobile apps hidden behind feature flags.

Elsewhere at Meta, there were layoffs for unusual reasons a week or two ago, when some people were let go from the social media giant’s LA site for misusing $25 meal credits intended for use when working late in the office. Instead, the vouchers were used to have meals delivered to homes, and spent on things like wine glasses and laundry detergent, as per the Financial Times. The FT reports that dismissals were for repeat incidents.

This kind of firing would probably not happen a few years ago, I suspect. Back when companies like Meta were struggling to hire enough talent, violations like misusing meal vouchers would likely have resulted in a warning, and no more.

It’s interesting to contrast that Meta had zero mass layoffs for the first 18 years of its existence from 2004 until November 2022. Since then, cuts have been regular but unpredictable in their timing. This story is a reminder just how much Big Tech is changing: companies no longer offer the kind of cozy job security that was taken for granted in the 2010s; even Google is no longer a “rest and vest” type of place.

US mandates “click-to-cancel”

16:23 • 6 days ago

Simon Willison

Video scraping: extracting JSON data from a 35 second screen capture for less than 1/10th of a cent

The other day I found myself needing to add up some numeric values that were scattered across twelve different emails. I didn't particularly feel like copying and pasting all of the numbers out one at a time, so I decided to try something different: could I record a screen capture while browsing around my Gmail account and then extract the numbers from that video using Google Gemini? This turn

The other day I found myself needing to add up some numeric values that were scattered across twelve different emails.

I didn't particularly feel like copying and pasting all of the numbers out one at a time, so I decided to try something different: could I record a screen capture while browsing around my Gmail account and then extract the numbers from that video using Google Gemini?

This turned out to work incredibly well.

AI Studio and QuickTime

I recorded the video using QuickTime Player on my Mac: File -> New Screen Recording. I dragged a box around a portion of my screen containing my Gmail account, then clicked on each of the emails in turn, pausing for a couple of seconds on each one.

I uploaded the resulting file directly into Google's AI Studio tool and prompted the following:

Turn this into a JSON array where each item has a yyyy-mm-dd date and a floating point dollar amount for that date

... and it worked. It spat out a JSON array like this:

[ { "date": "2023-01-01", "amount": 2... }, ... ]

I wanted to paste that into Numbers, so I followed up with:

turn that into copy-pastable csv

Which gave me back the same data formatted as CSV.

You should never trust these things not to make mistakes, so I re-watched the 35 second video and manually checked the numbers. It got everything right.

I had intended to use Gemini 1.5 Pro, aka Google's best model... but it turns out I forgot to select the model and I'd actually run the entire process using the much less expensive Gemini 1.5 Flash 002.

How much did it cost?

According to AI Studio I used 11,018 tokens, of which 10,326 were for the video.

Gemini 1.5 Flash charges $0.075/1 million tokens (the price dropped in August).

11018/1000000 = 0.011018
0.011018 * $0.075 = $0.00082635

So this entire exercise should have cost me just under 1/10th of a cent!

And in fact, it was free. Google AI Studio currently "remains free of charge regardless of if you set up billing across all supported regions". I believe that means they can train on your data though, which is not the case for their paid APIs.

The alternatives aren't actually that great

Let's consider the alternatives here.

I could have clicked through the emails and copied out the data manually one at a time. This is error prone and kind of boring. For twelve emails it would have been OK, but for a hundred it would have been a real pain. Accessing my Gmail data programatically. This seems to get harder every year - it's still possible to access it via IMAP right now if you set up a dedicated app password but that's a whole lot of work for a one-off scraping task. The official API is no fun at all. Some kind of browser automation (Playwright or similar) that can click through my Gmail account for me. Even with an LLM to help write the code this is still a lot more work, and it doesn't help deal with formatting differences in emails either - I'd have to solve the email parsing step separately. Using some kind of much more sophisticated pre-existing AI tool that has access to my email. A separate Google product also called Gemini can do this if you grant it access, but my results with that so far haven't been particularly great. AI tools are inherently unpredictable. I'm also nervous about giving any tool full access to my email account due to the risk from things like prompt injection. Video scraping is really powerful

The great thing about this video scraping technique is that it works with anything that you can see on your screen... and it puts you in total control of what you end up exposing to the AI model.

There's no level of website authentication or anti-scraping technology that can stop me from recording a video of my screen while I manually click around inside a web application.

The results I get depend entirely on how thoughtful I was about how I positioned my screen capture area and how I clicked around.

There is no setup cost for this at all - sign into a site, hit record, browse around a bit and then dump the video into Gemini.

And the cost is so low that I had to re-run my calculations three times to make sure I hadn't made a mistake.

I expect I'll be using this technique a whole lot more in the future. It also has applications in the data journalism world, which frequently involves the need to scrape data from sources that really don't want to be scraped.

Bonus: An LLM pricing calculator

In writing up this experiment I got fed up of having to manually calculate token prices. I actually usually outsource that to ChatGPT Code Interpreter, but I've caught it messing up the conversion from dollars to cents once or twice so I always have to double-check its work.

So I got Claude 3.5 Sonnet with Claude Artifacts to build me this pricing calculator tool (source code here):

You can set the input/output token prices by hand, or click one of the preset buttons to pre-fill it with the prices for different existing models (as-of 16th October 2024 - I won't promise that I'll promptly update them in the future!)

The entire thing was written by Claude. Here's the full conversation transcript - we spent 19 minutes iterating on it through 10 different versions.

Rather than hunt down all of those prices myself, I took screenshots of the pricing pages for each of the model providers and dumped those directly into the Claude conversation:

Tags: data-journalism, gmail, google, scraping, ai, generative-ai, llms, ai-assisted-programming, claude, gemini, vision-llms, claude-artifacts, claude-3-5-sonnet

12:32 • 6 days ago

Gemini API Additional Terms of Service

Gemini API Additional Terms of Service I've been trying to figure out what Google's policy is on using data submitted to their Google Gemini LLM for further training. It turns out it's clearly spelled out in their terms of service, but it differs for the paid v.s. free tiers. The paid APIs do not train on your inputs: When you're using Paid Services, Google doesn't use your prompts (includi

Gemini API Additional Terms of Service

I've been trying to figure out what Google's policy is on using data submitted to their Google Gemini LLM for further training. It turns out it's clearly spelled out in their terms of service, but it differs for the paid v.s. free tiers.

The paid APIs do not train on your inputs:

When you're using Paid Services, Google doesn't use your prompts (including associated system instructions, cached content, and files such as images, videos, or documents) or responses to improve our products [...] This data may be stored transiently or cached in any country in which Google or its agents maintain facilities.

The Gemini API free tier does:

The terms in this section apply solely to your use of Unpaid Services. [...] Google uses this data, consistent with our Privacy Policy, to provide, improve, and develop Google products and services and machine learning technologies, including Google’s enterprise features, products, and services. To help with quality and improve our products, human reviewers may read, annotate, and process your API input and output.

But watch out! It looks like the AI Studio tool, since it's offered for free (even if you have a paid account setup) is treated as "free" for the purposes of these terms. There's also an interesting note about the EU:

The terms in this "Paid Services" section apply solely to your use of paid Services ("Paid Services"), as opposed to any Services that are offered free of charge like direct interactions with Google AI Studio or unpaid quota in Gemini API ("Unpaid Services"). [...] If you're in the European Economic Area, Switzerland, or the United Kingdom, the terms applicable to Paid Services apply to all Services including AI Studio even though it's offered free of charge.

Confusingly, the following paragraph about data used to fine-tune your own custom models appears in that same "Data Use for Unpaid Services" section:

Google only uses content that you import or upload to our model tuning feature for that express purpose. Tuning content may be retained in connection with your tuned models for purposes of re-tuning when supported models change. When you delete a tuned model, the related tuning content is also deleted.

It turns out their tuning service is "free of charge" on both pay-as-you-go and free plans according to the Gemini pricing page, though you still pay for input/output tokens at inference time (on the paid tier - it looks like the free tier remains free even for those fine-tuned models).

Tags: gemini, llms, google, generative-ai, training-data, ai, fine-tuning

03:06 • 6 days ago

Wednesday, 16. October 2024

Simon Willison

files-to-prompt 0.4

files-to-prompt 0.4 New release of my files-to-prompt tool adding an option for filtering just for files with a specific extension. The following command will output Claude XML-style markup for all Python and Markdown files in the current directory, and copy that to the macOS clipboard ready to be pasted into an LLM: files-to-prompt . -e py -e md -c | pbcopy Tags: projects, python, l

files-to-prompt 0.4

New release of my files-to-prompt tool adding an option for filtering just for files with a specific extension.

The following command will output Claude XML-style markup for all Python and Markdown files in the current directory, and copy that to the macOS clipboard ready to be pasted into an LLM:

files-to-prompt . -e py -e md -c | pbcopy

Tags: projects, python, llms

23:29 • 6 days ago

2025 DSF Board Nominations

2025 DSF Board Nominations The Django Software Foundation board elections are coming up. There are four positions open, seven directors total. Terms last two years, and the deadline for submitting a nomination is October 25th (the date of the election has not yet been decided). Several community members have shared "DSF initiatives I'd like to see" documents to inspire people who may be conside

2025 DSF Board Nominations

The Django Software Foundation board elections are coming up. There are four positions open, seven directors total. Terms last two years, and the deadline for submitting a nomination is October 25th (the date of the election has not yet been decided).

Several community members have shared "DSF initiatives I'd like to see" documents to inspire people who may be considering running for the board:

Sarah Boyce (current Django Fellow) wants a marketing strategy, better community docs, more automation and a refresh of the Django survey. Tim Schilling wants one big sponsor, more community recognition and a focus on working groups. Carlton Gibson wants an Executive Director, an updated website and better integration of the community into that website. Jacob Kaplan-Moss wants effectively all of the above.

There's also a useful FAQ on the Django forum by Thibaud Colas.

Tags: django, dsf, jacob-kaplan-moss

23:01 • 6 days ago

IdM Laboratory

Credential Exchange Format/Protocolの新Working draft

こんにちは、富士榮です。
Authenticate 2024が10/14-16で開催されましたね。各社イベントに向けて色々と実装をして発表にこぎつける、というのは世の常ですが、これはもちろんスペックライターについても同じようです。

ということで満を持して発表されましたね。 https://fidoalliance.org/specifications-credential-exchange-specifications/
Credential Exchange Format Credential Exchange Protocol これらの新しいWorking draftが今週金曜日18日にリリースされるようです。
以前から触れていたCredential Exchange Specificationsですがパスキーのインポート・エクスポートのための仕様という冠で登場って感じですかね。
同期パスキーなど色々と新しい考え方が盛り込まれてきたFIDO関係ですが、NISTを含めちゃんと利用ガイドを整えていかないといけませんね。（まぁ、そもそも鍵はデバイスから出ないっていうのが原則だったわけなので、考え方を変えていかないといけません）

22:30 • 6 days ago

Simon Willison

Supercharge the One Person Framework with SQLite: Rails World 2024

Supercharge the One Person Framework with SQLite: Rails World 2024 Stephen Margheim shares an annotated transcript of the YouTube video of his recent talk at this year's Rails World conference in Toronto. The Rails community is leaning hard into SQLite right now. Stephen's talk is some of the most effective evangelism I've seen anywhere for SQLite as a production database for web applications,

Supercharge the One Person Framework with SQLite: Rails World 2024

Stephen Margheim shares an annotated transcript of the YouTube video of his recent talk at this year's Rails World conference in Toronto.

The Rails community is leaning hard into SQLite right now. Stephen's talk is some of the most effective evangelism I've seen anywhere for SQLite as a production database for web applications, highlighting several new changes in Rails 8:

... there are two additions coming with Rails 8 that merit closer consideration. Because these changes make Rails 8 the first version of Rails (and, as far as I know, the first version of any web framework) that provides a fully production-ready SQLite experience out-of-the-box.

Those changes: Ensure SQLite transaction default to IMMEDIATE mode to avoid "database is locked" errors when a deferred transaction attempts to upgrade itself with a write lock (discussed here previously, and added to Datasette 1.0a14 in August) and SQLite non-GVL-blocking, fair retry interval busy handler - a lower-level change that ensures SQLite's busy handler doesn't hold Ruby's Global VM Lock (the Ruby version of Python's GIL) while a thread is waiting on a SQLite lock.

The rest of the talk makes a passionate and convincing case for SQLite as an option for production deployments, in line with the Rails goal of being a One Person Framework - "a toolkit so powerful that it allows a single individual to create modern applications upon which they might build a competitive business".

Back in April Stephen published SQLite on Rails: The how and why of optimal performance describing some of these challenges in more detail (including the best explanation I've seen anywhere of BEGIN IMMEDIATE TRANSACTION) and promising:

Unfortunately, running SQLite on Rails out-of-the-box isn’t viable today. But, with a bit of tweaking and fine-tuning, you can ship a very performant, resilient Rails application with SQLite. And my personal goal for Rails 8 is to make the out-of-the-box experience fully production-ready.

It looks like he achieved that goal!

Via Hacker News

Tags: scaling, rails, sqlite, ruby, gil

22:24 • 6 days ago

[red-knot] type inference/checking test framework

[red-knot] type inference/checking test framework Ruff maintainer Carl Meyer recently landed an interesting new design for a testing framework. It's based on Markdown, and could be described as a form of "literate testing" - the testing equivalent of Donald Knuth's literate programming. A markdown test file is a suite of tests, each test can contain one or more Python files, with optionally s

[red-knot] type inference/checking test framework

Ruff maintainer Carl Meyer recently landed an interesting new design for a testing framework. It's based on Markdown, and could be described as a form of "literate testing" - the testing equivalent of Donald Knuth's literate programming.

A markdown test file is a suite of tests, each test can contain one or more Python files, with optionally specified path/name. The test writes all files to an in-memory file system, runs red-knot, and matches the resulting diagnostics against Type: and Error: assertions embedded in the Python source as comments.

Test suites are Markdown documents with embedded fenced blocks that look like this:

```py
reveal_type(1.0) # revealed: float
```

Tests can optionally include a path= specifier, which can provide neater messages when reporting test failures:

```py path=branches_unify_to_non_union_type.py
def could_raise_returns_str() -> str:
    return 'foo'
...
```

A larger example test suite can be browsed in the red_knot_python_semantic/resources/mdtest directory.

This document on control flow for exception handlers (from this PR) is the best example I've found of detailed prose documentation to accompany the tests.

The system is implemented in Rust, but it's easy to imagine an alternative version of this idea written in Python as a pytest plugin. This feels like an evolution of the old Python doctest idea, except that tests are embedded directly in Markdown rather than being embedded in Python code docstrings.

... and it looks like such plugins exist already. Here are two that I've found so far:

pytest-markdown-docs by Elias Freider and Modal Labs. sphinx.ext.doctest is a core Sphinx extension for running test snippets in documentation. pytest-doctestplus from the Scientific Python community, first released in 2011.

I tried pytest-markdown-docs by creating a doc.md file like this:

# Hello test doc

```py
assert 1 + 2 == 3
```

But this fails:

```py
assert 1 + 2 == 4
```

And then running it with uvx like this:

uvx --with pytest-markdown-docs pytest --markdown-docs

I got one pass and one fail:

_______ docstring for /private/tmp/doc.md __________
Error in code block:
```
10   assert 1 + 2 == 4
11   
```
Traceback (most recent call last):
  File "/private/tmp/tt/doc.md", line 10, in <module>
    assert 1 + 2 == 4
AssertionError

============= short test summary info ==============
FAILED doc.md::/private/tmp/doc.md
=========== 1 failed, 1 passed in 0.02s ============

I also just learned that the venerable Python doctest standard library module has the ability to run tests in documentation files too, with doctest.testfile("example.txt"): "The file content is treated as if it were a single giant docstring; the file doesn’t need to contain a Python program!"

Via Charlie Marsh

Tags: testing, rust, python, astral, markdown, ruff, pytest, uv

20:43 • 6 days ago

Decentral Gabe

The Greatly Exaggerated Demise of SSI: A Rebuttal to Premature Eulogies

💡 This is written in response to a post titled "Why Verifiable Credentials Aren't Widely Adopted & Why Trinsic Pivoted" A forewarning—there will be no tears here and, please, no hugging. credit, XKCD https://xkcd.com/386/ The post I'm responding to

💡 This is written in response to a post titled "Why Verifiable Credentials Aren't Widely Adopted & Why Trinsic Pivoted"

A forewarning—there will be no tears here and, please, no hugging.

credit, XKCD https://xkcd.com/386/

The post I'm responding to makes a point about people resisting its conclusions because their livelihoods depend on it. But let’s be real—Trinsic depends on this too, just as much as mine does. We’re both here, with skin in the game, and that's a good thing. I continue to work on figuring out how to make self-sovereign identity work at scale because I believe it's worth it. If I was not convinced of its value, I wouldn't be here debating it; I'd have moved on to easier, more profitable paths. That’s the commitment this industry requires—guts, patience, and a long-term perspective.

So let's set the stage clearly. This isn't just about egos or jobs; it's about a vision for the future of digital identity. And while you’ve pivoted, it's worth looking at the broader picture and challenging the notion that SSI is somehow "dead" just because it's hard. We’re not the first to face a daunting path—nor will we be the last.

The Market Isn't the Oracle You Think It Is

Your unwavering faith in the market's infallibility is touching. You proclaim, "The market is the objective reality," as if channeling the spirit of a laissez-faire economist at a startup retreat. But let's examine recent history.

Betamax was superior to VHS, but marketing muscle won the day. The QWERTY keyboard isn't ergonomic, yet it's entrenched due to historical happenstance. The invisible hand often leads us down paths of inefficiency and fragmentation, leaving a trail of superior but abandoned technologies in its wake.

Fragmentation and incompatible standards aren't signs of the market "choosing" the best option; they're about who can shout "MINE!" the loudest while elbowing competitors off the playground. Sometimes, it takes a guiding hand—a nudge from governments or standard bodies—to steer the market toward what's efficient and beneficial for all.

Pivoting Isn't Prophecy you can do better, hon

The leap from "We couldn't make it work" to "It can't work" is a logical chasm too wide to cross. Your pivot from SSI evangelist to skeptic, while understandable, isn't a revelation of universal truth.

This is akin to Friendster declaring social media a dead end, or Blockbuster declaring video streaming a passing fad. The inability to monetize a nascent technology doesn't negate its potential; it merely highlights the gap between visionary ideals and the brutal realities of running a startup. For example, certain decentralized identity projects have successfully implemented verifiable credentials in applications like educational certificates, employment verification, and gaining access to the financial system, demonstrating real-world utility and growing momentum. These efforts show that verifiable credentials are gaining traction in specific, focused use cases.

Let me be unequivocal: I commend you for having the guts to start a company, work hard for years, and continue to try to make it viable. That is nothing short of admirable. Your pivot might make sense for Trinsic, but it's hardly the canary in the coal mine for SSI.

SSI, like many transformative technologies before it, may simply be awaiting its "streaming moment" — that convergence of technological readiness, market understanding, and user demand that catapults it from interesting concept to indispensable tool.

The Power of Collective Standards

Dismissing government involvement as a mere "silver lining" is like calling the sun a "helpful light bulb." Let's not overlook crucial historical precedents.

The internet (ARPANET), GPS, and even time zone standardization were government initiatives. The metric system, USB, and Wi-Fi all benefited from coordinated efforts beyond market forces. These standards didn't become universal because the market willed them so, but because of coordination to prevent fragmentation.

It is important not to ignore this. Preventing the chaos of incompatible systems often requires patience and collaboration—yes, even more patience than the industry has shown to this point.

Scattered Seeds Grow Diverse Forests

Your lament that SSI's "heat map is too scattered" misses the point. This diversity isn't a bug—it's a feature. The applications sprouting in education, finance, healthcare, and agriculture worldwide are the seeds of a diverse forest, not a failed monoculture.

Yes, SSI's initial user experience may be challenging, but that's par for the course with transformative technologies. Remember when 2G networks made downloading a single image feel interminable? Or when dial-up internet's screeching was our daily soundtrack?

Every groundbreaking technology endures growing pains. Users adapt, technologists adapt, and the market adapts with them—especially when the endgame offers control, privacy, and empowerment.

Business Goals Aren't Universal Truths

There's a profound difference between "We couldn't make it work for our business" and "The technology is inherently flawed."

Your focus on immediate market adoption overlooks the long-term value of doing the right thing—the hard thing—the thing that maybe, honestly, is not suited for startups that need to show growth and profit now. Perhaps a successful SSI startup is better suited as a B-Corp.

While market challenges are real, the problems that led us to SSI have only grown more severe. Data breaches are no longer news; they're an expectation. Surveillance Capitalism has entrenched itself deeper into the fabric of our digital lives, commodifying our every interaction. Government-issued digital identities, on the hardware of monopolistic tech giants, laced with privacy-eroding mandates, threatening to erode what little privacy we have left.

SSI isn't just a novel technology; it is our defense against these escalating digital threats.

Focused Implementation

While the post laments the scattered nature of SSI adoption, I see this differently. There is a clear answer to the diversity challenge: focus on specific, high-impact use cases. Our net is intentionally not cast wide.

We don't require credentials to be used across different domains from which they were issued. That's a potential bonus, not a requirement. It does not undermine the original goal of the technology; optionality is a strength. It leaves the door open for future cross-domain adoption without dependence on it for current success.

This focused strategy allows us to deliver value in targeted areas while building the infrastructure for broader adoption. It’s a pragmatic approach that addresses immediate needs while paving the way for SSI’s long-term vision. An iterative approach helps build confidence in the SSI ecosystem, one application at a time.

The success of SSI won't come from boiling the ocean. We’re laying the groundwork for future interoperability while delivering value today.

A Vision Beyond The Fiscal Calendar

While some may chase immediate market opportunities, many of us remain committed to SSI's promise. Transformational change isn't measured in fiscal quarters but in steady progress over years, many years. It demands fortitude, collaboration, and a willingness to look beyond short-term gains.

To the SSI community: Let this be our rallying cry. We're not just building a product; we're creating the foundation for human-centric digital identity. Let's continue to collaborate on improving user experience, pushing interoperability, and engaging governments and standards organizations. Every effort, no matter how small, contributes to a future where secure, privacy-respecting, self-sovereign digital identity is the norm.

The most compelling technological advances come from those who persevere when the path gets steep. The future of digital identity—one that isn't controlled solely by governments or tech monopolies—is counting on us.

credit: Dune

Should you find yourself at the next IIW, or the one after that, perhaps hosting a session titled "I Was Wrong About Being Wrong About SSI," feel free to invite us—we'll be there. We won't cry, clap, or hug you. We'll simply nod, in recognition that the most profound insights often come from pushing through challenges, not pivoting away from them.

04:32 • 7 days ago

Tuesday, 15. October 2024

IdM Laboratory

信頼できるAIに関するG7のアクションプラン

こんにちは、富士榮です。 10/9〜11にローマで開催されたG7のラウンドテーブルでDFFT（Data Free Flow with Trust）文脈でTrustwothy AIに関するアクションプランについての宣言が出ていますね。こちらEUのプレスですが。 Data Protection Authorities Collaborate to Shape the Future of AI and Privacy https://www.edps.europa.eu/press-publications/press-news/press-releases/2024/g7-roundtable-2024-data-protection-authorities-collaborate-shape-future-ai-and-privacy_en ざっくりですが

こんにちは、富士榮です。

10/9〜11にローマで開催されたG7のラウンドテーブルでDFFT（Data Free Flow with Trust）文脈でTrustwothy AIに関するアクションプランについての宣言が出ていますね。

こちらEUのプレスですが。

Data Protection Authorities Collaborate to Shape the Future of AI and Privacy

https://www.edps.europa.eu/press-publications/press-news/press-releases/2024/g7-roundtable-2024-data-protection-authorities-collaborate-shape-future-ai-and-privacy_en

ざっくりですが、

第4回G7データ保護当局（DPA）ラウンドテーブルがローマで開催されたイタリアのデータ保護当局が主催し、カナダ、フランス、ドイツ、日本、英国、米国、欧州データ保護委員会（EDPB）、およびEDPSのプライバシーおよびデータ保護規制当局が一堂に会した倫理的で信頼性の高いAIの開発におけるデータ保護の重要性に関するグローバルな議論の形成に貢献することが目標だった DFFT、新興技術の影響、執行協力の3つの主要分野が焦点だった人工知能（AI）の倫理的かつ信頼性の高い開発に特に注目して議論した AI技術が信頼に足るものであり、責任を持って使用されることを保証する上で、データ保護当局が重要な役割を果たすことを強調する声明が採択された子供のプライバシー保護に向けた緊急の行動を呼びかける「AIと子供に関する声明」を発表した個人情報を保護する強固な国境を越えたデータ転送メカニズムの重要性が強調された DPAは2024/2025年に向けたアクションプランを承認し、2024年コミュニケで概説されたように、DFFT、新興技術、執行協力に引き続き重点的に取り組むことを表明したという感じです。
子供のプライバシーの話題など、ますます気をつけていかないといけませんね。こちらのポストにも書きましたが、ドイツテレコムの出しているAIと子供のプライバシーに関する動画はなかなかショッキングなところもありますので、しっかりと議論をしていってもらいたいところです。

22:30 • 7 days ago

Doc Searls Weblog

Identity as Root

This is from an email thread on the topic of digital identity, which is the twice-yearly subject* of the Internet Identity Workshop, the most leveraged conference I know. It begins with a distinction that Devon Loffreto (who is in the thread) came up with many moons ago: Self-sovereign identity is who you are, how you […]

Shot with a camcoder on a beach in the Caribbean during a 2001 Linux Journal Geek Cruise.

Self-sovereign identity is who you are, how you choose to be known, to whom, based on reasons of your own. It’s something you own, just like you own your body. Administrative identity is what organizations call you for their convenience (and secondarily, yours). We may call these labels “an ID,” but they are not who we are.

Here is what I wrote:

Humans have opposable thumbs. This makes them capable of tool use to a degree beyond immeasurable. Perhaps nothing, other than their brains, makes humans supremely capable as a species. It also makes them grabby. Try to teach Marxism to a three year old yelling “It’s mine!”

My mother’s favorite account of me as a small child was how I walked around constantly holding as many small toy trucks in my hands as I possibly could, unwilling to let them go. But this tendency was about control more than possession. I hadn’t yet learned to put my trucks in something I could carry around. I was unwilling to trust that a box or a bag was a working extension of my grabby little self.

I’m still a bit like that. “Your trucks” is what Joyce calls the electronic stuff I carry around. But I’m not alone. We conceive everything in terms that imply or involve forms of control, possession, or both. The English language, among many others, cannot get along without possessive pronouns: my, mine, our, ours, their, theirs, your, yours, hers, his. Even if ownership in the legal sense is not involved, responsibility is. Control is. When you drive a rental car, those are your wheels, your bumpers, your engine. You also think and talk about them with first person possessive pronouns.

Personal agency moves outward from that sense of control and responsibility over what is ours, including our selves.

This is why we need to start understanding personal identity, and how it works in the world, by recognizing that each of us is a self-sovereign human being. We are each, as William Ernest Henley put it in his poem Invictus, the captain of our “unconquerable soul.” Walt Whitman’s Song of Myself was a long-form explication of the same thing. (Which I wrote about way back in ’96, before there were blogs.)

This is not to deny that we are also profoundly social. But it is essential that we start with the personal.

Ownership is an abstract concept. So are rights. But we need both to operate civilization.

What makes ownership righteous is that it minimally abstract. We see this with the possessive pronouns “my” and “mine.” Again, this is rooted in our possessive nature, our opposable thumbs. We need to be self-possessed (and -sovereign), autonomous, and independent—before we are anything else, including our social selves.

In technical terms, it’s root.

So there ya go.

*But not the only subject. I’d say about half of the topics that come up at IIW are for topics other than identity.

20:39 • 7 days ago

Ben Werdmüller

Is Matt Mullenweg defending WordPress or sabotaging it?

[Mathew Ingram] Mathew Ingram's overview of the WordPress drama continues to be updated with new information. The hole just seems to be getting deeper and deeper. As he says: it's a mess. "It's pretty clear that Matt sees what he is doing as protecting WordPress, and forcing a no-good corporation to cough up some dough after years of taking advantage of the community (he say

[Mathew Ingram]

Mathew Ingram's overview of the WordPress drama continues to be updated with new information. The hole just seems to be getting deeper and deeper. As he says: it's a mess.

"It's pretty clear that Matt sees what he is doing as protecting WordPress, and forcing a no-good corporation to cough up some dough after years of taking advantage of the community (he says he has been trying to negotiate with WP Engine for more than a year now, while WP Engine says it gives back to WordPress in a number of ways.) To some observers like me, however — and to some other longtime members of the WordPress ecosystem — it looks like Matt has dragged the WordPress community into a legal mess with a variety of unforeseen and potentially serious consequences."

I still don't fully understand what prompted this sea change in how Matt has addressed the wider community, including WP Engine. I have this ongoing sense that there's another shoe left to drop, whether it's relating to stalling revenue at Automattic and pressure from its board (pure conjecture on my part, to be clear), or something else entirely. Without some strong motivating factor this just seems to be self-sabotage.

At this point I'm really curious to see what's next. All this drama has also made it clear that for the kind of CMS WordPress is - more of a framework than an out-of-the-box solution at this point, but with a strong ready-made editing and administration experience - there aren't many alternatives. That's not to denegrate other projects like Drupal, etc, because I think they're different kinds of CMSes. Ghost is much more narrowly focused, too. I think if WordPress had a real competitor in its space, this might all be playing out differently.

(If I was WP Engine and had run out of options to de-escalate, I'd be going ahead and forking WordPress right now. And what a mess that would be.)

[Link]

18:54 • 7 days ago

It feels like 2004 again.

[Anil Dash] Anil Dash makes a pertinent observation about the current state of the web: "At the start of this year, I wrote The Internet Is About To Get Weird Again, which began by calling back to the Internet of 2000. In thinking more about it, though, we more closely resemble the Internet of a few years later, where the crash of the dot-com bubble and the stock market had

[Anil Dash]

Anil Dash makes a pertinent observation about the current state of the web:

"At the start of this year, I wrote The Internet Is About To Get Weird Again, which began by calling back to the Internet of 2000. In thinking more about it, though, we more closely resemble the Internet of a few years later, where the crash of the dot-com bubble and the stock market had the same effect that the popping of the crypto bubble did: the casuals who were just trying to make a quick buck are much less likely to jump in the pool."

I agree.

The way I've been thinking about it is: There's everything to play for. We understand what can go wrong. We understand many of the needs, although we should always go out and learn more. But for the first time in a long time, the internet isn't calcified: there isn't a sense that the platforms people use are set. Anyone can come along and build something new, and it's absolutely possible for it to catch on.

And, as Anil says, the spirit of the web is more intact than it has been in a long time. Gone (hopefully) are the Wall Street-esque folks who are here to make a bunch of money; instead, we're left with the people who genuinely care about connecting and creating and making something good. That's what powered the web's heyday, and that's what has the potential to make a difference now.

Let's go make good stuff.

https://www.iso.org/standard/82772.html

[Link]

17:22 • 8 days ago

Riley Hughes

Why Verifiable Credentials Aren’t Widely Adopted & Why Trinsic Pivoted

A presentation I gave at IIW38, in April 2024. At the Internet Identity Workshop 38, I announced my talk with a description something like this: Five years ago at this conference, Trinsic’s first product (our self-sovereign identity wallet app) was launched. Today, I’m hosting a session called “SSI didn’t work. Trinsic is pivoting.” I knew it was a provocative title — and I fully consid

A presentation I gave at IIW38, in April 2024.

At the Internet Identity Workshop 38, I announced my talk with a description something like this:

Five years ago at this conference, Trinsic’s first product (our self-sovereign identity wallet app) was launched. Today, I’m hosting a session called “SSI didn’t work. Trinsic is pivoting.”

I knew it was a provocative title — and I fully considered that apostatizing this publicly could result in a mob of opposition. But I was not prepared for what actually happened.

Right after my announcement (and brief, surprised gasp from the crowd of ~350 people) one IIW veteran popped off his chair and clapped his way across the room to embrace me. Right before my session, the CEOs of two separate identity companies asked to broadcast the talk online (one via livestream, another via live Tweet). Right after my session during the closing circle (where the original crowd of ~350 reconvened), the cofounder of the conference wept as she thanked me. Another longtime identity veteran stood immediately after and claimed it was the most important session held at the event in the last 5 years. The next morning, a startup CEO called for memorializing a “Riley Session” as a new staple of the conference twice each year. More than a dozen people asked me to reprise my session.

I say this all descriptively, not boastfully. Far from feeling proud, I actually felt humbled to have had the opportunity to contribute something evidently valuable to a community which has given me so much. (It was at an IIW in 2018 that I realized I wanted to build a career in this industry.) That is the context for this post — to share apparently-useful insights that hopefully will result in getting the world closer to using better digital ID technologies.

The spoiler for this article: I’m no longer betting on verifiable credentials taking off. I mean this literally — 99% of my net worth is Trinsic stock, so the strategy I bet my company on is an objective revelation of my strongest convictions. While once central to Trinsic’s existence, our latest pivot demonstrates that I believe interoperable digital credentials won’t succeed in market anytime soon.

Specifically, my prognostication is that digital identities will continue to be deployed and adopted, but that fragmentation, incompatibility, and bespoke implementations will continue to dominate the market.

Now, I’ve been working on this problem for 7 years. Explaining how I landed on this conclusion requires expounding on years of the business equivalent of trench warfare. This post is a deep dive, not a summary. If you want a summary, I suggest dropping this post into your LLM of choice (I won’t be offended). I’ll do my best to be concise, but I also intend on being thorough. The structure of this post is as follows:

Assumptions

Brief History

Impossible Problems

Bull Case

End

Assumptions

Assumption #0: I’ve drunk the Kool-aid

As you read this post, your walls might go up. Especially if your salary depends on me being wrong. Just be aware of your biases and know that I am still Kool-aid drunk on the philosophy of self-sovereign identity. I want that future as much as anybody. But in my sincere pursuit to discover reality, I’ve learned some things that I hope others can benefit from.

Assumption #1: Terminology

I will use “verifiable credential” or “VC” in this post quite a bit. I’ll also use “self-sovereign identity” or “SSI” or “IDtech”. “Decentralized identity” or “DI” could be subbed in too for all I care. Fundamentally, I’m trying to communicate: A standard way to exchange trustworthy data in a user-centric/user-controlled way. But I’m intentionally avoiding being too pedantic, so don’t get overly caught up in semantics.

Assumption #2: The world is huge

The world is inconceivably big.

Consulting in SSI is a good way to make a living. But hand-to-hand combat is insufficient to get VCs adopted. The only way to scale something on the magnitude needed for SSI to work is a product with product/market fit. That, or regulatory obligation.

Imagine if companies needed Accenture to adopt artificial intelligence. Now consider how that differs from a world where any developer can pick up an API key from OpenAI and implement GPT4 in an afternoon. LLMs have product/market fit, SSI does not.

Assumption #3: The market is reality

Here’s how I think about startups & new product development. The market is unknowable in its entirety. If it was known how to build a successful product (in our case, a widely-adopted SSI wallet or an adoptable VC format), it would have been done by now. To make progress at the cutting edge, you must do something desirable and different. Andy Radcliff calls this being “non-consensus and right”.

Chamath describes a startup as an exercise in mentally modeling the “market” and building a product to fit that model. If your product is rejected, your model of the world is wrong and you need to iterate its form. If your product is adopted, your model of the world is right.

All this is to say, the market is the objective reality. It decides winners and losers. As I’ve sought to get product/market fit in this space, I seek to discover reality as impartially as I can. When I see evidence of something working, I pay attention; when I see evidence that something isn’t working, I try to absorb those data points too.

Assumption #4: The best technology gets adopted

The worth of software is its impact on humans. The most elegant, idealistic, best software is therefore worthless unless it’s adopted. Actually, if it’s not adopted, it’s not elegant, idealistic, or best — no consolation prizes. The market is the judge and if your product isn’t making an impact on people’s lives, it’s no better than a refrigerator on the moon.

Adoption isn’t just the most important thing, it’s the only thing. Thus, this is actually a tautology: by definition, what gets adopted is best.

Assumption #5: Don’t waste your time

I’ve thought a lot about the optimal point between discouraging and cautioning. Yes, I’ve tried a lot of things but it’s important to state that markets change. Maybe something is different now than when I tried it. I surely can’t turn over every stone. Running new or differentiated experiments in this market is probably fruitful–I don’t want to discourage innovation. But I do want it to shine a light on the graveyard that exists in this space and hopefully prevent people from doing things that are doomed to fail.

Brief history

I started my journey in identity in 2017 when I interviewed for a job at Sovrin Foundation. I started Trinsic with two cofounders in 2019 because of the relatively low adoption of Sovrin’s network. I thought if we made the technology dramatically easier to use, more companies would adopt SSI.

Me plus Tomislav & Michael, my cofounders, launching our product at IIW in 2019.

We built something good. Hundreds of developers were signing up every month and issuing verifiable credentials. One of those developers was Wayne Chang who, before cofounding Spruce, gave me the feedback that Trinsic was the best execution of any SSI product in the market. He asked something like:

“With a product like this (where a developer can get value for free) you are probably either using this as lead gen for $250k consulting deals, or you’re just really betting on verifiable credentials blowing up.”

I told him we were betting on VCs blowing up.

Our aim was to build a product that would accelerate VC adoption. In our view, if a customer needed consulting to succeed, it was a signal the product wasn’t good enough. (We did consult customers on how to succeed — but we treated this time as product discovery time, and incorporated the results immediately into the product roadmap.)

What happened is we acquired dozens of paying customers, but not enough were successful enough. We had roughly 1 breakaway customer story per 500 companies that signed up. The rest were either startups that failed to get traction or large enterprise R&D departments that never went to production.

In the meantime, the market shaped out in an undeniable way. Dozens of companies obtained product/market fit in the reusable identity space, none of which use the VC standard. Either none of the successful companies chose to use VCs, or none of companies that chose VCs succeeded (two statements of identical outcome but different presumed causation).

A slide I made in 5 mins immediately before my IIW38 presentation to make this point!

In fact one such company, which raised over $75m from top-tier venture capitalists to fuel its meteoric growth, said to me (paraphrasing):

“The SSI people had a good idea. So we’re applying that idea to get customers while they’re wasting their time in working groups. You can think of us as proprietary SSI.”

It’s been a few years since I was told that. I recently checked back in on them. They’re used by tens of millions of users.

Another reusable ID company used by several million end-users, which has raised 10s of millions, told me (paraphrasing):

“We are building a monopoly. Our goal is to own the identity layer of the internet. It might not work, but if it does, we’re going to make a lot of money.”

Both companies were started after 2019 (the year the VC data model became a W3C recommendation) and chose not to use standards.

This dynamic vexed me for months. Why was the market rewarding companies with inferior identity solutions?

It took me too long to conclude that SSI is the inferior identity solution. The market determines winners and losers, better and worse — and there are important reasons the market is rejecting verifiable credentials in favor of proprietary solutions.

After 4+ years of hypothesizing, experimenting, evaluating, and iterating, I found what I consider to be intractable problems that block adoption of verifiable credentials. I do not know the answers to these problems. If you’re building in the IDtech space, you should take these problems seriously because, in my view, these are the billion-dollar questions you’ll need to crack to have any meaningful success.

Impossible problems

Impossible problem #1: Interoperability is the premise, but doesn’t exist

Standards exist to enable compatibility. If interoperability isn’t important, simple alternatives to verifiable credentials exist (such as signing — or not signing — JSON). The ability to share data with an arbitrary 3rd-party in a standardized way (without bespoke integration or direct connection to issuing source) is the core purpose of VCs. But that doesn’t actually exist right now.

In other words, the reason for the existence of VCs is not present.

This is because interoperability at the application level requires technical interoperability of keys/signatures, public key resolution, exchange protocols, request/response formats, data models, semantics, and a host of underlying elements. (Not to mention nontechnical interoperability elements, like business models and trust frameworks.) Because a myriad of VC variants, exchange protocols, signature suites, etc. have emerged over the years, which must all line up to effectively interoperate, the only practical way to achieve interoperability with others is to use the same software.

Theoretically, if everyone in the industry agreed upon a single standard to use, interoperability would be present, and there would begin to be value to adopting VCs today — but there’s a problem: which standard should be adopted? and what if it’s the wrong one?

Prematurely standardizing is a risky endeavor. But in order to find out what kind of digital credential will have real product/market fit — and thus be adopted — the world needs to experiment, iterate, innovate, over trade-offs to discover what the market will adopt.

Do you see the problem? VCs need innovation, iteration, and experimentation to succeed. But they also need standardization, agreement, buy-in, and stagnation to have value. These two forces are directly opposed. I wrote about this dynamic more in a post several years ago.

An image I made in 2022 for a post I wrote about the tension between interop & innovation.

Standardization is essential for adoption. But adoption is essential for knowing what to standardize (there’s nothing worse than standardizing the wrong thing)! Prematurely standardizing before adoption is a classic “cart before the horse” scenario.

To be clear, none of this means that there is no reason to adopt VCs now — it means every reason to adopt VCs today is in anticipation of future value. Because today, verifiable credentials offer zero marginal value relative to simpler alternatives. To rationally adopt VCs, one must make a bet on the future. Some bold companies will be willing to take that very costly journey (like we did), but most will not.

Impossible problem #2: Initial UX is worse than baseline

Verifiable credentials are always a better UX than existing identity solutions at scale, but always a worse experience initially.

I had a mask-laden conversation with the late Vittorio Bertocci at the Internet Identity Workshop in 2021 where I was working to convince him of the merits of VCs as he advocated for federated approaches.

His point: VCs aren’t necessary to share data outside its original domain. Take, for example, Airbnb wanting to verify a digital driver’s license. Instead of using a VC, DMVs could become an IdP, and Airbnb could add a “verify with X driver’s license button” into their app. The tech exists today.

My point: Yes, but nobody does that today because it doesn’t work in person, doesn’t work offline, doesn’t work when presenting multiple credentials, doesn’t work when privacy needs to be preserved, etc. But most importantly, in this example Airbnb would need to add a “verify with X” button for all 50 states, which is a very bad UX — VCs would enable one universal verification button!

I’ve thought a lot about this interaction. While I like to think I helped nudge Vittorio a bit on the value VCs could provide, you’ll notice my points are basically only relevant at scale.

For a concrete example, take AI agents who need identity wallet capabilities. We ran a whole campaign about this at Trinsic and spoke to dozens of AI agent developers (see this landing page, watch the demo video!). It’s obvious the problem exists in the long-term. As a user, it’s a pain to provision an agent access to all my accounts, info, etc. needed for it to accomplish real tasks for me. But in the near-term, 100% of the AI developers we talked to just chose to use OAuth to connect to sources where users’ data exists already (e.g. Google).

If a user already has a wallet with a bunch of useful credentials in it, obviously sharing those is a better UX than redoing the onboarding process that was required to obtain those credentials. But acquiring the wallet with credentials in the first place will always cause extra friction upfront, which businesses and users are very reluctant toward.

This problem is compounded because businesses don’t just adopt things that are a little bit better than alternatives. In order to switch behaviors or risk their careers on unproven technologies, new solutions need to be better enough to justify that extra investment. And unfortunately for VCs, for every initial use case I’m aware of, the experience is objectively worse.

I shared more insights from Vittorio here.

Impossible problem #3: Heat map is too scattered

When I was raising money for Trinsic as a general-purpose verifiable credential platform, investors would often ask what the most prominent use case for verifiable credentials was — and more specifically, what the most common use case for Trinsic’s customers was. My answer sucked:

Our customers are building use cases in education, employment, financial services, healthcare, insurance, industrials, government, and agriculture. And more!

Once, in pursuit of a better answer to this question, we plotted the industries, geographies, and use cases of 100+ paying customers. The scatter plot was all over the place. Nothing stuck out prominently. It didn’t make sense to double down on a specific industry or use case because none dominated our demand. We didn’t even have geographic consistency — adoption existed from Myanmar to Peru to USA to Romania.

My cofounder Michael continuously sounded the alarm in my ear: in order for VCs to take off, a network effect needs to take hold. No network effect will exist if all adoption is completely scattered.

This problem doesn’t just affect Trinsic, it affects the entire VC space. Until pockets of adoption breed more adoption in a virtuous cycle, the rate of adoption will be capped to a linear value — and adoption to 1+ billion people requires exponential rates.

To drive this point home, I’ll cite an example from a packed room at a previous Internet Identity Workshop, circa 2020. In a room full of decentralized ID vendors, I asked for a raise of hands: “Who has ever heard of a verifiable credential being used for a purpose other than that for which it was issued?” (In other words, who has ever heard of a verifiable credential being used for the reason the VC standard was created.)

No hand went up.

I asked again in 2024. Still crickets.

Verifiable credentials won’t take off until this sort of thing is commonplace. This cross-context data sharing simply won’t happen until both sides of the exchange are interested in similar credentials, or are from a similar industry or geography. In other words, this won’t happen until the heat map of potential use cases lights up in a particular spot. But the current nature of the market is very scattered.

Bull case

This post outlines why I’m not betting on verifiable credentials taking off anytime soon. But if I had to articulate the bull case, there are two arguments here: government mandates and timing/direction of causality.

Governments

Governments represent the silver lining for verifiable credentials. While, in my view, the private sector lacks a viable adoption path for the reasons outlined above, governments have different incentives. Governments build utilities, and there is a good argument that interoperable digital ID is a government utility. Many governments in the West are investing in VCs, most notably eIDAS 2.0. Although even that is far from settled.

Government involvement is not limited to government-issued credentials. Virtually every private sector company in the verifiable credential world (except Trinsic, for what it’s worth) has been subsidized by governments funding their preferred VC standards (DHS funding different standards than the EU, for example). Various technologies through the years have succeeded due to government mandates or subsidies–nothing wrong with it inherently–but it is a telling aspect.

Governments compelling use of interoperable digital credentials represents the most likely success case for VCs I can think of. But that scenario exists in the medium-term, not the short-term.

Timing

The other argument says that verifiable credentials are simply too early. Many of the leading reusable ID ecosystems like IDme, Yoti, CLEAR, etc. were started and initially grew before the VC standard was finalized/stabilized. So one could argue these companies, if started today, could have had the same or greater success using verifiable credentials rather than a proprietary solution.

Unfortunately, recently companies like Incode, Footprint, Plaid, and many others launched “reusable ID” products and chose not to use VCs after extensive research. And have succeeded far beyond most SSI products. Which presents a difficult rebuttal to this argument.

One cannot rule out the possibility that companies can succeed betting on the VC standard. But it’s difficult to point to examples of VCs helping a business solve problems for customers (and thus make money) — and it’s trivial to point to dozens of examples of companies sinking millions of dollars into verifiable credential infrastructure (money which could have been spent acquiring customers). One hypothesis is that tinkerers are attracted to VCs while aggressive operators are less likely to experiment on new technologies or make speculative bets on the future.

End

I referenced Vittorio earlier, and I’ll cite him one last time here. In a blog post about his involvement in the CardSpace project at Microsoft, which is a spiritual ancestor of modern SSI, he said:

“When the user centric identity effort substantially failed to gain traction in actual products, with the identity industry incorporating some important innovations (hello, claims) but generally rejecting many of the key tenets I held so dear, something broke inside me. I became disillusioned with pure principled views, and moved toward a stricter Job to be done, user cases driven stance.”

I’ve not changed my convictions on the future I want to live in. Nor has my resolve wavered in my desire to build that future. But the market has spoken. Prematurely standardizing a pre-product/market fit technology won’t work — no matter how badly zealous advocates of SSI want it to. That is why Trinsic is no longer a general-purpose verifiable credential infrastructure company.

While I don’t expect a universal standard for digital IDs will happen anytime soon, new digital IDs continue to roll out every day. These digital IDs are sometimes built with VCs, other times with mDLs, but usually in proprietary ways. Some are bound with biometrics, others with passkeys. Some are focused on solving a problem in a specific industry, others in a specific geography.

Today, the Trinsic Identity Acceptance Network represents the future of Trinsic. Instead of helping companies create digital IDs (issue verifiable credentials or create wallets for users), we help businesses accept existing digital IDs. Our strategy is a bet on continued fragmentation of the ID landscape — a bet on no single standard winning out.

In fact, I’ve become convinced that Trinsic’s current strategy to help businesses accept digital credentials is actually a necessary stepping stone to a future world powered by interoperable digital credentials. We’re still ushering the world in that direction, just from a different angle.

Of course, I could be wrong, and I hope I am. My ears and eyes are wide open for new information. My only objective is to see reality as clearly as I can. If you have information you believe I’m missing, I’d love to hear from you and have a conversation. Sincerely! You can DM me on LinkedIn or Twitter, or shoot me an email riley@trinsic.id.

Thanks to Kim Hamilton Duffy, Gerald Glickman, Timothy Ruff, Eve Maler, James Monaghan, Dave Grantham, and Zachary Jones for giving feedback on a draft of this post.

16:49 • 8 days ago

The Pragmatic Engineer

Why techies leave Big Tech

A job in Big Tech is a career goal for many software engineers and engineering managers. So what leads people to quit, after working so hard to land these roles?

Hi – this is Gergely with the monthly, free issue of the Pragmatic Engineer Newsletter. In every issue, I cover challenges at Big Tech and startups through the lens of engineering managers and senior engineers. If you’ve been forwarded this email, you can subscribe here.

In case you missed it: the first two The Pragmatic Engineer Podcast episodes are out: Efficient scaleups in 2024 vs 2021 and AI tools for software engineers, but without the hype. Each episode covers approaches you can use to build stuff – whether you are a software engineer, or a manager of engineers. If you enjoy podcasts, feel free to add it to your favorite player.

Ask a hundred software engineers what their dream company is and a good chunk are likely to mention Google, Apple, Meta, Microsoft, and other global tech businesses. This is no surprise:

Brand value – few businesses in any sector are better-known than Big Tech

Compensation – pay is usually at the very top of the market. Ranges can get especially high in staff-and-above positions. We cover more on pay in The trimodal nature of tech compensation and in Senior-and-above compensation in tech

Scale – infrastructure used by hundreds of millions or billions of users, data storage measured in petabytes, and services which handle requests at the thousands per seconds, or above

With upsides like these and others, why walk out? To learn more, I asked several software engineers and engineering leaders who did precisely this. Personal experiences vary, but I wondered if there are any common threads in why people quit prestigious workplaces. Thanks to everyone who contributed.

In today’s deep dive, we cover:

Big Tech less stable than it was. Big Tech had few to no mass layoffs for years – but that’s all changed.

Professional growth in a startup environment. It’s hard to keep learning in some Big Tech environments, compared to at startups.

Closed career paths at Big Tech. It’s often more achievable to get to executive-level engineering positions at scaleups, than at global companies.

Forced out. The bigger the company, the more there’s politics and scope for workers to be victims of competing interests, personalities, and rivalries.

Scaleups get “too Big Tech.” Scaleups are nimble and move fast with few processes. Successful scaleups slow down and add more process.

Steep compensation drops. A falling stock price can make people consider leaving if it leads their compensation to also reduce. Also, when an initial equity grant vests out after 4 years.

Raw feedback. A former engineering leader at Snowflake shares their take on how people leave workplaces – or struggle to; golden handcuffs, a Big Tech hiring slowdown, a bifurcated market, and more.

Some reasons techies leave large and prestigious tech companies 1. Big Tech less stable than it was

Luiz Santana was a tech lead manager at Google in Germany, before leaving to cofound the health tech startup, Digitale Patientenhilfe. Before Google, he was a senior engineering manager at fintech N26, and head of engineering at ride-sharing app, FREE NOW. Luiz shares why he decided to say farewell to what looks like a techie’s dream job:

‘Some things helped me make the decision to leave Google:

The startup I got an offer from had raised healthy funding, meaning a good amount of runway

I managed to negotiate to join as a cofounder and CTO, which meant a healthy equity package.

The other two cofounders have a great track record with a previous startup. This gave me confidence.

‘Google changed a lot, which also made the decision easier:

Google had layoff tension at the time. In mid-2023, there were lots of small, unpredictable layoffs, which made Google feel less of a “secure” place to work.

The culture also changed visibly. There was cost cutting, ranging from small things like reduced snack selection, to some teams having trouble securing servers they needed for organic growth.

I realized I am no fan of promotion-driven culture, which I saw to result in outcomes I didn’t like.

‘Google makes it hard to leave. Some big factors held me back from quitting:

Compensation. The biggest challenge, by far! Google pays very well, and few if any companies can match the total package. In my case, I received my stock grant at half the stock price to what it was vesting at. This meant that my equity compensation was already worth double!

Brand. I have always been a big fan of Google products, and being associated with Google is a big positive in so many ways.

Risk. Staying at Google is lower risk – even with layoffs – than joining a startup is.

‘Personal circumstances made the decision to quit easier.

I had “layoff fatigue” keeping track of all the small layoffs in various teams.

In Germany, I was at higher risk of being laid off because I was not married at the time and do not have kids. There’s a “social criteria” for redundancies in Germany, and I was not in a protected bracket; if a layoff happened, I was a prime candidate.

I trusted the founders, and appreciated that they spent a lot of time with me, sharing their vision.

‘After a lot of back-and-forth, I finally pulled the trigger to join the startup. I’ve not looked back since!’

Luiz’s hunch about diminished job security echoes the reality. Since mid-2022, Big Tech has shattered its image for job security:

Meta let go ~25% of staff in 6 months in two separate layoffs. Before November 2022, the company had never done mass redundancies; then it did two.

Google never did repeat annual layoffs in its history until in 2024, following shock layoffs in 2023

Amazon made deep cuts in 2023. The company is also known for less job security due to using PIPs more than most other Big Tech companies. PIPs are used as part of meeting unregretted-attrition targets

Microsoft did large cuts in 2023 and small layouts since

Tesla did widespread layoffs in secret in 2022, hitting long-time employees with a 1-week severance package.

Apple and NVIDIA are the only two Big Tech companies not to do mass layoffs in the last two decades.

2. Professional growth in a startup environment

Benedict Hsieh is a software engineer based in New York City, who spent five years at Google, before quitting for a startup in 2015. Ben describes his journey:

‘I didn’t want to become a ‘lifer’ at Google. This was the specific reason I left Google: I felt I was headed in a direction of being stuck there for life. I was only learning Google-specific tech, and the position was not very demanding. I felt like I should be working harder and learning to create value on my own, instead of only functioning as a cog in the machine.

‘I’d stopped “exploring” and was mostly “exploiting.” There is a mental model I like called the explore-exploit tradeoff. Exploitation means you choose the best option based on your current knowledge. Exploration means you try out new technologies and approaches. Reflecting on my day-to-day work, it felt that almost all of it was “exploiting,” and I was doing very little “exploring.” It was too early in my career (and life) to stop exploring!

‘I think my mentality of worrying about not doing enough “exploring” is rare. Almost all my former peers are still at Google because the total compensation is really, really hard to beat!

‘Looking back, I was overconfident about how quickly I would grow in startup-land – both professionally and in the financial sense. I was willing to take the hit on significantly decreasing my total compensation, and getting a larger chunk of startup equity. I was impatient about hitting my “retirement number” by joining a fast-growing startup with much higher upside.

‘Also, to be frank, I figured that I could go back to working at Big Tech anytime I wanted: because I spent enough years there, and had a pretty good existing network.’

Ben joined a startup as a cofounder. The experience was not what he expected, as he wrote about:

‘I was miserable. We were working out of [my cofounder’s] unfinished apartment which was freezing cold in the middle of the winter and a constant reminder of all the things that weren't going well. I'm a low-conflict person who needs hours to calm down after an argument, where she preferred communicating via loud debate.

‘I was trying to learn all kinds of things that we needed for our business – how to work with clients, keep our servers up at all hours by myself, debug statistical anomalies in our data, or send out cold emails to find new business. I was the only one who could do these things, so I got them done. I woke up early in the morning and had trouble sleeping at night. Once I worked past midnight to compile a report for a client who'd requested a last-minute meeting in the morning, only for them to no-show, followed by an email two days later asking me why I hadn't found another way to send them their data. If I had asked my body what it wanted in that moment, it surely would have responded with incoherent screaming. It basically did that without being asked.

‘Our company folded in less than a year.

‘But in eight stressful and mostly unpleasant months I accomplished more than I had in the eight years before that. We made some money for our clients, and a minimal but nonzero amount for ourselves, and I was able to parlay the experience into an early position at a much more successful startup. More importantly, I learned how to just get things done when they need to be done, instead of feeling like a helpless bystander watching a car crash.’

Ben reports that the new startup he is working at is doing a lot better, and reckons he needed a “startup shock” to develop his professional skills beyond the (comparatively) neat and tidy confines of Google.

3. Closed career paths at Big Tech

A product manager based in Seattle worked in Big Tech for 14 years: 3 at Amazon, and 11 at Google, where they went from a product manager on a single product, to senior product manager, group product manager, and product lead for a portfolio of products. Despite promotions into influential positions, they quit the search giant for a fintech startup, as VP of Product. They asked to remain anonymous, and share:

‘I'd already decided to quit Google without a new gig lined up. This was because I couldn't find a new role that was a combination of interesting challenge, interesting people, and/or one that fulfilled my career goals. I had over 50 conversations inside Google for ~9 months.

‘I talked to many ex-Googlers and ex-Amazonians during interviews. I'd never heard of my current company prior to joining, but most people I met during the interview were ex-Googlers/Amazonians. They were tackling the worthy, difficult problem of building a truly modern fraud monitoring and management platform.

‘This company isn't a remuneration leader by any means. "Closing" a candidate – them accepting an offer – is a combination of:

A strong “sell” during interviews

Showcase the concentration of world-class talent at the company

Highlight that the team ships very fast – much faster than Big Tech!

Articulate interesting technical and product challenges the team overcomes

‘Despite not knowing about them, it turns out this business has a strong brand in the banking software sector. They have established business moats, and the more I learned, the more impressed I was.

‘The company is in the middle of an organizational turnaround that I get to be an active part of, as a VP. This challenge appeals to me because I get to work with a really motivated set of people who are focused on making a big difference within the company, but also across the financial industry.’

This journey from Big Tech middle-management into leadership at a scaleup, makes a lot of sense. Making the jump from engineering manager or product lead, to an executive position, is close to impossible at Big Tech because the change of scale is vast. An engineering lead might have 10-50 reports, but a VP or C-level will oftentimes have 10x more. There are exceptions, of course, like Satya Nadella, who rose through the ranks at Microsoft, from software engineer, through vice president, to CEO. But in general at large companies, getting promoted to the executive level is formidably difficult. Scaleups offer a more achievable path to C-level.

At the same time, tech professionals with managerial experience in Big Tech are often perfect fits for senior positions at scaleups. Recruitment like this can be a true win-win! A new executive gets to learn a lot by getting hands-on with strategy, attending behind-the-scenes meetings, liasing with the board and investors, and many other experiences that are simply off limits at Big Tech.

In exchange, the scaleup gets a seasoned professional who doesn’t panic when facing decisions potentially involving tens of millions of dollars, and who can make correct, well-informed decisions – which is what Big Tech managers do, usually.

4. Forced Out

Working at Big Tech is far from perfect; the larger the company, the more organizational politics there is, some of it bad.

Justin Garrison, former senior developer advocate at AWS, felt this after he posted an article that criticized the company, entitled Amazon’s silent slacking. In it, he wondered if Amazon’s sluggish stock price was the reason for its strict return to office (RTO) push, and whether it was a way to quietly reduce headcount via resignations. Justin shared other observations in the article:

“Many of the service teams have lost a lot of institutional knowledge as part of RTO. Teams were lean before 2023, now they’re emaciated.
Teams can’t keep innovating when they’re just trying to keep the lights on. They can’t maintain on-call schedules without the ability to take vacation or sick days.
The next logical step to reduce costs is to centralize expertise. It’s the reason many large companies have database administration, network engineering, or platform teams.
They’ll have to give up on autonomy to reduce duplication. Amazon has never had a platform engineering team or site reliability engineers (SRE). I suspect in 2024 they’ll start to reorg into a more centralized friendly org chart.”

Justin’s team was also hit by layoffs: his team was eliminated, but not his role. He was left in a limbo state of needing to find another role within the company, and was not offered severance. Justin suspected Amazon was aiming to avoid paying severance packages, and incentivised managers to put engineers on a performance improvement plan (PIP) and let them go without severance.

In the end, Justin didn’t want to go through what he predicted would be a demotivating, unfair process that would end in him being fired. So, he quit.

Afterward, he joined infrastructure startup Sidero Labs as head of product, building what they aim to make the best on-premises Kubernetes experience.

Ways out of Big Tech manager conflicts

There’s a saying about quitting that “people don’t leave bad companies, they leave bad managers.” It contains a kernel of truth: a bad manager is often reason enough to leave because it’s the most significant workplace relationship for most people.

At large companies, there is an alternative: internal transfers. As an engineer, if you feel held back by your manager or team, you can attempt to move. Internal transfers are usually a lot less risky– as someone changing jobs – than interviewing externally. With an internal transfer, you get to keep your compensation and network inside the company; in fact, you grow it. Also, your knowledge of internal systems and products is valuable.

There are usually a few requirements for an internal transfer to happen:

Minimum tenure: internal transfers are open to those at the company or in their current team for a year or more.

Good standing: performance reviews which meet expectations are needed to get to move, usually. This is to avoid low performers escaping to switching teams. Being on a performance improvement plan (PIP) is a blocker to moving at most companies.

Other teams’ headcounts: internal transfers can only happen when teams have the budget for your level. Internal transfers are a way to hire more efficiently.

Pass an interview: at many companies, internal transfers go through an internal interview. This is usually a lot more lightweight than external ones. The process usually depends on the manager. It might be a simple chat and review of your existing work, or be more competitive if there are other candidates. For example, at Microsoft/Skype, when I changed teams as a developer, my new manager had internal candidates do a software architecture interview.

Get approval from the existing team. At some places, this can be a thing! An existing manager can slow down a transfer, or even sometimes veto it. However, in practice, if an engineer and manager have a poor relationship but the engineer has decent standing, then the manager doesn’t have much reason to block their departure. Of course, a manager may be able to make the situation challenging enough that seeking opportunities externally seems like the better option.

5. Scaleups get “too Big Tech”

An engineering leader spent four years at Snowflake after joining in 2019, right before its IPO. They’ve asked to remain anonymous, and share why it was time to depart the data platform:

‘Snowflake became “too Big Tech” for my liking. When I joined, there was a lot of uncertainty within the company and teams moved quickly. We had to make rapid changes, and four years later, things looked different:

Stable teams

Mature and well-documented processes

Lots of internal committees

Ever-growing amount of documents

Endless program management work before starting anything meaningful

Lots of politics! Cliques formed and there was “empire building” in upper management.

‘I have to admit, none of this is for me; I’m more of a “move fast and build things” person. At the same time, I acknowledge that many people felt very comfortable with these changes, and thrive in them!

‘The reality is that the company became successful, quickly. I enjoyed being part of the ride and helping create this success, but the change in culture made it feel less close to me than the “old” culture.

“Working at a scaleup that became “Big Tech” made it so much easier to leave! I’m certain that having Snowflake on my resume gave me a huge head start on someone equivalent from a medium or lower tier company. If I didn’t have Snowflake on my resume, recruiters would have skipped over me, and hiring VPs would be extremely skeptical.

‘So while there have been lots of changes in culture thanks to the standout success of Snowflake, it gave a lot of career options to me and everyone who helped build Snowflake into what it is today.’

6. Steep compensation drops

Big Tech compensation packages usually have three components:

Base salary: the fixed sum in a paycheck

Cash bonus: awarded at the end of the year at some companies. Netflix is among the companies which do not award bonuses

Equity: awarded as an initial grant that vests over 4 years, usually. Most Big Tech companies offer equity refreshers

The more senior a position, the more of the compensation is in equity. Tech salary information site Levels.fyi maps how Microsoft’s positions offer considerably more equity, and how principal-and-above engineers usually make more in equity per year than in salary:

Microsoft’s typical US compensation packages for software engineers. Source: Levels.fyi Rising stock prices make it hard to hire away from public companies

Equity is converted from a dollar amount to the number of stocks on issue date. This means that if the stock value increases later, so does the grant value. If the stock goes down, so does the grant value, and total compensation with it.

This connection is why it’s close to impossible for a company to tempt NVIDIA employees to leave the chip maker, if they joined in the past four years and are still vesting out their initial grants: NVIDIA stock is worth 10x today than 4 years ago. So, let’s take an engineer who joined in October 2020 with a compensation package of $250K per year:

$150K base salary

$400K in equity (vesting $100K/year on the issue date)

Four years later, this engineer’s 2024 total compensation is around $1.15M, thanks to stock appreciation:

$150K base salary

$1M in equity vested in 2024 (thanks to that $100K/year grant being worth 10x, $1M/year!)

Falling stock price: big incentive to leave

Stock prices don’t only go up, they also go down; and when they do the equity value of comp packages drops significantly. We previously covered how low stock prices lead more people to leave listed tech companies in May 2022. From The Pulse:

Some tech stocks have had a terrible past 12 months. Here are some of the tech companies which have seen their stock prices decrease the most since a year ago:

Stitch Fix: -79% 📉

Redfin: -71% 📉

Pinterest: -65% 📉

Toast: -64% 📉

Robinhood: -61% 📉

Zoom: -61% 📉

Roku: -60% 📉

Opendoor: -56% 📉

Docusign: -48% 📉

In comparison, some Big Tech have done well:

Nvidia: +107% 📈

Tesla: +63% 📈

Apple: +47% 📈

Google: +37% 📈

Microsoft: +34% 📈

Oracle: +20% 📈

Let’s take a senior software engineer who offered a $350K/year package in March 2021. Let’s assume they got this compensation package at all of the above companies, and that the package consisted of:

$200K cash compensation (e.g. $170K base salary, $30K bonus target)

$150K/year stock compensation ($600K in stock, vesting over 4 years).

Here’s what their compensation would look like, assuming no cash compensation changes:
Stark difference in compensation outcomes due to stock performance

Back when these stock drops happened, my suggestion was this:

“If you’re an engineering manager at a company where the stock has dropped significantly: buckle up for a bumpy ride. Unless your company can deploy significant retention grants, you will likely see record attrition in the coming months. Make cases for these retainers, but know that companies have financial constraints: and this is especially the case if the stock underperforms for a longer period of time.
If you’re looking for a new position: at places that issue equity, you’ll need to take a bet on the trajectory of the company. Consider companies where you believe in the company, their products, and how those products will grow over the next several years.”

Over time, Big Tech stock has done much better than many recently IPO’d tech scaleups. The biggest stock drop happened at Meta, at the end of 2022. In just 6 months, the company’s stock price dropped from $330 to $88 – a 70% drop! Everyone who joined before 2022 saw their stock grants lose 50-70% of value on paper. Recovery was uncertain:

Meta’s stock price 2019-2022. A drop from mid-2022 hit equity packages

That year was probably one of the best times ever to hire away from Meta, due to its reduced stock price dragging down overall compensation. From early 2023, Meta’s stock rapidly recovered; employees’ issued with stock in 2022-2023 have seen its value multiple. From a total compensation point of view, it’s again hard to hire away from Meta:

Meta’s stock price since November 2022. Rising value means extra total compensation

We covered equity refresh targets per level in the US in Inside Meta’s engineering culture.

Four-year cliff

An event that frequently reduces compensation is the four-year vesting cliff, when the initial equity grant runs out at Big Tech. At senior engineer-and-above, and engineering-manager-and-above positions, these initial grants can be significant. It’s not uncommon for more equity to vest per year during the first four years of the initial grant vesting, than total compensation. The problem is that when this initial grant runs out, the compensation drops because the company does not “top up” with a similarly generous grant. This can mean a 10-40% drop in total compensation – pretty demoralizing!

As a manager, I dealt with the problem of engineers hitting 4 years’ tenure, and their annual earnings dropping 25-30%. The same happened to my own compensation package: in year 5 at Uber, I would have made about 30% less than in years 1-4, due to the initial equity grant running out, and lower annual refreshers. In the case of Uber, the stock price stayed relatively flat, and the drop in pay was the difference between revised compensation bands, and the equity which joiners had managed to negotiate.

Some Big Tech companies make the “cliff” less steep. Speaking with an engineering leader at Meta, they told me the annual refreshers offered at L6-and-above levels (staff engineer equivalent and above) are usually large enough to ensure no major compensation drop.

However, there are also companies like Amazon where only top performers receive top-up equity. This means that after four years, those without equity awards see a major compensation drop, as the compensation then only comprises salary, as Amazon doesn’t do cash bonuses. When this happens, it’s a signal that Amazon doesn’t particularly want to retain someone. It’s common for engineers to start applying externally when their equity is set to run out.

When a company’s stock price keeps increasing, the 4-year cliff becomes more painful. In Big Tech there are compensation targets for every engineering level. People earning above this target get very little or no equity refreshers, as they are already above target.

Going back to the example of NVIDIA, and the imaginary software engineer on $250K/year in 2020 ($150K salary, plus $100K/year stock), who’s on track to make $1.15M in 2024, thanks to NVIDIA’s stock price increase. That software engineer could see their compensation drop from $1.15M in 2024, to $150K in 2025, assuming no further equity refreshers. Even with an equity refresher of $400K over 4 years, their compensation will still drop from $1.15M in 2024 to $250K in 2025!

As a tech worker, it’s easy enough to rationalize that current compensation is outsized compared to other sectors; but you don’t need to be psychic to understand that a pay cut is demotivating; people are doing the same job as before for less money.

Assuming our engineer managed to save most of their gains from the incredible stock run, they might have a few million dollars in savings. This creates room for taking a risk, such as:

Joining another company for higher compensation (very small risk)

Joining a startup for lower compensation package but more equity (moderate risk)

Cofounding a startup, taking a steep cut on compensation, but a high equity stake (high risk)

7. Raw Feedback

The engineering leader who left Snowflake for becoming “too Big Tech” interviewed with several startups, and is in touch with peers still working in Big Tech. They share some unfiltered observations about people considering leaving big companies

Golden handcuffs

'Golden handcuffs' are a big thing at companies like Snowflake. I know plenty of people who are still riding out significant equity grants from the last few years that increased several times in value.

‘Salaries have stagnated across the industry, though. Back at Snowflake, we hired some people who were overpaid, compared to the current market. I know this because I hired some of them! We offered above the market because in 2021-2022 we were desperate to fill positions, like everyone else!

‘This is the problem with golden handcuffs: when you are lucky enough to have them, it’s hard to find anywhere offering more because you’re already above the market bands! So the only way to avoid a compensation cut is to stay.

Hiring slowdown

‘I have seen a slowdown in hiring across the tech industry, mostly at bigger companies. It also impacted people at the “lower end” of experience and domain expertise. If you are a very experienced engineer or engineering leader, or have some specific skills/knowledge that is in demand, the market is good in 2024!

‘Non-listed companies are still hiring more than public ones. I’ve talked with a decent number of strongly-growing companies and most want to hire experienced people.’ This observation tallies with one from the deep dive in August, Surprise uptick in engineering recruitment

‘I’m an example of the demand for experienced people. I have not been actively looking for jobs – but out of curiosity, I made myself open to inbounds from recruiters on LinkedIn. In two months, I had interviews with engineering VPs for series C and D companies. I am actually going to NYC next week for a half-day onsite as the final step for one role with a series D. I haven't actually actively applied to any jobs while doing so!

Bifurcated market

‘The current job market seems to be divided into two parts:

Experienced folks: If you are a senior, experienced person, especially with in-demand skills, there are options and the market is still moving steadily, if a bit slower than before

Junior folks: if you are more junior, or don't have unique experiences or skill sets, you are probably not going to see many opportunities in the current market

Risk takers favored:

‘There are two types of people when it comes to taking risks:

Builders and risk takers: people who like to build and grow teams and programs, who like taking risks, and jumping into the unknown with a bit of chaos. I’m someone who thrives on that; I get bored easily!

Incremental improvers seeking stability. Many people like to run things and make incremental improvements, from one stable job to another stable job.

‘In the current environment, big and stable companies are not hiring so much. So the people getting jobs are willing to take risks with less predictable companies, and jump into some chaotic situations.

Tech industry becoming ‘tiered’

‘An article by The Pragmatic Engineer covers the ‘tiering’ of the tech industry, which I experienced at first hand.

‘At my job before Snowflake, I was around “mid tier” at a financial software company. I would have been stuck in this “tier”, but got lucky in that Snowflake was desperate to hire tons of people in 2019.

Joining Snowflake immediately catapulted me into a much higher compensated group. Beforehand, I did not appreciate how massive the gap is between mid and top-tier companies! But I’m torn about this gap. On one hand, I really appreciate the compensation and career options. On the other hand, it irritates me how insular, incestuous, and hypocritical this is.

‘The upper tier literally feels like an old European aristocracy – and I’m saying this as someone who lives in the US! People help out their buddies, and are extremely suspicious of anyone not in their ‘club.’ It’s eye-opening to see how many people jump from company to company, taking their buddies with them. They all make lots of money, while keeping it exclusive and making sure it stays that way.’

Takeaways

Thank you to everyone who contributed to this look into why successful tech workers quit the most successful tech employers. When I joined Uber in 2016, it felt like the best-possible place I could have onboarded to. Back then, Uber had very positive media coverage, was called the most valuable startup in the world, and was the quickest to scale up in history. And yet, when I joined on the first 1:1 with my manager, the question I got from this was:

“So, what are you planning to do professionally after Uber?”

It was day one at the world’s most valuable startup; why was my manager asking about what I’ll do after this job? They later explained this question was because he’d been in the industry long enough to know that 99% of people don’t retire at their current company, and he wanted to be a supportive manager for future career goals. So if someone told him they might try to do a startup one day: he would try to get them involved in projects where they can do more zero-to-one building. If someone said they would like to get to a VP of engineering role at a scaleup later, he’d try to help them grow into a people manager. Everyone eventually leaves even the fastest-growing scaleups, or the most coveted Big Tech.

A smaller group departs into retirement, more commonly at companies like Microsoft and Amazon, where some engineers spend decades. But most people leave for other companies.

I hope the half dozen accounts from tech professionals who left Big Tech provide a sense of why people decide the most prestigious workplaces in tech are not for them.

Working at Big Tech can make leaving it much easier. This is counterintuitive because Big Tech pays so well, and the biggest reason against leaving is the compensation cut – at least in the short-term. However, the high pay allows people to save up a nest egg much faster, which provides the financial freedom to do something more risky like joining a startup and betting that the equity package will grow in value, or just taking a pay cut to join a company with more interesting work, or which they are passionate about.

Some people never stop growing professionally. A common theme in these accounts is feeling stagnant; most people felt they weren’t growing or being challenged. Some left because of frustration about doing more administrative busywork and less building.

Working at Big Tech is often a final goal, but a job in this elite group of workplaces can also be a stepping stone for pursuing new ambitions. I hope these accounts shed some light on the decision-making process and serve as a reminder that engineering careers are also about the journey, not just the destination.

16:23 • 8 days ago

Monday, 14. October 2024

Doc Searls Weblog

What goes in these structured wiring cabinets?

I need to install gear in these two structured wiring cabinets in the garage of the new house we are finishing. I don’t know exactly what to put in them and seek advice. The installed cables are: Blue CAT-6a Ethernet cables go to outlets (RJ-45 jacks) in four rooms. Internet will come from the city’s […]

Two in-wall structured wiring cabinets in our new garage.

I need to install gear in these two structured wiring cabinets in the garage of the new house we are finishing. I don’t know exactly what to put in them and seek advice.

The installed cables are:

Blue CAT-6a Ethernet cables go to outlets (RJ-45 jacks) in four rooms. Internet will come from the city’s new fiber optic system. Coaxial cables go to four possible TV locations. They will carry signals from the over-the-air TV antenna that will go on a pole outside. We will not have cable TV or Internet service.

Soon to be installed are:

A coaxial cable from the TV antenna. A fiber cable from the street.

Both will come underground and up into the garage through a conduit between the back of the house and the hole (not visible) in the left side of the left cabinet.

So here is what I think I need:

A patch panel for the coaxial cables, so I can either distribute the TV signal or patch it through to one TV at a time. An ONT (optical network terminal) to serve as the router between the fiber from outside and the four ethernet connections inside.

Here is what I don’t know:

What the fiber provider (GigabitNow Bloomington) will provide. I know they will need to terminate fiber from the street with an ONT that will go in one of the cabinets, but I don’t know whether they will provide, or I will need to get, a way to distribute service to the four Ethernet lines. I see ONTs sold online that have four Ethernet ports, but I don’t know if the provider will welcome my getting one or not. Whether I will need an in-line TV signal amplifier. My antenna, which I’ve already built and tested, has an amplifier already. The question is whether I will need more amplification, especially if I am distributing to more than one room, and not just patching through one room at a time. Exactly what the best practices are for attaching devices to the inside of these cabinets.

So that’s it for now. Thanks for any input you’ve got.

23:32 • 8 days ago

IdM Laboratory

ISO/IEC 18013-7が発行されました

こんにちは、富士榮です。マイナンバーカードとか免許証など、mDL/mdocの話題がつきませんが、そういえばISO/IEC 18013-7、Mobile driving license (mDL) addon functionsがリリースされました。 https://www.iso.org/standard/82772.html 全然どうでもいいんですが、イギリス英語なんですね。。。「licence」なんとなくlicenseって思ってましたがタイトルはlicenceでした。

こんにちは、富士榮です。

マイナンバーカードとか免許証など、mDL/mdocの話題がつきませんが、そういえばISO/IEC 18013-7、Mobile driving license (mDL) addon functionsがリリースされました。

全然どうでもいいんですが、イギリス英語なんですね。。。「licence」

なんとなくlicenseって思ってましたがタイトルはlicenceでした。

22:30 • 8 days ago

@_Nat Zone

デジタルアイデンティティ：世界の最新動向 – 2024年10月15日

今週のレポートでは、世界中のデジタルアイデンティティに関する最新の動向を探ります。アメリカの新しいサイバーセキュリティツールから発展途上国のデジタルID計画まで、世界各国がさまざまな分野でデジタル変革を進めています。アメリカ：SpyCloudの新しいサイバー犯罪調査ツールサイ…

今週のレポートでは、世界中のデジタルアイデンティティに関する最新の動向を探ります。アメリカの新しいサイバーセキュリティツールから発展途上国のデジタルID計画まで、世界各国がさまざまな分野でデジタル変革を進めています。

アメリカ：SpyCloudの新しいサイバー犯罪調査ツール

サイバーセキュリティソリューションのリーダーであるSpyCloudが、SpyCloud Investigationsポータル内に画期的な新機能を発表しました。IDLink高度分析と呼ばれるこの新機能は、サイバー犯罪調査にアイデンティティ分析を直接組み込むことを目的としています。

IDLinkの主な利点：

内部脅威分析の加速サプライチェーンリスク評価の強化脅威アクターの特定の改善

このツールは以下のチームにとってゲームチェンジャーになると期待されています：

サイバー脅威インテリジェンス（CTI）チームセキュリティオペレーション不正およびリスク防止アナリスト法執行機関

IDLinkは、漏洩したアイデンティティデータと潜在的なリスクとの関連性を迅速に洞察することで、サイバー犯罪調査の効率性と有効性を大幅に向上させることを目指しています。

アメリカ：NISTによるYotiの顔年齢推定の認識

米国国立標準技術研究所（NIST）が、Yotiの顔年齢推定（FAE, facial age estimation）モデルの精度がすぐれたものであることを確認しました。このモデルは2つの重要なカテゴリーで注目すべきランキングを達成しました：

「子供のオンライン安全」カテゴリーにおける13〜16歳の精度で1位 18〜30歳の「マグショット」画像の精度で2位

Yotiは、携帯電話で撮影された自撮り写真に特化してFAEモデルを最適化したと報告しています。この最適化により、Yotiの内部テストデータを使用して測定した場合、大幅に高い精度が得られました。

NISTからの recognition は、特に若年ユーザーのオンライン安全対策を強化する上で、顔年齢推定技術の可能性を強調しています。

フィジー：国家デジタルID計画の進行

フィジーが野心的な国家デジタルIDプログラムの開発に着手しました。主要な詳細は以下の通りです：

予想タイムライン：完了まで少なくとも26ヶ月リーダーシップ：以下のメンバーで構成される運営委員会マノア・カミカミカ副首相ビマン・プラサド教授（副首相）アリフ・アリ準備銀行総裁その他の省庁の代表者

フィジー内閣は6月にこのプロジェクトを承認し、包括的な国家デジタルIDシステムの確立に向けて重要な一歩を踏み出しました。このシステムは政府と民間セクターの両方のニーズに対応することを目的としており、フィジー国民がさまざまなサービスや機関とやり取りする方法を変革する可能性があります。

コモロ：デジタル公共インフラプロジェクト

コモロ連合が、アフリカ開発銀行（AfDB）からの実質的な資金援助を受けて、デジタル公共インフラ（DPI）プロジェクトを開始する準備を整えています。資金調達とプロジェクトの詳細は以下の通りです：

総資金：951万ユーロ（約1040万米ドル）アフリカ開発基金から402万ユーロ銀行の移行支援施設から549万ユーロ

プロジェクト名：「コモロ経済のデジタル化支援プロジェクト」

主な目的：

ガバナンスの改善公共サービスの質、手頃さ、アクセシビリティの向上デジタル政府システムの確立

具体的な成果物：

新しいデータセンターの建設と運用既存の二次データセンターのアップグレードデジタルイノベーションのためのインキュベーターの創設

プロジェクトの総コストは2253万ユーロ（2500万米ドル）と推定されており、追加の資金源や将来の投資計画があることを示唆しています。

グローバル：AI駆動の身元詐欺の増加

Signicatによる2024年の報告書「AI駆動の身元詐欺との戦い」が、AI駆動の詐欺の増加に関する警告的な統計を明らかにしました：

金融・決済セクターで検出された詐欺の試みの42.5%がAIを利用これらのAI駆動の試みの推定29%が成功

これらの数字は、詐欺師の手口の高度化と、金融業界における高度な不正検出・防止対策の緊急の必要性を浮き彫りにしています。

欧州連合：デジタル旅行資格イニシアチブ

欧州委員会が、シェンゲン圏の旅行文書のデジタル化に向けて重要な一歩を踏み出しました。パスポートとIDカードをデジタル化するための2つの提案が採択され、これらは総称して「EU Digital Travel application」と呼ばれています。このイニシアチブはEU市民と非EU市民の両方に適用されます。

提案の主な特徴：

デジタル旅行資格の使用に関する共通フレームワーク旅行者がデジタル旅行資格を作成・保存するための新しい「EU Digital Travel application」

利点：

シェンゲン圏への、および圏内の旅行がより簡単で安全に国境通過プロセスの現在の国境での物理的チェックからの合理化

実装：

アプリケーションは欧州委員会がeu-LISAの支援を受けて開発、EUレベルで提供されます生体認証パスポートまたはEU IDカードを持つすべての旅行者が利用可能になります

このイニシアチブは、欧州における旅行文書のデジタルファースト化への大きな転換を示しており、他の地域にも先例を示す可能性があります。

キプロス：デジタルIDカードの展開

キプロスは、10万枚のデジタルIDカードの購入を承認し、デジタル化への取り組みを進めています。ニコデモス・ダミアヌ研究副大臣が以下の詳細を発表しました：

最初の3万枚は無料で配布残りの7万枚は1枚15ユーロで発行デジタルIDカードは欧州委員会の承認を受けている市民はさまざまなデジタル政府サービスにアクセス可能になる

この動きは、キプロスがアイデンティティインフラを近代化し、より広範な欧州のデジタルアイデンティティイニシアチブに足並みを揃える決意を示しています。

フランス：デジタル医療カードの統合

フランスは、「carte Vitale」として知られる医療カードをデジタル化し、France Identitéアプリに統合する予定です。この開発の主なポイントは以下の通りです：

計画されている開始時期：2025年第1四半期現在の機能：ユーザは、物理的なcarte Vitaleを提示することによって、ユーザーがフランスの医療システムにアクセスし、医療費の償還を受けることを可能にしています。将来の機能：ユーザーはアプリを通じて医療カードをデジタルで提示可能になります。

France Identitéアプリには、すでに電子的な国民IDカードや運転免許証などの重要な文書が含まれています。この医療カードの追加により、フランス国民の個人識別情報とデータ管理がさらに一元化されます。Sopra Steria, Atos, iDAKTO and Idemia がフランスのデジタルアイデンティティスキームの技術を提供しています。

モルドバ：EUデジタルアイデンティティ標準への適合

モルドバは、デジタルID規制を欧州連合の標準に合わせる作業を進めており、特にEUデジタルアイデンティティ（EUDI）ウォレットとの互換性に焦点を当てています。この目標を達成するために：

モルドバはエストニアのソフトウェア会社Cyberneticaと提携プロジェクトは10月に終了予定

この適合は、モルドバが欧州のデジタルアイデンティティエコシステムとの統合に取り組んでいることを示しており、国境を越えたデジタル相互作用をよりスムーズにする可能性があります。

カザフスタン：国家生体認証システム

カザフスタンは、遠隔生体認証のための包括的な国家システムの確立を進めています。このシステムは、国内のすべての産業分野の生体認証データを統一したデータベースとして機能することを目指しています。

関与する主要組織：

National Information Technologies JSC（Nitec）、カザフスタンの電子政府インフラ運営者 BTS Digital、カザフスタンのデジタルIDスタートアップ

このイニシアチブは、カザフスタンがアイデンティティインフラを近代化し、安全な認証を必要とするさまざまなサービスを合理化する野心を反映しています。

おわりに

これらの発展から分かるように、世界中の国々がデジタルアイデンティティと関連技術において大きな進歩を遂げています。サイバーセキュリティ対策の強化から旅行文書や医療システムのデジタル化まで、これらのイニシアチブは、個人が政府サービスを利用し、旅行し、ますますデジタル化する世界で個人情報を管理する方法を再形成しています。

（出所）https://www.thinkdigitalpartners.com/news/2024/10/14/digital-identity-global-roundup-187/

14:31 • 9 days ago

Ben Werdmüller

Revisiting Known

I thought it would be fun to revisit Known, the open source publishing platform that powers my site. How it works Known allows a team or community to publish news on any topic to a single, searchable stream of content that’s easily accessible from any device. It is not a full CMS, and nor is it designed for independent publishers to sell subscriptions; instead, it’s optimized for publis

I thought it would be fun to revisit Known, the open source publishing platform that powers my site.

How it works

Known allows a team or community to publish news on any topic to a single, searchable stream of content that’s easily accessible from any device. It is not a full CMS, and nor is it designed for independent publishers to sell subscriptions; instead, it’s optimized for publishing to a single feed.

Every Known site is a single feed of content that any number of users can publish to. You can have one user, as my site does; you could have thousands, if you wanted.

The stream can also be filtered by hashtag, author, or content type — so you can choose to only view content on a certain topic, or only photos, or some combination thereof.

Each stream, filtered or not, is shown as a standard web page by default. These can be themed, but it’s also easy to view different interfaces. RSS and JSON are available for every screen you can view as a web page, and it would be easy to add low bandwidth HTML, for example. (I once added an interface type that displayed everything as a Star Wars crawl. It got old fast.)

When you log in, you get a little menu bar that lets you publish different kinds of content. It’s a little bit like Tumblr’s bar, but here, every type of content is powered by a plugin. You can download new content types created by other people, or you can write your own. On my site I’ve created a kind of blog post called an “aside”, which I’ve decided to make a distinct content type.

Hit the button, and you can compose right on the page.

Known supports an idea called POSSE: Publish on your Own Site, Syndicate Elsewhere. You can elect to syndicate a post to a third-party site by enabling the toggle for that site below the compose window. In this illustration I have two example webhooks, but people have written plugins for Mastodon, etc. (In the beginning, Known had plugins for Twitter, Facebook, and so on, but all those APIs locked down over time. The promo image, which you can see above, includes Foursquare and Flickr as options, which is a clue about the era it originated from.)

You can also compose using any application that supports the Micropub standard. I tend to write all my blog posts in iA Writer.

Known supports Webmention, so when you publish a post that links to a site, that site will be notified. You can even use webmention to respond to someone else’s post elsewhere and have a conversation across the web.

It’s free and open source, and intentionally runs on the same LAMP stack as WordPress. Be warned though; as the screenshots suggest, it’s now a little old.

A little history

Known was originally called Idno. (“What does it stand for?” someone once asked me. “I d’no,” I replied. This is the level of humor you can generally expect from me.)

I wrote the first version of it when my mother was recovering from a double lung transplant: she was in need of community but absolutely didn’t want to discuss her condition on Facebook. I’d previously written Elgg, an older open source social networking platform, so I decided to think about what a social community platform might look like in the era of the mobile, ubiquitous web. What would it look like for a community to publish to a place where it could continue to own its own content, on its own domain? (It seems like a quaint exploration now, but remember that this was 2013.)

I became friends with the indieweb folks, and met Erin Richey at an IndieWebCamp. We decided to collaborate on the project. It was her idea to submit it to Matter, where we took part in the third accelerator class. Along the way, we did some focus group testing (Erin’s instigation) and chose Known as a permanent name.

It was a startup for a couple of years; there was a paid, hosted version; a Known-powered site even won an award for KQED. But it wasn’t the kind of thing that excited investors, and we weren’t making enough money for it to be sustainable. Ultimately, I allowed myself to be acquihired by Medium, which allowed us to pay Matter back, and we both settled into new jobs. The day before my first Medium paycheck, I spent my last five dollars on gas. (Erin and I welcomed our actual child — a human one — two years ago. So there’s a coda.)

But there are still users out there, myself included, and the open source project is still alive. It’s been slower over the last few years, because I haven’t had much time to devote to it. (The main thing I’ve been looking at is a command line exporter to allow people to more easily take their content into WordPress, as well as some experiments with ActivityPub.) But it remains a core part of the operating system that powers my identity online, and the identity of others.

Lately I’ve been thinking that there’s a place for this model of publishing. The internal architecture needs to be overhauled; the Bootstrap-driven default template needs to go; but I think there’s really something to the model of letting communities publish to a simple, queryable feed of content that syndicates out to the world.

Perhaps it’s finally time for Known 2, with an easy upgrade path from the original? If you’re intrigued by the idea — or if you’re a Known user — I’d love to hear your thoughts.

01:28 • 9 days ago

Sunday, 13. October 2024

IdM Laboratory

そういえばGNAPがRFCになりました

こんにちは、富士榮です。
そういえばOAuth 3.0とかXYZとか言っていたGNAP（Grant Negotiation and Authorization Protocol）がRFC9635になりましたね。
https://www.rfc-editor.org/rfc/rfc9635.html
それに伴いGNAP WGはクローズされた模様です。 https://mailarchive.ietf.org/arch/msg/txauth/smCNPKB2vMPkMX9LCXONGZ-c5Oo/

うーん、まだ息してたんですね・・・（Justinに怒られそう）
ということで著者のJustinもブログ書いてますね。 https://justinsecurity.medium.com/gnap-a-conversation-of-authorization-5b603d850fe9

まぁしかしOAuth2.0の周辺仕様が多くなりすぎたのでシンプルにしましょう、というのは良かったのですが、フレームワークとプロファイルという意味でマイクロ化された仕様が組み合わさるOAuth2.0は複雑化する一方で柔軟性を提供して来たわけで、歴史の長さも含め広く浸透して来ているわけです。そこをシンプルではあるものの新しい仕組みで置き換えるのは、正しいかもしれませんが実際の普及という観点では非常に難しい話になりそうです。
今後、実際に使われていくかどうか、見守っていきましょう。

23:00 • 9 days ago

Ben Werdmüller

WordPress.org’s latest move involves taking control of a WP Engine plugin

[Wes Davis at The Verge] The feud between Automattic (or more specifically, Matt Mullenweg himself) and WP Engine is getting bonkers: "WordPress.org has taken over a popular WP Engine plugin in order “to remove commercial upsells and fix a security problem,” WordPress cofounder and Automattic CEO Matt Mullenweg announced today. This “minimal” update, which he labels a fork o

[Wes Davis at The Verge]

The feud between Automattic (or more specifically, Matt Mullenweg himself) and WP Engine is getting bonkers:

"WordPress.org has taken over a popular WP Engine plugin in order “to remove commercial upsells and fix a security problem,” WordPress cofounder and Automattic CEO Matt Mullenweg announced today. This “minimal” update, which he labels a fork of the Advanced Custom Fields (ACF) plugin, is now called “Secure Custom Fields.”"

What appears to have happened is this:

WP Engine was banned from the WordPress plugin portal. A flaw was found in its popular Advanced Custom Fields plugin and patched - but because it was banned from the portal, WordPress users couldn't get an automatic update. Rather than seed the patch, Automattic forked the plugin, renamed it, and took over the upgrade path in-place. All WordPress users of ACF that upgrade via the portal will now get Automattic's version, which removes all commercial ties to WP Engine.

Technically, Automattic (or anyone) can fork any open source plugin - that's what open source is all about. But seizing the upgrade path and swapping for the new version in-place in the portal is a pretty rotten move.

ACF is well-used in commercial sites and is often provided by agencies as a bedrock for their customizations. This isn't some sideline: for many users, ACF makes WordPress significantly more useful.

It's an existential issue for any open source plugin contributor. Again, forking is well within anyone's rights - but replacing the upgrade path is something only Automattic can do.

This is only muddied by the fact that the portal is technically owned by Matt alone, rather than Automattic. But the lines are blurry at best.

Whereas the feud had previously not created a risk to WordPress's functionality, for many serious users this is now a big problem. A stable platform with solid upgrade paths is a huge part of why people choose WordPress. Whatever's going on behind the scenes, this altercation has created huge risk for anyone who's thinking about making the leap (and, at the same time, may open up opportunities for other open source CMS vendors).

https://www.openid.or.jp/news/2024/10/openid-connect-for-identity-assurance.html

[Link]

16:45 • 10 days ago

Doc Searls Weblog

The iPhone 16 Pro Max, so far

A 5x telephoto shot with my new iPhone 16 Pro Max.

Holding the mic in this shot, taken with my new iPhone 16 Pro Max, is Mitch Teplitsky, a documentary filmmaker based in Bloomington, Indiana. Mitch has been reading this blog for the duration, and reached out when I showed up in town. The scene is the Pitchdox award event yesterday, which was by Hoodox at the Kan-Kan Cinema and Restaurant in Indianapolis. It’s one of the most delightful places I’ve ever been to: a great theater, bar, and restaurant, all in one. Here’s what my wife and I shared at the bar, between watching “We Strangers” and the event (thumbs up for both):

Frites, salad and two sliders at the Kan-Kan bar.

That was also shot with the Pro Max. (In fact, after the Hoodox event yesterday, Mitch asked if I had the new phone with me, because he had read my post about it that morning.)

So here is my review after one day with the Max and two weeks with its smaller brother, the Pro.

Getting the phone, setting it up, and transferring all the data (~135 GB) was simple, fast, and easy at the Indianapolis Apple Store. Nice.

I love the big screen, which is easy for me to read and poke (which, with my imperfect eyes and warped fingers, is a big plus).

The camera performance is great for a phone, but let’s be clear: it isn’t what I get from my Sony a7iv mirrorless 35mm SLR and any of its excellent lenses. But that kit is heavy, requires a bag, and screams “that guy is taking a picture!” when I use it. Everybody has phones and shoots pictures with them, so with the Pro Max I’m just another shooter in a crowd.

What the Pro Max does extremely well is provide useful pictures, like the two above, under many conditions. It’s also nice to have that big screen for showing pix to other people.

My only ding on the Max is a small one: In a side-by-side comparison with the Pro, the Max’s screen is dimmer at high angles off straight-on. Not by a lot, but by some. I think the brightness is also slightly less uniform—at least with this Max. This is not something anyone would notice without comparing the two phones side-by-side and looking at both from a low angle. But I do think the difference is there.

To sum up, I think the Pro Max is ideal for anyone who wants the largest iPhone screen and doesn’t mind the added weight (which does come with a bigger and more capacious battery). If you don’t need the extra stuff the pro cameras do, I suggest going for the plain iPhone. I loved the Pro when I had it, and all you give up with that one is the telephoto lens. One pro tip: get twice the storage you think you’ll need, because you’ll provably end up needing it.

But,,,,, if you can, wait another year for the iPhone 17, which will reportedly launch a new generation. I couldn’t wait (my iPhone 11 was out of storage), but maybe you could.

15:10 • 10 days ago

Michael Ruminer

AI “its got legs”

Compliments of FLUX schnell This last week I spoke with two folks on AI intersection within their work and how their work intersects with one another. One was a CPO for a startup that is bringing AI into their life sciences products in a forefront way. The other was a compliance manager for a pharma company. The second happens to be my wife. For me, their work intersects. They both certainly g

Compliments of FLUX schnell

This last week I spoke with two folks on AI intersection within their work and how their work intersects with one another. One was a CPO for a startup that is bringing AI into their life sciences products in a forefront way. The other was a compliance manager for a pharma company. The second happens to be my wife. For me, their work intersects. They both certainly got me thinking harder about AI agents. The first of the two was breaking down knowledge silos in organizations and I definitely saw a play for AI agents and the use of tools by those agents in their approach. The second needed some exuberant discussion on the potential of how AI can contribute to their organizational goals and needs; and without mansplaining, I was glad to help. :-) The conversations only reinforced in me where we are headed in AI adoption.

There were discussions on trend analysis, small language models, retrieval-augmented generation (RAG), and how AI agents and tools work to bring different knowledge repositories into a single interface with human consumable outputs, just to name a few topics. Surprisingly or not, a lot of these topics were not with the CPO but with the wife; putting the theory into practical examples. It was motivational to me.

I have been looking for good, practical, and real-world examples of AI agenic tools to work on as knowledge-expanding exercises; whether they work or, in the end, surface the immaturity of the technology. I don’t get especially motivated by the common and tired examples of financial analysis, LinkedIn scraping for recruiting, or travel booking scenarios that every agent example seems to rely on. These discussions were a step forward.

These topics reinforced with me that for AI and its agents — “its got legs”.

14:33 • 10 days ago

IdM Laboratory

OpenID Connect for Identity Assurance日本語版が公開

こんにちは、富士榮です。先日もお伝えしたとおり、OpenID Connect for Identity Assurance（通称OIDC4IDA）が正式化されましたが、早くもOpenIDファウンデーションジャパンの有志により日本語化が完了しています。こちらが日本語版のお知らせ https://www.openid.or.jp/news/2024/10/openid-connect-for-identity-assurance.html ぜひ読んでみましょう！

こんにちは、富士榮です。

先日もお伝えしたとおり、OpenID Connect for Identity Assurance（通称OIDC4IDA）が正式化されましたが、早くもOpenIDファウンデーションジャパンの有志により日本語化が完了しています。

こちらが日本語版のお知らせ

ぜひ読んでみましょう！

00:38 • 10 days ago

Saturday, 12. October 2024

Heres Tom with the Weather

21:27 • 10 days ago

IdM Laboratory

SIDI Hub - ベルリンレポートを読む（７）

こんにちは、富士榮です。なかなか終わらないSIDI Hubベルリンレポートです。再来週の東京サミットに間に合うかな・・・ようやくユースケースが終わり、相互運用性を確保するための最低限の要件（Minimum Requirements for Interoperability/MR4I）のパートです。 Minimum Technical Requirements - Debora To abstract the minimum requirement for global interoperability is an incredibly difficult task. We are looking not only at how to integrate different types of ID systems but also the abstract requ

こんにちは、富士榮です。

なかなか終わらないSIDI Hubベルリンレポートです。再来週の東京サミットに間に合うかな・・・

ようやくユースケースが終わり、相互運用性を確保するための最低限の要件（Minimum Requirements for Interoperability/MR4I）のパートです。

Minimum Technical Requirements - Debora

To abstract the minimum requirement for global interoperability is an incredibly difficult task. We are looking not only at how to integrate different types of ID systems but also the abstract requirements that underpin the goal. We believe that the goal is a Network of Networks - inclusive of many different networks because a founding principle of SIDI Hub is that we should not tell jurisdictions how to build their systems.

グローバルな相互運用性の最低要件を抽出することは、非常に困難な作業です。私たちは、異なるタイプのIDシステムを統合する方法だけでなく、その目標を支える抽象的な要件についても検討しています。私たちは、この目標は「ネットワークのネットワーク」であるべきだと考えています。なぜなら、SIDI Hubの設立理念は、管轄区域がシステムを構築する方法を指示すべきではないというものです。

相互運用性の課題は非常に難しい問題です。ネットワークのネットワーク、ネットワーク間を繋ぐためのネットワーク、考えてみるとまさにインターネットですね。

GAINもそうですが、やはりインターネットにアイデンティティのレイヤを載せていく、というアプローチが最終的には必要になるのではないかと思います。それまでの間はそれぞれのアイデンティティ・ネットワークを連携させるためのインターコネクトをどうデザインするのか、というところに着目し対応し続けないといけないのかもしれません。

In this effort, there are two protocol types:
Trust services layers (the control plane) ID info exchange (the data plane)
While the RP Registration conversation focused on the former, the SIDI Hub Minimum Technical Requirements session focused on the latter and did so using group exercises to explore nine scenarios in 3 groups (each group had 3x scenarios).
この取り組みには、2つのプロトコルタイプがあります。トラストサービスレイヤー（コントロールプレーン） ID情報交換（データプレーン） RP登録に関する会話では前者が中心でしたが、SIDI Hubの最低技術要件セッションでは後者が中心となり、3つのグループに分かれて9つのシナリオを検討するグループ演習が行われました（各グループには3つのシナリオがありました）。

相互運用性を担保するためには、ルール面（トラストフレームワークやガバナンスなど）とテクニカル面（データ構造、スキーマ、署名形式、通信プロトコルなど）の両方について相互に合意が取られることが必要ですが、こちらのパートではテクニカル面にフォーカスしています。一方でルール面については後ほど出てくるTrust Framework Mappingのワークストリームで対応を進めています。

The simple terms used in this diagram were taken to mean:
Federated = architectures built on standards-based federation, e.g., SAML and OIDC Wallet-Based = architectures built on a “three-party model” of issuer-holder-verifier in which something akin to a “wallet” plays a key role in data exchange, e.g., the EUDI ecosystem emerging in the European Union. API = architectures built atop another proprietary or custom API, e.g., the NIN in Nigeria

Each group, armed with their three scenarios, was asked to consider where interoperability might be achieved and the pros and cons of those different approaches.

この図で使用されているシンプルな用語は、次のような意味で使用されている。フェデレーション型 = 標準ベースのフェデレーション、例えばSAMLやOIDCを基盤とするアーキテクチャウォレット型 = 発行者、保持者、検証者の「三者モデル」を基盤とするアーキテクチャで、データ交換において「ウォレット」に似たものが重要な役割を果たすもの、例えば欧州連合で台頭しつつあるEUDIエコシステム。 API = 別の独自仕様またはカスタム仕様の API を基盤とするアーキテクチャ、例えばナイジェリアの NIN 3つのシナリオを武器に、各グループは相互運用性が実現できる可能性がある場所と、それらの異なるアプローチの利点と欠点を検討するように求められました。

大きく分けてアイデンティティを連携するための方式はフェデレーション、ウォレット、APIの3類型があるので、それらのアイデンティティシステム間を接続しようとすると、それぞれについてどのようなことが必要になるのかを検討していくわけです。

REPORTING GROUP 1:
Scenario 3:

Fix at source:
It works. It becomes a matter of the issuer deciding which one to support Different solutions on ToIP: We should not be stuck trying to find solutions in one shot. We are talking about networks of networks. We have one layer of understanding first before jumping to more. Privacy is the other side of the coin. We need to work on that for all the solutions we provide.
ソースでの修正：機能します。どちらをサポートするかは発行者が決定すればよいだけの問題です ToIPではさまざまなソリューションが提案されています：一発で解決策を見つけようとして立ち往生すべきではありません。私たちはネットワークのネットワークについて話しているのです。さらに踏み込む前に、まず理解すべきことが1つあります。プライバシーは表裏一体です。私たちが提供するすべてのソリューションにおいて、プライバシーにも取り組む必要があります。

このシナリオはWallet（ウォレット型）からRP（フェデレーション型）への接続を行う際にソース側となるWalletと宛先となるRPのそれぞれで対応するにはどうするか、という議論をしています。

まぁ、単純な話、ソース側のWalletがOP/IdPとしてRPに対してid_tokenやSAML Assertionを発行すればいい話です。もしくはRP側がIHV（Issuer/Holder/Verifier）モデルにおけるVerifierとしての機能をもてばいいわけです。

Scenario 6:

Where is the wallet? All solutions are similar; they are variations. Where is the user? Where is the user consent? Quality of data? How do you establish trust in the data and the issuer? ウォレットはどこにあるのか？すべてのソリューションは類似しており、バリエーションにすぎない。ユーザーはどこにいるのか？ユーザーの同意はどこにあるのか？データの品質は？データと発行者の信頼をどのように確立するのか？

次のシナリオはAPIとRPの間の連携です。

今回はソース側がAPIなので、単純にREST APIベースのIDサービスならOpenID Providerになればいいじゃないか、という話です。もしくはRP側がREST API Clientとして構成されれば問題ないですよね、という話。

Deboraも現地の声としてレポートしていますが、フェデレーション型のIdPはAPIベースのIdPの一類型でもあるので単にプロトコル合わせをしているだけですね。

Scenario 9:

Issue: who trusts the translator? Broker in the middle is the main issue. The broker ensures key management and key integrity. APIs to APIs need to be securely done
Two themes:
The trust layer is important privacy side of things and chain of trusts should be considered 問題：誰が翻訳者を信頼するのか？仲介者が存在することが主な問題である。仲介者は、鍵の管理と鍵の整合性を確保する。 APIとAPI間の通信は安全に行う必要がある
2つのテーマ：信頼レイヤーは重要であるプライバシーの側面と信頼の連鎖を考慮すべきである

API同士の連携においても互換性がなければ先ほどのフェデレーションーAPIの間の話と同じです。ここでブローカーモデルが登場しますが、いわゆるプロトコルコンバーターを中間に介在させることで、どうやって中間者を信頼するのか、直接的にソース・デスティネーションの間で信頼関係が作れなくなることをどう考えるのか、などの課題が浮き上がってきます。

ということで、まずは一つ目のグループでの議論の内容について見ていきました。

次回は2つ目のグループの議論も見ていきましょう。

00:51 • 11 days ago

Thursday, 10. October 2024

Ben Werdmüller

Insecure Deebot robot vacuums collect photos and audio to train AI

[Julian Fell at Australian ABC News] "Ecovacs robot vacuums, which have been found to suffer from critical cybersecurity flaws, are collecting photos, videos and voice recordings – taken inside customers' houses – to train the company's AI models." So in effect these robot vacuums are tiny spies inside your home, sending details about your living space and potentially your

[Julian Fell at Australian ABC News]

"Ecovacs robot vacuums, which have been found to suffer from critical cybersecurity flaws, are collecting photos, videos and voice recordings – taken inside customers' houses – to train the company's AI models."

So in effect these robot vacuums are tiny spies inside your home, sending details about your living space and potentially your family to some centralized data store.

This must be some terrible breach, right? A mistake? Code that should never have made it to production?

Not quite:

"The Chinese home robotics company, which sells a range of popular Deebot models in Australia, said its users are "willingly participating" in a product improvement program."
"[...] It also states that voice recordings, videos and photos that are deleted via the app may continue to be held and used by Ecovacs."

So, obviously, this is bad. The thing is, if any device is recording this kind of footage and sending it to a centralized datastore, it's reasonable to assume that it will eventually be compromised, either by a third party or the vendor themselves. It's not good that this is happening, but unless footage remains on your home network and never makes it to the internet, every device should be considered a security risk.

It's worth considering which devices could be quietly sending data to someone who can see them, and what implications that might eventually have. A simple rule of thumb is that if it's physically possible, someone will eventually do it.

#AI

[Link]

23:51 • 12 days ago

Jon Udell

Mix Human Expertise With LLM Assistance for Easier Coding

There are plenty of ways to use LLMs ineffectively. For best results, lean into your own intelligence, experience, and creativity. Delegate the boring and routine stuff to closely supervised assistants whose work you can easily check. Mix Human Expertise With LLM Assistance for Easier Coding Part of the LLM series at The New Stack.

Mix Human Expertise With LLM Assistance for Easier Coding

Part of the LLM series at The New Stack.

23:37 • 12 days ago

IdM Laboratory

SIDI Hub - ベルリンレポートを読む（６）

こんにちは、富士榮です。なかなかボリュームがあってなかなか終わらないベルリンレポートを引き続き見ていきます。 GAP分析の途中からです。ガバナンスの部分ですね。 We then discussed three essential points about governance: The need to think long-term: We cannot yet define what we will establish. A good starting point is champion use cases. Possible precedent for governing entity: there are organizations like the Global Fund or Gates Foundation set up t

こんにちは、富士榮です。
なかなかボリュームがあってなかなか終わらないベルリンレポートを引き続き見ていきます。

GAP分析の途中からです。ガバナンスの部分ですね。

We then discussed three essential points about governance:

The need to think long-term: We cannot yet define what we will establish. A good starting point is champion use cases.

Possible precedent for governing entity: there are organizations like the Global Fund or Gates Foundation set up to avoid complexity and time in inter-government negotiation & rule-making.

a. Another good example is GAVI, which was set up to channel vaccines from the rich north to the south with a focus on malaria and other diseases. Complex structure.

Engage the Global South: We cannot create use cases for them. We need to go to them and ask their needs; otherwise, how can we expect them to engage?

そして、ガバナンスについて3つの重要なポイントを議論した：

長期的に考える必要性：長期的な視点が必要である。出発点としては、チャンピオン・ユースケースが良い。世界基金やゲイツ財団のように、政府間の交渉やルール作りの複雑さや時間を避けるために設立された組織がある。 a. もう一つの良い例はGAVIで、マラリアやその他の病気に焦点を当て、豊かな北から南へワクチンを流すために設立された。複雑な構造。グローバル・サウスを巻き込む：グローバル・サウスとの関わり：彼らのためにユースケースを作ることはできない。彼らのところに行き、彼らのニーズを聞く必要がある。

これはなかなか難しいお題ですね。 SIDI Hub自体が現状は特定の法人ではなく国際コミュニティでしかないため、まずはこの状態をどうしていくのか？の戦略が必要になりそうです。そのためには成果物をどういう位置付けで何に使ってもらうことを想定するのか、という団体としての目指す姿、存在目的ですね。

The group then returned to the problem statement and how we might hone in on a methodology.

その後、グループは問題提起に戻り、どのように方法論に磨きをかけるかについて話し合った。

We discussed a number of risks inspired by the European Union’s EIDAS 2.0:
EU national ID: every country establishes and manages its own list and Do people want to use credentials across borders beyond Europe, e.g., California DL accepted by the Estonian gov? It seems that some are assuming that all these rules are going to be on the wallet. That’s not going to scale. We need to think about what kind of policies an issuer can give to a wallet There are a large number of trust marks, some regional, some functional. Agents in the wallets that will give users advice. How can we have wallets to work across jurisdictions?

EUのEIDAS2.0に触発された多くのリスクについて議論した：

EUの国民ID：すべての国が独自のリストを作成し、管理する。例えば、カリフォルニア州のDLがエストニア政府によって受け入れられるような。これらのルールはすべて財布の中にあると思い込んでいる人がいるようだ。それでは規模が拡大しない。発行者がどのようなポリシーをウォレットに与えることができるかを考える必要がある。トラストマークは地域的なものから機能的なものまで数多くある。ユーザーにアドバイスを与えるウォレットのエージェント。どのようにすればウォレットが法域を超えて機能するのか？

Walletモデルを考えるとやはり先行しているEU/eIDAS2.0を分析するアプローチになるのは自然かと思います。越境シナリオについても現実味がある地政学的な特色もありますし。

A member of the group asked, “Do we want RP registration at all?” and suggested a vote and working to clarify the problem statement. If yes, are we reinventing the wheel, or do we have what we need somewhere in the public sector?

グループのメンバーは、「RP登録を本当に必要としているのか？」と問いかけ、投票を行い、問題の明確化に取り組むことを提案した。もし必要だとしても、私たちは同じことを繰り返すのか、それとも必要なものは公共部門のどこかにあるのか？

リライングパーティの管理とスケーラビリティ・ガバナンスの問題はしばしば議論されてきましたが、ユースケース次第じゃないの？っていういつもの結論になりそうな予感しかしません。

The final discussion points in this section included:
User Protection: we need to identify the RP for every transaction. That does not mean that the RP is registered. We have a mechanism called attestations. We can replicate what we have today. RP Entitlement: In the EU, we are heading to Registration. Someone has to make a decision about who is entitled to do what. Recommendation to explore that question rather than the how. We need to solve this fundamental question now. BOLTS: Catalog business, Operational, Legal, Technical, and Social practices with respect to the Champion Use Cases and map risks.
このセクションの最後の議論のポイントは以下の通りです。
ユーザー保護：すべてのトランザクションの RP を特定する必要があります。ただし、RP が登録されるということではありません。アテステーションと呼ばれる仕組みがあります。現在行っていることを複製することができます。 RP 権限：EU では登録に向かっています。誰が何を実行する権利を有するのかについて、誰かが決定する必要があります。方法ではなく、その問題を調査することを推奨します。この根本的な問題は今すぐ解決する必要があります。 BOLTS：チャンピオンユースケースに関する業務、運用、法務、技術、および社会慣行をカタログ化し、リスクをマッピングする。

確かにRPが特定される状況でないとユーザは安心してサービス利用できません。そういう意味ではガバナンスが重要、っていう話（このセクションがそういうセクションですし）でしょう。

We did not cover the other two major rocks in detail and will return to those items in the workstreams and in future summits.
他の2つの主要な岩については詳しく取り上げなかったが、それらの項目についてはワークストリームや今後のサミットで再び取り上げる予定である。

まぁ、結局は業界やユースケースによってもガバナンスの主体や対象が異なるのに、国際的な相互運用ができるのか？っていうことです。そういう意味ではユースケースを特定してステークホルダーを明確化、その中で合意可能な範囲を探していく、というアプローチはしばらく続けないといけない気がします。

ようやく次はテクニカルな要求に関するセクションです。

23:22 • 12 days ago

Doc Searls Weblog

iPhone 16 Pro or Pro Max?

I got an iPhone 16 Pro twelve days ago. I have two more days to swap it for an iPhone 16 Pro Max, which will cost me $100 above the mint I already paid for the Pro with 1 TB of storage. Why so much storage? I want to maximize storage because this thing is […]

iPhone 16 Pro and Pro Max. This is from Apple’s page for both. I think it’s exaggerating the difference a bit. Not sure, though.

I got an iPhone 16 Pro twelve days ago. I have two more days to swap it for an iPhone 16 Pro Max, which will cost me $100 above the mint I already paid for the Pro with 1 TB of storage.

Why so much storage? I want to maximize storage because this thing is my main camera now, I shoot a lot, and I want a lot of archival photos on it as well. The iPhone 11 that this one replaced had 128 GB of storage and maxed out a long time ago. Frankly, I’d love it if Apple offered a phone with 2 TB of storage or more. I have 8 TB on this laptop, and my photos on it are already maxing it out.

The case for the Pro is that my hands are on the small side and gnarled with arthritis, and it fits nicely in my shirt and pants pockets. The case for the Pro Max is a bigger screen, which makes for easier work, for showing photos to others, for watching movies on planes, and other stuff like that. My eyes are also not getting better, and a bigger screen can help with that too. The battery on the Max is also bigger and lasts longer.

Earlier versions of the Pro Max also had functional advantages over the Pro models. For example, the oldest iPhone to run Apple Intelligence is the 15 Pro Max, not the 15 Pro. But functional advantages are gone with this generation. Other than the bigger size and slightly longer battery life, there is nothing the Pro Max can do that the Pro can’t.

Back to sizes.

The Pro Max weighs a little under 8 ounces, and the Pro a little over 7 ounces. That’s not much different to me. I’ve felt both and find them equally agreeable.

The Pro Max is 6.42 inches (163 mm) tall and 3.06 inches (77.6 mm) wide, while the Pro is 5.89 inches (149.6 mm) tall and 2.81 inches (71.5 mm) wide. So the difference is about a half inch in height and a quarter inch in width. Meaning that the Pro Max is a longer phone more than a wider one. That means the Max won’t really be hard for my hands to hold.

Here is something else to consider. I tend to keep a phone for several years:

3G in 2008 5 in 2012 7 in 2016 11 in 2019 (after I dropped the 7 into a pool while trying to rescue an insect) 16 Pro in 2024

That averages about four years per phone. So I might want to get the best phone I can at the start of each stretch. But would the best phone also be the biggest phone?

So, should I go to the trouble of making the 150-mile round trip to my nearest Apple Store (on the far side of Indianapolis from Bloomington) and laying down $100 for the Pro Max?

Not sure. I welcome advice.

22:40 • 12 days ago

The Pragmatic Engineer

The Pulse #110: VC-funded companies acting more like bootstrapped ones?

Also: first-ever double Nobel Prize wins for AI research, and an interesting cloud platform price comparison startup built on a budget

The Pulse is a series covering insights, patterns, and trends within Big Tech and startups. Notice an interesting event or trend? Send me a message.

Today, we cover:

Industry pulse. Google to be broken up? NVIDIA may compete with cloud providers, Twilio commits to full-remote work, third-party app stores coming to Android, and more.

VC-funded companies being more bootstrapped? More seed-stage VC-funded companies aim to become profitable fast in order to avoid future fund raising rounds. It’s like how bootstrapped companies operate, and reduced levels of available funding will keep this approach popular.

First-ever double Nobel Prize wins for AI research. Both the physics and chemistry Nobel prizes have been awarded for AI-related work. And three of the five prize winners have Google connections.

Interesting startup idea: benchmarking cloud platform pricing. A team of 3 developers built a neat platform called Spare Cores that makes cloud instance pricing more transparent. They currently track 275,000 up-to-date server prices across 4 cloud providers – and built all of this on top of a €150K ($165K) EU innovation grant in ten months.

1. Industry pulse Google to be broken up?

In August, the US Department of Justice (DOJ) ruled that Google broke antitrust law by paying other vendors to be the default search engine. Now, the process moves on to decide the penalty. The Department of Justice (DOJ) is suggesting the break-up of the search giant, in order to end the company’s monopoly in search.

The DOJ is seeking “behavioral and structural remedies that would prevent Google from using products such as Chrome, Play, and Android to advantage Google search and Google search-related products and features.”

It is down to the judge to decide the penalty, and whether to take the drastic decision to break up the tech giant, in the way the US government broke up oil companies in the early 20th century. The last time it happened to a Big Tech company was in 2000, to Microsoft. Back then, the business appealed and got the decision overturned in 2001.

It reads like the DOJ will push for Search to be a separate company, and products like Chrome, Play and Android to operate in a unit independently. It is a fair question if products like Chrome would be viable without being subsidized by the profitable ads business. Microsoft and Apple both subsidize their browser development from other parts of their businesses.

If Chrome, Android, and Play became a separate entity, this entity could feature search engines like Google, Bing, DuckDuckGo and other search businesses, could pay to be the default search engine for a limited time, or for a region.

Google will no doubt fight hard and do whatever it can to avoid being broken up.

NVIDIA to compete with AWS, GCP, and Azure on GPUs?

16:24 • 13 days ago

Werdmüller on Medium

It turns out I’m still excited about the web

The tech industry has gone down a dark path, but there are glimmers of hope. Continue reading on Medium »

The tech industry has gone down a dark path, but there are glimmers of hope.

[Charlotte Tobitt at Press Gazette]

16:18 • 13 days ago

Ben Werdmüller

It turns out I'm still excited about the web

I’m worried I’ve become cynical about technology as I’ve gotten older. But maybe technology really is worse. Someone asked me the other day: “what [in media and technology] are you excited about right now?” We both agreed that it was a surprisingly difficult question. And then came the follow-up: “Do you think it’s just because we’re older now, or is the web really less exciting?” And to b

I’m worried I’ve become cynical about technology as I’ve gotten older. But maybe technology really is worse.

Someone asked me the other day: “what [in media and technology] are you excited about right now?”

We both agreed that it was a surprisingly difficult question. And then came the follow-up:

“Do you think it’s just because we’re older now, or is the web really less exciting?”

And to be honest, I’m not sure.

I used to be so excited. If you sneak a glance at my high school yearbook, you’ll see that I wanted to be a journalist. Telling stories was my first love. It’s still where my brain feels the most comfortable. I love the flow state of writing more than doing just about anything else. That’s why I keep writing here, and why my long-term plan is to pivot from a technology career to one where I get to write all the time.

But in 1994 or so, I got distracted by the web: what an amazing medium for stories. Many of us share the experience of trying out a browser like NCSA Mosaic, discovering voices from all over the world, and getting stuck into writing our own HTML code without having to ask anyone for permission or buy a software license to get started. I vividly remember when we got the ability to add our own background images to web pages, for example. For a long time, I was a master at table-based layouts.

In the UK, where I grew up, you were effectively forced to pick your university degree at 16. You were required to choose three or four A-level subjects to focus on for your last two years of high school; then you had to apply to do a particular degree at each university, knowing that each degree had subject requirements. If you wanted to study English at university, you needed to have chosen the English A-level; good luck getting in if you hadn’t.

Specifically because I was distracted by the web, I put myself on the Computer Science track. Even then, I kept a Theater A-level, because I couldn’t imagine a world where there wasn’t some art and writing in my life. Most British universities correspondingly dismissed me for not being focused enough, but Edinburgh took me, so that’s where I went. Even while I was doing the degree, I built a satirical website that got over a million pageviews a day - in 2001. I blogged, of course, and although I haven’t kept a consistent platform or domain for all that time, I’ve been writing consistently on the web since 1998.

It was a platform I got to approach with a sense of play; a sense of storytelling; a sense of magical discovery as I met new people and learned from their creativity.

The web sits apart from the rest of technology; to me, it’s inherently more interesting. Silicon Valley’s origins (including the venture capital ecosystem) lie in defense technology. In contrast, the web was created in service of academic learning and mutual discovery, and both built and shared in a spirit of free and open access. Tim Berners-Lee, Robert Cailliau, and CERN did a wonderful thing by building a prototype and setting it free. As CERN points out on its page about the history of the web:

An essential point was that the web should remain an open standard for all to use and that no-one should lock it up into a proprietary system.

That ethos is how it succeeded; it’s why the web changed the world. And it’s why someone like me — over in Scotland, with no networks, wealth, or privilege to speak of — was able to break in and build something that got peoples’ attention. It’s also why I was interested to begin with. “The internet is people,” I used to say; more than protocols and pipes, the web was a fabric of interconnectedness that we were all building together. Even in the beginning, some people saw the web and thought, “this is a way I can make a lot of money.” For me, it was always a way to build community at scale.

And then Facebook — it always seems to be Facebook — became the first web company to reach a billion dollar valuation, in a year that happened to also see the launch of the iPhone. Building community at scale became finding customers at scale. There was a brief reprieve while global financial markets tumbled at the hands of terrible debt instruments that had been built on shaky foundations, and then the tech industry started investing in new startups in greater and greater numbers. Y Combinator, which had started a few years earlier, started investing in more and more startups, with higher and higher checks ($6,000 per founder for the first cohort, compared to half a million dollars per startup today). The number of billion-dollar-plus web startups grows by the hundreds every year.

The web I loved was swamped by a mindset that was closer to Wall Street. It’s been about the money ever since.

It’s so rare these days to find people who want to build that interconnectedness; who see it as a mission and a movement. People in tech talk excitedly about their total Compensation (which has earned its own shorthand acronym, TC), and less so what exciting thing they got to build, and what it allowed people to do. Maybe they’ll give you a line about what they allow for the enterprise or increasing some company’s bottom line, but it’s usually devoid of the humanist idealism that enchanted me about the early web.

I realized some time ago that the startups I personally founded in this era couldn’t have succeeded, because my focus was all wrong. I wanted to be paid to explore and build this wonderful platform, and was not laser focused on how to build investor value. I still want to be paid to build and explore, try and make new things happen, with a sense of play. That’s not, I’m afraid to say, how you build a venture-scale business.

So, let’s return to the question. Given this disillusionment, and my lack of alignment with what the modern tech industry expects of us, what am I excited about?

My cynicism has been tempered by the discovery that there are still movements out there that remind me of the web’s original promise — efforts that focus on reclaiming independence and fostering real community. Despite the commercialization of the web, these are still places where that original spirit of openness and community-building thrives.

The Indieweb is one. It’s an interdisciplinary group of people that advocates for everyone owning their own websites and publishing from their own domains. It’s happening! From the resurgence of personal blogs to new independent publications like Platformer and User Mag, many people see the value of owning their presence on the internet and their relationships with their community. Independence from sites like Facebook and Google is surging.

The other is the Fediverse: a way to have conversations on the web that isn’t owned by any single company or entity. The people who are building the Fediverse (through communities, platforms like Mastodon, cultural explorations) are expanding a patchwork of conversations through open protocols and collaborative exploration, just like the web itself was grown decades ago. It’s phenomenally exciting, with a rapidly-developing center of gravity that’s even drawing in some of the companies who previously were committed to siloed, walled-garden models. I haven’t been this enthused about momentum on the web for twenty years.

I was afraid I had become too cynical to find excitement in technology again. It wasn’t true.

While I’ve grown more cynical about much of tech, movements like the Indieweb and the Fediverse remind me that the ideals I once loved, and that spirit of the early web, aren’t lost. They’re evolving, just like everything else.

16:17 • 13 days ago

@_Nat Zone

情報セキュリティワークショップ in 越後湯沢 2024での講演のスライドです

本日（10月10日）、情報セキュリティワークショップ in 越後湯沢で「ID管理の死角：なぜ脅威は減らないのか？」と題して講演を行わさせていただきました。その最終販のスライドです。よろしくご査収ください

本日（10月10日）、情報セキュリティワークショップ in 越後湯沢で「ID管理の死角：なぜ脅威は減らないのか？」と題して講演を行わさせていただきました。

その最終販のスライドです。

よろしくご査収ください

2024-10-10_ID管理の死角

13:09 • 13 days ago

Ben Werdmüller

Why Reach journalists are being asked to write up to eight articles per day

[Charlotte Tobitt at Press Gazette] "Paul Rowland wrote in an email to staff on 27 September that article volumes were being talked about “a lot in newsrooms at the moment” and blamed, in part, the volatility from previously huge traffic referrers like Google and Facebook." "A separate email, sent by Birmingham Live editor Graeme Brown last month, suggested journalists shoul

"Paul Rowland wrote in an email to staff on 27 September that article volumes were being talked about “a lot in newsrooms at the moment” and blamed, in part, the volatility from previously huge traffic referrers like Google and Facebook."
"A separate email, sent by Birmingham Live editor Graeme Brown last month, suggested journalists should file at least eight stories per day unless they were newsgathering outside of the office."

Referrals from Facebook are down from 50% of traffic to 5%, and every newsroom is seeing similar declines from both social and search. But this is an insane way to deal with it: asking every journalist to file eight stories a day is a way to drive quality through the floor and exacerbate a downward spiral.

You can't just keep doing what you're doing but more of it. This change requires a rethink of platform and more ownership over newsroom technology: it's time to actually innovate around what it means to publish on the web, and to, finally, move from "audience" to "community".

To be blunt: every newsroom publishing on the web that doesn't do this will go away.

[Chris Person at Aftermath]

[Link]

13:02 • 13 days ago

Waffle House Index labels Hurricane Milton red, closes stores

[Ben Kesslen at Quartz] "Waffle House, the iconic American restaurant chain with over 1,600 locations known for cooking up Southern breakfast food, has developed an advanced storm center FEMA consults with." Stores in the path of Milton were closed in advance of the storm, which is rare for Waffle House, which is often the last store standing. It's been sophisticated about

[Ben Kesslen at Quartz]

"Waffle House, the iconic American restaurant chain with over 1,600 locations known for cooking up Southern breakfast food, has developed an advanced storm center FEMA consults with."

Stores in the path of Milton were closed in advance of the storm, which is rare for Waffle House, which is often the last store standing.

It's been sophisticated about storm predictions and response since Katrina:

"The chain also developed the Waffle House Storm Index, which was started after former FEMA Administrator Craig Fugate said, “If you get there and the Waffle House is closed? That’s really bad. That’s where you go to work.”"

As Pat Warner, a member of "the Waffle House crisis management team" said in the article, it's not about the extra sales Waffle House gets when it does re-open, often using generators and other emergency equipment. It's more to do with how this integrates the stores with their communities. They wouldn't do it if there wasn't a positive uplift for the business, but it comes across as a genuine desire to help.

#Society

[Link]

02:43 • 13 days ago

Forums Are Still Alive, Active, And A Treasure Trove Of Information

[Chris Person at Aftermath] "Over the years, forums did not really get smaller, so much as the rest of the internet just got bigger. Reddit, Discord and Facebook groups have filled a lot of that space, but there is just certain information that requires the dedication of adults who have specifically signed up to be in one kind of community. This blog is a salute to those foru

"Over the years, forums did not really get smaller, so much as the rest of the internet just got bigger. Reddit, Discord and Facebook groups have filled a lot of that space, but there is just certain information that requires the dedication of adults who have specifically signed up to be in one kind of community. This blog is a salute to those forums that are either worth participating in or at least looking at in bewilderment."

What an amazing index of indie forums still going strong on the web.

I'd love to do a survey of what they're powered by, and in turn, I'd love to read interviews of the product / engineering leads for each of these platforms. Are they individual developers, keeping the lights on out of love? Are they thriving companies? Something else? I'm fascinated that there's these corners of the web that haven't changed all that much in decades, but are full of life, supported by platforms that surely must have to evolve to deal with threats and abuse at the very least.

I love all of it. This kind of thing is what makes the web great.

#Culture

[Link]

02:25 • 13 days ago

Wednesday, 09. October 2024

IdM Laboratory

Windowsのパスキー対応の今後

こんにちは、富士榮です。いよいよ来週はAuthenticate 2024ですね。残念ながら参加できませんが。ということで、Authenticateに向けて各社パスキー周りの話題が進んできていそうです。 MicrosoftからもWindowsのパスキー対応について記事を公開しています。 Passkeys on Windows: Authenticate seamlessly with passkey providers https://blogs.windows.com/windowsdeveloper/2024/10/08/passkeys-on-windows-authenticate-seamlessly-with-passkey-providers/ こちらの機能がWindows Insiderチャネルで配信されるようです。久しぶりにWind

こんにちは、富士榮です。

いよいよ来週はAuthenticate 2024ですね。残念ながら参加できませんが。

ということで、Authenticateに向けて各社パスキー周りの話題が進んできていそうです。

MicrosoftからもWindowsのパスキー対応について記事を公開しています。

Passkeys on Windows: Authenticate seamlessly with passkey providers

https://blogs.windows.com/windowsdeveloper/2024/10/08/passkeys-on-windows-authenticate-seamlessly-with-passkey-providers/

こちらの機能がWindows Insiderチャネルで配信されるようです。久しぶりにWindows PCでも触ろうかな・・・

A plug-in model for third-party passkey providers Enhanced native UX for passkeys A Microsoft synced passkey provider

サードパーティプロバイダとの連携では1Passwordなどとの連携ができるようになるようです。3点目のMicrosoftが提供する同期ファブリックと連携できたりすると面白そうです。Credential Exchange Specificationが実装されてくると面白いと思います。

いずれにしても来週のAuthenticateで詳しく言及されるのかと思います。楽しみですね。

23:31 • 13 days ago

Justin Richer

GNAP: A Conversation of Authorization

After five years of standardization work, GNAP is now officially RFC9635! This long and intense process actually started a few years prior to that, when I was talking with a lot of folks in the security industry about some of the shortcomings of OAuth 2.0, and what we could do about them as an industry. These conversations led to the XYZ proposal (and implementations) which eventually led to the formation of the GNAP working group along with a bunch of others. In particular, the work that Fabien Imbault, Yaron Sheffer, Leif Johannsen, and Aaron Parecki put into the documents and conversations in the working group over these years.

I’m really proud of what we’ve built in GNAP. One of the core tenets of GNAP was to look at the world of OAuth and surrounding technologies and figure out how we could do a lot of that better. It’s been great to see GNAP getting applied in a bunch of places over the web, from payments to key management, and especially in places where OAuth doesn’t reach as well. While OAuth remains deeply entrenched over the world, and likely will be for some time, the community has learned many things from GNAP. Alot of things that started in GNAP have been making their way back to the OAuth ecosystem in some form.

The most obvious of this is RFC9396: OAuth Rich Authorization Requests. This replacement of OAuth’s scope parameter was a direct and intentional backport of what became GNAP’s resource access rights, which also acronyms to RAR. In the OAuth world, we don’t get some of the clean features of GNAP, like being able to substitute strings for objects as a shorthand, but a lot of the core enhancements are there.

We’re also seeing yet another intent registration addition to OAuth 2 (on top of the pushed authorization request, device grant type, and CIBA extensions), and this one mimics a lot of the flexibility of GNAP’s interaction system. It’s a more narrow use case in the OAuth specification, but it’s clear that the pattern that GNAP was built on is here to stay.

And then there’s RFC9421: HTTP Message Signatures. This is work that started independently from GNAP but grew up around the same time, and GNAP utilizes HTTP Message Signatures as a core security function. I don’t think we’d have gotten the signing spec to be as robust as it is without some of the GNAP key proofing use cases driving the discussion.

And finally, the GNAP Resource Servers document has just passed IESG review and is on its own way to becoming an RFC as well. This document represents key abstractions in how and RS and AS relate to each other, and I hope we can continue to build this out and pull the best ideas out into the world.

The GNAP working group is shutting down now that its core work is done, but GNAP is far from over. I look forward to seeing it grow into its spaces, and serve as a beacon of how a delegation protocol can be engineered and built.

19:50 • 13 days ago

Heres Tom with the Weather

We fix the fucking networks

“A lot of us remember what it was like to live and work on an Internet that was deeply flawed but not systematically designed to burn our emotions and time and safety for fuel.”

18:45 • 14 days ago

Just a Theory

PGXN v2 Update

A lot has happened in he five months since the last PGXN v2 update. The time has been used for continuing community discussions, planning, designs, and the start of implementation. Read on for a full accounting.

Speaking of PGXN news, I neglected to link to this post I wrote for the Tembo Blog last month, a fairly detailed accounting of what’s been happening on the PGXN v2 project:

Forgive me Postgres community, for it has been five months since my last PGXN v2 Update. In my defense, it has been super busy! The time went into ongoing community discussions, planning, designs, and the start of implementation. Join me below for the lowdown.

A few highlights:

PGXN RFCs Repository and rfcs.pgxn.org Binary Distributution POC and OCI POC Extension Ecosystem Summit API and ABI guidance pgxn_meta v0.1.0 PGXN Meta JSON Schemas project plan

There’s been quite a bit of activity since then, including the aforementioned PGXN RFC–5 — Release Certification. More soon!

More about… PGXN

17:14 • 14 days ago

PGXN Certifications RFC

A request for comments on a new PGXN RFC for signing releases, plus a link to an initial implementation.

A couple weeks ago, I drafted PGXN RFC–5 — Release Certification, which proposes to replace the simple inclusion of a SHA-1 hash digests in PGXN release META.json files with a JWS-signed release payload. From the introduction:

This RFC therefore proposes to extend v2 distribution metadata with a single additional property, certs, that contains one or more certifications that attest to the authenticity or other characteristics of a release on PGXN.

The certs value is an object that contains at least one property, pgxn, which itself contains a PGXN-generated RFC 7515 JSON Web Signature in the JWS JSON Serialization format. The pgxn property will allow clients not only to assemble the release URL and verify the downloaded file against checksums, but also validate it against a public key provided by PGXN.

The design allows multiple signatures, certifications, or other attestations, which in the future MAY allow authors or other entities to sign releases with their own keys. The new format appends a structure such as this to the distribution META.json file:
{ "certs": { "pgxn": { "payload": "eyJ1c2VyIjoidGhlb3J5IiwiZGF0ZSI6IjIwMjQtMDktMTNUMTc6MzI6NTVaIiwidXJpIjoiZGlzdC9wYWlyLzAuMS43L3BhaXItMC4xLjcuemlwIiwiZGlnZXN0cyI6eyJzaGE1MTIiOiJiMzUzYjVhODJiM2I1NGU5NWY0YTI4NTllN2EyYmQwNjQ4YWJjYjM1YTdjMzYxMmIxMjZjMmM3NTQzOGZjMmY4ZThlZTFmMTllNjFmMzBmYTU0ZDdiYjY0YmNmMjE3ZWQxMjY0NzIyYjQ5N2JjYjYxM2Y4MmQ3ODc1MTUxNWI2NyJ9fQ", "signature": "cC4hiUPoj9Eetdgtv3hF80EGrhuB__dzERat0XF9g2VtQgr9PJbu3XOiZj5RZmh7AAuHIm4Bh-rLIARNPvkSjtQBMHlb1L07Qe7K0GarZRmB_eSN9383LcOLn6_dO--xi12jzDwusC-eOkHWEsqtFZESc6BfI7noOPqvhJ1phCnvWh6IeYI2w9QOYEUipUTI8np6LbgGY9Fs98rqVt5AXLIhWkWywlVmtVrBp0igcN_IoypGlUPQGe77Rw" } } }

Review and feedback would be very much appreciated, especially on the list of unresolved questions toward the end.

Thanks to David Christensen and Steven Miller for the early reviews!

Meanwhile, I’ve released pgxn_meta v0.4.0, which adds support for this format, as well as code to rewrite PGXN v1 release fields to the new format. It doesn’t actually do signature verification, yet, as the server back end hasn’t been updated with the pattern and PKI. But I expect to modify it in response to feedback and get it implemented in early 2025.

More about… Postgres PGXN RFC JWS

16:26 • 14 days ago

The Pragmatic Engineer

Efficient scaleups in 2024 vs 2021: Sourcegraph (with CEO & Co-founder Quinn Slack)

On today’s episode of The Pragmatic Engineer, I’m joined by Quinn Slack, CEO and co-founder of Sourcegraph, a leading code search and intelligence platform.

The Pragmatic Engineer Podcast had a very warm reception with the first episode — thank you! Two weeks later, here is episode #2, with a CEO who still codes every day: Quinn Slack at Sourcegraph.

Listen now on Apple, Spotify, and YouTube.

Brought to you by:

• Paragon: Build native, customer-facing SaaS integrations 7x faster.

• Enterprise Ready Conferfence on 30 October: a one-day event in SF for product and engineering leaders shaping the future of enterprise SaaS.

On today’s episode of The Pragmatic Engineer, I’m joined by Quinn Slack, CEO and co-founder of Sourcegraph, a leading code search and intelligence platform. Quinn holds a degree in Computer Science from Stanford and is deeply passionate about coding: to the point that he still codes every day! He also serves on the board of Hack Club, a national nonprofit dedicated to bringing coding clubs to high schools nationwide. In this insightful conversation, we discuss:

How Sourcegraph's operations have evolved since 2021

Why more software engineers should focus on delivering business value

Why Quinn continues to code every day, even as a CEO

Practical AI and LLM use cases and a phased approach to their adoption

The story behind Job Fairs at Sourcegraph and why it’s no longer in use

Quinn’s leadership style and his focus on customers and product excellence

The shift from location-independent pay to zone-based pay at Sourcegraph

And much more!

Takeaways

Some of my main takeaways from this conversation are these.

1. As software engineers, it’s increasingly important to understand what value you add to the business. A big difference between 2021 and 2024 is how companies are much more focused on efficiency: meaning they are hiring more conservatively, and less likely to fund teams with headcount that don’t contribute to the core focus of the company.

As a developer or manager, try to figure out how much your team contributes in revenue, or savings, or other key goals to the company. Are you working in what the company would consider as a profit center, or what is more as a cost center? We did a deepdive on this topic in The Pragmatic Engineer: check out the article, linked in the show notes.

2. AI tools are great to eliminate the toil that we, developers face, day-to-day. There are AI tools that position themselves as their goal being “replacing developers.” I found it sympathetic that Quinn did not think this is the sensible path. His approach is to start by using AI tools with some of the “dumbest things” like generating the changelog for a software release – I mean, assuming you generate a changelog. And then you take tedious tasks where these tools could help, and see if you can automate some more.

Do this one step at a time: and it will actually help devs and teams, and it’s a lot more achievable than saying “let’s replace this whole complicated workflow with AI.”

3. The reality of location independent pay is that it stops being sensible above a certain company size. Sourcegraph was one of the few companies that offered the same base salary regardless of where people worked at. They did this until they grew to about 200 people, and switched this model to a location-indexed model.

Quinn was honest about why they did it: because keeping this would have not made sense for the company, from the business point of view. Basically, location-independent pay means the company can hire very easily in low-cost regions, but it’s hard or impossible to do this in high-cost regions. It also creates the incentive for employees to move to a low cost region where they can save more. In the end, I don’t know of any company with more than 200 people that pays location-independent: all large companies have some kind of indexing on location, and the best companies just pay the top of the local market. We cover more about compensation in the deepdive on The trimodal nature of software engineering salaries.

If you enjoy the podcast, I’d very much appreciate if you subscribe on your favorite podcast player, and leave a review. It helps the podcast be discovered by more people. Thank you!

Show notes

Where to find Quinn Slack:

• X: https://x.com/sqs

• LinkedIn: https://www.linkedin.com/in/quinnslack/

• Website: https://slack.org/

In this episode, we cover:

(01:35) How Sourcegraph started and how it has evolved over the past 11 years

(04:14) How scale-ups have changed

(08:27) Learnings from 2021 and how Sourcegraph’s operations have streamlined

(15:22) Why Quinn is for gradual increases in automation and other thoughts on AI

(18:10) The importance of changelogs

(19:14) Keeping AI accountable and possible future use cases

(22:29) Current limitations of AI

(25:08) Why early adopters of AI coding tools have an advantage

(27:38) Why AI is not yet capable of understanding existing codebases

(31:53) Changes at Sourcegraph since the deep dive on The Pragmatic Engineer blog

(40:14) The importance of transparency and understanding the different forms of compensation

(40:22) Why Sourcegraph shifted to zone-based pay

(47:15) The journey from engineer to CEO

(53:28) A comparison of a typical week 11 years ago vs. now

(59:20) Rapid fire round

The Pragmatic Engineer deepdives relevant for this episode:

• Inside Sourcegraph’s engineering culture: Part 1 https://newsletter.pragmaticengineer.com/p/inside-sourcegraphs-engineering-culture
• Inside Sourcegraph’s engineering culture: Part 2 https://newsletter.pragmaticengineer.com/p/inside-sourcegraphs-engineering-culture-part-2

Referenced:

• Sourcegraph: https://sourcegraph.com/

• SharePoint: https://www.microsoft.com/en-us/microsoft-365/sharepoint/collaboration

• Google Code Search: https://developers.google.com/code-search

• Steve Yegge on LinkedIn: https://www.linkedin.com/in/steveyegge

• Shopify: https://www.shopify.com/

• Tobi Lutke still writes code for Shopify: https://news.ycombinator.com/item?id=25003574

• Gitlab: https://about.gitlab.com/

• Scott Kirby on LinkedIn: https://www.linkedin.com/in/jscott-kirby/

• Beyang Liu on LinkedIn: https://www.linkedin.com/in/beyang-liu/

• Perl: https://www.perl.org/

• Booking.com: https://www.booking.com/

• Loom: https://www.loom.com/

• CleanShot: https://cleanshot.com/

• Kazam: https://launchpad.net/kazam

• Ollama: https://ollama.com/

• StarCoder: https://huggingface.co/blog/starcoder

• Llama: https://www.llama.com/

• The Years of Lyndon Johnson: The Path to Power; Means of Ascent; Master of the Senate; The Passage of Power: https://www.amazon.com/Robert-Caros-Years-Lyndon-Johnson/dp/038535147X

• The Life Of Mahatma Gandhi: https://www.amazon.com/Life-Mahatma-Gandhi-Louis-Fischer/dp/1784700401

• Nimitz at War: Command Leadership from Pearl Harbor to Tokyo Bay: https://www.amazon.com/Nimitz-War-Command-Leadership-Harbor/dp/0190062363/r

Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email podcast@pragmaticengineer.com.

15:48 • 14 days ago

Just a Theory

⛰️ Postgres Ecosystem Summit EU

The sequel to the successful PGConf.dev event, the Extension Ecosystem Summit EU will showcases some exemplary extension use cases.

Given the success of the Extension Ecosystem Summit at PGConf.dev back in May, my colleague Floor Drees has organized a sequel, the Extension Ecosystem Summit EU on Tuesday, October 22, at the Divani Caravel Hotel in Athens. That’s “Day 0” at the same hotel as PGConf.eu. Tembo, Percona, Xata, and Timescale co-sponsor.

While the May event took the form of an open-space technology (OST)-style unconference aimed at extension developers, the EU event aims to inform an audience of Postgres users about the history and some exemplary use cases for extensions. From the invite:

Join us for a gathering to explore the current state and future of Postgres extension development, packaging, and distribution. Bring your skills and your devices and start contributing to tooling underpinning many large Postgres installations.
Jimmy Angelakos - pg_statviz: pg_statviz is a minimalist extension and utility pair for time series analysis and visualization of PostgreSQL internal statistics. Adam Hendel (Tembo) - pgmq: pgmq is a lightweight message queue. Like AWS SQS and RSMQ but on Postgres. Adam is pgmq’s maintainer since 2023, and will present a journey from pure Rust → pgrx → pl/pgsql. Alastair Turner (Percona) - pg_tde: pg_tde offers transparent encryption of table contents at rest, through a Table Access Method extension. Percona has developed pg_tde to deliver the benefits of encryption at rest without requiring intrusive changes to the Postgres core. Gülçin Yıldırım Jelínek (Xata) - pgzx: pgzx is a library for developing PostgreSQL extensions written in Zig. Mats Kindahl (Timescale) - TimescaleDB (C), [pgvectorscale] (Rust) and pgai (Python): maintaining extensions written in different languages.

I will also deliver the opening remarks, including a brief history of Postgres extensibility. Please join us if you’re in the area or planning to attend PGConf.eu. See you there!

More about… Postgres PGXN Extensions PGConf Athens Summit

15:27 • 14 days ago

Tuesday, 08. October 2024

IdM Laboratory

SIDI Hub - ベルリンレポートを読む（５）

こんにちは、富士榮です。引き続きSIDI Hubベルリンレポートを読んできましょう。今回はユースケースをベースにしたGap分析です。Deboraがレポートしてくれています。相互運用性を担保する上で大きな障壁になりそうな課題として以下を挙げています。 We focused on three topics: Relying Party Registration: it is tackled in the scope of EUDIW and covered by Aadhaar, NIMC, and others on a country-by-country basis. But how does this interoperate across borders on a global scale? Issuing Authority Discover

こんにちは、富士榮です。

引き続きSIDI Hubベルリンレポートを読んできましょう。

今回はユースケースをベースにしたGap分析です。Deboraがレポートしてくれています。

相互運用性を担保する上で大きな障壁になりそうな課題として以下を挙げています。

We focused on three topics:
Relying Party Registration: it is tackled in the scope of EUDIW and covered by Aadhaar, NIMC, and others on a country-by-country basis. But how does this interoperate across borders on a global scale? Issuing Authority Discovery: ICAO centralized this for passports after many years. But how will this work for public and private sector issuers? Legal Entity Identifiers: the LEI (GLEIF) and DNS (ICANN) are two current examples. What is the best way to achieve legal entity linking?
We then facilitated a discussion, and the following summarizes the key points addressed in the room.

我々は3つのトピックに焦点を当てた：
リライングパーティの登録：EUDIWの範囲内で取り組まれており、AadhaarやNIMCなどが国ごとにカバーしている。しかし、世界規模で国境を越えてどのように相互運用するのか。発行機関の発見： ICAOは何年も経ってから、パスポートのためにこれを一元化した。しかし、公的機関や民間企業の発行者にとってはどのように機能するのだろうか。法的実体識別子：LEI（GLEIF）とDNS（ICANN）が現在の2つの例である。取引主体の連結を実現する最善の方法は何か？
その後、ファシリテーターによるディスカッションが行われ、その中で取り上げられたポイントを以下に要約する。

どれも頭の痛い問題ですね。特に2点目、3点目は答えが出そうにない課題ですねぇ。。いつまで経ってもIssuerのディスカバリは難しい問題です。ここで言っているのは単純に公開鍵を取得するためのURLのディスカバリだけじゃないですからね・・・どうやってIssuerが正当な機関であることを信じられるか、みたいな話です。また、識別子も非常に難しい問題です。DNSは比較的成功したモデルではありますが、それでも過去に使っていたドメインを別の機関が取得するという問題などもありますので、長期的に運用する上では非常に難しいかと思います。

それぞれ深掘りしていきます。

Relying Party (RP) Registration:
The group discussed the nature of Registration, its requirements, and how Trust establishment could work globally.
Are we focused only on foundational identity, or do we include functional identity systems? Example from Nigeria: the agency responsible for ID management is NIMC. In the case of foundational identity, one of the first things they do is a process of due diligence called Verification. RPs are registered mainly for the foundational part. It is specific to the country.

依拠当事者（RP）登録：
このグループでは、登録の性質、要件、および信頼性確立がグローバルにどのように機能するかについて議論した。
私たちは基盤的 ID のみに焦点を当てているのか、機能的 ID システムも含めるのか。
ナイジェリアの例：ID 管理を担当する機関は NIMC である。基盤的 ID の場合、最初に行うことの 1 つは、検証（Verification）と呼ばれるデュー・ディリジェンス・プロセスである。RP は主に基礎部分のために登録される。これはその国特有のものである。 Why are RPs registering? What are the requirements? What are the types of problems we are trying to solve?

Example of the mDL standard: the Trust ecosystem is only for Issuers. If I share my mDL with you, why should I trust you? This concern is especially relevant for a commercial vendor, e.g., Aadhaar they have to register all RP fingerprint devices with governments to know they are trusted

なぜRPは登録するのか？要件は何か？どんな問題を解きたいのか？

mDL標準の例：トラスト・エコシステムは発行者のためだけのもの。mDLを共有した場合、なぜ信用しなければならないのか？この懸念は特に商業ベンダーに関連する。例えば Aadhaar の場合、信頼できることを知るためにすべての RP 指紋デバイスを政府に登録しなければならない。

Should the solution be based on use cases? Should it be a risk-based approach?

It depends on the type of credentials, e.g., education with entity categories. A commercial entity doesn’t need your entire transcripts. ソリューションはユースケースに基づくべきか。リスク・ベースのアプローチにすべきか？

クレデンシャルのタイプ（例えば、エンティティ・カテゴリーを持つ教育）によって異なる。営利団体は成績証明書全体を必要としない。 Should it be public or private-led, or a combination of both?

Example, more public-led: ICAO

Example, more private-led: ICANN 公共主導か民間主導か、あるいは両者の組み合わせか？

より公共主導の例：ICAO

より民間主導の例：CANN Governance relates to funding the operating costs: would it be self-funded like ICAO? Should it be external funding? What are good reference models?

ICAOのように自己資金で運営するのか？外部資金とすべきか？良い参考モデルは何か？

Should it be global or regional?

AAMVA is in North America and only about driver's licenses

グローバルかリージョナルか

AAMVAは北米にあり、運転免許証に関するものだけである。

How would we approach the following:

Lifecycle management?

Type of data?

Legitimacy & KYB?

Policy enforcement?

以下について、どのようにアプローチしますか？

ライフサイクル管理？

データのタイプ？

正当性およびKYB？

ポリシーの施行？ Should we pursue an academic analysis of the options? オプションについて学術的な分析を行うべきでしょうか？

Who are the decision-makers, and why? 意思決定者は誰で、その理由は？

Is there a hierarchy or a pre-existing way to navigate views? 階層やビューをナビゲートする既存の方法はあるのでしょうか？

What is the appropriate role for:

Governments?

NGOs like the UN? Is the UN sufficiently independent?

Standards Organizations?

以下について適切な役割とはどのようなものか：

政府？

国連のようなNGO？国連は十分に独立しているか？

標準化団体？ What is required to achieve consensus? コンセンサスを得るために必要なことは何でしょうか？

We discussed that the Champion Use Cases will indicate the breadth of the issues we have to face if we go for the widest possible interoperability
チャンピオンユースケースは、最大限の相互運用性を実現しようとする場合に直面する問題の広がりを示すことになるだろう、という点について話し合いました。

当然ですが、相互運用を考えるとかなり幅広い議論が必要となりますね。

もう少しスコープを絞って議論をシャープにしていかないとまとまらない気もします・・・（少なくとも一気に全体ミーティングでまとまる量じゃない）

リライングパーティだけで上記ボリュームだったので、他にもガバナンスなどもあるので、この辺りは明日以降に。

23:09 • 14 days ago

The Pragmatic Engineer

What is Reliability Engineering?

A history of SRE practice and where it stands today, plus advice on working with reliability engineers, as a software engineer. A guest post by SRE expert and former Googler, Dave O’Connor

Hi, this is Gergely with a subscriber-only issue of the Pragmatic Engineer Newsletter. In every issue, I cover challenges at Big Tech and startups through the lens of engineering managers and senior engineers. To get articles like this in your inbox, every week, subscribe:

What is Security Engineering?

For software engineers, the job involves more than just building software systems; these systems must also be reliable. This is easy enough for a website with a small number of visitors; but the larger the system gets, the trickier reliability is to achieve. There’s a huge amount of complexity involved in making an app or website with tens, or hundreds, of millions of daily users work reliably for (almost) all of them.

Google pioneered the concept of Site Reliability Engineering (SRE), and it has become a pretty mainstream discipline with many mid size-and-above tech companies having dedicated SRE or reliability teams. To find out more about SRE and reliability engineering in general, I reached out to SRE veteran, Dave O’Connor. He was at Google in 2004 – working with the team, from where the SRE discipline emerged just a year before, in 2003.

Today, Dave covers:

Common terms. Why it’s called ‘reliability engineering’ instead of SRE, DevOps teams, etc.

History. From conception in 2003 at Google, to industry convergence in the 2010s.

Reliability engineering today. This is a discipline in growth mode. But near-perfect reliability increases cost exponentially, and most companies should not copy Google’s approach to the SRE role.

Four promises. A well-run reliability team sets and delivers on four promises: SLA/SLO/SLI, service substrate management, tech design involvement, and tactical heroism.

Working with reliability engineers. If there are reliability engineers in your workplace, find out their approach to the four promises, involve them in tech design discussions, and seek their help with reliability challenges.

Future of reliability engineering. Tooling is evolving, and a trend of companies considering quitting the cloud and being more opinionated about reliability practice.

Further reading. Article and book recommendations.

For related reading, see also these The Pragmatic Engineer deepdives:

Shipping to production

Healthy oncall practices

What is Data Engineering?

What is ML Engineering?

With this, it’s over to Dave.

Hi, I’m Dave. I’ve been a site reliability engineer (SRE) for 20 years, before many folks outside the Google ecosystem called it that. I joined the company in 2004, on a team tasked with (re)installing and (re)configuring machines in the fleet.

We quickly realized that due to sheer numbers, it was not a job that could be done by humans at the then-current scale, let alone at the scale expected. At the time, the common capability to run and manage more than a few hundred machines simply didn’t exist. Then began the chartering of what became known as ‘site reliability engineering’ at Google. The rest, as they say, is history.

Several years later, that history started to be told in Site Reliability Engineering (I contributed chapter 29), and various publications thereafter.

Since leaving Google in 2021 after 17 years as an SRE, I’ve led SRE and production groups at Elastic and Twilio, and I’m currently freelancing as a leadership practitioner for busy teams (SREs included), and as a coach for engineering leaders, focusing on reliability engineers. Check out my consulting services, and coaching practice.

1. Common terms

I use the term ‘reliability engineering’ in this article. This is because as a set of practices, it stands on its own and can be implemented via specialized ‘SREs’, the ‘DevOps’ model, or individually as software is built. Many publications go to great lengths to make this distinction, and the question of whether reliability should be done by specialized SREs, or everyone, is a resounding ‘it depends’. See my article 6 Reasons You Don’t Need an SRE Team for why I believe many companies don’t need a dedicated function.

As with any engineering specialization, anybody can do reliability engineering up to a point. The decision on hiring or building a dedicated SRE group is similar to the choice of whether to hire a dedicated QA, or an information security group. Does the business care enough about the outcomes to dedicate people and organizational headspace to it? Often, the answer is initially “no”. However, almost every enterprise encounters novel and domain-specific engineering challenges at some point, and in these situations dedicated SREs provide better outcomes, tailored to individual needs.

“Site reliability” as a name was coined by Google. The “site” in question was google.com. However, the role has grown since; for many years at Google I led the SRE group in charge of all the storage and databases. These were not ‘sites’ per se, but the name had stuck by that point.

As with most nascent engineering functions, folks who do reliability engineering go by many titles:

SREs

Production engineers

DevOps

Platform engineering

DevSecOps

Platform trust

Infrastructure engineering

… and other verbal gymnastics!

Reliability engineering goes by a lot of titles, the best-known being SRE

These titles all encompass pretty common practices. It’s also argued – correctly, if you ask me – that reliability engineering predates what Google did. Many of its common-sense or even specialized practices are taken from other disciplines. For example, the concept of the incident commander role for large-scale outages derives from the US Federal Emergency Management Agency (FEMA), founded in 1979.

2. History

As computing has evolved and gone increasingly online, our needs have scaled beyond the capability of humans to perform tasks, or even understand the scale at which we find ourselves doing things. When there are 100,000+ machines, it’s impossible to eyeball a graph of all data points like servers or machines. We can not take a well-understood administrative action on a machine if we have to replicate it 100,000 times.

In the early 2000s, many enterprises went from a manageable number of physical computers to large numbers of servers. These servers performed one function and were clustered, or they could take more generic workloads.

As Gavin McCance, Compute Services lead at CERN, put it, we started thinking of individual machines as cattle, not pets:

Transition from ‘cats’ to ‘cattle’. Source: CERN Data Centre Evolution by Gavin McCance

In the early days, the frame of reference for how many “a lot” of machines was, shifted almost monthly. At Google in around 2006, I went on a site visit to another large tech employer in Ireland, during which our group peered through a small window into “the largest datacenter in Ireland”, numbering thousands of machines. Cue some raised eyebrows and polite ‘ahems’ because we knew we had a room with twice as many machines, located half an hour’s drive away.

Google’s scaling ability lay in being able to assemble and power machines, but also in being able to pour concrete and purchase supporting equipment like generators, when supply chains simply weren’t set up for our scale. It represented an enormous uptick in the amount of real, difficult, and unsolved engineering problems in delivering services.

For 99% of enterprises, this kind of massive scaling up in servers isn’t (nor should be) a core competency. Hence the explosion in cloud computing because amassing the human expertise to understand and run all layers in the stack is far beyond the purview of most businesses.

The early SRE role

At places like Google, it made sense to build the expertise and technology to cover all layers of the stack, from the frontend serving infrastructure and network links, back to the physical machines and power infrastructure on the ground. This was for reasons that can be linked back to one thing: scale.

When I joined the tech giant the number of physical machines was in the process of sailing comfortably into six digits, crossing 100,000 and growing fast. By comparison, at most enterprises, a couple of hundred machines was considered a pretty large footprint.

This had two major forcing functions, both related to scale:

Tooling at the time was simply not set up to handle the scale at which Google needed to operate. There were no datacenter-scale tools (where ‘datacenter’ now means many thousands of machines) that could install, configure, or administer machines at our scale. At the control plane level, there were no job or workload management systems suited to the task. Machine virtualization was already in use in many folks’ production setups, but Containerisation, Kubenetes, Terraform, Prometheus, and many other familiar components didn’t exist yet. In fact, Kubernetes and Prometheus are based, in part, on later internal Google tools.

Also basically non-existent were SaaS options such as PagerDuty, and observability tools like Datadog and Honeycomb.

It would take entire buildings and campuses full of people to manually operate machines. We were capable of hiring a lot of people, but we knew that without a giant shift in the technology and tooling, the tools and ways to operate machines weren’t going to work. For places like Google, this meant we’d quickly run out of people to be able to deal with the complexity and workload. We needed toolchains and practices beyond what the industry could provide.

An additional forcing function for us was Google’s leadership’s almost fanatical desire for reliability and speed. Larry, in particular, cared a lot. Just as Gmail was launching and offering users an previously-unthinkable entire gigabyte of email storage, we were aiming for levels of precision and speed in serving content that were unheard of in most industries.

The fervent belief of Google’s founders was that speed and reliability mattered more than features. This belief was coupled with the understanding that we couldn’t achieve it traditionally, which made it an existential issue. The level of investment in building out all layers of the serving stack was a case of “because we can”, but also “because we have to, as nowhere else does what we need”.

There was never a question of whether traditional ‘ops’ would work at Google. We needed a specialized role, staffed by folks familiar with the problem space and engineering methods required to make it work.

In 2003, the SRE role was born. Ben Treynor Sloss had been tasked with building Google’s “production team” and in his own words, he built “what happens when you ask a software engineer to design an operations team.” This turned into the birth of the SRE function at Google. From the outset, SRE was staffed in varying measures by systems/operations experts and software engineers. A large part of the remit of the team was to build the tools and practices required to operate Google’s fleet.

I joined as one of the first non-US SREs, based in Ireland. My background is in systems administration, and my previous employer’s machine fleet numbered somewhere in the high double-digits. I was one of the newer, specialized breed of “sysadmins who code”. We didn’t have a snappy name, but did have the drive to embody the three virtues of ‘laziness, impatience and hubris.’

When I joined, my first gig was ‘babysitting’ Gmail’s machine fleet. Basically, the job was to ensure there were enough machines to serve storage and serving needs, and to juggle decisions on waiting for tools and processes to catch up, or building them. In particular, many practices for working in distributed teams containing up to nine time zones, came from the early experience of collaborating with our SRE and product development counterparts in Mountain View and other offices.

Industry Convergence

Eventually, other companies caught onto the scaling issues, especially the hyperscalers. Each had their own approach, but over time, the notion grew industry-wide that making things reliable was a real-life engineering discipline, not simply ‘ops’.

This step saw a number of terms coined to describe this engineering, including ‘DevOps’. At its core, this was the notion that the disciplines and practices of reliability engineering should be ingrained into the overall engineering organization. At places other than Google, this mostly took the form of combined developer/operations roles (i.e. “you build it, you run it”), which differed from Google’s implementation, but the practices were similar.

Around this time, Google started opening up about SRE, eventually publishing the first SRE book, and follow ups. Conferences such as USENIX SRECon, Devops Days, and other movements have solidified reliability engineering as a discipline that scales well beyond Google. Indeed, the company has become a consumer of many state-of-the-art developments.

3. Reliability Engineering Today

Reliability engineering is still in its growth and adoption phase. Unreliable software and systems which are slow or function incorrectly, are no longer tolerated by businesses and direct consumers. Fast, reliable internet access is becoming ubiquitous, and the services people use must be the same.

But aiming for near-perfect reliability scales costs exponentially. It’s estimated, based on experiences at AWS, that every “nine” of additional guaranteed availability (the difference between 99%, 99.9% and 99.99% uptime) scales overall costs by roughly ten times. This includes staffing, development and testing costs, and may only partially account for the opportunity costs of a necessarily slower release cycle. But slower release cycles aren’t for everyone! If you’re in a product space which can’t tolerate spending too much time and energy on testing and resiliency, the right answer may well be to aim lower.

It’s sensible practice for any organization to explicitly state how much they care about reliability. Know that it is not a race to the top: be realistic about balancing the price you’re willing to pay, with the reliability the business needs!

In the end, business outcomes win. I have been in several versions of the same meeting where a product owner demands a certain number of ‘nines’ of availability, but when probed on what the business outcome of falling below this target is, they don’t really have an answer. This especially applies to non-serving systems and data pipelines, which can be behind in processing by several hours with no ill effects. However, it’s often seen as easy or necessary to demand to-the-minute SLAs from the outset, without reference to the ‘North Star’ of business needs.

As in disciplines such as security, there is a tradeoff. The downside when things go wrong is bad, but we don’t have carte blanche to do absolutely everything for extra reliability. It may be possible to neglect these areas entirely and risk the enormous downside of a security incident or outage, or to pursue reliability goals at the expense of the core mission.

Try to avoid implementing what Google does for its SRE practice. One of the authors of the original SRE Book, Niall Murphy, famously tore up a copy of it during a keynote at SRECon in 2022. But far from disavowing the book’s content, he was sounding a note of caution about copying what Google does, wholesale.

Absorbing Google’s approach effectively is not about copying as much of it as possible, it’s about discovering which elements make sense for you and what you’re building.

Since departing Google, I’ve worked with more than one company with its own “SRE Book Club”. This is great because there’s a lot of knowledge contained therein. However, I never saw rooms full of database admins poring page-by-page over database design publications and figuring out which features to include wholesale in their own setup, and this definitely applies to a modern SRE practice. The Google model of building everything yourself is just one way. It worked in the 2000s, but likely wouldn’t work today.

The availability of technology and products that didn’t exist when Google was building SRE makes the tradeoffs a lot more understandable and explicit, in hindsight. I go into more detail on this topic in my article, “A Short History”.

4. Four promises

At its core, reliability engineering is this:

17:00 • 15 days ago

Ben Werdmüller

Earth’s ‘vital signs’ show humanity’s future in balance, say climate experts

[Damian Carrington at The Guardian] Meanwhile, while we're all paying attention elsewhere: "More and more scientists are now looking into the possibility of societal collapse, said the report, which assessed 35 vital signs in 2023 and found that 25 were worse than ever recorded, including carbon dioxide levels and human population. This indicates a “critical and unpredictabl

[Damian Carrington at The Guardian]

Meanwhile, while we're all paying attention elsewhere:

"More and more scientists are now looking into the possibility of societal collapse, said the report, which assessed 35 vital signs in 2023 and found that 25 were worse than ever recorded, including carbon dioxide levels and human population. This indicates a “critical and unpredictable new phase of the climate crisis”, they said."

And:

"“Climate change has already displaced millions of people, with the potential to displace hundreds of millions or even billions,” he said. “That would likely lead to greater geopolitical instability, possibly even partial societal collapse.”"

And:

"The assessment concludes: “Only through decisive action can we safeguard the natural world, avert profound human suffering, and ensure that future generations inherit the livable world they deserve. The future of humanity hangs in the balance.”"

In a world where everything seems amplified and like it's from some kind of comic book version of reality, making it hard to engage with it as actual truth, it's imperative that we don't gloss over this. We all have to change, and we all have to demand change.

#Climate

[Link]

15:06 • 15 days ago

Michael Ruminer

Sometimes It’s All About The Chunking

As I continue my study and experimentation with coding up AI solutions, and especially, at the moment, with Retrieval-Augmented Generation (RAG), I decided to work with a post from the Metadocs blog titled “Simple Agentic RAG for Multi Vector stores with LangChain and LangGraph”. It seemed it would cover two areas of interest, agentic operations and RAG. Little did I expect to learn a valuable les

It started with the prior mentioned post that referred to a prior post that it built upon. Following a link to that pre-requisite post, “Create a Langchain app with multiple vector store the easy way”, it in turn referenced an even earlier post as a pre-requisite. So down that rabbit hole I went. The earlier post was “Deploy a RAG app with Langchain in minutes”. I read the earliest of the three posts and it was a very simple RAG application. I coded it up ensuring I understood each line along the way. Most notable was that it was splitting the text into chunks on “\n\n”. I hadn’t looked at the source document they were providing as a sample. It turns out it was a text file of a US State of the Union address with a twist. Each sentence was followed by “\n\n” (two line feed carriage returns); an interesting if unrealistic formatting. I have my own example PDF that I have been using for testing out various RAG implementations and how it handles a specific prompt, so I copied two paragraphs from the document that contains the context I was after and formatted it with a “\n\n” after each sentence. Normally, I have been extracting the text from the PDF as part of the code and then chunking usually with recursive character text splitting, but I didn’t want to change this program since I was going to build on it. When done the results actually returned what I was after, a list of the 10 principles of SSI.

For no apparent reason, I decided to edit the text file and format with a single “\n” after each paragraph. This should return the same results if I edited the text split to represent this. It didn’t. I was, and still am, perplexed by this. It makes no sense that a double line feed split should return any different results than a single line feed results under the circumstances. I plan to revisit this as I believe I must be wrong despite trying multiple times. What was most important in the process was that with the right chunking, as simple as it was, I got the desired results when prompted whereas in all my past experiences it had failed. There was a difference in that I was reading a text file versus using a PDF text extractor and that I was using only two paragraphs focused on the context I wanted versus 70 pages of an academic paper that is probably very hard to extract even if the proper area of context was two clean paragraphs of text within that PDF. The real lesson for me is showing how chunking is so important. I suspect the major differential in the success was in the chunk divisions. I won’t rule out the simpler PDF as a contributor.

Next, I plan to try a few things in this naive RAG implementation before I move on to the multi-vector store — a PDF creation of just the two paragraphs that contain the needed context and split on paragraphs (\n) to see how that comes out. I’ll try the two paragraphs in a PDF with text extracted and chunked using RecursiveCharacterTextSplitter and separators=[“\n\n”, “\n”, “. “, “ “, “”] and a chunk size of 1000 and two different settings for overlap (0, and 200) as well as with SentenceTransformerEmbeddingFunction and then the default OpenAi embedding function. Let’s see how all those combinations work.

To recap, though I can’t explain why I got wildly different results depending on the parsing character used with the tex file format changed, I can suspect the simple chunking by sentence made a lot of difference. The other likely impacting result was clean simple text versus PDF-extracted text. I plan to experiment more and will report back the results. A take away for me, even if it was not entirely or even primarily the impactful element, was how important the chunking is for good results.

10:14 • 15 days ago

@_Nat Zone

NIST SP800-63B-4 2pdのパスワード基準は2017年から踏襲されている〜新しいのはフィッシング耐性の要求

なんか、「定期的に変更するな」　NISTがパスワードポリシーのガイドラインについて第2版公開草案を発表みたいな記事だのツイートだのが出回って「パスワード定期変更が禁止」されたのが新しいことみたいな言説が出回っているので言っておくが、2017年6月発行のNIST SP800-63…

なんか、「定期的に変更するな」　NISTがパスワードポリシーのガイドラインについて第2版公開草案を発表みたいな記事だのツイートだのが出回って「パスワード定期変更が禁止」されたのが新しいことみたいな言説が出回っているので言っておくが、2017年6月発行のNIST SP800-63 第３版ですでにそうなってるからね。新しいことではない。パスワードの文字の構成に対する規制も同様。

NIST SP 800-63B-4 2pd のガイドラインによると、パスワードの要件は以下のものを含みます：

長さ: パスワードは少なくとも8文字以上で、推奨される最低長は15文字です。パスワードの最大長は少なくとも64文字にする必要があります。文字の受付: ASCII印刷文字、スペース文字、Unicode文字を受け入れるべきです。Unicodeのコードポイントはパスワードの長さを評価する際に1文字としてカウントします。構成ルール: 特定の文字種を混ぜるなどの構成ルールを課すべきではありません。禁止リスト: 一般的に使用される、予想される、または漏洩したパスワードとの比較を行う必要があります。比較はパスワード全体で行い、部分文字列では行いません。保存: ソルトとハッシュを使った適切なパスワードハッシング方式で保存する必要があります。変更と複雑さ: パスワード定期変更を要求してはいけません。しかし、危殆化が検出されたときには変更を要求しなければなりません。利便性: 強力なパスワード選択のためのガイダンスを提供し、パスワードが拒否された場合には理由を通知すべきです7。

これらのガイドラインは、セキュリティとユーザーの利便性のバランスを考慮しています。

また、中程度以上のリスクのシステムにおいては、パスワードのみというのはダメで、AAL2として以下をはじめとする要求事項があります。

認証装置の管理: 請求者が、加入者アカウントに紐づけられた1つ以上の認証装置を管理しているという高い信頼性が求められます。 2つの異なる認証要素の所有と管理の証明が、安全な認証プロトコルを通じて必要となります。

暗号化:承認済みの暗号化技術を使用しなければなりません。AAL2で使用される認証装置は、承認済みの暗号化認証装置でなければなりません。

リプレイ攻撃からの保護:AAL2で使用される認証装置は、少なくとも1つはリプレイ攻撃に耐えるものでなければなりません。
認証済みチャネル:請求者と検証者の間の通信は、1つ以上の認証済み保護チャネルを介して行われなければなりません。

生体認証の使用:生体認証要素が使用される場合、特定の性能要件を満たさなければなりません.

フィッシング耐性: 検証者は、少なくとも1つのフィッシング対策認証オプションを提供しなければなりません。連邦機関は、職員、請負業者、およびパートナーに対し、連邦情報システムへのアクセスにフィッシング対策認証を使用することを義務付けなければなりません。

新しいとして注目すべきはむしろこの「フィッシング耐性」。事実上、パスワードオンリーはもちろん、パスワード＋OTPも禁じたと見るのが正しいでしょう。

00:59 • 15 days ago

Monday, 07. October 2024

@_Nat Zone

今週の世界のデジタルID最新動向 (10月8日)

概要詳細 SOURCE: https://www.thinkdigitalpartners.com/news/2024/10/07/digital-identity-global-roundup-186/

概要アメリカ合衆国:
ノースカロライナ州の控訴裁判所が、州立大学の学生と職員が投票時に大学発行のデジタルIDを使用することを禁止しました。これは新しい写真付きID義務化に関連する決定です。イギリス:
IT管理サービスプロバイダーのXalientがアイデンティティセキュリティ企業SailPointと提携しました。この協力により、グローバル企業のネットワークセキュリティ強化、重要資産の保護、規制基準の遵守を支援します。ナイジェリア:
政府が外国人居住者に対し、国民識別番号(NIN)の取得を義務付ける法改正を提案しました。これは expatriates を正式な課税システムに組み込むことが目的の一つです。ヨーロッパ:
Signicatが「オープンバンキングハブ」を立ち上げました。これは銀行口座を通じて個人情報を安全に確認する方法を提供します。ドイツ:
政府がスマートフォン用の国家デジタルウォレットの開発計画を発表しました。これはEU全域でのデジタル身分証明を可能にするプロジェクトの一環です。インド:
Aadhaar（国民ID）データ漏洩の疑いを受け、政府が関連ウェブサイトの閉鎖に動きました。イギリス:
ロイズ・バンキング・グループが生体認証ID検証企業Yotiに対する1000万ポンドの融資を株式に転換しました。ブータン:
世界初の国家デジタルIDプログラムとしてGlobal Acceptance Network(GAN)に参加しました。リトアニア:
RegTech企業iDenfyが電子マネー機関FinCauseと提携し、KYCプロセスの自動化を目指します。マレーシア:
ブロックチェーンプラットフォームZetrixが、中国国民の公式IDを海外でデジタル認証できるアプリケーションを発表しました。ボスニア・ヘルツェゴビナ:
2025年半ばまでに市民向けデジタルIDウォレットの開発を計画しています。パプアニューギニア:
新しいデジタルID、ウォレット、オンライン政府プラットフォームの試験運用を開始しました。ヨルダン:
アメリカとカナダに住むヨルダン人向けにデジタルID有効化サービスを開始しました。詳細アメリカ合衆国 (ノースカロライナ州):
ノースカロライナ州控訴裁判所が、州立大学ノースカロライナ大学チャペルヒル校の学生と職員による大学発行のデジタルIDの投票時使用を禁止しました。これは新しい写真付きID義務化法に関連する決定です。州選挙管理委員会（民主党多数）が大学のモバイルIDを承認したのに対し、共和党全国委員会と州共和党が訴訟を起こし、法律では物理的なIDカードのみが認められると主張しました。イギリス (Xalientの提携):
IT管理サービスプロバイダーのXalientがアイデンティティセキュリティ企業SailPointと提携しました。この協力により、XalientのアイデンティティコンサルティングサービスとSailPointの統合アイデンティティセキュリティプラットフォームを組み合わせ、グローバル企業のネットワークセキュリティ強化、重要資産の保護、規制基準の遵守を支援します。特にゼロトラストサービスを提供し、SailPointのIdentity Security Cloudを活用して、統合されたカスタマイズ可能なアイデンティティセキュリティソリューションを提供します。ナイジェリア:
連邦執行評議会（FEC）が、国内に居住する外国人に対し国民識別番号（NIN）の取得を義務付ける法改正を提案しました。現在、外国人居住者はこの生体認証デジタルIDを取得できません。この提案の主な目的の一つは、expatriates を正式な課税システムに組み込むことです。また、ナイジェリアで働き収入を得ている外国人を特定し課税することを目指す経済安定化法案も同時に提案されています。ヨーロッパ (Signicatのサービス):
Signicatが「オープンバンキングハブ」を立ち上げました。これは銀行口座を通じて個人情報を安全かつ同意ベースで確認する方法を提供します。消費者により広範な本人確認オプションを提供すると同時に、企業に対しては銀行口座の所有権、支払能力、口座情報の確認においてより高いセキュリティを提供します。このプロセスでは、個人が銀行口座を所有していることとその正当性を確認し、口座保有者の名前や銀行口座などのデータの完全性を提供します。ドイツ:
ドイツ政府がスマートフォン用の国家デジタルウォレット開発計画を発表しました。これはEU全域でのデジタル身分証明を可能にする欧州デジタルアイデンティティウォレット（EUDI Wallet）プロジェクトの一環です。このウォレットにより、求人応募、オンラインバンキング、行政手続きなど様々な活動において、スマートフォンを通じて安全かつ容易にデジタル本人確認が可能になります。また、身分証明データや公的文書をデジタルで保存し、電子署名を行うこともできるようになります。インド:
Aadhaar（国民ID）データの疑わしい漏洩を受けて、インド政府が関連ウェブサイトの閉鎖に乗り出しました。Aadhaar法では、個人情報を公開したり、どのようなドメインでも共有したりすることが禁止されています。インド固有識別番号庁（UIDAI）が問題のサイトについて警察に通報する責任を負っています。IT省は、センシティブな個人情報（AadhaarカードやPAN詳細を含む）を公開しているウェブサイトがあることを認識し、この問題を深刻に受け止めていると述べました。政府はサイバーセキュリティ実践の安全性に最高の優先順位を置いていると強調しています。イギリス（Lloyd’s Banking Groupの投資）:
Lloyd’s Banking Groupが生体認証ID検証企業Yotiに対する1000万ポンド（約1900万ドル）の融資を株式に転換しました。Yotiの最高経営責任者Robin Tombsは、顧客数で英国最大の銀行グループを株主として獲得したことで、英国市場における再利用可能なデジタルIDの採用を加速させるのに役立つと確信していると述べています。Yotiは生体認証ID検証、年齢確認、年齢推定サービスを提供しています。ブータン:
ブータン王国が、国家デジタルIDプログラムとしてGlobal Acceptance Network (GAN)に参加しました1。国家IDプログラムとしてGANに参加するのはブータンが初めての事例になります。
GANは検証可能なデータを利用可能、信頼可能、相互運用可能にすることを支援する分野横断的なイニシアチブです。2023年9月17日に30以上の創設メンバー組織とともにソフトローンチされました。ブータンはGANにエコシステムメンバーとして参加することで、自国のデジタルIDを他のデジタルIDシステムや信頼エコシステムと統合する位置に立ちました。これにより、ブータン国民が世界中で自分のデジタル資格情報を使用できるようになる可能性があります。リトアニア:
リトアニアを拠点とするRegTech企業iDenfyが、リトアニア銀行の規制を受ける新しく立ち上げられた電子マネー機関（EMI）FinCauseと提携しました。この協力を通じて、iDenfyはFinCauseのKnowYourCustomer（KYC）プロセスの自動化を目指しています。欧州経済地域（EEA）のB2B顧客やEEAで事業を展開するアジア企業向けに、シンプルな4ステップの認証プロセスを通じてより多くのコンバージョンを確保することを目的としています。マレーシア:
レイヤー1のパブリックブロックチェーンプラットフォームZetrixが、中国国民の公式IDを海外でデジタル認証・検証できるアプリケーションを発表しました2。ZCertサービスを通じて、Zetrixは中国国外の企業が中国国民から提示されたデジタルIDに含まれる情報の信憑性を確認する電子Know Your Customer（eKYC）プロセスを簡素化・促進します。このサービスは、Zetrixと中国の National public blockchain Xinghuo BIFとの統合によって可能になりました。中国国民は自分のデジタルIDをXinghuo BIFに公開することを選択でき、その後、海外の検証機関がZetrixのZCertサービスを通じてそのデータにアクセスできるようになります。ボスニア・ヘルツェゴビナ:
ボスニア・ヘルツェゴビナの市民向けデジタルIDウォレットが開発中で、2025年半ばまでの導入が予定されています。このプロジェクトはクロアチアのテクノロジースタートアップIdentity Consortiumが主導し、’EU4DigitalSME‘イニシアチブの一環として行われています。
目的は、安全な電子識別、認証、デジタル文書署名を通じて識別プロセスを近代化することです。EU4DigitalSMEは、ボスニア・ヘルツェゴビナの中小企業（SME）のデジタル変革を支援することに焦点を当てています。欧州連合の資金提供とドイツ政府の支援を受けて、このプログラムはデジタルインフラとサービスを近代化することで競争力を高め、より広範な欧州デジタル経済への統合を目指しています。
デジタルIDウォレットはIdentyumブランドの下でAdverta Proと協力して開発され、バックエンドシステムとモバイルアプリケーションの両方を提供します。このプラットフォームにより、ボスニアの市民はデジタルIDを使用して公共および民間のサービスにアクセスできるようになり、欧州の規制に準拠しつつ、ボスニアのEUデジタルエコシステムへの統合を促進します。パプアニューギニア:
パプアニューギニアが新しいデジタルID、ウォレット、オンライン政府プラットフォームを立ち上げ、今後数週間で10,000人のユーザーを対象にパイロット運用を開始する予定です3。
SevisPassデジタルID、SevisPortal、SevisWalletの試験運用が今週、限定的に開始されました。SevisPassは国民IDプログラムと既存の物理的IDから派生し、パプアニューギニアの住民はこれを使用してSevisPortalを通じて政府サービスにアクセスできます。SevisWalletアプリはSevisPassを保存します。
すでに7つの政府サービスがSevisPortalを通じて利用可能となっており、今後12〜18ヶ月の間に、国家経済を後押しするための「公共および民間サービスのデジタルエコシステム」が構築される予定です。国の情報通信技術省（DICT）の公共サービスICT運営委員会のSteven Matainaho委員長がこのように述べています。ヨルダン:
ヨルダンが、アメリカとカナダに住むヨルダン人向けにデジタルID有効化サービスを開始しました。
このサービスはSanadアプリを通じて利用可能で、IrisGuardとVFS Globalとのパートナーシップにより確立されました。海外に住むヨルダン人の政府サービスへのアクセスを向上させることを目的としています。
外務・在外居住者省が土曜日に発表したプレスリリースによると、この新サービスにより、海外に居住するヨルダン国民がVFS Globalのオフィスを通じてデジタルIDを有効化できるようになります。
このサービスは、米国とカナダに居住する約20万人のヨルダン人に恩恵をもたらすと期待されています。

SOURCE: https://www.thinkdigitalpartners.com/news/2024/10/07/digital-identity-global-roundup-186/

23:30 • 15 days ago

IdM Laboratory

SIDI Hub - ベルリンレポートを読む（４）

こんにちは、富士榮です。引き続きSIDI Hubベルリンイベントのレポートを見ていきます。今回はユースケース分析です。相互運用性を目指しましょう、といっても具体的なユースケースを見つけてボトムアップで考えていかないと進まないので、このワークストリームでは有用なユースケースを各地域ごとに発見して分析して行きます。 Champion Use Cases: Process and Progress to Date - Elizabeth The Champion Use Case workstream is in the process of identifying champion use cases and then prioritizing using an agreed framework. In Berlin, the Champio

こんにちは、富士榮です。
引き続きSIDI Hubベルリンイベントのレポートを見ていきます。

今回はユースケース分析です。相互運用性を目指しましょう、といっても具体的なユースケースを見つけてボトムアップで考えていかないと進まないので、このワークストリームでは有用なユースケースを各地域ごとに発見して分析して行きます。
Champion Use Cases: Process and Progress to Date - Elizabeth

The Champion Use Case workstream is in the process of identifying champion use cases and then prioritizing using an agreed framework. In Berlin, the Champion Use Cases workstream sought to do three things throughout the day: Ground Minimum Requirements conversations in salient use cases Add more use cases and more texture to the data already gathered Gain input on prioritization criteria

チャンピオン・ユースケース・ワークストリームは、チャンピオンのユースケースを特定し、合意されたフレームワークを用いて優先順位を決定しているところである。ベルリンでは、チャンピオン・ユースケース・ワークストリームは、一日を通して3つのことを行おうとした：
重要なユースケースにおける最低要件の会話の基礎固めより多くのユースケースを追加し、すでに収集されているデータにさらに質感を加える。優先順位付けの基準について意見を得る

これまでパリ、ケープタウン、ベルリン、ワシントンDC、そして東京の企画を通して見てきていますが、やはりユースケースに関する関心度、優先順位の置き方は地域によってかなり異なるイメージです。その意味で各地域を回りながら状況をヒアリングして回る、というSIDI Hubのアプローチは理にかなっていると思います。（どうしてもUSとEUだけで話が決まっていく傾向がある世界ですし）

つまり、せっかくなのでアジアからも意見を出していかないとダメですよ、ってことです。

Early in the day, we reviewed the inputs from other sources and past SIDI Hub events:
Paris Summit and write specific user stories W3C credentials working group EU Wallet use cases EU + US TTP bilateral analysis SIDI Hub Cape Town New input from SIDI Berlin
Wishing to spend the morning on technical requirements, SIDI Hub Berlin grounded further discussions in two use cases: Refugees and Opening a Bank Account. In this early session, presenters reviewed the outputs from the deep dive conducted at SIDI Hub Cape Town.

その日の早い段階で、他の情報源や過去のSIDI Hubイベントからのインプットを見直した：
パリサミットと具体的なユーザーストーリーの作成 W3Cクレデンシャル・ワーキンググループ EU ウォレットのユースケース EUと米国のTTP二国間分析 SIDIハブ・ケープタウン SIDI ベルリンからの新しいインプット
SIDIハブ・ベルリンは、午前中を技術的な要件に費やすことを希望し、2つのユースケースでさらなる議論を行った。「難民」と「銀行口座開設」である。この早いセッションで、発表者はSIDI Hub Cape Townで行われたディープダイブからのアウトプットをレビューした。

ベルリンでは先に書いたようにこれまでの取り組みについて確認し、その後、さらなるテーマの深掘りをしていっています。日本にいると難民のユースケースはリアリティがない人も多いと思いますが、これまで移民を受け入れてきたドイツや欧州諸国においては重要なキーワードなんだと思います。

こちら、難民のユースケースですね。

特徴として、自国の法的な身元証明が受けられない状況にあるので、UNHCRが発行する証明書を利用できるか？というのが大きなポイントになります。これはワシントンDCでも話があり、先日のクィックレビューでも書きましたが、どうしても出生からの流れを含め身元を証明することが困難であり、Identity Verificationをする際の照合先がないところから身分を付与していくことになります。その付与プロセス自体がどこまでの保証レベル（IAL/Identity Assurance Level）を持つのか？テロリストが混入している可能性や身元ロンダリングに悪用されていないかを踏まえて、どこまでVerifierが受け入れることができるのか？人権や人道支援の文脈を含めてどのように判断していくのかは非常に難しい話です。ただ、世界的に助けを求めている人々の数がますます増えている昨今、目を背けるべき案件ではありませんね。

こちらは銀行口座の開設のユースケースです。

こちらも移民のケースにも少々関わってきますが、国境を超えて別の国で銀行口座を開設するのは非常に難しい状態です。KYCが難しいのはもちろん、CDDについても元となる実績情報などが取得しにくい状態にあるので、どうしてもリスクベースで考えるとリジェクトもしくはネガティブな判断が下されがちです。こちらもAML/CFTの観点も踏まえて良い落とし所を作っていく必要がありそうです。

今回はここまでです。

この後、ギャップ分析が続きます。

22:30 • 15 days ago

FACILELOGIN

The lessons learned in building a secure, scalable, and reliable identity platform @ DevRev

Image credits: https://www.networkrailmediacentre.co.uk/resources/ap-0409 At DevRev, we are building an API-first dev-centric platform that gives you one unified view of your customers and products by connecting your data, teams, and users to our AgentOS platform and app experiences. This blog post shares some insights into how we built (and keep evolving) a secure, scalable, and reliable identity

Image credits: https://www.networkrailmediacentre.co.uk/resources/ap-0409

At DevRev, we are building an API-first dev-centric platform that gives you one unified view of your customers and products by connecting your data, teams, and users to our AgentOS platform and app experiences. This blog post shares some insights into how we built (and keep evolving) a secure, scalable, and reliable identity platform at DevRev. The decisions we made and the lessons we learned throughout this journey, we believe, would be equally applicable to most SaaS companies.

I would like to acknowledge and thank Akanksha Deswal, and the DevRev identity team, who were part of this journey, and instrumental in figuring out what works best for us.

An identity platform serves as a secure bridge between users and the services they access — whether apps, APIs, or microservices — delivering a seamless and trustworthy way to manage, verify, and protect user identities in increasingly interconnected environments. While the expectations for an identity platform can vary depending on specific use cases, at DevRev, we primarily focused on the following core areas.

The expectations for an identity platform can vary depending on specific use cases; and at DevRev, we primarily focused on the above core areas.

The decision to buy or build components of an identity platform depends on several factors, such as business requirements (both functional and non-functional), cost at scale, available resources, in-house expertise, and more. We wanted to maintain complete control over our object model (organizations, users, groups, etc.) while delegating more generic identity capabilities to an external identity service (identity provider). Having control over the foundational object model is essential for innovation, and allowing a third party to dictate its structure or limitations can stifle growth and introduce significant risks.

A few years ago, I enjoyed reading Ask Your Developer: How to Harness the Power of Software Developers and Win in the 21st Century by Jeff Lawson. In the book, Jeff reframes the classic ‘build vs. buy’ debate into a more urgent choice: ‘build vs. die.’ As every company evolves into a software company, the true competitive advantage lies in the software they build. When software becomes the critical interface between our services and our customers, not building it to meet the exact needs means risking irrelevance — or worse, failure.

Building doesn’t always mean starting from scratch. We shouldn’t reinvent the wheel if existing solutions meet our needs. Instead, focus on developing what’s core to our business — the components that give us a competitive edge. Everything else, the foundational components that enable us to build our vision, can be sourced from the digital supply chain. I touched on this concept in the blog post, The Next TCP/IP Moment in Identity.

We had no interest in building identity federation standards like OpenID Connect or SAML, nor did we want to develop authentication protocols in-house. These are commodity features supported by almost all identity providers. However, we wanted to ensure the external identity service handling federation and authentication remained a ‘thin’ layer, separate from our business logic. This approach minimizes vendor lock-in.

Vendor lock-in isn’t just about making it easy to switch providers — though that’s certainly part of it. More importantly, it’s about retaining the freedom to innovate independently, which was our primary focus. Within the DevRev platform, Janus serves as the identity service, managing the identity object model and interacting with the external identity provider during organization and user onboarding flows. For the rest of the DevRev platform components, the external identity provider remains a black box, only interacting with Janus.

Choosing the right identity provider requires a thorough and thoughtful evaluation. Our objective wasn’t to find an identity provider that perfectly matched the DevRev object model but rather to focus on its core constructs, their relationships, and the integration effort with the DevRev platform.

In the DevRev platform, each organization can choose its preferred authentication method. For example, one organization might opt for Google login, while another may use their identity provider via SAML. The external identity provider’s object model must support this flexible relationship between organizations and their connections. For instance, the connection to Google (social) is made using OpenID Connect, and ideally, we should be able to reuse that Google connection (or any other social connection) across multiple organizations.

This means the identity provider’s object model should allow connections to be defined independently of a specific organization and provide the ability to associate them with any org. During our evaluation, we encountered one identity provider that could only define a connection within the scope of an individual organization (or tenant), which led us to reject it.

Apart from key functional requirements for authentication and identity federation, we also worried about the scalability, availability, reliability, testability, compliance, and interoperability of the identity provider, and of course the cost at scale. We picked Auth0, mostly because of the lightweight organization model it had and it could meet our requirement to scale the number of organizations to 1 million. Most of the prominent identity providers do support almost all the key identity federation and authentication needs. Only in non-functional requirements and object relationships, you would find the differentiators.

In our hunt for an external identity provider, we mostly looked for a SaaS product. Going for a SaaS product gave us the freedom to focus on our core business problem, and delegate anything related to the external identity provider to the SaaS vendor. A key reason one would pick an on-prem product over a SaaS product is the freedom to do heavy customizations. Building heavy customization in the identity provider is something we wanted to avoid from day 1.

Then again, I am not completely ruling out identity provider on-prem deployments. It would depend on the industry you are in and your business-specific needs. For example, highly regulated industries would pick an on-prem identity provider to keep all their customer data within their boundaries. I led the development of the open-source WSO2 Identity Server for almost 14 years, before joining DevRev, and saw many large-scale successful on-prem deployments.

When picking a SaaS identity provider, non-functional requirements play a key role. We have little control over the non-functional requirements, and even from the SaaS vendor’s point of view, they will find it hard to accommodate change requests related to non-functional requirements; as it could require core changes in their product. For example, Auth0 offers an availability of four 9’s. When you create a tenant in Auth0, it’s bound to one region, and internally they create a replica of that tenant in another availability zone. However, when the primary Auth0 region is down, we do not have the control to switch to the corresponding availability zone. It’s the Auth0 team that decides when to switch based on the severity of the incident. Auth0 also offers a private cloud deployment, but, its a costly option.

Another limitation we found in Auth0 (and of course in most of the other prominent SaaS identity providers as well) replication of data between regions. For example, when you pin your tenant to the Auth0 US-1 region, all the API calls from any region globally must be routed to the US-1 region. Due to this limitation, we’ve witnessed increased latency levels in login API calls (OpenID Connect) for the users connecting from Asia. This hurts the app load time and at the moment we are in the process of building a solution in-house to minimize the impact.

The DevRev platform is built with 250+ microservices running on Kubernetes. Janus is one of them. Each service has its CI/CD pipeline, which deploys each commit to main, to Prod after running end-to-end tests in Dev and QA. We’ve integrated Auth0 into the Janus integration tests. We have a dedicated Auth0 tenant for that, and in the CI/CD pipeline, we connect to Auth0 via the API to create organizations and users; and then associate users with organizations. Once the tests are complete, we run a clean-up task to remove the resources created in Auth0, as it has certain resource restrictions.

There was a time when the clean-up task failed to run and unfortunately went unnoticed. This filled up Auth0 resources and soon reached its limits. Auth0 has a limit on the size of the app metadata that can be associated with a user. We used the app metadata to associate the DevRev-generated organization identifier with an Auth0 user. Once we hit the limit, the Auth0 system became unstable, and none of the data in the test tenant could be accessed either via the UI or the API. This became a blocker for the prod promotion of the Janus service and all other services that relied on Janus for testing. As a quick workaround (while the Auth0 team was working on recovering the tenant), we had to spin up a new Auth0 tenant and use that in our tests. Usually, these limits enforced by Auth0 are way beyond our production needs, and we encountered this issue due to the failed clean-up task from our end.

Auth0 has a rate limit on its management API. During our tests, we used this management API to create organizations and users. Due to the nature of the tests, we started to hit the rate limits (even with proper retries) which resulted in some flaky tests. As a solution, to this and the resource limit issue we discussed before, we ended up mocking Auth0 in our integration tests, rather than directly hitting the Auth0 test tenant.

We follow GitOps at DevRev to automate and manage infrastructure and application deployments. Git is the single source of truth. This helps us to take any change via a review process and audit all changes. We followed the same model with Auth0, where appropriate. For example, we use Auth0 actions to inject the DevRev-generated org/user identifiers into the tokens it produces. The actions are Javascript code and we keep it in a Git repo and deploy to the Auth0 Prod tenant via Dev and QA. This helped us avoid any manual edits from the Auth0 management console, and keep all three environments consistent.

Currently, not all operations on Auth0 are fully managed through GitOps. For example, we still update certain settings, such as token lifetimes for applications, directly through the Auth0 management console. Although these are infrequent tasks, it’s essential to have a complete audit trail in production for every operation and change made. While Auth0 provides logs, they are only retained for a month. To ensure long-term visibility, we push Auth0 logs to Datadog and set up Slack alerts for specific cases. Additionally, we’re exploring the use of the Auth0 Terraform provider to automate and manage all Auth0 operations via GitOps.

Passwords are painful, not just from the user’s point of view but also from the product side. If an app has to support password-based login, it must also build the supporting infrastructure to recover passwords, rotate them periodically, implement strong second-factor authentication, and so on. From day one, we made a design decision to avoid passwords.

We use Auth0 as an identity bridge. The contract between the DevRev app and Auth0 is defined by OpenID Connect. Auth0 helps us connect multiple other identity providers via SAML and OpenID Connect to authenticate users. These identity providers are either the social identity providers or the enterprise identity providers owned by DevRev customers. Following the identity bridge pattern helped avoid storing user passwords, and also introducing a new identity provider or an authentication method requires no changes to the DevRev app. It can be done transparently.

This graph shows the growth of unique monthly active Dev users by connection type over the last 12 months. The growth in active users is healthy, with enterprise connections starting to dominate.

The identity bridge pattern effectively addresses the common challenges posed by the spaghetti-identity and identity-silos antipatterns. Spaghetti identity occurs when a SaaS application tries to establish direct, point-to-point integrations with multiple identity providers, leading to complex, tangled connections. On the other hand, identity silos arise when an application is limited to a single federation or authentication protocol, making it difficult and costly to introduce new protocols in the future. By adopting the identity bridge pattern, organizations can simplify integration and enable flexibility in supporting multiple protocols without the downsides of these antipatterns.

It’s also worth mentioning how we leveraged the identity bridge pattern to overcome a limitation in Auth0. Auth0 supports passwordless authentication with email OTP; however, it cannot be used on its own with Auth0 organizations — only as a second factor. This limitation likely arises from Auth0’s architecture, as the organizations model was introduced later. To overcome this and use email OTP as a passwordless method, we spun up another Auth0 instance without organizations and then federated to it from the primary Auth0 tenant (which has organizations enabled). While this approach works, it is far from ideal. We anticipate switching to native passwordless email OTP within our primary Auth0 tenant as soon as that feature becomes available.

Auth0 provides the latency numbers it introduces during the login operations; however, that does not reflect the latency the users experience at the edge. To address this concern, and also to have better visibility into the OpenID Connect login flow, we started routing all login calls to Auth0 via our CDN provider, which is Fastly. On the Fastly Compute@Edge platform, we run an edge gateway, which pushes all the API stats to Google BigQuery for further analysis. There we would know the latency at the edge corresponding to all the login operations. These latency numbers are very much closer to what the users experience. The following graph shows the p90 latency numbers in milliseconds for the last 31 days against a set of selected regions, for the Auth0 token endpoint.

p90 latency numbers in milliseconds for the last 31 days against a set of selected regions, for the Auth0 token endpoint

All DevRev applications utilize the OpenID Connect authorization code flow. The token endpoint, highlighted in the graph above, is called once during the initial app load and periodically thereafter for token refreshes. While the periodic refreshes happen asynchronously and do not affect the user experience, the latency of the initial token request during page load is more critical.

Two main factors contribute to this initial latency:

Geographic latency — All users, regardless of location, connect to a centralized Auth0 instance in the US-East region. Web-worker-based login flow — The DevRev web app, a single-page application (SPA), leverages this flow, which adds complexity and impacts performance during authentication.

We opted for the web-worker-based login flow to enhance security by storing user tokens in browser memory more securely. However, web worker memory is scoped to a single browser tab, which means that even if a user has an active Auth0 session, opening a new tab requires calling both the authorize and token endpoints during the initial page load. Upon analyzing the stats, we found that 91.1% of the calls to the Auth0 authorize endpoint were from users who were already authenticated. To address this, we developed an edge-based solution targeting these 91.1% of users, aiming to reduce the latency at the edge to almost zero. As of this writing, this solution is being tested in our development and QA environments and will soon be available in production.

91.1% of the calls to the Auth0 authorize endpoint were from users who were already authenticated (page reloads). The users once logged in to the app, keep the login session. This graph shows the average number of active vs silent logins by weekday for the last 90 days. The edge-based solution we are working on will reduce the latency at the edge for the login calls for already authenticated users by almost 100%.

In addition to publishing stats related to Auth0 API performance from the edge to Google BigQuery, we also send logs to Datadog. This allows us to trigger alerts in case of Auth0 failures and detect downtimes early. Moreover, this helps us identify which users and organizations are impacted during Auth0 outages — events that have occurred a few times in the past couple of years.

The contract between all the DevRev apps and the identity provider is based on OpenID Connect. However, we didn’t find the ID token to be particularly useful in the case of an SPA. An SPA is simply a “dumb” app or a public client, which means it cannot make any trustworthy decisions on its own. The access token is the key! (If you’re interested in learning more about the difference between the ID token and the access token, please check out this video.)

We use self-contained access tokens (JWTs), which include the user context with every request. These tokens have been widely adopted in the industry and were formalized as an IETF standard in 2021 (RFC 9068). While Auth0 has yet to support RFC 9068, this has had no impact on our use cases. In fact, for an SPA, using reference tokens (the other alternative to self-contained access tokens — also known as opaque access tokens) is not a viable option. Only the issuer of an opaque token can validate it, and it’s impractical for the gateway intercepting API requests to communicate with the issuer for token validation, for each request.

When using self-contained tokens, we also need to implement support for revocation. Unlike self-contained tokens, when a reference token is revoked, that request goes to the token issuer. The consumer of the token (such as a gateway) would be aware of the revocation because it communicates with the token issuer each time it validates a token. However, there’s no point in communicating with the issuer to revoke a self-contained token unless there’s a reliable way to propagate the revocation decision to all dependent systems. To address this, we introduced the auth-tokens.delete endpoint, allowing us to revoke access tokens upon user logout. We also maintain a cache of the revoked token IDs, with a TTL based on the token’s expiration.

Returning to the topic of the ID token, the DevRev app relies on the dev-users.self API to identify the user. Additionally, another reason we can't rely on the ID token for user identification is that Auth0 cannot embed the necessary user information into the ID token, as we store only a minimal set of user data in Auth0.

Most of the support queries we receive from our customers related to login are due to some kind of an issue with the OTP delivery. Even though the number of such queries is quite low (compared to the success cases), when it happens, it’s painful to the users and hurts the business. Most of the DevRev customers at the stage of evaluation use email OTP-based login and then later switch to login with their identity provider, with SAML or OpenID Connect. Apart from our direct customers, we also offer a support portal for our customers’ customers (who we call Rev users), which also offers login with email OTP.

Most email OTP-related issues arise from one of the following causes:

The user’s email is added to the email provider’s global suppression list. The OTP email is marked as spam. The OTP email is rejected by the user’s mailbox due to internal domain restrictions, or other similar factors.

Email service providers (ESPs) maintain global suppression lists to prevent sending emails to addresses that have previously bounced (due to invalid addresses) or opted out of receiving emails from certain senders. If a user’s email address is on such a list, the OTP email won’t be delivered. This can happen if the user’s email previously encountered delivery issues, such as soft or hard bounces.

Initially, we used SendGrid as our ESP for sending OTP emails as well as marketing emails. Although we used two separate SendGrid accounts for these purposes, SendGrid maintains a global suppression list shared across accounts when emails originate from the same domain — devrev.ai in our case. This meant that if a user unsubscribed from marketing emails, they were automatically added to the global suppression list, preventing them from receiving OTPs. In some cases, users didn’t even unsubscribe themselves; certain plugins handled it automatically.

This behavior was undesirable, as it significantly impacted OTP delivery. To address the issue, we switched to AWS SES for sending OTP emails, which reduced global suppression list-related problems by nearly 90%.

We once encountered a particularly interesting issue related to email OTPs, and while it wasn’t a delivery problem, it’s worth mentioning. The user received the OTP email, but every time they tried entering the OTP, it failed. What made this case even more intriguing was that we discovered the OTP had already been attempted from a different IP address just before the user tried it. This explained why the OTP kept failing — the user was attempting to use an OTP that had already been used.

After further troubleshooting, we discovered the root cause. Along with the OTP in the email, we had also embedded an OTP link that allowed users to log in directly by clicking it. Although this link was commented out in the HTML template and not visible to the user, the organization this user belonged to had strict security protocols that automatically scanned and clicked on links embedded in incoming emails. This security bot didn’t distinguish between commented-out links and visible ones, so it clicked the hidden link, sending a request to the Auth0 server and invalidating the OTP before the user could use it. The issue was resolved by completely removing the commented-out OTP link from the email template.

Many email providers have spam filters that automatically classify certain emails as spam based on various criteria, such as the content of the email, sending frequency, or the sender’s reputation. If the OTP email is flagged as spam, it ends up in the user’s spam or junk folder, leading to the user not receiving it in their primary inbox. This could occur if the email provider deems the sender domain as suspicious or if the email contains certain keywords that trigger spam filters. After moving from SendGrid to AWS SES and then carefully designing the email template, we could get the number of spam emails almost down to 0.

Every organization and user in the DevRev platform is assigned a unique, immutable, and persistent pseudonym known as a DON (DevRev Object Name). Every resource in the platform has its own DON. The format of a DON is: don:<service>:<partition>:(<type>/<id>)+, inspired by Amazon Resource Names (ARN).

For example, don:identity:dvrv-us-1:devo/xyz is the DON of an organization. Here, identity refers to the service name, and dvrv-us-1 is the partition identifier, which indicates the organization’s location. For example, an organization in the us-east-1 region has dvrv-us-1 as its partition ID, while one in ap-south-1 has dvrv-in-1.

The type field of the DON defines the type of the object it is attached to. Every key entity within the DevRev platform is associated with an object, and all objects share a common ancestor known as the Atom object. For example, a devo object type represents a Dev organization, while devu represents a Dev user. A Dev organization is a company or team that uses the DevRev platform, and its members are referred to as Dev users. Customers of a Dev organization are called Rev users, who belong to Rev organizations (also known as workspaces). Both Rev organizations and Rev users operate within the scope of a Dev organization. For instance, the DON of a Rev organization would be: don:identity:dvrv-us-1:devo/xyz:revo/pqr.

All identity objects in the DevRev platform follow a predefined schema, ensuring consistency and standardization across all organizations. This common schema facilitates smooth integration and uniformity across the platform. However, recognizing that different organizations may have unique requirements, we also provide the flexibility to extend the Dev user and Rev user objects with custom fields specific to a given Dev organization.

Customizability is essential for organizations that need to tailor their identity models to accommodate unique workflows, industry requirements, or specific customer data needs. For instance, a Dev organization might need to capture additional fields such as department codes, job roles, or region-specific preferences that go beyond the standard schema. Moreover, when importing Rev user objects from external systems like Salesforce or Zendesk, these objects may come with their own set of attributes. The ability to extend and adapt the schema allows organizations to seamlessly integrate and migrate from these external systems to the DevRev platform.

All personally identifiable information (PII) of DevRev users is stored exclusively in Janus, making it the single source of truth for PII. Other services in the platform reference users through their DONs. For example, when a client calls the works.get API to retrieve a work item, the request goes through the DevRev gateway, which makes a gRPC call to the corresponding service. The service returns the work item data, including the DONs of the users who created and are assigned to the item.

The DevRev platform is built with 250+ microservices running on Kubernetes. Fastly Compute@Edge serves as the entry point at the edge for all DevRev services. At the origin, the DevRev API gateway intercepts all incoming traffic. Once the gateway verifies the JWT accompanying an API request, it dispatches the request to the appropriate service.

Since raw user DONs are not user-friendly, the gateway resolves them into user details, such as display name, email, and full name, by querying Janus (the API composition pattern). These user summaries are cached in Redis to avoid repetitive calls to Janus (from the gateway), ensuring faster response times for future requests.

A DON attached to a user or an organization is an immutable identifier. It’s not just immutable, but also resolvable by its own — or embeds information in it. For example, we use the partition ID embedded into the organization DON to route requests to the corresponding region of the organization, in our multi-regional routing implementation built at the edge. Designing a system based on immutable identifiers needs to be done with care because the cost of a bad design decision is high. For example, we embed the Dev organization ID into the user DON, which means we cannot move this user to another Dev organization or a user cannot be part of multiple Dev organizations. This is a conscious decision we made to scope all the resources in the DevRev platform under a Dev organization.

However, we made a mistake in our Rev user implementation. The Rev user DON embeds both the Dev organization ID and the corresponding Rev organization ID. This means, a Rev user cannot switch Rev organizations, or cannot be part of multiple Rev organizations. This was not a requirement initially, but as we kept learning from our customers, we figured that it was a too restrictive constraint to limit a Rev user to one Rev organization. We made the Rev user scoped at the Dev organization level and removed the Rev organization ID from the DON. This was a painful change.

The Dev organization ID (don:identity:dvrv-us-1:devo/xyz), which is embedded into a DON is not incremental. An incremental ID can expose sensitive information, such as the total number of organizations created over time. Malicious actors could exploit this to infer patterns about system growth or attempt ID enumeration attacks to access or manipulate unauthorized resources. Non-incremental IDs make it harder to guess or predict other organization IDs, improving security and reducing the risk of ID enumeration.

The randomly generated IDs provide global uniqueness, meaning IDs can be safely generated across distributed systems without the risk of collision. This allows for a more flexible, decentralized architecture where different systems or regions can generate IDs without coordinating with a central system.

We initially used a 48-bit random value as the Dev organization ID. Once base62 encoded, it becomes a string with either 8 or 9 characters. When we moved to the multi-regional deployment, we wanted to ensure the Dev organization ID is unique across all regions by adding a regional prefix, for example, “1” for us-east1. However, since the previously generated Dev organization IDs were already 8 or 9 characters long, this introduced a challenge. A new region could potentially generate an ID with the prefix that results in 9 characters, which could match an already generated ID from a different region, before the introduction of the regional prefix. To address this, we first ensured the base62-encoded 48-bit value was always 9 characters by adding padding, and then introduced a regional prefix on top of that, effectively making the Dev organization ID a fixed-length random string of 10 characters.

What does deleting a Dev organization mean? A Dev organization has many other resources linked to it within and outside the identity space (or Janus). Dev users, Rev users, workspaces, accounts, service accounts, system users, and preferences are all identity-related objects, while conversations, articles, tickets, and issues, for example, are objects outside the identity space.

This is a common problem in a distributed system. What we need is a cascade delete. Implementing a cascade delete in a distributed system requires careful consideration of consistency, availability, fault tolerance, and scalability. Unlike a traditional monolithic system, where database constraints and relationships can directly enforce cascade deletes, distributed systems involve multiple services, databases, and possibly different regions, making the process more complex.

When a Dev organization is deleted, we mark it as soft-deleted. A soft delete marks a Dev organization as deleted without actually removing it from the database. This is, in general, useful to avoid cascading failures in case other services depend on the deleted entity. When a Dev organization is marked as deleted, it triggers a CDC (change data capture) event, and we have a dedicated service (garbage collection service), listening to these events and taking actions based on the type of the object being deleted and the dependent object types.

Once the garbage collection service picks the event to soft delete an organization, it spins up a temporal workflow, which will trigger the cascade delete of the dependent objects after a retention period. As you have figured out already this is a recursive process, and we only need to specify the first level of dependencies.

In general, there are two common saga patterns for cascade delete implementation, orchestration-based saga and choreography-based saga. In the orchestration-based saga pattern, a central coordinator service orchestrates the cascade delete by calling each service that manages dependent objects in a defined sequence. It ensures the cascade completes across all services or triggers compensating actions (e.g., undo the deletion) if a step fails. In the choreography-based saga pattern, each service listens for events and performs its part of the cascade delete, triggering further events to continue the process. We picked the orchestration-based saga pattern over the choreography-based saga.

At DevRev, we leverage a range of databases — including MongoDB, PostgreSQL, and DynamoDB — each selected to address specific use cases. For all identity-related data managed by Janus and the STS (which we’ll discuss later in this blog), we rely on MongoDB. MongoDB is ideal for transactional (OLTP) workloads, excelling in scenarios that demand high write throughput and the ability to manage large volumes of concurrent transactions. Its built-in replication via replica sets provides automatic failover and data redundancy across multiple nodes, enhancing both reliability and fault tolerance. To ensure data consistency, we utilize a majority write concern and read from the primary node. Currently, we are focusing on improving read and write performance in this setup.

MongoDB also offers horizontal scaling through sharding, where each shard operates as a replica set. Data is distributed across shards using a shard key, which is a field (or combination of fields) that determines how MongoDB partitions and routes data. Although we currently operate with a replica set without sharding, we’ve designed each collection in the identity database with a Dev organization ID field that can serve as the shard key, allowing us to seamlessly enable sharding when needed.

Unlike traditional relational databases, which require predefined schema structures, MongoDB’s document-oriented nature supports schema-less design. This flexibility has allowed us to rapidly develop and iterate on identity objects without the need to define a schema or handle migrations upfront.

Along with MongoDB we also use Google BigQuery, which is a fully managed, serverless, and highly scalable data warehouse designed for fast SQL-based queries and real-time analytics on large datasets. We are using the MongoDB Kafka connector to capture change data capture (CDC) events from MongoDB and push them to BigQuery. This also acts as an audit for all changes at the data level.

SCIM support had always been planned but was deprioritized due to other tasks. We’re excited that it is now an integral part of the DevRev identity platform. SCIM offers a standardized protocol that streamlines the exchange of identity information between identity providers and external applications, significantly reducing manual processes and minimizing errors. It ensures that user data is synchronized and consistent across all systems. Furthermore, SCIM enhances security by automating the revocation of user access when employees leave or change roles, reducing the risk of unauthorized access and ensuring proper access management.

DevRev provides a SCIM endpoint for provisioning users and groups from an organization’s identity provider. Initial testing was done using Okta as the identity provider, and the DevRev SCIM integration will soon be available on the Okta Integration Network. Following that, we plan to expand availability to Azure and Google platforms. As a DevRev platform customer, the DevRev organization has already implemented SCIM for integration with Okta, streamlining identity management within the organization.

The identity bridge pattern (which we discussed before) facilitates trust mediation between the identity provider trusted by the DevRev platform (Auth0) and external identity providers that authenticate users from DevRev organizations. This pattern is primarily focused on human identities. To bridge the human identity into the services, we’ve introduced a security token service (STS). The STS implementation is inspired by the RFC 8693. This adds another level of indirection in trust, and all DevRev services only need to trust the STS. You can read more about STS from this blog: Securing north/south and east/west traffic @ DevRev.

STS is a key part of the DevRev identity architecture, which helps to make the identity provider a thin layer, which we discussed earlier in this blog. An Auth0 token tells us who the user is and does not embed any PII data other than the email address. However, when exchanging an Auth0 token for an STS-issued token, we have the flexibility to enrich the token with the data available in the DevRev platform.

The STS is capable of issuing tokens of different types. This graph shows the number of tokens issued by the STS for the last 30 days.

The STS is capable of issuing tokens of different types. For example Personal Access Tokens (PAT) for Dev users, Application Access Tokens (AAT) for service accounts, Rev session tokens for Rev users, and so on. Each of these token types is bound to a set of processing rules. For example, while issuing a Rev session token, the STS talks to Janus, and just-in-time provisions the Rev user, if they do not exist in the system. Another example would be how we handle delegation workflows. When a snap-in wants to act on behalf of a Rev user, it authenticates with its AAT (which represents the snap-in service account) and requests a Rev session token. Also, the STS-issued tokens are used in the multi-regional setup to facilitate certain region-to-region interactions.

In this blog post, we highlighted key aspects of the DevRev identity platform and the lessons we’ve learned throughout its development. This is an ongoing effort, and as we continue to gain insights from our customers, we constantly discover new ways to enhance the platform and tackle exciting challenges in the identity and security landscape.

The lessons learned in building a secure, scalable, and reliable identity platform @ DevRev was originally published in FACILELOGIN on Medium, where people are continuing the conversation by highlighting and responding to this story.

22:19 • 15 days ago

Kerri Lemole

Explaining Verifiable Credentials and Open Badges 3.0

Part 1: The Trust Model of Open Badges When the Open Badges v 0.5 specification was first launched by the Mozilla Foundation in 2012, the intention was to recognize learning that happens at any time, anywhere, and any how. The trust of the badges was based on 1) the persistent hosting of the credential data and 2) the content of the credential in the context it was being evaluated (including

Part 1: The Trust Model of Open Badges

When the Open Badges v 0.5 specification was first launched by the Mozilla Foundation in 2012, the intention was to recognize learning that happens at any time, anywhere, and any how. The trust of the badges was based on 1) the persistent hosting of the credential data and 2) the content of the credential in the context it was being evaluated (including evidence which was considered much more significantly in early open badges). While digital signatures were included in subsequent versions, the trust shifted to the badging platforms who would presumably check the identity of the issuers before they allowed them to use their platform.

The challenges with this have been:

1) assuming that the badges will always be available for the earner. Trust is based on the presumption that badging platforms persist and will continue a relationship with the issuer, ensuring that even if an earner’s email address has changed that they will still have access to their badges.

2) ensuring that once a badge has been awarded to an earner that its contents do not change. There is no assured permanence for web platforms. Databases can be altered purposely or accidentally. They can be hacked and some badging platforms allow their clients to change badge descriptions after they’ve been awarded.

With the most recent update of Open Badges, version 3.0, a new trust model has been adopted that makes it possible for earners to

have access to their digital credentials for as long as they think they will need them, decide which online identity they would like associated with their credentials, know that their badges will be perceived as being trustworthy because they have been digitally signed by an identified issuer and can prove that their badge data has not changed since they claimed it.

1EdTech Open Badges 3.0 (OBv3) is based on a standard developed and maintained by the World Wide Web Consortium (W3C) called Verifiable Credentials (VCs). The W3C was founded by the creator of the web, Tim Berners Lee, and is responsible for providing the standards that make the web work. VCs can be used to prove any claim online. OBv3 builds upon the VC standard with the purpose of proving claims made about educational achievements. These credentials have historically included badges and micro-credentials but can now also be used for degrees, certifications, and licenses. This means that earners could prove all of their achievements the same way across the web with convenience and security.

This blog post is Part 1 of a series of content explaining W3C Verifiable Credentials and Open Badges 3.0. Stay tuned for Part 2 — “How do Verifiable Credentials and Open Badges 3.0 Work?”

For more information on the DCC, including membership, projects, and events, visit our website, wiki, and Linkedin. To receive our newsletter and invitations to webinars and events sign up for our community mailing list.

Explaining Verifiable Credentials and Open Badges 3.0 was originally published in Digital Credentials Consortium on Medium, where people are continuing the conversation by highlighting and responding to this story.

20:36 • 15 days ago

@_Nat Zone

マイナ保険証利用時の手続き簡素化：10月7日から開始

厚生労働省は、マイナ保険証を使用する際の手続きを簡素化し、より使いやすくする改善を10月7日から順次開始します。主な変更点は以下の2つです：限度額適用認定証情報の提供同意プロセスの簡略化変更前：変更後：医療情報提供の同意方法の一本化変更前：変更後：これらの変更により…

厚生労働省は、マイナ保険証を使用する際の手続きを簡素化し、より使いやすくする改善を10月7日から順次開始します。主な変更点は以下の2つです：

限度額適用認定証情報の提供同意プロセスの簡略化

変更前：

「高額療養費制度を利用する方はこちら」をクリック別画面に移動「限度額情報を提供しますか」という質問に回答

変更後：

画面遷移と同意プロセスを省略自動的に限度額情報が提供される（出所）厚生労働省「マイナ保険証の利用促進等について」P.13 医療情報提供の同意方法の一本化

変更前：

手術歴、診療・薬剤情報、健診結果の3つの画面で個別に同意

変更後：

1つの画面ですべての情報提供に同意可能必要に応じて個別の同意画面にアクセスすることも可能

これらの変更により、マイナ保険証使用時の手続きがよりスムーズになり、利用者の利便性が向上することが期待されます。

（出所）厚生労働省「マイナ保険証の利用促進等について」P.13

16:56 • 16 days ago

Ben Werdmüller

Silicon Valley, the New Lobbying Monster

[Charles Duhigg at the New Yorker] "As the tech industry has become the planet’s dominant economic force, a coterie of specialists—led, in part, by the political operative who introduced the idea of “a vast right-wing conspiracy” decades ago—have taught Silicon Valley how to play the game of politics. Their aim is to help tech leaders become as powerful in Washington, D.C., a

[Charles Duhigg at the New Yorker]

"As the tech industry has become the planet’s dominant economic force, a coterie of specialists—led, in part, by the political operative who introduced the idea of “a vast right-wing conspiracy” decades ago—have taught Silicon Valley how to play the game of politics. Their aim is to help tech leaders become as powerful in Washington, D.C., and in state legislatures as they are on Wall Street."

This is a major change - it wasn't so long ago that journalists were remarking that tech was hopeless at influencing Washington.

That's not always a bad thing, but it sometimes very much is - for example when Silicon Valley lobbies politicians against crypto protections, or against privacy legislation, or prevent rideshare drivers from receiving standard workplace benefits.

What is certainly true, which this article takes pains to point out, is that tech is now one of the most powerful cohorts in politics. Future Presidencies - perhaps including this next one - will be chosen in large part based on tech's agenda. That's a new normal we need to get used to, and tech workers who care about equity need to deeply understand.

#Democracy

[Link]

16:44 • 16 days ago

The blogosphere is in full bloom. The rest of the internet has wilted

[ John Naughton ] "If you log into Dave Winer’s blog, Scripting News, you’ll find a constantly updated note telling you how many years, months, days, hours, minutes and seconds the blog has been running. Sometime tomorrow morning the year field will switch to 30." Running a blog for 30 years is no small feat. Dave Winer's Scripting News is a big deal that has an enduring co

[ John Naughton ]

"If you log into Dave Winer’s blog, Scripting News, you’ll find a constantly updated note telling you how many years, months, days, hours, minutes and seconds the blog has been running. Sometime tomorrow morning the year field will switch to 30."

Running a blog for 30 years is no small feat. Dave Winer's Scripting News is a big deal that has an enduring community which he's built from scratch over that time.

This also resonates:

"In my experience, most journalists failed to understand the significance of the blogosphere. This was partly due to the fact that, like Dr Johnson, they thought that “No man but a blockhead ever wrote except for money”, and so bloggers must be weird."

My position: everyone should blog. Every new voice adds something new to the conversation. And long-term bloggers like Dave have shown the way.

[Link]

13:22 • 16 days ago

If Harris Wins, Whether She Keeps Lina Khan Will Be Extremely Telling

[Karl Bode at TechDirt] "The Harris campaign has remained largely silent on whether Khan will be allowed to stick around. And it remains entirely unclear whether Harris will continue Biden’s support of something that, for once, at least vaguely resembles antitrust reform and a crackdown of concentrated corporate power." Many tech leaders - the article calls out Reid Hoffman

[Karl Bode at TechDirt]

"The Harris campaign has remained largely silent on whether Khan will be allowed to stick around. And it remains entirely unclear whether Harris will continue Biden’s support of something that, for once, at least vaguely resembles antitrust reform and a crackdown of concentrated corporate power."

Many tech leaders - the article calls out Reid Hoffman - have put open pressure on Harris to let go of Khan. FTC leaders often change between administrations, but I agree the premise that Lina Khan has actually done a pretty good job - and certainly better at anti-trust than we've seen in decades.

That's important because tech hasn't been a sideline industry for a long time. It's integrated into every aspect of how we live our lives and learn about the world. We should care about how much power an individual tech company (and its backers) can get, both to protect a competitive market and to ensure no one company has outsized influence on our democracy.

And as Karl Bode points out, it will say a lot about Harris's Presidency:

"Right now, Harris is remaining ambiguous about whether Khan will be allowed to stay at her post; allowing voters to fill in the blanks using vibes and their imagination. Whether Khan is kept in office, or replaced with yet another cookie cutter careerist, should prove pretty immediately telling in the new year."

We may find out soon.

Code: https://github.com/damienbod/GrpcAzureAppServiceAppAuth

[Link]

12:51 • 16 days ago

Damien Bod

Microsoft Entra ID App-to-App security architecture

This article looks at the different setups when using App-to-App security with Microsoft Entra ID (OAuth client credentials). Microsoft Entra App registrations are used to configure the OAuth clients and resources. For each tenant, an Enterprise application is created for the client App registration when the consent is granted. The claims in the access token […]

Use a single ME-ID App registration for client and resource

A single Microsoft Entra ID App registration can be used for both the client and the resource specification. The aud and the azp claims in the access token will have the same value in this setup. The roles claims are of no benefit. This could be considered a bad architecture because the ME-ID App registration is used for two separate intentions, the client requesting the access token and the resource providing the service. Deployment is simplified as only one App registration is required. The setup cannot be used for multiple clients.

The access token created for this setup will have the same value for the aud claim and the azp claim as the client and the resource are the same. Both of the claims should be validated to prevent other clients accessing the resource.

{ "aud": "19893e32-3f4d-4c5a-b5ca-27891cf75666", "iss": "https://login.microsoftonline.com/7ff95b15-dc21-4ba6-bc92-824856578fc1/v2.0", "iat": 1727688828, "nbf": 1727688828, "exp": 1727692728, "aio": "k2BgYDD6x3jLSW9uSapm1it3vtdnC66c40v83qzsVLhQga//6EsA", "azp": "19893e32-3f4d-4c5a-b5ca-27891cf75666", "azpacr": "1", "oid": "a517247c-86ba-43d9-8e02-7cf7918c15f4", "rh": "0.AR8AFVv5fyHcpku8koJIVlePwTI-iRlNP1pMtconiRz3VmaFAAA.", "roles": [ "access_as_app" ], "sub": "a517247c-86ba-43d9-8e02-7cf7918c15f4", "tid": "7ff95b15-dc21-4ba6-bc92-824856578fc1", "uti": "DBleEOFcqk-rcld809IQAA", "ver": "2.0" }

In the Azure portal, the Microsoft Entra ID App registration is configured to add an application App Role. This role is returned in the access token as shown above.. It is not required in this setup.

The access token can be validated using Microsoft.Identity.Web in a Web API.

builder.Services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme) .AddMicrosoftIdentityWebApi(builder.Configuration.GetSection("AzureAd")); builder.Services.AddAuthorization(options => { options.AddPolicy("ValidateAccessTokenPolicy", validateAccessTokenPolicy => { // Validate id of application for which the token was created // In this case the CC client application validateAccessTokenPolicy.RequireClaim("azp", "19893e32-3f4d-4c5a-b5ca-27891cf75666"); // only allow tokens which used "Private key JWT Client authentication" // // https://docs.microsoft.com/en-us/azure/active-directory/develop/access-tokens // Indicates how the client was authenticated. For a public client, the value is "0". // If client ID and client secret are used, the value is "1". // If a client certificate was used for authentication, the value is "2". validateAccessTokenPolicy.RequireClaim("azpacr", "1"); }); });

Use separate ME-ID App registrations for client and resource

This architecture allows for separation of clients and is required if more than one client can use the resource. The different clients can be identified in the application using the azp claim or the oid claim. The aud claim has the resource value and must be validated. The separate clients allows you to distribute different client certificates or secrets for each client. Audit logs can also be per client. The application can also implement separate business logic or authorization based on the client_id, object ID. (azp or oid)

The aud claim returns the resource App registration client_id and the azp returns the client App registration client_id. As two App registrations are used, the claims have different values. The roles claim is also returned in the access token. The roles permission is required to setup the client and resource relationship in the portal, but not required for authorization, if the other claims are validated correctly. It must be validated that only the allowed client acquired the access token for the resource and not any application access token from the same tenant.

{ "aud": "1a03257f-18a2-4cfa-81c1-d3cfaba6b09e", "iss": "https://login.microsoftonline.com/7ff95b15-dc21-4ba6-bc92-824856578fc1/v2.0", "iat": 1727685006, "nbf": 1727685006, "exp": 1727688906, "aio": "k2BgYFCeXa1fbNX34Odf08MOyzuu8k5eIO5xfYJi0rL41gkxx9QB", "azp": "912584f0-442c-41e3-87b3-3232edd82940", "azpacr": "1", "oid": "eb11b352-b9cc-489b-9e50-5eae9ee92e09", "rh": "0.AR8AFVv5fyHcpku8koJIVlePwX8lAxqiGPpMgcHTz6umsJ6FAAA.", "roles": [ "SharedAppRoleForAppToAppClients" ], "sub": "eb11b352-b9cc-489b-9e50-5eae9ee92e09", "tid": "7ff95b15-dc21-4ba6-bc92-824856578fc1", "uti": "Poa-FXigHkWg2mrtySMPAA", "ver": "2.0" }

Claim values returned in the access token:

aud

This MUST be validated and has the client_id from the resource ME-ID App registration.

roles: SharedAppRoleForAppToAppClients

This can be used to validate the access token, if multiple clients can be used to access the resource. This is not required, if the aud is validated. When using OAuth client credentials and an App-to-App client (no delegated user), the default scope is used to access the resource and all roles from the App registration are included in the access token.

azp

The azp can be used to lock down the resource to intended clients. This is different for each client. If you validate this, only the allowed clients can use the resource.

tid

The tid claim can be used to validate the tenant which requested the resource. It is important to validate this when using multi-tenant App registrations, if the resource is not for “All” Microsoft tenants. Only allow the tenants intended for the resource.

oid

The oid claim is the Microsoft Entra ID Enterprise application object ID created for the client App registration. If you are using only a single client and validate the OID, then you can prevent other Enterprise applications using the resource.

Application implementation in ASP.NET Core

The resource can be implemented using Microsoft.Identity.Web, if it is an ASP.NET Core Web API application.

// Add services to the container. builder.Services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme) .AddMicrosoftIdentityWebApi(builder.Configuration.GetSection("AzureAd")); builder.Services.AddAuthorization(options => { options.AddPolicy("ValidateAccessTokenPolicy", validateAccessTokenPolicy => { // Validate id of application for which the token was created // In this case the CC client application // Works with multi-tenant App registrations validateAccessTokenPolicy.RequireClaim("azp", builder.Configuration["AzureAd:ClientId"]!); // Value of Azure App registration where role is defined (resource) validateAccessTokenPolicy.RequireClaim("aud", builder.Configuration["AzureAd:Audience"]!); // Single tenant Enterprise application object ID // Only validate if locking down to a single Enterprise application. validateAccessTokenPolicy.RequireClaim("oid", builder.Configuration["AzureAd:Oid"]!); // only allow tokens which used "Private key JWT Client authentication" // https://docs.microsoft.com/en-us/azure/active-directory/develop/access-tokens // Indicates how the client was authenticated. For a public client, the value is "0". // If client ID and client secret are used, the value is "1". // If a client certificate was used for authentication, the value is "2". validateAccessTokenPolicy.RequireClaim("azpacr", "1"); }); });

The application configuration would look like this with your tenant ids.

"AzureAd": { "Instance": "https://login.microsoftonline.com/", "Domain": "damienbodhotmail.onmicrosoft.com", "TenantId": "7ff95b15-dc21-4ba6-bc92-824856578fc1", "ClientId": "912584f0-442c-41e3-87b3-3232edd82940", // aud claim in the access token, Azure App registration client ID "Audience": "1a03257f-18a2-4cfa-81c1-d3cfaba6b09e", "Oid": "eb11b352-b9cc-489b-9e50-5eae9ee92e09" },

Is the App Role required?

In no setup, is the App role required in the application implementation as the aud claim can be used to validate the resource. As the default scope is used, all roles in the resource App registration are included in the access token.

The role is required to add a permission in the client ME-ID App registration for a separate App registration resource specification. The role is required in the Microsoft Entra ID portal, if you have different clients App registrations and resources App registrations.

Multi client, Multiple resources

In more complex architectures, the solution might have multiple services. A single client can used multiple resources.

The Azure App roles are required to assign the resources to the clients. The application requesting the access token for the resource must specify the default scope of the Microsoft Entra App registration resource. This means that in Microsoft Entra ID, a separate access token is used for each resource. A client can request an access token for any resource where the role has been assigned. In the resource implementation, i.e. the API access token validation, the App roles can be used to authorize the request. The aud claim can also be used and the roles are not required.

Notes

If you are using multiple clients, separate the client and the resource specifications. If you only have a single client for an API, then you can just deploy the single Azure App registration for both client and resource. If you are a purist and require that the App registrations match the implementations and are architectural correct, use only one App registration for each application, one the client and one for the resource. If requirements are changed, you can always split this later without complication. Less is more.

Links

https://github.com/AzureAD/microsoft-identity-web/wiki

https://learn.microsoft.com/en-us/entra/identity-platform/

07:25 • 16 days ago

Sunday, 06. October 2024

IdM Laboratory

Entra IDを使ったパスワードレスでのオンボーディングシナリオ

こんにちは、富士榮です。
Entra IDもVerified IDやFIDOなど色々な要素が組み合わさってきているので、それらの機能をどうやって組み合わせて使うのが良いのか？という疑問が湧いてきます。
そんな時にパスワードレスでオンボーディングをするというシナリオに基づくデザイン〜実装ガイドがMicrosoftから発行されていますので、見てみようかと思います。
Phishing-resistant passwordless authentication deployment in Microsoft Entra ID こちらのドキュメントです。 https://learn.microsoft.com/en-us/entra/identity/authentication/how-to-plan-prerequisites-phishing-resistant-passwordless-authentication
全体像はこんな感じですね。

Onboarding step 1: Identity verification 最初のステップではEntra Verified ID（＋3rdパーティソリューション）を使って政府発行のIDなどで本人確認するところからスタートします。その後、PCのBootstrapではTAP（Temporary Access Pass）を使ってドメイン参加〜認証器のエンロールをする、という流れですね。（もしくは、最近PreviewになったGraph APIで事前にFIDO認証器をプロビジョニングしておく、という方法もありますね）
関連資料） Entra Verified ID https://learn.microsoft.com/en-us/entra/verified-id/remote-onboarding-new-employees-id-verification Temporary Access Pass https://learn.microsoft.com/en-us/entra/identity/authentication/howto-authentication-temporary-access-pass#enable-the-temporary-access-pass-policy Graph APIでもFIDOクレデンシャルのプロビジョニング https://learn.microsoft.com/en-us/entra/identity/authentication/how-to-enable-passkey-fido2#provision-fido2-security-keys-using-microsoft-graph-api-preview Onboarding step 2: Bootstrap a portable credential 前のフェーズでTAPでBootstrapし、最初のクレデンシャルのエンロールをするタイミングです。ここで重要なのはデバイスにバインドされたクレデンシャルではなくポータブルなクレデンシャルをエンロールすべきである、という点です。当然働き方・デバイスの使い方によって事情は異なりますが、最初のクレデンシャルがデバイスにバインドされてしまうと後々困ることになるからですね。
Onboarding step 3: Bootstrap local credentials on computing devices ポータブルなクレデンシャルがエンロールされれば、あとは個別のデバイスのセットアップを自由にできるわけです。この段階でデバイスごとのローカルクレデンシャルをエンロールしていきます。典型的にはWindows HelloのPINの生成ですね。要するにローカルの鍵ストアをオープンするための手段を作っていくところです。

まぁ、非常に典型的な話ではありますが、ドキュメントではもっと細かくパターン分けされたデザインが出てきますので、みなさんの仕事の仕方、デバイスの種類を考えて適切なデザインをしていってください。

22:30 • 16 days ago

Ben Werdmüller

Rules for Resters

[Ben Werdmuller on Medium] I find myself returning to this piece I wrote on Medium about building downtime into your work and lifestyle. It's important: Eight years into working in America, I’m still getting used to the macho culture around vacations. I had previously lived in a country where 28 days per year is the minimum that employers can legally provide; taking time off

[Ben Werdmuller on Medium]

I find myself returning to this piece I wrote on Medium about building downtime into your work and lifestyle. It's important:

Eight years into working in America, I’m still getting used to the macho culture around vacations. I had previously lived in a country where 28 days per year is the minimum that employers can legally provide; taking time off is just considered a part of life. The US is one of the only countries in the world that doesn’t guarantee any vacation at all (the others are Tonga, Palau, Nauru, Micronesia, Kiribati, and the Marshall Islands). It’s telling that American workers often respond to this simple fact with disbelief. How does anything get done?! Well, it turns out that a lot gets done when people aren’t burned out or chained to their desks.

When was the last time you took a real lunch hour? I don't think I've had one in at least five years. That's not a good sign.

#Business

[Link]

13:49 • 17 days ago

Saturday, 05. October 2024

IdM Laboratory

SIDI Hub - ベルリンレポートを読む（３）

こんにちは、富士榮です。粛々とSIDI Hub Tokyoの準備は進んでいるわけですが、始まるまでにちゃんとベルリンのレポートを読み終わっておきましょう。前回まででPart Oneの概要が終わったので今回からはPart Twoのセッションごとの詳細レポートを見ていきます。 SIDI Summit Introduction - Mark Haine まずはイントロです。Markがレポートしてくれています。 Debora Comparin (SIA), one of the founders and organizers of SIDI Hub, opened the day and welcomed participants to the third convening of SIDI Hub before EIC in Berlin. Afforded b

こんにちは、富士榮です。

粛々とSIDI Hub Tokyoの準備は進んでいるわけですが、始まるまでにちゃんとベルリンのレポートを読み終わっておきましょう。

前回まででPart Oneの概要が終わったので今回からはPart Twoのセッションごとの詳細レポートを見ていきます。

SIDI Summit Introduction - Mark Haine

まずはイントロです。Markがレポートしてくれています。

Debora Comparin (SIA), one of the founders and organizers of SIDI Hub, opened the day and welcomed participants to the third convening of SIDI Hub before EIC in Berlin. Afforded by its association with EIC, SIDI Hub is pleased to have expertise in the room that spans deep technical knowledge of transnational governance. Further attesting to this, representatives from Germany’s Federal Ministry of the Interior & Community and SPRIN-D, Germany’s Federal Agency for Disruptive Innovation, spoke to the importance of SIDI Hub’s focus on cross-border interoperability and open standards.

SIDIハブの創設者の一人であり、主催者でもあるデボラ・コンパリン（SIA）は、ベルリンのEICの前に開催されたSIDIハブの第3回会合への参加者を歓迎し、開会を宣言した。EICとの提携により、SIDIハブはトランスナショナル・ガバナンスに関する深い技術的知識を持つ専門家を会場に迎えることができた。このことをさらに証明するように、ドイツ連邦内務・地域省およびドイツ連邦破壊的イノベーション機関SPRIN-Dの代表者は、国境を越えた相互運用性とオープンスタンダードに焦点を当てたSIDI Hubの重要性を語った。

私も現地で参加しましたが、ドイツ政府の100％出資の機関であるSPRIN-Dのオフィスでイベントは開催されました。そのため、最初のWelcome keynoteはSPRIN-Dの方が担当しました。

Nick Mothershaw (OIX) reviewed the agenda, which emphasized the following:
1. Identifying Champion Use Cases
2. Identifying Major Barriers to Interoperability
3. Minimum Technical Requirements for Interoperability
4. Deepening our Trust Framework Analysis
5. Critical Research Questions

Nick Mothershaw（OIX）は、以下の点を強調したアジェンダをレビューした。
1 チャンピオンのユースケースの特定
2. 相互運用性に対する主な障壁の特定
3. 相互運用性のための最低技術要件
4. 信頼フレームワーク分析の深化
5. 重要な研究課題

そのあとはNickによるアジェンダの紹介がありました。

SIDI Strategy and Structure - Mark

続いてGailによるSIDI Hubのストラテジーとストラクチャの話です。

Gail Hodges provided an overview of the origins of SIDI Hub, which derived from an ID4Africa presentation and the “Human-Centric Digital Identity” paper. In particular, the problem of cross-border interoperability in the context of national Digital Identity strategies encompassing a wide range of technical architectures and governance models. Gail Hodges は、ID4Africa のプレゼンテーションと「人間中心のデジタル ID」論文から派生した SIDI ハブの起源について概要を説明した。特に、広範な技術アーキテクチャーとガバナンス・モデルを包含する各国のデジタル ID 戦略の文脈における国境を越えた相互運用性の問題について述べた。

この辺りはいつものGailのセッションなのですが、各国のデジタルID戦略をPublic Governance-Private Governance、Centralized-Decentralizedの２軸で4象限に分類し、現状のばらつきを表現しつつ、この環境のもとで国境を超えた相互運用性を達成することの必要性について話しているわけです。

Despite these challenges - and the ongoing need for domestic sovereignty - can one’s Digital Identity be as easy to present as an email, a phone number, or a passport? SIDI Hub seeks to build a blueprint for how we build Digital Identity ecosystems within and across ecosystems. The goal is for implementers to build interoperable Digital Identity credentials by default. But this, of course, requires measurement and metrics, policies, open standards, open source code (in many jurisdictions), and scientific analysis for best practice security.
このような課題があるにもかかわらず、そして国内主権の継続的な必要性があるにもかかわらず、デジタル ID は電子メール、電話番号、パスポートのように簡単に提示することができるのだろうか？SIDI ハブは、エコシステム内およびエコシステム間でデジタル ID エコシステムを構築する方法の青写真を構築することを目指す。目標は、実装者がデフォルトで相互運用可能なデジタル ID クレデンシャルを構築することである。しかし、これにはもちろん、測定と測定基準、ポリシー、オープン・スタンダード、オープン・ソース・コード（多くの法域で）、およびベスト・プラクティスのセキュリティのための科学的分析が必要である。

こんなバラバラな状態の中でもデジタルIDをメールや電話やパスポートのように国境を超えて世界中で相互運用できる状態にするにはやることがたくさんありますね。まさにこれがSIDI Hubがやろうとしていること、というわけです。

SIDI Hub is self-organized into five workstreams:
Champion Use Cases Trust Framework Mapping Minimum Requirements for Interoperability Metrics of Success Governance
As referenced above, SIDI Hub has no governance authority of its own. We therefore discussed where　decisions are made, which remain unchanged as a result of SIDI Hub, and how SIDI aims to support　them.

SIDI Hubは、以下の5つのワークストリームから構成される。
チャンピオンのユースケーストラストフレームワークマッピン相互運用のための最低要件成功の指標ガバナンス
上記で言及したように、SIDI Hub はそれ自体のガバナンス権限を持たない。そのため、SIDI Hubの結果として変わることのない意思決定がどこで行われるのか、また、SIDIがどのようにそれをサポートすることを目指しているのかについて議論した。

こちらはいつものSIDI Hubとは何なのか、という話と構成するワークストリームの話ですね。非常に難しい部分なのですがコミュニティなのでコンセンサスを取りながら意思決定をしていくというのが特徴でもあります。この辺りは今後変わっていくかもしれません。

今回はこのくらいです。ユースケース分析のセッションについて次回解説します。

23:00 • 17 days ago

Werdmüller on Medium

The two Fediverses

For some, it’s about growth. For others, it’s a movement. They must work together. Continue reading on Medium »

For some, it’s about growth. For others, it’s a movement. They must work together.

https://digitalidadvancement.org/news/2024-kim-cameron-awardee-reflections-matthew-spence/

20:26 • 17 days ago

Jon Udell

Geothermal power in the North Bay

I was aware of The Geysers, a geothermal field about 35 miles north of my home in Santa Rosa, but I never gave it much thought until my first bike ride through the area. Then I learned a number of interesting things. It’s the world’s largest geothermal field, producing more than 700 megawatts. It accounts … Continue reading Geothermal power in the North Bay

It’s the world’s largest geothermal field, producing more than 700 megawatts.

It accounts for 20% of California’s renewable energy.

The naturally-occurring steam was used up almost 30 years ago, and steam is now recharged by pumping in 11 million gallons of sewage effluent daily, through a 42-mile pipeline, from the Santa Rosa plain.

That daily recharge is implicated in the region’s frequent small earthquakes. (But nobody seems too worried about that, and maybe it’s a good thing? Many small better than one big?)

An article in today’s paper reports that AB-1359, signed last week by governor Gavin Newsom, paves the way for new geothermal development in the region that could add 600 megawatts of geothermal production.

How much electric power is that? I like to use WolframAlpha for quick and rough comparisons.

So, 2/3 of a nuke plant. 4/5 of a coal-fired power plant. These kinds of comparisons help me contextualize so many quantitative aspects of our lives. They’re the primary reason I visit WolframAlpha. I wish journalists would use it for that purpose.

18:42 • 18 days ago

IdM Laboratory

Kim Cameron Awardの受賞者によるIdentiverseへの参加レポート

こんにちは、富士榮です。先日お知らせしたVittorio Bertocciアワードと並行してDIAF（Digital Identity Advancement Foundation）が提供する個人向けの世界２大アイデンティティ・アワードであるKim Cameronアワードの受賞者であるMatthew SpenceがIdentiverse 2024への参加レポートを書いています。 https://digitalidadvancement.org/news/2024-kim-cameron-awardee-reflections-matthew-spence/ DIAFではアワード受賞者にIdentiverseやIIW、EICなどのカンファレンスへの参加をサポートしており、有能でやる気があっても費用面で課題がある若手などへのスポンサーをしています。ダイバーシティを確保のために

こんにちは、富士榮です。

先日お知らせしたVittorio Bertocciアワードと並行してDIAF（Digital Identity Advancement Foundation）が提供する個人向けの世界２大アイデンティティ・アワードであるKim Cameronアワードの受賞者であるMatthew SpenceがIdentiverse 2024への参加レポートを書いています。

DIAFではアワード受賞者にIdentiverseやIIW、EICなどのカンファレンスへの参加をサポートしており、有能でやる気があっても費用面で課題がある若手などへのスポンサーをしています。

ダイバーシティを確保のためにはこのような取り組みは非常に有用ですね。日本でも何かやれないかなぁ、、、と思いますが、まずは日本からもDIAFのアワードにApplyしてみる方が出てくることに期待です。

01:50 • 18 days ago

Thursday, 03. October 2024

IdM Laboratory

OpenID Connect for Identity Assuranceの仕様が承認されました

こんにちは、富士榮です。
Great newsです。先日より投票が開始されていたOpenID Connect for Identity Assuranceの仕様が最終化、承認されました。投票のお知らせ） https://idmlab.eidentity.jp/2024/09/openid-connect-for-identity-assurance.html
最終化に関する公式アナウンス） https://openid.net/final-openid-connect-for-identity-assurance-specifications-approved/

今回承認された仕様は以下のとおりです。 OpenID Identity Assurance Schema Definition 1.0 - https://openid.net/specs/openid-ida-verified-claims-1_0-final.html OpenID Connect for Identity Assurance Claims Registration 1.0 - https://openid.net/specs/openid-connect-4-ida-claims-1_0-final.html OpenID Connect for Identity Assurance 1.0 - https://openid.net/specs/openid-connect-4-identity-assurance-1_0-final.html
皆さん、使っていきましょう。

23:01 • 19 days ago

Doc Searls Weblog

Think Globally, Eat Here

Fifteenth in the News Commons series. This semester’s Beyond the Web salon series for the Ostrom Workshop and Hamilton Lugar School at Indiana University is themed Think Globally, Eat Here—Small Solutions for Big Tech Problems. I will give the opening talk, about the News Commons (subject of fourteen prior posts here) at noon (Eastern) next […]

Fifteenth in the News Commons series.

This semester’s Beyond the Web salon series for the Ostrom Workshop and Hamilton Lugar School at Indiana University is themed Think Globally, Eat Here—Small Solutions for Big Tech Problems. I will give the opening talk, about the News Commons (subject of fourteen prior posts here) at noon (Eastern) next Tuesday, October 10. If you’re in town, please attend in person. If not, join us by Zoom. Do that here.

Our plan is to prototype and prove locally what can apply globally for local news, starting with what Columbia Journalism Review called “news deserts” back in 2017—a label that has since caught on. There are many efforts toward seeding and watering these deserts, most prominently Press Forward, which is devoting $500 million to that challenge.

Bloomington is advantaged by not being one of those deserts, and instead having a talented pool of local journals, journalists, and organizations—including its legacy newspaper—all doing good work that could still be improved by putting to use some of the innovations I’ll be talking about, and by working together as a commons.

So join the conversation. I look forward to seeing you in the room or on the wall (because one whole wall is our Zoom screen).

[Later…] My talk is here on YouTube.

19:27 • 19 days ago

The Pragmatic Engineer

The Pulse #109: Open source business model struggles at Wordpress

Also: OpenAI’s biggest-ever fundraise even as key people keep quitting; why executive recruiters ignore tech professionals, and more

Today, we cover:

Industry pulse. Microsoft won’t follow Amazon in 5-day RTO, Cloudflare auto-mitigates world-record DDoS attack, California nearly regulates AI companies, Revolut starts selling performance management framework as a product, and more.

Open source business model struggles: Wordpress. Automattic, creator of Wordpress, is being sued by one of the largest WordPress hosting providers. The conflict fits into a trend of billion-dollar companies struggling to effectively monetize open source, and are changing tactics to limit their competition and increase their revenue.

OpenAI: biggest-ever fundraise, as key people keep quitting. OpenAI raised a $6.6B in funding on a $157B valuation, making it the largest fundraiser ever. And yet, cofounders and executives continue to depart. This time it’s the CTO, Chief Research Officer, the VP of Research. Also, OpenAI will most likely finally become a for-profit.

Why executive recruiters ignore tech professionals. Connecting with executive recruiters is a helpful strategy for engineering leaders. Senior product director shares key reasons these outreaches fail, and what you can do to improve chances.

1. Industry pulse Microsoft won’t follow Amazon in 5-day RTO

In Seattle, the two largest tech employers are Microsoft and Amazon. Both companies are willing and able to match compensation of each other, so when Amazon announced two weeks ago that it’s mandating a strict 5-days-from-office policy from 1 Jan 2025, the question was whether Microsoft would follow.

The company has internally confirmed it will not, as per Business Insider, Microsoft EVP Scott Guthrie told employees that Microsoft has no plans to tighten remote work policies. This means Microsoft workers can expect to spend 2-3 days per week in the office as a hybrid arrangement. I expect experienced Amazon software engineers who are unhappy about their workplace’s 5-days-per-week policy, and who live within commuting distance of a Microsoft office, to start applying for open positions at Microsoft.

Cloudflare auto-mitigates world-record DDoS attack

17:29 • 20 days ago

Wednesday, 02. October 2024

IdM Laboratory

SIDI Hub東京、前々夜祭を開きます

こんにちは、富士榮です。いよいよ今月末はSIDI Hub東京イベントです。が、小規模でのディスカッション中心、かつ英語イベントということもあり招待者に限りご参加いただくという形となります。そのため、日本のアイデンティティ関係者の皆さんにも概要を知っていただく場として前々夜祭としてイベントを開くことにしました。 https://openid.connpass.com/event/332975/ こちらは日本語で、かつ実際に活動をしている方からも話をしてもらえるようにしたいと思いますので、ぜひご参加ください。

こんにちは、富士榮です。
いよいよ今月末はSIDI Hub東京イベントです。が、小規模でのディスカッション中心、かつ英語イベントということもあり招待者に限りご参加いただくという形となります。
そのため、日本のアイデンティティ関係者の皆さんにも概要を知っていただく場として前々夜祭としてイベントを開くことにしました。
https://openid.connpass.com/event/332975/
こちらは日本語で、かつ実際に活動をしている方からも話をしてもらえるようにしたいと思いますので、ぜひご参加ください。

22:30 • 20 days ago

Tuesday, 01. October 2024

IdM Laboratory

Death and the Digital Estate（DADE）CGが発足

こんにちは、富士榮です。 DADE CG（Death and the Digital Estate Community Group）の発足がアナウンスされています。死後のデジタルアイデンティティや遺産について扱うコミュニティグループです。 https://openid.net/announcing-death-and-the-digital-estate-cg/ 4月のIIWの前日のOpenID Foundation Workshopで触れられていたコミュニティですね。 https://idmlab.eidentity.jp/2024/04/openid-foundation-workshopgoogle.html 当時AWSにいたDean Saxe（右から二人目。今はBeyond Identityに移籍）がChairを務めるようです。メーリングリス

こんにちは、富士榮です。
DADE CG（Death and the Digital Estate Community Group）の発足がアナウンスされています。死後のデジタルアイデンティティや遺産について扱うコミュニティグループです。 https://openid.net/announcing-death-and-the-digital-estate-cg/
4月のIIWの前日のOpenID Foundation Workshopで触れられていたコミュニティですね。 https://idmlab.eidentity.jp/2024/04/openid-foundation-workshopgoogle.html

当時AWSにいたDean Saxe（右から二人目。今はBeyond Identityに移籍）がChairを務めるようです。
メーリングリストへの参加なども受け付けていますので参加してみてはいかがでしょうか？

22:30 • 21 days ago

Doc Searls Weblog

2024_10_01 Postings

A radio item Over on my blog about infrastructure, I put up a brief post about WART, volunteer-powered community radio station with studios in a railroad caboose, that was lost in the flood that just devastated Marshall, North Carolina. Write once, publish everywhere Dave turned me on to Croissant today. Looks good. I’d even be […]

Croissants (the edible kind) on display at Peets in Santa Barbara.

A radio item

Over on my blog about infrastructure, I put up a brief post about WART, volunteer-powered community radio station with studios in a railroad caboose, that was lost in the flood that just devastated Marshall, North Carolina.

Write once, publish everywhere

Dave turned me on to Croissant today. Looks good. I’d even be willing to pay the monthly fee to post once across Bluesky, Mastodon, Threads, and Xitter. But it appears to be only for iOS mobile devices. I have some of those (including a new iPhone 16 Pro), but I mostly write on a computer. So I’ll hold out for real textcasting, like Tim Carmody talks up here. Because why should you have to post separately at all those places? Why should you have to go to a place at all, when you’ve got your own devices to write on and distribute from?

A heading convention

I started curating my photos (e.g. these) in the last millennium using this date-based naming convention: YYYY_MM_DD_topical-text_NNN.jpg (where the NNNs are just sequential numbers and the file type suffix could be .jpg, .arw, .cr2, .png or whatever. Same for folder titles.) So, because I don’t want a new title for every day I do this, I’m adopting the same convention, at least for now.

Not fast enough

In The End of Advertising, Michael Mignano says (in the subhead, and beyond), The business model that funded the internet is going away, and the open web will never be the same. He says AI is already killing it, by giving us answers to everything, and performing other handy tasks, without advertising to distract or annoy us. He also says AI services will attempt to invent ads, but that’s a losing proposition, mostly because it won’t work and we’ll hate it, but also because “content wants to be free.” (I submit that no art forms, ever, wanted to be called “content.”) I agree. I also agree that “Premium content will become even more premium.” He concludes, “the relationship between us and publishers will become much more transactional and direct. And we will feel it. Over time, it’ll be a new internet, and the open web will be a memory. Great content will still find a way to reach us, just like it always has. But we’ll look back on the first few decades of the internet as the golden age of content, when everything felt free.” Well, you’re reading some premium content right now, and it’s free. Thanks to what I do here, I can make money in other ways. We call those because effects.

21:37 • 21 days ago

The Pragmatic Engineer

Bug management that works (Part 1)

Finding and triaging bugs, fixing bugs on the spot instead of ‘managing’ them, and how to make time for bug fixing

Incident review and postmortem best practices

Before we start: The Pragmatic Engineer Podcast started off last week, with episode #1: AI tools for software engineers, but without the hype. New episodes come every other Wednesday. Thank you to everyone who listened to this first one. If you enjoy podcasts, please do add it on Apple, Spotify, YouTube or your in your favorite player – and you will have episodes automatically show up, while also helping the show.

How do you deal with bugs in software products you build? This topic seems very under-discussed, but it affects pretty much every software team. To find out what professionals think, with we reached out to two dozen engineering leaders and software engineers, who kindly shared approaches which work for their team and organization.

This topic is evergreen, and one that has outsized (potentially decisive!) impact on product quality and user experience.

In this issue, we cover:

Finding bugs. Dogfood products where possible, invest in test automation, stay close to users, and more.

Reporting bugs. Making it easy for users to report bugs, having an accessible engineering team, and scaling the reporting process by team or company size.

Bug triaging. Figure out which bugs are critical by using bug priority rubrics, consider priority definitions, and get inspired by Spotify’s bug prioritization framework.

Don’t manage bugs: fix them! See bug fixing as like gardening, not maintenance. Consider approaches like deleting all amassed bugs, and regularly pruning the backlog.

Zero bugs policy. An approach where all inbound bugs are either fixed on the spot: or the bug is deemed invalid. A subset of this approach is to fix all bugs for recently released features – which is what Figma does.

As a refresher, we have a few previous deepdives that are related to this topic:

Shipping to production

Healthy oncall practices

How Big Tech does Quality Assurance (QA)

QA approaches across the industry

Thank you to everyone who contributed insights to this article: Ahmed Saher (engineering manager), Anaïs van Asselt (senior QA engineer), Andrea Sipos (product leader), Bernd Kampl (Software Engineering Team Lead), Jason Diller (VP of Engineering), John Cutler (product leader), Magnus L. Udbjørg (CTO), Michał Borek (Principal Engineer), Rebecca Frost (QA leader), Rebecca Holm Ring (engineering leader), Ruben Weijers (engineering manager), Ryan Hanni (Director of Engineering), Serdar Biyik (engineering manager), Walter de Bruijn (Head of Engineering Productivity)

1. Finding bugs

How can we be confident that the software we release has no known issues? We need to validate that it works correctly and there are common approaches for this.

Dogfood products. The term “dogfooding” is the name of the common practice of devs and employees using a product while they are building it, pre-release. For example, when I worked at Uber, the company issued free credits for staff to use the internal beta app for rides and food deliveries. At Skype, we used running internal beta versions of Skype for all internal chat and video calling. The business gave Skype credits to employees, so we could dogfood paid features like landline calls. Spotify does the same, as Rebecca Holm Ring, a former engineering manager there, shares:

“Each employee at Spotify is expected to be on the master or internal Spotify release, and report any issues they experience. A problem here though is that most Spotify employees will be iOS users, and so the Android app doesn’t get nearly enough testing before rolling out.”

Invest in test automation. Anaïs van Asselt – senior QA at Choco – shares their approach:

“Proactively finding and fixing bugs before they reach production is crucial. We invest in various test automation techniques to act as quality gates. Additionally, we practice bug hunting and gradually roll out new features to a limited user base, allowing us to catch and fix bugs early, reducing their overall cost.”

At smaller companies, be close to users. These places tend to be closer to users and can use this to build a relationship with users who get invested in the product and the reporting of bugs. Bernd Kampl – Software Engineering Team Lead at Anyline, a smaller cartech AI company – shares:

“As we are still smaller, we usually find bugs when our customers report them. If that's not the case, we identify them mostly during day-to-day development work.”

Magnus Udbjørg is CTO of Testaviva, a 50-person startup in Denmark. His take is that it’s optimal to build trust with users so they report issues:

“To find, identify, and prevent bugs, we foster a culture of dialogue and mutual understanding. We believe our developers need to understand our users deeply, and sometimes, our users need to understand the technical constraints we face. Open communication is key to building trust.”

A fair question is why not do lots of testing, themselves? The smaller the company and the fewer the customers, the more it feels too expensive to invest a lot in testing, early on. Of course, there are always countercases, like how Figma spent nearly 3 years iterating on the first release, in order to get the performance of their collaborative, web-based editor right, to give users a fun “wow moment.” Worth noting that Figma is a product the dev team used continuously while developing it, getting lots of testing during the building phase. We cover Figma’s engineering culture in a deep dive.

Consider alpha and beta testing at larger companies. Alpha and beta testing is about giving customers access to unfinished, less stable versions of a product. “Alpha” usually refers to a latest build that has had little to no QA testing. “Beta” versions have had some testing, but not as much as a full release.

Rebecca Holm Ring shares how this worked at Spotify:

“There’s an Alpha testing, and Beta testing program, where external users are on the master branch and release branch respectively, and are expected to report any issues they encounter. With these versions of Spotify, it is also possible for a user to report a bug directly from the app, and logs will be attached as the bug report will be automatically added in JIRA.”

Automation: testing and code analysis. Unit tests, integration tests, end-to-end-tests, and other automated tests, are great ways to catch regressions, which is a software bug introduced into a feature after the feature was working correctly; the feature has ‘regressed’ into a faulty state.

This is true for static code analysis and other tools that automate quality assurance. We cover more on these methods in Shipping to production and QA approaches across the industry.

Code reviews. These serve multiple purposes, offering a second pair of eyes to double check code, spread knowledge, and follow not-yet-automated conventions, and more. Catching bugs before they make it into the codebase is an occasional side effect.

Even so, bugs can easily slip through code reviews, which are nowhere near a perfect way to defend against shipping bugs and regressions.

Define what a bug is. Users often report “bugs” when they mean missing features, so it can be helpful for teams to agree what a bug is and how to categorize them. In general, a bug is a flaw that results in a software product behaving incorrectly. Categorizations can be granular, like splitting bugs into concurrency bugs, syntax ones, arithmetic, logical errors, human errors and so on.

The simplest categorization is to split bugs into functional ones, when the behavior of the software is clearly wrong, and non-functional ones, when a bug is revealed in things like a system slowing down, increased latency, and other harder-to-spot issues.

It might be helpful to devise your own categorizations, based on the type of bugs you observe, and in a way that’s helpful for your product and organization.

2. Users reporting bugs

Gathering bugs can be a great source of data, providing a sense of product quality for feedback to teams, the organization, or company. However, data quality depends on how good the bug reporting process is – and how likely people are to report bugs!

Great reports and data come from simple, suitable processes. Features of useful bug reports:

Useful metadata (e.g. version, device, system metrics)

Relevant context (e.g. on mobile while connected to bluetooth speaker and poor connectivity, on a server in this region during lunch hour, on a debug build with these feature flags active, etc)

Straightforward to reproduce, or have reproduction steps

Reported by users who trust a reported bug will be fixed

Bad reports can create extra work and poor bug reporting processes can cause people to not commit to recording issues in the first place, and a spiral is created of deteriorating product quality, with the engineering team clueless of how bad things are.

To avoid an outcome like that, here are some processes tech companies use to support good bug reporting processes.

Make it easy to create quality bug reports. Walter de Bruijn, Head of Engineering Productivity at Miro suggests this is critical:

“The easier the reporting is, the more you will learn. For internally discovered bugs an internal #bugs Slack channel can go a long way.
One of my best recommendations is that you start logging bugs properly: you can’t manage what you can’t see. You need a good, well-documented, and known process for reporting bugs and a follow-up.”

QA leader Rebecca Frost on why quality bug reports count:

“Capture meaningful metadata on your bugs now to help with insights later.
Make sure your funnel for incoming bugs is helping not hurting you. If bugs are coming in from all over and the quality of the info on the bugs is a mess, consider routing bug reports through one team or group.”

Make the reporting process accessible. If creating a bug report is too complicated, it discourages reporting. There are ways to make it accessible:

Don’t make less technical users use unfamiliar tech jargon

Make it easy to find version numbers and other data required for bug reports. Even better, collect it automatically

Consider having engineers help support teams to know which questions to ask to get the right data for reports

A example of effective engineering support is at SF-based scaleup Ontra, as shared by director of engineering, Ryan Hanni:

“We have an Engineering Support team that sits between Product Support/Customer Success and R&D, which helps bridge the customer with R&D teams potentially further away from that context.”

Scaling bug reporting processes

There’s no one process that works best everywhere. Here are some common approaches by company size:

Smaller companies and startups: bug reports are usually simple, and the reporting process is lean because time is precious and knowledge is dense. Such workplaces are small enough that most tech folks can keep tabs on what’s happening, and people can submit bug reports pretty easily.

There’s rarely a need for formal processes. Here are some efficient, less formal ones:

Set up a #bug-reports channel on Slack/Teams or other chat product

Use a #bugs tag in the ticket system of choice to keep track of issues

Prefix bug fix pull requests or commits with [Bugfix]

… and similar, simple but effective conventions that anyone can start using

Mid-sized companies and scaleups: process matters more, and these places are big enough for it to be wasteful for everyone to keep tabs on reported bugs. There are also more bug reports, and it’s a time waster to get repeated information and metadata in bug reports.

Bug report templates and processes also matter. Good onboarding and documentation for processes and standards for bugs can have a big impact on efficiency.

Large companies: investing in automated processes is worthwhile due to the size and nature of the business:

Noisy bug reporting means lots of duplicate reports

“JIRA ping pong”: tickets created in JIRA are continually reassigned between teams and end up ‘returned to sender’

Time wasted discussing bugs: lots of overhead on debating whether a bug is a feature request, who’s best placed to fix it, etc

Here’s a good example of what JIRA ping pong looks like. Engineering manager Rebecca Holm Ring shares how it plays out a larger company:

“Bugs showing up in one user-facing feature could be caused by a number of different teams that own the deeper layers of the tech stack. Still, reports would always first land on the user facing team!
There would often be a game of ping pong, where bugs got sent back-and-forth between different teams, and sometimes found the right owners. In turn, these owners didn’t feel it was a high enough issue for them to fix. However, for the user facing team, it would often be a higher priority to get it resolved. As many different teams were involved in building the Spotify app, conflicting priorities often made bugs linger longer than they should have, and occasionally required intervention from the higher-ups to get these bugs prioritized by the owning team.”

At larger companies, some things can help deal with an ever-growing pile of bug reports, and improve processes and tooling:

Automation across the bug reporting process

Automated data collection for gathering context

Bug reporting templates for internal use

Automatic code/team attribution for inbound reports

Ryan Hanni, director of engineering at Ontra, shares examples of manual and mostly-automated processes he’s implemented at different stages of an org’s lifecycle:

Manual process:

Use a bug template in Trello, JIRA, or similar

Notify Eng Support and copy the bug description to their slack channel

→ Eng support aids in determining severity, and the team takes it from there

→ Eng Director uses a Google Spreadsheet to track all bugs, and includes details like report date and links

→ Eng Director does monthly follow ups on bugs, and adds details to the spreadsheet, like the timestamp of a fix. This spreadsheet is used for calculating stats like time-to-resolution for DORA metrics.

Partially Automated:

Use a bug template in Trello, JIRA, or similar.

With a press of a button, automatically post to Slack

→ Eng Support 'triages' the bug with the dev team in a thread

→ For high or medium severity bugs, support copies the details to an Airtable form and submits it

→ The form submitted uses automation to:

Post to Slack

Send an email to internal google groups for stakeholders and departments

Automatically adds bug information into a Airtable columns

→ Partially automated reporting:

Metrics for ‘Reported at’ date and severity are automatically filled in

Metrics are filed on a monthly basis to generate stats like DORA metrics and cycle time to fix (Time to Respond, Time to Fix, Total Time In Prod, etc).

→ When a bug is fixed, teams reply to the automated email detailing when it will be released to customers

Mostly Automated:

Use a bug template in Trello, JIRA, or similar.

→Use automation to send a message to Eng Support slack.

→ Triage happens with the Dev Team and Eng Support

→ Work moves through Jira columns (Kanban style), into a Done column

→ A release to production happens and the Jira card gets an automatic “released at” date

→ After a release to Prod, Jira sends a slack/email message with issue details to close the loop for a resolved bug of a given severity.

→ Reporting occurs by generating reports in JIRA. Run reports with all relevant metrics to the org to get information like cycle time, time to fix, response time, etc

→ Visuals for share outs on the bug fixing process is manually generated in Google Sheets

3. Bug triage

You now have a process for the bug reports to flow in, so the next step is to figure out which ones are critical, which are duplicates, and which ones not to bother with. Here are some common approaches:

16:34 • 22 days ago

Tuesday, 01. October 2024

Mike Jones: self-issued

OpenID Connect specifications published as ISO standards

I’m thrilled to report that the OpenID Connect specifications have now been published as ISO/IEC standards. They are: ISO/IEC 26131:2024 — Information technology — OpenID connect — OpenID connect core 1.0 incorporating errata set 2 ISO/IEC 26132:2024 — Information technology — OpenID connect — OpenID connect discovery 1.0 incorporating errata set 2 ISO/IEC 26133:2024 — […]

I’m thrilled to report that the OpenID Connect specifications have now been published as ISO/IEC standards. They are:

I submitted the OpenID Connect specifications for publication by ISO as Publicly Available Specifications (PAS) for the OpenID Foundation in December 2023. Following the ISO approval vote, they are now published. This should foster even broader adoption of OpenID Connect by enabling deployments in jurisdictions around the world that have legal requirements to use specifications from standards bodies recognized by international treaties, of which ISO is one.

Before submitting the specifications, the OpenID Connect working group diligently worked through the process of applying errata corrections to the specifications, so that the ISO versions would have all known corrections incorporated.

Having successfully gone through the ISO PAS submission process once, the OpenID Foundation now plans to submit additional families of final specifications for publication by ISO. These include the FAPI 1.0 specifications, and once they’re final, the eKYC-IDA specifications and FAPI 2.0 specifications.

Thanks to all who helped us achieve this significant accomplishment!

16:32 • 22 days ago

Tuesday, 01. October 2024

Doc Searls Weblog

Podcasts, Wallcasts, and Paycasts

Would a blog be a blog if it went behind a paywall, or if you needed a subscription to read it? Of course not. Blogs are on the open Web, and tend to stay there so long as they don’t move away from their original location. Same should go for podcasts. “Wherever you get your […]

The Ezra Klein Show, as it appeared on my podcast app this morning. It is now a wallcast.

Would a blog be a blog if it went behind a paywall, or if you needed a subscription to read it?

Of course not. Blogs are on the open Web, and tend to stay there so long as they don’t move away from their original location.

Same should go for podcasts. “Wherever you get your podcasts” certifies the open nature of podcasting.

But now the New York Times is putting all its podcasts archives behind a paywall, including Ezra Klein’s, above.

Arguments about the merits of this are all economic. Let’s instead look at classification. We need a new noun for restricted ‘casts such as Ezra’s, which are no longer podcasts in the pure sense. I suggest we call them wallcasts.

For subscription-only ‘casts, such as some on SiriusXM*, I suggest paycasts.

Bottom line: It can’t be a podcast if you have to pay for any of it, including archives.

By the way, it won’t matter if a Times subscription opens wallcast archives, as a subscription does for print archives. By putting their podcast archives behind a paywall, the Times is changing the DNA of those casts. A wallcast is not a podcast. Full stop.

Spread the words.

*SiriusXM’s paycasts include “SmartLess,” “Freakonomics Radio,” “The Joel Osteen Podcast,” “Last Podcast on the Left,” and “Andy Cohen’s Daddy Diaries.” They require a subscription to SiriusXM or its Podcasts+ service. Some, such as “Marvel’s Wastelanders” and “Marvel/Method also require a subscription. I’m not sure what kind. (FWIW, I’ve been a SiriusXM subscriber since 2005, but only listen to live subscription streams. I’ve never listened to any of its podcasts.) SiriusXM does have some shows in podcast form, however. Examples are “The Megyn Kelly Show,” “Best Friends with Nicole Byer and Sasheer Zamata,” and “Chasing Life with Dr. Sanjay Gupta.” I believe it also has some wallcasts. For example, “SmartLess” episodes are on the open Web, but early access and bonus episodes are behind a paywall. Or so it seems to me in the here and now. I invite corrections.

15:01 • 22 days ago

Michael Ruminer

Google’s NotebookLM, RAG and Then Some

I played around just a little bit with Google’s NotebookLM. It’s a Retrieval-Augmented Generation (RAG) tool. One thing it does is convert your content into a podcast format and it provides an impressive-sounding result. It's a little superficial on the content synthesis and definitely gimmicky, but it is a podcast after all. Its other features are your typical RAG chat interface. All I can say is

Here is a link to a single source document I provided it- Self Sovereign Identity A Systematic Review Mapping and Taxonomy and the link to the SSI Paper Podcast results.

11:53 • 22 days ago

Monday, 30. September 2024

IdM Laboratory

SIDI Hub - ベルリンレポートを読む（２）

こんにちは、富士榮です。しばらく別のネタばかりでSIDI Hubについてかけていませんでしたが、10月に入ったので東京開催秒読みということでベルリンレポートの続きを読んでいきます。前回からしばらく開きましたが、今回は続きです。 Users of a Trust Framework Analysis Tool A major output of the SIDI Hub 2024 strategy, led by the Open Identity Exchange (OIX), will be a Trust Framework Comparison Tool. This will be bolstered by further analysis and normalization of Trust Frameworks supported by SIDI Hub

こんにちは、富士榮です。

しばらく別のネタばかりでSIDI Hubについてかけていませんでしたが、10月に入ったので東京開催秒読みということでベルリンレポートの続きを読んでいきます。

前回からしばらく開きましたが、今回は続きです。

Users of a Trust Framework Analysis Tool
A major output of the SIDI Hub 2024 strategy, led by the Open Identity Exchange (OIX), will be a Trust Framework Comparison Tool. This will be bolstered by further analysis and normalization of Trust Frameworks supported by SIDI Hub. At the SIDI Berlin session, breakout groups shaped the value proposition and requirements for such a comparison tool, which will directly influence the final deliverable. Further information is found in the Rapporteur’s notes (next section).

信頼フレームワーク分析ツールのユーザー

Open Identity Exchange (OIX) が主導する SIDI Hub 2024 戦略の主な成果のひとつは、信頼フレームワーク比較ツールです。これは、SIDI Hub がサポートする信頼フレームワークのさらなる分析と標準化によって強化されます。SIDI Berlin セッションでは、分科会がこのような比較ツールの価値提案と要件を策定し、最終成果物に直接影響を与えることになります。詳細は、ラポータのメモ(次項)をご覧ください。

トラストフレームワークのマッピングに関して書かれていますね。

現在、各国で制定が進んでいるトラストフレームワークの相互運用が可能な状態にならないと国の間で相互運用性の担保ができなくなるのでここでいうマッピングは非常に重要です。OpenIDファウンデーションジャパンではOIXに協力する形で日本のトラストフレームワークのマッピングを支援しています。

BOLTS: Business, Operational, Legal, Technical, and Social
Given the above take-aways, which span Business, Operational, Legal, Technical, and Social forces that impact the global interoperability effort, the group will use a “BOLTS” framework as a core part of its Champion Use Case analysis.

BOLTS:ビジネス、運用、法律、技術、社会
グローバルな相互運用性への取り組みに影響を与えるビジネス、運用、法律、技術、社会の各分野における上記の要点を踏まえ、当グループは「BOLTS」フレームワークをチャンピオンユースケース分析の中核として使用します。

相互運用性を考える上では技術だけを考えていたは不十分です。ここにあるようにビジネス、運用、法律、社会を含めて考える必要がある、ということです。

Government Participation
A final point of reflection relates to the audience for SIDI Hub events. Given the light attendance from government officials in Berlin, the agenda skewed towards a technical audience that discussed technical possibilities. This is not ideal.
政府の参加
最後に、SIDI Hubのイベントの聴衆について考察したいと思います。ベルリンでの政府関係者の出席が少なかったため、技術的な可能性について議論する技術的な聴衆に偏ったアジェンダとなりました。これは理想的ではありません。

先に記載した通り、法律や社会についても検討が必要です。ベルリンでは政府機関の設備で開催したにもかかわらず確かにあまり多くの政府関係者が参加したわけではありませんでした。この辺りは日本開催をする際のバランスに関する考慮点となるでしょう。

SIDI Hub was founded to unite global audiences to define the users, benefits, and overall business case for globally interoperable digital identity to normalize approaches and define minimum requirements. It was, therefore, somewhat premature to attempt a solution-oriented agenda. With that said, the lessons were valuable, and SIDI Hub has had valuable contributions from European stakeholders through other avenues, e.g., the SIDI Paris Summit, eIDAS 2.0 documentation, etc. Regardless, the SIDI organizers have determined that baseline government participation will be a critical go/no-go criterion for the events planned in Washington, D.C., Tokyo, and Brazil.

SIDI Hubは、世界中のオーディエンスをまとめ、世界規模で相互運用可能なデジタルIDのユーザー、利点、全体的なビジネスケースを定義し、アプローチを標準化し、最低限の要件を定義するために設立されました。そのため、ソリューション志向のアジェンダを試みるには時期尚早でした。とはいえ、そこから得られた教訓は貴重であり、SIDIハブは、SIDIパリサミットやeIDAS 2.0文書など、他の手段を通じて欧州の利害関係者から貴重な貢献を得ることができました。それでも、SIDIの主催者は、ワシントンD.C.、東京、ブラジルで計画されているイベントについては、政府の基本的な参加が実施の可否を決定する重要な基準となると判断しました。

ベルリンでもユースケースの取りまとめ要件整備を行いました。次のワシントンDCや東京・ブラジルでの開催に向けて議論をしていく必要がありそうです。なお、ここに記載がある通りソリューションとして自立させるためのきっかけには早すぎるイメージはありました。しかし読者の皆さんは気にせずにアプライしてくださいね。

22:30 • 22 days ago

Doc Searls Weblog

When Radio Delivers

For live reports on recovery from recent Hurricane Helene flooding, your best sources are Blue Ridge Public Radio (WCQS/88.1) and iHeart (WWNC/570 and others above, all carrying the same feed). Three FM signals come from the towers on High Top Mountain, which overlooks Asheville from the west side: 1) WCQS, 2) a translator on 102.1 for WNCW/88.7, and 3) a translator on 97.7 for WKSF/99.9’s HD-2 stream. At this writing, WCQS (of Blue Ridge Public Radio) and the iHeart stations (including WKSF, called Kiss Country) are running almost continuous public service coverage toward rescue and recovery. Hats off to them.

Helene was Western North Carolina‘s Katrina—especially for the counties surrounding Asheville: Buncombe, Mitchell, Henderson, McDowell, Rutherford, Haywood, Yancey, Burke, and some adjacent ones in North Carolina and Tennessee. As with Katrina, the issue wasn’t wind. It was flooding, especially along creeks and rivers. Most notably destructive was the French Broad River, which runs through Asheville. Hundreds of people are among the missing. Countless roads, including interstate and other major highways, are out. Towns and communities—Spruce Pine, Swananoa, Chimney Rock, Mitchell, Lake Lure, and many others—have been wiped away, or are in ruins. Roads across the region are gone, or closed. Electric, water, gas, sewer, and other utilities are expected to be down in many places for weeks.

One public utility that is working enough for people to keep up with the news is radio. Many (perhaps most) stations are off the air, but some survive, and are providing constant service to residents and people out of the area who want to stay informed. I recommend Blue Ridge Public Radio (WCQS/88.1) and any of the local iHeart stations . All of the iHeart stations listed in the image above are carrying the same continuous live coverage, which is excellent. (I’m listening right now to the WWNC/570 stream.)

Of course, there’s lots of information on social media (e.g. BlueSky, Xitter, Threads), but if you want live coverage, radio still does what only it can do. Yes, you need special non-phone equipment to get it when the cell system doesn’t work, but a lot of us still have those things. Enjoy the medium while we still have it.

Item: WWNC just reported that WART/95.5 FM in Marshall, with its studios in a train caboose by the river, is gone (perhaps along with much of the town—for more on that, go here).

I hope when this is over that iHeart keeps one of its six stations (in that image above) on the Helene recovery case. Or that it partners with BPR or the Asheville Citizen-Times on continuing live radio service to the region while recovery continues.

More sources:

WISE/1310 stream WTMT/105.9 stream

[Later, October 7, 2024…] Nice piece in the Asheville Citizen-Times about how iHeart’s WWNC stayed on the air and became a go-to place for constant updates, and human connection, during recovery from Helene.

[Later, October 20, 2024…] iHeart seems done with its wall-to-wall coverage of Helene and its aftermath. All of the stations in the image above have returned to their normal programming and schedules—or so I gather. (Please correct me if I’m wrong.) All their websites say “Listen For The Latest News As The Cleanup & Recovery Efforts Continue” with a link to the station’s stream page at iHeart.com. Meanwhile, Blue Ridge Public Radio remains on the case. Normal programming is mostly news and public affairs anyway, so recovery service is in their wheelhouse. On the BPR website is a list of Helene resources, live updates at https://www.bpr.org/live-updates/live-blog-hurricane-helene-wnc, After Helene text briefings, and an After Helene newsletter (to which I just subscribed). I also just learned that WPVM/103.7, broadcasting from atop the Public Service Building downtown, has also been on the case and sending PSAs out to listeners.

The original version of this post was cross-posted on Trunkli, my blog on infrastructure.

20:29 • 22 days ago

Jon Udell

Making a Vote Forward checklist

In How and why to write letters to voters I discussed Vote Forward, my favorite way for those of us who aren’t in swing states to reach out to voters in swing states. The site works really well for adopting batches of voters, and downloading packets of form letters. As I close in on 1000 … Continue reading Making a Vote Forward checklist

With 50 bundles in play, many of which are farmed out to friends and neighbors who are helping with the project, it’s become cumbersome to keep track of which bundles are prepped (ready to mail) or not. Here is the checklist I needed to see.

VoteForward Dashboard Report mmorg: 1-UNPREPPED r23Pp: 2-UNPREPPED v9Kbo: 3-UNPREPPED wLMPw: 4-UNPREPPED 24L4o: 5-PREPPED 4nNnj: 6-PREPPED 5rQmV: 7-PREPPED ... YV4dL: 48-PREPPED zKjne: 49-PREPPED ZrKJz: 50-PREPPED

If you’re in the same boat, here’s a piece of code you can use to make your own checklist. It’s gnarly, if you aren’t a programmer I advise you not even to look at it, just copy it, and then paste it into your browser to have it open a new window with your report.

Vote Forward checklist maker (expand to copy) javascript:(function(){ // First part: Adjust height of divs with inline styles document.querySelectorAll('div[style]').forEach(div => { let inlineStyle = div.getAttribute('style'); if (inlineStyle.includes('position: relative')) { div.style.height = '20000px'; // Set the height to 20000px } }); // Introduce a delay before processing the list of items setTimeout(() => { const items = document.querySelectorAll('li.bundle-list-item.individual'); let dataList = []; // Iterate over the items to capture data-testid and ID items.forEach(item => { let dataTestId = item.getAttribute('data-testid'); // Use the id attribute of the input element to extract the ID const toggleInput = item.querySelector('input.slide-out-toggle'); const toggleId = toggleInput ? toggleInput.getAttribute('id') : ''; // Extract the ID part from the toggleId pattern "toggle-24L4o-PREPPED" const id = toggleId ? toggleId.split('-')[1] : 'ID not found'; // Remove "bundle-" and the number part from dataTestId, keeping only "PREPPED" or "UNPREPPED" dataTestId = dataTestId.split('-').pop(); // Extract only the "PREPPED" or "UNPREPPED" part // Push the data into the array dataList.push({ dataTestId, id }); }); // Sort first by whether it's PREPPED or UNPREPPED (descending for UNPREPPED first), // then by the ID within each group dataList.sort((a, b) => { if (a.dataTestId.includes("PREPPED") && b.dataTestId.includes("UNPREPPED")) { return 1; // UNPREPPED comes before PREPPED } else if (a.dataTestId.includes("UNPREPPED") && b.dataTestId.includes("PREPPED")) { return -1; } // Sort by ID if they belong to the same category return a.id.localeCompare(b.id); }); // Prepare the output string let output = ''; dataList.forEach((item, index) => { output += `${item.id}: ${index + 1}-${item.dataTestId}\n`; }); // Open a new window with the output in a text area for easy copying let newWindow = window.open('', '', 'width=500,height=500'); newWindow.document.write('<html><body><h2>VoteForward Dashboard Report</h2><pre>' + output + '</pre></body></html>'); newWindow.document.close(); }, 2000); // Adjust delay as needed })();

Here are instructions for Chrome/Edge, Safari, and Firefox. You might need to tell your browser to allow the popup window in which it writes the report.

Chrome/Edge: Open the VoteForward dashboard in your browser. Open the developer console: Windows/Linux: Press Ctrl + Shift + J. Mac: Press Cmd + Option + J. Paste the code into the console. Press Enter to run the code. Firefox: Open the VoteForward dashboard in your browser. Open the developer console: Windows/Linux: Press Ctrl + Shift + K. Mac: Press Cmd + Option + K. Paste the code into the console. Press Enter to run the code. Safari: Open the VoteForward dashboard in your browser. Enable the developer console (if it’s not already enabled): Go to Safari > Preferences. Click the Advanced tab. Check “Show Develop menu in menu bar” at the bottom. Open the developer console: Press Cmd + Option + C. Paste the code into the console. Press Enter to run the code.

It would be nice to have this as a built-in feature of the site but, as we come down to the wire, this may be a helpful workaround.

Thanks, again, to the Vote Forward team for all you do! It’s a great way to encourage voter turnout.

18:08 • 23 days ago

@_Nat Zone

マイナ保険証のスマートフォン搭載：2025年春に実現へ

厚生労働省は、2025年春をめどにマイナ保険証のスマートフォン搭載を開始する計画を発表しました。この新システムは、AndroidとiPhoneの両方に対応する予定です。主なポイント：利用方法：この計画は、9月30日の16時00分から18時30分に開催された第183回社会保障…

厚生労働省は、2025年春をめどにマイナ保険証のスマートフォン搭載を開始する計画を発表しました。この新システムは、AndroidとiPhoneの両方に対応する予定です。

主なポイント：

導入時期：2025年春を予定対応機種：Android（グーグル）とiPhone（アップル）医療機関側の準備：新たな読み取り機器の設置が必要

利用方法：

患者は医療機関の顔認証付きカードリーダーでスマホ利用を選択専用リーダーにスマホをかざす画面上で情報提供に同意して手続き完了（出所）厚生労働省「マイナ保険証の利用促進等について」P.15 (2024-09-30)

この計画は、9月30日の16時00分から18時30分に開催された第183回社会保障審議会医療保険部会で公表されました。厚生労働省は、物理的なマイナンバーカードがなくても医療機関を受診できるようになるため、マイナ保険証の利用が促進されると期待しています。

今後、実証事業を通じて普及を目指していく方針です。

16:27 • 23 days ago

Sunday, 29. September 2024

IdM Laboratory

Ruby-SAMLの脆弱性（XML署名ラッピング攻撃）

こんにちは、富士榮です。
XML署名ラッピングによるRuby-SAMLの脆弱性が報告されていますね。
https://ssoready.com/blog/engineering/ruby-saml-pwned-by-xml-signature-wrapping-attacks/
CVE-2024-45409としても登録されています。 https://nvd.nist.gov/vuln/detail/CVE-2024-45409
内容としては割とオーソドックスなXML署名の実装の問題っぽいですね。
XML署名の特徴としてJWSとは違いドキュメント全体ではなく要素を指定して署名をおこなうことができる点、署名された値そのものも当該XMLの内部に埋め込まれることが挙げられます。今回は部分的な署名を行うことができるという点について悪用された感じですね。上記の例で言うと、真ん中にあるalice@customer.comの値を含むid=dead[....]beefの部分に対して署名値が生成される一方で、攻撃者が偽の値をXML内に埋め込んだ同じid=dead[...]beefの部分は署名されない、ということが起きてしまうわけです。
この攻撃は2012年にJPCERTが以下のペーパーを出していたり、と過去もしばしば起きている話なので、実装する際は先人の知恵に頼りながらやっていけるといいでしょう。参考）JPCERTの資料 https://www.jpcert.or.jp/securecoding/2012/No.10_Apache_Axis.pdf

22:30 • 23 days ago

dockがmDLのWebinarをやるようです

こんにちは、富士榮です。パスポートや免許証のApple Wallet/Google Walletへの格納の話も多く、世の中はすっかりmDoc祭りですね。そんな中、各社も色々イベントやセミナーを仕掛けてきているわけですが、VCやWalletの界隈ではそろそろ老舗？になりつつあるdockもmDLに関するWebinarをやるようです。 https://www.dock.io/live-event-mdls-are-coming 13 US states have already rolled out mobile digital driver's licenses (mDLs), and many more are testing the waters. Why the buzz? These government-issued digital ID

こんにちは、富士榮です。
パスポートや免許証のApple Wallet/Google Walletへの格納の話も多く、世の中はすっかりmDoc祭りですね。
そんな中、各社も色々イベントやセミナーを仕掛けてきているわけですが、VCやWalletの界隈ではそろそろ老舗？になりつつあるdockもmDLに関するWebinarをやるようです。

https://www.dock.io/live-event-mdls-are-coming

13 US states have already rolled out mobile digital driver's licenses (mDLs), and many more are testing the waters. Why the buzz? These government-issued digital IDs promise game-changing benefits: enhanced privacy, smoother online transactions, and a streamlined process for everything from opening a bank account to securing a loan. So, here's the real question: how will mDLs transform remote ID verification?

米国ではすでに13の州でモバイル・デジタル運転免許証（mDL）が導入され、さらに多くの州で試験運用が行われている。なぜ話題になっているのか？これらの政府発行のデジタルIDは、プライバシーの強化、よりスムーズなオンライン取引、銀行口座の開設からローンの確保までの合理化されたプロセスなど、ゲームチェンジャー的なメリットを約束している。
mDLは遠隔地でのID認証にどのような変革をもたらすのだろうか？

なかなか興味深いですね。

例によって日本時間だと10月3日（木）AM1:00-という酷い時間ですが、興味のある方は参加してみると米国の様子などわかるかもしれませんね。

00:00 • 24 days ago

Friday, 27. September 2024

Bill Wendels Real Estate Cafe

Talk to Compensation Coach before signing showing agreement to maximize savings

#LegalHackers: Know about the new role emerging at the start of real estate transactions because homebuyers are now being asked to sign an agreement before… The post Talk to Compensation Coach before signing showing agreement to maximize savings first appeared on Real Estate Cafe.

#LegalHackers: Know about the new role emerging at the start of real estate transactions because homebuyers are now being asked to sign an agreement before…

The post Talk to Compensation Coach before signing showing agreement to maximize savings first appeared on Real Estate Cafe.

00:48 • 26 days ago

Thursday, 26. September 2024

Wrench in the Gears

What If The Totalitarian Game Is Actually A Participatory Play? Musings on CogNarr, Eusociality, and Synanon

Yesterday I put together a 40-minute video in which I talk about Karl Friston’s “Free Energy Principle,” a theory developed in the 1990s that centers the unique narrative world models we hold internally and how those models in consciousness are shaped by sensory input, the incoming information hurtling towards us through the feed every day. I relate Friston’s work to a new effort by John Boik, a Houston-based bioengineer with a background in Oriental medicine who is now pursuing open source software in AI assisted direct democracy and community currency programs. If you’ve been following my work around Michael Levin’s research you can probably see the connection between Boik’s former career in cancer research and cooperative digital money systems – morphogenetic fields linked to expanding cognitive light cones.

In a recent presentation for the Active Inference Institute, based at UC Davis in Daniel Ari Friedman’s Entomology Lab (Deborah Gordon, Stanford ant behavior expert is an advisor), Boik laid out his plans for CogNarr, a system that would structure human “stories” in ways that would be intelligible to computers and facilitate collaboration at a scale of a half dozen people to several million. This parallels Thomas Seeley’s work on distributed decision-making described in his book “Honeybee Democracy.”

I am settling in here in Hot Springs and I met a lovely elder who placed in my hands a memoir written by her son who is around my age. It describes his educational journey as a divergent learner and his family’s decade-long involvement with the “utopian” community Synanon based in Marin County, CA with ties to influential group psychologists and likely an experiment tied to the Human Ecology / MK Ultra Project. The main campus was on the site of an old lodge built by Marconi for wireless telegraphy.

While founded as an addiction treatment program that used attack therapy, “The Game,” the community later attracted intellectuals who were interested in a collective “utopian” lifestyle. I believe Synanon was a testbed for exploring the opportunities and limits of a hive / colony-oriented communal behavior. In other words Synanon was an ant computer lab. I get the feeling that while a lot of media attention was paid to increasingly unhinged adults with weapon stockpiles, including a rattlesnake placed in a lawyer’s mailbox, the primary area of investigation may have been the children.

It’s funny, with the cooler weather I’ve been enjoying my sunporch, but I’ve found that I have to keep an eye on the sugar ants. There are regularly 3-4 foragers exploring my Ikea Kallax furniture. One afternoon I got preoccupied with sewing and left a plate of sliced apples unattended for a few hours. Yowza – pheromonic stigmergy in action! It was shocking, but also quite fascinating to see how the superorganism works up close and personal. This video that is part of the Santa Fe Institute podcast offers helpful insights into this dynamic: “The Physics of Collectives.” I keep pushing the idea of the ant computer in front of people, because we are trained into linguistic and conceptual boxes that fuel polarity – libertarianism vs. socialism, etc. etc. when most of the roads are all using custom stories to take us to the same place – Conway’s Game of Life, counting down the iterations until we get the gliders coming on.

Abraham Maslow was involved with Synanon as well as being affiliated with Julius Stulman, Fritz Kunz, and Ervin Laszlo via the Foundation for Integrated Education. Maslow contributed a chapter to Stulman and Laszlo’s book “Emergent Man.” Synanon shut its doors in the 1990s, but spun out other programs including an abusive pioneer in the “troubled teen” space CEDU. Paris Hilton is an alumna of that program. Rory Kennedy, Bobby’s youngest sister, produced a documentary on Synanon that was released this spring by HBO. I speculate that this documentary as well as last year’s Netflix expose on San Patrignano are pre-bunking efforts to neutralize resistance to the institutional “wellness,” addiction and mental health treatment programs Bobby has planned for rural areas across the United States. My gut tells me these centers, like refugee camps, will be testbeds for tokenized eusocial behavioral experiments leading to advancements in collective decision making using emerging technologies. Nothing new under the sun, only this time we have nano-photonics in the mix to make social signaling more seamless and intuitive.

Here’s my video:

There were some good questions and comments on this video. Given how those tend to disappear, I chose to transcribe one exchange that provides some additional insight into how I’ve been thinking about things lately. I know my theory may sound far-fetched, but remember, consciousness is a coherence engine. I’ve been pulling in A LOT of disparate information for years. Trying to weave it into a meaningful narrative is tricky. Certainly the plot line that seems to be unfolding is NOTHING I would have ever expected. I’m not saying I’m correct, but I think there is enough information of concern that my thoughts do merit public consideration with respect to ethics, informed consent, and guard rails. The past four years have given me a crash course in the subtle practice of knowledge management refined by well-trained and well-funded social scientists and psychologists. All that I’m asking is that we actually engage in information streams outside of the ones being poured down our throats by mainstream media, academic gatekeepers, and social media feeds. But at the end of the day, I am proposing a pretty far out story. It makes sense that it’s going to be near to impossible to get much traction. Anyway, if you want to immerse yourself in the information fields that brought me to this place, I have linked a ton of my Kumu.io maps. Click away. I’m eager to hear what you come up with.

Commenter One:

“What I see is the constant reinforcement of being in mind, abdication of physical being, being in your body and developing a relationship with our humanness is the most important connection. I wonder if we stay truly connected to our human-manifestation would that stop people going automatic and getting lost with narrative.”

Commenter Two:

“This right here. There aro a whole slew of us that have been guided towards esoteric info to the point that we’ve started commenting “who’s here in 24?” To 10 plus year old videos. They think the algorithm has discovered that people really like this….I just think this is a test group of heavy users of you tube. A particular…group of people who would be prone to action in the physical realm. It’s a test for how well it neutralizes them. Doesn’t matter to them if it’s real or empowering or helpful or healing….just does it prevent action by a physical being and bend them to spread denying the physical. Many in that group have kids…many may be teaching their children to place the physical 2nd behind mental. And if that’s the case – why not climb into the box and never leave?”

Commenter Three:

“I would be curious to know more about this. So, to clarify, YouTube is pushing videos on a group of people to see if they will remain docile? I’ve often wondered about certain spiritual viewpoints, and if they serve to keep people from sticking up for themselves. Any specifics you have on this subject would be fascinating to me, thanks.”

Alison’s Comment:

“Robert Bosnak’s take is that apocalypse narratives are quite generative. I do see the ascension storyline as running parallel with the Christian imminent apocalypse storyline. Based on experiences I’ve had over the past several years as well as natural cycles, I sense that we don’t have a grasp of what spacetime really is and that it can loop back and feature recursion. If that is the case, the linear narratives of some transformative end point seem likely socially constructed for some purpose I can’t conjecture about. I’m skeptical of both. If you try on the theory I’ve been developing around heart-soul-based nested and networked human massive parallel processing with access to the field through quantum biology and companion microbiome management, then the system would need two phases – a foraging out-of-body ego-death phase where individuals or groups go into the field to retrieve information which is then processed in-vivo with support from bacterial colonies that live in and on us. I sense that both are required, but training and practice will be needed so people can do it as second nature.

Again, in both apocalypse narratives, the storyline and teaming aspect is super important. Consider the industries that have been built up around both from New Age TV and webinars and books, etc. to Christian megachurches and of course the Vatican’s deep dive into social impact and biotech and Oral Roberts setting up Bible verse magic AI mirrors in student dorms. It never occurred to me that “social impact” was meant to be a hub for ongoing remote cognitive-biological monitoring (now for “wellness”) that will enable society to become a noetic superorganism available for spiritual on-demand information processing. Just like I believe the Georgia Guidestones were an engineered memetic trope to focus negative energy and fear. I also think the apocalypse stories are meant to structure consciousness so that when the intended program rolls out it can be embedded in a dramatic arc each particular groomed group is already comfortable with. The question though…is it a cookbook?”

Commenter One Follow Up:

“So with the embodied imagination, which sounds a bit like shamanic journeying, it is the mind which is the ‘driver’ to facilitate this third state which is embodied. In the mirror’d virtual world it is presumably still the original mind inhabiting this new embodiment? I wonder also what happens to the energy field of the actual heart & body in the new embodiment. Is this measured by the galvanic response also? Narrative means each group (being coalesced) can happily volunteer to inhabit this state and the body can then be harnessed and utilised.”

Alison’s Reply:

“I just wanted to state I am working through this theory and don’t have everything. buttoned up – but as I picture it, it would be the physical body and etheric body working in tandem to facilitate the information processing – two cycles. So picture a bee scouting a site for a new hive. Bees do this “waggle dance” to communicate what they’ve found and make a case to their peers about why their location is good. Then, by consensus (think tokenized democracy) the bees come to a decision. So maybe we are using our etheric self to forage for information in the field, and we bring it back to “the collective,” but that collective isn’t a hive of bees, but rather a cosmos of microbiota and mitochondria who communicate not through dance vibrations, but perhaps through photonics in the fascia? Now picture all of us as agents linked in a parallel processing system via frequency facilitated by electroceuticals and liposomes, etc. in our “wellness” supplements. I’m kind of spitballing here and pardon me if I’m misinterpreting your framing, but I don’t think it’s – hey send your consciousness out into the ether and let the system use your physical body while you are gone. There is the issue of embodiment, narrative coherence, umwelt, and qualia. My logical self says that individuals need to maintain their integrity as agents over time in order to be productive in the collective decision making process. I think they want to be able to nest individual experiences and then aggregate them at a scale relevant to the particular “cognitive light cone” task. Think about ants – they are agents working within a social structure that facilitates complex emergence even if the individual ants have no clue what is going on at levels about their pay grade. As I have been researching Julian Gresser’s “intelligent heart” it seems that the torus field is useful for processing 4-D information. So that may be another reason, beyond the role of the microbiome, to integrate knowledge acquired through astral projection – to integrate the information in a space outside regular 3-D existence.

Also, Bosnak is very hip on group dreaming in VR “Asclepian” healing caves. My intuition tells me that tokens and the protocol layer could be an interdimensional language of manifestation. This paper shared with me today by Stephers has some interesting thoughts on consciousness and projecting geometry (sacred?). https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2020.01680/full “

Interactive Map: https://embed.kumu.io/745bb8ce3aad470199efea9e9528807d#untitled-map?s=bm9kZS1aUERrT3Vxbg%3D%3D

I mention psychodrama and Gurdjieff in the video. Here is a map I’ve been working on related to this:

Interactive map JG Bennet and Gurdjieff’s “The Work”: https://embed.kumu.io/0bf33678ed9f4126e528c09beb40370e#untitled-map?s=bm9kZS1RdE5jbkJ0Tg%3D%3D

This map links Maslow’s theories of self-actualization with Ocean Protocol (Trent McConaghy and Jim Rutt’s Game B), token engineering, game mechanics, sustainability, and AGI:

Interactive Map – Maslow Self-Actualization: https://embed.kumu.io/4011b2b1901acf8b24cbb1119bbb9e1c#untitled-map?s=bm9kZS1ObjBud3VEdg%3D%3D Source: https://embed.kumu.io/1ff16f690770c1d381e267f258358a32#untitled-map?s=bm9kZS1mVjhCN2ZOUw%3D%3D Source: https://archive.org/details/emergentmanhisch00stul/mode/2up Source: https://archive.org/details/emergentmanhisch00stul/mode/2up Source: https://archive.org/details/emergentmanhisch00stul/mode/2up

Source: https://embed.kumu.io/51fe5ee0c16765f2c748936cc86c28ee#untitled-map?s=bm9kZS1JbG83a1dOZQ%3D%3D Source: https://web.archive.org/web/20210531074023/https://littlesis.org/oligrapher/6930-arizona-state-games-and-education Source: https://embed.kumu.io/035ea6860f513bab36fa1a9815d93a74#untitled-map?s=bm9kZS1ENDF4aWZZZA%3D%3D Source: https://embed.kumu.io/d393e288b03d6a5ef05fc9bc6e51a6a6#untitled-map?s=bm9kZS05SndwcVNqQg%3D%3D Source: https://embed.kumu.io/f77a772b1d3ed24cf4140d0a3d270348#untitled-map?s=bm9kZS1HR0M1SWphVg%3D%3D Source: https://principledsocietiesproject.org/about_psp

Interactive map: https://embed.kumu.io/f662cd5969e9debc347b10601a5e3282#untitled-map?s=bm9kZS1Tc2dLUmd3Uw%3D%3D Source: https://embed.kumu.io/097cefb708139ffa65e906e397da3824#untitled-map?s=bm9kZS1hVzVyR1FaeA%3D%3D Source: https://embed.kumu.io/89566f995693843319e1b4970dac9dd1#untitled-map?s=bm9kZS1VSk9pY0JYTA%3D%3D

Lee Eyre, Boik’s collaborator on these papers, worked for Orange (ambient smart environments) for ten years. The video below is how the world looks when everything is run on a permission-based smart contract. This scenario presumes all agents have a digital identifier, which brings in Web3, blockchain, and spatial computing. Watch this with an eye towards interactions feeding into a personal story – qualia, umwelt.

LEDDA (Leda) is an interesting name / acronym choice here.

Source: https://www.youtube.com/watch?v=z9ZCjd2rqGY Source: https://www.researchgate.net/lab/Active-Inference-Institute-Daniel-Ari-Friedman Screenshot

Source: https://embed.kumu.io/f03f97508c7993eef8aefa20cd265e98#untitled-map?s=bm9kZS1SUU1ZTkdvdA%3D%3D

21:31 • 26 days ago

The Pragmatic Engineer

The Pulse #108: Elasticsearch unexpectedly goes open source again

Also: Amazon cuts the number of managers, engineers critiquing YouTube’s biggest critic, and key people keep quitting OpenAI.

The Pulse is a series covering insights, patterns, and trends within Big Tech and startups. Notice an interesting event or trend? Send me a message.

Today, we cover:

Industry pulse. Meta previews next-gen AI glasses, X probably needs to pay severance to engineers it laid off, on-prem datacenter threat for AWS, tech jobs drying up, Arc’s embarrassing secur…

16:23 • 27 days ago

Wednesday, 25. September 2024

The Pragmatic Engineer

AI tools for software engineers, but without the hype – with Simon Willison (co-creator of Django)

Simon is one of the best-known software engineers experimenting with LLMs to boost his own productivity: he’s been doing this for more than three years, blogging about it in the open.

The first episode of The Pragmatic Engineer Podcast is out. Expect similar episodes every other Wednesday. You can add the podcast in your favorite podcast player, and have future episodes downloaded automatically.

Listen now on Apple, Spotify, and YouTube.

Brought to you by:

• Codeium: Join the 700K+ developers using the IT-approved AI-powered code assistant.

• TLDR: Keep up with tech in 5 minutes

—

On the first episode of the Pragmatic Engineer Podcast, I am joined by Simon Willison.

Simon is one of the best-known software engineers experimenting with LLMs to boost his own productivity: he’s been doing this for more than three years, blogging about it in the open.

Simon is the creator of Datasette, an open-source tool for exploring and publishing data. He works full-time developing open-source tools for data journalism, centered on Datasette and SQLite. Previously, he was an engineering director at Eventbrite, joining through the acquisition of Lanyrd, a Y Combinator startup he co-founded in 2010. Simon is also a co-creator of the Django Web Framework. He has been blogging about web development since the early 2000s.

In today’s conversation, we dive deep into the realm of Gen AI and talk about the following:

Simon’s initial experiments with LLMs and coding tools

Why fine-tuning is generally a waste of time—and when it’s not

RAG: an overview

Interacting with GPTs voice mode

Simon’s day-to-day LLM stack

Common misconceptions about LLMs and ethical gray areas

How Simon’s productivity has increased and his generally optimistic view on these tools

Tips, tricks, and hacks for interacting with GenAI tools

And more!

I hope you enjoy this epsiode.

In this episode, we cover:

(02:15) Welcome

(05:28) Simon’s ‘scary’ experience with ChatGPT

(10:58) Simon’s initial experiments with LLMs and coding tools

(12:21) The languages that LLMs excel at

(14:50) To start LLMs by understanding the theory, or by playing around?

(16:35) Fine-tuning: what it is, and why it’s mostly a waste of time

(18:03) Where fine-tuning works

(18:31) RAG: an explanation

(21:34) The expense of running testing on AI

(23:15) Simon’s current AI stack

(29:55) Common misconceptions about using LLM tools

(30:09) Simon’s stack – continued

(32:51) Learnings from running local models

(33:56) The impact of Firebug and the introduction of open-source

(39:42) How Simon’s productivity has increased using LLM tools

(41:55) Why most people should limit themselves to 3-4 programming languages

(45:18) Addressing ethical issues and resistance to using generative AI

(49:11) Are LLMs are plateauing? Is AGI overhyped?

(55:45) Coding vs. professional coding, looking ahead

(57:27) The importance of systems thinking for software engineers

(1:01:00) Simon’s advice for experienced engineers

(1:06:29) Rapid-fire questions

Some takeaways:

If you are not using LLMs for your software engineering workflow, you are falling behind. So use them! Simon outlined a bunch a of reasons that hold back many devs from using these tools – like ethical concerns, or energy concerns. But LLM tools are here to stay, and those who use them get more productive.

It takes a ton of effort to learn how to use these tools efficiently. As Simon puts it: “You have to put in so much effort to learn, to explore and experiment and learn how to use it. And there's no guidance.” Also, in related research we did in The Pragmatic Engineer about AI tools, with about 200 software engineers responding, we saw some similar evidence. Those who have not used AI tools for 6 months, were more likely to be negative in their perception of these. In fact, a very common feedback from engineers not using these tools was “I used it a few times, but it didn’t live up to my expectations, and so I’m not using it any more”

Use local models to learn more about LLMs. Running local models has two bigger benefits:

Tou figure out how to do these! It’s less complicated than one would think, thanks to tools like HuggingFace. Go and play around with them, and try out a smaller local model.

You learn a LOT more about how LLMs work, thanks to local models being less capable. So it feels less “magic”. As Simon said, “ I think it's really useful to have a model hallucinate at you early because it helps you get that better mental model of, of, of what it can do. And the local models hallucinate wildly.”

Where to find Simon Willison:

• X: https://x.com/simonw

• LinkedIn: https://www.linkedin.com/in/simonwillison/

• Website: https://simonwillison.net/

• Mastodon: https://fedi.simonwillison.net/@simon

Referenced:

• Simon’s LLM project: https://github.com/simonw/llm

• Jeremy Howard’s Fast Ai: https://www.fast.ai/

• jq programming language: https://en.wikipedia.org/wiki/Jq_(programming_language)

• Datasette: https://datasette.io/

• GPT Code Interpreter: https://platform.openai.com/docs/assistants/tools/code-interpreter

• Open Ai Playground: https://platform.openai.com/playground/chat

• Advent of Code: https://adventofcode.com/

• Rust programming language: https://www.rust-lang.org/

• Applied AI Software Engineering: RAG: https://newsletter.pragmaticengineer.com/p/rag

• Claude: https://claude.ai/

• Claude 3.5 sonnet: https://www.anthropic.com/news/claude-3-5-sonnet

• ChatGPT can now see, hear, and speak: https://openai.com/index/chatgpt-can-now-see-hear-and-speak/

• GitHub Copilot: https://github.com/features/copilot

• What are Artifacts and how do I use them?: https://support.anthropic.com/en/articles/9487310-what-are-artifacts-and-how-do-i-use-them

• Large Language Models on the command line: https://simonwillison.net/2024/Jun/17/cli-language-models/

• Llama: https://www.llama.com/

• MLC chat on the app store: https://apps.apple.com/us/app/mlc-chat/id6448482937

• Firebug: https://en.wikipedia.org/wiki/Firebug_(software)#

• NPM: https://www.npmjs.com/

• Django: https://www.djangoproject.com/

• Sourceforge: https://sourceforge.net/

• CPAN: https://www.cpan.org/

• OOP: https://en.wikipedia.org/wiki/Object-oriented_programming

• Prolog: https://en.wikipedia.org/wiki/Prolog

• SML: https://en.wikipedia.org/wiki/Standard_ML

• Stabile Diffusion: https://stability.ai/

• Chain of thought prompting: https://www.promptingguide.ai/techniques/cot

• Cognition AI: https://www.cognition.ai/

• In the Race to Artificial General Intelligence, Where’s the Finish Line?: https://www.scientificamerican.com/article/what-does-artificial-general-intelligence-actually-mean/

• Black swan theory: https://en.wikipedia.org/wiki/Black_swan_theory

• Copilot workspace: https://githubnext.com/projects/copilot-workspace

• Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems: https://www.amazon.com/Designing-Data-Intensive-Applications-Reliable-Maintainable/dp/1449373321

• Bluesky Global: https://www.blueskyglobal.org/

• The Atrocity Archives (Laundry Files #1): https://www.amazon.com/Atrocity-Archives-Laundry-Files/dp/0441013651

• Rivers of London: https://www.amazon.com/Rivers-London-Ben-Aaronovitch/dp/1625676158/

• Vanilla JavaScript: http://vanilla-js.com/

• jQuery: https://jquery.com/

• Fly.io: https://fly.io/

Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email podcast@pragmaticengineer.com.

14:06 • 28 days ago

Tuesday, 24. September 2024

The Pragmatic Engineer

Software engineers training software engineers

What is it like to teach software engineers, full time? Reuven M. Lerner has done this for 15 years, and shares his hands-on learnings – including how to teach efficiently

Did you ever consider becoming a teacher of software engineers? I’m assuming many of us have not – simply because it’s an uncommon career path, and teaching rarely feels likely to be lucrative, compared to hands-on building (we previously covered Common engineering career paths as Big Tech and scaleups). But teaching software engineers is an interesting challenge for a few reasons:

Many engineers are good at learning by themselves, so may initially assume there’s little value in being taught by others

But, great teachers make a real difference in getting up to speed, including for software engineers

There’s demand at tech companies for innovative teaching approaches and new technologies for devs

To discover what being a full-time trainer of tech professionals is really like, I turned to software developer turned full-time trainer, Reuven M. Lerner.

Reuven worked as a developer for 15 years, and for the past decade and a half he's been a full-time instructor. He teaches Python, Pandas, and Git for a range of companies, including Apple, IBM, and Cisco. He does both corporate training, as well as online Python courses for individuals.

Today, Reuven takes us behind the scenes of technical training, covering:

Coding vs teaching it. You optimize software when coding, whereas with training you optimize how to best teach the writing of software.

Is training helpful? Many software engineers learn by themselves, and it can be hard to get dedicated time at work for training. But group courses boost communication across tech teams.

What makes effective teaching? Reuven’s thoughts, including on using interactive notebooks over slides, exercises above theory, and lots of pair programming.

Day to day. Teaching learners is just one part of being a trainer: sales, marketing, customer support, and continuous learning are also key.

Business of teaching. To work as a career, teaching must be a solid business. Reuven shares how he runs his operation, from closing new clients and educating decision makers, to collecting payment.

Advice for future trainers. Get comfortable with public speaking, go deep into a technology, don’t shy away from niches, and more.

With that, it’s over to Reuven:

When I got my computer science degree 30 years ago, I knew what my career would look like: I would develop software, eventually move up to manage other people, or maybe start my own business. Indeed, after writing software for Hewlett Packard and Time Warner’s “Pathfinder” website, I opened my own consulting shop, writing Web applications and running Linux-based servers.

Fast forward to today, and my career looks very different. I’m still self employed, but instead of developing software, I’m a full-time trainer in Python and Pandas. I teach at big companies like Apple, Arm, Cisco, and Western Digital, and at startups and financial institutions. I offer more than 30 courses, ranging from “Python for non-programmers,” and “Data analysis with Pandas,” to advanced practice workshops. Between these, I have a growing business of online courses and newsletters for people without access to company training programs.

I feel like I have the best of all worlds: I help people improve their careers, learn new technologies, and interact with smart people all over the world. Plus, I set my own schedule far in advance, have only a handful of meetings a month, spend time with my family, and get paid well — better, in fact, than many developers. I’ve never earned more, and I’ve never enjoyed my work more.

In this post, I introduce the world of tech training. I reveal how it operates, what I’ve found does (and doesn’t) work for training, how I run my business, and how you can explore the world of training.

How I became a full-time trainer

When I started consulting in 1995, I positioned myself as a coder and Linux expert. But some companies asked me not to develop software for them, but to teach their people how to do it. That was my first taste of training and I rather liked it, but saw it as just one part of my consultancy work. Indeed, I rarely spent more than 20 percent of my time on training.

In 2003, I started a PhD program, continuing to consult part-time in order to support my family. While working on my dissertation, a colleague suggested I concentrate on training, and offered to connect me with a company. I said yes – a decision which changed my career.

This training company marketed my Python courses, and filled up my calendar with training sessions. Soon, my schedule was full several months in advance. As convenient as it was to work with them, I also knew that they were keeping half the income.

When I finished my PhD in 2014 (after 11 years!) I left the training company and rebranded myself as a trainer. I’ve now been teaching Python, Pandas, and Git full time for around 15 years and absolutely love it.

My focus on Python turned out to be fortunate because it is used just about everywhere. Even hardware companies that mainly work in C, like Apple, Arm, and Western Digital, use Python on all sorts of internal testing and analysis projects. Financial institutions are moving to Python instead of Excel, and want help in making the switch. Companies doing numerical analysis with Matlab are tiring of the high per-seat licensing cost, and are moving to Python – and need help easing employees into a new environment.

I mostly teach people who are highly schooled and very smart, many of whom have engineering degrees and at least some experience of coding. In theory, their employer could buy them books or video courses, and ask them to learn Python solo. In practice, we all know this doesn’t work; we’re often too busy to use such materials. A timeboxed course, delivered in person and with everyone in the same place is the fastest option with the best results, and it helps establish best practices, instead of just learning the syntax.

1. Coding vs teaching it

How is my life and work different as a trainer, than as a coder? Some of the biggest differences:

As a trainer, my goals are fundamentally different from a full-time software engineer’s. A coder’s goal is to get new or improved functionality out the door. In contrast, my job is to help someone do their job better and faster by writing more idiomatic, maintainable, and efficient code quicker.

I spend much of my time thinking about code. However, I do not do this in the same way I did when working on software projects. I’m not trying to optimize software; I’m trying to optimize learning about writing software. I always seek to simplify and improve my explanations, and find stories, metaphors, and examples that improve my teaching. I’m constantly trying to understand how certain packages and techniques work, so I can explain and illustrate them better to students.

In many ways, I’m like a stand-up comedian. I teach so often, so I see which examples, explanations and exercises work, and which don’t. Just as a comedian changes their jokes from show to show and iterates repeatedly until they find what works, I’m constantly experimenting with what and how I teach, trying to find the optimal way to get information across.

I particularly enjoy using stories in my teaching. Good stories reinforce the ideas being taught, and also enliven classes on potentially dry, abstract topics.

Often, these stories come from personal experience. One recent example: Meta banned me from advertising my courses and newsletters on their platforms, apparently because they believe I was illegally trading in exotic animals (pythons and pandas – the irony!) This event was widely discussed on programming forums like Hacker News.

Python (left) vs a python (right.) Facebook doesn’t allow adverts for Python courses because they assume you’re selling serpents! Read more about this incident.

This was as bizarre and frustrating as it was amusing, but you can be sure I’ll tell this story every time I teach a course on machine learning, and the need to test models before deploying them to production.

When I was doing software projects, it was hard to set my schedule in advance. Typically, someone needs a software project done now, or they don’t want it at all. Talking to someone about a project six months hence is generally a non-starter.

By contrast, there’s almost never a training emergency. As such, training can be scheduled two, four, or even six months in advance. At the time of writing, I already have courses in my schedule for January 2025, and I’m talking to clients about scheduling beyond that.

This ability to plan ahead has improved my personal life and my business. I can now schedule vacations knowing when I will have training gigs. I also have a much better sense of how much I’ll earn in a given month; a much better situation than the feast-or-famine roller coaster of my first years of freelancing.

Shock news: training can pay far better than coding! On the topic of money, here’s a lesser-known detail about training I’ve experienced: It pays better, often far better, than coding because:

If you help 20 developers to become 10 percent more effective, that’s worth a lot of money. So it’s absolutely worthwhile for a company to invest in good, effective training.

The budget doesn’t come from R&D. Rather, it comes from HR, or from a special training budget. Whereas a company might balk at paying thousands of dollars per day for a developer, this is considered to be a normal rate for training services!

Training is usually done through companies with overheads like offices and employees in sales/marketing. A freelancer doesn’t have these costs. Companies will pay roughly the same for training regardless of the training vendor’s size and overheads. I’m a one-person company based in a home office, so I can basically pocket what other companies spend on their costs!

Hardly any meetings. This is another major difference between doing coding and providing training. I’ll typically speak with a new client two or three times before the first class takes place, and maybe once after the first session to wrap things up. But if they ask me to teach again, we just exchange some email, mainly about dates. If I have 4-5 meetings a month, that’s a lot – which means I can spend more time teaching and developing new course materials.

I do miss software projects. I’ve experienced first-hand that there’s nothing like pushing technological boundaries and launching a product, knowing that people around the world are using and enjoying it. And there’s a definite limit to the size and scope of things I can do on my own, rather than in a larger team.

That said, most projects I worked on weren’t pushing boundaries. And while many were exciting, completing them didn’t give me the same sense of purpose and fulfillment I get from teaching. Besides, now I get to write whatever code I want – and there is definitely code to write, whether as part of my courses or running the backend of my online store and newsletters.

My online store’s tech stack combines:

Podia: a SaaS where my video courses live

WooCommerce: an e-commerce SaaS handling payment and subscriptions

Drip: email marketing SaaS, used for two of my newsletters and marketing blasts. I use a fair amount of custom programming (“workflows”) here

Ghost: a CRM and email service used for Bamboo Weekly

GitHub: I create a new repo for each course I teach

Flask: a Python framework I run on a standalone server for one-time coupon codes

Discord: used for discussion among my members.

Zapier: an integrations platform I use to connect these systems. For example, someone subscribing to my Python+Data product is enrolled in all my courses, added to my Better Developers list, and is added to the appropriate GitHub repos.

Custom Python scripts: These help me set up and tear down environments when I give corporate training. Each class gets a new GitHub repo, as well as its own set of Jupyter notebooks. This, along with the “gitautopush” package, lets me work on my own computer and share the course contents with participants in a given course in near-real time.

Do I plan to consolidate these into a smaller number of services? Yes, absolutely. But one person can only do so much in a day. Between teaching, writing three weekly newsletters, responding to learners and researching new topics, I don’t have much time for major technological shifts. But I do have a roadmap; for example, I’ll soon move discussions from Podia to Discord, which seems to foster a greater sense of community.

2. Is training helpful?

I once met someone with a background in engineering and education. I told him what I did and he replied:

“Oh, so you’re an entertainer? Because we both know that you’re not giving any real educational value.”

This comment hurt. Still, I’m sure many developers who attend my classes also believe they could learn the same material as quickly and as well by themselves, and that my courses are a nice vacation from “real” work. I understand this, but here’s what I’ve learned from years of teaching.

Most people benefit from having someone explain things, including developers who could learn on their own! After I gave a talk at PyCon US this year, a developer told me my presentation answered questions they didn’t even know they wanted to ask.

I spend a lot of time thinking about the questions people might have beyond simple use cases and syntax, and I integrate them into my teaching. People could get these insights themselves, but it would take longer and not necessarily be contextualized appropriately.

Pressure at work stops many developers learning new things by themselves. One client of mine decided to save money and bought my video courses for their staff. When I came in to do a live Q&A based on the videos, the only person who had really watched them had red eyes, because he had finished at 2:30 a.m. In the end, we returned to in-person lectures.

Learning the nuances of a language is faster with an instructor. Python is a good example; I’m often told this language has such simple syntax that a course isn’t really needed, and it is true the language is pretty simple, with just a few core data structures. So how long can it really take for an engineer to figure it alone?

This argument is similar to saying chemistry is simple because there are only 118 chemical elements in the universe. Learning the nuances, use cases, limitations, and conventions takes time. This is as true for Python as for chemistry. Going beyond basic syntax is usually faster and more memorable with an instructor.

For example, when I teach Python I dive into the details of the += operator. I explain that even though it does what you expect, one should be careful when using it to concatenate strings. If preserving memory is important, then you should always use a combination of a list and the str.join method to conserve memory. I talk about the different ways to iterate over a dictionary, and why using the dict.keys method is almost always a bad idea. We discuss the difference between the “__str__” and “__repr__” methods, and when to use each (and why I think it’s OK to only define “__repr__”).

Having everyone take a course can improve workplace communication. If people learn solo they’ll understand different things, and choose their own styles/conventions. Giving the same training across a company ensures everyone has the same (or similar) skill levels and understanding, making communication easier within and across teams.

Hands-on exercises are the most efficient way I know how to teach. I’ve fine-tuned coding exercises over years to illuminate certain techniques, syntax, and conventions. I call these exercises “controlled frustration.” The goal is to solve a problem without a manager or deadlines adding to stress levels.

Learning from other people's mistakes is a great way to learn and in a group setting, this is much easier. As important as it is for students to do exercises, it’s also important to review the exercises together and learn from each other’s buggy code. Also, when I demonstrate how to solve a problem, I’m modeling a process they can apply to their jobs.

Companies rarely give people time to pick up new techniques and technologies. It is true there are plenty of developers who can learn on their own. The trouble is finding dedicated time to focus on learning. I’ve found people often enjoy being in advanced classes – especially exercise-only classes – where they can solve interesting problems they might not have the opportunity to do at work.

As a manager, when does it make sense to consider bringing in a trainer? If your team is adopting a new technology, or if you’re all a little shaky with using it, or you observe devs always going to ChatGPT (or StackOverflow – if you still use it!) to solve problems, then you might want to consider bringing in an instructor. A good instructor with plenty of experience can anticipate which mental models help engineers, and has exercises to take their understanding to the next level.

Also, training empowers members of staff; improving their communication skills and distributing knowledge across organizations. Six months after I taught a Git course at one company, an engineer told me he was now the Git expert in his group, and no longer had to guess what to do when they got in trouble. Not only did he feel great about himself and this new knowledge, but his group benefited from having a local expert.

3. What makes effective teaching?

16:18 • 29 days ago

Monday, 23. September 2024

Damien Bod

Implement a Geo-distance search using .NET Aspire, Elasticsearch and ASP.NET Core

This article shows how to implement a geo location search in an ASP.NET Core application using a LeafletJs map. The selected location can be used to find the nearest location with an Elasticsearch Geo-distance query. The Elasticsearch container and the ASP.NET Core UI application are setup for development using .NET Aspire. Code: https://github.com/damienbod/WebGeoElasticsearch Setup For […]

Code: https://github.com/damienbod/WebGeoElasticsearch

Setup

For local development, .NET Aspire is used to setup the two services and the HTTPS connections between the services. The services are configured in the Aspire AppHost project .

The Elasticsearch client is setup as a singleton and requires the connection configuration. This can be changed, if for example an API key is used instead. The connection URL is read from the configuration as well as the secrets.

using Elastic.Clients.Elasticsearch; using Elastic.Transport; namespace WebGeoElasticsearch.ElasticsearchApi; public class ElasticClientProvider { private readonly ElasticsearchClient? _client = null; public ElasticClientProvider(IConfiguration configuration) { if (_client == null) { var settings = new ElasticsearchClientSettings(new Uri(configuration["ElasticsearchUrl"]!)) .Authentication(new BasicAuthentication(configuration["ElasticsearchUserName"]!, configuration["ElasticsearchPassword"]!)); _client = new ElasticsearchClient(settings); } } public ElasticsearchClient GetClient() { if (_client != null) { return _client; } throw new Exception("Elasticsearch client not initialized"); } }

Create Index with mapping

The index cannot be created by adding a document because the mapping is created incorrectly using the default settings. The mapping can be created for the defined index using the Mappings extension from the Elastic.Clients.Elasticsearch Nuget package. This was added to the client project in the Aspire.Elastic.Clients.Elasticsearch package. The mapping is really simple and probably not complete for a production index, some keyword optimizations are required. The detailsCoordinates field is defined as a GeoPointProperty.

var mapping = await _client.Indices.CreateAsync<MapDetail>(IndexName, c => c .Mappings(map => map .Properties( new Properties<MapDetail>() { { "details", new TextProperty() }, { "detailsCoordinates", new GeoPointProperty() }, { "detailsType", new TextProperty() }, { "id", new TextProperty() }, { "information", new TextProperty() }, { "name", new TextProperty() } } ) ) );

The created mapping can be validated using the “IndexName”/_mapping GET request. This returns the definitions as a Json response.

https://localhost:9200/mapdetails/_mapping

Documents can be added to the Elasticsearch index using the IndexAsync method.

response = await _client.IndexAsync(dotNetGroup, IndexName, "1");

Search Query

A Geo-distance query is used to find the distance from the selected location to the different Geo points in the index. This using latitude and longitude coordinates.

public async Task<List<MapDetail>> SearchForClosestAsync( uint maxDistanceInMeter, double centerLatitude, double centerLongitude) { // Bern Lat 46.94792, Long 7.44461 if (maxDistanceInMeter == 0) { maxDistanceInMeter = 1000000; } var searchRequest = new SearchRequest(IndexName) { Query = new GeoDistanceQuery { DistanceType = GeoDistanceType.Plane, Field = "detailsCoordinates", Distance = $"{maxDistanceInMeter}m", Location = GeoLocation.LatitudeLongitude( new LatLonGeoLocation { Lat = centerLatitude, Lon = centerLongitude }) }, Sort = BuildGeoDistanceSort(centerLatitude, centerLongitude) }; searchRequest.ErrorTrace = true; _logger.LogInformation("SearchForClosestAsync: {SearchBody}", searchRequest); var searchResponse = await _client .SearchAsync<MapDetail>(searchRequest); return searchResponse.Documents.ToList(); }

The found results are returned sorted using the Geo-distance sort. This puts the location with the smallest distance first. This is used for the map display.

private static List<SortOptions> BuildGeoDistanceSort( double centerLatitude, double centerLongitude) { var sorts = new List<SortOptions>(); var sort = SortOptions.GeoDistance( new GeoDistanceSort { Field = new Field("detailsCoordinates"), Location = new List<GeoLocation> { GeoLocation.LatitudeLongitude( new LatLonGeoLocation { Lat = centerLatitude, Lon = centerLongitude }) }, Order = SortOrder.Asc, Unit = DistanceUnit.Meters } ); sorts.Add(sort); return sorts; }

Display using Leaflet.js

The ASP.NET Core displays the locations and the results of the search in a Leafletjs map component. The location closest to the center location is displayed differently. You can click around the map and test the different searches. The data used for this display is powered using the Geo-distance query.

Testing

The applications can be started using the .NET Aspire host project. One is run as a container, the other is a project. The docker container requires a Desktop docker installation on the host operating system. When the applications started, the containers need to boot up first. An optimization would remove this boot up.

Notes

Using Elasticsearch, it is very simple to create fairly complex search requests for your web applications. With a bit of experience complex reports, queries can be implemented as well. You can also use Elasticsearch aggregations to group and organize results for data analysis tools, reports. .NET Aspire makes it easy to develop locally and use HTTPS everywhere.

Links

https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-point.html

https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-geo-distance-query.html

https://leafletjs.com/

https://www.elastic.co/guide/en/elasticsearch/reference/current/explicit-mapping.html

Using Elasticsearch with .NET Aspire

08:56 • 30 days ago

Sunday, 22. September 2024

Kent Bull

KERI Series: Understanding Self-Addressing Identifiers (SAID)

What is a self addressing identifier, a SAID? What does this mean and how is a SAID created and verified? This post answers these questions. We show a generalized process for calculating SAIDs and delve into the encoding format for CESR-compliant self addressing identifiers. Examples with three popular algorithms, SHA2-256, SHA3-256, and Blake3-256, show specifics of applying the general process. This general process can be used for calculating SAIDs with other cryptographic algorithms.

For those who want to skim there are pictures below including bit diagrams that illustrate exactly what is happening.

What is a SAID?

Fundamentally, a SAID is a cryptographic digest of a given set of data and is embedded within the data it is a digest of. A CESR-style SAID pads the digest to 33 bytes and adds a type code into the padded digest to replace resulting Base64 pad characters. It looks like this:

HPJbVi6fZvGNCASDiwABn2wpQ0lI-2cR0yaoRErkD-j6

This is a SHA3-256 digest encoded in the CESR format.

What is the CESR format? It is the Base64 URL Safe encoding of the raw digest along with some front-padding of zero bits and a type code, as shown in detail below. From the above SAID, the ‘H’ character is the type code. The rest of the string is composed of Base64 URL Safe characters.

Why Base64? More Space

Why was Base64 encoding used rather than something like hex encoding? Because Base64 encoding allows maximally compact text encoding of data using a well-known encoding protocol of alphanumeric characters (0-9, a-z, A-Z, -_). As compared to hexadecimal (“hex”) encoding Base64 encodes 6 bits of data per Base64 character whereas hex encoding encodes 4 bits of data per Base64 character, so Base64 can store 50% more data in the same space compared to hex. This helps reduce bandwidth and power costs, optimizing performance overall.

Note on Hash or Digest Terminology

A note on terminology, sometimes digests are called hashes or hash values. The technical definition of the term hash refers to a hash function. Hash functions transform data into a fixed-size string. This fixed-size string is the digest, the output of a hash function.

Back to SAIDs, the fact that a SAID can be embedded in the data it is a digest of is why it is called “self addressing.” The digest is essentially a unique identifier of the data it is embedded in.

A SAID (Self-Addressing Identifier) is a special type of content-addressable identifier based on an encoded cryptographic digest that is self-referential.
Composable Event Streaming Representation ToIP Specification – Section 12.6 – Dr. Samuel M. Smith

What is a content addressable identifier? A content addressable identifier is an identifier derived from the content being stored which makes a useful lookup key in content addressable storage, such as IPFS or a key-value store database like LevelDB, LMDB, Redis, DynamoDB, Couchbase, Memcached, or Cassandra.

Embedding a digest changes the source data and hash, right?

How can the SAID digest could be accurate given that placing the SAID in the data it identifies changes the data, thus producing a different hash? The way SAIDs accomplish this is with a two step generation and embedding process.

Two step SAID generation and embedding process During SAID calculation the destination field of the SAID is filled with pound sign filler characters (“#”) up to the same length of the SAID. The digest is then calculated, encoded, and placed in the destination field.

The reverse occurs for verification of a SAID.

The SAID is replaced with filler ‘#’ characters up to the same length of the SAID. The digest is calculated, encoded and compared with the SAID

How does the generation step work? This question kicks off a larger discussion about CESR-style encoding of cryptographic digests using pre-padding and type codes. First, let’s start with some code examples that cut right to the chase. You can come back to these examples after reading the post if they don’t make sense to you at first.

Code examples with multiple algorithms

Let’s start with some code examples showing how to create a correct SAID including the appropriate pre-padding characters. For additional understanding come back and review these examples after you have read the sections on 24 bit boundaries, pad characters, and pad bytes.

For now, say you want to use other cryptographic digest algorithms to create your SAIDs. How would you go about doing that?

It is as easy as changing your hashing function and then using the corresponding type code from the CESR Master Code Table corresponding to your desired digest algorithm.

The following code examples in Python illustrate the process for each of the following algorithms, Blake2b-256, Blake3-256, and SHA2-256. The SHA3-256 algorithm is shown above in the example in the main body of the article.

Filler ‘#’ characters in digest ‘d’ field

The following examples all use the raw value that includes the filler ‘#’ pound sign characters for the digest field ‘d’ which will both be explained later. The “d” digest field is supposed to contain the same number of filler characters as the eventual SAID that will replace the filler characters.

Creating a Blake2b-256 SAID – Step By Step

For a Blake2b-256 SAID with Python you just change the hash function and specify a digest size.

import hashlib from base64 import urlsafe_b64encode raw_value = b'{"d":"############################################","first":"john","last":"doe"}' digest = hashlib.blake2b(raw_value, digest_size=32).digest() # <-- See the different algorithm blake2b padded_digest = b'\x00' + digest encoded = urlsafe_b64encode(padded_digest) b64_str_list = list(encoded.decode()) # convert bytes to string of chars for easy replacement of 'A' b64_str_list[0] = 'F' # replace first 'A' character with 'F' type code b64_str = ''.join(b64_str_list) # convert string of chars to string with .join() assert b64_str == 'FFfZ4GYhyBRBEP3oTgim3AAfJS0nPcqEGNOGAiAZgW4Q' assert len(b64_str) == 44 # length should still be 44 characters, 264 base64 bits, a multiple of 24 bits Creating a Blake3-256 SAID – Step By Step

Blake3-256 is even easier, though it requires the blake library

import blake3 from base64 import urlsafe_b64encode raw_value = b'{"d":"############################################","first":"john","last":"doe"}' digest = blake3.blake3(raw_value).digest() # <-- See the different algorithm blake3.blake3 padded_digest = b'\x00' + digest encoded = urlsafe_b64encode(padded_digest) b64_str_list = list(encoded.decode()) # convert bytes to string of chars for easy replacement of 'A' b64_str_list[0] = 'E' # replace first 'A' character with 'E' type code b64_str = ''.join(b64_str_list) # convert string of chars to string with .join() assert b64_str == 'EKITsBR9udlRGaSGKq87k8bgDozGWElqEOFiXFjHJi8Y' assert len(b64_str) = 44 # length should still be 44 characters, 264 base64 bits, a multiple of 24 bits Creating a SHA2-256 SAID – Step By Step

And finally SHA2-256 is also easy, just changing the hash function used:

import hashlib from base64 import urlsafe_b64encode raw_value = b'{"d":"############################################","first":"john","last":"doe"}' digest = hashlib.sha256(raw_value).digest() # <-- See the different algorithm sha3_256 padded_digest = b'\x00' + digest encoded = urlsafe_b64encode(padded_digest) b64_str_list = list(encoded.decode()) # convert bytes to string of chars for easy replacement of 'A' b64_str_list[0] = 'I' # replace first 'A' character with 'I' type code b64_str = ''.join(b64_str_list) # convert string of chars to string with .join() assert b64_str == 'IDuyELkLPw5raKP32c7XPA7JCp0OOg8kvfXUewhZG3fd' assert len(b64_str) == 44 # length should still be 44 characters, 264 base64 bits, a multiple of 24 bits

Now on to a visual introduction.

Visual Introduction to SAID

Here is a SAID using the SHA3-256 algorithm on the sample JSON object used in this post.

HPJbVi6fZvGNCASDiwABn2wpQ0lI-2cR0yaoRErkD-j6

Adding this SAID to a document looks like taking the following JSON,

computing the SAID, encoding it, and placing it in the SAID field, or digest field, which is the “d” field in this example:

The ‘H’ character is highlighted here to draw attention to the fact that is a special character. This special character is the type code in the CESR Master Code Table. This indicates the type of cryptographic algorithm being used, SHA3-256 in this case.

I see a problem…

Those new to calculating and encoding SAIDs often encounter a problem here. If you take the raw Base64 encoded value of the JSON value {"d":"","first":"john","last":"doe"} then you end up with the string value eyJkIjoiIiwiZmlyc3QiOiJqb2huIiwibGFzdCI6ImRvZSJ9, which is nowhere close to the value shown in the picture of HPJbVi6fZvGNCASDiwABn2wpQ0lI-2cR0yaoRErkD-j6. Why are they different?

Doing a plain Base64 encoding of the JSON bytes misses an important step, the encoding step referred to above. The rest of the post dives deep into this encoding as it shows and explains how to construct a correct, CESR-encoded, SAID digest and explains the rationale behind why CESR encoding is designed the way it is.

Five parts of a SAID (SAID)?

As mentioned earlier, a SAID is a cryptographic digest. Specifically, it is a kind of digest usable as a content addressable identifier, and it is embedded in the content it identifies. SAIDs were invented by Dr. Samuel Smith as a part of his work on key event receipt infrastructure (KERI), authentic chained data containers (ACDC), and composable event streaming representation (CESR).

To understand how SAIDs work you must learn the interplay of five different concepts including:

Bit boundaries – aligning on 24 bit boundaries using pre-padded bytes on the left/front of raw bytes Hash values – hashing input bytes with hashing functions to produce output hash values (digests) Encoding with the URL-safe variant of Base64 encoding, Using type codes to indicate type of hashing function and size of digest, The two-pass SAID calculation and embedding process.

This article specifically covers SAIDs that are encoded in the CESR format. These CESR-style SAIDs

use pre-padding of pad bytes for bit padding to align on 24 bit boundaries, are compatible with a variety of common hashing functions, are encoded in the URL-safe variant of Base64 encoding (a.k.a. Base64URL), substitute type codes from the CESR Master code table (section 12.4.2) for ‘A’ front zero characters and are calculated from and embedded in the data they identify. How does it work? How are SAIDs calculated?

The easiest way to understand a self addressing identifier is to create one. Starting with the JSON from above we walk through each of the five major concepts required to create a CESR encoded SAID.

7 Steps to Calculate and Embed a SAID

Briefly, the process is listed here. A detailed explanation and example follows this set of steps.

Get an object to calculate a SAID for with a digest field that will hold the SAID. In this case we use the JSON object below and the “d” field will hold the SAID. The field does not have to be empty though it can be. Prior to digest calculation it will be cleared and filled with the correct number of filler characters. Calculate the quantity of Base64 characters the final encoded bytes will take up and fill the digest field with that many ‘#’ characters. This value may be looked up from a parse table like the CESR Master Code Table based on the type of hashing function used. Replace the contents of the digest field, “d” in our case, with pound sign (“#”) characters up to the number of filler characters calculated in step 2. The calculated size and pad values used for this step are reused in step 4. Calculate a digest of the object with the filler ‘#’ characters added using the hash function selected. This will result in a quantity of digest bytes, specifically 32 bytes for the SHA3-256 algorithm. Calculate the quantity of pad bytes that when added to the digest bytes will give you a value length that is multiple of 24 bits. This math is shown below. For us this is 1 pad character giving us 33 bytes. This value may be looked up from a parse table like the CESR Master Code Table. Perform pre-padding by prepending the pad byte to the digest bytes to get padded raw bytes. Encode the padded raw bytes with the Base64 URL Safe alphabet. Pre-padding causes some characters at the start of the digest to be encoded as “A” characters which represent zero in the Base64 URL Safe alphabet. Substitute the type code for the correct number of “A” zero character(s) in the Base64 encoded string according to the CESR encoding rules from the CESR Master Code Table. Use the type code corresponding to the cryptographic hash algorithm used. In our case this is “H” because we are using the SHA3-256 algorithm. This is your SAID! Place the Base64 encoded, type code substituted string (your SAID!) into the digest field in your object. This makes your object self-addressing. 3 Steps to Verify a SAID Start with a SAID from an object you already have. Calculate the SAID for the object using the process shown above Compare the SAID you pulled out of the object with the SAID you calculated. If they match then the SAID verifies. Otherwise the SAID does not verify.

An illustration will make clear why and how this process is done. Let’s walk through an example with a small JSON object. The concept applies to any size JSON object and objects of any serialization format such as CBOR, MessagePack, arbitrary text, or otherwise.

Example walkthrough with JSON and SHA3-256 Create Step 1: Get an object with some data and a digest field

Starting with the JSON below we have a “d” field, or digest field, in which the SAID will eventually be placed. In our case it is empty though it could start with the SAID in the “d” field and the process would still work.

JSON being SAIDified:
{
  "d": "", 
  "first": "john",
  "last": "doe"
}

Create Step 2: Calculate the quantity of filler ‘#’ characters

The expected final size of the SAID must be known in advance in order to create a JSON object with a stable size. Calculating this quantity requires that you understand a major concept in CESR:

How to calculate pad sizes (quantity of pad bytes) and full sizes of values.

Understanding this calculation will get you most of the way towards understanding another major CESR concept called “fully qualified Base64 representation” of a cryptographic primitive. A digest is a kind of cryptographic primitive.

Knowing the size in advance, and having it be stable, is critical for CESR’s type, length, value (TLV) encoding scheme. This stable size is achieved by filling the digest field with the same number of pound sign ‘#’ characters as the size of the SAID, which looks like this:

Correct number of filler characters added to digest field

{
  "d": "############################################", 
  "first": "john",
  "last": "doe"
}

This enables the JSON to have the same size during and after the SAID calculation process, giving a stable size. In order to know the number of filler characters then you must calculate how many Base64 characters will be in the final SAID. Calculating how many Base64 characters are needed involves summing raw bytes and pad bytes needed to align on what is called a 24 bit boundary.

Final output has same size since Base64 characters count equals filler length

Aligning on this 24 bit boundary allows the final result with the SAID to have the same length as the version with the filler characters, 44 characters in our case:

{
  "d": "HPJbVi6fZvGNCASDiwABn2wpQ0lI-2cR0yaoRErkD-j6", 
  "first": "john",
  "last": "doe"
}

Remember when the “encoding” step was mentioned from above? That’s where this filler character and size calculation knowledge comes in. In this encoding step you learn about the CESR-style encoding using pre-padding, pre-conversion. Knowing how many filler characters to use requires understanding the concept of aligning on a 24 bit boundary. Aligning on a 24 bit boundary is where the pre-padding of CESR comes in. This calculation of pad bytes required to align on a 24 bit boundary is the primary difference between raw, or “naive”, Base64 encoding and CESR encoding.

First let’s delve into what a 24 bit boundary is, why it matters to Base64 encoded values, and then look at some diagrams that make Base64 post-padding and CESR pre-padding clear. In doing this we jump ahead a bit and show byte diagrams of the actual encoded digest since that will help introduce later steps.

24 bit boundary – from Base64

The 24 bit boundary comes from the Base64 encoding format standard, RFC4648, specifically section 4. The reason a 24 bit boundary matters is because you can only use whole Base64 characters; there is no such thing as a fractional Base64 character. A Base64 character represents 6 bits of your raw bytes. A single byte is 8 bits. How do you reconcile the 6 bit Base64 character encoding to the 8 bits of your raw bytes? This is where a little math comes in, specifically the least common multiple.

Section 4 of the Base64 RFC 4648 describes the 24-bit groups that are the origin of the 24-bit boundary:

The encoding process represents 24-bit groups of input bits as output strings of 4 encoded characters. Proceeding from left to right, a 24-bit input group is formed by concatenating 3 8-bit input groups. These 24 bits are then treated as 4 concatenated 6-bit groups, each
of which is translated into a single character in the base 64 alphabet.
RFC 4648 The Base16, Base32, and Base64 Data Encodings – Section 4

Using these 24-bit groups ensures the value coming out of a Base64 decoder is the same value you put in. Separating raw bits into these 24 bit groups is where the phrase “aligning on 24 bit boundaries” comes from.

Splitting the 8-bit groups up into 6-bit groups requires a little math because 8 does not split evenly into 6. The math equation to do this is the least common multiple (LCM). LCM is used to determine the lowest number that both 8 and 6 divide into evenly, which is 24, thus the need for 24-bit groups, or 24-bit boundaries. Any value that is encoded into Base64 characters must be padded to reach a multiple of 24 bits. These 24-bit groupings allows you to cleanly convert all of your 8-bit bytes in to 6-bit Base64 characters and back to bytes without missing any bits.

Yet, if we have a stream that does not align on a 24 bit boundary then how do we create that alignment?

Pad characters on the END of a string are the answer to this in Base64.

By adding the correct number of pad characters on the end of a Base64 stream then you always end up with a value aligned on a 24 bit boundary. The ‘=’ equals sign pad characters in a plain Base64 encoding indicate the quantity of pad bits that were used in the final Base64 character adjacent to the ‘=’ pad characters.

Pad bytes at the START of the raw bytes are the answer to this in CESR.

By prepending the correct number of pad bytes on the start of a set of raw digest bytes then you always end up with a value aligned on a 24 bit boundary. Since the pad bytes are all zero bits then the resulting encoded value will start with one or more ‘A’ characters since they correspond to all zero bits in the Base64 alphabet.

Pad characters Calculation

In a plain Base64 encoding when encoding an array of bytes into Base64 that does not align on a 24 bit boundary the correct number of Base64 pad characters ‘=’ must be included. Why? Because in order to avoid data corruption in the decoded value you must know the precise original value, which means knowing how many pad characters to strip off and how many pad bits to strip out of the Base64 character adjacent to the padding. The decoder of your Base64 character needs to know how many bits of the last character used were just padding and how many were a part of your raw value.

You must signal the end of your raw bytes somehow. If, instead, you ignore, drop, or omit pad characters then you will confuse a Base64 decoder into thinking that pad bits were a part of your raw bytes, which you want to avoid because that will give you a different output value than what your input value was, meaning you would experience data corruption.

Pad characters must be included with a plain or “naïve” Base64 encoded value so that a Base64 decoder can strip the correct number of pad bits from the output giving you your original input bytes when decoding from Base64 characters to raw bytes. This is the purpose that Base64 pad characters serve. The pad characters indicate how many pad byes were used to encode a value in Base64.

CESR uses pad bytes and characters in a similar way, yet on the front, and with pre-conversion padding, so the rules for identifying and stripping pad bits are slightly different.

Yet, let’s stick with Base64 padding for now and come back to CESR padding later. If you are starting to get confused or lost then skip ahead to the diagrams below and come back to this explanation.

ASIDE – Calculating the quantity of Base64 pad characters based on input byte quantity

For a SHA3-256 digest this count is 44 characters. See the math below for an explanation. This number may also be found in the CESR Master Code Table for the type of algorithm used. Since we measure every raw value in terms of bytes (8 bits) then there are three possible scenarios, detailed here in the Base64 RFC, for the number of pad bytes required and thus pad characters.

A value ending with a single byte (8 bits) beyond a 24 bit boundary requires two bytes (16 bits) to meet a 24 bit boundary. This will have two ‘=’ pad characters. This means that your 8 raw bits + the 16 padding bits (two bytes) will equal 24 bits, aligning your raw value on a 24 bit boundary. A value ending with two bytes (16 bits) beyond a 24 bit boundary requires one byte (8 bits) to align on a 24 bit boundary. This will have one ‘=’ pad character. Take the 16 bits + one pad byte (8 bits) to get to 24 bits to align on the 24 bit boundary. A value ending with three bytes is already aligned on a 24 bit boundary (3 * 8 = 24)

You can use the modulus operator ‘%’ to determine the number of ending bits you have. For 256 bits (32 bytes * 8 bits per byte) you end up with 16 bits, or two bytes, rule number two above. So we need the equivalent of one pad byte.

How Base64 handles pad bits

The way that Base64 handles the need for pad bytes is to split the last byte into two characters, add zero bits to the last Base64 character, and then add the correct number of pad ‘=’ equals sign characters to the final output to end up with groups of 4 Base64 characters, which aligns on a 24 bit boundary because 4 * 6 bits per Base64 character = 24 bits.

What this means for a SAID – Calculating Pre-pad Bytes for CESR

In CESR padding is handled a bit differently because it repurposes the pad characters for type codes in its TLV encoding scheme. This means that what would have been zero bits representing ‘A’ characters in the Base64 encoded CESR value gets replaced with the type code, also called derivation code, in the final CESR value. To accomplish this CESR does pre-padding prior to conversion to Base64 characters. What this means for SAIDs is that all digest bytes must be padded at the front of the digest bytes to reach a multiple of 24 bits. Compare this to Base64 padding which occurs at the end of the digest bytes. Both scenarios are pictured below, Base64 padding and CESR padding.

Since the SHA3-256 digest we start with is 32 bytes, or 256 bits (not a multiple of 24), then all we need to add is one byte to get to 264 bits, which is a multiple of 24, or 33 bytes.

Now once you know the quantity of bytes that align on a 24 bit boundary you can do a simple calculation to get to the number of pad characters for your digest. Since 6 bits of every byte are put into a Base64 character (6 bit groups) then you can divide your total number of bits (264) by 6 to get the number of Base64 characters of your final digest.

264 (bits) / 6 (bits per Base64 char) = 44 (Base64 chars)

This means the total length of the resulting SAID will be 44 Base64 characters. So, you need 44 filler ‘#’ pound sign characters in your digest field of your JSON object prior to calculating the SAID.

Fixed width output – why is it needed?

Consistent sizing of the resulting JSON object for stable size of the overall output is the primary reason for pad characters. In order to create the same size output both before and after the SAID is added into the JSON there must be an equivalently sized number of pound signs (44 in this case) placed into the same field where the SAID will go. This is used in CESR encoding because CESR data types are encoded with to a type, length, and value scheme (TLV scheme) that simplifies parsing. Size of the overall output is the length, or “L,” in TLV and it only works if you have a known width data.

{
  "d": "############################################", 
  "first": "john",
  "last": "doe"
}

Now that you know the rules for calculating the number of pad characters then we are ready to illustrate the calculation process with diagrams.

Diagram for plain “naïve” Base64 encoding of SHA3-256 digest

Base64 uses post-padding, post-conversion of pad characters, as shown in the diagram below. You start with the raw digest. All the boxes in this diagram represent the raw bytes of the digest. There is no padding yet because the value is raw and is not yet converted to Base64 characters.

Binary bits of 32 byte SHA3-256 digest of above JSON with ‘#’ filler

For those following along in code the raw bytes of the 32 byte SHA3-256 digest of the JSON above (with the ‘#’ filler characters) are represented in binary as follows:

1111001001011011010101100010111010011111011001101111000110001101000010000000010010000011100010110000000000000001100111110110110000101001010000110100100101001000111110110110011100010001110100110010011010101000010001000100101011100100000011111110100011111010

Take a look at the last two bytes are 11101000 and 11111010. This factors in to the last two characters adjacent to the pad character as you see below.

Encode this 32 byte digest to Base64 URL Safe and you get get:

What happened here is that four bits (1010) of the last byte ( 11111010) were encoded into the last character lowercase ‘o’ adjacent to the pad character. If you look at the value for lowercase o in the Base64 alphabet you will see that lowercase ‘o’ has the bit pattern 101000. Yet it only pulled four bits from the last byte of 11111010 so where did the last two bits (00) come from? They were added in by the Base64 encoder. These two pad bits are why the corresponding final value has a single equals sign ‘=’ pad character. That instructs the Base64 encoder to strip two bits from the last character during the decoding process:

IMPORTANT: Base64 does not add the padding to the raw bytes prior to conversion. Instead it adds the padding while converting the 6 bit groups of the raw bytes into Base64 characters.

Due to the fact that 32 bytes, 256 bits, does not evenly align on a 24 bit boundary, is not a multiple of 24, the Base64 encoder splits the last byte into two different Base64 characters since 8 bits does not evenly fit in one 6 bit group and must be spread across two 6-bit groups. Each of these 6 bit groups each get their own Base64 character. In this case, the last two bytes 11101000 and 11111010 get spread across the last two characters ‘P’ (001111) and ‘o’ (101000).

Because of how the math works when splitting the 8-bit byte groups into 6-bit Base64 character groups the ‘o’ character got four bits from the very end of the digest. Yet four bits is not enough for a Base64 character so the Base64 encoder adds two zero bits on the end, signified with white boxes containing zeroes. Before the pad character is added then we are at 43 Base64 characters (6 bit groups, 258 bits), which is not a multiple of 24 bits. When the pad character ‘=’ is added then we get to a 44 characters (264), which is a multiple of 24 bits, meaning the encoding completed successfully.

Base64 Encoded SHA3-256 Digest

With the fully padded value you end up with a valid, encoded, Base64 value that looks like the following bit diagram:

The C2 character at the end shares some bits with the raw bytes of the digest and also contains some padding zero bits. The last character, C1, is an equals sign ‘=’ pad character. The fact that there is one pad character indicates to the Base64 decoder that there are two zeroed pad bits to remove from the last character, ‘C2’, during decoding in order to get back to the original digest bytes.

‘=’ is wasted space?

You could consider the pad characters ‘=’ as wasted space that could be useful if repurposed. All of the pad bits used for the equals sign could represent something. This is exactly what CESR does except it moves the padding to the front of the bytes so that it can have a uniform TLV encoding format. TLV encoding formats require the type character to be at the front of the value, so using post-padding like Base64 does would not work.

Along these same lines, SAIDs do not use Base64-style padding because it does not enable separability of individual concatenated values due to the fact that there is no easy way to cleanly and reliably separate individual values out of a Base64 encoded stream of bytes. The CESR specification introduction mentions this:

This Composability property enables the round-trip conversion en-masse of concatenated Primitives between the text domain and binary domain while maintaining the separability of individual Primitives.
Composable Event Streaming Representation ToIP specification – Dr. Sam Smith

Now that you understand how the plain or “naïve” Base64 encoding works then we turn our attention to CESR style pre-padding.

CESR Byte Padding: Pre-padding, Pre-conversion

In CESR the padding of values occurs with the raw bytes prior to encoding to Base64 as shown below in the white box containing ‘B33.’

What this means is that the raw value, prior to conversion, already aligns on a 24 bit boundary. Due to this alignment pre-conversion then there will never be any Base64 pad characters ‘=’ in the output.

How many bytes to prepend?

How do you know how many bytes to prepend? With a similar calculation we did above to find the number of filler characters

Since the SHA3-256 digest we start with is 32 bytes, or 256 bits (not a multiple of 24), then all we need to add is one byte to get to 264 bits, which is a multiple of 24, or 33 bytes.

Again, once you know the quantity of bytes that align on a 24 bit boundary you can do a simple calculation to get to the number of pad characters for your digest. Since 6 bits of every byte are put into a Base64 character (6 bit groups) then you can divide your total number of bits (264) by 6 to get the number of Base64 characters of your final digest.

264 (bits) / 6 (bits per Base64 character) = 44 (Base64 Characters)

So 44 will be the quantity of filler characters to put into the JSON object in order to calculate a SAID.

What happens when prepending bytes for CESR style encodings?

When encoding a value that requires padding with CESR-style padding (up front), instead of ‘=’ at the end like Base64-style padding would produce you end up with ‘A’ characters on the front of your encoded value. You also end up with the one character adjacent to the ‘A’ character(s) including some pad bits and some raw bits, as shown below in the bit diagram.

The intermediate encoded value looks like the below value that is not yet a SAID. This is not yet a SAID because the ‘A’ character has not yet been replaced with a type code from the TLV scheme indicating this is a SHA3-256 digest.

This ‘A’ character represents all zero bits (000000) in the Base64 alphabet.

In binary the full, pre-padded digest value (all 33 bytes) looks like the following. Notice the zero bits at the front.

000000001111001001011011010101100010111010011111011001101111000110001101000010000000010010000011100010110000000000000001100111110110110000101001010000110100100101001000111110110110011100010001110100110010011010101000010001000100101011100100000011111110100011111010

The first two bytes are 00000000 and 11110010 which get encoded into Base64 as shown below. Six of the zero pad bits get encoded as an ‘A’ character and two of the zero pad bits get included in the capital ‘P’ character which also has four bits from the next raw byte of data from the digest.

Bit diagram of Base64 encoded, CESR pre-padded raw value.

This diagram illustrates how CESR does pre-padding with pad bytes of zero bits prior to performing a Base64 encoding on the fully padded raw value. The next diagram of characters shows the space a fully padded, encoded, CESR-style value would look like.

As you can see, the padding is at the front of the encoded value rather than the back like Base64 does. And the character with shared pad and raw bits is adjacent to the pad character at the front of the Base64 encoded value.

To get to the final SAID then you replace the ‘A’ character with the appropriate type code, or derivation code, yet we are getting ahead of ourselves a bit too much. Let’s now get into the calculation of the digest.

This step showed you how to calculate the appropriate number of filler ‘#’ pound sign characters to put in to the digest field in your JSON object. The next step shows you how to calculate a digest of that JSON object.

Creation Step 3: Calculate a digest of the data

When calculating a digest then you take the data with the correct number of filler characters added to the digest field and you simply take a digest of it. So in our case we would take a digest of the following

{
  "d": "############################################", 
  "first": "john",
  "last": "doe"
}

In Python taking a digest of this data would be as simple as the following:

import hashlib raw_value = b'{"d":"############################################","first":"john","last":"doe"}' digest = hashlib.sha3_256(raw_value).digest() # hash function ↑↑↑↑

This is a simple step and is very similar for any other algorithm such as SHA2-256, Blake3-256 or otherwise. You use the desired type of hash function.

The only other thing to be aware of here is that if you create a digest that is sized differently than 32 bytes, such as a SHA3-512 digest (64 bytes) then you need to also change the number of pad bytes, which gets into the next step.

Creation Step 4: Calculate the quantity of pad bytes

The calculation for the quantity of pad bytes is very similar to the calculation for the quantity of filler ‘#’ characters needed in Step 2. In fact, it is a subset of that calculation. The goal with pad characters is to make sure that the final value aligns on a 24 bit boundary as mentioned above.

For example, since the SHA3-256 digest we start with is 32 bytes, or 256 bits (not a multiple of 24), then all we need to add is one byte to get to 264 bits, which is a multiple of 24, or 33 bytes.

Deeper into Modulus Math for Pad Bytes

To get a bit deeper into the math, one way to do this calculation with the modulus operator is to find out how many characters are necessary to completely fill a 3 byte group. Since 3-byte groups are multiples of 24 then you can use a modulus calculation to see how far away you are from filling a three byte group by doing a modulus 3 operation in two steps:

Step 1: take bytes mod 3

32 bytes mod 3 = 2 (bytes)

meaning there are two bytes already in the last group of three (24 bit boundary).

Step 2: subtract bytes in group from group size

So to see how many bytes you must add to get to the 24 bit boundary (3 byte group) you subtract the quantity of bytes you have from the group size:

3 (group size) – 2 (bytes in group) = 1 (pad bytes needed to fill group)

Due to how modulus arithmetic works you will only ever have three possible values from this equation:

3 – (bytes mod 3) = 0 (pad bytes) 3 – (bytes mod 3) = 1 (pad bytes) 3 – (bytes mod 3) = 2 (pad bytes)

You never have to worry about three pad bytes because that would be an even multiple of 24 which means your raw value would already align on a 24 bit boundary and thus not need any pad bytes.

So, to review, for us the calculation of (3 - (32 mod 3)) = 1 pad byte gives us a single pad byte to be prepended to our raw value, as shown below in the ‘B33’ box.

As mentioned before, CESR does pre-padding, pre-conversion which means that the pad byte we found we need is added to the front of the array of raw bytes for the SHA3-256 digest. The next step covers encoding this padded raw value.

Creation Step 5: Base64 URL Safe Encode the padded raw bytes

Now that the raw value from Step 4 is properly padded then you encode it with Base64 URL Safe encoding. CESR uses Base64 URL Safe encoding rather than plain Base64 encoding so that CESR values can safely be used in URLs and filenames.

Now that you have the Base64 URL Safe encoded value then you are ready to finish off this SAID creation by replacing the ‘A’ pad character at the front of the encoded value with the appropriate value from the CESR Master Code Table.

Creation Step 6: Substitute Type Code for the front ‘A’ character(s)

When CESR pre-padded the raw value to get to a 24 bit boundary the purpose of that was to be able to repurpose the wasted space of the pad character for a type code in CESR’s TLV encoding scheme. The ‘A’ character at the front of the value in this scheme is considered to be a pad character. This pad ‘A’ character will be replaced with the appropriate type code, or derivation code in CESR parlance, from the CESR Master Code Table.

For a SHA3-256 digest that type code is ‘H’ as seen in the following subset of the CESR Master Code Table.

The substitution gives us a final value of HPJbVi6fZvGNCASDiwABn2wpQ0lI-2cR0yaoRErkD-j6 as seen in the following substitution diagram.

The substitution of the ‘A’ character with the ‘H’ character is the final part of what is called CESR encoding a raw digest value into a CESR-style self addressing identifier. This SAID is a front-padded, Base64 encoded, and type-code substituted, string of Base64 characters.

The final value can be created by the code as follows:

import hashlib from base64 import urlsafe_b64encode raw_value = b'{"d":"############################################","first":"john","last":"doe"}' digest = hashlib.sha3_256(raw_value).digest() padded_digest = b'\x00' + digest encoded = urlsafe_b64encode(padded_digest) b64_str_list = list(encoded.decode()) # convert bytes to string of chars for easy replacement of 'A' b64_str_list[0] = 'H' # replace first 'A' character with 'H' type code b64_str = ''.join(b64_str_list) # convert string of chars to string with .join() assert b64_str == 'HPJbVi6fZvGNCASDiwABn2wpQ0lI-2cR0yaoRErkD-j6' assert len(b64_str) == 44 # length should still be 44 characters, 264 base64 bits, a multiple of 24 bits Creation Step 7: Place the Front-Padded, Base64 encoded, Type-code Substituted string in the digest field

Now we can take this correctly padded, CESR encoded value and place it into the digest field in our JSON object, replacing the filler ‘#’ characters with the final, valid SAID:

{
  "d": "HPJbVi6fZvGNCASDiwABn2wpQ0lI-2cR0yaoRErkD-j6", 
  "first": "john",
  "last": "doe"
}

This takes us back to where we started off, with a valid SAID and a SAIDified JSON object.

What about verification?

What is nice about verification is that it is as simple as calculating the SAID again of a JSON object and comparing that to a SAID you are handed.

Verification Step 1: Start with a SAID from the object you already have

Say you are starting with the below object that has already had a SAID calculated and embedded in the digest field, the “d” field here.

{
  "d": "HPJbVi6fZvGNCASDiwABn2wpQ0lI-2cR0yaoRErkD-j6", 
  "first": "john",
  "last": "doe"
}

To get the SAID from this object you extract the value of the “d” field, giving you HPJbVi6fZvGNCASDiwABn2wpQ0lI-2cR0yaoRErkD-j6

Verification Step 2: Calculate the SAID of the object using the SAID creation steps

Verification is easy because all you need to do is take steps 1 through 6 above and re-calculate the SAID on the JSON object provided. Once you have recalculated the SAID, which will be HPJbVi6fZvGNCASDiwABn2wpQ0lI-2cR0yaoRErkD-j6 again, you can perform the comparison in step 3.

Verification Step 3: Compare the SAID from the object to the calculated SAID

If the SAID the object started with matches the SAID you calculated from the object then you know the object has not been changed and that the SAID is valid. Otherwise either your SAID is invalid or the object has changed.

Review Calculating a SAID

Now you understand how we SAIDify a JSON object by doing the following seven step process:

Start with a JSON object we want to add a SAID to that has a digest field. Calculate the quantity of Base64 characters the final, pre-padded, encoded raw digest bytes (SAID) will take up and fill the digest field with that many ‘#’ characters. Calculate a digest of the bytes of the JSON object after the ‘#’ filler characters are added. Calculate the quantity of pad bytes needed to align on a 24 bit boundary and prepend that to the raw bytes for a digest. Encode the padded raw bytes with the Base64URLSafe alphabet. Substitute the appropriate type code in place of the ‘A’ character(s) at the front of the encoded string. This final value is your SAID Place the final SAID value into the digest field of your JSON object.

Pre-padding and type code substitution prior to Base64 encoding is the essence of CESR-style self addressing identifiers. The steps above may seem overwhelming at first, though once you mentally anchor in that CESR pads at the start and that padding gives you ‘A’ characters you can reuse for type codes then you have mastered the fundamentals of what makes CESR style SAIDs work.

Verifying a SAID

Verification of a SAID is easy because you just calculate it again from the original JSON object, or other data object you are using. If the SAIDs match then it verifies; if they don’t then the data changed.

Extra Learning Alert – fully qualified Base64 primitive

And, as a nice side note, you happen to now know what the phrase “fully qualified base64 primitives” in KERIpy means. All that means is that your encoded value has been pre-padded, pre-conversion, and has had its type code added to the front, as we did here with substitution, with the exception that some CESR primitives

Give me a library please! I don’t want to manage these details

In case this article has convinced you that you do not ever again want to worry about the vagaries of aligning on 24 bit boundaries for Base64 or CESR values then you are in luck. There are multiple implementations of the SAID process that can meet your needs in a variety of different languages.

The Python reference implementation in Web Of Trust’s KERIpy’s Saider.saidify. The Human Colossus Foundation’s Rust implementation with WASM bindings for their JavaScript package. See their cool SAID generator and verifier demo here where you can try a whole list of different algorithms. SAIDify, my own Typescript implementation of the SAID creation process. Implementations Web Of Trust KERIpy Python

The Python example below from KERIpy shows a unit test showing the usage of the KERIpy Saider.saidify library code to calculate a SAID. The SAID is stored in the .qb64 property of Saider. The term qb64 stands for “qualified base64” which means a left-padded, Base64 encoded, type code substituted value as described above.

import json from keri.core.coring import MtrDex, Saider def test_saidify_john_doe(): code = MtrDex.SHA3_256 ser0 = b'{"d": "", "first": "john", "last": "doe"}' sad0 = json.loads(ser0) saider, sad = Saider.saidify(sad=sad0, code=code) assert saider.qb64 == 'HPJbVi6fZvGNCASDiwABn2wpQ0lI-2cR0yaoRErkD-j6' Human Colossus Foundation Rust SAID demo and test code

Start with their cool demo site of generating and verifying SAIDs:

If you want to dive into their code the linked test basic_derive_test shows the Rust code for the cool SAD macro #[derive(SAD, Serialize)]that can turn any Rust struct along with the #[said] field attribute for the SAID digest field into a self-verifying data structure.

    use said::derivation::HashFunctionCode;
    use said::sad::SAD;
    use said::version::format::SerializationFormats;
    use said::SelfAddressingIdentifier;
    use serde::Serialize; 

    #[test]
    pub fn basic_derive_test() {
        #[derive(SAD, Serialize)]
        struct Something {
            pub text: String,
            #[said]
            pub d: Option<SelfAddressingIdentifier>,
        }

        let mut something = Something {
            text: "Hello world".to_string(),
            d: None,
        };

        let code = HashFunctionCode::Blake3_256;
        let format = SerializationFormats::JSON;
        something.compute_digest(&code, &format);
        let computed_digest = something.d.as_ref();
        let derivation_data = something.derivation_data(&code, &format);

        assert_eq!(
            format!(
                r#"{{"text":"Hello world","d":"{}"}}"#,
                "############################################"
            ),
            String::from_utf8(derivation_data.clone()).unwrap()
        );

        assert_eq!(
            computed_digest,
            Some(
                &"EF-7wdNGXqgO4aoVxRpdWELCx_MkMMjx7aKg9sqzjKwI"
                    .parse()
                    .unwrap()
            )
        );
        assert!(something
            .d
            .as_ref()
            .unwrap()
            .verify_binding(&something.derivation_data(&code, &format)));
    }

SAIDify

If you want to use a Typescript library that is about 530 lines of code you can go with my SAIDify library. The below example shows how to use the library with Typescript.

Start with an NPM install

npm install saidify

And then you can use the saidify(data, label) function to SAIDify any JavaScript object you have as long as you indicate which field is the digest field, the “label” field, which defaults to the “d” field.

import { saidify, verify } from 'saidify' // create data to become self-addressing const myData = { a: 1, b: 2, d: '', } const label = 'd' const [said, sad] = saidify(myData, label) // said is self-addressing identifier // sad is self-addressing data console.log(said) // ...Vitest test assertion expect(said).toEqual('ELLbizIr2FJLHexNkiLZpsTWfhwUmZUicuhmoZ9049Hz') // verify self addressing identifier const computedSAID = 'ELLbizIr2FJLHexNkiLZpsTWfhwUmZUicuhmoZ9049Hz' const doesVerify = verify(sad, computedSAID, label) // can verify with original myData or sad // ...Vitest test assertion expect(doesVerify).toEqual(true) Conclusion

The key takeaways from calculating SAIDs are:

Use pre-padded bytes to align on a 24 bit boundary prior to encoding as Base64 characters. Substitute type codes in for the leading ‘A’ character(s) of a SAID. It is easy to chose different algorithms for the SAID calculation process. Just make sure you use a code on the CESR Master Code Table if you want to be CESR compliant. There are multiple implementations of the SAID algorithm you can use.

Now go make some SAIDs!

References: HCF oca-spec #58 RFC 4648: The Base16, Base32, and Base64 Data Encodings, specifically section 5 Composable Event Streaming Representation (CESR) ToIP Specification, specifically section 12.6 Self Addressing Identifier IETF draft specification SADs, SAIDs, and ACDCs video presentation by Daniel Hardman

19:27 • 1 months ago

Michael Ruminer

Anthropic’s Contextual Retrieval

One of the problems I have encountered in trying to do Retrieval Augmented Generation (RAG) where a complete single document was not uploaded for queries on that document alone is that chunks created from documents for embedding often lose context. I have often considered what would be the outcome if one were to put some limited but useful context along with each chunk. Now I know. Introducing Ant

One of my concerns with placing some additional context for each chunk is that you’d probably need to pass the entire document being chunked as context along with each individual chunk. That would be very expensive and slow. Now, with Anthropic’s Claude prompt caching, the cost and latency is reduced significantly and it seems actually doable for chunk contexting (is contexting a word?). An initial prompt including the full document could be done with that prompt cached for future prompt reference.

I plan to try this out.

Check out the Anthropic “Introducing Contextual Retrieval” post for greater details.

10:59 • 1 months ago

Saturday, 21. September 2024

Jon Udell

deo absente deum culpa

On a recent trip I saw this pair of Latin phrases tattooed on the back of a flight attendant’s arms: Left: Deo absente. Right: Deum culpa. I took Latin in middle school, and could guess what the combination might mean. It’s not a common construction, and a search seems to confirm my guess. Both Google … Continue reading deo absente deum culpa

On a recent trip I saw this pair of Latin phrases tattooed on the back of a flight attendant’s arms:

Left: Deo absente. Right: Deum culpa.

I took Latin in middle school, and could guess what the combination might mean. It’s not a common construction, and a search seems to confirm my guess. Both Google and Bing take you to a couple of Reddit posts in r/Latin.

Would this be the correct translation?

A song I like, Deus in absentia by Ghost, has that line in it intending to mean “In the absence of God”, so I was looking into alternate translations/syntax of the phrase intending to mean “In the absence of God; Blame/Fault God”. Would this make sense: “Deum in absente; Culpa Deus” or “Deus Culpa”?

Does the phrase “Deus In Absentia, Deus Culpa” make sense?

I’m using this for a tattoo and want to be absolutely sure it works in the sense of ‘In the absence of God, blame God’. All help appreciated!

Is that the same person I saw? If so, the responses in r/Latin seem to have guided them to the final text inked on their arms. And if so, the message is essentially what I had guessed. The intent of the message, though, is open to interpretation. I’m not quite sure how to take it. What do you think it means? Would it have been rude to ask?

05:24 • 1 months ago

Friday, 20. September 2024

Jon Udell

SQL translation from Postgres to SQLite and DuckDB

Powerpipe dashboards can now connect not only to Steampipe but also to SQLite and DuckDB. This creates a combinatorial explosion of possibilities, including dashboards that use SQL to visualize large datasets read from Parquet files by DuckDB. SQL Translation From Postgres to SQLite and DuckDB Part of the LLM series at The New Stack.

SQL Translation From Postgres to SQLite and DuckDB

Part of the LLM series at The New Stack.

19:59 • 1 months ago

Thursday, 19. September 2024

The Pragmatic Engineer

The Pulse #107: What does Amazon’s 5-day RTO mean for tech?

Amazon is the first Big Tech to mandate a strict 5-day return to office. What are the real reasons, will Amazon see a ‘brain drain’ as a result, and could other Big Tech companies follow?

The Pulse is a series covering insights, patterns, and trends within Big Tech and startups. Notice an interesting event or trend? Send me a message.

Today, we cover:

Industry pulse. OpenAI’s new model, LinkledIn opts everyone into GenAI training (save for those in the EU), Unity reverses runtime fees, what a startup “fire sale” looks like for employees, and more.

What does Amazon’s 5-day RTO mean for tech? The online retailer is the first Big Tech giant to mandate a strict, 5 days in the office policy. It says it wants things back exactly like they were before the pandemic. One reason is likely something less discussed: tax incentives from cities granted for staff being onsite, daily

Tech debt: further learnings. Two weeks ago, Lou Franco shared hard-earned lessons on keeping tech debt at bay. Lou shares two additional learnings: using tech debt payments to get into the flow, and how big rewrites need heavyweight support.

1. Industry pulse OpenAI’s new model: o1

Last week, OpenAI revealed its latest model called OpenAI o1. Until now, LLM models got bigger in how much data they were trained on, but o1 is a change of direction. It spends more time “thinking” before it responds: basically, it chains several LLM operations one after the other. As software engineer Simon Willison puts it:

“One way to think about these new models is as a specialized extension of the chain of thought prompting pattern — the “think step by step” trick that we’ve been exploring as a a community for a couple of years now, first introduced in the paper Large Language Models are Zero-Shot Reasoners in May 2022.”

The new model is far more capable in its responses – even though these responses take more time to process – and brings a new dimension in how to use language models. Because the model spends more compute time on answers, it has become more expensive to use: o1 costs 100x as much as GPT-4o mini, and 3-6x as much as GPT-4.

This approach feels like the biggest recent development in LLMs this year. It’s worth withholding judgment until we use it more, and learn more about its capabilities and tradeoffs. AI keeps moving fast, and OpenAI is at the forefront.

LinkedIn opts everyone into GenAI training, except in EU

15:48 • 1 months ago

Michael Ruminer

A Great AI RAG Resource

I came across a great AI Retrieval Augmented Generation resource. It is a Github repo: Advanced RAG Techniques: Elevating Your Retrieval-Augmented Generation Systems.I’ll just copy and paste their introduction here. “Welcome to one of the most comprehensive and dynamic collections of Retrieval-Augmented Generation (RAG) tutorials available today. This repository serves as a hub for cutting-edge t

“Welcome to one of the most comprehensive and dynamic collections of Retrieval-Augmented Generation (RAG) tutorials available today. This repository serves as a hub for cutting-edge techniques aimed at enhancing the accuracy, efficiency, and contextual richness of RAG systems.”

All I can say is, wow. It really covers a lot of ground. I plan to dig into it and will report back.

09:53 • 1 months ago

Tuesday, 17. September 2024

The Pragmatic Engineer

The Pragmatic Engineer: Three Years

How the newsletter evolved, popular topics the last year, and The Pragmatic Engineer Podcast launching

Hi, this is Gergely with a free issue of the Pragmatic Engineer Newsletter. In every issue, I cover topics related to Big Tech and startups through the lens of software engineers and engineering managers. To get weekly articles in your inbox, subscribe:

Meta: How the company built Threads with a small team in 6 months

This publication has turned three years old, which feels like the right time to pause and reflect on the past 12 months for The Pragmatic Engineer.

At time of publication, 759,402 readers subscribe to this newsletter. This is 300,000 readers up on a year ago, when it was just under 461,000. This figure far exceeds the size of the audience I assumed would be interested in software engineering deep dives into some fascinating and challenging topics, when I launched this publication. Thank you for your trust!

I’d like to extend a very special thanks to all paying subscribers: the publication runs on your support, and it enables us to deliver well-researched deep dives. Many paying subscribers expense this newsletter from their company learning and development budget. If you have such a budget, here’s an email to send to your manager. There’s also reduced prices for people in countries with lower average income than the US and Western Europe, and student discounts. If you enjoy the newsletter, you can also gift a subscription to others.

Speaking of deep dives: over the past year full subscribers received two articles almost every week: a deepdive on Tuesdays, and tech news in The Pulse on Thursdays. The last 12 months of issues adds up to around 5-7 books’ worth of reading (about 550,000 words). Free subscribers also got well over a nonfiction book’s worth of reading in their inbox: the first parts of the Tuesday articles, and a full article monthly.

Today, we look back, and peer in to the future:

How The Pragmatic Engineer has evolved

Popular, interesting topics

What I learned about readers!

Notable events

Looking ahead, and The Pragmatic Engineer Podcast

Before we jump in: if you’re someone who enjoys podcasts, I have a treat for you. The Pragmatic Engineer Podcast is launching with the first episode next week. To get the first episode in your podcast player, subscribe on:

YouTube — you can also watch the podcast announcement here

Spotify

Apple Podcasts

… or your favorite player!

1. How The Pragmatic Engineer evolved in the last year

Software engineering is an ever-changing field where the best teams continuously experiment with new approaches, and change how they work. I’m also an engineer at heart, so it’s only natural to do the same with the newsletter.

Here’s what’s changed in recent months; some of which you might have noticed, and other things that happened behind the scenes.

Unshared details from interesting tech companies

Some companies attract media attention due to how well they execute, and what this publication tries to do is learn how exactly they achieve it. I aim to bring details straight from the source, talking exclusively with engineering teams at companies in the spotlight. The last 12 months has seen exclusive articles on:

OpenAI: How they scaled ChatGPT, and Inside OpenAI: how does ChatGPT ship so quickly?

Anthropic: How the company built Artifacts (and details on how it operates)

Stripe: Inside Stripe’s engineering culture

Bluesky: How a tiny team built a Twitter/X competitor and Inside Bluesky’s engineering culture

Oxide: A hardware/software startup on “hard mode”

In terms of deep dives, it’s been the most prolific year yet for this publication. Doing lengthy articles which lift the lid on how leading teams get things done, can be challenging. Understandably, some companies do not normally share the details you’ve been able to read in this newsletter. At the heart of delivering articles like those above, is trust. Engineering leaders and software engineers ultimately feel they have scope to be candid with me, and that means a lot. Their trust in myself and The Pragmatic Engineer team is a very rewarding part of writing this newsletter.

More articles mixing deep research with pragmatic observations

Since February, it’s not just me doing the extensive research that goes into each article: Elin Nilsson is this publication’s first Tech Industry Researcher. Previously, Elin spent 7 years at Spotify, working in various mobile and platform engineering teams. Before, she interned at both Spotify as an Android developer and Google as a web/UX prototyper. She’s brought a whole lot of things onboard since she joined in February: like her infectious enthusiasm for nerding out about tech topics, diving deep into technologies and trends, and bringing order to the chaos of having too much information to process on any given topic.

Since she’s come onboard, there’s been more deeply researched pieces published than ever, and each one of them went into more depth. Some of the articles that started with Elin’s insightful research:

GenZ software engineers according to older colleagues, and what GenZ devs really think

Bluesky: How it was built, and Inside Bluesky’s engineering culture

How do AI software engineering agents really work?

AI tooling for software engineers: Reality check, Reality check (part 2) and Rolling LLM tools out company-wide (Part 3)

Thanks to Elin, we can take on ambitious projects that were simply not possible when the Pragmatic Engineer launched because the research can take months, We’re on a roll with Elin, so expect these to keep coming! You can always suggest interesting topics for us to potentially look into.

Goodbye “Scoop,” hello “Pulse”

During the first two years of this newsletter, there was usually an article every Thursday called “The Scoop,” which rounded up interesting things in the industry, and occasionally broke big tech stories. But as of a year ago, I am no longer doing this and the name change to “The Pulse” was part of this shift. I shared details about the change:

‘The Pulse’ just better represents the mission of these articles, which is to help you keep an “ear to the ground” on what’s happening across Big Tech and at startups – sectors which regularly affect the whole tech industry.
I receive and validate plenty of interesting information from insiders at these companies, but my main focus is analyzing what’s going on in our industry; the implications of that and the opportunities for software engineers and tech businesses.
Names matter, and I feel “The Scoop” suggests a more gossipy, tabloid newspaper-style approach than what these articles actually deliver. Since The Scoop’s first issue I’ve focused on relevant industry changes and what they mean. I hope “The Pulse” makes the intent of this series clear: to provide an ear to the ground, and the latest analysis of the state of our industry.

This change has been one I’m very happy with; it’s also partly why we’ve been able to publish more company deep dives straight from tech companies themselves.

When writing “The Scoop”, tech companies and devs whom I met were never quite certain if I had my “journalist” hat on (and would leak what they shared as news), or my “analytical” software engineer’s hat.

This made people reluctant to share information that could potentially identify their company, or affect their career. This was taxing for me personally and I decided I don’t want to be in the “gossip” business, but in the software engineering one. For that reason, the change from The Scoop was made.

But while writing it, I learned a lot about journalism and met a lot of reporters. It’s a fascinating world, but one I don’t feel is for me. Also, the time I save on no longer verifying information for news articles, means there’s more time to dive deep into relevant, interesting software engineering topics.

Industry Pulse

In the spirit of experimentation, I tried out a new section inside the Thursday “The Pulse” article. Industry Pulse is a roundup of tech events relevant for devs and EMs, with commentary.

It has delivered early insights on topics like The impact of Section 174 on the software industry, predicting pressure on commercial open source to make more money, the unfortunate “startup purge event,” analyzing why Amazon sunset several of its products in one fell swoop, and much more.

Readers feedback was very positive from the start. Also, I enjoy writing it and look forward to it every day: so it has stayed!

2. Popular, interesting topics

In case you’re looking for articles to read or re-read, here’s a selection of some standout ones from the past 12 months, in my opinion.

The Pragmatic Engineer Podcast: Trailer

Listen now (1 min) | A podcast covering software engineering at Big Tech and startups, from the inside

Welcome to The Pragmatic Engineer Podcast, hosted by Gergely Orosz, the author of The Pragmatic Engineer newsletter. In each episode, we dive deep into the world of software engineering, offering practical insights on scaling teams, engineering leadership, and navigating the evolving tech landscape. With industry veterans and successful engineers as guests, this podcast is perfect for anyone looking to level up their engineering career with real-world advice.

Subscribe to the podcast on YouTube, on Spotify, or Apple.

You can also subscribe to the newsletter here.

13:13 • 1 months ago

Monday, 16. September 2024

Heather Vescent

Ask A [cybersecurity] Futurist

Welcome to the inaugural “Ask a [cybersecurity] futurist column. This month’s question comes from Anna Borg. She asks: How can we change the “rules in the game” and make it less interesting to use the internet for attacks on organisations and societies including democracy. We will never “win the game” if we are forced to play defence play 24/7, 365 days per year. How do we change the incent

Welcome to the inaugural “Ask a [cybersecurity] futurist column. This month’s question comes from Anna Borg.

She asks:

How can we change the “rules in the game” and make it less interesting to use the internet for attacks on organisations and societies including democracy. We will never “win the game” if we are forced to play defence play 24/7, 365 days per year. How do we change the incentives for attackers?

I won’t sugar coat it, this is a complex and interconnected problem. Our increased cyber attacks are a symptom of a much larger problem. (Which means the solutions will not just address the cybersecurity problem at the core of this question, but bigger problems that society deals with.)

The Complexity of Cybersecurity Threats

For defense teams, the current state of cybersecurity feels like an endless battle, with organizations and societies constantly reacting to the latest threats. For hackers, it’s a landscape with abundant vulnerabilities if one is willing to put in the work to find (or create) the magic key — like a recent Advanced Persistent Teenager.

Cybersecurity is a long imbalanced game of whack-a-mole. Defenders have to guard everything, while attackers have the luxury of focusing. So how can you move out of a reactive mode to stay one step ahead of the next attack? And further, how can we make it even pointless for attackers to target us in the first place?

Understanding Hacker Motivation

To change the rules, we need to understand why cyber attacks are happening. What drives someone to become a black hat hacker? Or to start a black hat collective? Or join an elite nation state hacking team? I’ve spoken with security professionals, hackers and analyzed research data, and the motivations come down to: money, espionage, and the thrill of the con.

Viktor and Giselle from Better Call Saul

In a survey I conducted last year with identity-security experts, financial gain and espionage were the top motivations, followed by a desire for notoriety or the thrill of hacking. Most hackers today aren’t lone wolves; they’re part of organized collectives that pool their skills and resources to hit bigger, complex targets. They might even employ the Silicon Valley approved “AAS/as a service” business model.

There’s another factor that’s often overlooked: socioeconomic conditions. During my research for the paper Shifting Paradigms: Using Strategic Foresight to Plan for Security Evolution, I was genuinely surprised to hear about the overproduction of well-educated young people unable to find good jobs after their security education. There are very few well paid entry level security jobs — even in the US and developed nations.

Changing the Incentives

So how do we change the incentives to make cyber-attacks less appealing and less profitable for would-be attackers?

I’m going to skip over the obvious answer of creating/using better security technology. Everyone is racing to implement better tech solutions, but this is just a band aid — admittedly a really big band aid. I’m going to talk about non-tech solutions.

Economic Solutions

If it’s true that we are overproducing highly educated security people, could we solve some security problems by having full employment for all educated security experts?

One scenario in Shifting Paradigms envisioned this idea.

5.10 Scenario 9: Money, Cash, Hoes

In this scenario, whose name is inspired by the Jay Z song, security defensive work has become so attractive and well-paid that black-hat work is no longer attractive, and people with security-relevant technical skills are almost universally employed in defensive jobs. Governments can still hire offensive hackers, but criminal syndicates have a very hard time competing with white-hat employers.

Changes from Today: Employers pay great salaries, offer significant flexibility and benefits, and recruit extremely aggressively, especially in poor countries with good technical talent and in ungoverned spaces. There are many good entry level security jobs.

Could one of the most effective ways to reduce cybercrime to increase legitimate employment opportunities for those with technical skills? If well-paying jobs in cybersecurity or other tech fields were more readily available, especially in economically disadvantaged regions, would the incentive to engage in illegal activities diminish?

I think this scenario is illustrated in Wired’s recent article about Priscila Barbosa, Queen of the Rideshare Mafia, which describes the elaborate identity theft and fraud Priscila engaged in. Barbosa took IT classes back in her native Brazil and was a successful entrepreneur until the recession caused business to disappear. She came to find her fortune in the US — which she did. But because she overstayed her visa, she could not work legitimate jobs. I’d like to imagine, that if there was a legitimate way to work in the US, she would have never participated in criminal activities. And maybe, if there had been good tech jobs in Brazil, she might not have even needed to seek her fortune in the US.

In my view, Barbosa is a victim of economic lack of opportunity.

Disrupting [Criminal] Business Models

What are the business models that make cybercrime profitable? I remember Kim Cameron, Identity Architect, saying (not sure if this is his quote or he was quoting someone), that “the most profitable business model on the internet is crime. Back in the 90s and again now.”

Even entrepreneurial criminals have embraced the successful “as a Service” software development model. But to me, this echoes back to the economic themes. If there were legitimate economic paths half as successful, would there be less interest in criminal activity?

Nation-State Attacks

Nation-state hackers are a whole different beast. Their motivations are tied to geopolitical loyalty and belief. To reduce these types of attacks, we need to focus on improving global cooperation and reducing international tensions. This might sound idealistic, but fostering peace and global coexistence could significantly decrease the incentive for nations to engage in cyber warfare.

Reducing the Thrill

Then there’s the issue of hacking for fun, cultural subversion, or “the thrill of the con.” This is harder to combat, but one approach is to create more avenues for ethical hacking while embracing the dark side of hackers. Bug bounty programs, for example, allow hackers to flex their skills in a legal, constructive way. These programs are subversive in their own right but channel that energy into something positive.

Finding Solutions

Is the solution simply a transparent and interdependent global economy with legitimate well paid jobs so we can all live our best lives without stepping on someone to do so?

I don’t know about you, but that feels like a big and impossible solution — perhaps a bit too utopian. So, I want to leave Anna with some reasonable actions she or her company might take.

Entry level security roles: Increase well paid jobs & support development programs to transition people into well paid jobs. As part of this, eliminate the rockstar mentality, and hire someone who is good enough for the job and then treat them well. Embrace the Lulz factor: Embrace the shadow of cybersecurity by engaging and encouraging the exact hacker mindset but to improve security — and again, offer an economic incentive. Bug bounties are both subversive and helpful.

Both of these ideas could have a corporate initiative with a budget, KPI and program manager. They could be funded out of HR or professional development, IT or risk management line items. Companies could partner with universities, hacker collectives and government agencies, maybe sponsoring a hackathon or participating in a local cyber wargame activity.

Got Questions

If you could see the future of cybersecurity, what would you ask? Submit your question for a chance to be featured in the next “Ask a Cybersecurity Futurist” column.

If you found this insightful please consider booking me for a keynote talk or a lunch and learn session. Or sign up for the Future of Cybersecurity newsletter.

23:35 • 1 months ago

Michael Ruminer

RAG Disappointment and Azure Open AI Success With RAG

Recently, I have been working on programming small modules for Retrieval Augmented Generation (RAG) using Open AI. Also recently, I did a Coursera class on advanced RAG and have taken in several videos and posts on it. I have used samples from those sources and tested using various LLMs to generate simple Python to perform RAG with Open AI. In general , I have been disappointed with the outco

Disappointment

My general disappointments have come from the self-coded RAG attempts on a single PDF. The basic approach has been:

take a single PDF (not a small PDF — about 43,000 token when uploaded to Open AI API in whole and extract the text using PyPDF chunk the text using at best recursive text and/or sentence transformer; sometimes just naive simple character count split embed the chunks trying Chroma or text-embedding-3-small query the collection using Chroma or FAISS and in one instance also with a simple augmented prompt call the LLM with the initial prompt and context from the embeddings query the prompt was for a list of 10 principles in the document that were all outlined in a single paragraph

It’s nothing sophisticated by any stretch but they corresponded with the examples I had available.

The results — abysmal. I’m not really surprised. I’m not sure how such an approach could do very well with simple chunking, embeddings, and basically running off keyword proximity. But since these were supposed examples of the use of RAG I expected better results. There were only two times I received good results. One of those times I’ll outline below; the other was when I didn’t parse or embed and just passed the entire document as context. Of course, the latter worked well but that was not the exercise I was after.

Success

My successful attempt came when I didn’t create the code directly but used Microsoft Azure Open AI playground. I have no doubt that coding it up would have worked just as well since it relies on the Azure AI infrastructure and the code would be little more than passing a prompt to the Azure LLM instance and getting the results. Here is what it consisted of:

setup of an Azure Open AI instance using gpt-4o-mini model and, I think, the text-embedding-3-small embedding model setup an Azure AI Search instance with semantic search and indexing pointed to a blob storage container with a single PDF in it

I think that was it. I then went to the Azure Open AI Studio playground grounded the model with the search instance, provided my simple prompt, and got back the desired results. Was it the semantic search that made it work well? I suspect it helped a lot. I need to try it without the semantic search and see what happens. Sorry, I forgot to try that scenario.

Recap

All in all, I was very disappointed with the RAG results, as I coded it, especially as they were based on examples or AI generated single document RAG code. But, I was very pleased with the Azure test and I think the semantic search made all the difference.

13:15 • 1 months ago

Damien Bod

Using Elasticsearch with .NET Aspire

This post shows how to use Elasticsearch in .NET Aspire. Elasticsearch is setup to use HTTPS with the dotnet developer certificates and and simple client can be implemented to query the data. Code: https://github.com/damienbod/keycloak-backchannel Setup Two services are setup to run in .NET Aspire. The first service is the official Elasticsearch docker container and deployed […]

This post shows how to use Elasticsearch in .NET Aspire. Elasticsearch is setup to use HTTPS with the dotnet developer certificates and and simple client can be implemented to query the data.

Code: https://github.com/damienbod/keycloak-backchannel

Setup

Two services are setup to run in .NET Aspire. The first service is the official Elasticsearch docker container and deployed using dotnet developer certificates. The second service is an ASP.NET Core application using the Elastic.Clients.Elasticsearch Nuget package. The App.Host project is used to set this up and to link the services together.

Elasticsearch development server

The Elasticsearch container is configured in the program class of the App.Host project. The container is run using HTTPS and takes the Aspire parameters for configuration of the default account.

var elasticsearch = builder.AddElasticsearch("elasticsearch", password: passwordElastic) .WithDataVolume() .RunElasticWithHttpsDevCertificate(port: 9200);

The developer certificates needs to be created and copied to the specific folder inside the Elasticsearch docker container. This is implemented using a shared folder and the Elasticsearch xpack.security.http.ssl properties are set to match. The following three properties are used:

xpack.security.http.ssl.enabled xpack.security.http.ssl.certificate xpack.security.http.ssl.key using System.Diagnostics; using System.IO.Hashing; using System.Text; namespace Aspire.Hosting; // original src: https://github.com/dotnet/aspire-samples/tree/damianedwards/keycloak-sample/samples/Keycloak public static class HostingElasticExtensions { public static IResourceBuilder<ElasticsearchResource> RunElasticWithHttpsDevCertificate(this IResourceBuilder<ElasticsearchResource> builder, int port = 9200, int targetPort = 9200) { if (builder.ApplicationBuilder.ExecutionContext.IsRunMode) { builder .RunElasticWithHttpsDevCertificate() .WithHttpsEndpoint(port: port, targetPort: targetPort) .WithEnvironment("QUARKUS_HTTP_HTTP2", "false"); } return builder; } public static IResourceBuilder<TResource> RunElasticWithHttpsDevCertificate<TResource>(this IResourceBuilder<TResource> builder) where TResource : IResourceWithEnvironment { const string DEV_CERT_DIR = "/usr/share/elasticsearch/config/certificates"; if (builder.ApplicationBuilder.ExecutionContext.IsRunMode) { // Export the ASP.NET Core HTTPS development certificate & private key to PEM files, bind mount them into the container // and configure it to use them via the specified environment variables. var (certPath, _) = ExportElasticDevCertificate(builder.ApplicationBuilder); var bindSource = Path.GetDirectoryName(certPath) ?? throw new UnreachableException(); if (builder.Resource is ContainerResource containerResource) { builder.ApplicationBuilder.CreateResourceBuilder(containerResource) .WithBindMount(bindSource, DEV_CERT_DIR, isReadOnly: false); } builder .WithEnvironment("xpack.security.http.ssl.enabled", "true") .WithEnvironment("xpack.security.http.ssl.certificate", $"{DEV_CERT_DIR}/dev-cert.pem") .WithEnvironment("xpack.security.http.ssl.key", $"{DEV_CERT_DIR}/dev-cert.key"); } return builder; } private static (string, string) ExportElasticDevCertificate(IDistributedApplicationBuilder builder) { var appNameHashBytes = XxHash64.Hash(Encoding.Unicode.GetBytes(builder.Environment.ApplicationName).AsSpan()); var appNameHash = BitConverter.ToString(appNameHashBytes).Replace("-", "").ToLowerInvariant(); var tempDir = Path.Combine(Path.GetTempPath(), $"aspire.{appNameHash}"); var certExportPath = Path.Combine(tempDir, "dev-cert.pem"); var certKeyExportPath = Path.Combine(tempDir, "dev-cert.key"); if (File.Exists(certExportPath) && File.Exists(certKeyExportPath)) { // Certificate already exported, return the path. return (certExportPath, certKeyExportPath); } else if (Directory.Exists(tempDir)) { Directory.Delete(tempDir, recursive: true); } Directory.CreateDirectory(tempDir); var exportProcess = Process.Start("dotnet", $"dev-certs https --export-path \"{certExportPath}\" --format Pem --no-password"); var exited = exportProcess.WaitForExit(TimeSpan.FromSeconds(5)); if (exited && File.Exists(certExportPath) && File.Exists(certKeyExportPath)) { return (certExportPath, certKeyExportPath); } else if (exportProcess.HasExited && exportProcess.ExitCode != 0) { throw new InvalidOperationException($"HTTPS dev certificate export failed with exit code {exportProcess.ExitCode}"); } else if (!exportProcess.HasExited) { exportProcess.Kill(true); throw new InvalidOperationException("HTTPS dev certificate export timed out"); } throw new InvalidOperationException("HTTPS dev certificate export failed for an unknown reason"); } }

When the App.Host project is started, the Elasticsearch containers boot up and the server can be tested using the “_cat” HTTP Get requests or the default base URL will give a server information about Elasticsearch.

https://localhost:9200/_cat

Elasticsearch client

The Elasticsearch client was implemented using the Elastic.Clients.Elasticsearch Nuget package. The client project in .NET Aspire needs to reference the Elasticsearch server using the WithReference method.

builder.AddProject<Projects.ElasticsearchAuditTrail>( "elasticsearchaudittrail") .WithExternalHttpEndpoints() .WithReference(elasticsearch);

Elasticsearch can be queried used a simple query search.

public async Task<IEnumerable<T>> QueryAuditLogs(string filter = "*", AuditTrailPaging auditTrailPaging = null) { var from = 0; var size = 10; EnsureElasticClient(_indexName, _options.Value); await EnsureAlias(); if (auditTrailPaging != null) { from = auditTrailPaging.Skip; size = auditTrailPaging.Size; if (size > 1000) { // max limit 1000 items size = 1000; } } var searchRequest = new SearchRequest<T>(Indices.Parse(_alias)) { Size = size, From = from, Query = new SimpleQueryStringQuery { Query = filter }, Sort = BuildSort() }; var searchResponse = await _elasticsearchClient .SearchAsync<T>(searchRequest); return searchResponse.Documents; }

See the source code: https://github.com/damienbod/keycloak-backchannel/blob/main/AuditTrail/AuditTrailProvider.cs

Notes

With this setup, it is easy to develop using Elasticsearch as a container and no service needs to be implemented on the developer host PC. Setting up HTTPS is a little bit complicated and it would be nice to see this supported better. The development environment should be as close as possible to the deployed versions. HTTPS should be used in development.

Links

https://learn.microsoft.com/en-us/dotnet/aspire/search/elasticsearch-integration

https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html

https://www.elastic.co/products/elasticsearch

https://github.com/elastic/elasticsearch-net

https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-simple-query-string-query.html

04:24 • 1 months ago

Saturday, 14. September 2024

Mike Jones: self-issued

OAuth 2.0 Protected Resource Metadata draft addressing reviews since IETF Last Call

Aaron Parecki and I published a new version the “OAuth 2.0 Protected Resource Metadata” specification that addresses the review comments received since the IETF Last Call. Per the history entries, the changes were: Added metadata values declaring support for DPoP and mutual-TLS client certificate-bound access tokens. Added missing word caught during IANA review. Addressed ART, […]

Added metadata values declaring support for DPoP and mutual-TLS client certificate-bound access tokens. Added missing word caught during IANA review. Addressed ART, SecDir, and OpsDir review comments by Arnt Gulbrandsen, David Mandelberg, and Bo Wu, resulting in the following changes: Added step numbers to sequence diagram. Defined meaning of omitting bearer_methods_supported metadata parameter. Added internationalization of human-readable metadata values using the mechanism from [RFC7591]. Added resource_name metadata parameter, paralleling client_name in [RFC7591]. Added Security Considerations section on metadata caching. Used and referenced Resource Identifier definition. Added motivating example of an email client to intro.

The specification is available at:

https://www.ietf.org/archive/id/draft-ietf-oauth-resource-metadata-09.html

01:16 • 1 months ago

Friday, 13. September 2024

Georg C. F. Greve

Coming full circle: How Self-Sovereign Identity Could Lose Its Way

“A picture showing the knight of Self Sovereign Identity losing his way, going into the valley of doom where certain death awaits.” Self-Sovereign Identity (SSI) offers a robust and scale-able solution to the challenges of digital identity that is secure while also respecting privacy, freedom, and human rights. However, a mix of political and industry pressures, along with a push for quick ado

“A picture showing the knight of Self Sovereign Identity losing his way, going into the valley of doom where certain death awaits.”

Self-Sovereign Identity (SSI) offers a robust and scale-able solution to the challenges of digital identity that is secure while also respecting privacy, freedom, and human rights. However, a mix of political and industry pressures, along with a push for quick adoption, risks turning SSI into just another marketing buzzword.

Nowhere is this trend more visible than in the mobile Drivers License (mDL) and the ongoing conversations around the second version of European Unions “electronic IDentification, Authentication and trust Services” (eIDAS).

Old Interests, New Technology

The first version of eIDAS was an adoption failure. While it was not providing privacy by design, technologically it was sufficiently correct to function. It did so by giving a central role to Trust Service Providers (TSPs) and Certificate Authorities (CAs).

These intermediaries sought to exploit their government issued monopoly and consequently choked off adoption for the first version of eIDAS.

In doing so, they repeated the same mistake made on the wider web, where excessive pricing led to stagnation in the adoption of encrypted communications. In 2014, only 10 short years ago, the vast majority of all connections was not using any kind of transport layer security (TLS).

HTTPS adoption became mainstream only between 2015 and 2019 — via Matthew Greene, article linked below

It was the Snowden Revelations that changed everything. But not in the way the intermediaries had hoped.

While I am sure many were rubbing their hands, and expecting the pipe they were sitting on to finally start flowing, everyone in the industry knew that predatory pricing was to blame for the low adoption. So in November 2014, Let’s Encrypt was founded in order to provide for free what Certificate Authorities are selling at premium prices.

Today, Let’s Encrypt provides over 55% of all certificates used on the web. Which provides for two conclusions:

Let’s Encrypt has become the single point of failure for security on the internet. And the addressable market for TSPs has shrunk dramatically, and keeps shrinking.

Simultaneously, the costs for TSPs are growing. Not only in terms of technology cost. But also in terms of compliance cost. Because there is a whole industry of consultants and auditors that service TSPs, extracting six figure payments each year for initial certification and re-certification. Growing cost and shrinking markets are not a great prospect for any industry.

So when the new version of eIDAS came along, promising to switch to Self Sovereign Identity for privacy by design, security, and elimination of intermediaries, the TSPs deployed their lobbyists to Brussels to stop this threat to their remaining business.

The result was a bureaucratic monster, for which some people at DICE 2024 in Zurich voiced doubt whether it could even be implemented in practice.

Intermediaries are once again involved in every step. And all the technological choices are based on legacy Web2.0 technology shaped by the TSPs, without privacy by design, and with well known and regularly exploited security issues.

So TSP lobbying successfully defended their government mandated monopoly over the digital domain, and the fundamental issue that stopped adoption of the first version of eIDAS remains unresolved. Only the future will show whether this second attempt of the TSPs at monetizing all digital interactions will fare any better.

But that isn’t even the biggest problem.

The Web is a bad idea for personal identity

The web is a fundamentally unsound technology choice for personal identity, for a variety of reasons, starting with security.

The early web wasn’t really designed for privacy or security. Its original design was inspired by the need for institutional information sharing. Invented in 1989, security was added in 1995 when Netscape came up with Secure Sockets Layer (SSL), which led the way for today’s Transport Layer Security (TLS). This allowed encryption of communication, but all security was dependent on key management.

Because it was the only practical way at the time, intermediaries became the root of all trust for virtually all users. On top of those issues, secure communication requires two parties who may know nothing about one another to negotiate a secure key to use for encryption with one another. That negotiation must happen over the same channel that is being used for encrypted communication.

Which is a bit like yelling the combination of the key pad to your house through the open window over to your neighbour so he can come in and water the plants. If there are no eavesdroppers at that time: Great. If there are, they now also have the key. You’ll find out whether you were safe in a few weeks when you get back from vacation.

Diffie-Hellman key exchange was invented to solve that particular problem. But it is only secure if used correctly and keys are being rotated and refreshed often. Re-use of static keys or weak parameters can weaken security considerably.

On top of that, cryptography is an ever evolving field, and the need for web servers to support older browsers means that sometimes, outdated or known vulnerable settings are still being allowed. Managing all that is complex. It requires organisational level security teams.

And much like charging your modern Electric Vehicle using a coal powered steam engine to drive a generator, trying to fast track adoption for Self Sovereign Identity by using Web based Decentralized Identifier (DID) methods effectively negates all the conceptual and architectural progress that has been made with SSI.

All the underlying security issues our industry has struggled with for the past 30+ years, all the conceptual weaknesses of the Web, all the privacy issues and compromises also apply to SSI when using did:web and its descendants.

In practical terms this translates into most people once again finding themselves forced to entrust their personal identity to intermediaries in the form of custodial wallets and platforms. Just as they are doing today with Google, Facebook, Amazon, Apple, Microsoft.

Most people will rent usage of their own identities, but they will have no ownership or agency. And those intermediaries will once again have full visibility and control over everything people are doing online.

That is a steep price to pay. Is it worth it at least in terms of security?

Even these large organisations get it wrong often enough for data leaks and other issues to occur frequently. When using the Web for personal identity, these compromises will have consequences beyond anything we have seen so far. What happens when the compromise has enabled the attacker to provide legally valid signatures in your name to sign contracts, or consent to sharing your personal pictures with the wrong parties?

And that’s only the start of the valley of doom.

The Web is based on top of the Domain Name System (DNS). Which is a system in which you rent usage of certain human readable domains on a yearly basis. Without it, the Web falls apart.

So let’s assume enlightened users rent their own domains to become independent from the intermediaries. Now they can move between them in case one of them has issues, goes out of business, or gets acquired by a billionaire with a questionable political agenda. Just move your domain and you’re good, right?

Not so fast.

Not only is it possible you may lose your domain, either due to changes in DNS policy, legal aspects, such as trademarks, or because you’ve missed a yearly payment.

If a personal identity can be lost that easily, was it ever really yours?

But there is an even bigger issue.

The W3C Decentralized Identifier (DID) standard was designed to provide for unique, non-correlatable identifiers for each individual interaction. Which sounds great in theory. But how non-correlatable are did:tdw:johnsmith.eu:223499182 and did:tdw:johnsmith.eu:673403151?

Would you be able to tell they might both belong to the same person, whose name might be John Smith?

Because the domain is always an intrinsic part of each Web based DID method, none of them allow for personal ownership without complete loss of pseudonymity, and non-correlatability.

So whatever pseudonymity and privacy can be achieved leads back full circle to large intermediaries and custodial wallets — just like the federated identity infrastructure of today.

“Anakin Skywalker at a crossroads. Go down the Web based SSI method path, and become Darth Vader, or choose a world of privacy, freedom and security?” TL;DR: Don’t do it, Anakin!

Web technologies are a fundamentally poor choice for personal identity.

They can work in domains where privacy and non-correlatability are not required, for instance organisational identity. But for personal identity, they compromise almost everything that Self Sovereign Identity (SSI) was intended and designed for.

You cannot retrofit privacy and security onto an architecture that wasn’t built for them. Just look at Email. It has the same problem. Decades of adding layers of complexity and auxiliary systems have kept moving the problem somewhere else in the stack, to another component. Lots of snake oil has been sold. Some people got very rich this way. But the fundamental issues remain unsolved.

We should not repeat that same mistake for Self Sovereign Identity.

No system that is intended for personal identity should ever use did:web, did:tdw, or any of the other web based DID specifications. The concepts of SSI can solve all the issues above. They can provide people with real agency and control over their own identity. But not when using the Web.

The solution is simple.

Stop using Web based DID methods for personal identity!

Humans are not organizations. Stop using Web based DID methods for personal identity. did:web and its descendants are very useful for some organizational use cases, and for prototyping new ideas. They have NO place when it comes to personal identity, or use cases that require strong privacy or security.

There are (at least) three superior alternatives that come to mind, all of which can also provide a secondary channel for key negotiation.

Stay tuned.

08:59 • 1 months ago

@_Nat Zone

グーグルがパスポートのウォレット搭載を発表〜Google Walletで実現する新時代のデジタルID

米国時間2024年9月12日（日本時間9月13日）、グーグルが米国パスポートのグーグルウォレットへの搭載を発表しました。近年、日常生活のあらゆる面でデジタル化が進んでいますが、その中でも特に注目を集めているのがデジタルIDです。Googleは昨年から一部の州発行のデジタルIDを…

米国時間2024年9月12日（日本時間9月13日）、グーグルが米国パスポートのグーグルウォレットへの搭載を発表しました。

近年、日常生活のあらゆる面でデジタル化が進んでいますが、その中でも特に注目を集めているのがデジタルID1です。Googleは昨年から一部の州発行のデジタルIDをGoogle Walletに保存できる機能の展開を開始しましたが、さらなる進化を遂げようとしています。

まもなく、Google Walletに新しいタイプのデジタルIDを導入するベータテストが始まります。この新機能により、より多くの人々が、より多くの場所でデジタルIDを作成し保存できるようになります。注目すべきは、今回から米国パスポートを使用してデジタルIDを作成できるようになることです。

このデジタルIDパスは、一部のTSA（運輸保安局）チェックポイントで使用可能となり、国内旅行時の空港での時間と手間を省くことができます。

デジタルIDパスの作成プロセスは非常に簡単です：

Google Walletアプリで「米国パスポートでIDパスを作成」というプロンプトを選択パスポートの裏面にあるセキュリティチップをスキャンする指示に従う本人確認のためにセルフィービデオを撮影数分以内にIDパスの準備が整ったという通知を受け取る https://storage.googleapis.com/gweb-uniblog-publish-prod/original_videos/PassportinGoogleWallet_Revised_R3.mp4 Google Wallte へのパスポート登録の様子　（出所）Google

現在、このIDパスは一部のTSAチェックポイントでのみ受け付けられていますが、Googleはパートナー企業と協力して、将来的にはアカウント復旧、本人確認、さらにはレンタカーの利用など、より多くの場面でデジタルIDを使用できるよう取り組んでいます。

ただし、この技術はまだ初期段階にあるため、Google WalletのデジタルIDは現時点では物理的なIDの代替とはなりません。旅行時には引き続き物理的なIDの携帯が必要です。

セキュリティ面では、Google Walletは「プライベート・バイ・デザイン」原則によって個人情報を保護するように設計されたAndroidの高度なセキュリティを使用してデジタルIDを安全に保管します。IDパスは暗号化されて保存され、利用には指紋認証やPIN、パスコードでの認証が必要となります。また、本人確認時には共有される情報を事前に確認できるため、ユーザーが情報を管理できます。

さらに、カリフォルニア州のAndroidユーザーは既に州発行のID2や運転免許証をGoogle Walletアプリに保存できるようになっており、今後数か月以内にアイオワ州、ニューメキシコ州、オハイオ州でも同様のサービスが展開される予定です。

コロラド州では、MyColoradoアプリ内に新しいリーダーを導入し、州内の企業がデジタルIDを安全かつ簡単に受け入れられるようにする取り組みも始まっています。

このように、GoogleはデジタルIDの利便性と適用範囲を拡大し続けており、私たちの日常生活をよりスマートで効率的なものに変えようとしています。デジタルIDの未来に、今後も注目が集まりそうです。

（参考資料）

New Google Wallet features for travelers and commuters (2024-09-12) How we’re working to make digital identity a reality for everyone (2024-09-12)

05:01 • 1 months ago

Thursday, 12. September 2024

Bill Wendels Real Estate Cafe

Leverage $25K downpayment assistance to protect homebuyers & revitalize buyer agency movement

Three decades ago was invited to participate in a collaborative effort to develop President Bill Clinton’s National Homeownership Strategy. Those 100 recommendations raised homeownership from… The post Leverage $25K downpayment assistance to protect homebuyers & revitalize buyer agency movement first appeared on Real Estate Cafe.

Three decades ago was invited to participate in a collaborative effort to develop President Bill Clinton’s National Homeownership Strategy. Those 100 recommendations raised homeownership from…

The post Leverage $25K downpayment assistance to protect homebuyers & revitalize buyer agency movement first appeared on Real Estate Cafe.

23:39 • 1 months ago

The Pragmatic Engineer

The Pulse #106: Why does the EU tech sector lag behind the US?

Also: non-AI companies like Klarna want to look like AI ones, unusual details about Google, Amazon, and Microsoft levels, and more

The Pulse is a series covering insights, patterns, and trends within Big Tech and startups. Notice an interesting event or trend? Send me a message.

Today, we cover:

Industry pulse. Apple announces AI features you can’t use yet, Stripe is a hit with popular AI companies, Confluent buys an up-and-coming competitor before it grows into a threat, game studio…

16:00 • 1 months ago

@_Nat Zone

｢マイナ免許証｣25年3月導入へ　住所変更ワンストップで。更新時講習もオンラインに。

運転免許証とマイナンバーカードの一体化が、まもなく現実のものとなります。この画期的な変更は、私たちの日常生活に大きな影響を与える可能性を秘めています。警察庁が9月12日に発表した新しい方針によると、2025年3月から「マイナ免許証」が導入される予定です。これは、現在の運転免…

運転免許証とマイナンバーカードの一体化が、まもなく現実のものとなります。この画期的な変更は、私たちの日常生活に大きな影響を与える可能性を秘めています。

警察庁が9月12日に発表した新しい方針によると、2025年3月から「マイナ免許証」が導入される予定です。これは、現在の運転免許証の機能をマイナンバーカードに統合するという画期的な取り組みです。

マイナ免許証制度の概要

マイナンバーカードは、 2016年に導入された個人番号制度の一環として発行されている身分証明書です。このカードには、すでに様々な個人情報が ICチップに記録されていますが、新たに運転免許証の情報も追加されることになります。

具体的には、免許保有者の写真、免許の有効期間、取得している免許の種類、そして眼鏡着用などの条件といった情報がマイナンバーカードのICチップに記録されます。これにより、1枚のカードで身分証明と運転資格の証明が可能になります。

マイナンバーカードと運転免許証を一体化手続きは、運転免許センターや免許更新を扱う警察署で受け付ける方向で調整が進められています。これは、運転免許証の暗証番号を覚えている人は、スマホでできるようにして欲しいところではあります。

また、今の法律では免許取消や停止処分の対象者について、所在が分からず処分ができていない場合に警察官が対象者を見つけた場合、免許証を預かる制度があります。2025年3月24日以降はこの制度がなくなり、「出頭命令書」を出す制度に変わります。

マイナ免許証のメリット

この一体化には多くのメリットがあります。特に注目すべきは、住所変更手続きの簡素化です。現在、引っ越しの際には自治体と警察の両方に別々に届け出る必要がありますが、マイナ免許証では自治体での手続きだけで完結します。

さらに、免許更新時の利便性も向上します。優良運転者と一般運転者は、マイナポータルという個人向けサイトで講習動画を視聴するだけで、更新時の講習を受けたとみなされます。これにより、更新手続きにかかる時間と手間が大幅に削減されます。ただし、すべての手続きがオンラインで完結するわけではありません。写真撮影や視力検査などは、従来通り免許センターなどで実施されます。これは、安全運転に必要な基本的な身体能力を確認する重要な過程だからです。

マイナ免許証の導入は任意であり、既存の運転免許証も併用して使用することができます。しかし、マイナ免許証のみを保有する場合、新規取得時や更新時の手数料が比較的安くなるという経済的なメリットもあります。

免許取得免許更新現行の免許証2350円2850円マイナ免許証1550円2100円現行＋マイナ2450円2950円2025年3月24日以降の手数料懸念点の解消

この変更は、デジタル化が進む現代社会における行政サービスの効率化の一環と言えるでしょう。すでに「マイナ保険証」として健康保険証の機能を統合する取り組みも進められており、 2024年7月末時点でマイナンバーカード保有者の80％が登録を完了しています。

誤まった紐付けについて

しかし、このようなデジタル化の急速な進展には懸念の声も上がっています。特に、個人情報の管理や情報セキュリティに関する不安は根強く存在します。政府は、これらの不安に対して丁寧な説明と十分な対策を講じる必要があるでしょう。例えば、以前、マイナンバーカードと健康保険証の連携でミスが発生しました。マイナ免許証ではこのことの繰り返しを防ぐための対策が施されていることを訴えかけていく必要があるでしょう。現在検討されているのは、申請時に本人がマイナンバーカードと運転免許証の両方を持参し、係員が申請者の目の前で両方の顔写真が一致することを確認するということです。これでちゃんと脅威に対策できているのかは要検討ですが。

落とした・無くした時の再発行に時間がかかる点

また、マイナ免許証に一本化した場合、これを落としたりした場合の再発行にかかる時間の問題も心配の一つです。車の運転には免許証の携帯が必要で、これに対応するために現行の免許証は落としたりした時に即日再発行が可能ですが、マイナンバーカードは１ヶ月半ほどかかることもあります。この間運転できなくなったら困ります。

これは、実は、現行の運転免許証を再発行してもらうことで回避できます。結局２枚に戻ってしまいますが。

今後の予定

この新しいシステム（マイナ免許証）の運用開始に向けて、警察庁は道路交通法の改正を2025年3月24日に施行する予定です。さらに、2024年9月13日から30日間、この新しい運用方法について一般の人々から意見を募集します。

マイナ免許証の導入は、日本社会のデジタル化における重要なマイルストーンとなります。この変革が私たちの生活にどのような影響を与えるのか、今後の展開に注目が集まります。

参考記事

» 【マイナ免許証】来年3月24日開始へ　マイナカードと運転免許証一体化　現行免許証の選択・併用も可能だが手数料はマイナ免許証より割高に？（FNNプライムオンライン（フジテレビ系）） – Yahoo!ニュース https://t.co/5lCDeGcip8
— Nat Sakimura/崎村夏彦 (@_nat) September 12, 2024

｢マイナ免許証｣25年3月導入へ　更新時講習オンラインに：日本経済新聞https://t.co/vImZKSbCp9
— Nat Sakimura/崎村夏彦 (@_nat) September 12, 2024

03:06 • 1 months ago

Wednesday, 11. September 2024

Patrick Breyer

Neuester EU-Anlauf zur Chatkontrolle: Werden Messenger-Dienste in Europa gesperrt?

Ungarn hat den EU-Innenministern diese Woche einen neuen Vorschlag zur Einführung der Chatkontrolle vorgelegt, der von POLITICO geleakt wurde. Der Einsatz „künstlicher Intelligenz“ zur Suche nach verdächtigen Bildern und Chats soll Messenger- und Chatanbietern danach freigestellt bleiben. Zur Suche nach bekanntem illegalem Material bleibt es aber bei dem weltweit einzigartigen Vorhaben, die massenhafte, verdachtslose und nach BKA-Angaben höchst fehleranfällige Kontrolle aller privaten Chats für sämtliche Diensteanbieter verpflichtend zu machen, selbst wenn dazu die bisher sichere Ende-zu-Ende-Verschlüsselung von Messengerdiensten ausgehebelt werden muss. Die EU-Regierungen sollen sich bis zum 23. September zu dem Vorstoß positionieren, die EU-Innenminister sollen ihn am 10. Oktober beschließen. Die Messengeranbieter Signal und Threema haben bereits angekündigt, solche Überwachungsfunktionen keinesfalls in ihre Apps einzubauen und ihre Dienste in der EU gegebenenfalls komplett einzustellen.

“Statt Kindern durch sicherere Apps zu helfen sich im Netz vor Erpressung und Ausbeutung zu schützen, werden Missbrauchsopfer durch einen realitätsfremden und vor Gericht zum Scheitern verurteilten Gesetzentwurf im Regen stehen gelassen,” kritisiert Dr. Patrick Breyer, ehemaliger Europaabgeordneter der Piratenpartei und Mitverhandler der kritischen Position des Europäischen Parlaments. “Unsere Polizei mit großteils irrelevanten Hinweisen auf längst bekanntes Material zu überfluten, stoppt keinen Missbrauch, sondern erschwert seine Aufdeckung zusätzlich. Falls diese Chatkontrolle kommt, werden wir Bürger gängige sichere Messenger nicht mehr nutzen können – das bedeutet wir verlieren den Kontakt zu unseren Freunden und Kollegen in der ganzen Welt. Wollen wir wirklich, dass die EU weltweit führend beim Abhören unserer Smartphones und der flächendeckenden Überwachung der Chats von Millionen gesetzestreuer Bürger wird?”

Breyer ruft dazu auf, Druck auf Bundesinnenministerin Faeser und Bundesjustizminister Buschmann zu machen: “Nur dagegen zu sein und dann überstimmt zu werden, reicht nicht zur Verteidigung unseres digitalen Briefgeheimnisses und sicherer Verschlüsselung. Die Bundesregierung muss jetzt dringend Verbündete suchen, um eine Sperrminorität und eine grundlegende Überarbeitung zu erreichen. Die Sicherheit unserer Kommunikation steht auf dem Spiel!”

Info-Website: https://www.chatkontrolle.de

Hilf jetzt mit die Chatkontrolle zu stoppen

21:50 • 1 months ago

Tuesday, 10. September 2024

The Pragmatic Engineer

AI Tooling for Software Engineers: Rolling Out Company-Wide (Part 3)

Why are some companies not using GenAI yet? Also: AI guidelines and practical approaches for embracing LLM tools for software development on the individual developer, and organizational level

・住民記録システム標準仕様書【第5.1版】等の改正概要PDF
・住民記録システム標準仕様書【第5.1版】
・戸籍附票システム標準仕様書【第3.1版】

Before we start: you can now see use “table of contents” quick navigation on the right side of each article, when reading the newsletter on the web. Just click on the sidebar, and you can navigate this article — and all other The Pragmatic Engineer articles. See it in action on the web. Happy browsing!

There’s no shortage of big claims about what LLM tools will be able to do, or should be able to do in the software engineering field. But what do they actually do, right now? We asked software engineers who regularly use these tools, and engineering leaders who oversee these tools in their organizations.

This article is based on a survey of 216 professionals and is the third and final part of a mini-series on GenAI tooling. It covers how these tools are being used ‘day-to-day’ in tech workplaces, and what engineers think about them. Today, we cover:

AI usage guidelines. A quarter of respondents follow company-wide usage guidelines. Interestingly, a minority of companies have banned GenAI tools over security and copyright worries.

Internal LLMs at Meta, Microsoft, Netflix, Pinterest, Stripe. Large, fast-moving companies not only embrace GenAI tools, but build their own internal versions for their engineers. Vendors are starting to offer similar boxed products.

Reservations and concerns. Most common reservations, and how to overcome them – and why devs tend to start using LLMs regularly.

Advice for devs to get started with AI tools. Start small, verify outputs, don’t “outsource” coding and other advice.

Advice for engineering leaders to roll out AI tooling, org-wide. A roundup of how companies adopted these tools successfully: guidelines, tooling, training, and how these impact junior engineers.

Measuring the impact of GenAI tools. Most engineering leaders say these tools have no visible or measurable impact – at least not yet. We suspect this is because the impact is hard to measure: and not due to the lack of impact.

AI strategy. Why do companies incorporate GenAI into their software engineering workflow? Experimentation and hoping for increased productivity are two big reasons.

In Part 1 of this series, we covered:

Survey overview

Popular software engineering AI tools

AI-assisted software engineering workflows

The good

The bad

What’s changed since last year?

Part 2 was about:

What are AI tools similar to?

State of AI tooling in 2024: opinions

Critiques of AI tools

Changing views about AI tooling over time

Which tasks can AI already replace?

Time saved – and what it’s used for

Now, let’s dive into this final part of this mini-series.

1. AI usage guidelines across companies

We asked survey participants “how is AI tooling used for development at your company?” The responses reveal different approaches:

Common ways LLM tooling is used at tech companies

The most referenced approaches:

No formal guidelines. Around 25% of respondents (53 out of 216) say their company has no recommendations about AI tooling. People use it as they see fit.

Some guidelines. 25% (50 respondents) say their workplaces have rules and guidelines for AI tooling.

AI tools banned. Circa 12.5% (25 responses) say their businesses ban usage of AI tools, mostly due to concerns about code security, and potential copyright infringement. We previously covered how several open source projects have banned AI-generated code commits for this reason.

Working on guidelines. 7% of respondents (15 people) share that their company is trialing AI tooling, or is in the process of adopting guidelines.

Strongly encourage AI tool usage. 6% of respondents (12 people) work at places which encourage using these tools wherever possible.

Common features of guidelines across workplaces, based on survey responses:

Specifying which tools and LLM models may be used

No inputting of sensitive information into AI tools like ChatGTP

No entering of internal (closed-sourced) code into AI chat tools

It’s pretty clear some guidelines are responses to fears that LLMs may retain the data employees input and use it for training. This is also a reason why a handful of respondents shared that their companies go through the added complexity of running LLMs on their own infrastructure. It’s a reminder that LLM solutions which don’t store company data have a strong selling point for tech companies.

2. Internal LLMs at Meta, Netflix, Pinterest, Stripe

Only a fraction of respondents say their companies strongly encourage the use of LLM tools, but some of these are cutting-edge market leaders in tech. Let’s take a look at how a well-built internal LLM can help a business.

@_Nat Zone

戸籍のフリガナの届出は2026年5月26日まで。届け出がない場合、市区町村長により職権でフリガナ記載

2024年9月10日、戸籍にフリガナを記載する改正戸籍法の施行日が2025年5月26日と閣議決定されました。これにより、戸籍へのフリガナ記載に向けた準備が本格化します。改正の背景と目的この法改正の主な目的は、の３点です。現在の戸籍では氏名の読み方が不明確な場合があり、行政…

2024年9月10日、戸籍にフリガナを記載する改正戸籍法1の施行日が2025年5月26日と閣議決定されました。これにより、戸籍へのフリガナ記載に向けた準備が本格化します。

改正の背景と目的

この法改正の主な目的は、

行政のデジタル化基盤整備の促進本人確認情報としての利用各種規制の潜脱行為の防止

の３点です。

現在の戸籍では氏名の読み方が不明確な場合があり、行政手続きや本人確認の場面で不都合が生じていました。

新制度の概要

2025年5月26日以降、戸籍にカタカナでフリガナが記載されるようになります。主な変更点は以下の通りです：

新生児の場合、出生届に記載されたフリガナが戸籍に反映されます。すでに戸籍に氏名が記載されている人も、フリガナの届け出が必要になります。フリガナには一定の基準が設けられます。届け出の方法と期限制度開始から1年以内に「振り仮名の届出」をする必要があります。戸籍の筆頭者は氏名、それ以外の人は名前のフリガナを届け出られます。届け出は書面またはマイナポータル（予定）を通じて行うことができます。注意点届け出がない場合、市区町村長により職権でフリガナが記載される可能性があります。現在使っている読み方と異なる読み方が登録されてしまう可能性があります。フリガナは一般に認められる読み方でなければなりません。つまり、キラキラネームはNGです。逆に、一般の読み方以外の読み方を現在利用しておられる方は、パスポートや預金通帳など証拠を提示することによってそれを登録することができるようになります。経過措置として、一度のみ家庭裁判所の許可なしでフリガナの変更が可能です。自治体の対応

施行に向けて、自治体は以下の準備を進めています：

正確かつ迅速な通知発送：本籍地の市町村長が戸籍に氏名の振り仮名を記載する前提として、戸籍に記載される予定の氏名の振り仮名等を認識する機会を確保することとしています。具体的には、住民票において市町村が事務処理の用に供するため便宜上保有する情報等を参考に、本籍地の市町村長から皆様に、氏名の振り仮名に関する情報を通知することとしています。この通知は、改正法の施行日（令和７年５月26日）から遅滞なく送付することとしています。オンラインとアナログの併用:氏名の振り仮名の届出については、当該届出をする者の本籍地又は所在地の市町村に行うこととなりますが、窓口への出頭及び郵送による方法が可能です。また、マイナポータルを利用する方法についても現在検討中です。

この制度変更により、戸籍におけるフリガナの取り扱いが明確になり、行政手続きの効率化や本人確認の精度向上が期待されています。市民の皆様は、制度開始後1年以内にフリガナの届け出を行うことを忘れずに、新しい制度に対応していく必要があります。

詳しい情報は、法務省のサイトをご覧ください。

【9月11日追記】

総務省から住民記録システム標準仕様書【第5.1版】等の公表がありました

令和6年9月11日

住民記録システム標準仕様書【第5.1版】等の公表

「自治体システム等標準化検討会（住民記録システム等標準化検討会）」（座長：庄司昌彦武蔵大学社会学部教授）において、住民記録システム標準仕様書【第5.1版】及び戸籍附票システム標準仕様書【第3.1版】が取りまとめられましたので、公表します。

公表資料

※ 住民記録システム標準仕様書【第5.1版】及び戸籍附票システム標準仕様書【第3.1版】については、以下の総務省ホームページにおいても公開します。

総務省トップ > 組織案内 > 研究会等 > 自治体システム等標準化検討会
https://www.soumu.go.jp/main_sosiki/kenkyu/jichitaishisutemu_hyojunka/index.html

13:14 • 1 months ago

Monday, 09. September 2024

Michael Ruminer

My Surprising Llama 3.1 8B Post Stats

Back in July, I made a post here in Medium on running Llama 3.1 8B locally not long after the model dropped. It is my most viewed and most read post by between 1 and 2 orders of magnitude based on my poor general reach at this writing. It was surely the timeliness and high interest in all things Llama 3.1 at that moment. I looked at the stats and they were enlightening on the distribution of refer

Here are the stats:

The whole set was surprising to me. I made a post about this article only to LinkedIn so I was surprised at only one linkedin.com referral. I thought I had clicked on it more times than that. :-) It kind of makes me sad. Wow, Google. Not sure what I did to show up in some search results but I wish I did that more often. 146 in email, IM and direct. YCombinator… really. I’m honored. Gemini.google.com — interesting. Twitter — 2; very surprising as I didn’t post there and haven’t used Xitter in about a year for what would be obvious reasons for some people. Getliner.com — I don’t even know what that is. Sorry getliner.com. Identosphere.net — mildly surprised. I get picked up in the blog aggregator as I have in the past posted more reliably on digital identity and verifiable credentials.

12:00 • 1 months ago

Damien Bod

Implement OpenID Connect Back-Channel Logout using ASP.NET Core, Keycloak and .NET Aspire

This post shows how to implement an OpenID Connect back-channel logout using Keycloak, ASP.NET Core and .NET Aspire. The Keycloak and the Redis cache are run as containers using .NET Aspire. Two ASP.NET Core UI applications are used to demonstrate the server logout. Code: https://github.com/damienbod/keycloak-backchannel Setup The applications are run and tested using .NET Aspire. […]

Code: https://github.com/damienbod/keycloak-backchannel

Setup

The applications are run and tested using .NET Aspire. The UI applications are setup using the OpenID Connect code flow with PKCE and OAuth PAR. Two docker containers are used, one for the Redis cache and one for the Keycloak server.

.NET Aspire Setup

The AppHost project in .NET Aspire is used to configure the different services. The Keycloak container is run using HTTPS with develop certificates. The ASP.NET Core applications are run using development certificates as well. For the Keycloak server to access the docker host, it must register the same developer certificates from the host, or disable the certificate trust manger inside the container. This works good, but should only be setup like this in development. The different project must reference each other as required. To use Redis and Keycloak, the Aspire Nuget packages for these containers need to be installed.

var keycloak = builder.AddKeycloakContainer("keycloak", userName: userName, password: password, port: 8080) .WithArgs("--features=preview") // for more details regarding disable-trust-manager see https://www.keycloak.org/server/outgoinghttp#_client_configuration_command // IMPORTANT: use this command ONLY in local development environment! .WithArgs("--spi-connections-http-client-default-disable-trust-manager=true") .WithDataVolume() .RunWithHttpsDevCertificate(port: 8081); var cache = builder.AddRedis("cache", 6379) .WithDataVolume(); var mvcpar = builder.AddProject<Projects.MvcPar>("mvcpar") .WithExternalHttpEndpoints() .WithReference(keycloak) .WithReference(cache); var mvcbackchanneltwo = builder.AddProject<Projects.MvcBackChannelTwo>("mvcbackchanneltwo") .WithExternalHttpEndpoints() .WithReference(keycloak) .WithReference(cache);

Keycloak OpenID Connect client configuration

The Keycloak client should have the backchannel logout activated. The container uses the localhost applications from the docker host and so the host.docker.internal domain is used. The logout endpoint is implemented in the ASP.NET Core application.

ASP.NET Core Logout

Each ASP.NET Core application that supports the back-channel logout must have a server implementation and provide a web hook for the identity provider (Keycloak) logout event. If multiple instances are running, one can send a logout event to Keycloak. This ends the session on the identity provider and sends a logout post request to all server backends hosting the UI application. The logout event is handled and persisted to a distributed cache. For all other instances which request data from the server for the same user and session, the session is ended and the user must authentication again.

Logout request Webhook for logout event from Keycloak server for all instances Persist event to cache if missing and logout Logout UI on next HTTP request for user sessions

services.AddTransient<CookieEventHandler>(); services.AddSingleton<LogoutSessionManager>(); services.AddHttpClient(); services.Configure<AuthConfiguration>(configuration.GetSection("AuthConfiguration")); var authConfiguration = configuration.GetSection("AuthConfiguration"); builder.AddRedisDistributedCache("cache"); services.AddAuthentication(options => { options.DefaultScheme = CookieAuthenticationDefaults.AuthenticationScheme; options.DefaultChallengeScheme = OpenIdConnectDefaults.AuthenticationScheme; }) .AddCookie(options => { options.ExpireTimeSpan = TimeSpan.FromMinutes(60); options.Cookie.Name = "MvcPar"; options.EventsType = typeof(CookieEventHandler); }) .AddOpenIdConnect(OpenIdConnectDefaults.AuthenticationScheme, options => { options.Authority = authConfiguration["StsServerIdentityUrl"]; options.ClientSecret = authConfiguration["ClientSecret"]; options.ClientId = authConfiguration["Audience"]; options.ResponseType = OpenIdConnectResponseType.Code; options.Scope.Clear(); options.Scope.Add("openid"); options.Scope.Add("profile"); options.Scope.Add("email"); options.Scope.Add("offline_access"); options.ClaimActions.Remove("amr"); options.ClaimActions.MapJsonKey("website", "website"); options.GetClaimsFromUserInfoEndpoint = true; options.SaveTokens = true; options.PushedAuthorizationBehavior = PushedAuthorizationBehavior.Require; options.TokenValidationParameters = new TokenValidationParameters { NameClaimType = JwtClaimTypes.Name, RoleClaimType = JwtClaimTypes.Role, }; });

Note: The sample code in the repository was created using the IdentityServer4 Samples.

Redis cache

Redis Insight can be used to view the Redis cache data. Each time the application handles a new user and session logout event, it persists the event to the cache. If any further application instances are authenticated using this session and user, the application will sign-out as well on the next server event. The expiry time of the cache should be long enough so that an offline session cannot be opened after it expires.

Limitations

The back-channel logout only works on a per browser session because Keycloak creates new sessions for each browser. When the logout is received it is filtered and handled using the Keycloak session. If Keycloak can use a single session for all browsers of each user, then the logout can work for all active UI apps of the same user.

Links

https://www.keycloak.org/

https://openid.net/specs/openid-connect-backchannel-1_0.html

https://ldapwiki.com/wiki/OpenID%20Connect%20Back-Channel%20Logout

https://datatracker.ietf.org/meeting/97/materials/slides-97-secevent-oidc-logout-01

https://docs.microsoft.com/en-us/aspnet/core/fundamentals/app-state

https://docs.microsoft.com/en-us/azure/azure-cache-for-redis/cache-dotnet-core-quickstart

https://github.com/RedisInsight/RedisInsight

06:09 • 1 months ago

Sunday, 08. September 2024

Werdmüller on Medium

A conversation with myself about immigration

What is your ideal place? What are its values? Continue reading on Medium »

What is your ideal place? What are its values?

https://www.keycloak.org/

15:45 • 1 months ago

Friday, 06. September 2024

Michael Ruminer

Having AI Tools Generate Code For AI Tools

I recently posted on my experience with using a few Python examples of retrieval augmented generation (RAG) with an LLM (OpenAI API). They were underwhelming, though they did provide a lot of insight for me. This post is about my next experiment: let’s get AI tools to generate examples for me. The results were mixed but not for reasons you might think. Here’s what I found using ChatGPT, Microsoft

Note: you can find the generated code in my LLM_Generated_RAG Github repo. I will likely add to this repo as I try new prompts, LLMs, and approaches.

ChatGPT 4o

Let’s start with the most well-known, ChatGPT. I used a simple prompt to see what it would do. I didn’t provide prompts for different parts of the code. It should not be very extensive code so I didn’t feel the need to divide it into separate prompts. If you want to see my very elementary prompt pull up the ChatGPT related file in the repo.

The first thing I noticed is that it was using PyPDF2 which was deprecated in Decemeber of 2022. Not surprising as, almost certainly, a lot of what the model was trained on used PyPDF2. It actually created well-structured code with functions for the different discrete operations. What I realized later, due to the output of the Cursor tool, is that it created a more granular set of code than it had to. This is exactly what I was looking for but didn’t specify in the prompt. What do I mean? It did the following:

read the PDF and extracted the text split the document into chunks created embeddings for the chunks using the OpenAI text-embedding-ada-002 created an embedding for the query searched the chunks using faiss generated a response using the query and the chunk(s) as context

This is what I wanted. Turns out there was a very different way to interpret the prompt.

The downside. The code didn’t run out the box. Also, I believe it only used one chunk for the context to send along with the query. I have to investigate the code more closely to be sure on that single chunk thing. The embedding search chunk function I need to investigate more to understand it. I didn’t try to make the code run for now as that was not part of the experiment. I expect I will try to modify it soon to function. I’ll report back the results.

This granular set of actions was very different than what I got from Cursor.

Cursor

The Cursor prompt was not exactly the same as what I used for ChatGPT, which was bad of me to do. I got a little lazier, but the crux was the same. You can see that prompt here.

The results were very different. It did not provide the granular steps that ChatGPT did. It met the requirement more succinctly.

extract the text from the PDF, also using PyPDF2 pass the entire text to ChatGPT as context along with the prompt

When I first posted this I said it rain out of the gate, but that was wrong. It suffered from the same issue all three code generations did. It tried to use an old call of openai chat completion. With that fixed I believe it would, hypothetically, return the expected results. It would work similarly to how one does it in the ChatGPT interface. Not what I wanted, but I hadn’t specified to break it down so that I was creating embeddings etc. Fair game.

The downside, the context was too large (43,000+ tokens) for the tokens per minute limit of my OpenAPI account level and gpt-4o model (see attached note for an explanation). So I didn’t get to see the output, but have no doubt it would have produced similar results as if I had done it through the ChatGPT user interface.

Microsoft Copilot

What can I say? I don’t know what I would have gotten here because Copilot blocked the output. This is apparently because “GitHub Copilot checks code completion suggestions with their surrounding code of about 150 characters against public code on GitHub. If there is a match, or a near match, the suggestion is not shown to you.”

There is a flag you can set in your Github settings to turn this on or off. I checked and my flag and it is set to “Allowed” but you see the results.

I’ll continue to try to troubleshoot this issue and see if I can get something out of Copilot and update this post if I do.

In recap

In recap. One provided the detail I wanted even though I hadn’t specified it, a happy coincidence, but it did not run out of the gate. The second took a very different approach and would have provided the desired results in LLM response if my OpenAPI account level had supported a large enough tokens per minute for 43,000+ tokens in the context. But it wasn’t the code I was hoping for. Copliot just didn’t work for me for reasons I don’t yet understand. More experimentation to come.

20:07 • 2 months ago

Justin Richer

Making Bubbles: Re-connecting

If a set of accounts live in isolation forever, what happens to those accounts only matters within that isolated system. But when we make a bubble in our federated network, we aren’t just making a stand-alone system that can go live in a fully disconnected state. Over time, we expect things to re-connect, and when that happens, data needs to be harmonized across the boundaries. So many bubbles, so

So many bubbles, so many connections Data Synchronization

When multiple independent systems live together in the world and share data, inevitably that data is going to get out of sync. In a purely heirarchical system, we’re mostly concerned with building a consensus around the correct state of the shared data. We see this approach in distributed ledger systems, where nodes eventually come to a common view of what the shared state of the world ought to be.

But we don’t have that in a bubble-based architecture, because we don’t expect everyone to have the same view. Instead, we expect many different independent views to contribute to each other in a distributed fashion. This effectively means that for each bubble, it can send updates in and out to other systems. In most cases, there is a directionality to the data flow: one side is going to be treated as more authoritative than the other for a given context. What a bubble does when it’s on either end of that gap changes how we view the synchronization.

Account Updates From Above

When changes are pushed to us from an authoritative source, the simplest thing is to overwrite everything. After all, if it’s authoritative, why wouldn’t we just take that statement as truth? But the reality is that we’ve likely augmented our record of that user with additional details, overrides, and other localized changes that we don’t want to lose.

Local data (blue) shadows updates from the source (orange), in some cases

In these cases, we can shadow the data. In other words, we keep a copy of the source’s data separate from our own local view. When we get an update from that source, we can update our copy of the source’s data with wild abandon. We can then decide, by local policy, whether we want to adjust our overrides based on the update. Importantly, this decision is separate from accepting and processing the updated data from the source. This setup allows us to keep local information in the bubble at the same time that we sync from elsewhere.

Account Updates From Below

An authoritative bubble is sometimes going to want to pull updated information from the bubbles that live downstream. These are systems that we’ve sent accounts out to, and those systems might have something to tell us about our users. Maybe there’s an additional accreditation that’s been earned, or something to represent additional access details outside of our bubble, or even just an update to one of the core fields we sent down.

In any case, the downstream bubble is sending us additional data about a user, and we now have the chance to do something about it. If nothing else, we can store it and note it. If we want to, we can update the user’s record that we hold locally, and even go so far as to propagate that downward again to other bubbles.

Changing Trust

It’s not just user data that we can pass around, though that’s the most common piece we’d expect to see. The bubbles can also pass about other bubbles to each other, and incorporate what they learn into their own systems.

For many situations that fit the bubble architecture patterns, we expect to meet new peers in the field and make new decisions based on local context and requirements. These newly-discovered bubbles can then be propagated through the network, along with potentially interesting information about what the presenting bubble trusts it for.

Audit Logs

Now that we can identify users and the systems they come from, we can start to do one of the most interesting and perhaps complex jobs of a reconnected bubble: audit logs. When a bubble gets provisioned, that provisioning authority is likely to want to know what happens in that bubble during the disconnected spells. The bubble can package up the relevant audit log history and pass it along to the authorities that need it.

But auditing can go further than that: for any user that enters our bubble from somewhere else, we probably want to report back to the system that sent them to us. Since we know where we originally learned of them, and we know how to identify that system as a whole, we can filter and target the information we’re sending over. And we can do this while processing the updates they’re sending us about the account.

Conclusion

The bubble pattern sits between the always-connected and always-separate models, but the moment of reconnection gives us a profound opportunity to process user and trust data in ways that we may have never considered.

16:29 • 2 months ago

Thursday, 05. September 2024

The Pragmatic Engineer

The Pulse #105: More trad tech companies to move off the public cloud?

Also: CockroachDB joins the trend of going from open source to proprietary license, a software glitch nearly floods Amsterdam, and more.

The Pulse is a series covering insights, patterns, and trends within Big Tech and startups. Notice an interesting event or trend? Send me a message.

Today, we cover:

Industry pulse. Microsoft Recall to be unremovable, another “AI talent raid” by Amazon, Ticketmaster struggles to handle common load challenges – sometimes, rise and fall of OpenSea, and more.

More trad tech companies to move off public cloud? In a recent report, 83% of chief information officers say they want to move more workload to private clouds. Cloud computing has matured and is becoming more expensive, while private cloud technology is increasingly accessible.

CockroachDB: open source product goes proprietary. Another VC-funded open source product closes itself off, in hope of generating more revenue and profits. Many users need to open their wallets and buy annual licenses, or fork and maintain an older version of CockroachDB.

Software glitch almost floods Amsterdam. Dams near the Dutch capital that were supposed to be closed, mysteriously opened at night during a violent storm. Disaster was averted by a curious employee – and the workaround to avoid similar incidents includes 24-hour human monitoring of dams and water levels.

1. Industry pulse Microsoft Recall to be unremovable

Microsoft’s struggle with its controversial Recall feature continues. Recall is a “photographic memory” feature planned for Windows 11 which takes constant screenshots of a device, and makes it searchable. Its launch in June was a disaster, as Microsoft skipped basic security and privacy approaches – making these screenshots easily accessible for bad actors. A week after, Microsoft “recalled” Recall, confirming that it will not ship it with these glaring holes.

Now, the Redmond-headquartered tech giant is doubling down on Recall: it will be shipping on Windows 11, and the feature will have no option to be uninstalled. Users won’t be able to disable it.

The problem with making Recall core to Windows with no way to remove it or turn it off at the OS level, is that it could become a dealbreaker for enterprise clients which will not want an invasive, potentially insecure feature to run on machines by default. It’s also strange to see Microsoft so set on shipping this feature that is clearly designed to make some use of the hardware GPUs, even though the utility is still questionable. The company will surely keep getting bad press, and privacy-focused techies will have one more reason to stay clear of Windows.

Another “AI talent heist”

16:23 • 2 months ago

Jon Udell

Revisiting the Elm City project

“Communities that want to build comprehensive public calendars will be able to do so using a hybrid approach that blends existing iCalendar feeds with feeds synthesized from web calendars. It’s not a perfect solution, but with LLM assistance it’s a workable one. And who knows, maybe if people see what’s possible when information silos converge, … Continue reading Revisiting the Elm City project

Part of the LLM series at The New Stack.

15:08 • 2 months ago

Wednesday, 04. September 2024

Heres Tom with the Weather

At Bar del Pla in Barcelona

20:59 • 2 months ago

Tuesday, 03. September 2024

Michael Ruminer

My Failed (so far) AI RAG Code Experimentation

I have been wanting to experiment with creating some code performing retrieval augmented generation (RAG) for an LLM. So far I am 0 for 2. In both examples I tried, that were professed to work, it fell far short. I blame embedding and retrieval. I also blame myself… who else is there to blame. I know RAG works in some form because I can go onto ChatGPT upload a PDF and prompt about it with expecte

Experiment one was part of a Udemy course on RAG and mastering some better approaches than naive RAG. Specifically RAG with query expansion. It was a very short course but had a couple of cohesive Python examples that built out over the course material. It included the document to use for the augmentation. Additionally, unlike experiment two, it largely didn’t rely on a ton of abstraction of the underlying basic concepts. After reading and parsing the text from the PDF I used RecursiveCharacterTextSplitter and
SentenceTransformersTokenTextSplitter from Langchain to create my chunks. Added my chunks to Chroma db in memory using their default embedder. Took my query and performed the retrieval of 5 chunks from the Chroma db. No reranking performed. The returned chunks were at best partial results and at worst just not as good as expected. What I hadn’t noticed about the code from the instruction when I went through the class is that it never passed the chunks back as context to the LLM the second time along with the query to get an answer. Kind of an important part to be missing. I can tell from the returned chunks it would not have produced a satisfactory answer had the closing action before performed. I tried with differing chunk sizes and overlaps and never received better results. I tried with my own document and faired no better. I chalked it up to a disappointing and poor example. Perhaps this is why it didn’t go the final step to pass it all back to the LLM for a response.

I moved on to a second experiment that used a bit more abstraction by relying on Langchain significantly more. It was also doing naive RAG, not augmenting the prompt from the LLM initially in any way. This time it did have the pass to the LLM in the end to get the LLM response. Looking at the chunks it retrieved I could tell I wasn’t going to get a satisfactory response. I had asked it for a list of 10 principles that were specifically outlined in the document in a single paragraph. The best in all my attempts was that I got 2, maybe 3 of the list. Again I played around with chunk size and overlap and generally only got worse results. The results in the demonstration had not seemed much better when I evaluated it more closely.

All in all I need a third+ experiment. I am trying to get a response even remotely similar to what I get from the ChatGPT. Obviously, my examples are not high enough quality.

Do you have such an example?

Time to ask ChatGPT, Claude, Github Copliot, or Cursor — sounds like a job for Cursor — to create a python program for me and see how it functions.

19:17 • 2 months ago

The Pragmatic Engineer

Paying down tech debt

A guide for reducing tech debt effectively, and how to develop a mindset that welcomes the short-term benefits of eliminating it. A guest post by principal engineer Lou Franco

Q: “I’d like to make a better case for paying down tech debt on my team. What are some proven approaches for this?”

The tension in finding the right balance between shipping features and paying down accumulated tech debt is as old as software engineering. There’s no one answer on how best to reduce tech debt, and opinion is divided about whether zero tech debt is even a good thing to aim for. But approaches for doing it exist which work well for most teams.

To tackle this eternal topic, I turned to industry veteran Lou Franco, who’s been in the software business for over 30 years as an engineer, EM, and executive. He’s also worked at four startups and the companies that later acquired them; most recently Atlassian as a Principal Engineer on the Trello iOS app.

He’s currently an operating partner at private equity firm SilverTree Equity, and technical co-founder at a new startup. Lou says he isn’t delighted about the volume of tech debt accumulated during his career, but is satisfied with how much of it he’s managed to pay down.

In this guest post, Lou shares insights about how to approach tech debt. Later this year, he’s publishing a book on the subject. For updates on this upcoming release, subscribe here.

In this article, we cover:

Pay down tech debt to go faster, now. It’s common for less experienced engineering leaders to assume that focusing on features and ignoring tech debt is how to move faster. Lou used to agree, but not any more.

Use tech debt to boost productivity. Three examples of when tackling tech debt empowered engineering teams to move faster, right away.

Link tech debt and value delivery. When tech debt is tackled sensibly, it delivers business value. At Trello and Atalasoft, this was how Lou got engineers and management to appreciate the task.

Make tech debt’s effects visible. Dashboards are an effective way to visualize the impact of tech debt. A few examples.

Factor in time saved when thinking about productivity. Reducing tech debt typically improves coding, reviewing, and deployment for everyone.

Use tech debt payments to get into the flow. In a counter-intuitive observation: by making small, non-functional improvements, you gain more confidence in a new codebase, and can start to move faster.

Big rewrites need heavyweight support. Without the backing of management, a large-scale rewrite is likely to fail.

With that, it’s over to Lou:

1. Pay down tech debt to go faster immediately

What is tech debt?

I define tech debt as any problem in the codebase that affects programmers by making it harder to make necessary changes. As a programmer, I wanted to fix such issues because they slowed me down. But as a manager, I had to ensure the team delivered value to stakeholders. I’ve gone back and forth between these roles during my career, and made mistakes in both directions – but I also learned a lot about getting the balance right.

Reducing tech debt pays off immediately in faster builds

In 2010, I was head of development at Atalasoft, a company in the .NET developer tools space. I was obsessed with shipping, and spent all my time worrying about delivering the features in our roadmap. Over time, we improved at this, which showed up in our revenue growth and eventually led to an acquisition.

We were in a competitive market with more opportunities than we could handle. We had fewer than ten developers, but we were bootstrapped, so had to stay profitable and couldn’t just hire ahead of revenue.

The stakes got even higher after we were acquired. We had two years to deliver on an ambitious roadmap, for which there was an earnout bonus contingent upon delivery. If we didn’t deliver, we’d likely be classified as a failed acquisition. Our new owners had just had one such failure, which led to downsizing and an office closure.

My mindset was that any time spent on technical debt meant we’d fail to deliver on the roadmap. We couldn’t risk the deadline by wasting time cleaning up old messes, and had to choose between tech debt and roadmap delivery. In hindsight, I see this was wrong. I learned that the mindset of ignoring tech debt hurt my team.

Following an exit interview in which a departing engineer mentioned tech debt had contributed to their decision to leave, I started asking people during 1:1s how they felt about it. Their responses showed their frustration with me.

By then, I had been a developer for over fifteen years, and had worked in codebases with a lot of debt, so I knew what it was like. But by now, I was a manager who sometimes contributed code, but had forgotten what it was like to be thwarted by the codebase every day. To my team, I was part of the problem. They had been trying to tell me this, but I didn’t get it. Their departing colleague with nothing to lose in their exit interview finally got through to me and helped me understand the depth of the issue, and that it was slowing everyone down.

I learned an important lesson: the cost of tech debt is borne daily by your team, and you risk damaging motivation and raising attrition by ignoring it. Even if you have every reason to move forward without addressing tech debt, being an empathic manager requires you at least do something. Doing nothing – like I did – is not a good option.

So we started making changes. The biggest problems were with our build system and installer because they affected every developer and every product. It was a tangled bag of legacy code that needed constant maintenance, but it wasn’t very big, and I approved a plan to rewrite it with modern tools. It was a small experiment but paid off right away in quicker CI builds and an easier-to-modify codebase. Most importantly, I saw it didn’t derail our roadmap, so we took on other small initiatives.

This taught me another lesson about addressing technical debt. I had thought of it as something that might pay off in the long run. Might. This belief made it hard to justify doing it when I had to deliver on short-term goals. But instead, something else happened:

We paid off tech debt and increased productivity instantly! We had a build with faster feedback loops, less cognitive load, and which didn’t make developers frustrated when they had to add to it, which happened regularly. Updates were made with less code and without breaking things. It was an example of tech debt reduction paying off in increased developer productivity, right away.

Learning the cost of too much rewriting at Trello

I got my next lesson at Trello where I worked on the iOS app. The codebase was three years old when I joined in 2014. It had understandable tech debt because they needed to move fast, after going from 0 to 6 million sign ups. The devs working on it were founding engineers, working as described by The Pragmatic Engineer in Thriving as a Founding Engineer, and seeking product-market fit. Our biggest tech debt issue were some frameworks that made it fast to build a simple app, but held us back as the app got more complex.

Our own choices were influenced by the speed of Apple’s updates to iOS. The iOS 7 update completely changed the iOS design language and its network APIs. Later, iOS 8 introduced presentation controllers that gave developers much control over the animation when new views are shown. Unfortunately, the iOS 8 change broke our navigation code and caused crashes. These added up and started to make our code seem antiquated.

Left: how most iOS 6 apps looked like. Right: the iOS 7 version. iOS 7 introduced a ‘flatter’ look, and required lots of code changes to implement this new UI. Screenshot is from my app called Habits

Our code got even more complex when Apple decided to feature Trello on physical iPhones at Apple Stores. To be in stores, we needed a build that worked without an account or a network, so a mock backend was embedded in it for demo purposes. We didn’t want to maintain a separate codebase, so had a lot of random bits of demo-mode logic that stayed for years.

At Trello, I was coding every day and all this was in my face. Luckily, we were a small team of three developers, so my direct manager was also coding every day and was empathetic to the problems.

We did rewrites as we went, but sometimes went too far. To deal with the presentation controller problem of iOS 8, we developed a new paradigm for screen navigation inside the app, and rewrote all navigation to use it. This approach was the exact opposite of what I did at Atalasoft, where I’d ignored all tech debt.

Unfortunately, the approach of rewriting early turned out to be overkill. In hindsight, we could have just corrected the places that had crashed, and then lived with the code we had. Instead, we spent a few months designing and implementing a new, non-standard way of writing navigation code, but forgot a vital lesson that one of our founders, Joel Spolsky, identified in 2000 in Things You Should Never Do:

“We’re programmers. Programmers are, in their hearts, architects, and the first thing they want to do when they get to a site is to bulldoze the place flat and build something grand. We’re not excited by incremental renovation: tinkering, improving, planting flower beds.
There’s a subtle reason that programmers always want to throw away the code and start over. The reason is that they think the old code is a mess. And here is the interesting observation: they are probably wrong. The reason that they think the old code is a mess is because of a cardinal, fundamental law of programming:
It’s harder to read code than to write it.”

On the Trello engineering team, we were all very familiar with this article and quoted it to each other often, but it still sometimes bit us. The urge to rewrite a system instead of fixing it is strong, and we couldn’t resist! We should have addressed the few complex navigation cases that crashed our code without the full rewrite.

Size tech debt payment to be proportional to value. This is the biggest lesson I learned on this project.

I’ve seen both extremes of dealing with tech debt:

As a manager, I was overly resistant to devoting time to dealing with technical debt

As an engineer, I was exposed to its problems every day and didn’t resist the urge to pay it off enough

These two extremes form the fundamental tension of dealing with tech debt. As usual, there needs to be a balance, but finding it is not so easy.

The heuristic I use to pay tech debt these days is this: by reducing a specific tech debt, can I increase developer productivity and deliver business value right now?

If I can’t, then I don’t pay it down.

When the debt is so big that it couldn’t possibly deliver value now, or the value is invisible so nobody sees it, I do something else. Let me break down my heuristic…

2. Use tech debt to increase productivity

I try to pay down a little bit of tech debt regularly by making small cleanup commits as I go. I started doing this more intentionally after reading Kent Beck’s book, Extreme Programming Explained, in 1999, which introduced me to automated unit tests and continuous integration. Then, when I read Martin Fowler’s Refactoring, I started to see how to improve a codebase over time with very small, behavior-preserving changes checked by unit tests. In both books, and in others like Working Effectively with Legacy Code by Michael Feathers, and Kent Beck’s recent, Tidy First?, the authors stress that technical debt is inevitable, and that the main way to curtail it is to be constantly fixing it with small improvements enabled by unit tests and mechanical refactoring. I agree.

Unit tests, refactoring, and continuous integration are ubiquitous in the kinds of software I write, which are B2B SaaS productivity applications. Even making small improvements on an ongoing basis is common among my coworkers. It doesn’t take long, and there are usually quick wins to be had, like making the code more readable, or using a unit test to show how the code is supposed to work. Even in frontend code, Trello iOS adopted Model-View-ViewModel (MVVM) so we could test view-logic. We got the immediate productivity benefit of being able to run view code repeatedly without needing to manipulate a running app through several screens to check that our changes worked.

The issue is when the debt is large, which is where I struggled. My problem at Atalasoft was not with small improvements; it was with the bigger ones where I’d have to trade off current benefits like delivering features for the roadmap, for uncertain future benefits.

But I realized something.

You can get productivity benefits immediately, even with larger initiatives. If you do it right, you will deliver feature work faster and with higher quality. In fact, I view tech debt proposals that don’t deliver instant developer productivity gains as suspect.

Rewriting the build and installer at Atalasoft brought an immediate productivity boost. We had a backlog of problems and new additions, but the rewrite took one developer around a month, and when it was done many of the problems just went away because the new system was based on a framework wherein many problems could not occur, meaning we could close a bunch of reported bugs. The new system was unit testable, so we didn’t need to build and install the entire system during development to test our new changes while we were writing it. We also got more benefits later, but the instant benefits justified it.

At Trello, adding unit tests to a codebase helped me finish a project faster. When I joined in 2014, we were just about to start our internationalization (i18n) project, which I took on for the iOS app. One part was to write support for i18n-safe string interpolation (inserting variables or expressions into a string,) and pluralization (adjusting words to plural forms based on the number they refer to, to make the phrases grammatically correct) – which was only partially supported in iOS at the time. It’s standard string manipulation at its core, but in 2014 our iOS app didn’t have unit tests.

Without unit tests, if I had wanted to run the code, I’d need to run the app and then tap-tap-tap until I got to a specific string. I would have to do this for each kind of string I generated. But with unit-tests, I could just list all the examples with their expected results, and run tests in less than a second. So, I proposed to the team to add unit tests to our build and CI.

No one was against unit tests, but it hadn’t been a priority. Most of the code was UI or network code, for which unit tests are harder to write. But the code I was writing was highly testable, and in fact, it’s harder to write without tests. So, I added the unit test project to our workspace and wrote the string code. With the unit test project there, the other developers added tests to their work. I was there for six more years, and saw the benefits of the tests over time, especially in complex code like our sync engine. But that’s not why I did it: I added the unit tests to go faster immediately.

Also at Trello, creating an abstraction layer for the design system made us more productive. Eventually, we created a design system with a reduced set of fonts, colors, and other design attributes and specific rules for using them. Before, it was common to see hardcoded values in view controllers throughout the app, as each screen implemented the designer’s specification for that screen, which wasn’t always consistent. We could have just updated those lines to the new approved values, but it was the perfect time to make an abstraction for the design system itself. Doing this made it faster to write code that matched a design, and when a default in the design system changed, it would be reflected everywhere.

These three examples also adhere to another heuristic I use for finding the right balance with tech debt: coupling it with delivery of value.

3. Couple tech debt fixes with value delivery

18:56 • 2 months ago

Patrick Breyer

Chatkontrolle-Blitzentscheidung? Ungarn will beispiellose Pläne der EU zur Messenger-Massenüberwachung doch noch durchsetzen

Schon morgen früh sollen die EU-Regierungen den umstrittenen Gesetzentwurf zur Chatkontrolle, der im Juni nach massiven Protesten von der Tagesordnung genommen wurde, jetzt doch unterstützen. Erreichen will die neue ungarische Ratspräsidentschaft dies nach einer Meldung des Nachrichtendienstes Contexte mit einer von Politico geleakten Detailänderung, nämlich dem Verzicht auf die besonders fehleranfällige Suche nach unbekanntem Material im Zuge der verpflichtenden Chatkontrolle (freiwillig soll sie möglich bleiben). Geplant bleibt aber ansonsten die automatisierte Durchsuchung und gegebenenfalls Ausleitung privater Chats, darunter auch Ende-zu-Ende-verschlüsselte Chats, die zuvor klassifizierte verdächtige Fotos oder Videos enthalten sollen. Lehnt ein Nutzer diese „Upload-Moderation“ seiner Chats ab, könnte er keinerlei Bilder, Videos oder URLs mehr senden oder empfangen. Signal und Threema haben angekündigt, dass sie ihre Dienste in der EU einstellen würden, wenn sie gezwungen würden, die vorgeschlagene automatisierte Massenüberwachung (sogenanntes „Client-Side Scanning“) umzusetzen.

Der ehemalige Europaabgeordnete der Piratenpartei, Patrick Breyer, ruft die Bürger jetzt dazu auf sich an die EU-Regierungen zu wenden: „Im Juni gab es unter massivem öffentlichen Druck noch eine hauchdünne Sperrminorität zur Rettung des Digitalen Briefgeheimnisses, aber kaum bemerkte Blitzaktionen, zwischenzeitliche Regierungswechsel und Minimalzugeständnisse können das schnell ändern. Falls die Chatkontrolle kommt, werden wir gängige sichere Messenger ganz einfach nicht mehr nutzen können – das bedeutet wir verlieren den Kontakt zu unseren Freunden und Kollegen in der ganzen Welt“, warnt Breyer. „Wollen wir wirklich, dass Europa weltweit führend beim Abhören unserer Smartphones und der flächendeckenden Überwachung der Chats von Millionen gesetzestreuer Bürger wird? Das Europäische Parlament ist überzeugt, dass diese orwellsche Chatkontrolle das dringliche Anliegen eines besseren Kinder- und Opferschutzes verrät, weil sie unweigerlich vor Gericht scheitern wird. Es fordert deshalb einen wirklich wirksamen Kinderschutz durch sicherere Apps, proaktive Säuberung des Internets und eine Pflicht zur Löschung illegaler Inhalte – nichts davon ist in dem dem neuesten Orban-Vorstoß vorgesehen, zu dem sich die Regierungen morgen positionieren sollen. Jetzt liegt es an uns ihn zu stoppen!“

Die Chatkontrolle auf vermeintlich „bekannte“ illegale Inhalte beschränken zu wollen, bezeichnet Breyer als Augenwischerei: „Egal mit welchem Ziel – auch die Post darf nicht einfach jeden Brief verdachtslos öffnen und durchschnüffeln. Gerade die von US-Konzernen schon heute freiwillig praktizierte Durchleuchtung nach vermeintlich bekannten Inhalten führt zur Ausleitung tausender völlig legaler privater Chats, zur Überlastung der Strafverfolger und zur massenhaften Kriminalisierung Minderjähriger. Massenüberwachung ohne jeden Anlass zerstört das digitale Briefgeheimnis und sichere Verschlüsselung, auf die wir alle und auch unsere Kinder dringend angewiesen sind.“

Breyers Infoportal zur Chatkontrolle: chatkontrolle.de

Hilf jetzt mit die Chatkontrolle zu stoppen

12:55 • 2 months ago

Monday, 02. September 2024

Damien Bod

Implement ASP.NET Core OpenID Connect OAuth PAR client with Keycloak using .NET Aspire

This post shows how to implement an ASP.NET Core application which uses OpenID Connect and OAuth PAR for authentication. The client application uses Keycloak as the identity provider. The Keycloak application is hosted in a docker container. The applications are run locally using .NET Aspire. This makes it really easy to develop using containers. Code: […]

Code: https://github.com/damienbod/keycloak-backchannel

Setup

The standard Aspire Microsoft template was used to setup the .NET Aspire AppHost, ServiceDefaults projects. The Keycloak container service was added to the AppHost project using the Keycloak.AuthServices.Aspire.Hosting Nuget package. An ASP.NET Core Razor Page project was added as the UI client, but any project can be used like Blazor or an MVC application.

Keycloak Setup

The Keycloak Container is completely setup in the AppHost project. The Keycloak.AuthServices.Aspire.Hosting Nuget package is used to add the integration to .NET Aspire. For this to work, Docker Desktop needs to be installed in the development environment. I want to use the Keycloak preview features and initialized this using the WithArgs method. If using the Microsoft Keycloak package, the setup is almost identical.

var userName = builder.AddParameter("userName"); var password = builder.AddParameter("password", secret: true); var keycloak = builder.AddKeycloakContainer("keycloak", userName: userName, password: password, port: 8080) .WithArgs("--features=preview") .WithDataVolume() .RunWithHttpsDevCertificate(port: 8081);

I want to develop using HTTPS and so the Keycloak container needs to run in HTTPS as well. This was not so simple to setup, but Damien Edwards provided a solution which works great.

The RunWithHttpsDevCertificate extension method was added using his code and adapted so that the port is fixed for the HTTPS Keycloak server. This implementation requires the System.IO.Hashing Nuget package.

using System.Diagnostics; using System.IO.Hashing; using System.Text; namespace Aspire.Hosting; /// <summary> /// Original src code: /// https://github.com/dotnet/aspire-samples/blob/b741f5e78a86539bc9ab12cd7f4a5afea7aa54c4/samples/Keycloak/Keycloak.AppHost/HostingExtensions.cs /// </summary> public static class HostingExtensions { /// <summary> /// Injects the ASP.NET Core HTTPS developer certificate into the resource via the specified environment variables when /// <paramref name="builder"/>.<see cref="IResourceBuilder{T}.ApplicationBuilder">ApplicationBuilder</see>. /// <see cref="IDistributedApplicationBuilder.ExecutionContext">ExecutionContext</see>.<see cref="DistributedApplicationExecutionContext.IsRunMode">IsRunMode</see><c> == true</c>.<br/> /// If the resource is a <see cref="ContainerResource"/>, the certificate files will be bind mounted into the container. /// </summary> /// <remarks> /// This method <strong>does not</strong> configure an HTTPS endpoint on the resource. Use <see cref="ResourceBuilderExtensions.WithHttpsEndpoint{TResource}"/> to configure an HTTPS endpoint. /// </remarks> public static IResourceBuilder<TResource> RunWithHttpsDevCertificate<TResource>(this IResourceBuilder<TResource> builder, string certFileEnv, string certKeyFileEnv) where TResource : IResourceWithEnvironment { const string DEV_CERT_DIR = "/dev-certs"; if (builder.ApplicationBuilder.ExecutionContext.IsRunMode) { // Export the ASP.NET Core HTTPS development certificate & private key to PEM files, bind mount them into the container // and configure it to use them via the specified environment variables. var (certPath, _) = ExportDevCertificate(builder.ApplicationBuilder); var bindSource = Path.GetDirectoryName(certPath) ?? throw new UnreachableException(); if (builder.Resource is ContainerResource containerResource) { builder.ApplicationBuilder.CreateResourceBuilder(containerResource) .WithBindMount(bindSource, DEV_CERT_DIR, isReadOnly: true); } builder .WithEnvironment(certFileEnv, $"{DEV_CERT_DIR}/dev-cert.pem") .WithEnvironment(certKeyFileEnv, $"{DEV_CERT_DIR}/dev-cert.key"); } return builder; } /// <summary> /// Configures the Keycloak container to use the ASP.NET Core HTTPS development certificate created by <c>dotnet dev-certs</c> when /// <paramref name="builder"/><c>.ExecutionContext.IsRunMode == true</c>. /// </summary> /// <remarks> /// See <see href="https://learn.microsoft.com/dotnet/core/tools/dotnet-dev-certs">https://learn.microsoft.com/dotnet/core/tools/dotnet-dev-certs</see> /// for more information on the <c>dotnet dev-certs</c> tool.<br/> /// See <see href="https://learn.microsoft.com/aspnet/core/security/enforcing-ssl#trust-the-aspnet-core-https-development-certificate-on-windows-and-macos"> /// https://learn.microsoft.com/aspnet/core/security/enforcing-ssl</see> /// for more information on the ASP.NET Core HTTPS development certificate. /// </remarks> public static IResourceBuilder<KeycloakResource> RunWithHttpsDevCertificate(this IResourceBuilder<KeycloakResource> builder, int port = 8081, int targetPort = 8443) { if (builder.ApplicationBuilder.ExecutionContext.IsRunMode) { // Mount the ASP.NET Core HTTPS development certificate in the Keycloak container and configure Keycloak to it // via the KC_HTTPS_CERTIFICATE_FILE and KC_HTTPS_CERTIFICATE_KEY_FILE environment variables. builder .RunWithHttpsDevCertificate("KC_HTTPS_CERTIFICATE_FILE", "KC_HTTPS_CERTIFICATE_KEY_FILE") .WithHttpsEndpoint(port: port, targetPort: targetPort) .WithEnvironment("KC_HOSTNAME", "localhost") // Without disabling HTTP/2 you can hit HTTP 431 Header too large errors in Keycloak. // Related issues: // https://github.com/keycloak/keycloak/discussions/10236 // https://github.com/keycloak/keycloak/issues/13933 // https://github.com/quarkusio/quarkus/issues/33692 .WithEnvironment("QUARKUS_HTTP_HTTP2", "false"); } return builder; } private static (string, string) ExportDevCertificate(IDistributedApplicationBuilder builder) { // Exports the ASP.NET Core HTTPS development certificate & private key to PEM files using 'dotnet dev-certs https' to a temporary // directory and returns the path. // TODO: Check if we're running on a platform that already has the cert and key exported to a file (e.g. macOS) and just use those instead. var appNameHashBytes = XxHash64.Hash(Encoding.Unicode.GetBytes(builder.Environment.ApplicationName).AsSpan()); var appNameHash = BitConverter.ToString(appNameHashBytes).Replace("-", "").ToLowerInvariant(); var tempDir = Path.Combine(Path.GetTempPath(), $"aspire.{appNameHash}"); var certExportPath = Path.Combine(tempDir, "dev-cert.pem"); var certKeyExportPath = Path.Combine(tempDir, "dev-cert.key"); if (File.Exists(certExportPath) && File.Exists(certKeyExportPath)) { // Certificate already exported, return the path. return (certExportPath, certKeyExportPath); } else if (Directory.Exists(tempDir)) { Directory.Delete(tempDir, recursive: true); } Directory.CreateDirectory(tempDir); var exportProcess = Process.Start("dotnet", $"dev-certs https --export-path \"{certExportPath}\" --format Pem --no-password"); var exited = exportProcess.WaitForExit(TimeSpan.FromSeconds(5)); if (exited && File.Exists(certExportPath) && File.Exists(certKeyExportPath)) { return (certExportPath, certKeyExportPath); } else if (exportProcess.HasExited && exportProcess.ExitCode != 0) { throw new InvalidOperationException($"HTTPS dev certificate export failed with exit code {exportProcess.ExitCode}"); } else if (!exportProcess.HasExited) { exportProcess.Kill(true); throw new InvalidOperationException("HTTPS dev certificate export timed out"); } throw new InvalidOperationException("HTTPS dev certificate export failed for an unknown reason"); } }

Note: The AppHost project must reference all the services used in the solution.

Keycloak client configuration

See the razorpagepar.json file in the git repository. This is a Keycloak export of the whole client. This can be imported and updated.

The client is configured to use PAR.

ASP.NET Core OpenID Connect client using OAuth PAR

The client application uses the standard OpenID Connect client and requires OAuth PAR for authentication. This is a new feature in .NET 9. The repo has a Razor Page OpenID Connect example as well as an MVC client sample. This would be the same for a Blazor application.

services.AddAuthentication(options => { options.DefaultScheme = CookieAuthenticationDefaults.AuthenticationScheme; options.DefaultChallengeScheme = OpenIdConnectDefaults.AuthenticationScheme; }) .AddCookie() .AddOpenIdConnect(OpenIdConnectDefaults.AuthenticationScheme, options => { options.Authority = authConfiguration["StsServerIdentityUrl"]; options.ClientSecret = authConfiguration["ClientSecret"]; options.ClientId = authConfiguration["Audience"]; options.ResponseType = "code"; options.Scope.Clear(); options.Scope.Add("openid"); options.Scope.Add("profile"); options.Scope.Add("email"); options.Scope.Add("offline_access"); options.ClaimActions.Remove("amr"); options.ClaimActions.MapJsonKey("website", "website"); options.GetClaimsFromUserInfoEndpoint = true; options.SaveTokens = true; options.PushedAuthorizationBehavior = PushedAuthorizationBehavior.Require; options.TokenValidationParameters = new TokenValidationParameters { NameClaimType = JwtClaimTypes.Name, RoleClaimType = JwtClaimTypes.Role, }; });

Notes

.NET Aspire looks great and is easy to use in development. I am only learning this and must learn the details now. I have some issues using the containers and HTTPS and I don’t understand how the configuration works. I also don’t understand how this would work in production. Lots to learn.

Links

https://www.keycloak.org/server/features

https://github.com/NikiforovAll/keycloak-authorization-services-dotnet

https://openid.net/specs/openid-connect-backchannel-1_0.html

https://github.com/dotnet/aspire-samples/tree/main/samples

https://learn.microsoft.com/en-us/dotnet/aspire/get-started/aspire-overview

04:23 • 2 months ago

Sunday, 01. September 2024

Wrench in the Gears

Childhood Memories, An Unexpected Artifact, And Imaginal Openings

I write this post on the first anniversary of the death of my dear father Jerry Lee Hawver who grew up in an unstable, alcoholic household and through hard work and great people skills became a star athlete and executive with Proctor and Gamble. He and my mother raised my brother and me in the corporate suburbia of the New South of the 1970s and 80s, steadily climbing the ladder of the American dream. For my formative years that was my culture. The dynamic in our home was that I was the child who sought approval. My brother pushed the boundaries. Over time he and I have come together to meet in the center. We are both Hawver stock, and we “get ‘er done.”

In those early years of the feral-child 1970s, us kids had the run of the neighborhood yards and wood lots. I’ve spoken of hanging out by the creek lined with quartz-laden rip-rap looking for crawdads and swinging from giant vines and making “forts.” My best friend’s name was Andrea Hopkins. She and her brother Brandon lived in a house that was kitty-cross from our backyard. Her father had a British accent and they had a few antiques, which I thought made them rather a glamorous family. We would often spend the afternoon exploring, riding our Big Wheels around, playing freeze tag, and putting on plays in the garage. Sometimes Andi and I would get crafty and make little coil pots out of the red Kentucky clay and fill them with potions made of petals from her mom’s flower beds along her driveway. Of course, there were marigolds in the mix. In some ways my new “cottage” rancher is like a return to this liminal space of childhood imaginative possibility.

I realize now that early success is not necessarily a blessing. I did well in school after a disastrous second-grade year with a body-building teacher, Mrs. Weigand, in an open classroom school I had to take three buses to get to (desegregation). I have a strange memory of being shown a graphic movie of the horrors of smallpox in a tiny room with carpeted levels. That made no sense to me, but I later found out that the Flexners (of the Carnegie / Rockefeller Flexner Report) were from Louisville, so maybe we were part of some public health behavioral experiment.

I recovered the following year at the much more traditional Wilder Elementary School where we had a fabulous librarian who would take small groups of kids around the shelves and do short book pitches and place books into the hands of kids she knew would love them. It’s funny that there were quite a few paranormal kids’ books in the 70s – John Bellairs and Zilpha Keatley Snyder. The one book that really stayed with me was Margaret Anderson’s “To Nowhere And Back,” a time-travel adventure where a girl passes through an English hedge into another century to make friends with a poor servant girl on the other side.

I was never in the popular crowd, but I always had a circle of friends. Without realizing it, my identity became wrapped up in my idea of performance, of being a “good” person based on what society said good was. My life unfolded with relative ease, meeting my husband in college, raising a small family, doing my part to make my neighborhood and city a better place, so I thought. But things started to change when my child was in middle school as I began to realize society was being broken on purpose, and the social systems I’d built my identity around began to give way one by one. Eisenhower’s chosen head for ARPA was the President of Procter and Gamble Neil McElroy? Oh, and they ran contracts building nuclear missiles in the panhandle of Texas (Pantex)? You don’t say? Well that sure put Mr. Whipple and Mrs. Olson in a new light in a new light for me. If one’s identity is wrapped up in societal expectations and those start falling away, how does the center hold?

What does it mean to be a “good person” within the expectations of a profoundly flawed system? Here my parents had worked so hard to walk us up the mountain of success, who was I to pull off on a wayside and kick rocks? I was the good girl, wasn’t I? When my marriage fell apart and my child became estranged from me, the silver lining was the opportunity to begin to shed the “good girl” archetype. Not that I had become a “bad girl,” it’s just that I recognized such labels were insufficient and did not do justice to the richly evolving tapestry of my life. Our lives have dramatic arcs. I know it sounds trite, but if there are no bumps in the road, you can’t truly appreciate the parts when the ride is smooth and clear.

Last year we were trying to sort out the memorial service for my father. At the time I had been applying to all sorts of jobs with the idea that I would stay in Philadelphia and try to hold onto our family home that was almost paid off. In my perceived role of the “good mother,” I thought at some point my young adult child might return from the west coast and need a place to land. Life laughed at me and had the literal soles fall off my shoes on the walk to one interview at Wharton.

During the two-day drive from Philadelphia to North Carolina to be with my father in hospice I cried a lot. I thought about his strength, both physical and strength of character. He was a big man, and I was small, but we fit together, a chip off the old block. I was listening to a podcast this week on our relationship to death and ancestors and Perdita Finn said that sometimes when people go, it is because they can help the ones they love more from the other side. I truly believe my father has been with me every step of the way since that morning when I held his swollen big hands and I read to him from L’Engle’s “An Acceptable Time” and bathed his forehead in kisses, my cheeks wet, and he crossed over.

After a hurtful email exchange with my child and husband about the memorial service, it was clear that I was being removed from their lives with strong intention and there was nothing I could do about it. I am a planner. Hawvers like to be in control. We try to fix things. All of those things were off the table. It was up to me to build a new story outside of all the stories that had defined my life since I met my husband during our study abroad program in Venice in 1989. I’d never been alone, and now I was going to figure out how to be me, for me, not the good daughter, the model student, the diligent mother and wife. I told my husband I was going to leave Philadelphia and get a cozy cottage where I could garden and a kayak. I spoke it outloud, a reasonable wish, with conviction and with a few days to spare I got it all squared away.

It’s taken me a year to come around to this way of thinking, because there was a lot of pain to acknowledge and process. But now I realize I was being given space to unfold in a new way. I had a choice. I could hold onto the baggage I was carrying and imagine myself as a failure across many categories. The thing is, I actually like myself and I don’t feel shameful about the course I’ve taken. Regrets? Yes, of course I have some. Have I made missteps? Certainly. But not out of malice, out of lack of information, insight, and wisdom. I deeply miss the people I have lost, but I can learn to be, to become, outside of them, outside their stories and expectations, and that’s ok, too. Perhaps some day they will have grown in their own stories enough and there will be enough space to hold all of who we have become in some new kind of family. Or maybe that won’t happen in this version of the story. I have to learn to accept what comes with an open heart.

Before he left this Earth, my father navigated Alzheimer’s for about six years with numerous falls, escapades, ditched cars, eating light bulbs, jumping back yard fences, and taking apart cable boxes at his nursing home. As I wade into a broader understanding of consciousness, I open up space in my heart for what was really happening with my dad and his glitching interface with what we understand as reality.

Over the course of my research this past year I found a very interesting memoir written by Elmer Green of the former Menninger Clinic in Topeka, KS where my dad grew up. It’s called “The Ozawkie Book of the Dead: Alzhiemers Isn’t What We Think It Is” (PDF) and it is the story of how he cared for his wife Alyce, his research partner in biofeedback, during her journey through that illness. It involves the nature of the soul and interpenetrating realms and the process of learning and unlearning how to be in this space.

I think I’m going to read it again, because as terribly traumatic as it was to see my strong, capable father engage with the world in ways that were so extreme and problematic and inscrutable, having a year’s perspective and new awareness of proto-consciousness and the unified field and emergent complexity, I can sit with the hard parts of the process and realize that maybe what my father’s struggles were doing was giving me a window to look through for the answers I seek.

I was under the weather for about a week. It started with a vicious migraine and then some shooting nerve pain and gradually resolved into a typical cold. Let me tell you, it’s not fun to be feverish in Arkansas in August. Most of the month the daytime highs have lingered in the upper 90s. By the end of the week I was regrouping, less foggy headed, and I decided to take a hike to Balanced Rock and make a heart to honor Jerry Hawver’s life. The trail is on the other side of town. It’s only a three-mile walk, and the idea of balance seemed appropriate. A friend had sent a delightful care package with a hummingbird card and dried herbs from her garden in Arizona. I clipped the hummingbird and took out a selection of peppers to place in my wicker basket along with a heart trimmed from the collar of one of his shirts. It still smells of his aftershave.

I drove past the bath houses downtown then parked my Subaru on the gravel pull out along Cedar Glades Road, quickly crossed the blind curve, and started the climb. The online reviews said that the first half mile took some effort. It was steep, sure, but the challenge was walking on loose scree. For years I never gave much thought to being particularly careful while I was out and about, because there were always at least two of us. Now, it’s just me. So I have to be watchful about taking a misstep and turning an ankle. I took things slowly and on my way up I picked up a chunk of Novaculite that seemed to have been worked into a hand sized cutting tool or scraper. Idly I ran my fingers over it as I pressed on through the dry, dusty woods.

Where it leveled out, I paused for a moment and a wild red and black bug caught my eye. When I looked it up later, it turned out to be a velvet ant, a parasitic wasp species where the females are wingless. A minute later, I saw her partner flying down the trail. Fortunately I had the good sense to keep my distance, as I later found out their sting is many times worse than a hornet’s and they are commonly called “cow killers.”

An interesting synchronicity is that in one of my early blog posts I warned about hybrid “personalized” learning being like a cicada killer. A cicada killer is a large wasp that frequented the garden where I worked. Its reproductive cycle involved capturing their chosen prey on the wing, paralyzing it, dragging it into an underground burrow, and then laying eggs on the body where the larvae would eat their way out. It was rather grotesque imagery, but it served to make a point. Well, turnabout is fair play, and evidently these velvet ants parasitize the pupae of cicada killers. Hmm.

Despite the dryness you could still find a few ferns, and where the side trail to Balanced Rock branched off there were some wild plums dropping onto the path. I gathered a few things en route to supplement what I’d been gifted and brought from home. Not long after I arrived at the outcropping – more loose scree, more careful steps. I managed to get far enough down the slope to fully appreciate the balance embodied by these two impressive boulders.

Of course right as I got there, a couple arrived panting, and so we gave each other space. I pulled over to a little overlook in the shade and pulled out my basket of offerings. The arrangement of items in this particular heart is both loose and disordered while being vibrant and tactile. There are cracked nuts full of labyrinths, fractal turkey tail fungus, the first fall leaf offerings, and peppery spice.

I think that would suit my dad just fine. By the end everything was coming apart, which is a difficult thing for Hawvers. Have I said before that Hawvers like to be in control and have a plan? Well, we do. And the last years of Jerry’s life were anything but that. At the time there was a heaviness about it, this uncontrollable fraying and sense of loss that would not be diverted, but a year later, perhaps I can reimagine it as a cocoon tearing apart the restricting expectations my dad must have felt to be a provider and a fixer and the rock for all of those years. To transition to a new form meant things were going to fall apart. How else is renewal possible? In the looseness of MIA control systems there is the unrealized potential of a new blueprint and within the chaos, a kernel of what’s next.

I sat on that bit of bluff, and thanked my dad for all the things he provided for us. I can feel him giving me a big bear hug and me offering in return a big back scratch. I told him that I missed him deeply and thought of him every day. I also said that I knew he had been walking beside me, but that I was going to be ok, that if he felt held back in any way by me, he could go. I can just picture him high-fiving people all over heaven. I prayed that the creator would help me have the clarity to see the signs and find my purpose as I unfold into this new life. Then I packed up and headed back down the trail.

The walk back was uneventful with one exception. Near the far end I paused before an unusual artifact, a rusted out wash basin hanging from a knot in a tree at eye level, almost where a shaving mirror would be. It had been galvanized, but the rust had gotten to it and the entire bottom was eroded away. I lifted it off the knot to examine it more closely. It wasn’t large, a bit wider than a typical dinner plate and about four inches deep with a narrow rim punctured in one spot where you could hang it from a nail.

Now it wasn’t a totally bizarre addition to the landscape. You might find something like that tossed in a ravine out behind an old farmhouse. But there was no other debris around, no trash. I found it right along a trail on land that had been a national park for almost a century. It wasn’t tucked away in the brush; it was right in front of my face. It wasn’t a plastic bottle or even a face mask carelessly tossed trail side. No, this was a message for me, this wash basin, that if it weren’t for the rusted out bottom, would be right at home as part of a Cracker Barrel wall display.

Ok, so I had asked for the wisdom to notice the signs, but then what was I to make of this out-of-place, out-of-time basin? One story you might tell is one of obsolescence. When my life first fell apart during the lockdowns, that was the story I initially jumped into.

Who am I now that I am no longer my child’s mother, my husband’s wife, my mother’s accomplished daughter? I was a basin whose design was to hold things for others. Basins don’t function alone, they contain things. For decades my identity had contained many things, big and small. From carpool schedules and family vacation plans, to dinner menus and school registration procedures, civic campaigns, etc. etc. My existence was to be a helpful presence. Then somehow I picked up some rust. Perhaps at that school report card meeting in the summer of 2013. Rust wore away at the old functionality of the basin. Interestingly rust interferes with magnetism in metal. So as the basin’s rust surface area increased, it became less polarized. I imagine that might represent a shift towards non-dualism.

Was I meant to believe I was a failed basin? No, of course not. Because I believe in a benevolent creator that is loving and wants what is best for us. So the story I am choosing to tell myself about the basin is one that Alison Hawver might have imagined in third grade in Mrs. Nevius’s cardboard geodesic dome reading nook sitting on a carpet square. I am going to imagine that like the ugly duckling becoming a graceful swan, I have been a capable and trustworthy basin for many years, but slowly grew into a magical portal of imaginative potential, which given the state of the world right now, is a pretty awesome skill to have on offer. The twenty-first century needs some wiggle room for new possibilities to emerge, some fluidity to flex the boundaries of spacetime.

I’ve traded in the closed, hygienic system of wash basin culture for the jagged, unexpected potential of a back-to-nature, quantum vacuum, anti-mirror portal into proto-consciousness where we can dip in for messages from worlds with a better view than ours. A portal like the cottage hedge in the book “To Nowhere and Back” or the star watching rock from L’Engle’s Time Quintet that I read from at my father’s death bed.

Well, I’ve written all day, and the heat has finally broken with a gentle rain and a cool breeze here on the crystalline flank of Spa City. I hope you haven’t found this too self-indulgent or maudlin. There are lots of big feelings to navigate these days for all of us, and sometimes it’s useful to remember back to the times when all we had to do was make petal potions with our backyard buddies. If you quantum foam some interesting artifacts into your life, I’d love to hear about them and the story goes with that. Thanks for hanging out with me on this Labor Day weekend 2024. A toast to Jerry Hawver. I wouldn’t be who I am without him.

22:57 • 2 months ago

A Soft Landing on Marigold Circle

Dear readers, For those who are still following along, you’ve probably noticed it’s been quiet on the blog this summer. I did a short update on my channel a few weeks ago letting folks know I found a cozy home, a 1970s two-bedroom brick rancher on about a quarter acre on the outskirts of Hot [...]

Dear readers,

For those who are still following along, you’ve probably noticed it’s been quiet on the blog this summer. I did a short update on my channel a few weeks ago letting folks know I found a cozy home, a 1970s two-bedroom brick rancher on about a quarter acre on the outskirts of Hot Springs, AR. It is a sturdy little house with a sunny yard and quiet neighbors, mostly retirees who are kind about looking out for you without intruding and generously extend weekly invitations (in season) to share enormous slabs of delicious ice cold water melon. It’s nothing fancy, but it after a month I can say it suits me perfectly and I am grateful for the blessings of having a safe place to embark on this next phase of life.

My snail mail address is 123 Marigold Circle Hot Springs, AR 71913. I’ve already received a few letters and care packages, which are much appreciated. I’m loving the slower pace of things in my chosen exile. Maybe I can have a go setting up some old school pen pal correspondences if you care to drop me a line.

Marigold Circle is a loop at the end of Hobson Road that is tucked into the base of one of the ridges underlaid by that famous quartz crystal that comprise Hot Springs National Park, the oldest federally protected land in the country (1830s compliments of Andrew Jackson). This development of about thirty homes, locally known as “Little Chicago” is situated where West Mountain begins to transition into Music Mountain. For years Hot Springs was known as a mobsters’ hang out, some with Chicago ties. Oh, and the circle is a square (not a cul-de-sac), and I’m in the middle of the doughnut (aka the quantum vacuum, wink, wink).

I’m loving the breathing room provided by the small open lawn around my house that gives me space to take in the blue Arkansas skies, often punctuated by dramatic towering clouds. There is considerably less geoengineering here than in Philadelphia. It does happen, but much less. The forest that embraces our little development is a mix of pine and hardwoods, so I should have nice vistas at all times of the year. It’s not an off-the-grid, bug-out place; but it is quiet, there are good people here, and lots of places to hike and get out on the water. I bought an inflatable stand up paddle board that can convert into a basic kayak, which will be easier for me to transport. Now that I’ve got my life vest, I’m going to head out AFTER the Labor Day boating hoopla and soak up a bit of the calm of the shoreline of Lake Ouachita and the 1.8 million acres of national forest surrounding it. Water is the theme of the year.

My yard doesn’t have trees, but my neighbors do – a sweetgum and sycamore on the south, a maple to the north. It’s been a very hot month with daytime temps rarely getting out of the upper nineties and almost no rain. The poor gum tree, in particular, seems quite stressed. Having lived in a row house for the past thirty years, I didn’t own a rake, just a broom for the front walk, so I grabbed a solid wooden-handled Craftsman at Lowe’s this week. Even though I got a blower with my lawn mower purchase, I do hate the sound of them. For me Hot Springs is a great chance to slow down and get back to the basics. I slowly filled half a contractor bag with leaves yesterday and look forward to getting a better composting situation set up soon.

There’s a towering “axis mundi” oak across the street. It has been trimmed back by the power company, so it appears almost split in half, its impressive fractal branching pattern is on full display, which is right in line with my recent musings on complexity and fractals. This particular specimen is over sixty-feet high. Given the severe weather you can get here, I recently took out a measuring tape to see how much distance there was between its base and my eaves. Hopefully, if it ever falls, only the top ten feet will hit my house, enough to mess things up, but not totally destroy the house.

The one poor choice I’ve made so far (that I know of) is that I initially bought a terribly cheap, uncomfortable twin mattress. I don’t have sleep issues or chronic pain and don’t consider myself a mattress snob. Our old queen mattress that we had for over twenty years held up like a champ. It was just a basic Serta or something, but of course they don’t make them like they used to. I ended up buying a replacement with natural materials that should be arriving next week. I’ll put the old one in a mattress bag in the basement for tornado season. It’s the perfect use for a sad excuse of a sleeping surface, perhaps one day padded protection from falling timbers. If that badly-pruned oak ends up taking a nosedive through my living room, it could end up being the hero of my story.

My house is on a slightly fall-away lot, which is why I have a basement. Happily, it is still level enough to garden once I get a fence and raised beds set up to deter the many deer and bunnies. All the houses are modest one-stories, and so my little back sun porch has the feeling of a tree house. The neighbors feed the birds (and the deer). Most days I feel I’m simply a visitor in their avian kingdom. Marigold Circle is an ideal woodland edge habitat. There is a raucous group of jays, daredevil mocking birds and thrashers, lots of flickers, turtle doves, cardinals, house finches, and Carolina wrens. Some evenings or early mornings I get lucky to catch a glimpse of a hummingbird probing the magenta trumpets of the Four O’Clocks in my front dooryard. There are lots of dragonflies and butterflies and wasps, too. The wasps pollinate the big turkey fig out back that has been offering me a handful of sweet fruits every morning since I moved in.

I mentioned in my video that I’m weighing the extent to which it makes sense for me to continue to share my research moving forward. I’ve pretty much unplugged. I’m not on social media outside of the interactions I have on my channel and blog. I don’t have a television or radio. I have a portable “suitcase” record player and am gradually collecting bluegrass and folk on vinyl. I don’t check the news online. What information filters in comes to me through my small circle of friends. I’m less interested in blogging as a contact / “gotcha” sport, as tempting as that is with all the things we’ve been talking about now rolling out including Crypto Bobby, Trump’s new sidekick.

You might think that is irresponsible of me to check out, but I have come to the realization that I am not going to change the world through social activism or civic engagement. Trying to convince people to pay attention to the story I have attempted to surface doesn’t work. The learning curve is too steep and the feed does not operate in my favor. That said, maybe there is another way to tweak the seemingly chaotic but perhaps not entirely complex labyrinth of what we understand to be reality. Maybe it is my task for now to try and put my intuition and intellect into puzzling out some cheat codes and rather than throwing a wrench in the gears to stop the progress of so many problematic programs, learn to play living systems instrumentally with heart by adjusting initial conditions and tuning constraints. It may be a fool’s errand, but for now I turn inward – not giving up, but handing over control to the field of imaginative possibility as I try and train my eyes to see the clues and piece them together. Being quiet and reading and sewing for the time being is enough. I have put in my time trying to be loud. In the end it was unsatisfying, and even destructive to my wellbeing.

The idea of responding to breaking developments isn’t satisfying. Neither is critiquing personalities or asserting the superiority of my lens. Mostly, I want to continue to follow the intellectual meanderings that feed my soul. Neal Stephenson’s novel, “Anathem” revolves around a society of monastic thinkers who remove themselves from the world, so they can focus on weightier contemplations. I can see the logic in that. Stepping outside the noise has brought me a measure of peace and calm. Walking around barefoot, something I couldn’t do in Philadelphia, has felt great and is very grounding, too!

The physics of digital information is different. I think it comes so fast and furious and disconnected from our actual lived experience that it overwhelms our sense-making instincts both individually and collectively and ends up leaving us anxious and adrift. I feel like I know enough for the moment about the basics of web3, the mechanics of social impact policy and digital governance, and the Internet of Bio-Nanothings. Now I want to better understand the realm that we are in. Not in an esoteric, occult way that appeals to many; more like a Claude Shannon, information theory way. I think energy flow and information is the key to traversing the choice architecture of inner space. The more I discover, the less certain I am about anything. I do think there is merit in putting my thoughts down in writing with a date and time attached, even though I’m less and less invested in the concept of linear time. That said, the postings here will probably continue to be sparse.

The topics I currently find to be most compelling are quantum biology, proto-consciousness, complex non-linear dynamical systems, morphic fields, parallel processing, distributed collective intelligence, the observer effect, narrative structures, spatial computing, responsive environments, digital artifacts and deep learning. I am dipping into a lot of information flows along with friends and I’m not exactly sure what to say about my thought process yet. In addition to Karl Friston’s Free Energy Principle and Chris Fields’ Active Inference work, I am also very interested in the ideas of the late Jungian psychiatrist, dream analyst, and quantum consciousness theorist Gordon Globus. I am wrapping up his 2009 work “The Postmodern Mind” now.

I have a lot of ideas about non-traditional “computing” interfaces and humans-in-the-loop that have less to do with nanotechnology than with Qi, psi and plasma and etheric realms. As the head of standards for IEEE once said – it’s really all metaphysics!

Minimizing surprise and making solid predictions is an evolutionary imperative, according to Karl Friston’s free energy principle. We hold a model of how the world works in our heads, so that we can use it to make good choices navigating uncertainty. Many of those choices are fluid and socially normed. Some people are more comfortable changing their model than others. Some people deal with surprise by looking away. Then there are the odd balls like me who keep hoovering up information and trying to find patterns in it so that we can conjecture about stories that could make the whole messy pile hold together for a time at least, because there are always more surprises awaiting us.

It seems in my case, I became too much of a surprise for my beloved family, and the perceived cost of changing their models to accommodate my admittedly fringe world view was too high. I can’t say they were wrong. I will simply say I didn’t realize with each small step, new nugget of information, and adjustment that I was being removed from their world model. I simply did not appreciate how profound the cost would ultimately be.

I am still working to make sense of what happened to me in a way that is non-judgemental and centers compassion for all involved. I do believe understanding the power of the heart field is vital to navigating the turbulence of the world – not just in a sweet, saccharine sort of way but in a quantum processing sort of way that connects us straight to the divine nature of the cosmos.

I have another blog post I plan to work on tomorrow in honor of the year anniversary of my father’s passing. I’ve been under the weather for about a week, but it seemed to be resolving so a few days ago I took a short out and back hike to Balanced Rock on the other side of downtown. On the return leg I encountered an unusual artifact, which felt like it had some symbolic significance. My next writing will be a reflection on that.

I’ll close with an entertaining and informative talk my friend Sean posted this week about artifacts inspired by a discussion I’d posted with Lynn on Open Education Resources. Sean’s talk features a read aloud from a Mrs. Frizzle and the Magic School Bus book sponsored by the Department of Energy. It’s a good format, be sure to check it out.

All the best,

Alison McDowell

123 Marigold Circle

Hot Springs, AR 71913

PS: I just learned that marigolds go by the Latin name Tagetes erecta, which comes from Tages the Etruscan deity of divination. West is the direction of the underworld in Greek mythology, though I continue to imagine these other realms as being more about signals and information, which means it’s interesting that the ridge above me is West Mountain and Music Mountain.

03:14 • 2 months ago

Aaron Parecki

Thoughts on the Resiliency of Web Projects

I just did a massive spring cleaning of one of my servers, trying to clean up what has become quite the mess of clutter. For every website on the server, I either:

Documented what it is, who is using it, and what version of language and framework it uses Archived it as static HTML flat files Moved the source code from GitHub to a private git server Deleted the files

It feels good to get rid of old code, and to turn previously dynamic sites (with all of the risk they come with) into plain HTML.

This is also making me seriously reconsider the value of spinning up any new projects. Several of these are now 10 years old, still churning along fine, but difficult to do any maintenance on because of versions and dependencies. For example:

indieauth.com - this has been on the chopping block for years, but I haven't managed to build a replacement yet, and is still used by a lot of people webmention.io - this is a pretty popular service, and I don't want to shut it down, but there's a lot of problems with how it's currently built and no easy way to make changes switchboard.p3k.io - this is a public WebSub (PubSubHubbub) hub, like Superfeedr, and has weirdly gained a lot of popularity in the podcast feed space in the last few years

One that I'm particularly happy with, despite it being an ugly pile of PHP, is oauth.net. I inherited this site in 2012, and it hasn't needed any framework upgrades since it's just using PHP templates. My ham radio website w7apk.com is similarly a small amount of templated PHP, and it is low stress to maintain, and actually fun to quickly jot some notes down when I want. I like not having to go through the whole ceremony of setting up a dev environment, installing dependencies, upgrading things to the latest version, checking for backwards incompatible changes, git commit, deploy, etc. I can just sftp some changes up to the server and they're live.

Some questions for myself for the future, before starting a new project:

Could this actually just be a tag page on my website, like #100DaysOfMusic or #BikeTheEclipse? If it really needs to be a new project, then: Can I create it in PHP without using any frameworks or libraries? Plain PHP ages far better than pulling in any dependencies which inevitably stop working with a version 2-3 EOL cycles back, so every library brought in means signing up for annual maintenance of the whole project. Frameworks can save time in the short term, but have a huge cost in the long term. Is it possible to avoid using a database? Databases aren't inherently bad, but using one does make the project slightly more fragile, since it requires plans for migrations and backups, and If a database is required, is it possible to create it in a way that does not result in ever-growing storage needs? Is this going to store data or be a service that other people are going to use? If so, plan on a registration form so that I have a way to contact people eventually when I need to change it or shut it down. If I've got this far with the questions, am I really ready to commit to supporting this code base for the next 10 years?

One project I've been committed to maintaining and doing regular (ok fine, "semi-regular") updates for is Meetable, the open source events website that I run on a few domains:

events.indieweb.org events.oauth.net events.scim.cloud

I started this project in October 2019, excited for all the IndieWebCamps we were going to run in 2020. Somehow that is already 5 years ago now. Well that didn't exactly pan out, but I did quickly pivot it to add a bunch of features that are helpful for virtual events, so it worked out ok in the end. We've continued to use it for posting IndieWeb events, and I also run an instance for two IETF working groups. I'd love to see more instances pop up, I've only encountered one or two other ones in the wild. I even spent a significant amount of time on the onboarding flow so that it's relatively easy to install and configure. I even added passkeys for the admin login so you don't need any external dependencies on auth providers. It's a cool project if I may say so myself.

Anyway, this is not a particularly well thought out blog post, I just wanted to get my thoughts down after spending all day combing through the filesystem of my web server and uncovering a lot of ancient history.

01:36 • 2 months ago

Thursday, 29. August 2024

Aaron Parecki

OAuth Oh Yeah!

The first law of OAuth states that

The first law of OAuth states that the total number of authorized access tokens in an isolated system must remain constant over time. Over time.
In the world of OAuth, where the sun always shines, Tokens like treasures, in digital lines. Security's a breeze, with every law so fine, OAuth, oh yeah, tonight we dance online!
The second law of OAuth states that the overall security of the system must always remain constant over time. Over time. Over time. Over time.
In the world of OAuth, where the sun always shines, Tokens like treasures, in digital lines. Security's a breeze, with every law so fine, OAuth, oh yeah, tonight we dance online!
The third law of OAuth states that as the security of the system approaches absolute, the ability to grant authorized access approaches zero. Zero!
In the world of OAuth, where the sun always shines, Tokens like treasures, in digital lines. Security's a breeze, with every law so fine, OAuth, oh yeah, tonight we dance online!
Tonight we dance online! OAuth, oh yeah! Lyrics and music by AI, prompted and edited by Aaron Parecki

19:59 • 2 months ago

The Pragmatic Engineer

Leading Effective Engineering Teams: a Deepdive

What makes software teams effective, and how do the TL, EM and TLM roles differ? An excerpt from Addy Osmani’s new book: Leading Effective Engineering Teams

Programming note: instead of our usual The Pulse on Thursday, today we peek inside a book that I wrote the foreword for. Our regular schedule — and The Pulse — returns next week.

Addy Osmani is a software engineer and engineering leader. He’s worked at Google for 12 years, and is currently the Head of Chrome Developer Experience. Addy regularly shares insights on software engineering and engineering leadership, and is the author of several software engineering books. He also writes the newsletter .

When I was visiting the Bay Area, we met up with Addy, who gave a tour of the Chrome offices:

With Addy, in the Google Chrome team’s lobby, in Mountain View

As we talked, he shared how he’s been working on a book about engineering leadership – collecting a decade of lessons learned in motivating and leading engineering teams. The lessons span Addy’s earlier days – when he was a software engineer, and was mentoring fellow devs – through working as a tech lead manager (a specialist role present in larger tech companies like Google) – all the way to leading larger engineering teams. I was intrigued, and asked for an early look. In the end, not only did I read the book ahead of release, but I found it such a neat mix of “theory” and “practice” that I volunteered to write the foreword.

The book is now out, and I asked Addy if he’d be open to sharing two relevant parts with all newsletter readers. Addy said yes, and so in this issue we take a look inside the book, covering:

Context on the book. Why write this book, how long it took, and Addy’s favorite part of it.

What Makes a Software Engineering Team Effective? No company invested more research in this area than Google. A brief summary of Project Aristotle and Project Oxygen, and a collection of other useful team dynamics research.

Leadership roles: tech lead, engineering manager, and tech lead manager. A look at how these three roles overlap, and also differ. The tech lead manager (TLM) is unique to Google, and a few similar companies, and is the most unusual of the three.

Get the full book

My usual disclaimer: as with all my recommendations, I was not paid to recommend this book, and none of the links are affiliate ones. See my ethics statement for more details.

1. Context on the book

How did the idea for writing this book come around? This is what Addy told me:

“The idea for the book started percolating a few years back. I'd been in the trenches of engineering leadership at Google, and I was seeing patterns – ICs, leaders and managers all cared about being effective, but there wasn't a well defined resource on this topic. I would email folks guidance whenever they would ask, but there was a real disconnect between the academic world of management and the gritty reality of leading high-performing engineering teams.
People were hungry for practical guidance, not just theoretical frameworks. That's when I realized there was a need for a book that could bridge that gap.
So I started working on my notes that would eventually turn into this book.”

Writing a book takes a long time, and I was curious how much effort this one took. It was 1.5 years to write – plus the many years of experience to have something worthwhile to pen down. From Addy:

“Writing the book was a longer haul than I expected. Writing a book is a bit like building a complex software system - it requires careful planning, execution, and constant iteration.
While the actual writing took about a year and a half, the foundation was years in the making. It was essential to blend my personal experiences with rigorous research. I wanted to ensure the book wasn't just a collection of anecdotes but a practical guide backed by data and insights. I think as a team (O'Reilly, our editors and tech reviewers as well) we were able to land on something in the end that we all felt proud of.”

The book has seven chapters, and I was curious as to what Addy’s favorite part is, if there’s any. Addy said:

“The 3 E's framework (enable, empower, expand) is undoubtedly the cornerstone of the book. It's something I've been refining over years of leading teams. I'm particularly proud of how it provides a practical approach to building high-performing engineering organizations.
What excites me most is how the model can be adapted to different team sizes and company cultures. It's not a one-size-fits-all solution, but a flexible framework that can guide leaders at various stages of their career. I'm eager to see how others apply it and share their experiences.”

With this added context, let’s dive into two chapters of the book.

2. What Makes a Software Engineering Team Effective?

The below is from the beginning of Chapter 1 in the book.

Some teams seem to operate like well-oiled machines, churning out successes. Communication flows seamlessly, they meet deadlines with a smile, and they tackle challenges head-on. Conversely, other teams struggle to reach every milestone. Communication is chaotic, and meeting deadlines is a challenge. What makes the successful teams effective? It’s usually a mix of things: clear plans, honest talk, a healthy dose of trust, and a shared belief in what they’re doing. Some teams already have the rhythm and the steps down pat, while others are still figuring things out. But the good news is that everyone can learn the steps. Even the most stumbling crew can find its rhythm with a little practice.

This rhythm manifests itself in software engineering teams as their ability to produce useful products or product features by writing code, testing it, and releasing it to the world. Teams that do this regularly are said to be effective. So, to build great software, we must first build effective engineering teams.

Throughout my 25+ years of experience leading engineering teams at Google and other tech companies, I’ve seen firsthand how team dynamics can make or break a project. Building effective teams is not just about assembling the right technical skills; it’s about fostering a culture of collaboration, trust, and shared purpose. In this chapter, I’ll share some of the key lessons I’ve learned about what makes engineering teams successful, drawing on both research and my own experience in the trenches.

What makes an engineering team effective hinges on the key thing that distinguishes teams from groups. On the one hand, a group is a collection of individuals who coordinate their efforts. On the other hand, a team is a group that is bound by shared responsibilities and goals. Their members work together and share mutual accountability to solve problems and achieve common goals. When teams plan their work, review progress, or make decisions, they consider the skills and availability of all the members and not just those of one individual. This shared goal is what drives an effective team.

I have had the opportunity to observe or be a part of such teams at Google. These teams are passionate about achieving their goals. They find brainstorming sessions fun rather than stressful. Team members may write and test code on their respective machines, but they are collectively tuned in to a unified vision of what the code should achieve. There have been times when they had to resolve some difficult issues, but a culture of collaboration, innovation, and mutual respect helped to see them through such times.

Leaders are an important part of this picture. As a software engineering leader who wishes to make your team effective, you serve as an anchor that connects individual team members to the shared responsibilities and goals of the team. You provide the vision, direction, guidance, and environmental framework necessary to form this connection.

Although it’s possible to have a team without a leader, the team will go much further with the support of a good leader—and that’s where you come in!

Building an effective software engineering team takes work. Many factors can influence the success of a software engineering team, such as team composition, communication, leadership, and work processes. This chapter will explore what traits make teams effective and how to build them into your team. These traits will be things you can look for when hiring, but they’re also traits you can nurture in your existing team.

Research on What Makes Teams Effective

First, let’s examine what makes teams effective. To do so, let us look at some of the extensive research that has already been done on this topic.

Project Aristotle

Google conducted one of the best-known studies on effective software engineering teams, known as Project Aristotle. The project aimed to identify the factors that make some teams more successful than others. The study was based on the premise that the composition of a team was not the most critical factor in determining success but rather how team members interacted with each other.

Note: Before Project Aristotle, there was Project Oxygen, which looked into what traits make for a good manager. Some of the insights in this chapter were informed by the results of Project Oxygen, which I’ll talk about in detail in Chapter 4.

To determine what makes teams effective, the researchers first had to define what effectiveness means and how to measure it. They noticed that different roles had different perspectives on effectiveness. In general, whereas executives were interested in results such as sales numbers or product launches, team members thought that team culture was the key to team effectiveness. The team leaders indicated that ownership, vision, and goals were the most important measures.

Eventually, the researchers decided to study certain qualitative and quantitative factors that might impact team effectiveness, such as the following:

Team dynamics. Demographics, conflict resolution, goal setting, psychological safety

Personality traits. Extraversion, conscientiousness

Skill sets. Programming skills, client management

Researchers conducted interviews and reviewed existing survey data for 180 Google teams. They used this data to run 35 different statistical models and understand which of the many inputs collected impacted team effectiveness.

Project Aristotle identified five key dynamics that contribute to the success of software engineering teams (see Figure 1-1). These are listed next in the order of their importance:

Psychological safety

This was the most important factor identified by the researchers. It refers to the extent to which team members feel comfortable expressing their opinions and ideas without fear of retribution or criticism. Teams that have high levels of psychological safety tend to be more innovative and take more risks, which can lead to better outcomes. The researchers found that when teams feel safe, they:

Are less likely to leave the company

Are more likely to utilize the diverse ideas discussed by the team

Bring in more revenue and beat their sales targets

Tend to be rated highly on effectiveness by their leadership

Dependability

This refers to the extent to which team members can rely on each other to complete their work and meet deadlines. Teams in which individuals trust each other to be dependable are more likely to be efficient and effective in their work.

Structure and clarity

These are conditions under which team members clearly understand the project’s goals and their own individual roles and responsibilities. Team members who clearly understand what is expected of them tend to be more productive and focused.

Meaning

This refers to the extent to which team members feel that their work is meaningful and has a purpose. Teams with a strong sense of purpose tend to be more motivated and engaged.

Impact

This refers to how team members believe their work is making a difference and impacting the organization or society. Teams with a strong sense of impact are more committed to their work and the project’s success.

Figure 1-1. Google’s Project Aristotle: The five dynamics of effective teams

While Project Aristotle’s research was conducted within Google, the identified factors influencing team effectiveness could hold some relevance for teams in other contexts. By focusing on these five factors, software engineering teams can create an environment conducive to collaboration, innovation, and success. As I’ll discuss in Chapter 4, a good manager can foster these dynamics in their teams.

The researchers also discovered that variables such as team composition (size and colocation) or individual attributes (extroverted nature, seniority, tenure, etc.) did not contribute significantly to team effectiveness at Google. While these variables did not significantly impact team effectiveness measurements at Google, that doesn’t mean they’re unimportant, as indicated in the following section.

Other Research

While Project Aristotle is perhaps the best-known study on effective software engineering teams, many other studies have explored factors such as team composition, communication, leadership, and work processes. Here are a few key findings from some of these studies:

Smaller teams are better.

Although Project Aristotle did not recognize team size as relevant to team effectiveness, other studies have shown that smaller teams work better. As a team gets bigger, the number of links that need to be managed among members increases exponentially. Managing these multiple communication channels can be complicated. Many researchers have identified smaller teams containing less than 10 members as more likely to achieve success than larger teams.

Diversity can be beneficial.

It is sometimes suggested that team diversity may lead to communication and coordination problems. For example, a diverse team would usually consist of people from different family backgrounds. Those with young children are more likely to seek flexible work hours, leading to coordination challenges. However, others have found that diverse teams can be more innovative and effective. A study by Lu Hong and Scott Page of the University of Michigan found that groups of randomly selected (likely diverse) high-ability problem solvers can outperform groups comprising the best problem solvers. However, it’s important to note that diversity alone is not enough. Teams must also create an inclusive and respectful environment for all team members. For example, a team that is supportive of team members who need flexible work arrangements will be able to coordinate better than a team that is intolerant of members with such needs.

Clear communication is vital.

Effective communication is essential for effective teamwork. Studies have found that teams that communicate frequently and openly are more successful than those that do not. The idea of psychological safety is a shared belief among team members that they can freely express their thoughts, ideas, concerns, or even mistakes without fear of negative consequences or judgment. Its importance is backed up by the research from Project Aristotle. Clear communication also provides the glue to connect team members and establish structure and clarity within the team.

Leadership matters.

The leadership of a software engineering team can have a significant impact on its success. Google’s Project Oxygen showed that although teams could function without a leader, there is still a need for managers. It identified the essential traits that make for good managers and effective teams. I will talk about these traits in Chapter 4, but for now, it’s necessary to understand that there is a strong correlation between effective leadership and positive team outcomes.

Agility enables adaptability.

Agility is the ability to adapt quickly to changing circumstances. In software engineering, this means being able to pivot when requirements change or when unexpected issues arise. Agile teams are quick to adapt and can work swiftly and efficiently while maintaining high quality. A study by McKinsey & Company found that organizations that underwent successful agile transformations reported a significant improvement in efficiency, speed, customer satisfaction, innovation, and employee engagement, all of which are essential to effectiveness.

Colocation powers innovation.

The debate over whether colocation or remote work is better for software team effectiveness is ongoing, with both approaches having their own advantages and disadvantages. Multiple studies conducted at Harvard, Stanford, and others discuss the benefits of remote or hybrid work in terms of employee satisfaction and retention. However, other studies have shown that face-to-face interactions at the workplace, both planned and serendipitous, trigger the flow of knowledge, sharing of values, and exchange of ideas, which contribute to innovation.

While there may be trivial differences in the findings, we can build a theoretical picture of an ideal effective team based on the research findings discussed in this section. See Figure 1-2. By enabling psychological safety, clarity of structure and communication, dependability, meaningful work, and agility, software engineering teams can create an environment conducive to collaboration, innovation, and success.

You can now build on this understanding of dynamics and factors that influence the effectiveness of teams. The next things to consider are how the working environment can affect teams and how motivation can prime your team for success. As you go through the next sections, notice how the factors that affect teams pop up in various contexts.

3. Leadership Roles: TL, EM, TLM

The below is an excerpt from the middle of Chapter 7: Becoming an effective leader

Organizational structures in software engineering organizations differ widely depending on their culture and priorities. After a person has served as an engineer or senior engineer for a few years and gained the necessary expertise, there are typically two tracks open to them: technical or managerial. Each offers distinct leadership opportunities and requires individuals who can coach and guide their teams through challenges.

In this section, you will look at some typical roles across the industry and what they usually entail in terms of effective leadership. Note that these aren’t the only leadership roles in an organization.

Leadership roles in a team depend not only on the overall organizational structure but also on the size and complexity of the project. Larger teams could have one or many technical leads leading the development of different parts of a project. Additionally, such teams would have architects synchronize the efforts led by the technical leads and managers to plan and organize resources. You could also have a product manager who articulates what success looks like for a product and guides the team to make it a reality. Conversely, in small teams, these roles may be combined to have a manager with the technical expertise to lead the team.

Figure 7-2 shows how some of the different types of leadership roles may coexist in a software engineering team.

Figure 7-2. Relationships between various leadership roles in a software engineering team

Let’s take a closer look at some of these leadership roles.

Technical Lead

A technical lead is a hands-on role where you provide technical guidance and direction to the engineering team. The designation itself may vary across organizations. It may be a formal title in some workplaces, while it exists more informally in others. In some organizations, the position may be identified as a “software architect,” while in others, it may be referred to by titles like “principal engineer” or “lead software engineer.”

Irrespective of the name, tech leads play a crucial role in architectural decisions, code reviews, and mentoring junior team members. Technical leads often bridge the gap between the development team and management, ensuring alignment between technical strategies and business goals. Some of the responsibilities of a technical lead include the following:

Guide technical design and architecture

Tech leads play a vital role in shaping the technical direction of the project by providing guidance on design and architecture. A tech lead must leverage their expertise to ensure that the chosen technical solutions align with the project’s goals and adhere to industry best practices.

Set coding standards and best practices

Tech leads should take the initiative to establish coding standards and best practices within the development team. The tech lead role involves defining and enforcing these guidelines to contribute to overall code quality, maintainability, and consistency.

Lead troubleshooting of complex bugs and issues

Someone in the tech lead role leads the investigation and resolution of intricate technical issues and bugs. Their deep understanding of the codebase empowers them to troubleshoot effectively, ensuring the stability and reliability of the software.

Make key technical decisions with engineering trade-offs

Tech leads are responsible for making critical technical decisions, carefully weighing engineering trade-offs to align with project objectives. They consider factors such as performance, scalability, and maintainability to ensure the overall success of the software.

Do hands-on coding alongside the team

Despite their leadership role, tech leads often find themselves actively engaging in hands-on coding alongside their team members. This approach helps them mentor other engineers while staying connected with the codebase.

Serve as a mentor for development skills

Tech leads also act as overall mentors, guiding team members to enhance their development skills. They lead by example to foster a culture of continuous learning and professional development within the team.

Ensure deliverables meet the quality bar

Tech leads are accountable for the quality of deliverables, ensuring that the software meets established standards and requirements. They conduct thorough reviews and quality assessments to guarantee that the end product aligns with the defined quality bar.

Depending on the size of the project, the scope of these responsibilities will vary—from overseeing a single development team to having cross-team responsibilities.

Engineering Manager

An engineering manager typically oversees a team of software engineers, ensuring the successful delivery of projects. They are responsible for project planning, resource allocation, team productivity, performance, and career development, including that of the tech lead. This role often involves a mix of managerial tasks, such as performance evaluations and career development, along with technical oversight. In some companies, engineering managers may also be referred to as “development managers” or “technical managers.” To recap, an engineering manager’s key responsibilities include the following:

People management

Engineering managers should gear up to develop their skills in hiring, talent development, coaching, and mentoring. Engineering managers actively engage in the recruitment process, nurture their team members’ potential, provide guidance, and foster a culture of continuous learning within their team.

Manage processes

Engineering managers orchestrate critical processes such as sprint planning, retrospectives, and regular one-on-ones. They should ensure these processes are not just executed but tailored to their team’s needs, promoting collaboration, communication, and continuous improvement. They need to check that processes are not sidestepped.

Align team with organizational priorities

Engineering managers must ensure that their team is aligned with the broader organizational priorities. This involves effectively communicating context, goals, and expectations to team members while also shielding them from unnecessary distractions. By serving as a bridge between the team and the larger organization, the engineering manager helps team members focus on their work and deliver value.

Unblock resources

Engineering managers must actively work on unblocking resources needed for execution. They liaise with other departments, manage dependencies, and ensure that their team has the necessary tools, resources, and support to deliver on their commitments.

Technical oversight

While the engineering manager may not have any hands-on coding time, they should maintain their technical acumen. They engage in architecture discussions, ensuring technical decisions align with best practices and organizational goals. This technical oversight helps them guide their team to find sound technical solutions.

Stakeholder interaction

Engineering managers should engage with stakeholders, including having direct interactions with customers. They must understand project requirements, ensure proper communication channels, and act as a conduit between their team and external stakeholders. Engineering managers ensure that the team receives clear requirements from stakeholders.

Strategic work prioritization

Engineering managers must strategically prioritize work aligned with their team and company’s vision. This involves balancing project commitments with essential operational work, addressing technical debt, performing and maintenance in line with the organization’s strategy.

As you take on an engineering manager role, remember that you must broaden your responsibilities to include comprehensive people management, process leadership, and strategic alignment with organizational goals in addition to technical oversight. Unblocking your programmers is also an essential but slightly underrated aspect of managerial responsibilities.

Joel Spolsky, the cofounder of Stack Overflow and creator of Trello, once said, “Your first priority as the manager of a software team is building the development abstraction layer.”1 He further explains that if a developer is directly exposed to infrastructure issues like access to the project repo on GitHub or overriding a firewall for necessary project work, then the abstraction has failed.

Tech Lead Manager (TLM)

Tech lead managers (TLMs) are rare in many organizations. In Google, small or nascent teams usually have a TLM who can oversee a group of engineers, guiding them in project execution and ensuring the team’s productivity. This role involves a mix of technical leadership, project management, and people management. You will need a solid technical background to take up this role and should be able to contribute to technical discussions easily. You should be involved in technical design and communicate relevant design decisions to other teams and stakeholders.

TLMs are responsible for setting priorities, resolving technical challenges, and fostering a collaborative team culture. This role offers the opportunity to do both technical execution and people leadership. But it also comes with the challenge of balancing the two areas while not shortchanging either one. To help with this, TLMs will usually have a smaller number of direct reports as compared to engineering managers. TLM responsibilities include the following:

Blending people management with hands-on technical leadership

TLMs must balance their responsibilities as people manager and technical leader. This involves not only overseeing the professional development of the team but also actively participating in the technical aspects of projects, setting an example for team members.

Coach and develop engineers on coding skills

From a people management perspective, part of the TLM’s responsibility is nurturing their team, coaching, providing constructive feedback, and guiding engineers to enhance their technical proficiency. TLMs must also ensure individual contributors are challenged in their work and are on track to reach their personal career goals.

Establish technical standards and architecture

TLMs are responsible for setting technical standards and architecture. This entails defining and maintaining coding practices, architectural principles, design, and code reviews.

Help unblock developers when they are stuck

TLMs play a crucial role in unblocking developers when they encounter challenges. This involves providing technical guidance, removing impediments, and keeping upper management appraised of the project’s progress and resource needs.

Focus on higher-priority technical work

Sometimes, TLMs may need to concentrate on higher-priority technical initiatives. This could even involve hands-on coding or debugging. TLMs may have to delegate specific people management tasks to balance the other demands of their role. This strategic delegation ensures that both aspects of their role receive adequate attention.

Advocate for the team while coordinating cross-functionally

As the advocate for their team, TLMs engage in cross-functional coordination. This includes representing their team’s interests, ensuring effective communication across departments, and fostering collaboration to achieve collective goals.

Make technical decisions weighing various constraints

TLMs are decision makers in technical matters, which involves considering multiple constraints. This includes assessing factors such as project timelines, resource availability, and technical debt to make informed decisions that align with both short-term goals and long-term sustainability.

Provide mentorship and guidance

TLMs play a crucial role in mentoring and guiding team members to enhance their technical skills and professional development. By dedicating time to mentorship, TLMs foster a culture of continuous learning and growth within the team.

As you can tell from the preceding list, having really strong technical aptitude is critical in a TLM role. A TLM often asks intelligent questions and pushes the team to find answers. TLMs communicate a lot with various people, some of whom are purely technical and others of whom are business oriented. TLMs will thus have to switch their communication style constantly. A sign of success as a TLM is effectively balancing all the responsibilities while finding some extra time to write some code occasionally.

While there may be other roles or other names used to refer to these roles among software organizations, I have tried to discuss the key responsibilities of a team leader or manager in an engineering team in this section. However, responsibilities don’t dictate your ability to perform them. How do you know you have what it takes to lead your teams effectively? Find out by assessing yourself on key leadership traits in the next section.

Parting thoughts

In the preface of the book, Addy outlined who he wrote this book for:

“This book is for engineers wanting to move into leadership roles or engineering leaders who want evidence-based guidance to improve their effectiveness and that of their teams. It is a comprehensive guide to the strategies, frameworks, and best practices that I have found to be most effective in unlocking the full potential of engineering teams and driving transformative results. By sharing real-world examples, practical insights, and actionable advice, I aim to empower you with the tools and knowledge you need to become an exceptional engineering leader in your own right.
At the heart of this book lies a deep exploration of the key traits and behaviors that distinguish highly effective engineers and engineering leaders from their peers. These are the individuals who consistently deliver outstanding results, inspire their teams to reach new heights, and make a lasting impact on the projects and initiatives they lead. By understanding and embodying these characteristics, you, too, can set yourself apart and make a meaningful difference in your role.”

It’s a great time to transition into engineering leadership roles: as there are more and more in-depth resources where engineering leaders like Addy share their hard-earned experience, and way of thinking. Additionally, this book offers a peek at how effective managers at Google operate, and philosophies that are likely to be more common at Google – like the importance of physiological safety, balancing complex interpersonal dynamics, and empowering team members to take ownership of their work.

I hope you enjoyed this deepdive into a more theoretical overview of what we know about effective engineering teams, and a look at how companies like Google think about the TL, EM and TLM roles.

To read on, you can get the book (or e-book.)

And for more reading do check out some of Addy’s other books – including the free e-book titles Software Engineering: The Soft Parts, and The Developer Experience Book. You can also follow Addy on LinkedIn, where he shares learnings on software engineering several times per week.

As related reading, see these past The Pragmatic Engineer articles:

Engineering leadership skillset overlaps: how staff engineers, EMs, PMs, TLMs and TPMs overlap in Big Tech and high-growth startups.

Engineering career paths at Big Tech and scaleups. Levels at Big Tech, the most common software engineering career paths, and what comes after making it to Staff Engineer.

16:37 • 2 months ago

Wednesday, 28. August 2024

Phil Windleys Technometria

Digital Identity and Access Control

In response to a post on X about China's social credit system, Paul Conlon said:

Digital ID is ultimately about access control where those who impose the system are the ones determining what you are required to be and do.
Provision of resources and liberties become conditional upon the whims of the affluent. Doesn't sound safe or convenient to me.
From X
Referenced 2024-08-28T08:10:31-0400

How Paul said this struck me because I've been thinking a lot about access control lately. I believe that we build identity systems to manage relationships, but, as Paul points out, the ultimately utility of identity systems in many cases is access control.

This isn't, by itself, a bad thing. I'm glad that Google controls access to my GMail account so that only I can use it. But it doesn't stop there. If I use my Google account to log into other things, then Google ultimately controls my access to everything I've used it for. This is federation's original sin1.

Paul's comment points out the primary problem with how we build identity systems today: when access control is centralized, it inherently shifts power towards those who manage the system. This dynamic can lead to a situation where individuals must conform to the expectations or demands of those in control, just to maintain their access to essential services or resources. While we often accept this trade-off for convenience—like using Google to manage multiple logins—the broader implications are troubling.

The more we rely on federated identity systems, with their tendency to centralization, the more we risk ceding control over our digital lives, reducing our autonomy and increasing our dependence on entities whose goals may not align with our own. This is why the principles of self-sovereign identity (SSI) are so compelling. SSI proposes a model where individuals maintain control over their own identity, reducing the risks associated with centralized access control and enhancing personal freedom in the digital realm.

Critics of SSI will claim that giving people control over their identity means we have to accept their self assertions. Nothing could be further from the truth. When someone wants me to prove I'm over 18, I use a driver's license. The state is asserting my age, not me. But I'm in control of who I show that to and where. Sovereignty is about borders and imposes a system of relationships.

Now, China could use SSI to build the social credit system. One credential, controlled by the state, that is used to access everything. SSI makes individual control structurally possible, but can’t guarantee it. Technology alone can't solve this problem. As a society, we have to want a digital world, modeled on the physical one, where individuals are the locus of control and use information and assertions from a variety of credentials to build and and interact in peer-to-peer relationships. Until we value freedom and independence in the digital world, we will yield up control of our digital lives to others who will act in their own interests, not ours.

Notes

For similar reasons, I think federated social media systems are a bad idea too, but that's another blog post.

Photo Credit: Papers Please from DALL-E (public domain). Prompt: Draw a rectangular picture of police checking identity papers of people on the street

13:06 • 2 months ago

Bill Wendels Real Estate Cafe

Do HIDDEN listings trick homebuyers & sellers into double paydays, deny consumer savings?

Remember how much fun it was making up plays on the sandlot? Ever try fooling the opposing team with a trick play — a hidden… The post Do HIDDEN listings trick homebuyers & sellers into double paydays, deny consumer savings? first appeared on Real Estate Cafe.

Remember how much fun it was making up plays on the sandlot? Ever try fooling the opposing team with a trick play — a hidden…

The post Do HIDDEN listings trick homebuyers & sellers into double paydays, deny consumer savings? first appeared on Real Estate Cafe.

04:09 • 2 months ago

Tuesday, 27. August 2024

FACILELOGIN

How we built multi-regional routing at the edge @ DevRev

At DevRev, we are building an API-first dev-centric platform that gives you one unified view of your customers and products by connecting your data, teams, and users to our AgentOS platform and app experiences. This blog post shares some insights on how we built routing at the edge in our multi-regional deployment. The DevRev platform was initially deployed only in the US-East region. All data re

The DevRev platform was initially deployed only in the US-East region. All data related to organizations (or tenants) and their customers (workspaces) was stored in the US-East region. This raised two concerns, primarily regarding data residency and latency, which forced us to launch our 1st region outside the US, in Mumbai (ap-south-1), in March 2024, and now we are in the process of expanding our presence into multiple other regions.

The DevRev platform is built with 250+ microservices running on Kubernetes. Building the multi-regional architecture was a significant collaborative effort between the infrastructure and engineering teams. We went through numerous cycles of design, experimentation, and iteration to refine our approach. While future blog posts will explore the various design decisions in greater detail, this post focuses specifically on how we implemented routing at the edge.

Building multi-regional routing involves designing a system that efficiently directs traffic across geographically distributed regions to optimize performance, reliability, and adherence to data residency requirements. For instance, if performance is the key priority, traffic should be routed to the region with the lowest network latency relative to the user’s location. However, this requires replicating business data across multiple regions, which, while beneficial for reducing latency, is not the primary focus of our multi-regional strategy. Instead, we leverage Fastly as our CDN to cache relevant business data at globally distributed Fastly PoPs, ensuring lower latency by bringing the data closer to the user.

The primary goal of DevRev’s multi-regional deployment is to ensure data residency. Regardless of the user’s physical location, their requests are always routed to their organization’s designated home region. The home region is the specific region that hosts the organization’s data, and this region is chosen by the user when the organization is first created. Each organization is permanently tied to its selected region, maintaining compliance with data residency requirements.

Before we delve deeper into the routing design, let’s take a quick look at the DevRev high-level architecture. As shown in the following diagram, all the requests initiated from DevRev API clients are intercepted at the edge, by the corresponding Fastly PoP. All the API requests are first validated by a Compute service deployed in Fastly, at the edge. This will reject any unauthenticated requests at the edge itself, so the backend DevRev services only need to worry about the authenticated traffic.

The second level of enforcement is at the DevRev API Gateway. All the traffic to the DevRev platform flows through the API Gateway, and it centrally enforces authentication with JWT verification. The third level of enforcement happens at the service itself. We have implemented the Service Mesh pattern with Istio; and each Envoy proxy, which sits with the service itself, in the same Kubernetes pod, verifies the JWT that comes with inbound traffic.

Multi-regional routing logic is built into the Compute service running on Fastly. This is the same service that does request validation (along with JWT verification) at the edge.

APIs can be invoked either by authenticated users or through anonymous requests. For instance, when you access the DevRev web app, certain static resources are loaded before the user is authenticated. Similarly, some generic feature flags, which are not user-specific and remain unchanged across sessions, are also fetched without requiring authentication. Another example is the API endpoint https://api.devrev.ai/dev-orgs.public-info.get?slug=your_slug. This endpoint allows the web app to retrieve organization metadata, which is essential for initiating the login flow.

Apart from the authenticated and anonymous requests, there is also another type of request that does not carry a JWT, but some sort of a hint to indicate the region it needs to be routed to. For example, the web socket upgrade call carries an ID in the query string that indicates the region it belongs to. In the same way, the dev-orgs.create API carries a region hint in an HTTP header. Apart from the query string and the HTTP request headers, we also found some APIs that carry the region hint in the request body.

In summary, the routing at the edge has to deal with three types of requests:

The requests that carry a trusted JWT The requests that carry a region hint in the query string, the header, or the body The requests that carry neither a region hint nor a JWT

The following diagram shows what we built at a high level. At the edge (the Compute service) for the above three types of requests, we find the region and then dispatch the request to the corresponding backend. This happens at the Fastly PoP closer to the user, and the Fastly Compute service is aware of all the backends (or the regions) available to dispatch requests.

At the edge (the Compute service) finds the region and then dispatches the request to the corresponding backend. This happens at the Fastly PoP closer to the user, and the Fastly Compute service is aware of all the backends (or the regions) available to dispatch requests.

The Compute service running in Fastly is an edge gateway that intercepts all the requests. This is a module written in Rust, which compiles into a WebAssembly and then runs as a Compute service in Fastly.

In the Compute service code, we have a router that handles a given domain. For example, the api.devrev.ai domain has its router, and the app.devrev.ai domain has its own. Each router defines what needs to be validated in the request path, and a given router is attached to one backend. For example, the router behind the api.devreva.ai domain is attached to the AWS global accelerator for us-east-1 and sets the hostname to api.devrev.ai.

However, this router architecture was not sufficient to support multi-regional routing at the edge. Now, we need to find the backend attached to a router in the runtime based on the request context. In other words, we need to resolve the backend based on the request context. To support this model, we introduced a new construct called, resolver. A resolver is registered against a path, and it knows how to find the corresponding region based on the request context. If there is no resolver, then the default resolver is used, which finds the corresponding region based on a claim in the JWT. The default resolver can only be used for authenticated requests. All the unauthenticated requests (if allowed), are routed to the default region.

In the previous section, we discussed how authenticated requests, which include a JWT, are routed to the backend based on a claim within the token. But who is responsible for adding this claim to the JWT? Let’s unravel that mystery!

As shown in the first diagram, the DevRev web app uses a trusted identity provider, Auth0. Auth0 does not store any user credentials; it simply acts as an identity bus. When a user creates an organization in DevRev, we internally use the Auth0 API to create the corresponding Auth0 organization and update the metadata with the unique organization ID generated by DevRev. This unique organization ID includes the corresponding regional identifier.

When a user logs into their DevRev organization via Auth0 (after being federated through their identity provider or a social identity provider), an action within Auth0 reads the corresponding organization ID and adds it to the token it generates. The Compute service at the edge then reads this claim, determines the corresponding region, and dispatches the request accordingly.

By implementing multi-regional routing at the edge, we ensure that routing decisions are made as close to the user as possible, directly within the request path, and in a transparent manner. Neither the client nor the user needs to be aware of the underlying regional architecture. Each region operates independently, without requiring awareness of other regions, which allows us to seamlessly add new regions without affecting the existing ones.

How we built multi-regional routing at the edge @ DevRev was originally published in FACILELOGIN on Medium, where people are continuing the conversation by highlighting and responding to this story.

23:16 • 2 months ago

The Pragmatic Engineer

How Anthropic built Artifacts

The team behind Artifacts - an innovative new way to interact with Claude - shares how they built this innovative feature in just three months with a distributed team. Exclusive details.

An image picker and color extractor

In the past two months, Anthropic has started to gain momentum among software engineers. The company released its latest language model – Claude 3.5 Sonnet – on 20 June, which works noticeably better than other large language models (LLM) for coding-related work – and gives better results than ChatGPT models, which is wowing many developers. We touched on this observation in The Pulse #101: it’s the first time a company other than OpenAI potentially leads in LLM capability.

Anthropic also released a new feature, Artifacts, alongside Claude 3.5 Sonnet. It allows the creation of things like websites (single page React or HTML), code snippets, documents, diagrams, and more, with a prompt. It’s helpful for a variety of tasks, and also fun. For example, I gave Claude this prompt:

“Create a web application that allows me to tweak the color of The Pragmatic Engineer Logo. This logo is three rectangular bars that increase in height, and are in the color red.”

The result was what I asked for: a mini-web application with an interactive color picker:

The output of the prompt. I could have a blue logo, if I wish! Play with it here, or see a remixed version. Remixing means continuing with the Artifact, evolving it with further input

Other interesting examples include creating:

A game (e.g. “Pong”) with a prompt

A multi-monitor layout designer with a few minutes of giving instructions

A venture capital liquidation preference simulator with instructions provided

While the feature is small, it feels like it could a leap in using LLMs for collaborative work — as every Artifact can be shared, used by others, and “remixed”. Curious to learn how this product was built, I contacted the Anthropic team. Today, we look at the reality of building this feature, and we also get a sense of how Antrophic’s engineering team operates. We cover:

From the drawing board to shipping Artifacts. A scrappy prototype demonstrated on “WIP Wednesdays” kicked off what became Artifacts.

Tech stack. Streamlit, Node.js, React, Next, Tailwind, and Claude.

Using Claude to build Artifacts faster. The team not only dogfooded Claude, but used their LLM to build software faster, including Artifacts.

Timeline and team. A tiny team built and shipped this feature in just 3 months.

AI products and security. Security is part of everything Anthropic does. An explainer on how model security works, and the product security approach for Artifacts.

When is an idea a winner? Not even the engineers building this feature expected it to be as successful as it is.

GenAI paradigm shift? Artifacts is not a massive feature in itself, but it could pave the way for GenAI becoming a much more collaborative tool.

At the time of publishing, Artifacts is enabled by default for all Claude users on web and mobile. So you can try out this feature, if you prefer.

This is an in-depth look into how a complex engineering project was built. Read other, similar, real-world engineering challenge deepdives.

1. From drawing board to shipping Artifacts

Let’s jump into how Artifacts was built. For this, I talked with five current Anthropic employees involved in its creation:

Research scientist, Alex Tamkin, who built and demoed the first prototype

Product designer, Michael Wang, who joined iteration efforts early

Product engineer, Florian Scholz, who helped productionize Artifacts

Security engineer, Ziyad Edher, who helped evaluate security for Artifacts

Brand, Sam McAllister, who created several launch materials for Artifacts

Scratching the itch to move faster

In March 2024, research scientist Alex Tamkin was testing the website generation capabilities of Anthropic’s newest model, using these steps:

Prompt model to generate HTML code for a website

Copy generated code into an editor

Save file as an HTML

Open a web browser to view HTML file

The overhead wasn’t too bad from doing this once or twice. But Alex did it dozens of times, he recalled:

“This whole round-trip process was taking a lot of time. I kept thinking:
‘What if I could just see it right away?’
You know that feeling when you're cooking and you want to taste the sauce straight away, not wait for it to simmer? That's what I was after. I just wanted it to render on the screen immediately.”

So Alex put together a janky side-by-side interface, with Claude on the right and the realtime output on the left. He then showed this rough-around-the-edges demo to his team at a regular catchup session called “WIP Wednesday”:

Alex’s demo for WIP Wednesdays

The demo was a turning point, he says:

“I think this demo was when a lot of us realized: ‘oh wow, there's something here.’
Seeing it immediately on the screen, something sort of... clicks. You know how sometimes you don't realize you need something until you see it? That's what happened.
It wasn't just about making the process faster. It was about changing how we interact with Claude, making it more tangible, more immediate, more collaborative.”

First prototype

One demo participant was product designer Michael Wang, who then helped make the rough demo into a more production-ready experience. He says:

“I just kept replaying this demo from Alex over and over again in my head. So I started building a prototype, mainly to see how much we could actually pull off with some basic prompt engineering and instructions for Claude. Turns out, quite a bit. I had a proof of concept working much faster than I expected. And it just got my mind racing. Eventually, I felt like I had a pretty solid idea, and I posted it to Slack.”

Posting to internal Slack was a great idea, as it got the attention of many colleagues, including Anthropic’s CEO, Dario Amodei, who offered encouragement. After this, things moved quickly, says Michael:

“In about a week and a half, we had it ready for internal dogfooding. The entire company could start using it. It was a bit surreal seeing something go from an idea to a tool that everyone was experimenting with in such a short time. But that's often how it goes when you're working with Claude – sometimes things just click, and you find yourself building something you didn't even know was possible a week ago.”

Michael demoing a polished version of the feature that became Artifacts

As a more polished version took shape, Michael shared the demo internally, gathering even more feedback and encouragement from colleagues:

Michael sharing an update on Artifacts/Iteration Mode on the internal Anthropic Slack New engineer helps ship Artifacts

Engineers at Anthropic have a lot of autonomy, and are expected to take advantage of it. Product engineer Florian Scholz was just getting started at the company, when he saw the demo and decided to help ship the new feature. He recalls:

“Alex's first demo of Artifacts happened in my second week at Anthropic. I was still onboarding in the San Francisco office and adjusting to a very new environment, so I put it on the back burner at the time. Later, when Michael showed a working prototype, I jumped right in.
We all had a common realization that this feature was a step change. My immediate focus was on getting our infrastructure to a place where it was secure. We were concerned about preventing any issues that might arise from untrusted code generated by Claude. It was a pretty great introduction to the kind of fun challenges we face on product engineering at Anthropic.”

Demoing Artifacts with an 8-bit game

With the product ready to ship in beta, there was one last thing to do: create launch materials to showcase Artifacts, and how people can use it. This is where Sam McAllister came in, who leads Brand communications for Claude. After seeing the first prototype of Artifacts, he realized this feature was a truly differentiating UI layer. He’d been using Artifacts as it was built, and put together a demo to showcase the feature: generating an 8-bit game featuring a crab called “Claw’d:”

Screenshot of the “Claw’d” 2D, 8-bit style arcade game, generated by Artifacts. See how it was built in this 1-minute video

Anthropic recently shared a video on how they built Artifacts, which features interviews with research scientist Alex Tamkin and product designer Michael Wang.

2. Tech stack Prototyping phase

First version: When Alex built the early version of Artifacts – one that he showed to a few of his colleagues internally – he used Streamlit. This is a tool to turn Python data scripts into shareable web apps, quickly – to build a prototype for the team.

Using a dedicated prototyping framework to build a “visual proof of concept” turned out to be a helpful approach. It enabled quick feedback, and served as a reminder that the prototype was not (yet) production ready. Of course, not all prototypes become production features, and frameworks that allow research scientists to showcase their ideas are useful, as this case shows.

Second version: Node.js. After getting good feedback, Alex was ready to share the feature with the whole company to try out. For this stage, he decided to migrate the backend from Streamlit. He wanted to use a technology that would work better with more broader usage. Alex explains the reasoning:

“I migrated the app to a Node.js setup and implemented a side-by-side layout for rendering, which I felt improved user experience.
We hold ‘WIP Wednesdays’ meetings at Anthropic, where we share our works in progress with the wider team. Sharing work at a WIP Wednesday like this was a really nice forcing function. I worked late the night before in the office, super focused and just jamming on the prompt and the overall interaction pattern. I paired with Michael too, and he helped me debug what ended up being a simple CORS issue that I was having trouble with. At this point, Claude 3 Opus couldn't actually fix the issue on its own.”

Frontend

The technology used to build Artifact is a common-enough frontend stack used by many web teams:

React: frontend framework used to build the interface

Next.js: a React framework with performance and developer efficiency improvements that many React teams use

Tailwind CSS: a utility-first CSS framework to design delightful user interfaces

Using sandboxing primitives was a notable difference from how most web apps are built. Artifacts needs to isolate untrusted code in the sandbox; the team calls this approach a “secure playground.” As product engineer Florian Scholz puts it:

“This sandboxing approach gives us a clearly defined environment so we can deploy confidently. It's not a static sandbox, we're constantly pushing and expanding on its capabilities. Having this secure playground was instrumental in enabling us to ship so quickly.”

But how exactly did Anthropic build its sandbox; does it use browser sandboxing primitives like the Chrome V8 sandbox? Security engineer Ziyad Edher reveals details:

“We're not using any actual "sandbox" primitive per se.
We use iFrame sandboxes with full-site process isolation. This approach has gotten robust over the years. This protects users' main Claude.ai browsing session from malicious artifacts. We also use strict Content Security Policies (CSPs) to enforce limited and controlled network access.
These approaches protect user data from being exfiltrated through malicious artifacts. We're continuously working on hardening these environments as the browser ecosystem changes.”

Backend

Evolution has reduced the need for a more traditional backend, at least for something like Artifacts. Michael says:

“Our models have gotten so capable that a lot of what you'd normally build as backend logic, you can now just ask for! You give Claude the instructions, lay out the parameters, and get back exactly the structured data you need, formatted just the way you want.
A lot of people looking at Artifacts probably assume there's this incredibly complex backend system running the show.
The reality is, a huge chunk of Artifacts is ‘just’ presentational UI. The heavy lifting is happening in the model itself. It's not that traditional backend work disappears entirely, but the balance shifts. I think we're just scratching the surface of what this approach can do. As these models continue to evolve, who knows?”

3. Using AI to build Artifacts faster

The team behind Artifacts leaned on Claude heavily to build Artifacts. Here’s how research scientist Alex Tamkin used Claude 3 Opus:

“Claude 3 Opus was, at the time, our most intelligent model. The process was straightforward: I'd describe the UI I wanted for Claude, and it would generate the code. I'd copy this code over and render it. I’d then take a look at what I liked or didn't like, spot any bugs, and just keep repeating that process.
It was a really quick way to iterate on ideas!
When you can see something immediately on the screen, there's this moment where things just ‘click’. That's what I was aiming for with this approach – trying to get to those "a-ha!" moments faster.”

Screenshot of the multi-window iteration process to create a simple HTML5 web app with Claude 3 Opus, before Artifacts was ready

Florian Scholz, product engineer on the Artifact team, used Claude extensively, too. He says:

“Claude proved particularly useful as I went digging into the depths of obscure browser APIs and capabilities. I was using it to figure out how to implement specific interaction patterns, like configuring content security policy options, iFrame interactions, and DOM selection APIs. I used it for lots of areas where documentation can be thin or pretty complicated.
Since the launch of Sonnet and Artifacts, I've been using them to jam on experimental versions of new features and get them up and running. Claude usually gives me a good starting point and I can then pair with Claude and iterate from there. I find these tools helpful to avoid the “blank page” problem.”

Within Anthropic, Sonnet 3.5 was seen as a “game-changer,” and pushed the Artifacts team to be more ambitious. Product designer Michael Wang, shares:

“I'm almost always using Claude in my development process. Claude has become such an integral part of my workflow that I'm honestly not sure what I'd do if I couldn't use it anymore. I use it to scaffold out my code, have ongoing conversations about implementation details, and transform code as needed.
Claude 3.5 Sonnet wasn't ready to test during the initial prototyping phases of Artifacts. So at the time, I was primarily using Claude 3 Opus.
When we got an early peek at 3.5 Sonnet, it was a game-changer. Internally, folks were demoing entire Three.js or WebGL apps created by Sonnet in one shot. That's when I knew we could be a lot more ambitious with what we were building. Sonnet had a huge impact on our feature set in the month leading up to the launch. It really pushed us to expand what we thought was even possible with Artifacts.”

4. Timeline and team

Artifacts is one of the most talked-about releases from Anthropic this year, in software engineering circles, anyway! I asked product design engineer Michael Wang about the team size and timeline, from an idea all the way to production. This is how it played out:

“After Alex’s demo, I started working on the prototype on the main claude.ai repository on March 21 2024.
There was one person working on it full time, another part-time contributing on a regular basis. We had a few other helpful hands contributing at strategic points, and a bunch of other Anthropic employees dogfooding along the way.
The project shipped 3 months after the first demo on June 20. We shipped Artifacts alongside our most capable model yet, Claude 3.5 Sonnet.
The whole project felt kind of like a scrappy operation. But that's how some of the best stuff comes together, right?”

Previously in The Pragmatic Engineer, we covered small teams shipping impactful products – such as the dozen engineers who shipped Meta’s Threads app in 6 months. Still, Artifacts might be the scrappiest, high-impact product I’ve encountered! Congrats to everyone at Anthropic who helped build it.

Engineering, design, and research collaboration

16:26 • 2 months ago

Thursday, 22. August 2024

Patrick Breyer

Piratenpartei weist Herstellerkritik an Gamer-Bürgerinitiative gegen Computerspiele-Sterben zurück

Zum Auftakt der Branchenmesse Gamescom weist die Piratenpartei die Kritik des Herstellerverbands Video Games Europe an der EU-Bürgerinitiative zum Schutz von Videospielen vor Fernabschaltung durch den Hersteller zurück. Der Industrieverband bezeichnet den geforderten Weiterbetrieb vom Hersteller aufgegebener Spiele durch die Community als “unrealistisch” wegen der Kosten für Unternehmen und weil private Serverbetreiber Datenschutz und Sicherheit nicht gewährleisten könnten.

Dr. Patrick Breyer, ehemaliger Europaabgeordneter der Piratenpartei, hält dagegen:

Hersteller und EU-Kommission müssen lernen, dass Videospiele keine Konserven sind, die man einfach aus dem Regal nehmen und durch ein anderes Produkt ersetzen kann. Sie sind Teil unseres gemeinsamen digitalen Erbes und verbinden Menschen über Jahre und manchmal sogar Jahrzehnte hinweg. Deswegen ist ein Weiterbetrieb durch die Community nur folgerichtig.

Es gibt längst Spiele, die erfolgreich ehrenamtlich betrieben oder weiterbetrieben werden. Der Datenschutz gilt auch für private ehrenamtliche Betreiber. Gamer, die in eine Übertragung ihrer Daten einwilligen und über die neuen Rahmenbedingungen aufgeklärt werden, sollten nicht wegen möglicher Abstriche bevormundet und ganz ihres Lieblingsgames beraubt werden, sondern eine Wahl bekommen. Die aktuelle Situation, in der Hersteller sich gegenüber voll zahlenden Gamern die willkürliche kurzfristige Abschaltung von einem Tag auf den anderen vorbehalten, ist untragbar.

Die Bürgerinitiative richtet sich zwar speziell gegen das willkürliche Abschalten von Computerspielen durch die Hersteller, aber diese Praxis betrifft längst nicht mehr nur Spiele. Auch andere Software und digitale Geräte bis hin zu Zügen werden fernabgeschaltet. Wir Piraten wollen diesen Rechtsmissbrauch stoppen und gesetzlich den Weiterbetrieb sowie die Weiterentwicklung verwaister Software durch die Community ermöglichen. Dies ist auch im Interesse der Nachhaltigkeit.

Die Piratenpartei setzt sich mit Nachdruck für die europäische Bürgerinitiative „Stop Killing Videogames“ ein und ruft zur Unterzeichnung auf. Die Initiative zielt darauf ab, sicherzustellen, dass Videospiele nicht unspielbar werden, wenn die Unterstützung der Publisher endet. Sie hat kurz nach dem Start bereits etwa 300.000 von 1 Mio. erforderlichen Unterschriften erreicht.

„Initiativen wie ‚Stop Killing Videogames‘ sind essenziell, um die Rechte der Spielenden zu schützen und die Vielfalt der Gaming-Kultur zu bewahren. Wir Piraten fordern, dass vermeintlich nicht lukrative Spiele nicht deaktiviert, sondern der Öffentlichkeit als Open Source zur Verfügung gestellt werden und beispielsweise private Server ermöglicht werden“, erklärt Lukas Küffner, Bundesvorsitzender der Piratenpartei Deutschland.

Auf Anfrage Breyers hatte die EU-Kommission bestätigt, dass es bisher keinen Schutz vor willkürlicher Abschaltung durch Hersteller gibt.

Weitere Informationen zur Bürgerinitiative:
Info-Website
Aufzeichnung Piraten-Webinar
Piraten-Infos

07:38 • 2 months ago

Tuesday, 20. August 2024

Jon Udell

Building a Steampipe Plugin — and Powerpipe Dashboards — for WordPress

“Users of the WordPress API may enjoy the abstraction — and standardization — that a SQL interface provides. If you need to query multiple WordPress sites, Steampipe’s connection aggregator will be really handy. And if you want to integrate data from WordPress with data from other APIs wrapped by other plugins in the Steampipe hub, … Continue reading Building a Steampipe Plugin — and Powerpipe Dash

“Users of the WordPress API may enjoy the abstraction — and standardization — that a SQL interface provides. If you need to query multiple WordPress sites, Steampipe’s connection aggregator will be really handy. And if you want to integrate data from WordPress with data from other APIs wrapped by other plugins in the Steampipe hub, performing literal SQL JOINs across disparate APIs is a heady experience.” — Building a Steampipe Plugin — and Powerpipe Dashboards — for WordPress

Part of the LLM series at The New Stack.

18:31 • 2 months ago

Thursday, 15. August 2024

Heres Tom with the Weather

An error occurred

Classical pianist Jayson Gillham was pulled from an upcoming performance in Melbourne after dedicating a piece of music to the journalists of Gaza. The Melbourne Symphony Orchestra now says the cancellation was due to an error and they appear to be rescheduling the concert.

03:14 • 2 months ago

Monday, 12. August 2024

Damien Bod

Add a Swagger UI using a .NET 9 Json OpenAPI file

This post shows how to implement a Swagger UI using a .NET 9 produced OpenAPI file. The Swagger UI is deployed to a secure or development environment and is not deployed to a public production target. Sometimes, it is required to deploy the Swagger UI to a development deployment target and not the test or […]

Code: https://github.com/damienbod/WebApiOpenApi

Setup

The post uses the OpenAPI Json created by a .NET 9 ASP.NET Core application. See this blog:

Implementing an ASP.NET Core API with .NET 9 and OpenAPI

Two further packages were added to this project, one for the generation of the Swagger UI and the second package to generate the required security headers for an API using JWT Bearer tokens.

Swashbuckle.AspNetCore.SwaggerUI NetEscapades.AspNetCore.SecurityHeaders

The ASP.NET Core API application has already generated the OpenAPI definitions in a Json file. The Json can be used to create the UI. If the application is deployed to production, the Json file is not created and the security headers are deployed with the most restrictions. If the application is deployed for development, the Json is deployed and the security headers are weakened to allow this to work.

// Open up security restrictions to allow this to work // Not recommended in production //var deploySwaggerUI = app.Environment.IsDevelopment(); var deploySwaggerUI = app.Configuration.GetValue<bool>("DeploySwaggerUI"); app.UseSecurityHeaders( SecurityHeadersDefinitions.GetHeaderPolicyCollection(deploySwaggerUI)); // ... other middleware app.MapOpenApi("/openapi/v1/openapi.json"); if (deploySwaggerUI) { app.UseSwaggerUI(options => { options.SwaggerEndpoint("/openapi/v1/openapi.json", "v1"); }); }

The DeploySwaggerUI configuration is used to specify if the deployed version supports both a UI and an API or just an API with the most restrictive security settings.

{ // Open up security restrictions to allow this to work // Not recommended in production "DeploySwaggerUI": true,

Setup security headers

The security headers are setup so that if the deployment is for development, scripts and styles are allowed. The configuration allowing scripts is weak and not recommended for production.

namespace WebApiOpenApi; public static class SecurityHeadersDefinitions { public static HeaderPolicyCollection GetHeaderPolicyCollection(bool isDev) { var policy = new HeaderPolicyCollection() .AddFrameOptionsDeny() .AddContentTypeOptionsNoSniff() .AddReferrerPolicyStrictOriginWhenCrossOrigin() .AddCrossOriginOpenerPolicy(builder => builder.SameOrigin()) .AddCrossOriginEmbedderPolicy(builder => builder.RequireCorp()) .AddCrossOriginResourcePolicy(builder => builder.SameOrigin()) .RemoveServerHeader() .AddPermissionsPolicy(builder => { builder.AddAccelerometer().None(); builder.AddAutoplay().None(); builder.AddCamera().None(); builder.AddEncryptedMedia().None(); builder.AddFullscreen().All(); builder.AddGeolocation().None(); builder.AddGyroscope().None(); builder.AddMagnetometer().None(); builder.AddMicrophone().None(); builder.AddMidi().None(); builder.AddPayment().None(); builder.AddPictureInPicture().None(); builder.AddSyncXHR().None(); builder.AddUsb().None(); }); AddCspHstsDefinitions(isDev, policy); policy.ApplyDocumentHeadersToAllResponses(); return policy; } private static void AddCspHstsDefinitions(bool isDev, HeaderPolicyCollection policy) { if (!isDev) { policy.AddContentSecurityPolicy(builder => { builder.AddObjectSrc().None(); builder.AddBlockAllMixedContent(); builder.AddImgSrc().None(); builder.AddFormAction().None(); builder.AddFontSrc().None(); builder.AddStyleSrc().None(); builder.AddScriptSrc().None(); builder.AddBaseUri().Self(); builder.AddFrameAncestors().None(); builder.AddCustomDirective("require-trusted-types-for", "'script'"); }); // maxage = one year in seconds policy.AddStrictTransportSecurityMaxAgeIncludeSubDomains(maxAgeInSeconds: 60 * 60 * 24 * 365); } else { // allow swagger UI for dev policy.AddContentSecurityPolicy(builder => { builder.AddObjectSrc().None(); builder.AddBlockAllMixedContent(); builder.AddImgSrc().Self().From("data:"); builder.AddFormAction().Self(); builder.AddFontSrc().Self(); builder.AddStyleSrc().Self().UnsafeInline(); builder.AddScriptSrc().Self().UnsafeInline(); //.WithNonce(); builder.AddBaseUri().Self(); builder.AddFrameAncestors().None(); }); } } }

When the UI is deployed to in development mode, the Swagger UI is displayed and the user can enter a valid access token and use the APIs.

Notes

At present, the Swagger UI does not support script best practices and allowing this to work means deploying an unsecure web application with weak security. Any UI web application should use a strong CSP definition, for example like CSP nonces. An API has no UI and so should be locked down.

Links

https://github.com/martincostello/dotnet-minimal-api-integration-testing

https://learn.microsoft.com/en-us/aspnet/core/fundamentals/minimal-apis/aspnetcore-openapi

https://learn.microsoft.com/en-us/aspnet/core/web-api/action-return-types

https://github.com/RicoSuter/NSwag

https://swagger.io/

https://swagger.io/specification/

05:41 • 2 months ago

Saturday, 10. August 2024

Jon Udell

How LLMs Guide Us to a Happy Path for Configuration and Coding

“Some argue that by aggregating knowledge drawn from human experience, LLMs aren’t sources of creativity, as the moniker “generative” implies, but rather purveyors of mediocrity. Yes and no. There really are very few genuinely novel ideas and methods, and I don’t expect LLMs to produce them. Most creative acts, though, entail novel recombinations of known … Continue reading How LLMs Guide Us to a H

Part of the LLM series at The New Stack.

17:21 • 2 months ago

Friday, 09. August 2024

reb00ted

Notes from our DWebCamp session imagining the future of open social networking

At this year’s, DWebCamp, my friend and collaborator Kaliya aka IdentityWoman and I ran a session titled: “Envisioning the future: what if a single, decentralized, interoperable, open social network indeed emerges and wins over the proprietary platforms?” We did something that, at least for me, was new. We started with: Assume it has happened. We have won. Like on page 1 of a sci-fi story

At this year’s, DWebCamp, my friend and collaborator Kaliya aka IdentityWoman and I ran a session titled:

“Envisioning the future: what if a single, decentralized, interoperable, open social network indeed emerges and wins over the proprietary platforms?”

We did something that, at least for me, was new. We started with:

Assume it has happened. We have won. Like on page 1 of a sci-fi story, you wake up in the morning one day and the world you want is has suddenly arrived. At least 4 billion people interact with each other socially on an open, decentralized social network. All of the proprietary platforms have much smaller user numbers or have disappeared entirely; they have stopped mattering. Out of scope: how we get there, or which specific products/standards/technologies are being used.

We asked the question: “Describe what you see”. People who came to our session had a few minutes to meditate on that question, and then wrote down their thoughts, one at a time, on some stickies. They then announced what they wrote and put it on a big sheet of paper. Below is the electronic version of this paper (minus some items which I could not manage to decipher when transcribing):

Community focused and developed by the community Hate free zone Accessible resources (open source) for development by anyone Message flows automatically (I don’t have to decide who are the recipients) biofi.earth for decentralized technology Not harmfully powered Access to all medical data. Do I want all measurements? Which doctor may see what? How do I manage that? Serependipity / surprise me Anti-filter bubble Multi-modal “of one universe” Everything is forkable (in the open-source sense) Everything is fully transparent Attention management / needs to manage Personal and collective sense-making bubble Stigmergy / people marking How to make the network safe? Non-screen-centric “Netflix”-like subscription for all: distributed to builders / operators New funding models for open source Cooperative Attention Economy Fed by opt-in-able data sets Ability to opt-in/opt-out Personal digital exoskeleton / voice-centric Not one systems. Highly fragmented (needs are specific) A shared collaborative universe, not just “apps” Social “digital twin” objects Who you are is what you consume: food and information Photo gallery If it isn’t worth looking at for at least 10 seconds, it isn’t worth any time at all Tap to dive in, call, play, study / zoom out to explore and connect Non-repetitive Profile: Skills Expanse Reviews Map Faves Music Corporate news is dead, replaced by emergent editorial boards Threats community mapping / fear

What surprised me most is that there were so many comments that wanted to have a single “social universe” into which various apps and screens and voice are “windows”", rather than a single app. That’s very close to the original vision for the world-wide-web, and one that we’ve largely lost with all the apps we are using every day. But perhaps we can get it back?

Addition 2024-08-17: two more, which I don’t know how to transcribe.

00:00 • 3 months ago

Thursday, 08. August 2024

Just a Theory

Release: pgxn_meta v0.1.0

Today I released pgxn_meta v0.1.0, a Rust crate and executable for validating PGXN Meta v1 and v2 META.json files.

Following the recent spate of work drafting RFCs for a binary distribution format and Meta Spec v2, and bearing in mind the vote to implementing PGXN v2 apps in Rust, over the last few weeks I’ve switched gears to write some code.

First I wrote JSON Schemas for the v1 spec and then wrote a full test suite using the boon crate. Next I wrote and tested JSON Schemas for the v2 spec. This process informed revisions to the RFC, uncovering inconsistencies and unnecessary complexities.

With META.json file JSON Schema validation fully worked up, I decided to work the functionality into a proper Rust crate and CLI to eventually replace the old PGXN::Meta::Validator Perl module and its validate_pgxn_meta CLI. This turned out to be a decent Rust starter project, requiring a fairly simple crate and CLI, but also allowed me to develop patterns to build and release binaries for a variety of platforms and architecture.

As a result, I’m happy to announce the release today of the pgxn_meta crate and pgxn_meta CLI v0.1.0, available for download on these platforms, thanks to cross and houseabsolute/actions-rust-cross:

darwin-amd64 darwin-arm64 freebsd-amd64 freebsd-i686 illumos-amd64 linux-amd64 linux-arm linux-arm64 linux-i686 linux-powerpc linux-powerpc64 linux-powerpc64le linux-riscv64 linux-s390x linux-sparc64 netbsd-amd64 solaris-amd64 solaris-sparcv9 windows-amd64 windows-arm64 windows-i686

Download the archive file appropriate to your platform, decompress it, and put the pgxn_meta (or pgxn_meta.exe) binary in your path. Or use the universal binary installer (ubi) to install it:

ubi --project pgxn/meta --exe pgxn_meta --in ~/bin

And of course you can use cargo to compile it from source:

cargo install pgxn_meta

Usage is simple: just run pgxn_meta in a directory containing the META.json file to validate:

❯ pgxn_meta 
META.json is OK

And optionally pass it the name of the file, as in this example parsing a test file with no version property:

❯ pgxn_meta corpus/invalid.json 
Error: "corpus/invalid.json jsonschema validation failed with https://pgxn.org/meta/v2/distribution.schema.json#\n- at '': missing properties 'version'"```

That’s it!

What’s Next?

Now that I’ve implemented validation and figured out multi-platform binary support for Rust apps, my next tasks are to:

Implement a pattern to convert a v1 META.json to the v2 format Create a pattern to merge multiple META.json files into one Write code to build PGXS extension into trunk packages Develop patterns to satisfy third-party dependencies for multiple platforms

Should keep me busy for a few weeks. Updates as I have them.

More about… Postgres PGXN JSON Schema Rust Metadata

16:13 • 3 months ago

Wednesday, 07. August 2024

Just a Theory

To Preload, or Not to Preload

When should a Postgres extension be pre-loaded and when should it not? Should it be loaded in user sessions or at server startup? For the Tembo blog, I dug into this question and wrote up my findings.

The Tembo Blog published a post by yours truly last week about when to preload shared libraries and when not to:

Recently I’ve been trying to figure out when a Postgres extension shared libraries should be preloaded. By “shared libraries” I mean libraries provided or used by Postgres extensions, whether LOADable libraries or CREATE EXTENSION libraries written in C or pgrx. By “preloaded” I mean under what conditions should they be added to one of the Shared Library Preloading variables, especially shared_preload_libraries.

The answer, it turns out, comes very much down to the extension type.

I view this post as a kind of proto-chapter for an imagined book about developing extensions that I’d like to work on someday. I learned quite a lot researching it and responding to extensive feedback from more knowledgeable community members. It resulted in updates to the PGXN Meta preload property that I hope will inform binary distribution in the future. More on that soon.

More about… PGXN Postgres Extensions Preload Extensions Book

18:25 • 3 months ago

Tuesday, 06. August 2024

Damien Bod

Implementing an ASP.NET Core API with .NET 9 and OpenAPI

This post implements a basic ASP.NET Core API using .NET 9 and the Microsoft OpenAPI implementation. The OpenAPI Nuget package supports both Controller based APIs and minimal APIs. Until now, we used excellent solutions like NSwag to produce the API schemas which can be used to auto-generate client code. Code: https://github.com/damienbod/WebApiOpenApi Setup A .NET 9 […]

Code: https://github.com/damienbod/WebApiOpenApi

Setup

A .NET 9 project can be created using the .NET Web API templates. The required SDKs or Visual Studio version needs to be installed. The OpenAPI Json definitions can be created using the Microsoft.AspNetCore.OpenApi Nuget package. Microsoft learn docs have all the different possibilities for implementing this depending on your preferred development environment. Here’s an example using powershell:

Install-Package Microsoft.AspNetCore.OpenApi -IncludePrerelease

JWT OAuth authorization is used in this example for the API. I installed the following packages:

Microsoft.AspNetCore.OpenApi Microsoft.AspNetCore.Authentication.JwtBearer Microsoft.Extensions.ApiDescription.Server

The OpenAPI services are added to the project. The API uses JWT Bearer and OAuth and so the OpenAPI definitions should also add this. An BearerSecuritySchemeTransformer implementation was created like in the Microsoft documentation.

builder.Services.AddOpenApi(options => { options.AddDocumentTransformer<BearerSecuritySchemeTransformer>(); });

The BearerSecuritySchemeTransformer class implements the OpenAPI specific definitions. If using cookies, only the document.Info is required and this can be added directly in the services without an extra class. From the Microsoft docs:

internal sealed class BearerSecuritySchemeTransformer(IAuthenticationSchemeProvider authenticationSchemeProvider) : IOpenApiDocumentTransformer { public async Task TransformAsync(OpenApiDocument document, OpenApiDocumentTransformerContext context, CancellationToken cancellationToken) { var authenticationSchemes = await authenticationSchemeProvider.GetAllSchemesAsync(); if (authenticationSchemes.Any(authScheme => authScheme.Name == "Bearer")) { var requirements = new Dictionary<string, OpenApiSecurityScheme> { ["Bearer"] = new OpenApiSecurityScheme { Type = SecuritySchemeType.Http, Scheme = "bearer", // "bearer" refers to the header name here In = ParameterLocation.Header, BearerFormat = "Json Web Token" } }; document.Components ??= new OpenApiComponents(); document.Components.SecuritySchemes = requirements; } document.Info = new() { Title = "My API Bearer scheme", Version = "v1", Description = "API for Damien" }; } }

The middleware needs to be added to the pipeline. You should only add the OpenAPI in development mode unless you require this in production.

app.MapOpenApi("/openapi/v1/openapi.json");

ASP.NET Core supports two types of API implementations; Controller based and minimal APIs. Controller based APIs are used in this project. This API endpoint requires a valid access token and have the different Endpoint definitions.

[Authorize(AuthenticationSchemes = "Bearer")] [ApiController] [Route("[controller]")] public class WeatherForecastController( ILogger<WeatherForecastController> _logger) : ControllerBase { [EndpointSummary("This is a summary from OpenApi attributes.")] [EndpointDescription("This is a description from OpenApi attributes.")] [Produces(typeof(IEnumerable<WeatherForecast>))] [HttpGet("GetWeatherForecast")] public IActionResult Get() { _logger.LogDebug("GetWeatherForecast with OpenAPI definitions"); return Ok(Enumerable.Range(1, 5).Select(index => new WeatherForecast { Date = DateOnly.FromDateTime(DateTime.Now.AddDays(index)), TemperatureC = Random.Shared.Next(-20, 55), Summary = Summaries[Random.Shared.Next(Summaries.Length)] }) .ToArray()); }

You can also add definitions for POSTs and parameters or whatever you require.

[EndpointSummary("This is a second summary from OpenApi attributes.")] [EndpointDescription("This is a second description from OpenApi attributes.")] [Produces(typeof(IEnumerable<WeatherForecast>))] [HttpPost("PostWeatherForecast")] public IActionResult PostWeatherForecast( [Description("parameter post item using OpenApi")] WeatherForecast weatherForecast) { return Ok(weatherForecast); }

When the application is started, the Json OpenAPI definitions can be downloaded and used.

Notes

I am not sure how good the OpenAPI specifications are and need to validate how to define the different status codes in the specifications with the correct return types.

Links

https://github.com/martincostello/dotnet-minimal-api-integration-testing

https://learn.microsoft.com/en-us/aspnet/core/fundamentals/minimal-apis/aspnetcore-openapi

https://learn.microsoft.com/en-us/aspnet/core/web-api/action-return-types

https://github.com/RicoSuter/NSwag

12:13 • 3 months ago

Saturday, 03. August 2024

Randall Degges

I'm Writing Again

I can’t believe it’s been over two years since I last published an article here! Time really flies. The crazy part is that over these last two years, I’ve done more writing than ever; it just hasn’t been public. Several years back, I started a journaling habit, and since then, most of my writing energy has been redirected from my website to my private journal. While writing in private fee

I can’t believe it’s been over two years since I last published an article here! Time really flies.

The crazy part is that over these last two years, I’ve done more writing than ever; it just hasn’t been public.

Several years back, I started a journaling habit, and since then, most of my writing energy has been redirected from my website to my private journal.

While writing in private feels liberating, as I can dump every little thought in my head into an archive, I find myself missing the more structured and thought-out articles I used to push myself to publish.

There’s something magical about writing down your thoughts and sharing them with the world, even if nobody sees them. It forces you to rethink your position, clarify your thoughts, and distill the mess in your head into something direct and actionable.

Over the last couple of years, I have missed that feeling, and the itch to continue writing has returned.

All this is simply to say that I’m writing again. The title of my website has always been “Random Thoughts of a Happy Programmer,” so please stay tuned for more of my innermost thoughts and experiments.

07:00 • 3 months ago

Mike Jones: self-issued

Fully-Specified Algorithms Specification Addressing Feedback from IETF 120

Orie Steele and I have updated the “Fully-Specified Algorithms for JOSE and COSE” specification to incorporate feedback from IETF 120 in Vancouver. Specifically, the registrations for fully-specified Elliptic Curve Diffie-Hellman (ECDH) algorithms in draft 03 were removed, along with the previously proposed fully-specified ECDH algorithm identifiers, while continuing to describe how to create fully

The specification is available at:

https://www.ietf.org/archive/id/draft-ietf-jose-fully-specified-algorithms-04.html

05:13 • 3 months ago

Tuesday, 30. July 2024

Jon Udell

When not to LLM

Here’s the latest installment in the series on working with LLMS: https://thenewstack.io/choosing-when-to-use-or-not-use-llms-as-a-developer/ For certain things, the LLM is a clear win. If I’m looking at an invalid blob of JSON that won’t even parse, there’s no reason to avoid augmentation. My brain isn’t a fuzzy parser — I’m just not wired to see that kind … Continue reading When not to LLM

Here’s the latest installment in the series on working with LLMS: https://thenewstack.io/choosing-when-to-use-or-not-use-llms-as-a-developer/

For certain things, the LLM is a clear win. If I’m looking at an invalid blob of JSON that won’t even parse, there’s no reason to avoid augmentation. My brain isn’t a fuzzy parser — I’m just not wired to see that kind of problem, and that isn’t likely to change with effort and practice. But if there are structural problems with code, I need to think about them before reaching for assistance.

The rest of the series:

1 When the rubber duck talks back

2 Radical just-in-time learning

3 Why LLM-assisted table transformation is a big deal

4 Using LLM-Assisted Coding to Write a Custom Template Function

5 Elevating the Conversation with LLM Assistants

6 How Large Language Models Assisted a Website Makeover

7 Should LLMs Write Marketing Copy?

8 Test-Driven Development with LLMs: Never Trust, Always Verify

9 Learning While Coding: How LLMs Teach You Implicitly

10 How LLMs Helped Me Build an ODBC Plugin for Steampipe

11 How to Use LLMs for Dynamic Documentation

12 Let’s talk: conversational software development

13 Using LLMs to Improve SQL Queries

14 Puzzling over the Postgres Query Planner with LLMs

15 7 Guiding Principles for Working with LLMs

16 Learn by Doing: How LLMs Should Reshape Education

17 How to Learn Unfamiliar Software Tools with ChatGPT

18 Creating a GPT Assistant That Writes Pipeline Tests

19 Using AI to Improve Bad Business Writing

20 Code in Context: How AI Can Help Improve Our Documentation

21 The Future of SQL: Conversational Hands-on Problem Solving

22 Pairing With AI: A Senior Developer’s Journey Building a Plugin

23 How LLMs Can Unite Analog Event Promotion and Digital Calendars

24 Using LLMs to Help Write a Postgres Function

25 Human Insight + LLM Grunt Work = Creative Publishing Solution

20:21 • 3 months ago

Monday, 29. July 2024

Kerri Lemole

Do We Need Blockchain for LERs? No.

When explaining the W3C Verifiable Credential Data Model (VC), Open Badges 3.0, and LERs, I’m often asked about the role of blockchain and if I’m not asked, I feel compelled to explain because misconceptions and confusion are common. Blockchain is one of those technologies that is at once fascinating and cringey, useful and over-used. The business models are still experimental and when it comes down to it, there are simpler, more efficient, and more affordable methods for issuing data rich portable, verifiable digital credentials that meet the needs of issuers, give learners and earners easy access to their data, and a private way to prove that their digitally-signed achievements are theirs for as long as they need them.

You could say that I’ve spent some time thinking about this. In 2015, I co-founded a group called BadgeChain with other Open Badges community members that explored how Open Badges could be more verifiable and have clearer provenance if stored on blockchain. This coincided with Philipp Schmidt and Kim Hamilton Duffy’s leadership on Blockcerts, an open specification for issuing Open Badges on Bitcoin and Ethereum. Learning Machine continued with this work, which was later acquired by Hyland. In 2020 I was the lead researcher on the Education and Blockchain initiative with the American Council on Education and the US Department of Education where we reported on the potential uses of blockchain and then supported several projects that used blockchain or blockchain-like technology for education and training credentials.

I truly believed that we were creating the future and would help society by leveraging what seemed like an obvious use of distributed digital ledgers but what we all learned from this was:

Blockchains are expensive and energy-sucky ways to store and verify credentials It’s a headache to explain to and request public keys from credential recipients Storing data on a blockchain (especially long-living ones like Bitcoin and Ethereum), even when hashed, is risky for PII (Personal Identifiable Information) and means that once it’s on a chain, it can’t be changed or removed. So, if any mistakes are made or data has changed, there’s no way to remove it. It can create a vendor lock for the issuer and the recipient. When the vendor/network goes away, where do those credentials go? It doesn’t free data trapped in centralized silos. It creates new types of silos.

During this time and since, the Credentials Community Group at W3C (CCG) started developing the W3C Verifiable Credentials Data Model (VC). The use cases for this were originally sourced from the members of Open Badges community in 2017. The W3C Verifiable Credentials for Education Task Force was founded soon after by the Digital Credentials Consortium (DCC) and it was this task force which agreed to propose Open Badges 3.0 to 1EdTech in 2021. Now Open Badges 3.0 is final and W3C VC 2.0 is almost complete.

Issuing Open Badges 3.0 is not only about how the achievement is described or the structure of the metadata. The metadata is a critical component to the standard but the bigger upgrade is the alignment to W3C VCs so that digital education credentials like badges, micro-credentials, certifications, and even degrees could be stored, shared, verified, and used on the web in the same secure way as driver’s licenses and citizenship documents that are also issued as W3C VCs. The intention is for all of these credentials to be understandable and verified similarly so that they can be used together.

W3C VCs are cryptographically verifiable, meaning that verfiers can check via software that the data hasn’t changed since it’s been signed, validate the identity of the issuer, and confirm that the credential hasn’t expired or been revoked. They are digitally signed by the issuer and provided to the credential earner. From that point on, that credential belongs to the earner and whatever they do with it is their business only. It may be that issuers ask if earners are willing to tell them where they use their credentials, but inherently, the basic premise of VCs is that they can be used independently. Just because the data is sourced from the issuer does not mean how that data is used should be shared with the issuer if the earners don’t consent. Would you want the DMV to know every time you share your driver’s license? Should the DMV get paid if you use your driver’s license? Even if it was digital? Why should it be any different for education and training credentials?

Admittedly, there are interesting blockchain models to explore that are adjacent to or support W3C VCs and Open Badges 3.0. For example, we’re just starting to touch upon the usefulness of issuer registries and blockchain could be useful there; blockchain-like technologies like Fluree are fueling systems like the Trusted Learner Network (TLN) where distributed nodes of data can be accessed by students and alumni to produce VCs; systems like Ceramic and IPFS can be used for longevity and distribution of evidence and supporting files; some Decentralized identifier methods that use blockchain are being tested around the world.

A model where verifiers pay to access and verify credentials in a tokenized blockchain system compromises the privacy of the credential recipients and traps their credentials in a closed network. In the end, it’s the credential earners that get forced into a closed system where others could profit from their credentials that represent the investment of their time, work, and money. Also, what’s the incentive for verifiers to pay tokens for access to credentials when they could evaluate and accept other credentials on the web for free?

That is the opposite of the intention of W3C Verifiable Credentials and Open Badges 3.0. At the DCC along with many others, we promote the use of W3C VCs as distinct entities of digitally signed achievements that are in the hands of those who earn them, who choose how to use them, and who or what can have access to them. We need to make it easier and affordable for verifiers to get onboard and understand how VCs and Open Badges work so that they become commonplace and useful on the web for everyone.

Do We Need Blockchain for LERs? No. was originally published in Digital Credentials Consortium on Medium, where people are continuing the conversation by highlighting and responding to this story.

17:56 • 3 months ago

What Are Portable, Verifiable Digital Credentials?

The Digital Credentials Consortium (DCC) is advancing the use and understanding of portable, verifiable digital credentials in higher education through open source technology development and leadership, research, and advocacy. What are portable, verifiable digital credentials and why is it important that we advance their use? Digital credentials at their most basic definition are electronic

Digital credentials at their most basic definition are electronic assets that describe an achievement or a qualification. Credential Engine’s Credential Transparency Description Language Handbook describes a credential as “A qualification, achievement, personal or organizational quality, or aspect of an identity typically used to indicate suitability”. There are many different credential types that can be used in education, training, and industry as learning and employment records (LERs) to describe achievements and provide details on related skills and competencies.

When referring to credentials as “portable” and “verifiable” we are indicating that they have been created in alignment with the key web standards that are used to issue digitally signed, metadata rich digital credentials. The W3C Verifiable Credentials (VC) standard is the web standard for digital signatures and proving claims made on the web. Open Badges 3.0 is a standard conformant with W3C Verifiable Credentials that is specifically tailored to issue education and experience digital credentials. Both standards result in distinct electronic assets that can be held by individuals in wallets or other storage to be used later.

Through its participation in community standards organizations including the W3C and 1Edtech, the DCC has helped to develop these standards and is advancing their use and understanding so that individuals may have more access to their data and have the agency to freely and privately share and use their credentials. To accomplish this, the DCC is advocating that issuers of digital credentials use software and vendors that prioritize the use of the LER VC standards. Our members are issuing micro-credentials and even degrees as Open Badges and W3C Verifiable Credentials so that their students and alumni can have digitally signed, data rich markers of achievement that describe what they know and can do that are usable anywhere on the web (and in the world) for their lifetimes.

What Are Portable, Verifiable Digital Credentials? was originally published in Digital Credentials Consortium on Medium, where people are continuing the conversation by highlighting and responding to this story.

15:46 • 3 months ago

Sunday, 28. July 2024

Jon Udell

How and why to write letters to voters

If you don’t live in a swing state, but would like to do more than just send money to help encourage voter turnout in those places, what are your options? For me the best one is Vote Forward, which orchestrates letter-writing to registered voters. I sent hundreds of such letters in 2020 and am aiming … Continue reading How and why to write letters to voters

If you don’t live in a swing state, but would like to do more than just send money to help encourage voter turnout in those places, what are your options? For me the best one is Vote Forward, which orchestrates letter-writing to registered voters. I sent hundreds of such letters in 2020 and am aiming to do lots more, with help from friends, this time around.

Even if I lived in a swing state, I’m not someone who’d be comfortable knocking on doors. And the last thing I want to do is pester people in those places with yet another unwanted phone call or text message. The Vote Forward method is perfect for me personally, and I also think it’s the most clever and sensible way to encourage voters in other states. Here’s how it works.

You “adopt” voters in batches of 5 or 20. I just adopted my first 100: 20 in each of Ohio, Pennsylvania, Michigan, New Hampshire, and North Carolina. You download each batch as a PDF that prints 21 pages. Page one has the instructions and the list of registered voters’ names and addresses

The fact that you write the letters (and address the envelopes) by hand is a great idea. We receive very few hand-addressed letters nowadays, I think they have a pretty good chance of being opened. And once opened, the hand-written message is again unusual. The fact that somebody made the effort to do that signals a rare kind of authenticity.

Likewise, I think the nonpartisan tone of the message is unusual and conveys authenticity. I wish voting were mandatory in the US, as it is in Australia and elsewhere. However the chips fall in November, I would like to know that the result truly reflects what everyone thinks. My message last time was something like:

“… because it’s not really a democracy unless everyone’s voice is heard.”

Pages 2-21 are the letter templates. They look like this:

The hardest part for me was the handwriting. I famously struggled with cursive writing in fifth grade. By the time I reached high school I had reverted to printing. Then, in college, I realized that cursive is more efficient and relearned how to do it. I had to relearn all over again in 2020 because cursive was the fastest way to write all those letters. And I’ll probably have to relearn again this time. I suspect many in younger generations never learned cursive at all, in which case writing the letters by hand will be even harder. So: keep the message short!

If you’ve received a link to this post directly from me, it’ll come with an invitation to drop by our house, hang out on the porch, and help me complete batches of these letters. Otherwise, I hope you might try this method yourself, and/or share it with others. In the past week I’ve switched from doomscrolling to hopescrolling and that’s a huge relief. But I also want to do something tangible (again, beyond donations) and this will be my focus. It feels good to do the work, and will feel really good when I visit the post office sometime in October and drop off a big stack of hand-addressed envelopes.

But is it effective? That’s another thing I like about Vote Forward. They’ve made a sincere effort to measure the impact. And they are honest about the findings: the measurable effect is small. I’ll give them the last word here.

Why should we get excited about small differences?

Because getting people who don’t vote to show up at the polls (or mail in a ballot) is actually pretty hard. Most of the factors that affect whether people vote are tied to big, structural issues (like voter ID laws or polling place accessibility) or deep-seated attitudes (e.g., a lack of faith that elections matter). Given these obstacles, boosting turnout by even a small amount is a real achievement! And, when it comes to politics, we know that many races are decided by tight margins, so a small boost in turnout can translate into a meaningful difference in electoral outcomes.

21:31 • 3 months ago

Thursday, 25. July 2024

Mike Jones: self-issued

Fourth and Likely Last Implementer’s Draft of OpenID Federation Specification

The OpenID Foundation has approved the Fourth Implementer’s Draft of the OpenID Federation Specification. This is a major step towards having the specification become final. The previous Implementer’s Draft was in 2021. A lot has happened since then, largely motivated by feedback from actual implementations and deployments. Some highlights of progress made in the spec […]

The OpenID Foundation has approved the Fourth Implementer’s Draft of the OpenID Federation Specification. This is a major step towards having the specification become final.

The previous Implementer’s Draft was in 2021. A lot has happened since then, largely motivated by feedback from actual implementations and deployments. Some highlights of progress made in the spec since then are:

Changed name from OpenID Connect Federation to OpenID Federation, since Federation can be used for trust establishment for any protocol (including OpenID Connect). Introduced distinct Federation endpoints. Clearly defined and consistently used the terms Entity Statement, Entity Configuration, and Subordinate Statement. Clearly defined which claims can occur in which kinds of Entity Statements. Clearly defined Entity Types and the Federation Entity entity type. Enhanced description of Trust Mark issuance and usage. Defined relationship between metadata and metadata policy. Clearly defined interactions between policy operators. Defined where constraints may occur. Tightened descriptions of Automatic Registration and Explicit Registration. Added Historical Keys. Defined and used trust_chain JWS Header Parameter. Allowed Trust Chains to start with non-Trust Anchors. Clarified use of client authentication. Used OAuth Protected Resource Metadata. Consistent error handling. Added General-Purpose JWT Claims section. Comprehensive use of content types and media types. IANA registration of parameters, claims, and media types. Added and improved many diagrams. Substantial rewrites for increased consistency and clarity. Added Giuseppe De Marco and Vladimir Dzhuvinov as editors.

As a preview of coming attractions, I’ll note that profiles of OpenID Federation are being written describing how it being used in wallet ecosystems and how it is being used in open finance ecosystems. And we’re creating a list of implementations. Watch this space for future announcements.

Special thanks to all the implementers and deployers who provided feedback to get us to this point!

20:26 • 3 months ago

Wednesday, 24. July 2024

Altmode

Kimberley Trip, Day 21: Perth to Home

Wednesday, June 12, 2024 This being our last day in Perth, we had a leisurely start, although that was limited by the need to check out of our hotel room on time. We had breakfast in the hotel, finished packing, and checked our bags for the day at the front desk. We began with a […]

Wednesday, June 12, 2024

This being our last day in Perth, we had a leisurely start, although that was limited by the need to check out of our hotel room on time. We had breakfast in the hotel, finished packing, and checked our bags for the day at the front desk.

Entrance to WA Museum Boola Bardip

We began with a visit to the WA Museum Boola Bardip (Boola Bardip means Many Stories in the local indigenous language), which is located in an area designated as Perth’s cultural center, with theaters and performance venues as well as the Western Australia library and archives. The extensive museum has exhibit halls dedicated to wildlife, cultural, and historical themes. The exhibits, many of them interactive, were well-designed and informative.

It was lunchtime after that, and Kenna remembered a “healthy burgers” chain, Grill’d, that we tried and liked during last spring’s trip to Brisbane. We split a lamb burger, which was a nice change of pace.

We stopped by the Perth Town Hall but couldn’t see much because of an event. The Town Hall is the second-oldest building in Perth, so the receptionist suggested we check out the oldest building, the Old Court House Law Museum. The main room of the building was a small courtroom set up as it would have been in Perth’s early days. The museum also had several exhibits about the evolution of laws applied to Aboriginal people and women, which showed how much things have changed since the beginning of the 20th century and how far we still need to go.

Perth Mint

We continued on to the Perth Mint, where we took a short self-guided tour that explained the Mint’s history and how it related to the Australian gold mining industry. It was fun talking with one of the “coin geeks” there. Today, I learned that successive British monarchs face opposite directions when depicted on coins. Queen Elizabeth II faces right, but King Charles faces to the left when he appears on Australian coins.

We then went to Elizabeth Quay for a break and a small snack and watched our final Perth sunset of the trip from there. We walked back to the hotel, collected our bags, and took a taxi to the airport. Unfortunately, we arrived about an hour before check-in opened, but we waited that out, passed through security, and caught our 11:45 flight to Melbourne. After a short layover we caught a nonstop to San Francisco, a 13-hour flight but a comfortable one. It was good to get home, early on June 13. Now for the challenge of staying awake, to get onto California time!

This article is the final installment in a series about our recent trip to Indonesia and Australia. To see the introductory article in the series, click here.

16:00 • 3 months ago

Tuesday, 23. July 2024

Altmode

Kimberley Trip, Day 20: Rottnest Island

Tuesday, June 11, 2024 We have heard a lot about Rottnest Island, which is just off the coast of Perth. Rottnest (“Rotto” to the locals) is a recreational island with cabins for rent, several beaches with snorkeling (in the summer), restaurants, a golf course, skydiving, and more. It is also the home of about 10,000 […]

Tuesday, June 11, 2024

We have heard a lot about Rottnest Island, which is just off the coast of Perth. Rottnest (“Rotto” to the locals) is a recreational island with cabins for rent, several beaches with snorkeling (in the summer), restaurants, a golf course, skydiving, and more. It is also the home of about 10,000 quokkas. Quokkas are small marsupials billed as “the happiest animals on earth.”

We booked a round trip on the Rottnest Express last night, so we got an early start this morning and grabbed coffee and a breakfast wrap on the way to the pier. We got seats upstairs for both the view and the airflow. The trip started slow (about 10 knots) as we left the pier, increased to about twice that once we reached a broad section of the Swan River, and slowed again as we approached Fremantle. We stopped, picked up more passengers there, and then proceeded to Rottnest Island at a higher speed.

Although we had been equipped with a map, we weren’t sure where to go when we got to Rottnest. We stumbled on the main village and were heading out when we noticed some rain starting to fall. We returned to the village to get some snacks, and the sky opened up: it poured! We were thankful to be in the village as there weren’t a lot of other places to get out of the rain. While there, we noticed our first quokka and gates on the grocery entrance and other businesses designed to keep quokkas out.

Despite a lot of public education not to feed the quokkas, many of them hang out in the village to eat up the crumbs and other droppage from human diners. They are indeed cute, but the “always smiling” thing seemed a little overstated. They can be a little aggressive, too: when Kenna and I shared a chicken sandwich for lunch, a quokka climbed on the bench between us and tried to get to the sandwich. We moved rather quickly.

Osprey Nest at Cape Vlamingh, Rottnest Island

The rain continued off and on, limiting our ability to venture beyond the village until we boarded a bus for a “discovery” tour we had booked for the afternoon. The tour took us around the perimeter of the island, with excellent descriptive commentary from a guide on the bus. We stopped at several points to take pictures, most notably at the far west end facing the vast Indian Ocean, where there was also a large osprey nest.

Following our tour, we stopped briefly to pick up souvenirs (a t-shirt for me), and it was time for our return trip to Perth. Along the way, in one of the slower-speed segments, a couple of hydrofoil surfers managed to “draft” off our ship, keeping up for quite a long time. We were also tailed briefly by a few dolphins.

Returning to our hotel, we were exhausted but managed to venture out for a quick pizza dinner.

This article is part of a series about our recent trip to Indonesia and Australia. To see the introductory article in the series, click here.

16:00 • 3 months ago

Monday, 22. July 2024

Uncle Sams Blog

Harris’ Struggle to Connect Now Imperative

President Joe Biden, grappling with dementia, has finally stepped aside, leaving Vice President Kamala Harris to run in his place for re-election. This is the breath of fresh air Dems…

23:16 • 3 months ago

Altmode

Kimberley Trip, Day 19: Perth and Fremantle

Monday, June 10, 2024 Kenna and I slept in a bit today, then set out into downtown Perth. We grabbed breakfast at one of the many coffee shops. Perth seems to be quite a coffee city, but there are no Starbucks here. We walked to Elizabeth Quay and the Barrack Street jetty, from which many […]

Monday, June 10, 2024

Kenna and I slept in a bit today, then set out into downtown Perth. We grabbed breakfast at one of the many coffee shops. Perth seems to be quite a coffee city, but there are no Starbucks here.

We walked to Elizabeth Quay and the Barrack Street jetty, from which many tour boats originate. We picked up some information on Rottnest Island tours and were surprised when Lee and Colleen, two of our fellow passengers on the Orion, also showed up. We may see them again.

From there, we walked to Kings Park and its botanical garden. We made some wrong turns but eventually found it and discovered it is up quite a hill from downtown and has a great view. The botanical garden is nicely labeled and includes specific sections for areas like the Kimberley. From a distance, we saw Lee and Colleen here too. We could have spent a lot more time here, but we wanted to make it out to Fremantle, so we caught a bus to the train station and a train to Fremantle.

One of our first goals in Fremantle (“Freo” to the locals) was to visit the Indian Ocean for the first time. We went out to a nearby beach and touched the water, which, of course, felt like…water. Soon after that, we ran into a couple of other fellow passengers from the Orion, who had just returned from the Maritime Museum. They told us it was excellent, so we headed over there.

After a quick late lunch in their snack bar, we toured the exhibits. One memorable exhibit for me was Australia II, the Australian winner of the America’s Cup sailing race in 1983. It was a big deal at the time and continues to be a source of pride for Australians.

Fremantle hadn’t initially seemed much like other beach towns (Santa Cruz, Brighton UK, etc.) I have visited. But as we headed more into the city center, we began to see the makings of a beach town: some brewpubs, a game arcade, souvenir shops, etc. It was still somewhat more restrained than many.

Exhausted from several miles of walking, we returned to the train station and rode back to Perth, where we had more walking to do to get to the hotel. We opted to have dinner in the hotel’s pub, which was mediocre and surprisingly closed at 8 p.m. — earlier than most any pub I have seen.

This article is part of a series about our recent trip to Indonesia and Australia. To see the introductory article in the series, click here.

16:00 • 3 months ago

Sunday, 21. July 2024

Aaron Parecki

My IETF 120 Agenda

The sessions I will be attending and presenting at during IETF 120 in Vancouver

Here's where you can find me at IETF 120 in Vancouver!

Monday 9:30 - 11:30 • alldispatch • Regency C/D 13:00 - 15:00 • oauth • Plaza B 18:30 - 19:30 • Hackdemo Happy Hour • Regency Hallway Tuesday 15:30 - 17:00 • oauth • Georgia A 17:30 - 18:30 • oauth • Plaza B Wednesday 9:30 - 11:30 • wimse • Georgia A 11:45 - 12:45 • Chairs Forum • Regency C/D 17:30 - 19:30 • IETF Plenary • Regency A/B/C/D Thursday 17:00 - 18:00 • spice • Regency A/B 18:30 - 19:30 • spice • Regency A/B Friday 13:00 - 15:00 • oauth • Regency A/B My Current Drafts OAuth 2.1 OAuth for Browser-Based Apps OAuth Protected Resource Metadata OAuth for First-Party Apps OAuth Client ID Metadata Document Identity Assertion Authorization Grant OAuth Global Token Revocation

19:54 • 3 months ago

Altmode

Kimberley Trip, Day 18: Darwin to Perth

Sunday, June 9, 2024 Today started quite early because they needed to get us off the Orion early and prepare it in eight hours for the next cruise. We had our suitcases out for pickup at 7 a.m., ate breakfast, and exited the ship about 7:30. We had to go through Australian customs, which went […]

Sunday, June 9, 2024

Today started quite early because they needed to get us off the Orion early and prepare it in eight hours for the next cruise. We had our suitcases out for pickup at 7 a.m., ate breakfast, and exited the ship about 7:30. We had to go through Australian customs, which went smoothly for us but resulted in delays for some others, primarily some MDs who had medical supplies with them and one passenger who had bought a wooden oar somewhere along the way.

There were two busloads of passengers going to the airport. Our bus driver polled the passengers about our departure times and then took us on an impromptu bus tour of Darwin. Darwin is a surprisingly modern and clean city, and the driver was rightfully proud of his home town.

Darwin skyline

We arrived at the airport about three hours before our flight. Unfortunately, we couldn’t check our luggage until 90 minutes before flight time, so we had to wait outside security. There were enough of us there that we were able to use the time to reminisce about the cruise.

When the time came, we obtained our luggage tags and sent our bags down the conveyor. Unlike in the US, we just entered our names and destinations to check our luggage; no ID was required. Similarly, we went through security without showing an ID or our boarding passes, and there was no ID check getting onto the plane. This was very surprising to us!

The flight to Perth was uneventful. When we arrived in Perth, we found the climate to be very different. While it is the dry season in Darwin and the Kimberley coast, it is the wet season in Perth, which has a climate quite similar to the San Francisco Bay Area but shifted by six months. It was rainy and cloudy when we arrived.

When we got off the plane, we were surprised to see Dave’s friend Bruce and his wife waiting for us at the gate! That’s a consequence of not requiring a boarding pass to go through security and was a nice throwback to pre-9/11 practices.

We checked into our hotel, The Melbourne Hotel (yes, located in Perth), and found that we had booked a nicely outfitted but unusual guest room. For example, there was no door to any of the bathroom area; it was open to the bedroom. This is fine, but it takes a little getting used to.

Dave’s friend Bruce had made reservations for a lovely restaurant in North Fremantle,
Bib and Tucker. I had kangaroo for dinner, which had a nice flavor and is a very lean meat (accordingly cooked somewhat rare). Our hosts pointed out that the kangaroo they serve is farmed explicitly for that purpose.

After a very busy day, we said goodbye to Jan and Dave who left for the US that evening and to our hosts. We took the train from North Fremantle back to Perth, where we crashed.

This article is part of a series about our recent trip to Indonesia and Australia. To see the introductory article in the series, click here.

16:00 • 3 months ago

Saturday, 20. July 2024

Altmode

Kimberley Trip, Day 17: Return to Darwin

Saturday, June 8, 2024 Today is an at-sea day as we cross back over the Joseph Bonaparte Gulf to Darwin, where we will disembark from the Orion tomorrow morning. Again, the seas were rough, but not as much as the outgoing crossing. As with other at-sea days, there were lectures from the naturalists and an […]

Saturday, June 8, 2024

Today is an at-sea day as we cross back over the Joseph Bonaparte Gulf to Darwin, where we will disembark from the Orion tomorrow morning. Again, the seas were rough, but not as much as the outgoing crossing. As with other at-sea days, there were lectures from the naturalists and an information session on future cruises. The day ended with the Captain’s cocktail reception and dinner. The cocktail reception featured a slideshow of photos that passengers had submitted (unfortunately, I misunderstood how to submit mine). There was also an auction of a specially-decorated flag from the ship to benefit the crew welfare fund and of a didgeridoo from Goyma’s family, the proceeds from which went back to his family. Happily, both auctions produced high bids for the beneficiaries.

For dinner, Dave and Jan were kind enough to join us in the COVID dining area for our final dinner on the Orion. It was a particularly good dinner, and we witnessed a beautiful sunset and moonset.

This article is part of a series about our recent trip to Indonesia and Australia. To see the introductory article in the series, click here.

16:00 • 3 months ago

Friday, 19. July 2024

Altmode

Kimberley Trip, Day 16: Mitchell Falls

Friday, June 7, 2024 I checked in with Dr. Bob early this morning to remind him that I needed to be checked and cleared to take today’s helicopter flight to Mitchell Falls. He told me that I had been diagnosed too recently and had to be removed from the flight list. I was disappointed. I […]

Friday, June 7, 2024

I checked in with Dr. Bob early this morning to remind him that I needed to be checked and cleared to take today’s helicopter flight to Mitchell Falls. He told me that I had been diagnosed too recently and had to be removed from the flight list. I was disappointed. I pointed out that I had been diagnosed at 7:45 a.m. the previous day, and the cutoff time was given as 8:00. But after further consultation, he called back and said I was good to go because the helicopter ride was somewhat after 8:00.

We made a Zodiac boat transfer to a nearby beach. A few minutes later, three helicopters landed on the beach. We met our pilot and were given a safety briefing (tl;dr: STAY AWAY FROM THE SPINNING ROTORS). Dave and Jan were the other passengers on the helicopter with us. I had the left front seat next to the pilot (who sat on the right, as if we were flying on the left side of the sky).

With an open window next to me, the view was, as expected, gorgeous (both in the sense of attractiveness and the presence of gorges). We rode the 30 miles up the Mitchell River to the falls, mostly admiring the terrain as no particular animals were visible.

On arrival, we hiked up a short trail to get several good views of Mitchell Falls. We had enough time to explore the rocky terrain in the area before returning to the helicopter landing area for our trip back to the beach. The return trip was equally awesome.

For lunch, I joined in with others who had also tested COVID-positive and ate separately from others on the observation deck. This precaution was taken because, even though we were eating outside, eating is one of the times when it is easiest to spread the virus. The upper deck was enjoyable and quiet, although we noticed that several people who we knew had tested positive weren’t there.

After lunch, we had an informative lecture and a trip to another native art site. This schedule was coordinated with another (somewhat larger) ship in the area, Le Jacques Cartier, that we had also seen this morning when visiting Mitchell Falls. Unfortunately, the art site wasn’t big enough to accommodate both ships simultaneously, and we went second. The tide was very low when we went, so we had a very long (and delayed) zodiac boat ride to shore. We had to climb shore rocks to get to where the art was, which was slow for some visitors.

There was quite a bit of Indigenous art to look at, but before we could see it all, an urgent call was put out to return to the zodiac boats because the tide had fallen further. We climbed back down the rocks and waded out farther to reach the zodiac boats. We were disappointed not to have enough time to admire the rest of the art.

Today’s delicious dinner was an “Aussie BBQ” with all the trimmings, which I ate with my fellow COVID patients. After dinner, there was a talent show featuring many of the cooks, waiters, and porters from the hotel staff. It was a nice way to introduce them more personally.

This article is part of a series about our recent trip to Indonesia and Australia. To see the introductory article in the series, click here.

16:00 • 3 months ago

Thursday, 18. July 2024

Altmode

Kimberley Trip, Day 15: Isolation / Hunter River

Thursday, June 6, 2024 I felt a little stuffy this morning, and given the ship’s COVID outbreak, I decided to get tested. I called Dr. Bob, the ship’s doctor, who came by with a couple of swabs, one for me and one for Kenna. A few minutes later, he called back with the result: I […]

Thursday, June 6, 2024

I felt a little stuffy this morning, and given the ship’s COVID outbreak, I decided to get tested. I called Dr. Bob, the ship’s doctor, who came by with a couple of swabs, one for me and one for Kenna. A few minutes later, he called back with the result: I had tested positive, but Kenna was negative. He returned a little while later to check my vitals, listen to my lungs, and deliver Paxlovid, which I had taken on both of my previous COVID cases.

The ship’s protocol for handling this was straightforward: I needed to isolate myself in our room for 24 hours, and Kenna should stay away as much as possible. Breakfast, lunch, and dinner menus appeared on my cabin door. I selected meals, which were delivered to me in the room. Since the cruise ends on Sunday, I will need to wear a mask on the ship for a few days for the rest of the cruise.

Kenna went out to eat with the rest of the ship (specifically with Jan and Dave, whom we had spent time with) and then went on the day’s activities. The morning activity was a ride to see Thor’s Hammer, a tall sandstone rock formation. Kenna’s Zodiac was piloted by Karla, the expedition leader, and paired with Kurt, the geologist, in another Zodiac. Between the two, the group got a good geology lesson with specimens of quartz crystals that Karla borrowed from a rocky outcrop where they stopped.

The afternoon excursion was their best opportunity to see crocodiles as the tide was going out on Porosus Creek. They were still a little early, but after a while finally got a few sightings including a chance to watch a crocodile devour a crab, before having to return to the ship.

There was also a briefing for tomorrow’s helicopter trip. The helicopter company was consulted to see if they would permit masked Covid-positive people on the helicopters, and they said it was OK, provided they were diagnosed before 8 am today (I was diagnosed about 7:45). Whew. I will still need to be checked to see if I’m running a fever (I wasn’t today), so my fingers are crossed that I can go.

My day was mostly spent reading and doing consulting work since I had to stay in the cabin anyway.

In the evening, Kenna was told they had found another place for her to sleep to minimize her exposure. It is a cabin for harbor pilots to use when necessary. We appreciate the creativity of the hotel manager, Fran, in finding this for her. Fran has been amazing through this entire cruise, and this evening brought Kenna a night light to use in the temporary cabin.

This article is part of a series about our recent trip to Indonesia and Australia. To see the introductory article in the series, click here.

16:00 • 3 months ago

Werdmüller on Medium

The Silicon Valley Would-Be Vice President

They pushed JD Vance to serve their own interests. They’re wrong. Continue reading on Medium »

They pushed JD Vance to serve their own interests. They’re wrong.

13:32 • 3 months ago

Wednesday, 17. July 2024

Altmode

Kimberley Trip, Day 14: Montgomery Reef

Wednesday, June 5, 2024 Today’s breakfast came with an announced request to allow the COVID-infected people to pick up breakfast first, after which they would clean the serving utensils and invite the rest of us for breakfast. We were asked to wash our hands at a sink on the back deck to limit any further […]

Wednesday, June 5, 2024

Today’s breakfast came with an announced request to allow the COVID-infected people to pick up breakfast first, after which they would clean the serving utensils and invite the rest of us for breakfast. We were asked to wash our hands at a sink on the back deck to limit any further spread.

Today’s activities consisted of two outings in Zodiac boats, which we just toured and didn’t land anywhere. The morning cruise was around a few nearby islands and into a mangrove-lined area where we hoped to see some crocodiles. We didn’t see any, but we did see some birds and the stunning local geology, and we were treated to some more stories from Goyma, our indigenous guide, who happened to be on our zodiac boat.

Around lunchtime, I ran into Dr. Bob, our ship physician. He said there was a “trickle” of additional COVID cases but that some had recovered, none of the existing cases were severe, and most patients were taking Paxlovid. This is probably appropriate considering most passengers’ ages (we estimate the median age at 73).

After another ship repositioning, we were off on zodiac boats to Montgomery Reef, a unique reef formation. The reef is just below sea level at high tide but emerges more than several feet above water level, to the delight of birds and other creatures. We saw several turtles in the water and many birds (mostly egrets) on the reef and in the air. During the approximately 90 minutes we were there, the reef went from just emerging from the water to over 10 feet above water level, with very substantial amounts of water draining from the edges of the reef.

As we were leaving, we noticed a zodiac boat flying the Lindblad/National Geographic flag. Sure enough, we pulled up alongside it and were treated to tropical drinks served from a punchbowl before our ride back to the Orion.

Dinner was again outside and comfortable; meals will probably be outside for the rest of the cruise. After dinner, they arranged to shut off the lights in one area of the ship and held a star party. We easily saw the Southern Cross, as expected, but also the Big Dipper, except it was pointing down to Polaris, which was below the horizon.

This article is part of a series about our recent trip to Indonesia and Australia. To see the introductory article in the series, click here.

16:00 • 3 months ago

Tuesday, 16. July 2024

Altmode

Kimberley Trip, Day 13: Bigge Island

Tuesday, June 4, 2024 Bigge Island (pronounced “biggie”) is named for someone named Bigge although the island is big. It is the home for some more Gwion Gwion (ancient) and Wandjina (newer) rock art. Starting from the beach where we landed, we took a short hike to the art sites, with narration by our guides, […]

Tuesday, June 4, 2024

Bigge Island (pronounced “biggie”) is named for someone named Bigge although the island is big. It is the home for some more Gwion Gwion (ancient) and Wandjina (newer) rock art. Starting from the beach where we landed, we took a short hike to the art sites, with narration by our guides, particularly our indigenous guide, Goyma. I practiced some of the photography skills taught by our photo experts at some lectures earlier in the trip, with some success.

Mermaid Tree

After lunch (and some repositioning of the Orion), we landed at a different beach with a few boab trees. This included a huge and notable tree called the Mermaid Tree, around which a low platform had been built to protect its roots, much as has been done around some giant sequoia trees in California.

We then walked further along the trail to where a couple of “parade grounds” built by bower birds were located. The bower bird is very plain-looking, so to impress a potential mate, it creates a straw tunnel and decorates its approach with local materials. These aren’t nests, just ways to attract a mate.

Bower bird “parade grounds”

Returning to the ship, we found menus tucked into four doors across the hall, indicating additional COVID cases had been found. Dinner was moved outside to the aft deck, which was very comfortable and pleasant, but a lot of work for the staff since the galley is downstairs, necessitating a lot of shuttling back and forth.

This article is part of a series about our recent trip to Indonesia and Australia. To see the introductory article in the series, click here.

16:00 • 3 months ago

Monday, 15. July 2024

Just a Theory

RFC: PGXN Meta Spec v2

PGXN Meta Spec v2 represents a significant reworking of the original spec for PGXN distributions. It would very much benefit from feedback from Postgres extension and developers.

Two bits of news on the “PGXN v2” project.

PGXN RFCs: The Book

First, I’ve moved the RFC process (again, sorry) from PGXN Discussions, which were a bit fussy about Markdown formatting and don’t support inline comments, to the PGXN RFCs project, where use of pull requests on CommonMark Markdown documents address these issues. This process borrows heavily from the Rust RFCs project, right down to publishing accepted RFCs as a “book” site.

So I’d also like to introduce rfcs.pgxn.org, a.k.a., the PGXN RFCs Book.

It currently houses only one RFC: Meta Spec v1, dating from 2010. This document defines the structure of the META.json file required in archives published on PGXN.

But I expect many more RFCs to be drafted in the coming years, starting with draft RFC–2, the binary distribution RFC I POCed a few weeks ago. There has already been some great feedback in that pull request, in addition to the previous discussion. More eyes will make it even better.

PGXN Meta Spec v2 RFC

Last week I also iterated on the PGXN Metadata Sketch several times to produce draft RFC–3: Meta Spec v2. This represents a major reworking of the original spec in an attempt to meet the following goals:

Allow more comprehensive dependency specification, to enable packagers to identify and install system dependencies and dependencies from other packaging systems, like PyPI and CPAN Adopt more industry-standard formats like SPDX License Expressions and purls. Improve support multiple types of Postgres extensions, including apps, LOADable modules, background workers, and TLEs. Improve curation and evaluation via categories, badging, and additional download links.

There’s a lot here, but hope the result can better serve the community for the next decade, and enable lots of new services and features.

The proof will be in the application, so my next task is to start building the tooling to turn PGXN distributions into binary distributions. I expect experimentation will lead to additional iterations, but feel confident that the current state of both RFC–2 and RFC–3 is on the right track.

More about… Postgres PGXN RFC Trunk Distribution Metadata

19:15 • 3 months ago

Altmode

Kimberley Trip, Day 12: Vansittart Bay and Jar Island

Monday, June 3, 2024 The Orion was repositioned overnight at our next destination on the Kimberley coast. Fortunately, the seas were relatively calm (as compared with the trip to and from Darwin), and we didn’t need to work as hard to walk around. Our morning destination was the site of a World War II plane […]

Monday, June 3, 2024

The Orion was repositioned overnight at our next destination on the Kimberley coast. Fortunately, the seas were relatively calm (as compared with the trip to and from Darwin), and we didn’t need to work as hard to walk around.

Our morning destination was the site of a World War II plane wreck, a C-53 that had gotten lost on a trip from Perth to Broome. They neglected to bring along a navigator. The plane ran out of fuel, tried to land on a mud flat, and skidded into the forest, losing one of its wings and an engine. Fortunately, they were rescued after several days, and everyone survived. The metal frame and skin of the plane are relatively intact, and we had an opportunity to walk around a bit and examine them.

While in the area, we walked through the woods and saw several things described in our preparatory lectures: a boab tree, a paperbark tree (essential to the Aboriginals for many uses), and a potentially quite old termite hill.

We walked back to the beach across the mud flat, examining large sandstone formations along the way.

The afternoon destination was Jar Island, a nearby island featuring ancient Gwion Gwion indigenous rock art. We were given three options for our short hike: “easy,” “moderate,” and “difficult,” with the more challenging routes giving us more to see. We chose the difficult route, which, other than requiring a little use of hands to ascend and descend some rocks, was not that challenging. The rock art was of uncertain (but very old) age and was painted on the sandstone at the three sites we visited.

Returning to the beach just as the sun was beginning to set, we were surprised by an impromptu beach party (cocktails and music) that the staff had set up. The party was a lot of fun, and we returned to the Orion just as it was getting dark.

Upon our return, our festive mood was quickly dispelled by an announcement from the ship’s doctor that three cases of COVID had been found on board the ship. Masks were distributed (but were voluntary), and arrangements were made to isolate the affected people in their cabins for a minimum of 24 hours and to bring their meals to them. I had noticed someone wearing a mask picking up breakfast early that morning and was suspicious that there was an outbreak on the ship. It soon became evident who the affected people were, as dinner menus appeared in the document holders on their cabin doors. Two of the cabins across the hall from ours were included.

This article is part of a series about our recent trip to Indonesia and Australia. To see the introductory article in the series, click here.

16:00 • 3 months ago

Sunday, 14. July 2024

Moxy Tongue

Trump 2024

As Individual American citizens, we elect leaders, not committees. Functional literacy practitioners know the difference. Propaganda is a committee outcome. Mob Rule 101 Committee's are the dog of society, Individual people are the masters. Functional literacy doesn't get that twisted, ever. Freedom == Accountability There is work to do... I am committed t

As Individual American citizens, we elect leaders, not committees.

Functional literacy practitioners know the difference.

Propaganda is a committee outcome. Mob Rule 101

Committee's are the dog of society, Individual people are the masters.

Functional literacy doesn't get that twisted, ever.

Freedom == Accountability

There is work to do... I am committed to it.

Trump 2024 = Vote American Functional Literacy

We're playing a game of American chess; people can say "I hate those Trump pieces" and they can say "I hate those DEI marxist pieces"... doesn't change the game... win the board. People are forgetting the purpose of the game... we're not collecting game pieces. DEI practices are a marxist attack vector on the game board and purpose for playing the game, plainly stated.

Under CCP propaganda leadership it is openly stated, "America is not real, look at black women." The purpose for this statement is to manage Chinese people from believing that America is home to a civil society that functions better than the CCP.

In America 2020-2024... black women are carrying the CCP propaganda water by enforcing DEI practices within bureaucratic enforcement and influence campaigns. Marketing communications companies, in coordination with W2/W4 political hires within the White House and US Government are producing propaganda, setting casting practices for advertising inventories, and expressing a bias in media generally. This is a funded and strategically coordinated work product pursuing a change to the fabric of American society.

You are being hunted, as American citizens.

Program; or be programmed.

Reprogram; or be reprogrammed.

America is a diverse Nation. America is an inclusive Nation. Americans own equity, and the greatest inequity in the world is not knowing how to write an ownership contract for yourself. Functional American literacy is the most leveraged dream for a personal life where the liberty to pursue happiness is unfettered but for the size and nature of the dream being pursued. America invests a tremendous amount of resources improving itself, improving the lives of the people that pledge it allegiance. It matter not whether you bend a knee, because when you stand up for liberty, life and personal happiness, you stand for America. The solution is simple:

American people, Individuals all.

19:49 • 3 months ago

Thursday, 11. July 2024

Justin Richer

Making Bubbles: Three Stages of Identity

One of the key aspects to the bubbles model for federated identity systems is the fact that within the bubble, the account for each user is fully authoritative for that space. But since bubbles don’t exist in a vacuum, that same person probably has accounts that exist in other bubbles. In fact, the attributes in their account probably came from somewhere else to begin with. And of course, our bubb

Three stages of an identity in bubbles

With that model in mind, from the perspective of our bubble, we’ve got three distinct identity processing systems that all need to come together to make things work: the local identity management system for our bubble, something to process inbound accounts, and something to package accounts up for outbound transmission to somewhere else.

The Local Bubble

Within the bubble itself, we are using a cohesive IdAM system and are almost certainly using federation technology to connect out to a set of RP’s within the bubble. All of these systems can look towards one authoritative IdP within the bubble for the source of all account information.

Inside the bubble, we have tons of freedom for how we want to connect our users to our systems. While we probably want to use current best-of-class technologies like OpenID Connect and passkeys, we only really need to be compatible internally, using whatever makes the most sense for our environment.

The important thing here is that each user has an account that is accessible within the bubble at all times, and is not dependent on reaching out to anything outside the bubble for local authentication.

Inbound Processing

Most of the users in a bubble probably came from somewhere. If we onboard an account from an external system, it means that we’re creating an account based on a set of attributes from a known source. These attributes can come in with an assertion, credential, certificate, API call, or some other technology. The important thing, for us, is that we can now tie these attributes to a known account, and we can cache the attributes as we received them. A lot of these are going to be immensely useful — we won’t have to have every user type in all their attributes every time they connect into a new bubble.

But it’s not enough that we’re just making a cached copy. In many cases, we’ll want to override or update these attributes locally, but we don’t necessarily want to lose the data from the source when we do that override. After all, we don’t control the data source, and we want to know where all of our information came from.

We can use an overlay style data structure that lets us keep both updated data and the source data at the same time. Let’s say, for instance, that Patty O’Sullivan gets an account onboarded into the system, but it turns out that everyone inside the bubble just calls her Sully. We can create a local value that overrides the official value, but the official value doesn’t go away: it’s still sitting in its own structure. If we don’t have an override, when we look up an attribute we can follow a pointer to an upstream source and get it directly without having to copy it.

The approach also allows us to very efficiently take care of cases where we don’t have a need for referencing an attribute that was handed to us, or that we need to create a brand new attribute that doesn’t exist at the source. And in fact, this pattern can be applied up the chain, since our source might have gotten its information from somewhere else in the first place.

And we can just keep copying this pattern, even pointing at multiple sources at the same time. We can optimize this graph structure for both storage size and lookup efficiency, but more importantly it allows us to keep the data sources separate from each other in a meaningful fashion. We can tell where we’re getting each attribute value from, and we can differentiate between local updates and data copied from elsewhere.

This also means that we can put restrictions on data from different layers. For example, maybe we want a policy that needs an update on a cached value every so often. Or if I’m doing a local override of an important attribute, like one that gets used in security decision making, then I need to check that the override is still valid after a certain timeout. This can avoid a class of configuration errors that we see in the field, where something gets changed in order to solve an immediate problem, but never gets changed back when things de-escalate.

Outbound Packaging

And of course, we also want our bubble to be able to act as the source for some downstream receivers as well. In order to do that, we need to be able to package up our accounts and assert them outbound.

But wait a moment — isn’t that the same exact thing we’re doing inside of the bubble for our apps? Aren’t we already going through a federation process to connect on the inside? Shouldn’t we just use that same IdP again, since it’s already set up and has all the same accounts?

While it would be possible to re-use the same component, it makes more sense to have a dedicated IdP that only speaks to external receivers. This separation allows us to deliberately control which information we share and with whom, and without it being conflated with local policy, changes, overrides, and other concerns. When we’re talking to an external receiver, we likely want to give a very specific view of an account in this context, especially considering that we want to minimize the transmission of sensitive data across boundaries.

Stacking the Three Pieces Together

Each identity system we’ve talked about here has a distinct role to play. In this way, the three parts of a bubble system — inbound, local, and outbound — can work together to create a cohesive path for an account, its attributes, and the person who’s using it.

20:40 • 3 months ago

Mike Jones: self-issued

Fully-Specified Algorithms Specification Addressing Working Group Last Call Comments

Orie Steele and I have updated the “Fully-Specified Algorithms for JOSE and COSE” specification to incorporate working group last call (WGLC) feedback. Thanks to all who took the time to comment on the draft. Your feedback was exceptionally actionable and helped to substantially improve the specification. Responses to each WGLC comment thread were sent on […]

The updated draft attempts to discuss the full range of the problems created by polymorphic algorithm identifiers. Guided by working group feedback, it strikes an engineering balance between which of these problems to fix immediately in the specification and which to describe how future specifications can fix later as the need arises.

I look forward to discussing next steps for the specification at IETF 120 in Vancouver.

The specification is available at:

https://www.ietf.org/archive/id/draft-ietf-jose-fully-specified-algorithms-03.html

17:48 • 3 months ago

Patrick Breyer

Hilf jetzt mit die Chatkontrolle zu stoppen!

Die Chatkontrolle ist wieder auf der Tagesordnung der Regierungen der EU. Am 23. September sollen sich die EU-Regierungen zum neuesten Vorstoß positionieren. Die EU-Innenminister sollen die Chatkontrolle am …

Im Juni war es uns gelungen die beispiellosen Pläne mit einer extrem knappen „Sperrminorität“ der EU Regierungen aufzuhalten: Die Befürworter der Chatkontrolle erreichten damals 63,7% der 65%-Mehrheit, die im Rat der EU für eine qualifizierte Mehrheit erforderlich ist.

Zahlreiche Regierungen wie Frankreich, die sich früher gegen die Pläne gestellt hatten, haben ihren Widerstand bereits aufgegeben. Viele Regierungen wie die deutsche Bundesregierung, die immer noch kritisch zum Vorschlag stehen, melden nur noch geringfügigen Änderungsbedarf an (z.B. eine Ausnahme für Ende-zu-Ende-Verschlüsselung ), womit unsere private Kommunikation von den meisten Diensten immer noch massenhaft durchsucht und geleaked würde. Darum besteht jetzt aktuell die konkrete Gefahr, dass die benötigte Mehrheit für die massenhafte Durchleuchtung privater Kommunikation während der ungarischen Ratspräsidentschaft erreicht wird (Ungarn unterstützt den Vorschlag).

Deshalb müssen wir uns alle engagieren, unsere Stimmen gegenüber unseren Regierungen erheben und die Bevölkerung aufklären.

Regierungen, welche den Chatkontrolle-Vorschlag bisher unterstützt haben, müssen überzeugt werden ihre Position zu ändern Sich kritisch verhaltende Regierungen müssen dazu gebracht werden, umfassende und nicht nur kleine Änderungen am Vorschlag zu fordern, so wie sie das Europäische Parlament fordert.

Dieser Blogartikel erklärt verschiedene Möglichkeiten, wie du aktiv werden kannst. Jede*r kann sich einbringen. Um uns erfolgreich gegen die Chatkontrolle zu wehren, müssen wir uns alle engagieren!

Auf dieser Karte (die du gerne online teilen kannst!) kannst du die Positionen der EU-Mitgliedstaaten zur Chatkontrolle Stand 4. September 2024 laut eines geleakten Berichts sehen. Die Länder sind außerdem in der Tabelle unten aufgeführt.

Ist deine Regierung für die Chatkontrolle?
→ Verlange eine Erklärung und dass die Regierung ihren Kurs ändert.

Enthält sich deine Regierung?
→ Frage sie nach den Gründen und fordere, dass sie sich in Zukunft entschieden gegen die Chatkontrolle einsetzt.

Ist deine Regierung gegen die Chatkontrolle?
→ Prima, aber schau genaue hin: Einige Regierungen, wie z. B. die deutsche Bundesregierung, lehnen nur das Scannen verschlüsselter Kommunikation ab, sind aber mit dem anlasslosen Scannen anderer privater Kommunikation einverstanden, oder mit dem Ende anonymer Kommunikation durch verpflichtende Alterskontrollen, oder mit der Einführung eines Mindestalters für vermeintlich „riskante“ Messenger und Apps zur Kommunikation. Auch müssen Regierungen jetzt ihren Einfluss im Rat der EU besser geltend machen und sich mit anderen kritischen Staaten auf gemeinsame Forderungen verständigen und den Vorschlag damit grundliegend überarbeiten. Wenn diese Änderungen nicht umgesetzt werden, sollten die Regierungen von der Europäischen Kommission eine Rücknahme des Chatkontrolle-Vorschlags verlangen.

Wie deine Regierung zum aktuellen Chatkontrolle-Vorschlag steht Für den VorschlagNicht für den VorschlagUnentschieden / unklarBulgariaAustriaBelgiumCroatiaEstoniaCzech RepublicCyprusGermanyFinlandDenmarkLuxembourgItalyGreecePolandNetherlandsHungarySloveniaPortugalIrelandSwedenLatvia

Lithuania

Malta

Romania

Slovakia

Spain

Die sechs Länder, die sich kritisch geäußert haben, reichen noch nicht für eine Sperrminorität aus.

Das kannst du tun

Es gibt Maßnahmen, die du sofort und kurzfristig ergreifen kannst, und solche, die mehr Vorbereitung erfordern. Für den Anfang:

Bitte deine Regierung, von der Europäischen Kommission die Rücknahme des Vorschlags zur Chatkontrolle zu fordern. Weise sie auf einen gemeinsamen Brief hin, der gerade von Kinderrechts- und Digitalrechtsgruppen aus ganz Europa veröffentlicht wurde. Klicke hier, um den Brief und weitere Informationen zu finden. Aus Deutschland? Fordere die Bundesregierung dazu auf, die Chatkontrolle konsequent und vollständig abzulehnen und den eigenen Einfluss im Rat der EU zum Schutz unserer Grundrechte geltend zu machen. Aus einem anderen Land als Deutschland? Überprüfe das Abstimmungsverhalten deiner Regierung (siehe oben) und bitte sie um eine Begründung, falls sie für oder gegen die Chatkontrolle gestimmt hat. Sage ihnen, dass du als Bürger*in willst, dass sie den Vorschlag ablehnen, dass die Chatkontrolle von vielen Expert*innen kritisiert wird und dass keiner der bisher im Rat der EU vorgelegten Vorschläge akzeptabel ist. Fordere sie auf, die Privatsphäre deiner Kommunikation und deine IT-Sicherheit zu schützen. Teile diesen Aufruf zum Handeln online.

Die besten Anlaufstellen in deiner Regierung sind in der Regel das Innenministerium (federführend) und Ministerien für Justiz und für Digitalisierung / Telekommunikation / Wirtschaft. Außerdem gibt es für jedes Land eine Ständige Vertretung bei den EU-Institutionen. Die Kontaktdaten der Ständigen Vertretungen findest du, indem du hier klickst.

Es kann auch sehr helfen sich an die Abgeordneten in den nationalen Parlamenten zu wenden, da sie das Stimmverhalten ihres Landes festlegen können. Sprich deine politischen Vertreter*innen an. Ob es sich um die neu gewählten Abgeordneten des Europäischen Parlaments oder um lokale Gruppen der politischen Parteien handelt: Sie alle sollten erfahren, worum es bei der Chatkontrolle geht und dass du von der Politik erwartest, deine Grundrechte gegen den Chatkontrolle-Vorschlag zu verteidigen!

Tipp: Wenn du mit Politiker*innen in Kontakt treten willst, solltest du möglichst einen echten Brief schreiben, im Büro anrufen oder zu einer lokalen Parteiveranstaltung oder zu Öffnungszeiten des Büros gehen und dort ein persönliches Gespräch führen. Das hat eine noch größere Wirkung als eine E-Mail. Die Kontaktdaten findest du in der Regel auf den Websites der Politiker*innen und der Ortsgruppen der Parteien. Denk daran, dass du deinen Standpunkt entschlossen vertreten, dabei aber auch höflich bleiben solltest, da sie dir sonst nicht zuhören werden.

Hier findest du eine Argumentationshilfe zur Chatkontrolle. Und nachfolgend mehrere Erklärungen, warum die bisher von den Regierungen vorgesehenen Änderungen nicht ausreichen, um den Gefahren der Chatkontrolle entgegenzuwirken: von uns, von EDRi, von CDT.

Während wir den Kampf gegen die Chatkontrolle fortführen, müssen wir den Widerstand noch weiter ausbauen:

Erklär deinen Bekannten, warum das ein wichtiges Thema ist. Dieses kurze Video, das in alle europäische Sprachen übersetzt wurde, kann ein hilfreicher Anfang sein. Du kannst es frei nutzen und teilen! Auch verfügbar auf YouTube und PeerTube. Aktiv zu werden funktioniert besser und ist motivierender, wenn man zusammenarbeitet. Versuche also, Verbündete zu finden und Allianzen zu schmieden. Ob in einem lokalen Hackspace oder in einem Sportverein: Deine lokale Aktionsgruppe gegen Chatkontrolle kann überall an den Start gehen. Dann könnt ihr kreativ werden und überlegen, welche Art von Aktion am besten zu euch passt.

Werde jetzt aktiv. Der Widerstand gegen die Chatkontrolle sind wir alle!

14:46 • 3 months ago

#EuGoingDark-Überwachungsplan: Termine, Agenda, Hintergründe

Hoch kontrovers, intransparent und kaum hinterfragt: Zur Zeit bereiten Kommission und Rat der Europäischen Union ein neues, EU-weites Überwachungspaket vor. Der #EuGoingDark-Plan umfasst unter anderem die Wiedereinführung und Ausweitung der Vorratsdatenspeicherung von Kommunikationsdaten der Bürgerinnen und Bürger sowie konkrete Vorschläge zur Aushebelung der sicheren Verschlüsselung von Daten auf allen vernetzten Geräten von Autos bis Smartphone, Daten bei Diensteanbietern und Daten während der Übertragung.

„Journalistisch und politisch ist der #EuGoingDark-Plan nach wie vor ein blinder Fleck, obwohl eindeutige Dokumente vorliegen und das Ausmaß der geplanten Maßnahmen erschreckend ist“, sagt der Europaabgeordenete der Piratenpartei Dr. Patrick Breyer.

Dokumente

Themenseite zur »Hochrangigen Gruppe für den Zugang zu Daten für eine wirksame Strafverfolgung« mit Hintergründen, Presseberichten, Dokumenten und der Position von Dr. Patrick Breyer.

Zeitleiste

6. Juni 2023: Entscheidung der EU-Kommission zur Gründung der#EuGoingDark-Arbeitsgruppe Zwischen Juni 2023 und Juni 2024: Jeweils vier Treffen der #EuGoingDark-Arbeitsgruppe und ihrer drei Unterarbeitsgruppen, von denen aktuell drei auf der Website teilweise dokumentiert sind. 6. Juni 2024: Veröffentlichung von 42-Forderungen der #EuGoingDark-Arbeitsgruppe 1. Juli 2024: Beginn der ungarischen Ratspräsidentschaft, mit Ankündigung der Weiterverfolgung des #EuGoingDark-Programms („European consensus on the retention and access to law enforcementdata“) Für Q2 angekündigt: Zweiter Eurojust Data Retention Report 16. Juli 2024: Konstituierende Sitzung des neuen EU-Parlaments

Ab September:

Termin noch unbekannt: Abschließender Bericht zu den Empfehlungen der #EuGoingDark-Arbeitsgruppe 3.-5. September: Gruppe “Zusammenarbeit in Strafsachen” (COPEN) Termin noch unbekannt: Rede zur Lage der Union d. Präsiden*tin der Europäischen Kommission Termin noch unbekannt: Annahme des Arbeitsprogramms der Kommission für 2025 10-11 Oktober 2024: Rat (Justiz und Inneres) 16 – 17 October: EU -U.S. Senior Officials Meeting on Justice and Home Affairs November: Abschlusstreffen der #EuGoingDark-Arbeitsgruppe 25. November: Safe in Europe Forum 2024 Termin noch unbekannt: Gemeinsame Prioritäten der EU-Organe ab 2025 9-11 Dezember 2024: 140th Management Board Meeting of Europol 12-13 Dezember 2024 Rat (Justiz und Inneres) mit “exchange of views on Access to data for effective law enforcement” 1. Januar 2025: Beginn der polnischen Ratspräsidentschaft

Offene Fragen

Wer hat an den Treffen der EuGoingDark-Arbeitsgruppe und ihrer Unter-Arbeitsgruppen teilgenommen? Teilnehmerlisten sind nach wie vor Verschlusssache . Mullvad VPN hat berichtet, dass, mindestens ein ehemaliger US-Geheimdienstler teilgenommen hat. Warum wurden digitale Grundrechte-NGOs wie etwa EDRi auch auf Nachfrage nicht zu den Treffen der Arbeitsgruppe eingeladen obwohl diese laut Website eine „kollaborative und integrative Plattform für Interessengruppen“ ist? Wurden die Transparenzregeln der Gruppe mitten im Arbeitsprozess heimlich ausgetauscht und dabei massiv eingeschränkt? Die schriftliche Anfrage von Dr. Patrick Breyer dazu ist nach wie vor unbeantwortet. Warum sind die Arbeitsgruppe und ihre Unter-Arbeitsgruppen nicht im Register für Expertengruppen und ähnliche Foren (z. B. Beratungsgremien) der Kommission eingetragen, wodurch ihre Transparenz stark eingeschränkt ist? Wie bewerten die neuen, direkt und indirekt mit den betroffenen Themen befassten Kommissarinnen und Kommissare der EU-Kommission, die Europäischen Bürgerbeauftragte, die Datenschutzbeauftragten der EU und der Mitgliedsländer, IT-Sicherheitsexperten, Verbraucherschutzverbände und andere die Arbeitsweise und Pläne der #EuGoingDark-Gruppe?

Kontakt zur #EuGoingDark-Arbeitsgruppe

European Commission
Directorate-General Migration & Home Affairs
46 Rue de Luxembourg | B-1049 Brussels | Belgium
E-Mail: EC-HLG-GOING-DARK [:at:] ec.europa.eu.

Olivier Onidi
Deputy Director General
Directorate-General for Migration and Home Affairs

Carlo Van Heuckelom
Justice and Home Affairs Counsellor
Permanent Representation of Belgium to the European Union

14:12 • 3 months ago

Moxy Tongue

Root Administrator: Owner

DARPA enabled the creation of TCP/IP as means of overcoming limitations of networks communicating with other networks, to yield an "inter-network" outcome of communication portability, on battlefields and beyond. This is the origin of ARPAnet. NSF helped commercialize ARPAnet by formation of Network Solutions, Inc and deployment of ICANN, ARIN, to serve TCP/IP domain registry to public. This publ

NSF helped commercialize ARPAnet by formation of Network Solutions, Inc and deployment of ICANN, ARIN, to serve TCP/IP domain registry to public. This public internet of IP addresses and domain names was released in 1996 for public consumption.

Vint Cerf, co-creator of TCP/IP at DARPA, via private communication with author while at MCI, in question about why TCP/IP deployment via domain registries prioritized a renter model of cost support rather than an owner model, answered: "that was considered in 1976 and discarded".

As a result of this discarded consideration, and subsequent growth of TCP/IP use in American/Global civil societies, the original intent of DARPA within the engineering design choices made in 1976, prior and after, end "users" of this innovative technology have received a relationship structure via this infrastructure giving meaning and utility to data at scale ("Big Data"), and as of 2023, artificial intelligence systems deriving utility from public sources of data derived by human interactions. This relationship structure is well-informed by basic definitions in computing and economics, such as "renter", or "slave".

Data Slave: Renter

As a result of an engineering design process aimed at goals conceived of by DARPA, and subsequent deployment of public access technologies within American civil society in coordination with the National Science Foundation, "We The People" have received an architectural relationship to the utilities built atop this infrastructure. As can be viewed by inspecting the subsequent TCP/IP dependent solutions where human data interactions are enticed/suggested/served/supported/scaled, people have been packaged as "data product" to be sold to "customers of data context" by the "Master" of such web interactions as a webmaster may serve from time to time. Facebook and Google may serve as the leading edge of this infrastructure, where the relationship is best observed and harvested by social graphing systems.

Root Administrator: Owner

Sovereignty has borders; American Sovereignty is born of the people giving it authority, Individuals all. "We the People" exist as a data structure, a deeply personal one. In every zip code all over our land, in every home where families live and grow, in every pocket where every device is held by those Individual people, a data structure of relationships exists as the deeply held asset of American Sovereignty itself.

How should you be organized by default?

Own root. Every human life that acquires a birth certificate, whether for your child or another, or for yourself when needed, understands how a data structure works. Lawyers will understand that "Rights" are for people, not birth certificates. But try proving that at the border of American Sovereignty where it matters. Words don't accomplish much in those moments, especially legalese.

Structure yields results at the border. Root administration over the data structure of American Sovereignty is an edge-driven experience. It happens at the authentic border of authentic Sovereignty itself, or it doesn't. And when it doesn't, there is a reason. This is where failures carry meaning, and can instruct us on flaws in need of fixes.

When DARPA yielded TCP/IP and the NSF made it public, there were other examples that perhaps the notions of future utility were not adequately conceived, and were going to need fixes to flaws faster than originally understood. IPv4 -> IPv6 is one such example... the system was going to run out of addressable spaces to serve the inter-network to users. Other examples are the increasing absence of a default archival system, and IAM work to service the identity space where citizen identifiers are utilized broadly, as the ultimate data silo.

If given a choice, which one do you look to by default for more information:

Slave: Renter

Root: Owner

Without any additional information concerning architectural choices, infrastructure requirements, engineering design choices, or participatory patterns of engagement... which do you select?

Functional literacy for American citizens REQUIRES a new consideration of 1976 choices, once more.

13:53 • 3 months ago

Mike Jones: self-issued

OAuth 2.0 Protected Resource Metadata draft addressing shepherd comments

The “OAuth 2.0 Protected Resource Metadata” specification has been updated to address feedback from our document shepherd Rifaat Shekh-Yusef in advance of IETF 120 in Vancouver. All changes were strictly editorial. The specification is available at: https://www.ietf.org/archive/id/draft-ietf-oauth-resource-metadata-06.html

The specification is available at:

https://www.ietf.org/archive/id/draft-ietf-oauth-resource-metadata-06.html

03:08 • 3 months ago

Monday, 08. July 2024

Just a Theory

Introducing Go SQL/JSON Path and Playground

Introducing the SQL/JSON Path package, a PostgresSQL-compatible jsonpath parser and executor in Go. Also: a Wasm-based playground!

For a personal project, I needed to parse and execute PostgreSQL-compatible jsonpath expressions.1 So I’ve spent just about every spare evening and weekend the last several months porting Postgres jsonpath to Go, and it’s finally ready to ship.

Introducing Go SQL/JSON, featuring the path package. This project provides full support for all of the PostgresSQL 17 jsonpath features2 in the Go programming language. An example:

package main

import (
	"context"
	"encoding/json"
	"fmt"
	"log"

	"github.com/theory/sqljson/path"
	"github.com/theory/sqljson/path/exec"
)

func main() {
	// Parse some JSON.
	var value any
	err := json.Unmarshal([]byte(`{"a":[1,2,3,4,5]}`), &value)
	if err != nil {
		log.Fatal(err)
	}

	// Parse a path expression and execute it on the JSON.
	p := path.MustParse("$.a[*] ? (@ >= $min && @ <= $max)")
	res, err := p.Query(
		context.Background(),
		value,
		exec.WithVars(exec.Vars{"min": float64(2), "max": float64(4)}),
	)
	if err != nil {
		log.Fatal(err)
	}

	fmt.Printf("%v\n", res)
    // Output: [2 3 4]
}

I think the API is decent, but may implement better patterns as I discover them. Overall I’m quite satisfied with how it turned out, and just how well its implementation and performance compare to the original.

🛝 Playground

But why stop there? One of the nice things about this project is that Go supports compiling applications into WebAssembly (a.k.a. Wasm) via Go WebAssembly. Borrowing from the Goldmark project, I created and published the sqljson/path playground and populated the docs with links for all of its examples.

Now anyone can experiment with SQL/JSON path expressions, and share links to demonstrate patterns and techniques. The Playground is a stateless JavaScript/Wasm web application: data persists only in permalink URLs.3

🛝 Try this example Playground permalink right now!4

The Path Ahead

I’ve enjoyed learning how to implement a lexer, a goyacc parser, an AST, and an execution engine. The Playground was a bonus bit of fun!

I’m stoked to build cool stuff on this package, but don’t know whether anyone else will find it useful. If you do — or just enjoy messing about on the Playground, let me know!

“Whatever for,” you ask? Well, aside from wanting to see if I could do it, this post describes a POC. Now I’m working to create the real thing — done right and entirely from scratch. ↩︎

Well, nearly full. The only missing feature is the datetime(template) method. See also the comprehensive compatibility notes. ↩︎

And whatever data GitHub Pages collect 😔. ↩︎

JSON borrowed from MDM. ↩︎

More about… Postgres SQL/JSON JSON Path Go Playground

13:59 • 4 months ago

Patrick Breyer

BKA: Höchststand an Falschverdächtigungen aufgrund der freiwilligen Chatkontrolle

Von den Chats, die mithilfe der “freiwilligen Chatkontrolle” der US-Konzerne (allen voran Meta) an die Polizei geleakt werden, ist ein so geringer Anteil wie noch nie tatsächlich strafrechtlich relevant – zuletzt sogar weniger als die Hälfte. Dies ergibt sich aus dem heute veröffentlichten „Bundeslagebild Sexualdelikte“ des BKA. Danach ist die Zahl der zumeist automatisiert an das BKA ausgeleiteten Chatverläufe von 2022 auf 2023 massiv gestiegen (von 136.450 auf 180.300 Chats), die Zahl der für strafrechtlich relevant befundenen Chatverläufe aber zurückgegangen (von 89.850 auf 89.350 Chats).

„Die Zerstörung unseres Briefgeheimnisses nimmt immer dramatischere Ausmaße an“, kritisiert der Europaabgeordenete der Piratenpartei Dr. Patrick Breyer. „Erstmals sind mehr als die Hälfte der ausgeleiteten Chats völlig legal. Noch nie sind so viele private und intime Familienfotos, Partnerfotos, Sexting durch Kinder usw. in Hände gelangt, in die sie nicht gehören und in denen sie nicht sicher sind. Dass die Bundesregierung diese freiwillige Chatkontrolle unterstützt, ist der völlig falsche Weg. Unsere Kinder werden durch die Chatkontrolle massenhaft kriminalisiert, anstatt sie zu schützen – selbst wenn sie aus eigenen Stücken Selbstaufnahmen verschicken.

Nach meinen Informationen ist die Ursache dieser Rekord-Unzuverlässigkeit, dass einige Plattformen angefangen haben, Textchats nach Schlüsselwörtern zu scannen, denen jeder Bezug zu Kindesmissbrauch fehlt. Mit den privaten Schnüffelalgorithmen werden Millionen verdient, die Hersteller lobbyieren aber gegen jegliche Zuverlässigkeitsanforderungen, Transparenz und unabhängige wissenschaftliche Überprüfung.
Die von der Bundesregierung unterstützte eigenmächtige Chatkontrolle 1.0 ist ein gefährlicher Irrweg der US-Konzerne. Sie konnte die Menge des ‚verdächtigen‘ Materials an den teilnehmenden Plattformen nie eindämmen. Die Entlastung der Polizei von der Flut zum großen Teil falscher Verdachtsmeldungen würde Kapazitäten für die verstärkte Verfolgung und verdeckte Ermittlungen gegen organisierten Kindesmissbrauchs freimachen, was Kinder wirklich schützt und die Hintermänner stoppt. Big Tech unterlässt wirklich wirksame Schutzmaßnahmen durch sichere Gestaltung sozialer Netzwerke (Security by Design), weil sie ihren Profiten schaden würden. Warum werden Nutzer etwa nicht durchgängig und altersunabhängig gefragt, bevor ihre Fotos öffentlich sichtbar sind und sie für Fremde ansprechbar sind? Naheliegende Schutzmaßnahmen vor sexueller Annäherung sind wirksamer als jede ungezielte Massenüberwachung!”

Breyer klagt aktuell gegen die freiwillige Chatkontrolle vor dem Oberlandesgericht Schleswig. Auch die Klage eines Missbrauchsbetroffenen ist anhängig.

13:27 • 4 months ago

Sunday, 07. July 2024

Jon Udell

Seymour and Brownie

My family, on my dad’s side, were Jews from Poland and Ukraine. His parents came to America before the shit hit the fan, but I grew up knowing two people who weren’t so lucky. Seymour Mayer lived across the street during my teens. And Annie Braunschweig, who we knew as Brownie, had taken care of my sister and me as four- and five-year-old kids when our mom – unusually at that time – went back to work full-time teaching at a university. Both Seymour and Brownie were survivors of Nazi concentration camps, with tattooed numbers on their arms.

I never heard Seymour talk about it. Brownie rarely did, though I remember one story about a mother who tossed her swaddled baby to a stranger as the train was leaving to take her to the gas chambers.

Very few survivors remain. And there are not many of us who have known survivors. I’ve thought a lot, over the years, about what happens when that kind of personal connection ends, and living memories fall off the continental shelf into the deep ocean of history. I suspect the Holocaust may seem no more real, to many born in this century, than the Spanish Inquisition.

I don’t know if Seymour and Brownie ever read “It Can’t Happen Here” but I am pretty sure they’d have thought it absolutely can, they’d be even more horrified in this moment than many of us are, and they’d reject the fatalism that I see taking root among friends and acquaintances.

“It hasn’t happened yet,” they’d say, “you can still prevent it, do not despair prematurely, there is still time, but you must find a way to focus your efforts and unite all whose votes can matter.”

23:12 • 4 months ago

Saturday, 06. July 2024

Wrench in the Gears

The Undivided Universe of the Heart

That little voice whispered it was time to get out of the apartment and clear my head. I’d been working on additions to a map for several days, and it was going slower than I wanted it to go. My mother, in her 80s, moved into assisted living last month. As is to be expected, all the changes have been difficult for her. Compounding that, our communication lines seem to be experiencing a lot of psychic static; an ongoing family curse. So I’m trying to navigate that even as my own circumstances are still unsettled, at least for another month.

How does one undertake the responsibilities of a good daughter, when for some reason the gate to a mother’s heart continues to malfunction? Can you insert yourself into someone’s life if all signs indicate you’re not really welcome? Maybe not yet. Maybe things have to get worse before resolution is possible. When does the child take on the role of the mother? It doesn’t feel like a role that can be stepped into gracefully. Plus, I don’t have anyone physically here in my corner to lean on, which is not what I expected at this stage of the game. I guess I should see it as an opportunity to exercise my self-reliance muscles.

Despite offers of assistance, none were warmly welcomed. So I’m left with a lot of question marks. I send flowers and prayers across several thousand miles and try to come to terms with my situation. A friend found a small medallion in a parking lot with the inscription “Family Is Everything” only to later realize that it was a Disney branded trinket, which put a new spin on the pithy trope. No, not everything.

Every family comes with its own lessons, and oh those morphogenetic fields can be a bear. Perhaps the best thing is to offer what you can with an open, tender, and decidedly imperfect heart and let the pieces fall where they may. I think we do this more than once. Maybe my mother and I have done it a hundred times already. Maybe this round I fared better than in the last one, and in the next I’ll make more progress. It’s the journey, right? Not the destination…

So I drove across town to my rented PO box to pick up a book and some linen fabric I’d treated myself to. The colors are gray cashmere and Montana grape, which will make a lovely skirt and blouse for fall. I’m looking forward to pulling out my sewing machine and embroidery floss again. The purchase, which was spurred on by an emailed discount code, was my way of staking a claim on that future. The book was Bohm and Hiley’s “The Undivided Universe.”

Impatient and curious, I flipped through the pages on quantum consciousness in the parking lot as I placed the packages in the trunk and blanched at the equations therein. I really am hopeless with math and physics, but hopefully I can glean some insights on active information and the meaning of our oneness. As much as my mother exasperates me, I love her and I suspect she loves me too, just as I love my prodigal child. The signals are choppy – entangled human weather in the swirling gyre.

My inner voice said take the growing collection of feathers on the shelf above the desk here in the apartment. There is a strange giant gold hand with some books around it that clearly came from the decorator, books by the yard, not ones that were actually ever read by anyone. In the hand I’d placed a sprig of mistletoe that fell out onto the windshield of my car from the towering willow oaks that line the street where I’m staying. I have a small pile of stones, pinecones, and other bits and bobs. I put most of the feathers and a few of the stones in a soul-bright yellow canvas bag my friend Juliana entrusted to me during our Lititz foray. I decided on my way back I’d stop at “Space Odyssey” Allsopp Park and make a heartfelt intention for my mom during this time of her bumpy transition.

I think my hearts confound some people. They don’t know how to integrate the Alison that makes maps about inscrutable things with the Alison who walks around making patterns with acorns and leaves. I think some people think I ought to be embarrassed, perhaps not that I make them, but that I share them publicly.

Even though I used to consider myself a progressive, I began to see that I’m really more spiritually-minded than materialist inclined. I have my reasons for being skeptical of institutional faith communities, but I do feel the pull towards a benevolent creator. The gift of sacred cyclical beauty in nature is part of that. A woman on Facebook inspired my hearts. She posted photos spelling out “love” in sticks and stones. During that terrible period of the lockdowns, it felt like a meaningful affirmation and it still does.

If there is an implicate order, why not seek to communicate with it? For me, making hearts is like a guided meditation. I’m not a good meditator in the regular sense. My wheelhouse is symbol and story. For me, the practice works, and I share it, because it may work for other people too. Someone gifted me a version of their prayer, and I made it my own, and so may it be for others if it makes sense in your life. If not, feel free to immerse yourself in my maps instead. They are both valuable sides of the complex pattern that is Alison McDowell this time around.

I was still working at the garden during Covid, so I had ample materials to work with. Eventually, I decided laying out a heart shape was preferable to using words. Ever since I’ve made 3-4 hearts a month with nature’s findings. Sometimes I do it to neutralize fraught energy in a place I’ve researched, but not always. Sometimes I simply do it as a way to reaffirm my place in the world. With each I offer an intention for “life, health, and open paths” that Cliff Gomes taught me.

At first I hoarded materials, thinking that was what was needed; but with time, I realized the gift of a public heart is a reflection of a given place. Rather than bringing new materials to a spot, it was better to seek out the message from the location itself. What communication awaits? The trickiest part of heart-making is finding materials to outline the shape. From there, you can decorate inside and outside with a smattering of this or that.

When I got out of the car with my little yellow sack, I first saw a bountiful native pollinator garden full of sunflowers and bee balm and ironweed, a towering presence with deep purple flowers that I remember standing guard over the lower garden in high summer. I didn’t pick any of them though. The bees and wasps were enjoying them too much. I turned to head down the path and saw a giant wild wisteria vine clambering into a tree. The blossoms were spent, but there were a dozen or so soft, green velvet seed pods. That’s what I picked for the outline of today’s heart.

Wisteria is named for the Wistar family of Philadelphia, peers of John Bartram, glass makers, and doctors. In the late nineteenth-century the Wistar Institute, which is located on the campus of the University of Pennsylvania, became one of the first biomedical research labs in the world with a focus on vaccines. The first standardized experimental lab rat was created there, the Wistar rat.

Next, some ferny wild carrot tops caught my eye. I didn’t harvest any, but a wave of remembrance came over me for the summers of my childhood where I played in the dirt lots and tree lines of corporate suburbs in the making. Those summers in the late 1970s and early 1980s where carrot beings anchored themselves into the imaginative world building exercises of feral youth. Even as my adult world has fallen apart, that broken-ness gifted me the chance to renew my connection with the younger version of Alison, the one who would jump the fence onto a neighboring farm to walk the creek and pretend she lived in Narnia.

I left the pavement and took the high route up the wooded hillside. At the bottom I scooped up a few small pieces of bark from a fallen branch that was coated with a bluish-green lichen, still damp from the thunderstorm that rolled in this morning after weeks of dry heat. The storm knocked the dust down and you could still smell the earthiness of the soil.

The walk up was uneven and rocky. “We are shapes of the Earth” as John (or Tina) Trudell would say. Their refrain came to me as my sandals found footing among the loose stones. A few slices found their way into my bag, including a shard pierced by a round hole the size of a quarter. The path affirmed our mineral connection. Gray slate of ancient sea beds pitched vertically with cataclysmic force; the sheets splayed out like a library of plundered books with bindings broken, quires unmoored. The consciousness of those stones spoke my language of order lost to chaos, but with elegance and meaning, undertones of their present disheveled arrangement.

At the top of the hill was a signpost. I took a turn to the right and followed along the edge of a small ravine to where the Nandina grew. I thought about the Hot Springs park ranger berating the shrub for being an interloper, but it has a special place in my heart. After I left home, my parents moved into a house with an elaborately landscaped yard. There was a hedge of Nandina and I dug up a sprout and brought it home with me to Philadelphia where it flourished in our small patio garden. I would have to cut it back pretty vigorously every year and it never flowered like it did in the South, but it reminded me of my dad, so of course I grabbed a sprig and put it on the bag.

I also saw three purple mist flowers, blue boneset. I love purple, so I put them in, too. There were also huge stands of poke on the other side of the path. Their flower panicles were in just the early stages of white and green, before the intense purple berries arrive. The white, pink, and green pokeweed always reminds me of the preppy era when pink and green attire was all the rage in middle school.

During my walk, the woods were infused with R&B music from a distant picnic pavilion. When I pulled into the parking lot, I noticed a white pick-up truck towing a huge barbeque grill. There were a bunch of families having a cookout, which made me think about the big gatherings you see on summer weekends in Fairmount Park. It made me a tiny bit homesick, but in a good way.

When I used to live near John Coltrane’s Philadelphia house on 33rd Street, they had papers in the windows with the saying “one thought can create millions of vibrations and they all go back to God.” So, I go into nature with questions and an inner knowing that on the other side at least it will be ok; we will all be ok.

I can make an offering to try and harmonize the present moment of uncertainty. It may look like a milky quartz stone surrounded by lichen and feathers and pinecones and wisteria pods with a smattering of purple, because that’s what I like. That’s what little Ally liked, too. And we can weave a story of love even if the creaky lock is stuck, because one day it will open. The vibrations are slipping under the threshold and loosening the resistance. The path is open. We just need to find our way to the trailhead.

I’m not a nihilist. I’m studying the quantum and ontopoesis and the participatory universe, because I know that love finds a way. Meg knew that. Patience isn’t my strong suit, but this journey may carry across dimensions and I have to learn to endure and cultivate a sense of humor in the absurdity of it all.

Spoiler alert – love wins!

PS: I didn’t take my camera with me, so this heart will just have to live in your imagination.

01:51 • 4 months ago

Thursday, 04. July 2024

Werdmüller on Medium

The America I love

It’s not loud, but it exists. Continue reading on Medium »

It’s not loud, but it exists.